Examining Inter-Rater Reliability of a CMH Needs Assessment measure in Ontario

Transcription

1 Examining Inter-Rater Reliability of a CH Needs Assessment measure in Ontario CAHSPR, Halifax, ay 2011 Team: Janet Durbin, Elizabeth Lin, Carolyn Dewa, Brenda Finlayson, Stephen Gallant, April Collins CAH: Health Systems Research and Consulting Unit

2 Background Ontario implementing standardized client level assessment of need in community mental health organizations since 2008 Benefits Client care: standardization in reviewed needs, measurement; feedback; integration System - data repository for planning, research easure: Camberwell assessment of need plus other items About the Camberwell (CAN) Developed in 1995 (UK) (Phelan, 1995) Focus on persons with serious mental illness Holistic assessment, recovery philosophy 22 life domains Consumer and staff versions Large body of international research on reliability and validity, positive results for research applications (Hansson, 1995; Nielsen, 1999; Trauer, 2004; Fleury, 2006)

3 Camberwell Domains (24) ental health & risk Basic needs & functioning Social Psychological distress Accommodation quality Company Psychotic symptoms Benefits Intimate relationships Safety to self Self-care Sexual expression Safety to others Alcohol use Provide food Looking after home Drug use anage money Other Other addictions Telephone Information on Tx Physical health problems Education (literacy) Transport Child Care Other Dependents Daytime Activities Rating: 0 = no need; 1 = met need; 2 = unmet need; 9 = DK

4 Ontario Implementation 320 CH organizations - multiple programs & providers per org about 80% of orgs eligible to implement assessment about 40-50% of these are in process Provincial program responsible for implementation provide standardized training manuals, centralized training team, train-the-trainers Question achieving adequate data quality under current field conditions?

5 Reliability Study Studies support reliability of Camberwell ratings in research applications; less known about performance during routine care delivery Study Goals: 1. Assess potential for staff from different organizations to achieve consistent ratings of need (Camberwell) when presented with the same consumer information (staff assessment) 2. Assess how training influenced ratings

6 ethodology Data collection/sample: Tried to approximate data collection during routine care delivery (acdonald & Trauer, 2010) Developed 4 mock staff-consumer interviews, video-taped and mounted on web Invited trained program staff (26 orgs) to complete Camberwell need ratings for one of the interviews (random) Asked trainers to complete the ratings for all 4 interviews (gold standards) Data collection occurred during fall-winter

7 ethodology Analysis a. Consistency extent to which participants agree with each other on need for care b. Accuracy extent to which ratings agreed with expected responses (e.g., trainer goals) For each Camberwell item and case: a. Calculated % agreement, Kappa* b. Rated agreement as good, moderate, fair, slight (Landis & Koch, 1977) *Kappas selected to take uneven distribution of need across 4 cases into account (K free K m )

8 Kappa Interpretation (Landis & Koch, 1977) Kappa Value Level of agreement beyond chance <0 No agreement 0.0 to 0.20 Slight 0.21 to 0.40 Fair 0.41 to 0.60 oderate 0.61 to 0.80 Substantial 0.81 to 1.00 Almost perfect agreement

9 Sample Final sample (convenience) 123 participants out of 460 invited staff (27%) Program type 50% - case management; 50% - other Experience 1/3 with more than 10 years; 15% with < 2 years Education College 35%; University 65% Training provider 85% - in agency

10 Consistency Results (kappas) Safety to Self S Safety to Others S ental health and Alcohol S risk needs Drugs S Psychotic Symptoms Psychological Distress F Other Addictions F Physical Health F Child Care S Other Dependents S Education S Food Basic needs and Looking After Home Self-Care functioning oney Accommodation Telephone Daytime Activities Public Transport F Benefits N/A Company F Social needs Intimate relationships F Sexual Expression F Other Information on condition F 10

11 Accuracy Results Expert Trainer Agreement 79 out of 96 ratings Low - safety to self, benefits Accuracy (Kappas) Similar to consistency rating results Accuracy lower in 3 domains

12 Positive: Summary Reasonable reliability can be achieved Good performance in most mental health and risk domains, and in basic needs and functioning; more difficulty in social needs Rating results aligned with training intentions QI opportunity: Lower ratings may reflect: Domain & rating definitions double barreled, what is included Staff inexperience in rating social areas & lack of clear treatment response Defining divide between met & unmet need Explore need for training tailored to rater background, expectations

13 Limitations Summary Convenience sample study shows what is possible, not a measure of system performance Video format no opportunities to explore issues Certain domains lack variance across videos Source of rating variation not tested - e.g., train the trainers, background of raters, video case format Future Directions Training remains a priority Continued review to ensure clear definitions & messaging, support materials - domain intent document Awaiting CAN update

14 Thank you! Full report available: Provincial initiative contact: Appreciation to Ontario ental Health Foundation Participating community programs and staff Provincial program for CAN implementation