Ensuring Validity and Reliability in Accreditation Decision Making Slide 1
ACCME Policy Maintenance of internal consistency must remain a high organizational priority. Adopted November 2011 Slide 2
Our Challenge Our Responsibility Slide 3
Our Challenge Our Responsibility Not Arbitrary Reliability (Reproducibility) Score Would be the tomorrow as received today. Result of true differences between providers. Validity (Accuracy) Content Face Predictive Construct Concurrent Slide 4
The ACCME s Layered Strategies to Maximize Validity (Accuracy) and Reliability (Reproducibility) of Accreditation Decisions 1. A foundation of enterprise ethics and values. 2. Published rules and processes uniformly applied. 3. Outcomes rules based. Enterprise Values and Ethics Fair Valid Reliable Rules & Process Published Uniform Outcomes Criterion Internal Referenced Oversight The best care for everyone, every time! Slide 5
The ACCME s Strategies to Maximize Validity (Accuracy) and Reliability (Reproducibility) of Accreditation Decisions CORPORATE: A foundation of enterprise ethics and values 1. Fair 2. Consistent treatment of providers a standardized process 3. Professional staff (consistency, tenure, numbers) Slide 6
The ACCME s Strategies to Maximize Validity (Accuracy) and Reliability (Reproducibility) of Accreditation Decisions RULES Published Rules and processes uniformly applied. 1. One set of requirements for all providers (ex., Compliance Criteria, SCS) 2. Interpretations and explanations published on web site 3. Consensus process for Common interpretation of what compliance is @ margins 4. Designated staff responsible for communications internally (staff, ARC, surveyors) and externally (CME community) 5. Common understanding between staff responsible for communications internally (staff, ARC, surveyors) and externally (CME community) Slide 7
The ACCME s Strategies to Maximize Validity (Accuracy) and Reliability (Reproducibility) of Accreditation Decisions PROCESS INTERPRETATION OUTCOMES Standardization of providers experience during accreditation process Standardization of incoming database from Providers Validation and verification of information going to surveyors Within survey By Surveyors Between survey and ARC meeting At the ARC Staff Review after ARC but before DC At DC and BOARD After DC and Board Data Management consistency measures Slide 8
An example.. Findings from SSR congruent with Findings from PIP Combined opinion of SSR congruent with ARC Reviewer s opinion of SSR Provider s practices presented to the ARC by ARC reviewer to explain findings. Group mediated criterion referenced recommendation, with staff oversight Preliminary findings accurately describe provider s practices 100% of SSR, SRF and DRF S are reviewed by ARC Reviewer. 100% of ARC Reviewer s findings at the Criteria level checked for consistency w Interviewers aggregated findings. In ~10% of providers (~6 of 61), the interviewers and ARC reviewer were not in agreement on a finding on a Criterion In 33% of providers (18 of 54), the interviewers and ARC reviewer were not in agreement on a finding on a Criterion Disagreements were resolved through evidence based (referenced to ACCME written policy or ARC s manner of acting) reconciliation by staff in consultation with interviewers +/ after getting more information and/or by the ARC reviewer or the ARC at a later stage in the process. The ARC committee process altered an additional 56 (4%) of the 1,342 findings presented by ARC reviewers, By staff in files ~14 of 1,342 By ARC reviewer 6 of 1,342 By ARC 56 of 1,342 76 (5.6%) The ARC committee process altered 8 of the 61 provider s outcome recommendations (13%) presented by ARC members. Summary The ARC committee process altered an additional 28 (2%) of the 1,188 findings presented by ARC reviewers, By staff in files 11 of 1,188 By ARC reviewer 18 of 1,188 By ARC 28 of 1,188 57 (4.7%) The ARC committee process altered 7 of the 54 provider s outcome recommendations (13%) presented by ARC members. Slide 9
Can reliably distinguish between providers. Slide 10
Can drive improvement. Slide 11
Can drive improvement. Can recognize failure to improve. Slide 12
Variability in Outcomes Accreditation Decisions Nov. 2008 to Nov. 2012 (n=728) Accreditation with Commendation Nov. 2012 (n=54) 26% Probation Nonaccreditation Provisional Accreditation with Progress Report Accreditation 28% 4% 31% 11% 0 Slide 13
ACCME Policy Maintenance of internal consistency must remain a high organizational priority. Adopted November 2011 Slide 14
Ensuring Validity and Reliability in Accreditation Decision Making Slide 15
Thank you Slide 16