Playing By The Rules:

Transcription

1 Playing By The Rules: Who Decides Flagging Criteria For Outcomes and Why? SRTR Perspective Jon J. Snyder, PhD Director of Transplant Epidemiology Scientific Registry of Transplant Recipients Minneapolis Medical Research Foundation Minneapolis, Minnesota, USA Transplant Quality Institute Denver, CO 1

2 Who Sets Flag Boundaries? Short Answers: The OPTN s Membership and Professional Standards Committee (MPSC) sets the flag criteria for the OPTN. CMS s Survey and Certification team sets the flag criteria for CMS. SRTR performs outcomes evaluations for all transplant programs semiannually. SRTR provides the evaluations to the MPSC and CMS semiannually. SRTR provides a report to each transplant program indicating whether they meet either the MPSC or the CMS flag criteria. SRTR does not publicly report which programs meet either set of flag criteria. 2

3 Using Bayesian Assessments to Determine Performance Thresholds Bayesian models allow us to estimate the probability distribution for the performance of a particular program, which can be compared to identified thresholds or national norms Christiansen CL, Morris CN. Ann Intern Med. 1997;127:

4 Example PSR: First Year Survival With a Functioning Kidney Transplant 4

5 With a screening test, we can never be certain which programs are truly underperforming and which might be false positives (just a run of bad luck). 5

6 Moving the screening line lower = More True Positives and More False Positives More True Positives, but More False Positives 6

7 Moving the screening line higher = Less True Positives and Less False Positives Less False Positives, and Less True Positives 7

8 Designing the Flagging System in 2013 Consideration MPSC Decision in 2013 What is an acceptable false positive rate? 5% Should the false positive rate be associated with program volume? Should small volume programs have different false positive flagging rates than larger volume programs? What is the cost-benefit ratio when considering incorrectly flagging a program vs. missing an underperforming program? Should the algorithm try to do something to continue to identify underperformance in small programs without sacrificing much of the false positive rate? No Falsely flagging a program is 5 times worse than missing an underperforming program. Yes, the committee was concerned about pediatric programs which are generally small. 8

9 Simulating the Flagging System SRTR ran numerous simulations to find the optimal flagging system within the constraints the MPSC provided. Why simulate? When looking at real data, we do not know which programs truly have problems and which are false positives. In a simulation, we know precisely which programs are underperforming. The simulation then determines the true positive and false positive rates. SRTR simulated ~60,000 flagging systems. Each received a score based on their false positive and true positive rates. 9

10 Results of the ~60,000 Simulations. Previous MPSC 10

11 The Optimal Bayesian Flag Algorithm More than 75% of the bell would need to be to the right of 1.2 or more than 10% of the bell would need to be to the right of

12 The New Flag Boundary Achieved the 5% False Positive Goal 12

13 Three Example Program Evaluations with MPSC Evaluation Criteria Cutoffs 13

14 The Current MPSC Flag Space 14

15 The CMS Flag Space: Standard-Level Deficiency 15

16 The CMS Flag Space: Condition-Level Deficiency (current PSR and 1 of previous 4 PSRs) O/E=

17 Which Programs Should Be Reviewed? 17

18 MPSC / CMS Flag Spaces with Programs 18

19 Summary SRTR evaluates performance, but MPSC and CMS set their flag boundaries to meet their needs. SRTR recommends a search for an optimal flag boundary to best meet the stated goals of the review system, i.e., simulation studies to optimize true positives while holding false positives to an acceptable level. The regulatory agencies can tailor their review processes in conjunction with the flag boundary: If the boundary is willing to accept more false positives: less punitive review. If the boundary is willing to accept fewer false positives: more stringent review. 19

20 Playing By The Rules: Who Decides Flagging Criteria For Outcomes and Why? SRTR Perspective Thanks Jon J. Snyder, PhD Transplant Quality Institute Denver, CO 20