Measuring clinical impact: methodological issues and challenges

Transcription

1 Measuring clinical impact: methodological issues and challenges Panel discussion Samuel G. Schumacher McGill University

2 QUICK POLL 2

3 Question X? 3

4 Data on impact? We need studies on impact! Data on test characteristics and implementation setting are sufficient! 4

5 Final Outcomes? We need to look at final outcomes! Looking at key intermediate outcomes is sufficient! 5

6 RCTs? We need RCTs! Doing a mix of accuracy, modelling and OR is sufficient! 6

7 Settings? Doing a few large studies representative of major epidemiological settings is sufficient! We need many small operational research studies! 7

8 Dx or Dx+? We need to assess the combined impact of diagnostics and downstream effects! We should look at the impact of diagnostics, separately from the influence of downstream effects! 8

9 What are the challenges? 9 Ramsay 2010, Cobelens 2012, Lin 2012

10 What are the challenges? 1. A major obstacle to conducting impact assessments is a lack of consensus on what impact really means, and what patient-, population-, health systems-, and epidemiology-important outcomes should be measured to decide on impact. 9 Ramsay 2010, Cobelens 2012, Lin 2012

11 What are the challenges? 1. A major obstacle to conducting impact assessments is a lack of consensus on what impact really means, and what patient-, population-, health systems-, and epidemiology-important outcomes should be measured to decide on impact. 2. There is also a lack of guidance on the methods (i.e. study designs) to be used to measure them, which methods will be most rapid and cost-effective 9 Ramsay 2010, Cobelens 2012, Lin 2012

12 What are the challenges? 1. A major obstacle to conducting impact assessments is a lack of consensus on what impact really means, and what patient-, population-, health systems-, and epidemiology-important outcomes should be measured to decide on impact. 2. There is also a lack of guidance on the methods (i.e. study designs) to be used to measure them, which methods will be most rapid and cost-effective 3. [ ] the impact of a new, more accurate diagnostic tool on tuberculosis epidemiology [is] substantially affected by contextual factors unrelated to tool performance. 9 Ramsay 2010, Cobelens 2012, Lin 2012

13 Challenges & Lenses! 1. What is impact / which outcomes? 2. How to study it / which designs? 3. How to consider context? 1. test developer! 2. researcher! - primary research - systematic reviews 3. policy maker / guideline developer! - global - country level 4. Funders! 10

14 Challenges & Lenses! 1. What is impact / which outcomes? 2. How to study it / which designs? 3. How to consider context? 1. test developer! 2. researcher! - primary research - systematic reviews 3. policy maker / guideline developer! - global - country level 4. Funders! 11

15 1. What is impact / which outcomes? 12

16 What do other say? A quick look beyond the TB world 13

17 What do other say? A quick look beyond the TB world World Bank 13

18 What do other say? A quick look beyond the TB world World Bank UPSTF / AHRQ 13

19 What do other say? A quick look beyond the TB world World Bank UPSTF / AHRQ Epidemiology/ Computer Science 13

20 WORLD BANK INDEPENDENT EVALUATION GROUP E VALUATION CAPACITY DEVELOPMENT What do other say? A quick look beyond the TB world World Bank Interactive textbook at Impact Evaluation in Practice Paul J. Gertler, Sebastian Martinez, Patrick Premand, Laura B. Rawlings, Christel M. J. Vermeersch IMPACT EVALUATION- THE EXPERIENCE OF THE INDEPENDENT EVALUATION GROUP OF THE WORLD BANK UPSTF / AHRQ Epidemiology/ Computer Science 13

21 WORLD BANK INDEPENDENT EVALUATION GROUP E VALUATION CAPACITY DEVELOPMENT What do other say? A quick look beyond the TB world World Bank Interactive textbook at Impact Evaluation in Practice Paul J. Gertler, Sebastian Martinez, Patrick Premand, Laura B. Rawlings, Christel M. J. Vermeersch IMPACT EVALUATION- THE EXPERIENCE OF THE INDEPENDENT EVALUATION GROUP OF THE WORLD BANK UPSTF / AHRQ Figure 2 1. Application of USPSTF analytic framework to test evaluation * Epidemiology/ Computer Science 13

22 WORLD BANK INDEPENDENT EVALUATION GROUP E VALUATION CAPACITY DEVELOPMENT What do other say? A quick look beyond the TB world World Bank Interactive textbook at Impact Evaluation in Practice Paul J. Gertler, Sebastian Martinez, Patrick Premand, Laura B. Rawlings, Christel M. J. Vermeersch IMPACT EVALUATION- THE EXPERIENCE OF THE INDEPENDENT EVALUATION GROUP OF THE WORLD BANK UPSTF / AHRQ Figure 2 1. Application of USPSTF analytic framework to test evaluation * Epidemiology/ Computer Science 13

23 How to Assess Impact 3 important points 14

24 How to Assess Impact 3 important points 1. impact = causal effect on final outcomes comparison group (risk of bias) extrapolation (indirectness) 14

25 How to Assess Impact 3 important points 1. impact = causal effect on final outcomes comparison group (risk of bias) extrapolation (indirectness) 2. importance of mapping out the causal chain! logical framework connecting intermediate (surrogate) to final outcomes 14

26 How to Assess Impact 3 important points 1. impact = causal effect on final outcomes comparison group (risk of bias) extrapolation (indirectness) 2. importance of mapping out the causal chain! logical framework connecting intermediate (surrogate) to final outcomes 3. importance of context! need to understand reasons for heterogeneity 14

27 2. How to study it / which designs? 15

28 Study Design Examples RCTs 16

29 Study Design Examples Quasi-experimental studies Feasibility, diagnostic accuracy, and effectiveness of decentralised use of the Xpert MTB/RIF test for diagnosis of tuberculosis and multidrug resistance: a multicentre implementation study Catharina C Boehme, Mark P Nicol, Pamela Nabeta, Joy S Michael, Eduardo Gotuzzo, Rasim Tahirli, Ma Tarcela Gler, Robert Blakemore, William Worodria, Christen Gray, Laurence Huang, Tatiana Caceres, Rafail Mehdiyev, Lawrence Raymond, Andrew Whitelaw, Kalaiselvan Sagadevan, Heather Alexander, Heidi Albert, Frank Cobelens, Helen Cox, David Alland, Mark D Perkins 17

30 Study Design Examples Studies of clinical decision-making INT J TUBERC LUNG DIS 15(5): The Union doi: /ijtld Does solid culture for tuberculosis influence clinical decision making in India? N. Stall,* T. Rubin, J. S. Michael, D. Mathai, O. C. Abraham, P. Mathews, K. Thomas, M. John,* P. Daley * Schulich School of Medicine & Dentistry, University of Western Ontario, London, Ontario, Faculty of Medicine, Use of Drug-Susceptibility Testing for Management of Drug-Resistant Tuberculosis, Thailand, Eugene Lam, Sriprapa Nateniyom, Sara Whitehead, Amornrat Anuwatnonthakate, Patama Monkongdee, Apiratee Kanphukiew, Jiraphan Inyaphong, Wanlaya Sitti, Navarat Chiengsorn, Saiyud Moolphate, Suporn Kavinum, Narin Suriyon, Pranom Limsomboon, Junya Danyutapolchai, Chalinthorn Sinthuwattanawibool, and Laura Jean Podewils INT J TUBERC LUNG DIS 15(3): The Union SHORT COMMUNICATION Implementation of liquid culture for tuberculosis diagnosis in a remote setting: lessons learned P. Hepple,* J. Novoa-Cain, C. Cheruiyot, E. Richter, K. Ritmeijer *Médecins Sans Frontières, Manson Unit, London, UK; Médecins Sans Frontières Operational Centre Amsterdam ORIGINAL STUDIES Effect of Nucleic Acid Amplification for Mycobacterium tuberculosis on Clinical Decision Making in Suspected Extrapulmonary Tuberculosis* Renda Soylemez Wiener, MD; Phyllis Della-Latta, PhD; and Neil W. Schluger, MD, FCCP Daphne I. Ling, MPH,* Claire A. Crépeau, BScN, Marieke Dufresne, RN, Shazia Khan, BA, Caroline Quach, MD, MSc,* Nandini Dendukuri, PhD, Kevin Schwartzman, MD, MPH,* Dick Menzies, MD, MSc,* Larry C. Lands, MD, PhD, and Madhukar Pai, MD, PhD* 18

31 Study Design Examples uncontrolled/hypothetical/model-based 19

32 3. How to consider context? 20

33 Diagnostic Causal Chain & Spectrum of diagnostic research Test performance Clinical impact Patient impact decreased turn-around-time decreased time to Dx Speed reduced # of visits before TB Dx reduced % initial default Faster appropriate ATT Improved TB Dx test Rule-in cases where TB is correctly ruled-in confirmed TB-patients put on ATT - Improved treatmentoutcomes - Reduced morbidity - Reduced mortality increased accuracy reduced # of invasive diagnostic procedures Rule-out cases where TB is correctly ruled-out correct non-tb diagnoses made 21 reduced # of over-diagnoses and over-treatment

34 Diagnostic Causal Chain & Spectrum of diagnostic research Test performance Clinical impact Patient impact decreased turn-around-time decreased time to Dx Speed reduced # of visits before TB Dx reduced % initial default Faster appropriate ATT Improved TB Dx test Rule-in cases where TB is correctly ruled-in confirmed TB-patients put on ATT - Improved treatmentoutcomes - Reduced morbidity - Reduced mortality increased accuracy reduced # of invasive diagnostic procedures empiric ATT Rule-out cases where TB is correctly ruled-out correct non-tb diagnoses made 21 reduced # of over-diagnoses and over-treatment

35 Diagnostic Causal Chain & Spectrum of diagnostic research Test performance Clinical impact Patient impact decreased turn-around-time decreased time to Dx Speed reduced # of visits before TB Dx reduced % initial default Faster appropriate ATT Improved TB Dx test empiric ATT; little ZN-neg. TB Rule-in cases where TB is correctly ruled-in confirmed TB-patients put on ATT - Improved treatmentoutcomes - Reduced morbidity - Reduced mortality increased accuracy reduced # of invasive diagnostic procedures empiric ATT Rule-out cases where TB is correctly ruled-out correct non-tb diagnoses made 21 reduced # of over-diagnoses and over-treatment

36 Diagnostic Causal Chain & Spectrum of diagnostic research Test performance Clinical impact Patient impact decreased turn-around-time decreased time to Dx Speed reduced # of visits before TB Dx reduced % initial default Faster appropriate ATT Improved TB Dx test empiric ATT; little ZN-neg. TB Rule-in LTFU high cases where TB is correctly ruled-in confirmed TB-patients put on ATT - Improved treatmentoutcomes - Reduced morbidity - Reduced mortality increased accuracy reduced # of invasive diagnostic procedures empiric ATT Rule-out cases where TB is correctly ruled-out correct non-tb diagnoses made 21 reduced # of over-diagnoses and over-treatment

37 Diagnostic Causal Chain & Spectrum of diagnostic research Test performance Clinical impact Patient impact decreased turn-around-time decreased time to Dx Speed reduced # of visits before TB Dx reduced % initial default Faster appropriate ATT Improved TB Dx test empiric ATT; little ZN-neg. TB Rule-in LTFU high cases where TB is correctly ruled-in stock-outs confirmed TB-patients put on ATT - Improved treatmentoutcomes - Reduced morbidity - Reduced mortality increased accuracy reduced # of invasive diagnostic procedures empiric ATT Rule-out cases where TB is correctly ruled-out correct non-tb diagnoses made 21 reduced # of over-diagnoses and over-treatment

38 Diagnostic Causal Chain & Spectrum of diagnostic research Test performance Clinical impact Patient impact decreased turn-around-time decreased time to Dx Speed reduced # of visits before TB Dx reduced % initial default Faster appropriate ATT Improved TB Dx test Rule-in LTFU high cases where TB is correctly ruled-in stock-outs ART empiric ATT; little ZN-neg. TB confirmed TB-patients put on ATT - Improved treatmentoutcomes - Reduced morbidity - Reduced mortality increased accuracy reduced # of invasive diagnostic procedures empiric ATT Rule-out cases where TB is correctly ruled-out correct non-tb diagnoses made 21 reduced # of over-diagnoses and over-treatment

39 Diagnostic Causal Chain & Spectrum of diagnostic research logistical issues Test performance Clinical impact Patient impact decreased turn-around-time decreased time to Dx Speed reduced # of visits before TB Dx reduced % initial default Faster appropriate ATT Improved TB Dx test Rule-in LTFU high cases where TB is correctly ruled-in stock-outs ART empiric ATT; little ZN-neg. TB confirmed TB-patients put on ATT - Improved treatmentoutcomes - Reduced morbidity - Reduced mortality increased accuracy reduced # of invasive diagnostic procedures empiric ATT Rule-out cases where TB is correctly ruled-out correct non-tb diagnoses made 21 reduced # of over-diagnoses and over-treatment

40 Diagnostic Causal Chain & Spectrum of diagnostic research logistical issues Test performance Clinical impact Patient impact decreased turn-around-time decreased time to Dx LTFU very low Speed reduced # of visits before TB Dx reduced % initial default Faster appropriate ATT Improved TB Dx test Rule-in LTFU high cases where TB is correctly ruled-in stock-outs ART empiric ATT; little ZN-neg. TB confirmed TB-patients put on ATT - Improved treatmentoutcomes - Reduced morbidity - Reduced mortality increased accuracy reduced # of invasive diagnostic procedures empiric ATT Rule-out cases where TB is correctly ruled-out correct non-tb diagnoses made 21 reduced # of over-diagnoses and over-treatment

41 Panel Discussion 22

42 Panel Members Adithya Cattamanchi UCSF Claudia Denkinger FIND Afrânio Kritski Universidade Federal do Rio de Janeiro Andrew Ramsay TDR Grant Theron UCT 23

43 Challenges & Lenses! 1. What is impact / which outcomes? 2. How to study it / which designs? 3. How to consider context? 1. test developer! 2. researcher! - primary research - systematic reviews 3. policy maker / guideline developer! - global - country level 4. Funders! 24