ehealth EVALUATION AND DISSEMINATION RESEARCH

Transcription

1 ehealth EVALUATION AND DISSEMINATION RESEARCH Russ Glasgow, Ph.D. Kaiser Permanente Colorado Critical Issues in ehealth Research Conference June 10, 2005

2 ACKNOWLEDGEMENTS KPCO Colleagues RE-AIM Colleagues Liz Bayliss, M.D. Bridget Gaglio, MPH Marleah Jex, MPH Diane K. King, MS, OTR Alanna Kulchak Rahm, MS Barbara L. McCray Candace C. Nelson., MA Paul A. Nutting, MD, MSPH Diego Osuna, M.D. Caitlin O Neill, MS, RD Debra Ritzwoller, Ph.D. Anna Sukhanova, M.A. Holly Whitesides, B.S. Daniel Winn, BS Sheana S. Bull, Ph.D. David Dzewaltowski, Ph.D. Paul A. Estabrooks, Ph.D. Lisa M. Klesges, Ph.D. Marcia Ory, Ph.D., M.P.H. Deborah J. Toobert, Ph.D. Kaiser Permanente Colorado, Clinical Research Unit Mission: To Develop, Conduct, and Translate High-Quality Research into Practice

3 OVERVIEW 1. Journalist Questions and Terminology 2. Decision Maker Issues Practical ehealth Trials 3. Evaluation and RE-AIM Needs 4. The Road Ahead: Key Challenges and Opportunities

4 WHO, WHAT, WHEN, WHERE, AND HOW Who Gets Invited and Who Comes? (Digital Divide Stereotypes) - Define eligibility and exclusions - What percent of those invited participate? - What are the characteristics of participants? Compare to: a) Non-participants or b) Representative sample(s)

5 WHO, WHAT, WHEN, WHERE, AND HOW Key Issues for Representativeness: - Race, ethnicity, and SES - Computer experience - Health literacy* - Contextual issues related to health care setting - What are barriers to participation? *Nielson-Bohlman, et al. (Eds) (2004) Health Literacy, Institute of Medicine; The National Academies Press. Washington, DC.

6 WHO, WHAT, WHEN, WHERE, AND HOW How is the IT Program Designed? - Theoretical Basis and Behavioral Architecture - How User Friendly and Consumer- Based Is it? - Level of Interactivity - Adopted to Fit the Context Mooney K, Bergheim L. (2002) The Ten Demandments: Rules to Live By in the Age of the Demanding Customer. McGraw-Hill Danaher BG, et al (2005) J Med Internet Res 7(2):e12

7 WHO, WHAT, WHEN, WHERE, AND HOW How Do (Different) Participants Use the Technology? - Which components or sections, content? - Qualitative and quantitative syntheses - Moderator variables - Critical importance of formative evaluation and usability labs

8 WHO, WHAT, WHEN, WHERE, AND HOW What Happens Over Time? - Do patients stay involved? - Do they use the resources differently over time (initiation, change, maintenance)? - How long do outcomes persist? - Who drops out and why?

9 DECISION MAKING FOR DISSEMINATION Making Progress in the Relevance and Application of research; as well as in the process for acquiring and using it in Decision Making needs a cultural change by both researchers and decision makers. Jonathan Lomas

10 STUDIES TO TRANSLATE RESEARCH INTO PRACTICE Key elements of Practical Clinical Trials - Representative Patients - Multiple Settings - Controls address standard of care; other alternatives - Outcomes or measures relevant to clinicians and decision makers Tunis SR, Stryer DB, Clancy CM (2003) JAMA 290:

11 WHO, WHAT, WHEN, WHERE, AND HOW What Outcomes are Produced? - Behavior Change (patients and clinicians) - Biological Changes - Quality of Life and Unanticipated Consequences - Health Care Utilization and Patient/Provider Interactions - Economic Outcomes Glasgow, et al. (2003) Diabetes Care 26(8):

12 BEHAVIOR CHANGE Brief, practical measures* Often triangulate when no gold standard Focus on sensitivity to change Measures of patient, staff, change agents (e.g., family), system and policy changes * Glasgow, et al. (2005) Ann Fam Med 3:73-81

13 ECONOMIC OUTCOMES Use Standardized Methods Assess cost of intervention delivered* Estimate replication costs** Optional, more sophisticated analyses of cost-effectiveness, cost-utility, cost-benefit, return on investment Costs are not costs are not costs * Gold, et al. (2003) Cost-effectiveness in health and medicine. New York: Oxford Univ. Press ** Meenan, et al. (1998) Med Care 36:

14 IN DESIGNING FOR PRACTICAL ehealth TRIALS, be: Practical in intervention delivery Broad in what you measure Transparent (TREND*) in reporting Summarize results in terms understandable to clinicians (NNT) and policy makers * Des Jarlais, D.C., Lyles, C., Crepaz, N. (2004) Am J Public Health 94(3):

15

16 RE-AIM TO HELP PLAN, EVALUATE, AND REPORT STUDIES R Increase Reach E Increase Effectiveness A Increase Adoption I Increase Implementation M Increase Maintenance Glasgow, et al. Ann Behav Med 2004;27(1):3-12

17 SIMPLE QUESTIONS FOR DISSEMINATION 1. Who comes? (Reach and Representativeness) 2. What Outcomes are Produced? (Effectiveness) (Intended and Unintended) 3. Where Will Program Work? (Adoption and Representativeness) 4. How Consistently is Program Delivered? (Implementation) 5. How Long Will Effects Last? (Maintenance) Dzewaltowski, et al. (2004) Ann Beh Med 28(2):

18

19 RE-AIM AND RELATED GENERALIZATION ISSUES To Whom Do Results Apply? - At Setting, Clinician, and Patient Level - Contextual and Moderating Factors - Rakowski Focal Point Concept Rakowski W, Breslau ES (2004) Perspectives on behavioral and social science Cancer 101(5 Suppl):

20 SETTING LEVEL Which Settings and Health Professionals Adopt? - As important as individual level representativeness - Parallel questions about eligibility, participation as at individual level - Key issues: Low resource and rural settings Barriers to setting participation (silo effect) Sciamanna, et al. (2004) Infor Primary Care 12:40-48

21 RE-AIM AND RELATED GENERALIZATION ISSUES The 3 Rs of Translation/Dissemination - Representativeness (Reach, Adoption) - Robustness (Effectiveness across subgroups) Cronbach s generalization across persons, time, measures - Replicability (Implementation) in representative settings Cronbach LH, et al. (1972) The dependability of behavioral measurements: Theory of generalizability for scores and profiles. New York, John Wiley & Sons Shadish WR, et al. (2002) Experimental and quasi-experimental design for Generalized causal inference. Boston: Houghton Mifflin

22 Reach Effectiveness Adoption Implementation Maintenance RE-AIM Dissemination Focus Efficacy Focus See for displays and evaluation questions

23 NEW RE-AIM METRICS 1) Individual Level Impact (RE) = Reach x Composite Effectiveness a) Reach = [Participation rate Median ES differential characteristics ] b) Composite Effectiveness = [Median ES key outcomes -Median ES negative outcomes/qol Median ES differential impact ] Glasgow, et al. (2005) Evaluating the Overall Impact of Health Promotion Programs Health Education Research, In press.

24 NEW RE-AIM METRICS 2) Efficiency = Cost of Intervention (over control) [Reach x Composite Effectiveness] 3) Setting Level Impact (AI) = Adoption x Implementation [Multi-level Adoption (rate and robustness at setting and clinician levels) x Composite Implementation] Glasgow, et al. (2005) Evaluating the Overall Impact of Health Promotion Programs Health Education Research, In press.

25 RECOMMENDED PURPOSE OF FUTURE ehealth RESEARCH To determine the characteristics of interventions that can: - Reach large numbers of people, especially those who can most benefit - Be widely adopted by different settings - Be consistently implemented by staff members with moderate levels of training and expertise - Produce replicable and long-lasting effects (and minimal negative impacts) at reasonable cost

26 That Would Be Nice, But is it really feasible to do all this in a given report?

27 YES! See examples that do much or all of this. Ahern/RWJF ehealth Technologies Program M. Campbell, et al., (1999) Health Educ Res 14: R. Glasgow, et al., (2004) J Gen Int Med 18: D. Gustafson, et al., (2001) Med Info D. Toobert, et al., (2002) Health Educ Res 17:

28 CHALLENGES TO DISSEMINATION Hard to determine applicability to local setting Insufficient detail or specificity in protocol Protocol does not fit local expertise, patients, resources, time or culture Protocol not delivered as in EB study

29

30 THE ROAD AHEAD: DIRECTIONS AND CHALLENGES 1. Overcoming Silo Effects in ehealth, of: - One health condition - One target behavior - One technology - One setting - One discipline

31 THE ROAD AHEAD: DIRECTIONS AND CHALLENGES 2. Understanding the role of interpersonal contact in ehealth - Level, timing, and types of health professional contact - Role of peer support (chat rooms, bulletin boards, etc.) - Lay health coaches (Senior Net, etc.) Tate DF, Wing RR, Winett RA (2001) Using the Internet technology to deliver a Behavioral weight loss program. JAMA 285: Glasgow RE, McKay HG, et al. (2003) The D-Net Diabetes Self-Management Program Prev Med 36:

32 THE ROAD AHEAD: DIRECTIONS AND CHALLENGES 3. The 4 th R Respect for Creative Designs - ehealth interventions are dynamic, not static - Respect for designs in addition to traditional RCTs - Evaluation designs to fit the question - Documentation of how ehealth programs EVOLVE over time* *Rotheram-Borus MJ, Flannery ND (2004) Interventions that are CURRES: Cost-effective, useful In: Rehmschmidt, et al (Eds) Facilitating pathways: Care, treatment, and prevention in child and adolescent Health. (pp ) New York, Springer

33 THE ROAD AHEAD: DIRECTIONS AND CHALLENGES 4. Advice for Politicians (and ehealth Researchers) - Focus on the Denominator (of settings, clinicians, patients) - Everything is Contextual (customize and document it) - Plan for Generalization and Adaptation (don t hope for it) - Look for Interfaces with Policy - Think like and involve your Target Audience Klesges LM, et al. (2005) Ann Behav Med 29:66S-75S.

34 SOME MODELS FOR TRANSLATION RESEARCH SciPCH Recommendations CURRES Robinson, et al (1998) JAMA 280: Rotheram-Borus, et al (2004) Precede-Proceed Green & Kreuter (2005) Practical Clinical and Tunis, et al (JAMA, 2003) Behavioral Trials RE-AIM Glasgow, Davidson, Dobkin, Ockene, Spring (from EBBM committee, 2005; Ann Behav Med, in press)

35 CONCLUSION The world is complex, contextual, evolving, and multiply determined. Designs and measures for ehealth translational and dissemination research should address these characteristics