Critical Appraisal of RCT

Critical Appraisal of RCT

What is critical appraisal? Definition Process of systematically examining research evidence to assess its reliability (validity/internal validity), results and relevance (external validity) before using it to inform a decision Critical Appraisal Skills Program, Institute Health Sciences, Oxford Part of evidence based medicine(ebm) Allows us to make sense of research evidence to ensure practice is aligned with best evidence

Why we need critical appraisal? If health care professionals and managers are going to make the best decisions they need to be able to: Decide whether studies have been undertaken in a way that makes their findings reliable (validity/ internal validity). Make sense of the results. Know what these results mean in the context of the decision that needs to be made (relevance/ external validity/generalisability).

Target Population Study population Sample Population Randomised Controlled Trial Intervention Assess outcome Control Assess outcome Randomisation 5

Interventional Studies Before & after Non randomised controlled trial Randomised controlled trial Individual randomisation Cluster randomisation All RCT s should be reported according to CONSORT statement / checklist

Explaining the Study results Bias: could the results (of the study) be explained by a systematic error in its design? Confounding: could the results be explained by some other difference between the groups? Chance: could we have got these results by chance? True: The effect is due to the intervention

Bias in RCT Target population Selection well randomised? are groups similar? Performance Equally treated? Treatment Allocation Control Attrition complete follow-up? Detection independent and blind assessment of outcome? Measurements objective and standardised? Exposed Follow-up Outcomes Not exposed Follow-up Outcomes

Randomisation Generation of sequence Method of randomisation Central computer randomisation Using random numbers Allocation concealment Masking of sequence (The researcher not aware which intervention the next patient will get)

Assessing follow up bias: Flow of participants through each stage of randomised trial Su L et al. BMJ 2007;335:596 2007 by British Medical Journal Publishing Group

Bias In Assessing Outcome Influenced by patients expectations Make patient blind to treatment (single blind) Influenced by clinician / data collectors expectations Make clinician blind to treatment (double blind) Most important to use "blinded" outcome assessors when outcome is not objective. Influenced by analysts expectations Make analyst blind to treatment (triple blind) 11

Confounding Smoking Lung cancer Asbestos Exposure

Confounding in RCT Acupuncture Pain Severity of disease

Results Is the study powered to detect a meaningful difference? Did they analyse based on intention to treat principle? What are the results? (Direction of effect and size of effect) Were they analysed correctly Do the results make sense? What could have been the role of chance? Precision of results (Confidence Intervals) Statistically significant? p values Clinical/practical significance?

Power of Study Probability of rejecting the null hypothesis if the alternative hypothesis is true. Can the study detect an effect if there was one.

Contamination Or Crossover Participants randomised to the control group may unintentionally receive the intervention Participants randomised to the intervention group may NOT receive the intervention 16

Heart Protection Study 20,600 eligible subjects randomised Simvastatin 10,300 Control 10,300 Year 1 89% on Simvastatin Year 1 4% on Simvastatin What should we do? Year 5 82% on Simvastatin Year 5 32% on Simvastatin 17

Options 1. Analyse according to Treatment Received But groups are no longer randomised Allocation bias! 2. Analyse by Intention To Treat Preferred option Reduces risk of allocation bias May underestimate the effect Principle: Once a patient is randomized, s/he should be analyzed in the group randomized to - even if they discontinue, never receive treatment, or crossover. 18

What do the results mean?.. Direction of effect Size of effect Precision of results (Confidence intervals) Statistically significant (p values) Clinical significance

Relative Risk Intervention Arm (Group 2) Control Arm (Group 1) Risk in group 2 = A/A+B Risk in group 2 = 23/122 = 0.1885 Outcome + Outcome - 23 (A) 11 (C) 99 (B) 115 (D) Risk in Group 1 = C/C+D Risk in Group 1 = 11/126 = 0.0873 Relative Risk = Risk in Grp2/Risk in Grp1 Relative Risk = 23/122 11/126 = 2.16

Absolute risk and NNT Outcome + Outcome - Intervention Arm (Group 2) Control Arm (Group 1) 23 (A) 11 (C) 99 (B) 115 (D) Risk in group 2 = A/A+B Risk in group 2 = 23/122 = 0.1885 Risk in Group 1 = C/C+D Risk in Group 1 = 11/126 = 0.0873 Absolute Risk = Risk in Grp2 - Risk in Grp1 Absolute risk= 0.1885-0.0873 = 0.1012 NNT = 1/Absolute risk =1/0.1012 =9.88= 10

Is it due to chance? P value Confidence Interval

P-values (Hypothesis Testing) - in our study Exclusive breast feeding fraction/risk Risk in group 2 = 0.1885 Risk in group 1 = 0.0873 Relative Risk = 2.17 (P <0.05) The probability that this result would occur by chance is less than 5 in 100 statistically significant

Confidence Intervals (Estimation) - in our study Exclusive breast feeding fraction/risk Risk in group 2 = 0.1885 Risk in group 1 = 0.0873 Relative Risk = 2.17 (P <0.05) (95% CI, 1.05 4.43) The true value could be as low as 1.05 or as high as 4.43 - but is probably closer to 2.17 Since the CI does not include the no effect value of 1 the result is statistically significant

When we have eliminated everything else, whatever remains, no matter how incredible, must be the truth Sherlock Holmes (Arthur Conan-Doyle) 25

10 questions to help with a RCT is it trustworthy? Reliable what does it say? results will it help? relevance

Acknowledgements Dr. Tom Marshall - Department of Public Health, Epidemiology & Biostatistics Dr. Catherine Meads, Barts and the London School of Medicine and Dentistry Dr. Amanda Burls, CEBM, Oxford University Dr. Paul Glasziou, CEBM, Oxford University Dr. Dan Lasserson, CEBM, Oxford University Centre for Evidence Based Medicine Oxford University University of Birmingham References: Evidence based medicine Sharon Straus 4 th Edition 2010 Evidence based health care Muir Gray 3 nd Edition 2009