Rapid appraisal of the literature: Identifying study biases Rita Popat, PhD Clinical Assistant Professor Division of Epidemiology Stanford University School of Medicine August 7, 2007
What is critical appraisal? Balanced assessment of benefits and strengths of research against its flaws and weaknesses Assessment of research process and results To be undertaken by all health professionals as part of their work
Why should we critically appraise? Published research is not always valid we cannot take conclusions for granted, even if the article is published in a peer-reviewed journal. Published research is not always relevant the abstract may indicate relevance but you will need to read the complete article to judge its applicability to your own practice/circumstances. To improve clinical effectiveness, we need a systematic framework to interpret research, rather than relying on a haphazard or casual approach. 3
Key Steps To Effective Critical Appraisal 1. Are the results valid? Focus of today s lecture 2. What are the results? 3. How will these results help me work with my patients? 4
Outline Quick review of study designs What is validity? Identifying study biases that can threaten internal validity 5
Observational vs. Experimental Studies Experimental studies the investigator tries to control the environment in which the hypothesis is tested (the randomized, double-blind clinical trial is the gold standard) Observational studies the population is observed without any interference by the investigator 6
Why Observational Studies? Cheaper Faster Can examine long-term effects Hypothesis-generating Sometimes, experimental studies are not ethical (e.g., randomizing subjects to smoke) Sometimes, experimental studies are not possible examples randomizing subjects to gestational diabetes studying natural progression of a disease studying long term effects of drugs 7
THE LANCET Vol 359 January 5, 2002
Cohort Studies Target population Exposed (E+) Disease-free cohort Not Exposed (E-) TIME Disease (I E+ ) Disease-free Disease (I E- ) Disease-free 9
Measure of association in cohort studies: Relative Risk (RR) Exposure + Outcome + a Outcome - b a + b Exposure - c d c + d RR = a c a + c + b d = I I E E + Incidence (probability) of outcome among exposed Incidence (probability)of outcome among unexposed 10
Active Inactive T2DM + T2DM - 376 12560 985 23957 12936 24942 RR = 376 12936 985 24942 = 0.029 0.0395 = 0.73 Interpretation: Active women are 27% less likely to develop T2DM compared to inactive women Energy expenditure was at least 1000 kcal/wk. 11
Cohort Studies: Advantages & Disadvantages Advantages Allows you to measure true rates and risks of disease for the exposed and the unexposed groups. Temporality is correct Can be used to study multiple outcomes. Prevents bias in the ascertainment of exposure that may occur after a person develops a disease. Disadvantages Can be lengthy and costly! Loss to follow-up is a problem (especially if nonrandom). 12
Case-Control Studies Target population Cases (outcome +) Controls (outcome -) Exposed in past Not exposed Exposed in the past Not Exposed 13
Measure of Association in case-control studies: Odds Ratio (OR) Disease (D+) Cases No disease(d-) Controls Exposure (E+) a b No exposure(e-) c d a+c b+d OR = Odds of exposure among cases P( E D) Odds of exposure among controls = P( E D) P( E D ) P( E D ) = a c b d a+ c a+ c b + d b + d = ad bc 14
Cases Controls NSAIDS+ 561 971 NSAIDS- 71 74 OR = ( 561 )( 74 ( 71 )( 971 ) ) = 0.60 Interpretation: NSAIDs use is associated with a 40% reduction in the risk of colon cancer 15
Case-Control Studies: Advantages & Disadvantages Advantages: Cheap and fast Great for rare diseases Disadvantages: Exposure estimates are subject to recall bias (those with the disease are searching for reasons why they got sick and may be more likely to report an exposure) interviewer bias (interviewer may prompt a positive response in cases). Temporality is a problem (did exposure cause disease or disease cause exposure?) 16
Intervention Studies Target population Intervention group Disease-free cohort No intervention group TIME Disease Disease-free Disease Disease-free 17
Eligible participants Randomized Standard lifestyle recommendations Intensive Metformin Placebo Lifestyle (n = 1079) (n = 1073) (n = 1082) 18
DPP trial: Primary Outcome - Diabetes 19
Intervention Studies: Advantages & Disadvantages Advantages: Allows randomization (controls for confounding) Allows double-blind assessment (controls bias) Disadvantages: Can be lengthy and costly! Loss to follow-up is a problem (especially if nonrandom). Ethical limitations 20
Causation in human health & disease Association does not prove causation If a putative risk factor and the occurrence of an outcome are strongly associated with each other it does not provide evidence that the risk factor causes the disease, only implies that it is correlated with outcome Non-causal explanations may cause a spurious association study biases (measurement error, selection bias, confounding, sampling error) 21
Study validity INTERNAL VALIDITY Do we believe the results? EXTERNAL VALIDITY Can the results be applied to the target population i.e., beyond the subjects in the study? 22
Threats to Internal validity: Non-causal explanations due to study biases Confounding Selection bias Misclassification bias (measurement error) 23
Confounding Confounding is the defined as a distortion of an exposure-outcome association brought about by the association of another factor with both the outcome and exposure exposure outcome Confounding factor 24
JAMA. 2004;292:1188-1194 Research question - Is physical activity associated with risk of T2DM? Physical activity T2DM + BMI + Potential confounder 25
Why worry about confounding? Spurious association Exaggerate an association (over-estimate) Attenuate an association (under-estimate) Obscure an association 26
Methods for controlling confounding Design phase Randomization Matching Restriction Analysis phase: statistical adjustment for confounders 27
Controlling confounding by randomization
Identifying potential confounders Subset of Table 1 from JAMA. 2004;292:1188-1194
Statistical adjustment for confounding Crude HR =2.9/3.9 = 0.74 Is BMI a confounder of the relationship b/w physical activity and T2DM? Compare the crude hazard ratio (HR) to the adjusted HR. Weinstein et al. JAMA 292:1188-94
Statistical adjustment for confounding Weinstein et al. JAMA 292:1188-94 10% rule for identifying confounders: (Crude HR - Adjusted HR) X 100 10% Crude HR
True or false A randomized clinical trial design cannot be affected by bias due to confounding? Answer: False (if randomization is not done appropriately, then can introduce bias due to confounding) 32
Identifying bias due to confounding in a RCT Check the randomization procedure in the methods section (e.g., blocked randomization schemes when sample size is small) Check Table 1 see if groups are balanced If not, how was it handled? Was intention to treat analysis used? 33
JAMA. 2006;296:2441-2450
JAMA. 2006;296:2441-2450
Selection bias A form of sampling bias due to systematic differences between those who are selected for study (or agree to participate) and those who are not selected (or refuse to participate). 36
Selection bias Improper selection of cases or controls in a casecontrol study Subjects lost to follow-up varies according to both the exposure of interest and the outcome (e.g., in prospective cohort studies and clinical trials) Can affect any study design, although case-control studies more prone to selection bias Selection bias can cause either overestimates or underestimates of the true associations between the exposure and disease in the underlying population 37
Identifying Selection bias What are the response rates? Was follow-up complete? Ideally should have follow-up for at least 80% of the initial sample/cohort Does drop out differ in the groups being compared (e.g., treatment and placebo groups)? 38
Selection bias example: In this case-control study, response rate was ~80% in cases and ~60% in controls. Could there be selection bias especially among controls? 39
Selection bias example: Scenario: NSAID users in the base population were more likely to participate in this study than were non-nsaid users p 1 p 2 So observed proportion of NSAID users among controls (92.9%) is greater than true proportion of exposure Hence observed odds ratio of 0.54 is an overestimate (i.e., true odds ratio is greater than 0.54) [Note that Odds ratio = p 1 * (1- p 2 ) / p 2 *(1- p 1 )] 40
Some strategies for minimizing selection bias Careful enumeration and thorough attempts at recruiting all cases within the source population High standards for methods of control selection (population-based ideal) Minimizing non-response and refusals Minimizing loss to follow-up (in cohort and RCTs) 41
Information bias (aka measurement error) Misclassification of outcome Misclassification of exposure status 42
Information bias (measurement error) Imperfect definitions of study variables (outcome or predictors) or flawed data collection procedures Erroneous classification of outcome exposure 43
Information bias misclassification Gold-standard Outcome Outcome + - Study Outcome + Outcome - a TP c FN b FP d TN a+b c+d a+c b+d Sensitivity = a / (a+c) Specificity = d / (b+d) 44
Misclassification of the outcome Exposure + (treatment) Outcome + Outcome - a b Exposure (placebo) c d Non-differential misclassification occurs when the degree of misclassification of outcome is independent of exposure status Tends to bias the association toward the null Occurs when the sensitivity and specificity of the classification of outcome are same for exposed and non-exposed groups but less than 100% 45
Misclassification of the outcome Exposure + (treatment) Outcome + Outcome - a b Exposure (placebo) c d Differential misclassification occurs when the degree of misclassification differs between the groups being compared. May bias the association either toward or away from the null hypothesis Occurs when the sensitivity and specificity of the classification of outcome differ for exposed and nonexposed groups 46
Misclassification of the outcome: NSAIDs and colon cancer example Cases Controls NSAIDS+ 561 971 NSAIDS- 71 74 Most likely scenario in this study Is misclassification of outcome likely to be nondifferential or differential with respect to exposure? 47
JAMA. 2006;296:2441-2450 Study Measures The primary measures were the Medical Outcomes Study 36- Item Short- Form Health Survey (SF-36) bodily pain and physical function scales and the American Academy of Orthopaedic Surgeons MODEMS version of the Oswestry Disability Index (ODI). Secondary measures included patient self-reported improvement, work status, and satisfaction with current symptoms and with care. Symptom severity was measured by the Sciatica Bothersomeness Index (range, 0-24; higher scores represent worse symptoms). Is misclassification of outcome likely to be nondifferential or differential with respect to exposure?
Misclassification of the exposure Cases (outcome +) Controls (outcome -) Exposure + a b Exposure - c d Non-differential misclassification occurs when the degree of misclassification of exposure is independent of outcome/disease status Tends to bias the association toward the null Occurs when the sensitivity and specificity of the classification of exposure are same for those with and without the outcome but less than 100% 50
Misclassification of the outcome Cases (outcome +) Controls (outcome -) Exposure + a b Exposure - c d Differential misclassification occurs when the degree of misclassification differs between the groups being compared. May bias the association either toward or away from the null hypothesis Occurs when the sensitivity and specificity of the classification of exposure differ for those with and without the outcome 51
Misclassification of the exposure: NSAIDs and colon cancer example Cases Controls NSAIDS+ 561 971 NSAIDS- 71 74 Non-differential: poor recall in cases and controls Differential: cases recall NSAID use better than controls? Is misclassification of exposure likely to be nondifferential or differential with respect to outcome? 52
Some strategies for minimizing misclassification bias Test reliability and validity of instruments used to determine outcome and exposure Similar methods for determining outcome and exposure in all study subjects Train interviewers, blind interviewers to outcome status and study hypothesis ( interviewer bias) Blind subjects to study hypothesis ( recall bias) Use incident cases in case-control studies, not prevalent ( recall bias) Try to use objective measures (e.g., pharmacy records vs. self-report use of medications) 53
Summary : Study designs and biases Threats to internal validity Case-control Cohort RCT Confounding Generally present Generally present Not likely (due to randomization) Selection bias Likely (e.g., when low response rates) May occur due to differential loss to follow up May occur due to differential loss to follow-up Misclassification of exposure More likely to be differential Generally nondifferential Not likely; if exists then nondifferential (e.g., drop-in/drop out) Misclassification of outcome Most likely nondifferential Most likely nondifferential Most likely nondifferential 54
Once you are satisfied the study findings are valid You can now ask whether the association causal? Evaluate positive features of causation Temporality Strength of the association Dose-response relationship Consistency of findings Biologic plausibility 55
Best Experiment/RCT Prospective cohort Retrospective cohort Cost and ease Case-control Proof of cause Correlation/x-section Case series Case report Best 56
Critical appraisal tools Assist with systematic critique of published research papers Several tools available One of my favorite http://www.muhc-ebn.mcgill.ca/ebn_tools.htm#evidence and look for link to CASP Appraisal Tools 57
Critical appraisal of an article about therapy or prevention Primary guides: Was the assignment of patients to treatment groups randomized? Were all patients who entered the trial properly accounted for at the conclusion? Was follow up complete? Were patients analyzed in the groups to which they were randomized, that is, was an intention to treat analysis used? 58
Critical appraisal of an article about therapy or prevention Secondary guides: Were patients, health workers and study personnel blinded to treatment? Does the study provide evidence that blinding was effective? Were the groups similar at the start of the trial? Was there an adequate Table 1? If not, were adjustments made for differences? Aside from the experimental intervention, were the groups treated equally? 59
Critical appraisal of an article about therapy or prevention What were the results? How large was the treatment effect? How precise was the estimate of the treatment effect? Were confidence intervals given? Will the results help me with my patients? Will discuss these aspects on August 9 th, 2007! 60