Overview Critical Literature Evaluation and Biostatistics Ahl Ashley N. Lewis, PharmD, BCPS Clinical Specialist, Drug Information UNC Hospitals Background Review of basic statistics Statistical tests Clinical i trial design eig Reasons to Read and Evaluate Stay current in your practice area Learn from experience of others FDA is variable in requirements for approval Clinical i experience e can be misleading i Interpretations can be different Goals of Interpretation Establish the significance or importance of the trial Relate the results to the original objectives of the trial Compare data from the trial with data obtained from other trials Common Problems Encountered in the Literature Flawed study design Invalid statistical analysis Fraud, deception, and misrepresentation Unintentional errors Poorly conducted research Poorly written manuscripts Data dredging 1
Sample versus population Population: collection of all possible patients Sample: subset of members selected from the defined population under study Must be representative of the population Allocation Way subjects are chosen and assigned to their particular study group Nonrandom assignment Preferentially assign subjects to one of the study arms Will lead to bias Randomization Each subject has an equal and independent chance of being in any of the study arms Simple Randomization Single sequence of random assignments Groups may be unequal in size Blocked Randomization Used to ensure that comparison groups will be of approximately the same size Block numbers used to allocate patients Block size of 4: ABAB, BABA, AABB, BBAA Stratified Randomization Nonoverlapping groups based on key population characteristics Characteristics that may confound the results Random sample is selected from each stratum Cluster Randomization Groups, as opposed to individuals, are randomized to different interventions Controls Placebo control Historical control Cross over control Active control Standard treatment control Within patient comparison control Blinding Rationale To prevent clinicians from assessing/treating one patient group differently from the other To overcome the placebo effect T l i li To ensure equal patient compliance Limitations May be difficult to blind a medication with a distinctive taste, physiologic effect, or continuous titration Expensive and time consuming 2
Blinds Open label Single blind Double blind Double blind, double dummy trial Triple blind Total blind Bias Systematic error that enters a clinical trial and influences the outcomes Types of bias Selection Recall Interviewer Misclassification Channeling or confounding by indication Confounding Variable(s) that may influence the patient s condition and may affect the outcomes statistically Cigarette smoking causes lung cancer in a case control trial, drinking alcohol may confound the results Controlling for bias Selection of study population Means of collecting data Sources of information Controlling for confounding Randomization Matching Stratification Multivariate analysis Analysis Intention to treat Intended initial subjects assignments Estimates treatment under usual conditions Gives an conservative estimate of the treatment difference Last observation carried forward Analysis As treated Analyzed based on what was actually received Last observation carried forward Last observation carried forward Cleaner data regarding method effectiveness 3
Analysis Per protocol Analyzed based on who precisely followed the predefined protocol Provides superior data in regards to method effectiveness Provides more generous estimate of the treatment difference Does not portray real world if compliance is related to prognosis Interim analysis Analysis of the data at one or more time points prior to the official close of the study Ethical versus economic reasons Early stop rule Plan for stopping clinical trial based on safety or efficacy Data safety and monitoring board Mechanism for monitoring interim data in clinical trials in order to ensure the safety of the participating subjects Statistics Method of collecting, classifying, summarizing, and analyzing data Data can be categorized in several ways Continuous Discrete Descriptive Inferential Continuous Data Data with constant and defined units of measure Equal distance between units Two types Interval scale Arbitrary zero point Examples: Ratio scale Absolute zero Examples: Discrete Data Discrete data Nominal Categorical information Examples: Information classified by an assigned number or code to make data numeric Numbers are truly arbitrary with no particular order Ordinal Specific number of categories with an implied rank Examples: Distance is not necessarily equal between categories Types of Methods Parametric Data must follow normal or near normal distribution Data must be continuous Nonparametric Nonparametric Data studied deviate from normal distribution Data are ordinal or nominal 4
Variables Variable The characteristic being observed or measured Independent variable The intervention ti or what tis being bi manipulated Define the conditions under which the dependent variable is to be examined Dependent variable The outcome of interest within the study The outcome that intends to explain or estimate Variables Univariable analysis Applicable to a set of observations that contains one dependent variable and no independent variable Bivariable analysis Applicable to a set of observations that contains one dependent variable and one independent variable Multivariable analysis Applicable to a set of observations that contains one dependent variable and more than one independent variables Descriptive Data Summarizes data into a useful form Present, organize, and summarize data Measures of central tendency Mean Median Mode Example Data Set 1 1 + 2 + 3 + 4 + 5 = 15 Data Set 2 1 + 2 + 3 + 4 + 100 = 110 Mean = 3 Mean = 22 Median = 3 Median = 3 Normal Distribution MEAN = MEDIAN = MODE Descriptive Data Measures of variability Range Interquartile range Percentile Standard d deviation Meaningful only when calculated for normally distributed continuous data Describes the variability of data about the sample mean SD = variance Standard Deviation Gaddis et al. Ann Emerg Med. 1990;19:309 15. Inferential Statistics Standard Error of the Mean (SEM) = SD / [ (n)] It is computed from a single sample and is always smaller than SD Measure of the precision with which a sample mean (x) estimates the population mean (μ) The true but unmeasured mean of the larger population of interest will lie within two standard errors of the sample mean about 95% of the time 5
Standard Deviation Standard Error of the Mean X Sample = 100; SD = 4; x = 130 mmhg X Sample = 100; SD = 4; x = 130 mmhg SD versus SEM Effect of Weight Loss with Orlistat on Glucose Tolerance and Progression to Type 2 Diabetes in Obese Patients Meta analysis of 3 trials whose original endpoint was weight htloss with orlistat t Authors used SEM rather than SD SD versus SEM Orlistat group (n=359) 6.72 ±0.41 kg (mean ±SEM) (2 SEM: 5.9 to 7.54 kg) 6.72 ±7.79 kg (mean ±SD) (2 SD: 8.86 to 22.3 kg) Weight Loss Placebo group (n=316) 3.79 ±0.38 kg (mean ±SEM) (2 SEM: 3.03 to 4.55 kg) 3.79 ±6.75 kg (mean ±SD) (2 SD: 9.71 to 17.26 kg) Arch Intern Med. 2000;160:1321 6. Inferential Statistics Confidence intervals (CI) Provides a precise, objective way of specifying how good a sample estimate is taking into account sample size and variability 95% CI = x ± 2 SEM Inferential Statistics Confidence intervals The closer a point lies to the middle of the CI, the more likely it is representative of the population of interest Most commonly used for continuous data that are normally distributed Can also be calculated for medians, regression slopes, relative risks, hazard ratios 6
Inferential Statistics Confidence interval Relationship with sample size The width of the CI will when the sample size is increased The width of the CI will when the sample size is decreased Interpreting CIs Ratio World If you are evaluating any type of ratio and the CI includes one, then we conclude that statistical significance is not achieved Odds ratios, hazard ratios, relative risk All Other World If you are evaluating any other data and the CI includes zero, then we conclude that statistical significance is not achieved Continuous data BP, HR Hypothesis Testing Hypothesis Testing Statement of Hypothesis (H 1 ) Defines the research or alternative hypothesis State the Null Hypothesis (H 0 ) Hypothesis of no difference Either reject or fail to reject (accept) Errors in Hypothesis Testing Question Alpha False positive rate the investigators are willing to tolerate This is also known as type I error Concluding that there is a difference when one does not exist Beta False negative rate the investigators are willing to accept This is also known as a type II error Concluding that there is no difference when in fact there is a difference Accept Null Hypothesis Reject Null Hypothesis Null Hypothesis is True Null Hypothesis is False 7
Statistical Power The ability to detect a difference if a difference truly exists Power = 1 β Factors that influence power Sample size Alpha Inadequate power can cause a type II error If a study is performed and a difference is found, then power is irrelevant P values What is a p value? Probability of obtaining the observed difference between treatments in the study if there is no real difference between treatments t t in the larger population of interest How certain we are of the results Probability that the observed results in a study could be due to chance alone Interpreting P values For α = 0.05, the p value means that a difference will be due strictly to chance 5% of the time Small p value means there is strong evidence that some difference exists Does not necessarily mean that a large difference exists Sensitivity versus Specificity Sensitivity Ability of a test to reliably detect the presence of disease (positivity) Sensitivity (%) = 100 * TP/(TP + FN) Specificity Ability of a test to reliably detect the absence of disease (negativity) Specificity (%) = 100 * TN/(TN + FP) Sensitivity versus Specificity 250 clinically suspected to have DVT 150 actually do have DVT 130 have DVT by both gold standard test (GST) and new MRI (true positive group) Therefore, 20 proven to have DVT by GST had a negative new MRI (false negative group) 100 patients were judged to be disease free by GST Only 87 had a negative new MRI (true negative group) Remaining 13 were incorrectly classified by the new MRI (false positive group) Sensitivity versus Specificity Disease Evident No Disease Disease Evident TP (130) FN (20) No Disease FP (13) TN (87) Sensitivity (%) = 100 * TP/(TP + FN) = Specificity (%) = 100 * TN/(TN + FP) = 143 107 150 100 250 86.7% 87% Gaddis et al. Ann Emerg Med. 1990;19:591-7. 8
Predictive Value Predict the likelihood of disease in an individual Positive predictive value (PPV) Proportion of individuals who actually have the disease when the test indicates presence PPV (%) = 100 * TP/(TP + FP) Negative predictive value (NPV) Proportion of individuals who are truly free of the disease tested for when the test indicates the absence of disease NPV (%) = 100 * TN/(TN + FN) Disease Evident No Disease Predictive Value Disease Evident TP (130) FN (20) No Disease FP (13) TN (87) 143 107 150 100 250 PPV (%) = 100 * TP/(TP + FP) = 90.9% NPV(%) = 100 * TN/(TN + FN) = 81.3% Parametric Tests Statistical Tests Multiple (pairwise) comparison procedures (MCPs) Used to compare the means of the groups two at a time Also known as post hoc tests Less error associated with the use of MCPs compared with separate t tests Adjusts alpha to reduce the risk of type I error Examples Bonferroni correction Scheffe s method Tukey s least significant difference Newman Keuls Parametric Tests Only valid when used with continuous, normally distributed data Student s t test (uses t statistic) Independent samples Paired t test Matched samples ANOVA (uses F ratio) Three or more groups One way; two way; multiway; randomized block; repeated measures ANCOVA Controlling for the effects of confounders Nonparametric Tests Applies to non normal distributions or to data that do not meet the criteria for parametricity Continuous data, not normally distributed Ordinal data Nominal (categorical) data 9
Nonparametric Tests Mann Whitney U Test Ordinal or not normally distributed data Used for independent samples Similar to t test Wilcoxon Rank Sum is equivalent Wilcoxon Signed Rank Test Ordinal or non normally distributed data Used when data are matched or paired Nonparametric Tests Kruskal Wallis Test Ordinal or not normally distributed data Used for 3 or more groups with independent samples Similar to ANOVA Friedman Test Ordinal or not normally distributed data Used for 3 or more groups where subjects participate in more than one group Similar to randomized block ANOVA Nonparametric Tests Chi Square (χ 2 ) Compares the percentages between two or more groups Independent samples Most useful for nominal data, but can be used for ordinal data Useful to answer questions about rates, proportions, or frequencies Nonparametric Tests Fisher s Exact Test Used instead of χ 2 Matrix has an expected frequency < 5 Very small sample size McNemar s Test Used to compare nominal data from paired (matched) samples Also known as Cochran s Q Cochran Mantel Haenszel Test Used to compare nominal data while controlling confounding variables Survival Analysis Evaluating time to event Adjusts for the fact that subjects are followed for different lengths of time Censored data Patient lost to follow up or those that have not reached event of interest at the end Survival function When evaluated at time (t), the probability that the patient will survive until at least (t) Depicted by survival curves (Kaplan Meier) Survival Analysis Weeks 10
Survival Analysis Survival curve comparisons Log rank test Determine whether there is a statistically significant difference between the two curves Necessary when there are censored observations Useful for determining late differences in survival Assumption: proportional hazards Wilcoxon s test Determine whether there is a statistically significant difference between the two curves Useful for determining early differences Survival Analysis Cox Proportional Hazards Model Enables the difference between survival curves of particular groups to be tested while allowing for other factors Used to assess the effect of multiple variables on survival Separatesbaseline hazardfrom the variables Hazard ratio (HR) Relative risk of the complication based on comparison of event rates Correlation Analysis Quantify and define the strength of the relationship between variables This analysis tells us two things: Whether or not there is an association The strength of the association Correlation coefficient (r) Ranges from 1 to +1 1 = high negative or inverse relationship +1 = high positive or direct relationship Correlation Positive correlation r = +1 Does not assume a cause and effect relationship Correlation Correlation Negative correlation No correlation r = -1 r = 0 11
Correlation Analysis Correlation coefficient (r) Ranges from 1 to +1 1 = high negative or inverse relationship +1 = high positive or direct relationship Pearson Correlation Coefficient Used if the data are parametric Spearman Rank Coefficient Used if data are nonparametric Regression Analysis Allows prediction of the value of the outcome variable given the value of the intervention Coefficient of determination (r 2 ) Indicator of explained variance in one variable based on the second variable Varies from 0 to 1 The closer to 1 means the greater amount of variance in the dependent variable is explained by the independent variable Implies a cause and effect relationship Regression Analysis Linear Regression Continuous variables Logistic Regression Binary, categorical variables (eg, disease, no disease) Provides odd ratios for the data determining outcome measures of risk Regression Analysis Simple Regression One explanatory variable (intervention) Multiple Regression More than one explanatory variable Estimates of Effect Odds Ratio Odds Probability that an event will occur divided by the probability that it will not occur Odds ratio A ratio of the odds of an event in one group to those of another group Value < 1 = exposed group is less likely to experience the event of interest than unexposed group Value = 1 = exposed and unexposed groups are equally as likely to experience the event of interest Value > 1 = exposed group is more likely to experience the event of interest than unexposed group 12
Relative Risk Ratio of risk of an event occurring in the experimental group compared with the control group Indicates the risk of the event after the experimental treatment as a percentage of the original risk Value < 1 = therapy decreased the risk Value = 1 = no difference between treatments Value > 1 = therapy increased the risk RR = EER/CER Relative Risk Reduction Estimates the percentage of baseline risk that is removed as a result of therapy Percent reduction in the experimental group even rate (EER) compared with the control group event rate (CER) RRR of 0 indicates that there was no effect of the treatment compared with control RRR = 1 RR * 100 RRR = [(CER EER)/CER] * 100 Absolute Risk Reduction The difference in the event rate between a control group and an experimental group Provides the percentage of patients spared the adverse event as the result of treatment Changes with change in baseline risk ARR of 0 indicates no difference between comparison groups ARR = CER-EER Number Needed to Treat or Harm The number of patients who require treatment to prevent one event NNT or NNH assumes that the baseline risk is the same for all patients Do not extrapolate beyond the studied endpoints NNT = 1/ARR NNT = 1/[CER EER] Example of Measures of Effect Outcome No. (%) of patients Racemic albuterol Levalbuterol Total Hospitalized 123 (45%) 99 (36%) 222 (41%) Not hospitalized 150 (55%) 175 (64%) 325 (59%) Total 273 (100%) 274 (100%) 547 (100%) Odds: Racemic: 45%/55% = 82%; Levalbuterol: 36%/64% = 57% Odds ratio: Relative risk: 57%/82% = 69% 36%/45% = 80% Relative risk reduction: [45% - 36%]/45% = 20% Absolute risk: 45% with racemic albuterol; 36% with levalbuterol Critical Literature Evaluation and Biostatistics: Part II Ashley N. Lewis, PharmD, BCPS Clinical Specialist, Drug Information UNC Hospitals Absolute risk reduction: Number needed to treat: 45% - 36% = 9% 1/0.09 = 11 13
Trial Validity Clinical Trial Design Internal validity Problems with bias, confounding, measuring endpoints Does the study design accomplish what is suggested? External validity Ability to extrapolate to the population of interest Must have internal validity Can the results be applied to other groups, patients, or systems? NO internal validity = NO external validity Strength of Evidence Randomized clinical trials Follow up (cohort) study Increasing Case control study Strength Case series Case reports Study Perspectives Prospective followed forward in time Retrospective reviewed back in time What about meta analysis? Crossover Study Perspectives Each subject serves as his/her own control Subject receives both the study and control treatments Parallel Subject receives either the study or control medication throughout the study 2 or more groups are receiving treatment at the same time Basic Trial Design Descriptive Document and communicate experience: share ideas, programs, treatments, unusual events, and observations Begin search for explanation Explanatory Experimental Evaluate efficacy of therapeutic, educational, administrative interventions, Investigator controls allocation Observational Seek causes, etiologies, predicts, better diagnosis Investigator observes nature 14
Case control Design Exposure (+) Exposure ( ) Exposure (+) Exposure ( ) Cases Control Prospective Cohort Design (Follow up Design) Risk Factor (+) Risk Factor ( ) Study Population Disease or Outcome (+) Disease or Outcome ( ) Past Present Past Present Future Retrospective Cohort Design (Follow up Design) Risk Factor (+) Disease or Risk Factor Outcome (+) ( ) Study Population Risk Factor (+) Risk Factor ( ) Past Disease or Outcome ( ) Present Cross sectional Design simultaneous Risk or Factor (+) Free of disease or outcome Risk or Factor ( ) Study Population Risk or Factor (+) Have disease or outcome Risk or Factor ( ) Present Experimental Design Other Designs Study Population Present Experiment Intervention Control Disease or Outcome (+) Disease or Outcome ( ) Disease or Outcome (+) Disease or Outcome ( ) Future Superiority Detect differences between treatments Equivalence Confirm the absence of a meaningful difference(s) between treatments Noninferiority Designed to show that a treatment is no less effective than the existing treatment 15
Superiority Equivalence Committee for Proprietary Medicinal Products [July 2000, Cited November 2006]. Available from: www.emea.eu.int/pdfs/human/ewp/048299en.pdf Committee for Proprietary Medicinal Products [July 2000, Cited November 2006]. Available from: www.emea.eu.int/pdfs/human/ewp/048299en.pdf Noninferiority Trial Design Committee for Proprietary Medicinal Products [July 2000, Cited November 2006]. Available from: www.emea.eu.int/pdfs/human/ewp/048299en.pdf Conclusions of these trials are dependent upon the value chosen for Δ Choosing Δ can is difficult so watch for bias Reasons for pre definition of type of trial Ensure that comparator treatments, doses, patient populations, and endpoints are appropriate Allow sample size estimates based on correct power calculations Ensure that equivalence and noninferiority criteria are chosen Permits appropriate plans for analysis Switching the Objective Noninferiority to Superiority Properly designed and carried out with strict requirements for noninferiority Actual p values for superiority are presented to allow independent d assessment Analysis according to intention to treat principle Switching the Objective Superiority to noninferiority Lower end of the confidence interval provides a quantitative estimate of the minimum effect Protocol must contain a prospectively defined Δ for noninferiority 16
Systematic Review and Meta analysis Sum is greater than its parts This can be debated Combines the results of many trials to create a data set Calculation of effect size Interpretation of disparate results Statistical power for primary endpoints and subgroups Answers to new questions Systematic Review and Meta analysis Design issues Results are highly dependent on criteria for inclusion and exclusion of the previous studies and statistical methods to ensure validity Literature search conducted for inclusion of trials Funnel plot: way to visibly check for publication bias How treatment is/was assigned Tests for heterogeneity Criteria for pooling data and how they are measured Systematic Review and Meta analysis Funnel plot Measure precision against treatment effect Absence of publication bias is suggested by a symmetrical inverted funnel Systematic Review and Meta analysis Forest Plot Summary of results Each line represents a clinical trial Line is the CI Diamond is the result of the meta analysis Middle is the estimate of effect Width is the CI Critical Literature Evaluation and Biostatistics Ashley N. Lewis, PharmD, BCPS Clinical Specialist, Drug Information UNC Hospitals 17