Non-Randomized Trials ADA Research Toolkit ADA Research Committee 2011 American Dietetic Association. This presentation may be used for educational purposes
Learning Objectives At the end of this presentation the participant will be able to: Define a non-randomized trial and its components Describe the study design Describe the strengths and limitations of non-randomized trials Identify the main statistical tools used to analyze nonrandomized trials 2
Introduction Definition of a Non-Randomized Trial A study where participants have been assigned to the treatment, procedure, or intervention alternatives by a method that is not random. The investigator defines and manages the alternatives. (ADA terms on EAL) 6/1/2011
Introduction Non-randomized trials are a type of quasi-experimental design Quasi-experimental designs are those that do not meet the criteria for a true experimental design such as random assignment of participants to groups or having a control group. Portney L.G., Watkins MP. Foundations of Clinical Research. 3rd ed. Upper Saddle River, NJ: Pearson/Prentice Hall; 2009:223. 4
Distinguishing Features Uses natural groups or assigns participants to groups using a non-random procedure The investigator controls the exposure of groups to the intervention Prospective Confounders exist due to non-randomization 5
When to Use a Non-Randomized Trial When is it appropriate to use a non-randomized trial design? When the act of random allocation may reduce the effectiveness of the intervention Occurs when the effectiveness of the intervention depends on the participant s active participation which is influenced by their beliefs and preferences) When it would be unethical to do random allocation When it is impractical to do random allocation (e.g. cost or convenience factors) When there are legal or political obstacles to random allocation 6
Use of Controls There are different types of controls that can be used in non-randomized trials Concurrent controls: Treatment and control group participants are matched at the group level based on demographic and other characteristics, and receive different treatment conditions at the same time. www.hsrmethods.org/glossary 7
Use of Controls Example 1: Concurrent Control Population: Adults with Type 2 Diabetes Mellitus Inclusion criteria: age 20 65, female, noninsulin dependent, HbA1C < 8 Adults with Type 2 DM at outpatient hospital clinic A Standard Treatment Adults with Type 2 DM at outpatient hospital clinic B New Treatment 8
Use of Controls Historical controls: Investigators compare outcomes among a group of participants who are receiving a new treatment (experimental group) with outcomes among participants who received standard treatment in a previous period (control group). www.hsrmethods.org/glossary 9
Use of Controls Example 2: Historical Control Population: Children with Type 1 Diabetes Mellitus Inclusion criteria: age 8 18, insulindependent 50 Children with Type 1 DM not treated with insulin pumps whose management was well-documented at same clinic in the past Historical Control Group 50 Children with Type 1 DM at same outpatient hospital clinic treated with insulin pumps Treatment Group 10
Questions Answered What is the effectiveness of the intervention/exposure? Does it work? What is the magnitude of the effect? How large is the difference in the measured outcome (dependent) variable between the treatment group and the control group? What proportion of the sample/population will benefit? What percentage of the treatment group benefited? Which approach is better? Did the intervention group have a better outcome than the control group? 11
Generalizability (external validity) Generalizability is dependent on how representative the sample is of the reference population (the population to which results would apply External validity is the extent to which study findings can be generalized beyond the sample used in the study (Burns and Grove) Burns N, Grove SK. The Practice of Nursing Research: Appraisal, Synthesis and Generation of Evidence. 6th ed. St. Louis: Saunders Elsevier, 2009. 12
Internal Validity Definition The extent to which the effects detected in the study are a true reflection of reality, rather than being the result of the effects of extraneous variables (Burns and Grove) www.socialresearchmethods.net/kb/intval.php 13
Internal Validity Threats to internal validity Selection Bias: baseline differences between the control and intervention groups Known factors (confounding variables) that may have an effect on the outcome being measured (eg. age, education or income level) Unknown factors Differences in details of the intervention between groups Non-random dropout from the study Bias due to conscious and unconscious prejudice for or against the intervention on the part of participants, providers or evaluators 14
Internal Validity Ways non-randomized study designs can increase internal validity Blinding Blind participants so they do not know whether they are in the intervention group or the control group Blind providers so they do not know which participants are in the intervention group Blind evaluators so they do not know which participants are in the intervention group Statistically control confounding variables CAUTION: Due to lack of randomizaton, differences between the groups that are present before the intervention may affect the outcome of the study, thus limiting the conclusions regarding cause and effect. 15
Results Establish whether control and intervention groups are equivalent Compare baseline characteristics between groups Are differences statistically significant? If yes: may indicate systematic bias in group assignment If no: a) Groups are not different or b) If sample size is too small to provide adequate statistical power to detect a difference, groups may not be equivalent Are differences clinically significant? If yes: potential confounders exist If no: : due to nonrandom assignment, unknown confounders are still a threat to internal validity 16
Non-Randomized Trial (NRT) Are differences in the outcome variable between the control group and intervention group(s) significant? Outcome variables are measured quantitatively (ordinal, interval or ratio) Four levels of measurement Nominal: the numerical values just name the attribute uniquely Ordinal: the attributes can be ranked in order Interval: the distance between attributes has meaning Ratio: there is an absolute zero Level of measurement affects which statistical tests are appropriate www.socialresearchmethods.net/kb/measlevl.php 17
Were Data Adjusted for Confounders? Independent Variable Diet Intervention Weight Loss Outcome Variable Confounding Variables Exercise BMI Age Sex Confounding Variables: characteristics that differ between groups and may affect the results of the study 18
Typical Statistics Parametric statistics are used for normally* distributed data measured on the interval or ratio scales T-test: compares means of two groups Analysis of variance (ANOVA): compares means of three or more groups Multivariate analysis (e.g. MANOVA): analyzes relationships among three or more variables Logistic regression: predicts odds of a dichotomous outcome Nonparametric statistics are used for data measured on the nominal or ordinal scales that do not meet certain assumptions about population parameters such as a normal distribution Chi-square test: compares observed frequencies within categories to frequencies expected by chance *normal distribution: a symmetrical bell-shaped theoretical distribution that has defined properties 19
Key Points Appropriate when randomization not feasible No random assignment into groups Conclusions must take into account potential biases Can show associations and trends, but cannot test causeand-effect hypotheses 20
Resources Schwartz RP, Hamre R, Dietz, WH, et al. Office-Based Motivational Interviewing to Prevent Childhood Obesity. Arch Pediatr Adolesc Med. 2007;161:495-501. http://archpedi.ama-assn.org/cgi/content/full/161/5/495 21