Experimental Design Part II Keith Smolkowski April 30, 2008 Where Are We Now? esearch eview esearch Design: The Plan Internal Validity Statements of Causality External Validity Statements of Generalizability Designs Experimental Designs Quasi-experimental Designs Poor Designs Non-Experimental Designs 1 2 External Validity Statements of Generalizability Extent to which findings can be applied to other individuals or groups and other settings. Two aspects of external validity Population validity: extent to which you can generalize results to a specified larger group. Ecological validity: extent to which you can generalize results to other environmental conditions. A variety of threats to external validity identified by Glass and Bract (1968). Threats to External Validity Population Validity Population validity: generalization from a sample to a specific larger group Threats Inability to generalize from experimental sample to a defined population, the target population The extent to which personological variables interact with treatment effects 3 4 Generalize from experimental sample to a defined population Public schools and students by type of locale: 1996-97 Sample may differ from target population on location or environment Can only generalize to the population from which the sample was drawn Compare populations on relevant characteristics and hope results will generalize isky 5 regon Mississippi California 6
Interaction between personoligical variables and treatment effects Sample may differ from target population on personological variables Personological variables include locus of control, gender, SES, education, alcohol consumption, anxiety level, various academic or social skills, confidence level, and so on Public school student membership by racial & ethnic category regon Hawaii Mississippi 7 8 Threats to External Validity, Ecological Validity Explicit Description of IV Ecological Validity: Generalization from study conditions to a different environment 1) Explicit description of experimental treatment 2) Hawthorne effect 3) Multiple treatment interference 4) Novelty and disruption effects 5) Experimenter effect 6) Pretest sensitization 7) Posttest sensitization 8) Interaction of history and treatment 9) Measurement of dependent variable 10) Interaction of time of measurement and treatment effects 9 Give a complete and detailed description of method such that another researcher could replicate the procedure Write out steps Draw pictures Build a timeline 4. Infect with a cold virus 5. Lock treatment group in cold room 6. Have participants count symptoms 10 Multiple-Treatment Interference Hawthorne Effect Use multiple interventions in an uncontrolled manner Which treatment caused the change? Cannot generalize findings to situations with only one treatment Design studies that systematically separate the participants into different intervention groups I know you re watching me phenomenon When knowledge of the study influences performance by participants Aware of the aims or hypothesis eceive special attention (Not really the cause of the problem in the original Hawthorne study; see Gilbert, 1996) 11 12
Novelty and Disruption Effects Experimenter Effect Novel interventions may cause changes in DV simply because they are new or different Disruption effects occur when a disruption in routine inhibits performance Effectiveness or ineffectiveness of intervention due to the individual administering the intervention Importance of replication 13 14 Pretest Sensitization Posttest Sensitization Pretest may interact with intervention and produce different results than if the participants had not taken the pretest Pretest clues in the participants Applies to both groups Most common with self-report of attitudes or personality Posttest is itself a learning experience Participants performance affected by the the test Test extends the intervention Helps participants put the pieces together Practice in intervention was probably insufficient 15 16 Interaction of History and Treatment Effects Measurement of DV It may be difficult to generalize the finding outside of the time period in which the research was done Example: School Safety Study March to June 98: Intervene to improve safety in non-classroom settings May 98: Thurston Shooting Difficult to avoid! 17 esults limited to the particular mode of assessment used in the study Example: Project DAE Sample: 6 th graders Intervention: Police officer DAE s kids to say no to drugs Assessment officer plays a drug dealer: Wanna buy some drugs? esults: DAE 6 th graders say no DAE unrelated to later drug use 18
Interaction of Time of Measurement and Treatment Effects Threats to External Validity Population Validity eview esults may depend on the amount of time between intervention and assessment Last day of intervention A month later A year later Is assessment immediately after intervention better? Population validity: generalization from a sample to a specific larger group Threats Inability to generalize from experimental sample to a defined population, the target population The extent to which personological variables interact with treatment effects 19 20 Threats to External Validity, Ecological Validity eview Threats to Study Validity in Examples on Slides 18, 19, & 20 from Part I Ecological Validity: Generalization from your study to some other environment Generalization from research project to real world Lab to School Clinic to Home Threats Explicit treatment description Hawthorne effect Multiple treatment interference Novelty and disruption Experimenter effects Pretest sensitization Posttest sensitization Interaction of history & treatment Measurement of DV Interaction of time of measurement & treatment 21 a) Experimental treatment diffusion (i9), also called contamination. b) History (i1). c) Controls for differential selection (i6). d) A true placebo may control for experimental treatment diffusion (i9), compensatory equalization of treatments (i11) or resentful demoralization of the control group (i12). e) Multi-treatment interference (e2) threat to external validity. i = internal, p = population, e = ecological f) Not a typical threat to internal validity, but probably falls under maturation (i2). g) Differential selection (i6). h) Compensatory rivalry by the control group (i10). i) Generalization from sample to undefined population (i6; see also slides 5-8 in Part II). j) Statistical regression towards the mean (i5). k) Testing they become test wise (i3). l) Hawthorne effect (e3). 22 Group Designs Experimental Designs Experimental Show Causality Two or More Groups Comparisons Between Groups andom Assignment Equivalent Groups No Adjustment Unnecessary Manipulate IV Provide Intervention ne or More IVs Separate Groups for IVs Study Designs Quasi-Experimental Imply Causality Two or More Groups Comparisons Between Groups Not andom Assignment Nonequivalent Groups Must Adjust for Differences Manipulate IV Provide Intervention ne or More IVs Separate Groups for IVs Nonexperimental Show elationships Descriptive Correlational Causal Comparative Case Studies Single Group Pre-Post 23 Shows Causality equirements andomly assigned groups, two or more Manipulation of one IV (or more) Examples Post-only control group design Pre-post control-group design Solomon four group design Factorial designs 24
Post-nly Control Group Design Pre-Post Control Group Design Collect data () only after manipulation of the IV Interv n: X or X 1 Control: <blank> or X 2 Internal (causation): differential mortality External (generalization): potentially many X Alternate Specification X 1 X 2 Data collection () before & after intervention (X) Internal: attrition, treatment diffusion External: pretest sensitization, novelty X 25 26 Solomon 4-Group Design Factorial Designs Combination of post-only and pre-post designs Ideal but difficult to use equires larger sample size Cumbersome analysis Internal (causality): attrition External (generalizability): experimenter, disruption, but not pretest sensitization X X Similar to pre-post, but with two IVs X = intervention (drug) Y = setting (cold room) Internal: mortality External: interaction of testing & treatment X 1 X 1 X 2 X 2 Y 1 Y 2 Y 1 Y 2 27 28 Unit of Analysis: Individuals Unit of Analysis: Groups Standard procedure Sample individuals andomize individuals to groups Unit of analysis = individuals Analyze individuals Inference based on individuals But what if you cannot recruit and assign individuals, only intact groups? 29 Challenges to standard procedure Can only recruit intact groups, say, classrooms Intervention applies to groups, such as schools or communities Alternative procedure andomize groups (e.g., classrooms, cities) Unit of analysis = groups Intervention applies to groups Analyze group means (other options available) Inference based on groups equires more groups & more people 30
Quasi-Experimental Designs Static Group Comparison Quasi means resembling equirements Two or more groups, Not randomly assigned Manipulation of one or more IVs Examples Static-group comparison design Nonequivalent-group design Interrupted time series design Exactly like a post-only design, but no random assignment Threats to validity (examples) Internal: differential selection, mortality External: interaction of selection and treatment Not andom Design X 31 32 Nonequivalent Group Designs Interrupted Time Series Design Similar to experimental pre-post design, but not randomized Internal (causality): differential selection External: Interaction of testing and treatment Experimenter effect Not andom X 33 X Many assessments with IV in the middle Threats to Validity (Examples) Internal (causality): history, maturation External (generalizability): interaction of testing and treatment Contrast with true single-case research, which can achieve good internal validity (single-subject research sequence highly recommended) 34 egression Discontinuity Nonexperimental Designs Two-group design Assign to condition based on pretest Cut score must be used for assignment Accounts for selection bias Ex: Assign all students reading below 20 on Beck Depression Inventory to intervention Discontinuity in regression egress pretest on posttest Test for discontinuity at assignment cut score Show nly elationships equirements: Few Examples Descriptive Correlational Causal comparative design Case studies Single-group pre-post 35 36
Correlational Designs Causal Comparative Designs Also: ex post facto studies Correlation does not imply causation Determining the relationship between two variables Example: Teacher training and student performance in 1 st grade Variable 1: Hours spent in practicum Variable 2: Students reading scores 37 Studies temporal relationships Suspected cause: past event or experience Likely effect: present event or state Experimental control is not possible Shows relationships over time Cannot to establish causality Example: Gang membership School dropout may lead to gang membership Alternatives: poor school environment leads to both dropout and gang membership 38 ne group pretest-posttest Group Design eview Assess, intervene, assess a bad idea Threats to Validity (Examples) Internal Design History Maturation Testing X Interaction of selection and other factors External Interaction of testing and treatment Interaction of selection and treatment How do design types differ? Experimental? Quasi-experimental? Nonexperimental? Valid experiments What is internal validity? What is external validity? 39 40 Single-Case esearch (a digression) Experimental Validity eview Powerful, flexible, and parsimonious Very different from group designs 1 to 5 participants Each participant serves as his or her own control May achieve excellent internal and external validity with appropriate design Many experimental designs Multiple baseline ABAB Alternating treatments Beyond the scope of the current presentation Key sources Kennedy, Single Case Designs for Ed. esearch, 2005 Zhan & ttenbacher, Single subject research designs... in Disability and ehab., 2001, 23(1) Carr, A classroom demonstration of singlesubject research designs in Teaching of Psychology, 1997, 24(3) Fisher, Kelley, & Lomas, Visual aids and structured criteria for... single-case designs in JABA, 2003, 36(3) 41 Internal Validity Valid Statements about causality Can we draw conclusions about cause? External Validity Valid statements about generalization Can we expect the same results at other places or times, with other people, & with the intervention we reported? 42
Design eview the Plan Design: esearch Question Hypothesis Statement, esearch Question Design overview: timeline or design figure Participants: sampling & recruitment Intervention (IV): theory, implementation, fidelity, strength Data collection (DV): Measures with reliability & validity Carefully identify procedures & timing Intended analysis & power Critique: strengths & weaknesses 43 elationship between IV(s) and DVs Identifies the effect of the IV on the DV for the study sample Must represent falsifiable hypotheses Suggests empirical tests Implies measurable, well defined DVs State clearly and unambiguously (Kerlinger & Lee, 2000) 44 Design: verview & Sample Design: Independent Variable verview Draw design picture or timeline Define relationships among variables Sample To whom do I want to generalize? How will I sample participants? How do I assign participants? How large of a sample do I need? perational definition of IV Expected strength of IV Fidelity of Implementation the extent to which the treatment conditions, as implemented, conform to the researcher s specifications (G, B, & G, 1996, p. 481) Also called Treatment Fidelity 45 46 Design: Dependent Variable Design: Analysis & Critique Choose measures carefully Borrow measures in research literature Create new, which usually requires considerable pilot work eport reliability & validity Carefully identify procedures & assessment schedule 47 Choose an analysis method Factors to Consider esearch question Type of data collected Number of groups: treatment condition as well as other groups Type of design (e.g., pre-post, correlational) eport Power coming next Critique: strengths & weaknesses 48
Design: Power Analysis Design: Power Analysis (cont d) How large must my sample be? A big question Assumes true differences: how likely will we see them? eality & our glimpse of it eal world: not known ur sample: what we know We try to infer reality by what we know from our sample 49 A step back: two realities eality I: no difference in real world Assumption for statistical tests (not power) Type I error: we accidentally concluding we have a difference when it does not really exist The chance of Type I error: p-value or alpha (α) eality II: differences exist in real world Assumption for power Type II error: accidentally concluding we have no difference when one exists in reality The chance of Type II error: called beta (β) 50 Error Types Design: Computing Power Probability of Type I error: α Probability of Type II error: β What We Know from ur Sample (Test esults) No Difference Accept Null Groups Differ eject Null eal World Unknown to Us No Difference Correct Type I Error (α) Difference Type II Error (β) Correct 51 Important considerations Analysis: t-test, ANVA, correlation, etc. Alpha level: level of statistical significance chosen to reject null, often.05 or.01 Direction of hypothesis: one- or two-tailed Expected magnitude of effect, effect size Desired power: 1 β, often 80%, so β =.20 Attrition rate Consult Cohen (1988) or similar source for power tables or get G*Power (free) 52 Questions? esearch designs? Internal validity? Power? External validity? 53