Experimental Design
Previous Example New Tradition
Goal? New Tradition =?
Challenges Internal validity How to guarantee what you have observed is true? External validity How to guarantee what you have observed can be applied in other situations? Construct validity To what extend does your study measure the construct of interest?
Threats to Internal Validity: Biased Samples Group selection Self-identified Assigned by researchers Assigned arbitrarily Are the groups equal? Choosing groups based on their differences results in having groups that are different Difference should be less than 5%.
How About Matching? Almost impossible to perfectly match individual participants No identical participants Two many relevant factors/variables Some we know, but some we don t. Matching on pretest scores Selection by maturation interactions: participants growing in different ways How can you make sure equal scores mean subjects are equal?
So Trying to keep everything except the treatment constant is very difficult, if not impossible Selection is a big problem Internal validity is threatened. Only option is to rule out extraneous variables. Use random assignment Identify extraneous variables and then try to rule them out
Dilemma Attempts to rule out threats to internal validity may hurt external validity Not studying a heterogeneous group of participants Not studying participants in a naturalistic setting
You Need To Be cautious about accepting cause-effect statements If the study is not an experiment, the study s internal validity is probably threatened by at least one of 8 threats to validity argued by Campbell and Stanley
Some Threats To Internal Validity History: events that occur between treatments Testing: changes resulting from the practice and experience gained in the test Instrumentation: the change of the way subjects are measured Regression effects: selection of subjects based on extreme scores Mortality: the loss of subjects Selection: groups are selected differently Maturation: treatment effects are really due to natural growth or development Selection by maturation interactions: participants growing in different ways
Experimental Study Experimental designs can rule out most threats to internal validity.
What Is An Experimental Study?
Basic Logic Get a hypotheses Causal relationship Independent Variable Dependent Variable Manipulate the independent variable (IV) and measure the dependent variable (DV) Have experimental and control groups Similar, but treated differently Random selected Statistically analyze the difference Significant result or not?
Key Issue Apply treatment on one group but not the other. The only difference between two groups is the treatment. If DV shows any difference, it is due to the treatment! All other factors should be the same, at least in theory. Task, observation instrument, procedure. Subjects
HOW CAN WE GUARANTEE SUBJECTS IN TWO GROUPS ARE THE SAME?
Group Assignment Random assignment Select subjects for two groups from the same subject pool. Assign them to two groups randomly. Without random assignment, you do not have an experiment
Subject Recruitment General population Representative enough Considering confronting factors Subject pools Psych Department Methods to reach subjects Emails, flyers, ads in bulletin boards, class visit Incentive Always have backup subjects No show, failed study, in sufficient subjects, etc.
7 Steps to Assign Participants Randomly
Other Methods Toss a coin Tail experimental group Head control group Excel Generate random numbers and then sort.
Data Collection and Analysis Knowing what to measure Variables Calculating means Sample means Comparing sample means Statistic methods Other data Subjective judgment
Independent Variable Treatment Simple experiments Treatment: yes and no Experimental group: applied Control group: not applied Treatment: different levels and no Multiple experimental groups and one control group No treatment Could be a threat to construct validity
Manipulation IV Making the treatment observable and significant
Statistic Analysis Rule out random errors Errors = random errors + systematic errors by the treatment Statistically significant results You can declare the causal effects. May not be the same direction you expected Non-significant results You didn t find it But it doesn t mean it doesn t exist
Simple Experiment t test Could be between- or within-subject design
Questions Raised by Results Questions raised by non-significant results Enough participants? Participants homogeneous enough? Experiment sufficiently standardized? Data coded carefully? DV sensitive and reliable enough?
Pilot Study Testing various aspects of study Subjects IV DV Procedure and protocols With a smaller group of subjects With the SAME procedure and protocols Must have all experimental materials ready before the pilot study Issues to look at Procedure and protocols (e.g., instruction, task, etc.) Data collection method The number of subjects required
Questions Raised by Results (cont ) Questions raised by significant results External validity Do results generalize to other levels of the IV? Do other variables moderate the IV s effect? Construct validity (Was control group good enough?) Simple two-group experiments may not good enough Need multiple-group experiments to Improve construct validity Improve external validity Answer more questions
No Yes No Med High Amount of Treatment
Benefits of Multi-Group Experiment Better ability to estimate the effects of different amounts (levels) of a treatment Better ability to rule out confounding variables May help discover significant relationships May help more accurately map the functional relationship Summary: Multi-level experiments have more external validity than simple experiments
A straight line is not the only line that you can draw between two points.
Using Multiple Groups to Improve Construct Validity of Experiments Multi-level experiments can call attention to a possible problems with your control group* Using multiple imperfect control groups to make up for not having the perfect control group
Group Assignment in Multi-Group Experiment Similar procedure as in simple experiment Using random table Using a dice (rather than a coin) Using Excel
Statistics Methods t tests ANOVA
Control Group No treatment at all? May create noise Example: medicine Taking medicine may create psychological effect. Subjects may guess what you try to do Solution Provide "fake" treatment Similar to real treatment, in form only. Placebo pills Systems with similar appearance, but different treatments.