Previous Example. New. Tradition

Transcription

1 Experimental Design

2 Previous Example New Tradition

3 Goal? New Tradition =?

4 Challenges Internal validity How to guarantee what you have observed is true? External validity How to guarantee what you have observed can be applied in other situations? Construct validity To what extend does your study measure the construct of interest?

5 Threats to Internal Validity: Biased Samples Group selection Self-identified Assigned by researchers Assigned arbitrarily Are the groups equal? Choosing groups based on their differences results in having groups that are different Difference should be less than 5%.

6 How About Matching? Almost impossible to perfectly match individual participants No identical participants Two many relevant factors/variables Some we know, but some we don t. Matching on pretest scores Selection by maturation interactions: participants growing in different ways How can you make sure equal scores mean subjects are equal?

7 So Trying to keep everything except the treatment constant is very difficult, if not impossible Selection is a big problem Internal validity is threatened. Only option is to rule out extraneous variables. Use random assignment Identify extraneous variables and then try to rule them out

8 Dilemma Attempts to rule out threats to internal validity may hurt external validity Not studying a heterogeneous group of participants Not studying participants in a naturalistic setting

9 You Need To Be cautious about accepting cause-effect statements If the study is not an experiment, the study s internal validity is probably threatened by at least one of 8 threats to validity argued by Campbell and Stanley

10 Some Threats To Internal Validity History: events that occur between treatments Testing: changes resulting from the practice and experience gained in the test Instrumentation: the change of the way subjects are measured Regression effects: selection of subjects based on extreme scores Mortality: the loss of subjects Selection: groups are selected differently Maturation: treatment effects are really due to natural growth or development Selection by maturation interactions: participants growing in different ways

11 Experimental Study Experimental designs can rule out most threats to internal validity.

12 What Is An Experimental Study?

13 Basic Logic Get a hypotheses Causal relationship Independent Variable Dependent Variable Manipulate the independent variable (IV) and measure the dependent variable (DV) Have experimental and control groups Similar, but treated differently Random selected Statistically analyze the difference Significant result or not?

14 Key Issue Apply treatment on one group but not the other. The only difference between two groups is the treatment. If DV shows any difference, it is due to the treatment! All other factors should be the same, at least in theory. Task, observation instrument, procedure. Subjects

15 HOW CAN WE GUARANTEE SUBJECTS IN TWO GROUPS ARE THE SAME?

16 Group Assignment Random assignment Select subjects for two groups from the same subject pool. Assign them to two groups randomly. Without random assignment, you do not have an experiment

17 Subject Recruitment General population Representative enough Considering confronting factors Subject pools Psych Department Methods to reach subjects s, flyers, ads in bulletin boards, class visit Incentive Always have backup subjects No show, failed study, in sufficient subjects, etc.

18 7 Steps to Assign Participants Randomly

19

20

21

22

23

24

25 Other Methods Toss a coin Tail experimental group Head control group Excel Generate random numbers and then sort.

26 Data Collection and Analysis Knowing what to measure Variables Calculating means Sample means Comparing sample means Statistic methods Other data Subjective judgment

27 Independent Variable Treatment Simple experiments Treatment: yes and no Experimental group: applied Control group: not applied Treatment: different levels and no Multiple experimental groups and one control group No treatment Could be a threat to construct validity

28 Manipulation IV Making the treatment observable and significant

29 Statistic Analysis Rule out random errors Errors = random errors + systematic errors by the treatment Statistically significant results You can declare the causal effects. May not be the same direction you expected Non-significant results You didn t find it But it doesn t mean it doesn t exist

30 Simple Experiment t test Could be between- or within-subject design

31 Questions Raised by Results Questions raised by non-significant results Enough participants? Participants homogeneous enough? Experiment sufficiently standardized? Data coded carefully? DV sensitive and reliable enough?

32 Pilot Study Testing various aspects of study Subjects IV DV Procedure and protocols With a smaller group of subjects With the SAME procedure and protocols Must have all experimental materials ready before the pilot study Issues to look at Procedure and protocols (e.g., instruction, task, etc.) Data collection method The number of subjects required

33 Questions Raised by Results (cont ) Questions raised by significant results External validity Do results generalize to other levels of the IV? Do other variables moderate the IV s effect? Construct validity (Was control group good enough?) Simple two-group experiments may not good enough Need multiple-group experiments to Improve construct validity Improve external validity Answer more questions

34 No Yes No Med High Amount of Treatment

35 Benefits of Multi-Group Experiment Better ability to estimate the effects of different amounts (levels) of a treatment Better ability to rule out confounding variables May help discover significant relationships May help more accurately map the functional relationship Summary: Multi-level experiments have more external validity than simple experiments

36 A straight line is not the only line that you can draw between two points.

37 Using Multiple Groups to Improve Construct Validity of Experiments Multi-level experiments can call attention to a possible problems with your control group* Using multiple imperfect control groups to make up for not having the perfect control group

38 Group Assignment in Multi-Group Experiment Similar procedure as in simple experiment Using random table Using a dice (rather than a coin) Using Excel

39 Statistics Methods t tests ANOVA

40 Control Group No treatment at all? May create noise Example: medicine Taking medicine may create psychological effect. Subjects may guess what you try to do Solution Provide "fake" treatment Similar to real treatment, in form only. Placebo pills Systems with similar appearance, but different treatments.