More on Experiments: Confounding and Obscuring Variables (Part 1) Dr. Stefanie Drew

Transcription

1 1 More on Experiments: Confounding and Obscuring Variables (Part 1) Dr. Stefanie Drew

2 2 Previously, On Research Methods Basic structure of an experiment Now in more detail Internal validity issues Control Treatment Treatment A Explanation of null results

3 3 Reflect When interrogating an experiment, which is the most important validity to focus on? Experiment Which validity?

4 4 Remember: Threats to Internal Validity Three most common threats (involve alternative explanations) Design confounds Alternative explanation: another variable systematically varied with the IV Selection effects Alternative explanation: different IV groups have different types of participants Order effects (in within group design) Alternative explanation: outcome due to order of presentation of the IV levels And then there are more

5 5 What our book calls the really bad experiment One-group, pretest/posttest design One group of participants Single pretest Treatment Single posttest Participants Pre-test Treatment Posttest

6 6 One-group, pretest/posttest design Threats to Internal Validity Maturation Threats History Threats Regression Threats Attrition Threats Testing Threats Instrumentation Threats Combined Threats the good news: can avoid these by changing design!

7 7 Experimental Example Professor Z gives his students a pre-test for nervousness at the beginning of the semester. He finds that the students in the class are very nervous, so decides to start giving out candy at each lecture. After two weeks, he measures them again and finds that they are significantly less nervous. Did his candy therapy work? Students Nervousness semester start Gives Candy Nervousness Test after 2 weeks

8 8 Professor Z s Results 30 Anxiety Score Beginning of Semester Two Weeks Into Semester

9 9 Maturation Threats Maturation: change in experiment group that emerges spontaneously over time Experimental Example: What if the students improved on their own once they got used to the start of the semester? Spontaneous remission: occurs when disorders get better for unknown reason Image adapted from enology.umn.edu/2011/07/29/version/ on April 14, 2013

10 10 Image adapted from conniearnold.blogspot.com/2012/02/ups-and-downs-in-life.html on April 2, 2014 Preventing Maturation Threats Problem of one group pretest/posttest No way of knowing if improvements are due to maturation or treatments Experimental Example: Professor Z gives candy to only half of the class, and compares the two groups after two weeks Comparison Group If treatment group improves and comparison group does not, can rule out maturation effects

11 11 Image adapted from on April 2, 2014 History Threats History threats: historical or external event that happens to most members in treatment group at same time of treatment Must effect (nearly) everyone in the treatment group at the same time of treatment Experimental Example: What if campus promotes a program for inner peace during those two weeks?

12 12 Image adapted from conniearnold.blogspot.com/2012/02/ups-and-downs-in-life.html on April 2, 2014 Preventing History Threats Problem of one group pretest/posttest No way of knowing if improvements due to some historical event that effects everyone Comparison Group If treatment group improves and comparison group does not, can rule out history effects Experimental Example: Professor Z gives candy to only half of the class, and compares the two groups after two weeks

13 13 Regression Threats Regression threat: relates to regression to the mean, where any extreme measurement will be closer to average next measurement Everyday example: due to random factors, you have a particularly horrible day (oversleep, exit ramp is closed, drop your iphone, forgot your wallet at home) Experimental Example: What if, at time of pretest, students could have been having particularly high nervousness mood so will regress to mean for posttest Next day won t be as bad! Image adapted from tabmathletics.com/welcome-to-tabmathletics/regression-to-the-mean/ on April 14, 2013

14 14 Preventing Regression Threats Problem of one group pretest/posttest No way of knowing if improvements due to having a less extreme day on the posttest Comparison Group If both groups have equal extremes, but treatment group improves more, can rule out regression effects Experimental Example: Professor Z gives candy to only half of the class, and compares the two groups after two weeks Observe Pattern of Results* Image adapted from conniearnold.blogspot.com/2012/02/ups-and-downs-in-life.html on April 2, 2014 Group A Group B

15 15 Regression to the mean: Observing pattern of results No regression b/c both groups started equally extreme Regression possible b/c therapy started more extreme than non therapy No regression b/c regression can t make extremes cross mean to other extreme

16 16 Attrition Threats Attrition (aka mortality): systematic dropout of certain type of participant Experimental Example: What if the really nervous people couldn t take the class and dropped out? Threat to internal validity if it happens in a systematic manner E.g. only certain type of participants drop out Image adapted from blog.verishow.com/?p=306 on April 14, 2013

17 17 Preventing Attrition Threats Researchers Identify and Correct E.g. can remove dropped out participants from the group and retest Experimental Example: Professor Z removes the scores of those students that dropped the class when calculating the average pretest score

18 18 Testing Threats Wait a minute I remember this from before. Testing threat: occurs when participant changes over time from having been tested before A kind of order effect People can become more sensitized People may get tired, bored Experimental Example: After taking the pre-test, students figure out test is about nervousness, so try to improve for second test Image adapted from kirwaninstitute.osu.edu/standardized-testing-and-stereotype-threat/ on April 14, 2013

19 19 Preventing Testing Threats Use post-test only design Use alternate versions of the test for pretest and posttest Experimental Example: Professor Z gives candy to only half of the class, and compares the two groups after two weeks Comparison Group If comparison group also takes pretest and posttest, but treatment group improves more, can rule out testing effects

20 20 Image adapted from on April 2, 2014 Instrumentation Threats Instrumentation threats (aka instrument decay): measurement instrument changes over time Experimental Example: Professor Z becomes more lenient in rating nervousness for the second test

21 21 Prevention of Instrumentation Threats Use posttest only design Ensure pretest and posttest are equivalent Collect data from each instrument to be sure calibrated at same time Experimental Example: Professor Z creates a coding manual he follows for assessing results from teach test Train the coders multiple times, use coding manual Counterbalance order of tests Give some students version A and some B at pretest; then using version B and A at posttest

22 22 Combined Threats Selection-history threat: outside event systematically affects most people in one level of variable Experimental Example: Campus holds a wellness event and contacts people that they know eat junk food (including candy group) and gives them free yoga DVD Selection-attrition threat: only one experimental group experiences attrition Experimental Example: Most nervous people drop out only from candy group because they don t want to increase their anxiety with sugar

23 23 Any Experiment Threats to Internal Validity (even with comparison group) Observer bias Demand characteristics Placebo effects

24 24 Observer Bias & Demand Characteristics Observer bias: researchers expectations influence Affects internal validity Why? Affects construct Why? validity Demand characteristics: participants guess what study is about Oh! They re looking at how social I am! I should say Image adapted from on April 2, 2014

25 25 Preventing Observer Bias & Demand Characteristics Double blind studies: neither the participant nor the researchers evaluating know who is in treatment and who is in the comparison group Can also do masked design (aka blind study ) Only observers don t know which group participants are in Image adpated from asserttrue.blogspot.com/2013/03/how-blind-is-double-blind.html on April 14, 2013

26 26 Placebo Effects Placebo effects: E.g. sugar pills, placebo psychotherapies Not imaginary: can be physical or psychological If there is a placebo effect, no therapy group will not improve as much as placebo Image adapted from thoughtbroadcast.com/2011/02/13/the-placebo-effect-it-just-gets-better-and-better/ on April 14, 2013

27 27 The Undesired Dozen Internal Validity Threat Design confound Selection effect Order effect Maturation History Regression to the mean Attrition Testing Instrumentation Observer bias Demand characteristics Placebo effects Summary

28 28 So why even use experiments? Image adapted from onlyhdwallpapers.com/high-definition-wallpaper/question-mark-desktop-hd-wallpaper / on April 14, 2013

29 Exit Ticket EXIT