Review and Wrap-up! ESP 178 Applied Research Methods Calvin Thigpen 3/14/17 Adapted from presentation by Prof. Susan Handy

Transcription

1 Review and Wrap-up! ESP 178 Applied Research Methods Calvin Thigpen 3/14/17 Adapted from presentation by Prof. Susan Handy

2 Final Proposals Read instructions carefully! Check Canvas for our comments on assignment 4 later this week Use the proposal template! Talk with me or Dillon if you need help!

3 As a research consumer Do I believe it? Do I believe that they did a good job measuring things? What errors could there be in their measurements? Do I believe their explanation? Does it work they way they say it does? Is it a cause-and-effect relationship? Do I believe that these results hold beyond this study? Would it work this way in a different context? Measurement Validity Internal Validity External Validity

4 Research example UC Berkeley study on Does money make you mean? New York Times Privilege, Pathology and Power 1/1/16

5 Fancy cars are less likely to stop for pedestrians. Do you believe that being rich causes a lack of empathy? If you want to prove causality, what would you do? The Monopoly experiment note use of observations and interviews

6 RAZ: OK, but how does that happen? I mean, how does money change you? Like, say you come into a lot of it when you're like, you know, 50, what would happen? PIFF: Well, it would, for one, mean that you can afford a different kind of home. Maybe it means you can afford a bigger home where the people in your family would all occupy separate bedrooms. You'll have a bigger yard, potentially, or more space between your house and other people's homes. When you go to work, you may be less likely to take that bus or that carpool. When you get to work, you may be more likely to have a position of someone who's, say an overseer of other people as opposed to someone who works with one another in teams. And with that sort of increased self-focus, that increased control, you become less attuned to other people in your environment, less cooperative, less ethical, a whole slew of other things.

7 Types of Data Collection and Analysis Type of Question Explanatory Exploratory Type of Research Deductive Inductive Type of Data Quantitative Qualitative Sampling Probabilistic Random Representative Larger Non-probabilistic Convenient, purposeful Illustrative Smaller Data Collection Surveys Available data Observation In-depth interviews Focus groups Observation Data Analysis Statistical analysis Content analysis But it is not always so clear-cut!!!

8 Types of Questions Exploratory A question but no prior ideas about answer Not testing a hypothesis but starting with a framework Hypothesis testing but trying to build understanding Explanatory Accept or reject null hypothesis

9 Research Designs Cross-sectional Longitudinal Experimental Case Study Primary purpose Establish association between IV and DV Track changes over time in IV and DV Test effect of treatment (IV) on DV Understand relationship between IV and DV

10 Research Designs Quantitative Data Qualitative Data Cross-sectional X X Longitudinal X X Experimental X X Case Study X X

11 Quantitative Data by Research Design Surveys Observation Existing Data Cross-sectional X X X Longitudinal X X X Experimental X X X Case Study X X X Natural experiments Intervention studies

12 Descriptive Statistics Ratio variable Central tendency Mean, Median Mode Variation Standard deviation Variance Percentiles Statistics: Number of times children played outside in last 7 days N Valid 1000 Missing 0 Mean 6.1 Median 6 Mode 6 Std. Deviation 2.4 Variance 5.9 Percentiles

13 Descriptive Statistics Nominal or ordinal variable Frequency distribution graph or table Do you live on a cul-de-sac? Not at all Somewhat Very Entirely N % 45% 10% 5% 40%

14 Inferential Statistics Which regression model do you choose? Linear regression DV is a continuous ratio, without any nearby upper or lower boundaries Examples Height mw of electricity consumed by a household

15 Inferential Statistics Which regression model do you choose? Binomial logistic regression DV is either binary nominal yes/no bike/not bike purchased solar panels/did not Installed water-efficient toilt/did not integer ratio that could be considered a sequence of trials # of days played outside in the last week each day is a yes/no trial, adds up to between 0 and 7

16 Inferential Statistics Which regression model do you choose? Poisson regression DV is an integer ratio Must be greater than or equal to 0 No upper boundary Examples # of crashes at an intersection in a day # of toilet flushes in a day # of times a light is turned on

17 Inferential Statistics How to interpret a linear regression coefficient s: Estimate Standard error Linear Regression yy ii = αα + ββxx ii + εε ii εε ii ~NNNNNNNNNNNN 0, σσ 2 Ratio Nominal (binary) Nominal (multiple) Coefficients: Estimate Std. Error t value Pr(> t ) (Intercept) Ratio <2e-16 *** --- Signif. codes: 0 *** ** 0.01 * Residual standard error: on 998 degrees of freedom Multiple R-squared: 0.539, Adjusted R-squared: F-statistic: 1167 on 1 and 998 DF, p-value: < 2.2e-16 Coefficients: Estimate Std. Error t value Pr(> t ) (Intercept) <2e-16 *** Binary nominal <2e-16 *** --- Signif. codes: 0 *** ** 0.01 * Residual standard error: on 998 degrees of freedom Multiple R-squared: 0.316, Adjusted R-squared: F-statistic: on 1 and 998 DF, p-value: < 2.2e-16 Coefficients: Estimate Std. Error t value Pr(> t ) (Intercept) < 2e-16 *** Nominal e-12 *** Nominal e-07 *** Nominal < 2e-16 *** --- Signif. codes: 0 *** ** 0.01 * Residual standard error: 1.99 on 996 degrees of freedom Multiple R-squared: , Adjusted R-squared: F-statistic: 198 on 3 and 996 DF, p-value: < 2.2e-16

18 Inferential Statistics How to interpret a binomial logistic or poisson regression coefficient s: Estimate Standard error Binomial Logistic Regression Ratio Nominal (binary) Coefficients: Estimate Std. Error t value Pr(> t ) (Intercept) Ratio <2e-16 *** --- Signif. codes: 0 *** ** 0.01 * Residual standard error: on 998 degrees of freedom Multiple R-squared: 0.539, Adjusted R-squared: F-statistic: 1167 on 1 and 998 DF, p-value: < 2.2e-16 Coefficients: Estimate Std. Error t value Pr(> t ) (Intercept) <2e-16 *** Binary nominal <2e-16 *** --- Signif. codes: 0 *** ** 0.01 * Residual standard error: on 998 degrees of freedom Multiple R-squared: 0.316, Adjusted R-squared: F-statistic: on 1 and 998 DF, p-value: < 2.2e-16 Poisson Regression Nominal (multiple) Coefficients: Estimate Std. Error t value Pr(> t ) (Intercept) < 2e-16 *** Nominal e-12 *** Nominal e-07 *** Nominal < 2e-16 *** --- Signif. codes: 0 *** ** 0.01 * Residual standard error: 1.99 on 996 degrees of freedom Multiple R-squared: , Adjusted R-squared: F-statistic: 198 on 3 and 996 DF, p-value: < 2.2e-16

19 Qualitative Data by Research Design Interviews Focus Groups Observation Cross-sectional X X X Longitudinal X X X Experimental X X X Case Study X X X Individuals as cases In-depth interviews Observe individuals Communities as cases Key informant interviews Talk to community Observe community

20 Research Designs Internal Validity External Validity Cross-sectional L H Longitudinal M H Experimental H L Case Study H L Depending on sampling

21 For cross-sectional, focus on Criterion Association Non-spurious Time order Causal Mechanism Context Explanation If cause happens, effect happens; if no cause, no effect. No extraneous third variable that can explain association. Cause comes before effect. Logical explanation for how cause leads to effect. Understand the conditions under which the relationship holds

22 Is there a third variable that influences both IV and DV and makes it look like they re related when they re not?????

23 Is it possible that the effect comes before the cause, i.e. reverse causality? 23

24 For experiments, focus on the most likely ones Threat Selection bias Endogenous change History effects Contamination Treatment misidentification Mortality Definition when characteristics of treatment and control groups differ when subjects develop or change independent of treatment testing, maturation when something occurs during the experiment that influences outcomes when one group is aware of the other group and is influenced a process the researcher is not aware of is responsible for apparent effect of treatment expectancy effect, placebo effect, Hawthorne effect Participants drop out at differential rates by group for systematic reason

25 For external validity, think about how your sampling approach affects both Population Participants A Different Population

26 See handout on stratified versus cluster sampling

27 Don t forget to factor in your response rate! Random sampling Target Population Sampling frame error How good is coverage? Systematic sampling error?

28 For all studies, think about how to test for Measurement reliability = Consistent Measurement validity = Accurate

29 Reliability testing Do you get the same answer the second time? Test Re-test NOTE: You do not want to make your entire sample take the survey twice! NOTE: Before-and-after surveys cannot be used as a test-retest test!

30 Validity testing Translational How well is theoretical concept translated into measure? Face Validity makes sense on its face Content Validity fully covers the concept

31 Validity testing Criterion Does measure behave the way it should? Convergent Validity: Compare measure to a different measure of the same concept Does your child play outside frequently or infrequently? vs. How many times did your child play outside last week? vs. GPS devices to measure movement and location Second question in the survey

32 Measurement validity notes The goal is to establish the accuracy of individual measurements of the DV or IV Compare individual measure to gold standard for that individual Note the difference: Validity testing: different questions about the same thing on the same survey Reliability testing: the same question on different survey occasions Comparing your results to previously published studies is good to do, but it is not a measurement validity test!

33 Midterm 2 Study sheet and practice exam on website! Read the Brown and Handy article! Bring the article to the exam! No notes or highlighting!

34 Midterm 2 topics see study sheet! Types of errors Survey methods Scales Quantitative analysis Qualitative research Other methods Errors of observation Errors of non-observation Survey administration Survey questions As response categories As composite measures Descriptive statistics Inferential statistics Data collection methods Data analysis methods Available data Census, ACS! Case studies

35 As a research consumer Do I believe it? Do I believe that they did a good job measuring things? What errors could there be in their measurements? Do I believe their explanation? Does it work they way they say it does? Is it a cause-and-effect relationship? Do I believe that these results hold beyond this study? Would it work this way in a different context? Measurement Validity Internal Validity External Validity Emphasis on second midterm

36 To do don t pack up yet! Thursday, 3/16 Friday, 3/17 Friday, 3/24 Second midterm! READ ARTICLE BRING ARTICLE NO NOTES OR HIGHLIGHTING! You will submit the article with your exam. Bring a copy of your proposal: Workshop on proposals in section! Read our comments on Assignment 4 Final Proposal due at 6pm Early submissions encouraged! Submit via Canvas Please complete the course evaluation!