Other Designs. What Aids Our Causal Arguments? Time Designs Cohort. Time Designs Cross-Sectional

Transcription

1 Statistics in & Things to Consider Your Proposal Bring green folder of readings and copies of your proposal rough draft (one to turn in) Other Designs November 26, What Aids Our Causal Arguments? Theory "causes certainly are connected to effects; but this is because our theories connect them, not because the world is held together by cosmic glue. The world may be glued together by imponderables, but that is irrelevant understanding causal explanation." Hanson, Temporal Elements Design "No causation without manipulation" Rubin & Holland Classifying Other Kinds of Studies Purpose Descriptive Prediction Explanation Manipulation of the Independent Variable Experiment vs Non- Experiment When and How Often Data are Collected Longitudinal Time Series On whom are data collected 3 4 Time Designs Cross-Sectional s are drawn from different points along a developmental continuum but at one time. Time Designs Cohort Different samples from the same population are drawn at each time point. First Year New (2-5 yrs.) Experienced (6-15 yrs.) Veteran (16 yrs. +) Pre- Service Pre- Service First Year New (2-5 yrs.) Experienced (6-15 yrs.) Veteran (16 yrs. +)

2 Time Designs Longitudinal (Growth or Change) Studies The same individuals are repeatedly measured Singer et al. (1999) Pre- Service First Year New (2-5 yrs.) Experienced (6-15 yrs.) Veteran (16 yrs. +) Time Series Designs Used when time is an explicit element. 7 8 A Scientific Process Question Defining the Problem Review A rticulate Define Literature Theory H ypothesis Testing the Subject sam pling Instrum entation C ollecting D ata design Choosing analyses Piloting Conducting analyses Results or Findings Statistical Conclusion Validity The ability to trust what the statistics tell us Statistical Power The ability to detect true relationships Assumptions of Statistics The conditions under which the statistic will give us accurate results Parametric versus Non-parametric tests Unreliability Conclusions 9 10 Statistical Significance Testing The Problems with SST We misunderstand what it does tell us. It does not tell us what we want to know. We often overemphasize SST

3 Four Important Questions 1. Is there a real relationship in the population? Statistical Significance 2. How large is the relationship? Effect Size or Magnitude 3. Is it a relationship that has important, powerful, useful, meaningful implications? Practical Significance 4. Why is the relationship there??????? SST is all about... Sampling Error The difference between what I see in my sample and what exists in the target population. Simply because I sampled, I could be wrong. This is a threat to Internal Validity Population Population 1 Population 2 How it works: Treatment Situation A Control Treatment Situation B Control Carver, R. P. (1978). The case against statistical significance testing. Harvard Educational 1. Assume sampling error occurred; there is no relationship in the population. 2. Build a statistical scenario based on this null hypothesis 3. How likely is it I got the sample value I got when the null hypothesis is true? (This is the fabled p-value.) Review, 48, How it works (cont d): How unlikely does my result have to be to rule out sampling error? alpha (α). If p < α, then our result is statistically rare, is unlikely to occur when there isn t t a relationship in the population. What it does tell us What is the probability that we would see a relationship in our sample when there is no relationship in the population? Can we rule out sampling error as a competing hypothesis our finding?

4 What it does not tell us Whether the null hypothesis is true. Whether our results will replicate. Whether our research hypothesis is true. How big the effect or relationship is. How important the results are. Why there is a relationship. Carver s Scientific Method Consider Evidence Alternative Including the Null YES Do data support Collect Data research hypothesis? No NO Evidence 19 Carver, R. P. (1978). The case against statistical significance testing. Harvard Educational Review, 48, Carver s Corrupt Scientific Method Evidence YES Test Null Statistical Collect Data Significance? No NO Evidence Consider Alternative Excluding the Null Carver, R. P. (1978). The case against statistical significance testing. Harvard Educational What To Do Always use the phrase statistically significant Always report effect size or measures of association in addition to SST. Replicate and/or Cross-validate What does it mean? -- Discuss Practical Significance Review, 48, Statistical Methods in Psychology Journals (Wilkinson, 1999) Nice article Check your proposal to make sure it addresses the points made in this article Design Make design clear Do not cloak the design Each m of research has its own strengths, weaknesses and standards of practice. Population Define clearly How was the population sampled? Does it represent the target population? Describe recruitment procedures Emphasize inclusion and exclusion criteria If stratified explain why Note sample sizes subgroups Assignment Random vs. Nonrandom Control of confounds

5 Measurement Variables Explicitly state and explain how they are measured How do they vary? Instruments What evidence exists the usage of the scores? Reliability» Internal consistency, test-retest, etc. Validity» Predictive, concurrent, etc. Procedures Attrition How will you prevent it? How will you control threats to internal validity? Power and size Do you have a large enough sample? Based on previous studies Pilot work Power analysis Results Bee presenting results Report Complications Protocol violations Missing data Look at your data Make sure that there are no errors in the data Look at histograms, means, etc. Analysis KISS Keep it simple ****** Design studies that you know how to analyze Beware of computer programs You can perm incredibly complex analyses Assumptions Check them Normality, independence, etc tests Report p values Effect sizes Interval estimates Causality Be careful Alert reader to plausible rival hypotheses Discussion Try not to Overgeneralize Tear your study apart Relate to other studies Conclusions Speculation Use sparingly Do not interpret study s importance independent of other studies

6 Random Thoughts on the (In)credibility of Educational-Psychological Intervention (Levin, 2004) Random Selection vs. Random Assignment How are they different? The concept of evidence Show me the data (C)omparison (A)gain and again (R)elationship (E)liminating other plausible explanations Medical Comparison (Levin, cont.) Clinical Trials Phase I: Maximum tolerated dose Side effects, etc. Phase II: Biological activity Does the drug have the effect of interest? Phase III: Is the drug, or treatment, effective when compared to an alternative? Phase IV: Is the drug, or treatment, effective in the long run? In Pairs: Each person should: Summarize his/her study in 3 minutes or less. Discuss explicitly the role of causality in each study (5 Minutes). What in the design aids that causal argument? What inhibits the causal argument? Then read your partners methods section Does it make sense? Are steps in logical order? Does it use proper terminology? What threats to internal and external validity exist? Repeat with the other person. 33 Basic Outline Problem to be investigated Purpose & Justification Literature review Theory and definitions Methods Sampling Including human subjects. Instrumentation How you will measure each IV and DV.» Be sure to identify IVs and DVs Reliability & Validity Design Experiment, correlational, etc. Procedural Detail What will happen and when it will happen. Data Analyses Strengths and Limitations Internal and External. 34 Next Week Due: Final Paper Study Guide Questions Write one each chapter. 35 6