Study Design. Svetlana Yampolskaya, Ph.D. Summer 2013

Transcription

1 Study Design Svetlana Yampolskaya, Ph.D. Summer 2013

2 Study Design Single point in time Cross-Sectional Research Multiple points in time Study only exists in this point in time Longitudinal Research Study begins here Study ends here

3 Cross-Sectional Research Design Study population No Disease Disease factor absent factor present factor absent factor present Cross-sectional research design involves observation of the study population, or a representative subset, at one specific point in time

4 Cross-Sectional Research Design Cross-sectional study utilizes the information about variables that represents what is going on at only one point in time Measures are administered at one point in time The goal is to describe phenomena Can be used for group comparison

5 Cross-Sectional Research Design Advantages Quick Descriptive role Examine associations Disadvantages Temporal associations not clear Limited research questions that can be answered Shows association, not causality Rare events a problem. For example, quickly emerging diseases

6 Longitudinal Research Design A longitudinal study involves repeated observations of the same variables over long periods of time Longitudinal research involves Focus on time/duration Focus on change (or stability) Focus on specific trends Wave 1 Wave 2 Wave 3 Wave4

7 Longitudinal Research Design Types of longitudinal studies Trend studies - measures change in some general population over time; Often used to study social trends Cohort studies - track changes in a group that shared similar experience; Comparing the effect of the same event on different groups Panel studies observing/interviewing the same people more than once; Information collected on the same cases at more than one point in time Time-series design requires multiple observations of the same individuals before the intervention and multiple observation after the intervention

8 Longitudinal Research Design Trends Studies Sample 1 Survey A Sample 2 Survey A Sample 3 Survey A Trend studies can be viewed as a repeated cross-sectional design, asking the same questions to different samples of the target population at different points in time All surveys within the same study use the same materials and protocol Each consecutive survey uses a new sample, but samples are similar

9 Longitudinal Research Design Trends Studies

10 Longitudinal Research Design Trends Studies Advantages Examine changes over time Identifying a trend No attrition problem Cover a variety of different characteristics within the studied group Disadvantages Limited research questions that can be answered Shows changes over time, not causality No control for contextual factors

11 Longitudinal Research Design Cohort Studies Cohort is a group of people who share a common characteristic or experience within a defined period Each study collects data from different individuals within the same cohort A study design where one or more cohorts are followed prospectively and subsequent status evaluations with respect to a disease or outcome are conducted to determine which initial participants exposure characteristics (risk factors) are associated with it An observational design comparing individuals with a known risk factor or exposure with others without the risk factor or exposure looking for a difference in the risk (incidence) of a disease over time

12 Longitudinal Research Design Cohort Studies Recurrence of child maltreatment over time

13 Longitudinal Research Design Cohort Studies Advantages Examine changes over time for a group of people A temporal relationship between exposure and outcome can thus be established No attrition problem Retrospective cohort studies (use of pre-existing data) Disadvantages Limited research questions that can be answered Information on confounding variables may be unavailable Classification of individuals (exposure or outcome status) can be affected by changes in diagnostic procedures

14 Longitudinal Research Design Panel Studies Survey A Sample 1 Survey A Sample 1 Survey A Sample 1 A longitudinal study that examines changes in individuals across time Each study collects data from the same individuals The same measures are administered over time

15 Longitudinal Research Design Panel Studies Advantages Provides the opportunity for more in-depth analysis Establishing causation Identifying a trend Allows for assessing the stability and continuity of several attributes of a sample by repeatedly observing the same participants Disadvantages Attrition Expensive and time consuming Performance improvements that are unrelated to the normal patterns of development but familiarity with the measures

16 Longitudinal Research Design Time-series studies Used to analyze TRENDS of intervention effect Does effect occur immediately? Is there a delay between intervention and effect? Does effect continue to trend over time or stabilize? Does effect go away after a certain time? Data collected at regular intervals Both before and after intervention Timing of intervals depends on research being done Minutes, days, weeks, months

17 INTERVENTION Longitudinal Research Design Time-series studies A time-series design requires multiple observations of the same individuals Before the intervention and multiple observation after the intervention

18 Experimental Research Design Research design in which conditions are controlled so that 1 or more independent variables can be manipulated to test a hypothesis about a dependent variable. Allows evaluation of causal relationships among variables while all other variables are eliminated or controlled

19 Experimental Research Design Elements of Experimental Design: Pretest - measurement or observation of the dependent variable before treatment or intervention Posttest measurement or observation of the dependent variable after treatment or intervention Experimental or intervention group is a group of individuals who receives intervention or is exposed to the experimental treatment Control or comparison group is a group that does NOT receive intervention or is NOT exposed to the experimental treatment

20 Experimental Research Design Randomization - random assignment of participants to either control or experimental group based on chance Random assignment allows the assumption that the groups are identical with respect to all characteristics except the experimental treatment. It is used to make groups equivalent. Equivalence refers to an attempt to select and assign subjects to experimental and control groups in such a way so they can be assumed to be alike in all major respects Matching refers to selecting subjects to the control and experimental groups on the basis of matching certain characteristics, such as age, race, gender

21 Experimental Research Design Classical experimental design pretest posttest experimental and control groups randomization R random assignment O 1 pretest O 2 posttest X treatment or intervention N non-equivalent groups R O 1 X O 2 R O 1 O 2

22 Experimental Research Design Pre-experimental design Pre-experimental is a design, in which there is no random assignment and no comparison group Pre-experimental research is needed because there are many independent variables that we cannot manipulate, either ethically, or practically One group post-test-only design One group before and after design

23 Experimental Research Design Pre-experimental design One group post-test-only design A.K.A. after-only design X O 1 Experimental group is measured after being exposed to the experimental treatment No pre-measure is taken No control group

24 Experimental Research Design Pre-experimental design One group before and after design O 1 X O 2 Subjects in the experimental group are measured before and after the treatment is administered No control group Offers comparison of the same individuals before and after the treatment

25 Experimental Research Design Quasi-experimental design Two-group post-test only design. A.K.A., after-only design with control group or Static Group Design X O 1 O 1 Experimental group is measured after being exposed to the experimental treatment Control group is measured without having been exposed to the experimental treatment No pre-measure is taken No random assignment Major weakness is lack of assurance that the groups were equal on variables of interest prior to the treatment

26 Experimental Research Design Quasi-experimental design Two-group post-test only randomized controlled trial. A.K.A., After-Only with Control Experimental group is measured after being exposed to the experimental treatment Control group is measured without having been exposed to the experimental treatment No pre-measure is taken R X O 1 R O 1 Includes random assignment to groups Major weakness is lack of assurance that the groups were equal on variables of interest prior to the treatment

27 Experimental Research Design Internal Validity the degree to which a study supports the intended conclusion drawn from the results. Did, in fact, the experimental treatment or manipulation cause a change in the dependent variable? Threats to internal validity compromise our confidence in saying that a relationship exists between the independent and dependent variables. External Validity the degree to which the findings can be inferred to the population of interest or to other populations or settings; the generalizability of the results Threats to external validity compromise our confidence in stating whether the study s results are applicable to other groups

28 Experimental Research Design Threats to Internal Validity History events occurring during the experiment that are not part of the treatment Maturation biological or psychological processes within participants that may change due to the passing of time, e.g., aging, fatigue, hunger (e.g., The performance of first graders in a learning experiment begins decreasing after 45 minutes because of fatigue.) A physical or mental change in subjects Respondents growing older, wiser, stronger or more experienced between pretest and posttest Studies which last for long periods of time

29 Experimental Research Design Threats to Internal Validity Instrumentation Refers to lack of consistency in measuring instruments which may result in an invalid assessment of a treatment program Changes in testing instruments, raters, or interviewers including lack of agreement within and between observers Testing the effects of one test upon subsequent administrations of the same test A pretest may improve performance on a posttest, regardless of whether there is any treatment or not Testing is a problem when time between testing is short

30 Experimental Research Design Threats to Internal Validity Diffusion of treatment Occurs if the control group learns about the manipulation

31 Experimental Research Design Threats to Internal Validity Experimental mortality (attrition) A reduction in the number of subjects that remain in a study from beginning to end

32 Experimental Research Design Threats to Internal Validity Demand Characteristics: Participants that know they are being tested react to this. A.K.A. Hawthorne effect A situation in which subject s behavior is affected not by the treatment per se, but by their knowledge of participation in a study

33 Experimental Research Design Threats to Internal Validity Compensatory rivalry or Compensatory behavior - When subjects in some treatments receive goods or services believed to be desirable and this becomes known to subjects in other groups, social competition may motivate the latter to attempt to reverse or reduce the anticipated effects of the desirable treatment levels. Saretsky (1972) named this the John Henry effect in honor of the steel driver who, upon learning that his output was being compared with that of a steam drill, worked so hard that he outperformed the drill and died of overexertion.

34 Experimental Research Design Threats to Internal Validity Experimenter Effects: Experimenter influences the participants The way an experimenter looks, feels or acts may unintentionally affect study results, typically in the desired direction. Solution: Double-Blind Experiment neither subjects nor experimenters know which treatment subjects are receiving

35 Experimental Research Design Threats to Internal Validity Resentful Demoralization: If the control group learns about manipulation, they may become upset and give up and not try anymore

36 Experimental Research Design Threats to Internal Validity Statistical regression the fact that groups selected on the basis of extreme scores are not as extreme on subsequent testing Selection bias identification of comparison groups in other than a random manner Placebo effect participants in study who are given a mock stimulus may react as if they are being affected by the stimulus because they expect to be affected

37 Experimental Research Design Threats to External Validity External validity cannot exist without internal validity If the results of the study are not internally valid, there is nothing to generalize. Researchers should always be concerned about ensuring internal validity first

38 Experimental Research Design Threats to External Validity Population generalization: the extent to which your results generalize to people/animals beyond just the participants you tested Environmental: the extent to which your results generalize to situations or environment beyond those used in the current experiment Temporal: the extent to which your findings apply at all times, not just the specific time/season your study was conducted.

39 Laboratory versus Field Experiments Field experiment applies the scientific method to experimentally examine an intervention in the real world rather than in the laboratory A field experiment is conducted in the field, that is, in a real world situation Laboratory experiment is a method in which the phenomena of interest are investigated under controlled and regulated conditions

40 Laboratory vs. Field Experiments Factor Laboratory Field Environment Artificial Realistic Control High Low Reactive Error High Low Time Short Long Number of Units Small Large Ease of Implementation High Low Cost Low High

41 Experiments Ethical considerations

42