EXPERIMENTAL METHODS IN PSYCHOLINGUISTIC RESEARCH SUMMER SEMESTER 2015
RESEARCH CYCLE Course content
WHAT IS AN EXPERIMENT? A systematic research study in which the researcher: 1) directly varies (manipulates) some factor (or factors) the independent variable (IV) 2) holds all other factors constant the extraneous variables 3) observes the results of the variation (manipulation) the dependent variable (DV)
THE INDEPENDENT VARIABLE The factor of interest, the one that is being studied to see if it will influence behavior Also called a manipulated factor (or manipulation) because the experimenter has complete control over it Independent variables must have a minimum of two levels At the very least, an experiment involves a comparison between two situations (or conditions)
OPERATIONALIZING INDEPENDENT VARIABLE Research question: Are focused words faster to identify than nonfocused words? IV = focus Levels = Whether or not a word is focused Must clarify: Syntactic focus? Prosodic focus? Semantic focus? Must operationalize: Syntactic focus Clefting? Fronting? Other devices?
SUBJECT VARIABLES (NATURALLY DEFINED) Refer to already existing characteristics of the individuals participating in the study, such as Sex Age Socioeconomic status Cultural group etc. Subject variables are independent variables not manipulated by the experimenter Studies using subject variables are called quasi-experiments
THE DEPENDENT VARIABLE The variable that is measured (the outcome of the experiment) Research question: Are focused words faster to identify than non-focused words? We need measures of speed of word identification e.g., lexical decision time, naming, reading time Important to choose an appropriate task and DV to be able to answer the research question
CEILING AND FLOOR EFFECTS Ceiling effect: the average scores are so high that no difference can be determined This happens when the dependent measure is so easy that everyone gets a high score Floor effect: all the scores are extremely low because the task is too difficult, producing a failure to find any differences between groups.
EXTRANEOUS VARIABLES Variables that are not of interest but which might influence the behavior in some systematic way (can turn into confounding variables) A confound co-varies with the independent variable and could provide an alternative explanation of the results Confounded studies are uninterpretable extraneous variables must be controlled/held constant
POTENTIAL CONFOUNDS IN PSYCHOLINGUISTIC EXPERIMENTS Word frequency, length, concreteness Age of acquisition Transitional probability Verb biases Number of words in a sentence Ambiguity Ways to control for extraneous variables : norms, pre-tests, corpora studies When you can t control for them, make sure they are not associated (confounded) with your IV (the same in all conditions)
TASK Decide about the IV and DV Pay attention at how extraneous variables are controlled
EXAMPLE 1
EXAMPLE 1 ABSTRACT & MATERIALS Diagnostic: Heidi could imagine and create things because she won first prize at the art show. Causal: Heidi felt very proud and happy because she won first prize at the art show.
EXAMPLE 1 DESIGN AND PREDICTIONS IV: type of relationship Two levels: causal, diagnostic To obtain the two readings, the first clause was manipulated DV: Eye-tracking measures (reading time) - region in which differences occur Predictions: Delayed-integration hypothesis: differences should emerge at the end of the sentence Immediate integration hypothesis: differences should emerge at the beginning of the second clause
EXAMPLE 1 EXTRANEOUS VARIABLES The critical regions are held constant: Diagnostic: Heidi could imagine and create things because she won first prize at the art show. Causal: Heidi felt very proud and happy because she won first prize at the art show. Other possible confounds?
EXAMPLE 2
EXAMPLE 2 ABSTRACT
EXAMPLE 2 TYPES OF STRUCTURAL CHANGE a) The woman saw the famous doctor had been drinking quite a lot.
EXAMPLE 2 TYPES OF STRUCTURAL CHANGE b) Before the woman visited the famous doctor had been drinking quite a lot.
EXAMPLE 2 MATERIALS NP/S ambiguous: The American woman/saw the famous doctor/had been drinking/quite a lot. NP/S unambiguous: The American woman/saw that the famous doctor/ had been drinking/ quite a lot. NP/Z ambiguous: Before the woman/visited the famous doctor/had been drinking/quite a lot. NP/Z unambiguous: Before the woman/visited, the famous doctor/had been drinking/quite a lot.
EXAMPLE 2 CONTROLLING FOR CONFOUNDS Verb biases - corpus investigation to make sure that: The verbs were all biased toward the NP reading Each item had a verb pair which was as balanced as possible in terms of the degree to which the NP analysis was preferred over the alternative analyses Plausibility - pre-test to make sure that: Each NP reading was equally plausible
EXAMPLE 2 SUMMARY Independent variables Ambiguity (two levels: ambiguous, unambiguous)! Construction type (two levels: NP/S, NP/Z) 2 X 2 design = two factors, each one with two levels Dependent variable Reading time Controlled variables Verb biases Plausibility of the misanalysis
!!!!! EXAMPLE 2: RESULTS The American woman / saw (that) the famous doctor / had been drinking / quite a lot. (NP/S) Before the woman / visited(,) the famous doctor / had been drinking / quite a lot. (NP/Z) Ambiguous sentences took longer to read than unambiguous sentences Ambiguous NP/Z sentences were read more slower than ambiguous NP/S sentences " The type of structural change had an effect on reanalysis difficulty
TYPES OF DESIGNS Single factor design One IV with two or more levels/conditions (e.g., word length levels: 1 syllable, 2 syllables, 3 syllables) Factorial design Two or more factors, each with two or more conditions 2X2 design 2X3 design 2X2X2 design 3X2X2X2 design (don t do it! Split into two experiments ) Between or Within-subjects?
BETWEEN-SUBJECTS DESIGN In a between-subjects design the comparison of two or more conditions (levels of a factor) is a contrast between two or more different groups of individuals (each group is exposed to one condition) Advantage Participants are less likely to guess the purpose of the experiment Disadvantages Large number of participants needed Differences between conditions could be due to differences between the two groups (individual differences)
CREATING EQUIVALENT GROUPS To deal with potential confounds, steps must be taken to create equivalent groups (groups that are equal to each other in every important way except for the levels of the independent variable) Random assignment: every participant has an equal chance to be in any group If the IV is a subject variable, creating equivalent groups is almost impossible!
EXAMPLE 3
EXAMPLE 3
EXAMPLE 4
EXAMPLE 4
WITHIN-SUBJECTS DESIGN Each participant is tested in each level of the independent variable (also called repeated-measures design) Advantage Fewer participants needed More control on individual differences (any betweencondition individual difference variance disappears) Disadvantages Order (carryover) effects: practice, fatigue, catching-on
ORDER EFFECTS If you present the conditions in the same order to all subjects, the experience of one condition may affect performance on the other Practice effects: The more time you do a task, the better you get at it Fatigue effects: The longer you do a task, the more tired you get Catching-on: The longer you do a task, the more likelly it is that you will figure out the manipulation
COUNTERBALANCING Ensure that each item (e.g., word) is tested equally often in each condition and each subject receives an equal number of items in each condition Don t test all participants in one condition, then the next condition Don t test each participant on each item in each condition
COUNTERBALANCING In a 2 X 2 design (conditions: A, B, C, D), each version of each sentence is distributed across four groups (lists):
COUNTERBALANCING Simple example: 2 conditions, N items Need 2 versions (or lists), each with N items (N/2 in condition 1, remaining half in condition 2) More generally: M conditions, need some multiple of M items, and need M different versions You should not have 15 items and 4 conditions...(that means some subjects would contribute more to some conditions than others; bad, if there are true differences among subjects)
CONTROLLING CATCHING-ON EFFECTS Use filler items to hide the purpose of the experiment At least one filler item should appear between two experimental items Exact ratio between experimental/critical and filler items depends on an experiment
DESIGN: 2X2 WITHIN-SUBJECTS MATERIALS: 32 SETS OF ITEMS SIMILAR TO 8A-D (8a) NP/S ambiguous: The American woman/saw the famous doctor/had been drinking/quite a lot. (8b) NP/S unambiguous: The American woman/saw that the famous doctor/ had been drinking/ quite a lot. (8c) NP/Z ambiguous: Before the woman/visited the famous doctor/had been drinking/quite a lot. (8d) NP/Z unambiguous: Before the woman/visited, the famous doctor/had been drinking/quite a lot.
COUNTERBALANCING
SUMMARY: GENERAL DESIGN PRINCIPLES 1. Formulate your question clearly 2. Keep everything constant that you don t want to vary 3. Know how to deal with unavoidable extraneous variability 4. Use a within-subject design whenever possible 5. Counterbalance your materials
SUMMARY If you vary some factor(s) and successfully hold all else constant, the results can be attributed only to the factor varied A causal relationship between IV and DV can be established
HOMEWORK Identify and be prepared to talk about: 1) Theoretical basis/motivation for the study 2) The general research question 3) The Independent variables 4) The dependent variables 5) The procedure (what the participant had to do) and the type of design (within/between/mixed?) 6) Steps taken to eliminate potential confounds and order effects 7) Potential confounds that were not controlled for