Random Assignment/Selection DESIGNS AND THEIR LIMITATIONS. Limitations

Transcription

1 Designs and Their Limitations 153 Education level could be a moderator because it would explain the relationship between desire to quit and company citizenship. When you add education to the analysis, the association between desire to quit and company citizenship ceases to exist (Baron & Kenny, 1986). Number of hours studying is typically not manipulated by the researcher. The researcher might assign participants to studying different organizations of material. For example, some participants are told to read a story, others to read a story and take notes, and others to read a story, take notes, and predict what happens next. Reading the story has three versions, or levels, where each level is a different manipulated version of the IV. Random Assignment/Selection Random assignment occurs when each participant from a sample is randomly assigned to one of the experimental or control conditions. Random selection occurs when participants are randomly selected from the population of potential participants. Random assignment of participants is common, but random selection and then random assignment are rare in social science research. The random selection and assignment are an important assumption for null hypothesis testing (see Chapter 9). DESIGNS AND THEIR LIMITATIONS This section provides information about traditional single-subject, time series, and pre-, quasi-, or true experimental designs. We discuss them in this order because there is a natural progression in the core designs from single subject through true experimental. These designs are the most common that we see, and therefore, not all encompassing. They set the foundation for understanding. In addition to understanding how to proceed with each of the basic research designs in this chapter, it s important to understand that each has limitations or weaknesses. We present the limitations of experimental research design first and then discuss the types of experimental designs. We do not tell you each design s limitation as we describe it. Instead, we ask you to try to determine the limitations of each design. After the designs, we provide a table indicating each design s flaws. We have noticed that students understand the flaws in their own experiences with experimental research, and getting students to do that first appears to help make the connection to the research language much stronger. But that is anecdotal, nonexperimental evidence, ironically. Limitations Every study and essentially every design has limitations, weak spots. Depending on the type of argument you are trying to make, you are trying to reduce the number of limitations, to reduce the level of error in your conclusions. There are three categories that are crucial to understand during the design phase: internal validity,

2 154 Chapter 7 Experimental and Nonexperimental Research Design external validity, and statistical conclusion validity. These three validity topics are traditionally discussed within experimental designs; however, they apply to social science research in general. Validity is the quality of the evidence we use when we make a claim. Therefore, validity is an inference and is not an all-or-nothing proposition (Messick, 1989). Internal validity concerns causality, or cause and effect (A caused B). Causality, or the ability to make a causal claim or argument, has very specific criteria (Kenny, 1979, 2004): 1. Temporal Precedence: The cause must precede the effect we must be able to show this. Consider the hemline theory, also called the skirt length theory, which states that when women s skirts are short, the stock market booms, whereas longer skirts mean a tough economy. You might think that this would lead our economists and political leaders to urge women to wear mini-skirts exclusively. But, unless the research can show a causal link indicating the relationship between skirt length and the economy, we d better hold off on dictating economic decisions based on fashion trends. 2. Covariation of the Cause and Effect: A change the effect must occur. For foreign language education research, the relationship between enrolling in a total immersion language program (e.g., French) and French fluency is a research line of great interest to some. If we were to research this, we may observe a positive correlation between the two. Over time, we are also likely to observe that enrollment in the program increases French fluency. Finally, we are likely to observe that those students who enrolled in the immersion program earliest, say in kindergarten, have a better level of fluency than those who enrolled in high school, as early second-language acquisition leads to better second-language capacity (Lazaruk, 2007). If the pattern always followed this, then we could support a causal argument. However, we suspect that we would observe some students, who started language learning in high school, who have as good or better fluency than some who started in kindergarten. In general terms, effect B must follow cause A, and B must always follow A. 3. No Plausible Alternative Explanation there is not a third or fourth or other explanation for the change. For the depression study example we use for most of the designs below, a change in depression scores must occur after the intervention, must always occur after theintervention, andcannotbe due to a third variable such as medication. Yes, it is difficult to argue for causality. Therefore, you are trying to reduce the number of reasons readers have for denying your causal inferences and arguments, also known as rival hypotheses. Internal Validity Threats Internal validity threats are issues or concerns that develop when problems internal to the study, such as participants leaving, and the data negatively affect the quality

3 Designs and Their Limitations 155 of the causal argument. These threats fall into seven major categories. We use a blend of both the more recent names for these threats, as well as the original titles from Campbell and Stanley (1963). Historical Effects. Historical effects are unplanned instances or events that occur during a study that could affect the results. For example, Jim has had several fire alarms go off during studies in schools. One occurred while helping a friend complete the final experiment during her dissertation (well, the second to last experiment as it turned out). These events create problems in determining, and arguing for, why the results occurred. Maturation. Maturation is the natural developmental patterns that occur due to engaging in everyday life events. As the length of a study increases, the chance of a maturation effect increases. Studies with very young children and long-term, or longitudinal, studies are most susceptible to this validity problem. Change in performance over time could be due to the natural changes in motor skills as children grow. The researcher must show that maturation is not the sole reason for the change. Maturation can include age, experience, physical development, or anything that leads to an increase in knowledge and understanding of the world that is not related to the variables of study. Testing Effects. Testing effects occur because the participant has experience with the instrument or activity. For example, scores on the SAT may increase with a second administration simply because the test takers have experience with the test and format. Jim had to help someone practice several IQ test administrations; the person performed adequately on the first, but by the last one, he had increased his score to MENSA proportions. At the end, he didn t feel smarter, just tired and hungry. Instrumentation Threats. Instrumentation threats occur due to problems or other inconsistencies with the data collection method. This could be from the actual instrument (typos), the interviewer, observer change, or grader, for example. This is a common problem in longitudinal research, because instruments change over time. One research area where instruments change over time is intelligence tests. For example, the WISC-III to the WISC-IV test changed and the focus of interpretation changed from composite to level of index scores (Weiss, Saklofske, & Prifitera, 2005). It is also a common threat in cross-cultural research. For example, the Brief Symptom Inventory has a statement on it to check for depressive symptoms feeling blue. When translated into Spanish and used with a group of Central American immigrants, Kim found that the color blue was not used to describe sadness at all (Asner-Self, Schreiber, & Marotta, 2006). Regression to the Mean. The regression to the mean phenomenon demonstrates that individuals who score on the outer extremes (either very high or very low) of the score continuum will naturally score closer to the mean when retested. School

4 156 Chapter 7 Experimental and Nonexperimental Research Design districts are seeing this problem in repeated mandated testing. Their highest scoring students are dropping off on the next round of tests, when in actuality it is best explained by regression to the mean. Extreme group analysis (EGA) is highly susceptible to this problem. EGA occurs when the sample is at either end of the continuum of the phenomenon of interest. For example, you put students who are in the bottom group of reading ability in a reading program, and they show reading ability gains after the treatment. In reality, there is little evidence that the treatment worked, and it is likely that it is just a simple regression to the mean. Mortality. In certain types of studies, mortality does mean the death of participants, but for most of social science research, mortality occurs when participants decide to disengage from the study. Depending on which participants leave and how many, their departure can affect the results and lead to incorrect inferences from your data. For example, during data analysis, there appears to be no meaningful change over time for a group of students learning how to solve different volume problems. The lack of difference may be due to the departures because those participants with increasing scores left the study, but those who stayed were hoping that their scores would change. This is also an issue when participants are removed from a study during the data examination or cleaning phase. One should examine the demographics and response patterns of participants who leave the study and examine the types of participants who are removed from the data set. Selection Threat. The above internal validity threats are considered single-group threats because the focus is on the sample group of interest. However, multiplegroup designs (e.g., experimental and control group) are the norm and are subject to the same threats as single groups. The observed study differences are due to differences between groups that were preexisting or occurred during the study. A selection threat is generated when there are inconsistencies in the comparison groups, a sampling process problem that becomes a rival hypothesis. Within selection there are a few multiple-group threats, which we discuss next (Trochim, 2000). Selection-history occurs when one group experiences a non-experimentrelated event that the other group or groups do not. For example, a group of randomly assigned students in an educational intervention experience a fire alarm during the learning phase that the other groups do not experience. This happens more often than you would think. Selection-maturation occurs when one group matures faster than the other, for example, a higher proportion of females in one group compared to males during adolescent years for a study on social skill development. Selection-testing occurs when one group has previously taken the instrument. During an evaluation project, Jim discovered that the control group had been exposed to the content material and the test previous to his entry into the project, and subsequently, the control group outperformed the experimental group. Selection-mortality occurs when one group has a higher rate of leaving. Schoollevel data is prone to this type of problem based on mobility rate. Mobility rate is a percentage of students that enter and leave a school in a given time period. Students

5 Designs and Their Limitations 157 from families with lower socioeconomic status (SES) tend to have a higher mobility rate than students from families with a higher socioeconomic status. Because of this, researchers attempting to conduct studies over a semester, full school year, or longer have a difficult time because the lower SES students leave the school in higher proportions than the higher SES students. Selection-regression occurs when one group has a larger proportion of very high or very low scorers. An example of selection-regression is two groups of students who are in a summer reading improvement program. The participants in Group 2 started at a much lower initial average score compared to the first group on a reading comprehension test. At the posttest, Group 2 has much greater reading comprehension gains compared to Group 1 because of initial very low scores. Internal Social Effects Humans are social creatures. Humans run experiments. Social creatures interact during experiments. It is unavoidable. Jim has tested this during experiments by having the researcher flirt or act overly social as participants enter the lab. The results were different between those who had this extra treatment study versus those who did not. The larger issue is the inferences that cannot be made to the larger population or simply inference mistakes that occur when these social effects occur (Trochim, 2000). Diffusion or Imitation of Treatment. The knowledge of the study s purpose or the difference in treatment can lead to diffusion or imitation of treatment. Many years ago, this happened to a friend who had created a behavioral contract with one of her classes, which allowed the class to have a pizza party at the end of the school year, but not with the other class. This was done in order to increase the positive behaviors in the first class. Unfortunately, at an unsanctioned off-school-property event (i.e., house party) near the end of the implementation phase, one student in the pizza group was talking to another student in the pizza group and a third student in the no-pizza group heard. Party over, the deal had been diffused to the other group. You could also consider the house party a historical event. Compensatory Rivalry or the John Henry Effect. John Henry supposedly tried to lay railroad track faster than a machine and thus was in rivalry with the machine. In research, when the control group tries to outdo or compete with the experimental group, the control and experimental group may perform in a similar manner. In the pizza example, the no-pizza group could get mad and decide to behave better than the pizza group. Though this is an odd concept, their desire to prove the teacher wrong outweighs everything else. They essentially want to mess up the results of the program, or in this case to cause problems and retribution to the teacher by actually behaving better, and this desire outweighs the fact they are not getting pizza for positive behavior. A placebo treatment is often given so that everyone knows they are in the study, but participants do not know whether they are in the experimental or control group.

6 158 Chapter 7 Experimental and Nonexperimental Research Design Resentful Demoralization. The reverse of the compensatory rivalry is resentful demoralization. For the no-pizza group, individuals or the group as a whole may begin behaving even worse than before the study. Interestingly, the differences between the two groups increases and the intervention appears to be working, when in actuality the difference in the two groups has nothing to do with the study. Compensatory Equalization of Treatment. For the no-pizza group, the parents became involved in who receives pizza, and then all the students received a pizza party at the end. Obviously, changes in the research design and methodology have occurred at this moment. Part of being transparent means that the researcher had to then clearly record this change of events as possible threats to the validity of his or her findings. Interestingly, the experimental group still had fewer negative behaviors at the point of the diffusion, for weeks after the diffusion, after the announcement of pizza for everyone, and at the end of the study. The no-pizza group s negative behaviors skyrocketed after the announcement of pizza for everyone! What do you think that was all about? Certainly, that question would lead you to another research project! Novelty. Simply being a part of the study causes participants to increase interest, motivation, or engagement. Therefore, the treatmentmaybeeffectivebecauseit is a novelty and not because it is better than other treatments or no treatment. To deal with this situation, one should conduct the study over a longer period of time, so this effect can wear off. External Validity External validity concerns the inferences that can be made to the larger population in general or to other populations. In quantitative-focused research, one of the goals is to make inferences from the sample in the study to the population in general. Unfortunately, problems from the design perspective can reduce the validity of those inferences. Selection Treatment. Even with random assignment, external validity can be a problem when the sampling process creates a bias that interacts with the experimental variable of interest and the results do not represent the population as a whole. We see this problem in what we call convenient random sampling. For example, you could say that the participants were randomly sampled, but the sample is actually a convenience sample from a Psychology 101 student pool, a grade school by your house, a company where you know the owner, or the train you ride to work. These participants may or may not represent the whole population of interest and may interact with the experimental variable differently than the rest of the population. As a group, the participants are a nonrandom or volunteer group. However, the inferences to the larger population base are greatly limited. After all, what percentage of the U.S. population is similar to the kinds of people who take Psychology 101, who choose to take your study, and who need the points

7 Designs and Their Limitations 159 toward their grades? Kim did a study once looking at the mental health of Central American immigrant and refugee community members. She used an acculturation measure that had been normed on immigrant Latinos who were college students. Yet, a full 65.3% of the sample of people she studied had a high school education or less (and 33.7% of the sample had six years of education or less). Pretest Treatment Interaction. Pretest treatment interaction occurs when the pretest sensitizes the participants to components of the study, which affects the posttest scores. How much will depend on the sample of participants, the nature of the study, and the nature of the instruments used. The more unique or different the material or the study is from previous experience, the more likely this affect will be seen. If you believe this could happen due to the nature of your study, consider gathering data from other sources, such as school or employment records that are not part of the study but can give you information on prestudy knowledge of the participants. Multiple Treatment. Multiple treatment concerns studies where participants engage in more than one treatment. For example, students participate in Reading Programs A, B, and C. The problem occurs when one treatment interferes positively or negatively with the other treatments. An example would be if participants who had treatment A first performed worse on the reading test after treatment B, and those participants who had treatment C first performed better after treatment B. This can be solved in the design so that every possible combination of treatments is covered and any effects can be examined. Reactive or Participant Effects. Participants react to knowing that they are being studied or watched. We all do, and we change our behavior. This change in behavior is why reality television is not reality. In general, the term Hawthorne effect applies to any situation in which participants behavior is affected by knowing they are in the study or being watched. The name comes from a study at the Hawthorne Works plant of the Western Electric. The company was studying productivity and light intensity. As light intensity increased, production went up; and as light intensity decreased, production went down. The researchers, being good at observing and thinking, realized that it was not the light level that affected productivity, but the fact that attention was being paid to the workers. Specificity of Your Variables. In addition to developing stable measures as stated in the previous chapters, you must be specific about the definition of your variables and the details of how you had collected data that match that definition. You must give rich detail to the setting and the procedures you used, so that researchers can attempt to replicate or compare your study to other studies. Experimenter Effects (aka Rosenthal). As the researcher, you present a potential limitation. You can consciously, or unconsciously, affect the results by your engagement with the participants. Your gender, age, race, or outgoingness can affect the

8 160 Chapter 7 Experimental and Nonexperimental Research Design participants. The expectations for the study can also affect the participants when you engage with the experimental and control group differently and how you score their responses. For these reasons, the blind study or scoring is popular, where a third party not familiar with the study is executing the study or scoring the data. Many graduate students have begun their research careers as the third party in a set of studies. Finally, there are externalities that can affect the outcome of programs, and the ability to design a project that limits outside factors is crucial. Therefore, having a clear understanding of the threats during the design phase of your program can help you strengthen the structure of the study and avoid negative comments from critics who review your work. Ecological Validity Ecological validity is related to the realistic nature of the material or context used in the experiment. The specificity of variables, multiple treatments, pretest effect, and any Hawthorne effect can also be considered an ecological validity problem. After a decade of map and text studies, the research group Jim participated in finally conducted one that was similar to how students see maps and related text material in textbooks. The results supported the previous basic research work, but this series of studies needed to be completed from an ecological validity perspective. Statistical Validity Statistical validity concerns using the most appropriate statistical analysis in relation to the design, data, and of course, the research question. This requirement seems obvious when one reads it, but it is a common mistake. There must be systematic connections between your research questions, your sample, the design, collected data, the analysis, and the discussion. All of it has to make sense together. Not surprising, studies without the clear link are likely to languish unpublished, unwanted, or ignored by the savvy consumer of research (i.e., you). Now, if you want a challenge, try to identify all of the validity issues in Schuyler Huck s poem concerning the limitations of experimental designs. The poem can be found where it was published in the Journal of Experimental Education in The full reference can be found at the end of the chapter. Dr. Huck provides the answers in the article, so don t worry. EXPERIMENTAL DESIGNS Experimental designs have one thing in common: an independent variable (or variables) that is manipulated by the researcher. Each design can be described symbolically/graphically using a group of letters based on Campbell and Stanley (1963), where R = random assignment; O i = observations (the subscript tells which time period); G1, G2, G3, etc., = which group the participants are in; and X i = the treatment (i.e., the activity in which the participants are engaged in).