The use of students as participants in experimental research. Rachel Croson, University of Texas at Dallas

Transcription

1 The use of students as participants in experimental research Rachel Croson, University of Texas at Dallas Introduction Experimental methods have a long history in the behavioral sciences. Although most commonly used in psychology, within the last 30 years the experimental method has made great inroads into economics as well, culminating in a recent Nobel Prize won by Vernon Smith and Daniel Kahneman. Experiments are only starting as a recognized methodology in research in operations, and as is appropriate for an emerging area, researchers are questioning the existing norms domain. One of the major aspects is the choice of subject pool for our experiments. By far the vast majority of experiments in economics and psychology use students as participants. This discussion paper describes reasons for and against using students as experimental subjects. We also mention some previous research which has used non- student subject populations and which has provided a comparison of behavior between different subject pools. We conclude with recommendations for experimental and behavioral researchers in deciding which subject pool to use. Other fields Psychology routinely uses students in experiments. Common practice is to require some number of hours of experimental participation from students enrolled in the introductory psychology course. This practice has been adopted by many fields whose reference discipline is psychology as well (e.g. In contrast, economists tend to use students in experiments, but do not make participation mandatory. 1 Economists typically solicit volunteers to participate in experiments. These solicitations may happen in the classroom, or they may happen online, via posters around campus, or any of a number of ways. Instead of awarding class credit to student participants, economics experiments typically involve monetary earnings for participants. Using induced valuation, (Smith 1976) the earnings of a given participant is responsive to the decisions they make. Earnings are typically set so as to exceed the going wage for a given experiment. Note that both fields (along with many others) rely heavily on random assignment of subjects to treatments. This feature is critical in order to avoid biases. While it might be, for example, that students act differently than professionals in a given treatment, experimenters typically focus on the differences between multiple treatments. If participants are not randomly assigned to treatments (for example, if students from a morning class are in one treatment and students in an afternoon class are in another treatment), there may be systematic differences between the participants which causes the treatments to look different from each other (for example, morning- class students may have more impulse control than afternoon- class students). Some Considerations What are the advantages and disadvantages of using students as subjects in an experiment? When should you use students and when not? 1 Experiments run by professors in business schools often use MBA students rather than undergraduate students. While MBA students represent a middle- purposes of this discussion we put them in the same class as undergraduate students.

2 Type of experiment One issue to consider is the type of experiment you are running. Some experiments are designed to test theories, others to explore anomalies observed in the field, and others to testbed policies (Roth 1986). Different types of experiments call for different types of subject pools. Most theories are silent on the actors to whom they are designed to apply. For example, when one is testing a theory of incentives between principals and agents, there is no reason to suspect that the students are perfectly appropriate subjects. In contrast, if an experiment is designed to testbed a policy, one might expect that the impacts of that policy might be different on different subpopulations. For example, when one is considering a reform in welfare payments, it may very easily be true that the poor will react differently to the reform than will a typical college student. In these types of experiments, students are likely not the appropriate subjects. The middle ground is experiments designed to investigate anomalies observed in the field. If these anomalies are caused by some underlying and universal bias, or by the institutions that the field imposes on the actors, then student subjects will likely duplicate the anomaly in the lab, and allow the researcher to investigate institutional designs or debiasing techniques to improve behavior. If these anomalies are caused by specific features of the individuals in question (e.g. hedge fund managers tend to be quantitatively focused, which leads to a reduced ability to manage employees), then using student subjects will not replicate the anomaly in the lab. Availability and logistics A second set of considerations in choosing a subject pool is the availability of the participants, and the logistics of their participation. Students are (almost) always available and willing to participate in experiments. In contrast, recruiting high- level executives, production managers, or professional traders is often difficult or impossible. Secondly, one must decide how to engage alternative subject pools in the experiment. Students can be induced to participate (and to pay attention) with financial incentives. But a researcher will rarely have the type of incentives necessary to gain the attention of high- earning executives. Thus experiments run with these subject pools often involve other feedback, in the context of an executive education class or consulting project. This raises its own concerns (see Demand effects, below). Background knowledge needed A third consideration in the choice of subject pools is how much background knowledge the participants will need in order to complete the experiment. If one is to run an experiment on a complicated financial market, having participants who are experienced with the market will reduce the errors and confusion in the experiment. Novice participants, in contrast, may exhibit noisy behavior. Note that this concern is primarily aimed at reducing variance in the resulting experimental data, rather than necessarily affecting the mean outcomes. Demand effects A final consideration in choosing a subject pool is the avoidance of demand effects. A demand effect is said to occur when a participant in an experiment chooses an action simply to please the experimenter.

3 ht in their class). In this setting, participants often spend time and energy thinking about what the experimenter wants them to do, rather than doing what they would naturally do in a given situation. Instructions for the experiment can be written in such a way to cause or reduce demand effects. 2 Are there differences? There is now quite a lot of research on experimental differences between student participants and other, professional, participants. The following table presents an extremely incomplete list of papers (from experimental economics rather than psychology, marketing, strategy, or any other field), the domain in which the experiment was run, and a brief summary of the findings. At the bottom of the Table I have included a set of experiments which used non- student subjects but did not explicitly compare their behavior to those of students Insert Table 1 about here- - - As can be seen, the impact of subject pool on behavior varies considerably based on the task. Summary Should you use students as subjects in your experiments? The results from previous research suggest that in most settings it does not matter much; the results are much the same. That said, non- experimental reviewers and editors often criticize experimental research on the basis of its subject pool. One solution, often used, is to run the primary experiment with students, then replicate one treatment (usually the baseline) with a professional pool and demonstrate that the results are the same. If you have access to professionals, running experiments with them is a nice way to make a new contribution. However, you need to be careful about the typical considerations in all experimental designs, like random assignment to treatment, avoiding demand effects, and others. References 2 you would be would actually do in th 3 Note, no review such as this can ever hope to be entirely comprehensive. My goal in Table 1 was to give an overview of the types of experiments that others have run with different subject pools, and it is likely to be far from complete.

4 Article Domain Finding Alpert, 1967, Non- Businessmen as Surrogates for Businessmen in Behavioral Experiments. Anderson, Harrison, Lau and Rutstrom, 2004, Preference Heterogeneity in Experiments: Comparing the Field and the Lab. Bellemare and Kroger 2004, On Representative Social Capital. Block and Gerety Some Experimental Evidence on Differences between Student and Prisoner Reactions to Monetary Penalties and Risk. Cooper Are Experienced Managers Experts at Overcoming Coordination Failure? Cooper, Kagel, Lo and Gu Gaming Against Managers in Incentive Systems: Results with Chinese Students and Chinese Managers. Croson and Donohue Behavioral Causes of the Subjects rated appropriateness of a described firing decision. Then wrote a letter of advocacy either supporting or opposing the decision. Elicited risk and time preferences among the general Danish population and students at the University of Copenhagen and Copenhagen Business School. Played trust game with representative sample of the Dutch population and compared behavior with students at Tilburg University. Participants played auction with the possibility of collusion and with random probability of monetary penalties for collusive behavior. Subjects (managers) attempt to motivate other participants to coordinate after a history of failure in a weak- link coordination game. Undergraduate and Executive MBA students (with significant work experience) serve as managers. Firms choose productions, which causes central planners to set targets. Firms have an incentive to produce low in order to get future favorable targets. Subjects were either students, younger managers or older managers. A four- echelon supply chain game, played by undergraduate Student engineers supported manager more than all others. Military personnel, business managers, business non- managers, upper- division business students, all the same. Some differences in subject pools in the change of their opinion after writing the letter. Comparable degrees of risk aversion and time preference in the field and the lab. However, more variation in risk aversion measures in the field. No differences for time preference. Students trust less and reciprocate less than the general populat is thus closer to equilibrium. These differences are entirely explained by demographic differences. by the size of the penalty. by the probability of receiving the penalty. Executive MBAs more are more likely to use (effective) communication strategies while undergraduates use ineffective communications or incentives which ultimately fail. equilibrium predictions, and affected by context than was Professionals had the same (or worse) bullwhip behavior as

5 Bullwhip Effect and the Observed Value of Inventory Information. DeJong, Forsythe and Uecker A Note on the Use of Businessmen as Subjects in Sealed Offer Markets. Dyer, Kagel and Levin A Comparison of Naïve and Experienced Bidders in Common Value Auctions: An Experimental Analysis. Fehr and List The Hidden Costs and Returns of Incentives: Trust and Trustworthiness among CEOs. Glaser, Langer and Weber Overconfidence of Professionals and Lay Men: Individual Differences Within and Between Tasks? Guth, Schmidt and Sutter Bargaining Outside the Lab: A Newspaper Experiment of a Three- Person Ultimatum Game. Haigh and List Do Professional Traders Exhibit Myopic Loss Aversion? An Experimental Analysis. Harbaugh, Krause and Berry GARP for Kids: On the Development of Rational Choice Behavior. Harbaugh, Krause and Vesterlund Risk Attitudes of Children and Adults: Choices over Small and Large Probability Gains and Losses. students and professional logistics managers (members of the Council of Logistics Mangers). A principal- agent / labor market in which employers made offers to employees, who accepted or rejected them, and then exerted unmonitored effort. With some probability both parties suffered a loss. Common value auctions (placing bids on a job) were run with either students of professional construction contractors. Participants play a trust game with the opportunity for punishment. Costa Rican students and CEOs participated. Individuals provided estimates and confidence intervals for both general interest and financial forecasting questions. Traders and investment bankers and students participated. Newspaper readers submitted strategies in a three- party ultimatum game. Student and non- student samples were compared. Participants chose how much to invest in a risky or safe gamble; treatments varied the frequency of feedback. Undergraduates and professional traders from the Chicago Board of Trade participated. Participants chose between bundles of goods. Children aged 7-11 years and college undergraduates participated. Participants choose between multiple gambles, in gains and losses. Participants were children (age 5-13), undergraduates and adults. student participants. No significant differences between students and businessmen (partners in an auditing firm or corporate financial officers). Businessmen had higher variance of outcomes than students. Both students and professionals differences between subject pools. CEOs are more trusting and trustworthy than students (thus students are closer to equilibrium predictions). Experts are significantly more biased than student subjects. But overconfidence is not correlated (within an individual) across tasks. Students and non- students of the same age did the same thing. equilibrium. Traders are significantly more biased than students. Second graders violated axioms of choice significantly more than sixth graders, who were the same as college undergraduates. biased probability weighting than adults.

6 Harrison, Lau and Williams Estimating Individual Discount Rates in Denmark: A Field Experiment. Heinrich Does Culture Matter in Economic Behavior: Ultimatum Bargaining among the Machiguenga of the Peruvian Amazon. List and Haigh A Simple Test of Expected Utility Theory using Professional Traders. Montmarquette, Rulliere, Villeval and Zeigler Redesigning Teams and Incentives in a Merger: An Experiment with Managers and Students. Murnighan and Saxon Ultimatum bargaining by Children and Adults. Nagel, Bosch- Domenech, Satorra, Garcia- Montalvo One, Two, (Three), Infinity: Newspaper and Lab Beauty Contest Experiments. Potters and van Winden The Performance of Professionals and Students in an Experimental Stud of Lobbying. Participants responded to questions about money received now or later, and discount rates were computed. Participants were a national sample of the Danish population. Ultimatum bargaining game, done by remote Peruvian tribe and UCLA graduate students in the Department of Anthropology. Allais paradox questions to students and professional traders from the Chicago Board of Trade. A work task (exploring a function) done with teams, causing a social dilemma. Originally subjects were paired with others with similar payoffs occurred and teammates were subjects with a different payoff function. Participants were managers in a large pharmaceutical company which had just completed a merger and students. Ultimatum game played by children (kindergarten through sixth grade) and college students (played with money). Beauty Contest (iterated dominance) game played by the general public (newspaper readers) and students in a lab. A signaling (lobbying) game in which one participant sends a costly message to persuade his uninformed partner to take an action which would be Students have a higher discount rate than non- students. Thus students are less patient, controlling for age and other demographics. Machiguenga offer significantly less and reject significantly less, than students. Both students and traders violate the independence axiom, but traders (16 out of 64) are slightly better than students (16 out of 30). Fewer free riders among managers than students (students are closer to equilibrium play). Students learn more than the managers as they play the game. Children were more generous than college students overall, especially when the responder did not know the size of the ultimatum pie (college students give significantly less, while under complete information). Exist some subject pool differences; more heterogeneity in responses from the general population than from the students. No significant differences between the populations.

7 Sade, Schnitzlein and Zender Competition and Cooperation in Divisible Good Auctions: An Experimental Examination. Smith, Suchanek and Williams Bubbles, Crashes and Endogenous Expectations in Experimental Spot Asset Markets. Sutter and Kocher Trust and Trustworthiness across Different Age Groups. Bekkers, 2007, All- or- Nothing Dictator Games: A Field Experiment. Bellemare, Kroger and van Soest, Estimating Distribution- Based Preferences using a Large Scale Experiment with Probability Questions on Expectations. Binswanger Attitudes toward Risk: Experimental Measurement in Rural India Bosch- Domenech, Nagel and Sanchez- Andrez Social Capabilities Preserved in Alzheimer Patients. advantageous to himself. Participants were undergraduate students and public affairs officials. Compare discriminatory and uniform- price auctions. Participants were senior undergraduate and MBA students and finance industry professionals. Experimental asset markets show bubbles. Most experiments with students, but one (Experiment 10) uses professionals from the business community. Trust game played with different age groups (ages 8, 12, 16, Few differences reported. 40% of professionals made weakly dominated bids, while only 13% of students did. Other interesting subject pools, but no explicit student comparisons Representative sample of Dutch population played a dictator game with a charity as recipient. Participants earned their money (by filling out a survey) then could donate either all or none of it to an organization. endowment. Ultimatum and dictator decisions of a representative Dutch sample. Experimentally measured risk aversion among rural population in India. Used large stakes (for the population). Alzheimer patients and similarly- aged control subjects played a dictator game. Bubbles are robust in a variety of treatments, and remain so when business professionals are participants in the experiment. Trust increases from childhood to early adulthood, then stays constant. Trustworthiness is constant across ages. behavior is thus statistically indistinguishable from that of professionals and older adults. comparison. 5.7% gave their earnings, comparable to other studies with students when 2.6% give the entire endowment and 8.3% give more than half of their comparison. But the authors larger in the Dutch population as a whole than extrapolations based on student samples would closer to equilibrium. comparison. All but one of 118 participants are risk- averse. comparison. No differences are found among the tested groups. This population is somewhat

8 Carpenter, Daniere and Takahashi Cooperation, Trust and Social Capital in Southeast Asian Urban Slums. Social dilemma (VCM) games with costly sanctioning played by residents of Vietnamese and Thai slums. more generous (farther from equilibrium) than previous studies using students. comparison. Overall high contributions, higher in Vietnam than in Thailand.