2015 Russell Ackoff Doctoral Student Fellowship Proposal. Robert Mislavsky 2 nd Year Doctoral Student, OPIM

Transcription

1 2015 Russell Ackoff Doctoral Student Fellowship Proposal Robert Mislavsky 2 nd Year Doctoral Student, OPIM 1. Project title and descriptive summary: When Are Field Experiments Unethical? Evidence-based policy has become something of a buzzword in the academic, political, and business worlds (e.g., Young, Ashby, Boaz, & Grayson, 2002). Often, although archival methods are often used (particularly in economics) to determine causal impacts of policies, the best evidence to support the effectiveness of a policy comes from an experiment where participants are randomly assigned to receive different interventions and comparing results across conditions (List, 2011). Until recently, randomized trials in the social sciences have been mostly confined to laboratory settings with participant pools that are limited both in number and in their demographic characteristics (see Henrich, Heine, & Norenzayan, 2010 for a discussion of how this may affect the external validity of psychology experiments). However, technological advances (such as an increased ability to collect behavioral data through mobile devices) have made it easier to run large-scale, real-world experiments on a more general population. Recently, for example, researchers have run field experiments in a variety of domains, such as education (e.g., Fryer, 2010), energy use (e.g, Allcott & Rogers, 2012), retirement savings (e.g., Thaler & Benartzi, 2004), and charitable giving (e.g., Gneezy, Gneezy, Nelson, & Brown, 2010). There are numerous benefits of this research, as it could bring forth a number of smarter, more effective policies, while preventing ineffective (or worse, potentially harmful) policies from gaining wide-spread adoption and wasting resources that could be directed elsewhere. However, both organizational decision-makers and the general public have expressed significant resistance when it comes to field experiments. On the decision-maker side, for example, when asked if she would consider running an experiment that may improve her charity s performance, Heifer International 1 vice president Elizabeth Bintliff said [W]e re not about experiments. These are lives of real people (Kestenbaum & Goldstein, 2013). The general public is no more forgiving of what they may see as unethical experiments. In 2012, Facebook collaborated with an academic team to study the effects of emotional contagion on its users (Kramer, Guillory, & Hancock, 2014). When details of this experiment were published, the company faced substantial backlash. Many were upset about being treated like lab rats (Albergotti, 2014) while others accused the company of intentionally [making] thousands upon thousands of people sad. (Waldman, 2014) OKCupid, a popular dating website, garnered a similar reaction when they revealed the results of an experiment where they, in order to test the power of suggestion on users willingness to contact potential dates, arbitrarily changed the match ratings that users saw and tracked how that influenced their interactions (Rudder, 2014). Some have suggested that Facebook and OKCupid may have even violated the law in running these experiments (Grimmelmann, 2014). The aim of this research is to uncover the underlying cause of this resistance. Many academics and professionals have informally speculated on potential causes. For instance, Rey Junco of Harvard University attributes backlash against the Facebook experiment the potential for harm (Larson, 2014), while James Grimmelmann (2014) of the University of Maryland believes that the primary culprit the lack of informed consent. However, there has been no formal research examining this topic. Drawing on prior research on randomized 1 Heifer International is a charity that provides livestock and other assistance to poor villages in third-world countries.

2 clinical trials in medicine (see Emanuel, Wendler, & Grady, 2000 for a review) and fairness in market transactions (e.g., Kahneman, Knetsch, & Thaler, 1986), we propose that many of these lay theories are incorrect. Informed consent and a lack of harm are neither necessary nor sufficient to the conducting of ethical field research. Rather, in order to be considered ethical, each treatment in an experiment must have a reasonable expectation of improving outcomes for study participants relative to a status quo. This research has significant implications for a variety of stakeholders. Ideally, one product of this research would be the development of a general code of conduct for field researchers similar to those used by medical researchers (cf. World Medical Association, 2000). Most importantly, this would help protect the rights of people who may be placed in experiments without their knowledge or consent. It would also benefit key decision-makers and executives know when it is appropriate to run experiments that will not damage their organizations. Finally, it would benefit researchers by assisting them in designing research that would make it easier to find private- or public-sector partners to sponsor their studies. Planned Program of Studies. Topic Description Status Study 1: Pilot Study Our initial study is designed to provide an initial test of 2 hypotheses. The first hypothesis is that study participants view experiments that have the potential to make people worse off, relative to the status quo. This holds regardless of the absolute or relative magnitudes of harm and benefit. For example, although buying a product for a given price does not necessarily make someone worse off in the main, it makes them worse off compared to the status quo if the price is typically lower. Therefore, if some people are randomly assigned to pay the higher price and some the lower price (perhaps to study price elasticity), people would view that experiment as unethical. The second hypothesis is that, given the choice, participants would not participate in an experiment if there is any possibility of harm (again regardless of the relative magnitude of harms and benefits), and that decision to participate strongly correlates with the perceived ethics of the experiment. In Preparation (MTurk, N = 500) Study 2: Replication with Real-World Contexts To study this, we developed 3 brief, stylized scenarios describing different field experiments that companies could run. In these scenarios, we manipulate the outcome of different treatment arms (which are positive, negative, or status quo) and whether the experiment has different attributes considered to be relevant to an experiments ethics (e.g., informed consent, random assignment, deception). We will ask some participants to evaluate treatment arms individually and other participants to evaluate an experiment where participants are assigned to one of several treatment arms. Study 1 uses highly stylized and simplified scenarios to provide a clean, initial test of our hypotheses. In reality, however, people will not be evaluating field experiments using short descriptions. More often, they will learn about them in an academic article or in the context of a news story, which may also provide an opinion on the ethics of the experiment. In Study 2, we will bring the scenarios from Study 1 (and likely May 2015 (MTurk)

3 Study 3: Mechanism Development Studies 4 and 5: Lay and Expert Intuitions develop new ones) into these contexts. Instead of reading short scenarios, participants will read news stories and opinion pieces where we manipulate the individual components mentioned earlier (e.g., use of deception, informed consent) within the story to see if these still have an impact or if participants opinions is primarily driven by the opinions of others. We believe that, although the effects may be muted (as they often are when noise is added), potential for harm will still be the dominant factor when determining the perceived ethicality of field experiments. If the effects in Studies 1 and 2 prove to be robust, the next step would be understanding why these specific effects occur. Study 3 will be designed to test specific mechanisms and find additional boundary conditions if applicable (e.g., there is a certain context where informed consent is the most important component), which will be developed in part by using comments from users in Studies 1 and 2 (comments will be elicited at the end of the study). Finally, it will be important to understand common intuitions about what are the most important characteristics to keep in mind when designing an ethical field experiment and contrast them to our previous findings. It is particularly important to study the decisionmaking process of those who will be designing and running these experiments in the future. For instance, if decision-makers only believe that informed consent is necessary, they may think that they are unable to run experiments where it is unreasonable to get informed consent or, alternatively, believe that any experiment is acceptable as long as participants provide informed consent. Along these lines, we predict that both experts and the general public believe that deception and a lack of informed consent will be the most important influences of whether others view an experiment as ethical. Further, we predict that people believe others will tolerate the potential for harm in one condition, as long as the potential benefits of the other conditions are both large and likely enough. TBD (MTurk/WBL) TBD (MTurk/WBL/ Panel Survey) 2. Name of project advisor: Uri Simonsohn

4 3. Budget As outlined in the table below, the financial support of $4,000 that I am seeking will be used for two purposes: to fund data collection using the Wharton Behavioral Lab, Mechanical Turk, and a panel survey, and to share the results of my research at three conferences. I anticipate that this research will be of interest to the management and psychology audiences, and I hope to receive feedback on this project at three relevant conferences: the Association for Consumer Research (New Orleans, LA; October 2015), and the Society for Judgment and Decision Making (Chicago, IL; November 2015); and the Society for Personality and Social Psychology (San Diego, CA; January 2015). The budget outlined below reflects estimated costs of lodging (at conference room rates) and travel to and from the conferences. Expense category Explanation of Expense Research support $1,000 Subject payment: ~5 MTurk Experiments : ~$200 (15% * $0.50 avg. payment * 2,000 participants) Lab Study WBL participants: $300 (15% * $10 avg. payment *200 participants) Panel Survey: $500 Conference travel $1,150 (Association for Consumer Research; New Orleans, LA; October 2015) Flight - $400 Hotel (4 nights) - $450 Conference Registration - $300 $750 (Society for Judgment and Decision Making; Chicago, IL; November 2015) Flight - $300 Hotel (3 nights) - $350 Conference Registration - $100 $1,100 (Society for Personality and Social Psychology; San Diego, CA; January 2016) Flight - $500 Hotel (3 nights) - $400 Conference Registration - $200 TOTAL $4, Other sources of funding from OPIM department This research is not being supported by any other grants. The OPIM Department provides $800 annually toward doctoral student travel, but that money has already been allocated to attending one other conference (Academy of Management; Vancouver, BC; August 2015).

5 References Albergotti, R. (2014, June 30). Furor Erupts Over Facebook's Experiment on Users. Wall Street Journal. Retrieved March 15, 2015, from Allcott, H., & Rogers, T. (2012). The Short-Run and Long-Run Effects of Behavioral Interventions: Experimental Evidence from Energy Conservation. Retrieved from Emanuel, E. J., Wendler, D., & Grady, C. (2000). What Makes Clinical Research Ethical? JAMA, 283(20), doi: /jama Fryer, R. G. J. (2010). Financial Incentives and Student Achievement: Evidence from Randomized Trials. Retrieved from Gneezy, A., Gneezy, U., Nelson, L. D., & Brown, A. (2010). Shared social responsibility: a field experiment in pay-what-you-want pricing and charitable giving. Science (New York, N.Y.), 329(5989), doi: /science Grimmelmann, J. (2014, September 23). Illegal, Immoral, and Mood-Altering. Medium. Henrich, J., Heine, S. J., & Norenzayan, A. (2010). Most people are not WEIRD. Nature, 466(7302), 29. doi: /466029a Kahneman, D., Knetsch, J., & Thaler, R. H. (1986). Fairness as a Constraint on Profit Seeking: Entitlements in the Market. American Economic Review, 76(4), Retrieved from Kestenbaum, D., & Goldstein, J. (2013, August 23). Cash, Cows And The Rise Of Nerd Philanthropy. Retrieved March 15, 2015, from Kramer, A. D. I., Guillory, J. E., & Hancock, J. T. (2014). Experimental evidence of massive-scale emotional contagion through social networks. Proceedings of the National Academy of Sciences of the United States of America, 111(24), doi: /pnas Larson, S. (2014, July 28). Everyone's A Lab Rat In OkCupid's Labyrinth Of Love. Retrieved March 16, 2015, from List, J. A. (2011). Why Economists Should Conduct Field Experiments and 14 Tips for Pulling One Off. Journal of Economic Perspectives, 25(3), doi: /jep Rudder, C. (2014, July 28). We Experiment On Human Beings! Retrieved March 16, 2015, from Thaler, R. H., & Benartzi, S. (2004). Save More Tomorrow (TM): Using Behavioral Economics to Increase Employee Saving. Journal of Political Economy, 112(S1), S164 S187. Retrieved from

6 Waldman, K. (2014, June 28). Facebook s Unethical Experiment. Retrieved March 15, 2015, from _it_made_news_feeds_happier_or_sadder_to_manipulate.html World Medical Association. (2000). Declaration of Helsinki, ethical principles for medical research involving human subjects. 52nd WMA General Assembly, Edinburgh, Scotland. Young, K., Ashby, D., Boaz, A., & Grayson, L. (2002). Social Science and the Evidence-based Policy Movement. Social Policy and Society, 1(03), doi: /s