ETHICS OF PUBLICATION

Transcription

1 ETHICS OF PUBLICATION Ethics in Psychological Science Beyond how participants are treated, its of huge importance to be honest in reporting data Data most often comes from unverifiable sources (e.g., anonymous survey responses) Makes results extremely easy to fabricate Can just tweak a few data points Can make up data all together Without scientific integrity, this field cannot operate Fabricating Data Example: Diederik Stapel Tenured researcher at the prestigious Tilburg University in the Netherlands Over a decade of research;; about 150 publications Cited in many major news sources In 2011, his own grad students came forward, stating that He tweaked data He published data from experiments that never happened! 1

2 Fabricating Data Why did he do it? There are scarce resources, you need grants, you need money, there is competition. Normal people go to the edge to get that money. Science is of course about discovery, about digging to discover the truth. But it is also communication, persuasion, marketing. I am a salesman. I am on the road. People are on the road with their talk. It s like a circus. -Stapel, 2013;; New York Times Fabricating Data How was he able to do it? He was published in many wellrespected journals He collaborated with many wellrespected psychologists He oversaw over 50 doctoral students This field lacks over-sight! His data was never once checked by anyone other than him Why Lie? Professional progress in academia depends on attaining publishable findings Whether a study reveals what you want it to can be the difference between Getting hired or fired Getting a promotion Getting research grants Getting tenure With results being so easy to tweak or fabricate, it can be very tempting 2

3 Significance without Lying There are also legitimate methods that can be used to artificially increase significance E.g., trying to show that Head-Start program increases adult intelligence To better chances of finding significant effect Measure many different types of intelligence E.g., IQ, academic success, measures of logic, mathematics, verbal and spatial intelligence Compare different types of Head-Start programs E.g., examine head-start programs (compared to control groups) at 5 different schools Significance without Lying This seems like a good idea, right? Researchers are being very thorough in their measures However, by assessing so many different IVs (e.g., 5 head-start programs) and DVs (measures of intelligence), chances of Type-I Error go up For every test, there is a 5% chance of Type-I Error With 5 comparison schools and 6 measures of intelligence 150 potential comparisons! Likely to result in 7.5 Type-I errors (false positives) So, even if head-start had no effect, you could find about 7 8 effects just by chance in this study design Significance without Lying To illus t rat e how eas y it is t o f ind f als e pos it iv es Simmons et al. (2011) randomly assigned participants to listen to different types of music E.g., classical, pop, children s music And then report many different demographics E.g., age, race, gender, GPA Results showed that listening to classical music caused people to become younger 3

4 Significance without Lying Simmons et al. (2011) demonstrated that if you collect many measures, some are likely to show significant effects just due to chance Since there is a 5% likelihood for Type-I error for every test If you run 20 tests, 1 is likely to show a false positive Simmons demonstrated you can show just about any effect if you measure it in enough ways And then only report on the measures that show the effects you hypothesized Called fishing or casting a wide net Tweaking the Data There are many other ways to increase the likelihood of Type-I Error Continually check for significance during data collection and then stop collection once significance is attained This takes advantage of chance differences between small groups of participants Simmons et al. (2011) ran computer simulations testing for significant effects between groups that were sampled from the exact same distribution I.e., should not show significant differences Tweaking the Data Continually testing for significant difference during data collection Type-I error becomes less Opportunity for likely with Type-I Error larger samples p <.05 4

5 Tweaking the Data If a researcher began testing for significant differences after initial collection of 20 participants Actual likelihood of Type-I Error goes up to 22% The HORROR! That s way beyond the accepted 5% chance of error! No Accountability This is a big problem in any scientific field If researchers don t make their data public, how can we double-check their work or detect fraud? Wicherts, Borsboom, Kats, and Molenaar (2006) attempted to collect data from 249 randomly selected studies published in top-tiered journals ed authors repeatedly for 6 months Forwarded confirmation of their credentials Included petitions from several national ethics boards Assured researchers they wouldn t share their data No Accountability After 6 months of repeated requests Only 26% shared their data Most took months of prodding to comply From experimental researchers = 41% From clinical researchers = 22% From developmental researchers = 15% 5

6 No Accountability After 6 months of repeated requests Only 26% shared their data Most took months of prodding to comply From experimental researchers = 41% From clinical researchers = 22% From developmental researchers = 15% However, there was really no good reason to withhold the data And researchers rarely cited this as the reason for not sharing To some extent, this could have been to protect sensitive data No Accountability Goes against APA s own standard of ethics! Article 8.14.a of APA Ethical Standards: After research results are published, psychologists do not withhold data on which their conclusions are based from other competent professionals who seek to verify the substantive claims through reanalysis - APA, 2002 All IRBs require psychologists to follow APA ethical guidelines, so these psychologists are in breach of their own research contracts No Accountability Is this really such a big deal? Wicherts, Bakker, and Molenaar (2011) went on to double-check results from over 1,100 published articles Almost 50% contained some type of statistical error! In majority of cases, these errors increased the size of hypothesized effects In 17% of articles, errors caused non-significant effects to be found significant (i.e., Type-I Error)! Wicherts et al. found that those who refused to share their data were much more likely to publish such errors 6

7 No Accountability Not willing to share = More errors in reporting data Serious reporting errors: No Accountability Not willing to share = More errors in reporting data Errors resulting in Type-I Error: How big a problem is this? John, Loewenstein, and Prelec (2012) anonymously sampled over 2,000 US psychologists 70% admitted to cutting corners in reporting data E.g., not using appropriate tests/ criterion for significance 65% admitted to not reporting all dependent and independent variables 60% admitted to continually analyzing data during data collection and ending collection after finding significance 45% admitted to only reporting on findings that worked in their favor 7

8 How big a problem is this? John, Loewenstein, and Prelec (2012) anonymously sampled over 2,000 US psychologists 40% admitted to excluding data (e.g., outliers) only if they found the extra data-points hurt their results 30% admitted to re-phrasing projects/ papers to reflect findings that were not originally predicted 35% admitted to doubting their own integrity as a researcher About 1% admitted to fully manufacturing data That s about 20 researchers just like Diederik Stapel! Average Rating Across Fraud Types How big a problem is this? Respondents were even more willing to admit that other psychologists were committing acts of fraud What can we do? Just like we solved the problem of unethical treatment of participants in the 1970s, we need to solve unethical treatment of data now Increase accountability Having ethics boards check methods and data for errors, fraudulence, and techniques used to increase Type-I error Increase transparency Having researchers publically state variables/ hypotheses before data collection and then make their data public Decrease pressure to publish Scientists wouldn t feel so pressured to commit fraud if their careers didn t depend on attaining significant findings 8

9 What can we do? Psychologists want change! In a survey of over 1,000 research psychologists across 42 countries (Fiedler, Fuchs, & Jenny, 2013) Most researchers don t want to compete in a field where everyone is publishing data on steroids Researchers reported that 50 70% of their experiments meant to simply replicate established findings don t work! Because past findings are based on tweaked data Huge waste of time and money 65% of those surveyed want more transparency and accountability in reporting data, especially null findings What can we do? Psychologists want change! In a survey of over 1,000 research psychologists across 42 countries (Fiedler, Fuchs, & Jenny, 2013) 85% want all psychologists to report all of their findings, including null/ contradicting results And for psychologists to state their hypothesis in public record before data is collected 95%+ want publishers to be less demanding of flawless results that perfectly match researchers hypotheses Change is already happening! Leading psychologists (Jeffrey Spies, Brian Nosek) recently created the Center for Open Science Offers free online tools (Open Science Framework) to help psychologists conduct open and honest research Much like Google Docs, plus Psychologists can register their hypothesis before data-collection (records it in a public forum) Psychologists can also register their materials, methods, participant recruitment plan, etc Offers special tools for data entry/ analysis Creates publisher-friends results, graphs, citations, etc. 9

10 Change is already happening! OSF and other open forum services are now being required by many colleges and grants Conducting research on OSF offers publishers assurance that data was collected and reported honestly Researchers who use these services don t have to worry about others questioning the legitimacy of their work Many more psychological findings are being published in PLOS ONE Which gladly accepts null results from well done studies Is being used to uncover many well-established findings that do not reliably replicate Change is already happening! Statisticians are also busy creating methods of detecting tweaked data from previous publications Leif Nelson (2013) developed a method of meta-analysis that can detect tweaked data 95%+ of the time Simple method: P-curve analysis Plots how often p-values of results are safely low ( ) or dangerously high ( ) High p-value = High likelihood of Type-I Error If results rely on data on steroids, it s p-curve across studies is skewed toward.05 Change is already happening! Data that hasn t likely been tweaked 10

11 Change is already happening! Data that has likely been tweaked Change is already happening! These methods can be used On as few as five experiments examining the same effect Regardless of experiments sample size On data that has been published long ago Only requires knowing p-values, which are always reported Reveals p-hacking in virtually 100% of cases Leif Nelson and others have already convincingly ousted some psychologists The hunt is on for psychologists who consistently tweak data and established findings based on p-hacking Example Two lines of research on choice Barry Schwartz, Andrew Ward and others have found that more choices lead to More agonizing over making the right choice Greater expectations of outcomes of choice More upward counterfactuals ( I could have gotten something better ) Less satisfaction with the whole process Barry Schwartz has made a career from these findings Numerous papers and presentations, including TED talks His best-selling book, The Paradox of Choice 11

12 Example Two lines of research on choice However, virtually every economic model out there predicts that more choice leads to more satisfaction Greater likelihood of offering the right product for each person In open markets, competitors that offer more choices (Amazon, Ebay) almost always win out Schwartz theory, backed by experimental findings, seem to fly in the face of realworld findings Example Simonsohn, Nelson, & Simmons (2013) conducted p-hacking analyses on these lines of research On More choice = More satisfaction findings Experimental Evidence Field Evidence Left-skew = p-values around.01 on average Example Simonsohn, Nelson, & Simmons (2013) conducted p-hacking analyses on these lines of research On More choice = Less satisfaction findings Experimental Evidence Field Evidence Right-skew = p-values around.05 on average 12

13 Example Simmons, Nelson, & Simonsohn (2012) have also ousted Norbert Schwartz and Dirk Smeesters Smeesters found exposing people to red colors created contrast effects and blue colors assimilation effects Red color followed by picture of another person à Participants more likely to see dissimilarities between themselves and the other person Blue color à Participants more likely to see similarities Blue colors proposed to lessen prejudice and shown to do so in follow-up study Results revealed through p-curve analysis and other methods to be a product of fraudulent techniques Example Simmons, Nelson, & Simonsohn (2012) have also ousted Norbert Schwartz and Dirk Smeesters Because of this, Smeesters and Schwartz were both placed under formal investigations Smeesters has since confessed to massaging data to strengthen outcomes and resigned from his department Example Other psychologists have recently been found out to be publishing fraudulent data Lawrence Sanna claimed people behaved more prosocially when they were physically elevated ( coming from a higher place ) Found to be fraudulent by Simonsohn s statistical tools Ap Dijksterhuis claimed priming people with idea of professors made them perform better on tests Failed to replicate in 10+ well conducted tests published on PlusOne William Brown claimed body symmetry was related to dancing ability Reported by colleague who couldn t replicate findings 13

14 Replicating Past Studies Led by COS, Psychology Reproducibility Project is replicating 100 findings published in 2008 Launched in 2011, published in 2015 All scientists used COS to pre-register study design and provided their data for open analysis Found 61% of the studies could not replicate past results, with 15% having results totally contrary to those of the original studies In other words, published psychology findings are likely only around 40% valid and reliable Replicating Past Studies Believe it or not, that s still a relatively high level of replicability for any scientific domain Meta-analyses reveal that only 18% of studies finding positive effects of medications replicate Similarly, only 20% of findings in cancer studies replicate 30% of studies using DNA or RNA sequencing replicate Findings at the frontiers of science are inherently unreliable simply because they haven t yet been replicated But psychology, especially, has a problem working to replicate past results How will the field evolve? Now that it s becoming increasingly difficult to get away with tweaking data Psychologists are likely to focus less on findings with weak effect sizes that aren t likely to replicate Less reliance on surprising or sexy findings that might make headlines, but aren t based on good theory More interest in null results if they come from studies with good methodology (large samples, established measures, registered hypotheses) Hopefully greater interest in attaining nationally representative, non-college samples 14

15 How will the field evolve? Increased efforts to reproduce past findings Special replication sections of journals are being established to increase publication rate of such studies, even if they get null results Many of these also require researchers to use COS COS is continuing with replication projects Collaborative Replications and Education Project (CREP) formed to encourage undergrads to do the same Posted on Open Science Framework ( Foundations recently donated $4 million to Stanford to form the Center for Reproducible Neuroscience 15