Quntifying perceived impct of scientific publictions Filippo Rdicchi, Alexnder Weissmn, nd John Bollen Center for Complex Networks nd Systems Reserch, School of Informtics nd Computing, Indin University, Bloomington, IN 47408, USA To build the infrstructure needed for the survey, we took ll scientific rticles with publiction yer up to 2013 from the Web of Science (WoS) dtbse. We ssocited every rticle with the totl number of cittions ccumulted until 2013 in the WoS cittion netrxiv:1612.03962v1 [cs.dl] 12 Dec 2016 Cittions re commonly held to represent scientific impct. To dte, however, there is no empiricl evidence in support of this postulte tht is centrl to reserch ssessment exercises nd Science of Science studies. Here, we report on the first empiricl verifiction of the degree to which cittion numbers represent scientific impct s it is ctully perceived by experts in their respective field. We run lrge-scle survey of bout 2, 000 corresponding uthors who performed pirwise impct ssessment tsk cross more thn 20, 000 scientific rticles. Results of the survey show tht cittion dt nd perceived impct do not lign well, unless one properly ccounts for strong psychologicl bises tht ffect the opinions of experts with respect to their own ppers vs. those of others. First, reserchers tend to lrgely prefer their own publictions to the most ppers in their field of reserch. Second, there is only mild positive correltion between the number of cittions of top ppers in given reserch res nd expert preference in pirwise comprisons. This lso pplies to pirs of ppers with severl orders of mgnitude differences in their totl number of ccumulted cittions. However, when reserchers were sked to choose mong pirs of their own ppers, thus eliminting the bis fvouring one s own ppers over those of others, they did systemticlly prefer the most rticle. We conclude tht, when scientists hve full informtion nd re mking unbised choices, expert opinion on impct is congruent with cittion numbers. I. INTRODUCTION Metrics bsed on bibliogrphic dt incresingly inform importnt decision-mking processes in science, such s hiring, tenure, promotion, nd funding [1 6]. Mny, if not most, of these bibliometric indictors re derived from rticle-level cittion counts [7, 8], e.g. the h-index [9] nd the journl impct fctor [10], due to the wide vilbility of extensive bibliogrphic records. However, they rest on the frequently undeclred ssumption tht cittions re n ccurte nd relible reflection of scientific impct. Bornmnn et l. [11] nd Srigol et l. [12] find tht cittions cn serve s resonble pproximtions of socil indictors such s populrity or success, but they do not support the most importnt ssumption underlying most work in the re of cittionbsed impct indictors, nmely tht cittions quntify scientific impct. Here, we investigte this ssumption directly by determining whether cittion dt truly reflect scientific impct s it is perceived by expert scholrs in their respective fields. Such n ssumption is centrl in the long-stnding debte bout the proper use of cittion dt [13]. One common rgument ginst the use of cittion dt in ssessment exercises rests in prticulr on the doubts bout their vlidity s indictors of true scientific impct [14 16]. In fct, the literture frequently confounds impct, influence, nd rnk with cittion-derived metrics [17 19] lthough these metrics cn vry long multiple distinct dimensions [20, 21]. Determining whether cittions ctully indicte scientific impct is n empiricl question which cn not be resolved by theoreticl discussions of the perceived or pre- Electronic ddress: filirdi@indin.edu sumed benefits vs. demerits of cittion dt lone. For this reson, we decided to define nd empiriclly quntify novel post-publiction metric, nmely impct s perceived by the uthors themselves. Our gol is to understnd whether cittion numbers truly reflect prticulr dimension of impct: the influence or importnce of ppers in the dily prctice of reserchers. We designed lrge-scle survey t bigscience. soic.indin.edu to collect responses from thousnds of experienced scholrs in multitude of disciplines (Fig. A1). We sked reserchers to mke pirwise decision with regrds to their preference of one pper over the other. These decisions were mde under two conditions: whether the rticles were written by the expert scholrs themselves or by other scholrs. We then ggregted results over the entire popultion of respondents to quntify the degree of correltion between the pirwise preferences of respondents (i.e., perceived impct) nd the ctul difference in the number of cittions ccumulted (i.e., cittion impct) for the pir of ppers. Our results indicte tht, from the perspective of individul reserchers, their ssessment of impct vs. impct judged from cittion dt re only relted for pirs of their own ppers. Every time tht pper not co-uthored by the individul is involved in the estimtion of perceived impct, this comprison shows null or negtive correltion with cittion impct. II. METHODS
2 work s n indiction of its cittion impct. We identified ll rticles tht were ssocited with corresponding uthor(s) from three mjor public universities in the US: (1) Indin University (IU), (2) University of Michign (UMICH), nd (3) University of Minnesot (UMN). We chose IU to run the first pilot experiment since it is our home university ; this would mke initil dt vlidtion more strightforwrd. UMICH nd UMN were selected since they hve the lrgest number of publictions mong ll public universities in the US. All three universities host deprtments in lmost ll disciplines of the nturl nd socil sciences, offering reltively unbised smple of prticipnts in terms of field of expertise. Articles were mtched to their corresponding uthor(s) using the emil ddress(es) provided in the rticle metdt. For exmple, emil ddresses ending in indin.edu were tken s n indiction tht the uthors were bsed t IU. Similrly for UMICH nd UMN, we identified reserchers from those institutions by retining emil ddresses tht ended with umich.edu nd umn.edu, respectively. In our dt set, rticles published prior to 1995 did not provide uthor emil ddresses. After 1995, the proportion of rticles crrying t lest one emil ddress incresed rpidly ech yer. The vst mjority of rticles provide t lest one emil ddress. This my introduce potentil source of bis in our study towrds reltively publictions, but this procedure generted number of very importnt dvntges. First, the ssocition between rticles nd physicl persons ws virtully free from homonymy-induced errors. Second, nd very importnt for our purpose, emil ddresses llowed us to directly contct reserchers nd invite them to prticipte in our lrge-scle survey. We sent n emil messge tht contined unique nd customized URL to every potentil prticipnt (Fig. A2). This URL pointed to web pge in which the respondent ws presented with mximum of ten pirwise comprisons between scientific ppers. In ech comprison, we provided only the journl nd yer of publiction of the ppers, their title nd list of uthors. For every pir of ppers, the respondent ws sked to select the one rticle she/he believed to be more influentil for her/his own reserch. Note tht this tsk did not involve ny considertion of, nor informtion on, cittion dt for ny of the two ppers. We tilored the survey for every respondent; ppers were selected from three different pools tht were constructed using informtion from the publiction nd cittion record of the respondent: (i) Own publictions (OWN), (ii) Top rticles (TCD), nd (iii) Rndom ppers (RND). The OWN pool consisted of rticles written by the respondents themselves, i.e., they were ssocited with the emil ddress of the responding uthor. The pool of TCD rticles ws constructed s follows. First, we identified ll rticles ppering t lest once in the reference lists of publictions by the respondent. We eliminted from this set rticles tht ppered lso in the pool of OWN rticles. We then rnked the remining rticles bsed on their cittion impct, nd selected the top 10 rticles in the list. This procedure llowed us to populte the TCD pool with the most populr rticles in the respondent s re of reserch, but tht were not written by the respondent her/himself. The pool of RND ppers ws simply generted s the union of rticles (co-)uthored or (not just the top ) by ll potentil prticipnts, thus comprising minly rticles unknown to the respondent. This pool ws used only for the IU smple s control set to check for the presence of possible systemtic bises in the survey. Severl respondents were confused when presented with rndom ppers. We therefore decided to remove the RND pool from subsequent surveys mde t UMICH nd UMN. Although the IU respondents reported discomfort bout hving to mke selections from the RND pool of ppers, the dt generted ws useful to vlidte construction of the other two pools, nd to test for the bsence of systemtic bises in the visul formt used in the online survey (Fig. A3). Once the three rticle pools were generted, every comprison presented to respondents in their personlized survey ws composed of pirs of ppers tken t rndom from the vrious pools. This llowed us to collect informtion bout the preferences of respondents mong pirs of ppers within the sme pool nd cross different pools. We recorded the preferences expressed by every prticipnt, nd we used it to estimte the perceived impct of one pper with respect to the other in the comprison. We sent emils for prticiption to totl of 19, 546 reserchers (Tble A1). 1, 819 scholrs prticipted in the survey, for totl of 12, 000 pirwise comprisons mong 20, 661 distinct rticles. This is low but cceptble response rte for n online survey with self-selected smple [22]. Importntly, we didn t observe ny systemtic bis in the pool of prticipnts in terms of cdemic ge (Fig. A4), lthough we cn not exclude the presence of ll selection bis in the smple of reserchers who prticipted in the survey [23]. We remrk tht our estimte of the response rte is conservtive; some of the emil ddresses we used to contct reserchers might be no longer ctive, e.g., through retirement or chnge of ffilition. Note tht the resulting survey smple ws more thn ten times lrger thn tht of ttempt to chrcterize the fetures of the top 10 most scientific publictions uthored by biomedicl reserchers [24]. III. RESULTS Figure 1 summrizes the reltion between cittion nd perceived impct obtined from the nlysis of comprisons between one pper tken from the OWN nd the other from the TCD pool. Nturlly, pper in the TCD pool is very likely to hve cittion impct lrger thn n rticle of the OWN, s the mesure of the probbility P (c > c b p = o, p b = t) shows in Fig. 1. Here, c x denotes the totl number of cittions ccumulted by pper x, nd p x denotes the pool of the pper x (o stnds for OWN, nd t for TCD). Nonetheless, perceived impct
3 Higher number of cittions Own Top b Own Top c 0 5000 10000 Cittion difference Figure 1: Perceived impct of uthored vs. top ppers. We consider N = 2, 916 comprisons between one pper tken from the Own pool, nd the other one extrcted from the Top pool. Probbility P (c > c b p = o, p b = t) tht the pper written by the respondent ccumulted more cittions thn the top- rticle. To estimte such probbility, we consider only comprisons between ppers with different number of cittions, so tht the difference in cittions is lwys different from zero. We obtin P (c > c b p = o, p b = t) = 05, corresponding to stndrd score z = 53.36 with respect to the unbised binomil model. The error br in the grph is centered t the unbised expected vlue, nd hs height equl to twice the vlue of the stndrd devition σ = 09. b Probbility P ( b p = o, p b = t) tht the respondent preferred her/his own pper to the one tken from the pool of Top rticles. Such probbility mounts to P ( b p = o, p b = t) = 0.61, corresponding to z = 11.48. c Probbility P ot( b c > c b, c c b c) tht the respondent preferred the pper with more cittions. Ech point represents the probbility of preference for the pper with higher number of cittions for ll pirs of ppers whose cittion difference c is higher thn the vlue reported on the x-xis. The horizontl line indictes the nive expecttion of the unbised binomil model. Height of the error brs equls two stndrd devitions of the binomil model. of rticles written by respondents themselves is generlly higher thn the ones of ppers from the TCD pool, reveling strong bis towrds respondents own ppers. This is visulized in Fig. 1B, where we consider the probbility P ( b p = o, p b = t), i.e., the probbility tht n OWN rticle ws preferred to one from the TCD pool (the nottion x y denotes preference of pper x with respect to pper y). To quntify the sttisticl significnce of our mesurements, we use bsolute vlues of the stndrd score z with respect to n unbised binomil distribution, hence z = (P )/σ, where P is the ctul vlue of the probbility mesured in the experiment, σ = /N, nd N is the size of the smple (i.e., number of comprisons). Note tht, since only two possibilities re vilble, it doesn t mtter if we mesure the probbility P or its complementry probbility 1 P. The bsolute vlue of the stndrd score z doesn t depend on this choice. The sttisticl interprettion of the stndrd score for n unbised binomil distribution with sufficiently lrge smple sizes, s in our cse, is similr to the one vlid for the stndrd norml distribution, so tht p-vlues < 01 pproximtely correspond to z > 3, nd the p-vlue decreses exponentilly fst s z increses. The empiricl results of Fig. 1B re highly unlikely to hppen by chnce. In fct, we hve P ( b p = o, p b = t) = 0.61 nd N = 2, 916, leding to z = 11.48. One my wonder whether this result is ctully dependent on the difference in cittion impct between the two ppers or not. Fig. 1C points to possible nswer to this question. We consider the probbility P ot ( b c > c b, c c b c) tht respondents preferred the more pper mong the two rticles in the comprison s function of the difference c in cittions between the two ppers (for brevity we used the suffix ot to indicte the pools where the two ppers were tken from). Surprisingly, cittion nd perceived impct re negtively correlted, in sttisticlly significnt mnner, for wide rnge of vlues of the difference of cittions ccumulted by the two ppers. Only when the cittion impct of the TCD pper is much lrger thn the one of the OWN pper, we do not longer observe sttisticlly significnt preference. The fct tht scholrs tended to systemticlly prefer their own rticles regrdless of the comprison cn be interpreted s the consequence of n egocentric [25] or fmilirity bis [26]. Egocentric bis doesn t necessrily hve negtive connottion. The bis could be reconcilble with the fct tht reserchers bse most of their work on results of their own pst reserch, nd they might hve interpreted the generic question posed in the survey in this wy. Furthermore, since reserchers re inherently most fmilir with their own work, the uncertinty with regrds to their impct might be lower nd hence these rticles might enjoy the uthors preference over pper tht is less fmilir nd whose impct is thus more difficult to ssess relibly. To discount for the presence of n egocentric or fmilirity bis, we turn our ttention to results obtined from comprisons where both ppers were extrcted from the sme pool. We strt from the description of our findings regrding the TCD pool (Fig. 2). Here, positive correltion is observed between cittion impct nd perceived impct. Overll, ppers with higher cittion impct were preferred t higher rtes by reserchers, but only with probbility P tt ( b c > c b ) = 1 leding to stndrd score z = 2.18 (p-vlue = 2, Fig. 2). Also, perceived impct doesn t vry significntly with cittion impct even if the difference in cittions received by ppers in the comprison cn be lrger thn three orders of mgnitude (Fig. 2b). The result seems not dependent on the ge of ppers, s we do not observe ny systemtic preference (Fig. A5). We speculte tht, from the perspective of individul reserchers, citing top- rticle in their own reserch re my be interpreted s public homge to the collective populrity of the pper, nd thus reconcilble with cumultive dvntge principle [19, 27, 28], the conformity bis [29], or copying behviour [30, 31], rther thn true recognition of the importnce of the work for their own reserch. Figs. 1 nd 2 suggest tht the totl number of cittions received by pper doesn t relibly quntify the true impct or influence tht the pper hs for the ctul reserch of scholr. Such conclusion is, how-
4 0.8 b 0.6 0.4 0.2 More 0.2 Less0.4 0.6 5000 0.8 10000 Cittion difference Figure 2: Perceived impct of top- ppers. We consider only comprisons between two ppers tken from the Top pool. Probbility P tt( b c > c b ) tht the respondent preferred the pper with higher/lower number of cittions. Ppers with higher cittion counts re preferred with probbility P tt( b c > c b ) = 1, clculted over smple of size N = 5, 930, leding to stndrd score z = 2.18. Error br is centered t nd hs height equl to twice the stndrd devition σ of the unbised binomil model. Here σ = 07. b Probbility P tt( b c > c b, c c b c) tht the respondent preferred the pper with more cittions s function of the difference in cittions mong the ppers. The horizontl line indictes the nive expecttion of of the unbised binomil model. Height of the error brs equls two stndrd devitions of the binomil model. 0.8 b 0.6 0.4 0.2 More 0.2 Less0.4 00.6 50 0.8 100 Cittion difference Figure 3: Perceived impct of uthored ppers. We consider only comprisons between two ppers tken from the Own pool. A Probbility P oo( b c > c b ) tht the respondent preferred the pper with higher/lower number of cittions. Ppers with higher cittion counts re preferred with probbility P oo( b c > c b ) = 0.61 clculted over smple of size N = 1, 302, leding to stndrd score z = 7.60. Error br is centered t nd hs height equl to twice the stndrd devition σ of the unbised binomil model. Here σ = 14. B Probbility P oo( b c > c b, c c b c) tht the respondent preferred the pper with more cittions s function of the difference in cittions mong the ppers. The horizontl line indictes the nive expecttion of of the unbised binomil model. Height of the error brs equls two stndrd devitions of the binomil model. Probbility distribution 10 1 0.8 10 2 0.6 0.4 10 3 0.2 10 1 b 10 2 10 3 10 1 c 1 4 10 4 10 4 200 100 0 0.2 100 200 20000.4 1000 0 10000.62000 1000 500 0.8 0 500 1000 Cittion difference Figure 4: Cittion impct of rticles with higher perceived impct. We consider only comprisons between two ppers tken from the Own pool s in Fig. 3. We include here lso comprisons between ppers hving the sme number of cittions, so tht the difference of their cittion impcts cn be equl to zero. The gry brs stnd for the probbility distribution P oo(c c b = c b) of the difference in cittion impct between preferred nd non-preferred rticle in the comprison. The verge vlue of the distribution is 11.16, the stndrd devition is 130, nd the skewness is 29.28. The blck line serves s term of comprison s it represents the distribution P oo(c c b = c) of difference in cittion impct between pirs of ppers in the comprisons, irrespective of the preference expressed by reserchers. This distribution is by definition symmetric (null skewness), nd centered in zero (verge vlue equl to zero). Probbility distributions re normlized such tht the integrl below the curves equls one. b Sme s in pnel but for comprisons between one pper tken from the Own pool nd the other extrcted from the Top pool. Dt re the sme s those used in Fig. 1. The verge vlue of the distribution P ot(c c b = c b) is 267.50, the stndrd devition is 4762.69, nd the skewness is 1.36. The blck line stnds for the unconditionl probbility P ot(c c b = c). c Sme s in pnels A nd B but for comprisons between two ppers tken from the Top pool. Dt re the sme s those used in Fig. 2. The verge vlue of the distribution P tt(c c b = c b) is 99.47, the stndrd devition is 4255.18, nd the skewness is 12.34. The blck line stnds for the unconditionl probbility P tt(c c b = c). ever, overturned when we consider pirwise comprisons between rticles from the OWN pool (Fig. 3). In this cse, we observe significnt lignment between cittion impct nd expert impct s reserchers systemticlly preferred the highest rticles mong their own rticles. to the more rticle ws given with probbility P oo ( b c > c b ) = 0.61, corresponding to z = 7.60 (Fig. 3). Furthermore, we observed systemtic increse in the preference for the more pper s function of the difference of cittion impct between the ppers in the comprison (Fig. 3b). vlues sturte to lmost 0.75, if cittions ccumulted by ppers differ by 50 or more. This effect is even stronger when one ccounts for the generl tendency of preference for more publictions, nd the fct tht more publictions generlly exhibit lower cittion impct (Fig. A6). Overll, this finding conclusively supports the observtions by Ionnidis et l. in their smll-scle survey [24]. Our observtions re further supported by the results 10 2 10 3
5 presented in Fig. 4, where we consider the distribution of the difference in cittion impct between preferred nd non-preferred rticles in different types of comprisons. The distribution is clerly right-skewed for comprisons between ppers in the pool of OWN rticles (Fig. 4). The distribution for comprisons between rticles from the OWN nd TCD pools on the other hnd hs negtive verge vlue, nd is negtively skewed (Fig. 4b). Finlly, the distribution in Fig. 4c, for comprisons between ppers in the TCD pool, provides evidence of slight preference for rticles with higher number of cittions, but the preference is much less evident thn in the cse of Fig. 4. IV. CONCLUSIONS In summry, our work quntifies the degree of correltion between cittion impct nd new post-publiction metric, nmely impct s perceived by the uthors themselves. Our survey serves to understnd whether cittion numbers truly reflect the impct of ppers s it pertins to the dily prctice of reserchers. This is prticulr dimension of impct tht hs been not quntified before. The results from our lrge-scle survey generte one very importnt conclusion: cittion numbers pproximte with good ccurcy the perceived impct of scientific publictions, but only when psychologicl bises tht mke uthors prefer their own ppers bove those of others re removed. This conclusion cn be justified by the resonble ssumption tht the self-ssessment of one s own ppers is the most objective (nd lest bised) quntifiction of perceived impct since uthors re ssumed to know their own rticles best nd re thus the best judges of their comprtive impct. Another possible explntion could be the Goodhrt effect [32], nmely tht the preferences of uthors re shped by the knowledge of cittions ccumulted by their own ppers. As result, pirwise ssessments of their own ppers will be congruent with cittion dt. Acknowledgements We re grteful to ll reserchers who took prt in our survey. This work uses Web of Science dt by Thomson Reuters, provided by the Network Science Institute t Indin University. This work is funded by the Ntionl Science Foundtion (grnt SMA-1446078 nd SMA-1636636). [1] A.-W. Hrzing, The publish or perish book (Trm Softwre Reserch, 2010). [2] L. Bornmnn nd H.-D. Dniel, Scientometrics 68, 427 (2006), ISSN 0138-9130. [3] L. Bornmnn, G. Wllon, nd A. Ledin, PLoS One 3, e3480 (2008). [4] B. G. Lovegrove nd S. D. Johnson, Bioscience 58, 160 (2008). [5] S. Hornbostel, S. Böhmer, B. Klingsporn, J. Neufeld, nd M. von Ins, Scientometrics 79, 171 (2009). [6] L. Bornmnn, R. Mutz, W. Mrx, H. Schier, nd H.-D. Dniel, Journl of the Royl Sttisticl Society: Series A (Sttistics in Society) 174, 857 (2011). [7] J. Br-Iln, Journl of Informetrics 2, 1 (2008). [8] R. Vn Noorden, Nture 465, 864 (2010). [9] J. E. Hirsch, Proceedings of the Ntionl cdemy of Sciences of the United Sttes of Americ 102, 16569 (2005). [10] E. Grfield, JAMA: the journl of the Americn Medicl Assocition 295, 90 (2006). [11] L. Bornmnn nd H.-D. Dniel, Journl of Documenttion 64, 45 (2008). [12] E. Srigöl, R. Pfitzner, I. Scholtes, A. Grs, nd F. Schweitzer, rxiv preprint rxiv:1402.7268 (2014). [13] E. Grfield, Scientometrics 1, 359 (1979). [14] M. H. McRoberts nd B. R. McRoberts, Scientometrics 36, 435 (1996). [15] R. Adler, J. Ewing, nd P. Tylor, Sttisticl Science 24, 1 (2009). [16] M. H. McRoberts nd B. R. McRoberts, Journl of the Americn Society for Informtion Science nd Technology 61, 1 (2010). [17] S. Wuchty, B. F. Jones, nd B. Uzzi, Science 316, 1036 (2007). [18] F. Rdicchi, S. Fortunto, nd C. Cstellno, Proceedings of the Ntionl Acdemy of Sciences USA 105, 17268 (2008). [19] D. Wng, C. Song, nd A.-L. Brbási, Science 342, 127 (2013). [20] L. Bornmnn nd H. Dniel, Journl of Documenttion 6, 45 (2008). [21] J. Bollen, H. Vn de Sompel, A. Hgberg, nd R. Chute, PloS one 4, e6022 (2009). [22] K. B. Sheehn, Journl of Computer-Medited Communiction 6, 0 (2001). [23] J. Bethlehem, Interntionl Sttisticl Review 78, 161 (2010). [24] J. Ionnidis, K. W. Boyck, H. Smll, A. A. Sorensen, nd R. Klvns, Nture 514, 561 (2014). [25] M. Ross nd F. Sicoly, Journl of personlity nd socil psychology 37, 322 (1979). [26] R. B. Zjonc, Journl of personlity nd socil psychology 9, 1 (1968). [27] D. d. S. Price, Journl of the Americn society for Informtion science 27, 292 (1976). [28] A.-L. Brbási nd R. Albert, science 286, 509 (1999). [29] S. E. Asch, Psychologicl Monogrphs 70, 1 (1956). [30] P. Krpivsky nd S. Redner, Physicl Review E 71, 036118 (2005). [31] M. V. Simkin nd V. P. Roychowdhury, rxiv preprint cond-mt/0212043 (2002). [32] C. A. Goodhrt, Problems of monetry mngement: the UK experience (Springer, 1984).
1 Appendix Figure A1: Survey design. Screenshot from the online survey t bigscience.soic.indin.edu. The visuliztion shows n exmple of comprison presented to individul prticipnts. Ppers visulized in this exmple were rndomly chosen from our dtset.
2 Hello, You re receiving this emil becuse you re the corresponding uthor of severl scientific publictions. We re conducting survey, the lrgest of its kind, to determine the reltive importnce of scientific publictions s perceived by experienced reserchers. We ve generted survey specificlly for you tht consists of nywhere between 1 nd 10 pirs of ppers drwn from our dtbse. The publictions my be drwn from your own publictions or publictions you ve. For ech pir, we sk you to select the publiction tht you believe is more influentil for your own reserch. It should tke less thn two minutes to complete the ten comprisons. To prticipte, plese follow this link: bigscience.soic.indin.edu/ccess-token/{{ccess_token}} This survey is step towrds criticlly ssessing the wy the scientific community mesures the impct of its work. We hope you will prticipte nd help further this importnt reserch. This survey is being conducted by Alexnder Weissmn nd Filippo Rdicchi from the School of Informtics t Indin University. The reserch is sponsored by the NSF (Awrd #1446078) nd pproved by the Indin University IRB (Protocol #1407480218). For more informtion, plese write to Filippo Rdicchi: filirdi@indin.edu. With regrds, Alexnder Weissmn nd Filippo Rdicchi School of Informtics nd Computing Indin University Figure A2: Emil messge for prticiption in the survey. We sent this messge to ll potentil prticipnts in our survey. The emil messge contined unique token key for every individul prticipnt, pointing to her/his customized survey.
3 b c Own Rndom Top Rndom Rndom left Rndom right Figure A3: Sensitivity nlysis. Results of this figure re bsed on the experiment we conducted t Indin University, the only experiment where ppers from the Rndom pool were considered. Probbility P ( b p = o, p b = r) tht the respondent preferred the pper from the Own pool insted of the pper from the Rndom pool. Such probbility is P ( b p = o, p b = r) = 0.97 mesured over N = 587 totl comprisons, nd leding to stndrd score z = 22.50. b Sme s in pnel A but for comprisons between one pper from the Top pool nd the other from the Rndom pool. In this cse, we hve: P ( b p = t, p b = r) = 0.98, N = 578, nd z = 22.96. c Results for comprisons where both ppers were tken from the Rndom pool. We mesure the probbility P ( b g = L, g b = R) tht the respondent selected the pper ppering on the left (g x = L) or right (g x = R) side of the screen during the survey. We hve: P = 3, N = 550, nd z = 1.62.
4 Number of scholrs 4000 3000 0.8 0.6 2000 0.4 1000 0.2 Pot. prticipnts Respondents Response rte 0.15 0.10 5 0 0 1990 2000 0.2 20100.4 1990 0.6 2000.8 2010 Yer of first publiction b Figure A4: Prticiption in the survey. Totl number of potentil prticipnts (gry line) nd respondents (red line) to the survey s functions of the yer of their first publiction ppering in our dtbse. The first publiction corresponds to the oldest pper where the emil ddress of the scholr ws ppering in the rticle metdt. b Response rte (i.e., number of respondents divided by number of potentil prticipnts) s function of the yer of their first publiction in the dtbse. Institution Potentil prticipnts Respondents Response rte Indin University 2, 673 313 11.7% University of Michign 9, 560 889 9.3% University of Minnesot 7, 313 617 8.4% 19, 546 1, 840 9.4% Tble A1: Prticiption in the survey. Summry tble listing number of potentil prticipnts, number of respondents, nd response rte for individul institutions. In the the lst row of the tble we list vlues of the sme quntities for the entire survey.
5 More Less b 0 5 10 15 20 Pub. yer difference Higher number of cittions c More Less Figure A5: Perceived impct of top- publictions depending on the ge of ppers. We consider only comprisons between two ppers tken from the Top- pool s in Fig. 2 of the min text. Probbility P tt( b y > y b ) tht the respondent preferred the more/less pper (y x is the yer of publiction of pper x). To estimte the probbility, we consider only comprisons between ppers with different yers of publiction, so tht their difference is different from zero. More ppers re preferred with probbility P tt( b y > y b ) = 1 clculted over smple of size N = 5, 609, leding to stndrd score z = 5. Error br is centered t nd hs height equl to twice the stndrd devition σ of the unbised binomil model. Here σ = 07. b Probbility P tt( b y > y b, y y b y) tht the respondent preferred the more pper mong the two in the comprison s function of the difference of yer of publiction of the two ppers. The horizontl line indictes the nive expecttion of of the unbised binomil model. Error brs stnd for stndrd devition of the binomil model. c Probbility P tt(c > c b y > y b ) tht the more/less pper in the comprison hs ccumulted more cittions. To estimte the probbility, we considered only comprisons between ppers with different yers of publiction nd different cittion numbers, so tht the difference of both these numbers is different from zero. The probbility is P tt(c > c b y > y b ) = 0.42, clculted over smple of size N = 2, 742, leding to stndrd score z = 8.02. Stndrd devition computed ccording to the unbised binomil model is σ = 09. More Less b 0 5 10 Pub. yer difference Higher number of cittions c More Less Figure A6: Perceived impct of uthored publictions depending on the ge of ppers. We consider only comprisons between two ppers tken from the Own pool s in Fig. 3 of the min text. Probbility P oo( b y > y b ) tht the respondent preferred the more/less pper (y x is the yer of publiction of pper x). To estimte the probbility, we consider only comprisons between ppers with different yers of publiction, so tht their difference is different from zero. More ppers re preferred with probbility P oo( b y > y b ) = 6 clculted over smple of size N = 1, 207, leding to stndrd score z = 3.94. Error br is centered t nd hs height equl to twice the stndrd devition σ of the unbised binomil model. Here σ = 14. b Probbility P oo( b y > y b, y y b y) tht the respondent preferred the more pper mong the two in the comprison s function of the difference of yer of publiction of the two ppers. The horizontl line indictes the nive expecttion of of the unbised binomil model. Error brs stnd for stndrd devition of the binomil model. c Probbility P oo(c > c b y > y b ) tht the more/less pper in the comprison hs ccumulted more cittions. To estimte the probbility, we considered only comprisons between ppers with different yers of publiction nd different cittion numbers, so tht the difference of both these numbers is different from zero. The probbility is P oo(c > c b y > y b ) = 0.27, clculted over smple of size N = 550, leding to stndrd score z = 10.83. Stndrd devition computed ccording to the unbised binomil model is σ = 21.