The Quantification and Prediction of Graduate School Performance

Transcription

1 The Quantification and Prediction of Graduate School Performance by Christopher Zou A thesis submitted in conformity with the requirements for the degree of Doctorate of Philosophy Psychology University of Toronto Copyright by Christopher Zou 2016

2 The Quantification and Prediction of Graduate School Performance Abstract Christopher Zou Doctor of Philosophy Psychology University of Toronto 2016 In today s competitive job market, many students graduate with a bachelor s degree and move on to pursue higher education in hopes of standing out from the crowd. As the number of graduate students has steadily increased in the recent years, there is also growing interest in understanding how to cultivate successful graduate careers. Surprisingly, there has been little empirical work examining this topic. The current dissertation has two primary aims: 1. to quantify graduate school performance and 2. explore some of the potential predictors of success. Because publication output is believed to be the most valuable metric of success, the primary focus was on publications. Chapter 1 examines the publication trajectories of graduate students, postdoctoral fellows, and junior faculty members to establish a benchmark of success. Junior faculty members had twice the number of publications as post-doctoral fellows, who had twice the number of publications as senior graduate students. Chapter 2 and 3 aim to better accurately quantify a researcher s scientific output beyond just the number of publications with the introduction of a new bibliometric index called the zp-index. The zp-index, compared to other bibliometric indices, better discriminated the productive output of high-functioning academics. Finally, Chapter 4 identifies potential personality predictors of graduate school performance. Industriousness was a key variable that predicted publication success. Results across the studies ii

3 provide practical recommendations to apply to the current graduate school system and highlight some potential avenues for future research on this understudied topic. iii

4 Acknowledgments I would like to first and foremost thank my supervisor, Dr. Jordan B. Peterson, for giving me this opportunity to pursue my passion in psychology and research. Without him, none of this would have been possible and I owe him my deepest gratitude for taking the chance on me and the numerous research projects I have conducted under his wing. I would also like to thank my secondary supervisor, Dr. Judith P. Andersen, in supporting me throughout my graduate school career. Her continuous encouragement throughout this journey has been a great source of motivation for me to invest countless number of hours into my research. I also am deeply grateful for the guidance of my committee member, Dr. Wil Cunningham, who from the start has shown great excitement and deep insight into this line of work, and this dissertation has greatly improved with his involvement. My mentors have not only contributed to my own development as a better scientist, but also a better person. They have inspired me in so many ways and I truly feel lucky to have had the opportunity to work with these amazing individuals. I also must thank my parents, who have always supported me in pursuing my goals, whatever they may be. I would also like to thank the many friends I have made over the past number of years who made all of this really fun. Finally, I would like to thank the numerous research assistants who were involved in these projects. Each study in this dissertation was a collaborative effort and required countless hours of time and effort from so many people. It would have been impossible to complete alone, but with the combined efforts of my mentors and research assistants, the impossible became possible. I truly cannot thank them enough. iv

5 Table of Contents Acknowledgments... iv Table of Contents...v List of Tables... vii List of Figures... viii Chapter 1 Introduction The Quantification of Graduate School Performance zp-index The Prediction of Graduate School Performance Personality and Academic Performance Personality and Job Performance Supervisor s Impact on Student Performance Research Overview...11 Chapter 2 Study Study Method Results Discussion...7 Chapter 3 Study Study Method Results Authorship Value Work Distribution Discussion...15 Chapter 4 Study v

6 Study Method Results Discussion...22 Chapter 5 Study Study Method Results Discussion...30 Chapter 6 General Discussion...36 General Discussion Theoretical Contributions and Applications Limitations and Future Directions...37 References or Bibliography (if any)...62 vi

7 List of Tables Table 1. Perceived Work Distribution of Authorship by Position...52 Table 2. Publication Information of New Psychology Professors by Area...53 Table 3. Correlation Matrix of Different Indices of Scientific Output predicting H-index...56 Table 4. Ranking of Cognitive Psychology Professors by Scientific Output...58 Table 5. Correlations between Student and Supervisor Personality...59 Table 6. Predicting Number of Student Publications from Supervisor s Personality Factors...60 Table 7. Predicting Number of Student Publications from Supervisor s Personality Aspects...61 vii

8 List of Figures Figure 1. Graduate Student s Self-Reported Number of Publications across Year of Study...41 Figure 2. Graduate Student s CV-reported Number of Publications across Year of Study...42 Figure 3. Average Number of Publications among Post-Doctoral Fellows across Years of Post- Doctoral Fellowship...43 Figure 4. Average Number of Publications among Faculty Members Hired from Figure 5. Comparison of Publication Numbers between Faculty, Post-Docs and Senior Graduate Students...45 Figure 6. Perceived work distribution across authorship position among psychology researchers..46 Figure 7. Flow chart describing the selection process of new faculty members...61 Figure 8. Frequency Distribution of the Self-Reported Number of Publications among Graduate Students...48 Figure 9. Student Politeness predicting Number of Publications by Year of Study...49 Figure 10. Comparison of Student and Supervisor Personality Aspects...50 Figure 11. Supervisor Openness on the Number of Publications by Student Year of Study...51 viii

9 Chapter 1 Introduction Psychology is one of the most popular fields of study among university students. In 2006, in the United States alone, 90,000 students received their bachelor s degree in psychology, and this number is growing linearly at the rate of 20,000 additional students per year (US Department of Education, 2006). As interest in psychology grows, the number of students enrolled in graduate programs in psychology also increases. Approximately 20% of students enrolled in undergraduate psychology programs pursue a master s degree in psychology, and approximately 5% pursue a doctorate (Halonen, 2011). While some psychology graduate students enter the clinical stream, others are interested in the research stream, in hopes of one day obtaining an academic career as a tenure-track professor. However, as the number of graduate students in research-oriented psychology programs expands, the job market for an academic position becomes increasingly competitive. According to the most recent statistics, only 18% of PhD graduates in psychology attain an academic job within 5 years of obtaining their degree (National Science Board, 2012), and about 40% of doctorate recipients end up in non-academic jobs after completing their program (Weir, 2011). While the United States produces the second most number of science doctoral graduates after China, the proportion of those graduates who obtain tenured academic positions has been steadily decreasing from about 40% in 1981 to 30% in 2005 (Cyranoski, Gilbert, Ledford, Nayar, & Yahia, 2011). Given this heightened competition (see also, Lametti, 2012), it would be of great pragmatic benefit for both the graduate students and their supervisors to obtain a better understanding of the general academic requirements and achievements needed for the student to be competitive on the job market upon graduation. While there has been great interest in this topic, there is little consensus on what is known about graduate programs in psychology. This is due primarily to the lack of empirical research dedicated to this area, and the bulk of information tends to come from anecdotal evidence found on student forums and blog posts (e.g., The Grad Café Forums). For instance, even simple statistics, like the average number of publications among psychology graduate students, is difficult to find because there is little consensus about what that number actually is. The present 1

10 2 thesis aims to fill some of the gaps on two aspects of psychology graduate schools the quantification and prediction of graduate school performance. 1.1 The Quantification of Graduate School Performance Without a doubt, one of the most important requirements to obtain an academic position is to have publications, so many graduate students set a goal to publish prolifically. However, in the absence of more detailed information, such an unspecified goal may prove motivationally ineffective, and hence less than optimal worth pragmatically. Goal progress can often be facilitated by setting concrete goals, which can then be effectively broken down into smaller goals (Emmons, 1992). Accurate empirical knowledge about what standard of productivity allows for success in the academic job market would help graduate students and their supervisors create a more meaningful and effective plan of success over during the duration of the Master s/ph.d. training process. By all appearances, strong publication records lead to success in the academic job market. However, data supporting this supposition are surprisingly scarce. Valla s (2010) commentary in The APS Observer, Getting Hired, appears to provide the most information currently available. He assessed academic psychology employment ads posted in the APS Observer and Monitor from August 2006 to July 2008, and gathered information on 33 recent faculty hires. Valla concluded that the average number of publications among those hired as social psychology professors was around ten, about half of which were first-author publications (Valla, 2010). This number halved for cognitive psychologists and halved again for developmental psychologists (Valla, 2010). In addition to sheer quantity, it is also important to note that it is also crucial to produce high-quality papers. In fact, most newly hired professors had published at least one paper in a top-tier journal in their respective fields (see also, Bohannon, 2013). However, because Valla s (2010) piece in the magazine was meant to be a quick commentary, not much detail is given in terms of how the data was gathered. Additionally, his analyses were conducted on a small sample of professors, so we wanted to address these limitations. Chapter 2 adds to this literature by providing the most up-to-date information about the current academic productivity of graduate students based on self-reports and copies of their Curriculum Vitae (CV).

11 3 Another issue with the currently available information on graduate school performance is that it either uses a metric that is too simplistic (i.e., the number of publications) or inappropriate for use with newer researchers (e.g., h-index, g-index) in quantifying their scientific output. Research interest in the quantification of research productivity has led to the creation of a multitude of indices, widely cited by researchers across different fields (Bornmann & Daniel, 2009). The quantification of research productivity facilitates the comparisons across researchers on how impactful their work is to the field. It also helps others identify potential rising stars in a particular area of research or recognize who may be an expert in a specific field of study. However, the utility of quantifying an early researcher s research productivity has perhaps been underemphasized. For instance, the quantification of research productivity can also be beneficial for graduate students who are hoping to land a tenure-track position as a faculty member in a well-established institution. A productivity index might also help students compare their own progress to the progress of others, so that they can make the necessary adjustments to their graduate program to facilitate the attainment of their goals. Additionally, a research productivity index for emerging researchers may also be helpful to academic institutions when making hiring decisions, or choosing a candidate for a grant or award so that they can better rank potential candidates on a more objective criterion. Using an objective criterion for election procedures of scientific academies can ensure fairness and transparency in their selection process (Ball, 2005). Thus, there are numerous reasons to try and create and validate an index of early-stage research productivity. Therefore, we introduce the zp-index in Chapter 3, which characterizes not only the quantity of publications, but also its impact (using the impact factor of the journal the article is published in), and takes authorship position into account zp-index The simplest and the most common metric of scientific output is a simple count of the number of publications (Bateman & Hess, 2015), which has been shown to have some utility in predicting long-term success in the field (Simonton, 1997). Additionally, the production of good-quality papers is also seen as a requirement for hiring committees. However, there is a close relationship between quantity and quality (r = ~.8), in that more productive researchers tend to produce higher quality scientific work (Abramo, D Angelo & Di Costa, 2010). Despite this strong correlation, there are instances where the quality of publications, typically based on the number of times the paper has been cited, can provide information that greatly differs from the quantity

12 4 of publications. For instance, the ranking of research performance in economics departments across the United States differ substantially depending on whether the ranking was based on the quantity or quality of publications (Laband, 1985). Therefore, a metric of scientific output that incorporates both the quantitative and qualitative measures of publications is perceived to be a superior measure than quantity alone. Because of the limitations with the simple count of research papers, the h-index arose as the most prominent index of a researcher s productivity (Hirsch, 2005). The h-index uses information about the number of researchers publications along with their citation impact to derive the score. Specifically, the h-index refers to the number of articles (h) published by the individual with h number of citations. The h-index has been shown to have some predictive validity, as it can predict future scientific achievement (Hirsch, 2007), attainment of an NIH grant (Rezek, McDonald, & Kallmes, 2011), and even future Nobel Prize winners (Hirsch, 2005). While the h-index is a widely popular metric of scientific output, it also suffers from a number of critical limitations. First, the h-index does not take authorship position into account, despite authorship position being indicative of one s contribution to the paper (Wren, Kozak, Johnson, Deakyne, Schilling, & Dellavalle, 2007). Second, the h-index is restricted by the total number of publications, so two researchers with the same h-index can have greatly different number of citations (Cacioppo, 2008). Lastly, and most importantly for the present purposes: the h-index tends to increase with the number of years as a scientist, so the metric is less informative in the case of newer researchers (i.e., graduate students, post-doctoral fellows, new professors). Specifically, Kosmulski (2006) suggested that the h-index is a more suitable measure for mature scientists with at least 50 publications with a minimum h-index of 10. The primary reason why the h-index is not appropriate for newer researchers is that it relies primarily on citation count, which requires some time to accrue (Garfield, 2006). While other metrics of the h-index have been constructed, such as the g-index (Egghe, 2006), contemporary h-index, and the i10-index (Google Scholar Blog, 2011), they all suffer, for the purposes of early career evaluation, from their reliance on citation counts (Cacciopo, 2008). At the present moment, in fact, there is no established metric of scientific output that does not rely on citation counts. Therefore, there is no existing measure of scientific output that can be appropriately applied to assess new researchers. Chapter 4 demonstrates the application of the zp-index by

13 5 illustrating its utility in quantifying the productivity of new psychology professors and compares its predictive validity with commonly used scientometric indices (e.g., number of publications, h- index). 1.2 The Prediction of Graduate School Performance Many graduate students in psychology progress through their program with the primary goal of publishing prolifically, as publications are an important determinant of their future attainment of an academic job (Darley, Zanna, & Roediger, 2004; Stenstrom et al., 2013). While information about the predictors of graduate school performance can be quite fruitful, the topic has been understudied. Much of the literature that discusses the factors that influence a graduate student s performance has been based on anecdotal evidence from people s personal experiences with minimal and indirect empirical evidence (e.g., Croft, 2003; Darley et al., 2004). Furthermore, many of the studies do not focus exclusively on a particular field, which is an issue because each area of study has their own field-specific characteristics that change the structure of their graduate program. For instance, many graduate programs in the STEM fields require a much higher course load than a typical psychology graduate program. Therefore, if we want to know how to improve the performance of graduate students in psychology, then it is crucial to focus on empirical evidence from psychology students where there is virtually none. For the present study, we examined personality as one potential variable that may influence graduate school performance, and our primary focus for this work was on psychology graduate students. Much of the current research on the predictors of graduate school performance has focused on GRE scores and undergraduate GPA, which has shown to be valid predictors of graduate school performance (e.g., Kuncel, Hezlett, & Ones, 2001). However, there has also been growing interest in identifying non-intelligence related measures of performance due to fears that the current intelligence test may be disadvantageous to certain minority groups (Miller & Stassun, 2014; Salisbury, 2014). Although this is a common misperception of well-established intelligence tests (e.g., Kuncel & Hezlett, 2010), other variables that predict graduate school performance can work in conjunction with intelligence measures to better predict performance. In particular, personality is another potential variable that may influence a graduate student s productivity (Swaminathan, 2012). Personality has been shown to be a powerful predictor of

14 6 performance in a variety of domains, such as in relationships (Simpson, 1990), health (Marshall. Wortman, Vickers, Kusulas, & Hervig, 1994), and income (Judge et al., 2012). Thus, we hypothesized that personality would also influence graduate school performance. However, much of the evidence is based on indirect evidence drawing from other research in related areas, such as academic performance at the undergraduate level (e.g., Swaminathan, 2012), and so a direct link between personality and graduate school performance has not been demonstrated. By understanding which personality factors are related to graduate performance in which direction, professors can be particularly vigilant for the qualities that predict graduate school success. Additionally, current graduate students can also learn to embody some of the personality characteristics of successful graduate students to maximize their chances of success. Undergraduate students wondering whether they should pursue a graduate degree in psychology research can perhaps examine whether their personality is suited for the career path. In sum, there is potentially great utility in examining the relation between personality and graduate school performance for researchers involved at all levels of academia. Chapter 5 delves into this topic by examining how graduate students and their supervisor s personalities can influence the student s publication success Personality and Academic Performance While little research has focused on the predictors of graduate school performance, there is an abundance of work that has examined the predictors of academic performance among undergraduate students (Wolfe & Johnson, 1995). In particular, conscientiousness has been demonstrated to be a consistent positive predictor of GPA (r = ~.20; e.g., Noftle et al., 2007). There is also some mixed evidence that openness, neuroticism, and agreeableness may also predict academic performance (e.g., Komarraju et al., 2009), though their effects are not consistent (Noftle & Robins, 2007). Given the similarities between undergraduate and graduate programs, we could hypothesize that the predictors of undergraduate performance will also be related to predictors of graduate programs. However, there are some key differences between academic performance at the undergrad and the graduate level that should be taken into consideration. One clear difference is in the primary outcome variables. For undergraduate performance, success is most commonly assessed with GPA scores (e.g., Richardson et al., 2012), while graduate school performance is often quantified

15 7 by the number of publications (Darley et al., 2004). For instance, there have been a handful of articles that have examined the quantity of publications of successful graduate students (i.e., new professors; Stenstrom et al., 2013; Valla, 2010), which demonstrates the field s particular interest in publications. Additionally, undergraduate GPA is often comprised of a series of exams, while in contrast, there is minimal test-taking in research-oriented graduate programs. It is possible that the effects of personality on GPA are primarily driven by personality variables that predict poor performance on tests, such as neuroticism (e.g., Chamorro-Premuzic & Furnham, 2003; Schneider, 2014), which hamper student s ability to cope with anxiety. This would suggest that perhaps the variables that predict good test-takers are not necessarily going predict graduate school success. Second, undergraduate students often take classes with a number of different professors while graduate students typically work closely with one or sometimes two professors (Zanna & Darley, 2004). While it is possible, and quite likely, that many of the predictors of GPA will also predict research productivity, given these substantial differences between the structure of undergraduate and graduate programs, it is unclear which of the findings related to the predictors of GPA will apply to predictors of graduate school performance Personality and Job Performance While research on academic performance has remained quite separate from job performance, graduate schools share many similar characteristics with work and the overlap is far greater at the graduate level than at the undergraduate level. For instance, many jobs require employers to often interact with their primary supervisor, as is also the case in graduate school where graduate students often interact with their professors. Graduate programs in psychology also typically do not emphasize regular testing, and do not follow a regular academic schedule, which is also characteristic of a job. There is also a body of research that has demonstrated the utility of personality in predicting job performance. Like academic performance, conscientiousness is a moderate and consistent predictor of job performance (r = ~.20; e.g., Hurtz & Donovan, 2000). Additionally, depending on the context of the job, extraversion and openness to experience can also predict successful job performance (e.g., Barrick & Mount, 1991). However, there are some key differences between jobs and graduate schools that should be noted.

16 8 First, jobs follow a typical structured schedule whereas graduate programs in psychology often do not follow a regular 9-5 schedule. Second, jobs typically provide a consistent extrinsic reward (i.e., salary) on a biweekly basis while graduate schools do not. Thus, these two major structural differences between jobs and graduate schools may influence the relation between personality and successful performance. Therefore, it is unclear whether the personality predictors of job performance will also predict graduate school success Supervisor s Impact on Student Performance Another factor that has been overlooked in the literature is the impact of a graduate student s supervisor on their performance. As previously mentioned, graduate students typically work closely with one or sometimes two supervisors. It has been suggested that a good working relationship with the research supervisor is crucial in fostering an environment for success in graduate school (Darley et al., 2004). In related work on job performance, much of the work has focused on the validity of supervisor ratings (e.g., Hunter & Hunter, 1984), and factors that contribute to worker s ratings of their supervisor (e.g., Piedmont & Weinstein, 1994). Less work has focused on the characteristics of the supervisor that contribute to employee performance, although from the handful of studies that have examined this topic, the general consensus is that supervisor s behaviours do have a significant impact on performance. For instance, supervisory support has been shown to reduce stress and increase job satisfaction (Babin & Boles, 1996) and ethical leadership is positively related to manager performance (Kim & Brymer, 2011). There has been an abundance of work that has examined the impact of leadership styles on performance. For instance, a handful of studies have demonstrated the effectiveness of a transformational leadership style in boosting follower performance. For instance, a meta-analysis by Judge and Piccolo (2004) found a strong correlation between transformational leadership and follower job satisfaction (r =.58) and a moderate correlation with group performance (r =.26). Transformational leadership is distinguished from transactional leadership, where the former inspires followers and stimulates growth, while the latter is characterized the give-and-take relationship (Conger & Kanungo, 1998) 1. Additionally transformational leadership has been 1 Researchers of transformational leadership often define the concept by its outcome (e.g., inspiring others, building trust), making it difficult to separate the cause from the effect.

17 9 shown to be related to the big five personality traits, where extraversion, openness, agreeableness, and conscientiousness are somewhat correlated positively with transformational leadership styles (r = ~.2), while neuroticism is weakly negatively correlated (r = -.14; Bono & Judge, 2004). There have also been a handful of studies that have examined the role of supervisor personality on job satisfaction. Supervisor extraversion is positively related to employee job satisfaction and work motivation, whereas supervisor neuroticism is negatively related (Porter, Wrench, & Hoskinson, 2007; Smith & Canger, 2004). However, these studies did not examine how supervisor s personality directly relates to job performance. While job satisfaction and job performance are related, it is only moderately correlated with one another (r = ~.30; Judge, Thoresen, Bono, & Patton, 2001). Thus, a direct examination of supervisor s personality on worker performance would be beneficial. Hence, the present study examined the role of supervisor s personality on the publication performance of graduate students. There has also been some work that has examined the impact of teachers on student s performance. Once again, transactional leadership among teachers has some indirect effect on student academic performance (Koh et al., 1995). A greater match between the teacher s cognitive style (i.e., how information is received and interpreted) and the student s cognitive style boosts student performance (Cafferty, 1980) though, this learning styles hypothesis has not been strongly supported by empirical evidence (Pashler, McDaniel, Rohrer, & Bjork, 2008). Additionally, greater teacher qualifications also contributes positively to student achievement in reading and mathematics (Darling-Hammond, 2000). At the postsecondary level, more frequent faculty-student interactions and better quality of interactions were positively associated with academic performance (Anaya & Cole, 2001). Interestingly, more qualified (i.e., degree attained, level of professorship) and better experienced professors were found to negatively correlate with contemporaneous course performance, but positively correlated with follow-up course performance though these effects are generally small (~0.6% change in final percentage grade: Carrell & West, 2008). These results suggest that the characteristics of the teacher do have some influence in affecting student performance. There have also been a few studies that have examined the impact of teacher s personality on student evaluations (e.g., Isaacson, McKeachie, Wilbert, & Milholland, 1963; Rushton, Morgan,

18 10 & Richard, 2007). For instance, Murray and colleagues (1990) found that although the personality of effective teachers varied by course type (e.g. introductory, required honors), generally speaking, extraverted professors were rated more favourably, while neurotic supervisors were rated less favourably. They also specifically examined student ratings from graduate courses and found that effective teachers were less neurotic, but extraversion had no effect. Although the study only examined student ratings of teacher s effectiveness rather than student performance, their results illustrate that undergraduate courses differ from graduate courses to the extent where instructor extraversion influences student ratings in the former, but not the latter. Because the focus of the current study was on publication output, which is even more disparate from course grades, it is likely that the supervisor predictors of undergraduate course performance would not necessary behave in the same manner when predicting publication success. While previous studies shed some light on the potential characteristics of an effective academic supervisor, there are a few major differences in the organizational structure of academia and jobs that should be highlighted. First, previous work in examining the role of leadership on worker performance has typically focused on larger groups, whereas in graduate programs, each supervisor is responsible for supervising just a handful of students. Most of the work in education has also typically focused on elementary or high school settings (e.g, Brophy & Good, 1984), rather than on how faculty members influence college students at the undergraduate level (e.g., Anaya & Cole, 2001). Second, while the relationship between graduate students and their supervisors may begin as a leadership-follower relationship, it often develops into a more collaborative relationship as students progress through their graduate program. Therefore, it is unclear whether the student-supervisor relationship can be fully characterized in terms of leadership and following. The present study examined how the graduate student s perception of their supervisor s personality impacted the student s publication success. Because the nature of the student-supervisor relationship typically changes as the student progresses through the graduate program, it is possible that the type of effective supervision may depend on the seniority of the student. For instance, beginning graduate students may require a more authoritative, orderly and assertive professor. On the other hand, senior graduate students may benefit more from supervisors who foster a collaborative environment, through compassion

19 11 and flexibility. Thus, we also examined if the student s year of study moderates the relation between supervisor s personality and the student s publication success. 1.3 Research Overview The current program of research aims to quantify and predict post-graduate performance. Chapter 2 presents the most up-to-date information about the publication numbers of graduate students, post-doctoral fellows, and newer faculty members. Chapter 3 introduces the zp-index, which quantifies the quantity and quality of a new researcher s scientific output. Chapter 4 demonstrates the utility of the zp-index by applying the index among new psychology professors. In addition to better quantifying graduate performance, Chapter 5 delves into the predictors of graduate school performance by examining the influence of student and supervisor s personality on student productivity.

20 Chapter 2 Study 1 Study 1 While there has been a number of published statistics detailing the academic achievements of successful graduate students (i.e., graduate students who land a job anywhere in academia), in general, there are a number of problems with these statistics that need to be addressed. First, many of the available statistics are not applicable to individuals in the field of psychology. Based on an unpublished report by the National Science Foundation (NSF), over the period from 1988 and 2001, the social sciences (which includes psychology) ranked last behind other fields in science (e.g., medical sciences, computer sciences) in terms of the average number of publications during the time span (Javitz et al., 2010). Thus, the standard of academic achievements needs to be derived from psychology students and professors for the statistics to be truly relevant to psychology students. Second, the standards for professorship differ across time, as well as across fields. As previously noted, increasing numbers of students are studying psychology, many with an eye to a career. In addition, various economic and administrative pressures have been detrimentally affecting university hiring, in a cumulative manner, both chronically and phasically. In 1975, for example, 57% of faculty at US degree-granting institutions were tenured or on the tenure track. By 2009, this proportion had fallen by almost half. During the same period, part-time faculty proportions increased from 30 to approximately 50% (Weir, 2011). Additionally, the number of doctoral students in psychology have also increased considerably. Specifically, there are about 40% more doctoral graduates in 2013 when compared with 2003 (Brian, 2008). Based on the statistic that 50% of PhD students are interested in pursuing an academic research career (Sauermann & Roach, 2012), the average doctoral student in psychology is estimated to have just an 18% chance of landing a tenure-track position at graduation (Brian, 2015). Note that this is likely a fairly conservative estimate as it does not take into consideration the number of post-doctoral fellows and professors who are looking to switch academic institutions in the potential job market. 1

21 2 Furthermore, federal research spending has stagnated relative to inflation, since 2004 (Vastag, 2012). The recent recession particularly hurt the availability of assistant professorships. The number of posted advertisements for an assistant professorship in Geography, for example, dropped from 197 in 2007 to an all-time low of 65 positions in 2010 (Franklin & Ketchum, 2013). According to the NSF, during the period of 1973 to 2008, the number of full-time faculty positions declined in all fields in the sciences, but the declines were the steepest in the life sciences and psychology (National Science Board, 2012). Although the rate of hiring has been improving over the past few years (Franklin & Ketchum, 2013), there remains a large pool of applicants due to the accumulation of unemployed PhD graduate students from the economic downturn. All of this makes it even more important to provide accurate information about what qualifies an applicant for an academic job. A recent article published in the Perspectives on Psychological Science found that students who secured a job in academia had produced 5-6 publications, on average, of which 2-3 were firstauthor publications (Stenstrom, Curtis, & Iyer, 2013). However, these statistics were derived from self-reported numbers from graduate students. Additionally, the study was voluntary, so participation bias may well have systematically influenced some of the statistics. Furthermore, many reports (e.g., Carleton, Parkerson, & Horswill, 2012) only provide information about publication number, without taking position of authorship into account. Because the position of authorship of a paper indicates the relative contribution of each author, it is important for researchers to publish papers in which they are in the higher ranking of authorship (e.g., first authorship) (Wren, Kozak, Johnson, Deakyne, Schilling, & Dellavalle, 2007). Psychology researchers generally value authorship position, and in particular, seek first authorship positions (Zou & Peterson, in press), so for the present paper, we present the number of first-authored publications alongside the total number of publications. Chapter 2 s primary aim was to provide graduate students and their professors in psychology with a more accurate representation of the current standards in attaining a tenure-track academic position in Canada and the United States. In addition, we also provide useful information about the publication patterns of graduate students and post-doctoral fellows. Additionally, we were interested in characterizing students and post-doctoral fellows who were successful in attaining an academic job, to examine the typical standards that hiring committees may expect in the current job market. Therefore, we also collected data from professors who were recently hired, to

22 3 examine what graduate students should aim for if they are interested in pursuing an academic career. The present statistics provide the most up-to-date numbers from graduate students, postdoctoral fellows, and new faculty members. Furthermore, the information was obtained through both self-reports and CVs to examine whether there are any reporting biases or participation biases. 2.1 Method The present study utilizes four databases from three samples (graduate students, post-doctoral fellows, junior faculty members). A description of each database is detailed below: Graduate Student Survey. We recruited 1,283 psychology graduate students from the top 100 research-oriented psychology programs in the United States and 11 schools in Canada to participate in a short survey. This study was conducted over a school semester from September 2013 to January There was a generally uniform distribution of the year of study (MA = 63, PhD 1 = 217, PhD2 = 201, PhD 3 = 202, PhD 4 = 208, PhD 5+ = 325), and students came from a wide variety of areas (Development = 112, Social/Personality = 239, Cognitive = 167, Brain & Behavior = 116, Clinical = 375, Quantitative = 28, Other = 184). We selected schools from a list of research-oriented psychology programs from the National Academy of Sciences and the Social Psychology Network until we reached a targeted number of participants (>1,000) that provided enough power to detect small effects (Kenny, 1987). We obtained lists of all the graduate students from each institution. If this information was not readily available, the next school was chosen. Each student was individually ed with the survey link to increase rate of participation. By participating in the study, they were entered into a raffle with a total prize pool of $500. Each student was asked to report the number of peer-reviewed publications, including those in press. Additionally, students were asked to indicate how many out of the total number of publications were first-authored. Graduate Student CVs. Student CVs were collected over the course of We selected a list of all the schools (n = 100) included in the NAS database and an additional 11 schools in Canada listed in the Social Psychology Network. We searched each school and lab s websites to obtain a copy of the graduate student s full CV. If the student s CV was not found, we omitted them from the database. A total of 744 CVs were collected. More CVs were collected from the upper years of graduate school (MA = 32, PhD 1 = 80, PhD 2 = 91, PhD 3 = 144, PhD

23 4 4 = 118, PhD 5+ = 160, Unknown = 121) and students came from a variety of different subdisciplines of psychology (Social/Personality = 169, Cognitive = 225, Developmental = 37, Biological= 19, Quantitative = 14, Clinical = 213, Health psychology = 16, Industrial/Organizational = 38). Post-Doctoral Fellows. In conjunction with the collection of the graduate student CVs, we also collected CVs from post-doctoral fellows from the same sources (if available). We collected a total of 142 CVs. The year of post-doctoral training was dispersed (Year 1 = 6, Year 2 = 44, Year 3 = 42, Year 4 = 27, Year 5+ = 23, Unknown = 16), and most of the post-doctoral CVs were collected from cognitive psychologists (Social/Personality = 20, Cognitive = 99, Developmental = 5, Biological = 16, Quantitative = 6, Clinical = 3, Health Psychology = 1, Industrial/Organizational = 1). Note that because the focus of the paper was on psychology researchers, those who were pursuing to become a clinical psychologist were not included in the database. New Faculty Members. To obtain the most relevant numbers, we restricted our recruitment to all psychology professors hired from 2010 to 2013/2014 (some professors were hired a year in advance) in research-oriented universities from the top 92 American universities in psychology research according to the National Academy of Sciences (NAS) and 23 Canadian universities with a PhD program in psychology. All professors in the sample were hired as tenure-track professors (with the exception of certain Ivy league schools that do not offer tenuretrack assistant professorships). The CVs were collected from the professors personal website, their lab website, or academic social networking sites. If the CV was not available online, we directly contacted the professor. We collected a total of 201 CVs from associate or assistant professors. The largest proportion of new hires came from the cognitive psychology subdiscipline (Cognitive = 79, Social/Personality = 37, Clinical = 27, Developmental = 20, Biological = 14, Quantitative = 10, Other = 10). Because we were interested in the academic characteristics of professors before they are hired (i.e., successful graduate students), we only examined the number of publications before they were hired (including the year they were hired). For more details about the sample description and recruitment process, refer to Zou & Peterson, 2015.

24 5 For each database, we examined the average number of publications and the average number of first-authored publications. We also examined whether there were any differences in self-report data and the CV data by comparing the numbers from the Graduate Student Survey and the Graduate Student CVs. Additionally, we wanted to examine whether there were significant differences between graduate students who get hired into faculty positions and graduate students who have not yet been hired (i.e., senior graduate students, post-doctoral fellows). This could potentially address the question of whether academia hiring is based on actual scientific merit, or potentially on other extraneous factors such as the place of graduation (Stenstrom et al., 2013), and gender (Williams & Ceci, 2015). 2.2 Results The mean number of publications for graduate student s self-reported number of publications was 2.89 (SD = 3.53). As we would expect, the number of publications significantly increased with the year of study (r(1061) =.43, p <.001). About 38.4% (M = 1.11, SD = 1.68) of publications were first-authored. For instance, first-year PhDs had on average 1.15 publications (SD = 1.76) while fourth-year PhDs had on average 3.47 (SD = 2.82) publications. Figure 1 provides a breakdown of the number of publications by year of study. The mean number of graduate student s CV-reported number of publications was 3.08 (SD = 3.76). Following the patterns of the self-reports, the number of publications once again increased along with the year of study (r(619) =.30). For instance, first-year PhDs had on average 1.56 publications (SD = 3.12) while fourth-year PhDs had on average 3.69 (SD = 4.07) publications. About 38.9% (M = 1.20, SD = 1.91) of the total number of publications was first-authored, which did not differ from the self-report data. Figure 2 provides a breakdown of the number of publications across the different year of study. There were no significant differences in the number of total publications and the number of first authorships between the self-report (i.e., Graduate Student Survey) and the CV (i.e., Graduate Student CVs) data, suggesting that the effects of self-report biases are potentially minimal when collecting publication information. The mean number of publications among post-doctoral fellows was 8.06 (SD = 5.22). Due to the few number of post-doctoral fellows in health psychology (n =1), clinical psychology (n = 3), quantitative (n = 5), and developmental (n = 5), they were removed from the analysis. The mean number of post-doctoral training was 3.18 (SD = 1.27). This differed significantly by area (F(2,

25 6 124) = 5.83, p =.004), where post-doctoral fellows in cognitive psychology had significantly more years of training than social/personality psychologists (d = 1.04, SE =.31, 95% CI [.30, 1.78], p =.003). To reduce the effect of outliers (a handful of post-doctoral fellows with over 6 years of training), we merged anyone with over 6 years of post-doctoral training into one category. The number of publications significantly increased with the number of years of postdoctoral training (r(140) =.20, p =.02) and this effect held even when controlling for the area of psychology. Figure 3 provides a breakdown of the number of publications by years of postdoctoral training. Among new psychology professors, the number of publications did not differ by the year hired ( ), suggesting that the standards of academic achievement have not differed much in the recent years. The average number of publications was (SD = 8.34). 51.3% (M = 7.34, SD = 4.85) were first-authored. The number of publications also differed by whether the professor received post-doctoral training. Specifically, professors with post-doctoral training had on average 3.67 (95% CI [1.57, 5.74], t(196) = 3.45, p =.001) more publications and 2.57 (95% CI [1.38, 3.76], t(196) = 2.57, p <.001) more first-authored publications than professors without post-doctoral training. Figure 4 provides a breakdown of the number of publications by year hired. Because the job market comprises senior graduate students (PhD 4+) and post-doctoral fellows, we were interested in comparing their publication performance to each other and against junior faculty members. Additionally, because there were significant differences between junior professors with and without post-doctoral training, we separated these categories. We also controlled for area of psychology because its distribution was not uniform across the four samples. There was a significant difference across the different positions (F(3, 614) = 97.91, partial n 2 =.32, p <.001). New faculty members with post-doc had a significantly higher number of publications than new faculty members without post-doc, post-doctoral fellows, senior and graduate students (diff = 3.66, 95% CI [1.52, 5.79], SE =.81; diff = 5.75, 95% CI [4.02, 7.47], SE =.65, p <.001; diff = 9.63, 95% CI [8.10, 11.17], SE =.58, p <.001, respectively). New faculty members without a post-doc had a significantly higher number of publications than senior graduate students and post-doctoral fellows (diff = 5.98, 95% CI [4.05, 7.90], SE =.73, p <.001 and diff = 2.09, 95% CI [4.05, 7.90], SE =.73, p =.05 respectively). Post-doctoral fellows also had significantly more publications than senior graduate students (diff = 3.89, 95 % CI

26 7 [2.43, 5.34], SE =.55, p <.001). The same pattern was observed for first-authored publications. Figure 5 provides a breakdown of the comparison between these four groups. We also wanted to examine if there were any non-linear effects of publication numbers on academic position. In particular, hiring committees may bias against potential candidates with too many publications because it may be a marker of a shallow thinker. Because 50% of graduate students are typically interested in pursuing an academic career (Sauermann & Roach, 2012), they were omitted from the analysis. Additionally, we omitted new faculty members without post-doctoral fellowships for a fairer comparison between post-docs who were hired and who remained as post-docs. Because our outcome was binary (0 = post-doc, 1 = faculty), we conducted a logistic regression controlling for area. As expected, the number of publications predicted whether the post-doc was hired (OR = 1.16, 95% CI [1.10, 1.21], p <.001). There was no evidence of non-linear relation (i.e., quadratic or cubic) between the number of publications and hiring. We were also curious to see if the number of publications differed by school rankings. We ranked the schools on the New Faculty database based on the NAS rankings ( 2. There was no significant correlation between the school rankings and the number of publications. When examining the number of publications by ranking (i.e., top 10 vs. top 25 vs. others), there is some evidence that professors from elite schools had more publications at hiring than lower-ranked schools: professors from the top 10 schools (n = 4) had publications (SD = 7.48), the top 25 schools (13) had publications (SD = 7.03), and the others (n = 152) had on average publications (SD = 7.40). 2.3 Discussion The present study provide a genuinely accurate illustration of the typical progression of a psychology graduate student to a post-doctoral fellow to a junior faculty member at a top-tier 2 We used the.95 R scores to rank the schools. This is the regression-estimate of the 95 th percentile ranking of each department based on a number of important dimensions of faculty productivity (e.g., publications per allocated faculty, cites per publication, percent faculty with grants).

27 8 university. We provide the most up-to-date statistics and also provide a more detailed description by assessing the proportion of first-authored papers alongside its total number. We were also able to examine the progress of senior graduate students (i.e., PhD 4+), post-doctoral fellows, and junior faculty members, to compare groups who are simultaneously competing for the job market. Additionally, we obtained both self-reports and CV-reports to examine the impact of self-report biases in reporting publication numbers. A number of studies have attempted to examine the academic achievements that are necessary in obtaining an academic job in the current competitive market (e.g., Stenstrom et al., 2013). However, much of the data relied on small samples (e.g., Valla, 2010), and often did not distinguish between the types of academic job obtained (e.g., Stenstrom et al., 2013). It is important to note that the New Faculty CV data came from the top-tier colleges in psychology research, while the graduate student and the post-doc data were obtained from a wider range of schools listed on the NSF. Thus, it is likely that the differences between the faculty members and the other groups are somewhat exaggerated because top-tier schools often have higher standards for hiring. We focused only on the top end of the spectrum of faculty members because most academics are interested in pursuing a professorship on a tenure stream (rather than a lectureship and adjunct position). Therefore, our focus for the present paper was on the attainment of an academic position as a psychology professor in a top-tier university with a primary focus on research. Practical considerations also played a role: Faculty CV coding took an average of 1 hour due to the high number of publications and had to be kept to a manageable number. It would nonetheless be interesting for future researchers to examine how the publication rates differ by the different academic jobs. It is likely that the numbers of publications attained by junior faculty members in a top-tier university are among the highest in comparison with other academic jobs. There has been some work that has assessed non-academic factors that contribute to the hiring of professors, such as gender (Williams & Ceci, 2015), and PhD attainment from an elite university (Clauset, Arbesman, & Larremore, 2015). Anecdotal evidence also suggests that luck is often involved in the successful attainment of an academic job (Guo et al., 2013; proflikesubstance, 2013). However, it was clear that number of publications significantly increased as seniority increased: junior faculty members published the most, then post-doctoral fellows, then senior graduate students. In fact, the numbers of publications among new professors doubled that of

28 9 post-docs, which was double that of senior graduate students. Therefore, it would be incorrect to say that the hiring decisions are based on luck. However, that is not to say that other external factors do not come into play during hiring decisions. For instance, if two candidates have a similarly competitive CV, then it is likely that other factors become more prominent when making hiring decisions. Nevertheless, because there are significant and fairly large systematic differences between new faculty members, post-doctoral fellows, and senior graduate students, it would be unfair to claim that hiring committees neglect an individual s scientific merit when making decisions. We also have some evidence to suggest that higher-ranked schools have higher expectations of publishing than lower-ranked schools. However, this conclusion was based on a fairly restricted sample of faculty members from already established institutions (i.e., top 100 schools in North America). Whether this pattern would still hold once we include a wider range of schools remains to be seen. Additionally, we were only able to examine how the number of publications affects hiring decisions. However, there are other aspects of publishing, such as the quality of publications, which are also important (Valla, 2010). Additionally, these analyses are not necessarily able to capture all the nuances involved in hiring decisions. For instance, a researcher who publishes many papers in low-tier journals may be seen as someone who lacks quality control. There may be certain labs that are more generous in giving authorships, which job search committees may factor in to their decisions. These details are better assessed in a more direct study by surveying members of hiring committees, which would be an interesting and important avenue for future research. For the present study, our sample consisted only of professors who were hired in the department of psychology. However, psychology researchers with a PhD may also get hired in other departments, such as in human communications and philosophy. It would be interesting to examine if these other areas have similar, lower, or higher standards for academic positions than in psychology. As a preliminary exploration of this hypothesis, we gathered the CVs of new business professors with a psychology PhD. Because it is common for business schools to hire psychology graduates, we examined all the newly hired professors (i.e., after 2010) in business schools (from the same schools in the New Faculty database) that have graduated from a PhD program in psychology. We collected a total of 22 CVs and found no difference in the publication rates between the business professors and professors in social/personality

29 10 psychology 3 (Number of Publications: M = 10.09, SD = 4.24, 49.1% first authorship). Although we did not find differences between psychology professors and business professors from psychology PhDs, it is unclear if the differences will remain minimal in other fields where psychology professors are hired such as in mathematics and biology. This remains an open avenue for further investigation. Another novel finding from this study is that the self-report data from graduate students produced total publications and first authored paper estimates very similar to data derived from CVs. Previous studies that relied on self-reports are susceptible to self-report biases, where the number of publications may be slightly inflated. While this is unlikely because the number of publications is finite, and less susceptible to memory errors, a more serious concern is that there may be selection biases where only students who have published may be willing to participate in a study on publication performance. Our results suggest that there is minimal self-report bias in reporting the number of publications. Additionally, this finding is in line with a pilot study we conducted among psychology graduate students from the Greater Toronto Area (n = 123), where we found a near perfect correlation (r =.99, p <.001) between the self-reported number of publications and the CV-reported number. Although there was a significant mean difference when a paired t-test was conducted (diff =.14, 95% CI [.05,.22], t(122) = 3.28, p =.001), the difference was quite small, suggesting that the difference is likely not going to produce vastly different results. Thus publication information seems to be resilient to self-report biases, as is the case with self-reported GPA (Gramzow, Elliot, Asher, & McGregor, 2003). It is still possible that both studies suffer from the same selection bias, where students who have not published are a) less likely to participate in studies about publication performance and b) less likely to publicly post a copy of their CV. One potential method to address this limitation is to use Google Scholar to obtain publication information about graduate students who have not publicly made their CVs available. However, there are numerous challenges with this method, such as the deciphering of multiple researchers with the same names, and removing duplicate entries, which make the verification of citation counts an extremely time-consuming and 3 We selected social/personality psychology as our comparison group because business schools typically hire graduates from this particular area of psychology.

30 11 potentially error-ridden endeavour (Jacso, 2005; Jacso, 2008). Nevertheless, we were able to examine the publication numbers of an additional 37 professors for whom we did not obtain a full copy of their CV, but had a full publication list online, and found no significant difference in publication numbers (total number of publications: M = 13.14, SD = 9.72, 46.9% first authorship). This suggests that there is no evidence of a selection bias in our sample of new faculty members. However, whether this is also true for graduate students and post-doctoral fellows remains to be seen. Another potential method in obtaining publication information is to obtain informant reports from colleagues (e.g., their supervisors, fellow graduate students), although this too has its own advantages and disadvantages.

31 12 Chapter 3 Study 2 Study 2 Chapter 2 presented the number of publications along with the number of first-authored publications, but there are a number of issues with the number of publications as the sole indicator of scientific output. Namely, the number of publications do not take quality into account. It goes without saying that two publications in an obscure highly specialized lower-tier journal means something very different than two journal publications in Science. While there are indices that take into account the impact of a researcher s scientific output (e.g., h-index, g- index), these measures are not useful for newer researchers who typically have newer publications that have not accrued many citations. Thus, we created and assessed a potential new index of scientific output, called the zp-index, which avoids the use of citation counts, while also taking the quality of publications into account. Because citations take at least a few years to accrue, and many newer researchers have publications that are in press, we used the average citation count of articles within the journal (Impact Factor; IF) to calculate this index. Additionally, we took authorship position into account when calculating the zp-index, so that different positions of authorship will be given different weights. Typically, first authorship is given the most credit in a publication because it is believed that the first authors typically contribute to the largest proportion of the work in a publication (Wren et al., 2007). While last authorship is sometimes sought after (Cacioppo, 2008), this is typically not the case for early researchers, who are not yet well established. The zp-index is calculated, as follows: zp-index = n(in Xn) Where I = impact factor, X = weight given to the position of authorship. For each publication (n) each impact factor (In) is multiplied by a value dependent on authorship position (Xn), to take into account that contributions of each author systematically differ depending on the authorship position. While a number of researchers have suggested giving different weights to different role of authorship, all the proposed weights to date have not been empirically derived (Sekercioglu, 2008; Zhang, 2009), and so ultimately, were just estimated by the researchers themselves. It is

32 13 generally perceived that the first authors typically contribute to the greatest amount of work followed by the second author and so forth, but the exact proportions often depend on the area of study (Wren et al., 2007). However, to attribute more accurate weights to each position of authorship, we empirically derived the weights in the present chapter. Because authorship preferences can differ across different fields (APA Science Student Council, 2006; Tscharntke, Hochberg, Rand, Resh, & Krauss, 2007), and publication rates also differ across different fields (Bornman, Mutz, Neuhaus, & Daniel, 2008; Hirsch, 2005), we focused primarily on the field of psychology, which embodies characteristics of both Science, Technology, Engineering, and Mathematics (STEM) fields and the humanities. Psychology is also an extremely popular field of study that crosses different disciplines (Halonen, 2011), so a particular focus in this area can be of particular interest to many researchers. We distributed a short survey to ask researchers about their perceptions of the value of authorship and the typical work distribution across different positions of authorship. 3.1 Method We ed faculty members, graduate students, and post-doctoral students involved in psychology research in post-secondary institutions across Canada and the United States. We followed a list of PhD research programs in psychology from the National Academy of Sciences (NAS) (Lorden, Kuh, & Voytuk, 2011), along with the list of Canadian institutions listed on the Canadian Psychological Association (CPA; researchers (287 graduate students, 298 faculty members, 41 post-doctoral members) completed the survey. We included a number of surveys that are detailed below. Researchers were asked to indicate how important they think position of authorship is for a peerreviewed journal article on a scale from 1 (not at all important) to 7 (extremely important). They were also asked to indicate which position of authorship is typically sought after (first authorship, second authorship, third authorship, last authorship, other). Additionally, researchers were asked to indicate the typical work distribution of different authorship positions by distributing 100 points across first, second, and third (et al.) authorships. To shorten the survey and simplify the calculation of the zp-index, we divided the position of authorship into three groups (first, second, and third et al.,). Because sometimes last authorship can be sought after among established researchers (Cacioppo, 2008; Tscharntke et al., 2007), we asked respondents

33 14 to distribute the points from the perspective of new researchers (e.g., graduate students, postdoctorate students, newly hired professors). 3.2 Results Authorship Value 73% of psychology researchers reported that they believe authorship position in a peer-reviewed journal is either very important or extremely important. The importance of authorship position did not differ by the type of school membership (i.e., graduate student, faculty member, postdoc) and by the sub-discipline of psychology. 87% of researchers reported that the firstauthorship publications are most sought after in psychology. The preferred position of authorship slightly differed based on type of school membership (F(2, 623) = 10.51, p <.001), where graduate students generally preferred a first-authorship publication (94.4%), but the proportions of researchers with a first-author preference were slightly lower for faculty members (80.9%) and post-doctoral members (82.9%). The preferred authorship position also differed by the specific sub-disciplines of psychology (F(8, 614) = 4.80, p <.001), with the biggest discrepancy observed among biological psychologists, where just 66.1% of researchers indicated that first authorship was the most sought after. 25.0% of biological psychologists had a preference for the last-authorship position. The preferences of distribution of points were more similar across the other sub-disciplines of psychology Work Distribution Averaging across all researchers, the perceived amounts of work contribution for each authorship position were as follows: 62.14% (SD = 14.22) for first authorship, 22.57% (SD = 9.09) for second authorship, and 15.29% (SD = 9.73%) for third authorship and beyond (Figure 6). It is important to note that because we did not specify whether the third et al. referred to just the third author, or to all the remaining authors, it is possible that the contribution of the third authorship may be inflated. However, because the percentage of perceived contribution is fairly low when compared to the first and second author, this distinction will likely make little difference to researchers. Mirroring the differences in perception of the importance of authorship position across academic positions, the work distributions across authorship positions also slightly differed by academic

34 15 position [Table 1]. Percentage of work by first authors significantly differed by academic position (F(2, 623) = 4.79, p =.009). When controlling for multiple comparisons using a Bonferroni correction, post-doctoral fellows assigned a greater percentage to first authors compared to the percentage assigned by faculty members (difference = 6.98, 95% CI [1.32, 12.63], p =.009). Percentage of work by third authors were also significantly different by academic position (F(2, 623) = 5.46, p =.004), where faculty members tended to assign a greater percentage to third authors compared to the percentage assigned by post-doctoral fellows (difference = 4.59, 95% CI [0.72, 8.45], p =.014). While these differences were statistically significant, the differences were generally minor (< 10%), and thus we used the mean work distribution across all three groups for the subsequent analyses. 3.3 Discussion Interestingly, the pattern of work distribution closely resembles a geometric series with a constant ratio of ½. If the work distribution was assumed to be geometric, and the first authorship was given a value of 1, the second authorship would be given a value of ½ (1*1/2), and the third authorship would be given a value of ¼ (1/2*1/2). This point distribution in this model would be as follows: 57.14% (1/(1+1/2+1/4)) for first authorship, 28.57% (1/2/(1+1/2+1/4)) for second authorship, and 14.29% (1/4/(1+1/2+1/4)) for third authorship. Thus, because the work distributions across different positions of authorship closely mirrors that of a geometric function with a constant of ½, the zp-index for psychology researchers might be most simply formulated as: zp-index = n(in (½) (An 1) ) Where for n published journal articles, I = impact factor, A = position of authorship. Thus, if a new professor has a total of 3 publications, with 1 first-author paper in Psychological Science (IF = 4.43), and 1 first-author paper and 1 third-author paper in Journal of Personality and Social Psychology (IF = 4.88), then the professor s zp-index would be (4.43 x x x (1/2) (3-1) ). It is interesting to note that the perceived work distribution across authors also differed depending on the type of school membership. Faculty members perceived first authors to contribute to a lower proportion of the overall amount of work than perceived by graduate

35 16 students (difference = 5.12, 95% CI [.49, 9.76]) and post-doctoral members (difference = 6.98, 95% CI [2.35, 11.6]). On the other hand, faculty members attributed more credit to authors in the third-authorship et al position than graduate students (difference = 1.85, 95% CI [.28, 3.42]), and post-docs (difference = 4.59, 95% CI [1.43, 7.75]). However, all members on average attributed more than 60% of the work to first-authors and less than 17% of the work to third-authors et al, so the general patterns were not vastly different between graduate students, post-docs, and faculty members. Therefore, we maintained the geometric sequence of work distribution for all members of psychology.

36 Chapter 4 Study 3 Study 3 Chapter 3 empirically derived the perceived typical work distributions of each authorship position in psychology research. We were able to determine that the work distribution closely follows a geometric series, which helped us finalize the zp-index to incorporate the proper weights to each position of authorship. Study 2 was designed to construct an appropriate metric of scientific output for new researchers (i.e., zp-index). To demonstrate the utility of the zpindex, we gathered publication data of newly appointed faculty members in psychology. Additionally, because the impact factor is sometimes criticized by researchers (Seglen, 1997), we calculated an alternate version of the zp-index using another journal quality metric (i.e., SCImago Journal Rank) to demonstrate the flexibility of the zp-index. 4.1 Method Data collection for this study was conducted over the month of March We examined all psychology professors hired from 2010 to 2013/2014 (some were hired a year in advance) in research-oriented universities from the United States and Canada. We examined the top 100 psychology programs in the United States (i.e., 92 American universities), according to the National Academy of Sciences (NAS) rankings from 2011, and also included an additional 22 Canadian universities that offer a PhD program in psychology, for a total of 114 schools. The list of 22 Canadian universities were obtained from the Social Psychology Network ( All included professors were hired into positions that included a research component, and were tenure-track (with the exception of certain included Ivy League schools that do not offer tenure-track assistant professorships). Thus, the current sample does not include lecturers, adjunct, or visiting professors. The new professors curriculum vitae (CVs) were extracted from their personal website, their lab website, or academic social networking sites. If a full copy of the CV was not available online, the professor was directly contacted and asked to provide one. A total of 121 professors were ed to request a copy of their CV, and 51 (42.1%) professors sent a copy of their CV. See Figure 7 for a flowchart on 17

37 18 how the CVs were selected and obtained. The final sample consisted of 201 associate or assistant professors. To ensure accuracy, there were two independent groups of coders (N=15) that coded each CV. An additional spreadsheet was created to compile the impact factors (IFs) of journals that have published the professors work. Each coder was thoroughly trained on how each section of the CV should be coded. All the coders met with the principal investigator once every two to three weeks to ensure that the CVs were being coded reliably. The coding spreadsheets were visible on Google Drive, so that the principal investigator was able to oversee the coding process on a daily basis. Once the two groups of coders completed the coding of all the CVs, we subtracted one spreadsheet from the other, and jointly corrected any discrepancies. Two senior coders with over 20 hours of coding experience conducted the overlapping procedure. If there were any discrepancies that could not be resolved by the coders, they contacted the principal investigator, who made the final decisions. Each CV, on average, took two hours to code, and an additional hour to match and correct for any discrepancies. Each coder extracted the following information from each CV, identifying those accomplishments that occurred prior to the date of hiring. We recorded the number of peer-reviewed journal articles published (this number also included those published during the year of hiring itself, as most of those papers were likely in press or advanced preparation prior to hiring). We also recorded the publication position, for each paper (as first, second, or third author and beyond (e.g., fourth author, fifth author). Conference proceedings and abstract publications were omitted because there has been some concerns that they can be abused to pad the CV without merit (Gookins, 2012; Rohawani, 2013). We obtained the most recent 2-year ISI Impact Factor (IF) for each published or in press journal article on each professor s CV. If the IF was not available on the official website, we used the SCImago Journal & Country Rank (SJR; database to obtain the 2011 IF. If the IF could not be found on SJR, we used Publish or Perish (PoP; to compute the 2012 IF based on the Google Scholar database. In some instances, the IF could not be found on PoP, due to the low number of published articles (i.e., less than 10) from the journal (e.g., Current Trends in Neurology), or the journal was too new for the calculation of the IF (e.g., Clinical Psychological Science). In these cases, we coded the publication as a publication in a

38 19 journal with an unknown IF. Such publications (which made up 2.58% of the total number of publications) were omitted from the analysis. Because the limitations of the IF would also extend to the limitations of zp-index, we also assessed another metric of journal prestige. The SCImago Journal Rank (SJR) is an alternative metric of journal quality that addresses some of the limitations of the IF, namely that the IF does not take the quality of citations into account while the SJR attributes more weight to citations from important journals (Gonzalez-Pereira et al., 2010). The SJR also omits journal self-citations in its calculations, which often inflate the IF. We obtained the journal s SJR from the SCImago journal rank database. Like the IF, there were very few publications that did not have an SJR (5.91% of total publications). We used the SJR rank to calculate the adjusted zp-index (zpindexadj) by replacing the IF. Because publication rates differ by the different sub-disciplines of psychology, we categorized each professor into one of the following areas: social/personality psychology (e.g., relationships, intergroup relations), cognitive psychology (e.g., memory, perception), developmental psychology (e.g., aging), biological psychology (e.g., genetics, animal research), quantitative psychology (e.g., quantitative methods, statistics), clinical psychology (e.g., mental disorders), health psychology, and industrial/organizational psychology (e.g., marketing, management). We categorized the professor based on either a) the institution s categorization or description of the professor, b) the professor s biography on his or her webpage, or c) his or her usual venue for publication. If the professor could be categorized into multiple categories, we selected the category in which the professor had published more often. For instance, if a social neuroscientist published more often in the Journal of Personality and Social Psychology than in the Journal of Cognitive Neuroscience, then we categorized the individual as a social/personality psychologist. With the derivation of the zp-indices for new faculty members in psychology, we also examined how the zp-indices differs in its predictive validity from the simple number of publications. To examine how the new index predicts future scientific output, we gathered the most recent (March, 2015) H-index of the new professors from Google Scholar. Because citation counts take a number of years to accrue, we restricted our analyses to professors who were hired in 2010 and 2011 (n = 93), which would give professors in these cohorts about 4 to 5 years to accumulate

39 20 citations. We conducted a Spearman s rank correlation between the zp-indices, the number of publications, and the h-index. We also created additional indices to examine how each performs in comparison to the zpindices and the number of publications. First, we summed all the IFs with equal weights to create a zp-index that does not take authorship position into account (zp-indexna). We also repeated the calculation using SJRs (zp-indexadjna). Second, we calculated the zp-index assuming that all the journal publications have an impact factor of 1, to create a zp-index that does not take journal quality into account (zp-indexnq). Third, because first authorship is the most valued in psychology research, we summed only the IFs for first-authored publications and excluded non first-authorship publications (zp-indexfirst). The same procedure was conducted using SJRs (zpindexadjfirst). We examined the rank correlation between these additional indices along with the other measures. 4.2 Results Out of 201 new professors, the largest proportion of new hires took place in the cognitive psychology sub-discipline (32.7%), followed by social/personality psychology (19.6%) and then clinical psychology (14.0%). The average number of publications among newly hired professors in psychology was (SD = 7.28, skewness =.79, kurtosis =.41), and their average zp-index was (SD = 32.94, skewness =.76, kurtosis =.49). As expected, the number of publications, zp-index, and the zp-indexadj varied significantly with subject area (F(7, 193) = 7.56, p <.001, F(7, 193) = 8.67, p <.001, and F(7, 193) =5.36, p <.001 respectively). Specifically, researchers in biological psychology and clinical psychology had more publications and higher zp-indices than other subdisciplines of psychology. Table 2 presents detailed information about publication rates across the different subdisciplines of psychology. All measures of productivity were highly correlated with one another (r >.6 for all comparisons). The zp-index and the zp-indexadj were highly correlated (r =.93), which suggests that the SJR, an alternative journal quality metric to the IF, provides virtually the same information. Correlations between the indices remained very high even when taking authorship position into account, which suggests that researchers who publish a lot of first-authored publications also tend to publish a lot of publications in other authorship positions. Additionally, all the indices were good indicators of a researcher s future h-index (r >.5 for all indices), and there was no

40 21 evidence that one index had superior predictive validity than another. Table 3 displays the correlation matrix between all the calculated measures of scientific output along with their utility in predicting a researcher s future h-index. While there is a strong correlation between quantity and quality of research (Abramo et al., 2010), Laband (1985) demonstrated that the rankings of the top economic schools can differ depending on the metric of research output. Therefore, we examined whether rankings of professors can differ substantially depending on the metric used. Because there are fielddependent factors that can impact publication rates (Anauati et al., 2014; Bornmann & Daniel, 2008), we restricted our rankings to professors in cognitive psychology because of the highest number of professors in this subdiscipline. Because there is a strong correlation between the different variety of metrics of scientific output, we focused on the comparison between the number of publications (i.e., most commonly used metric) and the original zp-index. Table 4 provides the detailed rankings along with each professor s score. Looking at the rankings based on the number of publications against the rankings based on the zp-index, while 6 out of the 10 professors were in both rankings, their specific rankings tend to have shifted, and in some instances, have shifted dramatically. Additionally, another problem becomes apparent when using the number of publications because it is not a continuous measure like the zp-index, there are many more ties when ranking is calculated in that manner. In fact, because of the high number of ties, there are 17 researchers in the top 10 ranking based on number of publications compared to exactly 10 researchers in the top 10 ranking based on the zp-index because there were no ties. In Chapter 2, we illustrated some evidence that perhaps highly ranked schools have a higher bar set for the number of publications. However, another hiring strategy is to hire individuals based on the quality of publications rather than quantity. Because we have zp-scores for each professor in the dataset, we calculated an average impact score 4 for each faculty member by dividing their zp-score with the total number of publications. Interestingly, the average impact score was 4 It is not the average Impact Factor because the IF was divided by authorship position to compute the zp-score.

41 22 marginally correlated with the school rank (r(168) =.13, p =.09), suggesting that job search committees in higher-ranked schools may value quality over quantity. For future research, it would be interesting to examine which aspect of publications, quality or quantity, is more important in predicting hiring decisions. To test this question, a match unemployed comparison group (i.e., post-doctoral fellows, senior students interested in pursuing academia) is required. 4.3 Discussion Currently, there is no widely accepted measure of research productivity among the scientific community that does not utilize citation counts. This is problematic for newer researchers (e.g., graduate students, post-docs, new faculty members) who mostly have newer publications that have not yet received any citations. The present study introduces a new metric of scientific output, the zp-index, that does not utilize citation counts in its calculation, while also incorporating information about the quality of the publication along with the authorship position. We demonstrated how the zp-index can be used among new psychology professors. The zpindex was highly correlated with other indices of scientific output, so there was no empirical evidence that the zp-index should be favored over the simple total count of a researcher s publications. This is expected given that the correlation between quality and quantity of publications is very high (Abramo et al., 2010). However, when comparing the ranking of the professors based on the number of publications and the zp-index, the rankings do substantially differ depending on the metric used. Additionally, the comparison of rankings demonstrates that the number of publications may do an insufficient job in discriminating researchers scientific output, because there are too many researchers with equal number of publications. It might be noted here that because the zp-index s sensitivity can increase differentiation when it is used to rank scientists, what appear to be significant differences in ranking may arise as a consequence of smaller and potentially more random factors. The consideration of other markers of academic success, such as research funding, teaching ability and general administrative and societal contributions should help guard against potential over-weighting of ranking differences. While we did not find that the zp-index has higher predictive validity than the simple sum of citations, the zp-index clearly provides a more comprehensive assessment of research productivity, as well as one that is more in keeping with the implicit and explicit practices and

42 23 attitudes of the scientific community. First, as shown in Study 1, most researchers agree that authorship position is very important, and that the first authorship is typically sought after in psychology research. Thus, an index that takes authorship into account should be favored over one that does not consider authorship position. Second, the journal quality also matters to researchers. Many journal outlets publish some citation metric (e.g., impact factor) on their website to promote researchers to sublet to their journal. Additionally, researchers tend to perceive research from lower impact journals to be of lower quality (Saha, 2003), and there is even some evidence to validate this perception (Bordons, Fernández, & Gómez, 2002). Therefore, the zp-index would be generally favored over a simple sum of the number of publications because it takes the quality of the journal into account in its calculation. It is also important to note that the widely accepted h-index has not consistently been found to have superior predictive validity over other measures of scientific output (Bornmann et al., 2008; Mazloumian, 2012; Rezek et al., 2011). Rather, the h-index is a popular measure because the index incorporates aspects of research productivity (i.e., the number of publications and the number of citations) that the scientific community values. The creation of the zp-index follows a similar line of reasoning, where the assessment of a researcher s productivity incorporates aspects of a scientist s publication list (i.e., number of publications, quality of publications, authorship position) that matter to researchers and thus would be a more comprehensive index over the simple count of citations. While the number of publications seems to be a sufficient measure of a researcher s scientific output, it is not a favored measure among members in the scientific community because it is perceived to be overly simplistic. Therefore, the creation of a more widely accepted measure of research productivity that can be applied to newer researchers may still be necessary. Additionally, the sample from Study 2 was composed of all newly hired psychology professors, many of whom have published prolifically in their respective fields. However, if we were to compare graduate student s performance, for example, where the number of publications tends to be much fewer, then the choice of metric might make a substantial difference. The impact factor of each publication can make a more substantial difference if a researcher has fewer publications because their overall zp-index would be based on a few publications. Conversely, if a researcher has 30 publications, then the impact factor of 1 of their publications would make little difference to their overall zp-index, unless the impact factor of that one publication is very high.

43 24 Furthermore, as was observed in Study 2, when we select just a small subsample of researchers to compare with one another, the rankings differed substantially depending on the measure of scientific output (e.g., number of publications vs. zp-index). Thus, the use of the zp-index may be particularly useful in conjunction with other scientometrics in election procedures for scientific awards and jobs from a pool of candidates, as it can provide a more comprehensive assessment of a researcher s scientific merit than just the number of publications alone.

44 25 Chapter 5 Study 4 Study 4 Past studies on predictors of academic and job performance shed light into potential personality variables that influence graduate school performance. However, no study to date has demonstrated a direct link between personality and graduate school success. Thus, the present study collected data from graduate students in psychology in Canada and the United States to examine which personality factors predict graduate school productivity. The present study focused the number of publications, because such output is one of the most valuable metrics of graduate school performance (Darley et al., 2004), the focus on. Additionally, because graduate students work closely with one or two supervisors, we also examined whether the student s perception of their supervisor s personality might influence their publication success. Another advantage of the present work is that we assessed personality from the aspect level rather than the domain level typically examined in personality studies (DeYoung et al., 2007). Aspects act as an intermediary factor between the big five personality domains and their facets, with each domain splitting into related but still importantly distinct aspects. The agreeableness factor, for instance, is composed of two aspects: compassion and politeness. While no significant association has been found between agreeableness and political beliefs, aspect compassion is linked to liberalism while aspect politeness is linked to conservativism (Hirsch, DeYoung, Xu, & Peterson, 2010). Similarly, the Openness factor is composed of the aspects Openness and Intellect the former which is related to creativity but not fluid intelligence, and the latter is related to fluid intelligence but not creativity (Nusbaum & Silvia, 2011). Additional work in this area has also demonstrated that the Openness aspect is independently associated with verbal intelligence, while the Intellect aspect is equally related to verbal and nonverbal intelligence (DeYoung, Quilty, Peterson, & Gray, 2014). In the field of morality, the withdrawal aspect of neuroticism and the enthusiasm aspect of extraversion independently predicted greater likelihood of cooperation in a prisoner s dilemma while no significant association was found with the volatility aspect of neuroticism and the assertiveness aspect of extraversion (Hirsch & Peterson, 2009). Such work suggests that there may be certain instances where it would be useful to examine personality from the aspect level.

45 26 There is some evidence to suggest that an aspect level analysis would be fruitful when examining the personality predictors of publication success, particularly when assessing conscientiousness. Specifically, conscientiousness can be broken down into its two aspects: industriousness and orderliness. While conscientiousness has been shown to be a successful predictor of academic and job performance (e.g., Hurtz & Donovan, 2000; Noftle et al., 2007), a recent study has also demonstrated that perfectionism can have a negative impact on research productivity among psychology professors (Sherry et al., 2010). Additionally, perfectionism has also been linked to conscientiousness (e.g., Stoeber & Stoeber, 2009), which contrasts with conscientiousness positive relation with performance. However, perfectionism is more closely related to the orderliness aspect of conscientiousness than to industriousness (Roberts et al., 2005). Therefore, by splitting conscientiousness into its two aspects, the positive relation between industriousness and publication success may be strengthened, while the relation between orderliness and publication success may be nonsignificant or negative. These subtle but important details of personality may be missed if we were to use the traditional Big Five Inventory (John & Srivastava, 1999). Therefore, we expanded the traditional Big Five assessment of personality into its aspects for a more comprehensive examination of its link to publication success. 5.1 Method 1,283 graduate students from Canada and the United States participated in our study. There was a generally uniform distribution of the year of study (MA = 63, PhD 1 = 217, PhD2 = 201, PhD 3 = 202, PhD 4 = 208, PhD 5+ = 325), and students came from a wide variety of areas (Development = 112, Social/Personality = 239, Cognitive = 167, Brain & Behavior = 116, Clinical = 375, Quantitative = 28, Other = 184). We recruited participants from United States from the list of top 100 research-oriented psychology programs from the National Sciences Academy. We obtained lists of all the graduate students from each institution s official website. To boost participation, we individually ed graduate students participate in our study and tailored each to include their names. If the student agreed to participate, we gave them a link to a survey with the following measures: We asked students to self-report the number of peer-reviewed publications, including those in press. Based on a pilot study of 123 students conducted among psychology graduate students in the Greater Toronto Area, the correlation between self-reported number of publications and the

46 27 actual number of publications on their CV was.99 (p <.001). Although the paired samples t-test was significant, the inflation of self-reported publication numbers was minimal (diff =.14, 95% CI [.05,.22], t(122) = 3.28, p =.001), suggesting that self-reported number of publications are generally accurate. Students were also asked to fill out the Big Five Aspects Scale (BFAS, DeYoung et al., 2007) to assess their personality. Additionally, we also asked students to complete the BFAS on behalf of their professors. The informant version of the questionnaire changed the stem of the questionnaire from I to My supervisor. Because publication numbers differ by year of study and by area (Zou & Peterson, 2015), we asked participants to indicate their area of study within psychology, and their current year in the program. Once students completed the survey, they were given a chance to participate in a raffle with a prize pool of $500. They were then thanked and debriefed for participating in the study. 5.2 Results Publication numbers differ greatly depending on the year of study and the subdiscipline of psychology (see Chapter 2 and Chapter 4). Thus, these two variables were included in our models as control variables. We first examined the relation between personality and publication output at the level of the big five using partial correlations (omitting the effects of year and area). Extraversion and conscientiousness positively predicted the number of publications (r(1,035) =.06, p =.04 and r(1,035) =.10, p =.001, respectively), whereas neuroticism negatively predicted (r(1,035) = -.07, p =.02). Additionally, we also conducted a regression to control for the shared variance among the predictors. Because the number of publications is a count and is not normally distributed [Figure 8], we conducted a negative binomial regression to predict the number of publications from personality. In this model, extraversion and conscientiousness were once again positive predictors of the number of publications (IRR = 1.15, 95% CI [1.00, 1.33], p =.05 and IRR = 1.17, 95% CI [1.02, 1.35], p =.03, respectively). Interestingly, agreeableness arose as a negative predictor of the number of publications (IRR = 0.85, 95% CI [0.73, 0.99], p =.04). The negative effects of neuroticism became marginally significant (IRR =.91, 95% CI [0.81, 1.01], p =.09). We also examined for any potential moderation effects by year of study. We centered each continuous variable (BFAS scores, year of study) and included each interaction term one at a

47 28 time to reduce multicollinearity (Aiken, West, & Reno, 1991). Year of study did not moderate any of the effects of the big five personality factors on publication performance. We once again conducted partial correlations between the ten aspects and the number of publications, but this time, at the aspect level of analysis. Industriousness was significantly positively correlated with number of publications (r(1,035) =.15, p < 001), and assertiveness was marginally positively correlated (r(1,035) =.06, p =.06). Withdrawal was significantly negatively correlated with the number of publications (r(1,035) = -.10, p < 01), and openness was marginally negatively correlated with the number of publications (r(1,035) = -.06, p =.05). We also conducted a negative binomial regression to predict the number of publications from the personality aspects. Once again, we controlled for the year of study and the area of psychology. Industriousness was the only significant predictor of the number of publications, where industrious students published more (IRR = 1.29, 95% CI [1.10, 1.51], p <.001). The year of study moderated the effect of politeness on publication success, where a one-year increase in the year of study reduces the effect of politeness by 9.0% (Wald chi-square = 4.45, IRR =.91, 95% CI [.83,.99], p =.035). When plotting the interaction, politeness seems to have little influence on productivity among junior graduate students, but eventually becomes a detriment as students become more senior [Figure 9]. To examine the relation between student personality and perception of supervisor s personality, we first examined mean differences in personality scores [Figure 10]. All ten paired t-test comparisons were significant. Specifically, students perceived their supervisors to be less enthusiastic, compassionate, polite, orderly, withdrawn, volatile, and open than themselves. Conversely, students perceived their supervisors to be more assertive, industrious, and intellectual. Additionally, the zero-order correlations between student s self-ratings and their ratings of their supervisor s personality reveals that, with the exception of assertiveness, student s personality is positively correlated with their perception of their supervisor s personality (Table 5). Interestingly, student s self-report personality ratings systematically differed depending on year of study. Students became less polite (r(1,041) = -.06, p =.04), more volatile (r(1,041) =.07, p =.04), and more intellectual (r(1,041) =.07, p =.02) as they progressed through their graduate

48 29 program. Correlation between student ratings of supervisor personality and study year indicated that the latter were perceived as increasingly less enthusiastic (r(923) = -.11, p =.001), less agreeable (compassion: r(925) = -.11, p =.001; politeness: r(923) = -.12, p <.001), less conscientious (industriousness: r(924) = -.12, p <.001; orderliness: r(923) = -.08, p =.011), more neurotic (withdrawal: r(924) =.10, p =.002, volatility: r(923) =.09, p =.005), and less intellectual (r(923) = -.07, p =.03). We conducted a negative binomial regression examining the influence of student perception of supervisor personality on student publication. We first conducted the analysis by including only the supervisor s personality at the level of the big five in the regression model while controlling for area and year of study (Model 1; Table 6). Supervisor extraversion was the only variable that positively predicted the number of student publications. We then repeated the analysis predicting student s performance from supervisor s personality while controlling for the student s own ratings of their personality (Model 2; Table 6). This analysis can help control for rater biases, but at the same time, may also remove meaningful shared variance between the supervisor and the student. Nevertheless, supervisor extraversion was a significant predictor of the number of student publications. To examine the possibility that the year of study can moderate the relation between supervisor personality and student publications, we created interaction terms between each aspect and the year of study, and included them one at a time in the regression model after centering the continuous predictors. We did not include student personality scores as control variables, to free up degrees of freedom (as the previous analysis showed that its inclusion made a minimal difference). The effect of supervisor openness was moderated by student s year of study, where a one-year increase in the student s year of study increases the effect of openness by 12.3% (Wald chi-square = 5.37, IRR = 1.12, 95% CI [1.02, 1.24], p =.02). Plotting the interaction revealed that openness of the supervisor has a negative impact on early student performance, but had relatively little impact on later performance [Figure 11]. We repeated the analysis but at the aspects-level of personality. In the model with just the supervisor ratings, students with industrious supervisors and volatile supervisors published more, while those with assertive, polite, and withdrawn supervisors had fewer publications (Model 1; Table 7). When we included the student s own ratings into the model, all relations from the

49 30 previous analysis were retained, with the exception of volatility, which was reduced to marginal significance (Model 2; Table 7). Once again, moderation analysis by year of study showed that none of the supervisor BFAS variables were moderated by year of study, suggesting that the influential aspects of supervisor personality remain constant during graduate school. We also examined for the possibility that the difference in the perception of student and supervisor s personality might impact productivity. For each aspect, we saved standardized residuals from each supervisor s personality regressed on student s personality. These residuals were then absolute scored, then summed, to create an overall personality difference variable. Personality difference positively predicted the number of publications (Wald chi-square = 4.49, IRR = 1.03, 95% CI [1.00, 1.05], p =.03), but this effect became non-significant when year of study was included in the model. Year of study was positively related to the difference score (r(927) =.09, p =.007). 5.3 Discussion Given that there are large numbers of graduate students in the United States, it would be fruitful to examine which factors lead to their enhanced performance. Yet, little work has examined the factors that predict graduate school performance. The present study examined one potential factor, personality, in predicting publication success among graduate students enrolled in research-intensive psychology programs. We found that conscientiousness, with the industriousness aspect in particular, was consistently related to greater number of publications. This is in line with work demonstrating the benefits of conscientiousness on job (Barrick & Mount, 1991) and academic performance (Noftle et al., 2007). Our results also suggest that there may be a negative effect of neuroticism, and in particular, the withdrawal aspect may potentially hamper productivity. This finding also mirrors the handful of studies that have demonstrated the negative impact of neuroticism on work and academic performance (e.g., Chamorro-Premuzic & Furnham, 2003; Schneider, 2004; Tett, Jackson, & Rothstein, 1991). Interestingly, we also found that the year of study moderated the effect of politeness on the number of publications. Specifically, politeness does not seem to have an impact on publications at the junior level, but as students become more senior, politeness seems to have a detrimental effect on productivity. Because politeness is negatively correlated with power (Baxter, 2006), it is possible that politeness at the senior level indicates a lack of academic independence which

50 31 is assumed to be an integral aspect of academic success. Students likely receive less supervision as they progress through graduate school to develop independence from their supervisors to eventually run a lab of their own. However, it may be the polite students who face difficulties with this transition, and fail to gain the independence that is needed for a successful academic career. With that said, because this is the first study that has examined the personality predictors of publication success, these effects should be replicated. In addition to examining the influence of student s personality on the number of publications, we also examined how supervisor s personality may also impact student productivity. Based on our data, at the level of the big five, extroverted supervisors were beneficial in improving student performance. This is consistent with work that have demonstrated the benefits of having an extraverted supervisor increases job satisfaction, commitment, and reduce turnover intentions (e.g., Smith & Canger, 2004). Interestingly, openness of the supervisor was moderated by students year of study in graduate school. Specifically, supervisor openness does not have much of an effect on the productivity of senior graduate students, but seems to be a detriment for junior graduate students. One possible explanation is that while openness can be useful for stimulating new and creative ideas, the actual process of publishing requires tight adherence to a series of concrete steps. The open professor perhaps can be characterized as a supervisor who stimulate students with lots of new ideas and suggest many directions for future research, which may backfire for junior-level students who require more guidance in the actual process of writing a manuscript and submitting it for publication. Senior-level students who have gained more experience in the publishing process are likely better able to utilize their supervisor s ideas as general suggestions and channel their focus. Future research on effective research supervision can benefit from open-ended studies, which ask graduate students to describe the positive and negative aspects of their supervisor. With rich qualitative data, researchers can perhaps better illustrate what effective supervision really entails. At the aspects level, there was consistent evidence demonstrating that industrious and volatile supervisors can increase productivity among students, while polite, assertive, and withdrawn supervisors hamper student productivity. While we did not expect to find these relations, it is possible that the perception of these personality dimensions may be a marker of some aspect of

51 32 the student-supervisor relationship. For instance, as mentioned previously, politeness is negatively related to power (Baxter, 2006), and thus a polite supervisor may indicate a lack of authority, which can hamper worker productivity (Marshall, 1999). Conversely, volatility is positively related to power (Fast & Chen, 2009), which suggests that if students perceive their supervisors to be volatile, then they likely also perceive their supervisors to be an authority figure. Nevertheless, this is the first study to demonstrate a relation between supervisor personality and student productivity, so these results should be replicated to ensure the reliability of our findings. We also examined the relation between student self-ratings of personality and their ratings of supervisor personality. Student s personality was generally positively correlated with supervisor s personality, which suggests that students perceive themselves to be like their supervisors. This may indicate genuine similarity, but may also indicate a reporting bias that influences both the self-report and the informant report. Thus, it would be interesting to gather self-reports of supervisor personality to examine how the results might be influenced by shared method variance. While there was a positive correlation between student s self-ratings and their ratings of their supervisor, there were also systematic differences in their mean scores. Student s perception of their supervisors can be summarized as a respectable authoritative figure someone who is intelligence and competent, yet shows little emotion. However, due to the lack of the supervisor s own ratings of their personality, it is unclear whether this perception is accurate or the result of some systematic bias. We also did not find that a greater mismatch between a student and their supervisor s personality hampered student productivity. In fact, the results suggest that it may increase student productivity. However, this relation became nonsignificant when controlling for year of study. Because year of study was positively related to the observed difference, it is possible that the difference score may reveal greater accuracy in assessing supervisor personality. Early graduate students who are less familiar with their supervisors may be more likely to project their own personality onto their perceptions of their supervisor s personality. However, as time passes and students become better acquainted with their supervisor, they are better able to differentiate themselves from their supervisors. As previous researchers have demonstrated, self-informant

52 33 agreement of personality does improve with acquaintanceship length (e.g., Connolly, Kavvanagh, & Visweswaran, 2007) and people do use ratings of their own personality as a substantial source when making judgments of others, particularly when evaluating strangers (e.g., Beer & Watson, 2008). Therefore, our difference measure may essentially be a proxy measure of relationship closeness, where higher differentiation reflects a closer relationship. On the other hand, it is possible that students and their supervisor s personality changes as students progress through their graduate program. While personality change has been a hotly contested topic, there are some instances where personality change can be observed in a relatively short period of time, when there are strong environmental pressures to do so. For instance, medical students have repeatedly been demonstrated to decline in empathy as they complete medical school, presumably as a means of self-protection from getting too close to their patients (Hojat et al., 2004). In the current study, graduate students became less polite and more volatile, which may indicate a development of academic independence. Additionally, students also became more intelligent, presumably thanks to the wealth of knowledge accumulated through graduate school. However, because the study was cross-sectional, it is possible that the personality change may illustrate a cohort effect though cohort effects tend to be quite small (Trzesniewski & Donnelan, 2010) and the present study assesses a small span of just about 5 years. Furthermore, it is also unlikely for supervisor s true personality to change as their student matures, as they are often responsible for supervising multiple students throughout their career. An investigation of personality change using a longitudinal design can shed light on some of these unanswered questions. It is important to note that we did not obtain supervisor s self-reported personality. Because we only received informant reports of supervisor s personalities, we were not able to assess the accuracy of these reports. While there is some convergent validity between self- and informant reports of personality (e.g., Vazire, 2006), there are also systematic differences in informant reports that may reduce its validity (e.g., Leising et al., 2010). Regardless of the absence of the supervisor s self-reports, it is nevertheless still interesting to observe that student s perception of their supervisor can have an impact on the student s performance. It would be interesting for future studies to examine whether it is the perception of their supervisor s personality or the supervisor s actual personality that matters for the student s success. Perhaps it is possible that it

53 34 is only necessary for students to perceive their professor as being impolite to boost their productivity, rather than to reduce the actual politeness of supervisors. When we examined the predictors of student personality on publication success, it was the industriousness aspect of conscientiousness that was driving the effect, while orderliness was not a significant predictor of publication success. This would suggest that previous studies that have examined the influence of conscientiousness on performance may be somewhat attenuated by the inclusion of orderliness in its assessment of conscientiousness. The conscientiousnessperformance relationship is likely going to be strengthened if only the industriousness aspect was assessed. Thus, previous studies that have examined the link between personality and performance may be providing a more conservative estimate of its effect when assessed from the level of the big five. For example, in our study, the partial correlation between conscientiousness (i.e., average score of industriousness and orderliness) and publication numbers was.10 after controlling for the student s year of study and area. In comparison, the partial correlation between industriousness and publication output was.15, a 1.25 times increase ( /.10 2 ) in effect size. Hence, our study demonstrates that the effect of conscientiousness and performance may be greatly attenuated due to the inclusion of orderliness in the assessment of conscientiousness. However, it is also important to note that when we included all the aspects into a regression model, some of the factor-level relations disappeared. For instance, at the big five factor level, student extraversion was related to their performance. However, this effect seemingly disappeared when extraversion was split into its two aspects: assertiveness and enthusiasm. This may be due to the removal of the meaningful shared variance between the aspects. While the two aspects of a factor are somewhat independent, they also share variance that represent their comprised factor (DeYoung et al., 2007). Once this meaningful shared variance is removed in the regression model, it is possible that neither of the two aspects uniquely predict the outcome variable despite being related at the factor level. Therefore, we encourage researchers to examine personality at the five-factor level and the aspects-level in conjunction to avoid misinterpreting their data. It is also possible that the shared variance between the aspects represents systematic error variance. For instance, people are generally motivated to view themselves positively (e.g.,

54 35 Brown, 1986), and thus they may inflate their ratings on the two aspects, such as the politeness and compassion aspects of agreeableness. This inflation would in turn create shared error variance between the two aspects of agreeableness, which can be removed by removing the overall shared variance with regression analyses. Therefore, it is possible that the regression analyses reduces systematic error variance among the personality aspects. However, based on the current dataset, we are unable to tell to what extent is the shared variance between the aspects valid variance or systematic error variance. To tackle this question, another independent method of personality needs to be obtained (e.g., informant reports) to separate the shared method variance from the valid variance (Campbell & Fiske, 1959). This would be an interesting and important avenue for future personality research. Furthermore, when examining the impact of supervisor s personality on their student success, we found opposing relations of two aspects within one factor: neuroticism. Specifically, volatile supervisors had a positive impact on student publications, while withdrawn supervisors negatively impacted their productivity. If these two aspects were assessed as one neuroticism factor, it would be a nonsignificant predictor of publication success masking the effect of its individual aspects. Therefore, we encourage future researchers to utilize the BFAS when conducting personality research. While the scale is twice the length of the traditional BFI (John & Srivastava, 1991) at 100 items, the aspects level analysis can reveal relations that may be attenuated or masked at the factor level. Additionally, items in the BFAS can be averaged to produce the traditional Big Five scores, so that both levels of analyses can be conducted simultaneously.

55 36 General Discussion Chapter 6 General Discussion 6.1 Theoretical Contributions and Applications The present body of work provides a number of theoretical contributions to the field of scientometrics, educational psychology, personality psychology, and applied psychology. In Chapter 2, we provided a series of publication statistics of graduate students, post-doctoral fellows, and newly-hired faculty members in psychology. Although this topic has been of great interest to psychology researchers, no study to date has systematically collected this information. These statistics in Chapter 2 provide the most up-to-date and accurate numbers from a large pool. Additionally, the numbers were obtained from both self-reports and CVs though results suggested that there were minimal differences in the two methods. With these numbers, we hope that graduate students, supervisors, and administrators can tailor student experience in graduate programs to maximally increase job market competitiveness. In Chapter 3 and 4, we created a new scientometric index, the zp-index, which could be of substantive use to newer researchers. Previous indices of scientific output have mostly relied on citation counts. This sdisadvantage newer researchers, who have not had time to accumulate citations. Additionally, popular indices, such as the h-index (Hirsch, 2005), ignore authorship position, despite authorship position being an important aspect of publications. The zp-index uses the impact factor rather than the citation count to provide an estimate of the publication s impact. In Chapter 3, we empirically derived weights attributed to each authorship position so that more credit is given as a researcher goes up in authorship rank. In Chapter 4, we applied the zp-index among new faculty members, to demonstrate its utility. Although the zp-index did not predict the H-index better than mere number of publications, it does a better job of discriminating the scientific output of researchers. Furthermore, this lack of relationship could easily be attributed to the shortcomings of the H-index itself. Chapters 2 to 4 focused on quantification of graduate school performance. However, predicting such performance is also important. Research supervisors can use this information to identify potential graduate students who are most likely to succeed in graduate school. Undergraduate

56 37 students who are interested in pursuing graduate school can use this information to assess whether they are suited for academia. In consequence, Chapter 5 focused on personality as a potential variable that may influence graduate school success, namely, publication success. Results suggest that industrious students are more likely to publish. Additionally, supervisor s personality also had an impact on student success, where volatile and industrious supervisors boosted productivity, while polite, withdrawn, and assertive supervisors had the reverse effect. This suggests that the success of students is not only influenced by their own character traits, but also by their supervisors. As many researchers have already alluded to, a successful graduate school performance requires the joint effort of both the students and their supervisors. 6.2 Limitations and Future Directions There are a number of limitations that should be addressed for future research. First, we focused heavily on publication statistics in the present paper and omitted other valuable scientific contributions such as books, scholarships, conference presentations, and teaching credentials. Although we attempted to quantify these variables, one major concern was that it was difficult to distinguish whether the researcher purposely omitted a specific credential from their CV because of its absence, or because of the researcher s perception of its insignificance. For instance, a handful of CVs did not mention teaching credentials, which could suggest that these individuals did not have any teaching experience or that they omitted such information from their CV to highlight their research experience. Because publications make up the typical core of a CV, we were confident that a CV with no publications listed truly meant that the individual has not published. Another concern was that many CVs did not clearly distinguish between the different accomplishments. For instance, many CVs failed to make a distinction between conference presentations and conference posters, even though the former are much more difficult to obtain than the latter (Suldo, 2013). Thus, before we can accurately quantify the academic achievements other than publications, a standard CV format would need to be implemented. Another possibility is to distribute a comprehensive survey assessing these attributes for individuals to provide self-reports although such a survey would be somewhat tedious for researchers to complete alongside their everyday duties.

57 38 Our outcome measure across the studies also only included journal publications, while omitting other peer-reviewed publications from other sources (e.g., books, magazine articles). While these publications can and do make impactful contributions to the scientific community, it is difficult to make an overall assessment of the scientific merit of these sources as the circumstances surrounding their publications can greatly differ. For instance, many of these publications tend to be invited submissions, so researchers are not necessarily given an equal opportunity to publish in these mediums. However, it is important to acknowledge that non-journal publications can make useful contributions to the scientific community and should not be overlooked. Lastly, publication statistics differ by time and by area. Therefore, while the statistics presented in Chapter 2 will be applicable to current students for a number of years, future researchers should rerun the study once the data is out-dated. It is uncertain exactly how much time will pass before these statistics become obsolete, but it is an inevitable prospect of this type of work. In a similar vein, although our focus on the current study was on graduate students in psychology, a similar study would be extremely fruitful for other fields of studies whose primary focus is on publications. We encourage researchers in other disciplines to follow our procedures to provide their students with a benchmark for how they should progress in graduate school and beyond. The zp-index, per se, has a number of limitations that still should be considered and discussed. First, any limitations that apply to the impact factor (IF) or SCImago Journal Rank (SJR) will also apply to the zp-index (see Lawrence, 2007 & Seglen, 1997 for a more detailed discussion). However, in the absence of any information about the citation count, the IF and SJR appear to be the best indicators of the potential future influence of a paper. The zp-index adds additional flexibility, in that if another metric of journal quality becomes favoured, then the IF or the SJR scores can simply be replaced with the preferred metric. Second, the zp-index only assesses research publication. This is also a limitation of all other metrics of research productivity. Nonetheless, this remains a concern for newer researchers, who typically do not have a large number of publications. Other aspects of scientific achievement, such as conference posters, conference publications, and even teaching, may also need to be considered when examining a newer researcher s scientific output. It is difficult to incorporate other aspects of scientific achievement into the existing measures of research productivity because their scores are derived from citation counts. One potential advantage of the zp-index is

58 39 that these other domains of achievement can also be incorporated into the zp-index by examining how each accomplishment might be weighted through empirical comparison, for example, to a first-authorship publication in a journal with an impact factor of 1. Third, empirical validation of the zp-index was based on findings from psychology researchers. Future studies should examine how the relevant weights may differ in other fields. For instance, in some fields, particularly those in STEM, last authorship may be given more weight (Venkatraman, 2010). This is often also the case among psychology researchers. Fourth, like all other scientific productivity measures, the zp-index cannot be used without due caution to make comparisons across disciplines or even within different fields of one discipline because impact factors, publication rates, and citation counts are affected by field-dependent factors (Anauati et al., 2014; Bornmann & Daniel, 2008). Fifth, our results indicated that the inclusion of authorship information in the zp-index had little effect on the overall ranking of scientists, which might suggest that the specific weighting system used in the formula is not all that vital to its utility. However, it is important to note that our database was composed of professors who were hired in high-ranking institutions, who typically come from the more successful end of the spectrum. For graduate students who typically have just a few publications, inclusion of the authorship weighting system will often make a large difference to their zp-index. It would be interesting to examine how the zp-index functions among those who are at the very early stages of their careers, such as graduate students. Sixth, and finally, our outcome measure (i.e., h-index) used to examine the predictive validity of the zp-index is less than ideal, because the h-index itself is an imperfect indicator of a researcher s productivity. It can be argued that the zp-index may be a better indicator of a researcher s scientific output than the h-index, which would suggest that the ideal outcome measure we would want to accurately predict is the zp-index, rather than the h-index. H-indices can, for example, be inflated as a consequence of trendy publishing or controversial and not necessarily accurate results. Such factors lead to citation numbers that are poor indicators of genuine, long-term scientific merit. It is also possible that the zp-index might prove useful in providing additional unique predictive power when used in conjunction with other indicators of scientific output (i.e., h-index). Any evidence of unique predictive power would suggest that the zp-index may not only be useful for new researchers, but also for those who are more

59 40 established, providing additional information about scientific output that may be poorly captured by current research productivity metrics. Future studies should examine how the zp-index behaves in conjunction with the current measures of scientific indicators among established researchers in predicting long-term outcomes (e.g., attaining a research grant, receiving awards, producing successful graduate students).

60 Mean Number of Publications 41 Figure 1. Graduate Student s Self-Reported Number of Publications across Year of Study (n = 1,283) Total Publications First Authorship MA PhD 1 PhD 2 PhD 3 PhD 4 PhD 5+ Year of Study

61 Number of Publications 42 Figure 2. Graduate Student s CV-reported Number of Publications across Year of Study (n = 744) Total Publications First Authorship MA PhD1 PhD2 PhD3 PhD4 PhD5 Year of Study

62 Mean Number of Publications 43 Figure 3. Average Number of Publications among Post-Doctoral Fellows across Years of Post- Doctoral Fellowship (n = 142) Total Publications First Authorship & 2 3 & 4 5+ Year in Post-Doctoral Program

63 Number of Publications 44 Figure 4. Average Number of Publications among Faculty Members Hired from (n = 201) Total Publications First Authorship Year Hired

64 Number of Publications 45 Figure 5. Comparison of Publication Numbers between Faculty, Post-Docs and Senior Graduate Students Total Publications First Authorship Senior Grad Students (PhD 4+) Post-Doctoral Fellow New Faculty (Without Post- Doctorate) New Faculty (With Post-Doctorate) Academic Position

65 46 Figure 6. Perceived work distribution across authorship position among psychology researchers (n = 626) Third et al. 15% Second 23% First 62%

66 Figure 7. Flow chart describing the selection process of new faculty members 47

67 Figure 8. Frequency Distribution of the Self-Reported Number of Publications among Graduate Students (n = 1,167) 48