Further Validation of Survey-Effort Measures of Conscientiousness: Results from a Sample of High School Students

Transcription

1 Further Validation of Survey-Effort Measures of Conscientiousness: Results from a Sample of High School Students Gema Zamarro* University of Arkansas Malachi Nichols University of Arkansas Angela Duckworth University of Pennsylvania Sidney D Mello University of Notre Dame February 2017 [VERY PRELIMINARY DRAFT. PLEASE, DO NOT CIRCULATE OR CITE WITHOUT PERMISSION] Acknowledgements We would like to thank Julie Trivitt for her help in the early stages of this paper and Albert Cheng and Collin Hitt for their comments and feedback on our results. * Corresponding author. University of Arkansas. Address: 219B Graduate Education Building. College of Education and Health Professions Fayetteville, AR, USA Phone: gzamarro@uark.edu.

2 Abstract Researchers struggle to find reliable measures of relevant character skills, such as conscientiousness, grit or self-control, to use as part of program evaluations. For this aim, researchers mostly rely on data from self-reported psychometric scales. Although validated and relatively easy to collect, these self-reported measures, when used for evaluation purposes, can be affected by social desirability bias, reference group bias, and other threats to validity (Dobbie & Fryer, 2015; Krosnick, Narayan, & Smith, 1996; West et al., 2016). An alternative approach to self-reported measures of character skills is to obtain measures based on performance tasks. Though performance-task measures do not always suffer the same sources of biases as selfreports, they have limitations of their own (Duckworth and Yeager, 2015). Our paper focuses on validating parametrizations of survey effort measures as possible proxies for relevant character skills. Our work builds on previous research that has suggested that survey-effort measures could proxy for conscientiousness related character skills in a population of high school students as well as of adults (Markus and Schütz, 2005; Hedengren and Stratmann, 2012; Hitt, Trivitt and Cheng, 2016; Hitt, 2015; Huang et al., 2012; Johnson, 2005; Meade & Craig, 2012; Zamarro et al., 2016). Our contribution to previous research is twofold. Firstly, we are able to complement the work of Hitt, Trivitt and Cheng (2016) and Hitt (2015) on the validity, among adolescent students, of survey effort measures by studying the correlation of these measures with external reports on these character skills provided by teachers and with measures of student performance at the end of high school and college attendance. Secondly, we are able to also study the relationship between survey-effort measures and other direct performance task measures designed to capture diligence and tolerance of frustration through carefully designed tasks that high school seniors in our sample were asked to perform. For this aim, we study longitudinal data on high school seniors collected in Our results show promise for survey-effort measures to be used as proxy measures of character skills related to grit and self-control, when self-reports and teacher reports might be biased, as long as students are not aware that survey effort is being studied. Keywords: character skills, grit, conscientiousness, behavioral tasks, survey effort JEL codes: C83, C91

3 1. Introduction Though researchers are now aware of the potential importance of character skills, such as conscientiousness, grit or self-control, in determining relevant life outcomes like education, labor and health outcomes or criminal behavior (see, Almlund et al., 2011; ter Weel, 2008; Heckman, Stixrud, & Urzua, 2006), they struggle to find reliable measures of these skills for evaluation purposes. As a result, despite growing recognition of the importance of character skills, evaluations of education interventions rarely include measures of these skills. Ignoring the potential effects of education policies or interventions on character skills is not a trivial issue as it may lead to an incomplete appraisal of their benefits and costs. For example, evaluations of early childhood education programs, charter schools, and private school voucher programs have often found little to no impacts on student achievement and other indicators of cognitive ability but large gains in educational attainment, employment, income, and health outcomes as well as reductions in criminal behavior (Sass et al., 2016; Campbell et al., 2014; Chingos & Peterson, 2012; Cowen et al., 2013; Heckman et al., 2010; Heckman et al, 2013; Wolf et al., 2013). Researchers speculate that these differences in longer-run outcomes, despite no noticeable differences in achievement, are due to these programs potentially affecting their participants character skills. Evaluations of these programs based only on impacts on student s cognitive ability would have understated their benefits. In addition, character skills might mediate potential effects of such interventions, and a better understanding of their potential effect on relevant outcomes could help guide effective policy design. Despite the salience of character skills, it is difficult to measure them with validity. Researchers mostly rely on data from self-reported psychometric scales, where study participants are asked to answer a series of Likert-type items. Although validated by researchers and relatively easy to collect, these self-reported measures can be affected by important sources of bias like social desirability bias, reference group bias, and other threats to validity, when used for evaluation purposes (Dobbie & Fryer, 2014; Krosnick, Narayan, & Smith, 1996; West et al., 2016). Researchers have also recommended the use of teacher s or school personnel s ratings of students character skills or behaviors (Heckman et al., 2013). Despite promising, these ratings could potentially suffer from the same biases as self-reports if used for evaluation purposes. 1

4 Academic behavior measures through student grades, grade progression, absences and suspensions have also been used successfully to capture student s character skills (see, e.g. Jackson, 2013; Heckman et al., 2011). However, these measures are not always available for research purposes and are also not free of the risk of manipulation if used for evaluations. An alternative approach to the previously described measures of character skills is to obtain measures based on performance tasks. In a performance task, a researcher asks respondents to complete a carefully-designed task and interprets their behavior as some indication of an underlying character skill. Though performance-task measures do not always suffer the same sources of biases as previously described measures, they have limitations of their own. For one, tasks are generally very costly and difficult to collect in large samples. More importantly, it is not always clear that artificial tasks completed in a lab setting are generalizable to other contexts. Nor is it clear that behavioral tasks capture the non-cognitive skills that it purports to capture (Bardsley, 2008; Duckworth & Yeager, 2015; Falk & Heckman, 2009; Levitt & List, 2007). Finally, existing performance tasks are difficult to implement multiple times, as participants might show learning effects after having performed the task once. These limitations have generated calls to improve the measurement of character skills (Duckworth and Yeager, 2015). Our work is motivated by this call. Our paper focuses on validating parametrizations of survey effort measures as possible proxies for relevant character skills related to conscientiousness. We argue that questionnaires can be seen as a task and that by studying the effort that students put forward on surveys we can recover relevant character skills. These measures are not affected by the same types of biases that could affect self-reports and teachers reports and, as long as students and teachers are unaware that their survey effort is being measured, these measures would be less susceptible to direct manipulation. In this paper, we study longitudinal data on high school seniors collected in Our dataset contains a rich set of information on different measures of student s character skills including self-reported measures, teacher s reports, academic outcomes and direct performance task measures through two different tasks. This data is part of a long term project where seniors will be followed for six years after high school graduation, tracking their college enrollment status. At the moment, however, we have only access to the first wave of data and so, our paper can only study college enrollment during the first year after the baseline collection of data. 2

5 Our work follows previous research that has suggested that survey-effort measures could proxy for conscientiousness related character skills in a population of high school students as well as of adults (Markus and Schütz, 2005; Hedengren and Stratmann, 2012; Hitt, Trivitt and Cheng, 2016; Hitt, 2015; Huang et al., 2012; Johnson, 2005; Meade & Craig, 2012; Zamarro et al., 2016). However, our contribution to previous research is twofold. Firstly, we are able to complement the work of Hitt, Trivitt and Cheng (2016) and Hitt (2015) on the validity of survey effort measures among adolescents by studying the correlation of these measures with student s self-reported measures of different character skills and academic outcomes at the end of high school as well as college attendance one year after graduation but, more importantly, with external reports on these character skills provided by teachers. This is an important contribution as previous validation exercises have lacked external sources of information about student s character skills. Secondly, we are able to also study the relationship between survey-effort measures and other direct performance task measures designed to capture diligence and tolerance for frustration through carefully designed tasks that high school seniors in our sample were asked to perform. Our results show promise of survey-effort measures to be used as proxy measures of character skills related to grit and self-control when self-reports and teacher reports might be biased, as long as students are not aware that survey effort is being studied. The rest of this paper goes as follows. In section 2, we discuss the relevant literature including available prior research that has validated alternative measures of character skills related to conscientiousness and discuss the limitations of different measurement approaches. Next, in section 3, we describe the data and specific measures that are part of our validation exercise. In section 4, we describe the methods that we use to validate survey effort measures of character skills in our data. In section 5, we present the results of our analysis. Finally, in section 6, we discuss these results and offer concluding remarks about these measures of character skills and their place in future research. 2. Alternative Measures of Character Skills In the search to discover what factors produce desirable life time outcomes, researchers, practitioners, and policy makers have begun to highlight the importance of character skills such as conscientiousness, grit, diligence and self-control. In the area of academics, these types of 3

6 character skills have been found to be predictive of outcomes such as college persistence and degree attainment (Carneiro et al., 2007; Heckman, Strixrud, and Urzua, 2006). Within the labor market, these character traits and their facets including a person s social skills and work habits, have found to predict higher wages and increases in job productivity (Heckman et al., 2006; Lleras, 2008; Borghans et al., 2008). Lastly, these factors are also predictive of social behavior, decreases in teenage pregnancy, social behavior, and other positive health outcomes (Heckman et al., 2006; Carneiro et al., 2007; Moffitt et al., 2011). Out of the plethora of relevant noncognitive skills, conscientiousness and conscientiousness related skills such as grit or self-control have been the most studied and discovered to be a powerful predictors of outcomes including attainment, student achievement, college grades, job performance, and longevity in health (Lleras, 2008; Bowman & Matthews, 1960; Blair & Razza, 2007; Nyhus & Pons, 2005; Savelyev, 2010; and Almlund et al., 2011). Despite this research showing the relevance of conscientiousness and related character skills, researchers continue to struggle to find reliable measures of these skills for the purpose of evaluating the potential effects of education policies or interventions. Self-reported measures of conscientiousness, grit and related character skills Researcher s source of non-cognitive data mainly comes through self-reported psychometric scales where study participants answer a series of Likert-type questions. Because of their ease of use and relatively low cost, these scales are widely used. Scales such as the 12- item and the 8-item Grit scales are commonly used in determining a participant s pursuit of longterm goals (Duckworth et al., 2007). The grit scale has been shown to be correlated with conscientiousness and found to be associated with educational attainment in adults, predictive of GPA in university students, predictive of completing summer training for West Point cadets, and round attainment for Scripps National Spelling Bee participants, thus proving itself as a valid measure in multiple samples (Duckworth et al., 2007; Duckworth and Quinn, 2009). Additionally, the13-item Self-control scale, which measures the ability to change intrinsic motivation and the subsequent capability to restrain from undesirable external behavior, has been found to be predictive of GPA and lower probabilities of alcohol abuse within a sample of college students (Tangney, Baumeister, and Boone, 2004). Similarly, the Locus of Control scale, measuring an individual s perception of outcomes being within their control or out of their 4

7 control, was found predictive of high school graduation and college enrollment within a sample of eighth-graders (Rotter, 1966; Coleman and DeLeire, 2003). Despite their ease of use and predictability, these self-reported measures can be affected by social desirability bias, reference group bias, and other threats to their validity (Dobbie and Fryer, 2014; West et al., 2016; Knechtel et. al., 2015). In particular, self-reports of character skills only provide reliable comparative measures if two conditions are met. First, respondents must judge themselves on the same internal performance standard when reporting their level of skill. This is unlikely to be the case. For example, students who are more conscientious tend to judge themselves more harshly than do less conscientious students (Naumann & John, 2011). Moreover, educational interventions aimed to improve character skills in a group of students may also alter the individual standards students in that group use to judge their level of skill an issue commonly called reference group bias. For instance, in an evaluation of Boston charter schools, West et al. (2016) found that these schools seemed to have a negative effect on selfreported character skills, despite the fact that the schools appeared to have significant positive effects on test scores. This is a surprising result considering these schools focus on promoting character skills along with academic achievement. The authors hypothesize that these paradoxical results are driven by reference group bias; i.e., the standard for judging the character skills of those attending the charter schools was changed by the intervention. Similarly, after looking at the effects of enrollment in KIPP charter schools, Knechtel and team (2015) found no effects on student reported grit, academic confidence, or self-control despite positive effects on academic achievement. These results are unexpected because KIPP charter schools approach to learning includes a focus on character development. This finding presented the possibility that even though kids are making academic gains they are comparing themselves to the high standards set by their teachers or their hardworking peers, thus rating themselves lower in various measures than what they truly are. Lastly, Dobbie and Fryer (2014) found positive impacts of the Harlem Children s Zone on risky behavior (i.e. drug use and teen pregnancy) and academic performance on math and English but negative effects for self-reported measures of self-esteem and grit. Following suit, these findings are paradoxical because Harlem Children s Zone makes a concerted effort to change the culture of achievement, stressing the importance of hard work, grit, and persistence in achieving success (Dobbie and Fryer 2014, pg.6). The reduction in risky behavior assumes a positive change in character but the bias within the self- 5

8 reported measures inhibits such a finding. This highlights the necessity to improve the measurement of character skills to be able to capitalize on their promise as tools to inform educational practice and policy. A second condition for having valid self-reported measures of character skills is that respondents provide honest and reliable answers. We know this is not always the case. Some respondents expend low effort on surveys - they may provide thoughtless answers or skip questions entirely. The problem this creates for character skills research is that effort on surveys is likely related to the very skills that researchers are attempting to measure. For example, respondents who lack grit or self-control are unlikely to report that they lack those skills. This indicates that measurement error on surveys is related to the underlying skills we seek to measure, which then leads to invalid research findings. In addition to self-reports, external reports of participant behaviors and character skills have also been used as an alternative form of measurement. Several papers show the promise of using these types of measures, especially if reports came from those with close relationships to the subjects of the study. Using participants from the Kelly/Connolly Longitudinal Study on Personality and Aging, Jackson et al., (2015) look at the relationship between reports of participant s personality (i.e. extraversion, agreeableness, conscientiousness, emotional stability, and openness) made by their friends in the 1930 s and observed mortality rates in Male participants whose friends rated them as conscientious and open, and female participants whose friends reported them as emotionally stable and agreeable, were both found to live longer. Supporting the argument that external reports are a potential alternative way to measure character skills. Because of their consistent interactions with students over a given period of time, teachers are in a fortunate position for providing external evaluations of their students characteristics. Earlier research have shown teachers to be capable of identifying defiant behavior and prosocial behavior in adolescent children (Weir, 1980). Heckman et al., (2013) found that teacher reports of student externalizing behavior (i.e. lying, stealing, aggression and overall disruptive behavior) between the ages of 7 and 9 were predictive of later life crime at age 27 and employment at age 40. Additionally, teacher reports of student s academic motivation (i.e. student interest and persistence) between the ages of 7 and 9 were found to be associated with academic achievement at ages 8 and 14. However, despite their relevance, these external reports also have limitations and 6

9 are not free from the potential biases due to reference group bias or to misinterpretation of behavior (Duckworth and Yeager, 2015). Yet we feel that the addition of teacher reports to student selfreports, is a key asset of the dataset we use in this paper to better validate the use of survey-effort measures as potential proxies for relevant non-cognitive skills. Measures based on student s behaviors Measures based on academic behaviors such as student s grades, absences, credits earned, disciplinary infractions have also been used to measure the effect of policies and interventions on student s character skills. Kautz and Zanoni (2014) looked at the effect of the One Goal Program in Chicago, which is a program designed to promote college enrollment and completion through the building of non-cognitive skills, finding that One Goal participants experienced a decrease in absences, subsequent decreases in arrest rates and an increase in college enrollment and persistence. Angrist, Pathak and Walter (2011) reviewed the effects of attending a Massachusetts charter school on student discipline and absences. Charter school attendees found an increase in the duration of suspensions but decreases in the amount of unexcused absences, signaling mixed results. But due to the fact that over half of the charters examined fell into the category of No Excuses the results could be caused by the low tolerance for misbehavior within these schools or stricter discipline polices and not actual changes in student behavior. In a similar vein, an important question concerning these academic behavioral measures is whether they predict desirable outcomes similarly to their psychometric scale counterparts. Benda (2005) tried to answer this question by studying the predictability of a self-reported selfcontrol scale and measures of self-control based on student s behaviors, which included if the student had skipped school, been faulted in a wreck, or threatened someone within the year of reporting. The author found that the behavioral measure was a stronger predictor of drug use and crime than the self-reported measure. Additionally, Jackson (2013) looked at the effects of teachers on student cognitive and non-cognitive factors. Non-cognitive skills, which were derived from a weighted average of absences, suspensions, course grades and on-time course progression were found to predict high school dropout rates, graduation, AP course taking, sitting for the SAT, and college attendance. Lastly, in a sample of adolescents, precarious risky behaviors before the age of 15 defined as fighting, trying marijuana, daily smoking, regular 7

10 drinking and teenage sexual activity were also found to predict employment, heaving drinking, obesity, and voting as an adult (Heckman, Humphries, Urzua, Veramendi, 2011). Representing a potential change in underlying character, these real-life behavioral measures have been shown to be significant predictors of desirable outcomes. However, measures based on student s behaviors are seldom available for research and may be vulnerable to manipulation by teachers and school administrators if used for evaluation purposes (Kautz, Heckman, Diris, T. Well, and Borghan, 2014; Jackson, 2013). Performance task measures of conscientiousness, grit and related character skills Alternatively, character skills could also be measured using performance tasks, where respondents are asked to perform a specific carefully designed task to provide meaningful differences in behaviors as indicative of their level of a given skill. A classic example is Mischel and Ebbesen s (1970) famous Marshmallow Test, a performance task designed to measure self-control 1. Another example, of relevance for this paper, is the Academic Diligence Task (ADT) developed and validated among students by Galla et al. (2014). The original ADT was designed to mirror a real-world choice that students face when completing homework: the choice to remain engaged in tedious, but important assignments, and/or browse the Internet to play video games. Tested in a convenience sample of high school seniors they found that the number of questions answered correctly (i.e. productivity) and the time spent on task to be weakly but significantly correlated with self-reported conscientiousness (ρr= 0.08 and 0.09) and grit (ρr= 0.16 and 0.17). Additionally, productivity and the time spent working were predictive of GPA, academic achievement, on-time high school graduation, and college enrollment. Despite the potential of performance tasks to provide reliable measures of character skills that could then be used for evaluation purposes these types of measures also have disadvantages. Tasks are generally very costly and difficult to collect in large samples. More importantly, it is not always clear that artificial tasks completed in a lab setting are generalizable to other contexts. Nor is it clear that performance tasks could capture the non-cognitive skills that it aims to capture 1 In the Marshmallow Test, young children were presented with the options of eating one marshmallow or waiting until the experimenter returned (about 15 minutes) to get two marshmallows instead. 8

11 (Bardsley, 2008; Duckworth & Yeager, 2015; Falk & Heckman, 2009; Levitt & List, 2007). Finally, existing performance tasks are difficult to implement multiple times, as participants might show learning effects after having performed the task once. These limitations have generated calls to improve measurement of character skills (Duckworth and Yeager, 2015). Answering that call, researchers have begun to study the possibility of using measures of survey effort as proxy measures for character skills related to conscientiousness (Markus and Schütz, 2005; Hitt, 2015; Huang et al., 2012; Johnson, 2005; Meade & Craig, 2012; Zamarro et al., 2016). The idea behind this approach is that surveys often take effort to complete and they resemble everyday paperwork and tasks that people have to complete in their everyday life. For students, in particular, surveys administered in schools could resemble schoolwork or homework. Therefore, by studying how much effort people put forward in surveys, we could obtain proxy measures of relevant character skills related to conscientiousness. Two parametrizations of survey effort measures have shown promise as relevant proxies, item non-response and careless answering. Item non-response is defined as the percentage of questions skipped by a respondent on a survey. In earlier work Marcus and Schütz (2005) find item non-response to be related to agreeableness and openness but fail to find any association with conscientiousness in an online survey for website owners. Likewise, Zamarro et al. (2016) failed to discover any association of item non-response and self-reported personality traits among a representative sample of American adults interviewed over the internet. However, these measures are found to work better in paper-pencil surveys and in particular in populations of students. For instance, taking advantage of longitudinal nationally representative samples of adolescents and adults from the US and Germany, respectively, Hedengren and Stratmann (2012) find item non-response to be correlated with self-reported conscientiousness and a strong predictor of earnings and mortality risks. Furthermore, Hitt, Trivitt and Cheng (2016) used six different longitudinal nationally representative samples of U.S. youth to determine the relationship between the percentage of questions skipped and desirable outcomes. They found that item non-response is a significant predictor of educational attainment independent of cognitive ability. Still more work on the validity of this measure is warranted as to date no validation has been provided looking at its relationship with teacher reports or with other performance task measures of these skills. 9

12 Contrary to respondents who intentionally skip questions, some respondents answer randomly, inattentively, and overall carelessly (Johnson, 2005; Meade & Craig, 2012). This form of survey effort is captured through parametrizations of careless answering. Using two national longitudinal surveys, Hitt (2015) discovered that careless answering in adolescent respondents was associated with lower years of final education and decreases in the probability of high school completion, independent of cognitive ability. Building on previous work Zamarro et al. (2016), using data of a nationally representative internet panel of American adults, found that repeated careless answering behavior was negatively correlated with self-reported grit and selfreported conscientiousness and positively correlated with neuroticism, shedding light on its construct validity. They also determined that careless answering is a significant negative predictor of years of education and lower levels of income in the household. Providing reason that careless answering could be a promising proxy for conscientiousness and conscientiousness related skills. Through recent research, survey effort measures have proven themselves as promising proxy measures for conscientiousness and conscientiousness related skills (Hitt, Trivitt and Cheng 2016; Hitt, 2015; Zamarro et al., 2016). However, in spite of their demonstrated construct and criterion validity, these measures have yet to be studied in relation to external teacher evaluations and in comparison with other performance task measures of these character skills. It is our hope to fill this gap and further validate these two measures of survey effort. 3. DATA The data used is part of a longitudinal study on college persistence led by a research team at the University of Pennsylvania. As part of this project, in the spring of 2014, the research team conducted a baseline data collection from high school seniors (n=513) attending a public highschool in the northeast of the US. These students were then followed one year to track their college enrollment status. The overall project will follow them for five more years. Our analysis, however, is based on the baseline survey and first followed up data, which was made available to us. That is, we can only study college enrollment during the first year after the baseline collection of data on these high school seniors. 10

13 3.1 Character Skills Measures in Our Data Self-Reported Measures Grit The grit scaled used is a five-item scale modeled from the eight-item Grit-S scale (Duckworth and Quinn, 2009). The scale was shortened from eight to five items to allow for brevity within the survey. The scale asked the students to rate how true five statements described themselves on a five point Likert-type scale (Not at all true; Slightly true; Somewhat true; Very true; Completely true). These statements included, for example, I finish whatever I begin and I stay committed to my goals. Each student s question scores were averaged to develop a grit score for each respondent. Grit scores then range from 0 to 5 with a high score representing high values of grit. The Cronbach alpha measure of reliability of this scale was 0.80, therefore the construct can be considered reliable (Cronbach, 1955). Locus of Control The locus of control scale used is a four-item scale modeled from the Students Perception of Control Questionnaire (SPOCQ). The SPOCQ was developed by Wellborn et al., (1989) and was found to be a valid scale that significantly predicted college enrollment (Turner et al., 2008). Two of the questions used were modified to follow an if-then format. The scale asked the students to indicate how much they agree with four statements using a five-point Likert-type scale (Strongly Disagree; Disagree; Mostly Disagree; Mostly Agree; Agree; Strongly Agree). The statements included, Getting good grades is a matter of luck and If you get bad grades, it s not your fault. Student s scores from these questions were averaged to develop a locus of control score for each respondent. Scores could range from 0 to 6 with a high score meaning the student has high levels of locus of control. The Cronbach alpha of this scale in our sample was 0.55, indicating that it was somewhat reliable. 11

14 Self-Control Combined The Self-Control Combined scale is a combination of four Likert-types questions pertaining to Work skills and four Likert-type questions covering Interpersonal skills. These eight questions were developed from the KIPP Character Growth Card 2. The scale asked students to rate how true the eight statements described themselves on a five point Likert-type scale (Not at all true; Slightly true; Somewhat true; Very true; Completely true). The statements for Work skills included statements like I come to class prepared, I get to work right away, instead of waiting until the last minute and the statements relating to Interpersonal skills included statements like I allow others to speak without interruption and I control my temper. Scores from each question were totaled then averaged to develop a self-control combined score for each respondent. Average scores were also computed separately to represent self-control in work and self-control related to interpersonal skills. Scores could range from 0 to 5 with a high score meaning the student has high levels of self-control. The Cronbach alpha for the self-control combined measure was 0.76 which indicates the scale was reliable. Teacher Reports Additionally, teachers also completed surveys reporting on student behavior. Multiple teachers rated each student on each item of the grit and self-control scales described above and answered additional questions about classroom behavior and work ethic. As several teachers reported on a single child, each teacher s score was averaged for each student to give that student a unique construct score. High scores represent higher levels of that character skill. To measure the student s classroom behavior and work ethic, teachers were also asked to report on the students redirection and homework completion. For redirection the teachers were asked to pick the number of times the student required redirection within the last week with options: 0 times; 1 time; 2 times; 3 times; 4 times; or 5 or more times). Teachers were instructed to remember that redirection includes reminding the student to stay on task or to obey classroom expectations. Teachers reported redetection times were averaged for each student to give that 2 %20Supporting%20Materials.pdf 12

15 student a redirection score. A high number of redirects could represent a lack of diligence or negative classroom behavior. Finally, teachers were also asked about homework completion. In particular, teachers were asked what percentage, from 0 to 100%, of assignments the student completed with a passing grade and on time. The teachers reported values were then averaged to give the respective student an average percentage value. A higher percentage infers that the student has high levels of work ethic Survey Effort Measures Item Non-response The first way we parametrize survey effort is through measures of survey item nonresponse, following Hitt et al., (2016). We build two measures with this information. The Item non-response measure, which captures continuous item non-response rates, is built by dividing the total number of questions a student leaves blank by the number of answerable questions to which a student should have responded, given legitimate skips. In addition, we also build a Dichotomous Item Non-response measure which is just a binary indicator taking the value of 1 if the student leaves any answerable question blank, dependent on legitimate skips. The decision to build this dummy indicator is driven by the observation that a large proportion of students, 47%, completed the entire survey. Figure 1 shows the distribution of survey item non-response rates in our sample. Careless Answering The second way we parametrize survey effort is through measures of careless answering, following Hitt, (2015). The construction of this measure is explained through the following example. Consider a reliable, validated scale with let s say 5 questions. Thus, if the scale is reliable, each question will consistently measure the same underlying construct. Individual responses to each questions then would be well predicted by the responses to other questions in this same reliable scale. For instance, one would expect that responses to question 5 would be well predicted by student s responses to questions 1 through 4. Thus, we interpret deviations in responses from predicted values, given responses in other questions in the scale, as measures of careless answering. 13

16 In practice, we first identify all reliable scales, with Cronbach alpha coefficients of 0.6 and above, that students had to answer. We excluded self-reported scales that were used to validate survey effort measures in this paper (i.e. grit, self-control, and locus of control). A total of 10 scales were identified. For each scale, in our second step, we regress responses from each item in a scale on the average score of the rest of items in the scale. Third, residuals from each of these regressions are obtained to capture the extent to which the response to a particular item is unpredictable, based upon the responses that the individual and others in the analytic sample provided for other items in the scale. Absolute values of each of these residuals are then standardized to account for any differences across the items within the same scale. These standardized residuals are then averaged within scales and standardized again to take into account differences across scales (e.g., different total number of items, answer options). Finally, a composite careless answering score is obtained by averaging these standardized averages residuals at the student level. Higher values of this measure mean higher levels of carelessness or unpredictability in responding. Figure 2 shows the distribution of careless answering in our sample. As the measure is standardized by construction, mean and standard deviation are not very meaningful. We observe, however, quite some variation in the level of carelessness in students in this sample Direct Performance Task Measures Diligence Task The Diligence Task or the Academic Diligence Task (ADT) is a computer generated task where students are given the option to perform simple math problems, which have been prompted to be beneficial for them, or play computer games. This task was designed to measure academic diligence (Galla et al., 2014). The measure of academic diligence that we use in our analysis is the percentage of task assigned time a student spent completing math problems instead of playing video games. A higher percentage is representative of higher levels of diligence. 14

17 Frustration Task We also have information of a tolerance of Frustration task students were asked to complete as part of this study. During this task, students were asked to trace a shape using the mouse on their computer. However, the task was such that actual actions with the mouse produced movements in the screen in the opposite direction. This required students to present high levels of concentration when performing the task and promoted frustration. If the student stopped tracing or traced off of the shape, the student was required to restart the task. Our measure derived from this task captures the percentage of the task assigned time a student spent tracing. A higher percentage is seen to represent higher levels of determination, concentration and diligence Outcome Measures, Cognitive Ability and Other Relevant Information Available for our Analysis Our outcome measures include high school senior GPA ranging from 0- low to 100- high, a dummy indicating if the student graduated high school, a dummy indicating if the student attempted to take the SAT, the mean SAT score ranging from 600- lowest to highest which is the sum of the SAT critical reading, SAT math, and SAT writing scores. Furthermore, we constructed three dummy variables indicating if the student was continuously enrolled in college for one year after high school, if the student was continuously enrolled in a 4- year college one year following high school, and if that student was continuously enrolled fulltime in a 4-year college one year after high school, respectively. Additionally, we also estimate models to explain performance in the Math and Reading Keystone exams which are end of the year assessments designed to evaluate proficiencies in Algebra I and Literature. Results in these exams are part of student s state s graduation requirements and scores range from 1200 to A very important control variable in our analysis is the measure of a student s cognitive ability as we would like to find that our measures of survey effort are predictive of other measures of character skills and outcome variables even after controlling for cognitive ability. For this purpose we have measures of student s performance in the Kaufman Brief Intelligence Test (KBIT). The KBIT is a three part test consisting of vocabulary, matrix reasoning, and 15

18 riddles but only the matrix reasoning section was available in our data (Kaufman and Kaufman, 2004). The KBIT is presented using scaled scores ranging from 40 to 132. Finally, our analysis will also include student s demographic information such as age, gender, ethnicity, English Language Learner (ELL) and Special Education Status (SPED), Free and Reduced Lunch (FRL) status, and a measure of household income as control variables. 4. Empirical Strategy for Validation of Measures For survey effort measures to be valid, proxy measures of conscientiousness and conscientiousness related skills should show construct validity and some degree of criterion validity. That is, they should show correlations with other measures of similar character skills (construct validity) and they should also be correlated with other student outcome variables that are known to be correlated with the same latent skills (criterion validity). Following the literature described in section 2 above, we study Spearman correlations and partial correlations, controlling for cognitive ability and demographic information, for our measures of survey effort (i.e. non-response rates and measures of careless answering) and selfreported measures of grit, locus of control and self-control. We expect to find negative correlations with these self-reports. Importantly, we also study correlations with teacher reported self-control, grit, redirection and homework completion. In this case, we expect to find negative correlations with teacher s reported self-control, grit and homework completion and positive correlations with redirection. As a benchmark, we compare these estimated correlations with those observed for the direct performance task measures through the diligence task and the frustration task. As we argue that questionnaires can be seen as a performance task, we expect to find similar patterns of correlations with survey effort measures than with these direct performance task measures, as long as they are measuring related underlying character skills. The last set of analyses looks at criterion validity of survey effort measures. To do so, we estimate linear regression models and linear probability models to predict each of the following academic outcomes: Senior year GPA, high school graduation, attempted to take the SAT, SAT scores, Keystone Math and Reading scores, college enrollment in the 1 st year, college enrollment in a 4 year college and college enrollment full time in a 4 year college. We estimate separate 16

19 models for each of the survey effort measures, i.e. non-response rates, dichotomous nonresponse, and careless answering measures, following this type of specification: Academic_Outcome i = β 0 + β 1 Survey_Effort i + β 2 Cognitive_Ability i + β 3 X i + ε i (1) Our models control for cognitive ability measures through the use of the KBIT scaled score. Xi represents a vector of student demographic controls including age, ethnicity, gender, ELL status, FRL status, SPED status, and parental income. We report standardized regression coefficients for all models. For comparison, we also estimated models including direct performance task measures of diligence and persistence as well as using teacher s reports of student s grit, self-control, redirection and homework completion. 5. Results 5.1 Descriptive Statistics Table 1 presents descriptive statistics for our sample of 513 high school seniors. The sample is equally split among male and female students. Most of the students in our sample are either African American, White, or Asian in proportions of 41%, 36% and 20%, respectively. Only 3% of the students in our sample were Hispanic. Students in our sample reside in homes with an average household income of about $52,000. Also slightly more than half of our sample qualifies for free and reduced lunch. Additionally, Table 1 holds summary statistics for our outcome variables. On a 0 to 100 scale, the students averaged an 85 on their high school GPA and 95% of the sample graduated high school. Only half of the sample, however, attempted to take the SAT but about 60% enrolled in college after graduation. Of these, 43% enrolled in a 4- year college and 40% did so full time. Table 2 shows descriptive statistics for the character skills measures used in this paper. Concerning measures of survey effort, on average, students left blank 2% of the questions for which they were asked to respond. This item non-response rate is similar to those found by Hitt, Cheng and Trivitt (2016) in multiple national representative samples of adolescents. 47% of students in our data, however, answered all questions in the survey. Our careless answering measure, comprising a standardized average of residuals capturing inconsistent responses, ranges 17

20 from -2.3 to 4.9. This indicates considerable variation in the degree of care that students put into completing the surveys with some being more careful than the average (negative values) and some being considerably less careful (positive values). Concerning direct performance task measures, students devoted 64 percent of assigned time in the diligence task to engage in solving math exercises while they devoted 54 percent of the time to trace in the frustration task. If we look at self-reports and teacher reported measures, the average self-reported grit of students in our sample was An average that goes in line with those found by Duckworth and Peterson (2007) in several samples of convenience. Similarly, the students in our sample scored about 4.6 in the locus of control scale and 3.6 in the self-control combined scale. Additionlly, teachers reported an average of 3.53 in the levels of grit of the students of our sample, 3.72 in the level of work related self-control and 4.21 in the level of interpersonal selfcontrol. Teachers reported that students in our sample needed re-direction about 1 time during the last week, on average. Finally, teachers reported that students in our sample completed, on average, about 78% of the assigned homework on time and with a passing grade. 5.2 Relationship among Character Skills Measures As described in section 4 above, we first present evidence of construct validity of survey effort measures. Table 3 presents Spearman correlations among our proposed survey effort measures and student self-reported and teacher reported measures of character skills related to conscientiousness. As expected, item non-response is negatively correlated with both selfreported grit and self-control, as well as teacher reported grit and self-control. The first 2 columns of Table 5, show partial correlations among these measures once we control for student s cognitive ability and demographic information. A similar pattern is observed in this case but correlations with teacher reports become smaller suggesting that part of the raw correlations might be driven by student demographics and cognitive ability. Although the magnitudes of the correlations between survey-effort and these self-reported measures appear small, they are at least as large as the correlations reported in prior literature validating other performance-task measures of conscientiousness (Duckworth & Kern, 2011; Galla et al., 2014). Going back to Table 3, we observe that careless answering present negative correlations with self-reported student self-control as well as with teacher reported grit and work self-control. 18

21 As expected, in this case, we also observe a positive correlation with teacher reported redirection. That is, those students whose teachers identify them as needing more re-direction are also those that appear to respond more carelessly to the survey. The second column of Table 5 presents similar patterns in terms of partial correlations, once we control for student s cognitive ability and demographic information. Again, correlations with teacher reports become smaller once we add these controls. For comparison, Table 4 presents similar correlations as presented in Table 3 but focuses on direct performance task measures through the diligence task and frustration task results. Our measures of survey effort seem to be weakly correlated with performance in both the diligence and frustration tasks. The correlations are in the expected direction, lower levels of effort in the survey are related with lower performance in these tasks. Correlations are found to be bigger for the case of careless answering measures than for measures of item non-response. Looking at correlations between the performance on these tasks and self-reports, the diligence task performance presents the highest correlation with measures of locus of control while the frustration task presents highest correlations with measures of self-reported grit and self-control. Performance in both of these tasks is correlated with teacher reported levels of grit and selfcontrol. Correlations with teacher reported re-direction go in the expected direction with higher levels of re-direction reported leading to lower performance in these tasks. Finally, the last two columns of Table 5 present results for partial correlations among these measures. The patterns observed mostly remain, after controlling for student cognitive ability and demographic information, although correlations between performance in these tasks and teacher reports also become smaller, after adding the additional demographic and cognitive ability controls, and are smaller than those found with survey effort measures. 4.3 Relationship of Character Skills Measures and Academic Outcomes Next we present evidence of criterion validity, by studying the predictive power of our proposed survey effort measures on high school and college academic outcomes. As a reference, for comparison, we also study the predictive power of direct performance task measures as well as teacher reports. Table 6 presents the results of linear regression models for student academic outcomes, following specification (1) described above, when different survey effort measures and performance task measures in the diligence and frustration task are included as explanatory 19

22 variables. Similarly, Table 7 presents the results of linear regression models when each of the teacher reports are included as explanatory variables instead. Regressions that use SAT scores as dependent variable are limited to only those students that attempted the SAT test. Sample sizes vary depending on the available information for each individual regression but range from 392 to 458 observations and from 216 to 240 for SAT score regressions. Focusing on the results for survey effort measures, presented in Table 6, we observe that item non-response and the dichotomous item non-response variable are significant predictors in the expected direction of senior year GPA scores, the probability of attempting the SAT, SAT scores if they took the test, Keystone Math and Reading performance and college enrollment. A standard deviation increase in item non-response lead to an almost 0.20 standard deviations decrease in senior year GPA, keeping student cognitive ability and demographic information constant. We also estimated models where both item non-response rates and a dummy for leaving any question blank are included together. The idea of this specification is to study if both behaviors are potential drivers of academic outcomes. We find this is generally the case. Finally, careless answering is found to predict senior year GPA, attempting the SAT, Keystone Math and Reading scores, and college attendance. Concerning direct performance tasks, we find that both performance in the diligence and frustration task significantly predicted senior year GPA and Keystone Math and Reading scores. Performance in the diligence task also significantly predicted SAT scores and college enrollment but only marginally. Finally, performance in the frustration task significantly predicted the probability of attempting the SAT. For comparison, Table 7 studies the predictive power of teacher reports of student s character skills. Teacher reports appear to be a better predictor of student academic outcomes than survey effort or performance tasks. We find that all types of teacher reports considered significantly predict senior year GPA, the probability of high school graduation, attempting the SAT, performance in the Keystone reading and math tests and college enrollment. Additionally, all measures significantly predict the probability of attending college. Finally, the only significant predictor of SAT scores, among those who took the test, is teacher reported interpersonal self-control. Effect sizes are also generally larger than those found for survey effort or performance tasks. It should be stressed, however, that as it was the case with self-reports, teacher reports are subject to similar biases and manipulation and so, they might not be reliable measures for evaluation purposes. Also, they are often not available in researcher s datasets. 20

23 However, when available they seem to be good measures of students character skills. Survey effort measures, on the other hand, still showed predictive power and concurrent validity and so, are potentially a good proxy measure of relevant character skills when other measures are not available or when we suspect they might be affected by manipulation or other sources of bias. 6. Conclusions This paper uses data from a longitudinal study of high school seniors, collected in 2014, to study the potential of survey effort measures to serve as proxy measures of character skills related to conscientiousness. Surveys often resemble everyday paperwork and tasks that people have to complete in their everyday life and in particular, for students, surveys administered in schools, resemble schoolwork or homework. Therefore, we hypothesize that by studying how much effort people put forward on surveys we can learn relevant information about their character skills and, in particular, about those skills related to conscientiousness. We study the potential of two parametrizations of survey effort measures that have shown promise in previous research: item non-response and careless answering measures. We contribute to previous research that aimed to validate these measures in two ways. Firstly, we are able to complement the work of Hitt, Trivitt and Cheng (2016) and Hitt (2015) on the validity, among adolescent students, of survey effort measures by studying the correlation of these measures with teacher reports of student s skills and with academic outcomes at the end of high school and college attendance. Secondly, we are also able to study the relationship between survey-effort measures and other direct performance task measures designed to capture diligence and tolerance of frustration through carefully designed tasks that high school seniors in our sample were asked to perform. Our results show promise of survey-effort measures to be used as proxy measures of character skills related to grit and self-control. Both item non-response and careless answering measures show convergence validity through negative correlations with self-reported and teacher report measures of grit and self-control. Careless answering was also found to be positively correlated with teacher s reports of students needing redirection, even after controlling for student demographics and cognitive ability. Item non-response also presented criterion validity 21

24 through negative significant correlations with senior year GPA, the probability of attempting the SAT, SAT scores, performance on the Keystone Math and Reading tests, and the probability of enrolling in college. Careless answering only showed significant correlations with these academic outcomes. Direct performance tasks of diligence and tolerance of frustration also presented convergence and criterion validity through positive correlations with self-reported and teacher reported measures of grit and self-control and academic outcomes at the end of high school and college enrollment. A negative correlation was also observed between the performance in these tasks and careless answering patterns in the survey. Finally, for comparison, we also studied the predictive power of teacher reported measures with respect to student high school academic outcomes and college enrollment. Teacher reports were found to be better predictors of these academic outcomes than survey-effort or direct task measures, even after controlling for student demographic and cognitive ability. However, one should remember that teacher reports might be affected by the same sources of bias as self-reports if used for evaluation purposes. Therefore, we believe that this paper adds evidence to the potential of survey effort measures to provide meaningful information about student s character skills. These measures provide researchers and evaluators a relatively easy source of measuring respondent s character skills related to grit and self-control that lack the relevant forms of bias found in self-reported or teacher reported measures, as respondents are usually unaware they are being monitored on their survey effort. In addition, they open the opportunity to future character skills research of already collected data, even if these data did not include direct measures of character skills (see, e.g. Cheng, Zamarro, and Orriens, 2016). 22

25 References Almlund, M., Duckworth, A. L., Heckman, J. J., & Kautz, T. D. (2011). Personality Psychology and Economics. Handbook of the Economic of Education. Vol. 4, Angrist, J. D., Pathak, P. A., & Walters, C. R. (2011). Explaining charter school effectiveness (Working Paper No ). Cambridge, MA: National Bureau of Economic Research. Bardsley, N. (2008) Dictator game giving: altruism or artifact? Experimental Economics, 11 (2), Benda, B. B. (2005). The robustness of self-control in relation to form of delinquency. Youth & Society. 36 (4), Blair, C. and Razza, R., (2007). "Relating Effortful Control, Executive Function, and False Belief Understanding to Emerging Math and Literacy Ability in Kindergarten." Child Development 78(2): Borghans, L., ter Weel, B., & Weinberg, B.A. (2008). Interpersonal styles and labor market outcomes. Journal of Human Resources, 43(4), Bowman, P., and Matthews, C., (1960). "Motivations of Youth for Leaving School." Chicago, IL: University of Chicago Development Project. Campbell, F., Conti, G., Heckman, J. J., Moon, S. H., Pinto, R., Pungello, E., & Pan, Y. (2014). Early childhood investments substantially boost adult health. Science, 343(6178), Carneiro, P., Crawford, C., Goodman, A., & London School of Economics & Political Science, C. E. (2007). The Impact of Early Cognitive and Non-Cognitive Skills on Later Outcomes. CEE DP 92. Centre For The Economics Of Education. Cheng, A., Zamarro, G., and Orriens, B., (2016). Personality as a Predictor of Unit Nonresponse in Panel Data: An analysis of an Internet Based Survey. CESR-Schaeffer Working Paper No Available at SSRN: or 23

26 Chingos, M., and Peterson, P., (2012). The Effects of School Vouchers on College Enrollment: Experimental Evidence from New York City. Brookings Institution. Retrieved from: Coleman, M., and DeLeire, T., (2003), An Economic Model of Locus of Control and the Human Capital Investment Decision, Journal of Human Resources, 38 (3), Cowen, J. M., Fleming, D. J., Witte, J. F., Wolf, P. J., & Kisida, B. (2013). School vouchers and student attainment: Evidence from a state-mandated study of Milwaukee s Parental Choice Program. Policy Studies Journal, 41(1), Cronbach, L. J., & Meehl, P. E. (1955). Construct validity in psychological tests. Psychological bulletin, 52(4), Dobbie, W., & Fryer, R. G., Jr. (2014). The medium- term impacts of high-achieving charter schools on non-test score outcomes (NBER Working Paper No ). Cambridge, MA: National Bureau of Economic Research. Duckworth, A. L., & Kern, M. L. (2011). A meta-analysis of the convergent validity of selfcontrol measures. Journal of Research in Personality, 45, Duckworth, A. L., Peterson, C., Matthews, M. D., & Kelly, D. R. (2007). Grit: perseverance and passion for long-term goals. Journal of personality and social psychology, 92(6), Duckworth, A., and Quinn, P. (2009). Development and validation of the Short Grit Scale (GRIT S). Journal of personality assessment, 91 (2), Duckworth, A., and Yeager, D., (2015). Measurement matters assessing personal qualities other than cognitive ability for educational purposes. Educational Researcher, 44 (4), Falk, A., & Heckman (2009). Lab experiments are a major source of knowledge in the social sciences. Science, 326(5952), Galla, B. M., Plummer, B. D., White, R. E., Meketon, D., D Mello, S. K., & Duckworth, A. L. (2014). The Academic Diligence Task (ADT): Assessing individual differences in effort 24

27 on tedious but important schoolwork. Contemporary Educational Psychology, 39(4), Heckman JJ, Humphries JE, Urzúa S, Veramendi G. (2011). The effects of educational choices on labor market, health, and social outcomes. Unpublished manuscript. University of Chicago, Department of Economics. Heckman, J., Malofeeva, L., Pinto, R., and Savelyev, P. (2013). Understanding the mechanisms through which an influential early childhood program boosted adult outcomes. American Economic Review, 103(6), Heckman, J., Moon, S. H., Pinto, R., Savelyev, P., & Yavitz, A. (2010). Analyzing social experiments as implemented: A reexamination of the evidence from the High Scope Perry Preschool Program. Quantitative economics, 1(1), Heckman, J. J., Stixrud, J., & Urzua, S. (2006). The Effects of Cognitive and Noncognitive Abilities on Labor Market Outcomes and Social Behavior. Journal of Labor Economics, 24(3), Hedengren, D., & Strattman, T. (2012). The dog that didn t bark: What item nonresponse shows about cognitive and noncognitive abil- ity. Unpublished Manuscript. Retrieved from Hitt, C (2015). Just Filling in the Bubbles: Using Careless Answer Patterns on Surveys as a Proxy Measure of Noncognitive Skills (EDRE working paper ). Fayetteville, AR: Department of Education Reform, University of Arkansas. Hitt, C., Trivitt, J., & Cheng, A. (2016). When you say nothing at all: The predictive power of student effort on surveys. Economics of Education Review, 52, Huang, J., Curran, P., Keeney, J., Poposki, E., and DeShon, R., (2012). Detecting and Deterring Insufficient Effort Responding to Surveys. Journal of Business and Psychology, 27(1), Jackson, C. K. (2013). Non-Cognitive Ability, Test Scores, and Teacher Quality: Evidence from 9th Grade Teachers in North Carolina. Working Paper 18624, NBER. 25

28 Jackson, J. J., Connolly, J. J., Garrison, M., Levine, M., & Connolly, S.L. (2015). Your friends know how long you will live: A 75 year study of peer-rated personality traits. Psychological Science, 26(3), Johnson, J. A. (2005). Ascertaining the validity of individual protocols from Web-based personality inventories. Journal of Research in Personality, 39, Kaufman, A. S., & Kaufman, N. L. (2004). Kaufman Brief Intelligence Test, Second Edition. Bloomington, MN: Pearson, Inc Kautz, T., Heckman, J., Diris, R., T. Well, B., and Borghan, L., (2014), Fostering and Measuring Skills: Improving Cognitive and Non-cognitive Skills to Promote Lifetime Success, OECD Education Working Papers, No. 110, OECD Publishing. Kautz, T. and W. Zanoni (2014). Measuring and fostering non-cognitive skills in adolescents: Evidence from Chicago public schools and the onegoal program. Unpublished manuscript, University of Chicago, Department of Economics. Knechtel, V., M. A. Anderson, A. Burnett, T. Coen, M. Sullivan, C. C. Tuttle, and P. Gleason. (2015). Understanding the Effect of KIPP as It Scales, Vol. 2: Leadership Practices at KIPP, Princeton, N.J.: Krosnick, J. A., Narayan, S., & Smith, W. R. (1996). Satisficing in surveys: Initial evidence. New Directions for Evaluation, 70, Levitt, S., & List, J. (2007). What do laboratory experiments measuring social preferences reveal about the real world? The Journal of Economic Perspectives, 21(2), Lleras, C. (2008). Do skills and behaviors in High School matter? The contribution of noncognitive factors in explaining differences in educational attainment and earnings. Social Science Research, 37(3), Marcus, B. and Schütz, A. (2005), Who are the people reluctant to participate in research? personality correlates of four different types of nonresponse as inferred from self- and Observer Ratings. Journal of Personality, 73, Meade, A., and Craig, S., (2012). Identifying careless responses in survey data. Psychological Methods, 17(3),

29 Mischel, W., & Ebbesen, E.B. (1970). Attention in delay of gratification. Journal of Personality and Social Psychology, 16(2), Moffitt, T. E., Arseneault, L., Belsky, D., Dickson, N., Hancox, R. J., Harrington, H. L., Caspi, A. (2011). A gradient of childhood self-control predicts health, wealth, and public safety. Proceedings of the National Academy of Sciences, 108(7), Naumann, L.P., and John, O. P. (2011). Toward a domain-specific approach to cultural differences: The influence of cultural values and reference-group standards on selfreported personality. Unpublished manuscript. Sonoma State University, Department of Psychology. Nyhus, E. and Pons, E. (2005). The effects of personality on earnings. Journal of Economic Psychology, 26(3), Rotter, J. B. (1966). Generalized expectancies for internal versus external control of reinforcement. Psychological Monographs, 80(1), Sass, T., Zimmer, R., Gill, B., & Booker, K. (2016). Charter high schools effect on longterm attainment and earnings. Journal of Policy Analysis and Management, 35(3), Savelyev, P., (2010). "Conscientiousness, Education, and Longevity of High-Ability Individuals." Unpublished manuscript. University of Chicago, Department of Economics. Tangney, J. P., Baumeister, R. F., & Boone, A. L. (2004). High self-control predicts good adjustment, less pathology, better grades, and interpersonal success. Journal of Personality, 72, Turner, L. A., Pickering, S., Burke, J. R. (2008). The relationship of attributional beliefs to selfesteem. Adolescence, 33 (130), Weel, B. ter. (2008). The noncognitive determinants of labor market and behavioral outcomes: Introduction to the symposium. Journal of Human Resources, 43(4), Weir K., Stevenson J., and Graham P. (1980) Behaviroal deviance and teacher ratings of prosocial behavior: Preliminary findings. Journal of the American Academy of Child Psychiatry, 19 (1),

30 Wellborn, J.G., Connell, J.P., and Skinner, E. (1989). Students Perception of Control Questionnaire. University of Rochester. West, M. R., Kraft, M. K., Finn, A. S., Martin, R., Duckworth, A. L., Gabrieli, C. F. O., & Gabrieli, J. D. E. (2016). Promise and paradox: Measuring students non-cognitive skills and the impact of schooling. Educational Evaluation and Policy Analysis, 38(1), Wolf, P. J., Kisida, B., Gutmann, B., Puma, M., Eissa, N., & Rizzo, L. (2013). School vouchers and student outcomes: Experimental evidence from Washington, DC. Journal of Policy Analysis and Management, 32 (2), Zamarro, G., Cheng, A., Shakeel, D., and Hitt, C (2016). Comparing and Validation Measures of Character Skills: Finding from a Nationally Representative Sample (EDRE working paper ). Fayetteville, AR: Department of Education Reform, University of Arkansas. 28

31 Figure 1. Distribution of Item Non-Response Rates Figure 2. Distribution of Careless Answering Measures 29