Explaining Calibration in Classroom Contexts: The Effects of Incentives, Reflection, and Attributional Style. Douglas J. Hacker. University of Utah

Transcription

1 Calibration in Classrooms 1 Hacker, D. J., Bol, L., Bahbahani, K. (in press). Explaining calibration in classroom contexts: The effects of incentives, reflection, and attributional style. Metacognition and Learning. Running Head: CALIBRATION IN CLASSROOMS Explaining Calibration in Classroom Contexts: The Effects of Incentives, Reflection, and Attributional Style Douglas J. Hacker University of Utah Linda Bol Old Dominion University Kamilla Bahbahani Old Dominion University

2 Calibration in Classrooms 2 Abstract A 2 x 2 quasi-experimental design was used to investigate the impact of extrinsic incentives and reflection on students calibration of exam performance. We further examined the relationships among attributional style, performance, and calibration judgments. Participants were 137 college students enrolled in an educational psychology course. Results differed as a function of exam performance. Higher-performing students were very accurate in their calibration and did not show significant improvements across a semester-length course. Attributional style did not significantly contribute to their calibration judgments. Lower-performing students, however, were less accurate in their calibration, and students in the incentives condition showed significant increases in calibration. Beyond exam scores, attributional style constructs were significant predictors of calibration judgments for these students. The constructs targeting study and social variables accounted for most of the additional explained variance. The qualitative data also revealed differences by performance level in open-ended explanations for calibration judgments. Key Words: calibration, metacognitive monitoring, absolute accuracy, prediction, postdiction, attributional style

3 Calibration in Classrooms 3 Explaining Calibration in Classroom Contexts: The Effects of Incentives, Reflection, and Attributional Style Calibration is a measure of the degree to which a person s judged ratings of performance correspond to his or her actual performance (Keren, 1991; Lichtenstein, Fischhoff, & Phillips, 1982; Nietfeld, Cao, & Osborne, 2006; Yates, 1990). Ostensibly, the psychological processes underlying calibration entail a person monitoring his or her knowledge about a specific topic or skill and then self-assessing the extent of that knowledge in comparison to his or her performance on a criterion task. As such, calibration is a metacognitive process that people can use to regulate their behavior. For instance, students who can accurately assess the extent of their knowledge should be in a better position to intensify or redirect their studying for a test, provide self-guidance during reading for better comprehension, or generate self-feedback indicating that a new skill is being properly acquired. Calibration is calculated by taking the difference between a person s self-assessment of performance on a task and his or her actual performance on the task (Keren, 1991; Lichtenstein et al., 1982; Lin & Zabrucky, 1998; Schraw, Potenza, & Nebelsick-Gullet, 1993; Winne, 2004). Self-assessments made prior to performance are called predictions, and those made subsequent to performance are called postdictions. The more closely a person s predicted or postdicted performance matches his or her actual performance (i.e., the difference approaches zero), the better calibrated he or she is. For example, two people may judge that they will get 85% of the items on a test correct, but one gets 80% correct and the other gets 60% correct; the person who earned the 80% is the better calibrated of the two. The majority of calibration studies have been conducted in laboratory settings. These studies typically have strong internal validity and have provided essential information about

4 Calibration in Classrooms 4 calibration, but there is reason to believe that some of the findings generated from these studies may not generalize to naturalistic contexts, such as classrooms (Lundeberg & Fox, 1991; McCormick, 2003; Winne, 2004). In laboratory settings, the materials and procedures used to investigate calibration often involve word-pair associations that are presented over short periods of time, usually have little meaning for the participants, and are tested shortly after learning. These materials and procedures differ markedly from those used in classrooms in which learning is complex, is presented in multiple forms (e.g., lecture, reading, group participation) over long periods of time, is more motivating for students, and is tested at time intervals considerably longer than those used in laboratory studies. If the goal of psychological research is to understand cognition in the context of natural purposeful activity (Neisser, 1976, p. 7), then we need to acknowledge the differences between laboratory and classroom contexts. Although there is a continued need for laboratory work, there is also a need for researchers to move from the laboratory into more naturalistic environments. This move will compromise internal validity for external validity, but the compromise may be necessary to make psychology more practical (Parducci & Sarris, 1984, p. 10). Because there are still very few calibration studies that have been conducted in classrooms, there is much to be learned about calibration in this context. Therefore, our purpose was to conduct a quasi-experiment in four college classrooms to investigate (a) how well calibrated students can be, (b) whether incentives or reflective activities can improve calibration, and (c) whether several social-cognitive factors that are known to contribute to classroom learning also contribute to calibration.

5 Calibration in Classrooms 5 Classroom Studies to Improve Calibration Classroom studies to improve calibration have produced mixed results. In some instances, modest gains have been achieved (Hacker, Bol, Horgan, & Rakow, 2000; Nietfeld, Cao, & Osborne, 2006), but in other instances, gains have been negligible (Bol & Hacker, 2001; Bol, Hacker, O Shea, & Allen, 2005; Nietfeld, Cao, & Osborne, 2005). Moreover, strategies that have been shown to improve calibration in the laboratory do not seem to have the same effect in the classroom. For example, in the laboratory, the use of feedback has improved calibration (Arkes, Christensen, Lai, & Blumer, 1987; Glenberg, Wilkinson, & Epstein, 1987). However, in a classroom study by Nietfeld et al. (2005), feedback on exam scores did not lead to changes in students calibration; and in Bol and Hacker (2001), students who received teacher-generated feedback on practice tests were less accurate in comparison to students who engaged in a traditional review of course material without feedback. Failures to improve calibration have sometimes been attributed to lack of motivation on the part of participants. In an attempt to increase motivation, and consequently accuracy, Schraw, Potenza, and Nebelsick-Gullet (1993) provided participants in a laboratory study with extrinsic incentives, such as rewards. Schraw and colleagues found that providing extrinsic incentives improved both performance and calibration, suggesting that calibration is a controllable process that is subject to the self-efficacy beliefs and motivation of the learner. However, we could find no other study that has replicated this finding in a classroom context. In the present experiment, we attempted to replicate these findings by providing extrinsic incentives that entailed allotting an increasing number of extra credit points on tests to students who exhibited improved calibration.

6 Calibration in Classrooms 6 Interventions designed specifically to improve calibration have led to widely discrepant results. In a laboratory study conducted by Nietfeld and Schraw (2002), participants increased their monitoring accuracy after receiving short-term strategy training that did not require participants to explicitly monitor their training. In contrast, in the classroom study conducted by Nietfeld et al. (2005), in which explicit monitoring was implemented, no increases in accuracy were found. However, in a follow-up classroom study by Nietfeld et al. (2006), an intervention was used in which explicit monitoring focused on calibration, self-efficacy, and performance, and improvements in both calibration and performance were shown. Finally, in the Hacker et al. (2000) classroom study, a complex intervention was used that consisted of feedback, practice tests, and instruction on the benefits of accurate self-assessment. Increases in both prediction and postdiction accuracy were found but only for higher-performing students. To help resolve some of these discrepant findings, our primary goal for the present classroom study was to investigate whether we could improve students calibration through personal reflections on performance and extrinsic incentives. We used three interventions, each being administered three times across a 15-week course, once after each of three exams. Our interventions were developed by fully crossing two independent variables: reflection/no reflection and extrinsic incentives/no extrinsic incentives. Using a between-subjects design, we compared calibration of students across four conditions: (a) students who were asked to reflect on explanations for their calibration judgments but were not provided with extrinsic incentives to improve accuracy; (b) students who were not asked to reflect on their explanations for their calibration judgments but were provided with extrinsic incentives to improve accuracy; (c) students who were asked to reflect on their explanations and provided with extrinsic incentives to improve accuracy; and (d) students who were not asked to reflect on their explanations nor

7 Calibration in Classrooms 7 provided with extrinsic incentives. Also, because of the longitudinal nature of the study, we investigated the within subjects question of whether all students who received an intervention showed improved calibration across the semester. However, because previous studies have demonstrated that calibration varies with student performance (Barnett & Hixon, 1997; Bol & Hacker, 2001; Bol et al., 2005; Flannelly, 2001; Hacker et al., 2000; Kruger & Dunning, 1999, Winne & Jamison-Noel, 2002), our interventions may differentially affect calibration for higher-performing versus lower-performing students. Therefore, we addressed the following research questions; (1) Was calibration improved by our intervention? (2) Does calibration vary with student performance? (3) Do the effects of our interventions to improve calibration vary with student performance? Based on the literature previously discussed, for the between-subjects design we predicted that students who reflected on their explanations or received extrinsic incentives would show improved calibration compared to students who received neither. Students who both reflected on their explanations and received extrinsic incentives would show the greatest improvements in calibration. Moreover, for the within-subjects design component we predicted that students who participated in an intervention would show improvements in calibration across the semester, with students who both reflected and received incentives showing the greatest improvements. We further hypothesized that the effectiveness of our interventions would vary with student performance.

8 Calibration in Classrooms 8 The Role of Attributional Style As previously discussed, laboratory and classroom studies have shown that calibration tends to be relatively stable across time and tasks. Some studies have shown that the stability of students predictions and postdictions of performance is significantly greater than the stability of their performance (e.g., Hacker et al., 2000; Schraw et al., 1993). Such stability suggests that these metacognitive judgments may not be controllable processes as Schraw et al. (1993) have suggested, but rather, may be subject to people s stable and persistent personality traits or beliefs about their performance (Bol et al., 2005; Hacker & Bol, 2004; Hacker et al., 2000). Dunning, Johnson, Ehrlinger, and Kruger (2003) proposed that the stability in people s estimates of their performance arises from a top-down approach. That is, people begin with their preconceived beliefs about their skills and use those beliefs to estimate how well they will do on a task. In other areas of metacognitive research, researchers have found that people make some types of metacognitive judgments not by directly accessing their activated memories, but rather by constructing inferences that may bear a relation to the activated memories but are not directly connected to the memories (e.g., Koriat, 1993). For example, if asked about who the author is to a science fiction novel, a person may base a judgment-of-knowing not on actually knowing the author but on what he or she knows about science fiction novels in general. If the person knows a great deal, he or she will likely infer that the author is known, even though he or she may not know. Thus, the judgment-of-knowing is based on inferences that are generated about one s knowledge and not on actual retrieval of that knowledge. Similarly, we hypothesized that stability in prediction and postdiction judgments may be the result of inferential processes, in this case, inferences based on stable and persistent

9 Calibration in Classrooms 9 personality traits or beliefs about one s performance rather than actual or anticipated performance. Such traits or beliefs have been studied extensively in the social-cognitive literature as attributional style. Attributional style is the general way that individuals explain the causes for their successes or failures in daily events (Graham & Weiner, 1996). Students are said to exhibit a mastery orientation when they attribute their successes to factors internal to themselves, such as ability or effort, and their failures to temporary circumstances that are often under their control, for example, lack of studying (Dweck, 1986; Eccles, Midgley, & Adler, 1984). Students exhibit learned helplessness if they attribute their successes to external and uncontrollable factors, such as luck or an easy test. Conversely, they attribute their failures to internal causes, such as lack of ability, or to external negative factors that are beyond their control. Some support for our hypothesis is provided by Bol et al. (2005), who found significant relationships between calibration and students attributions: Overconfident predictions were related to external attributions and underconfident predictions to internal attributions. Social attributions also may be associated with metacognitive judgments. Jost, Kruglanski, and Nelson (1998) acknowledge that Social and cognitive biases appear to intermingle in the metacognitive realm (p. 140). In support of this thesis, Karabenick (1996) reported that the presence of co-learners questions in a social group setting elicited confusion in the participants, with greater feelings of confusion in knowing as the number of questions increased. Caravalho, Moisses, and Yuzawa (2001) presented participants with social comparative data from a fictitious group and found that the social comparisons had more impact on students with low versus high metacognitive ability and influenced confidence judgments for only low self-regulators. Puncochar and Fox (2004) examined students accuracy and confidence while completing quizzes in groups or individually. They showed that people who

10 Calibration in Classrooms 10 worked in groups to be more accurate and confident in their right answers than people who worked alone. Unfortunately, group confidence for wrong answers continued to increase across quizzes, a finding the authors coined the two heads are worse than one effect. The relationships between attributions and predictions or postdictions, however, may not be direct. Studies have shown that attributions vary with performance: Internal attributions are associated with higher performance and external attributions with lower performance (Bar-Tal & BarZohar, 1977; Borkowski, Carr, Rellinger, & Pressley, 1990; Boss & Taylor, 1989; Seligman, 1991). Other studies have shown that higher performance is associated with underconfidence in predictions and postdictions and lower performance with overconfidence, with the lowestperforming students showing gross overconfidence (e.g., Bol et al. 2005; Hacker et al., 2000). Thus, internal attributions may be associated with underconfident metacognitive judgments and external attributions with overconfident judgments, but these associations may depend on performance level. We addressed this possibility in our fourth research question: (4) Do the relations between attributional style and predictions or postdictions depend on performance level? In answering this question, we made the following predictions. Because higherperforming students tend to be underconfident in their judgments of performance, they must explain why they performed better than what they had judged. To explain the discrepancies between judged and actual performance, we predicted that they should be less inclined to attribute blame for their better performance to factors such as poor instruction, bad study efforts, or to social influences. In contrast, because lower-performing students tend to be overconfident in their judgments of performance, they must explain why they performed worse than what they

11 Calibration in Classrooms 11 had judged. Therefore, we predicted that they should be more inclined to attribute blame for their worse performance to poor instruction, study efforts, or to social influences. Method Participants Participants were 109 female and 28 male freshman, sophomores, or juniors enrolled in a 15-week college introductory educational psychology course. The course is required for most teacher education programs, and students enrolled in the course were either currently in a teacher education program or were about to apply for one. The lowest acceptable grade for the course is a C; therefore, student motivation to perform well was likely high. Students were predominantly white and middle-class. Participation in the study satisfied the research participation requirement for the course. Alternative ways to satisfy the requirement were provided. Design We used a quasi-experimental design in which our interventions were formed by fully crossing extrinsic incentives/no extrinsic incentives with reflection/no reflection: Students received only extrinsic incentives (n = 37), only reflection (n = 27), both extrinsic incentives and reflection (n = 36), or no incentives or reflection (n = 37). The four classroom sections of the course were randomly assigned to one of the four conditions. The textbook, curriculum, and instruction across the classrooms were standardized as were all exams, quizzes, and assignments. There were different instructors in each classroom; however, experimental materials relating to calibration and attributional style were administered by the first author, and instructors did not discuss the materials or the nature of the study with their students. Enrollment in the four sections of the course was open to all students so that there was no bias in academic ability among the classrooms. The curriculum of the course contained nothing about attributional style,

12 Calibration in Classrooms 12 and the instructors neither lectured about it nor encouraged their students to examine attributional style in their testing. An analysis of the four attributional style constructs examined in this study showed that there were no differences among the four groups on any of the constructs except for the task-centered construct, and only one group (i.e., the reflection group) differed from the other three. Measures Performance and calibration. Performance was measured using three exams that were administered across the semester course. Each exam consisted of 35 multiple-choice questions and assessed content covered in discussion groups, lectures, and the textbook. The first author developed all exam items based on a table of specifications aligned with course objectives. The mean scores and standard deviations for the three exams were, (3.97), (3.82), and (3.70), respectively. Calibration data were obtained by asking students prior to testing to predict how many of the 35 items they would get correct and immediately after the exam by asking how many of the items they had gotten correct. Prediction accuracy was calculated using the formula: Prediction - Performance (1 - ) Total Items X 100 Postdiction accuracy was calculated by substituting postdiction for prediction. The values produced by this formula correlate perfectly with the simple absolute value of the difference between prediction/postdiction and performance; however, because the values are expressed as a percentage, calibration is more intuitively clear. For example, a student who predicted a 33 on a 40-item test and earned a 35 would have a calibration score of 95%.

13 Calibration in Classrooms 13 Attributional style. We used an instrument (see Table 1) to measure students attributions regarding calibration that was developed by Bol et al. (2005). Eight items pertained to a taskcentered construct that concerned whether students focused on external factors to explain their calibration. Specific items asked students if they found test items tricky, if the test covered the content of the course, or how adequate the instruction was. Nine items pertained to a studentcentered construct and measured the degree to which students focused on internal factors for their calibration. Five of these items focused on students test-taking abilities, for example, whether students evaluated themselves as good test takers, and four items focused on students studying for the test, for instance, had they directed their studying to the right material. Because previous studies have suggested that students tend to be inaccurate in their predictions and postdictions, most items were worded negatively to elicit explanations of discrepancies between their judgments and their actual performance. Even when students make fairly accurate judgments, it is rare to achieve perfect accuracy. Therefore, the wording of the items would apply to all but the very rare student who may have perfect calibration. Our choice to use negative wording is further supported by Reivich and Gillham s (2003) revised, extended versions of the Attributional Style Questionnaire in which higher reliabilities were obtained when individuals were asked to report their explanations for negative events. Using a sample of 356 participants, Bol and colleagues (2005) examined the factor-analytic structure of the instrument and found that the items loaded on three distinct factors, each one corresponding to the proposed constructs. For the present study, we added a social-centered construct, which was intended to measure students attributions concerning social influences on calibration. Four items were used to measure such things as making performance comparisons with other students or whether the teacher s feedback had influenced their performance judgments.

14 Calibration in Classrooms 14 Students responded to the prompt, How would you explain any discrepancy between how well you thought you would do on the first [second, third] exam and how well you actually did? using a 5-point Likert-type scale ranging from strongly disagree to strongly agree (1-to-5, respectively). Therefore, the emphasis of students responses was placed on explaining why their actual performance was better or worse than their self-assessed performance. High scores on the task-centered construct indicated a strong emphasis on external causes for discrepancies; high scores on the studying construct indicated a greater emphasis on internal studying behaviors as causes; high scores on the testing construct indicated a greater emphasis on internal testing behaviors as causes; and high scores on the social-centered construct indicated greater emphasis on external social sources as causes. Low scores on all the constructs indicated that the internal or external causes played little or no role in the discrepancies. We conducted a principal components analysis followed by a principal axis factoring with a varimax rotation and eliminated items with loadings less than.30. The analysis produced a four-factor solution that accounted for 44% of the variability (See Table 2). The Kaiser- Meyer-Olkin measure of sampling adequacy was.79, exceeding the.60 value considered the lower value for good factor analysis. We also examined the variables for multicollinearity and singularity and found that neither was present. All eight of the task-centered items loaded on the first factor, four of the five student-centered testing items loaded on the second factor (one item did not have a clear factor loading and was eliminated from further analyses), all four of the student-centered studying items loaded on factor three, and all four of the social items loaded on factor four. The reliabilities (Cronbach s alpha) for the task-centered, student-centered testing, student-centered studying, and social-centered factors were.88,.64,.65, and.60, respectively, with an overall reliability of.82. With the exception of one item, our solution replicated Bol and

15 Calibration in Classrooms 15 colleagues (2005) three-factor solution, and the addition of the four social items added a clearly distinct factor. Because the present analysis generated the same number of factors, with the same items loading on the different factors for the two sample groups, we did not proceed with further statistical comparisons to confirm the stability and validity of our instrument (Tabachnick & Fidell, 1989, p. 642). Table 2 illustrates the similarities between the two samples. In addition to the attributional style instrument, we asked students in all conditions to respond to open-ended items. The items asked students to identify (a) factors that may have influenced their accuracy and (b) strategies they might employ for enhancing their accuracy. Procedure All students shared the following procedure. Immediately before taking an exam, students predicted their scores and recorded their predictions on sheets of paper. After the exam but before grading, they postdicted their scores and recorded their postdictions. Finally, after recording their postdictions, the first author scored their exams, calculated their calibration, and privately provided each student with his or her calibration scores for prediction and postdiction. For the extrinsic incentive condition, students were told prior to testing that they would receive extra points for greater accuracy: Four points if prediction and postdiction were both one or two points from the exam score, three points if both were three or four points from the exam score, two points if both were five or six points from the exam score, and one point if seven or more points from the exam score. When these students were provided with their calibration scores, they also were told how many extra points they had earned. For the reflection condition, students were asked to reflect on the accuracy of their judgments. Immediately after receiving their calibration scores, these students were given the attributional-style questionnaire. In addition to measuring students attributions, the

16 Calibration in Classrooms 16 questionnaire required that students reflect on explanations for the possible discrepancies between their judged and actual performance. By reflecting on the various internal and external causes for the discrepancies, students were provided with substantive information concerning their mastery of the course material and with guidance to increase their calibration. Students in the extrinsic incentives-plus-reflection condition received a combination of the two interventions just described. Finally, students in the fourth condition received neither incentives nor were asked to reflect. These students simply predicted and postdicted their scores and responded to the attributional-style questionnaire only after the third exam. At the end of the study all students received the same number of extra credit points as compensation for their participation in the study, which satisfied the research requirement for the course. Results Was Calibration Improved? We examined whether calibration was improved using both between- and within-subjects analyses. For the between-subjects analysis we compared the calibration scores for prediction and postdiction on the third exam across the four conditions. Because prior research has shown that calibration is moderated by performance, we equalized performance across the four conditions by using the third exam score as a covariate. We conducted an analysis of covariance (ANCOVA) for both prediction and postdiction accuracy. The means and standard deviations for each condition are shown in Table 3. Levene s test of equality of error variances indicated that the assumption of homogeneity for either prediction or postdiction was not violated. Our analyses showed no significant group differences for either prediction (p =.14) or postdiction accuracy (p =.20). No intervention led to improvements in calibration. Surprisingly, the

17 Calibration in Classrooms 17 students who did not engage in any intervention predicted and postdicted their performance with about the same degree of accuracy as the students who engaged in an intervention. We analyzed within-subjects calibration accuracy by collapsing across the three interventions and conducting a repeated measures analysis of covariance (RM-ANCOVA) for students prediction and postdiction accuracy (see Table 3). We equalized potential differences among the three different classrooms assigned to the three interventions by using either prediction or postdiction accuracy on the first exam as a covariate and then used prediction and postdiction accuracy on the second and third exams as repeated measures. Also, to eliminate the possibility of calibration differences due to moderating effects of performance, we conducted a paired samples t-test using the two exam scores and found no difference between exam 2 and exam 3 (M = 26.41, SD = 3.59; M = 26.65, SD = 3.52; p =.55). The RM-ANCOVA showed no significant within-subject effects for either prediction (p =.48) or postdiction accuracy (p =.57). Students did not improve their calibration accuracy across the second and third exams. Does Calibration Vary with Student Performance? Studies of calibration consistently have shown that accuracy varies with performance. Therefore, we conducted a similar RM-ANCOVA but with performance as a between-subjects variable. Our performance variable was a dichotomous variable created using a median split on the total of the three exam scores. The median score was 84. To eliminate the potential effects of outliers (i.e., abnormally low scores) for the lower performing group, we restricted the range of scores for both groups to 16. This resulted in the elimination of 5 students. Descriptive statistics on the two groups were: higher performing, n = 48, M = 89.46, SD = 4.05, range 84 to 100; lower performing, n = 46, M = 74.87, SD = 4.96, range 66 to 83.

18 Calibration in Classrooms 18 For prediction and postdiction accuracy, the RM-ANCOVAs showed only significant main effects for performance group, F(1, 91) = 28.46, p <.01, η 2 =.24, F(1, 91) = 12.34, p <.01, η 2 =.12, respectively. Interactions between the repeated measure and performance group were not significant nor were the interactions between the repeated measure and the covariate. Adjusted and unadjusted means and standard errors are shown in Table 4. Higher-performing students showed greater prediction and postdiction accuracy than lower-performing students, but neither higher- nor lower-performing students showed increases in accuracy from the second to third exams. For the higher-performing students, the high rates of accuracy and the constrained standard errors on the unadjusted means for both prediction and postdiction in comparison to the lower-performing students indicate that higher-performing students were nearing a ceiling on both measures. Therefore, whether increased accuracy was possible is uncertain. Do the Effects of Our Interventions to Improve Calibration Vary with Student Performance? Although our interventions did not significantly improve calibration, because we did find strong main effects for calibration by level of performance, we examined whether our interventions impacted calibration differently for lower- than higher-achievers. To test this we conducted a RM-ANCOVA with prediction or postdiction accuracy on exams two and three as the repeated measure, prediction or postdiction accuracy on exam one as a covariate, and both intervention and performance group as between-subjects variables. For prediction and postdiction we found the main effects for performance group, but for postdiction accuracy we also found a significant intervention x performance group x postdiction accuracy interaction, F(2, 87) = 3.89, p <.03, η 2 =.08. Table 5 shows that higher-performing students in all three conditions were consistent in their postdiction accuracy across the two exams, but lower

19 Calibration in Classrooms 19 performing students in the incentives and reflection-plus-incentives groups improved accuracy while the reflection group lost accuracy. To What Extent Does Attributional Style and Performance Contribute to Calibration Judgments? Apart from the question of whether calibration can be improved, we also were interested in identifying what contributes to people s prediction and postdiction judgments. Why do these judgments appear to be so stable across tasks and time? We hypothesized that stability in these judgments may be the result of inferential processes, in this case, inferences based on stable and persistent personality traits or beliefs about one s performance rather than actual or anticipated performance. We identified four attributional-style constructs that have been associated with classroom performance and that could explain such stability, and then examined whether these constructs made significant contributions to calibration beyond performance. First, however, if the attributional-style instrument that we used validly measured the four constructs, the following two predictions should be confirmed: Higher-performing students will be less inclined than lower-performing students to attribute blame for their performance to factors such as poor instruction, bad study efforts, or to social influences. Thus, higher-performing students should score lower on the four attributional-style constructs than lower-performing students. Because students who were assigned to the reflection conditions responded to the attributional-style questionnaire three times during the semester, we avoided biasing of the attributional constructs by those students by using students responses to the questionnaire from its first administration. Therefore, in the following analyses, we used the questionnaire responses, predictions, postdictions, and exam scores from the first exam for those students in the reflection and reflection-plus-incentives condition, and the questionnaire responses, predictions, postdictions, and exam scores from the third exam for those students in the

20 Calibration in Classrooms 20 incentives condition. This may have meant that the predictions and postdictions were biased toward those in the incentives condition, who made repeated calibration judgments by the third exam; however, we conducted two ANOVAs and found that there were no significant differences among the three groups for prediction (p =.69) or postdiction (p =.26). As indicated by the high Cronbach s alpha values (.85 for prediction,.78 for postdiction), students were remarkably consistent in their predictions and postdictions across the three exams. In fact, their consistency in making these judgments equaled or exceeded their consistency in performance (Cronbach s alpha =.77). We conducted a multivariate analysis of variance (MANOVA) using the dichotomous median split on performance as the independent variable and the four attributional-style constructs as the dependent variables. Levene s test of equality of error variances indicated that we did not violate the assumption of homogeneity for any of the variables. The analysis confirmed our prediction that all four measures were significantly lower for higher-performing students than for lower-performing students (see Table 6). We then investigated the unique contributions that the four attributional-style constructs made to students predictions and postdictions. Given that higher-performing students as compared to lower-performing students scored lower on the four constructs, we expected that any contributions these constructs made to students predictions and postdictions would be less for the former group than for the latter. Moreover, because of the strong relationships between exam score and prediction and postdiction that were found in this study (see Table 7), and is commonly found in other studies, we examined the unique contributions from exam score. We used standard multiple regression, with exam score entered first and the attributional-style constructs entered as a separate block.

21 Calibration in Classrooms 21 For higher-performing students, we found that exam score was a significant contributor to students predictions and postdictions; however, the attributional-style constructs did not significantly add to the R 2 value for either judgment. For lower-performing students, the attributional-style constructs were significant contributors beyond exam scores for both prediction and postdiction (see Tables 8 & 9). For prediction, the attributional variables increased the adjusted R 2 value from.04 to.24, and for postdiction, increased the adjusted R 2 value from.31 to.57. An examination of the four constructs shows that the social and studying variables were responsible for the increases in R 2 for both prediction and postdiction. In sum, for higher-performing students, how much they knew about the to-be-tested course material was the most significant contributor to their predictions and postdictions and attributional style did not add to this. For lower-performing students, beyond how much students knew about the to-betested material, the attributional constructs concerning their internal studying behaviors and external social influences strongly contributed to their predictions and postdictions. Qualitative Responses by Attributional Style and Performance In addition to quantitative analyses, we explored differences in attributional style by performance using students qualitative responses to the open-ended questions on the attributional-style questionnaire. The first question asked them to identify factors that influenced the accuracy of their predictions and postdictions. We generated a coding scheme of the students responses using the four attributional constructs and added an other category to include themes that did not fit within the four constructs or themes. Within each construct and the other category, we developed categories through multiple readings of the data. The development of categories was conducted by two researchers to strengthen the credibility of findings through triangulation. An inductive approach was employed in which the researchers

22 Calibration in Classrooms 22 identified topics among responses and clustered related topics. Category labels that best captured the meaning for a cluster of topics were generated. Once a set of categories was developed, we conducted trial codings to ensure that the categories were mutually exclusive and best represented the data. Because attributional styles differed by performance group, we divided students into higher- and lower-performing groups using the same median split that we used earlier and then examined responses for higher- and lower-performing students for each construct and other category. Responses were calculated as percentages of the number of responses coded by performance group, not on the number of students in the study. Only those categories that contained at least two percent of the responses were included. Idiosyncratic responses not represented by the categories were excluded. Table 10 presents each attributional construct, the categories within those constructs, and the percentages. We found that higher- and lower-performing students differed little in identifying taskcentered constructs as influencing their accuracy (19% for higher performers and 17% for lower). Most responses pertained to the students expectations about and reactions to tests. For example, one higher-performing student wrote, This test seemed a lot different than the previous two. A lower-performing student asserted, the exam is not at all like the quizzes. Also I learn better through discussion and this class was lacking in discussion. In both cases, the discrepancy between scores and predictions and postdictions was attributed to an external source, characteristics of the task, unexpected items, or test content. Differences between groups emerged on the student-centered constructs. Lowerperforming students were more likely to attribute discrepancies between scores and calibration judgments to student-centered studying sources (48% for lower performers vs. 31% for higher). The themes within this construct reflected how much or how well they studied, the study content,

23 Calibration in Classrooms 23 or how well they felt they knew the material. One lower-performing student said that her calibration was negatively influenced by, How well I was able to focus on the material while I studied I wasn t able to focus I didn t retain much and I didn t study enough. In reference to not studying the right or enough content, another lower-performing student said, I didn t know what the content was that was most important, so I guessed on what to study as well as how well I should do. These comments demonstrate students awareness that a lack of familiarity with course material influenced their abilities to predict and postdict their performance. A third theme pertained to how well students felt they knew the material. Many explained, I just based my predictions on how well I knew the material. Others used phrases such as how comfortable, confident, or prepared they were to make their predictions or postdictions. Again, students identified a relationship between their level of knowledge of the material and their calibration abilities. In contrast, higher-performing students were more likely than lower-performing students to attribute discrepancies between scores and calibration judgments to test-taking ability or performance sources (24% for higher performers vs. 19% for lower). However, the two groups differed little across the three themes within the student-centered testing construct. Two of the themes related to using prior quiz or test performance. One higher-performing student noted, I just figured I would do the same as I did on other exams in this class because I prepared in the same way. A third theme to emerge was anxiety and other stressors related to the test. For instance, one student was nervous before grading so I lowered my postdictions. A lowerperforming student wrote, I missed class last week. I don t take tests well. I have terrible test anxiety. Anxiety was at least as common in the higher performers as lower (7% vs. 5%).

24 Calibration in Classrooms 24 Given the prominent role that social attributes played in contributing to lower-performing students predictions and postdictions, as indicated in our earlier regression analyses, we were surprised that only higher-performing students (2%) provided open-ended comments categorized as social. For example, Others around had lower postdiction scores or got lower grades than they expected, so I lowered my postdiction 1 point. This illustrates at least some awareness of social influences among this group of students. There were numerous responses that did not fit into the attributional-style constructs, and higher-performing students tended to provide more of these other explanations than lowerperforming students (24% vs. 17%, respectively). When comparing the responses by themes within this category, we found that higher-performing students were more likely to attribute discrepancies between scores and calibration judgments to a lack of confidence or personal expectations. Given that these students scored well on the exams, these types of responses were unexpected. I usually do very well on tests, but I always underestimate my abilities. Another admitted, after taking tests I always worry I didn t do well because I second guess my answers and knowledge. Only responses from the higher-performing students were coded as wishing or hoping for high scores or to avoid disappointment. One student said, I just took a guess in hopes that I would get that many right. Another explained, I went low so I wouldn t be disappointed. Lower-performing students were a bit more likely to attribute the discrepancies between scores and calibration judgments to self-knowledge or intuition. One student wrote, I know myself and how much I studied. Another noted, I am so unpredictable. I can study a lot and bomb a test, or hardly study at all and do ok so I am never sure how well I ll do.

25 Calibration in Classrooms 25 Discussion Our goal for the present study was to understand cognition in the context of natural purposeful activity (Neisser, 1976, p. 7). In particular, we sought to expand on laboratory studies of calibration by examining calibration in classroom contexts, in this case, four classes of an undergraduate introductory educational psychology course. Because of the contextual differences between the laboratory and classroom most notably, more complex and multi-form learning, longer periods of learning, greater motivation, and longer delays between learning and testing we, along with others (e.g., Lundeberg & Fox, 1991; McCormick, 2003; Winne, 2004) believe that laboratory findings may not readily generalize to classroom contexts. Admittedly, the quasi-experiment that we conducted did not possess the degree of control that is possible in laboratory studies; however, we took several measures to hold as many variables constant across the four classes as possible: the classes were randomly assigned to an intervention; the teachers in the classes did differ, but each teacher had similar experiences teaching the course; the curriculum was standardized, as were the exams; student performance on the exams did not differ across the classes; the curriculum did not include material on attributional style; the instructors neither lectured about attributional style nor encouraged their students to examine it in the context of the course; and the teachers did not administer the interventions. We recognize that conducting research in naturalistic contexts will not provide the same degree of rigor as a true experiment, but we also recognize that quasi-experiments may be the best alternative available when investigating behavior and cognition in naturalistic contexts. Because classroom studies of calibration have produced mixed results, we sought to resolve some of these discrepant findings by investigating three aspects of calibration: (a) how well calibrated students can be, (b) whether incentives or reflective activities can improve

26 Calibration in Classrooms 26 calibration, and (c) whether several social-cognitive factors that are known to contribute to classroom learning also contribute to calibration. We used three interventions, each being administered three times across a 15-week course, once after each of three exams. Our interventions were developed by fully crossing two independent variables: reflection/no reflection and extrinsic incentives/no extrinsic incentives. Using a between-subjects design, we compared calibration of students across four conditions: (a) students who were asked to reflect on explanations for their calibration judgments but were not provided with extrinsic incentives to improve accuracy; (b) students who were not asked to reflect on their explanations for their calibration judgments but were provided with extrinsic incentives to improve accuracy; (c) students who were asked to reflect on their explanations and provided with extrinsic incentives to improve accuracy; and (d) students who were not asked to reflect on their explanations nor provided with extrinsic incentives. Also, because of the longitudinal nature of the study, we investigated in a within-subjects design whether students who received an intervention showed improved calibration across the semester. When all students were considered together, we found that our interventions did not impact calibration. Calibration was remarkably consistent across the 15 weeks, even for the control group who engaged in no intervention. However, when we examined calibration in terms of student performance, we found numerous differences between higher-performing and lowerperforming students. Higher-performing students students who scored above the median on the combined three examinations were well calibrated, that is, the discrepancies between predictions or postdictions and actual performance approached zero. These students were about 94% accurate in their predictions and postdictions, which means that their judgments differed from perfect

27 Calibration in Classrooms 27 accuracy by only 1 or 2 items on an exam. Given the constrained standard deviations for this group, their accuracy was approaching a ceiling. This degree of accuracy differs markedly from many laboratory studies that have shown accuracy rates ranging from 50-75%. Thus, in a classroom context, students who possess high levels of knowledge of the course content can be expected to be well calibrated. These findings suggest along with many other studies that highly accurate calibration may be possible only when a person possesses high knowledge of the content being judged (e.g., Dunning, Heath, & Suls, 2004; Hacker et al., 2000; Kruger & Dunning, 1999; Winne & Jamieson-Noel, 2002). In our investigations of attributional style, we found that the relations between attributional style and predictions or postdictions do depend on performance level. Our predictions were confirmed that higher-performing students would be less likely than lowerperforming students to attribute blame for the discrepancies between judged performance and actual performance to class instruction, study efforts, or social influences. These findings seem intuitively clear in that if students believe there is less of a discrepancy between judged and actual performance, there is less reason to find excuses for inaccurate judgments. Therefore, none of the attributional-style constructs contributed beyond exam score to higher-performing students predictions or postdictions. Lower-performing students students who scored below the median on the combined three examinations presented a much different picture. These students were not as well calibrated as higher-performing students, with mean prediction and postdiction accuracy at about 86-88%. Although these values are high, there was room for improvement, and in our analyses of the interactions among the interventions and student performance we did find some evidence of improvement. Lower-performing students in all conditions showed stable prediction accuracy