2013 Supervisor Survey Reliability Analysis

Transcription

1 2013 Supervisor Survey Reliability Analysis In preparation for the submission of the Reliability Analysis for the 2013 Supervisor Survey, we wanted to revisit the purpose of this analysis. This analysis is intended to assess the reliability and validity of the Supervisor Survey instrument based on research, in order to inform future versions of this and other instruments. This document provides information with respect to the factor structure and reliability of the instrument based on statistical analysis and other research. The factor structure is explored using a factor analysis, which serves to determine the underlying relationships of the individual items in the instrument for the purpose of combining items into a smaller number of scales that represent the larger constructs under study. The internal consistency reliability of the instrument and the factors is also explored, which looks specifically at whether all of the items on the instrument or in the factors are measuring the same construct. The process can also provide recommendations regarding scales that can be used in future statistical analyses, and items that do not seem to be measuring a specified construct and could potentially be deleted. We do not intend to advocate for the implementation of any specific changes or for any particular timeframe, but to provide information which could be used to inform these decisions. These decisions should be made by the entire Common Metrics Group and are beyond the scope of this document. Factor Analysis A factor analysis was performed on all Likert-scale items on Parts A through D of the Supervisor Survey. This factor analysis provides information about which items most contribute to the overall construct represented by each part of the instrument. This analysis can suggest which items could potentially be eliminated from each part if they do not contribute to the overall understanding of the construct. This does not necessarily mean the items should be eliminated however. As stated earlier, the statistical analysis only provides information to the researcher. The Common Metrics Group needs to make the ultimate decision about whether the items in the instrument provide valuable information to answer the research questions; and whether to retain, revise, or eliminate items identified in the factor analysis as reducing the instrument s reliability. Many of the 186 respondents for this survey did not provide an answer to every item. In addition, those who selected Unable to Respond for one or more items in each part were excluded from the analysis for that part, as this response did not provide useful information with respect to new teachers instructional practices. Only those respondents with a complete response set for each part were included in the factor analysis. The sample size for each part of the survey is included as Table 1. Table 1. Sample Size for Survey Parts Part Valid Responses Excluded Responses A B C D

2 For this data set, principal axis factor analysis with varimax rotation was conducted to evaluate the underlying structure of the items for each part of the Supervisor Survey. Several assumptions (determinant, KMO, Bartlett) were tested to ensure that factor analysis was appropriate for these data. All of these assumptions were met for all parts of the Supervisor Survey; the variables were highly correlated enough for analysis; and there were an adequate number of items to complete the factor analysis. The Kaiser criterion was used to determine how many factors to retain in each analysis. Results Factor Analysis Part A A factor analysis was completed for Part A, items 1 to 26. These items were all intended to relate to the first-year teacher s instructional practice. First, correlations between the items were calculated, as this provides information about how the items are related to each other. According to Cohen (1988), correlation coefficients between.3 and.49 represent a moderate correlation between two variables. Coefficients from.5 to 1.0 are strong correlations. Based on these guidelines, correlations range from weak to strong between the items in Part A. Correlation coefficients range from.101 to.811, with the majority of correlations being moderate to strong. Items with strong correlations are associated with each other and will likely be grouped together in the factor analysis, while those with low correlations are unlikely to load on the same factor (Leech, Barrett, & Morgan, 2011). For the 26 items in Part A, four factors emerged. The first factor accounted for 20% of the variance, the second accounted for 18%, while the third and fourth factors accounted for 13% and 10%, respectively. Twelve of the items loaded strongly on one factor: items 5, 6, 8, 9, 11, 12, 13, 16, 17, 18, 19, and 20. These items all address the teacher s ability to design instruction based on standards, individual student s abilities, and students needs, which could potentially result in one scale for these items to be used in an analysis. Seven items loaded strongly on a second factor: items 10, 21, 22, 23, 24, 25, and 26. With the exception of item 10, these could potentially result in a scale related to encouraging students higher-order thinking skills. In addition to loading with conceptually unrelated items, item 10 also had lower correlations than other items, suggesting that this item may not be a good fit for any of the scales in this part. Items 1, 2, 4, 14, and 15 created a third factor, primarily related to setting learning and curricular goals. Items 3 and 7 loaded onto a fourth factor and address the teacher s use of varietal strategies and activities to teach his/her subject matter. Item 7, which relates to a teacher s use of instructional strategies, also loaded ambiguously with Factor 1. The wording of items 3 and 7 may be the reason they failed to load in Factor 1 with other conceptually similar items related to designing classroom instruction. Items 9, 13, and 19 also loaded ambiguously between two factors. The cross-loading for these items indicate they may be redundant or could be revised to better align with other items that represent the same construct. Similar to the findings from the 2012 Supervisor Survey validity study, items 10, 21, 22, 23, 24, 25, and 26 loaded onto one factor creating a higher-order thinking skills scale. While items 9, 16, 17, 18, and 20 loaded together in both samples, as did items 1, 2, 4, 14, and 15, there was variation in the other items that loaded on the scales. Similar results from different respondent samples reinforces that these items could be used as scales in other statistical analyses, though

3 the mixed results of items that did not load together in both analyses suggest phrasing or repetition may be preventing respondents from interpreting those items in a consistent way. Recommendations Part A In future versions of the instrument, revising the wording of items to ensure that consistent phrasing is used throughout the instrument may help these items load on the factors more consistently with their conceptual meanings. For example, the items in this part include the terms instruction, learning tasks, activities, and learning opportunities, but it is not clear if these refer to the same or different portions of the classroom instruction. Using the same terminology for all items could increase the validity of the instrument. Alternatively, defining terms for respondents may help to ensure that items are interpreted as intended by the researchers (Babbie, 1990, Fowler, 2009). Creating scales based on the identified factors would reduce the number of variables used in subsequent analyses, potentially increasing the chances of finding significant effects in future research using these data. In particular, the higher-order thinking scale has been consistent across samples and could be used for more inferential statistics. Other scales have been less stable and may suggest the need for item revisions or deletions. If items are revised, additional factor analysis should be conducted to determine if factor loadings change as a result of any revisions. The results from the factor analysis are included in Table 2, with circles indicating items loadings on the four factors.

4 Table 2. Factor Matrix for Supervisor Survey Part A Factor Assessment to diagnose gaps Assignment difficulty Meet students needs Adjust based on assessments Modify assessments Social/emotional needs Role of standardized tests Real-life experiences Feedback to students Align with standards Prior knowledge Assessments for monitoring Students solve global problems Students solve abstract problems Students gain digital literacies Students work collaboratively Students analyze evidence Student self-assessment Educational technology Planning with objectives Learning goals Subject matter Long-range instructional plans Align strategies & goals Engage students Instructional strategies Factor Analysis Part B A factor analysis was completed for Part B, items 27 to 34. These items were all intended to relate to the first-year teacher s practices with diverse learners. Using Cohen s (1988) guidelines, all of these items were moderately to strongly correlated, with correlations ranging from.43 to.86. Similar to Part A, this potentially suggests that first-year teachers who are strong in one area related to practice with diverse learners are also strong in the other areas. The factor analysis suggests a one-factor solution which accounts for almost 70% of the variance. Similar to the 2012 instrument validity study findings, all items loaded strongest on only one factor, suggesting that Part B represents one scale related to diverse learners.

5 Recommendations Part B With over half of the respondents supplying incomplete or Unable to Respond answers for items in Part B, an analysis of these responses may provide information to help revise the instrument by removing items that many respondents were unable to answer. Specifically, there were three items which resulted in a substantial number of the responses being excluded from the analysis: 29. The teacher designs instruction to support students who are English language learners. 30. The teacher designs instruction for students with mental health needs. 34. The teacher effectively teaches students who have experienced trauma and/or displacement. It is highly probable, given that these items refer to very specific segments of the student population, that many teachers do not have any students in their classrooms that would fit these specific descriptions. Using broader terms, such as at-risk students, may provide opportunities for more respondents to give valid answers to these items, thus increasing the response rate for these items. In future versions of the instrument, revising the wording of items to ensure that consistent phrasing is used throughout may help respondents to consistently interpret and respond to the items. For example, four different phrases are used in Part B to refer to the teachers actions: differentiates, designs, advocates, and effectively teaches. It may be confusing for respondents when some items refer to the design of the instruction, while others refer to actual teaching. Providing definitions for terms or consistent language throughout the part will likely make it easier for respondents to accurately answer all items and increase instrument validity (Babbie, 1990, Fowler, 2009). Consistent factor loadings from different respondent populations in 2012 and 2013 indicate these items could be used as one scale to increase the chances for statistically significant findings as well as the options available when selecting inferential tests. If items are revised, additional factor analysis should be conducted to determine if factor loadings change as a result of any revisions. The results from the factor analysis for Part B are included in Table 3, with a circle indicating items loadings on the factor.

6 Table 3. Factor Matrix for Supervisor Survey Part B Factor Mental health needs Culturally and ethnically Traumatized students Differentiates learning Diverse backgrounds English language learners Gifted and talented Students needing special assistance.652 Factor Analysis Part C A factor analysis was completed for Part C, items 35 to 43. These items were all intended to relate to the classroom environment. Using Cohen s (1988) guidelines, except for one pair, all items are strongly correlated, with correlations ranging from.46 to.86. Again, this potentially suggests that first-year teachers who are strong in one area related to creating a classroom environment are also strong in the other areas. The factor analysis indicated a one-factor solution which accounted for 68% of the variance. All items loading strongly on only one factor suggested that Part C represents one scale related to classroom environment. Part C items loaded onto one factor on both the 2012 and 2013 instrument validity studies, further supporting that Part C represents one scale. Recommendations Part C Grouping items with similar constructs may promote ease of completion by respondents. For example, items 35, 42, and 43 all relate to the teacher s communication with students. Grouping these three items together on the instrument could potentially make it easier for supervisors to respond accurately to all items. In order to reduce the likelihood of inconsistent interpretations impacting results, it is recommended that each item pose one and only one question. For example, item 39 asks about the teacher s ability to foster self-control and self-discipline. Creating two items, one about self-control, and one about self-discipline, and providing definitions for each may provide more consistent and accurate responses for this item (Babbie, 1990, Fowler, 2009). Consistent factor loadings from different respondent populations indicate these items could be used as one scale to increase the chances for statistically significant findings as well as the options available when selecting inferential tests. If items are revised, additional factor analysis should be conducted to determine if factor loadings change as a result of any revisions. The results from the factor analysis are included in Table 4, with a circle indicating items loadings on the factor.

7 Table 4. Factor Matrix for Supervisor Survey Part C Factor Classroom environment Behavior management Student self-discipline Effective communication skills Effective listening techniques Behavioral expectations Physical environment Cultures interact positively Respects differences.723 Factor Analysis Part D A factor analysis was completed for Part D, items 44 to 51. These items were all intended to relate to professional responsibilities. Using Cohen s (1988) guidelines, these variables are all moderately to strongly correlated, with correlations ranging from.36 to.78. Again, this potentially suggests that first-year teachers who are strong in one area related to fulfilling their professional responsibilities are also strong in the other areas. The factor analysis suggests a one-factor solution which accounts for 57% of the variance. As all items loaded strongest on only one factor, this suggests that Part D represents one scale related to professional responsibilities. Similar results from the 2012 instrument validity findings support that all items in Part D represent one scale. Recommendations Part D In order to ensure common definitions of terms by both respondents and researchers, additional information in parentheses for terms that may be inconsistently understood may be helpful for respondents. For example, for items 48 and 49, providing examples of community and home resources, respectively, will help provide clarification for respondents with respect to how these terms are defined for this research. Similar factor loadings to the 2012 instrument validity results indicate these items could be used as one scale to increase the chances of finding statistical significance in future analyses, as well as increasing the options available in terms of appropriate inferential tests. If items are revised, additional factor analysis should be conducted to determine if factor loadings change as a result of any revisions. The results from the factor analysis are included in Table 5, with a circle indicating items loadings on the factor.

8 Table 5. Factor Matrix for Supervisor Survey Part D Factor Collaborates with colleagues Professional development Colleague feedback Needed improvements in teaching Community resources Home resources Engages parents/guardians Legal responsibilities.648

9 Instrument Reliability The reliability of the scales identified by the factor analyses was assessed using Cronbach s alpha. For Part A, Cronbach s alpha was calculated for all items together and the four potential scales suggested by the factor loadings. For Parts B, C, and D, Cronbach s alpha was calculated including all items for each part, as the factor analysis suggested only one scale for each part. The alphas for the scales, Part A 1 to 26, Part B 27 to 34, Part C 35 to 43, and Part D 47 to 51, all.90 or higher, indicate adequate internal consistency for these scales (Table 6). However, Cronbach s scores of.95 or higher reinforce previous findings that some items may be repetitive and could be deleted. The low alpha score of.74 for Factor 4 of Part A reinforces that these items have a conceptually weak alignment. Table 6. Reliability Estimates for Scales Part Items Scale Alpha A 1 to 26 Instructional practice.96 5, 6, 8, 9, 11 to 13, Instruction for standards/goals (Factor 1).93 10, 21 to 26 Encourage higher-order thinking (Factor 2).90 1, 2, 4, 14, 15 Learning/curricular goals (Factor 3).86 3, 7 Conceptually weak (Factor 4).74 B 27 to 34 Diverse learners.95 C 35 to 43 Classroom environment.95 D 47 to 51 Professional responsibilities.90 Factors that emerged from the analysis of Part A had lower alpha scores than factors from the 2012 analysis; particularly learning/curricular goals (Factor 3) as well as the conceptually weak Factor 4. Alpha coefficients for all of Part A as well as B, C, and D were similar to those from the previous analysis, supporting the adequate internal consistency but indicating some potentially repetitive items as previously mentioned. Correlations between Surveys In many instances, the items on the 2013 Supervisor Survey and the items on the 2013 Transition to Teaching Survey (TTS) Part C were virtually identical, as planned by the Common Metrics Group. In addition, the four main categories for these two instruments were conceptually the same: Instructional Practice, Diverse Learners, Classroom/Learning Environment, and Professional Responsibilities. In order to provide information regarding whether these two instruments are measuring similar constructs, bivariate correlations were used. These correlations provide information regarding whether or not there are linear relationships between two variables. The following steps were taken for this analysis. First, scale scores were calculated for each respondent for each instrument, based on the four categories listed above. The mean of all of the items for each category was calculated to result in this scale score. Responses of Unable to

10 Respond or Does Not Apply were coded as zero and excluded from the mean scale score calculations. For the TTS, Cronbach s alpha was calculated on these four scales to ensure they were appropriate for use in this analysis. As alpha values ranged from.75 to.97, these scales were considered adequate. Only respondents who had scale scores for both the TTS and the Supervisor Survey were included in this analysis, resulting in an n of 165. Bivariate correlations were then calculated to analyze the relationships between the corresponding scale scores for each instrument. According to Cohen (1988), correlation coefficients between.3 and.49 represent a moderate correlation between two variables. Coefficients from.5 to 1.0 are strong correlations. While Instructional Practice and Learning/Classroom Environment were significant at the.05 level, correlations were all very low (Table 7). Scatterplots were used to further examine the relationships between these variables. Despite the statistical significance, there were no visible relationships among the variables, indicating that a person s score on the TTS was not strongly related to their score on the same construct on the Supervisor Survey. Table 7. Correlations between Supervisor Survey and TTS Scales Transition to Teaching Survey Instructional Practice Diverse Learners Learning Environment Instructional Practice.185* Diverse Learners.142 Classroom Environment.200* Professional Responsibilities *p <.05 Supervisor Survey Professionalism.040 There are many potential reasons why these scales were not strongly correlated with each other. First, new teachers perceptions of their own performance, due to their relative inexperience, may simply be different from those of their more experienced supervisors. Second, the use of a 1 to 4 scale on both measures potentially reduces the sensitivity of the instruments. Increasing this to a 5- or 6-point scale may increase sensitivity, and result in different response patterns. Third, these four scales were chosen to match the Supervisor Survey, but the factor analysis of the 2012 TTS indicated 10 factors including instruction for higher-order thinking; classroom environment and behavior management; diverse learners and diverse learning; variety in instructional practices; facilitating effective learning environments; collaboration; alignment of instruction; increased understanding; meeting students needs; and one with weak conceptual alignment. TTS items not forming these four scales as anticipated could also be the cause of low correlations. Finally, while the items are conceptually similar on the two instruments, they may not measure the same constructs. Expert review of the items themselves would be needed to determine how the scales compare between the two instruments.

11 Conclusions The analysis presented here suggests that the 2013 Supervisor Survey has relatively good reliability as a measure of these constructs. Minor adjustments, as discussed in the recommendations for each part, could potentially increase the validity of the instrument. Further analysis, including focus groups and expert review of content, could provide valuable additional information to further improve this instrument. While statistical analysis can provide valuable information to researchers, decisions about the construction of the instrument must be made by the Common Metrics Group while considering the purpose and goals for the research project involved. While the analyses presented here are based on the particular data, sample, and instrument administration relative to this study, administering the instrument to a different sample at a different time could produce different results. The suggestions made here are intended to increase the reliability and validity of data gathered with this instrument, but additional analyses on data obtained from revised versions of the instrument would be necessary to determine how any changes impact validity and reliability. References Babbie, E. (1990). Survey research methods (2nd ed.). Belmont, CA: Wadsworth. Cohen, J. (1988), Statistical power analysis for the behavioral sciences (2nd ed.), New Jersey: Lawrence Erlbaum Associates Fowler, F. J. (2009). Survey research methods (4th ed.). ASRM Series, Vol. 1. Thousand Oaks, CA: Sage. Leech, N. L., Barrett, K. C., & Morgan, G. A. (2011). IBM SPSS for intermediate statistics use and interpretation (4 th ed.). New York, NY: Routledge.

12 Summary of 2013 Supervisor Survey Findings and Recommendations Findings for Part A Twelve of the items loaded strongly on one factor (5, 6, 8, 9, 11, 12, 13, 16, 17, 18, 19, and 20), potentially forming a scale related to a teacher s ability to design instruction based on standards, individual student s abilities, and students needs. Seven items loaded strongly on a second factor (10, 21, 22, 23, 24, 25, and 26), potentially resulting in a scale related to encouraging students higher-order thinking skills. These results were similar to the 2012 instrument validity findings. Five items loaded on a third factor (1, 2, 4, 14, and 15), primarily related to learning and curricular goals. Two items loaded onto a fourth factor (3 and 7). This factor had a weak conceptual alignment which may be due to item phrasing. Recommendations for Part A Use consistent wording to help items load on the factors more consistently with their conceptual meanings. Define terms to help to ensure that items are interpreted by respondents as intended by the researchers. Findings for Parts B to D For each of these parts, all items loaded strongly one factor. This suggests three scales, one for each part: diverse learners, classroom environment, and professional responsibilities. Recommendations for Part B Remove items that many respondents were unable to answer. Rephrase items by using more generic terms to describe broader segments of the student population to improve response rates. Recommendations for Part C Group items with similar constructs to promote ease of completion. Revise items to ensure each item poses one and only one question. Recommendations for Part D Provide additional information in parentheses for terms that may be inconsistently understood. This document is intended to provide information with respect to the reliability and validity of the instrument based on research, not to advocate for the implementation of any specific changes or for any particular timeframe. These decisions should be made by the entire Common Metrics Group and are beyond the scope of this document.