Reliability 2010 Internal Consistency Reliability - Internal Consistency

Size: px

Start display at page:

Download "Reliability 2010 Internal Consistency Reliability - Internal Consistency"

Wilfrid Cunningham
5 years ago
Views:

1 Reliability 2010 Internal Consistency Reliability - Internal Consistency How well do NSSE Deep scales measure a single latent construct? Purpose Based on the NSSE Psychometric Reliability Framework (2011), one way to estimate reliability, specifically the internal consistency, of NSSE results is by calculating Cronbach s alphas and intercorrelations for NSSE scales. The internal consistency of a set of items is an indicator of how well the items measure the same variable or construct. Assuming the scales effectively measure an underlying construct, we would expect to find high estimates of their internal consistency. Data In 2010, all randomly sampled students from participating U.S. institutions were included in this analysis. Results of all analyses were weighted by gender, enrollment status, and institution size. Because students need to respond to approximately three-fifths of the items within a scale to receive a scale score, the number of students used varied by class and scale. The numbers of respondents analyzed by class can be seen in Table 1. Table 1 Number of Respondents by Class Senior Higher Order 153, ,103 Integrative 157, ,370 Reflective 150, ,802 Deep Scale 153, ,269 Analyses were also run by gender, nine categories of student-reported major, and institution type. Numbers of respondents in these various subgroups can be seen in Table 8, Table 11, and Table 14, respectively, at the end of this document. Methods Cronbach s alpha measures the internal consistency of a group of items by measuring the homogeneity of the group of items it is an indication of how well the different items complement each other in their measurement of different aspects of the same variable or quality (Litwin, 2003, p. 22). Cronbach s alpha ranges in value between zero and one. Values closer to one indicate a higher internal consistency; values closer to zero indicate a lower internal consistency. McMillan and Schumacher (2001) suggest This report is part of NSSE s Psychometric Portfolio, a framework for presenting our studies of the validity, reliability, and other indicators of quality of NSSE data, available online at nsse.iub.edu/links/psychometric_portfolio. Updated 2/15/2011

2 that groups of items with an alpha below.70 should be used with caution. The internal consistency of a scale can also be examined with item-to-scale correlations and intercorrelations of items within a scale (DeVellis, 2003). If a group of items measures a single latent construct, we would assume that each item alone correlates with the scale overall and that items within such a scale are positively correlated. According to Clark and Watson (1995), the average inter-item correlation should fall somewhere between.15 and.50. A potential limitation of the results in this analysis is that higher correlations result from more heterogeneous groups of respondents, and the correlations reported here could only be generalized to students with characteristics similar to the students analyzed here. It is possible that the alphas and correlations for other subpopulations of students could be very different. Results Cronbach s alphas for the NSSE Deep scales and average inter-item correlations by class can be found in Table 2 below. Scale intercorrelations can be found in Table 7 at the end of this document. These results indicate a high degree of reliability for the overall Deep scale as well as the three Deep subscales. These Cronbach s alphas range between.702 for the first-year Integrative subscale and.859 for the senior overall Deep Scale. Overall, we see that Cronbach s alphas for seniors are slightly higher than for first-year students. The average inter-item correlations for the Integrative subscale and Deep scale fall within acceptable levels, but the correlations for the Higher Order subscale and Reflective subscale are a little high. For all three subscales and the overall Deep scale, the removal of any single item will lower the scale s overall Cronbach s alpha, and all of the corrected item-scale correlations are positive and high (see Table 3 through Table 6). All of these results suggest that these are reliable NSSE scales. Table 2 Scale Cronbach s Alphas by Class NSSE Scales Cronbach s α Senior Cronbach s α Average Inter-Item Correlation Senior Average Inter-Item Correlation Higher Order Integrative Reflective Deep Scale Cronbach s alphas for seniors are slightly higher than for first-year students across all of the Deep subscales and the Deep scale across gender, students self reported major category, and institutional Carnegie classification. Results from these subpopulation analyses show that differences by gender (Table 9), student-reported major category (Table 12), and institutional Carnegie classification (Table 15) are all trivial. Some minor trends could be detected such as female Cronbach s alphas tend to be higher for the Higher Order and Reflective subscales, alphas tend to be higher for business students, and alphas tend to be higher for students at Baccalaureate-Diverse institutions. All of these differences, however, are quite small. With few exceptions, Cronbach s alphas for all of these various subpopulations were at or above the.70 standard

3 Although there are trivial differences in the Cronbach s alphas across various subpopulations of firstyear and senior students, there are some very clear overarching trends across all of the analyses presented here. All three Deep subscales and the overall Deep scale have consistently high indications of reliability. References Clark, L. A. & Watson, D. (1995). Constructing validity: Basic issues in objective scale development. Psychological Assessment, 7, DeVellis, R. F. (2003). Scale development: Theory and applications (2 nd Edition). Thousand Oaks, CA: Sage Publications, Inc. Litwin, M. S. (2003). How to assess and interpret survey psychometrics, 2 nd edition. Thousand Oaks, CA: Sage Publications, Inc. McMillan, J. H. & Schumacher, S. (2001). Research in education: A conceptual introduction. New York: Longman. NSSE Psychometric Portfolio. (2011). Bloomington, IN: Center for Postsecondary Research, Indiana University, School of Education. nsse.iub.edu/links/psychometric_portfolio NSSE Psychometric Reliability Framework. (2011). Bloomington, IN: Center for Postsecondary Research, Indiana University, School of Education. nsse.iub.edu/_/?cid=

4 Additional Tables Analyses by Class Table 3 Higher Order Item-Scale Analyses Senior Corrected 2 Scale Item 1 Senior Cronbach s Cronbach s Alpha if Item Deleted Alpha if Item Deleted Corrected 2 Item- Scale Correlation Item-Scale Correlation analyze synthesz evaluate applying Table 4 Integrative Item-Scale Analyses Scale Item 1 Cronbach s Alpha if Item Deleted Senior Cronbach s Alpha if Item Deleted Corrected 2 Item- Scale Correlation Senior Corrected 2 Item-Scale Correlation integrat divclass intideas facideas oocideas Table 5 Reflective Item-Scale Analyses Scale Item 1 Cronbach s Alpha if Item Deleted Senior Cronbach s Alpha if Item Deleted Corrected 2 Item- Scale Correlation Senior Corrected 2 Item-Scale Correlation ownview othrview chngview Questions and response options for each scale item can be found in the 2010 NSSE Codebook 2 Corrected scale correlations are correlations between the item and the scale if the item is removed - 4 -

5 Table 6 Deep Item-Scale Analyses Senior Corrected 5 Scale Item 3 Senior Cronbach s Cronbach s Alpha if Item Deleted Alpha if Item Deleted Corrected 4 Item- Scale Correlation Item-Scale Correlation analyze synthesz evaluate applying integrat divclass intideas facideas oocideas ownview othrview chngview Table 7 Deep Intercorrelations by Class HOL INT RFL HOL INT RFL First-year intercorrelations are in the bottom, shaded half. Senior intercorrelations are in the upper, unshaded half. All p s < Questions and response options for each scale item can be found in the 2010 NSSE Codebook 4 Corrected scale correlations are correlations between the item and the scale if the item is removed - 5 -

6 Analyses by Class and Gender Table 8 Number of Respondents by Class and Gender Males Females Senior Males Senior Females Higher Order 68,674 84,421 80, ,955 Integrative 70,768 86,813 81, ,716 Reflective 67,577 83,247 79, ,730 Deep Scale 68,812 84,530 80, ,040 Table 9 Deep Cronbach's Alphas by Class and Gender NSSE Scales Male α Female α Senior Male α Senior Female α Higher Order Integrative Reflective Deep Scale Table 10 Deep Intercorrelations by Class and Gender Males Females HOL HOL INT INT RFL RFL First-year intercorrelations are in the bottom, shaded half. Senior intercorrelations are in the upper, unshaded half. All p s <

7 Analyses by Class and Major Table 11 Numbers of Respondents by Class and Major Higher Order Integrative Reflective Deep Scale s Arts & Humanities 16,280 16,295 16,273 16,298 Biological Sciences 12,509 12,526 12,517 12,526 Business 20,309 20,356 20,349 20,363 Education 11,860 11,873 11,877 11,878 Engineering 9,958 9,974 9,949 9,976 Physical Science 4,825 4,831 4,830 4,831 Professional 16,365 16,381 16,362 16,387 Social Science 16,492 16,508 16,505 16,509 Other 23,155 23,199 23,195 23,202 Seniors Arts & Humanities 22,424 22,453 22,448 22,448 Biological Sciences 12,606 12,616 12,596 12,617 Business 33,162 33,198 33,173 33,196 Education 15,955 15,963 15,954 15,964 Engineering 10,725 10,731 10,724 10,732 Physical Science 5,486 5,489 5,491 5,494 Professional 17,118 17,125 17,116 17,129 Social Science 24,152 24,155 24,156 24,157 Other 32,304 32,339 32,324 32,

8 Table 12 NSSE Scale Cronbach's Alphas by Class and Major Higher Order Integrative Reflective Deep Scale s Arts & Humanities Biological Sciences Business Education Engineering Physical Science Professional Social Science Other Seniors Arts & Humanities Biological Sciences Business Education Engineering Physical Science Professional Social Science Other

9 Table 13 NSSE Deep Scale Intercorrelations by Class and Major Arts & Humanities Biological Sciences HOL HOL INT INT RFL RFL Business Education HOL HOL INT INT RFL RFL Engineering Physical Science HOL HOL INT INT RFL RFL Professional Social Science HOL HOL INT INT RFL RFL Other HOL INT RFL HOL INT RFL First-year intercorrelations are in the bottom, shaded half. Senior intercorrelations are in the upper, unshaded half. All p s <

10 Analyses by Class and Institutional Carnegie Classification Table 14 Numbers of Respondents by Class and Institutional Carnegie Classification Higher Order Integrative Reflective Deep Scale s 1 Doc RU-VH 21,438 22,055 21,135 21,470 2 Doc RU-H 27,462 28,234 27,056 27,503 3 Doc DRU 11,168 11,514 11,011 11,187 4 Masters-L 41,030 42,308 40,387 41,098 5 Masters-M 14,586 15,011 14,397 14,619 6 Masters-S 5,224 5,386 5,147 5,230 7 Bac-AS 13,507 13,814 13,379 13,522 8 Bac-Diverse 9,732 10,046 9,542 9,753 9 Other 8,948 9,212 8,770 8,960 Seniors 1 Doc RU-VH 30,062 30,739 29,641 30,086 2 Doc RU-H 39,943 40,640 39,428 39,974 3 Doc DRU 12,608 12,835 12,439 12,625 4 Masters-L 52,404 53,315 51,759 52,454 5 Masters-M 15,848 16,083 15,666 15,861 6 Masters-S 5,709 5,785 5,654 5,710 7 Bac-AS 13,646 13,839 13,534 13,661 8 Bac-Diverse 8,678 8,789 8,580 8,686 9 Other 8,206 8,345 8,100 8,

11 Table 15 NSSE Deep Scale Cronbach's Alphas by Class and Institutional Carnegie Classification Higher Order Integrative Reflective Deep Scale s 1 Doc RU-VH Doc RU-H Doc DRU Masters-L Masters-M Masters-S Bac-AS Bac-Diverse Other Seniors 1 Doc RU-VH Doc RU-H Doc DRU Masters-L Masters-M Masters-S Bac-AS Bac-Diverse Other

12 Table 16 NSSE Deep Scale Intercorrelations by Class and Institutional Carnegie Classification Doc RU-VH Doc RU-H HOL HOL INT INT RFL RFL Doc DRU Masters-L HOL HOL INT INT RFL RFL Masters-M Masters-S HOL HOL INT INT RFL RFL Bac-AS Bac-Diverse HOL HOL INT INT RFL RFL Other HOL INT RFL HOL INT RFL First-year intercorrelations are in the bottom, shaded half. Senior intercorrelations are in the upper, unshaded half. All p s <

Reliability 2010 Internal Consistency Reliability - Internal Consistency

Reliability 2010 Internal Consistency Reliability - Internal Consistency How well do the NSSE benchmarks measure a single latent construct? Purpose Based on the NSSE Psychometric Reliability Framework