So far INFOWO Lecture M5 Homogeneity and Reliability Peter de Waal Department of Information and Computing Sciences Faculty of Science, Universiteit Utrecht Descriptive statistics Scores and probability distributions Hypothesis testing t-tests Correlation and prediction Measurement scales Survey Research Operationalisation Homogeneity and reliability Lecture M5: 1 / 26 Lecture M5: 2 / 26 Homogeneity Homogeneity Scores on a construct are determined indirectly: Literature: Chapter 7 (AB Book!) Topics Usefulness of a scale for a construct Importance of internal consistency Calculation of Cronbach s α SPSS Procedure Interpretation of analysis Construction of scale Score on a construct (latent variable) is constructed from: Indicator variables (hopefully relevant aspects of construct) Questions: To which degree are the answers of one person related? Are they internally consistent? To which degree are indicator variables homogeneous? Cronbach s α: Measure for homogeneity or internal consistency reliability Note: Cronbach s α has nothing to do with a significance level α! Lecture M5: Homogeneity Introduction 3 / 26 Lecture M5: Homogeneity Constructs 4 / 26
Construction of a scale for a construct 1 Recode negative questions into positive scales. 2 Make range of scales equal (e.g. all Likert 7 scales) 3 Investigate frequencies, missing values, odd values 4 Select a number of items to include in test scale, e.g. X, Y, Z. 5 Determine level of homogeneity: Calculate correlations Calculate Cronbach s α 6 If Cronbach s α value is not sufficient, then go back to Step 3. 7 If Cronbach s α value is sufficient, then construct a test scale using the selected items. Homogeneity: example In the INFOWO 2015 Questionnaire we asked the following questions, hoping to measure the construct Team attitude : I have good contacts with my fellow students I find working in teams hard (recode!) I always have a fellow student to fall back on if needed I prefer team assignments and tasks over individual assignments I d rather be assessed individually than in groups (recode!) I am comfortable with working in teams Everything was measured by Likert scale: I have good contacts with my fellow students strongly disagree 1 2 3 4 5 6 7 strongly agree Lecture M5: Homogeneity Constructs 5 / 26 Lecture M5: Homogeneity Constructs 6 / 26 Determine level of homogeneity: Correlation Determining consistency or homogeneity is based on correlations Correlation Measures and describes the strength and direction of a relationship between two variables Strong positive correlation between X and Y: Larger values of X are most often paired with larger values of Y. Strong negative correlation between X and Y: Larger values of X are most often paired with smaller values of Y. Weak correlation: Anything but the two strong correlation situations. Correlation: Formula s Correlation r = degree to which X and Y vary together cov(x, y) = degree to which X and Y vary separately s x s y Covariance = degree to which X and Y vary together cov(x, y) = SP n XY n 1 = i=1 (X i X)(Y i Y) n 1 Standard deviation = degree to which X and Y vary separately s x = SSX n 1 = n i=1 (X i X) 2, s y = n 1 Always: 1 r +1. (Recall Statistics Lecture S5) SSY n 1 Lecture M5: Homogeneity Correlation 7 / 26 Lecture M5: Homogeneity Correlation 8 / 26
Correlation: calculation in SPSS Analyze > Scale > Reliability Analysis > Statistics Correlation: calculation in SPSS (recoded variables) Analyze > Scale > Reliability Analysis > Statistics Lecture M5: Homogeneity Correlation 9 / 26 Lecture M5: Homogeneity Correlation 10 / 26 Determine level of homogeneity: Cronbach s α Cronbach s α: interpretation Measure for average item-to-total correlation : k: the number of items / variables used for the construct Cronbach s α = ( k ) s 2 1 item k 1 s 2 scale Standardised: when the variances of the individual items are approximately equal: Cronbach s α = k r 1 + (k 1) r, with There is no known statistical test to evaluate the significance of the value of Cronbach s α Rule of thumb: Value of α Qualification Decision > 0.80 Good Use scale to analyse and report 0.70 0.80 Adequate Use scale to analyse and report 0.60 0.70 Moderate May use scale, provide proper motivation < 0.60 Insufficient Use individual items and report where r is the mean correlation between the variables Lecture M5: Homogeneity Cronbach s α 11 / 26 Lecture M5: Homogeneity Cronbach s α 12 / 26
Homogeneity: Cronbach α In SPSS: Menu Analyze > Scale > Reliability Analysis > Statistics Homogeneity: Cronbach α In SPSS: Menu Analyze > Scale > Reliability Analysis > Statistics Also: click Statistics... Lecture M5: Homogeneity Cronbach s α 13 / 26 Lecture M5: Homogeneity Cronbach s α 14 / 26 Homogeneity: Interpretation of output Cronbach s α Homogeneity: Interpretation of output Cronbach s α Scale if question 1 deleted? (0.720) Scale if question 3 deleted? (0.783) So delete question 3 to improve! Suggestions for improvement? Delete question 1 to improve!.783.789 5 Lecture M5: Homogeneity Cronbach s α 15 / 26 Lecture M5: Homogeneity Cronbach s α 16 / 26
Homogeneity: Interpretation of output Cronbach s α Suggestions for improvement? No way to improve!.788 Lecture M5: Homogeneity Cronbach s α 17 / 26 Scale construction Choose meaningful combination of items (with sufficient Cronbach s α value) Construction of a scale: SPSS: Menu > Transform > Compute variable Score = question1 + question 2 + question 3 + + question k or Score = mean(question 1, question 2, question 3,..., question k ) Note that all questions must have the same Range, or be re-scaled to have the same Range. Re-use of scales from literature: You do not need to worry about homogeneity, but you may want to check it anyway. Use the exact same procedure, if you want to compare with others. (Compare the Rosenberg Self Esteem Scale with the 2015 Research questionnaire. Lecture M5: Homogeneity Cronbach s α 18 / 26 Homogeneity: example 2 Example 2: SPSS correlations We also asked some questions about a General happiness construct: Generally I am a happy person I am satisfied with the life I lead I often feel lonely (recoded) Life is hard (recoded) You can create your own luck I feel fine Lecture M5: Homogeneity Cronbach s α 19 / 26 Lecture M5: Homogeneity Cronbach s α 20 / 26
Homogeneity: Interpretation of output Cronbach s α Homogeneity: Interpretation of output Cronbach s α Suggestions for improvement? Construct from three questions Lecture M5: Homogeneity Cronbach s α 21 / 26 Lecture M5: Homogeneity Cronbach s α 22 / 26 Remark Homogeneity: Cronbach s α remarks Likert is an ordinal scale, not interval nor ratio scale, but An abstract construct created from a collection of Likert scale questions may sometimes be considered an interval scale. Number of items: Too many: Cronbach s α high automatically. Too few: Are you constructing a scale? Norm: 4-6 items If α remains small (lack of homogeneity): Do not construct scale, but... Consider items individually Features of items Degree of difference Positive and negative questions: Advantage: variation Disadvantage: double negations! Do not forget to recode Range of the scale Factor analysis Much more subtle For more info: INFOARM course Lecture M5: Homogeneity Cronbach s α 23 / 26 Lecture M5: Homogeneity Cronbach s α 24 / 26
Reliability Lessons learnt With Cronbach s α we are computing consistency between measurements Use this between measurements of the same variable measured at different points in time. Homogeneity: What is it? How to use it? Gives you an indication of the reliability of a measurement. Lecture M5: Homogeneity Cronbach s α 25 / 26 Lecture M5: Homogeneity Conclusion 26 / 26