Reliability and Validity checks S-005

Size: px

Start display at page:

Download "Reliability and Validity checks S-005"

Chloe Poole
5 years ago
Views:

1 Reliability and Validity checks S-005

2 Checking on reliability of the data we collect Compare over time (test-retest) Item analysis Internal consistency Inter-rater agreement

3 Compare over time Test-Retest reliability One sample at two (or more) times Very convincing in theory Often hard to do in practice Time interval? Memory effects? Special sample? Correlation of time-1 answers with time-2 answers Other approaches are often approximations of this idea Time 1 Time 2 Consistent? How strongly correlated?

into two sets (two halves) Correlate the scores on the two halves Often adjusted

4 Split-half reliability Easier than test-retest checks Requires only one time point Works when there is a scale or set of questions on a single topic Divide the items into two sets (two halves) Correlate the scores on the two halves Often adjusted by the Spearman-Brown correction Gives us an estimate of the test-retest reliability

5 Item-analysis approaches When there is a set of questions about a single topic Examine the answers to each item How many answered correctly? Percent correct? Item difficulty. Or, if there is no correct answer, look at how the answers were distributed Agree / neutral / disagree Examine the wrong answers that are chosen Find items that are too hard or too easy Or those that have little variability (too boring? too trivial?) Do you really need these? Sometimes these are very important Test publishers tend to delete the easy and hard items Correlate the item responses with the total responses High correlations indicate consistency Low correlations indicate different or weak items Negative correlations indicate something interesting Confusing wording? The item doesn t belong?

An item analysis Report for test questions Test developers and test publishers use some of these ideas for deciding which items should stay on their tests.

6 An item analysis Report for test questions Test developers and test publishers use some of these ideas for deciding which items should stay on their tests. Number not answering Skips or blanks Confusion about wording Too long or too difficult Item difficulty Item discrimination Check the distractor options (the wrong answers) What proportion answered the item correctly?.50 (50%) (seems about right).90 (90%) too easy?.12 (12%) too hard? The degree to which the items differentiate between those who know the material well and those who do not know the material well Compare top 25% with bottom 25% (e.g., top 25% score 80%, while bottom 25% score 40%) Item-total correlation coefficient correlation of item score with score on the entire test Are the wrong answers too obvious? Too silly? Are they really wrong? What percentage of students choose each option?

7 Internal consistency reliability When there is a scale or set of questions on a single topic Cronbach s coefficient alpha a measure of internal consistency Look at all of the items Check the average correlation Then adjust for the number of items Find items that do not correlate with others Check the item-total correlations If low, delete these or move them elsewhere Assess the overall internal consistency

8 Internal consistency reliability Comparing answers from different sources Compare similar questions that appear in different parts of the questionnaire Compare answers from different places during an interview Compare interview responses with questionnaire responses Compare questionnaires with actual observations

9 Inter-rater agreement Useful in checking on coding open-ended answers, observations, etc. We often try this on a sample or during a pilot study Check the overall percent agreement Sometimes we adjust for chance agreement -- Cohen s Kappa A very important step in lots of studies If agreement is high, then okay to rely on one primary coder or rater If not high, then perhaps we need more than one rater Or perhaps we need to revise or clarify the coding rules Then check on things again There are often several iterations here Keep going until the agreement is acceptable

10 Check out some examples Bayley Scales of Infant Development Inter-rater agreement example Internal consistency example Then try some clicker questions!

11 Observing students and teachers in classrooms. What type of reliability check is most important? 1. Inter-observer agreement (have more than one observer) 2. Time 1 - Time 2 (Observe at two or more times) 3. Consistency within the classroom sessions 4. Other 25% 25% 25% 25%

12 Coding transcripts from individual interviews What type of reliability check is most helpful? 1. Have multiple transcribers 2. Inter-rater agreement 3. Internal consistency checks 4. Other 0% 0% 0% 0%

13 Using answers from questionnaires. What type of reliability check is most important? 1. Inter-rater agreement 2. Internal consistency checks 3. Item-analysis checks 4. Other 0% 0% 0% 0%

14 Using a mix of open-ended and closed-ended questions on a questionnaire. Why is this a good idea? 1. Internal consistency checks 2. Makes replying less boring 3. Terry has said this about 50 times, so it must be a good idea 4. Other 5. All of the above 0% 0% 0% 0% 0%

Validity and reliability of measurements

Validity and reliability of measurements 2 Validity and reliability of measurements 4 5 Components in a dataset Why bother (examples from research) What is reliability? What is validity? How should I treat