Identifying Extraneous Threats to Test Validity for Improving Tests

Transcription

1 Identifying Extraneous Threats to Test Validity for Improving Tests Yi Du, Ph.D. Data Recognition Corporation Presentation at the 40th National Conference on Student Assessment, June 22,

2 If only it were so easy, we don t need testing professionals in this industry 2

3 State assessment designs may leave room for extraneous variable effects It may not be realistic that using a strictly experimental design which has randomized samples, pre- and post-tests, and comparison groups for a statewide assessment. Statewide assessments, in general, are based on an one sample post-test only design. This pre-experimental design provides the least amount of control and allows much room for extraneous variables possibly affecting the assessment outcomes. 3

4 An Example of Variable Map of An Assessment Age of student Gender of student Students SES X A Curriculum and textbook X B Instruction Approaches School and Classroom Environment O Classroom learning Y Student Math Achievement Students Ethnicity X C Students prior knowledge and skills 4

5 More Efforts Have Been Made in Providing Evidence on Instrument Validity Construct - consistency across concept, construct and instrument. Criterion-predictive - how well performance on one instrument predicts score on a second but different one at a future date. Criterion- concurrent - consistency between scores on a measure and a second independent instrument or classification. Content - adequacy of instrument in representing subject matter and skills. 5

6 Less Efforts Were Made in the Entire Test Process for Test Score Validity External Validity ensuring that both the samples and the test conditions under which the test is carried out are representative of the population and the situations to which the results are to apply; Statistical validity the appropriate choice of statistical methods, which is dependent on decisions made about measuring instruments and influenced by data collection process. Internal validity ensuring the design allows for the control of all other possible contributing of extraneous variables. 6

7 Test Score Validity May Be Beyond the Instrument Validity Planning Stage External Validity Statistical Validity Execution Stage State questions & hypotheses, identify variables Determine design structure Identify population & sample Design instruments & classify: operational definitions Select statistical test for resolving hypotheses Carry out plan, collect data Analyse data, draw conclusions & evaluate process Internal Validity Construct Validity 7

8 Possible extraneous variables in test process may affect test score validity Extraneous variables - those that could be competing independent variables, influencing the dependent variable, but are not of interest to the assessment. Item drifting Rater drifting Test/item overexposure Different choice of statistical/measurement tools Different administration approaches Test design effect 8

9 Possible Extraneous Variables Confounding in the Testing Confounding - occurs when the cause of the effect seen in the dependent variable is not unequivocally attributable to the independent variable alone. School program effectiveness may be confounded with the composition of students and faculty Students ethnicity background may be confounded with students social economic status Students scores on CR questions in social studies may be confounded with students writing levels. 9

10 Possible Identified Extraneous Variables Identified variables- those we know about that will affect the dependent variable. Students background: gender, age, education, social class. School characteristics Curriculum and instruction approaches 10

11 Possible Extraneous Variables Introduced by the Testing Itself Sampling: Sample representativeness Sample stability loss over time Time delay or ahead reduce sample quality Poor sampling procedures or non-random sampling Underestimate the sampling error variance or sample size with application of incorrect formulas, i.e., stratified sampling by using simple random sampling formulas Measuring growth with students loss over time Choices of analysis methods and procedures The impact of design effects The impact of equating error variance (Gary Phillips, NCME 2010) 11

12 Evidence of Extraneous Variable Threats (1) Rater Drifting 12

13 Rater Drifting from A Statewide Grade 8 Mathematics Assessment CR Anchor Item N 2008 Score 2009 Score Correl. Mean Dif SD Dif t-test Prob. Item Item These scores were from the same students responding to the same CR items. The only difference is the rater groups. It seems that raters in 2008 were more harsh than those in

14 Student Scores with Different Years of Raters 2009 Score Total 2008 Score Total

15 Comparison of the Mean Scale Score With and Without Rater Effect Adjustment in the Linking Linking with Rater Effect Adjustment Linking without Rater Effect Adjustment Mean SD Mean SD The mean scale scores were higher with rater effect adjustment than non-rater effect adjustment. 15

16 Proportion Difference (%) in Performance Level Classification N = (1.7%) students were affected Linking with Rater Effect Adjustment (Spring Operation) BB B P A Liking without Rater Effect Adjustment BB 12.8 B P 26.6 A

17 Rater Drifting from a Statewide Writing Assessment 2009 Operational Score Adjustment None Difficulty Rater Both 2008 Statistics N 50,000 50,000 50,000 50,000 50,000 % in Performa nce Level Equating Constants Mean SD Level Level Level Level Mult Add

18 Rater Drifting Rater Stringencies Different raters over years Possible different rater training over years Same raters change due to environmental factors Same raters change due to psychological factors 18

19 When Rater Drifting Occurs? Use CR items as anchors for over-year equating Use intact form which includes CR or essay items Use pre-equated forms which include CR or essay items 19

20 How to effectively identify and monitor rater drifting Use MC items only as anchors in equating Use trend score method (Tate, 1999) Rescore CR items or essays and add rater shift onto the equating constant Use auto-scoring? 20

21 Evidence of Extraneous Variable Threats (2) Item Drifting 21

22 Item Drifting Occurs Environmental factors Different curriculum and instructions approaches Different learning practices School or home environmental changes Societal environment Testing factors Item over-exposure Anchor items or items in intact forms were edited 22

23 Edits to Intact Forms or Anchor Items Item Used in a Spring Operational Test The first time that Linda saved someone s life, she knew they needed her help because A. a storm was coming. B. she heard yelling in the street. C. she saw signal lights from home. D. she was patrolling the harbor in a boat. Revised Item Used in the Later Year In 1990, when Linda saved someone s life, she knew they needed her help because A. a storm was coming. B. she heard yelling in the street. C. she saw signal lights from home. D. she was patrolling the harbor in a boat. 23

24 The Edits Reflected on the Raw-to-Scaled Score Table Raw Score Original RS-SS Table Scale Score SS SEM Updated RS-SS Table Scale Score Standard Error Difference

25 Evidence of Extraneous Variable Threats (3) Sampling 25

26 Sample Differences Result In Different Raw-to- Scale Score Tables Form A Form B Diff Sample N Median Mean SD N Median Mean Old Sample 4, , New Sample 4, , SD Median Mean (SE) *

27 Partial Raw-to-Scaled Score Tables for Old and New Samples Old Sample New sample Differences Form A Form B Form A Form B Old Sample New Sample

28 Evidence of Extraneous Variable Threats (4) Test Design Effect Slides # 35 through #37 were from and modified based on Score Drift and Design Effect by G. W. Phillips at NCME

29 How does the design effect impact estimates of sample size (n)? One of design effects is in the spiraling of field test forms associated with field testing. Let s assume that we have 12 field test forms that will be administered to 12,000 students in the state. If we could use a simple random sample (SRS) we would draw 12,000 students from across the state and administer each form to a random sub-sample of 1000 students. The standard error of the mean for each form would equal 5 scaled score units as Design 1 shows. 29

30 How does the design effect impact estimates of sample size (n)? The effect of clustering in field test samples Spiraling Design Forms are administered SE of the Mean Effective Sample Size and needed sample Design 1 (SRS) Random statewide sample /1000 Design 2 (cluster) Design 3 (cluster) Spiraled by students within classrooms Spiraled by classrooms within schools / / 4200 Design 4 (cluster) Spiraled by schools within districts /20800 Design 5 (cluster) Spiraled by districts /

31 How does the design effect impact estimates of sample size (n)? When we do sampling at the group level as designs 2 through 5, we need to add the cluster variance into the error variance. If we use the formulas used for SRS for sample size or sampling errors in the cluster sampling, we actually underestimate the error variance as the table shows. Consequently, their statistics (e.g., p-values, pointbiserials, different item function statistics, item parameter estimates, means, correlations, equating error, etc) have substantially more error associated with them. 31

32 How would we be aware of extraneous variables threats to the tests Be careful with every step of assessment process: Test design: Be careful with post-equating designs and pre-equating designs Rater drifting For anchor items For pre-equating forms For intact forms Be careful of editing anchor items and intact forms Be careful of changing the testing time, content subject sequence, and test format change over years Be careful of the choice of statistical methods 32

33 How would we be aware of extraneous variables threats to the tests Be careful of Sampling: Sample representativeness Poor sampling procedures or non-random sampling Appropriately estimate sample size or standard errors Be careful of not only the choice of statistical methods, but also different computation software, and different versions of the same software 33

34 What A Testing Professional Can do? We have responsibilities: statistical methods/tools can only solve the problems that we want to solve. It is up to the test professionals to construct the test that takes into account possible extraneous variables. We need to ensure the score validity as much as possible. More possible extraneous variables are accounted for, we actually ensure the validity of the test better. Find the balanced between the best and the ideal plan. In the real testing world with limited control of extraneous variables, we may have to realize some extraneous variables are controllable but not all of them. Thus, we need a balanced between the best plan and the ideal plan. 34

35 Thank you for your listening and your time! 35