Differential Performance of Test Items by Geographical Regions. Konstantin E. Augemberg Fordham University. Deanna L. Morgan The College Board

Transcription

1 Differential Performance of Test Items by Geographical Regions Konstantin E. Augemberg Fordham University Deanna L. Morgan The College Board Paper presented at the annual meeting of the National Council on Measurement in Education in New York, NY, March 27,

2 Abstract Differential item functioning (DIF) occurs when examinees of the same ability level from different groups have a different probability of answering a given item correctly. DIF analyses have been traditionally used for examining the fairness of tests with regard to specific populations, including but not limited to gender, age, ethnicity, cultural and language subgroups. The objective of this research was to compare the performance of items on a nationally administered test across geographical regions and individual states within the continental United States. The premise of this study is that certain geographically-based dissimilarities among examinees may potentially result in differential performance on certain items across various parts of the US. Particularly, some items may exhibit DIF due to the bias related to cultural differences, such as familiarity with specific content. For instance, it is possible that examinees in the southern parts of the country are less familiar with the words mittens or toboggan than their peers in the northern regions. On the other hand, it is also possible that examinees unequal performance on a given item may reflect true differences in knowledge of the subject, perhaps, due to discrepancies in school curricula or academic achievement across the regions. This study used data from a 2005 administration of a large scale test of Test A, Test B, and Test C. Individual items from the Test A, Test B, and Test C sections were examined for uniform DIF using Mantel-Haenszel (MH) (Mantel & Haenszel, 1959; Holland & Thayer, 1988) and Simultaneous Item Bias (SIBTEST) tests (Shealy & Stout, 1993), two of the most frequently used DIF identification methods. DIF analysis was performed on individual items across geographical regions, using 2

3 the classification denoted by the College Board regional offices in Table 1, and individual states, as indicated on the examinees records. Items were flagged for DIF based on the results of both MH and SIBTEST criteria. The classification of DIF was found to be highly inconsistent across the two detection methods, and final identification was based on the SIBTEST beta criteria. As a result of analyses across geographical regions, 2 items have been identified as exhibiting DIF of B-class. In the case of individual states, 10 items have been identified as functioning differentially with regard to the other states, with DIF of B-class (moderate) or C- class (high) degree of severity. Further analyses, such as examination of item content, and analyses of response patterns, have been performed to identify the possible causes of DIF in some items. Possible implications of findings were discussed, and recommendations for future research in this direction were suggested. 3

4 Differential item functioning (DIF) is observed when comparable (i.e., matched on ability) examinees from different groups have a different probability of answering a given item correctly. Thus, DIF implies that even after controlling for ability, an item appears to be more difficult for examinees from one group, as compared to examinees in other groups. The group, which is suspected to be disfavored by a given item, is called the focal group, whereas the group that is presumably favored by the item is usually referred to as the reference group (Roussos & Stout, 1996). When differences in correct response probability are found across all ability levels, it is said that this particular item has a uniform DIF. It is also possible that there is an interaction between the ability level and group membership, and as a result, DIF appears only on specific ability levels. In this case, it is said that the item exhibits non-uniform DIF (Gierl et al., 2000). Research suggests, however, that non-uniform DIF is rarely encountered in practice (Camilli & Shepard, 1994). Most DIF studies focus on the identification of items with differential performance, often followed by further examination of the flagged items and identification of the potential causes of DIF. Ultimately, DIF may be attributed to either item impact or item bias. Item impact refers to the differences in item performance across groups of interest due to actual differences in knowledge or another ability of interest (Clauser & Mazor, 1998). Item bias, on the other hand, indicates serious errors or flaws in the measurement of ability for members of a specific group (Camilli & Shepard, 1994). Among the most frequently used DIF identification methods are Mantel- Haenszel (MH) Chi-square and Simultaneous Item Bias (SIBTEST) tests. Mantel- 4

5 Haenszel is a non-parametric test that employs chi-square statistic to test the null hypothesis of no relationship between the test performance on a given item and group membership. Examinees are matched on test scores and compared across groups, using contingency tables. In addition to significance testing, it is also possible to classify the magnitude of DIF, using the effect size measure delta. MH delta is a logarithmic transformation of the ratio of the correct responses odds in a reference group to the odds for correct responses in a matched focal group. Provided that the overall test is significant, the absolute value of delta under 1 corresponds to negligible DIF, value of 1.5 and above suggests large degree of DIF, and intermediate values of delta indicate DIF of a moderate degree (Zieky, 1993). Alternatively, the SIBTEST employs the multidimensional DIF model of Shealy and Stout (1993), which looks at the differences in probability of correct responses between focal and reference groups (beta index), after matching respondents on true ability score. The non-zero difference is attributed to the influence of an additional nuisance dimension, and is considered an indication of DIF. The null-hypothesis of non-zero difference is rejected based on the significance level, adjusted for the number of comparisons (usually as.05 divided by the number of items assessed). In addition to the test of significance, SIBTEST produces for each item an estimate of the beta index. The severity of the bias is assessed, using the Stout-Roussos classification of DIF: (1) A-level DIF: the absolute value of beta index is less than.059; (2) B-level DIF: the absolute value of beta index is between.059 and.088, and (3) C-level DIF: the absolute value of beta index is equal or 5

6 higher than.088. The A-, B- and C- class DIF are also categorized accordingly as negligible, moderate, and large (Stout & Roussos, 1996). Sample This study used data from a 2005 administration of a large scale assessment, with all individual items from the Test A, Test B, and Test C sections included in the analyses. The original dataset contained 288,551 records. After removing duplicate entries and records with missing data, for example, records where an item(s) was missing or not reached, missing demographic data or other data needed to confirm geographic region or sample representation, 44,351 records for the Test A section, 92,323 records for the Test B section, and 160,914 for the Test C section were retained for further analyses. The original data set and the cleaned sample were compared by gender and ethnicity subgroup percentages as a way of checking if the sample was representative of the original. Subgroup percentages for the sample differed from the original by not more than 1% in any direction. Each respondent was assigned to one of the seven geographical regions based on his or her state of residence, in accordance with College Board regional offices affiliation (Table 1). Table 1. Geographical Region breakdown by state Region Region Name Constituent States 1 Midwest Illinois, Indiana, Iowa, Kansas, Michigan, Minnesota, Missouri, Nebraska, North Dakota, Ohio, South Dakota, West Virginia, Wisconsin 2 Middle States New York, Pennsylvania, New Jersey, Delaware, Maryland, Washington, D.C. 3 New England Connecticut, Rhode Island, Massachusetts, Vermont, New Hampshire, Maine 4 US Territories Puerto Rico, Virgin Islands 5 South Kentucky, Virginia, Tennessee, North Carolina, South 6

7 Carolina, Mississippi, Alabama, Georgia, Louisiana, Florida 6 Southwest Texas, New Mexico, Oklahoma, Arkansas 7 West California, Oregon, Washington, Alaska, Hawaii, Idaho, Montana, Wyoming, Colorado, Nevada, Utah, Arizona Distributions of respondents by regions are shown in Tables 2-4. Table 2. Respondents by regions, Test A section Region Region Name Frequency Percent Mean Score Standard Deviation 1 Midwest Middle States New England South Southwest West Table 3. Respondents by regions, Test B section Region Region Name Frequency Percent Mean Score Standard Deviation 1 Midwest Middle States New England South Southwest West Table 4. Respondents by regions, Test C section Region Region Name Frequency Percent Mean Score Standard Deviation 1 Midwest Middle States New England South Southwest West

8 Respondents from region 4 (U.S. Territories) were excluded from further analyses, due to the small sample size. Within each region, focal and reference groups were formed, with the focal group consisting of respondents from that particular region, and the reference group consisting of respondents from all other regions not in the focal group. Due to limitations of memory space and running time the reference group was sampled to be 5000 examinees which were representative of the regional makeup of the total sample by selecting randomly within region to match the number of examinees needed to produce the required proportion. Additionally, there was some concern that for regions with smaller sample sizes that the large difference in sample size may bias results and therefore a more similar sample size was desirable. The focal group was randomly selected to produce 5000 examinees where sample size permitted and where sample size was an issue the number of examinees used was the closest number of examinees available rounding every 500 examinees. Focal and reference group distributions for each region are shown in Tables 5-7. Table 5. Sample sizes for DIF analysis - Test A Region Region Name Focal Group (N) Reference Group (N) 1 Midwest 5,000 5,000 2 Middle States 5,000 5,000 3 New England 1,500 5,000 5 South 5,000 5,000 6 Southwest 3,000 5,000 7 West 5,000 5,000 8

9 Table 6. Sample sizes for DIF analysis - Test B Region Region Name Focal Group (N) Reference Group (N) 1 Midwest 5,000 5,000 2 Middle States 5,000 5,000 3 New England 3,500 5,000 5 South 5,000 5,000 6 Southwest 5,000 5,000 7 West 5,000 5,000 Table 7. Sample sizes for DIF analysis - Test C Region Region Name Focal Group (N) Reference Group (N) 1 Midwest 5,000 5,000 2 Middle States 5,000 5,000 3 New England 5,000 5,000 5 South 5,000 5,000 6 Southwest 5,000 5,000 7 West 5,000 5,000 Sampling for by State Comparisons For the DIF analysis across individual states, the state reported on the examinee s record was used as a basis of group identification. It was decided to focus only on states that had at least 1,000 examinees. For each selected state, focal and reference groups were formed, with the focal group consisting of a randomly selected subsample of respondents from that particular state, and the reference group consisting of a randomly selected but proportional subsample of respondents from all other states. Focal and reference group distributions for each selected state are shown in Tables

10 Table 8. DIF sample sizes by states, Test A section State Focal Group, N Reference Group, N CA 4,000 5,000 FL 3,000 5,000 GA 1,500 5,000 IL 1,000 5,000 IN 1,000 5,000 MA 1,200 5,000 MD 1,200 5,000 NC 1,200 5,000 NJ 3,000 5,000 NY 2,500 5,000 OH 1,200 5,000 PA 3,000 5,000 TX 2,500 5,000 VA 1,200 5,000 Table 9. DIF sample sizes by states, Test B section State Focal Group, N Reference Group, N AZ 4,500 5,000 CA 4,000 5,000 CO 1,000 5,000 CT 1,000 5,000 FL 4,000 5,000 GA 4,000 5,000 IL 1,500 5,000 IN 2,000 5,000 MA 2,000 5,000 MD 2,500 5,000 MI 1,000 5,000 NC 2,500 5,000 NJ 4,000 5,000 NY 4,000 5,000 OH 2,500 5,000 PA 4,000 5,000 SC 1,200 5,000 TX 4,000 5,000 VA 3,500 5,000 WA 1,500 5,000 10

11 Table 10. DIF sample sizes by states, Test C section State Focal Group, N Reference Group, N AZ 2,000 5,000 CA 4,000 5,000 CO 1,500 5,000 CT 1,500 5,000 FL 4,000 5,000 GA 4,000 5,000 IL 2,500 5,000 IN 4,000 5,000 MA 3,500 5,000 MD 4,000 5,000 MI 1,500 5,000 NC 4,500 5,000 NJ 4,500 5,000 NY 4,500 5,000 OH 4,000 5,000 PA 4,000 5,000 SC 2,500 5,000 TN 2,000 5,000 TX 4,500 5,000 VA 4,000 5,000 WA 2,500 5,000 DIF analyses Individual items from the Test A, Test B, and Test C sections were examined for DIF using SIBTEST statistical software. For each test section, individual items analyses were performed, with each item compared against the rest of the items in that section. Items were examined for DIF based on both beta and MH chi-square statistics, using the significance level adjusted for the number of comparisons (Roussos & Stout, 1996). Particularly, adjusted significance levels were.05/54 for Test A items,.05/67 for Test B items, and.05/49 for Test C items. An item was flagged for DIF only if it satisfied both MH and beta index significance criteria. 11

12 Results of DIF across geographical regions As a result of analyses, initially 32 items were identified as having DIF, with most items having negligible DIF (class A), and only 4 items with DIF of moderate (B-class) severity (Tables11-13). Table 11. Test A items flagged for DIF across regions Region Region Name item p Beta index MH Delta DIF class (SIB) DIF class (MH) 2 Middle States A A A A A A 6 Southwest A B 7 West A A A B A A Table 12. Test B items flagged for DIF across regions Region Region Name item p Beta index MH Delta DIF Class (SIB) DIF class (MH) 1 Midwest A A 2 Middle States A A A A A A 3 New England A A A A A A 5 South A A A A A A 6 Southwest A A A A A A A A A A 7 West B A A A A A 12

13 A A A A Table 13. Test C items flagged for DIF across regions Region Region Name item p Beta index MH Delta DIF Class (SIB) DIF Class (MH) 1 Midwest A A A A 2 Middle States A A A A 3 New England A A A A A A A A 5 South A A 6 Southwest A A A A A A A A B A A A 7 West A A A A A A A A The correlation between the two DIF statistics across all three sections was very high (r = -.92, p<.01), and similar to correlations reported in the literature (e.g., Dorans & Holland, 1993; Gierl et al., 1999). However, classification of DIF severity was inconsistent across the two detection methods: none of the 4 items were flagged as exhibiting moderate degree DIF by both statistics. According to DIF 13

14 classification based on MH statistics (Zieky, 1993), 2 items have been labeled as having B-class DIF. Classification of DIF based on beta (Roussos & Stout, 1996) resulted in 2 other items labeled as exhibiting B-class DIF. Similar inconsistencies in classification of DIF effect sizes across different DIF detection methods have been reported in the literature before (e.g., Gierl et al., 1999), and have been attributed to the fact that both measures use different metrics (Roussos & Stout, 1996). In addition, the MH test was found to produce fewer Type I errors than SIBTEST (Gierl et al., 1999; 2000). At the same time, it has been suggested that SIBTEST is more powerful, as compared to MH, and in general produces fewer Type II errors. It has also been shown that the MH test often produces inflated estimates of the effect size of DIF, resulting in a higher number of items with DIF of B- and C- class (Roussos & Stout, 1996). These differences lead to different implications (and applications of these methods) for researchers and test developers, with researchers emphasizing the more powerful SIBTEST, whereas test developers opt for the more conservative MH method (Gierl et al., 1999). Consequently, for the exploratory purposes of this study, it was decided to focus further only on items that have been flagged by SIBTEST (Table 14). Table 14. Items selected for further examination (DIF across geographical regions) Region Region Name Item SIB beta DIF class Test B Section 7 West B Test C Section 6 Southwest B Results of DIF across states 14

15 As a result of analyses, 70 items have been initially identified as having DIF of A-class or higher with regard to some states specific states (Tables 15-17). The correlation between two DIF statistics across all three sections was also very high (r = -.91, p<.01), and similar to correlations reported in the literature (e.g., Dorans & Holland, 1993; Gierl et al., 1999). Table 15 Test A items flagged for DIF State item Beta index MH Delta DIF Class (SIB) DIF class (MH) CA A A A A A A A A FL A B A A IL A B MD A B NC A C A A A B NJ A A A A A A A A NY A A B A A B A A A A PA A A TX A C VA A C A B 15

16 Table 16. Test B items flagged for DIF State item Beta index MH Delta DIF Class (SIB) DIF class (MH) AZ A A A A B A GA A A A A A A B A IL A B A A MA A A A A MD A A A A NC A A NJ A A A A A A A A NY A A A A A A B B A A OH A A B B PA A A A A A A A A TX A A B B VA A A A A 16

17 Table 17. Test C items flagged for DIF State item Beta index MH Delta DIF Class (SIB) DIF Class (MH) CA A A A A A A A A A A A A A A CO A A A A CT B B B A FL A A A A GA A A A A IL A A A A A A IN A A A A A A A A MA A A A A A A A A A A NC A A A A A A A A NJ A A A A A A 17

18 Table 17. Test C items flagged for DIF (continued) State item Beta index MH Delta DIF Class (SIB) DIF Class (MH) NY A A A A A A A A A A A A A A OH A A PA A A A A A A A A SC A A A A A A A A TN B B A C A A TX A A A A A A A A VA A A A A A A A A A A A A A A WA A A A A A A B A A A A A Again, classification of DIF was inconsistent across the two detection methods. A total of 15 items have been classified as having either B- or C-class DIF, 18

19 based on at least one of the classification methods. Based on MH statistics (Zieky, 1993), 5 out of 9 items have been labeled as B-class, and 4 items as C-class. Classification of DIF based on beta (Roussos & Stout, 1996) resulted in 6 items labeled as B-class DIF, and none as C-class. Only for 3 items was the classification of non-negligible (class B or C) DIF consistent. Again, only items flagged by the SIBTEST were selected for further examination (Table 18). Table 18. Items selected for further examination (DIF across individual states) State Item SIB beta DIF class Test A Section NY B Test B Section AZ B GA B NY B OH B TX B Test C Section CT B B TN B WA B Closer look at items with DIF across geographical regions The means and standard deviations for the flagged items across the focal and reference groups are shown in Table

20 Table 19. Means and standard deviations for flagged items, DIF across geographical states Region Region Name item Focal Group Reference Group M SD M SD Test B Section 7 West Test C Section 6 Southwest Both SIBTEST and MH are directional, with the sign of the value indicating direction of the DIF. A negative beta index indicates that the item favors focal group, whereas positive beta is an indication of advantage to the reference group. For the Mantel-Haenszel statistics, the opposite is true. Further analyses, such as analyses of response patterns, and examination of item content, have been performed to confirm the DIF in the selected items. The actual questions and answer choices are shown below. Item 28 (Test B Section) Much of this author s work, unfortunately, is with chapter often immediately following a sublime one. (A) mystical.. a superior (B) uneven.. a mediocre (C) predictable.. an eloquent (D) enthralling.. a vapid (E) flippant.. an intelligible 20

21 Item 32 (Test C Section) During seasons when ticks carrying Lyme disease are most prevalent, signs could be posted to deter hikers about their venturing into tick-infested areas. (A) about their venturing (B) from their venturing (C) from venturing (D) by not venturing (E) not to venture Results of DIF analyses suggest that items 28 and 32 moderately favored examinees from West and Southwest, accordingly. Analysis of the patterns of raw responses indeed confirms that. Adjusted residuals tests were computed to assess the strength of association between the regions and answer choices (Tables 20-21). The adjusted residuals measure strength of the association between the row and column categorical variables, with absolute value of adjusted residual of 3 suggesting a significant relationship. The sign of the residual also indicates the direction of association. In both tables, the relationship between region 7 (West) and the correct response choice was the strongest. Particularly, for item 28, adjusted residual of 14.7 in the corresponding cell suggests that examinees from the West tend to endorse choice B more likely than it would be expected at random. Similarly, adjusted residual of 10.1 in the corresponding cell suggests positive relationship between the endorsing correct response A and residence in region 6 (Southwest). 21

22 Table 20. Association between raw item response and region, Test B (* marks the correct response) Response (Adj. Resid.) Item Answer Region A B* C D E Table 21. Association between raw item response and region, Test C (* marks the correct response) Response (Adj. Resid.) Item Answer Region A* B C D E Interestingly, however, that although the largest negative residual in the cell for region 5 (South) and correct response B on item 28 of Test B section suggests that examinees from this region had a strong tendency to answer this item incorrectly, the item itself has not been flagged for DIF with regard to the South. Similarly, for item 32 of the Test C section, with adjusted residual of in the cell the correct response A and region 5, South should have been defined by DIF tests as a region that is disfavored by the item. Also, for the item 28, examinees from the region 3 (New England) also have shown a strong tendency for 22

23 endorsing correct response B, however, the item has not been flagged as functioning differently with regard to this region. These observations allow for speculation that either the definition of the regions was not proper and the variability from state to state is so great that it is canceled out within region, or DIF methods are not sensitive enough to detect item performance optimally at the regional level. Closer look at items with DIF across states The means and standard deviations for the flagged (by either method) items across the focal and reference groups are shown in Tables Table 22. Means and standard deviations for selected Test A items State Item Focal Group Reference Group M SD M SD NY Table 23. Means and standard deviations for selected Test B items State Item Focal Group Reference Group M SD M SD AZ GA NY OH TX Table 24. Means and standard deviations for selected Test C items State Item Focal Group Reference Group CT TN WA Item means across all sections (Tables 22-24) confirm the directionality of both MH and beta tests, with exception for one case. Test C item 16 (case of 23

24 Tennessee), is an example of Simpson s paradox (Simpson, 1951; Wagner, 1982). Simpson s paradox occurs when item favors one group over another, while the overall item impact (measured as item group means) suggests opposite. In the case of item 16 of the Test C section, a comparison of item means suggests that item favors examinees from Tennessee over examinees from the other states (item mean for Tennessee Carolina is.73, compared to item mean for other states of.70). Both beta and MH delta, however, indicate the opposite: the item gives slight advantage to examinees from other states. Further analyses, such as analyses of response patterns, and examination of items content, have been performed to identify the possible causes of DIF in the selected items. Of a special interest was item 51 of the Test B section, as this item was flagged for DIF with regard to several states. The actual question and answer choices are shown below. Test A Section, Item 20 k = 3wx m = (w -1)k If k and m are defined by the equations above, what is the value of m when w = 4 and x = 1? (A) 0 (B) 3 (C) 12 (D) 24 (E) 36 24

25 Test B Section, Item 51 Hawaii refers both to the group of islands known as the Hawaiian islands and to the largest island in that (A) flora (B) sierra (C) archipelago (D) flotilla (E) savanna For the following 3 Test C items the directions read: The following sentences test your ability to recognize grammar and usage errors. Each sentence contains either a single error or no error at all. No sentence contains more than one error. The error, if there is one, is underlined and lettered. If the sentence contains an error, select the one underlined part that must be changed to make the sentence correct. If the sentence is correct, select choice E. In choosing answers, follow the requirements of standard written English. Test C Section, Item 16 For months the press had praised Thatcher s handling of the international crisis, and A editorial views changed quickly when the domestic economy worsened. No error B C D E Key is A Test C Section, Item 18 In the aggressive society created by William Golding in Lord of the Flies, both Ralph A and Jack emerge early on as the leader of the lost boys. No error B C D E Key is C 25

26 Test C Section, Item 23 Of the two options, neither the system of appointing judges to the bench nor the A B process of electing judges are entirely satisfactory. No error C D E Key is D Test C Section, Item 32 [only relevant sentence from passage included in the paper for the purpose of brevity] (7)My father, by all accounts, sees taking time to listen as essential to any relationship, whether it involves family, friendship, or work.[passage continues]. In sentence 7, the phrase by all accounts is best replaced by (A) however (B) moreover (C) to my knowledge (D) like my sister (E) but nevertheless Results of DIF analyses suggested that Test B item 51 moderately favored examinees from New York (β = -.076) and Texas (β = -.081), and moderately disfavored examinees from Arizona (β =.074), Georgia (β =.074), and Ohio (β =.079) (Table 18). In Test A section, item 20 moderately favored examinees from New York (β = -.069). In Test C section, item 16 moderately disfavored examinees from Tennessee (β =.66), item 18 favored examinees from Connecticut (β = and β = , respectively), and item 23 disfavored examinees from Washington (β =.077). 26

27 Further examination of the patterns of examinees raw responses on respective items have confirmed that examinees from the states favored by the items tend to endorse correct responses more likely than examinees from the disfavored states. To examine the patterns of responses, adjusted residuals were computed to measure the strength of association between the state and raw answer responses (Tables 25-30). In case of Test B item 51 (Table 25), the strongest positive relationships between the correct response (row C) was with columns for New York (adjusted residual of 17.26), Texas (adjusted residual of 9.82), and Georgia (adjusted residual of 5.23) confirming that examinees from these states had a strong tendency to endorse the item correctly. On the other hand, the columns of Ohio and Arizona had the strongest negative relationship with response C, as compared to other answer choices. The strongest positive association of the column of Ohio with answer choice D (adjusted residual of 7.5) suggests that examinees from this state are less likely to endorse the correct response, and more likely to endorse response D ( flotilla ). Similarly, examinees from Arizona were more likely to endorse responses D and E ( flotilla and savanna, adjusted residuals of 4.14 and 3.09, respectively). Table 25. Association between raw item response and state, Test B (* marks the correct response), measured in adjusted residuals State Item AZ GA NY OH TX 51 A B C* D E

28 Patterns of association between the state and response choice have been consistent with DIF status of other flagged items (Tables 26-30). The only exception was item 16 of the Test C section, previously defined as a case of Simpson s paradox (Table 27). According to DIF test, strong negative association was expected to be found between the correct response choice A, and column Tennessee. However, the adjusted residual in the corresponding cell was positive and below 3.0, which suggests the association had direction opposite of what was expected, and overall was not significant. Table 26. Association between raw item response and state, Test A (*marks the correct response), measured in adjusted residuals State Item NY OTHER A B C* D Table 27. Association between raw item response and state, Test C (*marks the correct response), measured in adjusted residuals Item 16 State TN OTHER A* B C D E

29 Table 28. Association between raw item response and state, Test C (*marks the correct response), measured in adjusted residuals Item 18 State CT OTHER A B C* D E Table 29. Association between raw item response and state, Test C (*marks the correct response), measured in adjusted residuals Item 23 State WA OTHER A B C D* E Table 30. Association between raw item response and state, Test C (*marks the correct response), measured in adjusted residuals Item 32 State CT OTHER A* B C D E

30 Discussion and Suggestions for future research The purposes of this study were to examine the items of a large scale assessment for possible differential functioning with regard to certain geographical areas, including geographical regions and individual states. Based on the findings, the following conclusions and recommendations have been made. DIF statistical procedures Two popular DIF detecting methods, Mantel-Haenszel chi-square and SIBTEST beta tests have been used to identify differentially functioning items. Both methods have flagged a comparable number of items. However, in spite of strong correlation between the respective effect size measures, the classification of DIF varied considerably across procedures. Such inconsistencies in classification of DIF across detection methods have been reported in the literature before, and the selection of the items for the further examination is said to depend often on the ultimate purposes of the study. In this study, the final set of items was selected based on SIBTEST classification, as this procedure is recommended as more appropriate for research and exploratory purposes. Another way to enhance the objectivity of the selection process would be to add a third DIF detection method (e.g.., logistic regression), and to focus on items that have been flagged as exhibiting B- or C-class DIF by at least two of the procedures. It is also recommended to perform additional analyses of dimensionality (using, for example, DIMTEST or TESTFACT software) on flagged items at both regional and state levels to confirm the presence of secondary dimensions, as a source of DIF. 30

31 DIF with regard to geographical regions As a result of DIF analyses, only two items were flagged as functioning differently with regard to certain geographical regions. Both items exhibited DIF of moderate degree (B-class). In both cases (one item from Test B section, and one item from Test C section), items favored examinees from specific regions over the rest of examinees. Among the main possible limitations of analyses is the way in which the regions have been defined. The regions in the current study have been formed in accordance with the College Board regional office affiliation, and the number of the regions, and their configuration, may be not optimal for detecting potential geographically-based differences. It is recommended to consider other ways of grouping states into the regions, with the purpose of finding the configuration that is most appropriate for the purposes of this research. DIF with regard to individual states Substantially more (six) items with DIF have been detected when items were examined with regard to individual states. All flagged items exhibited DIF of moderate degree (B-class). Out of 10 cases of DIF, 6 represented items that were favoring examinees from particular states over other examinees, and only 4 have shown DIF in negative direction. Due to the sample size restrictions, only 20 states have been used in the analyses, so it is possible that flexing the sample size restrictions could result in more flagged items, and more clear patterns of differential functioning across the states. One may also try comparing each individual state against each other, as 31

32 opposed to using all other states as a reference group. At the end, however, DIF analysis at the regional level is the most appropriate for practical purposes, as it is less intensive and time-consuming. Identifying sources of DIF While DIF could be detected through quantitative methods, the identification of sources of differences in item performance often require qualitative means, such as review of the item content, and cognitive analysis (Gierl et al., 1999). In both cases, judgments of possible causes of the DIF were inconclusive, and further investigation is required. It is possible that differences in item performance are caused by specific differences in composition of the regions or states (e.g., proportion of racial or ethnic groups, or urban, suburban and rural populations, etc.). This hypothesis could be tested through additional analyses that take in consideration effects of demographic factors (e.g., logistic regression). It is also possible that DIF is a reflection of true differences in the knowledge, due to actual differences in curriculum or academic achievement across the states or regions. This hypothesis could be tested by combining the flagged items with other items in the section that measure the same specific content, and running DIF on the bundles of items. The DIF due to actual differences in knowledge of content could then be confirmed by identifying DIF at the bundle level (Nandakumar, 1993). Conclusion It is common to encounter anecdotal evidence of geographic based DIF when interacting with parents, teachers, administrators, and others with access to examinees ranging in experience from elementary school to college admissions 32

33 testing. This study was an attempt to empirically investigate these anecdotal reports in an effort to determine to what extent geographic location within the United States may affect examinee test performance. While the results from this study may not have identified a significant impact for the test data in consideration, which was based on examinees at the secondary level, it is suggested that additional research is needed on tests from across the range of the educational system. It seems that The anecdotal evidence of geographic DIF seems to be more pervasive in the early school years and it would be informative to determine if a greater impact is found on elementary school tests. At a minimum additional research on Geographic DIF needs to be undertaken to further explore this potential source of error. It is probably impractical to expect testing organizations to begin running DIF individually for the 50 states, however using regional classifications is probably reasonable once an optimal grouping of states within regions can be defined. The results of this study suggest it is an avenue that needs further exploration to ensure that tests are as fair to all students as possible. It is also important to evaluate whether differences are true cultural differences or evidence of the discrepancy between educational systems across the country and the need for a more standardized national curriculum. References Camilli, G., & Shepard, L.A. (1994). Methods for identifying biased test items. Newbury Park, CA: Sage. 33

34 Clauser, B.E., & Mazor, K.M. (1998). Using statistical procedures to identify differentially functioning test items. Educational Measurement: Issues and Practice, 17, Dorans, N.J., & Holland, P.W. (1993). DIF detection and description: Maentel- Haenszel and standardization. In P.W. Holland & H.Wainer (Eds.), Differential item functioning (pp ), Hillsdale, NJ: Erlbaum. Gierl M., Khaliq S.N., & Boughton, K. (1999). Gender differential item functioning in Test Aematics and science: Prevalence and policy implications. Paper presented at the symposium Improving Large-Scale Assessment in Education at the Annual Meeting of the Canadian Society for the Study Of Education, Quebec. Gierl, M., Jodoin, M.G., & Ackerman, T.A. (2000). Performance of Mantel- Haenszel, Simultaneous Item Bias Test, and Logistic Regression When the Proportion of DIF Items is Large. Paper presented at the Annual Meeting of the American Educational Research Association, New Orleans. Holland, P. W., & Thayer, D. T. (1988). Differential item functioning and the Mantel Haenszel procedure. In H. Wainer & H. I. Braun (Eds.), Test validity (pp ). Hillsdale, NJ: Erlbaum. Mantel, N., & Haenszel, W. (1959). Statistical aspects of the analysis of data from retrospective studies of disease. Journal of the National Cancer Institute, 22, Nandakumar, R. (1993). Simultaneious DIF amplification and cancellation: Shealy-Stout s test for DIF. Journal of Educational Measurement, 16, 34

35 Roussos, L.A., & Stout, W.F. (1996). Simulation studies of the effects of sample size and studied item parameters on SIBTEST and Mantel-Haenszel type I error performance. Journal of Educational Measurement, 33, Shealy, R., & Stout, W. F. (1993). A model-based standardization approach that separates true bias/dif from group differences and detects test bias/dif as well as item bias/dif. Psychometrika, 58, Simpson, E.H. (1951). The interpretation of interaction contingency tables. Journal of the Royal Statistical Society (Series B), 13, Wagner, C.H. (1982). Simpson s paradox in real life. American Statistician, 36, William Stout Institute for Measurement.(1999) Simultaneous item bias test (SIBTEST) version1.0. St. Paul, MN: Assessment Systems Corporation. Zieky, M. (1993). Practical questions in the use of DIF statistics in test Development. In P.W. Holland & H. Wainer (Eds.) Differential item Functioning (pp ). Hillsdale, NJ: Erlbaum. 35