Reporting Subscores from College Admission Tests

Transcription

1 A peer-reviewed elecronic journal. Copyrigh is reained by he firs or sole auhor, who grans righ of firs publicaion o he Pracical Assessmen, Research & Evaluaion. Permission is graned o disribue his aricle for nonprofi, educaional purposes if i is copied in is enirey and he journal is credied. Volume 14, Number 4, March 009 Reporing Subscores from College Admission Tess Per-Erik Lyrén, Umeå Universiy, Sweden The added value of reporing subscores on a college admission es (SweSAT) was examined in his sudy. Using a CTT-derived objecive mehod for deermining he value of reporing subscores, i was concluded ha here is added value in reporing secion scores (Verbal/Quaniaive) as well as subes scores. These resuls differ from a sudy of he SAT I and a sudy of a basic skills es and hus highligh he need for praciioners and researchers o gaher empirical evidence o suppor he reporing of subscores. The cause of he disparae resuls seems o be relaed o differences in he composiion of he ess raher han differences in he composiion of he examinee groups. Oucomes of assessmens are usually repored in he form of one or several scores. Many esing programs repor only a oal score, usually in he form of a composie. However, he las few decades have seen an increasing ineres in he reporing of subscores, which are derived from subsecions of ess. This is also rue for college admission ess. For insance, he SAT (College Board, 008) repors Criical Reading, Mahemaics, and Wriing scores, he ACT (ACT, 007) repors English, Mahemaics, Reading, and Science scores, and for he Psychomeric Enrance Tes (PET) a Verbal Reasoning score, a Quaniaive Reasoning score, and an English score are repored (Allalouf, 003). I is no uncommon ha ess ha were originally designed o produce reliable scores for ranking examinees are hen expeced o provide informaion ha can be useful for remediaion or oher purposes as well. For insance, some Israeli insiuions use he PET English score for placemen (Naional Insiue for Tesing and Evaluaion [NITE], 009). Monaghan (006) poins ou ha while assessmen organizaions and es developers wan o be responsive o he desire of examinees and oher paries wih an ineres in esing programs o repor subscores, hey also wan o preven hese subscores from being misused. Furher, he Sandards for Educaional and Psychological Tesing (American Educaional Research Associaion, American Psychological Associaion, & Naional Council on Measuremen in Educaion, 1999), saes: When inerpreaion of subscores, score differences, or profiles is suggesed, he raionale and relevan evidence in suppor of such inerpreaion should be provided (Sandard 1.1, p. 0). This means ha before es developers or praciioners decide on which scores o repor hey should gaher validiy evidence in suppor of such a provision. Tha validiy evidence may consis of logical evidence, procedural evidence, and empirical evidence (e.g. Haladyna & Kramer, 004). This aricle focuses on examining empirical evidence o suppor he reporing of subscores, specifically if he subscores mee cerain saisical crieria ha can be used o deermine heir poenial added value. APPROACHES TO SUBSCORE REPORTING When he Graduae Record Examinaion (GRE) Program (see e.g. Chalifour & Powers, 1988) sared developing subscores hey used wo subjecively defined crieria: (a) he subscores had o aain a reliabiliy of a leas.80, and (b) he disaenuaed inercorrelaions wih oher subscores had o be less han.90 (McPeek, Alman, Wallmark, & Wingersky, 1976). In an aemp o find logically meaningful subscores on he GRE Advanced Psychology Tes, McPeek e al. used wo mehods: conen analysis and facor analysis. The conen analysis was based on he conen areas defined

2 Pracical Assessmen, Research & Evaluaion, Vol 14, No 4 Page in he es specificaions and included iem and es analysis wih a focus on iem discriminaion, inercorrelaions, and reliabiliy, using he wo previously described subjecive crieria. The facor analysis was performed o examine if he subscores were essenially unidimensional, and if oher poenial groupings of iems (subscores) exised; however, because i was found ha only one predominan facor exised on he enire es i was deemed unnecessary o subjec each of he conen area subscores o facor analysis. I was concluded ha he subscores based on he conen analysis had considerable poenial o provide informaion for use in guidance and placemen; ye, his only applied o candidaes wih unusually high or low scores. Furher, he GRE Psychology Commiee found he hree subscores based on he facor analysis o be no useful for guidance and placemen. Longford (1990) used variance componens mehodology o examine he usefulness of reporing subscores for a college-level general educaion examinaion. Two versions of he es were examined: a shor version (one form wih 1 hour of esing ime) and a long version (hree forms wih 3 hours of esing ime). Longford concluded ha subscores were worh reporing on he long version only, while on he shor version, any subscore, or a linear combinaion of subscores, is indisinguishable from a less reliable version of he oal score (p. 109). Harris and Hanson (1991) examined English and Mahemaics subscores and oal scores on he P-ACT+ (American College Tesing, 1989) o deermine wheher he subscores provided differen and beer informaion for examinee-level decisions, specifically placemen, han he oal score. The mehods and measures used included a rue score mehod presened by Lord (1965) and Hanson (1989), disaenuaed correlaions among subscores, effecive weighs (Wang & Sanley, 1970), and a procedure for esimaing classificaion consisency indices described by Hanson and Brennan (1990). Harris and Hanson concluded ha neiher he English subscores nor he Mahemaics subscores provided informaion disinc from he respecive oal scores. Also, in a simulaed placemen siuaion he examined subscore did no provide beer informaion han he oal score did, a leas no wih any pracical significance. Tae (004) performed simulaions o examine wheher he addiional provision of muliple subscores offers useful diagnosic informaion for individual sudens. Specifically, he relaionship was examined beween he average error variance for subscores and hi raes for deecing imporan subscore differences on he one hand, and es dimensionaliy and he correlaional level among he subscores on he oher hand. I was found ha he average error variance of he oal score increased wih decreasing level of correlaion and increasing es dimensionaliy, while he average error variance of he individual subscores depended only on he number of subes iems and no on he level of correlaion. Also, he subscore differences depended only on he level of correlaion. Tae concluded ha he adequacy of he subscore performance, given a specific combinaion of dimensionaliy and correlaional level, depends on (a) wheher he subscores are o be used for absolue decisions or relaive decisions, and (b) he choice of esimaion mehod. Haladyna and Kramer (004) used several mehods and measures o examine he validiy of he inerpreaion and use of subscores from a es measuring knowledge of basic biomedical science, which is one par of an examinaion program for deniss (Join Commission on Naional Denal Examinaions, 004). The mehods and measures included (a) repeaed measures one-way analysis of variance o deermine if he subscore means suggesed differences in he difficuly of he subdomains, (b) inercorrelaions beween subscores, which provides evidence regarding he dimensionaliy of he iem responses; under condiions of unidimensionaliy he disaenuaed correlaions should be a leas.90, (c) confirmaory facor analysis, also used o assess dimensionaliy, (d) an iem analysis focused on discriminaion indices using boh he subscore and oal score as crierion scores; discriminaion indices based on he subscore should be differen from hose based on he oal score, and (e) he reliabiliy of differences among each pair of subscores, using a procedure recommended by Ryan (003). Haladyna and Kramer concluded ha validiy evidence can be garnered o suppor he inerpreaion and use of subscores ha may be used boh by failing candidaes for planning fuure remedial sudies and professional schools for he evaluaions of heir educaional programs (pp ). They noe, however, ha had he sudy focused on cogniive-process dimensions raher han conen dimensions he resuls migh have been differen. The idea of using he cogniive processes involved in iem solving when creaing subscores, raher han using he conen areas ha make up he es, was uilized in a sudy by Wainer, Sheehan, and Wang (000). They used

3 Pracical Assessmen, Research & Evaluaion, Vol 14, No 4 Page 3 he iem difficuly modeling approach described in Sheehan and Mislevy (1990) and Embreson (1998) o consruc subscales beer suied for diagnosic feedback on he Educaion in he Elemenary School Assessmen (EES), a par of he Praxis program (Dwyer & Villegas, 1993). When defining he subscales, he iems were classified according o (a) he primary skill area addressed and (b) he ype of informaion given and requesed in he iem sem. I was found ha he skill-area classificaion yielded informaion beer suied for remediaion han did he conen-area classificaion. All previously described crieria and mehods for examining he performance of subscores are, more or less, subjecive. However, Haberman (008; 005) proposed an objecive crierion derived from classical es heory (CTT). The crierion is based on he concepion ha here is value in reporing a cerain subscore if he observed subscore is a more reliable predicor of he rue subscore han he observed oal score is. Therefore, when applying his crierion one makes an objecive comparison; he size of he subscore measure relaive o he oal-score measure deermines wheher here is value in reporing ha subscore. Haberman (008) applied he crierion o subscores on he SAT I examinaion, which is used for college admission, and found ha none of he secion scores of SAT I mah or SAT I verbal provide any appreciable informaion concerning an examinee ha is no already provided by he mah or verbal oal score (p. 1). In addiion, Sinharay, Haberman, and Puhan (007) used Haberman s crierion o examine he value in reporing subscores on a basic skills ess primarily adminisered o prospecive or pracicing eacher s aides. They concluded ha here was no added value in reporing eiher of he subscores Reading Skills, Reading Applicaion, Mahemaics Skills, Mahemaics Applicaion, Wriing Skills, and Wriing Applicaion, or he combined Reading score, Mahemaics score, and Wriing score. Haberman, Sinharay, and Puhan (009) also used he crierion o examine subscores in he conex of a basic skills es used for eacher cerificaion, and Puhan, Sinharay, Haberman, and Larkin (008) used he crierion o examine subscores from eigh cerificaion ess. Like Haberman (008) and Sinharay e al., boh Haberman e al. and Puhan e al. concluded ha he subscores did no provide any useful informaion oher han already provided by he oal score. METHOD The poenial added value of reporing subscores on he SweSAT is examined by using he approach proposed by Haberman (008; 005). In addiion o providing an objecive crierion for deermining he value of repored scores, he use of he mehod is fairly sraighforward and is based on saisics ha are readily available for es scores (i.e. means, variances, correlaions, and reliabiliy coefficiens). Also, Sinharay e al. (007) noed ha anoher advanage of he measure suggesed is ha i is concepually very close o es reliabiliy so he measure will be inuiively appealing o he praciioners (p. 3). Purpose The main purpose of his sudy is o examine if here is added value in reporing subscores on he Swedish Scholasic Assessmen Tes (SweSAT), which is used for selecion o higher educaion in Sweden. The moivaion for performing he sudy is ha o his dae, Haberman s (008) sudy is he only one ha has used an objecive crierion o examine he value of reporing subscores on a college admission es. Consequenly, here is a need for more sudies examining similar ess in order o provide a more general picure of he value of reporing subscores. The moivaion for examining he SweSAT specifically is wofold. Firs, he SweSAT score reporing procedure is going hrough some changes a he momen. For example, a few years ago he reporing of subscores (corresponding o he five subess) o he examinees commenced. The moivaion for ha decision was ha he subscores could provide diagnosic informaion and hus be useful for remediaion; however, here was no empirical evidence o suppor he decision. Anoher change is ha secion scores will be provided and mos likely used for admission purposes, which has been a realiy for he SAT and ACT for decades. Second, examining he SweSAT allows for comparisons beween ess from wo differen counries, ha is, beween US ess and a non-us es. The SweSAT The Swedish higher educaion admission sysem is highly cenralized and a suden can be admied on he basis of hree measures: (a) he grade-poin average (GPA) afer he las year of high school, (b) he SweSAT score, and (c) crieria locally deermined by each universiy. Wha is raher unique is ha candidaes are admied on he basis of one of hese measures, no a combinaion of hem. According o admission regulaions, a leas one hird of candidaes should be admied on he basis of SweSAT scores, and in pracice his number is abou 40 percen.

4 Pracical Assessmen, Research & Evaluaion, Vol 14, No 4 Page 4 The SweSAT is a norm-referenced, paper-andpencil, muliple-choice es wih five subess: Vocabulary (WORD; 40 iems), Swedish Reading Comprehension (READ; 0 iems), English Reading Comprehension (ERC; 0 iems), Daa Sufficiency (DS; iems), and Diagrams, Tables, and Maps (DTM; 0 iems). The es is adminisered wice a year wih approximaely 30,000 50,000 examinees per adminisraion. As previously saed he SweSAT repors only an overall composie score for use in he admission process, while subes scores are provided o examinees only. The wo secion scores proposed for use in he admission procedure are a Verbal score (WORD + READ + ERC) and a Quaniaive score (DS + DTM). The moivaion for using secion scores is ha hey are assumed o increase he predicive validiy of he es, especially in programs where mahemaics and quaniaive reasoning dominae he curriculum, such as engineering. The five subess are inended o measure fairly disinc consrucs, so here is some logical evidence for reporing he subes scores. Furher, he SweSAT es developmen procedure is sandardized and rigorous, so here is some procedural evidence as well. However, here have been no sudies aimed a gahering objecive empirical evidence o suppor he reporing of subscores. Daa The daa consised of examinees scores from five consecuive adminisraions of he SweSAT; he 006 spring and fall adminisraions (06A and 06B), he 007 spring and fall adminisraions (07A and 07B), and he 008 spring adminisraion. Scores were available a he iem, subes, and es level. The number of examinees was 41,530, 9,787, 38,469, 6,610, and 37,43 for he five adminisraions respecively. Scores from all five separae subess of he SweSAT as well as he wo secion scores are examined. Subscore predicors According o Sinharay e al. (007), whenever a subscore is repored o an examinee he goal from a CTT perspecive is o predic he examinee s rue subscore S from his or her observed subscore S. This implies ha for a subscore o have added value i should provide a more accurae measure of he consruc i purpors o measure han is provided by he oal score (Haberman, 008). In erms of predicions, his means ha for a subscore o have added value, he rue subscore should be prediced beer by a predicor based on he observed subscore han by a predicor based on he oal score (Sinharay e al., 007, p. 3). If his condiion is no saisfied, hen any selecion decision made or diagnosic informaion provided on he basis of he subscore will have more errors han he corresponding one based on he oal score. The predicors proposed by Haberman (008) are based on linear regressions for approximaions of he rue subscore S. The regression formula for S s, he predicor based on he observed subscore S, is equivalen o Kelley s (1947) formula applied o subscores, S s = E( S) + ρ ( S [ S E( S)], where is he subscore reliabiliy. 1 Furher, he regression formula for S x, he predicor based on he observed oal score X, (see e.g. Equaion 45 in Holland & Hoskens, 003) is σ( S ) S x = E( S) + ρ ( S, X ) [ X E( X )], σ( X ) where σ(x ) is he sandard deviaion of he observed oal score, σ ( S ) = σ( S) ρ ( S is compued using he values of he observed subscore variance and he esimaed KR-0 reliabiliy, and ρ( S, X ) denoes he correlaion beween he rue subscore and he observed oal score. Furher, ρ ( S, X ) = ρ ( S, X ) ρ ( X, X ), (1) where ρ ( X, X ) is he oal score reliabiliy esimaed by he KR-0 approach and, X ) can be esimaed from [ σ( X ) ρ( S, X ) σ( S)[ ρ ( S ] ] ρ ( S, X ) = () σ ( X ) ρ ( S ρ ( X, X ) where σ (S) is he sandard deviaion of he observed subscore, ρ( S, X ) is he correlaion beween he observed subscore and he observed oal score, and and ρ ( X, X ) are he subscore and oal score reliabiliies esimaed by he KR-0 approach. I should be noed ha (1) and () are he only equaions ha have o be used when examining he subscores. (Complee derivaions are available from he auhor.) 1 Crierion for comparing predicors of he rue subscore Haberman (008) suggesed he proporional reducion of mean squared error (PRMSE) of he predicors S s and

5 Pracical Assessmen, Research & Evaluaion, Vol 14, No 4 Page 5 S x compared o he mean squared error (MSE) of he rivial predicor E(S) as a crierion for comparing predicors of rue subscores. The rivial predicor E(S) gives he same predicion of he rue subscore for all examinees (i.e. he average subscore). Before describing he PRMSE in more deail, he MSEs for he predicors S s and S x and he rivial predicor are described. The MSE of a predicor is reasonable o use in his conex because a larger MSE implies more error in he decisions made on he basis of ha score. The MSE for he rivial predicor E(S) is σ ( S ), he MSE for he predicor S s is σ ( S )[ 1 ρ ( S ], and he MSE for he predicor S x is σ ( S )[ 1 ρ ( S, X )]. Now, he PRMSE for any predicor A of S wih finie mean and variance is E([ S A] ) Ψ ( S, A) = 1, σ ( S ) so ha Ψ ( S, E( S )) = 0 (Haberman, 008). Then i can be shown ha he PRMSE for he predicor S s is, which is he subscore reliabiliy. Similarly, he PRMSE for he predicor S x is, X ), which is esimaed from (1) and () and can be hough of as he reliabiliy of he observed oal score as a measure of he subes consruc. Thus, for a subscore o have added value he subscore reliabiliy mus be larger han, X ), which makes sense in erms of correlaions as well (we expec, S ) o be larger han ρ( S, X ) for a subscore o have added value). Sinharay e al. conclude ha a subscore will be favored as he subscore reliabiliy increases, he oal score reliabiliy decreases, and he correlaion beween rue subscore and rue oal score decreases (which will happen when he subess measure very differen skills) (p. 4). Corroboraing he resuls. One way of corroboraing he resuls for he subes scores is by sudying he effecive weighs of he differen subess. An effecive weigh (Wang & Sanley, 1970; see also Peersen, Kolen, & Hoover, 1989; Kolen & Brennan, 004) is an index of he conribuion of he subes o he oal es. The more relaed a score is o he oher scores, he greaer is effecive weigh (Peersen e al., 1989). Also, he more relaed a score is o he oher scores, he more relaed ha score is o he oal score and hus he less he value of reporing ha score. The sandard definiion of an effecive weigh is he covariance beween he subes score and he oal score. However, because of he large differences in subes lenghs on he SweSAT here will be large differences in subes variances and covariances. Consequenly, he effecive weighs will be more dependen on es lengh han on he acual relaionships beween he subess. I herefore makes more sense o calculae he effecive weighs based on scores ha are placed on he same scale, for example by sandardizing scores o have uni variance. When doing so, he effecive weigh of a subes score is ε = 1+, i ρ ij j 1 where ρ is he correlaion beween ess i and j. ij RESULTS Table 1 shows he PRMSEs for subscores from he individual subess and wo secions for five forms of he SweSAT. The PRMSEs are larger for S s, he predicor based on he observed subscores, han for S x, he predicor based on he observed oal score, for hree subes scores (WORD, DS and DTM) and boh secion scores across all es forms. This implies ha here is added value in reporing hese scores. I seems clear ha he high subscore reliabiliy esimae conribues o he added value for he WORD subes, because he average PRMSE for S x is no paricularly low when compared o he oher subes scores. A high subscore reliabiliy esimae, as well as a relaively low PRMSE for S x, also conribues o he added value for he DS subscore. For he READ subscore he PRMSEs is smaller for S s han for S x across all five es forms, indicaing ha here is no added value in reporing he READ scores. The reason is no ha READ scores have low reliabiliy, bu raher ha he observed oal score correlaes well wih he rue READ score and hus is a good measure of reading comprehension. Compared o he oher subess he DTM subes has relaively low PRMSEs for boh predicors; i has he lowes average reliabiliy esimae and he second lowes correlaion beween observed oal score and rue subscore. In spie of he relaively low reliabiliy, here is sill added value in reporing DTM scores due o heir relaively weak relaionship wih he oal score. The PRMSEs for he predicors of he ERC rue score are quie similar, and differ on average by a mere 0.0 unis. Ye, he PRMSEs do indicae ha here may be value in reporing ERC scores.

6 Pracical Assessmen, Research & Evaluaion, Vol 14, No 4 Page 6 Table 1: Esimaed Proporional Reducion of Mean Squared Error (expressed as percenages) for Five Subes Scores and Two Secion Scores from Five Tes Forms Tes PRMSE Subes scores Secion scores form WORD DS READ DTM ERC V Q 06A , X ) B , X ) A , X ) B , X ) A , X ) Average ρ (, X ) S Noe. For each es form and subscore, he larger of he wo PRMSEs, indicaing he relaively beer predicor, is boldfaced. Table : Inercorrelaions and Effecive Weighs for he Five Subess Toal es WORD DS READ DTM ERC Effecive weighs WORD DS a READ DTM ERC Noe. The correlaions are averages from he five adminisraions of he SweSAT examined in his sudy. The effecive weighs are calculaed assuming he subes scores have been sandardized. a The disagreemen beween he value of he effecive weigh and he sum of he correlaions is due o rounding. Boh secion scores have raher high reliabiliy, bu he oal score seems o be more relaed o he Verbal score han o he Quaniaive score. This is no surprising as he Verbal secion has almos wice as many iems as he Quaniaive secion. Sill, here is added value in reporing boh secion scores. Table displays inercorrelaions and effecive weighs of he subess. The able shows ha he effecive weighs of he READ and ERC scores are similar and higher han hose for he WORD, DS and DTM scores. This indicaes ha here are less of he WORD, DS and DTM consrucs in he oal es han here are of he oher subes consrucs. DISCUSSION The general conclusion is ha here is added value in reporing all he examined subes and secion scores, wih he excepion of he READ score. The main reason for his divergence is ha he observed oal score is a beer predicor of he READ rue score han he READ observed score is. This implies ha he oal score is a beer (i.e. more reliable) measure of reading comprehension han he reading comprehension score

7 Pracical Assessmen, Research & Evaluaion, Vol 14, No 4 Page 7 iself. The mos probable explanaion is ha all he oher subess, possibly wih he excepion of he vocabulary es, require some degree of reading comprehension. This is confirmed by a sudy of he laen srucure of he SweSAT (Carlsed & Gusafsson, 005), which indicaed ha he Crysallized inelligence (Gc) facor has he sronges associaion wih he SweSAT scores and ha he wo reading comprehension scores (READ and ERC) are he ones mos srongly associaed wih his facor. Furher, he effecive weighs indicae ha i is more likely ha here is added value in reporing he WORD, DS, and DTM subscores han i is in reporing READ and ERC scores, which is consisen wih he main findings in his sudy. The prospecive use of Verbal and Quaniaive scores in he admission process is suppored by he resuls in his sudy. The use of secion scores in he admission process will mos likely lead o increased predicive validiy, which hopefully will lead o an increase in he oal amoun of validiy evidence gahered around he SweSAT. In he process, i is imporan o bear in mind he need for supporing documenaion of any major changes made o he es or in he proposed use or inerpreaions of he es scores. If secion scores are used for admission purposes hen he original purpose of he es, which is o be a es of general developed abiliy used equally across all educaional programs, will have changed. This is a major change in he proposed use and inerpreaion of he es scores, and herefore all users of he admission sysem, especially he colleges/universiies and he examinees, need o be informed abou such a change. Anoher imporan issue o consider when inroducing addiional scores is he issue of equaing. The five SweSAT subes scores are repored o individual examinees for remedial purposes only and hus need no be equaed wih scores from previous adminisraions. However, scores being used for admission purposes need o be equaed, and he criical quesion is how he score equaing procedure should be execued during he ransiion from reporing only he oal score o reporing he oal score and he wo secion scores. Issues regarding equaing designs and mehods need aenion and furher research. The resuls of his sudy are conradicory o he resuls found by Sinharay e al. (007) on he basic skills es and by Haberman (008) on he SAT I examinaion, who concluded ha here was no added value in reporing secion scores. The cause of he disparae resuls on wo such seemingly similar ess as he SAT I and he SweSAT is no apparen. However, here are wo poenial explanaions: (a) differences in he examinee groups, and (b) differences in he composiion of he ess. While boh ess are aken mainly by high school seniors, abou half of he SweSAT examinees are 1 years or older. The analysis was consequenly rerun on one adminisraion of he SweSAT wih he examinees who were a mos 0 years old; ye, here was no pracical difference in resuls. Regarding he composiion of he ess, here are differences. The mos noable difference is ha he SweSAT subes WORD makes up almos a hird of he oal score (40 of 1) and half of he Verbal score (40 of 80), while he examined version of he SAT I from 00 had no similar iems measuring vocabulary ou of conex bu 19 analogy iems. The analysis was rerun on one adminisraion of he SweSAT, wih he vocabulary es excluded. A comparison of he PRMSEs showed ha boh secion scores would be worh reporing in his case as well; however, only wo of he four subes scores (DS and ERC) would be worh reporing. In his case he mos defensible decision would be no o repor he subes scores, which is consisen wih he findings on he SAT I. The disparae resuls from examinaions of he SAT I scores and he SweSAT scores as well as he manipulaed SweSAT daa emphasize he need for praciioners and es developers o examine empirical evidence before deciding on which scores should be repored, irrespecive of wheher i concerns a es under developmen, an already exising es, or a es ha is undergoing composiional changes. There are some operaional diagnosic programs ha seem promising. For he SAT here is an online diagnosic score reporing sysem called SAT Skills Insigh, which is based on behavioral anchors. When a es-aker eners he score band ha he or she scored wihin, he sysem provides informaion including (a) academic skills, lised by skill group, ha are ypical of sudens who score wihin he same score band, (b) suggesions on how es akers can move beyond heir score bands in each conen secions, and (c) seleced sample quesions wih answers (College Board, 008). The Preliminary SAT/Naional Meri Scholarship Qualifying Tes (PSAT/NMSQT; College Board & Naional Meri Scholarship Corporaion, 008a) has a secion in is score repor called Improve Your Skills, where es-akers are given a personalized analysis of heir areas of weakness as well as specific suggesions for how o improve (College Board & Naional Meri Scholarship Corporaion, 008b). For he Graduae Managemen Admission Tes (GMAT) here is a sysem

8 Pracical Assessmen, Research & Evaluaion, Vol 14, No 4 Page 8 for providing diagnosic informaion called GMAT Focus, which is based on a decision heory model (Rudner & Taleno-Miller, 007), ha anchors he subscale resuls o he oal scale performance. Under his model one can repor, for example, Your responses o iems involving algebra were ypical of an examinee ha scored in he op 0 percenile of all es akers on he Quaniaive scale (Rudner & Taleno-Miller, 007). Rudner and Taleno-Miller compared he scores from decision heory o scores derived from iem response heory (IRT) models, and made an ineresing observaion abou he logical inconsisency in he use of overall scale IRT parameers for esimaing subscale abiliies: By he logic of IRT, when he iem parameers are on he same scale, i should no maer which se of iems are used o esimae abiliy. Thus all subses, all subskill esimaes, should be idenical. If hey are no, hen eiher here is error in he esimae, he iem parameers are wrong, or he model does no hold. Given he increased correlaions wih augmenaion, he facor srucures, he very high inercorrelaions, and he relaively low number of iems used o esimae subskill abiliy, i appears ha he variance in subscale scores is due o error. Providing such subscale scores o es-akers who wan o improve heir overall score is anamoun o elling hem o chase error. (p. 15, emphasis added) Reporing subskills for an IRT calibraed and scored es can be problemaic. Linear ess such as he SweSAT do no have his problem. Subscore reporing ha mees appropriae saisical crieria can clearly provide meaningful feedback o es akers. While his sudy examined saisical crieria for reporing subscores, i is also imporan o examine oher aspecs of he score reporing procedure. For insance, he quesion of wheher he examinees find he subscores useful for remediaion is mos relevan. Relaed quesions are wheher he examinees undersand heir scores, and wheher he scores are repored in a way ha faciliaes appropriae inerpreaions and preven misinerpreaions. Theses issues have recenly gained ineres in he conex of sae and naional assessmens (see e.g. Goodman & Hambleon, 004; Wainer, Hambleon, & Meara, 1999; Ryan, 003). Ye, lile or no research has been devoed o he issue of how o properly communicae scores on college admission ess o he examinees and oher ineresed paries. Hence, here is a gap ha needs o be filled. References ACT (007). ACT echnical manual. Iowa Ciy, IA: Auhor. Allalouf, A. (003). Scoring and equaing a he Naional Insiue for Tesing and Evaluaion (Based on Allalouf, A. (1999) Scoring and equaing a he Naional Insiue for Tesing and Evaluaion. Research Repor No. 69). Jerusalem: Naional Insiue for Tesing and Evaluaion. American College Tesing (1989). P-ACT+ Program echnical manual. Iowa Ciy, IA: Auhor. American Educaional Research Associaion, American Psychological Associaion, & Naional Council on Measuremen in Educaion (1999). Sandards for educaional and psychological esing. Washingon, DC: American Educaional Research Associaion. Carlsed, B., & Gusafsson, J.-E. (005). Consruc validaion of he Swedish Scholasic Apiude Tes by means of he Swedish Enlismen Baery. Scandinavian Journal of Psychology, 46(1), Chalifour, C., & Powers, D. E. (1988). Conen characerisics of GRE Analyical Reasoning iems (ETS Research Repor No. RR-88-07). Princeon, NJ: Educaional Tesing Service. College Board (008). The SAT Program handbook. New York: Auhor. College Board, & Naional Meri Scholarship Corporaion (008a). Official educaor guide o he PSAT/NMSQT. New York: College Board. Rerieved March 4, 009, from hp://professionals.collegeboard.com/profdownload/o fficial-educaor-guide-o-he-psa-nmsq.pdf College Board, & Naional Meri Scholarship Corporaion (008b). Undersanding 008 PSAT/NMSQT scores. New York: College Board. Rerieved March 4, 009, from hp://professionals.collegeboard.com/profdownload/u ndersanding-psa-nmsq-scores.pdf Dwyer, C. A., & Villegas, A. M. (1993). Guiding concepions and assessmen principles for he Praxis Series: Professional assessmens for beginning eachers. (ETS Research Repor No. RR-93-17). Princeon, NJ: Educaional Tesing Service. Edwards, M. C., & Vevea, J. L. (006). An empirical Bayes approach o subscore augmenaion: How much srengh can we borrow? Journal of Educaional and Behavioral Saisics, 31(3), Embreson, S. E. (1998). A cogniive design sysem approach o generaing valid ess: Applicaion o absrac reasoning. Psychological Mehods, 3(3), Goodman, D. P., & Hambleon, R. K. (004). Suden es score repors and inerpreive guides: Review of curren pracices and suggesions for fuure research. Applied Measuremen in Educaion, 17(), Haberman, S. J. (005). When can subscores have value? (ETS Research Repor No. RR-05-08). Princeon, NJ: Educaional Tesing Service. Haberman, S. J. (008). When can subscores have value? Journal of Educaional and Behavioral Saisics, 33(), 04 9.

9 Pracical Assessmen, Research & Evaluaion, Vol 14, No 4 Page 9 Haberman, S. J., Sinharay, S., & Puhan, G. (009). Reporing subscores for insiuions. Briish Journal of Mahemaical and Saisical Psychology, 6(1), Haladyna, T. M., & Kramer, G. A. (004). The validiy of subscores for a credenialing es. Evaluaion & The Healh Professions, 7(4), Hanson, B. A. (1989). Scaling he P-ACT+. In R. L. Brennan (Ed.), Mehodology used in scaling he ACT Assessmen and P-ACT+. Iowa Ciy, IA: American College Tesing. Hanson, B. A., & Brennan, R. L. (1990). An invesigaion of classificaion consisency indexes esimaed under alernaive srong rue score models. Journal of Educaional Measuremen, 7(4), Harris, D. J., & Hanson, B. A. (1991, April). Mehods of examining he usefulness of subscores. Paper presened a he annual meeing of he Naional Council on Measuremen in Educaion, Chicago. Holland, P. W., & Hoskens, M. (003). Classical es heory as a firs-order iem response heory: Applicaion o rue-score predicion from a possibly nonparallel es. Psychomerika, 68(1), Join Commission on Naional Denal Examinaions (004). Naional board denal examinaions echnical repor. Chicago: American Denal Associaion. Kelley, T. L. (1947). Fundamenals of saisics. Cambridge, MA: Harvard Universiy Press. Kolen, M. J., & Brennan, R. L. (004). Tes equaing, scaling and linking (nd ed.). New York: Springer. Longford, N. T. (1990). Mulivariae variance componen analysis: An applicaion in es developmen. Journal of Educaional and Behavioral Saisics, 15(), Lord, F. M. (1965). A srong rue score heory, wih applicaions. Psychomerika, 30, McPeek, M., Alman, R., Wallmark, M., & Wingersky, B. C. (1976). An invesigaion of he feasibiliy of obaining addiional subscores on he GRE Advanced Psychology Tes (GRE Board Professional Repor No. 74-4P). Princeon, NJ: Educaional Tesing Service. (ERIC Documen No. ED163090) Monaghan, W. (006). The facs abou subscores (ETS R&D Connecions No. 4). Princeon, NJ: Educaional Tesing Service. Rerieved January 9, 009, from hp:// ions4.pdf Naional Insiue for Tesing and Evaluaion (009). How your scores are used? Rerieved February 3, 009, from hp:// &selFol= &flag=0 Peersen, N. S., Kolen, M. J., & Hoover, H. D. (1989). Scaling, norming, and equaing. In R. L. Linn (Ed.), Educaional Measuremen (3rd ed.). New York: American Council on Educaional Measuremen/Macmillan. Puhan, G., Sinharay, S., Haberman, S. J., & Larkin, K. (008). Comparison of subscores based on classical es heory mehods (ETS Research Repor No. RR-08-54). Princeon, NJ: Educaional Tesing Service. Rudner, L. M., & Taleno-Miller, E. (007, April). Diagnosic esing using decision heory. Paper presened a he annual meeing of he Naional Council on Measuremen in Educaion, Chicago. Ryan, J. M. (003). An analysis of iem mapping and es reporing sraegies. Greensboro, NC: Souh Carolina Deparmen of Educaion. Ryan, J. M. (006). Pracices, issues, and rends in suden es score reporing. In S. M. Downing & T. M. Haladyna (Eds.), Handbook of es developmen. Mahwah, NJ: Lawrence Erlbaum Associaes. Sheehan, K. M., & Mislevy, R. J. (1990). Inegraing cogniive and psychomeric models o measure documen lieracy. Journal of Educaional Measuremen, 7(3), Sinharay, S., Haberman, S. J., & Puhan, G. (007). Subscores based on classical es heory: To repor or no o repor. Educaional Measuremen: Issues and Pracice, 6(4), 1 8. Tae, R. L. (004). Implicaions of mulidimensionaliy for oal score and subscore performance. Applied Measuremen in Educaion, 17(), Wainer, H., Hambleon, R. K., & Meara, K. (1999). Alernaive displays for communicaing NAEP resuls: A redesign and validiy sudy. Journal of Educaional Measuremen, 36(4), Wainer, H., Sheehan, K. M., & Wang, X. (000). Some pahs oward making Praxis scores more useful. Journal of Educaional Measuremen, 37(), Wang, M. W., & Sanley, J. C. (1970). Differenial weighing: A review of mehods and empirical sudies. Review of Educaional Research, 4, Yao, L., & Boughon, K. A. (007). A mulidimensional iem response modeling approach for improving subscale proficiency esimaion and classificaion. Applied Psychological Measuremen, 31(), Yen, W. M. (1987, June). A Bayesian/IRT index of objecive performance. Paper presened a he annual meeing of he Psychomeric Sociey, Monreal, Québec, Canada.

10 Pracical Assessmen, Research & Evaluaion, Vol 14, No 4 Page 10 Noes 1 Iniially, Sinharay e al. (007) also considered he observed subscore S as a predicor. However, hey showed ha S will always be inferior o S s as a predicor of S and herefore he predicor S is no discussed in his sudy. Acknowledgemens The auhor hanks Mas Hamrén for his daa collecion work, and Marie Wiberg, Chrisina Sage, Lawrence Rudner, and hree anonymous reviewers for heir valuable commens on he manuscrip. Ciaion Lyrén, Per-Erik (009). Reporing Subscores from College Admission Tess. Pracical Assessmen, Research & Evaluaion, 14(4). Available online: hp://pareonline.ne/gevn.asp?v=14&n=4. Corresponding Auhor Per-Erik Lyrén Deparmen of Educaional Measuremen (BVM) Umeå Universiy Umeå, Sweden per-erik.lyren [a] edmeas.umu.se