ABSTRACT. Professor Robert J. Mislevy Department of Measurement, Statistics and Evaluation

Transcription

1 ABSTRACT Titl of Documnt: RANDOIZATION-BASED INFERENCE ABOUT LATENT VARIABLES FRO COPLEX SAPLES: THE CASE OF TWO-STAGE SAPLING Tiandong Li Doctor of Philosophy Dirctd By: Profssor Rort J. islvy Dpartmnt of asurmnt Statistics and Evaluation In larg-scal assssmnts such as th National Assssmnt of Educational Progrss NAEP plausil valus asd on ultipl Imputations I hav n usd to stimat population charactristics for latnt constructs undr complx sampl dsigns. islvy 99 drivd a closd-form analytic solution for a fixd-ffct modl in crating plausil valus assuming a classical tst thory modl and a stratifid studnt sampl and proposd an analogous solution for a random-ffcts modl to applid ith a to-stag studnt sampl dsign. Th rsarch rportd hr xtnds th discussion of this random-ffcts modl undr th classical tst thory framor. Undr th simplifid assumption of non population paramtrs analytical solutions ar providd for multipl imputations in th cas of th classical tst thory masurmnt modl and to-stag sampling and thir proprtis ar vrifid in rconstructing population proprtis for th unosrval latnt varials. With th mor practical assumptions of unnon population and clustr mans this study mpirically xamins th rconstruction of population attriuts. Nxt

2 proprtis of sampl statistics ar xamind. Spcifically this rsarch xplors th impact of th varianc componnts and sampl sizs on th sampling varianc of th I-asd stimat for th population man. Findings includ significant prdictors and influntial factors. Last th rlationships tn th sampling varianc of th stimat of th population man asd on th imputations and thos asd on osrvations of th tru scor and th osrvd scor ar discussd. Th sampling varianc asd on th imputd scor is xpctd to th highr oundary of that asd on th osrvd scor hich is xpctd to th highr oundary of that asd on th tru scor.

3 RANDOIZATION-BASED INFERENCE ABOUT LATENT VARIABLES FRO COPLEX SAPLES: THE CASE OF TWO-STAGE SAPLING By Tiandong Li Dissrtation sumittd to th Faculty of th Graduat School of th Univrsity of aryland Collg Par in partial fulfillmnt of th rquirmnts for th dgr of Doctor of Philosophy Advisory Committ: Profssor Rort J. islvy Chair Dr. Fran F. Jnins Profssor Partha Lahiri Profssor Gorg B. acrady Rsarch Profssor Kith F. Rust

4 Copyright y Tiandong Li

5 DEDICATION This dissrtation is ddicatd to my parnts Zuntang Li and Lianin Zhang to my if Lanlan Yin and to my son to orn Alan Li. ii

6 ACKNOWLEDGEENTS First and formost I ish to convy my dpst gratitud to my dissrtation advisor and mntor Dr. Rort islvy ho shod m th ay to com a rsarchr and st an xcllnt xampl for m. His patinc ncouragmnt and nthusiasm mad this dissrtation possil. I ould also li to than Dr. Fran F. Jnins Dr. Partha Lahiri Dr. Gorg acrady and Dr. Kith Rust for thir guidanc and insightful suggstions. I ould not hav gottn this far ithout th continud support of my mployr Wstat and spcifically th support of th dirctor of th statistical group David organstin. Than you for providing m flxiility at or that allod m to accommodat my study ith my full tim jo. Also I ould li to than my collagus and frinds ho providd valual fdacs to my rsarch: Dr. Rort Fay Dr. Graham Kalton Dr. Konrad Non-Trauth and Dr. Judith Strnio. To my parnts Zuntang Li and Lianin Zhang your unconditional lov and continuous ncouragmnt supportd m throughout this long journy of dissrtation or. Than you for your faith in m through all ths yars. I hop that hat I hav accomplishd mas you happy and proud. ost importantly than you Lanlan for vrything you hav don to nsur my achivmnts. Your continuous undrstanding sacrific and lov mad this dissrtation possil. I than your parnts hom I considr my parnts as ll for thir slflss contriution to our family. lif. Finally I ddicat this thsis to our coming son Alan as my first gift in his iii

7 Tal of contnts List of Tals... vi List of Figurs... viii Chaptr : Introduction.... Bacground.... Rsarch Purposs and Qustions.... Organization of th Chaptrs... Chaptr : Litratur Rvi.... Tst Thory..... Classical Tst Thory..... Itm Rspons Thory ultipl atrix Sampling Clustrd Population and Random-Effct odl Complx Survy Sampling.... Randomization-asd Infrnc in Survy Sampling....6 ultipl Imputation for Latnt Varials in Complx Sampl Survys Th Sampling odl Th Population odl Th Latnt Varial odl Assumption for Imputation - issing at Random Th construction of ultipl Imputations for Latnt Varials... 7 Chaptr : ultipl Imputation Approach for Latnt Varials in To-Stag Sampls.... Gnral Form.... Th Cas of Classical Tst Thory... Chaptr 4 : Analytical Solution ith Knon Population Paramtrs Drivation of Expctation and Varianc of Imputd Clustr ans Drivation of Varianc of Within-Clustr Imputations Drivation of an and Varianc of th Imputations for th Clustrd Population... 6 Chaptr : Imputation ith Unnon Population an and Clustr ans Rsarch Qustions of th Simulation Study Construction of Imputations for th Cas of Unnon ans Data Gnration..... anipulatd Factors Population Varianc Componnts... iv

8 ... Sampl Sizs Numr of Imputations Numr of Rpatd Sampls..... Fixd Factor - Population Data Gnration Crat Simulatd Data on Osrvd Tst Scor X Gnrat Imputations Basd on Osrvd Valus Rpatd Sampls Random Sds Analysis Study thod Estimation Rsults Rsarch Qustion : Ho ar diffrnt amounts of varianc rconstruction trms incorporatd in th plausil valus to rcrat th population proprtis of th tru scor? Rsarch Qustion : Ho do th varianc componnts and sampl sizs impact th stimation rror of th I-asd stimat for th population man? 7... Th ratio of th varianc componnts is Th ratio of th varianc componnts is Rsarch Qustion : What ar th rlationships tn th sampling varianc of th imputations and thos of th tru scor and th osrvd scor? 9... Th ratio of th varianc componnts is Th ratio of th varianc componnts is Chaptr 6 : Conclusion Importanc of th Study ajor findings Limitations and Futur Rsarch... 9 Glossary.... Appndix A Rfrncs... v

9 List of Tals Tal. anipulatd Factors... Tal. Larg-sampl rlativ fficincy in % in units of standard dviations hn using a finit numr of propr imputations m rathr than an infinit numr as a function of th fraction of missing information Tal. Population statistics and corrsponding stimators asd on th imputd scors th tru scor and th osrvd scors Tal.4 Th varianc of th varianc rconstruction trms y simulation factors hn th ratio of th varianc componnts is.... Tal. Th varianc of th varianc rconstruction trms y simulation factors hn th ratio of th varianc componnts is Tal.6 Rang and man of th z-valu of th point stimats asd on th imputd scor th tru scor and th osrvd scor across th 8 cominations of simulation factors... 6 Tal.7 List of z-valus of th stimats hr th ias of th stimats asd on th osrvd scor may ngligil z-valus tn -4 and Tal.8 V for th plausil valus y th simulation factors Tal.9 Tal. U for th plausil valus y th simulation factors B for th plausil valus y th simulation factors Tal. Outlying rsiduals snss and urtosis in th rgrssion analysis hn varianc componnts ta valus and Tal. Paramtr stimats P-valus of F-tsts and smipartial s for th rgrssion modls on U B and V hn th ratio of th varianc componnts is... 7 Tal. Paramtr stimats P-valus of F-tsts and smipartial s for th rgrssion modl on th ratio varial U V / hn th ratio of th varianc componnts is Tal.4 V for th plausil valus y th simulation factors Tal. Tal.6 U for th plausil valus y th simulation factors B for th plausil valus y th simulation factors Tal.7 Outlying rsiduals snss and urtosis in th rgrssion analysis hn th varianc componnts ta valus and Tal.8 Paramtr stimats P-valus of F-tsts and smipartial s for th rgrssion modls on U B and V hn th ratio of th varianc componnts is ˆ ˆ ˆ vi

10 Tal.9 Paramtr stimats P-valus of F-tsts and smipartial s for th rgrssion modl on th ratio varials U V / hn th ratio of th varianc componnts is Tal. Th ratio of th sampling varianc asd on th osrvd scor ovr that of th tru scor hn th varianc componnts ta valus and Tal. Th ratio of th sampling varianc asd on th imputd data ovr that of th osrvd scor hn th varianc componnts ta valus and... 9 Tal. Th simulation conditions ith th ratio of th sampling varianc asd on th osrvd scor ovr that of th tru scor lor than for th data ith rpatd sampls rpatd sampls and rpatd sampls hn th varianc componnts ta valus and Tal. Th simulation conditions ith th ratio of th sampling varianc asd on th imputation ovr that of osrvd scors lor than for th data ith and rpatd sampls hn th varianc componnts ta valus and Tal.4 Paramtr stimats P-valus of F-tsts and smipartial ˆ ˆ s for th ratio varials Var /V hn th ratio of th varianc componnts is Tal. Th ratio of th sampling varianc asd on th osrvd scor ovr that of th tru scor hn th varianc componnts ta valus and 4... Tal.6 Th ratio of th sampling varianc asd on th imputd data ovr that of th osrvd scor hn th varianc componnts ta valus and 4... Tal.7 prsnts th paramtr stimats P-valus and smipartial ˆ for th ratio varials Var /V hn th varianc componnts ta valus and vii

11 List of Figurs Figur. : and Figur. : and Figur. : and Figur.4: and vs. sampl sizs K and I y and hr = or = or... 6 U U vs. sampl sizs K and I y and hr = or = or... 6 B vs. sampl sizs K and I y and hr = or = or B vs. sampl sizs K and I y and hr = or = or... 6 Figur.: V vs. sampl sizs K and I y and hr = or and = or Figur.6: V vs. sampl sizs K and I y and hr = or and = or Figur.7: U / V vs. sampl siz K and I y and hr = = or and = or Figur.8: U / V vs. sampl siz K and I y and hr = = or and = or... 7 Figur.9: and U vs. sampl siz K and I y and hr = = or 4 = or Figur.: and U vs. sampl siz K and I y and hr =4 = or 4 = or Figur.: and B vs. sampl siz K and I y and hr = = or 4 = or Figur.: and B vs. sampl siz K and I y and hr =4 = or 4 = or Figur.: V vs. sampl siz K and I y and hr = = or 4 and = or Figur.4: V vs. sampl siz K and I y and hr =4 = or 4 and = or viii

12 Figur.: U / V vs. sampl siz K and I y and hr = = or 4 and = or Figur.6: U / V vs. sampl siz K and I y and hr =4 = or 4 and = or ix

13 Chaptr : Introduction. Bacground In psychomtrics masurmnt modls provid a platform for xplaining thortical latnt constructs undrlying osrvd itm rsponss. Traditional masurmnt modls ar gnrally dvlopd undr th assumption of simpl random sampling SRS of individuals as th structurs of intrst concrn oftn complx ithin-prson pattrns of rspons. Hovr in larg-scal ducational assssmnts such as th National Assssmnt of Educational Progrss NAEP data ar collctd undr complx sampl dsigns hich includ th folloing thr componnts: unqual ights stratification and clustring Rust Krnz Qian & Johnson. In addition to rduc th tst urdn on rspondnts studnts ta only a sust of th tst itms. Th susts ar slctd using th multipl-matrix itm sampling mthod. This itm sampling dsign is handld y applying an Itm Rspons Thory IRT latnt-varial modl to stimat studnt proficincy or aility and th analyst calculats and rports rsults on th scal of th latnt varial. Hovr an fficint IRT stimator of individuals proficincy can sriously iasd in stimating th population distriution of th proficincy scors Lord 99; islvy Baton Kaplan & Shhan 99. To avoid th stimation of individual studnt latntvarial paramtrs hn stimating population charactristics population charactristics can calculatd asd on thir conditional xpctation in marginal analyss islvy 984. In addition this approach can jointly handl th latnt varial modl and complx studnt sampling. Bcaus th closd-form solution for th conditional xpctation is only availal for spcial cass an altrnativ calld plausil valus asd on Ruin s

14 987. ultipl Imputation I has n usd to allo scondary rsarchrs to stimat latnt trait distriutions in larg-scal ducational assssmnts. Although th mthods hav provn usful thy can difficult to undrstand in applications involving oth complx masurmnt modls and sampling dsigns. To provid intuition islvy 99 drivd a closd-form analytic solution for a fixd-ffct modl for I assuming a classical tst thory modl and a stratifid studnt sampl. Th rsults provid insight into th lmnts and proprtis of th procdur and ground intuition for mor complx applications. In th sam articl islvy proposd an analogous solution for a random-ffcts modl to applid ith a to-stag studnt sampl dsign. Hovr no proof or furthr discussion as providd on th charactr of th solution nor has on appard in th susqunt litratur. Rsarch prsntd hr fills in this gap providing analytic drivations for th ncssary componnts of th solution and dmonstrating its proprtis in a rang of circumstancs ith simulatd data. As such provid additional concptual grounding for practitionrs ho dvlop and/or us plausil valus. Th study dsign of NAEP is discussd as a rprsntativ xampl in this study.. Rsarch Purposs and Qustions Hr driv formulas for multipl imputations in th cas of th classical tst thory CTT masurmnt modl and to-stag sampling vrify thir proprtis in rconstructing population proprtis for th unosrval latnt varials or s and mpirically xamin th rconstruction of population attriuts and th proprtis of sampl stimats ith th mor practical assumptions of unnon population and clustr mans.

15 Spcifically th rsarch consists of to parts: Undr th simplifid assumptions of non population paramtrs analytic dmonstration is providd for th construction of multipl imputations of and drivations of dsird proprtis of th imputations for th cas of to-stag clustr sampl dsign. Undr th rlaxd assumption of unnon population and clustr mans simulation-asd dmonstration is providd for th construction of multipl imputations of and xploration of proprtis of th imputations for th cas of tostag clustr sampl dsign.. Organization of th Chaptrs This study is prsntd in six chaptrs. Chaptr to givs a rvi of th rlvant litratur. Th first fiv sctions rifly rvi aspcts - tst thory multipl matrix sampling clustrd population and random-ffct modl complx survy sampling and randomization-asd infrnc in survy sampling - provid th asis of th rsarch framor for this study hich is illustratd in sction.6 - ultipl Imputation for latnt varials in complx sampl survys. Chaptr thr gins th first part of th rsarch rsults ith an analytic discussion of th ultipl Imputation approach for latnt varials in to-stag sampls. It drivs th gnral form of th postrior distriution of I and th spcific cas of classical tst thory. Chaptr four givs th analytical solution hn th population paramtrs ar non to sho th rproduction of population charactristics ithin th I datast structur.

16 Chaptr fiv prsnts th scond part of th rsarch th simulation study of th situation hr th population and clustr mans ar not non. It consists of fiv susctions. Th first discusss th thr major rsarch qustions th simulation study is dsignd to xplor. Th scond sction xplains th construction of imputations for th cas of unnon mans. Nxt th study mthod and data gnration procss ar dscrid. Th fourth sction analyzs th simulation rsults and th fifth sction prsnts th analysis rsults in trms of th thr rsarch qustions posd at th outst of th simulation study. Chaptr six discusss th importanc of this study summarizs th major findings and addrsss th limitations of th study concluding ith som suggstions for futur rsarch. 4

17 Chaptr : Litratur Rvi. Tst Thory Educational tst thory provids statistical and mthodological tools to ma infrnc aout xamins' noldg sills and accomplishmnts. Sinc th first txt on tst thory pulishd y E. L. Thorndi in 94 rsarchrs hav xtndd tst thory from Classical Tst Thory CTT to gnralizaility thory itm rspons thory IRT and th analysis of rlationships among scors from diffrnt tsts including factor analysis structural quations modling and multitrait-multimthod analysis islvy 996. This rsarch uss a straightforard masurmnt modl classical tst thory CTT hich yilds closd-form solutions that support intuition for mor complicatd masurmnt modls such as IRT... Classical Tst Thory Th foundation of CTT as laid y Sparman 94a This modl as xtnsivly prsntd y Gullisn 9 and dvlopd mor rigorously y Lord and Novic 968. As shon in Crocr and Algina 986 th CTT modl nvisions an osrvd tst scor as th composit of to hypothtical componnts a tru scor and a random rror componnt xprssd in th form X i E. i i hr X i rprsnts th osrvd tst scor of th ith xamin; i th individual s tru scor; and E i a random rror componnt. Both X i and E i ar random varials in trms of rpatd osrvations for xamin i and i is a constant for xamin i. Th assumptions of th CTT modls ar as lo:

18 Th man of th random rror is zro E. Th corrlation tn tru and rror scors of a tst for a population of xamins is zro E Th corrlation tn rror scors from to paralll tsts is zro E i E E Undr assumption th rlationship of th variancs of th thr componnts in th CTT modl can shon to. X E Th rliaility cofficint dfind as th ratio of tru scor varianc to osrvd scor varianc can xprssd as XX. X hich shos th proportion of osrvd scor varianc xplaind y th tru scor varianc. In CTT scors ar otaind ovr a larg numr of itms and ar tratd as continuous. Th rliaility cofficint can approximatd y th stimats of th intrnal rliaility across itms.g. Cronach s alpha cofficint. CTT is a longstanding satisfactory mthod usd in th ara of standardizd tsting. An advantag ith CTT is that th modl rlis on a assumptions that ar asy to mt y standardizd tsting procdurs. In addition ith its linar structur and th additional assumption of normally distriutd rrors th CTT modl is rlativly simpl and asy to intrprt. W us this modl in this study for simplicity and th intuition that th rsults provid... Itm Rspons Thory Itm Rspons Thory is ssntially a mathmatical modl for th proaility of a corrct rspons to an itm givn th prson s proficincy paramtr and on or 6

19 mor paramtrs for ach itm islvy 989. Both th prson s proficincy and itm difficultis ar positiond on th sam latnt scals. A major advantag of IRT ovr CTT is th proficincy invarianc intrprtation ith rspct to slction of itms. That is th xpctd studnt proficincy scor is indpndnt of th st of itms administrd to him or hr. This fatur of th modl allos IRT to handl th ultipl atrix Sampl dscrid in th nxt sction. Although this study focuss on a simplr tst thory IRT is th modl that has actually n implmntd in largscal assssmnts including NAEP and hnc motivatd th choic of xrcising th ultipl Imputation discussd in sction.6.. ultipl atrix Sampling Along ith survy sampling of studnts ultipl atrix Sampling of tst itms is idly mployd in ducational assssmnt Educational Tsting Srvic & National Cntr for Education Statistics 999. In ultipl atrix Sampling random susampls of studnts ar administrd susts of th ntir pool of assssmnt itms. This dsign prmits a satisfactory prcision lvl in stimating population charactristics and a complt covrag of th assssmnt framor hil minimizing th tim urdn for ach studnt. Rsarchrs hav shon that population charactristics can stimatd accuratly ithout prcis masurmnt of individual studnts Lord 96; Lord t al. 968; Sirotni & Wllington 977. In fact th population itm-scor man is stimatd most fficintly hn ach studnt in th group is assignd on distinct itm from ach ojctiv rporting ara. Thrfor a highly dtaild curricular valuation ith - ojctivs can implmntd y administring a tst form of vn fr than itms for ach studnt as long as th studnts ho rciv itms from a givn ojctiv ar a rprsntativ sampl. Th 7

20 lngth of th tst is still ithin a rasonal limit. Ths findings in th 97s ld to th us of multipl matrix sampling dsigns in ducational assssmnt for fficintly stimating distriutions of prformanc in th population or supopulations in largscal assssmnts such as NAEP. Th typ of matrix sampling usd y NAEP is calld focusd alancd incomplt loc BIB spiraling. Th focusd part of NAEP s matrix sampling mthod rquirs ach studnt to ansr qustions from only on sujct ara. Th BIB part of th mthod nsurs that studnts rciv diffrnt intrlocing sctions of th assssmnt forms naling NAEP to chc for any unusual intractions that may occur tn diffrnt sampls of studnts and diffrnt sts of assssmnt qustions. Spiraling rfrs to th mthod y hich tst oolts ar assignd to pupils hich nsurs that any group of studnts ill assssd using approximatly qual numrs of th diffrnt vrsions of th oolt Educational Tsting Srvic & National Cntr for Education Statistics 999. Bcaus of BIB spiraling NAEP can sampl nough studnts to otain prcis rsults for ach qustion hil consuming an avrag of aout an hour and a half of ach studnt s tim. Th original NAEP survys in th 97s focusd on itm-lvl rsults. Bginning in th assssmnt of 984 hovr it as dsird to produc distriutions of proficincy in populations and supopulations of studnts. An IRT modl is dsiral in stimating studnt proficincy asd on data from multipl matrix itm sampling. Th numr of itms arrangd for ach studnt is too small to ma an accurat stimat of th proficincy hich typically rangs tn and itms in a givn rporting ara. Hovr population charactristics ar stimatd on IRT scals dirctly from survy rsponss through marginal stimation procdurs as discussd in sction.6.. Th plausil valus providd on pulic us data sts allo 8

21 scondary analysts to rproduc th official stimats and to carry out analyss of thir choic on th NAEP IRT scals.. Clustrd Population and Random-Effct odl Th population of studnts in ducational assssmnts is clustrd ithin naturally occurring organizational units such as classs schools and districts. This study is concrnd ith th population paramtrs in a to-lvl clustrd population. Th traditional population in statistical studis assums indpndnc of osrvations. Hovr hn studnts ar clustrd ithin natural units th rsponss from th sam clustr ar corrlatd ith ach othr in som dgr. For xampl studnts in on school may tnd to achiv highr assssmnt scors than studnts in anothr school in gnral. Thrfor th scors of studnts ar not indpndnt to ach othr. ultilvl modling allos rsarchrs to modl this nonindpndnc and vis th population structur as of potntial intrst Goldstin. Whil this study only dals ith a simpl cas of multilvl modling th so-calld random-ffcts modl and mixd-ffcts modl Elston & Grizzl 96 or th random-intrcpt modl Raudnush & Bry intrst in applying mor complx multilvl modling in larg-scal assssmnts has n incrasing.g. Braun Jnins & Grigg 6. In th simpl to-lvl modl in this study indpndnc ill assumd at th clustr lvl and ithin ach clustr. Th assumptions mad aout th variancs and covariancs ar strongr than a traditional analysis. This study considrs a random-ffcts modl ith qual clustr siz in th clustrd population. Th population varianc structur is assumd to consist of tn-clustr varianc and ithin-clustr varianc. asurmnt rror ill add a third lvl of varianc ithin studnts. 9

22 If clustrs ar indxd y and sujcts ithin a clustr ar indxd y i th osrvd scor X i in a CTT can rrittn as: X i i E i i E i.4 hr i is th tru scor of th ith xamin ithin th th clustr E i is th masurmnt rror of an individual prson ithin clustr and i is th dviation of th individual s tru scor from th clustr man. Th varianc of tru scor i in th population can xprssd in to componnts: tn-clustr varianc and ithin-clustr varianc that is X i in th population can xprssd in thr componnts:. Thus varianc of osrvd scor X. Th random-ffcts modl that islvy 99 proposd shos th distriution of th latnt varial as follos: ~ N.6 and z ~ N.7 i hr is th ovrall population man of th latnt varial i is th clustr man hn th clustr indx z quals and and rprsnt tn-clustr varianc and ithin-clustr varianc for th population. Hnc th distriution of i is as follos: z ~ N.8 i Ths modls sho th mchanism for ho studnt scors ar modld. As statd in th rsarch purposs this study xplors th proprtis of multipl imputations aa plausil valus in trms of rproducing th population statistics of th tru scor

23 shon aov hich includ population man clustr mans and th varianc componnts..4 Complx Survy Sampling As Boc 98 indicatd survy sampling dsigns gaind popularity in fficintly collcting social information during th 96s. Thy had alrady n mployd to collct information aout ducational aspirations and attituds. Whn proprly undrtan a sampl survy provids an ojctiv fficint and valid mthod of otaining th charactristics of an ntir population from only a small part of that population Franl & Franl 987. Complx sampl dsigns fatur at last on of thr componnts: unqual proaility of slction stratification and clustring.g. Cochran 977. Ths dsigns ar usually motivatd y cost constraints and administrativ rasons as ll as stimation accuracy for th population or su-population. In th naturally clustrd population in ducational assssmnts such as studnts in schools trating schools as th first-lvl sampling unit in a multi-stag clustr sampl of studnts savs administrativ costs and travling xpnss y not going to a larg numr of schools hich may only hav a f sampld studnts ach. Although a largr numr of studnts ill ndd to gain th sam accuracy lvl as from a Simpl Random Sampl a clustr dsign ill rduc th numr of schools on has to visit and thrfor proaly rducs th cost of data collction. As an xampl of multi-stag proaility sampling dsign th sampl for th NAEP 998 national assssmnt as dran via four stags of slction Rust t al. trating gographic aras and schools tc. as clustrs: th slction of Countis or groups of countis Primary Sampling Units PSUs; th slction of lmntary and

24 scondary schools ithin PSUs; th assignmnt of sssions y typ and of sampl typs to sampld schools; and 4 slction of studnts ithin schools and thir assignmnt to sssion typs.. Randomization-asd Infrnc in Survy Sampling As pointd out y Cassl Sarndal and Wrtman 977 to compting asic philosophis in th thory of infrnc for finit populations ar dsign-asd infrnc and modl-asd infrnc. Th dsign-asd infrnc ss th primary sourc of randomnss is th proaility ascrid y th sampling dsign to th various susts of th finit population {... N}. On th contrary in modl-asd thory of infrnc in survy sampling th valus associatd ith th N units of th population units ar vid as th ralizd outcom of random varials having an N-dimnsional joint distriution. Randomization-asd infrnc is usd for most of th or in this study. It is th traditional and dominating mod of infrnc in survy sampling folloing th milstons of litratur starting ith Nyman 94 and susqunt or including Hansn Huritz and ado 99 ahalanois 946 Kish 96 and Cochran 977 tc. As discussd in Kalton 98 ith th larg sampls typical of most survys survy practitionrs ar rluctant to us modl-asd stimators of dscriptiv paramtrs caus of th potntial stimation ias rsulting from any misspcifications of th modl. Hovr lmnts of th modl-asd approach ar rquird to implmnt Ruin s multipl imputation schm for latnt varials. For a finit population ith N units indxd y i th valus of a survy itm can dnotd as Y y y y N. To conduct randomization-asd infrnc y i s ar tratd as fixd ut unnon valus. Th statistics of intrst can rprsntd

25 as S SY Z hr Z is th vctor of dsign varials hich ar non for all units for osrvation. Lt IND = I I I N rprsnt th sampl indicator a vctor of random varials hr I i = if unit i in th population is in th sampl and I i = if unit i is not in th sampl. According to a sampl dsign-asd on th proaility of IND an uniasd sampl statistics s s Y Z IND and an stimator of sampling sampl varianc U U Y sampl Z IND can usually constructd. Clustring in th sampl dsign can rflctd y th lind proaility of I i for units in th sampl clustr. Infrncs from sampl statistics s to th population statistics S ar asd on th distriution of s in rpatd sampls of Y sampl undr an idntical sampl dsign. Randomization-asd infrncs ar thn asd on th normal distriution from larg- s S sampl approximations: ~ N. U Th pics of th thoris rvid in sctions.-. provid th asis of th rsarch framor for this study to illustratd in sction.6..6 ultipl Imputation for Latnt Varials in Complx Sampl Survys islvy 99 illustratd th thortical framor for th stimation of distriutions of latnt varials in finit populations hn th sampl is dran undr a complx sampling dsign. Latnt varials in a sampl survy ill tratd as survy varials ith missing valus for all rspondnts. By th natur of th latnt varial modl th assumption of missing at random AR is satisfid. Knoldg aout th latnt varial can fully rflctd y a postrior distriution givn th osrvd data namly th dsign sampling fram varial Z acground survy

26 varials Y and itm rsponss X. To stimat a scalar a crtain function of ths four typs of varials thr uilding locs ar ndd..6. Th Sampling odl Th first uilding loc is calld th sampling modl hich mas a randomization asd infrnc aout th population charactristics S from th sampl statistics. Whn is non th traditional randomization asd infrnc in sampling statistics rlis on th cntral limit thorm. Whn th sampl siz is larg th sampl statistics such as th sampl man hav th folloing distriution: s S Var s ~ N.9 As s cannot calculatd hn don t no th conditional xpctation may possily calculatd instad asd on th prdictiv or postrior distriution of th latnt varial Ruin 977:. hr all varials ar fixd hil Z is non and th valu of X and Y ill com non for sampld units asd on a sampl dsign. This approach mas it possil to stimat population charactristics of th latnt varials such as mans and proportions of studnts aov spcifid proficincy lvls dirctly from th osrvd rsponss avoiding th stps of calculating scors for individual studnts. To otain th prdictiv distriution using Bays Thorm th othr to uilding locs ar ndd th population modl and th latnt varial modl. 4

27 .6. Th Population odl Th population modl assums that th distriution of givn th survy collatral varial Y and th dsign varial Z is of th form hr th unnon paramtr of th distriution is rprsntd y. For xampl islvy 99 thoroughly discussd th fixd ffct modl in th prsnc of collatral survy varials. In modling practic th stratification dsign varials can tratd as collatral survy varials. Th conditional distriution for th fixd ffct population modl is dfind as. hr rprsnts th rgrssion paramtrs of Y on and. ith shoing th proportion of varianc of xplaind y Y. In this study th population modl follos th distriution of th to-lvl clustrd population discussd in sction.. Its paramtrs ar th population man th clustr mans and varianc at ach lvl..6. Th Latnt Varial odl Th latnt varial modl assums that th distriution of th itm rspons X givn th latnt varial is of th form hr th unnon paramtr of th distriution is rprsntd y. As discussd in sction.. this study uss a CTT modl as th latnt varial modl. Th unnon paramtr is th varianc of th rror trm. Using Bays thorm th postrior distriution of can xprssd as a function of th population modl and th latnt varial modl folloing islvy 99:

28 . hr th constant dpnds on and ut not. That is th postrior distriution can drivd from th normalizd product of th lilihood function of hich is th conditional proaility of X givn from th latnt varial modl and th conditional distriution for givn th acground and dsign varials from th population modl. That is.4 Undr th fixd ffct modl givn th sam CTT latnt varial modl and th fixd ffct population modl th postrior distriution is. hr as in Klly 9 and ith th conditional rliaility of X givn Y as..6.4 Assumption for Imputation - issing at Random Undr th framor of I to stimat charactristics of latnt varials in a sampl survy th latnt varials ar tratd as survy varials ith missing valus for all rspondnts. ost I mthods rquir th assumption of missing at random AR Ruin 977. That is th proaility that th osrvation is missing dos not dpnd on th valu of th missing osrvation givn th valus of th osrvd valus and th valu of any acground varials. Whn trating latnt varials as missing this assumption holds y natur as th latnt varials ar missing no mattr hat thir 6

29 valus ar. Thus all noldg aout sujcts latnt varials ar convyd y th prdictiv distriution upon hich th imputation ill asd..6. Th construction of ultipl Imputations for Latnt Varials Basd on th framor dscrid in th prvious sction th population charactristics ar stimatd using th conditional xpctation in th sampling modl. Hovr closd-form solutions for th intgral quations can only calculatd for spcial cass islvy Johnson & urai 99. For xampl th closd form of th postrior distriution is not availal for data analysis hn th latnt varial modl is an IRT masurmnt modl hich is usd in NAEP and othr ducational assssmnts. As an altrnativ mthod stochastic or ont Carlo intgration asd on random dras from postrior distriutions of ach sampld studnt is mployd in stimating th conditional xpctation in ducational assssmnts. Although th dvlopmnt of arov chain ont Carlo CC mthods sm to ovrcom th computational difficultis dscrid aov to a larg xtnt Rao choos to dvlop th postrior distriution in a closd form in a cas in hich this is possil to add insight to th I procss. Also non as ultipl Imputations random dras from postrior distriutions ar carrid out svral tims to form sts of plausil valus. Additionally I or th plausil valus provids complt data sts that th standard statistical mthods can applid to y scondary rsarchrs. With th multiply-imputd data sts ach of th imputd complt data sts is analyzd y standard mthods including randomization-asd stimats of population statistics and accompanying sampling variancs. Infrncs aout statistics of intrst ill mad asd on th comination of stimats of ithin-imputation and tnimputation variancs. 7

30 Spcifically Ruin s stimats for a statistic and its sampling varianc calculatd using I is carrid out as follos for th latnt varial situation modld as.. Estimat th postrior distriution of th paramtrs of th latnt varial modl and of th population modl or.. Crat imputd datasts. a. Randomly dra a valu for th m-th data st from.. For ach rspondnt i in th m-th data st dra a valu from th prdictiv distriution. Th rsulting valus ar th imputd valus.. Using th multipl imputd data calculat th point stimat and varianc of th statistics S. Ruin s formulation of th varianc of a statistic asd on m psudo datasts starts ith th ithin-imputation sampling distriution of th statistic N s m U m s hr m is th point stimat of som statistic of intrst calculatd on imputation U m st m and is th stimat of sampling varianc trating th imputations as if thr r non tru valus. Th folloing statistics can calculatd: man of th s stimats m U m and ithin imputation stimat of sampling varianc as avrags ovr psudo data sts tn imputation varianc B and total varianc V hr 8

31 9 s s m U U m.6 s s B m and B U V / Th dvlopmnt of th form of th imputation for this study is providd in chaptr.

32 Chaptr : ultipl Imputation Approach for Latnt Varials in To-Stag Sampls islvy 99 proposd th multipl imputation mthod along ith th to-stag random-ffct modl hich as suggstd to rproduc population varianc componnts of th tru scor. As no dtaild discussion as providd aout ho this imputation as formd this chaptr discusss this issu xplicitly.. Gnral Form For th to-stag sampl th population modl can rittn as to lvls: Th clustr lvl modl:. Th xamin lvl modl:. For a givn form of th latnt varial modl can construct th postrior distriution from hich th multipl imputations ill dran.. Th Cas of Classical Tst Thory As this study mploys th CTT modl and th clustrd population th postrior distriution of i can uilt in to stags: th postrior distriution of th clustr man of th tru scor conditioning on th individual osrvd scors in clustr and highr lvl paramtrs including and ; and th postrior distriution of th tru scor of individual prson conditioning on th clustr

33 man all individual osrvd data and highr lvl paramtrs including and. Givn th latnt varial modl xprssd as lo. and.4 hr I is th sampl siz of th clustr and th population modl shon in sction. normal postrior distriutions can drivd. Within clustrs th postrior distriution is. and tn clustrs.6 Givn th normal assumption of th population modl and th latnt varial modl at oth stags th postrior distriutions ar rsolvd to for th tru scor of individual prson ithin clustrs.7 hr is th ithin-clustr xamin-lvl rliaility cofficint; and for th clustr man of th tru scor.8 hr is th clustr-lvl rliaility cofficint and I is th numr of sujcts in a clustr. Basically th postrior man at oth th individual lvl and th clustr lvl is a ightd avrag of th population man and th man of th appropriat osrvd data.

34 An imputation for th clustr man is x g g hr is a random dra from N and an imputation for th latnt varial i is x i [ x g ] f i.9 f hr i is dran from N g. Random trms and f i ar dran from normal distriutions ith variancs qual to th postrior variancs of and i rspctivly. By adding ths to trms th variancs of th imputations for clustr mans and for individual scors ar uniasd. Ths to trms ar rfrrd to as varianc rconstruction trms in th rst of th thsis. Th nxt chaptr drivs formulas to sho th uniasdnss of stimats asd on th imputation. That is th xpctd valus of imputations so constructd hn population paramtrs ar non corrctly rproduc th population latnt-varial charactristics.

35 Chaptr 4 : Analytical Solution ith Knon Population Paramtrs At th first stag of or construct imputations ith th highr lvl paramtrs and s tratd as non in ordr to dmonstrat th rproduction of population charactristics ithin th I datast structur. islvy 99 dmonstratd that th us of ithr maximum lilihood or Baysian stimats for individuals θs producd iasd rsults for population varianc componnts. Th sam papr proposd an approach to gnrating multipl imputations in th to-stag random-ffcts modl hich r suggstd to rproduc varianc componnts ut no proof has vr n shon in th litratur. Th rsarch in this dissrtation provids rsults for th random-ffcts modl that ar analogous to islvy s analysis rsults for th fixd ffcts modl. This chaptr shos that th ithin clustr stimator ~ and population stimator ~ i usd in th imputation ar uniasd hn th population paramtrs and ar non. Th postrior distriution of givn th osrvd scors and non paramtrs is drivd analytically. Th rsulting quations illustrat th dsiral proprtis of th imputations for th cas of a to-stag clustr sampl dsign. Th rsult can provid intuitiv undrstandings for mor complx cass.

36 4 4. Drivation of Expctation and Varianc of Imputd Clustr ans Th proof in this sction shos that th xpctd valu and varianc of th imputd clustr man ~ ar uniasd stimats of population man and tnclustr varianc. ] [ ~ g E x E g x E E 4. and / / / / / ] [ ~ I I I I I g Var E Var Var Var g Var E Var g Var x Var g x Var Var 4. Th drivation of th varianc of th imputd clustr man ~ also shos ho th varianc componnts ar rflctd in this statistic. Th clustr lvl rliaility cofficint rprsnts th shrinag of th clustr man stimats asd on th postrior stimats x toard th population man and th lvl of

37 shrinag of th varianc. By construction th varianc of th random componnt is th postrior varianc hich is qual to th portion of th varianc shrun. or shrinag toard th population man corrsponds to a rlativly largr proportion of tn-clustr varianc from th random addd trm. According to th dfinition g of rlativly largr varianc ithin clustrs and masurmnt rror varianc corrspond to a largr proportion of tn-clustr varianc from th random componnt Ruin s I stimats. g hnc a largr sampling varianc of th clustr man asd on Th ithin-clustr sampl siz I is anothr factor in this formula a largr clustr siz mas largr hnc th proportion of varianc from th random componnt g smallr. 4. Drivation of Varianc of Within-Clustr Imputations Within clustr population can tratd as a population ithout clustring. Th ithin-clustr varianc calculatd as follos provs that th ithin-clustr varianc of imputations is an uniasd stimat of th population ithin-clustr varianc. ~ Var Var[ x i i' Var x ' i ' ' Var f i ' i ' f ] 4. Th drivation of th varianc of th imputd individual scors ithin a clustr ~ ' also shos ho th varianc componnts ar rflctd in this statistics. i

38 Th ithin-clustr rliaility cofficint rprsnts th shrinag of th stimat of individual scors asd on th postrior stimats x toard th i population clustr man and th lvl of shrinag of th varianc. By construction th varianc of th random componnt f i is th postrior varianc hich is qual to th portion of th varianc shrun. or shrinag toard th population clustr man corrsponds to gratr proportion of ithin-clustr varianc from th random addd trm. According to th dfinition of rlativly largr masurmnt rror varianc corrspond to largr proportion of ithin-clustr varianc from th random componnt f i hnc a largr sampling varianc for th individual scors ithin clustrs asd on Ruin s I stimats. 4. Drivation of an and Varianc of th Imputations for th Clustrd Population This proof shos that th xpctd valu and varianc of th imputations ar uniasd stimats of th man and varianc of th population ith a clustrd structur. ~ E i E{ x E x i i [ x [ E x g ] f i E g } ] E f i 4.4 and 6

39 7 ]/ [ ]/ cov [var ] [ ] cov[ ] [ } ] [ { ~ q p i i i i i i i i i I n I q p x x I x f Var g x Var x Var g x x f Var g x Var x Var f g x x Var Var 4. As shon aov th drivation of th varianc of th imputd individual scors in a clustrd population comins th rsults from sctions 4. and 4.. Th rliaility cofficints from ach stag and rprsnt th shrinag at th stag. or shrinag corrsponds to a highr proportion of varianc from th random addd trm at th corrsponding stag. In th nxt chaptrs ill furthr study th impact of th varianc componnts to th sampling varianc of th man of th imputation in a mor complx cas imputation ith unnon population man and clustr mans. In this mor complx cas th analytic rsults r too unildy to driv dirctly so a simulation study as conductd.

40 Chaptr : Imputation ith Unnon Population an and Clustr ans To achiv th goal of this study a simulation study as dsignd and carrid out hich not only allod prfct control of factors undr considration in th stimation procdur ut also mad it possil to compar th stimats to th tru population valus. This chaptr dscris th mthodological framor and th application procss of th simulation study in fiv sctions. Th first sction stats th thr rsarch qustions xplicitly. Th scond sction xtnds th construction of multipl imputations to th cas in hich nithr nor s ar non. This amounts to adding stags of Baysian stimation for ths highr-lvl paramtrs and draing random valus from th postrior distriution in th construction of ach I data st. In sction. th gnration of multipl data sts of imputd θs undr a clustr sampl dsign ith qual clustr siz using I is dscrid in dtail including discussions of th manipulatd factors th fixd population and th actual gnration of simulatd data. Sction.4 spcifis an analysis mthod asd on th simulatd data. Finally Sction prsnts analysis rsults to addrss th thr rsarch qustions. Th data gnration and analyss r carrid out using th R languag and icrosoft Excl.. Rsarch Qustions of th Simulation Study Th purpos of th simulation study is to xamin th proprtis of stimats of population charactristics otaind from I in th cas of unnon population man and clustr mans. Th statistics of intrst includ th point stimats of th population man clustr mans ovrall population varianc and tn-clustr and 8

41 poold-ithin-clustr variancs asd on th plausil valus for tru scors. Th study addrsss th folloing rsarch qustions:. Ho ar diffrnt amounts of varianc rconstruction trms incorporatd to construct ach st of plausil valus to rcrat th population proprtis of th tru scor?. Ho do th varianc componnts and sampl sizs impact th sampling varianc of th I-asd stimat for th population man?. What ar th rlationships tn th sampling varianc of th stimat of th population man asd on th imputations and thos asd on osrvations of th tru scor and th osrvd scor?. Construction of Imputations for th Cas of Unnon ans In chaptr 4 th population paramtrs and non for th imputation gnratd in th random-ffcts modl ith r assumd to masurmnt rrors. Ths simplifying assumptions allod us to driv closd forms of th stimats that illustratd th proprtis of th imputations. Th rlationships illustratd in ths calculations add insight to th structur and maning of th lmnts usd in th construction of imputations. Hovr ths population-lvl paramtrs ar alays unnon in practic although thy may stimatd from th information in prvious rsarch and currnt data. To invstigat th modl ith unnon population paramtrs a Baysian mthod as applid in a simulation study. In this part of th rsarch too into account that th location paramtrs namly th population man and th clustr mans ar not non hil still 9

42 ping and non. In practical applications ths varianc componnts ill nd to stimatd concurrntly or from prvious rsarch. Analyss ith unnon varianc componnts rmain a topic for futur study. Simpl closd-form drivations ar no longr availal ut y using ll-chosn simulations can furthr dmonstrat additional proprtis of th imputations and add additional insight for potntial usrs for th CTT cas as ll as for cass that ar mor complx and lss transparnt such as itm rspons thory modls. In invstigating th modl ith unnon population location paramtrs a Baysian procdur ith non-informativ prior distriution on ths paramtrs as considrd. As Glman t al. 4 indicatd y using noninformativ prior distriutions infrncs ar not affctd y information xtrnal to th currnt data. This mthod can approximatd y stimating population paramtrs asd on th osrvd data. Spcifically th population man as stimatd ith a sampl man from th simulatd data and th varianc of th sampl man as calculatd hr th stimat of is dnotd y ˆ and th varianc of th stimat y Vˆ ˆ x. Thn plausil valus for r constructd y draing a valu from this postrior distriution for th sampl man in a normal approximation for ach psudo datast of plausil valus. That is for ach psudo datast m a random numr ~ m from ˆ x N ˆ Vˆ as dran. By doing this th I procdur uilt th appropriat amount of uncrtainty in stimating th unnon into th construction of th plausil valus.

43 . Data Gnration Th simulation study cratd imputd data sts for a varity of contrasting conditions ith notaly diffrnt sizs of varianc componnts and sampl sizs at diffrnt lvls of th dsign. In particular th folloing varials r cratd squntially y randomly draing from corrsponding distriutions: to gnrat a data st of sampld X cratd clustr mans of tru scors individual tru scors i and individual osrvd scors X i ; to produc sts of imputations asd on ach data st of X for ach of m psudo data sts of plausil valus cratd ~ ~ imputd clustr mans of individual scors m and imputd individual scors i m hr th suscripts indicat th m th imputd scor for th i th simul in clustr. Th ntir procss dscrid aov as rpatd a larg numr of tims to crat rpatd sampls y using diffrnt random sds in th random dra at ach stp of slction. As a rsult for ach rpatd sampl a st of i X ~ ~ i m and i m r cratd and m data sts of plausil valus r savd in m data sts... anipulatd Factors Whn gnrating simulation data to rflct contrasting conditions in th study th manipulatd factors includ four groups: varianc componnts sampl sizs th numr of imputations and th numr of rpatd sampls. Th valus of ths factors ar summarizd in th tal lo. Cominations of varianc componnts and sampl sizs ar usd for ach of th thr rsarch qustions. Th numr of imputations and numr of rpatd sampls ar slctd among th conditions to ffctivly addrss ach rsarch qustion. Not that th full cross-classification of th

44 factors mntiond aov is not usd for ach rsarch qustion. Th chosn conditions ar discussd in mor dtail in th corrsponding sctions. Tal. anipulatd Factors Factors # of Conditions Dscription Varianc componnts = Sampl sizs Numr of clustrs K 4 Clustr siz I 4 # of imputations # of rpatd sampls... Population Varianc Componnts A id rang of ratios tn varianc componnts as slctd for th simulation study. This rang mor than covrs commonly osrvd ratios in social rsarch y addrssing a id numric rang of th ratio. Th ratios ar rflctd y th valus of th componnts as th aslin condition sts all thr varianc componnts to qual to on hich is rprsntd y =. Othr conditions sho inflation of crtain componnts hich ar rprsntd y = and. Th implications of ths vry diffrnt structurs for th lmnts of imputation ill pointd out. In addition to th xtrm valus of th ratios shon aov to rprsnt situations commonly sn in practic th simulation also usd a st of modrat valus hr th ratio of th varianc componnts is 4. Spcifically th thr

45 varianc componnts ar st to = and 4 4. All th cominations of varianc componnts r usd in th invstigation of all thr rsarch qustions. In xamining rsarch qustion to this id rang of varianc ratios as usd to fully valuat th ffct of th rlativ siz of th varianc componnts in th population on Ruin s I-asd stimat of th varianc of th population man stimats. For othr rsarch qustions ths ratios rprsnt a sufficint covrag of possil situations.... Sampl Sizs Folloing th sam schm as for th varianc componnts sampl sizs in th simulation study r slctd to rprsnt a larg rang of valus covring mor than normally osrvd in social rsarch. For xampl smallr sampl sizs could appar in practic spcially hn analyzing su-domains of th population. To rflct such cass th minimum sampl siz is st to at oth sampling stags. As this study assums normal distriution at oth lvls of th clustrd population sampling distriution ith small sampl sizs is also normal. This study usd th cominations of sampl sizs ith and at ach sampling stag; that is K = and I = ; K = and I = ; ; K = and I = ; ; K = and I = hr K is th numr of clustrs and I is th numr of osrvations ithin ach clustr. Ths cominations of sampl sizs r usd for all thr rsarch qustions. Ths sttings impact th rliaility of clustr mans and th prcision of sampl stimats.

46 ... Numr of Imputations To dtrmin th numr of imputations ndd for applications Ruin 987 p.4 illustratd th rlationship tn th numr of imputations and rlativ fficincy RE of th stimator from I as follos: ~ s i ~ s m i m. Dfind as th fficincy hn using a finit numr of propr imputations m rathr than an infinit numr RE can xprssd as a function of th xpctd fraction of missing information and th numr of imputations m. ~ rprsnts th conditional varianc of point stimats asd on s i an infinit numr of imputations of i givn th osrvd x i and ~ rprsnts th conditional varianc asd on m imputations. s m i m For point stimats in a larg sampl th REs achivd for various valus of m and rats of missing information ar shon in Tal. Ruin 987 p.4. Tal. Larg-sampl rlativ fficincy in % in units of standard dviations hn using a finit numr of propr imputations m rathr than an infinit numr as a function of th fraction of missing information. m