A Comment on Variance Decomposition and Nesting Effects in Two- and Three-Level Designs

Transcription

1 DISCUSSION PAPER SERIES IZA DP No A Commnt on Varianc Dcomposition and Nsting Effcts in Two- and Thr-Lvl Dsigns Spyros Konstantopoulos Novmbr 007 Forschungsinstitut zur Zukunft dr Arbit Institut for th Study of Labor

2 A Commnt on Varianc Dcomposition and Nsting Effcts in Two- and Thr-Lvl Dsigns Spyros Konstantopoulos Northwstrn Univrsity and IZA Discussion Papr No Novmbr 007 IZA P.O. Box Bonn Grmany Phon: Fax: Any opinions xprssd hr ar thos of th author(s) and not thos of th institut. Rsarch dissminatd by IZA may includ viws on policy, but th institut itslf taks no institutional policy positions. Th Institut for th Study of Labor (IZA) in Bonn is a local and virtual intrnational rsarch cntr and a plac of communication btwn scinc, politics and businss. IZA is an indpndnt nonprofit company supportd by Dutsch Post World Nt. Th cntr is associatd with th Univrsity of Bonn and offrs a stimulating rsarch nvironmnt through its rsarch ntworks, rsarch support, and visitors and doctoral programs. IZA ngags in (i) original and intrnationally comptitiv rsarch in all filds of labor conomics, (ii) dvlopmnt of policy concpts, and (iii) dissmination of rsarch rsults and concpts to th intrstd public. IZA Discussion Paprs oftn rprsnt prliminary work and ar circulatd to ncourag discussion. Citation of such a papr should account for its provisional charactr. A rvisd vrsion may b availabl dirctly from th author.

3 IZA Discussion Papr No Novmbr 007 ABSTRACT A Commnt on Varianc Dcomposition and Nsting Effcts in Two- and Thr-Lvl Dsigns Multilvl modls ar widly usd in ducation and social scinc rsarch. Howvr, th ffcts of omitting lvls of th hirarchy on th varianc dcomposition and th clustring ffcts hav not bn wll documntd. This papr discusss how omitting on lvl in thr-lvl modls affcts th varianc dcomposition and clustring in th rsulting twolvl modls. Spcifically, I usd th ANOVA framwork and providd rsults for simpl modls that do not includ prdictors and assumd balancd nstd data (or dsigns). Th rsults ar usful for tachr and school ffcts rsarch as wll as for powr analysis during th dsigning stag of a study. Th usfulnss of th mthods is dmonstratd using data from Projct STAR. JEL Classification: C00 Kywords: varianc dcomposition, nstd dsigns, clustring Corrsponding author: Spyros Konstantopoulos School of Education and Social Policy Northwstrn Univrsity 10 Campus Driv Evanston, IL 6008 USA spyros@northwstrn.du

4 Many populations of intrst in ducation, psychology, and th social scincs hav multilvl structur (.g., studnts ar nstd within classrooms and classrooms ar nstd within schools, individuals ar nstd within nighborhoods, which ar nstd within citis). Bcaus individuals within aggrgat units (.g., classrooms or schools) ar oftn mor alik than individuals in diffrnt units, this nstd structur producs what is calld in th sampling litratur clustring ffcts (s,.g., Kish, 1965). First, clustring ffcts nd to b takn into account whn analyzing data with nstd structurs. For xampl, on shortcoming of ignoring dpndncis in th data is that th stimatd standard rrors of th rgrssion cofficints ar typically undrstimatd, lading to libral tsts of significanc and an inflatd probability of making a Typ I rror. In sampling mthodology th clustring ffcts ar capturd by th dsign ffct that is usd to corrct th standard rrors of rgrssion stimats (s Cochran, 1977; Kish, 1965; Lohr, 1999). In ducation and th social scincs th ffcts of clustring hav bn wll addrssd by multilvl modls th last 0 yars (Goldstin, 003; Longford, 1993; Raudnbush & Bryk, 1986, 00; Snijdrs & Boskr, 1999). Such modls tak into account th clustring ffcts in th stimation of th standard rrors of th rgrssion cofficints. Espcially in twolvl modls with on lvl of clustring rsarchrs hav shown th dgr of undrstimation of th standard rrors of th stimats that taks plac whn on ignors th dpndncy of th data at th scond lvl (.g., Raudnbush & Bryk, 1986, 00). Scond, clustring ffcts nd to b takn into account whn dsigning studis with nstd structur that do not follow a simpl random sampling schm. Multilvl or nstd dsigns ar asily undrstood by rcognizing th sampling usd at diffrnt lvls of th population hirarchy. For xampl, th clustring ffct in a two-lvl dsign that follows a twostag clustr sampling (.g., sampl schools in th first stag, and thn sampl studnts within

5 ths schools at th scond stag) is typically dfind via an intraclass corrlation (s Cochran, 1977; Lohr, 1999). This intraclass corrlation is involvd in computations of statistical powr (an important aspct of study dsign) that ar prformd at th dsigning stag of a study (s Donnr & Klar, 000; Hdgs & Hdbrg, 007, Murray, 1998; Raudnbush & Liu, 000). In two-lvl dsigns with on lvl of clustring, rsarchrs hav documntd th importanc of including th intraclass corrlation in powr computations and hav discussd th ovrstimation in statistical powr that taks plac whn on ignors th ffct of clustring at th scond lvl (Hdgs & Hdbrg, 007; Murray, 1998; Raudnbush & Liu, 000; Snijdrs & Boskr, 1999). Although two-lvl modls ar common practic in ducation and th social scincs, statistical analyss do not always involv two lvls. For xampl, ducational rsarchrs hav dmonstratd th usfulnss of applying thr-lvl modls to nstd achivmnt data (s.g., Bryk & Raudnbush, 1988; Ny, Konstantopoulos, & Hdgs, 004; Rowan, Corrnti, & Millr, 00). Ths studis hav shown mpirically that thr ar important clustring ffcts at th scond and at th third lvl of th hirarchy. In addition, in study dsign clustring ffcts can tak plac at th scond and at th third lvl of th hirarchy. Considr a thr-lvl dsign that follows a thr-stag clustr sampling (.g., sampl schools in th first stag, sampl classrooms in th scond stag, and thn sampl studnts within ths classrooms at th third stag). Th clustring ffcts ar in this cas typically dfind via two intraclass corrlations, on at th scond and on at th third lvl (s Cochran, 1977; Lohr, 1999). Ths intraclass corrlations ar involvd in computations of statistical powr that ar prformd at th dsigning stag of thr-lvl studis with two lvls of nsting (s.g., Konstantopoulos, in prss). Whn a thr-lvl dsign follows a thr-stag sampling schm th total varianc in th outcom is dcomposd into thr componnts: th within lvl- btwn lvl-1 unit (.g.,

6 btwn studnts within classrooms) varianc, σ ; th within lvl-3 btwn lvl- unit (.g., btwn classrooms within schools) varianc, τ ; and th btwn lvl-3 unit (.g., btwn schools) varianc, ω. Thn, th total varianc in th outcom is dfind as σ σ τ ω T = + +. (1) In such thr-lvl dsigns two intraclass corrlations ar ndd to dscrib th varianc componnt structur. Ths ar dfind as th scond lvl intraclass corrlation: τ ρ = σ T () and th third lvl intraclass corrlation ω ρ = 3 σt (3) whr th subscripts and 3 indicat th lvl of th hirarchy. Howvr, it is not uncommon in practic to omit on lvl and trat data (or dsigns) with thr sourcs of variation (.g., within-classroom, btwn-classroom, and btwn-school) as data (or dsigns) with two lvls of variation (.g., within-school, btwn-school variation, or within-classroom, btwn-classroom variation). That is, frquntly, analyss of nstd data ar conductd without including all lvls of th hirarchy. Omitting a lvl of th hirarchy in analyss is somtims a mattr of convninc and othr tims a ncssity. In ducation, for instanc, classroom idntifirs ar not always availabl and thus th ducational rsarchr in such cass may conduct analyss mploying two-lvl modls (whr studnts ar nstd within schools). Similarly, in th dsigning phas of a study somtims information about clustring ffcts (such as intraclass corrlations) is not availabl for all lvls of th hirarchy. In such

7 cass, powr analyss ar conductd omitting on (or mor) lvls of th hirarchy du to lack of information. Whn on sourc of variation (or clustring) is ignord in thr-lvl modls, ithr at th scond or at th third lvl, th rmaining two lvls of variation hav to absorb that variation, bcaus th total variation in th outcom rmains constant. Howvr, th mchanism of th varianc dcomposition among two- and thr-lvl data (or dsigns) is not that clar. In particular, considr data (or study dsigns) that follow a thr-lvl nstd structur with two lvls of nsting. Suppos that th rsarchr trats th data (or th dsign) mploying two-lvl modls, by ignoring ithr th scond or th third lvl. Howvr, omitting a lvl will affct th stimats of th varianc componnts and th clustring ffcts for th rmaining lvls. This papr uss ANOVA rsults for balancd dsigns and provids drivations about how th varianc dcomposition in thr-lvl modls is changd whn only two-lvls of th hirarchy ar takn into considration. I discuss th simplst multilvl modl whr no covariats ar includd at any lvl for two distinct cass. First I show how th varianc dcomposition taks plac whn th middl lvl is omittd, and thn I show how th varianc dcomposition taks plac whn th third lvl is ignord. A Two-Way Nstd Random Modl Suppos that th data (or th dsign) follow indd a thr-lvl structur with two lvls of nsting (at th scond and third lvl). Considr a thr-lvl unconditional modl with no covariats at any lvl of th hirarchy. Within th ANOVA framwork this is a two-way nstd random modl (.g., studnts ar nstd within classrooms, and classrooms ar nstd within

8 schools). Th structural modl quation for th l th lvl-1 unit in th k th lvl- unit in th j th lvl-3 unit is Y = μ + β + γ + ε, (4) jkl j jk jkl whr μ is th grand man, β j is th random ffct of th lvl-3 unit j (j = 1,, m), γ jk is th random ffct of lvl- unit k (k = 1,, p) within lvl-3 unit j, and ε jkl is th rror trm of th lvl-1 unit l (l = 1,, n) within lvl- unit k, within lvl-3 unit j. Th lvl-1, lvl-, and lvl-3 random ffcts ar normally distributd with a man of zro and variancs σ, τ, and ω rspctivly. Following Kirk (1995) and Sarl, Caslla, and McCullogh (199) I dfin th total sums of squars in this thr-lvl modl as SS = SS + SS + SS 3 (5) T 1 whr th subscripts 1,, and 3 indicat th lvl of th hirarchy. Th xpctd valu of th sums of squars at th first lvl is ESS ( 1) mpn ( 1) σ, (6) = th xpctd valu of th sums of squars at th scond lvl is ESS ( ) = mp ( 1)( σ + n τ ), (7) and th xpctd valus of th sums of squars at th third lvl is ESS ( ) = ( m 1)( σ + nτ + pn ω ), (8) 3 assuming m lvl-3 units, p lvl- units, and n lvl-1 units. Cas A: Omitting th Scond Lvl of th Hirarchy First, suppos that th scond lvl (.g., classroom) of th thr-lvl structur is omittd, and th modl is rducd to two-lvls (.g., studnt and schools). Thn, th sums of

9 squars at th scond lvl (.g., school) and at th first lvl (.g., studnt) of thr rsulting twolvl modl ar dfind as and SS = SS 3, (9) SS 1 = SS1+ SS (10) rspctivly. Th objctiv is to comput th xpctd valus of th first and scond lvl variancs 1, σ ω. Spcifically, using quations 7, 8, and 10 th xpctd valu of th first lvl varianc σ 1 is ESS ( 1) + ESS ( ) mp ( 1)( σ + nτ ) + mpn ( 1) σ np ( 1) E( σ 1 ) = = = σ + τ. (11) m( pn 1) m( pn 1) pn 1 Th abov quation indicats that whn th middl lvl (in a thr-lvl structur) is omittd th first lvl varianc in th rsulting two-lvl modl is th sum of th first lvl varianc and a portion of th scond lvl varianc in th thr-lvl modl. Notic that whn n (.g., th numbr of studnts within ach classroom) bcoms infinitly larg th trm np ( 1) τ 0 pn 1 tnds to zro, and whn p (.g., th numbr of classrooms pr school) bcoms infinitly larg th trm np ( 1) τ τ tnds to τ. This suggsts that whn th numbr of lvl-1 units is pn 1 larg and th numbr of lvl- units is small (in a thr-lvl structur) th middl lvl varianc dos not affct much th first lvl varianc of th rsulting two-lvl modl. Similarly, sinc EMS ( ) EMS ( 1) E( ω ) = pn (1)

10 and ESS ( ), ( m 1) 3 = EMS3 = EMS ( ) ( ) 1 ESS ( ) EMS ( 1) = mpn ( 1) (13) and using quations 8 and 11 th xpctd valu of ω is ESS ( 3)/( m 1) E( σ 1 ) n 1 E( ω ) = = ω + τ. (14) pn pn 1 Th abov quation indicats that whn th middl lvl (in a thr-lvl structur) is omittd th scond lvl varianc of th rsulting two-lvl modl is th sum of th third lvl varianc and a portion of th scond lvl varianc in th thr-lvl modl. Notic that whn n (.g., th numbr of studnts within ach classroom) bcoms infinitly larg th trm n 1 τ 1 pn 1 tnds to on, and whn p (.g., th numbr of classrooms pr school bcoms infinitly larg th trm n 1 pn 1 0 τ tnds to zro. This suggsts that whn th numbr of th middl-lvl units (in a thr lvl structur) is larg th middl-lvl varianc τ dos not affct much th scond lvl varianc of th rsulting two-lvl modl. Howvr, whn th numbr of lvl-1 units (in a thr-lvl structur) is larg th scond lvl varianc of th rsulting two-lvl modl is th sum of th scond and th third lvl variancs in th thr-lvl modl. Notic that th sum of quations 11 and 14 is np ( 1) n 1 ( ws ) + ( bs E σ E ω ) = σ + τ + ω + τ = σ + τ + ω. pn 1 pn 1 It is straightforward to driv th clustring ffct in this cas. Suppos that th nsting ffct is xprssd via an intraclass corrlation ρ. Thn, using quation 11 and 14 it follows that

11 n 1 ω + 1 τ ω pn n 1 ρ = = = ρ 3+ ρ, (15) σ σ + τ + ω 1 1 ω pn + which indicats that whn th numbr of lvl-1 units n (in a thr-lvl structur) is quit larg th intraclass corrlation in th rsulting two-lvl modl is th sum of th intraclass corrlations at th scond and at th third lvl in th thr-lvl modl. Howvr, whn th numbr of th middl-lvl units p (.g., classrooms) is quit larg th intraclass corrlation in th rsulting two-lvl modl is simply th intraclass corrlation at th third lvl in th thr-lvl modl. Cas B: Omitting th Third Lvl of th Hirarchy Scond, suppos that th third lvl (.g., school) in th thr-lvl structur is omittd, and th modl is rducd to two-lvls (.g., studnts and classrooms). Thn, th sums of squars at th scond lvl (.g., classroom) and at th first lvl (.g., studnt) of th rsulting two-lvl modl ar dfind as and SS = SS + SS 3, (16) SS 1 = SS 1 (17) rspctivly. Th objctiv is to comput th xpctd valus of th first and scond lvl variancs σ 1, τ. Th xpctd valu of th first lvl varianc in th rsulting two-lvl modl is simply th first lvl varianc in th thr-lvl modl, namly 1 E( σ )= σ. (18) Similarly, sinc

12 ( ) ( 1) ( )/( 1) ( ( EMS EMS ESS mp Eσ 1) E τ ) = = (19) n n and ESS ( ) = ESS ( ) + ESS ( 3 ), (0) and using quations 6, 7, and 18, and 0 th xpctd valu of τ is { σ τ σ τ ω } mp ( 1)( + n ) + ( m 1)( + n + pn ) /( mp 1) E( τ ) = σ n n which rducs to 1 ( p E τ ) = τ + 1 ω. (1) mp 1 Th abov quation indicats that whn th third lvl (in a thr-lvl structur) is omittd, th scond lvl varianc in th rsulting two-lvl modl is th sum of th scond lvl varianc and a portion of th third lvl varianc in th thr-lvl modl. Notic that whn p (.g., th numbr of classrooms within ach school) bcoms infinitly larg th trm p 1 ω mp tnds to zro, and whn m (.g., th numbr of schools) bcoms infinitly larg th trm p 1 1 ω ω tnds to mp 1 ω. This suggsts that whn th numbr of th middl-lvl units (in a thr-lvl structur) is larg th third lvl varianc dos not affct much th scond lvl varianc in th rsulting two-lvl modl. Howvr, whn th numbr of lvl-3 units (in a thrlvl structur) is larg th scond lvl varianc in th rsulting two-lvl modl is th sum of th scond and third lvl variancs in th thr-lvl modl. Also, notic that in this cas th first lvl varianc in th thr-lvl modl and th first lvl varianc in th rsulting two-lvl

13 modl ar th sam. As with th prvious cas A, suppos that th clustring ffct is xprssd via an intraclass corrlation ρ. Thn, using quations 18 and 1 it follows that ρ p 1 τ ω τ mp = = σ1 + τ p 1 σ + τ ω mp, () which indicats that whn th numbr of lvl-3 units is quit larg th intraclass corrlation in th rsulting two-lvl modl is th sum of th intraclass corrlations at th scond and third lvls in th thr-lvl modl. Exampl To show th usfulnss of th mthods prsntd in this papr I usd data from Projct STAR (s Ny, Hdgs, & Konstantopoulos 000). Spcifically I ran a thr-lvl unconditional modl (no covariats at any lvl) using kindrgartn data from projct STAR to modl mathmatics achivmnt. I standardizd th outcom so that its total varianc is on. Thr wr about 18 studnts pr classroom (n = 18), four classrooms pr school (p = 4) and 79 schools (m = 79). Th rsults of th thr-lvl analysis indicatd that th variancs at th first, scond, and third lvls wr rspctivly 0.709, 0.16, First, suppos that th middl lvl (.g., classrooms) is omittd. Thn, using quations 11 and 14 I computd th first and scond lvl varianc of th rsulting two-lvl modl (.g., studnts nstd within schools) as 18(4 1) σ 1 = = 0.805, 7 1 and

14 18 1 ω = = Th two-lvl HLM analyss of th data providd almost idntical varianc componnts stimats: 0.80 for th first lvl varianc, and 0.0 for th scond lvl varianc. Bcaus th total varianc is on in this xampl, th clustring ffct xprssd as an intraclass corrlation is as wll, which is about /3 of th sum of th clustring ffcts (= = 0.91) in th thr-lvl modl. Spcifically, 18 1 ρ = = Scond, suppos that th third lvl (.g., schools) is omittd. Thn, using quations 18 and 1 I computd th first and scond lvl varianc of th rsulting two-lvl modl (.g., studnts nstd within classrooms) as and σ 1 = 0.709, 4 1 = = τ. Th two-lvl HLM analyss of th data providd almost idntical varianc componnts stimats: 0.71 for th first lvl varianc, and 0.9 for th scond lvl. A small discrpancy in th stimats is xpctd sinc th lvl-1, lvl-, and lvl-3 units wr computd approximatly (assuming a balancd dsign). Bcaus th total varianc is on in this xampl, th clustring ffct xprssd as an intraclass corrlation is 0.89 as wll, which is about th sum of th clustring ffcts (= = 0.91) in th thr-lvl modl. This is xpctd sinc th numbr of lvl-3 units was larg in this cas ( m = 79). Spcifically,

15 ρ = = In sum, this papr showd that omitting a lvl in thr-lvl modls affcts th varianc dcomposition and clustring ffcts in th rsulting two-lvl modls. Th rsults ar prsntd in algbraic xprssions that ar asy to us. Ths rsults ar usful for ducation and social scinc rsarchrs (.g., in tachr and school ffcts rsarch) sinc thy indicat what part of th middl lvl (.g., classroom) varianc (in a thr-lvl modl) is distributd to th first (.g., within school) and th scond (.g., btwn school) lvl (in th rsulting two-lvl modl), or what part of th third lvl (.g., school) varianc (in a thr-lvl modl) is includd in th scond lvl (.g., btwn classroom) varianc (in th rsulting two-lvl modl). Ths rsults ar also usful for powr computations during th dsigning stags of a study, sinc thy provid a guid about how th clustring ffcts chang from thr- to two-lvl dsigns.

16 Rfrncs Bryk, A. S., & Raudnbush, S. W. (1988). Toward a mor appropriat concptualization of rsarch on school ffcts. A thr-lvl hirarchical linar modl. Amrican Journal of Education, 97, Cochran, W. G. (1977). Sampling tchniqus. Nw York: Wily. Donnr, A., & Klar, N. (000). Dsign and analysis of clustr randomization trials in halth rsarch. London: Arnold. Goldstin, H. (003). Multilvl statistical modls (3 rd d.). London: Arnold. Hdgs, L. V., & Hdbrg, E. (007). Intraclass corrlation valus for planning group randomizd trials in Education. Educational Evaluation and Policy Analysis, 9, Kirk, R. E. (1995). Exprimntal dsign: Procdurs for th bhavioral scincs (3 rd d.). Pacific Grov, CA: Brooks/Col Publishing. Konstantopoulos, S. (in prss). Th powr of th tst for tratmnt ffcts in thr-lvl clustr randomizd dsigns. Journal of Rsarch on Educational Effctivnss. Lohr, S. L. (1999). Sampling: Dsign and analysis. Duxbury Prss. Longford, N T. (1993). Random cofficint modls. Nw York: Oxford Univrsity Prss. Murray, D. M. (1998). Dsign and analysis of group-randomizd trials. Nw York: Oxford Univrsity Prss. Ny, B, Hdgs, V. E., & Konstantopoulos, S. (000). Th ffcts of small classs on acadmic achivmnt: Th rsults of th Tnnss class siz xprimnt. Amrican Educational Rsarch Journal, 37, Ny, B, Konstantopoulos, S., & Hdgs, V. E. (004). How larg ar tachr ffcts? Educational Evaluation and Policy Analysis, 6, Raudnbush, S. W., & Bryk, A. S. (1986). A hirarchical modl for studying school ffcts. Sociology of Education, 59, Raudnbush, S. W., & Bryk, A. S. (00). Hirarchical linar modls. Thousand Oaks, CA: Sag Publications. Raudnbush, S. W., & Liu, X. (000). Statistical powr and optimal dsign for multisit randomizd trails. Psychological Mthods, 5,

17 Rowan, B., Corrnti, R., & Millr, R. J. (00). What larg scal, survy rsarch tlls us about tachr ffcts on studnt achivmnt: Insights from th Prospcts study of lmntary schools. Tachrs Collg Rcord, 104, Sarl, S. R., Caslla, G., & McCuloch, C. E. (199). Varianc componnts. Nw York: Wily. Snijdrs, T. A. B., & Boskr, R. J. (1999). Multilvl analysis. London: Sag.