Incorporating Within-Study Correlations in Multivariate Meta-analysis: Multilevel Versus Traditional Models

Transcription

1 Incorporating Within-Study Correlations in Multivariate Meta-analysis: Multilevel Versus Traditional Models Alison J. O Mara and Herbert W. Marsh Department of Education, University of Oxford, UK Abstract The multilevel meta-analysis approach is compared with the traditional meta-analytical approaches, known as fixed effects and random effects models, for dealing with data with multiple outcomes. It is common for research to examine multiple outcomes (e.g., multidimensional constructs), and so it is important to establish whether multilevel models offer an improvement in modelling such datasets. A description and comparison of the underlying models is followed by the results of a simulation study comparing the three meta-analytic models under different statistical conditions. The results of the simulation study suggest that the maximum likelihood multilevel approach is in general superior to the fixed-effects and random effects approaches, as it provides good estimates of the parameters and generally has better coverage. In particular, the fixed and random effects are shown to have higher risk of Type II error. Interestingly, the inclusion of the covariance between the two outcomes (the cornerstone of the multivariate multilevel approach; Kalaian & Raudenbush, 1996) did not seem to improve the estimation of the parameters beyond the regular multilevel model. In general, all models performed better when the number of studies included in the meta-analysis was larger. Implications for the inclusion of different types of within-study correlations are discussed. Corresponding author: Alison O Mara, alison.omara@education.ox.ac.uk, Department of Education, University of Oxford, 15 Norham Gardens, OXFORD, OX2 6PY Acknowledgement: This research was supported by a grant from the United Kingdom Economic and Social Research Council (ESRC RES ).

2 Advances in knowledge come from the integration of a vast body of research (Schmidt, 1992). Broadly, meta-analysis can be defined as a quantitative review of the results of independent studies addressing a related research question (Normand, 1999). Meta-analysis seeks to systematically synthesise the results from multiple studies to facilitate comparison across studies (Rosenthal, 1987), with the goal of gaining a greater understanding of related research. Methodological approaches to meta-analysis are being developed faster than they can be incorporated into practice, and most of these developments have been made in the last two decades (Viechtbauer, 2005). Early meta-analyses were based on a fixed-effects model (Hedges & Vevea, 1998), which assumes that all samples are drawn from the same population of studies so that there is only one true (homogeneous) population effect (Cohn & Becker, 2003; Erez, Bloom, & Wells, 1996). Fixed effects meta-analyses were an improvement on traditional literature reviews because they allowed the researcher to model differences between studies, rather than just report whether results of various studies were statistically significant or not. However, there is a problem with fixed effects models: Results of fixed-effects analyses apply only to the actual data considered so the results cannot be generalised to other studies (Raudenbush, 1994). This is because fixed effects models assume that there is no systematic variability between studies (i.e., they are homogeneous). In reality, this is rarely the case. Hafdahl (2007) noted that, Given the ubiquity of between-studies variation in meta-analytic data and the theoretical and practical value of relating it to study features, models that incorporate this heterogeneity are indispensable (p. 200). More recently, researchers have argued for a random-effects model, in which variability between studies is assumed to reflect both sampling error and variability in the population of effects; this is the random effects model (Raudenbush, 1994). The inclusion of between-study variance estimates makes the findings from random effects model analyses appropriate to generalize to other studies that were not included in the meta-analysis (Lipsey & Wilson, 2001). The procedure for the random-effects model is similar to fixed-effects models, except that a random error variance component is added to the variance associated with each effect size, which takes into account variability in the population effects (Lipsey & Wilson, 2001). Although random effects models represented an important extension of in meta-analytic methodology, it still suffers a potentially critical flaw. An important statistical assumption of both fixed and random effects models is that the outcomes are independent of each other. This

3 means that if a meta-analyst wishes to consider multiple outcomes from the same study, this assumption is violated (i.e., effect sizes from the same study are not independent and therefore likely to be more highly correlated). As such, traditional meta-analytic methods (fixed and random effects models) are not appropriate for dealing with multidimensional constructs consisting of multiple outcomes, as they use incorrect standard errors in the analyses. A promising new direction for dealing with this issue is the multilevel model approach to metaanalysis (Hox, 2002; Kalaian & Kasim, 2008a, 2008b; Kalaian & Raudenbush, 1996), outlined below. First, we will examine how meta-analysts using traditional methods have dealt with the multiple outcome issue in the past, to see why a new approach is necessary. Approaches to Dealing with Multiple Outcomes in Traditional Meta-Analytic Models As a result of the assumption of independence in fixed and random effects models, metaanalysts using these traditional methods are forced to consider only one outcome per study. This inevitably has important theoretical implications, as well as statistical ramifications (see Gleser & Olkin, 1994, for a more detailed discussion of the problem with dependent effect sizes). Choose one outcome of interest. Many traditional meta-analyses focus on only one outcome that which is of most interest to the researcher. For instance, the researcher might only look at how an educational programme improves overall school grades, but not be interested in the affect of the programme on standardised achievement tests. This approach is appropriate for some research questions that are very specific and deal with a narrow range of outcomes. However, this strategy does not allow a detailed look at how the independent variables of interest (e.g., an educational programme) affect different outcomes (e.g., grades in mathematics and grades in English). Averaging the effect sizes. A common approach when multiple indicators of the same construct (e.g., different measures of job satisfaction) is to establish an independent set of effect sizes by calculating the average of the effect sizes in the study. However, the dependent variables need to be almost perfectly correlated for this method to work, because the mean effect size gives an estimate that is lower than expected if the correlation is moderate or low (Rosenthal & Rubin, 1986). In addition to a high statistical correlation, the outcomes (or indicators) should be conceptually similar and on the same metric to make the results meaningful. Similarly, averaging distinct domains of self-concept (e.g., maths ability self-concept and verbal ability self-concept) may not make sense when forming a global, overall self-concept score, since these domains have

4 been shown to be nearly uncorrelated and hence very distinct (see Marsh & O Mara, 2008 for review). As such, the practice of averaging outcomes is usually counter-productive as potentially valuable information is lost or the resulting score might even be meaningless. In particular, it is highly problematic for multidimensional constructs in which the multiple dimensions are specifically designed to measure different facets of the construct and are highly differentiated (e.g., multiple dimensions of self-concept for which the average correlation among different selfconcept domains is only about.20). Separate meta-analyses on each outcome. If multiple outcomes are of interest to the researcher, then one approach is to conduct separate analyses on each outcome. However, this approach does not allow contrasts between outcomes, thereby restricting the questions you can ask. Thus, for example, if the meta-analyst specifically wanted to compare the effects of a math intervention on math and verbal self-concepts, it would be important to consider both facets of self-concept in the same analysis. Therefore, this may not always make sense for the research question under consideration (Rosenthal & Rubin, 1986). Shifting unit of analysis (Cooper, 1998). Cooper suggested that the outcomes of the included studies could be aggregated depending on the level of analysis of interest the study or outcome level. At the study level, all effect sizes from within a study are aggregated to produce one outcome per study. For each moderator analysis, effect sizes are aggregated based upon the particular moderator (or predictor) variable, such that each study only includes one effect size per outcome on that particular variable. Although this strategic compromise does not eliminate the problem of independence, this approach minimizes violations of assumptions about the independence of effect sizes, whilst preserving as much of the data as possible (Cooper, 1998). It is probably the most defensible way of dealing with multiple outcomes in fixed and random effects models in cases where the researcher is explicitly interested in comparing different outcomes. However, this too has problems. Within an aggregated dataset, the same problems with averaging the outcomes arise. In addition, each set of analyses will be based on a different set of effect sizes, making interpretation of different analyses within the same meta-analysis difficult. Indeed, the original presentation of this strategy was a short suggestion presented by Cooper (1998) as an expedient compromise to an apparently serious limitation to traditional approaches to meta-analysis, but with no statistical or empirical justification. Despite being little more than a note in passing, Cooper s suggestion has been widely cited as a justification for the

5 aggregation of effect sizes depending on the unit of analysis. However, little research has evaluated whether this method is indeed an improvement on the alternatives. Although the present investigation does not evaluate the shifting unit of analysis approach, an alternative that is based on sound statistical reasoning is tested. In summary, there are no truly satisfactory ways of dealing with multiple outcomes when the researcher is explicitly interested in comparing and contrasting different outcomes in the traditional meta-analytic models. In response to this, researchers are now exploring multilevel modelling approaches to meta-analysis. Approaches to Dealing With Multiple Outcomes in Multilevel Modelling Multilevel (or hierarchical linear) models were applied to meta-analysis to account for the dependencies in multivariate meta-analytic datasets (Goldstein, 2003; Hox, 2002; Marsh, O Mara, & Malmberg, 2008; Raudenbush & Bryk, 2002). The idea here is that multiple effect sizes from the same study can be included in the analyses, without averaging or aggregating and thereby losing information. Multilevel analyses (in general) are increasingly used in the social sciences for analysing hierarchically ordered data in which the assumption of independence is violated (Hox, 2002). In typical educational data, for example, students (level 1) are nested within classes (level 2). Students within the same class are usually more similar to each other than to students from different classes violating the assumption of independence consistent with random sampling (Goldstein, 2003). Meta-analysis is a special type of multilevel data in which the multiple effect sizes from a given study (level 1) are nested within the studies included in the meta-analysis (level 2) (Hox & de Leeuw, 2003). Multilevel model meta-analyses are surprisingly uncommon, despite being first suggested by Raudenbush and Bryk in The lack of published metaanalyses utilizing this method does not stem from scepticism about its potential efficacy; rather, a distinct lack of prescriptive instructions on how to conduct a meta-analysis using multilevel modelling (Van den Noortgate & Onghena, 2003) has made researchers hesitant. Only in the last few years have more detailed methodological notes been published (e.g., Hox & de Leeuw, 2003; Van den Noortgate & Onghena, 2003; Hox, 2002; Raudenbush & Bryk, 2002). Three (or more) level models. In addition to the issue of multiple outcomes, meta-analyses can violate the assumption of independence by having more than one treatment group or sample reported in the same study. It is not uncommon to have more than two levels in the data structure

6 as would be the case when there are multiple studies within the same publication, or multiple publications from the same research team. A multilevel approach is more explicit in identifying the different levels, determining how much variance is explained at each level, and evaluating characteristics that are specific to different levels (and even interactions between effects that might occur at different levels). Although random effects models do include an estimate of the between-studies variance, they are not really able to incorporate true hierarchical data particularly when there are more than two levels in the structure. Known population correlation. More recently, the multilevel meta-analytic model has been extended to include multivariate outcomes (Kalaian & Raudenbush, 1996; Kalaian & Kasim, 2008a, 2008b). By accounting for this nested structure and including an estimate of the correlation between the outcomes within a study, it is proposed that the problems with dependencies within the dataset are minimised (Kalaian & Kasim, 2008a, 2008b). This claim is yet to undergo rigorous testing using simulated data, as previous work (e.g., Berkey, Anderson, & Hoaglin, 1996; Kalaian & Raudenbush, 1996; Kalaian & Kasim, 2008a, 2008b) has been based on real data. However, despite the intuitive appeal of multivariate multilevel model meta-analyses, in practice they can be difficult to implement. The primary reason for this is that often the withinstudy correlations are not known (see Riley, in press, for a review). Typically, the primary studies included in the meta-analysis do not report the correlations between the outcomes of interest, nor do they report the raw data so that a meta-analyst could calculate the correlation (Riley, Thompson, & Abrams, 2008). In a meta-analysis of Scholastic Aptitude Test (SAT) scores, Kalaian and Kasim (2008a; see also Kalaian & Raudenbush, 1996) did not have access to the within-study correlations, so they used the correlation between the two outcomes (maths and verbal achievement) published in the SAT manual. In this case, this approach is reasonable because all of the included studies reported the exact same outcomes using the exact same measure and based on similar samples. Thus, even though it is likely that there will be slight study-to-study variations in the actual sample correlation, it is reasonable to assume that the sample correlations would not differ much from the population correlation published in the test manual. This assumption becomes less realistic when (a) different studies use different instruments to measure the outcomes, (b) the population correlation between the outcomes is not known, or (c) the samples in the included

7 studies differ from the population. This means that in some cases, the estimate of the correlation is little better than an educated guess. In summary, whilst Kalaian and colleagues research provides a reasonable compromise to this problem in the context of SAT scores, this situation does not generalize to other research. Hence, meta-analysts are left with the problem of dealing with the estimation of the correlation between multiple outcomes at level 1 (i.e., the level of the individual participant). Unknown population correlation. An alternative is to use the known correlations from some studies to estimate the unknown correlations in others. For example, Berkey et al. (1996) used the correlations between outcomes in one clinical trial as the basis for the correlations in any other study that did not report the correlation. They conducted a sensitivity analysis in combination with this to show that they would have reached a similar conclusion had the correlations come from a different source. Using known correlations could also involve mean imputation. For instance, if a meta-analysis yielded 20 studies, and half of them reported the correlation between the outcomes, then one could use the average of the known as the fixed value correlation of the unknown. This requires a reasonable number of known correlations (or access to the original raw data; see Riley, in press) for this to be appropriate. An alternative approach could be to use more sophisticated approaches to missing data, such as multiple imputation. Riley et al. (2008) suggested an approach for bivariate random effects model metaanalysis (BRMA) when the within-study correlations are not known. This model allows for individual weightings for each included study, but includes only one overall correlation parameter, ρ. This overall correlation is a hybrid measure of the within-study and between-study correlations. This seems a promising approach. However, so far the model is confined to bivariate studies with complete data it is not clear how to estimate the overall correlation with more than two outcomes, or when some of the studies include only one (or some) of the outcomes of interest. Adding more outcomes will make the modelling process very complex. Perhaps most importantly, this model assumes that the within-study correlations will be equal across all studies. Although this may be reasonable in many cases, it is not clear whether having different samples across the studies (which could potentially be from different populations) and, perhaps, different measures of the target constructs, will mean that the correlation between the outcomes is the same in every study.

8 In summary, multilevel models are the most recent and sophisticated development in metaanalytic methodology. They have greatly expanded the number and types of research questions that can be appropriately addressed through meta-analysis. They promise to resolve the issue of non-independent outcomes within meta-analytic datasets, although this promise is undermined by the difficulty in acquiring or estimating the within-study correlations. This issue is yet to be fully evaluated. Meta-analytic Approaches to Research Synthesis In the following sections, the basic components of the analytical strategies in meta-analysis (i.e., effect sizes, variances, and covariances) are summarised. This is followed by an overview of the underlying models of the major techniques in the fixed effects (FE), random effects (RE) and multilevel (ML) approaches. For ease of reading, please note the following notation used in the present study. p was used to denote the number of outcome measures in each of the primary experimental studies. In this bivariate model, each of the primary studies under review has two outcome measures (p = 2), so that the total number of outcome measures across all k studies is equal to 2k (or pk, the number of outcomes multiplied by the number of studies). i is used to denote the study from which the data are drawn. k represents the number of studies included in the meta-analysis. Effect Sizes, Variances, and Covariances Effect sizes. An effect size is a standardised numerical outcome for comparing the studies. There are numerous ways of calculating an effect size (see Lipsey & Wilson, 2001). Here we will be examining the standardised mean difference effect size. In the present study, an effect size is defined as the difference between two groups (e.g., experimental and control groups) in standard deviation units. This is calculated as (Glass, 1976; Rosenthal, 1994; Lipsey & Wilson, 2001) d ip Y Y2ip = (1) s 1ip pooled where Y 1i and Y 2i are the means of the outcome scores for group 1 and group 2, respectively, and s pooled is the pooled standard deviation (square root of the pooled within-groups variances), from the i th study. s pooled is estimated as follows s p _ pooled = ( s 1ip ( n 1i 1)) + ( s ( n 1i + n 2i 2ip ( n 2) 2i 1)) (2)

9 where s 1ip and s 2ip are group 1 and 2 s estimated standard errors for the outcome measure in study i for outcome p, respectively. n 1i and n 2i are the sample sizes for groups 1 and 2 in study i, respectively. Variances and weighting. The variance of the effect size in meta-analysis is the sampling error variance of the original study. It is the square of the standard error of the effect size. The standard error of the effect size is defined as the standard deviation of the sampling distribution (Lipsey & Wilson, 2001). The variance is based on the sample sizes for the two groups and the effect size, as seen below. The sampling variance (v ip ) of the effect size, d ip, for study i can be estimated as (Hedges, 1981) 2 ( n1 i + n2i ) dip vip= + ( n n ) 2( n + n 1i 2i 1i 2i ) (3) where n 1i and n 2i are the sample sizes for groups 1 and 2 in study i, respectively. d 2 ip is the square of effect size p for study i. The variance is very important in meta-analysis, as it is used in weighting the effect sizes. Because different studies have different sample sizes and standard errors of the effect sizes, simply finding the average of the effect sizes may give unwarranted weight to studies with small samples or large sampling error. So, meta-analysts weight the effect sizes based on the inverse of the variance so that more weight is given to effect sizes that are considered to be more precise, such as those with a small standard error and those from studies with larger samples. The weighting formulas for the different models are given in the next section. Covariances. It has been proposed that the covariances between any pair of the multiple effect sizes needs to be estimated to account for the correlation between outcomes (e.g., Kalaian & Raudenbush, 1996). The covariance between the effect sizes from each primary study can be estimated as follows (Kalaian & Kasim, 2008a) σ ip, ip ( n1+ n2) ' = r ( n n ) 1 2 ip, ip' dipd + 2( n 2 ip' ip, ip' 1 r + n 2 ) (4)

10 where r ip,ip is the correlation coefficient between the outcome measures in each of the primary studies, d ip and d ip represent the two effect sizes; in a bivariate model, the effect sizes would be denoted as d i1 and d i2. The covariance term is only included in the ML model. The Three Meta-Analytic Models (Fixed, Random, and Multilevel Models) Fixed effects models. As noted earlier, FE models assume that all of the variance between the effect sizes is attributable to sampling error (v ip ), because the effects are assumed to all come from the same population (Hedges & Olkin, 1985). Thus, the formula for the observed effect size in an FE model is given by (5) d i = δ + e i where d i is the observed effect size in study i, δ is the true population effect, and e i is the residual due to sampling variance in study i (equivalent to the square root of v ip ). In the FE model, a weighted average of the effect sizes is calculated. The weight is given by w ipf 1 = v ip (6) where v ip is the value calculated using Equation (3). The overall mean effect size is given by d p ( wipf d = w ipf ip ) (7) In addition to calculating the overall weighted mean effect size, a test of the homogeneity of the effect sizes is conducted. This indicates whether the effect sizes are similar to each other, and are drawn from the same population of effect sizes. That is, it asks the question: Are the differences among the effect sizes bigger than might be expected by chance? If the test is significant, it means that the data are heterogeneous, and so this assumption of the FE model is violated. The Q-statistic follows a chi-square distribution, and is calculated as where d ip is the observed effect size p in study i, is the mean effect size p from study i, and d ip Q p ( d ip dip ) = w ip w ip is the weighting for effect size p in study i. If this value is significant when compared to a 2 (8)

11 chi-square table, the meta-analysts should conduct RE analyses instead, but may also wish to model moderators of the effect sizes to determine the source/s of the variance. Random effects model. In addition to the sampling error, RE models also assume that there are some genuine between-study differences, which contribute to the variation in the effect sizes. The general formula for the RE model is d = δ + u + e ip where δ is the mean true population effect size, u ip is the deviation of the true study effect size from the mean true effect size, and e ip is the residual due to sampling variance in study i. The RE model takes into account the between-study variability by including a random error variance component in the weighting. The random error variance component, v θ, is given by Q where Q is the Q-statistic calculated to test the homogeneity of the model (Equation 8), k is the number of studies, and w ipf is the effect size weight, calculated based on FE models (Equation 6). The random error variance component, v θ, is then used in the calculation of the weighting. (9) (10) 1 (11) + The weighted overall mean effect size is then calculated as in Equation (7), but with w ipr rather than w ipf. Multilevel model. Like the RE model, the ML model assumes that there is sampling error variance (i.e., within-study variance) and between-study variance. The formula for the observed effect size in a ML model is d γ + u + e ip = 0 ip where d ip is the observed effect size p in study i, γ 0 is the mean true population effect size, u ip is the deviation of the true study effect size p from the mean true effect size, and e ip is the residual due to sampling variance in study i. The ML model differs from the RE model in the way that the parameters are estimated. It uses an iterative process, so that the model is run repeatedly, improving the estimation each time, until the estimates fail to significantly improve (known as ip ip ( k 1) p vθ p = w 2 ipf wipf w ipf w ipr = v ip v θ ip (12)

12 convergence ). The process through which this occurs is detailed in Goldstein (2003), Hox (2002), and Raudenbush and Bryk (2002). The estimation procedure appropriate in meta-analysis is called restricted maximum likelihood, which means that the variance components are estimated after removing the fixed effects from the model. This model is preferred particularly for meta-analysis (see Bryk & Raudenbush, 1992) because the significance of the between-study variance is critical in deciding whether all the studies report the same outcome, and the restricted maximum likelihood model is more accurate in estimating the variance components (Hox, 2002). The alternative estimation procedure, full information maximum likelihood, has also been found to have downwardly biased estimates of the fixed parameters in meta-analysis (e.g., Turner, Omar, Yang, Goldstein, & Thompson, 2000; Van den Noortgate & Onghena, 2003). ML models have been suggested to give more precise and less biased estimates of between-study variance than traditional techniques based on Hedges and Olkin s (1985) formula. This is particularly true for comparisons of standard errors in FE models (Bateman & Jones, 2003; Hox, 2002; Van den Noortgate & Onghena, 2003). Hox (2002) notes that, in RE weighted regression, an estimate of the residual between-study variance must be provided by the metaanalyst before conducting the analysis, whereas multilevel modelling programmes use iterative maximum likelihood estimation, taking into account the information contained in the explanatory variables in the model (p. 144). Since the estimation of standard errors is improved in multilevel modelling, the confidence intervals will be more precise, regardless of heterogeneity. However, like the RE model, ML models incorporate the random error variance component, and so the confidence intervals will be more similar in the two models than the FE model. The difference between RE and ML models is likely to increase with greater standard errors as proposed by Hox, in press). The Present Investigation Standard approaches to multilevel model meta-analyses (referred to here as the general ML model) have been cited as a logical way of representing meta-analytic data (e.g., Hox, 2002). As an extension of this model, the multivariate ML approach has been hailed as the best way to deal with multivariate outcomes (Kalaian & Kasim, 2008a), as it explicitly models the covariance between the multiple outcomes. Although conceptually this seems reasonable, the practicality of knowing (or estimating) the various within-study correlations is likely to be problematic.

13 The purpose of the present investigation is twofold. Firstly, we wish to test whether the general ML model is actually an improvement on the FE and RE model in terms of estimating the overall mean effect sizes and their standard errors. Secondly, we seek to evaluate the multivariate ML model compared to the general ML model, to determine whether the inclusion of the covariance between the multivariate outcomes in the multivariate model improves the estimates beyond the general ML model (which does not include the covariance term in the model). These issues will be examined through a simulation study. The methods will be compared on the amount of bias in the estimates, the accuracy and variability of the estimates, and the frequency of Type I and Type II error under a host of conditions. It is hypothesised that the ML will perform better than the FE and RE across all conditions. It is also hypothesised that the multivariate ML model will produce better estimates than the general ML model when the covariance in the fitted model matches the population covariance. The precise predictions for each of the outcomes (bias, variability, and coverage) will be elaborated on at the end of the Method section. Method Simulation Design Statistical simulation is a numerical technique of experimenting with random sampling from probability distributions. In a typical simulation study, a series of computer-generated datasets are analysed under a host of different (statistical) conditions. The purpose of simulation is generally to determine how a particular statistical model behaves under various conditions, and therefore whether its properties operate in a way that is appropriate for particular research applications. For example, it is possible to test whether having a larger number of studies in the meta-analysis improves the estimate of the overall mean effect size. Because the dataset is computer-generated, the true sampling distribution based on the population generating model specified by the researcher (which will be referred to as the population distribution here) is known for the particular set of conditions. This allows researchers to test to see if the model is able to accurately reflect that generated distribution (Castro & Gaviria, 2000). Simulation studies involve generating multiple independent simulated data sets. In the present research, 1,000 datasets were generated per condition, which is considered a sufficient number for guaranteeing the precision of the analysis (e.g., Castro and Gaviria, 2000). The effect sizes of interest were standardised mean differences (i.e., the difference between two groups on

14 an outcome; described further below). The datasets were moderately independent, in that the same set of 1,000 simulated independent datasets were used to compare the three statistical methods (ML, FE, and RE) for the same condition, but a different set of datasets was generated for each condition investigated. This situation can be viewed as analogous to a matched pair design where the within sample variability is eliminated and therefore are sensitive to detecting any differences between methods (Burton, Altman, Royston, & Holder, 2006, p. 4282). The simulation conditions were determined by six characteristics of the meta-analyses and their included studies (see Table 1). In total, 144 conditions (with 432 cells representing the three models) were tested. <<insert table 1 about here>> Number of studies (k = 20 or 100). With an increased number of studies, there is more information from which to calculate the parameters of interest. Therefore, we would expect that the accuracy of the estimates would increase with increasing k, particularly for RE (Raudenbush, 1994) and ML models which assume between-study variance. Whether 20 or 100 studies are considered a small or large sample of studies, respectively, depends on the discipline. In the social sciences, 20 would represent a small-medium meta-analysis, whereas 100 would represent a medium-large sample (cf. the sample sizes of the 302 meta-analyses included in Lipsey & Wilson s 1993 meta-meta-analysis). Mean group sizes (n 1 = 80 or 100). It is unlikely that most studies included in a metaanalysis have the same mean sample size for the two groups. For example, it is often common that a treatment group has a smaller number of participants than a control group, due to costs, accessibility, or other reasons. The sample sizes of the two groups (n 1 and n 2 ) are important, because they are used in the calculations of every aspect of the meta-analysis, including the weighted mean effect size, the sampling variance, and critically, the covariance. Thus, it is important to test whether the condition n 1 n 2 has a differential effect when the covariance term is included in the model. The numbers of participants in the two groups were generated on a Poisson distribution (following Castro & Gaviria, 2003) under two conditions: one in which the mean of the n s were equal (mean n 1 = 100, mean n 2 = 100) and one in which the mean of the n s for the two groups were not equal (mean n 1 = 80, mean n 2 = 100). The group sample sizes are equal within a study, so that the two effect sizes are based on the same sample sizes within a study. It is acknowledged that this may not necessarily be the case in real data situations.

15 Overall true effect size (mean δ 1 = 0 or 0.5). The size of the two outcomes is crucial in calculating the covariance, so it was important to test whether an imbalance in the means of the overall true effect sizes impacted the parameter estimation. In one condition, the population means of the effect sizes were equal for the two outcomes (mean δ 1 = 0.5 and mean δ 2 = 0.5), and in the other condition, the means were unequal (mean δ 1 = 0 and mean δ 2 = 0.5). These values were chosen as they were both seen to be realistic and common overall mean effect sizes in the social sciences. An effect size of zero represents no difference between the two groups, which is the null hypothesis, whereas an effect size of.5 represents a.5 standard deviation higher score for group 1 (e.g., the treatment group) compared to group 2 (e.g., the control group). Variance in true effect sizes (σ 2 1 = 0.1 or 0.2). The variance (σ 2 1 ) in the population of effects for each outcome will determine the spread of the effect sizes. In one condition, the population variances of the effect sizes were equal (mean σ 2 1 = 0.1 and mean σ 2 2 = 0.1), and in the other condition, the population variances were unequal (mean σ 2 1 = 0.2 and mean σ 2 2 = 0.1). This is important, as the variances will determine the between-study heterogeneity. Correlation in the fitted (ML) model (r = 0, 0.2, or 0.8). The FE and RE models assume that the data are independent, and so do not include an estimate of the covariance between the effects. The ML approach to meta-analysis is proposed to yield better estimates of the data when information about the correlation between the outcomes is included in the model. As shown earlier, the ML model includes the covariance, which incorporates the correlation between the outcomes (see Equation 4). Each pair of effect sizes will have their own covariance that is distinct from study to study, even if the correlation is constant across studies, because of the way in which the covariance is calculated (Equation 4). It was important to test the model when the fitted within-study correlation was zero, as this is equivalent to a ML model that does not include the covariance estimate. It was also desirous to test if the model could determine correct parameter estimates when the correlation between outcomes was low (r =.2) or high (r =.8). It is possible that the inclusion of the covariance in the fitted model (based on the correlation between the outcomes) only makes a substantial difference when the within-study correlations are high. 2 Covariance in true effect sizes ( σ = 0, 0.005, or 0.02). This refers to the covariance 12 between the two outcomes in the simulated data, and therefore represents the population covariance. The correlations in the fitted model (r = 0,.2, or.8), when substituted into Equation

16 4 for the δ s and n s specified above, correspond roughly with covariances of 0,.005 and.02, respectively. For example, when a correlation of.2 is entered into Equation 4 when δ 1 and δ 2 are.5 and n 1 and n 2 are 80 or 100, the covariance will equal approximately.005. Therefore, the population data were generated such that the two outcomes had a covariance of 0,.005, or.02. This was so that the population covariances would correspond with the zero, low, and high correlations in the fitted model. In summary, by selecting these six factors (Table 1), it was intended to determine whether certain features of the studies or the meta-analyses had an impact on whether the inclusion of the correlation in the model improved the parameter estimates for the ML model compared to the FE and RE models (which do not incorporate the within-study correlation). The conditions selected were considered to be representative of typical social science meta-analytic data. Generating the Data The simulation was performed using MLwiN 2.10 (Beta10; Rasbash, Browne, Healy, Cameron, & Charlton, 2008). The data were generated using macros developed by the first author. The effect sizes were simulated by the programme based on the population conditions specified in Table 1. The variances to be fitted in the model were calculated based on the simulated effect sizes and the randomly generated sample sizes. Analysing The Simulated Data The variances, weights, and standard errors were calculated based on Hedges and Olkin (1985), described in the previous section. The fitted model for the FE was that described in Hedges (1994). The fitted model for the RE was the method of moments approach described in Raudenbush (1994). Also see Hedges and Vevea (1998), Overton (1998), and Lipsey and Wilson (2001) for details and comparisons of both the FE and RE models used here. For the ML approach, the multivariate model described by Kalaian and Raudenbush (1996) and Kalaian and Kasim (2008a, 2008b) was used. In the ML model, restricted maximum likelihood estimates were calculated because full maximum likelihood estimates are downwardly biased (e.g., Turner, Omar, Yang, Goldstein, & Thompson, 2000; Van den Noortgate & Onghena, 2003). In simulation research, three of the primary methods for testing the ability of a model to estimate the true population parameters are coverage, bias, and accuracy. The coverage, bias, and precision were calculated in MLwiN 2.10 (Beta 10), and then imported into SPSS15.0 for analysis.

17 Bias. Bias is the deviation in an estimate from the true population quantity, and indicates the accuracy of the estimator for the effect. The basic assessment of bias is the difference between the average estimate and the true value, and is given by δ = ˆ β β (13) B where β is the true value for the estimate of interest, ˆ β = ˆ T i =1 βi /, T is the number of simulations performed, and βˆ i is the estimate of interest within each of the I = 1,..., T simulations. The smaller the bias, the closer the estimate is to the true value; the more accurate the estimate. RMSE. The mean square error (MSE) indicates the overall variability of the model. It is defined as the average squared difference between the estimate and the true value, and includes both bias and efficiency (Collins et al., 2001). MSE is given by ( ˆ β ˆ)) β 2 2 β ) + ( SE( (14) The smaller the MSE, the better, as it indicates greater accuracy of the estimates. It is common to report the square root of the MSE (referred to as root mean square error, or RMSE), because this is in the same scale as the parameter and makes it easier to interpret (Burton et al., 2006; Collins et al., 2001). Again, smaller RMSEs are preferred. Empirical SD and the bias of the SE. Precision can be assessed by examining the empirical standard deviation (SD). The empirical SD is the SD of the parameter estimates across the 1,000 iterations. A small empirical SD means that the estimates of that approach are not very variable. In this respect, precision differs from the RMSE in that the RMSE is based on deviations between each estimate and the true population parameter, whilst the empirical SD is based on deviation between each estimate and the mean estimate. Thus, RMSE reflects a combination of accuracy and precision, whilst the empirical SD reflects precision without regard to accuracy. It is also useful to evaluate the difference between the empirical standard deviation and the estimated standard error (mean of the estimated standard errors for a given parameter estimate provided by the model, averaged across the different simulation samples) can reveal whether the estimated standard error is biased. This occurs when the empirical standard deviation varies systematically from the corresponding from the estimated standard error.

18 Coverage. The coverage of a confidence interval is the proportion of times that the true specified parameter value is included in the obtained confidence interval. The confidence intervals are obtained by ˆ β ± Z SE( ˆ β ) (15) i 1 α / 2 i where βˆ i is the estimate of interest within each of the I = 1,..., T simulations, ( ˆ i ) SE β is the standard error of the estimate of interest within each simulation and Z 1 α / 2 is the 1-α/2 quantile of the standard normal distribution. For the 95% confidence interval, Z 1 α / 2 is The coverage indicates in part the accuracy of the standard error, and should be approximately equal to the nominal coverage rate. For example, a 95% confidence interval would warrant a 95% coverage rate. Coverage is linked to Type I and Type II error. Type I error is the error of rejecting a correct null hypothesis. It occurs when a difference is observed, when in truth there is not a difference. Type II error is the error of accepting a false null hypothesis, and occurs when a difference is not observed, when there actually is a difference. For a nominal coverage rate of 95%, coverage rates greater than 95% suggest that the results are too conservative, leading to a loss of statistical power with higher than expected Type II errors. Coverage rates lower than 95 per cent can lead to higher than expected type I errors. Burton et al. (2006) suggested that, with 95% confidence intervals and 1,000 generated datasets, the coverage value should lie between 93.6 and 96.4, as this is the range of two standard errors around the nominal coverage probability. However, if there is systematic bias, the interpretation of coverage rate is complicated: an accurate standard error associated with a large bias will result in population value not being in the CI. Hypotheses and Research Questions Bias. In the present study, it is hypothesised that the ML will show smaller bias in the parameter estimation than the RE and FE models when estimating multivariate outcomes. This is because the iterative process in the ML model ensures greater accuracy in the parameter estimates. The FE model is likely to show greater bias in the parameter estimation than the ML or RE, as the model does not include the between-studies variance estimate, and so is less likely to estimate the true population values. In addition, there is no prediction regarding the difference between the multivariate ML model and the general ML model in terms of bias.

19 RMSE. In the present study, it is hypothesised that the ML will have a smaller RMSE than the FE and RE models. This is proposed because the iterative process ensures more accurate standard errors in the ML model, and so the estimates are more precise. The RE model may show a higher RMSE than both ML and FE models. This is hypothesised because the RE model s standard errors tend to be larger than in the FE model as they include the random error variance component. The RE model s RMSEs may be larger than in the ML model because the ML model will have smaller bias (which is included in the calculation of the RMSE), and because the RE model does not have an iterative process to increase precision. Also, it is predicted that the multivariate ML model will have a smaller RMSE than the general ML model because it will estimate a more accurate error structure. Empirical SD and bias of the SE. We make no predictions regarding differences between the models in terms of the empirical standard deviation (SD). In terms of the difference between the empirical SD and the estimated standard error (SE), it is predicted that the difference will be smallest for the ML and largest for the RE for the same reasons as for the RMSE. We anticipate that the multivariate ML model will have a smaller bias of the SE than the general ML model because it will estimate a more accurate error structure. Coverage. It is hypothesised that the ML model will show no tendency towards either Type I or Type II error; i.e., the coverage will be between 93.6 and The FE model is expected to be prone to Type I error (i.e., coverage <93.6), because it does not include a random error variance component and so has smaller confidence intervals. Smaller confidence intervals mean that it is more likely to be statistically significant, and therefore more likely to falsely reject the null hypothesis. The RE model is assumed to be more similar in terms of coverage to the ML, as both include a random error variance component especially when heterogeneity is relatively small. However, in the conditions where heterogeneity is large (i.e., when the variance of δ 1 is 0.2 in the present study), then the RE model may be too conservative and show greater Type II errors. This means that the RE model is more likely to falsely accept the null hypothesis. Finally, it is hypothesised that the multivariate ML model will have better coverage than the general ML model because it will estimate a more accurate error structure. Results and Discussion No problems were encountered in estimating the coefficients of the MLM, FE, or RE models; the estimation procedure converged in all 144,000 simulation data sets. The bias,

20 RMSE, and coverage for each of the three models were estimated for each generated dataset for all 432 cells ( ) in the design. To determine which of the study s conditions contributed to the bias, the RMSE, the coverage, and the empirical standard deviation, we conducted ANOVAs using the bias, RMSE, coverage, and empirical standard deviation of the estimator as the dependent variables and each manipulated condition (method; number of studies, k; group size, n; population effect size; σ 2 1 ; covariance in the population; and covariance in the fitted model) as a factor. The ANOVAs were conducted at the cell level, with each cell average being treated as one observation (so that the highest-order interaction could not be separated from the error). Assessment of Bias The bias indicates whether the estimated mean is systematically different from the true population mean. To determine the bias, the cell mean for each of the 1,000 repetitions was calculated, and then the population value was subtracted from this mean (see Equation 13). For the ML approach, across the 144 cell means, the bias ranged in magnitude from to (M = -.001, SD =.000) for δ 1 and to.002 (M =.000, SD =.002) for δ 2 (see Table 2). As predicted, the bias for the FE approach was larger, with values ranging from to (M = -.010, SD =.010) for δ 1 and to (M = -.013, SD =.002) for δ 2. The bias for the RE approach was similar to the ML approach, with values ranging from to (M = -.002, SD =.001) for δ 1 and to.001 (M = -.003, SD =.002) for δ 2. In contrast to the ML approach and the RE approach, the FE approach slightly underestimated the effect sizes estimates (i.e., the true population mean was typically larger than the estimated mean). However, it is clear that the biases for all three models are quite small. <<insert Table 2 about here>> The source of the bias was further investigated by conducting an ANOVA, with the bias for δ 1 as the dependent variable. The values related to δ 1 were used because all of the changes in the conditions affected δ 1 only (i.e., variance, size of the overall mean effect size) or δ 1 and δ 2 equally (i.e., n, k, covariance, correlation). Thus, any differences would be observed most predominantly in δ 1. The factors in the ANOVA were the method (ML, FE, or RE) plus the six conditions described above (k, n, population effect size, σ 2 1, covariance in the population, and the correlation used in the fitted model). Although all factors were included in the 7-way model,

21 only the main effects and the 2-way, 3-way, and 4-way interactions were tested. This is because higher order interactions would have too few cases per cell, would be difficult to interpret, and were always non-significant or extremely small in size. The results of the ANOVA suggest that the largest moderator of the bias was the main effect of method (η 2 =.335), followed by the population mean effect size method interaction (η 2 =.330), the main effect of the population mean effect size (η 2 =.247), the variance method interaction (η 2 =.027), the population mean effect size variance method interaction (η 2 =.027), the main effect of variance (η 2 =.016), the population mean effect size variance interaction (η 2 =.014), and the main effect of k (η 2 =.003). All other main and interaction effects had η 2 values that were zero when rounded to three decimal places (i.e., less than 1/10 of 1% of the variance explained) and deemed to be substantively negligible. The R 2 for this model was due to the fact that analyses were based on cell means, but the total variance of the bias was very small (=.021). Importantly, the method used explained the most variance in the biases. The FE model had the largest biases (see Table 2), whilst RE and ML biases were quite similar. This means that, on average, the FE tends to underestimate the true population value. Explaining the second largest amount of variance was the interaction between the population mean effect size and the method. Oddly, when the population mean effect size, the three models have roughly the same bias; when the effect size increases, the FE model has a larger bias (see Figure 1). This could suggest that the FE model is as accurate at detecting the alternative hypothesis (that the two groups constituting the effect size calculation are not equal). The main effect for the population mean effect size also explained a large proportion of the variance in the biases. An inspection of the cell means indicates that bias is larger (mean bias = -.024, p <.001) for meta-analyses in which the population mean is.5, compared to meta-analyses in which the population mean is zero (mean bias = -.002, p <.001). It is arguable that, despite being statistically significant, these differences are substantively trivial. However, it might be the case that this difference is not trivial when the population mean is even higher than.5. Future simulations could test a broader range of population mean effect sizes to determine this. The second largest interaction effect for bias was the variance method interaction. From Figure 2 we can see that the bias for the FE model is consistently larger (in magnitude), but increases when the variance is larger. This means that the FE model is more prone to underestimate the mean effect size when the variance is higher.