Meta-analysis using HLM 1. Running head: META-ANALYSIS FOR SINGLE-CASE INTERVENTION DESIGNS

Transcription

1 Meta-analysis using HLM 1 Running head: META-ANALYSIS FOR SINGLE-CASE INTERVENTION DESIGNS Comparing Two Meta-Analysis Approaches for Single Subject Design: Hierarchical Linear Model Perspective Rafa Kasim Kent State University

2 Meta-analysis using HLM 2 Abstract In spite of the criticism for lack of validity, researchers still use single-subject designs to evaluate school counseling when working with individual students, autism treatment and many other individualized therapeutics interventions. The accumulation of many studies on one area of research creates the need for comparing the efficacy of the findings from these studies. Researchers developed several meta-analyses techniques to synthesize findings from single subjects studies, including a recent application of the HLM to conducting meta-analyses of single-subject designs. Two alternatives for estimating and modeling effect sizes from single subject steadies are presented through a comparison of two approaches that utilize Hierarchical Linear Modeling.

3 Meta-analysis using HLM 3 Introduction Elegant experimental studies characterized by random assignment, control groups, and pre- and post-intervention descriptive data, some cannot be applied to a research investigate individualized treatments or interventions. On the contrary, research studies involving individualized treatments are more likely to employ samples of convenience from applied settings where a single intervention is examined and withdrawal of treatment is unethical. Finally, such intervention studies have employed single-case methodology, where results are expressed in graphic displays that do not readily lend themselves to traditional meta-analytic investigations (McConnell, 2002). Single-subject designs became a common method for analyzing and evaluating the effectiveness of individualized therapeutic interventions. Although single-case studies are cumbersome for use in meta-analyses, such designs nevertheless have high statistical control and excellent internal validity, and allow for analyses of trends in data across time (Kratochwill, Mott, & Dodson, 1984; McConnell, 2002). Quantitative analysis of multiple single-case studies is particularly valuable in the areas of individual therapeutic interventions, areas that are characterized by diverse populations that challenges study replication (Van den Noortgate & Onghena, 2003; White et al., 1989). Despite the complexity involved, examination of intervention effects across multiple single-case studies using meta-analysis can contribute to research practice. Meta-analysis is a preferred methodology for deriving objective conclusions about the effectiveness of several individual therapeutic interventions across a series of studies (White, Rusch, Kazdin, & Hartmann, 1989). Previously employed methods for

4 Meta-analysis using HLM 4 conducting meta-analyses of single-case studies have included the conversion of singlecase graphic displays to quantitative data for the calculation of effect-sizes (Clark & Stewart, 1997); a piecewise regression technique (Center, Skiba, & Casey, ); percent of non-overlapping data between baseline and treatment phases (Scruggs, Mastropieri, & Casto, 1987); calculation of effect sizes for changes between phases (Corcoran, 1985; White, Rusch, Kazdin, & Hartmann, 1989); and pooling data from multiple baseline and treatment phases (Busk & Serlin, 1992). The majority of these developments focused on finding appropriate methods for estimating effect sizes for the different methods adopted in single-subject designs (Corcoran, 1985). This paper focuses on comparing two alternatives for estimating effect sizes using Hierarchical Linear Models (HLM), and modeling their variability across several singlecase studies. One alternative estimates and models the effect sizes using the actual repeated observations within the phases of single-case studies in HLM repeated measure designs (Van der Noortgate & Onghena, 2003). The other uses the V-Known procedure of HLM described by Raudenbush and Bryk (2002) on calculated effect sizes by Busk and Serlin s (1992) method for calculating effect sizes from single-case studies, and moderate the variability of the effect sizes by subject or study characteristics. Effect Size in Single-Subject Design Single subject design can be described as a special form of repeated measures designs. Data from such designs often lack the property of independent observations that can lead to the possibility of having autocorrelations across measures. Researchers often put forward assumptions such as normality and sphericity that pertain to the nature of the

5 Meta-analysis using HLM 5 data in the populations of such designs. Such assumptions are often hard to achieve, and, unless they can be validated, misleading findings may result (Myers & Well, 2003). In its simplest form of (AB) design, a single-subject design involves repeatedly observing a subject on some outcome over a period of time during a baseline period (phase A) and during the period of an intervention (phase B). The objective is to have a consistent assessment of the effectiveness of the intervention. There may be an unequal number of observations recorded in phases A and B. Variations of the design can take the form of having multiples of each of the two phases, such as ABAB design, or having an additional phase (C) that represents a different intervention from the one in B, such as ABC, in the same study. Ultimately, observations from the phases are compared either graphically (Busk & Serlin, 1992) or by using some permutation tests (Edgington, 1995) to report on the effectiveness of an intervention. In 1992, Busk and Serlin described three approaches for calculating effect size for measuring treatment effectiveness in single-case study. These approaches vary according to the assumptions needed for such designs. Using the first approach where no assumptions are made, an effect size, similar to Glass s d effect size (1976), is found for each subject by dividing the difference between the means from the two phases of the study by the standard deviation of the observed values in phase A under the null hypothesis of no intervention effect. Effect sizes calculated under this approach have a binomial distribution if they are estimating a common population effect size with some effect sizes being positive and other are negative (Busk & Serlin, 1992). Using the binomial sign-test model, a confidence interval can be established for the population median effect size.

6 Meta-analysis using HLM 6 In the second approach described by Busk and Serlin assuming equal variances and similar distributions for the observations in both phases, an effect size for each subject is found by dividing the difference between the means from the two phases by the square root of the pooled variance from both phases. Pooling variances under the given assumption allows for the attaining a more precise estimate for the denominator of the effect size. The assumption of similar population distributions of the outcome for the two phases with equal variances suggests that the distribution of the effect sizes is symmetrical, a property that can be adopted under the Wilcoxon model to find confidence intervals for the population median of the effect sizes (Busk & Serlin, 1992). For the third approach, effect sizes are obtained the same way as in the second approach under the assumption of normally distributed observations in both phases and the sphericity assumption. The estimated effect sizes have a non-central t-distribution that can be used to build confidence intervals for each study effect size as well as a confidence interval for an overall effect size. For this study we use effect sizes estimated by Busk and Serlin s third approach in the variance-known (V-known) modular of hierarchical linear models (HLM) (Raudenbush & Bryk, 2002). Variance estimates of the effect sizes are based on Hedges, 1994, approach for correlated observation. Purpose and Method The purpose of this study is to compare two approaches of conducting metaanalysis for single-subject studies via HLM when studying the effectiveness of therapeutic interventions. While the two approaches rely on using HLM for combining data from single-subjects studies, the difference between them is in their treatment of the

7 Meta-analysis using HLM 7 data within HLM. In one approach, the actual observations from single-subject studies are used in a two level HLM analysis. In the second approach, separate effect size based on Busk and Serlin s (1992) method with its estimated variance for each subject is used in the V-known version of HLM. A more detailed description of the two approaches is presented in the next two sections. There are at least three benefits of using HLM in combining data for singlesubject studies. One is obtaining better estimates of treatment effectiveness by borrowing information from other similar studies (Radenbush & Bryk 2002). HLM estimates of effect sizes are weighted combination of the individual effect size estimated weight by its precision and the overall effect size from all the studies in the combined data. A second benefit is the accommodation of dependent repeated observations. An important assumption required by many statistical procedures and certainly hard to deny in single-subject studies due to their design nature. Finally, the ability to model the systematic variability in studies effect sizes by subjects or studies characteristics. Data Data for this study come from a larger data set under current analysis by the author. The full data set consists of 233 participants with autism who were given one of seven different forms of intervention. Three outcomes were observed on two different scales (percentages and frequencies). Participants were identified by the type of treatment received, the outcome measured, and the scale of the data (percent versus frequency). For the purpose of this study, data from 35 participants with frequency data observed on academic/communication outcome. Twenty three subjects were introduced to the Ecological and Milieu type of intervention, while the remaining 12 were introduced to

8 Meta-analysis using HLM 8 the Peer Supported type of intervention. Data from all 35 participants were combined to form one sample to examine effect size estimate properties from a larger sample size. The two approaches also were compared separately for the two types of interventions, under moderate (23 participants) and small (12 participants) sample sizes. Applying Two Level HLM Model for Combining Single-subject Data Hierarchical linear models are statistical methods for analyzing hierarchically structured data where variables are measured at different levels. For example, measures might be available on patients as well as doctors where patients are nested within doctors. This hierarchy in the data can create a lack of independent observations. The use of the hierarchical linear model accounts for the dependency between the scores on the first level. The basic structure of HLM in its simplest form (two levels HLM) consists of two or more regression models for its two levels. Characteristics measured on level-one units are used in a regression equation to model an outcome of interest within each level-two units. Estimates of the regression coefficients from level-one model are then used as outcomes in one or more regression equations to be modeled by level-two characteristics (variables). Data from single-subject studies fit nicely into the structure of hierarchical data. Repeated observations on the individual from the phases of each study constitute the first level of the HLM model as (1)

9 Meta-analysis using HLM 9 where Y ij represents observation i for subject j. The indicator (intervention) ij takes a value of 1 if observation i for subject j is part of phase B of the study and 0 if it is part of phase A of the study. Finally, ε ij is a random error term for observation i that is normally distributed with mean zero and equal variance σ 2 ε. The model intercept β oj is the expected observed score for subject j at phase A and β 1j is the expected change in the observed score for subject j as a result of phase B of the study. The part β oj + β 1j (intervention) ij represents the expected observed score for subject j at phase B of the study. Variations in level one model parameters,β oj andβ 1j across the j subjects can then be introduced in a second level model as (2) where γ oo can be interpreted as the average observed scores across all subjects for phase A of the study and γ 1o is the average change in the observed scores across all subjects as a result of phase B of the study. Both U oj and U 1j are random parts for phase A and B respectively associated with subject j. Interest is mainly focused on the fixed effect parameters γ oo and γ 1o and the variance and covariance, σ 2 uo, σ 2 u1, and σ 2 uou1 for the first level parameters, β oj andβ 1j. Estimates of these parameters provide an overall effect-size estimate for the intervention and variance estimate for subjects specifics effect sizes with model specification that allow moderating the variability for these subject or study specific effect sizes

10 Meta-analysis using HLM 10 Using Glass s definition of the effect size d, the fixed effect γ 1o represents the numerator part of the overall effect size from single-subject studies. To get the overall effect size, γ 1o is divided by the square root of the pooled within subject variance σ 2 ε. Thus, the HLM estimate of the overall effect size, d is defined as (3) Similarly we estimate the subject specific effect size as (4) From equation (2) variability in subjects specific effect sizes can be estimated as (5) Where σ 2 u1 and σ 2 ε are the variances of β 1j and within subject variance, respectively and they can be estimated by the model. Large estimate of indicates large variability in the estimated subjects specific effect sizes around the overall effect size d. This variability can be modeled by study or subject characteristics, such as subject s age and gender or study intervention type or duration. Applying the V-known Version of HLM for Combining Single-Subject Data This approach differs from the 2 levels HLM approach for combining singlesubject data in two ways. First, actual observations from each subject are summarized

11 Meta-analysis using HLM 11 into subjects specific effect sizes with their reliability estimates (their variances) before their use in the HLM analysis, virtually reducing the first level of HLM model to a simple measurement model. Estimates for subjects specific effect sizes are calculated using Busk and Serlin s 1992 third approach for calculating effect sizes for single-subject studies. One drawback in using the V-known HLM approach for combining data from several single subject studies is the treatment of dependent observations for each subject. Assumptions of normality and sphericity, which are often difficult to validate, must be present in the data to address the issue of independent observations. One advantage of using the V-known HLM approach for combining data from several single subject studies is its use of standardized statistics (effect sizes in our case) that are unit free. This makes it well suited for combining studies that use different outcome measures, even though they are viewed as measures of the same construct. This is a case usually encountered in the larger data set, where some of the studies reported the percentages of outcome occurrence and some reported the frequencies of the outcome occurrence as data. To use this approach, under the normality and sphericity assumptions, estimated effect size, for each subject is found as where and are the means of the observations in phase B and A respectively. The term is the pooled, within subject standard deviation. Each may be viewed as statistic estimating a the true subject s specific effect size where

12 Meta-analysis using HLM 12 The standard error of estimate depends on the number of observations within each phase of the study. Hedges in 1994 showed that the statistics is approximately normally distributed with variance Once subjects specific effect sizes estimated by with their variances, the first level of HLM reduces to a measurement model as for j= 1,., J subjects, where is the sampling error associated with. In the second level model the true effect size for each subject can be expressed as an overall effect size plus some unique random part attributed to subject j.. The second level model can expand to include subject characteristics that help explain part of the randomness in the unique part as follows

13 Meta-analysis using HLM 13 where are some subject s characteristics. The coefficient in the expanded model is the overall effect size adjusted for subject s characteristics. Analysis and Discussion of the Results Tables 1, 2, and 3 shows the result of the analysis when the two approaches (Vknown HLM and two level HLM) applied, in three ways, on observations collected from 35 subjects in single-subjects studies. First, in order to obtain large sample estimates of the effect sizes, observation from all 35 subjects were combined by ignoring the two kinds of interventions the subjects received in phase B. Three overall effect sizes (ES) were obtained under each approach. Unadjusted, adjusted for age, and adjusted for gender overall effect sizes are presented in the first, second, and third columns of Table 1, respectively. Insert Table 1 here Second, observations from 23 subjects who received ecological and milieu type of intervention were analyzed by the two approaches. The focus of this analysis is to see whether intervention type and a moderate number of subjects produce different patrons in the estimates of the effect sizes. Again, unadjusted, adjusted for age, and adjusted for

14 Meta-analysis using HLM 14 gender overall effect sizes are presented in the first, second, and third columns of Table 2, respectively. Insert Table 2 here Finally, observations from 12 subjects who received peer supported type of intervention were again analyzed by the two approaches. Because all the 12 subjects were male no gender adjustment was required. Unadjusted, and adjusted for age overall effect sizes are presented in the first and second columns of Table-3, respectively. Insert Table 3 here Tables 1 and 2 reveal that, except when adjusting for age, effect sizes estimates from the V-known HLM approach were in general higher than those produced by twolevel HLM approach. This can be attributed to the differences in the estimation of the effect size make up by the two approaches. The two-level HLM model uses individuals raw data to produce weighted shrunken estimates (empirical Bays estimates) of level-one intervention effect parameters β 1j, and level two overall intervention fixed effect γ 1o. Each of these two parameters made up the numerator of subject s specific effect size and the overall effect size respectively. Further variance estimates from the two levels HLM analysis accounts for two types of variations. Within and between subject variations which cause the denominator of the effect size to be large, therefore, producing smaller effect sizes. Effect sizes variances in level one V-known HLM approach were assumed known and fixed. Their estimates were used in the HLM model to produce an overall effect size. The standard deviation used in the denominator of the effect size does not

15 Meta-analysis using HLM 15 account for the within subject variation. Thus, the denominator of the effect size is based on a smaller standard deviation than the one in two levels HLM, which explain the reason for having larger effect sizes. In addition, estimation of the first level variances in the V- know approach does not account for the correlations between observations for each subject. This can lead to under estimation of these variances. Numbers enclosed in parentheses in the three tables represent the variance of subjects specific effect sizes around the overall effect size. For the V-known HLM approach these numbers were directly estimated by the program and provided as part of the output. For the two levels HLM approach they were not immediately available and were estimated using equation (4). Clearly, variances from the V-known approach are much smaller than those from the two levels HLM approach. This is can be attributed to the same facts presented above about the differences in structure of the variance components of the two approaches. To summarize, the advantage of using one approach over the other depends on several considerations. First, unlike group comparison studies, single-subject studies usually provide the meta-analyst with raw data. This is clearly an advantage that one should considered when combining data from single-subject studies. Of the two approaches the two levels HLM approach utilizes all the information the raw data provide. Therefore, it accounts for all possible variations in the data when calculating effect sizes. Second, one drawback of using raw data is that when the outcome is measured differently across subjects, data can not be combined across subjects. Under such circumstances, V-known HLM approach becomes an alternative method for the meta-analyst.

16 Meta-analysis using HLM 16 Finally, both approaches provide maximum likelihood estimates for the effect sizes which are based on large sample theory with normality assumption. Estimates derived from small number of subjects, using these two approaches, might not have the same desirable properties of maximum likelihood estimates.

17 Meta-analysis using HLM 17 Table - 1 Over All Effect Sizes (ES) and their estimated variances for Academic/Communication outcome Type of HLM Analysis ES ES Adjusted for Age ES Adjusted for Gender Male Female Using the raw data in two levels HLM (3.3407) (3.4340) (0.5940) (3.2847) Using ES with V- known HLM (0.0013) (0.0014) (0.0013) (0.0013) Note: 1-Numbers in parentheses are the estimated variances of the ES. 2- Number of cases = 35 Table 2 Effect Sizes (ES) and their estimated variances of Ecological and Milieu intervention on Academic/Communication outcome Type of HLM Analysis Using the raw data in two levels HLM Using ES with V- known HLM ES (2.9456) (0.0018) ES Adjusted for Age Notes: 1- Numbers in parentheses are the estimated variances of the ES. 2- Number of cases = 23. ES Adjusted for Gender Male Female (3.0422) (0.5808) (4.7209) (0.0019) (0.0019) (0.0019) Table 3 Effect Sizes (ES) and their estimated variances of Peer Supported intervention on Academic/Communication outcome Type of HLM Analysis Using the raw data in two levels HLM Using ES with V- known HLM ES ( ) (0.0009) ES Adjusted for Age ( ) (0.0011) Notes: 1-Numbers in parentheses are the estimated variances of the ES 2-All subjects are male. 3- Number of subjects = 12.

18 Meta-analysis using HLM 18 References Busk, P.L., & Serlin, R.C. (1992). Meta-analysis for single-case research. In T.R. Kratochwill & J.R. Levin (Eds.), Single-case research design and analysis: New directions for psychology and education (pp ). Mahwah, NJ: Erbaum. Center, B.A., Skiba, R.J., & Casey, A. ( ). A methodology for the quantitative synthesis of intra-subject design research. Journal of Special Education, 19, Corcoran, K.J. (1985). Aggregating the idiographic data of single-subject research. Social Work Research and Abstracts, 21, Glass,G.V. (1976). Primary, secondary, and meat-analysis of research. Educational Researcher, 5, 3-8 Hedges, L.V. (1994). Fixed Effects Models. In H. Cooper, and L. V. Hedges (Eds.), The handbook of Research Synthesis, New York: Russell Sage Foundation. McConnell, S.R. (2002). Interventions to facilitate social interaction for young children with autism: Review of available research and recommendations for educational intervention and future research. Journal of Autism and Developmental Disorders, 32, Myers, J.L & Well, A.D. (2003). Research Design and Statistical Analysis. Lawrence Erlbaum Associates, Publishers. Raudenbush, S.W. & Bryk, A.S. (2002). Hierarchical Linear Models: application and data analysis methods. 2 nd edition. Sage Publications, Inc. Ritvo, E.K. & Freeman, B.J. (1978). National society for autistic children; Definition of the syndrome autism. Journal of Autism and Childhood Schizophrenia, 8, Scruggs, T.E., Mastropieri, M.A., & Casto, G. (1987). The quantitative synthesis of singlesubject research: Methodology and validation. Remedial and Special Education, 8, Van den Noortgate, W., & Onghena, P. (2003). Combining single-case experimental data using hierarchical linear models. School Psychology Quarterly, 18, White, D.M., Rusch, F.R., Kazdin, A.E., & Hartmann, D.P. (1989). Applications of metaanalysis in individual-subject research. Behavioral Assessment, 11,