Putting the Patient in Patient Reported Outcomes: A Robust Methodology for Health Outcomes Assessment

Transcription

1 Putting the Patient in Patient Reported Outcomes: A Robust Methodology for Health Outcomes Assessment May 2014 Abstract When analyzing many health-related quality-of-life (HRQoL) outcomes, statistical inference is often based on the summary score formed by combining the individual domains of the HRQoL profile into a single measure. Through a series of Monte Carlo simulations, this paper illustrates that reliance solely on the summary score may lead to biased estimates of incremental effects, and I propose a novel two-stage approach that allows for unbiased estimation of incremental effects. The proposed methodology essentially reverses the order of the analysis, from one of aggregate, then estimate to one of estimate, then aggregate. Compared to relying solely on the summary score, the approach also offers a more patient-centered interpretation of results by estimating regression coefficients and incremental effects in each of the HRQoL domains, while still providing estimated effects in terms of the overall summary score. I provide an application to the estimation of incremental effects of demographic and clinical variables on HRQoL following surgical treatment for adult scoliosis and spinal deformity. Word, Table, and Figure Count: Approximately 4950 words of body text (excluding footnotes), 6 tables, 2 figures Running Head: Putting the Patient in PROMs 1

2 JEL Classification: I10, C24, C25, C34, C35, C51 Keywords: patient-reported outcome measures, quality-adjusted life-years, cost-effectiveness, comparative-effectiveness Funding: This project was supported by grant number XX from the Agency for Healthcare Research and Quality. The content is solely the responsibility of the author and does not necessarily represent the official views of the Agency for Healthcare Research and Quality. 2

3 1 Introduction Improving the efficiency of health care delivery hinges on accurate methodologies for economic evaluation and comparative effectiveness. Accompanying results must also be sufficiently parsimonious so as to ensure the appropriate interpretation and dissemination of findings. To this end, substantial research has been devoted to the appropriate analysis of health-related quality-of-life (HRQoL) and, more generally, patient-reported outcome measures (PROMs). The U.K. s National Health Service (NHS) explicitly mandates the use of such data in health care decision making, and the U.S. appears to be following suit with substantial investment in the Patient-Centered Outcomes Research Institute (PCORI) created under the Patient Protection and Affordable Care Act (PCORI, 2012; Selby et al., 2012; Devlin et al., 2010; Department of Health, 2008). PCORI was specifically created to promote and ultimately fund the development of comparative effectiveness research in health care, although they are statutorily prohibited from funding cost effectiveness research aimed at estimating costs per qualityadjusted life years (QALYs). For the purposes of economic evaluation and comparative effectiveness, PROMs are of interest for several reasons. First, they are outcome measures rather than process measures, the latter of which dominate the quality measures reported by the Centers for Medicare and Medicaid Services (CMS) and the National Committee for Quality Assurance (NCQA, 2008). Only recently has CMS started closely tracking outcome measures such as 30-day readmissions and mortality. Second, PROMs can be consistently studied across a range of conditions and treatment options, offering a more appropriate comparison of treatments than is typically available with purely clinical outcome measures. Third, a patient s self-reported HRQoL is generally considered to be a valuable health outcome measure and one which providers should routinely seek to improve (Porter, 2010; Ahmed et al., 2012). A recent article in the Wall Street Journal described HRQoL data as [helping] medical providers see the big picture...and makes for happier, healthier patients, stating that increased reliance on HRQoL measures was transforming 3

4 health care (Landro, 2012). Finally, and perhaps most importantly, PROMs offer the potential for truly patient-centered care, allowing providers to administer and evaluate health care based on outcomes elicited directly from patients themselves (Porter, 2010). Despite the growing awareness and use of PROMs, I argue in this paper that existing methodologies for analyzing HRQoL data are deficient because they rely solely on the HRQoL summary score in estimating incremental effects. Specifically, the most common approach to analyzing HRQoL data is to combine individual HRQoL domains into a single summary score using some existing scoring algorithm. These summary or index scores are often then used as weights over time in order to estimate QALYs (Powell, 1984; Austin, 2002; Manca et al., 2005; Drummond et al., 2005; Brazier & Ratcliffe, 2007; Gray et al., 2011; Basu & Manca, 2012). Aside from normative concerns regarding which weights to use, an analysis based solely on the summary scores is flawed for at least three reasons. First, relying on the summary scores comes with an inherent loss of information and may ultimately bias incremental effects estimates (Mortimer & Segal, 2008; Gutacker et al., 2012; Parkin et al., 2010). For example, in many HRQoL outcome measures, there exists variation in the underlying domain scores that is not reflected in the summary score (Brazier & Ratcliffe, 2007; Gray et al., 2011). This loss of variation is inherent to the scoring process and not due to any specific algorithm. Second, the empirical distribution of summary scores is often subject to significant floor or ceiling effects and may also be multi-modal, necessitating empirical methodologies more complicated than a simple linear regression (Austin, 2002; Manca et al., 2005; Basu & Manca, 2012; Hernández Alava et al., 2012). The extent to which alternative distributional assumptions regarding the summary score approximate the true distribution will vary by application. Third, and perhaps more importantly, the reliance on summary scores reflects a fundamental divide between the actual outcomes effected versus the outcomes being analyzed. For researchers interested in the effect of some covariate on HRQoL, these effects occur by definition at the individual domain level since this is the level at which respondents are asked about their quality of life (e.g., the physical functioning 4

5 or mental health domains of a larger HRQoL profile). Effects on the summary score are somewhat artificial as they exist only by combining the individual domains and associated effects. It is unclear a priori whether the effects estimated at each domain and then combined to form an effect on the summary score would yield the same result as an analysis based solely on the summary score. In fact, as the findings in Section 3 indicate, the order of estimation and aggregation to the summary score is an important (but unappreciated) aspect of statistical inference. As a result, there is growing concern in the literature regarding the appropriateness of HRQoL summary scores as the outcome of interest (Sculpher & Gafni, 2001; Brazier et al., 2009). For example, Gutacker et al. (2012) considers an ordered probit model in analyzing EQ-5D scores, accounting for baseline quality-of-life through the panel structure and exploiting the ordered probit construct to explicitly model individual domain scores. The authors avoid an analysis based solely on the summary scores. Devlin et al. (2010) considers an alternative classification system and a health profile grid, each of which exploit rankings of EQ-5D health states and attempt to summarize patient outcomes based on those reporting an unequivocal improvement, worsening, or no change in health. The studies of Gutacker et al. (2012), Devlin et al. (2010), and others illustrate concern surrounding the appropriateness of relying solely on summary scores in estimating the effects of an intervention and other covariates on a patient s well-being. However, in avoiding the summary scores entirely, these approaches are silent as to the incremental effects on the summary score and offer little in terms of comparing results across other studies (where summary scores remain the primary outcome of interest). This paper proposes a novel two-stage estimator (2SE) that first estimates regression coefficients and incremental effects based on the full HRQoL profile and then re-interprets these effects in terms of the summary score. Through a series of Monte Carlos simulations, the paper illustrates how a reliance solely on the summary score may lead to biased incremental effects estimates, while the 2SE is shown to restore the unbiased estimation of incremental effects. The proposed methodology essentially 5

6 reverses the order of the analysis, from one of aggregate, then estimate to one of estimate, then aggregate. The 2SE also allows for a more patient-centered discussion wherein the incremental effects of treatment or other covariates are domain-specific and more applicable to areas of health deemed most important to a given patient. Importantly, by re-interpreting the incremental effects in terms of summary scores, the 2SE maintains the parsimonious interpretation that has proven so valuable in the applied cost- and comparative-effectiveness literature. I then apply the 2SE along with other common estimators in the literature to a prospective, multi-center dataset on HRQoL outcomes for adult scoliosis and spinal deformity patients. The current paper therefore contributes to the growing empirical literature on the appropriate analysis of HRQoL outcomes. This analysis is also broadly related to theoretical econometric research surrounding the differences between marginal effects calculated from multivariate estimation versus marginal effects calculated from univariate outcomes formed by collapsing the underlying multivariate outcomes (Mullahy, 2011). I discuss the empirical framework and 2SE in Section 2. Details of the Monte Carlo exercise are presented in Section 3, with an application presented in Section 4. Section 5 concludes. 2 Methodology The primary goal of the current analysis is to accurately estimate the effect of a covariate, x, on a patient s HRQoL summary score. For consistency with the empirical application in Section 4, I adopt the SF-6D as the measure of HRQoL; however, the intuition and methodological contribution of the paper extends to similar metrics such as the EQ-5D. 6

7 2.1 Summary of the SF-6D The SF-6D is a six-dimensional health profile derived from a subset of responses from the SF-36 or SF-12 (Brazier et al., 2002; Brazier & Ratcliffe, 2007). The six dimensions of health classified by the SF-6D are: 1) physical functioning; 2) role limitations; 3) social functioning; 4) pain; 5) mental health; and 6) vitality. Each domain is characterized numerically with a range of integers, where a 1 indicates the best value in each domain. The worst value in each domain varies, with values up to 6 in the physical functioning and pain domains, values up to 5 in the social functioning, mental health, and vitality domains, and values up to 4 in the role limitations domain. The patient s full SF-6D profile is therefore characterized by a series of six integers, with the best health state represented by {1, 1, 1, 1, 1, 1} and the worst health state represented by {6, 4, 5, 6, 5, 5}. Taking all possible combinations of responses, the SF-6D defines 18,000 unique health states. Each health state can then be converted into a single index score using available scoring algorithms that essentially assign weights to each domain and interactions between domains. Following the algorithm in Brazier & Ratcliffe (2007), the resulting SF-6D index score ranges from 0.30 to 1.0, with 0.30 representing the poorest health state, {6, 4, 5, 6, 5, 5}, and 1 representing the best health state, {1, 1, 1, 1, 1, 1}. The scoring algorithm from Brazier & Ratcliffe (2007) is reproduced in Table 1. Table 1 The appropriate algorithm to calculate a summary score remains an area of debate in the literature (Parkin et al., 2010). Importantly, the proposed methodology relies on the scoring algorithm only to reinterpret the estimated incremental effects in terms of the summary score. Although the estimated incremental effects will certainly differ depending on the scoring algorithm adopted, the focus of this paper is on highlighting the bias introduced when relying solely on the summary score. To this end, the intuition underlying this analysis extends broadly to other scoring algorithms, including some 7

8 of the more recent literature on HRQoL crosswalks intended to convert responses from one HRQoL instrument into those of another instrument (Dakin, 2013). 2.2 The Two-Stage Estimator The proposed 2SE applies when one is interested in estimating the incremental effect of some covariate on a summary score, which is itself derived from a combination of individual responses. Several alternative models have also been proposed to estimate such effects, including ordinary least squares (OLS), variations of the classic Tobit model, censored least-absolute deviations models, Beta MLE, and Beta QMLE models (Powell, 1984; Austin, 2002; Basu & Manca, 2012). 1 Rather than rely on the univariate outcome, the 2SE first estimates the coefficients of interest based on the underlying SF-6D responses and then re-interprets the coefficients in terms of the summary score. 2 The 2SE first models each individual health domain using an ordered probit model (Gutacker et al., 2012), where the response in each domain intuitively follows from a latent index variable, y id = x i β d + ε id. (1) Here, x i denotes a set of independent variables possibly including a constant term, d denotes the relevant health domain, d = 1,..., 6, and ε id is assumed to follow a normal distribution with µ = 0 and σ = 1. In general, ε id could be correlated across domains. Such correlation could be accounted for in the proposed methodology (e.g., by adopting a composite marginal likelihood estimation for multivariate ordered probit or logit models as in Bhat et al. (2010)); however, such an approach would only impact the efficiency of the estimated coefficients and would not impact the point estimates. As such, I simplify the analysis by assuming zero cross-equation correlation. 1 We ignore issues of selection or the role of baseline HRQoL in order to focus solely on the estimation of incremental effects in settings where standard regression models are considered appropriate. 2 Since QALYs generally reflect health states as well as the time spent in each health state, I do not treat QALYs as synonymous with the HRQoL summary scores; however, as Basu & Manca (2012) indicates, it is relatively common in practice that researchers estimate QALYs based on a single followup survey administered at one year after treatment, in which case the summary score is equivalent to a QALY. 8

9 Denote by y id the observed response for patient i in domain d. For example, in the physical functioning domain (d = 1), y i1 {1,..., 6}. As y i1 crosses several unknown thresholds (denoted by α j ), the observed response moves up the health status ranking such that y i1 = 1 for α 0 < y i1 < α 1 and y i1 = 6 for y i1 α 5. Note that the ordering from best to worst or worst to best is irrelevant provided the appropriate adjustments are made when estimating summary scores. Since most statistical software programs estimate ordered discrete choice models such that a higher value is better, I adopt a worst to best ordering in the analysis, which I then convert to a best to worst ordering to apply the scoring algorithm. More compactly, the observed dependent variable, y id, takes the form y id = j if α d,j 1 y id α d,j, j = {1,..., }, (2) where differs across domains as discussed previously. Importantly, even with a well-behaved distribution of latent variables, yid, the ordered discrete choice framework can generate distributions with strong floor and ceiling-effects via different threshold values, α j. As a result, the estimation of ordered, discrete dependent variable models can avoid the distributional and statistical difficulties present in models based solely on the summary scores. I estimate separate ordered probit models for each HRQoL domain, and the results of each model are used to form predicted probabilities of responses, denoted ˆP d ij, for person i, response j, and domain d. In the physical functioning domain, the regression results therefore provide six predicted probabilities for each person, ˆP ( P F i1, ˆP i2 P F,..., ˆP ) i6 P F. Continuing this process across all six domains yields a total of 31 predicted probabilities - one for each possible response in each domain - for each person. Applied to HRQoL measures like the SF-6D, one difficulty surrounds the most severe category, where Brazier et al. (2002) defines most severe as any one of the following responses: a level of 4 or more in the physical functioning, social functioning, mental health, or vitality domains; a level of 3 or more in the role limitation domain; or a level of 5 or more in the pain domain. The probability of a most severe health status can then be calculated 9

10 following the principle of inclusion and exclusion for probability. 3 With a slight abuse of notation, the inclusion-exclusion principle states that the probability of the union of N non-mutually exclusive events is given as: N P (A 1 A 2... A N ) = P (A 1 ) P (A N ) + ( 1) n+1 P ( n events). (3) Applied to the SF-6D, I denote by A P F the outcomes of the physical functioning domain that enter into the most severe indicator, and similarly by A RL for the role limitations domain, A SF for the social functioning domain, A P for the pain domain, A MH for the mental health domain, and A V for the vitality domain. Since only one value can be reported in each domain, these terms enter directly into equation 3, where n=2 P (A P F ) = P r(p F = 4) + P r(p F = 5) + P r(p F = 6), P (A RL ) = P r(rl = 3) + P r(rl = 4), P (A SF ) = P r(sf = 4) + P r(sf = 5), P (A P ) = P r(p ain = 5) + P r(p ain = 6), P (A MH ) = P r(mh = 4) + P r(mh = 5), and P (A V ) = P r(v = 4) + P r(v = 5). An estimate of P (A 1 A 2... A 6 ), denoted ˆP (Most Severe), can therefore be obtained by applying the inclusion-exclusion principle to the individual estimates of the probabilities of each outcome in each domain, ˆP d ij. Based on the scoring algorithm in Table 1, the probability estimates from the ordered probit estimation can then be 3 A similar term which combines the scores across several individual domains also appears in the EQ-5D scoring algorithm (Shaw et al., 2005; Agency for Healthcare Research and Quality, 2005). 10

11 converted to a predicted SF-6D summary score, Ŝi: Ŝ i = ( ˆP P F i2 ( ˆP RL i2 + ˆP ) i3 P F + ˆP RL i ˆP i4 P F ) ˆP RL i ˆP P F i ˆP P F i6 (4) SF ˆP i2 ( ˆP P ain i2 ( ˆP MH i2 ( ˆP V i2 + ˆP V i3 + ˆP V i4 ˆP SF ˆP SF i i ) P ain + ˆP i ) MH MH + ˆP i ˆP i4 ) ˆP i5 V ˆP P ain i ˆP SF i5 ˆP P ain i5 ˆP MH i ˆP P ain i ˆP (Most Severe). In an of itself, the predicted summary score is of little value. If researchers were interested only in the value of a respondent s summary score, then clearly the observed summary score formed from the observed responses would be most relevant. The predicted summary score is instead critical to the estimation of incremental effects via the method of recycled predictions (Oaxaca, 1973; Graubard & Korn, 1999; Basu & Rathouz, 2005; Basu, 2005; Glick, 2007; Kleinman & Norton, 2009). For example, if we are interested in the average effect of a one standard deviation increase in x on respondents summary scores, the 2SE would proceed as follows. First, estimate ordered probit models in each domain and form the predicted summary score based on the observed independent variables, Ŝ i x i. Second, replace x i with the hypothetical values of interest, x i = x i + σ x, and based on the same coefficients estimated from the ordered probit models, form the predicted summary scores for these hypothetical values, Ŝ i x i. Taking the difference in each predicted summary score, Ŝ i x i Ŝi x i, and averaging across all individuals provides an estimate of the average effect of a one standard deviation change in x. This recycled predictions method (also referred to as predictive margins) also avoids the difficulty of computing and interpreting marginal effects in nonlinear models (Norton et al., 2004) and can be particularly valuable when the variable of interest is interacted with other covariates. 11

12 By definition, the predicted probabilities from the first stage regressions are estimates of the true probabilities and are therefore uncertain. To accommodate this variation, standard errors and confidence intervals around the incremental effects are estimated via bootstrap, where each iteration of the bootstrap includes both stages of the 2SE. Uncertainty surrounding the parameters in the ordered probit model is therefore incorporated into the final estimated effects. 3 Simulation I simulate data consistent with the latent index model discussed above. Alternatively, authors sometimes simulate summary scores directly under a series of different distributional assumptions (e.g., Basu & Manca (2012)); however, in application, the level of measurement is always at the individual HRQoL domain, and summary scores are only generated after converting the individual domain scores. Simulation based on the underlying HRQoL domains is therefore more consistent with the likely DGPs encountered in practice. 3.1 Data In practice, the distribution of summary scores is often highly skewed, censored, and multi-modal. For example, in a large study of laparoscopic-assisted versus abdominal hysterectomy (the EVALUATE trial), the observed distributions in both treatment arms were highly left-skewed with strong ceiling-effects at 1 (Basu & Manca, 2012; Garry et al., 2004; Sculpher et al., 2004). Basu & Manca (2012) reproduces graphs from several additional applications in which the summary score distributions are similarly skewed, censored, or bi-modal. To reflect the breadth of distributions encountered in practice, I simulate data under several alternative DGPs. The DGPs are intentionally over-simplified in order to generate distributional properties of interest and to focus specifically on the estimation of 12

13 incremental effects. In all cases, I simulate a latent continuous variable for each HRQoL domain (d = 1,..., 6), denoted yid, as a function of a single independent variable, x i, and a normal i.i.d. error term, ε id. Denote by γ the intercept coefficient and by β the coefficient on x. Then the D 1 vector of latent HRQoL values, yi, is as follows: yi = γ + βx i + ε i, where ε N (0 D 1, I D D ), x U[0, 1], γ = I D 1, and β = 1.5 I D 1. Observed HRQoL values, y id for d (1, 2, 3, 4, 5, 6), are then generated based on the value of the latent value, y id, relative to the 1 vector of threshold values in each domain, α d, where = 6 in the physical functioning and pain domains, = 4 in the role limitations domain, and = 5 in the social functioning, mental health, and vitality domains. Alternative specifications of α are used to generate different distributional properties of the summary scores. Specifically, I consider five different threshold values corresponding to each of five distributions of interest. In each domain, threshold values are set to specific quantiles of the empirical distribution of the latent variable, F (y d ). Denoting the τ j th quantile by q y d (τ j ) for all j {1,..., }, data are simulated under the following alternative specifications of τ j : 1. τ = [.1,.3,.5,.7,.9, 1] in the physical functioning and pain domains, τ = [.1,.3,.6,.8, 1] in the social functioning, mental health, and vitality domains, and τ = [.1,.4,.8, 1] in the role limitations domain. These values for τ generate a bell-shaped distribution between 0.3 and 1, illustrated in panel (a) of Figure τ j = 0.5 j, which generates a right-censored distribution, illustrated in panel 13

14 (b) of Figure τ j = 0.25 j, which generates a heavily right-censored distribution, illustrated in panel (c) of Figure 1. ( ) 4. τ j = j + j, which generates a left-censored distribution, illustrated in panel (d) of Figure τ j = 0.5 ( 1 j ) + j, which generates a heavily left-censored distribution, illustrated in panel (e) of Figure 1. Figure Monte Carlo Results The focus of the Monte Carlo study is to compare incremental effects in the summary score domain calculated with existing regression methods to the incremental effects calculated using the 2SE. The primary hypothesis is that an ordered discrete choice model (e.g., an ordered probit or logit) can better accommodate the idiosyncratic properties of distributions encountered in practice. By modeling HRQoL domains directly and then re-interpreting in terms of the summary score, the results are therefore (arguably) more robust to a wide range of distributions relative to models based solely on the summary score. For each of the five DGPs discussed above, I simulate 1,000 datasets consisting of N = 500 observations (patients). I estimate coefficients with four alternative estimators: 1) 2SE; 2) standard OLS; 3) the Beta MLE model proposed in Basu & Manca (2012); and 4) the Beta QMLE also proposed in Basu & Manca (2012). In all cases, incremental effects are calculated using the method of recycled predictions as discussed previously, interpreted as the average change in summary scores following a one standard deviation change in x. The results are summarized in Table 2. 14

15 Table 2 The 2SE consistently provides accurate estimates of the true incremental effect across a range of alternative distributions. By comparison, incremental effects estimated with OLS are downward (upward) biased in the presence of sufficient ceiling (floor) effects. The Beta MLE and QMLE estimators perform better than OLS; however, the Beta MLE estimator still provides biased estimates in the presence of uniformly distributed summary scores with mild ceiling effects (DGP 2). In addition, Beta MLE and Beta QMLE estimators are both less accurate relative to the 2SE, where estimates from the latter are generally centered around the true effects while estimates from the Beta MLE and Beta QMLE models differ from the true effect by 10% or more on average. The 2SE also provides the lowest RMSE in all cases, although the differences in RMSE across estimators are minimal and statistically insignificant. 4 As discussed in Basu & Manca (2012), if the true marginal effect is relatively small and the data are subject to strong ceiling or floor effects, biases in marginal effects may be relatively minor. I therefore simulated additional datasets with β = 5 I D 1 rather than β = 1.5 I D 1. I focus on DGPs 3 and 5 above (strong ceiling and floor effects, respectively), where any bias would be most apparent. Results are summarized in Table 3. Here, the 2SE provides accurate estimates of the true incremental effect, while all other estimators yield biased estimates. Differences in RMSE are also larger relative to those in Table 2, with the 2SE again providing the minimum RMSE in all cases. Table 3 4 Although the efficiency of these estimates will clearly depend on the overall model fit, the results are qualitatively unchanged when considering alternative simulations in which the model fit is intentionally reduced (via a larger variance in the distribution of the error term, ε). Moreover, there would be no reason in practice to propose a different set of independent variables for the 2SE compared to another estimator such as standard OLS or Beta MLE. Concerns regarding the choice of covariates therefore apply equally to all estimators considered in the analysis. Results are similarly unchanged when allowing for non-zero cross-equation correlation across HRQoL domains. Results from these sensitivity analyses are excluded for brevity but available upon request. 15

16 4 Application to Scoliosis Surgery I apply the proposed 2SE to the estimation of the effect of observed pre-operative variables on post-operative HRQoL and summary scores following surgical treatment for adult spinal deformity (ASD). Surgical treatment of ASD is one of the lesser studied but fastest growing and most expensive areas of spine surgery, affecting as much as 32% of the adult population and up to 60% of the elderly (Robin et al., 1982; Schwab et al., 2003, 2005, 2008). 4.1 Data The data for this study were collected from a multi-center, prospective database maintained by the International Spine Study Group (ISSG). The dataset consists of 209 adult scoliosis and spinal deformity patients undergoing surgery at any participating ISSG member site, with institutional review board approval obtained at all centers. For purposes of this application, I limit the analysis to the following covariates: 1) age; 2) gender; 3) baseline SF-6D scores; 4) total number of vertebrae fused at surgery (i.e., the number of levels fused); and 5) surgical approach. The outcome of interest is patients HRQoL one year after surgery. Summary statistics are provided in Table 4. Table Results Coefficient estimates are provided in Table 5. Although the coefficients in the ordered probit regressions do not easily compare to those from the OLS, Beta MLE, and Beta QMLE regressions, the ordered probit analysis immediately allows a more patientcentered interpretation than is provided by the other estimators. To the extent that a given patient s preferences are such that certain health domains are more important 16

17 than others, the results may support a more meaningful discussion for shared decisionmaking purposes. The ordered probit analysis also reveals important differences across health domains that are not identified in the other estimators. Namely, the role of age, gender, levels fused, and baseline HRQoL clearly differs across health domains, with age having a significant positive impact in some domains, a significant negative impact on others, with no significant impact on overall HRQoL. Similarly, gender and surgical approach are estimated to have no significant impact on overall HRQoL despite having a significant effect on the role limitations domain. Table 5 The impact of baseline HRQoL is also more clearly represented with the ordered probit results. For example, post-operative mental health scores are influenced heavily by a patient s baseline mental health score, much more so than in the other health domains. This is consistent with the underlying nature of the disease, which can have major negative effects on a patient s daily activities and body image, but may not generally impact a patient s overall mental health. As such, for two patient s with an identical SF-6D index score, a patient with lower baseline mental health will have relatively less opportunity for HRQoL improvement following surgery. This interpretation would not be available with the standard empirical framework based solely on the summary scores (Manca et al., 2005). Incremental effects estimated from the method of recycled predictions are summarized in Table 6. For binary variables such as Female and Posterior Approach, the incremental effect represents the predicted change in summary scores for women relative to men and for patient s with a posterior approach relative to a combined anterior/posterior approach, respectively. For age, the incremental effect represents the predicted change in the summary score following a one-year increase in age at surgery; and for levels fused and each HRQoL domain, the incremental effects represent the predicted change in summary scores following a one-unit increase (improvement) from the 17

18 median (e.g., an increase from 9 to 10 levels fused or from a baseline physical functioning domain score of 4 to 3). As should be the case given the well-behaved distribution of summary scores, the incremental effects for age, gender, levels fused, and surgical approach are similar for all estimators considered. Table 6 The results from Table 6 also illustrate the loss of variation when estimating effects based solely on the summary score. For example, an improvement from 4 to 3 or from 3 to 2 in a patient s baseline role limitations domain will have no impact on the patient s summary score because the scoring algorithm is such that the score does not vary along these values of the role limitations domain. A similar scenario unfolds for certain values of the physical functioning, pain, mental health, and vitality domains. Because of this loss of variation due to the scoring algorithm, incremental effects estimates for the role limitations or mental health domains are not available when relying solely on the summary score in the current application. By modeling each domain separately, the 2SE avoids this problem and allows for a more complete estimation of incremental effects at all values of each baseline HRQoL domain. 5 5 Discussion This paper develops a new two-stage estimator (2SE) for analyzing HRQoL outcomes which offers important benefits relative to existing methodologies. Primarily, the paper illustrates how a reliance solely on the summary score may lead to biased incremental effects estimates, while the 2SE is shown to restore the unbiased estimation of incremental effects. The proposed methodology essentially reverses the order of the 5 Such differences could be avoided somewhat by including each baseline HRQoL domain score as a covariate in the OLS, Beta MLE, and Beta QMLE regressions; however, this is not the standard approach adopted in the literature. Moreover, this approach would not fully resolve the differences, as incremental effects under the 2SE remain higher in the mental health and vitality domains, and lower in the pain domain. Results of this analysis are not included but are available upon request. 18

19 analysis, from one of aggregate, then estimate to one of estimate, then aggregate. The 2SE also allows for a more patient-centered discussion wherein the incremental effects of treatment or other covariates are domain-specific and more applicable to areas of health deemed most important to a given patient. Importantly, the 2SE offers a unified framework by which to estimate incremental effects at the individual domain level while still interpreting these same effects in terms of the overall summary score. The improvements offered by the 2SE come at some cost. Namely, the 2SE is analytically more difficult to implement than a standard OLS and perhaps more complicated than the Beta MLE, Beta QMLE, and other estimators relying solely on the summary score. The 2SE also requires sufficient sample size (larger than standard OLS) in order to estimate the ordered dependent variable models. However, as shown through the Monte Carlo exercise, the standard estimators are less robust to the idiosyncratic distributional properties of summary scores than is the 2SE. Moreover, the 2SE allows for an interpretation in terms of summary scores just as the OLS, Beta MLE, and Beta QMLE models do. The added computational burden therefore falls solely on the analyst rather than the end-user of the results. As such, the proposed 2SE offers an improvement over existing estimators with no additional complexity for the end-user. In light of the growing use of patient-reported outcome measures for purposes of provider comparison and quality reporting (Nuttall et al., 2013), the proposed 2SE should be considered as an alternative estimator for analysis of HRQoL outcomes in practice. 19

20 References Agency for Healthcare Research and Quality Calculating the U.S. Populationbased EQ-5D Index Score. Ahmed, Sara, Berzon, Richard A, Revicki, Dennis A, Lenderking, William R, Moinpour, Carol M, Basch, Ethan, Reeve, Bryce B, Wu, Albert W, et al The use of patient-reported outcomes (PRO) within comparative effectiveness research: implications for clinical practice and health care policy. Medical Care, 50(12), AHRQ Sample. Healthcare Cost and Utilization Project (HCUP), National Inpatient Austin, P.C A comparison of methods for analyzing health-related quality-of-life measures. Value in Health, 5(4), Basu, A., & Manca, A Regression Estimators for Generic Health-Related Quality of Life and Quality-Adjusted Life Years. Medical Decision Making, 32(1), Basu, Anirban Extended generalized linear models: simultaneous estimation of flexible link and variance functions. Stata Journal, 5(4), Basu, Anirban, & Rathouz, Paul J Estimating marginal and incremental effects on health outcomes using flexible link and variance function models. Biostatistics, 6(1), Bhat, C.R., Varin, C., & Ferdous, N A comparison of the maximum simulated likelihood and composite marginal likelihood estimation approaches in the context of the multivariate ordered-response model. Advances in Econometrics, 26, Brazier, J., & Ratcliffe, J Measuring and valuing health benefits for economic evaluation. Oxford University Press, USA. Brazier, J., Roberts, J., & Deverill, M The estimation of a preference-based measure of health from the SF-36. Journal of health economics, 21(2),

21 Brazier, John E, Dixon, Simon, & Ratcliffe, Julie The role of patient preferences in cost-effectiveness analysis. Pharmacoeconomics, 27(9), Dakin, Helen Review of studies mapping from quality of life or clinical measures to EQ-5D: an online database. Health and quality of life outcomes, 11(1), 151. Department of Health Guidance on the Routine Collection of Patient Reported Outcome Measures (PROMs). Devlin, N.J., Parkin, D., & Browne, J Patient-reported outcome measures in the NHS: new methods for analysing and reporting EQ-5D data. Health economics, 19(8), Drummond, M.F., Sculpher, M.J., & Torrance, G.W Methods for the economic evaluation of health care programmes. Oxford University Press, USA. Garry, Ray, Fountain, Jayne, Mason, Su, Hawe, Jeremy, Napp, Vicky, Abbott, Jason, Clayton, Richard, Phillips, Graham, Whittaker, Mark, Lilford, Richard, et al The evaluate study: two parallel randomised trials, one comparing laparoscopic with abdominal hysterectomy, the other comparing laparoscopic with vaginal hysterectomy. British Medical Journal, 328(7432), Glick, H Economic evaluation in clinical trials. Oxford University Press, USA. Graubard, Barry I, & Korn, Edward L Predictive margins with survey data. Biometrics, 55(2), Gray, A.M., Clarke, P.M., Wolstenholme, J., & Wordsworth, S Applied Methods of Cost-effectiveness Analysis in Healthcare. Oxford Univ Pr. Gutacker, N., Bojke, C., Daidone, S., Devlin, N., & Street, A Analysing Hospital Variation in Health Outcome at the Level of EQ-5D Dimensions. 21

22 Hernández Alava, Mónica, Wailoo, Allan J, & Ara, Roberta Tails from the peak district: adjusted limited dependent variable mixture models of EQ-5D questionnaire health state utility values. Value in Health, 15(3), Kleinman, Lawrence C, & Norton, Edward C What s the risk? A simple approach for estimating adjusted risk measures from nonlinear models including logistic regression. Health services research, 44(1), Landro, L The Simple Idea That Is Transforming Health Care. The Wall Street Journal. Manca, A., Hawkins, N., & Sculpher, M.J Estimating mean QALYs in trial-based cost-effectiveness analysis: the importance of controlling for baseline utility. Health economics, 14(5), Mortimer, D., & Segal, L Comparing the incomparable? A systematic review of competing techniques for converting descriptive measures of health status into QALY-weights. Medical decision making, 28(1), 66. Mullahy, J Marginal Effects in Multivariate Probit and Kindred Discrete and Count Outcome Models, with Applications in Health Economics. Tech. rept. National Bureau of Economic Research. NCQA National Committee for Quality Assurance (NCQA). HEDIS and quality measurement: technical resources. Norton, Edward C, Wang, Hua, & Ai, Chunrong Computing interaction effects and standard errors in logit and probit models. Stata Journal, 4, Nuttall, David, Parkin, David, & Devlin, Nancy Inter-provider Comparison of Patient-reported Outcomes: Developing and Adjustment to Account for Differences in Patient Case Mix. Health Economics. 22

23 Oaxaca, Ronald Male-female wage differentials in urban labor markets. International economic review, 14(3), Parkin, D., Rice, N., & Devlin, N Statistical analysis of EQ-5D profiles: does the use of value sets bias inference? Medical Decision Making, 30(5), PCORI Draft National Priorities for Research and Research Agenda: version 1. Porter, Michael E What Is Value in Health Care? Medicine, 363(26), PMID: New England Journal of Powell, J.L Least absolute deviations estimation for the censored regression model. Journal of Econometrics, 25(3), Robin, G., Span, Y., Steinberg, R., Making, M., & Menczel, J Scoliosis in the elderly: a follow-up study. Spine, 7(4), Schwab, Frank, Dubey, Ashok, Pagala, Murali, Gamez, Lorenzo, & Farcy, Jean P Adult scoliosis: a health assessment analysis by SF-36. Spine, 28(6), Schwab, Frank, Dubey, Ashok, Gamez, Lorenzo, El Fegoun, Abdelkrim Benchikh, Hwang, Ki, Pagala, Murali, & Farcy, J-P Adult scoliosis: prevalence, SF- 36, and nutritional parameters in an elderly volunteer population. Spine, 30(9), Schwab, Frank J, Lafage, Virginie, Farcy, Jean-Pierre, Bridwell, Keith H, Glassman, Stephen, & Shainline, Michael R Predicting outcome and complications in the surgical treatment of adult scoliosis. Spine, 33(20), Sculpher, Mark, & Gafni, Amiram Recognizing diversity in public preferences: The use of preference sub-groups in cost-effectiveness analysis. Health economics, 10(4), Sculpher, Mark, Manca, Andrea, Abbott, Jason, Fountain, Jayne, Mason, Su, & Garry, Ray Cost effectiveness analysis of laparoscopic hysterectomy compared with 23

24 standard hysterectomy: results from a randomised trial. British Medical Journal, 328(7432), Selby, J.V., Beal, A.C., & Frank, L The Patient-Centered Outcomes Research Institute (PCORI) national priorities for research and initial research agenda. JAMA: The Journal of the American Medical Association, 307(15), Shaw, J.W., Johnson, J.A., & Coons, S.J US valuation of the EQ-5D health states: development and testing of the D1 valuation model. Medical care, 43(3),

25 6 Tables and Figures Table 1: Scoring Algorithm for SF-6D a Starting value = 1.0 (perfect health) Physical Functioning (PF) PF=2 or PF= PF= PF= PF= Role Limitations (RL) RL=2 or RL=3 or RL= Social Functioning (SF) SF= SF= SF= SF= Pain (P) P=2 or P= P= P= P= Mental Health (MH) MH=2 or MH= MH= MH= Vitality (V) V=2 or V=3 or V= V= Combination of Domains Most Severe a Algorithm based on Brazier & Ratcliffe (2007). Most Severe denotes any one of the following responses: a level of 4 or more in the physical functioning, social functioning, mental health, or vitality domains; a level of 3 or more in the role limitation domain; or a level of 5 or more in the pain domain. 25

26 Figure 1: Empirical QALY Distributions in Monte Carlo Study Frequency SF-6D Index Score Frequency SF-6D Index Score (a) (d) ( ) τ PF,Pain = [.1,.3,.5,.7,.9, 1] τ j = j + j τ SF,MH,V = [.1,.3,.6,.8, 1] τ RL = [.1,.4,.8, 1] Frequency SF-6D Index Score Frequency SF-6D Index Score (b) τ j = 0.5 j τ j = 0.5 ( (e) ) 1 j + j Frequency SF-6D Index Score (c) τ j = 0.25 j 26

27 Table 2: Incremental Effects Estimates under Alternative DGPs a Model Incremental Effect St. Dev. Mean % Bias Lower % Bias Upper % Bias RMSE DGP 1: τ PF,Pain = [.1,.3,.5,.7,.9, 1], τ SF,MH,V = [.1,.3,.6,.8, 1], τ RL = [.1,.4,.8, 1] True Effect Two-stage Approach % % 11.64% OLS % -8.89% 17.18% Beta MLE % -4.84% 25.44% Beta QMLE % -6.66% 19.96% DGP 2: τ j = 0.5 j True Effect Two-stage Approach % % 11.48% OLS % % 8.39% Beta MLE % 28.34% 76.59% Beta QMLE % -4.26% 25.24% DGP 3: τ j = 0.25 j True Effect Two-stage Approach % % 15.21% OLS % % -1.40% Beta MLE % % 23.44% Beta QMLE % -5.97% 32.68% DGP 4: τ j = 0.25 ( 1 j ) + j True Effect Two-stage Approach % % 11.14% OLS % -2.40% 24.52% Beta MLE % -2.67% 25.71% Beta QMLE % -3.23% 22.88% DGP 5: τ j = 0.5 ( 1 j ) + j True Effect Two-stage Approach % % 11.19% OLS % 2.21% 32.65% Beta MLE % -1.05% 28.53% Beta QMLE % -0.26% 28.56% a Results based on 1,000 bootstrap iterations for N = 500 observations in each DGP. Upper % bias and lower % bias denote the upper and lower 95% confidence intervals of the percent difference between the estimated incremental effect and the true incremental effect. RMSE=root mean squared error. 27

28 Table 3: Incremental Effects Estimates with Larger True Effect a Model Incremental Effect St. Dev. Mean % Bias Lower % Bias Upper % Bias RMSE DGP 3: τ j = 0.25 j True Effect Two-stage Approach % -7.35% 6.17% OLS % % % Beta MLE % % -6.09% Beta QMLE % 19.12% 39.04% DGP 5: τ j = 0.5 ( 1 j ) + j True Effect Two-stage Approach % -3.65% 4.52% OLS % 29.10% 50.09% Beta MLE % 21.11% 45.85% Beta QMLE % 18.95% 36.38% a Results based on 1,000 bootstrap iterations for N = 500 observations in each DGP, with data simulated using β = 5 I D 1 rather than β = 1.5 I D 1. Upper % bias and lower % bias denote the upper and lower 95% confidence intervals of the percent difference between the estimated incremental effect and the true incremental effect. RMSE=root mean squared error. 28

29 Table 4: Summary Statistics for ISSG Data (N=209) Variable Mean Standard Deviation Age Levels Fused Count Percent Female % Posterior Approach 71 34% Baseline Post-operative Count Percent Count Percent Physical Functioning Domain PF=1 0 0% 0 0% PF=2 10 5% 27 13% PF= % 61 29% PF= % 35 17% PF= % 74 35% PF=6 14 7% 12 6% Role Limitations Domain RL=1 8 4% 23 11% RL= % 80 38% RL=3 4 2% 7 3% RL= % 99 47% Social Functioning Domain SF= % 79 38% SF= % 44 21% SF= % 50 24% SF= % 26 12% SF= % 10 5% Pain Domain P=1 1 0% 15 7% P=2 12 6% 27 13% P= % 70 33% P= % 47 22% P= % 35 17% P= % 15 7% Mental Health Domain MH= % 83 40% MH= % 64 31% MH= % 34 16% MH= % 23 11% MH=5 14 7% 5 2% Vitality Domain V=1 4 2% 6 3% V= % 69 33% V= % 67 32% V= % 37 18% V= % 30 14% 29

30 Table 5: Regression Results a OLS Beta Beta Ordered Probit OLS MLE QMLE Outcome: QALY QALY QALY PF RL SF P MH V Age 0.00* * ** * 0.01* 0.00 (0.00) (0.00) (0.00) (0.01) (0.01) (0.01) (0.01) (0.01) (0.01) Female *** * (0.02) (0.11) (0.10) (0.21) (0.22) (0.21) (0.20) (0.23) (0.21) Levels Fused * -0.03* (0.00) (0.01) (0.01) (0.02) (0.02) (0.02) (0.02) (0.02) (0.02) Posterior Approach ** (0.02) (0.09) (0.09) (0.18) (0.19) (0.19) (0.18) (0.19) (0.18) Baseline HRQoL SF-6D Index 0.57*** 2.49*** 2.54*** (0.07) (0.40) (0.37) PF 0.49*** (0.09) RL 0.36*** (0.08) SF 0.39*** (0.07) P 0.43*** (0.07) MH 0.59*** (0.08) V 0.42*** (0.07) RMSE a Results based on OLS, Beta MLE, Beta QMLE, and Ordered Probit regressions. Beta MLE and QMLE estimation follows the procedure and code available from Basu & Manca (2012). Standard errors in parenthesis, * p<0.1. ** p<0.05. *** p<0.01. RMSE: root mean squared error. 30

31 Table 6: Incremental Effects a OLS Beta MLE Beta QMLE 2SE Age (0.001) (0.001) (0.001) (0.001) Female (0.022) (0.024) (0.022) (0.021) Levels Fused (0.002) (0.002) (0.002) (0.002) Posterior Approach (0.019) (0.021) (0.019) (0.018) Baseline HRQoL PF (0.002) (0.002) (0.001) (0.002) RL (0.001) SF (0.000) (0.000) (0.000) (0.002) P (0.004) (0.004) (0.003) (0.003) MH (0.002) V (0.001) (0.001) (0.001) (0.001) a Incremental effects on QALYs estimated via the method of recycled predictions following OLS, Beta MLE, Beta QMLE, and 2SE (Oaxaca, 1973; Graubard & Korn, 1999; Basu & Rathouz, 2005; Basu, 2005; Glick, 2007; Kleinman & Norton, 2009). Beta MLE and QMLE estimation follows the procedure and code available from Basu & Manca (2012). Bootstrapped standard errors in parenthesis based on 1,000 iterations. 31