WHO S ASSESSMENT OF HEALTH CARE INDUSTRY PERFORMANCE: RATING THE RANKINGS

Transcription

1 WHO S ASSESSMENT OF HEALTH CARE INDUSTRY PERFORMANCE: RATING THE RANKINGS ELLIOTT PARKER and JEANNE WENDEL * Department of Economcs, Unversty of Nevada, Reno, NV, USA SUMMARY Ths paper examnes the econometrc methodology used by World Health Organzaton (WHO) researchers to develop a template for rankng health system effcency n 191 countres, as reported n the World Health Report 2000 (WHR). Complementng recent crtques on the qualty of the WHR data, we ask whether the methodology would be approprate for accomplshng the WHR goal, even f the data problems were resolved. We dentfy three econometrc problems and then re-estmate the health producton functon and the resultng effcency ratos, usng avalable WHO data. We conclude that the WHR strategy of rankng countres accordng to econometrc effcency ratos wll not produce polcy-relevant results untl health nputs, outputs, and producton relatonshps are clarfed. KEY WORDS World Health Organzaton; WHO; World Health Report 2000; producton effcency; comparatve health care system effcency * Correspondence to: Jeanne Wendel, Department of Economcs /030, Unversty of Nevada, Reno, NV , USA. Tel.: ; e-mal: wendel@unr.edu

2 WHO S ASSESSMENT OF HEALTH CARE INDUSTRY PERFORMANCE: RATING THE RANKINGS SUMMARY Ths paper examnes the econometrc methodology used by World Health Organzaton (WHO) researchers to develop a template for rankng health system effcency n 191 countres, as reported n the World Health Report 2000 (WHR). Complementng recent crtques on the qualty of the WHR data, we ask whether the methodology would be approprate for accomplshng the WHR goal, even f the data problems were resolved. We dentfy three econometrc problems and then re-estmate the health producton functon and the resultng effcency ratos, usng avalable WHO data. We conclude that the WHR strategy of rankng countres accordng to econometrc effcency ratos wll not produce polcy-relevant results untl health nputs, outputs, and producton relatonshps are clarfed. INTRODUCTION The World Health Report (WHR) 2000 [1], enttled "Health systems: Improvng performance", presents the results of an ambtous body of work undertaken n pursut of a laudable goal. In an effort to stmulate acton that wll eventually mprove the overall performance of health systems [2, p 17], researchers at the World Health Organzaton defne health nputs and outputs, buld a fve-year data set for 141 of ther 191 member countres, estmate a health producton functon, and use these estmates to calculate rankngs of overall effcency for each country. Antcpatng the potental polcy mplcatons of such rankngs, we ask two questons: How relable and useful are these rankngs? Does ths report provde a template for analyzng nternatonal health system effcency? In ths paper, we focus on the econometrc methodology used to create the effcency rankngs. Blendon, et al. [3] and Wllams [4] have crtqued the WHR data set. Wllams focused on the large proporton of the data set that was estmated rather than observed, whle 1

3 Blendon, et al., focused on the WHR relance on expert opnons to proxy publc vews, wthout nput from publc opnon surveys. We ask a dfferent queston: f the data qualty ssues were resolved, would the WHR econometrc approach provde a sold framework for understandng, measurng, and analyzng health system effcency? We dentfy three problems n the specfcaton of the WHR producton functon, and we use ther data to re-estmate health care effcency and assess the practcal relevance of the methodologcal ssues. Because our results rase several troublng ssues, we conclude that these nternatonal effcency comparsons should not be used to gude management or polcy decsons. The next secton of ths paper presents an overvew of the WHR methodology. The subsequent secton dentfes two ntal problems, aggregaton and functonal form, and reestmates the WHR effcency rankngs, usng both a producton functon approach and a cost functon approach. The subsequent secton focuses on the more fundamental, problem posed by mssng varables that are nadequately proxed by dummy varables. Fnally, the concludng secton summarzes our fndngs. THE WHR METHODOLOGY: AN OVERVIEW The WHR analyss focuses on a composte measure of health care system performance, n whch the most fundamental component of ths measure s Dsablty-Adjusted Lfe Expectancy (DALE). Addtonal components shown n Table 1 nclude a measure of the responsveness of the health care system to demand for non-clncal qualty characterstcs (RESP), along wth three measures of relatve equalty: the dstrbuton of DALE (DIST), the dstrbuton of RESP 2

4 (RDIS), and the farness of fnancng (FFIN). These fve measures are normalzed nto the (0,1) nterval and combned nto a lnear composte measure wth the weghts ndcated n the table. Ths composte measure of health care system output (Q) s modeled as the outcome of a translog producton functon of two explanatory varables, health care expendtures and educatonal attanment. The frst explanatory varable s the prmary varable of nterest, whle the second s ncluded to capture the effects of non-controllable non-health-system nputs nto the producton of health. The fxed effects model also ncludes 190 dummy varables to capture nter-country economc, demographc, poltcal, publc health, and cultural dfferences. A second translog producton functon, wth dentcal ndependent varables, s estmated for DALE alone. The WHR authors estmate the parameters of ths producton functon, and use the results to calculate an overall effcency rato for each country. The WHR data set ncludes fve years of data for 141 countres, and one year of data for the remanng 50 countres. The WHR authors modfy the tradtonal concept of techncal effcency, to develop the overall effcency scores. Techncal effcency, also called multfactor productvty or total factor productvty n the lterature, s usually defned as the rato of weghted outputs to weghted nputs, normalzed to the (0,1) nterval. In a normal producton process, however, zero nputs would lead to zero outputs. Notng that ths would not apply to the provson of health care (because lfe expectancy would not fall to zero n the absence of health care and educaton), the authors of the WHR defne a measure of overall producton effcency relatve to the mnmum lfe expectancy that would occur wth zero health care expendtures: ( Q Q WTE ) ( =, Q Q 3

5 where Q = average observed DALE for the th country, Q ( = mnmum DALE expected wth zero health care expendtures, and Q ) = maxmum potental DALE consstent wth actual health care expendtures. Thus, overall effcency measures the extent to whch a country succeeds n realzng ts potental to ncrease health above the level expected to preval n the absence of any health care, gven ts level of health care expendtures. Ths mnmum level of the health composte measure s developed from a combnaton of estmaton and assumpton. The mnmum level of DALE s estmated from 1908 data on 25 countres. Ths estmaton s based on two key assumptons: that no country enjoyed the benefts of a functonng health system n 1908, and that 1908 DALE can be modeled as a functon of lteracy. The 1908 relaton between DALE and lteracy s used to estmate the hypothetcal level of DALE that would have been observed n each country n n the absence of any health system. The zero-nput values of the remanng four performance measures are assumed to be equal to zero (for DIST and RESP) or one (for RDIS and FFIN). Producton effcency measures estmated by WHR, for both the producton of DALE alone (ths WHR DALE effcency estmate s denoted WDE) and the producton of the composte output (WCE denotes ths WHR composte effcency estmate), are provded n the WHR annex tables for all 191 WHO countres. The rank correlaton coeffcent between the two effcency measures s 0.87, reflectng the fact that the composte measure s hghly correlated wth DALE. Results of the World Health Organzaton s Global Programme on Evdence for Health Polcy are reported n the WHR [1], wth more detaled nformaton provded n 30 workng papers posted on the WHO webste. 4

6 RE-ESTIMATING THE RANKINGS: ARE THEY ROBUST? Ths secton focuses on two of the econometrc problems, napproprate aggregaton of heterogeneous performance measures pror to estmaton and the logcally nconsstent functonal form. We examne the practcal relevance of these ssues by frst re-specfyng and re-estmatng the producton functon, and then estmatng a cost functon that provdes a more drect approach for addressng the two ssues. The re-estmated producton functon yelds effcency measures and rankngs that are postvely correlated wth the WHR measures and rankngs. The cost functon, however, yelds effcency rankngs that are uncorrelated wth the WHR rankngs. Frst Specfcaton Issue: Inapproprate Aggregaton of the Composte Output The frst problem embedded n the WHR specfcaton s napproprate aggregaton of heterogeneous performance measures pror to estmaton of the producton functon. The WHR composte health system performance measure consttutes a vald output measure only f the rght-hand-sde varables exert equal mpacts on each of the components of the composte measure. For a producton process n whch goods are weghted by margnal values and producers equate the margnal products of nputs across products, t s usually vald to specfy aggregate producton as a functon of common nputs. The fve health system outputs are unprced, however, and three of them measure relatve equalty rather than output quanttes. The WHR study offers nether theoretcal nor emprcal evdence to support the mplct assumpton that these measures meet the equal-mpact requrement for aggregaton pror to estmaton. We employ two strateges to assess the practcal relevance of ths ssue. Frst, we estmate separate producton functons for each of the fve outputs to permt dfferental mpacts of the explanatory varables on the performance measure components, and test for equalty of the 5

7 coeffcents n the fve equatons. Ths strategy facltates comparson wth the WHR results, but provdes only a partal soluton to the aggregaton problem, because subsequent calculaton of fve effcency measures employs the WHR weghts to compute a weghted composte effcency measure. We subsequently address ths second problem by estmatng a cost functon that models health expendtures as a functon of nput prces and the fve outputs. Second Specfcaton Issue: Logcally Inconsstent Functonal Form The WHR translog functonal form s logcally nconsstent wth the explct assumpton that lfe expectancy would reman postve f healthcare expendtures fell to zero. In contrast to ths assumpton, the functon Q=f(X) s transformed by the translog form to lnq=g(lnx), whch mples the condton f(0)=0 (or, n logs, g(- ) - ). Thus, the translog producton functon does not defne Q when one of the nputs (X) s zero, even though the authors of WHR argue Q would be postve when health expendtures are zero. Ths nconsstency forces the WHR authors to estmate the mnmum from out-of-sample data, nstead of usng the regresson results to estmate the mnmum. We address ths problem by specfyng a non-homothetc Generalzed Leontef functon of the form: q 1 α k 0 + α km X m, + β kmn X m, X n, + ε k 2 = k,, m m n, where ndexes countres 1-191, k ndexes the fve health system outputs (q) ncluded n the composte measure (DALE, DIST, RESP, RDIS, and FFIN ) and m and n ndex the nputs X (whch are assumed by WHR to nclude only health expendtures H and educatonal attanment E). Ths functonal form fts the dmnshng non-homothetc property apparent n the data, both 6

8 vsually and from prelmnary Box-Cox estmates, and t permts output to be postve when nput quanttes are zero, thus avodng the logcal nconsstency of employng the translog producton functon n ths case. Because ths functonal form can assume non-zero values when H equals zero, ths also permts drect estmaton of the mnmum health output assocated wth zero health system nputs. Re-estmaton of the Producton Functon To assess the practcal relevance of these specfcaton ssues, we re-estmate the producton functon usng: (1) fve separate equatons nstead of one composte equaton, (2) a nonhomothetc Generalzed Leontef producton functon, (3) the one year of data publcly avalable from WHR 1 (see Table 2), and (4) no dummy varables (whch mples that the country characterstcs wll be captured n the error term ε). We nclude only the two ndependent varables (H and E) ncluded n the WHR producton functon, n both the full second-order verson and a frst-order verson that facltates nterpretaton of the margnal effects. Estmaton results for these two versons of the model are reported n Table 3. We do not present ths model as a template for ratng health system effcency; rather, we use the estmaton results to assess the econometrc ssues dentfed n the prevous secton. The re-specfed producton functons yeld reasonable results: the coeffcents of the explanatory varables are postve and statstcally sgnfcant n most cases. Whle ndvdual coeffcents are not all statstcally sgnfcant, the jont hypothess that ether of the two varables can be dropped s rejected. Confrmng our hypothess that the composte output was napproprately aggregated pror to estmaton, these results reveal sgnfcant dfferences n the mpacts of health expendture and 7

9 educaton on the fve separate output components. The jont hypothess that relatve values of both the coeffcents for H and the coeffcents for E are equal to the weghts gven n Table 1 s rejected wth an F-statstc equal to 45, whch exceeds the crtcal value of 2.42 for a two-taled 5 percent test. Thus, the fve performance measures should not be aggregated nto a composte measure pror to estmaton, at least not wth the weghts gven n WHR. Calculatng Producton Effcency Rankngs How do these econometrc ssues affect the effcency ratos and country rankngs? We calculate overall effcency for each country for our producton functon and cost functon results, usng the WHR effcency defnton. Calculatng the producton functon effcency ratos requres computaton of maxmum and mnmum values, followed by computaton of overall effcency. Usng the frst set of reported producton functon estmatons, wth both frst and second-order effects but excludng addtonal varables, we calculate the maxmum attanable output for each performance measure, gven current levels of X = [H, E] and no country-specfc dfferences: 1 α ˆ ˆ ˆ k 0 + α km X m, + β kmn X m, X n, + 2 max ( ˆ ε ). qˆ k, k, m m n = Mnmum output for each component s calculated as output gven zero health care expendture,.e. X = [0, E], whch smplfes to: ( q 1 = α ˆ ˆ ˆ mn ( ˆ k 0 + α ke E + β kee E + ε k, ). 2 k, These quanttes permt computaton of an overall effcency rato for each performance measure. The numerator s the dfference between expected output and the mnmum output 8

10 (expected wth zero health system nputs); the denomnator s the dfference between effcent output and the mnmum output: q ( q k, k, PE k, = ) ( qk, qk,. Usng the WHR weghts, the resultng effcency measures are used to calculate the composte effcency ndex: CPE = 0.25 PE1, PE2, PE3, PE4, PE5,. From ths ndex we derve country rankngs; correlatons between our calculated effcency rankngs and those gven n WHR are reported n Table 4. The correlaton coeffcent for the rankngs produced by the WCE (the WHR measure of composte producton effcency) and the RCE (our revsed measure of composte producton effcency) s Whle t s sgnfcantly greater than zero, the hypothess that the correlaton between WCE and RCE equals one s rejected 2. On average, countres move 36 places n rank between the two effcency measures. To examne the sources of the dfferences between the two sets of effcency ratos, we calculate an alternatve measure, RCE2 by (1) usng our Generalzed Leontef functonal form, (2) estmatng only one producton functon for the composte measure Q, and (3) calculatng the ( mnmum Q usng the WHR-assumed mnmum values for the non-dale outputs (0 for DIST and RESP, 1 for RDIS and FFIN). 3 The correlaton between the RCE and RCE2 rankngs s.96, as reported n Table 4. The hypothess that the correlaton between RCE and RCE2 equals one cannot be rejected; hence the dfference between the WCE and RCE rankngs does not stem from dfferences n the method used to estmate mnmum health. Instead, t reflects changes to ether the data set or the functonal form. Whle the correlaton between RCE2 and WCE s slghtly 9

11 hgher than the correlaton between RCE and WCE, the dfference s not statstcally sgnfcant. Table 4 also ncludes the correlaton coeffcents between the effcency rankngs for the DALE-only measures WDE (gven n WHR) and our revsed DALE-only measure RDE, as derved from the estmaton of our frst equaton, along wth those between these rankngs and ther assocated composte measure. The correlaton between the WHR results and our results s approxmately 0.7, whether output s measured as the WHR composte or DALE-only. Estmatng a Cost Functon and Calculatng Cost Effcency Rankngs The cost functon provdes a more drect approach for addressng these econometrc ssues. It provdes a more compact sngle-equaton representaton of the relatonshp between the prmary nput H and the fve health system outputs, t avods the necessty of calculatng a composte effcency score, t permts postve health output n the presence of zero health nputs, and t provdes effcency scores that do not rely on calculaton of mnmum outputs. The cost functon approach s useful here because t s approprate for multple outputs, can be estmated n the presence of fxed nputs, and wll permt drect estmaton of cost effcency as the rato of the mnmum potental cost to the observed cost of producng observed output gven observed nput prces. Because drect estmaton of cost effcency elmnates the need to calculate mnma, there s no need to use the non-homothetc Generalzed Leontef form; the translog form orgnally used n WHR wll suffce. Ths functon s of the general form: 1 ln C = γ + γ m m Z m, + m θ n mn Z m, Z n, + v 2 0, 10

12 where Z (subscrpted by m and n) s a vector of outputs, fxed nputs, and varable nput prces, usually expressed n logarthms. Cost, measured as health care expendtures H, s specfed as a functon of the fve performance measures (DALE, DIST, RESP, RDIS, and FFIN). We assume one fxed nput, (educaton E), and two varable nputs (captal and labor). We assume that the real cost of captal vares over tme but does not dffer sgnfcantly across countres, and cannot be dstngushed from the constant because we are usng cross-secton data. To proxy the relatve wage of labor nputs, we nclude Gross Domestc Product per Capta (G), whch s adjusted by the authors of WHR to reflect purchasng power party. Recognzng the potentally dramatc mpact of ncludng G n the cost functon, we also estmate ths basc cost functon wthout G to facltate comparson wth the WHR results. Because a cost functon s lnearly homogeneous n nput prces, ths specfcaton mples that the elastcty of H wth respect to G must le n the [0,1] nterval. Further, the specfcaton of E as a fxed nput mples that the elastcty of H wth respect to E s negatve, and the elastcty of H wth respect to each of the fve outputs should be postve. Because theory provdes no a pror assumptons about returns to scale for the fve performance measures, we test whether these should enter lnearly or logarthmcally; the results ndcate we should accept the former and reject the latter. Thus, we specfy the elements of Z as [lne, lng, DALE, DIST, RESP, RDIS, and FFIN]. Wth seven elements n Z, we have 36 parameters to estmate and report. Estmatng the full equaton usng OLS, many of the 28 second-order (θ) terms are nsgnfcant, but we reject the hypothess that these θ terms are all jontly equal to zero. Retestng ths jont hypothess for the θ parameters wth ntal t-statstcs less than one, ndcates that we cannot reject ths hypothess 11

13 at any reasonable level of confdence. The basc model therefore ncludes all terms wth ntal t- statstcs greater than one, for whch estmaton results are presented n Table 5. Calculatng elastctes from the estmated coeffcents, we fnd that the elastcty of H wth respect to G s postve and sgnfcant (.e., the jont hypothess that ts frst and second-order coeffcents all equal zero can be rejected). However, the elastcty of H wth respect to E has a postve rather than a negatve mean (though ts range ncludes negatve observatons) and t s sgnfcantly dfferent from zero. The cost elastctes for both DALE and DIST have a postve mean (wth some negatve values n ther ranges), and they are sgnfcant at 5 percent and 10 percent respectvely. The cost elastcty for RDIS has a negatve mean that s sgnfcantly dfferent from zero, whle the cost elastctes for both RESP and FFIN are not sgnfcantly dfferent from zero. These results reconfrm the hypothess that the fve performance measures should not be aggregated nto a composte measure pror to estmaton of the producton functon. To calculate cost effcency rankngs, we frst calculate the mnmum attanable health care expendture gven observed values of Z as: 1 ln C ˆ ˆ ˆ = γ + γ m m Z m, + m θ n mn Z m, Z n, + 2 mn ( vˆ ), 0 and the cost effcency ndex then smply equals: CE ( = exp ( ln H ln H ). The correlaton coeffcent between the resultng cost effcency rankngs and the orgnal WHR producton effcency rankngs equals 0.04, and not sgnfcantly dfferent from zero. The cost effcency rankngs are also uncorrelated wth the rankngs from our producton functon estmatons. 12

14 To check whether ths lack of correlaton s an artfact of our specfcaton, we estmate seven addtonal versons of the cost functon. We consder the effects of excludng G, excludng all second-order coeffcents, and/or excludng three performance varables (DIST, RDIS, and FFIN) that measure dstrbutonal equty rather than output quantty and qualty, and we consder all possble combnatons of these exclusons. The correlaton coeffcents between these varous cost functon effcency rankngs and the WHR rankngs range from 0.10 to The average absolute change n rank for all estmated versons of the cost functon ranges from 64 to 70 places. The correlaton s not sgnfcantly dfferent from zero n any of these cases. The nsgnfcant correlatons between the cost functon effcency scores and the WHR effcency scores, along wth the substantal average changes n rank, are partcularly troublng because the cost functon provdes a more drect approach, compared wth the producton functon, for modelng the relaton between health expendtures and multple health system outputs. Detaled examnaton of the cost functon results (not reported here, but avalable upon request) also hghlghts the relevance of the excluded soco-economc varables. The cost elastcty of G s always postve and sgnfcant, but the cost elastcty of E s only negatve (as hypotheszed) when G s excluded. When G s excluded, the cost elastctes of DALE, DIST, and RESP are postve and sgnfcant. When G s ncluded, however, only the elastcty for RESP s consstently postve and sgnfcant. THE MISSING VARIABLES PROBLEM The cost functon results hghlght the sgnfcance of the WHR decson to exclude from the producton functon the economc, demographc, poltcal, publc health, and cultural condtons that are wdely expected to exert substantal mpacts on the producton of health. The WHR 13

15 researchers argue that the producton functon should not nclude such varables for three reasons [1]. Frst, the 190 country dummy varables ncluded n the fxed effects model account for these mpacts. Second, these addtonal varables represent controllable varables, whch should not be ncluded n a producton functon. Thrd, a Hausman test ndcates that ncluson of GDP per capta would not ncrease the explanatory power of the model. Ths modelng strategy does not, however, yeld results that can be meanngfully appled to the stated goals of the WHR Study. An OLS regresson of the WHR composte effcency ratos on three varables (GDP per capta (G), fertlty rates (F), and reported HIV/AIDs nfecton rates (I)) yelds an R-square of 0.66 (see Table 6), wth each explanatory varable sgnfcant at the 5 percent level. Ths type of effcency measurement system, whch ndcts countres wth low ncomes, hgh brthrates and hgh reported HIV nfecton rates, suggests that country leaders can utlze health system resources more effcently by reducng brthrates, preventng dsease and stmulatng economc growth. Ths mplcaton s not very helpful, and not consstent wth the stated WHR goal of stmulatng actons that wll eventually mprove the overall performance of health systems [2, p 17]. Measurng progress toward ths goal requres a measurement system that focuses on how well each country s health system utlzes ts resources, gven the level of economc development, fertlty rates, and HIV nfecton rates. The WHR decson to exclude non-controllable varables (other than educaton) s not consstent wth the WHR defnton of the health system boundary. WHR 2000 researchers defne the boundary of the health system by focusng on the concept of a health acton [7, p 4]. The prmary ntent of a health acton s to mprove or mantan health. WHR 2000 researchers explan ths concept va an example: measures taken to ncrease educaton of young grls are not 14

16 health system nputs, even though ncreased female educaton may mprove health outcomes because health mprovements are not the prmary ntent. Accordng to ths defnton, expendtures desgned to ncrease development, for example, are not health system nputs; hence the measurement of health system effcency should be ndependent of a country s level of development. Do the Dummy Varables Substtute for the Relevant Socoeconomc Varables? The WHR approach of estmatng a fxed effects model, nstead of ncludng relevant socoeconomc varables n the producton functon, s problematc for two types of reasons. Frst, the data set s not suffcent for estmatng the fxed effects model and, second, ths approach does not mathematcally separate country characterstcs from country neffcency. The WHR data set s not suffcent to estmate a fxed effects model because (a) the fve-year observaton wndow s too short to observe meanngful ntra-country varaton n varables such as lfe expectancy, educatonal attanment, and GDP per capta, (b) only one year of data s avalable for approxmately one-fourth of the 191 countres [2, p 7], and (c) a substantal porton of the fve-year data set s estmated rather than observed. The estmaton of much of the data set s partcularly problematc n ths stuaton, because some of the potentally relevant (but excluded) varables are employed as ndependent varables n the estmatng equatons. RESP and RDIS, for example, are estmated from nformaton developed n a one-tme survey of knowledgeable ndvduals n 35 countres (See Valentne [10] and Wllams [4]). Ths nformaton s used to estmate relatonshps between the components of responsveness and ndependent explanatory varables such as geographc access, percent of populaton below the poverty level, GDP per capta, and educaton. These 15

17 relatonshps are used to estmate fve years of data for RESP and RDIS for all 191 countres. Thus, some dependent varables are constructed as lnear combnatons of potentally relevant (but ntentonally excluded) explanatory varables. Wllams [4] dscusses the data estmaton procedures n more detal and draws the concluson that the data s nsuffcent to support the WHR research program. We pont here to the consequent modelng problem: f the mult-year data set does not nclude ndependent observatons for each year, country dummy varables wll not account for country characterstcs n a meanngful way. Whle fxed effects models are approprate for estmatng producton functons n some stuatons, ths approach s problematc n ths stuaton because (a) the mssng varables are expected to exert substantal mpacts on the producton of health and (b) the stated goal s to develop effcency measures that provde ncentves for effcent resource use wthn the health care system. Inter-country dfferences n health-care requrements are not dstngushed n a fxed effects model, however, from nter-country dfferences n neffcency. The dummy varables smply measure the average resdual for each country; they do not prevent relevant country characterstcs factors from affectng the estmaton of effcency. Incluson of both a dummy varable and an error term n the fxed effects model separates country characterstcs nto two terms, but t s not clear that t wll do ths by solatng country-specfc neffcency n the error term. Includng soco-economc varables n the producton functon gves explct consderaton to country characterstcs that mpact the producton of health. Should Socoeconomc Varables be Excluded Because they are Controllable? The WHR Report argues that soco-economc varables should not be ncluded n the producton functon because they are not controllable. Instead, the WHR producton functon 16

18 ncludes educaton as a controllable varable. Ths varable, accordng to WHR, measures educaton as a fxed nput nto the process of producng health, and serves as a proxy for nonhealth systems nputs [8, p. 70]. Ths logc suffers from three problems. Frst, the concept of controllable and uncontrollable nputs s problematc n the context of an nternatonal health care producton functon because t begs the questons, controllable by whom? and controllable to what extent? Second, educaton s lkely to proxy development. Snce development s frustratngly non-controllable, the WHR logc would exclude t from the producton functon. Thrd, excludng non-controllable nputs from the producton functon yelds effcency measures that are strongly nfluenced by varables that are outsde the sphere of nfluence of each country s health system, as defned by WHR [8, p 4]. These measures are not, therefore, useful for assessng the effcency wth whch each health system utlzes ts resources. Does the WHR Hausman Test Justfy Excluson of Per-Capta GDP? After argung that per-capta GDP (G) s not a conceptually approprate explanatory varable, the WHR study authors conducted a statstcal test to assess whether G should be ncluded n the producton functon. They regressed G on the ndependent varables, [8, p. 12] and then ncluded the resdual from ths regresson n the producton functon. They found that the resdual coeffcent s statstcally nsgnfcant; hence nether G nor the resdual are ncluded n the WHR model. The Effects of Includng Socoeconomc Varables n the Producton Functon We assess two ssues. Frst, we repeat the Hausman test wth a producton functon that does not nclude the dummy varables. As reported below, the estmated coeffcent on G s 17

19 sgnfcant n ths case. Second, we examne the mpact of ncludng three soco-economc varables on the producton functon and effcency estmaton results. To assess the practcal relevance of the decson to exclude socoeconomc varables, we reestmate the frst-order verson of the fve producton functons gven earler, addng G, F, and I to each equaton. These results, whch are reported n Table 7, confrm that ncludng per-capta GDP (G), fertlty (F), and reported HIV/AIDS nfecton rates (I ) exerts sgnfcant mpacts on the estmaton results. When these varables are ncluded n the producton functon, the coeffcent on health care expendtures becomes nsgnfcant n three of the equatons (DALE, DIST and FFIN) and sgnfcantly negatve n another (RDIS). The coeffcent for H s only postve n the RESP equaton, where t s sgnfcant at 10 percent. Because the effect of H s no longer sgnfcantly postve, calculaton of the effcency scores s problematc. How can we measure the effcency of health care expendtures f they do not mprove health performance? We calculate t nonetheless, to assess the magntude of the mpact, and fnd that ncluson of the mssng varables G, F, and I n the producton functon alters the effcency rankng dramatcally. The correlaton coeffcent (ρ WCE/RCE3 ) between ths new revsed composte effcency rankng (RCE3) and the orgnal WHR rankngs (WCE) drops to Whle ths correlaton s sgnfcantly greater than zero, addng three socoeconomc varables to the producton functon reduced the correlaton between our re-estmated effcency scores and the WHR scores by approxmately 50 percent. Ths correlaton coeffcent reducton s statstcally sgnfcant: the confdence ntervals for the two correlaton coeffcents (ρ WCE/RCE3 and ρ WCE/RCE ) do not overlap. The average change n rank between the orgnal WHR rankngs and these new rankngs s 51 places out of 191 countres. 18

20 CONCLUSION We conclude that the effcency rankng effort, whle laudable, s premature. The results are not robust wth respect to two fundamental decsons: to focus on a producton functon, rather than a cost functon, and to exclude all socoeconomc explanatory varables except educaton. The strong statstcal relatonshp among the WHR effcency measures, HIV/AIDs nfecton rates, fertlty rates, and G undermnes the WHR vson that the effcency scores can be used to measure and nduce ncreased productvty of health care expendtures. An nternatonal ad allocaton strategy desgned to reward countres wth effcent health care systems, as measured by the WHR methodology, would channel ad to countres wth low reported HIV/AIDs nfecton rates, low fertlty rates, and hgh GDP per capta; clearly, such a system would not provde ncentves for more effcent use of health system resources. The WHR s strategy of ncludng only controllable varables n the producton functon s wdely used, and s approprate for studes desgned to analyze exogenous factors that affect producton effcency. Ths two-step approach (frst estmate effcency wthout consderaton of the exogenous factors, then analyze the mpacts of the exogenous factors on the effcency measures) does not provde the nformaton needed to meet the WHR goals, however. Meetng these goals wll requre effcency measures that answer the queston: how well does each country s health care system utlze ts resources, gven ts socoeconomc characterstcs. The results of our paper pont to the fundamental problem that plagues every effort to analyze health producton: whle many products, such as steel, are largely produced wthn reasonably well-defned ndustres, health s produced both nsde and outsde the health care ndustry. The relatonshp between non-health system nputs and health outputs has been well 19

21 documented, but t s not well understood. Ths knowledge gap may lead researchers to omt such varables from effcency analyss, but falure to nclude relevant nputs n the health producton functon s lkely to dstort effcency results. The complexty of the ssues at hand are llustrated by the fact that GDP per capta may be postvely correlated wth health system output measures for three dstnct reasons. Frst, hgh GDP per capta facltates provson of publc health measures (such as clean water, mproved nutrton, santaton and vaccnatons) and personal consumpton patterns (such as adequate det) that are nputs nto the publc and household producton of health. Second, GDP per capta proxes labor costs. Falure to nclude a measure of nter-country wage dfferences wll bas the health expendture coeffcent f countres wth hgh wage rates experence relatvely hgh health care expendtures that prmarly reflect hgh nput prces, rather than hgh nput quanttes. Thrd, patents n hgh-gdp countres may have above-average wllngness to pay for types of health care that do not extend lfe expectancy. As ncome rses, health care delvery may broaden ts focus beyond prolongng lfe to nclude more qualty of lfe measures that allevate pan, ncrease jont moblty, or address reproductve problems. Such health care expendtures would appear, n the WHR template, as neffcent resource utlzaton. Thus, producng effcency measures that wll accomplsh the WHR goal wll requre better clarfcaton of health system nputs, the relatonshp between nputs and outputs, and actual health system outputs. 1 The complete data set, wth fve years of data for 141 of 191 countres, was requested from Dr. Murray and others n the WHR research project, but t was not provded to us. Dr. Evans gracously provded a porton of Evans, et al. [5] whch had been cleared by WHO. 2 We focus our dscusson on the correlatons between sets of country rankngs because the of polcy relevance of these rankngs. We test whether these rankng correlatons are sgnfcantly greater than zero. However, all other hypothess tests are conducted for correlatons among the sets of effcency ratos. 20

22 3 The logc of these assumptons s murky because the last three performance measures are conceptually meanngless n the absence of a functonng health care system (see Murray, et al. [7], p 8). 21

23 REFERENCES 1. World Health Organzaton. The World Health Report 2000: Health Systems: Improvng Performance. Geneva, World Health Organzaton, Tandon A, Murray C, Lauer J, Evans D. Measurng Overall Health System Performance for 191 Countres, Geneva, World Health Organzaton, Global Programme on Evdence for Health Polcy, Dscusson Paper Seres, No. 30, Blendon RJ, Km M, Benson JM, The publc versus the World Health Organzaton on health system performance. Health Affars, May/June Wllams A, Scence or marketng at WHO? A Commentary on World Health Health Economcs 2001; 10: Evans DB, Bendb LM, Tandon A, Lauer J, Ebener S, Hutubessy RCW, Asada Y, and Murray CJL. Estmates of ncome per capta, lteracy, educatonal attanment, absolute poverty, and ncome Gn coeffcents for the World Health Report Geneva: World Health Organzaton, Global Programme on Evdence for Health Polcy, Dscusson Paper No. 7, World Health Organzaton. Global stuaton of the HIV/AIDS pandemc, end Geneva: World Health Organzaton, emc/ dseases/ hv/ Documents/ Report1999.pdf. 7. Murray CJL and Frenk J. A WHO Framework for Health System Performance Assessment. Geneva: World Health Organzaton, Global Programme on Evdence for Health Polcy, Dscusson Paper No. 6, Evans DB, Tandon A, Murray CJL, and Lauer JA. The Comparatve Effcency of Natonal Health Systems n Producng Health: An Analyss of 191 Countres. Geneva: World Health Organzaton, Global Programme on Evdence for Health Polcy, Dscusson Paper No. 29, Valentne NB, de Slva A, Murray CJL. Estmatng Responsveness Level and Dstrbuton for 191 Countres: Methods and Results, Geneva: World Health Organzaton, GPE Dscusson Paper Seres No. 22, 2000.

24 Table 1. Components of the WHR 2000 composte output for health system performance Composte Measure Weght DALE Dsablty-Adjusted Lfe Expectancy 25.0% DIST Dstrbuton of DALE 25.0% RESP Responsveness 12.5% RDIS Dstrbuton of Responsveness 12.5% FFIN Farness n Fnancng 25.0%

25 Table 2. Data set Varable Year Source Performance Measures: DALE and DIST 1999 Annex Table 5 of WHR 2000 RESP and RDIS 1999 Annex Table 6 of WHR 2000 FFIN 1997 Annex Table 7 of WHR 2000 WHR Effcency Measures: WCE Composte producton effcency Annex Table 10 of WHR 2000 WDE Producton effcency for DALE Annex Table 10 of WHR 2000 Exogenous Varables: G GDP per capta, n thousands of Annex 1 of Evans, et al. [5] purchasng-power-party US Dollars E Educatonal attanment 1997 Annex 1 of Evans, et al. [5] H Health care expendtures per capta, 1997 Annex Table 8 of WHR 2000 calculated as the product of G and the health expendture share, ncludng total publc and prvate health care expendtures A Share of populaton aged 65 or older 1999 Annex Table 2 of WHR 2000 F Fertlty rate 1999 Annex Table 2 of WHR 2000 I Percentage of populaton wth reported HIV/AIDS nfecton 1999 Table 2 of WHO 2000 [6]

26 Table 3. Producton functon re-estmatons Independent Dependent Varables Varables DALE DIST RESP RDIS FFIN frst and second order: Constant (0.094) ** (0.087) ** (0.038) ** (0.063) ** (0.074) ** H (0.186) ** (0.172) ** (0.075) (0.124) ** (0.145) E (0.105) (0.097) (0.042) ** (0.070) ** (0.082) H/ (0.130) (0.120) ** (0.052) (0.086) * (0.101) H E (0.087) ** (0.080) ** (0.035) (0.058) ** (0.068) E/ (0.056) (0.052) ** (0.023) ** (0.037) (0.044) R frst-order only: Constant (0.031) ** (0.029) ** (0.012) ** (0.021) ** (0.022) ** H (0.023) ** (0.022) ** (0.009) ** (0.016) ** (0.016) ** E (0.015) ** (0.014) ** (0.006) ** (0.010) ** (0.011) R Note: Standard errors n parentheses *, ** Dfference from zero statstcally sgnfcant at 10% and 5% levels (two-taled), respectvely

27 Table 4. Correlaton coeffcents among producton functon effcency rankngs WHR 2000 composte effcency ndex (WCE) Re-estmated composte effcency ndex (RCE) WHR 2000 DALE-only effcency ndex (WDE) Re-estmated DALE-only effcency ndex (RDE) Re-estmated composte effcency ndex wth G,F,I (RCE3) Measure WCE 1.00 RCE RCE WDE RDE RCE

28 Table 5. Cost functon estmaton Estmated Standard Estmated Standard Parameter Coeffcent Error Parameter Coeffcent Error γ0 (CONSTANT) (3.124) θ G1 (lng DALE) (0.371) ** γ E (lne) (0.627) ** θ E1 (lne DALE) (0.727) γ1 (DALE) (4.173) ** θ E4 (lne RDIS) (0.762) ** γ 2 (DIST) (5.504) θ 11 (DALE 2 ) (4.750) * γ3 (RESP) (7.171) θ 12 (DALE DIST) (2.089) * γ4 (RDIS) (1.093) ** θ 15 (DALE FFIN) (4.543) γ 5 (FFIN) (3.202) θ 25 (DIST FFIN) (6.178) * γg (lng) (0.250) ** θ 35 (RESP FFIN) (7.834) θge (lng lne) (0.128) ** R Note: Standard errors n parentheses *, ** Dfference from zero statstcally sgnfcant at 10% and 5% levels (two-taled), respectvely

29 Table 6. Determnants of WHR 2000 composte effcency Dependent Varable: WCE OLS regresson Independent Standard Varable Coeffcent Error Constant (0.031) ** G (0.002) ** F (0.007) ** I (0.072) ** R Note: Standard errors n parentheses ** Dfference from zero statstcally sgnfcant at 5% level (two-taled)

30 Table 7. Producton functon wth addtonal varables Independent Dependent Varables Varables DALE DIST RESP RDIS FFIN WHR varables + addtonal varables, frst-order only: Constant (0.063)** (0.070)** (0.029)** (0.052)** (0.059)** H (0.050) (0.055) (0.023)* (0.041)** (0.047) E (0.015)** (0.016)** (0.007)** (0.012)** (0.014)* G (0.015)** (0.016)** (0.007)** (0.012)** (0.014) F (0.019)** (0.021)** (0.009) (0.015)** (0.017)** I (0.028)** (0.031)** (0.013) (0.023) (0.026) R Note: Standard errors n parentheses *, ** Dfference from zero statstcally sgnfcant at 10% and 5% levels (two-taled), respectvely