Can Subjective Questions on Economic Welfare Be Trusted?

Transcription

1 Publc Dsclosure Authorzed Polcy Research Workng Paper 6726 WPS6726 Publc Dsclosure Authorzed Publc Dsclosure Authorzed Can Subjectve Questons on Economc Welfare Be Trusted? Evdence for Three Developng Countres Martn Ravallon Krsten Hmelen Kathleen Beegle Publc Dsclosure Authorzed The World Bank Development Research Group Poverty and Inequalty Team December 2013

2 Polcy Research Workng Paper 6726 Abstract Whle self-assessments of welfare have become popular for measurng poverty and estmatng welfare effects, the methods can be deceptve gven systematc heterogenety n respondents scales. Lttle s known about ths problem. Ths study uses specally-desgned surveys n three countres, Tajkstan, Guatemala, and Tanzana, to study scale heterogenety. Respondents were asked to score stylzed vgnettes, as well as ther own household. Dverse scales are n evdence, castng consderable doubt on the meanng of wdely-used summary measures such as subjectve poverty rates. Nonetheless, under the dentfyng assumptons of the study, only small bases are nduced n the coeffcents on wdely-used regressors for subjectve poverty and welfare. Ths paper s a product of the Poverty and Inequalty Team, Development Research Group. It s part of a larger effort by the World Bank to provde open access to ts research and make a contrbuton to development polcy dscussons around the world. Polcy Research Workng Papers are also posted on the Web at The authors may be contacted at kbeegle@worldbank.org. The Polcy Research Workng Paper Seres dssemnates the fndngs of work n progress to encourage the exchange of deas about development ssues. An objectve of the seres s to get the fndngs out quckly, even f the presentatons are less than fully polshed. The papers carry the names of the authors and should be cted accordngly. The fndngs, nterpretatons, and conclusons expressed n ths paper are entrely those of the authors. They do not necessarly represent the vews of the Internatonal Bank for Reconstructon and Development/World Bank and ts afflated organzatons, or those of the Executve Drectors of the World Bank or the governments they represent. Produced by the Research Support Team

3 Can Subjectve Questons on Economc Welfare Be Trusted? Evdence for Three Developng Countres Martn Ravallon, Krsten Hmelen and Kathleen Beegle 1 Key words: Subjectve welfare, subjectve poverty, scales, heterogenety, vgnettes JEL classfcatons: C81, D60, I32 Sector board: Poverty Reducton (POV) 1 Ravallon s wth Georgetown Unversty, Beegle and Hmelen are wth the World Bank. Ths paper would not have been possble wthout the work of the Tajkstan Lvng Standards Survey Team, ncludng Dane Steele, Sasun Tsrunyan, Vladmr Kolchn, Farhod Khamdov, Oleksy Ivaschenko, and staff of the Natonal Statstcs Commttee of Tajkstan (Goskomstat), Economc Development Intatves n Bukoba, Tanzana, ncludng Joachm De Weerdt and Albertus Kamanz, Centro de Estudos y Documentacón de la Frontera Occdental n Huehuetenango, Guatemala, and the Programa Regonal de Segurdad Almentara y Nutrconal para Centroamérca for fundng the Guatemala porton of feldwork. Helpful comments on ths paper were receved from Dens Cogneau, Sylve Lambert and Clauda Senk. All vews are those of the authors and do not reflect the vews of ther employers ncludng the World Bank or ts member countres.

4 1. Introducton Wdely used measures of subjectve welfare ask survey respondents to rate ther economc welfare, satsfacton wth lfe or happness on an ordnal scale. These measures have found nnumerable applcatons n the psychologcal and socal scences and have recently become popular n economcs. 2 However, dfferent people may well have dfferent deas about what t means to be rch or poor, or satsfed or not wth one s lfe, leadng them to nterpret survey questons on subjectve welfare dfferently. 3 For example, the Young Lves Project (2009) reports the comment of a sx-year old n rural Vetnam, named Duy, as sayng that We are nearly rch as we have a new cupboard, but we haven t got a washng machne. Duy clearly has a dfferent dea of what t means to be rch than those n Vetnam more famlar wth the lvng condtons of the truly rch. Survey respondents can be expected to nterpret subjectve questons relatve to ther personal frame-of-reference, whch wll depend on latent aspects of ther own knowledge and experence. Two mportant applcatons of subjectve welfare data llustrate why ths matters. The frst s ther applcaton n the nterpersonal comparsons of welfare requred for poverty measurement. Measures of subjectve poverty are becomng common. 4 These measures tell us what proporton of survey respondents place themselves on the bottom rung (or possbly second lowest rung) of a welfare ladder from poor to rch. But f the rungs of the welfare ladder are not understood the same way by dfferent respondents t s unclear what meanng can be attached to such measures. The second applcaton relates to the many studes of the covarates of subjectve welfare. 5 In now standard practce, a lnear or ordered probt (OP) regresson s run of the survey 2 The relevant economcs lterature s revewed by Frey and Stutzer (2002), D Tella and MacCulloch (2006) and Dolan et al. (2008). The psychologcal lterature on subjectve welfare s revewed n Dener et al. (1999) and Furnham and Argyle (1998). An alternatve approach s to ask what level of ncome s needed to attan a gven poston on a ladder, such as not beng poor. Ths s the Leyden method devsed by van Praag (1968). Whle we do not use ths type of data here, the same concerns about bas arse n the Leyden method. 3 Whle ths paper focuses on heterogenety n scales, there are other concerns wth survey desgn. For example, Cont and Pudney (2011) fnd that mnor re-desgns n questons on satsfacton of lfe/work led to large changes n answers, partcularly for women, fndng that dstortons n responses nfluence fndngs wth respect to correlates of women s job satsfacton. For an overvew of the concerns about nferrng welfare effects from subjectve data see Ravallon (2012). 4 Examples nclude Mangahas (1995), Ravallon and Lokshn (2002), Carletto and Zezza (2006), and Posel and Rogan (2013). 5 Examples nclude van de Stadt et al. (1985), Clark and Oswald (1994, 1996), Kapteyn et al. (1998), Oswald (1997), Wnkelmann and Wnkelmann (1998), Pradhan and Ravallon (2000), McBrde (2001), Ravallon and 2

5 responses aganst ndvdual and household characterstcs, such as age, gender, martal status, ncome, educaton, employment status, and household demographcs. Such regressons offer the prospect of dentfyng varous welfare effects and trade-offs of nterest (ncludng to polcy makers) under seemngly weaker dentfyng assumptons than requred by wdely-used methods that rely solely on objectve crcumstances, such as ncome or consumpton. We can agree n prncple that a person s economc welfare does not only depend on the household s current consumpton or ncome, but s also nfluenced by the sze and demographc composton of the famly and characterstcs such as educaton and employment. Prces are mssng for these other attrbutes. Subjectve data offer a soluton for dentfyng the trade-offs and constructng a composte ndex of welfare based on the regresson s predcted values. But can that soluton be trusted? The OP estmator assumes that the thresholds the values of the underlyng welfare metrc at whch ordnal responses on the stpulated scales change are constant parameters, the same for all respondents. We defne scale heterogenety as any stuaton n whch ths assumpton does not hold,.e., that the thresholds are dosyncratc. If there s such heterogenety and t s correlated wth the covarates n subjectve welfare regressons then based nferences about the underlyng welfare functon wll be drawn from the regressons found n the lterature. (Ths concern arses n addton to more famlar concerns about the possble endogenety of regressors, whch create confoundng correlatons wth the error term n the underlyng contnuous varable for subjectve welfare.) Concerns about such systematc measurement errors n subjectve questons have prompted some observers to warn aganst ther use as dependent varables. Bertrand and Mullanathan (2001, p.70) conclude that: subjectve varables cannot reasonably be used as dependent varables, gven that the measurement error lkely correlates n a very causal way wth the explanatory varables. Ths dsmsses a great many past and potental applcatons usng subjectve welfare questons. But s such a negatve assessment really warranted? It would be far to say that the potental problem of systematc scale heterogenety has receved lttle more than passng attenton n the extensve emprcal lterature makng subjectve welfare comparsons. Dolan et al. (2008) survey the fndngs of a large number of papers runnng Lokshn (2001, 2002, 2010), Graham and Pettnato (2002), Senk (2004), Luttmer (2005), Ferrer--Carbonell (2005), Graham and Felton (2006), Herrera et al. (2006), Bshop and Luo (2006), Kngdon and Knght (2006, 2007), Fafchamps and Shlp (2009), Knght and Gunatlaka (2010, 2012), Castlla (2010) and Posel and Rogan (2013). 3

6 regressons for subjectve welfare but do not explctly dscuss the potental for bas due to systematc dfferences n scales (though they do note concerns about the possble endogenety of some regressors). 6 A seemngly wdely-held vew s reflected n the authortatve survey paper by Frey and Stutzer (2002), whch notes the scope for scale heterogenety n self-reported welfare responses but clams that ths does not nvaldate regresson models for such data. That clam s hard to defend on a pror grounds gven the aforementoned concerns about bas. It would seem premature ether to gnore the problem (followng the advce of Frey and Stutzer) or to abandon subjectve poverty/welfare regressons knowng only that there s a potental for bas (followng Bertrand and Mullanathan). More evdence s needed. Askng survey respondents to place vgnettes descrbng hypothetcal stuatons on the same scale has been used to address scale heterogenety n a few studes of subjectve data on health status, poltcal effcacy, and job satsfacton. 7 Followng ths approach, Beegle, Hmelen and Ravallon (BHR) (2012) used vgnettes to study frame-of-reference effects on subjectve welfare and offered varous tests for confoundng effects of scale heterogenety usng data for Tajkstan. 8 BHR found that, despte the exstence of scale heterogenety, subjectve welfare regressons that gnored the problem stll gave qute smlar results to those that address t. The present paper makes two contrbutons, correspondng to the two applcatons descrbed above. Frst, we propose a new measure of subjectve poverty, anchored to a household vgnette desgned for descrbng a famly that most would consder poor n the specfc settng. Instead of countng as poor those who put themselves on (say) the lowest rung of a welfare ladder, wth uncertan meanng and comparablty, subjectve poverty s measured by comparng the household s self-assessed welfare to ts assessment of the welfare of ths specfc poor household descrbed n the vgnette. In other words, we ask for explct nter-personal comparsons of welfare aganst a common reference. The paper dscusses ths approach n theoretcal terms and provdes applcatons to three developng countres, Tajkstan, Guatemala 6 Some papers run lnear regressons for the ordnal responses on subjectve welfare rather than an OP. The assumpton of constant scales s explct n the OP but the problem s clearl stll present n the lnear models. 7 Kng et al. (2004) and Kng and Wand (2007) desgned vgnettes to establsh common ponts on the heterogeneous reference scales regardng poltcal effcacy n Chna and Mexco. Krstensen and Johansson (2008) used vgnettes n anchorng subjectve scales for job satsfacton. Kapteyn et al. (2008) use vgnettes to compare lfe satsfacton between respondents n the U.S. and the Netherlands. Bago d Uva et al. (2008) used them for correctng self-assessed health data for reportng bas. 8 Ths can be thought of as a contrbuton to ongong efforts to employ qualtatve data to help valdate standard objectve metrcs of welfare and poverty. For an overvew of varous approaches see Shaffer (2013). 4

7 and Tanzana. In studyng the scale-heterogenety problem n relatvely poor settngs we can reasonably expect to obtan a more complete characterzaton of lvng standards than would be possble wth a short vgnette n a rch country. Related to our new measure of subjectve poverty, we propose a measure of scale heterogenety among the poor. We fnd consderable scale heterogenety and substantally hgher poverty rates wth our new measure, but the emprcal determnants of subjectve poverty turn out to be very smlar to past methods gnorng scale heterogenety. Second, the paper tests the robustness of the conclusons of BHR wth regard to the extent of bases n standard regresson models for subjectve welfare. Here we follow what appears to be the most common approach n the lterature, whereby the survey responses are nterpreted as ordnal ndcators of a latent contnuous welfare metrc. Wth the addtonal assumptons of constant thresholds (the levels of welfare at whch ordnal responses swtch along the scale) and a normally dstrbuted error term n the latent welfare varable, an OP s then wdely used to model the data to retreve the parameters of the underlyng welfare functon. In common wth past work, the scales dentfed n the survey queston are not treated as havng any welfare sgnfcance they are merely the survey nstrument used to help dentfy the underlyng welfare metrc. The dfference wth past work s that we use the vgnette scores to relax the assumpton that the thresholds n the welfare space are constant across respondents. BHR also addressed ths ssue but only had the vgnette data for Tajkstan, for whch they found that respondents wth dfferent socoeconomc backgrounds tended to use systematcally dfferent scales n respondng to subjectve welfare questons. Ths prompted us to nvestgate the ssue elsewhere. We decded to pck two very dfferent poor areas of the world, n Guatemala and Tanzana. In both cases the study areas are clearly poor, but they are not unusually solated or equal, so t can be expected that people wll have some knowledge of the range of lvng standards n ther socetes. Usng the vgnettes developed for ths study n these qute dfferent settngs, we confrm the fndng of BHR that subjectve welfare regressons are reasonably robust to scale heterogenety. These fndngs put applcatons usng subjectve poverty and economc welfare data as dependent varables on a frmer foundaton. We begn wth a descrpton of our data. Secton 3 presents our approach to measurng subjectve poverty usng vgnettes. That secton also tests whether dfferent covarates emerge, 5

8 compared to past measures of subjectve poverty. Secton 4 turns to our tests of the robustness of regressons for subjectve economc welfare. Secton 5 concludes, also notng some caveats. 2. Survey data on subjectve economc welfare We study subjectve economc welfare, as measured by survey responses to the followng queston: Imagne a 6-step ladder where on the bottom, the frst step, stand the poorest people, and the hghest step, the sxth, stand the rch. On whch step are you today? Concerns about scale heterogenety also arse wth questons related to broader welfare concepts such as happness or satsfacton wth lfe. Our methods may well be adapted to these broader concepts, although t s lkely to be harder to devse credble and practcal vgnettes for such questons, gven that so many varables could be deemed relevant. Respondents were asked to place themselves on the subjectve welfare ladder descrbed above. Later n the questonnare they were asked to place four vgnettes, each descrbng a hypothetcal household, on the same ladder, and fnally to (agan) place ther own household on the ladder, after scorng the vgnettes. In askng the own-welfare questons both before and after the questons about the vgnettes, we are able to test whether the vgnettes alter the respondents perceptons of ther welfare. The vgnette questons may focus the respondent to thnk about, and possbly revse, the scale they have n mnd n reportng ther subjectve welfare (smlarly, see Hopkns and Kng, 2010). For all three countres, we developed the vgnettes n consultaton wth local counterparts. The actual vgnettes from the questonnares (translated nto Englsh) are gven n the Appendx. The vgnettes were desgned to capture representatve snapshots of varous levels of welfare n each country. The frst vgnette was desgned to present a scene that almost anyone n the country concerned would deem to be one of poverty. Ths was not ntended to be the poorest magnable desttute household, but rather an undenably poor famly wth a sustanable lvelhood. The second vgnette ndcated a famly wth mproved condtons, though a famly that some would stll consder poor n that settng. The thrd was ntended to represent a famly from the mddle class and the fourth an affluent famly. The characterstcs ncorporated n the vgnettes vared across countres and ncluded land holdngs, educaton, det, clothng, and the ablty to heat the home durng the wnter. The vgnettes were developed n a clear expected herarchy of domnance wth respect to economc welfare, wth all aspects of soco-economc 6

9 status ncreasng monotoncally. Ths structure was used to mnmze the effects of multdmensonalty, whch can lead to the perverse sequencng of the vgnettes wth respect to the ntended orderng f respondents place dfferent values on varous characterstcs contaned n the vgnettes. Such perverse rankngs are nconsstent wth our theoretcal model (outlned below), whch mples that respondents wll agree on the orderng of the vgnettes. Ths does not seem to be a concern for our vgnettes snce there were very few nstances of ncorrect orderng. 9 The most common characterstc of respondents who perversely order the vgnettes s a low level of educaton of the household head. These cases of ncorrect orderng were excluded from the analyss. In Tajkstan, the subjectve welfare experment was embedded n the 2007 Tajkstan Lvng Standards Measurement Survey (TLSMS). The sample conssts of 4,860 households ntervewed n September-November The sample s desgned to be representatve at the natonal, urban and rural levels, and at the oblast (admnstratve regon) level. Data were collected n two vsts. In addton to the standard questons common to mult-topc household questonnares, subjectve welfare modules were developed and asked n the frst vst. In Guatemala, the Impacto de las Remesas y la Mgracon sobre la Segurdad Almentara y Nutrconal (IRMSAN) survey was done n the department of Huehuetenango, Guatemala, located n the Western Hghlands and borderng the Mexcan state of Chapas. Ths area s entrely rural and s characterzed by hgh levels of both poverty and temporary outmgraton. The survey has a sample sze of 1,222 households ntervewed from Aprl-August The survey covered sx mcro-regons wthn the muncpaltes of Culco, San Gaspar Ixchl, Santa Ana Husta, and Jacaltenango selected purposvely to capture geographcal heterogenety. The survey was stratfed to over-sample mgrant households. Four months pror to feldwork, a household census was conducted n the regon. Probablty weghts are used n the analyss to adjust for the dfferng probabltes of selecton. In Tanzana, the Kagera Subjectve Welfare Survey was purposvely desgned for evaluatng the potental use of anchorng vgnettes n subjectve well-beng measurements. Felded n November-December 2007, the survey conssts of 450 households randomly selected from Ngara dstrct of Kagera regon on the border wth Rwanda. Ths s a poor area n Tanzana 9 In Tajkstan, 89 of the 4,860 households n the sample had ncorrect codng. In Guatemala the correspondng number was 28 out of 1,222 and n Tanzana t was 3 out of

10 but also an area where nfrastructure (especally roads) has mproved n recent tmes (after the genocde n Rwanda). Ths has opened up the Kagera area, ncludng the ntroducton of nformaton about lvng standards elsewhere. Whle the Tajkstan and Guatemala surveys were entrely verbal, n the case of Tanzana, the survey teamed used a paper dagram of the ladder of lfe onto whch respondents placed blocks for each of the household vgnettes and ther own household. Ths was developed after plotng showed that verbal-only mplementaton dd not work well. Half-steps were also allowed (scorng at, for example, 2.5). In Table 1 we compare the pre- and post-vgnette responses of the household s subjectve welfare. Whle there are some off-dagonal elements, the correspondence s qute strong. Most respondents place themselves n the same poston n the pre- and post-vgnette. (The footnote to each panel of the table gves summary statstcs on the strong correlaton n the cross-tab.) In all three cases, the modal responses are around the second or thrd rung of the ladder. Notce that very few respondents put themselves on the top rung of the ladder; pre-vgnette the proporton s only 0.2%, 0.1% and 0.4% for Tajkstan, Guatemala and Tanzana respectvely. Ths could well reflect an unwllngness of the rch to reveal ther true economc poston. 3. Subjectve poverty measures The vgnettes allow us to make explct at least a sub-set of the characterstcs that defne poverty. Ths s n contrast to the standard approach to measurng subjectve poverty n whch no characterstcs are explct. 3.1 Theoretcal representaton Let SW denote the subjectve welfare of respondent n a sample of N respondents. Ths s a contnuous but latent varable. The survey provdes an ordnal response welfare, whch s assumed to be ncreasng n R = 1 f R = k f SW 1 SW as follows: R on subjectve < τ for =1,..,N (1.1) τ < k 1 k SW τ for =1,..,N and k=2,,k (1.2) 8

11 1 2 where the thresholds are τ < τ <... < τ K for the K possble ordnal responses (wth K=6 n our case). In standard applcatons of subjectve welfare data the thresholds are taken to be fxed across all whle here we take them to be heterogeneous. SW The standard subjectve poverty measure desgnates a respondent to be poor f < τ ( R = 1). Usng ths approach we see n Table 1 that 7.5%, 25% and 17% of the 1 samples n Tajkstan, Guatemala, and Tanzana respectvely are deemed to be poor (postvgnettes). However, such measures lack concrete meanng n terms of lvng standards when the scales vary across respondents. Wth scale heterogenety, the resultng subjectve poverty measures need not be welfare consstent n that rankngs across dfferent respondents n terms of ther R s need not accord wth ther rankngs n terms of SW. Someone respondng that R = 2 (say) could well have a lower value of SW than someone respondng that R = 1. Planly ths could only be ruled out on a pror grounds f the scales are constant. The vgnettes help address ths problem by fxng a set of welfare-relevant characterstcs so as to lmt the varance n the underlyng dosyncratc scales. We assume that SW s a stable functon of a vector of varables, Z, so we can wrte SW = SW Z ). However, Z need not be fully observable. The vgnette characterstcs are nterpreted as a subset of the characterstcs n the vector Z. We can partton that vector as Z = ( Z v, Z o ) where characterstcs dentfed n the vgnettes and values, denoted ( o Z are other characterstcs. v Z s the vector of v Z can take varous v Z j where j=1,, J, whch defne each of the J vgnettes, where n our case J=4. v o The values taken by SW ( Z j, Z ) for vgnette j at any gven v Z j are treated as a random varable, the dstrbuton of whch reflects the respondent s uncertanty about the omtted characterstcs, o Z. We make two key dentfyng assumptons. Frst we assume nternal consstency between vgnette and own-welfare assessments. Specfcally, t s assumed that each respondent uses the same subjectve welfare functon n assessng the welfare of the vgnettes as for the respondent s 9

12 own welfare and that each respondent uses the same scales (albet personal scales) when assessng own-welfare and the vgnette welfare. 10 Secondly, we assume that each respondent s ordnal responses on subjectve welfare of each vgnette, denoted j R, are generated by an underlyng contnuous varable gven by ther expected values of the welfare of that vgnette and that respondents share a common dstrbuton of the unobserved vgnette characterstcs, Under these assumptons, o Z.e., 11 j v o v v SW = E[ SW ( Z, Z ) Z = Z ] for j=1,..,j (2) n scorng vgnette welfare are gven by: j j SW s a stable functon of the vgnette characterstcs and responses j 1 R = 1 f SW < τ (3.1) j R j k 1 j k = k f τ SW < τ for k=2,,k (3.2) Thus, even though the thresholds vary across respondents, all wll agree on the orderng of the vgnettes. Probably the greatest concern about these dentfyng assumptons s the possblty of heterogenety n the dstrbutons of the omtted characterstcs n each vgnette. Ths could yeld dsagreements among partcpants n the orderng of the vgnettes. However, as noted n the prevous secton, ths was rare emprcally; 98% n Tajkstan and Guatemala and over 99% n Tanzana gave the same, expected, orderng of the vgnettes. Wth ths set-up, our proposed vgnette-consstent measure of subjectve poverty says that respondent s poor f (and only f) 1 R R.e., the respondent s no better off than the poorest vgnette. (More generally, one can choose any reference vgnette r 1 and say that a person s poor f r R R.) Ths stll does not guarantee that anyone deemed to be poor by ths crteron has a lower SW Z ) than those not deemed to be poor. That would requre that the vgnettes provde ( a complete descrpton of the welfare-relevant characterstcs. The uncertantes about those characterstcs and the practcaltes of surveyng (notably the length of the questonnare) wll 10 The latter part of our assumpton s dentcal to the assumpton of response consstency n Kng et al. (2004). 11 One can relax the latter assumpton to allow for addtve dosyncratc dfferences n the level of subjectve k welfare assgned to a gven vgnette. Ths error term can then be subsumed n the τ s. Consstency requres that the same addtve error term appears n own-welfare. 10

13 undoubtedly preclude such completeness. However, our model does mply a form of consstency n terms of expected welfare, n the specfc sense that 1 R < R f (and only f) j SW < SW. Equalty of the ladder rungs ( R = 1 R ) does not, of course, assure that j SW = SW. In that respect, our measure s no dfferent to standard measures of subjectve welfare n that t s purely ordnal, whle the underlyng levels of subjectve welfare are taken to be contnuous. The vgnette-consstent subjectve poverty rate can be no lower than the ordnary raw poverty rate (the proporton wth R = 1). The vgnette-consstent rate wll pck up all those who put themselves on the lowest rung. But t wll also nclude those who thnk vgnette 1 s not n fact on the lowest rung and yet stll rate ther own household at or below that vgnette. Clearly such people have a very dfferent frame-of-reference and perceve the exstence of even greater deprvatons than descrbed by vgnette 1. The dfference between the poverty count based on R = 1 and that based on 1 R R reflects the extent to whch subjectvely poor people (by our defnton) put vgnette 1 above the lowest rung of the ladder. If everyone who s counted as poor by our new defnton ( R 1 R ) agrees that vgnette 1 s on the lowest rung of the ladder then the two measures wll be equal. Note that there s no obvous bass for makng cross-country comparsons usng ether method of measurng subjectve poverty. Here we are nterested n comparng the two methods wthn each country. From ths approach, a smple measure of scale heterogenety among the poor can be proposed. Consder frst the proporton of those who are subjectvely poor by our defnton and put vgnette 1 on the lowest rung. (We gnore dfferences n the rankngs gven to vgnette 1 among those who assgn R 1 > 1.) Denote ths proporton by: N = ( R = 1R R ) P = (4) N 1 1( R = R 1 ) where 1(.) takes the value 1 f the term n parentheses s true and zero otherwse. A natural measure of scale heterogenety among the poor (SHP) s then the scaled varance: SHP 4P(1 P) [0,1] (5) 11

14 At the lower bound of 0 all the poor put vgnette 1 on the same rung or none do. At the upper bound of 1 the varance of poor peoples rankngs of vgnette 1 s at ts maxmum, for whch half put vgnette 1 on the lowest rung. 3.2 Emprcal mplementaton Table 2 provdes the dstrbutons of respondents subjectve welfare levels relatve to each vgnette. The self-assessments provded after hearng the vgnettes are probably of greater nterest n ths context, so we focus the dscusson on those results. We fnd that 14% of the Tajkstan respondents mplctly felt that they were no better off than the poorest vgnette household, whch can rarely afford meat, has lmted heatng and warm clothng durng the wnter, has poorly clothed chldren, who are sent to work when reachng secondary-school age. (The full detals are gven n the Appendx.) By contrast, only 7.5% put themselves on the lowest rung. Thus we fnd consderable heterogenety n scales among the poor. Of those who judge ther own welfare to be no greater than that of the poorest vgnette, over half (56%) put that household above the lowest rung of the welfare ladder. The measure of scale heterogenety among the poor (usng equaton 5) n Tajkstan s thus very near the maxmum varance at SHP= For the Guatemala sample, 32% of survey respondents reported themselves as no better off than vgnette 1, whch lves n an adobe house wth one room and no latrne, electrcty nor runnng water, eatng beans and tortllas, and s unable to afford meat or eggs. (Recall that the sample s from a poor area.) Ths s closer to the poverty rate of 25% based solely on the subjectve welfare responses. In contrast to Tajkstan, we fnd that about three-quarters (73%) of the Guatemala respondents who were no better off n ther percepton than the poorest vgnette placed that household on the lowest rung of the welfare ladder. The measure of scale heterogenety among the poor s In Tanzana 25% of the sample put ther own welfare at or below that of a famly of three llterate adults and three chldren, only one of whch s n prmary school, lvng n a mud house wth no furnture, wth no pped water, no land and engaged n casual agrcultural labor. The famly has one small meal a day and very rarely eats meat or fsh. Correctng for scale heterogenety usng the vgnettes thus entals a szable ncrease n the subjectve poverty rate, 12

15 from 17% to 25%. For the Tanzana data, about one-thrd (36%) of the poor put vgnette 1 above rung 1, gvng SHP= The choce of vgnette 1 as the poverty lne for these calculatons s natural, but t s of nterest also to consder the mplcatons of usng vgnette 2 nstead. Usng the raw (uncorrected) data the poverty rates are then 34%, 74%, and 44% respectvely, whle the correspondng vgnette-consstent rates are 61%, 84%, and 79%. Whle cross-country comparablty s questonable (as already noted), t s at least notable that the rankng of the three countres n terms of poverty s dentcal, wth and wthout the vgnette correcton for scale heterogenety. The Guatemala sample s the poorest, followed by Tanzana, and Tajkstan the least poor. Ths holds for both vgnettes 1 and 2 as the reference. The two measures of subjectve poverty also share smlar covarates. Table 3 gves probts for the two measures of subjectve poverty the measure n whch one deems the respondent to be poor f she puts herself on the lowest rung and our proposed new measure. Despte the substantal scale heterogenety we fnd among the subjectvely poor, there s a strkng smlarty n the two probts. There are only a few cases n whch a coeffcent s sgnfcant n one and not the other, and the szes of the coeffcents are generally smlar. 4. Tests for bas n subjectve welfare regressons The subjectve poverty measures dscussed above naturally gnore rankngs among the non-poor, n keepng wth the usual focus axom n poverty measurement. We turn next to regresson analyss of the full range of welfare rankngs. Frst we ask whether vgnette responses are correlated wth covarates commonly found n subjectve welfare regressons n the lterature, ncludng objectve measures of economc welfare. We assume an ordered probt specfcaton, whch has become standard n the lterature. However, t should be noted that ths specfcaton requres a further restrcton on the nature of the heterogenety problem, as descrbed n the model represented by equatons (2) and (3). Specfcally, we also need to assume that the heterogenety takes the form of homogeneous shfts k n the scales, such that τ τ k 1 scale-transformed, contnuous varable, s constant across all for each k. Thus we can defne a new, j* SW, by addng an approprate (ndvdual- and 13

16 vgnette-specfc) constant to j SW such that the scales become constant across respondents when appled to the transformed varable, as requred by the ordered probt specfcaton. Our specfcaton for the determnants of ths transformed contnuous varable (generatng the ordnal categorcal responses on each vgnette) s as follows: SW j* = β ln PCE + π X + ε (j=1,4; =1,..,N) (6) j j j Here PCE denotes per capta expendture, X s a vector of other household-level varables and ε s a normally dstrbuted error term. The latent contnuous varable dscrete response on the scale from 1-6 for each vgnette wth constant scales. j* SW then generates a Table 4 summarzes the OP estmates. There are a number of systematc covarates, although the pseudo R 2 s are low, at approxmately In two of the three countres the pseudo R 2 s hghest for the poorest vgnette; n the thrd (Tanzana) t s roughly the same for the poorest and the least poor and both are hgher than for the mddle vgnettes. There s lttle sgn of a clear pattern n one drecton. For vgnettes 3 and 4 (but not 1 and 2), we fnd a postve and statstcally sgnfcant relatonshp between lnpce and the vgnette rankngs. Both poor and rch tend to agree that vgnettes 1 and 2 are poor, but rcher households are more lkely to gve a hgh welfare rankng to the better-off households descrbed by vgnettes 3 and 4. On the other hand smallholders tend to rate the poorest vgnette hgher than do other households. Geographc characterstcs are more lkely to be sgnfcant for the vgnettes hgher on the consumpton scale (vgnettes 3 and 4). How much do these effects bas the regressons often found n the lterature? Our second test tres to assess the robustness of a standard regresson for own-reported subjectve welfare, employng wdely-used covarates from the lterature. Analogously to (6) we assume that: SW = β ln PCE + πx + ε (=1,..,N) (7) Ths s the latent contnuous varable for the subjectve welfare of respondent, whch generates a dscrete response on the scale from 1-6. We refer to the estmated β as the economc gradent n subjectve welfare. We do not attempt to make the covarates X dentcal across the three countres. There s lttle obvous reason to do so, and reasons for adaptng the model to each context. (For example, mgraton s an mportant factor n the regon of study n Guatemala. And geographc effects are less relevant to our Tanzana data, as they come from just one dstrct.) 14

17 To test the robustness of the standard test based on (7), we allow for systematc covarates of the vgnettes under the assumpton of nternal consstency descrbed n Secton 3.1, but now relaxng our assumpton of homogeneous shfts n the scales. To do so we employ the method of Compound Herarchcal Ordered Probt, proposed by Kng et al., (2004), dubbed CHOPIT. In the standard model, one postulates the exstence of a seres of common cut-off ponts n the SW space that generate the observed ordnal responses (as n any OP). Instead, CHOPIT postulates that these thresholds are functons of a vector of observed covarates. The extra nformaton on the vgnette responses provdes the bass for dentfcaton, under the nternal consstency assumpton specfcally that the thresholds for a respondent s self-assessed welfare are determned dentcally as for that respondent s thresholds n the vgnette responses. Followng Kng et al. (2004), the thresholds are assumed to be gven by: τ = γ ln PCE + δ X (8.1) 1 k k 1 k k τ = τ + exp( γ ln PCE + δ X ) for k=2,,k (8.2) k k The dentfyng assumpton s that the same parameters, γ and δ, and (hence) the thresholds k τ determne the ordnal responses on the vgnettes. Such response consstency s a natural assumpton to make. Wthout the vgnettes, dentfcaton would only be possble under questonable assumptons about the nonlnearty of the functonal forms nvolved. Thus we are able to model determnants of the thresholds separately to those of the latent contnuous varable for subjectve welfare. 13 We begn by testng for an economc gradent n subjectve welfare. Table 5 gves the estmated economc gradent (the regresson coeffcents on log consumpton per capta) for varous specfcatons, comparng OP and CHOPIT, but wthout any controls n the subjectve welfare regresson. As can be seen from Table 5, the OP and CHOPIT estmates of the coeffcents of the equaton for subjectve welfare turn out to be qute close. Correctng for scale heterogenety attenuates the economc gradent (lower β ) for Tajkstan and Guatemala but 12 Notce that no error terms appear n the followng equatons. These are taken to be subsumed n the overall error term ε n equaton (7). 13 We mplemented the CHOPIT analyss usng the R statstcal analyss program, usng the programs 'anchors,' 'rgenoud,' and 'Zelg.' Further nformaton and documentaton on these packages s avalable at and The R code s avalable on request. 15

18 ncreases t n Tanzana. The post-vgnette coeffcent estmates are closer than the pre-vgnette estmates for Tajkstan and Tanzana but there s lttle dfference for Guatemala. Table 6 presents the results for an extended specfcaton ncludng varous other covarates often found n subjectve welfare regressons n the lterature, for both OP and CHOPIT. Agan we provde both OP and CHOPIT for both pre and post vgnettes. And, as found n the prevous table, the results are qute smlar between the two for each country. (For brevty, we do not comment on the regressons themselves, though they accord reasonably well wth our prors based on smlar regressons n the lterature.) Whle there are rather few dfferences between the OP and CHOPIT results, the more notable dfferences are as follows: () Tajkstan: The sgnfcant negatve effect n the OP of beng a female-headed household s not robust to allowng for scale heterogenety, although here too there s more smlarty n the post-vgnette case, wth a sgnfcant negatve effect ndcated (comparng the OP and CHOPIT coeffcents n Table 6(a)). Nor s the sgnfcant negatve effect of beng Russan robust (whether pre- or post-vgnette). Household sze has a stronger (postve) effect on subjectve welfare when we use CHOPIT. And one of the geographc effects (lvng n urban Khatlon) becomes much stronger. () Guatemala: The economc gradent n subjectve welfare falls when we adjust for scale heterogenety (Table 6(b)). A stronger female respondent effect emerges when we correct for bas usng CHOPIT, as does the (negatve) effect of beng sngle rather than marred. The geographc effects also change. () Tanzana: The economc gradent n subjectve welfare rses when we adjust for scale heterogenety and a much larger (negatve) effect of chldren emerges (Table 6(c)). Agan, we found that a number of covarates were sgnfcant predctors of the scales k (sgnfcantγˆ s), as revealed by the vgnettes although there s lttle clear pattern. Table 7 summarzes these results and s self-explanatory. 5. Conclusons There are a pror grounds for questonng past applcatons of survey-based subjectve assessments of welfare n measurng and modelng poverty and n calbratng welfare functons. Not only are the nterpretatons gven by households to the scales used n the survey questons lkely to vary castng doubt on the meanng of the derved measures but there are reasons to 16

19 expect these dfferences to confound nferences about welfare mpacts. For example, poorer people may well have more lmted horzons n lfe, stemmng from more lmted experences wth the extent of the dspartes n levels of lvng n socety as a whole. Such dfferences n knowledge and experence could well translate nto a dfference n the nterpretaton gven to the scales used n questons on subjectve welfare. In partcular, poorer people mght be expected to use lower thresholds for defnng poverty. Ths would confound efforts to measure poverty and dentfy welfare effects usng subjectve data. Some observers have concluded that such data should not be used as dependent varables castng doubt on a large lterature, and warnng aganst future applcatons. But s ths a serous problem for the many applcatons of subjectve welfare data, ncludng n measurng subjectve poverty and estmatng welfare effects? The paper has tred to answer that queston. An approach to measurng and modelng subjectve poverty and welfare has been proposed that takes scale heterogenety serously. Our approach reles on carefully desgned vgnettes on hypothetcal households, whch were added to household surveys ncludng questons on own-welfare and relevant covarates. Respondents scored the vgnettes on the same ladder used to report ther own subjectve economc welfare. For measurng subjectve poverty, nstead of askng what proporton of respondents say that they are on the lowest rung of the welfare ladder, we ask what proporton report that ther own welfare s no greater than the poorest vgnette. In dong so, subjectve poverty measures ncrease gven the presence of scale heterogenety. The ncrease s large n two of the three countres. There s consderable scale heterogenety among the poor. Whle the overall poverty rankng of the three countres s unaffected, the large change n levels resultng from addressng scale heterogenety rases concerns over usng a smple subjectve welfare measure to assess and compare poverty. However, on comparng regresson models for our new method of measurng subjectve poverty wth the past method gnorng scale heterogenety, we fnd lttle dfference n the coeffcents on the covarates of poverty or n ther statstcal sgnfcance. Whle there s ample scale dversty, ts systematc component does not serously confound nferences from prevalng methods that gnore ths problem. We do fnd some sgnfcant covarates for vgnette responses among a set of regressors commonly used to explan subjectve welfare, although the effects defy any smple pattern of bas. To explore further the extent of the overall bas n estmates of the regresson coeffcents of 17

20 subjectve welfare on standard covarates, we have compared them wth a model that explctly allows for the heterogenety n scales. For ths purpose, the thresholds were modeled as functons of covarates, assumng consstency between own-welfare scorng and scorng of the vgnettes. Whle some dfferences are notable, when taken overall, our results suggest qute smlar factors nfluencng subjectve welfare when comparng the standard regressons wth those augmented to allow for systematc scale heterogenety. Ths holds n the data studed for all three countres. Our fndngs suggest that scale heterogenety s a serous concern when usng subjectve welfare data for makng nter-personal comparsons of welfare. The meanng of wdely-used subjectve poverty measures appears hghly questonable. However, more encouragngly, scale heterogenety s not as great a concern as some observers have clamed for usng subjectve poverty and welfare measures as dependent varables n stuatons n whch there s no opton but to assume constant scales. It seems that, despte scale heterogenety, one can learn somethng that s reasonably robust about trade-offs from such data trade-offs that are otherwse hard to dentfy. It should be emphaszed that we have delberately focused here on subjectve economc welfare n developng countres. Applcatons n rch countres or on broader concepts of welfare such as happness or satsfacton wth lfe may well ental greater latent heterogenety n scales, although for happness t s possbly less obvous that ths heterogenety would be correlated wth standard regresson covarates. One mght try to develop vgnettes for rchcountry settngs or for broader welfare concepts. However, even asde from the dffcultes of havng long vgnettes n surveys, addng many more dmensons nto the vgnettes wll make t less lkely that an unambguous welfare rankng of the vgnettes s possble. So t mght reasonably be argued that we have pcked the low-lyng frut of the scale-heterogenety problem n subjectve welfare questons. How best to reach the hgher frut remans as a topc for future research. 18

21 Table 1: Pre-vgnette and post-vgnette subjectve welfare rankngs Tajkstan Post-vgnette Pre-vgnette poorest rchest Total 1 poorest , , , rchest Total 359 1,272 2, ,771 Note: Pearson ch2(25) = 8.0e+03 Pr = 0.000; lkelhood-rato ch2(25) = 4.9e+03 Pr = 0.000; Cramér's V = ; gamma = ASE = 0.009; Kendall's tau-b = ASE = Guatemala Post-vgnette Pre-vgnette poorest rchest Total 1 poorest rchest Total ,220 Note: Pearson ch2(25) = 1.1e+03 Pr = 0.000; lkelhood-rato ch2(25) = Pr = 0.000; Cramér's V = ; gamma = ASE = 0.022; Kendall's tau-b = ASE = Tanzana Post-vgnette Pre-vgnette poorest rchest Total 1 poorest rchest Total Note: Half-steps ncluded n the survey queston have been aggregated nto one step (the one below) to facltate comparsons. Pearson ch2(45) = Pr = 0.000; lkelhood-rato ch2(45) = Pr = 0.000; Cramér's V = ; gamma = ASE = 0.036; Kendall's tau-b = ASE =

22 Table 2: Subjectve economc welfare relatve to the vgnettes Tajkstan Guatemala Pre-Vgnette Self-Assessment Tanzana At or below vgnette At or below vgnette At or below vgnette At or below vgnette Post-Vgnette Self-Assessment At or below vgnette At or below vgnette At or below vgnette At or below vgnette Note: The table gves the proporton of respondents n each country who rated ther own subjectve economc welfare at or below the level they assgned to each vgnette (as gven n the Appendx). 20

23 Table 3(a): Probts for alternatve measures of subjectve poverty n Tajkstan (1) (2) Probt for placng oneself on step 1 Probt for placng oneself at or below vgnette 1 Coeff. s.e. Coeff. s.e. log per capta real consumpton *** *** Household Head Demographcs female headed household age of hh head Ethncty (Reference: Tajk) Uzbek Russan * Other 0.615*** *** Educaton (Reference: No Educaton) Prmary Basc General Secondary Specal Seconday Techncal Secondary Hgher Educaton Graduate School Household Characterstcs log household sze *** *** no. of elderly (65+) any mgrant(s) n the hhs no. of employed Agrculture / Fshng / Forestry Manufacture / Mnng Servces n.a. n.a ** Constructon Publc Admnstraton / Educaton / Health Sales and Servces * * Other Landholdng (Reference: No Land) Small Holdng Medum Holdng ** ** Large Holdng *** ** Geography Sogd Urban Sogd Rural ** Khatlon Urban *** Khatlon Rural 0.473*** *** RRP Urban 0.486*** ** RRP Rural 0.499*** *** Gbao Urban Gbao Rural Pseudo R Note: *** p<0.01, ** p<0.05, * p<0.1; Col (1): there were no households wth heads n servces. 21

24 Table 3(b): Probts for alternatve measures of subjectve poverty n Guatemala (1) (2) Probt for placng oneself on step 1 Probt for placng oneself at or below vgnette 1 Coeff. s.e. Coeff. s.e. Household Characterstcs log annual per capta consumpton *** *** log household sze *** *** female headed household * Respondent Characterstcs respondent s head of household * respondent s female * age of respondent year of educaton of respondent *** ** respondent s employed sngle / never marred ** *** unon *** *** separated/dvorced wdowed Muncpalty (reference: Culco) San Gaspar Ixchl Santa Ana Husta *** *** Jacaltenango Mgraton characterstcs any mgrant LA (excludng respondent) * any mgrant US (excludng respondent) mgrated nternally mgrated n Latn Amerca 0.466*** *** mgrated to USA Pseudo R Note: *** p<0.01, ** p<0.05, * p<0.1

25 Table 3(c): Probts for alternatve measures of subjectve poverty n Tanzana (1) (2) Probt for placng oneself on step 1 Probt for placng oneself at or below vgnette 1 Coeff. s.e. Coeff. s.e. Household Characterstcs log per capta consumpton *** *** log household sze share of women share of chldren under age share of chldren between 6 and share of members 55 and older log land sze (n acres) *** *** Household Head Characterstcs household head female 1.087*** *** household head age household head years of educaton * Respondent Characterstcs respondent female 1.121*** *** respondent age respondent years of educaton ** Pseudo R Note: *** p<0.01, ** p<0.05, * p<0.1 23