Discussant: Heinz-Herbert Noll GESIS, Social Indicators Research Centre, Mannheim. Session STS 045 ISI World Congress, Dublin, August 25, 2011

Transcription

1 Obtaining weights: from objective to subjective approaches in view of more participative methods in the construction of composite indicators of well-being Discussant: Heinz-Herbert Noll GESIS, Social Indicators Research Centre, Mannheim Session STS 045 ISI World Congress, Dublin, August 25, 2011

2 Indices should be based on micro-data rather than aggregated data, allowing disaggregation and application of full set of statistical analysis! Composite Indices Pros: allow to condense statistical information and to measure multi-dimensional phenomena at a uni-dimensional scale make it easier to communicate relevant information and to catch the attention of the media and the public Cons: methodological difficulties and flaws risk of simplification level of differentiation possibly not adequate to the information demands of an increasingly complex world Indices should only be used as add-ons to differentiated indicator systems

3 Weighting Different choices: yes / no (incl. equal weighting; implicit weighting) if yes, objective (extracted from data) vs. subjective (expert/population ratings) weights fixed vs. changing/dynamic weights individual weights vs. average weights (in case of population ratings)

4 Advantages / Disadvantages: Subjective weights: closer to reality, but problem of arbitrariness, particularly in case of expert ratings Objective weights: less arbitrary, but sometimes difficult to communicate and comprehend Fixed weights: more transparency (weighting scheme easier to communicate), better comparability (across time, countries, cities etc.) of index values changes/differences in index values due to 'real' changes/differences rather than changes/differences of weights; Analogy to price-index: using weights of either base year (Laspeyres Index) or current year (Paasche Index) Changing/dynamic weights: closer to reality, but intransparent and difficult to comprehend; index mixes changes/differences of indicator values and weights!

5 Subjective Importance Weighting Based on Population Ratings Sometimes suggested as the gold standard (e.g. Hagerty/Land: 35; Sharpe/ Andrews: 8) However, raises also questions, such as: - are importance ratings stable across time? - do importance ratings vary across population subgroups within countries? - do importance ratings vary across countries? - are importance ratings independent observations?

6 Importance Ratings of Selected Life Domains, West-Germany (Scale 1-7, 1=unimportant, 7= very important) Importance of Family & Children Occupation & Work Leisure & Recreation Friends Mean SD Mean SD Mean SD Mean SD ,2 1,5 5,7 1,6 5,8 1,3 5,4 1, ,1 1,6 5,4 1,8 5,6 1,4 5,4 1, ,3 1,4 5,6 1,7 5,7 1,3 5,5 1, ,1 1,6 5,3 1,8 5,6 1,4 5,5 1, ,3 1,3 5,7 1,5 5,8 1,2 5,7 1, ,2 1,4 5,6 1,6 5,8 1,3 5,7 1, ,0 1,7 5,2 2,0 5,5 1,5 5,6 1,3 Database: ALLBUS; own calculations

7 Importance Ratings of Selected Life Domains, West-Germany 1998 (Scale 1-7, 1=unimportant, 7= very important) Importance of Age Family & Children Occupation & Work Leisure & Recreation Friends Mean SD Mean SD Mean SD Mean SD ,7 1,8 5,7 1,6 5,8 1,3 5,9 1, ,1 1,7 5,9 1,6 5,5 1,4 5,6 1, ,2 1,6 5,3 2,0 5,5 1,4 5,5 1, ,1 1,6 3,7 2,3 5,1 1,6 5,6 1,3 Relatives Religion & Church Politics & Public Life Neighbourhood Mean SD Mean SD Mean SD Mean SD ,9 1,6 3,3 2,0 3,9 1,6 4,1 1, ,0 1,6 3,6 2,0 4,3 1,5 4,7 1, ,1 1,5 4,0 1,9 4,3 1,7 5,0 1, ,4 1,5 4,7 1,9 4,0 1,8 5,0 1,6 Database: ALLBUS; own calculations

8 Database: European Values Study 2008; own calculatios 3,5 Importance of family (means) Scale: 1 (very important) - 4 (not at all important) 3 2,5 2 1,5 1

9 3,5 Importance of work (means) Scale: 1 (very important) - 4 (not at all important) 3 2,5 2 1,5 1 Database: European Values Study 2008; own calculatios

10 Database: European Values Study 2008; own calculatios 3,5 Importance of friends (means) Scale: 1 (very important) - 4 (not at all important) 3 2,5 2 1,5 1

11 Database: European Values Study 2008; own calculatios 3,5 Importance of leisure time (means) Scale: 1 (very important) - 4 (not at all important) 3 2,5 2 1,5 1

12 3,5 Importance of religion (means) Scale: 1 (very important) - 4 (not at all important) 3 2,5 2 1,5 1 Database: European Values Study 2008; own calculatios

13 Importance Ratings - seem to be pretty stable across time - seem to vary somewhat across subgroups within societies - seem to vary considerably across societies Consequences / limitations for the use of weights based on national averages and fixed (across time / countries) weights?

14 Are importance ratings independent from satisfaction ratings? Usually importance as well as satisfaction distributions skewed important & satisfied = most frequent combination positive correlation Previous research (Borg/Noll 1990) suggests: Importance ratings not independent from satisfaction ratings, rather there seems to be interdependency between the two! Dissonance feelings and dissonance reduction strategy as explaining mechanism

15 Importance Two different sorts of associations, depending on characteristics of respondents (internal/external locus of control; good/poor health) and domain: high low

16 Source: Borg/Noll 1990

17 Source: Borg/Noll 1990

18 Message: Importance ratings seem to vary with different levels of satisfaction The association between importance and satisfaction seems to vary across domains The association between importance and satisfaction seems to be different across groups of respondents Consequences for the usability of subjective importance ratings as weights?

19 Conclusions Weighting raises many questions There is obviously no single best solution Non or equal weighting sometimes seems to be better solution than (unequal) weighting, but also implicates strong assumptions concerning substitutability of index components Importance weighting based on population survey data has been suggested as sort of best practice (concerning subjective weighting), but also raises considerable doubts

20 Alternatives? User weighting (e.g. OECD Better Life Index) does not claim to approximate reality, but rather provides a tool to explore and demonstrate the consequences of allocating different value / weight to index components; Doing without composite indices by using different approaches of condensing and synthesizing information, like key indicators, dashboards etc.