The State of the Art in Indicator Research

Transcription

1 International Society for Quality-of-Life Studies (ISQOLS) The State of the Art in Indicator Research Filomena Maggino

2 The State of the Art in Indicator Research I 1. Developing indicators defining the conceptual framework hierarchical design 2. Developing a system of indicators functions of systems of indicators elements characterising a system of indicators characteristics of indicators within a system (classification, quality, problems in selection) analysis of indicators within a system

3 Introduction Measuring processes by: a fundamental process lenght, volume a deriving process density, velocity a defining process socio-economic status

4 Introduction In social sciences, the measurement process requires indicators: what is INDICATOR? indicator who or what indicates index any thing that is useful to indicate

5 Introduction INDICATOR not a simply crude statistical information but a measure organically connected to a conceptual model index indicator when its definition and measurement occur in the ambit of a conceptual model and is connected to a defined aim

6 Contents 1. Developing indicators 2. Developing a system of indicators

7 1. 1. Developing indicators 2. Developing a system of indicators

8 Defining the conceptual framework Measuring well-being of societies different classification of conceptual frameworks (A) conceptual references

9 Defining the conceptual framework Functioning and capability to select goods and services that one desires economic indices (observed or estimated amounts, ) Normative ideals social indicators (living conditions, ) Subjective experiences subjective indicators (subjective reactions and perceptions)

10 Defining the conceptual framework Societal well-being is related to Functioning and capability to select goods and services that one desires Normative ideals Subjective experiences Societal well-being Strategies of observation What should be observed through what at which level/s Income, considered as a mean to achieve an acceptable standard of living a set of characteristics inspired by normative aims, grounded in moral values or policy goals Individual s cognitive and affective reactions to his/her whole life (or specific domains) and societies observed or estimated amounts living conditions subjective reactions and perceptions Macro (national) and/or micro (individual) macro (e.g. national) and micro (individual) micro (individual) Obtained measures economic indices and national accounts social indicators subjective indicators

11 Defining the conceptual framework Measuring well-being of societies different classification of conceptual frameworks (B) different perspectives

12 PROCESSES Defining the conceptual framework Growth Progress Development. CONDITIONS availability of resources distribution of resources impact of policies GOALS sustainability quality of life well-being

13 Defining the conceptual framework Measuring well-being of societies different classification of conceptual frameworks (C) different viewpoints

14 Defining the conceptual framework individuals Quality of life societies Quality of societies

15 Defining the conceptual framework Quality of life resources approach capabilities approach subjective well-being approach basic needs approach objective living conditions and subjective well-being approach

16 Defining the conceptual framework Quality of societies liveability and quality of nations Societal integration, solidarity and stability social cohesion social exclusion social capital sustainability human development social quality

17 Defining the conceptual framework An example of a comprehensive approach European System of Social Indicators EUSI

18 Defining the conceptual framework The concepts considered by EUSI Three pillars: (i) quality of life (ii) economic and social cohesion (iii) sustainability

19 Defining the conceptual framework (i) Quality of life concept (micro level) Level of Subjective well-being high low Objective high well-being dissonance living conditions low adaptation deprivation

20 Defining the conceptual framework (ii) Economic and social cohesion concept Two goal dimensions has been distinguished: 1. reduction of disparities and inequalities and fighting social exclusion 2. strengthening of connections and social ties including the enhancement of social capital.

21 Defining the conceptual framework (iii) Sustainability concept The sustainable development is referred to the World Bank s four capital approach. Four goal dimensions are the enhancement and preservation of social, human, produced and natural capital. For each type of capital two aspects have been considered: (i) (ii) preservation or enhancement of social capital of present generations and provision for future generations.

22 Defining the conceptual framework Measuring well-being of societies different classification of conceptual frameworks (D) objective vs subjective

23 Defining the conceptual framework Before defining the two components clarifying the meaning of adjectives objective and subjective. They can be referred to different concepts: conceptual framework measurement and analysis method observed / measured characteristics

24 Defining the conceptual framework Quantitative macro micro macro O b j e c t i v e S u b j e c t i v e micro macro Qualitative

25 Defining the conceptual framework Quantitative macro micro macro O b j e c t i v e what S u b j e c t i v e micro macro Qualitative

26 Defining the conceptual framework Quantitative macro micro macro O b j e c t i v e in which way S u b j e c t i v e micro macro Qualitative

27 Defining the conceptual framework Objective characteristics Demographic and socio-economic characteristics Observable acquired knowledge - sex - age - civil/marital status - household - educational qualification - skills - cognition - occupation - geographical mobility (birthplace / residence / domicile) - social mobility (original family status) - know-how - competences Micro level Individual living conditions (resources) Social capital Observable behaviours and life style - standards of living - financial resources (income) - housing - social relationships - freedom to choose one's lifestyle - working and professional conditions and status - state of health - activities (work, hobby, vacation, volunteering, sport, shopping, etc.) - engagements (familiar, working, social, etc.) - habits (schedule, using of public transport and of means of communication, diet, etc.) - public life (participation, voting, etc)

28 Defining the conceptual framework Objective characteristics Social conditions Social exclusion Social inclusion Disparities, equalities/inequalities, opportunities Informal networks, associations and organisations and role of societal institutions Macro level Structure of societies Political setting Institutional setting Human rights, democracy, freedom of information, etc. Educational system Health system Energy system Economical setting Income distribution, etc. Environmental conditions Decisional and institutional processes

29 Defining the conceptual framework Subjective characteristics - verbal comprehension and fluency - memory (rote, visual, meaningful, etc.) Abilities / capacities intellectual - numerical facility - reasoning (deductive and inductive) - ability to seeing relationships - special orientation - perceptual speed Micro level Personality traits special - social traits - motives - mechanical skills - artistic pursuits - personal conceptions - adjustment - physical adroitness - personality dynamics Interests and preference Values Sentiments Attitudes cognitive evaluations (beliefs, evaluations opinions) affective perceptions (satisfaction and emotional states i.e., happiness) behavioural intentions

30 Defining the conceptual framework Subjective characteristics How to measure subjective characteristics?

31 Defining the conceptual framework Subjective characteristics Measuring approaches Performance Inventory / self inventory Self-reported measures Observational methods Projective techniques Physiological measures Dimensions that can be measured abilities personality values interests attitudes opinions abilities behaviour social traits motivations adjustment attitudes

32 Defining the conceptual framework Measuring approaches Performance Inventory / self inventory Self-reported measures Observational methods Projective techniques Physiological measures Dimensions that can be measured abilities personality values interests attitudes opinions abilities behaviour social traits motivations adjustment attitudes Subjective characteristics Measures - execution of an assigned task; - outcome evaluated with reference to specified criteria of success individual answer to a certain number of submitted stimuli subject s answer expressed in terms of agreement, preference, etc. to a particular statement referring to the characteristic to be measured observation made on the subject by an external and neutral observer individual s reaction to one situation constructed but not completed relationship between subjective traits and physiological processes Notes intra-individual and inter-individual comparisons to be evaluated - requires strong interpretative approaches. - interpretation of answers strictly connected to researcher s experiences (subjective method) - method considered not scientifically applicable and relevant (not standardizable) - useful to integrate other approaches scientific evidence of this connection is not completely demonstrable

33 Defining the conceptual framework Subjective characteristics What are the difficulties in measuring subjective characteristics?

34 Defining the conceptual framework individual factors as: social desirability Subjective characteristics response set (stereotyped answers) researcher s attitude semantic factors (interpretative discordance between subject and researcher) situational factors different situations presence or not of other persons different context at home, at work, in the street, etc.

35 Defining the conceptual framework Objective and subjective characteristics What is their relationship?

36 Defining the conceptual framework Objective and subjective characteristics housing financial conditions personality traits health work Subjective well-being professional status social networks education health enviromental conditions

37 Defining the conceptual framework Relationships between subjective and objective components Objective characteristics descriptive / background components Subjective characteristics evaluative

38 Defining the conceptual framework Relationships between subjective and objective components Objective living conditions Subjective well-being Social and economic development Quality-of-life improvement

39 Defining the conceptual framework Relationships between subjective and objective components Comparison of objective conditions Subjective well-being Standards of comparison previous experiences with other people with aspirations Ambits of comparison Housing Work Family Friends

40 Defining the conceptual framework Relationships between subjective and objective components Comparison of objective conditions Subjective well-being smaller the perceived gap higher the subjective well-being through different comparators with reference to different ambits (housing, work, family, friends, etc.).

41 Defining the conceptual framework Relationships between subjective and objective components Multiple discrepancies approach Subjective well-being perceived gap between what one has wants what others have one has had in the past one expected to have one expected to deserve expected with reference to needs happiness not dependent on living conditions

42 Defining the conceptual framework Relationships between subjective and objective components Disposition approach Stable individual characteristics (personality traits) Subjective well-being

43 Defining the conceptual framework Relationships between subjective and objective components Causal approach (I) Subjective well-being = reactive state to the environment bottom-up The sum of the reactive measures for the defined ambits allows subjective well-being to be quantified

44 Defining the conceptual framework Relationships between subjective and objective components Causal approach (II) Individual stable traits Subjective well-being top-down

45 Defining the conceptual framework Relationships between subjective and objective components Causal approach Subjective well-being Two components : a long-period component (top-down effect), a short-period component (bottom-up effect) up-down

46 Hierarchical design CONCEPTUAL MODEL HIERARCHICAL DESIGN AREAS TO BE INVESTIGATED LATENT VARIABLES ELEMENTARY INDICATORS

47 Hierarchical design COMPONENT THE QUESTION TO BE ANSWERED DEFINITION OF THE COMPONENT CONCEPTUAL MODEL Phenomena to be studied defines the phenomenon to be studied, the domains and the general aspects characterizing the phenomenon Process of abstraction AREAS/PILLARS TO BE INVESTIGATED LATENT VARIABLES ELEMENTARY INDICATORS (E.I.)

48 Hierarchical design COMPONENT THE QUESTION TO BE ANSWERED DEFINITION OF THE COMPONENT CONCEPTUAL MODEL AREAS/PILLARS TO BE INVESTIGATED Aspects defining the phenomenon Each area represents each aspect allowing the phenomenon to be specified consistently with the conceptual model LATENT VARIABLES ELEMENTARY INDICATORS (E.I.)

49 Hierarchical design COMPONENT THE QUESTION TO BE ANSWERED DEFINITION OF THE COMPONENT CONCEPTUAL MODEL AREAS/PILLARS TO BE INVESTIGATED LATENT VARIABLES Elements to be observed Each variable represents each element that has to be observed in order to define the corresponding area. The variable is named latent since is not observable directly ELEMENTARY INDICATORS (E.I.) theoretical assumptions (dimensionality) empirical statements Their definition requires:

50 Hierarchical design COMPONENT THE QUESTION TO BE ANSWERED DEFINITION OF THE COMPONENT CONCEPTUAL MODEL AREAS/PILLARS TO BE INVESTIGATED LATENT VARIABLES ELEMENTARY INDICATORS (E.I.) In which way each element has to be measured Each indicator (item, in subjective measurement) represents what is actually measured in order to investigate each variable. appropriate techniques They are defined by: a system allowing observed values to be interpreted and evaluated

51 Hierarchical design conceptual model area I area II area III area... variable 1 variable 2 variable... variable... variable... variable... E.I. a E.I. b E.I.... E.I.... E.I.... E.I....

52 Hierarchical design The hierarchical design is completed by the definition of relationships between latent variables and corresponding indicators (model of measurement) latent variables (managing the complexity)

53 Hierarchical design Two different conceptual approaches: models with reflective indicators models with formative indicators

54 Hierarchical design Models with reflective indicators indicators functions of the latent variable changes in the latent variable are reflected in changes in the observable indicators top-down explanatory approach

55 Hierarchical design Models with formative indicators indicators causal in nature changes in the indicators determine changes in the definition / value of the latent variable bottom-up explanatory approach

56 2. 1. Developing indicators 2. Developing a system of indicators

57 Introduction Application of the hierarchical design system of indicators Not a collection of indicators Each indicator measures and represents a distinct aspect of the observed phenomenon

58 Introduction system of indicators key elements conceptual framework system architecture dimensions to be measured units to be monitored measuring and monitoring procedures

59 Introduction system of indicators characteristics Objectivity equal, comparable results Quantification quantitative values Efficiency and fidelity communication of results Economicity simple, standardized, up-to-datable information Generalization exportability of the system Joint development shared by all the actors

60 Introduction system of indicators formal criteria Comprehensiveness Consistency Non redundancy parsimoniousness

61 Functions of systems of indicators

62 Functions Functions can be seen in cumulative terms (each requires the previous) : Monitoring Reporting Forecasting Program/performance evaluation Accounting Assessment

63 Functions Monitoring capacity of the system to monitor changes over time and meet the need of improving knowledge Reporting Forecasting Program/performance evaluation Accounting Assessment

64 Functions Monitoring Reporting monitoring + analysis + interpretation Forecasting Program/performance evaluation Accounting Assessment

65 Functions Monitoring Reporting Forecasting trends in observed reality Program/performance evaluation Accounting Assessment

66 Functions Monitoring Reporting Forecasting Program/performance evaluation problem definition, policy choice and evaluation of alternatives and program monitoring Accounting Assessment

67 Functions Monitoring Reporting Forecasting Program/performance evaluation Accounting supporting decision concerning the allocation and the destination of resources Assessment

68 Assessment certificate or judge subjects (individuals or institutions) by discriminating their performances or to infer functioning of institutions, enterprises or systems. Functions Monitoring Reporting Forecasting Program/performance evaluation Accounting

69 Elements defining a system of indicators

70 Elements i. Aims ii. Structure iii. Analytical approaches iv. Interpretative and evaluating models

71 Elements i. Aims Conceptual goals Operative objectives Planning actions indicators function - input measuring resources available in the system and indicating some sort of inputs into a process - process monitoring the basic progress of implementing the actions defined (intermediate output) and outlined at strategic level - output/outcome monitoring direct results of actions - impact monitoring progress and improvement towards goals and objectives achievement

72 Elements ii. Structure Vertical from local to higher level Horizontal different ambits at the same level Local monitoring internal organization of the level and referring to external parameters

73 Elements iii.analytical approaches consistently with aims and structure

74 Elements iv.interpretative and evaluating models according to a specific reference frame. Possibility to define and identify particular standard-values, which can be defined a priori, according to the objectives or empirical observations (e.g. surveys).

75 Characteristics of indicators within a system

76 Characteristics Classification Purposes Governance contexts Perspectives of observation Forms of observation Levels of communication

77 Characteristics Classification Purposes descriptive (describing a reality) explicative (interpreting a reality) predictive (identifying trends) normative (supporting decisions) problem oriented (testing hypotheses) evaluating practical directionable actionable (process advancement effect)

78 Characteristics Classification Governance context Public debates Policy guidance Administrative guidance

79 Characteristics Classification Perspective of observation Conglomerative approach measures capturing advances made by the society as a whole Deprivational approach assessing status of the deprived Both needed for an adequate understanding

80 Characteristics Classification Forms of observation Status indicators capturing a reality in a particular moment Trend indicators observing a reality along time (longitudinal design of observation)

81 Characteristics Classification Level of communication Cold indicators complex and difficult, for specialists Hot indicators simple and easy Warm indicators good balance between quality, comprehensibility and resonance

82 Characteristics Quality I. Methodological soundness II. Integrity III. Serviceability IV.Accessibility

83 Characteristics An indicator should be able to: Define and describe Observe unequivocally and stably Record by a degree of distortion as low as possible Adhere to the principle of objectivity Reflect adequately the conceptual model Meet current ad potential users needs Be observed through realistic efforts and costs Reflect the length of time between its availability and the event of phenomenon it describes Be analyzed in order to record differences and disparities Be spread (I) METHODOLOGICAL SOUNDNESS (II) INTEGRITY (III) SERVICEABILITY (IV) ACCESSIBILITY

84 Characteristics Prerequisity of quality Legal and institutional environment (coordination within and across institutions) Resources available for statistics work Quality awareness informing statistical work

85 Characteristics Problems in selecting indicators Different issues need to be addressed in selecting and managing indicators, into a complex system for monitoring, reporting and accounting

86 Characteristics Problems in selecting indicators Michalos (2006) identified 15 different issues related to the combination of social, economic and environmental indicators. The issues collectively yield over 200,000 possible combinations representing at least that many different kinds of systems:

87 Characteristics Problems in selecting indicators Settlement/aggregation area sizes Time frames Population composition Domains of life composition Objective versus subjective indicators Positive versus negative indicators Input versus output indicators Benefits and costs Measurement scales Report writers Report readers Quality-of-life model Distributions Distance impacts Causal relations

88 Analysis of indicators within a system

89 Analysis ANALYTIC APPROACH OBJECTIVE Monitoring to monitor developments of a specific condition (e.g. environmental conditions) Reporting to report the results as they are obtained in a hierarchical procedure of decision-making Trend to clarify development trend Evaluation to record and evaluate the effects of planned and performed initiatives and actions Benchmarking to compare between performances of the considered units (e.g. countries) Assessment to clarify the impacts of planned and undertaken initiatives and actions

90 Flight desk Needs to - Know destination ( aim) - Share destination ( democracy) - Know pre-conditions ( ) - Know flight conditions ( monitoring) - Communicate flight conditions - Know how to manage emergency -.

91 Presentation designer: