International Society for Quality-of-Life Studies (ISQOLS) The State of the Art in Indicator Research Filomena Maggino filomena.maggino@unifi.it
The State of the Art in Indicator Research I 1. Developing indicators defining the conceptual framework hierarchical design 2. Developing a system of indicators functions of systems of indicators elements characterising a system of indicators characteristics of indicators within a system (classification, quality, problems in selection) analysis of indicators within a system
Introduction Measuring processes by: a fundamental process lenght, volume a deriving process density, velocity a defining process socio-economic status
Introduction In social sciences, the measurement process requires indicators: what is INDICATOR? indicator who or what indicates index any thing that is useful to indicate
Introduction INDICATOR not a simply crude statistical information but a measure organically connected to a conceptual model index indicator when its definition and measurement occur in the ambit of a conceptual model and is connected to a defined aim
Contents 1. Developing indicators 2. Developing a system of indicators
1. 1. Developing indicators 2. Developing a system of indicators
Defining the conceptual framework Measuring well-being of societies different classification of conceptual frameworks (A) conceptual references
Defining the conceptual framework Functioning and capability to select goods and services that one desires economic indices (observed or estimated amounts, ) Normative ideals social indicators (living conditions, ) Subjective experiences subjective indicators (subjective reactions and perceptions)
Defining the conceptual framework Societal well-being is related to Functioning and capability to select goods and services that one desires Normative ideals Subjective experiences Societal well-being Strategies of observation What should be observed through what at which level/s Income, considered as a mean to achieve an acceptable standard of living a set of characteristics inspired by normative aims, grounded in moral values or policy goals Individual s cognitive and affective reactions to his/her whole life (or specific domains) and societies observed or estimated amounts living conditions subjective reactions and perceptions Macro (national) and/or micro (individual) macro (e.g. national) and micro (individual) micro (individual) Obtained measures economic indices and national accounts social indicators subjective indicators
Defining the conceptual framework Measuring well-being of societies different classification of conceptual frameworks (B) different perspectives
PROCESSES Defining the conceptual framework Growth Progress Development. CONDITIONS availability of resources distribution of resources impact of policies GOALS sustainability quality of life well-being
Defining the conceptual framework Measuring well-being of societies different classification of conceptual frameworks (C) different viewpoints
Defining the conceptual framework individuals Quality of life societies Quality of societies
Defining the conceptual framework Quality of life resources approach capabilities approach subjective well-being approach basic needs approach objective living conditions and subjective well-being approach
Defining the conceptual framework Quality of societies liveability and quality of nations Societal integration, solidarity and stability social cohesion social exclusion social capital sustainability human development social quality
Defining the conceptual framework An example of a comprehensive approach European System of Social Indicators EUSI
Defining the conceptual framework The concepts considered by EUSI Three pillars: (i) quality of life (ii) economic and social cohesion (iii) sustainability
Defining the conceptual framework (i) Quality of life concept (micro level) Level of Subjective well-being high low Objective high well-being dissonance living conditions low adaptation deprivation
Defining the conceptual framework (ii) Economic and social cohesion concept Two goal dimensions has been distinguished: 1. reduction of disparities and inequalities and fighting social exclusion 2. strengthening of connections and social ties including the enhancement of social capital.
Defining the conceptual framework (iii) Sustainability concept The sustainable development is referred to the World Bank s four capital approach. Four goal dimensions are the enhancement and preservation of social, human, produced and natural capital. For each type of capital two aspects have been considered: (i) (ii) preservation or enhancement of social capital of present generations and provision for future generations.
Defining the conceptual framework Measuring well-being of societies different classification of conceptual frameworks (D) objective vs subjective
Defining the conceptual framework Before defining the two components clarifying the meaning of adjectives objective and subjective. They can be referred to different concepts: conceptual framework measurement and analysis method observed / measured characteristics
Defining the conceptual framework Quantitative macro micro macro O b j e c t i v e S u b j e c t i v e micro macro Qualitative
Defining the conceptual framework Quantitative macro micro macro O b j e c t i v e what S u b j e c t i v e micro macro Qualitative
Defining the conceptual framework Quantitative macro micro macro O b j e c t i v e in which way S u b j e c t i v e micro macro Qualitative
Defining the conceptual framework Objective characteristics Demographic and socio-economic characteristics Observable acquired knowledge - sex - age - civil/marital status - household - educational qualification - skills - cognition - occupation - geographical mobility (birthplace / residence / domicile) - social mobility (original family status) - know-how - competences Micro level Individual living conditions (resources) Social capital Observable behaviours and life style - standards of living - financial resources (income) - housing - social relationships - freedom to choose one's lifestyle - working and professional conditions and status - state of health - activities (work, hobby, vacation, volunteering, sport, shopping, etc.) - engagements (familiar, working, social, etc.) - habits (schedule, using of public transport and of means of communication, diet, etc.) - public life (participation, voting, etc)
Defining the conceptual framework Objective characteristics Social conditions Social exclusion Social inclusion Disparities, equalities/inequalities, opportunities Informal networks, associations and organisations and role of societal institutions Macro level Structure of societies Political setting Institutional setting Human rights, democracy, freedom of information, etc. Educational system Health system Energy system Economical setting Income distribution, etc. Environmental conditions Decisional and institutional processes
Defining the conceptual framework Subjective characteristics - verbal comprehension and fluency - memory (rote, visual, meaningful, etc.) Abilities / capacities intellectual - numerical facility - reasoning (deductive and inductive) - ability to seeing relationships - special orientation - perceptual speed Micro level Personality traits special - social traits - motives - mechanical skills - artistic pursuits - personal conceptions - adjustment - physical adroitness - personality dynamics Interests and preference Values Sentiments Attitudes cognitive evaluations (beliefs, evaluations opinions) affective perceptions (satisfaction and emotional states i.e., happiness) behavioural intentions
Defining the conceptual framework Subjective characteristics How to measure subjective characteristics?
Defining the conceptual framework Subjective characteristics Measuring approaches Performance Inventory / self inventory Self-reported measures Observational methods Projective techniques Physiological measures Dimensions that can be measured abilities personality values interests attitudes opinions abilities behaviour social traits motivations adjustment attitudes
Defining the conceptual framework Measuring approaches Performance Inventory / self inventory Self-reported measures Observational methods Projective techniques Physiological measures Dimensions that can be measured abilities personality values interests attitudes opinions abilities behaviour social traits motivations adjustment attitudes Subjective characteristics Measures - execution of an assigned task; - outcome evaluated with reference to specified criteria of success individual answer to a certain number of submitted stimuli subject s answer expressed in terms of agreement, preference, etc. to a particular statement referring to the characteristic to be measured observation made on the subject by an external and neutral observer individual s reaction to one situation constructed but not completed relationship between subjective traits and physiological processes Notes intra-individual and inter-individual comparisons to be evaluated - requires strong interpretative approaches. - interpretation of answers strictly connected to researcher s experiences (subjective method) - method considered not scientifically applicable and relevant (not standardizable) - useful to integrate other approaches scientific evidence of this connection is not completely demonstrable
Defining the conceptual framework Subjective characteristics What are the difficulties in measuring subjective characteristics?
Defining the conceptual framework individual factors as: social desirability Subjective characteristics response set (stereotyped answers) researcher s attitude semantic factors (interpretative discordance between subject and researcher) situational factors different situations presence or not of other persons different context at home, at work, in the street, etc.
Defining the conceptual framework Objective and subjective characteristics What is their relationship?
Defining the conceptual framework Objective and subjective characteristics housing financial conditions personality traits health work Subjective well-being professional status social networks education health enviromental conditions
Defining the conceptual framework Relationships between subjective and objective components Objective characteristics descriptive / background components Subjective characteristics evaluative
Defining the conceptual framework Relationships between subjective and objective components Objective living conditions Subjective well-being Social and economic development Quality-of-life improvement
Defining the conceptual framework Relationships between subjective and objective components Comparison of objective conditions Subjective well-being Standards of comparison previous experiences with other people with aspirations Ambits of comparison Housing Work Family Friends
Defining the conceptual framework Relationships between subjective and objective components Comparison of objective conditions Subjective well-being smaller the perceived gap higher the subjective well-being through different comparators with reference to different ambits (housing, work, family, friends, etc.).
Defining the conceptual framework Relationships between subjective and objective components Multiple discrepancies approach Subjective well-being perceived gap between what one has wants what others have one has had in the past one expected to have one expected to deserve expected with reference to needs happiness not dependent on living conditions
Defining the conceptual framework Relationships between subjective and objective components Disposition approach Stable individual characteristics (personality traits) Subjective well-being
Defining the conceptual framework Relationships between subjective and objective components Causal approach (I) Subjective well-being = reactive state to the environment bottom-up The sum of the reactive measures for the defined ambits allows subjective well-being to be quantified
Defining the conceptual framework Relationships between subjective and objective components Causal approach (II) Individual stable traits Subjective well-being top-down
Defining the conceptual framework Relationships between subjective and objective components Causal approach Subjective well-being Two components : a long-period component (top-down effect), a short-period component (bottom-up effect) up-down
Hierarchical design CONCEPTUAL MODEL HIERARCHICAL DESIGN AREAS TO BE INVESTIGATED LATENT VARIABLES ELEMENTARY INDICATORS
Hierarchical design COMPONENT THE QUESTION TO BE ANSWERED DEFINITION OF THE COMPONENT CONCEPTUAL MODEL Phenomena to be studied defines the phenomenon to be studied, the domains and the general aspects characterizing the phenomenon Process of abstraction AREAS/PILLARS TO BE INVESTIGATED LATENT VARIABLES ELEMENTARY INDICATORS (E.I.)
Hierarchical design COMPONENT THE QUESTION TO BE ANSWERED DEFINITION OF THE COMPONENT CONCEPTUAL MODEL AREAS/PILLARS TO BE INVESTIGATED Aspects defining the phenomenon Each area represents each aspect allowing the phenomenon to be specified consistently with the conceptual model LATENT VARIABLES ELEMENTARY INDICATORS (E.I.)
Hierarchical design COMPONENT THE QUESTION TO BE ANSWERED DEFINITION OF THE COMPONENT CONCEPTUAL MODEL AREAS/PILLARS TO BE INVESTIGATED LATENT VARIABLES Elements to be observed Each variable represents each element that has to be observed in order to define the corresponding area. The variable is named latent since is not observable directly ELEMENTARY INDICATORS (E.I.) theoretical assumptions (dimensionality) empirical statements Their definition requires:
Hierarchical design COMPONENT THE QUESTION TO BE ANSWERED DEFINITION OF THE COMPONENT CONCEPTUAL MODEL AREAS/PILLARS TO BE INVESTIGATED LATENT VARIABLES ELEMENTARY INDICATORS (E.I.) In which way each element has to be measured Each indicator (item, in subjective measurement) represents what is actually measured in order to investigate each variable. appropriate techniques They are defined by: a system allowing observed values to be interpreted and evaluated
Hierarchical design conceptual model area I area II area III area... variable 1 variable 2 variable... variable... variable... variable... E.I. a E.I. b E.I.... E.I.... E.I.... E.I....
Hierarchical design The hierarchical design is completed by the definition of relationships between latent variables and corresponding indicators (model of measurement) latent variables (managing the complexity)
Hierarchical design Two different conceptual approaches: models with reflective indicators models with formative indicators
Hierarchical design Models with reflective indicators indicators functions of the latent variable changes in the latent variable are reflected in changes in the observable indicators top-down explanatory approach
Hierarchical design Models with formative indicators indicators causal in nature changes in the indicators determine changes in the definition / value of the latent variable bottom-up explanatory approach
2. 1. Developing indicators 2. Developing a system of indicators
Introduction Application of the hierarchical design system of indicators Not a collection of indicators Each indicator measures and represents a distinct aspect of the observed phenomenon
Introduction system of indicators key elements conceptual framework system architecture dimensions to be measured units to be monitored measuring and monitoring procedures
Introduction system of indicators characteristics Objectivity equal, comparable results Quantification quantitative values Efficiency and fidelity communication of results Economicity simple, standardized, up-to-datable information Generalization exportability of the system Joint development shared by all the actors
Introduction system of indicators formal criteria Comprehensiveness Consistency Non redundancy parsimoniousness
Functions of systems of indicators
Functions Functions can be seen in cumulative terms (each requires the previous) : Monitoring Reporting Forecasting Program/performance evaluation Accounting Assessment
Functions Monitoring capacity of the system to monitor changes over time and meet the need of improving knowledge Reporting Forecasting Program/performance evaluation Accounting Assessment
Functions Monitoring Reporting monitoring + analysis + interpretation Forecasting Program/performance evaluation Accounting Assessment
Functions Monitoring Reporting Forecasting trends in observed reality Program/performance evaluation Accounting Assessment
Functions Monitoring Reporting Forecasting Program/performance evaluation problem definition, policy choice and evaluation of alternatives and program monitoring Accounting Assessment
Functions Monitoring Reporting Forecasting Program/performance evaluation Accounting supporting decision concerning the allocation and the destination of resources Assessment
Assessment certificate or judge subjects (individuals or institutions) by discriminating their performances or to infer functioning of institutions, enterprises or systems. Functions Monitoring Reporting Forecasting Program/performance evaluation Accounting
Elements defining a system of indicators
Elements i. Aims ii. Structure iii. Analytical approaches iv. Interpretative and evaluating models
Elements i. Aims Conceptual goals Operative objectives Planning actions indicators function - input measuring resources available in the system and indicating some sort of inputs into a process - process monitoring the basic progress of implementing the actions defined (intermediate output) and outlined at strategic level - output/outcome monitoring direct results of actions - impact monitoring progress and improvement towards goals and objectives achievement
Elements ii. Structure Vertical from local to higher level Horizontal different ambits at the same level Local monitoring internal organization of the level and referring to external parameters
Elements iii.analytical approaches consistently with aims and structure
Elements iv.interpretative and evaluating models according to a specific reference frame. Possibility to define and identify particular standard-values, which can be defined a priori, according to the objectives or empirical observations (e.g. surveys).
Characteristics of indicators within a system
Characteristics Classification Purposes Governance contexts Perspectives of observation Forms of observation Levels of communication
Characteristics Classification Purposes descriptive (describing a reality) explicative (interpreting a reality) predictive (identifying trends) normative (supporting decisions) problem oriented (testing hypotheses) evaluating practical directionable actionable (process advancement effect)
Characteristics Classification Governance context Public debates Policy guidance Administrative guidance
Characteristics Classification Perspective of observation Conglomerative approach measures capturing advances made by the society as a whole Deprivational approach assessing status of the deprived Both needed for an adequate understanding
Characteristics Classification Forms of observation Status indicators capturing a reality in a particular moment Trend indicators observing a reality along time (longitudinal design of observation)
Characteristics Classification Level of communication Cold indicators complex and difficult, for specialists Hot indicators simple and easy Warm indicators good balance between quality, comprehensibility and resonance
Characteristics Quality I. Methodological soundness II. Integrity III. Serviceability IV.Accessibility
Characteristics An indicator should be able to: Define and describe Observe unequivocally and stably Record by a degree of distortion as low as possible Adhere to the principle of objectivity Reflect adequately the conceptual model Meet current ad potential users needs Be observed through realistic efforts and costs Reflect the length of time between its availability and the event of phenomenon it describes Be analyzed in order to record differences and disparities Be spread (I) METHODOLOGICAL SOUNDNESS (II) INTEGRITY (III) SERVICEABILITY (IV) ACCESSIBILITY
Characteristics Prerequisity of quality Legal and institutional environment (coordination within and across institutions) Resources available for statistics work Quality awareness informing statistical work
Characteristics Problems in selecting indicators Different issues need to be addressed in selecting and managing indicators, into a complex system for monitoring, reporting and accounting
Characteristics Problems in selecting indicators Michalos (2006) identified 15 different issues related to the combination of social, economic and environmental indicators. The issues collectively yield over 200,000 possible combinations representing at least that many different kinds of systems:
Characteristics Problems in selecting indicators Settlement/aggregation area sizes Time frames Population composition Domains of life composition Objective versus subjective indicators Positive versus negative indicators Input versus output indicators Benefits and costs Measurement scales Report writers Report readers Quality-of-life model Distributions Distance impacts Causal relations
Analysis of indicators within a system
Analysis ANALYTIC APPROACH OBJECTIVE Monitoring to monitor developments of a specific condition (e.g. environmental conditions) Reporting to report the results as they are obtained in a hierarchical procedure of decision-making Trend to clarify development trend Evaluation to record and evaluate the effects of planned and performed initiatives and actions Benchmarking to compare between performances of the considered units (e.g. countries) Assessment to clarify the impacts of planned and undertaken initiatives and actions
Flight desk Needs to - Know destination ( aim) - Share destination ( democracy) - Know pre-conditions ( ) - Know flight conditions ( monitoring) - Communicate flight conditions - Know how to manage emergency -.
filomena.maggino@unifi.it Presentation designer: trapani.marco@gmail.com