Issues and Challenges around Integrating Qualitative and Quantitative Data James Thomas 1

Transcription

1 Issues and Challenges around Integrating Qualitative and Quantitative Data James Thomas 1 InCQuiRES Sheffield, 6 September 2012 (1) The EPPI-Centre is part of the Social Science Research Unit at the Institute of Education, University of London, UK 1

2 Acknowledgements Colleagues at the EPPI-Centre, SSRU, Institute of Education, London Angela Harden (Integrating qualitative and quantitative research in systematic reviews: Issues and challenges. Presentation 2010) (2)

3 Outline Why integrate different types of literature in a systematic review? Mixed methods primary research & systematic reviews Examples (3)

4 Example context: the DH health promotion & public health reviews facility Since 1995 our aim has been to address the needs of DH Policy Research Programme: DH policy teams Knowledge base of high quality research for policies directed at improving population health and wellbeing and reducing inequalities Ensure that policy decisionmaking can be informed by all available and robust scientific evidence Department of health policy research programme Reviews facility (us) (4)

5 Early work Reviews of effectiveness in health promotion ( transferring and translating the model of evidence-based health care to HP & PH ) Sexual health; smoking; workplace Policy teams were positive, but wanted more assistance than a straight translation could offer (5)

6 What is the model we were translating? Typical caricature of Cochrane reviews: Narrow in scope and range of included methods Aim to answer a single question with a single answer (6)

7 Traditional /Cochrane approach to knowledge use SRS Treatment A v Control SRS Treatment B v Control What the most effective treatment for condition X SRS Treatment C v Control (7)

8 But Policymakers do not come to us with a single narrow question They begin with a particular problem More work is done in a single review Aggregation of many focused single reviews is possible but problematic Gaps / non- commensurate / difficult to tailor for a particular situation (8)

9 Principles Policy and practice concerns often precede, or go beyond, questions of effectiveness. They also ask does it vary according to? What does... mean to...? Different types of questions require different combinations of study types to be included. E.g. process evaluations and trials Different combinations of study types demand different methods of synthesis. E.g. synthesis of correlational and qualitative studies (9)

10 Complex reviews of social research: Start from a given problem (often a known outcome and population) Aim to identify a range of possible solutions (i.e. interventions) Rarely aim to come to a single answer Acknowledge that there are rarely replications of interventions in social research Use heterogeneity to better understand factors that influence the impact of interventions Contain detailed and complex conceptual frameworks Often require methodological development (10)

11 Drivers for integration - summary Greater recognition of the value of qualitative research in evidence based health care Empty reviews Patient perspectives and experiences Systematic reviews of complex interventions Issues of process and implementation Extension of evidence-based health care to other areas of public policy Growing tradition for integration in primary research (mixed methods research) Dedicated research funding for methodological research Establishment of dedicated methods groups (e.g. Cochrane Qualitative Methods Research Group) (From presentation by Angela Harden 2010) (11)

12 Challenges of integration New and evolving field Scale of task Lack of training and expertise Paradigm wars Lack of worked examples Innovation required (From presentation by Angela Harden 2010) (12)

13 Designs are most effective and have the greatest potential for innovation when they are dictated by the nature of the research problem. W. Paul Vogt (2008) The Dictatorship of the Problem: Choosing Research Methods. Methodological Innovations online, 3 (1) (13)

14 Reviews & mixed methods research Reviews that answer practical policy / practice questions can find they need to integrate qualitative / quantitative data The substantial body of literature on mixed methods in primary research is a useful resource (14)

15 What is Mixed Methods Research? Mixed methods research is formally defined here as the class of research where the researcher mixes or combines quantitative and qualitative research techniques, methods, approaches, concepts or language into a single study. Mixed methods research also is an attempt to legitimate the use of multiple approaches in answering research questions, rather than restricting or constraining researchers choices (i.e., it rejects dogmatism). It is an expansive and creative form of research, not a limiting form of research. It is inclusive, pluralistic, and complementary, and it suggests that researchers take an eclectic approach to method selection and the thinking about and conduct of research. (pp ) Johnson and Onwuegbuzie (2004) (15)

16 The fit between mixed methods & EIP&P (1)... developing a mixed method strategy fits with the political currency accorded to practical enquiry that speaks to policy and policymakers and that informs practice (Hammersley 2000), while scientific research may require closer attention to and justification of the methods used and the types of data generated in reaching conclusions. [Disciplinary / paradigm specific vs interdisiplinary?] In Britain we have seen a whole industry of mixed method research created around evidence based policy and over a longer time frame in the evaluation of policy (Ritchie 2003; Tashakorri and Teddlie 2003a). However the downside to this is that researchers have less and less leeway to define their own research questions and to follow their own ideas. (16) From: Brannen J (2006) Mixed methods research: a discussion paper. NCRM Methods Review Papers, NCRM/005

17 The fit between mixed methods & EIP&P (2) Some social scientists are concerned with generating understandings at the micro level while others are concerned with the macro level. Micro: emphasise the agency of those they study through an emphasis upon studying subjective interpretations and perspectives Macro: concerned with larger scale patterns and trends and seek to pose structural explanations. If one is to transcend conceptually the micro and the macro levels then methods must be developed to reflect this transcendence (Kelle 2001) (17) From: Brannen J (2006) Mixed methods research: a discussion paper. NCRM Methods Review Papers, NCRM/005

18 Purposes of combining different sorts of data / results Triangulation (?) Elaboration or expansion Initiation Complementarity Contradictions Contextualisation (18)

19 Combining data Combining different types of data requires some means of enabling them to speak to one another. Typical techniques include: Matrices Qualitising numeric data Quanititising qualitative data Another key question is the order and relative weight given to each type of data (19)

20 Taxonomy of paradigm stances (Creswell 2011) Incommensurability (cannot be mixed) A-paradigmatic (can be mixed and matched in different ways) Complementary strengths (not incompatible, but are different and should be kept separate) Dialectic (paradigms are important in different ways leading to useful tensions & insights) Alternative paradigm ( mixed methods paradigm; foundation in e.g. pragmatism) (20)

21 Mixed methods paradigms and methods /approaches to synthesis Review method / approach Developed by / example Bayesian synthesis Roberts et al (2002) Voils C, et al (2009) Leeman J et al (2011) Narrative synthesis Popay et al (2006) Arai et al (2007) Noyes & Popay Mixed methods synthesis Thomas et al (2004) Harden & Thomas (2005 / 2010) Candy et al (2011) Critical interpretive synthesis Dixon-Woods et al (2006) Mixed methods paradigm perspective Alternative or single paradigm stance Complementary strengths stance Complementary strengths stance Dialectical stance Alternative or single paradigm stance Meta narrative review Greenhalgh et al (2006) Dialectical stance (21) Realist synthesis Pawson (2006) Alternative or single paradigm stance

22 Configuration & aggregation New (ish) work in SRs has argued that the qualitative / quantitative binary divide conceals more than it reveals Suggests a better heuristic is aggregate / configure Voils CI, Sandelowski M, Barroso J, Hasselblad V: Making sense of qualitative and quantitative findings in mixed research synthesis studies. Field Methods 2008, 20:3 25. Sandelowski M. Voils CJ, Leeman J, Crandlee JL (2011) Mapping the Mixed Methods-Mixed Research Synthesis Terrain Journal of Mixed Methods Research Gough D, Oliver S, Thomas J (2012) An Introduction to Systematic Reviews. London: Sage Gough D; Thomas J; Oliver S (2012) Clarifying differences between review designs and methods. Systematic Reviews. 1(28) (22)

23 Aggregation in reviews Aggregation refers to adding up (aggregating) findings from primary studies to answer a review question to indicate the direction or size of effect and our degree of confidence in that finding (23)

24 Configuration in reviews Configuration involves the arrangement (configuration) of the findings of primary studies to answer the review question. to offer a meaningful picture of what research is telling us... across a potentially wide area of research (24)

25 Philosophy: Relation to theory: Approach to synthesis: Methods: Quality assessment: Product: Idealist Idealist Generate Explore Test Test Configuring Iterative Theoretical search Value study contributions Emergent concepts Realist Aggregating A priori Exhaustive search Avoid bias Magnitude & precision Review use: Enlightenment Instrumental

26 For example - Meta-analysis Heterogeneity Aggregation / configuration A problem in fixed effect Allowance made for in random effects For exploration Fixed & random effects models are pure aggregation Sub-group & metaregression both aggregate & configure Synthesis of qualitative research Expected. Key challenge is judging when contexts / concepts are too dissimilar to be compared. Similar concepts are aggregated ; most synthesis involves configuration (26) Realist review Meta narrative review Necessary to obtain different contexts etc Necessary to obtain diversity in paradigms Aggregates within CMO configurations; configures between them Aggregation within paradigms; configuration between them Mixed methods review Necessary and desirable Aggregation within types, configuration between AND aggregation and configuration across data types

27 (27) EXAMPLES

28 Example 1. using a matrix to combine two syntheses How might a systematic review of qualitative research support or add insight to a metaanalysis? Product of qualitative synthesis can form the conceptual framework within which heterogeneity can be explored in a meta-analysis Thomas J, Harden A, Oakley A, Oliver S, Sutcliffe K, Rees R, Brunton G, Kavanagh J (2004) Integrating qualitative research with trials in systematic reviews: an example from public health. British Medical Journal 328: ( (28)

29 Mixed method reviews Review question e.g. What is known about the barriers to, and facilitators of, fruit and veg intake amongst children aged 4 to 10 years? MAPPING (193 studies in 272 reports) Trials (N=33) 1. Application of inclusion criteria 2. Quality assessment 3. Data extraction 4. Statistical meta-analysis Views studies (N=8) 1. Application of inclusion criteria 2. Quality assessment 3. Data extraction 4. Thematic synthesis Trials and views Mixed methods synthesis (29)

30 Cross study synthesis via a matrix Children s views Outcome evaluations Recommendation for interventions Do not promote fruit and vegetables in the same way Brand fruit and vegetables as an exciting or childrelevant product, as well as a tasty one Reduce health emphasis in messages to promote fruit and vegetables particularly those which concern future health Good quality No soundly evaluated interventions 5 soundly evaluated interventions identified 5 soundly evaluated interventions identified Other No other interventions identified 5 other interventions 6 other interventions identified (30)

31 Portions Cross study synthesis: an example of sub-group analysis Increase (standardised portions per day) in vegetable intake across trials 0.6 Little or no emphasis on health messages Wardle Liquori Henry Anderson Reynolds Auld Auld (b) Baranowski Perry Study (31)

32 Alternative analytical approach to using a matrix Candy B, King M, Jones L, Oliver S (2011) Using qualitative synthesis to explore heterogeneity of complex interventions. BMC Medical Research Methodology. 11(124) Rather than partitioning variance in a standard meta-analysis, correspondence is explored using Qualitative Comparative Analysis (QCA) (32)

33 This method of synthesis across study types: (33) preserves the integrity of the findings of the different types of studies allows us to integrate quantitative estimates of benefit and harm with qualitative understanding from people s lives allows the exploration of heterogeneity in ways in which it would be difficult to imagine in advance protects against data dredging

34 Example 2: Bayesian approaches Roberts et al (2002) Factors affecting uptake of childhood immunisation: a Bayesian synthesis of qualitative and quantitative evidence. Lancet 360: Qual -> quant Voils C, Hassselblad V, Crandell J, Chang Y, Lee E, Sandelowski M (2009) A Bayesian method for the synthesis of evidence from qualitative and quantitative reports: the example of antiretroviral medication adherence. J Health Serv Res Policy. 14(4): Quant qual Leeman J, Chang Y, Voils CI, Crandell JL, Sandelowski M (2011) A Mixed-Methods Approach to Synthesizing Evidence on Mediators of Intervention Effects West J Nurs Res 33: Data augmentation (34)

35 Bayesian mixed methods synthesis (1) Roberts et al (2002) Factors affecting uptake of childhood immunisation: a Bayesian synthesis of qualitative and quantitative evidence. Lancet 360: Bayesian statistics: different perspective Synthesis is part of decision-making process Evidence is not approached in isolation Pre-existing beliefs, subjective judgement and other evidence all shape interpretation Structure of analysis is similar to the previous mixedmethods example Key factors thought to influence the uptake of childhood immunisation were identified from experts and qualitative literature Quantitative studies (mostly surveys) were examined to assess the extent that they met the above factors Bayesian meta-regression models were used in order to assess whether each factor was a determinant (35)

36 Prior probabilities Pre-existing subjective beliefs of 5 researchers were combined with qualitative evidence Rankings for each factor were combined to form prior probabilities (36)

37 How the results were generated The prior probabilities were combined with probabilities arising from the quantitative studies Using the qualitative studies enabled the quantitative results to be modified and also suggested variables which were not identified as being important by the quantitative studies (37)

38 Bayesian mixed methods synthesis (2 & 3) Voils C, et al (2009) A Bayesian method for the synthesis of evidence from qualitative and quantitative reports: the example of antiretroviral medication adherence. J Health Serv Res Policy. 14(4): Key differences between this and Roberts et al: equal treatment of qual & quan; Qualitised the statistical studies Leeman J et al (2011) A Mixed-Methods Approach to Synthesizing Evidence on Mediators of Intervention Effects West J Nurs Res 33: Approach to data augmentation (38)

39 Summary of examples Transformation method Ordering MM Paradigm Aggregation / configuration Example 1 Matrix Qual -> quant Dialectical Both complex (explain!) Example 2 Quantitising Qualitising Quantitising Qual -> quant Equal Qual -> quant Alternative or single paradigm stance No qual synthesis, but still complex (39)

40 References Brannen J (2006) Mixed methods research: a discussion paper. NCRM Methods Review Papers, NCRM/005 Creswell J (2011) Mapping the Developing Landscape of Mixed Methods Research in Teddlie C, Tashakkori A: SAGE Handbook of Mixed Methods in Social & Behavioral Research. New York: Sage. Harden A, Thomas J (2005) Methodological issues in combining diverse study types in systematic reviews. International Journal of Social Research Methodology. 8(3): Harden A, Thomas J (2010) Mixed methods and systematic reviews: examples and emerging issues. In: Tashakkori A, Teddlie C SAGE Handbook of Mixed Methods in Social & Behavioral Research. pages Johnson RB & Onwuegbuzie AJ (2004). Mixed methods research: A research paradigm whose time has come. Educational Researcher, 33, Thomas J, Kavanagh J, Tucker H, Burchett H, Tripney J, Oakley A (2007) Accidental injury, risk-taking behaviour and the social circumstances in which young people live: a systematic review. London: EPPI-Centre, Social Science Research Unit, Institute of Education, University of London. (40)