Establishing confidence in qualitative evidence. The ConQual Approach

Transcription

1 Establishing confidence in qualitative evidence The ConQual Approach

2 Objectives To review the JBI conceptualisation of what constitutes evidence for decision making To provide an overview of the JBI approach to Levels of Evidence (LoE) and Grades of Recommendation (GoR) To refresh your knowledge of the JBI approach to qualitative systematic review (Meta aggregation) To overview the JBI approach to establishing confidence in the findings of qualitative systematic reviews (ConQual Approach) To practically apply the ConQual approach to qualitative systematic review findings

3 What is evidence? What constitutes evidence to inform practice

4 JBI and Evidence

5 What is evidence? the basis of belief; the substantiation or confirmation that is needed in order to believe that something is true (Pearson et al 2005) In healthcare this relates not just to effectiveness, but to feasibility, appropriateness and meaningfulness (i.e. different questions require different research evidence to answer them)

6 FAME Feasibility (practicality or viability) Appropriateness ( fit with context or setting) Meaningfulness (patient/provider experience) Effectiveness (achieves the intended result)

7 Levels of Evidence and Grades of Recommendations

8 Is all evidence created equal? If not, how do we determine what good evidence is? How can we be confident that the evidence we are using is the best?

9 Current JBI Levels of Evidence Levels of evidence for: Diagnosis

10 Current JBI Levels of Evidence Levels of evidence for: Diagnosis Prognosis

11 Current JBI Levels of Evidence Levels of evidence for: Diagnosis Prognosis Economic Evaluations

12 Current JBI Levels of Evidence Levels of evidence for: Diagnosis Prognosis Economic Evaluations Effectiveness

13 Current JBI Levels of Evidence for Meaningfulness

14 Grades of Recommendation Grades of Recommendation are used to assist healthcare professionals when implementing evidence into practice. Recommendations assigned a grade

15 JBI Grades of recommendations

16 Group Discussion What questions might these GoR leave unresolved?

17 Group Discussion - Answers What questions might these GoR leave unresolved? Quality of the primary research Strength of primary research findings Level of confidence in primary research findings Degree to which primary research findings represent the participants voices Degree to which the primary research findings are congruent with each other

18 Current JBI GoR JBI and collaborating entities currently assign a Grade of Recommendation to all recommendations made in its resources. These Grades are intended to be used alongside the supporting document outlining their use.

19 Advice

20 Implications for HDR All systematic reviews must have recommendations for practice with a GRADE assigned in their systematic reviews

21 Systematic Review of Qualitative Evidence Knowledge refresher of the JBI Approach: Meta aggregation

22 JBI and Qualitative Evidence Systematic review of qualitative evidence important: To generate evidence for practice in a systematic, transparent and robust way To provide recommendations for policy or practice arising from review findings

23 JBI and Qualitative Evidence The JBI approach = Meta aggregation Meta aggregation focuses on study findings not study data This means differing methodologies (e.g. phenomenology, ethnography or grounded theory), using different methods, can be mixed in a single synthesis of qualitative studies as long as they focus on the same phenomena of interest

24 Meta-aggregation Aim of meta aggregation: is to assemble findings categorize these findings into groups on the basis of similarity in meaning aggregate these to generate a set of statements that adequately represent that aggregation

25 Defining Findings A finding is a verbatim extract of the authors analytic interpretation accompanied by either a participant voice, or fieldwork observations or other data. Illustration: Direct quotation of participant voice, field work observations or other supporting data

26 Example Finding/Illustration Finding Illustrations From: Chase et al 1997

27 Assigning a level of credibility to findings Unequivocal findings accompanied by an illustration that is beyond reasonable doubt and therefore not open to challenge Credible findings accompanied by an illustration lacking clear association with it and therefore open to challenge Not supported when 1 nor 2 apply and when most notably findings are not supported by the data. Should not be included in synthesis to inform practice

28 Defining Categories Categorization involves repeated, detailed examination of the assembled findings The reviewer identifies groups of findings on the basis of similarity in meaning to create categories an explanatory statement is created by the reviewer to assist interpretation and understanding of the context, promoting auditability

29 Defining synthesized findings In meta aggregation a synthesized finding is an overarching description of a group of categorized findings that allow for the generation of recommendations for practice Synthesized findings are expressed as indicatory statements that can be used to generate recommendations for policy or practice.

30 Establishing confidence in qualitative evidence

31 Confidence in the evidence Confidence is defined as the belief, or trust, that a person can place in the results of the research Considering that our Synthesised Findings are statements that should assist in informing practice / decision making: How confident are we that the quality of evidence supports a particular decision or recommendation?

32 What increases or decrease our confidence in the results of a SR? Quantitative Qualitative

33 What increases or decrease our confidence in the results? Quantitative Study design Risk of bias Indirectness Heterogeneity Imprecision Publication bias Effect size Plausible bias Dose response effect Qualitative

34 What influences our confidence in the results of a SR? Quantitative Study design Risk of bias Indirectness Heterogeneity Imprecision Publication bias Effect size Plausible bias Dose response effect Qualitative Type of research Dependability Credibility

35 Analogous criteria for paradigmatic assumptions Quantitative Qualitative Reliability Dependability Internal Validity Credibility External Validity Transferability

36 Dependability [Reliability] Appropriateness of methodology, methods and implementation of the research methods, regardless of paradigm The focus of dependability is on achieving consistent quality rather than repeatability Should be logical, traceable and clearly documented

37 Credibility [Internal validity] Credibility addresses whether a finding has been represented correctly Assessment of credibility is multi dimensional, including goodness of fit and representativeness Credibility is auditable the process may be based upon researcher confirmation, member checks, peer checks, second researcher analysis, or observation

38 Transferability [External validity] Findings are not generalizable in the quantitative sense of the word generalization is narrowly conceived in terms of sampling and statistical significance. qualitative research is directed toward naturalistic or idiographic generalizations, or the kind of generalizations made about particulars Schofield (1990) describe qualitative metasynthesis as cross-case generalizations created from the generalizations made from, and about, individual cases. Sandelowski et al(1997)

39 ConQual Approach

40 ConQual

41 The ConQual Approach Rankings of confidence can be: High Moderate Low Very Low All findings start off as high System allows synthesized findings to be downgraded based on the dependability and credibility of individual findings

42 The ConQual Approach: Dependability

43 The ConQual Approach: Credibility

44 Summary of Findings table Includes: Context Synthesized Findings Type of research Dependability Score Credibility Score ConQual Score Reasons behind decisions

45 Qualitative SoF Table Systematic review title: Population: Phenomena of interest: Context: Synthesised Finding Type of research Dependability Credibility ConQual* Comments

46 Qualitative SoF Table Systematic review title: A comprehensive systematic review on the experience of lateral/horizontal violence in the profession of nursing Population: Licenced and Student Nurses in any setting Phenomena of interest: The experience of LV/HV in the profession of nursing Context: Nurses in any practice setting Synthesised Finding Type of research Dependability Credibility ConQual* Comments There is a wide array of coping behaviors that are employed, either successfully or unsuccessfully, by victims of lateral/horizontal violence. In more cases than not, people just coped the best way they could. Qualitative (HIGH) (MODERATE) (LOW) LOW

47 JBI Reviews > It is recommended that all JBI systematic reviews of qualitative evidence include a summary of findings table. These should appear following the executive summary in JBI systematic reviews in the appropriate format

48 Activity Determining the ConQual Score

49

50 Synthesised Finding 1 Organizational influences Enabling nurses to demonstrate accountability for their own learning, along with clear organizational systems that provide resources, time, adequate staffing and support, demonstrates encouragement for and the value of nurses learning and education. Type of Research Dependibility Credibility CONQUAL Score Comments

51 Synthesised Finding 2 Relational dynamics Nurses value their peers, expert nurses, preceptors, mentors and educators facilitating and encouraging their learning and professional development. Type of Research Dependibility Credibility CONQUAL Score Comments

52 Summary of Findings Table

53 Summary The main reasons for continuing with a Levels of Evidence system are as follows: To assist in assigning a pre rankings to studies when conducting systematic reviews. For resources such as evidence summaries which require a rapid review and classification of literature (for example, the Levels of Evidence can provide information on the most appropriate study design to search for when asking a clinical question). For educational purposes for health professionals.

54 Summary GoR are helpful, but arguably are the equivalent of surrogate outcome measures they tell part of the story, not the full story

55 Summary Establish confidence in the findings of qualitative evidence Study Design Dependability Credibility

56 Useful tips to remember Clearly document and record your process Clearly record the studies that inform the synthesised finding This makes it easier to rank Dependibility Clearly record the credibility ranking for each of the findings that inform the synthesised finding This makes it easier to rank Credibility