CRITICAL APPRAISAL AP DR JEMAIMA CHE HAMZAH MD (UKM) MS (OFTAL) UKM PHD (UK) DEPARTMENT OF OPHTHALMOLOGY UKM MEDICAL CENTRE

Transcription

1 CRITICAL APPRAISAL AP DR JEMAIMA CHE HAMZAH MD (UKM) MS (OFTAL) UKM PHD (UK) DEPARTMENT OF OPHTHALMOLOGY UKM MEDICAL CENTRE MINGGU PENYELIDIKAN PERUBATAN & KESIHATAN PPUKM

2 Lecture content Introduction What is critical appraisal? Steps of critical appraisal

3 Introduction Continuous medical education is essential part in a doctor s life Acquiring up-to-date knowledge and skills for clinical practice Providing patients with the best possible standard of care Revalidation and appraisal of competency

4 However Information overload Number of new research articles published each year are continually increasing > than 12,000 new articles per year RCTs are added to the MEDLINE database each week

5 Evidence-based medicine Findings of scientific research should be apply to individual patients as part of a doctor s clinical decision-making process Therefore, doctors must be able to: Select and appraise scientific literature that is relevant to their field Understand the implications of research findings for individual patients Elicit patients' own preferences Develop an appropriate management plan based on the combination of this information

6 Bad science In June 2008, the Sunday Express published an article about the link between suicides and phone masts The increase in deaths among young people in Britain s suicide capital could be linked to radio waves from dozens of mobile phone transmitter masts near the victims homes. Dr Roger Coghill, who sits on a Government advisory committee on mobile radiation, has discovered that all youngsters who have killed themselves in Bridgend, South Wales, over the past 18 months lived far closer than average to a mast (Johnston 2008) Ben Goldacre, a medical doctor and author of the weekly Bad Science column in the Guardian, investigated the claim made by the Sunday Express article and found out the following: I contacted Dr Coghill, since his work is now a matter of great public concern, and it is vital his evidence can be properly assessed. He was unable to give me the data. No paper has been published. He himself would not describe the work as a study. There are no statistics presented on it, and I cannot see the raw figures. In fact Dr Coghill tells me he has lost the figures. Despite its potentially massive public health importance, Dr Coghill is sadly unable to make his material assessable. (Goldacre 2008)

7 Behind the headlines Some news are genuinely based on valid studies, but jump to wrong conclusions by failing to consider some important aspects, such as the study design and the level of evidence of the original research. In July 2008, an article was published on the Daily Mail claiming that there is a link between vegetarian diet and infertility (Daily Mail Reporter 2008). The article was based on a cross-sectional study on soy food intake and semen quality published in the medical journal Human Reproduction (Chavarro et al. 2008) Behind the Headlines, A study reported that eating tofu can significantly lower your sperm count Limitation of the study: it was small, and mainly looked at overweight or obese men who had presented to a fertility clinic. It focused only on soy (soya) intake, Daily Mail s claim that there is a causal link between eating a vegetarian diet and reduced fertility is misleading. (NHS Knowledge Service 2008)

8 Bias in the location and selection of studies If a study had not obtained positive results, would it have been published - and quoted in the news? When reviewing the literature published in scientific/medical journals, we should consider that papers with significant positive results are more likely to submitted and accepted for publication (publication bias); published in a major journal written in English (Tower of Babel bias); published in a journal indexed in a literature database, especially in less developed countries (database bias); cited by other authors (citation bias); published repeatedly (multiple publication bias); and quoted by newspapers! (Egger & Smith 1998; Gregoire, Derderian, & Le Lorier 1995)

9 Hierarchy of evidence

10 Lecture content Introduction What is critical appraisal? Steps of critical appraisal

11 Critical appraisal Defined as "...application of rules of evidence to a study to assess the validity of the data, completeness of reporting, methods and procedures, conclusions, compliance with ethical standards, etc. The rules of evidence vary with circumstances. Last JE (Ed.; 2001) A Dictionary of Epidemiology (4th Edn). New York: Oxford University Press

12 Con t A systematic process used to identify the strengths and weaknesses of a research article whether to use the results of a study in clinical practice No 'gold-standard' instrument for critical appraisal Criteria are not static Different study designs are prone to various sources of systematic bias Increased awareness of the potential influence of other non methodological factors, such as conflicts of interest

13 Lecture content Introduction What is critical appraisal? Steps of critical appraisal

14 Ten steps of critical appraisal Is the study question relevant? Does the study add anything new? What type of research question is being asked? Was the study design appropriate for the research question? Did the study methods address the most important potential sources of bias? Was the study performed according to the original protocol? Does the study test a stated hypothesis? Were the statistical analyses performed correctly? Do the data justify the conclusions? Are there any conflicts of interest?

15 Is the study s research Q relevant? Relevant to the researcher s own field of work Addresses an important topic and adds to what is already known about that subject Subjective It might be crucial to some, but irrelevant to others

16 Does the study add anything new? New ideas and knowledge are developed on the basis of previous work Standing on the shoulders of giants Two types of work: Study that make a substantive new contribution to knowledge - rare Study that makes an incremental advance can also be of value E.g. a study might increase confidence in the validity of previous research by replicating its findings, or might enhance the ability to generalize a study by extending the original research findings to a new population of patients or clinical context

17 What type of research Q do the study pose? The most fundamental task Identifying the specific research question that an article addresses - this will determine the optimal study design and have a major bearing on the importance and relevance of the findings Well-developed research question usually identifies 4 components: Population Intervention Comparator Outcomes

18 Con t Clinical research questions fall into 2 categories: 1. Questions about the effectiveness of treatment Relate to whether one treatment is better than another in terms of clinical effectiveness (benefit and harm) or cost-effectiveness 2. Questions about the frequency of events Refer to the incidence or prevalence of disease or other clinical phenomena, risk factors, diagnosis, prognosis or prediction of specific clinical outcomes and investigations on the quality of health care

19 Was the study design appropriate for the research question? Study design Study design Qualitative Quantitative Document Clinical trial Observational Passive observation Randomised Cohort Participant observation In depth interview Non randomised Case-control Cross sectional Focus group

20 Cross sectional Measures prevalence of health outcomes/determinants of health/both in a population at a point in time Strengths Allows comparison of many different variables at the same time Limitation Unable to provide definite information about cause-andeffect relationships

21 Case control Strengths Relatively economical and quick. Ideal for studying rare diseases or diseases that can be defined specifically Able to examine the association between multiple exposures and the disease of interest Limitation Unable to examine multiple diseases that might be associated with an exposure Selection of appropriate controls can be challenging Recall bias Cannot be used absolute risk for disease after an exposure (uses OR)

22 Cohort A longitudinal study (prospective or retrospective) Strengths For examining disease after rare and highly specific exposures Able to monitor the occurrence of multiple diseases potentially caused by an exposure Allows direct measurement of the absolute risk of developing a disease after an exposure Limitation Selection bias and confounding factor Can only examine a single exposure or group of related exposures Cannot be used if the population at risk is not known

23 Randomised controlled trial (RCT) Gold standard of study design Most rigorous way of: determining whether a causeeffect relation exists between treatment and outcome for assessing the cost effectiveness of a treatment Sequence generation - a rule to allocate interventions to participants Limitations: Ethical and practical concerns More costly and time consuming than other studies Allocation concealment secure implementation of the schedule of random assignment

24 Did the study methods address the key potential sources of bias? Bias deviation from the truth Can be attributed to: chance (e.g. a random error) affect the precision of the study does not influence the results in any particular direction to the study methods (systematic bias) e.g. how the study is conducted, be it how study participants were selected, how data was collected, or through the researchers' analysis or interpretation. results in the overestimation or underestimation of the 'truth has a direction

25 Type of bias Type of bias Selection bias Description Systematic differences between baseline characteristics of the groups that are compared. Performance bias Systematic differences between groups in the care that is provided, or in exposure to factors other than the interventions of interest. Detection bias Attrition bias Reporting bias Systematic differences between groups in how outcomes are determined. Systematic differences between groups in withdrawals from a study. Systematic differences between reported and unreported findings. **Different study designs are prone to varying sources of systematic bias

26 Eg. In RCT:

27 Was the study performed in line with the original protocol? Deviations from the planned protocol can affect the validity or relevance of a study Failure to recruit the planned number of participants Sampling bias - those who actually were recruited might be different from those who weren't for some reason Reduces the power of the study changes to the inclusion and exclusion criteria Variation in the provided treatments or interventions changes to the employed techniques or technologies Changes to the duration of follow-up

28 Does the study test has a stated hypothesis? Study hypotheses must be identified a priori Developed from theory or previous experience Post-hoc analysis produces high false positive findings 1 in 20 associations tested will be significant (positive) by chance alone at significance level of 5% (P = 0.05)

29 Con t Sample size depends on hypothesis Whether it is a equivalence or noninferiority trial vs. a superiority trial Check all data relevant to the stated study objectives have been reported, and that selected outcomes have not been omitted

30 Were the statistical analyses performed correctly? Tools used in the statistical analysis and the rationale for this approach should be included in the 'Method' section Approach to dealing with missing data and the statistical techniques applied In the 'Results' section, Defaulted or lost in follow-up and missing data should be clearly identified

31 Do the data justify the conclusions? Whether the conclusions are based on the accumulated data Overemphasis - Statistically significant findings are too small to be of clinical value Under emphasis - Dismissing large and potentially important differences between groups that are not statistically significant, because of small sample size Generalisation of findings to broader groups of patients or contexts than was reasonable given in their study sample Whether statistically significant associations have been misinterpreted to imply a cause and effect

32 Are there any conflicts of interest? Conflicts of interest occur when personal factors have the potential to influence professional roles or responsibilities Decision such as which studies will be conducted in their unit, which patients will be invited to participate in a study and whether certain clinical occurrences should be reported as adverse events affects validity of study These decisions require researchers to act with integrity and not for personal or institutional gain Check for declaration about the source of funding for the study and, if a potential conflict of interest had been identified for a statement about how this conflict was managed. Judge whether the declared factors are important and might have influenced the validity of the study's findings

33 Evidence based medicine Formulate your questions into a format according to PICO tool (Population, Intervention, Comparison, Outcome) to easily do literature review Literature review - if you can find a pre-appraised resource (e.g. systematic review), you can miss out the next step Critical appraisal of your results Decide what action to take from your findings Evaluate your new or amended practice

34 Conclusion Critical appraisal is a systematic process through which the strengths and weaknesses of a research study can be identified This process enables us to assess the study's usefulness and whether its findings are trustworthy The most important component of critical appraisal is careful assessment of the study design (Don t forget evaluation of the statistical methods used, interpretation of the findings and potential conflicts of interest) Consideration of the importance of the research to our own patients will help us identify the most relevant, high-quality studies available to guide in our clinical practice