Are those results meaningful?

Size: px

Start display at page:

Download "Are those results meaningful?"

Scot Harvey
5 years ago
Views:

Evidence based practice Are those results meaningful? How to understand and apply the results of studies in optometric practice.

based 1 Dr Lisa Nivison-Smith 1 NHMRC. 1999 A guide to the development, implementation and evaluation of clinical practice guidelines.

papers estimated to be out-of-date/irrelevant in 10 yrs Overview Considerations before reading a research paper Methods for accessing research papers Tips for critically evaluating research papers 1

1 Evidence based practice Are those results meaningful? How to understand and apply the results of studies in optometric practice.provide a statement of the scientific basis on which the guidelines were developed and clearly specify the level, quality, relevance and strength of the evidence on which the recommendations are based 1 Dr Lisa Nivison-Smith 1 NHMRC A guide to the development, implementation and evaluation of clinical practice guidelines. Canberra The (scary) rate of publication Global scientific output doubles every 9 years 1 In optometry/ophthalmology 2 97,326 articles published btw % increase every year 50% of current papers estimated to be out-of-date/irrelevant in 10 yrs Overview Considerations before reading a research paper Methods for accessing research papers Tips for critically evaluating research papers 1 RV Noorden, Nature News Blog, Huang W et al (2013) BMC Ophthalmology. 13:25 Overview Considerations before reading a research paper Methods for accessing research papers Tips for critically evaluating research papers Types of research articles Primary research Experimental studies Clinical trials Surveys Secondary research Non-systematic reviews Systematic reviews Meta-analyses 1

Assessing article impact Weight given to different studies when making decisions about clinical interventions Ia Ib Systematic reviews (meta-analyses) of RCT Randomised clinical trials

always indicate positive impact II III IV V Cohort studies Case-control studies Case-series Opinion CJA Bradshaw, ConversationBytes.com 18.11.13 Wakefield et al.

On the antibacterial action of cultures of a penicillium, with special reference to their use in the isolation of B. influenzæ.

citations for papers in a certain journal Can be inflated by reviews, a small number of highly cited papers or self-citations First author: contributed most original work to the project

2 Assessing article impact Weight given to different studies when making decisions about clinical interventions Ia Ib Systematic reviews (meta-analyses) of RCT Randomised clinical trials Impact metrics reflect a paper s effect in its field Citation count Number of times a paper is referenced by other researchers Dependent on field of research and age of paper Does not always indicate positive impact II III IV V Cohort studies Case-control studies Case-series Opinion CJA Bradshaw, ConversationBytes.com Wakefield et al. (1998) Ileallymphoid-nodular hyperplasia, nonspecific, and pervasive development disorder in children. The Lancet. 251: citations Fleming A. (1929). On the antibacterial action of cultures of a penicillium, with special reference to their use in the isolation of B. influenzæ. British Journal of Experimental Pathology 10 (31): citations Assessing article impact Author reputation Impact metrics reflect a paper s effect in its field Citation count Number of times a paper is referenced by other researchers Dependent on field of research and age of paper Does not always indicate positive impact Journal impact factor Average number of citations for papers in a certain journal Can be inflated by reviews, a small number of highly cited papers or self-citations First author: contributed most original work to the project Last author: generally heads research group and can have established reputations in fields Kawasaki et al (2010), Ophthalmology, 117;921 2

$Considerations before reading a research paper Methods for accessing research papers Tips for critically evaluating research papers Methods to obtain scientific papers \ Subscription Journals Papers$ are only available to those with a subscription Open access journals All articles freely accessible to the public Journals charge author processing fees to cover costs Hybrid journals authors chose

are only available to those with a subscription Open access journals All articles freely accessible to the public Journals charge author processing fees to cover costs Hybrid journals authors chose

3 Different types of research have different relative strengths as clinical evidence Impact metrics can reflect a paper s effect on a field but does not necessarily indicate paper quality Overview Considerations before reading a research paper Methods for accessing research papers Tips for critically evaluating research papers Methods to obtain scientific papers \ Subscription Journals Papers are only available to those with a subscription Open access journals All articles freely accessible to the public Journals charge author processing fees to cover costs Hybrid journals authors chose between open access or subscription Some articles are free for certain time periods Predatory Journals Questionable open-access publishers Signs of predatory journals/publishers Inconsistent/unreliable records no academic/contact information on editor, staff or review board false impact factors or other metrics mimics other journals/publishers 3

Many articles can be accessed for free via open access or other online sources Some open access journals do not provide rigorous peer-review and should be avoided Overview Considerations before

4 Many articles can be accessed for free via open access or other online sources Some open access journals do not provide rigorous peer-review and should be avoided Overview Considerations before reading a research paper Methods for accessing research papers Tips for critically evaluating research papers Case study 1 Case study 1 9 year old Asian myopic female BCVA OU: 6/6 Rx OD: DS, OS: DS Yearly routine examinations Mother reports myopia progression over past 3 years Is my daughter short-sighted because she plays too much with my iphone? Ia Ib II III IV V Systematic reviews (meta-analyses) of RCT Randomised clinical trials Cohort studies Case-control studies Case-series Opinion 4

trials Cohort studies Case-control studies

for myopia Correlation vs Causation Younger

causation Causation requires intervention

reporting scientific findings Does near work

5 Ia Ib II III IV V Systematic reviews (meta-analyses) of RCT Randomised clinical trials Cohort studies Case-control studies Case-series Opinion Case study 1: Risk factors for myopia Correlation vs Causation Younger cohort Older cohort Correlation does not imply causation Causation requires intervention studies Commonly implied by media when reporting scientific findings Does near work causes myopia? Is being outdoors necessary for correct visual development? 5

Case study 2 Correlation does not equal causation

be entirely co-incidental even if causality makes

diagnosed with AMD Case study 2 Should I take

Ia Systematic reviews (meta-analyses) of RCT Ib

Case-control studies IV Case-series V Opinion

6 Case study 2 Correlation does not equal causation Similar changes in two scientific variables could be entirely co-incidental even if causality makes sense 56 year old Caucasian female Medications: Nil BCVA: 6/7.6+2 OD 6/6- OS Amsler grid: Unremarkable Previously diagnosed with AMD Case study 2 Should I take supplements for my AMD? Ia Systematic reviews (meta-analyses) of RCT Ib Randomised clinical trials II Cohort studies III Case-control studies IV Case-series V Opinion Ia Systematic reviews (meta-analyses) of RCT Ib Randomised clinical trials II Cohort studies III Case-control studies IV Case-series V Opinion 6

7 Case study 2 Cited: 199 Case study 2: Supplements for AMD ω3 fatty acids Chong et al (2008) Arch Ophthalmology. 126:826 Case study 2: Supplements for AMD Meta-analyses Fish intake Advantages Easy to assess results from multiple studies addressing same research question formally Reasons for inconsistencies can be identified Points to consider Although a large number of studies/patients considered result may mostly come from 1 or 2 studies Potential for bias from inclusion/exclusion criteria Meta-analyses large randomized clinical trial 1. Chong et al (2008) Arch Ophthalmology. 126:826 1 LeLorier, et al, (1997) New Engl J Med;337:

Case-control studies IV Case-series V Opinion Ia Systematic reviews (meta-analyses) of RCT Ib Randomised

8 Meta-analyses help understand large data on single research questions but do not replace the value of individual primary studies Ia Systematic reviews (meta-analyses) of RCT Ib Randomised clinical trials II Cohort studies III Case-control studies IV Case-series V Opinion Ia Systematic reviews (meta-analyses) of RCT Ib Randomised clinical trials II Cohort studies III Case-control studies IV Case-series V Opinion Case study 2: Supplements for AMD 8

Case study 2: Supplements for AMD Relative risk Absolute risk

4% Zinc 21% 4.7% - 21.3% Anti-oxidants + zinc 25% 4.5% - 20.

clinically useful if absolute risk is already low Positive

prefer papers on positive results Researchers don t see the

reporting of null results 1 Fanelli (2012) Scientometric.

jpg Case study 3 Scientific papers tend to report and promote

9 Case study 2: Supplements for AMD Relative risk Absolute risk reduction No supplements 0% 6% 27% Anti-oxidants 17% 5.0% % Zinc 21% 4.7% % Anti-oxidants + zinc 25% 4.5% % Relative risk Absolute risk Reducing relative risk may not be clinically useful if absolute risk is already low Positive outcome bias Publishing bias towards positive results 1 Journals prefer papers on positive results Researchers don t see the reward in papers on negative results New measures to encourage reporting of null results 1 Fanelli (2012) Scientometric. 90: Kaplan and Irvin (2015) PloS One 10(8): e data/assets/image/0009/47529/publishingnegativeresults.jpg Case study 3 Scientific papers tend to report and promote positive/significant findings even though null findings are also clinically important Case study 3 Did you know my OCT can detect glaucoma better? 9

Ia Systematic reviews (meta-analyses) of RCT Ib Randomised clinical trials II Cohort studies III Case-control studies IV Case-series V Opinion Ia Systematic reviews (meta-analyses) of RCT

specifically refers to statistics Indicates likelihood of a result occurring by chance Large sample sizes can lead to significant p values despite small differences Statistical

10 Ia Systematic reviews (meta-analyses) of RCT Ib Randomised clinical trials II Cohort studies III Case-control studies IV Case-series V Opinion Ia Systematic reviews (meta-analyses) of RCT Ib Randomised clinical trials II Cohort studies III Case-control studies IV Case-series V Opinion Case study 3: OCT parameters for glaucoma Statistical significance significant specifically refers to statistics Indicates likelihood of a result occurring by chance Large sample sizes can lead to significant p values despite small differences Statistical significance does not imply clinical significance Low p-values does not exclude poor methodology Specific clinical measures such as specificity and sensitivity are more relevant for comparing clinical tests 10

Case study 4 Case study 4 64 year old Caucasian female normal VA but reports difficulty with night vision (doesn t drive at night) vision loss

family Case study 4 Do you advise the patient that they may have or be a carrier of RP?

There is considerable controversy whether Vit A has a beneficial effect in slowing down the RP Vitamin A may be detrimental Search: retinitis

the National Institutes of Health (NIH), USA Additional testing will reveal changes consistent with reticular pigmentation which is not a

11 Case study 4 Case study 4 64 year old Caucasian female normal VA but reports difficulty with night vision (doesn t drive at night) vision loss in the family 64 year old Caucasian female normal VA but reports difficulty with night vision (doesn t drive at night) vision loss in the family Case study 4 Do you advise the patient that they may have or be a carrier of RP? RP diagnosis requires electrophysiological confirmation (next lecture) Do you advise them that Vit A may slow down the condition? There is considerable controversy whether Vit A has a beneficial effect in slowing down the RP Vitamin A may be detrimental Search: retinitis pigmentosa treatment Two of the five top hits refer to Vitamin A One is from the American Academy of Ophthalmology (AAO) and the second from the National Institutes of Health (NIH), USA Additional testing will reveal changes consistent with reticular pigmentation which is not a typical rod-cone dystrophy Search: vitamin A retinitis pigmentosa AAO provides information about Vitamin A (April, 2015) All first page hits refer to potential beneficial effects of Vitamin A or DHA (w-3 fatty acid) 11

Vitamin A and ω-3 fatty acids (DHA) Vitamin A and ω-3 fatty acids (DHA) All 3 major clinical trials: Trial 1 (1993): author wrote to the editor highlighting that grouping subgroups lead to a

RP progression Only 3 eligible trials Berson et al (vitamin A) Berson et al (DHA) Hoffman et al 2004 (DHA) Cochrane review agrees Massoff & Fishman 2010 Case study 4 Do you advise the patient that

There is controversy whether Vitamin A has beneficial effects in slowing RP Vitamin A may be detrimental Do you refer and advise that further tests are required?

12 Vitamin A and ω-3 fatty acids (DHA) Vitamin A and ω-3 fatty acids (DHA) All 3 major clinical trials: Trial 1 (1993): author wrote to the editor highlighting that grouping subgroups lead to a significant finding Trial 2 ( DHA) and Trial 3 ( lutein), did not show improvement (depends how you plot the data) They conclude that none of the supplements (Vit A; DHA; lutein) modify RP progression Only 3 eligible trials Berson et al (vitamin A) Berson et al (DHA) Hoffman et al 2004 (DHA) Cochrane review agrees Massoff & Fishman 2010 Case study 4 Do you advise the patient that they may have or be a carrier of RP? RP diagnosis requires electrophysiological confirmation Do you advise them that Vitamin A may slow down the condition? There is controversy whether Vitamin A has beneficial effects in slowing RP Vitamin A may be detrimental Do you refer and advise that further tests are required? Additional testing will reveal changes consistent with reticular pigmentation which is not a typical rod-cone dystrophy Case study 4 Do you advise the patient that they may have or be a carrier (X-linked) of RP? RP diagnosis requires electrophysiological confirmation Do you advise them that Vitamin A may slow down the condition? There is controversy whether Vitamin A has beneficial effects in slowing RP Vitamin A may be detrimental Do you refer and advise that further tests are required? Additional testing will reveal changes consistent with reticular pigmentation which is not a typical rod-cone dystrophy Cone-rod dystrophies Radu et al IOVS 2008 Mutation in the ABCR (ABCA4) gene Causes atypical retinitis pigmentosa Transporter cannot export retinoids out of photoreceptor outer segment Accumulation of retinoids in photoreceptor outer segment and subsequent accumulation in RPE Damage via elevated A2E levels At 11 months, albino mice with ABCA4 mutation showed elevated photoreceptor degeneration after Vitamin A supplementation 12

13 NIH update 2008 Some evidence based recommendations are not always consistent with the evidence base Summary Most papers are now freely accessible Determining the type of research is important to understand level of evidence Caution must be taken using impact metrics Some issues to consider when appraising a paper Correlation causation Statistical significance clinical significance Positive outcome bias Meta-analyses do not always supersede individual studies 13

Year 4 Results For a Phase 1 Trial of Voretigene Neparvovec in Biallelic RPE65- Mediated Inherited Retinal Disease

8:00 AM Year 4 Results For a Phase 1 Trial of Voretigene Neparvovec in Biallelic RPE65- Mediated Inherited Retinal Disease Albert M. Maguire, MD OBJECTIVE Assess maintenance of functional vision/visual