EBP and SLP: speechbite helps clinicians access and appraise the evidence speechbite.com

Transcription

1 EBP and SLP: speechbite helps clinicians access and appraise the evidence speechbite.com Leanne Togher, Elizabeth Murray, Emma Power, Kate Smith, Patricia McCabe, Natalie Munro & Melissa Brunner Speech Pathology, Faculty of Health Sciences, The University of Sydney, Sydney, Australia ASHA Convention, Chicago, USA 15 th November 2013

2 Acknowledgements/financial disclosure 2

3 Official speechbite launch We are happy today to launch the new updated speechbite website!

4 speechbite has evolved! speechbite 2013

5 Easier navigation - Larger print So what s new about the new speechbite? - Responsive web design so that you can search speechbite on your tablets and mobiles Better searching options - Search straight from the home page - New intervention categories (e.g., cognitive rehabilitation, social/pragmatic therapy. - New client population categories (e.g., multilingual, at risk of a communication disorder. Better options for managing search results - Export your results to EndNote or save them to your computer.

6 What s new about the new speechbite? Clinical Practice Guidelines have been added - 30 guidelines have been added from the past 5 years - Linked to guidelines listed on the ASHA compendium providing access to AGREE II ratings Get monthly updates on new research added to speechbite - Enter your details in our Stay Informed page and we ll send you a monthly update of the latest papers in key areas of speech pathology

7 What is speechbite? Evidence Based Practice speechbite aims to make critical appraisal of published intervention research quicker and easier Figure: Chapter 1, in Hoffman, T., Bennett, S. & Del Mar, C. Evidence-based practice across the health professions. Elsevier Australia, speechbite has evolved! speechbite 2013

8 Steps in conducting EBP ACCESS: Search for evidence APPRAISE: Critically appraise the evidence for its validity and relevance APPLY: Make a decision by integrating evidence with clinical experience ASSESS: Evaluate performance after acting on evidence ASK: Pose an answerable question

9 What is speechbite? Unique in SLP: Only database of evidence based speech language pathology treatment literature Comprehensive: Repeated sensitive searches across multiple databases e.g. Medline, PsychInfo, journal alerts Empirical data: All listed studies contain empirical data SLP focused: Only includes trials where the population have a communication or swallowing disorder or are at risk Rated designs: Group comparison and single case experimental design studies rated for risk of bias

10 Some facts and figures Freely available worldwide More than 1 million hits in 2013 from over 120 countries Top 5 countries: Australia, USA, UK, Germany, Canada Over 6000 unique users per month 100% increase in searches in 2013 Incorporated into Speech Pathology Australia Competency Occupational Based Standards (CBOS) (2011) document Taught in every speech pathology university curriculum across Australia Utilized by Australian Speech Pathology EBP interest group networks

11 How does a paper get listed on speechbite? Identify of all potentially relevant reports from 9 generic databases every month via autosearch: Medline, PsycINFO, CINAHL, AMED, ERIC, Embase, LLBA, Cochrane, Google Scholar Screen for eligibility Index reports for SLP Practice Area, intervention type, population, service delivery model, age group, and study design Rate for methodological quality Enter citation details, abstract, indexing terms and ratings on speechbite database

12 Papers accepted onto speechbite must fulfill the following criteria: The treatment is currently part of speech pathology practice or could become part of speech pathology practice. The intervention need not be carried out by speech pathologists. The paper is published as a fulllength paper (not an abstract) in a peer-reviewed scientific journal The population treated are representative of those whom the intervention may be applied to in the course of speech pathology practice. The population treated will have, or be at risk of, a communication disorder and/or dysphagia. The trial includes evaluation of at least one intervention and includes empirical data regarding treatment efficacy

13 Searchable domains on speechbite SLP Practice Areas Speech - E.g., articulation, perception, dysarthria Language - E.g., Language impairment acquired, developmental, discourse, pragmatics, semantics, syntax Swallowing - Adult dysphagia, paediatric feeding Voice Fluency Literacy - Preliteracy - Reading, writing, spelling Age ranges Under 2 Under 5 Children (up to 16 years) Adolescents Adults

14 Searchable intervention types listed on speechbite Speech Language Stuttering Swallowing/feeding Voice Literacy/pre literacy Computer based AAC Social pragmatic therapy Surgical Pharmacological Counseling Complementary Education Aural Rehabilitation Assistive devices/technology Cognitive rehabilitation

15 Some of the populations indexed on speechbite Alzheimers and other dementias Attention deficit disorder Autism Cancer Cerebral palsy Child onset fluency disorder Cleft lip/palate/craniofacial Congenital Degenerative disorders/diseases Hearing and visual impairment Intellectual disability Language disorders (developmental) Laryngectomy Mental Health Multilingual NEW!! Speech sound disorders (developmental) Stroke Tracheostomy Traumatic brain injury At risk NEW!!

16 Research Designs listed on speechbite Study types Systematic Reviews Randomised Controlled Trials (RCTs) Non-randomised Controlled Trials (NRCTs) Case Series Single Case Designs - Experimental (SCEDs) - Non-experimental Clinical Practice Guidelines speechbite has evolved! speechbite 2013 Images: psdgraphics.com

17 Levels of Evidence Level of Evidence I II III Study Design Evidence obtained from a systematic review of all relevant randomized controlled trials Evidence obtained from at least one properly designed randomised controlled trial. Evidence obtained from well-designed pseudorandomised controlled trials (alternate allocation or some other method). (NHMRC 2010) IV Evidence obtained from non-randomised controlled trials: such as comparative studies (including systematic reviews of such studies) with concurrent controls and allocation not randomised, cohort studies, case-control studies, or interrupted time series with a control group. Evidence obtained from comparative studies with historical control, two or more single arm studies, or interrupted time series without a parallel control group. Evidence obtained from case series, either post-test or pretest/post-test

18 Systematic Review - overview of primary studies Study Designs - with explicit statements of objectives, materials, and methods - conducted according to explicit and reproducible methodology - includes meta analyses - a mathematical synthesis of the results of two or more primary studies that are similar in hypotheses and methodology

19 Study Designs Randomised Control Trial (RCT): - comparison of at least two interventions where - Allocation is random - Outcomes are compared between groups Non Randomised Control Trial: - comparison of at least two interventions - non-random allocation

20 Study Designs Case series: also known as pre-post - one intervention given to group(s) - outcomes measured in participants before and after exposure Single Subject Design (SSD): - same (single) participant - assessing treatment effects over several different points in time as treatments are systematically applied or withdrawn

21 Overview of papers listed on speechbite (n=4000) N = 4000 treatment research articles

22 Overall 92.9% of RCTs and non-rcts on speechbite are rated (n=1106).

23 Systematic Reviews Single Case Experimental Design Randomized Controlled Trials Treatment Studies Non- Group Comparison Group Studies Comparison Studies Randomized Controlled Trials Rated for Methodological Case Series Quality 23

24 No Criterion Score Eligibility criteria were specified External Validity Subjects were randomly allocated to interventions Allocation was concealed Intervention groups were similar at baseline regarding: a. Key outcome measure(s) b. Severity of the condition being treated c. Important prognostic indicators identified Blinding of all subjects PEDro-P Internal Validity Blinding of all therapists who administered the therapy Blinding of all assessors who measured at least one key outcome Indication of risks of bias Measures of at least one key outcome were obtained from more than 85% of the subjects initially allocated to groups All subjects for whom outcome measures were available received the treatment or control condition as allocated. When this was not possible, data was analysed by intention to treat Results of between-intervention group statistical comparisons are reported for at least one key outcome The study provides both point measures and measures of variability for at least one key outcome Overall Score /10 24

25 - Group comparison trials (RCTs speechbite and ratings: PEDro-P scale NRCTs) - Ratings out of 10 - Average PEDro-P rating on speechbite is 4 - Score of 6+ considered robust methodological quality speechbite has evolved! speechbite 2013 Image:

26 PEDro-P Scale Based on a Delphi list (Verhagen et al 1998) + 2 additional items Expert consensus A trial of quality - external validity (1) + - method score (10): internally valid & sufficient information to make results interpretable Not a measure of the validity of study s conclusions or that treatment is useful Reliability reported (Maher et al., 2003; Murray et al., 2013)

27 PEDro-P scale External Validity Criterion 1 Participant eligibility and recruitment source stated Biases prior to treatment phase Criterion 2 Random allocation to groups Criterion 3 Allocation concealment Criterion 4 Groups similar at BL (demographic, severity and o/come) Performance biases during treatment Criterion 5 Blinding of participants Criterion 6 Blinding of therapist Criterion 7 Blinding of assessors Biases at data analysis Criterion 8 Outcome measures from more than 85% of participants Criterion 9 Intention to treat analysis Criterion 10 Between-group statistical comparison Criterion 11 Measures of variability 27

28 Benchmarking speechbite s use of the PEDro-P scale for rating group comparison studies Retrospective investigation of the first 100 RCTs and NRCTs rated until August 2009 Papers: - published between 1973 and Average RCT PEDro score = 5.17 (SD 1.4). - Were rated by 2 raters independently and a 3 rd consensus rater if needed Participants: - 17 members of volunteer raters network - Underwent training and achieved over 82% (pass mark) on accuracy test - Rated on average two papers a month (range 1-6) - Knew were rating for speechbite but did not know were part of reliability study

29 Results Database Overall reliability (ICC 1, 1) Confidence interval (95%) speechbite PEDro OTseeker PsycBITE Not published Not published NB: Poor = < 0.40, Fair-Good = , Excellent = > 0.75 (Fleiss, 1986) Reliability of speechbite PEDro-P raters is in the good to excellent range Average discrepancy = < 1 per paper (Murray et al 2013) 29

30 Key Definitions Key outcome variables - provides the primary measure of the effectiveness of the therapy, pertaining to the primary aim of the study. - most studies report more than one variable as an outcome measure Prognostic indicators - variables that influence the effect of outcome irrespective of equivalence of outcome measures Blinding - the person in question (subject, therapist or assessor) did not know to which group the subject had been allocated. - being unable to distinguish between treatments in the study

31 Criterion 1: Eligibility The key aim is to provide information so that a comparable study could be conducted. This criterion is satisfied if the report describes: - the SOURCE of subjects AND - a list of specific CRITERIA (either inclusion and/or exclusion criteria)

32 Criterion 1: (Drummond & Walker, 1995, col 2. para. 3, p284) Rating Demonstration Source details Exclusion criteria

33 Criterion 2: Random Allocation allocating individuals to groups based on chance - precise method of randomisation need not be specified - considered random if the report states that allocation was random - can be block randomisation, stratified, matching pairs not satisfied by quasi-randomisation - other methods of allocation which are not statistically random, but systematic e.g. by date of birth, days of the week, hospital record number, or alternation )

34 Criterion 2: Example (Drummond & Walker, 1995, col. 1 para. 3, p285) Random allocation

35 Criterion 3: Concealed allocation person who determined if a subject was eligible for inclusion in the trial was unaware at the time of his/her group assignment Satisfied if allocation was concealed or indicates a method that achieves this: - E.g sealed envelopes, - Allocation conducted by an independent person (e.g. telephone randomisation, researcher off-site, or other researcher blinded to study etc)

36 Criterion 3: Example (Drummond & Walker, 1995, col. 1 para. 3, p285) Concealed allocation

37 Crit. 4: Groups similar at baseline Statistically similar results at baseline for: Primary measure(s) of effectiveness - At least one key outcome measure (not just demographics etc) Severity measure(s) of condition treated - At least one measure of severity of condition treated (that is not the above single primary measure of effectiveness) Important prognostic variables - demographic and clinical variables which affect outcome regardless of treatment group

38 Crit. 4: Groups similar at baseline 3 question test: - 1) what are the key outcome, severity, and prognostic variables? - 2) has at least one key outcome measure and one additional severity measure been compared at baseline, with no significant group differences? - 3) are there significant differences in important prognostic variables that are not controlled for in the outcome analysis?

39 Criterion 4 example: (Drummond & Walker, 1995, Table 4 p287) Key outcome measures equivalent Severity measure equivalent

40 Criterion 4 example: (Drummond & Walker, 1995, col. 1 para. 2, p286) Prognostic variables: difference in age among groups

41 Criterion 4 example: (Drummond & Walker, 1995, col. 1 para. 4, p288) but controlled for through ANCOVA

42 Blinding involves: Criteria 5 & 6: Blinding of participants/ therapists - Subject/therapist being unaware of group allocation, and - being unable to tell the difference between treatments offered to different groups Met if paper states that subject and/or therapist is blind and if blinding is possible for the study

43 Criteria 5 & 6 example: (Drummond & Walker, 1995, p285) Clear differences in treatment offered to groups

44 Criterion 7: Blinding of Assessor Assessor is unaware of group assignment Satisfied if blinding is mentioned and assessors are blinded for both pre- and postintervention assessments Where disclosures are made, the assessor is not unblinded to more than 10% of participants

45 Criterion 7: (Drummond & Walker, 1995, col. 2 para. 3, p285) Blind assessor

46 Criterion 8: Retention of >85% Key outcome(s) results are based on > 85% of sample - For at least one key outcome measure Total sample from when randomised to groups - Subtract drop-outs and calculate - Infer from degrees of freedom

47 Criterion 8: (Drummond & Walker, 1995, p ) Initial n = 65 3 month n = 62 6 month n = 60 N for outcomes = 62/65*100 =95%

48 Criterion 9: Intention to Treat Data analysed according to the group subjects were allocated to, regardless of treatment received Satisfied if : - ITT mentioned (or described), - stated that all subjects received treatment as allocated - stated that data where available is analysed according to group allocation Not satisfied by flow diagrams showing same numbers progressed from start to finish (as does not preclude switching between groups)

49 Criterion 9: (Drummond & Walker, 1995) No mention of intention-to-treat analysis (or description that participants were analysed according to their assigned group) no statement that all patients received the treatment they were allocated to Group allocation was random, thus cannot assume that participants stayed within the assigned groups

50 Criterion 10: Between group comparisons At least one post-treatment outcome measure is compared between groups Analysis can involve: - Comparing change over time between groups - Hypothesis testing (e.g one group outcome > another group) - Comparing the relative risk ratio etc Not satisfied if analyses are just showing change over time within each group

51 Criterion 10: (Drummond & Walker, 1995, p288)

52 Criterion 11: Point measures & measures of variability Some quantification of size of treatment effect, and of the level of error associated - Typically mean and SD or median and IQR Shown for at least one post-treatment outcome measure Can be provided graphically or in text or tables

53 Criterion 11 example: (Drummond & Walker, 1995, p287)

54 Scoring summary (Drummond & Walker 1995) 1. Eligibility criteria specified - (Y) p284 c.2 para.3 Internal Validity: 2. Random allocation to groups (Y) p285 c.1 para.3 3. Allocation concealment (Y) p285 c.1 para.3 4. Groups similar at baseline (Y) p286-7, Table 4 5. Blinding of participants (N) 6. Blinding of therapist (N) 7. Blinding of assessors (Y) p285 c.2 para.3 8. Outcome measures from > 85% (Y) p286-7, Table 4 9. Intention to treat analysis (N) 10. B/w-group statistical comparison (Y) p288, Table Measures of variability (Y) p287, Table 5 SCORE: 7/10

55 speechbite ratings Consensus ratings scored by 2-3 different raters Tell us the believability (validity) of studies Ratings are designed to give you an idea of the scientific rigour used in the study so that you can quickly identify the highest quality studies speechbite has evolved! speechbite 2013 Image:

56

57 This is the PEDro-P Rating This study has a rating of 10/10! = VERY strong, robust methodological quality speechbite has evolved! speechbite 2013

58 This is the PEDro-P Rating This study has a rating of 6/10 = robust methodological quality speechbite has evolved!. speechbite 2013

59 This is the PEDro-P Rating This study has a rating of 1/10 = low methodological quality speechbite has evolved! speechbite 2013

60 Single case experimental speechbite trials (SCEDs) ratings: RoBiNT scale Revised scale recently released in 2013 Ratings to be out of 30 (instead of 12!) speechbite ratings of SCEDs are about to recommence... Reference: Tate RL, Perdices M, Rosenkoetter U, Wakim D, Godbee K, Togher L, McDonald S (2013). Revision of a method quality rating scale for single-case experimental designs and n-of-1 trials: The 15-item Risk of Bias in N-of-1 Trials (RoBiNT) Scale. DOI: / Published online: 09 Sep speechbite has evolved! speechbite 2013 Image:

61 How do you search? Searching speechbite Review your results - Which are the better papers to read? - Why (what is your rationale for this)? speechbite has evolved! speechbite 2013 Image:

62 Tips for searching speechbite speechbite has evolved! speechbite 2013

63 Tips for searching speechbite speechbite has evolved! speechbite 2013

64 Incorrect spelling the database will not match When your search gets no results... misspelled words There may be no articles that meet the inclusion criteria for speechbite in the areas that you are searching A lack of published research in that area Your search terms may be too narrow speechbite has evolved! speechbite 2013

65 speechbite has evolved! speechbite 2013

66 Search example 1 You need to find information about best practice for speech therapy with a 4 year old child that has an acquired brain injury. How would you use speechbite to help plan evidence-based therapy? speechbite has evolved! speechbite 2013

67 You need to find information about best practice for speech therapy with a 4 year old child that has an acquired brain injury. How would you use speechbite to help plan evidence-based therapy? speechbite has evolved! speechbite 2013

68 speechbite has evolved! speechbite 2013

69 Search example 2 You are looking for research on stuttering treatment for a teenage client. How would you use speechbite to find it? speechbite has evolved! speechbite 2013

70 You are looking for research on stuttering treatment for a teenage client. How would you use speechbite to find it? speechbite has evolved! speechbite 2013

71 Which papers would you look at first? Why (what is your rationale)? speechbite has evolved! speechbite 2013

72 In Summary Free website no login or password required Search the entire scope of speech language pathology intervention literature Obtain ratings of research method quality Learn how to critically appraise research articles Save time! speechbite has evolved! speechbite 2013 Image:

73 By knocking on our door at The University of Sydney: - Cumberland Campus S Block, Room Got S142, questions? there s a sign Contact our door us! saying speechbite - We are in the office Tuesday, Thursday and Friday By calling: Through social media: - on Twitter - Like speechbite on Facebook By ing: Kate.Smith@sydney.edu.au Melissa.Brunner@sydney.edu.au Images: has evolved! JiD0I/AAAAAAAAAxg/tZpYaPkOk1g/s1600/old-telephone.jpg; speechbite 2013

74 speechbite has evolved! speechbite 2013