Program. What is a Critically Appraised Topic? (cont.) What is a Critically Appraised Topic? Oliver Wendt Developing Critically Appraised Topics

Transcription

1 CAT 101: Developing Critically Appraised Topics for Evidence-based Clinical Decision Making Oliver Wendt, Ph.D. Program 1. What is a Critically Appraised Topic (CAT)? 2. How to construct a CAT 3. Example for a CAT taken from augmentative and alternative communication (AAC) 4. Other approaches for critical appraisal 5. Validation Criteria 6. Limitations and difficulties 7. Tools for dissemination 8. Software to develop CATs 9. Questions What is a Critically Appraised Topic? 1. Preferred categorization format for quick studies in evidence-based practice (EBP) 2. Defined as a brief summary of a search and critical appraisal of the literature related to a focused clinical question, which should be kept in an easily accessible place so that it can be used to help make clinical decisions (Center for Evidence-based Emergency Medicine, n. d. a.). 3. Describes the best research evidence to date, evaluates the findings and summarizes the results into 1-2 pages (Shannon, 2001) What is a Critically Appraised Topic? (cont.) 4. Key parts of a CAT include: 1. Purpose 2. Reviewer 3. Date of completion 4. Proposed Re-evaluation 5. Well-built questions 6. Search strategies and results 7. Evidence retrieved 8. Appraisal 9. Conclusions Systematic review of research versus CAT Characteristics Systematic review CAT Goal Focus Authors Search Critical, in-depth assessment and evaluation of existing research using methods to reduce likelihood of bias; provide overview on current research base Can have wide scope, comprehensive, focus on larger body of research Written by content and methodology experts Comprehensive search strategy, exhaustive search Create concise summary of best evidence most central for specific clinical scenario Narrowed down to specific clinical question Clinicians and practitioners Limited to best and most current evidence, time-saving strategies Systematic review of research versus CAT (cont.) Characteristics Systematic review CAT Effort/ Efficiency Commonalities Design Dissemination Potential for bias Don t leave a stone unturned Time-limited (high efficiency preferred) Critical appraisal of the quality of retrieved evidence Can be quantitative (meta-analysis) Journals, books, scholarly publications Depends on comprehensiveness and transparency of methods Brief narrative summary, descriptive CAT databases, institutional collection Rather high 1

2 Why do we need CATs? Almost impossible for practitioner to keep up with all the new research in a field Research shows that even seasoned health-care clinicians and practitioners encounter up to 5 knowledge needs for every in-patient, and about 2 needs for every three out-patients (Sauvé, 1995) Knowledge needs raise questions about the best course of action Critically appraised summaries of the best evidence for common clinical or educational questions address these needs Gold standard would be a systematic review, but this takes too much time and effort (up to 1-3 yrs.) and is not practicable for clinicians (Ziegler et al., 2001) Evidence-based decision making in clinical scenarios demands time-efficient, up-to-date evidence review CATs as part of the EBP process The EBP process: 1. Developing a well-built question 2. Selecting evidence sources & executing the search strategy 3. Examining and synthesizing the evidence Write CAT 4. Applying the evidence 5. Evaluating the application of the evidence Revise CAT 6. Disseminating the findings (i.e., share CAT) (Sackett et al., Steps 1-5; Schlosser, 2003) How to construct a CAT 1. Formulate a well-built question Identify gaps in knowledge ( knowledge needs ) from client encounters Is this problem about an intervention, assessment or diagnosis? Translate the problem into an answerable question Avoid vague, unanswerable questions ( Is new intervention better than traditional approach? ; What does better mean? In what population and setting?) 1. Formulate a well-built question (cont.) Use PICO framework (Schlosser, Koul & Costello, 2005; Schlosser & Raghavendra, 2004): P Patient or problem being addressed I Intervention or exposure being considered C Comparison intervention or exposure O Outcomes of interest 2. Search for best evidence Locate best & most current evidence first Seek out reviews before individual research studies Work down the hierarchy of sources Database of Reviews of Effects (DARE) Cochrane Database of Systematic Reviews General purpose databases Look for peer-reviewed evidence before nonpeer-reviewed evidence Analyze those articles most pertinent to your question Evaluate evidence How good is the evidence (i.e., internal validity)? Quality and appropriateness of design Treatment integrity Interobserver agreement What is the social importance or significance of the evidence (i.e., social validity)? What is the transportability of the evidence (i.e., external validity)? Apply hierarchy of evidence Different fields have different hierarchies, e.g., AAC field: (1) participants with disabilities, and (2) participants without disabilities (Schlosser & Raghavendra, 2004) 2

3 1. Meta-analysis of (a) single-subject experimental designs (Schlosser & Lee, 2000), (b) non RCTs, 2a. One well-designed non 2b. One single-subject 2c. One single-subject RCT experimental design Š one experimental design Š intervention multiple interventions i. Multi-group pretestposttest design without (e.g., Reid & Hurlbut, design(e.g., Bennett et al., i. Multiple baseline design i. Parallel treatments control group (Layton, 1977; Sigafoos et al., 1986; Schlosser et al., 1988; Yoder & Layton, 1996) 1995) 1988) ii. Basic within-group ii. Multiple probe design ii. Adapted alternating design, crossover design, (e.g., Schlosser et al., treatments design (e.g., complex 2000) Schlosser et al., 1998) counterbalanced design iii. Multi-group time series iii. ABAB design (e.g., iii. Alternating treatments designs Vaughn & Horner, 1995) design (e.g., Remington & Clarke, 1993) iv. Factorial designs iv. ABA design (Schweigert iv. ABAC A/ACAB design & Rowland, 1992) (e.g., Koul et al., 1997) v. Latin square designs v. Non-concurrent multiple v. A-B-BC-B-BC/A-BC-B- (Hupp & Mervis, 1981) baseline design BC design (Parson & LaSorte, 1992) vi. Posttest-only control vi. ABAB design (e.g., group design, mu ltigroup posttest-only Burberry, 1989) Rotholz, Berkowitz, & design vii. Single-group time series design viii. Separate sample pretestposttest design 3. Quantitative reviews that are non meta-analytic (Mirenda, 1997) 4. Narrative reviews (Doherty, 1985; Schlosser & Sigafoos,2002) 5. Preexperimental group designs (e.g., one-shot case study, one -group pretest-posttest design, and the static group comparison) and single-case studies (e.g., AB d esigns, case studies) (Blischak, 1995; Dowden et al., 1986; Glennen & Calculator, 1982) 6. Respectable opinion (Augmentative Communication News, Perspectives, ISAAC Bulletin, Opinions of expert presenters, textbook authors) (Schlosser & Raghavendra, 2004) Quality and appropriateness of design Experimental vs. pre-experimental designs Treatment integrity The degree to which an independent variable is implemented as intended (Schlosser, 2002) Treatment integrity has the potential to enhance, diminish, or even destroy the certainty of evidence (Schlosser, 2003, p. 273) Evaluation Checklist for Planning and Evaluating Treatment Integrity Assessments (Schlosser, 2003) Interrater reliability on dependent measures Degree to which two independent observers agree on what is being recorded Acceptable levels of observer consistency: > 80% Assessment of methodological quality Certainty of evidence Conclusive, preponderant, suggestive, inconclusive (Simeonsson, 1994) Social Validity Are consumers satisfied with the results, all of the results [positive or negative], including those that were unplanned? Behavioral treatment programs are designed to help someone with a problem. Whether or not the program is helpful can be evaluated only by the consumer. Behavior analysts may give their opinions, and these opinions may even be supported with empirical objective behavioral data, but it is the participants and other consumers who want to make the final decision about whether a program helped solve their problems (Wolf, 1978, p. 210). (cont.) Transportability (i.e., External Validity) Generality of Conditions to evaluate the exact fit between the research conditions and the conditions of their practical situation. Generality of Subjects/Participants will the research results work for my client even if this client might differ from the study participants in a number of characteristics? Generality of Communication Partners Generality of Settings Generality of the Treatment Agent Generality of Materials (cont.) Apply appraisal checklist if available for particular type of evidence For example, checklists exist for internal validity evaluation of single-subject experimental designs, treatment integrity, and systematic reviews (see Schlosser, 1999, 2002; Schlosser, Wendt, Sigafoos, in press) May not be available for all types of research May not be field tested by practitioners yet to determine utility 3

4 4. Write the CAT The CAT is a 1-2 page summary of your findings Keep it at an easily accessible place so that other practitioners can rely on it when encountering the same or similar clinical question (Sauvé, 1995) High quality CAT should be disseminated by submitting to CAT bank or professional journal Example for a CAT in AAC (Schlosser, Wendt, & Blischak, in prep.) Start with scenario and identify knowledge need Sam is a 4-year old child who was recently diagnosed with autism. He is unable to meet his daily communication needs in his preschool through natural speech. He has recently learned to imitate words such as mama, dada, quack-quack, and bye-bye. To date, Sam does not use these words to communicate. His family and the staff at his preschool anticipate many of his needs and consistently respond to his prelinguistic communication behaviors such as touching objects or leading people to objects he wants or activities that he would like to do. Sam s parents hope that he will eventually speak and be included in a classroom with typically-developing children. Prior to Sam s diagnosis, his parents were not interested in exploring other forms of communication because they had serious concerns about the impact on Sam s potential for developing speech. Their current goals for Sam are that he improve his speech and communicate more effectively through whatever means are appropriate and supportive of speech development. Which AAC approach(es) best support natural speech production? Title Impact of AAC Intervention on Natural Speech Production in Learners with Autism Type of CAT Related to treatment/intervention Well-built question Sam is a 4-year old child who was recently diagnosed with autism. He is unable to meet his daily communication needs in his preschool through natural speech. He has recently learned to imitate words such as mama, dada, quack-quack, and bye-bye...his parents want to know, which AAC approach(es) best support natural speech development? Reviewer/s Ralf W. Schlosser and Doreen M. Blischak Search Database Searches (to identify reviews) DARE ( ): autism and communication; yielded 6 references but none focused on AAC and autism; CINAHL (inclusive 2002): autism and augmentative and alternative communication and systematic review [DT] and similar combinations; yielded 3 reviews but not focused on autism ERIC (- 2002): autism and communication and review; autism and augmentative and alternative communication; yielded 5 reviews on autism and AAC some of which included sections on natural speech production but did not focus on this issue Database Searches (to identify reviews) Medline (- 2003): autism and communication and review: 4 reviews, some of which included parts on natural speech production. PsycINFO (- 2002): autism and augmentative communication, and review: 3 reviews, some of which included parts on natural speech production. Hand Searches Special issue on speech output in Augmentative and Alternative Communication (2003): 1 relevant review and 1 relevant study ISAAC proceedings 2000 & 2002: 1 systematic review ASHA Convention Abstracts 2001 & 2002: 1 systematic review Millar et al. (2003) Schwartz et al. (2003) Database Searches (to identify studies) CINAHL (- 2002): autism AND augmentative and alternative communication ERIC (- 2002): autism AND communication, autism AND augmentative and alternative communication Medline (- 2003): autism AND communication AND ( clinical trials [MeSH terms] OR research design [MesH terms] OR evaluation study [MeSH terms] OR clinical trial [PT]) PsycINFO (- 2003): autism and augmentative communication and (LA=ENGLISH) and ((PT=CLINICAL-TRIAL) or (PT=EMPIRICAL-STUDY) or (PT=EXPERIMENTAL-REPLICATION) or (PT=FOLLOWUP- STUDY) or (PT=META-ANALYSIS) or (PT=TREATMENT- OUTCOME-STUDY)) 4

5 Date The search was completed August 10, 2003; Proposed re-evaluation: August 2006 Citations (only those that are relevant) Relevant Narrative Reviews (only sections): Bondy & Frost (1998); Bondy & Frost (2001); Goldstein (2002); Koul, Schlosser, & Sanscribrian (2001); Mirenda (2001); Mirenda & Erickson (2000); Schlosser (2003a); Schlosser (2003c); Schlosser & Blischak (2001); Sigafoos, & Drasgow (2001); Sundberg, & Michael (2001) Relevant Systematic Reviews (to locate appraised studies): Correa, N., & Nye, C. (2001). Sign language and autism: A quantitative synthesis of single-subject research. Poster presented at the Annual Convention of the American Speech-Language Hearing Association. ASHA Leader, 6(15), 189. Relevant Systematic Reviews (to locate appraised studies): Schwartz, J. B., Nye, C., & Correa, N. (2003). The efficacy of teaching sign language to children with autism. Unpublished manuscript. University of Central Florida, Orlando, FL. Millar, D., Light, J., & Schlosser, R. (2003). The impact of AAC on natural speech development: A meta-analysis. Unpublished manuscript. Penn State Univ, College Park, PA. Millar, D., Light, J., & Schlosser, R. (2000). The impact of AAC on natural speech development: A meta-analysis. In Proceedings of the 9th biennial conference of the International Society for Augmentative and Alternative Communication (pp ). Washington, DC: ISAAC. Relevant Studies (Suggestive evidence or better): Anderson, A. E. (2002). Augmentative communication and autism: A comparison of sign language and the Picture Exchange Communication System. Doctoral Dissertation, University of California at Santa Barbara, 2001). Dissertation Abstracts International: Section B: The Sciences and Engineering, 62, Relevant Studies (Suggestive evidence or better): Charlop, C. M. H., Carpenter, M., Le, L., LeBlanc, L. A., & Kellet, K. (2002). Using the picture exchange communication system (PECS) with children with autism: Assessment of PECS acquisition, speech, social-communicative behavior, and problem behavior. Journal of Applied Behavior Analysis, 35, Kouri, T. A. (1988). Effects of simultaneous communication in a childdirected treatment approach with preschoolers with severe disabilities. Augmentative and Alternative Communication, 4, Parsons, C. L., & LaSorte, D. (1993). The effect of computers with synthesized speech and no speech on the spontaneous communication of children with autism. Australian Journal of Human Communication Disorders, 21, Yoder, P. J., & Layton, T. L. (1988). Speech following sign language training in autistic children with minimal verbal language. Journal of Autism and Developmental Disorders, 18, Relevant Studies (Inconclusive evidence): Bondy & Frost (1994); Bonta & Waters (1983); Dyches (1998); Fulwiler & Fouts (1976); Kravits et al. (2002); Schwartz et al. (1998); Sigafoos et al. (2003) excluded Summary and appraisal of studies Studies were appraised in terms of design, interobserver agreement, and treatment integrity Depending on how these three issues were addressed, outcomes were appraised as Conclusive Preponderant Suggestive Inconclusive 5 studies remained and were all ranked as suggestive : certain speech outcomes are plausible and are within the realm of possibility as the result of AAC intervention; minor flaws in design, interobserver agreement, and/or treatment integrity Study; Subjects AAC (n, age, a ) Anderson 6 (2002); PECS vs. manual signing Charlop et al. (2002); PECS children (2 to 4) 3 children (3, 5, 12) Summary and Appraisal of Studies Results Signing resulted in more vocalizations during correct responding a t post treatment than PECS; vocalizations during and after treatment were associated with initial vocal imitation levels. John mastered 3 items with speech during signing and 1 item during PECS intervention. Ryan mastered one item with speech in s igning and none in PECS. Spontaneou s speech and imitation increased even with nov el (a) partners, (b) settings, and (c) stimuli. Appraisal Suggestive: Sets were equated in terms of preference; a teaching criterion was used; treatment integrity and interobserver agreement are strong; because vocalizations were measured only du ring correct responding, there is no comparable baseline. Suggestive: The speech design rules out threats to internal validity; strong interobserver agreement data; treatment integrity data are lacking 5

6 Study; AAC Kouri (1988) Parsons & LaSorte (1993); SGD Š synthetic speech Yoder & Layton (1988); Manual signing Subjects (n, age, a ) 1 child (3) 6 children (4 to 6) 60 children (5) Results A PND of 24 was yielded, rendering the effects unreliable Intervention without speech outpu t produced no chang e in terms of spontaneous utterances. When speech output was added, vocalizations increased Across conditions, pretreatment vocal imitation skills predicted speech produc tion. Speech alone, simultaneous, and alternating yielded better speech produc tion than signing alone. Appraisal Suggestive: No treatment integrity data Suggestive: The speech design was mapped on to the intervention design. Order effects were controlled across subjects with an n of only 3 (between -group n requirements not satisfied); no treatment integrity data Suggestive: Strong de sign; lack of treatment integrity data; the evidence would have been stronger if they documented increases in signing as well. Applicability the n is < 9 per experiment, and only 1 study per recommendation thus, generality may be compromised; however, several participants are of the same age as Sam three out of four studies included participants with demonstrated vocal imitation skills at onset (just like Sam) Conclusions it is plausible that: PECS increases spontaneous natural speech (Charlop et al., 2002) manual signing increases natural speech production to a greater extent than does PECS (Anderson, 2002); Conclusions cont d it is plausible that: simultaneous communication and alternating between speech alone and simultaneous communication yield better speech production than signing alone (Yoder & Layton, 1988); computer-based instruction with synthetic speech output increases natural speech production compared to computer-based instruction without speech output (Parsons & LaSorte, 1993); and, pre-treatment vocal imitation skills predict natural speech production (Anderson, 2002; Yoder & Layton, 1988). Further approaches to critical appraisal Journal clubs Example: University of Copenhagen, Dept. of General Practice General practitioners from local community invited to meet for 2-hrs-session once a month; anyone can join Each practitioner gets a copy of an article and an appraisal checklist Everybody reads same article and evaluates it, then group discussion to arrive at final conclusion about the evidence However, this has not produced completely written CATs that can be stored and shared beyond the club (Makela & Witt, 2005) Validation Criteria for CATs Is the CAT valid? 1. Was the CAT focused by a well-built question? 2. What was the explicit and sensible process used to identify and select the evidence? 3. Is it unlikely that relevant studies were missed? 4. Was the evidence appraised the best available to answer the question? 5. Were the appropriate validity criteria applied to the evidence appraised? 6. Are the dates clearly stated? Date of search, date of publication, date of expiry. (Dawes, 2005) Validation Criteria for CATs (cont.) What is the CAT s message? 7. How strong is the message? 8. Is it expressed in terms likely to be helpful in clinical management? Will the CAT s message help me in the care of my clients? 9. Can I apply the message in my client setting, to my clients? 10. Were all clinically important outcomes, benefits, harms and costs discussed? (Dawes, 2005) 6

7 Validation Criteria for CATs (cont.) Criteria for authority of authors, disclosure of competing interests, and feedback mechanisms 11. Is the academic or training level of the authors or commentators clearly stated? 12. Have the authors, site developers, and sponsors disclosed all competing interests? 13. Is there a mechanism to contact the authors? (Dawes, 2005) Limitations of CATs CATs can have shortcomings: Individual CATs can be wrong or inaccurate CATs appear first as drafts without peer review These first drafts may contain inferior evidence, or errors of fact, calculations, or interpretation can be turned into educational benefit: revise CATs in group discussions with clinicians or in other educational events Individual, one-article CATs contain only a single element of the relevant research literature These CATs are not comprehensive explorations for all useful articles Non-representative of the entire body of evidence Limitations of CATs (cont.) Individual CATs may have a short shelf life Become obsolete as soon as newer, better evidence becomes available Without constant updating for newer, better evidence, their clinical bottom line becomes out of date Good quality CATs specify update or expiration dates after which they should be considered obsolete (Centre for Evidence-Based Medicine, 2005) Possible difficulties Time pressure and heavy caseload of clinicians Development of the necessary skills for Formulating well-built, searchable question Searching the literature Critical appraisal (evaluating and rating evidence) can be overwhelming for busy clinicians Resources for instant access to the evidence where and when it is needed is lacking in most clinical/educational settings (Sutherland, 2004) Educational value Because they are client-centered and based on real-life clinical scenarios, CATs have appeal to clinical learners at all levels from student clinicians to faculty members (Sauvé, 1995) Focus on research evidence: Promotes the acquisition and polishing of literaturesearching skills Enhances critical appraisal and research evaluation skills Fosters translation of research into clinical practice Writing CATs can be part of pre-service training Include in educational curriculum, teaching units Established practitioners and researchers can benefit from journal clubs where designs, results and relevance of published articles are discussed and written up as a CAT Tools for dissemination It takes too long for each clinician to perform a critical appraisal of evidence for all clinical questions In the medical and health care fields professionals haves started systems to share CATs via CAT banks Make the information needed by clinicians more widely available and accessible CAT checked by a second person/expert 7

8 Tools for dissemination (cont.) CAT Crawler Dawes (2005) identified 13 major CAT banks for health care fields in North America, e.g., EBM journal ( ACP Journal club ( Evidence-Based on Call ( Meta-search engine, the CAT Crawler, available to search eight of the major CAT banks at one time Web-based application Finds relevant CATs following key word input High precision and recall EBP relevant journals for the SLP field New, secondary journals for review and critical appraisal of research in communication disorders: Communicative Disorders Review (since June 2006) Editor: Raymond D. Kent Publishes systematic and critical reviews, metaanalyses, and scholarly commentaries No CATs per se, but expert reviews that can be subject of a CAT Evidence-Based Communication Assessment and Intervention (started in 2007) Editors: Ralf W. Schlosser and Jeff Sigafoos Publishes CATs, original articles advancing review methodology, experiential accounts by clinicians, structured abstracts on research evidence Evidence-based Communication Assessment and Intervention Selects and appraises the latest and highest quality studies and reviews related to Assessment, intervention, diagnosis, and prognosis published across 60+ professional journals in speechlanguage pathology and related fields Published by Informa Healthcare 4 issues/year ISSN (print) (online) Evidence-based Communication Assessment and Intervention Seeks to publish original and exemplary Critically Appraised Topics (CATs) Full peer review To be published, CATs must meet criteria for content, currency of information, attribution, and documentation Software for writing CATs In the medical community, software has been written to facilitate the process of creating CATs and help clinicians save time Stand-alone programs: CATmaker ( Web-based forms: Criticial Appraisal Skills Programme (CASP) ( 8

9 CATmaker CATmaker (cont.) Available through the Oxford-Centre for Evidence Based Medicine For Windows or MacOS Prompts clinician for clinical question, search strategy and key information about the study Provides on-line critical appraisal guides for assessing validity and applicability of evidence Helps formulate clinical bottom line summary Creates 1-page summary print-out to store or publish on the web as html-file Reminder function to update CATs CATmaker (cont.) CASP Available through the Oxford-Centre for Evidence Based Medicine For Windows or MacOS Prompts clinician for clinical question, search strategy and key information about the study Provides on-line critical appraisal guides for assessing validity and applicability of evidence Helps formulate clinical bottom line summary Creates 1-page summary print-out to store or publish on the web as html-file Reminder function to update CATs BUT: tailored for medical research literature; limited utility for some research (e.g., single-subject designs) Further issues and discussion Questions??? What formats should CATs have in different subdisciplines of communication disorders? What resources and supports are needed to facilitate critical appraisal of evidence (e.g., time, EBP infrastructure, etc.)? How to teach CAT writing to current and future clinicians? How will we disseminate and share CATs within our community? Experiences of clinicians when writing CATs 9

10 References Center for Evidence-based Emergency Medicine (n. d. a.). Critically appraised topics bank. Retrieved January 11, 2002, from Centre for Evidence-Based Medicine (n. d. a.). What is a CAT? Retrieved December 7, 2005 from Dawes, M. (2005). Critically appraised topics and evidence-based medicine journals. Singapore Medical Journal, 46(9), 442. Makela, M., & Witt, K. (2005). How to read a paper: Critical appraisal of studies for application in healthcare. Singapore Medical Journal, 46(3), Sackett, D. L., Richardson, W. S., Rosenberg, W., & Haynes, R. B. (1997). Evidencebased medicine: How to practice and teach EBM. New York: Churchill Livingstone. Sauvé, S., Lee, H. N., Meade, M. O., Lang, J. D., Farkouh, M., Cook, D. J., & Sackett, D. L. (1995). The critically appraised topic: A practical approach to learning critical appraisal. Annales of the Royal College of Physicians and Surgeons of Canada, 28(7), Schlosser, R. W. (1999). Comparative efficacy of interventions in augmentative and alternative communication. Augmentative and Alternative Communication, 15, Schlosser, R. W. (2002). On the importance of being earnest about treatment integrity. Augmentative and Alternative Communication, 18, Schlosser, R. W. (2003). The efficacy of augmentative and alternative communication: Toward evidence-based practice. San Diego, CA: Academic Press. References (cont.) Schlosser, R. W., Costello, J., & Koul, R. (2005). Asking well-built questions for evidencebased practice in augmentative and alternative communication. Journal of Communication Disorders, 40(3), Schlosser, R. W., & Raghavendra, P. (2004). Evidence-based practice in augmentative and alternative communication. Augmentative and Alternative Communication, 20, Schlosser, R.W., Wendt, O., & Blischak, D.M. (manuscript in preparation). Effects of augmentative and alternative communication on natural speech production in autism: II. Developing a critically appraised topic. Schlosser, R. W., Wendt, O., Angermeier, K. L., & Shetty, M. (2005). Searching for evidence in augmentative and alternative communication: Navigating a scattered literature. Augmentative and alternative communication, 21 (4), Schlosser, R.W., Wendt, O., & Sigafoos, J. (in press). Not all systematic reviews are created equal: Considerations for appraisal. Evidence-based Communication Assessment and Intervention. Shannon, S. (2001). Critically appraised topics (CATs). Canadian Association of Radiologists Journal, 52(5), Sutherland, S. E. (2004). An introduction to systematic reviews. The Journal of Evidence- Based Dental Practice, 4, Wolf, M. M. (1978). Social validity: The case for subjective measurement, or how applied behavior analysis is finding its heart. Journal of Applied Behavior Analysis, 11, References (cont.) Ziegler, S., Lühmann, D., Raspe, H., & Windeler, J. (2001). Entscheidungsunterstützung durch Kurzstellungnahmen zur Evidenzlage. Zeitschrift für ärztliche Fortbildung und Qualitätssicherung, 95, Contact information Oliver Wendt, Ph.D. Department of Speech, Language and Hearing Sciences, and Department of Educational Studies BRNG 5156, Purdue University West Lafayette, IN , USA Phone: (+1) Fax: (+1) olli@purdue.edu 10