Searching for and Finding Evidence in Augmentative and Alternative Communication (AAC): Navigating a Scattered Literature Oliver Wendt, MS, Doctoral Candidate Purdue University American Speech-Language Hearing Association, Division 12: Augmentative and Alternative Communication (DAAC), 7th Annual Conference Presentation based on Schlosser, R. W., Wendt, O., Angermeier, K. L., & Shetty, M. (2005). Searching for evidence in augmentative and alternative communication: Navigating a scattered literature. Augmentative and alternative communication, 21 (4), 233255. What is Evidence-based Practice (EBP)? • “Evidence-based medicine is the conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients” (Sackett et al., 1996). • “…the integration of best and current research evidence with clinical/educational expertise and relevant stakeholder perspectives to facilitate decisions for assessment and intervention that are deemed effective and efficient for a given direct stakeholder” (Schlosser & Raghavendra, 2003, p. 256) The EBP Process 1. Developing a well-built question 2. Selecting evidence sources & executing the search strategy 3. Examining and synthesizing the evidence 4. Applying the evidence 5. Evaluating the application of the evidence 6. Disseminating the findings (Sackett et al., 1996 - Steps 1-5) Is this what it is like to look for evidence in AAC? The Role of the Search • AAC literature is scattered – It takes knowledge and skills (K & S) • Consequences of an inadequate search – Ignore pertinent evidence • (Erroneously) conclude that there is no evidence available – Over- or underestimate the support for a particular intervention – Misinterpret the applicability of the evidence Potential Sources of Evidence • • • • Textbooks Journals Newsletters Databases with pre-filtered evidence – "…an individual or group of individuals with expertise in a particular substantive area has reviewed and presented the methodologically strongest data in the field" (Melnyk and FineoutOverholt, 2002, p. 263, based on Guyatt and Rennie, 2002). • Database of Abstracts of Reviews of Effects (DARE) – http://www.york.ac.uk/inst/crd/welcome.html • General Purpose Databases – CINAHL, ERIC, LLBA, MEDLINE, PsycINFO • Internet • General Purpose Databases CINAHL Cumulative Index of Nursing and Allied Health Literatures ERIC Educational Resources Information Center http://www.eric.ed.gov/ Medline MEDLINE http://www.ncbi.nlm.nih.gov/entrez/query.fcgi LLBA PsycInfo Linguistics and Language Behavior Abstracts E-equivalent of Psychological Abstracts Assumptions for EBP Searches Characteristics Goal Type of search Search strategy Effort/Efficiency Use of quality fil ters Use of content keywords Systematic Reviews To retrieve all evidence and seek to avoid missing studies outside oneΥs regular purview Exhaustive searchΣ Multi-faceted DonΥt leave a stone unturned EBP To retrieve the best and most curre nt evidence first Typically no Best evidence searchΣ Multi-faceted Time-limi ted (high effi ciency preferred) Yes Yes Yes Assumptions Continued • Locate best & most current evidence first – Work down the hierarchy of sources • Database of Reviews of Effects (DARE) • Cochrane Database of Systematic Reviews • Seek out reviews before individual research studies • Look for peer-reviewed evidence before nonpeer-reviewed evidence About Quality Filters Database CINAHL Quality filters • Topical subheadings (SH): –diagnosis, evaluation, review, systematicreview, therapy, therapeutic use, utilization • Document type (DT): –Clinical trials, research ERIC • Publication type (PT): –Statistical data, journal article, evaluative/feasibility, research/technical • Keyword: – case studies, meta-analysis, review, quasiexperimental design Database LLBA Medline Quality filters –Journal articles only, Latest update • Publication type (PT): –Meta-analysis, review, academic review, tutorial • Medical subject heading (MeSH): –Clinical trials, research, research design, guidelines, consensus development conference, data collection, evaluation study, comparative study PsycINFO • Form/Content type (CT): –Clinical trial, double-blind design, empirical study, experimental replication, literature review, longitudinal study, meta-analysis, prospective study, single-blind design, treatment outcome study, treatment outcome Example 1 - Comparative Efficacy 1. Begin with scenario: – A team of practitioners and family members, serving a 6year old child with developmental disabilities (i.e., severe to profound mental retardation), have deemed it appropriate to introduce manual signing. They are, however, unsure what instructional strategy is most effective and efficient to yield successful expressive use and receptive learning 2. Consider your question: – “What instructional strategies are most effective and efficient in yielding expressive signing and receptive speech?” Example 1 Continued 3. Extract general keywords/limiters (quality filters) from question – Direct extractions • • • • • • • • – Manual signing, signing Developmental disabilities Instruction (treatment, intervention, therapy) Effectiveness Efficiency Expressive use Receptive speech Young children Extrapolations • • • Comparative (treatment) studies Group studies Single-subject experimental designs Example 1 Continued 4. Determine and prioritize appropriate evidence sources: • • • (a) DARE (b) General-purpose databases (prioritize: PsycINFO) (c) Internet (if needed) Studiesa CINAHL ERIC Barrera (1980) Barrett (1987) X Benne tt (1 986) X Berkowitz ( 1990) X MED- PSYC - LINE INFO Science X X X X X X X X X X X Belfiore (1993) LLBA Web of X Brady ( 1996) X X X Brady ( 1978) X X X X Carr (1984) X X X X Clarke (1986) X X X X X X X Clarke (1988) Conaghan (1992) X X X Studiesa CINAHL ERIC MED- PSYC - LINE INFO Dalrymple (1992) X X Duker (1994) X X Goodman (1993) X Hurlbut (1 982) Ia cono (1995) X X X Lim (1998) X X X X X X X X X X X X (1984) Kozleski (1991) Science X X Ia cono (1993) Konstantareas Web of X Ducker (1986) Horn (1996) LLBA X X X X X Studiesa CINAHL MED- PSYC - LINE INFO Science Linton (1 984) X X McNaughton (1993) X Oliver (1983) ERIC Web of X X Parsons (1993) Reichle (2000) LLBA X X Remington (1983) X X X X X X Remington (1993a) X X Remington (1993b) X X Rotholz (1989) X X X Schlosser (1995) X X X X X Schlosser (1998) X X X Studiesa CINAHL ERIC MED- PSYC - LINE INFO LLBA Web of Science Sigafoos (1995) Sigafoos (1992) X Sisson ( 1984) X Soto (1993) X X X Sundb erg (1990) X Wraikat (1991) X X X X X Wells (1981) Wherry (1983) X X X Watters (1981) Webster (1973) X X Van Acker (1995) Vaughn (1995) X X X X X X X X X X X X X – AAC-relevant journals and their inclusion in various databases CINAHL 1. Advances in Speech Language Pathology x 2. Augmentative and Alternative Communication x DARE x 3. Australasian Journal of Special Education 6. Analysis of Verbal Behavior LLBA x x x x MEDLINE PsycINFO Web of Science x x 4. American Journal on Mental Retardation 5. American Journal of Speech-Language Pathology Practice ERIC x x x x x x x (from 2003) x x x x CINAHL DARE ERIC PsycINFO Web of Science x x x x x x 9. Assistive Technology x x 10. Australia and New Zealand Journal of Developmental Disabilities 11. Australian Journal of Human Communication Disorders 12. Autism x 7. Aphasiology 8. Applied Psycholinguistics x x LLBA MEDLINE x x x CINAHL DARE ERIC LLBA 13. Behaviour Change 14. Behavior Modification 15. Behavioral Residential Treatment 16. British Journal of Developmental Disabilities 17. Communication Disorders Quarterly 18. Education and Training in Mental Retardation 19. Education and Treatment of Children 20. Exceptional Children x x MEDLINE PsycINFO Web of Science x x x x x x x x x x x x x x x x x x x x 21. Exceptionality x x x 22. Focus on Autism and Other Developmental Disabilities x x x x x x x x CINAHL 23. Journal of Developmental and Physical Disabilities 24. Journal of Experimental Child Psychology 25. International Journal of Disability, Development and Education 26. International Journal of Language and Communication Disorders 27. Journal of Applied Behavior Analysis 28. Journal of Applied Research in Intellectual Disabilities 29. Journal of Autism and Developmental Disorders 30. Journal of Behavioral Education 31. Journal of Childhood Communication Disorders 32. Journal of Communication Disorders DARE ERIC LLBA MEDLINE x Web of Science x x x x x x x x x x x x x x x x x x x x x x x x x x x x PsycINFO x x x x x x x x x x x x x x x CINAHL DARE ERIC LLBA MEDLINE PsycINFO Web of Science x x x x x x x x x x x x x x x x 36. Journal of Special Education x x x x x 37. Journal of Special Education Technology 38. Journal of Speech, Language, and Hearing Research 39. Journal of the Association for Persons with Severe Handicaps 40. Journal of Visual Impairment and Blindness 41. Language, Speech, and Hearing Services in Schools 42. Mental Handicap Research x x x x x x x x x x x x x x x x x x x x x x x 33. Journal of Intellectual Disability Research 34. Journal of Psycholinguistic Research 35. Journal of Mental Deficiency Research x CINAHL DARE ERIC 43. Mental Retardation 44. Mental Retardation and Learning Disabilities Bulletin x 45. Remedial and Special Education x 46. Research in Developmental Disabilities 47. Sign Language Studies 48. Topics in Early Childhood Special Education x LLBA MEDLINE x x Web of Science x x x x x PsycINFO x x x x x x x x x x Example 1 Continued 5. Implement search for reviews - DARE • Augmentative communication, alternative communication, assistive technology, sign language, manual sign 6. Examine results (reviews) - DARE augmentative communication/All fields OR alternative communication/All fields OR assistive technology/All fields - 2 Hits (1-2) (Record 1) Promoting generalization and maintenance in augmentative and alternative communication: a meta-analysis of 20 years of effectiveness research. Schlosser R W, Lee D L. AAC: Augmentative and Alternative Communication 2000; 16(4): 208-226. (Record 2) Speech and language therapy to improve the communication skills of children with cerebral palsy. Pennington L, Goldbart J, Marshall J. Speech and language therapy to improve the communication skills of children with cerebral palsy (Cochrane Review). In: The Cochrane Library, Issue 4, 2004. Chichester, UK: John Wiley & Sons, Ltd • sign language/All fields OR manual sign/All fields - No Hits 5. Implement search for reviews - PsycINFO Evid ence Tab le Study Psyc INFO? Table 1 Comparative Studies Invo lving Una ided AAC Modes: Sim ult aneou s Comm unication, Sign-alone Instruction, and Oral Instruction Barrett (1987) McDona ld (1977) Remington (1983) Sisson (1984) Wells (1981) Yes No Yes Yes Yes Clarke (1986) Conaghan (1992) Ducke r (1986) Duke r (1994) Goodman (1993) Linton (1994) Remington (1993a) Remington (1993b ) Benne tt (1986) Dalr ymple (1992) Iacono (1986) Wolery (1994) Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes No Yes Table 2. Comparative studies invo lving una ided approache s: Determi ning the nec essary cond iti ons for sim ultaneous comm unication Table 3. Comparison studies invo lving una ided approache s: Other instruc tiona l strategies Schlosser, R. W., & Sigafoos, J. (2006). Augmentative and alternative communication interventions for persons with developmental disabili ties: Narrative review of comparative single-subject experimental studies. Research in Developmental Disabilities, 27, 1-29. • How to get the other ones not in database? – McDonald, L. (1977). A comparison of three methods of word imitation training with Down’s Syndrome children under six years of age. Unpublished master’s thesis. University of Manitoba, Winnipeg. Canada. • ---> AAC Thesis and Dissertations (Lloyd, Koul, & Arvidson, 1993, 1994; Lloyd, Arvidson, & Koul, 1996) – Iacono, T., & Parsons, C. (1986). A comparison of techniques in teaching signs to the intellectually disabled using an alternating treatments design. Australian Journal of Human Communication Disorders, 14, 23-34. • ----> Handsearch Prepare search - PsycINFO Study Keywo rds (sub jects) “compar *” Form/C ont ent type Classifi cation Barrett (1987) Comm unication skill s training, moderate mental retardation, speech th erapy Yes Empiri cal study Speech & language therapy Benne tt (1986) Manua l Communication ; Mental Retardation; Multi ple Disabiliti es; Special Edu cation Studen ts; Teaching Methods Seve re Mental Retardation; Sign Language; Vocabulary; Language Arts Educa tion; Special Educ ation Studen ts Comm unication Skills Training; Pheny lketonuria; Seve re Mental Retardation; Sign Language; Verbal Comm unication Hea ring disorders, mental retardation, positi ve reinfo rceme nt, practice, sign language, comm unication skill s training, gene rali zation (learning ) Yes Empiri cal study Special & Remedial Educa tion Alt ernati ng Empiri cal study Special & Remedial Educa tion Yes Empiri cal study Speech & Language Ther apy Yes Empiri cal study Interpersona l & Cli ent-centered humanistic therapy Clarke (1986) Clarke (1988) Conaghan (1992) Study Keywo rds (sub jects) “compar*” Form/C ont ent type Classifi cation Dalr ymple (1992) Moderate Mental Retardation; Practice; Rein forcement; Severe Mental Retard ation; Sign Language; Nonverb al Learning Behav ior Modifi cation; Down s Synd rome; Instit utiona li zed Mentally Retarded ; Reinfo rcement; Sign La nguage Comm unication Skills; Profound Mental Retardation; Severe Mental Retardation; Stim ulus Control Rein forcement; Severe Mental Retardation; Sign Language; Special Educa tion Studen ts Mental retardation, ove rcorrection, positi ve reinfo rceme nt, practice, sign l ang uage, Comprehen sion, learning disabiliti es, seve re mental retardation, sign l ang uage; special educa tion studen ts; comm unication skill s training, selective Attention Yes Emp iri cal study Rehab il itation Yes Emp iri cal study Hospital & Impatient Services Dif fe rential effectiveness, Vs. Emp iri cal study Behav ior Ther apy & Behav ior Modifi cation Yes Emp iri cal study Special & Remedial Educa tion Alt ernating Emp iri cal study Rehab il itation Yes Emp iri cal study Special & Remedial Educa tion Ducke r (1986) Duke r et al. (1994) Good man (1993) Lin ton (1994) Remington (1993a) Study Keywo rds (sub jects) “compar*” Form/C ont ent type Classifi cation Remington (1993b) Comm unication Skills Training; Comprehens ion; Seve re Mental Retardation; Sign Language; Special Educa tion Studen ts; Selective Attention Comm unication skill s training, mil d mental retardation, moderate mental retardation, oral comm unication, sign language Mental retardation, ve rbal learning, videotape instruction, oral comm unication, sign language, Articulation (Speech); Comm unication Skills Training; Sever e Mental Retardation; Sign Language; Speech Ther apy Delayed Dev elop me nt; Nami ng; Preschoo l Studen ts; Read ing; Teaching Methods Yes Empiri cal study Special & Remedial Educa tion Yes Empiri cal study Behav ior therapy & behav ior modific ation Yes Empiri cal study Special & Remedial Educa tion Yes N/A Speech & Language The rapy Yes Empiri cal study Special & Remedial Educa tion Sisson (1984) Watkins et al. (1990) Wells (1981) Wolery (1993) • What can we glean from this backward process? – Sign language • Most widely and consistently used KW • If not used, other alternatives are – Manual communication – Communication skills training – ( ) Mental retardation • is best to capture the population – Quality filter: • Empirical study (only one without it) • “compar*” as a free-text word works for most 5. Implement search for reviews & 6. Examine Results - PsycINFO -12 references, but none focused on topic (autism, not treatment focused) ---> refine search -3 references, but none focused on topic (autism, not treatment focused) ----> Need to look for individual studies 5. Implement search for studies - PsycINFO Add limiters (quality filters) 6. Examine the results (studies) - PsycINFO 6. Examine results (& revise & search again) revise search (and implement again) 6. Examine the results (studies) - PsycINFO Example 1 Lessons • Use the thesaurus to build your search • When you found an appropriate reference, check it out in terms of indexing • Use synonyms or terms that describe similar concepts • Be mindful - terminology changes, but the indexing may not be retro-active • Don’t forget the “TX” option • Consider to use truncation • Trust no one! Indexers are not perfect. • “Pearl Growing” can be a beneficial EBP search strategy (if you have a relevant article to start with) Pearl Growing • Pearl Growing involves the following process – (1) Find a relevant article; – (2) find the terms under which the article is indexed in database-1; – (3) find other relevant articles in database-1 by using the index terms in a Building Block query; – (4) repeat 2-3 in other databases; – (5) repeat 1-4 for other relevant articles; and – (6) end when articles retrieved provide diminishing relevance. An Illustration of Pearl Growing • Question – What strategies are most effective and efficient for introducing manual signs to children with developmental disabilities in terms of expressive signing, and/or expressive natural speech, and/or receptive speech? • (1) Our Pearl – Clarke, S., Remington, B., and Light, P., 1988, The role of referential speech in sign learning by mentally retarded children: A comparison of total communication and sign-alone training. Journal of Applied Behavior Analysis, 21, 419-426. • (2) Under which terms is the pearl indexed in database 1 Medline? – Medical Subject Heading terms (italic = most relevant): • Child; Child, Preschool; Communication Methods, Total; Comparative Study; Female; Humans; Imitative Behavior; Male; Manual Communication; Mental Retardation/rehabilitation; Rehabilitation; Research Support, Non-U.S. Gov't; Sign Language; and Speech. – the “pearl” was not indexed with a particular publication type • The MeSH term “comparative study” can serve to filter out studies that involve only one treatment. – Checked the MeSH database http://www.nlm.nih.gov/mesh/MBrowser.html to identify other relevant terms to capture this population • found “developmental disabilities” and “autistic disorder.” – (3) find other relevant articles using terms in Medline • First goal is to identify reviews: – [(communication methods, total OR sign language OR manual communication) AND (mental retardation OR developmental disabilities OR autistic disorder)] using “child,” “human,” and sequentially “meta-analysis,” “review,” & “practice guideline” as limiters. » Using “review”- Yield: 9 entries - three were applicable (Bondy & Frost, 1998; Howlin, 1988; Wilken, 1996). • Second goal is to identify individual studies: – [(communication methods, total OR sign language OR manual communication) AND comparative study AND (mental retardation OR developmental disabilities OR autistic disorder) AND speech)] using “child” and “human” as limiters » Yield: 3 additional relevant references (Brady & Smouse 1978; Layton 1988; Wells 1981). » PT use varied from case report to clinical trial to RCT – Repeat search without “speech” » Yield: 9 references - 4 were applicable (Barrera & Sulzer-Azaroff 1983; Kahn 1981; Sisson & Barrett 1984; Wherry & Edwards, 1983) » PT use varied from none to case report to clinical trials to RCT » This variation in quality filters along with the absence of any additional keywords made it unnecessary to conduct more MEDLINE searches. – (4) repeat steps 2-3 in PsycINFO • This pearl is indexed under the following Descriptors: – communication skills training, phenylketonuria, sign language, severe mental retardation, and verbal communication – Form/Content Type of “empirical study” as a quality filter – This provided some valuable keywords (in italics) and one quality filter to start with – We supplemented other relevant keywords to better describe the population of developmental disabilities using the thesaurus • First goal is to identify reviews: – [(DE "Sign Language" or DE "Verbal Communication") and (DE "Communication Skills Training") and (DE "Mental Retardation" or DE "Developmental Disabilities" or DE "Pervasive Developmental Disorders")] using literature review and meta-analysis as quality filters. » Yield: No hits • Second goal is to identify individual studies: – As above with limiters: preschool child, schoolage child, and adolescent » Yield: 7 hits with 6 meeting our relevancy criteria (Bonta & Watters 1983; Braam & Poling 1983; Bucher 1983; Gaines et al., 1988; Light et al., 1990; Remington et al.,1990). » All of these hits were indexed under the form/content type of “empirical study” (i.e., a quality filter), just like the “pearl.” » An analysis of keywords revealed no viable additional ones and so we abandoned the search for more entries Pearl Growing Summary Features Purpose Information Source Needed Functions Representativeness Effort Pearl Growing 1. To aid a current EBP search 2. One or a few relevant article/s – the “pearl/s” (a very small sample of the population of relevant studies) Identifies relevant keywords Identifies relevant quality filters Suggests indexing problems May not get typical keywords and quality filters (i.e., a “pearl” in terms of content may not be one in terms of keywords and quality filters) Fairly time efficient Example 2 - Speech Production • 1. Start with scenario • Sam is a 4-year old child who was recently diagnosed with autism. He is unable to meet his daily communication needs in his preschool through natural speech. He has recently learned to imitate words such as “mama,” “dada,” “quack-quack,” and “bye-bye.” To date, Sam does not use these words to communicate. His family and the staff at his preschool anticipate many of his needs and consistently respond to his prelinguistic communication behaviors such as touching objects or leading people to objects he wants or activities that he would like to do. Sam’s parents hope that he will eventually speak and be included in a classroom with typically-developing children. Prior to Sam’s diagnosis, his parents were not interested in exploring other forms of communication because they had serious concerns about the impact on Sam’s potential for developing speech. Their current goals for Sam are that he improve his speech and communicate more effectively through whatever means are appropriate and supportive of speech development. • 2. Consider your question: • His parents want to know, “Which AAC approach(es) best support natural speech production?” Example 2 Continued 3. Extract potential keywords/limiters from question – Direct extractions • • • • • • • • Autism Presymbolic (Prelinguistic) Instruction (treatment, intervention, therapy) Augmentative communication, alternative communication Effectiveness/efficiency Preschool children Settings: special education, toward inclusion Speech production – Extrapolations • • • • • Prognosis Comparative (treatment) studies Group studies Single-subject experimental designs Social validation Example 2 Continued 4. Prioritize your search – DARE – (Conference Proceedings) – General-purpose databases 5. Implement your search - DARE Broaden your search 6. Examine Results 5. Implement search for reviews - PsycINFO 6. Examine the results (reviews) -PsycINFO 5. Implement search for reviews - ERIC 6. Examine the results (reviews) - ERIC 5. Implement search for reviews - Medline – First go for meta-analyses - Broaden to other reviews 5. Implement search for reviews - CINAHL 6. Examine the results (reviews) - CINAHL 5. Implement search for reviews -CINAHL 6. Examine the results (reviews) - CINAHL 5. Implement search for reviews - CINAHL 6.Examine the results (reviews) - CINAHL Searching conference proceedings latest reviews Special issue on speech output: • Yielded 1 relevant review • Blischak, D. M., Dyson, A. T., & Lombardino, L. J. (2003). Use of speech-generating devices: In support of natural speech. Augmentative and Alternative Communication, 19, 29-35. ISAAC proceedings 2000, 2002, 2004: Yielded 1 systematic review Millar, D., Light, J., & Schlosser, R. (2000). The impact of AAC on natural speech development: A meta-analysis. In Proceedings of the 9th biennial conference of the International Society for Augmentative and Alternative Communication (pp. 740-741). Washington, DC: ISAAC. ASHA Convention Abstracts 2001, 2002, 2003: Yielded 1 systematic review Correa, N., & Nye, C. (2001). Sign language and autism: A quantitative synthesis of single-subject research. Poster presented at the Annual Convention of the American Speech-Language Hearing Association. ASHA Leader, 6(15), 189. Let’s take stock of reviews • 2 systematic reviews – 1 on AAC, natural speech production & DD – 1 on manual signing and autism • ---> examine for individual studies • 13 narrative reviews – Mirenda (2003), Blischak et al. (2003), Goldstein (2002), Mirenda (2001), Schlosser & Blischak (2001), Koul et al. (2001), Sigafoos & Drasgow (2001), Bondy (2001), Bondy (1998), Mirenda & Schuler (1988), Hedbring (1985), Kiernan (1983), Bonvilllian (1981) • ---> examine for sections on autism & natural speech production & individual studies 5. Implement search for studies - PsycINFO Add limiters (quality filters) Drop the peer-reviewed part Zero in on speech production 6. Examine results (studies) - PsycINFO (and revise strategy) 5. Implement search for studies again - PsycINFO Add relevant free-text terms 6. Examine the results (studies) again - PsycINFO 5. Implement search for studies - ERIC 6. Examine the results (studies) - ERIC 5. Implement search for studies - CINAHL 6. Examine the results (studies) - CINAHL Taking stock of studies • From the two systematic reviews: – Barrera, Lobato-Barrera, & Sulzer-Azaroff, 1980, Benaroya, Wesley, Ogilvie, Klein, & Meaney, 1977; Bondy & Frost, 1994; Bonta & Watters, 1983; Casey, 1978; Fulwiler & Fouts, 1976; Kouri, 1988; Yoder & Layton, 1988 • From database searches: – Ganz (2004), Tincani (2004), Sigafoos et al. (2003), Charlop-Christy et al. (2002), Anderson (2002), Forsey (1996), Buday (1995), Kouri (1988), Yoder & Layton (1988), Shimizu (1988), Carr (1984), Ferrarese (1982), Creekmore (1982), Konstanteras (1979) EBP Search Example 3: AAC and Aphasia 1. Begin with Scenario: A clinician has a patient with aphasia on the caseload and needs to know what AAC intervention strategies have been used in similar cases and what were the results. 2. Consider your question: What AAC intervention strategies have been used successfully with patients with aphasia? – type of aphasia not further specified at the beginning of the search to screen a broad range of studies that can be narrowed down later EBP Search Example 3: AAC and Aphasia (cont.) 3. Extract general keywords/quality filters from question – Direct extractions • • • • • • • • Aphasia AAC Augmentative and alternative communication Communication Aided communication Unaided communication Sign language Intervention EBP Search Example 3: AAC and Aphasia (cont.) 4. Determine and prioritize appropriate evidence sources: • Start with specialized databases for pre-filtered evidence: – • Cochrane/DARE: 6 hits on “augmentative and alternative communication”, none related to aphasia; 243 hits on “aphasia” but AAC not included in any of them General databases better choice, most appropriate for aphasia related topics: – – – (a) PsycINFO (b) MEDLINE (c) Cumulative Index of Nursing and Allied Health Literatures (CINAHL) EBP Search Example 3: AAC and Aphasia (cont.) 5.(a) Implement Search - PsycINFO: • Select appropriate keywords, chose major PsycINFO headings: – – – To locate aphasia relevant entries: 1. Enter “aphasia” (already major heading); perform search To locate AAC relevant entries: 2. Enter “augmentative communication”, map it to subject headings, then also chose “sign language”; ignore “nonverbal communication” and “communication systems” because no additional results; perform search 3. Enter “symbolism” (major heading) to retrieve symbolrelated entries; perform search EBP Search Example 3: AAC and Aphasia (cont.) 5. (a) Implement Search - PsycINFO (cont.): – Combine results from (2.) and (3.) to summarize all AAC entries using OR operator – Combine this summary with (1.) using AND operator to obtain results relevant to AAC and aphasia 89 entries EBP Search Example 3: AAC and Aphasia (cont.) 5. (a) Implement Search - PsycINFO (cont.): – Narrow down results using appropriate quality filters: • Click on “Limit”, on next screen check general filters “English Language” and “Peer Reviewed Journals” 64 entries, peer-review quality • Several filters available under “Form/Content Types”, select: “empirical study” 47 entries, “quantitative study” 2 entries, “literature review” 3 entries, “clinical case report” 15 entries “qualitative study” 1 entry EBP Search Example 3: AAC and Aphasia (cont.) 6. (a) Examine Results • Final results for PsycINFO search: 64 entries from peer-reviewed journals “empirical study” 47 entries, “quantitative study” 2 entries, “literature review” 3 entries, “clinical case report” 15 entries “qualitative study” 1 entry Need to review these results in terms of content (relevance to individual case) and research design (quality of evidence) Save search for later use, print out or e-mail search results EBP Search Example 3: AAC and Aphasia (cont.) EBP Search Example: AAC and Aphasia (cont.) EBP Search Example 3: AAC and Aphasia (cont.) 5. (b) Implement Search - MEDLINE – Select appropriate keywords, consider Medical Subject Headings (MeSH): To locate aphasia relevant entries: – 1. Enter “aphasia” (already a MeSH term); perform search To locate AAC relevant entries: – 2. Enter “graphic symbols”, map it to subject headings, then chose MeSH terms “Communication Aids for Disabled”, “Symbolism” and also check “graphic symbols” as keyword; ignore “communication” because it reveals too many unrelated entries; perform search – 3. Enter “manual communication” (MeSH term), perform search – 4. Enter “communication methods, total” (MeSH term), perform search EBP Search Example 3: AAC and Aphasia (cont.) 5. (b) Implement Search - MEDLINE (cont.) – Combine results from (2.), (3.), and (4.) to summarize all AAC entries using OR operator – Combine this summary with (1.) using AND operator to obtain results relevant to AAC and aphasia 69 entries – Narrow down results using appropriate quality filters: • First, try to locate pre-filtered evidence checking “Limit to: ‘Review articles’ and ‘Systematic Reviews’” reveals no results, field not advanced to that level yet, need to descend in evidence hierarchy • Click on “Limit”, on next screen chose filters “English Language” and under Publication Types chose “Case Reports” EBP Search Example: AAC and Aphasia (cont.) 6. (b) Examine results: 18 “case reports” on AAC and aphasia, 13 are new entries not listed in the original 62 from PsycINFO “case reports” not clearly defined, there is still the need to review these studies in terms of design quality (e.g., whether experimental single subject study or narrative report) review titles and abstracts (if needed) to identify those most relevant to your case Save search for later use, print out or e-mail search results EBP Search Example 3: AAC and Aphasia (cont.) EBP Search Example: AAC and Aphasia (cont.) EBP Search Example 3: AAC and Aphasia (cont.) 5. (c) Implement Search - CINAHL: – Select appropriate keywords, chose major CINAHL headings: To locate aphasia relevant entries: – 1. Enter “aphasia” (already major heading); perform search To locate AAC relevant entries: – 2. Enter “AAC”, map it to subject headings, then chose “Alternative and Augmentative Communication” and “Communication Aids for Disabled”; ignore “non-verbal communication” because it does not lead to additional results; perform search – 3. Enter “sign language” (major heading) to retrieve unaided communication modes; perform search EBP Search Example 3: AAC and Aphasia (cont.) 5. (c) Implement Search - CINAHL (cont.): – Combine results from (2.) and (3.) to summarize all AAC entries using OR operator – Combine this summary with (1.) using AND operator to obtain results relevant to AAC and aphasia 33 entries EBP Search Example 3: AAC and Aphasia (cont.) 5. (c) Implement Search - CINAHL (cont.): – Narrow down results using appropriate quality filters: • Click on “Limit”, on next screen check general filters “English” and “Research” 16 entries, research-based • Under “Journal Subsets” select “Peer Reviewed Journals” 13 entries, peer-review quality • Look at quality filter “Publication types”, several options: “clinical trial”, “systematic review” and “review” reveal no hits, “case study” results in 4 entries, if these are applicable distinguish between single-subject research designs and narrative case reports • Interesting, but not yet applicable (no results at this point): Under “Special Interest Category” filter on “Evidence Based Practice” EBP Search Example 3: AAC and Aphasia (cont.) 6. (c) Examine Results - CINAHL: 16 research based entries, 13 of which did not appear in MEDLINE, and 8 not in either MEDLINE or PsycINFO 13 peer-reviewed 4 “case studies” Again, need to review for quality of research design Review titles and abstracts (if needed) to identify those most relevant to your case Save search for later use, print out or e-mail search results EBP Search Example 3: AAC and Aphasia (cont.) EBP Search Example: AAC and Aphasia (cont.) – Locating in press-articles through • Publisher websites (e.g., MetaPress, Science Direct, etc.) EBP Search Example 3: AAC and Aphasia (cont.) 6. (a, b, c) Examine overall results from PsycINFO, MEDLINE, and CINAHL: • 83 unique entries covering range of aphasia subtypes, interventions, research designs, and range in quality (!) Need to review these results in terms of content (relevance to individual case) and research design (quality of evidence) Start with information embedded in title, then read abstract, and if the content is still unclear retrieve complete article to review in further detail Example 3 Lessons • No single database covers all relevant literature on a given topic • Each database requires slightly different search strategy re: combination of key words and phrases • Even if correct keywords are used, irrelevant results may still show up (accuracy of indexing), • Databases vary in terms of available filters – Not all filters are helpful • Databases use different terms and sometimes ambiguous labels to classify filtered search results – Example: “clinical case reports” can cover qualitative case study or single-subject research design – Filtered results still need to be reviewed by hand in further detail re: quality of the research design EBP Search Example 4: Efficacy of AAC in Autism 1. Begin with scenario: A researcher aims to identify the research base for AAC in autism, wants to investigate effectiveness of AAC interventions for individuals with autism, and decides to conduct quantitative synthesis of intervention literature (meta-analysis) EBP Search Example 4: Efficacy of AAC in Autism 2. Consider your question(s): - What AAC interventions (e.g., aided, unaided, partner training, etc.) are effective in terms of behavior change, generalization, and maintenance in individuals with autism? - What AAC interventions are effective in yielding specific outcomes (e.g., language acquisition, functional communication training, participation, etc.)? - Which AAC interventions are more effective than others in general and in yielding specific outcomes? - What are research gaps and methodological gaps in the intervention literature? Note: more comprehensive approach of systematic review EBP Search Example 4: Efficacy of AAC in Autism 3. Extract general keywords/quality filters from question – Direct extractions • • • • • • • • • – AAC Augmentative and alternative communication Communication Aided communication Unaided communication Manual signing, signing Aided Language Stimulation Augmented Input Autism Extrapolations • • Group studies Single-subject experimental designs EBP Search Example: Efficacy of AAC in Autism • Note different approach of searches for systematic reviews (as outlined above), particularly: • • • Multi-faceted, i.e., different searches and search strategies complement one another Targeted, but broad and very comprehensive because of need to locate and finally evaluate all existing research studies out there What can be learned from systematic reviews to enhance searches for EBP? - Evidence for AAC scattered widely, often scarce and difficult to locate through traditional channels (e.g., databases) Search strategies common for systematic reviews can be of benefit when evidence is hard to find improve EBP search through multiple strategy approach EBP Search Example 4: Efficacy of AAC in Autism 4. Determine and prioritize appropriate evidence sources: Focus on computerized database searches covering • • • • • Cumulative Index to Nursing and Allied Health Literature (CINAHL) Dissertation Abstracts International Educational Resources Information Center (ERIC) Language and Linguistics Behavior Abstracts (LLBA) PsychINFO EBP Search Example 4: Efficacy of AAC in Autism 5. Implement search (a) Computerized databases with selected keywords as outlined above Plus additional search strategies (multi-faceted search): (b) Hand Search - "AAC Theses and Dissertations” • • • published report "AAC Theses and Dissertations" (Lloyd, Koul, & Arvidson, 1993, 1994) in Journal of AAC includes abstracts and bibliographic information of 73 theses and dissertations facilitates the further retrieval of unpublished theses EBP Search Example 4: Efficacy of AAC in Autism 5. Implement search (cont.) (c) Hand Search - Journals • 39 professional journals, identified as possible publication outlets for AAC interventions • Covers journals that may not be indexed in databases • Hand search does not rely on key word indices provided by the journals; the accuracy of these indices always depends (e.g., judgment of authors, indexers/editor) • How to hand search: systematic search of table of contents for relevant titles; if titles appear relevant, examine the abstract for further relevance EBP Search Example 4: Efficacy of AAC in Autism 5. Implement search (cont.) • Selection criteria rather than quality filters; applied by researcher when reviewing obtained literature: • Intervention must pertain to AAC (according to ASHA 1989 definition; Facilitated Communication literature excluded) • Intervention target must include individuals with autism spectrum disorders (ASD) • Experiment employs single-subject or group design • Experiment needs to be written as an article in a refereed journal, documents available through ERIC, or unpublished MS/doctoral theses • Dated between 1976 and 2004 EBP Search Example 4: Efficacy of AAC in Autism 5. Implement search (cont.) • Selection criteria rather than quality filters; applied by researcher when reviewing obtained literature: • Intervention must pertain to AAC (according to ASHA 1989 definition; Facilitated Communication literature excluded) • Intervention target must include individuals with autism spectrum disorders (ASD) • Experiment employs single-subject or group design • Experiment needs to be written as an article in a refereed journal, documents available through ERIC, or unpublished MS/doctoral theses • Dated between 1976 and 2004 EBP Search Example 4: Efficacy of AAC in Autism 6. Examine Results • • • Literature search revealed a total of 270 publications related to AAC & autism (excluding FC) Empirical studies: 134 – Descriptive clinical case studies: 7 – group designs: 13 – longitudinal designs: 3 (2 descr./1 experim.) – single-subject designs: 81 – design flawed, etc. : 30 95 studies met inclusion criteria re: research design quality Example 4 Lessons • • • • Searches for systematic reviews are different from EBP searches, more comprehensive and multiple search strategies that complement one another Electronic databases, even if combined, do not reveal all available evidence – AAC relevant journals may not be indexed – Database indeces may not be accurate “Nontraditional ways” to search can reveal further useful evidence, i.e., hand searches and ancestry searches, but practitioners have to decide how feasible – Hand search in example lead to 28 additional studies that did not show up in databases Research evidence can also be found in unpublished literature such as PhD and MS theses (although not peer reviewed); 6 PhD and 1 MS theses located in example Example 4 Lessons (cont.) • In this particular case of AAC & autism: – – Search reveals lots of anecdotal evidence, less data-based Considerable amount of studies with flaws in research design or insufficient (pre-experimental) design quality Researchers and clinicians need to be cautious in interpreting and evaluating these findings, need to keep EBP guidelines and evidence hierarchy in mind Further Sources for EBP Searches • ProQuest Digital Dissertations formerly known as Dissertation Abstracts International (DAI) • Linguistics and Language Behavior Abstracts (LLBA) • Web of Science • Internet – – – – General search engines Specialized search engines Meta-searches and meta-search engines Evaluation of Internet information ProQuest Digital Dissertations (PQDD) • PQDD is an online version of the print counter-part Dissertation Abstracts International (DAI) • Dissertations and theses may contain valuable research findings • Exclusion can introduce bias in the EBP process • Most theses undergo internal peer-review process, i.e., acceptance by committee • PQDD holds more than 2 million entries from 1861 to the present (updated each semester) ProQuest Digital Dissertations (PQDD) • More than 1.7 million full-text versions available in paper or microform, can be ordered online • More than 450,000 full-text versions available in digital format for online download from Dissertation Express website – Entries from 1997 forward • Only last 2 years can be accessed free of charge • Broader searches require subscription • Downloading full-text versions often requires additional fee payment Example 5: AAC & Early Intervention 5. Implement search - CINAHL 6. Examine the results - CINAHL Example 5 continued 5. Implement search - PsycINFO 6. Examine the results - PsycINFO ProQuest Digital Dissertations (PQDD) • Brief search example: AAC & Early intervention • Keywords: – “augmentative communication” 65 hits – “alternative communication” 86 hits – “augmentative and alternative communication” 51 hits – “AAC” 156 hits, but many non-relevant – “sign language” 621 hits – “sign language” AND NOT “deaf” 204 hits • No filters available for research design quality ProQuest Digital Dissertations (PQDD) Online Demo 5. Implement search - expand to the internet (ixquick http://www.ixquick.com) • 5. Implement search - expand to the inter-net (Dogpile http://www.dogpile.com) • 5. Implement search - expand to the internet (scirus http://www.scirus.com) ISI Web of Science • Five high-quality databases containing information from journals in all research areas – Three relevant for AAC: • Science Citation Index Expanded • Social Sciences Citation Index • Arts & Humanities Citation Index – Contain references cited by authors of indexed journals – Option to use forward citation search where cited references are used as search terms – Can’t delimit citations to meet design criteria ISI Web of Science (cont.) – May also search by topic, author, source title, and address – Searching by author may be more cost-efficient because of possible fees for forward citation search – Requires either an individual or university library subscription ISI Web of Science (cont.) Searching for evidence on the Internet • Internet offers a vast array of information pertaining to AAC • Challenge when obtaining too many sites of interest but different quality • Lack of quality control is a major concern • Points to consider when conducting evidence searches – Knowing which search engines will yield best results – Knowing how to assess the quality of the retrieved information General search engines • Search engine: – Software program built on top of large databases containing web page files – Matches the database content to specified search terms – Databases compiled through “robots”, ”spiders”, or “crawlers (programs searching and indexing webpages) – Each search engine uses own set of criteria to decide what to include in its database – Different ways to organize search results for users, e.g., link popularity vs. common themes General search engines (cont.) • Web directories: – Instead of “spiders” human editors review and index links – Guidelines for web sites to be included in index – Directory editor looks at quality of a site: functionality, content and design – Directory indexes • small number of, but higher quality links – Example: Yahoo! (in the past), Open Directory Project (http://dmoz.org) General search engines (cont.) • Hybrid search engines: – Latest generation – Combination of traditional search engine with a directory – Top ten search sites are hybrids – Example: Google uses Open Directory Project to supplement its automatically generated listings (Shapiro & Lehoczky, 2004) General search engines (cont.) • Most popular search engines by percentage of home and work users (as of June 2004) – Google, http://www.google.com (41,6%) – Yahoo, http://www.yahoo.com (31,5%) – MSN, http://www.msn.com (27.4%) – AOL Search, http://www.aolsearch.com (13.6%) – Ask Jeeves, http://www.ask.com (7.0%) – Overture, http://www.overture.com (5.1%) (Source: Nielsen//Netratings, 2004) – Others: Altavista (http://www.altavista.com); Hotbot (http://www.hotbot.com); Lycos (http://www.lycos.com) • Currently, Google and Yahoo leading the market General search engines (cont.) • Search engines differ significantly in terms of power and user-friendliness of their interfaces • Advanced search options offered: – Boolean operators (AND, NEAR, OR) – Truncation (e.g., communicat*) – Search within results (only Google, Hotbot, and Lycos) General search engines (cont.) • Internet directories (e.g., dmoz): good for beginning searches and to distinguish between different categories, but may not be up to date and limited in scope • Not every item appears in every search engine • Even the most current search engines cover less than half of all the content on the internet (e.g., Google accesses only 15-16%) - in fall of 2005 Yahoo reported to possess an index of 19 billion web documents; Google’s index at the time was estimated at 8 billion documents (Sherman, 2005) General search engines (cont.) • No recent updates on index sizes by either Yahoo or Google • Google, Yahoo and other search engines only access “surface web” static web documents that are accessible to search engines • “Deep web” 50x bigger than “surface web”, consists of web documents dynamically generated by database servers: this information tends to be more recent, more topic-focused, and more relevant, but these sites are difficult to retrieve for search engine crawlers (Sherman, 2005) Repeat search using different search engines! Specialized search engines • Consider for locating higher quality research evidence on the internet • Scirus (http://www.scirus.com) – – – – – Focuses on scientific content only Searches the web and electronic journal sources Locates more peer reviewed articles Recognizes formats such as PDF and Postscript Most beneficial to locate research data Specialized search engines (cont.) • Google Scholar (http://www.scholar.google.com) – Peer-reviewed papers, theses, books, preprints, abstracts and technical reports from all broad areas of research are available – Results are ordered by how relevant they are to the query – Automatically analyzes and extracts references and presents them as separate results, even if the documents they refer to are not online Meta-searches and meta-search engines • Simultaneously explore the databases of multiple sets of individual search engines • Summarize results from a broad range of different search engines • Search is more comprehensive without loosing the general overview perspective • Meta-search engines – Ixquick (http://www.ixquick.com); features include coverage of 12 most familiar search engines and directories, result rankings, and ratings for most appropriate individual search engine – Copernic (http://www.copernic.com) – DogPile (http://www.dogpile.com) – Metacrawler (http://www.meta-crawler.com) Evaluation of information located on the Internet • A systematic evaluation of the obtained web documents should be performed to determine the quality and usefulness of the information or data presented • Evaluation of validity is necessary because there is a lack of quality control on the internet • Five criteria for evaluation – Accuracy – Authority – Objectivity – Currency – Coverage Accuracy • Need for content to be valid and “without errors of fact, interpretation, or judgment” (Eysenbach et al., 2002) • Information needs to be disclosed re: – Who authored the web document – Whether or not that person is qualified to write the information provided – If the provided content is reliable and error-free – What kind of sources are cited Authority • Overall credibility of the author and publisher of a web document • To determine authority look at: – Header or footer for an affiliation – The Internet domain (e.g., .edu, .com, .ac.uk, .org, and .net) • Credentials of the author should be reviewed Authority: Review of Top Level Domains Domain Meaning educational .edu .gov .org .com .net .mil .ca ac.uk .fl.us Example aac.unl.edu U.S. government www.nih.gov organization/association www.ussaac.org commercial (mostly) www.prentrom.com internet service provider www.assistivetech.net U.S. military www.bethesda.med.navy.mil Canada www.apraxia.ca UK (academic) callcentre.education.ed.ac.uk U.S., Florida State aten.scps.k12.fl.us Objectivity • Depending on website’s purpose (advertising, advocacy, opinion, scholarship, etc.) the provided information may be biased • To determine objectivity consider: – Why the document was created – For whom it was written – What opinions, if any, are expressed by the author • Disclosure of ownership and sponsorship are essential Currency • Look at when the web document or its web site was first produced and updated • Check the number of dead links located on the web page – If links cannot be accessed, there is a high likelihood that information on the web page from which the link was accessed is no longer current Coverage, comprehensiveness, and accessibility of information • Web documents containing more detail most likely provide more information and increased coverage of a topic • Credibility increases with links to reputable web sites • Many web pages contain links to biased sites and advertisements • Accessibility of web pages is important. Accessible sites provide a text equivalent for all non-text elements to be read in screenreaders to users with visual impairments Look out for • Schlosser, R. W., Wendt, O., Angermeier, K. L., & Shetty, M. (2005). Searching for evidence in augmentative and alternative communication: Navigating a scattered literature. Augmentative and alternative communication, 21 (4), 233-255. • Schlosser, R. W., Wendt, O., Bhavnani, S., & NailChiwetalu, B. (under review). Benefits of “adapted pearl growing” for evidence-based practice and implementing systematic reviews: A short report. Manuscript submitted for publication. References • American Speech-Language-Hearing Association (ASHA) (2003). Evaluating Web sites. Retrieved November 23, 2003 from the ASHA web site: http://www.asha.org/sitehelp/websites.htm • Beaven, O. (2002). Searching the literature. In J. V. Graig, & R. L. Smyth (Eds.), The evidence-based practice manual for nurses (pp. 45-85). Edinburgh: Churchill Livingstone. • Beck, S. (1997). The good, the bad, and the ugly: or, why it’s a good idea to evaluate web sources. Retrieved March 22, 2004, from the New Mexico State University Library web site: http://lib.nmsu.edu/instruction/evalcrit.html • Blair-Lockney, E. (2003). AJSLP reaches major milestone – MEDLINE indexing. The ASHA Leader, April 1, 2003, 3. • Boynton, J., Glanville, J., McDaid, D., Lefebvre, C. (1998). Identifying systematic reviews in MEDLINE: developing an objective approach to search strategy design. Journal of Information Science, 24,137-157. • Chamberlain, E. (2003). Bare bones 101: A basic tutorial on searching the web. Retrieved March 10, 2004, from the University of South Carolina, Beaufort Library web site: http://www.sc.edu/beaufort/library/bones.html • Cook, D. J., Mulrow, C. D., & Haynes, R. B. (1997). Synthesis of best evidence for clinical decisions. Annals of Internal Medicine, 126, 376-380. • Cook, D. J., Sackett, D.L., Spitzer, W. O. (1995). Methodologic guidelines for systematic reviews of randomized control trials in health care from the Potsdam Consultation on MetaAnalysis. Journal of Clinical Epidemiology, 48, 167-171 • Cooper, H. & Hedges, L. V. (1994). The handbook of research synthesis. New York: Russell Sage Foundation. • Corrall, C. J., Wyer, P. C., Zick, L. S. et al. (2002). How to find evidence when you need it, part 1: databases, search programs, and strategies. Annals of Emergency Medicine, 39, 302-306. • De Vet, C. W. H., de Bie, R., van der Heijden, G., Verhagen, A., Sijpkes, P., & Knipschild, P. (1997). Systematic reviews on the basis of methodological criteria. Physiotherapy, 83, 284-289. • Dickson, R. (1999). Systematic reviews. In S. Hamer & G. Collinson (Eds.), Achieving evidence-based practice. A handbook for practitioners (pp. 41-60). Edinburgh: Tindall. • Eysenbach, G., Powell, J., Kuss, O., Sa, E.-R. (2002). Empirical studies assessing the quality of health information for consumers on the world wide web: A systematic review. Journal of the American Medical Association, 287, 2691-2700. • Gallagher, P. E., Allen, T. Y., & Wyer, P. C. (2002a). How to find evidence when you need it, part 2: A clinician’s guide to MEDLINE: The basics. Annals of Emergency Medicine, 39, 436-440. • Gallagher, P. E., Allen, T. Y., & Wyer, P. C. (2002b). How to find evidence when you need it, Part 3: A clinician’s guide to MEDLINE: Tricks and special skills. Annals of Emergency Medicine, 39, 547-551. • Golper, L. A., Wertz, R. T., Frattali, C. M., Yorkston, K., Meyers, P., Katz, R., Beeson, P., Kennedy, M. R. T., Bayles, K., & Wambaugh, J. (2001). Evidence-based practice guidelines for the management of communication disorders in neurologically impaired individuals: Project introduction. Minneapolis, MN: Academy of Neurologic Communication Disorders and Sciences. • Guyatt, G., & Rennie, D. (2002). Users' guides to the medical literature. Essentials of evidence-based clinical practice. Chicago, AMA Press. • Hass, U., Andersson, A., Brodin, H., Persson, J. (1997). Assessment of computeraided assistive technology: analysis of outcomes and costs. Augmentative and Alternative Communication, 13, 125-135. • Haynes, R. B., Walker, C. J., McKibbon, A., et al. (1994). Performances of 27 MEDLINE systems tested by searches with clinical questions. Journal of the American Medical Informatics Association, 1, 285-295. • Haynes, R. B., Wilcynski, N., McKibbon, A., et al. (1994). Developing optimal search strategies for detecting clinically sound studies in MEDLINE. Journal of the American Medical Informatics Association, 1, 447-458. • Haynes, R. B., Wilcynski, N., McKibbon, A., et al. (1994). Developing optimal search strategies for detecting clinically sound studies in MEDLINE. Journal of the American Medical Informatics Association, 1, 447-458. • Helewa, A., & Walker, J. M. (2000). Critical evaluation of research in physical rehabilitation: Towards evidence-based practice. Philadelphia: W.B. Saunders Company. • Humphris, D., Littlejohns, P., Victor, C., O’Halloran, P., & Peacock, J. (2000). Implementing evidence-based practice: Factors that influence the use of research evidence by Occupational Therapists. British Journal of Occupational Therapy, 63, 516-522. • Jackson, G. B. (1980). Methods for integrative reviews. Review of Educational Research, 50, 438-460. • Kapoun, J. (1998). Teaching undergrads Web evaluation: A guide for library instruction. College & Research Libraries News, 59 (7), 522-523. • Kunka, J. L. (2004). Research and the Internet. Retrieved March 10, 2004, from Purdue University, Online Writing Lab web site: http://owl.english.purdue.edu/workshops/pp/index.html • Kuster, J. M. (2002, May). Web-based information resources for evidence-based practice in speech-language pathology. Perspectives on Language Learning and Education, 9 (1), 6-14. • Lloyd, L. L., Arvidson, H., & Koul, R. (1996). AAC Master’s and doctoral theses update. Augmentative and Alternative Communication, 12, 200-208. • Lloyd, L. L., Koul, R., & Arvidson, H. (1993). AAC Theses and Dissertations. Augmentative and Alternative Communication, 9, 196-225. • Lloyd, L. L., Koul, R., & Arvidson, H. (1994). Master's and doctoral theses update. Augmentative and Alternative Communication, 10, 61-65. • Lohr, K. N. (1994). Guidelines for clinical practice: Applications for primary care. International Journal for Quality in Health Care, 6, 17-25. • Lou, J. Q. (2002). Searching for the evidence. In M. Law (Ed.), Evidence-based rehabilitation (pp. 71- 94). Thorofare, NJ: Slack Incorporated. • McAuley, L., Pham, B., Tugwell, P., & Moher, D. (2000). Does the inclusion of grey literature influence estimates of intervention effectiveness reported in meta-analyses? The Lancet, 356, 1228-1231. • Melnyk, B. M., & Fineout-Overholt, E. (2002). Key steps in implementing evidence-based practice: Asking compelling, searchable questions and searching for the best evidence. Pediatric Nursing, 22, 262-266. • Millar, D., Light, J. C., & Schlosser, R. W. (2004). The impact of augmentative and alternative communication on speech development: A best evidence research review. Manuscript submitted for publication. • Morrisey, L. J., & DeBourgh, G. A. (2001). Finding evidence: Refining literature searching skills for the advanced practice nurse. AACN Clinical Issues, 12, 560-577. • NHS Centre for Reviews and Dissemination (CRD) (2001, March). Undertaking systematic reviews of research on effectiveness (CRD Report No. 4, 2nd ed.). Retrieved June 18, 2003, from the University of York, Centre for Reviews and Dissemination web site http://www.york.ac.uk/inst/crd/report4.htm • NHS Centre for Reviews and Dissemination (2003). DARE guide to searching. Version 2003:1. York, UK: Author. • Nielsen//NetRatings (2004). Top 15 Search Destinations. Home and Work Users, June 2004. Retrieved November 1, 2004, from http://www.nielsen-netratings.com/news.jsp • Raghavendra, P. (2000). Evidence-Based Practice: Where are we in AAC? Proceedings of the 9th Biennial Conference of the International Society for Augmentative and Alternative Communication (pp. 742-743). Washington DC.: International Society for Augmentative and Alternative Communication. • Sackett, D. L., Haynes, R. B., Tugwell, P. (1985). Clinical epidemiology. A basic science for clinical medicine. Boston: Little, Brown & Co. • Scherer, S., & Smith, M. B. (2002). Teaching evidence-based practice in academic and clinical settings. Cardiopulmonary Physical Therapy, 13, 23. • Schlosser, R. W. (2003). The efficacy of augmentative and alternative communication: Toward evidence-based practice. San Diego, CA: Academic Press. • Schlosser, R. W., & Raghavendra, P. (2004). Evidence-based practice in augmentative and alternative communication. Augmentative and Alternative Communication, 20, 1-21. • Schlosser, R. W., & Sigafoos, J. (in press). Augmentative and alternative communication interventions for persons with developmental disabilities: Narrative review of comparative single-subject experimental studies. Research in Developmental Disabilities. • Shapiro, Y., & Lehoczky, E. (2004). Search engines. Retrieved November 1, 2004, from http://www.searchengines.com/search_engines_101.html • Sherman, C. (2005). Search Day. Electronic newsletter. Retrieved January 24, 2006, from http://searchenginewatch.com/searchday/ • White, H. D. (1994). Scientific communication and literature retrieval. In H. Cooper, & L. V. Hedges (Eds.), The handbook of research synthesis (pp. 42–55). New York: Russell Sage Foundation. For further information contact • Oliver Wendt: – E-mail: olli@purdue.edu • Ralf W. Schlosser: – E-mail: r.schlosser@neu.edu • Handout available on Purdue AAC website: http://www.edst.purdue.edu/aac (click on “Recent Presentations”)