Survey Instruments for Knowledge, Skills, Attitudes and Behaviour Related to Evidence-based Practice in Occupational Therapy: A Systematic Review

Transcription

1 RESEARCH ARTICLE Survey s for Knowledge, Skills, Attitudes and Behaviour Related to Evidence-based Practice in Occupational Therapy: A Systematic Review Helen Buchanan 1 *, Nandi Siegfried 2 & Jennifer Jelsma 1 1 Department of Health & Rehabilitation Sciences, University of Cape Town, Cape Town, South Africa 2 Independent Clinical Epidemiologist, Cape Town, South Africa Abstract The purpose of this study was to evaluate, through a systematic review, assessment instruments for evidence-based practice (EBP). The specific objectives were to (1) identify survey instruments testing EBP knowledge, skills, attitudes and behaviour; (2) determine the attributes by each instrument; (3) evaluate the psychometric properties of the instruments; and (4) evaluate the methodological quality of the instruments. Using the Cochrane approach, searches were conducted in Pubmed, EBSCOHost and Scopus from inception to February Papers were screened by two independent assessors, and data were extracted by one researcher. Forty papers reporting 34 instruments met the inclusion criteria and were included in the qualitative synthesis. Most instruments EBP behaviour (n = 33) and attitudes (n = 21). This review provides a single source of information to enable researchers to select the most robust descriptive instruments to measure EBP learner attributes. s used only with occupational therapists may have resulted in some instruments being missed. For further research, it is recommended that attention is given to developing objective instruments with a focus on knowledge and skills. Copyright 2015 John Wiley & Sons, Ltd. Received 11 March 2015; Revised 10 June 2015; Accepted 10 June 2015 Keywords evidence-based practice; occupational therapy measurement instruments; research utilization *Correspondence Helen Buchanan, Department of Health & Rehabilitation Sciences, F45 Old Groote Schuur Hospital Building, University of Cape Town, Observatory, 7925, Cape Town, South Africa. helen.buchanan@uct.ac.za Published online 3 July 2015 in Wiley Online Library (wileyonlinelibrary.com) DOI: /oti.1398 Background Health professionals globally have been encouraged to implement evidence-based practice (EBP). In response to this call, most undergraduate and post-graduate health professional curricula now include modules that enable students to acquire the necessary knowledge, skills, attitudes and behaviour to implement EBP. Continuing professional development programmes have similarly targeted qualified health professionals through educational workshops and seminars. Within occupational therapy, there has been an appeal for the global adoption of EBP with local application to enhance relevance (Illott et al., 2006), but there is insufficient evidence to establish the extent to which this challenge has been taken up. Furthermore, some authors have suggested that EBP is not yet a global reality (Bannigan, 2011; Buchanan, 2011). Surveys of professional groups provide information about the strengths and learning needs of the group Occup. Ther. Int. 23 (2016) John Wiley & Sons, Ltd. 59

2 Evidence-based Practice Survey s Buchanan et al. (Eller et al., 2003), which is required to plan appropriate methods and strategies to increase the uptake of EBP. To date, an occupational therapy survey to determine the extent to which EBP is being implemented internationally has not been carried out. Such an enterprise would be useful not only in measuring the progress of the profession towards becoming evidence-based but also in providing important information for developing further action plans towards realizing this goal. To yield useful findings from a survey, measurement instruments with satisfactory psychometric properties are required to ensure that the relevant attributes are and with accuracy. In 2007, a summary of the current state of the art on evaluation of training in [evidence-based health care (EBHC)] (Nabulsi et al., 2007, p 469) outlined three domains within the learner (affective, cognitive and behavioural) that should be in order to evaluate the outcomes of an EBP training activity comprehensively. Unfortunately, terminology referring to EBP learner outcomes has been used inconsistently in the literature (Nabulsi et al., 2007). As a result, the terms used in this paper will first be defined. The affective domain encompasses attitudes that include views, perceptions, beliefs and intentions relating to EBP (Nabulsi et al., 2007). Attitudes may also incorporate a health professional s agreement/ acceptance of the evidence, their perceived clinical applicability of the evidence, and their motivation and sense of self-efficacy to adopt EBP (Menon et al., 2009, p 1025). The cognitive domain includes the knowledge and skills required to implement the steps of EBP. Knowledge is defined as the acquisition of awareness or facts, data, information, ideas or principles to which one has access through formal or individual study, research, observation, experience or intuition (Wojtczak, 2002, p 451). Skills incorporate knowledge by performing EBP steps in some type of clinical scenario, such as with a standardized patient, written case, computer simulation, [objective standardized clinical examination] or direct observation (Shaneyfelt et al., 2008, p 1117). Skills require competence in specific areasoutside the practice environment, while behaviour encompasses the actual performance of EBP in practice (Shaneyfelt et al., 2008, p 1117). Examples of the latter include searching databases for evidence, accessing information sources and using evidence to select an intervention in the actual practice setting. High-quality evidence should be used to inform the choice of an instrument. However, information about instruments that may be used to measure EBP learner outcomes is scattered across a variety of databases and journals making it difficult to obtain a comprehensive picture of the best instruments available. Furthermore, some instruments may only be appropriate for undergraduate students, while others may not be applicable to occupational therapy because of fundamental differences in the scope of practice of different health professions. It is currently unclear how EBP learning differs between health professionals. As several occupational therapy-specific survey instruments have been described in the literature, a review and a description of these would prove useful as a single source of information. A systematic review on instruments measuring EBP knowledge and skills in occupational therapists was published in 2010 (Glegg and Holsti, 2010), but a review focussing on instruments to measure all learner outcomes, including behaviour, has not been carried out. Considering the importance of EBP in affirming the contribution of occupational therapy, it is critical that occupational therapists engage actively in implementing evidence in everyday practice. To monitor and measure this process, suitable instruments are needed to (1) describe the current state of the art; (2) plan appropriate interventions to improve EBP knowledge, skills, attitudes and behaviour; and (3) establish the effectiveness of these interventions. To evaluate whether interventions are effective, instruments with satisfactory validity, reliability and clinical utility (Corr and Siddons, 2005) are needed for ongoing monitoring. This review therefore sets out to (1) identify survey instruments to measure EBP knowledge, skills, attitudes and behaviour (use); (2) describe the aspects of EBP learning by each instrument; (3) evaluate the psychometric properties of identified instruments; and (4) evaluate the methodological quality of the identified instruments. Methods Criteria for considering studies for this review Type of study Descriptive studies that used instruments measuring knowledge, skills, attitudes or behaviour related to EBP were included. Systematic reviews, either of 60 Occup. Ther. Int. 23 (2016) John Wiley & Sons, Ltd.

3 Buchanan et al. Evidence-based Practice Survey s instruments measuring these aspects or reviews of EBP surveys, were excluded, but the reference lists of included papers were checked to identify studies that may have been missed. Mixed methods studies were included if they contained a quantitative component and complied with the aforementioned criteria. Types of participants Studies had to include participants who were qualified occupational therapists or occupational therapy students (undergraduate or post-graduate). Studies of rehabilitation professionals were included if occupational therapists were part of the sample. Aspects of EBP learning Studies that outcomes related to EBP knowledge, skills, attitudes and behaviour were included. Papers that focussed on these aspects in a specific area of practice (e.g. stroke) rather than EBP in general were excluded. Reference lists of included studies were searched to identify papers that may have been missed. Data collection and analysis Citations for identified papers were imported into Endnote, and duplicates were removed. Selection of studies Two researchers screened paper titles and abstracts and applied the eligibility criteria. Papers that did not meet the eligibility criteria were excluded. Reasons for exclusion were documented. The full text of the remaining studies was retrieved, and the inclusion criteria were applied to identify relevant papers. Once this process had been completed independently, the researchers met to reach consensus on the included and excluded studies. Any discrepancies were discussed until consensus was reached. If the full text paper was not available, it was excluded. Search methods for identification of studies Using the Cochrane approach, searches were conducted in Pubmed, EBSCOHost and Scopus from their inception to February The following databases were searched simultaneously in EBSCOHost: Africa-Wide Information, CINAHL, ERIC, Health : Nursing/academic edition, MEDLINE, PsycARTICLES and PsycINFO. No limits were set. The search terms used were: ( occupational therapy OR occupational therapy practice OR OT) AND (tool OR survey OR instrument OR test OR measure OR scale OR questionnaire) AND ( evidence-based practice OR evidence based practice OR EBP OR evidence-based-medicine OR evidence-based) AND (knowledge OR awareness OR skills OR attitudes OR perceptions OR behaviour OR practice OR ability OR uptake OR implementation OR research use OR research utilisation OR research utilization ). Each set of terms was first searched individually after which the results sets were combined. The terms were used as MeSH and as text. No limits were set, and no attempt was made to identify unpublished materials. Data extraction and management One researcher conducted data extraction of all eligible studies, which were summarized in tables depicting their descriptive characteristics, learner attributes and a quality assessment of each instrument. Rating of the methodological quality of included instruments On extracting the information on each instrument, it became clear that the process of development of most instruments did not follow test development procedures, and reporting of psychometric properties was generally of a low standard. Therefore, criteria that were considered fundamental for survey instruments were selected from the COnsensus-based Standards for the selection of health Measurement INstruments checklist (Terwee et al., 2012), and a basic assessment and rating of each property were performed. Aspects to evaluate the clinical utility of the instrument (Corr and Siddons, 2005) were added to the quality rating criteria. s were rated on a 4-point scale on seven aspects relating to their measurement properties. The criteria used in the quality evaluation are available in Table I. Properties that were not reported were classified as unclear. Occup. Ther. Int. 23 (2016) John Wiley & Sons, Ltd. 61

4 Evidence-based Practice Survey s Buchanan et al. Table I. Criteria used for the quality evaluation based on the COSMIN checklist (Terwee et al., 2012) Rating scale Measurement property Excellent (E) Good (G) Fair (F) Poor (P) Internal consistency: Was an internal consistency statistic calculated for each sub-scale separately? Reliability (intra-rater reliability): Was an intraclass correlation coefficient (for continuous variables) or a Kappa (for dichotomous/nominal/ ordinal variables) calculated? Content validity: Was there an assessment of whether all items refer to relevant aspects of the construct being? Structural validity: Was exploratory or confirmatory factor analysis performed? Hypothesis testing: Were hypotheses regarding correlations or mean differences formulated a priori? Cross-cultural validity: Were both the original language in which the instrument was developed and the language in which it was translated described? Did the translators work independently of each other? Were items translated forward and backward? Clinical utility: Were the clarity of the instructions, the format /acceptability of the instrument and the time taken to complete the questionnaire tested? Internal consistency statistic calculated for each sub-scale separately ICC or Kappa calculated Assessed if all items refer to relevant aspects of the construct being Exploratory or confirmatory factor analysis performed Multiple hypotheses formulated a priori Both source language and target language described Translators worked independently Multiple forward and backward translations The clarity of instructions, format/ acceptability of the instrument and time taken to complete were tested Minimal number of hypotheses formulated a priori Assumable that translators worked independently Multiple forward but one backward translation Two of the described aspects of utility were tested Poor description of the relevant aspects of the construct being Hypotheses vague or not formulated but possible to deduce what was expected Unclear whether translators worked independently One forward and one backward translation One of the described aspects of utility were tested Internal consistency statistic NOT calculated for each sub-scale separately No ICC/Pearson/ Spearman correlations calculated. No Kappa calculated only percentage agreement Did not assess if all items refer to the relevant aspects of the construct being No exploratory or confirmatory factor analysis performed Unclear what was expected language not known Translators did not work independently Only a forward translation No mention was made of any aspects of utility COSMIN, consensus-based standards for the selection of health measurement instruments; ICC, intraclass correlation coefficient. Results The search identified 50 citations in Pubmed, 277 in EBSCOHost and 68 in Scopus. After removing duplicates, 351 references were imported into Endnote. Further duplicates were identified in Endnote and removed, leaving 326 citations. These were independently screened by two reviewers, and 286 papers were excluded based on the title and abstract. The full text of the remaining 40 papers was screened for eligibility. Thirteen papers were excluded on study type, two of which were systematic reviews, and one was excluded 62 Occup. Ther. Int. 23 (2016) John Wiley & Sons, Ltd.

5 Buchanan et al. Evidence-based Practice Survey s as the full text was not available. The original studies included in the systematic reviews (n = 2) and the reference lists of the included papers (n = 26) were screened. A further 14 papers met the inclusion criteria, bringing the total to 40 papers. As some papers (n =5) reported different aspects of a single study, the 40 papers accounted for 35 studies. The PRISMA flow diagram for the study (Moher et al., 2009) is shown in Figure 1. Description of the studies Of the 35 studies reported in the included papers, most designs were cross-sectional surveys (n = 33), one of which was a follow-up study in which the same survey was sent to the same group of occupational therapists 6 years apart to determine whether attitudes had changed over time (Karlsson and Törnquist, 2007). Two further studies described the development of an instrument (Salbach and Jaglal, 2011; Upton and Lewis, 1998). In the 33 cross-sectional studies, surveys were distributed by post (n = 22), hard copy (n = 4), (n = 2), and post (n = 1) and online (n = 2). The distribution method was unclear in two studies (Closs and Lewin, 1998; Heiwe et al., 2011). Details of the included studies are summarized in Table II. Studies are reported in alphabetical order by author and then publication year. Only the outcomes of relevance to this review (knowledge, skills, attitudes and behaviour) are included. Figure 1 Study flow diagram Occup. Ther. Int. 23 (2016) John Wiley & Sons, Ltd. 63

6 Evidence-based Practice Survey s Buchanan et al. Table II. Characteristics of included papers (n = 34) and Bennett et al. (2003); Pettingill et al. (1994) Modified from McColl et al. (1998) Australia Members of OT Australia (n = 649) Bennett et al. (2007) Developed for study modelled on existing survey Multi-national (41 countries) Users of the OTseeker evidence database (included 447 qualified OTs (n = 250) and students (n = 197)) Brown Research Knowledge # Attitudes and Practices of Research (KAP) Survey (Van Mullem et al., 1999) Australia, Taiwan and UK Qualified OTs listed on national OT databases who worked with children (n = 696) (online) Attitudes Self-report Use Confidence in skills Barriers to use Access to OTseeker at work Frequency and main reason for use Use of other databases Perception whether OTseeker improved ability to find relevant evidence Contribution to changes in practice containing 4 sections with 5-point rating scales, fixedresponse categories and an open-ended question Self-report online questionnaire with 14 questions, mainly dichotomous or fixed-response categories and an open-ended question Content validity tested in a pilot NR NR Content validity: 0.84 (Van Mullem et al., 1999) NR Feedback on acceptability and clarity obtained in a pilot test NR NR Trialled with clinicians and minor changes made Test retest reliability for subscales: Internal consistency: α = (Van Mullem et al., 1999); α = (Eller et al., 2003) Easy and inexpensive (Van Mullem et al., 1999) 64 Occup. Ther. Int. 23 (2016) John Wiley & Sons, Ltd.

7 Buchanan et al. Evidence-based Practice Survey s Table II. (Continued) and Edmonton Research Orientation Survey (EROS; Pain et al., 1996) Barriers to Research Utilization Scale (BARRIERS; Funk et al., 1991) Brown et al. Research Knowledge (2010a) # Attitudes and Practices of Research (KAP) Survey (Van Mullem et al., 1999) Edmonton Research Orientation Survey (EROS; Pain et al., 1996) Barriers to Research Utilization Scale (BARRIERS; Funk et al., 1991) Participation in research and research orientation to practice the EBP sub-scale indicates research utilization Perceived barriers to research utilization et al. (2009) 38 items rated on a 5-point Likert scale; an overall score and sub-scale scores are calculated As for Pain et al. (2004) As for Pain et al. (2004) NR 4 sub-scales and 28 items rated on a 4- point scale Sub-scales confirmed by factor analysis (Funk et al., 1991) Factor analysis identified 4 factors similar to those of Funk et al. but only 22 of the 28 items were retained Internal consistency: 0.72 to 0.80 for the first 3 factors and 0.65 for the 4th factor Test retest reliability: preliminary evidence (Funk et al., 1991) NR Occup. Ther. Int. 23 (2016) John Wiley & Sons, Ltd. 65

8 Evidence-based Practice Survey s Buchanan et al. Table II. (Continued) and Brown et al. Research Knowledge (2010b) # Attitudes and Practices of Research (KAP) Survey (Van Mullem et al., 1999) Caldwell et al. (2007) Questionnaire developed for study London, UK OT graduates from 3 universities (n = 50) Cameron et al. (2005) Questionnaire based on Humphris et al. (2000) United States and Puerto Rico Members of the American Occupational Therapy Association (n = 131) Closs and Lewin (1998) Barriers to Research Utilization Scale (BARRIERS; Funk et al., 1991) and additional questions on barriers and facilitators UK Dieticians (n = 12), OTs (n = 24), physiotherapists (n = 51) and speech therapists (n = 15) (unclear) Preparation and skills training for EBP, access to databases, experience of research changing practice, views on EBP, confidence in EBP Application of EBP to intervention and attitudes towards EBP Perceived barriers to research utilization Self-report with closed questions and 5-point Likert scales Literature used to develop questionnaire NR Minor changes to wording based on pilot study (n = 20) Self-report questionnaire with 2 sections demographic information and 9 questions rated on a 5-point scale 28 items with 5 response options Questionnaire evaluated by 3 experts Factor analysis of pilot study data (n = 30) noncontributing items removed NR NR Wording modified for UK context based on feedback from pilot 66 Occup. Ther. Int. 23 (2016) John Wiley & Sons, Ltd.

9 Buchanan et al. Evidence-based Practice Survey s Table II. (Continued) and Cooke et al. (2008) Questionnaire developed for study North England, UK OTs in 2 UK Councils with Social Services Responsibilities (not disaggregated from social workers) Curtin and Jaramazovic (2001) Questionnaire developed for the study South West and South East England and Channel Islands, UK OTs who supervised students (n = 500) Dopp et al. (2012) Questionnaire based on Humphris et al. (2000); Parahoo (2000) and Dysart and Tomlin (2002) Netherlands Members of the Dutch Association of Occupational Therapists (n = 100) ( ) ( and postal) Access to and use of research information Views and perceptions of EBP, self-rated involvement in types of EBP activities Frequency of using evidence sources, barriers to implementing EBP and attitudes to EBP Self-report questionnaire containing 6 sections. Formats for questions included tick box options, 5-point Likert scales and open-ended Self-report questionnaire with a mixture of open, closed and Likert scale items Self-report questionnaire with 7-point or 5-point Likert scale response options NR NR Piloted on a small number of staff and changes made before use changes not specified Content based on focus group discussions and previous research Draft questionnaire tested in cognitive interviews, revised and then tested in a postal pilot study Content validity assumed as based on focus group findings No validity testing performed Content evaluated in a pilot study (n =6) Face validity supported by expert opinion No reliability testing performed Internal consistency good use of sources: α = 0.789; barriers: α = 0.795; attitudes: α = Cognitive interviews to check thoughts about questionnaire Format evaluated in a pilot study (n =6) Occup. Ther. Int. 23 (2016) John Wiley & Sons, Ltd. 67

10 Evidence-based Practice Survey s Buchanan et al. Table II. (Continued) and Dysart and Tomlin (2002) Questionnaire developed for the study United States Random sample of members of the American Occupational Therapy Association (n = 209) Eller et al. (2003) Research Knowledge Attitudes and Practices of Research (KAP) Survey (Van Mullem et al., 1999) New Jersey, United States Nurse (n = 538) and non-nurse health professionals (n = 208) including OTs (hard copy) EBP skills and attitudes, facilitators and barriers to EBP, access to and frequency of using EBP resources, frequency of research implementation Knowledge, willingness to engage in (attitudes) and ability to perform (practices) activities related to: identifying clinical problems, establishing current best practice, implementing research in practice and communicating research Self-report with 3 sections containing dichotomous, 4- point scale, 5-point Likert scale and openended response options NR NR Piloted with 6 OTs and revisions made (details not stated) Wording of some items may have been confusing 33-item self-report consisting of 5 factors; items rated on a 3-point scale and sub-scale scores determined Construct validity: factor analysis revealed 5 factors NR NR 68 Occup. Ther. Int. 23 (2016) John Wiley & Sons, Ltd.

11 Buchanan et al. Evidence-based Practice Survey s Table II. (Continued) and Gilman (2011) Adapted from Powell and Case-Smith (2010) Oregon, United States Recent OT Masters graduates (n = 26) Gosling and Westbrook (2004) Graham et al. (2013) Questionnaire designed for the study Adapted from Bennett et al. (2003) New South Wales, Australia Allied health professionals at randomly selected hospitals (physiotherapists [n = 228], OTs [n = 118], speech pathologists [n = 77], dieticians [n = 78], clinical psychologists [n = 59], pharmacists [n = 84] and social workers [n = 146]) New Zealand OTs with practice certificates who consented to be contacted (n = 473) (online) (hard copy) As for Powell and Case-Smith (2010) Use of an online evidence database, searching skills, success finding evidence, impact on clinical practice, perceived barriers to using the database As for Powell and Case-Smith (2010) with minor wording changes, additional response options included for 2 items and 4 new items added 25-item questionnaire, comprising closeended questions with various response options, e. g. yes/no, 3 6 point rating scales Input from academics but no validation performed NR NR NR NR Minor changes made to wording of some questions to improve clarity Perceptions of skills Self-report Attitudes questionnaire with Behaviour 5-point Likert scale, categorical and open-ended questions NR NR Cognitive interviews used to improve clarity Occup. Ther. Int. 23 (2016) John Wiley & Sons, Ltd. 69

12 Evidence-based Practice Survey s Buchanan et al. Table II. (Continued) and Heiwe et al. (2011) Questionnaire translated from Jette et al. (2003) with some modifications Sweden OTs employed in a large university hospital (n = 57) Hu et al. (2012) Questionnaire developed for study UK OTs working in rural/remote areas in an NHS Trust (n = 64) Humphris et al. (2000) Questionnaire based on a qualitative study and Pettingill et al. (1994) South Thames region, UK OTs in 7 Acute NHS Trusts in South Thames region (n = 66) Karlsson and Törnquist (2007) Adaptation and translation of an instrument by Eckerling et al. (1988) Two central county districts in Sweden OTs registered with the Swedish Occupational Therapy Association (n = 425 at baseline and n = 442 at follow-up) (unclear) Follow-up Attitudes, beliefs, knowledge and behaviour towards EBP Attitudes towards and involvement in EBP Research availability and utilization, attitudes towards research and barriers and facilitators to EBP Perceptions of attitudes to, perceived ability to perform and current involvement in research-related activities Self-report questionnaire comprising 5 sections and 51 statements. Most items rated on a 5-point Likert scale Self-report questionnaire consisting of 4 sections with yes/no and 5-point Likert Scale response options Self-report questionnaire with 4 sections and items with yes/no and 5-point Likert scale response options Self-report; 5 dimensions of research-related activities each containing 4 statements rated on a 5-point Likert scale Face validity established with a group of experts Based on a literature review Content validity established through professional opinion Rigorous 3-step translation process followed 4 original dimensions and 4 research activities chosen based on extensive literature review conducted by Ehrenfeldt and Eckerling (1991). Validity not explicitly tested for this study NR NR NR NR NR NR Internal consistency for 4 dimensions: α = (Eckerling et al., 1988) and α = (Ehrenfeldt and Eckerling, 1991). Present study: α = 0.82 (role), α =.62 (ability), α = 0.53 (intent), and α = 0.80 (engagement) Trial version completed by occupational therapists and students not involved in the study (n = 30) and minor changes made to instructions and wording 70 Occup. Ther. Int. 23 (2016) John Wiley & Sons, Ltd.

13 Buchanan et al. Evidence-based Practice Survey s Table II. (Continued) and Lyons et al. Research Knowledge (2010) # Attitudes and Practices of Research (KAP) Survey (Van Mullem et al., 1999) Barriers to Research Utilization Scale (BARRIERS; Funk et al., 1991) Edmonton Research Orientation Survey (EROS; Pain et al., 1996) UK Qualified paediatric OTs on the College of Occupational Therapists database (n = 145) Intention to engage in research-related activities in the future Attitude to research-related activities rated on a 7-point scale Questions were added to measure actual research engagement as facilitators for research, proficiency in reading scientific papers and use of scientific journals Occup. Ther. Int. 23 (2016) John Wiley & Sons, Ltd. 71

14 Evidence-based Practice Survey s Buchanan et al. Table II. (Continued) and Lyons et al. Research Knowledge (2011) # Attitudes and Practices of Research (KAP) Survey (Van Mullem et al., 1999) McCluskey (2003) Barriers to Research Utilization Scale (BARRIERS; Funk et al., 1991) Edmonton Research Orientation Survey (EROS; Pain et al., 1996) Adapted from Upton and Lewis (1998) Australia Qualified paediatric OTs (n = 138) New South Wales, Australia OTs attending an EBP workshop (n = 67) McKenna et al. (2005) Questionnaire based on 2 previous surveys (McCluskey (2003)) Australia OTs from all States and territories and those at 95 facilities in Queensland and New South Wales (n = 213) (hard copy) Knowledge, skills and attitudes related to EBP, use of EBP, perceived barriers to EBP and solutions Use and perceptions of an online database (OTseeker) and its impact on knowledge and practice Self-report with 21 items containing tick box or 3-point scale response options and openended questions Demographic information and 2 sections: use of OTseeker and its impact on practice; and perceptions of the utility of OTseeker. Response options included dichotomous, multiple fixedresponse categories and 10-point rating scales NR NR Feedback on wording, layout and response options; 10 minutes to complete NR NR Questionnaire piloted on occupational therapists who were not members of OT Australia (n = 3) and minor changes made to improve the clarity of some questions 72 Occup. Ther. Int. 23 (2016) John Wiley & Sons, Ltd.

15 Buchanan et al. Evidence-based Practice Survey s Table II. (Continued) and Metcalfe et al. (2001) Barriers and Attitudes to Research in the Therapies (BART) derived from previous studies (Closs and Lewin, 1998; Metcalfe et al., 2000) Northern and Yorkshire regions, UK Pain et al. (2004) General Use of Research (adapted from Varcoe and Hilton (1995)) Province in the West, Canada Knowledge Acquisition Survey developed for the study Individual semistructured interviews OTs registered with the Council for Professions Supplementary to Medicine (n = 159) OTs in 2 large urban and 2 rural/small urban areas (n = 58) Attitudes and barriers to research Self-rated use of research Availability and use of information sources Use of research in practice Self-report questionnaire with 3 sections: perceived importance of research (PIR; 7 questions) and perceived barriers (PB; 22 questions). Scores for sections 2 and 3 ranged from 7 to +7 and 22 to +22, respectively. No further details provided. 10 self-report items rated on a 4-point scale and summed to give an overall total Self-report. No other details stated Factor analysis revealed 2 factors in the PIR sub-scale and 6 factors in the PB sub-scale Established content validity with peer review (Varcoe and Hilton, 1995) Good internal consistency for the perceived importance of research scale (α = 0.63) and high internal consistency for the perceived barriers scale (α = 0.78) Internal consistency: α = 0.87 NR NR NR NR NR Research use rated on a 7-point scale by participants and interviewer NR IRR ranged from 0.80 to 0.91 NR Occup. Ther. Int. 23 (2016) John Wiley & Sons, Ltd. 73

16 Evidence-based Practice Survey s Buchanan et al. Table II. (Continued) and Edmonton Research Orientation Survey (EROS; Pain et al., 1996) Self-rated knowledge of research concepts Participation in research and research orientation to practice the EBP sub-scale indicates research utilization Self-report 2-part questionnaire consisting of 38 items rated on a 5-point Likert scale Evidence of construct validity Scores correlated with research involvement and training levels Internal consistency: α = 0.93 NR Principal component analysis 4 factors Significant relationship between higher scores and higher education levels, and research participation and training Overall mean correlation with other research participation indicators: Occup. Ther. Int. 23 (2016) John Wiley & Sons, Ltd.

17 Buchanan et al. Evidence-based Practice Survey s Table II. (Continued) and Philibert et al. (2003) Questionnaire based on similar studies in other professionals (e.g. Kirk et al. (1976); McKee et al. (1987)) United States OTs with AOTA membership in 5 States (n = 328) Pollock et al. (2000) Questionnaire developed for study Scotland, UK Stroke rehabilitation professionals (physiotherapists [n = 27], OTs [n = 26], nurses [n = 22], speech and language therapists [n = 6] and other professionals [n = 5]) s of knowledge guiding practice Attitudes towards and use of research in practice Confidence in reading and understanding literature, conducting searches and appraisal, attitudes to EBP, perceived facilitators to EBP and reported use of EBP Self-report questionnaire containing 4 sections. Items included Likerttype scales with varying response options and 2 open-ended questions Self-report containing 20 statements classified under ability (4 statements), opportunity (8 statements) and implementation (8 statements). Level of agreement with each statement rated on a 5-point Likert scale Item-total correlations for attitude items showed a significant relationship to the total score (r 0.56, p < 0.001; Kirk et al., 1976) Content validity was tested in a pilot study (n =5) Barriers identified in 4 focus groups formed into statements by an 2 independent assessors, then combined and independently categorized by 4 assessors and followed by a consensus process Internal consistency for attitudes towards and use of research were 0.89 and for 0.78, respectively Minor modifications based on pilot study (n = 31) to establish the clarity of questions and opinions about the questionnaire in general NR NR Occup. Ther. Int. 23 (2016) John Wiley & Sons, Ltd. 75

18 Evidence-based Practice Survey s Buchanan et al. Table II. (Continued) and Pomeroy et al. (2003) Questionnaire developed for the study North-West region, UK Physiotherapists (n = 7), nurses (n = 2), OTs (n = 2), speech and language therapist (n = 2) Powell and Case-Smith (2003) Questionnaire developed for study United States Graduates of an OT Bachelors programme at Ohio State University (n = 85) Powell and Case-Smith (2010) As aforementioned United States Graduates of an OT Masters programme at Ohio State University (n = 43) ( ) EBP knowledge and skills (formulating a clinical question, searching for evidence, critical appraisal, synthesizing information, understanding statistics, extracting clinical information from papers and evaluating own practice) Information needs, use of online databases, success finding information, ability to analyse and apply information in practice As for Powell and Case-Smith (2003) Self-report on changes in knowledge and skills related to EBP rated on a 5-point scale and an open-ended question on involvement in EBP NR NR NR Self-report questionnaire containing tick box, yes/no and open-ended questions NR NR NR As for Powell and Case-Smith (2003) NR NR NR 76 Occup. Ther. Int. 23 (2016) John Wiley & Sons, Ltd.

19 Buchanan et al. Evidence-based Practice Survey s Table II. (Continued) and Salbach and Jaglal (2011) Evidence-based Practice Confidence (EPIC) Scale Toronto, Canada Qualified OTs Step-wise process to develop and validate the questionnaire Salls et al. (2009) Questionnaire developed for the study from Jette et al. (2003) and Dysart and Tomlin (2002) United States AOTA members and nonmembers licenced to practice in Pennsylvania (n = 930) Confidence in ability to implement EBP (self-efficacy) Attitudes, knowledge and use of EBP 11-item self-report scale with 11 response options per item (ranging from no confidence [0%] to completely confident [100%]) Four sections: access to EBP resources (5 questions), frequency of using EBP resources (4 questions), knowledge of and attitudes towards EBP (13 questions using a 4-point rating scale) and perceived supports and barriers to EBP (2 questions with unordered response choices) Content of scale based on literature after which face and content validity were evaluated by experts in the field of EBP Questionnaire developed with input from an expert reviewer NR Revised scale tested in cognitive interviews to check comprehensibility after which minor changes were made to wording of instructions and some items NR Pilot tested with a convenience sample of occupational therapists (n = 10) and modified based on their feedback Wording of some questions may have been confusing or prompted responses in a particular manner Occup. Ther. Int. 23 (2016) John Wiley & Sons, Ltd. 77

20 Evidence-based Practice Survey s Buchanan et al. Table II. (Continued) and Stronge and Cahill (2012) Modified Knowledge, Attitude and Behaviour (KAB) questionnaire (Johnston et al., 2003) Ireland Final year OT students at the 4 universities (n = 86) Sweetland and Craik (2001) Questionnaire designed for study UK OTs registered with the National Association of Neurological Occupational Therapists (NANOT; n = 125) (hard copy) Self-rated knowledge, attitudes to and future use of EBP Frequency of using EBP and factors influencing its use Self-report containing subjective and objective questions. Consists of 4 subscales: knowledge (5 items), attitudes (6 items) and future use of EBP (9 items) rated on a 6- point scale and 17 additional questions on sources of evidence and demographics Self-report using closed questions with scaled responses and tick box formats. Construct validity demonstrated through correlations with other EBP measures Concurrent validity demonstrated (Johnston et al., 2003) Factor analysis revealed four distinct areas (knowledge, attitudes, personal application and use and future use) Questionnaire based on information from literature and interviews with occupational therapy stroke experts Internal consistency: α = Effect size of 0.33 (p < 0.01) for increase in EBP knowledge in 2nd year medical students at 8 months (Johnston et al., 2003) 10 minutes to complete NR Piloted on NANOT committee members and regional contacts (n = 20) minor modifications made Recommendations after the study were improvement of wording of some questions and clearer definitions of levels of evidence for frequency of using evidence 78 Occup. Ther. Int. 23 (2016) John Wiley & Sons, Ltd.

21 Buchanan et al. Evidence-based Practice Survey s Table II. (Continued) and Upton and Lewis (1998) Questionnaire on EBP and clinical effectiveness UK NA Questionnaire development and validation Upton (1999a) *# Upton (1999b) *# Upton and Upton (2006) Questionnaire by Upton and Lewis (1998) As for Upton (1999a) As for Upton (1999a) Questionnaire developed from Upton and Lewis (1998) Wales, UK Podiatrists (n = 38), OTs (n = 84), physiotherapists (n = 135) and speech therapists (n = 38) As for Upton (1999a) UK Allied health professions and health science services including OTs (n = 86) Perceived knowledge of EBP and its individual steps, frequency of completing EBP steps, attitudes to EBP and barriers and solutions to implementing EBP As for Upton and Lewis (1998) As for Upton and Lewis (1998) As for Upton and Lewis (1998) Five sections with varied response formats (visual analogue scales, semantic differentials, Likert-type scales) and a section for open comments As for Upton and Lewis (1998) Face validity: high (items informed by literature and clinician interviews) Content validity: good (based on expert discussion and pilot studies) Criterion validity: no external reference for comparison As for Upton and Lewis (1998) Test retest reliability: Internal consistency: α = As for Upton and Lewis (1998) Feedback from pilot to improve design, clarity of instructions and wording; lengthy As for Upton and Lewis (1998) As for Upton and Lewis (1998) Self-report containing items rated with a visual analogue scale, 7-point scale or 5-point scale As for Upton and Lewis (1998) As for Upton and Lewis (1998) As for Upton and Lewis (1998) As for Upton and Lewis (1998) As for Upton and Lewis (1998) As for Upton and Lewis (1998) Occup. Ther. Int. 23 (2016) John Wiley & Sons, Ltd. 79

22 Evidence-based Practice Survey s Buchanan et al. Table II. (Continued) and Waine et al. (1997) Edmonton Research Orientation Survey (EROS; Pain et al., 1996) Alberta, Canada Members of the Alberta Association of Registered OTs (n = 293) *Number of OTs not disaggregated from other (non-nursing) professions. # Articles report different aspects of a single study. *#Articles report different aspects of a single study. EBP, evidence-based practice; OTs, occupational therapists; NA, not applicable; NR, no data reported. As for Pain et al. (2004) Perceived barriers to research Resources available to support research As for Pain et al. (2004) Added a 3rd section with 6 items on perceived barriers to research rated on a 5-point Likert scale and an item on resources available to support research Validity data for EROS determined on an earlier version of the scale No validity data on barriers items Reliability data for EROS determined on an earlier version of the scale No reliability data on barriers items NR 80 Occup. Ther. Int. 23 (2016) John Wiley & Sons, Ltd.

23 Buchanan et al. Evidence-based Practice Survey s Location of studies Studies were conducted in nine countries with two multi-national studies. Most were undertaken in the UK (n = 14), followed by the United States (n = 8), Australia (n = 5), Canada (n = 3) and Sweden (n = 2). One study each was carried out in Ireland, the Netherlands, New Zealand, Puerto Rico and Taiwan. Description of instruments The included papers used 34 instruments, all of which were self-reports. Two studies incorporated more than one instrument. Nine studies used existing instruments, while the remainder either developed new ones (n = 15) or modified existing instruments to meet the study aims and objectives (n = 16). Four instruments were used in more than one study the Knowledge Attitudes and Practice of Research survey (Van Mullem et al., 1999; two studies), the BARRIERS scale (Funk et al., 1991; two studies), the Edmonton Research Orientation scale (Pain et al., 1996; three studies) and the questionnaire developed by Powell and Case-Smith (2003; two studies). Aspects of EBP learning s were classified by a researcher using the definitions for learner outcomes (knowledge, skills, attitudes and behaviour) presented at the beginning of this paper. Table III shows each instrument with the attribute/s. Most instruments EBP behaviour (n = 33) and attitudes (n = 21), while relatively few knowledge (n = 8) or skills (n = 3). Methodological quality of included studies The quality ratings of included instruments are shown in Table IV. The number of instruments rated per category of the quality rating scale for each measurement property is shown in Table V. In general, there was limited consideration of measurement properties. Eighteen instruments reported no validity and reliability testing at all. In the cases where instruments were modified for a study, the researchers assumed that previously established properties still applied and many failed to test validity in the study setting. Only eight instruments (n = 34) had three or more properties rated as excellent. These are summarized in Table VI. All eight instruments aspects of EBP behaviour, while five aspects of knowledge and attitudes. Only one instrument EBP skills. Discussion This systematic review includes 35 studies that used 34 instruments. In general, testing of the psychometric properties of the included instruments was inadequate. Only nine instruments had at least one aspect of validity and reliability tested. This variability in the quality of instruments was similarly found in a systematic review of instruments measuring EBP knowledge and skills in occupational therapists (Glegg and Holsti, 2010). If instruments have not demonstrated their validity, the study results are questionable because an instrument needs to be reliable and valid to generate accurate results (Kielhofner, 2006). Validity and reliability are interconnected; an unreliable instrument cannot be valid, and a reliable instrument is not necessarily valid because, although it may produce consistent results, it may not be measuring what is intended (Kielhofner, 2006). As expected, the instruments that had undergone more investigation of their psychometric properties were those that were used in more than one study. This finding supports Bowling s (2009) recommendation to use existing instruments rather than develop new ones. However, as the properties of an instrument are contextually dependent, validity and reliability need to be tested in the setting in which the study will be conducted even when validated and reliable instruments are used without changes (Boynton and Greenhalgh, 2004; Streiner and Norman, 2008). This is particularly important considering that all the instruments used in the included studies were developed and tested in high income countries, which may differ substantially from middle-income and low-income contexts. Many studies modified instruments used in previous studies. It is concerning that where this was carried out, the researchers assumed that the properties of the original instrument still applied. Ideally, a previously validated instrument should be used in its original form, but this may not be possible if the context in which the instrument was developed differs from that in Occup. Ther. Int. 23 (2016) John Wiley & Sons, Ltd. 81

24 Evidence-based Practice Survey s Buchanan et al. Table III. s from included studies classified by EBP learner attribute (n = 34) and name of instrument (if applicable) EBP knowledge Skills in EBP Attitudes to EBP EBP behaviour Bennett et al. (2003) Attitudes Use in practice, perceived Confidence in skills barriers to use Bennett et al. (2007) Use of OTseeker and other databases, perceived ability to find relevant evidence, perceived contribution of OTseeker to changes in practice Caldwell et al. (2007) Views on relevance and key aspects of EBP Use of, and access to, databases, experience of research changing practice Confidence to engage in EBP Cameron et al. (2005) Value of EBP Application of EBP to intervention, resources to support EBP Cooke et al. (2008) Access to, and use of, research information Curtin and Jaramazovic (2001) Views and perceptions of EBP Involvement in EBP activities Dopp et al. (2012) Attitudes to EBP Access to, and use of, evidence sources, barriers to implementing EBP Dysart and Tomlin (2002) Attitudes towards research findings and their relevance to practice, valuing understanding and application of research to practice confidence using Access to, and use of EBP resources, use of research to develop or alter treatment plans, facilitators and barriers to EBP databases and the internet and doing critical appraisal Funk et al. (1991); BARRIERS Scale Perceived barriers to research utilization Gilman (2011) Information needs for practice, sources of evidence-based information used, use of online databases, success finding information, perceived ability to analyse and apply information in practice Gosling and Westbrook (2004) Graham et al. (2013) Attitudes to research and EBP Confidence in EBP skills Use of an online evidence database Success finding evidence, impact of database on practice, perceived barriers to using the database s of evidence used Barriers and facilitators to EBP 82 Occup. Ther. Int. 23 (2016) John Wiley & Sons, Ltd.

25 Buchanan et al. Evidence-based Practice Survey s Table 3. (Continued) and name of instrument (if applicable) EBP knowledge Skills in EBP Attitudes to EBP EBP behaviour Heiwe et al. (2011) Hu et al. (2012) Humphris et al. (2000) Karlsson and Törnquist (2007) Knowledge of information sources Attitudes and beliefs towards EBP Perceived benefits and limitations of EBP Confidence in finding and appraising literature relevant to practice Views and perceptions of EBP Attitudes towards research and its use Perceptions and attitudes towards research-related activities as part of OT Perceived ability to perform research-related activities Access to, and use of, evidence sources, applying practice guidelines Involvement in EBP activities Accessing and implementing research to practice, barriers and facilitators to EBP Current and intended future involvement in research-related activities, barriers to engaging in research-related activities McCluskey (2003) Self-rated EBP knowledge Self-rated EBP skills Attitudes to EBP Frequency of completing EBP steps, Confidence in knowledge and skills related to EBP Perceived barriers and solutions to EBP McKenna et al. (2005) Metcalfe et al. (2001); BART Pain et al. (1996); EROS Pain et al. (2004); adapted General Use of Research Pain et al. (2004); Knowledge Acquisition Survey Pain et al. (2004); Individual semi-structured interviews Philibert et al. (2003) Perceived increase in knowledge as a result of the information on OTseeker (online database) Self-rated knowledge of research concepts Perceptions of OTseeker Perceived importance of research Access to and reasons for using OTseeker, information used in clinical decision-making, change in practice from using OTseeker Perceived barriers to research Research utilization Self-rated use of research Availability and use of information sources Use of research in practice Pomeroy et al. (2003) Perceived improvement in EBP skills Confidence in searching for, reading, understanding and appraising literature Attitudes towards research in practice s of knowledge to guide practice, use of research in practice Pollock et al. (2000) Attitudes to EBP Involvement in EBPrelated activities, reported use of EBP, perceived barriers to EBP Involvement in EBP Use of EBP Occup. Ther. Int. 23 (2016) John Wiley & Sons, Ltd. 83