Deirdre J. Knapp, Teresa L. Russell, Carrie Byrum, and Shonna Waters

Transcription

1 FR Entry-Level Practice Analysis Update for Physical Therapist Assistant Licensure Examinations Offered by the Federation of State Boards of Physical Therapy Deirdre J. Knapp, Teresa L. Russell, Carrie Byrum, and Shonna Waters Prepared for: Federation of State Boards of Physical Therapy (FSBPT) 509 Wythe Street Alexandria, VA February 14, 2007

2

3 FR Entry-Level Practice Analysis Update for Physical Therapist Assistant Licensure Examinations Offered by the Federation of State Boards of Physical Therapy Deirdre J. Knapp, Teresa L. Russell, Carrie Byrum, and Shonna Waters Prepared for: Federation of State Boards of Physical Therapy (FSBPT) 509 Wythe Street Alexandria, VA February 14, 2007

4

5 ENTRY-LEVEL PRACTICE ANALYSIS UPDATE FOR PHYSICAL THERAPIST ASSISTANT LICENSURE EXAMINATIONS OFFERED BY THE FEDERATION OF STATE BOARDS OF PHYSICAL THERAPY Acknowledgments A large number of people, in addition to the authors, were instrumental in conceptualizing and completing the work described in this report. Dr. Cynthia Searcy, FSBPT s Managing Director of Assessment, both oversaw the work as well as provided invaluable guidance and assistance throughout the process. Her contributions to this effort were critical factors in the work s successful completion. Members of the supporting groups of experts organized by FSBPT are listed individually in the appendix. We would like to particularly thank Rebecca Porter, PT, Ph.D., who participated in both the Oversight Panel and the Policy Group and provided an extra measure of additional support between meetings. Of course, we also thank all the physical therapists who completed practice analysis surveys in support of this important effort. We would also like to acknowledge the support and insights of Mark Lane, PT (FSBPT s Vice President for Professional Standards and Assessment) throughout the course of this project. From HumRRO, we wish to thank Vasileios Papadimitriou, Pam O Quinn, Peter Ramsberger, LaVonda Murray, Christoper Lewis, Mary Warthen, Dan Furr, and Jeff Barnes for their efforts. Thanks also go to Brian Katz, formerly of HumRRO. Human Resources Research Organization (HumRRO) i

6 ENTRY-LEVEL PRACTICE ANALYSIS UPDATE FOR PHYSICAL THERAPIST ASSISTANT LICENSURE EXAMINATIONS OFFERED BY THE FEDERATION OF STATE BOARDS OF PHYSICAL THERAPY Table of Contents Acknowledgments... i Introduction...1 Supporting Expert Groups...1 Oversight Panel... 2 PTA Task Force... 2 Policy Group... 2 Overview of Technical Approach...3 Clear Differentiation of Work Activities and Knowledges... 4 Two Surveys... 5 Survey Rating Scales... 6 Survey Administration and Analysis... 6 Blueprint Construction and Review... 6 Work Activities...6 Knowledges...8 Blueprint Weights...8 Finishing Touches and Final Reviews...9 Initial Information Gathering...9 Review of Existing Documentation... 9 NPTE Item Writer Interviews Survey Development...10 Structure Development of Work Activity and Knowledge Lists First Drafts...10 Interviews...10 Oversight Panel Input...11 PTA Task Force Input...11 Post-Meeting Revisions...11 Construction of Web-Based and Paper Surveys Survey Pilot Test...12 Incentives for Participating Administration Process Response Rates Results and Subsequent Revisions to Surveys and Administration Process Substantive Edits...13 Survey Length...14 Knowledge Categories...14 Final Surveys ii Human Resources Research Organization (HumRRO)

7 Table of Contents (Continued) Survey Administration...16 Administration Process and Sampling Plan Response Rates, Data Processing, and Final Sample Size...17 Sample...19 Description Representativeness Content Analysis of Open-Ended Responses...25 Initial Data Analysis and Blueprint Construction...25 Work Activities Compute Criticality Index...25 Check for Differences in Ratings by Years of Experience...26 Interrater Agreement...27 Determination of an Empirical Decision Rule for Deleting Activities...28 Knowledges Check for Differences in Ratings by Years of Experience...28 Determination of an Empirical Decision Rule for Deleting Knowledges...29 Interrater Agreement...29 Oversight Panel Review and Input...30 Work Activities Knowledges PTA Task Force Review and Input...31 Work Activities Knowledges Blueprint Weighting Exercise Linkage Exercise Policy Group Review and Input...33 Finalizing the Blueprint and Associated Materials...33 Suggestions for Future Studies...35 Process Development of a Practice Analysis Updating System Conclusion...37 References...38 Human Resources Research Organization (HumRRO) iii

8 Table of Contents (Continued) List of Appendices Appendix A - Supporting Expert Group Members... A-1 Appendix B - FSBPT Pilot Test Participant Feedback Questions...B-1 Appendix C - Physical Therapist Assistant Work Activity Survey...C-1 Appendix D - Physical Therapist Assistant Knowledge Survey... D-1 Appendix E - Content Analysis of Survey Comments...E-1 Appendix F - Annotated Work Activities with Survey Results...F-1 Appendix G - Annotated Knowledges with Survey Results... G-1 Appendix H - Constructed Weights... H-1 Appendix I - Linkage Results...I-1 Appendix J - Blueprint with Category Descriptions...J-1 Appendix K - Final Work Activities... K-1 Appendix L - Final Knowledges...L-1 List of Tables Table 1. Response Rate for Pilot FSBPT Work Activity and Job Knowledge Surveys...13 Table 2. PTA Work Activity Category Structure (Survey Version)...15 Table 3. PTA Knowledge Category Structure (Survey Version)...16 Table 4. Percent of Contacts Accessing an On-Line or Paper Survey...17 Table 5. Final Response Rates by Survey...18 Table 6. PTA Knowledge Survey Background Frequencies and Percentages...20 Table 7. PTA Knowledge Survey Background Means and Standard Deviations...22 Table 8. PTA Work Activity Survey Background Frequencies and Percentages...23 Table 9. PTA Work Activity Survey Background Means and Standard Deviations...24 Table 10. PTA Work Activity Absolute Value of Subgroup (1& 5) Effect Sizes...26 Table 11. PTA Work Activity Statements with Subgroup 1 to 5 Effect Sizes greater than Table 12. Estimates of Inter-Rater Reliability for Work Activity Rating Scales...27 Table 13. Cross-tabulation of Mean Importance and Mean Frequency Ratings for PTA Work Activities...28 Table 14. PTA Knowledge Statement Ratings Absolute Value of Subgroup (1& 4) Effect Sizes..29 Table 15. PTA Knowledge Statements with Subgroup 1 to 4 Effect Sizes greater than Table 16. Estimates of Inter-Rater Reliability for Knowledge Rating Scales...29 Table 17. PTA Holistic Weights...32 Table 18. Final Work Activity Categories...34 Table 19. Final Knowledge Categories...35 iv Human Resources Research Organization (HumRRO)

9 Table of Contents (Continued) List of Figures Figure 1. Overview of project approach... 3 Figure 2. Practice analysis timeline Figure 3. Screenshot of work activity survey Section 2 instructions... 7 Figure 4. Screenshot of knowledge survey Section 2 instructions Figure 5. Knowledge testability rating scale Human Resources Research Organization (HumRRO) v

10

11 ENTRY-LEVEL PRACTICE ANALYSIS UPDATE FOR PHYSICAL THERAPIST LICENSURE EXAMINATIONS OFFERED BY FSBPT Introduction The Federation of State Boards of Physical Therapy (FSBPT) develops and administers the licensure testing program for Physical Therapists (PTs) and Physical Therapist Assistants (PTAs). FSBPT administers the National Physical Therapy Examinations (NPTEs) to candidates for licensure or certification in all 50 states, the District of Columbia, Puerto Rico, and the Virgin Islands. Approximately every 5 years, FSBPT updates the practice analysis studies in support of updating blueprints for the PT and PTA licensure examinations. The subject of this report is the 2006 PTA practice analysis. This project was conducted with contractual support from the Human Resources Research Organization (HumRRO). HumRRO is a non-profit personnel research and consulting firm dedicated to creating quality testing and training programs that improve human, occupational, and organizational effectiveness. Credentialing examinations are designed to be content-valid, meaning that test content corresponds closely with occupational requirements. Despite efforts to develop a content validity index (e.g., Hambleton, 1980; Lawshe, 1975; Rovinelli & Hambleton, 1977), content validity cannot be quantified with a single number. Rather, it is the documented process of test design and development that demonstrates the extent to which an examination assesses the domains of knowledge and skill that it should. The primary foundation for a content-valid examination is a job analysis study (also known as a practice analysis, role delineation study, or occupational analysis depending on the context in which the analysis is being conducted). Job analysis findings are used to develop a test blueprint that specifies the format and content of the examination (e.g., number of items representing each task or knowledge category). Typically, items written to match the blueprint specifications are evaluated by several subject matter experts (SMEs) to further verify their appropriateness for inclusion on the test. Thus, the purpose of the present job/practice analysis study was to help ensure the PTA NPTE covers important, current, entry-level PTA work requirements in appropriate proportions. A particular goal was to create an updated test blueprint that would be easy to interpret and use and result in high quality multiple-choice questions. Supporting Expert Groups The physical therapist assistant practice analysis update was conducted with the help of multiple groups identified by FSBPT to play key roles in the process. These included an Oversight Panel, a PTA Task Force, and a Policy Group. Individual members of these groups are listed in Appendix A. Human Resources Research Organization (HumRRO) 1

12 Oversight Panel FSBPT contacted highly experienced PTs and PTAs familiar with the NPTE test development process, test blueprint, and current professional issues. Six physical therapists with experience in PT or PTA education and/or clinical practice agreed to participate as Oversight Panel members; most of them had extensive experience in academic education and administration. Four of the six panel members were female and they ranged in experience from 19 to 37 years (median = 24 years). Their specialty areas included orthopedics, cardiovascular and pulmonary, neurology, pediatrics, geriatrics, wound care, and modalities. The role of the Oversight Panel was to work with FSBPT and HumRRO to think through major project issues, fine tune details of the technical approach, and provide subject matter expert input to the process. The panel met for two in-person meetings and project staff conferred with the panel via teleconference several times during the course of the project. PTA Task Force FSBPT appointed a 15-member PTA Task Force to serve as the primary source of subject matter expert input at key stages of the practice analysis. In particular, the Task Force was primarily responsible for (a) developing lists of relevant entry-level PTA work activities and associated knowledge areas and (b) translating survey results into a new NPTE blueprint. Nominees for the Task Force were solicited from American Physical Therapy Association (APTA) section chairs as well as board members and administrators of FSBPT member jurisdictions. An important goal was to represent PTAs from a variety of specialties, work settings, and geographic locations. The Task Force included 11 female members and experience ranged from 2 to 30 years (median = 11 years). Seven regions of the U.S. were represented, along with 12 different specialties, and nine distinct work settings. The Task Force met for two in-person meetings during the course of the project. The PTA Task Force meetings were held in conjunction with a similarly constituted PT Task Force so that their efforts could be coordinated as needed. Policy Group FSBPT anticipated that some aspects of the PT and PTA practice analysis approach and results might be controversial and in any case would benefit from a final review by a group of key stakeholder representatives. This Policy Group included 12 individuals. Some participants represented PT and PTA licensing boards, educators, clinicians, and employers. There was also a public member, a member of the FSBPT Ethics and Legislation Committee, a member of APTA s Advisory Panel on Practice, and a representative from the Commission on Accreditation for Physical Therapy Education (CAPTE). Finally, the group included liaisons from the FSBPT Board and the Technical Advisory Group. The Policy Group met in-person once toward the end of the practice analysis study. 2 Human Resources Research Organization (HumRRO)

13 Overview of Technical Approach There are a seemingly infinite number of ways to conduct a job analysis, and the literature is rife with opinions about the right way to do it (e.g., Gael, Cornelius, Levine, & Salvendy, 1988; Harvey, 1991; Raymond, 2001). Moreover, there are a variety of ways to describe different aspects of a typical job analysis study that differ more in semantics than in substance. This is particularly true when looking at the literature associated with different application areas (e.g., credentialing, education, training, personnel testing). A full-scale job analysis typically involves gathering information from available documents (e.g., job descriptions, educational or training curricula) and from job incumbents (and often their supervisors or employers) through job observation, interviews, workshops, and electronic or paper-based surveys. A fairly traditional job analysis will identify the important activities performed by a job incumbent (e.g., by identifying critical tasks, task domains, or responsibilities). Often, the job analysis will also identify the knowledge, skills, and abilities (KSAs) or competencies required for successful performance of the job. A job analysis may also identify important contextual features of the job (e.g., characterizing the work environment). In the credentialing arena, the scope of the job analysis is often limited to the identification of tasks or task domains and knowledge and/or skill requirements relevant for a licensure or certification examination (thus, for example, excluding cognitive abilities and personality characteristics). Regardless of which job descriptors are used, most job analyses will attempt to prioritize job requirements, and to identify the most critical ones. It is also common practice to link critical tasks with KSAs to further define how these two types of descriptors are related. Figure 1 shows the major steps in the 2006 PTA practice analysis and a project timeline is provided in Figure 2. Note that we coordinated work on the PTA and PT practice analyses so that differences (especially with regard to the wording of work activity and knowledge statements) would reflect substantive differences rather than arbitrary differences in phrasing. The practice analysis method differed from that used in previous FSBPT-sponsored PTA practice analyses (ACT 1995, 2002) in several ways so in this section we discuss the rationale behind key aspects of our approach. Initial Information Gathering Survey Development Survey Pilot Test Survey Administration Data Cleaning Data Analysis Blueprint Construction Figure 1. Overview of project approach Human Resources Research Organization (HumRRO) 3

14 Date Dec 2005 Dec 2005 Jan 2006 Jan 2006 Jan 21-22, 2006 Jan 27-29, 2006 Feb May 2006 June 2006 July 2006 Aug Sep 2006 Sep 2006 Oct 19 20, 2006 Oct 27 29, 2006 Nov 13, 2006 Dec 2006 Activity Talk with NPTE item writers to discuss current exam blueprints Review resource materials to generate lists of work activities and knowledges Meet with local PTAs (and PTs) to review preliminary work activity and knowledge lists Meet with Oversight Panel to discuss project approach and review work activity and knowledge lists Meet with PTA (and PT) Task Force to review and revise work activity and knowledge lists Develop draft work activity and knowledge surveys Conduct survey pilot study Produce final surveys Administer surveys Conduct data analysis Meet with Oversight Panel to review survey results and discuss test blueprint structure Meet with PTA (and PT) Task Force to review survey results, conduct blueprint weighting exercise, and conduct work activity/knowledge linkage exercise Meet with Policy Group to review process and resulting blueprint Put finishing touches on blueprint; prepare final report Figure 2. Practice analysis timeline. Clear Differentiation of Work Activities and Knowledges Work activities (also often called tasks) describe what people do on their job. Knowledges, skills, abilities, and other personal characteristics are what enable people to perform these work activities. They are two distinct ways of characterizing work requirements. In the prior practice analysis studies, work activity information was collected on surveys and the associated knowledge and skill requirements were identified in a separate process. The resulting distinctions between the two types of characterizations were not clear. In contrast, working with a small number of local SMEs and with input from FSBPT staff, we used available materials (e.g., the prior PTA practice analysis survey, the NPTE test blueprint) to develop draft work activity and knowledge lists and categories at the start of the project. By developing the work activity and knowledge lists simultaneously, we could more effectively ensure conceptual distinctions between the two characterizations of work requirements. Additionally, we paid particular attention to the knowledge statements in an effort to make them as easily interpretable as possible and to craft them in a way that would make them maximally testable on a multiple-choice examination. Careful attention to the issue of 4 Human Resources Research Organization (HumRRO)

15 interpretability and testability in the practice analysis process had the goal of making the resulting test blueprints easier for test item developers and candidates to use and to enhance the quality of the multiple-choice examinations. Indeed, it was with this in mind that we determined that we should limit the list to knowledge areas rather than skills since skills are not generally testable on traditional multiple-choice examinations. We also needed to create meaningful work activity and knowledge categories. The categories provided a way to easily conceptualize the requirements of the PTA profession and helped to organize the practice analysis surveys. The knowledge categories would also eventually become the primary rubric categories in the new blueprint. Two Surveys There are differences of opinion about collecting KSA data using a survey approach. For example, Raymond (2001) acknowledged several advantages to collecting KSA data using a survey, including the outcome that resulting test blueprints are much more directly linked to the input from a large sample of practitioners; however, he also expressed the belief that KSAs are relatively abstract and thus less easily interpreted by survey respondents than observable task statements. While we agree with the concerns expressed by Raymond (2001), we feel that their merit is highly dependent on the occupation and KSAs in question. For occupations with reasonably highly educated incumbents, our experience is that practitioners are not likely to have problems inferring KSA requirements. Moreover, our review of the 2002 NPTE test outlines suggested that knowledge statements could be written for easy review and interpretation on a survey. In our view, the benefits of having more direct input from a larger number of representative SMEs far outweighed the concerns associated with using this approach. Accordingly, we developed two surveys one to gather information about entry-level work activities from entry-level PTAs and one to gather information about entry-level knowledges from experienced PTAs and PTs who supervise PTAs. This approach had a significant added benefit of reducing the length of a single survey containing both the work activities and knowledges. During their first workshop, the Oversight Panel established definitions for the target sample for the practice analysis surveys. First, it was determined that an entry-level PTA should be defined as an individual with 3 months through 5 years of experience. Three months was chosen because the established probationary period is 90 days, and 5 years was chosen to ensure inclusiveness and allow consistency with the previous survey. Second, experienced PTAs were defined as individuals who have been licensed for more than 5 years and who spend at least 50% of their time in the delivery of physical therapy services in a clinical setting. Human Resources Research Organization (HumRRO) 5

16 Survey Rating Scales In comparison to prior PTA practice analysis surveys, we used fewer rating scales to gather information about work activities. Previously, respondents made three judgments for each work activity: (a) frequency, (b) consequences, and (c) acquisition (entry-level or advanced). Because we collected work activity data only from entry-level PTAs, an acquisition rating indicating whether the work activity was entry-level or advanced was not really necessary. We used a frequency scale similar to that used previously and an importance scale that was conceptually similar to the consequences scale used on prior surveys. Knowledge statements had not previously appeared on the practice analysis survey. We collected a single, holistic importance rating for each knowledge statement. The Oversight Panel was instrumental in finalizing the rating scales, which are shown in screenshots of the final surveys (see Figures 3 and 4). Survey Administration and Analysis The surveys were offered in both electronic and paper formats, with most respondents opting for the Internet-based version. As with most other surveys of this type, the response rate in the 2000 practice analysis study was on the low side (approximately 27%). To provide the kind of sample sizes that would provide reliable and valid inferences from the data and meet the expectations of the profession, we thus selected a very large sample of PTAs (approximately 7,500) to receive the surveys. FSBPT also offered an ipod TM drawing as an incentive to participate. Achieving reasonably large numbers of returned surveys allowed us to be relatively strict about the data we included in our final analysis samples. For example, although they were still eligible for the ipod drawing, respondents who were retired or not currently employed at least part time were excluded from the final sample. That is, we could afford to lose some data to help ensure the quality of the inferences based on the extent to which the retained responses represented the population we were seeking to sample. Blueprint Construction and Review The survey data were analyzed and used in conjunction with input from the Oversight Panel and PTA Task Force and eventually the Policy Group to generate the new NPTE blueprint. Work Activities The new blueprint was intended to be knowledge-based. The work activities were still important, however, because they served as the foundation for inferring the relevant knowledges to include on the survey. We also conducted a linkage exercise in which we identified the knowledge areas that were required to perform important entry-level work activities. This served as a safeguard against including knowledge areas in the final blueprint that should not be there. It could also be use to create a roadmap that lists the work activity areas most often linked with each knowledge-based blueprint category. Such a roadmap could be used as a job aid for NPTE item writers. 6 Human Resources Research Organization (HumRRO)

17 Figure 3. Screenshot of work activity survey Section 2 instructions. Figure 4. Screenshot of knowledge survey Section 2 instructions. Human Resources Research Organization (HumRRO) 7

18 Thus, a first step in creating the blueprint and associated item writer roadmap was to drop work activities that were infrequently performed and rated relatively unimportant by entry-level PTAs. Work activities were flagged for deletion based on an empirical decision rule, then the SME groups reviewed the results. They made limited exceptions to the survey results, usually with the rationale that the original phrasing of certain work activities led to a disconnect between the intended meaning of the statement and how survey respondents interpreted it. Knowledges Again, we used an iterative process that incorporated the survey results with the input from the Oversight Panel and PTA Task Force. The first step was to identify knowledges that should be deleted because they failed to meet a minimum average importance rating on the survey. In a few cases, the empirical decision was over-ridden, generally because relevant content was covered by other retained knowledge statements or was integrated into other related statements. Because the knowledges were to serve as the foundation for the new NPTE blueprint, we were particularly concerned about the extent to which each could be adequately assessed in a multiple-choice test format. Because the Oversight Panel members had considerable experience with NPTE item development, they were asked to judge the extent to which each knowledge was testable on the NPTE. This process resulted in a very intense examination and discussion of the knowledge statements which in turn resulted in several revisions to clarify wording and some reorganization of the knowledges across categories. Blueprint Weights We included a section in the knowledge survey that asked respondents to indicate the percentage of NPTE content that should be devoted to each knowledge category. Specifically, respondents were asked to apportion 100 points across the various categories. This provided a direct, holistic judgment about examination content priorities from persons who are currently practicing in the profession. We included a similar category judgment exercise on the work activity survey. In this case, however, respondents were asked to allocate 100 points across the categories to reflect the relative importance of each category. We did not have a specific use for the work activity category data, but thought it would be wise to collect it given that final decisions about how the exam blueprint would be constructed had not yet been made. Thus we had three sources of information to help determine how to distribute items across content areas on the NPTE: (a) holistic weights which were the mean knowledge category weights from the survey respondents, (b) constructed weights computed by summing the mean knowledge ratings for the knowledge statements within each category, and (c) the judgment of the Oversight Panel and Task Force members. Changes made to the knowledge categories themselves and deletion of unimportant and untestable knowledge statements made the holistic judgments from the survey less useful than they otherwise would have been. Constructed weights are heavily influenced by the number of knowledge statements in each category. This is a problem when the individual statements vary in their level of specificity (which was an issue here) and the degree to which they can be tested with multiple-choice test questions. 8 Human Resources Research Organization (HumRRO)

19 Therefore, we used a weighting process that relied largely on SME judgment tempered with the survey response data. The process is analogous to a modified Angoff standard setting approach. After spending a day reviewing and discussing the survey results, individual knowledge statements, and associated categories, the PTA Task Force members independently allocated 100 points to each knowledge category (and for subcategories within each category, as applicable). They shared and discussed their judgments in a facilitated discussion, and were given the constructed weight estimates. Then the Task Force members made a second and final round of judgments. The final weights were rounded means of the second round of judgments by the Task Force members. Finishing Touches and Final Reviews To complete the blueprint package, we developed knowledge category descriptions and added brief statements to the blueprint to clearly articulate what the NPTE is designed to cover and what it not designed to cover. The blueprint and knowledge category descriptions were provided to the Oversight Panel for final review and approval prior to submission to the FSBPT Board for adoption. Initial Information Gathering Having provided an overview of the practice analysis and blueprint process, the next several sections describe each major project step in some detail. Review of Existing Documentation FSBPT provided HumRRO staff with available job information and guidance regarding where to obtain sources of job information (e.g., web sites) about the PTA occupation. The sources provided to or obtained by HumRRO consisted of a variety of materials ranging from reports from the prior practice analysis to study guides for students. Some of the documentation provided for review included: 2002 PTA Practice Analysis Reports 2004 NPTE Examination Development Policies 2005 Form Rollout Report 2004 Psychometric Audit of NPTE PTA test blueprints (2002 and 1996) Descriptions of Responsibilities for Item Bank Review Committee and Exam Construction & Review Committee 2002 Pass Point Standard Setting Study and powerpoint slides Item Writer Training Slides Physical Therapist Assistant Examination Review Guide to Physical Therapist Practice A Normative Model of Physical Therapist Assistant Education The information collected was reviewed to gain a better understanding of the PTA occupation and the NPTE. It also provided information used by project staff to generate an initial list of PTA work activities and associated knowledges. Human Resources Research Organization (HumRRO) 9

20 NPTE Item Writer Interviews Because HumRRO staff had not been previously involved in the development of NPTE test items, we wanted to learn from those who had worked with the 2002 blueprint prior to commencing with the remainder of the project. Thus, as an initial step in the practice analysis, HumRRO staff conducted telephone interviews with two SMEs who had previously worked on the PTA NPTE (and two SMEs who had worked with the PT NPTE) to discuss the 2002 blueprint. Questions were designed to help determine the difficulty or ease of writing test items or concepts underlying test items for various blueprint outline categories. Other questions probed the interviewees perception of the conceptual structure of the content outline as well as their reactions to the idea of an alternate categorization scheme based in part on systems. Survey Development Structure The surveys included the following sections: Section 1: Respondent Background Information Section 2: Work Activities, or Section 2: Knowledge Statements Section 3: Weighting of Work Activity Categories, or Section 3: Weighting of Knowledge Categories There was also a place at the end of the survey in which respondents could write in comments or specify work activities or knowledge areas they felt were missing from the survey. For Section 1, we started with the prior survey (administered in 2001) and revised the background questions as needed with input from the Oversight Panel and Task Force. For Section 3, we created a fairly simple process for obtaining judgments from respondents about the relative importance of the knowledge (or work activity) categories. Development of the work activity and knowledge lists needed for Section 2, as described below, was a more involved process. First Drafts Development of Work Activity and Knowledge Lists Project staff generated an initial list of PTA work activities and associated knowledges using available written documentation. This included the prior practice analysis survey and exam blueprint, as well as other relevant documentation. Interviews We were interested in getting at least some subject matter expert (SME) input on the initial draft lists prior to review by the Oversight Panel and PTA Task Force. HumRRO staff 10 Human Resources Research Organization (HumRRO)

21 interviewed three local PTs (and one PTA via teleconference) to conduct this preliminary review. We obtained the interviewees feedback on the broad categorization schemes on the draft work activity and knowledge lists, the draft work activity and knowledge statements (rewording as necessary for clarity), and the question of whether there were work activities or knowledges missing from the lists. This input was used to refine both the PT and PTA lists. Oversight Panel Input The first 2-day Oversight Panel workshop focused primarily on discussion of major issues associated with the project. Specifically, issues related to survey sample characteristics were determined, definitions were established for entry-level and experienced PT/PTAs, and ratings scales to be used on the work activities and knowledge surveys were discussed. In addition, the preliminary work activities and knowledge lists were reviewed and discussed. These lists were provided to the Oversight Panel for their review prior to the workshop. PTA Task Force Input Prior to their first workshop, Task Force members were provided with workshop materials, instructions, and an explanation of the meeting agenda. This afforded them an opportunity to become familiar with the workshop requirements and decision-making tasks prior to the workshop, thus maximizing the efficiency of the workshop itself. During the workshop, the Task Force reviewed and revised the draft work activity and knowledge lists. To ensure a comprehensive review of each list, HumRRO asked questions similar to those listed below to elicit feedback. Is the list comprehensive in terms of the work activities/knowledges required to perform successfully at the entry-level? What additional work activities/knowledges are required to perform successfully at the entry-level? Are there any work activities/knowledges currently on the list that are not required to perform successfully at the entry-level? Does the wording of each work activity and knowledge statement accurately reflect what is needed to perform successfully? Is each work activity and knowledge statement grouped in an appropriate category? Is each work activity and knowledge statement relevant to all work settings? Will the knowledge categories (and system categories, if applicable) serve well as test blueprint categories? Post-Meeting Revisions After the Task Force meeting, HumRRO and FSBPT staff performed additional clean-up of the lists and categories. This included a follow-up with the revised lists to the Oversight Panel and Task Force as well as a teleconference with the Oversight Panel to review changes. Construction of Web-Based and Paper Surveys Human Resources Research Organization (HumRRO) 11

22 HumRRO constructed web-based surveys that could be accessed online by any PTA or PT with access to the Internet. The web-based survey was designed to be hosted on HumRRO s server. It included several desirable features, including a pause feature that allowed respondents to start the survey and finish it at a later time. Note also that there was a single link to the web surveys for both PTAs and PTs. Respondents were directed to the appropriate survey (PTA knowledge, PTA work activity, PT knowledge, PT work activity) based on information embedded in the access code 1 that they were provided in advance. We also constructed paper-based surveys for distribution to individuals who preferred that format. The length and complexity of the surveys warranted considerable review and pretesting of the draft surveys by HumRRO and FSBPT staff. Survey Pilot Test The purpose of the pilot test was to (a) try out the survey administration procedures and (b) evaluate the clarity of the survey instructions and items. Although we reviewed the data that were collected, we did not make changes based on analysis of the data. Rather, in the pilot test versions of the surveys, respondents were asked for feedback about the survey and the initial contact letter they received asking them to participate. We used this input and information about the response rates to revise the surveys and administration procedures. Incentives for Participating In this type of survey, response rates tend to be low. We attempted to make the surveys as short as possible as well as simple and appealing. In addition, FSBPT offered an ipod drawing from among individuals who completed a survey. The initial plan was to offer 10 ipods. Administration Process Our goal was to administer the pilot test surveys in a manner that would as closely as possible mirror plans for the full scale administration. After receiving a database of addresses for licensed or certified PTAs (which FSBPT staff constructed using input from member jurisdictions), we followed the steps listed below: Selected a random sample of 100 (50 each, entry-level and experienced PTAs). Created a contact letter with FSBPT input. Assigned random access codes to each sample member with the first number indicating group membership (i.e., entry-level PTA, experienced PTA). Created individualized letters and envelopes using mail merge. Mailed contact letters. Created reminder postcard with access codes and mailed approximately 2 weeks later. 1 We wanted to collect PTA knowledge data from experienced PTs who supervise PTAs. Thus, some PTs who indicated that they supervise PTAs in a question posed in Section 1 of the PT knowledge survey were redirected to the PTA knowledge survey when they reached Section Human Resources Research Organization (HumRRO)

23 The contact letters were mailed on June 20, 2006 and the reminder postcards were mailed on June 29, Although we stated that the survey would be available for 2 weeks, we did not actually suspend the survey. That is, we kept the web survey link live until it was replaced with the revised survey so we could continue to collect data. Our hope was that any changes we made to the surveys would be minor enough that we could include the pilot test data in the full scale sample. This turned out to be the case. Response Rates As mentioned, contact letters were sent to 50 individuals in each group (entry-level PTAs, experienced PTAs). We pilot tested two parallel surveys for PTs (again, 50 from each group) at the same time. Ten letters were returned as undeliverable. Only one paper survey (for an entry-level PTA) was requested and it was returned completed. If we adjust the original sample size to account for the undeliverable letters (i.e., = 190), the overall response rate was 18.5%. This figure includes responses ultimately discarded because of excessive missing data. Table 1. Response Rate for Pilot FSBPT Work Activity and Job Knowledge Surveys Group Number Responded Percent Responded Entry-level PT 6 12% Experienced PT 11 22% Entry-level PTA 8 16% Experienced PTA 12 24% Overall % Note that response rates were slightly higher for PTA respondents and higher for experienced PTs and PTAs than for entry-level PTs and PTAs. This pattern also held for the fullscale administration described later. Results and Subsequent Revisions to Surveys and Administration Process Appendix B shows the questions posed to pilot test participants on the web surveys and on the paper-based scannable forms. We reviewed all respondent comments with FSBPT staff and jointly determined the changes that would be made based on the pilot test experience. Substantive Edits Respondents identified very few problems with the surveys. We made minor edits to three items in the Background section based on respondent comments. On the work activities surveys, a few respondents found the instructions for completing the ratings in Section 2 on the web survey unclear. These instructions were revised accordingly. On the knowledge surveys, a couple of respondents indicated that it was difficult to distinguish ratings related to the endocrine system from ratings related to the metabolic system. The PTA Task Force had considered combining the items for these two systems, so we went ahead and made that change. We were somewhat concerned that the weighting exercises in Section 3 would be difficult for respondents, but the pilot test respondents reported little difficulty with this section. Human Resources Research Organization (HumRRO) 13

24 Survey respondents were asked to indicate any content they felt was missing from Section 2 of their surveys. They offered few suggestions. In response to a suggestion to add basic pharmacology, we edited the knowledge of medical management statements for each system to re-order the examples so that medications were not listed last. Survey Length Our biggest concern was the length of the surveys. Section 2 of the of the work activity survey included 157 statements all of which were rated on two scales. There were 165 statements on the knowledge survey, which were rated on one scale. Despite the length, several respondents commented positively about the survey and little was said about the length. That said, 30% of the people who accessed the web survey did not complete it, and most of those individuals stopped in Section 2. We made two changes to the surveys to address our continuing concerns about their length. First, we added motivational comments throughout Section 2 of the web surveys. These comments were intended to help respondents gauge their progress and urge them to keep working to the end. Second, we developed an alternate version of Section 2 for each survey that essentially reversed the order of the statements. We figured that this might improve the quality of the data (to the extent that fatigue sets in when rating the later statements in the section) and would give us more data on the later statements on the surveys (assuming we include the data of individuals who did not finish the surveys in the analyses). In addition to these changes, FSBPT increased the incentive drawing of 10 ipods to 20 ipods (or the cash equivalent). We also greatly increased the number of people who would be asked to complete the surveys, from about 4,000 total to 21,000 total (across both PTs and PTAs). Knowledge Categories Although not identified as a problem by survey respondents, further internal review of the knowledge categories indicated that they needed additional clarification. Knowledge statements were not reorganized, but some category titles were edited. Final Surveys Copies of the final paper-based knowledge and work activity surveys are provided in Appendixes C and D. Screenshots of the web-based surveys were shown previously in Figures 3 and 4. There were 157 work activities and 165 knowledge statements, all of which were organized into categories, and in some cases subcategories. Table 2 summarizes the category structure used on the survey for the work activities and Table 3 summarizes the organizing structure used on the knowledge survey. 14 Human Resources Research Organization (HumRRO)

25 Table 2. PTA Work Activity Category Structure (Survey Version) Patient/Client Assessment (70) Information Gathering (2) Systems Review (5) Data Collection (63) Aerobic Capacity/Endurance (3) Anthropomorphic (3) Arousal, Attention, & Cognition (5) Assistive & Adaptive Devices (4) Circulation (3) Nerve Integrity (4) Environmental & Community Integration (4) Gaits, Locomotion, & Balance (2) Integumentary Integrity (5) Joint Integrity (4) Motor Function (3) Muscle Performance (1) Neuromotor Development & Sensory Integration (3) Orthotic, Protective, & Supportive Devices (5) Pain (1) Posture (1) Prosthetic Devices (4) Range of Motion (3) Reflex Integrity (2) Sensory Integrity (2) Respiration (1) Interventions (79) Procedural Interventions (63) Therapeutic Exercise/ Therapeutic Activities (8) Functional Training (4) Manual Therapy Techniques (7) Devices & Equipment (6) Airway Clearance Techniques (5) Integumentary Repair & Protection Techniques (6) Electrotherapeutic Modalities (3) Physical Agents (21) Mechanical Modalities (3) Non-Procedural Interventions (16) Communication (5) Documentation (7) Education (4) Patient/Client & Staff Safety (8) Emergency Procedures (3) Environmental Safety (3) Infection Control (2) Total (157) Note. The number of statements in each category/subcategory is noted in parentheses. Human Resources Research Organization (HumRRO) 15

26 Table 3. PTA Knowledge Category Structure (Survey Version) Systems (118) Cardiac, Vascular, & Pulmonary (24) Musculoskeletal (23) Neuromuscular & Nervous (18) Integumentary (13) Metabolic & Endocrine (12) Gastrointenstinal (10) Genitourinary (12) Multi-System (6) Equipment & Therapeutic Modalities (13) Equipment & Devices (8) Therapeutic Modalities (5) Patient Management (14) Psycho-Social & Cultural Issues (5) Environmental Factors (1) Safety (6) Teaching & Learning (2) Practice Management (20) Evidence-Based Practice (6) Business Guidelines (14) Total (165) Note. The number of statements in each category/subcategory is noted in parentheses. Survey Administration Administration Process and Sampling Plan We selected a relatively large stratified random sample of 21,000 names from the mailing list which contained 244,909 PTs and PTAs. Our goals were to ensure that the final sample of respondents would be large enough to provide stable results and representative of the population of PTs and PTAs. We expected that the stratified random sample of 21,000 would easily provide large, stable, representative databases for each survey. The response rate in the last PTA practice analysis was 27% (ACT Inc., 2002). If we were to obtain a comparable response rate in the current practice analysis, we would receive 3,570 surveys, about 900 for each of the four surveys, more than would be necessary to achieve our goals. As with the pilot study, whether someone was entry-level or experienced was gauged based on the date of license as specified in the database. We knew that this would not be completely accurate, so we oversampled entry-level categories to account for individuals who obtained a new license in a jurisdiction after having experience elsewhere. Next, we drew a stratified sample so that the number selected from each jurisdiction would reflect its contribution to the population. The target samples were assigned as follows: Entry-level PT = 7,500 Experienced PT = 6,000 Entry-level PTA = 4,500 Experienced PTA = 3, Human Resources Research Organization (HumRRO)

27 We mailed initial contact letters on August 2, 2006, and follow-up postcards 2 weeks later. 2 The letter and postcard provided participants their unique access code and directed them to the website where the survey was hosted. A toll-free number was given so that individuals without Internet access could obtain a paper-and-pencil survey. We distributed paper surveys to 247 individuals. The survey closed on September 10. Because of the relatively low volume of paper surveys, the data were hand-entered into the web survey database with appropriate quality control checks. Response Rates, Data Processing, and Final Sample Size We planned to administer the PTA knowledge survey to PTs who reported that they supervised PTAs. As we monitored survey returns, it became evident that we were getting plenty of PTA knowledge data, but the PT knowledge survey responses were lower than desired. Therefore, we changed the web survey to direct all subsequent experienced PTs to the PT knowledge survey. Table 4 traces the returns of pilot test and final surveys in combination. As shown, 26.71% of the individuals who were contacted (i.e., 20,071) accessed either an on-line survey or a paper and pencil version (i.e., 5,361). Approximately 34% of the experienced PTs and PTAs accessed the survey compared to roughly 21% of the entry-level PTs and PTAs. Table 4. Percent of Contacts Accessing an On-Line or Paper Survey Respondent Type Status Entry Experienced Entry Level Experienced Unregulated Level PT PT a PTA PTA PTA b Total Letters Sent 7,483 6,001 4,549 3, ,282 Number Returned as ,211 Undeliverable Number Contacted 6,932 5,717 4,296 3, ,071 Number Accessed 1,454 1, , ,361 Survey Percent Accessed Survey 20.98% 33.62% 21.67% 34.05% 23.47% 26.71% a Experienced PTs were directed to either the PT knowledge survey or the PTA knowledge survey. b Unregulated PTAs were directed to either the work activity or knowledge survey depending upon experience. As a convenience for respondents (and in attempt to bolster the response rate), we had designed the on-line survey such that respondents could save their data and return to complete the survey. Unfortunately this also results in missing data since some respondents never return to the survey after accessing it the first time. Our next step, therefore, was to clean and screen the data to (a) ensure that responses were in range and appropriate for the response format, (b) ensure that respondents met criteria for inclusion for the questionnaire based on their years of experience and employment status, and (c) identify respondents with too much missing data for inclusion in subsequent analyses. 2 The state of Hawaii did not provide addresses to FSBPT, so 100 contact letters (and follow-up postcards) were sent in bulk for distribution by the state licensure board. Human Resources Research Organization (HumRRO) 17

28 Table 5 shows how the 5,361 individuals accessing the surveys were distributed across surveys. As shown, numbers accessing each survey were quite large (e.g., 936 for the PTA work activity survey and 1,409 for the PTA knowledge survey). We applied a 10% missing data rule for each survey. 3 That is, respondents had to have completed 90% or more of the work activity or knowledge ratings, depending upon the survey being administered. 4 To ensure the credibility and appropriateness of the samples, we also dropped (a) respondents who were not currently working either full or part time (Background Question #9) or (b) did not fit within the experience limits for the survey based on the year of first licensure. 5 Table 5 provides the number of respondents dropped as a result of the missing data, employment status, and experience level screens. Table 5. Final Response Rates by Survey Survey Status PT Work PT PTA Work PTA Activity Knowledge Activity b Knowledge a,b Total Number Accessed Survey 1,454 1, ,409 5,361 Eliminated by missing data screen Number Completed 1,067 1, ,249 4,397 Actual Response Rate 15.39% 24.70% 17.53% 36.10% 21.91% Eliminated by either employment status or experience level screen Final Analysis Sample 1,023 1, ,126 4,063 Usable Response Rate 14.76% 22.72% 16.11% 32.54% 20.24% a 357 experienced PT respondents who indicated that they supervise PTAs were directed to the PTA Knowledge Survey. b Of the 23 unregulated PTAs in the full sample, five responded to the PTA WA survey, and 18 responded to the PTA KS survey. After data cleaning, four were in the PTA WA final sample and 14 were in the PTA KS final sample. 3 Before applying the missing data rule, we ran standard data quality checks to ensure that responses were appropriate and in range for the questions. The on-line survey controlled the response boundaries, therefore, out-ofrange or inappropriate responses were extremely rare (i.e., less than 10 out of almost 5,361 cases had an inappropriate response). In those rare occurrences, we set the data to missing for that item. 4 We eliminated respondents with more than 10% missing data separately for Sections 2 and 3 of the survey. Some respondents completed Section 2 but did not complete Section 3. We retained all respondents who had <10% missing data on Section 2, regardless of whether they completed Section 3. We also eliminated respondents who did not meet our criterion for inclusion on Section 2 but did meet the criterion for inclusion on Section 3 because we wanted people to respond to Section 3 after having examined the specific tasks/knowledges in Section 2. 5 Recall that work activities respondents were to have 5 years or less of experience while knowledge survey respondents were to have more than 5 years of experience. The on-line survey branched individuals to specific surveys based on their responses to a background question about the number of years of experience they had since they were first licensed. Another survey question asked respondents to report the year in which they were first licensed. We used the second survey question as a check to ensure that the respondent was indeed entry-level (for the work activities survey) or experienced (for the knowledge survey). Specifically, for the work activities survey, we dropped respondents who were first licensed before 2001 (i.e., had more than 5 years of experience). For the knowledge survey, we dropped respondents who were first licensed after 2001 (i.e., had less than 5 years of experience). 18 Human Resources Research Organization (HumRRO)

29 In all, the data collection, cleaning, and screening steps resulted in data bases that were of very high quality. As shown in Table 5, the final analysis samples were large (i.e., 696 for the PTA work activity survey and 1,126 for the PTA knowledge survey). We were, therefore, confident that the final analysis samples were large enough to provide stable data representative of PTA work. The screening and cleaning procedures also ensured that the respondents were appropriate for each survey and that they appeared to have responded conscientiously to it. In the remainder of this report, we will focus on the PTA results. The PT results are documented in a separate report. Sample Description Tables 6 and 7 describe the final sample of work activity survey respondents. Tables 8 and 9 present data that describe the final sample of knowledge survey respondents on which subsequent analyses were based. Note that the knowledge survey sample included 74% experienced PTAs and 26% PTs who supervise PTAs. Representativeness To truly evaluate the representativeness of the survey samples, we would need information about the populations of entry-level and experienced PTAs. Such information is not available. Comparing our samples with the samples from preceding PTA practice analysis surveys is of limited value because the representativeness of those samples has also not been independently evaluated. Moreover they are based on a single survey administered to all PTAs rather than two surveys administered separately to entry-level PTAs and to experienced PTAs and PTs. 6 Nonetheless, a comparison of background variables shared among the surveys does show considerable consistency in gender, racial/ethnic group, and highest degree earned. The principal areas of responsibility and type of facilities also showed consistent patterns across surveys. As you expect, the percent of time in direct patient care was higher for entry-level respondents compared to the knowledge survey respondents (84% versus 75% spending more than 81% of time). The comparable figure for the prior survey was 73%. When the 2006 survey sample descriptions were presented to the Oversight Panel and Task Force members, several participants noted an apparent under-representation of respondents from the northeast region of the country. This prompted us to review the sampling plan and led to the discovery that this region had indeed been undersampled. The state of New York had provided FSBPT with information about just one-quarter of their licensed PTs and PTAs. To obtain the appropriate sample, then, it would have been necessary to oversample the New York portion of the database by a factor of four. This was not done. Once this problem was discovered, we drew another sample from New York and reopened the survey. We recomputed estimates to demonstrate that the undersampling did not have a substantive impact on the final results and blueprint. These analyses are detailed in an addendum to the present report. 6 In the 2001 survey, 83% of the respondents roughly met the 5-year definition of entry-level. Human Resources Research Organization (HumRRO) 19

30 20 Human Resources Research Organization (HumRRO) Table 6. PTA Knowledge Survey Background Frequencies and Percentages Respondent's Job Frequency Percent Years of Experience Frequency Percent PT to 9 yrs PTA to 14 yrs Total 1, to 24 yrs or more yrs Gender Frequency Percent Total 1, Female Male Employment Status Frequency Percent Total 1, Full-time salaried/hourly Part-time salaried/hourly Race/Ethnicity Frequency Percent Full-time self employed American Indian or Alaskan Native Part-time self employed Asian or Pacific Islander Total 1, African American or Black (Not of Hispanic Origin) White (Not of Hispanic Origin) Area of Responsibility Frequency Percent Hispanic/Latino Administration Other Consultation 5.44 Missing 3 Direct patient care 1, Total 1, Research Sales/marketing 2.18 PT Highest Degree Earned Frequency Percent Teaching/clinical education Baccalaureate/Bachelor's Missing 4.36 Master's Total 1, Current Doctoral Student Doctoral Percentage Direct Patient Care Frequency Percent Medical Doctor or Doctor of Dental Surgery % Total % % % % Total 1,

31 Human Resources Research Organization (HumRRO) 21 Table 6. (Continued) PTA Highest Degree Earned Frequency Percent Census Region of the Country for Primary Practice Frequency Percent High School/Grandfathered Northeast Associate Midwest or More Associate Degrees South Current Baccalaureate/Bachelor's Student West Baccalaureate/Bachelor's Degree Missing Current Master's Student Total 1, Master's Doctoral Other Total

32 22 Human Resources Research Organization (HumRRO) Table 7. PTA Knowledge Survey Background Means and Standard Deviations Percent Time Spent in Each Facility n M SD Percent of Patient Population by Age n M SD Academic institution 1, years and younger 1, Acute care hospital 1, to 65 years old 1, Health and wellness facility 1, years and older 1, Health system or hospital-based outpatient facility or clinic 1, Not Applicable 39 Industry 1, Missing 0 Patient's home/home care 1, Total 1, Private outpatient office or group practice 1, Rehabilitation hospital 1, Research center 1, School system 1, SNF/ECF/ICF 1, Other Facility 1, Missing 0 Total 1, Percent of Time Spent in Patient Care Services n M SD Acute care 1, Ambulatory/out-patient care 1, Chronic/long-term care 1, Critical care 1, Emergency care 1, Home care 1, Hospice care 1, In-patient rehabilitation 1, Prevention/wellness/health promotion 1, School-based care 1, Sub-acute care 1, Other patient care services 1, Not Applicable 10 Missing 0 Total 1,

33 Human Resources Research Organization (HumRRO) 23 Table 8. PTA Work Activity Survey Background Frequencies and Percentages Gender Frequency Percent Employment Status Frequency Percent Female Full-time salaried/hourly Male Part-time salaried/hourly Total Full-time self employed Part-time self employed Race/Ethnicity Frequency Percent Total American Indian or Alaskan Native Asian or Pacific Islander Area of Responsibility Frequency Percent African American or Black (Not of Hispanic Origin) Administration 6.86 White (Not of Hispanic Origin) Consultation 2.29 Hispanic/Latino Direct patient care Other Research 3.43 Total Sales/marketing 1.14 Teaching/clinical education 3.43 PTA Highest Degree Earned Frequency Percent Missing 3.43 Associate Total Current Baccalaureate/Bachelor's Student Baccalaureate/Bachelor's Degree Percentage Direct Patient Care Frequency Percent Master's % Other % Total % % Years of Experience Frequency Percent % <1.5 yrs Total to 2.5 yrs to 3.5 yrs Census Region of the Country for Primary Practice Frequency Percent 3.5 to 4.5 yrs Northeast to 5yrs Midwest Total South West Missing Total

34 24 Human Resources Research Organization (HumRRO) Table 9. PTA Work Activity Survey Background Means and Standard Deviations Percent Time Spent in Each Facility n M SD Percent of Patient Population by Age n M SD Academic institution years and younger Acute care hospital to 65 years old Health and wellness facility years and older Health system or hospital-based outpatient facility or clinic Not Applicable 35 Industry Missing 1 Patient's home/home care Total Private outpatient office or group practice Rehabilitation hospital Research center School system SNF/ECF/ICF Other Facility Missing 1 Total Percent of Time Spent in Patient Care Services n M SD Acute care Ambulatory/out-patient care Chronic/long-term care Critical care Emergency care Home care Hospice care In-patient rehabilitation Prevention/wellness/health promotion School-based care Sub-acute care Other patient care services Not Applicable 5 Missing 1 Total

35 Content Analysis of Open-Ended Responses Work Activities Survey At the end of the survey, we asked respondents to tell us about any relevant entry-level work activities they felt should be added to the survey and to provide any comments on the survey. We content analyzed write-in responses from 35% (n = 240) of the respondents. Results of this content analysis were provided to the Task Force and is shown in Appendix E. There were no consistently identified work activities that appeared to be missing from the survey list. Knowledge Survey Forty-four percent (n = 499) of the knowledge survey respondents wrote comments. Appendix E provides the summary information that was provided to the Task Force to assist them in interpreting the survey results. The bottom line was that the write-in comments did not consistently identify salient knowledges that were missing from the survey. Nothing was mentioned by more than 4% of either sample and most suggested additions were actually already on the survey or were not suitable for inclusion on the list (e.g., personality characteristics). Initial Data Analysis and Blueprint Construction Before finalizing the survey data analysis plan, the Oversight Panel met via teleconference to provide feedback on potential blueprint structures. They favored a structure that included systems divided into subcategories. This common vision of the ultimate blueprint helped guide the remainder of the work to construct it. Compute Criticality Index Work Activities Recall from Figure 3 that the work activities were rated on both frequency and importance. For some analyses, it was useful to combine this information into a single score. We thus computed an overall index of criticality as follows: Criticality = Frequency + 2 * Importance This index weights importance twice as heavily as frequency. This index was recommended by HumRRO and accepted by the Oversight Panel for two major reasons. First, the work activities that are affected the most by different weighting strategies are the ones that are either (a) infrequently performed but nevertheless important or (b) frequently performed and unimportant. As shown later in Table 13, none of the PTA work activities fell into this latter category; few ever do. Second, job experts are usually concerned that low-frequency, highly important work activities will be overlooked. Weighting importance more heavily than frequency addresses this concern. Human Resources Research Organization (HumRRO) 25

36 Check for Differences in Ratings by Years of Experience The 5-year definition of entry-level spans a good bit of time, so there was an interest in examining the extent to which respondents with more or less experience in this 5-year window differed in their responses to the survey. Therefore, we computed mean criticality ratings on work activities for five subgroups of respondents based on years of experience: 1. <1.5 yrs experience (licensed in 2005 or 2006) to 2.5 yrs experience (licensed in 2004) to 3.5 yrs experience (licensed in 2003) to 4.5 yrs experience (licensed in 2002) to 5 yrs experience (licensed in 2001) The correlations between the vectors of mean criticality scores were very high (i.e.,.99). This means that regardless of experience level, respondents criticality scores were similar. We also checked to see whether there were any large differences in importance ratings on individual work activity statements. We calculated the effect sizes between all adjacent levels of experience. The effect size (d) is a standardized difference score; d of 1.0 indicates that there is one standard deviation difference between the means of the two subgroups. A common rule of thumb is that.20 is a small effect,.50 is a moderate effect, and.80 is a large effect (Cohen, 1988). None of the effect sizes were large or even moderate using Cohen's definition. Table 10 provides frequency distributions for the number of effect sizes within different ranges (these are absolute values of effect sizes). Although we looked at effect sizes for all levels, for the sake of brevity, only the Level 1-5 comparison is presented here. Table 10. PTA Work Activity Absolute Value of Subgroup (1& 5) Effect Sizes Absolute Value of Effect Size (Subgroups 1 & 5) Number of Work Activities.00 to to to to.40 1 Total 157 While there were not large differences between the subgroups, we did take a look at the items with the largest differences. The statement with the largest effect size between levels 1 and 5 for PTAs (i.e., d >.30) is listed in Table 11. Note that the entry-level PTAs rated these items higher than the experienced PTAs did. 26 Human Resources Research Organization (HumRRO)

37 Table 11. PTA Work Activity Statements with Subgroup 1 to 5 Effect Sizes greater than.30 Criticality by PTA/PT Years of Experience Level 1 Level 5 < 1.5 yrs 4.5 to 5 yrs # n M SD N M SD 103 Apply dressings (e.g., hydrogels, vacuumassisted closure, wound coverings) Interrater Agreement We computed intraclass correlation coefficients (ICCs; McGraw & Wong, 1996; Shrout & Fleiss, 1979) to estimate the level of concordance among survey respondents. We computed two types of ICCs, consistency and agreement. Consistency coefficients indicate the extent to which individuals concur on the relative level of the rating (e.g., Task X is more important than Task Y and less important than Task Z). Agreement coefficients are more stringent. They indicate how well the individuals agree on the absolute rating (e.g., Importance of Task X equals 3 and importance of Task Y equals 5. ). As shown in Table 12, the agreement coefficients are slightly lower than the consistency ICCs. We have reported consistency and agreement ICCs estimated for a single rater and for 500 raters. The 500-rater ICCs, all 1.00, estimate the level of agreement in our sample because we had more than 500 raters. 7 The data are highly consistent across raters and strengthen our confidence in the data. The single rater estimates can be used to compare the reliability of the different rating scales. While the single rater reliabilities for both frequency and importance were high, the frequency ratings were somewhat more reliable than the importance ratings. Table 12. Estimates of Inter-Rater Reliability for Work Activity Rating Scales Rating Scale Number of Consistency ICC Agreement ICC Items 1-Rater 500-Raters 1-Rater 500-Raters Frequency Importance Criticality Weights The software program limited the number of raters to 500 raters with complete data (i.e., no missing data). We identified individuals with no missing data and randomly selected 500 for this analysis. Also, the ICCs were computed for Shrout & Fleiss s (1979) case 2 which is appropriate when raters are selected from a larger population and each rater rates all items. In case 2, we want to be able to generalize conclusions to the larger population. Human Resources Research Organization (HumRRO) 27

38 Determination of an Empirical Decision Rule for Deleting Activities Table 13 cross-tabulates the mean frequency and importance ratings for the PTA work activities. There were 157 work activities on the survey. HumRRO evaluated four alternative decision rules for deleting work activities (various combinations of mean frequency and mean importance ratings less than 1 or 2) and initially recommended a rule in which a work activity would be dropped if the mean frequency rating was less than 1 and the mean importance rating was less than 2. Table 13. Cross-tabulation of Mean Importance and Mean Frequency Ratings for PTA Work Activities Mean Frequency Mean Importance Rating Total # of Rating 0 thru thru thru thru thru 5 Statements 0 thru thru thru thru thru Total # of Statements Knowledges We conducted parallel analyses on the knowledge survey data, as detailed below. Check for Differences in Ratings by Years of Experience We computed mean importance ratings on knowledges for four subgroups of knowledge survey respondents based on years of experience: 1. 5 to 9 yrs experience to 14 yrs experience to 24 yrs experience or more yrs experience The correlations between the vectors of importance score means were very high (i.e.,.99). This means that regardless of experience level, respondents tended to rate the importance of the knowledges similarly. We also checked to see whether there were any large differences in importance ratings on individual knowledge items. We calculated the effect sizes between all adjacent levels of experience. None of the effect sizes were large or even moderate using Cohen's definition. Table 14 provides frequency distributions for the number of effect sizes within different ranges (these are absolute values of effect sizes). Although we looked at effect sizes for all levels, only the Level 1-4 comparison is presented here. 28 Human Resources Research Organization (HumRRO)

39 Table 14. PTA Knowledge Statement Ratings Absolute Value of Subgroup (1& 4) Effect Sizes Absolute Value of Effect Size (Subgroups 1 & 4) Number of Knowledges.00 to to to to.40 3 Total 165 While there were not large differences between the subgroups, we did take a look at the items with the largest differences. The three statements with the largest effect sizes between levels 1 and 4 (i.e., d >.30) are listed in Table 15. As shown, more experienced respondents (with 25 years or more of experience) rated these items slightly higher than less experienced respondents did. But, the differences in means were fairly small. Table 15. PTA Knowledge Statements with Subgroup 1 to 4 Effect Sizes greater than.30 Importance by Years of Experience Level 1 Level 4 5 to 9 years 25 or more yrs Knowledge n M SD N M SD 035 Knowledge of appropriate types of musculoskeletal system data collection techniques and their applications(e.g., manual muscle testing, isokinetic testing) Knowledge of joint function and mobility Knowledge of physiological response of the musculoskeletal system to various types of tests/measures Determination of an Empirical Decision Rule for Deleting Knowledges We examined two possible cut scores on mean importance 2.5 (half-way between Minimally Important and Important) and 3.0 (Important). After reviewing the results of the two cut scores, the recommended cut was set at 2.5. Interrater Agreement We computed intraclass correlation coefficients (McGraw & Wong, 1996; Shrout & Fleiss, 1979) to estimate the level of agreement among survey respondents. The agreement estimates were high and added to our confidence in the quality of the data. Table 16. Estimates of Inter-Rater Reliability for Knowledge Rating Scales Rating Scale Number of Consistency Agreement Items 1-Rater 500-Rater 1-Rater 500-Rater Importance Weights Human Resources Research Organization (HumRRO) 29

40 Oversight Panel Review and Input Work Activities The Oversight Panel reviewed the survey results (including the mean and standard deviation of ratings on each work activity) and recommended empirical decision rules for deletion of work activities. There was some discomfort with activities that would be dropped based on the empirical decision rule suggested by HumRRO. Accordingly, they adopted a slightly more stringent rule, which was to drop an activity if the mean frequency rating was less than 1 and the mean importance rating was less than 1.5. Even with this more stringent rule, the group retained a small number of work activities the fell beneath the cutoff. In most cases, they rephrased the statements to clarify their meaning. They also adjusted the way in which the work activities were categorized to make the categories and subcategories easier to interpret and work with. Knowledges The Oversight Panel also reviewed the knowledge results. As with the work activities, they evaluated the recommended decision rule for deleting knowledge statements (which they endorsed). They did not retain any statements that did not meet the empirical rule, but did drop some statements that did. In many cases, the content of a dropped statement was covered by a retained statement or integrated into another statement. The panel also reorganized some of the knowledges. To minimize problems writing items in the future, the Oversight Panel, whose members were experienced item writers and reviewers, judged the testability of each retained knowledge statement. First, each panel member independently rated each statement on a 3-point scale (see Figure 5). After making independent ratings, the group discussed their ratings and reached consensus on the knowledge statements that were problematic. This exercise prompted a close examination of many of the knowledge statements and resulted in identification of additional knowledges that needed at least minor rephrasing and the deletion of additional knowledges that, although important, were not viewed as testable on a multiple-choice examination. How much of the content reflected in this statement is reasonably testable with fact-based multiple-choice test items? 1. Little or None 2. Some 3. Most Figure 5. Knowledge testability rating scale. 30 Human Resources Research Organization (HumRRO)

41 PTA Task Force Review and Input Work Activities We provided the PTA Task Force members with the work activity results annotated with the changes (and associated rationale for those changes) made by the Oversight Panel and applicable changes the PT Task Force made to their work activities. The PTA Task Force reviewed the material and made additional changes, which were also annotated for subsequent review by the Policy Group. The work activities with the associated survey ratings data (e.g., means and standard deviations), annotated with changes made to individual statements, are shown in Appendix F. The appendix has three tables, as follows: Table F.1 lists the final work activities following review and revision. Activities that were retained despite falling beneath the decision rule cutoff are noted. Table F.2 lists work activities that were dropped based on the empirical decision rule. Table F.3 lists work activities that were dropped even though they met the empirical decision rule. Content for these activities was incorporated into other retained statements. Knowledges We provided the PTA Task Force members with the knowledge list annotated with the changes (and associated rationale for those changes) made by the Oversight Panel and relevant changes the PT Task Force made to their knowledge list. The PTA Task Force reviewed the material and made additional changes, which were also annotated for subsequent review by the Policy Group. The knowledge statements with the associated survey results are shown is provided in Appendix G. The appendix has three tables, as follows: Table G.1 lists the final knowledge statements following review and revision. Table G.2 lists knowledges that were dropped because they did not meet the empirical decision rule for importance. Table G.3 lists knowledges that were dropped because they are not reasonably testable across jurisdictions in a multiple-choice format or were of questionable relevance to patient protection. Blueprint Weighting Exercise After spending considerable time reviewing and discussing the individual knowledge statements, knowledge categories, and associated survey results, the Task Force participated in a blueprint weighting exercise. The exercise was conducted as follows: Participants independently allocate 100 points across the blueprint categories. The judgments made by individual participants were written on a whiteboard. The constructed weights based on the survey responses were added to the whiteboard. A HumRRO staff facilitated a discussion of the judgments, ensuring that large discrepancies among the initial judgments were explored in some detail. Human Resources Research Organization (HumRRO) 31

42 The participants made a second round of independent judgments. The holistic weights were based on the mean of the second round of judgments. This process was repeated for the systems categories, which all had three to four subcategories. The constructed weights used in this exercise are provided in Appendix H. The final (unrounded) holistic weights derived from the exercise are shown in Table 17. Table 17. PTA Holistic Weights Mean SD # Test Items Cardiac, Vascular, & Pulmonary Systems Fundamental Sciences Tests & Measures Interventions Musculoskeletal System Fundamental Sciences Tests & Measures Interventions Neuromuscular & Nervous System Fundamental Sciences Tests & Measures Interventions Integumentary System Fundamental Sciences Tests & Measures Interventions Metabolic System & Endocrine System Fundamental Sciences Tests & Measures Interventions Gastrointestinal System Multi-System Equipment & Devices Therapeutic Modalities Safety, Protection, & Professional Roles Teaching & Learning Research & Evidence-Based Practice Linkage Exercise The purposes of the linkage exercise were to (a) ensure that no knowledge areas made it on the blueprint that could not be demonstrably linked to work activities and (b) provide information that would facilitate item writing efforts. In the linkage exercise, we asked Task Force members to rate the importance of various knowledge areas for performing various types of work activities effectively. The relevance of each knowledge area to each work activity category was judged using a 3-point scale. They entered their judgments directly into Excel spreadsheets. The final results are shown in Appendix I. All knowledge categories were linked to multiple work activity categories (mean linkage rating > 2.00). 32 Human Resources Research Organization (HumRRO)

43 Policy Group Review and Input The Policy Group met in November HumRRO staff reviewed the practice analysis process and results. Participants were asked to react to this information from the ultimate perspective of public safety. The group reacted positively to the process and results of the practice analysis. Their discussions in relation to the PTA practice analysis results prompted the following changes to the PTA blueprint, intended primarily to help clarify the content of the blueprint categories. Changed "Fundamental Sciences" to "Clinical Application of Physical Therapy Principles and Foundational Sciences" to convey that NPTE items measure application of foundational sciences in the context of physical therapy. Changed "Tests and Measures" to "Data Collection" to ensure consistency in NPTE and physical therapy terminology for physical therapist assistants. Added a purpose statement to the NPTE for PTAs to emphasize the focus of the exam and to ensure that the NPTE meets best practices for licensure examinations. Re-oriented the blueprint to present the systems as subcategories rather than as categories to better represent the array of knowledge required for entry-level physical therapist assistant activities. In addition to reviewing the practice analysis, FSBPT staff urged members of the Policy Group to think about how to address important areas that are not reasonably tested on the NPTE (e.g., ethics, communication skills). An example strategy used in some jurisdictions is to have local jurisprudence examination to cover some of this type of material. Another strategy is requiring continuing education in these areas. Although not directly impacting the PTA blueprint, the Policy Group did determine that the following changes should be made to the final work activity list. Additional changes (specifically, elimination of four more work activities for the same reasons noted below) were approved by the FSBPT Board of Directors in February Eliminated the activity pertaining to Grade I spinal mobilization from the list of entry-level PTA activities while the Policy Group did not unanimously recommend this change, FSBPT's Board of Directors, after considering feedback from this group, approved the modification. Combined Grade I and Grade II peripheral mobilization/manipulation activities in the following statement: Perform peripheral joint range of motion (e.g., active, passive, Grade I-II mobilization). Eliminated the example "end feel" from activities pertaining to performing tests of spinal and joint mobility. Finalizing the Blueprint and Associated Materials Considerable care was given to reviewing the knowledge category titles to make sure that they conveyed the appropriate information to users of the NPTE blueprint. The Oversight Panel helped re-title blueprint the blueprint categories as noted above. We also worked with the Oversight Panel to develop descriptions of the blueprint categories and added explanatory statements to the blueprint that emphasizes the type of material that is and is not on the NPTE. Human Resources Research Organization (HumRRO) 33

44 Simple listings of the final retained work activities and knowledges are provided in Appendixes K and L, respectively. As summarized in Table 18, there are 146 work activities sorted into three major categories and 34 subcategories. As summarized in Table 19, there are 146 knowledges sorted into 12 categories. Table 18. Final Work Activity Categories Patient/Client Assessment: Information Gathering (2) Systems Review (5) Data Collection Cardiovascular and Pulmonary (6) Anthropomorphic (2) Arousal, Attention, & Cognition (4) Assistive & Adaptive Devices (4) Nerve Integrity (2) Environmental & Community Integration/Reintegration (4) Gait, Locomotion, & Balance (2) Integumentary Integrity (5) Joint Integrity/Range of Motion (3) Motor Function (3) Muscle Performance (1) Neuromotor Development & Sensory Integration (3) Orthotic, Protective, Supportive, and Prosthetic Devices (9) Pain (1) Posture (1) Reflex Integrity (1) Sensory Integrity (2) Evaluation & Diagnosis (15) Development of Prognosis, Plan of Care, & Goals (5) Interventions: Procedural Interventions Therapeutic Exercise/Therapeutic Activities (11) Functional Training (4) Manual Therapy Techniques (4) Devices & Equipment (6) Integumentary Repair & Protection Techniques (3) Electrotherapeutic Modalities/Physical Agents (12) Mechanical Modalities (3) Non-Procedural Interventions Communication (5) Documentation (7) Education (3) Patient/Client & Staff Safety: Emergency Procedures (3) Environmental Safety (3) Infection Control (2) Note. The number in parentheses is the number of work activities in the category. 34 Human Resources Research Organization (HumRRO)

45 Table 19. Final Knowledge Categories Cardiac, Vascular, & Pulmonary Systems: Clinical Application of Physical Therapy Principles and Foundational Sciences (20) Data Collection (4) Interventions (5) Musculoskeletal System: Clinical Application of Physical Therapy Principles and Foundational Sciences (17) Data Collection (4) Interventions (5) Neuromuscular and Nervous Systems: Clinical Application of Physical Therapy Principles and Foundational Sciences (13) Data Collection (5) Interventions (7) Integumentary System: Clinical Application of Physical Therapy Principles and Foundational Sciences (8) Data Collection (4) Interventions (6) Metabolic & Endocrine Systems: Clinical Application of Physical Therapy Principles and Foundational Sciences (6) Data Collection (1) Interventions (3) Gastrointestinal System: Clinical Application of Physical Therapy Principles and Foundational Sciences (1) Multi-System: Clinical Application of Physical Therapy Principles and Foundational Sciences (8) Equipment & Devices (7) Therapeutic Modalities (5) Safety, Protection, & Professional Roles (12) Teaching & Learning (3) Evidence-Based Practice (2) Note. The number in parentheses is the number of knowledge statements in the category. A small number of statements appear in more than one category. Suggestions for Future Studies No matter how well a practice analysis is done, there is still the requirement to periodically update it. With that requirement in mind, we offer suggestions for the next physical therapist assistant practice analysis study. Process Surveying just entry-level PTAs to obtain descriptive information about what they do on their jobs and experienced PTAs and PTs to obtain information about requisite entry-level knowledge requirements worked well. This approach allowed us to make the best use of each group s experience and minimized the length of a survey that would have other included both work activities and knowledges. Human Resources Research Organization (HumRRO) 35

46 Given the relatively small number of respondents who opted for paper-based surveys, it would seem reasonable to offer future surveys only via the Internet. Continuing to offer incentives for participation would seem to be a good idea. The differences between response rates for entry-level versus experienced respondent warrant consideration before the next practice analysis update. If these are generational differences in response rates, then it is possible that overall response rates will continue to drop as the currently entry-level PTAs become more experienced. Alternatively, the explanation could be the surveys themselves. The work activities survey is somewhat longer 157 statements rated on two rating scales versus 165 knowledge statements rated on a single rating scale. Development of a Practice Analysis Updating System In our experience with job and practice analysis, we have found that almost everyone would like to have a means of readily adapting to future changes in the profession and almost no one has such a system. Most organizations conduct a full scale job or practice analysis every 5 to 10 years depending upon how rapidly the profession is changing. An updating system could be used to (a) make interim adaptations to the blueprint and (b) determine when a full scale practice analysis is needed. The cornerstone of the updating system would be the idea that some areas of practice are newly emerging and others are becoming outmoded. Regulatory activities can also make interim changes to a blueprint necessary. At the time of the practice analysis, a newly emerging work activity or knowledge may not yet be prevalent enough in practice to be placed on the blueprint. Outmoded activities or knowledge may still be in use for several years because clinics have the equipment available and the training to perform them. They may be placed on the blueprint with the idea that eventually they would be eliminated. The updating system would, thus, require first reviewing our notes and discussion materials to identify: Knowledges and work activities that were not included on the surveys or on the blueprint because they are newly emerging and not yet prevalent enough for inclusion. Knowledges and work activities that were included on the blueprint but are expected to be outmoded in the near future. Knowledges and work activities that were e that could be affected by trends in regulations. The next step would be to develop a means of updating information on the flagged items. For items that were either not yet prevalent enough for inclusion in the blueprint or still too prevalent to remove, FSBPT could conduct a short web-based survey every 2 to 3 years, like an abbreviated version of the practice analysis, to determine whether their status should be changed. We planned a process similar to this for air traffic controllers with the U.S. Federal Aviation Administration (Knapp, Morath, Quartetti, & Ramos, 1997). Regulatory changes should be tracked by a committee, perhaps assembled in an ad hoc fashion as key changes near. Next, the results would need to be folded into the blueprint through a structured process. Importantly, the updating system should also include a mechanism for determining that a full scale practice analysis is necessary. 36 Human Resources Research Organization (HumRRO)

47 Conclusion The 2006 PTA practice analysis update resulted in a new NPTE blueprint that is a valid representation of entry-level PTA requirements that are reasonably testable on a well-constructed multiple-choice examination. Moreover, there is reason to believe that the new blueprint structure will be easier for users (item writers, test candidates, educators) and more durable over time. The blueprint reflects the input of a large and representative sampling of physical therapist assistants and the careful review and consideration by stakeholder groups. Examinations based on the new blueprint will be offered starting in Human Resources Research Organization (HumRRO) 37

48 References ACT Inc. (2002). An analysis of physical therapy practice in the U.S. Iowa City, IA: Author. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Earlbaum Associates. Gael, S., Cornelius, E.T., Levine, E., & Salvendy, G. (1988). The job analysis handbook for business, industry, and government. New York: John Wiley. Hambleton, R.K. (1980). Test score validity and standard setting methods. In R.A. Berk (Ed.), Criterion-referenced measurement: The state of the art. Baltimore: Johns Hopkins University Press. Harvey, R.J. (1991). Job analysis. In M.D. Dunnette & L.M. Hough (Eds.), Handbook of industrial and organizational psychology (2 nd Ed. Vol. 2). Palo Alto, CA: Consulting Psychologists Press. Knapp, D.J., Morath, R., Quartetti, D.A., & Ramos, R.A. (1997). A strategic approach to the analysis of air traffic controller jobs. Fairfax, VA: Caliber Associates, Inc. Lawshe, C.H. (1975). A quantitative approach to content validity. Personnel Psychology, 28, McGraw, K.O., & Wong, S.P. (1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods, 1, Raymond, M.R. (2001). Job analysis and the specification of content for licensure and certification examinations. Applied Measurement in Education, 14(4), Rovinelli, R.J., & Hambleton, R.K. (1977). On the use of content specialists in the assessment of criterion-referenced test item validity. Dutch Journal of Educational Research, 2, Shrout, P.E., & Fleiss, L.L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86, Human Resources Research Organization (HumRRO)

49 Appendix A Supporting Expert Group Members Oversight Group Members Peggy DeCelle Newman - PT, MHR John Greany - PT, PhD Rebecca Porter - PT, PhD Judy White - PT, MS, DPT Peter Zawicki - PT, MS Mari Bosworth - PT Physical Therapist Assistant Task Force Members Candy Bahner - PT, MS Frank Reicherter - PTA George Ibrahim - PT, DPT Joan E. Stassel - PT, MS John Linberger, Jr. - PTA, BS Kathleen H. Barnes - PT, MS Kimberly McGaughey - PT, MPT, CCI Kimberly Rieff - PTA Kristen Messina - PTA Manson Wong - PT, MS Marian S. Over - PT, CCI Monica Fowers - PTA Nancy Brox - PTA Phyllis Lyn MacAlpine - PTA, MS Tamara Klintworth Kirk - PT, CCI Betsy Schultz - BA, PTA Policy Group Members Scott Oppler - PhD Peg Hiller - PT Roy Junkins - PTA Tom Mohr - PT, PhD Rebecca Porter - PT, PhD Lisa Shoaf - PT, PhD Besty Schultz PTA, BA Mary Jane Harris - PT, MS Jean-Pierre Viel - PT, MSPT, OCS Julia Works - JD Suzanne O Brien - PT, MS, NCS Linda Gustafson - PT Human Resources Research Organization (HumRRO) A-1

50

51 Appendix B FSBPT Pilot Test Participant Feedback Questions Web Survey In front of the General Introduction: This is a pilot version of the 2006 practice analysis survey, which is being completed by a relatively small number of people. At the end of each section, you will be asked to provide feedback. The more specific your feedback (e.g., suggested rewording to clarify a question), the more helpful that will be. At the end of the survey, you will be asked to provide any additional feedback you have about the survey process. Before the practice analysis survey is sent to a much larger group of physical therapists assistants, it will be revised as needed based on this pilot test. If there are no major changes, we will incorporate your data into the final analysis sample. Note that by completing this pilot survey, you will qualify for the ipod drawing. Thank you for participating in this very important pilot study. At the end of Section 1 (Background): Is the information provided in the introduction clear and complete? If not, what information is unclear/incomplete? Did you have trouble answering any of the background questions? If so, which ones and why? At the end of Section 2 (Work Activities): Are the instructions for completing this section clear? Is the frequency rating scale easy to use? Is the importance scale easy to use? Please identify by number any work activity statements that were unclear to you and briefly describe what was unclear. At the end of Section 2 (Knowledge Statements): Are the instructions for completing this section clear? Is the importance scale easy to use? Please identify by number any knowledge statements that were unclear to you and briefly describe what was unclear. Human Resources Research Organization (HumRRO) B-1

52 At the end of Section 3 (Category Weighting): Are the instructions for completing this section clear? At the end of the Survey: We have tried to identify every relevant work activity/knowledge performed [needed] by entry-level PTAs, but recognize that we may have missed some. Please use the space below to tell us about any relevant entry-level work activity [knowledge] that we failed to include on this survey. Also, if you have any comments about the survey itself, please add them below. [Note that this question appears in the final surveys as well as the pilot surveys.] Please provide any other feedback you have regarding this survey. Please provide any feedback you have regarding the initial contact letter you received asking you to participate in the survey. B-2 Human Resources Research Organization (HumRRO)

53 Paper Survey This is a pilot version of the 2006 practice analysis survey, so we would like for you to tell us about any problems you may see as you complete the survey. We will then make revisions to the survey as needed before it is sent to a larger group of respondents. Please use the bottom part of this page to provide feedback on any sections of the survey that you wish to comment on. We are interested in any comments you might have (e.g., regarding the clarity of instructions, ease of using the rating scales). We also ask for your feedback on the initial contact letter you received asking for your participation and your estimate of how long it took you to complete the survey. If you wish to suggest edits to specific questions, feel free to write them directly on the survey. Feedback on the survey: Feedback on the initial contact letter you received asking you to participate: About how long did it take you to complete this survey? Human Resources Research Organization (HumRRO) B-3

54

55 Appendix C Physical Therapist Assistant Work Activity Survey Human Resources Research Organization (HumRRO) C-1

56 C-2 Human Resources Research Organization (HumRRO)

57 Human Resources Research Organization (HumRRO) C-3

58 C-4 Human Resources Research Organization (HumRRO)

59 Human Resources Research Organization (HumRRO) C-5

60 C-6 Human Resources Research Organization (HumRRO)

61 Human Resources Research Organization (HumRRO) C-7

62 C-8 Human Resources Research Organization (HumRRO)

63 Human Resources Research Organization (HumRRO) C-9

64 C-10 Human Resources Research Organization (HumRRO)

65 Human Resources Research Organization (HumRRO) C-11

66 C-12 Human Resources Research Organization (HumRRO)

67 Human Resources Research Organization (HumRRO) C-13

68

69 Appendix D Physical Therapist Assistant Knowledge Survey Human Resources Research Organization (HumRRO) D-1

70 D-2 Human Resources Research Organization (HumRRO)

71 Human Resources Research Organization (HumRRO) D-3

72 D-4 Human Resources Research Organization (HumRRO)

73 Human Resources Research Organization (HumRRO) D-5

74 D-6 Human Resources Research Organization (HumRRO)

75 Human Resources Research Organization (HumRRO) D-7

76 D-8 Human Resources Research Organization (HumRRO)

77 Human Resources Research Organization (HumRRO) D-9

78 D-10 Human Resources Research Organization (HumRRO)

79 Human Resources Research Organization (HumRRO) D-11