2015 Exam Committees Report for the National Physical Therapy Examination Program

Transcription

1 2015 Exam Committees Report for the National Physical Therapy Examination Program

2 Executive Summary This report provides a summary of the National Physical Therapy Examination (NPTE) related activities in 2014 and This is a high level review of major exam components; development, administration, security, quality assurance and oversight, program highlights and communications. This report is intended for the Member Boards of the Federation of State Boards of Physical Therapy (FSBPT). More detailed information on the examination is available from FSBPT by request. During the time period covered within this report, the NPTE program continued to meet test industry standards for licensure examinations. In July 2014, the NPTE program received the highly coveted Stage One reaccreditation by the Buros Institute for demonstrated adherence to test industry standards. FSBPT carried out its cyclical activities for launching updated versions of the NPTE on January 1, 2015, including developing new test items, developing exam forms adhering to the NPTE blueprints, conducting analyses to update item bank statistics, and administering feedback and evaluation surveys for examination committee members and item writers to identify areas for improvement. FSBPT also continues to actively investigate possible security breaches to fulfill its responsibility to ensure that the inferences made on the basis of NPTE scores are both reliable and valid. FSBPT & NPTE Purpose FSBPT is committed to establishing and implementing a strong foundation of laws and regulatory standards in physical therapy, effective tools and systems to assess entry level and continuing competence, and public and professional awareness of resources for public protection. Ensuring the excellence, reliability, defensibility, security and validity of the National Physical Therapy Examination (NPTE) and related examinations is a major task in fulfillment of FSBPT s mission. The NPTE is the key tool used by the FSBPT in the protection of the public. It is designed to measure the knowledge and cognitive skills necessary for minimal entry level competence for Physical Therapists (PT) and Physical Therapist Assistants (PTA) in the provision of safe and effective physical therapy services. Exam Development The exam development process begins when a practice analysis is completed. We then can build content outlines, write items and build forms. Our most recent practice analysis was completed in Throughout 2013 development activities focused on creating items and building forms based on the 2011 practice analysis. In 2015, all exams administered adhere to the most recent practice analysis. There are many people involved in the process. Item Writers: Volunteers who write test items for the NPTE based on the test content outlines or blueprints. FSBPT hosted the following workshops for item writers: o o On site item writing workshops for beginning item writers, Off site item writing workshops for beginning item writers, and

3 o Item writing task force workshops for experienced item writers. Item Writer Coordinators (IWCs): Experienced item writers who provide guidance to item writers during the item development process and are responsible for verifying that new items meet the NPTE criteria. Task Forces: Groups of experienced item writers and committee members who meet to accomplish special item writing tasks, such as practice test item development, writing items areas that are under represented in the item bank, reviewing rejected but salvageable items, or developing item templates. FSBPT staff: Three in house PTs, two professional editors, and additional support staff who review items and draft forms in order to reduce the administrative load on volunteers, conduct statistical analyses, and facilitate volunteer meetings. Item Bank Review Committees (IBRCs): Committees of volunteers who perform final review and revision of items to meet NPTE standards and are made eligible for inclusion in the item bank. Examination Development Committees (EDCs): Committees of volunteers who conduct a final review, revision, and approval of each draft test form to ensure items are independent of each other and there is adequate coverage of each content area. A large component of exam development work is writing and reviewing individual items. The item development process is shown in the flow chart below.

4 Item Development Item Writer Items revised until final action taken Item Writer Coordinator Approve Editorial Review IBRC Approve Content and Editorial Review EDC Discarded Items Statistics fail to meet criteria Reject Request additional review/revision Usable Items Statistics meet criteria Psychometric Review Pre test Approve Editorial Review Overview of Test Development Procedures Once the items have been written and reviewed, the test forms can be developed. For the NPTE there are multiple test development phases, with the work of each phase being performed by psychometricians who specialize in testing and measurement and content experts who are representative of the profession in terms of area of expertise, practice setting, demographic makeup, and geographic representation. The steps in the test development procedures, including the assembly and review of operational test forms and accompanying pre test blocks, are illustrated below.

5 NPTE Form Development Process Examination Administration There are two exam types within the NPTE program; the NPTE for PTs and the NPTE for PTAs. The NPTE PT is a 250 item examination with 200 scored test questions (items) and 50 pre test or experimental items. The NPTE PTA is a 200 item examination with 150 scored items and 50 pre test items. Under standard administration conditions, candidates are allowed five hours for the NPTE PT and four hours for the NPTE PTA. The examination itself is divided into sections of 50 items. Candidates are allowed to take breaks during the examination subject to security procedures that are explained to candidates in the candidate handbook and at the test center. FSBPT utilizes Prometric as our test delivery vendor. Prometric and FSBPT work collaboratively in establishing and enforcing policies and procedures that apply to candidates on test dates. FSBPT and Prometric also work together in establishing test dates and predicting where, geographically, demand for testing appointments will be. In 2013, FSBPT announced the test dates for 2014 and 2015.

6 This year, and for the coming years, FSBPT has 4 testing events for the NPTE for PTs and 4 testing events for the NPTE for PTAs. Each testing event is one or two days long. During a testing event is the only time of the year that candidates may sit for the NPTE. Candidates taking the NPTE go to a test center of their choosing (subject to availability at the time of registration) and take a computer based examination. Prometric staff members at test centers enforce strict security policies to protect the NPTE. The Technical Advisory Panel (TAP) monitor the effectiveness and efficiency of all policies related to registering candidates for and administering the NPTE, as well as reporting examination results to candidates, licensing authorities, and other stakeholders. The TAP convenes twice per year to review the NPTE annual report, relevant policies, examination security issues, and NPTE related research. The following data show the 2014 candidate volume and pass rates from the 2015 NPTE Annual Report Test dates and candidate volumes NPTE PT NPTE PTA 2014 Administrations Date Volume Date Volume January January 29 2,498 January 15 1,490 April April 30 3,085 April 9 1,470 July July ,163 July 8 3,996 October October 29 2,993 October 8 2,093 Total 14,739 9,049 Pass Rates 2014 Physical Therapist Category 1 st time candidates Graduates of US accredited PT programs (n = 8,777) 1 st time candidates Graduates of non US PT programs (n = 1,123) 1 st time candidates Graduates of all PT programs % Passing 91.1% 37.4% 85.7% 2014 Physical Therapist Assistant Category % Passing 1 st time candidates Graduates of US 86.3% accredited PTA programs (n = 7,440) 1 st time candidates Graduates of non US PTA 67.7% programs (n = 347) 1 st time candidates Graduates of all PTA 85.5% programs Additional pass rate information is available on FSBPT s public website (

7 Additional Services In addition to overseeing the process to register candidates for and administer the NPTE, FSBPT s examination administration team provides resources related to the NPTE for candidates, jurisdictions and schools. Candidate services include online registration, the ability to purchase FSBPT s PEAT (Practice Exam and Assessment Tool), free score reports, performance feedback reports, online exam withdrawals, and score transfers. Jurisdiction services include access to an online processing system for approving registrations and receiving scores, a variety of pass rate reports, and processing requests for candidates requesting testing accommodations. School services include the ability to purchase FSBPT s PEAT for a group of students at one time, access to an online reporting system where there are three types of pass rate reports available, a faculty newsletter, and presentations on the NPTE program. Exam Security Exam security starts from the moment a candidate registers with FSBPT to take the NPTE. During the registration process candidates must read and acknowledge the NPTE Security Agreement and General Terms of Use. Security continues at the testing center where there are strict policies to verify each candidate s identity, including an electronic verification of a passport or driver s license and digital fingerprint capture. Candidates are required to place all personal belongings in a locker in the test center waiting area. Before candidates can enter the testing room they must show that their pockets are empty and a metal detecting wand is used to ensure candidates do not have concealed electronic devices. When candidates sit down to test, they see another security statement that they must acknowledge acceptance of before the exam begins. During the examination all candidates are videotaped and proctors actively monitor testing rooms. When the examination is complete, before candidates leave their computer testing station, a screen pops up to remind them that they are bound by the terms of the security agreement. Exam security efforts continue even after the examination is complete. FSBPT uses an outside consultant to conduct extensive data forensics to identify potential security breaches. In this way we monitor for people who likely had access to exam questions prior to the exam. Additionally, FSBPT has developed an internal score screening process that went into effect January 1, This process allows us to quickly identify issues with scores and prevent reporting scores we cannot validate. For NPTE administrations that are identified as potential security issues, FSBPT conducts an investigation that may involve reviewing the security camera video of the administration, interviewing the candidate, and conducting additional follow up statistical analysis to determine the validity of the NPTE score. Quality Assurance & Oversight Staff FSBPT employs high quality staff with experience and qualifications to support the psychometric, operational and physical therapy knowledge requirements of a high stakes examination program. Staff

8 members have developed operational policies and procedures that focus on quality assurance throughout all stages of exam development and administration. Technical Advisory Panel The Federation contracts with four external testing experts to provide ongoing advice on the development, administration, research, and security of the NPTE through participation in FSBPT s Technical Advisory Panel (TAP). The TAP meets with FSBPT staff twice a year. The TAP members review examination processes and statistical reports, make recommendations for improvements, and recommended areas of further research. Accreditation The Federation first received Stage One and Stage Two accreditation by the division of the Buros Center for Testing responsible for accreditation (the Buros Institute for Assessment Consultation and Outreach, or BIACO) in The Stage One accreditation recognizes that FSBPT has demonstrated the capacity, procedures, and processes that will likely support the valid interpretations of scores for the defined purpose of its testing program. The Stage Two accreditation recognizes that the test forms developed under the procedures accredited in Stage One demonstrate a match to test specifications, acceptable item performance, adequate score reliability, appropriate item exposure, and statistical model fit. The Federation received Stage One reaccreditation by BIACO in 2014 for the NPTE PT and NPTE PTA. This accreditation evaluated procedures beginning with the practice analyses conducted in 2011 and extending through the item and test development work that produced test forms administered during In 2015, FSBPT expects to receive Stage Two accreditation of the NPTE test forms administered and developed in Program Highlights Increased exam committee s capabilities to conduct green, paperless reviews. In recognition of the increasing volume of items, forms, and pretest blocks the exam committees are reviewing, FSBPT staff have created innovative ways to conduct reviews electronically rather than on paper. This process reduces the amount of pre meeting reproduction work, post meeting transcription work, and provides a more secure process for conducting reviews. Started a process to reduce exam committee overlap. Assessment staff members have been working to consolidate the Exam Development Committee (EDC) and Item Bank Review Committee (IBRC) into a single committee. NPTE exam committees consist of experts of similar qualifications and backgrounds. Moreover, the committees perform very similar duties (e.g., item and form reviews, item editing). These facts beg the question, Why have two committees? Consolidating the committee will (a) provide a deeper pool of potential volunteers to staff each meeting, (b) provide staff and volunteers some ability to assign volunteers to tasks that best suit their interests and abilities, (c) reduce redundancy in committee work, and (d) provide volunteers a better understanding of the development process in order to facilitate meeting workflow.

9 Updated School Reports. In 2013, FSBPT conducted a survey of PT and PTA program administrators to determine their perceptions of the strengths and weaknesses of the school reports, and followed that up with a task force who reviewed and provided feedback on possibilities for future school reports. In March of 2015, FSBPT introduced a new school report system that is more in line with current industry reporting standards, better utilizes graphics to aid in report interpretation, provided licensing information where available, and addresses many of the shortcomings of the previous reports. A followup survey in June of 2015 determined that program directors much preferred the new reports. Implemented enhanced score screening for security. The validity of the NPTE for determining competent PTs and PTAs is directly related to the security of the examination. Each year FSBPT conducts research and implements initiatives to improve the security of the examination. This year, FSBPT supplemented existing processes of identifying scores that need additional review before being released with more intensive, sophisticated analyses. Communication Providing accurate and timely information on the NPTE is an essential service of FSBPT. A summary of some of our major communication activities follows. Workshops for Educators The Federation of State Boards of Physical Therapy conducted two Faculty Workshops in 2014, both onsite in Alexandria, VA. The workshops included instruction and practice in writing high quality multiplechoice questions, information and opportunities for discussion on how the NPTE is developed, as well as discussing strategies for increasing pass rates. The workshops were well attended and positively received. Presentations FSBPT made several presentations to external groups on initiatives and efforts relevant to the NPTE. The purpose of these presentations is to get peer feedback on new initiatives before they are implemented in the NPTE program. The presentations are as follows Society for Industrial and Organizational Psychology on how to properly utilize and motivate subject matter expert volunteers Association of Test Publishers on protecting Personally Identifying Information and on defending cut scores National Council on Measurement in Education on (a) identifying possible security issues by examining person fit indices. Item Writer Newsletters Newsletters with helpful item writing tips and updates on the NPTE are distributed to all volunteers who have participated in item writing during the most recent two years, as well as to all examination committee members and the FSBPT Board of Directors.

10 Faculty Newsletters Quarterly newsletters with the latest information and updates on the NPTE are distributed to all program directors and any faculty members who request to be included on the mailing list.