A 2-Year Study of Gram Stain Competency Assessment in 40 Clinical Laboratories

Transcription

1 Microbiology and Infectious Disease / GRAM STAIN COMPETENCY ASSESSMENT A 2-Year Study of Gram Stain Competency Assessment in 40 Clinical Laboratories Nancy Goodyear, PhD, 1 Sara Kim, PhD, 2,3 Mary Reeves, 1 and Michael L. Astion, MD, PhD 1 Key Words: Gram stain; Competency assessment; Computer-assisted instruction DOI: /40WD3015CH1RYH58 Abstract We used a computer-based competency assessment tool for Gram stain interpretation to assess the performance of 278 laboratory staff from 40 laboratories on 40 multiple-choice questions. We report test reliability, mean scores, median, item difficulty, discrimination, and analysis of the highest- and lowestscoring questions. The questions were reliable (KR-20 coefficient, 0.80). Overall mean score was 88% (range, 63%-98%). When categorized by cell type, the means were host cells, 93%; other cells (eg, yeast), 92%; gram-positive, 90%; and gram-negative, 88%. When categorized by type of interpretation, the means were other (eg, underdecolorization), 92%; identify by structure (eg, bacterial morphologic features), 91%; and identify by name (eg, genus and species), 87%. Of the 6 highest-scoring questions (mean scores, 99%) 5 were identify by structure and 1 was identify by name. Of the 6 lowest-scoring questions (mean scores, <75%) 5 were gram-negative and 1 was host cells. By type of interpretation, 2 were identify by structure and 4 were identify by name. Computer-based Gram stain competency assessment examinations are reliable. Our analysis helps laboratories identify areas for continuing education in Gram stain interpretation and will direct future revisions of the tests. Competency assessment (CA) is an integral component of laboratory operations. CA results in improved laboratory performance through identification of targets for improvement of processes or training. CA also allows laboratories to provide documentation of laboratory quality for staff, management, surveyors, and customers. In addition, CA is mandated by the Clinical Laboratory Improvement Amendments (CLIA) regulations and included in the College of American Pathologists and Joint Commission on Accreditation of Healthcare Organizations requirements. 1-3 CLIA regulations specify that laboratories must assess competency by different methods, including direct observation, review of test and quality control results, instrument records, and written testing. 1 Gram stain CA is of particular importance because Gram stains often are performed in urgent care settings where patient care and treatment decisions are made based on the results. Gram stain misinterpretation has been reported widely. 4-8 Yuan et al 9 recently demonstrated that Gram stain misinterpretation can have significant clinical impact. Gram stains other than urethral or endocervical are classified as high complexity under CLIA. 10 The Department of Laboratory Medicine, University of Washington, Seattle, has been developing laboratory tutorials and computerbased CA tools for the last 10 years Computer-based CA is an objective method that functions effectively as a supplement to other methods of CA. This work is a logical extension of our previous work. In this study, we detail the performance of 278 laboratory staff from 40 facilities on the Web-based Gram Stain CA examination. We identify and discuss the CA questions that were most and least problematic for the laboratory staff studied. Overall, the study demonstrates successful wide-scale implementation of 28 Am J Clin Pathol 2006;125: DOI: /40WD3015CH1RYH58

2 Microbiology and Infectious Disease / ORIGINAL ARTICLE computer-based CA and points out key areas that US laboratory staff found to be problematic. Materials and Methods Description of the CA System The University of Washington Department of Laboratory Medicine and Medical Training Solutions ( Seattle, WA) have collaborated to produce an online CA system comprising more than 30 different CA topics to date. The minimum system requirements are Microsoft Internet Explorer, version 6.0 (Microsoft, Redmond, WA); Windows Media Player 6.4 (Microsoft); Microsoft Windows 98 (Microsoft); a Pentium-class processor (Intel, Santa Clara, CA); and a 16-bit color display with resolution of A sound card and speakers are recommended but not required, and a printer is necessary if paper copies of examination results are desired. 15 Administrative features of the system include the ability to assign examinations to individual employees and to send automated reminders, including direct links to the examinations, to employees. Examinations are scored automatically, and individual or laboratory results are available immediately for review or printing. Each examination consists of 10 multiple-choice questions, written and reviewed by content experts and educators in the Department of Laboratory Medicine. Each question is accompanied by an image, which is integral to answering the question. Image 1 shows an example of a question from the Gram Stain examination. After answering the question, the examination taker is shown the correct answer, with an explanation of the question. There are 2 examination periods for each year (January 1 June 30 and July 1 December 31). New examination questions are written for each testing period. Subjects Subjects included laboratory staff from facilities in the United States who completed 4 consecutive Gram stain examinations administered in 2003 and We applied 2 criteria for including test scores in the analysis: (1) Only users who took all 4 tests were included. (2) Data from clinical laboratories were included in the study. We obtained approval from the University of Washington Institutional Review Board to conduct the study. Test Items We combined 10 items from each of the 4 Gram stain tests and based our analysis on a total of 40 items. We analyzed the items using 2 different categorizations. The first set of categories were based on cell type and included the following: gram-positive, 12 questions; gram-negative, 15 questions; host cells, including WBCs, RBCs, squamous epithelial cells, and columnar epithelial cells, 11 questions; and other cells, including yeast and no organisms seen, 10 questions. Some questions fit more than 1 category, eg, a question may have asked about WBCs and gram-positive cocci in an image. The second categorization was based on the type of interpretation required and included the following: identify by structure, in which the test taker was asked to identify cell types, Gram reaction, or bacterial morphologic features, 22 questions; identify by name, in which the test taker was asked to correlate the image to organism identification by genus or genus and species, 15 questions; and other, which included Image 1 Screen capture from the Gram Stain Competency examination showing a typical explanation screen for an imagebased multiple-choice question. Am J Clin Pathol 2006;125: DOI: /40WD3015CH1RYH58 29

3 Goodyear et al / GRAM STAIN COMPETENCY ASSESSMENT questions about overdecolorization or underdecolorization, 3 questions. The identify by structure category requires only 1 cognitive step, that of identifying the structures in the image. Questions in the identify by name category require an additional step, that of matching the structure(s) identified in the image with organism names. This requires consideration of clinical significance, knowledge of the Gram reaction and morphologic features for many organisms, and distinguishing subtle differences in morphologic features. Each question was classified into only 1 of the 3 categories. Data Analysis Scores recorded on a server were transferred to Microsoft Excel (Microsoft) and SPSS version (SPSS, Chicago, IL) for analysis. We conducted the following data analyses: KR-20 Reliability We calculated the reliability of the test to determine its internal consistency based on dichotomous test scores (1 for correct; 0 for incorrect). Descriptive Statistics We calculated the mean and median scores for the overall test and for the cell and interpretation types covered in the test. Item Difficulty and Discrimination To assess the quality of individual items and the test as a whole, we performed item difficulty and item discrimination analyses. For each test item, we calculated the percentage of users who answered the items correctly to determine the level of difficulty of the items. We used the data to identify easy items (defined as >95% of the users answering correctly) or difficult (<75% of the users answering correctly). Then we derived an item discrimination based on correlations between a user s total score on the overall test and that user s score on an item. The discrimination index is used to determine whether an item appropriately differentiates the ability of high- and low-performing users. We applied the following standard to judge the discriminatory quality of the items: good, more than 0.30; fair, 0.10 to 0.30; and poor, less than An ideal question would include a correct answer with a strong positive correlation and distracter items with negative correlations. In other words, users who attained a high score on the overall test would tend to choose the correct answer and, therefore, not select distracter items. Subset Analysis We present 6 questions that were answered correctly by 99% or more of the users, and we highlight in detail 6 items that were answered correctly by fewer than 75% of the users to illustrate why these items were challenging to them. Results and Discussion Scores of 278 users from 40 laboratories were analyzed. The number of users per laboratory ranged between 1 and 25. The average overall score on the combined 40-question examination was 88% (range, 63%-98%). Reliability We obtained a KR-20 coefficient of 0.80 (out of 1.00), which suggests that test items are dependable for assessing a user s mastery of Gram stain content. Descriptive Statistics We calculated the mean and median scores and ranges based on users scores on 40 test items. Table 1 and Table 2 present the percentage of correct answers selected by 278 users, based on categorization by cell type and type of interpretation, respectively. Overall, users correctly identified approximately 90% of the items. When questions are categorized by cell type (Table 1), users attained the highest scores in host cells (93%) followed by other cells (92%), gram-positive cells (90%), and gram-negative cells (88%). When categorized by type of interpretation (Table 2), users attained the highest scores in the other category (92%), followed by identify by structure (91%), and identify by name (87%). Item Difficulty and Discrimination Of 40 items, 6 were answered correctly by 99% or more of users, suggesting that these items were easy. Examination of the item discrimination index for these items revealed the level of discrimination was poor (<0.10). When categorized by cell type, these questions were spread across all 4 categories: host cells, 3; gram-negative, 3; other cells, 2; and gram-positive, 1 (note that 3 questions were classified into more than 1 category). When categorized by type of interpretation, only 1 question was in the identify by name category, 5 were in the identify by structure category, and none were in the other category. There were 6 items that were answered correctly by fewer than 75% of the users, suggesting that these items were difficult. All of these items showed fair to good discrimination levels (>0.10). Five items were in the gram-negative category (range, 67%-74%), and 1 was in the host cells category. When classified by type of interpretation, 2 of the difficult items were classified as identify by structure and 4 as identify by name. We found item difficulty and discrimination to be useful tools for assessing the quality of the examinations. Based on these study results, items with poor discrimination ( 0.10) will be reviewed when new questions are written. Although all questions are retired at the end of each examination period, some new questions will be similar in style and content to 30 Am J Clin Pathol 2006;125: DOI: /40WD3015CH1RYH58

4 Microbiology and Infectious Disease / ORIGINAL ARTICLE Table 1 Ability of 278 Users to Identify Cell Types in a 40-Question Examination * Cell Type Mean (%) Median (%) Minimum Score (%) Maximum Score (%) Host cells (n = 11) Gram-positive (n = 12) Gram-negative (n = 15) Other (n = 10) * The 40 questions sort into 48 items when categorized by cell type because some questions fit more than 1 category. The lowest individual score for the 278 users. The highest individual score for the 278 users. Table 2 Ability of 278 Users From 40 Laboratories to Perform Different Types of Gram Stain Interpretations * Type of Interpretation Mean (%) Median (%) Minimum Score (%) Maximum Score (%) Identify by name (n = 15) Identify by structure (n = 22) Other (n = 3) * Each of the 40 questions corresponded to 1 type of interpretation. The lowest individual score for the 278 users. The highest individual score for the 278 users. previous questions. Items with poor discrimination will be avoided or revised to improve discrimination. Subset Analysis Of the 6 questions with mean scores of 99% or more, the only one that fell in the identify by name category asked the test taker to identify Neisseria gonorrhoeae from a classic image with many WBCs, one of which is packed with intracellular gram-negative diplococci (Image 1). Four questions involved recognizing WBCs, epithelial cells, budding yeast, gram-negative rods, and gram-positive cocci. One question involved differentiating background material from true bacteria ( no organisms seen ). The 6 questions with mean scores of 75% or less are shown in Table 3, with the percentage of users who selected each possible answer and the item discrimination indices for each item. The discrimination indices for all correct answers ranged between and and for all distracter items, between and Therefore, these 6 questions are judged to be of good quality even if they were difficult for the users. All 6 questions, accompanied by images, are available at Questions 3, 10, and 22 involved variations in the morphologic features of gram-negative rods or diplococci. For question 3, 18% of users selected Clostridium species instead of the correct answer, Klebsiella species. Although at first glance this seems an unlikely error, Clostridium species are well known to overdecolorize easily, and the specimen source was a biliary abscess, which is consistent with an anaerobic infection. Question 10 involved a different sort of distinction. The correct answer was Moraxella catarrhalis, a gram-negative diplococcus, but 13% of users answered Haemophilus influenzae, a tiny gram-negative coccobacillus. An additional 10% answered no infection present, which may reflect the infrequency of M catarrhalis as a pathogen in their institution. Only 1 WBC was present in the image, which may have led them to believe the organism was most likely an upper respiratory tract contaminant such as a saprophytic Neisseria species. The subtle morphologic distinction in question 22 involved Klebsiella pneumoniae appearing as short, wide rods, several of which exhibit bipolar staining. The bipolar staining rods were thought to be Neisseria species by 11% of users and Pseudomonas aeruginosa, usually longer and thinner, by 15% of users. The difficulty in correctly answering questions 6 and 9 seems to stem from confusion about the appearance of Fusobacterium nucleatum and the term fusiform. This is an uncommon isolate, although it is quite distinctive in appearance. In question 6, 30% of users answered F nucleatum instead of the correct answer, Serratia marcescens. The rods in this image are much shorter and wider than true Fusobacterium species. It is possible that users were misled by background material. In question 9, 16% of users described the appearance of the gram-negative rods as long and narrow, and 13% described them as curved. Neither answer is completely false; however, fusiform also incorporates the needlelike appearance of the rods. Question 24 asked users to distinguish columnar epithelial cells from squamous epithelial cells. One other question in the sample set required the same differentiation (question not Am J Clin Pathol 2006;125: DOI: /40WD3015CH1RYH58 31

5 Goodyear et al / GRAM STAIN COMPETENCY ASSESSMENT Table 3 Questions Correctly Answered by Fewer Than 75% of Users Image * Description Question and Multiple Choice Options Users Selecting Option (%) Item Discrimination WBCs; few GNRs WBCs; few GNRs; moderate pink background Long, thin GNRs, some curved; 2 WBCs 3. This directly stained biliary abscess fluid specimen suggests an infection caused by which of the following? a. Clostridium spp b. Haemophilus spp c. Neisseria spp d. Klebsiella spp This direct Gram stain is from an infected surgical wound. Which of the following is the most likely identification? a. Coagulase-negative staphylococci b. Fusobacterium nucleatum c. Serratia marcescens d. Legionella spp This is a direct Gram stain of a peritonsillar abscess. Which of the following best describes the organism seen? a. Gram-negative rods, fusiform b. Gram-negative rods, curved c. Gram-negative rods, long, narrow d. Gram-negative rods, coccobacilli WBC; many GNDC; streaky 10. This direct Gram stain of sputum suggests an infection caused by: background a. Haemophilus influenzae b. Klebsiella pneumoniae c. Moraxella catarrhalis d. No infection present WBCs; GNRs, short, some 22. This direct Gram stain from a urine specimen suggests infection with: bipolar staining a. Haemophilus influenzae b. Klebsiella pneumoniae c. Neisseria spp d. Pseudomonas aeruginosa WBCs; columnar epithelial 24. Evaluate this direct Gram stain from a bronchial wash specimen. cells; no organisms a. WBCs, columnar epithelial cells b. WBCs, epithelial cells, GPC c. WBCs, gram-positive cocci, GNRs d. WBCs, squamous epithelial cells GNDC, gram-negative diplococci; GNR, gram-negative rod; GPC, gram-positive cocci; spp, species; WBC, white blood cell. * See for questions and images as they appear in the Gram stain competency test. Correct answer in bold. shown); the mean score for this question was 91%. In this question, the columnar epithelial cells were more classic in appearance than in question 24, very oblong, and included visible villi. In question 24, the cells are more rounded; however, they can be distinguished from squamous epithelial cells by their size relative to several WBCs in the field and in the large size of their nuclei relative to the cytoplasmic volume. It also is possible that many subscribing laboratories do not distinguish between columnar and squamous epithelial cells in reporting routine Gram stains; therefore, their employees lack experience in this area. Conclusion We successfully implemented a computer-based CA tool in 40 laboratories. In this study, we looked at the performance of 278 laboratory staff who took Gram stain competency examinations in 4 examination periods during 2 years. Our data show that the items on the Gram stain CA examination are reliable overall. Questions with the poorest responses involved distinguishing columnar and squamous epithelial cells, identifying fusiform organisms, and distinguishing a variety of gram-negative morphologic features. Our program allows laboratories to determine their own benchmarks or pass rates for each examination to accommodate differences in practice in different laboratories; however, laboratory managers may consider interventions in the areas identified as difficult. The ability to show only 1 microscopic field is a limitation of this test system. Our ongoing analysis of questions results in high-quality CA tools. Our future plans include examining other CA test topics (eg, safety), performing a more extensive analysis involving a larger pool of test questions and subjects, and continuing quality improvement of the CA tests based on these results. From the Departments of 1 Laboratory Medicine, 2 Medical Education and Biomedical Informatics, and 3 Family Medicine, University of Washington, Seattle. The University of Washington receives licensing revenue, in relation to the competency assessment Web site, from Medical Training Solutions. 32 Am J Clin Pathol 2006;125: DOI: /40WD3015CH1RYH58

6 Microbiology and Infectious Disease / ORIGINAL ARTICLE Address reprint requests to Dr Goodyear: Dept of Laboratory Medicine, Box , University of Washington, Seattle, WA Acknowledgments: We thank the many faculty and staff from the Department of Laboratory Medicine who write and review the examination questions for the Web site and the staff at Medical Training Solutions for development and distribution. References 1. Medicare, Medicaid, and CLIA programs: regulations implementing the Clinical Laboratory Improvement Amendments of 1988 (CLIA). 57 Federal Register 7002 (1992). 2. College of American Pathologists. Commission on Laboratory Accreditation Laboratory General Checklist. Northfield, IL: College of American Pathologists; 2004: Joint Commission on Accreditation of Healthcare Organizations. Hospital Accreditation Standards. Oakbrook Terrace, IL: Joint Commission on Accreditation of Healthcare Organizations; 2005:HR-13-HR Harrington BJ, Plenzler M. Misleading Gram stain findings on a smear from a cerebrospinal fluid specimen. Lab Med. 2004;35: Parry CM, White RR, Ridgeway ER, et al. The reproducibility of sputum Gram film interpretation. J Infect. 2000;41: Bloch MJ, Cembrowski GS, Lembesis GJ. Longitudinal study of error prevalence in Pennsylvania physicians office laboratories. JAMA. 1988;260: Al Balooshi N, Jamsheer A, Botta GA. Impact of introducing quality control/quality assurance (QC/QA) guidelines in respiratory specimen processing. Clin Microbiol Infect. 2003;9: Zarakolu P, Sahin Hodoglugil NN, Aydin F, et al. Reliability of interpretation of gram-stained vaginal smears by Nugent s scoring system for diagnosis of bacterial vaginosis. Diagn Microbiol Infect Dis. 2004;48: Yuan S, Astion ML, Schapiro J, et al. Clinical impact associated with corrected results in clinical microbiology testing. J Clin Microbiol. 2005:43: US Food and Drug Administration, Center for Devices and Radiological Health. On-Line Searchable CLIA Database. Available at: Accessed April Phillips C, Henderson PJ, Mandel L, et al. Teaching the microscopic examination of urine sediment to second year medical students using the Urinalysis-Tutor computer program. Clin Chem. 1998;44: Astion ML, Kim S, Nelson A, et al. A two year study of microscopic urinalysis competency using the Urinalysis- Review computer program. Clin Chem. 1999;45: Kim S, Henderson PJ, Phillips C, et al. A Web-based competency assessment system for microscopic urinalysis. Clin Chem. 2002;48: Astion ML, Kim S, Terrazas E, et al. Characteristics of educational software use in 106 clinical laboratories. Am J Clin Pathol. 2002;118: Medical Training Solutions. On-line System Requirements. Available at: SystemRequirements.aspx. Accessed April Office of Educational Assessment, University of Washington. Available at: Accessed April Am J Clin Pathol 2006;125: DOI: /40WD3015CH1RYH58 33