Do Admissions Data and NBDE Part I Scores Predict Clinical Performance Among Dental Students?

Transcription

1 Do Admissions Data and NBDE Part I Scores Predict Clinical Performance Among Dental Students? Sang E. Park, D.D.S., M.M.Sc.; Srinivas M. Susarla, B.A.; Ward Massey, B.D.S., Ph.D. Abstract: The purpose of this study was to evaluate possible associations between a variety of measures used to evaluate didactic knowledge and clinical performance within a predoctoral dental program. In this study, clinical performance was assessed by clinical productivity and clinical proficiency across four different competency areas: operative dentistry, major restorative dentistry, fixed prosthodontics, and removable prosthodontics. Predental and preclinical predictors were undergraduate GPAs (overall and science), DAT subtest scores (including the Perceptual Ability Test, PAT), and performance on subtests of Part I of the National Board Dental Examination. The sample consisted of eighty-four students at the Harvard School of Dental Medicine who graduated during the period Associations between predictors and outcomes were first evaluated individually. Any associations that were near statistically significant (p 0.15) were then included in a multiple linear regression model. The criterion for statistical significance in the multiple linear regression model was p While a number of measures were associated in bivariate analyses, few predictors were statistically significantly associated with clinical outcomes in the multiple regression analyses. Those predictors that were associated with clinical outcomes were also not consistently associated with the different outcomes studied. These data indicate that, within this study population, there is little to no uniform association between preclinical didactic performance and measurements of clinical productivity and clinical proficiency. It is possible that the overlap in skill sets required for success in the predental/preclinical and clinical areas is minimal. Dr. Park is Senior Tutor, Office of Dental Education, Harvard School of Dental Medicine; Mr. Susarla is a D.M.D. Candidate, Harvard School of Dental Medicine; and Dr. Massey is Professor in the Department of Endodontics, Prosthodontics, and Operative Dentistry, Baltimore College of Dental Surgery, University of Maryland. Direct correspondence and requests for reprints to Dr. Ward Massey, Baltimore College of Dental Surgery, University of Maryland, 666 West Baltimore Street, Baltimore, MD 21201; phone; fax; wlm001@dental.umaryland.edu. Key words: DAT, PAT, NBDE Part I, clinical outcomes Submitted for publication 11/30/05; accepted 1/27/06 All dental schools in the United States utilize the applicant s general undergraduate grade point average (GPA), the undergraduate science grade point average (SGPA), and the Dental Admission Test (DAT) scores as factors for admissions. 1 The DAT is a necessary tool to compare students from different schools as it is the only national standardized test available for dental school admissions. The DAT consists of a Quantitative Reasoning Test, a Reading Comprehension Test, a Survey of the Natural Sciences, and a Perceptual Ability Test. Upon matriculation, the only other national standardized tests are the National Board Dental Examination (NBDE) Parts I and II, which are required for graduation and for a license to practice dentistry in the United States. In addition, a large proportion of postgraduate programs use the NBDE Part I score to evaluate a candidate s fitness for placement in residency programs and use this score as a proxy measure of overall ability. To optimize the admissions and graduation process, it is critically important that dental schools be able to predict the future academic performance and clinical competency of students in the dental curriculum. The dental literature is replete with the effects of different variables on predicting student performance at dental school. Many studies have assessed the relationship between predictor variables and student performance in preclinical or restorative laboratory courses, but there have been few studies emphasizing clinical performance overall. Predental and preclinical predictors have largely shown insignificant correlation with preclinical laboratory course performance. 2-9 Since the introduction of the Perceptual Ability Test (PAT), a number of studies have been conducted to determine the effectiveness of the PAT in predicting student preclinical and clinical performance. Studies that examined the predictive value of the DAT subtest scores (including the PAT score) on clinical courses have shown inconclusive or conflicting results Little information is available in the literature regarding the relationship of predental and preclinical predictors to productivity and success in the clinical environment of a dental school. 518 Journal of Dental Education Volume 70, Number 5

2 While preclinical predictors typically evaluate the fund of biomedical and basic science knowledge, it is not unreasonable to suggest that success in evaluating and solving scientific and biomedical problems has direct correlations to addressing, evaluating, and solving problems in a clinical setting. Although some of the skills required to effectively treat patients in a clinical setting may not have direct associations with more academic measures, there is likely some overlap between the skill sets required for academic success and those for clinical success. Students who succeed academically are typically highly motivated, organized, creative problem-solvers. These same attributes can be plausibly linked to those who perform well in clinical settings. The purpose of this study was to assess the validity of the undergraduate academic records (GPA, science GPA, DAT, and PAT) and the National Board Dental Examination (NBDE) Part I in predicting clinical performance of student cohorts tracked over a two-year period in a comprehensive care clinic. In this study, we evaluated the hypothesis that predental (undergraduate GPAs, DAT scores) and preclinical (NBDE Part I scores) would have significant predictive ability with regard to clinical performance, as measured by number of procedures within a specific discipline and competency scores. Materials and Methods This study was a retrospective cohort study at the Harvard School of Dental Medicine (HSDM). All data regarding predental performance (overall GPA, science GPA, and DAT scores), preclinical performance (NBDE Part I), and clinical performance (clinical evaluations in specific disciplines) were obtained from the office of the registrar. The registrar assigned random identification numbers to each student enrolled in the study, such that no identifying information (name, social security number, etc.) was given to the study authors. The project received appropriate approval from the HSDM/HMS Institutional Review Board. The study variables were broadly categorized into predictors and outcomes. Study variables that were classified as predictors were further classified as predental or preclinical. Predental predictors were overall GPA, science GPA, and performance on all the subtests of the Dental Admission Test. The preclinical predictor was performance on all subtests of the National Board Dental Examination Part I. Outcome variables consisted of various measures of students performance and productivity during the clinical years. Clinical performance and productivity were evaluated in four disciplines: operative, major restorative, and removable and fixed prosthodontics. The students performance on all operative treatments involving direct restorative materials (e.g., composite/amalgam restorations) were included as outcome measures. Major restorative procedures that were evaluated included core/ foundation restorations in direct restorative materials, all cast cores, onlay and inlay restorations, and indirect veneers in porcelain or resin. Fixed prosthodontic procedures included single crowns, fixed partial dentures, endosseous implants, and implant crowns. Removable prosthodontic procedures included complete and partial prostheses, overdentures, and implant-retained complete prostheses. Clinical productivity was evaluated by examining the number of procedures completed by each student within a given category. Clinical performance was evaluated using the HSDM Clinical Evaluation System. In this system, all procedures are evaluated with an overall score of 1 (best) to 5 (least). Scores are defined as follows: 1=student performs beyond the expectations for a predoctoral student; 2=student performs within the expectations for a fourth-year student who is about to graduate; 3=student performs within the expectations for a student at the end of the third year; 4=student performs within the expectations for a student at the beginning of the third year; and 5=student is unable to complete the task without instructor assistance. Over the course of the study, data were entered into a statistical database (SPSS v11.0, SPSS Inc., Chicago, IL). Descriptive statistics were computed for all study variables. Bivariate analyses were computed to evaluate associations between the predictors and outcomes. Any associations with p 0.15 in bivariate analyses were included in multiple linear regression models. 12 In the multiple linear regression models, p-values 0.05 were considered statistically significant. Goodness of fit for the models was evaluated using the R-squared value for each model. Results The sample included eighty-four students who were enrolled at the Harvard School of Dental Medicine during the period , corresponding to the graduating classes of 2002 through Dur- May 2006 Journal of Dental Education 519

3 ing this time period, ninety-nine students graduated from HSDM. Fifteen students (15.2 percent) were excluded from the sample because they had incomplete data for one or more predictors/outcomes. Descriptive statistics for the study population are summarized in Table 1. The mean total and science GPAs for the sample were 3.7±0.22 (range: 3.2 to 4.1) and 3.6±0.24 (range: 3.0 to 4.1), respectively. The mean DAT subtest scores were 19.5±2.4 (range: 13.0 to 27.0) for the Perceptual Ability Test, 21.1±2.7 (range: 16.0 to 28.0) for quantitative reasoning, 21.7±2.4 (range: 16.0 to 27.0) for reading comprehension, 20.8±2.2 (range: 16.0 to 28.0) for biology, 22.7±2.5 (range: 17.0 to 28.0) for general chemistry, and 22.8±2.6 (range: 17.0 to 27.0) for organic chemistry. The mean NBDE Part I scores for anatomical sciences, biochemistry and physiology, microbiology and pathology, and dental anatomy and occlusion were 92.5±3.9 (range 81.0 to 99.0), 93.8±3.4 (range: 82.0 to 99.0), 94.6±3.6 (range: 85.0 to 99.0), and 95.4±3.9 (range: 83.0 to 99.0), respectively. On average, students performed 56.0±17.9 (range: 20.0 to 120.0) operative procedures during their clinical education at HSDM. Major restorative procedures were the second most frequently performed, followed by removable and fixed prosthodontic procedures, which were roughly even. The mean clinical performance scores were clustered near 2.0, indicative of the level of competency expected for a student at the end of his or her D.M.D. education. Table 2 summarizes the bivariate associations between the predictor variables and clinical productivity. The number of operative procedures was associated with total GPA, DAT biology, DAT general chemistry, and NBDE dental anatomy and occlusion. The number of major restorative procedures was associated with total GPA and DAT biology. The number of removable prosthodontic procedures was associated with DAT biology and DAT general chemistry. The number of fixed prosthodontic procedures was associated with DAT biology, NBDE anatomical sciences, and NBDE biochemistry and physiology. Table 1. Descriptive statistics for study variables N Minimum Maximum Mean SD Predental Predictors Total GPA Science GPA DAT Scores PAT Score Quantitative Reasoning Reading Comprehension Biology General Chemistry Organic Chemistry Preclinical Predictors NBDE Part I Scores Anatomical Sciences Biochemistry/Physiology Microbiology/Pathology Dental Anatomy/Occlusion Clinical Productivity Number of Operative Procedures Number of Major Restorative Procedures Number of Removable Procedures Number of Fixed Procedures Clinical Performance Average Operative Grade * Average Major Restorative Grade Average Removable Prosth Grade Average Fixed Prosth Grade *Note that the clinical evaluation system used allows for five levels of proficiency (1-5) with a proficiency grade of 1 being the highest and 5 the lowest. 520 Journal of Dental Education Volume 70, Number 5

4 Table 2. Bivariate correlations between predictors and clinical productivity Operative Major Restorative Removable Fixed Predental Predictors Total GPA Science GPA DAT Scores PAT Score Quantitative Reasoning Reading Comprehension Biology General Chemistry Organic Chemistry Preclinical Predictors NBDE Part I Scores Anatomical Sciences Biochemistry/Physiology Microbiology/Pathology Dental Anatomy/Occlusion Note: Associations in bold type met the criterion for inclusion in the multiple regression model. The multiple linear regression models for clinical productivity are listed in Table 3. After adjusting for the effects of multiple covariates on clinical productivity for operative procedures and fixed prosthodontic procedures, there were no statistically significant associations. Both the total GPA and DAT biology scores were statistically significantly associated with the number of major operative procedures. DAT general chemistry was associated with the number of removable prosthodontic procedures. Bivariate associations between predictor variables and clinical performance are summarized in Table 4. Mean performance on operative procedures was associated with science GPA, PAT score, and DAT biology. Performance on major restorative procedures was associated with total GPA and NBDE dental anatomy and occlusion. Clinical performance on removable prosthodontic procedures was associated with PAT score, NBDE biochemistry and physiology, and NBDE dental anatomy and occlusion. None of the predictor variables met the criterion for association in bivariate analyses with performance on fixed prosthodontic procedures. The multiple linear regression models corresponding to clinical performance outcomes are summarized in Table 5. Performance on operative procedures was associated with science GPA and DAT biology. After adjusting for the effects of multiple predictor covariates, there were no statistically significant predictors of clinical performance for major restorative or removable prosthodontic procedures. Discussion The purpose of this study was to evaluate the ability of predental and preclinical benchmarks to predict clinical performance in a cohort of students at the Harvard School of Dental Medicine. The primary study hypothesis was that predental (undergraduate GPAs, DAT scores) and preclinical (NBDE Part I scores) would have significant predictive ability with regard to clinical performance, as measured by number of procedures within a specific discipline and competency scores. To evaluate this study hypothesis, academic records were examined in a retrospective manner for a cohort of students at HSDM during the period The result of this study revealed a lack of any statistically significant associations between the predictors and the clinical outcomes for productivity and proficiency. In addition, some intuitively plausible associations were not found. The PAT subtest, a subject of some debatable utility in dental education, was not associated with clinical productivity or proficiency in the multiple linear regression models. Similarly, the dental anatomy and occlusion subtest score was not associated with any of the clinical outcomes after controlling for the effects of other covariates. A number of possibilities could explain this lack of association. First, it is possible that the sample size evaluated here may be insufficient to test the complex effects of multiple covariates on multiple May 2006 Journal of Dental Education 521

5 clinical outcomes. The relatively small variance in outcome measures seen in this cohort increases the difficulty of evaluating the incremental effects of the predictors on the outcome in the setting of a small sample size. Also, the sample used here is somewhat homogeneous and highly selected: all students evaluated were students at HSDM and had similar academic backgrounds (thus, the distribution of score values in this population may not be representative of all dental students). Another possibility is that there is little to no association between these predictors and clinical Table 3. Multiple regression models for predictors and clinical productivity Operative Major Restorative Removable Fixed B p-value B p-value B p-value B p-value Constant < Total GPA N/A N/A N/A N/A Science GPA N/A N/A N/A N/A N/A N/A N/A N/A Biology General Chemistry N/A N/A N/A N/A Anatomical Sciences N/A N/A N/A N/A N/A N/A Biochemistry/Physiology N/A N/A N/A N/A N/A N/A Dental Anatomy/Occlusion N/A N/A N/A N/A N/A N/A Table 4. Bivariate correlations between predictors and clinical performance Operative Major Restorative Removable Fixed Predental Predictors Total GPA Science GPA DAT Scores PAT Score < Quantitative Reasoning Reading Comprehension Biology < General Chemistry Organic Chemistry Preclinical Predictors NBDE Part I Scores Anatomical Sciences Biochemistry/Physiology Microbiology/Pathology Dental Anatomy/Occlusion Associations in bold type met the criterion for inclusion in the multiple regression model. Table 5. Multiple regression models for predictors and clinical performance Operative Major Restorative Removable B p-value B p-value B p-value Constant 3.3 < < <0.01 Total GPA N/A N/A N/A N/A Science GPA <0.01 N/A N/A N/A N/A PAT Score N/A N/A Biology N/A N/A N/A N/A Biochemistry/Physiology N/A N/A N/A N/A Dental Anatomy/Occlusion N/A N/A Journal of Dental Education Volume 70, Number 5

6 outcomes precisely because the clinical outcomes are dependent upon skill sets that may vary significantly from the skill sets required to perform well on standardized examinations and didactic work. One can consider different possibilities for student skill set pairings (Figure 1). Some students may have significant success with didactic work but lack the psychomotor skills to perform at an equivalent level clinically. Alternatively, there may be some students for whom didactic learning is a significant challenge, but who have excellent hand skills. Associations between the predictors and clinical outcomes in these students may be inconsistent, as students may be able to successfully compensate based on their stronger skill area (that is, a student with excellent hand skills may be able to compensate for a lack of didactic knowledge and vice versa). While certain knowledge gaps may be evident between students at different times during the course of their dental education, these gaps will eventually be filled with knowledge gained from clinical experience. In other words, all students reach the same point of clinical competence prior to graduation. What may differ between students is the rate at which competency is achieved, which was not evaluated in this study, but is a potential direction for future research. With regard to clinical productivity, it may be that once clinical thresholds for a given procedure are achieved, students feel little to no incentive to seek out additional procedures and may end up transferring patients to other students. Finally, it may be that clinical evaluations cannot be effectively compared to standardized examinations due to the subjective nature of faculty grading for clinical procedures and variability between clinical faculty in terms of the amount of assistance given to students. In this regard, a significant shortcoming is the clustering of faculty evaluations within a narrow range of positively skewed scores, which diminishes our ability to detect differences between subjects, assuming that such differences exist. The combination of a relatively narrow distribution of predictor values (GPAs, DAT, and NBDE Part I scores) and narrow clinical evaluation scores most certainly skews the statistical analysis and, in the setting of a relatively small sample size, decreases the power of the study. It remains to be determined whether or not preclinical predictors dictate clinical performance in den- Psychomotor Skills High Low High A B Didactic Skills Low C D A student in Category A would be expected to perform exceptionally on both didactic and clinical examinations. A student in category D would be expected to perform marginally on both clinical and didactic evaluations. Students in categories B and C will have inconsistent performance correlations, as additional didactic knowledge may compensate for a lack of technical ability and vice versa. Figure 1. Possible pairings for skill sets among dental students May 2006 Journal of Dental Education 523

7 tal school. Future research should be aimed at examining these associations in larger cohorts and across different populations of dental students. In addition, evaluations over finite time periods to evaluate the rate at which clinical proficiency is achieved may give some insight into the more subtle effects that these predictors have on clinical outcomes. Acknowledgments The authors would like to acknowledge the invaluable assistance of Melissa Dunphy-Gooley and Carole Chase. REFERENCES 1. American Dental Association. 2002/03 survey of predoctoral dental education. Volume 2: tuition, admission, and attrition. Chicago: American Dental Association, Gray S, Deem L. Predicting student performance in preclinical technique courses using the theory of ability determinants of skilled performance. J Dent Educ 2002; 66: Kramer GA, Kubiak AT, Smith RM. Construct and predictive validities of the Perceptual Ability Test. J Dent Educ 1989;53: Coy K, McDougall H, Sneed M. Issues regarding practical validity and gender bias of the Perceptual Ability Test (PAT). J Dent Educ 2003;67: Dworkin SF. Dental Aptitude Test as performance predictor over four years of dental school: analyses and interpretations. J Dent Educ 1970;34: Boyle AM, Santelli JC. Assessing psychomotor skills: the role of the Crawford small parts dexterity test as a screening instrument. J Dent Educ 1986;50: Manhold JH, Manhold BS. Final report of an eight-year study of the efficacy of the dental aptitude test in predicting four year performance in a new school. J Dent Educ 1965;29: Boyd MA, Wood WW, Conry RF. Prediction of preclinical operative dentistry performance in two instructional methods. J Dent Educ 1980;44: Phipps GT, Fishman R, Scott RH. Prediction of success in a dental school. J Dent Educ 1968;32: Gray SA, Deem LP, Straja SR. Are traditional cognitive tests useful in predicting clinical success? J Dent Educ 2002;66: Kramer GA. Predictive validity of the Dental Admission Test. J Dent Educ 1986;50(9): Susarla SM, Dodson TB. Risk factors for third molar extraction difficulty. J Oral Maxillofac Surg 2004;62 (11): Journal of Dental Education Volume 70, Number 5