One of the aims of postgraduate/postdoctoral

Advanced Dental Education The Effect of Awareness of American Board of Orthodontics Criteria on Treatment Outcomes in a Postgraduate Dental Clinic Rahime Burcu Nur Yilmaz, DDS, PhD; Didem Nalbantgil, DDS, PhD; Fulya Ozdemir, DDS, PhD Abstract: The aims of this study were to evaluate the posttreatment outcomes in a postgraduate orthodontic clinic following a course on American Board of Orthodontics Cast and Radiograph Evaluation (ABO-CRE); to compare the outcomes of postgraduate students who took the course before and after finishing treatment of their cases; and to assess if the need for orthodontic treatment as determined by the Index of Orthodontic Treatment Need (IOTN) at the beginning of treatment affected students final scores. A course on ABO-CRE was given to second- (group A), third- (group B), and fourth- (group C) year postgraduate students at Yeditepe University, Istanbul, Turkey, in 2012. Pre- and posttreatment plaster models of 253 cases (group A) were treated by students in 2011-12. An additional 251 (group B, 2012-13) and 341 (group C, 2013-14) cases were evaluated in the first and second years after the course, respectively. The models were graded retrospectively using the ABO-CRE and IOTN. The results showed that the total mean scores on the posttreatment plaster models were significantly higher in the pre-course group than the first- and second-year post-course group (p<0.05 and p<0.01, respectively). The borderline cases (grade 3) received a lower score on the ABO-CRE than the cases with need (grade 4) (p<0.01) and severe need (grade 5) (p<0.01) for orthodontic treatment. Increasing awareness by giving information about the ABO-CRE significantly improved the posttreatment success of these postgraduate students. After the course, treatment outcomes in the following year were better than two years later, suggesting it may be useful to teach the course annually to refresh students knowledge. Dr. Yilmaz is Teaching Assistant, Department of Orthodontics, Faculty of Dentistry, Yeditepe University, Istanbul, Turkey; Dr. Nalbantgil is Associate Professor, Department of Orthodontics, Faculty of Dentistry, Yeditepe University, Istanbul, Turkey; and Dr. Ozdemir is Professor, Department of Orthodontics, Faculty of Dentistry, Yeditepe University, Istanbul, Turkey. Direct correspondence to Dr. Rahime Burcu Nur Yilmaz, Yeditepe University, Faculty of Dentistry, Department of Orthodontics, Bagdat Cad. No: 238, 34728 Göztepe, Istanbul, Turkey; +902163636044; drburcunur@gmail.com. Keywords: orthodontics, advanced dental education, postdoctoral dental education, patient care, clinical practice, clinical skills Submitted for publication 12/3/15; accepted 2/29/16 One of the aims of postgraduate/postdoctoral programs in orthodontics is the maintenance and development of optimal standards of patient care for clinical practice. However, monitoring clinical skills by means of treatment outcomes is a challenge. The success of the treatment outcome is multifactorial, depending on factors such as severity of malocclusion at initiation of treatment, treatment need(s) at onset, patient cooperation, the individual response to treatment, and the skill of the clinician. 1 Therefore, methods to assess the outcomes of orthodontic treatment have been developed by many researchers. 2-7 One of the most investigated evaluation methods in orthodontics in the last decade is the American Board of Orthodontics Cast and Radiograph Evaluation (ABO-CRE), the grading system developed to objectively assess the quality of the finished result with respect to the proper arrangement of teeth after orthodontic treatment. 5,6 After introducing the grading system, the ABO also introduced an index called the Discrepancy Index (DI) for classifying the complexity of cases. This index was used as a supplement for choosing cases that are submitted to the ABO and to give direct recommendations such as selection of two cases with DI of 25 and above, six cases with DI 16 and above, and two cases with DI 7 and above. 8 Studies have used the DI criteria to determine the relationship between CRE and DI regarding predictability of the quality of treatment outcomes. Pulfer et al. and Cansunar and Uysal, for example, reported that some of the criteria were affected by pretreatment DI components, even though no statistical correlation was found. 9,10 Orthodontic treatment need has been assessed with other occlusal indices such as the Handicapping September 2016 Journal of Dental Education 1091

Labio-Lingual Deviation Index, Treatment Priority Index, Occlusal Index, Standardized Continuum of Aesthetic Need Index, and Index of Orthodontic Treatment Need (IOTN). 11,12 The need for orthodontic treatment was classified by IOTN with both dental health components (DHC) and aesthetic components (AC). 13,14 The method of the ABO-CRE reflects more the DHC of the scoring IOTN, so using both of these methods enables a more reliable comparison of need regarding the onset and outcome of treatment. For postgraduate orthodontic students, quality of treatment outcomes assessment would be useful for educational improvements and could have an impact on their attitudes and success. 15 Therefore, indicators were developed within the scope of educational programs to form a protocol for improved success as well as to evaluate results with treatment needs at the onset of care in mind. The aims of this study were to evaluate the posttreatment outcomes of a postgraduate clinic after postgraduate orthodontics students took a course on the ABO-CRE; to compare the outcomes of those students who attended the course before and after finishing treatment of their cases; and to determine if the need for orthodontic treatment as determined by the IOTN at the beginning of treatment affected their final scores. Materials and Methods This study was approved by the ethical committee of Yeditepe University, Istanbul, Turkey (#553). The data used in this retrospective study consisted of records of the patients treated by postgraduate students in the Department of Orthodontics clinic at Yeditepe University for the years 2011-14. The postgraduate program in orthodontics at this university is a minimum of four years. The postgraduate students have to deliver a minimum of 60 finished patient records before becoming candidates for graduation. The records are collected and assessed by one lecturer appointed by the head of the department. In 2012, a course about the ABO-CRE criteria was introduced into the postgraduate curriculum, and further one-on-one training with each student was carried out by one lecturer (RBNY). When the course was introduced, there were seven second-year students, five third-year students, and five fourth-year students. While the students in the fourth year had already completed treatment on their patients (group A), treatments by the second- (group C) and third- (group B) year students were ongoing. Groups B and C delivered the records of their finished cases in the years 2012-13 and 2013-14, respectively (Figure 1). For equal distribution, the elimination of potential bias, and competence of postgraduate students, only cases finished in the fourth year were included. A total of 1,031 finished cases were delivered by the students between 2011 and 2014. However, cases that were left unfinished due to the patient s will (n=25), had been started in another clinic (n=27), had been started by another postgraduate student in previous years or were debonded in the second or third years of postgraduate training (n=61), and had incomplete records (n=73) were excluded from the study. The total study sample was thus comprised of 845 optimally finished cases (i.e., not finished due to personal reasons of the patient). The records of 253 finished cases were evaluated before the course (group A, 2011-12), and 251 (group B, 2012-13) and 341 (group C, 2013-14) cases were evaluated in the two years after the course (Figure 1). In 2010, the principal investigator (RBNY), a teaching assistant in the Department of Orthodontics, was calibrated with the head of the department, who was knowledgeable about the evaluation system. Similar to the Yang-Powers et al. calibration method, 16 the two examiners evaluated ten sets of models twice. If more than two discrepancies were recorded for one component, it was corrected by reevaluation of the model. The teaching assistant began to assess posttreatment plaster models (n=300) in the postgraduate clinic to develop a quality indicator for the department in 2010. In the year that followed, all finished cases in the clinic were assessed by the same investigator, for a total of more than 2,000 cases until 2015. The principal investigator was also enrolled in a national oral health epidemiologic survey conducted with 3,040 individuals in seven regions in Turkey to analyze the national orthodontic treatment need using IOTN. 12 In our study, after collection of pre- and posttreatment plaster models as well as the posttreatment panoramic radiographs, one investigator evaluated all the parameters for the final score on treatment outcomes and determined the category of the need for orthodontic treatment for each case by using the ABO-CRE and IOTN, respectively. The posttreatment plaster models were scored in regard to the ABO-CRE, which consists of eight evaluation criteria (alignment, marginal ridges, buccolingual inclination, occlusal relationship, occlusal contacts, overjet, interproximal contacts, and root angulations). 1092 Journal of Dental Education Volume 80, Number 9

Figure 1. Illustration of treatment finishing and course attendance time for each study group (A, B, and C) Any deviation from the defined ideal was noted as a subtraction point. A score of 0 indicated ideal cases; one or two points were subtracted for each deviation. The higher the score, the worse the treatment outcomes. The total subtraction sum of each case was noted as the total ABO-CRE score (Figure 2). 5,14 The DHC of the IOTN was used to identify the treatment need on the pretreatment plaster casts. The clinical component DHC categorizes the need for orthodontic treatment into five grades: grades 1 and 2 represent no/slight need, grade 3 shows moderate, and grades 4 and 5 demonstrate need/severe need. Each grade was assigned according to the severity of the worst occlusal trait. 13,14 Forty cases in each group were collected by selecting every fourth case on the patient name list. Consequently, a total of 120 cases were randomly selected from the main sample group (845 cases) and rescored according to the ABO-CRE and DHC of IOTN to assess intraexaminer reliability. The second assessment was performed eight weeks after the first one. All statistical analyses were performed using the Statistical Package for Social Science, Version 22 (IBM Corp., Istanbul, Turkey) for Windows. To analyze the data, descriptive statistical methods (mean value, prevalence ratio, standard deviation) were used; to compare more than two groups, a oneway ANOVA test was carried out. The Tukey HSD test was used to identify the group that was causing statistical differences. For all statistical analyses, the significance level was set at p<0.05. To evaluate intrarater reliability for ABO-CRE scores and the degree of nonrandom agreement between assessments for DHC of IOTN grades, Intraclass Correlation Coefficient (ICC) and Kappa tests were used, respectively. Results The total sample group (n=845) consisted of data for 320 (37.9%) male and 525 (62.1%) female patients. This group was divided into three groups: group A (253 cases, 29.9% of total group), group B (251 cases, 29.7% of total group), and group C (341 cases, 40.4% of total group). The ICC of the ABO- CRE scores showed a very high intrarater agreement for first and second measurements (ICC: 0.998; 95% CI: 0.998-0.999). The nonrandom agreement between observers for the DHC of IOTN was 0.869 according to the Kappa test (Kappa: 0.869; 95% CI: 0.786-0.951). The mean ABO-CRE score for the total sample was 27.04±11.12. The mean ABO-CRE scores of the posttreatment plaster models were significantly higher in the pre-course group (group A) than the first- (group B) and second- (group C) year postcourse groups (p=0.001 and p=0.024, respectively; September 2016 Journal of Dental Education 1093

Figure 2. Case examples with same ABO-CRE scores at posttreatment stage but different IOTN grade at onset of treatment (IOTN: case A>case B; ABO-CRE: case A=case B) Table 1). No differences were evident between groups B and C (p=0.200 and p>0.05). The maximum/minimum ABO-CRE scores for groups A, B, and C were 82/5, 56/2, and 64/4, respectively. The percentage of ABO-CRE scores in groups A, B, and C that were less than 30 were calculated as 37.94% (96/253), 27.09% (68/251), and 28.15% (96/341), respectively. All cases were borderline (11.5%) or had need/ severe need (62.5%/26%) for orthodontic treatment classified by the IOTN. The borderline cases (grade 3) presented a significantly lower subtraction score of ABO-CRE than the cases with need (grade 4) and severe need (grade 5) for orthodontic treatment (p<0.01). No significant differences were found between the need (grade 4) and severe need (grade 5) groups (p=0.651 and p>0.05, respectively) (Table 2, Figure 2). Discussion The ABO grading system was introduced as an objective method to measure posttreatment outcomes with the evaluation of dental casts and panoramic radiographs. 5 Before introduction of this system, each practitioner evaluated his or her own work subjectively before seeking board certification; therefore, the reason for failure had probably been inadequate knowledge of the criteria for passing. James pointed out that the passing rate has significantly increased since introduction of the criteria and since the candidates began scoring their cases before the exam and selecting cases that are more likely to pass. 17 While the ABO-CRE is a clinical examination tool used to rank orthodontic treatment outcomes, it has also been used in previous studies to increase the reliability and precision of assessment of treatment outcomes in postgraduate clinics. 15,18 Thus, clinical performance and treatment outcomes can be evaluated reliably with the help of the ABO-CRE. After introduction of the grading system, research has focused on the treatment outcome assessment. 1,7,9,10 Some studies evaluated the outcomes in university postgraduate clinics, comparing those outcomes with the outcomes in private clinics and in samples that passed the board. 1,10,15,16 Other studies were interested in the relation between pretreatment complexity and treatment outcomes. 9,10,16 How- 1094 Journal of Dental Education Volume 80, Number 9

Table 1. Comparison of ABO-CRE scores for groups A, B, and C ABO-CRE Group Maximum Value Minimum Value Mean±SD p-value A 82 5 28.97±11.98 a B 56 2 24.89±0.47 b 0.001* C 64 4 26.57±11.03 b a,b Same superscript letters indicate no statistically significant difference (one-way ANOVA test, Tukey HSD test). *Significant at p<0.01 Table 2. Comparison of ABO-CRE scores according to DHC of IOTN ABO-CRE IOTN Maximum Value Minimum Value Mean±SD p-value 3 58 4 21.82±9.71 a 4 82 2 27.48±10.72 b 0.001* 5 68 4 28.26±12.08 b a,b Same superscript letters indicate no statistically significant difference (one-way ANOVA test, Tukey HSD test). *Significant at p<0.01 ever, only two studies used outcomes assessment in attempts to improve an educational program to enhance outcomes. 15,18 Furthermore, to the best of our knowledge, no previous research has assessed if the outcomes were enhanced after educational interventions such as lecture courses or one-on-one training. Overall, the requirement of repetition of courses and education had not previously been evaluated. Therefore, our main goals were to evaluate the improvement in posttreatment outcomes after using the ABO-CRE in a postgraduate clinic and to compare three postgraduate student groups who participated in the course after and before treatment finishing of their patients. The treatment outcomes were categorized as passing, undetermined, and failing cases according to ABO-CRE scores. 10 This grouping did not guarantee that the candidate passed the board, but experience has shown that scores of less than 20 commonly pass, scores of 20-30 are undetermined, and scores of more than 30 usually fail. The scores on posttreatment cases that were lowered by 30 or fewer points probably would pass the board certification process. 18 In our study, the cases with scores of 30 or less ranged between 62% and 72%. These percentages were fairly high as Yang-Powers et al. noted that only 19.6% of university and 46.9% of private practice cases would be able to pass the board certification with ABO-CRE scoring. 16 In our study, only the optimal finished cases (not finished because of personal reasons such as moving to another city) were evaluated, which may be responsible for the higher passing rate. In our study, the improvement of ABO-CRE scores after introduction of the scoring system was approximately four points. Knierim et al. evaluated 437 cases treated in a postgraduate clinic over a three-year period, 15 and Pinskaya et al. compared those data with the results of a second study of 521 cases. 18 In between these study periods, the variables of the clinical protocol were to educate students about the problems identified in the earlier study, to start collecting records six months prior to finishing the treatment to gain time to correct the discrepancies, and to evaluate cases more frequently by the clinic doctor. The results of the two studies showed that ABO-CRE scores improved by 9.4 points (mean score of 34.59 in 1998-2000 to 25.19 in 2001-03), which was higher than our recorded scores. The reasons for the more significant improvement in those studies may be due to those students extremely high scores in the first year (mean score 34.59), which were approximately six points higher than in our first-year evaluation (mean score 28.97). The study of Knierim et al. included both optimally finished and unfinished cases, 15 whereas in our study only the optimal finished cases were evaluated. September 2016 Journal of Dental Education 1095

Campbell et al. also pointed out that if the unfinished cases were included in the study group, ABO-CRE score of the total sample would be higher. 1 The protocol for finishing cases in the postgraduate clinic at our university include some of the improvements from Knierim et al. s study. 15 The cases have to be discussed with the referent doctor at almost every appointment, and the finishing of every case has to be approved by the clinical director. Panoramic radiographs are taken during the finishing stage to enable corrections before debonding. Moreover, the progress and posttreatment cases are routinely discussed with all members of the department as case reports, so that students learn from the experiences of their colleagues. The results of our study showed that the ABO- CRE scores improved from the pre-course group (group A mean score: 28.97) to the first-year (group B mean score: 24.89) and second-year (group C mean score: 26.56) post-course groups. A slight increase in the score occurred between the first-year (group B mean score: 24.89) and second-year (group C mean score: 26.56) post-course groups, but differences were not significant (p>0.05). In order to retain the information learned in the course on ABO criteria, repetition and further training may be needed each year, because in the second year the scores were higher than the first-year scores. Nevertheless, according to our results, the treatment outcomes of the postgraduate students improved significantly compared to the scores before the course. The index of need for orthodontic treatment, as defined by Brook and Shaw, classified malocclusions and thereby the need for orthodontic treatment in terms of occlusal features. 13 The percentages of the borderline, need, and severe need groups for orthodontic treatment in our study sample were 11.5%, 62.5%, and 26%, respectively. Patients with severe need for orthodontic treatment such as craniofacial syndromes generally need multidisciplinary therapy approaches. Although recently one-year clinical fellowships in craniofacial centers have been developed to enable practitioners to gain experience in the treatment of cases with severe treatment need, many private practitioners still refer such cases to universities and hospital centers. 19,20 Therefore, it is not a surprise that our sample contained no cases with no or mild need for orthodontic treatment. At the same time, the increase in ABO-CRE mean score in relation to the increase in the need for orthodontic treatment was also an expected result. Interestingly, the cut-off between the borderline and need/severe need for the finished cases was also indicated by the ABO-CRE scores. While the mean ABO-CRE score for the borderline category of DHC of IOTN was 21.82, the ABO-CRE scores increased significantly in the need (mean score 27.48) and severe need (mean score 28.26) groups. A potential limitation of our study is that it did not compare each evaluation criterion of ABO- CRE at onset and finishing stage. Furthermore, the relations among elements of IOTN with treatment outcomes were not evaluated. It may be advisable in future research to assess the changes in alignment, marginal ridges, buccolingual inclination, occlusal relationship, occlusal contacts, overjet, interproximal contacts, and root angulations points separately to determine the most common mistakes among postgraduate students and consequently to improve educational programs in these areas. Finally, since this study took place at one university, its findings may not be generalizable to postgraduate students in other programs. Conclusion The following conclusions can be drawn from this study. Being informed about and trained in ABO-CRE improved the treatment outcomes of these postgraduate students significantly. After the course, the mean ABO-CRE score in the following year was less than that two years later, meaning the treatment outcomes were better immediately after the course. Therefore, it may be useful to teach a course on ABO-CRE annually. The grade of the need for orthodontic treatment at the onset of treatment had an effect on the final ABO-CRE scores at posttreatment. Therefore, for fair-minded evaluation of the final scores of postgraduate students, the treatment needs of cases at onset should also be considered. Moreover, it may be helpful to educate students about both of these evaluation systems together while scoring the treatment outcomes. REFERENCES 1. Campbell CL, Roberts WE, Hartsfiled JK, Qi R. Treatment outcome in a graduate orthodontic clinic for cases defined by the American Board of Orthodontics malocclusion categories. Am J Dentofacial Orthod 2007;132:822-9. 2. Cons NC, Jenny J, Kohout FJ. DAI: the dental aesthetic index. Iowa City: College of Dentistry, University of Iowa, 1986. 1096 Journal of Dental Education Volume 80, Number 9

3. Richmond S, Shaw WC, O Brien KD, et al. The development of the PAR index (peer assessment rating): reliability and validity. Eur J Orthod 1992;14:125-39. 4. Richmond S, Shaw WC, Roberts CT, Andrews M. The PAR index (peer assessment rating): methods to determine outcome of orthodontic treatment in terms of improvement and standards. Eur J Orthod 1992;14:180-7. 5. Casko JS, Vaden JL, Kokich VG, et al. Objective grading system for dental casts and panoramic radiographs. Am J Orthod Dentofacial Orthop 1998;114(5):589-99. 6. Hoybjerg AJ, Currier GF, Kadioglu O. Evaluation of 3 retention protocols using the American Board of Orthodontics cast and radiograph evaluation. Am J Orthod Dentofacial Orthop 2013;144:16-22. 7. Onyeaso CO, Begole EA. Relationship between index of complexity, outcome and need, dental aesthetic index, peer assessment rating index, and American Board of Orthodontics objective grading system. Am J Orthod Dentofacial Orthop 2007;131(2):248-52. 8. Cangialosi TJ, Riolo ML, Owens SE, et al. The ABO discrepancy index: a measure of case complexity. Am J Orthod Dentofacial Orthop 2004;125:270-8. 9. Pulfer RM, Drake CT, Maupome G, et al. The association of malocclusion complexity and orthodontic treatment outcomes. Angle Orthod 2009;79:468-72. 10. Cansunar HA, Uysal T. Relationship between pretreatment case complexity and orthodontic clinical outcomes determined by the American Board of Orthodontics criteria. Angle Orthod 2014;84:974-9. 11. Güray E, Ertaş E, Orhan M, Doruk C. An epidemiologic survey using treatment priority index (TPI) on primary school children in Konya. Turkish J Orthod 1997;7:195-200. [in Turkish] 12. Nur RB, Ilhan D, Fişekçioğlu E, et al. Total and interregional differences of the need for orthodontic treatment in Turkey: epidemiologic surveillance analysis. Turkish J Orthod 2014;27:1-8. [in Turkish] 13. Brook PH, Shaw WC. The development of an index of orthodontic treatment priority. Eur J Orthod 1989;11: 309-20. 14. Shaw WC, Richmond S, O Brien K D. The use of occlusal indices: a European perspective. Am J Orthod Dentofacial Orthop 1995;107:1-10. 15. Knierim K, Roberts WE, Hartsfield J. Assessing treatment outcomes for a graduate orthodontics program: follow-up study for the classes of 2001-03. Am J Orthod Dentofacial Orthop 2006;130:648-55. 16. Yang-Powers LC, Sadowsky C, Rosenstein S, BeGole EA. Treatment outcome in graduate orthodontics clinic using the American Board of Orthodontics grading system. Am J Orthod Dentofacial Orthop 2002;122:451-5. 17. James RD. Objective cast and panoramic radiograph grading system. Am J Orthod Dentofacial 2002;122:450. 18. Pinskaya YB, Hsieh TJ, Roberts WE, Hartsfield JK. Comprehensive clinical evaluation as an outcome assessment for a graduate orthodontic program. Am J Orthod Dentofacial Orthop 2004;126:533-43. 19. Vig KW, Mercado AM. Overview of the orthodontic care for children with cleft lip and palate, 1915-2015. Am J Orthod Dentofacial Orthop 2015;148:543-56. 20. McCarthy JG. Development of craniofacial orthodontics as a subspecialty at New York University Medical Center. Semin Orthod 2009;15:221-4. September 2016 Journal of Dental Education 1097