Challenges exist in dental hygiene programs. Dental Hygiene Faculty Calibration Using Two Accepted Standards for Calculus Detection: A Pilot Study

Transcription

1 Allied Dental Education Dental Hygiene Faculty Calibration Using Two Accepted Standards for Calculus Detection: A Pilot Study Lisa J. Santiago, RDH, MS; Jacqueline J. Freudenthal, RDH, MHE; Teri Peterson, EdD; Denise M. Bowen, RDH, MS Abstract: Faculty calibration studies for calculus detection use two different standards for examiner evaluation, yet the only therapeutic modality that can be used for nonsurgical periodontal treatment is scaling/root debridement or planing. In this study, a pretest-posttest design was used to assess the feasibility of faculty calibration for calculus detection using two accepted standards: that established by the Central Regional Dental Testing Service, Inc. (CRDTS; readily detectible calculus) and the gold standard for scaling/root debridement (root roughness). Four clinical dental hygiene faculty members out of five possible participants at Halifax Community College agreed to participate. The participants explored calculus on the 16 assigned teeth (64 surfaces) of four patients. Calculus detection scores were calculated before and after training. Kappa averages using CRDTS criteria were at pretest and at posttest. Kappa scores using the scaling/root debridement or planing standard were at pretest and at posttest. The scores indicated improvement from moderate (Kappa= ) to substantial agreement (Kappa= ) following training using the CRDTS standard. Although this result differed qualitatively and Kappas were significantly different from 0, the differences for pre- to post-kappas for patient-rater dyads using CRDTS were not statistically significant (p=0.778). There was no difference (p=0.913) in Kappa scores pre- to post-training using the scaling/root debridement standard. Despite the small number of participants in this study, the results indicated that training to improve interrater reliability to substantial agreement was feasible using the CRDTS standard but not using the gold standard. The difference may have been due to greater difficulty in attaining agreement regarding root roughness. Future studies should include multiple training sessions with patients using the same standard for scaling/root debridement used for evaluation of students. Ms. Santiago is Dental Hygiene Instructor, Halifax Community College and was a graduate student, Idaho State University, at the time of this study; Prof. Freudenthal is Associate Professor, Department of Dental Hygiene, Idaho State University; Dr. Peterson is Assistant Professor, College of Business and was Statistical Consultant, Division of Health Sciences, Idaho State University at the time of this study; Prof. Bowen is Professor Emeritus, Department of Dental Hygiene, Idaho State University. Direct correspondence to Prof. Jacqueline Freudenthal, Department of Dental Hygiene, Idaho State University, Mail Stop 804, Pocatello, ID 83209; ; freujacq@isu.edu. Keywords: dental hygiene, dental hygiene faculty, dental hygiene education, allied dental education, faculty development, calibration, calibration training Submitted for publication 7/13/15; accepted 12/26/15 Challenges exist in dental hygiene programs and dental schools to ensure consistency in faculty assessments of students clinical performance. Over time, researchers have sought methods to increase reliability in evaluation of student performance in many facets of dental and dental hygiene education. Findings of early faculty calibration studies suggested standardization in clinical techniques may improve reliability in clinical evaluation. 1-3 Knight argued that students need to be aware of ideal performance standards and be given opportunities to perform skills that mimic situations similar to those they will perform after graduation in order to achieve competence. 4 Studies have concluded that discussion, comparison of technique, and continual practice all components of faculty calibration training have the potential to improve reliability. 2,5-10 Faculty calibration for calculus detection presents a distinctive challenge relative to the existence of two established standards for evaluation of student competence. The standard established by the Central Regional Dental Testing Service, Inc. (CRDTS) and similar agencies is described as readily detectable calculus, a definite jump or bump which is easily detected with one or two strokes; a deposit that easily binds or catches the explorer; ledges or ring formations; spiny or nodular formations. 11 CRDTS reports successful calibration of its examiners for August 2016 Journal of Dental Education 975

2 dental hygiene testing using this standard. 12 However, the gold standard accepted for immediate evaluation of adequacy of scaling/root debridement or planing in clinical practice and education differs because removal of clinically detectable calculus is necessary for the health of adjacent tissues Plessas described the evolution of philosophy affecting the approach to periodontal instrumentation in a recent evidence-based review of nonsurgical periodontal therapy. 13 Historically, a primary objective of periodontal instrumentation was the removal of calculus and diseased cementum. Consequently, vigorous hand instrumentation and root planing were considered necessary to remove infected cementum and achieve a glassy smooth root surface. Subsequently, other studies showed that bacterial endotoxins were not firmly attached on the root surface and that periodontal health can be achieved without excessive removal of cementum by scaling/ root planing The term periodontal debridement has been used to describe gentle but thorough instrumentation (using power-driven and hand scaling) aimed at the removal of plaque, endotoxin, and calculus but not cementum Although there is no evidence that complete root smoothness, such as a glassy surface, is necessary for gingival health, the gold standard used for immediate evaluation of periodontal instrumentation is removal of calculus and root roughness without excessive cementum removal Elimination of as much root roughness as necessary to achieve a smooth root surface and attain gingival health is advocated. 20,21 Pattison and Pattison warned that clinicians, educators, and researchers should not shift too far from this standard with the periodontal debridement approach rather than root planing. 20 The standard of root roughness is commonly accepted for clinical evaluation of students removal of subgingival calculus deposits and likely presents a greater challenge for faculty calibration than the standard used by testing agencies such as CRDTS for evaluation of minimal competence required for licensure. Challenges cited have included working subgingivally without vision, clinical judgment to differentiate between calculus and root morphology, and disparities among clinicians in tactile sensitivity. 21 Although no data exist documenting the relationship between faculty calibration and clinical dental hygiene instrumentation learning outcomes, inconsistency in evaluation potentially could distract student learning, lead to students concerns or frustrations, and detract from the ultimate goal of reaching competence. 22,23 Students understanding of clinical expectations may be impacted if clinical assessments are inaccurate and variable due to individual faculty members using their own standards instead of the gold standard established and accepted by the profession. 24 Faculty assessment methods are necessary components of clinical evaluation systems because these methods assist clinical faculty in making judgments about students performance in relationship to attaining the profession s gold standards. Faculty members should be consistent when making judgments regarding students performance and need to evaluate students based on the gold standard. If faculty judgment is skewed, the chance of rater agreement is compromised. 2 The lack of interrater agreement among dental and dental hygiene faculty members is widely recognized; however, results of studies regarding calculus detection are mixed. Recently, a study by Partido et al. tested intra- and interrater calculus scores with the use of a dental endoscope. 25 These investigators conducted pre- and posttests with faculty members (N=6) assigned to control and training groups. Training included an ODU 11/12 explorer, typodonts with simulated calculus deposits produced by the manufacturer, and dental endoscopy. Partido et al. pre-established a standard for mastery at 80% accuracy with the answer key and a 0.80 Kappa score, consistent with the Kappa average required for regional clinical board examiners. The results showed significant increases in interrater reliability levels for the training group pretest to posttest. No significant between-group differences were found in intrarater reliability, as both groups improved. Findings of other studies have indicated calibration was unable to improve agreement in faculty evaluation of calculus detection. Garland and Newell conducted a study of faculty calibration using typodonts with simulated calculus and ODU 11/12 explorers and concluded there was no effect on interrater reliability levels of clinical faculty (N=12) for calculus detection after calibration training, although Kappa averages indicated substantial agreement prior to calibration training, leaving little margin for improvement. 23 Those authors primary recommendation for future studies was to conduct training using patients for more realistic evaluation results. An earlier study by Pippin and Feil examined interrater agreement among clinical examiners (N=10) that explored and scored subgingival calculus using patients and simulated calculus in typodonts. 26 Their findings indicated fair agreement, with low Kappa 976 Journal of Dental Education Volume 80, Number 8

3 scores of 0.33 for patients and 0.34 for typodonts after calibration. Substantial agreement (Kappa ) or nearly perfect agreement ( ) is desired for calibration of faculty examiners. 27 Authors of recent faculty calibration studies that did not use endoscopy have suggested the need for objective measures of calculus detection in patients to attain effective faculty calibration training for subgingival calculus detection techniques. 23,25 Research studies assessing the value of dental and dental hygiene faculty calibration and the impact of interrater and intrarater reliability on student learning and satisfaction have presented conflicting findings. 1,6,8,23 Although some researchers were unsuccessful in reaching successful calibration, all agreed that the faculty training was beneficial and concluded that different approaches or multiple calibration sessions might be needed. However, the specific time and mechanism needed for calibration remain undetermined. The aim of our pilot study was to evaluate the feasibility of dental hygiene faculty calibration during calculus detection using two different accepted standards. Calibration training was conducted clinically with patients to address the primary recommendation made by Garland and Newell to replicate conditions of students clinical evaluation standards. 23 The absence of calculus assessed by root smoothness or roughness without excessive removal of cementum is the gold standard for scaling/root debridement or planing used by most, if not all, dental hygiene and periodontal educational programs for evaluation of calculus removal. This standard may not be equally replicated when using replacement teeth for typodonts with simulated calculus. Endoscopy was not used in this pilot study due to the anticipated increase in time and procedures required for the patients. Use of endoscopy for faculty training also is not feasible for many dental hygiene programs due to the cost of the dental endoscope. Inclusion criteria for patients (N=4) were as follows: at least18 years of age; last prophylaxis more than six months prior to the study; availability of a full mouth series of radiographs exposed within the last 12 months; blood pressure 160/100 as clinical policy requires these patients be referred for medical evaluation; and a minimum of 24 teeth present in the dentition. Exclusion criteria included patients reporting uncontrolled systemic diseases, although none did. After obtaining informed consent, patient eligibility was confirmed. All patients had supragingival and subgingival deposits. One patient with gingivitis had light, grainy calculus on most proximal surfaces; two patients with early periodontitis had light and moderate deposits in posterior areas and heavy calculus in the mandibular anterior sextant; one patient had moderate periodontitis with moderate calculus generalized. Thus, patients used for calibration represented various types of deposits commonly found, as well as various periodontal conditions including recession, normal sulcus depths, and pocket depths. No anesthesia was used. Materials and Methods The Idaho State University Human Subjects Committee granted this study exempt status. This pilot study measured pre- and post-training interrater reliability of dental hygiene faculty members at Halifax Community College. The standard single-group, pretest-posttest research design used three sessions (Figure 1). All full- and part-time clinical faculty members (N=5) were invited to participate via a letter. Figure 1. Faculty calculus detection calibration study protocol August 2016 Journal of Dental Education 977

4 During the pretest session, patients health histories and blood pressure readings were updated, and the clinical faculty members were assigned alphabetically to one of two groups. This half-mouth assignment was made for the comfort of the patients by limiting the number of times each surface would be explored by the faculty participants in each group. The ODU 11/12 explorer was selected because of its familiarity to the faculty participants and for consistency with the CRDTS examination and previous studies of faculty calibration for calculus detection. 23,25 This explorer also was designed to explore normal sulci or deep periodontal pockets in both anterior and posterior teeth. 19 Sterilized, new explorers and full mouth radiographs were displayed on view boxes for each patient in order to assess calculus deposits. Each group was randomly assigned two quadrants to explore for presence or absence of calculus deposits. All teeth present were explored on the left and right sides of the mouth, except for third molars; however, six teeth (24 surfaces) on each side of the mouth for a total of 12 teeth (48 surfaces per half mouth) were selected for data analysis to ensure the same teeth were present in all of the patients mouths. Participants explored the two randomly assigned quadrants and recorded C for calculus deposits or R for root roughness using the CRDTS Dental Hygiene Full Mouth Patients Screening worksheet. 11 The box was left blank where no calculus or roughness was detected. No discussion of findings was allowed between participants during pre- or posttests. A one-hour 15-minute calibration workshop was conducted one week after the pretest (Table 1). The posttest occurred immediately following the training. To establish consistency in differentiating between the calculus deposits required by CRDTS and root roughness, the participants repeatedly explored these various areas of grainy and obvious calculus with the tooth out of the typodonts and inserted into the typodonts. They then reached consensus regarding whether to record a C for an obvious deposit or R for root roughness for each deposit. The participants discussed assigned scenarios depicting challenging instances encountered during calculus detection, such as anatomical landmarks of a tooth that can be mistaken as calculus, existing restorations, root proximity and roughness, and malaligned teeth. Open discussion about the pretest also took place, and the use of air was encouraged. Patient preparation, armamentarium, and recording protocols for the posttest were identical to those described for the pretest. Each clinical faculty member explored the same half mouth she had explored during the pretest. All patients were invited to return to discuss findings and receive preventive dental hygiene care. A data set comprised of all pretest and posttest calculus detection data was constructed in Excel for Table 1. Faculty calibration training process Topic Description Time PowerPoint presentation regarding goals of calibration training CRDTS criteria review Video by Nield-Gehrig Scenario discussions Kilgore periodontal dentoform Discussed definition and purpose of calibration, importance of calculus detection calibration, and potential impact on faculty as well as students. Reviewed criteria for qualifying calculus deposits developed by CRDTS for board examination candidates. Video was shown to demonstrate proper technique for use of ODU 11/12 explorer. Video demonstrated how to determine the correct working end, tip positioning, insertion points, adaptation, activation using overlapping oblique and horizontal strokes, neutral wrist position, and correct handle position. The trainer and participants discussed challenging cases and situations that participants had experienced during calculus detection. This typodont, currently used by the participants, had grainy and readily detectable calculus present with an answer key. Participants demonstrated proper use of the ODU 11/12 explorer to detect calculus. Some removed teeth to visibly confirm presence of calculus. 10 minutes 10 minutes 10 minutes 30 minutes 15 minutes CRDTS=Central Regional Dental Testing Service, Inc. 978 Journal of Dental Education Volume 80, Number 8

5 statistical analysis using SPSS version (2012). Cohen s kappa coefficients were calculated at the pre- and posttests for each of the standards to test the following hypotheses: 1) there would be no difference in interrater reliability of calculus detection scores following calibration training designed to enhance agreement on assessment of calculus deposits based on the CRDTS standard (readily detectible calculus); and 2) there would be no difference in interrater reliability of calculus scores following calibration training designed to enhance agreement on assessment of calculus deposits based on the standard established for evaluation of adequacy of scaling/root debridement or planing. A paired sample t-test was used to compare kappa scores pre- to post-training at the 0.05 alpha level. Results The convenience sample consisted of four dental hygiene members of the clinical faculty who routinely evaluate students calculus removal and volunteered for the study (this was four out of a total five possible participants). The sample included two full-time and two part-time faculty members. One participant had two years of clinical teaching experience, and three had eight to ten years of clinical teaching experience. All participants had over 20 years of clinical practice experience as dental hygienists. A Cohen s Kappa was used to assess interrater reliability of faculty calculus detection scores before and following calibration training based on the standard established by CRDTS (readily detectible calculus). 11 The pretest Kappa score was compared to at posttest, indicating improvement from moderate (Kappa= ) to substantial agreement (Kappa= ) (Table 2). 27 Although this result is qualitatively different and the Kappas are significantly different from 0, when comparing the pre- to post-kappas for patient-rater dyads using CRDTS, there was no statistically significant increase (p=0.778) in the Kappa scores. This lack of statistical significance could be a result of the small sample size in this study. Table 2 also shows the Kappa scores based on the gold standard established for immediate evaluation of adequacy of scaling/root debridement or planing and root smoothness. These Kappa scores were at pretest and at posttest. Both scores indicated slight to fair agreement; no qualitative or quantitative improvement was shown. Using a paired t-test, this minor change was not statistically significant (p=0.913). Discussion Studies of faculty calibration for calculus detection present a unique challenge in design and measurement relative to the existence of two different standards used for the evaluation of calculus removal. The problem with the lack of interrater agreement among dental and dental hygiene faculty members has been widely recognized in several areas of clinical evaluation. Partido et al. suggested that problems of inconsistency in calculus detection among faculty members can be reduced with calibration training. 25 Others have found faculty calibration for calculus detection to be ineffective using patients and typodonts. 23,26 Our pilot study followed the primary recommendation of Garland and Newell to evaluate calculus in patients with varying amounts of deposits rather than typodonts with simulated calculus in order to provide realistic tactile sensations for faculty calibration. 23 The patients selected for this study exhibited different degrees of calculus deposits representing conditions students may encounter in the clinical setting, from light and grainy to readily detectable. The results showed a difference posttraining in the participants interrater reliability when the CRDTS criteria were used; however, there was no difference when all types of deposits were assessed, despite time spent discussing the associated challenges during training. Table 2. Kappa scores for testing dental hygiene faculty members interrater reliability (N=4) Standard for Calibration Pretest/Posttest Kappa Statistic Standard Error p-value CRDTS Pre Post Root roughness Pre Post August 2016 Journal of Dental Education 979

6 Previous studies used typodonts with simulated calculus on varied numbers of teeth for faculty calibration training. Garland and Newell handcrafted calculus to simulate light, moderate, and heavy deposits. 23 Partido et al. used typodont teeth prepared with calculus deposits by the manufacturer. 25 These manufacturer-produced deposits tend to be moderate to heavy, similar to the type of deposits described by the CRDTS standard. The gold standard for evaluating adequacy of scaling/root debridement or planing in clinical practice and education differs from that used by CRDTS because removal of all clinically detectable calculus is necessary for the health of adjacent tissues There is no expectation that complete root smoothness be attained, so initial evaluation of root smoothness or roughness followed by tissue reevaluation is recommended. This gold standard of root smoothness is accepted universally for immediate clinical evaluation of students removal of calculus deposits and likely presents a greater challenge for faculty calibration than the standard for removal of readily detectable calculus used by testing agencies such as CRDTS for evaluation of minimal competence required for licensure. The results of our pilot study were that, while calibration to substantial agreement was attained for detection of readily detectable calculus, it was not attained for evaluation of root roughness using the same training. Students have anecdotally reported frustration with differences in evaluation of scaling/root debridement or planing. 22 This procedure requires a higher level of skill and judgment than removal of readily detectable calculus. The results of calculus detection following faculty calibration using the CRDTS criteria showed substantial agreement among faculty calculus detection scores after training, although the Kappa score of 0.80 required by CRDTS was not achieved. 21 CRDTS reports successful calibration of examiners. 12 The calibration training in our pilot study utilized similar training techniques employed by CRDTS, including a PowerPoint presentation and discussion regarding policy, procedures, and protocol for evaluating and recording calculus deposits detected during exploration. 11 The use of typodonts with artificial calculus for practice and application of principles was discussed in the workshop, and a video demonstration of the proper use of the ODU 11/12 explorer was shown. However, the level of training provided by CRDTS exceeds the training provided during our study. CRDTS conducts multiple training sessions for continuous examiner calibration. 28 Results of previous dental and dental hygiene faculty calibration studies for other clinical procedures have concluded that multiple training sessions may be needed to attain agreement. 1,10,24,29 The faculty training prior to the posttest did not emphasize individualized technique in detecting root roughness other than the video demonstration of using the ODU 11/12 explorer and discussion of case studies presenting related challenges. The subjects explored examples of root roughness and obvious deposits on typodonts, and the majority of the time was spent discussing situations that would help faculty members differentiate root roughness from other conditions and past experiences in determining root roughness. Exploring root roughness in patients mouths and discussing agreement and differences, as well as evaluating exploring techniques, may enhance calibration training for student evaluation using the gold standard. The participants informal comments after the posttest indicated that exploring for calculus deposits using the root roughness standard, in particular, was very stressful, as they tended to second guess themselves. Determining whether root roughness is calculus requiring additional periodontal instrumentation poses a significant challenge; however, dental hygiene students performance commonly is evaluated using this standard in the clinical setting. CRDTS has determined it is extremely difficult to achieve 80% agreement with root roughness due to widely varying opinions regarding the presence and extent of root roughness. 12 The CRDTS dental hygiene examination is an entry-level competency assessment for licensure; therefore, examiners are required to demonstrate a high level of interrater reliability based on the defined criteria for qualifying calculus and calculus removal. When measuring interrater agreement for calculus detection using the CRDTS criteria, our study found substantial agreement post-training. When using the standard for scaling/root debridement or planing, the participants did not achieve the goal of substantial interrater agreement. This difficulty in clinical judgment regarding root roughness in clinical practice and potential damage when the goal of instrumentation is complete root smoothness has resulted in a shift from immediate final evaluation of root planing to the periodontal debridement approach, requiring an ultimate evaluation based on tissue response, such as gingival inflammation or bleeding on probing, four to six weeks following nonsurgical periodontal therapy. 15,17, Journal of Dental Education Volume 80, Number 8

7 Based on the results of this pilot study, the question arises: should calculus detection competencies be based on students ability to identify readily detectible calculus, instead of immediate evaluation of root roughness, with the final competency assessment based on tissue response at reevaluation? Another alternative might be to assess competence using the standard of readily detectable calculus, coupled with a non-graded evaluation system for root roughness for initial evaluation of scaling/root debridement or planing to allow for faculty discussion and assistance. Student competencies should be developed to address the different standards and the challenges they pose for experienced faculty members. The question of how to fairly and accurately assess calculus detection is one that should be discussed by dental and dental hygiene educators and studied further. Limitations of this study included the small sample size; however, the number of tooth surfaces evaluated provided adequate power for statistical analysis. The convenience sample of clinical faculty members from a single dental hygiene program also limits generalizability of results. Future research should include multiple training sessions with patients during faculty calibration training for clinical evaluation of subgingival calculus detection. This approach would enhance external reliability of results as a full range of calculus deposits could be represented; training could be focused on achieving the gold standard; and interrater reliability of root roughness could be assessed. Additional pilot studies are needed to confirm the feasibility of attaining agreement with the gold standard followed by a large-scale study involving multiple institutions to provide support or reject the effectiveness of calibration training. Training sessions should include pre-study materials, be conducted with patients at regular intervals, and provide clinical faculty members the opportunity to review and improve exploration techniques and discuss specific cases and scenarios. Conclusion Although this pilot study was small, some conclusions can be drawn and recommendations made for future research. This study compared two accepted calculus detection standards used in examiner calibration with mixed results. Although changes pre- to post-training were not statistically significant, successful faculty calibration occurred when measuring calculus detection using the CRDTS standard. Non-substantial results occurred with the gold standard of root roughness. Unfortunately, little documentation exists for the approaches or number of sessions capable of producing high rater reliability. The question remains as to what standard should be employed when evaluating student competence in calculus removal, considering the apparent difficulty in calibrating faculty detection scores. Acknowledgments The authors would like to thank the dental hygiene faculty participants in this project. Disclosure The authors reported no conflicts of interest related to this study or article. REFERENCES 1. Dahlström L, Keeling SD, Friction JR, et al. Evaluation of a training program intended to calibrate examiners of temporomandibular disorders. ACTA Odontol Scand 1994;52: Flack VF, Atchison KA, Hewlett ER, White SC. Relationships between clinician variability and radiographic guidelines. J Dent Educ 1995;75(2): Biller IR, Kerber PE. Reliability of scaling error detection. J Dent Educ 1980;44(4): Knight GW. Toward faculty calibration. J Dent Educ 1997;61(12): Christie C, Bowen D, Paarmann C. Effectiveness of faculty training to enhance clinical evaluation of student competence in ethical reasoning and professionalism. J Dent Educ 2007;71(8): Assaf AV, Meneghim C, Zanin L, et al. Effects of different diagnostic thresholds on dental caries calibration: a 12-month evaluation. Community Dent Oral Epidemiol 2006;34: Jacks ME, Blue C, Murphy D. Short- and long-term effects of training on dental hygiene faculty members capacity to write SOAP notes. J Dent Educ 2008;72(6): Lanning SK, Pelok SD, Williams BC, et al. Variation in periodontal diagnosis and treatment planning among clinical instructors. J Dent Educ 2005;69(3): Lanning SK, Best AM, Temple HJ, et al. Accuracy and consistency of radiographic interpretation among clinical instructors in conjunction with a training program. J Dent Educ 2006;70(5): Alexander AG, Leon AR, Ribbons JW, Morganstein SI. Evaluation of a training program for the PMA and the gingival index. J Periodontal Res 1972;7(4): Central Regional Dental Testing Service. CRDTS dental hygiene full mouth patient screening worksheet At: Mouth%20%20Patient%20Screening%20Worksheet.pdf. Accessed 3 March Central Regional Dental Testing Service. An evaluation of the Central Regional Dental Testing Service s national August 2016 Journal of Dental Education 981

8 dental hygiene examination At: uploads/2010%20crdts%20dh%20technical%20 Report.pdf. Accessed 3 March Plessas A. Nonsurgical periodontal treatment: review of the evidence. Oral Health Dent Manag 2014;13(1): Apatzidou DA. Modern approaches to non-surgical biofilm management. Front Oral Biol 2012;15: Drisko CH. Nonsurgical periodontal therapy. Periodontol ;25: Smart GJ. The assessment of ultrasonic root surface debridement by determination of residual endotoxin levels. J Clin Periodontol 1990;17: Newman MG, Takei HH, Klokkevold PR, Carranza F. Carranza s clinical periodontology. 11 th ed. St. Louis: Saunders, Hodges KO. Decision making related to nonsurgical periodontal therapy. In: Darby ML, Walsh MM, Bowen DM, eds. Dental hygiene theory and practice. 4 th ed. St. Louis, MO: Elsevier, Neild-Gehrig JS. Fundamentals of periodontal instrumentation and advanced root instrumentation. 7 th ed. Philadelphia: Lippincott Williams & Wilkins, Pattison AM, Pattison GL. Periodontal instrumentation transformed. Dimens Dent Hyg 2003;1(2): Osborn JB, Lenton PA, Lunos SA, Blue CM. Endoscopic vs. tactile evaluation of subgingival calculus. J Dent Hyg 2014;88(4): Henzi D, Davis E, Jasinevicius R, et al. Appraisal of the dental school learning environment: the students view. J Dent Educ 2005;69(10): Garland KV, Newell KJ. Dental hygiene faculty calibration in the evaluation of calculus detection. J Dent Educ 2009;73(3): John V, Lee SJ, Prakasam S, et al. Consensus training: an effective tool to minimize variations in periodontal diagnosis and treatment planning among dental faculty and students. J Dent Educ 2013;77(8): Partido BB, Jones AA, English DL, et al. Calculus detection calibration among dental hygiene faculty utilizing dental endoscopy. J Dent Educ 2015;79(2): Pippin DJ, Feil P. Interrater agreement on subgingival calculus detection following scaling. J Dent Educ 1992;56(5): Viera AJ, Garrett JM. Understanding inter-observer agreement: the kappa statistic. Fam Med 2005;37(5): Personal communication, K. Cobb, Executive Director, Data and Dental Exam Administrator, Central Regional Dental Testing Service, Inc., January Haj-Ali R, Feil P. Rater reliability: short-and long-term effects of calibration training. J Dent Educ 2006;70(4): Journal of Dental Education Volume 80, Number 8