DO NOT DUPLICATE. The Inter-rater Reliability of the Clinical Signs and Symptoms Checklist in Diabetic Foot Ulcers FEATURE

Transcription

1 FEATURE The Inter-rater Reliability of the Clinical Signs and Symptoms Checklist in Diabetic Foot Ulcers Sue E. Gardner, PhD, RN; Rita A. Frantz, PhD, RN; Heeok Park, MSN, RN; and Melody Scherubel, BSN The Clinical Signs and Symptoms Checklist is a tool designed to measure 12 clinical signs and symptoms of localized chronic wound infection. Since its initial development, this Checklist has been revised to include sanguineous drainage. To examine the inter-rater reliability of the revised Clinical Signs and Symptoms Checklist in diabetic foot ulcers, an observational, cross-sectional study was conducted in conjunction with a larger study examining the validity of each sign and symptom for identifying infection in diabetic foot ulcers. Two independent nurse observers assessed 64 diabetic foot ulcers in 64 patients using the Checklist. The reliability of each item was calculated using percent agreement and the Kappa coefficient. Total percent agreement ranged from 76% to 100%, and Kappa statistics ranged from.34 to Study findings confirm that the Clinical Signs and Symptoms Checklist is a reliable tool for identifying the clinical signs and symptoms of localized infection in diabetic foot ulcers. KEYWORDS: wound infection, foot ulcer, diabetic foot, clinical signs and symptoms 46 OstomyWound Management Ostomy Wound Management 2006;53(1):46 51 Dr. Gardner is a Core Investigator, Center for Research in the Implementation of Innovative Strategies in Practice (CRIISP), Iowa City VA Medical Center; and an Assistant Professor, The University of Iowa College of Nursing, Iowa City, Iowa. Dr. Frantz is a Professor and Ms. Park is a doctoral student, The University of Iowa College of Nursing. Ms. Scherubel is a staff nurse, Iowa City VA Medical Center. Please address correspondence to: Sue E. Gardner, PhD, RN, 320 NB, The University of Iowa, College of Nursing, Iowa City, IA 52242; sue-gardner@uiowa.edu. This study was funded in part by the Nursing Research Initiative, Health Services Research and Development, Department of Veteran s Affairs (NRI ) and with support from the Gerontological Nursing Interventions Research Center NIH #P30 NR03979 (PI: Toni Tripp-Reimer, The University of Iowa College of Nursing) and the Hartford Center for Geriatric Nursing Excellence, The John A. Hartford Foundation (PI: Kathleen Buckwalter, The University of Iowa College of Nursing). Its contents are solely the responsibility of the authors and do not necessarily represent the official views of the Department of Veteran s Affairs or the National Institute of Nursing Research.

2 The Clinical Signs and Symptoms Checklist (CSSC) 1 is a tool designed to objectively measure 12 clinical signs and symptoms of localized chronic wound infection. Five of these signs and symptoms are commonly known as the classic signs of infection (pain, erythema, heat, edema, and purulent exudate); the other seven are signs specific to wounds healing by secondary intention (serous exudate, sanguineous drainage, delayed healing, discoloration of granulation tissue, friable granulation tissue, wound base pocketing, foul odor, wound breakdown). 2 Each sign and symptom is represented with a specific descriptor; the conceptual definitions and the development of their descriptors have been reported elsewhere. 1 During first testing, the reliabilities of the items on the tool were found to range between 0.53 and Since its initial development, the CSSC has been revised to include sanguineous drainage as a sign or symptom because wound drainage with a bloody composition did not fit the descriptors for serous exudate or purulent exudate (see Figure 1). Adding sanguineous drainage enhanced discrimination regarding wound drainage. The reliability of the revised CSSC has not been examined. Furthermore, the reliability of the CSSC has not been examined in a homogenous sample of chronic wounds. The purpose of this study was to examine the reliability of the revised CSSC in a sample of diabetic foot ulcers. The findings reported here were obtained in conjunction with a larger study designed to examine the validity of each sign and symptom for identifying infection in diabetic foot ulcers. Methods and Procedures Participants. In this observational, cross-sectional design, two members of the research team independently assessed patients with diabetic foot ulcers for the presence of signs and symptoms of infection using the CSSC. In addition to the principal investigator (PI) a Registered Nurse, three other Registered Nurses were trained in CSSC use in order to examine its reliability. All participating nurses had training and experience in chronic wound management two of the nurses were certified wound care nurses and the other two had more than 3 years of wound care experience. Each nurse reviewed the items on the CSSC and practiced using this tool to assess wounds in conjunction with the PI. Competency was typically achieved with fewer than five practice wounds. Study subjects included patients with diabetic foot ulcers. Subjects were screened and enrolled based on the following criteria: 1) 18 years of age or older, 2) presence of a full-thickness diabetic foot ulcer, 3) white blood count >1,500 cells/mm 3, 4) platelet count >125,000/mm, and 5) no coagulapethies. If a subject had more than one diabetic foot ulcer, one was randomly selected for inclusion in the study. Human subject approval was obtained from the Institutional Review Board at each site before data collection commenced. Informed consent was obtained from all patient participants. Subjects were enrolled in the study from August 2001 through August Setting. A Department of Veteran s Affairs Medical Center and a university-associated tertiary hospital served as settings for the study. Wound assessment. The CSSC was used to assess the presence of the clinical signs and symptoms of infection. Decisions regarding presence of signs and symptoms on the CSSC required observing, palpating, and manipulating the wound as well as observing the dressing for character of wound exudate. After the dressing was removed, the study ulcer was cleansed by gently rubbing the surface with saline-soaked coarsemesh gauze. 1 The dressing, which typically had been in Ostomy Wound Management 2007;53(1):46 51 KEY POINTS Clinical assessment of infection in chronic wounds remains an important challenge and efforts to develop an assessment method that is non-invasive, valid, reliable, and easy to use with a variety of wounds and treatment modalities must continue in order to improve the quality of patient care. After adding a wound drainage item to a previously tested instrument developed to assess clinical signs and symptoms of infection, all individual items were assessed by two nurses for 64 diabetic foot ulcers. The results confirm that, despite variations for some commonly used assessment variables, the instrument items are generally reliable. January 2007 Vol. 53 Issue 1 47

3 CLINICAL SIGNS AND SYMPTOMS CHECKLIST (CSSC) 1 Increasing pain in the ulcer area: subject s subjective report of perceived discomfort increases in level of peri-ulcer pain since the ulcer developed. Ask subject to select the most appropriate statement for current level of ulcer pain from the following choices: 1) I am not able to detect pain in ulcer area, 2) I am having less ulcer pain now than I have had in the past, 3) The intensity of ulcer pain has remained the same since the ulcer developed, or 4) I have more ulcer pain now than I have had in the past. Circle the number corresponding to the most appropriate statement. If number 4 is selected, place a check in the box to the right; mark N/A if the subject is not able to respond to question Erythema: presence of bright or dark red skin or darkening of normal ethnic skin color immediately adjacent to the ulcer opening Edema: presence of shiny, taut skin or pitting impressions in the skin adjacent to the ulcer but within 4 cm of the ulcer margin. Assess pitting edema by firmly pressing the skin within 4 cm of ulcer margin with a finger, releasing, and waiting 5 seconds to observe indentation Heat: detectable increase in skin temperature of the skin adjacent to the ulcer but within 4 cm of the ulcer margin as compared to the skin 10 cm proximal to the wound. Assess differences in skin temperature using the back of the examiner s hand or the wrist Purulent exudate: presence of tan, creamy, yellow, or green thick fluid on a dry gauze dressing removed from the ulcer 1 hour after placement. The wound is cleansed before placing the gauze dressing in the ulcer Sanguineous drainage: presence of bloody fluid on a dry gauze dressing removed from the ulcer 1 hour after placement Serous exudate: presence of thin, watery fluid on a dry gauze dressing removed from the ulcer 1 hour after placement Delayed healing of the ulcer: subject or caregivers report no change or an increase in the volume or surface area of the ulcer over the past 4 weeks. Ask subject or caregiver if the ulcer has filled with tissue or is smaller around than it was 4 weeks ago. If it has/is not, place check in box to the right Discoloration of granulation tissue: granulation tissue that is pale, dusky, or dull in color Friable granulation tissue: bleeding of granulation tissue when gently manipulated with a sterile cottontipped applicator Pocketing at base of wound: presence of smooth, non-granulating pockets of ulcer tissue surrounded by beefy red granulation tissue Foul odor: putrid or distinctively unpleasant smell as assessed by the examiner Wound breakdown: small open areas in newly formed epithelial tissue not caused by re-injury or trauma place overnight, was retained for assessment of the presence and type of wound exudate. Study ulcers were not treated with dressings and or topical treatments (eg, IF PRESENT Figure 1. The Clinical Signs and Symptoms Checklist. hydrocolloids or silver sulfadiazine) that would interfere with the ability to characterize wound exudate; hydrocolloids and silver sulfadiazine interact with 48 OstomyWound Management

4 wound fluids in a manner that makes the wound fluid appear falsely purulent. Two members of the research team trained in the use of the CSSC independently assessed and recorded the presence of each clinical sign and symptom of chronic wound infection to determine inter-rater reliability. Statistical analysis. Inter-rater reliability of each sign and symptom was analyzed using percent agreement (ie, total number of concordant observations divided by total number of paired observations) and Kappa coefficient (ie, observed agreement minus expected agreement divided by one minus expected agreement). Three measures of percent agreement were used because estimates are influenced by the manner in which agreement is defined and total agreement can be inflated by a high percentage of nonoccurrence agreements when few occurrence agreements occur. Because percent agreement does not correct for chance agreements, Kappa statistics, which correct for chance agreement, also were calculated for each Checklist item. Results Participants and wounds. Patient participants (N = 64, 40 from the tertiary hospital, 24 from the VA facility; mean age 55 years, SD ± 11.4 ) were predominantly Caucasian (n = 63; 98%) and male (n = 49; 77%). The subjects had non-arterial, neuropathic diabetic foot ulcers located on the plantar surface of the foot. The most commonly used dressing (n = 44; 69%) was moistened saline gauze covered with a Telfa (Tyco Healthcare/Kendall, Mansfield, Mass), which prevented evaporation and drying of the gauze. The second most frequently used dressing was dry gauze (n = 18; 28%). From a sample of 64 diabetic foot ulcers, 64 paired, independent observations were made using the CSSC. Checklist findings. Table 1 presents the reliability findings for each item on the CSSC, including total percent agreement (TPA), occurrence percent agreement (OPA), non-occurrence percent agreement (NPA), and Kappas. All of the items had TPA values >70%. In addition, all items had Kappa values > Sanguineous drainage (0.385), discoloration of granulation tissue (0.381), and foul odor (0.345) had the lowest Kappa values. An overall Kappa value for the CSSC was not calculated. TABLE 1 RELIABILITY OF THE INDIVIDUAL ITEMS ON THE CLINICAL SIGNS AND SYMPTOMS CHECKLIST Signs and Symptoms Increasing pain Erythema Edema Heat Purulent exudate Serous exudate Sanguineous drainage Delayed healing Discoloration Friable granulation Pocketing Foul odor Wound breakdown TPA 96% 84% 92% 86% 82% 80% 80% 90% 76% 92% 94% 88% 100% OPA 60% 56% 64% 42% 61% 78% 78% 81% 37% 50% 40% 25% 100% NPA 96% 80% 91% 84% 75% 33% 33% 83% 72% 91% 94% 88% 100% Kappa TPA = total percent agreement; OPA = occurrence percent agreement; NPA = nonoccurrence percent agreement N = 64 observations with two observers Discussion The measurement literature generally suggests an agreement of 70% is necessary, 80% is adequate, and 90% is good. 3,4 A Kappa >0.80 generally is almost perfect agreement, between 0.60 and 0.80 substantial agreement, between 0.40 and 0.60 moderate agreement, and between 0.20 and 0.40 fair agreement. 5 Based on TPA and Kappa values, all of the items on the CSSC, including the newer sanguineous drainage item, had fair to almost perfect reliability estimates. These findings are consistent with initial findings on the reliability of the CSSC, 1 although the sample used in this study was twice as large and more homogenous in terms of chronic wound type. In addition, the reliability of the items on the CSSC compare favorably with other studies that have examined inter-clinician agreement with respect to wound infection status. Wirthlin et al 6 found that agreement between surgeons with respect to wound cellulitis/infection and the presence of erythema were and 0.22, respectively. Cutting 7 found that agreement between nurses with respect to the infection status of granulating wounds was only 47.5%. January 2007 Vol. 53 Issue 1 49

5 Neither of these studies employed a structured tool from which to identify the specific signs and symptoms of infection. Consistent with the findings from the initial study, 1 the signs of heat, discoloration of granulation tissue, and foul odor had only moderate or fair agreement. These signs require more subjective judgment than other signs and symptoms because they include the characteristics of touch, color, and smell. The newly included sign (sanguineous drainage) also had only fair agreement, which may indicate the descriptor of this item needs to be revised. It is important to note that the majority of ulcers in this study were dressed with gauze dressings, which are more conducive to assessing type of wound exudate/drainage than other dressing types such as hydrocolloids. 1 Hydrocolloids interact with the wound fluid, producing a creamy substance that is difficult to distinguish from purulent exudate. Gauze dressings allow more direct observations of exudate/drainage color and consistency. In clinical settings that use non-gauze dressings for wound care, wound exudate/drainage can best be assessed by cleansing the ulcer, dressing the ulcer with a dry gauze dressing for an hour, and then completing the assessment of the wound for type of wound exudate. 1 In addition, the findings of this study are based on the assessments of nurses trained in chronic wound assessment and the use of the CSSC. Other clinicians, such as physical therapists, physicians, or podiatrists trained in wound assessment and the use of the CSSC, should be able to achieve similar results. Finally, the fact that most ulcers were plantar neuropathic diabetic foot ulcers may have resulted in all having similar signs and symptoms. This similarity may have inflated the inter-rater reliability findings reported here. Nevertheless, the findings on inter-rater reliability reported here are similar to the findings reported on the CSSC in a mixed sample of chronic wounds. Also, the fact that most of the subjects were Caucasian may have made erythema easier to identify than it may be in persons of color. This may have inflated the inter-rater reliability for erythema. The findings from our first study 1 also were based on a primarily Caucasian sample. Implications for Practice The early identification of infection in diabetic foot ulcers is essential to preventing infection-related complications, such as amputation. The Infectious Disease Society of America (IDSA) recently published guidelines 8 specific to identifying and managing diabetic foot infections. The IDSA states that wound infections must be diagnosed clinically on the basis of local (and occasionally systemic) signs and symptoms of inflammation. 8 Specifically, the guidelines regarding a diabetic foot ulcer note the presence of purulent exudate or the presence of two or more signs of inflammation, such as redness, warmth, swelling or induration, and pain or tenderness. The items on the CSSC relevant to these signs include purulent exudate, increasing pain, erythema, heat, and edema. The identification of purulent exudate had substantial inter-rater reliability using the CSSC, as did increasing pain, erythema, and edema. However, the routine assessment of diabetic foot ulcers often does not include a systematic approach to identifying these specific signs and symptoms of infection. Incorporating the CSSC into wound assessment may improve clinician skill and accuracy in identifying the specific signs and symptoms (eg, purulent exudate and signs of inflammation) delineated as most important by the IDSA. Although the findings from this study suggest that the CSSC may improve the ability to consistently identify signs and symptoms of diabetic foot ulcer infection, the findings reported here do not address the validity of these signs and symptoms as accurate indicators of infection in diabetic foot ulcers. A study 9 designed to examine the validity of clinical signs as indicators of diabetic foot ulcer infection revealed that the IDSA definition (purulent exudate or two or more signs of inflammation) is a better indicator of infection status among diabetic foot ulcers than any one sign or symptom alone. Moreover, the signs specific to secondary wounds were better indicators of infection in diabetic foot ulcers than the classical signs of infection. More study is needed to identify which signs and symptoms, or combinations of signs and symptoms, are most indicative of diabetic foot ulcer infection. 50 OstomyWound Management

6 Conclusion A study to examine the inter-rater reliability of the revised CSSC in diabetic foot ulcers found that interrater reliability was lower but still substantial regarding the more subjective signs and symptoms of infection (eg, heat, discoloration of granulation tissue, and foul odor) than the inter-rater reliability regarding secondary signs and symptoms such as wound breakdown, pocketing, and increasing pain. A systematic approach to wound infection assessment potentially can improve diagnosis and subsequent treatment, making instruments such as the CSSC important tools in the management of diabetic foot wounds specifically and chronic wounds in general. - OWM References 1. Gardner SE, Frantz RA, Troia C, et al. A tool to assess the clinical signs and symptoms of localized infection in chronic wounds: development and reliability. Ostomy Wound Manage. 2001;47(1): Cutting KF, Harding KG. Criteria for identifying wound infection. J Wound Care. 1994;3(4): Hartmann D. Considerations in the choice of interobserver reliability estimates. J Applied Behav Anal. 1977;10(1): House A, House B, Campbell M. Measures of interobserver agreement. Calculation formulas and distribution effects. J Behav Assess. 1981;3(1): Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33(1): Wirthlin DJ, Buradagunta S, Edwards RA, et al. Telemedicine in vascular surgery: feasibility of digital imaging for remote management of wounds. J Vasc Surg. 1998;27(6): Cutting KF. Identification of infection in granulating wounds by registered nurses. J Clin Nurs. 1998;7(6): Lipsky BA, Berendt AR, Deery HG, et al. IDSA guidelines: diagnosis and treatment of diabetic foot infections. Clin Infect Dis. 2004;39(7): Gardner SE, Frantz RA, Park H, Scherubel M. Diagnostic validity of clinical signs in identifying localized diabetic foot ulcer infection. Manuscript in preparation. My standards haven t changed. My budget has. Face it - budgets are shrinking, and they re not increasing any time soon. My Solution - I switched to DermaRite. I save money without sacrificing quality of care. DermaRite performs just like the name brands, is very cost effective, and I don t have to compromise my care plan or quality of life. It s comforting to know I don t have to choose between quality and price point. DermaRite makes it possible to have both. TAKE THE DERMARITE 1 WEEK CHALLENGE We re so sure you ll love our skin and wound care products that we ll supply you with a weeks worth of 2 products absolutely free. Give us your toughest residents - really put us to the test. See what you think after a week. We believe you ll agree that DermaRitee beats the competition. Visit us on the web at for more details. DermaRite Is The Cost Effective Alternative To Name Brands January 2007 Vol. 53 Issue 1 51