EPI Case Study 2: Reliability, Validity, and Tests of Agreement in M. Tuberculosis Time to Complete Exercise: 30 minutes LEARNING OBJECTIVES At the completion of this Case Study, participants should be able to: Know how to construct 2-by-2 tables Compare screening techniques using the Kappa Statistic Distinguish between reliability and validity Identify groups at high risk for TB infection Identify methods used to screen for TB infection ASPH EPIDEMIOLOGY COMPETENCIES ADDRESSED C.2. Identify the principles and limitations of public health screening programs C. 3. Describe a public health problem in terms of magnitude, person, place, and time C. 6. Apply the basic terminology and definitions of epidemiology C. 7. Calculate basic epidemiologic measures C. 9. Draw appropriate inference from epidemiologic data C. 10 Evaluate the strengths and limitations of epidemiologic reports Please evaluate this material by clicking here: http://www.zoomerang.com/survey/?p=web229tew5zpt4 This material was developed by the staff at the Global Tuberculosis Institute (GTBI), one of four Regional Training and Medical Consultation Centers funded by the Centers for Disease Control and Prevention. It is published for learning purposes only. Permission to reprint excerpts from other sources was granted. Case study author(s) name and position: George Khalil, MPH (work done as MPH candidate) Marian R. Passannante, PhD Associate Professor, University of Medicine & Dentistry of New Jersey, New Jersey Medical School and School of Public Health and Epidemiologist, NJMS, GTBI For further information please contact: New Jersey Medical School Global Tuberculosis Institute (GTBI) 225 Warren Street P.O. Box 1709 Newark, NJ 07101-1709 or by phone at 973-972-0979 Suggested citation: New Jersey Medical School Global Tuberculosis Institute. /Incorporating Tuberculosis into Public Health Core Curriculum./ 2009: Epidemiology Case Study 2: Reliability, Validity, and Tests of Agreement in M. Tuberculosis INSTRUCTOR S GUIDE Version 1.0. Date Last Modified: November 16, 2009 1
Introduction In the United States, no vaccines are given to prevent the transmission of tuberculosis (TB) due to their current lack of efficacy. 1 Because scientists are still working to create a more efficient vaccine, the generally accepted approach to TB control relies on screening, surveillance, and contact investigations. 2 Identifying and treating persons with latent TB infection (LTBI) at high risk for developing TB is part of the current TB elimination strategy in the United States. is essential in the identification phase of this strategy (2). The most common method of screening is with purified protein derivative (PPD), a type of tuberculin skin test (). Infection with Mycobacterium tuberculosis (M. tuberculosis) produces a delayed-type hypersensitivity reaction to certain antigenic components of the organism that are contained in extracts of culture filtrates called tuberculins. The skin test is injected in the forearm and read by a trained clinician after 48 to 72 hours. The size of the reaction is measured in millimeters and interpreted according to its size, using cutoff points corresponding to the degree of induration. 3 Another type of screening test is called interferon gamma assay (IGA), which measures the production of the cellular interferon gamma by T-cells after sensitization with M. tuberculosis antigens. Although researchers believe that interferon tests are preferable to the s, they are much more expensive. 4 This exercise will be based on the following study. Gerald H. Mazurek; Philip A. LoBue; Charles L. Daley; John Bernardo; Alfred A. Lardizabal; William R. Bishai; Michael F. Iademarco; James S. Rothel, Comparison of a Whole-Blood Interferon Assay With Tuberculin Skin Testing for Detecting Latent Mycobacterium tuberculosis Infection JAMA. 2001;286:1740-1747. 5 Sections of this document have been reprinted with permission of the journal (permission pending). Context Identifying persons with latent tuberculosis infection (LTBI) is crucial to the goal of TB elimination. A whole-blood interferon (IFN- ) assay, the Quanti-FERON- TB test, is a promising in vitro diagnostic test for LTBI that has potential advantages over the tuberculin skin test (). Objectives To compare the with the and to identify factors associated with discordance between the tests. Design and Setting Prospective comparison study conducted at 5 university affiliated sites in the United States between March 1, 1998 and June 30, 1999. Participants A total of 1226 adults (mean age, 39 years) with varying risks of Mycobacterium tuberculosis infection or documented or suspected active TB, all of whom underwent both the and the. Main Outcome Measure Level of agreement between the and the. Date Last Modified: November 16, 2009 2
METHODS The study was conducted at 5 sites: Boston University School of Medicine, Mass; Johns Hopkins School of Hygiene and Public Health, Baltimore, Md; University of California at San Francisco; New Jersey Medical School, Newark; and University of California at San Diego, using a common protocol. These sites were randomly coded as A-E in the analysis. Ethical approval for the study was obtained from the institutional review boards at the Centers for Disease Control and Prevention (CDC), which supported the study, and the 5 study sites prior to enrolling any subjects. All participants provided written informed consent. Persons recruited for the study were 18 years or older and included persons requesting a preemployment or preschool enrollment ; persons being screened with a because they were considered to be at high risk for LTBI; persons in whom TB was clinically suspected and who had received fewer than 6 weeks of anti-tb therapy; and persons who previously had active TB, confirmed by a positive culture, and who had completed a course of multidrug anti-tb therapy within the prior 2 years. Subjects were excluded from the study if they self-reported as pregnant or HIVpositive; had a history of severe reaction to tuberculin; were immunocompromised due to leukemia, lymphoma, or Hodgkin disease; or had taken immunosuppressive drugs (eg,orticosteroids,methotrexate, azathioprine) during the preceding 3 months. After providing written informed consent, enrolled persons completed a detailed questionnaire about possible risk factors for exposure to M tuberculosis. Subjects were also asked to indicate results of any prior, whether they had received BCG vaccination, details of any contact with a person having TB, any risk factors associated with HIV infection, and whether they had any other medical conditions. When applicable, data were also collected from medical records about findings on chest radiography, results and dates of cultures for mycobacteria, and details of treatment for TB. Data were collected on subjects age, race, place of birth, residence outside of the United States, and residence or work (paid or unpaid) in a health care setting, prison, homeless shelter, drug rehabilitation unit, or other group housing. Based on responses to the questionnaire and a review of available medical records, persons were categorized into 4 study groups: (1) low-risk for LTBI, subjects receiving preemployment or preschool enrollment with no identified risks for LTBI; (2) highrisk for LTBI, asymptomatic subjects with risk of LTBI including contacts of patients with TB; persons from countries where tuberculosis is prevalent (>10 cases per 100000 population) 26 ; intravenous drug users; persons who lived, worked, or volunteered on a regular basis in a homeless shelter, prison, drug rehabilitation unit, hospital, or nursing home; and persons determined to be at increased risk by prior local investigations;(3) TB suspects, subjects being evaluated for active TB who had received fewer than 6 weeks of anti-tb therapy; and (4) culture-confirmed TB, subjects who completed treatment for culture-confirmed TB within the prior 2 years. Date Last Modified: November 16, 2009 3
Persons enrolled during preemployment or preschool enrollment examinations were assigned to group 2 if risk factors for LTBI were identified during questioning. However, to maintain the integrity of group 1 as truly low risk for LTBI, persons considered to be at high-risk for LTBI at enrollment were assigned to group 2 even when risk factors were denied. Reference 26 in Mazurek et al: World Health Organization. Global Tuberculosis Control. Geneva, Switzerland: WHO; 1999.WHO/ CDS/CPC/TB/99.259. For the purposes of this exercise, we will examine the data from Group 1 (low-risk group for TB infection) and Group 2 (high-risk group). Table 1 compares the responses to both and IFN tests for Groups 1 (low risk) and 2 (high risk). Table 1 Response to and IFN- tests in high- and low-risk Groups 5 Question 1 A. For Group 1, create a 2X2 table to test the association between and IFNreadings. Show percent for marginal totals. B. For Group 2, create a 2X2 table to test the association between and IFNreadings. Show percent for marginal totals. Use the following 2x2 tables: A. B.. Positive Negative Positive Negative Date Last Modified: November 16, 2009 4
A. Group 1 Positive 1 1 2 (2.0%) Negative 7 89 96 (98.0%) 8 (8.1%) 90 (91.8%) 98 B. Group 2 Positive 146 79 225 (23.8%) Negative 73 649 722 (76.2%) 219 (23.1%) 728 (76.8%) 947 Question 2 A. For Group 1, calculate an overall percent agreement by and and interpret. B. Do the same for Group 2. Note: Percent agreement can be calculated as (a+d)/(a+b+c+d) x 100 and is called p o (or proportion of agreement observed). A. Group 1 Positive 1 a 1 b 2 (2.0%) Negative 7 c 89 d 96 (98.0%) 8 (8.1%) 90 (91.8%) 98 B. Group 2 Positive 146 a 79 b 225 (23.8%) Negative 73 c 649 d 722 (76.2%) 219 (23.1%) 728 (76.8%) 947 A. p o or % agreement for Group 1 = (1 + 89)/(1+1+7+89) x 100=91.8%; This means that the tests agreed in 91.8% of the screenings. B. p o or % agreement for Group 2 = (146 + 649)/ (146 +79+ 72+649) x 100=83.9%; This means that the tests agreed in 83.9% of the screenings. Date Last Modified: November 16, 2009 5
Question 3 For Groups 1 and 2, create2x2 tables of percent agreement by screening test expected by chance alone and calculate its percent agreement. To do this you can calculate the expected values for cells a and d and then calculate the percent agreement expected by chance using the formula (a expected value + d expected value )/(a+b+c+d) x 100. This is called p e (or proportion of agreement expected). Remember: Expected values= (row total x column total)/grand total Percent agreement expected by chance alone for Group 1 Positive 0.16-2 Negative - 88.16 96 8 90 98 % agreement= (0.16+88.16)/98) X 100=90.1% Percent agreement expected by chance alone for Group 2 Positive 52.03-225 Negative - 555.03 722 219 728 947 % agreement= (52.03+555.03)/947) X 100=64.1% A. For Group 1, the agreement expected (p e ) by chance alone would be 90.1% of all screenings. B. For Group 2, the agreement expected (p e ) by chance alone would be 64.1% of all screenings. Date Last Modified: November 16, 2009 6
Question 4 A. What is the kappa statistic of the 2 tests for Group 1 (low risk for TB)? B. And for Group 2 (high risk for TB)? κ = (percent observed agreement) (percent agreement by chance alone) 100%- (percent agreement expected by chance alone) A. For Group 1, κ= 91.8-90.1= 0.17 100-90.1 B. For Group 2, κ= 83.9 64.1= 0.55 100 64.1 Table 2 6 Source: Understanding interobserver agreement: the kappa statistic. Viera AJ, Garrett JM. Fam Med. 2005;37:362. Permission granted to reprint table. Question 5 Rate the overall reliability of the screening tests. Does prevalence seem to have an impact on the kappa value? The tests have moderate agreement for those at high risk of developing TB, but only slight agreement for low-risk groups. Date Last Modified: November 16, 2009 7
According to Viera and Garret (2005) Kappa is affected by prevalence of the finding under consideration much like predictive values are affected by the prevalence of the disease under consideration. 5 For rare findings, very low values of kappa may not necessarily reflect low rates of overall agreement. 6 ( page 262) Reference 5 in Viera and Garret: Feinstein AR, Cicchetti DV. High agreement but low kappa: I. The problems of two paradoxes. J Clin Epidemiol 1990;43:543-9. Question 6 Is there a limitation in comparing the IFN- assay with the? If so, what is it? There is an inherent problem in comparing the two because as the authors state, Discrepancies encountered may be the result of limitations in the and not in the IFN-gamma assay. 5 (page 1745) Validity and Reliability A useful screening test is both reliable (reproducible or accurate) and valid (precise). In Figure 1 6, assuming the goal is to hit the middle of the target, B and D are reliable (also called reproducible) but only D is both reliable and accurate. Unfortunately, to test for validity the truth has to be known or there has to be a gold standard and in TB screening there is no gold standard. So, in this exercise we are only looking at reliability of two screening tests. Source: Understanding interobserver agreement: the kappa statistic. Viera AJ, Garrett JM. Fam Med. 2005;37:361. Permission granted to reprint Figure 1. Date Last Modified: November 16, 2009 8
Works Cited 1. Development of new vaccines for tuberculosis. MMWR Recomm Rep. 1998;47 (RR-13):1-6. 2. The role of BCG vaccine in the prevention and control of tuberculosis in the United States: A joint statement by the Advisory Council for the Elimination of TB and the Advisory Committee on Immunization Practices. MMWR Recomm Rep. 1996;45 (RR-4):1-18. 3. Ayub A, Yale SH, Reed KD, Nasser RM, Gilbert SR. Outpatient practice management tips. Testing for Latent Tuberculosis. Accessed February 8, 2008. http:www.clinmedres.org/cgi/content/full/2/3/191 4. Madariaga MG, Jalali Z, Swindells S. Clinical utility of interferon gamma assay in the diagnosis of tuberculosis. J Am Board Fam Med. 2007;20:540-547. 5. Mazurek GH, LoBue PA, Daley CL, et al. Comparison of a whole-blood interferon gamma assay with tuberculin skin testing for detecting latent mycobacterium tuberculosis infection. JAMA. 2001;286:1740-1747. 6. Viera J, Garrett JM. Understanding interobserver agreement: the kappa statistic Fam Med. 2005;37:360-363. Date Last Modified: November 16, 2009 9