Dr. Ruby Senie is Professor of Clinical Epidemiology and Sociomedical Sciences in the Mailman School of Public Health.

Size: px

Start display at page:

Download "Dr. Ruby Senie is Professor of Clinical Epidemiology and Sociomedical Sciences in the Mailman School of Public Health."

Arnold Burns
5 years ago
Views:

1 Screening Introduction Screening is the examination of asymptomatic people in order to classify them as likely or unlikely to have a particular disease. Epidemiologists study the screening of diseases for two reasons: the first is to determine the validity of a screening test, and the second is to evaluate the effectiveness of a screening test in a population. This exercise will lead you through the design and analysis of a study to determine the validity of a screening test in Epiville. At the conclusion of this exercise, you will read an article and answer questions about a study designed to evaluate the effectiveness of a screening test in a population. Faculty Highlight: Ruby Senie Dr. Ruby Senie is Professor of Clinical Epidemiology and Sociomedical Sciences in the Mailman School of Public Health. Dr. Senie has devoted more than 20 years to research on breast cancer risk and prognostic factors. Dr. Senie is principal investigator of an NCI-funded project titled the "Metropolitan New York Registry of families at increased risk of breast cancer." Dr. Senie also has a special interest in racial/ethnic disparities associated with detection and prognosis of breast and other cancers, and is principal investigator of the NY component of the NCI-funded Asian American Network for Cancer Awareness Research and Training, which has supported outreach to community-based organizations to provide educational programs and opportunities for cancer screening. Read more about Dr. Senie's work: 1. Islam, N. Kwon, SC, Senie R, Kathuria, N. Breast and cervical cancer screening among South Asian women in New York City. Journal of Immigrant and Minority Health. July (3): Senie, R. Santella, R. Ahsan, H. The metropolitan New York Registry and CFRBCS. Unique resources for breast cancer research. Annals of Epidemiology. October (7):462. Step 1: Learning Objectives Describe the characteristics of diseases appropriate for screening. Distinguish between primary, secondary, and tertiary prevention. Calculate and interpret sensitivity, specificity, positive predictive value, and negative predictive value. Learn the relationship between sensitivity and specificity when determining the cutpoint ( criterion of positivity ) of a screening test. Understand the relationship between disease prevalence and positive predictive value. Define length-time bias, lead-time bias, and volunteer bias. Step 2: Student Role - Your Plan of Action A goal of the Epiville Department of Health (EDOH) is to reduce the morbidity caused by the neurological disorder Susser Syndrome (SS). A drug treatment has been found to be successful in alleviating the symptoms of Susser Syndrome in patients that already have clinical presentation and diagnosis of the disease. Now there is reason to 1

2 believe that if the treatment is begun before clinical symptoms first appear, the treatment can prevent irreversible neurological damage and slow down the progression of the disease accompanied by the worsening of the neurological symptoms. In order to start treatment before the clinical symptoms of Susser Syndrome appear, it is necessary to implement a screening test to identify persons in the detectable pre-clinical phase. A laboratory test called SussStat has been developed that measures a biomarker for asymptomatic neurological damage. The director of the EDOH, Dr. Melissa Zapp, suggests that your next project should be to conduct a study to see how well this new test identifies persons with Susser Syndrome prior to the symptomatic stage. Step 3: Study Design Before you can begin your study, you must write a proposal to secure funding for the project. In the proposal, you must describe how a screening program would fit into the prevention activities of the EDOH. 1. A screening program for Susser Syndrome would fall under what type of prevention activity? a. Primary Incorrect. The disease process would have already been initiated among the individuals identified by the screening program. Primary prevention is characterized by individual or community efforts to stop the disease process from initiating (Aschengrau p. 404). b. Secondary Correct. The rationale for the screening program is that it will identify asymptomatic persons with Susser Syndrome and provide early treatment so that the expression of the disease will be less severe, which is characteristic of secondary prevention (Aschengrau p. 405). c. Tertiary Incorrect. A tertiary prevention program is implemented after a clinical diagnosis has been confirmed to alleviate suffering and minimize complications of the disease. The screening program you hope to design for Susser Syndrome will identify persons who are asymptomatic and have not yet been diagnosed. As luck would have it, your study is funded by a generous grant from DoseEmAll Pharmaceuticals. You plan to recruit 1000 subjects with no symptoms of Susser Syndrome. Since you want to ensure that enough individuals in your study will develop Susser Syndrome, you decide to recruit workers from Glop Industries, where an agent suspected to cause Susser Syndrome is manufactured. The EDOH has conducted a study which demonstrated a relationship between exposure to chemicals involved in the production of Superclean and a higher risk of Susser Syndrome. At enrollment, you collect a blood sample from each participating subject to perform the new SussStat test. One year later, you have each subject return to undergo thorough clinical assessment for SS, using the gold standard of diagnosis set by the EDOH. Step 4: Data Collection The SussStat procedure tests for irregularities in the DNA, called DNA adducts, that are formed when a chemical contained in Glop Indutstry s Superclean product bonds to DNA. It is this process which researchers suspect may lead to Susser Syndrome. 2

3 You receive a spreadsheet with the DNA adduct data from the laboratory, and now must use these data to categorize each subject into the categories of Test Positive and Test Negative for Susser Syndrome. The manufacturer of SussStat suggests that levels of DNA adducts lie on a continuum. The manufacturer suggests dichotomizing SussStat level at 12 x Those who test at 12 x 10-8 or higher will be considered screened positive, and everyone else will be considered screened negative. You use this manufacturer-suggested cutpoint to categorize your subjects into Test Positive and Test Negative for SS. When each subject returns for the one-year clinic visit, the clinician evaluates them and uses the diagnostic criteria put forth by the EDOH to categorize each subject into Disease Positive and Disease Negative. After receiving these data from the study physician, you are now ready to compare the SussStat results to the Clinician Gold Standard! In order to prepare the data for analysis, you classify each subject into a 2x2 table: Gold Standard SussStat Positive Negative [ Click here for the Interactive Exercise. ] Step 5: Data Analysis Gold Standard SussStat Positive Negative You are now ready to compute some statistics that will tell you how well SussStat performs compared to the gold standard of clinical diagnosis. Sensitivity and specificity are commonly used measures of the validity of a screening of a test (Aschengrau p. 410). Validity is the ability of a test to correctly categorize persons into their true disease status. The measures of positive predictive value (PPV) and negative predictive value (NPV) describe how well a positive screening test result predicts presence or absence of a disease in a particular population. The PPV and NPV are measures of a screening program s feasibility (Aschengrau p. 414). 2. Calculate the sensitivity, specificity, PPV, and NPV. a. Sensitivity =

4 b. Specificity = 0.96 c. PPV = 0.69 d. NPV = Which is the best interpretation of the sensitivity of SussStat? a. The probability that SussStat correctly categorized an individual as not having Susser Syndrome. Incorrect.This is an interpretation of specificity. b. Of those who tested positive on the SussStat test, the percent of persons that developed Susser Syndrome. Incorrect. This is the PPV of the test, PPV = TP/(TP+FP). The denominator of sensitivity is not those who tested positive, but rather those who truly have the disease. c. The probability that SussStat correctly categorized an individual as having Susser Syndrome. Correct. Sensitivity = TP/(TP+FN). 4. Which is the best interpretation of the specificity of SussStat? a. The probability of obtaining a false negative if you truly do not have Susser Syndrome. Incorrect. It is the probability that SussStat correctly categorized an individual as having Susser Syndrome, or the probability of obtaining a true negative. b. Of those who never develop Susser Syndrome, the percent that tested negative on the SussStat essay. Correct. Specificity= TN / TN + FP. c. Of those who tested negative of the SussStat screening test, the percent that actually developed Susser Sydrome. Incorrect. Specificity= TN / TN + FP. 5. How could you interpret the positive predictive value (PPV)? a. Of those persons who developed Susser Syndrome, the percent that tested positive on the SussStat test. Incorrect. The is the interpretation of sensitivity. b. The probability of not developing Susser Syndrome given a negative SussStat test result. Incorrect. This is the NPV. 4

5 c. Of those who tested positive, the percent that develop Susser Syndrome. Correct. PPV = TP/(TP+FP). You are concerned that there are too many false positives when the manufacturer-suggested cutpoint of DNA adducts is used to define the pre-clinical SS and you want to see if you can reduce the number by changing the cutpoint, or criterion of positivity, of the SussStat test. Click here for an interactive exercise demonstrating how raising or lowering the cutpoint changes the measures you calculated for SussStat. [ Click here for the Interactive Exercise. ] 6. What is a consequence if SussStat has a low sensitivity? a. You will miss the opportunity to correctly diagnose and treat people with Susser Syndrome. Correct. A low sensitivity means a lot of false negatives. b. You will treat too many people who don t actually have the disease, which is costly and stressful to the subjects and also puts people at risk from possible side effects. Incorrect. This is the consequence of a low specificity. c. Both a and b. Incorrect. Only one of the answers listed is correct. You are happy with the screening test s ability to identify Susser Syndrome, but you are now considering to what groups in Epiville you should target your screening program. 7. Using the sensitivity and specificity measures you calculated in Question 2 above, calculate what the PPV would be if you screen the total population of Epiville where you estimate the prevalence of SS to be only 1%. (hint: draw a new 2x2 table and write the new column totals for the Clinician Gold Standard using the new true population prevalence of 1%, given a total N=1000 Then use the specificity and sensitivity proportions you calculated previously to fill in the rest of the 2x2 table.) NOTE: ONLY USE THE SENSITIVITY AND SPECIFICITY VALUES ROUNDED TO 2 DECIMAL PLACES THAT APPEAR IN THE ANSWER TO QUESTION 2. Answer = 0.18 You make note of the fact that as the prevalence of the disease decreases, the PPV of the screening test decreases. You recommend to the EDOH that a screening program be introduced among workers of the Glop Industries since they have a higher prevalence of the disease, and therefore SussStat will be most effective in that group because it will detect a larger proportion of actual cases among individuals with positive results (Aschengrau p. 415). The measures you calculated above describe the validity of the SussStat test. In contrast, reliability is the ability of a test to give the same result on repeated testing, i.e., consistency. (Aschengrau p. 410) Reliability can also be computed to describe the extent to which two tests agree with each other. A common measure of reliability is the kappa statistic. 5

6 Intellectually curious? Learn how to calculate Kappa. How to Calculate Kappa What if the clinical diagnosis of Susser Syndrome was not considered a gold standard? In this case we cannot measure validity with sensitivity and specificity and we would instead measure reliability by calculating kappa. Kappa tells us the extent to which SussStat and the clinician agree with each other beyond what you might expect to see based on chance alone. The formula for Kappa is: We calculate observed agreement by calculating the frequency with which the two measurements agreed: We calculate expected agreement by first calculating the expected values of the cells in the 2x2 table using the marginal frequencies, then using those cell numbers to calculate the frequency with which the two measurements are expected to agree: How to calculate expected cell frequencies: Clinical Diagnosis SussStat Positive (a+b)(a+c)/n (a+b)(b+d)/n a+b Negative (c+d)(a+c)/n (c+d)(b+d)/n c+d a+c b+d N When two measurements agree by chance only, kappa = 0. When the two measurements agree perfectly, kappa = 1. Say instead of considering the Clinician rating of Susser Syndrome a gold standard, you wanted to see how well the lab test agreed with the clinician s categorization. Using the same 2x2 table as you used in Question 2, calculate Kappa. Scroll down for the answer. ANSWER New table showing expected values: 6

7 Clinical Diagnosis SussStat Positive (130)(100)/(1000) = 13 (130)(900)/(1000) = Negative (870)(100)/(1000) = 87 (870)(900)/(1000) = Observed agreement = ( ) / 1000 = Expected agreement = ( ) / 1000 = Kappa = ( ) / ( ) = Interpretation: The SussStat test and the clinician had a probability of agreeing who had SusserSyndrome beyond chance of (good agreement). Step 6: Discussion Questions 1. You have probably had the experience of being screened many times (e.g., cholesterol test, glucose level test, blood pressure measurement). What is more important information from a PATIENT s point of view (PPV/NPV vs. Sensitivity/Specificity), and what is more important information from a PHYSICIAN s point of view (PPV/NPV vs. Sensitivity/Specificity). 2. Think of hypothetical screening programs for diseases of interest to you; find one example of a disease where high sensitivity would be more important, and one example where high specificity would be more important. 3. What are some strategies to reduce lead time bias in studies evaluating the efficacy of screening programs? Question for the intellectually curious Observational studies of screening programs often compare the mortality and survival time in individuals who choose to be screened to individuals who do not choose to be screened. What are some potential biases that could arise in observational studies of screening programs? 7

Screening for Disease

Screening for Disease An Ounce of Prevention is Worth a Pound of Cure. Actually, an ounce of prevention is better than a pound of cure, but if prevention hasn t been effective, perhaps early identification