Evidence-based Diagnosis: A Workshop on Evaluating and Using Medical Tests Small Group 2: Thursday, June 9, 4:00 6:00. Problems Without Answers

Transcription

1 Evidence-based Diagnosis: A Workshop on Evaluating and Using Medical Tests Small Group 2: Thursday, June 9, 4:00 6:00 Problems Without Answers A B C D E F G H I Objectives: Incorporation Bias Verification Bias Double Gold Standard Bias Spectrum Bias Distinction between Prognostic and Diagnostic Tests Calibration vs. Discrimination in Prognostic Risk Estimates Estimating the Value of a Prognostic Tests Combining tests and test non-independence Logistic regression, Recursive Partitioning, and Validation A. Incorporation Bias B. Verification Bias C. Double Gold Standard Bias D. Spectrum Bias E. Distinction between Prognostic and Diagnostic Tests F. Calibration vs. Discrimination in Prognostic Risk Estimates G. Estimating the Value of a Prognostic Tests Problem Topic Difficulty 1 Ultrasound for Intussusception Intermediate X 2 Ectopic Pregnancy Intermediate X 3 Ultrasound for DVT X X X X 4 ABCD2 Score Intermediate X 5 Revisit SG-1, Problem 6. Serum Lactate for Sepsis Intermediate X 6 Coronary Artery Calcium Intermediate X 7 Tumor Marker Intermediate X X 8 CT and Ultrasound for Appendicitis in Children Intermediate X 9 Revisit SG-1, Problem 5, Part c. Wells score and D-Dimer for PE Intermediate X 10 Revisit SG-1, Problem 8, Part e. CT Brain and Lumbar Puncture for SAH Beginner X H. Test non-independence I. Logistic regression, Recursive Partitioning, and Validation 1

2 1. Ultrasound for Intussusception (Same as EBD Problem 5.4) The abstract of the study of ultrasound to diagnose intussusception discussed in Chapter 5 is excerpted below.(1) Eshed I, Gorenstein A, Serour F, Witzling M. Intussusception in children: can we rely on screening sonography performed by junior residents? Pediatr Radiol Feb;34(2):134-7 BACKGROUND: Ultrasonography (US) is an important tool in the screening and diagnosis of patients with suspected intussusception. MATERIALS AND METHODS: Between January 1999 and February 2003, 151 patients with suspected intussusception underwent screening US. The mean age of the patients was 13.8 months. RESULTS: Sixty-five patients had both US and air enema. Forty-four patients had a positive US result; 37 (84%) were true positive and 7 (16%) were false positive. Twenty-one patients had a negative US result; 18 (86%) were true negative and 3 (14%) were false negative. Eighty-six patients [with negative ultrasound scans] underwent screening US only and were then kept under observation in the emergency room. They were all diagnosed as having a non-surgical condition [i.e., as true negatives]. The total accuracy rate was 93%, sensitivity was 84%, specificity was 97%, positive predictive value was 93% and negative predictive value was 94%.. (For the questions below assume that the statement, They were all diagnosed as having a nonsurgical condition, means that none of the 86 patients who only had a negative screening ultrasound and clinical follow-up (but no air enema) were felt to have an intussusception. Also assume that if air enema was performed at all, it was performed immediately after the ultrasound. Parts a through c review material covered in chapter 3.) a) Create a 2x2 table that summarizes the results of the study. (You can check your answer in box 5.1) b) Check the authors calculations of sensitivity, specificity and accuracy of ultrasound for diagnosis of intussusception in this study. c) Can positive and/or negative predictive value be estimated from a study with this sampling scheme? If so, what are they? If not, why not? d) The authors did not do air enemas on all of the children; in some they just watched them. Name the bias this could cause (using the terminology from this text). 2

3 e) Assume that intussusception NEVER resolves spontaneously, i.e. that nobody who would have had a positive enema (if one were done) would ever have negative clinical follow-up. Also assume no new cases of intussusception develop after the air enema. What would be the effect of the bias you named above on estimates of sensitivity and specificity of ultrasound from this study? f.) Now repeat part (e) assuming that intussusception DOES sometimes spontaneously resolve, i.e. that some of those with negative clinical follow-up would have had a positive enema, if one were done. (Maintain the assumption that no new cases develop.) g. Now repeat part (e), only this time assume that intussusception can develop during a short follow-up period after the enema has been done. 2. Ectopic Pregnancy (Same as EBD Problem 5.3) Women presenting to the emergency department with abdominal pain and a positive pregnancy test may have an ectopic pregnancy (about 10%), an abnormal intrauterine pregnancy (about 30%) or a normal intrauterine pregnancy (about 60%). An excerpt from the abstract of a study on this topic is reprinted below (2): " The objectives of this study were to determine the optimal cutoff value and utility of a single serum beta human chorionic gonadotropin hormone (HCG) level in assessing the likelihood of ectopic pregnancy. A retrospective chart review was performed at an urban county hospital. The optimal cutoff value was determined by comparing all available patients diagnosed with ectopic pregnancy and patients diagnosed with threatened abortion in the Emergency Department who subsequently delivered a baby at the same hospital. The patients diagnosed with threatened abortion.. who subsequently delivered a baby at the same hospital were patients who presented with abdominal pain but ultimately turned out to have a normal intrauterine pregnancy. These patients tend to have higher HCGs than patients with abnormal intrauterine (non-ectopic) pregnancies. No women with abnormal intrauterine pregnancies were included in the non-ectopic study sample. The authors found the sensitivity of an HCG <40,000 miu/ml for ectopic pregnancy was 99%, and the specificity was 85%. (That is, 85% of the women who subsequently delivered a baby at the same hospital had an HCG 40,000 miu/ml.) a.) Is spectrum bias a potential problem in this study? Why or why not? b.) Is the sensitivity estimate too high, too low, or about right? c.) Is the specificity estimate too high, too low, or about right? 3

4 3. Ultrasound for DVT Emergency physician-performed bedside compression ultrasound for deep venous thrombosis (DVT) has been touted as a very sensitive test to guide further workup and anticoagulation therapy. a) If it s a very sensitive test, it should be used as follows: i) If the bedside test shows a DVT, the patient should be treated with anticoagulation ii) If the bedside test is normal, the patient can be sent home without further treatment or testing Two studies (3, 4) of a combined 146 patients showed perfect (100%) sensitivity and high specificity (pooled specificity = 95%) for ED resident-performed bedside compression ultrasound relative to a gold-standard of color-flow duplex ultrasound performed by radiologists blinded to the bedside ultrasound result.. Both these studies used convenience sampling; the pooled prevalence of DVT was 40%. A third similar study (5)of 183 patients showed much lower sensitivity (70%) and specificity (89%) relative to the same gold standard. This study used consecutive sampling and the prevalence of DVT was 15% One possible explanation for the higher accuracy of the bedside ultrasound in the first two studies is that the individuals performing the ultrasound were better at it. Although this is possible, assume that the people performing the index test had the same level of expertise. b) Explain why sensitivity, specificity, and prevalence were all higher in two studies using convenience sampling. Choose from incorporation bias, verification bias, double gold standard bias, and spectrum bias. Explain your answer. 4

5 The study(5) that used consecutive sampling also stratified results by clinical risk of DVT as follows: Clinical Risk of DVT Low Moderate Bedside Ultrasound Reference Standard DVT(+) DVT(+) DVT( ) Subtotal Negative % Positive % Negative % Positive % Negative % High Positive % Total % c) Based on these results, what is the probability of DVT in a patient with both a low clinical risk and a negative scan? d) Use Newman s Rule of 3,5,7, to calculate a 95% upper bound on your answer to (c). We used the table above to calculate the specificity of the bedside ultrasound at each of the 3 clinical risk levels. Clinical Risk of DVT Bedside Specificity Low 0.94 Moderate 0.84 High 0.77 e) Why does the specificity go down as the clinical risk goes up? (Hint: In order to do the bedside compression ultrasound, you can t help but interview and examine the patient subconsciously developing your own clinical risk assessment.) 5

6 4. ABCD2 Score The ABCD2 Score was developed to stratify risk of stroke in patients after a transient ischemic attack (TIA).(6) For your information, here is how the ABCD2 score is calculated. Risk Factor Points Age 60 years 1 Blood Pressure SBP 140 or DBP 90 1 Clincal features of TIA Unilateral weakness (with or without speech impairment) 2 Speech impairment without unilateral weakness 1 Duration TIA duration 60 minutes 2 TIA duration minutes 1 Diabetes Diabetes diagnosed by a physician 1 Total ABCD2 Score 0 7 Score % of TIA Patients 2-day Stroke Risk 90-Day Stroke Risk % 1.0% 3.1% % 4.1% 9.8% % 8.1% 17.8% One of the main reasons for hospitalizing a patient after TIA is to enable treatment with thrombolytics if the patient has a subsequent stroke. a) If you are willing to admit 25 patients to the hospital for 2 days in order to avoid discharging one who goes home to have a stroke. What is your ABCD2 score cutoff for hospitalization? 6

7 b) The above table of 2-day stroke risks can be converted into an ROC table and an ROC curve. Without doing any calculations, what do you expect the AUROC to be? i) < 0.5 ii) iii) iv) We will convert the table of 2-day risks above into an ROC table and calculate the area under it. First, order the results from most to least abnormal: Score % of TIA Patients 2-day Stroke Risk % 8.10% % 4.10% % 1.00% Next, calculate the individual cell percentages. Score D+ D- % of TIA Patients % 19.30% 21% % 43.16% 45% % 33.66% 34% 3.89% 96.11% % Then, calculate the column percentages. Score D+ D % 20.08% % 44.90% % 35.02% % % 7

8 Finally, change them to cumulative percentages. Score D+ D % 20.08% % 64.98% % % c) Use the above ROC Table to plot the ROC curve on the grid below. Sensitivity 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100 % 1 - Specificity 5. Revisit SG-1, Problem 6. Serum Lactate for Sepsis 6. Coronary Artery Calcium (Same as EBD Problem 7.4) Greenland et al(7) recently compared the Framingham Risk Score (FRS), obtained from history and physical examination and lipid levels, with a Coronary Artery Calcium Score (CACS) obtained from CT scanning in 1461 asymptomatic adults at least 45 years old. The FRS is an estimate of the 10-year risk of nonfatal myocardial infarction or death. The authors found that the CACS was predictive of this combined outcome among those with a FRS more than 10%, but not in those with an FRS less than 10%, and they recommended against doing CACS when the FRS was less than 10%. In a letter to the editor, Pletcher et al(8) wrote: 8

9 "In fact, such an interaction would be difficult to detect, and this study adds little evidence, given the low number of persons in the study with an FRS < 10% (n=98) and the low number of events in this subgroup (n=1)." a) Do you think this lack of power affects the conclusion that CACS is not indicated in this low risk group? b) Pletcher et al also point out that the FRS was less predictive of events in the Greenland et al study, compared with previous studies, and postulated that this could occur if treatment decisions based on the FRS blunted its predictive ability. What could the authors do to address this possibility? Tumor Markers 7. (Same as EBD Problem7.6) TP53 is the gene for tumor-suppressor protein p53. In a multi-center 7-year prospective cohort study, disruptive TP53 mutations in tumor DNA (i.e., mutations leading to loss of function of p53) were associated with reduced survival after surgical resection in patients with squamouscell cancer of the head and neck. (9). a. Of the 420 subjects, 232 had died by the end of the follow-up period. Of these, 121 died from head and neck cancer, 62 from other causes, and 49 from unknown causes. The authors used overall survival as the outcome for all analyses. How would the use of overall (vs. cause-specific) survival affect the results? b. How else could the authors have handled the subjects who died of other and unknown causes? One question that arises for genetic tests is how much new information they provide. For example, if disruptive TP53 mutations worsened prognosis by leading to more advanced stage at presentation, much of the prognostic information from TP53 might be captured from stage at presentation. In fact, in this study, the nodal stage at presentation was highly predictive of survival. 9

10 Bivariate (just one variable plus the outcome) and multivariate hazard ratios for nodal stage (N1-N3 vs N0 or NX) and TP53 (disruptive mutation vs. no mutation) are shown in the table below. Prognostic Factor HR (95% CI) Bivariate Multivariate Nodal Stage N1-N3 2.0 ( ) 2.4 ( ) Disruptive TP53 mutation 1.7 ( ) 1.7 ( ) c) What can you conclude about whether the TP53 gene provides new information about prognosis in head and neck cancer patients? d) Assuming the hazard ratios reported in this study are valid and generalizable, what else would you need to know in order to decide whether to order this test on your patients? 8. CT and Ultrasound for Appendicitis in Children Kaiser et al (10) did a randomized study to evaluate computed tomography and ultrasound as diagnostic tools for acute appendicitis in children. 600 children with high clinical suspicion for acute appendicitis were enrolled; 283 were randomized to undergo US imaging only, while 317 children had both US and CT imaging (US was always done prior to CT scan) a) Use the results summarized in "Table 4" below to create two 2 2 tables to calculate the positive likelihood ratios for CT and US results alone. 10

11 b) If your patient s prior probability of acute appendicitis was 10%, what would be the posterior probability, given positive (both) US and CT scan results? Assume that the two tests are independent. c) Now create a 4 2 table like Table 8.2 and calculate the likelihood ratios for each possible US and CT result combination. Re-calculate the posterior probability for a patient with a prior probability of 10% and positive results on both tests. d. Are the two tests independent? Please explain both how you know and suggest a possible biologic reason for your answer. 9. Revisit SG-1, Problem 5, Part c. Wells score and D-Dimer for PE. 10. Revisit SG-1, Problem 8, Part e. CT Brain and Lumbar Puncture for SAH 11

12 References 1. Eshed I, Gorenstein A, Serour F, Witzling M. Intussusception in children: can we rely on screening sonography performed by junior residents? Pediatr Radiol Feb;34(2): Marill KA, Ingmire TE, Nelson BK. Utility of a single beta HCG measurement to evaluate for absence of ectopic pregnancy. J Emerg Med May-Jun;17(3): Farahmand S, Farnia M, Shahriaran S, Khashayar P. The accuracy of limited B-mode compression technique in diagnosing deep venous thrombosis in lower extremities. Am J Emerg Med Jan Jang T, Docherty M, Aubin C, Polites G. Resident-performed compression ultrasonography for the detection of proximal deep vein thrombosis: fast and accurate. Acad Emerg Med Mar;11(3): Kline JA, O'Malley PM, Tayal VS, Snead GR, Mitchell AM. Emergency clinicianperformed compression ultrasonography for deep venous thrombosis of the lower extremity. Ann Emerg Med Oct;52(4): Johnston SC, Rothwell PM, Nguyen-Huynh MN, Giles MF, Elkins JS, Bernstein AL, et al. Validation and refinement of scores to predict very early stroke risk after transient ischaemic attack. Lancet Jan 27;369(9558): Greenland P, LaBree L, Azen SP, Doherty TM, Detrano RC. Coronary artery calcium score combined with Framingham score for risk prediction in asymptomatic individuals. Jama Jan 14;291(2): Pletcher MJ, Tice JA, Pignone M. Use of coronary calcification scores to predict coronary heart disease. Jama Apr 21;291(15):1831-2; author reply Poeta ML, Manola J, Goldwasser MA, Forastiere A, Benoit N, Califano JA, et al. TP53 mutations and survival in squamous-cell carcinoma of the head and neck. N Engl J Med Dec 20;357(25): Kaiser S, Frenckner B, Jorulf HK. Suspected appendicitis in children: US and CT--a prospective randomized study. Radiology Jun;223(3):