Predictive Models Michael W. Kattan, Ph.D. Department of Quantitative Health Sciences and Glickman Urologic and Kidney Institute
Treatment for clinically localized prostate cancer Trade off: Substantial risk of immediate, long term, side effects Potential gain in life expectancy and avoidance of progression symptoms
Why do patients need estimates of treatment benefit and harm 1. To make a good treatment decision 2. For legal reasons informed consent 3. To understand why you are better!
Should you pick your patient s treatment, even if he asks you to? No. You are trying to balance quantity and quality of life for the patient without really formally assessing his preferences. Shared decision making provides other benefits. Else, Patient is at increased risk for regret.
Whom/how to treat? Some treatment options: 1. AS 2. ORP+LND 3. ORP+LND+NEO 4. ORP-LND 10. BRACHY 11. BRACHY+NEO 12. CRYO 13. HIFU 5. EBRT 6. EBRT+LNRT 7. EBRT+LNRT+HORM 8. LRP 9. LRP+NEO
Why do you tend to advocate a particular treatment? You think predicted outcomes are better 1. BCR 2. METS 3. DEATH 4. IMPOTENCE 5. INCONTINENCE 6. BOWEL DYSFUNCTION 7. LENGTH OF STAY 8. RETURN TO WORK
The patient needs a Metagram Treatment options Outcomes BCR AS RRP 3 4 5 6 7 8 HIFU METS 3 10 Must tailor the probabilities to the individual patient.
When The Patient Wants A Prediction, What Options Does The Clinician Have? Predict based on knowledge and experience Deny ability to predict at the individual patient level Quote an overall average to all patients Assign the patient to a risk group, i.e. high, intermediate, or low Apply a model
Preoperative Nomogram for Prostate Cancer Recurrence Points 0 10 20 30 40 50 60 70 80 90 100 PSA 0.1 1 2 3 4 6 7 8 9 10 12 16 20 30 45 70 110 T2a T2c T3a Clinical Stage T1c T1ab T2b 2+3 3+ 2 4+? Biopsy Gleason Grade 2+ 2 3+3 3+ 4 Total Points 0 20 40 60 80 100 120 140 160 180 200 60 Month Rec. Free Prob..96.93.9.85.8.7.6.5.4.3.2.1.05 Instructions for Physician: Locate the patient s PSA on the PSA axis. Draw a line straight upwards to the Points axis to determine how many points towards recurrence the patient receives for his PSA. Repeat this process for the Clinical Stage and Biopsy Gleason Sum axes, each time drawing straight upward to the Points axis. Sum the points achieved for each predictor and locate this sum on the Total Points axis. Draw a line straight down to find the patient s probability of remaining recurrence free for 60 months assuming he does not die of another cause first. Instruction to Patient: Mr. X, if we had 100 men exactly like you, we would expect between <predicted percentage from nomogram - 10%> and <predicted percentage + 10%> to remain free of their disease at 5 years following radical prostatectomy, and recurrence after 5 years is very rare. Kattan MW et al: JNCI 1998; 90:766-771. 1997 Michael W. Kattan and Peter T. Scardino
CaPSURE Heterogeneity within Risk Groups Preoperative Nomogram Predicted Probability 1.0 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0.0 Nomogram Values by Prostate Cancer Risk Group Low Intermediate High Risk Group J Urol. 2005 Apr;173(4):1126-31
Making a nomogram Usually a regression model (Cox or logistic) Try machine learning techniques (neural nets, optimized trees like CART) Keep continuous variables continuous but relax linearity assumptions P-values for predictors don t matter No variable selection or univariable screening Bottom line is its predictive accuracy
Patients used in the preoperative recurrence nomogram validation USA Cleveland Clinic 1168 N LSU 583 UCLA 617 USC 1501 Europe Hamburg 1134 Rotterdam 475 Australia Sydney 754 Total 6232 Patients Graefen et al., JCO, 2002.
Predicted vs. Actual Freedom from Recurrence Range of Nomogram Predicted Probabilities Percent Free from Recurrence at 5 Years Predicted Actual N 87 86 2873 64 64 855 39 42 343 14 17 209 Graefen et al., JCO, 2002.
Nomogram Validation by Concordance Index (AUC) Coin Toss Perfect Discrimination 0.5 0.6 0.7 0.8 0.9 1.0 Preoperative Nomogram International Validation 1. Randomly select 2 patients a. One of whom fails (reaches the event of interest) b. The other must survive longer 2. Concordance index is the proportion of these pairs in which patient who fails first also had worse nomogram prediction. Graefen et al., JCO, 2002.
Continuous Models vs. Staging/Grouping Systems Model Comparator CI (M vs C) Preop L/I/H Risk Groups 0.67 vs. 0.64 Preop + IL6/TGFβ1 L/H Risk Groups 0.84 vs. 0.73 Pre XRT L/I/H Risk Groups 0.76 vs. 0.69 Melanoma SLN+ AJCC Stage 0.69 vs. 0.66 Pancreatic Ca AJCC Stage 0.64 vs. 0.56 Gastric Ca AJCC Stage 0.77 vs. 0.75 Breast Ca NPI Groups 0.69 vs. 0.64 Sarcoma CART Groups 0.77 vs. 0.74
When The Patient Wants A Prediction, What Options Does The Clinician Have? Predict based on knowledge and experience Deny ability to predict at the individual patient level Quote an overall average to all patients Assign the patient to a risk group, i.e. high, intermediate, or low Apply a model
Urologists vs. Preoperative Nomogram 10 case descriptions from 1994 MSKCC patients presented to 17 urologists In addition to PSA, biopsy Gleason grades, and clinical stage, urologists were provided with patient age, systematic biopsy details, previous biopsy results, and PSA history. Preoperative nomogram was provided. Urologists were asked to make their own predictions of 5 year progression-free probabilities with or without use of the preoperative nomogram. Concordance indices: Nomogram = 0.67 Urologists = 0.55, p<0.05 Ross P et al., Semin Urol Oncol, 2002.
Nomogram for predicting the likelihood of additional nodal metastases in breast cancer patients with a positive sentinel node biopsy Vanzee K, et al., Ann Surg Oncol., 2003.
Breast Cancer Prediction: 17 Clinicians vs. Nomogram on 33 Patients Nomogram AUC 0.72 Sensitivity: Proportion of women with positive nodes predicted to have positive nodes Clinician AUC 0.54 Specificity: Proportion of women with negative nodes predicted to have negative nodes
ROC Curves Individual Clinicians and Nomogram Sensitivity 0.0 0.2 0.4 0.6 0.8 1.0 Areas 0.75 0.72 Nomogram 0.68 0.65 0.65 0.63 0.59 0.58 0.55 0.55 0.53 0.52 0.50 0.49 0.47 0.43 0.42 0.40 0.0 0.2 0.4 0.6 0.8 1.0 1-Specificity
Biases in Human Prediction Data Acquisition Process Output availability inconsistent wishful thinking selective perception heuristics illusion of control base rate insensitive non-linear response frequency conservative illusory correlation environment data representation sources Feedback recall, overconfident, hindsight bias, chance adapted from Hogarth, 1988
When The Patient Wants A Prediction, What Options Does The Clinician Have? Predict based on knowledge and experience Deny ability to predict at the individual patient level Quote an overall average to all patients Assign the patient to a risk group, i.e. high, intermediate, or low Apply a model
PATIENTS AND METHODS mrna isolated from 79 radical prostatectomy specimen of primary prostate tumors Recurrent patients: 37 Criteria for recurrence: 3 consecutive PSA rises Non-recurrent patients: 42 Criteria for non-recurrence: undetectable PSA 60 months Median follow-up: 72 months (range 59-105) Stephenson A, et al., Cancer, 2005.
NOMOGRAM VS GENE MODEL - Results of Leave-One-Out Cross-Validation - 1.0 Survival Functions GENE EXPRESSION MODEL Survival Functions POST-OPERATIVE NOMOGRAM 1.0.8.6.4.2 Progression-Free Survival.8 GENE.6 REC REC-censored NR NR-censored.4.2 N 0.0 0 12 24 36 48 60 72 84 -censored 0.0 0 12 24 36 48 60 72 84 Months from Prostatectomy (months) Concordance Index: 0.75 0.84 Time from Prostatectomy (months) Stephenson A, et al., Cancer., 2005.
Paclitaxel, i.v. estramustine phosphate, and carboplatin for castrate, metastatic prostate cancer Solit D, et al., Cancer, 2003.
Nomograms for clinical trial design Example: CALGB 90203, preoperative therapy for patients at high risk of failure following surgery for prostate cancer Points 0 10 20 30 40 50 60 70 80 90 100 PSA 0.1 1 2 3 4 6 7 8 9 10 12 16 20 30 45 70 110 T2a T2c T3a Clinical Stage T1c T1ab T2b 2+3 3+ 2 4+? Biopsy Gleason Grade 2+ 2 3+3 3+ 4 Total Points 0 20 40 60 80 100 120 140 160 180 200 60 Month Rec. Free Prob..96.93.9.85.8.7.6.5.4.3.2.1.05 < 60% Eastham et al., Urology, 2004.
Software to facilitate real-time predictions Software for the Palm Pilot, PocketPC, and Windows Desktop Computers Software is free from http://www.nomograms.org or www.clinicriskcalculators.org Individual predictions should have a family of models behind them.
Conclusions Your patients need, and deserve, predictions of treatment outcomes. Formulas, such as those behind nomograms and risk calculators, are the most accurate choice for predictions because they are tailored to the patient. Nomogram predictions assist in: Patient counseling Shared decision-making Informed consent Clinical trial design and analysis Free software facilitates the computations.