MODEL DRIFT IN A PREDICTIVE MODEL OF OBSTRUCTIVE SLEEP APNEA

Transcription

1 MODEL DRIFT IN A PREDICTIVE MODEL OF OBSTRUCTIVE SLEEP APNEA Brydon JB Grant, Youngmi Choi, Airani Sathananthan, Ulysses J. Magalang, Gavin B. Grant, and Ali A El-Solh. VA Healthcare System of WNY, The University at Buffalo, ECMC, and Emory University

2 Introduction Model drift has been defined as the deterioration in a model s predictive properties over time It is a well-recognized phenomenon in various branches of science such as meteorology but has received little formal attention in the medical literature. Clinical prediction models are being used increasingly, therefore model drift is something that needs attention.

3 Background Obstructive sleep apnea is a common disorder that causes daytime sleepiness and is associated with hypertension, stroke and coronary arterial disease, and with increased mortality in severe cases.

4 Obstructive Sleep Apnea The syndrome is associated with episodes of reduced or cessation of breathing during sleep. The severity of the condition is determined by the apnea-hypopnea index (AHI), which measures the number of respiratory disturbances/hour during an overnight sleep study Clinical diagnosis by expert physician is inaccurate

5 Severity of Obstructive Sleep Apnea by AHI AHI events/hour Severity <5 Normal 5 - <15 Mild 15 - <30 Moderate > 30 Severe

6 Previous Study (1996 data) Setting: Sleep lab at University affiliated hospital Patients Derivation group: 189 patients. Validation group: 80 patients. Method: Responses to a questionnaire used as predictors, AHI from sleep study as a target. Artificial neural network (ANN) (El-Solh et al, Validity of neural network for sleep apnea. Sleep 22: , 1999)

7 Artificial Neural Network (ANN) Ten models were trained The derivation group was divided into 10 subgroups, nine used for training and the tenth group used for testing during training to prevent overfitting. Architecture and training methods were not altered The final model was selected was the median performance of the 10 models

8 Present Study (2002 data) Setting: Sleep lab at private facility Patients Derivation group of 529 patients Validation group of 200 patients. Method: Responses to self-administered questionnaire used as predictors, AHI from sleep study as the target. Artificial neural network (ANN) Multiple adaptive regression splines (MARS)

9 Both Studies Diagnostic accuracy was measured by the area under the receiver operator characteristic curve with 95% CI range 0.5 (useless test) to 1.0 (perfect test) C-index provides the same information which was calculated by a bootstrap method Three thresholds of disease were used, 5, 15 and 30 events/hour.

10 Artificial Neural Network MODEL Dataset AHI = 5 AHI = 15 AHI = ANN 1996 (n=80) ( ) ( ( )

11 Artificial Neural Network MODEL 1996 ANN 1996 ANN Dataset 1996 (n=80) 2002 (n=200) AHI = 5 AHI = 15 AHI = ( ) ( ( ) ( ) ( ) ( )

12 Artificial Neural Network Summary the accuracy of the ANN has deteriorated over time: model drift

13 Causes of Model Drift # Change Reason 1 Change in disease characteristics Change in virulence of infectious agent 2 Definition of the disease Definition of respiratory events is different Diagnostic methodology Administration of predictive instrument Spectrum of patients referred Use of pressure sensors/ thermocouples Administered vs. selfadministered questionnaire Probable change: increased awareness

14 Causes of Model Drift In this case, all are possible except #1 Retrain the ANN Improvement would suggest causes #2, #3 and #5 predominate (disease definition, diagnostic methodology or patient spectrum) Failure to improve would suggest cause #4 (changes in the predictive instrument)

15 Effect of Retraining ANN MODEL New ANN 1996 ANN Dataset 2002 (n=200) 2002 (n=200) AHI = 5 AHI = 15 AHI = ( ) ( ) ( ) ( ) ( ) ( )

16 Artificial Neural Network (ANN) Summary the accuracy of the ANN was not improved by retraining changes in the predictive instrument are more likely to the the cause of model drift.

17 Are Other Predictive Methods Improve Accuracy? Analytic approach Use multiple adaptive regression splines (MARS) to develop a predictive model

18 ANN vs MARS MODEL New ANN MARS Dataset 2002 (n=200) 2002 (n=200) AHI = 5 AHI = 15 AHI = ( ) ( ) ( ) ( ) ( ) ( )

19 Multiple Adaptive Regression Splines (MARS) Summary the accuracy of MARS was better than ANN MARS used only responses to 3 items ANN used responses to 12 items

20 No Model Drift With MARS MODEL MARS MARS Dataset AHI = 5 AHI = 15 AHI = (n=200) ( ) ( ) ( ) 1996 (n=80) ( ) ( ) ( )

21 Multiple Adaptive Regression Splines (MARS) In this study, the more robust predictor variables were selected by the MARS model ANN models with high predictive accuracy may be more susceptible to model drift

22 Are Highly Predictive Models Susceptible to Model Drift? To test this hypothesis we generated 20 datasets. Developed MARS model for each dataset. Measured the area under the ROC with threshold of 15 events/hr for both the 1994 dataset and 2002 dataset.

23 Model Size and Diagnostic Accuracy With MARS Models 0.9 Area Under ROC Dataset Number of Predictors

24 Model Size and Diagnostic Accuracy With MARS Models Increased number of predictors variables is more variable diagnostic accuracy

25 Model Drift With MARS Models 0.9 Area Under ROC Dataset 2002 Dataset Number of Predictors

26 Model Drift With MARS Models Improved diagnostic accuracy with 1996 compared with 2002 dataset: SE vs SE (p=0.001) The higher accuracy with the 1996 dataset suggests that these patients are easier to distinguish between those with and without sleep apnea than the 2002 dataset: change in patient spectrum

27 Conclusions The ANN using the questionnaire is susceptible to model drift. Recalibration failed to rectify the problem suggesting that changes in the administration of the questionnaire was the most likely cause. The MARS model appeared to be less susceptible to model drift than the ANN

28 Conclusions The better diagnosis accuracy with MARS models on 1996 dataset suggests that the model drift associated with the MARS model is due to a change in the patient spectrum. The results are consistent with less clinically apparent cases of sleep apnea being referred to sleep labs.

29 Conclusions Therefore, in this study model drift may be attributed to two factors: Self-administration of the predictive instrument Change in spectrum of patients referred Clinical predictive models should be tested for model drift.