The Latest Technology from CareFusion Contents 1 Introduction... 2 1.1 Overview... 2 1.2 Scope... 2 2.1 Input Recordings... 2 2.2 Automatic Analysis... 3 2.3 Data Mining... 3 3 Results... 4 3.1 AHI comparison... 4 3.2 ODI Comparison... 6 4 Comparison on using automatic vs. manual scoring.7 4.1 AHI threshold of 10... 8 5 Summary... 9 Introduction 1.1 Overview Analysis comparison is made to validate the automatic scoring performed by the Noxturnal application compared to manual scoring of full PSG data. The compares focused on the comparison of the Apnea Hypopnea Index (AHI) values and the Oxygen Desaturation Index (ODI) values. 1.2 Scope This document validates the quality of the automatic analysis performed by the Noxturnal software application.
2 Methodology 2.1 Input Recordings The input recordings consisted of 1134 Polysomnography (PSG) recordings, collected using the Embla N7000 recorder. The input recordings had all gone through the process of being manually scored by Sleep Technicians with RPSGT certification, and then reviewed by a Physician. The manually scored events included Sleep Stages and Respiratory events such as Apneas, Hypopneas and Desaturation. Sleep Technicians base their scoring on the AASM scoring standard. In 2007, the AASM released an updated standard on which the Noxturnal Automatic Analysis bases its rules. The rules for all of the manual scorings, however, were based on the older AASM standard and therefore had discrepancies compared to the 2007 standard. The most notable discrepancy was in the scoring of Hypopneas. The 2007 AASM standard states that a Hypopnea should be scored where there is a drop of 30% from the baseline followed by a 4% Desaturation from pre-event baseline whereas in the manual scoring, the older definition was used where a Hypopnea had to be followed by a 4% Desaturation and/or an electroencephalographic arousal. To minimize the effects of the discrepancy in scoring rules, each of the input recordings were pre-processed in which all manually scored Hypopneas that were not consistent with the AASM 2007 standard (i.e. were not followed by a Desaturation event), were removed. The full pre-screening and pre-processing of the set of input recordings was as follows: 1. Include only those recordings that have an estimated Sleep Duration of between 4 and 10 hours 2. Exclude recordings where any of the input traces, necessary for the automatic analysis, are missing or where there is an artifact or movement scored in more than 60% of the recording period. 3. For each included recording, in the manual scoring, remove any Hypopnea event, manually scored, that is not followed by a Desaturation event. The pre-screening excluded 77 recordings or 6.8% of the total recordings resulting in 1057 recordings with pre-processed manual scorings, used for comparison. 2.2 Automatic Analysis The default Respiratory AASM scoring that ship with the Noxturnal software application bases its rules on the AASM 2007 scoring manual. It uses a cannula or belt flow and SpO2 signals to score Apneas, Hypopneas and Desaturation events. It uses pressure from the nasal cannula to score Snore events and position/activity signals to score Position and Movement. The AHI and ODI values are then calculated from the automatically scored values and displayed in the Study Results sheet within Noxturnal or in Noxturnal s Respiration Report. The AHI is defined as the number of Apneas/Hypopneas per hour of sleep and the ODI is defined as the number of Desaturation Events per hour of sleep. Since the Noxturnal automatic analysis does not perform sleep staging; the application must estimate the index period (hours of sleep) to use in calculating the AHI and ODI values. To do this, the position and movement events are used.
2.3 Data Mining The methodology of collecting AHI values from manual and automatic scoring for each of the input recordings was as follows: 1. Open up the recording 2. Export the Recording Identifier 3. Activate the original full PSG manual scoring 4. Export the AHI, ODI and Sleep Duration parameters 5. Clear the active scoring 6. Activate the pre-processed (AASM compliant), full PSG manual scoring 7. Export the AHI, ODI and Sleep Duration parameters 8. Clear the active scoring 9. Run the automatic analysis (Respiratory AASM) 10. Export the AHI, ODI and Sleep Duration parameters 11. Collect the exported parameters into Excel where each line contains a recording identifier and the parameters exported for that particular recording.
3 Results 3.1 AHI comparison The following plot shows the correlation between the manual full PSG scoring (AASM compliant) AHI values and the AHI values calculated from the Noxturnal automatic analysis: Comparing the 1057 recordings, there is a 96% correlation between AHI calculated from Noxturnal compared to the manual scoring. The following plot shows the Bland & Altman comparison:
We see from the Bland & Altman plots that the results between the manual and automatic analysis is excellent and that on average the Noxturnal automatic analysis is scoring 0.9 less events than the manual scoring.
3.2 ODI Comparison The following plot shows the correlation between the manual full PSG scoring (AASM compliant) ODI values and the ODI values calculated from the Noxturnal automatic analysis: Comparing the 1057 recordings, there is a 97% correlation between ODI calculated from Noxturnal compared to the manual scoring.
The following plot shows the Bland & Altman comparison: Again we see from the Bland & Altman plots that the results between the manual and automatic analysis is excellent and that on average the Noxturnal automatic analysis is on average scoring 1.3 more events than the manual scorings.
4 Comparison on using automatic vs. manual scoring in the classification of patient population To estimate whether a patient needs further investigation or treatment for Sleep Disordered Breathing an AHI threshold is set and if the AHI value for a patient is above that threshold he or she is determined to be positive for further investigation or treatment. If the AHI value is below that threshold he or she is determined negative for further investigation or treatment. By comparing the AHI values to a threshold and determining positive/negative results from that comparison, we can calculate True Positives (TP), True Negatives (TN), False Positives (FP) and False Negatives (FN). From those values we can calculate: Sensitivity, Specificity, Positive Predictive Value (PPV) and Negative Predictive Value (NPV) as comparison parameters between automatic and manual scoring of events. The standard AHI threshold value used to determine whether a patient needs further treatment or not is the threshold of 10 Apneas/Hypopneas per hour. 4.1 AHI threshold of 10 By using an AHI threshold of 10 the following results were obtained from the manual scoring: Positives = 502 Negatives = 555 The following tables summarize the results: TP: 435 FP: 27 Sensitivity: 87% Specificity: 95% FN: 67 TN: 528 PPV: 94% NPV: 88%
5 Summary The comparison between manual and automatic scoring revealed that the Noxturnal application scores AHI and ODI events in a substantially equivalent manner to the manual scoring on full PSG recordings obtained using Embla N7000 recorders.