Automated Detection of Videotaped Neonatal Seizures of Epileptic Origin

Transcription

1 Epilepsia, 47(6): , 2006 Blackwell Publishing, Inc. C 2006 International League Against Epilepsy Automated Detection of Videotaped Neonatal Seizures of Epileptic Origin Nicolaos B. Karayiannis, Yaohua Xiong, Guozhi Tao, James D. Frost, Jr., Merrill S. Wise, Richard A. Hrachovy, and Eli M. Mizrahi Department of Electrical and Computer Engineering, University of Houston, Peter Kellaway Section of Neurophysiology, Department of Neurology, and Department of Pediatrics, Baylor College of Medicine, and Michael E. DeBakey Veterans Affairs Medical Center, Houston, Texas, U.S.A. Summary: Purpose: This study aimed at the development of a seizure-detection system by training neural networks with quantitative motion information extracted from short video segments of neonatal seizures of the myoclonic and focal clonic types and random infant movements. Methods: The motion of the infants body parts was quantified by temporal motion-strength signals extracted from video segments by motion-segmentation methods based on optical flow computation. The area of each frame occupied by the infants moving body parts was segmented by clustering the motion parameters obtained by fitting an affine model to the pixel velocities. The motion of the infants body parts also was quantified by temporal motion-trajectory signals extracted from video recordings by robust motion trackers based on block-motion models. These motion trackers were developed to adjust autonomously to illumination and contrast changes that may occur during the video-frame sequence. Video segments were represented by quantitative features obtained by analyzing motion-strength and motion-trajectory signals in both the time and frequency domains. Seizure recognition was performed by conventional feedforward neural networks, quantum neural networks, and cosine radial basis function neural networks, which were trained to detect neonatal seizures of the myoclonic and focal clonic types and to distinguish them from random infant movements. Results: The computational tools and procedures developed for automated seizure detection were evaluated on a set of 240 video segments of 54 patients exhibiting myoclonic seizures (80 segments), focal clonic seizures (80 segments), and random infant movements (80 segments). Regardless of the decision scheme used for interpreting the responses of the trained neural networks, all the neural network models exhibited sensitivity and specificity >90%. For one of the decision schemes proposed for interpreting the responses of the trained neural networks, the majority of the trained neural-network models exhibited sensitivity >90% and specificity >95%. In particular, cosine radial basis function neural networks achieved the performance targets of this phase of the project (i.e., sensitivity >95% and specificity >95%). Conclusions: The best among the motion segmentation and tracking methods developed in this study produced quantitative features that constitute a reliable basis for detecting neonatal seizures. The performance targets of this phase of the project were achieved by combining the quantitative features obtained by analyzing motion-strength signals with those produced by analyzing motion-trajectory signals. The computational procedures and tools developed in this study to perform off-line analysis of short video segments will be used in the next phase of this project, which involves the integration of these procedures and tools into a system that can process and analyze long video recordings of infants monitored for seizures in real time. Key Words: Motion segmentation Motion-strength signal Motion tracking Motion-trajectory signal Neonatal seizure Neural network Quantitative feature Video recording. Clinical seizures in the neonate may be the first, and perhaps the only, sign of central nervous system dysfunction. Of every 1,000 infants, between 1.5 and 5.5 will experience seizures within the first month of life (1 8). An increase in the occurrence of neonatal seizures has occurred in premature infants and infants of low birthweight. Seizures can occur in 20% of infants hospital- Accepted March 3, Address correspondence and reprint requests to Dr. N. B. Karayiannis, Department of Electrical and Computer Engineering, University of Houston, N308 Engineering Building 1, Houston, TX , U.S.A. karayiannis@uh.edu ized in the neonatal intensive care unit (NICU) (3,6,9,10). With the occurrence of neonatal seizures comes an increased risk of death and, in survivors, an increased risk of neurologic impairment, developmental delay, and postnatal epilepsy (3,4,9 15). Accurate and rapid detection of neonatal seizures in the newborn may initiate prompt treatment of possible causes of central nervous system disorders and the seizures themselves. The long-term goal of this research is the development of a stand-alone automated system that could be used as a supplement in the NICU to provide 24-h/day noninvasive monitoring of infants at risk for seizures. The development 966

2 DETECTION OF VIDEOTAPED NEONATAL SEIZURES 967 of a seizure-detection system involves the following three tasks: (a) the extraction of quantitative motion information from video recordings of infants monitored for seizures, (b) the selection of quantitative features that convey some unique behavioral characteristics of neonatal seizures, and (c) the training of neural-network classifiers to distinguish neonatal seizures from infant behaviors not associated with seizures. During the first phase of this project, we built on the results of our preliminary study (16) by developing new computational tools and procedures that would provide the building blocks of an automated seizure-detection system (17 24). Motion was quantified by motion-strength signals extracted by using motion-segmentation methods (19,20) and by motion-trajectory signals extracted by using motion-tracking methods (21 24). The study carried out during the first phase of this project indicated that the development of an automated seizure-detection system can benefit from the use of both motion-segmentation and motion-tracking methods, which produced complementary results (18). Moreover, this study evaluated the specific methods developed to perform motion segmentation and motion tracking. The outcome of this evaluation revealed that the motion-segmentation methods based on optical-flow computation outperformed those based on clustering and morphologic filtering (18). In addition, the robust motion trackers based on block-motion models outperformed those based on adaptive block matching and predictive block matching (18). Seizure detection was attempted during the first phase of the project by training feed-forward neural networks (FFNNs) using quantitative features obtained by analyzing motion-strength and motion-trajectory signals in the time domain. The sensitivity and specificity of the trained FFNNs exceeded 80% and 90%, respectively, which were their target values for the first phase of this project (18). During the first phase of the project, the three main tasks involved in automated seizure detection were carried out by applying independently the necessary computational tools and procedures to analyze short video segments containing either neonatal seizures of the myoclonic and focal clonic types or infant movements not associated with seizures. The results of our ongoing project Video Technologies for Neonatal Seizures provided evidence suggesting that the analysis of motion in video can facilitate the automated detection of neonatal seizures (17 24). This study also revealed that accomplishing the goals of this project requires the development of more advanced computational tools and procedures than those used during the first phase of this project (18). We describe the results of a study aimed at accomplishing the first objective of the second phase of this project, which is to enhance the reliability and improve the accuracy of the computational tools and procedures developed during the first phase of the project and to integrate them into the development of an automated system trained to detect neonatal seizures from short video segments of infants monitored for seizures. The sensitivity and specificity targets were both increased for the second phase of the project to 95% to improve the likelihood of accomplishing the second objective of the second phase of the project, which is to build a functioning prototype of a computerized video-monitoring system capable of autonomously detecting neonatal seizures of the myoclonic and focal clonic types in long video recordings of neonates monitored for seizures. METHODS This study relied on analog video recordings selected from a database developed by the Clinical Research Centers for Neonatal Seizures (CRCNS) in Houston, Texas, established by the National Institute of Neurological Disorders and Stroke (25). The clinical seizures included in the CRCNS database have been characterized and classified in terms of their electrographic and behavioral features by a team of clinical neurophysiologists and neonatal electroencephalographers, who studied each video recording together with simultaneously recorded EEGs. The analog video recordings contained in the CRCNS database were digitized with a temporal sampling rate of 30 frames/s, producing sequences of frames of size pixels. The temporal rate of 30 frames/s captured sudden and rapid motion associated with neonatal seizures, whereas the spatial-sampling rate used to digitize the analog video produced video frames of satisfactory resolution for motion analysis. This section describes the computational tools and procedures developed to perform the main tasks involved in automated seizure detection. This study relied on 240 video segments of lengths varying from 7.5 to 20 s, selected and labeled by physicians from a set of video recordings of 54 patients exhibiting myoclonic seizures (80 segments), focal clonic seizures (80 segments), and random infant movements (80 segments). The set of video segments classified as random infant movements by the physicians was used as a comparison group in tests of the ability of the automated system to detect myoclonic and focal clonic seizures. Included in this category were a variety of behaviors (hypnic jerks, startle responses, kicking, etc.) associated with motor activity involving the extremities or body or both that occurred asleep, awake, or during arousal from sleep, and which, in the opinion of the team of clinical neurophysiologists, although exhibiting comparable amounts of motion, were not a result of seizure activity. Extraction of quantitative motion information from video recordings of neonatal seizures Motion in video recordings of neonatal seizures can be quantified by extracting temporal motion-strength and motion-trajectory signals (18).

3 968 N. B. KARAYIANNIS ET AL. Extraction of motion-strength signals based on motion segmentation Motion-strength signals quantify motion by measuring the area of each frame occupied by moving body parts affected by seizures. As the seizure progresses in time, the area measurements A produce temporal motion-strength signals A(t). The extraction of motion-strength signals requires an automated procedure capable of segmenting the infants moving body part(s) at each frame of the sequence. Motion segmentation was performed in this study based on optical-flow computation. Optical flow is the term used to indicate the velocity field generated by the relative motion between an object and the camera in a sequence of frames. Optical-flow computation often relies on regularization theory, which seeks a smooth estimate of the velocity field by imposing a smoothness constraint on the estimated velocities. Our work produced a method for the development of regularized optical-flow computation methods based on a broad variety of smoothness constraints, which include as special cases those based on first-order and second-order spline functionals and a smoothness constraint relying on the squared laplacian operator (19). The solution of the minimization problem was obtained by using elements of calculus of variations in the form of a set of partial differential equations (PDEs). The development of practical optical-flow computation methods was attempted by considering a variety of approximations derived for the differential operators involved in the resulting PDEs. As an alternative, we also proposed a discrete formulation of the optical-flow problem. This formulation relied on the discrete approximation of a family of quadratic functionals, which include as a special case the function involving the squared laplacian considered in this study. Motion-strength signals were obtained after the computation of the velocity fields by measuring the area at each frame containing all pixels with velocities exceeding a certain threshold (19). Our subsequent studies indicated that the procedure used in (19) for quantifying motion is sensitively dependent on the selection of the threshold value. As an alternative, we developed more sophisticated and robust motion-segmentation methods, which relied on clustering (20). According to the segmentation approach followed in this work, each video frame was divided into 3 3 nonoverlapping blocks. For each of these blocks, a parametric motion model was fitted to the velocities produced by optical-flow computation. The model parameters were estimated for each block by minimizing the error between the velocities provided by the model and those produced by optical-flow computation. Blocks corresponding to large values of the error were considered to be unreliable; as such, they were excluded from the clustering process that followed. The rest of the blocks produced parameter vectors (i.e., vectors composed of the model parameters), which were clustered by the k-means algorithm. The clustering process was initialized by computing σ = ( M ) 1/2 i=1 σ i 2, where σ 2 i denotes the variance of the ith element of the M-dimensional parameter vector. One of the initial prototypes was formed as the centroid of all parameter vectors with magnitude above 2σ. The other prototype contained all zero elements and was introduced to represent the cluster of parameter vectors of low magnitude. These prototypes were used to initialize the k-means algorithm, which operated in an iterative fashion until it converged to the final prototypes, which define a certain partition of the parameter vectors. All blocks assigned to the cluster represented by the final prototype with magnitude above a threshold ϑ were considered as belonging to the moving body part. The blocks assigned to the final prototype with magnitude below the threshold ϑ were considered as belonging to the background. Extraction of motion-trajectory signals based on motion tracking Motion-trajectory signals track the body part(s) affected by a seizure and are obtained by projecting the movement of an anatomic site located on the body part onto a twodimensional plane. As the seizure progresses in time, the projections X and Y to the horizontal and vertical axes, respectively, produce temporal motor-activity signals X(t) and Y(t) representing motion of the body part of interest. The motion-trajectory signal Z = Z(t) can be computed in terms of the motor-activity signals X = X(t) and Y = Y(t) as Z = (X X) 2 + (Y Y ) 2, where X and Y denote the means of X = X(t) and Y = Y(t), respectively. The extraction of motion-trajectory signals from video requires (a) a computational procedure for selecting anatomic sites of interest on the infants moving body parts, and (b) an automated method for tracking selected anatomic sites throughout the video-frame sequence. Motion tracking in video was initialized by an automated procedure capable of selecting anatomic sites on the moving body part (17,21). This procedure was based on optical-flow computation for the early identification of motion onset. The velocity fields produced by optical-flow computation provided an estimate of motion in a sequence of frames. The areas of the video frames that contain moving body parts were segmented in this study by thresholding the magnitudes of the velocity vectors. Thresholding was followed by morphologic filtering to eliminate or at least significantly reduce insignificant patches due to noise and other recording imperfections. The initial location of the anatomic site selected for tracking was determined as: (a) the center of the velocity patch with the largest area, and (b) the center of the velocity patch with the largest

4 DETECTION OF VIDEOTAPED NEONATAL SEIZURES 969 average velocity. However, the tracking algorithm failed occasionally to track the anatomic sites selected according to both aforementioned schemes, which have the tendency to place the initial anatomic site in homogeneous areas of the moving body parts. The experiments indicated that tracking a site lying in a homogeneous area of the moving body part is a much more challenging task than tracking a site located in an area rich in texture, such as an area closer to the edges. Thus the selection of an anatomic site for tracking can benefit from the results of texture analysis performed for the region around the initial site location. In this study, the selection of an anatomic site for tracking relied on texture analysis based on the concept of the entropy, which was defined in terms of the co-occurrence matrix by Kermad and Collewet (26). This automated procedure was developed even further to perform tracking of multiple anatomic sites located on moving body parts. This is necessary because neonatal seizures are frequently associated with motion of multiple extremities. In some cases, different velocity patches produced by optical-flow computation were actually located on the same body part, thus resulting in redundant tracking. This problem was overcome by systematically eliminating overlapping velocity patches identified in all subsequent frames, so that only patches representing sites located on different body parts were retained. The procedure outlined previously was used to select an anatomic site located on an infant s moving body part, which was then tracked throughout the entire video-frame sequence. Motion tracking was performed in this study by motion trackers based on novel minimization approaches (22) and a variety of block-motion models, which include a pure translation model, an affine model, and fractional models (23). The reliability of motion tracking was enhanced by developing robust motion trackers specifically designed to suppress the effect of noise and other recording imperfections (23). The motion trackers proposed in (23) were developed under the assumption that the illumination and contrast remain constant during the entire video recording. This assumption is rarely true in practice, especially in NICUs where infants are typically monitored for seizures. This problem was addressed by developing robust motion trackers capable of adjusting to illumination and contrast changes in video recordings (24). This was accomplished by relying on a motion-tracking model that can capture and quantify contrast and brightness changes in the video frames. The motion trackers were developed in the first phase of the project by assuming that the tracked block of pixels is mapped onto a new block of pixels in the next frame. The shape and location of the new block were determined by a parametric block-motion model, whereas the intensities were assumed to be unchanged from one frame to the next. To accommodate illumination and contrast changes between adjacent frames, the shape and location of the new block of pixels were again determined by means of a parametric block-motion model, but the intensity of its pixels was assumed to be a linear function of their intensity at the previous frame. The multiplicative and additive parameters involved in the linear intensity model represent the brightness and contrast changes from frame to frame, respectively. In the absence of illumination and contrast changes, the brightness and contrast parameters would take their baseline values 1 and 0, respectively. The model parameters representing the brightness and contrast changes were adjusted by a data-based computational procedure similar to that developed for the free parameters of the block-motion model (23). The deviation of the brightness and contrast parameters from their baseline values are indicative of illumination or contrast changes or both. Thus the proposed procedure equips the resulting motion trackers with the ability to keep tracking anatomic sites of interest by absorbing illumination and contrast changes. The most reliable and accurate among these motion trackers was that based on a fractional block-motion model; this motion tracker provided the basis for computing the motion-trajectory signals in this study. Selection of quantitative features from video recordings of neonatal seizures This section describes the selection of quantitative features that convey some unique behavioral characteristics of neonatal seizures and nonseizure infant behaviors from temporal motion-strength and motion-trajectory signals extracted from video recordings. Quantitative features can be extracted based on a global view of the temporal signals to represent some of their key properties and also to reveal their relation with the underlying clinical event. As an example, such quantitative features should provide the basis for discriminating the sustained and rhythmic motion of the infants extremities that characterizes focal clonic seizures and the rapid and jerky movements of the infants extremities that are typical manifestations of myoclonic seizures (27). The quick displacement of a body part of interest is represented by a sharp spike in the corresponding motionstrength signal. The same displacement is represented in the motion-trajectory signal by an abrupt rise or fall, which corresponds to a spike of short duration in the gradient of the motion-trajectory signal. Thus the extraction of quantitative features was facilitated by computing the gradient of motion-trajectory signals, which produced temporal signals that have morphology similar to that of motionstrength signals. The motion-strength signals and the gradient of motion-trajectory signals provided the basis for validating the data set selected for this study by comparing the level of motion contained by the video segments of random infant movements with that contained by the video segments of myoclonic and focal clonic seizures. This was done by computing the average power per unit

5 970 N. B. KARAYIANNIS ET AL. time of the motion-strength signals and the gradient of motion-trajectory signals. Quantitative features based on analysis of temporal motion signals in the time domain Both motion-strength and motion-trajectory signals were analyzed in the time domain to compute three quantitative features: energy ratio, variance of time intervals, and maximum spike duration, which are briefly described later (18): For a motion-strength signal containing N samples, the autocorrelation was computed by shifting the signal with respect to itself by up to 0.6 N samples. For a motiontrajectory signal of length N samples, the autocorrelation was computed by shifting the gradient of the signal with respect to itself by up to 0.6 N samples. The energy ratio was calculated from the autocorrelation sequence computed from a motion-strength or the gradient of a motiontrajectory signal as the ratio of the energy contained by the last 75% of the samples of the autocorrelation sequence to the energy contained by the first 25% of samples of the autocorrelation sequence. The feature energy ratio is expected to take large values for clonic movements due to their rhythmicity. In contrast, the feature energy ratio is expected to take small values for isolated rapid and jerky movements associated with myoclonic seizures. The variance of the time intervals was obtained from the motion-strength signals based on the lengths of the time intervals between any two adjacent spikes, and from the gradient of motion-trajectory signals based on the lengths of the time intervals between any two adjacent extrema. This feature cannot be obtained if a motion-strength signal contains one or two spikes or if the gradient of the motion-trajectory signal contains one or two extrema. In such cases, the variance was assigned an arbitrary large value. This feature variance of time intervals is a measure of rhythmicity, because rhythmic movements produce variance values close to zero. The maximum spike duration provides a quantitative measure of the speed of movements. The maximum spike duration was obtained directly from the motion-strength signals and from the gradient of the motion-trajectory signals. Because a step-like displacement in the motiontrajectory signal would be represented as a spike in its gradient, the feature maximum spike duration is expected to take the smallest values for seizures involving rapid movements of short duration. Such values can differentiate the movements associated with seizures from random infant movements, which are typically slower and produce spikes of longer duration. Quantitative features based on analysis of temporal motion signals in the frequency domain Additional quantitative features were established more recently by analyzing motion-strength and motiontrajectory signals in the frequency domain: 10% Spectral power frequency: This feature quantifies the rate of reduction of the power spectral density (PSD) of motion-strength signals for frequency values near the origin. This feature is obtained by determining the upper bound F10 of the frequency band [0, F10] that contains 10% of the total spectral power. Motion-strength signals extracted from myoclonic seizures contain an isolated sharp spike or a small number of sharp spikes. It is, therefore, expected that the PSD of such signals would reduce slowly near the origin because the spectrum of a spike of near-zero duration is almost flat. Thus myoclonic seizures are expected to produce relatively high values of the feature F10. Motion-strength signals extracted from focal clonic seizures typically contain a sequence of nearperiodic spikes. Thus it is expected that the spectrum of such signals also would contain a sequence of spikes. This implies that the power would reduce quickly near the origin, producing relatively low values of the feature F10. Random infant movements are typically slower than those associated with myoclonic seizures. Thus it is expected that random infant movements would produce lower values of the feature F10 that could be used to discriminate myoclonic seizures from random infant movements. 95% Spectral power frequency: This feature relies on the distribution of the power of the gradient of motiontrajectory signals, also called the spectral power. The spectral power distribution can be quantified by computing the PSD of the gradient of motion-trajectory signals extracted from video. The spectral power corresponding to myoclonic seizures is expected to spread over a wide band of frequencies. This is consistent with the fact that the rapid and jerky movements characterizing myoclonic seizures correspond to isolated sharp spikes in the gradient of motion-trajectory signals. In contrast, the sustained and rhythmic motion associated with focal clonic seizures lead to quasi-periodic motion-trajectory signals. Thus most of the spectral power of such signals is typically contained by a narrow frequency band. This is expected to differentiate focal clonic seizures from both myoclonic seizures and random infant movements. In practice, video recordings of infants monitored for seizures can be quantified in terms of the spectral properties mentioned earlier by determining the frequency band [0, F95] that contains 95% of the total spectral power. The upper bound F95 is called the 95% power spectral frequency. According to this discussion, focal clonic seizures are expected to produce the lowest values of the frequency F95, whereas the highest values of F95 are expected to come from myoclonic seizures. Detection of neonatal seizures by using neural networks The development of an automated seizure-detection system is essentially the problem of classifying a set of temporal signals that describe neonatal seizures and clinical events not associated with seizures. Neural networks

6 DETECTION OF VIDEOTAPED NEONATAL SEIZURES 971 provide a solid basis for the development of a seizuredetection system because of their versatility and flexibility (28,29). A straightforward approach to seizure detection is to use a set of quantitative features extracted from the temporal signals to train an artificial neural network model. This work relied on conventional feed-forward neural networks (FFNNs), quantum neural networks (QNNs), and cosine radial basis function neural networks (RBFNNs) trained by two different algorithms. The neural network models were trained and tested in 50 trials by using training and testing sets of equal sizes. The training and testing sets were obtained by random permutations of the data set of 240 video segments. Each of the training and testing sets contained 120 video segments (40 segments of myoclonic seizures, 40 segments of focal clonic seizures, and 40 segments of random movements). The inputs used for training the neural-network models were formed in terms of features extracted from temporal motion-strength and motion-trajectory signals. Neural network models used for seizure detection The development of an automated seizure-detection system was first attempted by relying on conventional FFNNs (28,29). FFNNs have been a natural choice as trainable pattern classifiers because of their functionapproximation capability and generalization ability (28). The function-approximation capability allows them to form arbitrary nonlinear discriminant surfaces, whereas the generalization ability allows them to respond consistently to data they were not trained with. Seizure detection was performed by FFNNs trained with a single layer of 20 sigmoid hidden units, which exhibited the best overall performance. The number of hidden units was chosen in these experiments by trial and error in an attempt to ensure that the FFNNs implement the mapping defined by the training set without compromising their generalization ability. The FFNNs were trained by the error back-propagation learning algorithm, which is based on gradient descent (28,29). The development of an automated seizure-detection system also was attempted by training QNNs (30,31). QNNs are feed-forward neural network models that were developed in an attempt to deal with the inability of conventional FFNNs to deal effectively with the uncertainty typically involved in pattern-classification tasks (32). The capacity of QNNs for identifying uncertainty in data classification arises from their ability to create graded internal representations of the sample information provided by the training data. The sample information is encoded into graded internal representations by choosing multilevel activation functions for the hidden units, instead of the conventional sigmoid activation functions. The activation functions of the multilevel hidden units are formed by superimposing n s shifted sigmoid functions. The locations of the superimposed sigmoid functions determine the n s quantum levels. The QNNs were trained with n h = 20 multilevel hidden units, each containing n s = 4 quantum levels. The development of an automated seizure-detection system was finally attempted by training radial basis functions neural networks (RBFNNs). The true potential of RBFNNs was revealed by recent studies, including the development of an axiomatic approach for constructing reformulated RBFNNs suitable for gradient descent learning (33,34). This approach reduces the development of reformulated RBFNNs to the selection of admissible generator functions that determine the form of the radial basis functions. Linear generator functions of a special form produced cosine RBFNNs, that is, a special class of reformulated RBFNNs constructed by radial basis functions that are referred to as cosine radial basis functions (34). The cosine RBFNNs were trained by the original learning algorithm developed by Karayiannis and Randolph-Gips (34) and by the learning algorithm proposed recently by Karayiannis and Xiong (35) for training cosine RBFNNs capable of identifying uncertainty in the classification of multidimensional data. The new learning algorithm can autonomously determine the size of the cosine RBFNN required for implementing a desired input output mapping; this is done during the training process by exploiting an inherent advantage of cosine radial basis functions, which diminish when their only free parameter approaches zero. When cosine RBFNNs were trained by the original learning algorithm, the number of cosine radial basis functions was determined in terms of the number M = 120 of training examples as c = 5M = 600 and remained fixed during the learning process. When RBFNNs were trained by the new learning algorithm, the initial number of cosine radial basis functions was also determined in terms of the number M of training examples as c in = 5M = 600. A cosine radial basis function was eliminated if the value of its corresponding free parameter reduced below a threshold during the training process. Interpreting the responses of neural network models trained to detect neonatal seizures The development of trainable classifiers requires the formation of a training set, which involves the selection of a scheme that establishes a correspondence between the output units of the neural network model and the classes involved. Such a correspondence influences the development of schemes used for assigning class labels to the inputs of the trained neural network model by interpreting its responses. These issues were investigated in this study by relying on conventional FFNNs, which are considered to be the standard among the available feed-forward neural network models. FFNNs have been extensively evaluated in pattern-classification applications, and their behavior and properties have been extensively documented.

7 972 N. B. KARAYIANNIS ET AL. Seizure detection was initially performed by training FFNNs with two sigmoid output units representing the classes neonatal seizure and random infant movement. Label assignment relied on a winner-take-all decision scheme based on the responses ŷ 1 and ŷ 2 of the output units representing the classes neonatal seizure and random infant movement, respectively. More specifically, this decision scheme assigned to the input the label neonatal seizure if ŷ 1 > ŷ 2 and the label random infant movement if ŷ 1 < ŷ 2. Seizure detection was also performed by training FFNNs with three sigmoid output units representing the classes myoclonic seizure, focal clonic seizure, and random infant movement. The inputs of the trained FFNNs were labeled as neonatal seizures or random infant movements based on the responses ŷ 1 ŷ 2, and ŷ 3 of the output units representing the classes myoclonic seizure, focal clonic seizure, and random infant movement, respectively. Class label assignment relied on two alternative decision schemes, which are outlined subsequently: The decision scheme 1 assigned to the input the label neonatal seizure if max{ŷ 1, ŷ 2 } > ŷ 3 and the label random infant movement if max{ŷ 1, ŷ 2 } < ŷ 3. According to decision scheme 2, the input was labeled as a neonatal seizure if (ŷ 1 + ŷ 2 )/2 > ŷ 3 and as random infant movement if (ŷ 1 + ŷ 2 ) /2 < ŷ 3. These two decision schemes are expected to assign different class labels to ambiguous inputs that produce comparable responses ŷ 1, ŷ 2 and ŷ 3. When dealing with ambiguous inputs, the decision scheme 1 is likely to assign the neonatal seizure label, thus improving the sensitivity at the expense of specificity. In contrast, the decision scheme 2 is expected to improve the specificity at the expense of sensitivity because it is more likely to assign the class label random infant movement to ambiguous inputs. RESULTS This section describes the extraction of quantitative motion information from video and compares the performance of various neural network models trained to detect neonatal seizures of the myoclonic and focal clonic types and to distinguish them from random infant behaviors. Extraction of quantitative motion information from video The experimental results presented in this section validate the data used in this study and illustrate the process of obtaining quantitative features by analyzing motionstrength and motion-trajectory signals extracted from video recordings. Measuring the motion levels of video segments The data set used in this study was validated by computing the power per unit time of motion-strength signals and the gradient of motion-trajectory signals. These motion measures are plotted by using a logarithmic scale in Fig. 1 for 80 segments of myoclonic seizures, 80 segments of focal clonic seizures, and 80 segments of random infant movements. According to Fig. 1, the power values obtained for video segments of random infant movements were comparable with those obtained for focal FIG. 1. A scatterplot of the power values obtained for 80 segments of myoclonic seizures, 80 segments of focal clonic seizures, and 80 segments of random infant movements.

8 DETECTION OF VIDEOTAPED NEONATAL SEIZURES 973 FIG. 2. a: Selected frames from the video recordings of a myoclonic seizure (MCS), focal clonic seizure (FCS), and random infant movement (RIM). b: Motion-strength signals A(t) extracted from the video recordings. c: Autocorrelation sequences computed from the motion-strength signals. d: Motion-strength signals together with the time intervals between the spikes. e: Power spectral density obtained from motion-strength signals. Energy ratio: (MCS), (FCS), (RIM); variance of time intervals: 150 (MCS), 5.4 (FCS), 9.8 (RIM); maximum spike duration: 10 (MCS), 12 (FCS), 58 (RIM); 10% spectral power frequency: 0.19 Hz (MCS), 0.10 Hz (FCS), 0.07 Hz (RIM). clonic seizures, an outcome consistent with the similarities between random infant movements and the sustained and rhythmic movements associated with focal clonic seizures. Fig. 1 also indicates that the power values obtained for random infant movements were generally higher than those obtained for myoclonic seizures; this is consistent with the differences between random infant movements and the isolated rapid and jerky movements associated with myoclonic seizures. The results shown in Fig. 1 validate the data set used in this study because the power measure computed from temporal motion signals extracted from video segments is directly related to the level of motion in these video segments. Quantitative features based on motion-strength signals Fig. 2 shows a collection of intermediate results obtained in the process of selecting quantitative features from motion-strength signals. These signals were extracted by the motion-segmentation approach based on clustering of motion parameters obtained for the velocity fields

9 974 N. B. KARAYIANNIS ET AL. produced by the discrete formulation of the optical flow problem. Fig. 2(a) shows selected frames of the video recordings of a myoclonic seizure, a focal clonic seizure, and a random infant movement. Fig. 2(b) shows the motionstrength signals extracted to quantify the motion of the infants body parts, which are enclosed in Fig. 2(a) by a box. Fig. 2(b) reveals the behavioral characteristics of myoclonic and focal clonic seizures, which can be exploited to differentiate such seizures from random infant movements. The motion-strength signal extracted from the myoclonic seizure contains two spikes of short duration, which represent the two rapid and jerky movements of the infant s left hand that were observed in this video segment. The series of periodic spikes obtained from the focal clonic seizure represent the sustained and rhythmic motion of the infant s left hand and right leg. The motionstrength signal obtained for the random movements of an infant s left hand, right hand, and right leg can easily be distinguished from those obtained for seizures. Fig. 2(c) shows the autocorrelation sequences computed from the motion-strength signals shown in Fig. 2(b). According to Fig. 2(c), the autocorrelation sequence obtained from the myoclonic seizure decays quickly to values close to zero and remains close to zero most of the time. In contrast, the autocorrelation sequence computed from the focal clonic seizure takes nonzero values most of the time, a fact that is consistent with the rhythmicity of motion associated with seizures of this type. The autocorrelation sequence obtained from the random infant movement is closer to that obtained from the focal clonic seizure. However, the proportion of the energy contained by its last 75% of samples is lower than that corresponding to the focal clonic seizure. The value of the feature energy ratio was for the random infant movement, which can be compared with the energy ratio value of corresponding to the focal clonic seizure. Fig. 2(d) shows the motion-strength signals together with the time intervals between the spikes. The intervals between the spikes shown for the focal clonic seizure in Fig. 2(d) were of comparable length. This outcome, which is consistent with the rhythmicity of motion associated with focal clonic seizures, is the reason that the focal clonic seizure produced the lowest value of the feature variance of time intervals. The highest variance value was obtained for the myoclonic seizure, whereas the random infant movement produced a variance value between the two extremes corresponding to the two types of seizures. The random infant movement produced the longest spikes, an outcome that is consistent with the fact that random infant movements are typically slower than the rapid and jerky movements associated with neonatal seizures. Fig. 2(e) shows the PSD obtained for all three video recordings, which contained a dominant spike that was located at the origin. According to Fig. 2(e), the rate of reduction of the dominant spike was relatively low for the myoclonic seizure. In contrast, the dominant spike declined very quickly for the focal clonic seizure and the random infant movement. As a result, these events typically produce lower values of the feature F10 than those obtained for myoclonic seizures. The PSD of the focal clonic seizure contained numerous spikes of substantial amplitude after the dominant spike. In contrast, the PSD of the random infant movement contained a sequence of weak spikes after the dominant spike. This can differentiate random infant movements from focal clonic seizures. Quantitative features based on motion-trajectory signals Fig. 3 shows some examples of intermediate results obtained in the process of selecting quantitative features from motion-trajectory signals extracted by a robust motion tracker that relied on the fractional block-motion model. Fig. 3(a) shows selected frames of the video recordings of a myoclonic seizure, a focal clonic seizure, and a random infant movement. Fig. 3(b) shows the motiontrajectory signals extracted to quantify the motion of the infants body parts, which are enclosed in Fig. 3(a) by a box. The rapid and jerky movements of the infant s left hand associated with the myoclonic seizure are represented by the abrupt fall before frame 150 and the needlelike fluctuation just before frame 250. The saw-like signal obtained for the movements of the infant s left leg associated with the focal clonic seizure represents the sustained and rhythmic motion observed in the corresponding video recording. Finally, the random movement of the infant s right hand can easily be distinguished from seizures of both myoclonic and focal clonic types by the lack of any structure of the corresponding motion-trajectory signal. Fig. 3(c) shows the autocorrelation sequences computed from the gradient of motion-trajectory signals. According to Fig. 3(c), the autocorrelation sequence computed for the myoclonic seizure decays very quickly and takes values close to zero for large time intervals. This is consistent with the rapid and jerky movements that are the signature of myoclonic seizures. In contrast, the rhythmic movements that characterize focal clonic seizures produce autocorrelation sequences that do not decay to zero, such as that shown for the focal clonic seizure in Fig. 3(c). Random infant movements can produce a great variety of autocorrelation sequences. Fig. 3(d) shows the signals produced by computing the gradient of the motion-trajectory signals shown in Fig. 3(b) together with the time intervals between the extrema. The gradient of the motion-trajectory signals shown in Fig. 3(d) for the myoclonic and focal clonic seizures contains spikes of short duration, consistent with the rapid movements associated with neonatal seizures. Figure 3(d) also describes a situation in which the random infant movement produces short spikes together with spikes of longer duration. The rhythmic movements

10 DETECTION OF VIDEOTAPED NEONATAL SEIZURES 975 FIG. 3. a: Selected frames from the video recordings of a myoclonic seizure (MCS), focal clonic seizure (FCS), and random infant movement (RIM). b: Motion-trajectory signals Z(t) extracted from the video recordings. c: Autocorrelation sequences computed from the gradient of motion-trajectory signals. d: Signals produced by computing the gradient of the motion-trajectory signals. e: Power spectral density computed from the gradient of the motion-trajectory signals. Energy ratio: (MCS), (FCS), (RIM); variance of time intervals: 80.0 (MCS), 6.39 (FCS), 39.6 (RIM); maximum spike duration: 6 (MCS), 17 (FCS), 23 (RIM); 95% spectral power frequency: 10.5 (MCS), 6.1 (FCS), 9.1 (RIM). associated with focal clonic seizures correspond to time intervals between the extrema that are almost equal in length. This is the reason that focal clonic seizures lead to small values of the feature variance of time intervals compared with myoclonic seizures and random infant movements. Such clinical events produce relatively large values of the feature variance of time intervals because they correspond to irregular time intervals between the extrema, as indicated by Fig. 3(d). Fig. 3(e) shows the PSD of the gradient of motiontrajectory signals shown in Fig. 3(d). The spectral power was spread over the widest frequency band for the gradient of the motion-trajectory signal extracted from the video recording of the myoclonic seizure. In contrast, the spectral power of the gradient of the motion-trajectory signal extracted from the focal clonic seizure was spread over the narrowest frequency band. The myoclonic seizure produced the highest value of the 95% spectral power frequency. The lowest value of F95 was that obtained for the focal clonic seizure, with the value F95 obtained for the random infant movement between the two extremes corresponding to the myoclonic and focal clonic seizures. However, the value of F95 obtained for the random infant movement was close to that obtained for the myoclonic

11 976 N. B. KARAYIANNIS ET AL. seizure. This indicates that the feature 95% spectral power frequency would be particularly useful for differentiating focal clonic seizures from both myoclonic seizures and random infant movements. Automated detection of neonatal seizures based on neural networks In the first set of experiments, seizure detection was attempted by training FFNNs with two sigmoid output units representing the classes neonatal seizure and random infant movement. Seizure detection was also attempted by training FFNNs with three sigmoid output units representing the classes myoclonic seizure, focal clonic seizure, and random infant movement. The results of the 50 trials provided the basis for computing the sample mean and standard deviation for the sensitivity and specificity. These values were used to obtain the 95% confidence intervals for the sensitivity and specificity of the trained FFNNs on the testing sets, which are shown in Table 1. The same results were also used to test whether the trained FFNNs achieved the desired performance levels of the second phase of this project. This was done by performing a one-tailed t test of the hypothesis that the sensitivity and the specificity exceeded their target value of 95% (36). The FFNNs trained with two output units exhibited sensitivity and specificity >90%. However, they did not even come close to achieving the specificity target value of 95%. Overall, the FFNNs trained with two output units were outperformed by the FFNNs trained with three output units representing the classes myoclonic seizure, focal clonic seizure, and random infant movement. According to Table 1, the sensitivity and specificity of the FFNNs trained with three output units depend on the decision scheme used for class-label assignment. When classlabel assignment relied on decision scheme 1, the sensitivity of the trained FFNNs exceeded 95%, whereas their specificity exceeded 90%. When class-label assignment relied on decision scheme 2, the sensitivity of the FFNNs was <95% but >90%. In this case, their specificity was close to its target value of 95%, although the hypothesis specificity >95% was not validated by the t test. TABLE 1. Sensitivity and specificity (95% confidence intervals) produced by FFNNs Sensitivity Specificity Structure of neural network (%) (%) Two output units 95.4 ± ± 1.3 Three output units (Decision Scheme 1) 97.3 ± ± 1.1 Three output units (Decision Scheme 2) 92.3 ± ± 0.9 The FFNNs were trained with two output units and three output units to detect neonatal seizures by using four features extracted from motion-strength signals (energy ratio, variance of time intervals, maximum spike duration, and 10% spectral power frequency) and four features extracted from motion-trajectory signals (energy ratio, variance of time intervals, maximum spike duration, and 95% spectral power frequency). Table 2 summarizes the performance of various neural network models, which were trained to detect neonatal seizures based on quantitative features obtained from motion-strength signals. When class-label assignment relied on decision scheme 1, the sensitivity of the QNNs was lower than that of the FFNNs. This was compensated by an improved specificity of the QNNs. The specificity of the QNNs was also higher than that of the FFNNs when classlabel assignment relied on decision scheme 2. In this case, the sensitivity of FFNNs and QNNs was almost the same. When class label assignment relied on decision scheme 1, the RBFNNs trained by using the original learning algorithm exhibited higher sensitivity than FFNNs and QNNs. In this case, the specificity of these RBFNNs was comparable to that exhibited by FFNNs but lower than that exhibited by QNNs. When class-label assignment relied on decision scheme 2, the specificity of RBFNNs trained by the original learning algorithm was higher than that exhibited by FFNNs but lower than that exhibited by QNNs. In this case, the sensitivity of RBFNNs was higher by almost 4% than that exhibited by both FFNNs and QNNs. Compared with the RBFNNs trained by the original learning algorithm, the RBFNNs trained by the new learning algorithm exhibited lower sensitivity and specificity. Nevertheless, the RBFNNs trained by the new learning algorithm exhibited the sensitivity and specificity values shown in Table 2 when tested with fewer radial basis functions. In this set of experiments, the original learning algorithm trained cosine RBFNNs with c = 600 radial basis functions. In contrast, the RBFNNs trained by the new learning algorithm in 50 trials contained 150 ± 20 radial basis functions, which is just a fraction of the number c in = 600 of radial basis functions used to initialize the learning process. Table 3 summarizes the performance of various neural network models, which were trained to detect neonatal seizures based on quantitative features obtained from motion-trajectory signals. The sensitivity of the QNNs was slightly lower on average than that of the FFNNs when class-label assignment was based on decision scheme 1. When decision scheme 2 provided the basis for classlabel assignment, the specificity of the QNNs was slightly higher on average than that of the FFNNs. Nevertheless, in either case, the performance differences among QNNs and FFNNs were not statistically significant. The RBFNNs trained by the new learning algorithm outperformed those trained by the original learning algorithm. This is an important experimental outcome, given that the RBFNNs trained by the new learning algorithm contain fewer radial basis functions. In this case, the cosine RBFNNs trained by the new learning algorithm contained 176 ± 46 radial basis functions, that is, a fraction of the c in = 600 radial basis functions used to initialize the learning process. The RBFNNs trained by the new learning algorithm exhibited similar performance with FFNNs and QNNs. These three neural network models exhibited different sensitivity and