An improved procedure for the extraction of temporal motion strength signals from video recordings of neonatal seizures

Transcription

1 Image and Vision Computing 24 (2006) An improved procedure for the extraction of temporal motion strength signals from video recordings of neonatal seizures Nicolaos B. Karayiannis *, Guozhi Tao Department of Electrical and Computer Engineering, University of Houston, N308 Engineering Building 1, Houston, TX , USA Received 22 December 2004; received in revised form 8 September 2005; accepted 15 September 2005 Abstract This paper presents a procedure developed to extract quantitative motion information from video recordings of neonatal seizures in the form of temporal motion strength signals. Temporal motion strength signals are obtained from a sequence of video frames by measuring the displacement areas of the infants moving body part(s) from frame to frame of the video sequence. The proposed motion segmentation procedure relies on the application of non-linear filtering, vector clustering, and morphological filtering to the differences between adjacent frames. The experiments indicate that temporal motion strength signals constitute a satisfactory representation of videotaped clinical events and may be used for automated seizure recognition. q 2005 Elsevier B.V. All rights reserved. Keywords: Clustering; Median filtering; Morphological filtering; Motion segmentation; Motion strength signal; Neonatal seizure; Video recording 1. Introduction Seizures occur in approximately 2 5/1000 live births, depending upon studied populations and methodology [1 6]. In fact, the incidence of seizures in infants weighing less than 1500 g is 57.5/1000 live births compared to 3.5/1000 live births for all birthweights [4]. Similarly, Scher et al. [7] reported that seizures occurred in approximately 4% of premature infants less than 30 weeks conceptional age, although some have reported the incidence in this population to be as high as 20%. These studies indicate that seizure occurrence represents the most frequent clinical sign of central nervous system dysfunction in the newborn [8,9]. The prompt identification of clinical seizures when they occur in the newborn, the subsequent evaluation of their etiology, and the institution of etiology-specific therapy may significantly reduce associated morbidity. Despite the importance of seizure recognition, most neonatal intensive care units and nurseries have limited resources for seizure identification. Neonatal seizures are often brief and may not be recognized since nurses and physicians cannot provide continuous surveillance of all * Corresponding author. Tel.: C ; fax: C address: karayiannis@uh.edu (N.B. Karayiannis) /$ - see front matter q 2005 Elsevier B.V. All rights reserved. doi: /j.imavis infants at risk for clinical seizures. These factors illustrate the clear need for improved seizure surveillance methods that supplement direct observation by nurses and physicians, and that are practical and economically feasible. Early attempts to characterize neonatal seizures involved primarily bedside observation and brief EEG recordings. Portable EEG/video/- polygraphic monitoring techniques have allowed investigators to assess and characterize neonatal seizures at the bedside and have permitted retrospective review. However, these techniques are relatively expensive, are generally used for only a few hours of monitoring, and are not routinely available in many centers [10 17]. The long-term goal of the study outlined in this paper is the development of a stand-alone automated system that could be used as a supplement in the neonatal intensive care unit to: (1) provide 24-h a day non-invasive monitoring of infants at risk for seizures, and (2) facilitate the analysis and characterization of videotaped neonatal seizures by physicians during retrospective review. The development of such a system requires automated procedures for extracting quantitative motion information from video recordings of infants monitored for seizures. The study described in this paper involved short video recordings of neonatal seizures of the myoclonic and focal clonic types, which affect the infants extremities [14], and random movements of the infants extremities not associated with seizures. Motion in video recordings of infants monitored for seizures can be quantified by extracting temporal motor activity and

2 28 N.B. Karayiannis, G. Tao / Image and Vision Computing 24 (2006) motion strength signals [18,19]. Strength of motion mainly refers to the speed of the infants moving body part(s), which can be indirectly measured by estimating the displacement areas of the infants moving body part(s) from frame to frame of a video sequence. As the seizure progresses in time, the displacement area measurements A produce temporal motion strength signals A(t). The extraction of motion strength signals requires an automated procedure capable of segmenting the displacement of the infants moving body part(s) at each frame of the sequence. Segmentation of motion in an image sequence is one of the most challenging problems in computer vision. Segmentation based on motion information is unlikely to achieve an accurate result without the help of spatial information. In fact, many researchers proposed spatiotemporal segmentation procedures where spatial segmentation played an important role in the overall process [20 24]. Thompson [20] was the first to use a similarity constraint based on contrast and motion to achieve motion segmentation. This algorithm relies on a region-merging procedure, which is realized in two phases: In the first phase, adjacent regions with the same velocity label are always merged. Unlabeled regions and regions with different labels can be merged but only if they satisfy similarity conditions based on contrast information. In the second phase, velocity estimates are well determined so regions with different labels are not allowed to merge. The merging operation is applied only on the regions that satisfy the similarity conditions based on contrast information but have the same velocity label. Dufaux et al. [21] proposed a new algorithm for the representation of a scene in terms of a moving object. Spatial pre-filtering is applied in the beginning to facilitate segmentation of the frames by producing constant luminance regions. The morphological OPENING and CLOSING operators are adopted for this task. Following the pre-processing stage, the algorithm begins with static segmentation and splits or merges regions based on motion information. Static segmentation is performed by applying the k-means clustering algorithm on the luminance values. For each of the resulting static regions, affine motion parameters are computed. Motion estimation is applied on a region characterized by coherent motion. Regions that are not well compensated are further split. Afterward, regions with similar motion are merged in the motion parameter space by applying clustering. Clustering in the motion parameter space results in regions characterized by coherent motion that can be identified as the moving object. Choi et al. [22] presented a morphological spatio-temporal segmentation algorithm that incorporates luminance and motion information simultaneously and uses morphological tools such as morphological filters and the watershed algorithm. The procedure consists of three steps: joint marker extraction, boundary decision, and motion-based region fusion. The algorithm is implemented as follows: The frames are operated by the morphological OPENING and CLOSING filters to facilitate the segmentation process. The intensity markers are identified by labeling flat regions that have size larger than a given threshold. Then the motion markers are extracted to split the intensity marker for which the affine motion parameters are not accurate enough. The extraction of motion and intensity markers is followed by the boundary decision stage, which is used to deal with pixels not yet assigned to any region. The watershed algorithm is used as a region-growing tool based on the joint similarity measure. The joint similarity measure used for spatio-temporal segmentation is the weighted sum of the intensity difference and the motion difference. Finally, redundant regions are eliminated using motion-based region fusion. Regions created at stage two are merged together based on the consistency of an affine transformation. Fusion of motion segmentation and image spatial segmentation can solve the over-segmentation problem since it relies on illumination segmentation to determine the object boundary. However, despite its advantages, fusion of motion segmentation and spatial segmentation is likely to trigger repeated splitand-merge procedures; this could increase the computational requirements of the resulting algorithms. If the motion boundary is not consistent with the intensity values of the object boundary, which is possible during complex movements, the results of spatial segmentation could mislead the motion segmentation algorithm. This may be the case in video recordings of neonatal seizures if only some small part of the infant s limb is moving, The extraction of quantitative motion information from video recordings of neonatal seizures was initially attempted by using a motion segmentation procedure based on subband decomposition of video, which was followed by non-linear filtering and scalar clustering [18]. This procedure was tested on a few video recordings during an early study and produced motion strength signals consistent with the videotaped clinical event. However, additional testing revealed that the original procedure is sensitive to noise. This paper presents an improved motion segmentation procedure developed for the extraction of motion strength signals. Motion segmentation was attempted in this study by combining the most promising among the tools and methodologies mentioned above into a motion segmentation procedure especially tailored to quantify video recordings of neonatal seizures. The proposed procedure was designed to overcome some of the deficiencies of the original procedure developed for the analysis of video recordings of neonatal seizures [18]. An additional constraint imposed on the development of the proposed motion segmentation procedure was that of low computational and storage requirements. This constraint was deemed necessary to ensure that the system under development would be capable of analyzing videotaped neonatal seizures in real time. 2. Methods The extraction of quantitative information from videotaped seizures must focus only on the moving parts of the infant s body that are affected by the seizure [18,19]. The extraction from video recordings of visual information that is relevant only to the seizure can be accomplished by the procedure outlined in this section.

3 N.B. Karayiannis, G. Tao / Image and Vision Computing 24 (2006) Detection of motion In this study, motion was detected on a new frame sequence obtained by subtracting the current frame from the previous frame. This operation can be seen as temporal high pass filtering applied on the original frame sequence. The sequence of frames produced by temporal high pass filtering is expected to contain only the traces of the moving body parts. However, the experiments indicated that the frames of this sequence contain the moving body parts but they are also corrupted by spiky noise. The presence of spiky noise can be attributed to recording imperfections, illumination and contrast changes, and distortion introduced by the conversion of analog video recordings to digital video sequences. The noise appears as spurious positive and negative spikes of large magnitude compared with the signal. Removing random spiky noise is essential since this kind of noise can mislead the clustering algorithm used in the next stage for motion segmentation. Spiky noise was removed in this study by applying median filtering, which is very effective in removing this type of noise without blurring the edges of the objects [25]. Fig. 1(a) shows selected frames from the video recording of a focal clonic seizure. Fig. 1(b) shows the new frames produced by applying temporal high pass filtering to the original sequence of frames followed by the application of a 2-D median filter of size 3!3 pixels. This was the smallest nontrivial filter window proven experimentally to be capable of eliminating spiky noise with the least impact on the visual information contained by the video frames. The new frames corresponding to frames 82 and 150 show the infant s right leg, which is involved in the seizure. In frame 82, the right leg is moving to the bottom and to the right of the frame. In frame 150, the right leg is moving to the top and to the left of the frame. The infant s right leg is not visible in the difference frame computed on frames 1 and 190 since there was no motion in these frames Displacement area estimation Following median filtering, the frame sequence produced from the original sequence of frames by applying temporal high pass filtering was segmented in order to isolate the moving body parts from background noise and other irrelevant clusters of pixels. Motion segmentation was performed in the original study by an adaptive version of the k-means algorithm, which clustered all pixels of each frame from the sequence formed by the difference frames in kz3 clusters [18,19]. The main drawback of such a segmentation procedure is that it ignores the spatial correlation within each frame since it forms clusters of individual pixels [26]. The use of scalar clustering also degrades the resistance of the segmentation procedure to noise. The robustness of the procedure developed to extract motion strength signals was improved in this study by employing a segmentation technique that relies on vector clustering instead of scalar clustering. This procedure clustered vectors of length 9 that contained the pixels of a 3!3 window. In the sequence of frames produced by temporal high pass filtering, the pixels of each frame with non-zero values belong to displacement areas of the infants moving body part(s). Note that the infant s body and the background have different intensity values in the original frames. Thus, if an object moves from a certain position in the previous frame to a new position in the current frame, the intensity differences in the old and new positions of the object will have different signs. In the original study, the pixels corresponding to the old and new positions were segmented together [18,19]. In the proposed approach, the pixels corresponding to the old and new positions of the moving object formed two different segments. This was initially attempted by employing the k-means algorithm to partition the available vectors into kz3 clusters, which were represented by the prototypes p 1, p 2, and p 3. The clustering process was initialized to ensure that the first cluster contain vectors composed of pixels of negative intensity values of large magnitude, the second cluster contain vectors composed of pixels of positive intensity values of large magnitude, and the third cluster contain vectors composed of background pixels. More specifically, each entry of the prototype p 1 (p 2 ) that represented the first (second) cluster was selected to be the minimum (maximum) among the corresponding entries of the available vectors. The prototype p 3 that represented the third cluster was selected as the centroid of p 1 and p 2, that is, p 3 Z 0.5(p 1 Cp 2 ). Following cluster formation, a non-negligible proportion of vectors composed of background pixels were closer to the prototypes p 1 and p 2 instead of the prototype p 3, which represented the cluster of vectors composed of background pixels. As a result, such vectors were assigned to the displacement areas of the infants moving body part(s); this resulted in overestimation of the strength of motion. In order to prevent this from happening, the displacement areas were alternatively estimated by employing the k-means algorithm to partition the available vectors into kz4 clusters. The prototypes p 1 and p 2 representing the first and second clusters were initialized as described above. The prototypes p 3 and p 4 representing the third and fourth clusters, respectively, were initialized as p 3 Z0.5p 1 and p 4 Z0.5p 2. This approach reduced considerably the proportion of vectors composed of background pixels that were assigned to the displacement areas of the infants moving body part(s). The segmentation process was completed by assigning to all pixels belonging to the cluster of the highest intensities the same intensity value of 255 (corresponding to white color). The pixels belonging to the cluster of the lowest intensities were all assigned the same intensity value of 0 (corresponding to black color). Finally, the pixels corresponding to the other two clusters were all considered background and were assigned the same intensity value of 128 (corresponding to gray color). Fig. 1(c) shows the frames corresponding to the four frames in Fig. 1(a) following segmentation. Segmentation eliminated most spurious clusters of pixels in frames 1 and 190, which contained no moving body parts. Segmentation also eliminated the clusters of pixels with low absolute intensity values from frame 82 and 150, which led to a better definition of the displacement of the moving body part. The traces of

4 30 N.B. Karayiannis, G. Tao / Image and Vision Computing 24 (2006) Fig. 1. Extraction of temporal motion strength signals: (a) selected frames from the video recording of a focal clonic seizure, (b) frames produced by applying median filtering on the new sequence offrames produced by temporal high pass filtering, (c) frames produced by clustering, and (d) frames produced by morphological filtering. the infant s right leg are shown in frame 82 and 150 as white and black patches in a gray background. It is clear from Fig. 1(c) that the two black and white segments correspond well with the motion of the right leg, which moves along the horizontal direction. Moreover, the relative locations of the black and white segments reveal the direction of motion. Nevertheless, the segmented frames shown in Fig. 1(c) contain some spurious black and white patches of relatively small sizes. These patches are rather isolated and do not correspond to the moving body part. These patches can be eliminated, or at least reduced, by applying non-linear binary operations developed for mathematical morphology [25,27] Morphological filtering The frames produced by the segmentation process outlined above still contained spurious white patches (i.e. small groups of pixels assigned high intensity values by the segmentation process) as well as spurious black patches (i.e. small groups of pixels assigned low intensity values by the segmentation

5 N.B. Karayiannis, G. Tao / Image and Vision Computing 24 (2006) process). Such patches do not typically belong to body parts affected by the seizure and they resemble salt-and-pepper noise. The reduction of such spurious patches was attempted by relying on mathematical morphology [25,27]. More specifically, spurious patches were reduced by employing the OPENING and CLOSING morphological operations. The OPENING of an object A by a structuring element X, denoted as A+X, is the erosion of X by followed by the dilation of the result by X. In mathematical terms, A+XZ ða1xþ4x, where A1X and A4X denote the erosion and dilation of the object A by the structuring element X, respectively. The CLOSING of an object A by a structuring element X, denoted as A X, is the dilation of A by X followed by the erosion of the result by X.In mathematical terms, A XZ(A4X)1X. OPENING smoothes the contour of an object and breaks narrow bridges. CLOSING also smoothes the contour of an object. However, in contrast with OPENING, CLOSING fuses narrow breaks, eliminates small holes, and fills gaps in the contour. In this study, each of the segmented frames (containing three intensity levels) produced two binary frames. One of the binary frames contained only the white patches of the segmented frame while the rest of the frame (including the black patches) was considered to be background. The other binary frame contained only the black patches of the segmented frame while the rest of the frame (including the white patches) was considered to be background. The spurious black and white patches were reduced by applying a sequence of morphological operators to the binary frames. More specifically, each of the two binary frames was operated first by the OPENING morphological operator followed by the CLOSING morphological operator. The size of the structuring element was selected as the best compromise after testing and comparing circles within square windows of sizes 3!3, 5!5, and 7!7. The circular structuring element within a window of size 3!3 was not effective in eliminating spurious segments of size larger than 3!3 pixels. The circular structuring element within a 7!7 window altered the shape of the segments corresponding to the displacement of the infants moving body part(s). As a result, both OPENING and CLOSING operators employed the same structuring element, which was selected to be a circle within a 5!5 square window. The OPENING operator reduced the size of the big black and white patches corresponding to the moving body parts and eliminated the black and white patches that were relatively small in size, irregular in shape, and isolated from each other. The CLOSING operator that followed restored to some degree the size and shape of the black and white patches that belong to body parts affected by the seizure. The binary frames resulting after the application of these two morphological operators were subsequently used to restore the segmented frame. This was accomplished by copying the black and white patches left intact after the morphological operators and considering the rest of the frame to be background. Fig. 1(d) shows the frames produced by applying morphological filtering to the segmented frames shown in Fig. 1(c). It is clear that morphological filtering eliminated most of the noisy black and white patches that were not located on the infant s moving body part. Motion of the infant s right leg was clearly identified in frames 82 and 150. Comparison of Fig. 1(c) and (d) indicates that the size of the black and white patches generated by the motion of the infant s right leg was reduced slightly after the application of the OPENING and CLOSING morphological operators. The location of the black and white patches in Fig. 1(d) clearly reveals the direction of motion. Finally, morphological filtering eliminated most of the noisy black and white patches that can be seen in frames 1 and 190 of Fig. 1(c) Extraction of motion strength signals The motion strength signals were obtained by measuring the displacement areas of the infants moving body part(s). In order to improve the robustness of the proposed procedure to noise, the extraction of motion strength signal exploited the black and white patches produced by the segmentation process and refined by morphological filtering. Note that the black and white patches represent the areas occupied by the moving body part in two successive frames of the sequence. Since the black and white patches represent motion of the same body part between successive frames, their areas are expected to be equal or, at least, sufficiently close to each other. In the presence of noise, the area occupied by either the black patches or the white patches may be abnormally high. In order to reduce the sensitivity of the proposed procedure to noise, the displacement areas of the moving body part(s) were calculated as the minimum of the areas occupied by black patches and white patches Selection of quantitative features This section describes quantitative features obtained by analyzing motion strength signals extracted from video recordings of infants monitored for seizures. Such quantitative features were selected to convey some unique behavioral characteristics of neonatal seizures and non-seizure infant behaviors. Quantitative features can be extracted based on a global view of the temporal signals to represent some of their key properties and also to reveal their relationship with the underlying clinical event. As an example, such quantitative features should provide the basis for discriminating the sustained and rhythmic motion that characterizes focal clonic seizures and the rapid and jerky movements that are typical manifestations of myoclonic seizures. Analysis of motion strength signals in the time domain resulted in the following quantitative features: variance of time intervals, energy ratio, and maximum spike duration. The variance of the time intervals between the extrema, or simply the variance of time intervals, can be obtained from motion strength signals by computing the variance of the time intervals between any two adjacent spikes. The presence of needle-like spikes in motion strength signals reveals rapid motion of the infants body part(s) since a spike represents an abrupt increase of the displacement areas of the infants moving body part(s). This feature was introduced to measure

6 32 N.B. Karayiannis, G. Tao / Image and Vision Computing 24 (2006) the rhythmicity of the infants movements based on the observation that rhythmic movements would produce variance values close to zero. The variance of time intervals can be useful for distinguishing myoclonic seizures from focal clonic seizures since it is expected to take small values for focal clonic seizures but considerably higher values for myoclonic seizures. The feature energy ratio relies on the autocorrelation sequence, which may be used to measure the rhythmicity of motion manifested as quasi-periodic spikes in motion strength signals. The energy ratio of the autocorrelation sequence, or simply the energy ratio, was calculated as the ratio of the energy contained by the last 75% of the samples of the autocorrelation sequence to the energy contained by the first 25% of samples of the autocorrelation sequence. The energy ratio is expected to take large values for motion strength signals produced for focal clonic seizures due to the rhythmicity of motion that is their signature. In contrast, the energy ratio is expected to take small values for myoclonic seizures due to the fact that their autocorrelation sequence decays very fast while most of its energy is contained by a few samples near the origin. The maximum spike duration provides a quantitative measure of the speed of the infants movements. The maximum spike duration is expected to take the smallest values for myoclonic seizures, which are typically associated with rapid movements of short duration. Such values can differentiate neonatal seizures from random infant movements, which are typically slower and produce spikes of longer duration. Fig. 2. Temporal motion strength signals produced for the video recording of a focal clonic seizure affecting the infant s right leg: (a) selected frames of the sequence, (b) segmented frames produced by the proposed procedure, (c) motion strength signal produced by the original procedure, and (d) motion strength signal produced by the proposed procedure.

7 N.B. Karayiannis, G. Tao / Image and Vision Computing 24 (2006) Results This section presents some results of an experimental study, which relied on a large database containing a broad variety of neonatal seizures. This database, which was developed by the Clinical Research Centers for Neonatal Seizures (CRCNS), contains video/eeg/polygraphic recordings of several hundred individual seizures, which have been characterized and classified by a team of physicians in terms of their electrographic and behavioral features [28]. The video recordings were acquired by a single analog camera mounted above the infant s bed. The exact distance between the camera and the bed varied depending on the individual patient s body size, but was within the range of 0.9 and 1.5 m. The infant was positioned at the center of the bed just below the camera. A fixed zoom setting was used in the acquisition of the individual video recordings, which guarantees that the spatial resolution remained constant throughout each video recording. The CRCNS database contains simultaneous EEG and analog video recordings on a split screen. The analog video recordings were digitized using a PXC200A frame grabber. The temporal sampling rate was 30 frames/s, which is considered high enough to capture sudden and rapid motion. The digitized frames were of spatial resolution 352!240 pixels. Following the elimination of the EEG signals, the video recordings produced sequences Fig. 3. Temporal motion strength signals produced for the video recording of a myoclonic seizure affecting the infant s left leg: (a) selected frames of the sequence, (b) segmented frames produced by the proposed procedure, (c) motion strength signal produced by the original procedure, and (d) motion strength signal produced by the proposed procedure.

8 34 N.B. Karayiannis, G. Tao / Image and Vision Computing 24 (2006) of frames of size 203!240 pixels. No compression of any form was used at any stage of data acquisition and processing. Although this increased considerably the storage hardware required for this study, this choice was deemed necessary to ensure that the results of the experiments were not affected by data distortion due to a lossy compression scheme. The proposed procedure was tested and evaluated on a set of 240 video recordings of 54 patients exhibiting myoclonic seizures (80 cases), focal clonic seizures (80 cases), and random infant movements (80 cases) Extraction of motion strength signals The original procedure [18] and the procedure proposed in this paper were evaluated on the video recordings of neonatal seizures and normal infant behaviors not associated with seizures. Figs. 2 4 show the temporal motion strength signals extracted by both procedures from video recordings of focal clonic seizures, myoclonic seizures, and random infant movements in three different patients. The locations of the moving body parts during the clinical event are shown in representative frames of each video recording. The frames of the video recordings shown in Figs. 2 4 can be used as a reference to verify the consistency of the temporal signals with the corresponding clinical events. The values of the signals corresponding to the frames shown at the top of each figure are indicated by dots, while the moving body part in each video recording is shown within a box. Figs. 5 7 show the motion strength signals extracted by the proposed procedure in video Fig. 4. Temporal motion strength signals produced for the video recording of a random movement of the infant s left leg: (a) selected frames of the sequence, (b) segmented frames produced by the proposed procedure, (c) motion strength signal produce by the original procedure, and (d) motion strength signal produced by the proposed procedure.

9 N.B. Karayiannis, G. Tao / Image and Vision Computing 24 (2006) recordings of 10 additional patients exhibiting myoclonic seizures, focal clonic seizures, and random infant movements. Fig. 2 shows the motion strength signals produced by the original procedure and the procedure proposed in this paper for a focal clonic seizure affecting the infant s right leg. The motion strength signals produced for the focal clonic seizure shown in Fig. 2(a) reveal the rhythmicity that is the signature of such clinical events. Comparison of Fig. 2(c) and (d) indicates that the two procedures employed in the experiments identified motion at the same frames of the video recording. This becomes clear by the locations of the spikes produced by the two procedures. However, there are some noteworthy differences between the motion strength signals produced by the two procedures. The original procedure identified strong motion between frames 100 and 150. In contrast, the procedure proposed in this paper identified only weak motion in this time interval. On the other hand, the proposed procedure identified stronger motion right before and after frame 100. Similar differences can also be seen in the signals produced by the original and the proposed procedures for the time intervals between frames 0 and 50 and between frames 150 and 200. Frame-by-frame inspection of the video recording indicated that the motion of the infant s right leg was best represented by the motion strength signal produced by the proposed procedure. During the myoclonic seizure shown in Fig. 3, the infant s left leg is moving rapidly to the right and to the top of the frame between frames 10 and 16 (Fig. 3 shows frames 14 and 16 in this interval). The infant s left leg is moving slowly between frames 20 and 150. After frame 150, the infant s left leg moves slowly to the left and to the bottom of the frame for a number of frames and then it remains stable. Both procedures identified that the left leg moved between frames 10 and 16, which is indicated by the sizable spike in this time interval shown in Fig. 3(c) and (d). However, the proposed procedure produced a higher spike; this indicates that, according to the proposed method, the moving body part occupied a larger area of the frames. According to Fig. 3(d), the proposed procedure identified no motion after frame 17. In contrast, the original procedure produced some weaker spikes after the big spike between frames 10 and 16; these spikes can be attributed to Fig. 5. A single frame of the video recording, the segmented frame, and the motion strength signal produced by the proposed procedure for: (a) a myoclonic seizure affecting the infant s left hand, (b) a myoclonic seizure affecting the infant s left leg, (c) a myoclonic seizure affecting the infant s left leg, and (d) a myoclonic seizure affecting the infant s right hand.

10 36 N.B. Karayiannis, G. Tao / Image and Vision Computing 24 (2006) Fig. 6. A single frame of the video recording, the segmented frame, and the motion strength signal produced by the proposed procedure for: (a) a focal clonic seizure affecting the infant s left hand, (b) a focal clonic seizure affecting the infant s right hand, (c) a focal clonic seizure affecting the infant s right hand, and (d) a focal clonic seizure affecting the infant s left leg. noise that interferes with the extraction of the motion strength signal or to some weak motion that is not visually detectable when reviewing the video recording frame-by-frame. Nevertheless, even if the weak spikes in Fig. 3(c) represent motion, this motion is too weak to be attributed to the myoclonic seizure. Note that the typical signature of myoclonic seizures is the rapid motion of the infants extremities. Fig. 4 shows the motion strength signals produced by these procedures for the video recording of a random movement of an infant s left leg. The original procedure produced multiple sharp spikes throughout the duration of the video recording. The most significant spikes were those produced between frames 0 and 100. Frame-by-frame inspection of the video recording indicated that most of the spikes produced by the original procedure do not represent actual motion, but instead were caused by the susceptibility of the original procedure to noise and other recording imperfections. Such signals would be misleading if used for seizure recognition, since such sharp spikes are typical signatures of myoclonic seizures. Compared with the original procedure, the procedure proposed in this paper produced fewer isolated spikes between frames 50 and 100. Frame-by-frame inspection of the video recording verified that the motion strength signal produced by the proposed procedure constitutes a better and more accurate representation of the videotaped event. Fig. 5 shows the temporal motion strength signals produced by the proposed procedure for a set of four myoclonic seizures. In the myoclonic seizure shown in Fig. 5(a), the infant s left hand is moving from frame 68 to frame 74. However, the movement was hardly detectable at frame 70. The proposed procedure identified motion at frames 68 and 69 as indicated by the motion strength signal shown in Fig. 5(a). The proposed procedure correctly indicated that the fastest motion occurred between frames 71 and 74 (Fig. 5(a) shows frame 73). Fig. 5(b) shows the motion strength signal obtained for the same myoclonic seizure shown in Fig. 5(a), which also affected the infant s left leg. The left leg is moving to the left and to the bottom of the frame between frames 68 and 76 (Fig. 5(b) shows the infant at frame 70). The motion of the infant s left leg was quantified correctly by the proposed procedure, as indicated by the spikes in the motion strength signal shown in Fig. 5(b). This

11 N.B. Karayiannis, G. Tao / Image and Vision Computing 24 (2006) Fig. 7. A single frame of the video recording, the segmented frame, and the motion strength signal produced by the proposed procedure for: (a) a random movement of the infant s right leg, (b) a random movement of the infant s left hand, (c) a random movement of the infant s left hand, and (d) a random movement of the infant s left hand. motion strength signal indicates that there is no motion at frame 72, a fact that was verified by inspecting the video recording frame-to-frame. In the myoclonic seizure shown in Fig. 5(c), the infant s left leg is moving from frame 73 to frame 78. The proposed procedure quantified correctly the movement in this time interval. The proposed procedure did not reveal motion between frames 116 and 125 and between frames 204 and 244. Very slow motion during these time intervals was identified visually only after viewing the video recording at various speeds. Note that such slow motion was not detectable during an inspection of the video recording frame-by-frame. In the myoclonic seizure shown in Fig. 5(d), there is rapid motion of the infant s right hand between frames 100 and 105. During this time interval, the infant s right hand moves toward the bottom of the frame, stops suddenly, and then moves toward the top of the frame. This motion was captured correctly by the proposed procedure, as indicated by the two sharp spikes present in the motion strength signal shown in Fig. 5(d). Fig. 6 shows the temporal motion strength signals produced by the proposed procedure for a set of four focal clonic seizures. In the focal clonic seizure shown in Fig. 6(a), the proposed procedure captured the motion of the infant s left hand and quantified the rhythmicity that is the defining characteristic of focal clonic seizures. The proposed procedure missed a brief and slow movement toward the camera between frames 1 and 3 and some insignificant motion between frames 15 and 16. The highest spike produced by the proposed procedure was that located at frame 96. Frame-by-frame inspection of the video recording indicated that this was the most significant movement during this clinical event. Fig. 6(b) shows the motion strength signal produced by the proposed procedure for a focal clonic seizure affecting the infant s right hand. Fig. 6(b) indicates that the proposed procedure produced some isolated spikes that are almost periodic. This is consistent with the rhythmicity of motion caused by focal clonic seizures. Overall, the proposed procedure was successful in detecting and measuring the salient motion throughout the video recording but missed weak motion that was barely visible during a frame-by-frame inspection of the video recording. The focal clonic seizure shown in Fig. 6(c) affected the infant s right hand. The proposed procedure missed some weak motion of the infant s right hand between frames 110 and 115. Nevertheless, the proposed procedure captured the salient motion

12 38 N.B. Karayiannis, G. Tao / Image and Vision Computing 24 (2006) associated with this seizure and produced a motion strength signal that reveals the rhythmicity of the movements associated with this seizure. The focal clonic seizure shown in Fig. 6(d) affected the infant s left leg. The motion strength signal produced by the proposed procedure captured the rhythmicity of the movements associated with this seizure. In fact, the proposed procedure captured all movements of the infant s left leg. Fig. 7 shows the temporal motion strength signals produced by the proposed procedure for a set of four random infant movements. The proposed procedure captured the random movement of the infant s right leg shown in Fig. 7(a) well before frame 50. The infant s right leg is also moving between frames 53 and 64, but the proposed procedure captured only the relatively fast movement between frames 60 and 64. After frame 64, there is no motion. Fig. 7(b) shows the temporal motion strength signals produced by the proposed procedure for the video recording of a random movement of an infant s left hand. The proposed procedure produced spikes that represent the movements of the infant s left hand. These spikes are not as sharp as those produced for myoclonic seizures. This is consistent with the motion patterns that differentiate random movements from the rapid movements caused by myoclonic seizures. According to Fig. 7(c), the proposed procedure captured correctly the random movements of the infant s left hand from frame 210 to frame 222 and between frames 227 and 230. The proposed procedure missed the movement between frames 235 and 240. However, this movement was weaker and slower compared with the other movements observed in this video recording, which were accurately captured by the proposed procedure. In the video recording shown in Fig. 7(d), the infant s left hand is moving between frames 55 and 87 and from frame 223 to frame 238. The proposed procedure captured the significant motion from frame 60 to frame 86 and between frames 226 and 236. The proposed procedure missed only some insignificant motion observed between frames 104 and Evaluation of motion strength signals This section presents a statistical evaluation of the motion strength signals produced by the proposed procedure. In accordance with the ultimate goal of this project, the criterion used for this evaluation was the ability to separate the classes myoclonic seizure, focal clonic seizure, and random infant movement based on quantitative features obtained by analyzing motion strength signals. The proposed procedure was evaluated using a set of 240 short video recordings of 54 patients exhibiting myoclonic seizures (80 cases), focal clonic seizures (80 cases), and random movements (80 cases). This evaluation relied on the Fisher ratio (FR), which is defined for a two-class problem as [29] f ij Z ðm ikm j Þ 2 s 2 ; jsi; (1) i Cs2 j where m i (m j ) is the mean and s 2 i ðs 2 j Þ is the variance of the samples belonging to the class C i ðc j Þ. The Fisher ratio f ij provides a measure of the separability of the two classes C i and C j, with increasing values of the FR revealing improving class separability. Since the data set used in the experiments contains an equal number nz80 of cases from each of the three classes, the FR can also be used to test whether a feature provides a sufficient basis for separating any pair of the classes. This can be done by performing a t-test of the hypothesis that there is a statistically significant difference between the mean values of this feature for those classes [30]. For a given feature, this hypothesis p can be tested for a pair of classes C i and C j by comparing t ij Z ffiffiffiffiffiffi nf ij with the critical t-value corresponding to vzncnk2z158 degrees of freedom (df). With az0.05, a two-tailed t-test indicates that theaforementionedhypothesisistrueiff ij R The separability of multiple classes can be measured by the generalized Fisher ratio (GFR), which can be defined for a multi-class problem as [31] F Z 1 cðck1þ P ciz1 P c l jz1 i l j f ij jsi P ciz1 P c jz1 jsi l i l j ; (2) where c is the number of classes, l i and l j are the mixing proportions for classes C i and C j, respectively, and f ij is the FR for the classes C i and C j. Table 1 shows the FR values for the three features selected from motion strength signals produced by the procedure proposed in this paper. The FR was computed for three pairs of the classes: myoclonic seizure, focal clonic seizure, and random infant movement. Table 1 also shows the GFR values for the same features. Note also that some of the entries of Table 1 were shaded based on the outcomes of the t-test to indicate that the corresponding feature provides no sufficient basis for separating the corresponding pair of classes. Because several of the variables had skewed distributions, a nonparametric Mann Whitney test was also performed [30]. Nevertheless, in each case, the outcome of this test was consistent with that of the t-test. According to Table 1, the features selected from motion strength signals produced by the proposed procedure can be rated in terms of class separability as follows: variance of time intervals (highest GFR value), energy ratio, and maximum spike duration (lowest GFR value). Finally, Table 1 indicates that the most challenging problem is to distinguish random movements from either myoclonic or focal clonic seizures. The above conclusions are consistent with Fig. 8, which shows a scatter plot of the energy ratio and the variance of time intervals obtained for motion strength signals extracted by the proposed procedure from the video recordings of 40 cases of myoclonic seizures, 40 cases of focal clonic seizures, and 40 cases of random infant movements. According to Fig. 8, most of the myoclonic seizures produced high values of the variance of time intervals. The variance of time intervals can also differentiate focal clonic seizures from myoclonic seizures. However, this feature is not particularly reliable for distinguishing focal clonic seizures from random infant movements, a fact that is also consistent with the outcome of the statistical test. The feature energy ratio provides

13 N.B. Karayiannis, G. Tao / Image and Vision Computing 24 (2006) Table 1 FR and GFR values computed for three features selected from motion strength signals extracted from video recordings of 80 myoclonic seizures, 80 focal clonic seizures, and 80 random infant movements by the procedure proposed in this paper Class separability measure Fisher ratio Fisher ratio Fisher ratio Generalized Fisher ratio a satisfactory basis for discriminating between myoclonic seizures and focal clonic seizures and for distinguishing myoclonic seizures from random infant movements. However, this feature can only have a weak contribution to the discrimination between focal clonic seizures and random infant movements. This observation is in agreement with the statistical test and the corresponding FR value shown in Table 1. Overall, Fig. 8 also illustrates why differentiating random infant movements from either myoclonic seizures or focal clonic seizures is by far a more challenging problem than distinguishing myoclonic from focal clonic seizures. 4. Discussion Classes Variance of time intervals Energy ratio Maximum spike duration Focal clonic seizure/myoclonic seizure Myoclonic seizure/random movement Focal clonic seizure/random movement All three classes This paper outlined a procedure proposed for extracting quantitative motion information from video recordings of neonatal seizures in the form temporal motion strength signals. This procedure employs non-linear filtering, vector clustering, and morphological filtering. The outcome of the experimental study indicates that the proposed procedure is less susceptible to noise than the original procedure. The experiments indicated that the temporal motion strength signals produced by the proposed procedure capture and quantify the differences between myoclonic and focal clonic seizures. In the case of myoclonic seizures, the motion strength signals are consistent with the rapid and jerky movements that are the typical signatures of such events. The motion strength signals produced for such seizures contain a significant spike and occasionally a few weaker spikes. In the case of focal clonic seizures, the temporal motion strength signals contain multiple spikes that correspond very well with the rhythmicity that characterizes the movements of such clinical events. The motion strength signals produced by the proposed procedure for random movements of the infant s body parts contain few isolated spikes, which differentiate such movements from those caused by focal clonic seizures. The spikes produced for random movements are typically wider than those produced for myoclonic seizures; this provides the basis for distinguishing random movements of the infant s extremities and those corresponding to myoclonic seizures. The experimental study outlined in this paper also evaluated the results of the proposed procedure using as criterion the separability of the classes myoclonic seizure, focal clonic seizure, and random infant movement. This evaluation revealed a weakness of the proposed procedure when its results are used for distinguishing focal clonic seizures from random infant movements. This can be attributed to the fact that the proposed procedure occasionally underestimated the duration of the infants movements since it missed the beginning and the end of those movements. A potential solution to this problem is to extract motion strength signals by alternative motion segmentation methods based on optical flow computation [32]. This would be sensible only if the performance gain offered by optical flow computation is high enough to compensate for its computational requirements, which exceed by far those of the procedure proposed in this paper. The discrimination between focal clonic seizures and random infant movements may also be reinforced by combining motion strength signals with motion trajectory signals extracted from video recordings by motion trackers based on adaptive block matching [33] or block motion models [34,35]. Acknowledgements Fig. 8. Scatter plot of the energy ratio and the variance of time intervals obtained for motion strength signals extracted by the procedure proposed in this paper from video recordings of myoclonic seizures, focal clonic seizures, and random infant movements. The authors would like to thank Drs James D. Frost, Jr, Merrill S. Wise, and Eli M. Mizrahi, from the Department of Neurology of the Baylor College of Medicine, for providing the video recordings and assisting with the formation of the data set used in the evaluation of the methods. This work was supported by the National Institute of Biomedical Imaging and Bioengineering under Grant 1 R01 EB00183.