Evaluation of New Online Automated Embolic Signal Detection Algorithm, Including Comparison With Panel of International Experts

Transcription

1 Evaluatio of New Olie Automated Embolic Sigal Detectio Algorithm, Icludig Compariso With Pael of Iteratioal Experts Marisa Culliae, BSc; Greg Reid, BSEE; Ralf Dittrich; Zolta Kaposzta, MD; Rob Ackerstaff, PhD; Vike Babikia, MD; Dirk W. Droste, MD; Doald Grossett, MD; Mario Siebler, MD; Luc Valto, MD; Hugh S. Markus, DM Backgroud ad Purpose The cliical applicatio of Doppler detectio of circulatig cerebral emboli will deped o a reliable automated system of embolic sigal detectio; such a system is ot curretly available. Previous studies have show that frequecy filterig icreases the ratio of embolic sigal to backgroud sigal itesity ad that the icorporatio of such a approach ito a offlie automated detectio system markedly improved performace. I this study, we evaluated a olie versio of the system. I a sigle-ceter study, we compared its performace with that of a huma expert o data from 2 cliical situatios, carotid steosis ad the period immediately after carotid edarterectomy. Because the huma expert is curretly the gold stadard for embolic sigal detectio, we also compared the performace of the system with a iteratioal pael of huma experts i a multiceter study. Methods I the sigle-ceter evaluatio, the performace of the software was tested agaist that of a huma expert o 20 hours of data from 21 patiets with carotid steosis ad 18 hours of data from 9 patiets that was recorded after carotid edarterectomy. For the multiceter evaluatio, a separate 2-hour data set, recorded from 5 patiets after carotid edarterectomy, was aalyzed by 6 differet huma experts usig the same equipmet ad by the software. Agreemet was assessed by determiig the probability of agreemet. Results I the 20 hours of carotid steosis data, there were 140 embolic sigals with a itesity of 7 db. With the software set at a cofidece threshold of 60, a sesitivity of 85.7 ad a specificity of 88.9 for detectio of embolic sigals were obtaied. At higher cofidece thresholds, a specificity 95 could be obtaied, but this was at the expese of a lower sesitivity. I the 18 hours of post carotid edarterectomy data, there were 411 embolic sigals of 7-dB itesity. Whe the same cofidece threshold was used, a sesitivity of 95.4 ad a specificity of 97.5 were obtaied. I the multiceter evaluatio, a total of 127 evets were recorded as embolic sigals by at least 1 ceter. The total umber of embolic sigals detected by the 6 differet ceters was 84, 93, 108, 92, 63, ad 78. The software set at a cofidece threshold of 60 detected 90 evets as embolic sigals. The mea probability of agreemet, icludig all huma experts ad the software, was 0.83, ad this was higher tha that for 2 huma experts ad lower tha that for 4 huma experts. The mea values for the 6 huma observers were averaged to give P 0.84, which was similar to that of the software. Coclusios By usig the frequecy specificity of the itesity icrease occurrig with embolic sigals, we have developed a automated detectio system with a much improved sesitivity. Its performace was equal to that of some huma experts ad oly slightly below the mea performace of a pael of huma experts (Stroke. 2000;31: ) Key Words: carotid artery diseases cerebral embolism observer variatio sigal processig, computer-assisted ultrasoography Trascraial Doppler ultrasoud (TCD) ca be used to detect asymptomatic emboli i the cerebral circulatio. 1 Embolic sigals have bee detected i patiets with a wide variety of potetial embolic sources, icludig symptomatic ad asymptomatic carotid steosis, 2,3 atrial fibrillatio, 4,5 ad prosthetic cardiac valves. 6 They have also bee detected durig ad after surgical procedures, icludig carotid edarterectomy 1,7 ad cardiopulmoary bypass. 8 Received Jauary 31, 2000; fial revisio received March 21, 2000; accepted March 21, From the Departmet of Cliical Neuroscieces (M.C., R.D., Z.K., H.S.M.), St George s Hospital Medical School, Lodo, UK; Nicolet-EME GmbH (G.R.), Kleiostheim, Germay; the Departmet of Neurology (R.D., D.W.D.), Muster, Germay; the Departmet of Cliical Neurophysiology (R.A.), St Atoius Hospital, Utrecht, Netherlads; the Departmet of Neurology (V.B.), Bosto Uiversity of Medicie, Bosto, Mass; the Departmet of Neurology (D.G.), Souther Geeral Hospital, Glasgow, UK; the Departmet of Neurology (M.S.), Uiversity of Dusseldorf, Dusseldorf, Germay; ad the Departmet of Neurology (L.V.), Uiversity of Toulouse, Toulouse, Frace. Correspodece to Prof Hugh Markus, Cliical Neurosciece, St George s Hospital Medical School, Cramer Terrace, Lodo, SW17 ORE, Eglad. h.markus@sghms.ac.uk 2000 America Heart Associatio, Ic. Stroke is available at

2 1336 Stroke Jue 2000 Icreasig evidece suggests that at least i certai situatios, embolic sigals do have cliical sigificace, ad embolic sigals have bee foud to be predictive of icreased stroke ad trasiet ischemic attack risk i patiets with asymptomatic or symptomatic carotid steosis 9 11 ad i patiets durig the immediate post carotid edarterectomy period. 12 The techique has a umber of promisig applicatios, icludig the selectio of high-risk patiets for appropriate surgical ad pharmacological itervetio, determiig the pathophysiology of stroke i idividual cases, assessig the effectiveess of ovel atiplatelet therapies, ad perioperative moitorig. The major techical impedimet to its widespread cliical use is the lack of a reliable automated method of embolic sigal detectio. The prevalece of embolic sigals may be low, requirig may hours of patiet recordigs to detect oly 1 or 2 embolic sigals. Curretly, the gold stadard is to record the Doppler sigal oto tape ad review it later i real time. This is extremely time-cosumig ad oly practical for research studies. Previous attempts at producig a automated system have failed to reach the same level of sesitivity ad specificity as the curret gold stadard of the huma aalyst The most promisig system to date has bee a eural etwork, 13 but although good levels of specificity were achieved, sesitivity of the system remaied relatively low. The major difficulty i previous systems has bee the detectio of low-itesity embolic sigals ad their differetiatio from ormal Doppler speckle. Therefore, systems have teded to perform better for the more itese embolic sigals see i patiets with prosthetic cardiac valves but have bee ureliable for the less itese sigals detected i patiets with carotid steosis ad atrial fibrillatio. Ay sigal-processig system that will improve the embolic sigal to Doppler blood sigal itesity ratio is likely to aid detectio of these low-itesity sigals. A characteristic feature of embolic sigals is that they are frequecy-focused, havig a maximum itesity over a arrow frequecy rage. Recet work has demostrated that the embolic sigal to Doppler blood sigal itesity ratio ca be icreased by 3 db by the use of frequecy-filterig techiques. 16 For automated sigal detectio, it is also essetial to determie which characteristics allow optimal discrimiatio of embolic sigals from both Doppler speckle ad artifact. Usig a ovel sigal aalysis approach, which provided both high temporal ad frequecy resolutio, we determied these characteristics i a previous study. 17 The frequecy-filterig approach that we developed was computatioally itesive; therefore, we adapted the approach by usig the fast Fourier trasform (FFT) as a frequecy filter, aalyzig the output from each frequecy bi idepedetly. I a offlie system, we obtaied a high sesitivity for embolic sigals usig this approach. 17 I the preset study, we describe the implemetatio of this system olie ad its testig i a umber of situatios. We have tested it with the use of large amouts of data from 2 cliical situatios, carotid artery steosis ad the period after carotid edarterectomy. The curret gold stadard for embolic sigal detectio is the experieced huma observer; therefore, i additio, we have compared the performace of the software with that the performace of a pael of experieced huma observers from a umber of ceters with extesive research experiece i the techique. Subjects ad Methods Trascraial Doppler Recordigs All recordigs were made from the middle cerebral artery (MCA). A Nicolet/EME Pioeer TC4040 TCD machie was used for all recordigs with a stadard 2-MHz trasducer. Depth settigs were betwee 45 ad 54 mm, ad a sample volume of 5 mm was used. Doppler sigals were recorded oto digital audio tape for subsequet aalysis. All aalysis was performed by a experieced observer blided to patiet group; all cadidate sigals were the reviewed by a secod observer, ad embolic sigals were icluded oly if both observers agreed. Stadard criteria o the FFT spectral display were used to idetify embolic sigals i additio to the characteristic chirpig or clickig soud. 18 Data for Sigle-Ceter Evaluatio First Patiet Group: Carotid Steosis TCD recordigs were obtaied from the ipsilateral MCA i 21 patiets with 70 carotid artery steosis. The group comprised 18 symptomatic ad 3 asymptomatic patiets. Recordigs were performed for 1 hour i 19 patiets ad 30 miutes i 2 patiets, resultig i a 20-hour data set. Secod Patiet Group: Carotid Edarterectomy Ipsilateral MCA recordigs were made i 9 patiets after carotid edarterectomy, startig 30 miutes after ski closure ad cotiuig for 2 hours, resultig i a 18-hour data set. No patiets were patched. Data for Multiceter Evaluatio Recordigs, startig at least half a hour after ski closure, were made from the ipsilateral MCA of patiets after carotid edarterectomy. The recordig duratio was 20 miutes i 3 patiets ad 30 miutes i 2 patiets, resultig i a 2-hour data set. Six idetical copies of these data sets were made ad set to participatig ceters. Embolic Sigal Detectio Algorithm Desig The algorithm uses a covetioal FFT to aalyze the quadrature audio sigal. To make the algorithm idepedet of user iterferece, the algorithm computes a additioal FFT, ot displayed to the user, based o the audio sigal. To achieve high time ad frequecy resolutio, this FFT is always 64 poit ad is computed oce every millisecod. A Haig fuctio is used to widow the FFT with the overlap fixed at 89, regardless of the FFT displayed to the user. The algorithm cotiually calculates a average backgroud sigal level over a rage of FFT colums from 750 ms before each evet to 750 ms after each evet. The backgroud level is determied by usig a 2D media filter over the whole sigal, except for those frequecies immediately adjacet to zero frequecy. At the same time, ay evets that are of sigificat area (o the time-frequecy surface) ad 3 db i itesity above the backgroud level are further aalyzed as embolic sigal cadidates. Cadidate evets are aalyzed as 3D volumes i itesity-time-frequecy space just above the backgroud-level time-frequecy surface. For each possible evet foud, the followig parameters were measured: 1. Peak itesity volume: The volume uder the itesity frequecy curve is cosidered i 1-ms periods, each of which costitutes a itesity volume. The maximum of these is referred to as the peak itesity volume. 2. Peak itesity: This is the peak itesity at ay frequecy or time measured above the backgroud level surface. 3. Evet frequecy disorder: This parameter measures how iterally cosistet or repeatable the shape of the evet volume is, measured from the highest to the lowest frequecy coordiates of the evet volume.

3 Culliae et al Olie Embolic Sigal Detectio Evet time disorder: This parameter measures how iterally cosistet or repeatable the shape of the volume is, measured from the highest to the lowest time coordiates of the evet volume. 5. Itesity volume reflectio ratio: This is the ratio of total itesity volume of the evet to the itesity volume of the equivalet area reflected across the zero frequecy lie. 6. Evet distace to zero frequecy: This is the frequecy differece betwee the evet ceter ad zero frequecy; evets with reverse flow directios always have egative distace. 7. Teardrop shape of evet area ear zero frequecy: It is ofte the case that a artifact ca be cut off by TCD high-pass filters, resultig i a teardrop shape i the spectral display. 8. Localizatio i frequecy: This is a iverse measure of the extet i frequecy of a evet. 9. Nearby high-itesity speckle iterferece: The presece of speckle ear a evet i time or frequecy idicates that the evet is ulikely to be a embolus if the earby speckle ad the evet are of similar itesity. The optimal parameters were derived from previous detailed aalysis of embolic sigals. 17 This demostrated that a artifact could be differetiated by the bidirectioality of the itesity icrease ad by the observatio that the maximal itesity was at low frequecy. The evet disorder parameters metioed were developed from the use of oliear forecastig, which has show that embolic sigals are highly ordered i time ad frequecy. Previous work has used the time domai data with this techique to distiguish embolic sigals from speckle ad artifact. 19 This algorithm differs by workig o the frequecy domai (with use of FFT data). Each of these parameters has some value i predictig whether a evet is a embolic sigal or a artifact sigal or either. For example, time or frequecy disorder teds to be low for embolic sigals but high for speckle ad artifact, whereas the itesity volume reflectio ratio is ear 1 oly for a artifact. Each parameter is coverted ito a idex represetig a estimate of the probability of a evet beig a embolic sigal or a artifact. All the parameters are aggregated by usig stadard techiques from fuzzy logic. This results i both a embolus probability (or cofidece level) ad a artifact probability, expressed as a percetage, that a give sigal is a embolic sigal or that it is a artifact. Evets with a high artifact probability are the idetified as such ad rejected as embolic sigals. Remaiig evets with a high embolus probability are idetified as embolic sigals. Each evet (embolic sigal or artifact) is saved to disk alog with its frequecy coordiates, time extet coordiates, decibel itesity, ad embolus probability level, allowig the evet to be viewed ad further aalyzed offlie. Data Aalysis Sigle-Ceter Aalysis All 38 hours of data were played through the automated software. All embolic sigals saved by the software were reviewed by a experieced huma observer. The exact time ad appearace of each sigal detected by the software was matched agaist the times recorded by the huma expert. Each sigal detected by the software was classified as a true positive sigal if it appeared as a characteristic embolic sigal ad matched the time oted by the huma aalysis. The software detected a few additioal sigals. These were reviewed ad classified as false positives if they did ot match the stadard criteria for embolic sigals. 18 There were a few sigals that were detected by the software ad appeared to be clear embolic sigals but that had bee missed by the huma observer. These were classified as true embolic sigals. The itesity of each embolic sigal was determied from the color-coded itesity scale as previously described. 9 Iterobserver agreemet is relatively poor for embolic sigals of very low itesity; therefore, may ceters use a itesity threshold as oe criterio for embolic sigal detectio. 18 Therefore, we used our stadard threshold of 7 db as oe criterio. Sigals detected by the software that were characteristic of embolic sigals 18 but were below the 7-dB itesity threshold were classified as extra sigals. All other saved sigals were classified as false-positive sigals. Sesitivity ad specificity were the calculated with the TABLE 1. Number of True-Positive ad False-Positive Embolic Sigals Detected With Correspodig Sesitivity ad Specificity for Software Set at Differet Embolus Probability Thresholds, Without Use of Decibel Threshold Embolus Probability Threshold, True False Sesitivity, Specificity, The total umber of embolic sigals was 140. threshold for sigal detectio set at differet embolus probability levels i icremets of 5. Multiceter Aalysis Each ceter aalyzed the tape by playig the recordig back through a Nicolet-EME TCD machie ad moitorig both the audio sigal ad visual Doppler spectra with use of the fastest sweep speed available to achieve best FFT temporal overlap. They were asked to apply stadard criteria for embolic sigal detectio 18 as they would for research or cliical studies i their departmet. No itesity threshold was specified. This is because absolute measuremets of itesity vary with the method of measuremet 18 ad would ot be the same i each ceter. The exact time of each embolic sigal was the oted by the observer. All ceters were blided to patiet details. The results were retured to the coordiatig ceter, where a idepedet observer had also aalyzed the same recordig. The same 2-hour tape was aalyzed by the olie software. The times of all embolic sigals detected by all observers, icludig the software, were oted. A itesity value for each true embolic sigal was determied from the itesity color-coded spectral display as previously described 9 ad calculated by the coordiatig ceter. Iterobserver agreemet was determied by a method based o the proportio of specific agreemet, as has previously bee used to examie iter-rater agreemet i embolic sigal detectio. 20 This allows calculatio of the probability that a specified observer will detect a embolic sigal compared with the performace of 1 or more other observers. A value of 1 idicates perfect agreemet; 0, o agreemet. The iclusio of a itesity threshold as oe criterio for embolic sigal idetificatio was ot possible because of the measuremet of itesity by use of differet methodologies i differet ceters. However, the probability tests were performed both with ad without the applicatio of a 7-dB itesity threshold by usig the itesity measuremets from the origial aalysis by the coordiatig ceter. Results Sigle-Ceter Evaluatio First Patiet Group: Carotid Steosis Huma aalysis of the data detected 134 embolic sigals with itesity 7 db. Software aalysis of the data detected a additioal 6 embolic sigals that were 7 db but had bee missed by the huma observer, makig the total umber of true positive sigals 140. Whe the threshold for detectio was set at a embolus probability value of 65, a sesitivity of 85.7 ad a specificity of 88.9 were obtaied. Table 1 shows how the performace varied accordig to the embolus

4 1338 Stroke Jue 2000 TABLE 2. Number of True-Positive ad False-Positive Embolic Sigals Detected With Correspodig Sesitivity ad Specificity for Software Set at Differet Embolus Probability Thresholds, With >7-dB Threshold Embolus Probability Threshold, True False Sesitivity, Specificity, The total umber of true embolic sigals was 411. probability or cofidece threshold at which the software was used. At higher thresholds, a specificity of 95 could be obtaied, but this was at the expese of a lower sesitivity. Of those 20 true embolic sigals missed at a embolus probability threshold settig of 65, 6 were missed because of the presece of coicidet artifact, leadig to categorizatio as a artifact, whereas 7 were missed probably because they were low-itesity embolic sigals that the software failed to detect. I the 7 remaiig cases, the reaso for the failure to detect the embolic sigal was uclear. Secod Patiet Group: Carotid Edarterectomy Huma aalysis of the data detected 402 embolic sigals of 7-dB itesity. Software aalysis of the data detected a additioal 9 embolic sigals that were 7 db but had bee missed by the huma observer, makig the total umber of true positive sigals 411. After aalysis of the first patiet group, we foud that usig a embolus probability threshold of 65 produced the same sesitivity as usig a embolus probability threshold of 60 but that it icreased specificity. For the purpose of this aalysis, the miimum threshold used was 65. With use of this threshold, a sesitivity of 95.4 ad a specificity of 97.5 were obtaied. Table 2 shows the effect of alterig the threshold o the sesitivity ad specificity. Of those 19 true embolic sigals missed at a embolus probability threshold settig of 65, 8 were missed because TABLE 3. Proportio of Specific Agreemet Betwee Idividual Ceters of the presece of a coicidet artifact, leadig to categorizatio as a artifact, whereas 11 were missed probably because they were low-itesity embolic sigals that the software failed to detect. The mea SD itesity of the embolic sigals, as defied above, icludig a 7-dB itesity threshold, was sigificatly higher i the carotid edarterectomy group tha i the carotid steosis group: versus db (P ). Multiceter Evaluatio A total of 127 evets were recorded as embolic sigals by at least 1 ceter. The total umber of embolic sigals detected by the 6 differet ceters were 84, 93, 108, 92, 63, ad 78. With the software set at a embolus probability threshold of 60, 90 evets were detected as embolic sigals. The probability that a observer from a secod ceter would detect a embolic sigal if a observer from 1 ceter had also detected a embolic sigal is show i Table 3. The values for each ceter have bee averaged to give a mea value, as show i the rightmost colum of Table 3. The mea probability of agreemet value for the software was 0.83, ad this was higher tha that for 2 ceters ad lower tha that for 4 ceters. The mea values for the 6 huma observers were averaged to give a probability of agreemet value of 0.84, which was similar to that of the software. Two ceters detected rather differet umbers of embolic sigals, with ceter 3 detectig 108 evets ad ceter 5 detectig 63 evets. O review, ceter 5 had missed some clear embolic evets, whereas ceter 3 had icluded some evets that appeared ot to fulfill the stadard criteria for embolic sigals. 18 I view of this ad to evaluate the software agaist the most rigorous criteria, amely, the best huma experts, the aalysis was repeated with the data from ceters 3 ad 5 omitted. The results are show i Table 4. The values for each ceter have bee averaged to give a mea probability value as show i the rightmost colum of Table 4. The mea value for the software was 0.85, ad this was higher tha that for 1 ceter ad lower tha that for 2 ceters. The mea values for the 4 huma observers were averaged to give a value of 0.87, which was oly slightly higher tha that for the software. There was a highly sigificat relatioship betwee the proportio of ceters agreeig that a certai sigal was a Software Mea Ceter 1 x x x x x x Software x 0.83 Numbers 1 to 6 represet huma ceters. Mea values for each ceter ad software i right-most colums.

5 Culliae et al Olie Embolic Sigal Detectio 1339 TABLE 4. Proportio of Specific Agreemet Betwee 4 Idividual Ceters Software Mea Ceter 1 x x x x Software x 0.85 Numbers 1, 2, 4, ad 6 represet huma ceters. Mea values for each ceter ad software are show i rightmost colums. embolic sigal ad of a certai itesity (Spearma 0.667, P 0.001). All embolic sigals detected by 1 or more ceters were reevaluated. ad their itesities were measured by the stadard method used i the coordiatig ceter. 9 The aalysis was the repeated for all ceters usig a 7-dB itesity threshold as a additioal criteria i the defiitio of a embolic sigal. The results comparig the performace of the software with that of all ceters are show i Table 5. The mea probability value for the software was 0.87, ad this was higher tha that for 2 ceters ad lower tha that for 4 ceters. The mea values for the 6 huma observers were averaged to give a value of 0.88 which was similar to that of the software. As discussed above, the aalysis was repeated after omittig data from ceters 3 ad 5, which had reported otably more ad otably less embolic sigals, respectively, tha the other ceters. The results are show i Table 6. The mea value for the software was 0.87, ad this was higher tha that for 1 ceter ad lower tha that for 2 ceters. The mea values for the 4 huma observers were averaged to give a value of 0.90, which was oly slightly higher tha that of the software. TABLE 6. Proportio of Specific Agreemet Betwee 4 Idividual Ceters ad Software Whe >7-dB Itesity, as Part of Defiitio of Embolic Sigal, Is Used Software Mea Ceter 1 x x x x Software x 0.87 Numbers 1, 2, 4, ad 6 represet huma ceters. Mea values for each ceter ad software are show i rightmost colums. Discussio The preset study tested a ew olie algorithm for the automated detectio of embolic sigals that is based o a ovel approach that uses the fact that embolic sigals have a maximum itesity over a arrow frequecy rage. We demostrated cosistetly good performace across 2 cliical situatios, with the software compared with expert observers i 1 ceter ad with the software compared with a iteratioal pael of experts from a umber of ceters. The results are cosiderably better tha those of ay previously published automated detectio system, 21 ad the levels of sesitivity ad specificity obtaied are of the order of those required if automated aalysis is to replace the traied huma observer i the detectio of embolic sigals. I the first half of the study, the software was aalyzed i a sigle ceter ad compared with the performace of traied huma observers. We studied data sets from 2 cliical situatios, carotid artery steosis ad the period after carotid edarterectomy. I both situatios, good levels of detectio were obtaied, with a sesitivity ad specificity of 85.7 ad 88.9, respectively, for carotid steosis ad 95.4 ad 97.5, respectively, for the period after carotid edarterectomy. This was with the threshold for detectio set at 65. Higher specificity could be obtaied whe the threshold was raised, but this was at a loss of sesitivity. I may cliical situatios, the frequecy of embolic sigals is low. I patiets with carotid steosis ad atrial fibrillatio, the media umber per hour i embolic sigal positive patiets is oly 1 to 3. 3,4 I such situatios, the sesitivity must be high; therefore, a threshold set at 65 would be optimal. This the requires a experieced observer to check the saved segmets after aalysis. This is facilitated because the software saves the detected segmets with the correspodig audio soudtrack so that they ca be rapidly reviewed. This setup is similar to that curretly used by 24-hour ECG moitorig aalysis systems. TABLE 5. Proportio of Specific Agreemet Betwee Idividual Ceters Whe >7-dB Itesity, as Part of Defiitio of Embolic Sigal, Is Used Software Mea Ceter 1 x x x x x x Software x 0.87 Numbers 1 to 6 represet huma ceters. Mea values for each ceter ad software are show i rightmost colums.

6 1340 Stroke Jue 2000 The performace of the software was better for the embolic sigals detected after carotid edarterectomy tha for those i patiets with carotid steosis. Our cliical impressio was that the embolic sigals recorded i the post carotid edarterectomy patiets were of greater itesity, ad this was cofirmed by quatitative sigal aalysis. This is likely to be the reaso for the improved performace of the software i the post carotid edarterectomy group. However, the mea itesity values i the post carotid edarterectomy patiets were below those that are commoly foud i patiets with prosthetic heart valves or durig itervetioal radiological procedures, whe the majority of embolic sigals are believed to be due to gaseous emboli. We believe that this is due to the fact that embolic sigals i the post carotid edarterectomy settig represet larger solid emboli, ad this is supported by their marked reductio after admiistratio of the atiplatelet aget S-itrosoglutathioe. 22 This emphasizes that it is importat that a automatic detectio system be tested o the data set that it will be subsequetly be used o. Oe might expect the performace of this system to be eve better i patiets with prosthetic heart valves, i whom embolic sigals are of eve greater itesity, 23 but this eeds testig. Similarly, the software eeds evaluatig i patiets with atrial fibrillatio, i whom embolic sigals are ifrequet ad of low itesity 4 ; i this group, its performace may ot be as good. Our iitial pilot data suggest a sesitivity of oly about 50 i this group. I the sigle-ceter study, we evaluated the detectio of embolic sigals defied as havig a itesity of 7 db. This is the stadard itesity threshold that we use i all our studies. The presece of embolic sigals, defied by usig this threshold, has bee show to be predictive of stroke ad trasiet ischemic attack risk i patiets with carotid artery steosis. 9 Iterobserver agreemet is ot as good for embolic sigals of very low itesity, ad this was cofirmed i our multiceter study. The use of a itesity threshold has bee show to icrease reproducibility without too great a loss of sesitivity, ad its use is recommeded i recet cosesus criteria. 18 A difficulty with this approach i the evaluatio of a automated system is that the software may detect embolic sigals that have a itesity below the threshold. This occurred i the preset study, but for the purpose of the study, these sigals were excluded ad couted as either true positives or true egatives. However, i olie use, the software calculates the itesity of ay evets its saves; therefore, istructig it to detect embolic sigals oly above a certai threshold is possible. It should be remembered that the absolute itesity value depeds o the method i which it is calculated ad, i particular, how the relative itesity of both the embolic sigal ad backgroud are calculated. 24 The 7-dB threshold was determied by usig previously described methods from the color-coded itesity scale 24 ad ot calculated by the computer algorithm. Therefore, the absolute value of a appropriate threshold for use by the software is ot ecessarily 7 db but could be easily determied. I the secod part of the preset study, we evaluated the software agaist a pael of experieced huma observers. We felt that this was appropriate because the preset gold stadard for embolic sigal idetificatio is the huma observer. Although geerally high levels of agreemet betwee huma experts have bee foud i previous work, 20,25 there is some iterobserver variatio. Therefore, we determied whether the software was as good as a experieced huma observer. Whe compared for all 6 ceters, the performace of the software was very similar, with a mea proportio of specific agreemet of 0.83 compared with 0.84 for the averaged value for the huma experts. Two ceters performed sigificatly differetly from the other ceters; oe reported fewer embolic sigals, whereas the other reported additioal embolic sigals, which the other ceters did categorize as embolic sigals. This reflects the fact that very-low-itesity embolic sigals may occur, ad oly certai ceters icluded these as defiite sigals, which they would report as embolic sigals whe performig the techique for research ad cliical studies. Essetially, some ceters seem to be usig a implicit itesity threshold. To provide the most rigorous test of the software, we reevaluated it agaist the best humas experts, with the data from these 2 outlyig experts removed. Eve this selected group of huma experts performed oly slightly better tha the software, with a averaged value of 0.87 compared with 0.84 for the software. Embolic sigals that were most commoly disagreed o betwee ceters, icludig the software, were those of low itesity, as reported i previous studies, 20,25 ad this emphasizes the beefit of applyig a itesity threshold. Our results also demostrate that despite the publicatio of detailed cosesus criteria for the detectio of embolic sigals, 18 a miority of experieced research ceters are ot applyig these rigorously. This emphasizes the eed for cotiuig iterceter reproducibility studies as part of a ogoig quality cotrol program. There is also the potetial problem of huma error, particularly whe large amouts of data have to be aalyzed. Eve i our sigle-ceter evaluatio, we foud a umber of typical embolic sigals of itesity 7 db that had bee missed by the huma observer but were detected by the software; this costituted 4 of the carotid steosis embolic sigals ad 2 of the carotid edarterectomy embolic sigals. This emphasizes a major potetial advatage of a automated system over the huma observer; it does ot suffer from fatigue. The software missed oly 5 to 10 of embolic sigals i the differet data sets. I some cases, these were lowitesity sigals. The detectio of these may be improved by adjustmets to the algorithm, but the detectio of some may require a differet method of sigal aalysis that maximizes the embolic sigal to Doppler blood sigal itesity ratio further. For example, the wavelet trasform is particularly suited to the aalysis of short-duratio trasiet sigals, ad pilot data suggest that it describes embolic sigals better tha the FFT. 26 I other cases, the software missed embolic sigals that occurred at the same time as artifact. I these cases, the use of a multigate system may improve detectio. This could work i a offlie mode, with detectio occurrig i 1 chael but with 2 chaels beig saved for subsequet review by the huma expert. I ucertai cases, embolic sigals could the be idetified by the time delay occurrig betwee the proximal ad distal chaels. 27 This would also help distiguish betwee embolic sigals ad artifact o the

7 Culliae et al Olie Embolic Sigal Detectio 1341 rare occasios o which a predomiatly, but ot exclusively, uidirectioal sigal occurs with maximum itesity at low velocity; such sigals ca result from emboli or, rarely, artifact. For the first time, this automated system provides a method with sufficiet performace for routie cliical use. However, potetial limitatios eed to be bore i mid. First, we have oly validated its use i 2 situatios, carotid steosis ad the period after carotid edarterectomy. Its performace eeds to be similarly evaluated i other situatios. It may work less well for the very few ifrequet sigals see i patiets with atrial fibrillatio. It may also work less well i patiets with the more itese embolic sigals occurrig durig, rather tha after, itervetioal procedures, such as cardiopulmoary bypass, carotid edarterectomy, ad cerebral agiography. A proportio of the emboli i such situatios are believed to arise from gaseous bubbles ad to result i embolic sigals of much higher itesity. This ca lead to receiver overload ad a degree of aliasig. This appears as a bidirectioal itesity icrease ad may lead to mistake idetificatio of embolic sigals as artifact. However, modificatio of the algorithm ad the use of a TCD system with sufficiet dyamic rage should overcome this problem. Secod, although we tested the algorithm o a large amout of uselected routie cliical data, it may work less well o certai data sets. Third, the saved sigals eed to be reviewed by a huma expert for optimal performace; therefore, the use of the software eeds to be combied with appropriate traiig. Despite these potetial limitatios, this automated system is sigificatly better tha previous published approaches ad, for the first time, provides a system whose performace is similar to that of the huma expert, the curret gold stadard. Ackowledgmets This study was supported i part from a project grat from the British Heart Foudatio (PG96176). We thak Dr Jae Molloy for help with patiet moitorig. Refereces 1. Specer MP, Thomas GI, Nicholls SC, Sauvage LR. Detectio of middle cerebral artery emboli durig carotid edarterectomy usig trascraial Doppler ultrasoography. Stroke. 1990;21: Siebler M, Kleischmidt A, Sitzer M, Steimetz H, Freud HJ. Cerebral microembolism i symptomatic ad asymptomatic high-grade iteral carotid artery steosis. Neurology. 1994;44: Markus HS, Thomso ND, Brow MM. Asymptomatic cerebral embolic sigals i symptomatic ad asymptomatic carotid artery disease. Brai. 1995;118: Culliae M, Waiwright R, Brow A, Moagha M, Markus HS. Asymptomatic embolizatio i subjects with atrial fibrillatio ot takig aticoagulats: a prospective study. Stroke. 1998;29: Sliwka U, Job FP, Wissuwa D, Diehl RR, Flachskampf FA, Harath P, Noth J. Occurrece of trascraial Doppler high-itesity trasiet sigals i patiets with potetial cardiac sources of embolism: a prospective study. Stroke. 1995;26: Rams JJ, Davis DA, Lolley DM, Berger MP, Specer M. Detectio of microemboli i patiets with artificial heart valves usig trascraial Doppler: prelimiary observatios. J Heart Valve Dis. 1993;2: Gaut ME. Trascraial Doppler: prevetig stroke durig carotid edarterectomy. A R Coll Surg Egl. 1998;80: Gallagher EG, Pearso DT: Ultrasoic idetificatio of sources of gaseous microemboli durig ope heart surgery. Thorax. 1973;28: Molloy J, Markus HS. Asymptomatic embolizatio predicts stroke ad TIA risk i patiets with carotid artery steosis. Stroke. 1999;30: Siebler M, Nachtma A, Sitzer M, Rose G, Kleischmidt A, Rademacher J, Steimetz H. Cerebral microembolism ad the risk of ischemia i asymptomatic high-grade iteral carotid artery steosis. Stroke. 1995; 26: Valto L, Larrue V, Pavy le Trao A, Massabuau P, Geraud G. Microembolic sigals ad risk of early recurrece i patiets with stroke or trasiet ischemic attack. Stroke. 1998; 29: Levi CR, O Malley HM, Fell G, Roberts AK, Hoare MC, Royle JP, Cha A, Beiles BC, Chambers BR, Bladi CF, Doa GA. Trascraial Doppler detected cerebral microembolism followig carotid edarterectomy: high microembolic sigal loads predict postoperative cerebral ischemia. Brai. 1997;120: Siebler M, Rose G, Sitzer M, Beder A, Steimetz H. Real-time idetificatio of cerebral microemboli with US feature detectio by a eural etwork. Radiology. 1994;192: Droste DW, Hagedor G, Notzold A, Siemes HJ, Sievers, HH, Kaps M. Bigated trascraial Doppler for the detectio of cliically silet circulatig emboli i ormal persos ad patiets with prosthetic cardiac valves. Stroke. 1997;28: Kemey V, Droste DW, Hermes S, Nabavi DG, Schulte-Altedoreburg G, Siebler M, Rigelstei EB. Automatic embolus detectio by a eural etwork. Stroke. 1999;30: Markus HS, Reid G. Frequecy filterig improves ultrasoic embolic sigal detectio. Ultrasoud Med Biol. 1999;25: Markus H, Culliae M, Reid G. Improved automated detectio of embolic sigals usig a ovel frequecy filterig approach. Stroke. 1999; 30: Riglestei EB, Droste DW, Babikia VL, Evas DH, Grosset DG, Kaps M, Markus HS, Russell D, Siebler M. Iteratioal Cosesus Group o Microembolus Detectio: cosesus o microembolus detectio by TCD. Stroke. 1998;29: Keue RWM, Stam CJ, Tavy DLJ, Mess WH, Titulaer BM, Ackerstaff RGA. Prelimiary report of detectig microemboli sigals i the trascraial Doppler time series with oliear forecastig. Stroke. 1998;29: Markus H, Blad JM, Rose G, Sitzer M, Siebler M. How good is iterceter agreemet i the idetificatio of embolic sigals i carotid artery disease? Stroke. 1996;27: Va Zuile EV, Mess WH, Jase C, Va der Tweel I, va Gij J, Ackerstaff RGA. Automatic embolus detectio compared with huma experts: a Doppler ultrasoud study. Stroke. 1996;27: Molloy J, Marti JF, Baskerville PA, Fraser SCA, Markus HS. S-Nitrosoglutathioe reduces the rate of embolizatio i humas. Circulatio. 1998;98: Grosset DG, Georgiadis D, Kelma AW, Lees KR. Quatificatio of ultrasoud emboli sigals i patiets with cardiac ad carotid disease. Stroke. 1993;24: Markus HS, Molloy J. The use of a decibel threshold i the detectio of embolic sigals. Stroke. 1997;28: Markus HS, Ackerstaff RG, Babikia VL, Bladi C, Droste D, Grosset D, Levi C, Russell D, Siebler M, Tegeler C. Iter-ceter agreemet i readig Doppler embolic sigals: a multiceter iteratioal study. Stroke. 1997;28: Aydi N, Padayachee S, Markus HS. The use of the wavelet trasform to describe embolic sigals. Ultrasoud Med Biol. 1999;25: Molloy J, Markus HS. Multigated Doppler ultrasoud i the detectio of emboli i a flow model ad embolic sigals i patiets. Stroke. 1996;27: