Clustering & Classification of ERP Patterns: Methods & Results for TKDD09 Paper Haishan Liu, Gwen Frishkoff, Robert Frank, & Dejing Dou

Clustering & Classification of ERP Patterns: Methods & Results for TKDD09 Paper Haishan Liu, Gwen Frishkoff, Robert Frank, & Dejing Dou Created: 01/21/2009 by HL Last edit: 01/27/2009 by GF Dataset: https://trac.nic.uoregon.edu/ntk/attachment/wiki/nemotechnicalreports/kdd07.rar 1. Summary There are two parts to this report: 1) clustering results for the 4 LP1, LP2 datasets, 3 target ERP patterns, and 14 pattern attributes; and 2) clustering and cluster-based classification results for WL3a data using 6 target ERP patterns (8 originally; see Notes in Section XX), and XX pattern attributes. [Summary of results -- maybe copy summary tables for best results & summarize in a few sentences. Note that a priori specfication of #clusters is helpful. Discuss consistency of results, or lack thereof, across 4 LP datasets. Talk about basis for selection of metrics to use for clustering & classification. Note results for WL3a classification based on expert vs. autolabeled data.] 2. Clustering LP1 and LP2 data For the LP1 and LP2 experiments, there are four datasets: LP1g1, LP1g2, LP2g1, LP2g2; and two clustering techniques: (a) manually specifying the number of clusters and (b) automatic determination of the number of clusters. These parameters resulted in 4x2=8 experiments. 2.1 Three (3) Pattern Rules used for Autolabeling for LP1 and LP2 data The input to the clustering consists of labeled data for 3 early patterns (patterns with peak latencies between ~0-250 ms after stimulus onset). For these case studies, the LP1 and LP2 data (i.e, individual observations for each subject, condition, and tpca factor) were automatically labeled by RMF using the rules that GF specified in ERP_Rules_09-02.doc. The 3 rules are listed below: Rule #1 (pattern PT 1 = P100-visual component of the ERP) Let ROI=occipital (average of left occipital, right occipital) For any n, FA n = PT 1 iff 80ms < TI-max (FA n ) < 150ms AND temporal criterion #1 IN-mean(ROI).4 mv AND min variance criterion IN-mean(ROI) > 0 spatial criterion #1 Rule #2a (pattern PT 2 = N100-visual component of the ERP) Let ROI=occipital (average of left occipital, right occipital) For any n, FA n = PT 2 iff 150ms < TI-max (FA n ) < 220ms AND temporal criterion #2a IN-mean(ROI).4 mv AND min variance criterion IN-mean(ROI) < 0 spatial criterion #2a Rule #2b (pattern PT 2 = laten1/n2-visual component of the ERP) Let ROI=occipital-posterior temporal (average of left occipital, left posterior temporal) For any n, FA n = PT 2 iff 220ms < TI-max (FA n ) < 300ms AND temporal criterion #2b IN-mean(ROI).4 mv AND min variance criterion Robert Frank Comment: Rules are consistent with ERP_Rules_09 02.doc Robert Frank Comment: Rule 1 criteria consistent with ERP_Rules_09 02.doc Robert Frank Comment: Rule 2a criteria consistent with ERP_Rules_09 02.doc Robert Frank Comment: Rule 2b criteria consistent with ERP_Rules_09 02.doc

IN-mean(ROI) < 0 spatial criterion #2b 2.2 Metrics used for Clustering of LP1 and LP2 data We used 13 metrics to summarize the temporal and spatial attributes of the 3 ERP patterns in datasets LP1 and LP2, as shown in Table 2. Note that, where ROI is a pre-defined scalp region, is not used in the clustering. Table 1. Metrics used for clustering of LP1 and LP2 Metric Label Brief Definition Temporal Spatial TI-max Peak latency (in ms) x IN-mean (LOCC) Mean intensity over LOCC scalp region x IN-mean (ROCC) Mean intensity over ROCC scalp region x IN-mean (LPAR) Mean intensity over LPAR scalp region x IN-mean (RPAR) Mean intensity over RPAR scalp region x IN-mean (LPTEM) Mean intensity over LPTEM scalp region x IN-mean (RPTEM) Mean intensity over RPTEM scalp region x IN-mean (LATEM) Mean intensity over LATEM scalp region x IN-mean (RATEM) Mean intensity over RATEM scalp region x IN-mean (LORB) Mean intensity over LORB scalp region x IN-mean (RORB) Mean intensity over RORB scalp region x IN-mean (LFRON) Mean intensity over LFRON scalp region x IN-mean (RFRON) Mean intensity over RFRON scalp region x 2.3 LP1g1 Dataset 2.3.1. Data structure #Observations = 126 (#Subj*#Cond) #Subjects = 21 #Conditions = 6 #PCA Factors Retained for Autolabeling (Patt/Fac Matching) = 15 Pattern rules as specified in Section 2.1. Table 2: Autolabeling Results Summary for LP1group1 Factor Factor Factor 8 Factor Factor Factor NObs 5 7 9 12 15 Rule 1 #Nonmatch 51 102 153 (P100) #Match 75 24 99 of 126 %Match 60% 20% 80% Rule 2a (N100) #Nonmatch 33 33 #Match 93 93 %Match 74% 74% Rule 2b #Nonmatch 16 30 50 96 Haishan Liu Comment: As Dejing pointed out, this fraction is wrong. Can t add two fractions together. Should be 99/(99+153)=39% GF: The reason for adding these percentages was to show that some observations must have been belonged to multiple factors, which means that we need to be cautious about which observations are input to the clustering. Robert Frank 1/26/09 2:45 PM Comment: You can add percentages if they are w.r.t the same base. 75 = 60 % of 126, and 24 = 20% of 126. So 99 = 80% of 126. The base is 126, the # of raw ERP observatios in LP1g1. If the % in the last column is < 100 %, then PCAautlabel is stating that for some raw ERP observations, the pattern of interest is not present in their tpca factors. If the last column % is > 100, then autolabel is stating that for some raw ERP observations, the pattern of interest is present in 2 or more factors. Perhaps we can show in the last column that the #Match is out of 126 (# of raw ERP observations). Also, I am not certain we should sum across the # Nonmatch columns, and suggest deleting that row.

(N1/N2) #Match 110 96 76 282 %Match 87% 76% 60% *223% Modal Factor (Rule) X (P100) X (N100) X (N1/N2) 474 (matches) [[DD: Is there any factors match more than one rule? It seems Factor 5 only match to P100, Factor 7 only match to N100 and Factor 9,12 and 15 only match to N1/N2. ]] [RF: I believe the phenomena of a single factor matching more than one rule occurred in WL3a. Also, the P100 (Rule 1) was captured by factors 5 and 12, while N1/N2 (Rule 2b) was captured by factors 8, 9 and 15.] 2.3.2. Case Study #1: Clustering LP1g1 data using expert specification of # target patterns HL manually set the number of clusters to 3, since there are 3 target patterns (see Sec. 2.1). Only those observations that belonged to the 3 modal factors (Factors 5, 7, and 8; see Table 2) were used in the clustering. Hence, the total N is 75+93+110=278. The following run information gives the settings for clustering of these data in WEKA using the EM algorithm. Scheme: weka.clusterers.em -I 100 -N 3 -M 1.0E-6 -S 100 Relation: LP1group1_Subj21_NN_NW_WC_WN_WR_WU_pattern_factors_modalweka.filters.unsupervised.attribute.Remove-R1-4-weka.filters.unsupervised.attribute.Remove-R24-26 Instances: 278 Attributes: 24 [GF PLEASE RERUN CLUSTERING WITH THE ATTRIBUTES SPECIFIED IN TABLE 1] [[DD We agreed to use 14 (13?) attributes to re-run the tests. I think Haishan can do it Monday]] ROI TI-max IN-mean (LOCC) IN-mean (ROCC) IN-mean (LPAR) IN-mean (RPAR) IN-mean (LPTEM) IN-mean (RPTEM) IN-mean (LATEM) IN-mean (RATEM) IN-mean (LORB) IN-mean (RORB) IN-mean (LFRON) IN-mean (RFRON) SP-cor

IN-max SP-max SP-max ROI IN-min SP-min SP-min ROI Ignored: Pattern Test mode: Classes to clusters evaluation on training data === Model and evaluation on training set === EM == Number of clusters: 3 Class attribute: Pattern Classes to Clusters: 0 1 2 <-- assigned to cluster 75 0 0 P1 0 36 57 P2a 2 52 56 P2b Cluster 0 <-- P1 Cluster 1 <-- P2b Cluster 2 <-- P2a Incorrectly clustered instances : 94.0 33.8129 % [[DD: I think the result is similar good (P1 and P2a) or bad (P2b) as kdd 07 replications]] 2.3.3. Case Study #2: Clustering LP1g1 data without expert specification of # target patterns For this next analysis, HL let WEKA discover the number of patterns/clusters automatically. Only those observations that belonged to the 3 modal factors (Factors 5, 7, and 8; see Table 2) were used in the clustering. Hence, the total N is 75+93+110=278. The following run information gives the settings for clustering of these data in WEKA using the EM algorithm. === Run information === Scheme: weka.clusterers.em -I 100 -N -1 -M 1.0E-6 -S 100 Relation: LP1group1_Subj21_NN_NW_WC_WN_WR_WU_pattern_factors_modalweka.filters.unsupervised.attribute.Remove-R1-4,28-30 Instances: 278

Attributes: 24 [GF PLEASE RERUN CLUSTERING WITH THE ATTRIBUTES SPECIFIED IN TABLE 1] [[DD Again, we agreed to do it]] ROI TI-max IN-mean (LOCC) IN-mean (ROCC) IN-mean (LPAR) IN-mean (RPAR) IN-mean (LPTEM) IN-mean (RPTEM) IN-mean (LATEM) IN-mean (RATEM) IN-mean (LORB) IN-mean (RORB) IN-mean (LFRON) IN-mean (RFRON) SP-cor IN-max SP-max SP-max ROI IN-min SP-min SP-min ROI Ignored: Pattern Test mode: Classes to clusters evaluation on training data === Model and evaluation on training set === EM == Number of clusters selected by cross validation: 5 Class attribute: Pattern Classes to Clusters: 0 1 2 3 4 <-- assigned to cluster 0 75 0 0 0 P1 30 0 1 52 10 P2a 25 0 28 26 31 P2b Cluster 0 <-- No class Cluster 1 <-- P1 Cluster 2 <-- No class Cluster 3 <-- P2a Cluster 4 <-- P2b

Incorrectly clustered instances : 120.0 43.1655 % [[DD: it is hard to say whether clustering without a number of clusters is better or worse than the 3-cluster clustering. One interesting question is that how close autolabeling is to gold standard? We may discuss it on Thursday]] 2.4. LP1g2 Dataset 2.4.1. Data structure #Observations = 120 (#Subj*#Cond) #Subjects = 20 #Conditions = 6 #PCA Factors Retained for Autolabeling (Patt/Fac Matching) = 15 Pattern rules as specified in Section 2.1. Table 3: Autolabeling Results Summary for LP1group2 Factor 5 Factor 3 Factor Factor 10 Factor NObs 8 14 Rule 1 #Nonmatch 42 42 (P100) #Match 78 78 of 120 Rule 2a (N100) Rule 2b (N1/N2) %Match 65% 65% #Nonmatch 5 59 64 #Match 115 61 176 %Match 96% 51% *147% #Nonmatch 30 79 109 #Match 90 41 131 %Match 75% 34% *109% Modal Factor (Rule) X (Fac5/P1) X (Fac3/N1) X (Fac10/N2) 385 (matches) 2.4.2. Case Study #3: Clustering LP1g2 data using expert specification of # target patterns HL manually set the number of clusters to 3, since there are 3 target patterns (see Sec. 2.1). Only those observations that belonged to the 3 modal factors (Factors 3, 5, and 10; see Table 3) were used in the clustering. Hence, the total N is 78+115+90=283. The following run information gives the settings for clustering of these data in WEKA using the EM algorithm. Scheme: weka.clusterers.em -I 100 -N 3 -M 1.0E-6 -S 100 Relation: LP1group2_Subj21_NN_NW_WC_WN_WR_WU_pattern_factors_modalweka.filters.unsupervised.attribute.Remove-R1-4,28-30 Instances: 283

Attributes: 24 [GF PLEASE RERUN CLUSTERING WITH THE ATTRIBUTES SPECIFIED IN TABLE 1] [[DD Agreed.]] ROI TI-max IN-mean (LOCC) IN-mean (ROCC) IN-mean (LPAR) IN-mean (RPAR) IN-mean (LPTEM) IN-mean (RPTEM) IN-mean (LATEM) IN-mean (RATEM) IN-mean (LORB) IN-mean (RORB) IN-mean (LFRON) IN-mean (RFRON) SP-cor IN-max SP-max SP-max ROI IN-min SP-min SP-min ROI Ignored: Pattern Test mode: Classes to clusters evaluation on training data === Model and evaluation on training set === EM == Number of clusters: 3 Class attribute: Pattern Classes to Clusters: 0 1 2 <-- assigned to cluster 5 110 0 P2a 2 0 76 P1 86 0 4 P2b Cluster 0 <-- P2b Cluster 1 <-- P2a Cluster 2 <-- P1 Incorrectly clustered instances : 11.0 3.8869 %

[[DD: It is similar as kdd 07 replication that LP1 group 2 data are much more distinguishable then group 1 data. I understand we set 4 clusters in kdd replication and used different number of input]] 2.4.3. Case Study #4: Clustering LP1g2 data without expert specification of # target patterns For this next analysis, HL let WEKA discover the number of patterns/clusters automatically. Only those observations that belonged to the 3 modal factors (Factors 3, 5, and 7; see Table 3) were used in the clustering. Hence, the total N is 78+115+90=283. The following run information gives the settings for clustering of these data in WEKA using the EM algorithm. === Run information === Scheme: weka.clusterers.em -I 100 -N -1 -M 1.0E-6 -S 100 Relation: LP1group2_Subj21_NN_NW_WC_WN_WR_WU_pattern_factors_modalweka.filters.unsupervised.attribute.Remove-R1-4,28-30 Instances: 283 Attributes: 24 [GF PLEASE RERUN CLUSTERING WITH THE ATTRIBUTES SPECIFIED IN TABLE 1] [[DD Agreed]] ROI TI-max IN-mean (LOCC) IN-mean (ROCC) IN-mean (LPAR) IN-mean (RPAR) IN-mean (LPTEM) IN-mean (RPTEM) IN-mean (LATEM) IN-mean (RATEM) IN-mean (LORB) IN-mean (RORB) IN-mean (LFRON) IN-mean (RFRON) SP-cor IN-max SP-max SP-max ROI IN-min SP-min SP-min ROI Ignored: Pattern Test mode: Classes to clusters evaluation on training data

=== Model and evaluation on training set === EM == Number of clusters selected by cross validation: 13 Class attribute: Pattern Classes to Clusters: 0 1 2 3 4 5 6 7 8 9 10 11 12 <-- assigned to cluster 8 9 0 49 0 10 8 4 0 5 17 5 0 P2a 0 0 4 0 72 0 0 0 0 0 0 0 2 P1 1 1 0 0 0 19 1 9 1 4 0 0 54 P2b Cluster 0 <-- No class Cluster 1 <-- No class Cluster 2 <-- No class Cluster 3 <-- P2a Cluster 4 <-- P1 Cluster 5 <-- No class Cluster 6 <-- No class Cluster 7 <-- No class Cluster 8 <-- No class Cluster 9 <-- No class Cluster 10 <-- No class Cluster 11 <-- No class Cluster 12 <-- P2b Incorrectly clustered instances : 108.0 38.1625 % [[DD: I think the result is worse than pre-set up number of clusters. Here it actually shows the domain knowledge are helpful for data mining]] 2.5. LP2g1 Dataset 2.5.1. Data structure: #Observations = 144 (#Subj*#Cond) #Subjects = 24 #Conditions = 6 #PCA Factors Retained for Autolabeling (Patt/Fac Matching) = 15 Pattern rules as specified in Section 2.1. HL came up with the autolabeling results summary for LP2 data according to the LP1 examples [GF?? -- We should review this to be sure the results summary is correct]. [[DD: I have some doubt too. Haisan, can you explain it a little more. Why LP1 examples can be used for LP2 data?]] [RF: I have attached a copy of the GrandAverageStats_LP2-Gp1 spreadsheet, which I believe references this data, and the

results summary is a bit different.] HL chose modal factors for the target ERP patterns based the percentage of observations that matched a given rule for each of the latent factors. All the experiments were conducted using only observations that were captured by the modal factors. The cluster-to-class assignment tables in the results are highlighted. Table 4: Autolabeling Results Summary for LP2group1 Factor 3 Factor 5 Factor 7 Factor 8 NObs Rule 1 #Nonmatch 42 42 (P100) #Match 84 84 Rule 2a (N100) Rule 2b (N1/N2) %Match 67% 67% #Nonmatch 42 87 129 #Match 84 39 123 %Match 67% 31% 49% #Nonmatch 27 27 #Match 99 99 %Match 79% 79% Modal Factor (Rule) X (P100) X (N100) X (N1/N2) 306 (matches) 2.5.2. Case Study #5: Clustering LP2g1 data using expert specification of # target patterns HL manually set the number of clusters to 3, since there are 3 target patterns (see Sec. 2.1). Only those observations that belonged to the 3 modal factors (Factors 3, 5, and 8; see Table 4) were used in the clustering. Hence, the total N is 99+84+84=278. The following run information gives the settings for clustering of these data in WEKA using the EM algorithm. Scheme: weka.clusterers.em -I 100 -N 3 -M 1.0E-6 -S 100 Relation: LP2group1_Subj21_NN_NW_WC_WN_WR_WU_pattern_modalweka.filters.unsupervised.attribute.Remove-R1-4,28-30 Instances: 267 Attributes: 24 [GF PLEASE RERUN CLUSTERING WITH THE ATTRIBUTES SPECIFIED IN TABLE 1] [[DD Agreed]] ROI TI-max IN-mean (LOCC) IN-mean (ROCC) IN-mean (LPAR) IN-mean (RPAR) IN-mean (LPTEM) IN-mean (RPTEM) IN-mean (LATEM) IN-mean (RATEM) IN-mean (LORB)

IN-mean (RORB) IN-mean (LFRON) IN-mean (RFRON) SP-cor IN-max SP-max SP-max ROI IN-min SP-min SP-min ROI Ignored: Pattern Test mode: Classes to clusters evaluation on training data === Model and evaluation on training set === EM == Number of clusters: 3 Class attribute: Pattern Classes to Clusters: 0 1 2 <-- assigned to cluster 84 0 0 P1 2 34 48 P2a 10 34 55 P3 Cluster 0 <-- P1 Cluster 1 <-- P2a Cluster 2 <-- P3 Incorrectly clustered instances : 94.0 35.206 % [[DD: I would say that the result is similar to LP1g1, not bad but not very good]] 2.5.2. Case Study #6: Clustering LP2g1 data without expert specification of # target patterns For this next analysis, HL let WEKA discover the number of patterns/clusters automatically. Only those observations that belonged to the 3 modal factors (Factors 3, 5, and 8; see Table 3) were used in the clustering. Hence, the total N is 99+84+84=278. The following run information gives the settings for clustering of these data in WEKA using the EM algorithm. Scheme: weka.clusterers.em -I 100 -N -1 -M 1.0E-6 -S 100

Relation: LP2group1_Subj21_NN_NW_WC_WN_WR_WU_pattern_modalweka.filters.unsupervised.attribute.Remove-R1-4,28-30 Instances: 267 Attributes: 24 [GF PLEASE RERUN CLUSTERING WITH THE ATTRIBUTES SPECIFIED IN TABLE 1] [[DD Agreed]] ROI TI-max IN-mean (LOCC) IN-mean (ROCC) IN-mean (LPAR) IN-mean (RPAR) IN-mean (LPTEM) IN-mean (RPTEM) IN-mean (LATEM) IN-mean (RATEM) IN-mean (LORB) IN-mean (RORB) IN-mean (LFRON) IN-mean (RFRON) SP-cor IN-max SP-max SP-max ROI IN-min SP-min SP-min ROI Ignored: Pattern Test mode: Classes to clusters evaluation on training data === Model and evaluation on training set === EM == Number of clusters selected by cross validation: 3 Class attribute: Pattern Classes to Clusters: 0 1 2 <-- assigned to cluster 84 0 0 P1 2 34 48 P2a 10 34 55 P3 Cluster 0 <-- P1

Cluster 1 <-- P2a Cluster 2 <-- P3 Incorrectly clustered instances : 94.0 35.206 % [[DD: It is interesting Weka choose the same number of clusters as autolabeling results]] 2.6. LP2g2 Dataset 2.6.1. Data structure: #Observations = 144 (#Subj*#Cond) #Subjects = 24 #Conditions = 6 #PCA Factors Retained for Autolabeling (Patt/Fac Matching) = 15 Pattern rules as specified in Section 2.1. Table 5: Autolabeling Results Summary for LP2group2 Factor Factor Factor Factor Factor Factor Factor NObs 2 3 6 7 8 10 14 Rule 1 #Nonmatch 125 33 69 42 (P100) #Match 1 93 57 151 %Match 0% 40% Rule 2a #Nonmatch 50 112 162 (N100) #Match 76 14 90 Rule 2b (N1/N2) %Match 36% #Nonmatch 50 38 27 #Match 76 88 164 %Match 65% Modal Factor (Rule) X (P100) X (N100) X (N1/N2) 405 (matches) 2.3.2. Case Study #7: Clustering LP2g2 data using expert specification of # target patterns HL manually set the number of clusters to 3, since there are 3 target patterns (see Sec. 2.1). Only those observations that belonged to the 3 modal factors (Factors 5, 6, and 7; see Table 5) were used in the clustering. Hence, the total N is 93+76+76=267. [GF I would use Factor 7 as the modal factor for the N2 pattern, not Factor 14. Factor 14 is noiser]. [[DD: Either Factor 7 or Factor 14 is ok for me because I do not have enough domain knowledge to choose. Gwen, could you please explain why Factor 14 is noiser although it has high percentage matching to Rule 2b? On the other hand, if we know it is a noiser, why list in table 5?]] [RF: I need to double-check, but I believe the factors are in order of decreasing variance accounted for, so the higher # factors tend to be noisier. However, regardless of a factor s SNR, we applied PCAautolabel to the first 15 factors: If any one of them was flagged as capturing a pattern of interest in one or more raw ERP observations, it would appear in the table.]

The following run information gives the settings for clustering of these data in WEKA using the EM algorithm. Scheme: weka.clusterers.em -I 100 -N -1 -M 1.0E-6 -S 100 Relation: LP2group1_Subj21_NN_NW_WC_WN_WR_WU_pattern_modalweka.filters.unsupervised.attribute.Remove-R1-4,28-30 Instances: 267 Attributes: 24 [GF PLEASE RERUN CLUSTERING WITH THE ATTRIBUTES SPECIFIED IN TABLE 1] [DD Agreed] ROI TI-max IN-mean (LOCC) IN-mean (ROCC) IN-mean (LPAR) IN-mean (RPAR) IN-mean (LPTEM) IN-mean (RPTEM) IN-mean (LATEM) IN-mean (RATEM) IN-mean (LORB) IN-mean (RORB) IN-mean (LFRON) IN-mean (RFRON) SP-cor IN-max SP-max SP-max ROI IN-min SP-min SP-min ROI Ignored: Pattern Test mode: Classes to clusters evaluation on training data === Model and evaluation on training set === EM == Number of clusters selected by cross validation: 3 Class attribute: Pattern Classes to Clusters: 0 1 2 <-- assigned to cluster 84 0 0 P1 2 34 48 P2a

10 34 55 P3 Cluster 0 <-- P1 Cluster 1 <-- P2a Cluster 2 <-- P3 Incorrectly clustered instances : 94.0 35.206 % 2.6.3. Case Study #8: Clustering LP2g2 data without expert specification of # target patterns For this next analysis, HL let WEKA discover the number of patterns/clusters automatically. Only those observations that belonged to the 3 modal factors (Factors 5, 6, and 14; see Table 5) were used in the clustering. Hence, the total N is 93+76+88=267. [GF I would use Factor 7 as the modal factor for the N2 pattern, not Factor 14. Factor 14 is noiser]. [[DD Gwen may help us understand this a little more]] The following run information gives the settings for clustering of these data in WEKA using the EM algorithm. Scheme: weka.clusterers.em -I 100 -N -1 -M 1.0E-6 -S 100 Relation: LP2group2_Subj21_NN_NW_WC_WN_WR_WU_pattern_moadalweka.filters.unsupervised.attribute.Remove-R1-4,28-30 Instances: 257 Attributes: 24 [GF PLEASE RERUN CLUSTERING WITH THE ATTRIBUTES SPECIFIED IN TABLE 1] [[DD Agreed]] ROI TI-max IN-mean (LOCC) IN-mean (ROCC) IN-mean (LPAR) IN-mean (RPAR) IN-mean (LPTEM) IN-mean (RPTEM) IN-mean (LATEM) IN-mean (RATEM) IN-mean (LORB) IN-mean (RORB) IN-mean (LFRON) IN-mean (RFRON) SP-cor IN-max SP-max SP-max ROI

IN-min SP-min SP-min ROI Ignored: Pattern Test mode: Classes to clusters evaluation on training data === Model and evaluation on training set === EM == Number of clusters selected by cross validation: 5 Class attribute: Pattern Classes to Clusters: 0 1 2 3 4 <-- assigned to cluster 0 57 0 36 0 P1 33 0 19 0 24 P2a 3 0 42 0 43 P2b Cluster 0 <-- P2a Cluster 1 <-- P1 Cluster 2 <-- No class Cluster 3 <-- No class Cluster 4 <-- P2b Incorrectly clustered instances : 124.0 48.249 %

3. Clustering & Classification of WL3a data 3.1 Eight (8) Pattern Rules used for Autolabeling for WL3a data The input to the clustering consists of labeled data for 8 ERP patterns (patterns with peak latencies between ~0-900 ms after stimulus onset). For these case studies, the WL3a data (i.e, individual observations for each subject, condition, and tpca factor) were automatically labeled using the rules as 3.2 Metrics used for Clustering of WL3a data We used 14 metrics to summarize the temporal and spatial attributes of the 8 ERP patterns in dataset WL3a, as shown in Table 2. Note that, where ROI is a pre-defined scalp region, is not used in the clustering. Table 6. Metrics used for clustering of WL3 data Metric Label Brief Definition Temporal Spatial Functional TI-max Peak latency (in ms) x TI-duration Duration (in ms) x IN-mean (LOCC) Mean intensity over LOCC scalp region X IN-mean (ROCC) Mean intensity over ROCC scalp region X IN-mean (LPAR) Mean intensity over LPAR scalp region X IN-mean (RPAR) Mean intensity over RPAR scalp region X IN-mean (LPTEM) Mean intensity over LPTEM scalp region X IN-mean (RPTEM) Mean intensity over RPTEM scalp region X IN-mean (LATEM) Mean intensity over LATEM scalp region X IN-mean (RATEM) Mean intensity over RATEM scalp region X IN-mean (LORB) Mean intensity over LORB scalp region X IN-mean (RORB) Mean intensity over RORB scalp region X IN-mean (LFRON) Mean intensity over LFRON scalp region X IN-mean (RFRON) Mean intensity over RFRON scalp region X Pseudo Known Condition (Diffwave) x RareMisses-RareHits Condition (Diffwave) x RareHits-Known Condition (Diffwave) x Pseudo RareMisses Condition (Diffwave) x 3.3 Clustering WL3a data using expert specification of # target patterns Input: WL3a_PCAautolabel_2007Feb07v4.xls Pattern factors are extracted according to the auto-labeling results (column N) Preprocessing: Combined all sheets except Attribute4Mining in the input file into a single sheet. Add a new column at the end of the new sheet called pattern. Filter out non-pattern factors according to the value in the Pattern Present column. [[GF Please clarify whether observations were filtered using Pattern Present (expert labeling ) or Fac=Patt (autolabeling). [[DD Haishan, can you explain this?]]

Data structure: #Observations = 144 (#Subj*#Cond) #Subjects = 36 #Conditions = 4 #PCA Factors Retained for Autolabeling (Patt/Fac Matching) = 15 Pattern Rules used for Autolabeling: See Appendix B of Frishkoff, Frank, et al., 2007 (Computational Intelligence & Neuroscience) Table 7. Autolabeling Results Summary: (Grand Average Mean %Match GrandAverageStats.xls_WL-3a.xls -> Column H) Rule 1 (P100) Rule 2a (N100) Rule 2b (N1/N2) Rule 3 (N3) Rule 4 (P1r) Rule 5 (MFN) Rule 6 (N4) Rule 7 (P300) Fac4 Fac3 Fac10 Fac7 Fac2 Fac8 Fac9 Fac11 Fac13 Fac15 NObs #Nonmatch 25 94 119 #Match 119 50 169 %Match 83% 35% 118% #Nonmatch 25 25 #Match 119 119 %Match 83% 83% #Nonmatch 70 44 114 #Match *74 100 174 %Match 51% 69% 120% #Nonmatch 83 75 59 217 #Match *61 69 85 215 %Match 42% 48% 59% 149% #Nonmatch 51 107 54 212 #Match *93 37 90 220 %Match 65% 26% 63% 151% #Nonmatch 111 91 95 85 382 #Match 33 53 49 59 194 %Match 23% 37% 34% 41% 135% #Nonmatch 130 93 223 #Match 14 *51 65 %Match 10% 35% 45% #Nonmatch 58 62 130 250 #Match 86 82 14 182 %Match 60% 57% 10% 127% Robert Frank Comment: Statistics are missing from GranAvgStat spreadsheet. Need to recomputed. 2.6.3. Case Study #9: Clustering WL3a data without expert specification of # target patterns HL set the number of clusters to six (6), since there are 6 latent pattern-related factors (see Table 7). Only those observations that belonged to the 6 modal factors (Factors 2, 3, 4, 7, 9, and 10; see Table 7) were used in the clustering. Hence, the total N is 119(P1/Fac4)+119(N1/Fac3)+74(N2/fac10)+61(N3/Fac7)+53(MFN/Fac2)+82(P3/Fac9)=508.

[[DD It seems some factors match more than one rule. I am little confused. Why the number of clusters should be 6? Since we have 8 pattern rules, the ideal case is that we can have 8 clusters. Otherwise, we have no way to generate classification rules based on clustering result.]] [RF: With respect to 1 factor matching more than 1 rule, take for instance Factor 2. Rules 5, 6 and 7 have overlapping temporal windows, so the factor s Ti-max, which is subject and condition invariant, can meet all three rule criteria. Although the MFN (Rule 5), N4 (Rule 6) and P300 (Rule 7) have different spatial criteria, the spatial topography of a given factor in tpca, such as Factor 2, is subject and observation specific: Factor 2 can satisfy a given rule s spatial criteria in one ERP observation (subject and condition) and still satisfy another rule s very different spatial criteria in some other ERP observation. It also depends on whether or not the spatial criteria of the rules of the patterns in question are mutually exclusive: Rules 6 and 7 have mutually exclusive spatial criteria, but Rules 5 and 6 as one pair, and Rules 5 and 7 as another, do not. Moreover, the extent to which a factor s multiple rule matches correspond to identical or distinct ERP observations can affect whether or not the factor is actually describing more than one pattern. I think that collapsing the number of clusters to 6, based on the PCAautolabel results, is a judgement call; There may be more than 6 patterns described by the 6 factors. Gwen, do my remarks seem reasonable?] [GF I m in total agreement with Bob s comments. It is a judgment call. It may also be informative to note that for Factor 2, I ONLY selected observations meeting criteria for one rule (the MFN). Imagine that we didn t know that the N400 had been observed in other experiments as a distinct pattern. Then we would not know that some observations that meet the MFN criteria actualy contain information that CAN be captured with more than one pattern rule. It s possible, in principle, that any of the latent temporal PCA factors could confound more than one pattern. Similarly for Factor 7 (I chose observations matching the N3 somewhat arbitrarily, and because the N3 is of greater interest to me than the P1r at the moment ). So, if my reasoning is correct, I think it is possible to explain and justify the decision to summarize the patterns in the WL3a data using only 6 expert-defined pattern rules (because we only selected observations matching these 6 patterns). **IF CLUSTERING WITHOUT PRESPECIFICATION OF NUMBER OF CLUSTERS SUGGESTS THERE ARE MORE PATTERNS AND IF WE THE RESULT IS BELIEVEABLE THAN I BELIEVE THIS WOULD SHOW HOW DATA MINING CAN ADD TO OUR CERTAINTY THAT MORE PATTERNS EXIST (WHICH WE ALREADY BELIEVE, BUT ARE HARD-PRESSED TO SHOW USING TPCA WITH THESE DATA).** ] The following run information gives the settings for clustering of these data in WEKA using the EM algorithm Scheme: weka.clusterers.em -I 100 -N 8 -M 1.0E-6 -S 100 Relation: WL3a_PCAautolabel_2007Feb16_merged_with_pattern_auto_label- weka.filters.unsupervised.attribute.remove-r1-4-weka.filters.unsupervised.attribute.remove-r1-4- weka.filters.unsupervised.attribute.remove-r2-8-weka.filters.unsupervised.attribute.remove-r28-30,35-50 Instances: 615 [GF PLEASE RERUN CLUSTERING WITH 508 OBSERVATIONS AS SPECIFIED ABOVE] [[DD again, if we do not consider Factor 11, 13, 15 whatever the matching percentage is, why we list them in the table]]

Attributes: 32 [GF PLEASE RERUN CLUSTERING WITH THE ATTRIBUTES SPECIFIED IN TABLE 6] [[DD agreed and we believe it is the best domain knowledge so far]] NGOODS IN-LOCC IN-ROCC IN-LPAR IN-RPAR IN-LPTEM IN-RPTEM IN-LATEM IN-RATEM IN-LORB IN-RORB IN-LFRON IN-RFRON SP-cor TI-max TI-begin TI-end TI-duration IN-max to Baseline IN-min to Baseline IN-max SP-max SP-max ROI IN-min SP-min SP-min ROI Pseudo-Known RareMisses-RareHits RareHits-Known Pseudo-RareMisses Ignored: Pattern Test mode: Classes to clusters evaluation on training data === Model and evaluation on training set === EM == Classes to Clusters: 0 1 2 3 4 5 6 7 <-- assigned to cluster 0 80 0 0 0 0 39 0 P100 0 0 9 53 57 0 0 0 N100 0 0 0 74 0 0 0 0 N2 0 0 44 0 10 0 0 7 N3

0 0 75 0 11 0 0 7 P1r 53 0 0 0 0 0 0 0 MFN 14 0 0 0 0 0 0 0 N4 0 0 0 0 0 82 0 0 P3 Cluster 0 <-- MFN Cluster 1 <-- P100 Cluster 2 <-- P1r Cluster 3 <-- N2 Cluster 4 <-- N100 Cluster 5 <-- P3 Cluster 6 <-- No class Cluster 7 <-- N3 Incorrectly clustered instances : 187.0 30.4065 % 3.4 Cluster-based classification of WL3a data using expert specification of # target patterns For the WL3a experiments, HL used 6-cluster EM clustering algorithm and based on the result of which HL conduct the classification process [GF?? -- I thought we were going to use 6 clusters, since there are only 6 modal factors? Can HL explain his procedure for deriving 8 clusters instead of 6?]. The rules derived are highlighted. [[DD I guess Haishan was looking that there are 8 pattern rules because we finally hope compare data mining rules with expert rules. The number of classes (cluters) would better be eight. Even we use 6 modal factors, we can still set up the number of clusters as 8 because the autolabeling already show that one factor can match more than one patterns. ]] [GF BUT note that I am suggesting only to select observations that match one of the 6 pattern rules see explanation above.] Input: WL3a_PCAautolabel_2007Feb07v4.xls Pattern factors are extracted according to the auto-labeling results (column N) Preprocessing: Generating class label: Apply the AddCluster filter in the Preprocess tab in Weka Explorer. In the parameter panel of the filter, choose EM as the clusterer and set the # of clusters to 8 in EM parameter. This procedure attaches a new column at the end of the file with the assigned cluster values to each factor. Result: === Run information === Scheme: weka.classifiers.trees.j48 -C 0.25 -M 2 Relation: WL3a_PCAautolabel_2007Feb16_merged_with_pattern_auto_label- weka.filters.unsupervised.attribute.remove-r1-4-weka.filters.unsupervised.attribute.remove-r1-4- weka.filters.unsupervised.attribute.remove-r2-8-weka.filters.unsupervised.attribute.remove-r28-30,35-50-weka.filters.unsupervised.attribute.remove-r32-

weka.filters.unsupervised.attribute.addcluster-wweka.clusterers.em -I 100 -N 8 -M 1.0E-6 -S 100 Instances: 615 [GF PLEASE RERUN CLUSTERING WITH 508 OBSERVATIONS AS SPECIFIED ABOVE] Attributes: 32 [GF PLEASE RERUN CLUSTERING WITH THE ATTRIBUTES SPECIFIED IN TABLE 6] [[DD - Agreed]] NGOODS IN-LOCC IN-ROCC IN-LPAR IN-RPAR IN-LPTEM IN-RPTEM IN-LATEM IN-RATEM IN-LORB IN-RORB IN-LFRON IN-RFRON SP-cor TI-max TI-begin TI-end TI-duration IN-max to Baseline IN-min to Baseline IN-max SP-max SP-max ROI IN-min SP-min SP-min ROI Pseudo-Known RareMisses-RareHits RareHits-Known Pseudo-RareMisses cluster Test mode: 10-fold cross-validation === Classifier model (full training set) === J48 pruned tree ------------------ TI-max <= 276 TI-max <= 102 IN-RORB <= -2.106243: cluster2 (71.0/1.0)

IN-RORB > -2.106243 Pseudo-Known <= 0.857294: cluster7 (43.0/3.0) Pseudo-Known > 0.857294: cluster2 (5.0/1.0) TI-max > 102 TI-max <= 230 IN-min <= -6.520502 <= -1.457642 SP-min <= 20: cluster4 (3.0/1.0) SP-min > 20: cluster5 (61.0/5.0) > -1.457642: cluster3 (9.0/1.0) IN-min > -6.520502 IN-LORB <= -0.054975 SP-cor <= 0.226834: cluster3 (3.0) SP-cor > 0.226834: cluster5 (2.0/1.0) IN-LORB > -0.054975 IN-max <= 7.110824: cluster4 (104.0) IN-max > 7.110824 SP-min <= 92: cluster5 (7.0) SP-min > 92: cluster4 (4.0/1.0) TI-max > 230 IN-LOCC <= 3.138674 IN-LPTEM <= -3.02711 IN-LFRON <= 1.149881 SP-cor <= 0.4684: cluster3 (3.0) SP-cor > 0.4684: cluster5 (2.0) IN-LFRON > 1.149881: cluster5 (15.0) IN-LPTEM > -3.02711 Pseudo-Known <= -2.264061: cluster5 (2.0) Pseudo-Known > -2.264061: cluster3 (116.0/1.0) IN-LOCC > 3.138674 SP-min <= 76: cluster8 (14.0) SP-min > 76: cluster3 (2.0) TI-max > 276 TI-max <= 408: cluster1 (67.0) TI-max > 408: cluster6 (82.0) Number of Leaves : 20 Size of the tree : 39 Time taken to build model: 0.16 seconds === Stratified cross-validation === === Summary === Correctly Classified Instances 569 92.5203 % Incorrectly Classified Instances 46 7.4797 % Kappa statistic 0.912 Mean absolute error 0.0225 Root mean squared error 0.1305

Relative absolute error 10.6148 % Root relative squared error 40.0698 % Total Number of Instances 615 === Confusion Matrix === a b c d e f g h <-- classified as 67 0 0 0 0 0 0 0 a = cluster1 0 72 0 0 0 0 5 0 b = cluster2 0 0 120 4 6 0 0 3 c = cluster3 0 0 3 104 8 0 0 0 d = cluster4 0 0 7 6 71 0 0 0 e = cluster5 0 0 0 0 0 82 0 0 f = cluster6 0 3 0 1 0 0 39 0 g = cluster7 0 0 0 0 0 0 0 14 h = cluster8 Quick Reference: Cluster toclass assignment: Cluster 0 < MFN Cluster 1 < P100 Cluster 2 < P1r Cluster 3 < N2 Cluster 4 < N100 Cluster 5 < P3 Cluster 6 < No class Cluster 7 < N3 Haishan Liu Comment: Cluster index is generated by weka preprocessor starting from 1. The cluster index in the cluster to class assignment starts from 0, which is generated by weka clustering module. I think there is a one to one correspondence between these indices, i.e., 0< >1, 1< >2 This can be further verified by comparing the data mining rules with the expert rules. === Rules Derived From the Tree === 1. TI-max <= 276 & TI-max <= 102 & IN-RORB <= -2.106243 ===> cluster2 (71.0/1.0) 2. TI-max <= 276 & TI-max <= 102 & IN-RORB > -2.106243 & Pseudo-Known <= 0.857294 ===> cluster7 (43.0/3.0) 3. TI-max <= 276 & TI-max <= 102 & IN-RORB > -2.106243 & Pseudo-Known > 0.857294 ===> cluster2 (5.0/1.0) 4. TI-max <= 276 & TI-max > 102 & TI-max <= 230 & IN-min <= -6.520502 & <= - 1.457642 & SP-min <= 20 ===> cluster4 (3.0/1.0) 5. TI-max <= 276 & TI-max > 102 & TI-max <= 230 & IN-min <= -6.520502 & <= - 1.457642 & SP-min > 20 ===> cluster5 (61.0/5.0) 6. TI-max <= 276 & TI-max > 102 & TI-max <= 230 & IN-min <= -6.520502 & > - 1.457642 ===> cluster3 (9.0/1.0) 7. TI-max <= 276 & TI-max > 102 & TI-max <= 230 & IN-min > -6.520502 & IN-LORB <= -0.054975 & SP-cor <= 0.226834 ===> cluster3 (3.0) 8. TI-max <= 276 & TI-max > 102 & TI-max <= 230 & IN-min > -6.520502 & IN-LORB <= -0.054975 & SP-cor > 0.226834 ===> cluster5 (2.0/1.0) 9. TI-max <= 276 & TI-max > 102 & TI-max <= 230 & IN-min > -6.520502 & IN-LORB > -0.054975 & IN-max <= 7.110824 ===> cluster4 (104.0) 10. TI-max <= 276 & TI-max > 102 & TI-max <= 230 & IN-min > -6.520502 & IN-LORB > -0.054975 & IN-max > 7.110824 & SP-min <= 92 ===> cluster5 (7.0) 11. TI-max <= 276 & TI-max > 102 & TI-max <= 230 & IN-min > -6.520502 & IN-LORB > -0.054975 & IN-max > 7.110824 & SP-min > 92 ===> cluster4 (4.0/1.0) 12. TI-max <= 276 & TI-max > 102 & TI-max > 230 & IN-LOCC <= 3.138674 & IN-LPTEM <= -3.02711 & IN-LFRON <= 1.149881 & SP-cor <= 0.4684 ===> cluster3 (3.0) 13. TI-max <= 276 & TI-max > 102 & TI-max > 230 & IN-LOCC <= 3.138674 & IN-LPTEM <= -3.02711 & IN-LFRON <= 1.149881 & SP-cor > 0.4684 ===> cluster5 (2.0) 14. TI-max <= 276 & TI-max > 102 & TI-max > 230 & IN-LOCC <= 3.138674 & IN-LPTEM <= -3.02711 & IN-LFRON > 1.149881 ===> cluster5 (15.0) 15. TI-max <= 276 & TI-max > 102 & TI-max > 230 & IN-LOCC <= 3.138674 & IN-LPTEM > -3.02711 & Pseudo-Known <= -2.264061 ===> cluster5 (2.0) 16. TI-max <= 276 & TI-max > 102 & TI-max > 230 & IN-LOCC <= 3.138674 & IN-LPTEM > -3.02711 & Pseudo-Known > -2.264061 ===> cluster3 (116.0/1.0) 17. TI-max <= 276 & TI-max > 102 & TI-max > 230 & IN-LOCC > 3.138674 & SP-min <= 76 ===> cluster8 (14.0)

18. TI-max <= 276 & TI-max > 102 & TI-max > 230 & IN-LOCC > 3.138674 & SP-min > 76 ===> cluster3 (2.0) 19. TI-max > 276 & TI-max <= 408 ===> cluster1 (67.0) 20. TI-max > 276 & TI-max > 408 ===> cluster6 (82.0) [GF (1/24/2009): Dejing, please rewrite rules so they can be aligned with expert rules as we discussed 8 days ago.] [[DD (1/26/2009): It seems we need ask Haishan to re-run the tests based on new metrics in table 6 and also Gwen suggested that the number clusters/patterns will be 6. If that is the case, how can we compare 6 cluster/classification rules with 8 expert rules. I can do the comparison for the kdd 07 replication report because we will not change the metrics and number of clusters there.]] [GF (1/27/2009): We would compare data mining results with 6 pattern rules that match the 6 patterns that tpca shows clearly can be separated in these data: P100, N100, N2, N3 (or P1r I chose N3 for my own reasons), MFN, and P300.]