Tumor Phylogenetic Lineage Separation by Medoidshift Clustering with Non-Positive Kernel

Transcription

1 Tumor Phylogenetc Lneage Separaton by Medodshft Clusterng wth Non-Postve Kernel Lu Xe School of Computer Scence Carnege Mellon Unversty Pttsburgh, PA Commttee Dr. Russell Schwartz, Dr. Kathryn Roeder, Dr. Roy Maxon Abstract Background. Computatonal approaches have been wdely used for dssectng tumor subtypes through mxture models. One successful approach nvolves formulatng ths as a problem equvalent to dentfyng smplces n a hgh-dmensonal space of genomc pont clouds, where the vertces of a smplex are assumed to represent unque cell clones such as healthy stem cells, tumor progentors and tumor subtypes. The hghly heterogeneous nature and the complex evolutonary trajectores of tumors pose sgnfcant challenges to conventonal methods for unmxng tumor composton, whch can be smplfed by usng clusterng methods to separate tumor evoluton lneages. Am. The goal of ths project s to desgn a clusterng method that addresses the specfc challenges posed by the propertes of tumor genomc data: 1) the number of cluster s unknown; 2) data ponts are clustered accordng to ther belongng smplces; 3) the data clouds from two clusters may overlap; 4) there s no exstng probablstc model yet developed to descrbe the dstrbuton of the data ponts nsde a cluster; 5) the edges of two smplces may form sharp angles. Methods. The new clusterng method s based on standard medodshft but coupled wth specalzed non-postve kernel functons. It converges on vertces of the smplces nstead of densty maxma, and classfy data ponts wth respect to the dstance to the convergng vertces. The convergence s guaranteed when the shadow kernel functon s non-ncreasng and concave. Data. Synthetc datasets are produced from 5 hypothetcal tumor progresson scenaros. The method was also appled on two real breast cancer datasets and one real lung cancer dataset. Results. The new method shows consstently better clusterng than stand medodshft n all test scenaros, and mprovements over k-medods n certan applcatons. It also captures some nterestng structural characterstcs of the real cancer dataset that could not be detected by pror automated approaches to tumor mxture separaton. Conclusons. The medodshft-based method descrbed n ths work better handles the structural constrants of subdvdng smplcal complex by fndng cluster centers at vertces of the smplces, whereas standard medodshft s heavly nfluenced by the densty on shared margns and may fal to dstngush adjacent smplces. Comparng to the prevously used k-medods method, the new method removed the often dffcult model selecton queston of choosng k, whch requres a pror knowledge of the number of oncogenetc lneages. 1

2 1 Background Genomcs research has dramatcally mproved our understandng of the nature of tumor progresson and the means of possble treatment. Tumor progresson, or tumorgeness, s an evolutonary process by whch tumor cells accumulate successve mutatons that lead to decreased growth control, ncreased nvasveness and eventually metastass. There has been much nterest n reconstructng ths process of evoluton because of ts relevance to dentfyng the drvng factors of mutaton and predctng drug response and future prognoss. Our understandng of tumor progresson has been radcally reshaped by the applcaton of new technologes for probng the genome, gene and proten expresson profles of tumors, whch have made t possble to dentfy key sub-types of tumors that may be clncally ndstngushable yet have very dstnct prognoses and responses to treatments [1, 2, 3, 4]. Uncoverng these tumor sub-types has helped drve the development of novel therapeutcs, known as targeted therapeutcs, that are more specfcally targeted to the partcular genetc defects that cause each cancer [5, 6, 7]. Despte the profound mpact molecular genetcs had on cancer research, however, we are only startng to embrace the full complexty of tumor evoluton. As a result of that, some recognzed sub-types reman poorly defned and many patents do not fall nto any currently recognzed sub-type [8]. Nevertheless, clncal treatment of cancer could receve consderable beneft from better technques to dentfy sub-types mssed by the prevalng expresson clusterng approaches, ther dagnostc sgnatures, and the genes essental to the pathogencty of partcular sub-types [8]. More sophstcated computatonal models have stemmed from the feld of phylogenetcs, where tumors are not consdered as merely random collectons of mutated cells, but rather evolvng populatons. The poneerng work of Desper et al. nferred tumor pylogenes, or oncogenetc trees, by treatng observed tumors as leaf nodes n a speces tree, estmatng evolutonary dstances from genomc and gene expresson profles, and applyng a varety of methods to obtan reasonable models of the major progresson pathways by whch tumors evolve across a patent populaton. [9, 10, 11]. Maxmum lkelhood estmaton has also been used to work wth common measurements of tumor state [12]. An alternatve to the above tumor-by-tumor phylogenetcs, the cell-by-cell approach reles nstead on heterogenety between ndvdual cells wthn sngle tumors to dentfy lkely pathways of progresson [13, 14, 15, 16, 17]. The Pennngton model forms tumor evoluton as a Stener tree problem wthn ndvdual patents, and uses pooled data from many patents to buld a global consensus network descrbng common evolutonary pathways across a patent populaton [13, 14]. It s based on the assumpton that tumors preserve remnants of ther earler cell populatons as they develop, and any gven tumor sample wll therefore consst of a heterogeneous combnaton of cells at dfferent stages of progresson along a common pathway, as well as possbly contamnaton by healthy cells of varous knds. Ths assumpton s supported by numerous studes usng recent technques such as fluourescence n stu hybrdzaton (FISH) and sngle cell sequencng. [15, 16, 17, 18]. Each of the above two methods has ts shortcomngs. The tumor-by-tumor approach overlooks the ntratumor heterogenety that can provde valuable clues to tumor progresson, and the cell-by-cell approach consders the ntratumor heterogenety nformaton but at the cost of lmtng the number of probes per cell and therefore the resoluton of measure. The more recent sngle-cell sequencng technque ncreased the resoluton of a sngle cell measurement to the genomc level, but so far the data s affected by nose and the sample sze s lmted by the hgh cost [19, 20, 21, 22, 23]. 2 Related works Schwartz et al. proposed a gap-brdgng method that computatonally nfers cell populaton from tssue-wde measurements by usng unmxng, a mathematcal formalsm of the problem of separatng fundamental components of mxed samples n whch each observaton s presumed to be an unknown convex combnaton of several hdden components. [8]. In the context of tumor phylogeny, each component corresponds to a cell clone ( node ) on the underlyng oncogenetc tree. Ther specfc approach vews components as vertces of a mult-dmensonal smplex that encloses the observed ponts, whch makes unmxng essentally the problem of nferrng the vertces and boundares of the smplex from a survey of the ponts nsde t [24]. Tollver et al. revsed the 2

3 unmxng model and derved a soft geometrc unmxng algorthm that robustly handles nosy observatons [25]. Instead of focusng on sub-structural nformaton of the data clouds, other popular mxture nference methods are centered around the probablty densty estmaton that best fts the observed data [26]. Both optmzed for data acqured from sngle-nucleotde polymorphsm (SNP) array, ABSOLUTE fts optmal CNV model and uses a karyotype lkelhood model as a pror [27], and ASCAT uses addtonal nformaton such as varant allele frequency (VAF) based upon the avalablty of the raw sequencng data [28]. Some methods also reles on sequencng and VAF data, such as PyClone that uses herarchcal Bayesan clusterng [29] and ScClone that uses Bayersan mxture modelng [26]. CNAnorm s a another method desgned for sequencng data and nfers heterogenety and copy numbers separately, but t reles on the assumpton that tumor s largely monoclonal [30]. Assumng that the tumor sub-populatons can be dstngushed by CNV nformaton, the THetA algorthm poses heterogenety nference as a maxmum lkelhood mxture decomposton problem [31]. The above numerous strateges towards deconvoluton of tumor genomc data can typcally resolve up to approxmately 10 dstnct cell types, whch lmts ther ablty to resolve fner detals of cellular heterogenety wthn tumors [32]. To address ths ssue, Roman et al. proposed a methodologcal mprovement on genomc mxture modelng by explotng the fact that mxed genomc data from cells evolvng accordng to an evolutonary tree model would be expected to have fner mathematcal substructures than a sngle unform smplex assumed by pror work. In partcular, pont clouds produced by representng tumors as ponts n a genomc space (e.g., by gene expresson or gene copy numbers) would be expected to yeld smplcal complexes: conjunctons of low-dmensonal sub-smplces, correspondng to dstnct tumor subtypes, joned to one another va lower-dmensonal surfaces correspondng to shared ancestral cell populatons ncludng healthy cells [32]. Reconstructng these smplcal complexes can be consdered a specal case of the technque of manfold learnng [33]. In the work of Roman et al., k-medods clusterng s employed to separate pont clouds from dfferent sub-smplces, but t requres the pre-selecton of cluster number k that corresponds to the number of lneages n the unknown oncogenetc tree. In ths project, the structural-based clusterng approach s extended wth a medodshft-based method that s desgned to better dentfy clusters of ponts on dstnct low-dmensonal subspaces of the full manfold and remove the often dffcult task of model selecton regardng the choce of k. The standard medodshft method automatcally seeks the densty centers of the pont clouds known as modes, but t may break down n the context of separatng ndvdual smplces from smplcal complex due to the possblty that densty centers may locate on the conjuncton surface. The general methodology of medodshft provdes a good platform for desgnng problem-specfc classfer for tumor genomc data, as t s free from any pre-assumpton on data dstrbuton or cluster number, and guaranteed to converge wth only one-tme computaton of parwse dstances [34], 3 Problem descrpton The project s amed to desgn a problem-specfc algorthm for clusterng tumor samples wth respect to ther belongng smplces n a smplcal complex embedded n hgh-dmensonal featurespace. The vertces of each smplex are assumed to represent the cell clones that map to the nodes on the oncogenetc tree of the partcular type of tumor, and the number of shared nodes by two lneages determnes the number of vertces shared by two smplces. The nput dataset s organzed as a matrx whose rows and columns are assocated wth samples and measurements of certan features, respectvely. 4 Method 4.1 Algorthms and mplementaton Besdes the development of medodshft wth non-postve kernel (NPK), ths project also nvolves standard medodshft and k-medods as competng algorthms, and adjusted rand ndex as a measure of clusterng results. NPK-medodshft, standard medodshft and ARI are scrpted n MATLAB R2014b, and k-medods s mplemented as a bult-n functon of MATLAB R2014b wth k-means++ 3

4 seedng. The bult-n functon graphallshortestpaths of MATLAB R2014b s used whenever the shortest L2-squared path s needed as dstance metrc. 4.2 Standard medodshft revsted The dervaton of the medodshft algorthm manly follows the work of Shekh et al. [34] and Comancu et al. [35]. Smlar to meanshft, the medodshft algorthm s desgned to fnd the modes of estmated kernel densty, but only at data ponts rather than n the whole contnuous parameter space: f(x ) = 1 c n Φ(z j ) (1) j Here Φ( ) s the kernel functon usng profle notaton, and x, x j are data ponts such that 1 n, 1 j n. The factors c and h are the normalzer and kernel bandwdth, respectvely. The dstance measure z j can be any non-negatve dstance metrc between data ponts x and x j, for example, the scaled L2-squared dstance: z j = x x j 2 0 (2) h To formulate a medodshft task, frst the orgnal kernel, or shadow kernel Φ( ) must be translated nto a medodshft kernel by: where the teratve update rule s: ϕ( ) = Φ ( ) (3) y p+1 = arg mn y {x 1...x n} n ( x y 2 ϕ ( x y p h 2)) (4) Here y p s the medod at p th step. In fact, because the sequence of y belongs to the set of data ponts, one teraton of the above mnmzaton wll provde enough nformaton to trace out the convergence trajectory for all data ponts. The trace termnates when y k+1 = y k. Shekh et al. proved that medodshft s guaranteed to converge when the shadow kernel functon Φ( ) s convex. The resultng sequence y has the property that f(y p+q ) > f(y p ) for all q > 0 (Theorem 2.1 n [34]). Equvalently, a convex shadow kernel means ϕ ( ) 0, or ϕ(x) s a monotoncally non-ncreasng functon on x 0 because of the symmetry of Φ( ) and ϕ( ). 4.3 Medodshft wth non-postve kernel (NPK) As dscussed n the background secton, the standard medodshft clusterng may not work well for separatng smplces from a smplcal complex due to the possblty that the smplces, despte occupyng dstnct subspaces, may yeld pont clouds so overlappng that wll result n a super cluster that ncludes all adjacent smplces. By the structural characterstcs of a smplcal complex, a better strategy s to fnd the class centers located at densty mnma, often one vertex that s farthest from the densty center for each smplex, and group the ponts n the contanng smplex nto one cluster. Intutvely, ths can be acheved by reversng the sgn of the kernel functon n Equaton 3: φ( ) = Φ ( ) (5) 4

5 The convergence of medodshfht wth φ( ) s guaranteed wth a concave shadow kernel Φ( ) and the same update rule n Equaton 4 wth φ( ) nstead of ϕ( ). To prove that, t s suffcent to show that the resultng sequence y has the property that f(y p+q ) > f(y p ) for all q > 0. From Equaton 1, f(y p+1 ) f(y p ) = 1 c n ( Φ ( x y p+1 h 2) Φ ( x y p 2)) (6) h and the concavty of Φ( ), f(y p+1 ) f(y p ) 1 h 2 c n ( φ ( x y p h 2)( x y p+1 2 x y p 2)) (7) By the update rule n Equaton 4 and the termnaton at y p+1 = y p, n ( x y p+1 2 φ ( x y p 2)) < h n ( x y p 2 φ ( x y p h 2)) (8) Combnng the two above nequaltes, we conclude that f(y p+1 ) < f(p) for all p, thus the sequence f(y) s strctly decreasng. Ths guarantee of convergence wll however break down f a hard cut-off s appled. Hard cut-offs, such as Φ(x) = 0 f x > 1, are common for other kernels but they wll break the concavty of Φ(x). Although the convergence only requres concavty, non-decreasng Φ( ) such as log(z j + 1) wll result n an estmated kernel densty that s nversely proportonal to data densty, makng a decreasng sequence of f( ) converge at the densty maxma. A nonncreasng Φ( ) mples a non-postve frst dervatve,.e. φ( ) 0, and the concavty of Φ( ) mples φ ( ) 0,.e. φ( ) s non-ncreasng stage medodshft To suppress the nose n data clouds, a standard medodshft s used as a generc denoser before usng NPK medodshft. Wthout knowng the underlyng nose dstrbuton, we collapse the data clouds by steppng one teraton forward wth standard medodshft, and then use NPK medodshft to cluster the remanng ponts. Labelng s preserved when trackng the trajectory of medods across the two stages. 4.5 Assessment of clusterng results It s a non-trval work to fnd a good measure that tells the qualty of clusterng, as the values of true postve (TP), true negatve (TN), false postve (FP) and false negatve (FN) are hard to defne when the number of clusters gven by medodshft s not always the same as ground truth. One promsng measure s the adjusted Rand ndex (ARI), whch defnes TP, TN, FP and FN accordng to the parwse relatons among data ponts [36]: TP: a par of ponts that belong to the same true cluster are also n the same nferred cluster. TN: a par of ponts that belong to dfferent true clusters are not n the same nferred cluster. FP: a par of ponts that belong to dfferent true clusters are n the same nferred cluster. FN: a par of ponts that belong to the same true cluster are not n the same nferred cluster. Assume there are v true clusters denoted by V, 1 v, w nferred clusters denoted by W j, 1 j w, and m j = V W j, then the value of ARI s computed as [37]: ARI = ( mj ) 2 t3 (9) t 1 + t 2 t 3 v w j 5

6 where t 1 = 1 2 v ( ) V ; t 2 = w ( ) Wj ; t 3 = 8t 1t 2 2 n(n 1) j (10) One advantage of usng ARI s ts strngent penalty for random clusterng and trval clusterng. As mentoned above, standard medodshft often creates a trval cluster that ncludes all data ponts, whch wll be rated as low as 0 by ARI but stll as much as 0.5 by Jaccard ndex, another popular measure of clusterng results [38, 39]. 4.6 Kernel functons and bandwdth selecton In ths project, the followng shadow kernel and kernel are used wth the standard medodshft method and the frst stage of 2-stage medodshft: { 1 Φ 0 (z j ) = 2 (1 z j) 2 z j 1 0 z j > 1 { 1 zj z ϕ 0 (z j ) = j 1 0 z j > 1 (11) and the followng shadow kernel and kernel are used wth NPK medodshft and the second stage of 2-stage medodshft: Φ 1 (z j ) = z j exp(z j ) φ 1 (z j ) = 1 exp(z j ) (12) To provde a relatvely farer comparson across datasets wth varyng sparsty and dstance metrc, the kernel bandwdth was not set to a fxed value, but rather decded by the mean of parwse dstance and the choce of a bandwdth factor h c. The followng s an example for L2-squared dstance: h 2 = h c n 2 and the dstance metrc s therefore scaled as: n n x x j 2 (13) j n 2 x x j 2 z j = n n h c j x (14) x j 2 5 Dataset 5.1 Synthetc scenaros Evaluaton of the effectveness of the methods s complcated by the lack of ground truth mxture compostons from real heterogeneous tumor data. Synthetc datasets generated from smulated scenaros wth known mxture components, mxture fractons, and smplcal structures therefore provde a relatvely far testbed to quantfy effectveness of the methods. Fve tumor evoluton scenaros are consdered here, each nvolvng a model of tumors evolvng nto two or three subtypes: Scenaro T2E conssts of a model n whch a healthy state evolves nto an early precancerous state that subsequently branches nto two subtypes of late state (Fgure 1 A). Ths model results n a structure of two trangles sharng an edge (Fgure 1 F). One trangle has 50 data ponts and the other has

7 Fgure 1: The smulated tumor progresson scenaros for producng synthetc datasets. Subfgures A, B, C, D, E show fve oncogenetc trees of the synthetc model (T2E, T2V, Q2V, L3V, TLV, respectvely), each wth healthy cells at root, tumor progentor cells as nternal nodes, and tumor subtypes at leaves. Ther correspondng smplcal complex structures are shown n subfgures F, G, H, I, J, respectvely, and synthetc data ponts are randomly drawn from the enclosed smplces. Note that the structures are embedded n 10-Dmensonal space whle llustrated n 3-D for dagrammatc vew. Scenaro T2V conssts of two subtypes each wth ts own early and a late progresson stages (Fgure 1 B), yeldng a structure of two trangles joned at a vertex (Fgure 1 G). One trangle has 50 data ponts and the other has 100. Scenaro Q2V conssts of a more complex two-subtype model, n whch each subtype s defned by early, ntermedate, and late progresson stages (Fgure 1 C). The resultng structure s a par of tetrathedra joned at a vertex (Fgure 1 H). One tetrahedron has 50 data ponts and the other has 100. Scenaro L3V assumes an ancestral healthy cell subpopulaton that dverges nto three dscrete subtypes each wth a sngle progresson state (Fgure 1 D). The result s a smplcal complex structure consstng of three lnes joned at a vertex (Fgure 1 I). Each of the lnes have 25, 50, 75 data ponts, respectvely. Scenaro TLV conssts of two subtypes wth unequal number of ntermedate stages: one s the drect descendant of healthy cell, the other has one precancerous stage (Fgure 1 E). The resultng structure s a lne and a trangle sharng a vertex (Fgure 1 J). The lne has 25 data ponts and the trangle has 100 data ponts. The structures are embedded n 10-Dmensonal space by placng the vertex that represents healthy cells at the orgn, and each of the other vertces on a unque base vector wth unt dstance away from the orgn. Under noseless condton, data ponts are unformly randomly placed nsde the enclosed smplex; and under nosy condtons, Gaussan noses wth standard devaton σ = 0.05, 0.1, 0.2 are added to each dmenson of each data pont. 5.2 Real tumor data Usng array comparatve genomc hybrdzaton (acgh), a molecular cytogenetc technque for analysng copy number varatons (CNVs) relatve to plody level n the DNA of a test sample compared to a reference sample, Navn et al. assayed CNVs of 83,055 genes from 87 breast tumor samples, formng an 87-by-83,055 matrx [40]. Prncple component analyss (PCA) reduced the 7

8 dmensonalty down to 86, wth the projected space consstng of the frst 10 prncple components (PC) capturng approxmately 90% of the data covarance. Jones et al. measured the expresson levels of genes n 91 lung cancer cells by complementary DNA (cdna) mcroarray [41]. The pre-process of the dataset follows prevous work n Schwartz et al. [8], where the data ponts were translated from log space to lnear space and the upper and lower lmts are set to 2 5 and 2 5, respectvely. The values outsde the lmts were set to the closer lmt, and two outlers were dscard from the dataset. The dmensonalty was reduced to 88 by PCA. Retreved from the cancer genome atlas (TCGA), the thrd dataset conssts of 1100 breast tumor samples profled at 20,243 expresson levels wth RNASeq [42], a technology that uses the capabltes of next-generaton sequencng to reveal a snapshot of RNA presence and quantty from a genome at a gven moment n tme. Three outlers were dscarded, and the remanng data ponts were normalzed by dmenson-wse standard devaton. The dmensonalty was reduced to 1096 by PCA. For all three datasets, the data ponts projected n the PC space were lnearly scaled to ft n a range of [0, 1] n each dmenson. 6 Results The standard medodshft, NPK medodshft and 2-stage medodshft methods were appled to all synthetc scenaros (Fgure 2), where data ponts are randomly generated for 100 replcates per scenaro per choce of σ. For the frst 2 methods, h c takes the values from 0.2 to 2 wth a step sze of 0.2, and for 2-stage medodshft, h c s fxed at 1 for the frst stage and ranges from 0.2 to 2 wth a step sze of 0.2 for the second stage. The results of the three methods obtaned at h c = 1 are then compared aganst k-medods wth the data generated under the same scenaros and levels of nose (Fgure 3). By applyng NPK medodshft (wth h c = 1) on the frst 3-PC space of the real acgh data, one can fnd 3 lne-shaped clusters joned at one vertex, and the cluster centers located on the dsjoned endponts of the lnes (Fgure 4 B). The spatal dstrbuton of the data ponts projected n the frst 3- PC space may nspre people usng 3-medods clusterng, whch yelds ndeed qute smlar clusters for acgh data, but wth cluster centers located at the densty centers (Fgure 4 A). Appled to the frst 5-PC space of the acgh cancer data, NPK medodshft (wth h c = 1) fnds one more cluster that resdes near the orgn n the frst 3-PC space (Fgure 5 A). Although ndstngushable n the frst 3-PC space, the new cluster shows consderable dvergence n the addtonal PC spaces (Fgure 5 B, C) from the other clusters. The dstrbuton of cdna data ponts n the frst 3-PC space shows a smlar 3-arm shape to the acgh case, wth the dfference beng arched arms and sharper angles between each par of arms (Fgure 4 C, D). Shortest L2 squared path s used as dstance measure between ponts n the acgh data, and NPK medodshft (wth h c = 1) classfed each arm as ts own cluster wth centers located agan on the dsjoned endponts of the arms (Fgure 4 D). The applcaton of 3-medods faled to gve desrable clusters that could separate the arms (Fgure 4 C). The data clouds n TCGA dataset have a much noser appearance whch makes the separaton of potental clusters a much harder task than n the prevous two real datasets (Fgure 6). Coupled wth shortest L2-path dstance metrc and kernel bandwdth factor h c = 1, the applcaton of 2-stage medodshft on TCGA dataset dscovered 3 clusters n the frst 4-PC space (Fgure 6). Despte the nosness, the red cluster shapes lke a tetrahedron n the frst 3-PC space, and a flat dsk n the PC space; and the blue and black clusters appear to be a trangle and a rod respectvely n both PC and PC spaces (Fgure 6). 7 Dscusson Comparng across the fve scenaros, fndng the rght clusters for Q2V and T2V s much easer than for the others, because under these scenaros the two smplces share only one vertex whle havng most of the data clouds away from the jonng vertex. Scenaro T2E s a much harder case due to ts densty center beng placed on the jonng edge. In theory, TLV s an easer case than L3V, whch s 8

9 Fgure 2: Comparng clusterng results of synthetc Scenaros Q2V (B, C, D, E), T2V (G, H, I, J), TLV (L, M, N, O), L3V (Q, R, S, T) and T2E (V, W, X, Y) by standard medodshft (red curves), NPK medodshft (blue curves) and 2-stage medodshft (black curves). Gaussan nose s added to the datasets wth σ = 0 (B, G, L, Q, V), 0.05 (C, H, M, R, W), 0.1 (D, I, N, S, X), and 0.2 (E, J, O, T, Y). For each subfgure of errorbars, the vertcal axs ndcates the value of ARI for every subfgure, the horzontal axs ndcates the kernel bandwdth factor, and the errorbars show one standard devaton above and below the mean ARI value. The structural llustratons of the scenaros are shown n Subfgures A, F, K, P, U, respectvely. In general, 2-stage medodshft (black curves) gves the hghest ARI values, whle standard medodshft gves the lowest and most bandwdth-senstve ARI. proved by k-medods, but NPK medodshft has a slghtly lower success rate n clusterng TLV than L3V, whch s caused by occasonally mstakng the two vertces n the trangle as two endponts n dfferent smplces. All methods perform poorly at nose level 0.2 under all scenaros. Comparng across the frst three methods, the standard medodshft yelds low and h c -senstve ARIs for all scenaros, whle 2-stage medodshft gves overall the hghest and most stable ARIs. Scenaro TLV s relatvely senstve to kernel bandwdth wth NPK medodshft (Fgure 2 L, M, N, P), whch prefers the choce of smaller bandwdth value under all nose condtons. The performance of NPK medodshft s close to 2-stage medodshft n term of ARI value under Scenaro L3V (Fgure 2 Q, R, S, T). Addng k-medods to the comparson, t s obvous that the k-medods method has slght advantages over 2-stage medodshft under Scenaros Q2V (Fgure 3 B, C, D, E), T2V (Fgure 3 G, H, I, J) and TLV (Fgure 3 L, M, N), but slght dsadvantages under the other two harder scenaros (Fgure 3 Q, R, S, V, W, X). 9

10 Fgure 3: Comparng clusterng results of synthetc Scenaros Q2V (B, C, D, E), T2V (G, H, I, J), TLV (L, M, N, O), L3V (Q, R, S, T) and T2E (V, W, X, Y) by standard medodshft, NPK medodshft, 2-stage medodshft and k-medods wth the correct k. Gaussan nose s added to the datasets wth σ = 0 (B, G, L, Q, V), 0.05 (C, H, M, R, W), 0.1 (D, I, N, S, X), and 0.2 (E, J, O, T, Y). For each subfgure of boxplots, the vertcal axs ndcates the value of ARI, the horzontal axs corresponds to the lst of methods, the red bars ndcate the medan, the boxes nclude the frst to the thrd quarter quantle, the whskers reach to extrema, and the red pluses mark the outlers. The structural llustratons of the scenaros are shown n Subfgures A, F, K, P, U, respectvely. K- medods gves the hghest ARI under scenaros Q2V, T2V and TLV, whle 2-stage medodshft and NPK medodshft gve hgher ARI under scenaros L3V and T2E. Applcatons of NPK medodshft and 2-stage medodshft on real cancer datasets show reasonable clusterng results whch one would expect from ther sub-structural appearances. In the cases of acgh and cdna datasets, where data clouds spread n 3 dstnct arms n the frst 3-PC space, the NPK medodshft performs superor than standard medodshft even k-medods, whch s n accordance wth the smlar synthetc scenaro L3V. In the cases of cdna and TCGA datasets where nter-cluster dstances may be shorter than ntra-cluster dstances for some data ponts measured n Eucldean space, usng shortest path as dstance metrc mght be able to create the correct clusterng as n other cases of manfold learnng [33]. In comparson, standard medodshft faled to produce any desrable clusterng wth a range of bandwdth when beng appled to the frst 3-PC space of real acgh and cdna data(fgure 9). Affected 10

11 Fgure 4: Comparng clusterng results of the acgh (A, B) and cdna (C, D) data projected n the space consstng of the frst 3 PC by 3-medods (A, C) and NPK medodshft (B, D). Data ponts are colored wth respect to ther belongng clusters, and the crcles ndcate the correspondng cluster centers. The 3-medods method found the cluster center located at local densty maxma, whle NPK medodshft found them located at vertces farthest away from the conjont pont. by the fluctuaton n the spatal densty of the data cloud, a small bandwdth may lead to too many smaller clusters, and a large bandwdth may nclude two or more clusters n a super cluster. K-medods works reasonably well wth the correct number of clusters. In the applcatons on acgh and cdna datasets, t seems obvous that k should be set to 3 whle clusterng the frst 3-PC space wth k-medods method n both cases, but addng more PCs to the space hnders the determnaton of cluster numbers by vsually checkng the dstrbuton of data ponts, and the use of k-medods method therefore requres the nference of k by model selecton under these condtons. Medodshft-based methods do not rely on predefned cluster numbers; nstead the number of nferred clusters s nfluenced by kernel bandwdth, and such nfluence s less promnent wth NPK medodshft. 8 Lmtatons When developng the synthetc scenaros, the vertces of the smplces are assumed to occupy dstnct subspaces n the synthetc scenaros, resultng n perpendcular edges joned at the orgn. Ths assumpton follows as a consequence of embeddng a low-dmensonal manfold n a much hgher dmenson, where t s less lkely to have more than one vertex n one dmenson. Another smplfcaton n the scenaros s that the vertces have equal dstance from the orgnal, whch follows the sole crteron for classfyng data ponts wth respect to the found cluster centers beng mnmzng the dstance from each data pont to the center of ts belongng cluster. As a consequence of that, when a vertex s chosen to be the center of a smplex by NPK medodshft, the correctness of clusterng reles on the dstance from that vertex to the decson boundary. Takng scenaro L3V for an example, the resultng clusters mght be skewed f the lnes have dfferent length measured n 11

12 certan dstance metrc. A possble remedy for more complcated stuaton n manfold learnng s to develop more sophstcated metrcs. Lots of factors n the real world may affect the accuracy of clusterng, wth an mportant one beng the nosy observatons. In the synthetc scenaros, Gaussan nose s added wth standard devaton up to σ = 0.2, and the k-medods and 2-stage medodshft can produce meanngful clusters up to σ = 0.1 for some scenaros. It s non-trval to estmate the nose nduced by expermental measures wth dfferent genomc proflng technques; Su et al. estmated σ to range from 0.05 to 0.38 for RNA-seq [43]. When these datasets usually have much hgher dmensonalty than the number of data ponts, PCA may serve as a denoser but complcates the quantfcaton of the remanng nose after PCA. Two other factors - the surface shared by two or more smplces and the number of data ponts n each smplex - are nherent to the structure of the tumor phylogenetc tree and the source of the samples. The results from clusterng the synthetc datasets show the tendency of a less accurate clusterng f a hgher proporton of the precancerous states s shared among the lneages. The synthetc scenaros are created delberately wth unequal cluster sze, whch accounts for the uncertanty n the possble real applcatons to some extent. The accuracy of every method ncreases substantally when data ponts are evenly dstrbuted across the smplces (results not shown). The clusterng results are nfluenced by more PCs addng to the space (Fgure 7, 8), but whether the fluctuaton n clusterng s a reflecton of actual oncogenetc meanng or merely drven by nose requres further oncogenetc analyss. 9 Concluson The work n ths project fts n the framework of tumor phylogenetc study by facltatng unmxng tumor heterogenety wth the separaton of possble phylogenetc lneages. The new-ntroduced NPK medodshft method works as a desrable classfer that handles the specal challenges n clusterng generc tumor genomc data, and s less dependent on the a pror knowledge and expermental source of the data. Non-conventonal kernels such as Φ 1 ( ) and φ 1 ( ) may have lttle meanng n most statstcal applcatons that nvolve kernel densty estmaton, but they serve the purpose n fndng data ponts located at geometrc extrema of the smplces. Applcaton on the synthetc scenaros proves that 2-stage medodshft can produce clusters of smlar qualty to k-medods wth known cluster numbers, and outperforms standard medodshft method n general. NPK medodshft and 2-stage medodshft captured prevously undscovered sub-spacal characterstcs of the pont clouds from real breast cancer data, but the bologcal meanng of the dscovered clusters n real dataset requres verfcaton from unmxng and further oncogenetc studes. 10 Future works The results suggest further work may be needed to buld more realstc synthetc scenaros that can better reflect the nature of tumor genomc data. Some drectons seem promsng n makng mprovements n synthetc data generaton, for example, dervng a better data generatng method that approxmates the nose n real measurements, and desgnng new scenaros wthout the assumptons of vertces occupyng ndvdual dmenson and equal dstance from vertces to the orgn. In case of clusterng data wth more complcated structures, one mght consder fndng a problemspecfc dstance metrc, but mght also beneft from a problem-specfc kernel functon. Unlke standard kernel densty estmaton where the choce of kernel functons plays a much less mportant role than that of the kernel bandwdth, couplng medodshft wth dfferent NPKs may lead to unexpected clusterng results. For example, by usng a flat kernel φ 1 ( ) = 1 medodshft wll smply create a sngle trval cluster centered at the pont that has the largest total dstance to the other ponts: y = arg mn y {x 1...x n} n x y 2 (15) 12

13 The use of φ 1 (z j ) = zj r wll result n undesrable nsenstvty to the choce of bandwdth: y p+1 = arg mn y {x 1...x n} = arg mn n y {x 1...x n} h 2r = arg mn y {x 1...x n} 1 x y 2( x y p 2r h 2r ) n x y 2 x y p 2r n x y 2 x y p 2r (16) The NPK φ 1 (z j ) = exp( z j ) 1 has some nterestng features. When h s too small, exp( z j ) 1 approxmates -1 and creates trval clusterng. When h s too large, exp( z j ) 1 approxmates z j and becomes nsenstve to h. Nevertheless, the results from clusterng tumor genomc data, even the canddates from unmxng each cluster, are nconclusve of any bologcal meanng untl further oncogenetc valdatons have been performed. Acknowledgment The author apprecates the advsory of Dr. Russell Schwartz and the effort of the commttee members. Specal thanks to Theodore Roman for answerng countless questons, and to Dr. Roy Maxon for formulatng the layout of ths report. Ths work s funded by NIH Award R01CA and the Department of Computatonal Bology, School of Computer Scence, Carnege Mellon Unversty. References [1] Golub T, Slonm D, Tamayo P, Huard C, Gaasenbeek M, et al. (1999) Molecular classfcaton of cancer: class dscovery and class predcton by gene expresson montorng. Scence, 286: [2] Perou C, Sorle T, Esen M, Rjn van der M, Rees S, et al. (2000) Molecular portrats of human breast tumors. Nature, 406: [3] Sorle T, Perou C, Tbshran R, Aas T, Gesler S, et al. (2001) Gene expresson profles of breast carcnomas dstngush tumor subclasses wth clncal mplcatons. Proceedngs of Natonal Academy of Scence, 98: [4] Sorle T, Tbshran R, Parker J, Haste T, Marron J, et al. (2003) Repeated observaton of breast tumor subtypes n ndepednent gene expresson data sets. Proceedngs of Natonal Academy of Scence, 100: [5] Pegram M, Konecny G, Slamon D. (2000) The molecular and cellular bology of HER2/neu gene amplfcaton/overexpresson and the clncal development of herceptn (trastuzumab) therapy for breast cancer. Cancer Treatment Research, 103: [6] Atkns J, Gershell L. (2002) From the analyst s couch: Selectve antcancer drugs. Nataure Revews Cancer, 2: [7] Bld A, Pott A, Nevns J. (2006) Opnon: Lnkng oncogenc pathways wth therapeutc opportuntes. Nature Revews Cancer, 6: [8] Schwartz R, Shackney S. (2010) Applyng unmxng to gene expresson data for tumor phylogeny nference. BMC Bonformatcs, 11(42): [9] Desper R, Jang F, Kallonem O, Moch H, Papadmtrou C, et al. (1999) Inferrng tree models for oncogeness from comparatve genome hybrdzaton data. Journal of Computatonal Bology, 6: [10] Desper R, Jang F, Kallonem O, Moch H, Papadmtrou C, et al. (2000) Dstance-based reconstructon of tree models for oncogeness. Journal of Computatonal Bology, 7:

14 [11] Desper R, Khan J, Schaffer A. (2004) Tumor classfcaton usng phylogenetc methods on expresson data. Journal of Theoretcal Bology, 228: [12] von Heydebreck A, Gunawan B, Fzes L. (2004) Maxmum lkelhood estmaton of oncogenetc tree models. Bostatstcs, 5: [13] Pennngton G, Smth C, Shackney S, Schwartz R. (2006) Expectaton-maxmzaton method for the reconstructon of tumor phylogenes from sngle-cell data. Computatonal Systems Bonformatcs Conference, [14] Pennngton G, Smth C, Shackney S, Schwartz R. (2007) Reconstructng tumor phylogenes from sngle-cell data. Journal of Bonformatcs and Computatonal Bology, 5: [15] Smth C, Pollce A, Gu L, Brown K, Sngh S, et al. (2000) Correlatons among p53, Her- 2/neu and ras overexpresson and aneuoplody by multparameter flow cytometry n human breast cancer: evdence fro a common phenotypc evolutonary pattern n nfltratng ductal carcnomas. Clncal Cancer Research, 6: [16] Janocko L, Brown K, Smth C, Gu L, Pollce A, et al. (2001) Dstnctve patterns of Her-2/neu, c-myc and cycln D1 gene amplfcaton by fluorescence n stu hybrdzaton n prmary human breast cancers. Cytometry, 46: [17] Shackney S, Smth C, Pollce A, Brown K, Day R, et al. (2004) Intracellular patterns of Her- 2/neu, ras, and plody abnormaltes n prmary human breast cancers predct postoperatve clncal dsease-free survval. Clncal Cancer Research, 10: [18] Shah S, Morn R, Khattra J, Prentce L, Pugh T, et al. (2009) Mutatonal evoluton n a lobular breast tumour profled at sngle nucleotde resoluton. Nature, 461: [19] Navn N, Kendall J, Troge J, Andrews P, Rodgers L, et al. (2011) Tumour evoluton nferred by sngle-cell sequencng. Nature, 472(7341): [20] Wang D, Bodovtz S. (2010) Sngle cell analyss: the new fronter n omcs. Trends n botechnology, 28(6): [21] Tao Y, Ruan J, Yeh S, Lu X, Wang Y, et al. (2011) Rapd growth of a hepatocellular carcnoma and the drvng mutatons revealed by cell-populaton genetc analyss of whole-genome data. Proceedngs of the Natonal Academy of Scences, 108(29): [22] Hou Y, Song L, Zhu P, Zhang B, Tao Y, et al. (2012) Sngle-cell exome sequencng and monoclonal evoluton of a jak2-negatve myeloprolferatve neoplasm. Cell, 148(5): [23] Xu X, Hou Y, Yn X, Bao L, Tang A, et al. (2012) Sngle-cell exome sequencng reveals snglenucleotde mutaton characterstcs of a kdney tumor. Cell, 148(5): [24] Ehrlch R, Full W. (1987) Sortng out geology - unmxng mxtures. Use and Abuse of Statstcal Methods n the Earth Scences, Oxford Unversty Press, [25] Tollver D, Tsourakaks C, Subramanan A, Shackney S, Schwartz R. (2010) Robust unmxng of tumor states n array comparatve genomc hybrdzaton data. Bonformatcs, 26(ISMB): [26] Dng L, Wendl M, McMchael J, Raphael B. (2014) Expandng the computatonal toolbox for mnng cancer genomes. Nature Revews, 15: [27] Carter S, Cbulsks K, Helman E, McKenna A, Shen H, et al. (2012) Absolute quantfcaton of somatc DNA alteratons n human cancer. Nature Botechnology, 30: [28] van Loo P, Nordgard S, Lngjærde O, Russnes H, Rye I, et al. (2010) Allele-specfc copy number analyss of tumors. Proceedngs of the Natonal Academy of Scence, 107(39): [29] Shah S, Roth A, Goya R, Oloum A, Ha G, et al. (2012) The clonal and mutatonal evoluton spectrum of prmary trple-negatve breast cancers. Nature, 486: [30] Gusnanto A, Wood H, Pawtan Y, Rabbtts P, Berr S. (2012) Correctng for cancer genome sze and tumour cell content enables better estmaton of copy number alteratons from nextgeneraton sequence data. Bonformatcs, 28(1): [31] Oesper L, Mahmoody A, Raphael B. (2013) THetA: nferrng ntra-tumor heterogenety from hgh-throughput DNA sequencng data. Genome Bology, 14: R80. 14

15 [32] Roman T, Nayyer A, Fasy B, Schwartz R. (2014) A smplcal complex-based approach to unmxng tumor progresson data. Submtted [33] Rowes S, Saul L. (2000) Nonlnear dmensonalty reducton by locally lnear embeddng. Scence, 290(5500): [34] Shekh Y, Khan E, Kanade T. (2007) Mode-seekng by medodshft. IEEE 11th Internatonal Conference on Computer Vson (ICCV). [35] Comancu D, Meer P. (2002) Mean Shft: A Robust Approach Toward Feature Space Analyss. IEEE Transactons on Pattern Analyss and Machne Intellgence, 24(5): [36] Hubert L, Arabe P. (1985) Comparng parttons. Journal of Classfcaton, 2: [37] Kuncheva L, Hadjtodorov S. (2004) Usng Dversty n Cluster Ensembles. IEEE SMC Internatonal Conference on Systems, Man and Cybernetcs. [38] Wagner S, Wagner D. (2007) Comparng Clusterngs - An Overvew. [39] Denœud L, Garreta H, Guénoche A. (2006) Comparson of dstance ndces between parttons. t Studes n Classfcaton, Data Analyss, and Knowledge Organzaton, [40] Navn N, Krasntz A, Rodgers L, Cook K, Meth J, et al. (2010) Inferrng tumor progresson from genomc heterogenety. Genome research, 20(1): [41] Jones M, Vrtanen C, Honjoh D, Myosh C, Satoh Y, et al. (2004) Two prognostcally sgnfcant subtypes of hgh-grade lung neuroendocrne tumours ndependent of small-cell and large-cell neuroendocrne carcnomas dentfed by gene expresson profles. Lancet, 363: [42] Wensten J, Collsson E, Mlls G, Shaw K, Ozenberger B, et al. (2013) The cancer genome atlas pan-cancer analyss project. Nature genetcs, 45(10): [43] Su Z, Łabaj P, L S, Therry-Meg J, Therry-Meg D, et al. (2014) A comprehensve assessment of RNA-seq accuracy, reproducblty and nformaton content by the sequencng qualty control consortum. Nature Botechnology, 32(9):

16 Fgure 5: Resultng clusters of the acgh data projected n the space consstng of the frst 5 PC by NPK medodshft. Data ponts are colored wth respect to ther belongng clusters. The data ponts are shown n 3D spaces consstng of the 1st-2nd-3rd (A), 2nd-3rd-4th (B), and 3rd-4th-5th (C) PC. Despte ndstngushable n Subfgure A, the magenta cluster dverse from the other clusters n Subfgures B and C. 16

17 Fgure 6: Resultng clusters of TCGA data projected n the space consstng of the frst 4 PC by 2-stage medodshft. Data ponts are colored wth respect to ther belongng clusters. The data ponts are shown n dfferent vew angles n 3D spaces consstng of the 1st-2nd-3rd (A, C, E, G) and 2nd-3rd-4th (B, D, F, H) PC. The clusters appear to have dstnct shapes and spacal occupaton. 17

18 Fgure 7: Resultng clusters of acgh data projected n the space consstng of the frst 8 PC by medodshft wth non-postve kernel. Data ponts are colored wth respect to ther belongng clusters. The data ponts are shown n 3D spaces consstng of the 1st-2nd-3rd (A), 2nd-3rd-4th (B), 3rd-4th-5th (C), 4th-5th-6th (D), 5th-6th-7th (E), and 6th-7th-8th (F) PC. The ncreased dmensonalty nfluences the clusterng result wth the rsk of fttng nose. 18

19 Fgure 8: Resultng clusters of cdna data projected n the space consstng of the frst 6 PC by medodshft wth non-postve kernel. Data ponts are colored wth respect to ther belongng clusters. The data ponts are shown n 3D spaces consstng of the 1st-2nd-3rd (A), 2nd-3rd-4th (B), 3rd-4th-5th (C), and 4th-5th-6th (D) PC. The ncreased dmensonalty nfluences the clusterng result wth the rsk of fttng nose. 19

20 Fgure 9: Comparng clusterng results of acgh data projected n the space consstng of the frst 3 PC by standard medodshft wth kernel bandwdth factors of 0.5 (A), 1.0 (B), 1.5 (C), 2.0 (D), 2.5 (E), and 3.0 (F). Cluster centers are marked wth red crcles. The clusterng s senstve to the choce of bandwdth factor, and t fals to gve desrable clusterng n all cases. 20

21 Fgure 10: Comparng clusterng results of cdna data projected n the space consstng of the frst 3 PC by standard medodshft wth kernel bandwdth factors of 0.5 (A), 1.0 (B), 1.5 (C), 2.0 (D), 2.5 (E), and 3.0 (F). Cluster centers are marked wth red crcles. The clusterng s senstve to the choce of bandwdth factor, and t fals to gve desrable clusterng n all cases. 21