Evaluation of Gene Selection Using Support Vector Machine Recursive Feature Elimination Committee: Advisor: Dr. Rosemary Renaut Dr. Adrienne C. Scheck Dr. Kenneth Hoober Dr. Bradford Kirkman-Liff John Huynh E-mail: jahuynh@yahoo.com
Contents Introduction Data Support Vector Machine Feature Selection Hypothesis & Experimental design Result Conclusion Future work Experience Reference
Terminology Sample is data set including Gene = feature = attribute = column Example = data point = slide = array = row r_1 r_2 r_i r_m x1 x_11 x_21 x_i1 x_m1 x2 x_12 x_22 x_i2 x_m2...... xj x_1j x_2j x_ij x_mj...... xn x_1n x_2n x_in x_mn Class c_1 c_2 c_i c_m
Meningioma Dr. Adrienne C. Scheck s Lab, BNI (Barrow Neurological Institute) Meningioma: 20% of primary intracranial tumor Mortality/Morbidity: In one series by Coke et al, the overall survival rate for all patients at 5 and 10 years were 87% and 58%, respectively. Medial Sphenoid Wing Meningioma
Meningioma Correlating clinical process, microarray, NMR, and FISH with WHO classification grade I, II, and III. Tubercullum sellae meningioma
Anatomy Meningioma is tumor of arachnoid.
Histology Neuron & Purkinje cell (cerebellum) Neuroglial cells Astrocytes: nurture, support Protoplasmic astrocytes (gray matter) Fibrous astrocytes (white matter) Oligodendrocytes: myelin, support Microglia: immune system in brain Ependymal cells: epithelium Blood vessels
Meningioma - Histopathology Meningioma: whorl-like structure + psammoma bodies WHO grade I: benign WHO grade II: (atypical) A meningioma with increased mitotic activity or three or more of the following features: increased cellularity, small cells with high nucleus: cytoplasm ratio, prominent nucleoli, uninterrupted patternless or sheet-like growth, and foci of spontaneous or geographic necrosis. WHO grade III: (anaplastic) A meningioma exhibiting histological features of frank malignancy far in excess of the abnormalities present in atypical meningioma.
BNI Meningioma Data Affymetrix HG-U133 Plus 2.0 with 54,675 genes. Small data set with many genes Grade Primary Recurrence Total I 15 3 18 II 7 0 7 III 0 1 1 Total 22 4 26
BNI Meningioma Data Plan A: consider data as large data set Plan B: consider data as small data set Grade Train Test I 11 4 II 5 2 Total 16 6 Total 15 7 22
BNI Meningioma Data High quality
Microarray Gene expression- Microarray Pattern of gene expressions for each tissue Oligo-microarray vs cdna High density Fixed probe length (25) In-situ synthesis
Microarray Microarray explores gene expression in global scale. PM & MM
Lymphoma Data Amersham cdna microarray with 7129 genes Tissue = bone marrow, blood ALL: acute lymphocytic leukemia AML: acute myelogenous leukemia Incidence: peak 2-3 yrs old: 80/1,000,000; 2400 new/yr/usa, 31% of all cancers ALL AML Total Train Test 27 20 11 14 38 34 Total 47 25 72
Lymphoma Data Good quality Large sample size, smaller feature dimension
Inducer Problem The purpose of learning machine is to find the most accurate classifier by learning in the training set and testing in the testing set. It is the minimizing problem of the error function E in mathematics. Let call f is learning algorithm, data points X = {x1, x2,, xi,, xm} in Rn, target {y1, y2,, yi,, ym} in Y = {-1, +1} f: X Y xi f(xi) E = (yi - f(xi))2. Testing set requirement: the testing set must be never seen in the training process; otherwise the correctness of the testing phase is unexpected high.
Support Vector Machine Map data into the feature space Learn in the feature space Return the result to the output space Learning function f (xi) = xi w + b f(xi) > 0 for yi = +1, f(xi) < 0 for yi = -1 f(xi) = 0 for decision boundary Output space Input space Feature space
SVM Characteristics Maximum margin Low computer cost: Kernel function costs O(n). Training cost: the worst case costs O(nsv3 + nsv2m + nsvmn); the best case costs O(nm2). Testing cost: O(n).
Linear SVM - Separable Case No kernel = scalar dot product Margin = 2/ w minimizing w2 Constraints (xi w+b)yi >0
Linear SVM - Non-Separable Case Introduce slacks ξi to adjust the choosing of support vector when needed. This means adding a constraint C on the Lagrangean multipliers C = 100 in our experiment.
Non-Linear SVM There is no linear decision boundary in the input space
Non-Linear Support Vector Machine Introduce kernel function to map data into Euclidean high dimensional space: dot product.
Non-Linear Support Vector Machine Now the data and weight are in the hyperspace. Training and testing processes are in the high dimensional space
Problem of Microarray Data Instance space F1x F2 x x Fi x x Fn The training set must be a large enough subset of instance space. Over-fitting problem of small data set: the inducer performs well in training set, but acts poorly in test set. The computational cost of high dimensional data is so high (n = 54675). Multiple testing correction: FDR, SAM, Classical analysis methods are not suitable.
Feature Selection Benefits of feature selection are reducing Computer cost Over-fitting Feature selection actually is a search algorithm in the feature space to find the optimal feature subset. Given an inducer I, and a data set D with features X1,, Xi,, Xn from a distribution D over the labeled instance space, an optimal feature subset, Xopt, is a subset of the features such that the accuracy of the induced classifier C = I(D) is maximal (Kohavi97).
Feature Selection: How? Filter method vs wrapper method Feature ranking criteria Correlation coefficient Weight
Recursive Feature Elimination RFE is a top-down (backward) wrapper using weight as feature ranking criterion. Eliminate One feature in every loop: slow A subset in every loop: fast Are they the same optimal subsets? Is the feature ranking criteria are the same?
Feature Selection Meaning Create nested subsets Let define Rate of elimination Surviving subset Note that the feature selection module includes an inducer so the training set must be never seen in both Feature selection module Evaluation module (Kohavi97)
Full Two Factorial Experiment Design The evaluation cost is the accuracy. The evaluation methods: independent test and cross-validation. The inducer is SVM for both feature selection and evaluation (Guyon02). The factor A (row) is the rate of elimination. The factor B (column) is the surviving subset
Software Design Preprocessing data: linear normalization + log2 transformation (prep.java) SVM, feature selection and evaluation: Matlab 6.5R13
Result: Lymphoma The optimal subset is 32 genes.
Result: Lymphoma Box Plots Box Plots
Result: Lymphoma ANOVA Tsuc Vsuc
Result: Meningioma The optimal subset is 4.
Result: Meningioma Box Plots Small Plan: 4 Large Plan: 2 Large Small
Result: Meningioma ANOVA Correct choice is 4. Index 37881 22501 50198 16979 Probe 238018_at 222608_s_at 1564431_a_at 21552_x_at
Conclusion No interaction between the rate of elimination and the feature optimal subset Small data set: rely on cross-validation
Future Works More published data set: large + small, difficult + easy How small is small? Evaluation method for small data set: master gene lists + LOOCV Over-fitting and cross-validation
Experience Not all the data mining task will be success. Business focus: communication, learning, negotiation, team work, leadership, Understand and live with data: a high dimensional small data set Never alternate the data in preprocessing process (time cost) Experimental design: good planning Observation + Think + Reaction = Strategy Loop, deal with the fact, not with who. Repeatable: Document experiment results and analysis Welcome new idea: good + bad; read, read, and read Never seen rule of test data, evaluation algorithm, over-fitting Feature selection SVM Software
References (Blum) Avrim L. Blum and Pat Langley, Selection of Relevant Features and Examples in Machine Learning, http://citeseer.ist.psu.edu/blum97selection.html. (Burges98) Christopher J.C. Burges, A Turtorial on Support Vector Machines for Pattern Recognition, (1998), Web-print: http://citeseer.ist.psu.edu/397919.html. (Golub99) Golub et al, Molecular Classication of Cancer: Class Discovery and Class Prediction by Gene Expression Monitoring, Science 286 (1999), 531-7, http://www.broad.mit.edu/mpr/publications/projects/leukemia/golub et al 1999.pdf. (Guyon02) Isabelle Guyon et al., Gene Selection for Cancer Classication using Support Vector Machines, Machine Learning 46 (2002), no. 1-3, 389422, Web-print: http://citeseer.ist.psu.edu/guyon00gene.html. (Gunn98) Steve R. Gunn, Support Vector Machines for Classification and Regression, (1998), http:// www.ecs.soton.ac.uk/~srg/publications/pdf/svm.pdf (Kohavi97) Ron Kohavi and George H. John, Wrappers for Feature Subset Selection, Artifcial Intelligence 97 (97), 273-324. (Soroin03) Soroin Dr aghici, Data Analysis Tools for DNA Microarrays, Chapman and Hall/CRC, 2003. WHO Classification http://neurosurgery.mgh.harvard.edu/newwhobt.htm
Question?