CLUSTERING is always popular in modern technology

Transcription

1 Max-Entropy Feed-Forward Clusterng Neural Network Han Xao, Xaoyan Zhu arxv: v1 [cs.lg] 11 Jun 2015 Abstract The outputs of non-lnear feed-forward neural network are postve, whch could be treated as probablty when they are normalzed to one. If we take Entropy-Based Prncple nto consderaton, the outputs for each sample could be represented as the dstrbuton of ths sample for dfferent clusters. Entropy-Based Prncple s the prncple wth whch we could estmate the unknown dstrbuton under some lmted condtons. As ths paper defnes two processes n Feed-Forward Neural Network, our lmted condton s the abstracted features of samples whch are worked out n the abstracton process. And the fnal outputs are the probablty dstrbuton for dfferent clusters n the clusterng process. As Entropy-Based Prncple s consdered nto the feed-forward neural network, a clusterng method s born. We have conducted some experments on sx open UCI datasets, comparng wth a few baselnes and appled purty as the measurement. The results llustrate that our method outperforms all the other baselnes that are most popular clusterng methods. Keywords Feed-Forward Neural Network, Clusterng, Max-Entropy Prncple, Probablstc Models I. INTRODUCTION CLUSTERING s always popular n modern technology of artfcal ntellgence. It s a large branch of machne learnng algorthms. No matter n theoretcal or practcal aspects, clusterng s always frutful and useful. The applcatons based on hgh qualty clusterng methods are so many, and most of them are mportant n ether ndustral or ndvdual usages. For example, n the area of busness, clusterng machnes help to fnd the users group [1], and n the area of bology, they can help to dscover genes and speces. In natural language processng, clusterng could help to dscover the group of Morphologcally Related Chnese Words [2]. Even n the task of dscoverng better conference paper keywords, clusterng could be very useful [3]. Above, clusterng s very mportant, and a better clusterng machne could not only mprove the basc belef of theores, but also promote the ntellgent products. Meanwhle, neural networks are also a large branch of learnng algorthms, and at present the deeper archtectures are becomng popular and attractve for promotng the ablty of neural network. The neural network, especally feed-forward neural network, could extract the features and characterstcs automatcally from the orgnal nput Xao s wth the State Key Lab. of Intellgent Technology and Systems, Natonal Lab. for Informaton Scence and Technology, Dept. of Computer Scence and Technology, Tsnghua Unversty, Beng , PR Chna (e-mal:xaoh12@mals.tsnghua.edu.cn) Zhu s wth the State Key Lab. of Intellgent Technology and Systems, Natonal Lab. for Informaton Scence and Technology, Dept. of Computer Scence and Technology, Tsnghua Unversty, Beng , PR Chna (e-mal:zxy-dcs@tsnghua.edu.cn) data. However, tradtonal feed-forward neural networks are supervsed learnng machne that can only complete the tasks such as regresson or classfcaton, whch means these methods need a lot of labelled data to tran the network, whle unsupervsed learnng tasks could hardly be completed by feed-forward neural networks. Though self-organzed mappng or Kohenon network s also unsupervsed learnng machne, they should belong to shadow models whch can not be mproved by the deeper archtecture. For these reasons, n ths paper, we would lke to propose an unsupervsed multlayer feed-forward neural network whch could beneft from both the extracton of features and unlabelled data. In the learnng process of supervsed learnng, the target functon s based on the costs or losses of labelled data. However, when we deal wth the problem of unlabelled data, what we hold are only the data manfolds. For ths cause, we must select other prncple, wth whch we could estmate the dstrbuton under the condtons of each layers data abstracton. So n ths paper, we have treated the output of neural layer as two knds of nformaton form. The outputs of each layer are both abstracted data features and probablty dstrbutons. For the latter form, Entropy-Based Prncple s consdered. In detal, the outputs of non-lnear feed-forward networks are postve, whch are the abstracted form of sample features and also could be vewed as probablty dstrbuton of dfferent components n data manfold. When the Entropy-Based Prncple s ntroduced nto the feed-forward neural network, we maxmze the entropy of clusterng layers probablty dstrbuton and mnmze the entropy of abstracton layers dstrbuton. As to the nference of ths neural network, we make the sample nto the component whch corresponds to the mnmum output neuron. Noted that n our model, ths mnmum output neuron also means the maxmum probablty neuron. By ths way, we could propose an unsupervsed multlayer feed-forward neural network. Our experments are conducted on sx UCI open datasets, that are Glass, Banknote Authentcaton, Whte Wne Qualty, Red Wne Qualty, Image Segment and MAGIC Gamma Telescope. We select four most popular clusterng algorthms as baselnes namely famous K-Means, densty-based method, herarchcal clusterng method and expectaton maxmum clusterng method. The results of our experments could prove the effectveness of our method. Our experments llustrate that our clusterng method works well and outperforms common clusterng methods. In concluson, the Entropy-Based unsupervsed learnng algorthm based on feed-forward neural network would be reasonable and behave well.

2 The man contrbuton of ths paper ncludes: 1) Ths paper studes two knds of propertes for the output of each neuron n network layers, whch are abstracted data features and cluster probablty dstrbuton. Wth ths theoretcal analyss, the Entropy-Based Prncple s appled nto the feed-forward neural network to make t as an unsupervsed learnng machne whch needs only unlabelled data. 2) An optmzaton problem s formulated based on above motvatons. Clusterng algorthm s desgned as a soluton of ths optmzaton problem. II. BACKGROUND AND RELATED WORK Clusterng methods have always been a very huge branch of modern machne learnng or data mnng methods and t s one of the man topc of unsupervsed learnng. [4] and [5] had surveyed the algorthms of clusterng methods. There are four knds of most popular clusterng algorthms. The frst knd of popular clusterng method s Dstance-Based Clusterng methods, whch are focusng on the dstance or smlarty between samples or centrods. The famous K-Means and X-Means Clusterng methods are ncluded n ths sort of algorthms as a specal applcaton of Vector Quantzaton, besdes the graph-based methods are also popular methods whch leverage the graph dstance as the metrc. The Fuzzy C-Means are based on Fuzzy Set Theores as ts metrc. Neural network model such as Self-Organzed Mappng(SOM), and ARTS network also belong to ths knd of clusterng algorthm. The second knd of popular clusterng method s the Herarchcal Clusterng Methods, and ths knd of methods are focusng on the teratve process to splt dfferent components n data manfolds. Methods such as Sngle lnkage, Complete lnkage, Group average lnkage, Medan lnkage, Centrod lnkage, Ward s method, Integer Lnear Programmng Herarchcal Clusterng [6], BIRCH, CURE and ROCK are also n ths knd of algorthms. The thrd popular knd of method s Densty-Based Methods. The methods are based on the local metrc between samples, whch can dscover many components n dfferent shapes, styles and forms. The famous DBSCAN method, ADBSCAN [7] and OPTICS are also n ths sort of clusterng algorthms. The fnal knd of methods are based on the probablstc vewpont, the famous EM clusterng method, mxed Gaussan Dstrbuton, Bayesan non-parametrc multlevel clusterng [8] and Evolutonary soft co-clusterng [9] pertan to ths sort. Besdes, the clusterng methods based on nformaton theory also belong to the fnal sort [10]. There are also some other clusterng methods, whle the above mentoned s the most popular ones. For an overvew of ablty compared between our model and common popular models, we select these four knds of methods as our baselnes. Feed-Forward neural networks are also a large branch of machne learnng algorthms, they have extraordnary data abstracton ablty especally n the deep archtectures. However, t s manly used n supervsed learnng, whch requres labelled data. The deep archtecture could provde data abstracton ablty and t could extract the features automatcally. Thus the clusterng tasks meetng the multlayer feed-forward neural network would become better than common neural network clusterng methods, such as SOM and also better than common popular clusterng methods. Recently such stated n [11], gong to deep catches many eyes, snce not only ust addng the hdden layers could gan an mprovement n performance, but deep neural networks can also automatcally select features and amazngly complete the comprehenson mssons. [12] had appled deep network nto natural languages, and many works such as [13] and [14] had appled deep network nto mage processng. Obvously, deep learnng s one of the hottest topc n today s machne learnng theores and methods. Before [15] and [16] proposed the fast unsupervsed and supervsed methods, mult-layer neural networks are hard to tran, ths knd of dffculty s analysed and solved n [17] and [18], for the same cause our method could also perform well n deep archtecture. Besdes our work s a knd of prncple, whch could both works for shadow or deep archtecture of feed-forward neural networks. In a word, we can cluster the data wth deep or shadow archtecture of Feed-Forward Neural Network, and a deeper archtecture would beneft more. Feed-forward neural network gans a powerful feature extracton ablty, but few works could make t a clusterng algorthm before ths paper. As we have analysed about ths stuaton, we conclude for two reasons. Frstly, tradtonal feed-forward neural network s the learnng machne for the labelled data, and there s no prncple or tranng target to make t able to learn from the unlabelled data. Based on ths motvaton, we propose Entropy-based prncple whch nvolves Mn-Entropy and Max-Entropy Prncple to overcome ths dffcult. Secondly, SOM and ARTs lead the road of neural clusterng computng, t seems that only SOM or recursve neural network could be able to the clusterng problem. However, wth Entropy-based Prncple, feed-forward neural network could also acheve ths ablty. Besdes t outperforms most common clusterng models wth ts amazng feature extracton ablty. Max-Entropy Prncple s always used n probablty dstrbuton estmaton where the dstrbuton s unknown but s lmted by some constraned condtons. The Max-Entropy Prncple could be leveraged nto logstc regresson, and also be appled nto feed-forward neural networks, whch s proposed n ths paper. The outputs of non-lnear feed-forward neural networks are postve, thus they can be treated as both the abstracton of orgnal data and the probablty dstrbuton of correspondng components n data manfolds. The abstracton of data manfold s the constraned condton, under whch the outputs of each layer are the unknown probablty dstrbutons to be estmated. Wth ths motvaton, Feed-Forward neural networks wth Max-Entropy Prncple are able to cluster unlabelled data. Due to ts strong ablty of the abstracton process, t could be better than popular clusterng methods. Our methods are both varfed wthn theortcal apsects and practcal aspects.

3 III. TWO PROCESSES IN FEED-FORWARD NEURAL A. A Bref Illustraton NETWORK In ths secton, we propose a novel vewpont to revew Feed-Forward Neural Network, whch explans the network wth two knds of processes that are Abstracton and Clusterng Process. As the process that Fg. 1 llustrates, the teratve operatons of feed-forward neural nformaton processng could be treated as abstracton processes, where the features of sample are transducted from orgnal feature space to abstracted feature space. In the stage of abstracton, each neuron would play a role as a lnear regresson learners, the new coordnated system s constructed by these lnear regresson learners wth non-lnear output functon as ts coordnates, and these lnear regresson learners could be vewed as the bass of the coordnated system for a new feature space. As the abstracton process n Fg. 1 shows, the hdden neurons correspond to those dashed lnes n orgnal space, and the dstance of samples to these lnes construct the coordnated system of new feature space for the next process. The abstracton process could make the essence of data more easy and smple to be revealed. In ths way, we could dscover clusters n abstracted data space where the essence of data could be revealed more easly. When the cluster nformaton s dscovered, we can estmate the probablty dstrbuton of samples for dfferent clusters n clusterng process. In the non-lnear feed-forward neural network, the outputs of each layer have two propertes. The frst s abstracted feature whch s dscussed n above and the second s probablty dstrbuton for some components n data manfolds. As we know, each neuron catches some characterstcs n data manfolds, whch we could explan as that each neuron catches some knd of data clusters n data manfolds. For a sample n a specfc cluster, the output neurons gve out the probablty dstrbuton of correspondng clusters, based on whch we could select the most possble cluster that the sample should belong to. The key of clusterng process whch could make the probablty dstrbuton for clusters n data manfold reasonable, les to the tranng process, or we say the tranng prncple. When we estmate the unknown probablty dstrbuton for data manfold, the Max-Entropy Prncple works well, and the clusterng algorthm s desgned based on ths prncple. B. Abstracton Process n Mn-Entropy Vewpont In the abstracton process, each neuron behaves as a lnear regresson learner, whch catches part of data characterstcs. Ths seems a lttle lke the densty-based methods, the abstractng neuron works as the densty detector for data manfold. The output of each neuron could be treated as abstracted features or probablstc dstrbutons. The output of a neuron s smaller when a sample s more near to ts correspondng regresson hyper-plane, whch llustrates that the lnear regresson learner works better. From the vewpont of data abstracton, when we would lke to enhance the abstracted ablty, we must optmze a target whch could Fg. 1. Two Processes n Clusterng Feed-Forward Neural Networks. make more samples near the only one correspondng lnear regresson learner. Ths part of target s supposed to have the followng form. N a 1 O 1 O Mn J = ( Na =1 1 O )log( Na =1 1 O ) (1) =1 In above formula, the N a s the neuron number n abstracton layer, and O s the -th unt output of ths layer or we say the dstance from sample to -th lnear regresson learner, whch s expressed n below part of ths paper. In mathematcal aspect, the target encourages that mnorty outputs of neurons become smaller and maor outputs of neurons become bgger, whch means that data samples get closer to one of the lnear regresson learners and the abstracted ablty of the abstractng neurons becomes better. Thus mnmzng ths target, we make the data densty detectors or we say our lnear regresson learners catch more essental data characterstcs. Also, we could revew ths target that we mnmze the entropy of the abstracton processes n probablstc or system vewpont. As we know, the entropy of system means the uncertan level of the system, and n ths stuaton, we want the data densty could be analysed clearly enough to belong to only one determned lnear regresson learner, whch means that we have to prefer a mnmum of system uncertan level. For ths reason, we must apply Mn-Entropy Prncple to abstracton process to gan the abstracton ablty. C. Clusterng Process n Max-Entropy Vewpont When the data densty manfold could be analysed by lower layers whch play the role of abstracton process, the output layer could behave as a clusterng process, where we must determne how a sample belongs to those components that are detected from the abstracton process. Ths s a probablstc estmaton problem, n whch the 1 O means the degree of the sample nearng the component as the same as the degree

4 of component-sample membershp. For ths sake, we apply the Max-Entropy Prncple to ths estmaton problem, wth a dstrbuton such lke (1 O 1,1 O 2,1 O O n ). We obtan the target as followng: N c 1 O 1 O Max J = ( Nc =1 1 O )log( Nc =1 1 O ) (2) =1 In above formula, the N c s the neuron number n clusterng layer, and O s the -th unt output of ths layer. D. Dffcultes n Clusterng And Soluton n Ths Paper In a common vewpont, there s a man dffculty n clusterng algorthms that the shape of cluster or the flexblty of clusterng model. Inflexble model can only characterze smple shape of cluster. For dstance based clusterng models, they own a fxed dstance expresson, whch leads to nflexblty of clusterng model. Meanwhle, for probablstc clusterng models, the assumpton of cluster shape or we say the flexblty of model means very mportant to these methods. Thus, many clusters wth dfferent and pecular shapes could not be detected well. In ths paper, our model could overcome ths dffculty n clusterng by abstracton and clusterng process provded by feed-forward neural network. As we know, non-lnear feed-forward neural network could express many knds of functons whch could be treated as many knds of smlarty or dstance metrc. For ths pont, our method provdes a very flexble clusterng model or a very varous cluster shape assumpton. In the abstracton process of our model, the data s transducted from one knd of extracted feature expresson to another knd of extracted feature expresson, and n the last knd of feature space, the data manfold or the cluster shape s very regular and easy to be revealed by Max-Entropy Prncple. Ths pont benefts from the motvaton and results of the deep neural networks. As we argued n ths subsecton, our model could both provde enough flexble clusterng ablty and detect many dfferent and pecular shapes. So our model s not lmted by ths knd of nflexble factor and overcomes ths dffculty to acheve a better result. IV. ALGORITHMS FOR MAX-ENTROPY FEED-FORWARD CLUSTERING NEURAL NETWORK In the whole vewpont, the last layer that the output layer behaves as a clusterng layer, and other layers that are the hdden layers could be treated as abstracton layers. From the prevous secton, we could work out the whole target as followed: N 1 O 1 O Max J = ( N =1 1 O )log( N =1 1 O )+ λ =1 L l L ( l=1 =1 1 O,l Ll =1 1 O,l )log( 1 O,l Ll =1 1 O ),l In above formula, the N means the unt number of output layer, and the L l means the unt number of l-th hdden layer. The L means the number of layers. The O means -th neuron output n output layer, and O,l means the -th neuron output n l-th hdden layer. In ths formula, we have the output of each layers as below: O = σ(<, O L >) (3) O,l = σ(< w l, O l 1 >) (4) In above two formulas, the σ means the non-lnear functon appled nto the clusterng neural network and the O l means the output vector of l-th layer that s composed by O l and O means the output vector of output layer. Besdes, the operator < x, y > means an nner product operator between x and y. In the vector-matrx form, we have followng formulaton. < x, y >= x T y The gradent descent optmzaton method s appled to solve the problem, so we must work out the dervatve of the target, for bref llustraton, we ntroduce the δ functon. δ out,l δ out,l 1 = δ l,l = O +N 1 N =1 O (N N =1 O ) 2 1 O (1 log( N =1 1 O ))σ (< w out, O >) (5) L l =1 δ out,l w l s ()σ (< w l 1, O l 1 >) (6) = O,l +L l 1 L l =1 O,l (L l L l =1 O,l) 2 (1 log( =1 1 O,l Ll =1 1 O ))σ (< w l, O l >) (7),l L l δ,l 1 l = δ,l l wl s ()σ (< w l 1, O l 1 >) (8) In above formulas, the w s the weght vector for correspondng neuron, and O l s the output vector of the l-th layer, whch s composed by all the outputs of neurons n the l-th layer. Wth the expresson of δ functon, we could reduce the complexty of computaton, and express the updatng equaton very brefly. J = δ out w out,l yl (9) J w l = δ out,l yl λ L δ,l s yl (10) In above formula, σ s the non-lnear functon and σ s the dervatve form of non-lnear functon. Hence the updatng equaton s obtaned as followng: = s=l +α J w l = wl +α J w l (11) (12) In above equaton, the α s the learnng rate, so the clusterng algorthm s obtaned. Our tranng algorthm as algorthm 1 shows, holds a

5 Algorthm 1 Tranng Algorthm For Feed-Forward Clusterng Neural Networks set randrom values to weghts repeat To nfer the neural network usng Algorthm 2 for all w out output layer do = for all w l hdden layer do +α J w l = w l +α J w l untl All Samples Convergence computaton complexty as O( NeuronNumber 2 LayerNumber 2 ) The complexty s proportonal to the number of neurons n each layer and also the number of layers. For the reason that the number of layer s fxed and small, our algorthm could run as fast as common back-propagaton neural network. As prevous secton stated, the re-desgned nference method s acheved as to modfy the normal feed-forward neural network nference algorthm, n whch we obtan the cluster that a sample belongs to by selectng the neuron whch output the maxmum of probablty notated as 1 O. That s to say that we select the mnmum output neuron as the cluster of our sample. Ths re-desgned nference algorthm or we say clusterng algorthm s almost the basc back-propagaton neural network nference algorthm. Only for the last step, we choose the mnmum dstance to the clustered component as our clusterng choce. Ths algorthm could also run as fast as common back-propagaton neural network nference method. Algorthm 2 Clusterng Algorthm As Inference n Feed-Forward Clusterng Neural Networks for all each sample x t do for all each hdden neuron hdden layer do y = σ(< w l, ol >) for all each output neuron output layer do t = σ(<, o L >) return Order Number of Mnmum Output of Neurons n Output Layers. TABLE I. The Purty of Dfferent Datasets For Dfferent Method Dataset K-Means Densty Her. EM Ours Glass BankNote Whte Wne Red Wne Image Segment MAGIC V. COMPARISON AMONG POPULAR CLUSTERING MODELS The connecton between dstance based methods wth our model s relatvely obvous. The tranng algorthm could be treated as a process to acheve a better dstance functon whch s encoded n the weghts of neural networks. Then the nference and clusterng process use ths learned dstance functon that our neural network, to cluster data. There would be a fact that the dstance of our model s traned from the dataset tself, wth flexble form and reasonable tranng prncple. So our model would not be easly affected by dfferent shapes of cluster or fxed dstance expresson. The connecton between densty-based methods wth our model s also relatvely obvous. Our method could be treated as a mult-layer or herarchcal densty-based methods. One layer densty based method could extract some clusterng nformaton, and ths paper apply many layers densty based method whle each layer could extract more clusterng nformaton than the prevous one. Ths process s ust the abstracton process of neural network. Wth ths process, our model that could be treated as herarchcal densty-based method would be more precous than common densty based methods. The connecton between herarchcal methods wth our model s not very obvous. However, as we know, herarchcal methods also need a knd of fxed metrc for lnkage weghts, whch n our model s changed to a flexble and expressve dstance formulaton that s obtaned by the tranng algorthm. For ths key pont, our model would gan the flexble dstance adaptve to the specfc dataset. Thus our model would be more competent than common herarchcal methods. The connecton between probablstc methods wth our model s relatvely obvous. Our model s ust a probablstc model or nformaton-theoretcal model. However, common probablty models are based on the orgnal data, the features of whch are rough. But ths paper onts probablstc models wth neural network. Thus, our model would abstract rough features to refned features whch could be easly analysed. VI. EXPERIMENTS We have conducted experments for the effectveness of Feed-Forward Clusterng Neural Networks. Each group of experments has acheved good results to prove our methods and our theores are effectve.

6 A. Expermental Settngs We have selected sx open UCI datasets, whch are often used n clusterng tasks. Banknote Authentcaton. The data were extracted from mages that were taken from genune and forged banknote-lke specmens and Wavelet Transform tool were used to extract features from mages. there are 1,372 tems wth 5 attrbutes for bnary classes. Glass. The study of classfcaton of types of glass was motvated by crmnologcal nvestgaton. At the scene of the crme, the glass left can be used as evdence. We use ths dataset to testfy our clusterng method. There are 214 nstances wth 10 attrbutes for 6 classes. Red Wne Qualty And Whte Wne Qualty. The two datasets are related to red and whte varants of the Portuguese Vnho Verde wne. There are 1,599 nstances wth 11 attrbutes n the frst dataset for 4 classes. And there are 4,899 nstances wth 11 attrbutes n the second dataset for 5 classes. Image Segmentaton. The nstances were drawn randomly from a database of 7 outdoor mages. The mages were hand-segmented to create a classfcaton for every pxel. There are 2,310 nstances wth 18 attrbutes for 7 classes. MAGIC Gamma Telescope. It s generated to smulate regstraton of hgh energy gamma partcles n a ground-based atmospherc Cherenkov gamma telescope usng the magng technque. There are 19,020 tems wth 11 attrbutes for bnary classes. B. Effectveness of Our method For evaluaton of effectveness about our clusterng algorthms, we choose four baselnes, all of whch are most famous and popular methods. Baselnes and our model are lsted as followed. 1) K-Means Algorthm, whch s the dstance based clusterng methods, and the number of clusters s fxed, we try many settngs for ths methods, and work out the almost best purty results, and ths method s mplemented by WEKA, notated as K-Means. 2) Densty-Based Algorthm, whch s the densty estmaton clusterng method, t can detect many shapes of clusters, but the number of clusters s fxed and ths method s mplemented by WEKA, notated as Densty-Based. We also try many settngs for ths methods, and work out the almost best purty results. 3) Herarchcal-Based Algorthm, whch s the herarchcal clusterng method, the number of clusters s fxed, and ths method s also mplemented by WEKA, notated as Herarchcal. Many settngs are tred for ths method to work out the almost best purty results. 4) EM Algorthm, whch must be provded wth the number of cluster, and ths method s also mplemented by WEKA, notated as EM. Ths method s based on probablstc prncple. Dfferent settngs and parameters are tred to work out the almost best purty. 5) Our Algorthm, whch s two layer clusterng feed-forward neural network. Cluster number of our model s fxed, so we try some dfferent settngs Fg. 2. The beam means the purtes of Glass for dfferent hdden node number whch s showed n x-axs. And The lne means the target functon values. The left y-axs value means target value and the rght y-axs means purty. Fg. 3. The beam means the purtes of Whte Wne Qualty for dfferent hdden node number whch s showed n x-axs. And The lne means the target functon values. The left y-axs value means target value and the rght y-axs means purty. to work out the almost best purty of t. and t s mplemented by ourselves, and ths model s notated as Ours. We apply a popular evaluaton method for clusterng wth classfcaton data to testfy our clusterng algorthms. The purty s showed n below formula, and the hgher the score s got, the better the system s. Purty = 1 N k max w k c (13) In above formula, the N s the number of nstances. the w k s the set of samples n k-th cluster and the c s the set of samples n -th real class. Table 1 shows the purty of dfferent methods for dfferent datasets. It clearly and obversely that our model s better than others. The effectveness about our model s verfed, and the detaled analyss s lsted as followed: 1) The reasons leadng to the expermental result s analysed n Secton V. That s to say, our theoretcal analyss s verfed. 2) Our method outperforms other popular clusterng methods. The effectveness of our method s verfed, C. Network Structure Studes In ths subsecton, we study the effect of the neural network structure. 1) Expermental Settngs: We choose the classcal three UCI datasets, whch are Glass, Whte Wne Qualty and Red Wne Qualty, wth dfferent hdden node number to study the structure effect. Noted that the results on other datasets are smlar n the aspect of trend. We also apply the purty that s defned above as our measurement. 2) Results And Dscusson: The results are shown n the Fg. 2, 3 and 4, where more x-axs value s, more dense the neural network s. From the trend of these fgures, we could conclude and explan some key ponts as followng: 1) The purty of clusterng has the same trend wth target values whle hdden node changes. We could choose the hdden node number based on ths pont. 2) Less hdden nodes often lead to unfttng problem whle more hdden nodes often lead to overfttng problem. We shall chose a sutable hdden node number to avod unfttng problem or overfttng problem. Fg. 4. The beam means the purtes of Red Wne Qualty for dfferent hdden node number whch s showed n x-axs. And The lne means the target functon values. The left y-axs value means target value and the rght y-axs means purty.

7 Our clusterng feed-forward neural network may be stuck n unfttng problem and overfttng problem. However, we could use the trend to mprove the clusterng effect. But n the whole vewpont, our method would also be robust enough to cluster the uncertan data. In concluson, our model s more flexble and more abstractng able than those common popular clusterng methods. Jontng the abstracton ablty of feed-forward neural networks and the probablty estmaton of Max-Entropy Prncple works well. [15] G. Hnton, S. Osndero, and Y.-W. Teh, A fast learnng algorthm for deep belef nets, Neural computaton, vol. 18, no. 7, pp , [16] Y. Bengo, P. Lambln, D. Popovc, and H. Larochelle, Greedy layer-wse tranng of deep networks, Advances n neural nformaton processng systems, vol. 19, p. 153, [17] X. Glorot and Y. Bengo, Understandng the dffculty of tranng deep feedforward neural networks, n Internatonal Conference on Artfcal Intellgence and Statstcs, pp , [18] H. Larochelle, Y. Bengo, J. Louradour, and P. Lambln, Explorng strateges for tranng deep neural networks, The Journal of Machne Learnng Research, vol. 10, pp. 1 40, VII. CONCLUSION In ths paper, we propose a method to cluster, ontng Entropy-Based Prncple wth Feed-Forward Neural Networks. We treat the Feed-Forward Clusterng Neural Network as two processes. In the abstracton process, Mn-Entropy Prncple s appled for more abstracted features and n the clusterng process, Max-Entropy Prncple s appled for dstrbuton estmaton of clusters n data manfold. We compare four knds of popular clusterng methods wth our model, and conclude that our model would perform better for flexble metrc representaton and varous shape assumpton. Experments are conduct on sx open UCI datasets and four baselnes are selected. Results show our model outperforms other common clusterng methods and sutable hdden node selecton would avod unfttng problem or overfttng problem to perform better. REFERENCES [1] H. Kano, Y. Tsubo, and H. Kashma, Clusterng crowds., n AAAI, [2] C.-L. Lee, Y.-N. Chang, C.-L. Lu, C.-Y. Lee, and J. Y.-. Hsu, Semantc clusterng of morphologcally related chnese words, [3] K. H. Moran, B. C. Wallace, and C. E. Brodley, Dscoverng better aaa keywords va clusterng wth communty-sourced constrants, n Twenty-Eghth AAAI Conference on Artfcal Intellgence, [4] R. Xu, D. Wunsch, et al., Survey of clusterng algorthms, Neural Networks, IEEE Transactons on, vol. 16, no. 3, pp , [5] C. C. Aggarwal and P. S. Yu, A survey of uncertan data algorthms and applcatons, Knowledge and Data Engneerng, IEEE Transactons on, vol. 21, no. 5, pp [6] S. Glpn, S. Nssen, and I. N. Davdson, Formalzng herarchcal clusterng as nteger lnear programmng., n AAAI, [7] S. T. Ma, X. He, J. Feng, and C. Böhm, Effcent anytme densty-based clusterng, SIAM, [8] V. Nguyen, D. Phung, X. Nguyen, S. Venkatesh, and H. H. Bu, Bayesan nonparametrc multlevel clusterng wth group-level contexts, arxv preprnt arxv: , [9] W. Zhang, S. J, and R. Zhang, Evolutonary soft co-clusterng, n Proceedngs of the 2013 SIAM Internatonal Conference on Data Mnng, pp , SIAM, [10] G. V. Steeg, A. Galstyan, F. Sha, and S. DeDeo, Demystfyng nformaton-theoretc clusterng, arxv preprnt arxv: , [11] Y. Bengo, Learnng deep archtectures for a, Foundatons and trends R n Machne Learnng, vol. 2, no. 1, pp , [12] R. Collobert and J. Weston, A unfed archtecture for natural language processng: Deep neural networks wth multtask learnng, n Proceedngs of the 25th nternatonal conference on Machne learnng, pp , ACM, [13] D. Cresan, U. Meer, and J. Schmdhuber, Mult-column deep neural networks for mage classfcaton, n Computer Vson and Pattern Recognton (CVPR), 2012 IEEE Conference on, pp , IEEE, [14] Y. He, K. Kavukcuoglu, Y. Wang, A. Szlam, and Y. Q, Unsupervsed feature learnng by deep sparse codng, arxv preprnt arxv: , 2013.