Vision Research 50 (2010) Contents lists available at ScienceDirect. Vision Research. journal homepage:

Transcription

1 Vision Research 5 (2) Contents lists availale at ScienceDirect Vision Research journal homeage: Biologically lausile saliency mechanisms imrove feedforward oject recognition Sunhyoung Han *, Nuno Vasconcelos Electrical and Comuter Engineering, University of California, San Diego, 95 Gilman Drive, La Jolla, CA , United States article info astract Article history: Received 6 Octoer 29 Received in revised form 3 Feruary 2 Keywords: Discriminant saliency HMAX Oject recognition Visual attention Biologically lausile recognition Neural models The iological lausiility of statistical inference and learning, tuned to the statistics of natural images, is investigated. It is shown that a rich family of statistical decision rules, confidence measures, and risk estimates, can e imlemented with the comutations attriuted to the standard neurohysiological model of V. In articular, different statistical quantities can e comuted through simle re-arrangement of lateral divisive connections, non-linearities, and ooling. It is then shown that a numer of roosals for the measurement of visual saliency can e imlemented in a iologically lausile manner, through such rearrangements. This enales the imlementation of iologically lausile feedforward oject recognition networks that include exlicit saliency models. The otential of comined attention and recognition is illustrated y relacing the first layer of the HMAX architecture with a saliency network. Various saliency measures are comared, to investigate whether () saliency can sustantially enefit visual recognition and (2) the enefits deend on the secific saliency mechanisms imlemented. Exerimental evaluation shows that saliency does indeed enhance recognition, ut the gains are not indeendent of the saliency mechanisms. Best results are otained with to-down mechanisms that equate saliency to classification confidence. Ó 2 Elsevier Ltd. All rights reserved.. Introduction * Corresonding author. Fax: address: shan@ucsd.edu (S. Han). The effectiveness and seed of iological solutions to the oject recognition rolem have long een a source of insiration for recognition algorithms. The introduction of the ack-roagation algorithm (Rumelhart, Smolenksy, Mcclelland, & Hinton, 986) estalished a framework for the automated design of recognition networks, and was highly successful for a numer of rolems. In articular, convolutional networks were shown to e highly cometitive with the est non-iological classifiers for tasks such as hand-written character recognition (Lecun, Bottou, Bengio, & Haffiner, 998). More recent results, y Thore, Fize, and Marlot (996), on the aility of human sujects to categorize natural scenes, showed that such tasks can e erformed with high accuracy (close to 94%) and very quickly (in less than 5 ms). The fact that such low recognition times leave no room for roagation of feedack across cortical areas, reinforced the significance of feedforward networks in visual recognition, at least in its early stages. It also surred a renewed interest in the family of feedforward architectures, of which the most recent oular element is the HMAX network of Riesenhuer and oggio (999) and Serre et al. (27). This network emulates the organization of the visual system as a cascade of layers of simle and comlex cells (Huel & Wiesel, 962), and has een recently shown to achieve state-ofthe-art erformance for a numer of recognition tasks (Mutch & Lowe, 28). There are, however, two imortant limitations of the HMAX model. First, ecause the organization of the network lacks a clear comutational justification, HMAX networks also lack a rinciled otimality criterion and training algorithm. This limits their relevance as an exlanation for the underlying iological comutations. Second, HMAX networks do not account for the sychohysical evidence on the imortant role layed y visual attention in to-down rocesses such as oject recognition (Yarus, 967). This limitation has een somewhat mitigated y research on recognition within multi-oject dislays, which comlements the HMAX network with serial attention mechanisms (Miau, aageorgiou, & Itti, 2; Walther & Koch, 26). In these methods, saliency is comuted with an indeendent ottom-u network, which () acts as a front-end to the HMAX network, selecting atches of the visual field to recognize (Miau et al., 2) or (2) modulates the connections of some HMAX units, serially directing attention to different roto-ojects in the field of view (Walther & Koch, 26). None of these works can account for the role of to-down attention in recognition, or the enefits of saliency in single oject dislays. These enefits have een documented in the comuter vision literature (Kadir & Brady, 2; Mikolajczyk & Schmid, 24; See & Lew, 23), ut with recourse to interest-oint detectors that are not iologically lausile. Within the HMAX literature, it has een shown that limiting the satial ooling erformed y some of the HMAX units can lead to non-trivial recognition /$ - see front matter Ó 2 Elsevier Ltd. All rights reserved. doi:.6/j.visres

2 2296 S. Han, N. Vasconcelos / Vision Research 5 (2) imrovements (Mutch & Lowe, 28). This, however, has een done in a somewhat ad-hoc form, y restricting the recetive fields of these units to a re-defined window size. To the est of our knowledge, no formal connection has een estalished etween HMAX itself and visual attention. In this work, we suggest a modification of the HMAX architecture that makes the connection etween recognition and visual saliency exlicit. We start y investigating the iological lausiility of statistical inference and learning tuned to the statistics of natural images. Building on rior work y Gao and Vasconcelos (29), we show that a rich family of statistical decision rules, confidence measures, and risk estimates, can e imlemented with the comutations attriuted to the standard neurohysiological model of V (Carandini, Heeger, & Movshon, 997; Carandini et al., 25; Heeger, 992; Huel & Wiesel, 962): a comination of linear filtering, divisive normalization, non-linearities, and satial ooling. In fact, it is shown that all these comutations have recise statistical meaning, contriuting to an overall roailistic interretation where simle cells comute osterior roailities and comlex cells estimate statistical risks. It follows that a numer of statistical oerators can e imlemented with iological hardware, through simle re-arrangement of lateral divisive connections, non-linearities, and ooling. We next estalish a connection to saliency mechanisms, y showing that various roosals for the measurement of visual saliency, from oth the iological and comuter vision literatures, can e imlemented with iologically lausile reconfigurations of the standard neurohysiological model. By relacing the first layer of the HMAX architecture with these saliency networks, we conduct a rigorous exerimental study of three questions at the intersection of attention and feedforward oject recognition: () whether saliency enefits visual recognition, (2) whether the gains deend on the tye of saliency considered (e.g. to-down vs. ottom-u) or even the secific saliency algorithms, and (3) whether max-ased ooling has an advantage over the classical linear oerator. We note that the goal is not to investigate whether saliency is eneficial as a means to serialize recognition when there are multile ojects within the field of view, as has een done in Miau et al. (2), Walther and Koch (26), or whether there are gains in comlementing recognition with an indeendent saliency ath. Instead, we consider the question of whether saliency is intrinsically imortant for recognition, even when there is a single oject in the field of view, as is suggested y comuter vision research. Or, in other words, whether in addition to its redominant role within the where athway, saliency also lays a role within the what athway of oject recognition. It is shown that the addition of saliency can significantly imrove recognition erformance, ut that this is not indeendent of the saliency rincile adoted. Best results are otained with to-down saliency mechanisms that equate saliency to classification confidence. jxj X ðx; a; Þ ¼ ð 2aCð=Þ e aþ where CðzÞ ¼ R e t t z dt, t > is the Gamma function, a a scale arameter, and a arameter that controls the shae of the distriution. The arameters a, can e learned in multile ways, including the method of moments (Huang & Mumford, 999), maximum likelihood (Do & Vetterli, 22), or Bayesian maximum a osteriori (MA) estimation (Gao & Vasconcelos, 29). We adot the latter, using a (Gamma distriuted) conjugate rior for the scale arameter a. Given a samle of training oservations D ¼fx ;...; x n g, this leads to Gao and Vasconcelos (29) ^a MA ¼ j X n j¼! jx j j þ m ; with j ¼ n þ g where g and m are rior hyer-arameters. The details of the rior are not crucially imortant, as its role is simly to regularize the feature resonses, so as to revent a null scale estimate. In our imlementation we use g = and m = 3. The MA estimate of the shae arameter is more comlex. However, for natural images this arameter tends to e fairly stale, usually taking values etween.5 and.8 (Srivastava, Lee, Simoncelli, & Zhu, 23). We have found =.5 to maximize the likelihood of a large samle of resonses of a set of Gaor filters to a random collection of natural images. This is illustrated in Fig., which shows the log roaility histogram of the Gaor resonses and the MA GGD fit for =.5. This value was used in all exeriments reorted in this work Statistical inference The iological lausiility of roailistic inference with GGD stimuli was studied in Gao and Vasconcelos (29). This work has shown that, for such stimuli, the fundamental comutations of roailistic inference and learning can e imlemented with the standard comutational model of simle and comlex cells (Carandini et al., 997, 25; Heeger, 992; Huel & Wiesel, 962). In what follows, we extend the rocedures introduced y Gao and Vasconcelos (29) to show that a much roader set of comutations, summarized in Tale, is iologically lausile. These comutations are descried in the second column of the tale. Although their iological imlementation turns out to e ossile with sutle modifications to the comutations of Gao and Vasconcelos (29), namely the introduction of various non-linear- 2 histogram fitted GGD ðþ ð2þ 2. Method We study the iological lausiility of statistical inference tuned to the statistics of natural images. We start y reviewing some known roerties of these statistics, then consider statistical inference, and finally the learning rolem Natural image statistics Various authors have shown that the emirical distriution of the resonse X of a and-ass filter to a wide variety of natural imagery is accurately modeled y the generalized Gaussian distriution (GGD) (Buccigrossi & Simoncelli, 999; Do & Vetterli, 22; Huang & Mumford, 999). This distriution is defined as Fig.. Histogram of resonses of a set of Gaor filters to a collection of natural images, and its MA fit y the GGD model with =.5.

3 S. Han, N. Vasconcelos / Vision Research 5 (2) Tale Oerations of statistical inference under the GGD model. w(x) is defined as wðxþ ¼ 2 log x x Oeration Definition Under GGD statistics Notes Single oservation x inference Neg. log-likelihood (NLL) log X (x) l a ðxþ ¼ jxj a þ K K ¼ log 2aCð=Þ Log-likelihood ratio (LLR) log XjY ðxjþ XjY ðxjþ gðxþ ¼l a ðxþ l ðxþ a Target osterior (T) YjX (jx) r[g(x)] r(x) = ( + e x ) Information I(Y; X = x) n{r[g(x)]} nðxþ ¼log 2 þ x log x þð xþlogð xþ Measures of detection confidence LLRC(x) wfyjx ~ ðjxþg wfr½gðxþšg ~ wðxþ ~ wðxþ; x :5 ¼ ; x <:5 IC(x) ~ nfyjx ðjxþg ~ nfr½gðxþšg ~ nðxþ; x :5 nðxþ ¼ ; x <:5 Emirical risks ased on samle R ¼fx ;...; x ng Exected NLL E X [ log X (x)] n n i¼a;ðx i Þ H[X] h i Exected LLR E X log XjY ðxjþ n XjY ðxjþ n i¼ gðx iþ KL[ X (x)k XjY (xj)] KL[ X (x)k XjY (xj)] MI E X [I(Y; X = x)] n n i¼ nfr½gðx iþšg I(Y; X) Exected confidence (LLR) E X [LLR(x)] n ~ n i¼ wfr½gðx i ÞŠg KL[ Xjy (xj)k XjY (xj)] Exected confidence (MI) E X [IC(x)] n ~ n i¼ nfr½gðx i ÞŠg ities, this extension sustantially roadens the scoe of the underlying comutational framework. For examle, the oerations now considered are critical to the design of networks that address to-down rolems such as oject recognition. In fact, as will e shown in Section 3.3, the erformance of such to-down networks can e quite sensitive to the recise choice of statistical inference rincile, and associated non-linearities. Tale is organized in three sections. The first reorts to inference from a single oservation x. It starts with the most atomic comutation of statistical inference: the evaluation of the log roaility log X (x) of an oservation x. A ercetual system can use this roaility to make otimal decisions regarding the classification of x with resect to a target and a null hyothesis. These are identified y a class lael Y that takes the values Y = for the target and Y = for the null hyothesis. Otimal decision-making is frequently defined in the minimum roaility of error (ME) sense, under which the otimal rocedure is the Bayes decision rule (Duda, Hart, & Stork, 2). This consists of thresholding the log-likelihood ratio (LLR) log XjYðxjÞ XjY ðxjþ and selecting the target hyothesis whenever this ratio is aove threshold. An equivalent imlementation of this decision rule is to choose the target hyothesis when the osterior target roaility, YjX (jx), is aove /2. The rocess is illustrated in Fig. 2, for an oject recognition rolem where the target is the class of airlanes. Given a set of examle images from this class, and a set of examles from the null hyothesis (in this case any oject other than a lane), the visual system relies on a set of andass (e.g. Gaor) filters to extract visual features characteristic of the two classes. The GGDs that est fit the distriutions, XjY (xji),i 2 {,}, of filter resonses under the two hyotheses are then estimated. Given a new image, the corresonding features are extracted, and the LLR of (3) is comuted, using these GGDs. Thresholding this quantity then roduces a inary ma that indicates the locations of the target within the visual field. The LLR is one of various quantities that lay an imortant role in statistical inference and otimal decision making. Tale includes a numer of others, which we review in more detail in the remainder of this section. A grahical illustration of these measures, in the context of oject recognition, is resented in Figs. 4 and 5. An alternative otimality criteria for decision making, commonly referred as infomax (Linsker, 988), is to maximize the ð3þ information aout the class lael Y. This criterion underlies many classification rocedures roosed in the machine learning literature, including logistic regression and some forms of oosting (logitboost) (Friedman, Hastie, & Tishirani, 2; Hastie, Tishirani, & Friedman, 2). Its maximization has also een roosed as a fundamental rincile for the organization of ercetual systems (Barlow, 2; Linsker, 988). In this case, inference is ased on the information IðY; X ¼ xþ ¼ X i YjX ðijxþ log X;Yðx; iþ X ðxþ Y ðiþ that the oservation x rovides aout the class Y. The second section of Tale refers to the evaluation of confidence measures. These comlement the decision that x elongs to the target class (target detection), y quantifying how confident the classifier is aout this decision. Oviously, the confidence measure should e derived from the rincile used for inference. This leads to two confidence measures ased on the likelihood ratio ( 2 LLRCðxÞ ¼ log YjX ðjxþ ; YjX ðjxþ YjXðjxÞ :5 ð5þ YjX ðjxþ <:5; and the information measure ICðxÞ ¼ IðY; X ¼ xþ; YjXðjxÞ :5 YjX ðjxþ <:5: An imortant roerty is that, in oth cases, the confidence measure is one-sided, i.e. non-zero only if x is classified as a target. Although undesirale for the ottom-u rolems considered in Gao and Vasconcelos (29), we will see that this roerty ecomes quite imortant for success in to-down rolems, such as recognition. The two measures can e exressed as a transformation of the osterior target roaility YjX (jx). As indicated in Tale, these transformations are ~wðxþ ¼ ( log x 2 ; x :5 x ; x <:5 for LLRC(x) and log 2 þ x log x þð xþlogð xþ; x :5 ~nðxþ ¼ ; x <:5 for IC(x). They are shown in Fig. 3. ð4þ ð6þ ð7þ ð8þ

4 2298 S. Han, N. Vasconcelos / Vision Research 5 (2) Learning e.g. the identification of the most discriminant Gaor filters for a articular detection rolem or (2) the determination of the entrotarget (Y=) non target (Y=) filtering GGD estimation X Y (x ) X Y (x ) Recognition LLR Target? x X Y (x ) > X Y (x ) Fig. 2. Oject recognition with the LLR measure. The learning stage is shown at the to of the figure. Gaor filtering is alied to examles of the target and null class. In this examle, the target is the class of airlane ojects. The roaility distriutions of the filter resonses are then modelled with the GGD distriution. This enales the detection of ojects from the target class in reviously unseen images, as shown at the ottom. Given the filter resonses to an unseen image, and the GGD estimates learned during training, the LLR is comuted at each location of the visual field. Simle thresholding of this measure roduces a inary ma indicating the region of the vision field covered y the oject Fig. 3. Non-linear transformations of the osterior target roaility that roduce the information I(Y; X = x) (left) and the confidence measures LLRC(x) (center) and IC(x) (right). The third section of Tale addresses the characterization of the random variale X. This enales tasks like () feature selection,

5 S. Han, N. Vasconcelos / Vision Research 5 (2) y of X, e.g. to evaluate the uncertainty of the feature resonses. This characterization usually requires the comutation of emirical averages of the statistical inference oerators discussed aove, from a samle of oservations R ¼fx ;...; x n g. Such averages are emirical estimates of oular statistical risks, which are referenced in the right-most column. These include the entroy Z H½XŠ ¼ X ðxþ log X ðxþdx ¼ E X ½ log X ðxþš ð9þ the mutual information IðY; XÞ ¼ X Z X;Y ðx; iþ log X;Yðx; iþ i X ðxþ Y ðiþ dx ¼ E X½IðY; X ¼ xþš or the Kullack-Leiler (KL) divergence Z KL½ X ðxþkq X ðxþš ¼ X ðxþ log XðxÞ Q X ðxþ dx ¼ E X log XðxÞ Q X ðxþ ðþ ðþ Once again, each inference rincile leads to a different risk. For examle, the exected LLR is a difference of two KL divergences KL½ X ðxþk XjY ðxjþš KL½ X ðxþk XjY ðxjþš ð2þ while the exected value of the information measure I(Y; X = x) is the mutual information I(Y; X) etween the oservation X and the class lael Y. Finally, it is also ossile to rely on exectations of the confidence measures of (5) and (6). These can e seen as onesided versions of the KL difference and mutual information, which only average samle oints identified as elonging to the target class (y the Bayes decision rule). Such averaging is equivalent to comuting exectations with resect to the target class conditional distriution YjX (xj), rather than X (x). It, for examle, simlifies the KL difference of (2) into the more standard KL divergence KL[ XjY (xj)k XjY (xj)]. Again, their one-sided nature makes these risks articularly effective for to-down rolems, such as target detection or recognition. All risks ased on KL divergences or mutual informations measure the discriminant ower of X for target detection, and can e used for feature selection. When X ={X,...,X k } is a set of andass features, the deendencies of the feature resonses to natural images tend to carry little information aout the class lael (Vasconcelos & Vasconcelos, 29). This can e exloited to simlify the joint mutual information of the features with the class lael into IðX; YÞ X k IðX k ; YÞ ð3þ and justifies the comutation of the overall discriminant ower of X y adding the discrimination measures derived from each feature channel. We use this rocedure to integrate the emirical risks of Tale across feature channels Inference under the GGD When X follows a GGD, the comutations aove can e simlified into the form shown in the third column of Tale. Here, all equations assume that X is either a GGD random variale of arameters (a, ), or a GGD random variale when conditioned on the class Y. In this case, the class conditionals XjY (xji) have arameters (a i, i ), i 2 {, }. It is also assumed that Y () = Y () = /2, ut this could e generalized into any lael distriution. As noted y Gao and Vasconcelos (29), the form of the negative log-likelihood l aðxþ ¼jxj a þ K ð4þ is a straightforward consequence of (). It follows that large values of jxj indicate the locations of visual stimuli of low roaility within the field of view. This is illustrated in Figs. 4 and 5a c, which resent two images, the magnitude jxj of their convolution with a Gaor filter, and the NLL l aðxþ for the MA GGD fit with =.5. Note that the latter emhasizes details of the oject or ackground which have very distinctive aearance from the rest of the image. In this sense, the log-likelihood oerator ehaves as an interest oint oerator, similar to a numer of interest oint oerators currently oular in comuter vision (& Stehens, 988; Kadir & Brady, 2; Mikolajczyk & Schmid, 24; See & Lew, 23). By definition, the LLR is a difference of two negative log likelihoods. It can e written as gðxþ ¼log XjYðxjÞ XjY ðxjþ ¼ jxj jxj þ T; ð5þ a a where T ¼ log a a. Figs. 4d and 5d show the LLR for motorike detection on the images of (a). In oth cases, a was learned from a collection of ike images (target hyothesis), and a from a random collection of natural images (null hyothesis). The LLR emhasizes the region of the motorike, which is aroximately uniformly highlighted, and inhiits the ackground. Simle alication of Bayes rule leads to the well known relation YjX ðjxþ ¼ ¼ r½gðxþš þ XjY ðxjþ XjY ðxjþ where r(x) = ( + e x ) is the sigmoid function. Hence, the target osterior is a sigmoidal transformation of the LLR. Similarly, (4) can e written as I(Y;X = x)=n[ YjX (jx)], with the non-linearity nðxþ ¼log 2 þ x log x þð xþ logð xþ shown in Fig. 3. The alication of these non-linearities to the images of Figs. 4d and 5d are shown in Figs. 4 and 5(e f). They rema the LLR into the range [ ]. While Gao and Vasconcelos (29) have comined r(x) and n(x) into a single non-linearity, there are non-trivial enefits in decouling the two comonents. Note, in articular, that while the sigmoidal transformation maintains the emhasis on the ike region, the non-linearity associated with the information measure re-emhasizes some of the ackground. This is due to the fact that the latter is insensitive to the sign of the LLR (or, equivalently, to the sign of YjX (jx) /2). In a strict information theoretic sense, the asence of an oject is as informative as its resence for oject detection (the classifier is simly very confident in the assignment of the image ixels to the ackground class). This is, however, undesirale for oject detection, where the role is to detect oject, and not ackground. When the two non-linearities are decouled, this rolem can e corrected y resorting to the measures of classification confidence of (5) and (6), which can e comuted y comosition of the sigmoid with the non-linearities of (7) and (8). The result, shown in Figs. 4 and 5(g h) is a strong suression of regions that elong to the ackground. This suression enales very non-trivial gains in recognition accuracy, as will e shown in Section 3.3. Finally, all emirical risks can e comuted y averaging some comination of these non-linearities. In summary, as noted in Tale, most oerations of statistical inference with GGD stimuli are non-linear maings of the LLR g(x) of(5) Biological lausiility Gao and Vasconcelos (29) have shown that, given a samle R from a GGD distriution and using the estimate of (2) in (4), l jxj aðxþ ¼j j jx jj þ m þ K ð6þ

6 23 S. Han, N. Vasconcelos / Vision Research 5 (2) (a) () (c) (d) (e) (f) (g) (h) Fig. 4. (a) An image, () magnitude of Gaor resonses, (c) NLL, (d) LLR, (e) target osterior roaility, (f) information I(Y; X = x), (g) LLRC(x), and (h) IC(x). The ars on the side of each image show the range of values corresonding to the ixel amlitudes. The asolute value of x can e comuted y half-wave rectification, i.e. as jxj = x + + x where x + = max(x, ) and x = max( x, ). This leads to the sequence of comutations attriuted to simle cells y the standard neurohysiological model of V (Carandini et al., 997, 25; Heeger, 992; Huel & Wiesel, 962): linear filtering to roduce a filter resonse x, half-wave rectification, and divisive normalization y the resonses of other cells. For simlicity, we omit the decomosition into the rectified comonents (x +, x ) from all equations and network diagrams, working with jxj instead. The comination of asolute value and divisive normalization as in (6) has recently een found to sustantially imrove the recognition accuracy of classical convolutional networks (Jarrett, Kavukcuoglu, Ranzato, & LeCun, 29; into, Cox, & DiCarlo, 28; into, Doukhan, DiCarlo, & Cox, 29). However, no rinciled justification has een given for the imortance of these oerations. The discussion aove suggests that this imortance follows from their interretation as estimators of the fundamental quantity of statistical inference (log roaility). The network reresentation of the simle cell is shown in Fig. 6. Since the LLR is the difference of two log-roailities, given two samles R and R from the null and target class, resectively, it follows that gðxþ ¼ j jxj x j 2R jx j j þ m jxj j x j 2R jx j j þ m þ T ð7þ This leads to the iologically lausile imlementation of the LLR with the network of Fig. 7. The main difference with resect to the network of Fig. 6 is that the filter resonses are now differentially normalized y the units in the two dashed oxes. These oxes ool the resonse of other cells in a region T where the training samle R is collected. The ottom (to) units collect ositive (negative) examles, roducing an estimate of the GGD scale for the target class (null hyothesis). The region T localizes the cell comutations. If T is the entire field of view, the GGD models are average distriutions for the feature resonses across the latter. For smaller T, the cell resonse is tuned to the statistics of a su-region of the field of view. Hence, the LLR can e comuted y a differentially normalized simle cell. This romted (Gao & Vasconcelos, 29) to roose the LLR network as a model for simle cells. There are, however, two significant advantages in further including a sigmoidal non-linearity at the network outut, as is now roosed in Fig. 7. First, this turns the cell into an estimator of the osterior target roaility YjX (jx), a more central quantity to the comutations of Bayesian

7 S. Han, N. Vasconcelos / Vision Research 5 (2) (a) () (c) (d) (e) (f) (g) (h) Fig. 5. (a) An image, () magnitude of Gaor resonses, (c) NLL, (d) LLR, (e) target osterior roaility, (f) information I(Y; X = x), (g) LLRC(x), and (h) IC(x). Fig. 6. The NLL is comuted y a simle cell that normalizes a feature resonse x y the resonses of its satially neighoring units. decision theory than the LLR. Second, it strengthens the iological lausiility of the simle cell model, y accounting for the saturation effects that are well known to hold for simle cell oututs, ut are not relicated y the LLR. Most risk estimates in the lower third of Tale consist of ooling some non-linear transformation of the osteriors r[g(x i )], within some region R of the field of view. This makes the associated comutations good candidates for comlex cells. An examle is the MI, for which the ooling oeration is reresented in Fig. 8. This network ools the resonses of its afferent simle cells, after assing them through the non-linearity n(). As shown in Fig. 3, this Fig. 7. A LLR unit divisively normalizes a feature resonse x differentially, using the oututs of two units that estimate GGD arameters under the target and null hyothesis. With the inclusion of the outut non-linearity r(), this unit comutes osterior target roailities. non-linearity is very close to quadratic, making the network a very good aroximation of the standard energy model of comlex cells y Adelson and Bergen (985). The remaining emirical exectations of Tale can e imlemented y relacing n() with the non-linearities ~ wðþ or ~ nðþ, also shown in Fig. 3. The only excetion

8 232 S. Han, N. Vasconcelos / Vision Research 5 (2) is the entroy network, which does not rely on the LLR g(x). In this case, the comlex cell ools the resonse of the NLL units in R Saliency R C S Fig. 8. A comlex unit ools the resonses of simle units within some region R, after assing them through a non-linearity. A numer of roosals for the measurement of visual saliency can e imlemented y the networks of Tale. We consider two ottom-u saliency methods, ased on the detection of rare features, and a to-down aroach, discriminant saliency, which accounts for the classes of the ojects to detect Detection of rare features A numer of authors have advocated the detection of features of low roaility as a criterion for visual saliency (Bruce & Tsotsos, 26; Rosenholtz, 999; Zhang et al., 28). As discussed aove, this criterion can e imlemented with the NLL unit of Fig. 6. The detection of low roaility features is also closely related to the most oular strategy for the detection of interest oints in comuter vision. A numer of detectors from this literature identify image structure such as corners (Harris & Stehens, 988), locations of strong image derivatives (Mikolajczyk & Schmid, 24), wavelet coefficients of large magnitude (See & Lew, 23), or local maxima of image entroy (Kadir & Brady, 2) that have low roaility of occurrence. The features that elicit a strong resonse y NLL units generalize all these tyes of structure. For examle (see Tale ), the comination of NLL units with a comlex cell that ools its afferents linearly measures the entroy of the underlying feature resonses. It should e noted, however, that NLL units are technically not feature detectors, since they only comute the likelihood of feature resonses. One ossiility to transform them into detectors is to consider a discriminant version, that tests two hyotheses. Under the null hyothesis, x follows a GGD distriution X (x) of arameters (a, ) estimated from the visual field. Under the alternative hyothesis, x follows a non-informative distriution X (x) /. The likelihood ratio is g(x)/ log X (x) and the osterior YjX (jx)=r( log X (x)) = r(jxj/a + K). The null hyothesis is rejected when jxj a is large, i.e. large resonses are etter exlained y the non-informative distriution. This imlies that such resonses are rare within the field of view. From an imlementation oint of view, the discriminant unit is identical to the NLL of Fig. 6, with the addition of an outut sigmoid. We denote this comination as a rare feature detector (RFD) Discriminant saliency Discriminant saliency is defined with resect to a target and a null hyothesis. In the oject detection context, the target is the class of ojects to detect while the null hyotheses encomasses all stimuli outside that class. Locations of the visual field that can e assigned to the target class with minimal roaility of error (.) (.) (.) are declared salient, with degree of saliency equal to the classification confidence (Gao & Vasconcelos, 29; Mahadevan, & Vasconcelos, 27; Gao, Han, & Vasconcelos, 29) SðxÞ ¼ IðY; X ¼ xþ if YjXðjxÞ >:5 ; otherwise; ð8þ This is the IC(x) measure of Tale. If multile resonses {x,...,x K } from feature X are availale, the saliency of X is defined as IðX; YÞ ¼ K i Sðx iþ, i.e. the exected confidence (MI) measure of the tale. Saliency measurements derived from multile feature channels are comined with (3). The last third of Tale suggests a numer of other discriminant ossiilities for measuring feature saliency: KL difference, mutual information I(X;Y), or KL divergence. These measures differ from the exected confidence (MI), adoted y discriminant saliency, in relatively small details (mostly non-linearities). Such details could nevertheless e of consequence. For examle, Jarrett et al. (29) has found that simly taking the asolute value of the outut of each unit of a classical convolutional network can roduce drastic imrovements in its recognition accuracy. The discussion aove shows that these details can also comletely alter the semantics of the network comutations. For examle, unlike the exected confidence (MI), the MI does not emhasize feature resence and could identify as salient a feature that is always asent from the target class. This is desirale for ottom-u saliency (Gao & Vasconcelos, 29) ut not necessarily for to-down alications, such as oject detection or localization. We evaluate the erformance of these measures in the following section, where it is shown that the choice of non-linearities can indeed have a significant imact on recognition erformance. 3. Results HMAX networks emulate the organization of the visual system y a cascade of two layers of simle and comlex cells. We investigated the role of saliency in recognition y relacing the first HMAX layer with a saliency network. Under HMAX, this layer is quite simle: simle units erform filtering, and comlex units ool simle unit resonses within a satial neighorhood, using a maximum oerator. While these simle units have no roailistic interretation, max-ased comlex units are an interesting alternative to the samle averages of Tale. They act more like a feature selection mechanism: rather than averaging resonses, max-ased ooling identifies the location of most salient resonse. This aears natural for detection-ased saliency measures, e.g. the RFD. By relacing the first HMAX layer with a saliency network we can thus investigate three questions:. Is saliency imortant for visual recognition? 2. How do the various saliency criteria comare on an ojective task, such as oject recognition? 3. Is there an advantage in using max vs. the classical linear ooling? In the roader neural network literature, there have een recent showings that some details of the network comutations, e.g. what tye of non-linearities or normalization is erformed, can have a sustantial imact in recognition accuracy (Jarrett et al., 29; into et al., 29). As discussed aove, the statistical interretation of these oerations makes it ossile to assign semantics to all comutations, with resect to otimality rinciles for discrimination, statistical inference, measurement of information, etc. This enales a more efficient search for otimal comutations than trial-and-error (Jarrett et al., 29), or rute-force otimization (into et al.,

9 S. Han, N. Vasconcelos / Vision Research 5 (2) ). To study these questions we erformed a numer of exeriments, which are discussed in the remainder of this section. 3.. Exeriments We start with a simle synthetic rolem that rovides intuition on the enefits of to-down discriminant saliency for recognition, and then resent more extensive exeriments on the Caltech enchmark, commonly used to evaluate oject recognition erformance. All exeriments were ased on the HMAX network, whose first layer was relaced y a saliency network. On Caltech we tested all saliency measures in the lower third of Tale, as well as RFD, and the saliency detector of Itti, Koch, and Nieur (998). For comleteness, we also evaluated the use of a classical sigmoidal layer (no comlex units or ooling, simle units a comination of filtering and a sigmoid) in the first HMAX layer, and the HMAX network itself. To investigate the advantages of max over linear ooling, all saliency networks were imlemented with oth. On the synthetic exeriment we comared an HMAX network, HMAX with first layer relaced y a ottom-u saliency network of RFD units (HMAX + RFD), and HMAX with first layer relaced y a to-down saliency network of exected confidence(llr) units (HMAX + EC). In all exeriments, for saliency units that involve divisive normalization, the ooling region T of the normalizing units was the whole image. In the case of ottom-u saliency (NLL or RFD units) the normalization is erformed on-line, i.e. dividing y neighoring resonses to the image to recognize. For to-down saliency (LLR units) the normalizing coefficients are learned during training, when the network is exosed to images from the target and null hyotheses. For comlex units, the ooling region R was as secified in Mutch and Lowe (28). The second layer of the HMAX network consists of a set of radial asis function (RBF) units, centered at rototyes randomly samled from the resonses of the first HMAX layer, during training. On Caltech we used the imlementation of Serre, Wolf, Bileschi, Riesenhuer, and oggio (27), which includes 475 RBF units. On the synthetic exeriment we used a smaller network of units. For LLR units, training roduces two divisive normaliza- tion arameters a i er oject class. For a given RBF rototye, the arameters of the afferent simle units are set to the values a i ðþ with which was learned (i.e. the arameters learned from the image class which originated ). Other than these modifications, the network is exactly as descried in Mutch and Lowe (28) Synthetic rolem To gain some insight on the role of discriminant saliency in recognition, we considered the simle rolem of learning to differentiate underlined from non-underlined characters. This was formulated as a two-class recognition rolem, involving the stimuli of Fig. 9. Each network was trained with the to two images of the figure, using underlined Xs as examles from the target class, and regular Xs as examle non-targets. This made the classes identical u to a salient feature of the underlining concet (the underline ar). The network was then used to classify 2 test images, containing either targets or non-targets. To increase the difficulty of the task, the character used on the test images (Y) was different from that used for training (X), and random noise was added to all images. The recognition accuracy achieved y the three networks was 9% for HMAX + EC, 55% for HMAX+RFD, and 5% for HMAX. The suerior erformance of the network with to-down saliency can e understood y analyzing the intermediate network resonses, shown in Figs. 9 and. Consider the resonse of the first network layer, shown in Fig. 9. The HMAX network only has access to Gaor filter resonses, which are very similar for target and non-target. This makes it very difficult for the susequent HMAX stages to distinguish etween the two classes. Because none of the arts of the underlined Xs o-out within the target dislays, the saliency resonse of RFD is asically a contrast enhanced version of the filter resonses. This does not imrove the recognition accuracy sustantially, since contrast variaility is not the reason for the oor erformance of HMAX on this classification rolem (although it can e a source of concern for rolems involving natural images where, as we will see in the next section, HMAX + RFD tends to Target Non target Test Filter resonse (HMAX) RFD LLR Non target Target Fig. 9. Detection of underlined characters. To row: Training examles from target and non-target class. Bottom rows: Examles of test stimuli from the target and non-target class, and layer resonses from the three networks considered.

10 234 S. Han, N. Vasconcelos / Vision Research 5 (2) atch for S2 layer Outut of S2 layer HMAX LLR Target Non Target Non Fig.. To: Most discriminant filter (the four orientation channels are shown) of the second network layer, for HMAX (left) and HMAX + EC (right). By most discriminant it is meant that this is the filter given larger weight y the linear SVM classifier at the network outut. Bottom: Examle outut of the simle cells in layer 2, to target and non-target stimuli. outerform HMAX). Hence, the erformance of HMAX and HMAX + RFD is asically identical. The underline ar is, however, salient in the to-down sense, since it is the only art that distinguishes the target and non-target examles. Because the units of the HMAX + EC network comute the LLR etween target and non-target hyothesis, they roduce a strong resonse to underline ars (lausile under target, ut not lausile under the non-target hyothesis) and a weak resonse to everything else (equally lausile, or non-lausile, under the two hyotheses). The network has thus learned that horizontal ars are discriminant features for the detection of underlined characters, and thus salient. Its first layer acts as a detector of these ars, and its very different resonses to targets and non-targets are easily detected y the susequent network stages. Fig. resents the most discriminant filter of the second layer (four orientation channels shown), for the HMAX and HMAX + EC networks. Note how the filter of HMAX + EC is a detector of horizontal ars, a roerty that does not hold for the other networks. In result, the outut of the second layer of HMAX + EC is uniformly large for underlined carachters, and almost null for non-targets. This is unlike the other two networks, whose second layers resond to oth targets and non-targets. It is thus not surrising that HMAX + EC achieves a sustantially higher recognition accuracy Caltech exeriments To evaluate the imact of the various saliency rinciles on the classification of natural images, we erformed a numer of exeriments on Caltech. All exeriments were ased on the exerimental rotocol of Mutch and Lowe (28). We considered the multiclass recognition task, where 3 images er class are used for training and a maximum 5 of the remaining for test. In all exeriments the reorted recognition rate is the average over five indeendent runs, with different train and test sets (randomly samled images). Tale 2 resents the recognition accuracy achieved with each variant of the first network layer. A grahic dislay of these rates, as well as the associated error ars, is shown in Fig.. A few interesting oservations can e made. First, the two exected confidence criteria achieve the est results. Their erformance is similar, ut EC(LLR) attains slightly higher recognition rates. These methods can e imlemented with simle units that comute the target osterior roaility, i.e. a comination of a differentially and divisively normalized (LLR) unit and a sigmoid r(). The gains with resect to the remaining networks can e very significant. Second, saliency criteria ased on rare features (ENLL and RFD) erform worse than saliency criteria ased on discrimination (the exected confidence measures). On the other hand, oth rare feature criteria have clearly etter erformance than sigmoid or HMAX. This suggests that rare feature (interest oint) detection can e useful when statistics of the target oject class are not availale. Note that, under the rare feature criteria, none of the two network layers requires class-secific training. While the same holds for the saliency detector of Itti et al. (998), its erformance (5.8%) is sustantially weaker than those of ENLL or RFD. Third, the one-sided confidence measures EC(LLR) and EC(MI) erform sustantially etter than their two sided counterarts, such as the ELLR or the MI used in Gao and Vasconcelos (29). This imlies that the choice of non-linearities (e.g. ~ n instead of n or ~ w instead of w) can have a very non-trivial imact in recognition accuracy. It aears to e articularly imortant for the cells to fire only when the target is resent. Fourth, for most networks, maxased ooling has inferior erformance to averaging. This imlies that it is imortant to fully characterize features, and not only select locations where they are informative for the classification. The only Tale 2 Recognition rates on Caltech, using 3 training examles er class. All areviations are the same as in Tale. Furthermore, EC means exected confidence, ELLR exected LLR, ENLL exected NLL, RFD rare feature detection. Network Simle units Comlex units Divisive normalization Non-linearity ooling NLL LLR r() n() w() ~ nðþ wðþ ~ Sum Max Accuracy EC (LLR) EC (MI) ELLR MI ENLL RFD Itti et al. (998) 5.8 Sigmoid 42 HMAX

11 S. Han, N. Vasconcelos / Vision Research 5 (2) EC(LLR) EC(MI) ELLR MI ENLL RFD Sum Max HMAX +sigmoid HMAX Itti Model sigmoid Fig.. Recognition rates on Caltech, using 3 training examles er class. network for which max ooling consistently achieves etter erformance is HMAX (where the lack of sohistication of the simle units makes the network with average ooling linear). Furthermore, maxased ooling is rone to large erformance variaility. For examle, the EC(MI) network dros from 6% to 53% recognition rate when averaging is relaced y max ooling. Finally, the classical sigmoid layer has the worst erformance of all considered. However, the simle addition of a ooling stage can imrove erformance consideraly, esecially when comined with max ooling Comarison to state-of-the-art results To the est of our knowledge, the current state-of-the art results for oject recognition with HMAX networks are those resented in Mutch and Lowe (28). This work reorted significant imrovements over the ase HMAX erformance, through a numer of enhancements to the original network. Some of these involved additional training, e.g. to select features, others are heuristics that were shown to imrove erformance. Tale 3 resents the contriutions y these enhancements, as reorted in Mutch and Lowe (28). As can e seen from the tale, the simle use of the saliency layer, without any further otimization, outerforms the gains of all enhancements of Mutch and Lowe (28). One of these imrovements is a feature selection stage. Rather than Tale 3 Multiclass classification results for categories. Model 5/cat 3/cat Base model of Serre et al. (27) sarse S2 inuts Mutch and Lowe (28) inhiited S/C oututs Mutch and Lowe (28) limited C2 invariance Mutch and Lowe (28) feature selection Mutch and Lowe (28) 5 56 EC (LLR) with sum ooling feature selection descried in Mutch and Lowe (28) Convolutional net of into et al. (28) 42 + second HMAX layer 56 Convolutional net of Jarrett et al. (29) 56 + random filters 63 + unsuervised filters 64 + ack-roagation filters 66 Lazenik et al. (26) Zhang et al. (26) using 475 randomly samled rototyes, a larger set of 2, are collected. The network is trained with this larger set, and a suort vector machine is used to select the most discriminant 475. When we retrained the network containing the saliency layer in this manner, the erformance increased to 64%, as oosed to the 56% reorted y Mutch and Lowe. While we have not yet exerimented with any of their other suggestions, or erformed any other otimization, these results suggest that the inclusion of saliency can significantly oost the erformance of feedforward oject recognition. In the roader area of convolutional networks, recent studies have addressed the role of non-linearities and normalization in oject recognition (Jarrett et al., 29; into et al., 28). These works advocate the use of divisive normalization as a form of contrast normalization, that imroves the roustness of the neural network when trained from small samles, as is the case of Caltech (Jarrett et al., 29). This is a strictly ottom-u exlanation for the role of divisive normalization, and comarale to the ENLL and RFD saliency measures discussed in this work. Comarison with these methods should e erformed with care, since the network arameters are not the same. For examle, while it has ecome somewhat oular to claim that method of into et al. (28) eats the state-of-the-art in comuter vision, the truth is that its imlementation is far from the standard in this area. For examle, while (for comutational efficiency) most comuter vision imlementations rely on a relatively small set of filters (e.g. Gaor filters at four orientations) and a relatively small numer of network oututs (475 for the first HMAX network (Serre et al., 27), 2, for enhanced HMAX (Mutch & Lowe, 28)), this method relies on a much larger filter set (2 orientations), and a much larger outut dimensionality (86,4 6,4). The network has a single layer and is comlemented y a classifier that comines a rincial comonents analysis of very disutale iological lausiility, and an SVM. While the recognition accuracy originally reorted y the authors is of 65% (3 images er category), our imlementation with () the Gaor filter front-end and (2) the outut dimensionality used y the HMAX networks only achieved 42%. Further inclusion of the second HMAX layer raised recognition erformance to 56%. We note that this is consistent with the results of Tale 2, as the network of into et al. (28) is similar to the RFD network. Hence, it is not surrising that the results are in etween those of ENLL (58.2%) and RFD (55.%). Similar erformance was documented y Jarrett et al. (29), who have otained 55.8% accuracy with a two layer network

12 236 S. Han, N. Vasconcelos / Vision Research 5 (2) including divisive normalization in the two layers (as oosed the one we tested, where only the first layer was modified). This work has tested a numer of extensions, including the use of filters learned from the training data, in oth a ottom-u and to-down manner. All results reorted are lower than those achieved with the EC(LLR) network, excet when the filters are trained in a discriminant manner. Note that, in this case, the convolutional network has two layers of trained filters and divisive normalization, network training is orders of magnitude more comlex than that required y the saliency network (ack-roagation for the former vs. the individual tuning of the divisive normalization weights of each simle cell, according to (2), for the latter), and the gains are very marginal (65.5% vs. 64%). The filters of the EC(LLR) network could also have een trained in a discriminant manner, ut we have not attemted to erform this otimization. For comleteness, we also reort the state-of-the-art results on Caltech from the roader recognition literature in comuter vision, where iological lausiility is not a constraint. We consider here only methods that use a single image reresentation, and are therefore comarale to the networks roosed aove. In this class, the est erformance in the literature is in the range of 65 66% (Lazenik, Schmid, & once, 26; Zhang, Berg, Maire, & Malik, 26) and arely suerior to the 64% now reorted for the iologically lausile networks. Oviously, etter erformance should e attainale y comining multile image reresentations, e.g. y adding features that cature color or shae roerties to the set of Gaor functions that we consider in this work. This is indeed a oular strategy in the comuter vision literature, where it has een shown that sustantial imrovements over (Lazenik et al., 26; Zhang et al., 26) can e achieved with suort vector machines comining multile kernels (Gehler & Nowozin, 29; Varma & Ray, 27). Such cominations of multile image reresentations could also e alied to the networks that we have roosed, ut are eyond the scoe of this work. 4. Discussion and conclusion Overall, the results resented aove suort three main conclusions: saliency (attention) has a significant ositive imact on recognition, ut this imact is largest when saliency is discriminant (of a to-down nature). Unsuervised learning of interest oints does not erform as well, although it consistently achieves etter erformance than no saliency at all (standard HMAX); max-ased ooling does not aear to have an advantage over averaging, indicating that selecting discriminant features is more imortant than locating them exactly. It could e argued that relacing the raw filter oututs with discriminant saliency measures is simly a form of normalization, whose enefits have already een ointed out in the literature (Jarrett et al., 29; into et al., 28). While normalization has advantages of its own, as shown y the gains of the ENLL and RFD networks over their sigmoidal counterart, this is not the whole story. The results aove show that non-trivial additional gains can e otained with intelligent normalization, which tunes the cell resonses according to the target recognition class, at a very marginal cost in comutation. This is a to-down saliency oeration. To illustrate the enefits of this tye of saliency for classification of natural images, we examined the intermediate comutations of the different tyes of networks. Fig. 2 shows the outut of the saliency layer for an examle image of the accordion class. The figure shows the saliency mas roduced for four Gaor orientation channels. The first row resents the magnitude jxj of the Gaor resonses (no saliency rocessing), the second row the outut of the NLL units (ottom-u rocessing), and the third row that of the LLR units trained for accordion detection (to-down saliency). Note that oth tyes of saliency units reinforce the contrast of certain areas of the image, leading to a more distinctive visual signature than the simle magnitude of Gaor resonses. The resonses of the two tyes of saliency units are, nevertheless, quite different. NLL has no knowledge of the accordion class, and simly highlights visual features that have low roaility within the field of view. These tend to e the keyoards that aear on each side of the instrument. The diagonal edges, which are a distinctive attern of the accordion oject ut lentiful on this image, are suressed. This imlies that there is some loss of information, a limitation of ottom-u saliency for recognition: universal saliency criteria Fig. 2. An image from the accordion class, and corresonding saliency oututs for Gaor channels of four orientations. To row: Magnitude of the Gaor resonses. Center: Saliency mas roduced y NLL units. Bottom: Saliency mas of LLR units.