An expressive three-mode principal components model for gender recognition

Transcription

1 Journal of Vson (4) 4, An expressve three-mode prncpal components model for gender recognton James W. Davs Hu Gao Department of Computer and Informaton Scence, Center for Cogntve Scence, Oho State Unversty, Columbus, OH, USA Department of Computer and Informaton Scence, Oho State Unversty, Columbus, OH, USA We present a three-mode expressve-feature model for recognzng gender (female, male) from pont-lght dsplays of walkng people. Prototype female and male walkers are ntally decomposed nto a subspace of ther three-mode components (posture, tme, and gender). We then apply a weght factor to each pont-lght trajectory n the bass representaton to enable adaptve, context-based gender estmatons. he weght values are automatcally learned from labeled tranng data. We present experments usng physcal (actual) and perceved (from perceptual experments) gender labels to tran and test the system. Results wth 4 walkers demonstrate greater than 9% recognton for both physcally and perceptually-labeled tranng examples. he approach has a greater flexblty over standard squared-error gender estmaton to successfully adapt to dfferent matchng contexts. Keywords: gender classfcaton, recognton, three-mode prncpal components, style analyss, bologcal moton Introducton Popularzed by Johansson n the early 97s (Johansson, 973), pont-lght dsplays provde an deal means to study the contrbuton of moton to the percepton of bologcal (and mechancal) movements. In classc experments examnng human movements, the vsual detals of the human body were hdden except for several brght pontlghts located at the major lmb jonts (shown aganst a dark background). Although sometmes dffcult to nterpret from sngle or multple statc mages (Johansson, 973; Kozlowsk & Cuttng, 977), when vewed n a sequence, the movng pont-lghts convey a vvd and compellng percept of human movement. Observers can further extract many subtle yet nformatve style propertes of the movement. People can recognze gender from gat and facal/head moton (Barclay, Cuttng, & Kozlowsk, 978; Cuttng, Profftt, & Kozlowsk, 978; Hll & Johnston, ; Kozlowsk & Cuttng, 977; Mather & Murdoch, 994; roje, ), dentfy ndvduals from gat patterns and arm movements (Beardsworth & Buckner, 98; Cuttng & Kozlowsk, 977; Hll & Pollck, ), nfer emotons from dance and arm movements (Brownlow, Dxon, Egbert, & Radclffe, 997; Dttrch, roscanko, Lea, & Morgan, 996; Pollck, Paterson, Bruderln, & Stanford, ; Walk & Homan, 984), and estmate dynamcs such as the weght of a lfted object (Bngham, 987; Bngham, 993; Runeson & Frykholm, 98; Runeson & Frykholm, 983). Regardng gender recognton from pont-lghts, both human percepton and computatonal algorthms for dscrmnatng female from male walkng styles have been explored n the past several years. Results have shown gender recognton performance by humans n the range of 46 86% for dfferent actons, ages, and vews (Barclay et al., 978; Hrashma, 999; Kozlowsk & Cuttng, 977; Montepare & Zebrowtz-McArthur, 988; Runeson & Frykholm, 983; roje, ). A recent pattern recognton framework (roje, ) reported a hgher 9.5% recognton rate on adult walkers. Much of the effort on gender recognton has focused on the manual dentfcaton of key features that enable the perceptual classfcaton between female and male walkng styles. Factors related to speed, arm swng, shoulder-hp lengths, nverson, and body sway have been examned. However, to date there s no conclusve evdence as to whch features actually drve the dscrmnaton process. No sngle feature s lkely to be suffcent, but rather t seems multple combned features are nvolved. In ths work, we evaluate our new expressve threemode pattern recognton approach (Davs, Gao, & Kannappan, ; Davs & Gao, 3a; Davs & Gao, 3b) for recognzng the gender of adult pont-lght walkers. he term expressve refers to the framework s ablty to weght the representatonal unts dfferently to best express (capture) the ntrnsc style/gender varatons (nstead of manually assgnng the key features). he framework automatcally learns the key features for a gven representaton (e.g., pont-lght trajectores) by tunng a weght value for each representatonal unt (e.g., each trajectory) to bas the numercal estmaton of class labels to match the target tranng labels (e.g., FEMALE =, MALE = +). do:.67/4.5. Receved September 9, 3; publshed May 6, 4 ISSN ARVO

2 Journal of Vson (4) 4, Davs & Gao 363 he approach frst constructs an effcent prncpal components analyss (PCA) representaton of pont-lght trajectores for a prototype (average) female and male walker. A large set of gender-labeled pont-lght walkers are then used to automatcally learn whch pont-lght trajectores (and n what combnaton) n the prototype PCA representaton best express the gender of the walkers. he nonexpressve trajectores are removed from consderaton, and the remanng expressve trajectores are weghted n the PCA representaton to bas the gender estmaton method to produce the desred gender labels. he approach combnes the benefts of PCA and machne learnng nto a sngle, robust gender classfer. he generalty of the approach s found n the ablty to tran the system to recognze dfferent gender contexts of the walkers. he typcal gender context s to recognze the true physcal gender of the walkers (s female/male). An alternate context for the system could be to recognze the perceved gender of the walker (appears female/male). For example, a female walker could be consstently perceved by several observers to have a male-lke gat pattern (appearance vs. truth). Both of these contexts may have partcular appled relevance. For automatc vsual survellance, recognzng the physcal gender s of most concern, whereas a model of the perceved gender s most mportant for computer anmaton tools to gve the best appearance of gender. Models of perceved gender are also mportant for studyng how humans dscrmnate gender. As dfferent expressve weghts may be requred for dfferent recognton contexts (physcal or perceved gender), the approach s desgned to automatcally learn the weght values for a specfc context of labeled tranng data. We demonstrate the applcablty of our framework for gender recognton by tranng and evaluatng our model wth 4 pont-lght walkers ( female, male) labeled wth ther true physcal gender and also wth ther perceved gender (obtaned from a perceptual classfcaton task). Results of our expressve PCA model show a hgher classfcaton rate attaned for both contexts than wth a standard non-expressve PCA model (wth no weghts). Related work Gender recognton from pont-lghts has receved much attenton durng the past few decades. he frst major experment was presented n Kozlowsk and Cuttng (977) wth sx walkers (three females, three males) of approxmately the same heght and weght recorded at a sagttal vew. hey demonstrated that human observers could classfy the gender of the walkers wth an average recognton rate of 63%, ncludng one female walker who was dentfed as male by most partcpants. Alteratons such as varyng the arm swng, changng the walkng speed, and occludng portons of the body were examned and found not to sgnfcantly nfluence recognton performance. Interestngly, they suggest that gender recognton s even possble from vewng only two movng ankle ponts (Kozlowsk & Cuttng, 977; Kozlowsk & Cuttng, 978). A further study of gender recognton from pont-lghts was presented n Barclay et al. (978), where temporal and spatal factors were examned. It was reported that successful gender recognton requred exposure to approxmately two walkng cycles. he renderng speed was also found to have a strong nfluence over recognton. When the movements were recorded at normal walkng speeds, but played back at an abnormally slow rate (about 3 tmes slower), the recognton rate dropped to almost chance level. Degraded dsplays, n whch the pont-lghts were dffused nto one brght blob, were also examned and shown to degrade recognton performance to chance level. he effect of nverson on the pont-lghts was also nvestgated n Barclay et al. (978). Interestngly, the gender assgnments were sgnfcantly reversed (.e., male walkers were seen as females, and vce versa). A vew-based explanaton was proposed based on the shoulder-hp rato, n whch men tend to have broader shoulders and smaller hps than women. Wth a perceptual bas to uprght fgures, the shoulder ponts n the nverted dsplay would be perceved by the observer as hp ponts (and the nverted hp ponts as shoulder ponts), thus causng the nterchange of the gender labels. Addtonal studes n Cuttng et al. (978) supported the shoulder-hp rato concept and proposed a related center-of-moment feature of the torso. he shoulder-hp rato and center-of-moment features (Barclay et al., 978; Cuttng et al., 978) are manly based on the structural dfferences between male and female walkers. However, there are certanly dynamc vsual features of movement that contrbute to recognton. By settng structural and dynamc features nto conflcton usng a synthetc pont-lght walker, experments n Mather and Murdoch (994) found that shoulder sway was an effectve cue to gender at the frontal vew (as supported by Murray, Kory, & Sepc, 97). Structural and dynamc nformaton were also compared n roje (), where dynamc-only stmul (movements appled to average postures) produced better results than wth structural nformaton (postures usng averaged motons). Most of the aforementoned experments on recognzng the gender of walkers were conducted usng a sde-vew presentaton of the walkers to observers. Other experments have been conducted to examne the effect of vew angle on gender recognton performance. In Hrashma (999), Mather and Murdoch (994) and roje (), t was found that a frontal-vew presentaton of the walkers consstently provded better gender recognton results than at a sde vew. In Montepare and Zebrowtz-McArthur (988), gender recognton usng dfferent age groups of the pont-lght walkers were examned. Chld, adolescent, young adult, and older adult walkers showed an average recognton rate of 55%, 7%, 64%, and 69%, respectvely. However, n a second experment wth a dfferent group of adults, the average recognton rate dropped to 46%. Experments n

3 Journal of Vson (4) 4, Davs & Gao 364 Pollck, Lestou, Ryu, and Cho () examned other pont-lght movements, such as knockng, lftng, and wavng, whch resulted n chance-level performance (a Neural Network demonstrated better performance). An average gender recognton rate of 75% was reported n Runeson and Frykholm (983) for complex actons ncludng walkng, sttng, rsng, standng on a char, and jumpng. hey addtonally explored the nfluence of the actor s natural, emphatc, and deceptve gender movement ntenton, whch showed that the natural movements yelded the best gender recognton rate (86%). Smlar actons were examned n Crawley, Good, Stll, and Valent () usng young chldren (4 5 years old), but resulted n unrelable gender recognton performance. A recent pattern recognton approach for gender recognton was presented n roje (), where a two-stage PCA framework was mplemented to decompose male and female walkng data nto an Egenspace, from whch a lnear classfer was used for gender recognton. he data conssted of three-dmensonal (3D) moton-capture trajectores of 4 walkers ( females, males). he frst PCA was appled to each walker to decompose the moton pattern nto a posture bass. A walker descrpton vector was constructed by concatenatng the mean posture, frst 4 posture components, and 4 snusodal parameters for modelng the perodc nature of the projecton coeffcents n the posture bass. A second PCA was then appled on the 4 descrpton vectors (one for each walker) to further reduce the walker dmensonalty. For gender recognton, a lnear classfer was appled to the projecton coeffcents of the walkers n the second PCA space. he approach yelded a recognton rate of 9.5%. Style analyss In relaton to gender recognton, much work n computer vson and computer anmaton/modelng has been devoted to the general modelng of movement styles. In computer vson, a Parameterzed-HMM was used by Wlson and Bobck (999) to model spatal pontng gestures by addng a global varaton parameter n the output probabltes of the HMM states. A blnear model was used n enenbaum and Freeman (997) for separatng perceptual content and style parameters, and was demonstrated wth extrapolaton of fonts to unseen letters and translaton of faces to novel llumnates. In Davs (), an approach to dscrmnate chldren from adults based on varatons n relatve strde length and strde frequency over varous walkng speeds was presented. Addtonally, n Davs and aylor (), the regulartes n the walkng motons for several people at dfferent speeds were used to classfy typcal from atypcal gats. Morphable models were employed n Gese and Poggo () to represent complex moton patterns by lnear combnatons of prototype sequences and used for movement analyss and synthess. In Brand (), non-rgd objects were modeled from vdeo by usng sngular value decomposton (SVD) wth constrant and uncertanty factors. Analytcal global transformatons were employed n Yacoob and Black (999) for recognzng atomc actvtes usng PCA. A method for recognzng skll-level was presented n Yamamoto, Kondo, Yamagwa, and Yamanaka (998) to determne the ablty of skers by rankng propertes such as synchronous and smooth motons. In computer anmaton and modelng, a Fourer-based approach wth basc and addtonal factors (walk; brsk) was employed n Unuma, Anjyo, and akeuch (995) to generate human moton wth dfferent emotonal propertes (e.g., a happy walk). An HMM wth entropy mnmzaton was used by Brand and Hertzmann () to generate dfferent state-based anmaton styles. An N-mode factorzaton of moton-capture data for extractng person-specfc moton sgnatures was descrbed n Vaslescu () to produce anmatons of people performng novel actons based on examples of other actvtes. SVD was used n Mason, Gomez, and Ebner () to model reach-to-grasp hand postures. A movement exaggeraton model usng measurements of the observablty and predctablty of jont angle trajectores was presented n Davs and Kannappan () to warp motons at one effort nto ncreasng efforts usng only selected trajectores. An approach usng PCA to represent anmaton sequences was presented n Alexa and Muller (). In Ch, Costa, Zhao, and Badler (), the EMOE character anmaton system used the effort and shape components of Laban Movement Analyss to descrbe a parameterzaton for generatng natural synthetc gestures wth dfferent expressve qualtes. Pror work on expressve three-mode PCA We outlned the fundamental three-mode PCA concepts usng expressve features for style analyss n Davs et al. (Davs, Gao, & Kannappan, ; Davs & Gao, 3a; Davs & Gao, 3b). he method was examned wth walkng style varatons caused by carryng load and pace. In Davs and Gao (3a), we addtonally conducted a lmted ntal experment on the physcal gender dfference of walkers. In the present work, we present a specal case of the general three-mode framework for recognzng the gender of pont-lght walkers. Specfcally, we examne physcal and perceptual recognton contexts for several pont-lght walkers. Also, we present new experments related to perceptual parameterzaton of the walkers usng perceptual classfcaton experments. Our approach to gender recognton s most related to the PCA gender classfcaton method of roje (), yet there are mportant dfferences. Frst, we use D, rather than full 3D, trajectores for the experments. Second, we normalze the walkng speed and heght of the walkers to remove these bases on the gender recognton task. hrd, because our PCA model s based on a sngle three-mode factorzaton of the posture, tme, and gender, our representaton enables us to embed an expressve weght on each pont-lght trajectory wthn the PCA representaton to adapt the estmaton of the gender labels to the tranng

4 Journal of Vson (4) 4, Davs & Gao 365 data (smlar to a neural network approach). hus our approach can be easly tuned to dfferent recognton contexts (actual vs. perceved gender). heoretcally, a smlar weghtng scheme could be used n other approaches. However, wth a step-wse factorzaton (roje, ), the walker data are typcally rasterzed and represented wth bass vectors and projecton coeffcents. We feel that operatng the expressve feature approach on these projecton coeffcents would be relevant only f the projectons on the bass vectors capture some nherent, meanngful, decomposton of the orgnal data (.e., the decomposton provdes meanngful representatonal unts). hough ths s possble, t s not generally clear that a PCA bass tself wll provde a meanngful doman for feature weghts when the data for each posture are rasterzed together. Fourer analyss has a smlar concern (though low and hgh frequency components could be mportant features, as shown n Unuma et al., 995), and a frequency-based analyss may not be best-suted to nonperodc actons (e.g., lftng or throwng). In our expressve three-mode approach, we assgn weghts to each trajectory, not to each posture bass vector, and thus a more semantc and part-wse lnk to the actual moton s retaned. Expressve three-mode PCA model Human movements can be vsually descrbed as specfc body postures changng sequentally over tme. hus, n ts most basc sense, movements have two vsual modes: posture and tme. If we further consder a stylstc component assocated wth the movement, such as the varaton n walkng due to gender, we have an addtonal thrd mode (one can easly argue for addtonal modes). In our approach, we explot ths tr-modal nature of female/male walkers (posture, tme, and gender) wth an effcent threemode PCA representaton that s sutable to ncorporatng tunable weghts on trajectores to drve the recognton process to context-based matchng crtera. he data for multple stylstc performances of a partcular acton can be naturally organzed nto a 3D cube Z (see Fgure a), wth the rows n each frontal plane/matrx Z k composed of the pont-lght trajectores (segmented and normalzed to a fxed length) for a partcular style ndex k. he matrx data for each varaton k could alternatvely be rasterzed nto a column vector and placed nto an ordnary two-mode matrx (each column a style example), but ths smply gnores the underlyng three-mode nature of the data (posture, tme, and style). Many tmes t s preferable to reduce the dmensonalty of large data sets for ease of analyss (or recognton) by descrbng the data as lnear combnatons of a smaller number of latent, or hdden, prototypes. PCA and SVD are standard methods for achevng ths data reducton, and have been successfully appled to several two-mode (matrx) problems (e.g., Alexa & Muller, ; Black, Yacoob, Jepson, & Fleet, 997; Bobck & Davs, 996; L, Dettmer, & Shah, 997; Murase & Nayar, 995; urk & Pentland, 99; Yacoob & Black, 999). hree-mode factorzaton hree-mode factorzaton (ucker, 966) s an extenson of the tradtonal two-mode PCA/SVD n that t produces three orthonormal bass sets for a gven three-mode data set arranged n a 3D cube Z (see Fgure a). For gender recognton, we consder the style dmenson as a bnary gender mode of FEMALE or MALE. We begn by reducng the cube Z to a prototype data cube usng two walkng sequences, matrces Z f and Z m, that represent the average female and average male walkng styles. Each prototype s constructed by averagng multple walkng examples of each gender class. o ensure proper algnment when averagng, one walk cycle of each example s extracted (at the same walkng phase), heght-normalzed, and tme-normalzed to a specfc duraton N. Further detals of ths preprocessng stage are presented later. Wth a total of M pont-lght trajectores per person and N frames n the sequence (tme-normalzed walk cycle), each prototype s represented as a trajectory matrx of sze M N. We then subtract the prototype mean ( Zf + Z m) / from the two gender prototype matrces Z f and Z m and place them nto the frst and second (last) frontal plane of the cube Z (see Fgure b). he dmensonalty of Z s therefore M N. he three-mode factorzaton of Z decomposes t nto three orthonormal matrces P,, and G that span the column (posture), row (tme), and slce (gender) dmensons of the cube (see Fgure c). he core C s a matrx that represents the complex relatonshps of the components n P,, and G for reconstructng Z. he desred column and row spaces can be found usng SVD. We outlne the technque n Appendx A. posture style tme Z (a) Zk posture gender Z (b) tme Zm Zf posture s P gender s t C - G t tme Fgure. (a). General three-mode arrangement of stylstc moton data. (b). hree-mode arrangement of two gender prototypes. (c). hree-mode factorzaton of gender prototypes. he posture bass P s able to represent any body posture (of pont-lghts) at any partcular tme for ether gender prototype (.e., column bass for Z ). he tme bass represents any temporal trajectory (of any pont-lght) for ether gender prototype (.e., row bass for Z ). Lastly, the gender bass G represents the gender-related changes be- (c)

5 Journal of Vson (4) 4, Davs & Gao 366 tween the two prototypes for any posture at any partcular tme (.e., slce-lne bass for Z ). ypcally, each mode needs only to retan ts frst few components (meetng some modal varance crtera) to capture most of the ft to Z. As there are only two meansubtracted gender prototypes, the normalzed gender bass s constraned to be G = [-, ] /, sgnfyng the female ( / ) and male (/ ) prototype sdes of the mean. he complete three-mode factorzaton of Z, n flattened matrx form, can be concsely wrtten as F = ( [ Zf Zm ] = PC( G j gˆ αj ). (9) ), () = j= where s the Kronecker product (Kroonenberg, 983) and [ Zf Z m ] s a matrx wth the columns of Z f followed by the columns of Z m. he core matrx C can then be solved by smply re-arrangng Equaton as C = P [ Zf Zm]( G ), () where C need not be dagonal, as s requred n two-mode PCA/SVD. Related methods for solvng ths three-mode factorzaton can be found n Kroonenberg and Leeuw (98) and Vaslescu and erzopoulos (). hree-mode gender estmaton From Equaton, each gender prototype ( Z f, Z m ) can be reconstructed as { f, m} { f, m} Z = PCg, (3) where Z { f, m} corresponds to ether gender prototype. he gender parameter g sgnfes the gender wth g f = / for FEMALE and g = / for MALE m Zf = PC (4) Zm = PC. (5) We can wrte Equaton 3 as a summaton of threemode bass elements and solate the gender parameter from the remanng factored terms wth s t Z P C g (6) = j{ f, m} p pq { f, m} jq p= q= = g s t P C jq (7) { f, m} p pq p= q= = g α, (8) { f, m} j where the ndces, j correspond to the elements n the respectve posture and tme dmensons ( M and j N). o determne the gender for a new walker wthn ths framework, we need only to estmate ts correspondng gender parameter. For ths, a mnmzaton of the threemode PCA reconstructon error for the new walker can be employed. Followng Equaton 8, the unknown gender parameter ĝ for a new walker Y (already mean-subtracted wth the model) can be estmated by fndng the value of ĝ that mnmzes the sum-of-squared-error (SSE) reconstructon M N Y Settng the dervatve of F to zero and re-arrangng the equaton, the resultng gender parameter ĝ s gven by gˆ j = Yj αj, () α j j where the gender parameter s computed by the normalzed projecton of Y onto the bass. he fnal gender can be assgned by examnng the sgn of ĝ, choosng FEMALE f t s negatve and MALE f postve (.e., selectng the nearest gender prototype). Expressve three-mode gender estmaton Gender estmaton usng Equaton could have equally been acheved by rasterzng the gender prototype data nto a MN matrx (each column s a rasterzed gender prototype), performng a standard two-mode PCA, and estmatng the gender parameter for a new walker by computng and thresholdng ts projecton coeffcent. he three-mode formulaton (Equaton ), however, enables us to easly embed tunable weght factors (on trajectores) to nfluence the estmaton of the gender parameter. he mnmzaton of Equaton 9 seeks a value of the gender parameter ĝ that reduces the reconstructon error n a squared-error manner. Hence, any pont-lght trajectores havng large magntude dfferences from the model wll sgnfcantly nfluence the mnmzaton process (outlers are a common problem n SSE mnmzatons). However, only certan trajectores may carry the most expressve and consstent nformaton regardng the gender dfferences across several walkng examples. Furthermore, the most expressve trajectores could have smaller magntude dfferences n comparson to the remanng trajectores, thus attenuatng ther mpact n an SSE gender estmaton process. What s needed s a method to weght trajectores dfferently to enable the most expressve trajectores to drve the estmaton process. Usng the three-mode reconstructon error equaton (Equaton 9), we ntroduce a weght factor n the range on each of the M pont-lght trajectores wth M N Y j j = j= F = E ( gˆ α ). ()

6 Journal of Vson (4) 4, Davs & Gao 367 he new expressve gender parameter estmaton s gven by gˆ E j = = E = E E Y j j α E α j j α j j () (3), (4) N Y j j α = j. As the denomnator α where = E j j n Equaton 3 s a constant for a gven set of factors E, we fold ths term nto the fnal expressve weghts E n Equaton 4. If we set each expressve weght to ( j ) j α n Equaton 4, the resultng gender parameter estmaton reverts to the prevous SSE method (Equaton ). However, wth non-unform values for E, the approach s capable of producng other non-sse gender estmatons accordng to a specfc recognton context. Learnng expressve weghts We present a learnng-based method to determne the approprate values of the expressve weghts E by mnmzng a second error functon that compares the computed gender parameters ĝ (usng Equaton 4) wth labels ( g =± ) pre-assgned to K dfferent tranng examples K k = J = ( g gˆ ) (5) K k M k k k= = k = ( g E ). (6) o solve for the expressve weghts n Equaton 6, we employ a fast teratve gradent descent algorthm (Burden & Fares, 993) of the form ( n+ ) = J E E( n) η( n) E, (7) wth the gradents J examples J E K k k m mk k= m= E computed over the K tranng M = - ( g E ). (8) he learnng rate η s re-computed at each teraton (va nterpolaton of the error functon) (Burden & Fares, 993) to yeld the best ncremental update. he expressve weghts are ntalzed to the default SSE formulaton, where each weght s ntally set to ( ) m j α mj, whch therefore guarantees a fnal soluton (even at a local mnma) wth a smaller error than produced wth SSE (.e., wth no expressve weghts). he weghts are also confned to be postve (by defnton) n each teraton. Experments conducted wth random values n the locus of the SSE values ( E randomly set between and, and the expressve weghts E ntalzed to j α E ( m E m mj ) showed farly consstent convergence n experments wth our data. Followng termnaton of Equaton 7, the gender parameter for a walker can be estmated wth Equaton 4 usng the newly learned expressve weghts. he general gradent descent algorthm determnes a local mnmum for a mult-parameter error functon by searchng through the parameter space to fnd values that yeld the mnmum error. he algorthm evaluates the error functon wth the current parameter estmates and updates the parameters by a small amount n the opposte drecton of the error gradent to reduce the error functon. hs updatng process s repeated untl t converges or reaches a maxmum number of teratons. It s certanly possble to use other mnmzaton technques to estmate the expressve weghts n our formulaton. If the tranng set contans enough examples (more than the number of expressve weghts), the error functon could n fact be solved lnearly. However, the matrx nverson step could produce negatve weghts. Interpretaton of expressve weghts Our framework offers a method to learn numerc weghtngs of representatonal unts to bas the estmaton of the gender parameter from a gven set of labeled tranng data. In essence, the approach combnes PCA and Neural Network learnng technques nto a sngle framework. In the experments presented n ths work, we employ low-level D poston trajectores as the representatonal unts. hose trajectores assgned a zero-valued expressve weght can be nterpreted as non-expressve gender features n the model. For the remanng trajectores wth non-zero expressve weghts, we cannot defntvely state that trajectores wth larger weghts are gven more nfluence n the classfcaton task. he magntude dfferences between the nput and reconstructed trajectores drectly nfluence the magntude of the weghts durng the numercal mnmzaton procedure. One possblty to correct ths mght be to frst normalze the trajectory data n some manner as to remove any effects due to scale across the dfferent trajectores. herefore, wth the current trajectory representaton, we make no partcular clam that the resultng non-zero expressve weght magntudes are ndcatve of what s beng used n human percepton. However, f a sutable perceptually-based representaton, perhaps usng normalzed jontangles, s employed n the framework, the resultng weght magntudes computed wth the approach may potentally show hgh-level correlatons ndcatve to what humans are usng for dscrmnaton. hs wll requre further nvestgaton.

7 Journal of Vson (4) 4, Davs & Gao 368 Walkng data o conduct the experments, we employed a set of female and male moton-capture walkng movements collected by N. roje at the BoMotonLab of Ruhr- Unversty, Bochum, Germany. he partcpants were mostly students and staff n the Psychology Department, aged between and 38 years (average age of 6 years). A set of 38 retroreflectve markers was attached to the body of each walker usng a standard marker confguraton for human fgure moton-capture. Partcpants were each asked to walk on a treadmll and adjust the speed to the most comfortable settng. en gat cycles were recorded after the partcpant was walkng for at least 5 mn (the partcpant was not notfed when recordng was to begn). A Vcon moton-capture system wth 9 hgh-speed CCD cameras was used to capture the 3D marker postons wthn mm spatal resoluton at -FPS temporal samplng. hs moton-capture data was prevously used (see roje, ), however the data for each person were transformed (usng BodyBulder bomechancal modelng software) nto a stck-fgure skeleton consstng of 5 vrtual 3D ponts located approxmately at the major jont locatons of the body. o elmnate any artfacts that may arse when computng such skeletons, we nstead chose to select most of our pont-lghts drectly from the orgnal marker set. We frst selected markers at the major arm and leg jont locatons. hen we averaged the 4 head markers (left-front, rght-front, left-rear, rght-rear) nto a sngle head pont. Smlarly, the left and rght hp markers (left-front, left-rear; rght-front, rght-rear) were averaged on each sde (producng left-hp and rght-hp ponts). he result was a set of 3 pont-lghts mostly located on the body surface (except for the head and hps). o suppress any nose from the moton-capture system, we smoothed the trajectores wth a ffth-order, zero-phase forward-and-reverse lowpass Butterworth flter wth cut-off at 6 Hz. Each walkng sequence was then rotated to have all partcpants facng the same forward drecton (postve Z- axs). he center-of-rotaton (root locaton) for each person was selected as the average center poston between the hps throughout the walkng sequence. he angle of rotaton was computed such that the average root orentaton was facng drectly forward. o remove any potental bas due to person heght, the stature of each person was normalzed by scalng the pontlghts wth the average length of the person s left and rght tbas (dstance between knee and ankle markers). Addtonally, one walk cycle was extracted from each sequence (at the same walkng phase) by detectng cyclc curvature peaks n the left-knee trajectory. Each walk cycle was then tme-normalzed to a fxed duraton usng splne nterpolaton to remove any bas of walkng speed on the gender recognton task. Each cycle was tme-normalzed to N = 5 frames to avod under-samplng (longest cycle sequence of the walkers was 44 frames at 3 FPS). o make a contnuously repettve walk cycle (for the perceptual experments), we used a smple approach that removes the dscontnuty between the last and frst frame of the cycle for each trajectory x(t) by dstrbutng the error δ = x() x(n) throughout the trajectory as ( t ) δ xt () = xt () +, (9) N where t denotes the frame number ( t N). he approach dstrbutes the dscontnuty usng small shfts throughout the trajectory to algn the startng and endng postons wthout the loss of hgh-frequency nformaton. hs smple, yet effectve, method produces seamless walkng cycles wthout notceable vsual dstorton (other Fourer-based technques could also be employed). Lastly, the 3D trajectores were orthographcally projected nto D at the frontal vew. he forward vewng drecton was prevously found to yeld the best gender dscrmnaton rate by human observers (Hrashma, 999; roje, ). he frontal vew also avods the problem of estmatng the shoulder-hp rato or center-of-moment (Barclay et al., 978; Cuttng et al., 978). Example pont-lght mages from our dataset for three female and three male walkers are shown n Fgure. (a) (b) Fgure. Pont-lght mages at the frontal vew. (a). hree female walkers (#, #8, and #6). (b). hree male walkers (#, #3, and #37).

8 Journal of Vson (4) 4, Davs & Gao 369 Gender recognton by human observers Before testng our three-mode approach for gender recognton, we frst examned the capablty of human observers to recognze the gender of our pont-lght walkers. he perceptual classfcaton labels produced from ths experment wll also be used to tran our expressve threemode model n the perceptual recognton context. Partcpants Ffteen students (5 female, male) from Oho State Unversty were recruted as partcpants for the experment. All Englsh-speakng partcpants had normal or corrected-to-normal vson. her ages ranged from to 34 years (average age of 5 years). Some partcpants had prevously been exposed to pont-lght stmul, but not to the dsplays used n ths experment. Methods A computer program was mplemented to dsplay the 4 female/male walkers and to collect the observer responses of the perceved gender of the walkers. Each sequence was rendered as black ponts aganst an off-whte background (see Fgure 3). Because the tme-normalzed walk cycles (length of N = 5 frames) appear abnormally slow f rendered at 3 FPS, we used a slghtly faster renderng speed of 36 FPS determned from the longest natural cycle tme of the walkers (.4 s). Each sequence was looped contnuously whle presented to the observer. he heght of each pont-lght walker was scaled to 7% of the screen resoluton heght (8 4 resoluton, wth -n. dagonal vewable Fgure 3. Screen-shot of the computer dsplay used for the gender-recognton task. montor). he root locaton of each walker was randomly postoned wthn a small crcle at the center of screen (wth radus % of the screen resoluton heght). hese dsplay parameters were used to prevent any explct poston or sze comparson between the walkers. he pont-lght dsplay was generated usng C++ and OpenGL wth antalasng. Each observer was seated approxmately 6 cm from the montor, whch corresponded to a vsual angle of approxmately deg for the heght of the pont-lght fgure. For each dsplayed walker sequence, the partcpant was asked to select a gender label usng the keyboard, pressng the F key to select FEMALE or M for MALE. o confrm/save the choce and load the next stmul, the partcpant was requred to press the Enter key. he 4 sequences were presented n random order for each tral. he progress was shown n the bottom-left of the computer dsplay (though not requred for the experment), and no tme restrcton was enforced. Each walker labeled by the partcpant as FEMALE was assgned a numerc label of, and each sequence labeled as MALE was assgned +. Each partcpant was pad $5, and an addtonal ncentve of 5 cents per correct gender selected after exceedng randomchance performance was pad at the end of the experment (maxmum possble payment of $5). o ensure that each partcpant was able to perceve the movng pont-lghts as a walkng person (requred before determnng the gender), we ntroduced a prevew stage before the actual experment. A sample pont-lght walkng sequence was shown to each partcpant and told that t contans a person walkng on a treadmll wth markers attached to the major lmbs. he partcpant must verbally confrm the presence of a walkng person before begnnng the experment (all partcpants could easly recognze the dsplay). As not to bas the gender recognton task, walker # (male) was selected for the prevew, because t resulted n an ambguous gender assgnment from an earler plot study wth 7 observers. Results he average recognton rate for the 4 walkng sequences across all 5 observers was 69%. he result s sgnfcantly above chance performance (t(39) = 5.5, p <.). Prevous experments employng a frontal vew of walkers (as n ths experment) reported rates of 64% (Hrashma, 999) and 76% (roje, ). hough the experments n roje () were conducted usng the same walkers as n ths experment (both datasets were derved from the same moton-capture data), the dfference n recognton rates could potentally be explaned by dfferent stmul presented to the observers. In roje (), bomechancal modelng software was employed to create a vrtual stck-fgure skeleton from the full marker set. We used 3 pont-lght trajectores, of whch pont-lghts were drectly chosen from the orgnal marker data (the remanng 3 were averaged from other orgnal

9 Journal of Vson (4) 4, Davs & Gao 37 markers). Addtonally, roje () presented several walkng cycles, whereas we only presented one (looped) walk cycle at a fxed speed. o examne the ndvdual walker results, we computed a gender consstency value for each walker by averagng the numercal values (±) assgned by the 5 observers. A consstency value of corresponds to total agreement of the walker as FEMALE, a value of + corresponds to total agreement as MALE, and a value near zero corresponds to AMBIGUOUS. We present the gender consstences from the perceptual experment wth the 4 pont-lght walkers n Fgure 4. Walkers #- are true females, and walkers #-4 are true males. here appears to be a slght bas toward perceved maleness n the walkers. here were three walkers (#, #4, #34) whose gender labels were unanmously selected by all the observers. Interestngly, walker #4 was a female that was labeled as male by all partcpants. hs clearly demonstrates the potental dfferences between perceved and actual gender. Several other walkers were stll dffcult to label (wth consstency values near zero). Prevous studes (Barclay et al., 978; Cuttng et al., 978) have suggested the shoulder-hp rato as a factor nfluencng gender recognton. We computed ths rato usng the D wdth of the shoulder s and hp h n the frst mage of each walker sequence. We note that there could, however, be more dscrmnatve structural nformaton n later frames (though t should not change drastcally at the frontal vew). he average shoulder-hp rato s/h was.7 ±.6 for females and.9 ±.4 for males, and were sgnfcantly dfferent (two-taled t test: t(38) = 3.4, p <.). he dfferences n our shoulder-hp ratos wth prevous measurements (Cuttng et al., 978) are lkely due to the nward placement of the front and back hp markers on the body (not at the maxmal hp wdth). he two hp pont-lghts (left, rght) were created by averagng the back and front hp markers on each sde. herefore the calculated hp wdth would be shorter than the actual hp wdth (thus ncreasng the shoulder-hp rato). he lack of a strong correlaton between the shoulderhp ratos for the walkers and the gender consstency values (r =.34) suggests that ths factor alone does not account for the perceptual gender choces. We also computed the center-of-moment for the walkers, usng C m = s/(s+h). Even though sgnfcantly dfferent for females and males (twotaled t test: t(38) = 3.4, p <.), ts correlaton wth the gender consstency values was also low (r =.34). Comparson to statc dsplay In addton to the dynamc pont-lght stmul, we presented observers wth only one frame from each pont-lght sequence to examne the nfluence of moton for the gender recognton task. We recruted 5 new students (3 female, male) not nvolved wth the prevous experment as partcpants. her ages ranged from to 9 years (average age of 4 years). Maleness Consstency Femaleness Walker Fgure 4. Gender consstency values for the 4 walkng sequences (females: #-; males: #-4). A total of 4 sngle-frame pont-lght mages of the female and male walkers comprsed the stmul. As the walkng poses look very smlar at the frontal vew (as opposed to the sdevew), we chose to employ only the ntal frame n the walk cycle rather than multple statc frames (as n Kozlowsk & Cuttng, 977). We agan admt that statc dfferences could potentally arse at other frames/poses. Examples mages are shown n Fgure. he same computer program (now only dsplayng a sngle frame), prevew, and compensaton method were employed as n the prevous dynamc experment. he gender selectons were collected from the partcpants and averaged nto gender consstency values (see Fgure 5a). As expected, few walkers were strongly dentfed wth ther true gender. he average recognton rate for the 4 statc mages across all 5 observers was 57%, and was above chance (t(39) =.6, p <.5). Many of the walkers were ambguous to label gven one statc frame, yet walkers #, #3, and #37 were correctly recognzed at 87%, 87%, and 93%, respectvely. In comparson, the overall statc and dynamc rates were sgnfcantly dfferent (two-taled t test: t(78) =.6, p <.5), wth the statc recognton rate almost % lower than acheved wth the dynamc dsplays. We also calculated the absolute value of the dfference between the statc and dynamc consstency values (see Fgure 5b). A large dfference magntude close to for a walker ndcates a strong gender nconsstency between the statc and dynamc cases, and a value close to ndcates that the two stmul provded a smlar (strong or weak) gender consstency. Interestngly, several walkers (e.g., #7, #, #5, and #9) had farly strong gender conflctons between the statc and dynamc cases. Overall, the dynamc stmul appear to gve more gender-related nformaton than the sngle frame case.

10 Journal of Vson (4) 4, Davs & Gao 37 Maleness.8.8 Consstency Absolute Consstency Dfference Femaleness Walker Walker (a) (b) Fgure 5. Statc dsplay expermental results. (a). Gender consstency values for the 4 statc walkng mages. (b). Dfferences between statc and dynamc consstency values. Gender recognton usng expressve three-mode PCA We now examne our expressve three-mode PCA model for gender recognton wth the same 4 female/male walker data used n the perceptual experments. We demonstrate the learnng flexblty of the framework by recognzng gender based on the actual physcal label (s female/male) and the perceved label (appears female/male). Recognzng physcal gender o compute the optmal three-mode PCA model and to avod overfttng, we employed a leave-one-out crossvaldaton technque. In ths method, we smultaneously vared the percentage modal ft of the three-mode P and bass sets (posture and tme) from 5% 95% (n 5% ncrements). For example, an 85% modal ft means that we accumulate the top bass vectors n the posture bass P untl 85% of the varance n the data s captured. We apply the same crteron for the bass. he gender bass G remans fxed to [-, ] /. For each percent modal ft of P and, we constructed 4 dfferent models, each usng 39 tranng examples by leavng one dfferent example out of the set. For each model (39 examples at a partcular modal ft), we created the gender prototypes, computed the three-mode PCA for the prototypes, and ran the learnng algorthm to compute the expressve weghts (examples labeled wth ther true gender). We emprcally selected a lmt of,5 teratons for the gradent descent learnng algorthm as t provded satsfactory convergence of the expressve weghts for our data set (n both recognton contexts). he tranng error for the model was computed by examnng the sgn of the computed gender parameter value for each of ts 39 labeled tranng examples ( : FEMALE, +: MALE). he testng (valdaton) error for the model was smlarly computed usng only the sngle left-out example. For each modal ft, we then computed the average tranng and testng errors of the 4 leave-one-out models. he cross-valdaton tranng and testng errors at each modal ft are shown n Fgure 6a. o select the optmal modal ft for the data, we chose the ft (75%) that corresponded to the smallest average testng error (5%). We then constructed a sngle expressve three-mode model at ths modal ft. Frst, the prototypes were created from the full tranng set. Next, the bass sets P and were computed at the selected modal ft (75%), and were of dmenson 6 3 and 5 3, respectvely (the core C was therefore of sze 3 3). he resultng three-mode PCA captured 98% of the overall data varance n the two gender prototypes. he expressve weghts for ths model were generated by averagng the 4 sets of expressve weghts computed at the selected cross-valdaton modal ft (75%). We show the average expressve weghts ± SD n Fgure 7. Some weghts were zero, sgnfyng that they were not relevant to the gender assgnments. he larger magntude weghts appear to have a sgnfcant devaton across the 4 leave-one-out sets, showng the mpact of the sngular leftout examples. However, as prevously mentoned, t s dffcult to assgn any hgh-level nterpretaton to the larger magntude weghts. he mappng of the 6 weghts to the pont-lghts s shown n Fgure 8. Results o evaluate the resultng expressve three-mode PCA model, we frst computed the raw (unthresholded) gender parameters usng Equaton 4 for all 4 walkers, and com-

11 Journal of Vson (4) 4, Davs & Gao ranng - Exp est - Exp 5 45 ranng - SSE est - SSE 4 4 Error Rate (%) Error Rate (%) Modal Ft (%) (a) (b) Modal Ft (%) Fgure 6. Cross-valdaton tranng and testng errors for dfferent modal fts of physcal gender. (a). Expressve model. (b). Nonexpressve SSE model. pared the results to the assgned ± physcal gender values. he target (tranng) labels and computed values from the expressve model are shown n Fgure 9a. For comparson, we employed the same cross-valdaton technque on a three-mode PCA model wthout any expressve weghts (.e., usng the default SSE estmaton). he cross-valdaton errors are shown n Fgure 6b. A constant testng error s present, though dfferent examples resulted n the errors across the modal fts. In ths case, t appears that early generalzaton (at 5% ft) was acheved for the SSE model (wth a 3% testng error). he unthresholded gender parameters produced from the fnal non-expressve SSE model and the target values are shown n Fgure 9b. he gender estmatons wth our expressve model appear much closer to the desred gender values (average dfference =.3) than the alternatve SSE estmatons (average dfference =.7). hresholdng the gender parameter values at zero produced a 9.5% classfcaton rate wth our expressve model, and a much lower 7% classfcaton rate for the non-expressve SSE verson. Although the testng errors durng cross-valdaton for the expressve and SSE models were smlar (expressve model was slghtly better), the tranng errors for the expressve model were sgnfcantly less than wth SSE. Furthermore, we appled a generalzed set of expressve weghts (averaged from the 4 crossvaldaton models at the selected modal ft). Both of these 5 [,] Expressve Feature Weghts 5 5 [3,4] [9,] [5,6] [,] [5,6] [7,8] [7,8] [3,4] [9,] [3,4] rajectory Fgure 7. Average expressve weghts ± SD from the crossvaldaton set for physcal gender. [,] [5,6] Fgure 8. Pont-lghts [x,y] labeled wth expressve weght ndex.

12 Journal of Vson (4) 4, Davs & Gao arget Estmaton 4 3 arget Estmaton Gender Gender Walker Walker (a) (b) Fgure 9. Physcal gender parameter estmaton results. (a). Expressve estmaton. (b). Non-expressve SSE estmaton. (walkers -: female; -4: male) factors enabled the expressve model to acheve a smaller recognton error than wth SSE. Recognzng perceptual gender An advantage to our framework s that the model can adapt to a dfferent labelng of the same underlyng tranng data. o demonstrate ths capablty, we also traned our model to produce gender estmatons more smlar to the classfcaton results attaned from human observers of the data. We employed the gender consstency values from the earler dynamc perceptual experment to label the walkng data. For each walker n the tranng set, we assgned a perceptual gender label of / (FEMALE) f t had a negatve gender consstency or / (MALE) otherwse. he perceptual tranng set resulted n 5 perceved-females (ncludng 4 true females) and 5 perceved-males (ncludng 9 true males). We then used the same cross-valdaton technque (over multple modal fts) outlned n the prevous secton to construct the optmal expressve PCA model. In ths perceptual context, the two prototypes were constructed usng the perceved-female and perceved-male classes (not the true gender). he average cross-valdaton tranng and testng errors for the expressve model at dfferent modal fts are shown n Fgure a. he optmal cross-valdaton set was found at an 8% modal ft (testng error of 8%). For the optmal three-mode model, the prototypes were computed from all of the perceved-females and percevedmales. he bass sets P and (at 8% modal ft) were of dmenson 6 4 and 5 4, respectvely (the core C was therefore of sze 4 4). he resultng three-mode PCA captured 98% of the overall data varance n the two gender prototypes. he expressve weghts were generated by averagng the 4 sets of expressve weghts computed at the 8% cross-valdaton ft. We show the average weghts ± ranng - Exp est - Exp Error Rate (%) Error Rate (%) Modal Ft (%) Modal Ft (%) ranng - SSE est - SSE (a) (b) Fgure. Cross-valdaton tranng and testng errors for dfferent modal fts of perceptual gender. (a). Expressve model. (b). Nonexpressve SSE model.

13 Journal of Vson (4) 4, Davs & Gao 374 SD n Fgure. As before, some weghts are zero, and we also see a larger varaton n the hgher magntude weghts. As the tranng data for the physcal and perceptual labels are n fact dfferent, we expect the resultng weghts to also be dfferent. here s an unexplaned asymmetry n the arms, though ths may be mostly due to the varaton n the cross-valdaton weghts. However, we do not yet have a hgh-level nterpretaton of the cause for the weght dfferences between the two contexts. Results he resultng gender parameter estmatons for the expressve model are shown n Fgure a. For comparson, we also computed the optmal SSE model for the perceptual data usng the cross-valdaton technque (see Fgure b), and show ts gender parameter estmatons n Fgure b. As n the prevous physcal gender case, our expressve model produced gender parameter values much closer to the desred perceptual values (average dfference =.33, smlar to the physcal gender results) than dd the alternatve SSE estmaton approach (average dfference =.68). hresholdng the gender parameter values at zero produced a 9% classfcaton rate for our expressve model and 77.5% for the non-expressve SSE model. he correlaton of the expressve perceptual gender parameter wth the gender consstency values was r =.89 (SSE correlaton was r =.69). Expressve Feature Weghts rajectory Fgure. Average expressve weghts ± SD from the crossvaldaton set for perceptual gender. Consstency weghtng o account for the gender ambguty that occurs for some walkers (havng gender consstency values near zero), we can attenuate the nfluence of those walkers and gve the remanng walkers wth hgh consstency magntudes more emphass when learnng the expressve weghts. Gven the set of K tranng examples and ther assgned perceptual genders g k, we slghtly alter the prevous matchng error functon (Equaton 6) by usng ther consstency magntudes ω k to bas the mnmzaton procedure to those examples havng more relable matches across the observers J K M = ω ( g E k). () p k k k= = he correspondng perceptual gradent s then J E K M - ( ) p = ωk k gk Em mk k= m=. () hs new gradent s used as before n the gradent descent procedure (Equaton 7) to determne the approprate expressve weghts for the perceptually-labeled walkers. Wth ths new approach, the perceptually ambguous walkers wll be mostly dsregarded when learnng the ex- 4 3 arget Estmaton 4 3 arget Estmaton Gender Gender Walker Walker (a) (b) Fgure. Perceptual gender parameter estmaton results. (a). Expressve estmaton. (b). Non-expressve SSE estmaton. (walkers #-5: perceved-female; #6-4: perceved-male)