Fast ensemble representations for abstract visual impressions

Transcription

1 Received Oct 2 Accepted Sep 2 Pulished Nov 2 Fst ensemle representtions for strct visul impressions Allison Ymnshi Lei, Ann Kosovichev 2 & Dvid Whitney DOI:./ncomms OPEN Much of the richness of perception is conveyed y implicit, rther thn imge or feture-level, informtion. The perception of nimcy or lifelikeness of ojects, for exmple, cnnot e predicted from imge level properties lone. Insted, perceiving lifelikeness seems to e n inferentil process nd one might expect it to e cognitively demnding nd seril rther thn fst nd utomtic. If perceptul mechnisms exist to represent lifelikeness, then oservers should e le to perceive this informtion quickly nd relily, nd should e le to perceive the lifelikeness of crowds of ojects. Here, we report tht oservers re highly sensitive to the lifelikeness of rndom ojects nd even groups of ojects. Oservers percepts of crowd lifelikeness re well predicted y independent oservers lifelikeness judgements of the individul ojects comprising tht crowd. We demonstrte tht visul impressions of strct dimensions cn e chieved with summry sttisticl representtions, which underlie our rich perceptul experience. University of Cliforni Berkeley, Whitney L, 2 Tolmn Hll, Berkeley, Cliforni 2, USA. 2 Northestern University, Huntington Ave, Boston, Msschusetts 2, USA. Correspondence nd requests for mterils should e ddressed to A.Y.L. (emil: ylei@gmil.com). NATURE COMMUNICATIONS : DOI:./ncomms

2 NATURE COMMUNICATIONS DOI:./ncomms Even t first glnce, oservers report tht their visul perception seems rich nd complete. At lest some of this richness my e supported y ensemle or summry sttisticl representtions. Prior work hs shown tht ensemles support gist level interprettions of scenes 2 ; however, until now, ensemle perception hs only een demonstrted for explicit visul dimensions such s size, orienttion, motion nd fces. Wht mechnism underlies the visul experiences tht go eyond simple visul fetures, textures nd explicit dimensions in scenes? Oservers cn t lest cognitively interpret strct content from scenes, including emotionl nd socil informtion,. For exmple, oservers cn report the inferred emotionl pin from photogrph of serious cr ccident, or the implied sense of urgency in photogrph of pnicking crowd. One striking strct impression tht oservers report is nimcy, or liveliness, of n oject or scene. These sorts of high-level perceptul impressions re sed on visul informtion ut re not directly ville from the imge content itself (t lest not in well-defined or strightforwrd wy) 2. Reserchers often ssume tht these high-level impressions must e cognitive: requiring ttention to contextul informtion, deliertion out mening, oserver specific lerning, or other potentilly slow or seril processes. An lterntive possiility is tht these high-level visul impressions, such s lifelikeness, might e specified quickly nd utomticlly. In fct, some reserchers hve reported rpid processing of single strct items. However, to dte, no visul mechnism hs een proposed to support the rpid extrction of strct informtion from groups of ojects in visul scene. Here we explore whether ensemle coding supports rpid strct impression formtion. Until now, ensemle perception hs only een demonstrted to operte on sic visul dimensions,, 2. However, even fleeting glimpses of visul environments engender rich Disply s Low Response 2 High Figure Stimuli nd tril sequence in Experiment. () Exmple imges illustrting the diversity of stimuli used in the experiment (out of totl). The imges exhiit wide vriety of physicl fetures, nd some stimuli my e ssocited with oth living nd non-living ttriutes. Some imges used in the ctul experiments re replced in these figures due to copyright. () Experiment tril sequence. Prticipnts viewed rndom stimulus for s nd then rted it on Likert scle of lifelikeness using slider r. perceptul impression tht cnnot e esily explined y the summry sttistics of sic visul fetures. For exmple, our impression of the liveliness nd energy depicted in photo of mounted nimls t nturl history museum compred to photo of those nimls t the zoo is not well explined y ensemle perception of colour (similr), texture (similr) or even iologicl motion informtion (irrelevnt in the cse of sttic pictures). The context lone lso does not give wy the nswer, s it depends on n interction etween the ojects nd context, mong other fctors. Museums, for exmple, cn e more or less nimte thn zoos, depending on things like whether uslod of children is rriving. In this study, we tested whether summry sttisticl perception cn precisely represent the virncy or lifelikeness of rndom sets of stimuli. We find tht oservers perceive the verge lifelikeness of crowds of ojects, demonstrting tht ensemle or summry sttisticl perception my underlie our perception of strct visul experiences. This process could provide link etween summry sttisticl representtions for sic fetures, ojects nd the kind of gist perception tht oservers report in their first glnce impressions of visul scenes. Results Individul oject lifelikeness rtings. In the first experiment, 2 prticipnts on Mechnicl Turk rted the lifelikeness of different sttic stimuli on ten-point Likert scle (see Methods). The prticipnts freely viewed ech stimulus one time; no stimulus ws repeted. In the instructions, lifelikeness (or nimcy) ws explicitly defined s how reltively live the item in the photogrph ppered. In ech tril, rndomly selected stimulus (vrious ojects, people, nimls, insects, food, nd so on) ws displyed for s. After the stimulus disppered, slider r ppered with the words Plese rte the previously shown picture on scle of, with one representing the lowest possile lifelikeness rting nd ten representing the highest possile lifelikeness rting. Figure depicts the tril sequence. Prticipnts were not given time limit to rte items, nd they were not llowed to skip ny items. After the response, the experiment dvnced to the next tril. Prticipnts completed trils in totl, nd ech prticipnt viewed the stimuli in rndomly generted sequence. We evluted prticipnts consistency in rting lifelikeness y using n intr-clss correltion coefficient test, or ICC 2. Specificlly, we used mixed, two-wy ICC model to mesure consistency cross the verge rtings. The test yielded n ICC within the excellent rnge, ICC ¼. (ref.22).this ensures tht oservers greed on the lifelikeness of ojects, nd suggests tht lifelikeness ws rted similrly cross oservers. After confirming inter-rter reliility, ech stimulus ws ssigned the verge vlue of the 2 prticipnts rtings for tht prticulr item. The lifelikeness rtings rnged from. to.. Ensemle coding lifelikeness in oject groups. In the second experiment, we creted groups of stimuli y rndomly drwing six ojects without replcement from the -item stimulus set tht hd een rted in the first experiment. This process yielded 2 groups contining stimuli ech (Fig. 2; see Methods). Ech of the 2 groups ws ssigned single, independently determined predicted lifelikeness rting, clculted y verging the rtings of the six ojects comprising the group (from Experiment dt). From now on, the result of this clcultion will e referred to s the predicted lifelikeness vlue of the group. The predicted lifelikeness rtings of the 2 groups were normlly distriuted round men of, with rnge from 2. to.2. Twenty new Mechnicl Turk sujects prticipted in the second experiment. Their tsk ws to judge the verge 2 NATURE COMMUNICATIONS : DOI:./ncomms

3 NATURE COMMUNICATIONS DOI:./ncomms ARTICLE Crowd lifelikeness rtings Crowd lifelikeness rtings s S S2 2 r =, P <. 2 r =.2, P <. 2 2 S 2 r =., P <. 2 Predicted lifelikeness rtings 2 Predicted lifelikeness rtings lifelikeness of ech group of six ojects. The methods were identicl to the first experiment, except prticipnts freely viewed group of stimuli for s, nd were sked to Plese rte the verge nimcy of the previously shown group. Prticipnts were not given time limit to rte the group, nd were not llowed to skip ny groups of stimuli. Importntly, stimuli were displyed in two wys. In the whole set condition, prticipnts viewed the entire set of six stimuli. In the suset conditions, prticipnts viewed susets of the whole set. Specificlly, in the suset conditions, either, 2 or stimuli from the entire set were displyed to the prticipnt. The susets were rndomly chosen from the whole set (see Methods). The whole set nd suset conditions were rndomly interleved throughout the experiment, nd every prticipnt viewed the crowds in rndomly generted order. Ech prticipnt responded to 2 possile groups of ojects t ech of the four set size conditions (, 2, nd ), for totl of trils. If prticipnts were le to extrct the verge lifelikeness from group of visully distinct stimuli, we would expect the prticipnts rtings in the whole set condition to correlte with the predicted lifelikeness of the group (sed on rtings of individul ojects mde y independent oservers in the first experiment). For the set of 2 unique groups, we conducted Person correltion test etween prticipnts men lifelikeness rtings of the groups in the second experiment nd the predicted lifelikeness of the groups from the first experiment. Figure 2 2 S r =.2, P <. Figure 2 Experiment 2 stimuli nd results. () In the whole set condition, disply of six stimuli ws presented for s. Prticipnts then rted the verge lifelikeness of the group of stimuli using slider r. () Experiment 2 results for four representtive sujects. The oservers rtings (lck circles) of crowd lifelikeness (y xis) were well predicted y the verge of individul item rtings derived from n independent group of oservers in Experiment (x xis), using Person correltion, n ¼ 2. shows four representtive sujects dt in the whole set condition. This nlysis ws performed individully for ll 2 prticipnts. The verged Fisher z vlue cross prticipnts, (z ¼.; r z, Po., n ¼ 2), suggests tht prticipnts were le to extrct ensemle lifelikeness. While the results indicte tht prticipnts perceived ensemle lifelikeness, it is criticl to determine whether the oservers ctully integrted lifelikeness informtion from multiple items in the group or merely rndomly smpled single item from the group. The suset conditions llow us to simulte wht prticipnts responses in the whole set condition would look like if they rndomly selected single stimulus from the group of pictures or rndomly smpled smll susets from the group of pictures 2,2,2. The suset conditions mke n importnt prediction: If prticipnts engged in ensemle coding, their rtings of lifelikeness in the whole set condition would e more highly correlted with the predicted lifelikeness vlue of the entire group compred to the suset conditions. The correltions should increse monotoniclly s set size increses. We will refer to this outcome s the suset effect. The logic is s follows: When lifelikeness informtion out the whole set of stimuli is present, oservers will use it. By the sme logic, if prticipnts integrte the presented ojects into n ensemle, correltions will decrese with smller suset sizes. This is ecuse the rndomly selected susets will not lwys e representtive of the overll lifelikeness of the group. Importntly, we will only oserve this outcome if the prticipnts ctully integrte lifelikeness informtion from multiple ojects. Conversely, if prticipnts sed their lifelikeness judgement on single rndomly smpled oject, their performnce (tht is, correltion etween oservers crowd rtings of lifelikeness nd the predicted rtings) cross the suset conditions would not show n improvement s more informtion ecme ville (Fig., left). Insted, their performnce would symptote t smll suset size, indicting tht they were using susmpling strtegy to ccomplish the tsk. Figure illustrtes the two expected ptterns of performnce for rndom susmpling nd for integrting percent of the items, n extreme form of ensemle coding. Suset condition control results. For ech suject nd suset condition, we clculted Fisher z (correltion) etween prticipnts rtings of set lifelikeness nd the predicted lifelikeness of the whole set (from Experiment ). The verge Fisher z scores cross suset conditions were well fit y liner regression, r 2 ¼., Po., n ¼, illustrting tht prticipnts exhiited incresing correltions s set sizes ecme lrger (Fig. ). These results confirm tht sujects were not engging in strtegy of smpling one oject or relying on the most extreme oject to ccomplish the lifelikeness rting tsk. Insted, these results indicte tht prticipnts used most of the ville informtion hllmrk of ensemle coding. Additionlly, we conducted permuttion test on the Fisher-trnsformed dt to compre prticipnts correltion vlues etween the four-oject suset nd the whole set. Prticipnts exhiited higher correltion in the whole set thn the four-oject suset, Po., n ¼ 2. This indictes tht prticipnts integrted informtion from more thn four ojects in the set. Fst ensemle lifelikeness perception. The dt from the second experiment indicted tht prticipnts perceive ensemle lifelikeness in sets of ojects viewed for s. An open question, in Experiment, is how quickly ensemle lifelikeness perception cn operte. To test this, nd to confirm tht the results of the second experiment extend to lortory setting, we replicted the NATURE COMMUNICATIONS : DOI:./ncomms

4 NATURE COMMUNICATIONS DOI:./ncomms Rndom susmpling one item Integrting % of the set Mgnitude of correltion Mgnitude of correltion + ms- s # of items in disply Averge fisher z..2 n = 2 2 # of items in disply # of items in disply Figure Hypotheticl nd empiricl results for Experiment 2. () Hypotheticl outcomes for the suset conditions, compring prticipnts performnce during rndom susmpling of one oject or during ensemle coding (n extreme version of ensemle coding in which % of the ojects in the set re integrted). Left: If the prticipnt rndomly smples lifelikeness informtion from just one item in the set, the mgnitude of the correltion should remin reltively constnt, even when more informtion ecomes ville ecuse the prticipnt does not use the new informtion. Right: In contrst, if the prticipnt integrtes lifelikeness informtion from every item s it ecomes ville to them, the correltion etween prticipnts lifelikeness rtings nd the predicted lifelikeness rtings of the crowd should increse s more informtion (more items) ecomes ville. () Experiment 2 Results. The Fisher z scores increse s the numer of items displyed increses. This pttern indictes tht prticipnts integrted the ville informtion nd did not use rndom susmpling technique to ccomplish the tsk. Error rs represent s.e.m. second experiment t five different exposure durtions with experienced psychophysicl oservers (see Methods). Susets of stimuli (s in the second experiment) were presented for durtions of ms, 2 ms, ms, s or s. In ech tril, prticipnts foveted on fixtion cross, nd viewed up to six stimuli, which were displyed isoeccentriclly round the fixtion cross (see Fig., lso see Methods). The different exposure durtions nd suset conditions (, 2,, stimuli per set) were rndomly interleved (2 trils ech), totlling trils per prticipnt. We oserved highly roust correltions etween prticipnts estimtes of ensemle lifelikeness nd the predicted lifelikeness vlues of the crowds for ll exposure durtions (Fig. ). The prticipnt s Fisher z scores in ech exposure durtion condition were well fit y liner regression (lowest r 2 ws., in the ms set durtion condition, P ¼., n ¼ ) illustrting the incresing correltions for lrger set sizes. To confirm tht ensemle lifelikeness perception does not rely on visul or monitor persistence, we conducted two control experiments tht replicted the results of this third experiment with ckwrd msking of the riefest displys (Experiments nd, Supplementry Fig., Supplementry Tle 2). These results * Person r s s ms # of items in disply # of items in disply # of items in disply Suject Suject 2 Suject Suject Person r Person r 2 ms ms 2 2 # of items in disply # of items in disply Figure Ensemle coding lifelikeness t different disply durtions. () Exmple disply in Experiment. Prticipnts viewed groups of up to six stimuli presented for ms to s, then rted the verge lifelikeness of the group. () Prticipnts consistently integrted multiple items cross disply durtions. reinforce the results of Experiment 2 (dt from Mechnicl Turk) within controlled lortory setting, indicting tht it is possile to formulte n ensemle percept of visully distinct items. These results lso provide hint tht prticipnts integrted multiple stimuli into their estimtes of ensemle lifelikeness, even for riefly presented sets. Although this is intriguing evidence tht ensemle lifelikeness perception my e fst process, there remin unnswered questions. First, how mny stimuli re integrted in rief glnce? And, second, do sujects rely on explicit memory of the stimuli? To ddress these questions, we conducted follow-up experiment. Limited explicit memory for set memers. The third experiment indicted tht oservers perceive ensemle lifelikeness even for riefly presented sets. Although this could indicte fst perceptul process, n lterntive is tht oservers recll the items in the set nd use this memory to mke their judgement of lifelikeness. It is therefore necessry, in Experiment, to determine whether group lifelikeness perception occurs without explicit memory for individul items in the group. To ddress this, we needed more precise estimte of how mny stimuli oservers integrted into their ensemle lifelikeness percept, nd we needed to mesure memory cpcity for the ojects in the sets. To this end, we presented sets of ojects in two conditions, mesured in seprte runs. In the first condition, we replicted the third experiment, presenting sets of, 2, or ojects for 2 ms. Like in Experiment, we mesured correltions etween prticipnts estimtes of ensemle lifelikeness nd the predicted lifelikeness vlues of the crowds from independent oservers. In the second condition, the sets of stimuli were similr ut the oservers performed memership identity tsk. In ech tril, sujects viewed six stimuli displyed isoeccentriclly from centrl Person r Person r NATURE COMMUNICATIONS : DOI:./ncomms

5 NATURE COMMUNICATIONS DOI:./ncomms ARTICLE fixtion cross for 2 ms. After the set disppered, two items ppered on the screen. One item ws memer of the previous set, nd one item ws lure (drwn from the full set of imges). In two-lterntive forced choice tsk, prticipnts chose which stimulus ws memer of the previous set y pressing one of two keys on keyord. Memory cpcity ws estimted from the proportion correct in this tsk (see Methods). Oservers prticipted in trils in ech condition. Results for the first condition replicted nd extended the third experiment. The verge Fisher z score ws.2, r z, Po., n ¼. We lso fit liner model to prticipnt s Fisher z scores (collpsing cross the 2 trils in Experiment nd trils in Experiment ) during the 2 ms exposure condition nd found significnt positive liner trend (liner model r 2 ¼., Po., n ¼ ). On verge, prticipnts integrted up to six stimuli in their estimtes of ensemle lifelikeness (see Fig. ). A permuttion test compring prticipnts Fisher z vlues etween the four-oject suset nd the whole (six oject) set reveled significnt difference, P ¼.2, n ¼, indicting tht oservers integrted up to six ojects in their estimtes of ensemle lifelikeness. A regression nlysis 2 complemented these findings, nd indicted tht prticipnts integrted ll disply items into their ensemle percept (Supplementry Tles nd 2). The second condition mesured memory cpcity for ojects in the riefly presented sets. On verge, prticipnts exhiited n effective memory cpcity of. items, when sets of six stimuli were presented (verticl dshed lines in Fig.,; see Methods). Tken together, the results demonstrte tht perception of group lifelikeness cnnot e sed on explicit memory lone, or on cognitive clcultion tht relies on the explicit memory of ech item. Ensemle lifelikeness cn e perceived even when the individul set memers re lost or forgotten. Perceiving group lifelikeness over time. The first four experiments required prticipnts to extrct n ensemle percept from items rrnged in sptil rry, demonstrting tht ensemle lifelikeness perception involves some degree of sptil integrtion. Averge fisher z.2 2 # of items in disply Memory cpcity Ensemle coding performnce n = * Person r Person r S 2 # of items in disply S 2 # of items in disply S2 2 # of items in disply S 2 # of items in disply Figure Limited explicit memory of ll items. () Oservers perceived ensemle lifelikeness in the 2 ms condition in Experiments nd y integrting five or more ojects (indicted y the significnt difference in performnce when four nd six items re viewed). Seprte memory cpcity testing reveled tht prticipnts only reclled less thn two items per disply, on verge (lck dshed line). Error rs represent s.e.m. () Individul suject dt. Becuse of eye nd oject motion, however, we often encounter ojects nd crowds in dynmic situtions. In Experiment, we tested whether oservers cn perceive ensemle lifelikeness in temporl sequence 2,2. In this fifth experiment, six prticipnts in the lortory viewed the sme sets of stimuli viewed y Mechnicl Turk sujects in the second experiment. However, s shown in Fig., these items were displyed sequentilly over time, rther thn simultneously over spce. After the items disppered, lnk screen ppered, during which prticipnts rted verge the lifelikeness of the set. Prticipnts used uttons lelled on the computer keyord, where represented the lowest lifelikeness rting nd represented the highest lifelikeness rting. There were four conditions, corresponding to four different set sizes (, 2, or ojects per set). The whole set condition contined six items displyed fovelly with sptil jitter (see Methods) for ms ech. In the suset conditions, prticipnts viewed ech item for longer durtion to equlize the totl visile stimulus durtion (see Methods). Note tht, in mny wys, incresing the exposure durtion for the items in the smller susets ctully works ginst the suset effect (Fig. right pnel), s the longer exposure could fcilitte recognition or memory. Therefore, performnce might e expected to increse Crowd lifelikeness rtings r =. 2 2 Predicted lifelikeness rtings ms per item Averge Fisher z.2 Time n = * 2 # of items in disply Figure Exmple stimuli nd results for Experiment. () In the whole set condition, prticipnts viewed groups of six stimuli, sequentilly presented for ms ech. Prticipnts then rted the verge lifelikeness of the previously seen group. () Experiment results. Left: Crowd lifelikeness of temporlly presented groups is well predicted y the verge of individul item rtings. This grph depicts the rtings (lck circles) of one representtive prticipnt who judged 2 unique temporl crowds in the whole set condition. There is high correltion etween the rtings of verge crowd lifelikeness (y xis) nd predicted rtings of the crowds generted from single-item rtings (x xis), n ¼ 2. Right: Averge results for six oservers. The x xis represents the numer of items displyed in the set. The y xis represents the mgnitude of the Fisher z score. The Fisher z score in the whole set condition indictes tht prticipnts were le to perceive ensemle lifelikeness in sequentilly presented groups of items. Moreover, Fisher z scores increse s the numer of items displyed increses. This pttern rules out the possiility tht prticipnts engged in rndom susmpling to ccomplish the tsk. Error rs represent s.e.m. NATURE COMMUNICATIONS : DOI:./ncomms

6 NATURE COMMUNICATIONS DOI:./ncomms (or sty constnt) in the smll suset conditions. Despite this, if prticipnts truly integrte lifelikeness informtion from most of the items displyed, we will still oserve the predicted suset effect (Fig. right pnel) ecuse the suset is not truly representtive of the verge of the whole crowd. Ech prticipnt completed trils in totl (2 possile groups of ojects set sizes). If oservers re le to extrct the verge lifelikeness from group of sequentil stimuli, we would expect lifelikeness rtings in the whole set condition in this experiment to correlte with the predicted lifelikeness of the crowd sed on the rtings of individul ojects mde y independent oservers (from Experiment ). Once gin, we conducted ivrite correltion test etween prticipnts lifelikeness rtings of the temporl crowds presented in Experiment nd the predicted lifelikeness vlues. Figure left pnel shows the results from one representtive prticipnt, Person r ¼., Po.. We trnsformed the Person correltion coefficients to Fisher z scores nd verged cross prticipnts (Fisher z ¼., r z ¼.22, Po., n ¼ ). A liner regression of verge group performnce versus set size demonstrted tht prticipnts performnce ecme more correlted s set size incresed, r 2 ¼., P ¼., n ¼ (Fig., right pnel). A permuttion test reveled significnt difference etween the four-item suset nd the whole set, Po., n ¼, with prticipnts exhiiting higher correltion in the whole set versus the four-item suset. This indictes tht prticipnts integrted five or more items into their ensemle percept. It lso confirms tht prticipnts were not using rndom susmpling strtegy, nd revels tht prticipnts cn ensemle code lifelikeness temporlly s well s sptilly. Visul short-term memory control. A common question in ensemle coding literture is whether ttention or wreness of single ojects is necessry to formulte n ensemle percept. Mny previous experiments indicte tht prticipnts re le to formulte remrkly precise ensemle percept of low-level stimuli (simple shpes, orienttion or motion) or high-level stimuli (fce, iologicl motion), even when performnce t memership identity tsks is reltively poor,2. Ensemle nimcy perception ppers to follow similr pttern: Experiment found tht oservers hd memory cpcity of less thn two effective ojects, nd yet they integrted more thn five into their ensemle percept. However, tht experiment employed sptil rrys. An open question is whether the oservers lck informtion out the individul ojects in the temporl rry (Experiment ). In the sixth experiment, we therefore investigted whether prticipnts were le to formulte n ensemle percept of temporl sequence of ojects without specific memory of every item in the group. In the sixth experiment, we lso incresed the set size to 2 items. This llowed us to ensure tht the results remined roust with lrge set sizes. In this experiment, we pseudo-rndomly generted crowds (see Methods) for ech different prticipnt. Five prticipnts in the lortory viewed temporlly presented sets of 2 stimuli. Stimuli were presented fovelly with sptil jitter (see Methods). The suset conditions consisted of, 2, nd -item susets. Ech stimulus ws shown for ms, with ms interstimulus intervl (ISI) in ll set sizes. Whole (2-item) set nd suset conditions were rndomly interleved. Oservers prticipted in totl of trils, with 2 trils per set size condition. A lyout of the experimentl tril sequence is depicted in Fig.. First, prticipnts viewed the temporl disply of stimuli. Prticipnts then performed two rndomly ordered tsks. In one tsk, they rted the verge lifelikeness of the group. In the other tsk, prticipnts performed 2AFC memership identity tsk (identicl to Experiment ). In the 2AFC tsk, two stimuli were shown on the screen side y side. One of the stimuli ws lure; the other stimulus ws rndomly selected item from the previously seen set. Prticipnts chose which oject ws memer of the set (see Methods). The order of the tsks ws rndomized. Tht is, in some trils, the prticipnts performed the memership identity tsk directly fter viewing the crowd, nd then rted the lifelikeness of the crowd. In other trils, prticipnts rted the lifelikeness of the crowd directly fter viewing the crowd, nd then performed the memership identity tsk. Prticipnts were not given time limit to perform either tsk (memership identity nd lifelikeness rtings). As soon s the prticipnts entered their response, the other tsk ppered. As in the previous experiments, we correlted prticipnts lifelikeness rtings of the whole set with the predicted lifelikeness rtings sed on n verge of the individul ojects within the set (s determined y independent oservers in the first experiment). In this prticulr experiment, the whole set ws twice the size of whole sets in previous experiments (tht is, 2 versus items). We still oserved roust correltion etween the prticipnts lifelikeness rtings nd the predicted lifelikeness of the group (Fig. ; Fisher z ¼.22, r z, Po., n ¼ ). A representtive prticipnt s dt re shown in Fig.. We exmined prticipnts performnce in the different set sizes y exmining prticipnts verge Fisher z scores in the suset conditions. The verge Fisher z vlues cross the different set sizes were well fit y liner model, suggesting tht performnce improved s set size incresed, r 2 ¼.2, Po., n ¼ (Fig. ). Additionlly, permuttion test compring Fisher z vlues etween the eight-item suset nd the whole (2-item) set indicted significnt difference, P ¼.2, n ¼, with prticipnts exhiiting higher correltion in the whole set versus the eight-item suset. This indictes tht prticipnts integrted more thn eight items into their ensemle percept of lifelikeness. We nlysed the numer of items prticipnts rememered in ech tril y clculting mesure of memory cpcity (see Methods). Prticipnts memory improved in the smller susets, ut plteued for the lrger set sizes. Prticipnts rememered pproximtely item in the -item disply, 2 items in the 2-item disply, items in the -item disply, nd items in oth the -item set nd the 2-item disply (verge effective memory cpcity (MC) in the -item set ¼. (SEM ¼.); MC in the 2-item set ¼.2 (SEM ¼.); MC in the -item set ¼ 2. (SEM ¼.); MC in the -item set ¼. (SEM ¼.), MC in the 2-item set ¼. (SEM ¼.)). Even though prticipnts rememered pproximtely items in oth the -item disply nd the 2-item disply, their ensemle coding performnce continued to sustntilly increse etween set sizes nd 2. This indictes ensemle coding without explicit memory for individul items in the temporl disply. This pttern of results closely mirrors the results in Experiment, where explicit memory for individul ojects in sptil rry of ojects ws not necessry for perception of ensemle lifelikeness. Binry versus grded representtions. Experiments 2, nd suggested tht prticipnts perceived ensemle lifelikeness over spce nd over time. Experiments nd replicte these findings, nd suggest tht explicit memory of individul ojects ws not required to form ensemle representtions of lifelikeness. However, it remins unknown how prticipnts extrct the verge lifelikeness of the group. One potentil strtegy is tht prticipnts perceive lifelikeness s ctegoricl nd inry (either live or not) nd then verge the items (we refer to this s the inry strtegy). Alterntively, oservers might perceive grded NATURE COMMUNICATIONS : DOI:./ncomms

7 NATURE COMMUNICATIONS DOI:./ncomms ARTICLE Disply Response Wht is the verge nimcy? Time ms per item ms ISI Which item ws prt of the group? Order Rndomized Figure Tril sequence for the whole set condition. First, prticipnts viewed 2 rndom stimuli displyed sequentilly for ms per item, with ms ISI. Next, prticipnts viewed two-lterntive forced choice memership identity tsk nd prticipnts were sked to rte the verge lifelikeness of the group. Prticipnts were not given time limit to complete oth tsks. The order of the two tsks ws rndomized throughout the experiment, so tht sometimes prticipnts performed the lifelikeness rting first nd sometimes prticipnts performed the memership identity tsk first. Crowd lifelikeness rtings r =. 2 2 Predicted lifelikeness rtings or reltively nunced differences in the lifelikeness of individul items to chieve precise ensemle percept of the group s whole. We refer to this strtegy s grded verging strtegy. Both perceptul strtegies could result in n ensemle representtion, nd oth strtegies my e relied upon under different circumstnces, ut they mke different predictions. To investigte this, in Experiment, we used design similr to the sixth experiment except tht prticipnts were instructed to Averge Fisher z # of items in disply * Memory cpcity Ensemle coding performnce n = Figure Results of Experiment. () Crowd lifelikeness of lrge groups is well predicted y the verge of individul item rtings from independent oservers. This grph depicts the rtings (lck circles) of one representtive prticipnt judging 2 unique temporl groups. There is high Person correltion, r ¼. etween the rtings of verge crowd lifelikeness (y xis) nd predicted rtings of the groups generted from single-item rtings (x xis). () Averged dt for five oservers. The x xis represents the numer of items displyed in the set. The y xis represents the mgnitude of the Fisher z score. The Fisher z score in the whole set condition indictes tht prticipnts re le to perceive ensemle lifelikeness even in lrge groups of items (in this cse 2 items). The Fisher z scores increse s the numer of items displyed increses. This pttern rules out the possiility tht prticipnts engged in rndom susmpling techniques to ccomplish the tsk. Error rs represent s.e.m. Finlly, the dshed line represents the memory cpcity limit in this experiment. Prticipnts rememered on verge five items in ech disply. However, their ensemle coding performnce continued to significntly increse etween nd 2 items, indicting tht ensemle coding performnce is not solely dependent on explicitly rememered items. count either the numer of living or non-living items in ech group (counterlnced cross prticipnts). The tril procedure ws s follows: Prticipnts viewed temporlly displyed crowd, then fter ech disply, prticipnts verlly reported their count to the experimenter. After their verl report, prticipnts rted the ensemle lifelikeness of the group nd performed the memership identity judgement in rndomized order, similr to Experiment (see Fig. for tsk lyout). The counting tsk llowed us to crete inry verging model: simultion of wht prticipnts performnce would look like if they were simply ssigning inry vlues (for exmple, s nd s, or s nd s, nd so on) to the non-living nd living items respectively, nd then verging these numers. Oservers prticipted in 2 trils totl. Both strtegies (inry verging nd grded verging) will, unsurprisingly, produce similr pttern of results. However, if prticipnts extrct grded informtion, we should still oserve slight decrement in performnce in the inry verging simultion compred to their ctul ensemle coding judgement. This is exctly wht we found. We fit liner regression through the origin of prticipnts ensemle coding performnce nd the performnce predicted from the inry verging model. Prticipnts grded verging ws etter fit (grded verging r 2 ¼., inry verging r 2 ¼.). To ensure tht prticipnts ensemle rting judgements were not ised y the counting tsk, we lso compred the sme inry verging simultion for ech prticipnt to their verging performnce in seprte tsk. This seprte mesure of ensemle coding performnce ws identicl to Experiment (excluding the counting tsk), nd the result ws similr (grded verging r 2 ¼., inry verging r 2 ¼.). Thus, our results suggest tht prticipnts did not solely rely on inry ensemle coding strtegy when viewing sets of sequentilly presented stimuli. We lso tested whether oservers use grded versus inry verging when judging nimcy in sptil rrys of stimuli. We found tht prticipnts did not rely on inry ensemle coding strtegy (Experiment, Supplementry Fig. 2). Additionl control experiments ensured tht prticipnts did not merely rely on low-level fetures or imge sttistics to determine lifelikeness (Experiments nd 2; Supplementry Figs nd ). Insted, prticipnts relied on configurl informtion to ssess the liveliness of group. Discussion Our results indicte tht oservers cn perceive the verge lifelikeness of groups of ojects in frction of second. This is NATURE COMMUNICATIONS : DOI:./ncomms

8 NATURE COMMUNICATIONS DOI:./ncomms Disply Response Wht is the verge nimcy? Time ms per item ms ISI How mny living items did you see? Which item ws prt of the group? Order rndomized Figure Tril sequence in Experiment. Prticipnts were instructed to count the numer of living items in the disply. They viewed 2 rndom stimuli displyed sequentilly for ms per item with ms ISI. Then, prticipnts verlly reported their count to the experimenter. Afterwrd, the prticipnts performed oth the 2AFC memership identity tsk nd rted the verge lifelikeness of the group. the first evidence tht ensemle coding my contriute to our first glnce or gist impression of strct ttriutes like nimcy or liveliness. Our results show tht gist impressions of visul scenes re rich: they encompss more thn sprse sttisticl summry of concrete physicl dimensions. Until now, it ws unknown whether individuls could extrct n ensemle code from perceptul impressions tht were not immeditely specified y the visul fetures in the imge. Our experiments demonstrte tht individuls cn extrct ensemle percepts out strct visul interprettions, suggesting tht lifelikeness is n explicitly coded perceptul dimension. Moreover, these representtions re remrkly consistent cross oservers, suggesting tht lifelikeness is shred visul percept. Our results provide link etween summry sttisticl representtion of sic visul fetures, nd the virnt, complex perceptions tht oservers report experiencing in their first impressions of visul scene. Our findings revel tht ensemle perception of lifelikeness is chieved extremely rpidly. While previous work hs shown tht oservers ctegorize stimuli in rief time period (for exmple, niml or non- niml, ), our study shows tht oservers cn perceive reltive lifelikeness (tht is, whether one stimulus is more life-like thn nother) on similrly rpid timescle for groups s well. These results prllel the rpid time scle reported in previous ensemle coding experiments using stimuli with explicit physicl dimensions 2,2, highlighting the remrkle efficiency of ensemle representtions tht support strct visul impressions. Our findings suggest tht lifelikeness is n explicitly coded perceptul dimension tht is continuous s opposed to dichotomous. One prior study hs investigted whether nimcy is strictly dichotomous representtion, or whether nimcy is represented s continuum. While this prior study focused on single repeted stimuli shown for longer exposure durtions, our findings extend this question to groups of heterogeneous ojects tht were riefly presented. Our prticipnts extrcted grded ensemle percept of group lifelikeness. Becuse of the rpid timescle, the judgements of lifelikeness in our experiment would not llow for cognitive resoning or socil processes. Consistent with this, explicit memory of the ojects in the sets ws not sufficient to ccount for the numer of ojects integrted into the ensemle percept. Our results suggest tht grded representtions of oject nd crowd lifelikeness emerge s sic, shred visul percept, ville during rudimentry nd rpid visul nlysis of scenes. Animcy, s generl construct nd topic of cognition reserch, is extremely complex. Numerous contextul, cognitive nd socil mechnisms come into ply when determining whether n oject exhiits nimte qulities. Specificlly, when mking judgements out nimcy, theory of mind, contextul cues, nd cognitive strtegies 2 contriute significntly to nimcy evlutions. These complexities help explin why there re reltively few greed-upon opertionl definitions of nimcy or lifelikeness. In contrst to the miguity of the terms nimcy or lifelikeness, our results show tht the ensemle perception of lifelikeness in groups of sttic ojects ws surprisingly consistent cross oservers. When stimuli were presented for rief durtions, oservers reched remrkle consensus on the verge lifelikeness even regrding ojects tht exhiit seemingly miguous qulities. This consistency suggests tht similr percept of lifelikeness is commonly ville to oservers who glnce t scene. Numerous cognitive nd socil mechnisms my come online lter, nd oservers my refine their percepts of lifelikeness when given longer periods to evlute items nd context. However, in first-glnce impression of the environment, oservers shre reltively unified, consistent percept of lifelikeness. Methods Prticipnts. In totl we tested helthy prticipnts with norml or corrected-to-norml vision. In Experiment, we tested 2 prticipnts on Amzon Mechnicl Turk (men ge ¼., SD ¼.; 2 mles, femles). In Experiment 2, we tested 2 new prticipnts on Amzon Mechnicl Turk (men ge ¼., SD ¼.; mles, femles). In Experiments nd, we tested four experienced psychophysicl oservers (men ge ¼ 2. SD ¼.; 2 mles, 2 femles). In Experiment, we tested six experienced psychophysicl oservers, including one uthor (men ge ¼ 2., SD ¼.; mles, femles). In Experiment, we tested five experienced psychophysicl oservers, including one uthor (men ge ¼ 2, SD ¼.; mle, femles). In Experiment, we tested eight oservers in the lortory (men ge ¼ 2.2, SD ¼.; femles, mles). In Experiment, we tested three experienced psychophysicl oservers (men ge ¼ 2, SD ¼.2; 2 femles, mle). In Experiment, we tested two experienced psychophysicl oservers (men ge ¼ 2, SD ¼.; mle, femle). In Experiment, we tested four experienced psychophysicl oservers (men ge ¼ 2. SD ¼.; 2 mles, 2 femles). In Experiment, we tested three experienced psychophysicl oservers (men ge ¼ 2. SD ¼.; 2 femles, mle). In Experiment 2, we tested prticipnts from Amzon Mechnicl Turk (men ge ¼. SD ¼.; femles, mles, for those who reported demogrphic informtion). Amzon Mechnicl Turk oservers who filed to complete ll of the experimentl trils were utomticlly excluded from the experiment nd ny susequent dt nlysis. Some oservers who were tested in the lortory prticipted in multiple experiments. All prticipnts, with the exception of the one uthor, were nïve s to the purpose of the experiment. Ech prticipnt provided informed consent in ccordnce with the IRB guidelines of the University of Cliforni t Berkeley. Stimuli. We used the stimuli from the Mssive Visul Memory Stimulus dtse. This stimulus set contins coloured photos of diverse ojects including electronics, household items, food, plnts, people, nimls, insects, vehicles, furniture nd mny other items on white ckground. As the dtse my e hevily ised towrds living or non-living ojects, we first very roughly lnced NATURE COMMUNICATIONS : DOI:./ncomms

9 NATURE COMMUNICATIONS DOI:./ncomms ARTICLE or flttened the distriution of living nd non-living stimuli. To ccomplish this, the dtse ws corsely divided into living nd non-living stimuli y one oserver. This does not reflect n ojective mesure or reference seline of lifelikeness, ut ws done to simply increse the likelihood tht prticipnts potentilly viewed rod distriution of items. Prticipnts were unwre of this step or tht the experimentl stimuli were pproximtely lnced in this wy. Of course, individul oservers might still perceive the distriution of stimuli s hevily ised towrds or wy from living or non-living ctegories. From these divided stimuli, we rndomly chose nominlly living nd nominlly non-living items. Figure shows representtive suset of the imges used in the experiment. The stimuli were presented either using Qultrics (r 2) for online prticipnts or Psychophysics Toolox in Mtl for lortory experiments. Prticipnts on Amzon Mechnicl Turk were sked to plce their personl computer monitor in centred position in front of them, nd were sked to mintin cler, unostructed view of the pictures nd sit n rm s length wy from the computer screen. Prticipnts in the lortory viewed stimuli on. cm imc LCD monitor with resolution of 2,, nd Hz refresh rte. Prticipnts st with the screen positioned centrlly in front of them t viewing distnce of cm. Ech stimulus ws presented in white ox, with oundries sutending.. of visul ngle. In Experiment 2, ech group of imges ws rrnged on grid with three stimuli on the top row nd three stimuli on the ottom row within pixel grid. The loction of ech stimulus ws rndomly determined within the grid. The size of ech stimulus ws 2 2 pixels, nd prticipnts were llowed to freely view stimuli. In Experiments,,, nd, prticipnts were instructed to fovete on fixtion cross. The visul ngle etween the fixtion cross nd stimuli ws.. In Experiments nd, ech stimulus ws presented sequentilly t the centre of the screen, with sptil jitter of up to. on the verticl xis,. on the horizontl xis. Across ll the ove-mentioned experiments, the mximum nd minimum luminnce in the pictures ws 2. nd. cd/m 2 respectively. The mximum Michelson contrst ws.2. The visul ngle nd Michelson contrst in the remining experiments (Experiments, 2 nd 2) ws not mesurle, s these experiments were conducted on Mechnicl Turk. The stimuli were rndomized in the following mnner to crete crowds of stimuli: In Experiments 2 nd we rndomly drew from the entire stimulus rry ( items) without replcement to generte 2 displys of stimuli. This rndom method yielded rod rnge of predicted lifelikeness vlues for the crowds, from 2. to.2. Ech prticipnt viewed the 2 crowds in rndom order. In Experiments,, nd the crowds of ojects were rndomly generted for ech prticipnt on ech tril. In Experiments, nd, we pseudo-rndomly drew from the originl stimulus set ( items) to generte crowds of 2 stimuli. The one constrint ws tht one-third of the rndomly drwn groups were elow predicted men lifelikeness of, one-third of the groups hd predicted lifelikeness men of, nd one-third of the groups hd predicted men lifelikeness ove. This method ensured similrly rod rnge of predicted lifelikeness rtings for the crowds, despite the fct tht we incorported twice the numer of items for ech disply. In Experiment, we pseudo-rndomly drew from the entire stimulus set, with the constrint tht oth sides of the disply contined the sme numer of nimte items, either 2 nimte items nd innimte item on ech side of the disply, or 2 innimte items nd nimte item on ech side. In Experiment 2, we rndomly generted sets of stimuli, ech contining crowds of rndomly selected ojects nd crowds of scrmled ojects (Supplementry Fig. ). Amzon Mechnicl Turk prticipnts were rndomly ssigned to one of the five stimulus sets. Procedure. Experiments 2 2 incorported the following generl tril lyout: First, prticipnts viewed group of stimuli. Next, prticipnts were sked to rte the verge lifelikeness of the groups of stimuli. The different experiments included different disply durtions, numer of stimuli nd response methods. Specificlly, in Experiment 2, six stimuli were shown for s. In Experiment, six stimuli were shown for ms, 2 ms, ms, s, nd s in interleved trils. In Experiments, nd, six stimuli were shown for 2 ms. In Experiment, six stimuli were shown sequentilly for ms ech in the whole set condition. To equlize totl durtion, the suset condition stimuli were shown for longer ( ms per item in the -item suset condition, ms per item in the 2-item suset condition, ms per item in the -item suset condition). In Experiments nd, twelve, eight, four, two nd one, stimuli were shown sequentilly for ms in interleved trils. In Experiment, twelve stimuli were shown sequentilly for ms in the whole set condition. To equlize totl durtion, the suset condition stimuli were shown for longer ( ms per item in the -item suset condition, 2 ms per item in the - item suset condition, ms per item in the 2-item suset condition,, ms per item in the -item suset condition). Across ll sequentilly presented displys (Experiments,, nd ), the ISI ws ms. In ll the experiments, fter the disply disppered, prticipnts were required to rte the verge lifelikeness of the stimuli. In Experiments, 2 nd 2, prticipnts used slider r to rte the lifelikeness of the stimuli. The slider r ppered fter the disply disppered, nd llowed the prticipnts to rte the nimcy of the disply. Ech end of the slider r lso hd written cues reminding the prticipnt tht represented the lowest nd represented the highest possile nimcy or lifelikeness. The scle ws integer sed (tht is, deciml rtings were not ville to the prticipnt). In Experiments,,,,,,, nd, prticipnts used keyord uttons to rte the lifelikeness of the stimuli, with representing the lowest lifelikeness nd representing the highest lifelikeness. In Experiment, prticipnts lso used keyord uttons to indicte whether the crowd with the highest verge lifelikeness ws displyed on the left or right side of the screen. Across ll experiments, prticipnts were not given time limit to mke their response. In ddition to sking prticipnts to rte the verge lifelikeness of stimuli, the memory experiments (Experiments nd ) lso included memory test during the response phse. Two ojects were presented side y side; one ws lure nd the other ws memer of the previously seen set. Prticipnts used keyord utton to indicte whether the memer of the set ws displyed the right or the left side of the screen. The loction (right or left) of the correct memer ws rndomized throughout the experiment. Prticipnts memory cpcity ws estimted using the following formul: MC ¼ I P, where MC represents working memory cpcity, I represents the numer of items in the set, nd P represents (proportion correct.) 2. Dt vilility. All relevnt dt re ville from the uthors. References. Noë, A., Pesso, L. & Thompson, E. Beyond the grnd illusion: wht chnge lindness relly teches us out vision. Vis. cogn., (2). 2. Alvrez, G. A. & Oliv, A. Sptil ensemle sttistics re efficient codes tht cn e represented with reduced ttention. Proc. Ntl Acd. Sci. USA, (2).. Prkes, L., Lund, J. & Angelucci, A. Compulsory verging of crowded orienttion signls in humn vision. Nt. Neurosci., (2).. Wtmniuk, S. N. J. & Duchon, A. The humn visul system verges speed informtion. Vision Res. 2, (2).. Hermn, J. & Whitney, D. Rpid extrction of men emotion nd gender from sets of fces. Curr. Biol., (2).. Sperer, D., Premck, D. & Premck, A. J. (eds). Cusl cognition: A multidisciplinry dete. (Clrendon Press, Oxford, UK, ).. Heerlien, A. S. in Understnding events: From perception to ction (eds Shipley, T. & Zcks, J.) (Oxford University Press, New York, NY, USA, 2).. Lng, P. J., Brdley, M. M. & Cuthert, B. N. Interntionl Affective Picture System (IAPS): Technicl Mnul nd Affective Rtings. (Ntionl Institute of Mentl Helth Center for the Study of Emotion nd Attention, Ginesville, FL, ).. Dsser, V., Ulek, I. & Premck, D. The Perception of Intention. Science 2, ().. Bker, C., Sxe, R. & Tenenum, J. Action understnding s inverse plnning. Cognition, 2 (2).. Tvres, P., Lwrence, A. D. & Brnrd, P. J. Pying ttention to socil mening: n fmri study. Cere. Cortex, (2). 2. New, J., Cosmides, L. & Tooy, J. Ctegory-specific ttention for nimls reflects ncestrl priorities, not expertise. Proc. Ntl Acd. Sci. USA, (2).. Vrtick, P., Snder, D. & Vuilleumier, P. Influence of dult ttchment style on the perception of socil nd non-socil emotionl scenes. J. Soc. Pers. Relt. 2, (22).. Grühn, D. & Scheie, S. Age-relted differences in vlence nd rousl rtings of pictures from the Interntionl Affective Picture System (IAPS): do rtings ecome more extreme with ge? Behv. Res. Meth., 2 2 (2).. Schupp, H. T. et l. Affective picture processing: the lte positive potentil is modulted y motivtionl relevnce. Psychophysiology, 2 2 (2).. Hermn, J. & Whitney, D. in From perception to consciousness: Serching with Anne Treismn (eds Wolfe, J. & Roertson, L.) (Oxford University Press, New York, NY, 22).. Mr, R. A. & Mcre, C. N. in Empthy nd Firness Novrtis Foundtion Symposium 2 (eds Bock, G. & Goode, J.) 2 (John Wiley nd Sons, Chichester, UK, 22).. Chong, S. C. & Treismn, A. Representtion of sttisticl properties. Vision Res., (2).. Hermn, J. & Whitney, D. Seeing the men: ensemle coding for sets of fces. J. Exp. Psychol. Hum. Percept. Perform., (2). 2. Sweeny, T. D., Hroz, S. & Whitney, D. Perceiving group ehvior: sensitive ensemle coding mechnisms for iologicl motion of humn crowds. J. Exp. Psychol. Hum. Percept. Perform., 2 (2). 2. McGrw, K. O. & Wong, S. P. Forming inferences out some intrclss correltion coefficients. Psychol. Meth., (). 22. Cicchetti, D. V. Guidelines, criteri, nd rules of thum for evluting normed nd stndrdized ssessment instruments in psychology. Psychol. Assessment, 2 (). NATURE COMMUNICATIONS : DOI:./ncomms