An Intuitive Model of Perceptual Grouping for HCI Design

Transcription

1 An Intuitive Model of Perceptul Grouping for HCI Design Ruth Rosenholtz MIT Cmridge, MA Nthniel R. Twrog MIT Cmridge, MA Ndj Schinkel-Bielefeld MIT Cmridge, MA Mrtin Wttenerg IBM Wtson Reserch Cmridge, MA ABSTRACT Understnding nd exploiting the ilities of the humn visul system is n importnt prt of the design of usle user interfces nd informtion visuliztions. Good design enles quick, esy nd veridicl perception of key components of tht design. An importnt fcet of humn vision is its ility to seemingly effortlessly perform perceptul orgniztion ; it trnsforms individul feture estimtes into perception of coherent regions, structures, nd ojects. We perceive regions grouped y proximity nd feture similrity, grouping of curves y good continution, nd grouping of regions of coherent texture. In this pper, we discuss simple model for rod rnge of perceptul grouping phenomen. It tkes s input n ritrry imge, nd returns structure descriing the predicted visul orgniztion of the imge. We demonstrte tht this model cn cpture spects of trditionl design rules, nd predicts visul percepts in clssic perceptul grouping displys. Author Keywords Perceptul orgniztion, grouping, good continution, proximity, similrity, Gestlt, contour integrtion. ACM Clssifiction Keywords H5.2. User interfces, theory & methods. INTRODUCTION Design of user interfces nd informtion grphics is poorly understood, nd somewht hit-or-miss in terms of effectiveness. A numer of issues influence the success of design, nd these run the gmut of the underlying humn ehvior. A design must e good cognitively (cn the user esily understnd the semntic structure of the design?), perceptully (cn they effortlessly interpret the visul informtion present in the design?), nd socilly (does the design fit into the user s workflow? will they wnt to use it?). Here we focus on perceptul spects of design. Permission to mke digitl or hrd copies of ll or prt of this work for personl or clssroom use is grnted without fee provided tht copies re not mde or distriuted for profit or commercil dvntge nd tht copies er this notice nd the full cittion on the first pge. To copy otherwise, or repulish, to post on servers or to redistriute to lists, requires prior specific permission nd/or fee. CHI 2009, April 4 9, 2009, Boston, MA, USA. Copyright 2009 ACM /09/04 $5.00. Perhps the most importnt spect of humn vision for design is perceptul orgniztion. Perceptul orgniztion refers to phenomen in which the visul system quickly nd seemingly effortlessly trnsforms individul feture estimtes into perception of coherent regions, structures, nd ojects. These phenomen were first studied in detil y the Gestlt psychologists, who produced set of qulittive Gestlt principles tht govern pttern perception [1, 2], including ut not limited to: the tendency of things to group if they re nery (the Gestlt lw of proximity); if they shre similr fetures (the lw of similrity), or re smooth nd continuous (the lw of good continution). The duls of perceptul grouping re importnt phenomen in their own right: we quickly nd effortlessly perceive oundries etween certin visul textures, perceive edges etween coherent regions in n imge, nd quickly detect unusul items tht seem to pop out from the ckground. Exmples of perceptul grouping phenomen re given in Figure 1. Following the visul system s rules of visul orgniztion mkes interprettion of visul spects of designs effortless: user esily sees which lels refer to which prts of di e c Figure 1: Perceptul grouping exmples, including grouping y proximity & similrity (, ), nd grouping y good continution (c, d). (e) A user interfce; wht is the percept? (f) A grph, from [5]. Will user perceive the trend of the dt? d f 1331

2 grm, notices trend in dt, nd mkes connections etween the overview nd the detil in mp. Good designs use the nturl perceptul processing power of the rin, nd interprettions of such designs re fst, roust to instruction, nd cross-culturl [3]. With poor visul design, the grouping structure my not mtch the structure of the informtion, leding to confusing displys [for exmples, see 4, 5]. A user might see columns of informtion where in fct there were intended to e rows; incorrectly group two regions of grphic tht hve no reltion to ech other, nd so on. In Figure 1f, do the + s t the grid points interfere with perceiving the dt curves? However, models for how users extrct mening from visul displys re incomplete. Designers often eyell it i.e. try to judge perceptul groupings using their own visul systems, perhps using tricks such s squinting t the design or viewing it from fr wy to ring out corser scle groupings. Vrious reserchers hve suggested generl guidelines for design [3, 5, 6, 7]. Mny existing models in use in the HCI nd informtion grphics field re specific to prticulr types of displys e.g. lphnumeric [8], text documents [9, for review], visul sic dilog oxes [10], nd sketch editing with vectorized input [11]. In ddition, some work hs een done to trnslte very sic rules of thum out prettentive (i.e. fst nd effortless) visul processing to the design of visuliztions [12]. Rules of thum, sed on simple ehviorl experiments, re useful in understnding nd guiding design. However, designers my hve difficulty pplying them to more complex displys. Idelly, one would prefer model tht could predict the likely perceptul groups for n ritrry design. Such model would e most useful if its mechnisms nd output were esy to understnd, s this trnsprency would id designer in mking chnges to poor design. In this pper, we drw on tools from sttistics s well s recent work in computer vision to propose model of perceptul grouping tht is simple to understnd nd implement, yet effective. This model cn predict imge segmenttion, contour integrtion, segmenttion of orienttionsed textures, grouping y similrity nd proximity in stndrd Gestlt displys, segmenttion of nturl imges, nd grouping in more complex digrms. PREVIOUS WORK A gret ody of computer vision work exists on the topic of perceptul grouping. However, much of it is indequte for user interfce (UI) designs nd informtion grphics. After mny yers of humn nd computer vision reserch, results of imge segmenttion models re still often quite poor. For one thing, mny of these models hve inherent ises ginst extended groups. A clssic result is the tendency to predict the perceptully invlid segmenttion of pure lue sky into 3-4 seprte regions. Furthermore, the etter models often do not esily lend themselves to intuitions. For exmple, the normlized cut lgorithm [13], which works firly well, trets imge segmenttion s grph prtitioning prolem, with ech pixel node. The similrity etween two pixels determines the weight etween the corresponding nodes. This weight cn e thought of s the tightness of spring. The lgorithm prtitions this spring-mss system into regions tht will tend to move independently. This is n evoctive description, ut it does not lend itself to esy intuitions out the predicted segmenttion, nor how one might chnge disply to otin different grouping. In informtion grphics, often interesting groupings form etween non-physiclly djcent items. We wnt to know, for instnce, whether user will esily perceive the ssocition etween colored lines on plot nd the colors in legend. Will it e ovious to user tht set of uttons on remote control perform relted functions (Figure 1e)? The vst mjority of computer vision lgorithms group only contiguous regions. It is uncler how well these lgorithms cn extend to group over gps etween items. This is serious prolem for pplying these lgorithms to UI designs. Contour integrtion lgorithms do group cross gps [14, 15, 16, 17], ut hve not een extended to grouping sed upon other Gestlt principles. Techniques tht cluster in luminnce or color spce, for exmple k-mens nd non-prmetric equivlents [18, 19], will group cross gps in spce. However, these techniques, s commonly used, tke this loosening of the proximity constrint too fr; they will tend to group independently of proximity. Another difficulty is tht typicl perceptul orgniztion models do not produce hierrchicl grouping, though see [20, 21, 22, 23, 9]. Something like hierrchicl percept clerly exists [24, 25]. In text document, for instnce, individul letters group to form lines of text, which form prgrphs, which form columns, nd so on. There hs een little work compring models of perceptul grouping with humn perception, though for prtil ttempts see, for exmple, [26, 27, 28]. In prt this is due to the lck of quntittive dt on perceptul orgniztion. More seriously, though, mny of the lgorithms perform too poorly to predict even wht qulittive dt exist. Predicting wide rnge of simple qulittive perceptul orgniztion ehvior would e significnt step forwrd. A more sutle difficulty with existing models of perceptul grouping is tht nerly ll hndle only one kind of perceptul grouping, e.g. they only find oundries etween regions of nturl imges [20, 21, 22, 26, 29, 30, 31, 32, 33], perform grouping y proximity [23, 9], segment textures [27, 34, 35, 36, 28], or do contour integrtion [14, 15, 16, 17]. Certinly if seprte models for the different grouping phenomen were required, we would use seprte models. However, we will demonstrte tht this is not necessry. Advntges of single, unifying model include the fct tht the outputs re comptile, nd thus it will e esier to comine them to get n overll picture of the perceptul structure of disply. Furthermore, the difficulty of getting 1332

3 intuitions out wht will group is significntly reduced if there is essentilly one model to understnd insted of four. The rin itself contins the est perceptul orgniztion system in existence, nd it is generlly elieved to use similr mechnisms to perform relted tsks. More recent models hve ecome more unified lrgely due to unified vision of the purpose of perceptul orgniztion. (Although see, for exmple [37], which cptures numer of perceptul orgniztion phenomen sed on unified view of the underlying neurl opertions.) This vision echoes tht of Helmholtz [38], who rgued tht wht we perceive is our mind's est guess of wht is in the world, sed on summry dt (the input imge) nd prior experience. By this rgument, the gol of perception is to crete wht the computer vision field refers to s genertive model for n imge: wht processes creted the imge, nd where ech of the processes opertes. Perceptul grouping concerns itself with the ltter. Bsed on this view of perceptul grouping, numer of computtionl models hve een developed for imge segmenttion [39, 13], edge detection including texture segmenttion [28, 26, 36, 40], contour integrtion [41, 42, 43, 44] nd distinguishing figure from ground [45]. Our model similrly finds groups y ttempting to infer the processes tht generted the disply. THE MODEL We motivte our model in detil, since the intuitions to e gined pply to design s well. The Representtion is Key Why is it difficult to predict perceptul groupings? Mny perceptul grouping prolems pper difficult to explin vi simple processing in the imge domin. Consider the Gestlt disk rry exmple in Figure 1. The typicl percept in this figure is of n rry of disks with three groups: the gry disks on the left, the white disks on the right, nd the ckground. It is this grouping we hope to mimic. In determining wht regions form groups, oth proximity, i.e. difference in (x, y), nd similrity, i.e. difference in feture (here, luminnce) vlue, re relevnt. Virtully ny simple lgorithm cn group the pixels in ech individul disk, since they re touching nd contin pixels with exctly the sme luminnce. The difficulty comes from vritions in luminnce, nd ridging the gps etween individul disks to group them in spite of the lck of sptil djcency. However, sptil informtion is lso importnt you do not wnt to group together ll pixels of the sme luminnce, regrdless of how fr prt they re. One possile solution is to lur the imge, then check whether this hs merged the disks [20, 21, 22, 23, 29, 30, 31]. See Figure 2. This is reltively old ide in imge processing. The lur mkes the lgorithm somewht roust to vrition in luminnce, nd lso llows grouping cross smll sptil gps. This pproch does resonly well t cpturing grouping y proximity when ll foreground items shre the sme contrst with the ckground. However, when items hve different contrsts, s in Figure 1, this pproch performs poorly. Blurring the imge not only merges neighoring regions, ut lso mixes the ckground luminnce with tht of the disks, s seen in Figure 2. This mixing mkes such methods overly sensitive to the difference etween foreground nd ckground. Proximl high contrst items will tend to group with ech other, wheres lower contrst items will tend to group with either neighoring high contrst items or with the ckground. Contrst does influence the sptil extent over which discontiguous items my group. However, lgorithms tht lur in c Figure 2: () Right side of Figure 1, t successive levels of lur (,c). Note tht the disks merge in (c). spce overestimte this influence, mking mid-gry disks on lck ckground fr less likely to group with ech other thn white disks, nd this is not the cse. Consider insted the exmple of Figure 1c. The typicl percept is two smooth curves tht cross in the middle. Here the pixels tht form groups re contiguous; how do we seprte those pixels into two seprte curves, given tht they touch nd re the sme color? Agin, this is tricky in the imge domin. We need to split the imge into two contours in spite of the sptil proximity. Where the two curves cross, the locl difference in orienttion is sufficient to suggest percept of two contours. The key in oth of these exmples is to represent the originl imge in higher dimensionl spce tht incorportes oth spce nd feture dimensions. For exmple, Figure 3 shows the representtion of Figure 1 in (x, y, L) spce, where L corresponds to mesure of luminnce, e.g. the L component of CIEL** [46]. This representtion is n indictor function, with 1 s t ll points (x, y, L) such tht there exists point in the originl imge with the given (x, y) loction nd the given luminnce, L. The representtion hs 0 s t ll other points in the 3-D spce. (Note: in Figure 3 we continue to show elements in their originl luminnce, to clrify the reltionship to the originl imge.) This representtion in higher dimensionl feture spce explicitly incorportes, in wy tht the originl imge did not, the relevnce of oth sptil proximity nd feture similrity. One cn imgine tht simple grouping lgorithm could esily find the desired perceptul groups. The gry disks lie ner ech other nd not ner nything else. One cn mke similr oservtion out the white disks, nd out the ckground. If one lurred in this higherdimensionl spce, the gry disks would merge to form 1333

4 Figure 3: () Representtion of Figure 1 in (x, y, L) spce. () A slice y=y 0 through this spce (old lines), superimposed over the sme slice, lurred. Hshed lck lines show lo oundries, which define 3 groups. single group, the white disks nother, nd the ckground third. Note lso tht the predicted groupings will e quite roust to the mount of lur chosen. We hve moved from complicted 2D prolem to seemingly more trctle 3D prolem. The right representtion cn mke perceptul orgniztion seem much esier. A higher-dimensionl representtion like this mkes imge smoothing computtionlly simpler [33], nd representtion in oth sptil coordintes nd 3-dimensionl color coordintes [e.g. 47, 39] forms the sis for n imge segmenttion technique known s men-shift. In the cse of contour integrtion (e.g. Figure 1cd), orienttion is the key feture. The pproprite representtion uses (x, y, θ) spce, where θ is the locl orienttion estimte (Figure 4). In (x, y, θ) the two curves of Figure 1c do not even come ner ech other, mking it lmost impossile to group them in ny wy other thn the desired percept. Agin, the right representtion cn mke perceptul grouping prolem much esier. We use steerle filtering [48] to extrct the est locl orienttion t given scle. The technique descried y [49] returns, t ech scle, two imges, pproximtely corresponding to kcos2θ nd ksin2θ, where k indictes the strength or confidence in the orienttion. Regions with strong, single orienttions yield vlues of k ner 1, wheres regions with poor contrst or multiple orienttions, such s corners, yield vlues closer to 0. We mp ech pixel to point (x, y, θ), where θ is the estimted orienttion t pixel (x, y). However, rther thn plcing simply 1 t point (x, y, θ), we insted put the weight, k. This cuses our contour integrtion module to ignore unoriented or wekly oriented regions in the imge. Figure 4: Representtion of Figure 1c in (x, y, θ) spce, from two viewing ngles. A Simple Blurring Merges Regions Into Groups A good representtion is key to mking perceptul grouping trctle. Given this representtion, how do we decide wht groups? Wheres lurring in the imge domin hs undesirle effects, in the higher-dimensionl spce, simple lurring performs well t joining seprte elements of perceived group into single lo. It does so in wy tht mimics humn perception. Figure 3 shows crtoon exmple of this lur through slice of the disk rry exmple of Figure 1. Note tht the lur correctly joins the light gry disks into single lo, nd the sme for the drk gry disks nd for the ckground. Additionl lurring is not necessry for finding groups in the curve-crossing exmple of Figures 1c nd 4, ut the point of contour integrtion is often to connect seprted contour elements, s in the more complicted contour integrtion exmple of Figure 1d. Here lurring in (x, y, θ) spce helps to join the seprted contour elements. Note tht in Figure 3, we lur in oth the feture dimension, L, nd in the sptil dimension, x. One cn think of lurring in the feture dimension s joining regions tht re sufficiently similr in feture. Fetures, such s luminnce, need to differ y certin mount in order to e perceived s different y humn oserver. If the lur we pply in the model is smller thn this just noticele difference, prts of the imge whose fetures re for humns not distinguishle my e incorrectly segmented. This sets lower limits on the mount of feture lur. Edges of the Blos Define Perceptul Groups Blurring leds to los in the higher-dimensionl representtion. Neighoring regions in (x, y, feture) spce will merge into single lo, corresponding to predicted perceptul group. Incresing the lur leds to igger los, nd lrger-scle grouping of more disprte regions. Therefore, simply chnging the lur from smll to lrge produces hierrchy of groupings (this my not produce strict hierrchy, ut s [23] points out, neither does perception). The lst step involves finding the oundries of coherent los, nd thus leling groups. This is chllenge. Perceptul groups tend to correspond to rod, firly flt regions in the higher-dimensionl spce. One wnts to find the oundries of those regions while ignoring smller umps in the los. Such smller, noisy umps cn occur 1334

5 due to feture vriility within group, nd due to discontiguous groups. Finding meningful oundries while ignoring noisy umps is the clssic prolem of edge detection in computer vision. A numer of existing edge detection lgorithms generlize well to our higher-dimensionl spce. We use roust version of the Mrr & Hildreth [50] edge detection lgorithm. Informl testing suggests tht this method works well cross rnge of perceptul grouping exmples, nd is resonly roust to choice of prmeters. At this point, it is worth mentioning the reltionship of our model to two relted suggestions for finding perceptul groups or imge segmenttions. Logn [51] provided thumnil sketch of n lgorithm for grouping y similrity nd proximity. Though his description differs gretly from ours, he effectively suggested estimting surfce much like our lurred, higher-dimensionl representtion, which he clled the CODE surfce, then looking t slice L = L 0. Groups would derive from finding connected components ove some threshold, T, within tht slice. Logn suggested exmining multiple vlues of L 0 nd T to extrct grouping hierrchy. This differs from our proposed model oth in looking t only one slice of the (x, y, feture) representtion t time, nd in tht it finds los y thresholding rther thn edge detection. As result, his suggestion is fr more sensitive to choice of prmeters thn our model. Furthermore, given choice of prmeters L 0 nd T will not tend to select meningful groups throughout the imge. Our method lso requires choice of prmeters, ut degrdes more grcefully, s the necessry threshold for Logn will depend upon the unknown group size, nd operting slice t time is n unnturl nd less relile wy of understnding the full higher-dimensionl lo structure. Also closely relted is computer vision technique for segmenting color imges, known s men-shift [47]. Menshift typiclly represents the imge in (x, y, color) spce, with color typiclly in CIEL*** color coordintes. Men-shift then lurs with seprle Gussin kernel (mening one cn lur first with 1-D Gussin in x, then, 1-D lur in y, then L, nd so on). It finds groups y finding peks of the resulting function, nd points ssocited with ech pek. This technique hs proven resonly effective nd efficient t segmenting nturl scenes [39], ut hs not een pplied to contour integrtion or perceptul grouping of disjoint regions. Its relince on finding peks in the higher dimensionl representtion mkes it highly sensitive to the noisy umps we wish to ignore. In prctice, men-shift hs difficulty grouping rod, homogeneous res into single group. This is prolemtic enough in nturl imges, in which men-shift often produces very roust over-segmenttions tht re perceptully invlid. In informtion visuliztions or user interfce designs, in which lrge flt regions ound, nd groups my not e sptilly contiguous, men-shift will likely hve difficulty deriving perceptully vlid groups. Further Discussion of the Model Our perceptul grouping lgorithm, then, represents the imge in higher dimensionl (x, y, feture) spce, lurs to merge regions into coherent los, nd performs edge detection to find meningful groups. How cn this lgorithm perform grouping y similrity nd proximity, nd contour integrtion? Interpreting the lgorithm in terms of the sttisticl gol of the visul system gives us clue. The visul system likely ims to infer genertive model of the imge: wht processes produced the imge, nd where they operte. Figure 1, for instnce, cn e thought of s generted y one rndom process, ctive on the left side of the imge, which produced drk gry disks, one rndom process tht produced light gry disks on the right, nd one process tht produced the drk ckground. (Very little is rndom out these processes, ut the visul system evolved to hndle the rndomness inherent in nturl imges, e.g. of trees, leoprd skin, grss, etc.) To extrct this interprettion of the imge, one could gther smples from throughout the imge, to estimte the distriution of fetures cross spce. This distriution is the proility density function (pdf) tht generted the imge. In this cse, the pdf hs 3 modes, corresponding to the 3 groups. Our representtion of the imge in (x, y, feture) spce cn e thought of s histogrm estimte of the underlying pdf, with ech point in the imge corresponding to smple. Blur in the feture dimensions prllels stndrd sttisticl technique for etter estimting the pdf, known s Przen windowing. In sttistics, choice of the mount of lur is clssic unsolved prolem. Too much lur overly smoothes the pdf estimte, which cn led to underestimtes of the numer of modes. Too little lur leds to noisy pdf estimte, with too mny modes. Wht the rin my do is try rnge of lur, nd construct hierrchy of possile groups, from fine segmenttion sed on smll mounts of lur through corser segmenttion sed on lrger mounts of lur. Perhps we hve hierrchicl grouping percept ecuse the rin is not sure of the est prmeters. Blurring the pdf estimte in the (x, y) dimensions is justified sed on the prior tht processes in nturl scenes tend to e loclized, so tht if one point in scene cme from given process, tht sme process likely lso generted neighoring points. Blurring in the (x, y) dimensions in effect llows us to improve our pdf estimte y collecting more smples, likely to e from the sme process, from neighoring res of the imge. The lur in (x, y), then, should e relted to the proility of finding pixels from the sme process t given distnce from the current pixel. Our exmintion of hnd-leled groups in nturl imges (dt from [26]) suggests tht lur for grouping y similrity nd proximity should e pproximtely isotropic. Groups in the world do not, y nd lrge, tend to extend more in one direction thn nother, though there is slight preference for horizontl nd verticl directions. Furthermore, lur should e independent of 1335

6 the locl feture vlue, since e.g., righter regions re not, in generl, lrger thn drk regions. We pproximte these constrints with 2-D seprle, isotropic Gussin lur. A different lur my e more pproprite for informtion grphics, given how they differ from nturl scenes. However, it is often ssumed tht perceptul orgniztion processes in the rin re dpted to processing nturl imges. For contour integrtion (Figures 1c & 4), other priors re more pproprite. Contour segments more likely come from the sme process when they re roughly co-circulr, with preference for low curvture "circles", i.e. for colliner segments [41, 14]. Here we ssume simple prior, in which locl orienttion estimte of α t point (x, y) indictes tht we should look for dditionl smples from the sme process in roughly direction α. This mens in prctice tht for ech slice θ=α in (x, y, θ) spce, we lur with n nisotropic Gussin oriented t α degrees. We hve found tht n elongted Gussin with n spect rtio of t lest 10 works quite well. Though this explicitly prefers colliner contours, in prctise it does surprisingly well t joining roughly co-circulr contour segments (n oservtion lso recently mde y [52]). RESULTS In this section, we show results of our lgorithm, first on stndrd Gestlt displys, then on more complicted informtion grphics. In wht follows, we present only the results y the grouping module(s) most pplicle to the given disply. We discuss future work on comining the results in the Discussion. Ech predicted group is represented y colored lo or los superimposed over the originl imge. These grouping imges re est viewed in color. Grouping By Proximity nd Similrity Figure 5 displys the grouping-y-luminnce-similrity predictions on severl simple Gestlt disk rrys. In Figure 5 the verticl spcing etween disks is significntly less thn the horizontl spcing. Becuse of this, the typicl hierrchicl percept is individul disks, which group into 5 columns, nd, on lrger scle, simply n rry of disks. It is difficult to perceive rows of disks. Our lgorithm correctly predicts this result. (The fine scle percept of individul disks is predicted, ut not shown.) Figure 5d shows similr Gestlt rry, in which rows of disks lternte in luminnce. The similrity of disks within row, nd the difference cross rows, overrides the column percept of grouping y proximity. We perceive rows of disks, nd it is difficult to see columns, s predicted y the lgorithm. (The corse scle percept of n rry of disks is predicted ut not shown.) Grouping Contours y Good Continution Figure 6 displys good continution predictions on severl stimuli. Figure 6 shows Venn digrm, in which the ovious percept is of two circles overlpping, s predicted y our lgorithm. Note tht neither top-down knowledge of circle is required, nor preference for closed contours. Figure 6c shows stndrd contour integrtion stimulus. Figure 6d shows the contours most slient to humn oservers, from [41]. Figure 6e shows the most slient contours predicted y our lgorithm. Overll, the lgorithm's performnce is quite similr to the expected perceptul orgniztion. In pilot experiments we hve tested our lgorithm more quntittively ginst humn performnce t detecting, in less thn 200 ms, whether the lrgest contour is on the left or right of fixtion. Our model performs quite well t predicting the results of these experiments. Text nd Texture Exmples Textures, i.e. ptterns of orienttion, size, contrst, etc., re often used to visulize high dimensionl dt, ecuse of the ese with which the visul system notices similrities nd differences in such ptterns. Figure 7 shows the results of grouping y similr orienttion in texture. Note tht the lgorithm correctly predicts the percept. Figure 5: Grouping y luminnce similrity. () & (d) Originl imges (500x500 pixels), with dominnt column nd row percept, respectively. () & (e) Results of the lgorithm on () & (d), respectively. Sptil lur σ = 20 pixels, luminnce lur σ L = 4% of the rnge of luminnce vlues. (c) & (f) show results of the lgorithm for σ = 32 pixels, σ L = 4%. Figure 6: () Venn digrm. () Contour grouping results on (). (c) Field of oriented elements demonstrting grouping y good continution, from Geisler et l (2001). (d) Contours selected y Geisler et l. s eing present in (c). (e) Most slient (longest) contours from our contour grouping lgorithm. 1336

7 CHI 2009 ~ Cognitive Modeling nd Assessment Figure 9 shows vritions of plot y Puling. In Figure 9 one cn clerly recognize the shpe of the curve, s pointed out in [5] nd shown y our lgorithm (). In Figure 9c, the dshed lines hve een removed, nd it is hrd to see the peks in the grph s the dots re quite scttered. In 9e, Tufte hs inserted crosses in the ckground to fcilitte reding of the ctul vlues of the dt, t the cost of modest mount of dditionl clutter. The curves re still cler, s predicted y the perceptul orgniztion lgorithm. Figure 7: Segmenttion results () on texture defined y line orienttion (). Oservers cn rpidly loclize the segmenttion oundry in () [28]. A specil kind of texture for UI design is text. Figure 8 shows our predicted hierrchicl segmenttion of simple prgrph of text. As we progress up the hierrchy, we predict the percept of letters, then words, then lines, nd finlly the entire prgrph long with its heder. Informtion Grphics The previous grouping exmples re highly relevnt to designers, s similr groupings occur in UI designs nd informtion visuliztions. Even more relevnt, we here test the lgorithm on severl informtion grphics from Tufte [5]. These exmples re prticulrly interesting, since we hve Tufte s description of the true percept. Figure 10 shows section of Mrey s trin schedule. The verticl xis represents loction, i.e. trin sttions etween Pris nd Lyon. The horizontl xis denotes time. Digonl lines indicte key informtion: trins trveling from one sttion to nother over time. The version with the lowcontrst grid is much esier to red [5]. This is lso reflected in the model prediction; for the high-contrst grid, the model finds mny grid lines ut not ll the digonl lines representing the trins. For the low contrst grid ll digonl lines re found, ut not the grid lines. Stops of trins result in offsets of the digonl lines. However, if they re smll, they do not lter grouping into single trin. c d e f c d e Figure 8: () Originl text, long with the lgorithm s predicted hierrchicl grouping of letters (), words (c), lines (d), nd heder grouped with lower prgrph (e). Figure 9: Vritions on grph (, c, e) from [5], long with the predicted contours (, d, f, respectively). 1337

8 CHI 2009 ~ Cognitive Modeling nd Assessment c Figure 12: () Remote control. () Grouping y (color) similrity nd proximity. d DISCUSSION & CONCLUSIONS Figure 10: Sections from two vritions of Mrey s [5] trin schedules (,c) nd the contour integrtion results (,d). Figure 11 shows cncer rtes mong white femles, y U.S. county. Figure 11 shows the grouping-y-proximity predicted y our lgorithm. The groups found gree well with Tufte s expliction of the percept: cluster of high cncer rtes in the Northest, with dditionl outlier clusters in southern Cliforni nd northern Minnesot. With our model in hnd, one could hve predicted numer of Tufte s oservtions on these designs. Clerly this should e useful for designers, s it enles generliztion of design rules to ritrry nd complex displys. Figure 12 shows finl exmple of n erly version of the lgorithm tht uses color s feture for grouping y similrity. Buttons of similr color re predicted to group if they re sufficiently close to ech other. We hve presented conceptully simple model for perceptul grouping. The key ide is to trnslte complicted two-dimensionl imge, in which segmenttion is difficult, into higher-dimensionl representtion where strightforwrd methods yield good results. Our prticulr technique uses high-dimensionl lur opertion, which is simple to implement nd understnd. We my exmine hierrchy of groupings y vrying the degree of lur. In this work, we hve extrcted fetures, e.g. orienttion, t single scle. The visul system is known to extrct fetures t multiple scles, nd in future versions we will incorporte this. This is kin to stnding frther wy from disply, prior to extrcting groups. Extrcting fetures t multiple scles is not equivlent to chnging the extent of the lur in (x, y, feture) spce, though in some cses they my hve similr effects on the predicted groupings. We hve focused on model sed on luminnce nd orienttion. A long line of reserch on the sic fetures influencing perceptul orgniztion [53], however, suggests mny other cndidte fetures. Figure 11: () Cncer rtes for white femles, from [5]. () Groupings found y the lgorithm. We hve djusted the sturtion in the results to emphsize interesting groups. One issue in integrting multiple fetures is how to generlize the lur opertion. For exmple, we would clerly like to model the effect of color, ut it is not ovious how one should lur in the color dimensions. A unit step nywhere in CIEL** spce hs pproximtely the sme perceptul discriminility, yet it seems unlikely tht isotropic, seprle filtering in L, *, * is optiml, s it implies tht the likelihood of grouping two colors together depends only upon their discriminility, not on whether they differ in hue, sturtion, or luminnce. In the nturl world, luminnce vries within given oject due to shding, wheres hue chnges occur less frequently. Such distinctions my well e present (whether lerned or hrdwired) in our perceptul system, Thus work on sttistics of segmenttion of nturl imges [e.g. 26, 54] my lend insight into how est to lur in color spce. Other likely fetures include some mesure of size or shpe. A simple stnd-in my e contrst energy, s in [55, 27, 28], though ultimtely more complicted mesure of shpe my e required. An open question generliz- 1338

9 tion of the issue rised ove with dimensions of color concerns how these multiple fetures should e comined. One might, for instnce, comine ll the fetures into feture vector, nd find groups using single high-dimensionl representtion (x, y, θ, L, *, *, ). However, this might depend upon whether such dimensions re integrle or seprle, s discussed in [3]. Groupings sed on different fetures might require more complicted comintion rules, since [56] shows tht when color nd geometric form led to different texture segmenttions, color domintes. A relted open question is how one should comine the results from, sy, grouping y similrity, with grouping y good continution. Contour integrtion my serve to fix under-segmenttion when fetures of neighoring items re too similr, nd we re exploring using our predicted contours to modulte the results of grouping y similrity. Finlly, some perceived groupings hve greter strength thn others, mening they re more likely to e perceived. We re currently exploring, with some success, mesure of grouping strength sed upon how stle group is to chnges in lgorithm prmeters. Top-down influences re no dout lso importnt, though our current lgorithm does quite it without them. We might incorporte top-down effects in our frmework y rewrding groups tht mtched fmilir shpes, either in the imge or in the higher-dimensionl representtion. Our simple lgorithm works well t predicting grouping in Gestlt displys, s well s informtion visuliztions like digrms. This lgorithm, nd the intuitions ssocited with it, should e of use to designers wishing to ensure tht the structure of the informtion presented grees, s ner s possile, with the likely perceptul structure of disply. ACKNOWLEDGMENTS Thnks to Johnnes Wolter for help with color segmenttion. N.S-B ws funded y Germn Acdemic Exchnge Service (DAAD) fellowship nd y Foxconn. REFERENCES 1. Wertheimer, M. Lws of Orgniztion in Perceptul Forms. Hrcourt Brce Jovnovich, London, Koffk, K. Principles of Gestlt Psychology. Hrtcourt, New York, Wre, C. Informtion Visuliztion: Perception for Design. Elsevier, Sn Frncisco, Kosslyn, S. M. Understnding chrts nd grphs. Applied Cognitive Psychology, 3 (1989), Tufte, E. R. The Visul Disply of Quntittive Informtion. Grphics Press, CT, Clevelnd, W. The Elements of Grphing Dt. Wdsworth, Monterey, CA, Mckinly, J. D. Applying theory of grphicl presenttion to the grphic design of user interfces. In Proc. SIGGRAPH (UIST) (1988), ACM Press, Tullis, T. S. A computer-sed tool for evluting lphnumeric displys. INTERACT 84 (1984), Vincent, L. Current topics in pplied morphologicl imge nlysis. In Kendll, W. S. et l., eds., Current Trends in Stochstic Geometry nd Its Applictions. Chpmn & Hll, Shneidermn, B., Chimer, R., Jog, N., Stimrt, R., & White, D. Evluting sptil nd texture style of displys. In McDonld, L. W. nd Lowe, A. C., eds., Disply Systems: Design nd Applictions. John Wiley & Sons, Chichester, U.K., Sund, E. Symolic construction of 2-D scle-spce imge. IEEE Trns. Pttern Anlysis & Mchine Intelligence, 12, 8 (1990), Heley, C.G., Booth, K.S., & Enns, J.T. Hrnessing prettentive processes for multivrite dt visul-iztion. Proc. Grphics Interfce '93 (1993), Shi, J. & Mlik, J. Normlized cuts nd imge segmenttions. IEEE Conf. on Computer Vision nd Pttern Recognition ( 1997), Field, D. J., Hyes, A., & Hess, R. F. Contour inte-grtion y the humn visul system: evidence for locl ssocition field. Vision Reserch, 33 (1993), Prent, P. & Zucker, S. W. Trce inference, curvture consistency, nd curve detection. IEEE Trns. Pttern Anlysis & Mchine Intelligence, 2, 8 (1989), Mumford, D. Elstic nd computer vision. In Algeric Geometry nd Its Applictions. Springer-Verlg, NY, Grosserg, S. & Mingoll, E. Neurl dynmics of perceptul grouping: textures, oundries, nd emergent segmenttions. Perception & Psychophysics, 38 (1985), Dud, R. O. & Hrt, P. E. Pttern Clssifiction. Wiley, NY, Ohlnder, R., Price, K., & Reddy, R. Picture segmenttion y recursive region splitting method. Computer Grphics nd Imge Processing, 8 (1978), Witkin, A. P. Scle-spce filtering. Proc. 8th Int. Conf. on Artificil Intelligence (1983), Koenderink, J. J. The structure of imges. Biologicl Cyernetics, 50 (1984), Lindeerg, T. Scle-Spce Theory in Computer Vision. Kluwer Acdemic, Dordrecht, Netherlnds, Wttenerg, M. & Fisher, D. A model of multi-scle 1339

10 perceptul orgniztion in informtion grphics. Infovis 2003 ( 2003), Plmer, S. E. Hierrchicl structure in perceptul representtion. Cognitive Psychology, 9 (1977), Nvon, D. Forest efore trees: The precedence of glol fetures in visul perception. Cognitive Psychology, 9 (1977), Mrtin, D., Fowlkes, C., & Mlik, J. Lerning to detect nturl imge oundries using locl rightness, color, nd texture cues. IEEE Trns. Pttern Anlysis nd Mchine Intelligence, 26, 5 (2004), Mlik, J. & Peron, P. A computtionl model of texture segmenttion. Proc. Computer Vision nd Pttern Recognition (1989), Rosenholtz, R. Significntly different textures: A computtionl model of pre-ttentive texture segmenttion. Proc. Europ. Conf. Computer Vision (2000), Peron, P. & Mlik, J. Scle-spce nd edge detection using nisotropic diffusion. IEEE Trns. Pttern Anlysis & Mchine Intelligence, 12 (1990), Gemn, S. & Gemn, D. Stochstic relxtion, Gis distriutions, nd the Byesin restortion of imges. IEEE Trns. Pttern Anlysis & Mchine Intelligence, 6 (1984), Tomsi, C. & Mnduchi, R. Bilterl filtering for gry nd color imges. IEEE Int. Conf. on Computer Vision (1998), Durnd, F. & Dorsey, J. Fst ilterl filtering for the disply of high-dynmic-rnge imges. ACM Trns. on Grphics, 21, 3 (2002), Pris, S. & Durnd, F. A fst pproximtion of the ilterl filter using signl processing pproch. Proc. Europen Conf. Computer Vision ( 2006), Julesz, B. A theory of prettentive texture discrimintion sed on the first order sttistics of textons. Biologicl Cyernetics, 41 (1981), Beck, J. Texturl segmenttion, second-order sttistics, & texturl elements. Biol. Cyern., 48 (1983), Voorhees, H. & Poggio, T. Computing texture oundries from imges. Nture, 333 (1988), Li, Z. Pre-ttentive segmenttion in the primry visul cortex. Sptil Vision, 13 (2000), Helmholtz, H. Hndook of Physiologicl Optics. Vol. 3, The Perceptions of Vision. Opticl Society of Americ, Rochester, Pris, S. & Durnd, F. A topologicl pproch to hierrchicl segmenttion using men shift. IEEE Conf. Computer Vision & Pttern Recognition ( 2007), Ruzon, M. A. & Tomsi, C. Edge, junction nd corner detection using color distriutions. IEEE Trns. Pttern Anlysis & Mch. Intell., 23, 11 (2001), Geisler, W. S., Perry, J. S., Super, B. J., & Gllogly, D. P. Edge co-occurrence in nturl imges predicts con-tour grouping performnce. Vision Reserch, 41 (2001), Estrd, F. J. & Elder, J. H. Multi-scle contour extrction sed on nturl imge sttistics. IEEE Conf. Computer Vision & Pttern Recognition Workshop (2006), Schinkel-Bielefeld, N. Contour integrtion models predicting humn ehvior. University of Bremen, Mhmud, S., Willims, L. R., & Thorner, K. K. Segmenttion of multiple slient contours from rel imges. IEEE Trns. on Pttern Anlysis & Mchine Intelligence, 25 (2003), Ren, X., Fowlkes, C., & Mlik, J. Figure/ground ssignment in nturl imges. Proc. Europen Conf. on Computer Vision ( 2006), C.I.E. Recommendtions on uniform color spces, color difference equtions, psychometric color terms. Supp. No. 2 to CIE pul. 15 (E ) 1971/(TC-1.3.) (1978). 47. Comnciu, D. & Meer, P. Men shift: roust pproch towrd feture spce nlysis. IEEE Trns. Pttern Anlysis & Mchine Intelligence, 24 (2002), Freemn, W. T. & Adelson, E. H. The design nd use of steerle filters. IEEE Trns. Pttern Anlysis & Mchine Intelligence, 13 (1991), Lndy, M. S. & Bergen, J. R. Texture segregtion nd orienttion grdient. Vis. Reserch, 31 (1991), Mrr, D. nd Hildreth, E. C. Theory of edge detection. Proc. Royl Society, London B, 207 (1980), Logn, G. The CODE theory of visul ttention: An integrtion of spce-sed nd oject-sed ttention. Psychologicl Review, 103, 4 (1996), Wtt, R., Ledgewy, T., nd Dkin, S. C. Fmilies of models for Gor pths demonstrte the importnce of sptil djcency. Journl of Vision, 8, 7 (2008), htp://journlofvision.org/8/7/23/. 1340