THE ENCODING OF PARTS AND WHOLES

THE ENCODING OF PARTS AND WHOLES IN THE VISUAL CORTICAL HIERARCHY JOHAN WAGEMANS LABORATORY OF EXPERIMENTAL PSYCHOLOGY UNIVERSITY OF LEUVEN, BELGIUM DIPARTIMENTO DI PSICOLOGIA, UNIVERSITÀ DI MILANO-BICOCCA, MARCH 18, 2013

Some examples

Aviezer, Trope & Todorov (2012). Body cues, not facial expressions, discriminate between intense positive and negative emotions. Science, 338(6111), 1225-1229.

Visual hierarchy features parts wholes, e.g. objects faces bodies scenes

Cortical hierarchy: Mainstream view based on single-unit recordings (Hubel & Wiesel) tuning properties of different types of cells in different areas of the brain functional specialization hierarchical organization confirmed in human fmri (modules, maps) standard view in several approaches Felleman, D. J., & Van Essen, D. C. (1991). Distributed hierarchical processing in the primate cerebral cortex. Cerebral Cortex, 1, 1-47. Grill-Spector, K., & Malach, R. (2004). The human visual cortex. Annual Review of Neuroscience, 27, 649-677. Serre, T., Oliva, A., & Poggio, T. (2007). A feedforward architecture accounts for rapid categorization. Proceedings of the National Academy of Science of the USA, 104, 6424-6429.

Felleman, D. J., & Van Essen, D. C. (1991). Distributed hierarchical processing in the primate cerebral cortex. Cerebral Cortex, 1, 1-47.

Grill-Spector, K., & Malach, R. (2004). The human visual cortex. Annual Review of Neuroscience, 27, 649-677.

Serre, T. et al. (2007). A feedforward architecture accounts for rapid categorization. PNAS, 104, 6424-6429.

Alternative views possible Gestalt theory (Berlin school: Wertheimer, Köhler, Koffka) wholes are not only more than the sum of the parts wholes are also different from the sum of the parts 2-sided dependency wholes come first (e.g., global precedence effect) more recent views Hochstein, S., & Ahissar, M. (2002). View from the top: Hierarchies and reverse hierarchies in the visual system. Neuron, 36, 791-804. Bar, M. et al. (2006). Top-down facilitation of visual recognition. Proceedings of the National Academy of Science of the USA, 103, 449-454. interesting characteristics from viewpoint of Gestalt theory wholes come first highly interactive highly dynamic

Hochstein, S., & Ahissar, M. (2002). View from the top: Hierarchies and reverse hierarchies in the visual system. Neuron, 36, 791-804.

Bar, M. et al. (2006). Top-down facilitation of visual recognition. PNAS, 103, 449-454.

The problem How to understand the relationships between parts and wholes in visual experience How to understand the encoding of parts and wholes in the hierarchy of visual cortex How to understand the relationships between cortical encoding and visual experience

My proposal There are different types of Gestalts with their own relationships between parts and wholes, both in visual experience and in their neural encoding Some Gestalts seem to be encoded in low-level areas based on feedback from higher-order regions Kourtzi, Z., Tolias, A. S., Altmann, C. F., Augath, M., & Logothetis, N. K. (2003). Integration of local features into global shapes: Monkey and human fmri studies. Neuron, 37, 333-346. Murray, S. O., Boyaci, H., & Kersten, D. (2006). The representation of perceived angular size in human primary visual cortex. Nature Neuroscience, 9, 429-434.

My proposal There are different types of Gestalts with their own relationships between parts and wholes, both in visual experience and in their neural encoding Some Gestalts seem to be encoded in lower-level areas based on feedback from higher-level areas Other Gestalts seem to be encoded in higher-level areas, while the parts are encoded in lower-level areas without suppression of the parts with suppresion of the parts

Preservative versus eliminative Gestalts 1. preservative Gestalts functional wholes arise spontaneously and parts become less functional but the encoding of these wholes at higher levels of the cortical hierarchy does not suppress the encoding of the parts 2. eliminative Gestalts wholes dominate and parts disappear from experience wholes emerge in higher areas of the brain and encoding of parts is then suppressed

Preservative Gestalts excellent example: configural-superiority effect Pomerantz et al. key papers: Pomerantz, J. R., Sager, L. C., & Stoever, R. J. (1977). Perception of wholes and their component parts: Some configural superiority effects. Journal of Experimental Psychology: Human Perception and Performance, 3, 422-435. Pomerantz, J. R., & Portillo, M. C. (2011). Grouping and emergent features in vision: Toward a theory of basic Gestalts. Journal of Experimental Psychology: Human Perception and Performance, 37, 1331-1349. neural basis?

Kubilius et al. (2011) Kubilius, J., Wagemans, J., & Op de Beeck, H. P. (2011). Emergence of perceptual Gestalts in the human visual cortex: The case of the configural superiority effect. Psychological Science, 22(10), 1296-1303. behavioral results fmri decoding results

Behavioral results parts corner whole

Scanning protocol

fmri results: Retinotopic mapping

MVPA: decoding

fmri results: decoding

Discussion behavioral configural-superiority effect neural configural-superiority effect: better coding of wholes than parts in higher shapeselective regions better coding of parts than wholes in lower-level retinotopic regions general conclusions: at least some Gestalts emerge only at higher stages of visual information processing feedforward processing may be sufficient to produce some Gestalts

Two examples of eliminative Gestalts Motion silencing Suchow, J. W., & Alvarez, G. A. (2011). Motion silences awareness of visual change. Current Biology, 21(2), 140-143. doi:10.1016/j.cub.2010.12.019 Poljac*, E., de-wit*, L., & Wagemans, J. (2012). Perceptual wholes can reduce the conscious accessibility of their parts. Cognition, 123, 308-312. (*joint first authors) doi:10.1016/j.cognition.2012.01.001 Bistable diamond Fang, F., Kersten, D., & Murray, S. O. (2008). Perceptual grouping and inverse fmri activity patterns in human visual cortex. Journal of Vision, 8(7):2, 2-9. doi: 10.1167/8.7.2 de-wit, L. H., Kubilius, J., Wagemans, J., & Op de Beeck, H. P. (2012). Bi-stable Gestalts reduce activity in the whole of V1 not just the retinotopically predicted parts. Journal of Vision, 12(11):12, 1-14. doi:10.1167/12.11.12

Suchow & Alvarez (2011) Suchow, J. W., & Alvarez, G. A. (2011). Motion silences awareness of visual change. Current Biology, 21(2), 140-143. doi:10.1016/j.cub.2010.12.019 Best Illusion of the Year 2011

Demonstration

More demonstrations

Methods Stimuli: 100 dots first stationary, then rotating back and forth for 30 2 phases alternating every 3 s Task: observers had to adjust the rate of change during the stationary phase to match the apparent rate of change in the moving phase rate of change ( silencing factor ) between 0.1 (static perceived as changing slower) and 10 (static perceived as changing faster)

Results

Interpretation Suchow & Alvarez: local mechanisms with small receptive fields because a fast-moving object spends little time at any one location, a local detector is afforded only a brief window in which to assess the changing object alternative interpretation: objecthood when a good whole is formed, the details of the parts are fundamentally less accessible to conscious perception

Our study Poljac*, E., de-wit*, L., & Wagemans, J. (2012). Perceptual wholes can reduce the conscious accessibility of their parts. Cognition, 123, 308-312. (*joint first authors) doi:10.1016/j.cognition.2012.01.001 motivation: to test this alternative interpretation with biological motion

Why biological motion? a prototypical case of a complex hierarchical stimulus (Johansson, 1973; Cutting & Proffitt, 1982) multiple elements, each with their own spatio-temporal trajectories organized quickly and efficiently in a hierarchical configuration, in which the motion of the local elements are coded relative to a more global structural description the perceptual Gestalt is constructed automatically by the visual system (Thornton & Vuong, 2004) the construction of the perceptual whole implies a more efficient representation of the relationships between the parts (Tadin et al., 2002) inversion allows control over low-level motion trajectories (Sumi, 1984; Pavlova & Sokolov, 2000)

Demonstrations

Methods Stimuli: motion captured point-light treadmill walkers (Vanrie & Verfaillie, 2004) 70 colored dots ( confetti walker ) upright, inverted, phase-scrambled Task: adjust rate of change in the test figure until it matches the rate of change in the comparison figure Direct comparison of dynamic and static comparison figure: scrambled test figures: upright or inverted

Results

Discussion on top of the effect of static vs moving, there is a clear effect of configurality ( goodness of the whole percept) cost of objecthood: the more strongly the parts are integrated into the perception of a whole object, the less accessible the changing features of the parts are (e.g., also embedded figures)

Eliminative Gestalts excellent example: bistable diamond Murray et al. key papers: Murray, S. O., Kersten, D., Olshausen, B. A., Schrater, P., & Woods, D. L. (2002). Shape perception reduces activity in human primary visual cortex. Proceedings of the National Academy of Sciences, 99, 15164-15169. Fang, F., Kersten, D., & Murray, S. O. (2008). Perceptual grouping and inverse fmri activity patterns in human visual cortex. Journal of Vision, 8(7):2, 2-9. doi: 10.1167/8.7.2

Demonstration

Nice features perceptual bi-stability: parts seen to move vertically whole seen to move horizontally switching relatively slow, perceptual states rather clear stable individual differences studied rather extensively at psychophysical level, e.g. Lorenceau, J. & Shiffrar, M. (1992). The influence of terminators on motion integration across space. Vision Research, 32, 263-273.

Murray et al.: Design present bistable diamonds ask observers to indicate perception of parts (line segments) or whole (diamond) record BOLD responses (fmri) in different areas and relate these to the reported percepts

Murray et al.: Results

Discussion convincing demonstration of inverse activity patterns in V1 and LOC interpretation? perception of parts suppressed by perception of whole predictive coding framework: explaining away however in a recent follow-up study, we have shown that the reduction of activity in V1 is global, not retinotopically specific

Conclusion the encoding of parts, wholes and their relationships constitutes a serious challenge to the visual system the visual system appears to have developed flexible mechanisms with different characteristics sometimes wholes are encoded in low-level areas (feedback?) sometimes wholes are encoded in high-level areas, while parts are preserved in low-level areas sometimes wholes are encoded in high-level areas, while parts are suppressed in low-level areas further research is needed to establish the specific properties of these cases (computational reasons, boundary conditions, etc.)

THANK YOU JOHAN.WAGEMANS@PSY.KULEUVEN.BE WWW.GESTALTREVISION.BE