What can films teach us about vision? Professor Tom Troscianko Department of Experimental Psychology University of Bristol
A general question What is a movie and why is it fun to watch?
Some scientific questions How do we perceive a changing world? Is the world as depicted in film markedly different from the one outside the cinema? How do any such differences affect perception when viewing a film?
A practical question Is it worth buying a large TV?
The traditional approach to vision Attention Models in the head
Fixation and saccades Our fovea is the only high-resolution area within the visual field Meaning that rapid shifts of fixation saccades are required to gain information about objects in different parts of the visual field
Stev http://www.dakinlab.org/dakinlab/main.html
Stev X
Stev
Dynamic scene inspection Show Iain Gilchrist videos
Models in the head? So - we make eye movements and traditional accounts of vision assumed that we construct a detailed model of the world in our head It feels to us that this is what is happening But change blindness experiments suggested otherwise
If we have a detailed representation in our head, then we should have no difficulty in noticing changes in a scene which occur across interruptions of processing tasks
The trans-saccadic method Blackmore SJ, Brelstaff G, Nelson K, Troscianko T (1995) Is the richness of our visual world an illusion? - Trans-saccadic memory for complex scenes. Perception 24, 1075-1081.
Basketball video Show video Monkey Business
More modern accounts of vision We pick up information about a scene from eye movements and this is processed in a manner which guides action Fixations pick up information about: Items with which we are about to interact The gross geometry of the scene Image locations which provide answers to questions that we are currently asking
Ilya Repin (ca. 1884) Unexpected visitor
Yarbus (1914-86).
Yarbus (1914 86).
Scanpaths from Yarbus (1967)
Scanpaths from Yarbus (1967) Free viewing Material circumstances Age of people Doing what before arrival? Clothes Position of people and objects How long has visitor been away?
Tatler et al (2010)
Tatler et al (2010)
Sensorimotor theory of vision O'Regan, J.K., & Noe, A. (2001). A sensorimotor account of vision and visual consciousness. Behavioral and Brain Sciences, 24(5), 939-1031. The central idea of our new approach is that vision is a mode of exploration of the world that is mediated by knowledge of what we call sensorimotor external memory that can be probed at will by the
Cinematic viewing The visual world of the cinema is quasistable It keeps jumps to a new shot with a mean frequency around 5 sec relevant perceptual processes can survive However, the basic signal is unstable What does vision science predict the effect of this global instability to be?
Vision science prediction If an instability occurs as a result of a saccade, we perceive stability in spite of most receptor signals being very different (Bridgeman 2010) visual instability will make the world appear unstable Such an unstable stimulus is likely to be aversive can cause migraine at the right temporal frequency (Arnold Wilkins)
Movie: continuity error Show video Office
Ceci pas la verité!
What are the timing statistics of movies? Cutting JE, DeLong J, Nothelfer CE (2010) Attention and the evolution of Hollywood film. Psychological Science doi:10.1177/0956797610361679 Studies shot length statistics for 150 films from 1935 to 2005 Measured autocorrelation function for shot length setting the actual narrative aside, perhaps being engrossed in a film is, in part, to allow its 1/f temporal
Cutting paper
Film shots are getting shorter! Salt, B. (2009). The Shape of 1959. The New Review of Film and Television Studies, 7(4)
Unstable world of movies now with a story!
Perception of films The autocorrelation argument suggests be modified to allow for an unstable world whose temporal statistics conforms to a certain law We shall return to test this issue directly Meanwhile, we need a measure of
What can we measure? We shall points during the movie -
How to measure presence We measured subjective ratings of presence using a simple line-bisection task Each time a sound beeped or a light flashed, subjects made a mark on a line to indicate how much they X low presence high presence
What film to use? Hasson et al (2004) Science 209, 1631-1641 United Artists, Sergio Leone director) fmri study Established three main types of scene giving reliably different profiles of response in the visual cortex: Faces (Fusiform gyrus, FFA) Landscapes/buildings (Colleteral sulcus, PPA) Hand-related scenes (Post-central sulcus) Therefore, if we use this movie and the points that give high activation in FFA and PPA, we will be studying the brain when it has a different pattern of response.
What film to use?
FFA scenes
FFA scenes
PPA scenes
PPA scenes
Screen size experiments Thanks to Lee De-Wit for the picture
Presence results Screen size p<0.001 Scene type p<0.001 NS interaction
Presence and memory The effect of screen size on presence and recognition memory for scenes
Presence results Screen size p<0.005 Scene type p<0.05 NS interaction
Memory results Screen size p<0.001 Scene type p<0.001 Size x type interaction p<0.001
Presence, reaction time, and arousal The effect of screen size on presence, reaction time, and arousal
What can we conclude? Presence ratings appear to reflect the degree of arousal of the viewer similar pattern of results to RT and pupil dilation Whereas recognition memory appears to operate on different principles Rated presence is a reliable measure of the degree of immersion in a movie
And now to something more interesting than screen size Nobody has properly studied the difference between colour and black & white movies Matched groups saw one or other movie
show installed a stereo projection system for 250k in the Watershed cinema In 3D and 2D
Avatar
Finally: what is the effect of visual instability on presence? The shoot-out scene experiment We used shoot-out scenes from obscure Western movies made in early 1960s Created a 26 minute showreel and tested presence on two groups of 21 subjects
Presence and mean shot length
Presence and mean shot length: replication with Good, Bad, Ugly
Presence and mean shot length in Avatar 2D r = -.626, p <.001 3D r =-.526, p =.003
Editing and presence The problem with the data up to now is that a movie editor can choose to put short shots in the more exciting parts of the movie What we need is a means of manipulating shot length independently of narrative Achieved this by making a video recording of 30 mins of a stage play with three static cameras Control condition: uncut video/sound from one camera Edit 2: like Edit 1, but in different time phase
Editing and presence: stage play
Editing and presence The presence in the edited conditions is significantly higher than in the control condition without edits These edits are not relevant to the narrative, being taken from an unrelated movie Suggesting that adding visual discontinuities
So: what does all this mean? The result that short shots give high presence is hard to explain with existing theories of vision such as the Stable World Assumption So what is going on Could it be that things which are unusual and tending towards being aversive/unpleasant get us excited?
Aversive stimuli studied by Arnold Wilkins
Wilkins asked people to classify modern art
A repetition
Design Debbie Ayles, 2004
A further repetition
A similar study with photographs from: Pretty J, Peacock J, Sellens M, Griffin M. The mental and physical health outcomes of green exercise. International Journal of Environmental Health Research 2005; 15: 319-337.
Fernandez and Wilkins (2008)
So: what does all this mean? Wilkins and colleagues have identified that systematic deviations from normality in static images can produce an aversive response In a similar manner, movies have a temporal frequency spectrum which is unnatural Both types of stimulus (the Wilkins images and movies) produce high excitation in visual cortex This may be associated with high degrees of arousal, and therefore presence Like movies, the Wilkins images are compelling (Wilkins, private communication)
So: what does all this mean? So: A stimulus which violates the properties of natural scenes can be arousing The editing of movies therefore may make them more arousing The skill lies in making the visual story possible to follow in spite of these discontinuities
Op-art
Bridget Riley
Conclusions Films are immersive and arousing The degree of arousal is (partly) determined by the viewing situation The arousal in films is augmented by the temporal discontinuities (edits) in the film Films tend towards being aversive stimuli, but the (mild) levels of aversion appear to result in arousal/excitement we get excited by a world which seems slightly dangerous but we know we are safe!
A practical answer a large TV!
Flirting with danger: adult version
Flirting with danger: