To appear in: Handbook of Color Psychology, Andrew J. Elliot and Mark D. Fairchild, editors. Cambridge University Press

Transcription

1 FUNDAMENTALS OF COLOR VISION I: COLOR PROCESSING IN THE EYE Andrew Stockman 1 and David H. Brainard 2 1 University College London, Institute of Ophthalmology 2 University of Pennsylvania, Department of Psychology To appear in: Handbook of Color Psychology, Andrew J. Elliot and Mark D. Fairchild, editors. Cambridge University Press Introduction The ultimate purpose of vision, and indeed of color vision, is to provide us with a representation of the physical objects and lights in our three-dimensional environment. A complete understanding of vision must incorporate: (i) a description of the physical properties of these objects and lights (sometimes called the distal stimulus for vision); (ii) a description of how information about those physical properties is carried by light to the eyes and imaged by the optics to form the retinal image (sometimes called the proximal stimulus for vision); (iii) a description of how the retinal photoreceptors respond to the retinal image; (iv) a description of how the photoreceptor responses are transformed by visual processing into the way we see the world our perceptual representation; and lastly (v) a description of the nature of that perceptual representation. In this chapter, we focus on the initial stages of the visual process, primarily as it relates to color. We consider the formation of the retinal image, the transduction of the image by the mosaic of cone photoreceptors, the processing of the cone responses by retinal circuitry, and how retinal signals flow through the lateral geniculate nucleus (LGN) on their way to the visual cortex. The properties of each stage limit the information available to subsequent stages. These limits and the corresponding loss of information at each stage fundamentally shape color perception.

2 We emphasize two major themes. First, when two physically distinct stimuli cannot be discriminated by an observer, they are assumed to have effectively identical representations at some stage of visual processing such that whatever information could have been used to distinguish them has been lost somewhere in the processing stream (Brindley, 1970; Geisler, 1989). Understanding the causes of information loss can explain a number of fundamental facts about human color vision. This type of interpretation is most secure when the stage of processing under consideration is welldefined, which in essence means that the responses of the stage are determined by the signals arising from the visual stimulus combined with internal and external noise. The retina satisfies this criterion to a reasonable approximation. A second theme is that the representation of information changes as it is passed from one stage of visual processing to the next. These transformations affect both what information survives at each stage and also what information is most explicitly available to the following stage. In this chapter we focus primarily on links between information loss at early stages of visual processing and psychophysical performance. Not all aspects of color perception, however, can be understood in terms of information loss alone: understanding the nature of how color is represented is also important. For example, we would ultimately like to associate the experience of subjective percepts of hue with particular patterns of neural activity in the brain, but any theory that achieves this is unlikely to be concerned solely with information loss. Although we are a long way from having satisfactory theories of this sort, a goal of this chapter is to provide an outline of the format in which color information is passed to the cortex. Wavelength, visual optics and image formation Vision begins with light. For the purposes of color vision, its key physical property is that it consists of a mixture of individual components that differ in two respects: their wavelength and their power (Newton, 1704). Thus we can characterize the light arriving at a retinal location by its spectral power distribution, P( λ ), which gives the amount of light power at each wavelength,, in the 2

3 visible spectrum. Power can be thought of as a measure of how much light of each wavelength arrives at a location per unit time. For most practical purposes, the visible spectrum can be considered to lie between approximately 390 nm and 730 nm, although some tabulations of color data extend as far as 360 and 830 nm; and if intense enough lights are used measurements of sensitivity can be made in the near infra-red (Brindley, 1955). Our visual systems are insensitive to electromagnetic radiation outside of this interval (e.g., x-rays, radio waves). In this introductory chapter, we will not consider the physics of light in additional detail (see Purcell, 1965; Hecht, 1990) nor how light interacts with matter to form the retinal image of objects in a scene (see Adelson & Bergen, 1991; Pharr & Humphreys, 2010). To provide some intuition about wavelength and color, the top panel of Figure 1 shows a rendered image of a scene. Images of this scene were acquired for each of 31 narrow bands of wavelength ranging between 400 nm and 700 nm at 10-nm intervals, of which three are shown in the middle panel of the figure. Roughly speaking the brightness in each of the three grayscale images represents the power in its narrow band of wavelengths. That is, the images represent P( λ ) at each location in the image for the sample wavelengths 440 nm, 550 nm, and 660 nm, respectively. As the hue of the rendered image in the top panel varies, the relative intensities in the three narrowbands vary systematically. Image regions that appear blue tend to be lighter in the 440-nm band than in the 660-nm band, image regions that appear green tend to be lightest in the 550-nm band, and image regions that appear red tend to be lighter in the 660-nm band. Comparison of the middle panels of Figure 1 also indicates a regularity of natural images: the same basic spatial structure is evident in each of the narrowband planes (Chakrabarti & Zickler, 2011; Ruderman, Cronin, & Chiao, 1998). This regularity in space, together with similar regularities in time, means that there are correlations across wavelength in the information available about the spatial structure of stimuli in the environment. Although it is not an idea we will treat in this chapter, the notion that vision often exploits regularities in the visual input to improve the quality of perceptual representations is central 3

4 to modern thinking about perception (for examples related to color, see Attneave, 1954; Barlow, 1961; Buchsbaum & Gottschalk, 1983; Atick, 1992; Atick, Li, & Redlich, 1992; Geisler, 2011). Consequently, it should not be surprising that the visual processing of color depends strongly on the spatial and temporal structure of the stimulus. Indeed, such interactions begin as early as the formation of the retinal image and continue through the early stages of color processing. [Insert Figure 1 about here.] The retinal image of a scene is formed by light entering the pupil of the eye, and it is this image that provides the brain with information about the visual world. Thus the quality of the retinal image imposes an initial constraint on vision. Two important factors limit that quality. The first is diffraction: light passing through the eye s pupil interferes with itself to produces a blurred pattern on the retina. Human eyes have a circular pupil that varies in diameter between about 2 and 8 mm, with the size constricting as the light level increases. The amount of optical blurring can be measured in several ways: by the spatial spread in the image of a point source of light (the point spread function or PSF), or by the smallest separation between adjacent elements of a striped object that can just be resolved as separate stripes. The stripes are specified in terms of the number of stripes or cycles for each unit of visual angle. The diffraction-limited resolution depends on pupil size and, for a 2-mm pupil is about 1.05 cycles per minute of arc (Westheimer, 1964), but this diffraction limit improves as the pupil diameter increases. Consequently, the deleterious effects of diffraction get worse as light level increases and the pupil constricts. The second factor that limits image quality is optical aberrations. Even at their best, the eye s optical focussing elements, primarily the cornea and lens, are imperfect and contain aberrations (Westheimer, 1986; Charman, 2010). Aberrations come in various sorts: some arise because of variations in the curvature of the cornea and some chromatic aberrations arise because the focussing power of the cornea and lens depend upon wavelength, so that not all the wavelengths from an object can be in focus in the retinal image at the same time. Aberrations combine with 4

5 diffraction and the net effect is a retinal image that is more blurred than diffraction alone would predict (except for the smallest pupil sizes where diffraction predominates). The amount of blurring depends on pupil diameter (with greater blurring at larger pupil diameters because of increased aberrations), on retinal position (with greater blurring at peripheral retinal locations), on the wavelength of the incident light (because of chromatic aberrations), on the distance to which the optics of the eye are accommodated (focussed), and on the distance from the eye to the various objects in the scene (depth of field). The blurring due to diffraction and optical aberrations are usefully summarized by the point spread function (or PSF). This is the retinal image formed by a very small point of light. Because any image may be thought of as the superposition of the images formed by a collection of point sources, knowledge of the point spread function allows the straightforward computation of the retinal image by superimposing the PSF corresponding to each point source. When we consider color, an important aspect of the optics is that they contain chromatic aberrations. These occur because the refractive index of the cornea and lens depend on wavelength (Westheimer, 1986; Charman, 2010). This dependence in turn means that the focussing power of the eye s optics depends on wavelength and that the size of the PSF is wavelength dependent. Typically, the eye is accommodated (the shape of the lens is adjusted), so that wavelengths in the middle of the visible spectrum are in best focus. This means that the image at other wavelengths is blurred. This wavelength-dependent effect is largest for the shorter wavelengths as illustrated in the bottom panels of Figure 1. The interaction of optical blur with wavelength means that color and spatial information become intertwined even before the photoreceptors. Additional differences between an idealized image and the actual retinal image include wavelength-dependent absorption by media within the eye (primarily the lens and, for central vision, the macular pigment) and scatter of light within the eye (the amount of which is also wavelength dependent). The typical absorption of the lens and macular pigments for a 2-deg field of view are 5

6 shown in the lower inset of Figure 3 below, but note that lens absorption increases systematically with age. The photoreceptor array and retinal sampling The retinal image itself is not directly accessible to the visual system. Rather, it is represented by three broad classes of light-sensitive cells in the human retina. These are the rod and cone photoreceptors, and the recently-discovered intrinsically-photosensitive retinal ganglion cells (iprgcs). Each class of light-sensitive cell is deployed in a mosaic across the retina, and thus each class samples the retinal image at a set of discrete locations. The rods are highly sensitive to light and are primarily responsible for vision at low (scotopic) light levels. At typical daylight intensities (photopic light levels), the rod system saturates and thus provides little or no information about spatial or temporal variations in the retinal image. Although there is an intermediate range of light levels (mesopic light levels) where both rods and cones contribute to vision, we will not consider rods further in this chapter. See Buck (2014) for an introduction to rods and their interactions with cones. The presence of iprgcs in mammalian retina is a relatively recent discovery (for reviews, see Do & Yau, 2010; Berson, 2014). These cells contain the photopigment melanopsin. Although they play important roles in several aspects of visual function, such as the control of pupil size (Gamlin et al., 2007) and the regulation of the circadian clock (Berson, 2003), what if anything they contribute to conscious visual perception is less clear (but see Zaidi et al., 2007; Brown et al., 2012; Horiguchi, Winawer, Dougherty, & Wandell, 2013). As with the rods, we will not consider the iprgcs further in this chapter we consider only cones. The cones are less sensitive than rods and do not saturate easily. It is the cones that are primarily responsible for our perception of color at mesopic and photopic light levels the range of light intensities typically experienced outdoors between dawn and dusk when we can see color (Hood & Finkelstein, 1986). There are three classes of cones, distinguished by their different sensitivities to 6

7 light of different wavelengths. These are referred to as the long (L), medium (M) and short (S) wavelength-sensitive cones because of their relative sensitivity to different wavelengths. As we discuss in more detail below, it is the presence of three classes of cones that enables trichromatic color vision. Cones of each class effectively sample the retinal image via their own submosaic, and the three submosaics are interleaved. Recent advances in adaptive optics (AO) can now be used to measure and correct for the eye s optical aberrations and allow diffraction-limited imaging of the retina. By dilating the pupil to minimize the effects of diffraction and by using AO to reduce aberrations it is possible to resolve the mosaic of cone photoreceptors in the living human eye (Roorda & Williams, 1999; Roorda, 2011). Figure 2 shows AO images of small patches of the retinae of individual subjects. Overlaid on the images are colored dots indicating the class of each imaged cone red dots for L-cones, green for M-cones, and blue for S-cones. The cone type was identified by comparing images obtained at different wavelengths after differential or selective bleaching of photopigments (Roorda & Williams, 1999; Hofer & Williams, 2014). [Insert Figure 2 about here.] Several features of the cone mosaic are noteworthy (Hofer & Williams, 2014). First, for all subjects, the S-cones are sparser than the L- and M-cones, making up only about 5-10% of the cone population, and are spaced in a quasi-regular fashion (Curcio et al., 1991). One conjecture about the reason for the sparse spatial sampling by the S cones, and their absence from the very center of the fovea (Willmer, 1944; Williams, MacLeod, & Hayhoe, 1981), is that the S-cones respond best to short wavelengths. As discussed above, chromatic aberration in the eye s optics lead to a sufficiently blurred short-wavelength image that most of the spatial information available at short wavelengths can be obtained via sparse spatial sampling (Yellott, Wandell, & Cornsweet, 1984; Garrigan et al., 2010). Second, the L- and M-cones are interleaved with each other in a random or nearly random arrangement (Hofer, Carroll, Neitz, Neitz, & Williams, 2005). Third, there is large individual 7

8 variation in the relative numbers of L- and M-cones; the L:M cone ratio in Figure 2 ranges from 1:2.7 (HS; a protan carrier a female, one of whose two X chromosomes is missing the L-cone photopigment gene) to 16.5:1 (BS) (Hofer et al., 2005). [Interestingly, red-green color vision as assessed by standard tests is typically normal across this large range of L:M cone ratios (Pokorny, Smith, & Wesner, 1991; Brainard et al., 2000; Hofer et al., 2005)]. Though the chromatic aberrations of the eye s optics and the interleaved nature of the three cone submosaics mean that color and spatial information is intertwined at the first stage of visual processing, progress can be made by studying color vision using spatially-extended stimuli and neglecting the role of the optics and the packing of the mosaic. We next review several such studies and return to the dependence of color vision on the spatial structure of the stimulus at the end of the chapter. Color vision at the photoreceptors Many of the essential properties of human color vision are determined or heavily influenced by the properties of the cone photoreceptors; among these are: color matching and trichromacy, spectral sensitivity, individual differences ranging from normal variability to color vision deficiencies, and cone-specific ( first-site ) adaptation. Although these properties were originally discovered and formalized through psychophysical experimentation, in many cases we now have a firm understanding of their molecular basis. Phototransduction and univariance Color vision begins with the absorption of a photon by a single cone photopigment molecule, which initiates a cascade of steps that lead to ion channel closure in the photoreceptor membrane and the electrical hyperpolarization of the photoreceptor. Voltage-dependent neurotransmitter release at the cone pedicles then transfers this signal through bipolar cells to the rest of the visual system. Although the molecular transduction steps in the receptor are complicated, they are becoming increasingly well understood. Phototransduction begins in the receptor with a photopigment 8

9 molecule made up of a transmembrane opsin [a G (GTP-binding) protein-coupled receptor protein] covalently bound to the chromophore, 11-cis-retinal. The absorption of a photon by the chromophore initiates a rapid conformational change of the opsin into the activated photoproduct, metarhodopsin- II. Metarhodopsin-II interacts with transducin (the G-protein) releasing the transducin α-subunit. The α-subunit activates phosphodiesterase (the effector molecule), which then catalyzes the removal of the nucleotide cyclic guanosine monophosphate (cgmp). The lowered level of cgmp leads to the closure of cyclic-nucleotide-gated ion-channels in the receptor membrane, blocking the influx of Ca 2+ and Na + ions into the photoreceptor. This in turn causes the interior of the photoreceptor, already electrically negative with respect to the exterior of the cell, to become more negative (i.e., to hyperpolarize), thus reducing the release of the glutamate neurotransmitter at the cone pedicle and producing the neural visual response (Perlman & Normann, 1998; Pugh, Nikonov, & Lamb, 1999; Pugh & Lamb, 2000; Burns & Baylor, 2001; Arshavsky, Lamb, & Pugh, 2002). While these steps might seem far removed from the psychology of color vision, they fundamentally limit color perception. Crucially, the first step of this cascade is binary: a photon either isomerizes the 11-cis-retinal or it does not, and the isomerization is the same whatever the wavelength (energy) of the absorbed photon. Thus, a photoreceptor effectively counts photons. What does vary with photon wavelength is the probability that an isomerization will occur. However, since the number of isomerizations depends jointly on the probability of photon absorption and on the number of photons reaching the photopigment (i.e., on light wavelength and light intensity), the two are confounded; individual photoreceptors are blind to wavelength. Put another way: a change in the rate of photon absorption could be due to a variation in wavelength but could equally be due to a variation in light intensity. This property leads to what is known as the Principle of Univariance (Mitchell & Rushton, 1971). The principle that changes in the number of photons absorbed in a cone can be caused indistinguishably by both intensity and wavelength was inferred from psychophysical experimentation long before the details of the transduction cascade had been unraveled. Univariance 9

10 can also be demonstrated directly via cone photocurrent recordings (Baylor, Nunn, & Schnapf, 1987). The sensitivity of a cone class to a particular wavelength of monochromatic light is proportional to the probability that a cone of that class absorbs an incident photon of that wavelength. The function relating the sensitivity of a cone class to wavelength is called its spectral sensitivity. The three cone classes are named to indicate the relative location of their peak sensitivity along the wavelength axis (see the upper panel of Figure 3) as long-, medium-, and short-wavelength-sensitive cones (L-, M- and S-, respectively). The L- and M-cone spectral sensitivities are broad and overlap throughout the entire visible spectrum. The S-cone spectral sensitivity, on the other hand, is negligible in the long wavelength region above about 560 nm. The breadth of the spectral sensitivities means that there is no light that stimulates only one cone class. [Insert Figure 3 about here] Monochromacy, dichromacy, trichromacy, and color matching A consequence of univariance is that color vision at the photoreceptor level is relatively straightforward to understand. Indeed, if we had only a single cone class, color would be reduced to a single dimension and would be monochromatic: for such a system, a light of any spectral composition (often called a test light in color matching experiments) can be made to look identical to (i.e., to match) any other light simply by adjusting its intensity so that the two lights produce the same number of absorbed photons. This is the sort of vision we have when the light levels are low and only rods, with their single class of photopigment, operate. Under such low-light scotopic conditions, everything appears to have the same hue and objects vary only in lightness. With two univariant cone classes, color vision would be two-dimensional and would be dichromatic. For a dichromatic system, a test light of any spectral composition can be matched by a mixture of two other lights. The two mixture lights, often called primaries to distinguish them from the test light, must have the property of independence, which means that the appearance of one of 10

11 them cannot be matched by any intensity of the other. Independence is required so that the number of absorbed photons can be independently adjusted in each of the two cone classes to match their responses to any test light. With three classes of cone photoreceptor, color vision becomes three dimensional and is said to be trichromatic. For such a system, a light of any spectral composition can be matched by a mixture of three other primary lights; provided, again, that no one of the primaries can be matched by a combination of the other two. Trichromacy is the usual situation in human color vision, because color-normal observers have three univariant cone classes with different spectral sensitivities (see Figure 3). This limitation to just three dimensions means that many pairs of lights that are physically very different from one another can appear identical. Such pairs are known as metamers and arise whenever the members of the pair produce the same photon absorption rates in each of the three cone classes. A consequence of trichromacy and metamerism is that many colors can be reproduced using three-color, additive projection systems, such as color televisions, monitors and projectors. Historically, trichromacy was established in color matching experiments well before the existence of three univariant cone classes was known to underlie it (Grassmann, 1853). Now that the three cone classes have been characterized, however, it is more straightforward to relate color matching directly to the properties of the cones. Nonetheless, the laws originally derived from color matching remain central to our understanding color vision (e.g., Krantz, 1975). For the purposes of human color vision, lights can be specified with respect to three primaries used in a color matching experiment, an example of which is illustrated in the lower panel of Figure 4. Typically, an observer is presented with a centrally fixated bipartite field. Half the field is the "test" field and is illuminated by a test light. The test light can be of any wavelength distribution. The other half-field is the "comparison" field. It can be illuminated by a mixture of three chosen primary lights; these are typically lights that look red (R), green (G), and blue (B) but any three 11

12 independent lights would do. For each test, the observer adjusts the intensities of the three primary lights with the option of using all three in the comparison field or using two in the comparison field and mixing one with the test so that the test field and comparison field are indistinguishable. In general, to complete the match, one of the primary lights must be switched to the other side and mixed with the test. [Insert Figure 4 about here] One important set of matching results is shown in the upper left-hand panel of Figure 4. The data were obtained by Stiles & Burch (1959) using monochromatic primary lights with wavelengths of 645, 526, and 444 nm. The upper left-hand panel of Figure 4 has three curves each showing the relative amount of one of the three primaries required to match a monochromatic test light at the wavelength shown along the abscissa. Each curve corresponds to one of the primaries and the curves, known as color-matching functions (CMFs), are referred to as: rr (λλ), gg (λλ) and bb (λλ), respectively. Notice that the CMFs shown in Figure 4 can have a negative value. Physically, there is, of course, no negative light but, in the context of color matching experiments, a negative value for a primary means that the primary has been added to the test light to complete the match, and it is the mixture of that primary and the test that looks identical to a combination of the other two primaries. Thus matching the test light in the context of a color-matching experiment is not what is ordinarily meant by matching, since although the two halves of the bipartite field can be made to appear indistinguishable neither half need end up looking like the test light alone. Color matching functions are usually determined for monochromatic test lights, as with those shown in Figure 4. Since any given light can be considered as a weighted mixture of monochromatic lights, we can determine the amounts of the three primaries needed to match that light by summing the amounts needed to match each of its monochromatic constituents. This empirically determined linearity property of color matching (Grassmann, 1853; Krantz, 1975) greatly simplifies 12

13 characterization, since it is only necessary to actually measure matches for the monochromatic lights to predict the matches for any test light (see for example Wandell, 1995). Consequently, CMFs are usually measured and tabulated with respect to monochromatic test lights. Cone spectral sensitivities We can match a test light in color-matching experiments using just three primaries because we need only equate the number of photons absorbed by the three cone classes in the two half fields. Cone spectral sensitivities underlie all trichromatic color matching functions, including those shown in Figure 4. The underlying linking hypothesis can be stated formally: After an observer has adjusted the three primaries to achieve a color match with a given test light, the matched half-fields produce identical quantal absorption rates in the observer s L-cones, M-cones and S-cones. But, how can the L-, M- and S-cone spectral sensitivities at each wavelength be recovered from color-matching data? If there were three lights that each uniquely stimulated each of the three photopigments and we were to use them as primaries in a color matching experiment, then the resulting CMFs would provide direct measurements of the cone spectral sensitivities. Such coneisolating primaries do not exist, however, because the cone spectral sensitivities are broad and overlap (see Figure 3). Nevertheless, color matching data measured with real primaries place constraints on the cone spectral sensitivities. The intuition is straightforward: color matching data establish pairs of physically different lights that produce the same absorptions in each cone photopigment; consequently, only cone sensitivities that predict equal absorptions for all measured matches can be consistent with the color matching data. Although we will not provide the derivation here (for details see Brainard & Stockman, 2010), it can be shown that the only candidate cone spectral sensitivities consistent with a set of color matching functions are those where each cone sensitivity is given as a weighted sum of the those color matching functions (i.e., they must be linear transformations of the CMFs). This result follows 13

14 from the constraint described in the previous paragraph and the empirical fact, noted above, that color matches are linear in the spectra of the light to be matched (Grassmann, 1853; Krantz, 1975). Much effort in color vision research has been expended in determining the appropriate weightings of the color matching functions required to produce the cone spectral sensitivities. These can be obtained in several ways: from comparisons between normal color matches and those made by dichromatic observers who are missing one of the three cone classes (e.g., Maxwell, 1856; König & Dieterici, 1886; Vos & Walraven, 1971; Estévez, 1979), from direct measurements of cone spectral sensitivities (e.g., Smith & Pokorny, 1975; Stockman, MacLeod, & Johnson, 1993; Stockman & Sharpe, 2000), or, in the case of S-cones, from direct analysis of color matching data (Bongard & Smirnov, 1954; Stockman et al., 1993; Stockman & Sharpe, 2000). The derived cone spectral sensitivities are also referred to as the "fundamental" CMFs that underlie all other color matches. (Historically, the fundamental CMFs were described as such because they correspond to three primary lights that, were it physically possible to produce them, would individually excite each of the three cone classes separately to produce what were hypothesized to be three Grundempfindungen or fundamental sensations. The term cone fundamentals is now also used to describe estimates of cone spectral sensitivities derived as a weighted combinations of color matching functions.) The validity of a set of cone fundamentals depends not only on the correct choice of weighting coefficients, but also on the CMFs from which they are derived. Unfortunately, the available measured CMFs vary widely in quality (Stockman & Sharpe, 1999). A set of cone fundamentals commonly used for the past several decades in vision research has been those of Smith & Pokorny (1975), which are transformations of the Judd-Vos modified 2-deg CMFs (Vos, 1978). The fundamentals adopted more recently by the CIE as physiologically-relevant 2-deg and 10-deg fundamental CMFs (CIE, 2006) are those proposed by Stockman & Sharpe (2000). The latter are based on the Stiles & Burch 10-deg CMFs (1959). These CMFs, which were measured from 14

15 approximately 390 to 730 nm in 49 subjects (and in 9 subjects from 730 to 830 nm), are probably the most secure and accurate set of color matching data. The upper panel of Figure 3 shows the Smith & Pokorny and the Stockman & Sharpe cone fundamentals for 2-deg vision plotted as dashed and solid lines, respectively. The differences between them, which are apparent mainly at short-wavelengths for the L- and M-cone functions and at longer wavelengths for the S-cone functions, arise mainly because of flaws in the Judd-Vos 2-deg CMFs, which are corrected versions of the CIE 1931 functions (Stockman & Sharpe, 1999). Note also that despite well-documented problems with the original CIE deg functions, they are still used extensively in colorimetry and photometry, in part because the 1931 standard is incorporated into many colorimetric measurement devices. The initiating, univariant step of vision is also the primary determinant of the spectral sensitivity of each cone class, since a cone s spectral sensitivity is related to the probability that photons of different wavelengths will isomerize 11-cis-retinal. These probabilities depend principally upon how closely the energy of each photon (given by EE = hcc/λλ, where h is Planck s constant, c is the speed of light, and λ is the wavelength) matches the energy required to initiate the isomerization. Importantly, the initiation energy varies between cone classes because of key amino acid differences in parts of the opsin molecule that surround the chromophore. These differences modify the isomerization energy and thus the spectral sensitivity of the photoreceptor (Deeb, 2005). Note that cone fundamentals are specified with respect to light energy arriving at the cornea rather than at the photopigment. Consequently, they depend on the absorption of photons by the optical media (see the lower panel of Figure 3 for the typical optical densities of the macular and lens pigments) and on the density of photopigment in the photoreceptor outer segment (see, Brainard & Stockman, 2010). Optical density quantifies the transmissivity of a material, such as macular pigment, as -log10(iout/iin), where Iin is the incident light intensity and Iout is the light intensity after it has passed through the material. Thus, a material that reduces the light by a factor of 2 has an optical 15

16 density of 0.3, by a factor of 10, 1 and so on. Optical densities are useful because they can be added to give the total optical density. Because of the macular and lens pigments and the photopigment optical density, cone fundamentals are not exactly the same as the absorbance spectra of the cone photopigments at the retina. Further, because the density of the optical media varies with age, with retinal location, and from individual to individual, cone fundamentals can only be representative averages, albeit tightly constrained ones. These variations in optical density with retinal location lead to different fundamentals for 2 and 10 field sizes. For further details, see Brainard & Stockman (2010). More generally, the CIE standard provides formulae for computing cone fundamentals that are tailored to retinal location and age as well as other individual observer parameters (CIE, 2006). Molecular genetics, genetic variability and color vision deficiencies The spectral sensitivity of each cone class depends on the sequence of amino acids in its photopigment opsin. A watershed in human color vision research was the isolation and sequencing of the L-, M- and S-cone photopigment opsin genes by Jeremy Nathans and his colleagues in 1986 (genes that are now designated OPN1LW, OPNL1MW, and OPNL1SW, respectively) that provided a solid molecular basis for human trichromacy (Nathans, Thomas, & Hogness, 1986; Nathans, Piantanida, Eddy, Shows, & Hogness, 1986). As expected from the inheritance patterns of red-green color vision deficiencies, OPN1LW and OPNL1MW were found to lie next to each other on the X chromosome (at position Xq28), while OPNL1SW, an autosomal gene, was found to lie on chromosome 7 (at position 7q32). A more surprising result was the genetic (and confirmatory behavioral) evidence for multiple versions of the OPN1LW and OPNL1MW genes. These different versions give rise to variations in the L- and M-cone spectral sensitivities of individual observers and also to some red-green color vision deficiencies (Nathans, Piantanida, et al., 1986; Neitz, Neitz, & Jacobs, 1991; Merbs & Nathans, 1992a, 1992b; Winderickx et al., 1992; Sharpe, Stockman, Jägle, & Nathans, 1999). The potential for this variability arises because OPN1LW and OPNL1MW differ by only a few amino 16

17 acids and lie adjacent to each other on the X chromosome. They are therefore are particularly susceptible to alterations (crossovers) during meiosis, which can give rise to the more common types of color vision deficiencies: genes can be either lost (due to intergenic nonhomologous recombination), rendered non-functional (due to missense or nonsense mutations or coding sequence deletions), or altered (due to intragenic recombination between genes of different types to produce hybrid genes), or altered because of point mutations. Phenotypically, the results of major gene alterations are usually dichromacy or anomalous trichromacy. In dichromacy, color vision is reduced to two dimensions either because one of the cone pigments is lost or because a hybrid cone pigment is too similar to another cone pigment The forms of dichromacy known as protanopia and deuteranopia correspond to the lack of L-cones and M-cones, respectively. In anomalous trichromacy, one of the cone pigments is altered so that its spectral sensitivity shifts towards that of another cone, but trichromacy is not impaired in the sense that three primaries are still required in color-matching experiments even though the amounts of each are different from those needed by normal trichromats. The common forms of anomalous red-green trichromacy are known as protanomaly and deuteranomoly, corresponding to whether the L-cones or the M-cones, respectively, are displaced in spectral sensitivity. These spectral shifts are caused by the inheritance of hybrid photopigment opsin genes fusion genes produced by intragenic crossing over and containing the coding sequences of both L- and M-cone pigment genes. The severity of protanomaly and deuteranomoly in an individual depends on the size of the spectral shift of their affected photopigment. Minor alterations in OPN1LW and OPNL1MW that have resulted in consistent small differences in the normal population are known as polymorphisms. The most common genetic polymorphism is the substitution of alanine for serine at position 180 of the L-cone photopigment gene. This substitution produces a shift of several nanometers towards shorter wavelengths in the L-cone spectral sensitivity (Sharpe et al., 1999). These shifts lead to measureable differences in color- 17

18 matching experiments but are too small to lead to changes that are noticeable outside the laboratory, and individuals with variants of these polymorphisms are considered to have normal color vision. Tritanopia is the third type of dichromacy. It is a rare autosomal dominant disorder that arises due to missense mutations in the OPNL1SW gene (Went & Pronk, 1985; Weitz, Miyake, et al., 1992; Weitz, Went, & Nathans, 1992). The dominant inheritance pattern (i.e., only the gene on one of the two chromosomes needs to be defective to produce the disorder) indicates that the presence of the defective photopigment causes a loss of function or loss of S-cones. See the chapters on Variation in normal color vision and Color vision deficiencies in this book for more details. Color vision after photopigment isomerization Understanding the trichromatic representation provided by the cone photopigment isomerization rates allows us to determine when two physically different spectra provide identical input to subsequent visual processing. On the other hand, there are many aspects of visual processing that are not accounted for by the photopigment isomerization rates in the three cone classes. As we discuss in this section, the isomerization rates are transformed within individual cones into a contrast code, and then recombined by post-receptoral processes, so that the code that leaves the retina is substantially reduced and changed from the code encoded by the cone isomerizations. First-site adaptation and contrast-coding An important function of the cone phototransduction cascade is the regulation of sensitivity. Without sensitivity regulation, cones that are sensitive enough to respond effectively to a few photon absorptions at low light levels would soon reach their maximum response as an increasing light level led to increased rates of photon absorption. At its maximum response, a cone is referred to as saturated, since changes in photon absorptions have no immediate effect on the hyperpolarization of the photoreceptor. Thanks to sensitivity regulation (and to photopigment bleaching, which 18

19 depletes the number of photosensitive photopigment molecules at very high light levels), cones rarely saturate under conditions of steady illumination (Alpern, Rushton, & Torii, 1970). Many of the molecular mechanisms of sensitivity regulation acting within the photoreceptor are now reasonably well understood. Regulation is achieved mainly by mechanisms that shorten the time over which the reactions initiated by photon absorptions persist thus shortening the so-called integration time either by speeding up reactions or by terminating them more quickly, and by mechanisms that restore depleted molecules or extend the response range (Arshavsky et al., 2002; Hamer, Nicholas, Tranchina, Lamb, & Jarvinen, 2005; Pugh et al., 1999; Stockman, Langendörfer, Smithson, & Sharpe, 2006). The mechanism for avoiding saturation by shortening the integration time has the effect of lowering sensitivity to the steady-state background while minimizing sensitivity losses to spatial and temporal variation (e.g., MacLeod, 1978) thus preserving much of the information available in the retinal image. Perhaps the most important consequence of sensitivity regulation within the cones is its effect on the information transferred to the rest of the visual system. Cone sensitivity regulation at low temporal frequencies and for steady fields converts the photon absorption rate encoded by photopigment isomerization into what is known as a contrast code. Once the level is high enough, sensitivity for low-temporal-frequency stimuli is roughly inversely proportional to the background light intensity, so that the contrast remains constant. At higher temporal frequencies, on the other hand, contrast can increase with light adaptation (de Lange, 1958; Kelly, 1961). Constant contrast at lower frequencies is known as Weber s Law: the just-noticeable change in intensity, ΔI, is proportional to the background intensity, I. Put another way: the ratio ΔI / I is constant so that ΔI increases as I increases. For example, if we consider the L cones, then the change in cone response to an increment in L-cone photon absorption rate of L relative to L-cone background absorption rate, Lbg, is given by the L-cone contrast CL = L/Lb g. Similar expressions apply for the M and S cones. And it is in the form of cone-contrasts that information moves up the visual system. The 19

20 conversion of absorption rates to a contrast code has an important adaptive function, since the range of cone absorption rates encountered in the environment can be >10 6 (Hood & Finkelstein, 1986), while the range of contrasts encountered for any single scene is generally much smaller (Xiao, DiCarlo, Catrysse, & Wandell, 2002; Heckaman & Fairchild, 2009). The conversion from absorption rates to contrast is revealed psychophysically by measurements of visual difference. A difference threshold is the smallest stimulus change along some parameterized dimension that is just detectable by an observer (Cornsweet, 1970). For example, one could measure how much power for a circular spot of one wavelength is required for that light to be detected when it is seen against a larger circular background of some specified wavelength and power (Stiles, 1949). The measurement of difference thresholds is well-suited for exhibiting the behavioral correlates of information loss early in the visual pathway, since any physical differences between the two perceptually indistinguishable stimuli must have been lost at some stage or stages of the pathway (Brindley, 1970). When discussing threshold measurements, results are often cast in terms of sensitivity, which is the reciprocal of threshold (i.e., the lower the threshold, the higher the sensitivity). When a difference threshold is measured for detecting a light presented against a background, the thresholds approximately follow Weber's Law: over a range of adapting background intensities the threshold for detecting increments seen primarily by a single cone class is roughly constant when it is expressed as in terms of the contrasts of the detecting class (see Stockman & Brainard, 2010). The notion that the signals leaving the cones provides a contrast code (once the light intensity is high enough for cone sensitivity regulation to come into play) is important when considering the effects of post-receptoral processing. A contrast code can be implemented within cone photoreceptors by a reciprocal reduction in sensitivity as the light level increases. The apparently direct and simple link between a contrast code implemented by a reciprocal sensitivity adjustment and the experimental finding that behavioral thresholds follow Weber s Law 20

21 hides some important linking assumptions that should be made explicit. In particular, behavioral thresholds depend on the signal to noise ratio. Consequently, for the changes in the cone signal with adaptation to predict behavioral thresholds implies that that the limiting noise must be roughly independent of adaptation (and thus relatively unaffected by the reciprocal sensitivity adjustment); for cones this seems to be approximately the case (Angueyra & Rieke, 2013). In psychophysical models of sensitivity regulation, adaptation that is cone-selective is known as first-site adaptation; that is, sensitivity measures that stimulate a single cone type depend only on the level of excitation in that cone type. By contrast, if adaptation measured psychophysically depends on the levels of excitation in more than one cone type, it is called second-site adaptation. Attempts to relate this seemingly simple psychophysical distinction to physiology reveals its potential complexity. Sensitivity regulation acting within the transduction cascade of individual cones can reasonably be assumed to be cone-specific, and therefore to give rise to first-site adaptation. However, once signals from different cones are combined, which first occurs at the cone pedicles via gap junctions and horizontal cell feedback (see below), complications arise. First, as we shall see in the next section, there are multiple, successive postreceptoral sites at which second-site adaptation can occur that have different cone inputs with different input weights and different signs (e.g., some will be cone-opponent and some non-cone-opponent). Consequently, second-site adaptation can, in principle, take on several forms. Second, adaptation mediated at a second-site could mimic the effects of first-site adaptation because of its cone inputs and/or because of the experimental conditions. Postreceptoral retinal physiology Figure 5 shows the principal cell types found in the retina, with the exception of the amacrine cells and the rods. Put simply, photoreceptor cells drive bipolar cells, which in turn drive retinal ganglion cells, the axons of which form the optic nerve that carries neural signals primarily to the lateral geniculate nucleus (LGN), the pretectal nucleus, the superior colliculus and the 21

22 suprachiasmatic nucleus. Horizontal and amacrine cells (the latter not shown) make lateral connections between photoreceptor, bipolar, and ganglion cells. Retinal cells typically have dendrites that receive input from other neurons, cell bodies, and axons that transmit the neuronal output to the next neural stage. Cells can also influenced by feedback from later neural stages, as for example when amacrine cells synapse on the axon terminals of bipolar cells. The dendrites, cell bodies and axons of the different retinal cells make up the successive layers of the retina. Starting with the layer furthest from the light, the outer nuclear layer (ONL), contains the cell bodies of rods and cones. Next, the outer plexiform layer (OPL), contains cone pedicles that make synaptic contacts with bipolar and horizontal cells. Next, the inner nuclear layer (INL) contains horizontal, bipolar and amacrine cell bodies. Then, in the inner plexiform layer (IPL), the bipolar cell axons make synaptic contacts with ganglion and amacrine cell dendrites. Lastly, the ganglion cell layer (GCL) contains the cell bodies of ganglion cells and displaced amacrine cells. [Insert Figure 5 about here] There are also some finer details to this structure. In particular, the IPL is stratified into an OFF layer, nearest the bipolar cells, where connections are made between OFF cone bipolar and OFF ganglion cells, and an ON layer where connections are made between the corresponding ON cells (see Figure 5). ON and OFF refer to the effects of light falling on the receptive field of the cell that is, falling on the region of space relative to fixation in which light can alter the cell s response. For most retinal cells, receptive fields are roughly circular with a central disc and an annular antagonistic surround. If light falling in the central disc increases the cell s response (and light falling in its antagonistic surround reduces it) the cell is an ON cell. OFF cells show the opposite characteristics. The number and type of connections at the cone pedicle makes it one of the most complex synapses in the central nervous system (Haverkamp, Grunert, & Wassle, 2000). The L- and M-cones jointly contact diffuse ON bipolar cells (DBON) via inhibitory synapses (which invert the sign of the 22

23 cone signals) and contact diffuse OFF bipolar cells (DBOFF) via excitatory synapses (which preserve the sign of the cone signals), with both types of contacts occurring in a non-cone-class-selective manner (e.g., Polyak, 1941; Boycott & Dowling, 1969; Boycott & Wässle, 1991; Wässle & Boycott, 1991; Grünert, Martin, & Wässle, 1994). Recall that cones hyperpolarize (their response becomes more negative) with increasing photon absorption. Thus, at light onset, when a photoreceptor contributing to the center of a cell s receptive field hyperpolarizes, ON cells depolarize and OFF cells hyperpolarize. The existence of parallel ON and OFF channels is an important feature of visual pathways. L- and M-cones also make inhibitory (sign-inverting) contacts with midget ON bipolar cells (MBON) and excitatory (sign-preserving) contacts with midget OFF bipolar cells (MBOFF) (e.g., Polyak, 1941; Boycott & Dowling, 1969; Kolb & Dekorver, 1991; Calkins, Schein, Tsukamoto, & Sterling, 1994). In the central retina, single L- or M-cones innervate the center responses of single midget bipolar cells, but there are multiple innervations in the periphery (Kolb & Marshak, 2003). Thus in central retina, the centers of the receptive fields of midget bipolars preserve the spectral information encoded by L and M cones, although the cells are cone-opponent because of their mixed-cone surrounds. S-cones make inhibitory contacts with ON S-cone bipolar cells (Mariani, 1984; Kouyama & Mashak, 1992). Their contacts with OFF bipolar cells are more controversial, but there is evidence at least in macaque retina for contacts between S-cones and midget OFF bipolar cells (Klug, Herr, Ngo, Sterling, & Schein, 2003; Field et al., 2010) (not shown in Figure 5). For simplicity, gap junctions between cones (e.g., Raviola & Gilula, 1973; Tsukamoto, Masarachia, Schein, & Sterling, 1992) are not shown in Figure 5. Gap junctions connect the pedicles of adjacent cones and allow various molecules and ions to pass between the two cells and produce a form of electrical cross-talk between them (e.g., DeVries, Qi, Smith, Makous, & Sterling, 2002; Hornstein, Verweij, & Schnapf, 2004). For cones, the cross-talk is mainly between M- and L-cones; 23

24 the S-cones have fewer gap junctions than M- and L-cones (Ahnelt, Keri, & Kolb, 1990) and their coupling to other cones seems to be non-functional (Hornstein et al., 2004; Li & DeVries, 2004). The main function of cone-coupling is thought to be noise reduction (DeVries et al., 2002). Lateral inhibitory connections between cones are made by the H1 and H2 horizontal cells. The H1 and H2 cells both contact M- and L-cone pedicles, while 15% of H1 cells and 100% of H2 cells contact S-cone pedicles (Ahnelt & Kolb, 1994; Goodchild, Chan, & Grünert, 1996). Physiologically, however, an S-cone response has only been found in H2 cells (Dacey, Lee, Stafford, Pokorny, & Smith, 1996). The horizontal cells, through feedback, provide spatially-opponent inhibitory surrounds for cones and bipolar cells. The surrounds created by horizontal cell feedback for individual cones near the fovea means that bipolar cells will generally be cone-opponent simply because the surround signals come from more than one cone class (Boycott, Hopkins, & Sperling, 1987; Paulus & Kröger-Paulus, 1983; Lennie, Haake, & Williams, 1991; Packer, Verweij, Li, Schnapf, & Dacey, 2010). By the term coneopponent in this context, we mean that the cones in the surround collectively have a different spectrally sensitivity from the cones in the center. The degree to which the bipolar cells are coneopponent depends on the number and classes of cones in the center and the number and classes of cones in the surround. In the fovea, where single cones contact the centers of MBON and MBOFF cells, the cells will be cone-opponent (Kolb & Dekorver, 1991; Calkins et al., 1994), simply because the horizontal cell input for M- and L-cone surrounds is not cone-selective. For DBON and DBOFF cells, which are contacted by multiple cones (Hopkins & Boycott, 1995), however, cone-opponency will be weaker, but nevertheless strong cone opponency will be found in some diffuse bipolar cells by chance. Midget bipolar cells connect to ganglion cells of the same type (MGCON and MGCOFF); and together they form the substrate of the parvocellular pathway that projects to the cortex via the parvocellular layers of the LGN; diffuse bipolar cells connect to parasol ganglion cells (PGON and 24

25 PGOFF) of the same polarity and together they form the substrate of the magnocellular pathway that projects to the cortex via the magnocellular layers of the LGN (Leventhal, Rodieck, & Dreher, 1981; Perry, Oehler, & Cowey, 1984; Rodieck, Binmoeller, & Dineen, 1985). There are, in addition, other types of retinal ganglion cell that could be important for processing L- and M-cone signals (e.g., Rodieck, 1998). Of several candidate S-cone pathways through the retina, the one that is best characterized is an ON pathway mediated by SBON bipolar cells (Mariani, 1984; Kouyama & Mashak, 1992) and the distinctive bistratified blue-yellow (SBG) ganglion cells (Dacey & Lee, 1994; Calkins, Tsukamato, & Sterling, 1998; Herr, Klug, Sterling, & Schein, 2003; Dacey & Packer, 2003) that project to the koniocellular layers of the LGN (Martin, White, Goodchild, Wilder, & Sefton, 1997; Hendry & Reid, 2000; Szmajda, Grunert, & Martin, 2008; Tailby, Solomon, & Lennie, 2008). Despite the lack of a distinct anatomical substrate, OFF S-cone signals have been variously reported in recordings from primate ganglion and/or LGN cells in the magnocellular (Derrington, Krauskopf, & Lennie, 1984), parvocellular (Wiesel & Hubel, 1966; Marrocco, 1976; Derrington et al., 1984; Valberg, Lee, & Tigwell, 1986), and koniocellular (Szmajda, Buzás, FitzGibbon, & Martin, 2006; Tailby, Solomon, & Lennie, 2008) streams. The origin of OFF S-cone signals has been linked to contacts between S-cones and MBOFF cells (Klug et al., 2003; Field et al., 2010) found in macaque, but these contacts have not been found in marmoset (Lee, Telkes, & Grünert, 2005). A sign-inverting amacrine cell that converts the S-cone ON signal to an OFF signal has been identified but so far only in ground squirrel (Chen & Li, 2012; Sher & DeVries, 2012); this may reflect a difference between rodent and primate, as cone-opponent S-OFF ganglion cells have been identified in guinea pig that do not appear to exist in primate (Yin, Smith, Sterling, & Brainard, 2009). A giant ganglion cell with an S-cone OFF input, which is an iprgc since it also contains melanopsin, has been reported in primate (Dacey et al., 2005). 25

26 The LGN is a layered structure in the thalamus: its two ventral layers (1 and 2), contain neurons with large cell bodies and are known as the magnocellular layers, while the four dorsal layers, which contain neurons with small cell bodies, are known as the parvocellular layers (3, 4, 5 and 6) (e.g., Hubel & Wiesel, 1977). The koniocellular layers are intercalated between the magnocellular layers (Casagrande, 1994; Hendry & Reid, 2000). There are two lateral geniculate nuclei, one on each side of the brain. By partially crossing over at the optical chiasm, retinal neurons from the left visual field project to the right LGN, and those from the right visual field project to the left LGN. Cells from the ipsilateral eye (the eye on the same side as the LGN) project to layers 2, 3 and 5, while those from the contralateral eye (on the opposite eye) project to layers 1, 4 and 6 (e.g., Hubel & Wiesel, 1977). The receptive-field properties of LGN cells tend to be similar to those of the ganglion cells to which they connect (e.g. Derrington et al., 1984; Lankheet, Lennie, & Krauskopf, 1998; Smith, Lee, Pokorny, Martin, & Valberg, 1992). Thus, the different streams established in the retina are maintained through the LGN to the cortex with very little convergence or divergence, but signals from the LGN are modulated by cortical feedback (Briggs & Usrey, 2011). There are many excellent reviews of color physiology and parallel processing (e.g., Dowling, 1987; Wässle & Boycott, 1991; Rodieck, 1998; Gegenfurtner & Kiper, 2003; Wässle, 2004; Lennie & Movshon, 2005; Solomon & Lennie, 2007; Nassi & Callaway, 2009; Conway et al., 2010). The S- cones and their pathways have been the subject of several recent reviews published in a single volume (Miyagishima, Grünert, & Li, 2014; Dacey, Crook, & Packer, 2014; Martin & Lee, 2014; Marshak & Martin, 2014). More information is available from an on-line resource for retinal physiology and anatomy, Webvision at Recombination of cone signals: Chromatic and achromatic processing Figure 6 shows a diagrammatic model of the early visual pathways. This model condenses and simplifies the details of the physiology and provides a framework that allows us to relate major features of post-receptoral processing to psychophysical data in a qualitative fashion. A quantitative 26

27 treatment is beyond the scope of this chapter and would require explicit consideration not just of the transformations embodied by the model but also of the response noise at each stage of processing as well as an analysis of the information carried by populations of neurons (see, for an introduction to more advanced ideas, Stockman & Brainard, 2010). Figure 6 has its antecedents in earlier models of color vision that emphasized post-receptoral combination of signals from different classes (e.g., L-M, S-(L+M) and L+M pathways, see Figure 7.3 of Boynton, 1979; Ingling & Drum, 1973) and also draws on more recent ideas about how neural processing, occurring after the LGN, might further recombine signals from different types of units (e.g. Lennie & D'Zmura, 1988; Billock, 1991; De Valois & De Valois, 1993; Kingdom & Mullen, 1995). Although this model is incomplete and may be incorrect in some respects, we find it helpful in framing possible links between physiology and psychophysics. [Insert Figure 6 about here] The left side of Figure 6 shows outputs from the L-, M- and S-cones and their connections as three repeated clusters (top, middle and bottom truncated colored triangles for L, M and S-cones, respectively). These are intended to highlight the early retinal processing of signals from each cone class. The three larger cones are central cones and the four smaller cones around each larger cone symbolize a network of surround cones in the ratio of two L-cones to one M-cone, which is approximately the mean L:M cone ratio found in human retina (e.g., Cicerone & Nerger, 1989; Carroll, Neitz, & Neitz, 2002; Vimal, Smith, Pokorny, & Shevell, 1989; Hofer et al., 2005; Sharpe, Stockman, Jagla, & Jägle, 2011). The S-cones are overrepresented in the figure, since they are absent from the central foveola and represent only 5-10% of the total cone population (Curcio et al., 1991). The feedback from M- and L-cones can, in principle, be mediated by H1 or H2 cells (see above). The feedback from S-cones to L- and M-cones, which is mediated by H2 cells, is shown as dashed lines, since its functional significance not fully established (Dacey et al., 1996). L-, M-, and S-cone surround feedback to S-cones via H2 cells is also shown. 27

28 The three cone clusters each have ON and OFF outputs producing L-ON, L-OFF, M-ON, M- OFF, S-ON and S-OFF center responses with opposing surround responses assumed to be derived from the random connections from horizontal cells (e.g., Rodieck, 1998; Dacey, 2000; Wässle, 2004; Field & Chichilnisky, 2007). Established pathways from the cones are shown as solid lines, more speculative pathways as dashed lines. The colored concentric circles shown by each output represent the chromatic and spatial center-surround opponency encoded at each cone output. Each output is cone-opponent because there are different cone classes in its surround, and it is spatially-opponent because the surround is larger than the single cone center input. The color of each central disc is chosen to indicate very loosely the wavelength of a large stimulus likely to produce an increase in the response of the cell. We have labeled all the surrounds as LM to designate the fact that they receive input from both L- and M-cones. The degree to which S-cones contribute to each type of surround is less certain, but there is certainly the possibility of some S-cone input to the surrounds via H2 cells. Note that although we assume an S-OFF system (shown by the dashed line), its anatomical substrate is not yet established. The existence of an S-OFF system in humans, which is separate from the S-cone ON bipolars, is supported by the finding that sufferers of congenital stationery night-blindness, with a GRM6 mutation that leads to a lack of functional ON-bipolars, are not tritanopic (Schmidt, Neitz, & Neitz, 2014). Moreover, S-OFF signals are strongly represented in the cortex (for review see Xiao, 2014). In the middle column, the left side of each diagram illustrates the ON and OFF signals as they flow into the parvo-, magno-, and koniocellular layers of the LGN, while the wiring diagram illustrates speculation as to how the signals that arrive at each layer might be recombined demultiplexed by processing that occurs subsequently in visual cortex, in a manner that would be at least superficially consistent with a number of physiological and psychophysical results. Demultiplexing refers to the fact that the input signals contains both spatial and color information but that at the output the signals contain either color information or achromatic spatial information 28

29 but not both. Such demultiplexing produces the receptive fields shown in the column on the right of the figure. The parvocellular layers receive ON and OFF input with centers from both L- and M-cones. Although we assume random cone connections in the surrounds of parvocellular neurons, how selective the surround inputs actually are remains controversial (Reid & Shapley, 1992; Lee, Kremers, & Yeh, 1998; Dacey, 2000; Mullen & Kingdom, 2002; Diller et al., 2004; Solomon, Lee, White, Rüttiger, & Martin, 2005; Vakrou, Whitaker, McGraw, & McKeefry, 2005; Buzas, Blessing, Szmajda, & Martin, 2006; Jusuf, Martin, & Grünert, 2006). As noted above, for central vision the midget/parvocellular cells will be cone-opponent since only a single cone feeds each unit s center. The parvocellular pathway encodes both cone-opponent and non-opponent information (e.g., Ingling & Drum, 1973; Kelly, 1983; Ingling & Martinez, 1983; Lennie, 1984; Martinez-Uriegas, 1985; Merigan & Eskin, 1986). In the top panel of the middle column, we illustrate this key idea by indicating how the signals from an ON-unit with one type of central cone input added to an OFF-unit with a different central cone input can produce cone-opponent L-M and M-L units having receptive fields with no spatial opponency (note that the structure of these receptive fields would be different if the opponent surrounds of the input units were cone-selective rather than unselective as shown). In the lower parvocellular panel, and in the magnocellular panel, we illustrate how adding ON-units with different central cone inputs can be combined to produce ON L+M spatially-antagonistic units, and how by similarly adding OFF-units with different central cone inputs OFF L+M spatiallyantagonistic units can be produced (Lennie & D'Zmura, 1988; Billock, 1991; De Valois & De Valois, 1993; Kingdom & Mullen, 1995). Unlike the circuitry suggested for the magnocellular and koniocellular pathways shown in Figure 6, which is retinal, the demultiplexing circuitry suggested for the parvocellular pathway is most likely cortical. In the bottom panel, the S-ON signal passing through the koniocellular layers is shown, with its output speculatively combined with an LM surround at a later stage. An S-OFF pathway is not shown in the figure but is likely to exist. 29

30 The iconic representation on the right side of Figure 6 illustrates the concept that as wavelength information leaves the cones and flows from the retina through the LGN, the trichromatic code of L-, M-, and S-cone isomerization rates is transformed into two forms: coneopponent and non-cone-opponent. Cone-opponent channels signal L-M and M-L, while non-coneopponent channels signal L+M. Although they are usually simplified as S-LM and LM-S, the precise nature of the cone-opponent channels that carry S-cone signals is less certain. Although there are probably both S-ON and S-OFF varieties, the signals that oppose the S-cone signals will depend on their physiological substrates, which remain unclear (see above). As discussed above, first-site adaptation also means that we should conceive of the postreceptoral cone signals as contrast signals, with their effective magnitudes for each cone class being normalized by the background isomerization rate for that cone class. Interestingly, the discovery of the physiological transformation of cone to opponent signals was foreshadowed in the 19 th century by Hering (1878, 1920), who noted that certain pairs of colors, red versus green and blue versus yellow, are phenomenologically opposed in the sense that they are mutually exclusive and cannot coexist for a single patch of light. He postulated that color is represented by the responses of three different mechanisms each driven by paired color-opponent processes (red/green, blue/yellow and white/black). Hurvich and Jameson (Hurvich, 1981) elaborated this observation using a psychophysical procedure known as hue cancellation, from which they inferred spectral sensitivities of red-green and blue-yellow color-opponent mechanisms. However, though superficially similar, cone opponency as indicated by physiological measurements in the retina and LGN, and as revealed by color discrimination measurements, has a significantly different spectral signature from the color opponency derived from hue cancellation (e.g., Guth, 1991; De Valois & De Valois, 1993). In the next subsection, we briefly review some ways in which the cone-opponent and nonopponent mechanisms illustrated schematically on the right side of Figure 6 can be connected to 30

31 psychophysical results. Our review is illustrative, as the relevant literature is quite large and a full treatment is beyond the scope of an introductory chapter. For a more extensive review, see Stockman & Brainard (2010). Psychophysical correlates of cone-opponent and non-cone-opponent codes Consider a color detection experiment, in which threshold is measured for a long lasting (200-ms), moderately large (1.2-deg diameter) target flash seen against a uniform background. The target flash is manipulated so that it produces contrasts in each of the different classes of cones in various proportions, with some of the contrasts being increments in intensity (+ signs) and some decrements in intensity (- signs). Figure 7 shows the results of such an experiment where only the L- and M-cone contrasts were varied, in a cone-contrast space. The data are replotted from Figure 4 of Stromeyer, Cole & Kronauer (1985). Each plotted point (yellow dotted circle) represents the mean of several threshold measurements, with the position of a point along the abscissa indicating threshold L-cone contrast ( L/L) and the position along the ordinate indicating threshold M-cone contrast ( M/M). Directions in this plot reflect the ratio of changes in L- and/or M-cone contrast. As indicated by the arrows in the upper left quadrant, the horizontal stimulus direction represents changes in L-cone contrast alone, whereas the vertical direction represents changes in M-cone contrast alone. For example, the two data points on the abscissa either side of the origin for which the M-cone contrasts are close to 0% are the threshold contrasts for L-cone only changes; positive L-cone contrast threshold for increments in contrast, negative for decrements. Similarly thresholds for M-cone only changes are shown on the ordinate where ΔL/L is close to zero. As indicated by the arrows in the lower right quadrant, stimulus directions parallel to the positive diagonal represent changes in L-cone or M-contrast together (L+M), which might be expected to preferentially excite non-cone-opponent mechanisms. Thus the two points along the positive diagonal farthest away from the origin show the L- and M-cone contrasts for incremental and decremental threshold stimuli that stimulate both the L- and M-cones together with equal 31

32 contrast. (For convenience, we will use the term luminance modulations to describe these and related stimuli in the rest of this chapter. This usage is a simplification, because we neglect consideration of how S-cone signals co-vary with those from L- and M-cones.) Directions parallel to the negative diagonal reflect opposing changes in L-cone or M-cone contrast (L-M), which might be expected to preferentially excite cone-opponent mechanisms. The two points along the negative diagonal show the L- and M-cone contrasts for threshold stimuli that stimulate both the L- and M-cones together with opposite contrast. (We will use the term redgreen isoluminant modulations to refer to these and related stimuli where there is no change in the luminance direction, again neglecting a number of considerations that come into play when one defines isoluminance more precisely.) The data shown in Figure 7 may be used to reason about the nature of the underlying mechanisms. What is immediately clear is that signals from the L- and M-cones must interact to produce these results, because the amount of L-cone stimulation required to reach threshold depends heavily on how strongly the M-cones are stimulated (and vice versa): when the two cone classes are stimulated together (luminance modulation) approximately 2.5 times more L-cone contrast is required at threshold than when the L-cones are stimulated alone (compare the threshold L-cone contrast for the points having zero M-cone contrast with the L-cone contrast needed at the ends of the ellipse where the L- and M-cone contrasts are equal). Moreover, the lowest threshold (assessed in terms of the distance from the origin to the threshold point in the cone-contrast space) is obtained for the red-green isoluminant modulation, where the L- and M-cone contrasts are of the same size but opposite in sign. A natural interpretation of these data (e.g., Wandell, 1995; Stockman & Brainard, 2010) is that the detection of stimuli that can just be detected by L-cone excitation depends on the responses of cone-opponent L-M and M-L mechanisms (see also Chaparro, Stromeyer, Huang, Kronauer, & Eskew, 1993). As illustrated in Figure 6, such mechanisms can be formed from signals carried through the parvocellular layers of the LGN. 32

33 Cone-opponent mechanisms, of course, would be very insensitive to luminance modulations, because in that case the ON L- and OFF M- contrast signals that are combined to produce the L-M opponent signal would tend to cancel each other as would the OFF L- and ON M- signals producing the M-L signal. But, the luminance modulations could be detected by a non-coneopponent mechanism, formed from signals passing through either the parvocellular or the magnocellular pathway (or both; we neglect here consideration of the possibility that S-cone opponent mechanisms mediate the detection, but the same argument would apply to those as well). Moreover, because the long axis of the ellipse lies along the luminance direction, it is clear that under this assumption the non-cone-opponent mechanism is much less sensitive than the coneopponent mechanism. Indeed, the non-cone-opponent mechanisms would be expected to be relatively insensitive in this experiment because the stimuli used were large (relative to ganglion cell receptive fields) so that the spatial antagonism of the non-cone-opponent mechanisms would render them so. [Insert Figure 7 about here.] We can go further. A cone-opponent mechanism that summed L- and M-cone contrasts with equal weight but opposite sign would have a constant response for any set of stimuli where M/M = - L/L + k: straight lines with a slope of one in cone-contrast space with the constant k determining the distance from the origin. Many of the threshold measurements shown in Figure 7 are well approximated by straight lines of this form. Indeed, the dashed lines shown in Figure 7 along the upper and lower flanks of the data have the form M/M = - L/L ± Thus, all the threshold points lying close to this line can be explained under the hypothesis that they are detected by coneopponent mechanism with equally weighted L and M cone contrast. If this hypothesis is correct (but see Poirson, Wandell, Varner, & Brainard, 1990), the data reveal not only the action of the underlying cone-opponent mechanism but also the nature of this mechanism: it opposes L- and M- contrast with equal weight. 33

34 Threshold data can also reveal separate effects of first-site and second-site adaptation, when thresholds are measured on different backgrounds. Figure 8 (the data are replotted from Fig. 4 of Chaparro, Stromeyer, Chen, & Kronauer, 1995) shows the effects of changing the wavelength of the adapting field from 525 nm (green circles) to 579 nm (yellow squares) to 654 nm (red triangles). (Only the data along the flanks of the three ellipses are shown.) Similar to the data shown in Figure 7, contrast thresholds along the flanks are well-described as straight lines with positive unity slopes for all three backgrounds, consistent with detection by a cone-opponent mechanism that sums L- and M-cone contrasts with equal and opposite sign. This regularity when the cone signals are expressed as contrast is consistent with first-site adaptation, which converts isomerization to a contrast-code separately in each cone class. The displacement of the threshold contour away from the origin as the adapting field wavelength increases to 654 nm is characteristic of second-site adaptation. The idea is that the 654-nm background produces a larger signal in the cone-opponent mechanism than the other two backgrounds, and that this signal desensitizes the mechanism and leads to higher overall thresholds without a change in the shape of the threshold contours. Indeed, it is this feature of the data that suggests that in addition to their role at the first-site, chromatic backgrounds can produce second-site adaptation at a second-site without altering the relative sensitivities of the mechanism to M- and L- cone contrast inputs. The equality of cone contrast weights into the inferred L-M cone-opponent mechanism ( M/M = - L/L) is found under a variety of different conditions (e.g., Eskew, Stromeyer, & Kronauer, 1994; Chaparro, Stromeyer, Kronauer, & Eskew, 1994; Chaparro et al., 1995; Giulianini & Eskew, 1998). [Insert Figure 8 about here.] There is an extensive literature on cone-opponent and non-cone-opponent detection and discrimination mechanisms, which are often referred to as chromatic and luminance (or achromatic) 34

35 mechanisms (for reviews, see Eskew, McLellan, & Giulianini, 1999; Eskew, 2008; Stockman & Brainard, 2010). Another manifestation of the cone-opponent versus non-cone-opponent code can be seen in the results of psychophysical experiments that probe second-site adaptation using habituating stimuli (Krauskopf, Williams, & Heeley, 1982). Habituation refers to the loss of sensitivity following prolonged exposure or adaptation; in this case to a temporally varying adapting stimulus. Again, threshold contours like those shown in Figures 7 and 8 are measured, but both before and after the subject has been habituated to a temporally-varying adapting stimulus that is modulated in color space in either the same direction as the test stimulus (for which threshold is to be measured), or in a different direction. The results show that for habituating stimuli that are modulated in a color direction that excites principally the L-M and M-L cone-opponent mechanisms, threshold is selectively elevated for test stimuli that excite those same mechanisms, but the threshold is not elevated if the habituating modulation has no component in the test direction. The same thing happens with habituating stimuli that principally excite non-cone-opponent or luminance mechanisms. Critically, habituating stimuli that are modulated in intermediate color directions directions that excite both cone-opponent and non-cone-opponent mechanisms lead to nonselective increases in threshold for all test modulations regardless of the direction of the test modulation. These results imply the existence of second-site sensitivity regulation that occurs at or after the recombination of cone signals into cone-opponent and non-cone-opponent codes. The same paradigm may also be used to probe for the existence of higher-order mechanisms whose properties are determined by additional recombination that occurs after the LGN (Krauskopf, Williams, Mandler, & Brown, 1986; D'Zmura, 1991; Tailby, Solomon, Dhruv, & Lennie, 2008; Eskew, 2009; Hansen & Gegenfurtner, 2013). Psychophysics can also be used to study the more detailed properties of S-cone opponent and non-cone-opponent mechanisms. Techniques used for the latter include heterochromatic flicker 35

36 photometry, heterochromatic modulation photometry, minimally distinct border, and minimum motion, details of which can be found in several papers (Dresler, 1953; Boynton & Kaiser, 1968; Guth, Donley, & Marrocco, 1969; Le Grand, 1972; Wagner & Boynton, 1972; Wyszecki & Stiles, 1982; Pokorny, Smith, & Lutze, 1989; Lennie, Pokorny, & Smith, 1993; Stockman & Sharpe, 1999; Stockman & Sharpe, 2000). All of these techniques rely in some way on the assumption that coneopponent mechanisms are relatively insensitive to higher temporal or spatial frequencies. Such reduced sensitivity may be the result of processing that occurs after the LGN. As illustrated by the discussion of the data shown in Figure 7, the non-cone-opponent mechanisms may also be studied by choosing stimuli with cone contrasts that silence the other mechanisms, a technique known as silentsubstitution (Estévez & Spekreijse, 1974). In the case of the data shown in Figure 7, only the L-M and M-L mechanisms are likely to be silenced when the L- and M-cone contrasts are modulated together; both non-cone-opponent and S-cone opponent mechanisms are still stimulated. It is also possible to arrange modulations that silence multiple specified mechanisms (Brainard & Stockman, 2010; Vienot & Brettel, 2014). As indicated by the fact that both parvocellular and magnocellular pathways carry non-coneopponent information, sensitivity to luminance modulations as assessed psychophysically has a complex neural substrate. The existence of two distinct non-cone-opponent mechanisms has been frequently discussed (Ingling & Martinez-Uriegas, 1983; Ingling & Martinez, 1983; Ingling & Martinez-Uriegas, 1985; Ingling & Tsou, 1988). The need for a parvocellular-based non-coneopponent system is also suggested by the fact that the retinal distribution of magnocellular cells is too coarse to support the observed psychophysical acuity for spatial luminance modulations (e.g., Lennie & D'Zmura, 1988). Non-cone-opponent mechanisms based on magnocellular and parvocellular cells should also have different temporal properties, since magnocellular cells respond faster and are thus likely to be more responsive to higher temporal frequencies than parvocellular cells (Derrington & Lennie, 1984). 36

37 Relative cone numerosity and the cone-opponent and non-cone-opponent codes Recall that a striking feature of the cone mosaics shown in Figure 2 is the large individual variation the ratio of L- to M-cones in color normal observers. Such differences are not revealed by color matching behaviour measured for large (e.g., 2 diameter) uniform fields, because these stimuli cover many cones of each class even for the most asymmetric ratios observed, as a consequence of which the full trichromatic code remains available. In fact, more generally, the perceptual consequences of variation in L:M ratio appear to be small. It is possible, however, to observe systematic differences across individuals with different cone ratios in their sensitivity to luminance modulations as measured with the flicker ERG, probably because the flicker ERG represents a signal summed over all stimulated L- and M-cones (Brainard et al., 2000; Carroll et al., 2002; Hofer et al., 2005). The spectral sensitivity to luminance modulation thus represents a weighted sum of the known L- and M-cone spectral sensitivities with weights that reflect the proportion of L- and M- cones in each individuals retina. The correlation between flicker ERG and psychophysical estimates of sensitivity to luminance modulations (Chang, Burns, & Kreitz, 1993; Kremers, Scholl, Knau, Berendschot, & Sharpe, 2000) in turn suggests that one consequence of variation in L:M ratio is a shift in the effective spectral sensitivity of the underlying non-cone-opponent mechanisms, consistent with the hypothesis that these draw randomly on the L- and M-cones in the mosaic. In contrast to the dependence of sensitivity to luminance modulations on the L:M cone ratio in the mosaic, the wavelength of unique yellow (a stimulus that by assumption silences an L-M/M-L opponent mechanism somewhere along the visual pathways and thus looks uniquely yellow neither reddish-yellow nor greenish-yellow) varies little across observers with different cone ratios (Brainard et al., 2000). This may be because a long-term adaptation process normalizes the relative L- and M- cone input, so that the adapted mechanism is not driven by the mean spectrum that reaches the eye from natural images (Yamauchi et al., 2002; Neitz, Carroll, Yamauchi, Neitz, & Williams, 2002). 37

38 Temporal aspects of color vision The top panel in Figure 9 shows the contrast sensitivity (the reciprocal of threshold contrast) plotted as a function of temporal frequency (Hz) for a luminance modulation (inverted black triangles). To obtain these measurements, the stimulus was flickered sinusoidally around a constant mean background intensity level at each temporal frequency and its contrast (defined as the maximum excursion of the stimulus from the background divided by the mean intensity of the background) was varied to find the threshold for seeing flicker. Such measurements produce a function known as a temporal contrast sensitivity function (CSF). The particular luminance modulation used for the measurements in Figure 9 differed from the one used in the measurements for Figure 7; for the data in Figure 9 the stimulus modulated the contrast of all three cone classes equally and in phase. Such a luminance modulation is also known as an isochromatic modulation, and when seen against an achromatic background such modulations appear achromatic. In addition to silencing L-M and M-L cone-opponent mechanisms, an isochromatic modulation is also thought to silence S-cone opponent mechanisms. The temporal CSF for the isochromatic modulation has an inverted u-shape (also known as a bandpass shape) and peaks near 10 Hz for the stimulus conditions used (Figure 9; de Lange, 1958; Kelly, 1961). Above we introduced the idea that first-site sensitivity regulation shortens the integration time of cone photoreceptors as light level increases. One psychophysical consequence of this shortening is a change in the form of the temporal contrast-sensitivity function with increasing overall light level. As described just above, for low spatial-frequency isochromatic modulations, the temporal luminance CSF has a bandpass shape. Critically, as background intensity increases the temporal frequency of peak sensitivity increases, as does the prominence of the peak (de Lange, 1958; Kelly, 1961). These effects are consistent with a shortening of the integration time in cone sensitivity regulation. A second psychophysical correlate of the speed-up in processing can be observed through measurements of the lag in processing between stimuli presented separately to the 38

39 two eyes when a difference in background light level between the eyes is introduced (e.g., Stockman et al., 2006). [Insert Figure 9 about here.] The top panel of Figure 9 also shows measurements of the temporal CSF for a red-green isoluminant modulation (red circles) designed to silence the non-cone-opponent and S-cone opponent mechanisms. By assumption this CSF reveals the temporal properties of the L-M and M-L cone-opponent mechanisms (Kelly & van Norren, 1977). The red-green isoluminant temporal CSF is restricted to lower temporal frequencies than the luminance CSF. It is also much less bandpass in form, with a very broad peak at much lower frequencies near 3 Hz (see Stockman & Brainard, 2010). Spatial aspects of color vision Dependence of color matches on spatial frequency Our discussion of color matching was developed for spatially uniform stimuli. If instead we consider stimuli with spatial structure, differential blurring of the retinal image as a function of wavelength makes the computation of matches more complex. To illustrate this concept, consider the stripes of a sinusoidal grating produced by spatial sinusoidal modulations across a uniform background. At very low spatial frequencies, blur has a negligible effect and a color match between two sinusoids of different spectral composition can be predicted directly using the methods described for spatially uniform fields. As the spatial frequency of the grating increases, however, differential blurring of two spectral metamers can break the color match, because the blurring depends on the wavelength of the components of the metamers and not upon their color appearance. Marimont and Wandell (1994) discuss this phenomenon in detail and provide a method for predicting color matches for spatially complex stimuli. Spatial MTFs To assess the overall ability of the visual system to resolve spatial structure, a fundamental psychophysical measurement is that of the spatial contrast sensitivity function (de Lange, 1958; Kelly, 1966; Robson, 1966; Campbell & Robson, 1968). This is the spatial analogue of the temporal 39

40 CSF discussed above. The stimulus is a sinusoidal grating (often with its contrast windowed by a Gaussian envelope, such that the contrast decreases toward the edges of the grating). Contrast threshold is measured as a function of the spatial frequency of the grating. And, of course, the color direction of the spatial modulation can be chosen to try to isolate different underlying mechanisms. Figure 9 (bottom panel) plots the spatial CSF for two different color directions. Normalized contrast sensitivity is plotted against spatial frequency (cycles per degree of visual angle or c/deg). Both axes are logarithmic. The measurements were made using an interferometric technique that eliminates the blurring introduced by the eye s optics and thus exposes the neural limits of spatial resolution. For the isochromatic luminance modulation (black triangles), the contrast of all three cone classes was modulated together. As noted above, this is thought to silence the action of coneopponent mechanisms while strongly driving non-cone-opponent mechanisms. The maximum spatial frequency that the visual system can resolve for isochromatic modulations is about 60 c/deg (Williams, 1985) the measurements shown were not extended to higher spatial frequencies, which can be detected via spatial patterns (aliasing) that are produced by the sampling of the interferometric gratings by the photoreceptor mosaic. Generally, contrast sensitivity increases as the gratings become coarser below 50 c/deg, until it reaches a peak at about 5 c/deg (for reasonably bright adapting conditions). At still lower frequencies, contrast sensitivity decreases producing the familiar band-pass shape (Robson, 1966; Kelly, 1966). Performance is different when a red-green isoluminant modulation stimulates the L- and M-cones such that the contrasts silence the non-cone-opponent mechanisms (red squares). First, the shape of the CSF is low pass, with highest sensitivity found at the lowest spatial frequencies measured. Indeed, when the data are normalized as they are here, to take into account the different contrasts seen by the L- and M-cones between the isochromatic and isoluminant modulations, overall sensitivity is higher for the red-green isoluminant modulation at low frequencies (see Sekiguchi, 40

41 Williams, & Brainard, 1993b; Chaparro et al., 1993). A comparable difference is also apparent in the data plotted in Figure 7, above. Second, the falloff in sensitivity with increasing spatial frequency is more rapid for red-green than for isochromatic gratings, and red-green isoluminant stimuli at spatial frequencies above about cycles/deg (depending on observer) cannot be discriminated from uniform fields, even in the absence of optical blur (Sekiguchi, Williams, & Brainard, 1993a). [This limit is lower still when measured through un-bypassed optics (Mullen, 1985; Anderson, Mullen, & Hess, 1991).] The spatial CSF when only S-cone contrast is modulated (not shown) falls off faster than for redgreen isoluminant modulations for conventional viewing where the S-cone image is more blurred than the L- and M-cone image (Humanski & Wilson, 1993, 1992). Interestingly, however, the role of neural factors in shaping the spatial CSF is about the same for S-cone and red-green isoluminant modulations. This conclusion is inferred from experiments that bypass the optics or minimize chromatic aberration and from analyses that take into account information loss due to the sparser sampling of the S-cone sub-mosaic (Mullen, 1985; Sekiguchi et al., 1993b). The spatial CSF does not capture all aspects of spatial resolution. For example, although signals seen by S-cones can be discriminated from uniform fields at moderate spatial frequencies if the gratings are focussed to eliminate the effects of chromatic aberration, such gratings are not seen veridically. Rather, the coarse sampling by the S-cones leads to aliasing; that is, the S-cone stimuli are detected as splotchy low spatial frequency patterns rather than as gratings (Williams & Collier, 1983). In addition, it is important to note that the spatial CSF depends on a number of ancillary factors, such as the luminance of the background as well as the size and temporal properties of the stimulus. There has been some progress characterizing how the spatial CSF varies systematically with color direction (Poirson & Wandell, 1993, 1996). 41

42 Foveal and peripheral differences Most high-resolution seeing is done with the fovea, using eye movements to scan the fovea across the scene to gather high-resolution information. Indeed, a major function of peripheral vision is probably to help guide foveation (Carpenter, 1977). Many aspects of vision are less acute for stimuli imaged on peripheral retina, with a major exception being our ability to process both rapid flicker and movement and our ability to see with the rods at night. Perhaps not surprisingly, two major factors governing the fall-off in performance are the optics of the eye and the sampling by the retinal mosaic. As eccentricity increases, so does the amount of optical blurring (Jennings & Charman, 1981; Williams, Artal, Navarro, McMahon, & Brainard, 1996; Navarro, Moreno, & Dorronsoro, 1998). In addition, cone density decreases with eccentricity (Østerberg, 1935; Curcio, Sloan, Kalina, & Hendrickson, 1990). These two factors contribute to an overall decrease in spatial resolution with eccentricity. More interesting for present purposes is the fact that chromatic discrimination also falls off with eccentricity. The top panel of Figure 10 shows the contrast sensitivity for eight degree disks as a function of the eccentricity of the disk center for modulations in three color directions: isochromatic (inverted black triangles), red-green isoluminant (red circles), and S-cone (blue squares). The data are psychophysical measurements of subjects ability to discriminate color and luminance changes from uniform gray backgrounds replotted from Hansen et al. (2009). Of note is that sensitivity for isoluminant red-green contrast falls off more rapidly than sensitivity for luminance or S-cone contrast (see also Mullen & Kingdom, 2002; Mullen, Sakurai, & Chu, 2005). Moreover, there is evidence in the periphery for an asymmetry in red-green detection, with sensitivity greater for stimuli that would drive an LON-MOFF unipolar (rectified) mechanism compared to those that would drive an MON-LOFF unipolar mechanism (see Stromeyer, Lee, & Eskew, 1992; Newton & Eskew, 2003; Sakurai & Mullen, 2006). 42

43 One idea about the faster falloff in sensitivity for red-green stimuli derives from an important difference in the midget ganglion cell system across eccentricities. As described above, in central retina, each midget cell center receives input from a single cone. This means that this system, in essence, can preserve the chromatic information available in the cone mosaic, since in the centers the signals from L- and M-cones remain unmixed. In the periphery, however, midget centers draw on multiple cones, and it appears that these centers generally mix signals from L- and M-cones though the mixing may not be entirely random (e.g., Diller et al., 2004; Field et al., 2010). Any mixing would not affect the information available for luminance-mediated discriminations, but would reduce the information available for red-green isoluminant discriminations. Thus the more rapid fall-off in isoluminant red-green color discrimination may reflect the fact that L-M cone opponent signals are preserved in the retina not through specialized retinal circuitry but rather because this preservation can piggy-back on the primate s foveal single-cone center high-resolution midget ganglion cell system. Increasing mixing of L and M-cone signals with eccentricity alone, however, cannot account for the differential loss of the LON-MOFF mediated sensitivity relative to MON-LOFF sensitivity in the periphery, which suggests that other factors must also be involved. [Insert Figure 10 about here.] Conclusion We have considered color processing in the eye as a succession of stages: the formation of the retinal image, the transduction of the image by the mosaic of cone photoreceptors, and the processing of the cone responses by retinal circuitry. The properties of each successive stage place fundamental limits on the information available to subsequent stages, and an understanding of those limits is essential to any understanding of color perception. Our approach has been to link optics, molecular processes, molecular genetics, physiology, and psychophysics. The convergence of information from these different disciplines has led to a much clearer understanding of early color processing, yet much remains to be discovered particularly in the context of color processing as a whole. In such a 43

44 short chapter, we have not touched on many complexities such as the detection of increments versus decrements, the effects of visual non-linearities, additional richness in second-site adaptation, disagreements between thresholds and appearance in the details of opponency, more elaborated multi-stage models, and so on. Color processing after the LGN is treated in the chapter by Gegenfurtner and Ennis. Acknowledgements We thank Bruce Henning for comments on several versions of this manuscript, and Rhea Eskew for comments and help with Figure 7. We are also grateful to E.J. Chichilnisky, Dennis Dacey, Ana Radonjić, Fred Reike and Sam Solomon for helpful comments on a near-final draft. Supported by the National Institutes of Health, RO1 EY10016 (DHB), by the Biotechnology and Biological Sciences Research Council (AS), and by the Engineering and Physical Sciences Research Council (AS). References Adelson, E. H., & Bergen, J. R. (1991). The plenoptic function and the elements of early vision. In M. S. Landy & J. A. Movshon (Eds.), Computational Models of Visual Processing (pp. 3-20). Cambridge: MIT Press. Ahnelt, P. K., Keri, C., & Kolb, H. (1990). Identification of pedicles of putative blue sensitive cones in human and primate retina. Journal of Comparative Neurology, 293, Ahnelt, P. K., & Kolb, H. (1994). Horizontal cells and cone photoreceptors in primate retina: a Golgi-light microscope study of spectral connectivity. Journal of Comparative Neurology, 343, Alpern, M., Rushton, W. A. H., & Torii, S. (1970). Signals from cones. Journal of Physiology, 207(2), Anderson, S. J., Mullen, K. T., & Hess, R. F. (1991). Human peripheral spatial resolution for achromatic and chromatic stimuli: limits imposed by optical and retinal factors. Journal of Physiology, 442, Angueyra, J. M., & Rieke, F. (2013). Origin and effect of phototransduction noise in primate cone photoreceptors. Nature Neuroscience, 16(11),

45 Arshavsky, V. Y., Lamb, T. D., & Pugh, E. N., Jr. (2002). G proteins and phototransduction. Annual Review of Physiology, 64, Atick, J. J. (1992). Could information theory provide an ecological theory of sensory processing. Network-Computation in Neural Systems, 3(2), Atick, J. J., Li, Z. P., & Redlich, A. N. (1992). Understanding retinal color coding from first principles. Neural Computation, 4(4), Attneave, F. (1954). Some informational aspects of visual perception. Psychological Review, 61, Autrusseau, F., Thibos, L., & Shevell, S. K. (2011). Chromatic and wavefront aberrations: L-, M- and S-cone stimulation with typical and extreme retinal image quality. Vision Research, 51(21-22), Barlow, H. B. (1961). Possible principles underlying the transformations of sensory messages. In W. A. Rosenblith (Ed.), Sensory communication (pp ). Cambridge, MA: M.I.T. Press. Baylor, D. A., Nunn, B. J., & Schnapf, J. L. (1987). Spectral sensitivity of cones of the monkey Macaca fascicularis. Journal of Physiology, 390, Berson, D. M. (2003). Strange vision: ganglion cells as circadian photoreceptors. Trends Neurosci, 26(6), Berson, D. M. (2014). Intrinsically photosensitive ganglion cells. In L. M. Chalupa & J. S. Werner (Eds.), The New Visual Neurosciences (pp ). Cambridge, MA: MIT Press. Billock, V. A. (1991). The relationship between simple and double opponent cells Vision Research, 31(1), Bongard, M. M., & Smirnov, M. S. (1954). Determination of the eye spectral sensitivity curves from spectral mixture curves. Doklady Akademiia nauk S.S.S.R., 102, Boycott, B. B., & Dowling, J. E. (1969). Organisation of the primate retina: Light microscopy. Philosophical Transactions of the Royal Society of London. Series B, 255, Boycott, B. B., Hopkins, J. M., & Sperling, H. G. (1987). Cone connections of the horizontal cells of the rhesus monkey's retina. Proceedings of the Royal Society of London. Series B, 229(1257), Boycott, B. B., & Wässle, H. (1991). Morphological classification of bipolar cells of the primate retina. European Journal of Neuroscience, 3(11), Boynton, R. M. (1979). Human Color Vision. New York: Holt, Rinehart and Winston. Boynton, R. M., & Kaiser, P. (1968). Vision: the additivity law made to work for heterochromatic photometry with bipartite fields. Science, 161,

46 Brainard, D. H. (1998). Hyperspectral image data. Retrieved May 1, 2014, from Brainard, D. H., Hofer, H., & Wandell, B. A. (2013). Wavefront optics: Matlab toolbox for analyzing wavefront optics data; especially human adaptive optics measurements. Retrieved May 1, 2014, from Brainard, D. H., Roorda, A., Yamauchi, Y., Calderone, J. B., Metha, A., Neitz, M., Neitz, J., Williams, D. R., & Jacobs, G. H. (2000). Functional consequences of the relative numbers of L and M cones. Journal of the Optical Society of America A, 17(3), Brainard, D. H., & Stockman, A. (2010). Colorimetry. In M. Bass, C. DeCusatis, J. Enoch, V. Lakshminarayanan, G. Li, C. Macdonald, V. Mahajan & E. van Stryland (Eds.), The Optical Society of America Handbook of Optics, 3rd edition, Volume III: Vision and Vision Optics (pp ). New York: McGraw Hill. Briggs, F., & Usrey, W. M. (2011). Corticogeniculate feedback and visual processing in the primate. Journal of Physiology, 589(1), Brindley, G. S. (1955). The colour of light of very long wavelength. Journal of Physiology, 130, Brindley, G. S. (1970). Physiology of the Retina and the Visual Pathway (2nd. ed.). Baltimore: Williams and Wilkins. Brown, T. M., Tsujimura, S., Allen, A. E., Wynne, J., Bedford, R., Vickery, G., Vugler, A., & Lucas, R. J. (2012). Melanopsin-Based Brightness Discrimination in Mice and Humans. Current Biology, 22(12), Buchsbaum, G., & Gottschalk, A. (1983). Trichromacy, opponent colours coding and optimum colour information transmission in the retina. Proceedings of the Royal Society of London. Series B, 220(1218), Buck, S. L. (2014). The interaction of rod and cone signals: pathways and psychophysics. In L. M. Chalupa & J. S. Werner (Eds.), The New Visual Neurosciences (pp ). Cambridge, MA: MIT Press. Burns, M. E., & Baylor, D. A. (2001). Activation, deactivation and adaptation in vertebrate photoreceptor cells. Annual Review of Neuroscience, 24, Buzas, P., Blessing, E. M., Szmajda, B. A., & Martin, P. R. (2006). Specificity of M and L cone inputs to receptive fields in the parvocellular pathway: random wiring with functional bias. Journal of Neuroscience, 26(43),

47 Calkins, D. J., Schein, S. J., Tsukamoto, Y., & Sterling, P. (1994). M and L cones in macaque fovea connect to midget ganglion cells by different numbers of excitatory synapses. Nature, 371(6492), Calkins, D. J., Tsukamato, Y., & Sterling, P. (1998). Microcircuitry and mosaic of a blue-yellow ganglion cell in the primate retina. Journal of Neuroscience, 18(9), Campbell, F. W., & Robson, J. G. (1968). Application of Fourier analysis to the visibility of gratings. Journal of Physiology, 197(3), Carpenter, R. H. S. (1977). Movements of the eyes. London: Pion. Carroll, J., Neitz, J., & Neitz, M. (2002). Estimates of L:M cone ratio from ERG flicker photometry and genetics. Journal of Vision, 2(8), Casagrande, V. A. (1994). A third parallel visual pathway to primate area V1. Trends Neurosci, 17(7), Chakrabarti, A., & Zickler, T. (2011). Statistics of real-world hyperspectral images. Paper presented at the Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Chang, Y., Burns, S. A., & Kreitz, M. R. (1993). Red-green flicker photometry and nonlinearities in the flicker electroretinogram. Journal of the Optical Society of America A, 10(6), Chaparro, A., Stromeyer, C. F., III, Chen, G., & Kronauer, R. E. (1995). Human cones appear to adapt at low light levels: Measurements on the red-green detection mechanism. Vision Research, 35(22), Chaparro, A., Stromeyer, C. F., III, Huang, E. P., Kronauer, R. E., & Eskew, R. T., Jr. (1993). Colour is what the eye sees best. Nature, 361(6410), Chaparro, A., Stromeyer, C. F., III, Kronauer, R. E., & Eskew, R. T., Jr. (1994). Separable red-green and luminance detectors for small flashes. Vision Research, 34(6), Charman, N. (2010). Optics of the eye. In M. Bass, C. DeCusatis, J. Enoch, V. Lakshminarayanan, G. Li, C. Macdonald, V. Mahajan & E. van Stryland (Eds.), The Optical Society of America Handbook of Optics, 3rd edition, Volume III: Vision and Vision Optics (pp ). New York: McGraw Hill. Chen, S., & Li, W. (2012). A color-coding amacrine cell may provide a blue-off signal in a mammalian retina. Nature Neuroscience, 15(7), Cicerone, C. M., & Nerger, J. L. (1989). The relative numbers of long-wavelength-sensitive to middle-wavelength-sensitive cones in the human fovea centralis. Vision Research, 29(1),

48 CIE. (2006). Fundamental chromaticity diagram with physiological axes Part 1. Technical Report Vienna: Central Bureau of the Commission Internationale de l' Éclairage. Commission, I. E. (1999). srgb Standard, International Electrotechnical Commission Standard Conway, B. R., Chatterjee, S., Field, G. D., Horwitz, G. D., Johnson, E. N., Koida, K., & Mancuso, K. (2010). Advances in color science: from retina to behavior. Journal of Neuroscience, 30(45), Cornsweet, T. (1970). Visual Perception. New York: Academic Press. Curcio, C. A., Allen, K. A., Sloan, K. R., Lerea, C. L., Hurley, J. B., Klock, I. B., & Milam, A. H. (1991). Distribution and morphology of human cone photoreceptors stained with anti-blue opsin. Journal of Comparative Neurology, 312(4), Curcio, C. A., Sloan, K. R., Kalina, R. E., & Hendrickson, A. E. (1990). Human photoreceptor topography. Journal of Comparative Neurology, 292(4), D'Zmura, M. (1991). Color in visual search. Vision Research, 31(6), Dacey, D. M. (2000). Parallel pathways for spectral coding in primate retina. Annual Review of Neuroscience, 23, Dacey, D. M., Crook, J. D., & Packer, O. S. (2014). Distinct synaptic mechanisms create parallel S- ON and S-OFF color opponent pathways in the primate retina. Visual Neuroscience, 31(Special Issue 02), Dacey, D. M., & Lee, B. B. (1994). The 'blue-on' opponent pathway in primate retina originates from a distinct bistratified ganglion cell type. Nature, 367(6465), Dacey, D. M., Lee, B. B., Stafford, D. K., Pokorny, J., & Smith, V. C. (1996). Horizontal cells of the primate retina: cone specificity without spectral opponency. Science, 271(5249), Dacey, D. M., Liao, H.-W., Peterson, B. B., Robinson, F. R., Smith, V. C., Pokorny, J., Yau, K.-W., & Gamlin, P. D. (2005). Melanopsin-expressing ganglion cells in primate retina signal colour and irradiance and project to the LGN. Nature, 433(7027), Dacey, D. M., & Packer, O. S. (2003). Colour coding in the primate retina: diverse cell types and cone-specific circuitry. Current Opinion in Neurobiology, 13(4), de Lange, H. (1958). Research into the dynamic nature of the human fovea-cortex systems with intermittent and modulated light. I. Attenuation characteristics with white and colored light. Journal of the Optical Society of America, 48, De Valois, R. L., & De Valois, K. K. (1993). A multi-stage color model. Vision Research, 33(8),

49 Deeb, S. S. (2005). The molecular basis of variation in human color vision. Clinical Genetics, 67(5), Derrington, A. M., Krauskopf, J., & Lennie, P. (1984). Chromatic mechanisms in lateral geniculate nucleus of macaque. Journal of Physiology, 357, Derrington, A. M., & Lennie, P. (1984). Spatial and temporal contrast sensitivities of neurones in the lateral geniculate nucleus of macaque. Journal of Physiology, 357, DeVries, S. H., Qi, X., Smith, R. A., Makous, W., & Sterling, P. (2002). Electrical coupling between mammalian cones. Current Biology, 12(22), Diller, L., Packer, O. S., Verweij, J., McMahon, M. J., Williams, D. R., & Dacey, D. M. (2004). L and M cone contributions to the midget and parasol ganglion cell receptive fields of macaque monkey retina. Journal of Neuroscience, 24(5), Do, M. T., & Yau, K. W. (2010). Intrinsically photosensitive retinal ganglion cells. Physiological Reviews, 90(4), Dowling, J. E. (1987). The retina, an approachable part of the brain. Cambridge, Mass.: Harvard University Press. Dresler, A. (1953). The non-additivity of heterochromatic brightness. Transactions of the Illuminating Engineering Society, 18, Eskew, R. T., Jr. (2008). Chromatic detection and discrimination. In T. D. Albright & R. H. Masland (Eds.), The Senses: A Comprehensive Reference, Volume 2: Vision II (pp ). San Diego: Academic Press Inc. Eskew, R. T., Jr. (2009). Higher order color mechanisms: A critical review. Vision Research, 49(22), Eskew, R. T., Jr., McLellan, J. S., & Giulianini, F. (1999). Chromatic detection and discrimination. In K. Gegenfurtner & L. T. Sharpe (Eds.), Color vision: From Genes to Perception (pp ). Cambridge: Cambridge University Press. Eskew, R. T., Jr., Stromeyer, C. F., III, & Kronauer, R. E. (1994). Temporal properties of the redgreen chromatic mechanism. Vision Research, 34(23), Estévez, O. (1979). On the fundamental database of normal and dichromatic color vision. (Ph.D.), Amsterdam University. Estévez, O., & Spekreijse, H. (1974). A spectral compensation method for determining the flicker characteristics of the human color mechanisms. Vision Research, 14, Field, G. D., & Chichilnisky, E. J. (2007). Information processing in the primate retina: circuitry and coding. Annual Review of Neuroscience, 30,

50 Field, G. D., Gauthier, J. L., Sher, A., Greschner, M., Machado, T. A., Jepson, L. H., Shlens, J., Gunning, D. E., Mathieson, K., Dabrowski, W., Paninski, L., Litke, A. M., & Chichilnisky, E. J. (2010). Functional connectivity in the retina at the resolution of photoreceptors. Nature, 467(7316), Gamlin, P. D. R., McDougal, D. H., Pokorny, J., Smith, V. C., Yau, K. W., & Dacey, D. M. (2007). Human and macaque pupil responses driven by melanopsin-containing retinal ganglion cells. Vision Research, 47(7), Garrigan, P., Ratliff, C. P., Klein, J. M., Sterling, P., Brainard, D. H., & Balasubramanian, V. (2010). Design of a trichromatic cone array. PLoS Comput Biol, 6(2), e Gegenfurtner, K. R., & Kiper, D. C. (2003). Color Vision. Annual Review of Neuroscience, 26, Geisler, W. S. (1989). Sequential ideal-observer analysis of visual discriminations. Psychological Review, 96(2), Geisler, W. S. (2011). Contributions of ideal observer theory to vision research. Vision Research, 51(7), Giulianini, F., & Eskew, R. T., Jr. (1998). Chromatic masking in the ( L/L, M/M) plane of conecontrast space reveals only two detection mechanisms. Vision Research, 38, Goodchild, A. K., Chan, T. L., & Grünert, U. (1996). Horizontal cell connections with shortwavelength-sensitive cones in macaque monkey retina. Visual Neuroscience, 13(5), Grassmann, H. (1853). Zur Theorie der Farbenmischung Annalen der Physik und Chemie, 165, Grünert, U., Martin, P. R., & Wässle, H. (1994). Immunocytochemical analysis of bipolar cells in the macaque monkey retina. Journal of Comparative Neurology, 348(4), Guth, S. L. (1991). A model for color and light adaptation. Journal of the Optical Society of America A, 8(6), Guth, S. L., Donley, N. V., & Marrocco, R. T. (1969). On luminance additivity and related topics. Vision Research, 9(5), Hamer, R. D., Nicholas, S. C., Tranchina, D., Lamb, T. D., & Jarvinen, J. L. P. (2005). Toward a unified model of vertebrate rod phototransduction. Visual Neuroscience, 22(4), Hansen, T., & Gegenfurtner, K. R. (2013). Higher order color mechanisms: Evidence from noisemasking experiments in cone contrast space. Journal of Vision, 13(1), Hansen, T., Pracejus, L., & Gegenfurtner, K. R. (2009). Color perception in the intermediate periphery of the visual field. Journal of Vision, 9(4),

51 Haverkamp, S., Grunert, U., & Wassle, H. (2000). The cone pedicle, a complex synapse in the retina. Neuron, 27(1), Hecht, E. (1990). Optics (2nd ed.). Reading, MA: Addison-Wesley Publishing Company. Heckaman, R. L., & Fairchild, M. D. (2009). Jones and Condit redux in high dynamic range and color. Color and Imaging Conference, 2009(1), Hendry, S. H. C., & Reid, R. C. (2000). The koniocellular pathway in primate vision. Annual Review of Neuroscience, 23, Hering, E. (1878). Zur Lehre vom Lichtsinne. Sechs Mittheilungen an die Kaiserliche Akademie der Wissenschaften in Wien. Wien: Carl Gerold's Sohn. Hering, E. (1920). Grundzüge der Lehre vom Lichtsinn. Springer: Berlin. Herr, S., Klug, K., Sterling, P., & Schein, S. (2003). Inner S-cone bipolar cells provide all of the central elements for S cones in macaque retina. Journal of Comparative Neurology, 457(457), Hofer, H. J., Carroll, J., Neitz, J., Neitz, M., & Williams, D. R. (2005). Organization of the human trichromatic cone mosaic. Journal of Neuroscience, 25(42), Hofer, H. J., & Williams, D. R. (2014). Color vision and the retinal mosaic. In L. M. Chalupa & J. S. Werner (Eds.), The New Visual Neurosciences (pp ). Cambridge, MA: MIT Press. Hood, D. C., & Finkelstein, M. A. (1986). Sensitivity to light. In K. Boff, L. Kaufman & J. Thomas (Eds.), Handbook of perception and human performance (Vol. 1, pp ). New York: Wiley. Hopkins, J. M., & Boycott, B. B. (1995). Synapses between cones and diffuse bipolar cells of a primate retina. Journal of Neurocytology, 24(9), Horiguchi, H., Winawer, J., Dougherty, R. F., & Wandell, B. A. (2013). Human trichromacy revisited. Proceedings of the National Academy of Science U.S.A., 110(3), E260-E269. Hornstein, E. P., Verweij, J., & Schnapf, J. L. (2004). Electrical coupling between red and green cones in primate retina. Nature Neuroscience, 7(7), Hubel, D. H., & Wiesel, T. N. (1977). Functional architecture of macaque monkey visual-cortex. Proceedings of the Royal Society Series B-Biological Sciences, 198(1130), Humanski, R. A., & Wilson, H. R. (1992). Spatial frequency mechanisms with short-wavelengthsensitive cone inputs. Vision Research, 32(3), Humanski, R. A., & Wilson, H. R. (1993). Spatial-frequency adaptation: Evidence for a multiplechannel model of short-wavelength-sensitive-cone spatial vision. Vision Research, 33(5-6), Hurvich, L. M. (1981). Color Vision. Sunderland, Massachussets: Sinauer. 51

52 Ingling, C. R., Jr., & Drum, B. A. (1973). Retinal receptive fields: correlations between psychophysics and electrophysiology. Vision Research, 13(6), Ingling, C. R., Jr., & Martinez-Uriegas, E. (1983). The relationship between spectral sensitivity and spatial sensitivity for the primate r-g X-channel. Vision Research, 23(12), Ingling, C. R., Jr., & Martinez-Uriegas, E. (1985). The spatiotemporal properties of the r-g X-cell channel. Vision Research, 25(1), Ingling, C. R., Jr., & Martinez, E. (1983). The spatio-chromatic signal of the r-g channels. In J. D. Mollon & L. T. Sharpe (Eds.), Colour Vision: Physiology and Psychophysics (pp ). London: Academic Press. Ingling, C. R., Jr., & Tsou, H. B.-P. (1988). Spectral sensitivity for flicker and acuity criteria. Journal of the Optical Society of America A, 5(8), Jennings, J. A., & Charman, W. N. (1981). Off-axis image quality in the human eye. Vision Research, 21(4), Jusuf, P. R., Martin, P. R., & Grünert, U. (2006). Random wiring in the midget pathway of primate retina. Journal of Neuroscience, 26(15), Kelly, D. H. (1961). Visual responses to time-dependent stimuli I. Amplitude sensitivity measurements. Journal of the Optical Society of America, 51, Kelly, D. H. (1966). Frequency doubling in visual responses. Journal of the Optical Society of America, 56(11), Kelly, D. H. (1983). Spatiotemporal variation of chromatic and achromatic contrast thresholds. Journal of the Optical Society of America, 73(6), Kelly, D. H., & van Norren, D. (1977). Two-band model of heterochromatic flicker. Journal of the Optical Society of America, 67(8), Kingdom, F. A. A., & Mullen, K. T. (1995). Separating colour and luminance information in the visual system. Spatial Vision, 9(2), Klug, K., Herr, S., Ngo, T. N., Sterling, P., & Schein, S. (2003). Macaque retina contains an S-cone OFF midget pathway. Journal of Neuroscience, 23(30), Kolb, H. (1970). Organization of the outer plexiform layer of the primate retina: electron microscopy of Golgi-impregnated cells. Philosophical Transactions of the Royal Society of London. Series B, 258(823), Kolb, H., & Dekorver, L. (1991). Midget ganglion cells of the parafovea of the human retina: a study by electron microscopy and serial reconstructions. Journal of Comparative Neurology, 303(4),

53 Kolb, H., & Marshak, D. (2003). The midget pathways of the primate retina. Documenta Ophthalmologica, 106(1), König, A., & Dieterici, C. (1886). Die Grundempfindungen und ihre Intensitäts-Vertheilung im Spectrum. Sitzungsberichte Akademie der Wissenschaften, Berlin, 1886, Kouyama, N., & Mashak, D. W. (1992). Bipolar cells specific for blue cones in the macaque retina. Journal of Neuroscience, 12(4), Krantz, D. H. (1975). Color measurement and color theory: I. Representation theorem for Grassmann structures. Journal of Mathematical Psychology, 12, Krauskopf, J., Williams, D. R., & Heeley, D. W. (1982). Cardinal directions of color space. Vision Research, 22(9), Krauskopf, J., Williams, D. R., Mandler, M. B., & Brown, A. M. (1986). Higher order color mechanisms. Vision Research, 26(1), Kremers, J., Scholl, H. P. N., Knau, H., Berendschot, T. T. J. M., & Sharpe, L. T. (2000). L/M -cone ratios in human trichromats assessed by psychophysics, electroretinography and retinal densitometry. Journal of the Optical Society of America A, 17(3), Lankheet, M. J. M., Lennie, P., & Krauskopf, J. (1998). Distinctive characteristics of subclasses of red-green P-cells in LGN of macaque. Visual Neuroscience, 15(1), Le Grand, Y. (1972). Spectral luminosity. In D. Jameson & L. M. Hurvich (Eds.), Visual Psychophysics, Handbook of Sensory Physiology (Vol. VII/4, pp ). Berlin: Springer- Verlag. Lee, B. B., Kremers, J., & Yeh, T. (1998). Receptive fields of primate retinal ganglion cells studied with a novel technique. Visual Neuroscience, 15(1), Lee, S. C. S., Telkes, I., & Grünert, U. (2005). S-cones do not contribute to the OFF-midget pathway in the retina of the marmoset, Callithrix jacchus. European Journal of Neuroscience, 22, Lennie, P. (1984). Recent developments in the physiology of color vision. Trends Neurosci, 7(7), Lennie, P., & D'Zmura, M. (1988). Mechanisms of color vision. CRC Critical Reviews in Neurobiology, 3(4), Lennie, P., Haake, P. W., & Williams, D. R. (1991). The design of chromatically opponent receptive fields. In M. S. Landy & J. A. Movshon (Eds.), Computational models of visual processing (pp ). Cambridge, Mass.: MIT Press. Lennie, P., & Movshon, J. A. (2005). Coding of color and form in the geniculostriate visual pathway. Journal of the Optical Society of America A, 10(10),

54 Lennie, P., Pokorny, J., & Smith, V. C. (1993). Luminance. Journal of the Optical Society of America A, 10(6), Leventhal, A. G., Rodieck, R. W., & Dreher, B. (1981). Retinal ganglion-cell classes in the oldworld monkey - morphology and central projections. Science, 213(4512), Li, W., & DeVries, S. H. (2004). Separate blue and green cone networks in the mammalian retina. Nature Neuroscience, 7(7), MacLeod, D. I. A. (1978). Visual sensitivity. Annual Review of Psychology, 29, Mariani, A. P. (1984). Bipolar cells in monkey retina selective for the cones likely to be bluesensitive. Nature, 308(5955), Marimont, D. H., & Wandell, B. A. (1994). Matching Color Images - the Effects of Axial Chromatic Aberration. Journal of the Optical Society of America A, 11(12), Marrocco, R. T. (1976). Sustained and transient cells in monkey lateral geniculate nucleus: conduction velocites and response properties. Journal of Neurophysiology, 39(2), Marshak, D. W., & Martin, P. R. (2014). Short wavelength-sensitive cones and the processing of their signals. Visual Neuroscience, 31(Special Issue 02), Martin, P. R., & Lee, B. B. (2014). Distribution and specificity of S-cone ( blue cone ) signals in subcortical visual pathways. Visual Neuroscience, 31(Special Issue 02), Martin, P. R., White, A. J., Goodchild, A. K., Wilder, H. D., & Sefton, A. E. (1997). Evidence that blue-on cells are part of the third geniculocortical pathway in primates. European Journal of Neuroscience, 9(7), Martinez-Uriegas, E. (1985). A solution to the color-luminance ambiguity in the spatiotemporal signal of primate X cells. Investigative Ophthalmology and Visual Science (supplement), 26, 183. Maxwell, J. C. (1856). On the theory of colours in relation to colour-blindness. A letter to Dr. G. Wilson. Transactions of the Royal Scottish Society of Arts, 4, Merbs, S. L., & Nathans, J. (1992a). Absorption spectra of human cone pigments. Nature, 356, Merbs, S. L., & Nathans, J. (1992b). Absorption spectra of the hybrid pigments responsible for anomalous color vision. Science, 258(5081), Merigan, W. H., & Eskin, T. A. (1986). Spatio-temporal vision of macaques with severe loss of Pb retinal ganglion cells. Vision Research, 26, Mitchell, D. E., & Rushton, W. A. H. (1971). Visual pigments in dichromats. Vision Research, 11(10),

55 Miyagishima, K. J., Grünert, U., & Li, W. (2014). Processing of S-cone signals in the inner plexiform layer of the mammalian retina. Visual Neuroscience, 31(Special Issue 02), Mullen, K. T. (1985). The contrast sensitivity of human colour vision to red-green and blue-yellow gratings. Journal of Physiology, 359, Mullen, K. T., & Kingdom, F. A. A. (2002). Differential distributions of red-green and blue-yellow cone opponency across the visual field. Visual Neuroscience, 19(1), Mullen, K. T., Sakurai, M., & Chu, W. (2005). Does L/M cone opponency disappear in human periphery? Perception, 34, Nassi, J. J., & Callaway, E. M. (2009). Parallel processing strategies of the primate visual system. Nat Rev Neurosci, 10(5), Nathans, J., Piantanida, T. P., Eddy, R. L., Shows, T. B., & Hogness, S. G. (1986). Molecular genetics of inherited variation in human color vision. Science, 232(4747), Nathans, J., Thomas, D., & Hogness, S. G. (1986). Molecular genetics of human color vision: The genes encoding blue, green and red pigments. Science, 232(4747), Navarro, R., Moreno, E., & Dorronsoro, C. (1998). Monochromatic aberrations and point-spread functions of the human eye across the visual field. Journal of the Optical Society of America A, 15(9), Neitz, J., Carroll, J., Yamauchi, Y., Neitz, M., & Williams, D. R. (2002). Color perception is mediated by a plastic neural mechanism that is adjustable in adults. Neuron, 35(4), Neitz, M., Neitz, J., & Jacobs, G. H. (1991). Spectral tuning of pigments underlying red-green color vision. Science, 252(5008), Newton, I. (1704). Opticks: or a treatise of the reflexions, refractions, inflexons and colours of light. London: Samuel Smith and Benjamin Walford. Newton, J. R., & Eskew, R. T., Jr. (2003). Chromatic detection and discrimination in the periphery: A postreceptoral loss of color sensitivity. Visual Neuroscience, 20(5), Østerberg, G. A. (1935). Topography of the layer of rods and cones in the human retina. Acta Ophthalmologica (supplement) 6, Packer, O. S., Verweij, J., Li, P. H., Schnapf, J. L., & Dacey, D. M. (2010). Blue-yellow opponency in primate S cone photoreceptors. Journal of Neuroscience, 30(2), Paulus, W., & Kröger-Paulus, A. (1983). A new concept of retinal colour coding. Vision Research, 23(5), Perlman, I., & Normann, R. A. (1998). Light adaptation and sensitivity controlling mechanisms in vertebrate photoreceptors. Progress in Retinal and Eye Research, 17(4),

56 Perry, V. H., Oehler, R., & Cowey, A. (1984). Retinal ganglion cells that project to the dorsal lateral geniculate nucleus in the macaque monkey. Neuroscience, 12(4), Pharr, M., & Humphreys, G. (2010). Physically based rendering: From theory to implementation (2nd ed.). San Francisco: Morgan Kaufmann Publishers. Poirson, A. B., & Wandell, B. A. (1993). Appearance of colored patterns: pattern-color separability. J Opt Soc Am A Opt Image Sci Vis, 10(12), Poirson, A. B., & Wandell, B. A. (1996). Pattern-color separable pathways predict sensitivity to simple colored patterns Vision Research, 36(4), Poirson, A. B., Wandell, B. A., Varner, D. C., & Brainard, D. H. (1990). Surface characterizations of color thresholds. Journal of the Optical Society of America A, 7(4), Pokorny, J., Smith, V. C., & Lutze, M. (1989). Heterochromatic modulation photometry. Journal of the Optical Society of America A, 6(10), Pokorny, J., Smith, V. C., & Wesner, M. F. (1991). Variability in cone populations and implications. In A. Valberg & B. B. Lee (Eds.), From Pigments to Perception (Vol. 203, pp ). New York: Plenum. Polyak, S. L. (1941). The Retina. Chicago: University of Chicago. Pugh, E. N., Jr., & Lamb, T. D. (2000). Phototransduction in vertebrate rods and cones: molecular mechanisms of amplification, recovery and light adaptation. In D. G. Stavenga, W. J. de Grip & E. N. Pugh (Eds.), Handbook of biological physics, Vol. 3, Molecular mechanisms of visual transduction (pp ). Amsterdam: Elsevier. Pugh, E. N., Jr., Nikonov, S., & Lamb, T. D. (1999). Molecular mechanisms of vertebrate photoreceptor light adaptation. Current Opinion in Neurobiology, 9(4), Purcell, E. M. (1965). Electricity and Magnetism. New York: McGraw-Hill Book Company. Raviola, E., & Gilula, N. B. (1973). Gap junctions between photoreceptor cells in the vertebrate retina. Proceedings of the National Academy of Sciences of the United States of America, 70, Reid, R. C., & Shapley, R. M. (1992). Spatial structure of cone inputs to the receptive fields in primate lateral geniculate nucleus. Nature, 356(6371), Robson, J. G. (1966). Spatial and temporal contrast sensitivity functions of the visual system. Journal of the Optical Society of America, 56, Rodieck, R. W. (1998). The First Steps in Seeing. Sunderland, Mass.: Sinauer. Rodieck, R. W., Binmoeller, K. F., & Dineen, J. (1985). Parasol and midget ganglion-cells of the human retina. Journal of Comparative Neurology, 233(1),

57 Roorda, A. (2011). Adaptive optics for studying visual function: A comprehensive review. Journal of Vision, 11(5). Roorda, A., & Williams, D. R. (1999). The arrangement of the three cone classes in the living human eye. Nature, 397(6719), Ruderman, D. L., Cronin, T. W., & Chiao, C. C. (1998). Statistics of cone responses to natural images: implications for visual coding. Journal of the Optical Society of America A, 15(8), Sakurai, M., & Mullen, K. T. (2006). Cone weights for the two cone-opponent systems in peripheral vision and asymmetries of cone contrast sensitivity. Vision Research, 46(26), Schmidt, B. P., Neitz, M., & Neitz, J. (2014). Neurobiological hypothesis of color appearance and hue perception. Journal of the Optical Society of America A, 31(4), A195-A207. Sekiguchi, N., Williams, D. R., & Brainard, D. H. (1993a). Aberration-free measurements of the visibility of isoluminant gratings. Journal of the Optical Society of America A, 10(10), Sekiguchi, N., Williams, D. R., & Brainard, D. H. (1993b). Efficiency in detection of isoluminant and isochromatic interference fringes. Journal of the Optical Society of America A, 10(10), Sharpe, L. T., Stockman, A., Jagla, W., & Jägle, H. (2011). A luminous efficiency function, V*(λ), for daylight adaptation: a correction. Color Research & Application, 36, Sharpe, L. T., Stockman, A., Jägle, H., & Nathans, J. (1999). Opsin genes, cone photopigments, color vision and colorblindness. In K. Gegenfurtner & L. T. Sharpe (Eds.), Color vision: From Genes to Perception (pp. 3-51). Cambridge: Cambridge University Press. Sher, A., & DeVries, S. H. (2012). A non-canonical pathway for mammalian blue-green color vision. Nature Neuroscience, 15(7), Smith, V. C., Lee, B. B., Pokorny, J., Martin, P. R., & Valberg, A. (1992). Responses of macaque ganglion cells to the relative phase of heterochromatically modulated lights. Journal of Physiology, 458, Smith, V. C., & Pokorny, J. (1975). Spectral sensitivity of the foveal cone photopigments between 400 and 500 nm. Vision Research, 15, Solomon, S. G., Lee, B. B., White, A. J., Rüttiger, L., & Martin, P. R. (2005). Chromatic organization of ganglion cell receptive fields in the peripheral retina. Journal of Neuroscience, 25(18),

58 Solomon, S. G., & Lennie, P. (2007). The machinery of colour vision. Nature Reviews Neuroscience, 8(4), Stiles, W. S. (1949). Incremental thresholds and the mechanisms of colour vision. Documenta Ophthalmologica, 3, Stiles, W. S., & Burch, J. M. (1959). NPL colour-matching investigation: Final report (1958). Optica Acta, 6, Stockman, A., & Brainard, D. H. (2010). Color vision mechanisms. In M. Bass, C. DeCusatis, J. Enoch, V. Lakshminarayanan, G. Li, C. Macdonald, V. Mahajan & E. van Stryland (Eds.), The Optical Society of America Handbook of Optics, 3rd edition, Volume III: Vision and Vision Optics (pp ). New York: McGraw Hill. Stockman, A., Langendörfer, M., Smithson, H. E., & Sharpe, L. T. (2006). Human cone light adaptation: from behavioral measurements to molecular mechanisms. Journal of Vision, 6(11), Stockman, A., MacLeod, D. I. A., & Johnson, N. E. (1993). Spectral sensitivities of the human cones. Journal of the Optical Society of America A, 10(12), Stockman, A., & Sharpe, L. T. (1999). Cone spectral sensitivities and color matching. In K. Gegenfurtner & L. T. Sharpe (Eds.), Color vision: From Genes to Perception (pp ). Cambridge: Cambridge University Press. Stockman, A., & Sharpe, L. T. (2000). Spectral sensitivities of the middle- and long-wavelength sensitive cones derived from measurements in observers of known genotype. Vision Research, 40(13), Stromeyer, C. F., III, Cole, G. R., & Kronauer, R. E. (1985). Second-site adaptation in the red-green chromatic pathways. Vision Research, 25(2), Stromeyer, C. F., III, Lee, J., & Eskew, R. T., Jr. (1992). Peripheral chromatic sensitivity for flashes: a post-receptoral red-green asymmetry. Vision Research, 32(10), Szmajda, B. A., Buzás, P., FitzGibbon, T., & Martin, P. R. (2006). Geniculocortical relay of blue-off signals in the primate visual system. Proceedings of the National Academy of Sciences of the United States of America, 103(51), Szmajda, B. A., Grunert, U., & Martin, P. R. (2008). Retinal ganglion cell inputs to the koniocellular pathway. Journal of Comparative Neurology, 510(3), Tailby, C., Solomon, S. G., Dhruv, N. T., & Lennie, P. (2008). Habituation reveals fundamental chromatic mechanisms in striate cortex of macaque. Journal of Neuroscience, 28(5),

59 Tailby, C., Solomon, S. G., & Lennie, P. (2008). Functional asymmetries in visual pathways carrying S-cone signals in Macaque. Journal of Neuroscience, 28(15), Tsukamoto, T., Masarachia, P., Schein, S. J., & Sterling, P. (1992). Gap junctions between the pedicles of macaque foveal cones. Vision Research, 32(10), Vakrou, C., Whitaker, D., McGraw, P. V., & McKeefry, D. (2005). Functional evidence for conespecific connectivity in the human retina. Journal of Physiology, 566(Pt 1), Valberg, A., Lee, B. B., & Tigwell, D. A. (1986). Neurones with strong inhibitory S-cone inputs in the macaque lateral geniculate nucleus. Vision Research, 26(7), Vienot, F., & Brettel, H. (2014). The Verriest Lecture: Visual properties of metameric blacks beyond cone vision. Journal of the Optical Society of America. A, 31(4), A Vimal, R. L. P., Smith, V. C., Pokorny, J., & Shevell, S. K. (1989). Foveal cone thresholds. Vision Research, 29(1), Vos, J. J. (1978). Colorimetric and photometric properties of a 2-deg fundamental observer. Color Research and Application, 3, Vos, J. J., & Walraven, P. L. (1971). On the derivation of the foveal receptor primaries. Vision Research, 11(8), Wagner, G., & Boynton, R. M. (1972). Comparison of four methods of heterochromatic photometry. Journal of the Optical Society of America, 62(12), Wandell, B. A. (1995). Foundations of Vision. Sunderland, MA: Sinauer. Wässle, H. (2004). Parallel processing in the mammalian retina. Nature Reviews Neuroscience, 5(10), Wässle, H., & Boycott, B. B. (1991). Functional architecture of the mammalian retina. Physiological Reviews, 71(2), Weitz, C. J., Miyake, Y., Shinzato, K., Montag, E. D., Zrenner, E., Went, L. N., & Nathans, J. (1992). Human tritanopia associated with two amino acid substitutions in the blue-sensitive opsin. American Journal of Genetics, 50(3), Weitz, C. J., Went, L. N., & Nathans, J. (1992). Human tritanopia associated with a 3rd amino-acid substitution in the blue-sensitive visual pigment. American Journal of Human Genetics, 51(2), Went, L. N., & Pronk, N. (1985). The genetics of tritan disturbances. Human Genetics, 69(3), Westheimer, G. (1964). Pupil size and visual resolution. Vision Research, 4(1-2),

60 Westheimer, G. (1986). The eye as an optical instrument. In K. R. Boff, L. Kaufman & J. P. Thomas (Eds.), Handbook of perception and human performance (Vol. 1, pp. 1-20). New York: Wiley. Wiesel, T. N., & Hubel, D. (1966). Spatial and chromatic interactions in the lateral geniculate body of the rhesus monkey. Journal of Neurophysiology, 29(6), Williams, D. R. (1985). Aliasing in human foveal vision. Vision Research, 25(2), Williams, D. R., Artal, P., Navarro, R., McMahon, M. J., & Brainard, D. H. (1996). Off-axis optical quality and retinal sampling in the human eye. Vision Research, 36(8), Williams, D. R., & Collier, R. J. (1983). Consequences of spatial sampling by a human photoreceptor mosaic. Science, 221, Williams, D. R., MacLeod, D. I. A., & Hayhoe, M. M. (1981). Foveal tritanopia. Vision Research, 19(9), Willmer, E. N. (1944). Colour of small objects. Nature, 153, Winderickx, J., Lindsey, D. T., Sanocki, E., Teller, D. Y., Motulsky, A. G., & Deeb, S. S. (1992). A Ser/Ala polymorphism in the red photopigment underlies variation in colour matching among colour-normal individuals. Nature, 356, Wyszecki, G., & Stiles, W. S. (1982). Color Science: concepts and methods, quantitative data and formulae (2nd ed.). New York: Wiley. Xiao, F., DiCarlo, J. M., Catrysse, P. B., & Wandell, B. A. (2002). High dynamic range imaging of natural scenes. Color and Imaging Conference, 2002(1), Xiao, Y. (2014). Processing of the S-cone signals in the early visual cortex of primates. Visual Neuroscience, 31(Special Issue 02), Yamauchi, Y., Williams, D. R., Brainard, D. H., Roorda, A., Carroll, J., Neitz, M., Neitz, J., & Jacobs, G. H. (2002). What determines unique yellow, L/M cone ratio or visual experience? Paper presented at the 9th Congress of the International Colour Association, Proceedings of SPIE, Yellott, J. I., Jr., Wandell, B. A., & Cornsweet, T. N. (1984). The beginnings of visual perception: the retinal image and its initial encoding. In I. Darien-Smith (Ed.), Handbook of Physiology: The Nervous System (Vol. III, pp ). New York: Easton. Yin, L., Smith, R. G., Sterling, P., & Brainard, D. H. (2009). Physiology and morphology of coloropponent ganglion cells in a retina expressing a dual gradient of S and M opsins. Journal of Neuroscience, 29(9), Zaidi, F. H., Hull, J. T., Peirson, S. N., Wulff, K., Aeschbach, D., Gooley, J. J., Brainard, G. C., Gregory-Evans, K., Rizzo, J. F., Czeisler, C. A., Foster, R. G., Moseley, M. J., & Lockley, S. 60

61 W. (2007). Short-wavelength light sensitivity of circadian, pupillary, and visual awareness in humans lacking an outer retina. Current Biology, 17(24),

62 Figure Legends Please note that we intend to retain copyright for Figures 1, 5-10, which will appear in our forthcoming book. Please list as Authors copyright. Permissions needed for Figure 2 from Journal of Neuroscience. Figure 1. Spectral information for color. Top image. RGB rendering of a hyperspectral image. The hyperspectral image consists of 31 monochromatic images acquired at wavelengths between 400 nm and 700 nm in 10 nm steps (Brainard, 1998). The image is rendered according to the srgb standard of the International Electrotechnical Commission (1999). Middle images. Three separate monochromatic image planes from the underlying hyperspectral image. The planes are for 440 nm, 550 nm, and 660 nm. Note that although the same basic structure is visible in each monochromatic image plane, the intensity relations across planes differ between image locations. These differences indicate differences in spectra, and it is such spectral differences that provide the information for color vision. Bottom images. Effect of blurring by the eye s optics on monochromatic image planes. The optics of the eye contain chromatic aberrations, so that spatial structure is differentially blurred across wavelengths in the retinal image. The bottom three images show the same 440 nm, 550 nm, and 660 nm image planes after blurring by a model of the eye s optics. To make the images, we assumed that the scene was flat and viewed foveally with an angular subtense of 1, and used the model of the eye s optical aberrations described by Autresseau et al. (2011) as computed for a 3 mm pupil with the Wavefront Optics Toolbox (Brainard, Hofer, & Wandell, 2013). Figure 2. Human cone mosaics. The L, M, and S cones sample the retinal image via three interleaved submosaics. Each image shows an image of the cone mosaic measured in an individual living human eye, for a small patch of central retina. L-cones indicated as red, M- as green, and S- 62

63 as blue. Each image shows the mosaic for a different subject or retinal location. Reprinted from Figure 4 of Hofer et al. (2005). Figure 3. Comparisons between estimates of the 2-deg L-, M- and S-cone spectral sensitivities by Stockman & Sharpe (2000) (solid colored lines) and by Smith & Pokorny (1975) (dashed lines). The lower inset shows the lens pigment optical density spectrum (black line) and the macular pigment optical density spectrum (magenta line) from Stockman and Sharpe (2000). Figure 4. A monochromatic test field of wavelength λ can be matched by a mixture of red (645 nm), green (526 nm) and blue (444 nm) primary lights, one of which must be added to the test field to complete the match (lower panel). The amounts of each of the three primaries required to match monochromatic lights spanning the visible spectrum are the red, rr (λλ), gg (λλ) and bb (λλ) CMFs (red, green and blue lines, respectively) shown in the upper left-hand panel. These CMFs were measured using 10-deg diameter fields by Stiles & Burch (1959). A negative sign means that that primary must be added to the target to complete the match. CMFs can be linearly transformed from one set of primaries to another and to the cone fundamentals. Although the fact that many of the negative values are of very small magnitude may make it difficult to discern in the figure, one of the color matching functions is negative for all test wavelengths other than those of the primaries. Shown in the upper right-hand panel are the are 10-deg, rr (λλ), gg (λλ) and bb (λλ) CMFs linearly transformed to yield the ll (λλ), mm (λλ) and ss (λλ) Stockman & Sharpe (2000) 10-deg cone fundamental CMFs (red, green and blue lines, respectively). Figure 5. The principal cone pathways in the human retina. Both L- and M-cones contact the diffuse bipolar cells (DB) and midget bipolar cells (MB) with which they make inhibitory (sign-inverting or 63

64 ON) or excitatory (sign-preserving or OFF) connections. The S-cones make inhibitory (signinverting or ON) connections with the S-cone bipolar cells (SB). The H1 and H2 horizontal cells make lateral inhibitory connections between cones, but the H1 cells avoid contacts with S-cone. The midget bipolar cells (MB) contact midget ganglion cells (MGC) of the same polarity. The diffuse bipolar cells (DB) contact parasol ganglion cells (PG) of the same polarity. S-cone bipolar cells (SB) contact S-cone bistratified ganglion cells (SBG), which also receive an input from DBOFF cells. Not shown are rods, gap junctions between photoreceptors, OFF S-cone connections, and amacrine cells. Amacrine cells make lateral connections at the level of the bipolar and ganglion cell synapses. Design inspired by figures from Rodieck (1998). Figure 6. Diagrammatic network model of the early visual pathways. Left column: ON and OFF outputs from the L-, M- and S-cones represented by the larger red, green and blue truncated triangles, respectively. The four smaller cones around each larger cone symbolize a surround network of two L-, one M- and one S-cones mediated by horizontal cells. The S-cone surround is shown as a dashed line, since the S-cone surround influence is uncertain. The concentric circles beside each output represent cone- and spatially-opponent center-surround receptive fields. Middle column: parvocellular, magnocellular and koniocellular connections and outputs, including a demultiplexing scheme for the parvocellular circuits that extracts spatially non-opponent color (top) and spatially opponent luminance (bottom) information. The magnocellular circuit is shown having inputs from only three cones but in central retina about 5-7 cones contact diffuse bipolar cells (Kolb, 1970). The koniocellular pathway shows only an S-cone ON pathway. An S-cone OFF pathway remains speculative but is likely to exist. Note that the circuitry suggested in the parvocellular case, which is contained within the box delineated by the dashed line, is likely to be cortical, whereas the circuitry in the magnocellular and koniocellular cases is retinal. Right column: illustrative representations of the receptive fields at the outputs of the various circuits. The sizes of the receptive 64

65 fields at the outputs of the pathways are approximate and not to scale. They are larger than the sizes of the receptive fields at the cone outputs because of convergence. Figure 7. Detection thresholds. Typical detection contour plotted in cone-contrast space. Each point indicates threshold in one polar direction in the cone-contrast color space, represented as the contrast at threshold seen by the L-cones (abscissa, L/L) and M-cones (ordinate, M/M). Data replotted from Figure 4 (Observer CFS) of Stromeyer, Cole & Kronauer (1985). The data are detection thresholds for 1.2 diameter, 200-ms duration flashes presented on a larger 524-nm background of 3200 trolands. The red line through the points is an ellipse fitted to the data. The dashed lines have been fitted to the upper and lower flanks of the data and have slopes of 1, consistent with detection mediated by cone-opponent mechanisms with equal but opposite cone contrast weights ( M/M = - L/L ± ). The double-headed arrows in the upper left quadrant illustrate changes in only L-cone contrast (L, horizontal arrow) or in only M-cone contrast (M, vertical arrow). The double-headed arrows in the lower right quadrant illustrate changes in L-cone or M-contrast together (L+M, positive-diagonal arrow) or opposing changes in L-cone or M-contrast (M-L, negative-diagonal arrow). Figure 8. Second-site desensitization of L-M by steady fields. L-M detection contours in cone contrast space for subject AC measured by Chaparro et al. (1995) on 3000 td fields of 525 nm (green circles), 579 nm (yellow squares) and 654 nm (red triangles) replotted from their Figure 4. The target was 2.2 in diameter and of 200 ms duration and was presented in the center of a 6.2 diameter uniform adapting field. The straight contours fitted to thresholds have slopes of 1, consistent with detection by a cone-opponent mechanism with equal and opposite M- and L-cone contrast weights. 65

66 Figure 9. Temporal and spatial contrast sensitivity functions. Top panel. Temporal contrast sensitivity functions for isochromatic [or luminance] (black inverted triangles) and red-green isoluminant [or chromatic] (red circles) temporal modulations. The data are threshold measurements for a flickering 1.8 uniform field viewed foveally replotted from Figure 1 of Kelly (1977). The retinal illuminance was 860 photopic trolands. Bottom panel. Spatial contrast sensitivity functions (CSFs) for foveally-viewed isochromatic (black triangles) and red-green isoluminant (red squares) sinusoidal modulations. These data were collected using stimuli produced by a dual-wavelength interferometer, which eliminated blurring by the eye s optics. The data are plotted after normalization to the performance of an ideal observer computed from the cone isomerization rates of the retinal cone mosaic, and thus take into account the differences in the L- and M-cone contrasts for the luminance and isoluminant stimuli. The smooth curves fit to the data are of no theoretical significance. Data replotted from Figure 9 of Sekiguchi (1993b), observer NS. The retinal illuminance was 500 photopic trolands. Figure 10. Differences between fovea and periphery. Decline in color contrast detection sensitivity with eccentricity. Top panel. Contrast sensitivity for isochromatic [luminance] (black inverted triangles), red-green isoluminant [chromatic] (red circles), and S-cone mediated (blue squares) stimuli measured as a function of eccentricity. Bottom panel. Isoluminant red-green (red circles) and S-cone-mediated (blue squares) sensitivity from top panel, normalized by luminance sensitivity at the same eccentricity. The smooth curves fit to the data are of no theoretical significance. Data replotted from Figure 3 of Hansen et al. (2009). 66

67 440 nm 550 nm 660 nm 440 nm 550 nm 660 nm Figure 1

68 Figure 2