Visual perception: a radical new framework

     What is visual perception? Why and how do we see the things we do? Take the perception of color. Everybody knows what color is: we have no problem deciding that a red table is red. But when philosophers and scientists try to explain how sensations of color are rooted in the physical world―and many have tried―they soon run in to trouble. The fact is that a satisfactory account of why and how we see the colors we do has never been offered. This caveat applies to other visual qualities such as brightness, form, depth, and speed (to name a few), and arguably to the perceptual qualities we experience in audition or any other sensory modality.

     Most contemporary work on vision and visual perception is concerned with how brain activity correlates with the features of visual stimuli and subjective reports. Over the last six decades, neuroscientists have had tremendous success showing that the neuronal components of the eye and specific brain regions are sensitive to certain aspects of the physical world like color or line orientation. The central idea is that input-level neurons detect and encode retinal image features, and that successive stages of visual system processing eventually represent the physical qualities of the world as conscious percepts. A fuller explanation of how these brain systems process the information in images, the thinking goes, will eventually explain why we see what we do.

     There is, however, good reason to think that this general assumption is mistaken. The problem is that retinal images can't specify the objects in the world that generate visual sensory stimulation. To better understand this problem, known as the “inverse problem,” consider the following figure: 

Figure 1. The light stimulus arising from different objects in the world that have different sizes, are at different distances and in different orientations. These differences notwithstanding, all three objects produce the same two-dimensional image on the retina. How, then, do we manage to routinely see differences in size, distance and orientation, and behave successfully in response? (From the Purves lab website).

As this example illustrates, three different objects at different distances and orientations can give rise to the same retinal image. There is, in principle, no way to get from the retinal image―the two-dimensional pattern of light―back to the three dimensional object(s) in the world. Any view that assumes this is the way the visual system works is suspect from the outset.

     We have the same problem with the perception of color, in which illumination (the amount and kind of light illuminating an object) and reflectance (the amount and kind of light an object tends to reflect) are entangled in any light stimulus. A conventional account is that spectral information, represented in the responses of three different types of cone cells in the human retina, is processed by the visual system to determine sensations of color. But the fact is that a red apple looks red when it's illuminated by the sun at different times of day―even though the spectra of the reflected light waves are quite different. While the reflectance of the apple does not change its illuminance does, and our visual system has no way of knowing which changed and which didn't. This phenomenon, known as color constancy, is a good example of how the physical characteristics of a stimulus do not (indeed, cannot) determine what we actually see.

      The inverse problem seems impossible to solve by a logical parsing of stimulus features, but somehow the brain gets around it and generates appropriate behaviors in response to stimuli. How could this be done?

     Over the last decade, Dale Purves and his colleagues at Duke University have developed a framework aimed at answering the question I just posed (I am a new student in the Purves lab). Their conclusions are surprising. According to this work, what we see is determined by how we have behaved in response to similar patterns of light in the past. The idea is that our ancestors responded, and we respond, to patterns of light by trial and error. Through natural selection over evolutionary time and neural plasticity over individual lifetimes, neural circuitry that generated successful responses to any given light pattern has been preserved, linking that stimulus pattern to successful behavior. Because the relationship between sensory images and properties of objects cannot be known, the investigators have argued that this is the only plausible strategy to generate object-appropriate behavior from sensory images.

      The idea is simple, but the ramifications for how we think about perception are great. On this view the sensation of color, for example, does not represent any objective reality. Color, and all other perceptual qualities, are simply biological adaptations that reflect a long history of trial-and-error interactions with the world. The visual system incorporates this species and individual history into its design. As a result, the argument goes, what we see is a reflexive response no different than a “knee jerk”.

     The evidence for this counterintuitive understanding of vision is the remarkable power it brings to predicting what we actually see. Check out the color effects below: although the physical characteristics of the target squares are surprising, the colors we see are consistent with a trial-and-error strategy for vision.

Figure 2. The small panels at the bottom show what the middle squares would look like by themselves. (From Purves and Lotto (2003)).

Sources:
Purves lab website

Brains: How they seem to work

Purves D, Lotto RB, Williams SM, Nundy S, Yang, Z, Philos Trans R Soc Lond B, 356:285-297. (2001)  Why we see things the way we do: evidence for a wholly empirical strategy for vision (PDF).