Predictive Processing and the Brain's Shortcuts

For most of the history of vision science, the standard picture ran from the bottom up. Light hits the eye, signals climb up through the visual system, and somewhere near the top a recognisable image appears, a bit like an assembly line running in one direction. Predictive processing turns that picture round. It treats your brain as a prediction engine that is constantly generating a model of what it expects to see, and that uses the signals from your eyes mainly to check and correct that model, not to build perception from scratch (Clark, 2013).

Unconscious inference

The idea goes back to the nineteenth-century physicist and physiologist Hermann von Helmholtz. He argued that perception is "unconscious inference". Your eyes never deliver an unambiguous picture, so your brain infers the most likely cause of what it receives, drawing on past experience, much as a detective reasons from partial evidence (Helmholtz, 1867). He didn't have the mathematics we have today, but his core point, that perception is a matter of inference and not transcription, anticipated the modern theory.

Today the idea is usually expressed in terms of Bayesian probability. The brain holds "priors", which are expectations about the world built up from experience, and it combines them with the incoming evidence to reach its best current guess about what's out there (Kersten, Mamassian & Yuille, 2004). Importantly, the evidence isn't always trusted to the same degree. When it's noisy or ambiguous, the prior dominates, and when it's clear, perception follows the evidence. Karl Friston has developed this into a broader theory in which perception, action and learning all serve to reduce the brain's prediction errors about its surroundings (Friston, 2005).

Prediction error

One surprising claim of the framework is that what travels up the visual hierarchy is mainly the error, not the raw image: the mismatch between what higher areas predicted and what lower areas actually detected. When the prediction is right, there's very little left to send. When it's wrong, the error travels upwards and the model gets updated. Rajesh Rao and Dana Ballard showed that a network working this way reproduces some puzzling properties of real neurons in the visual cortex (Rao & Ballard, 1999). It also gives a job to the many connections running from higher areas back down to V1, which I've described in the role of the visual cortex: you can think of them as predictions sent down to be checked against what arrives from the eyes.

Why this explains illusions so well

If perception is prediction shaped by expectation, then illusions are exactly what you'd expect when an image triggers a strong prior that happens to be wrong.

In the checker shadow illusion, your brain's strong expectation that shadows darken surfaces makes it judge the shaded square as lighter than the light coming from it suggests. In the hollow face illusion, the expectation that faces bulge outwards is strong enough to overrule direct evidence that the mask is hollow. Motion gives a particularly neat example. Yair Weiss, Eero Simoncelli and Edward Adelson showed that many motion illusions follow naturally if the brain assumes, other things being equal, that things tend to move slowly. When a moving pattern is faint or low in contrast, the evidence is weak, the slow-motion prior wins, and the pattern looks slower than it really is (Weiss, Simoncelli & Adelson, 2002). The Necker cube fits too: with no evidence favouring either of two equally likely interpretations, your perception alternates between them.

Expectation can also be measured directly. The duck-rabbit tends to look like a rabbit at Easter, and the poll I've added to the spinning dancer page tests one specific prior: our preference for seeing figures from above.

A leading theory, still being tested

Predictive processing is one of the most influential ideas in neuroscience and cognitive science today, and it has been applied to perception, movement, mental illness and even consciousness. It isn't settled, though. Researchers are still debating how priors are learned, how they're implemented in neurons, and how far the framework can be stretched before it stops making predictions that can be tested. As a way of understanding illusions, however, it explains something the older bottom-up story couldn't: why the brain would ever produce a confident, vivid and wrong perception, instead of simply failing to see. If you'd like to step back to the wider question of why perception and reality come apart at all, have a look at why do optical illusions happen.

Sources and further reading

1. Helmholtz, H. von (1867). Handbuch der physiologischen Optik. Leopold Voss. Find this reference on Google Scholar
2. Rao, R. P. N., & Ballard, D. H. (1999). Predictive coding in the visual cortex: A functional interpretation of some extra-classical receptive-field effects. Nature Neuroscience, 2(1), 79–87. Find this reference on Google Scholar
3. Weiss, Y., Simoncelli, E. P., & Adelson, E. H. (2002). Motion illusions as optimal percepts. Nature Neuroscience, 5(6), 598–604. Find this reference on Google Scholar
4. Kersten, D., Mamassian, P., & Yuille, A. (2004). Object perception as Bayesian inference. Annual Review of Psychology, 55, 271–304. Find this reference on Google Scholar
5. Friston, K. (2005). A theory of cortical responses. Philosophical Transactions of the Royal Society B, 360(1456), 815–836. Find this reference on Google Scholar
6. Clark, A. (2013). Whatever next? Predictive brains, situated agents, and the future of cognitive science. Behavioral and Brain Sciences, 36(3), 181–204. Find this reference on Google Scholar