Dopamine neurons factor ambiguity into predictions enabling us to 'win big and win often'

To make winning decisions, animals cannot consult a hard-wired playbook or receive helpful advice. Instead, success depends on their ability to learn, from triumphs and from mistakes.

Today in the journal Current Biology, a team led by Professor Adam Kepecs of Cold Spring Harbor Laboratory (CSHL) adds an important new dimension to our understanding of how the brain learns, focusing specifically on situations in which perceptually ambiguous information is used to make predictions.

What if it's gotten foggy and we can barely see the sign up ahead, or the announcement over the loudspeaker is garbled? In circumstances like these, how does the brain adjust the predictions that inform decisions?

Kepecs and colleagues describe how dopamine-releasing neurons, which are understood to produce critical teaching signals for the brain, weigh the ambiguity of sensory information when they assess how successfully past experiences have guided a new decision. Their findings indicate that these neurons are even more sophisticated than previously thought.

"Dopamine neurons are not just any neurons in the brain," Kepecs says. "They are neurons that sit in the midbrain and send connections to large swaths of the brain. Dopamine neurons compare the predicted outcomes to actual outcomes, and send the discrepancy between these as an error feedback to other parts of the brain. This is exactly what you would want learning signals to do. And we've found that what dopamine neurons do is in perfect agreement with what is theoretically required for a powerful learning algorithm."

This is such an effective way of learning that the same strategy, called reinforcement learning, has been incorporated into many types of artificial intelligence, allowing computers to figure out how to do things without instructions from a programmer.

Kepecs and his team added what computer scientists call a belief state (a probability distribution variable) to a mathematical model of reinforcement learning. The previously used model is good at explaining how an animal learns from experience -- but only when the sensory stimuli available to the animal are straightforward, easy to interpret. "It's been unclear whether or how dopamine neurons factor perceptual ambiguity into their teaching signal," Kepecs says. "Our revised model generates an estimate of the probability that a given choice is correct -- this is the degree of confidence about the decision."

"If you can evaluate your decision process -- the noise and everything else that's contributed to it -- and in addition to making a binary choice assign a graded level of confidence to the decision, it's essentially a prediction about your accuracy," he says. "It's a completely different computation than just using the cues to recall past experiences to make predictions."

Armin Lak, a British Iranian postdoctoral researcher working with Kepecs, developed a computer model that predicted how neurons would behave if they were performing this calculation. Then they compared those predictions to the activity of dopamine neurons in the brains of monkeys while they performed a task in which they made a decision informed by visual cues that were sometimes clear, and sometimes not.

The activity of the neurons was recorded by Kepecs' collaborators Kensaku Nomoto and Masamichi Sakagami, neuroscientists at Tamagawa University in Tokyo, who were studying how the brain uses sensory information to make decisions. In the task, monkeys watched a screen filled with moving dots, and had to decide whether more dots were moving to the left or to the right, receiving a juice reward when they got the answer right.

When Kepecs and Lak analyzed the data, they found that the activity of the dopamine neurons closely fit the predictions of their new model -- indicating that these neurons were likely evaluating the reliability of the information used during each decision. Applying the computational model was like looking at the data through a new microscope, Kepecs says. "By looking at it this way, you can see things that you couldn't see before in the data. Suddenly it just made sense. This system is much smarter than we thought before."

Surprisingly, Kepecs says, the confidence-signaling neurons became active a fraction of a second before the monkeys' made a decision. This early activity suggests the neurons may influence the immediate decision making process -- a role that the team will explore in future research.