A reward is valued more if you choose it yourself: New quirky byproduct of learning from reward

"We weren't looking to explain anything about choice bias to start off with," said lead author Jeffrey Cockburn, a graduate student in the research group of senior author Michael Frank, associate professor of cognitive, linguistic and psychological sciences. "This just happened to be the behavioral phenomenon we thought would emerge out of this credit assignment model."

So the next time a friend raves about the movie he chose and is less enthusiastic about the just-as-good one that you chose, you might be able to chalk it up to his basic learning circuitry and a genetic difference that affects it.

Modeled mechanism

The model, developed by Frank, Cockburn and co-author Anne Collins, was based on prior research on the function of the striatum, a part of the brain's basal ganglia (BG) that is principally involved in representing reward values of actions and picking one. "An interaction between three key BG regions moderates that decision making process. When a rewarding choice has been made, the substantia nigra pars compacta (SNc) releases dopamine into the striatum to reinforce connections between cortex and striatum, so that rewarded actions are more likely to be repeated. But how does the SNc reinforce just the circuits that made the right call? The authors proposed a mechanism by which another part of the subtantia nigra, the SNr, detects when actions are worth choosing and then simultaneously amplifies any dopamine signal coming from the SNc."

"The novel part here is that we have proposed a mechanism by which the BG can detect when it has selected an action and should therefore amplify the dopamine reinforcing event specifically at that time," Frank said. "When the SNr decides that striatal valuation signals are strong enough for one action, it releases the brakes not only on downstream structures that allow actions to be executed, but also on the SNc dopamine system, so any unexpected rewards are amplified."

Specifically, dopamine provides reinforcement by enhancing the responsiveness of connections between cells so that a circuit can more easily repeat its rewarding behavior in the future.

But along with that process of reinforcing the action of choosing, the value placed on the resulting reward becomes elevated compared to rewards not experienced this way.

Experimental evidence

That prediction seemed intriguing, but it still had to be tested. The authors, who also included postdoctoral researcher Anne Collins, identified both behavioral and genetic tests that would be telling.

They recruited 80 people at Brown and elsewhere in Providence to play a behavioral game and to donate some saliva for genetic testing.

The game first presented the subjects pictures of arbitrary Japanese characters that would have different probabilities of rewards if chosen ranging from a 20 percent to 80 percent chance of winning a point or losing a point. For some characters, the player could choose a character to discover its resulting reward or penalty, whereas for others, its result was simply given to them. After that learning phase, the subjects were then presented the characters in pairs and instructed to pick the one they thought had the highest chance of winning based on what they had learned.

The researchers built the game so that for every character a player could choose, there was an equally rewarding one had merely been given to them. On average, players showed a clear choice bias in that they were more likely to prefer rewarding characters that they had chosen over equally rewarding characters they had been given.

Notably, they exhibited no choice bias between unrewarding characters suggesting that choice bias emerges only in relation to reward, one of the key predictions of their model. But they wanted to further test whether the impact of reward on choice bias was related to the proposed biological mechanism: that striatal dopaminergic learning is enhanced to chosen rewards.

The genetic tests focused on single-letter differences in a gene called DARPP-32, which governs how well cells in the striatum respond to the reinforcing influence of dopamine.

People with one version of the gene have been shown in previous research to be less able to learn from rewards, while people with other versions were less driven by reward in learning.

"The reason why this gene is interesting is because we know something about the biology of what it does and where it is expressed in the brain," Frank said. "It's predominant in the striatum and specifically affects synaptic plasticity induced by dopamine signaling. It's related to the imbalance to which you learn from really good things, or not so good things.

"The logic was if the mechanism that we think describes this choice bias and credit assignment problem is accurate then that gene should predict the impact of how good something was on this choice bias phenomenon," he said.

Indeed, that's what the data showed. People with the form of the gene that predisposed them to be responsive to big rewards also showed more choice bias from the most strongly rewarded characters. Interestingly, the other people also showed choice bias, but more strongly for those characters that were more mediocre. This pattern was mirrored by the authors' model when it simulated the effects of DARPP-32 on reward learning imbalances from positive vs. negative outcomes.

For some people, the plums are sweeter if they picked them.