Smell tells intruder mice how to behave

"There seems to be a specific part of the brain that says, yeah, this is someone's house, and you need to act accordingly," says lead investigator Larry Zweifel, assistant professor of pharmacology at the University of Washington.

The hypothalamus, which is known to be the seat of social behavior in the brain, may contain additional sets of distinct cells wired to respond to other specific social contexts. Together, these signals tell an animal how to behave. The study, one of the first to look at what drives the behavior of subordinate mice rather than aggressors, could help researchers better understand disorders in social behavior in humans. For instance, individuals with autism, schizophrenia, depression, and social anxiety all experience disruptions in social behavior.

"The basic circuitry that regulates social function in a mouse is essentially the same as it is in humans," says Zweifel. "We're studying these discrete cells in the parts of the brain that regulate social behavior in mice to get some insight into where things go wrong to cause social behavioral deficits."

Zweifel's team came across the cells--neurons in the ventral premammillary nucleus of the hypothalamus--because they are genetically programmed to release dopamine, a neurotransmitter his team studies. It turns out that they don't release dopamine. But the team couldn't help but be intrigued when they saw that these cells were strongly activated only to male smells.

The team followed up by exposing male mice to a set of social contexts. A control group experienced no encounter. Others were either intruders into a cage occupied by either a male or female or were intruded upon in their own residence by either a male or female. The cells were activated in intruders only upon entering the cage of a male resident. Odors from both the resident animal and the cage, which contained bedding marked by the resident's urine, drove the cellular activity.

Most studies have focused on aggressive behavior because of its relevance to human social struggles, but in this social context, an intruder into another's cage is by definition a subordinate. "The parts of the brain that control aggression have been fairly well established," says Zweifel. "But what's it like if you're the animal that's encroaching into another's territory?"

Zweifel's team found that the intruder mouse's immediate impulse is to get more information. "They sniff and explore, like dogs in a park," says Zweifel. "Who are you, and are you fun to play with?"

Animals sniff and explore in other social situations, too, but these specific cells have nothing to do with it. "If you introduce a male into a female's cage, social investigation goes through the roof," says Zweifel. "But these cells aren't activated in this context."

To pin down the exact role of these cells, Zweifel's team inhibited them so that they would not be activated. This reduced exploration in male intruders. They also activated these cells artificially using optogenetics, the genetic engineering of cells to respond to light input. Activation increased exploration in males, causing them to sniff and explore mice they'd known since birth as if they were strangers.

In females, however, the cells were not activated by same-sex odorants, suggesting that these cells are not wired to control social exploration in females. "We don't know what they're doing in females yet," says Zweifel.

While sensory input induces these hypothalamic neurons to fire, the social control happens downstream in the medial amygdala and other regions of the hypothalamus. Zweifel's team intends to look there next to learn more about how social behavior is wired in the brain.