Mini-microscopes reveal brain circuitry behind social behavior

Instincts such as mating or fighting are innate behaviors generally thought to be hardwired into an animal's brain. But now, two studies that map brain activity in living mice reveal that social experiences can influence brain responses to other mice. The results, recently reported in the journals Nature and Cell, show how and where in the brain some instincts are shaped by learning, says Howard Hughes Medical Institute (HHMI) Investigator David Anderson of California Institute of Technology.

"We're starting to get a sense of what happens between the part of the brain that takes in sensory information and the part that produces behavior," Anderson says.

Neuroscientists want to understand how the brain converts sights, sounds, and smells into pictures of the outside world. For animals, smell also provides clues about the age and sex of others nearby; this information can trigger instinctive behaviors. Male and female mice sharing a cage will mate, for example, while two males will often fight for territory.

Anderson and HHMI Investigator Catherine Dulac of Harvard University have previously identified brain regions controlling social behaviors in mice. Anderson has used genetic and optogenetic approaches to identify brain regions responsible for the control of aggressive and mating behaviors. And Dulac has used similar approaches to study the neural pathways involved in smell-related social behaviors in male and female mice.

Now, the researchers have visualized brain activity in awake mice as they interacted normally with mice and other stimuli. Working independently, Dulac and Anderson mapped patterns of brain activity in different brain regions while mice sniffed, ignored, fought, or mated with other mice. Dulac also tracked brain activity triggered by predator and infant odors.

Lasting brain changes

Dulac and her colleagues tracked brain activity in the medial amygdala, an almond-shaped structure that transmits smell signals to the hypothalamus. First, the researchers used a genetic trick to introduce a protein that lights up in active brain cells. Then, the team mounted lightweight microscopes on the heads of individual mice and looked at which brain cells were active when each camera-wearing mouse met another mouse. A thin glass rod implanted in the amygdala collected light from active brain cells and served as the microscope's lens. The researchers recorded neural activity while videotaping the mice's behavior in different social situations.

Dulac's team saw that different clusters of neurons lit up when mice met a member of the opposite sex. In males and females, these sex-specific neural patterns were quite different, Dulac says. And, surprisingly, the act of mating actually transformed the brain's activity patterns. After housing virgin mice in a cage with a mouse of the opposite sex for 15 days, the mice had long-lasting changes in the brain, Dulac and colleagues discovered.

The key to this discovery was an ambitious experiment that had researchers tracking mouse behavior and neural activity (via the head-mounted mini-microscopes) regularly for more than three months straight -- a technical feat that scientists had not attempted before.

Sexual experience strengthens the brain's responses to the opposite sex's odors, and improves an animal's ability to tell males and females apart, the researchers found. That's a sign that experience -- learning -- can help shape an animal's instincts, Dulac says.

"It was surprising to see patterns of brain activity thought to be instinctively activated by odor actually change with experience -- and stay changed for over a month," Dulac says.

In male mice, the hormone oxytocin -- known for its role in maternal and social bonding -- is likely involved in regulating these long-term changes to the brain, Dulac's team found. In female mice, pregnancy also changed brain activity patterns. During and after pregnancy, a whiff of predator odor (soiled rat bedding) didn't trigger as big of a neural response as it did in mice before pregnancy. That finding stood out, Dulac says, because pregnant females and new mothers have been shown to have reduced responses to stressors. Her team, which collaborated with HHMI Investigator Mark Schnitzer at Stanford University and Venkatesh Murthy at Harvard, reported their results October 26, 2017, in Cell.

Probing the hypothalamus

At Caltech, Anderson and his colleagues also wanted to visualize the neural circuitry involved with social behavior in male mice. The researchers, including Stanford's Schnitzer, used the same microscopic technique as Dulac's team but implanted the lens in the ventromedial hypothalamus, an evolutionarily ancient structure involved in social behavior. Anderson's team imaged the activity of a specific population of neurons that produce the estrogen receptor, which is well known for its influence on social behaviors.

Anderson's team placed the microscopes on virgin, socially isolated male mice and let them interact in an alternating manner with five different females and five different males, each for two minutes, for several consecutive days. The researchers imaged the same neurons across multiple trials and multiple days, and correlated changes in neural activity with changes in social behaviors, such as sniffing, mounting, and attacking.

During the males' initial encounters with male or female visitors, researchers observed little mating or fighting, and the same neurons lit up in response to both sexes. But with continued social experience, the males gradually began to mate with female visitors, and then to attack male visitors. At the same time, more neurons began to respond specifically to one or the other sex, and fewer to both.

"We watched these activity patterns change in real time as a mouse's brain learned to tell the difference between males and females," Anderson says.

In a different set of experiments, Anderson's team showed that just a brief experience with a female mouse could make a big difference in a virgin male's brain, as well as in his aggressive behavior. As little as 30 minutes of sexual experience was enough to promote female- and male-specific neural activation patterns when tested 24 hours later. The short tryst also caused males to exhibit aggression the next day, whereas males without this experience were non-aggressive. Thirty minutes of experience with a male had no such effect.

The results suggest that although mating and fighting are innate behaviors, mice's brains have to learn to tell the difference between males and females before they can properly exhibit both of these behaviors, Anderson says. "There's a learned component to these instinctive behaviors."

His team's findings also reveal that neural activity in the hypothalamus is dynamic, and can be shaped by experience, Anderson adds. Those properties indicate that this evolutionarily ancient region of the brain may be more similar to newer brain regions than previously thought, he says. He and colleagues reported their work October 18, 2017, in Nature.

Although Dulac's and Anderson's teams examined different brain regions, both researchers observed a similar relationship between sex-specific neural activity and social behavior.

But it's not yet possible to say whether the activation patterns observed by the two groups in different brain regions are influencing each other, Anderson says. The connections between the hypothalamus and the amygdala are complicated, and experiments to follow information flow between the two are on the edge of what's technically possible, he adds.

But altogether, the current work has given researchers a detailed look into the neural circuitry underlying mouse social behavior, Dulac says. "It's wonderful to have a flurry of information about what the brain of an animal says as it meets another animal and how that changes with different social experiences. For me, this is a bit of a dream come true," she says.