Researchers reconstruct spoken words as processed in nonhuman primate brains

"What we've done is to record the complex patterns of neural excitation in the secondary auditory cortex associated with primates' hearing specific words," said Arto Nurmikko, a professor in Brown's School of Engineering, a research associate in Brown's Carney Institute for Brain Science and senior author of the study. "We then use that neural data to reconstruct the sound of those words with high fidelity.

"The overarching goal is to better understand how sound is processed in the primate brain," Nurmikko added, "which could ultimately lead to new types of neural prosthetics."

The brain systems involved in the initial processing of sound are similar in humans and non-human primates. The first level of processing, which happens in what's called the primary auditory cortex, sorts sounds according to attributes like pitch or tone. The signal then moves to the secondary auditory cortex, where it's processed further. When someone is listening to spoken words, for example, this is where the sounds are classified by phonemes -- the simplest features that enable us to distinguish one word from another. After that, the information is sent to other parts of the brain for the processing that enables human comprehension of speech.

But because that early-stage processing of sound is similar in humans and non-human primates, learning how primates process the words they hear is useful, even though they likely don't understand what those words mean.

For the study, two pea-sized implants with 96-channel microelectrode arrays recorded the activity of neurons while rhesus macaques listened to recordings of individual English words and macaque calls. In this case, the macaques heard fairly simple one- or two-syllable words -- "tree," "good," "north," "cricket" and "program."

The researchers processed the neural recordings using computer algorithms specifically developed to recognize neural patterns associated with particular words. From there, the neural data could be translated back into computer-generated speech. Finally, the team used several metrics to evaluate how closely the reconstructed speech matched the original spoken word that the macaque heard. The research showed the recorded neural data produced high-fidelity reconstructions that were clear to a human listener.

The use of multielectrode arrays to record such complex auditory information was a first, the researchers say.

"Previously, work had gathered data from the secondary auditory cortex with single electrodes, but as far as we know this is the first multielectrode recording from this part of the brain," Nurmikko said. "Essentially we have nearly 200 microscopic listening posts that can give us the richness and higher resolution of data which is required."

One of the goals of the study, for which doctoral student Jihun Lee led the experiments, was to test whether any particular decoding model algorithm performed better than others. The research, in collaboration with Wilson Truccolo, a computational neuroscience expert, showed that recurrent neural networks (RNNs) -- a type of machine learning algorithm often used in computerized language translation -- produced the highest-fidelity reconstructions. The RNNs substantially outperformed more traditional algorithms that have been shown to be effective in decoding neural data from other parts of the brain.

Christopher Heelan, a research associate at Brown and co-lead author of the study, thinks the success of the RNNs comes from their flexibility, which is important in decoding complex auditory information.

"More traditional algorithms used for neural decoding make strong assumptions about how the brain encodes information, and that limits the ability of those algorithms to model the neural data," said Heelan, who developed the computational toolkit for the study. "Neural networks make weaker assumptions and have more parameters allowing them to learn complicated relationships between the neural data and the experimental task."

Ultimately, the researchers hope, this kind of research could aid in developing neural implants the may aid in restoring peoples' hearing.

"The aspirational scenario is that we develop systems that bypass much of the auditory apparatus and go directly into the brain," Nurmikko said. "The same microelectrodes we used to record neural activity in this study may one day be used to deliver small amounts of electrical current in patterns that give people the perception of having heard specific sounds."

The research was supported by the U.S. Defense Advanced Research Projects Agency (N66001-17-C-4013) and a private gift to Brown.