Brain regions that encode words, grammar, story identified

Researchers from CMU's Machine Learning Department performed functional magnetic resonance imaging (fMRI) scans of eight people as they read a chapter of that Potter book. They then analyzed the scans, cubic millimeter by cubic millimeter, for every four-word segment of that chapter. The result was the first integrated computational model of reading, identifying which parts of the brain are responsible for such subprocesses as parsing sentences, determining the meaning of words and understanding relationships between characters.

As Leila Wehbe, a Ph.D. student in the Machine Learning Department, and Tom Mitchell, the department head, recently reported in the online journal PLOS ONE, the model was able to predict fMRI activity for novel text passages with sufficient accuracy to tell which of two different passages a person was reading with 74 percent accuracy.

"At first, we were skeptical of whether this would work at all," Mitchell said, noting that analyzing multiple subprocesses of the brain at the same time is unprecedented in cognitive neuroscience. "But it turned out amazingly well and now we have these wonderful brain maps that describe where in the brain you're thinking about a wide variety of things."

Wehbe and Mitchell said the model is still inexact, but might someday be useful in studying and diagnosing reading disorders, such as dyslexia, or to track the recovery of patients whose speech was impacted by a stroke. It also might be used by educators to identify what might be giving a student trouble when learning a foreign language.

"If I'm having trouble learning a new language, I may have a hard time figuring out exactly what I don't get," Mitchell said. "When I can't understand a sentence, I can't articulate what it is I don't understand. But a brain scan might show that the region of my brain responsible for grammar isn't activating properly, or perhaps instead I'm not understanding the individual words."

Researchers at Carnegie Mellon and elsewhere have used fMRI scans to identify activation patterns associated with particular words or phrases or even emotions. But these have always been tightly controlled experiments, with only one variable analyzed at a time. The experiments were unnatural, usually involving only single words or phrases, but the slow pace of fMRI -- one scan every two seconds -- made other approaches seem unfeasible.

Wehbe nevertheless was convinced that multiple cognitive subprocesses could be studied simultaneously while people read a compelling story in a near-normal manner. She believed that using a real text passage as an experimental stimulus would provide a rich sample of the different word properties, which could help to reveal which brain regions are associated with these different properties.

"No one falls asleep in the scanner during Leila's experiments," Mitchell said.

They devised a technique in which people see one word of a passage every half second -- or four words for every two-second fMRI scan. For each word, they identified 195 detailed features -- everything from the number of letters in the word to its part of speech. They then used a machine learning algorithm to analyze the activation of each cubic centimeter of the brain for each four-word segment.

Bit by bit, the algorithm was able to associate certain features with certain regions of the brain, Wehbe said.

"The test subjects read Chapter 9 of Sorcerer's Stone, which is about Harry's first flying lesson," she noted. "It turns out that movement of the characters -- such as when they are flying their brooms -- is associated with activation in the same brain region that we use to perceive other people's motion. Similarly, the characters in the story are associated with activation in the same brain region we use to process other people's intentions."

Exactly how the brain creates these neural encodings is still a mystery, they said, but it is the beginning of understanding what the brain is doing when a person reads.

"It's sort of like a DNA fingerprint -- you may not understand all aspects of DNA's function, but it guides you in understanding cell function or development," Mitchell said. "This model of reading initially is that kind of a fingerprint."

A complementary study by Wehbe and Mitchell, presented earlier this fall at the Conference on Empirical Methods in Natural Language Processing, used magnetoencephalography (MEG) to record brain activity in subjects reading Harry Potter. MEG can record activity every millisecond, rather than every two seconds as in fMRI scanning, but can't localize activity with the precision of fMRI. Those findings suggest how words are integrated into memory -- how the brain first visually perceives a word and then begins accessing the properties of the word, and fitting it into the story context.