First AI system for contactless monitoring of heart rhythm using smart speakers

But what about tracking something even smaller: the minute motion of individual heartbeats in a person sitting in front of a smart speaker?

UW researchers have developed a new skill for a smart speaker that for the first time monitors both regular and irregular heartbeats without physical contact. The system sends inaudible sounds from the speaker out into a room and, based on the way the sounds are reflected back to the speaker, it can identify and monitor individual heartbeats. Because the heartbeat is such a tiny motion on the chest surface, the team's system uses machine learning to help the smart speaker locate signals from both regular and irregular heartbeats.

When the researchers tested this system on healthy participants and hospitalized cardiac patients, the smart speaker detected heartbeats that closely matched the beats detected by standard heartbeat monitors. The team published these findings March 9 in Communications Biology.

"Regular heartbeats are easy enough to detect even if the signal is small, because you can look for a periodic pattern in the data," said co-senior author Shyam Gollakota, a UW associate professor in the Paul G. Allen School of Computer Science & Engineering. "But irregular heartbeats are really challenging because there is no such pattern. I wasn't sure that it would be possible to detect them, so I was pleasantly surprised that our algorithms could identify irregular heartbeats during tests with cardiac patients."

While many people are familiar with the concept of a heart rate, doctors are more interested in the assessment of heart rhythm. Heart rate is the average of heartbeats over time, whereas a heart rhythm describes the pattern of heartbeats.

For example, if a person has a heart rate of 60 beats per minute, they could have a regular heart rhythm -- one beat every second -- or an irregular heart rhythm -- beats are randomly scattered across that minute but they still average out to 60 beats per minute.

"Heart rhythm disorders are actually more common than some other well-known heart conditions. Cardiac arrhythmias can cause major morbidities such as strokes, but can be highly unpredictable in occurrence, and thus difficult to diagnose," said co-senior author Dr. Arun Sridhar, assistant professor of cardiology at the UW School of Medicine. "Availability of a low-cost test that can be performed frequently and at the convenience of home can be a game-changer for certain patients in terms of early diagnosis and management."

The key to assessing heart rhythm lies in identifying the individual heartbeats. For this system, the search for heartbeats begins when a person sits within 1 to 2 feet in front of the smart speaker. Then the system plays an inaudible continuous sound, which bounces off the person and then returns to the speaker. Based on how the returned sound has changed, the system can isolate movements on the person -- including the rise and fall of their chest as they breathe.

"The motion from someone's breathing is orders of magnitude larger on the chest wall than the motion from heartbeats, so that poses a pretty big challenge," said lead author Anran Wang, a doctoral student in the Allen School. "And the breathing signal is not regular so it's hard to simply filter it out. Using the fact that smart speakers have multiple microphones, we designed a new beam-forming algorithm to help the speakers find heartbeats."

The team designed what's called a self-supervised machine learning algorithm, which learns on the fly instead of from a training set. This algorithm combines signals from all of the smart speaker's multiple microphones to identify the elusive heartbeat signal.

"This is similar to how Alexa can always find my voice even if I'm playing a video or if there are multiple people talking in the room," Gollakota said. "When I say, 'Hey, Alexa,' the microphones are working together to find me in the room and listen to what I say next. That's basically what's happening here but with the heartbeat."

The heartbeat signals that the smart speaker detects don't look like the typical peaks that are commonly associated with traditional heartbeat monitors. The researchers used a second algorithm to segment the signal into individual heartbeats so that the system could extract what is known as the inter-beat interval, or the amount of time between two heartbeats.

"With this method, we are not getting the electric signal of the heart contracting. Instead we're seeing the vibrations on the skin when the heart beats," Wang said.

The researchers tested a prototype smart speaker running this system on two groups: 26 healthy participants and 24 hospitalized patients with a diversity of cardiac conditions, including atrial fibrillation and heart failure. The team compared the smart speaker's inter-beat interval with one from a standard heartbeat monitor. Of the nearly 12,300 heartbeats measured for the healthy participants, the smart speaker's median inter-beat interval was within 28 milliseconds of the standard monitor. The smart speaker performed almost as well with cardiac patients: of the more than 5,600 heartbeats measured, the median inter-beat interval was within 30 milliseconds of the standard.

Currently this system is set up for spot checks: If a person is concerned about their heart rhythm, they can sit in front of a smart speaker to get a reading. But the research team hopes that future versions could continuously monitor heartbeats while people are asleep, something that could help doctors diagnose conditions such as sleep apnea.

"If you have a device like this, you can monitor a patient on an extended basis and define patterns that are individualized for the patient. For example, we can figure out when arrhythmias are happening for each specific patient and then develop corresponding care plans that are tailored for when the patients actually need them," Sridhar said. "This is the future of cardiology. And the beauty of using these kinds of devices is that they are already in people's homes."

This research was funded by the National Science Foundation.