Phoneme project creates new haptic communications future

Hong Tan, a professor of electrical and computer engineering, was the lead professor on the haptic research that developed a method to receive messages by learning to interpret signals such as a buzzing sensation and others through the skin on the forearm.

The research results were presented Friday (June 15) at the Proceedings of EuroHaptics 2018 conference in Pisa, Italy.

The yearlong project was done with researchers from the Massachusetts Institute of Technology and Facebook Inc.

Tan, founder and director of Purdue's Haptic Interface Research Laboratory, said while the research lends itself to being used by hearing-impaired and visually impaired users, the study is also being looked at on general terms for any number of possible uses.

"We are collaborating with Facebook through the company's Sponsored Academic Research Agreement. Facebook is interested in developing new platforms for communication and the haptic research we are doing has been promising," she said.

"I'm excited about this ... imagine a future where you're able to wear a sleeve that discreetly sends messages to you -- through your skin -- in times when it may be inconvenient to look at a text message," Tan said. "I'm really hoping this takes off as a general idea for a new way to communicate.

"When that happens, the hearing-impaired, the visually-impaired, everyone can benefit."

In the study, subjects used a material cuff encircling the forearm from the wrist to below the elbow. The instrument, wrapped around the test subject's non-dominant arm, featured 24 tactors that, when stimulated, emitted a vibration against the skin, changing quality and position in the process.

Tan said the 39 phonemes (units of sound in a language that distinguish one word from another) in the English language were mapped using signals from specific tactors. The sounds of consonants such as K, P and T were stationary sensations on different areas of the arm while vowels were indicated by stimulations that moved up, down or around the forearm.

For video on vowel sensations, go to https://www.youtube.com/watch?v=CYfqcdnvMyE&feature=youtu.be.

"We used anything that can help you establish the mapping and recognize and memorize it," Tan said. "This is based on a better understanding of how to transmit information through the sense of touch in a more intuitive and effective manner."

Twelve subjects learned haptic symbols through the phoneme-based method at a schedule of 100 words in 100 minutes for the research, while 12 others learned using a word-based system with the haptic signals on their arm.

Research results showed the phonemes worked better than a word-based approach by providing a more consistent path for user learning in a shorter period of time. Performance levels varied greatly among the test subjects for each method, but with the phoneme-based approach at least half could perform at 80 percent accuracy while two subjects reached 90 percent accuracy.

Tan said using phonemes was more efficient compared to letters, noting there are less phonemes in a word compared with the number of letters.

She said the reasoning behind the project is the thought that there are many ways to encode speech into feelings. The project goal is to demonstrate one way that actually works.

"For this research, the learning progress is one of the key things," Tan said. "With this, not only do we have a system that works, but we're able to train people within hours rather than months or even years."

For the study, a concentrated, intense training regimen was developed where the test subjects worked for about 10 minutes a day. They were then tested on their progress.

"It is more efficient than if they sit here for three hours to study," Tan said. "You can't keep good concentration for that long."

"People who are hearing-impaired may be motivated to spend additional time to training themselves, but the general public probably doesn't have the patience," she added.