Mobile control with facial gestures

Modern mobile devices are usually controlled by means of a touch screen. In everyday life, however, there are many situations in which this type of operation cannot be implemented. If you are wearing gloves or have your hands full with purchases, the use of smartphones, etc. is difficult, at best. Scientists from the Fraunhofer IGD are therefore looking for alternative concepts for managing mobile devices. Operation via language is an obvious choice. However, challenges such as environmental noise as well as social acceptance impose narrow limits on language-based operation. The Fraunhofer experts' solution: operation via head and face gestures, such as winking, smiling or nodding.

EarFS measures a smile in a person's ear

In the context of researching contactless operation for mobile scenarios, the Rostock researchers have evaluated various technologies with which the movements of a person's head and face can be interpreted. In this process, great importance was placed on practical use in everyday life. For example, systems that read gestures using sensors directly on one's face are very accurate and able to recognize a large number of gestures. They are so conspicuous and unpleasant to wear, though, that they are not suitable for daily use in public. Instead, systems are needed which are as inconspicuous as possible, such as EarFS, a proprietary development of the Fraunhofer IGD. This is a special ear plug that measures the muscular currents and distortions of the ear canal which occur during facial movements. The sensor detects even the smallest movements in the face through the way the shape of the ear canal changes and measures muscle currents that arise during the movement of the face or head.

"The challenge was that these currents and movements are sometimes very small and need to be intensified," explains Denys Matthies, scientist at the Fraunhofer IGD. "In addition, the sensors cannot be interfered with by other movements of the body, such as vibrations during walking or external interferences. To solve this problem, an additional reference electrode was applied to the earlobe which records the signals coming from outside." The signals detected inside the ear are matched with the signals coming from the outside -- the remaining useful signal enables clear facial gesture identification, even if the person wearing the EarFS is in motion.

Numerous application and further development possibilities

EarFS not only allows microinteractions with a smartphone, such as accepting and rejecting phone calls or operating the music player. The evaluation of facial movements also allows conclusions about weariness, exertion and other emotional states of the user. As a result, smartphones could warn drivers when signs of weariness and fatigue are registered or could switch automatically to mute when drivers are concentrating attentively. The use of the technology is also imaginable in the medical field. For example, it could help people with locked-in syndrome to communicate more easily by allowing them to operate computers with facial movements. However, this is not the end of the potential for EarFS. "Differential amplification sensing, or, in other words, the amplification of muscle currents and ear canal distortions while simultaneously filtering out external signals, fills a gap in research," says Matthies. "With the technology, we can also read activities at other parts of the body and separate them from external signals: this opens up further possibilities for us, including the complementary operation of machines in Industry 4.0."