Computers That Understand How You Feel

Dialogue systems are computer systems which communicate with humans and which are used especially for information provision such as in the speaking computer that provides travel information. Normally these computers do not take human emotions into account even though this is an important component of human interaction. The problem with human emotions is that they are often difficult to interpret, and that is especially so for a computer.

Raising one’s voice can, for example, indicate enthusiasm but it can also be a sign of anger. We require extra information to be sure which of the two emotions is present. Human beings are trained to combine various types of information (which may be quite vague) and still be able to draw the correct conclusions. Dealing with uncertainties is however difficult to program into computer software.

Taking emotions into account

Bui developed a dialogue system that – unlike others – could take emotions into account. To do this, he used a mathematical technique developed in the 1960s for controlling factory processes called Partially Observable Markov Decision Process (POMDP). He demonstrated that this technique was suitable for integrating the user’s emotions into a dialogue system because it could deal with uncertainties.

The method performs better than existing systems as long as it is tested with small-scale dialogue problems. However, for larger problems the method requires too much calculating power. That is why Bui developed a hybrid strategy which combines the Dynamic Decision Network (DDN) technique with POMDP. In contrast to the latter, the DDN-POMDPs split dialogue systems into two levels. They make a choice between looking far ahead and seeing whether the necessary calculating power is available.

To illustrate the effectiveness of the DDN-POMDP, Bui applied it to a navigation system for emergency services that took the stress experienced by the user into account. The navigation system receives input from a separate stress module that measures an emergency worker’s stress levels, taking these into account when the user is in communication with the system. Whenever the user’s stress levels become raised, the system will anticipate, for example, that the user is more likely to make mistakes and for that reason will request confirmation more often.