AI improves touchscreen interfaces for users with impairments

"The majority of available user interfaces are targeted at average users. This "one size fits all" thinking does not consider individual differences in abilities -- the ageing and disabled users have a lot of problems with daily technology use, and often these are very specific to their abilities and the circumstances," says postdoctoral researcher Jussi Jokinen at Aalto University.

"There are ways to automatically optimize the user interface, but this is efficient only if we have a realistic model of the user. Previously, designers did not have detailed models that are based on psychological research and can be used to predict, how different individuals perform in interactive tasks," he goes on.

Example problems for users with disabilities

• Essential tremor hinders the user's ability to point accurately, leading to enormous difficulties with touch-based interfaces. An optimizer can increase the size of the user interface elements and group functions together to adapt to the screen size constraints.
• Dyslexia makes proofreading typed text and reading words of the user interface more time consuming and error-prone. An optimizer can adjust the number of text in the user interface and introduce aids for making sure that the typed text is correct.
• Dementia decreases the ability to think and remember, making the use of most everyday user interfaces difficult or impossible. An optimizer can suggest designs, which minimize the memory load and require as little previous knowledge from the user as possible. They prioritize frequent or important tasks.

The basis of their work is a new predictive model of interaction, which predicts how individual abilities affect text entry on a touch screen. The model combines psychological research on finger pointing and eye movements, allowing it to predict text entry speed, typing errors, and proofreading.

To showcase the model, researchers used it to simulate a user with essential tremor. They predicted that typing with a smartphone that has the normal Qwerty keyboard is almost impossible for such a user, because more than half of all typed keys are typos. "After this prediction, we connected the text entry model to an optimizer, which iterates through thousands of different user interface designs. No real user could of course try out all these designs. For this reason it is important that we could automatise the evaluation with our computational model," Jokinen tells.

The result of the optimization was a text entry interface, which the model predicted to be superior for users with essential tremor. There were almost no text entry errors in the typing any more. After the model's prediction, the optimized layout was tested with a real user with essential tremor. The model's prediction and real-life observations coincided, and the user was able to type almost error-free messages. "This is of course just a prototype interface, and not intended for consumer market. Our work as researchers is to come up with solutions," Jokinen reminds. "I hope that designers pick up from here and with the help of our model and optimizer create individually targeted, polished interfaces."

The model can be used to simulate a variety of users in different interactive tasks. "We started with text entry, which is an everyday task. We chose to simulate and optimize for essential tremor, because it makes text entry very difficult. Now that we have confirmed the validity and usefulness of the model, it can be extended to other use cases and disabilities. For example, we have models for simulating how being a novice or an expert with an interface impacts user's performance. We can also model how memory impairments affect learning and everyday use of interfaces. The important point is that no matter the ability or disability, there must be a psychologically valid theory behind modelling it. This makes the predictions of the model believable, and the optimization is targeted correctly," Jokinen sums up.

The results of the research are published in IEEE Pervasive Computing Journal. The research was a collaboration between Aalto University, Finland, and Kochi Technical University, Japan. The research group at Aalto, led by professor Antti Oulasvirta, focuses on computational optimization methods and mathematical models of human behaviour to design user interfaces.