Web-ifying the real world

In the future, real life will be web-ified. A host of advanced tools and techniques conceived at the leading edge of several scientific disciplines will be able to capture everyday interactions and apply the online tools of knowledge discovery to better understand them.

This web of human-to-human interaction will be the result of advanced automatic speech-to-text transcription, as well as all-points analyses of gestures, distances between people, intonation, gaze, expression, and body language. Once captured, these and other signifiers of the social dynamic will offer a vast data store to which the researchers will apply search and analyses of every sort.

It can be the basis for new forms of as yet unimagined interaction and study, and it will, the researchers hope, make all forms of social interaction more beneficial in government, education, politics and business.

Getting there

We are not there yet, but we are getting there fast. And the effort has already led to significant commercial success.

Web-fying real life is the vision of the AMI consortium, which stands for Augmented Multiparty Interaction. The consortium is the force behind two highly successful European projects, AMI and AMIDA (http://www.amiproject.org/).

Both projects were well funded, featured a who's who of world-class research institutions and industrial partners like Noldus, Philips, the IDIAP Research Institute in Switzerland, the Universities of Sheffield, Edinburgh and Twente and TNO in the Netherlands, among others.

They remain a key exercise in the corporate world because of their potential, a potential often unrealised because participants' recollection of what was said varies, their focus of interest diverges, or because their responsibility lies in one small area.

Good target

The AMI consortium started with meetings. The team sought to capture all meeting information, apply web technologies for knowledge discovery, and use artificial intelligence (AI) and advanced signal processing techniques to make meetings more effective via a wide variety of tools to sort and present relevant information, such as decisions taken, views expressed, topics discussed and ideas generated.

For all their venerable tradition, meetings have a bad reputation, the butt of innumerable jokes, dismissed as exercises in time wasting and conflict that add confusion rather than clarity. Yet they remain an essential component for business.

But there is no complete solution that overcomes the weaknesses and enhances the strengths. As such, they offer a ready route for technology transfer and commercialisation. They also present many difficult, and consequently interesting, research problems.

The AMI project had as its ambition to create a browser for corporate meetings, a single interface for people to access text, video and audio of an AMI-powered meeting, including capture of presentations, documents, whiteboard exercises and notes taken by participants.

"We wanted to have a complete, easily accessed record of every aspect of the meeting. Over time, this information makes a database that can be accessed as part of a project archive, or as a search for specific types of meeting," explains Prof. Hervé Bourlard, director of the IDIAP Research Institute, and co-coordinator of the AMI project, together with Prof. Steve Renals of the University of Edinburgh.

AMIDA built on this work to create tools for Augmented Multiparty Interaction Distance Access. Here meeting participants can follow and intervene real time during a meeting. It also enables people to catch up if they are late, with a complete summary of meeting progress to date.

The summary is developed in real time by focusing on important sentences and phrases. Importance is established by the machine based on sentence length, keywords and the status of the speaker, for example the chairperson of the meeting. A remote participant can quickly get up to speed.

Social dynamics

Another example is social dynamics. A big problem for people accessing a meeting from a distance is the lack of socially dynamic context provided by text on a page. Gestures, for example, convey vast information that we intuitively acquire, but remains largely absent from the printed word. AMIDA developed software to record social dynamics of a meeting in real time.

The software can track gestures, indicate gaze and head movements, and add speech bubbles to indicate who is speaking to whom.

All this data is captured and available for later analysis and review. But beyond this value, the technology has a host of other potential applications like robotics, AI, and image recognition.

For example, there are thousands of CCTV cameras in most modern cities, but they currently act passively -- they can be consulted after a violent crime, but it is very difficult to track them in real time. If software can detect gestures indicating violence or aggression, however, the police could be alerted to a possible crime in progress.

Or at a large, fractious public meeting, the chairperson might not notice someone has a hand up to speak, but AMIDA's software could capture that. Or a waiter with a Bluetooth headset could be alerted to a customer behind him trying to get his attention.

These are just the obvious blue-sky applications where AMI could lead. The consortium studied dozens more -- most of which have potential commercial applications.

"Many people might, mistakenly, believe that this is the end," explains Bourlard. "We have many significant research breakthroughs that deliver a suite of technologies which represent a large advance on the state of the art. Our work has put us in the lead internationally, and many of the technologies are currently in various stages of commercialisation."

End of the beginning

"But this is not the end," Bourlard states emphatically. "This is just the beginning. We have taken technologies and enhanced them, then combined them in useful and effective new ways. But this is merely a platform from which we can make really impressive advances. We now have an effective set of working tools, but it is what those tools will enable us to do that really matters."

"Companies like Microsoft and NTT have caught on to the significance of what we are doing and they are working hard to catch up. If we stop now it will not take them long to overtake us. On the other hand, other companies, like EADS and Nokia are developing collaborations with the AMI consortium."

This is the point where Bourlard starts talking about web-ifying the real world, using AMI technologies already developed, and yet to come, to capture all the intangible and information-rich social data about human-to-human interactions -- information that remains largely invisible in the historical record.

Bourlard describes enthusiastically a scenario with large meetings or brainstorming sessions which capture all the intangible date. "So people can back track and see how a decision was made, what were the circumstances that led to it."

It all means more accountability and transparency, and should in the near future lead to a better decision-making processes. Already, AMIDA-powered technology is transforming the worlds of business and education.

The AMIDA project received funding from the ICT strand of the EU's Sixth Framework Programme for research.