When cars go to driving school

Drivers go to school to learn to anticipate emerging situations and respond appropriately. Why shouldn't cars do the same? That's the question Florentin Wörgötter and his colleagues at the EU-fundedresearch programme DRIVSCO asked themselves three years ago.

Their answer was that, with state-of-the-art sensors, image processors, and learning algorithms, a car that smart could be built.

The result, now tested in a prototype vehicle, is a system that tracks a driver's every move, matches those actions with what it "sees" down the road, and learns how that driver normally handles situations such as upcoming curves or other vehicles ahead.

With its infrared headlights, stereo cameras, and advanced visual processing the system can actually see better at night than a human driver. It has proved its worth by providing early warnings of hazards a human driver had not yet seen or reacted to.

"What we wanted was a system that learns to drive during the day by correlating what it sees with the actions a driver takes," says Wörgötter. "Then at night the system could say, 'Slow down, a curve is coming up!' -- a curve the human didn't see. Now we have a prototype that does this."

Sharper, smarter computer vision

When artificial intelligence researchers first set out to give machines vision they had no idea what an enormous problem they were taking on. A scene that makes perfect sense to us -- a clearly defined roadway curving into the distance, trees and signs slipping by, other vehicles scattered in the lanes ahead, some moving at our speed, others pulling away or looming closer -- starts as nothing more than a sea of coloured pixels to a computer.

The DRIVSCO researchers drew their inspiration from what's been learned in recent years about how the brains of humans and other animals do such a remarkable job of making sense of the patterns of light dancing over their retinas. A key feature turns out to be constant two-way feedback between higher- and lower-level visual areas.

As we drive, high-level visual areas that store complex perceptions such as 'car getting closer' or 'person crossing the road' are constantly active. These areas send messages -- feedback -- that interact with incoming signals representing more basic features such as edges, colours, and movement. When there's a match, an object pops out of the background, complete with perceptions of its size, location, and movement.

"How the visual front-end of DRIVSCO works was very much inspired by the visual cortex of vertebrates," says Wörgötter.

"The feedback mechanism, where higher-level modules interact with modules that detect simpler features, solves the very difficult problem of detecting independent objects even when you and they are moving at the same speed."

School time for smart cars

Having provided their prototype car with an advanced vision system, the DRIVSCO team next set out to make it smart enough to learn how to drive.

"The idea was that cars should be able to learn from the driver to be capable of driving autonomously," says Wörgötter. Since cars aren't legally allowed to drive themselves, he adds, the system limits itself to providing a warning when the driver isn't responding to an upcoming situation as expected.

In the future, however, the system might also provide more insistent feedback. For example, Wörgötter says, if you were heading off the road it would make steering much stiffer in that direction and much lighter in the direction that would get you back on track.

The system learns by building up a huge database of associations between the driving situations it sees -- for example heading into curves at various speeds -- and the actions the driver takes in terms of steering and speed changes.

The system looks at the scene, analyses it, and matches it with the actions the driver is taking. This cycle repeats 20 times per second. "So you get a whole stream of vision-action links," Wörgötter says.

Like a person learning to drive, the system gets better over time. After processing terabytes of information, the DRIVSCO system was able to produce consistent real-time predictions of how a particular driver would handle most highway or country road situations. City driving situations are still too complex for it to master.

Wörgötter was particularly pleased when the system proved able to generalise what it had learned to new roads and novel situations.

The system also showed that it could learn individual driving styles. "An old granny may not want to drive as fast as Michael Schumacher," Wörgötter says. "This is quite important in the car industry."

Wörgötter expects that after the current financial crisis eases, major car manufacturers will want to incorporate DRIVSCO's vision and learning advances into their high-end vehicles.

"The ability to learn from a driver is quite new," Wörgötter says. "I think it has great potential as a commercial product."