Augmented Reality Goggles May Offer Mere Mortals X-Ray Vision

Augmented
Reality Goggles May Offer Mere Mortals "X-Ray Vision"

"X-ray vision," once the province of comic book
superheroes, is coming to a mere mortal near you: A budding
technology called augmented reality uses computers and cameras to
mesh views from different perspectives, giving the wearer of
augmented reality goggles something like superhuman vision. Now,
a group of computer scientists at the University of Rochester --
one of only a handful of groups working on augmented reality --
has proposed a new way for cameras and computers to cooperate to
produce an image that very convincingly merges computer-generated
images and real-world videos.

This ability to seamlessly combine a visible object, like a
wall, with a whole database of images you can't see, such as an
infrared view or a blueprint of the wall's interior, has many
applications in such fields as medicine, entertainment,
maintenance, and defense. For example, instead of having to
glance back and forth between a patient and an MRI view of his
brain tumor, surgeons at Brigham and Women's Hospital in Boston
can see on a monitor the MRI view superimposed over a video image
of the patient. The next step may be augmented reality goggles --
eyewear that superimposes the MRI view directly over the naked
eye's view of the patient so that surgeons can even more
precisely eradicate only the tumor.

"It sounds futuristic, but this technology's a lot closer
than you think," says James Vallino, a Rochester graduate
student who has been working on augmented reality with Assistant
Professor Kiriakos Kutulakos. "It's been evolving slowly but
steadily, led by military, design, and medical projects, some of
which already use augmented reality."

A version of augmented reality already makes it possible for
weather forecasters to appear on the air in front of computerized
maps. There are many other high-tech applications being
investigated by the Rochester researchers and others at
institutions such as MIT, Carnegie Mellon University, and
UNC-Chapel Hill:

A soldier on the ground looking through augmented
reality goggles could see infrared images of the landscape
gleaned from a reconnaissance aircraft far above. A group of
enemy troops just beyond the top of a ridge can't be seen by the
soldier but can be detected by the aircraft -- which can then
alert the soldier by superimposing its find onto the soldier's
more limited view.

Doctors, who must often glance back and forth from
patient to medical imaging devices while performing surgery, are
benefiting from being able to see everything they need to know at
once. For example, augmented reality goggles superimpose an
ultrasound image of a developing fetus on an external view of its
mother's body, giving doctors an unprecedented view of babies as
they develop in the womb. Physicians may even don augmented
reality goggles to mix and match medical images from endoscopy,
ultrasound, MRI scans, CT scans, and x-rays -- giving them the
best views for carrying out treatment.

Repairs to buildings or complex machinery can be made
much simpler by augmented reality: If a wall's innards are stored
as a computerized blueprint, that blueprint can be superimposed
on a repairman's view of the wall to pinpoint where hidden ducts,
beams, and wiring lie. Augmented reality could help homeowners
envision home improvements before they're even started. A
remodeling consultant discussing a new deck or addition with a
homeowner would aim a video camera at the house and have the
augmented reality system show what the home would look like when
the work is completed.

The biggest problem faced by the few labs now studying
augmented reality is how to flawlessly coordinate the video and
computer graphic images in an augmented reality sequence.
"Most people want to align the video image and the
computer-generated image by building a super-accurate computer
model of the three- dimensional setting in which augmentation
takes place," says Vallino. "If you're working in a
small room or an uncomplicated space, this is easy to do. But
what if you're designing goggles for a soldier on a battlefield?
Things are always changing, and it's impossible to model that
setting."

"In previous approaches the environment had to be tightly
controlled for augmented reality to work," Kutulakos adds.
"We're now trying to move outside of the lab into less
structured environments."

The Rochester group takes a unique approach to this problem of
camera-computer coordination. Its approach uses what's called an
affine set of axes to place computer graphics into videos using
visible natural landmarks as reference points. "You could
get the computer to precisely place a three-dimensional computer-
generated character on top of a train," Vallino says.
"When the train moves, the resulting motion of landmarks
around the train is detected to keep the character correctly
positioned in the image despite the motion. Augmented reality
through such tracking is much easier to manage than the laborious
frame-by-frame manipulation now used in Hollywood to make scenes
like computerized Martians wandering through the White
House."

Vallino and Kutulakos are joined in the augmented reality
research by Chris Brown, a professor of computer science. The
research is funded by the U.S. Department of Defense's Defense
Advanced Research Projects Agency (DARPA) and the National
Science Foundation.