An innovative mixed light field technique for immersive projection mapping

Projection mapping (PM) is a fascinating technology that provides an immersive visual experience by projecting computer-generated images onto physical surfaces, smoothly merging real and virtual worlds. It allows us to experience augmented reality without the need for special glasses. As a result, PM is in high demand in various fields including enhanced stage productions, trying on clothing and make-up, and educational demonstrations.

Despite its potential, current PM methods face challenges in bright environments with ambient lighting. Ambient lighting drowns the entire scene in light, reducing the contrast of PM targets. This is why conventional PM solutions mainly function in dark environments. However, even in dark environments, PM fails to provide a natural scene, as in a dimly lit scene, only the PM target is well-lit while the rest of the surroundings remain dark, causing it to appear overly bright. Additionally, non-PM objects appear too dark, breaking the immersion.

To address these issues, a team of researchers from Japan, led by Associate Professor Yoshihiro Watanabe from the Department of Information and Communications Engineering at Tokyo Institute of Technology, has recently developed an innovative new mixed light field approach for achieving PM in brightly lit environments. "In this approach, instead of using normal ambient light, we introduced a mixed light field in which a ray-controllable light avoids the PM target while adequately lighting other areas within the scene while the PM projector exclusively illuminates the target," explains Dr. Watanabe. Their findings were published in the journal IEEE Transactions on Visualization and Computer Graphics and will be presented at the31stIEEE Conference on Virtual Reality and 3D User Interfaces in Orlando, Florida USA, with the presentation scheduled for March 19th at 1:30 PM local time(UTC-4).

At the center of this novel approach lies the ray-controllable lighting unit. This unit reproduces a range of ambient lighting scenarios while also avoiding illuminating the PM target. To achieve this, the researchers developed a novel kaleidoscopic array comprising an array of mirrors positioned behind a lens array, which, in turn, was placed in front of a projector. This setup allowed the projector to produce a high-density light field, crucial for ray-controllable lighting.

Furthermore, to avoid illuminating the PM target, the researchers deployed a camera to capture images of the scene and identify the pixels from the projector that illuminated the PM targets, subsequently turning them off. To identify these pixels, they employed a simple binary-search-based method, resulting in effective mixed light field.

This innovative approach allowed them to achieve high-contrast PM presentations in brightly lit surroundings. Notably, it preserved the natural appearance and shadows of ordinary non-PM objects, addressing a key challenge in PM technology. Through several captivating augmented scenes, the researchers showcased the seamless coexistence of PM targets and ordinary objects, providing an immersive visual experience.

While the researchers identified some limitations, such as artefacts and the low efficiency of binary search algorithms with large PM targets, they have already identified potential solutions and are actively working to expand this approach in the future.

"Our experiments prove the effectiveness of using this technique for achieving natural PM presentations with accurate lighting for all objects. Mixed light field has the potential to usher PM for various practical day-to-day applications, such as attractions, for support in manufacturing and trying on make-up," says Dr. Watanabe, highlighting the applications of their technology.

Overall, this approach marks a significant step for PM technology, paving the way for immersive augmented experiences in future.