'Unmaking' a move: Correcting motion blur in single-photon images

When taking a picture with a regular CMOS camera, like the ones on smartphones, the camera sensor is open to a large influx of photons during a predefined exposure time. Each pixel in the sensor grid outputs an analog value that depends on the number of photons that hit that pixel during exposure. However, this type of imaging has few ways to deal with moving objects; the movement of the object has to be much slower than the exposure time to avoid blurring. In contrast, single-photon cameras capture a rapid burst of consecutive frames with very short individual exposure times. These frames are binary -- a grid of 1s and 0s that respectively indicate whether one photon arrived at each pixel or not during exposure. To reconstruct an actual picture from these binary frames (or bit planes), many of them have to be processed into a single non-binary image. This can be achieved by assigning different levels of brightness to all the pixels in the grid, depending on how many of the bit planes had a "1" for each pixel.

Besides its higher speed, the completely digital nature of single-photon imaging allows for designing clever image reconstruction algorithms that can make up for technical limitations or difficult scenarios. At Tokyo University of Science, Japan, Professor Takayuki Hamamoto has been leading a research team focused on taking the capabilities of single-photon imaging further. In the latest study by Prof. Hamamoto and his team, which was published in IEEE Access, they developed a highly effective algorithm to fix the blurring caused by motion in the imaged objects, as well as common blurring of the entire image such as that caused by the shaking of the camera.

Their approach addresses many limitations of existing deblurring techniques for single-photon imaging, which produce low-quality pictures when multiple objects in the scene are moving at different speeds and dynamically overlapping each other. Instead of adjusting the entire image according to the estimated motion of a single object or on the basis of spatial regions where the object is considered to be moving, the proposed method employs a more versatile strategy.

First, a motion estimation algorithm tracks the movement of individual pixels through statistical evaluations on how bit values change over time (over different bit planes). In this way, as demonstrated experimentally by the researchers, the motion of individual objects can be accurately estimated. "Our tests show that the proposed motion estimation technique produced results with errors of less than one pixel, even in dark conditions with few incident photons," remarks Prof. Hamamoto.

The team then developed a deblurring algorithm that uses the results of the motion estimation step. This second algorithm groups pixels with a similar motion together, thereby identifying in each bit plane separate objects moving at different speeds. This allows for deblurring each region of the image independently according to the motions of objects that pass through it. Using simulations, the researchers showed that their strategy produced very crisp and high-quality images, even in low-light dynamic scenes crowded with objects coursing at disparate velocities.

Overall, the results of this study aptly showcase how greatly single-photon imaging can be improved if one gets down to developing effective image processing techniques. "Methods for obtaining crisp images in photon-limited situations would be useful in several fields, including medicine, security, and science. Our approach will hopefully lead to new technology for high-quality imaging in dark environments, like outer space, and super-slow recording that will far exceed the capabilities of today's fastest cameras," says Prof. Hamamoto. He also states that even consumer-level cameras might timely benefit from progress in single-photon imaging.