Researchers use artificial intelligence to boost image quality of metalens camera

Metalenses are ultrathin optical devices -- often just a fraction of a millimeter thick -- that use nanostructures to manipulate light. Although their small size could potentially enable extremely compact and lightweight cameras without traditional optical lenses, it has been difficult to achieve the necessary image quality with these optical components.

"Our technology allows our metalens-based devices to overcome the limitations of image quality," said research team leader Ji Chen from Southeast University in China. "This advance will play an important role in the future development of highly portable consumer imaging electronics and can also be used in specialized imaging applications such as microscopy."

In Optica Publishing Group journal Optics Letters, the researchers describe how they used a type of machine learning known as a multi-scale convolutional neural network to improve resolution, contrast and distortion in images from a small camera -- about 3 cm × 3 cm × 0.5 cm -- they created by directly integrating a metalens onto a CMOS imaging chip.

"Metalens-integrated cameras can be directly incorporated into the imaging modules of smartphones, where they could replace the traditional refractive bulk lenses," said Chen. "They could also be used in devices such as drones, where the small size and lightweight camera would ensure imaging quality without compromising the drone's mobility."

Enhancing image quality

The camera used in the new work was previously developed by the researchers and uses a metalens with 1000-nm tall cylindrical silicon nitride nano-posts. The metalens focuses light directly onto a CMOS imaging sensor without requiring any other optical elements. Although this design created a very small camera the compact architecture limited the image quality. Thus, the researchers decided to see if machine learning could be used to improve the images.

Deep learning is a type of machine learning that uses artificial neural networks with multiple layers to automatically learn features from data and make complex decisions or predictions. The researchers applied this approach by using a convolution imaging model to generate a large number of high- and low-quality image pairs. These image pairs were used to train a multi-scale convolutional neural network so that it could recognize the characteristics of each type of image and use that to turn low-quality images into high-quality images.

"A key part of this work was developing a way to generate the large amount of training data needed for the neural network learning process," said Chen. "Once trained, a low-quality image can be sent from the device to into the neural network for processing, and high-quality imaging results are obtained immediately."

Applying the neural network

To validate the new deep learning technique, the researchers used it on 100 test images. They analyzed two commonly used image processing metrics: the peak signal-to-noise ratio and the structural similarity index. They found that the images processed by the neural network exhibited a significant improvement in both metrics. They also showed that the approach could rapidly generate high-quality imaging data that closely resembled what was captured directly through experimentation.

The researchers are now designing metalenses with complex functionalities -- such as color or wide-angle imaging -- and developing neural network methods for enhancing the imaging quality of these advanced metalenses. To make this technology practical for commercial application would require new assembly techniques for integrating metalenses into smartphone imaging modules and image quality enhancement software designed specifically for mobile phones.

"Ultra-lightweight and ultra-thin metalenses represent a revolutionary technology for future imaging and detection," said Chen. "Leveraging deep learning techniques to optimize metalens performance marks a pivotal developmental trajectory. We foresee machine learning as a vital trend in advancing photonics research."