Smart algorithm cleans up images by searching for clues buried in noise

Strong beams, which yield clearer images, are damaging to specimens. On the other hand, weak beams can give noisy, low-resolution images.

In a new study published in Nature Machine Intelligence, researchers at Texas A&M University describe a machine learning-based algorithm that can reduce graininess in low-resolution images and reveal new details that were otherwise buried within the noise.

"Images taken with low-powered beams can be noisy, which can hide interesting and valuable visual details of biological specimens," said Shuiwang Ji, associate professor in the Department of Computer Science and Engineering. "To solve this problem, we use a pure computational approach to create higher-resolution images, and we have shown in this study that we can improve the resolution up to an extent very similar to what you might obtain using a high beam."

Ji added that unlike other denoising algorithms that can only use information coming from a small patch of pixels within a low-resolution image, their smart algorithm can identify pixel patterns that may be spread across the entire noisy image, increasing its efficacy as a denoising tool.

Instead of solely relying on microscope hardware to improve the images' resolution, a technique known as augmented microscopy uses a combination of software and hardware to enhance the quality of images. Here, a regular image taken on a microscope is superimposed on a computer-generated digital image. This image processing method holds promise to not just cut down costs but also automatize medical image analysis and reveal details that the eye can sometimes miss.

Currently, a type of software based on a machine-learning algorithm called deep learning has been shown to be effective at removing the blurriness or noise in images. These algorithms can be visualized as consisting of many interconnected layers or processing steps that take in a low-resolution input image and generate a high-resolution output image.

In conventional deep-learning-based image processing techniques, the number and network between layers decide how many pixels in the input image contribute to the value of a single pixel in the output image. This value is immutable after the deep-learning algorithm has been trained and is ready to denoise new images. However, Ji said fixing the number for the input pixels, technically called the receptive field, limits the performance of the algorithm.

"Imagine a piece of specimen having a repeating motif, like a honeycomb pattern. Most deep-learning algorithms only use local information to fill in the gaps in the image created by the noise," Ji said. "But this is inefficient because the algorithm is, in essence, blind to the repeating pattern within the image since the receptive field is fixed. Instead, deep-learning algorithms need to have adaptive receptive fields that can capture the information in the overall image structure."

To overcome this hurdle, Ji and his students developed another deep-learning algorithm that can dynamically change the size of the receptive field. In other words, unlike earlier algorithms that can only aggregate information from a small number of pixels, their new algorithm, called global voxel transformer networks (GVTNets), can pool information from a larger area of the image if required.

When they analyzed their algorithm's performance against other deep-learning software, the researchers found that GVTNets required less training data and could denoise images better than other deep-learning algorithms. Furthermore, the high-resolution images obtained were comparable to those obtained using a high-energy light beam.

The researchers noted that their new algorithm can easily be adapted to other applications in addition to denoising, such as label-free fluorescence imaging and 3D to 2D conversions for computer graphics.

"Our research contributes to the emerging area of a smart microscopy, where artificial intelligence is seamlessly integrated into the microscope," Ji said. "Deep-learning algorithms such as ours will allow us to potentially transcend the physical limit posed by light that was not possible before. This can be extremely valuable for a myriad of applications, including clinical ones, like estimating the stage of cancer progression and distinguishing between cell types for disease prognosis."

This research is funded by the National Science Foundation, the National Institutes of Health and the Defense Advanced Research Projects Agency.