Mathematics and fine art: Digitizing paintings through image processing

In a paper published this month in the SIAM Journal on Imaging Sciences, authors Gloria Haro, Antoni Buades and Jean-Michel Morel propose a far simpler technique that can achieve reliable reproductions of paintings using fusion of photographs taken from different angles through statistical methods. The simple photographic procedure eliminates the need for sophisticated illumination and acquisition requirements. The postproduction process, while intensive, is fully automated.

The fusion of multiple images of a painting from well-chosen angles can eliminate glare, highlights and motion blur. Robust statistical methods reduce noise and compensate for optical distortion, thus addressing the problem of uncontrolled illumination and destructive reflection that tends to be seen in many digitized paintings.

One of the main advantages of the method described is that image fusion obviates the need for a high-performance camera. "This article demonstrates the possibility of acquiring a good quality image of a painting from amateur snapshots taken in bursts from different angles, in normal museum illumination," says senior author Jean-Michel Morel. "The photographing procedure is simple and can be done with a commercial hand-held camera by an amateur photographer." Thus, paintings can be digitized even under poor light conditions, and this includes museum pieces that may be protected by glass screens that reflect light from other objects in the room.

The only requirement is for the photographer to take as many shots from as many angles as possible, making sure that no particular detail is affected by glare in a majority of the photographs. "This acquisition is then followed by an intensive (but fully automatic) post-production chain, whose mathematical and algorithmic definition is precisely the object of the article," author Morel explains.

First, each series of images taken from the same position at the same instant -- termed "bursts" -- is fused to get single views with little noise and motion blur. A weighted average is used to obtain the sharpest possible image. Fusion of multiple images taken from the same position allows removal of noise and compression artifacts, while permitting emphasis of details. This is called burst denoising.

Highlights -- regions of an image where extreme brightness causes information to be lost -- can also be removed using photographs taken from various viewpoints. Finally, Poisson editing, a technique which allows seamless blending of images, and robust gradient statistics are used to fuse all final views from each burst. This brings images taken from different angles containing many reflections, highlights and shadows, into one final frontal-view image.

Fusion of images also does away with the need for controlled illumination, which is absolutely necessary in the case of single photograph techniques as used in the Google Project. The authors point out that with a single photograph it is much harder to eliminate glare and reflectance noise, which they find evident in white saturated regions caused by reflections in some photographs in Google's database. The image processing chain described in this paper takes the best advantage of a group of photographs and can work with as few as three. "The image fusion algorithm described in the paper can be applied to a video of the painting as well, though it would be very slow. An extension of the technique to video therefore requires a technological effort," Morel explains.

Likening photography of a painting to viewing it at a museum (several walkarounds are needed to find an optimal viewing position), the authors assert that a quality image of a painting with a complete view cannot be obtained by a single photograph, or even multiple photographs taken from the same position, regardless of the capabilities of a camera.

Once an image clear of imperfections caused by setup and illumination is obtained, color mapping, contrast, and other subjective features are left to photographers and artists. As the authors explain, "There is no absolute definition of a "good" photograph of a painting. This is an artistic question that goes beyond the scope of image processing as a technique."

The methods described in the paper could potentially be used in image restoration. In addition, these methods can be applied to 3-D objects, though the process is slow and success cannot be guaranteed. Future research will be directed toward better digitization of three-dimensional art using these techniques. "The digital recreation of 3D pieces of art is the subject of very active research to resolve its obviously more complex issues," Morel says.

This paper describes just one of many uses for the highly versatile field of image processing. View a video overview of image processing from SIAM's Annual Meeting: http://youtu.be/hie_52HxL68