Reproducing paintings that make an impression

"RePaint" uses a combination of 3-D printing and deep learning to authentically recreate your favorite paintings -- regardless of different lighting conditions or placement. RePaint could be used to remake artwork for your home, protect originals from wear-and-tear in museums, or even help companies create prints and postcards of historical pieces.

"If you just reproduce the color of a painting as it looks in the gallery, it might look different in your home," says Changil Kim, one of the authors on a new paper about the system, which will be presented at ACM SIGGRAPH Asia in December. "Our system works under any lighting condition, which shows a far greater color reproduction capability than almost any other previous work."

To test RePaint, the team reproduced a number of oil paintings created by their artist collaborator. The team found that RePaint was more than four times more accurate than state-of-the-art physical models at creating the exact color shades for different artworks.

At this time the reproductions are only about the size of a business card, due to the time-costly nature of printing. In the future the team expects that more advanced, commercial 3-D printers could help with making larger paintings more efficiently.

How it works While 2-D printers are most commonly used for reproducing paintings, if you have a blue sky or a purple dress in your work of art, you might be out of luck. That's because 2-D printers have a fixed set of just four inks (cyan, magenta, yellow, and black).

The researchers, however, found a better way to capture a fuller spectrum of Degas and Dali. They used a special technique they developed called "color-contoning," which involves using a 3-D printer and 10 different transparent inks stacked in very thin layers, much like the wafers and chocolate in a Kit-Kat bar. They combined their method with a decades-old technique called "halftoning," where an image is created by tons of little ink dots, rather than continuous tones. Combining these, the team says, better captured the nuances of the colors.

With a larger color scope to work with, the question of what inks to use for which paintings still remained. Instead of using more laborious physical approaches, the team trained a deep learning model to predict the optimal stack of different inks. Once the system had a handle on that, they then fed in images of paintings, and used the model to determine what colors should be used in what particular areas for specific paintings.

Despite the progress so far, the team says they have a few improvements to make before they can whip up a dazzling dupe of "Starry Night." According to mechanical engineer Mike Foshey, they couldn't completely reproduce certain colors like cobalt blue due to a limited ink library.

In the future they plan to expand this library, as well as create a painting-specific algorithm for selecting inks. They also can hope to achieve better detail to account for aspects like surface texture and reflection, so that they can achieve specific effects such as glossy and matte finishes.

"The value of fine art has rapidly increased in recent years, so there's an increased tendency for it to be locked up in warehouses away from the public eye," says Foshey. "We're building the technology to reverse this trend, and to create inexpensive and accurate reproductions that can be enjoyed by all."

Kim and Foshey worked on the system alongside lead author Liang Shi, MIT professor Wojciech Matusik, former MIT postdoc Vahid Babaei, now Group Leader at Max Planck Institute of Informatics, Princeton University computer science professor Szymon Rusinkiewicz, and former MIT postdoc Pitchaya Sitthi-Amorn, now a lecturer at Chulalongkorn University in Bangkok, Thailand.