Colorizing images with deep neural networks

The research, entitled "Real-Time User Guided Colorization with Learned Deep Priors," is authored by a team at UC Berkeley led by Alexei A. Efros, Professor of Electrical Engineering and Computer Sciences. They will present their work at SIGGRAPH 2017, which spotlights the most innovative in computer graphics research and interactive techniques worldwide. The annual conference will be held in Los Angeles, 30 July to 3 August.

In prior work, the team trained a deep network on big visual data (a million images) to automatically colorize grayscale images, with no user intervention. While the results were sometimes very good, it was prone to certain artifacts. One major limitation was that the color of many objects -- for example, shirts -- may be inherently ambiguous. The system could only ultimately decide on one possibility.

"The goal of our previous project was to just get a single, plausible colorization," says Richard Zhang, a coauthor and PhD candidate, advised by Professor Efros. "If the user didn't like the result, or wanted to change something, they were out of luck. We realized that empowering the user and adding them in the loop was actually a necessary component for obtaining desirable results."

The new network is trained on a grayscale image, along with simulated user inputs. The system improves upon previous automatic colorization systems by enabling the user, in real-time, to correct and customize the colorization. The user provides guidance by adding colored points, or "hints," which the system then propagates to the rest of the image. The network also learns common colors for different objects and makes appropriate recommendations to the user. Though the new system is only trained on natural images -- for instance, elephants are typically brown or gray -- the system is also happy to follow the user's whims, enabling out-of-the-box coloring. For example, a pink elephant -- though unnatural -- is not off limits.

To evaluate the system, the researchers tested their interface on novice users, challenging them to produce a realistic colorization of a randomly selected grayscale image. Even with minimal training and limited time -- just one minute per image -- these users quickly learned how to produce colorizations that often fooled real human judges in a real vs. fake test scenario.

Deep networks are being more heavily used in graphics. Perhaps after conquering remaining challenges, such as streamlining memory usage and hardware requirements, along with integrating with existing image editing tools, a system like this one could find its way into commercial tools for image manipulation.