Given a grayscale photograph as input, this paper attacks the problem of hallucinating a plausible color version of the photograph. This problem is clearly underdetermined, so previous approaches have either relied on significant user interaction or resulted in desaturated colorizations. We propose a fully automatic approach that produces vibrant and realistic colorizations. We embrace the underlying uncertainty of the problem by posing it as a classification task and use class-rebalancing at training time to increase the diversity of colors in the result. The system is implemented as a feed-forward pass in a CNN at test time and is trained on over a million color images. We evaluate our algorithm using a "colorization Turing test," asking human observers to choose between a generated and ground truth color image. Our method successfully fools humans on 32% of the trials, significantly higher than previous methods. Moreover, we show that colorization can be a powerful pretext task for self-supervised feature learning, acting as a cross-channel encoder.

Automatic colorization of grayscale images has been a topic of interest since Sziranyi et al. first proposed the problem. The problem is inherently ambiguous: a grayscale image corresponds to many plausible color images. Despite this inherent ambiguity, most existing methods focus on producing a single "correct" colorization, often resulting in desaturated outputs because the loss function (typically L2 in Lab space) encourages averaging over the possible colors. We take a fundamentally different approach: we embrace the ambiguity by treating colorization as a classification problem over quantized color values, rather than a regression problem. This allows the network to hedge its bets and produce multimodal distributions over possible colors, ultimately leading to more vivid and plausible results.

We train a CNN to map from a grayscale input to a distribution over quantized color value outputs using the CIE Lab color space. Given the lightness channel L as input, our network learns to predict the corresponding a and b color channels. We quantize the ab output space into bins with grid size 10 and keep the Q = 313 values which are in-gamut. The architecture is based on VGG-net, with all pooling layers removed and replaced by strided convolutions, followed by a series of dilated convolutions. For each pixel, the network predicts a probability distribution over the 313 possible color bins, and the final colorization is obtained by mapping these distributions to point estimates in ab space.

Quantizing the color space and treating the prediction as a classification problem offers several advantages. We use a multinomial cross-entropy loss rather than L2 regression loss. However, the distribution of ab values in natural images is strongly biased towards desaturated values — backgrounds like sky, ground, and walls dominate. To address this, we rebalance the loss based on the rarity of the color at training time. We use a weighting term that is inversely proportional to the color class probability, smoothed by a temperature parameter lambda. This class rebalancing is critical for producing colorful results, as without it the network would converge to producing grayish outputs.

We train on 1.3M images from ImageNet and evaluate with several metrics. For our "colorization Turing test," we show pairs of images — one real, one colorized — to AMT workers and ask them to identify the fake. Our method achieves a fooling rate of 32.3%, compared to 22.1% for the baseline L2 regression approach. A perfect colorization would achieve 50%. We also evaluate using PSNR and perceptual metrics. While L2 regression achieves higher PSNR (because it produces mean colors), our classification approach produces results that are perceptually preferred. Additionally, we demonstrate that features learned through colorization achieve competitive performance on ImageNet classification (44.5% top-1 accuracy) when used as a self-supervised pre-training task, outperforming several other self-supervised methods.

We conduct ablation studies to analyze the effect of each component. Without class rebalancing, the fooling rate drops from 32.3% to 24.1%. Using regression instead of classification reduces the fooling rate further. The annealed-mean mapping for converting probability distributions to point estimates in ab space provides a good balance between vividness and spatial consistency. We also analyze failure cases: the method sometimes produces semantically incorrect colors (e.g., blue bananas) and struggles with scenes that have ambiguous object-color associations.

We have presented a system for fully automatic image colorization that produces realistic and vibrant results by treating the problem as multinomial classification with class rebalancing. Through a "colorization Turing test," we showed that our results fool human observers more often than previous methods. Importantly, we have also demonstrated that colorization is a viable pretext task for self-supervised representation learning, opening up new directions for learning visual features without manual annotation. Future work may explore the use of conditional generative models to handle the multimodality more explicitly.