We investigate conditional adversarial networks as a general-purpose solution to image-to-image translation problems. These networks not only learn the mapping from input image to output image, but also learn a loss function to train this mapping. This makes it possible to apply the same generic approach to problems that traditionally would require very different loss formulations. We demonstrate that this approach is effective at synthesizing photos from label maps, reconstructing objects from edge maps, and colorizing images, among other tasks. Indeed, since the release of our pix2pix software, a large number of internet users have posted their own experiments with our system, further demonstrating its wide applicability.

Many problems in image processing, computer graphics, and computer vision can be posed as "translating" an input image into a corresponding output image. Just as a concept may be expressed in either English or French, a scene may be rendered as an RGB image, a gradient field, an edge map, a semantic label map, etc. In analogy to automatic language translation, we define automatic image-to-image translation as the task of translating one possible representation of a scene into another. Traditionally, each of these tasks has been tackled with separate, special-purpose machinery. One key concern with CNNs is that they require the design of effective loss functions — naive Euclidean distance produces blurry results because it is minimized by averaging all plausible outputs.

GANs offer a solution: instead of manually designing a loss function, we can specify only a high-level goal, like "make the output indistinguishable from reality," and then automatically learn a loss function appropriate for satisfying this goal. Conditional GANs (cGANs) learn a mapping from observed image x and random noise vector z to y: G: {x, z} -> y. The generator G is trained to produce outputs that cannot be distinguished from "real" images by an adversarially trained discriminator D. Our primary contribution is to demonstrate that conditional GANs produce reasonable results across a wide variety of image-to-image translation tasks, and to present a simple framework sufficient to achieve good results, along with an analysis of several important architectural choices.

Image-to-image translation has traditionally employed per-pixel classification or regression, treating output space as "unstructured". In contrast, structured losses penalize the joint configuration of the output rather than individual pixels. Prior work conditioned GANs on discrete labels, text, and images. Previous image-conditional models addressed image prediction from a normal map, future frame prediction, and image generation from sparse annotations. However, our framework differs fundamentally: nothing is application-specific. This makes our setup considerably simpler than most others. Architecturally, we distinguish our approach through two choices: a "U-Net"-based generator and a convolutional "PatchGAN" discriminator.

The conditional GAN objective combines an adversarial loss where the generator G tries to minimize against the discriminator D trying to maximize: L_cGAN(G,D) = E[log D(x,y)] + E[log(1 - D(x,G(x,z)))]. We also find it beneficial to mix the GAN objective with a more traditional loss like L1 distance: L_L1(G) = E[||y - G(x,z)||_1]. We use L1 rather than L2 as L1 encourages less blurring. The final objective is: G* = arg min_G max_D L_cGAN(G,D) + lambda * L_L1(G). Regarding noise: the generator simply learned to ignore the noise z when provided as an explicit input. Instead, we provide noise only in the form of dropout applied at both training and test time.

A defining feature of image-to-image translation is that input and output differ in surface appearance, but both are renderings of the same underlying structure. Therefore, the structure in the input is roughly aligned with the structure in the output. We employ an encoder-decoder with skip connections, following the general shape of a "U-Net". Specifically, we add skip connections between each layer i and layer n-i, where n is the total number of layers. Each skip connection simply concatenates all channels at layer i with those at layer n-i. This allows low-level information shared between input and output to shuttle directly across the net, bypassing the bottleneck layer.

It is well established that the L1 loss already ensures low-frequency correctness. For the GAN discriminator, we therefore only need to model high-frequency structure. This motivates restricting the discriminator to only penalize structure at the scale of local image patches. We term this discriminator a PatchGAN: it classifies whether each N x N patch in an image is real or fake, running this discriminator convolutionally across the image, averaging all responses to provide the ultimate output. This design assumes independence between pixels separated by more than a patch diameter, effectively modeling the image as a Markov random field. Testing patch sizes from 1x1 to 286x286 shows that 70x70 patches achieve optimal results. Smaller patches lack spatial sharpness; full image discriminators show diminishing returns.

Experiments test diverse tasks: semantic labels to photo, architectural labels to photo, maps to aerial photos, BW to color, edges to photo, sketch to photo, day to night, thermal to color, and photo inpainting. Evaluation uses Amazon Mechanical Turk studies and FCN-score (whether synthesized images fool pre-trained classifiers). Ablation studies show L1 alone produces blurry results, cGAN alone introduces artifacts, and combining both (L1+cGAN) reduces artifacts and blurriness. U-Net substantially outperforms basic encoder-decoder with skip connections proving essential. AMT studies show map-to-photo results fool participants 18.9% of trials with L1+cGAN versus 0.8% with L1 alone. Colorization achieves 22.5% fooling rate, slightly below specialized prior work at 27.8%.

Conditional adversarial networks are a promising approach for many image-to-image translation tasks, especially those involving highly structured graphical outputs. These networks learn a loss function adapted to the task and data at hand, which makes them applicable across a wide variety of settings. The release of pix2pix has sparked significant community engagement, with artists and researchers applying it to creative applications beyond the original scope, demonstrating the framework's promise as a general-purpose image-to-image translation tool.