We present a method for synthesizing high-resolution photorealistic images from semantic label maps using conditional generative adversarial networks (conditional GANs). Conditional GANs have enabled a variety of applications, but the results are often limited to low-resolution and still far from realistic. In this work, we generate 2048 x 1024 visually appealing results with a novel adversarial loss, as well as new multi-scale generator and discriminator architectures. Furthermore, we extend our framework to interactive visual manipulation with both semantic and instance-level control. To validate the importance of the different components, we perform detailed ablation studies and comparisons against existing methods.

Image-to-image translation using conditional GANs has been demonstrated on a wide range of tasks, such as synthesizing photos from label maps, reconstructing objects from edge maps, and colorizing images. However, existing methods produce limited resolution results (typically up to 256 x 256). Generating higher resolution images requires larger networks with deeper or wider architectures, which makes GAN training notoriously unstable. The memory constraints of current hardware further limit the achievable resolution.

We address this problem with a coarse-to-fine generator architecture, a multi-scale discriminator architecture, and a robust adversarial learning objective function. Specifically, we decompose the generator into two sub-networks: G1 (global generator) operates at 1024 x 512 resolution, and G2 (local enhancer) outputs images at 2048 x 1024. For the discriminator, we use 3 discriminators that operate at different image scales. We also introduce an instance-level feature encoder that enables diverse image synthesis and instance-level semantic manipulation.

Generative adversarial networks (GANs) have been applied to various image generation and manipulation tasks. Pix2pix proposes a general framework for image-to-image translation using conditional GANs with a U-Net generator and a PatchGAN discriminator. However, the results of pix2pix are limited to 256 x 256 resolution. Cascaded Refinement Networks (CRN) use a multi-resolution refinement approach to synthesize images at 2048 x 1024 resolution, but the images lack fine details and the method does not use an adversarial loss. Our approach combines the benefits of adversarial training with multi-scale architectures to produce high-resolution results with rich details.

Our generator is decomposed into two sub-networks: a global generator network G1 and a local enhancer network G2. The global generator operates at 1024 x 512 resolution and consists of a convolutional front-end, a set of residual blocks, and a transposed convolutional back-end. The local enhancer network has a similar architecture but operates at the full 2048 x 1024 resolution. Crucially, the feature map output from the last layer of G1 is fed as input to G2, so the local enhancer builds upon the global structure established by G1 and adds local details. During training, we first train G1 at lower resolution, then fix G1 and train G2, and finally fine-tune both jointly.

High-resolution image synthesis demands a discriminator with a large receptive field. Rather than using a deeper network (which increases capacity but may cause overfitting and training instability), we use 3 discriminators D1, D2, D3 that have identical network structures but operate at different image scales. Specifically, D1 operates on the original scale, D2 at half resolution, and D3 at quarter resolution. The discriminators at coarser scales have a larger effective receptive field and can guide the generator to produce globally consistent structures, while the finest-scale discriminator encourages fine-grained details.

The full objective combines the GAN loss with a feature matching loss. The feature matching loss is based on the discriminator: we extract features from multiple layers of each discriminator and match intermediate representations between real and synthesized images. This stabilizes training by providing additional supervision beyond the binary real/fake signal. The total loss is: min_G ((max_{D1,D2,D3} sum_{k=1,2,3} L_GAN(G,D_k)) + lambda * sum_{k=1,2,3} L_FM(G,D_k)), where lambda controls the relative importance of the feature matching loss.

We evaluate our method on Cityscapes (2048 x 1024), NYU Indoor RGBD, ADE20K, and Helen Face datasets. For quantitative evaluation, we use semantic segmentation accuracy as a proxy metric: we run a pre-trained segmentation model on the synthesized images and measure how well the predicted labels match the input. On Cityscapes, our method achieves pixel accuracy of 83.78% and mean IoU of 0.6389, significantly outperforming pix2pix (78.34% / 0.3948) and CRN (70.55% / 0.3483). In human perceptual studies, our results are preferred over pix2pix 93.8% of the time and over CRN 86.2% of the time.

Ablation studies reveal the contribution of each component. Using GAN loss only achieves 58.90% preference versus our full method; adding feature matching raises this to 68.55%. The instance boundary map provides significant improvement, with 64.34% preference for realism when included. Architecturally, our coarse-to-fine generator achieves 83.78% pixel accuracy, outperforming both U-Net (77.86%) and CRN encoder (78.96%). The multi-scale discriminator improves mean IoU from 0.5775 (single-scale) to 0.6389 (three-scale). The system runs at 20-30 milliseconds per 2048 x 1024 image on an NVIDIA 1080Ti GPU.

We have presented pix2pixHD, a conditional GAN framework for synthesizing high-resolution photorealistic images from semantic label maps. Our key contributions include a coarse-to-fine generator, a multi-scale discriminator architecture, and a feature matching loss that together enable 2048 x 1024 resolution synthesis with significantly improved quality. We further demonstrate interactive semantic manipulation at the instance level, including object removal, insertion, and appearance transfer. Our results suggest that high-resolution conditional image synthesis is a viable foundation for interactive content creation tools.