With recent progress in graphics, it has become more tractable to train models on synthetic images, potentially avoiding the need for expensive annotations. However, learning from synthetic images may not achieve the desired performance due to a gap between synthetic and real image distributions. To reduce this gap, we propose Simulated+Unsupervised (S+U) learning, where the task is to learn a model to improve the realism of a simulator's output using unlabeled real data, while preserving the annotation information from the simulator. We develop a method for S+U learning that uses an adversarial network similar to Generative Adversarial Networks (GANs), but with synthetic images as inputs instead of random vectors. We make several key modifications including a 'self-regularization' term, a local adversarial loss, and updating the discriminator using a history of refined images.

Large labeled training datasets are becoming increasingly important with the recent rise in high capacity deep neural networks. However, labeling such large datasets is expensive and time-consuming. An alternative to labeling real images is to use synthetic data from simulators, which can automatically generate annotations. While this approach is appealing, learning from synthetic images can be problematic due to a gap between synthetic and real image distributions. The goal of this work is to improve the realism of synthetic images from a simulator using unlabeled real data. The refined images should add realism, preserve annotation information, and be generated without artifacts.

We introduce SimGAN, which refines synthetic images from a simulator using a neural network which we call the 'refiner network'. The key idea is to use a combination of an adversarial loss that fools a discriminator network into classifying refined images as real, together with a self-regularization loss that penalizes large changes between the synthetic and refined images. This combination ensures that the refined images look realistic while retaining the annotation information. Our key contributions include: (i) proposing the S+U learning methodology; (ii) training refiner networks using combined losses; (iii) implementing modifications to stabilize GAN training; and (iv) demonstrating improvements through qualitative, quantitative, and user study experiments.

The GAN framework learns two networks (a generator and a discriminator) with competing losses. Various extensions have been proposed including CoGAN, InfoGAN, and style transfer applications. Regarding synthetic data usage, applications span gaze estimation, text detection, font recognition, object detection, hand pose estimation, scene recognition, semantic segmentation, and human pose estimation. Our approach differs from domain adaptation methods in that it bridges the gap between image distributions through adversarial training at the pixel level, rather than adapting features for specific tasks. This makes it task-independent — the refined images can be used for any downstream task.

The refiner network R_theta should produce outputs that look like real images in appearance while preserving the annotation information. The discriminator D_phi minimizes cross-entropy loss distinguishing real from refined images. The refiner loss combines an adversarial component that fools D into classifying refined images as real, with a self-regularization loss: lambda * ||psi(R_theta(x_i)) - psi(x_i)||_1, where psi is a feature transform (identity in the simplest case). The refiner is a fully convolutional neural network without striding or pooling, modifying the synthetic image on a pixel level while preserving its global structure. This design ensures the spatial resolution remains unchanged and the refined image stays close to the original.

Rather than using a global discriminator that classifies the entire image as real or fake, we implement a discriminator network that classifies all local image patches separately, outputting a w x h dimensional probability map of patches belonging to the fake class. This local adversarial loss encourages the refiner to model the local statistics of real images rather than over-emphasize global structure, which can lead to obvious artifacts like unnatural depth boundaries when using a global discriminator. The local approach also provides more training signal per image, as each patch contributes independently to the loss.

We introduce a method for improving the stability of adversarial training by updating the discriminator using a history of refined images rather than only the ones in the current mini-batch. We maintain a buffer of previously generated refined images. During each discriminator update, we sample b/2 images from the current refiner network, and sample an additional b/2 images from the buffer, then update the buffer by randomly replacing some entries. This prevents the discriminator from overfitting to the most recent refiner output and provides a more stable training signal that smooths out the adversarial dynamics.

We evaluate SimGAN on two tasks: gaze estimation and hand pose estimation. For gaze estimation, training on refined images achieved a 22.3% absolute percentage improvement over synthetic baselines, and state-of-the-art results on the MPIIGaze dataset with a relative improvement of 21%. A user study found that subjects chose the correct label (real vs. refined) only 517 times out of 1000 trials (p=0.148), meaning they were not able to reliably distinguish real images from refined synthetic ones. For hand pose estimation, training on refined synthetic data outperforms the model trained on real images with supervision by 8.8%. Ablation studies confirmed that the history buffer prevents severe artifacts, and the local adversarial loss is superior to global approaches, removing obvious unrealistic depth boundary artifacts.

We proposed Simulated+Unsupervised learning to add realism to the simulator while preserving the annotations of the synthetic images. Using a combination of adversarial loss and self-regularization loss, along with local adversarial training and a history buffer, our method demonstrates state-of-the-art results without any labeled real data. The refined images are visually indistinguishable from real images by human observers. Future work includes exploring modeling the noise distribution to generate more than one refined image for each synthetic image, and investigating refining videos for temporal consistency.