We present an unsupervised visual feature learning algorithm driven by context-based pixel prediction. We train a convolutional neural network to generate the contents of an arbitrary image region conditioned on its surroundings. In order to succeed at this task, the model needs to both understand the content of an image, as well as produce a plausible hypothesis for the missing parts. We find that the combination of reconstruction loss and adversarial loss produces results that are both coherent to the context and sharp in appearance. The resulting features prove effective for CNN pre-training for classification, detection, and semantic segmentation, as well as for semantic inpainting applications.

Can convolutional neural networks learn to understand visual structure from context alone? Humans can effortlessly imagine the missing parts of a scene — we know what a room looks like behind a piece of furniture, or what a face looks like behind sunglasses. This suggests that "natural images, despite their diversity, are highly structured" and that this structure can be learned. We propose Context Encoders — an encoder-decoder architecture trained to predict missing image regions from their context. Unlike denoising autoencoders that handle localized, low-level corruption, context encoders must reason about large-scale semantic content to fill in substantial missing regions. Our approach parallels word2vec's use of context in natural language processing, where predicting surrounding words leads to rich semantic representations.

Self-supervised learning methods create proxy tasks from unlabeled data to learn useful representations. Doersch et al. proposed predicting relative positions of image patches — a discriminative task choosing among 8 discrete spatial configurations. In contrast, our approach uses generative prediction, providing a much richer supervisory signal (approximately 15,000 real values per example versus 8 discrete choices). Classical image inpainting methods such as texture synthesis handle small holes by propagating nearby textures, but fail for large missing regions that require semantic understanding. Our method leverages GAN frameworks for context-conditioned generation, combining reconstruction and adversarial losses to produce semantically meaningful and visually sharp completions.

The Context Encoder follows an encoder-decoder pipeline. The encoder, derived from AlexNet, processes 227x227 images through five convolutional layers and pooling to produce a 6x6x256 feature representation. A critical innovation is the "channel-wise fully-connected layer" connecting encoder to decoder: it allows information to directly propagate from one corner of the feature map to another while reducing parameters from over 100 million to m*n^4 compared to standard fully-connected layers. The decoder comprises five up-convolutional layers with ReLU activations, progressively upsampling features toward the original resolution.

The training objective combines two losses. The reconstruction loss (L2) measures the normalized masked L2 distance between the generated and ground-truth pixels: this captures the overall structure but produces blurry results by averaging multiple output modes. The adversarial loss, following the GAN framework, trains a discriminator to distinguish real from generated image regions. Critically, the discriminator remains unconditioned on the mask, as conditioning causes it to exploit perceptual discontinuities at mask boundaries. The joint loss balances reconstruction (lambda_rec = 0.999) and adversarial (lambda_adv = 0.001) components, with the adversarial loss selecting particular modes from the output distribution, producing sharper results.

Three masking strategies are explored to prevent the network from learning trivial low-level features based on mask boundaries. Central region masking uses fixed square masks that work well but learn non-generalizable low-level features. Random block masking employs multiple overlapping masks covering up to 1/4 of images. Random region masking uses arbitrary shapes from the PASCAL VOC dataset, completely removing sharp boundaries. Random region masking produces the most generalizable features, as it forces the network to learn semantic representations rather than relying on edge completion heuristics.

For semantic inpainting on the Paris StreetView dataset, context encoders trained with joint loss produce semantically meaningful completions with mean L1 loss of 9.37% and PSNR of 18.58 dB. The method outperforms nearest-neighbor inpainting and matches Content-Aware Fill on semantic content, though underperforms on pure texture regions. For feature learning transfer: classification on PASCAL VOC 2007 achieves 56.5% mAP, competitive with self-supervised baselines (Doersch et al.: 55.3%) but below ImageNet pre-training (78.2%). Detection via Fast R-CNN achieves 44.5% mAP, outperforming autoencoders (41.9%). Segmentation via FCN achieves 30.0% on PASCAL VOC 2012, significantly outperforming random initialization (19.8%). Training requires only 14 hours on a Titan X GPU compared to weeks for competing methods.

Context Encoders advance both semantic inpainting and self-supervised feature learning. By training networks to predict missing image regions, we demonstrate that "a model needs to both understand the content of an image, as well as produce a plausible hypothesis for the missing parts". The combination of reconstruction and adversarial losses produces visually compelling results, and the learned features transfer effectively to downstream tasks. Our work opens the door for using pixel prediction as a powerful pretext task for unsupervised representation learning.