Recent deep learning based approaches show promising results for image inpainting of large missing regions. However, existing methods often generate distorted structures or blurry textures inconsistent with surrounding areas. This paper proposes a new deep generative model-based approach which can not only synthesize novel image structures but also explicitly utilise surrounding image features as references during network training to make better predictions. The model is a feed-forward, fully convolutional neural network which can process images with multiple holes at arbitrary locations and with variable sizes. Experiments on multiple datasets including faces (CelebA, CelebA-HQ), textures (DTD), and natural images (ImageNet, Places2) demonstrate superior results.

Image inpainting — filling in missing pixels — has applications in photo editing, image-based rendering and computational photography. Early patch-based methods (such as PatchMatch) work well for stationary backgrounds but cannot hallucinate novel contents for complex, non-repetitive structures like faces. Recent deep CNN and GAN approaches formulate inpainting as conditional image generation but often create boundary artifacts, distorted structures and blurry textures.

The authors identify that CNNs struggle with long-range correlations between distant contextual information and the hole regions. Standard convolutional layers have limited receptive fields and process all spatial locations uniformly, without explicitly borrowing information from known regions. The key contribution is a novel contextual attention layer that explicitly attends to and borrows feature patches from distant spatial locations, using surrounding patches as convolutional filters to match and reconstruct missing content.

Traditional diffusion and patch-based methods use variational algorithms or patch similarity to propagate information from background regions to holes. PatchMatch accelerated this with efficient nearest-neighbour search. Deep learning methods like Context Encoders first applied deep networks to inpainting large holes. Iizuka et al. improved results using global and local discriminators plus dilated convolutions. In the realm of attention modelling, Spatial Transformer Networks predict affine transformations, appearance flow predicts offset vectors, and deformable convolutional networks learn spatially attentive kernels. The proposed contextual attention differs by explicitly matching patches rather than learning deformations.

The network adopts a coarse-to-fine architecture: the first stage produces a rough completion using a dilated convolutional network with reconstruction loss; the second stage refines results with both reconstruction and adversarial losses. Key design improvements include a thin/deep architecture with fewer parameters, use of ELU activations instead of ReLU, and removal of batch normalisation. Training employs Wasserstein GAN with gradient penalty (WGAN-GP) rather than DCGAN, achieving better stability. A novel spatially discounted reconstruction loss weights pixels as gamma^l where l is the distance to the nearest known pixel, reducing ambiguity in the hole centre and achieving 100x training speedup on Places2.

The contextual attention layer operates via a match-and-attend mechanism. First, 3x3 patches are extracted from background regions and used as convolutional filters. Foreground-background similarity is measured using normalised inner product (cosine similarity). A scaled softmax across the background dimension produces attention scores, and the background patches serve as deconvolutional filters for reconstruction. An attention propagation step enforces spatial coherency through left-right then top-down convolution. The unified network has two parallel encoders after the coarse stage: one for content generation via dilated convolution, another for contextual attention, with outputs merged into a single decoder.

Experiments are conducted on Places2, CelebA, CelebA-HQ, DTD textures, and ImageNet. Qualitative results show that contextual attention generates more realistic results with much less artifacts than the baseline. Attention map visualisations reveal adaptive information borrowing from semantically relevant surrounding areas. Quantitative results on Places2 show improvements in l1 loss (8.6% vs. 9.4% baseline) and PSNR (18.91 vs. 18.15). The model has only 2.9M parameters — half of prior work — and runs at 0.2 seconds per 512x512 frame. Ablation studies confirm that WGAN-GP outperforms DCGAN and LSGAN, reconstruction loss is essential, and contextual attention outperforms Spatial Transformer Networks and appearance flow.

The coarse-to-fine framework with the contextual attention module significantly improves image inpainting results by learning feature representations for explicitly matching and attending to relevant background patches. The approach bridges the gap between traditional patch-based methods and modern deep generative models, combining the explicit spatial correspondence of the former with the learning capacity of the latter. Future work includes progressive growing for very high-resolution applications and extensions to image editing and super-resolution.