We consider image transformation problems, where an input image is transformed into an output image. Recent methods for such problems typically train feed-forward convolutional neural networks using a per-pixel loss function between the output and ground-truth images. Parallel work has shown that high-quality images can be generated by defining and optimizing perceptual loss functions based on high-level features extracted from pretrained networks. We combine the benefits of both approaches, and propose the use of perceptual loss functions for training feed-forward networks for image transformation tasks. We show results on two tasks: style transfer, where we achieve comparable performance to the optimization-based method of Gatys et al. but are three orders of magnitude faster, and single-image super-resolution, where we show that replacing a per-pixel loss with a perceptual loss gives visually more pleasing results.

Many classic problems in computer vision can be framed as image transformation tasks, where a system receives some input image and transforms it into an output image. Each of these tasks can be addressed with convolutional neural networks. A common approach is to train a feed-forward CNN using a per-pixel loss function that measures the difference between the output and ground-truth. Although such approaches produce reasonable results, per-pixel losses do not capture perceptual differences between output and ground-truth images. Two images that are perceptually quite different may have a small per-pixel loss, and two images that are perceptually very similar may have a large per-pixel loss. Recently, Gatys et al. showed that neural style transfer can be performed using optimization to find an image that minimizes a perceptual loss; however this optimization process is very slow, requiring hundreds of iterations of forward and backward passes through a pretrained network.

Our system consists of two components: an image transformation network f_W and a loss network phi that is used to define several loss functions. The image transformation network is a deep residual CNN parameterized by weights W; it transforms input images x into output images y = f_W(x). The loss network phi is used to define a feature reconstruction loss l_feat and a style reconstruction loss l_style. For each input image x, we have a content target y_c and a style target y_s. For style transfer, the content target is the input image itself and the style target is a fixed style image. The feature reconstruction loss is the (squared, normalized) Euclidean distance between feature representations: l_feat = ||phi_j(y) - phi_j(y_c)||^2. The style reconstruction loss is based on the Gram matrices of the feature maps.

For style transfer, we train one network per style. Once trained, the network can stylize any input image in a single forward pass, taking approximately 20 milliseconds for a 256x256 image, compared to the tens of seconds required by the optimization-based approach of Gatys et al.. The results are qualitatively comparable to those produced by the optimization approach. Our trained networks successfully capture the texture, color, and compositional patterns of the target style while maintaining the high-level structure of the content image. We use a network architecture based on residual blocks, with two stride-2 convolutions for downsampling, followed by five residual blocks and two fractionally-strided convolutions for upsampling.

For single-image super-resolution, we train networks with upsampling factors of x4 and x8. We compare our perceptual loss approach against standard per-pixel loss (MSE). While per-pixel loss achieves higher PSNR values, the results tend to be blurry and lack fine detail. In contrast, our perceptual loss produces sharper images with more plausible high-frequency detail, even though the PSNR may be slightly lower. This observation aligns with the growing recognition that PSNR and per-pixel metrics do not always correlate well with human perceptual quality. The super-resolution network uses the same architecture as the style transfer network but with different loss functions.

In this work, we have combined the benefits of feed-forward image transformation networks and optimization-based image generation by training feed-forward networks with perceptual loss functions that depend on high-level features from a pretrained loss network. We have demonstrated compelling results on style transfer and single-image super-resolution, showing that perceptual losses produce results that are visually superior to those trained with per-pixel losses alone. Our method enables real-time style transfer and super-resolution, making these techniques practical for interactive applications. We believe that the ideas presented here are applicable to a broad range of image transformation tasks.