We propose a deep learning method for single image super-resolution (SR). Our method directly learns an end-to-end mapping between the low/high-resolution images. The mapping is represented as a deep convolutional neural network (CNN) that takes the low-resolution image as the input and outputs the high-resolution one. We further show that traditional sparse-coding-based SR methods can also be viewed as a deep convolutional network. But unlike traditional methods that handle each component separately, our method jointly optimizes all layers. Our deep CNN achieves superior performance compared with state-of-the-art methods.

Single image super-resolution (SR) aims at recovering a high-resolution image from a single low-resolution image, and is inherently an ill-posed problem since there are multiple high-resolution images that can produce the same low-resolution image after downsampling. Traditional approaches include interpolation-based, reconstruction-based, and learning-based methods. Among them, sparse-coding-based methods have been the most successful, but they involve complex optimization and hand-crafted features.

Our SRCNN consists of three operations: (1) Patch extraction and representation: the first convolutional layer extracts patches from the low-resolution input and represents each patch as a high-dimensional vector; (2) Non-linear mapping: the second layer maps these vectors non-linearly to another set of high-dimensional vectors representing high-resolution patch representations; (3) Reconstruction: the final layer aggregates these representations to form the high-resolution output image. Each operation corresponds to a convolutional layer with ReLU activation.

We establish a theoretical connection between SRCNN and traditional sparse-coding-based SR. The three stages of sparse-coding SR — patch extraction, sparse coding, and reconstruction — correspond exactly to the three layers of our SRCNN. However, SRCNN optimizes all three stages jointly through end-to-end learning, whereas sparse coding treats them as independent modules optimized separately. This joint optimization is the key advantage of our approach.

We evaluate SRCNN on standard benchmarks: Set5, Set14, and BSD200. For upscaling factors of 2x, 3x, and 4x, SRCNN consistently outperforms state-of-the-art methods including Bicubic, Sparse Coding (SC), and Anchored Neighborhood Regression (ANR) in terms of PSNR and SSIM. For example, at 3x upscaling on Set5, SRCNN achieves 32.75 dB PSNR, compared to 31.42 dB for SC and 30.39 dB for Bicubic. The network is also fast, running at real-time speed.

We have proposed SRCNN, a simple yet effective deep learning approach for single image super-resolution. By establishing the connection to sparse coding, we provide a principled understanding of why CNNs work well for SR. SRCNN achieves state-of-the-art performance with fast execution speed. We believe that deeper and more sophisticated network architectures will further push the boundaries of image super-resolution.