Since convolutional neural networks (CNNs) perform well at learning generalizable image priors from large-scale data, these models have been extensively applied to image restoration and related tasks. Recently, another class of neural architectures, Transformers, have shown significant performance gains on natural language and high-level vision tasks. While the self-attention mechanism in the Transformer model is key to its success, its computation grows quadratically with the spatial resolution, therefore making it infeasible to apply to most image restoration tasks involving high-resolution images.

In this paper, the authors propose an efficient Transformer model, Restormer, that can capture long-range pixel interactions while remaining applicable to large images. The key designs include multi-Dconv head transposed attention (MDTA) that efficiently aggregates local and non-local pixel interactions, and gated-Dconv feed-forward network (GDFN) that performs controlled feature transformation to suppress less informative features. Restormer achieves state-of-the-art results on 16 benchmark datasets across tasks including image deraining, single-image motion deblurring, defocus deblurring, and image denoising.

Image restoration aims to reconstruct a high-quality image from its degraded observation corrupted by noise, blur, rain streaks, and other adverse factors. This is an inherently ill-posed problem as multiple solutions can map to the same degraded input. CNNs have become the preferred approach because they learn generalizable image priors from large-scale data. However, CNNs present two key limitations: restricted receptive fields that prevent long-range dependency modeling, and static filter weights that cannot flexibly adapt to input content.

Self-attention mechanisms, the core component of Transformer models, compute responses as weighted sums across all positions, thus addressing both CNN limitations. Transformers have demonstrated state-of-the-art performance in NLP and high-level vision tasks. However, their complexity grows quadratically with spatial resolution, rendering them infeasible for high-resolution image restoration. Recent Transformer applications in restoration either apply self-attention to small 8x8 spatial windows or divide images into non-overlapping 48x48 patches. These approaches contradict the goal of capturing true long-range relationships, particularly on high-resolution images.

In this work, the authors propose Restormer, an efficient Transformer capable of capturing global connectivity while remaining applicable to large images. The main contributions are threefold: (1) proposing Restormer, an encoder-decoder Transformer enabling multi-scale local-global representation learning on high-resolution images without disintegrating them into patches; (2) introducing multi-Dconv head transposed attention (MDTA) that efficiently aggregates local and non-local pixel interactions by applying self-attention across channels rather than spatial dimensions; and (3) presenting gated-Dconv feed-forward network (GDFN) that performs controlled feature transformation with a gating mechanism to suppress less informative features. The model achieves state-of-the-art performance on 16 benchmark datasets across deraining, deblurring, and denoising.

Data-driven CNN architectures have significantly outperformed conventional restoration approaches. Encoder-decoder U-Net architectures dominate the field due to their hierarchical multi-scale representation and computational efficiency. Techniques such as skip connections, spatial and channel attention modules, and multi-stage progressive designs have further enhanced restoration effectiveness. Notable approaches include MPRNet with multi-stage progressive restoration and MIRNet with multi-scale residual blocks, both demonstrating the value of hierarchical feature processing for capturing details at multiple scales.

Transformers, initially developed for sequence processing in NLP, have been adapted across vision tasks including image recognition, segmentation, and object detection. Vision Transformers (ViT) decompose images into patch sequences, learning mutual relationships and capturing long-range dependencies with strong input adaptability. Despite successful applications to low-level vision problems like super-resolution, colorization, denoising, and deraining, computational complexity remains prohibitive for high-resolution outputs. Recent methods employ strategies reducing complexity, such as applying self-attention within local image regions using Swin Transformer design. However, this restricts context aggregation to local neighborhoods, contradicting the primary motivation for self-attention over convolutions — particularly unsuitable for image restoration requiring global context awareness.

Several Transformer-based restoration methods have been proposed. IPT (Image Processing Transformer) uses a large-scale pre-trained Transformer but requires enormous computational resources with over 115M parameters. SwinIR applies Swin Transformer blocks with shifted window attention and achieves strong results in super-resolution and denoising, yet its local window attention limits the effective receptive field. Uformer introduces a U-shaped Transformer with LeWin attention, applying self-attention within non-overlapping windows. These methods demonstrate the potential of Transformers for restoration, but none achieves truly global self-attention at linear complexity on full-resolution feature maps.

Given a degraded image of dimensions H x W x 3, Restormer first applies a convolution to obtain low-level feature embeddings of dimension H x W x C. These features pass through a 4-level symmetric encoder-decoder, producing deep features of H x W x 2C dimensions. The encoder hierarchically reduces spatial size while expanding channel capacity. The decoder progressively recovers high-resolution representations from low-resolution latent features. Feature downsampling and upsampling use pixel-unshuffle and pixel-shuffle operations respectively. Encoder features concatenate with decoder features via skip connections, followed by 1x1 convolution to reduce channels. A refinement stage at the original resolution further enriches deep features before a final convolution generates the residual image that is added to the degraded input.

Conventional self-attention exhibits O(W²H²) time and memory complexity for W x H pixel images, rendering it infeasible for restoration tasks involving high-resolution imagery. MDTA achieves linear complexity by applying self-attention across channels rather than spatial dimensions, computing cross-covariance across channels to generate attention maps that encode global context implicitly. Instead of the massive HW x HW spatial attention maps in conventional self-attention, MDTA produces compact C x C transposed-attention maps, where C is the number of channels — typically orders of magnitude smaller than HW.

MDTA incorporates depth-wise convolutions to emphasize local context before feature covariance computation. From layer-normalized input tensors, MDTA generates query (Q), key (K), and value (V) projections enriched with local context through two stages: first, 1x1 convolutions aggregate pixel-wise cross-channel context; then, 3x3 depth-wise convolutions encode channel-wise spatial context. The multi-head structure divides channels into separate heads, each learning parallel attention maps independently. This design implicitly models contextualized global relationships between pixels while computing covariance-based attention maps, complementing convolutional strengths within the pipeline.

The computational advantage of MDTA is significant. For an input feature map of spatial dimensions H x W and C channels, conventional self-attention produces attention maps of size HW x HW, leading to O(H²W²) complexity. In contrast, MDTA reshapes Q and K such that the dot-product generates C x C transposed-attention maps, resulting in O(HWC) complexity — linear in the number of pixels. For a typical restoration setting with 256 x 256 input and 48 channels at level-1, this represents a reduction from approximately 4.3 billion operations to 3.1 million — a factor of over 1000x.

Conventional feed-forward networks (FFN) in Transformers operate identically at each pixel location using two 1x1 convolutions: first expanding feature channels by a factor gamma (typically 4), then reducing to original dimensions, with non-linearity applied in hidden layers. GDFN introduces two fundamental modifications: a gating mechanism and the incorporation of depth-wise convolutions. The gating mechanism functions as the element-wise product of two parallel linear transformation paths, one activated with GELU non-linearity. This design controls information flow through hierarchical levels, allowing each level to focus on complementary fine details.

Specifically, the GDFN formulation involves: the input tensor undergoes layer normalization, then splits into two parallel transformation paths. The first path applies point-wise (1x1) convolution followed by 3x3 depth-wise convolution and GELU activation. The second path applies similar point-wise and depth-wise convolutions without activation. These two paths are combined via element-wise multiplication (gating), followed by a final point-wise convolution and residual connection. The depth-wise convolutions encode spatially neighboring pixel information, useful for learning local image structure enabling effective restoration. Since GDFN performs more operations than a standard FFN, the expansion ratio is reduced (gamma = 2.66 instead of 4) to maintain similar parameter count and computational cost.

CNN-based restoration models typically train on fixed-size image patches. However, Transformer models trained on small cropped patches may not encode global image statistics, yielding suboptimal performance on full-resolution test images. Progressive learning addresses this by training networks on smaller patches in early epochs and gradually larger patches in later epochs. This strategy resembles curriculum learning, where networks progress from simpler to more complex tasks requiring fine image structure and texture preservation.

Mixed-size patch training through progressive learning enhances performance at test time across different image resolutions — a common scenario in restoration tasks. Specifically, training begins with 128 x 128 patches and batch size 64, then advances through progressively larger patch-batch pairs: (160², 40), (192², 32), (256², 16), (320², 8), (384², 8) at designated iteration milestones [92K, 156K, 204K, 240K, 276K] across 300K total iterations. As patch sizes increase, batch sizes decrease to maintain consistent computational cost per optimization step.

Image Deraining. Restormer achieves consistent and significant performance gains across five deraining datasets compared to existing approaches. Compared to SPAIR, the previous best method, Restormer advances the state-of-the-art by 1.05 dB averaged across datasets, with individual gains reaching 2.06 dB on Rain100L. Visual results demonstrate that Restormer effectively removes rain streaks while preserving fine structural content that other methods tend to over-smooth.

Single-Image Motion Deblurring. Restormer outperforms all compared approaches on both synthetic (GoPro, HIDE) and real-world (RealBlur-R, RealBlur-J) datasets. Averaged across all datasets, Restormer provides a 0.47 dB boost over MIMO-UNet+ and 0.26 dB over the previous best MPRNet. Crucially, Restormer achieves this with 81% fewer FLOPs than MPRNet. Compared to the Transformer model IPT, Restormer shows 0.4 dB improvement with 4.4x fewer parameters and runs 29x faster. Despite being trained solely on the GoPro dataset, the method demonstrates strong generalization to other benchmarks.

Image Denoising. For Gaussian denoising across synthetic benchmarks and noise levels (sigma = 15, 25, 50), Restormer achieves state-of-the-art under both single-model and noise-level-specific training. At challenging noise level sigma = 50 on Urban100, Restormer achieves 0.37 dB gain over DRUNet (CNN-based) and 0.31 dB over SwinIR (Transformer-based), with 3.14x fewer FLOPs and 13x faster inference than SwinIR. For real image denoising, Restormer is the only method surpassing 40 dB PSNR on both SIDD and DND datasets. On SIDD, it obtains 0.3 dB over MIRNet and 0.25 dB over Uformer.

Ablation Studies. Systematic ablation experiments are conducted on Gaussian color denoising (Urban100, sigma = 50). The MDTA module provides 0.32 dB gain over the baseline, and removing depth-wise convolutions results in PSNR drops, confirming the importance of local context. The gating mechanism in GDFN yields 0.12 dB gain over conventional FFN, with depth-wise convolutions providing additional benefits, totaling 0.26 dB gain. The overall Transformer block improvements amount to 0.51 dB. Progressive learning provides better results than fixed patch training while maintaining similar training time. Under similar parameter and FLOP budgets, deep-narrow models outperform wide-shallow counterparts in accuracy.

Restormer presents an efficient Transformer for high-resolution image restoration through carefully designed architectural innovations. Multi-Dconv head transposed attention (MDTA) models global context by applying self-attention across channels rather than spatial dimensions, achieving linear complexity versus the quadratic complexity of conventional self-attention. Gated-Dconv feed-forward networks (GDFN) introduce gating mechanisms for controlled feature transformation. Both modules incorporate depth-wise convolutions encoding spatially local context, complementing CNN strengths within the Transformer pipeline.

Extensive experiments across 16 benchmark datasets demonstrate state-of-the-art performance for image deraining, single-image motion deblurring, defocus deblurring (single-image and dual-pixel), and image denoising (Gaussian and real-world). The model effectively removes degradations while preserving fine image structure and texture details across diverse restoration tasks. These results confirm that Transformer architectures, when equipped with appropriate computational efficiency mechanisms, represent a powerful paradigm for low-level vision tasks that have long been dominated by convolutional approaches.