Convolutional neural networks (CNNs) are inherently limited to model geometric transformations due to the fixed geometric structures in their building modules. In this work, we introduce two new modules to enhance the transformation modeling capability of CNNs, namely deformable convolution and deformable RoI pooling. Both are based on the idea of augmenting the spatial sampling locations in the modules with additional offsets and learning the offsets from the target tasks, without additional supervision. The new modules can readily replace their plain counterparts in existing CNNs and can be easily trained end-to-end by standard backpropagation. Extensive experiments validate the effectiveness of our approach on object detection and semantic segmentation tasks.

A key challenge in visual recognition is how to accommodate geometric variations or transformations in object scale, pose, viewpoint, and part deformation. There are generally two approaches: building training datasets with sufficient variation (data augmentation), or using transformation-invariant features and algorithms. Both are limited: data augmentation is costly and cannot cover all transformations, while hand-crafted invariances may not generalize. The authors argue that the geometric transformation modeling should be internal and adaptive, learned from the data rather than fixed by design.

CNNs are built on fixed-structure modules: convolutions sample at fixed grid locations, pooling reduces over fixed spatial bins, and RoI pooling separates features into fixed spatial partitions. This fixed structure limits their capacity to handle unknown geometric transformations. The authors propose to make these structures adaptive by learning spatial offsets. Deformable convolution adds 2D offsets to each sampling position of the regular grid, enabling free-form deformation of the sampling pattern. Deformable RoI pooling adds offsets to each bin position, enabling adaptive part localization. Both modules are lightweight, adding minimal parameters and computation overhead.

The work is distinguished from several related approaches. Unlike Spatial Transformer Networks (STN), which perform global parametric warping, deformable convolution performs local, dense, per-position sampling. Unlike Active Convolution, which learns static offsets shared across all locations, deformable convolution uses dynamic, input-dependent offsets that vary for each spatial position. The approach generalizes atrous (dilated) convolution as a special case where offsets are fixed integers. Compared to Deformable Part Models (DPM), the proposed method is simpler, more end-to-end trainable, and integrated into modern deep architectures.

In standard convolution, the sampling grid R defines a fixed set of offsets (e.g., R = {(-1,-1), (-1,0), ..., (1,1)} for a 3x3 kernel). Deformable convolution augments each position p_n in R with a learnable offset Delta_p_n, so the output becomes: y(p_0) = sum_{p_n in R} w(p_n) * x(p_0 + p_n + Delta_p_n). Since the offsets are typically fractional, bilinear interpolation is used: x(p) = sum_q G(q, p) * x(q), where G is a bilinear interpolation kernel. The offsets are learned through a separate convolutional layer applied to the input feature map, producing 2N offset values (x and y for N sampling positions). This is fully differentiable and trained end-to-end.

Deformable RoI pooling extends the same idea to region-of-interest pooling. In standard RoI pooling, an input region is divided into k x k fixed bins. Deformable RoI pooling adds offsets to the center of each bin, enabling the network to adaptively focus on the most discriminative parts of an object. The offsets are computed by a small FC network that takes the standard pooled features as input and outputs 2k^2 normalized offsets. This also extends to position-sensitive (PS) RoI pooling, where offsets enable adaptive part localization beyond the fixed grid partition. Both variants show that learned offsets correlate with object scale and shape, confirming the adaptive receptive field behavior.

Experiments integrate deformable modules into ResNet-101 with Faster R-CNN, R-FCN, and DeepLab. On COCO object detection, deformable convolution achieves 11-13% relative improvement across methods. On PASCAL VOC semantic segmentation, significant mIoU gains are observed. The computational overhead is minimal: adding deformable layers to 1-6 convolutional layers adds negligible parameters and runtime. Ablation studies confirm that: (1) more deformable layers yield better results up to a saturation point; (2) the improvements are particularly strong for large and deformable object categories; (3) the learned offsets visually correspond to object structure, expanding for large objects and contracting for small ones.

This paper presents the first demonstration that learning dense spatial transformations in deep CNNs is effective for sophisticated vision tasks. Deformable convolution and deformable RoI pooling are simple, lightweight modules that augment existing networks with adaptive geometric transformation modeling. The learned offsets show meaningful correspondence with object geometry, and the approach achieves consistent improvements across detection and segmentation frameworks with minimal overhead. The work opens a new direction for building transformation-adaptive neural network components.