Both convolutional and recurrent operations are building blocks that process one local neighborhood at a time. In this paper, we present non-local operations as a generic family of building blocks for capturing long-range dependencies. Inspired by the classical non-local means method in computer vision, our non-local operation computes the response at a position as a weighted sum of the features at all positions. This building block can be plugged into many computer vision architectures. On the task of video classification, non-local models achieve competitive results on Kinetics and Charades datasets. In addition, non-local models are shown to improve object detection and pose estimation on COCO.

Capturing long-range dependencies is of central importance in deep neural networks. For sequential data (e.g., speech, language), the dominant approach is recurrent operations, which process elements sequentially and propagate information through hidden states. For image data, long-range dependencies are modeled by stacking many convolutional layers, gradually increasing the receptive field. Both approaches have fundamental limitations: recurrent operations are sequential and thus computationally inefficient, and convolutional stacking requires many layers to cover large distances, making optimization difficult and multi-hop dependencies hard to capture.

We propose non-local operations as an efficient, simple, and generic solution. A non-local operation computes the response at a position as a weighted sum of features at all positions in the input feature maps. The set of positions can be in space, time, or spacetime, making the formulation applicable to image, sequence, and video problems alike. Our non-local operations are related to self-attention mechanisms recently explored in machine translation, and we show that the embedded Gaussian instantiation of our non-local operation is equivalent to the self-attention form used in Transformers.

The classical non-local means algorithm computes a filtered output as a weighted average of all pixels, where weights depend on patch similarity. This inspired bilateral filtering and other non-local denoising methods. In the deep learning era, self-attention in NLP computes attention weights across all positions in a sequence. Relation networks model interactions between entities. Our work bridges these traditions by formulating non-local operations as a generic neural network building block applicable beyond any single domain.

A generic non-local operation is defined as: y_i = (1/C(x)) * sum_j f(x_i, x_j) * g(x_j), where i is the index of an output position, j enumerates all possible positions, f computes a scalar pairwise affinity between positions i and j, g computes a representation of the input at position j, and C(x) is a normalization factor. The key property is that the operation considers all positions (j) regardless of their distance from position i, in contrast to convolution which only operates on a local neighborhood.

The paper explores several instantiations of the pairwise function f: (1) Gaussian: f(x_i, x_j) = exp(x_i^T * x_j), corresponding to a softmax-based similarity; (2) Embedded Gaussian: f(x_i, x_j) = exp(theta(x_i)^T * phi(x_j)), where theta and phi are learned embeddings — this form is equivalent to the self-attention mechanism in Transformers; (3) Dot product: f(x_i, x_j) = theta(x_i)^T * phi(x_j). Surprisingly, all three instantiations achieve comparable results, suggesting that the generic non-local behavior is the key factor rather than the specific choice of f.

The non-local operation is wrapped into a non-local block that can be inserted into existing architectures: z_i = W_z * y_i + x_i, where the residual connection (+x_i) allows inserting the block into pre-trained models without breaking initial behavior. The implementation uses a bottleneck design (halving the number of channels) and spatial subsampling of the pairwise computation for efficiency. This design makes the block model-agnostic: it can be plugged into ConvNets, recurrent networks, or hybrid architectures with minimal modification.

Video classification: On Kinetics, a non-local ResNet-101 I3D model achieves 77.7% top-1 accuracy, compared to 73.1% for the baseline ResNet-101 — an improvement of +4.6 percentage points with only 1.2x increase in FLOPs. Non-local blocks are shown to be complementary to both 3D convolutions and deeper architectures: adding non-local blocks to a ResNet-50 with 3D conv provides gains on top of the already-improved 3D baseline. On Charades, the non-local model achieves state-of-the-art results.

Object detection and pose estimation: On COCO, adding 1 non-local block to a Mask R-CNN baseline leads to ~1 point increase in AP for both object detection and instance segmentation. For keypoint detection, the improvement is +1.4 AP when non-local blocks are added to both the backbone and the head. These results demonstrate that non-local operations provide benefits beyond video understanding, improving spatial reasoning in static image tasks as well.

We have presented non-local operations as a generic, flexible, and efficient building block for capturing long-range dependencies in deep neural networks. Our approach generalizes the classical non-local means in computer vision to a learnable, neural network module. We have demonstrated that the embedded Gaussian form is equivalent to self-attention, connecting our work to the Transformer architecture. Experiments on video classification, object detection, instance segmentation, and pose estimation consistently show improvements, validating the broad applicability of non-local operations across tasks and modalities.