In this work, we revisit the global average pooling layer proposed in Network in Network, and shed light on how it explicitly enables the convolutional neural network to have remarkable localization ability despite being trained on image-level labels. While this technique was previously proposed as a means for regularizing training, we find that it actually builds a generic localizable deep representation that can be applied to a variety of tasks. Despite the simplicity of our approach, we achieve 37.1% top-5 error for weakly-supervised object localization on ILSVRC 2014, which is remarkably close to the 34.2% top-5 error achieved by fully supervised methods. We demonstrate that our network is able to localize the discriminative regions used for classification across different tasks.

A remarkable property of convolutional neural networks is that, despite being trained on image-level classification labels without any bounding box annotations, the convolutional layers still retain substantial spatial information about objects. This information is typically lost when fully-connected layers are used for classification. Recent works such as Network in Network (NIN) and GoogLeNet have replaced fully-connected layers with global average pooling (GAP), originally motivated as a structural regularizer to prevent overfitting. We show that with a little tweaking, the network can retain its remarkable localization ability until the final layer, enabling single forward-pass localization across diverse tasks without explicit localization training.

Weakly-supervised object localization has been approached through various methods, including multiple instance learning (MIL) and attention-based mechanisms. Oquab et al. proposed using global max pooling (GMP) for localization, but this only captures the single most discriminative point rather than the full extent of the object. In contrast, "average pooling encourages the network to identify the complete extent of the object" because it aggregates information from all spatial locations, incentivizing the network to activate broadly over the object region. For CNN visualization, prior methods such as backpropagation-based saliency maps and deconvolution provide insights into network behavior, but they are computationally expensive and do not directly relate to the classification decision.

The Class Activation Map (CAM) is derived as follows. For a given image, let f_k(x, y) represent the activation of unit k in the last convolutional layer at spatial location (x, y). For a given class c, the class score before softmax is S_c = sum_k w^c_k * F_k, where F_k is the global average pool of f_k and w^c_k is the weight connecting the k-th feature map to class c. The class activation map is then simply: M_c(x, y) = sum_k w^c_k * f_k(x, y). This map "directly indicates the importance of the activation at spatial grid (x, y) leading to the classification of an image to class c".

The class activation map can be upsampled to the input image resolution to produce a heatmap highlighting the regions most relevant to the predicted class. By thresholding the heatmap and computing the bounding box of the largest connected component, we obtain an object localization without any bounding box supervision. This localization emerges as a natural byproduct of classification training with GAP. Crucially, the same network can generate different activation maps for different classes, revealing what regions the network focuses on for each category. This provides an interpretable visualization of the CNN decision-making process.

On ILSVRC 2014 weakly-supervised localization, GoogLeNet-GAP achieves 43% top-5 localization error, significantly outperforming global max pooling variants and backpropagation-based approaches. For fine-grained recognition on CUB-200 birds, using CAM to automatically crop bounding boxes achieves 67.8% accuracy, approaching fully-supervised crop methods. The method also demonstrates strong pattern discovery capabilities: when applied to scene recognition, CAM highlights relevant objects (e.g., beds in bedrooms, bookshelves in libraries) that contribute to scene classification. Furthermore, the technique enables informative error diagnosis: when the classifier makes mistakes, CAM reveals what the network was "looking at," providing actionable insights for model improvement.

We have shown that Class Activation Mapping provides a simple yet powerful way to leverage global average pooling for discriminative localization. Despite the simplicity of this technique, it is remarkably effective and can be applied to a variety of computer vision tasks for fast and accurate localization. The ability to generate class-specific heatmaps opens avenues for understanding and interpreting CNN decisions, bridging the gap between high classification performance and model transparency.