Recognition algorithms based on convolutional networks (CNNs) typically use the output of the last layer as a feature representation. However, the information in this layer may be too coarse spatially to allow precise localization. On the other hand, earlier layers may be precise in localization but will not capture semantics. To get the best of both worlds, we define the "hypercolumn" at a pixel as the vector of activations of all CNN units above that pixel. Using hypercolumns as pixel descriptors, we show results on three fine-grained localization tasks: simultaneous detection and segmentation, where we improve state-of-the-art from 49.7 mean AP^r to 60.0, keypoint localization, where we get a 3.3 point boost, and part labeling, where we get a 6.6 point gain over a strong baseline.

Convolutional neural networks have led to impressive performance on a range of recognition tasks. For tasks such as image classification, the final layer of a CNN provides a compact feature vector that captures high-level semantic information. However, when we turn to fine-grained localization tasks — such as object segmentation, keypoint prediction, and part labeling — the final layer's spatial resolution is too coarse. A typical CNN may reduce spatial resolution by a factor of 16 or more through pooling and striding, making it fundamentally difficult to produce pixel-precise outputs from the top layer alone.

A natural observation is that information at different resolutions lives in different layers of the CNN. Early layers detect edges and corners at high spatial resolution, while later layers capture complex patterns and category-level semantics at low spatial resolution. We propose to leverage this multi-scale hierarchy by defining the hypercolumn at a pixel as the vector of all CNN unit activations above that pixel. This hypercolumn representation naturally combines fine spatial details from lower layers with rich semantic information from higher layers, enabling precise and semantically meaningful pixel-level predictions.

Multi-scale feature representations have a long history in computer vision. Classical approaches such as image pyramids and SIFT descriptors capture information at multiple scales. In the deep learning era, Fully Convolutional Networks (FCN) by Long et al. also combine predictions from multiple layers, but through additive skip connections rather than concatenation. Our hypercolumn approach differs in that we concatenate features from all layers into a single high-dimensional descriptor for each pixel, preserving the distinct information each layer provides. Concurrent work on semantic segmentation has explored similar multi-scale ideas, but typically for dense prediction tasks with limited application to instance-level problems like simultaneous detection and segmentation.

Given an input image, we pass it through a CNN and extract activations from selected layers. For a target pixel location, the hypercolumn is defined as the vector of activations of all units that lie above that pixel. Since different layers have different spatial resolutions, we resize all feature maps to a common resolution of 50 x 50 using bilinear interpolation, then concatenate the features from selected layers — specifically, pool2, conv4, and fc7 — into a single vector for each pixel location. This concatenated vector serves as a rich, multi-scale descriptor that encodes both low-level spatial details and high-level semantic information.

A naive approach would train 2,500 separate classifiers for each location in the 50 x 50 grid, which is computationally prohibitive and prone to overfitting. Instead, we train classifiers on a coarse K x K grid (e.g., K = 5 or K = 10) and interpolate classifier outputs using bilinear interpolation. Crucially, this interpolation operates on the classifier weight functions rather than the output values, enabling efficient computation. By further decomposing the feature vector into blocks corresponding to each layer, we apply the linear classifier independently to each block before upsampling, converting expensive dense operations into efficient convolutions followed by interpolation.

We evaluate hypercolumns on three fine-grained localization tasks using PASCAL VOC benchmarks. For simultaneous detection and segmentation (SDS) on PASCAL VOC 2012, our System 1 achieves 52.8 mean AP^r at 0.5 overlap and 33.7 at 0.7 overlap. Our improved System 2 with the O-Net architecture reaches 60.0 mean AP^r at 0.5 overlap and 40.4 at 0.7 overlap, substantially surpassing the previous state-of-the-art of 49.7. For keypoint prediction on PASCAL VOC 2009, we achieve a mean APK of 18.5, a 3.3 point improvement over prior work. For part labeling, we observe substantial improvements across seven object categories with an average gain of 6.6 points.

Ablation studies reveal that all three selected layers (pool2, conv4, fc7) contribute meaningfully to performance. Removing any single layer degrades results, confirming that the multi-layer fusion is not redundant. The grid resolution analysis shows that K = 5 grids recover full performance, while K = 1 grids (location-independent classifiers) lose 2.4 points, validating the importance of spatial awareness in the classifier. Across all three tasks, hypercolumns significantly outperform top-layer-only baselines, confirming that early-layer features carry crucial information for localization.

We have presented hypercolumns — a simple yet effective pixel-level feature representation that aggregates information across all layers of a convolutional network. By treating the activations above each pixel as a rich descriptor, hypercolumns bridge the gap between high-level semantics and low-level spatial precision. Our experiments across three diverse fine-grained localization tasks demonstrate consistent and significant improvements over approaches that rely solely on top-layer features. The hypercolumn concept is general and can be applied to any CNN architecture, suggesting it as a fundamental building block for pixel-level recognition tasks that require both semantic understanding and precise spatial reasoning.