Existing deep neural networks (DNNs) require a fixed-size (e.g., 224x224) input image. This requirement is "artificial" and may reduce the recognition accuracy for the images or sub-images of an arbitrary size/scale. In this work we equip the networks with another pooling strategy, "spatial pyramid pooling", to eliminate the above requirement. The new network structure, called SPP-net, can generate a fixed-length representation regardless of image size/scale. Pyramid pooling is also robust to object deformations. With these advantages, SPP-net should improve all CNN-based image classification methods in general. SPP-net achieves state-of-the-art accuracy on the datasets of ImageNet 2012, Pascal VOC 2007, and Caltech101.

We note that the requirement of fixed input size comes only from the fully-connected layers that demand fixed-length vectors. On the other hand, the convolutional layers accept inputs of arbitrary sizes and produce outputs proportional to the input size. If we can remove the fixed-size constraint from the fully-connected layers, we can build networks that accept variable-size inputs. This observation is the motivation for our spatial pyramid pooling layer.

Spatial pyramid pooling has a long history in computer vision, originating from Lazebnik et al.'s spatial pyramid matching (SPM) for bag-of-words models. SPM partitions the image into divisions from finer to coarser levels and aggregates local features in each division. We adopt this idea but apply it to deep CNN features instead of traditional descriptors, creating a bridge between traditional computer vision and deep learning.

The spatial pyramid pooling layer is placed on top of the last convolutional layer. It pools the features and generates fixed-length outputs, which are then fed into the fully-connected layers. Specifically, we use a multi-level pooling structure with bins of sizes 1x1, 2x2, 3x3, and 6x6, yielding a fixed concatenated vector of (1+4+9+36) x 256 = 12,800 dimensions for a 256-channel feature map. This fixed-length representation encodes both global and local spatial information.

For image classification, SPP-net provides two key benefits. First, it allows multi-size training: we can train the network on images of different sizes (e.g., 180x180 and 224x224), improving scale invariance. Second, at test time, we can apply the network to the full image without cropping or warping. On ImageNet 2012, multi-size training with SPP-net improves the top-1 error by 0.5% over single-size training, and full-image testing further reduces error by 0.6%.

For object detection, SPP-net brings a dramatic speedup over R-CNN. R-CNN extracts features from each of the ~2,000 candidate regions independently, requiring ~2,000 forward passes through the CNN. In contrast, SPP-net computes the convolutional feature map for the entire image only once, then applies spatial pyramid pooling on each candidate region's sub-feature-map. This shared computation makes SPP-net 24-102x faster than R-CNN at test time. On PASCAL VOC 2007, SPP-net achieves comparable accuracy to R-CNN while being dramatically faster.

We have presented SPP-net, a deep network equipped with spatial pyramid pooling that removes the fixed-size input constraint of conventional CNNs. SPP-net achieves state-of-the-art results in classification and detection while providing significant speedups for detection. The spatial pyramid pooling layer is a flexible and general component that can potentially benefit other CNN architectures and tasks. In ILSVRC 2014, our SPP-net methods ranked #2 in object detection and #3 in image classification among all 38 teams.