We present a method for detecting objects in images using a single deep neural network. Our approach, named SSD, discretizes the output space of bounding boxes into a set of default boxes over different aspect ratios and scales per feature map location. At prediction time, the network generates scores for the presence of each object category in each default box and produces adjustments to the box to better match the object shape. Additionally, the network combines predictions from multiple feature maps with different resolutions to naturally handle objects of various sizes. SSD is simple relative to methods that require object proposals because it completely eliminates proposal generation and subsequent pixel or feature resampling stages and encapsulates all computation in a single network. This makes SSD easy to train and straightforward to integrate into systems that require a detection component. Experimental results on the PASCAL VOC, COCO, and ILSVRC datasets confirm that SSD has competitive accuracy to methods that utilize an additional object proposal step and is much faster, while providing a unified framework for both training and inference. For 300x300 input, SSD achieves 74.3% mAP on VOC2007 test at 59 FPS on a Nvidia Titan X, and for 512x512 input, SSD achieves 76.9% mAP, outperforming a comparable state-of-the-art Faster R-CNN model.

Current state-of-the-art object detection systems are variants of the following approach: hypothesize bounding boxes, resample pixels or features for each box, and apply a high-quality classifier. This pipeline has prevailed on detection benchmarks since the Selective Search work through the current leading results on PASCAL VOC, COCO, and ILSVRC detection achieved by Faster R-CNN. Although accurate, these approaches are too computationally intensive for embedded systems and, even with high-end hardware, too slow for real-time applications. Often, detection speed for these approaches is measured in frames per second (FPS), and even the fastest high-accuracy detector, Faster R-CNN, operates at only 7 FPS.

There have been many attempts to build faster detectors by attacking each stage of the detection pipeline, but so far, significantly increased speed comes only at the cost of significantly decreased detection accuracy. Our work introduces the first deep network based object detector that does not resample pixels or features for bounding box hypotheses and is as accurate as approaches that do. This results in a significant improvement in speed for high-accuracy detection. The fundamental improvement in speed comes from eliminating bounding box proposals and the subsequent pixel or feature resampling stage.

The SSD approach is based on a feed-forward convolutional network that produces a fixed-size collection of bounding boxes and scores for the presence of object class instances in those boxes, followed by a non-maximum suppression step to produce the final detections. The early network layers are based on a standard architecture used for high quality image classification (truncated before any classification layers), which we call the base network. We then add auxiliary structure to the network to produce detections with the following key features: multi-scale feature maps for detection — we add convolutional feature layers to the end of the truncated base network. These layers decrease in size progressively and allow predictions of detections at multiple scales.

Convolutional predictors for detection — each added feature layer (or optionally an existing feature layer from the base network) can produce a fixed set of detection predictions using a set of convolutional filters. For a feature layer of size m x n with p channels, the basic element for predicting parameters of a potential detection is a 3x3xp small kernel that produces either a score for a category, or a shape offset relative to the default box coordinates. Default boxes and aspect ratios — we associate a set of default bounding boxes with each feature map cell, for multiple feature maps at the top of the network. The default boxes tile the feature map in a convolutional manner, so that the position of each box relative to its corresponding cell is fixed.

The key difference between training SSD and training a typical detector that uses region proposals is that ground truth information needs to be assigned to specific outputs in the fixed set of detector outputs. Once this assignment is determined, the loss function and back propagation are applied end-to-end. Training also involves choosing the set of default boxes and scales for detection as well as the hard negative mining and data augmentation strategies. Matching strategy: during training we need to determine which default boxes correspond to a ground truth detection. We begin by matching each ground truth box to the default box with the best jaccard overlap. Unlike MultiBox, we then match default boxes to any ground truth with jaccard overlap higher than a threshold (0.5).

Hard negative mining: after the matching step, most of the default boxes are negatives, especially when the number of possible default boxes is large. This introduces a significant imbalance between the positive and negative training examples. Instead of using all the negative examples, we sort them using the highest confidence loss for each default box and pick the top ones so that the ratio between the negatives and positives is at most 3:1. We found that this leads to faster optimization and more stable training. Data augmentation is crucial to improving performance. We generate additional training examples by random cropping, horizontal flipping, and photometric distortions.

To understand the performance of our SSD model in detail, we carried out a comprehensive analysis using PASCAL VOC 2007. In all experiments, we use VGG-16 as the base network, pre-trained on ILSVRC CLS-LOC dataset. We compare against Fast R-CNN and Faster R-CNN. Our SSD300 model achieves 74.3% mAP at 59 FPS while SSD512 achieves 76.9% mAP at 22 FPS. For reference, Faster R-CNN achieves 73.2% mAP at 7 FPS. This means SSD300 is 8.4x faster than Faster R-CNN while achieving higher mAP. Our SSD512 model outperforms Faster R-CNN in both accuracy and speed.

We also evaluate SSD on the COCO dataset. We compare against the state-of-the-art Faster R-CNN and ION on various metrics including mAP@0.5 and mAP@[0.5:0.95]. Our SSD300 achieves 23.2% mAP@[0.5:0.95] compared to Faster R-CNN's 21.9%. However, SSD has relatively worse performance on smaller objects compared to larger objects. This is not surprising because those small objects may not even have any information at the very top layers. Increasing the input size (e.g., from 300x300 to 512x512) can help improve detecting small objects.

This paper introduces SSD, a fast single-shot object detector for multiple categories. A key feature of our model is the use of multi-scale convolutional bounding box outputs attached to multiple feature maps at the top of the network. This representation allows us to efficiently model the space of possible object shapes. We demonstrate that given appropriate training strategies, a larger number of carefully chosen default bounding boxes results in improved performance. We build SSD models with at least an order of magnitude more box predictions sampling location, scale, and aspect ratio, than existing methods. We demonstrate that SSD achieves accuracy that is competitive with state-of-the-art object detectors while being significantly faster, making it the ideal candidate for real-time object detection systems.