We present a new large-scale dataset for advancing the state of the art in object recognition by placing the question of recognition in the context of the broader question of scene understanding. This is achieved by gathering images of complex everyday scenes containing common objects in their natural context. Objects are labeled using per-instance segmentations to aid in precise object localization. Our dataset contains photos of 91 object types that would be easily recognizable by a 4 year old. With a total of 2.5 million labeled instances in 328k images, the creation of our dataset drew upon extensive crowd worker involvement.

One of the primary goals of computer vision is the understanding of visual scenes. Scene understanding involves numerous tasks including recognizing what objects are present, localizing the objects in 2D and 3D, determining the objects' and scene's attributes, characterizing relationships between objects and providing a semantic description of the scene. Current object recognition datasets address some of these tasks, but not all, and they do not place object recognition in the context of scene understanding.

We address this by collecting a dataset with the following properties: (1) a large number of object instances per category, (2) a large number of categories, (3) a large number of instances with per-instance segmentation, (4) images showing objects in their natural contexts, and (5) five captions per image. The combination of these properties makes the COCO dataset unique in its breadth and depth of annotation.

Several datasets have been instrumental in advancing object recognition. PASCAL VOC provides high-quality annotations for 20 categories, while ImageNet offers over 14 million images spanning more than 20,000 categories. However, PASCAL VOC has limited categories and ImageNet's annotations are primarily bounding boxes without segmentation masks. SUN provides scene-level annotations but lacks per-instance segmentation. Our COCO dataset bridges these gaps by combining rich per-instance annotations with natural scene contexts.

The COCO dataset collection process involved several stages. First, we selected 91 common object categories that are easily recognizable and have a clear visual identity. Category selection was guided by the principle that objects should be relevant to everyday life and should occur frequently in natural images. We then used Amazon Mechanical Turk (AMT) workers to collect images and annotate them. Each image was annotated with instance-level segmentation masks, bounding boxes, and five textual captions.

A key design choice is our emphasis on non-iconic images. Unlike datasets that primarily contain iconic views of objects (centered, well-lit, unoccluded), COCO images depict objects in natural, often cluttered environments. This makes the recognition task more challenging but also more representative of real-world visual understanding. Statistics show that COCO images contain an average of 7.7 object instances per image, compared to fewer than 3 in PASCAL VOC.

We provide extensive analysis of the dataset properties. COCO contains 328,000 images with 2.5 million labeled instances across 91 categories. The dataset is split into training (83k images), validation (41k images), and test (81k images) sets. We also establish baseline results for detection and segmentation tasks using state-of-the-art methods. The results show that current methods perform significantly worse on COCO than on PASCAL VOC, indicating the increased difficulty and room for improvement.

We have introduced Microsoft COCO, a new large-scale dataset for object detection, segmentation, and captioning. Through its emphasis on non-iconic imagery, rich per-instance annotations, and natural scene contexts, COCO provides a challenging benchmark that pushes the boundaries of current recognition systems. We believe COCO will be an important resource for the computer vision community for years to come.