Traditional approaches for learning 3D object categories have been predominantly trained and evaluated on synthetic datasets due to the unavailability of real 3D-annotated category-centric data. The main goal of this work is to facilitate advances in this field by collecting real-world data in a magnitude similar to existing synthetic counterparts. The principal contribution is a large-scale dataset called Common Objects in 3D (CO3D), with real multi-view images of object categories annotated with camera poses and ground truth 3D point clouds. The dataset contains 1.5 million frames from nearly 19,000 videos capturing objects from 50 MS-COCO categories. The authors exploit this dataset to conduct one of the first large-scale "in-the-wild" evaluations of several new-view-synthesis and category-centric 3D reconstruction methods. Finally, they contribute NerFormer — a novel neural rendering method that leverages the Transformer to reconstruct an object given a small number of its views.

The vision community's progress on 3D object understanding has been largely driven by synthetic datasets such as ShapeNet and ModelNet. While these enable controlled evaluation, they fail to capture the complexity and diversity of real-world objects: real objects exhibit complex textures, varying illumination, partial occlusions, and diverse backgrounds. Existing real-world 3D datasets are either small in scale (DTU, Tanks and Temples) or limited to specific categories (cars, faces). This creates a significant gap between method development on synthetic data and deployment in real-world applications.

Synthetic 3D datasets like ShapeNet (51,300 models, 55 categories) provide clean 3D meshes but lack photorealism. Objectron provides 3D bounding boxes for real videos but no point clouds or dense geometry. DTU captures controlled scenes with 124 scenes but in laboratory settings only. Tanks and Temples provides 21 large-scale outdoor scenes but focuses on scenes rather than object categories. CO3D differs fundamentally: it provides category-centric, in-the-wild, multi-view coverage of 50 diverse categories at a scale (19,000 sequences) that dwarfs all existing real alternatives.

The CO3D dataset contains 1.5 million frames from nearly 19,000 videos of objects from 50 MS-COCO categories. Each video captures an object from multiple viewpoints through a smooth camera trajectory around the object. The data processing pipeline includes: COLMAP-based SfM to estimate camera poses for each video, Multi-View Stereo (MVS) to generate dense 3D point clouds, and PointRend-based instance segmentation to extract foreground object masks. Quality filtering removes sequences with poor SfM reconstruction or insufficient viewpoint coverage. The resulting annotations include per-frame camera intrinsics and extrinsics, foreground masks, and dense 3D point clouds.

CO3D defines two primary benchmark tasks. The single-sequence new-view synthesis task evaluates how well a method can render novel views after training on a single video sequence of an object. The category-level new-view synthesis task evaluates whether a model trained on many sequences of a category can generalize to unseen instances. This second task is particularly challenging as it requires learning category-level 3D priors from diverse real-world instances. Evaluation metrics include PSNR, SSIM, and LPIPS on held-out views. Baseline methods evaluated include NeRF, IDR, and SRN, providing the first standardized comparison of these methods on real-world, in-the-wild data.

The authors introduce NerFormer, a Transformer-based neural rendering method designed for the few-view category-level reconstruction task. Given a small number of source views (e.g., 5-10) of an unseen object, NerFormer predicts novel views by leveraging cross-attention between target ray features and source image features. The Transformer architecture naturally handles variable numbers of input views and can aggregate information across views through attention. NerFormer is trained on many sequences within a category to learn category-level priors that enable reconstruction from sparse observations.

Experiments reveal significant insights. On single-sequence new-view synthesis, NeRF achieves reasonable quality but struggles with sequences containing sparse viewpoints or complex backgrounds. On category-level synthesis, all methods show substantially lower performance than on synthetic benchmarks, confirming the significant gap between synthetic and real-world evaluation. NerFormer demonstrates competitive performance on the few-view category-level task, particularly when given 5-10 source views. The experiments establish that current methods have significant room for improvement on in-the-wild 3D reconstruction, validating CO3D's role as a challenging and informative benchmark.

CO3D represents a significant step toward real-world evaluation of 3D object understanding methods. With 1.5 million frames, 19,000 sequences, and 50 categories, it is orders of magnitude larger than existing real-world 3D datasets. The benchmark experiments reveal that current methods still have substantial room for improvement on in-the-wild data. NerFormer shows promise for category-level few-view reconstruction using Transformer-based cross-view aggregation. The authors hope CO3D will catalyze progress in 3D understanding by providing a standardized, large-scale real-world benchmark.