The authors investigate whether extensive labeled datasets are truly necessary for training effective CNNs. They propose using hundreds of thousands of unlabeled videos from the web to learn visual representations. The core insight leverages visual tracking as a supervisory signal — patches connected by a track likely belong to the same object and should share similar visual representations. A Siamese-triplet network with a ranking loss function is designed to enforce this constraint. Training on 100K unlabeled videos plus VOC 2012 data, their ensemble achieves 52% mAP (without bounding box regression) on VOC 2007 detection, which comes "tantalizingly close" to the ImageNet-supervised baseline of 54.4% mAP.

The remarkable success of deep CNNs in computer vision is heavily dependent on large labeled datasets like ImageNet. Creating such datasets requires millions of human annotations — a process that is expensive, time-consuming, and difficult to scale to new domains. Meanwhile, billions of hours of video are uploaded to the web every day, representing an enormous source of free visual data with inherent temporal structure. The authors argue that temporal coherence in videos provides a natural supervisory signal: objects maintain their identity across frames, so patches tracked across time should yield invariant representations. This idea builds on the slowness principle in neuroscience — that useful visual features should change slowly over time.

Self-supervised learning methods exploit inherent data structure as supervision. Context prediction (Doersch et al.) uses spatial relationships between image patches. Autoencoder-based methods learn representations through reconstruction objectives that may focus on low-level details. For video-based learning, prior work on temporal coherence used simple L2 distance penalties that are difficult to optimize and may collapse representations. Metric learning approaches, particularly Siamese networks, provide a more principled framework for learning similarity-preserving embeddings. The proposed method combines the temporal signal from videos with the discriminative power of triplet-based metric learning.

The architecture is a Siamese-triplet network consisting of three weight-sharing CNN streams. For each training triplet: the anchor patch is a region in frame t, the positive patch is the same region tracked to frame t+k, and the negative patch is a random region from a different video. The network is trained with a ranking loss that enforces: D(anchor, positive) < D(anchor, negative) - margin, where D is the cosine distance in the embedding space. This formulation ensures that tracked patches (same object across time) are embedded closer together than unrelated patches, learning representations that are invariant to the transformations an object undergoes across video frames.

Training data is generated by applying an unsupervised visual tracker (based on IDT — Improved Dense Trajectories) to 100,000 unlabeled videos from YouTube. The tracker produces dense correspondences between patches across frames, providing millions of training triplets automatically. To handle tracker noise, the authors use a temporal gap of 30 frames between anchor and positive, ensuring sufficient visual transformation while maintaining identity. The CNN architecture follows AlexNet/CaffeNet with modifications for the triplet input. Training proceeds with stochastic gradient descent with momentum, and careful hard negative mining is employed to select the most informative negative examples.

On Pascal VOC 2007 object detection, the video-trained features achieve 52% mAP (ensemble, no bounding box regression), compared to 54.4% for ImageNet-supervised pre-training — a gap of only 2.4 percentage points. This result is "tantalizingly close" to supervised performance, suggesting that strong labeled supervision may not be strictly necessary. Single-model performance reaches 44.5% mAP without fine-tuning and 47.4% with fine-tuning. The representations also demonstrate competitive performance on surface normal estimation tasks on the NYU depth dataset, showing generalization to geometric understanding beyond object recognition. Nearest-neighbor retrieval confirms that the learned features capture semantic object similarity rather than low-level visual similarity.

This work demonstrates that visual representations learned from unlabeled videos through temporal coherence can approach the quality of ImageNet-supervised pre-training. The Siamese-triplet architecture with ranking loss over tracked patches provides an effective framework for converting temporal consistency into a discriminative learning signal. The 2.4% mAP gap to supervised baselines suggests that the visual world itself — captured through freely available video — contains sufficient structure for learning powerful representations. Future directions include exploring larger video datasets, better tracking algorithms, and combining temporal and spatial self-supervision signals.