Object tracking is one of the most important problems in computer vision, yet there is no established standard benchmark for evaluating tracking algorithms. Existing evaluations are fragmented, using different sequences, metrics, and protocols, making fair comparison difficult. In this paper, we present a large-scale benchmark for online single-object tracking that includes 50 fully annotated sequences with 11 attributes covering major challenges such as illumination variation, scale change, occlusion, deformation, and fast motion. We evaluate 29 tracking algorithms using one-pass evaluation (OPE), temporal robustness evaluation (TRE), and spatial robustness evaluation (SRE), and provide comprehensive analysis of tracker performance under each attribute.

The field of visual object tracking has seen rapid progress over the past decade, with dozens of new algorithms proposed each year. However, this progress is difficult to quantify objectively because researchers often evaluate on their own selected sequences with different evaluation metrics. A paper may report improvements on 5-10 sequences that happen to favor its method, while failing on sequences not shown. This "evaluation crisis" hinders scientific progress and makes it impossible for practitioners to choose the right tracker for their application.

In the object detection community, the PASCAL VOC challenge has served as a standard benchmark since 2005, enabling fair comparison and measurable year-over-year progress. Similarly, the Middlebury benchmark has driven advances in optical flow and stereo. The tracking community lacks an equivalent benchmark. Our goal is to fill this gap by providing a comprehensive, standardized evaluation platform for visual tracking.

Prior tracking evaluations include PETS workshops (focused on surveillance), the VOT challenge (initiated in 2013, concurrent with this work), and various ad hoc evaluations in individual papers. PETS primarily addresses multi-object tracking in fixed-camera scenarios, which is quite different from the generic single-object tracking problem we address. The Visual Tracker Benchmark by Babenko et al. evaluates a small number of trackers on limited sequences without systematic attribute annotation. Our benchmark improves on these efforts by combining large-scale evaluation, detailed per-attribute analysis, and multiple robustness evaluation protocols.

Our benchmark consists of 50 sequences selected to cover a wide variety of tracking challenges. Each sequence is annotated with frame-level bounding boxes and tagged with 11 attributes: illumination variation (IV), scale variation (SV), occlusion (OCC), deformation (DEF), motion blur (MB), fast motion (FM), in-plane rotation (IPR), out-of-plane rotation (OPR), out-of-view (OV), background clutter (BC), and low resolution (LR). Each sequence may have multiple attributes, enabling fine-grained analysis of which challenges cause which trackers to fail.

We propose three complementary evaluation protocols. One-Pass Evaluation (OPE) runs each tracker once from the ground-truth initial frame — this is the standard protocol. Temporal Robustness Evaluation (TRE) initializes trackers from 20 different starting frames throughout each sequence, measuring sensitivity to the temporal starting point. Spatial Robustness Evaluation (SRE) perturbs the initial bounding box with shifts and scale changes, measuring sensitivity to initialization accuracy. Performance is reported via success plots (overlap threshold vs. success rate) and precision plots (location error threshold vs. precision), with area under the curve (AUC) as the primary ranking metric.

We evaluate 29 tracking algorithms spanning diverse approaches: generative methods (IVT, L1APG), discriminative methods (Struck, SCM), sparse representation (ASLA), correlation filters (CSK, MOSSE), and part-based models (DPM-based). Under OPE, Struck achieves the highest overall AUC of 0.474, followed by SCM (0.453) and ASLA (0.434). The attribute-based analysis reveals that no single tracker dominates across all attributes: Struck excels on fast motion and motion blur, while SCM performs best on background clutter and occlusion.

The TRE and SRE evaluations provide additional insights. Under TRE, rankings remain largely consistent with OPE, suggesting that top trackers are robust to temporal starting position. Under SRE, some trackers show significant sensitivity to bounding box perturbation — in particular, generative trackers like IVT degrade more than discriminative trackers when initialization is noisy. We also observe that speed-accuracy trade-offs vary widely: real-time trackers like MOSSE (615 FPS) score much lower than Struck (20 FPS) but are orders of magnitude faster.

We have presented a comprehensive benchmark for online single-object tracking with 50 sequences, 11 attributes, and 29 evaluated trackers. Our findings suggest that the choice of tracker should be guided by the specific challenges of the application scenario, as no single method excels in all situations. We release our benchmark including all sequences, annotations, evaluation code, and tracker results to the community, and hope it will serve as a standard platform for future tracking research.