Sparse representation-based tracking methods have shown promising results, but a critical challenge remains in how to effectively manage and update the template dictionary during tracking. In this paper, we treat template updating as an online incremental dictionary learning problem and propose a multi-lifespan dictionary learning framework. The key idea is to maintain dictionary atoms with different lifespans — short-lived atoms that capture recent appearance changes, and long-lived atoms that preserve stable features. This strategy balances adaptability to appearance variations with robustness against drift. We develop both generative and discriminative observation models within a Bayesian particle filter framework. Experiments on ten challenging sequences demonstrate that our approach achieves state-of-the-art tracking performance.

Visual object tracking is a fundamental problem with broad applications in surveillance, autonomous driving, human-computer interaction, and activity recognition. A tracker must handle appearance changes due to illumination variation, pose change, partial occlusion, and background clutter. Sparse representation-based methods have gained popularity by representing the target as a sparse linear combination of dictionary templates. However, the effectiveness of these methods hinges on the quality of the template dictionary.

Existing template update strategies fall into two extremes: static dictionaries that never update and fully adaptive dictionaries that update every frame. Static dictionaries fail to capture appearance variations over time, while fully adaptive ones are prone to drift — gradually incorporating tracking errors into the template set. We propose a principled middle ground through multi-lifespan dictionary learning, where dictionary atoms are assigned different lifespans reflecting their temporal relevance and reliability. This enables the tracker to adapt to genuine appearance changes while maintaining a stable reference from the initial frames.

Sparse representation was first applied to tracking by Mei and Ling, who formulated target appearance as a sparse linear combination of template patches with trivial templates for occlusion handling. Incremental visual tracking (IVT) by Ross et al. maintains a PCA subspace representation that is incrementally updated. Online dictionary learning has been explored by Mairal et al. for general tasks but not specifically adapted for tracking with temporal coherence and drift prevention. Our approach combines the representational flexibility of sparse coding with a principled temporal management strategy that has no direct precedent in the tracking literature.

We formulate the template management problem as online dictionary learning. Given a sequence of observed target patches {y_1, y_2, ..., y_t}, we seek a dictionary D = [d_1, ..., d_K] such that each observation can be well approximated by a sparse combination: y_t ≈ D * alpha_t with ||alpha_t||_0 small. The dictionary is updated incrementally using block coordinate descent: at each frame, we fix the dictionary and solve for sparse codes, then update the dictionary atoms to minimize reconstruction error. This avoids the computational cost of retraining from scratch at each frame.

The core innovation is the multi-lifespan management of dictionary atoms. We partition the dictionary into three groups: (1) permanent atoms initialized from the first frame that are never replaced, preserving the original target appearance; (2) long-lifespan atoms that are updated slowly with a large decay factor, capturing gradual appearance changes such as illumination drift; and (3) short-lifespan atoms that are frequently replaced with recent observations, handling rapid appearance changes such as pose variation. The combination of these three temporal scales ensures that the dictionary retains memory of the original target while remaining adaptive to both slow and fast appearance changes. The replacement strategy for short-lived atoms prioritizes atoms with the lowest activation frequency.

We evaluate on ten challenging video sequences widely used in the tracking community, covering scenarios with heavy occlusion (FaceOcc, Girl), illumination change (Car4, David), fast motion (Deer, Jumping), and background clutter (Singer, Football). We compare against eight state-of-the-art trackers including IVT, L1-tracker, MIL, OAB, SemiBoost, FragTrack, TLD, and VTD. Using both center location error (CLE) and overlap rate as metrics, our method achieves the best average performance across all sequences. On the occlusion-heavy sequences, our method shows particularly strong robustness — the permanent atoms prevent catastrophic drift after occlusion recovery. Ablation studies confirm that removing any lifespan group (permanent, long, or short) degrades performance, validating the necessity of the three-level temporal hierarchy.

We have presented a multi-lifespan dictionary learning framework for robust visual object tracking. By partitioning the dictionary into permanent, long-lifespan, and short-lifespan atoms, our method effectively balances the stability-adaptability trade-off that plagues sparse representation-based trackers. The online incremental learning procedure ensures real-time feasibility, and the multi-lifespan strategy provides principled protection against model drift. Experimental results on challenging sequences validate the approach. Future directions include extending the framework to deep feature representations and developing adaptive lifespan allocation that automatically determines the optimal temporal scale for each atom.