Machine learning based approaches to tracking have typically addressed the problem by learning a model of the tracked object's appearance online during test time. This online learning approach limits the expressiveness of the model and requires careful engineering. We propose a tracker that learns to track objects in a purely offline manner, using a deep regression network trained on a large set of videos with tracking annotations. Our tracker, GOTURN (Generic Object Tracking Using Regression Networks), does not perform any online model updating, and thus tracks at 100 frames per second. Despite its simplicity and speed, GOTURN outperforms state-of-the-art online learning approaches on the VOT2014 benchmark. We also show that by training on a large and diverse dataset, the tracker learns to generalize to track novel objects without further training.

Object tracking is one of the fundamental problems in computer vision. Most state-of-the-art trackers learn a model of the object's appearance online, updating the model each frame. This online learning is computationally expensive and limits the tracker's speed. Furthermore, online learning typically uses only the current video as training data, which limits the capacity of the learned model. We take a fundamentally different approach: we train a neural network offline to learn a generic relationship between an object's appearance and its motion. At test time, the network takes as input a crop of the previous frame (centered on the object) and a crop of the current frame (a search region), and directly outputs the bounding box coordinates of the object in the current frame.

GOTURN uses a two-branch CNN architecture. Both branches use the first five convolutional layers of CaffeNet (similar to AlexNet) pre-trained on ImageNet. The previous frame crop (target patch) and current frame crop (search region) are processed by the two branches independently, and their features are concatenated and passed through three fully-connected layers. The output is a 4-dimensional vector representing the bounding box coordinates (top-left and bottom-right corners) of the object in the search region. The entire network is trained end-to-end with an L1 loss on the bounding box coordinates. This direct regression approach eliminates the need for proposal generation or classification-based detection.

We train GOTURN on the ALOV300 video tracking dataset combined with the ImageNet detection dataset. Using ImageNet detection data is crucial: we generate synthetic "tracking" pairs by applying random transformations to detection images, which greatly increases the diversity and volume of training data. The network is trained using stochastic gradient descent with a batch size of 50. A key training detail is the data augmentation strategy of random translation and scale changes to simulate the motion patterns that occur during tracking. Training takes approximately 2 days on a single GPU.

We evaluate GOTURN on VOT2014 and OTB benchmarks. On VOT2014, GOTURN achieves an accuracy of 0.62 and a robustness score of 2.02, outperforming all compared online trackers. The speed advantage is dramatic: GOTURN runs at 100 FPS on GPU, which is at least 5x faster than any competing deep tracker. On the OTB benchmark, GOTURN achieves competitive performance despite no online model updates. The tracker struggles with extreme appearance changes and long-term occlusions, which is expected given the absence of online adaptation. However, for applications requiring real-time performance with good accuracy, GOTURN represents an excellent trade-off.

We have presented GOTURN, a tracker that learns offline to track generic objects through deep regression, achieving 100 FPS with no online updating. By leveraging large-scale offline training data, GOTURN learns a general-purpose motion model that generalizes to novel objects. This work demonstrates that offline learning can be a viable and efficient alternative to online learning for visual tracking.