The problem of arbitrary object tracking has traditionally been tackled by learning a model of the object's appearance exclusively online, using as sole training data the video itself. Despite the success of these methods, their online-only approach inherently limits the richness of the model they can learn. Recently, several attempts have been made to exploit the expressive power of deep convolutional networks. However, when using deep features for tracking, training is typically done using a classification-based approach with stochastic gradient descent, which is too slow for real-time tracking. We equip a basic tracking algorithm with a novel fully-convolutional Siamese network trained end-to-end on the ILSVRC15 dataset for object detection in video. Our tracker operates at frame-rates beyond real-time (86 FPS) and, despite its extreme simplicity, achieves state-of-the-art performance in multiple benchmarks.

We propose to learn a similarity function offline using large datasets, and then apply this function at test time to track arbitrary objects. The core idea is to train a Siamese network that takes two image patches as input — an exemplar (the target template) and a search region — and produces a scalar-valued score that indicates how similar the two patches are. At test time, the exemplar is cropped from the first frame using the ground-truth bounding box, and the search region is extracted from subsequent frames. The position of maximum similarity in the output score map indicates the estimated target location. This approach sidesteps the need for online model updating via SGD, which is the bottleneck of most deep learning based trackers.

A Siamese network applies an identical transformation phi to both inputs and then combines their representations using a function g. In our formulation, the function g is a cross-correlation layer. Given the exemplar image z and search image x, the response map is computed as: f(z, x) = phi(z) * phi(x) + b, where * denotes cross-correlation. The key advantage of our formulation is that it is fully-convolutional with respect to the search image. This means that rather than evaluating the similarity at a single position, we can evaluate it densely across the entire search region in a single forward pass. The embedding function phi is a CNN without any fully-connected layers, using a modified AlexNet architecture with five convolutional layers.

We train our network on the ILSVRC15 video object detection dataset, which contains more than 4000 sequences and over 1.3 million annotated frames. We use pairs of frames from the same video to create training examples. The exemplar is a 127x127 crop centered on the target in the first frame, and the search image is a 255x255 crop from a later frame. We train using logistic loss, where each position in the score map is treated as a binary classification problem (target present or absent). We use stochastic gradient descent with momentum for optimization. The training takes only 2 days on a single GPU, and the resulting model generalizes well to tracking arbitrary objects not seen during training.

We evaluate SiamFC on OTB-2013, OTB-2015, and VOT-2015 benchmarks. On OTB-2013, our tracker achieves an AUC score of 0.608, which is comparable to more complex methods like MDNet (0.648) while running at 86 FPS compared to MDNet's 1 FPS. On VOT-2015, SiamFC ranks among the top performers with an expected average overlap of 0.292. The tracker is remarkably fast: it runs at 86 FPS with 3 scales and 58 FPS with 5 scales on a Titan X GPU. While our tracker does not achieve the absolute best accuracy on all benchmarks, its combination of speed, simplicity, and competitive accuracy makes it an attractive option for applications requiring real-time performance.

We have presented a tracker based on fully-convolutional Siamese networks that achieves state-of-the-art tracking accuracy while operating at frame rates far exceeding real-time requirements. The simplicity of the approach — learning a similarity function offline and applying it at test time without online model updates — makes it easy to implement and integrate. This work establishes that Siamese networks are a powerful paradigm for visual object tracking, and we believe it will inspire further research in combining offline training with online tracking.