Convolutional neural networks (CNNs) have recently shown outstanding image classification performance on large-scale datasets. The success, however, comes at the expense of requiring millions of parameters and extensive annotated training data. In this paper, we show that image representations learned with CNNs on large-scale annotated datasets can be efficiently transferred to other visual recognition tasks with limited amount of training data. We design a method to reuse layers trained on the ImageNet dataset to compute mid-level image representations for the PASCAL VOC dataset. The transferred representation leads to significantly improved results for object and action classification, outperforming the current state of the art on PASCAL VOC 2007 and 2012 classification benchmarks.

The recent success of deep convolutional neural networks on the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) has demonstrated that learned representations can dramatically outperform hand-designed features like SIFT and HOG. However, training such networks from scratch requires millions of labeled examples and significant computational resources. Many practical visual recognition tasks — such as recognizing actions in still images or classifying objects in specific domains — have far fewer labeled examples available. This raises the fundamental question: can the generic visual features learned on ImageNet transfer to new tasks with different statistics and much less training data? We argue that CNN features, particularly those from mid-level layers, capture generic visual patterns (edges, textures, parts) that are useful across many visual recognition tasks.

Transfer learning has a long history in machine learning, with the core idea of leveraging knowledge from a source domain to improve learning in a target domain. In computer vision, early work focused on transferring hand-crafted features or learned classifiers. The success of AlexNet on ImageNet opened the door to transferring deep learned representations. Concurrent with our work, Donahue et al. (DeCAF) and Razavian et al. also demonstrated the effectiveness of CNN features as generic descriptors. Our work differs in two key ways: we explicitly study which layers transfer best and we propose a multi-scale sliding window approach for object and action localization that does not require region proposals.

Our approach starts with a CNN pre-trained on ImageNet (1.2 million images, 1000 classes) using the AlexNet architecture. The key idea is to remove the last classification layer (trained for 1000 ImageNet categories) and replace it with new adaptation layers for the target task. Specifically, we freeze the weights of the first several convolutional layers (which capture generic low- and mid-level features) and train only the newly added layers on the target dataset. This strategy has two advantages: (1) the frozen layers serve as a powerful generic feature extractor, benefiting from the rich patterns learned from millions of ImageNet images, and (2) training only the top layers requires far fewer labeled examples and less computation, making it feasible for small-scale target datasets.

For tasks requiring spatial localization (e.g., object and action classification where the target may occupy only a portion of the image), we employ a multi-scale sliding window approach. The image is processed at multiple scales, and at each scale the CNN is applied to overlapping fixed-size windows. The final classification score for each image location is the maximum over all scales and spatial positions. Unlike region-based methods such as R-CNN that require an external proposal generator, our approach applies the CNN directly to the image in a dense, exhaustive manner. While computationally more expensive, this avoids potential errors from missed proposals and provides a principled way to handle objects at different scales.

We evaluate on PASCAL VOC 2007 and 2012 for both object classification and action classification. For object classification on VOC 2007, our method achieves 77.7% mAP, surpassing the previous state of the art. On VOC 2012, we achieve 78.7% mAP. For action classification, the transferred features also significantly outperform methods based on hand-crafted features. Importantly, we conduct layer transfer analysis: features from mid-level layers (e.g., fc6, fc7) transfer better than features from the final classification layer, supporting the hypothesis that mid-level representations are more generic. We also show that even with only hundreds of labeled examples per class on the target task, the transferred features provide significant benefits over training from scratch.

We have demonstrated that mid-level image representations learned by CNNs on large-scale datasets can be effectively transferred to new visual recognition tasks with limited training data. Our results confirm that CNN features, particularly from intermediate layers, encode generic visual knowledge that generalizes across datasets and tasks. The transferred features consistently outperform hand-crafted alternatives and significantly reduce the need for task-specific training data. This work contributes to the growing evidence that deep learned representations serve as powerful general-purpose visual features, analogous to the role of SIFT and HOG in the previous decade but with substantially greater discriminative power.