Recent results indicate that the generic descriptors extracted from convolutional neural networks are very powerful. This paper adds to the mounting evidence that a simple baseline of using CNN features with a linear classifier can achieve surprisingly strong performance across a wide range of visual recognition tasks. We evaluate OverFeat features extracted from a pre-trained deep CNN on several benchmarks for image classification, scene recognition, fine-grained recognition, attribute detection, and image retrieval. In most cases, these off-the-shelf CNN features surpass highly-tuned, task-specific systems that use hand-crafted features and complex pipelines. Our results suggest that features learned from large-scale image classification transfer effectively to diverse recognition problems, establishing a strong baseline for future research.

The computer vision community has long relied on hand-crafted feature descriptors such as SIFT, HOG, and Fisher Vectors for visual recognition. These features, combined with carefully designed encoding schemes and classifiers, have achieved impressive results on various benchmarks. However, the emergence of deep convolutional neural networks, particularly AlexNet's landmark victory on the ImageNet challenge in 2012, has fundamentally changed the landscape. A key question remains: do these deep features only work well on the specific task they were trained for, or do they generalize to other visual recognition problems?

In this paper, we provide a comprehensive empirical study demonstrating that CNN features extracted from a network pre-trained on ImageNet classification serve as an astounding baseline for a wide range of recognition tasks. Our approach is deliberately simple: we extract activations from a specific layer of the pre-trained OverFeat network, treat them as generic feature vectors, and train a simple linear SVM classifier on top. No fine-tuning of the CNN is performed. This simplicity is intentional — it allows us to isolate the contribution of the learned features from any task-specific adaptation, providing a clean measurement of feature transferability.

We use the OverFeat network, a deep CNN trained on the ImageNet 2012 classification task with 1.2 million training images across 1,000 categories. For feature extraction, we consider activations from different layers of the network: early convolutional layers that capture low-level features such as edges and textures, middle layers that encode mid-level patterns, and the fully-connected layers (fc6 and fc7) that produce high-level semantic representations. Each input image is resized and center-cropped to the network's expected input dimensions. The extracted feature vectors are then L2-normalized and used as input to a linear SVM, with the regularization parameter selected via cross-validation.

To further improve performance, we also explore data augmentation at test time, where we extract features from multiple crops and flips of each test image and average the resulting predictions. Additionally, we investigate the effect of multi-scale feature extraction, where the input image is processed at different resolutions and the resulting features are concatenated. These simple enhancements, combined with the base CNN features, provide a strong yet computationally efficient framework that requires no task-specific neural network training.

We evaluate CNN features on a diverse set of benchmarks. On Caltech-101 (object recognition), the CNN baseline achieves 86.5% accuracy, outperforming Fisher Vector approaches (85.5%) and spatial pyramid pooling methods (83.0%). On MIT Indoor-67 (scene recognition), CNN features reach 58.4% accuracy, surpassing the previous state-of-the-art of 51.4% by a substantial margin. On CUB-200 (fine-grained bird recognition), the CNN baseline achieves 51.7%, competitive with specialized part-based models (56.8%) despite using no part annotations. On Oxford Flowers-102, CNN features achieve 86.8% accuracy, approaching fine-tuned CNN results (87.4%).

For image retrieval on Oxford Buildings and Paris Buildings, CNN features achieve mAP of 55.7% and 67.5% respectively, competitive with Fisher Vector and VLAD-based retrieval systems. For attribute detection on H3D, CNN features yield an average AUC of 87.3%, outperforming hand-crafted attribute classifiers. Across all benchmarks, the fc7 layer features generally provide the best results, suggesting that higher-level semantic representations transfer more effectively than low-level or mid-level features. The consistent strong performance across such diverse tasks provides compelling evidence for the universality of CNN-learned representations.

We analyze which layers produce the most transferable features. Our experiments reveal a clear trend: features from deeper layers (fc6, fc7) consistently outperform those from earlier convolutional layers across all recognition tasks. This suggests that the higher layers learn increasingly abstract and semantically meaningful representations that are less tied to the specific training task. Interestingly, the performance difference between fc6 and fc7 is small (typically less than 1%), indicating that both layers capture similar levels of abstraction. We also observe that fine-tuning the CNN on the target task can further improve results by 2-5% on most benchmarks, but even without fine-tuning, the off-the-shelf features remain remarkably competitive.

We have demonstrated that CNN features extracted from a pre-trained deep network provide an astoundingly strong baseline for a wide variety of visual recognition tasks. Without any task-specific training, these off-the-shelf features match or surpass carefully engineered systems on classification, scene recognition, fine-grained recognition, attribute detection, and image retrieval. Our results provide strong evidence that deep CNN representations learned from large-scale supervised classification are highly transferable and can serve as universal visual descriptors. We recommend that future work in visual recognition report results using CNN features as a baseline, and we anticipate that the gap between off-the-shelf and fine-tuned CNN features will motivate further research into transfer learning and domain adaptation.