We present an object detection approach that leverages Fisher vector (FV) representations derived from SIFT and color descriptors. A key contribution is the use of tentative segmentation masks to suppress background clutter in the Fisher vector computation. By reweighting local descriptors according to their likelihood of belonging to the foreground object, we obtain a cleaner, more discriminative representation that significantly improves detection performance. We further incorporate segmentation-based hypothesis generation for efficient candidate window proposal, and add contextual information through full-image Fisher vectors and cross-category rescoring. Experiments on the PASCAL VOC 2007 and 2010 benchmarks show that our method surpasses state-of-the-art detection results.

Object detection in natural images remains a fundamental challenge in computer vision. The dominant paradigm involves extracting local descriptors (e.g., SIFT, HOG) from candidate windows and encoding them into a fixed-dimensional representation for classification. Fisher vectors have emerged as one of the most effective encoding schemes, achieving state-of-the-art results in image classification. However, when applied to object detection, FVs suffer from a critical limitation: the pooling of descriptors within a bounding box inevitably includes background features that dilute the object representation.

Our key insight is that even rough, imperfect segmentation can significantly improve detection when used to reweight the contribution of local descriptors in the Fisher vector computation. Rather than treating all descriptors within a bounding box equally, we assign higher weights to descriptors that are more likely to belong to the foreground based on tentative segmentation hypotheses. This approach is complementary to other detection improvements and can be combined with contextual rescoring and segmentation-based window proposals for further gains.

Fisher vectors were introduced by Perronnin and Dance for image classification, extending the Fisher kernel framework to visual recognition. Chatfield et al. provided comprehensive comparisons showing FVs outperform Bag of Visual Words (BoVW) and VLAD encodings. For object detection, the Deformable Part Model (DPM) by Felzenszwalb et al. remains the dominant approach using HOG features and latent SVMs. Selective Search by Uijlings et al. proposed segmentation-based region proposals as an alternative to exhaustive sliding windows. Our work bridges these lines by bringing the discriminative power of Fisher vectors to detection while using segmentation to focus the representation on the object.

Given a set of local descriptors {x_1, ..., x_T} extracted from a candidate window, we encode them using a Fisher vector with respect to a Gaussian Mixture Model (GMM) with K components. The FV captures the first and second order statistics of how the descriptors deviate from the GMM. Specifically, for each GMM component k, we compute the gradient of the log-likelihood with respect to the mean (mu_k) and variance (sigma_k), yielding a 2KD-dimensional representation where D is the descriptor dimension. We use SIFT descriptors (D=128) and color descriptors, with a GMM of K=256 components, and apply power normalization and L2 normalization.

The core innovation is to reweight local descriptors based on tentative foreground-background segmentation. For each candidate window, we generate multiple segmentation hypotheses using GrabCut initialized with the bounding box. Each segmentation provides a soft foreground probability map p(x_t) for each descriptor location. The weighted Fisher vector is then computed by multiplying each descriptor's contribution by its foreground probability: replacing the uniform 1/T weighting with p(x_t) / sum(p(x_t)). This effectively focuses the Fisher vector on foreground descriptors while suppressing background noise. When multiple segmentation hypotheses are available, we concatenate the weighted FVs from different segmentations, allowing the classifier to select the most useful segmentation.

We evaluate on the PASCAL VOC 2007 and 2010 detection benchmarks using mean average precision (mAP). On VOC 2007, our method achieves 41.7% mAP, surpassing the DPM baseline (33.4%) by a large margin and outperforming other Fisher vector-based detectors. The segmentation-driven reweighting alone improves the baseline FV detector by 3.8 mAP points, confirming the effectiveness of background suppression. Adding contextual rescoring provides an additional 1.5 mAP improvement. On VOC 2010, we achieve 36.8% mAP, again setting a new state of the art. Per-category analysis reveals the largest gains on categories with high intra-class variation and cluttered backgrounds, such as "chair" and "potted plant," where segmentation-based foreground focusing is most beneficial.

We have presented a segmentation-driven approach to object detection that leverages Fisher vector representations enhanced by foreground-background segmentation. By reweighting local descriptors based on their foreground likelihood, we obtain cleaner object representations that lead to significant detection improvements on PASCAL VOC benchmarks. The approach demonstrates that even imperfect segmentation can serve as a powerful tool for improving feature encoding in object detection. Combined with contextual rescoring and segmentation-based proposals, our method achieves state-of-the-art performance. Future work includes exploring learned segmentation models and extending the framework to deep feature representations.