We present an approach that combines image retrieval with machine learning for robust face detection. Our method leverages a large database of faces with bounding rectangles and facial landmark locations to train exemplar-based classifiers. Rather than using traditional sliding-window techniques, we employ a voting-based method where these classifiers evaluate test images through image retrieval, with face locations identified by selecting the modes from the voting maps. The framework handles challenging conditions without requiring explicit modeling of facial variations. Beyond detection, we extend the methodology to face validation (to remove false positives) and face alignment/landmark localization. Our exemplar-based approach achieves state-of-the-art performance on benchmark datasets and generalizes to other tasks including attribute recognition and object detection.

Face detection is one of the most well-studied problems in computer vision, with the Viola-Jones detector achieving practical real-time performance over a decade ago. However, detecting faces in the wild — under large pose variations, partial occlusion, extreme lighting, and low resolution — remains challenging. Traditional detectors use sliding-window approaches with hand-crafted features (Haar, HOG, LBP) and cascade classifiers. While effective for frontal faces, these approaches struggle with non-frontal views and require separate models for different pose ranges.

We propose an exemplar-based approach that leverages a large database of annotated face images to implicitly handle all sources of variation. The key idea is to train a set of exemplar-specific linear classifiers, each associated with a face image that has known bounding box and landmark annotations. At test time, these classifiers vote for potential face locations in a Hough-transform-like scheme. This approach offers several advantages: it naturally handles pose variation (different exemplars cover different poses), provides both detection and alignment simultaneously, and can be easily updated by adding new exemplars without retraining.

Exemplar-based methods have gained renewed interest with works on Exemplar-SVM for object detection and scene retrieval approaches. The idea of training one classifier per exemplar allows explicit association between detections and specific training examples, enabling attribute transfer and fine-grained recognition. For face detection specifically, deformable part models (DPM) have shown strong results by modeling faces as collections of parts with spatial constraints. Face alignment methods range from Active Shape Models (ASM) to regression-based approaches. Our work unifies detection and alignment within a single retrieval-based framework, avoiding the need for separate specialized models.

Our face database contains N annotated face images, each with a bounding box and K landmark points. For each exemplar face i, we train a linear SVM classifier w_i using HOG features: the positive example is the exemplar itself, and negatives are sampled from non-face images via hard negative mining. At test time, given an input image, we densely compute HOG features and score each location with all N classifiers. High-scoring locations indicate that the local appearance matches a particular exemplar. These scores form N response maps, from which we construct a unified voting map for face center locations.

Each exemplar classifier that fires at a location casts a vote for the face center, offset by the known displacement from the exemplar's detection window to its face center. The votes from all exemplars are accumulated in a Hough voting space. We identify face detections by finding modes (peaks) in the voting map using mean-shift clustering. This voting approach is inherently robust: even if some exemplar matches are incorrect, the correct matches will produce coherent votes that reinforce each other, while incorrect votes are spatially scattered. The mode locations give face center estimates, and the associated exemplars provide bounding box and landmark predictions through simple geometric transfer.

A key advantage of our exemplar-based framework is that detection naturally provides alignment. Since each matched exemplar carries annotated landmark positions, we can transfer these landmarks to the detected face through the geometric mapping between the exemplar and the detection. When multiple exemplars vote for the same face, we compute the median landmark positions across all contributing exemplars for robust estimation. For face validation, we train a secondary classifier on features extracted from the aligned face, which effectively removes false positives by verifying that the detection exhibits face-like structure when properly aligned. This reduces the false positive rate by over 50% while maintaining recall.

We evaluate on FDDB (2,845 images, 5,171 faces) and AFW (205 images with diverse poses) for face detection, and LFPW and LFW for face alignment. Using a database of approximately 20,000 exemplar faces, our method achieves competitive detection performance on FDDB, outperforming the Viola-Jones and DPM-face baselines. On AFW with large pose variations, our method significantly outperforms all baselines, demonstrating the benefit of exemplar diversity. For alignment on LFPW, we achieve mean error below 5% of inter-ocular distance, competitive with specialized alignment methods. The framework also generalizes to pedestrian detection and car detection, showing consistent improvements over vanilla Exemplar-SVM.

We have presented an exemplar-based framework that unifies face detection, alignment, and validation through image retrieval. By leveraging a large annotated database and voting-based aggregation, our approach implicitly handles the full range of facial variation without requiring explicit modeling of pose, lighting, or expression. The framework is flexible, easily extensible, and applicable beyond face detection. Future directions include scaling to larger databases with efficient indexing and incorporating deep features to replace HOG representations.