In modern face recognition, the key pipeline consists of four stages: detect, align, represent, and classify. In this paper, we revisit both the alignment and representation steps. For alignment, we employ explicit 3D face modeling in order to apply a piecewise affine transformation that corrects for out-of-plane rotations. For representation, we use a nine-layer deep neural network involving more than 120 million parameters, which are trained on the largest facial dataset to date — four million facial images belonging to more than 4,000 identities. The network uses locally connected layers without weight sharing rather than the standard convolutional layers. Our method reaches an accuracy of 97.35% on the Labeled Faces in the Wild (LFW) dataset, reducing the error of the current state of the art by more than 27%, closely approaching human-level performance.

Face recognition in unconstrained environments is a long-standing challenge in computer vision. Despite decades of research, machines still fall behind humans on face verification benchmarks, particularly when faces exhibit large variations in pose, illumination, expression, and occlusion. Recent progress in deep learning provides new tools to address this gap. Our work shows that by combining an effective 3D alignment procedure with a large-scale deep neural network trained on an unprecedented amount of face data, we can dramatically close the gap between machine and human performance. The key insight is that face-specific alignment combined with deep representations yields substantially better features than generic approaches.

Alignment is a crucial preprocessing step. We employ a 3D face alignment pipeline that proceeds as follows: first, we detect six fiducial points (two eyes, nose tip, and three mouth points) using a face detector. Second, we fit a generic 3D face model to the detected fiducial points through an affine camera model. Third, we apply a piecewise affine transformation based on the Delaunay triangulation of the 2D projected fiducial points to warp the face to a frontal canonical coordinate system. This 3D-based alignment effectively removes out-of-plane rotations, generating a frontalized face image that is then cropped to a 152x152 pixel input for the neural network.

The DeepFace architecture consists of nine layers: the first three are standard convolutional layers (C1, C2, C3) with max-pooling, followed by a locally connected layer (L4) that does not share weights across spatial positions, then another locally connected layer (L5), and finally two fully connected layers (F6, F7) and a softmax output layer. The critical design choice is the use of locally connected layers instead of convolutional layers in the upper regions: since different face regions (e.g., eyes, nose, mouth) have different local statistics after alignment, weight sharing across positions is inappropriate. The entire network contains over 120 million parameters, with the locally connected layers accounting for the majority. The F7 layer produces a 4096-dimensional face descriptor used for verification.

The network is trained on a dataset of four million facial images belonging to 4,030 identities, collected from Facebook's social network. Training is performed as a multi-class classification task where each identity is a class. The network is optimized using stochastic gradient descent with momentum. After training, the softmax layer is removed, and the F7 layer activations serve as the face representation. For face verification, two face representations are compared using either the weighted chi-squared distance or a Siamese network configuration where the absolute difference of the two descriptors is fed into a learned classifier. Optionally, Principal Component Analysis (PCA) is applied to reduce dimensionality and remove correlations.

We evaluate primarily on the Labeled Faces in the Wild (LFW) benchmark, the standard testbed for unconstrained face verification. DeepFace achieves 97.35% accuracy, compared to 95.17% for the previous state of the art (Tom-vs-Pete classifiers). This represents a relative error reduction of more than 27%. We also evaluate on the YouTube Faces (YTF) dataset, where DeepFace achieves 91.4% accuracy, also setting a new state of the art. Notably, human performance on LFW is estimated at 97.53%, meaning DeepFace approaches within 0.18% of human accuracy. The 3D alignment step contributes approximately 1% improvement over 2D affine alignment, confirming its importance for handling pose variation.

We have presented DeepFace, a face verification system that closes the gap to human-level performance by combining 3D face alignment with a large-scale deep neural network. Our results demonstrate that careful alignment and sufficiently large training data, combined with a deep architecture that respects the structure of the aligned face, can produce face representations that rival human perception. The use of locally connected layers, motivated by the non-stationary statistics of aligned faces, proves to be a key architectural choice. Looking ahead, we believe that further improvements in training data scale, network depth, and alignment accuracy will continue to push the boundaries of face recognition.