DeepFace: Closing the Gap to Human-Level Performance in Face Verification

What the paper set out to do

Face verification is the task of deciding whether two photographs show the same person. A conventional system splits this into stages: detect the face, align it so that facial features land at consistent positions, compute a numerical representation of the face, and finally compare two representations to decide whether they belong to one identity. DeepFace, which appeared at CVPR 2014 on pages 1701 to 1708, concentrates on the second and third of these stages: it revisits both the alignment stage and the representation stage of the classic detect-align-represent-classify pipeline^[1].

Alignment through a 3D model

Two photos of the same person rarely line up. Head pose, framing and rotation differ, and a representation computed on unaligned crops will treat the same face as two different patterns. The paper's alignment step therefore uses explicit 3D face modelling and then applies a piecewise affine transformation^[1]. In plain terms, a three-dimensional model of a face serves as a reference, and the photograph is warped toward it. A piecewise affine transformation does this region by region rather than with a single global warp: the image is divided into pieces, and each piece is mapped with its own simple geometric transformation, which allows a closer fit than one transformation applied everywhere.

A large, locally connected network

The face representation is produced by a nine-layer deep neural network with more than 120 million parameters^[1]. Its architecture departs from the usual recipe in one notable way: several of its layers are locally connected layers without weight sharing, used in place of standard convolutional layers^[1]. The difference is easy to state. In a convolutional layer one small filter is slid across the image, so the same weights are reused at every position; in a locally connected layer each position has its own set of weights. That gives the network more freedom to model position-dependent structure, at the cost of many more parameters to store and train.

The network was trained on an identity-labelled dataset of four million facial images belonging to more than 4,000 identities^[1]. The supervision is identity: images are labelled with who the person is, and the network learns to tell those identities apart. The representation that results from this training is then what the system compares when it has to decide whether two photographs match.

Results

On Labeled Faces in the Wild, a standard benchmark for face verification on unconstrained photographs, the method reaches 97.35% accuracy and reduces the error of the previous state of the art by more than 27%^[1]. The second number is the more informative one. Once accuracy is already high, each further point is hard to win, because most of the easy cases are already solved; what remains is the error rate, the share of pairs the system still gets wrong. Cutting that remaining error by more than 27% means a large share of the mistakes earlier systems made have disappeared.

Why it matters

The title makes the ambition explicit: closing the gap to human-level performance in face verification. Verification on unconstrained photographs is hard precisely because the photographs are unconstrained: the same face appears under different lighting, at different angles, at different distances, and at different moments in a person's life. The paper's answer is to treat two parts of the established pipeline as problems to be solved properly rather than as fixed preprocessing: alignment gets an explicit 3D model, and the representation gets a very large learned network trained on millions of labelled images^[1]. The reported figures put the method clearly ahead of the previous state of the art on a widely used benchmark^[1].