Real-Time Human Pose Recognition in Parts from a Single Depth Image

What the paper set out to do

Reading a person's pose from a camera image — finding where the body's joints sit in three dimensions — is one of the classic problems in computer vision. This paper asks a deliberately narrow question: how much of the pose can be recovered from a single depth image, with no temporal information at all? A depth image is like an ordinary photograph except that each pixel stores the distance from the sensor to the nearest surface rather than a colour, so the shape of a person stands out while texture and lighting do not. Because the method uses no temporal information, it does not track a person across successive frames or use the previous frame to guess the next; each frame is treated on its own ^[1].

The abstract states the goal plainly: the method quickly and accurately predicts 3D positions of body joints from a single depth image using no temporal information ^[1].

The idea: classify pixels into body parts

The central move is to introduce an intermediate representation. Instead of predicting joint positions directly, the system first labels pixels with the body part they belong to, which maps the difficult pose estimation problem into a simpler per-pixel classification problem ^[1]. The intermediate step is what makes the rest tractable: deciding which part a pixel belongs to is a fairly local judgement that can be made from local depth evidence, whereas a joint position in 3D is a global quantity that depends on the whole body.

Training such a classifier needs data that covers many poses, body shapes and items of clothing. The abstract reports that a large and highly varied training dataset lets the classifier estimate body parts invariant to pose, body shape and clothing ^[1]. In other words, the variety in the training data is what keeps the labelling useful when a new person stands in a new posture wearing something the system has not seen.

Getting from part labels back to joints is a second step. The system generates confidence-scored 3D proposals of several body joints by reprojecting the classification result and finding local modes ^[1]. A mode is simply a place where the reprojected evidence piles up; the confidence score records how strongly the system supports that particular proposal.

Where the training data came from

The supplementary material accompanying the paper at CVPR 2011 gives more detail on the randomized synthetic rendering pipeline used to build training data ^[2]. That pipeline used 3D models of 15 varied base characters, both male and female, from child to adult ^[2]. Varying the base characters is one way to obtain the range of body shapes that the classifier is asked to be robust to.

Results

The headline number is speed: the system runs at 200 frames per second on consumer hardware ^[1]. That is far faster than the frame rates of ordinary cameras, and it is achieved without special hardware and without temporal smoothing across frames.

Accuracy was evaluated on both synthetic and real test sets, where the system showed high accuracy, and it reached state-of-the-art accuracy compared with related work ^[1]. Testing on real data matters here because the training pipeline is synthetic: the results indicate that what the classifier learned carried over from rendered depth images to depth images captured by a real sensor.

Why it matters

The paper was identified as the CVPR 2011 Best Paper in a Microsoft Research Faculty Summit 2011 presentation ^[3]. The same presentation lists the authors as Jamie Shotton, Andrew Fitzgibbon, Mat Cook, Toby Sharp, Mark Finocchio, Richard Moore, Alex Kipman and Andrew Blake ^[3].

Its contribution is a reformulation as much as a result. Turning full-body pose estimation into per-pixel part classification, and then turning part labels back into joint proposals by reprojection and mode finding, replaces a hard structured 3D problem with a classification problem that can be solved fast ^[1]. Combined with a randomized synthetic rendering pipeline that supplies large and varied training data ^[1][2], the work showed that a single depth frame — no history, no markers — carries enough information for full-body pose, and that the computation fits comfortably within the budget of consumer hardware at 200 frames per second ^[1].