Figure 2. Overall pipeline. Our method takes the entire image as the input for a two-branch CNN to jointly predict confidence maps for  body part detection, shown in (b), and part affinity fields for parts association, shown in (c). The parsing step performs a set of bipartite  matchings to associate body parts candidates (d). We finally assemble them into full body poses for all people in the image (e).

The image is first analyzed by a convolutional network  (initialized by the first 10 layers of VGG-19 [26] and finetuned),  generating a set of feature maps F that is input to  the first stage of each branch. At the first stage, the network  produces a set of detection confidence maps S  1 = ρ  1  (F)  and a set of part affinity fields L  1 = φ  1  (F), where ρ  1  and  φ  1  are the CNNs for inference at Stage 1. In each subsequent  stage, the predictions from both branches in the previous  stage, along with the original image features F, are  concatenated and used to produce refined predictions

Figure 4. Confidence maps of the right wrist (first row) and PAFs  (second row) of right forearm across stages. Although there is confusion  between left and right body parts and limbs in early stages,  the estimates are increasingly refined through global inference in  later stages, as shown in the highlighted areas.

Fig. 4 shows the refinement of the confidence maps and  affinity fields across stages. To guide the network to iteratively  predict confidence maps of body parts in the first  branch and PAFs in the second branch, we apply two loss  functions at the end of each stage, one at each branch respectively.  We use an L2 loss between the estimated predictions  and the groundtruth maps and fields. Here, we weight  the loss functions spatially to address a practical issue that some datasets do not completely label all people. Specifically,  the loss functions at both branches at stage t are:

where S  ∗  j  is the groundtruth part confidence map, L  ∗  c  is the  groundtruth part affinity vector field, W is a binary mask  with W(p) = 0 when the annotation is missing at an image  location p. The mask is used to avoid penalizing the  true positive predictions during training. The intermediate  supervision at each stage addresses the vanishing gradient  problem by replenishing the gradient periodically [31]. The  overall objective is

We first generate individual confidence maps S  ∗  j,k for  each person k. Let xj,k ∈ R2 be the groundtruth position of  body part j for person k in the image. The value at location  p ∈ R2  in S  ∗  j,k is defined as,

where σ controls the spread of the peak. The groundtruth  confidence map to be predicted by the network is an aggregation  of the individual confidence maps via a max operator,

We take the maximum of  the confidence maps instead  of the average so that the  precision of close by peaks  remains distinct, as illustrated  in the right figure. At test time, we predict confidence  maps (as shown in the first row of Fig. 4), and obtain body  part candidates by performing non-maximum suppression.

Given a set of detected body parts (shown as the red and  blue points in Fig. 5a), how do we assemble them to form  the full-body poses of an unknown number of people? We  need a confidence measure of the association for each pair  of body part detections, i.e., that they belong to the same  person. One possible way to measure the association is to  detect an additional midpoint between each pair of parts  on a limb, and check for its incidence between candidate  part detections, as shown in Fig. 5b. However, when people  crowd together—as they are prone to do—these midpoints  are likely to support false associations (shown as green lines  in Fig. 5b). Such false associations arise due to two limitations  in the representation: (1) it encodes only the position,  and not the orientation, of each limb;
(2) it reduces the region  of support of a limb to a single point.

To evaluate fL in Eq. 5 during training, we define the  groundtruth part affinity vector field, L  ∗  c,k, at an image point  p as

L ∗ c,k(p) = ( v if p on limb c, k 0 otherwise. (8)
Here, v = (xj2,k − xj1,k)/||xj2,k −xj1,k||2 is the unit vector in the direction of the limb. The set of points on the limb is defined as those within a distance threshold of the line segment, i.e., those points p for which

0 ≤ v · (p − xj1,k) ≤ lc,k and |v⊥ · (p − xj1,k)| ≤ σl
where the limb width σl is a distance in pixels, the limb length is lc,k = ||xj2,k − xj1,k||2, and v⊥ is a vector perpendicular to v.

The groundtruth part affinity field averages the affinity fields of all people in the image,

L ∗ c (p) = 1 nc(p) X k L ∗ c,k(p), (9)

where nc(p) is the number of non-zero vectors at point p across all k people (i.e., the average at pixels where limbs of different people overlap).

During testing, we measure association between candidate part detections by computing the line integral over the corresponding PAF, along the line segment connecting the candidate part locations. In other words, we measure the alignment of the predicted PAF with the candidate limb that would be formed by connecting the detected body parts. Specifically, for two candidate part locations dj1 and dj2 , we sample the predicted part affinity field, Lc along the line segment to measure the confidence in their association:

where p(u) interpolates the position of the two body parts dj1 and dj2 ,

E = Z u=1 u=0 Lc (p(u)) · dj2 − dj1 ||dj2 − dj1 ||2 du, (10)

In practice, we approximate the integral by sampling and summing uniformly-spaced values of u.

p(u) = (1 − u)dj1 + udj2 . (11)

If we consider a single pair of parts j1 and j2 (e.g., neck and right hip) for the c-th limb, finding the optimal association reduces to a maximum weight bipartite graph matching problem [32]. This case is shown in Fig. 5b. In this graph matching problem, nodes of the graph are the body part detection candidates Dj1 and Dj2 , and the edges are all possible connections between pairs of detection candidates. Additionally, each edge is weighted by Eq. 10—the part affinity aggregate. A matching in a bipartite graph is a subset of the edges chosen in such a way that no two edges share a node. Our goal is to find a matching with maximum weight for the chosen edges

max Zc Ec = max Zc X m∈Dj1 X n∈Dj2 Emn · z mn j1j2 , (12)
s.t. ∀m ∈ Dj1 , X n∈Dj2 z mn j1j2 ≤ 1, (13)
∀n ∈ Dj2 , X m∈Dj1 z mn j1j2 ≤ 1, (14)
where Ec is the overall weight of the matching from limb type c, Zc is the subset of Z for limb type c, Emn is the part affinity between parts d m j1 and d n j2 defined in Eq. 10. Eqs. 13 and 14 enforce no two edges share a node, i.e., no two limbs of the same type (e.g., left forearm) share a part. We can use the Hungarian algorithm [14] to obtain the optimal matching.

With these two relaxations, the optimization is decomposed simply as: max Z E = X C c=1 max Zc Ec. (15)

We therefore obtain the limb connection candidates for each limb type independently using Eqns. 12- 14. With all limb connection candidates, we can assemble the connections that share the same part detection candidates into full-body poses of multiple people. Our optimization scheme over the tree structure is orders of magnitude faster than the optimization over the fully connected graph [22, 11].

We evaluate our method on two benchmarks for multiperson pose estimation: (1) the MPII human multi-person dataset [2] and (2) the COCO 2016 keypoints challenge dataset [15]. These two datasets collect images in diverse scenarios that contain many real-world challenges such as crowding, scale variation, occlusion, and contact. Our approach set the state-of-the-art on the inaugural COCO 2016 keypoints challenge [1], and significantly exceeds the previous state-of-the-art result on the MPII multi-person benchmark. We also provide runtime analysis to quantify the efficiency of the system. Fig. 10 shows some qualitative results from our algorithm.

In Table 4, we report self-comparisons on a subset of the COCO validation set, i.e., 1160 images that are randomly selected. If we use the GT bounding box and a single person CPM [31], we can achieve a upper-bound for the top-down approach using CPM, which is 62.7% AP. If we use the state-of-the-art object detector, Single Shot MultiBox Detector (SSD)[16], the performance drops 10%. This comparison indicates the performance of top-down approaches rely heavily on the person detector. In contrast, our bottom-up method achieves 58.4% AP. If we refine the results of our method by applying a single person CPM on each rescaled region of the estimated persons parsed by our method, we gain an 2.6% overall AP increase. Note that we only update estimations on predictions that both methods agree well enough, resulting in improved precision and recall. We expect a larger scale search can further improve the performance of our bottom-up method. Fig. 8 shows a breakdown of errors of our method on the COCO validation set. Most of the false positives come from imprecise localization, other than background confusion. This indicates there is more improvement space in capturing spatial dependencies than in recognizing body parts appearances.

Moments of social significance, more than anything else, compel people to produce photographs and videos. Our photo collections tend to capture moments of personal significance: birthdays, weddings, vacations, pilgrimages, sports events, graduations, family portraits, and so on. To enable machines to interpret the significance of such photographs, they need to have an understanding of people in images. Machines, endowed with such perception in real time, would be able to react to and even participate in the individual and social behavior of people.

In this paper, we consider a critical component of such perception: realtime algorithms to detect the 2D pose of multiple people in images. We present an explicit nonparametric representation of the keypoints association that encodes both position and orientation of human limbs. Second, we design an architecture for jointly learning parts detection and parts association. Third, we demonstrate that a greedy parsing algorithm is sufficient to produce highquality parses of body poses, that maintains efficiency even as the number of people in the image increases. We show representative failure cases in Fig. 9. We have publicly released our code (including the trained models) to ensure full reproducibility and to encourage future research in the area.