# AlphaPose: Whole-Body Regional Multi-Person Pose Estimation and Tracking in Real-Time Hao-Shu Fang $^{ID}$, Jiefeng Li $^{ID}$, Hongyang Tang, Chao Xu, Haoyi Zhu $^{ID}$, Yuliang Xiu $^{ID}$, Yong-Lu Li $^{ID}$, and Cewu Lu $^{ID}$, Member, IEEE Abstract—Accurate whole-body multi-person pose estimation and tracking is an important yet challenging topic in computer vision. To capture the subtle actions of humans for complex behavior analysis, whole-body pose estimation including the face, body, hand and foot is essential over conventional body-only pose estimation. In this article, we present AlphaPose, a system that can perform accurate whole-body pose estimation and tracking jointly while running in realtime. To this end, we propose several new techniques: Symmetric Integral Keypoint Regression (SIKR) for fast and fine localization, Parametric Pose Non-Maximum-Suppression (P-NMS) for eliminating redundant human detections and Pose Aware Identity Embedding for jointly pose estimation and tracking. During training, we resort to Part-Guided Proposal Generator (PGPG) and multi-domain knowledge distillation to further improve the accuracy. Our method is able to localize whole-body keypoints accurately and tracks humans simultaneously given inaccurate bounding boxes and redundant detections. We show a significant improvement over current state-of-the-art methods in both speed and accuracy on COCO-wholebody, COCO, PoseTrack, and our proposed Halpe-FullBody pose estimation dataset. Our model, source codes and datasets are made publicly available at . Index Terms—Human pose estimation, pose tracking, whole-body pose estimation, hand pose estimation, realtime, multi-person ## 1 INTRODUCTION FULL body human pose estimation is a fundamental challenge for computer vision. It has many applications in human-computer interaction [^1], film industry [^2], action recognition [^3], etc. In this work, we focus on the problem of multi-person full body pose estimation. In conventional body-only pose estimation, recognizing the pose of multiple persons in the wild is more challenging than recognizing the pose of a single person in an image [^4], [^5], [^6], [^7], [^8]. Previous attempts approached this problem by using either a top-down framework [^9], [^10] or a bottom-up framework [^11], [^12], [^13]. Our approach follows the top-down framework, which first detects human bounding boxes and then estimates the pose within each box independently. For top-down based methods, although their performances are dominant on common benchmarks [^14], [^15], such methodology has some drawbacks. Since the detection stage and the pose estimation stage are separated, i) if the detector fails, there is no cue for the pose estimator to recover the human pose, and ii) current researchers adopt strong human detectors for accuracy, which makes the two step processing slow in inference. To solve these drawbacks of the top-down framework, we propose a new methodology to make it efficient and reliable in practice. To alleviate the missing detection problem, we lower the detection confidence and NMS threshold to provide more candidates for subsequent pose estimation. The resulted redundant poses from redundant boxes are then eliminated by a parametric pose NMS, which introduces a novel pose distance metric to compare pose similarity. A data-driven approach is applied to optimize the pose distance parameters. We show that with such strategy, a top-down framework with YOLOV3-SPP detector can achieve on par performance with the state-of-the-art detectors while achieving much higher efficiency. Furthermore, to speed up the top-down framework during inference, we design a multi-stage concurrent pipeline in AlphaPose, which allows our framework to run in realtime. Beyond body-only pose estimation, full body pose estimation in the wild is more challenging as it faces several extra problems. For both top-down framework and bottom-up framework, the currently most used representation for keypoint is the heatmap [^16]. And the heatmap size is usually the quarter of the input image due to the limit of computation resources. However, for localizing the keypoints of body, face and hands simultaneously, such representation is unsuitable since it is incapable of handling the large scale variation across different body parts. A major problem is referred to as the quantization error. As illustrated in Fig. 1, since the heatmap representation is discrete, both the adjacent grids on heatmap may miss the correct position. This is not a problem for body pose estimation since the correct --- ![Image](images/1640c3afccb2ade7693ea24734932f67d6be6009ccb0602a1981990833e591a8.jpg)
Fig. 1. The quantization error caused by heatmap (green and blue lines). With our symmetric integral keypoints regression (pink line), we can resolve the localization error.
area are usually large. However, for fine-level keypoints on hands and face, it is easy to miss the correct position. To solve this problem, previous methods either adopt additional sub-networks for hand and face estimation [^17], or adopt ROI-Align to enlarge the feature map [^18]. However, both methods are computation expensive, especially in multi-person scenario. In this paper, we propose a novel symmetric integral keypoints regression method that can localize keypoints in different scales accurately. It is the first regression method that can have the accuracy on par with heatmap representation while eliminate the quantization error. Another problem for the full body pose estimation is the lack of training data. Unlike the frequent studied body pose estimation with abundant datasets [^14], [^19], there is only one dataset [^18] for the full body pose estimation. To promote development in this area, we annotate a new dataset named Halpe for this task, which includes extra essential joints not available in [^18]. To further improve the generality of top-down framework for full body pose estimation in the wild, two key components are introduced. We adopt a Multi-Domain Knowledge Distillation to incorporate training data from separate body part datasets. To alleviate the domain gap between different datasets and the imperfect detection problem, we propose a novel part-guided human proposal generator (PGPG) to augment training samples. By learning the output distribution of a human detector for different poses, we can simulate the generation of human bounding boxes, producing a large sample of training data. At last, we introduce a pose-aware identity embedding to enable simultaneous human pose tracking within our top-down framework. A person re-id branch is attached on the pose estimator and we perform jointly pose estimation and human identification. With the aid of pose-guided region attention, our pose estimator is able to identify human accurately. Such design allows us to achieve realtime pose estimation and tracking in an unified manner. This manuscript extends our preliminary work published at the conference ICCV 2017 [^20] along the following aspects: - We extend our framework to full body pose estimation scenario and propose a new symmetric integral keypoint localization network for fine-level localization. - We extend our pose guided proposal generator to incorporate with multi-domain knowledge distillation on different body part datasets. - We annotate a new whole-body pose estimation benchmark (136 points for each person) and make comparisons with previous methods. We propose the pose-aware identity embedding that enable pose tracking in our top-down framework in a unified manner. - This work documents the release of AlphaPose, which achieves both accurate and realtime performance. Our library facilitated many researchers and has been starred for over 6,000 times on GitHub. ## 2 RELATED WORK In this section, we first briefly review papers in multi-person pose estimation, which provides background knowledge of human pose estimation. In Section 2.2 we review related works in multi-person whole-body pose estimation and discuss a key issue in the current literature. In Section 2.3 we review integral regression based keypoint localization and clarify our improvements toward previous works. In Section 2.4 we review pose tracking and summarize the connection and differences between previous works and our method. ### 2.1 Multi Person Pose Estimation Bottom-Up Approaches. Bottom-up approaches are also called part-based approaches in the early. These approaches first detect all possible body parts in an image and then group them into individual skeletons. Representative works [^10], [^11], [^12], [^13], [^21], [^22], [^23] are reviewed. Chen et al. [^11] present an approach to parse largely occluded people by graphical model which models humans as flexible compositions of body parts. Gkiox et al. [^10] use k-poselets to jointly detect people and predict locations of human poses by a weighted average of all activated poselets. Pishchulin et al. [^12] propose DeepCut to first detect all body parts, and then label, filter and assemble these parts via integral linear programming. A stronger part detector based on ResNet[^24] and a better incremental optimization strategy is proposed by Insafutdinov et al. [^13], named DeeperCut. Openpose [^17], [^25] introduces Part Affinity Fields (PAFs) to encode association scores between body parts with individuals and solves the matching problem by decomposing it into a set of bipartite matching subproblems. Newell et al. [^26] learn an identification tag for each part detected to indicate which individuals it belongs to, named associative embedding. Cheng et al.[^27] use a powerful multi-resolution network[^28] as backbone and high-resolution feature pyramids to learn scale-aware representations. OpenPif-paf [^22], [^23] proposes a Part Intensity Field (PIF) and a Part Association Field (PAF) to localize and associate body parts respectively. While bottom-up approaches have demonstrated good performance, their body-part detectors can be vulnerable since only small local regions are considered and face the scale variation challenge when there are small persons in the image. Top-Down Approaches. Our work follows the top-down paradigm like others[^9], [^20], [^28], [^29], [^30], [^31], which first obtains the bounding box for each human body through object detector and then performs single-person pose estimation on August 16, 2026 at 10:27:07 UTC from IEEE Xplore. Restrictions apply. --- cropped image. Fang et al.[^20] propose symmetric spatial transformer network to solve the problem on imperfect bounding boxes with huge noise given by human body detector. Mask R-CNN[^29] extends Faster R-CNN[^32] by adding a pose estimation branch in parallel with existing bounding box recognition branch after ROIAlign, enabling end-to-end training. PandaNet [^33] proposes an anchor based method to predict multi-person 3D pose estimation in a single shot manner and achieved high efficiency. Chen et al.[^30] use a feature pyramid network to localize simple joints and a refining network which integrates features of all levels from previous network to handle hard joints. A simple-structured network [^31] with ResNet[^24] as backbone and a few deconvolutional layers as up-sampling head shows effective and competitive results. Sun et al.[^28] present a powerful high-resolution network, where a high-resolution subnetwork is established in the first stage, and high-to-low resolution subnetworks are added one by one in parallel in subsequent stages, conducting repeated multi-scale feature fusions. Bertasius et al.[^34] extend from images to videos and propose a method for learning pose warping on sparsely labeled videos. Although state-of-the-art top-down approaches achieve remarkable precision on popular large-scale benchmark, The two step paradigm makes them slow in inference compared with the bottom-up approaches. In addition, the lack of library-level framework implementation hinders them from being applied to the industry. Thus we present AlphaPose in this paper, in which we develop a multi-stage pipeline to simultaneously process the time-consuming steps and enable fast inference. One-Stage Approaches. Some approaches need neither post joints grouping nor human body bounding boxes detected in advance. They locate human bodies and detect their own joints simultaneously to improve low efficiency in two-stage approaches. Representative works include CenterNet [^35], SPM [^36], DirectPose [^37], and Point-set Anchor [^38]. However, these approaches do not achieve high precision as top-down approaches, partly because body center map and dense joint displacement maps are high-semantic nonlinear representations and make it difficult for the networks to learn. ### 2.2 Whole-Body Keypoint Localization Unified detection of body, face, hand and foot keypoints for multi-person is a relative new research topic and few methods have been proposed. OpenPose [^17] developed a cascaded method. It first detects body keypoints using PAFs [^25] and then adopts two separate networks to estimate face landmarks and hand keypoints. Such design makes it time inefficient and consumes extra computation resources. Hidalgo et al. [^39] propose a single network to estimate the whole body keypoints. However, due to its one-step mechanism, the output resolution is limited and thus decreases its performance on fine-level keypoints such as faces and hands. Jin et al. [^18] propose a ZoomNet that used ROI\_align to crop the hand and face region on the feature maps and predict keypoints on the resized feature maps. All these methods adopt the heatmap representation for keypoint localization due to its dominant performance on body keypoints. However, the mentioned quantization problem of heatmap would decrease the accuracy of face and hand keypoints. The requirement of large-size input also consumes more computation resources. In this paper, we argue that soft-argmax presentation is more suitable for whole-body pose estimation and proposed an improved version of soft-argmax that yields higher accuracy. Jin et al. [^18] also extended the COCO dataset to whole-body scenario. However, some joints like head and neck are not presented in this dataset, which is essential in tasks like mesh reconstruction. Meanwhile, the face annotation is incompatible with that in 300LW. In this paper, we contribute a new in-the-wild multi-person whole-body pose estimation benchmark. We annotate 40K images from HICO-DET [^40] with Creative Common license $^{1}$ as the training set and extend the COCO keypoints validation set (6K instances) as our test set. Experiments on this benchmark and COCO-Wholebody demonstrate the superiority of our method. ### 2.3 Integral Keypoints Localization Heatmap is a dominant representation for joint localization in the field of human pose estimation. The read-out locations from heatmaps are discrete numbers since heatmaps only describe the likelihood of joints occurring in each spatial grid, which leads to inevitable quantization error. As denoted in Section 2.2, we argue that soft-argmax based integral regression is more suitable for whole-body keypoints localization. Several previous works have studied the soft-argmax operation to read continuous joint locations from heatmaps [^41], [^42], [^43], [^44], [^45], [^46]. Specifically, Luvizon et al. [^43], [^46] and Sun et al. [^45] apply the soft-argmax operation to single-person 2D/3D pose estimation successfully. However, two drawbacks exist in these works that decrease their accuracy in pose estimation. We summarize the drawbacks as asymmetric gradient problem and size-dependent keypoint scoring problem. Details of these problems are provided in Section 3.1, as well as our proposed new gradient design and keypoint scoring method. By solving these problems, we provide a new keypoint regression method with higher accuracy, and it shows good performance in both whole-body pose estimation and body-only pose estimation. ### 2.4 Multi Person Pose Tracking Multi person pose tracking is extended from multi person pose estimation in videos, which gives each predicted keypoint the corresponding identity over time. Similar to pose estimation literature, it can be divided into two categories: top-down $^{[31]}$, $^{[47]}$, $^{[48]}$, $^{[49]}$, $^{[50]}$, $^{[51]}$, $^{[52]}$, $^{[53]}$, $^{[54]}$ and bottom-up $^{[23]}$, $^{[55]}$, $^{[56]}$. Based on the bottom-up pose estimation methods, $^{[55]}$, $^{[56]}$ use the detected keypoints to build temporal and spatial graphs which aims to link corresponding individual body by solving an optimization problem. However, the prerequisite of temporal and spatial graphs prevent graph-cut optimization from running in online manner, which makes them quite time-consuming and memory-inefficient. $^{[49]}$ utilizes a 3D MaskRCNN to estimate person tubes and poses simultaneously. $^{[50]}$ proposes forward and backward bounding box propagation strategy to eliminate the issue of missed detection. --- ![Image](images/185ceea2376ac8e93f99505c0fc14d0bd7291708024d1cf15e7bf66d6cdd1e87.jpg)
Fig. 2. Illustration of our full-body pose estimation and tracking framework. Given an input image, we first obtain (i) human detections using off-the-shelf object detectors like YoloV3 or EfficientDet. For each detected human, we crop and resize it and forward it through pose estimation and tracking networks to obtain full body human pose and Re-ID features. The backbone of these two networks can either be separated for adaptation to different pose configurations, or share the same weights for fast inference (thus misaligned in the figure). The (a) symmetric integral regression is adopted for fine-level keypoint localization. We adopt (b) pose NMS to eliminate redundant poses. The (c) pose-guided alignment (PGA) module is applied on the predicted human re-id feature to obtain pose-aligned human re-id features. The (d) multi-stage identity matching (MSIM) utilizes the human poses, re-id features and detected boxes to produce the final tracking identity. During training, the proposed method can achieve a higher accuracy in the training.
input of these methods is a whole video sequence, which cannot achieve online tracking. Some other top-down methods allow to input a single frame, and then use designed poseflow $^{[47]}$, GCN $^{[48]}$, [^52], optical flow $^{[31]}$ or transformer [^54] for identity matching. Yang et al. [^53] predict current poses given historical pose sequences and merge them with the pose estimation results from the current frame. A drawback of these methods is that they rely on the spatial continuity of the poses only, which may not be satisfied when the online image stream is unstable or humans are moving rapidly. Specifically, [^57] proposes to use re-ID feature to tackle tracking problem. Our tracking method also explicitly adopts the human re-ID features to solve this problem. Compared with [^57], we design a pose-guided re-ID feature extraction to avoid potential background noise. Moreover, we design a multi-stage information merging method to utilize the boxes, poses, and re-ID features simultaneously. ## 3 WHOLE-BODY MULTI PERSON POSE ESTIMATION The whole pipeline of our proposed method is illustrated in Fig. 2. In this section, we introduce the details of our pose estimation, which is shown in the top row of Fig. 2. ### 3.1 Symmetric Integral Keypoints Regression As mentioned in Section 2.3, there exist two problems in conventional soft-argmax operation for keypoint regression. We illustrate them in the following sections and propose our novel solution. #### 3.1.1 Asymmetric Gradient Problem The soft-argmax operation, also known as integral regression, is differentiable, which turns heatmap based approaches into regression based approaches and allows end-to-end training. The integral regression operation is defined as: where x is coordinate of each pixel and $p_x$ denotes the pixel likelihood on heatmap after normalization. During training, the loss function is applied to minimize the $\ell_1$ norm between the predicted joint locations $\hat{\mu}$ and ground-truth locations $\mu$: $\mathcal{L}_{reg} = \|\mu - \hat{\mu}\|_1$. The gradient of each pixel can be formulated as: $$ \hat{\mu}=\sum x\cdot p_{x}, $$ Notice that the gradient amplitude is asymmetric. The absolute value of the gradient is determined by the absolute position (i.e., x) of the pixel instead of the relative position to the ground truth. It denotes that given the same distance error, the gradient becomes different when the keypoint locates at a different position. This asymmetry breaks the translation invariance of the CNN network, which leads to performance degradation. $$ \frac{\partial\mathcal{L}_{r e g}}{\partial p_{x}}=x\cdot\mathrm{s g n}(\hat{\mu}-\mu). $$ Amplitude Symmetric Gradient. To improve the learning efficiency, we propose an amplitude symmetric gradient (ASG) function in backward propagation, which is an approximation to the true gradient: $$ \begin{array}{r}{\delta_{A S G}=A_{g r a d}\cdot\mathrm{s g n}(x-\hat{\mu})\cdot\mathrm{s g n}(\hat{\mu}-\mu),}\end{array} $$ where $A_{grad}$ denotes the amplitude of gradients. It is a constant that we manually set as 1/8 of the heatmap size and we give the derivation in the next paragraph. Using our symmetric gradient, the gradient distribution is centred at the predicted joint locations $\hat{\mu}$. In the learning process, this symmetric gradient distribution can better utilize the advantage of heatmaps and approximate the ground-truth in August 16, 2026 at 10:27:07 UTC from IEEE Xplore. Restrictions apply. --- locations in a more direct manner. For example, assume the predicted location $\hat{\mu}$ is higher than the ground truth $\mu$. On one hand, the network tends to suppress the heatmap values on the right side of $\hat{\mu}$, because they have positive gradients; on the other hand, the heatmap values on the left side of $\hat{\mu}$ will be activated because of their negative gradients. Stable Gradients for ASG. Here, we conduct a Lipschitz analysis to derive the value of $A_{grad}$ and show that ASG can provide a more stable gradient for training. Recall that f denotes the objective function that we want to minimize. We say that f is L-smooth if: $$ \|\nabla_{\theta}f(\theta+\Delta\theta)-\nabla_{\theta}(\theta)\|\leq L\|\Delta\theta\|, $$ where $\theta$ is the network parameters and $\nabla$ denotes the gradient. The objective function can be re-written as: $$ \nabla_{\theta}f=\nabla_{\theta}\mathcal{L}(\mu,h(z))=\nabla_{z}\mathcal{L}(\mu,h(z))\nabla_{\theta}z, $$ where z denotes the logits that are predicted by the network, and $\hat{\mu} = h(z)$ denotes the composition of the normalization and soft-argmax functions. Here, we assume the gradient of the network is smooth and only analyze the composition function, i.e.: $$ \|\nabla_{z}\mathcal{L}(\mu,h(z+\Delta z))-\nabla_{z}\mathcal{L}(\mu,h(z))\|. $$ In the conventional integral regression, we have: $$ \nabla_{z}\mathcal{L}(\mu,h(z))=(x-\hat{\mu})\cdot p_{x}. $$ In this case, Eqn. (6) is equivalent to: $$ \|(x-\hat{\mu}-\Delta\hat{\mu})(p_{x}+\Delta p_{x})-(x-\hat{\mu})\cdot p_{x}\|. $$ Note that x can be a arbitrary position on the heatmap. Denoting the heatmap size as W, we have $\|x - \hat{\mu}\| \leq W$ over the whole dataset. Therefore, we derive the Lipschitz constant of integral regression as: $$ \begin{aligned}&\|\nabla_{z}\mathcal{L}(\mu,h(z+\Delta z))-\nabla_{z}\mathcal{L}(\mu,h(z))\|\\ \leq&\|W(p_{x}+\Delta p_{x})-W p_{x}\|=W\|\Delta p_{x}\|\\=&W\cdot L_{s}\cdot\|\Delta z\|,\end{aligned} $$ where $L_{s}$ is the Lipschitz constant of the normalization function [^58], [^59]. It shows that the conventional integral regression multiplies a factor W to the Lipschitz constant of normalization. Similarly, we can derive the Lipschitz constant of the proposed amplitude symmetric function. First, the gradient of the logits is: $$ \begin{align*}|\nabla_{z}\mathcal{L}(\mu,h(z))|&=|A_{grad}\cdot p_{x}\cdot\left(1+\sum_{x_{i}<\hat{\mu}}p_{x_{i}}-\sum_{x_{i}>\hat{\mu}}p_{x_{i}}\right)|\\&\leq2\cdot A_{grad}\cdot p_{x}.\end{align*} $$ We set $A_{grad} = W/8$ to make the average norm of the gradient the same as integral regression. Specifically, $$ E_{x}[|(x-\hat{\mu})p_{x}|]=E_{x}[|x-\hat{\mu}|]p_{x}=\frac{W}{4}\cdot p_{x}. $$ The Lipschitz constant of the proposed amplitude symmetric function derived as: It shows that the Lipschitz constant of the proposed method is 4-times smaller than the original integral regression when $A_{grad} = W/8$, which indicates that the gradient space is more smooth and the model can be optimized more easily. $$ \begin{aligned}&\|\nabla_{z}\mathcal{L}(\mu,h(z+\Delta z))-\nabla_{z}\mathcal{L}(\mu,h(z))\|\\ \leq&\|2A_{grad}(p_{x}+\Delta p_{x})-2A_{grad}p_{x}\|=\frac{W}{4}\|\Delta p_{x}\|\\ =&\frac{W}{4}\cdot L_{s}\cdot\|\Delta z\|.\end{aligned} $$ #### 3.1.2 Size-Dependent Keypoint Scoring Problem Before conducting soft-argmax, the element-sum of the predicted heatmaps should be normalized to one, i.e., $\sum p_x = 1$. Prior works [^45], [^46] adopt soft-max operation, which works well in single-person pose estimation but remains a large performance gap with the state-of-the-arts in multi-person pose estimation [^31], [^45], [^60]. This is because in multi-person cases, we need not only the joint locations, but also the joint confidence for pose NMS and calculating the mAP. In previous methods, the maximum value of the heatmap is taken as the joint confidence, which is size-dependent and not accurate. If we adopt the one-step normalization such as soft-max, the maximum value of the heatmap is inversely proportional to the scale of the distribution, which highly depends on the projected size of the body joint. Therefore, a large-size joint (e.g., left-hip) will generate a smaller confidence value than a small-size joint (e.g., nose), which harms the reliability of the predicted confidence values. Two-Step Heatmap Normalization. To decouple confidence prediction and integral regression, we propose a two-step heatmap normalization manner. In the first step, we perform element-wise normalization to generate the confidence heatmap C: where $z_x$ denotes the un-normalized logits value of location x, $c_x$ denotes the confidence heatmap value of location x. Hence, the joint confidence can be indicated by the maximum value of the heatmap: $$ c_{x}=\mathrm{sigmoid}(z_{x}), $$ Since we use an element-wise operation sigmoid for the first step of normalization and don't force the sum of C to be one, the maximum value of C won't be affected by the size of the joint. In this way, the predicted joint confidence is only related to the predicted location. In the second step, we perform global normalization to generate the probability heatmap P: $$ conf=\max(\mathbf{C}). $$ $$ p_{x}=\frac{c_{x}}{\sum\mathbf{C}}. $$ The element-sum of the probability heatmap P is one, which ensures the predicted joint location $\hat{\mu}$ is within the heatmap boundary and stabilizes the training process. To sum up, we obtain the joint confidence through the first step and obtain the joint location on the heatmap generated by the second step. An ablation study is carried out in August 16, 2026 at 10:27:07 UTC from IEEE Xplore. Restrictions apply. --- Section 6.6 to show the effectiveness of our normalization method. ### 3.2 Multi-Domain Knowledge Distillation Beyond our novel symmetric integral regression, the performance of network can further benefit from extra training data. Except for annotating a new dataset (detailed in Section 6.1), we also adopt multi-domain knowledge distillation to train our network. Three additional datasets are adopted, namely 300Wface [^61], FreiHand [^62] and InterHand [^63]. The details of these datasets will be introduced in Section 6.1. Combining these datasets, our network are able to predict face and hand keypoints accurately for in-the-wild images. During training, we construct each training batch by sampling different datasets with a fixed ratio. To be specific, 1/3 of the batch are sampled from our annotated dataset, 1/3 from the COCO-fullbody and the remaining are equally sampled from 300Wface and FreiHand. For each sample, we apply dataset-specific augmentation, which is introduced in the next section. Although these domain specific datasets are able to provide accurate intermediate supervision, their data distribution are quite different from in-the-wild images. To solve this problem, we extend our pose-guided proposal generator in [^20] to full body scenario and conduct data augmentation in a unified manner. ### 3.3 Part-Guided Proposal Generator For the two-stage pose estimation, the human proposals generated by the human detector usually produce a different data distribution from the ground-truth human boxes. Meanwhile, the spatial distribution of the face and hand are also different between the full-body images in the wild and the part-only images in the dataset. Without proper data augmentation during training, the pose estimator may not work properly in the testing phase for the detected human. To generate training samples with similar distribution to the output of the human detector, we propose our part-guided proposal generator. For different body parts with a tight surrounded bounding box, the proposal generator generates a new box that is inline with the distribution of the output of human detector. Since we already have the ground truth bounding box for each part, we simplify this problem into modeling the distribution of the relative offset between the detected bounding box and the ground truth bounding box varies across different parts. To be more specific, there exists a distribution $$ P(\delta x_{min},\delta x_{max},\delta y_{min},\delta y_{max}|p) $$ where $\delta x_{min}/\delta x_{max}$ is the normalized offset between the left-est/right-est coordinate of a bounding box generated by human detector and the coordinates of the ground truth bounding box: $$ \delta x_{min}=\frac{x^{detect}_{min}-x^{gt}_{min}}{x^{gt}_{max}-x^{gt}_{min}}, $$ $$ \delta x_{max}=\frac{x^{detect}_{max}-x^{gt}_{max}}{x^{gt}_{max}-x^{gt}_{min}}, $$ ![Image](images/e15ee5b6a36920a64372b6c47db542c846fa2a03080f899e995b41fea320bdfc.jpg)
Fig. 3. Distributions of bounding box offsets for several different body parts. The dotted boxes denote the ranges of approximated uniform distributions. Best viewed in color.
and similarly is $\delta y_{min}$, $\delta y_{max}$, p is the ground truth part type. If we can model this distribution, we are able to generate many training samples that are similar to human proposals generated by the human detector. To achieve that, we adopt an off-the-shelf object detector [^64] and generate human detection for our Halpe-FullBody dataset. For each instance in the dataset, we separate the annotations of face, body and hand. For each separated part, we calculate the offsets between its tightly surrounded bounding box and the detected bounding box of the whole person. Since the box variances in horizontal and vertical directions are usually independent, we simplify the modeling of original distribution into modeling $$ P_{x}(\delta x_{min},\delta x_{max}|p), $$ After processing all the instances in Halpe-FullBody, the offsets form a frequency distribution, and we fit the data to a Gaussian mixture distribution. For different body parts, we have different Gaussian mixture parameters. We visualize the distributions and their corresponding parts in Fig. 3. $$ P_{y}(\delta y_{m i n},\delta y_{m a x}|p). $$ During the training phase of the pose estimator, for a training sample belonging to part p, we can generate additional offsets to its ground-truth bounding box by dense sampling according to $P_x(\delta x_{min}, \delta x_{max} | p)$ and $P_y(\delta y_{min}, \delta y_{max} | p)$ to produce augmented training proposals. In practice, we found that sampling in an approximated uniform distribution (the dotted red boxes in the Fig. 3) can also produce similar performance. ### 3.4 Parametric Pose NMS For the top-down approaches, a main drawback is the early commitment problem: if the human detector fails to detect a person, there is no recourse for the pose estimator to recover it. Most top-down based methods [^28], [^30], [^31], [^65] would meet this problem since they set the detection confidence to a high value to avoid redundant poses. On the contrary, we set the detection confidence to a low value (0.1 in our experiments) to ensure a high detection recall. In this case, human detectors inevitably generate redundant detections for some people, which results in redundant pose estimations. Therefore, pose non-maximum suppression (NMS) is required to eliminate the redundancies. Previous methods [^11], [^66] are in August 16, 2026 at 10:27:07 UTC from IEEE Xolore. Restrictions apply. --- either not efficient or not accurate enough. In this paper, we propose a parametric pose NMS method. Similar to the previous subsection, the pose $P_i$, with m joints is denoted as $\{\langle k_i^1, c_i^1\rangle, \ldots, \langle k_i^m, c_i^m\rangle\}$, where $k_i^j$ and $c_i^j$ are the $j^{th}$ location and confidence score of joints respectively. NMS Scheme. We revisit pose NMS as follows: first, the most confident pose is selected as reference, and some poses close to it are subject to elimination by applying elimination criterion. This process is repeated on the remaining poses set until redundant poses are eliminated and only unique poses are reported. Elimination Criterion. We need to define pose similarity in order to eliminate the poses which are too close and too similar to each others. We define a pose distance metric $d(P_i, P_j|\Lambda)$ to measure the pose similarity, and a threshold $\eta$ as elimination criterion, where $\Lambda$ is a parameter set of function $d(\cdot)$. Our elimination criterion can be written as follows: $$ f( Ⅲ P_{i},P_{j}|\Lambda,\eta)=\mathbb{1}[d(P_{i},P_{j}|\Lambda,\lambda)\leq\eta] $$ If $d(\cdot)$ is smaller than $\eta$, the output of $f(\cdot)$ should be 1, which indicates that pose $P_i$ should be eliminated due to redundancy with reference pose $P_j$. Pose Distance. Now, we present the distance function $d_{pose}(P_i, P_j)$. We assume that the box for $P_i$ is $B_i$. Then we define a soft matching function $$ \begin{aligned}&K_{Sim}(P_{i},P_{j}|\sigma_{1})=\\&\left\{\begin{aligned}&\sum_{n}\tanh\frac{c_{i}^{n}}{\sigma_{1}}\cdot\tanh\frac{c_{j}^{n}}{\sigma_{1}},&&\text{if}k_{j}^{n}\text{is within}\mathcal{B}(k_{i}^{n})\\&0&&otherwise\end{aligned}\right.\end{aligned} $$ where $\mathcal{B}(k_{i}^{n})$ is a box center at $k_{i}^{n}$, and each dimension of $\mathcal{B}(k_{i}^{n})$ is 1/10 of the original box $B_{i}$. The tanh operation filters out poses with low-confidence scores. When two corresponding joints both have high confidence scores, the output will be close to 1. This distance softly counts the number of joints matching between poses. The spatial distance between parts is also considered, which can be written as $$ H_{S i m}(P_{i},P_{j}|\sigma_{2})=\sum_{n}\exp[-\frac{\left(k_{i}^{n}-k_{j}^{n}\right)^{2}}{\sigma_{2}}] $$ By combining Eqns. (17) and (18), the final distance function can be written as $$ d(P_{i},P_{j}|\Lambda)=K_{Sim}(P_{i},P_{j}|\sigma_{1})+\lambda H_{Sim}(P_{i},P_{j}|\sigma_{2}) $$ where $\lambda$ is a weight balancing the two distances and $\Lambda = \{\sigma_1, \sigma_2, \lambda\}$. Note that the previous pose NMS [^11] set pose distance parameters and thresholds manually. In contrast, our parameters can be determined in a data-driven manner. Optimization. Given the detected redundant poses, the four parameters in the eliminate criterion $f(P_i, P_j|\Lambda, \eta)$ are optimized to achieve the maximal mAP for the validation set. Since exhaustive search in a 4D space is intractable, we optimize two parameters at a time by fixing the other two parameters in an iterative manner. Once convergence is achieved, the parameters are fixed and will be used in the testing phase. ## 4 MULTI PERSON POSE TRACKING In this section, we introduce our multi-person pose tracking method shown in the middle row of Fig. 2. We attach a person re-ID branch on the pose estimator. Thus the network can estimate human pose and re-ID feature simultaneously. A Pose-Guided Attention Mechanism (PGA) is adopted to enhance the person identity feature. Finally, human proposal information (identity embedding, box and pose) are integrated by our designed Multi-Stage Identity Matching (MSIM) algorithm to achieve online real-time pose tracking. ### 4.1 Pose-Guided Attention Mechanism Person re-ID feature can be used to identify the same individual from a lot of human proposals. In our top-down framework, we extract re-ID feature from each bounding box produced by object detector. However, the quality of the re-ID feature will be reduced by the background in the bounding box, especially when there exists other people's bodies. In order to solve this problem, we consider using the predicted human pose to construct a region where the human body is concentrated. Thus, the Pose-Guided Attention (PGA) is proposed to force the extracted features focusing on the human body of interest, and ignore the impact of the background. The insight of PGA is elaborated in ablation studies (Section 6.8). The pose estimator generates k heatmaps where k means the number of keypoint for each person. Then the PGA module will transform these heatmaps to an attention map ($m_{A}$) with a simple conv layer. Note that the $m_{A}$ has same size with re-ID feature map ($m_{id}$). Therefore, we could obtain the weighted re-ID feature map ($m_{wid}$): $$ m_{w i d}=m_{i d}\odot m_{A}+m_{i d} $$ where $\odot$ means Hadamard product. Finally, the identity embedding $(emb_{id})$ which is a 128 dimension vector is encoded by a fully-connection layer. ### 4.2 Multi-Stage Identity Matching For a video sequence, Let $H_t^i$ denote the i-th human proposal of t-th frame. As described above, $H_t^i$ has several features: pose $(P_t^i)$, bbox $(B_t^i)$ and identity embedding $(E_t^i)$. Considering that all these features can determine the identity of a person, we design MSIM algorithm to assign the corresponding id for $H_t^i$. Assuming that the detection and tracking results of the previous t-1 frames have been obtained and stored in the tracking pool $P_t$. First, a kalman filter is used to finetune the detection features in the current frame thus make trajectories more smooth. Then we perform the first stage matching by computing the affinity matrix $M_{emb}^i$ among identity embedding of the t-th frame and all embeddings existed in $P_t$. The matching rules are as follows: $$ \left\{\begin{aligned}{}&{{}l i n k(p,q),}&{\mathrm{i f~}}&{{}M_{e m b}^{t}[p][q]=m i n(M_{e m b}^{t}[p])}\\ {}&{{}}&{\mathrm{a n d~}}&{{}M_{e m b}^{t}[p][q]\leq\mu_{e m b}}\\ {}&{{}H_{t}^{p}}&{\mathrm{k e e p~u n t r a c k e d},}&{{}\quad\mathrm{o t h e r w i s e}}\\ \end{aligned}\right. $$ where $\text{link}(p, q)$ means $H_t^p$ shares the same trajectory with the q-th human proposal in $P_t$. $\mu_{emb}$ is the threshold. Here we set $\mu_{emb}$ as 0.7 following [^67]. At the second stage, we consider both position and shape constraints for those untracked human proposals in (21). In August 16, 2026 at 10:27:07 UTC from IEEE Xplore. Restrictions apply. --- ![Image](images/f20f9f737145900f9d195fb90448e24819fde6aba9677c29cf511c9648a32012.jpg)
Fig. 4. System architecture of AlphaPose. Our system is divided into five modules, namely (a) data loading module that can take images, video or camera stream as input, (b) detection module that provides human proposals, (c) data transformation module to process the detection results and crop each single person for later modules, (d) pose estimation module that generates keypoints and/or human identity for each person, (e) post processing module that processes and saves the pose results. Our framework is flexible and each module contains several components that can be replaced and updated easily. Dashed box denotes optional components in each module. See text for more details and best viewed in color.
Specifically, We use IOU metric between bboxes as position constraint and normalized pose distance as shape constraint. For two human proposals $H_i^j$ and $H_{t-\delta}^j$, we first resize their bbox to same scale and get the center point c of each bbox. Then we compute the normalized pose vector by subtracting the center from each keypoint coordinates. Finally, we can obtain the normalized pose distance ( $dist_{np}$) by Eqn. (19). Therefore the fusion distance matrix of shape and location can be written as: $$ M_{f}^{t}=\left(1-I O U\right)+\lambda_{n p}\times d i s t_{n p} $$ Where IOU and $dist_{np}$ denote the matrix formed by IOU-function and normalized pose distance among untracked proposals and Pl. $\lambda_{np}$ is weight to balance location and shape distance matrix. Here we also use a threshold $\mu_{f}$ to filter unmatched proposals like Eqn. (21) and empirically set it as 0.5. In order to match the tracklets that are not very similar with previous frames, we appropriately lower the threshold and repeat the above stage. If there is still no matched proposal, we think that this is a newly tracklet, so a new id will be assigned to it. ### 4.3 Joint Training Strategy In order to simplify the training process of the whole network, we simultaneously train the pose estimator and the re-ID branch. Our network is trained on COCO $^{[14]}$ and PoseTrack $^{[68]}$ dataset. PoseTrack has both pose and identity annotation, while COCO only has pose annotation. Therefore, when training on COCO, the gradient contributed by the re-ID branch does not participate in back propagation. We follow the loss balanced strategy in [^69] to jointly optimize pose and identification sub-task. ## 5 ALPHAPOSE In this section, we present AlphaPose $^{2}$, the first jointly whole-body pose estimation and tracking system. ### 5.1 Pipeline A drawback of two-step framework is the limitation of the inference speed. To facilitate the fast processing of large-scale data, we design a five-stage pipeline with multi-processing implementation to speed up our inference. Fig. 4 illustrates our AlphaPose pipelining mechanism. We divide the whole inference process into five modules, following the principle that each module consumes similar processing time. During inference, each module is hosted by an independent process or thread. Each process communicates with subsequent processes with a First-In-First-Out queue, that is, it stores the computed results of current module and the following modules directly fetch the results from the queue. With such design, these modules are able to run in parallel, resulting in a significant speed up and enabling real-time application. ### 5.2 Network For our two-step framework, various human detector and pose estimator can be adopted. In the current implementation, we adopt off-the-shelf detectors include YOLOv3 [^64] and EfficientDet [^75] trained on COCO [^14] dataset. We do not retrained these models as their released model already work well in our case. For the pose estimator, we design a new backbone named FastPose, which yields both high accuracy and efficiency. The network structure is illustrated in Fig. 5. We use ResNet [^24] as our network backbone to extract features from the input cropped image. Three Dense Upsampling Convolution (DUC) [^76] modules are adopted to upsample the extracted features, followed by a $1 \times 1$ convolution layer to generate heatmaps. The DUC module first applies 2D convolution to the feature map with dimension $h \times w \times c$ and then reshapes it to $2h \times 2w \times c$ via a PixelShuffle [^77] operation. To further boost the performance, we also incorporate deformable convolution operator into our ResNet backbone following [^78] to improve the feature extraction. Such network is named as FastPose-DCN. --- ![Image](images/bc7b7161eaad48995b9feed4fb14dd977a1ae7e8c2e52f7fdadbc6f4bcd94c1f.jpg)
Fig. 5. Network architecture of FastPose. First, ResNet is adopted as the network backbone. Then, DUC modules are applied for up-sampling. Finally, a $1 \times 1$ convolution is utilized to generate heatmaps.
### 5.3 System AlphaPose is developed based on both PyTorch [^79] and MXNet [^80]. Benefiting from the flexibility of PyTorch, AlphaPose supports both Linux and Windows system. AlphaPose is highly optimized for the purpose of easy usage and further development, as we decompose the training and testing pipeline into different modules and one can easily replace or update different modules for custom purpose. For the data loading module, we support image input by specifying image name, directory or a path list. Video file or stream input from camera are also supported. For the detection module, we adopt YOLOX [^81] YOLOV3-SPP [^64], EfficientDet [^75] and JDE [^67]. Detecting results from other detectors are also supported as a file input. Other trackers like [^82] can also be incorporated. For the data transform module, we implement vanilla box NMS and soft-NMS [^83]. For the pose estimation module, we support SimplePose [^31], HRNet [^28], and our proposed FastPose with different variants like FastPose-DCN. Our Re-ID based tracking algorithm is also available in this module. For the post processing module, we provide our parametric pose NMS and the OKS-based NMS [^65]. Another tracker PoseFlow [^47] is available here and we support rendering for images and video. Our saving format is COCO format by default and can be compatible with OpenPose [^17]. One can easily run AlphaPose with different settings by simply specifying the input arguments. ## 6 DATASETS AND EVALUATIONS ### 6.1 Datasets Halpe-FullBody. To facilitate the development of whole body human pose estimation, we annotate a full body keypoints dataset named Halpe-FullBody $^{3}$. For each person, we annotate 136 keypoints, including 20 for body, 6 for feet, 42 for hands and 68 for face. The keypoint format is illustrated in Fig. 6. Note that since there are two popular definition for the face keypoints (see Fig. 7), we only annotate the visible lower jaw of the face (green dots in Fig. 7) so as to be compatible with ![Image](images/1a6fca9614a1b2f42ee56b42ef98155fb190a9c887ac8ab7ded493ad9b802d88.jpg) ![Image](images/f48197698f5b235da7c456f7733bad039be2f6082be0307ba741e7ee2fe71c19.jpg)
(a)
(b)
![Image](images/eb6b21227853c9a7cd4f631808694951428678fd3c9cba00552e030e08967b46.jpg)
(c)
Fig. 6. Annotated keypoint format in Halpe-FullBody dataset for (a) body and foot, (b) face, (c) hand respectively. Zoom in for details of the face annotation.
these two definition. For the images, our training set uses the training images of the HICO-DET [^40] dataset and our testing set uses the COCO-val set. In total, our dataset contains 50K instances for training and 5K images for testing. Table 1 compares our dataset with previous popular datasets on human pose estimation. COCO-WholeBody. As a concurrent work, Jin et al. annotates 133 whole body keypoints based on the COCO dataset. They share a similar keypoints definition with us, except that the head, neck and hip points are missing in their annotation. The total training set contains 118K images with 250K instances, and the test set contains 5K images. We also evaluate our algorithm on this dataset. COCO COCO dataset is a standard benchmark for human keypoints prediction. It contains 17 keypoints of human body without face, hand and foot annotations. In total, there are 118K images for training, 5K for validation and 41K for testing. We train our algorithm on the COCO 2017 train set and compare our FastPose network and ![Image](images/882bcc63b284fbafe7ae643623d05eb82b0ca9efe9dd016cda0b8ffb30882587.jpg)
(a)
![Image](images/3b23d0e781a161b53c4d734cf5e9dfe9883376169efd3d986eea8cb565a11ae3.jpg)
(b)
Fig. 7. Two different definition of face keypoints on the lower jaw. The green dots represent the same definition and the red dots indicate their differences. In (a) and (b), the left definition is commonly used in 2D annotated dataset like [^18], [^61], [^72], while the right definition is used in 3D face alignment task like [^84].
---
TABLE 1
Overview of Some Popular Public Datasets for 2D Keypoint Estimation in RGB Images
DataSet#KptWildBodyHandFaceHOITotal
KptKptKptInstances
MPII [^15]16✓✓40K
CrowdPose [^19]14✓✓80K
PoseTrack [^68]15✓✓150K
COCO [^14]17✓✓250K
OneHand10K [^70]21✓✓10K
FreiHand [^62]21✓130K
MHP [^71]21✓80K
WFLW [^72]98✓✓10K
AFLW [^73]19✓✓25K
COFW [^74]29✓✓1852
300W [^61]68✓✓3837
COCO-WholeBody [^18]133✓✓✓✓250K
Halpe-FullBody136✓✓✓✓✓50K
Kpt stands for keypoints, and #Kpt means the annotated number. "Wild" denotes whether the dataset is collected in-the-wild. "HOI" denotes human-object-interaction body-part labels. symmetrical integral loss with previous state-of-the-arts models on the COCO 2017 test-dev set. PoseTrack. PoseTrack is a large scale dataset for multi-person pose estimation and tracking. It is built on the raw videos provided by MPII Human dataset $^{[15]}$. There are more than 1356 video sequences of PoseTrack and they are split into train, val, test. Each annotated person has 17 keypoints similar with COCO, but there are two different keypoints compared with COCO, which are 'top head' and 'bottom head'. Other annotations share the same format with COCO. We train our method on PoseTrack-2018 set and compare it with previous methods on both PoseTrack-2017-val and PoseTrack-2018-val sets. 300Wface, FreiHand and InterHand are used as supplemental datasets to improve the generalization ability of our model. 300Wface [^61] contains 300 indoor and 300 outdoor in-the-wild images. For each face, 68 keypoints are annotated. FreiHand [^62] contains 33K unique hand samples for training, each contains 21 keypoints. InterHand [^63] contains 2.6M images with interacting hands, where each hand also has 21 keypoints. ### 6.2 Evaluation Metrics and Tools Halpe-FullBody. We extend the evaluation metric of the COCO keypoints to full body scenario. COCO defines a Object Keypoint Similarity controlled by a per-keypoint constant k. For our newly added keypoints, we set the k for the feet, face and hand as 0.015. Same as COCO, we report $AP^{0.5:0.95:0.05}$ as the main result and the detailed results for body, foot, face and hand are also reported. COCO-WholeBody. COCO-WholeBody adopts the same metric as ours except that the constant k is different from us for some keypoints. COCO. We adopt the standard AP metric of COCO dataset for fair comparison with previous works. PoseTrack. In fact, multi-person pose tracking can be regarded as the combination of multi-person pose estimation and multi-object tracking. Thus, the evaluation metric should follow these two tasks. Mean Average Precision (mAP) $^{[12]}$ is used to measure frame-wise human pose accuracy. To evaluate the tracking performance, the MOT $^{[85]}$ metric is applied to each of the body joints independently. Then the final tracking performance is obtained by averaging each joint mot metric. The PCKh $^{[15]}$ (head-normalized probability of correct keypoint) is one of the most commonly used metric to evaluate whether a body joint is predicted correctly. Here it can determine which predicted joint is matched with groundtruth joint. To evaluate the tracking result on posetrack validation dataset, we use the official tool named poseval $^{4}$ and report Multiple Object Tracker Accuracy (MOTA), Multiple Object Tracker Precision (MOTP), Precision and Recall. ### 6.3 Implementation Details We conduct our experiments with PyTorch [^79]. We train the network with batch 32 for 270 epochs. The initial learning rate is 0.01 and we decay it on epoch 100 and 170 by 0.1. The pose guided proposal generator is applied after epoch 200. After the entire network is trained, we freeze the backbone and only finetune the re-ID branch on posetrack dataset for 10 epochs. Learning rate in the fine-tuning phase is 1e-4. We adopt Adam [^86] optimizer during training. All experiments are conducted on 8 Nvidia 2080Ti GPUs. ### 6.4 Evaluation for Full Body Pose Estimation We first evaluate the performance of our model on Halpe-FullBody and COCO-WholeBody dataset. Since Halpe-FullBody is a new dataset, we retrain several state-of-the-art models and compare the results with us. Table 2 gives the final results. YOLOV3 is adopted as human detector for all the top-down based models. We can see that top-down methods can achieve higher accuracy compared to the bottom-up methods. However, due to the quantization error introduced by heatmap, conventional SPPEs decrease a lot on the fine-level body parts like face and hand. Equipped ---
TABLE 2
Whole-Body Pose Estimation Results on Halpe-FullBody Dataset
MethodInput Sizefull-bodyfootfacehandbody
APAP $^{{50}}$AP $^{{75}}$AP $^{{L}}$AP $^{{M}}$ARAPARAPARAPARAPAR
OpenPose-default [^17]N/A0.2760.5280.2580.3560.3100.3700.4380.6520.4820.4950.1400.2090.5140.575
OpenPose-maxacc [^17]N/A0.2810.5310.2650.3630.3180.3810.4560.6770.4820.4960.1420.2110.5260.590
SN [^39]N/A0.2330.6060.1280.2110.3540.3620.4810.6800.3440.4190.0300.0710.5630.624
HRNet [^28]256×1920.3870.7820.3460.3930.4320.5220.5810.7490.4290.5580.1040.2040.6050.713
Simple [^31]256×1920.4090.7820.3910.4170.4350.5060.7060.7820.4440.5360.1410.2330.6480.691
ZoomNet384×2880.4270.8030.4120.4460.4330.5130.7020.7780.5050.5690.1360.2100.6480.699
FastPose50-hm256×1920.4170.7840.4060.4260.4390.5160.7300.8030.4320.5360.1630.2580.6580.701
FastPose50-si256×1920.4410.7720.4440.4700.4460.5320.7060.7810.4910.5800.2070.2940.6500.699
FastPose152-si256×1920.4510.7850.4570.4750.4600.5370.7240.7910.5080.5900.1990.2940.6510.699
FastPose50-dcn-si256×1920.4620.7950.4770.4910.4640.5480.7390.8100.5080.5890.2140.3010.6720.717
FastPose50-dcn-si $^{{*}}$256×1920.4840.8260.5050.4970.5080.5650.7330.8100.5370.5960.2260.3300.6780.721
For fair comparisons, results are obtained using single-scale testing. “OpenPose-default” and “OpenPose-maxacc” denotes its default and maximum accuracy configuration respectively. “hm” denotes the network uses heatmap based localization, “si” denotes the network uses our symmetric integral regression. “*” denotes model trained with multi-domain knowledge distillation and PGPG. FastPose50 denotes our FastPose network with ResNet50 as backbone and so is FastPose152. “dcn” denotes that the deformable convolutional layer [^78] is adopted in the ResNet backbone. by our novel symmetrical integral loss function, our FastPose models achieve the best accuracy. Notably, we can see that FastPose50-si yields 2.4 mAP (5.7% relatively) higher than its heatmap-based counterpart. The improvements mainly come from the face and hands. It demonstrates that the quantization error of heatmap affects the fine-level localization of face and hand keypoints, and our symmetric integral regression works well on such cases. On the COCO-WholeBody dataset, our FastPose embedded with symmetrical integral loss function also outperforms previous state-of-the-art methods by a large margin, especially on the face and hands. Notably, our FastPose achieves the highest accuracy given a smaller input size. The model complexity is also much lower than previous methods. It demonstrates the superiority of our network structure and the novel loss. Some qualitative results of full body pose estimation is shown in Fig. 8. ### 6.5 Evaluation for Conventional Body Pose Estimation We also conduct experiments on the conventional body-only pose estimation task to demonstrate the effectiveness of our method, although it is not our main focus. We train our models on the COCO dataset and evaluate it on COCO test-dev set. The results are reported in Table 4. For the heatmap based methods, we can see that our FastPose backbone can achieve on par performance with the state-of-the-art method, given a smaller input size and weaker human detector. It demonstrates the superiority of our FastPose network. Note that since our goal is to present a new baseline model like SimplePose [^31], we conduct these experiments to prove the accuracy and efficiency of our model. Further pursuing higher accuracy with speed and resources trade-off is not our goal in this paper and we leave them for future research. For the regression based methods, our method achieves the state-of-the-art performance with the lowest GFLOPS.
TABLE 3
Whole-Body Pose Estimation Results on COCO-WholeBody Dataset
TypeMethodInput SizeGFLOPswhole-bodybodyfootfacehand
APARAPARAPARAPARAPAR
Bottom-UpOpenPose [^17]N/AN/A0.3380.4490.5630.6120.5320.6450.4820.6260.1980.342
SN [^39]N/AN/A0.1610.2090.2800.3360.1210.2770.3820.4400.1380.336
PAF [^25]N/AN/A0.1410.1850.2660.3280.1000.2570.3090.3620.1330.321
PAF-body [^25]N/AN/A--0.4090.470------
AE [^26]N/AN/A0.2740.3500.4050.4640.0770.1600.4770.5800.3410.435
AE-body [^26]N/AN/A--0.5820.634------
HRNet [^28]384\times28816.00.4320.5200.6590.7090.3140.4240.5230.5820.3000.363
HRNet-body [^28]384\times28816.0--0.7580.809------
Top-DownZoomNet384\times28820.00.5410.6580.7430.8020.7980.8690.6230.7010.4010.498
FastPose50-si256\times1925.90.5540.6250.6730.7170.6360.7180.7570.8180.4250.515
FastPose152-si256\times19213.20.5690.6410.6840.7300.6720.7500.7650.8240.4430.532
FastPose50-dcn-si256\times1926.10.5770.6500.6930.7400.6900.7650.7590.8200.4530.538
For fair comparisons, results are obtained using single-scale testing. We only report the input size and GFLOPS of the pose model in top-down based approaches and ignore the detection model. “hm” denotes the network uses heatmap based localization, “si” denotes the network uses our symmetric integral regression. FastPose50 denotes our FastPose network with ResNet50 as backbone and so is FastPose152. “dcn” denotes that the deformable convolutional layer [^78] is adopted in the ResNet backbone. Authorized licensed use limited to: ShanghaiTech University. Downloaded on August 16, 2026 at 10:27:07 UTC from IEEE Xplore. Restrictions apply. --- ![Image](images/c1920e5463a2d9b6595f5ab64ba74abb12c6dbdf5e51f1333084c312f3466a11.jpg) ![Image](images/f33919083238b37f2a1126e1eaa433ef0383fe3a1cb0783f7e21a0476958ad24.jpg) ![Image](images/222fe34749f2579484b0534ab148036b966f5551a4111568aac1a0044564c9ec.jpg) ![Image](images/87df99bdeaed723cc021f39cb8cc38728d14138ebfa677975a97f65ab93a3a71.jpg) ![Image](images/c04e5351a7f907b9da501d1513b0d756e8ce0f87b5fb53f3f5ca33ae4daa7117.jpg) ![Image](images/d35c2b5e1d90637dc9f6ab0d4f6d024477a34e4ae9bab6557c8d8774daef9aed.jpg) ![Image](images/b5e7f039a1f54b37531f06a09b1104624f629d67fe1b4188ba6e4f876f18b8ef.jpg) ![Image](images/a0f8bb14d05af710422ce432f705767ed70d80e45f3d19a06c5804b0fa2e6b64.jpg) ![Image](images/b68bfb9ffc1e9426704b66dcc6de939b964636054ec214de1d4f114d4760df33.jpg) ![Image](images/c2e8e14e7ba4eada676acee462d551ece677624744f3fa7a85f452156cf219fc.jpg) ![Image](images/00a81b5798b17fe638fc105b984406c03201e46a1a782618764136a69bfeb64a.jpg)
Fig. 8. Qualitative results of AlphaPose on the full-body pose estimation task. Zoom in for more details and best viewed in color.
TABLE 4
Body Pose Estimation Results on COCO Test-Dev Set
MethodsBackboneDetectorInput SizeGFLOPsAPAP $^{{50}}$AP $^{{75}}$AP $^{{M}}$AP $^{{L}}$AR
DetectionG-MRI [^65]ResNet-101Faster-RCNN353 \times 25757.00.6490.8550.7130.6230.7000.697
RMPE [^20]PyraNetFaster-RCNN320 \times 25626.70.7230.8920.7910.6800.786-
CPN [^25]ResNet-InceptionFPN384 \times 288-0.7210.9140.8000.6870.7720.785
PAF-body [^25]----0.6180.8490.6750.5710.6820.665
AE [^26]----0.6550.8680.7230.6060.7260.702
SimplePose [^31]ResNet-50Faster-RCNN256 \times 1928.90.7020.9090.7830.6710.7590.758
HRNet [^28]HRNet-32Faster-RCNN384 \times 28816.00.7490.9250.8280.7130.8090.801
HRNet [^28]HRNet-48Faster-RCNN384 \times 28832.90.7550.9250.8330.7190.8150.805
FastPose-hmResNet-50YOLO-v3256 \times 1925.90.7180.9190.8030.7280.7420.773
FastPose-dcn-hmResNet-50YOLO-v3256 \times 1926.10.7260.9220.8120.7370.7490.781
FastPose-dcn-hmResNet-101YOLO-v3256 \times 1929.80.7270.9220.8130.7360.7510.781
RegressionIntegral [^45]ResNet-101Faster-RCNN256 \times 25617.80.6780.8820.7480.6390.740-
CenterNet [^35]Hourglass-2 stacked---0.6300.8680.6960.5890.704-
SPM [^36]Hourglass-8 stacked---0.6690.8850.7290.6260.731-
Point-set Anchor [^38]HRNet-W48---0.6870.8990.7630.6480.753-
FastPose-siResNet-50YOLO-v3256 \times 2567.90.6490.8650.7280.6690.6630.716
FastPose-siResNet-101YOLO-v3256 \times 25612.80.6790.8760.7510.6750.7140.723
FastPose-dcn-siResNet-101YOLO-v3256 \times 25613.10.6900.9010.7730.7290.6900.775
For fair comparisons, results are obtained using single-scale testing. “hm” denotes the network uses heatmap based localization, “si” denotes the network uses our symmetric integral regression. Authorized licensed use limited to: Shanghai Tech University. Downloaded on August 16, 2026 at 10:27:07 UTC from IEEE Xplore. Restrictions apply. ---
TABLE 5
Ablation Studies on Halpe Fullbody Dataset and COCO Dataset
ModuleHalpe Fullbody (mAP)COCO (mAP)
two-step hm-norm44.169.5
one-step hm-norm38.167.1
w. SIKR44.169.5
w.o SIKR42.364.6
w. P-NMS44.169.5
w.o P-NMS43.768.2
w. PGPG48.4 $^{*}$N/A
w.o PPGG47.1 $^{*}$N/A
“hm-norm” denotes heatmap normalization. “*” denotes results trained with additional data from Multi-Domain Knowledge Distillation. Compared to [^45], our network serves as a new baseline for future research. ### 6.6 Ablation Studies for Pose Estimation To evaluate the effectiveness of our proposed module for pose estimation, we also conducted ablation experiments on COCO and Halpe-Fullbody dataset. We adopt FastPose50 as base network and report the numbers on COCO validation and Halpe-Fullbody test set respectively. The results are summarized in Table 5. Heatmap Normalization. We elucidated the essence of our two-step heatmap normalization for applying the integral-based method in the multi-person scenario in Section 3.1. Here we conduct an ablation experiment to show the performance gap of different heatmap normalization methods. We can see that when comparing the conventional one-step heatmap normalization (soft-max) to our two-step heatmap normalization, the performance in multi-person pose estimation decrease for 6 mAP and 2.4 mAP on Halpe-fullbody and COCO datasets, respectively. It demonstrates that the two-step normalization can alleviate the size-dependent effect and improve performance. SIKR Module. We compare our symmetric integral function with the original integral regression [^45]. For both full body pose estimation scenario and conventional body pose estimation, we can see that our symmetric integral function greatly outperforms the original integral regression. Pose NMS Module. Without Pose-NMS, multiple human poses will be predicted for a single person. The redundant poses will decrease the model performance. From Table 5, we can see that our model decreases for 0.4 mAP and 1.3 mAP for Halpe Fullbody and COCO dataset respectively. PGPG Module. Proper data augmentation is needed during training to ensure the generalization ability at testing phase. For Halpe Fullbody dataset, we compare the results of FastPose50-dcn trained with and without PGPG module. Table 5 shows that without our part guided proposal generation, the performance would decrease due to the domain variance in training. ### 6.7 Evaluation for Pose Tracking To verify that our system is sufficient for multi-person pose tracking task, we apply it to the posetrack validation dataset. Table 6 shows the comparison with other state-of-the-art methods. The backbone we adopted is the FastPose152 and
TABLE 6
Evaluation Result on Posetrack Validation Dataset
DataMethodmAPMOTAfpsResSro
2017Det&Track[^49]60.655.21.2-✓
PoseFlow[^47]66.558.3$10^{{*}}$-✓
JointFlow[^87]69.359.80.2N/A✗
Fast[^57]70.363.212.2N/A✗
TML++[^88]71.561.3--✗
STAF[^55]72.662.73.0N/A✓
FlowTrack[^31]76.765.43.0$384 \times 288$✓
PGPT[^52]77.268.41.2$384 \times 288$✓
Yang et.al.[^53]81.173.4-$384 \times 288$✗
Ours-UNI76.165.511.3$256 \times 192$✓
Ours-SEP76.965.78.9$256 \times 192$✓
2018MDPN[^51]71.750.6-$384 \times 288$✗
STAF[^55]70.460.93.0N/A✓
OpenSVAI[^89]69.762.4--✗
LightTrack[^48]71.264.60.7$384 \times 288$✓
KeyTrack[^54]74.366.61.0$384 \times 288$✗
PGPT[^52]76.867.11.2$384 \times 288$✓
Yang et.al.[^53]77.969.2-$384 \times 288$✗
Ours-UNI74.064.410.9$256 \times 192$✓
Ours-SEP74.764.78.7$256 \times 192$✓
“Res” denotes input resolution of pose network and “Src” denotes whether source code is available. “Ours-UNI” denotes results trained with a shared backbone of pose and re-ID branch and “Ours-SEP” denotes results trained with separated backbones. The “*” in fps means not including detection time. The mAP value is obtained after tracking post-precessing. the detector is YoloX. We can see that our model outperforms most methods in both mAP metric and MOTA metric, and our speed is quite fast. This near real-time processing speed can be applied to various scenarios in our real life. It is worth noting that there are some other methods $^{[49]}$, $^{[50]}$ that have achieved good results on the posetrack dataset, but they mainly consider the overall timing information of the video, which means that they are not strictly an online algorithm. Therefore our method is not directly compared with theirs. $^{[52]}$, $^{[53]}$, $^{[54]}$ achieve higher accuracy compared with our results. But they use very high resolutions for input and
TABLE 7
The Ablation Study Results of Proposed Pose Tracking Method
expSettingheadshouelbhipkneeTotal
PGAmAP
w/77.775.475.369.068.174.7
w/o78.075.675.569.368.674.9
MOTA
w/74.072.664.666.858.964.7
w/o73.471.864.266.158.463.5
MSIMmAP
No-GT77.775.475.369.068.174.7
GT-Box81.381.081.580.881.581.3
GT-Pose100.0100.0100.0100.0100.0100.0
MOTA
No-GT74.072.664.666.858.964.7
GT-Box75.875.575.875.476.775.9
GT-Pose93.893.693.793.894.493.9
--- ![Image](images/a33a4de40b3fbcb2ed0005bfa3d2aacc45e09976fee9b3f68cf8df44facdd353.jpg)
Fig. 9. Qualitative results of AlphaPose on the full-body pose tracking task. Zoom in for more details and best viewed in color. The colors of persons denote their tracking ID. The image order is denoted by the time arrow. See text for more analysis.
output, which consumes a lot of memory and is computationally expensive. Our method achieves satisfactory accuracy while running efficiently. ### 6.8 Ablation Studies for Pose Tracking In order to verify the effectiveness of each part of the tracking algorithm, we have designed several sets of ablation experiments. ![Image](images/86381a13f86be60d743918a722973578f9232083d7d0d864fecf02f750c2aac0.jpg)
Fig. 10. Visualization of the role of PGA module. When there is no PGA module, some background area will also have a high response. The results of adding PGA module show that the feature response is more concentrated on the target person. Notably, from figure (b) we can see that when two people are close, the feature response focus on the target person with the aid of PGA (zoom in for more details).
PGA Module. The function of PGA module is to assist in extracting more effective re-ID features with the help of the keypoint information. As a comparison, we remove the PGA module in our framework, which means the human pose and re-ID feature are fed into MSIM directly. Tests on PoseTrack dataset show that tracking performance will decrease after removing PGA module which is reported in Table 7. At the same time, we visualized the extracted re-ID features with or without PGA module shown as Fig. 10. Since the detection result is usually a larger box than the original size of the human, the background in the box has a large proportion. However, the background information makes the human identity embedding carry useless features. This intuitively explains that the advantage of PGA is that it can better focus attention on the target person's area. MSIM. To further verify the performance of our model, we add different level information into the Network. Specifically, we set up several sets of experiments, respectively using GT box, GT pose. These results are reported in Table 7. The results show that if we replace the human detector and pose estimator with more accurate network, our tracking performance will be further improved. ## 7 FULL BODY POSE TRACKING In the above sections, we demonstrate the effectiveness of our methods on both full body pose estimation and pose tracking. Since our tracking algorithm is general, it is also applicable to the whole-body scenario. We adopt a weakly supervised strategy by training on both PoseTrack dataset and Halpe-FullBody dataset. Some qualitative results of full body pose tracking is shown in Fig. 9. We can see that both full body pose estimation and pose tracking yield high accuracy given the heavily crowded scene. And our method is insensitive to the size variance of humans. Specifically, when a person is occluded by others and re-appear, our method still gives the correct identity (e.g., the person with black shorts on the right). ## 8 LIBRARY ANALYSIS In this section, we compare our AlphaPose library with other popular open source library in both pose estimation in August 16, 2026 at 10:27:07 UTC from IEEE Xplore. Restrictions apply. --- ![Image](images/86e457922c720cd53fe014ced7b8362428e89c38cd47ec12dde74882fc3e2987.jpg)
(a)
![Image](images/03ae3aa1d927d6a6148f167cd32e1bf8bb42e48e2b563b1cb8bce0162bba523d.jpg)
(b)
Fig. 11. Speed/Accuracy comparison of different pose estimation and tracking libraries. (a) Pose estimation results obtained on COCO-Whole-Body validation set and COCO validation set. (b) Pose tracking results obtained on PoseTrack18-val set.
and pose tracking. The results are obtained on a single Nvidia 20080Ti GPU. Fig. 11 shows the speed-accuracy curve of different libraries. From Fig. 11a we can see that our method has the highest accuracy and yields the highest efficiency on whole-body and body-only pose estimation. Although a drawback of our top-down based approach is that the running time would increase as the persons in the scene increase, our parallel processing pipeline greatly redeem this deficiency. According to the statistics by OpenPose [^17], our library is more efficient than it when there are less than 20 persons in the scene. From Fig. 11b we can see that our pose tracking achieves on-par performance with the state-of-the-art library while running with high efficiency. ## 9 CONCLUSION In this paper, we propose a unified and realtime framework for multi-person fullbody pose estimation and tracking. To the best of our knowledge, it is the first framework that serves this purpose. Several novel techniques are presented to achieve this goal and we demonstrate superior performance in both efficacy and efficiency. A new dataset that contains full body keypoints (136 keypoints for each person) is annotated to facilitate the research in this area. We also present a standard library that is highly optimized for easy usage and hope that it can benefit our community. For our future research, we will also include 3D keypoints and mesh to our library. ## ACKNOWLEDGMENTS version of AlphaPose. Hao-Shu Fang would like to thank the support from Baidu, MSRA and ByteDance Fellowship. Hao-Shu Fang and Jiefeng Li are contribute equally to the manuscript. Chao Xu and Haoyi Zhu are contribute equally. We appreciate Chenxi Wang for help developing the MXNet version and Yang Han for developing the Jittor ## REFERENCES [^1]: K. Wang, R. Zhao, and Q. Ji, "Human computer interaction with head pose, eye gaze and body gestures," in Proc. IEEE 13th Int. Conf. Autom. Face Gesture Recognit., 2018, pp. 789–789. [^2]: T. B. Moeslund, A. Hilton, and V. Krüger, “A survey of advances in vision-based human motion capture and analysis,” Comput. Vis. Image Understanding, vol. 104, no. 2/3, pp. 90–126, 2006. [^3]: B. Pang, K. Zha, and C. Lu, “Human action adverb recognition: ADHA dataset and a three-stream hybrid model,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. Workshops, 2018, pp. 2388–238809. [^4]: B. Sapp, A. Toshev, and B. Taskar, "Cascaded models for articulated pose estimation," in Proc. Eur. Conf. Comput. Vis., 2010, pp. 406–420. [^5]: M. Sun, P. Kohli, and J. Shotton, "Conditional regression forests for human pose estimation," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2012, pp. 3394–3401. [^6]: L. Ladicky, P. H. Torr, and A. Zisserman, "Human pose estimation using a joint pixel-wise and part-wise formulation," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2013, pp. 3578–3585. [^7]: A. Newell, K. Yang, and J. Deng, "Stacked hourglass networks for human pose estimation," 2016, arXiv:1603.06937. [^8]: S.-E. Wei, V. Ramakrishna, T. Kanade, and Y. Sheikh, “Convolutional pose machines,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2016, pp. 4724–4732. [^9]: L. Pishchulin, A. Jain, M. Andriluka, T. Thormählen, and B. Schiele, "Articulated people detection and pose estimation: Reshaping the future," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2012, pp. 3178–3185. [^10]: G. Gkioxari, B. Hariharan, R. Girshick, and J. Malik, "Using k-poselets for detecting people and localizing their keypoints," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2014, pp. 3582–3589. [^11]: X. Chen and A. L. Yuille, "Parsing occluded people by flexible compositions," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2015, pp. 3945–3954. [^12]: L. Pishchulin et al., "DeepCut: Joint subset partition and labeling for multi person pose estimation," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2016, pp. 4929–4937. [^13]: E. Insafutdinov, L. Pishchulin, B. Andres, M. Andriluka, and B. Schiele, "DeeperCut: A deeper, stronger, and faster multi-person pose estimation model," in Proc. Eur. Conf. Comput. Vis., 2016, pp. 34–50. [^14]: 2016. [Online]. Available: [^15]: M. Andriluka, L. Pishchulin, P. Gehler, and B. Schiele, “2D human pose estimation: New benchmark and state of the art analysis,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2014, pp. 3686–3693. [^16]: J. J. Tompson, A. Jain, Y. LeCun, and C. Bregler, "Joint training of a convolutional network and a graphical model for human pose estimation," in Proc. Conf. Neural Inf. Process. Syst., 2014, pp. 1799–1807. [^17]: Z. Cao, G. Hidalgo, T. Simon, S.-E. Wei, and Y. Sheikh, "OpenPose: Realtime multi-person 2D pose estimation using part affinity fields," 2018, arXiv:1812.08008. [^18]: S. Jin et al., “Whole-body human pose estimation in the wild,” in Proc. Eur. Conf. Comput. Vis., 2020, pp. 196–214. [^19]: J. Li, C. Wang, H. Zhu, Y. Mao, H.-S. Fang, and C. Lu, "CrowdPose: Efficient crowded scenes pose estimation and a new benchmark," in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognition., 2019, pp. 10863–10872. [^20]: H.-S. Fang, S. Xie, Y.-W. Tai, and C. Lu, "RMPE: Regional multi-person pose estimation," in Proc. IEEE Int. Conf. Comput. Vis., 2017, pp. 2334–2343. [^21]: J. G. Umar Iqbal, "Multi-person pose estimation with local joint-to-person associations," in Proc. Eur. Conf. Comput. Vis. Workshops, 2016, pp. 627–642. [^22]: S. Kreiss, L. Bertoni, and A. Alahi, "PifPaf: Composite fields for human pose estimation," in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2019, pp. 11 977–11 986. --- [^23]: K. Sven, B. Lorenzo, and A. Alexandre, "OpenPifPaf: Composite fields for semantic keypoint detection and spatio-temporal association," IEEE Trans. Intell. Transp. Syst., vol. 23, no. 8, pp. 13498–13511, Aug. 2022. [^24]: K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition," in Proc. IEEE Conf. Comput. Vis. Pattern Recognition., 2016, pp. 770–778. [^25]: Z. Cao, T. Simon, S.-E. Wei, and Y. Sheikh, "Realtime multi-person 2D pose estimation using part affinity fields," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2017, pp. 1302–1310. [^26]: A. Newell, Z. Huang, and J. Deng, “Associative embedding: End-to-end learning for joint detection and grouping,” in Proc. Int. Conf. Neural Inf. Process. Syst., 2017, pp. 2274–2284. [^27]: B. Cheng, X. Bin, W. Jingdong, S. Honghui, S. H. Thomas, and Z. Lei, "HigherHRNet: Scale-aware representation learning for bottom-up human pose estimation," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2020, pp. 5386–5395. [^28]: K. Sun, B. Xiao, D. Liu, and J. Wang, "Deep high-resolution representation learning for human pose estimation," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2019, pp. 5693–5703. [^29]: K. He, G. Gkioxari, P. Dollár, and R. Girshick, "Mask R-CNN," in Proc. IEEE Int. Conf. Comput. Vis., 2017, pp. 2980–2988. [^30]: Y. Chen, Z. Wang, Y. Peng, Z. Zhang, G. Yu, and J. Sun, "Cascaded pyramid network for multi-person pose estimation," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2018, pp. 7103–7112. [^31]: B. Xiao, H. Wu, and Y. Wei, "Simple baselines for human pose estimation and tracking," in Proc. Eur. Conf. Comput. Vis., 2018, pp. 466–481. [^32]: S. Ren, K. He, R. Girshick, and J. Sun, "Faster R-CNN: Towards real-time object detection with region proposal networks," in Proc. Int. Conf. Neural Inf. Process. Syst., 2015, pp. 91–99. [^33]: A. Benzine, F. Chabot, B. Luvison, Q. C. Pham, and C. Achard, "PandaNet: Anchor-based single-shot multi-person 3D pose estimation," in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2020, pp. 6856–6865. [^34]: G. Bertasius, C. Feichtenhofer, D. Tran, J. Shi, and L. Torresani, “Learning temporal pose estimation from sparsely-labeled videos,” in Proc. Int. Conf. Neural Inf. Process. Syst., 2019, pp. 3021–3032. [^35]: Z. Xingyi, W. Dequan, and K. Philipp, "Objects as points," 2019, arXiv:1904.07850. [^36]: X. Nie, J. Feng, J. Zhang, and S. Yan, "Single-stage multi-person pose machines," in Proc. IEEE Int. Conf. Comput. Vis., 2019, pp. 6951–6960. [^37]: Z. Tian, H. Chen, and C. Shen, "DirectPose: Direct end-to-end multi-person pose estimation," 2019, arXiv:1911.07451. [^38]: F. Wei, X. Sun, H. Li, J. Wang, and S. Lin, "Point-set anchors for object detection, instance segmentation and pose estimation," in Proc. Eur. Conf. Comput. Vis., 2020, pp. 527–544. [^39]: G. Hidalgo et al., “Single-network whole-body pose estimation,” in Proc. IEEE/CVF Int. Conf. Comput. Vis., 2019, pp. 6982–6991. [^40]: Y.-W. Chao, Y. Liu, X. Liu, H. Zeng, and J. Deng, "Learning to detect human-object interactions," in Proc. IEEE Winter Conf. Appl. Comput. Vis., 2018, pp. 381–389. [^41]: C. Finn, X. Y. Tan, Y. Duan, T. Darrell, S. Levine, and P. Abbeel, "Learning visual feature spaces for robotic manipulation with deep spatial autoencoders," 2015, arXiv:1509.06113. [^42]: K. M. Yi, E. Trulls, V. Lepetit, and P. Fua, "LIFT: Learned invariant feature transform," in Proc. Eur. Conf. Comput. Vis., 2016, pp. 467–483. [^43]: D. C. Luvizon, D. Picard, and H. Tabia, "2D/3D pose estimation and action recognition using multitask deep learning," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2018, pp. 5137–5146. [^44]: A. Nibali, Z. He, S. Morgan, and L. Prendergast, "3D human pose estimation with 2D marginal heatmaps," in Proc. IEEE Winter Conf. Appl. Comput. Vis., 2019, pp. 1477–1485. [^45]: X. Sun, B. Xiao, F. Wei, S. Liang, and Y. Wei, “Integral human pose regression,” in Proc. Eur. Conf. Comput. Vis., 2018, pp. 536–553. [^46]: D. C. Luvizon, H. Tabia, and D. Picard, “Human pose regression by combining indirect part detection and contextual information,” Comput. Graph., vol. 85, pp. 15–22, 2019. [^47]: Y. Xiu, J. Li, H. Wang, Y. Fang, and C. Lu, “Pose flow: Efficient online pose tracking,” 2018, arXiv:1802.00977. [^48]: G. Ning and H. Huang, "LightTrack: A generic framework for online top-down human pose tracking," 2019, arXiv:1905.02822. [^49]: R. Girdhar, G. Gkioxari, L. Torresani, M. Paluri, and D. Tran, "Detect-and-track: Efficient pose estimation in videos," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2018, pp. 350–359. [^50]: M. Wang, J. Tighe, and D. Modolo, "Combining detection and tracking for human pose estimation in videos," 2020, arXiv:2003.13743. [^51]: H. Guo, T. Tang, G. Luo, R. Chen, Y. Lu, and L. Wen, "Multi-domain pose network for multi-person pose estimation and tracking," in Proc. Eur. Conf. Comput. Vis. Workshops, 2018, pp. 209–216. [^52]: Q. Bao, W. Liu, Y. Cheng, B. Zhou, and T. Mei, “Pose-guided tracking-by-detection: Robust multi-person pose tracking,” IEEE Trans. Multimedia, vol. 23, pp. 161–175, Mar. 2020. [^53]: Y. Yang, Z. Ren, H. Li, C. Zhou, X. Wang, and G. Hua, "Learning dynamics via graph neural networks for human pose estimation and tracking," in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recog-nit., 2021, pp. 8074–8084. [^54]: M. Snower, A. Kadav, F. Lai, and H. P. Graf, "15 keypoints is all you need," in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2020, pp. 6738–6748. [^55]: Y. Raaj, H. Idrees, G. Hidalgo, and Y. Sheikh, “Efficient online multi-person 2D pose tracking with recurrent spatio-temporal affinity fields,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2019, pp. 4620–4628. [^56]: S. Jin, W. Liu, W. Ouyang, and C. Qian, "Multi-person articulated tracking with spatial and temporal embeddings," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2019, pp. 5664–5673. [^57]: J. Zhang et al., “FastPose: Towards real-time pose estimation and tracking via scale-normalized multi-task networks,” 2019, arXiv:1908.05593. [^58]: B. Gao and L. Pavel, “On the properties of the softmax function with application in game theory and reinforcement learning,” 2017, arXiv:1704.00805. [^59]: H. Gouk, E. Frank, B. Pfahringer, and M. J. Cree, “Regularisation of neural networks by enforcing Lipschitz continuity,” Mach. Learn., vol. 110, no. 2, pp. 393–416, 2021. [^60]: J. Li et al., “Human pose regression with residual log-likelihood estimation,” in Proc. IEEE/CVF Int. Conf. Comput. Vis., 2021, pp. 11005–11014. [^61]: C. Sagonas, E. Antonakos, G. Tzimiropoulos, S. Zafeiriou, and M. Pantic, “300 faces in-the-wild challenge: Database and results,” Image Vis. Comput., vol. 47, pp. 3–18, 2016. [^62]: C. Zimmermann, D. Ceylan, J. Yang, B. Russell, M. Argus, and T. Brox, "FreiHAND: A dataset for markerless capture of hand pose and shape from single RGB images," in Proc. IEEE/CVF Int. Conf. Comput. Vis., 2019, pp. 813–822. [^63]: G. Moon, S.-I. Yu, H. Wen, T. Shiratori, and K. M. Lee, "InterHand2.6M: A dataset and baseline for 3D interacting hand pose estimation from a single RGB image," 2020, arXiv:2008.09309. [^64]: J. Redmon and A. Farhadi, "YOLOv3: An incremental improvement," 2018, arXiv:1804.02767. [^65]: G. Papandreou et al., "Towards accurate multi-person pose estimation in the wild," in Proc. IEEE Conf. Comput. Vis. Pattern Recognition., 2017, pp. 4903–4911. [^66]: X. Burgos-Artizzu, D. Hall, P. Perona, and P. Dollar, “Merging pose estimates across space and time,” in Proc. Brit. Mach. Vis. Conf., 2013, pp. 58.1–58.11. [^67]: Z. Wang, L. Zheng, Y. Liu, Y. Li, and S. Wang, "Towards real-time multi-object tracking," 2019, arXiv:1909.12605. [^68]: M. Andriluka et al., “PoseTrack: A benchmark for human pose estimation and tracking,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2018, pp. 5167–5176. [^69]: A. Kendall, Y. Gal, and R. Cipolla, "Multi-task learning using uncertainty to weigh losses for scene geometry and semantics," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2018, pp. 7482–7491. [^70]: Y. Wang, C. Peng, and Y. Liu, "Mask-pose cascaded CNN for 2D hand pose estimation from single color image," IEEE Trans. Circuits Syst. Video Technol., vol. 29, no. 11, pp. 3258–3268, Nov. 2019. [^71]: F. Gomez-Donoso, S. Orts-Escolano, and M. Cazorla, "Large-scale multiview 3D hand pose dataset," Image Vis. Comput., vol. 81, pp. 25–33, 2019. [^72]: W. Wu, C. Qian, S. Yang, Q. Wang, Y. Cai, and Q. Zhou, "Look at boundary: A boundary-aware face alignment algorithm," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2018, pp. 2129–2138. [^73]: M. Koestinger, P. Wohlhart, P. M. Roth, and H. Bischof, "Annotated facial landmarks in the wild: A large-scale, real-world database for facial landmark localization," in Proc. IEEE Int. Conf. Comput. Vis. Workshops, 2011, pp. 2144–2151. --- [^74]: X. P. Burgos-Artizzu, P. Perona, and P. Dollár, “Robust face landmark estimation under occlusion,” in Proc. IEEE Int. Conf. Comput. Vis., 2013, pp. 1513–1520. [^75]: M. Tan, R. Pang, and Q. V. Le, "EfficientDet: Scalable and efficient object detection," in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognition., 2020, pp. 10781–10790. [^76]: P. Wang et al., “Understanding convolution for semantic segmentation,” in Proc. IEEE Winter Conf. Appl. Comput. Vis., 2018, pp. 1451–1460. [^77]: W. Shi et al., “Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2016, pp. 1874–1883. [^78]: J. Dai et al., “Deformable convolutional networks,” in Proc. IEEE Int. Conf. Comput. Vis., 2017, pp. 764–773. [^79]: A. Paszke et al., “PyTorch: An imperative style, high-performance deep learning library,” in Proc. Int. Conf. Neural Inf. Process. Syst., 2019, pp. 8026–8037. [^80]: T. Chen et al., “MXNet: A flexible and efficient machine learning library for heterogeneous distributed systems,” 2015, arXiv:1512.01274. [^81]: Z. Ge, S. Liu, F. Wang, Z. Li, and J. Sun, “YOLOX: Exceeding YOLO series in 2021,” 2021, arXiv:2107.08430. [^82]: B. Pang, Y. Li, Y. Zhang, M. Li, and C. Lu, "TubeTK: Adopting tubes to track multi-object in a one-step training model," in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2020, pp. 6308–6318. [^83]: N. Bodla, B. Singh, R. Chellappa, and L. S. Davis, “Soft-NMS—Improving object detection with one line of code,” in Proc. IEEE Int. Conf. Comput. Vis., 2017, pp. 5562–5570. [^84]: X. Zhu, Z. Lei, X. Liu, H. Shi, and S. Z. Li, "Face alignment across large poses: A 3D solution," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2016, pp. 146–155. [^85]: A. Milan, L. Leal-Taixé, I. Reid, S. Roth, and K. Schindler, "MOT16: A benchmark for multi-object tracking," 2016, arXiv:1603.00831. [^86]: D. P. Kingma and J. Ba, "Adam: A method for stochastic optimization," 2014, arXiv:1412.6980. [^87]: A. Doering, U. Iqbal, and J. Gall, "Joint flow: Temporal flow fields for multi person tracking," 2018, arXiv:1805.04596. [^88]: J. Hwang, J. Lee, S. Park, and N. Kwak, “Pose estimator and tracker using temporal flow maps for limbs,” in Proc. Int. Joint Conf. Neural Netw., 2019, pp. 1–8. [^89]: G. Ning, P. Liu, X. Fan, and C. Zhang, "A top-down approach to articulated human pose estimation and tracking," in Proc. Eur. Conf. Comput. Vis., 2018, pp. 227–234. ![Image](images/b3aad764024b68bbc47b4d91b0f905f98915b92648f2a10ffe5b937e44654be5.jpg) Hao-Shu Fang received the BE degree in computer science from Shanghai Jiao Tong University, in 2019. He is currently working toward the PhD degree with the Department of Computer Science and Engineering, and MVIG Lab, Shanghai Jiao Tong University. His research interests include computer vision and robot intelligence. He was awarded the CCF-CV Rising Scholar Award in 2019, Baidu Fellowship in 2019, MSRA Fellowship in 2020, and ByteDance Fellowship in 2020. ![Image](images/1c5f6b3dd2a05f611c4dee73f42a89402bda8685d25a689cfa7a4816e0125da7.jpg) Jiefeng Li received the BS and MS degrees from Shanghai Jiao Tong University, Shanghai, China, in 2018 and 2020. He is currently working toward the PhD degree with Machine Vision and Intelligence Group, Shanghai Jiao Tong University. His research interests include computer vision, machine learning, and robotics. ![Image](images/8ca72044fd6f1db1702c2d1ad60d9ca545850d60c80b93db8c24b8ef46764859.jpg) Hongyang Tang received the BS degree from the Department of Automation, Tsinghua University, in 2019. He is currently working toward the postgraduate degree with the Machine Vision and Intelligence Group, Shanghai Jiao Tong University. His research interests include computer vision and machine perception. ![Image](images/1abb57cb178dfded395256ada8e797c2c8bcd0f7f459b972ebf7d9a534cf45b3.jpg) Chao Xu received the bachelor of engineering degree from the School of Cyber Science and Engineering, Shanghai Jiao Tong University (SJTU), and the master of engineering degree from the Department of Computer Science and Engineering, Shanghai Jiao Tong University (SJTU), in 2021. He is currently working as algorithm engineer with Tencent GY-Lab, Shanghai. His main research interests include computer vision, pose estimation, and cloud computing. ![Image](images/c7ec7fd1f74e3bda38627f8d3c1551a9ae90675fdbd1e4c68c7f37934e685efa.jpg) Haoyi Zhu is currently working toward the undergraduate degree majoring in artificial intelligence with the School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University, Shanghai, China. His research interests include computer vision, pose estimation, and robotics. ![Image](images/f58009c59863169c6ae1c5710c9b0b3a53e1aa1160230e97bfa606346abed1a0.jpg) Yuliang Xiu received the BEng degree from Shandong University, Jinan, China, in 2016, and the MSc degree from Shanghai Jiao Tong University, Shanghai, China, in 2019. He is currently working toward the PhD degree with the Perceiving Systems, Max Planck Institute for Intelligent Systems, Tuebingen, Germany. His research lies in the intersection of vision and graphics, especially in 3D Human Digitalization. ![Image](images/60a8ec27f1d2ad312f3b9614a71c0817671712d5d5b56773f786dc693b0c37ae.jpg) Yong-Lu Li received the BS degree in automation from the Beijing University of Chemical Technology, Beijing, China, in 2012, and the MS degree in control engineering from the Institution of Automation, Chinese Academy of Science and the Harbin University of Science and Technology, Beijing, in 2017. He is currently working toward the PhD degree with the Machine Vision and Intelligence Group, Shanghai Jiao Tong University. His research interests include computer vision and robotics. ![Image](images/ec9ff829ba72f471487cb0689234b616b8f7445b8ceaa08cb36af47f03175b82.jpg) Cewu Lu (Member, IEEE) received the PhD degree from the Chinese University of Hong Kong, supervised by Prof. Jiaya Jia. He is a professor with Shanghai Jiao Tong University (SJTU). Before he joined SJTU, he was a research fellow with Stanford University working under Prof. Fei-Fei Li and Prof. Leonidas J. Guibas. He was a research assistant professor with the Hong Kong University of Science and Technology with Prof. Chi Keung Tang. He was selected as MIT TR35 in 2019 and served as an area chair for CVPR'20 and ICCV'21. His research interests fall mainly in computer vision, deep learning, and robotics. For more information on this or any other computing topic, please visit our Digital Library at .