# 基于空间和时间Transformer的三维人体姿态估计 Ce Zheng$^{1}$, Sijie Zhu$^{1}$, Matias Mendieta$^{1}$, Taojiannan Yang$^{1}$, Chen Chen$^{1}$, Zhengming Ding$^{2}$$^{1}$Center for Research in Computer Vision, University of Central Florida, USA $^{2}$Department of Computer Science, Tulane University, USA {cezheng, sizhu, mendieta, taoyang1122}@knights.ucf.edu; ; ## Abstract Transformer architectures have become the model of choice in natural language processing and are now being introduced into computer vision tasks such as image classification, object detection, and semantic segmentation. However, in the field of human pose estimation, convolutional architectures still remain dominant. In this work, we present PoseFormer, a purely transformer-based approach for 3D human pose estimation in videos without convolutional architectures involved. Inspired by recent developments in vision transformers, we design a spatial-temporal transformer structure to comprehensively model the human joint relations within each frame as well as the temporal correlations across frames, then output an accurate 3D human pose of the center frame. We quantitatively and qualitatively evaluate our method on two popular and standard benchmark datasets: Human3.6M and MPI-INF-3DHP. Extensive experiments show that PoseFormer achieves state-of-the-art performance on both datasets. Code is available at ### 1. Introduction Human pose estimation (HPE) aims to localize joints and build a body representation (e.g. skeleton position) from input data such as images and videos. HPE provides geometric and motion information of the human body and can be applied to a wide range of applications (e.g. human-computer interaction, motion analysis, healthcare). Current works generally can be divided into two classes: (1) direct estimation approaches, and (2) 2D-to-3D lifting approaches. Direct estimation methods [31, 29] infer a 3D human pose from 2D images or video frames without intermediate estimating the 2D pose representation. 2D-to-3D lifting approaches [25, 5, 43, 38] infer 3D human pose from an intermediate estimated 2D pose. Benefiting from the excellent performance of state-of-the-art 2D pose detectors, 2D-to-3D lifting approaches generally outperform direct estimation methods. However, the mapping of these 2D poses to 3D is non-trivial; various potential 3D poses could be generated from the same 2D pose due to depth ambiguity and occlusion. To alleviate some of these issues and preserve natural coherence, many recent works have integrated temporal information from videos into their approaches. For example, [25, 5] utilize temporal convolutional neural networks (CNNs) to capture global dependencies from adjacent frames, and [33] uses recurrent architectures to similar effect. However, the temporal correlation window is limited for both of these architectures. CNN-based approaches typically rely on dilation techniques, which inherently have limited temporal connectivity, and recurrent networks are mainly constrained to simply sequential correlation. Recently, the transformer [37] has become the de facto model for natural language processing (NLP) due to its efficiency, scalability and strong modeling capabilities. Thanks to the self-attention mechanism of the transformer, global correlations across long input sequences can be distinctly captured. This makes it a particularly fitting architecture for sequence data problems, and therefore naturally extendable to 3D HPE. With its comprehensive connectivity and expression, the transformer provides an opportunity to learn stronger temporal representations across frames. However, recent works [12, 36] show that transformers require specific designs to achieve comparable performance with CNN counterparts for vision tasks. Specifically, they often require either extremely large scale training datasets [12], or enhanced data augmentation and regularization [36] if applied to smaller datasets. Moreover, existing vision transformers have been limited primarily to image classification [12, 36], object detection [4, 50], and segmentation [41, 47], but how to harness the power of transformers for 3D HPE remains unclear. To begin answering this question, we first directly apply the transformer on 2D-to-3D lifting HPE. In this case, we view the entire 2D pose for each frame in a given sequence as a token (Fig. 1(a)). While this baseline approach is functional to an extent, it ignores the natural distinction of --- ![Image](images/c1b01345d0d504ce1b663110388e0a368239cd281ba8d3d38a85a44c519a4d96.jpg)
Figure 1. Two baseline approaches.
spatial relations (joint-to-joint), leaving potential improvements on the table. A natural extension to this baseline is to instead view each 2D joint coordinate as a token, and provide an input formed with these joints from across all frames of the sequence (Fig. 1(b)). However, in this case, the number of tokens becomes increasingly large when long frame sequences are used (up to 243 frames and 17 joints per frame is common in 3D HPE, the number of tokens would be $ 243 \times 17 = 4131 $). Since the transformer computes direct attention with each token to another, the memory requirement of the model approaches an unreasonable level. Therefore, as an effective solution to these challenges, we propose PoseFormer, the first pure transformer network for 2D-to-3D lifting HPE in videos. PoseFormer directly models the spatial and temporal aspects with distinct transformer modules for both dimensions. Not only does PoseFormer produce strong representations across the spatial and temporal elements, it does so without inducing enormous token counts for long input sequences. On a high level, PoseFormer simply takes a sequence of detected 2D poses from an off-the-shelf 2D pose estimator, and outputs the 3D pose for the center frame. More specifically, we build a spatial transformer module to encode local relationships between the 2D joints in each frame. The spatial self-attention layers consider the position information of 2D joints and return a latent feature representation for that frame. Next, our temporal transformer module analyzes global dependencies between each spatial feature representation, and generates an accurate 3D pose estimation. Experimental evaluations on two popular 3D HPE benchmarks, Human3.6M [16] and MPI-INF-3DHP [27], show that PoseFormer achieves state-of-the-art performance on both datasets. We visualize our estimated 3D pose compared with the state-of-the-art approach, and find that PoseFormer produces smoother and more reliable results. Also, visualizations and analyses of PoseFormer's attention maps are provided in the ablation study to understand the internal workings of our model and demonstrate its effectiveness. Our contributions are three-fold: - We propose the first pure transformer-based model, PoseFormer, for 3D HPE under the category of 2D-to-3D lifting. cal relationships between human body joints, and the temporal transformer module captures the global dependencies across frames in the entire sequence. - We design an effective Spatial-Temporal Transformer model, where the spatial transformer module encodes lo- - Without bells and whistles, our PoseFormer model achieves state-of-the-art results on both Human3.6M and MPI-INF-3DHP datasets. ### 2. Related Works Here we specifically summarize 3D single-person single-view HPE methods. Direct estimation approaches infer 3D human pose from 2D images without intermediating the estimating 2D pose representation. 2D-to-3D lifting approaches utilize the 2D pose as input to generate the corresponding 3D pose, which is more popular among state-of-the-art methods in this domain. Any off-the-shelf 2D pose estimator can be effectively compatible with these methods. Our proposed method, PoseFormer, also follows the 2D-to-3D lifting pipeline, and therefore we will focus mainly on such methods in this section. 2D-to-3D Lifting HPE. 2D-to-3D lifting approaches leverage 2D poses estimated from input images or video frames. OpenPose [3], CPN [6], AlphaPose [13], and HRNet [35] have been extensively used as the 2D pose detectors. Based on this intermediate representation, the 3D pose can be generated with a variety of methods. Martinez et al. [26] proposed a simple and effective fully connected residual network to regress 3D joint locations based on the 2D joint locations from just a single frame. However, instead of estimating 3D human pose from monocular images, videos can provide temporal information to improve accuracy and robustness [49, 10, 32, 8, 2, 44, 38]. Hossain and Little [33] proposed a recurrent neural network using Long Short-Term Memory (LSTM) cells to exploit temporal information in the input sequence. Several works [10, 2, 21] utilized spatial-temporal relationships and constraints such as bone-length and left-right symmetry to improve performance. Pavllo et al. [32] introduced a temporal convolution network to estimate 3D pose over 2D keypoints from consecutive 2D sequences. Based on [32], Chen et al. [5] added a bone direction module and bone length module to ensure temporal consistency across video frames, and Liu et al. [25] utilized an attention mechanism to recognize significant frames. However, the previous state-of-the-art methods (e.g. [25, 5]) rely on dilated temporal convolutions to capture global dependencies, which are inherently limited in temporal connectivity. Additionally, the majority of these works [25, 5, 33, 32] project the joint coordinates to a latent space using simple operations, without considering the kinematic correlations of human joints. GNNs in 3D HPE. Naturally, a human pose can be represented as a graph where the joints are the nodes and the bones are the edges. Graph Neural Networks (GNNs) have also been applied to the 2D-to-3D pose lifting problem. --- ![Image](images/9de4b62975b479f49e0c731acb9f40e94e9168f8669cd873c44a6a8b79bf5720.jpg)
Figure 2. (a) Temporal transformer baseline. (b) Spatial-temporal transformer (PoseFormer) architecture, which consists of three modules. A spatial transformer module for extracting features with considering joint correlations of each individual skeleton. A temporal transformer module for learning global dependencies of entire sequence. A regression head module regresses the final 3D pose of the center frame. The illustration of the transformer encoder is followed by ViT $ ^{[12]} $.
and provided promising performance [9, 45, 24]. Ci et al. [9] proposed a framework, named Locally Connected Networks (LCNs), which leverages both fully connected networks and GNN operations to encode the relationship between local joint neighborhoods. Zhao et al. [45] tackled a limitation of Graph Convolutional Network [19] (GCN) operations, specifically how the weight matrix is shared across nodes. The semantic graph convolution operation was introduced to learn channel-wise weights for edges. For our PoseFormer, the transformer can be viewed as a type of graph neural network with a unique, and often advantageous, graph operation. Specifically, a transformer encoder module essentially forms a fully-connected graph, where the edge weights are computed using input-conditioned, multi-headed self-attention. The operation also includes the normalization of node features, a feedforward aggregator across attention head outputs, and residual connections which enable it to scale effectively with stacked layers. Such an operation can be advantageous in comparison to other graph operations. For example, the strength of the connection between nodes is determined by the self-attention mechanism of the transformer, rather than predefined through an adjacency matrix as with the typical GCN-based formulations employed in this task. This allows the model flexibility to adapt the relative importance of joints to each other with each input pose. Additionally, the comprehensive scaling and normalization components of the transformer are likely advantageous in mitigating the over-smoothing effect that troubles many GNN operation variants when numerous layers are stacked together [48]. Vision Transformers. Recently, there is an emerging interest in applying transformers to vision tasks [17, 14]. Carion et al. [4] presented a DEtection TRansformer (DETR) for object detection and panoptic segmentation. Dosovitskiy et al. [12] proposed a pure transformer architecture, Vision Transformer (ViT), which achieves state-of-the-art performance on image classification. However, ViT was trained on large-scale datasets ImageNet-21k and JFT-300M that requires huge computation resources. Then, a data-efficient image transformer (DeiT) [36] was proposed which builds upon the ViT with knowledge distillation. For regression problems such as HPE, Yang et al. [40] proposed a transformer network, Transpose, which only estimates 2D pose from images. Lin et al. [23] combined CNNs with transformer networks in their method METRO (MEsh TransfOrmer) to reconstruct the 3D pose and mesh vertices from a single image. In contrast to our approach, METRO falls under the category of direct estimation. Also, temporal consistency is neglected in METRO, which limits the robustness of its estimations. Our spatial-temporal transformer architecture exploits keypoint correlation in each frame and preserves natural temporal coherence in videos. ### 3. Method We follow the same 2D-to-3D lifting pipeline for 3D HPE in videos as [26, 32, 25, 5]. The 2D pose of each frame is obtained by an off-the-shelf 2D pose detector, then 2D pose sequences of consecutive frames are used as input for estimating the 3D pose of the center frame. Compared to the previous state-of-the-art models, which are based on CNNs, we produce a highly competitive convolution-free transformer network. #### 3.1. Temporal Transformer Baseline As a baseline application of a transformer in 2D-to-3D lifting, we treat each 2D pose as an input token and employ a transformer to capture global dependencies among the inputs as illustrated in Fig. 2(a). We will refer to each input token as a patch, similar in terminology to ViT [12]. For the input sequence $ X \in \mathbb{R}^{f \times (J \cdot 2)} $, $ f $ is the number of frames of the input sequence, $ J $ is the number of joints of each 2D pose, and 2 indicates joint's coordinate in 2D space. $ \{\mathbf{x}^i \in \mathbb{R}^{1 \times (J \cdot 2)} | i = 1, 2, \ldots, f\} $ indicates the input vector of each frame. The patch embedding is a trainable linear --- projection layer to embed each patch to a high dimensional feature. The transformer network utilizes positional embeddings to retain positional information of the sequence. The procedure can be formulated as: $$ Z_{0}=[\mathbf{x}^{1}E;\mathbf{x}^{2}E;\ldots;\mathbf{x}^{f}E]+E_{p o s}. $$ After embedding through a linear projection matrix $ E \in \mathbb{R}^{(J\cdot2) \times C} $ and summing with the positional embedding $ E_{pos} \in \mathbb{R}^{f \times C} $, the input sequence $ X \in \mathbb{R}^{f \times (J\cdot2)} $ becomes $ Z_0 \in \mathbb{R}^{f \times C} $, where $ C $ is the embedding dimension. $ Z_0 $ is sent to the Temporal Transformer Encoder. As the core function of the transformer, self-attention is designed to relate different positions of the input sequence with embedded features. Our transformer encoder is composed of Multi-head Self Attention blocks with multilayer perceptron (MLP) blocks as in [12]. LayerNorm is applied before every block and residual connections are applied after every block [39, 1]. Scaled Dot-Product Attention can be described as a mapping function that maps a query matrix $ Q $, key matrix $ K $ and value matrix $ V $ to an output attention matrix. $ Q $, $ K $, $ V \in \mathbb{R}^{N \times d} $, where $ N $ is the number of vectors in the sequence and $ d $ is the dimension. A scaling factor of $ \frac{1}{\sqrt{d}} $ is utilized within this attention operation for appropriate normalization, preventing extremely small gradients when large values of $ d $ lead dot products to grow large in magnitude. Thus the output of the scaled dot-product attention can be expressed as: $$ Attention(Q,K,V)=\mathrm{Softmax}(QK^{\top}/\sqrt{d})V. $$ In our temporal transformer, $d = C$ and $N = f$. The $Q$, $K$ and $V$ are computed from the embedded feature $Z \in \mathbb{R}^{f \times C}$ by linear transformations $W_Q$, $W_K$ and $W_V \in \mathbb{R}^{C \times C}$: $$ Q=Z W_{Q},\quad K=Z W_{K},\quad V=Z W_{V}. $$ Multi-head Self Attention Layer (MSA) utilizes multiple heads to model the information jointly from various representation subspaces with different positions. Each head applies scaled dot-product attention in parallel. The MSA output will be the concatenation of $ h $ attention head outputs. $$ \mathrm{MSA}(Q,K,V)=\mathrm{Concat}(H_{1},H_{2},\ldots,H_{h})W_{out} $$ $$ \begin{array}{l}where\quad H_{i}=Attention(Q_{i},K_{i},V_{i}),i\in[1,...,h]\end{array} $$ The Temporal Transformer Encoder structure of $L$ layers given our embedded feature $Z_0 \in \mathbb{R}^{f \times C}$ can be represented as follows: $$ Z_{\ell}^{^{\prime}}=\mathrm{M S A}(\mathrm{L N}(Z_{\ell-1}))+Z_{\ell-1},\quad\ell=1,2\ldots L $$ $$ Z_{\ell}=\operatorname{M L P}(\operatorname{L N}(Z_{\ell}^{^{\prime}}))+Z_{\ell}^{^{\prime}},\quad\ell=1,2,\ldots L $$ $$ Y=\mathrm{L N}(Z_{L}), $$ where $ \text{LN}(\cdot) $ denotes the layer normalization operator (same as in ViT). The temporal transformer encoder consists of $ L $ identical layers and the encoder output $ Y \in \mathbb{R}^{f \times C} $ keeps the same size as encoder input $ Z_0 \in \mathbb{R}^{f \times C} $. In order to predict the 3D pose of center frame, the encoder output $ Y \in \mathbb{R}^{f \times C} $ is shrunk to a vector $ \mathbf{y} \in \mathbb{R}^{1 \times C} $ by taking the average in the frame dimension. Finally, an MLP block will regress the output to $ \mathbf{y} \in \mathbb{R}^{1 \times (J \cdot 3)} $, which is the 3D pose of the center frame. #### 3.2. PoseFormer: Spatial-Temporal Transformer We observe that the temporal transformer baseline mainly focuses on global dependencies between frames in the input sequence. The patch embedding, a linear transformation, is utilized to project joint coordinates to a hidden dimension. However, the kinematic information between local joint coordinates is not strongly represented in the temporal transformer baseline because a simple linear projection layer is not able to learn attention information. One potential workaround is to view each joint coordinate as an individual patch and feeding the joints from all frames as input to the transformer (see Fig. 1(b)). However, the number of patches would increase rapidly (frames $f$ multiplied by the number of joint $J$), resulting in a model computational complexity of $O((f \cdot J)^2)$. For example, if we use 81 frames and 17 joints for each 2D pose, the number of patches would be 1377 (ViT model uses 576 patches (input size $= 384 \times 384$, patch size $= 16 \times 16$)). To learn local joint correlations efficiently, we employ two separated transformers for spatial and temporal information, respectively. As shown in Fig. 2(b), PoseFormer consists of three modules: spatial transformer module, temporal transformer module, and regression head module. Spatial Transformer Module. The spatial transformer module is to extract a high dimensional feature embedding from a single frame. Given a 2D pose with $ J $ joints, we consider each joint (i.e. two coordinates) as a patch and follow the general vision transformer pipeline to perform the feature extraction among all patches. First, we map the coordinate of each joint to a high dimension with a trainable linear projection, which is referred to as the spatial patch embedding. We sum this with the learnable spatial positional embedding [12] $ E_{SPos} \in \mathbb{R}^{J \times c} $, and therefore the input $ \mathbf{x}_i \in \mathbb{R}^{1 \times (J \cdot 2)} $ of the $ i $-th frame becomes $ z_0^i \in \mathbb{R}^{J \times c} $, where $ 2 $ indicates 2D coordinate in each frame and $ c $ is the spatial embedding dimension. The resulting joint sequence of features $ z_0^i $ are fed into the spatial transformer encoder which applies the self-attention mechanism to integrate information across all joints. For the $ i $-th frame, the output of spatial transformer encoder with $ L $ layers will be $ z_L^i \in \mathbb{R}^{J \times c} $. Temporal Transformer Module. Since the spatial transformer module encodes high dimensional features for each individual frame, the goal for the temporal transformer module is to model dependencies across the sequence of frames. For the i-th frame, the output of the spatial trans- --- former $ z_L^i \in \mathbb{R}^{J \times c} $ is flattened as a vector $ \mathbf{z}^i \in \mathbb{R}^{1 \times (J \cdot c)} $. We then concatenate these vectors $ \{z^1, z^2, \ldots, z^f\} $ from the $ f $ input frames as $ Z_0 \in \mathbb{R}^{f \times (J \cdot c)} $. Before the temporal transformer module, we add the learnable temporal positional embedding [12] $ E_{TPos} \in \mathbb{R}^{f \times (J \cdot c)} $ to retain frame position information. For the temporal transformer encoder, we apply the same architecture as the spatial transformer encoder, which consists of multihead self-attention blocks and MLP blocks. The output of the temporal transformer module is $ Y \in \mathbb{R}^{f \times (J \cdot c)} $. Regression Head. Since we use a sequence of frames to predict the 3D pose of the center frame, the output of the temporal transformer module $ Y \in \mathbb{R}^{f \times (J \cdot c)} $ needs to be reduced to $ \mathbf{y} \in \mathbb{R}^{1 \times (J \cdot c)} $. We apply a weighted mean operation (with learned weights) on the frame dimension to achieve this. Finally, a simple MLP block with Layer norm and one linear layer returns output $ \mathbf{y} \in \mathbb{R}^{1 \times (J \cdot 3)} $ which is the predicted 3D pose of the center frame. Loss Function. To train our spatial-temporal transformer model, we apply the standard MPJPE (Mean Per Joint Position Error) loss to minimize the error between the predicted and ground truth pose as $$ \mathcal{L}=\frac{1}{J}\Sigma_{k=1}^{J}\|p_{k}-\hat{p}_{k}\|_{2}, $$ 其中 $p_{i}$ ted 三维关节 ## 4. 实验 ### 4.1. 数据集与评估指标 我们在两个常用的3D HPE数据集Human3.6M [^16] 和 MPI-INF-3DHP [^27] 上评估了我们的模型。 Human3.6M [^16] 是单人3D HPE领域应用最广泛的室内数据集。共有11名专业演员完成静坐、行走、打电话等17种动作，每个受试者的视频在室内环境下从4个不同视角录制。该数据集包含360万帧视频帧，带有由高精度标记式动作捕捉系统采集的3D真值标注。遵循已有研究 [^32] [^25] [^5]，我们采用相同的实验设置：全部15种动作用于训练和测试，模型在五个分组（S1、S5、S6、S7、S8）上训练，在两个受试者（S9和S11）上测试。 MPI-INF-3DHP [^27] 是一个难度更高的3D姿态数据集，同时包含受限室内场景和复杂户外场景。共有8名演员在14个相机视角下完成8种动作，覆盖了更多样的姿态。MPI-INF-3DHP提供了包含6名受试者、涵盖不同场景的测试集。我们遵循 [^22] [^5] [^38] 中的设置。 针对Human3.6M数据集，我们使用最通用的评估指标（MPJPE和P-MPJPE）[^46]，将我们的估计结果与3D真值姿态比对来评估性能。MPJPE（平均每关节位置误差，Mean Per Joint Position Error）是估计关节与真值之间以毫米为单位的平均欧氏距离，我们将MPJPE称为协议1。P-MPJPE是估计3D姿态与真值经过后处理刚性对齐后的MPJPE，它对单个关节预测错误的鲁棒性更强，我们将P-MPJPE称为协议2。 针对MPI-INF-3DHP数据集，我们使用MPJPE、150mm范围内正确关键点百分比（PCK）[^22] [^5] [^38] 以及曲线下面积（AUC）作为评估指标。 ### 4.2. 实现细节 我们使用Pytorch实现了本文提出的方法[^30]。训练和测试使用两块NVIDIA RTX 3090 GPU完成。实验中我们选取了三种不同的帧序列长度，即$f = 9$、$f = 27$、$f = 81$。关于帧数和对应结果的详细分析将在消融实验（第4.4节）中讨论。遵循[^32] [^25] [^5]的设置，我们在训练和测试阶段都采用水平翻转姿态作为数据增强方法。我们使用Adam[^18]优化器训练模型共130轮，权重衰减系数为0.1。我们采用指数学习率衰减策略，初始学习率为2e-4，每个epoch的衰减因子为0.98。我们将批量大小设置为1024，在Transformer编码器层使用随机深度[^15]，概率为0.1。对于2D姿态检测器，我们遵循[^7] [^32] [^25]的设置，在Human3.6M数据集上使用级联金字塔网络（CPN）[^5]，并遵循[^28] [^22]的设置，在MPI-INF-3DHP数据集上使用真实2D姿态作为输入。 ### 4.3 与现有最优方法的对比 Human3.6M数据集。我们在表1中报告了测试集（S9和S11）全部15种动作的实验结果，最后一列给出了整个测试集的平均性能。遵循2D到3D姿态提升方法的通用设置，我们使用CPN网络作为2D姿态检测器，再将检测得到的2D姿态作为输入进行训练和测试。在协议1和协议2下，PoseFormer分别以6.1%和6.4%的幅度大幅优于我们的基线方法（即第3.1节中的时间Transformer基线）。这清楚地证明了使用空间Transformer对每一帧的关节相关性进行高效建模的优势。如表1（上半部分）所示，在协议1下，PoseFormer获得了最低的平均MPJPE，为44.3mm。与基于Transformer的方法METRO [^23]相比，后者通过单张图像估计3D姿态，忽略了时间一致性，PoseFormer将MPJPE降低了约18%。对于协议2，如表1（下半部分）所示，我们同样取得了最佳的整体结果。此外，PoseFormer在拍照（Photo）、坐下（SittingDown）、遛狗（WalkDog）和抽烟（Smoke）等难度较高的动作上也能得到更准确的姿态预测结果。与其他简单动作不同，这些动作的姿态变化更快，部分远距离帧之间存在较强的相关性，在这种情况下全局依赖发挥着重要作用。 ---
Table 1. Quantitative comparison of Mean Per Joint Position Error between the estimated 3D pose and the ground truth 3D pose on Human3.6M under Protocols 1&2 using the detected 2D pose as input. Top-table: results under Protocol 1 (MPJPE). Bottom-table: results under Protocol 2 (P-MPJPE). f denotes the number of input frames used in each method, $*$ indicates that the input 2D pose is detected by the cascaded pyramid network (CPN), and $\dagger$ denotes a Transformer-based model. (Red: best; Blue: second best)
Protocol 1Dir.Disc.Eat.GreetPhonePhotoPosePurch.SitSitD.SomkeWaitWalkD.WalkWalkT.Average
Dabral et al. [^11]ECCV'1844.850.444.749.052.961.443.545.563.187.351.748.552.237.641.952.1
Cai et al. [^2] (f = 7)ICCV'1944.647.445.648.850.859.047.243.957.961.949.746.651.337.139.448.8
Pavllo et al. [^32] (f = 243)*CVPR'1945.246.743.345.648.155.144.644.357.365.847.144.049.032.833.946.8
Lin et al. [^22] (f = 50)BMVC'1942.544.842.644.248.557.152.641.456.564.547.443.048.133.035.146.6
Yeh et al. [^42]NIPS'1944.846.143.346.449.055.244.644.058.362.747.143.948.632.733.346.7
Liu et al. [^25] (f = 243)*CVPR'2041.844.841.144.947.454.143.442.256.263.645.343.545.331.332.245.1
SRNet [^43]*ECCV'2046.647.143.941.645.849.646.540.053.461.146.142.643.131.532.644.8
UGCN [^38] (f = 96)ECCV'2041.343.944.042.248.057.142.243.257.361.347.043.547.032.631.845.6
Chen et al. [^5] (f = 81)*TCSVT'2142.143.841.043.846.153.542.443.153.960.545.742.146.232.233.844.6
METRO [^23] (f = 1)†CVPR'21---------------54.0
Baseline (f = 81)*†43.847.943.845.549.755.744.345.857.766.347.445.448.632.533.847.2
PoseFormer (f = 81)*†41.544.839.842.546.551.642.142.053.360.745.543.346.131.832.244.3
Protocol 2Dir.Disc.Eat.GreetPhonePhotoPosePurch.SitSitD.SomkeWaitWalkD.WalkWalkT.Average
Pavlakos et al. [^31]CVPR'1834.739.841.838.642.547.538.036.650.756.842.639.643.932.136.541.8
Hossain et al. [^33]ECCV'1835.739.344.643.047.254.038.337.551.661.346.541.447.334.239.444.1
Cai et al. [^2] (f = 7)ICCV'1935.737.836.940.739.645.237.434.546.950.140.536.141.029.632.339.0
Lin et al. [^22] (f = 50)BMVC'1932.535.334.336.237.843.033.032.245.751.838.432.837.525.828.936.8
Pavllo et al. [^32] (f = 243)*CVPR'1934.136.134.437.236.442.234.433.645.052.537.433.837.825.627.336.5
Liu et al. [^23] (f = 243)*CVPR'2032.335.233.335.835.941.533.232.744.650.937.032.437.025.227.235.6
UGCN [^38] (f = 96)ECCV'2032.935.235.634.436.442.731.232.545.650.237.332.836.326.023.935.5
Chen et al. [^5] (f = 81)*TCSVT'2133.135.333.435.936.141.732.833.342.649.437.032.736.525.527.935.6
Baseline (f = 81)*†33.637.135.436.737.842.233.934.747.053.438.234.337.625.327.837.0
PoseFormer (f = 81)*†32.534.832.634.635.339.532.132.042.848.534.832.435.324.526.034.6
作用，而Transformer的注意力机制在这一方面具备独特优势。 为了进一步探究我们方法的性能下界，我们直接使用真实2D姿态作为输入，以缓解噪声2D姿态数据带来的误差。实验结果如表2所示。使用干净的2D姿态数据后，MPJPE从44.3mm降低到31.3mm，降幅约为29.7%。PoseFormer在9个动作上取得了最高分，在6个动作上取得了第二高分，与SRNet [^43]相比，平均分数提升了约2%。 在图3中，我们还对比了Human3.6M测试集S11中动作Photo上误差最大的若干单个关节的MPJPE。PoseFormer在这些困难关节上取得了比[^32] [^5]更好的性能。 ![Image](images/914567d134d69d0765eb7563b808d7a6c81af9fecbe52d15abebedae1754d748.jpg)
Figure 3. The average joint error comparison across all the frames of the Human3.6M test set S11 with the Photo action.
MPI-INF-3DHP数据集。表3报告了PoseFormer与其他方法在MPI-INF-3DHP上的定量结果。与Human3.6M相比，该数据集包含的训练样本更少，且部分样本采集于户外场景。由于该数据集的序列长度通常更短，我们将9帧2D姿态作为模型输入。我们的方法再次在全部三个评价指标（PCK、AUC和MPJPE）上取得了最佳性能。 定性结果。我们还提供了3D估计姿态与真实姿态之间的视觉对比。我们在Human3.6M测试集S11的Photo动作上评估了PoseFormer，该动作是最具挑战性的动作之一（所有方法在该动作上都有较高的MPJPE）。如图4所示，与现有最先进方法[^5]相比，我们的PoseFormer获得了更准确的预测结果。 #### 4.4. 消融实验 为了验证PoseFormer各个组件的贡献，以及超参数对模型性能的影响，我们在协议1下使用Human3.6M数据集开展了大量的消融实验。 PoseFormer的设计。我们在表4中研究了空间变换器，以及空间变换器和时间变换器的位置嵌入的影响。我们输入9帧由CPN检测得到的2D姿态（J = 17）来预测3D姿态。为了公平比较每个模块的影响，所有架构参数都保持固定；空间变换器的嵌入维度为$17 \times 32 = 544$，空间变换器编码器层数为4。对于时间变换器，其嵌入维度与空间变换器保持一致（即544），我们同样使用4个时间变换器层。为了验证我们的时空设计的效果，我们首先与本文3.1节中我们最初采用的变换器基线进行对比。表4的结果表明，我们的时空变换器带来了显著的性能提升（MPJPE从52.5降至49.9），这是因为它对关节间相关性的建模能力更强。这也与表1中f = 81时的结果（基线 vs. PoseFormer）一致。接下来，我们评估了位置嵌入的影响。我们探究了四种可能的组合：无位置嵌入、仅空间位置嵌入、仅时间位置嵌入，以及同时包含空间位置和时间位置嵌入。 ---
Table 2. Quantitative comparison of Mean Per Joint Position Error between the estimated 3D pose and the ground truth 3D pose on Human3.6M dataset under Protocol 1 (MPJPE) using the ground truth 2D pose as input. (Red: best; Blue: second best)
SGPANLTD.Disc.Eat.GreetPhonePhotoPosePurch.SitSitD.SomkeWaitWalkD.WalkWalkT.Average
Hossain et al. [^33]ECCV'1835.240.837.237.443.244.038.935.642.344.639.739.740.232.835.539.2
Pavllo et al. [^32] (f = 243)CVPR'19---------------37.2
Liu et al. [^25] (f = 243)CVPR'2034.537.133.634.232.937.139.635.840.741.433.033.833.026.626.934.7
SRNet [^43]ECCV'2034.832.128.530.731.436.935.630.538.940.532.531.029.922.524.532.0
Chen et al. [^5] (f = 243)TCSVT'2132.3
PoseFormer (f = 81)30.033.629.931.030.233.334.831.437.838.631.731.529.023.323.131.3
![Image](images/e80ef713a928a7d705135d4c7a7e05b3b4e32d73b8ec76caf774ef55d7a34ac0.jpg)
Figure 4. Qualitative comparison between our method (PoseFormer) and the SOTA approach Chen et al. [^5] on Human3.6M test set S11 with the Photo action. The green arrows highlight locations where PoseFormer clearly has better results.
Table 3. Quantitative comparison with previous methods on MPI-INF-3DHP. The best scores are marked in bold.
PCK $\uparrow$AUC $\uparrow$MPJPE $\downarrow$
Mehta et al. [^27]3DV'1775.739.3117.6
Mehta et al. [^28]ACM ToG'1776.640.4124.7
Pavllo et al. [^32] (81 frames)CVPR'1986.051.984.0
Pavllo et al. [^32] (243 frames)CVPR'1985.551.584.8
Lin et al. [^22] (25 frames)BMVC'1983.651.479.8
Li et al. [^20]CVPR'2081.246.199.7
Chen et al. [^5] (81 frames)TCSVT'2187.954.078.8
PoseFormer (9 frames)88.656.477.1
Table 4. Ablation study on different components in PoseFormer. The evaluation is performed on Human3.6M (Protocol 1) using detected 2D pose as input. (T: Temporal only; S-T: Spatial-temporal)
Input length (f)TS-TSpatial Pos EmbTemporal Pos EmbMPJPE
9✓✗✗✓52.5
9✗✓✗✗51.6
9✗✓✓✗50.7
9✗✓✗✓50.5
9✗✓✓✓49.9
Table 5. Ablation study on different architecture parameters in PoseFormer. The evaluation is performed on Human3.6M (Protocol 1) using detected 2D pose as input. c is the spatial transformer patch embedding dimension. $L_S$ and $L_T$ indicate the number of layers in the spatial and temporal transformers, respectively.
c161616323232484848
$L_S$246246246
$L_T$246246246
MPJPE52.851.750.452.449.950.351.750.450.5
置嵌入。对比这些组合的结果可以明显看出，位置嵌入提升了模型性能。在空间模块和时间模块都应用位置嵌入后，我们获得了最佳的整体结果。 架构参数分析。我们探究了多种参数组合，以找到最优的网络
Table 6. Comparison on computational complexity, MPJPE, and inference speed (frame per second (FPS)). The evaluation is performed on Human3.6M under Protocol 1 using detected 2D pose as input. FPS is based on a single GeForce GTX 2080 Ti GPU.
fParameters (M)FLOPs (M)MPIPEFPS
Hossain and Little [^33]-16.9533.8858.3-
Pavllo et al. [^32]278.5617.0948.81492
Pavllo et al. [^32]8112.7925.4847.71121
Pavllo et al. [^32]24316.9533.8746.8863
Chen et al. [^5]2731.8861.745.3410
Chen et al. [^5]8145.5388.944.6315
Chen et al. [^5]24359.1811644.1264
PoseFormer99.5815049.9320
PoseFormer279.5945247.0297
PoseFormer819.60135844.3269
架构，结果如表5所示。c代表空间变换器中的嵌入特征维度，L表示变换器编码器使用的层数。在PoseFormer中，空间变换器的输出会被展平，然后与时间位置嵌入相加，形成时间变换器编码器的输入。因此，时间变换器编码器中的嵌入特征维度为$c \times J$。我们模型的最优参数为c = 32、$L_S = 4$和$L_T = 4$。 计算复杂度分析。我们在表6中给出了不同输入序列长度(f)下的模型性能、总参数量、每帧估计浮点运算量(FLOPs)以及输出每秒帧数(FPS)。我们的模型在序列长度增加时可获得更高的准确率，同时总参数量并未出现明显增长。这是因为帧数仅影响时间位置嵌入层，该层并不需要大量参数。与其他模型相比，我们的模型总参数量更少，同时具备竞争力 --- 表现。我们遵循[^5]中的相同设置，在单块GeForce RTX 2080 Ti GPU上测试了不同模型的推理FPS。虽然我们模型的推理速度并非绝对最快，但该速度对于实时推理来说仍然是可接受的。完整的3D HPE处理流程为：先由2D姿态检测器检测出2D姿态，再由我们的方法估计出3D姿态。常用2D姿态检测器的FPS通常低于80，这意味着我们模型的推理速度不会成为整个流程的瓶颈。 ![Image](images/284eda9d5d220fdd24d273a34ac5d3065c55236d87fa39f4c6ec49b52f9b41fa.jpg)
Figure 5. Visualization of self-attentions in the spatial transformer. The x-axis (horizontal) and y-axis (vertical) correspond to the queries and the predicted outputs, respectively. The pixel $w_{i,j}$ (i: row, j: column) denotes the attention weight of the j-th query for the i-th output. Red indicates stronger attention. The attention output is normalized from 0 to 1.
![Image](images/35390d60c3320f394f29cf39fe3a43bd8a90734cda872d662e09d08bc212de02.jpg)
Figure 6. Visualization of self-attentions in temporal transformer. The x-axis (horizontal) and y-axis (vertical) correspond to the queries and the predicted outputs, respectively. The pixel $w_{i,j}$ (i: row, j: column) denotes the attention weight of the j-th query for the i-th output. Red indicates stronger attention. The attention output is normalized from 0 to 1.
注意力可视化。为了说明多头自注意力块的注意力机制，我们在Human3.6M测试集S11上针对特定动作（坐下）对我们的模型进行评估，并分别对空间transformer和时间transformer的自注意力图进行可视化，结果如图5和图6所示。对于空间自注意力图，x轴对应17个关节的查询，y轴表示注意力输出。如图5所示，不同注意力头输出不同的注意力强度，代表它们在输入关节之间学习到了不同的局部关联。我们发现3号注意力头聚焦于15号和16号关节，也就是右肘和右手腕。5号注意力头建立了左腿与左臂的关联（4、5、6号关节和11、12、13号关节），这些关节同属于身体左侧区域，而7号注意力头则聚焦于身体右侧（1、2、3号关节和12、13、14号关节）。 对于图6中的时间自注意力图，x轴对应81帧的查询，y轴表示注意力输出。不同注意力头学习到了不同的长程全局依赖关系。3号注意力头的注意力高度关联中心帧右侧的若干帧（例如第58、62和69帧）。7号注意力头即使在距离很远的情况下，也能捕捉到第1、20、22、42、78帧之间的依赖关系。空间和时间注意力图表明，PoseFormer成功对关节之间的局部关系建模，同时也捕捉到了整个输入序列的长程全局依赖关系。
Table 7. MPJPE evaluation on HumanEva test set. FT indicates using pre-trained model on Human3.6M for fine tuning.
walkjog
S1S2S3S1S2S3
PoseFormer $f = 43$16.311.047.125.015.215.1
PoseFormer $f = 43$ FT14.410.246.622.713.413.4
小数据集泛化能力。先前的研究如[^12]得出结论：transformer在数据量不足的情况下训练时，泛化性能较差。我们针对我们的模型开展实验，研究transformer在小型数据集HumanEva [^34]上的学习能力。与Human3.6M（超过100万帧）相比，该数据集规模小得多（少于5万帧）。表7展示了从零开始训练，以及使用在Human3.6M上预训练的模型进行微调的结果。我们发现微调后性能得到了大幅提升，这符合[^12] [^36]以往的观察结论：transformer在大规模数据集上预训练后可以取得良好的性能。 #### 5. 结论 本文提出了PoseFormer，这是一种纯基于Transformer的从2D视频进行3D姿态估计的方法。空间Transformer模块对2D关节之间的局部关系进行编码，时间Transformer模块则可捕获任意帧之间不受距离影响的全局依赖关系。大量实验表明，我们的模型在两个主流3D姿态数据集上取得了当前最优性能。 致谢：本工作得到了美国国家科学基金会项目编号1910844的部分支持。 --- ### 参考文献 [^1]: Alexei Baevski and Michael Auli. Adaptive input representations for neural language modeling. arXiv preprint arXiv:1809.10853, 2018. 4 [^2]: Y. Cai, L. Ge, J. Liu, J. Cai, T. Cham, J. Yuan, and N. M. Thalmann. Exploiting spatial-temporal relationships for 3d pose estimation via graph convolutional networks. In ICCV, 2019. 2,6 [^3]: Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. Realtime multi-person 2d pose estimation using part affinity fields. In CVPR, 2017. 2 [^4]: Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European Conference on Computer Vision, pages 213–229. Springer, 2020. 1, 3 [^5]: Tianlang Chen, Chen Fang, Xiaohui Shen, Yiheng Zhu, Zhili Chen, and Jiebo Luo. Anatomy-aware 3d human pose estimation with bone-based pose decomposition. IEEE Transactions on Circuits and Systems for Video Technology, 2021. 1, 2, 3, 5, 6, 7, 8 [^6]: Yilun Chen, Zhicheng Wang, Yuxiang Peng, Zhiqiang Zhang, Gang Yu, and Jian Sun. Cascaded pyramid network for multi-person pose estimation. In CVPR, 2018. 2 [^7]: Yilun Chen, Zhicheng Wang, Yuxiang Peng, Zhiqiang Zhang, Gang Yu, and Jian Sun. Cascaded pyramid network for multi-person pose estimation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7103–7112, 2018. 5 [^8]: Y. Cheng, B. Yang, B. Wang, Y. Wending, and R. Tan. Occlusion-aware networks for 3d human pose estimation in video. In ICCV, 2019. 2 [^9]: H. Ci, C. Wang, X. Ma, and Y. Wang. Optimizing network structure for 3d human pose estimation. In ICCV, 2019. 3 [^10]: Rishabh Dabral, Anurag Mundhada, Uday Kusupati, Safeer Afaque, Abhishek Sharma, and Arjun Jain. Learning 3d human pose from structure and motion. In ECCV, 2018. 2 [^11]: Rishabh Dabral, Anurag Mundhada, Uday Kusupati, Safeer Afaque, Abhishek Sharma, and Arjun Jain. Learning 3d human pose from structure and motion. In Proceedings of the European Conference on Computer Vision (ECCV), pages 668–683, 2018. 6 [^12]: Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR, 2021. 1, 3, 4, 5, 8 [^13]: Hao-Shu Fang, Shuqin Xie, Yu-Wing Tai, and Cewu Lu. Rmpe: Regional multi-person pose estimation. In ICCV, 2017. 2 [^14]: Kai Han, Yunhe Wang, Hanting Chen, Xinghao Chen, Jianyuan Guo, Zhenhua Liu, Yehui Tang, An Xiao, Chunjing Xu, Yixing Xu, et al. A survey on visual transformer. arXiv preprint arXiv:2012.12556, 2020. 3 [^15]: Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In 欧洲计算机视觉会议，第646–661页，Springer，2016年。 5 [^16]: C. Ionescu, D. Papava, V. Olaru, and C. Sminchisescu. Human3.6m: Large scale datasets and predictive methods for 3d human sensing in natural environments. IEEE TPAMI, 2014. 2, 5 [^17]: Salman Khan, Muzammal Naseer, Munawar Hayat, Syed Waqas Zamir, Fahad Shahbaz Khan, and Mubarak Shah. Transformers in vision: A survey. arXiv preprint arXiv:2101.01169, 2021. 3 [^18]: Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014. 5 [^19]: Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations (ICLR), 2017. 3 [^20]: Shichao Li, Lei Ke, Kevin Pratama, Yu-Wing Tai, Chi-Keung Tang, and Kwang-Ting Cheng. Cascaded deep monocular 3d human pose estimation with evolutionary training data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020. 7 [^21]: Z. Li, X. Wang, F. Wang, and P. Jiang. On boosting single-frame 3d human pose estimation via monocular videos. In ICCV, 2019. 2 [^22]: Jiahao Lin and Gim Hee Lee. Trajectory space factorization for deep video-based 3d human pose estimation. In BMVC, 2019. 5, 6, 7 [^23]: Kevin Lin, Lijuan Wang, and Zicheng Liu. End-to-end human pose and mesh reconstruction with transformers. arXiv preprint arXiv:2012.09760, 2020. 3, 5, 6 [^24]: Kenkun Liu, Rongqi Ding, Zhiming Zou, Le Wang, and Wei Tang. A comprehensive study of weight sharing in graph networks for 3d human pose estimation. In ECCV, 2020. 3 [^25]: Ruixu Liu, Ju Shen, He Wang, Chen Chen, Sen-ching Cheung, and Vijayan Asari. Attention mechanism exploits temporal contexts: Real-time 3d human pose reconstruction. In CVPR, 2020. 1, 2, 3, 5, 6, 7 [^26]: Julieta Martinez, Rayat Hossain, Javier Romero, and James J. Little. A simple yet effective baseline for 3d human pose estimation. In ICCV, 2017. 2, 3 [^27]: Dushyant Mehta, Helge Rhodin, Dan Casas, Pascal Fua, Oleksandr Sotnychenko, Weipeng Xu, and Christian Theobalt. Monocular 3d human pose estimation in the wild using improved cnn supervision. In 2017 international conference on 3D vision (3DV), pages 506–516. IEEE, 2017. 2, 5, 7 [^28]: Dushyant Mehta, Srinath Sridhar, Oleksandr Sotnychenko, Helge Rhodin, Mohammad Shafiei, Hans-Peter Seidel, Weipeng Xu, Dan Casas, and Christian Theobalt. Vnect: Real-time 3d human pose estimation with a single rgb camera. ACM Transactions on Graphics (TOG), 36(4):1–14, 2017. 5,7 [^29]: Gyeongsik Moon and Kyoung Mu Lee. I2l-meshnet: Image-to-lixel prediction network for accurate 3d human pose and mesh estimation from a single rgb image. In ECCV, 2020. 1 #### [^30]: Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Al- --- ban Desmaison、Luca Antiga和Adam Lerer。PyTorch中的自动微分，2017年。 5 [^31]: Georgios Pavlakos, Xiaowei Zhou, and Kostas Daniilidis. Ordinal depth supervision for 3D human pose estimation. In CVPR, 2018. 1, 6 [^32]: Dario Pavllo, Christoph Feichtenhofer, David Grangier, and Michael Auli. 3d human pose estimation in video with temporal convolutions and semi-supervised training. In CVPR, 2019. 2, 3, 5, 6, 7 [^33]: Mir Rayat Imtiaz Hossain and James J. Little. Exploiting temporal information for 3d human pose estimation. In The European Conference on Computer Vision (ECCV), 2018. 1, 2, 6, 7 [^34]: L. Sigal, A. Balan, and M. J. Black. HumanEva: Synchronized video and motion capture dataset and baseline algorithm for evaluation of articulated human motion. IJCV, 2010. 8 [^35]: Ke Sun, Bin Xiao, Dong Liu, and Jingdong Wang. Deep high-resolution representation learning for human pose estimation. In CVPR, 2019. 2 [^36]: Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. arXiv preprint arXiv:2012.12877, 2020. 1, 3, 8 [^37]: Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. 2017. 1 [^38]: Jingbo Wang, Sijie Yan, Yuanjun Xiong, and Dahua Lin. Motion guided 3d pose estimation from videos. In European Conference on Computer Vision, pages 764–780. Springer, 2020. 1, 2, 5, 6 [^39]: Qiang Wang, Bei Li, Tong Xiao, Jingbo Zhu, Changliang Li, Derek F Wong, and Lidia S Chao. Learning deep transformer models for machine translation. arXiv preprint arXiv:1906.01787, 2019. 4 [^40]: Sen Yang, Zhibin Quan, Mu Nie, and Wankou Yang. Transpose: Towards explainable human pose estimation by transformer. arXiv preprint arXiv:2012.14214, 2020. 3 [^41]: Linwei Ye, Mrigank Rochan, Zhi Liu, and Yang Wang. Cross-modal self-attention network for referring image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10502–10511, 2019. 1 [^42]: Raymond A Yeh, Yuan-Ting Hu, and Alexander G Schwing. Chirality nets for human pose regression. 2019. 6 [^43]: Ailing Zeng, Xiao Sun, Fuyang Huang, Minhao Liu, Qiang Xu, and Stephen Lin. Srnet: Improving generalization in 3d human pose estimation with a split-and-recombine approach. In ECCV, 2020. 1, 6, 7 [^44]: Tianshu Zhang, Buzhen Huang, and Yangang Wang. Object-occluded human shape and pose estimation from a single color image. In CVPR, 2020. 2 [^45]: Long Zhao, Xi Peng, Yu Tian, Mubbasir Kapadia, and Dimitris N. Metaxas. Semantic graph convolutional networks for 3d human pose regression. In CVPR, 2019. 3 [^46]: Ce Zheng, Wenhan Wu, Taojiannan Yang, Sijie Zhu, Chen Chen, Ruixu Liu, Ju Shen, Nasser Kehtarnavaz, and Mubarak Shah。基于深度学习的人体姿态估计：综述，2020年。 5 [^47]: Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip HS Torr, et al. Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers. arXiv preprint arXiv:2012.15840, 2020. 1 [^48]: Kaixiong Zhou, Xiao Huang, Yuening Li, Daochen Zha, Rui Chen, and Xia Hu. Towards deeper graph neural networks with differentiable group normalization. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 4917–4928. Curran Associates, Inc., 2020. 3 [^49]: Xingyi Zhou, Qixing Huang, Xiao Sun, Xiangyang Xue, and Yichen Wei. Towards 3d human pose estimation in the wild: A weakly-supervised approach. In ICCV, 2017. 2 [^50]: Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint arXiv:2010.04159, 2020. 1