{
  "paper_id": "afbf79188568b451b57b44bd43e23b30",
  "paper_title": "3D Human Pose Estimation with Spatial and Temporal Transformers",
  "paper_authors": [
    "Ce Zheng",
    "Sijie Zhu",
    "Matias Mendieta",
    "Taojiannan Yang",
    "Chen Chen",
    "Zhengming Ding"
  ],
  "publication_date": "",
  "publication_venue": "2021 IEEE/CVF International Conference on Computer Vision (ICCV)",
  "abstract": "Transformer architectures have become the model of choice in natural language processing and are now being introduced into computer vision tasks such as image classification, object detection, and semantic segmentation. However, in the field of human pose estimation, convolutional architectures still remain dominant. In this work, we present PoseFormer, a purely transformer-based approach for 3D human pose estimation in videos without convolutional architectures involved. Inspired by recent developments in vision transformers, we design a spatial-temporal transformer structure to comprehensively model the human joint relations within each frame as well as the temporal correlations across frames, then output an accurate 3D human pose of the center frame. We quantitatively and qualitatively evaluate our method on two popular and standard benchmark datasets: Human3.6M and MPI-INF-3DHP. Extensive experiments show that PoseFormer achieves state-of-the-art performance on both datasets. Code is available at https://github.com/zczcwh/PoseFormer",
  "keywords": [
    "3D human pose estimation",
    "transformer",
    "spatial-temporal transformer",
    "video-based pose estimation",
    "2D-to-3D lifting"
  ],
  "paper_institutions": [
    "Center for Research in Computer Vision, University of Central Florida, USA",
    "Department of Computer Science, Tulane University, USA"
  ],
  "output_language": "zh",
  "provider": "doubao",
  "model": "ark-code-latest",
  "prompt_template": "simple",
  "extracted_at": "2026-08-16T19:47:44.605225Z",
  "template_tag": "simple",
  "summary": "### 摘要\n\n本文提出PoseFormer，首个纯Transformer架构的2D-to-3D视频人体姿态估计方法，旨在解决现有基于CNN或RNN方法在捕获长期时间依赖上的局限。该方法采用2D-to-3D lifting流程，输入从现成2D检测器获得的连续帧2D姿态序列，通过空间Transformer模块建模单帧内关节间的局部关系，再通过时间Transformer模块捕获整个帧序列的全局依赖，最后利用回归头预测中心帧的3D姿态。核心机制为将每帧的关节位置作为token，分别进行空间和时间维度的自注意力建模，并通过可学习的位置编码保留空间与时间顺序信息。实验在两个标准基准数据集Human3.6M和MPI-INF-3DHP上验证，采用MPJPE、P-MPJPE、PCK和AUC等指标。在Human3.6M上，使用CPN检测的2D姿态作为输入时，平均MPJPE达到44.3mm（Protocol 1），优于此前最优方法；使用真实2D姿态作为输入时平均MPJPE降至31.3mm。在MPI-INF-3DHP上，PCK达到88.6%，AUC为56.4%，MPJPE为77.1mm，均取得最佳结果。PoseFormer的空间Transformer能够学习关节间有价值的连接，时间Transformer可捕获远距离帧的依赖，消融实验验证了各组件及超参数的有效性。该方法适用于需要时序平滑性的视频3D姿态估计场景，例如人机交互、运动分析和医疗健康等，但受限于2D检测器质量以及较长序列带来的计算开销（FLOPs随帧数线性增加），不过推理速度仍可满足实时应用。主要机制为缩放点积注意力：Attention(Q,K,V)=Softmax(QK^T/√d)V，配合多头部自注意力和残差连接，分别用于空间和时间Transformer编码器。\n\n### 关键要点\n- PoseFormer是首个纯Transformer的2D-to-3D视频人体姿态估计方法，不依赖卷积架构。\n- 模型包含空间Transformer模块和时间Transformer模块，分别建模关节间关系与帧间依赖。\n- 每帧的每个2D关节坐标作为空间token，通过可学习线性投影和空间位置编码形成嵌入。\n- 时间Transformer将空间编码后的特征序列作为输入，并添加时间位置编码以保留帧序信息。\n- 损失函数采用标准MPJPE（平均每关节位置误差），公式为 L=(1/J)Σ||p_k - p̂_k||₂。\n- 在Human3.6M上，使用CPN检测的2D姿态（f=81）时，Protocol 1的MPJPE为44.3mm，优于基线6.1%。\n- 与忽视时序的Transformer方法METRO相比，PoseFormer的MPJPE降低约18%。\n- 在MPI-INF-3DHP上，使用9帧输入即获得最佳PCK（88.6%）、AUC（56.4%）和MPJPE（77.1mm）。\n- 使用真实2D姿态作为输入时，Human3.6M上MPJPE降至31.3mm，比SRNet提升约2%。\n- 消融实验表明，同时使用空间和时间位置编码可显著提升性能（MPJPE从51.6降至49.9）。\n- 注意力可视化显示，空间注意力头关注不同身体部位（如右肘/右腕、左侧肢体），时间注意力头捕获远距离帧的依赖。\n- 模型参数较少（f=81时约9.6M），推理速度269 FPS（RTX 2080 Ti），不成为实时系统瓶颈，但2D检测器通常低于80 FPS。\n- 在小型数据集HumanEva上，从零训练效果一般，但经过Human3.6M预训练微调后性能大幅提升，验证了Transformer需要大数据预训练的特性。"
}