{
  "paper_id": "f545d72d989becc423cb5c2e5ca81c50",
  "paper_title": "WHAM: Reconstructing World-grounded Humans with Accurate 3D Motion",
  "paper_authors": [
    "Soyong Shin",
    "Juyong Kim",
    "Eni Halilaj",
    "Michael J. Black"
  ],
  "publication_date": "",
  "publication_venue": "2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)",
  "abstract": "The estimation of 3D human motion from video has progressed rapidly but current methods still have several key limitations. First, most methods estimate the human in camera coordinates. Second, prior work on estimating humans in global coordinates often assumes a flat ground plane and produces foot sliding. Third, the most accurate methods rely on computationally expensive optimization pipelines, limiting their use to offline applications. Finally, existing video-based methods are surprisingly less accurate than single-frame methods. We address these limitations with WHAM (World-grounded Humans with Accurate Motion), which accurately and efficiently reconstructs 3D human motion in a global coordinate system from video. WHAM learns to lift 2D keypoint sequences to 3D using motion capture data and fuses this with video features, integrating motion context and visual information. WHAM exploits camera angular velocity estimated from a SLAM method together with human motion to estimate the body's global trajectory. We combine this with a contact-aware trajectory refinement method that lets WHAM capture human motion in diverse conditions, such as climbing stairs. WHAM outperforms all existing 3D human motion recovery methods across multiple in-the-wild benchmarks. Code is available for research purposes at http://wham.is.tue.mpg.de/.",
  "keywords": [
    "3D human pose and shape estimation",
    "global coordinate system",
    "video-based method",
    "moving camera",
    "foot sliding",
    "SLAM",
    "contact-aware trajectory refinement",
    "AMASS dataset",
    "2D keypoint lifting",
    "feature integration"
  ],
  "paper_institutions": [
    "Carnegie Mellon University",
    "Max Planck Institute for Intelligent System"
  ],
  "output_language": "zh",
  "provider": "doubao",
  "model": "ark-code-latest",
  "prompt_template": "simple",
  "extracted_at": "2026-08-16T19:47:23.657139Z",
  "template_tag": "simple",
  "summary": "### 摘要\n\n本文提出WHAM（World-grounded Humans with Accurate Motion）方法，旨在从移动相机拍摄的单目视频中，在全局坐标系中准确且高效地重建3D人体运动。现有方法主要存在四个问题：多数方法在相机坐标系中估计人体；全局坐标估计常假设平坦地面并产生脚滑；高精度方法依赖昂贵的优化流程，无法实时应用；视频方法的精度反而不如单帧方法。WHAM采用两阶段训练：首先利用大规模运动捕捉数据集AMASS合成2D关键点序列，预训练运动编码器和解码器，学习从2D关键点序列提升（lift）到3D姿态；随后在真实视频数据集上微调，通过特征集成网络将图像特征与运动特征融合，补充密集视觉上下文，提高姿态和形状精度。为估计全局轨迹，WHAM利用SLAM或陀螺仪获得的相机角速度，与运动特征结合，通过轨迹解码器递归预测全局朝向和速度，并采用接触感知的轨迹细化模块，基于脚地接触概率调整速度，消除脚滑并支持楼梯等非平坦地形。WHAM在3DPW、RICH、EMDB等多个野外基准上均达到最先进水平，超越所有现有图像和视频方法，同时保持时间平滑，且运行速度高（网络部分200 fps），适用于实时应用。核心机制包括：将相机角速度拼接至运动特征以分解相机运动，以及通过接触感知速度修正和轨迹细化网络实现无滑动的全局轨迹重建。未来方向包括利用SLAM提供场景信息进行联合优化，以及与束调整结合等。\n\n### 关键要点\n- WHAM是首个将3D运动上下文与视频上下文有效融合的3D人体姿态和形状回归方法，克服了视频方法精度低于单帧方法的反常现象。\n- 利用AMASS大规模运动捕捉数据预训练网络，通过合成2D关键点序列（采用动态虚拟相机投影）实现2D到3D的关键点提升，为后续轨迹估计奠定基础。\n- 特征集成网络采用残差连接，将预训练图像编码器提取的视觉特征与运动特征融合，补充关键点稀疏性，提升姿态和形状精度。\n- 全局轨迹解码器以运动特征和相机角速度（来自SLAM或陀螺仪）为输入，递归预测全局朝向和速度，从而分解相机运动与人体运动。\n- 接触感知轨迹细化模块根据脚地接触概率调整根速度，减少脚滑，并利用细化网络更新朝向和速度，支持楼梯等非平坦地面场景。\n- 在3DPW、RICH、EMDB三个野外数据集上，WHAM（ViT）在MPJPE、PA-MPJPE、PVE等指标上均优于现有方法，且加速度误差低于单帧方法，证明其精度和光滑性。\n- 在EMDB 2全局轨迹评估中，WHAM（使用DROID-SLAM）的W-MPJPE_100为343.9mm，RTE为4.6%，远优于GLAMR、TRACE、SLAHMR等基线，且脚滑和抖动显著降低。\n- 消融实验表明，特征集成、2D-3D提升预训练、神经网络初始化、相机角速度、轨迹细化模块均对性能有贡献；去除轨迹细化会增加脚滑和轨迹误差。\n- 训练数据包括AMASS、3DPW、Human3.6M、MPI-INF-3DHP、InstaVariety，并实验性地加入BEDLAM合成数据集，进一步提升精度。\n- WHAM网络在线运行，除预处理外速度达200 fps，远快于优化方法，适合实时应用，如游戏、AR/VR、自动驾驶、体育分析和人机交互。\n- 局限性：依赖AMASS运动分布，对分布外运动泛化有限；数据合成主要假设全身在视野内，部分遮挡情况处理欠佳。未来可结合SLAM场景信息进行联合优化，并探索手机实时部署。"
}