{
  "paper_id": "f1a7fca0f12af1748558079fae16a021",
  "paper_title": "HUM3DIL: Semi-supervised Multi-modal 3D Human Pose Estimation for Autonomous Driving",
  "paper_authors": [
    "Andrei Zanfir",
    "Mihai Zanfir",
    "Alexander Gorban",
    "Jingwei Ji",
    "Yin Zhou",
    "Dragomir Anguelov",
    "Cristian Sminchisescu"
  ],
  "publication_date": "",
  "publication_venue": "unknown",
  "abstract": "Autonomous driving is an exciting new industry, posing important research questions. Within the perception module, 3D human pose estimation is an emerging technology, which can enable the autonomous vehicle to perceive and understand the subtle and complex behaviors of pedestrians. While hardware systems and sensors have dramatically improved over the decades – with cars potentially boasting complex LiDAR and vision systems and with a growing expansion of the available body of dedicated datasets for this newly available information – not much work has been done to harness these novel signals for the core problem of 3D human pose estimation. Our method, which we coin HUM3DIL (HUMAN 3D from Images and LiDAR), efficiently makes use of these complementary signals, in a semi-supervised fashion and outperforms existing methods with a large margin. It is a fast and compact model for onboard deployment. Specifically, we embed LiDAR points into pixel-aligned multi-modal features, which we pass through a sequence of Transformer refinement stages. Quantitative experiments on the Waymo Open Dataset support these claims, where we achieve state-of-the-art results on the task of 3D pose estimation.",
  "keywords": [
    "autonomous driving",
    "perception",
    "human pose",
    "key points",
    "skeletal representation"
  ],
  "paper_institutions": [
    "Google Research",
    "Waymo Research"
  ],
  "output_language": "zh",
  "provider": "doubao",
  "model": "ark-code-latest",
  "prompt_template": "simple",
  "extracted_at": "2026-08-16T19:47:39.954815Z",
  "template_tag": "simple",
  "summary": "### 摘要\n本文提出 HUM3DIL，一个面向自动驾驶场景的轻量级半监督多模态 3D 人体姿态估计网络。该问题源于自动驾驶车辆在复杂真实环境中对行人细微姿态理解的需求，现有 RGB 单模态方法在遮挡、光照变化和距离变化下精度不足，且计算开销大。相关研究多依赖图像或深度信息，很少有效利用 LiDAR 点云与图像的互补性。HUM3DIL 将 LiDAR 点云投影到图像平面，生成深度图并与 RGB 图像拼接，通过轻量 U-Net 提取像素对齐的多模态特征；对 LiDAR 点使用随机傅里叶特征嵌入，并与相机内参、图像特征拼接成 token 序列，经多层 Transformer 编码器处理，最终由 MLP 回归 3D 关节位置。模型采用半监督训练，利用大量 2D 标注和少量 3D 标注，损失函数包含 3D 重构损失与 2D 重投影损失。在 Waymo Open Dataset v1.3.2 的 WHKP 子集上，HUM3DIL 以 6.72 cm 的 3D MPJPE 和 8.33 像素的 2D MPJPE 大幅优于 ContextPose、THUNDR 等基线，且参数量仅为对比方法的 1/5 至 1/14，单张推理仅需 8 毫秒。该网络适用于车载实时感知，能提供细粒度人体关键点信号以支持行为分析和预测。其核心机制是结合像素对齐图像特征与点云信息的 Transformer 多模态融合，并通过半监督损失充分利用标注资源。\n\n### 关键要点\n- HUM3DIL 融合 RGB 图像与 LiDAR 点云，利用像素对齐多模态特征进行 3D 人体关键点预测。\n- 采用深度图与 RGB 拼接作为 U-Net 输入，深度通道提供稀疏但关键的深度线索，有助于在遮挡和拥挤场景中解耦目标。\n- LiDAR 点通过随机傅里叶特征（RFF）嵌入高维空间，以捕捉高频空间信息。\n- Transformer 编码器处理 token 序列，token 由相机内参、图像特征和傅里叶嵌入拼接而成，并加入可学习的关键点 token 用于回归。\n- 半监督训练利用大量 2D 关键点标注和少量 3D 标注，损失为 3D 重构损失与 2D 重投影损失的加权和，训练批次中按比例混合 2D/3D 样本。\n- 在 Waymo Open Dataset v1.3.2（WHKP）上，HUM3DIL 的 3D MPJPE 为 6.72 cm，2D MPJPE 为 8.33 像素，明显优于 THUNDR（9.62 cm）和 ContextPose（10.82 cm）。\n- 模型参数量仅 5.3M，约为对比方法的 1/5 至 1/14，推理时间 8 ms/帧，适合车载部署。\n- 消融实验显示移除 2D 监督损失使 3D 误差升至 8.62 cm，移除 RGB 或深度输入分别导致 3D 误差升至 8.06 和 7.63 cm，证明多模态和半监督机制均有益。\n- 失败案例主要出现在极端遮挡、部分视野和异常外观场景，误差在目标过近或过远时增大，但重建结果仍具有解剖合理性。\n- 当前模型仅处理单个人体，未利用多人和时序信息，未来工作可引入时序一致性以进一步稳定预测。"
}