{
  "paper_id": "4256cfb50f18a3ae8be51fd0de763dc1",
  "paper_title": "Multi-modal 3D Human Pose Estimation with 2D Weak Supervision in Autonomous Driving",
  "paper_authors": [
    "Jingxiao Zheng",
    "Xinwei Shi",
    "Alexander Gorban",
    "Junhua Mao",
    "Yang Song",
    "Charles R. Qi",
    "Ting Liu",
    "Visesh Chari",
    "Andre Cornman",
    "Yin Zhou",
    "Congcong Li",
    "Dragomir Anguelov"
  ],
  "publication_date": "",
  "publication_venue": "2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)",
  "abstract": "3D human pose estimation (HPE) in autonomous vehicles (AV) differs from other use cases in many factors, including the 3D resolution and range of data, absence of dense depth maps, failure modes for LiDAR, relative location between the camera and LiDAR, and a high bar for estimation accuracy. Data collected for other use cases (such as virtual reality, gaming, and animation) may therefore not be usable for AV applications. This necessitates the collection and annotation of a large amount of 3D data for HPE in AV, which is time-consuming and expensive. In this paper, we propose one of the first approaches to alleviate this problem in the AV setting. Specifically, we propose a multi-modal approach which uses 2D labels on RGB images as weak supervision to perform 3D HPE. The proposed multi-modal architecture incorporates LiDAR and camera inputs with an auxiliary segmentation branch. On the Waymo Open Dataset, our approach achieves a ~22% relative improvement over camera-only 2D HPE baseline, and ~6% improvement over LiDAR-only model. Finally, careful ablation studies and parts based analysis illustrate the advantages of each of our contributions.",
  "keywords": [
    "3D human pose estimation",
    "autonomous driving",
    "multi-modal learning",
    "weak supervision",
    "LiDAR",
    "camera",
    "modality fusion",
    "auxiliary segmentation"
  ],
  "paper_institutions": [
    "Waymo LLC",
    "Google Research"
  ],
  "output_language": "zh",
  "provider": "doubao",
  "model": "ark-code-latest",
  "prompt_template": "simple",
  "extracted_at": "2026-08-16T19:47:48.696931Z",
  "template_tag": "simple",
  "summary": "### 摘要\n自主驾驶（AV）场景下的3D人体姿态估计（HPE）与动画、游戏、虚拟现实等应用相比具有显著差异，主要体现在3D数据分辨率和范围、缺少密集深度图、LiDAR的特定失效模式、相机与LiDAR的相对位置以及更高的精度要求。现有其他场景数据集不适用，且大规模采集三维标注代价高昂。为此，本文提出一种多模态方法，利用RGB图像上的2D关键点标签作为弱监督进行3D HPE，无需昂贵的3D标注。所提架构级联了基于CNN的相机网络和基于PointNet的点网络：相机网络从RGB图像预测2D热图，通过模态融合将纹理特征采样到LiDAR点云；点网络回归3D关键点，并配有辅助逐点分割分支以增强监督。模型仅用2D标签生成伪3D标签（对近邻3D点按2D距离加权平均）和逐点点标签进行训练。在Waymo Open Dataset上，该方法在OKS@3D和MPJPE上比仅相机模型（2D预测后提升）相对提升约22%，比仅LiDAR模型相对提升约6%；在内部数据集上OKS@2D也达到最高。消融研究验证了模态融合和分割分支的互补增益，且对困难部位（肘、腕）改善尤为明显。未来可探索更大规模训练、处理长尾情况，并关注在真实驾驶场景中的实时性和鲁棒性。\n\n### 关键要点\n- 明确提出自主驾驶场景下3D人体姿态估计面临的独特挑战，如传感器特性、数据标注成本和场景多样性，与现有其他应用场景有本质区别。\n- 提出一种多模态级联架构，利用相机网络（ResNet50+反卷积）生成2D热图，并通过高斯平滑和采样将纹理特征融合到LiDAR点云，增强点网络输入。\n- 采用PointNet风格的点网络作为主回归器，输入为每个点坐标（3维）与采样热图（K维）的拼接，输出K个3D关键点坐标。\n- 设计辅助逐点分割分支，为每个LiDAR点预测其最近关键点类型，通过交叉熵损失提供细粒度监督，提升模型对点级特征的利用。\n- 提出弱监督标签生成方法：基于2D关键点标签和点-像投影，对每个2D关键点邻近的3D点按距离加权平均生成伪3D标签，并计算可靠性权重以调整损失。\n- 使用温度参数T的softmax加权公式生成伪3D坐标，并定义可靠性r_k = exp(-T_r * min_i ||x_i^(2)-y_k^(2)||^2) 来降低不准确标签的影响。\n- 训练时仅对可见关键点施加损失，但实验表明模型仍能对遮挡部位给出合理预测，体现了泛化能力。\n- 在Waymo Open Dataset（3D评估，986个行人标注）和内部数据集（155,182训练/42,199测试，纯2D标注）上进行实验，使用OKS@3D、MPJPE和OKS@2D作为指标。\n- 多模态模型在Waymo上OKS@3D达到63.14%，优于仅相机的51.74%和仅LiDAR的59.58%；MPJPE降低至10.32cm，较仅LiDAR的10.80cm进一步优化。\n- 在内部数据集上，多模态模型的OKS@2D为82.94%，优于仅相机的78.19%和仅LiDAR的77.53%，表明点云深度信息也辅助2D预测。\n- 消融研究显示，加入辅助分割分支带来约1.8%相对提升，加入模态融合带来约6%相对提升，两者叠加效果最佳，证明其互补性。\n- 相机图像尺寸和骨干网络的影响实验表明，更大尺寸（256x256）和更深骨干（ResNet50）对困难关键点（肘、腕）改善显著，但对其他点可能略降，需要权衡。",
  "available_templates": [
    "deep_read",
    "simple"
  ]
}