{
  "paper_id": "723fafa805bdd28a145c9421cb624b4f",
  "paper_title": "Accurate detection and 3D localization of humans using a novel YOLO-based RGB-D fusion approach and synthetic training data",
  "paper_authors": [
    "Timm Linder",
    "Kilian Y. Pfeiffer",
    "Narunas Vaskevicius",
    "Robert Schirmer",
    "Kai O. Arras"
  ],
  "publication_date": "",
  "publication_venue": "2020 IEEE International Conference on Robotics and Automation (ICRA)",
  "abstract": "While 2D object detection has made significant progress, robustly localizing objects in 3D space under presence of occlusion is still an unresolved issue. Our focus in this work is on real-time detection of human 3D centroids in RGB-D data. We propose an image-based detection approach which extends the YOLO v3 architecture with a 3D centroid loss and mid-level feature fusion to exploit complementary information from both modalities. We employ a transfer learning scheme which can benefit from existing large-scale 2D object detection datasets, while at the same time learning end-to-end 3D localization from our highly randomized, diverse synthetic RGB-D dataset with precise 3D groundtruth. We further propose a geometrically more accurate depth-aware crop augmentation for training on RGB-D data, which helps to improve 3D localization accuracy. In experiments on our challenging intralogistics dataset, we achieve state-of-the-art performance even when learning 3D localization just from synthetic data.",
  "keywords": [
    "RGB-D fusion",
    "3D localization",
    "YOLO v3",
    "synthetic data",
    "human detection"
  ],
  "paper_institutions": [],
  "output_language": "zh",
  "provider": "doubao",
  "model": "ark-code-latest",
  "prompt_template": "simple",
  "extracted_at": "2026-08-16T19:46:54.416186Z",
  "template_tag": "simple",
  "summary": "### 摘要\n本文针对遮挡条件下人类3D质心的实时检测问题，提出了一种基于YOLO v3的RGB-D融合方法。该方法通过添加3D质心回归损失和中级特征融合，有效利用了RGB和深度信息的互补性。采用迁移学习策略，利用大规模2D检测数据集预训练，并结合高度随机化的合成RGB-D数据集进行端到端3D定位学习。同时，提出一种深度感知的裁剪增强方法，改善3D定位精度。在具有挑战性的内部物流数据集上，仅使用合成数据训练即可达到最先进的性能，验证了该方法在复杂室内场景中的有效性。\n\n### 关键要点\n- 该方法扩展了YOLO v3架构，增加3D质心回归输出，实现端到端3D定位，无需依赖点云表示。\n- 通过中级特征融合，同时处理RGB和深度流，在特征提取阶段结合互补信息，对缺失深度和遮挡具有鲁棒性。\n- 迁移学习策略利用ImageNet和MS COCO预训练权重，复用2D检测能力，不影响2D性能。\n- 合成数据集包含15,000帧RGB-D图像，涉及6个场景、700多个遮挡物，并使用Kinect v2噪声模型模拟真实传感器。\n- 深度感知增强根据缩放因子调整深度值和输入图像，保持度量尺度不变，显著提升3D定位准确率。\n- 在合成验证集上，加入RGB+D融合后3D AP从46.5%提升至60.6%（0.25m阈值）。\n- 真实世界测试中，仅用合成数据训练的RGB+D模型在0.25m阈值下peak-F1达到72.5%，优于多种基线。\n- 结合合成和真实数据（S+R）时，0.25m和0.5m阈值下peak-F1分别达到78.6%和95.3%。\n- 与基于点云的方法相比，本文方法在部分遮挡和缺失深度场景下表现更稳健，可覆盖更广的RGB视野。\n- 该方法在GPU上运行速度达25 Hz，满足实时性要求，适用于AGV等机器人平台。\n- 3D质心回归使用L1损失用于深度，二元交叉熵用于图像坐标，并限制质心在检测框内。\n- 真实数据标注利用离线3D姿态估计生成弱质心标签，无需手动3D注释。"
}