{
  "paper_id": "530a73329e48bf883f25d81de7dc54fb",
  "paper_title": "BEVPose: Unveiling Scene Semantics through Pose-Guided Multi-Modal BEV Alignment",
  "paper_authors": [
    "Mehdi Hosseinzadeh",
    "Ian Reid"
  ],
  "publication_date": "",
  "publication_venue": "2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)",
  "abstract": "In the field of autonomous driving and mobile robotics, there has been a significant shift in the methods used to create Bird's Eye View (BEV) representations. This shift is characterised by using transformers and learning to fuse measurements from disparate vision sensors, mainly lidar and cameras, into a 2D planar ground-based representation. However, these learning-based methods for creating such maps often rely heavily on extensive annotated data, presenting notable challenges, particularly in diverse or non-urban environments where large-scale datasets are scarce. In this work, we present BEVPose, a framework that integrates BEV representations from camera and lidar data, using sensor pose as a guiding supervisory signal. This method notably reduces the dependence on costly annotated data. By leveraging pose information, we align and fuse multi-modal sensory inputs, facilitating the learning of latent BEV embeddings that capture both geometric and semantic aspects of the environment. Our pretraining approach demonstrates promising performance in BEV map segmentation tasks, outperforming fully-supervised state-of-the-art methods, while necessitating only a minimal amount of annotated data. This development not only confronts the challenge of data efficiency in BEV representation learning but also broadens the potential for such techniques in a variety of domains, including off-road and indoor environments.\n",
  "keywords": [
    "BEV representation",
    "self-supervised learning",
    "multi-modal fusion",
    "pose supervision",
    "semantic segmentation",
    "autonomous driving"
  ],
  "paper_institutions": [],
  "output_language": "zh",
  "provider": "doubao",
  "model": "deepseek-v4-flash",
  "prompt_template": "simple",
  "extracted_at": "2026-08-18T10:30:15.781015Z",
  "template_tag": "simple",
  "summary": "### 摘要\n\n本文针对自动驾驶与移动机器人中鸟瞰图（BEV）表示学习依赖大量昂贵标注数据的问题，提出BEVPose框架。与现有基于Transformer和全监督多传感器融合的BEV方法不同，BEVPose利用相机与激光雷达的传感器位姿作为自监督信号，通过对比学习对齐多模态BEV特征，从而在无需分割标注的情况下学习蕴含几何与语义信息的潜在BEV嵌入。方法上，多视角相机图像经LSS式深度分布提升为相机BEV，激光雷达点云经体素化和垂直池化得到激光雷达BEV，两者拼接后经MLP编码融合；核心对齐机制是以激光雷达BEV为参考、相机BEV为查询，在已知相对位姿下最大化正样本一致性并最小化负样本一致性，其对比损失为 L_align = -log( exp(s(B_cam,B_lidar;T_c^l+)/τ) / Σ_j exp(s(B_cam,B_lidar;T_c_j^l)/τ) )，其中相似度通过对齐后对应BEV特征的逐点内积求和得到。实验采用nuScenes数据集，图像分辨率256×704，激光雷达体素0.1m，深度范围1–80m、158个离散深度值，使用Swin-Transformer、FPN和VoxelNet作为骨干，并用AdamW优化；在BEV地图语义分割任务上，BEVPose在100%微调数据下mIoU达67.8%，超过全监督BEVFusion的62.7%，且仅用约44%标注数据即可匹配BEVFusion，用20%数据微调仍能达到59.7% mIoU。消融实验表明相机+激光雷达联合微调最佳，0.1m体素和158个深度值更优；PCA可视化显示所学嵌入能捕捉道路、交叉口等高层语义与几何结构，传感器缺失鲁棒性实验也表明框架在单模态输入下仍能保持可用性能。未来工作将引入时间约束、扩展至越野与室内环境、2D/3D目标检测以及深度传感器和雷达等模态；实际应用中，BEVPose尤其适合标注数据稀缺的非城市、越野和室内场景，但当前未显式利用双目视觉或RGB-D等相机约束，这是后续改进方向。\n\n### 关键要点\n\n- BEVPose提出以传感器位姿作为自监督信号的多模态BEV预训练框架，旨在减少BEV表示学习对昂贵分割标注的依赖。\n- 方法将多视角相机图像通过LSS式深度分布估计提升为相机BEV，并将激光雷达点云体素化后经3D编码和垂直池化生成激光雷达BEV。\n- 相机BEV与激光雷达BEV经拼接和逐单元最大池化后送入MLP BEV编码器，以处理多模态特征在BEV网格中的潜在错位。\n- 核心对比对齐损失以激光雷达BEV为参考、各相机BEV为查询，利用已知相对位姿构造正负样本，并通过对齐后BEV特征内积和计算相似度。\n- 负样本采样策略围绕其他无重叠相机的位姿分布进行，以挑选高似然但错误的难负样本，引导模型向真实位姿对应的对齐方向优化。\n- 实验采用nuScenes数据集，包含850个训练/验证场景和150个测试场景，使用6个相机和1个激光雷达，分割任务关注6类背景“stuff”标注。\n- 实现设置包括Swin-Transformer图像骨干、FPN颈部、VoxelNet激光雷达骨干、图像尺寸256×704、激光雷达体素0.1m、深度范围1–80m且离散为158个深度值，优化器为AdamW。\n- 在100%微调数据下，BEVPose的mIoU达到67.8%，超过BEVFusion的62.7%以及LSS、CVT、CenterPoint、MVP等单模态和多模态基线。\n- 数据效率实验中，BEVPose仅用约44%的标注训练数据即可匹配BEVFusion全量监督性能，用20%数据微调时mIoU为59.7%，仍接近全监督结果。\n- 消融研究表明相机+激光雷达联合微调优于单一模态，0.1m体素和158个深度离散值设置最佳；PCA可视化显示所学BEV嵌入能呈现道路、交叉口等语义几何结构。\n- 传感器缺失鲁棒性实验显示，相机-only模式覆盖更广但几何精度较低，激光雷达-only模式几何更准但范围有限，融合模式在传感器缺失时仍保持稳定性能。\n- 当前局限是未利用双目视觉或RGB-D等显式多视图相机约束，未来计划引入时间轨迹约束、拓展到越野/室内环境、2D/3D检测以及深度传感器和雷达等模态。"
}