{
  "paper_id": "dc02efe9d33f332980969537dc7bbe88",
  "paper_title": "HumanoidPano: Hybrid Spherical Panoramic-LiDAR Cross-Modal Perception for Humanoid Robots",
  "paper_authors": [
    "Qiang Zhang",
    "Zhang Zhang",
    "Wei Cui",
    "Jingkai Sun",
    "Jiahang Cao",
    "Yijie Guo",
    "Gang Han",
    "Wen Zhao",
    "Jiaxue Wang",
    "Chenghao Sun",
    "Lingfeng Zhang",
    "Hao Cheng",
    "Yujie Chen",
    "Lin Wang",
    "Jian Tang",
    "Renjing Xu"
  ],
  "publication_date": "",
  "publication_venue": "unknown",
  "abstract": "The perceptual system design for humanoid robots poses unique challenges due to inherent structural constraints that cause severe self-occlusion and limited field-of-view (FOV). We present HumanoidPano, a novel hybrid cross-modal perception framework that synergistically integrates panoramic vision and LiDAR sensing to overcome these limitations. Unlike conventional robot perception systems that rely on monocular cameras or standard multi-sensor configurations, our method establishes geometrically-aware modality alignment through a spherical vision transformer, enabling seamless fusion of 360° visual context with LiDAR's precise depth measurements. First, Spherical Geometry-aware Constraints (SGC) leverage panoramic camera ray properties to guide distortion-regularized sampling offsets for geometric alignment. Second, Spatial Deformable Attention (SDA) aggregates hierarchical 3D features via spherical offsets, enabling efficient 360°-to-BEV fusion with geometrically complete object representations. Third, Panoramic Augmentation (AUG) combines cross-view transformations and semantic alignment to enhance BEV-panoramic feature consistency during data augmentation. Extensive evaluations demonstrate state-of-the-art performance on the 360BEV-Matterport benchmark. Real-world deployment on humanoid platforms validates the system's capability to generate accurate BEV segmentation maps through panoramic-LiDAR co-perception, directly enabling downstream navigation tasks in complex environments. Our work establishes a new paradigm for embodied perception in humanoid robotics. The codes will be released soon.\n",
  "keywords": [
    "人形机器人",
    "全景感知",
    "LiDAR",
    "跨模态感知",
    "BEV语义分割",
    "球面可变形注意力",
    "HumanoidPano"
  ],
  "paper_institutions": [
    "Beijing Innovation Center of Humanoid Robotics",
    "Hong Kong University of Science and Technology (Guangzhou)",
    "Hong Kong University of Science and Technology",
    "Beijing Institute of Technology",
    "ETH Zurich",
    "Nanyang Technological University"
  ],
  "output_language": "zh",
  "provider": "doubao",
  "model": "deepseek-v4-flash",
  "prompt_template": "simple",
  "extracted_at": "2026-08-18T10:32:06.811782Z",
  "template_tag": "simple",
  "summary": "### 摘要\n\n人形机器人因仿生结构存在自遮挡和视场受限问题，传统单目或标准多传感器感知方案难以适应其形态约束；已有工作多依赖高程图、深度图像或模仿学习，缺乏大范围语义理解能力。针对这一挑战，HumanoidPano提出一种混合跨模态感知框架，将360°全景视觉与LiDAR深度感知协同融合，通过球面视觉Transformer建立几何感知的模态对齐。其核心机制可表示为M=F_seg(I,D)，其中I为输入全景图像，D为深度/LiDAR深度图，M为输出的BEV语义分割图；框架通过球面几何约束(SGC)利用全景相机射线属性引导畸变正则化采样偏移，通过空间可变形注意力(SDA)在球面偏移下聚合3D特征并完成360°到BEV的融合，并通过全景增强(AUG)联合图像与BEV空间变换增强特征一致性。实验采用360BEV-Matterport数据集（源自Matterport3D，512×1024全景图，20个导航相关类别，合成深度保证几何一致），在MiT-B0、MiT-B2和MSCA-B骨干上以500×500 BEV、10m×10m范围训练，使用Acc、mRecall、mPrecision和mIoU评估，取得SOTA结果，例如MiT-B2+增强在val集mIoU达44.54%，test集达38.49%，并在俯仰/滚转扰动下比360BEV更鲁棒。真实部署在Tiangong人形机器人（1.65m、23 DoF）上，集成Insta360 X4全景相机与Livox Mid-360 LiDAR，以10Hz在Nvidia Jetson Orin AGX上处理，在杂乱室内环境中验证了BEV语义建图与下游导航能力；同时方法需应对LiDAR深度稀疏、动态关节运动引起的外参漂移等约束。未来工作将进一步优化多视角与全景相机感知算法，并采集和标注更多人形机器人相关场景数据。\n\n### 关键要点\n\n- HumanoidPano 是针对人形机器人设计的通用跨模态感知框架，通过融合全景视觉与LiDAR来缓解自遮挡和视场受限带来的感知退化。\n- 框架包含全景图像编码、全景畸变感知注意力（PD Attention）和BEV解码器，其中PD Attention由球面几何约束（SGC）与空间可变形注意力（SDA）组成。\n- SGC 将3D参考点通过球面坐标变换（Φ=tan⁻¹(y/x)，Θ=tan⁻¹(x/(z·cosΦ))）映射到全景图，并预测球面角偏移以生成畸变感知的采样位置。\n- SDA 以BEV查询对应的2D索引为参考点，从全景特征图中采样并加权求和，实现360°到BEV的几何完整对象表示融合。\n- 全景增强（AUG）在训练时对全景图像和BEV语义图联合执行随机翻转、混合和旋转，并要求BEV空间变换与图像空间变换保持几何对齐。\n- 实验使用360BEV-Matterport数据集，该数据集提供512×1024全景RGB图、20类BEV语义标注和射线投射生成的合成深度，适合验证跨模态融合。\n- 在360BEV-Matterport val集上，HumanoidPano（MiT-B2+增强）达到Acc 78.48%、mRecall 58.32%、mPrecision 61.52%、mIoU 44.54%，超过360BEV对应结果。\n- 在test集上，HumanoidPano（MSCA-B+增强）的mIoU为40.41%，且不使用增强时MSCA-B骨干mIoU为40.27%，均优于360BEV的39.10%。\n- 在俯仰和滚转扰动下，HumanoidPano+增强的mIoU下降22.40%，小于360BEV的24.78%，表明其对相机姿态扰动更鲁棒。\n- 消融实验显示，SDA、SGC和AUG逐步叠加可将test集mIoU从31.81%提升至35.37%，验证了三个组件的贡献。\n- 真实部署采用Insta360 X4全景相机和Livox Mid-360 LiDAR，在Tiangong人形机器人上以10Hz运行，验证了穿门和避障等复杂室内导航场景中的BEV语义建图能力。\n- 方法采用LiDAR而非结构光深度相机，因为后者难以提供大范围探测；LiDAR深度较稀疏，但动态关节运动引起的外参漂移需通过在线标定协议补偿。"
}