{
  "paper_id": "e2930c53d5c000987759a2fff7bec8ee",
  "paper_title": "Real-Time LiDAR 3D Semantic Segmentation via Multi-View and Cross-Modal Compact Featuring Two-Branch Knowledge Distillation",
  "paper_authors": [
    "Yun Zhang",
    "Kun Qian",
    "Zihan Zhang",
    "Min'ao Zhang",
    "Hai Yu"
  ],
  "publication_date": "2026",
  "publication_venue": "MDPI",
  "abstract": "Simultaneous online mapping and semantic segmentation using handheld scanners supports various environmental inspection and measurement tasks. For such scanners, combing visual and LiDAR data is beneficial for improving the segmentation performance. But the direct fusion of multi-modal and multi-view features faces challenges in terms of both real-time performance and robustness. To address these challenges, this paper proposes a multi-view and cross-modal knowledge distillation method for supporting runtime LiDAR-only semantic segmentation. The proposed method hierarchically compacts multi-view and cross-model priors and distills them into two branches to improve segmentation accuracy. In addition, we design an improved data augmentation technique based on PolarMix for rendering more realistic point cloud scenes. The experimental results on the SemanticKITTI and nuScenes datasets demonstrate that the mIoU of our approach outperforms the state-of-the-art knowledge-distillation-based methods. In addition, mapping experiments using a handheld scanner demonstrate the proposed method's superior real-time performance and accuracy.",
  "keywords": [
    "LiDAR",
    "semantic segmentation",
    "multi-view and cross-model fusion",
    "knowledge distillation",
    "handheld 3D scanner"
  ],
  "paper_institutions": [
    "School of Automation, Southeast University, Nanjing, China",
    "Southeast University Shenzhen Research Institute, Shenzhen, China",
    "Information and Communication Research Institute, China Electric Power Research Institute Co., Ltd., Nanjing, China"
  ],
  "output_language": "zh",
  "provider": "doubao",
  "model": "glm-5.3",
  "prompt_template": "simple",
  "extracted_at": "2026-09-17T09:41:10.498169Z",
  "template_tag": "simple",
  "summary": "### 摘要\n本文针对手持LiDAR扫描仪在线建图与语义分割中多模态、多视角特征直接融合所导致的实时性与鲁棒性难题展开研究：相机易受光照变化（如过曝）影响且难以获取远距离精确深度，LiDAR虽提供大范围精确空间数据但缺乏颜色纹理且点云稀疏无序，而现有点、体素、投影三类视角方法分别受无序性、量化误差与投影误差制约，多视角/多模态融合方法又需为每种输入配置专用编码器，计算与内存开销大。为此，论文提出一种支持运行时仅LiDAR推理的多视角与跨模态知识蒸馏方法：主分支为基于体素的轻量级分割网络（编码器基于SPVCNN、稀疏卷积基于TorchSparse++，层维度16/32/64/128）负责实时推理，辅助分支仅在训练时通过基于互注意力的三步门控机制（GM）分层压缩距离视图与RGB图像等多视角跨模态先验，并将融合特征双向蒸馏至两分支，其核心门控融合公式为 F̄ = Σᵢ split[softmax(Σᵢ σ(MLP(Fᵢ)))]ᵢ·Fᵢ（σ为sigmoid函数）；同时提出改进的APolarMix数据增强算法，通过地面点选择、按权重与距离插入实例、高度/方位角校正及点云墙遮挡消除生成更真实的点云场景。实验选用SemanticKITTI（43,551帧、每帧约13万点）与nuScenes（14亿标注点）两大权威基准验证泛化性，并以自建手持/背包扫描平台（Livox Avia LiDAR与相机10Hz采集、IMU 200Hz）验证真实场景实时性；采用mIoU指标，方法在SemanticKITTI上取得73.5% mIoU与19.8 FPS、在nuScenes上取得81.7% mIoU，均超越2DPASS、PVKD等最先进知识蒸馏方法，消融实验进一步证明单编码器门控融合较专用多编码器减少52.7%参数、APolarMix将mIoU从69.36%提升至69.85%，支撑了紧凑特征融合与真实增强两项创新的有效性。未来工作计划探索利用时空信息增加辅助分支公共点数量，并指出高效点云压缩与高速语义分割的协同是关键方向；实际部署需应对传感器标定误差、极端光照、设备位姿不稳定导致的建图畸变等约束，端到端延迟测试表明建图占主要延迟（71.57ms），整体最低帧率13.97Hz仍满足实时流水线需求。\n\n### 关键要点\n- 方法核心：提出双分支特征增强框架，主分支为基于体素的轻量级LiDAR-only分割网络（SPVCNN架构）负责训练与推理，辅助分支仅在训练时融合多视角（距离视图）与跨模态（RGB图像）特征并蒸馏至两分支，实现运行时仅LiDAR的高实时推理。\n- 核心机制：采用基于互注意力的三步门控融合（LWFAF特征增强融合、辅助分支融合、分支间融合），门控融合公式为 F̄ = Σᵢ split[softmax(Σᵢ σ(MLP(Fᵢ)))]ᵢ·Fᵢ，其中σ为sigmoid函数。\n- 可扩展性设计：融合更多模态或视角时仅需增加对应LWFAF模块，参数量、GPU内存与训练时间基本不变（V+I为24.9M/13.9GB，V+IR为24.8M/13.8GB），保证网络可低成本扩展。\n- 数据增强：APolarMix在PolarMix基础上引入地面点选择、按类别权重与距离约束插入实例、高度/方位角校正及点云墙遮挡消除，将mIoU从69.36%逐步提升至69.85%，且仅在训练阶段使用不影响推理时间。\n- 数据集设置：SemanticKITTI含43,551帧LiDAR数据（每帧约13万点），nuScenes含14亿标注点（850训练验证场景+150测试场景），自建数据集由手持/背包扫描平台采集（Livox Avia LiDAR与相机10Hz、IMU 200Hz），分别用于基准对比与真实实时性验证。\n- 实现细节：主分支编码器层维度为16/32/64/128，辅助分支采用ResNet34编码器，体素大小在SemanticKITTI为0.05、nuScenes为0.1，RGB与距离图像分别缩放至640×320和64×2048，batch size为2、学习率0.24、训练80轮，采用SGD优化器与交叉熵+Lovázs损失。\n- 主要结果：在SemanticKITTI上取得73.5% mIoU与19.8 FPS，超过2DPASS（72.9%、14.33 FPS）与PVKD（71.2%）；在nuScenes上取得81.7% mIoU与91.4% FW IoU，超过2DPASS的80.8%。\n- 消融证据：单编码器门控融合（SE GATED）相比多编码器（ME GATED）减少52.7%参数、2.5GB GPU内存与21.31小时训练时间，mIoU仅下降0.08，验证了紧凑融合设计的有效性。\n- 超参权衡：体素大小从0.025增至0.1时mIoU仅从69.91%降至68.92%（98.4%），推理时间从310s降至132s（42.6%），最终选择0.05以平衡精度与实时性；参数量翻倍仅提升mIoU 0.13%但推理时间增加56.59%。\n- 蒸馏路径：将第三级门控融合特征蒸馏至两分支并辅以辅助分支到主分支的蒸馏，相比仅辅助到主分支蒸馏将mIoU从68.76%提升至69.85%，证明双向蒸馏可间接增强主分支性能。\n- 实时性验证：端到端延迟测试（20,000帧）显示标定25.05ms、体素化16.3ms、推理41.3ms、建图71.57ms，建图占主要延迟，最低帧率13.97Hz，流水线并行下每帧功耗13.39W。\n- 应用约束：相机过曝、多传感器标定误差、设备位姿不稳定引起的建图畸变是实际约束；训练数据采集需在光照稳定的白天进行并启用自动增益与自动曝光，运行时仅依赖LiDAR避免多传感器在线融合的实时性瓶颈。"
}