{
  "paper_id": "2844f7cbdfbdc12dd2a2721459f9e55d",
  "paper_title": "Align and Blend: A Unified Multi-Modal LiDAR Segmentation Network",
  "paper_authors": [
    "Chuanxu Wang",
    "Jiajiong Li",
    "Xin Chen",
    "Da Song",
    "Binghui Wang"
  ],
  "publication_date": "",
  "publication_venue": "IEEE Transactions on Intelligent Transportation Systems",
  "abstract": "The multimodal fusion of point cloud data is a challenging task in 3D computer vision, particularly in fine object segmentation. In this study, we propose an innovative unified multimodal LiDAR segmentation network, named Align and Blend (A2Blend for short), which skillfully integrates three representative forms of point clouds: point view, voxel view, and range view. The core innovation of A2Blend lies in addressing two primary tasks: Align and Blend. For the \"Align\" task, we have carefully designed a learnable cross-modal association module, whose core is a unique \"Cross-Modal Triplet Alignment Loss\" mechanism. To the best of our knowledge, this is the first application of such a mechanism in the field of LiDAR segmentation. This mechanism applies principles from contrastive learning, promoting the tight clustering of similar semantic groups both within and across modalities, while increasing the separation between dissimilar groups by expanding the distance between similar and dissimilar sample clusters in the feature space. This significantly enhances the discriminative power and representational capacity of the feature embeddings. For the \"Blend\" task, we propose a fusion strategy that integrates intragroup self-attention within modalities and intergroup self-attention across modalities. This approach combines key concepts from standard self-attention and cross-self-attention mechanisms to achieve more comprehensive multimodal fusion. The segmentation performance on two large-scale outdoor datasets and one indoor dataset surpasses that of most state-of-the-art algorithms.",
  "keywords": [
    "Multi-modal LiDAR segmentation",
    "align and blend",
    "cross-modal triplet alignment loss",
    "self attention"
  ],
  "paper_institutions": [
    "Qingdao University of Science and Technology",
    "Ocean University of China"
  ],
  "output_language": "zh",
  "provider": "doubao",
  "model": "glm-5.3",
  "prompt_template": "simple",
  "extracted_at": "2026-09-17T09:39:10.728612Z",
  "template_tag": "simple",
  "summary": "### 摘要\n\nLiDAR语义分割旨在为每个输入点分配语义标签，是自动驾驶、数字城市和服务机器人等领域的关键技术，而多模态点云数据的融合（尤其是精细目标分割）仍是3D计算机视觉中的挑战性难题。现有方法多聚焦RGB与距离图像的双视图融合或简单多模态拼接，往往忽视各模态间的固有差异，且RGB数据在许多数据集中缺失、依赖标定矩阵的硬关联易受标定误差影响；受CLIP、MaskCLIP、PointCLIP等对比学习工作的启发，本文提出统一多模态LiDAR分割网络A2Blend，首次将对比学习应用于3D分割的多模态融合任务，巧妙整合点视图、体素视图与距离视图三种点云表示，并围绕两大核心任务展开：“Align”任务设计可学习跨模态关联模块（LCA）与语义引导多模态特征解耦模块，首次提出跨模态三元组对齐损失 L_cmtal = L_Dclpr + L_Dclpv，其基于标准对比损失 L(Z,Z⁺,Z⁻) = −log(exp(Zᵀ·Z⁺/τ)/(exp(Zᵀ·Z⁺/τ)+Σₖ exp(Zᵀ·Z⁻/τ)))（τ为可学习温度因子），在特征空间中拉近相似语义簇、推远不相似簇，显著增强特征嵌入的判别力与表示能力；“Blend”任务结合模态内组内自注意力与跨模态组间交叉自注意力实现更全面的多模态融合。实验选取两个大规模室外数据集SemanticKITTI（19类）与nuScenes（16类）以及室内数据集S3DIS，以同时验证室外驾驶场景与室内密集遮挡场景下的性能与泛化能力，采用类间IoU与mIoU作为评价指标；实现上LCA采用半UNet结构（距离与体素分支各四次下采样，特征维度32/64/128/256），体素分辨率恒定0.05m，点分支采用PointNet式MLP，PV结构采用完整SPVNAS架构，SGD端到端训练（SemanticKITTI/nuScenes分别50/70个epoch，RTX 3090训练约一周）。结果显示A2Blend在SemanticKITTI测试集达71.9 mIoU（超RPVNet 1.6、PVKD 0.7），nuScenes测试集达80.8 mIoU（超RPVNet 3.2、AF2S3 2.5），S3DIS Area 5上结合Swin3D达74.1 mIoU（超Swin3D 1.6、仅落后PTv3 0.6），消融实验证明Align模块提升2.2/2.5 mIoU、Blend模块提升1.3/1.4 mIoU，有力支持了对比学习对齐与注意力融合两项创新的有效性。未来工作将优化算法1（小批量特征聚类与填充）以进一步提升模型性能；实际部署中模型以24.3M参数、132.2ms延迟在精度与效率间取得竞争性平衡，但室内密集遮挡场景需将调整后特征送入Swin3D等下游网络，且与使用额外RGB模态的Uniseg相比在自行车、建筑等类别仍有0.2–0.3 mIoU差距。\n\n### 关键要点\n\n- 提出A2Blend统一多模态LiDAR分割网络，首次将对比学习应用于3D分割的多模态融合任务，整合点视图、体素视图与距离视图三种点云表示，并摒弃许多数据集中缺失、采集成本高的RGB模态。\n- 核心创新分为“Align”与“Blend”两大任务：Align通过跨模态三元组对齐损失在特征空间实现跨模态对齐，Blend通过组内自注意力与组间交叉自注意力实现多模态融合。\n- 设计语义引导多模态特征解耦模块，基于真实标签对三模态特征进行软分组聚类与一致填充，解决了百万级无序点上逐点划分锚点、正负样本不可行的难题。\n- 跨模态三元组对齐损失由点-距离（L_Dclpr）与点-体素（L_Dclpv）两部分对偶对比损失构成，温度因子τ可学习；总损失还结合加权交叉熵损失与Lovázs-Softmax损失，超参数β_wce、β_ls、β_cmtal在SemanticKITTI与nuScenes上分别设为1/1/2.5与1/1/7.5。\n- Blend融合策略在模态内使用标准自注意力捕获同语义簇内特征关联，跨模态采用多头交叉注意力（MCSA）在特征簇级别建模语义关系，最终经Merg与Cat操作输出融合特征并送入后续点-体素分割网络。\n- 实验基于SemanticKITTI（22条序列、19类）、nuScenes（1000个驾驶场景、16类）与S3DIS（272个房间、6大区域）三个公开大规模数据集，采用类间IoU与mIoU作为评价指标。\n- 部署细节：LCA采用半UNet结构且无预训练提取器，距离分支输入尺寸为64×2048（SemanticKITTI）与32×2048（nuScenes），体素分辨率恒定0.05m，点分支使用维度32与256的两个MLP，PV结构采用完整SPVNAS架构。\n- 训练设置：SGD优化器端到端训练，SemanticKITTI与nuScenes分别训练50与70个epoch，初始学习率1.0、一个warm-up epoch后余弦退火，动量0.9、权重衰减0.0001，RTX 3090训练约一周，点分支采用随机翻转、缩放及LaserMix、PolarMix数据增强。\n- 结果：SemanticKITTI测试集71.9 mIoU，超RPVNet 1.6、SDSeg3D 1.5、GASN 1.2、PVKD 0.7，在汽车、摩托车手、道路等9个类别领先；nuScenes测试集80.8 mIoU，超RPVNet 3.2、AF2S3 2.5、GASN 0.4，16类中7类分割更优。\n- 消融实验：体素编码器贡献最大（较距离编码器在两数据集上分别提升13.3与12.7 mIoU），Align模块提升2.2/2.5 mIoU，Blend模块提升1.3/1.4 mIoU，Blend融合较Add基线提升1.4 mIoU而Concat仅提升0.2 mIoU。\n- 效率与约束：A2Blend参数量24.3M、延迟132.2ms，优于Cylinder3D、SPVCNN与RPVNet并在精度与延迟间取得竞争性平衡；室内S3DIS因密集遮挡不直接使用完整框架，而是将调整后特征送入Swin3D下游网络（仅用单次对偶对比损失）验证泛化性，取得74.1 mIoU。\n- 局限与未来方向：与使用额外RGB模态的Uniseg相比在自行车、建筑等类别存在0.2–0.3 mIoU差距；未来将优化算法1（小批量特征聚类与填充）以进一步提升模型性能。",
  "available_templates": [
    "deep_read",
    "simple"
  ]
}