{
  "paper_id": "9203d11d1418f111f5fa784c24c66a46",
  "paper_title": "TransFusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformers",
  "paper_authors": [
    "Xuyang Bai",
    "Zeyu Hu",
    "Xinge Zhu",
    "Qingqiu Huang",
    "Yilun Chen",
    "Hongbo Fu",
    "Chiew-Lan Tai"
  ],
  "publication_date": "",
  "publication_venue": "2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)",
  "abstract": "LiDAR and camera are two important sensors for 3D object detection in autonomous driving. Despite the increasing popularity of sensor fusion in this field, the robustness against inferior image conditions, e.g., bad illumination and sensor misalignment, is under-explored. Existing fusion methods are easily affected by such conditions, mainly due to a hard association of LiDAR points and image pixels, established by calibration matrices.\n\nWe propose TransFusion, a robust solution to LiDAR-camera fusion with a soft-association mechanism to handle inferior image conditions. Specifically, our TransFusion consists of convolutional backbones and a detection head based on a transformer decoder. The first layer of the decoder predicts initial bounding boxes from a LiDAR point cloud using a sparse set of object queries, and its second decoder layer adaptively fuses the object queries with useful image features, leveraging both spatial and contextual relationships. The attention mechanism of the transformer enables our model to adaptively determine where and what information should be taken from the image, leading to a robust and effective fusion strategy. We additionally design an image-guided query initialization strategy to deal with objects that are difficult to detect in point clouds. TransFusion achieves state-of-the-art performance on large-scale datasets. We provide extensive experiments to demonstrate its robustness against degenerated image quality and calibration errors. We also extend the proposed method to the 3D tracking task and achieve the $1^{st}$ place in the leaderboard of nuScenes tracking, showing its effectiveness and generalization capability. [code release]",
  "keywords": [
    "3D object detection",
    "LiDAR-camera fusion",
    "Transformer",
    "soft association",
    "autonomous driving"
  ],
  "paper_institutions": [
    "Hong Kong University of Science and Technology",
    "ADS, IAS BU, Huawei",
    "City University of Hong Kong"
  ],
  "output_language": "zh",
  "provider": "doubao",
  "model": "ark-code-latest",
  "prompt_template": "simple",
  "extracted_at": "2026-08-16T19:46:20.020062Z",
  "template_tag": "simple",
  "summary": "### 摘要\n本文针对自动驾驶中LiDAR与相机融合进行3D目标检测时，面对图像质量退化（如光照不足）和传感器标定误差时的鲁棒性问题，提出了一种基于Transformer的软关联融合框架TransFusion。现有融合方法通常通过标定矩阵建立LiDAR点与图像像素的硬关联，导致对图像缺陷和标定误差敏感。TransFusion采用卷积骨干提取LiDAR BEV特征与图像特征，并以Transformer解码器作为检测头：第一层解码器利用稀疏的输入依赖且类别感知的对象查询从点云生成初始3D边界框，第二层解码器通过空间调制交叉注意力（SMCA）将对象查询与图像特征自适应融合，SMCA使用2D高斯掩码约束注意力区域，公式为$M_{ij} = \\exp(-\\frac{(i - c_x)^2 + (j - c_y)^2}{\\sigma r^2})$，从而软性地确定从图像中获取信息的位置和内容。此外，还设计了图像引导的查询初始化模块，将图像特征投影到BEV平面以提升对点云中难检测物体的召回。在nuScenes和Waymo数据集上，TransFusion取得了当时的最优性能，nuScenes检测mAP达到68.9%、NDS 71.7%，并扩展到3D跟踪任务获得nuScenes跟踪排行榜第一名。实验验证了其在夜间、图像缺失和传感器错位等退化条件下的鲁棒性。未来可探索将软关联融合策略应用于3D分割等其他任务，并考虑更强大的点云密度的适应。\n### 关键要点\n- TransFusion提出用软关联代替硬关联进行LiDAR-相机融合，增强对图像质量和标定误差的鲁棒性。\n- 检测头采用两层Transformer解码器：第一层从LiDAR特征生成初始框，第二层通过交叉注意力融合图像特征。\n- 对象查询采用输入依赖和类别感知的初始化，基于中心热图选取top-N候选，并加入类别嵌入。\n- 空间调制交叉注意力（SMCA）利用2D高斯掩码限制注意力区域，公式为$M_{ij} = \\exp(-\\frac{(i - c_x)^2 + (j - c_y)^2}{\\sigma r^2})$。\n- 图像引导的查询初始化将图像特征沿高度压缩后投影到BEV，通过交叉注意力与LiDAR BEV特征融合，提升对稀疏点云中小物体的检测。\n- 训练分两阶段：先用LiDAR训练第一解码器20轮，再联合训练融合模块和图像引导模块6轮。\n- 在nuScenes测试集上，TransFusion的mAP为68.9%，NDS为71.7%，优于所有对比方法；TransFusion-L（仅LiDAR）mAP为65.5%。\n- 在Waymo验证集上，融合策略主要提升行人和骑行者的mAPH，整体mAPH为65.5%，较LiDAR-only提升有限，因Waymo点云更密。\n- 在3D跟踪任务上，TransFusion获得nuScenes跟踪排行榜第一，AMOTA为71.8%，远超CenterPoint（63.8%）。\n- 鲁棒性实验显示，在夜间场景mAP提升6.0%，远高于硬关联基线（CC和PA分别+0.2和+1.8）。\n- 当所有6张图像被丢弃时，TransFusion的mAP仅下降3.9%至61.7%，而CC下降23.8%至39.5%，PA下降17.2%至47.0%。\n- 在传感器错位1m时，TransFusion的mAP仅下降0.49%，而PA和CC分别下降2.33%和2.85%。\n- 消融实验表明，输入依赖初始化比学习式查询位置极大提升性能（NDS从33.8%升至66.8%），类别感知嵌入带来约3%的NDS提升。"
}