{
  "paper_id": "eb85173c6b0c70529afe29eb9a1fddc0",
  "paper_title": "Multispectral Pedestrian Detection using Deep Fusion Convolutional Neural Networks",
  "paper_authors": [
    "Jörg Wagner",
    "Volker Fischer",
    "Michael Herman",
    "Sven Behnke"
  ],
  "publication_date": "2016-04-27",
  "publication_venue": "ESANN 2016 proceedings, European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning, Bruges (Belgium)",
  "abstract": "Robust vision-based pedestrian detection is a crucial feature of future autonomous systems. Thermal cameras provide an additional input channel that helps solving this task and deep convolutional networks are the currently leading approach for many pattern recognition problems, including object detection. In this paper, we explore the potential of deep models for multispectral pedestrian detection. We investigate two deep fusion architectures and analyze their performance on multispectral data. Our results show that a pre-trained late-fusion architecture significantly outperforms the current state-of-the-art ACF+T+THOG solution.",
  "keywords": [
    "multispectral pedestrian detection",
    "deep fusion",
    "convolutional neural networks",
    "thermal imaging",
    "late fusion",
    "early fusion",
    "KAIST benchmark"
  ],
  "paper_institutions": [
    "Robert Bosch GmbH",
    "University Bonn - Computer Science VI, Autonomous Intelligent Systems"
  ],
  "output_language": "zh",
  "provider": "doubao",
  "model": "ark-code-latest",
  "prompt_template": "simple",
  "extracted_at": "2026-08-16T19:47:27.929845Z",
  "template_tag": "simple",
  "summary": "### 摘要\n本论文针对自动驾驶系统中的多光谱行人检测问题，提出利用深度卷积神经网络融合可见光与热红外图像。文中介绍了两种融合架构：早期融合（EarlyFusion）在像素级拼接输入，晚期融合（LateFusion）则通过独立子网络提取特征后在全连接层融合。基于KAIST多光谱基准数据集（包含95.3k对图像，其中训练集50.2k对，测试集45.1k对），采用R-CNN框架，并以ACF+T+THOG作为基线。实验结果表明，预训练的晚期融合架构在合理子集上显著优于基线，全天的log平均漏检率为43.80%，白天为46.15%，夜晚为37.00%，相比基线（50.48%/51.27%/47.40%）均有明显提升。早期融合架构则大多未能超越基线，归因于数据量不足、预处理近似热通道不准确及对齐误差等。该研究表明深度模型在多光谱行人检测中的潜力，未来可探索更精确的跨模态预训练策略和应对对齐误差的鲁棒方法。\n\n### 关键要点\n- 本文首次将深度卷积神经网络应用于可见光与热红外图像的多光谱行人检测，提出早期融合和晚期融合两种架构。\n- 早期融合架构基于CaffeNet，将四通道输入（RGB+热）在像素级拼接，并将卷积层滤波器数量乘以4/3补偿通道数，全连接层神经元设为2048，首层卷积步长设为2。\n- 晚期融合架构使用两个独立的CaffeNet子网络分别处理可见光和热图像，热子网络卷积滤波器减半，两子网络的全连接层输出（各2048维）拼接后经4096维融合层、ReLU、dropout和二元分类层输出。\n- 由于缺乏大规模多光谱数据集，预训练采用可见光数据：用红色通道近似热通道，在ImageNet上预训练卷积层，并用CALTECH数据集进行微调。\n- 在KAIST训练数据上，采用每第二帧采样，并划分92%训练、8%验证；对晚期融合采用先分别优化子网络再联合微调的两阶段训练策略。\n- 实验在KAIST合理子集（all/day/night）上评估，采用log-average miss rate指标，并遵循[2]的标准化协议（固定高宽比0.41）。\n- 预训练的晚期融合架构（LateFusion+PreTraining）在所有测试集上均优于基线，全/白天/黑夜的漏检率分别为43.80%/46.15%/37.00%，相比ACF+T+THOG（50.48%/51.27%/47.40%）提升显著。\n- 未预训练的晚期融合（LateFusion）在夜晚（41.58%）优于基线，但整体（51.30%）略逊于基线；预训练对两种架构均有显著帮助。\n- 早期融合架构（无论是否预训练）大多不敌基线，作者认为原因是数据量有限、预训练时热通道近似不当导致无法学习互补特征，以及数据中存在非系统性对齐误差。\n- 晚期融合对对齐误差更鲁棒，因为它在空间信息重要性降低的高级特征层融合，而早期融合在像素级拼接更容易受对齐误差影响。\n- 热模态在夜间更有优势，可见光模态在白天更有优势，融合两者可提升全天候性能。\n- 该工作的未来方向包括改进跨模态预训练策略（如可见光到热的迁移学习）、处理对齐误差的鲁棒融合方法，以及探索更复杂的融合层级。"
}