{
  "paper_id": "0dc5f6edee50a49b6f7a389101235ad7",
  "paper_title": "LiDAR-NeRF: Novel LiDAR View Synthesis via Neural Radiance Fields",
  "paper_authors": [
    "Tang Tao",
    "Yixing Lao",
    "Longfei Gao",
    "Guangrun Wang",
    "Dayang Hao",
    "Peng Chen",
    "Hengshuang Zhao",
    "Xiaodan Liang",
    "Mathieu Salzmann",
    "Kaicheng Yu"
  ],
  "publication_date": "2024",
  "publication_venue": "Proceedings of the 32nd ACM International Conference on Multimedia (MM '24)",
  "abstract": "We introduce a new task, novel view synthesis for LiDAR sensors. While traditional model-based LiDAR simulators with style-transfer neural networks can be applied to render novel views, they fall short of producing accurate and realistic LiDAR patterns because the renderers rely on explicit 3D reconstruction and exploit game engines, that ignore important attributes of LiDAR points. We address this challenge by formulating, to the best of our knowledge, the first differentiable end-to-end LiDAR rendering framework, LiDAR-NeRF, leveraging a neural radiance field (NeRF) to facilitate the joint learning of geometry and the attributes of 3D points. However, simply employing NeRF cannot achieve satisfactory results, as it only focuses on learning individual pixels while ignoring local information, especially at low texture areas, resulting in poor geometry. To this end, we have taken steps to address this issue by introducing a structural regularization method to preserve local structural details. To evaluate the effectiveness of our approach, we establish an object-centric multi-view LiDAR dataset, dubbed NeRF-MVL. It contains observations of objects from 9 categories seen from 360-degree viewpoints captured with multiple LiDAR sensors. Our extensive experiments on the scene-level KITTI-360 dataset, and on our object-level NeRF-MVL show that our LiDAR-NeRF surpasses the model-based algorithms significantly.",
  "keywords": [
    "LiDAR-NeRF",
    "NeRF-MVL",
    "LiDAR View Synthesis"
  ],
  "paper_institutions": [
    "Shenzhen Campus of Sun Yat-sen University",
    "The University of Hong Kong",
    "Cainiao Group",
    "University of Oxford",
    "Research Institute of Multiple Agents and Embodied Intelligence, Peng Cheng Laboratory",
    "CVLab, EPFL",
    "Westlake University"
  ],
  "output_language": "zh",
  "provider": "doubao",
  "model": "glm-5.3",
  "prompt_template": "simple",
  "extracted_at": "2026-09-17T09:36:43.251027Z",
  "template_tag": "simple",
  "summary": "### 摘要\n\n本文提出了一个新任务——LiDAR传感器的新视角合成，旨在给定多视角LiDAR观测的情况下，从任意新视角渲染物体或场景的点云。传统基于模型的LiDAR模拟器（如LiDARsim和PCGen）依赖显式3D网格重建与游戏引擎光线投射，多步骤流程限制了可扩展性，且因忽略LiDAR点的重要属性（如光线丢失、穿透玻璃等）而产生不真实的LiDAR图案；而现有NeRF方法仅将LiDAR用于辅助训练，仍以图像为输出，生成新LiDAR视角的任务此前未被探索。受NeRF在RGB新视角合成中成功的启发，作者提出首个可微分端到端LiDAR渲染框架LiDAR-NeRF：将点云转换为360度距离伪图像，每个伪像素对应一条激光束，用神经网络编码3D信息并预测距离、强度和ray-drop（光线丢弃）概率三种属性，核心机制是沿光线进行体渲染积分，如期望深度 $\\hat{D}(\\mathbf{r})=\\sum_{i=1}^{N}T_{i}(1-\\exp(-\\sigma_{i}\\delta_{i}))t_{i}$，其中 $T_i$ 为累积透射率，并利用多视角一致性学习准确几何。针对NeRF只关注单个像素、在低纹理区域忽略局部信息导致几何不佳的问题，进一步提出结构正则化，将梯度损失限制在低纹理区域以保留局部结构细节。为评估方法，作者建立了以物体为中心的多视角LiDAR数据集NeRF-MVL（因现有自动驾驶数据集视角多样性不足，仅适合场景级合成），包含9个常见交通类别、超过76k帧、两种采集车辆、三种LiDAR设置和两条采集路径，由真实自动驾驶车辆多LiDAR传感器采集并精细标定。实验在场景级KITTI-360数据集（4个静态郊区序列，每序列64帧取4帧评估）和物体级NeRF-MVL上进行，采用C-D（倒角距离）、F-score、RMSE、阈值精度δ1/δ2/δ3、SSIM和MAE等指标：场景级上LiDAR-NeRF（iNGP+结构正则化）取得C-D 0.081、F-score 92.49、δ1 82.18、SSIM 0.626，全面优于LiDARsim（C-D 0.951）和PCGen（C-D 0.187）；物体级上Ours-iNGP取得C-D 0.005、F-score 98.50、δ1 98.86，大幅超越基线，验证了可微分渲染与结构正则化的有效性。该方法还可用于场景编辑，将NeRF-MVL物体融合进KITTI-360场景产生真实遮挡效果，在自动驾驶仿真等实际场景中具有应用潜力，且无需显式3D重建与渲染引擎，简单有效，有望启发新LiDAR视角合成的未来研究。\n\n### 关键要点\n\n- 提出首个可微分端到端LiDAR渲染框架LiDAR-NeRF，无需显式3D重建和游戏引擎即可渲染带点强度和ray-drop概率的新颖点云，填补了新LiDAR视角合成任务的空白。\n- 将LiDAR点云转换为360度距离伪图像表示，每个伪像素对应一条真实激光束，编码距离（3D坐标）、强度（反射光量）和ray-drop概率（光线丢弃概率）三种属性。\n- 核心机制沿光线进行体渲染积分计算期望深度、强度和ray-drop概率，公式为 $\\hat{D}(\\mathbf{r})=\\sum_{i=1}^{N}T_{i}(1-\\exp(-\\sigma_{i}\\delta_{i}))t_{i}$，并利用多视角一致性帮助网络产生准确几何。\n- 提出结构正则化方法 $\\mathcal{L}_{reg}=\\|\\hat{G_M}(\\mathbf{R})-G_M(\\mathbf{R})\\|_1$，将梯度损失限制在低纹理区域（如地面），保留局部结构细节，引导NeRF产生更精确的几何估计。\n- 总损失函数包含四项：距离L1损失、强度L2损失、ray-drop L2损失和结构正则化项，通过权重系数λ平衡各目标。\n- 建立NeRF-MVL数据集：以物体为中心的多视角LiDAR数据集，包含9个常见交通类别物体、超过76k帧、两种采集车辆、三种LiDAR设置（含前部和顶部两种旋转式64束LiDAR）和两条方形采集路径，传感器位姿经过精细标定并提供lidar2world矩阵。\n- 场景级实验在KITTI-360数据集的4个静态郊区序列上进行，每序列64帧、取4个等距帧用于评估；物体级实验使用NeRF-MVL的7.3k帧精简版覆盖九个类别。\n- 在KITTI-360场景级合成中，LiDAR-NeRF（iNGP+结构正则化）取得C-D 0.081、F-score 92.49、RMSE 3.615、δ1 82.18、SSIM 0.626、MAE 0.096，在所有指标上显著优于LiDARsim（C-D 0.951）和PCGen（C-D 0.187）基线。\n- 在NeRF-MVL物体级合成中，Ours-iNGP取得C-D 0.005、F-score 98.50、RMSE 1.305、δ1 98.86、SSIM 0.879、MAE 1.057，在全部九个类别上以大幅优势超越基线渲染器。\n- 消融实验表明iNGP的混合3D网格结构与多分辨率哈希编码比原始NeRF更具表达力、性能更优，故被选为基础架构；结构正则化在定量和定性上均显著改善几何估计与LiDAR图案真实性。\n- LiDAR-NeRF能准确模拟激光束穿透车窗玻璃到达座椅等物理行为，而基线方法因显式重建和光线投射忽略表面材质与法线方向的影响，导致物体表面高估和光线丢失，产生不真实图案。\n- 该方法可扩展至场景编辑应用，将NeRF-MVL中的新物体重渲染并融合到KITTI-360场景中，产生真实的遮挡效果和一致的LiDAR图案；在纹理丰富的物体级数据上无需结构正则化，该正则化主要针对大规模低纹理场景设计。"
}