# PRNet: Point-Range Fusion Network for Real-Time LiDAR Semantic Segmentation Xiaoyan Li $^{1,3*}$, Gang Zhang $^{2*\dagger}$, Tao Jiang $^{3}$, Xufen Cai $^{4}$ and Zhenhua Wang $^{2}$ $^{2}$Damo Academy, Alibaba Group $^{1}$University of Chinese Academy of Sciences $^{3}$Institute of Computing Technology, Chinese Academy of Sciences $^{4}$Beijing School , , , ## Abstract Accurate and real-time LiDAR semantic segmentation is necessary for advanced autonomous driving systems. To guarantee a fast inference speed, previous methods utilize the highly optimized 2D convolutions to extract features on the range view (RV), which is the most compact representation of the LiDAR point clouds. However, these methods often suffer from lower accuracy for two reasons: 1) the information loss during the projection from 3D points to the RV, 2) the semantic ambiguity when 3D points labels are assigned according to the RV predictions. In this work, we introduce an end-to-end point-range fusion network (PRNet) that extracts semantic features mainly on the RV and iteratively fuses the RV features back to the 3D points for the final prediction. Besides, a novel range view projection (RVP) operation is designed to alleviate the information loss during the projection to the RV, and a point-range convolution (PRConv) is proposed to automatically mitigate the semantic ambiguity during transmitting features from the RV back to 3D points. Experiments on the SemanticKITTI and nuScenes benchmarks demonstrate that the PRNet pushes the range-based methods to a new state-of-the-art, and achieves a better speed-accuracy trade-off. ## 1 Introduction LiDAR semantic segmentation provides a crucial point-level perception of the surrounding environments for applications, e.g. autonomous driving and moving robots. Such applications require the LiDAR semantic segmentation model to run in real-time to support timely downstream object detection and planning. However, the LiDAR point clouds come in an unstructured and sparse format, which brings challenges for efficient and effective processing. Different data representations (the 3D point, voxel, and range view) have been used to ![Image](images/8089a1b3e7ac7c1422a07a747147590c4a237a047789b5967aeb9c6f08ef6237.jpg)
Figure 1: mIoU vs. runtime on the validation set of both SemanticKITTI and nuScenes. All methods are trained and evaluated based on the official code. The experiments are conducted with PyTorch FP32 on NVIDIA RTX 2080Ti GPU. Note that PRNet-2C has twice as many RV feature channels as PRNet-1C. The definition of mean Intersection over Union (mIoU) is given in the Section 4.1.
accomplish this task, which further results in differences in efficiency and accuracy as shown in Figure 1. The point-based methods [Qi et al., 2017a; Qi et al., 2017b; Thomas et al., 2019; Hu et al., 2020] directly consume the unstructured 3D point clouds, which are the most complete sources of 3D information. However, as shown in Figure 1, point-based methods run slowly, since it has to adopt inefficient components for searching neighbors and aggregating features within unstructured points. The latter voxel-based methods [Choy et al., 2019; Tang et al., 2020; Cheng et al., 2021; Zhang et al., 2020; Zhu et al., 2021] discretize the point clouds into voxel cells and utilize the 2D or 3D convolution networks for feature extraction. These methods achieve relatively higher performance than point-based methods, but their efficiency (speed) still acts as a significant bottleneck for real-world applications. The range-based methods adopt the most compact representation of the LiDAR point clouds, namely the RV representation, to ensure their efficiency. The previous range-based methods [Cortinhal et al., 2020; Milioto et al., 2019; Xu et al., 2020; Razani et al., 2021] first project 3D points. --- ![Image](images/655fe00b038d3f20ff424199dc63c79b2a0c9281b39287b8517d1a756be69124.jpg)
Figure 2: The information loss during the projection from 3D points to the RV: for the points of traffic-sign and vegetation in the same RV pixel, existing range-based methods only keep the nearest traffic-sign point. The semantic ambiguity from the RV predictions back to 3D points labels: the occluded vegetation points are assigned by the label of traffic-sign. The results are acquired by the official SalsaNext with kNN as post-processing.
to the RV, and then perform the 2D semantic segmentation on the RV with the well-optimized 2D convolution networks, and finally transmit the 2D RV labels back to 3D points. Despite their efficiency, their performance is much poorer than the voxel-based counterparts. We analyze two major reasons in Figure 2 and propose to solve these issues as the following. First, the information loss during the projection from 3D points to the RV is nonnegligible. As shown in Figure 2, multiple LiDAR points may be projected to the same RV pixel, but existing methods only keep the nearest point. For example, SalsaNext [Cortinhal et al., 2020] drops a significant proportion of 22% points on the SemanticKITTI dataset. To better encode the RV features, a novel range view projection (RVP) operation is proposed to aggregate features of the points in the same RV pixel through max-pooling. Moreover, the features of the RV and points are fused for the final per-point prediction to further minimize the information loss. Second, the semantic ambiguity results in the wrong predictions, when the RV labels are transmitted back to the 3D points. The RV squeezes the distance r dimension and the 3D points far away in the 3D space may correspond to adjacent or even the same pixel in the RV space, which implies that these 3D points would be probably assigned to the same label. To ameliorate this problem, existing methods adopt k-Nearest-Neighbor (kNN) as post-processing, voting for the final prediction. But sometimes the kNN also fails, as shown in Figure 2, the occluded vegetation regions are assigned by the label of traffic-sign. To address this issue fundamentally, we propose to transmit the RV features instead of labels back to 3D points and introduce a novel point-range convolution (PRConv), which transmits the RV features with dynamic per-point hyper-parameters and allows distinct 3D point features although they are corresponding to the same RV pixel. In summary, we propose an end-to-end point-range fusion network (PRNet) for real-time LiDAR semantic segmentation. Different from the previous range-based methods that predict 2D RV semantic labels, the proposed method first projects the features from 3D points to the RV for the latter feature extraction and then fuses the RV features back to 3D points for the final per-point prediction. In this framework, the RVP operation aims to alleviate the information loss during the projection from 3D points to the RV, while the PRConv is proposed to mitigate the semantic ambiguity during the feature transmission from the RV back to 3D points. Experiments on the SemanticKITTI and nuScenes benchmarks show that the proposed PRNet outperforms existing range-based methods by a large margin. Compared with the top-ranking methods, the PRNet achieves a better speed-accuracy trade-off. ## 2 Related Work Point-based Methods. The point-based methods directly consume raw point clouds without any quantization, but their performance and efficiency are relatively worse. The pioneering work, PointNet [Qi et al., 2017a], adopts the shared multilayer perception (MLP) to extract per-point features and the global max-pooling to integrate global features. To learn richer local structures, many subsequent works have been introduced. PointNet++ [Qi et al., 2017b] proposes the stacked set abstraction layers to learn hierarchical point features. KPConv [Thomas et al., 2019] proposes a novel spatial kernel-based point convolution to extract local structure. However, since the farthest point sampling (FPS) widely used in these networks is inefficient in both computation and memory cost, these methods are limited to indoor scenes and cannot be directly extended to autonomous driving with a large number of 3D points. RandLA-Net [Hu et al., 2020] replaces the FPS with random sampling to improve efficiency, but it is still far away from real-time processing. Voxel-based Methods. The voxel-based methods quantize the 3D points into structured voxels, where the mature 2D or 3D convolution networks are applied. To reduce the computation and memory cost, sparse convolution [Choy et al., 2019] is applied only on these non-empty voxels. Considering that the voxelization brings quantization errors, the following works [Tang et al., 2020; Cheng et al., 2021; Xu et al., 2021] fuse the voxel features with the fine-grained point features. SPVNAS [Tang et al., 2020] further adopts neural architecture search and achieves better results with lower computation cost. AF2S3Net [Cheng et al., 2021] design the attentive feature fusion module (AF2M) and adaptive feature selection module (AFSM) to efficiently extract local and global structures simultaneously. Recent RPVNet [Xu et al., 2021] fuses the features from the points, voxel, and RV in a single framework to alleviate quantization errors, and achieves the best results on the SemanticKITTI and nuScenes benchmarks. Besides, different 3D space partition strategies are proposed. PolarNet [Zhang et al., 2020] uses the polar grid representation and Cylinder3D [Zhu et al., 2021] follows the cylindrical partition. Although the voxel-based methods dominate the LiDAR semantic segmentation benchmarks, they cannot run in real-time on mobile platforms. Range-based Methods. The range-based methods are more attractive because they are fast and easy-deployed by utilizing highly optimized 2D convolutions. These methods first project 3D points to the RV space and then perform semantic segmentation on the RV. RangeNet++ [Milioto et al.]. --- ![Image](images/892b646fd23287c6afa5e9240cd89ee86eca5bea9b2f54305ea104305f18cdde.jpg)
Figure 3: The point-range fusion network (PRNet). It takes the 3D point features as input and extracts semantic features mainly on the RV. The output point features are acquired by fusing the RV features back to the 3D points. The PRNet can be stacked for multiple times to increase its representation capacity.
al., 2019] proposes an accelerated kNN as post-processing to deal with the semantic ambiguity from the RV predictions back to 3D points labels. SqueezeSegV3 [Xu et al., 2020] proposes the spatially adaptive convolution to apply different convolution parameters for different locations on the RV. SalsaNext [Cortinhal et al., 2020] designs a novel encoder-decoder network and adopts Lováz-Softmax loss that can directly optimize the mean Intersection over Union (mIoU) metric. More recently, Lite-HDSeg [Razani et al., 2021] designs lightweight harmonic dense convolutions for both efficiency and effectiveness. ## 3 Methodology To achieve accurate and real-time LiDAR semantic segmentation, the method needs to extract semantic features efficiently while keeping the LiDAR point clouds information as much as possible. Inspired by the recent range-based methods that extract features at a fast speed and the point-based methods that can preserve point information, we propose a novel end-to-end point-range fusion network (PRNet). The proposed method extracts semantic features mainly on the RV, and then fuses the features with 3D point features for enhancement, as shown in Figure 3. The PRNet consists of four steps: 1) the range view projection (RVP) projects the input point features to the RV feature maps; 2) a 2D FCN is applied to the 2D RV feature maps to efficiently extract semantic features; 3) the point-range convolution (PRConv) transmits features from the RV back to 3D points; 4) the point fusion (PF) module fuses the features from the RV and the 3D points to ensure complete information. Additionally, the PRNet can be stacked M times to get stronger features. Finally, the features are fed to a single fully connected layer for the final per-point semantic prediction. In the following subsections, we first illustrate the four components of the proposed PRNet in Section 3.1, 3.2, 3.3 and 3.4. Then, the loss function is shown in Section 3.5. ### 3.1 Range View Projection In the range-based methods, the 3D point features are needed to be projected to the 2D RV feature maps. Unlike the previous methods that directly ignore the occluded points, the RVP is proposed to aggregate all point features projected to the same RV pixels by max-pooling. In detail, it first transforms the $k^{th}$ 3D point from the cartesian space $\boldsymbol{p}_{k}^{3D} = (x_{k}, y_{k}, z_{k})$ to the spherical space $\boldsymbol{p}_{k}^{sph} = (r_{k}, \theta_{k}, \phi_{k})$ by applying, $$ \begin{pmatrix}r_{k}\\\theta_{k}\\\phi_{k}\end{pmatrix}=\begin{pmatrix}\sqrt{x_{k}^{2}+y_{k}^{2}+z_{k}^{2}}\\\arcsin\left(\frac{z_{k}}{\sqrt{x_{k}^{2}+y_{k}^{2}+z_{k}^{2}}}\right)\\\arctan\left(y_{k},x_{k}\right)\end{pmatrix}, $$ where $r_k$, $\theta_k$, $\phi_k$ denote the distance, zenith and azimuth angle respectively. Then, it acquires the corresponding 2D RV coordinates $\boldsymbol{p}_k^{RV} = (u_k, v_k)$ by discretizing $\theta_k$ and $\phi_k$ and ignoring $r_k$, as the following, $$ \left(\begin{matrix}{u_{k}}\\ {v_{k}}\\ \end{matrix}\right)=\left(\begin{matrix}{\frac{1}{2}[1-\phi_{k}\pi^{-1}]W}\\ {[1-(\theta_{k}+f_{u p})f^{-1}]H}\\ \end{matrix}\right), $$ where $f = f_{up} + f_{down}$ is the LiDAR vertical field-of-view. W, H are the predefined width and height of the RV feature maps. After grabbing the coordinates on the RV, the 3D features can be transposed to the corresponding 2D position. However, there may be more than one points that fall in the same RV pixel $(h, w)$, and an aggregation manner needs to be determined. The traditional range-based methods only select --- the feature from the nearest 3D point, which leads to the above-mentioned information loss. To tackle this problem, the RVP gathers the features $\mathcal{F}_{k,c}^{3D}$ of the 3D points that fall in the pixel $(h, w)$ by max-pooling to form the RV features $\mathcal{F}_{h,w,c}^{RV}$, $$ \mathcal{F}_{h,w,c}^{R V}=\operatorname*{m a x}_{\forall k\operatorname{s.t.}\lfloor u_{k}\rfloor=h,\lfloor v_{k}\rfloor=w}\mathcal{F}_{k,c}^{3D}. $$ Theoretically, the complete 3D points information can be retained within the RV features $\mathcal{F}^{RV}$ by passing information from different points through different feature channels. ### 3.2 2D Fully Convolutional Network The 2D FCN is applied to the RV features for semantic feature extraction. Its architecture follows the conventional encoder and decoder structure. The encoder network has multiple down-sampling stages and does not apply down-sampling along the height dimension. The decoder network conducts up-sampling while fusing high-level and low-level feature maps. The detailed architecture is shown in the supplementary material. ### 3.3 Point-Range Convolution To avoid semantic ambiguity, unlike the previous range-based methods that directly predict semantic labels on the RV, the proposed method transmits the RV features back to the 3D points and then performs per-point prediction. Considering that different 3D points may correspond to the same 2D pixel, the transmission module must be able to produce dynamic features according to its 3D and 2D positions to avoid ambiguity. Therefore, the PRConv is proposed to fulfill this function by first predicting a set of offsets and weights, and then doing a $3 \times 3$ convolution on the displaced positions according to the predicted offsets and weights. Intuitively, as shown in Figure 2, to aggregate features for the occluded vegetation points, the kernel position should be adjusted to the nearby non-occluded vegetation regions, and the kernel amplitude should be reduced if the corresponding kernel position falls on the traffic-sign. Formally, for the $k^{th}$ 3D point, its features $\mathcal{F}_k^{3D}$ are acquired by aggregating the RV features $\mathcal{F}^{RV}$ within a $3 \times 3$ window as the following, $$ \mathcal{F}_{k}^{{3D}}=\sum_{n=1}^{9}\boldsymbol{w}_{n}\cdot\mathcal{F}^{{R V}}(\boldsymbol{p}_{k}^{{R V}}+\boldsymbol{p}_{n}+\Delta\boldsymbol{p}_{k,n})\cdot\Delta m_{k,n}, $$ where $p_n \in \{(-1, -1), (-1, 0), \ldots, (1, 1)\}$ defines a $3 \times 3$ window, and $w_n$ denotes the convolution parameters applied on the $n^{th}$ location. To enable the PRConv to alleviate the semantic ambiguity, the kernel offset $\Delta p_{k,n}$ is used to adjust kernel position, and the kernel weight $\Delta m_{k,n}$ is introduced for modulating the kernel amplitude. As illustrated in Figure 3, both $\Delta p_{k,n}$ and $\Delta m_{k,n}$ are predicted dynamically according to features from both the RV and the 3D points. Specifically, they are estimated following a three-step manner: 1) a 2D convolution layer is applied on the 2D feature maps $\mathcal{F}^{RV}$ to get $\mathcal{F}^{RV'}$; 2) the 3D features $\mathcal{F}_k^{3D\leftarrow RV'}$ for the $k^{th}$ point is obtained by bilinear sampling among the $2 \times 2$ 2D neighbours of the corresponding RV position $p_k^{RV}$ on the feature map $\mathcal{F}^{RV'}$; 3) a MLP takes a concatenation of the original 3D point features $\mathcal{F}_k^{3D}$ and the interpolated features $\mathcal{F}^{3D\leftarrow RV'}$ as its input and outputs 27-channel ( $3 \times 3 \times 3$) vector per 3D point, where the first 18 channels denote the kernel offset $\Delta p_{k,n}$ and the remaining 9 channels are activated by a sigmoid function to obtain the kernel weight $\Delta m_{k,n}$. ### 3.4 Point Fusion The point fusion (PF) module fuses the features from the RV and the 3D points to enhance per-point features. For efficiency, it consists of a feature concatenation and two MLP layers. The output point features can serve as the input of the next stacked PRNet or the input of the prediction header. If more than one PRNets are stacked, we impose supervision on each to facilitate the training process. For inference, the output of the last PRNet is used for the final prediction. ### 3.5 Loss Function The amounts of data for different categories are highly unbalanced in the LiDAR semantic segmentation dataset (e.g. SemanticKITTI, nuScenes). For example, the proportions of road, building, and car are hundreds times than those of motorcyclist and traffic-sign. Therefore, we apply the weighted cross entropy (WCE) $\mathcal{L}_{wce}$ that emphasizes the rare categories as the following, $$ \begin{align*}\alpha_{c}=&\frac{1}{F_{c}+0.001}\\\mathcal{L}_{wce}=&-\frac{1}{N}\sum_{n=1}^{N}\sum_{c=1}^{C}\alpha_{c} y_{n}^{c}\log(\hat{y}_{n}^{c}),\end{align*} $$ where $y_n^c$ ( $y_n^c \in \{0,1\}$) and $\hat{y}_n^c$ ( $\hat{y}_n^c \in [0,1]$) are the ground-truth and the predicted probability of the $c^{th}$ class on the $n^{th}$ point. $F_c$ is the frequency, and $\alpha_c$ is the weight of the $c^{th}$ class. We also adopt the Lovázs-Softmax loss [Berman et al., 2018] $\mathcal{L}_{ls}$ to directly optimize the mean Intersection over Union (mIoU) metric. According to [Aksoy et al., 2020; Cortinhal et al., 2020], it can improve the mIoU performance of the LiDAR semantic segmentation task. The total loss $\mathcal{L}_{total}$ is the sum of the two loss terms and is defined as $$ \mathcal{L}_{t o t a l}=\mathcal{L}_{w c e}+3\mathcal{L}_{l s}. $$ ## 4 Experiments We evaluate the effectiveness and efficiency of the proposed PRNet on the public nuScenes [Caesar et al., 2020] and SemanticKITTI [Behley et al., 2019] single-scan LiDAR semantic segmentation benchmarks. ### 4.1 Experimental Setup Datasets. nuScenes for the LiDAR semantic segmentation is a newly released benchmark with 1,000 scenes collected in Boston and Singapore. Each LiDAR scan is collected by a ---
MethodsmIoURuntime(ms)barrierbicyclebuscarconstructionmotorcyclepedestriantraffic-conetrailertruckdriveableother_flatsidewalkterrainmanmadevegetation
RangeNet++ [Milioto et al., 2019]65.578.666.021.377.280.930.266.869.652.154.272.394.166.663.570.183.179.8
SalsaNext [Cortinhal et al., 2020]72.226.274.834.185.988.442.272.472.263.161.376.596.070.871.271.586.784.4
PolarNet [Zhang et al., 2020]71.067.574.728.285.390.935.177.571.358.857.476.196.571.174.774.087.385.7
AMVNet [Liong et al., 2020]77.2-----------------
Cylinder3D [Zhu et al., 2021]76.175.776.440.391.293.851.378.078.964.962.184.496.871.676.475.490.587.4
RPVNet [Xu et al., 2021]77.6-78.243.492.793.249.085.780.566.066.984.096.973.575.976.090.688.9
PRNet-1C [ours]75.32076.938.690.992.144.879.876.662.560.179.897.074.476.275.789.587.9
PRNet-2C [ours]78.04378.040.992.593.454.185.480.663.269.884.797.375.177.776.191.089.3
Table 1: Class-wise and mean IoU of the proposed PRNet and its competitors on the nuScenes validation set. Runtime measurements are taken on a single NVIDIA RTX 2080Ti GPU.
Velodyne HDL-32E 360° rotating LiDAR with 32 beams vertically. It splits 28,130 samples for training, 6,019 for validation, and 6,008 for testing. After merging similar categories and removing rare categories, 16 categories are used for the official evaluation. SemanticKITTI contains 43,552 LiDAR scans from 22 sequences collected in Germany. Each LiDAR scan is collected by a Velodyne HDL-64E 360° rotating LiDAR with 64 beams vertically. The training set (19,130 scans) consists of sequences from 00 to 10 except 08, and the sequence 08 (4,071 scans) is used for validation. The rest sequences (20,351 scans) from 11 to 21 are only provided with LiDAR point clouds and are used for the online leaderboards. The dataset is annotated with 28 categories, but it merges categories with different motion states to acquire 19 valid categories for the single-scan LiDAR semantic segmentation benchmark. Evaluation Metric. To evaluate the proposed PRNet and its competitors, we follow the official metric, namely mean Intersection over Union (mIoU), as the following, $$ m I o U=\frac{1}{C}\sum_{c=1}^{C}\frac{T P_{c}}{T P_{c}+F P_{c}+F N_{c}}, $$ where $TP_{c}$, $FP_{c}$, $FN_{c}$ are the true positive, false positive and false negative of the $c^{th}$ category, respectively. C is the total number of classes. Network Setup. In the experiments, we adopt two stacked PRNets, and each has a similar network architecture but different parameters. The dimensions of the input point features for these two nets are 5, and 64, respectively. The first 5 feature channels contain x, y, z, intensity and r. As shown in Figure 3, the first MLP layer of each PRNet outputs 64 feature channels. The details of the 2D FCN can be seen in the supplementary material. The two stacked PRNets output 64, 96 point feature channels, respectively. Finally, the segmentation results are acquired by applying an FC layer to the output of the last stacked PRNet. Note that PRNet-2C just doubles the RV feature channels of PRNet-1C. For SemanticKITTI, the parameters of LiDAR vertical field-of-view are set as $f_{up} = 3^{\circ}$ and $f_{down} = -25^{\circ}$. The input size of the RV branch is $64 \times 2048$. For nuScenes, we adopt the same configuration except $f_{up} = 20^{\circ}$ and $f_{down} = -40^{\circ}$. Training Details. All experiments are conducted with PyTorch FP32 on NVIDIA RTX 2080Ti GPU. The proposed PRNet is trained from scratch for 48 epochs with a batch size of 16 on 8 GPUs. Stochastic gradient descent (SGD) serves as the optimizer with a weight decay of 0.001, a momentum of 0.9, and an initial learning rate of 0.02, which is decayed by 0.1 every 10 epochs. Following the convention, the data augmentation strategies include random flipping along the x and y axes, random global scale sampled from $[0.95, 1.05]$, random rotation around the z axis, random Gaussian noise $\mathcal{N}(0, 0.02)$, and instance CutMix [Xu et al., 2021]. ### 4.2 Comparisons With the State-of-the-Arts Results on nuScenes. We report the performance of the proposed PRNet on the newly released nuScenes validation set. As shown in Table 1, our PRNet-2C achieves the best performance among all competing methods, and even outperforms the most competitive RPVNet for most categories. In terms of running time, the PRNet-2C is much faster compared with all methods except the SalsaNext, but SalsaNext is slower and has poorer mIoU performance than the PRNet-1C. Results on SemanticKITTI. The proposed PRNet is compared with the state-of-the-arts on the SemanticKITTI test set. As shown in Table 2, the methods are grouped as point-based, voxel-based, and range-based methods from top to bottom. We find that PRNet-2C outperforms all point-based and range-based methods by a large margin, and it can be comparable with the top-ranking voxel-based methods, while it runs much faster. Though the proposed PRNet-2C performs poorer compared with the top-ranking RPVNet on some hard categories, it runs 4 times faster with performance comparable with RPVNet on most categories. Limitation Discussion. For the categories that have fewer training samples and are easily confused with other categories, the proposed method performs worse than the voxel-based RPVNet, e.g. motorcyclist that is easily confused with bicyclist and motorcycle. The reason is that the proposed range-based method mainly encodes context on the 2D RV, but this can be remedied by introducing images. ### 4.3 Ablation Studies We make ablative analyses on the SemanticKITTI validation set to figure out the effectiveness of the proposed components. ---
MethodsmIoURuntime(ms)carbicyclemotorcycletruckother-vehiclepersonbicyclistmotorcyclistroadparkingsidewalkother-groundbuildingfencevegetationtrunkterrainpoletraffic-sign
Point-based Methods
PointNet [Qi et al., 2017a]14.6-46.31.30.30.10.80.20.20.061.615.835.71.441.412.931.04.617.62.43.7
PointNet++ [Qi et al., 2017b]20.1-53.71.90.20.90.20.91.00.072.018.741.85.662.316.946.513.830.06.08.9
RandLA-Net [Hu et al., 2020]53.9521.894.226.025.840.138.949.248.27.290.760.373.720.486.956.381.461.366.849.247.7
KPConv [Thomas et al., 2019]58.8-96.030.242.533.444.361.561.611.888.861.372.731.690.564.284.869.269.156.447.4
Voxel-based Methods
PolarNet [Zhang et al., 2020]54.374.393.840.330.122.928.543.240.25.690.861.774.421.790.061.384.065.567.851.857.5
AMVNet [Liong et al., 2020]65.3-96.259.954.248.845.771.065.711.090.171.075.832.492.469.185.671.769.662.767.2
SPVCNN [Tang et al., 2020]63.8187-------------------
SPVNAS [Tang et al., 2020]67.0-97.250.650.456.658.067.467.150.390.267.675.421.891.666.986.173.471.064.367.3
Cylinder3D [Zhu et al., 2021]67.817897.167.664.059.058.673.967.936.091.465.175.532.391.066.585.471.868.562.665.6
DRINet [Ye et al., 2021]67.56296.957.056.043.354.569.475.158.990.765.075.226.291.567.385.272.668.863.566.0
AF2S3Net [Cheng et al., 2021]69.7-94.565.486.839.241.180.780.474.391.368.872.553.587.963.270.268.553.761.571.0
RPVNet [Xu et al., 2021]70.3168 $^{{*}}$97.668.468.744.261.175.974.473.493.470.380.733.393.572.186.575.171.764.861.4
Ranged-based Methods
RangeNet++ [Milioto et al., 2019]52.282.391.425.734.425.723.038.338.84.891.865.075.227.887.458.680.555.164.647.955.9
SqueezeSegv3 [Xu et al., 2020]55.9124.392.538.736.529.633.045.646.220.191.763.474.826.489.059.482.058.765.449.658.9
SalsaNext [Cortinhal et al., 2020]59.527.491.948.338.638.931.960.259.019.491.763.775.829.190.264.281.863.666.554.362.1
Lite-HDSeg [Razani et al., 2021]63.8-92.340.055.437.739.659.271.654.193.068.278.329.391.565.078.265.865.159.567.7
PRNet-1C [ours]65.230/24.3 $^{{*}}$95.559.758.253.847.366.372.325.192.367.477.121.691.567.082.667.469.459.464.6
PRNet-2C [ours]67.263/43.5 $^{{*}}$95.762.160.056.348.471.075.028.692.466.375.921.892.972.284.470.468.663.571.0
Table 2: Class-wise and mean IoU of the proposed PRNet and its competitors on the SemanticKITTI test set. Runtime measurements are taken on a single NVIDIA RTX 2080Ti GPU, while $*$ means that it uses NVIDIA Tesla V100 GPU.
FrameworkBackboneRVPPRConvPFMmIoURT(ms)
aBaselineSalsaNext159.027.4
bPRNet-1CSalsaNext✓160.229.2
cSalsaNext✓✓161.229.6
dSalsaNext✓✓✓162.130.8
eOurs✓✓✓161.814.6
fOurs✓✓✓263.730
Table 3: The proposed framework analysis on the SemanticKITTI validation set. M is the number of the stacked PRNets. RT is the abbreviation of running time.
Effects of the Framework. This analysis is very important to illustrate the insight of our motivation. It starts with the baseline model, SalsaNext [Cortinhal et al., 2020], which is an open-sourced and top-ranking range-based method. As shown in Table 3, we progressively add the proposed components to the SalsaNext until it becomes the same as the proposed PRNet. It can be discovered that: 1) it achieves +1.2 mIoU gains when the post-processing kNN is replaced by the proposed PRConv (a, b); 2) the RVP outperforms the traditional range projection strategy that only keeps the nearest point within an RV pixel, by +1.0 mIoU (b, c). Based on the official code of SalsaNext, we find that only 78% of the whole 3D points are kept during projection to the RV, while the RVP keeps as much 3D points information as possible; 3) the feature fusion of the RV and the 3D points further improves the mIoU by +0.9 (c, d); 4) our 2D backbone is more efficient than that of SalsaNext (d, e, f); 5) two stacked PR-Nets perform better than a single one (e, f). Variants of the PRConv. We set the baseline as the bilinear sampling (BS) that acquires the point features by interpolating the four neighbors of the corresponding RV pixel. Afterwards, we add the kernel offset and the kernel weight.
TypeOffsetWeightmIoURuntime(ms)
aBS60.720
bPRConv61.124.3
c✓62.926.6
d✓✓63.730
Table 4: The PRConv analysis on the SemanticKITTI validation set. BS stands for bilinear sampling.
one by one to figure out the proposed PRConv. As shown in Table 4, the PRConv without the kernel offset and the kernel weight improves the +0.4 mIOU, compared with the baseline. Moreover, adding the kernel offset into PRConv leads to the biggest mIoU jump of +1.8. Finally, the complete PRConv equipped with the kernel offset and weight outperforms the baseline by +3.0 mIoU. ## 5 Conclusion In this paper, we present an end-to-end PRNet for real-time LiDAR semantic segmentation, which leverages the strengths of the point-based and range-based methods. The PRNet extracts features mainly on the RV and produces semantic predictions in the 3D space by fused features originating from both the 3D space and the RV space. To minimize the information loss, we propose the RVP operation for the RV projection. To avoid semantic ambiguity, a novel PRConv is proposed to transmit the RV features back to the 3D points dynamically. Experimental results on the SemanticKITTI and nuScenes benchmarks demonstrate the effectiveness and superiority of the proposed components. The PRNet has resolved two long-plagued issues for the range-based methods and demonstrates the potential of the range-based methods for both effectiveness and efficiency. Future works lie in integrating image information to further improve its performance. --- ## References [Aksoy et al., 2020] Eren Erdal Aksoy, Saimir Baci, and Selcuk Cavdar. Salsanet: Fast road and vehicle segmentation in lidar point clouds for autonomous driving. In 2020 IEEE Intelligent Vehicles Symposium (IV), pages 926–932. IEEE, 2020. [Behley et al., 2019] Jens Behley, Martin Garbade, Andres Milioto, Jan Quenzel, Sven Behnke, Cyrill Stachniss, and Jurgen Gall. Semantickitti: A dataset for semantic scene understanding of lidar sequences. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9297–9307, 2019. [Berman et al., 2018] Maxim Berman, Amal Rannen Tri, and Matthew B Blaschko. The lovász-softmax loss: A tractable surrogate for the optimization of the intersection-over-union measure in neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4413–4421, 2018. [Caesar et al., 2020] Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11621–11631, 2020. [Cheng et al., 2021] Ran Cheng, Ryan Razani, Ehsan Taghavi, Enxu Li, and Bingbing Liu. 2-s3net: Attentive feature fusion with adaptive feature selection for sparse semantic segmentation network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12547–12556, 2021. [Choy et al., 2019] Christopher Choy, Jun Young Gwak, and Silvio Savarese. 4d spatio-temporal convnets: Minkowski convolutional neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3075–3084, 2019. [Cortinhal et al., 2020] Tiago Cortinhal, George Tzelepis, and Eren Erdal Aksoy. Salsanext: Fast, uncertainty-aware semantic segmentation of lidar point clouds. In International Symposium on Visual Computing, pages 207–222. Springer, 2020. [Hu et al., 2020] Qingyong Hu, Bo Yang, Linhai Xie, Stefano Rosa, Yulan Guo, Zhihua Wang, Niki Trigoni, and Andrew Markham. Randla-net: Efficient semantic segmentation of large-scale point clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11108–11117, 2020. [Liong et al., 2020] Venice Erin Liong, Thi Ngoc Tho Nguyen, Sergi Widjaja, Dhananjai Sharma, and Zhuang Jie Chong. Amvnet: Assertion-based multiview fusion network for lidar semantic segmentation. arXiv preprint arXiv:2012.04934, 2020. [Milioto et al., 2019] Andres Milioto, Ignacio Vizzo, Jens Behley, and Cyrill Stachniss. Rangenet++: Fast and accurate lidar semantic segmentation. In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 4213–4220. IEEE, 2019. [Qi et al., 2017a] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 652–660, 2017. [Qi et al., 2017b] Charles R Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. arXiv preprint arXiv:1706.02413, 2017. [Razani et al., 2021] Ryan Razani, Ran Cheng, Ehsan Taghavi, and Liu Bingbing. Lite-hdseg: Lidar semantic segmentation using lite harmonic dense convolutions. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 9550–9556. IEEE, 2021. [Tang et al., 2020] Haotian Tang, Zhijian Liu, Shengyu Zhao, Yujun Lin, Ji Lin, Hanrui Wang, and Song Han. Searching efficient 3d architectures with sparse point-voxel convolution. In European Conference on Computer Vision, pages 685–702. Springer, 2020. [Thomas et al., 2019] Hugues Thomas, Charles R Qi, Jean-Emmanuel Deschaud, Beatriz Marcotegui, François Goulette, and Leonidas J Guibas. Kpconv: Flexible and deformable convolution for point clouds. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6411–6420, 2019. [Xu et al., 2020] Chenfeng Xu, Bichen Wu, Zining Wang, Wei Zhan, Peter Vajda, Kurt Keutzer, and Masayoshi Tomizuka. Squeezesegv3: Spatially-adaptive convolution for efficient point-cloud segmentation. In European Conference on Computer Vision, pages 1–19. Springer, 2020. [Xu et al., 2021] Jianyun Xu, Ruixiang Zhang, Jian Dou, Yushi Zhu, Jie Sun, and Shiliang Pu. Rpvnet: A deep and efficient range-point-voxel fusion network for lidar point cloud segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 16024–16033, 2021. [Ye et al., 2021] Maosheng Ye, Shuangjie Xu, Tongyi Cao, and Qifeng Chen. Drinet: A dual-representation iterative learning network for point cloud segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7447–7456, 2021. [Zhang et al., 2020] Yang Zhang, Zixiang Zhou, Philip David, Xiangyu Yue, Zerong Xi, Boqing Gong, and Hassan Foroosh. Polarnet: An improved grid representation for online lidar point clouds semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9601–9610, 2020. [Zhu et al., 2021] Xinge Zhu, Hui Zhou, Tai Wang, Fangzhou Hong, Yuexin Ma, Wei Li, Hongsheng Li, and Dahua Lin. Cylindrical and asymmetrical 3d convolution networks for lidar segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9939–9948, 2021.