Received April 21, 2020, accepted May 5, 2020, date of publication May 8, 2020, date of current version May 21, 2020. Digital Object Identifier 10.1109/ACCESS.2020.2993299 # Learning-Based Human Segmentation and Velocity Estimation Using Automatic Labeled LiDAR Sequence for Training WONJIK KIM $^{1,2}$, (Member, IEEE), MASAYUKI TANAKA $^{1,2}$, (Member, IEEE), MASATOSHI OKUTOMI $^{1}$, (Member, IEEE), AND YOKO SASAKI $^{2}$, (Member, IEEE) $^{1}$Department of Systems and Control Engineering, School of Engineering, Tokyo Institute of Technology, Tokyo 152-8550, Japan $^{2}$Artificial Intelligence Research Center, National Institute of Advanced Industrial Science and Technology, Tokyo 135-0064, Japan Corresponding author: Wonijk Kim () Corresponding author: Wonjik Kim () ## ABSTRACT In this paper, we propose an automatic labeled sequential data generation pipeline for human segmentation and velocity estimation with point clouds. Considering the impact of deep neural networks, state-of-the-art network architectures have been proposed for human recognition using point clouds captured by Light Detection and Ranging (LiDAR). However, almost all conventional datasets are either a collection of single LiDAR scanning with label information or sequential LiDAR scanning without label information. This limitation has disturbed the progress of research to date. Therefore, we have developed an automatic labeled sequential data generation pipeline, in which we can control any parameter or data generation environment with pixel-wise and per-frame ground truth segmentation and pixel-wise velocity information for human recognition. Our approach uses a precise human model and reproduces a precise motion to generate realistic artificial data. We present more than 7K sequences, where each sequence consists of 32 frames generated by the proposed pipeline. With the proposed sequence generator, we confirm that human segmentation performance is improved when using the sequential data compared to when using the data from a single LiDAR scan. We also evaluate our data by comparing with data generated under different conditions. In addition, we estimate pedestrian velocity with LiDAR by only utilizing data generated by the proposed pipeline. INDEX TERMS Artificial intelligent, machine learning, computer vision, LiDAR. ### I. INTRODUCTION Robot navigation depends on real-time, precise, and robust sensing. A robot should recognize its surrounding environment and objects such as pedestrians and other robots. The robot is also required to be robust for various kinds of situations. Therefore, LiDAR is often employed to acquire accurate 3D information with a high sampling frequency. For example, LiDAR has been utilized for mapping processes [^1] and robotics applications including robot navigation [^2]. Human recognition with LiDAR is a very important task in robot navigation. Deep neural networks have had a strong impact on computer vision. Although they have been mainly applied for single image processing [^3] [^4] [^5], neural networks for processing sequential images are now being actively researched [^6] [^7] [^8]. One important task is semantic segmentation, which focuses on the segmentation of object-like areas using sequential images with predefined class labels. Since deep neural networks have achieved high performance in the image segmentation field, various network architectures have also been proposed for handling the sequential data [^9], [^10]. However, the performance of the learning based approach strongly depends on the training dataset. Almost all sequential images datasets are not labeled at pixel-level in the entire frames. Therefore, many learning based semantic sequential images segmentation methods take weakly supervised learning to overcome the lack of ground truth [^9], [^10]. On the other hand, in this study, we tackled data collection by generating sequential data. We also focused on human segmentation with sequential data collected by LiDAR. In this paper, we define the 'frame' as data from a single LiDAR scan, and the 'sequence' as sequential data from constant LiDAR scanning. --- Human recognition with LiDAR data has been researched vigorously [^5], [^11], [^12]. According to [^12], the human recognition performance drastically decreases as distance between a human and LiDAR increases. Because the number of points is inversely proportional to the square of the distance between a human and LiDAR, humans in the distance are difficult to recognize only with the very sparse LiDAR data. On the other hand, the sequence provides not only shape but also motion information, the recognition accuracy of distant humans can be improved by utilizing this motion information. We have confirmed that the sequence-based approach improves the accuracy of human detection when compared to the frame-based approach. Collecting a sufficient amount of labeled data requires a significant cost in terms of both time and money. In this study, we develop an automatic 3D LiDAR data sequence generation pipeline for human detection and velocity estimation. In comparison to single frame data generation, data sequence generation is more challenging because we need to take into account the temporal consistency of the sequence. For example, all objects should be temporally continuous. In addition, the generated data sequence should be spatially and temporally realistic to ensure the network is accurately trained. In [^11], a method to generate a single frame of realistic 3D LiDAR data for the human detection has been proposed. In that method, a statistical human shape model [^13] is used to build different types of precise human shapes. That model is very useful for generating the single frame of LiDAR data. However, this information is insufficient for generating the realistic LiDAR data sequence. For sequence generation, it is necessary to consider 1) the human walking model, 2) human trajectory, and 3) the sensor position and pose. In our study, we incorporate a walking model and an observed walking trajectory. Combining these information considering the LiDAR trajectory, the 3D LiDAR data sequence is generated with human labels. One of our goals is to accelerate the research of learning-based sequential human segmentation with point cloud data. For that purpose, we have generated a large amount of labeled sequential data (more than 7K sequences) with the proposed data generation pipeline. We have trained several neural networks with different training policies. The trained networks have been evaluated in comparison with actual data collected by a real LiDAR sensor. All generated data and labeled real data are provided in the following URL. Trained network weight and test sample code also included. The main contributions of our work are: - We have proposed a fully automated labeled LiDAR sequence generation pipeline with data publication. • We have confirmed experimentally that human segmentation performance can be improved and velocity estimation can be done by using sequential LiDAR data. each step in the data generation procedure is explained in detail. The network architecture for the training sequence is described in IV. The specific policy of the training and experimental results are then discussed in Section V. Finally, we conclude our research with an outline of further improvements and potential future applications in Section VI. The remainder of this paper is organized as follows. We briefly review related work in Section II. In Section III, ### II. RELATED WORK ### A. DATASET OF DEPTH MAP After the release of the Microsoft Kinect in 2010 [^14], several RGB-D datasets have been published. RGB-D datasets for human recognition have also been provided, such as that for the re-identification of a person with RGB-D sensors [^15], BIWI RGBD-ID dataset [^16], and UPCV Gait dataset [^17]. As the Kinect cannot measure depths greater than 10 m, LiDAR sensors were employed to handle depths over 10 m. In addition, LiDAR sensors are used in auto driving technology. In this field, the KITTI dataset [^18] is widely used by many researchers [^19], [^20]. However, KITTI only provided 93K+ of depth data without labeling. Collecting labeled depth maps is still challenging. Pixel or point-wise labeling for 3D depth data is usually a challenging task that involves significant costs. Under the circumstances, the video game Grand Theft Auto was deployed to collect data [^5], [^21], [^22]. This approach may reduce the cost of data construction, but limitations still remain. Grand Theft Auto is not designed for research purposes; therefore, we cannot control specific properties of the circumstances of the simulation, such as the human body type and model deployment location. Automatic Labeled LiDAR Data [^11], [^12] have been released with 1M+ data including depth, xyz coordinates and pixel-wise human labels. Automatic Labeled LiDAR Data may cover demands in the frame domain; however, its application is not sufficient to address the requirements of the sequence domain. The SementicKITTI dataset [^23] provides manually labeled segmentation information of the entire KITTI dataset [^18]. However, the amount of KITTI data is limited and the labeling tasks need massive efforts. In addition, the datasets of depth map cannot handle velocity information because they only contain single frames, and not sequences on the time axis. ### B. DATASETS FOR RGB SEQUENTIAL IMAGES SEGMENTATION Following the increase in sequential images dataset demands, several datasets have been published. For example, the Freiburg-Berkeley Motion Segmentation dataset [^24] has been proposed for motion segmentation. In addition, SegTrack v1 [^25] and v2 [^26] have also been published for sequential images segmentation specialized with fast motion and complex changing shapes. In the case of sequential images object segmentation, the DAVIS challenge [^27] has been presented and updated since 2016. Accordingly, RGB sequential images datasets have been vigorously proposed and updated. On the other hand, to the best of our knowledge, fully labeled sequential images segmentation datasets for --- ![Image](images/96e6f42ea8f7b7b038d040ee08f2d3ed8af38ab54777efb632dd257b2c221f1b.jpg)
FIGURE 1. Overview of our data generation pipeline. Bold words represent controllable parameters. (Green boxes: Human walking model) Human models are built by weight and height, then they are combined with a human walking motion. Thereafter, they are deployed following the sampled trajectory. (Blue boxes: Background sequence) Sensor trajectory and background LiDAR data are refined with a frame length of sequence generation. (Yellow boxes: LiDAR data generation) Human walking models are synthesized with depth map sequentially. The synthesized pixel-wise depth maps are labeled based on the information from the human model deployed location. Then, pixel-wise velocity information of sensor origin is generated according to human velocity, velocity command, and labeled information.
LiDAR with large scale have not been proposed. Our research focuses on filling this gap. To address these problems, we constructed a sequential data generation pipeline enabling us to change any parameters and environments. Further, the proposed generator creates human labels and velocity information in the process, thereby incorporating this normally human related task into the computation cost. As a result, as long as sufficient computational resources are available, we can generate sequential data continuously. ### III. SEQUENTIAL DATA GENERATION PIPELINE One of the main contributions of this paper is the automatic generation of labeled sequences of LiDAR data considering the precise human model and motion, without involving manual labeling. Unlike frame generation, all of the object data must be connected in time-series for generating a sequence. The sequence generation pipeline comprises three steps: 1) background sequence collection, 2) human walking model generation, and 3) sequential LiDAR data generation. The details are described in the following subsections, and an overview of the pipeline is illustrated in Fig. 1. ### A. SENSOR TRAJECTORY AND BACKGROUND SEQUENCE EXTRACTION From the given sensor trajectory and background sequence, we extract a specified length of data. First, the start time is randomly selected. Then, we cut out the sensor trajectory and associated background sequence from that start time with the specified time length. Those data are used in the sequential LiDAR data generation process described in Section III-C. The extracted sensor trajectory is also used for the velocity calculation and the human depth map generation process detailed in Section III-C. The standardized sensor trajectory is also utilized for velocity calculation and the human depth map generation process detailed in Section III-C. We can employ any LiDAR data as background LiDAR data in simulations [^5], [^28], in real time [^29], [^30], and taken by ourselves. The sensor trajectory is determined automatically when the LiDAR data are determined. ### B. HUMAN WALKING MODEL For the human walking model, first, we build a human model based on the human body database [^13]. The walking sequence is also observed. We can generate the human walking mode by combining the observed walking sequence and the human model build with height and weight. We independently observed human trajectory. Assuming that the human model, the walking model, and the human trajectory can be sampled independently, we can construct various kinds of human walking models with associated human trajectories. However, since we considered them independently, it is necessary to contemplate the relationship between human velocity and stride length in order to construct a precise human walking model. For the human model database, we employed a ‘DhaibaWorks’ [^13] to build precise, functional 3D human models. DhaibaWorks supports editing and visualizing basic models such as 3D meshes and skeletal structures, including human models with motion [^31]. Using DhaibaWorks, we can easily generate a specific human model by setting human parameters such as height, weight, and action status [^32]. ### C. SEQUENTIAL LiDAR DATA GENERATION In the sequential LiDAR data generation step, random human models, walking trajectories, and LiDAR trajectory are sampled. Then, the depth map of the constructed human model is synthesized. Thereafter, the synthesized depth map of the human model and the background depth map associated to the LiDAR trajectory are combined to generate the training depth map for human segmentation. After generating a depth map, human models are resampled by the relationship between --- ![Image](images/881e8cfe2fba5eb71a0158b4e0f1693706f1373e825bafd2174c41f04f226242.jpg)
(a) Example 01
![Image](images/414a4915008c38f8904bd77f70dbf818437e213e0b868a7a495e9780707b5532.jpg)
(b) Example 02
FIGURE 2. Examples of human models built by DhaibaWorks. Figure (b) is at the moment of maximum distance between the left and right heels.
![Image](images/6693bd11e0886a15009e5d7dd2be469023ed012efb2ea8f036651bc553b1af94.jpg)
FIGURE 3. Coordinate systems. An uppercase character indicates that its origin is the ground and a lowercase character indicates that its origin is the sensor.
stride length and observed velocity in trajectory information. Thereafter, updated human models and the LiDAR position are relocated based on their trajectory information. With this loop flow, we can generate the LiDAR sequence. The entire coordinate system in this study is illustrated in Fig. 3. To synthesize the human model depth map, we virtually located the LiDAR sensor at $(X_{s}, Y_{s}, Z_{s}, Q_{s})$ of ground origin, where $Q_{s}$ is the sensor quaternion. Then, the human model depth map was synthesized, virtually inserting the human model at $(X_{h}, Y_{h}, 0)$ of ground origin, where $(X_{h}, Y_{h})$ is a sampled position in the human trajectory. The human model direction is also determined by the sampled human trajectory. Once the geometrical positions of the LiDAR sensor and the human model are provided, the associated depth map can easily be synthesized. Once the human model depth map is synthesized, this is simply combined with the background depth map by pixelwise minimum depth selection. The depth map taken by the LiDAR sensor usually includes holes or missing pixels whose depth could not be measured. We leave these holes as they are for the synthesis process because these types of holes are equally obtainable in a real sensing process. In addition, the human labeling task can be simultaneously performed because we know which pixels correspond to the human model depth map. In the velocity calculation process, pixel-wise velocity information of sensor origin is generated. Human velocity of ground origin $V_{h}$, and sensor velocity of ground origin $V_{s}$ can be calculated by sensor trajectory. In addition, the transform function from the ground origin to the sensor origin can also be calculated using the sensor trajectory. Therefore, the velocity information for each pixel can be generated as follows, where p is a pixel in a frame: $$ \begin{aligned}\boldsymbol{v}_{p}=\begin{cases}\boldsymbol{V}_{h}-\boldsymbol{V}_{s}&if~human~pixel\\-\boldsymbol{V}_{s}&if~background~pixel\end{cases}\end{aligned} $$ Figure 4 shows examples of the generated sequence. Detailed information for the generating Fig. 4 is described in Section V-A. IV. NETWORK ARCHITECTURE FOR SEQUENCE TRAINING To utilize the generated sequences for training data, we designed a network architecture for human segmentation and velocity estimation with depth sequence as shown in Fig. 5. The architectures of feature extraction and decoding network shown in Fig. 5-(b) and (c) are inspired by Fully Convolutional Network (FCN) [^3]. FCN shows effectiveness for image segmentation [^3] and is also known to be valid for LiDAR data [^11], [^12]. However, since the FCN only treats a single frame as input, we expanded the FCN architecture to feature extraction network and decoding network for handling multiple frames efficiently. The architecture for segmentation is described in black ink in Fig. 5. First, we sampled depth frames from the sequence. Thereafter, each input was computed by weight shared convolutional layers for producing the features. Next, we concatenated each feature in the channel axis. Then we adopted another convolutional layer to concatenate features as a decoder. For activation, we employed softmax for the segmentation task. With this architecture, we can tune the network by feeding the sampled depth sequence and the label of the last frame in the sampled sequence. We used categorical cross-entropy for the loss function of the segmentation task. The architecture for velocity estimation is described in red ink in Fig. 5. We concatenated the segmentation result and concatenated features in the channel axis. Then we adapted the other convolutional layers to re-concatenate the features as a decoder. For activation, we employed linear for the velocity estimation task. We used the Mean Square Error (MSE) for the loss function of the velocity estimation. In addition, we did not consider defect pixels in the input scene for MSE calculation. For sequential data learning, we randomly selected a piece of sequential data from the training dataset at every step. Then, a certain frame length of sequentially sampled training data could be generated from the selected sequence. ### V. EVALUATION For evaluation, we generated 1,108 sequences as described in table 1, then 1,000 sequences were used for training and 108 sequences were used for validation and estimation. As a result, 32,000 frames were used for training. We only used 108 final frames in each sequence for estimation. We also prepared 0.1K of manually labeled real data for evaluation. --- ![Image](images/e55c640ee59bd26413eebc59eb282b614b9b31bafa6b50ba8c75d4b7d6fe71c8.jpg)
(b) Point cloud of frame #16/32
(c) Point cloud of frame #32/32
FIGURE 4. Example frames of a generated sequence in point cloud. Blue points denote the background and red points denote a human. Notice that human models are walking following their individual trajectory and the background scene is also changing following the LiDAR trajectory.
![Image](images/424991271c3bc03d1cea07f313b2d9f51d32fc0245752a19b1d6820544621b2d.jpg)
(a) Entire network
(b) Feature extraction network architecture
(c) Decoding network architecture
FIGURE 5. Network architecture for learning sequence. This architecture can be trained with different numbers of input frames. The convolutional layer parameters are denoted as "Conv-{number of channels} {kernel size}". All convolutional layers are activated with a ReLU function except "Conv-2 (1,1)" in decoding network.
TABLE 1. Data generation properties.
DescriptionSpecification
Number of detector pairs32
Limitation in horizontal scanning1024
Vertical scanning range+10.67 to -30.67°
Angular resolution in vertical1.33°
Angular resolution in horizontal0.2°
Human number in one frame1 to 30
Output velocity unitmm/s
Frame number in one sequence32
Data sampling rate10 [Hz]
In this Section, we employed an Intersection over Union (IoU) in this experiment. An IoU was calculated as positive as a human. We computed an IoU as a True Positive (TP), False Positive (FP), and False Negative (FN) as follows: $$ IoU=\frac{TP}{TP+FP+FN} $$ Since this study focuses on human segmentation, the IoU, which only evaluates human detection performance, is an appropriate metric. For evaluating the performance in a specific distance range, IoU is calculated by considering only the evaluation distance range on the generated test data. ### A. DATA GENERATION SETTING The background sequence was collected in the Miraikan 3rd floor using a HDL-32E LiDAR. In the background sequence collection step, the velocity command to the LiDAR equipped robot and the coordinate transformation matrix from the ground origin to the LiDAR origin were also recorded. --- ![Image](images/20aad5ff99a5a1853123897f6c7a846bc472337cf8a2fb44148facdcd99a8b23.jpg)
FIGURE 6. LiDAR and human trajectories. The green points denote the floor, red points denote LiDAR trajectory, and blue points denote human trajectory. X marks denote the sensor installation position, and 10 random human trajectories are illustrated for example. The xy coordinate for ground origin is also shown. The entire length of the figure in the X-axis is approximately 50 [m].
TABLE 2. Real data for evaluation.
DescriptionSpecification
LocationMiraikan [^33]
Date2017.Aug.04
LiDAR modelVelodyne HDL32E
LiDAR height from ground800 [mm]
Sequence number100
Labeling methodManual
TABLE 3. Human model specification.
Height (mm)12001400160017001800
Weight (kg)15 20 3020 30 4040 50 7050 60 8050 70 90
The LiDAR trajectory is estimated using 'Real-Time 6DoF Monte-Carlo Localization' [^34]. the estimated LiDAR trajectory is shown in Fig. 6. In a real world environment, human beings may have multiple postures including standing, walking, and running. However, we assume that the human is always walking during the sequence generation for this study. Walking motion data is required to build an artificial human walking model. We used one cycle of walking data consists of 230 frames that was collected in [^35]. We estimated the walking stride length required for the walking human model sampling explained in Section III-C based on a maximum distance between the left heel to the right heel of the human model. For constructing the human model, we take fifteen typical combinations of height and weight as summarized in Table 3. We believe that these combinations cover a variety of relevant scenarios. For gathering human trajectory, the HOKUYO UTM-30LX sensor [^36] was installed at a fixed laser sensor position as shown in Fig 6. With the HOKUYO UTM-30LX, we can obtain the sequential human location, direction, and velocity. Therefore, we used collected visitors' trajectories gathered on Sep 21, 22 and Dec 06, 07 in 2018 [^37]. As a result, 70,300 different kinds of walking trajectories were utilized for sequence generation. Fig. 6 shows an example of the walking trajectories collected. By observing real walking trajectories, we can also avoid deploying human models into unreachable areas. With this information, we now have 230 frames of walking motion data, fifteen combinations of the human model, and 70,300 kinds of trajectories. As such, a total of 242,535,000 different types of human walking models can be generated. For generating velocity information, we used the human velocity command linked to a LiDAR equipped robot, and the coordinate transformation matrix from the ground origin to the sensor origin. The human velocity is recorded as xy coordinates for ground origin. Therefore, we describe the human velocity as the xy coordinate of the sensor origin using the coordinate transformation matrix. Then, we can obtain the pixel-wise human velocity by substituting each human velocity to a whole pixel in each human label. For background velocity, we assumed that the opposite direction of velocity commands of ground origin are the same as the background velocity of sensor origin. Therefore, pixel-wise background velocity can be calculated by considering the velocity command. As a result, we can obtain the pixel-wise velocity map in xy coordinates by composing human and background velocities of sensor origin. We denote that the x-axis of sensor origin points to the forward direction of the LiDAR. The parameters for the LiDAR data generation are summarized in Table 1. These parameters contain depth, xyz coordinates, human label, and velocity map in HDF5 format. We also provide further specific information in the shape of an xml file. Xml files contain a human number in the depth scene, location, weight, and height of each human model. The sampling rates of both human trajectory and LiDAR trajectory are 10 [Hz]. ### B. TRAINING PARAMETERS We only used generated data for training data in this study. From the Table 1, the size of the input frame is $32 \times 1024$. We employed Adam [^38] with the learning rate = 0.001 and decay = 0.001 for the optimizer. We also set the weight to human label in every training data as background pixel number/human pixel number for each of the scenes. In addition, we set the weight to categorical cross-entropy loss for segmentation as 10,000, MSE for the background velocity estimation was set to 1, and MSE for human velocity estimation was set to 1,001. ### C. EXPERIMENTAL RESULTS We evaluated a single type of network for learning frame, and four types of networks for learning sequence. For those networks, we set the parameters as 1,000 for the epoch, 1 for batch size, and 150 for step for epoch. First, we trained the single frame network which is a reference of frame domain learning with the same architecture in Fig. 5. Thus, we set the sampling frame length as 2, 4, 8, and 16 for learning sequence. The manually labeled 0.1K of real data and 108 sequences generated by the proposed pipeline were used for the evaluation. The specific information about real data is ---
TABLE 4. Comparison of the networks for human segmentation. Criteria range denotes the evaluation distance range. For example, '0 to 4 [m]' means that the distance range between 0 [m] and 4 [m] from LiDAR was only considered for metric calculation. We would also like to point out that defect pixels in the depth map are not considered for evaluation.
Criteria rangeSingle frameSequence
2-frame4-frame8-frame16-frame
Generated data0 to $\infty$ [m]0.43580.58690.54610.55230.6443
0 to 4 [m]0.63010.77740.72780.72500.8141
4 to 8 [m]0.23460.40540.37400.38310.4695
8 to $\infty$ [m]0.07250.13230.12220.12720.1622
Real data0 to $\infty$ [m]0.50970.55880.55310.55500.5883
0 to 4 [m]0.64020.68010.68370.68860.7041
4 to 8 [m]0.40670.45430.45210.43200.4668
8 to $\infty$ [m]0.18830.22340.19710.19650.2309
![Image](images/38439c9055539a00d2fce874d0c95e638e4107ada4891a76485ec279f734bd74.jpg)
(a) Ground truth of generated data in point cloud
![Image](images/4d77368c5622f2fa2efb81b26b0cccc3130ee8705ec26052977db8967be7a673.jpg)
(b) Prediction with 16-frame in point cloud
FIGURE 7. An example of segmentation with 16-frame. In (a), blue points denote the background and red points denote a human label. In (b), blue points denote that the estimated result is the background and red points denote that the estimated result is a human.
described in Table 2. Note that the velocity unit is unified as [m/s] and the training data do not include test data. ## 1 ) HUMAN SEGMENTATION Table 4 shows the evaluation results. From the result, the ‘Sequence’ shows performance improvement than ‘Frame’ in every criteria range. In the case of the single frame, it shows comparable performance in ‘0 to 4 [m]’. However, the performance of the single frame drastically decreases as the criteria range recedes. This decrease problem can be resolved by using a multi frame. For example, there is an improvement of more than 23.4% in ‘4 to 8 [m]’ and 8.9% in ‘8 to ∞ [m]’ in 16-frame. Furthermore, we can observe that utilizing sequential data also improves human segmentation performance in ‘0 to 4 [m]’ and improves the overall network performance. According to these experimental results, we conclude that utilizing the multi frame can improve performance for recognizing far human than a single frame in a human segmentation task with LiDAR sensing. Inform that the results of 16-frame are used for examples in Fig. 7 and 8 because 16-frame shows the best score in Table 4.
TABLE 5. Comparison of networks trained with different data.
human modeltrajectorywalking modelIoU
✓0.2276
✓✓0.3318
✓✓0.3395
✓✓✓0.3814
To confirm the effect of the sequence feature, we compared four different datasets as shown in Table 5. In the data that excludes the trajectory and walking model, all human walking models are fixed in a specific point and state. In the data without a walking model, all human walking models are also sliding along their trajectories in a fixed state. In case of the data without trajectory, all human walking models are walking in a fixed position. We trained the 16-frame with 100 sequences in each dataset and evaluated this with real data for the IoU. The results are shown in Table 5. According to Table 5, the network trained by data with all information shows the best score. We can also observe that the scores of the network trained by data excluding the walking model and trajectory are higher than data without both trajectory and walking model. Then, we assume that consideration of trajectory and walking model affects performance improvement.
TABLE 6. Effect of velocity estimation in 16-frame.
NetworkVelocity estimation
w/ow/
Generated data0.57090.6443
Real data0.52260.5883
We also confirmed the effect of velocity estimation on proposed network architecture. Table 6 shows the IoU comparison using the proposed architecture in 16-frame. From the results of the table 6, we can see that training using not only human labels but also velocity information shows higher --- ![Image](images/64a5493f3562df114b193892afe9c549974da1dc2cf3331439d41b274f63f536.jpg)
(a) Color board
(b) Depth frame #01
![Image](images/17fc545263e0e17d3f0f2198c23f4ad2755f2f50fcf54791cc85a4e569d7074a.jpg)
(c) Human velocity #01
![Image](images/ad828bb0965811f960f376dbc153bfcd36f30ea302a393d2a45bdffa6074a43b.jpg) ![Image](images/5249bde355cb1ff8ac25d5d698b858a130bac7cf342eb4befdc552ed670eda97.jpg)
(d) 16-frame estimation #01
(e) Depth frame #02
![Image](images/96f94f997e9965d31b956ef42e18ec31d81499dfbb03973d1265cf0d27bb4ac2.jpg) ![Image](images/2a1510783a3f95de3b9fb63bab9ee26b0b60a4802ed98674e20cc0f7103e2a1f.jpg) ![Image](images/90a93268d096fd60f7ac19fa973a1a0d79f12c392ac242deb4e71ac135689def.jpg)
(f) Human velocity #02
(h) Depth frame #03
(g) 16-frame estimation #02
![Image](images/4cf6b43c72d37463fb57797aec303d8820a8aeb08211dc24bcb522c41845ea64.jpg)
(i) Human velocity #03
![Image](images/bfd04babb37d517d6e513f7c02bdaddf477eafadb3f338a88a63e1b5559f65ca.jpg)
(j) 16-frame estimation #03
FIGURE 8. Estimation of human velocity. Figures (d), (g), and (j) show the estimation results of human velocity for depth frames (b), (e), and (h), respectively, while (c), (f), and (i) show the corresponding ground truth human velocity. The meaning of color is shown in Fig. (a).
TABLE 7. Comparison of networks in real data.
NetworkIoU
Single network [^12]0.3490
Ensemble network [^12]0.3656
Proposed in 16-frame0.5883
performance than the training using only human labels. This indicates that the performance of human segmentation can be improved by directly considering the velocity information rather than simply inputting sequential data. From the human segmentation results, we assume that the network can extract the difference features in the time axis from sequential data, and the features can improve the segmentation task in far distance. In addition, we also assume that a network within the frame domain is capable of learning the human shape only, whereas the sequence domain is capable of learning both the human shape and its movement. Accordingly, we conclude that the performance decrease problem with distance can be solved by taking time-series information into consideration. The example of human segmentation is illustrated in Fig. 7. In recent study [^12], ensemble network was proposed and evaluated using same real data. Therefore, the proposed network was compared with the results reported in [^12] as shown in Table 7. We can confirm that the proposed network shows significantly better score than other networks. ## 2 ) ESTIMATION OF HUMAN VELOCITY As shown in Fig. 5, the network can estimate the human segmentation and pixel-wise velocity map. Then, we can derive the velocity map of the segmented area. The examples of human velocity estimation are illustrated in Fig. 8. According to the Fig. 8, the estimated velocities show similar tendency with ground truth. As a consequence, we conclude that velocity estimation with LiDAR only is a feasible task. ### VI. CONCLUSIONS In this paper, we propose a fully automated sequence generation pipeline using a precise human model and motion for human detection with velocity estimation using LiDAR. With this process, we can easily generate labeled data with any properties for LiDAR. Following this result, we conclude that sequential data can improve the performance of human segmentation when compared to data collected from the frame domain only. Furthermore, we also confirm the possibility of velocity estimation with LiDAR only. We present 0.1K of labeled real data, and more than 7K of generated sequences with human labels and velocity maps. With these sequences, we were able to confirm the effectiveness of using sequential data over frame domain data. We have considered two main points with regard to future improvements of our work. The first relates to pipeline improvement. Although we used a confirmed method to produce the human model, this is not entirely representative of the real world. As such, we will take into consideration pose, fashion, and other conditions for more accurate simulations. Because we made the human model walk, we will also try to apply different walking poses, stride lengths and other conditions. In addition, it should be noted that networks were only trained with generated data in this study. A comparison of network performance in manually labeled training data and generated training data would be beneficial to this study. However, based on our investigations, this particular experiment is expected to incur significant costs. The second point for improvement relates to utilizing sequences. Because generated data are sequential, this can be applied to many tasks such as human tracking and trajectory prediction. Our future works include tuning the network architecture and multimodal extension as in [^39], [^40]. Future work will elaborate on how these points for improvement can be achieved. ## REFERENCES [^1]: B. Schwarz, “LIDAR: Mapping the world in 3D,” Nature Photon., vol. 4, no. 7, p. 429, 2010. --- [^2]: S. Niijima, Y. Sasaki, and H. Mizoguchi, “Autonomous navigation of electric wheelchairs in urban areas on the basis of self-generated 2D drivable maps,” in Proc. IEEE/ASME Int. Conf. Adv. Intell. Mechatronics (AIM), Jul. 2018, pp. 1081–1086. [^3]: J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks for semantic segmentation,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2015, pp. 3431–3440. [^4]: C. Farabet, C. Couprie, L. Najman, and Y. LeCun, “Learning hierarchical features for scene labeling,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 35, no. 8, pp. 1915–1929, Aug. 2013. [^5]: B. Wu, A. Wan, X. Yue, and K. Keutzer, “SqueezeSeg: Convolutional neural nets with recurrent CRF for real-time road-object segmentation from 3D LiDAR point cloud,” 2017, arXiv:1710.07368. [Online]. Available: [^6]: C. Feichtenhofer, H. Fan, J. Malik, and K. He, “SlowFast networks for video recognition,” 2018, arXiv:1812.03982. [Online]. Available: [^7]: D. Tran, L. Bourdev, R. Fergus, L. Torresani, and M. Paluri, “Learning spatiotemporal features with 3D convolutional networks,” in Proc. IEEE Int. Conf. Comput. Vis. (ICCV), Dec. 2015, pp. 4489–4497. [^8]: C. Gu, C. Sun, D. A. Ross, C. Vondrick, C. Pantofaru, Y. Li, S. Vijayanarasimhan, G. Toderici, S. Ricco, R. Sukthankar, C. Schmid, and J. Malik, "AVA: A video dataset of spatio-temporally localized atomic visual actions," in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., Jun. 2018, pp. 6047–6056. [^9]: S. Hong, D. Yeo, S. Kwak, H. Lee, and B. Han, “Weakly supervised semantic segmentation using Web-crawled videos,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jul. 2017, pp. 7322–7330. [^10]: G. Seguin, P. Bojanowski, R. Lajugie, and I. Laptev, “Instance-level video segmentation from object tracks,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2016, pp. 3678–3687. [^11]: W. Kim, M. Tanaka, M. Okutomi, and Y. Sasaki, “Automatic labeled LiDAR data generation based on precise human model,” in Proc. Int. Conf. Robot. Autom. (ICRA), May 2019, pp. 43–49. [^12]: W. Kim, M. Tanaka, M. Okutomi, and Y. Sasaki, “Automatic labeled LiDAR data generation and distance-based ensemble learning for human segmentation,” IEEE Access, vol. 7, pp. 55132–55141, 2019. [^13]: M. Mochimaru, M. Kouchi, N. Miyata, M. Tada, Y. Yoshida, K. Aoki, K. Kawachi, and Y. Motomura, “Dhaiba: Functional human models to represent variation of shape, motion and subjective assessment,” SAE Tech. Paper 2006-01-2345, 2006. [^14]: Microsoft. Accessed: Jul. 17, 2019. [Online]. Available: [^15]: I. B. Barbosa, M. Cristani, A. Del Bue, L. Bazzani, and V. Murino, “Re-identification with RGB-D sensors,” in Proc. Eur. Conf. Comput. Vis. Springer, 2012, pp. 433–442. [^16]: M. Munaro, A. Basso, A. Fossati, L. Van Gool, and E. Menegatti, “3D reconstruction of freely moving persons for re-identification with a depth sensor,” in Proc. IEEE Int. Conf. Robot. Autom. (ICRA), May 2014, pp. 4512–4519. [^17]: D. Kastaniotis, I. Theodorakopoulos, C. Theoharatos, G. Economou, and S. Fotopoulos, “A framework for gait-based recognition using Kinect,” Pattern Recognit. Lett., vol. 68, pp. 327–335, Dec. 2015. [^18]: A. Geiger, P. Lenz, and R. Urtasun, “Are we ready for autonomous driving? The KITTI vision benchmark suite,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2012, pp. 3354–3361. [^19]: C. Premebida, J. Carreira, J. Batista, and U. Nunes, “Pedestrian detection combining RGB and dense LIDAR data,” in Proc. IEEE/RSJ Int. Conf. Intell. Robots Syst., Sep. 2014, pp. 4112–4117. [^20]: D. Levi, N. Garnett, and E. Fetaya, “StixelNet: A deep convolutional network for obstacle detection and road segmentation,” in Proc. Brit. Mach. Vis. Conf., 2015, pp. 1–109. [^21]: M. Johnson-Roberson, C. Barto, R. Mehta, S. N. Sridhar, K. Rosaen, and R. Vasudevan, “Driving in the matrix: Can virtual worlds replace human-generated annotations for real world tasks?” in Proc. IEEE Int. Conf. Robot. Autom. (ICRA), May 2017, pp. 746–753. [^22]: S. R. Richter, V. Vineet, S. Roth, and V. Koltun, “Playing for data: Ground truth from computer games,” in Proc. Eur. Conf. Comput. Vis. Springer, 2016, pp. 102–118. [^23]: J. Behley, M. Garbade, A. Milioto, J. Quenzel, S. Behnke, C. Stachniss, and J. Gall, “SemanticKITTI: A dataset for semantic scene understanding of LiDAR sequences,” in Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), Oct. 2019, pp. 9297–9307. [^24]: T. Brox and J. Malik, “Object segmentation by long term analysis of point trajectories,” in Proc. Eur. Conf. Comput. Vis. Springer, 2010, pp. 282–295. [^25]: D. Tsai, M. Flagg, and J. Rehg, “Motion coherent tracking using multi-label MRF optimization,” in Proc. Brit. Mach. Vis. Conf., 2010, pp. 190–202. [^26]: F. Li, T. Kim, A. Humayun, D. Tsai, and J. M. Rehg, “Video segmentation by tracking many figure-ground segments,” in Proc. IEEE Int. Conf. Comput. Vis., Dec. 2013, pp. 2192–2199. [^27]: S. Caelles, A. Montes, K.-K. Maninis, Y. Chen, L. Van Gool, F. Perazzi, and J. Pont-Tuset, “The 2018 DAVIS challenge on video object segmentation,” 2018, arXiv:1803.00557. [Online]. Available: [^28]: APOLLO. Accessed: Jul. 27, 2019. [Online]. Available: [^29]: A. Geiger, P. Lenz, C. Stiller, and R. Urtasun, “Vision meets robotics: The KITTI dataset,” Int. J. Robot. Res., vol. 32, no. 11, pp. 1231–1237, Sep. 2013. [^30]: M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele, “The cityscapes dataset for semantic urban scene understanding,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2016, pp. 3213–3223. [^31]: Y. Endo, M. Tada, and M. Mochimaru, “Hand modeling and motion reconstruction for individuals,” Int. J. Autom. Technol., vol. 8, no. 3, pp. 376–387, May 2014. [^32]: Y. Endo, M. Tada, and M. Mochimaru, “Estimation of arbitrary human models from anthropometric dimensions,” in Proc. Int. Conf. Digit. Hum. Modeling Appl. Health, Saf., Ergonom. Risk Manage. Springer, 2015, pp. 3–14. [^33]: Miraikan. Accessed: May 27, 2018. [Online]. Available: [^34]: A. Ohsato, Y. Sasaki, and H. Mizoguchi, “Real-time 6DoF localization for a mobile robot using pre-computed 3D laser likelihood field,” in Proc. IEEE Int. Conf. Robot. Biomimetics (ROBIO), Dec. 2015, pp. 2359–2364. [^35]: Y. Kobayashi, H. Hobara, T. A. Heldoorn, M. Kouchi, and M. Mochimaru, “Age-independent and age-dependent sex differences in gait pattern determined by principal component analysis,” Gait Posture, vol. 46, pp. 11–17, May 2016. [^36]: HOKUYO. Accessed: Feb. 19, 2019. [Online]. Available: [^37]: Y. Sasaki and J. Nitta, “Long-term demonstration experiment of autonomous mobile robot in a science museum,” in Proc. IEEE Int. Symp. Robot. Intell. Sensors (IRIS), Oct. 2017, pp. 304–310. [^38]: D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” 2014, arXiv:1412.6980. [Online]. Available: [^39]: O. Erkent, C. Wolf, C. Laugier, D. S. Gonzalez, and V. R. Cano, “Semantic grid estimation with a hybrid Bayesian and deep neural network approach,” in Proc. IEEE/RSJ Int. Conf. Intell. Robots Syst. (IROS), Oct. 2018, pp. 888–895. [^40]: D. Feng, C. Haase-Schutz, L. Rosenbaum, H. Hertlein, C. Glaser, F. Timm, W. Wiesbeck, and K. Dietmayer, “Deep multi-modal object detection and semantic segmentation for autonomous driving: Datasets, methods, and challenges,” IEEE Trans. Intell. Transp. Syst., to be published. ![Image](images/c69f1939355a29300cc7b298ab140664a45f2ee8b13ae6b46af7f6f94d827768.jpg) WONJIK KIM (Member, IEEE) received the Bachelor of Engineering degree from the Department of Control and Systems Engineering, Tokyo Institute of Technology, in 2018, and the Master of Engineering degree from the Department of Systems and Control Engineering, Tokyo Institute of Technology, in 2020, where he is currently pursuing the Ph.D. degree. He was a Research Assistant with the Artificial Intelligence Research Center, National Institute of Advanced Industrial Technology (NIAI). Science and Technology, in 2018. His research interests include artificial intelligence, machine learning, robotics, and optimization. --- ![Image](images/ab81cc545e6b0546a9c0ef5513c2e7b6647917071373f04dac0bbe766c98384e.jpg) MASAYUKI TANAKA (Member, IEEE) received the bachelor's and master's degrees in control engineering and the Ph.D. degree from the Tokyo Institute of Technology, in 1998, 2000, and 2003, respectively. From 2003 to 2004, he was a Software Engineer with Agilent Technology. From 2004 to 2008, he was a Research Scientist with the Tokyo Institute of Technology. From 2008 to 2016, he was an Associate Professor with the Graduate School of Science and Engineering, Tokyo Institute of Technology. From 2013 to 2014, he was a Visiting Scholar with the Department of Psychology, Stanford University. From 2016 to 2017, he was an Associate Professor with the School of Engineering, Tokyo Institute of Technology. Since 2017, he has been a Senior Researcher with the National Institute of Advanced Industrial Science and Technology. MASATOSHI OKUTOMI (Member, IEEE) received the B.Eng. degree from the Department of Mathematical Engineering and Information Physics, The University of Tokyo, Tokyo, Japan, in 1981, the M.Eng. degree from the Department of Control Engineering, Tokyo Institute of Technology, Tokyo, in 1983, and the D.Eng. degree from the Tokyo Institute of Technology for his research on stereo vision. He joined the Canon Research Center, Canon Inc., Tokyo, in 1983. From 1987 to 1990, he was a Visiting Research Scientist with the School of Computer Science, Carnegie Mellon University, USA. Since 1994, he has been with the Tokyo Institute of Technology, where he is currently a Professor with the Department of Systems and Control Engineering, School of Engineering. ![Image](images/d76b41921756ae23aec465002801abd59196afb59d7820541bce8ae81dca6d62.jpg) ![Image](images/c4314ed98012cf55c5b55514cc4b29aee0115248a558338d46b8dd9ffea169e3.jpg) YOKO SASAKI (Member, IEEE) received the Ph.D. degree in mechanical engineering from the Tokyo University of Science for her research on auditory system for a mobile robot, in 2009. She joined the Digital Human Research Center, National Institute of Advanced Industrial Science and Technology (AIST). Since 2015, she has been a Senior Researcher with the Artificial Intelligence Research Center, AIST. Her work involves developing autonomous mobile robot and its sensing. and navigation technology, especially in human living environment.