# Convolutional Networks for Semantic Heads Segmentation using Top-View Depth Data in Crowded Environment Daniele Liciotti, Marina Paolanti, Rocco Pietrini, Emanuele Frontoni, Primo Zingaretti, Department of Information Engineering (DII), Università Politecnica delle Marche Via Brecce Bianche, 12, 60131, Ancona, Italy Email: , , , , Abstract—Detecting and tracking people is a challenging task in a persistent crowded environment (i.e. retail, airport, station, etc.) for human behaviour analysis of security purposes. This paper introduces an approach to track and detect people in cases of heavy occlusions based on CNNs for semantic segmentation using top-view depth visual data. The purpose is the design of a novel U-Net architecture, U-Net3, that has been modified compared to the previous ones at the end of each layer. In particular, a batch normalization is added after the first ReLU activation function and after each max-pooling and up-sampling functions. The approach was applied and tested on a new and public available dataset, TVHeads Dataset, consisting of depth images of people recorded from an RGB-D camera installed in top-view configuration. Our variant outperforms baseline architectures while remaining computationally efficient at inference time. Results show high accuracy, demonstrating the effectiveness and suitability of our approach. ### I. INTRODUCTION Detecting and tracking people is an important and challenging task for many interactive and intelligent systems, such as visual surveillance and human-computer interaction. In recent years, this has been a topic of considerable research and robust methods have been developed to track isolated or a small number of humans for which only transient occlusion exists [^1]. However, tracking in more crowded situations, where a lot of people are present, is a problem that remains largely open due to many serious issues, such as the exhibition of persistent occlusion, change of appearance, dynamic and complex background [^2]. In fact, all these issues cause extreme problems when encountered with the crowded environment, since conventional surveillance technologies have difficulty in understanding the image [^3]. In these situations, sensors viably adopted are RGB-D cameras because of their reliability, availability and affordability. Researchers have demonstrated the great value (both in efficiency and accuracy) of depth camera in coping with severe occlusions among humans and complex background [^4]. The combination of high-resolution depth and visual information opens up new challenges and opportunities for activity recognition and people tracking for many applications. Reliable depth maps can provide valuable additional information to significantly improve tracking and detection results [^5]. When dealing with 2D images, video, and even 3D data the semantic segmentation is proved to be of the high-level task. It paves the way towards the complete scene understanding and it is being tackled using deep learning architectures, usually Convolutional Neural Networks (CNNs), which are surpassing other approaches by a large margin in terms of accuracy and sometimes even efficiency [^6], [^7], [^8], [^9]. Semantic segmentation is understanding an image at pixel level. For this reason, each pixel of the image has to be assigned to an object class. In addition to object recognition, this technique is used to define the boundaries of each object. Silberman et al. [^10] were the first to deal with the task of semantic full image labeling using depth information. After the release of the NYU dataset, the indoor RGB-D pixel-wise semantic segmentation has gained popularity [^11]. This dataset has proved the benefit of the depth channel to improve segmentation. Their approach used features such as RGB-SIFT, depth-SIFT and pixel location as input to a neural network classifier to predict pixel. Other datasets employed for semantic image segmentation present the objects centered into the images, under nice lightening conditions [^12], [^13]. A common thread in the approaches adopted by Gupta et al. [^14] and Hermans et al. [^15] is the use of hand engineered features for classification of either RGB or RGBD images. The first cited authors above use boundary detection and hierarchical grouping before performing category segmentation. The second work uses Random Forest as classifier and it focuses on real-time joint reconstruction and semantic segmentation. As previously stated, the success of CNNs for object classification has led researchers to exploit their feature learning capabilities for segmentation. Three subcategories were developed: patch-wise training [^16], [^17], end-to-end learning of pixel-wise classification [^18], [^19], [^20] and multiscale semantic segmentation [^21], [^19], [^22], [^23]. In [^16], patch-wise classification is used. This paper outlines a method for the detection and tracking of people in depth images, acquired with a low- --- resolution Time-of-Flight (ToF) camera. This depth sensor is placed perpendicular to the ground in order to provide distance information from a top-view position. The paper introduces an approach to track and detect people in the presence of persistent and temporarily heavy occlusion with CNNs for semantic segmentation using top-view depth data in crowded environment. While many studies have performed semantic segmentation for RGB-D images, most existing methods focus on either only medical images or only object recognition in indoor environment. To the best of our knowledge, this is the first exploitation of data acquired from an RGB-D camera in top-view configuration for people detection and tracking in a crowded environment. Our work differs from the references above in both our approach to segmentation and recognition. The approach has been applied to a newly collected dataset (TVHeads) of depth images of people recorded from an RGB-D camera installed in top-view configuration. The goal is to design a novel U-Net architecture, the so-called U-Net3, that has been modified compared with the previous ones, at the end of each layer. In particular, a batch normalization is added after the first ReLU activation function and after each max-pooling and up-sampling functions. The approach presented in this paper does not require persons to be isolated, i.e. not occluded, when they first enter the scene. In order to identify a person, we require that at least the head region must be visible. We assume an RGB-D camera in top-view configuration and people passing under this camera walk on a known ground plane with no significant false alarms due to shadows, reflections, or other reasons. To briefly summarize, the main contributions of this work are: (i) a demonstration that semantic segmentation can be applied in real-time for people detection and tracking in crowded environment; (ii) a novel U-Net3 design for semantic segmentation; (iii) a challenging new dataset of depth images in a heavy crowded environment (more than 3 people per square meter) collected by an RGB-D camera in top-view configuration, hand-labelled with ground truth semantic segmentations. The application of our approach to TVHeads dataset yields good results in terms of precision, recall and F1-score and demonstrates the effectiveness of the proposed approach. The paper is organized as follows: Section II introduces our approach consisting of a modified U-Net (Subsection II-A1), a state-of-art CNNs comparison (Subsections II-A2, II-A3, II-A4) and gives details on the “TV-Heads Dataset” (Subsection II-B). Final sections present results (Section III) and conclusions (Section IV) with future works. ### II. METHODS In this section, we introduce a novel modified U-Net architecture, the so called U-Net3, for heads images segmentation. We also present the Convolutional Networks for image Segmentation taken from the literature as well as the dataset used for the comparison. Further details are given in the following subsections. The architecture proposed is comprehensively evaluated on the TVHeads dataset. Details on the data collection and ground truth labelling are discussed in Subsection II-B. ### A. Convolutional Networks for image Segmentation 1) U-Net3: The U-Net3 $^{1}$ architecture is illustrated in Figure 1. We build upon a more elegant architecture, modifying and extending the ones proposed in [^23], [^24] such that it works with very few training images and yields more precise segmentations. It is composed of two main parts: a contracting path (left side) and an expansive path (right side). The first path follows the typical architecture of a convolutional network. It consists of the repeated application of two $3 \times 3$ convolutions (unpadded convolutions), each followed by a ReLU and a $2 \times 2$ max pooling operation with stride 2 for downsampling. At each downsampling step, the number of feature channels is doubled. Every step in the expansive path consists of an upsampling of the feature map followed by a $2 \times 2$ convolution (“up-convolution”). It halves the number of feature channels, a concatenation with the correspondingly cropped feature map from the contracting path, and two $3 \times 3$ convolutions, each followed by a ReLU. At the final layer a $1 \times 1$ convolution is used to map each 32-component feature vector to the desired number of classes. Compared with [^23], [^24], U-Net3 architecture proposed in this work has been modified at the end of each layer. The main idea is to add a batch normalization after the first ReLU activation function and after each max-pooling and upsampling functions. ![Image](images/e317b6dd5050c6b54917ce2dcadd965fcbf572c7f0065a53fa67772027986eab.jpg)
Fig. 1: Illustration of implemented U-Net3 architecture.
2) SegNet: SegNet was presented in Vijay et al. in [^20]. The architecture consists of a sequence of non-linear processing layers (encoders) and a corresponding set of decoders followed by a pixelwise classifier. Typically, each encoder consists of one or more convolutional layers with batch normalization and a ReLU non-linearity, followed by non-overlapping max pooling and sub-sampling. The sparse encoding, due to the pooling process, is up-sampled in the decoder using the max pooling indices in the --- encoding sequence. One key ingredient of the SegNet is the use of max-pooling indices in the decoders to perform upsampling of low resolution feature maps. This has the important advantages of retaining high frequency details in the segmented images and also reducing the total number of trainable parameters in the decoders. The entire architecture can be trained end-to-end using stochastic gradient descent. The raw SegNet predictions tend to be smooth even without a CRF based post-processing. 3) ResNet: A ResNet is composed of a sequence of residual units [^25]. The output $x_{n}$ of the n-th RU in a ResNet is computed as: $$ x_{n}=x_{n-1}+F(x_{n-1};W_{n}) $$ where $F(x_{n-1}; W_{n})$ is the residual, which is parametrized by $W_{n}$. In this way, instead of computing the output $x_{n}$ directly, F only computes a residual that is added to the input $x_{n-1}$. This design can be referred as skip connection, since there is a connection from the input $x_{n-1}$ to the output $x_{n}$ that skips the actual computation F. It has been empirically observed that ResNets have superior training properties over traditional feedforward networks. This can be explained by an improved gradient flow within the network. 4) FractalNet: Fractal network is introduced by Larsson et al. in [^26]. Let C denotes the index of a truncated fractal $f_{C}(\cdot)$ (i.e., a few stacked layers) and the base case of a truncated fractal is a single convolution: $$ f_1(z)=conv(z) $$ According to the expansion rule: $$ z^{\prime}=conv(z) $$ $$ f_{C+1}(z)=c o n v(c o n v(z^{\prime})\oplus f_{C}(z^{\prime})) $$ can be defined recursively the successive fractals, where $\oplus$ is a join operation and $\text{conv}(\cdot)$ is a convolution operator. Two blobs are merged by the join operation $\oplus$. As these two blobs contain features from different visual levels, joining them can enhance the discrimination capability of our network. Generally, this operation can be summation, maximization and concatenation. In order to enlarge the receptive field and enclose more contextual information, down-sampling and up-sampling operations are added in the above expansion rule. In particular, a max pooling with a stride of 2 and a deconvolution also with a stride of 2 are added. After the down-sampling operation, the receptive field of a fractal becomes broader. When combining different receptive fields through the join operation, the network can harness multi-scale visual cues and promote itself in discriminating. ### B. TVHeads Dataset The TVHeads $^{2}$ dataset contains depth images of people recorded from an RGB-D camera installed in top-view configuration. The heavy crowded dataset is defined as a scenario where at least 3 people per square meter are passing in the area of the camera, with some cases of physical contacts between people entering the area of interest. In particular, the purpose of this dataset is to localize the heads of people who come across the view area of the camera. A total of 1815 depth images (16 bit) with a dimension of $320 \times 240$ pixels was collected. Additionally, after the image preprocessing phase, the depth images are also converted with an appropriate scaling. In this way it is possible to obtain an image (8 bit) where the heads silhouette is highlighted, by improving image contrast and brightness. The ground truth was manually labelled by 6 human annotators with a majority policy. Figure 2 shows an example of a dataset instance that includes the three images described above, i.e. 16 bit depth image, 8 bit depth image and the corresponding ground truth. ### III. RESULTS AND DISCUSSIONS In this section, the results of the experiments conducted on TVHeads dataset are reported. In addition to the performance of U-Net3, here is also presented the performance of different approaches taken from literature and based on CNNs in order to solve the problem of heads image segmentation. Each CNN implementation is trained with two types of depth images in order to highlight the heads silhouette, improving the images contrast and brightness: 16-bit (original depth images) and 8-bit (scaled images). The general procedure for training neural network based models is to take the dataset and split it into three parts: training, validation, and test. In this case, train, test and validation are chosen respectively to learn model parameters. Once this is complete, the best model is also evaluated over the never before seen test set. Furthermore, different combinations of hyper-parameters are tested, but in this case a learning rate equal to 0.001 and an Adam optimization algorithm have been used. 1) Performance Evaluation: Semantic segmentation performances are divided into two different tables. Table I shows Jaccard [^27] and Dice [^28] indices for training and validation respectively. Table II reports the results in term of accuracy, precision, recall and F1-score. Both tables refer to a learning process conducted during 200 epochs. In Table I, the best CNN architecture is the U-Net3 8-bit version. Indeed, Jaccard index reaches a value equal to 0.962520 regarding train performances. The second best is Fractal Net 16-bit version also obtaining higher values regarding validation performances. For each network is highlighted in red color the best bit version model. As in the previous case, also in the Table II the best CNN architecture, in terms of accuracy, is U-Net3 8-bit version, while Fractal Net 16-bit version exceeds slightly in precision, recall and F1-score metrics. In Figure 3 is shown the trend of Jaccard index for each CNN during the fit procedure. It is easy to see how the FractalNet and ResNet reach high values immediately. --- ![Image](images/6ed026c38b46390dc4c2f1bf5a019f5a7bd0ac00c1fe251dfd5ac8802ded2645.jpg)
(a) 8 bit Depth image.
![Image](images/1b84caf647ebc56883de1028505e61933fc09d60ebd665e890d28b905e4e5a08.jpg)
(b) Ground truth.
Fig. 2: TVHeads Dataset. It consists of 8 bit scaled depth image (Figure 2a) and the corresponding ground truth (Figure 2b).
TABLE I: Jaccard and Dice indices of different CNN architectures.
NetBitJaccard TrainJaccard ValidationDice TrainDice Validation
Fractal [^26]80.9604640.9480000.9798330.973306
160.9616360.9477620.9804430.973180
U-Net [^23]80.8968040.8693990.9455950.930138
160.8944100.8694870.9442620.930188
U-Net2 [^24]80.9238230.9390860.9604030.968586
160.9235370.9382080.9602490.968119
U-Net380.9625200.9313550.9809020.964458
160.9615400.9299240.9803930.963690
SegNet [^20]80.8841820.8237310.9385310.903347
160.8841620.8277450.9385200.905756
ResNet [^25]80.9321600.8563370.9648890.922609
160.9334360.8482400.9655720.917889
TABLE II: Semantic segmentation results of different ConvNet architectures.
NetBitAccuracyPrecisionRecallF1-Score
Fractal [^26]80.9944140.9914000.9931200.992235
160.9944370.9926670.9932970.992970
U-Net [^23]80.9926620.9464750.9504830.948408
160.9925690.9450830.9489570.946938
U-Net2 [^24]80.9931560.9700130.9692060.969568
160.9931650.9678840.9705570.969123
U-Net380.9945720.9904510.9903870.990419
160.9945590.9893820.9894110.989396
SegNet [^20]80.9926830.9463040.9531360.949625
160.9926990.9462370.9533420.949658
ResNet [^25]80.9937890.9683990.9683740.968359
160.9938190.9687650.9692560.968992
after a few epochs. Instead, the U-Net3 increases its value more slowly. The classic U-Net is always below all other networks. 2) Qualitative Evaluation: Table III presents qualitative semantic segmentation results. The table shows the predicted images for each architecture. The best results are obtained by using the U-Net3 (8 and 16 bit), while typical U-Nets and SegNet provide more smooth and rounded images. ![Image](images/3c2e8107523deb97e9c9530b415fabb9912b26b1f5f3a22bcfcd98505381c642.jpg)
Fig. 3: Jaccard index trends.
shapes of the heads. As can be seen, the performance of all networks is good, with F1-Scores ranging that exceed the 90%, thus demonstrating the effectiveness and the suitability of the proposed approach. The comparison shows that U-Net3 performs better than the previous U-Nets design. The system is able to perform at 10 fps on a single core CPU ensuring real-time performances. ### IV. CONCLUSION Humans detection and tracking in a crowded situation, where large number of people are present and exhibit persistent occlusion, represents a challenging but rewarding task. In this paper, U-Net3 for semantic heads segmentation has been introduced, using depth images acquired from an RGB-D camera installed in top-view configuration. We compare the performance of the novel network with different approaches taken from literature based on CNNs in order to solve the problem of heads images segmentation: U-Net, U-Net2, FractalNet, SegNet and ResNet. U-Net3 approach is able to learn a high level representation of image content and to achieve high precision and recall. The experiments on TVHeads dataset yield high accuracies and demonstrate the effectiveness and suitability of our approach. Further investigation will be devoted to improve our approach by employing a larger dataset. Moreover, we will extend the evaluation by --- ![Image](images/8fcc86dbeef759040f3ce46bde1ff33e9fdee6799f232c81ec5f828f7e491cd2.jpg)
(a) Fractal Net 8-bit.
![Image](images/4052fe194c36327f109f236964512f184d404c2bfa4f499421646cf58772c85f.jpg)
(b) Fractal Net 16-bit.
![Image](images/9772b074fbcdf08d1f3092d3334fc7190d71553f63eb7c65670b3eecc61f0278.jpg) ![Image](images/64c35db9980786d9b6038f4f0d205bd9c912992dee86c4ae9a7b7087d016d355.jpg)
(c) U-Net 8-bit.
(d) U-Net 16-bit.
![Image](images/81fd5fce22728a8f9a9f8ae78f1d08687ede6cf7cb0de3c13f333160a9697b31.jpg)
(e) U-Net2 8-bit.
![Image](images/d92d7f6bf1c29c445c00a7e80e7adaaaa6991bdc41934e1dae4ca05bea510793.jpg)
(f) U-Net2 16-bit.
![Image](images/c8364da2077c1ad2f814eefc5e9cabc117e28acee3fec6d0318f551cf8b10f27.jpg)
(g) U-Net3 8-bit.
![Image](images/2f1e107ead2bd2dd2c1351058c5ae3f12044b65d748fc42cfd85ddf093f09c4b.jpg)
(j) SegNet 16-bit.
(h) U-Net3 16-bit.
![Image](images/3f33e34209787cc127afbf3e0b6633d3f06cd750668515946c40b5728466bb82.jpg) ![Image](images/6c27456cff5347271ac649685bc272f6eae40f207b0c32d5d02f9b21781e264a.jpg)
(i) SegNet 8-bit.
![Image](images/c7e17fe6435c3204aab4ec8dd84323e381d7b925a2c49e2bcc59db1a24c3947e.jpg)
(k) ResNet 8-bit.
Fig. 4: Jaccard index results.
![Image](images/b14a8c6b7e8581d5e5e756919d6ee2ac5c190a9d712ca3fce9be27ee36665306.jpg)
(1) ResNet 16-bit.
comparing our U-Net3 with other CNNs existing in the literature. A more complex dataset with other crowded scene will be also collected to propose a more challenging test environment. ## REFERENCES [^1]: T. Zhao, R. Nevatia, and B. Wu, “Segmentation and tracking of multiple humans in crowded environments,” IEEE transactions on pattern analysis and machine intelligence, vol. 30, no. 7, pp. 1198–1211, 2008. [^2]: J. Liu, Y. Liu, G. Zhang, P. Zhu, and Y. Q. Chen, "Detecting and tracking people in real time with rgb-d camera," Pattern Recognition Letters, vol. 53, pp. 16–23, 2015. [^3]: W. Ye and Z. Zhong, “Robust people counting in crowded environment,” in Robotics and Biomimetics, 2007. ROBIO 2007. IEEE International Conference on. IEEE, 2007, pp. 1133–1137. [^4]: M. Sturari, D. Licotti, R. Pierdicca, E. Frontoni, A. Mancini, --- Label ![Image](images/9ce93e0ec7bd0b840b1faf46122f58f8c80d54d81fa83264ecef52e431612d4c.jpg) ResNet [^25] M. Contigiani, and P. Zingaretti, "Robust and affordable retail customer profiling by vision and radio beacon sensor fusion," Pattern Recognition Letters, 2016. [^5]: D. Licotti, M. Paolanti, E. Frontoni, and P. Zingaretti, “People detection and tracking from an rgb-d camera in top-view configuration: Review of challenges and applications,” in International Conference on Image Analysis and Processing. Springer, 2017, pp. 207–218. [^6]: M. Paolanti, C. Kaiser, R. Schallner, E. Frontoni, and P. Zingaretti, “Visual and textual sentiment analysis of brand-related social media pictures using deep convolutional neural networks,” in International Conference on Image Analysis and Processing. Springer, 2017, pp. 402–413. [^7]: M. Sturari, M. Paolanti, E. Frontoni, A. Mancini, and P. Zingaretti, “Robotic platform for deep change detection for rail safety and security,” in Mobile Robots (ECMR), 2017 European Conference on. IEEE, 2017, pp. 1–6. [^8]: B. Hariharan, P. Arbeláez, R. Girshick, and J. Malik, “Simultaneous detection and segmentation,” in European Conference on Computer Vision. Springer, 2014, pp. 297–312. toni, “Mobile robot for retail surveying and inventory using visual and textual analysis of monocular pictures based on deep learning,” in Mobile Robots (ECMR), 2017 European Conference on. IEEE, 2017, pp. 1–6. [^9]: M. Paolanti, M. Sturari, A. Mancini, P. Zingaretti, and E. Fron- [^10]: N. Silberman and R. Fergus, “Indoor scene segmentation using a structured light sensor,” in Computer Vision Workshops (ICCV Workshops), 2011 IEEE International Conference on. IEEE, 2011, pp. 601–608. [^11]: N. Silberman, D. Hoiem, P. Kohli, and R. Fergus, “Indoor segmentation and support inference from rgbd images,” Computer Vision–ECCV 2012, pp. 746–760, 2012. [^12]: S. Gould, R. Fulton, and D. Koller, “Decomposing a scene into geometric and semantically consistent regions,” in Computer Vision, 2009 IEEE 12th International Conference on. IEEE, 2009, pp. 1–8. [^13]: C. Liu, J. Yuen, and A. Torralba, “Sift flow: Dense correspondence across scenes and its applications,” IEEE transactions on pattern analysis and machine intelligence, vol. 33, no. 5, pp. 978–994, 2011. [^14]: S. Gupta, P. Arbelaez, and J. Malik, “Perceptual organization and recognition of indoor scenes from rgb-d images,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2013, pp. 564–571. [^15]: A. Hermans, G. Floros, and B. Leibe, “Dense 3d semantic mapping of indoor scenes from rgb-d images,” in Robotics and Automation (ICRA), 2014 IEEE International Conference on. IEEE, 2014, pp. 2631–2638. [^16]: C. Farabet, C. Couprie, L. Najman, and Y. LeCun, “Learning hierarchical features for scene labeling,” IEEE transactions on pattern analysis and machine intelligence, vol. 35, no. 8, pp. 1915–1929, 2013. [^17]: D. Grangier, L. Bottou, and R. Collobert, “Deep convolutional networks for scene parsing,” in ICML 2009 Deep Learning Workshop, vol. 3, 2009. [^18]: J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks for semantic segmentation,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp. 3431–3440. [^19]: H. Noh, S. Hong, and B. Han, "Learning deconvolution network for semantic segmentation," in Proceedings of the IEEE International Conference on Computer Vision, 2015, pp. 1520–1528. [^20]: V. Badrinarayanan, A. Kendall, and R. Cipolla, “Segnet: A deep convolutional encoder-decoder architecture for image segmentation,” CoRR, 2015. [^21]: F. Yu and V. Koltun, “Multi-scale context aggregation by dilated convolutions,” arXiv preprint arXiv:1511.07122, 2015. [^22]: L.-C. Chen, Y. Yang, J. Wang, W. Xu, and A. L. Yuille, “Attention to scale: Scale-aware semantic image segmentation,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 3640–3649. [^23]: O. Ronneberger, P. Fischer, and T. Brox, "U-net: Convolutional networks for biomedical image segmentation," arXiv preprint arXiv:1505.04597, 2015. [^24]: H. Ravishankar, R. Venkataramani, S. Thiruvenkadam, P. Sudhakar, and V. Vaidya, “Learning and incorporating shape models for semantic segmentation,” in International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 2017, pp. 203–211. [^25]: K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778. [^26]: G. Larsson, M. Maire, and G. Shakhnarovich, “Fractalnet: Ultra-deep neural networks without residuals,” arXiv preprint arXiv:1605.07648, 2016. [^27]: P. Jaccard, “Etude comparative de la distribution florale dans une portion des alpes et des jura,” Bull Soc Vaudoise Sci Nat, vol. 37, pp. 547–579, 1901. [^28]: L. R. Dice, “Measures of the amount of ecologic association between species,” Ecology, vol. 26, no. 3, pp. 297–302, 1945.