# Fourier Features For Person Detection in Depth Data

Viktor Seib $^{\circledR}$, Guido Schmidt, Michael Kusenbach, and Dietrich Paulus

Active Vision Group (AGAS), University of Koblenz-Landau, Universitätsstr. 1, 56070 Koblenz, Germany

{vseib,guidoschmidt,mkusenbach,paulus}@uni-koblenz.de <http://agas.uni-koblenz.de>

Abstract. A robust and reliable person detection is crucial for many applications. In the domain of service robots that we focus on, knowing the location of a person is an essential requirement for any meaningful human-robot interaction. In this work we present a people detection algorithm exploiting RGB-D data from Kinect-like cameras. Two features are obtained from the data representing the geometrical properties of a person. These features are transformed into the frequency domain using Discrete Fourier Transform (DFT) and used to train a Support Vector Machine (SVM) for classification. Additionally, we present a hand detection algorithm based on the extracted silhouette of a person. We evaluate the proposed method on real world data from the Cornell Activity Dataset and on a dataset created in our laboratory.

Keywords: People detection · Silhouette detection · Hand detection · Fourier features · Service robots

## 1 Introduction

The ability of service robots to properly react to the commands given by a user highly depends on the robot's capability to reliable detect the position of the interacting person. Apart from the position of the person itself, also the position of its hands is of large interest for a natural interaction. This comes from the customary practice of people to use gestures for interaction. Important gestures that a service robot needs to be aware of in its daily routine is pointing to a position or object of interest, raising a hand to call for attention or waving the hand to call the robot.

With the availability of affordable RGB-D cameras the problem of person detection can be addressed by the combination of RGB camera data, as well as geometrical information based on depth data (figure 1). We propose an approach that exploits these information by calculating 2 novel features. In the first step, a model-based search is performed on the input data to find possible person candidates. For each candidate, 2 features are computed: the Frontal Feature and the Width Feature. To obtain a low dimensional feature vector, the computed features are transformed to the frequency domain using Discrete Fourier Transform.

---

![Image](images/b58317529a6a43abceff32b16da703101f92e848879d5eec3af9e5ce3031766d.jpg)

<div style="text-align: center;">(a)</div>

![Image](images/81a99cdc6e380c45a973e90e0b55dbf028501f90c4ca65c54eac38a742d10203.jpg)

<div style="text-align: center;">(b)</div>

<div style="text-align: center;">Fig. 1. Example RGB image (a) and the corresponding depth image (b) taken from the Cornell Activity Dataset [^14].</div>

(DFT) and a Support Vector Machine (SVM) is trained on the obtained Fourier coefficients. The presented approach enables us to detect standing and sitting persons in different poses and orientations towards the camera. Additionally, for frontal facing persons hands are detected using skin color extraction from the face and distances between possible hand candidates on the previously detected person's silhouette. The presented approach is successfully applied on the service robot Lisa in our lab.

In the following section 2 we briefly review related work on people detection and especially people detection on depth data. The section 3 and section 4 describe our proposed approach in detail and introduce the proposed features. Finally, section 5 presents evaluation results obtained from the Cornell Activity Dataset [^14] and a dataset acquired in our lab and discusses the obtained results. This paper is concluded by section 6 where a summary and an outlook to future work is presented.

## 2 Related Work

In the past years, several approaches for person detection and tracking using depth data were proposed. However, the actual devices that data stems from are changing. A few years ago, laser sensors or costly stereo camera systems were used. For instance, Bertozzi et al. [^1] use a combination of an RGB stereo setup combined with infrared stereo cameras. They extract histograms of oriented gradients (HOG) for classification. The HOG feature is not only used for RGB data, as for example by Dalal and Triggs [^5]. It was also adapted to depth data, resulting in the histograms of oriented depths (HOD) detector introduced by Spinello and Arras [^13]. Also combinations of HOG and HOD are used as for instance demonstrated in [^7].

Machine learning approaches are also exploited for shape matching e.g. the approach described by Xia et al. [^11]. Xia et al. use chamfer distance matching to find candidate regions for people locations and examine these with a three

---

![Image](images/ae3d4826cad72e10ef892d0753345210355cf7a6db923e32f3e9a41890ac2639.jpg)

<div style="text-align: center;">Fig. 2. The resulting depth histogram and the corresponding extracted depth slices are represented as binary images.</div>

dimensional head model. Other approaches use distance transformations and extract a joint representation of detected people, as Lee et al. show in [^10].

Paisitkriangkrai et al. [^12] introduce spatial pooling on a set of covariance descriptors for feature extraction. Choi et al. [^4] introduce a Monte Carlo based Particle Filter. The applied descriptor is based on HOG and on a shape vector is derived from the upper body part of a person candidate. This approach is capable of handling dynamic scenes.

Similar to the approach of Choi et al., our algorithm is also based on shape extraction of the shoulder region and the head. However, we use 2 features specifically designed to detect silhouettes from depth data. As proposed by Hordern and Kirchner [^8], we also transform the features with DFT before using them for SVM training and classification.

## 3 Person Detection

In this section we describe our algorithm for silhouette detection. In the following, we employ a right-handed coordinate system, where the x-axis points forward, the y-axis to the left and the z-axis to the top.

### 3.1 Candidate Detection

To find candidates for person detection, we first compute a point histogram along the depth axis (x-axis) and then perform a model based search. The input point cloud is divided into equally thick slices along the depth axis. In our experiments we use a thickness of $a = 0.1m$ for each slice. Each slice corresponds to a histogram bin, holding the number of points in that slice. The idea behind this histogram is that potential objects form local point clusters in the scene. To find these objects local minima inside the histogram are found. The objects are expected to be located between these extracted local minima.

In the following step, the point cloud is resliced at locations corresponding to minima in the histogram as shown in figure 2. For every depth slice, a binary image I is generated by only considering points located inside the corresponding

---

![Image](images/48b5dac7bda83cba7c5a12b3fd60997aa865d7a5b236c0ccbec6a0c0f7d71736.jpg)

<div style="text-align: center;">Fig. 3. The RGB data of a person's silhouette (a) and the corresponding binary image (b and c) is shown. The head model assumptions are illustrated in (b), whereas the occupancy of test regions is shown in (c).</div>

slice. A model-based search is performed on the set of binary images $I_{0}\ldots I_{n}$ to find person candidates.

First, local maxima $I_i(v_h, w_h)$ on the vertical axis in each binary images $I_i$ are extracted. These are possible head positions, which need to be validated against our model assumptions. Since the binary images were created from point clouds, a 3D point is known for every valid (i.e. “white”) pixel $I_i(v, w)$. This permits us to define model parameters in metric space rather than in pixels. In the following, the function $\mathcal{P}(I(v, w))$ is used to obtain the corresponding 3D point $p = (x, y, z)^T$ from a pixel $I(v, w)$ in the binary image. Further, the inverse function $\mathcal{Q}(p)$ is used to obtain the pixel in the binary image corresponding to a 3D point p in the organized point cloud. The actual mapping of a color pixel to a depth pixel (and vice-versa) depends on the dataset (see section 5).

Our model assumption consists of a typical head width $\theta_{w}$ and height $\theta_{h}$ (given in a range of minimal and maximal valid sizes) and includes the approximated calculation of the possible head dimensions, as shown in figure 3. For a local maximum $I_{i}(v_{h}, w_{h})$ to be selected as candidate, the following propositions must hold. First, there has to be a valid pixel $I(v_{m}, w_{m})$ (middle point), defined as

 $$ I(v_{m},w_{m})=\mathcal{Q}(p_{v_{m},w_{m}})=\mathcal{Q}(p_{v_{h},w_{h}}-(0,0,\frac{\theta_{h}}{2})^{T}). $$

in the binary image, corresponding to a point $\frac{\theta_{h}}{2}$ below the maximum. Equation 1 requires that at least the upper half of the head must be visible.

Two other points, the left and the right border point of the head, $I(v_l, w_l)$ and $I(v_r, w_r)$, have to be set in the binary image at the same height as the middle point $I(v_m, w_m)$, fulfilling the constraint

 $$ \theta_{h,m i n}<||\mathcal{P}(I(v_{l},w_{l}))-\mathcal{P}(I(v_{r},w_{r}))||<\theta_{h,m a x}. $$

The found value  $\theta_w = ||\mathcal{P}(I(v_l, w_l)) - \mathcal{P}(I(v_r, w_r))||$ is the hypothetical head width. As a result of processing every binary image according to this model-based search, a set of object hypotheses is obtained possessing human head-like

---

![Image](images/69a1127acf58ea9cf50d3afd255d90f30e004c572baea852903bf1594de15864.jpg)

<div style="text-align: center;">(a)</div>

![Image](images/c4a10e4b0575d7c9e830ac27dedaa804dbb49285bda602d192217603c57e19cc.jpg)

<div style="text-align: center;">(b)</div>

![Image](images/2340f19f7765b41aefda85cc0d369e8c7bbd91dcea0c772ce20d0ce453cbe759.jpg)

<div style="text-align: center;">(c)</div>

<div style="text-align: center;">Fig. 4. The image section R used to extract the frontal feature is shown in (a). The contained part of the silhouette is shown in (b). Image (c) shows the frontal silhouette from the side. The horizontal line in (c) is the mean distance value in region R.</div>

![Image](images/948ec93b854503cb9f088b048dcdcf47e794acfc8e9f62567fd560f35a1afcab.jpg)

<div style="text-align: center;">Fig. 5. The extracted values  $d_w$ and  $r_w$ for the frontal features are shown. This figure corresponds to figure 4(c) rotated by  $90^\circ$ to the left. The axes are labeled according to the same coordinate system used for feature extraction. The resulting curve is the input data for the DFT.</div>

characteristics at their highest position. Note that this model-based approximation of a head is computationally faster than fitting a circle around local maxima of the binary images.

This set of hypotheses is further refined by calculating the percentage of filled pixels in the binary image in four test regions around the detected hypothetical head (figure 3). The first region represents the area above the highest point of the head with a height of $\frac{\theta_h}{2}$ and the width $\theta_w$. Further, 2 equal regions, one located on the left the other located to the right of the head, are examined. The fourth and last region is located below the head and extends to the bottom of the corresponding binary image. For every region, the percentage of filled pixels in the binary image is calculated. Ideally, the areas above, left and right of the head should be empty. On the other hand, the area below the head should be completely filled, representing a hypothetical torso and legs of a person. Because of possible occlusions by objects in front of the torso, pixels from depth slices in front of the detected candidate are also considered when determining the ratio of filled pixels in the bottom area. After applying these model-based rules, the

---

search space for possible persons is reduced from the whole input point cloud to a set of person candidates.

### 3.2 Features

For further classification, we propose to extract 2 features for each candidate: the Frontal Feature and the Width Feature. Both features are well suited to differentiate between persons and objects and are described in the following.

Frontal Feature. The idea of our Frontal Feature is to encode the frontal depth contour of the face and upper body of a person. This feature is computed using the previously extracted binary image of the head and the torso and the corresponding point cloud. Therefore, the position, width and height of the head are known. We extract an image section R which is horizontally centered at the head. Vertically, R starts at $I_i(v_h, w_h)$ being the highest point, with the parameter b determining the sections height (figure 4). We obtained best results with $b = 0.5m$ in our experiments. The resulting region covers the candidate's head and upper body. For each pixel row w inside the binary image section R, the mean depth value $r_w$ of all points in this row is computed as

 $$ r_{w}=\frac{\sum_{v=0}^{V}\mathcal{P}_{x}(R(v,w))}{V} $$

where V is the width of R in pixels, $R(v, w)$ is a pixel in R and $\mathcal{P}_x(R(v, w))$ is a function that returns the x-value (i.e. the value on the forward axis) of the point at pixel $R(v, w)$. This provides some stability against rotations and side movements of the head, as well as sensor noise. Further, we calculate the distance $d_w$ of each row w to the highest point $I(v_h, w_h)$ as

 $$ d_{w}=\frac{\sum_{v=0}^{V}\mathcal{P}_{z}(R(v,w))}{V}-\mathcal{P}_{z}(I(v_{h},w_{h})) $$

where $\mathcal{P}_z$ is defined similar to $\mathcal{P}_x$, but returning the z-value (i.e. the vertical coordinate). Now we have obtained a mean depth value $r_w$ and a vertical distance $d_w$ to the top point for each row. The resulting frontal contour of a person is visualized in figure 4 for a person facing the camera. The computed values $r_w$ and $d_w$ from R are shown in figure 5. Depending on the person's orientation towards the camera, this feature provides different profiles.

Width Feature. This feature benefits from the fact that persons facing the sensor will have a broader shoulder section compared to their head width. Again we use b = 0.5m for the height of the section of interest R, its width is set to the width of the binary image. We now extract the leftmost $\mathcal{P}(R(v_l, w_l))$ and the rightmost $\mathcal{P}(R(v_r, w_r))$ valid point in each row of the binary image, obtaining the row width $b_w$ from

 $$ b_{w}=||\mathcal{P}_{y}(I(v_{l},w_{l}))-\mathcal{P}_{y}(I(v_{r},w_{r}))||, $$

---

where $\mathcal{P}_{y}$ returns the value on the horizontal axis. Further, we again use the vertical distance $d_{w}$ of each row to the highest point of the head $I(v_{h}, w_{h})$.

### 3.3 Fourier Feature Vectors

Both features can be seen as functions of the distance $d_{w}$ from the top point of the candidate head. In case of the Frontal Feature function values are the average depth values $r_{w}$ and in case of the Width Feature the width $b_{w}$ of the candidate silhouette. These two features are computed for each candidate and transformed into the frequency domain using the Fourier transform. Since we have discrete functions (discretized by the lines of the binary images), we use the Discrete Fourier transform (DFT). The actual feature vector is composed of the resulting Fourier coefficients. We can adapt the length of the feature vector by omitting high frequency Fourier coefficients. While the computed features originally had a length of 50, after the Fourier transform we retain only the first 6 coefficients. Figure 7 depicts several sinusoids corresponding to the first Fourier coefficient. Unlike in the work of Hordern et al. [^8], where the feature vector is divided by the constant component and therefore is scale invariant, we

![Image](images/b2402da7c23b382bd1b66fff08ac71963a2b84eeebd714ba9da8f6a2a8e73b32.jpg)

<div style="text-align: center;">Fig. 6. Omitting the constant component of the Fourier Transform results in a distance independent feature vector sinusoid (shown in red), in contrast to the distance dependent sinusoid with the constant component (shown in green).</div>

![Image](images/f139bf7081e2070a3a2389db00b48790f3ab1213aa1281ce4db9e45ef43e582c.jpg)

<div style="text-align: center;">(a)</div>

![Image](images/8825d0713b03ae7cd5f84458b4e315469dd7b3008d3bd52b282bb254ad7db383.jpg)

<div style="text-align: center;">(b)</div>

<div style="text-align: center;">Fig. 7. The sinusoids corresponding to the first Fourier coefficient of several human silhouettes are depicted in (a). The outliers stem from partially visible silhouettes at image borders. Image (b) shows the first sinusoids of different objects.</div>

---

omit the constant component from the feature vector without division. In our case this leads to the feature vector being invariant to the distance of the person to the sensor (figure 6). Our features are based on metric data and scale is an important factor in the process of people detection. We therefore do not want to have a scale invariant feature in order not to detect e.g. toys like dolls that have a similar silhouette, but different size as a human. The resulting feature vector is used to train a support vector machine (SVM) with a radial basis function kernel (RBF).

## 4 Hand Detection

Since we are interested in interacting with people, we focus on detecting hands of upraised and outstretched arms. These are signs of calling for attentions and thus should be recognized by a service robot.

![Image](images/2fe4f66f90d37fd9b667b1f120a970b5b4c41fa1860298efbc6d86e0118f5f26.jpg)

<div style="text-align: center;">(a)</div>

![Image](images/b91b5eb56f7c0ea82222cf004f023e83da131431bc0a9d63eefc73c92654b83c.jpg)

<div style="text-align: center;">(b)</div>

![Image](images/24a80231dc43b514e16f8f5e80959a60fc8a1b6651e88959fec5aa9a95e607c9.jpg)

<div style="text-align: center;">(c)</div>

<div style="text-align: center;">Fig. 8. Illustration of the geometric model to determine the distance between any silhouette point  $\mathcal{P}(M(v,w))$ and the centroid s. Image (a) shows the RGB data of the silhouette, while (b) shows the binary image. In image (c) the distance matrix D is depicted.</div>

Starting from the already known highest pixel $I(v_h, w_h)$ of a detected person, a region growing is performed on the binary image. We use the 4-neighborhood for region growing. For 2 neighboring pixels $I(v_a, w_a)$ and $I(v_b, w_b)$, with $I(v_a, w_a)$ being part of the region, $I(v_b, w_b)$ has to fulfill 2 criteria to be added to the region. First, its corresponding 3D point $\mathcal{P}(I(v_b, w_b))$ must not be farther away from the 3D point of a point in the region $\mathcal{P}(I(v_a, w_a))$ than the threshold value $c_1$:

 $$ |\mathcal{P}(I(v_{a},w_{a}))-\mathcal{P}(I(v_{b},w_{b}))||<c_{1}. $$

Additionally, it has to be within the range $c_2$ of the corresponding 3D point of the highest point of the head, $\mathcal{P}(I(v_h, w_h))$:

 $$ ||\mathcal{P}(I(v_{h},w_{h}))-\mathcal{P}(I(v_{b},w_{b}))||<c_{2}. $$

---

In contrast to the initial binary image I, the resulting binary image M contains only the silhouette of the person (without possible background clutter). In the following, the hand is detected using two cues: by taking into account the distance of the hand from the body and exploiting skin color.

### 4.1 Distance Cue

To exploit the distance cue, the detected body in the silhouette binary image M is reduced to a single vertical line through the centroid of the silhouette. The centroid is calculated from all 3D points of the silhouette using the points p as

 $$ p=\begin{cases}{\mathcal{P}(M(v,w))}&{\mathrm{i f~}M(v,w)=1}\\ {(0,0,0)^{T}}&{\mathrm{e l s e}}\\ \end{cases}. $$

The centroid s is then computed as the mean of all points

 $$ s=\frac{\sum_{v=0}^{V}\sum_{w=0}^{W}\mathcal{P}(M(v,w))}{\sum_{v=0}^{V}\sum_{w=0}^{W}M(v,w)} $$

where V and W represent the image width and image height, respectively. Note that in the denominator of equation 9 the number of the valid pixels in the silhouette binary image M is computed, since all pixels in M are either 1 or 0. We represent the silhouette's centroid line through the point s by a directional vector $\boldsymbol{a} = (0, 0, 1)^T$ parallel to the vertical axis of the coordinate reference frame. Now we can calculate a directional vector $\boldsymbol{b}_{v,w}$ pointing from an arbitrary point $\mathcal{P}(M(v,w))$ in the silhouette image to the centroid s as

 $$ \boldsymbol{b}_{v,w}=s-\mathcal{P}(M(v,w)). $$

As shown in figure 8 (b), the distance d of a given point $M(v, w)$ can then be calculated by the cross product

 $$ d_{vw}=\left\|a\times b_{v,w}\right\| $$

and a distance matrix D created according to

 $$ D(v,w)=\begin{cases}{d_{v w}}&{\mathrm{i f}\mathcal{P}(M(v,w))=1}\\ {0}&{\mathrm{e l s e}}\\ \end{cases}. $$

### 4.2 Skin Color

We use the skin color as the second cue for hand detection. The detected person's face color is used as a reference color. Please note that at this point the person's silhouette is already segmented. The skin color detection is thus not influenced by clutter in the environment.

To extract a representing skin color, we convert the RGB-data of the face region to HSV and then create a color histogram. The HSV color space has

---

<div style="text-align: center;">Table 1. Dataset samples from the Cornell Activity Dataset [^14] used to evaluate the people detection algorithm</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td colspan="4">Person 1</td><td colspan="4">Person 2</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>subset</td><td style='text-align: center; word-wrap: break-word;'>action</td><td style='text-align: center; word-wrap: break-word;'>pose</td><td style='text-align: center; word-wrap: break-word;'># images</td><td style='text-align: center; word-wrap: break-word;'>subset</td><td style='text-align: center; word-wrap: break-word;'>action</td><td style='text-align: center; word-wrap: break-word;'>pose</td><td style='text-align: center; word-wrap: break-word;'># images</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>(A)</td><td style='text-align: center; word-wrap: break-word;'>brushing teeth</td><td style='text-align: center; word-wrap: break-word;'>standing</td><td style='text-align: center; word-wrap: break-word;'>1351</td><td style='text-align: center; word-wrap: break-word;'>(A)</td><td style='text-align: center; word-wrap: break-word;'>still</td><td style='text-align: center; word-wrap: break-word;'>standing</td><td style='text-align: center; word-wrap: break-word;'>482</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>(B)</td><td style='text-align: center; word-wrap: break-word;'>drinking water</td><td style='text-align: center; word-wrap: break-word;'>standing</td><td style='text-align: center; word-wrap: break-word;'>1746</td><td style='text-align: center; word-wrap: break-word;'>(C)</td><td style='text-align: center; word-wrap: break-word;'>drinking water</td><td style='text-align: center; word-wrap: break-word;'>standing</td><td style='text-align: center; word-wrap: break-word;'>508</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>(C)</td><td style='text-align: center; word-wrap: break-word;'>wearing contact lenses</td><td style='text-align: center; word-wrap: break-word;'>standing</td><td style='text-align: center; word-wrap: break-word;'>440</td><td style='text-align: center; word-wrap: break-word;'>(F)</td><td style='text-align: center; word-wrap: break-word;'>opening pill container</td><td style='text-align: center; word-wrap: break-word;'>standing</td><td style='text-align: center; word-wrap: break-word;'>210</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>(E)</td><td style='text-align: center; word-wrap: break-word;'>working on laptop</td><td style='text-align: center; word-wrap: break-word;'>sitting</td><td style='text-align: center; word-wrap: break-word;'>1265</td><td style='text-align: center; word-wrap: break-word;'>(I)</td><td style='text-align: center; word-wrap: break-word;'>cooking (stirring)</td><td style='text-align: center; word-wrap: break-word;'>standing</td><td style='text-align: center; word-wrap: break-word;'>442</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>(H)</td><td style='text-align: center; word-wrap: break-word;'>talking on the phone</td><td style='text-align: center; word-wrap: break-word;'>standing</td><td style='text-align: center; word-wrap: break-word;'>1442</td><td style='text-align: center; word-wrap: break-word;'>(L)</td><td style='text-align: center; word-wrap: break-word;'>writing on white board</td><td style='text-align: center; word-wrap: break-word;'>standing</td><td style='text-align: center; word-wrap: break-word;'>609</td></tr><tr><td colspan="4">Person 3</td><td colspan="4">Person 4</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>subset</td><td style='text-align: center; word-wrap: break-word;'>action</td><td style='text-align: center; word-wrap: break-word;'>pose</td><td style='text-align: center; word-wrap: break-word;'># images</td><td style='text-align: center; word-wrap: break-word;'>subset</td><td style='text-align: center; word-wrap: break-word;'>action</td><td style='text-align: center; word-wrap: break-word;'>pose</td><td style='text-align: center; word-wrap: break-word;'># images</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>(A)</td><td style='text-align: center; word-wrap: break-word;'>still</td><td style='text-align: center; word-wrap: break-word;'>standing</td><td style='text-align: center; word-wrap: break-word;'>374</td><td style='text-align: center; word-wrap: break-word;'>(A)</td><td style='text-align: center; word-wrap: break-word;'>still</td><td style='text-align: center; word-wrap: break-word;'>standing</td><td style='text-align: center; word-wrap: break-word;'>276</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>(D)</td><td style='text-align: center; word-wrap: break-word;'>still</td><td style='text-align: center; word-wrap: break-word;'>sitting</td><td style='text-align: center; word-wrap: break-word;'>474</td><td style='text-align: center; word-wrap: break-word;'>(F)</td><td style='text-align: center; word-wrap: break-word;'>opening pill container</td><td style='text-align: center; word-wrap: break-word;'>standing</td><td style='text-align: center; word-wrap: break-word;'>215</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>(H)</td><td style='text-align: center; word-wrap: break-word;'>opening pill container</td><td style='text-align: center; word-wrap: break-word;'>standing</td><td style='text-align: center; word-wrap: break-word;'>245</td><td style='text-align: center; word-wrap: break-word;'>(G)</td><td style='text-align: center; word-wrap: break-word;'>opening pill container</td><td style='text-align: center; word-wrap: break-word;'>standing</td><td style='text-align: center; word-wrap: break-word;'>346</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>(M)</td><td style='text-align: center; word-wrap: break-word;'>walk around</td><td style='text-align: center; word-wrap: break-word;'>standing</td><td style='text-align: center; word-wrap: break-word;'>511</td><td style='text-align: center; word-wrap: break-word;'>(J)</td><td style='text-align: center; word-wrap: break-word;'>cooking (choping)</td><td style='text-align: center; word-wrap: break-word;'>standing</td><td style='text-align: center; word-wrap: break-word;'>390</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>(Q)</td><td style='text-align: center; word-wrap: break-word;'>wearing contact lenses</td><td style='text-align: center; word-wrap: break-word;'>standing</td><td style='text-align: center; word-wrap: break-word;'>447</td><td style='text-align: center; word-wrap: break-word;'>(K)</td><td style='text-align: center; word-wrap: break-word;'>working on computer</td><td style='text-align: center; word-wrap: break-word;'>sitting</td><td style='text-align: center; word-wrap: break-word;'>411</td></tr></table>

been proved to be a reliable color space for skin and hand detection [^9], [^3], [^6]. However, the exact choice of the color space is not critical, since we are not actually detecting skin color, but rather comparing the face color with possible hand locations on the segmented silhouette.

The generated histogram will show a clustering of the skin color. To clean the histogram from colors originating from the hair, eyebrows, eyes or lips, only the high-valued classes of the histogram are used. Starting with the highest valued class, further high-valued classes are added, until the number of pixels in the valid classes exceeds a predefined threshold. During detection, each pixel with a color inside the valid histogram classes will be regarded as skin color. The resulting colors and the corresponding pixels are stored in a separate binary image S. Both cues are combined to a hand confidence map M as shown in the following Equation:

 $$ M_{h a n d}(v,w)=\alpha\cdot D(v,w)+\beta\cdot\frac{S(v,w)}{m a x(S)} $$

with the 2 weights $\alpha$ and $\beta$ and the normalization value $max(S)$ with the maximum histogram value. Note that no separate normalization for the distance value is needed, since it is already in metric space. For a validation of possible hands the maximum inside $M_{hand}$ is computed. Regions in the neighborhood of the first maximum, as well as the maximum itself are excluded and the remaining values are used to compute a second maximum, resulting in both hands of a person.

## 5 Evaluation and Results

The people detection algorithm was evaluated with sample sets from the Cornell Activity Dataset [^14]. The dataset was recorded with the Microsoft Kinect. The provided RGB and depth images are aligned, so no manual alignment is

---

<div style="text-align: center;">Table 2. Person detection results with different features applied</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td colspan="5">Frontal Feature</td></tr><tr><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'>Person 1</td><td style='text-align: center; word-wrap: break-word;'>Person 2</td><td style='text-align: center; word-wrap: break-word;'>Person 3</td><td style='text-align: center; word-wrap: break-word;'>Person 4</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Precision</td><td style='text-align: center; word-wrap: break-word;'>0.679</td><td style='text-align: center; word-wrap: break-word;'>0.958</td><td style='text-align: center; word-wrap: break-word;'>0.606</td><td style='text-align: center; word-wrap: break-word;'>0.887</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Recall</td><td style='text-align: center; word-wrap: break-word;'>0.999</td><td style='text-align: center; word-wrap: break-word;'>0.904</td><td style='text-align: center; word-wrap: break-word;'>0.940</td><td style='text-align: center; word-wrap: break-word;'>0.960</td></tr><tr><td colspan="5">Width Feature</td></tr><tr><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'>Person 1</td><td style='text-align: center; word-wrap: break-word;'>Person 2</td><td style='text-align: center; word-wrap: break-word;'>Person 3</td><td style='text-align: center; word-wrap: break-word;'>Person 4</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Precision</td><td style='text-align: center; word-wrap: break-word;'>0.684</td><td style='text-align: center; word-wrap: break-word;'>0.887</td><td style='text-align: center; word-wrap: break-word;'>0.468</td><td style='text-align: center; word-wrap: break-word;'>0.973</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Recall</td><td style='text-align: center; word-wrap: break-word;'>0.976</td><td style='text-align: center; word-wrap: break-word;'>0.904</td><td style='text-align: center; word-wrap: break-word;'>0.996</td><td style='text-align: center; word-wrap: break-word;'>0.955</td></tr><tr><td colspan="5">Frontal &amp; Width Feature</td></tr><tr><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'>Person 1</td><td style='text-align: center; word-wrap: break-word;'>Person 2</td><td style='text-align: center; word-wrap: break-word;'>Person 3</td><td style='text-align: center; word-wrap: break-word;'>Person 4</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Precision</td><td style='text-align: center; word-wrap: break-word;'>0.684</td><td style='text-align: center; word-wrap: break-word;'>0.983</td><td style='text-align: center; word-wrap: break-word;'>0.575</td><td style='text-align: center; word-wrap: break-word;'>0.973</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Recall</td><td style='text-align: center; word-wrap: break-word;'>0.977</td><td style='text-align: center; word-wrap: break-word;'>0.860</td><td style='text-align: center; word-wrap: break-word;'>0.885</td><td style='text-align: center; word-wrap: break-word;'>0.964</td></tr></table>

![Image](images/1d96cde9132a098ed4556871aeb4941d464a08d00f6a4d0ceea532a7e8d499a1.jpg)

![Image](images/b2e788dcc2820cbb99091e8b7b0880be3e1a1ca4520239f9d9d98cc84be38a71.jpg)

![Image](images/92edf1a32bd4864dff12497ec2376dbef505ccbb275a96935c6442fff1b17f11.jpg)

<div style="text-align: center;">Fig. 9. Precision and recall results on the three different configurations</div>

necessary. We used the CAD-60 subset, which originally consists of 60 RGB-D videos of 4 humans (2 male, 2 female) performing different home activities. For the evaluation, 5 subsets of each person from the dataset were used as indicated in table 1. Since the tested dataset was provided to evaluate action recognition, we had to manually annotate the different images with the correct locations of the persons. This dataset was used to evaluate only the people detection, because the dataset does not provide sufficient hand gestures. We used 60% of the images for training and the remaining 40% for testing. The evaluation was performed 3 times: using only the Frontal Feature, only the Width Feature and finally using both features. The results are reported in table 2 and figure 9.

As can be seen from table 2 the recall for the Frontal Features as well as for the Width Feature is in all cases above 90%. However, based on the results it can not be definitely decided which features is best. On the other hand, when both features are combined the recall decreases, which speaks in favor of only using one of the proposed features. In general, the precision is below the recall in all cases. This means that our algorithm rather oversees a person instead of detecting a person where there is none. This behavior is largely due the parameterization of the candidate detection step as described in section 3 and can be changed by adjusting the parameters. Still, certain poses could not be detected by our algorithm. Mainly, persons standing close to high objects or bending over

---

objects have a low recognition rate. Further, problems occurred when the head was occluded.

To evaluate the hand detection, we used our own dataset with 457 test images. The sensor was calibrated using well established techniques [^15], [^2] to provide accurate RGB and depth image alignment to create the dataset. The hand detection performed with a recall of 73% and a precision of 79%. In this case, main problems arose due to complicated lighting situations, where skin colors differ between hand and face and therefore hands cannot be detected properly.

## 6 Conclusions and Outlook

In this work we proposed a system for people detection based on RGB-D data. Person candidates are extracted from the input point cloud. Two novel Fourier features are computed on the candidate silhouettes and used for classification. The experiments indicate that using only one of the features at a time leads to better results than using a combination of both features. When only one feature is used, recall is above 90% in all cases, while precision is lower, but still at a high value. The hand detection algorithms achieve a recall and precision above 70%. These promising results support our approach for silhouette and hand detection.

In its current state, our algorithm is challenged by people bending over, since the resulting pose does not support our model of a head being directly above the torso. Our future work will concentrate on further improving the algorithm and adjust our geometric human model. We will extend it to not only check the test areas below the head, but also in a diagonal direction to be able to detect bending people.

## References

[^1]: Bertozzi, M., Broggi, A., Del Rose, M., Felisa, M., Rakotomamonjy, A., Suard, F.: A pedestrian detector using histograms of oriented gradients and a support vector machine classifier. In: Intelligent Transportation Systems Conference, ITSC 2007, pp. 143–148. IEEE (2007)

[^2]: Brown, L.G.: A survey of image registration techniques. ACM Computing Surveys (CSUR) 24(4), 325–376 (1992)

[^3]: Cerlinca, T.L., Pentiuc, S.G., Vatavu, R.D., Cerlinca, M.C.: Hand posture recognition for human-robot interaction. In: Proceedings of the 2007 Workshop on Multimodal Interfaces in Semantic Interaction, pp. 47–50. ACM (2007)

[^4]: Choi, W., Pantofaru, C., Savarese, S.: Detecting and tracking people using an rgb-d camera via multiple detector fusion. In: 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops), pp. 1076–1083. IEEE (2011)

[^5]: Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR 2005, vol. 1, pp. 886–893 (June 2005)

[^6]: Ghosh, S., Zheng, J., Chen, W., Zhang, J., Cai, Y.: Real-time 3d markerless multiple hand detection and tracking for human computer interaction applications. In: Proceedings of the 9th ACM SIGGRAPH Conference on Virtual-Reality Continuum and its Applications in Industry, pp. 323–330. ACM (2010)

---

[^7]: González, D.I.R., Hayet, J.-B.: Fast Human Detection in RGB-D Images with Progressive SVM-Classification. In: Klette, R., Rivera, M., Satoh, S. (eds.) PSIVT 2013. LNCS, vol. 8333, pp. 337–348. Springer, Heidelberg (2014)

[^8]: Hordern, D., Kirchner, N.: Robust and efficient people detection with 3-d range data using shape matching. In: Australasian Conference on Robotics and Automation (2010)

[^9]: Kakumanu, P., Makrogiannis, S., Bourbakis, N.: A survey of skin-color modeling and detection methods. Pattern Recognition 40(3), 1106–1122 (2007)

[^10]: Lee, S.J., Nguyen, D.D., Jeon, J.W.: Design and Implementation of Depth Image Based Real-Time Human Detection. Journal of Semiconductor Technology and Science 14(2), 212–226 (2014)

[^11]: Xia, L., Chen, C.-C., Aggarwal, J.K.: Human Detection Using Depth Information by Kinect. In: International Workshop on Human Activity Understanding from 3D Data in Conjunction with CVPR (HAU3D) (2011)

[^12]: Paisitkriangkrai, S., Shen, C., van den Hengel, A.: Strengthening the Effectiveness of Pedestrian Detection with Spatially Pooled Features. CoRR, abs/1407.0786 (2014)

[^13]: Spinello, L., Arras, K.O.: People detection in RGB-D data. In: IEEE/RSJ Int. Conf. on (2011)

[^14]: Sung, J., Ponce, C., Selman, B., Saxena, A.: Human activity detection from rgbd images. plan, activity, and intent recognition, 64 (2011)

[^15]: Zhang, Z.: A flexible new technique for camera calibration. IEEE Transactions on Pattern Analysis and Machine Intelligence 22(11), 1330–1334 (2000)