# A Novel Approach of Counting People Based on Stereovision and DSP Zhu Qiuyu, Tang Li, Jiang Yiping School of Communication and Information Engineering Shanghai University Shanghai 200072, China e-mail: Deng Wei-jun Shanghai Transportation Investment Information Technology CO., LTD Shanghai 200433, China e-mail: Abstract—A novel approach for counting people passing through a region where is surveilled by a parallel-fixed binocular camera is proposed in this paper. Firstly, two virtual counting lines are set to obtain four line space-time gray images. Then an integrated algorithm based on binocular stereovision and regional gray projection is proposed to extract and split overlapped moving objects in complex counting environment. At last, the object matching and counting is accomplished in space-time disparity maps. The counting algorithm takes full advantage of the objects' 2D and 3D information, and is implemented on the DSP hardware system. The practical test results show that the proposed system can operate in real-time and achieve a counting accuracy of over 96% in crowded people environment. Keywords-people counter; disparity map; object detection; object segmentation; stereovision; TMS320DM642 ### I. INTRODUCTION Many enterprises, such as transport system, shopping malls and exhibition halls, are very much concerned with the number of people who travel through their transport systems, shop in their markets, or attend the show. A people counting system is very important for them, especially at the aspects of optimization of management, security monitoring and improving service quality. Therefore, the study of automatic counting system has great practical value. At present, image processing techniques for counting people have become a research hotspot. A mass of methods for counting pedestrians by analyzing a series of images captured with a video camera have been proposed [^1] [^2] [^3]. However these approaches can't correctly deal with the complex counting situation with crowded pedestrians, variable illumination and severe shadows. Because the person is hard to be extracted from the background using a single camera under the complex environments, stereovision [^4] [^5] is used to deal with the problems that are difficult to be solved by a single camera. In the paper, we proposed a novel people counting approach based on stereovision. A real-time counting hardware system is also realized using the high performance DSP chip TMS320DM642 [^6] of Texas Instrument (TI). ### II. THE PROPOSED COUNTING SYSTEM In the proposed scheme, the stereo camera is set on the top of the gate and kept the cameras' optical axis vertical downwards, as shown in Fig. 1. In this way, the passing people could be observed just from overhead, and the overlapping problem can be reduced significantly. Two virtual counting lines (in-line L1 and out-line L2) are set in the monitored area to reduce the amount of calculation and judge the direction of moving people. Image data of the counting line are captured, and then the newest 100 rows data are arranged along the time axis to form an image. Thus, we can obtain four images from two counting lines, as shown in Fig. 2, which are called as line space-time images in this paper. The counting algorithm has three steps: object detection according to peoples' height; object segmentation by the gray and depth of pedestrians; object matching and counting in the disparity map. ![Image](images/b559f7d1497ec6976398b912ea5a30c8f06bc4ce46b5c889463732d975e81059.jpg)
Figure 1. Setting of the counting system using the stereo camera
![Image](images/c5488a4a11cd936e174fbf004beb8aa8f2e688a5cfe0dc8a2db02172d1f98ffd.jpg) ![Image](images/0318a7eb906fd75db3ae72e392df67e11aae510113f6c1a998ee94c7adea5d49.jpg)
Figure 2. Line space-time images of two counting lines: (a) L1 of left camera; (b) L2 of left camera; (c) L1 of right camera; (d) L2 of right camera.
--- ### III. OBJECT DETECTION In the proposed system, we use a parallel stereo camera to extract objects. As we know, the equation [^7] between parallax and object distance is $p = bf/d$, where b and f are the baseline distance and focal length, d is the vertical distance from a spatial object point to the image plane of camera, and p is the parallax of a spatial point. The passing people are detected according to their height. ### A. Disparity map In this paper, we use area-based image matching to get the dense disparity map. As L1 and L2 are epipolar lines, matching windows is reduced to 1-D. The normalized cross correlation (NCC) criterion [^8] is used to find the optimal matching pixel. In order to accelerate disparity computing, the possible object pixels are firstly detected by line frame difference, and then the parallax of these pixels are evaluated by a 1-D window (its width is w) correlation matching as follows: $$ \rho(d)=\frac{\displaystyle\sum_{m=-w}^{m=w}(f_{r}(j+m,t)-\overline{f_{r}(j,t)})(f_{l}(j+d+m,t)-\overline{f_{l}(j+d,t)})}{\sqrt{\sum_{m=-w}^{m=w}(f_{r}(j+m,t)-\overline{f_{r}(j,t)})^{2}}\sqrt{\sum_{m=-w}^{m=w}(f_{l}(j+d+m,t)-\overline{f_{l}(j+d,t)})^{2}}} $$ Where $f_{r}$ and $f_{l}$ are the right and left line space-time image intensities, $\overline{f_{r}}$, $\overline{f_{l}}$ are their average value over the correlation window and d represents the window displacement along the epipolar line in the left line space-time image. $\rho$ represents similarity degree of two matching windows. The parallax d corresponds to the displacement of maximum $\rho(d)$. After the quadratic interpolation of NCC value $\rho(d)$, the optimal sub-pixel matching position can be obtained in the right line space-time image. Thus, the disparity maps of the two space-time images are obtained. ### B. Detecting Object Region In the counting system, objects that we are interested in are closer to the sensor than background. The closer the object is apart to the sensor, the bigger the parallax we obtain. Therefore, if there are people in the line space-time image, there will be some brighter regions in the disparity map. In order to eliminate background interference and detect moving objects, we set a threshold $T_{d}$ to segment the peoples' heads in the disparity map. Object region is detected using (2). $$ d\left(i,j\right)=\left\{\begin{aligned}&g\left(i,j\right)&\quad&g\left(i,j\right)\geq T_{d}\\ &0&\quad&others\end{aligned}\right. $$ Where $g(i,j)$ and $d(i,j)$ are the gray before and after object detection in the disparity map. $T_{d}$ is determined by specific application environment and binocular camera installation height. After the thresholding and object labeling, we get the object's top-left $(y_{T}, x_{L})$ and bottom-right $(y_{B}, x_{R})$ positions. Two disparity maps corresponding to Fig. 2 (a) and (b) are shown in Fig. 3. ![Image](images/74fd7a0e7d59c55bd20b6c66d1a26f2d31f9606614782f2b09fd0fbba0710293.jpg)
(a)
![Image](images/916a29c6fac5fac43dd37be44d207878ee0a9e22f5d92b75496ec9f159d56f97.jpg)
(b)
Figure 3. Disparity maps corresponding to the left line space-time image: (a) in-line L1; (b) out-line L2
### IV. OBJECTS SEGMENTATION When the crowded people pass through a counting scene, many existed counting methods are difficult to achieve an good accuracy. Because each moving object often tends to be merged in crowded situation, this problem can confuse counting and make the counting error increase. In this paper, firstly, objects are detected according to their height. This step can eliminate some ill effects, such as illumination changes and shadow. But when there are two shoulder-to-shoulder people who have almost the same stature or the one is taller than the other by a head, we can't segment correctly merged objects only by objects' parallax. To solve this problem, a novel method integrated gray projection of object region is used to segment the multi-people merging pattern. Based on the analysis of experimental results, each object belongs to one of three cases, and it is handled separately as follows (s is object's area and $w = x_{R} - x_{L}$ is object's width): 1) If $w \geq w_{max}$ and $s \geq s_{min}$, split object firstly using object region projection which will be discussed latter, and then segment the object by its parallax; 2) If $\min \leq w \leq w_{\max}$, segment object only according to parallax; 3) If don't satisfy the above situations, discard the object. Where $[w_{min}, w_{max}]$ and $s_{min}$ are respectively the width range and minimum area of single-people in disparity maps. Supposed that an object rectangle size is $M \times N$ (M and N are respectively the height and width of the rectangle) and its bottom-left corner coordinate is $(i,j)$. The segmentation is carried out by gray projection along the direction of moving people. The projection is computed for each column as (3) and (4) $$ h(n)=255-\sum_{m=i-M+1}^{i}k(m,n)f(m,n)\bigg/N_{0}\qquad n\in(j,j+N)\ ( $$ $$ k\left(m,n\right)=\left\{\begin{aligned}&1&\quad&d\left(m,n\right)\neq0\\ &0&\quad&d\left(m,n\right)=0\\ \end{aligned}\right. $$ --- where $f(m,n)$ and $d(m,n)$ are the line gray space-time image and disparity map intensity, $N_{0}$ is the number of pixel whose parallax is nonzero. An example is shown in Fig. 4. The merged pattern, which assumes that people have darker hair, can be successfully segmented according to the above method.
O
![Image](images/a3b4e5b4b47ae5e042cdc4ca92a5b29ef3839fcd64edb50497052c27b3ca4a3a.jpg)
O
![Image](images/d891224fa0b5769714fa97aec0c23db6f5dcfda712b2921333a26a6005168572.jpg)
(a)
![Image](images/407470b947ed69cabe2a894f431bab4c6bc6af4b1bfbf5835586b28ffa98f1fe.jpg)
(b)
(c)
O
![Image](images/2be64ac79320f62d6916c642eb40f711cbac4b77ab60ed97d3b403b065798d69.jpg)
Figure 4. Disparity maps corresponding to the left line space-time image: (a) line space-time image; (b) disparity map; (c) projection histogram of object region; (d) the result of segmentation
### V. OBJECT MATCHING AND COUNTING When an object appears in in-line disparity map, we should search the same person who appears in the out-line disparity map. Sometimes, there are two or more objects within the matching region, the optimal matching object is chosen by using the cost function: $$ F\left(x,y_{n}\right)=aS\left(x,y_{n}\right)+bD_{x}\left(x,y_{n}\right)+cD_{y}\left(x,y_{n}\right) $$ This cost function is used to measure similarity between the object x in the in-line disparity map and the object $y_{n}$ in the out-line disparity map. $S(x,y_{n})$, $D_{x}(x,y_{n})$ and $D_{y}(x,y_{n})$ respectively represent the difference of the average parallax, distance at horizontal direction and distance at time direction between x and $y_{n}$. a, b and c are the weighing, which are set to 0.4, 0.4, 0.2 respectively according to experiments. The optimal matching object is $y_{n}$ with minimum cost. As we've got a pair of matching objects, this person's direction can be judged by its relative position in two disparity maps. If a person comes in the counting scene, he will firstly appear in in-line disparity map, and vice versa. ### VI. SYSTEM HARDWARE ARCHITECTURE AND SOFTWARE Many counting systems based on stereovision and global moving object tracking can't meet real-time demands. Due to small amount of calculation of the proposed algorithm, algorithm is easy to work in real-time. We use the high performance DSP chip TMS320DM642 of Texas Instrument (TI) to design and realize a real-time counting hardware system. Fig. 5 shows its main architecture. ![Image](images/289d5b8237973fa1c5e0db4485cd3a7d02dafccfb57737aa53b947386f331bc6.jpg)
Figure 5. Diagram of hardware system
The two synchronoused analog video signals from binocular camera are captured by the system. An integrated infrared LED light is used to get high-quality gray images at night. The algorithm of counting people can be realized on DM642 chip. The counting results will be transmitted to supervising computer through McBsp serial port. And the video can be shown on monitor after having been processed by video encoding. ![Image](images/09a2cb7311a13606c1c88d1f6e7426d756350b09b2b370e1412997fb9615d68f.jpg)
Figure 6. Software processing flowchart of the proposed counting method
--- For the software programming, C language and assemble language in integrated development environment CCS and DSP/BIOS are used to implement image processing and people counting algorithm. The program flowchart of this counting system is described in Fig. 6. ### VII. EXPERIMENTAL RESULTS A large number of practical tests show that the paper's counting algorithm and hardware system can realize real-time counting with frame rate at 25 frames per second and has very good stability. In order to verify the efficiency of counting algorithm, we test this system in multiple experiments with different situations of pedestrian congestion and different counting backgrounds. In table 1, index number 1, 2, and 3 respectively represents the situations of one person after another, two persons closing to each other or side by side, and more than two people together passing through the counting scene. E1 means an ideal background environment, and E2 represents a complex counting scene with serious illumination changes and shadows. The counting results indicate that the error rate is less than 4% in the all above-mentioned counting environments.
TABLE I. COUNTING RESULTS IN DIFFERENT ENVIRONMENTS
No.Environ-mentActual persons (in/out)Counting results (in/out)Error rate (%) (in/out)
1E1250/256250/2550.00/-0.39
E2200/201199/200-0.50/-0.50
2E1211/220207/217-1.90/-1.36
E2198/200194/198-2.02/-1.00
3E1240/243247/235+2.92/-3.29
E2210/210218/216+3.81/+2.86
### VIII. CONCLUSIONS In the paper, an automatic counting system based on stereovision and DSP has been proposed. The counting approach adopts two counting line's space-time images to improve reliability of counting algorithm, and reduce amount of calculation. The 2D gray image data and the 3D height information of people are integrated to detect moving objects, and segment merged multiple people by object region gray projection and its parallax. The experimental results show that the algorithm has high counting accuracy, and the hardware system meets the demands of real-time and stability. The system is reliable and robust to illumination changes, shadow interference, and crowded people. ## REFERENCES [^1]: J. W. Kim, K. S. Choi, B. D. Choi, J. Y. Lee and S. J. Ko, “Real-Time System for Counting the Number of Passing People Using a Single Camera”, Lecture Notes in Computer Science, vol. 2781, pp. 466-473, 2003. [^2]: T. H. Chen, C. W. Hsu, “An automatic bi-directional passing-people counting method based on color image processing”, IEEE 37th Annual 2003 International Carnahan Conference on Security Technology, pp. 200-207, Oct. 2003. [^3]: X. W. Xu, Z. Y. Wang, Y. H. Liang, and Y. Q. Zhang, "A rapid method for passing people counting in monocular video sequences", Proceedings of the sixth International Conference on Machine Learning and Cybernetics, vol. 3, pp. 1657-1662, Aug. 2007. [^4]: Terada K, Yoshida D, Oe S, and Yamaguchi J, “A method of counting the passing people by using the stereo images”, IEEE International Conference on Image Processing, vol. 2, pp. 338-342, 1999. [^5]: Yahiaoui T, Meurie C, Khoudour L and Cabestaing F, "A people counting system based on dense and close stereovision", International Conference on Image and Signal Processing, pp. 59-66, Jul. 2008. [^6]: TMS320DM642 Video/Imaging Fixed-point Digital Signal Processor Data Manual, Literature Number: SPRS200E, Texas Instruments Incorporated. July 2002-Revised March 2004. [^7]: D. Beymer, “Person Counting Using Stereo”, Proc. IEEE Workshop on Human Motion, pp.127-133, 2000. [^8]: Fua P, "A parallel stereo algorithm that produces dense depth maps and preserves image features", Machine Vision and Applications, Vol.6(1), pp.35-49, 1993.