# Hand Pose Understanding With Large-Scale Photo-Realistic Rendering Dataset Xiaoming Deng $^{ID}$, Yinda Zhang, Jian Shi, Yuying Zhu, Dachuan Cheng, Dexin Zuo, Zhaopeng Cui, Ping Tan $^{ID}$, Senior Member, IEEE, Liang Chang $^{ID}$, and Hongan Wang Abstract—Hand pose understanding is essential to applications such as human computer interaction and augmented reality. Recently, deep learning based methods achieve great progress in this problem. However, the lack of high-quality and large-scale dataset prevents the further improvement of hand pose related tasks such as 2D/3D hand pose from color and depth from color. In this paper, we develop a large-scale and high-quality synthetic dataset, PBRHand. The dataset contains millions of photo-realistic rendered hand images and various ground truths including pose, semantic segmentation, and depth. Based on the dataset, we firstly investigate the effect of rendering methods and used databases on the performance of three hand pose related tasks: 2D/3D hand pose from color, depth from color and 3D hand pose from depth. This study provides insights that photo-realistic rendering dataset is worthy of synthesizing and shows that our new dataset can improve the performance of the state-of-the-art on these tasks. This synthetic data also enables us to explore multi-task learning, while it is expensive to have all the ground truth available on real data. Evaluations show that our approach can achieve state-of-the-art or competitive performance on several public datasets. Index Terms—Hand pose estimation, photo-realistic synthetic dataset, physical-based rendering, multi-task CNN. ### I. INTRODUCTION ## A S AN important task in computer vision, recovering hand pose is essential for many applications in the Manuscript received July 18, 2019; revised February 29, 2020 and October 20, 2020; accepted March 15, 2021. Date of publication April 7, 2021; date of current version April 14, 2021. This work was supported in part by the National Key Research and Development Program of China under Grant 2019YFC1521100 and in part by the Distinguished Young Researcher Program, Institute of Software, Chinese Academy of Sciences. The associate editor coordinating the review of this manuscript and approving it for publication was Dr. Senem Velipasalar. (Xiaoming Deng, Yinda Zhang, and Jian Shi contributed equally to this work.) (Corresponding author: Xiaoming Deng.) Xiaoming Deng, Yuying Zhu, Dexin Zuo, and Hongan Wang are with the Beijing Key Laboratory of Human-Computer Interactions, Institute of Software, Chinese Academy of Sciences, Beijing 100190, China (e-mail: ; ). ![Image](images/aea45bb45e3828357fe0e5020b001db8a52ef27588d5ae50a3c0352dcc0e0017.jpg)

Photo-Realistic Rendering Hand mesh model

Fig. 1. The overview of our work. We build a large-scale photo-realistic hand pose database with a variety of hand shapes and poses, and render millions of synthetic images with different kinds of ground truths. Our dataset with multiple annotations enable us to design a multi-task CNN structure that predicts depth and 3D hand pose from input RGB image in a cascaded way.

area of human-computer interaction, augmented reality, and robotics [^1]. One type of the most popular approaches to address this problem are deep learning based methods, which learn the hand pose from large-scale dataset. One key bottle-neck of deep learning based hand pose methods is the lack of large-scale hand pose dataset with diverse and accurate annotations. Existing datasets for hand pose understanding are very limited in terms of number of frames, number of subjects, level of annotations and annotation accuracy. For example, BigHand2.2M [^2] is the largest public available hand pose dataset, which only contains depth and 3D joint annotations. In order to collect BigHand2.2M, the subjects have to wear a marker-based tracker on the hand to get ground truth. As a result, the color image could be contaminated and cannot be applied to color-based hand pose estimation task. RHD [^3] contains diverse annotations, but it only has about 44K images. Moreover, RHD is not photo-realistic and has a great domain gap to real color images. Collecting large-scale hand pose datasets is not trivial, and it needs accurate 3D hand pose estimation as well as labor-intensive efforts for semantic segmentation. To reduce the cost for building real datasets, synthetic data has been widely employed in recent works [^3] [^4] [^5] [^6]. Compared to real data, synthetic data is easy to scale up and provides --- high-quality ground truth. Another advantage of synthetic data is the availability of diverse supervisions. For example, besides the 3D hand pose ground truth, many other kinds of supervision can be obtained from virtual hand models, e.g. depth and semantic segmentation, which could be potentially useful for pose estimation. One of the major concerns about the synthetic data is the domain gap on the input side, e.g. color, depth, between the rendered synthetic data and real data. The other problem is how hand pose understanding related tasks are affected by large-scale datasets. To address these problems, we develop a large-scale synthetic dataset (about 5.52M images), namely PBRHand. Images are rendered from textured hand meshes under a variety of hand poses. To ensure the reality, we capture high-fidelity 3D hand models and pose sequence with professional 3D scanners and digital glove. We randomly sample a variety of camera viewpoints and environment map illuminations, and employ physically based rendering (PBR) to generate photo-realistic images. As for the ground truth, we generate accurate 3D skeleton hand pose, pixel-wise semantic segmentation (e.g. finger, palm) and depth maps. We use our data to pretrain convolutional neural network for three hand pose understanding related tasks, 2D/3D hand pose from color, depth estimation from color image, and 3D hand pose from depth. The experiments demonstrate that for these hand pose related tasks our data can improve the network performance of the current state-of-the-art. We also show that for 3D pose from single color image the dataset enables us to pretrain our model with multiple supervisions and delivers better performance after finetuning on real datasets (see Fig. 1). Specifically, we present a cascaded multi-task deep neural network based on PBRHand for hand pose estimation from color image, which can effectively learn from an auxiliary depth recovery task. Compared to Weakly-sup [^7] that also leverages depth map for auxiliary supervision during the hand pose estimation, our cascaded multi-task method adopts a different way to use depth map. Inspired by the success of hand pose from depth, we consider depth map/feature can be used as important input for hand pose, and we conduct an in-depth analysis on the effect of different cascaded forms (feature cascade, input/output cascade), and find that the depth feature can improve the 3D hand pose estimation from color (See Section VI.B). Weakly-sup [^7] presents a weakly-supervised network for 3D hand pose estimation. They predict 3D hand pose from the input color image, and then generate depth map from the predicted 3D hand pose. The generated depth map serves as weak supervision for 3D pose regression. Moreover, as shown in Table II of our supplementary document, our method achieves better performance than [^7]. The major contributions of this work are as follows: 1) We develop a high-quality synthetic hand pose dataset, namely PBRHand. It contains a large number of photo-realistic synthetic color images with various ground truths (depth maps, semantic segmentation labels and hand joints); hand pose related tasks. We also show that the rendering quality is important for hand pose estimation; 2) We demonstrate that the networks pretrained on our PBRHand achieve state-of-the-art performances on three 3) Our PBRHand also enables us to explore multi-task learning. We present a cascaded multi-task deep neural network based on PBRHand for hand pose estimation from color image. Our network can effectively learn from an auxiliary depth recovery task for 3D hand pose estimation. The network achieves the state-of-the-art or competitive performance on public datasets. The rest of the paper is organized as follows. In Section II, we introduce related works. In Section III, we introduce the pipeline of generating synthetic hand pose dataset. In Section IV, we investigate the effect of our dataset on three typical hand pose related task. In Section V, we present our multi-task model for hand pose estimation in detail. In Section VI, we show comparison experiments and ablation study. Conclusions are given in Section VII. ### II. RELATED WORK Building large-scale dataset is a key issue for hand pose estimation using convolutional neural networks. One direction is to construct real hand pose datasets, in which the images are captured with a real color or depth camera, and the joints are annotated manually, by model fitting methods [^16], or jointly using both methods [^17]. Constructing large-scale dataset with manual annotation is almost prohibited, due to the fact that it usually requires excessive human efforts, results in inaccurate annotations, and it is also time-consuming for scaling up. ICVL, NYU and MSRA dataset are three main depth image benchmark datasets for hand pose performance evaluations. ICVL dataset [^17] adopts human pose tracking and manual refinement for annotation. However, the total numbers of frames and subjects are small, and its issue of annotation accuracy is found in [^18]. MSRA dataset [^19] uses the same strategy for annotation. Its scale is larger, but the annotation accuracy is still not high. NYU dataset [^16] is a larger benchmark with a large view coverage, large variation in articulation. It relies on first model-based fitting, and then particle swarm optimization to get the joint annotation. Manually adjusting annotation and re-initialization are required. Oberweger et al. [^18] also propose a semi-automated method for labeling a hand-depth video with the 3D joints. The dataset currently contains only 4 subjects and about 63K RGBD frames. Recently, Yuan et al. propose BigHand2.2M [^2]. BigHand2.2M is currently the largest real dataset for depth, and the hand joints are annotated with magnetic markers. The dataset is of less diversity with respect to hand shape with only 10 subjects. Moreover, it is not suitable for color-based hand pose estimation tasks. Datasets such as Stereo [^20], Dexter+Object [^10], Dexter Ego [^4], MPII+NZSL [^21] for color based hand pose estimation are usually either relatively small or contain only partial annotations. Recently, FreiHands [^9] is created as a large-scale real dataset of color images with hand pose/shape labels; However, the background is synthetic, and the annotation is semi-automatic and requires human-in-the-loop verification. ---

TABLE I

DATASET COMPARISON. 'PARTSEG' IS ABBREVIATION FOR HAND PART SEGMENTATION. 'FINGERTIPS' MEANS 5 FINGERTIPS ARE LABELED, AND IT CAN BE EXTRACTED FROM 'JOINTS'. OUR DATASET HAS BETTER DATA VARIATIONS, QUALITY OF RENDERED IMAGES AND FULL ANNOTATIONS

Dataset	Content	Type	Frames	Subjects	Viewpoints
Stereo [^8]	RGB, depth, joints	real	18K	6	3rd
FreiHands [^9]	RGB, joints	real	170K	-	3rd
Dexter+Object [^10]	RGB, depth, fingertips	real	3,014	2	3rd
Dexter Ego [^4]	RGB, depth, fingertips	real	3,190	4	ego
FingerPaint [^11]	depth, fingertips	synthetic	100k	1	full
HandNet [^12]	depth, 6 joints	real	212k	10	3rd
MHP [^13]	depth, joints	real	80k	9	3rd
Bighand2.2M [^2]	depth, joints	real	2.2M	10	full
RHD [^3]	RGB, depth, joints, partseg	Blender [^14]	44K	20	full
Our PBRHand	RGB, depth, joints, partseg	Mitsuba [^15]	5.52M	50	full

The other direction is to build synthetic dataset, which has shown promising effect for convolutional neural network training of many computer vision tasks, indoor scene understanding [^5], intrinsic image [^6], optical flow [^22], [^23], object pose estimation [^24], and human pose estimation [^25]. These methods employ synthetic dataset with ground-truth annotations for network pretraining, and then finetune on real dataset. Synthetic data for human hand is especially challenging due to the fact that it is generally hard to capture high-quality hand shapes and hand motions from real subjects as well as generate photo-realistic images to narrow the gap between the real images and synthetic images. Zimmermann and Brox [^3] used a rendered hand dataset named RHD with various ground truth for pose estimation. The dataset is almost limited with the number of frames (44K) and the number of human subjects (20). In contrast, we provide 125 times of images in RHD, and our rendered color images are more photo-realistic than the color images in RHD. SynthHands [^4] contains depth and color images from an egocentric viewpoint, and the images are rendered using Unity. The dataset is the first synthetic dataset with hand interaction with objects. GAN-ratedDataset [^26] translates synthetic images in SynthHands to real images using geometrically consistent CycleGAN. The dataset contains more than 330K color images of hands with 2D and 3D hand pose annotation. The MSRC FingerPaint hand pose dataset [^11] contains 100K synthetic depth frames for one subject, and its hand poses are limited by random sampling from six articulations [^2]. Our dataset proposed in this paper differs from previous datasets. It is the first large-scale hand dataset with photo-realistic RGB, depth, semantic segmentation, and 3D hand joints. The whole data generation pipeline is of little cost and can be scaled up easily. We show that our data with shape, pose, texture and illumination variations can benefit hand pose understanding problems. ### III. PBRHAND HAND POSE DATASET We build a large-scale hand pose dataset, named PBRHand (physically based rendered hand). It contains millions of photo-realistic color images and various kinds of accurate ground truths which could be potentially helpful for hand pose related tasks. Recently, deep learning methods have been successfully applied to hand pose estimation problem, and several good datasets are publicly available (Table I). Those datasets made great contributions to improve the quality of this task. However, there are still some limitations. First, for real data, it is extremely expensive to capture and label large-scale datasets for training complex deep neural networks. Current real datasets [^4], [^20] are only built with thousands of frames and quite few subjects, which limits the generalization of deep neural networks. Recently, Zimmermann and Brox [^3] build a synthetic dataset named rendered Hand Pose Dataset (RHD), which contains more frames, poses and subjects. The data variation of this synthetic dataset is generally better than existing real datasets. However, since they use low-quality 3D human models and unrealistic rendering method, there exists a large gap between the dataset and real-world images. To bring further improvement to hand pose understanding, we choose to build a new dataset. To make the data realistic and diverse, we first use professional devices to capture hand models and poses from human subjects. We manually clean and label those scanned 3D hand models. After that, we apply photo-realistic rendering to generate millions of images with multiple ground truths. Overall, our PBRHand contains 5.52 million synthetic color images with 3D hand pose, semantic segmentation and depth. Fig. 2 shows several examples of our photo-realistic hand dataset. Table I shows the comparison of our dataset and other datasets. ### A. 3D Hand Model and Pose To collect high-fidelity mesh models, we use a hand-held 3D scanner Artec $^{\circledR}$ Eva scanner [^27] to capture dense 3D point cloud of hands in standard upright pose (Fig. 3 (a)). For the diversity of hand shapes between human, we collect hand shapes from 50 people, including 30 males and 20 females of different ages, careers and races. We scan a human hand at rest pose by moving Eva scanner around the hand, soften point cloud and remove noise of each frame using 'Smooth Brush' and 'Eraser' tools in Eva capture software, then fuse multiple scans into a completed mesh using the 'Fusion' tool. Though Artec Eva scanner can provide color texture for a captured mesh, we found the texture was blurry. --- ![Image](images/43a479994fda03012e2cabb3310ee1cd6f8307b30eda4d25228d4876872090cb.jpg)

Fig. 2. Our photo-realistic hand dataset by posing real hand models with real hand motion data. Examples of rendered photo-realistic hand color images are shown on the top rows. Examples of hand poses, rendered depths and semantic segmentation are shown on the bottom rows.

![Image](images/c97e6f8021ff35ac49b3c1662a83a6cec1c6b2e7badd9ebe4abc484147456dc0.jpg)

(a)

![Image](images/4a34dea77a8449c60d4fa7315e9767df0b2fe86e49279558c07058df1b260e34.jpg)

(b)

![Image](images/d53bc24a6bf62630b66e0d9ac51a2ff7e789becde24d8c228702240db6517867.jpg)

(c)

Fig. 3. Hand model generation. We show captured hand point cloud with 3D scanner in (a), rewired mesh in (b), and mesh deformation under several poses in (c). We use real hand mesh models instead of hand meshes designed by artists, which are broadly used in previous hand pose estimation literature.

![Image](images/3a6ca433e89298992a3290224b26ed1ef710a7e01eb66c3278de47be008b36e9.jpg)

Fig. 4. Textured hand shape. left: original texture; right: improved by designer.

in some areas and there were also some artifacts. Therefore, we employ professional designers to manually refine the hand textures for high reality and rich details (Fig. 4). In order to simplify the rigging and texture mapping process, we rewire the surface line to fit natural hand topology of a standard hand mesh template using nonrigid ICP [^28] (Fig. 3 (b)). In our current hand model, we do not add arm part. Adding arm while still maintaining the reality is a challenging task. Capturing 3D models for arm and dealing with skin, pose, and potential interaction with clothes requires huge effort. This is a great direction for future work. The point clouds are then fitted to triangulated textured mesh using UV mapping [^29]. The texture maps of our hand models are manually created by professional engineers according to real hand texture images. To capture realistic hand poses, we summarize 1,840 key hand poses for human-computer interactions inspired by existing datasets such as NYU [^16], Dexter+Object [^10] and EgoDexter [^4]. We ask subjects to show these poses wearing a Measurand $^{\circledR}$ ShapeHand glove, which captures skeleton poses. The key pose set covers typical hand gestures such as grasping, pinch, abduction-abduction of all fingers together, flexion-extension of all fingers etc. The subjects are also asked to move hand freely to further increase the pose diversity. The captured hand poses usually contain noises, so we employ professional designers to refine local joint position and orientation. It makes hand poses right and reduces artifacts. The collected hand poses are refined and semi-automatically re-targeted onto the collected hand mesh models using rigging and deformation [^30], so that the hand meshes can be transformed automatically into the collected pose set. Fig. 3 and Fig. 5 show some examples of hand mesh deformation under several poses. The captured hand poses usually contain noises, so we employ professional designers to refine local joint position and orientation. It can reduce hand mesh artifacts such as mesh penetrations during the model re-targeting. We first manually align the standard hand pose to hand meshes (only one mesh per subject) by adjusting joint local offsets. And then we bind the skeleton and mesh in Autodesk $^{®}$ Maya, and we can generate meshes automatically under different poses by deformation. However, the deformation may have distortions to the smooth-binded skin, we employ professional artists to manually check the data, adjust the skin weights and control --- ![Image](images/f7baef93005503d5ec495ee9cc367cff4b4c3002d1c5149dbe710b2532bbadfb.jpg)

Fig. 5. Hand shapes and poses. With hand models of 50 subjects and 1,840 gestures, we synthesize 92K hand meshes with various hand shapes and poses.

TABLE II

	Capture hand shape and post-process	Create hand texture	Rig and deform hand shape with pose
Time	10	60	15

MANUAL WORK (IN MINUTE) OF HAND SHAPE CAPTURE AND TEXTURE CREATION FOR EACH SUBJECT

the deformation, and make sure that the rigged hand meshes are of high quality. Moreover, with the current data collection framework, we are able to capture high quality pose and shape and create photo-realistic texture within a reasonable amount of time. Firstly, we collect a hand pose set of 1,840 poses using a Measurand $^{\circledR}$ ShapeHand glove within less than two hours. Secondly, we collect hand shapes, create hand textures, and retarget the hand poses for 50 subjects. The manual work for each subject costs about 85 minutes. Table II shows the detailed time of manual work for each subject. With high-fidelity hand meshes and realistic pose, we employ a widely used physically-based renderer Mitsuba [^15] to generate photo-realistic images. We pick camera viewpoints with random distances in [^400] [^800] mm away from the center of the mass of the hand model, and sample 3 camera positions with the zenith angle in $[\frac{1}{6}\pi, \frac{2}{3}\pi]$ and the azimuth angle in $[0, 2\pi)$. To simulate realistic illumination effect, we collect 20 HDR environment maps, where indoor and outdoor scenes with various illumination conditions are included. Fig. 6 shows these environment maps. The perspective camera has a resolution of $640 \times 480$, horizontal field of view of 54 degrees. Given a camera position, our hand models are rendered under a random subset of the environment maps using path tracing. With the hand meshes, textures, environment maps and random sampled viewpoints, we obtain more than 5.52 million hand images by photo-realistic rendering. Fig. 2 illustrates the synthetic image, depth, segmentation and joints of our dataset. ### B. Physically Based Rendering In order to compare whether photo-realistic rendering is helpful for hand pose understanding tasks, we also render from ![Image](images/11c86c67e41259680c43e36a407f886b198c0086fdeab6253c62565f81bc6977.jpg)

Fig. 6. HDR environment maps used for rendering our photo-realistic hand images. They consist of 10 indoor scenes and 10 outdoor scenes under different lighting conditions.

![Image](images/39dc48368f099d26360e608378baf90e357717f18cafc7fe486926af2b382cab.jpg)

(a) PBRHand-OpenGL

(b) PBRHand

Fig. 7. Hand image rendered with different methods. (a) OpenGL. (b) Physically-based rendering.

Unrealistic rendering with directional lights using OpenGL. Our dataset rendered with OpenGL is named PBRHand-OpenGL. Basic shading effects, multiple directional lights are included, but no global illumination, or shadow is included. Fig. 7 shows examples with OpenGL and physically-based rendering. We only show the hand regions to better illustrate the effect of different rendering methods on hand regions. ### C. Annotation For each sample in our PBRHand dataset, we provide rendering color images with semantic segmentation, depth image and 3D joint coordinates. When modeling hand shape, designers manually label 15 hand parts' segmentation on the hand meshes of a standard pose, the segmentation is stored as a texture map, and the segmentation of hand mesh undergoing different poses can be automatically labeled during the hand pose retargeting process. The semantic segmentation is also generated by rendering the albedo of the hand mesh with semantic label encoded texture map. The depth is extracted from z-value of 3D position in camera space. The depth --- ![Image](images/0fc61061927a4f80af852c735036d00bed6008796c7037d4a42fb5e56cc02c88.jpg)

(a) Hand pose space

![Image](images/b92fb571c5e535abd2a9efa1cdc455a1371f15c369bc21e0003b1e6d10050227.jpg)

(b) Hand shape space

Fig. 8. 2D t-SNE of hand pose and hand shape space for PBRHand, BigHand2.2 and RHD dataset.

is rendered using “distance field” integrator followed by a post-processing that converts the ray distance to camera center into depth with camera intrinsic matrix. When modeling hand poses, hand meshes are deformed according to the pose of its corresponding skeleton. We calculate the joint locations in the camera space with the transformation and forward kinematics. ### D. Hand Pose/Shape Space Following Bighand2.2M [^2], we visualize the hand pose/shape space, and compare our PBRHand dataset with the existing datasets using 2D t-SNE. Fig. 8 shows the projected hand pose space and hand shape space of our PBRHand dataset, RHD [^3] and BigHand2.2M [^2]. We observe that our dataset has better coverage in hand shape space, and the hand pose distribution of our dataset and Bighand2.2M is better than RHD. It is consistent with the intuition that the pose space is simpler than the shape space, where the hand shape can be more difficult to be captured and the hand shape is different between subjects. ### IV. HAND POSE UNDERSTANDING RELATED TASKS In this section, we first investigate three fundamental hand pose related tasks: (1) 2D/3D hand pose estimation from color image; (2) depth estimation from color image; (3) 3D pose from depth image. For these tasks, we show how the state-of-the-art model pretrained with our synthetic dataset compares with the model pretrained with other synthetic datasets. ### A. Dataset and Evaluation Metrics In this paper, we evaluate hand pose estimation performance for different hand pose understanding tasks on several public datasets: Stereo dataset [^8], MPII-NZSL dataset [^21], NYU dataset [^16], Dexter+Object dataset [^10], Rendered Hand Pose dataset (RHD) [^3], and Task3 of Hands 2019 Challenge (Hands-2019) [^31]. In this section, we use Stereo dataset, MPII-NZSL dataset and NYU dataset for evaluation. In Section V, we use Stereo, RHD, Dexter+Object and Hands 2019 datasets for evaluation. We briefly review them in the following and present the evaluation metrics. 1) Stereo Hand Pose Dataset (Stereo) [^8]: is the largest public real image dataset with fully annotated 3D hand joints using color images. We use the color-depth subset of Stereo SK captured from a RGBD camera. It provides color and depth images, 3D annotations for 21 hand keypoints. The dataset is separated into an evaluation set of 3,000 images and a training set of 15,000 images. 2) MPII+NZSL Dataset [^21]: contains 2800 images with 2D hand pose annotations. There are about 2000 and 800 images for training and testing. 3) NYU Hand Dataset [^16]: is a depth image based hand pose dataset. We use NYU dataset to compare the effect of our scanned hand shape and the parametric MANO model on 3D hand pose estimation from depth. However, we do not evaluate on the color images of NYU dataset, because these images are registered with depth and thus corrupted [^3]. 4) Rendered Hand Pose Dataset (RHD) [^3]: is a synthetic dataset, which consists of 41,258 images for training and 2,728 images for evaluation. The images are rendered using Blender. The dataset provides 2D and 3D joint annotations, depth map, and part segmentation. 5) Dexter+Object Dataset [^10]: consists of 6 sequences with 2 actors, and varying interactions with a simple object shape. Fingertip positions and cuboid corners were manually annotated for all sequences. Hand pose estimation on Dexter+Object Dataset is very challenging, since no training dataset is available and there are heavy occlusions due to the interaction between hand and object. 6) Task3 of Hands 2019 Challenge Dataset (Hands-2019) [^31]: contains images from 3 subjects manipulating 4 objects for training, and contains images from 5 different subjects manipulating 6 different objects for testing. Hands-2019 dataset is also challenging due to the heavy occlusion of objects. a) Evaluation metrics: We conduct three hand pose understanding related tasks, and for each task we adopt the standard evaluation metrics in literature. To evaluate 2D and 3D hand pose accuracy from color image, we follow common metrics [^3] and use average End-Point-Error (EPE), Area Under the Curve (AUC) and percentage of correct keypoints (PCK) over different thresholds. For 3D hand pose from color image, we follow [^3] to estimate normalized 3D joint coordinates relative to a reference point, which can be selected as a hand root joint or the 3D center-of-mass (CoM) of hand foreground region estimated from the aligned depth of the color image. In the evaluation of Stereo, RHD and Dexter+Object, we use 3D CoM as the reference point. In the evaluation of Hands-2019, we use the provided ground-truth wrist locations for the test images as the reference point. During the evaluation, the absolute joint positions are obtained by adding the predicted normalized 3D joints with the reference point. For MPII+NZSL dataset, we follow [^21], [^32] to use handsize-normalized PCK for evaluation. For depth estimation from color, we use mean absolute error (MAE) to evaluate the difference between the ground truth and estimated depths on the hand foreground. To evaluate 3D hand pose from depth, we use the standard metrics proposed in [^17], the mean joint errors and the percentage of --- ![Image](images/d2f27a33a557e75931acd526318886fcff487cd58ccac3b4e2d7bab03105fdf4.jpg)

(a) 2D hand pose results with PoseNet.

![Image](images/20ed59f33d88075ec2291e3b33831d05323ad7fac9b60f85408518a9f99ea86b.jpg)

(b) 3D hand pose results with PoseNet + PosePrior.

Fig. 9. (a) 2D hand pose results with PoseNet. (b) 3D hand pose results with PoseNet + PosePrior. PoseNet and PoseNet + PosePrior are both pretrained on different datasets and finetuned on Stereo dataset.

test images that have all predicted joint errors within a given distance threshold from the ground truth. ### B. 2D/3D Pose From Color Image 1) Method: We use the network architecture in CHP3D [^3] for 2D/3D hand pose estimation from a single color image. The network consists of three blocks, HandSegNet, PoseNet and PosePrior network. HandSegNet is a segmentation network to segment hand mask. With the hand mask predicted by HandSegNet, the input image is cropped and normalized in size, and fed to PoseNet to get the score maps of 2D hand joints. The PoseNet adopts a cascaded encoder-decoder network similar to [^33] to predict score map of the input image. PosePrior recovers 3D hand joints using the score map of 2D hand joint as input. 2) Training: Similar to CHP3D [^3], we first initialize the first 16 layers of PoseNet network with pretrained weights of [^33] and all the other layers with random weights, and train the PoseNet on the Stereo dataset [^8] and the synthetic dataset. Following [^3], we use the score map predicted by PoseNet to train PosePrior. We keep PoseNet fixed and train on Stereo hand pose dataset (Stereo) [^8]. To highlight the difference between physical-based rendering and unrealistic OpenGL rendering techniques, we train on PBRHand + Stereo, PBRHand-OpenGL + Stereo and RHD [^3] + Stereo. The networks are trained with a batch size of 16 and using ADAM solver. The learning rate is 0.0001 for the first 10K iteration, 0.00001 for the following 10K iterations and 0.000001 until the end. 3) Experiment: We train with the code released by [^3] and test on the Stereo dataset [^8]. Firstly, we compare the performance of 2D keypoint estimation using PoseNet trained on our PBRHand dataset, PBRHand-OpenGL dataset and the RHD dataset. Secondly, we compare the performance of 3D joints using PoseNet + PosePrior. Fig. 9(a) and Table III show the performance of 2D pose estimation on the Stereo

TABLE III

PERFORMANCE OF 2D HAND POSE FROM COLOR IMAGE ON STEREO DATASET WITH DIFFERENT TRAINING DATASETS. '-' IN THE FIRST COLUMN MEANS NO PRETRAINING. AUC IS SHOWN IN PERCENTAGE POINTS

Pre-train Dataset	PoseNet
Pre-train Dataset	AUC(0-30pix.) $\uparrow$	EPE (pix.) $\downarrow$
-	76.1	8.732
RHD	79.9	7.500
PBRHand	80.8	6.467
PBRHand-OpenGL	76.9	8.203

TABLE IV

PERFORMANCE OF 3D POSE REGRESSION FROM COLOR IMAGE ON STEREO DATASET WITH DIFFERENT TRAINING DATASETS. '-' IN THE FIRST COLUMN MEANS NO PRETRAINING. AUC IS SHOWN IN PERCENTAGE POINTS

Pre-train Dataset	PoseNet + PosePrior
Pre-train Dataset	AUC(20-50mm) $\uparrow$	EPE (mm) $\downarrow$
-	97.9	9.142
RHD	98.7	8.626
PBRHand	99.3	7.716
PBRHand-OpenGL	97.8	9.502

dataset with different training settings. Fig. 9(b) and Table IV show the performance of 3D pose from color image on Stereo dataset with different training settings. We observe that the EPE and AUC (20-50mm) with the 3D pose model pretrained on our PBRHand dataset are better than the model without pretrain, the models pretrained on RHD dataset [^3] and PBRHand-OpenGL dataset. Moreover, the 2D hand pose with the model pretrained on PBRHand is also better than the model pretrained on RHD and PBRHand-OpenGL. Fig. 10 shows 2D hand pose results on Stereo dataset with different training datasets. We can see that the 2D hand pose with the model --- ![Image](images/be4cf0164de998550c069e9f222f75734815067b109c8c6cf574bb7dcfbc8407.jpg)

(a) Input image

(b) Hand pose with model pretrained on RHD (c) Hand pose with model pretrained on PBRHand-OpenGL (d) Hand pose with model pretrained on PBRHand

Fig. 10. Hand pose results on Stereo dataset with different training datasets. The pretrained model on PBRHand provides more reasonable hand skeleton joint structure.

TABLE V

PERFORMANCE OF 2D HAND POSE FROM COLOR IMAGE ON MPII+NZSL DATASET USING POSENET TRAINED UNDER DIFFERENT TRAINING DATASETS. '-' IN THE FIRST COLUMN MEANS NO PRETRAINING. AUC IS SHOWN IN PERCENTAGE POINTS

Pre-train Dataset	PoseNet
Pre-train Dataset	AUC(0-0.5 hand size) $\uparrow$	EPE (pix.) $\downarrow$
-	65.5	12.197
RHD	76.1	12.197
PBRHand	76.7	11.803
PBRHand-OpenGL	75.2	12.708

pretrained on PBRHand provides reasonable hand skeleton joints and aligns better with the hand foreground than those with the models pretrained on RHD and PBRHand-OpenGL. We also test the effect of our PBRHand dataset on MPII+NZSL hand pose dataset [^21]. MPII+NZSL only contains 2D hand pose annotations, and we evaluate the effect of different synthetic datasets on 2D hand pose using PoseNet. Fig. 11 and Table V show the performance of 2D pose estimation on the MPII+NZSL dataset with the PoseNet models trained on different datasets. We observe that the 2D hand pose with the model trained on PBRHand is better than those with the model without pretraining, the models pretrained on RHD dataset and PBRHand-OpenGL dataset. Therefore, our PBRHand dataset synthesized by physically-based rendering improves the performance of color-based hand pose estimation upon the state-of-the-art method. ### C. Depth Estimation From Color Image 1) Method: We utilize a stacked hourglass network to regress depth estimation from color image similar to human depth estimation task in [^34]. The network formulates the ![Image](images/879f872fd4671fb8d2ecbd9d1b0192ba3291f26355c650860e1787fcbdda19a7.jpg)

Fig. 11. 2D hand pose results with PoseNet pretrained on different datasets and finetuned on MPII+NZSL dataset.

TABLE VI

PERFORMANCE OF DEPTH ESTIMATION FROM COLOR IMAGE (IN mm) ON STEREO DATASET WITH DIFFERENT TRAINING DATASETS

Training Data	Stereo	RHD+Stereo	PBRHand+Stereo
MAE $\downarrow$	5.877	4.571	4.541

depth estimation from a color image as a pixel-wise regression problem. Following the image preprocessing in [^34], we first normalize the depth values of hand foreground pixels to $[-1, 1]$. We find the 3D center of the mass (CoM) of the hand region using the depth map, then the depth map is normalized by subtracting CoM and divided by a constant 100. The network adopts convolutional layers with residual connections and 3 stacked ‘hourglass’ modules, and each followed module takes the prediction of the previous module as input. 2) Training: The training process consists of two stages, pretraining and finetuning. We pretrain the model on the synthetic datasets (PBRHand and RHD), then finetune the pretrained model on the Stereo pose dataset [^8] similar to Zhang et al. [^5]. We use RMSProp optimizer to train and finetune the model. At the pretraining stage, the initial learning rate is set to $5 \times 10^{-4}$, and it will be smoothly decayed to 0.9 scale every 80k steps. We train 140k steps at pretraining stage and finetuning stage. We adopt data augmentation such as rotation, adding noises and brightness adjustment. The batch size is set to 16. 3) Experiment: In order to verify the effect of PBRHand on depth estimation, we compare the depth recovery performance on the Stereo dataset [^8] using models trained on our PBRHand dataset, RHD and Stereo datasets. Table VI shows the comparison of depth estimation on the Stereo dataset. We can see that the model trained on PBRHand+Stereo achieves the best performance. Fig. 12 shows several qualitative results. We can see that the model pretrained with our PBRHand dataset recovers more accurate depth images. ### D. 3D Hand Pose from Depth Image In this experiment, we aim to compare the effect of synthetic training datasets with our scanned hand models in PBRHand and a parametric hand model MANO [^35] on 3D hand pose --- ![Image](images/71fa18e48e28d47e778608033cb278d9071f493534bf12fc1f56aae5a2187c50.jpg) 2.7072

(a)

3.0745

(b)

(c)

Fig. 12. Depth estimation from color image on stereo dataset. (a)(b)(c) show the predicted depth and error map of predicted depth using models pretrained on our PBRHand dataset, pretrained on RHD, and without pretraining, respectively. The model pretrained with our PBRHand dataset recovers more accurate depth images.

estimation. Since the original MANO model does not contain photo-realistic textures to support color based hand pose related tasks, we conduct the comparison using 3D hand pose estimation from depth. During the comparison, we generate two synthetic training datasets PBRHand-Sub and MANO, which both consist of 10 hand models and 1840 hand pose. The PBRHand-Sub dataset, a subset of our full PBRHand dataset, adopts scanned hand shape models, and the hand shape models in the MANO dataset is generated via the MANO hand model [^35]. 2) Training: We conduct our comparison experiments on NYU dataset [^16]. To highlight the difference between scanned hand models and MANO hand models for 3D hand pose estimation, we train DenseReg [^36] from scratch on NYU dataset, PBRHand-Sub + NYU dataset, and MANO + NYU dataset. The network are trained with a batch size of 40 and using ADAM solver. The initial learning rate is 0.001, and the exponential decay rate of the momentum is set to $\beta_{1} = 0.5$. We do not use data augmentation during the training process. 1) Method: We use the network architecture in DenseReg [^36] for 3D hand pose estimation from a single depth. We choose DenseReg [^36] as the baseline model, because DenseReg is one of the state-of-the-art networks with the released code. The network adopts both the 2D and 3D properties of a depth to recover 3D hand pose via pixel-wise regression. 3) Experiment: We train with the code released by [^36] and test on the NYU hand dataset. We get three DenseReg models trained on NYU dataset, PBRHand-Sub + NYU dataset, and MANO + NYU dataset, and compare their 3D hand pose ![Image](images/832c326e4cff1bbd73e186d461cf7713092d0a3a7a56c880369192e37012a96a.jpg)

Fig. 13. 3D hand pose results with DenseReg trained on NYU dataset, PBRHand-Sub + NYU dataset, and MANO + NYU dataset.

TABLE VII

PERFORMANCE OF 3D HAND POSE FROM DEPTH IMAGE ON NYU DATASET WITH DIFFERENT TRAINING DATASETS. AUC IS SHOWN IN PERCENTAGE POINTS

Dataset	NYU	MANO + NYU	PBRHand-Sub + NYU
Mean error (mm) $\downarrow$	12.06	12.55	11.75
AUC(0-80mm) $\uparrow$	64.5	63.4	66.5

estimation performance. Fig. 13 and Table VII compare the 3D hand pose results on NYU dataset with different training datasets. The model trained with PBRHand-Sub + NYU dataset achieves better performance than the model trained with MANO + NYU dataset and the model trained on NYU dataset, which indicates that our hand model is more realistic than MANO and the rendered data is more beneficial to the pre-training. ### V. MULTI-TASK HAND POSE ESTIMATION FROM COLOR With the help of our dataset, we further propose a multi-task deep neural network for hand pose estimation, which jointly predicts depth and hand joints in a cascaded fashion. We show how our multi-task model and our synthetic dataset compare with the state-of-the-art models pretrained with other synthetic datasets. Fig. 14 illustrates the basic structure of our network. In the first stage, the network predicts a depth map from input hand image. And in the second stage, the network uses latent features from first stage to predict 3D hand joints. We select depth as an auxiliary task based on the fact that depth is closely related to 3D hand pose estimation [^2]. ### A. Data Preparation We denote the training samples as $\{(I_i, D_i, J_i)\}_{i=1}^N$, where $I_i$ is the i-th detected hand region, $D_i$ and $J_i$ are the corresponding hand depth map and 3D hand joint locations in the camera's coordinate system. Following commonly used data normalization methods [^3], the input color image is normalized to $[-1, 1]$. Considering the sizes of human hands, we choose a constant value $a = 100$ mm to normalize depth and 3D --- ![Image](images/5cdf4299f0bfb3c778e71de1f088ad4b55b2fc2498a0674bfb0ef197a50b1db1.jpg)

Fig. 14. Our cascaded multi-task network for hand pose estimation from a single color image. The network uses cropped and resized hand regions as input. Our network has two stages, the first stage for depth recovery, and the second stage for hand pose recovery, which iteratively generates depth and hand pose.

hand joints. The depth image is first subtracted by the average depth of the hand region and then normalized with $\alpha$. For 3D hand joints, we first find the 3D center of the mass (CoM) of the hand region using the depth map. Each hand joint is then normalized by subtracting CoM and divided by $\alpha$. This CoM is only used to crop the hand but not directly used as input of the network in anyway. The network output is a 3D pose relative to a CoM, and the CoM is only required to compare with the ground truth. Hereafter, we denote $D_i$ and $J_i$ to be the normalized depth map and joints, respectively. ### B. Cascaded Multi-Task Network Our network takes a color image with the resolution of $128 \times 128 \times 3$ as input. We first use several convolutional blocks to extract image features. After that, in the depth prediction stage, we use an U-net [^37] to generate latent depth features. U-net is well known for its capability to extract both low-level and high-level features and widely used in segmentation, detection and pixel-wise regression. Then the depth features are passed to several convolutional blocks to generate depth map. In the joint prediction stage, the network first concatenates image features and depth features. And then, another U-net is employed to fuse the input features. Finally, after several convolutional blocks, downsampling and fully connected layers, the network outputs normalized 3D hand joint positions. Our network predicts depth and joints in a cascaded fashion, where joints depends on not only input image features but latent depth features as well. More details about the network can be found in the appendix. ### C. Loss Functions In the training phase, the total loss function for the proposed cascaded network is defined as: $$ L_{t o t a l}(\hat{J},\hat{D})=L_{p o s e}(\hat{J},J^{*})+\lambda_{d}L_{d e p t h}(\hat{D},D^{*}) $$ where the loss $L_{total}$ is a combination of hand pose loss $L_{pose}$ and depth recovery loss $L_{depth}$. The parameter $\lambda_{d}$ is the weight of $L_{pose}$ and $L_{depth}$, and set to 1.0. 1) Loss Function for Hand Pose: We use Euclidean distance to evaluate the difference between the ground truth and estimated 3D hand joints. $$ L_{pose}(\hat{\boldsymbol{J}},\boldsymbol{J}^{*})=\frac{1}{n}\sum_{i=1}^{n}||\hat{\mathbf{j}}_{i}-\mathbf{j}_{i}^{*}|| $$ where $\hat{J} = \{\hat{\mathbf{j}}_i\}$ and $J^* = \{\mathbf{j}_i^*\}$ are the predicted joint positions and ground truth joints, respectively, n is the total number of joints, which is typically 21 in most datasets. 2) Loss Function for Depth Recovery: Similar to the image regression work [^38], we use $L_{1}$ norm to evaluate the loss between the ground truth and estimated depth maps on the hand foreground region. $$ L_{d e p t h}(\hat{\boldsymbol{D}},\boldsymbol{D}^{*})=\frac{1}{m}\sum_{i=1}^{m}|\boldsymbol{M}\odot(\hat{\mathbf{d}}_{i}-\mathbf{d}_{i}^{*})| $$ where $\hat{D} = \{\hat{\mathbf{d}}_i\}$ and $D^* = \{\mathbf{d}_i^*\}$ are the predicted and ground truth depth maps, respectively. M is the hand foreground binary mask, m is the number of valid pixels in the hand foreground binary mask, and $\odot$ represents element-wise multiplication. ### D. Implementation Details Our method focuses on hand pose estimation, where hand regions are detected and cropped following the way in [^39], and the cropped color images and depth maps are resized to $128 \times 128$ pixels. During the training stage, we perform data augmentation on the training datasets using 2D rotation, scaling for the hand region, depth image and ground truth hand joints. We use in-plane rotation around z-axis of the camera's coordinate system. The in-plane rotation angle is randomly sampled from the interval $[-5, 5]$ degrees. We keep the aspect and scale hand color images with factors randomly chosen from [0.84, 1.16]. Our implementation is based on Tensorflow [^40]. We use RMSProp optimizer with 0.001 learning rate, 0.9 weight decay. We use 16 batch size and train on our PBRHand dataset for 10 epochs and finetune on the training set of each evaluating dataset for 20 epochs. ### VI. EXPERIMENT In this section, we first compare our multi-task hand pose estimation model with state-of-the-art, then conduct ablation study on our method. ### A. Comparison With State-of-the-Art Methods We first evaluate our cascaded multi-task network and compare with state-of-the-art methods on the Stereo dataset (Fig. 15 and Table VIII). Our network is pretrained on --- ![Image](images/3487632db44d8c1d033cf9a4c8b8dcc370a2daa270c371a217338e11ef2ab29c.jpg)

Fig. 15. Comparison of different methods on stereo dataset. Our method outperforms all other methods and achieves the highest AUC.

TABLE Ⅷ

EPE AND AUC (BETWEEN 20 - 50mm) EVALUATION ON RHD AND STEREO DATASETS. OUR CASCADED NETWORK WITH PRETRAINING ON OUR DATASET SHOWS CONSISTENTLY BETTER PERFORMANCE THAN OTHERS. OURS: OUR CASCADED NETWORK. AUC IS SHOWN IN PERCENTAGE POINTS

	RHD		Stereo
	EPE $\downarrow$	AUC $\uparrow$	EPE $\downarrow$	AUC $\uparrow$
CHP3D+RHD [^3]	35.60	67.0	12.2	94.80
Weakly-sup [^7]	-	88.7	-	99.4
Spurr [^41]	-	84.9	-	98.6
Ours+RHD	17.35	88.93	7.03	99.68
Ours+PBRHand	17.12	90.39	6.39	99.79
Ours wo pretrain	17.35	88.93	7.07	99.62

our PBRHand dataset, and then finetuned on the Stereo training set. Weakly-sup [^7], Latent2.5D [^32], Spurr [^41], Mueller et al. [^4] and CHP3D [^3] are the state-of-the-art color-based hand pose methods using the same experiment setting (i.e. the same training and testing datasets) as our method. For Stereo dataset, the area under the curve (AUC) of our network are higher than the results of the compared methods, Weakly-sup [^7], Latent2.5D [^32], CHP3D [^3], Spurr [^41], and Mueller [^4]. More evaluations can be found in the supplementary document. We then use the synthetic RHD dataset to further verify the effectiveness of both the synthetic data and our model. Table VIII shows the comparison results. To compare on the dataset, we pretrain our cascaded model with RHD and our PBRHand, respectively. We use both testing set from RHD and Stereo for evaluation. We first train models on RHD or our dataset, and then finetune for the Stereo dataset. For Stereo dataset, the model pretrained on our PBRHand (6.39mm) achieves better performance than the model pretrained on RHD (7.03mm), which shows our synthetic dataset contributes more to the pretraining. To compare about the models, we pretrain CHP3D [^3] and our model using RHD, finetune and test on Stereo datasets. Again, our model (7.03mm) achieves an error ## TABLE IX EPE AND AUC (BETWEEN 0-100mm) ON DEXTER+OBJECT DATASETS. OUR CASCADED NETWORK WITH PRETRAINING ON OUR DATASET SHOWS CONSISTENTLY BETTER PERFORMANCE THAN OTHERS. OURS: OUR CASCADED NETWORK. ** MEANS THE EPE AND AUC FOR THE ABSOLUTE 3D POSES AS LATENT 2.5D [^32]. AUC IS SHOWN IN PERCENTAGE POINTS

	EPE $\downarrow$	AUC $\uparrow$
Mueller [^4]	-	56.0
Latent 2.5D [^32]	45.54 $^{*}$	57.00 $^{*}$
Ours	43.31	57.20

TABLE X

COMPARISON OF MEAN JOINT ERRORS (IN mm) ON HANDS-2019 ONLINE CHALLENGE [^31]. OUR CASCADED NETWORK PRETRAINED ON PBRHAND PERFORMS BETTER THAN OTHERS. OURS: OUR CASCADED NETWORK

	EXTRAP. $\downarrow$	INTERP. $\downarrow$	OBJECT $\downarrow$	SHAPE $\downarrow$
User:yhasson [^42]	38.42	7.38	31.82	15.61
User:lin84[^43]	31.51	19.15	30.59	23.47
Ours	36.70	14.23	40.51	32.88
Ours+PBRHand (User:Fractality)	28.81	6.61	22.88	20.76
Ours+RHD	30.31	11.24	30.59	24.63

5.17mm lower than [^3] (12.2mm), and also performs better than Weakly-sup [^7] and Spurr [^41]. We also evaluate the performance on RHD testing set with CHP3D + RHD, Ours + RHD, Ours + PBRHand. As shown in Table VIII, the 3D EPE with Ours + RHD (17.35mm) and Ours + PBRHand (17.12mm) is 18.25mm and 18.48mm lower than CHP3D + RHD (35.6mm). We also evaluate our method on the Dexter+Object dataset and compare to the state-of-the-art Latent 2.5D [^32] and Mueller [^4]. Table IX shows the numerical evaluations. We can observe that our method trained on RHD+Stereo performs better than Mueller [^4] (Ours AUC:0.572, Mueller AUC:0.560). Since Latent 2.5D has not released the code yet, we use the reported scores of the model which is trained on RHD+Stereo. We also train our model on RHD+Stereo, and we observe that our method achieves comparable performance to Latent 2.5D on Dexter+Object dataset (Ours EPE:43.31, AUC:0.572, Latent 2.5D EPE:45.54, AUC:0.570). Finally, we also evaluate our method on Hands-2019 online challenge [^31], which is especially challenging due to occlusions of object. For Hands-2019, we submit our hand pose results online, and adopt the evaluation protocol in [^31] to calculate mean joint errors for four splits of the test set of Hands-2019 (whether the test split has hand shapes or objects present in the training set), i.e. EXTRAP, INTERP, OBJECT, and SHAPE (refer to [^31] for details). EXTRAP, OBJECT and SHAPE are three data splits to verify the generalization power of the model for hand shapes and objects, and EXTRAP is the key test split for evaluation in Hands-2019 competition. Table X shows the comparison to the state-of-the-art methods [^42], [^43] (submitted by 'User:yhasson' at 2019-10-09 and 'User:lin84' at 2019-10-07). Their methods train networks on a large-scale synthetic dataset of hands grasping objects [^42] or hands with neighboring objects [^43]. ---

TABLE XI

COMPARISON OF 3D EPE (IN mm) AND AUC (BETWEEN 20mm TO 50mm) ON STEREO AND RHD DATASETS USING OUR SINGLE TASK AND CASCADED MULTI-TASK METHODS WITH OR WITHOUT PRETRAIN. AUC IS SHOWN IN PERCENTAGE POINTS

	Stereo hand dataset		RHD dataset
	EPE $\downarrow$	AUC $\uparrow$	EPE $\downarrow$	AUC $\uparrow$
single wo pretrain	8.93	98.70	18.77	86.81
single w pretrain	7.13	99.64	18.21	87.90
cascade wo pretrain	7.07	99.62	17.35	88.93
cascade w pretrain	6.39	99.79	17.12	90.39

TABLE XII

COMPARISON OF MEAN 3D EPE (IN mm) AND AUC (BETWEEN 0mm TO 100mm) ON DEXTER+OBJECT DATASET [^10]. WE USE DIFFERENT TRAINING DATA COMBINATION AS WELL AS OUR SINGLE TASK AND CASCADED MULTITASK NETWORKS. FROM THE TABLE WE CAN OBSERVE THAT OUR CASCADED NETWORK TRAINED ON PBRHAND ONLY PERFORMS THE BEST (HIGHEST AUC AND LOWEST EPE). AUC IS SHOWN IN PERCENTAGE POINTS

	single		cascade
	EPE $\downarrow$	AUC $\uparrow$	EPE $\downarrow$	AUC $\uparrow$
Stereo	50.04	51.40	52.35	48.50
RHD	49.05	51.20	48.80	51.40
RHD + Stereo	51.78	50.40	43.31	57.20
PBRhand	43.30	57.40	40.70	59.53

Compared to the synthetic data used in [^42], [^43], our PBR-Hand dataset does not contain objects, thus it has domain gap to Hands-2019 dataset. Our method pretrained on PBRHand ('User:Fractality') can still achieve better performance on the data splits of EXTRAP, INTERP, and OBJECT than [^42]. Compared to [^43], our method performs better in all the four test data splits. We also observe that our cascaded network performs better when pretrained on our PBRHand compared to RHD dataset. These results indicate the generalization ability our model for hand shape and object. ### B. Ablation Study To further understand the task and the network, we conduct several comparison experiments using our multi-task network and our single task network with/without pretrain. 1) Does Pre-Training Help?: From Table XI, we can observe that by pretrained on our PBRHand dataset the average errors of hand joints with cascaded network (with finetune) drop by 0.68mm and 0.23mm for Stereo and RHD datasets. From Table XII, considering the same network with different training data, we can observe that the networks trained on PBRHand provide consistently better performance on Dexter+Object dataset. We can see that pretraining on our PBRHand dataset is an effective way to enhance the performance of color based hand pose estimation for single task or multi-task networks. 2) Does Multi-Task Help?: In order to evaluate the multi-task networks for hand pose estimation, we compare the baseline single-task network, which shares the same network architecture as the cascaded task network. The difference is ![Image](images/192551951813c399027b5090cdbbe73b4fdc3ffa4bd1dcad480c93e682c53637.jpg)

(a) Input image

(b) Ground truth pose

Fig. 16. Examples of results with our method on stereo dataset. (a) input image. (b) the ground truth pose. (c) hand pose with our cascaded network. (d) hand pose with single joint network w/o depth cascade.

that the single task network does not adopt auxiliary depth recovery task and does not use the depth loss. Fig. 16 and Table XI show the accuracy of 3D hand joints with and without multi-task supervisions. We can observe that results with our cascaded network is consistently better than results with single task network. For Stereo and RHD, the average errors of hand pose using cascaded network with pretraining drop about 2.72mm, 1.09mm compared to the single task network with pretraining, and the average errors of the cascaded network without pretraining drop about 1.86mm, 1.42mm, compared to the single task network without pretraining. For Dexter+Object dataset (See Table XII), by comparing networks trained on the same datasets, we can observe that our cascaded multi-task network performs better than others. First, it is generally better than the single joint task network (except trained only on Stereo dataset, which might be caused by the limited size and diversity of the dataset). One possible explanation is that multi-task cascaded training can provide more cues (e.g. depth features) and improve the feature extraction on early layers (color image features in our network). We see that our cascaded multi-task network is useful for the training with and without pretraining. 3) Feature Cascade vs. Input/Output Cascade: The proposed cascaded network uses image features and depth features instead of input image and output depth. We evaluate the effects of different cascade inputs on hand pose estimation performance. We compare the following settings: $RGB + D$, $RGB_{feature} + D$, $RGB + D_{feature}$, D, $D_{feature}$ and the proposed $RGB_{feature} + D_{feature}$. RGB and $RGB_{feature}$ mean the input color image and the feature extracted with first three convolution layers of our network. D and $D_{feature}$ --- ![Image](images/d20e864df5de59b61d1066a1b95183fb5e0b1e2644c6c35fd02262b7f6033179.jpg)

Fig. 17. Qualitative results on stereo and RHD datasets. From left to right: input color image and the predicted joint; ground truth depth; predicted depth image; difference between ground truth and the prediction.

are the recovered depth with depth recovery network and the depth-aware feature extracted with the first U-net in Fig. 14. Table XIII shows that feature cascade would produce the best result for joint prediction. More ablation studies can be found in the supplementary document. ### C. Runtime and Qualitative Experiments Our network contains about 26.7M parameters, and the testing time is 180 fps on a computer with a NVIDIA Titan X GPU, which also has the potential to run on embedding systems in real-time. Fig. 17 shows our results on Stereo and RHD dataset, with our model pretrained on PBRHand and finetuned on Stereo and RHD training dataset. More qualitative results ## TABLE XIII CASCADE EVALUATION. WE TRIED DIFFERENT COMBINATION OF DATA THAT CASCADED FOR JOINT ESTIMATION. $RGB_{feature} + D_{feature}$ PRODUCES THE BEST RESULT, WHICH IS THE CHOOSEN NETWORK STRUCTURE. ONLY PASS D TO JOINT NETWORK WOULD IMPROVE DEPTH PREDICTION, BUT CANNOT PRODUCE THE BEST JOINT PREDICTION. AUC IS SHOWN IN PERCENTAGE POINTS

	Depth		Joint
Cascade type	RMSE $\downarrow$	MAE $\downarrow$	EPE $\downarrow$	AUC $\uparrow$
RGB + D	9.08	7.59	8.10	99.17
RGB $_{feature}$ + D	9.52	7.41	8.13	99.32
RGB + D $_{feature}$	15.46	12.82	8.55	98.96
D	8.42	6.63	7.78	99.47
D $_{feature}$	8.93	6.94	7.13	99.59
RGB $_{feature}$ + D $_{feature}$	8.68	6.63	7.07	99.68

using captured real data can be found in the supplementary document. --- ![Image](images/b505bb37a1c3e8296526101f0c10eae78084b5bf4b240c883cf3eac219789302.jpg) ![Image](images/09747b8e0eb4124893831d6636171cd4acf535daefb833b4a88ca0a60135e610.jpg)

Fig. 18. Full details of our multi-task cascaded network. Kernel size of the convolutional layers and deconvolutional layers is 3. Alpha value of the Leaky RELU [^44] activation function is 0.2. To avoid zero patterns caused by up-sampling, each deconvolutional layer is followed by two convolutional layers.

### VII. CONCLUSION In this paper, we construct a large-scale hand pose dataset with photo-realistic rendering color images and various ground truths. Based on the dataset, we first verify on three hand pose related tasks, and find that pretraining on our photo-realistic rendering dataset can enhance the performance of the state-of-the-art models. With the help of our dataset, we further design a cascaded multi-task network to recover 3D hand pose from a color image with an intermediate depth supervision. We find that our dataset benefits the model pretraining by providing various ground truths and our cascaded network can improve the performance, both enhancing the pose estimation results. ## APPENDIX ## DETAILS OF OUR NETWORK ARCHITECTURE Fig. 18 illustrates the details for our network architecture. ## REFERENCES [^1]: A. Erol, G. Bebis, M. Nicolescu, R. D. Boyle, and X. Twombly, "Vision-based hand pose estimation: A review," Comput. Vis. Image Understand., vol. 108, nos. 1–2, pp. 52–73, Oct. 2007. [^2]: S. Yuan, Q. Ye, B. Stenger, S. Jain, and T.-K. Kim, "BigHand2.2M benchmark: Hand pose dataset and state of the art analysis," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jul. 2017, pp. 2605–2613. [^3]: C. Zimmermann and T. Brox, “Learning to estimate 3D hand pose from single RGB images,” in Proc. IEEE Int. Conf. Comput. Vis. (ICCV), Oct. 2017, pp. 4913–4921. [^4]: F. Mueller, D. Mehta, O. Sotnychenko, S. Sridhar, D. Casas, and C. Theobalt, "Real-time hand tracking under occlusion from an egocentric RGB-D sensor," in Proc. IEEE Int. Conf. Comput. Vis. (ICCV), Oct. 2017, pp. 1154–1163. [^5]: Y. Zhang et al., “Physically-based rendering for indoor scene understanding using convolutional neural networks,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jul. 2017, pp. 5287–5295. [^6]: J. Shi, Y. Dong, H. Su, and S. X. Yu, “Learning non-Lambertian object intrinsics across ShapeNet categories,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jul. 2017, pp. 1685–1694. [^7]: Y. Cai, L. Ge, J. Cai, and J. Yuan, "Weakly-supervised 3D hand pose estimation from monocular RGB images," in Proc. Eur. Conf. Comput. Vis., 2018, pp. 666–682. [^8]: J. Zhang, J. Jiao, M. Chen, L. Qu, X. Xu, and Q. Yang, "3D hand pose tracking and estimation using stereo matching," 2016, arXiv:1610.07214. [Online]. Available: [^9]: C. Zimmermann, D. Ceylan, J. Yang, B. Russell, M. J. Argus, and T. Brox, "FreiHAND: A dataset for markerless capture of hand pose and shape from single RGB images," in Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), Oct. 2019, pp. 813–822. [^10]: S. Sridhar, F. Mueller, M. Zollhöfer, D. Casas, A. Oulasvirta, and C. Theobalt, “Real-time joint tracking of a hand manipulating an object from RGB-D input,” in Proc. Eur. Conf. Comput. Vis., 2016, pp. 294–310. [^11]: T. Sharp et al., “Accurate, robust, and flexible real-time hand tracking,” in Proc. 33rd Annu. ACM Conf. Hum. Factors Comput. Syst., Apr. 2015, pp. 3633–3642. [^12]: A. Wetzler, R. Slossberg, and R. Kimmel, “Rule of thumb: Deep derotation for improved fingertip detection,” 2015, arXiv:1507.05726. [Online]. Available: [^13]: F. Gomez-Donoso, S. Orts-Escolano, and M. Cazorla, “Large-scale multiview 3D hand pose dataset,” 2017, arXiv:1707.03742. [Online]. Available: 14] Blender. (2018). Blender. [Online]. Available: [^15]: W. Jakob. Mitsuba Renderer. Accessed: 2018. [Online]. Available: --- [^16]: J. Tompson, M. Stein, Y. Lecun, and K. Perlin, “Real-time continuous pose recovery of human hands using convolutional networks,” ACM Trans. Graph., vol. 33, no. 5, pp. 1–10, Sep. 2014. [^17]: D. Tang, H. J. Chang, A. Tejani, and T.-K. Kim, "Latent regression forest: Structured estimation of 3D articulated hand posture," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2014, pp. 3786–3793. [^18]: M. Oberweger, G. Riegler, P. Wohlhart, and V. Lepetit, “Efficiently creating 3D training data for fine-hand pose estimation,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2016, pp. 4957–4965. [^19]: X. Sun, Y. Wei, S. Liang, X. Tang, and J. Sun, "Cascaded hand pose regression," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2015, pp. 824–832. [^20]: J. Zhang, J. Jiao, M. Chen, L. Qu, X. Xu, and Q. Yang, "A hand pose tracking benchmark from stereo matching," in Proc. IEEE Int. Conf. Image Process. (ICIP), Sep. 2017, pp. 982–986. [^21]: T. Simon, H. Joo, I. Matthews, and Y. Sheikh, “Hand keypoint detection in single images using multiview bootstrapping,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), vol. 2, Jul. 2017, pp. 1145–1153. [^22]: N. Mayer et al., “A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2016, pp. 4040–4048. [^23]: N. Mayer et al., “What makes good synthetic training data for learning disparity and optical flow estimation?” Int. J. Comput. Vis., vol. 126, no. 9, pp. 942–960, Sep. 2018. [^24]: H. Su, C. R. Qi, Y. Li, and L. J. Guibas, “Render for CNN: Viewpoint estimation in images using CNNs trained with rendered 3D model views,” in Proc. IEEE Int. Conf. Comput. Vis. (ICCV), Dec. 2015, pp. 2686–2694. [^25]: G. Rogez and C. Schmid, “MoCap-guided data augmentation for 3D pose estimation in the wild,” in Proc. Adv. Neural Inf. Process. Syst., 2016, pp. 3108–3116. [^26]: F. Mueller et al., “GANerated hands for real-time 3D hand tracking from monocular RGB,” 2017, arXiv:1712.01057. [Online]. Available: [^27]: Artec. Artec Eva. Accessed: 2018. [Online]. Available: [^28]: B. Amberg, S. Romdhani, and T. Vetter, “Optimal step nonrigid ICP algorithms for surface registration,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2007, pp. 1–8. [^29]: Wikipedia. UV Mapping. Accessed: 2018. [Online]. Available: [^30]: I. Baran and J. Popović, “Automatic rigging and animation of 3D characters,” ACM Trans. Graph., vol. 26, no. 3, p. 72, Jul. 2007. [^31]: Task 3 of Hands19 Challenge: RGB-Based 3D Hand Pose Estimation While Interacting With Objects. Accessed: 2019. [Online]. Available: [^32]: U. Iqbal, P. Molchanov, T. Breuel, J. Gall, and J. Kautz, “Hand pose estimation via latent 2.5 D heatmap regression,” in Proc. Eur. Conf. Comput. Vis., 2018, pp. 118–134. [^33]: S.-E. Wei, V. Ramakrishna, T. Kanade, and Y. Sheikh, “Convolutional pose machines,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2016, pp. 4724–4732. [^34]: G. Varol et al., “Learning from synthetic humans,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jul. 2017, pp. 109–117. [^35]: J. Romero, D. Tzionas, and M. J. Black, “Embodied hands: Modeling and capturing hands and bodies together,” ACM Trans. Graph., vol. 36, no. 6, pp. 1–17, Nov. 2017. [^37]: O. Ronneberger, P. Fischer, and T. Brox, "U-net: Convolutional networks for biomedical image segmentation," in Proc. Int. Conf. Med. Image Comput. Comput.-Assist. Intervent., 2015, pp. 234–241. [^36]: C. Wan, T. Probst, L. Van Gool, and A. Yao, “Dense 3D regression for hand pose estimation,” 2017, arXiv:1711.08996. [Online]. Available: [^38]: P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros, “Image-to-image translation with conditional adversarial networks,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jul. 2017, pp. 1125–1134. [^39]: M. Oberweger and V. Lepetit, "DeepPrior++: Improving fast and accurate 3D hand pose estimation," in Proc. IEEE Int. Conf. Comput. Vis. Workshops (ICCVW), Oct. 2017, pp. 585–594. [^40]: M. Abadi et al., “TensorFlow: Large-scale machine learning on heterogeneous distributed systems,” 2016, arXiv:1603.04467. [Online]. Available: [^41]: A. Spurr, J. Song, S. Park, and O. Hilliges, “Cross-modal deep variational hand pose estimation,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., Jun. 2018, pp. 89–98. [^42]: Y. Hasson et al., “Learning joint reconstruction of hands and manipulated objects,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2019, pp. 11807–11816. [^43]: L. Yang, S. Li, D. Lee, and A. Yao, “Aligning latent spaces for 3D hand pose estimation,” in Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), Oct. 2019, pp. 2335–2343. [^44]: A. L. Maas, A. Y. Hannun, and A. Y. Ng, “Rectifier nonlinearities improve neural network acoustic models,” in Proc. ICML, 2013. ![Image](images/4e8d61e837afe6dc7297b73656c9452bc6c637c4b212c835a903ef5acce527f8.jpg) Xiaoming Deng received the bachelor's and master's degrees from Wuhan University, in 2001 and 2004, respectively, and the Ph.D. degree from the Institute of Automation, Chinese Academy of Sciences (CAS), in 2008. After a two-year Postdoctoral training with the Institute of Computing Technology, CAS, he joined the Institute of Software, CAS, in 2010, where he is currently an Associate Professor. From 2012 to 2013, he was a Research Fellow with the National University of Singapore. His main research topics are in computer vision, and specifically related to 3D reconstruction, human motion tracking and synthesis, and deep learning. Yinda Zhang received the bachelor's degree from the Department of Automation, Tsinghua University, the master's degree from the Department of Electrical and Computer Engineering, National University of Singapore, under the supervision of Prof. P. Tan and Prof. Shuicheng Yan, and the Ph.D. degree from Princeton University, advised by Prof. Thomas Funkhouser. He is currently a Research Scientist at Google. He is currently working on 3D context model, 3D deep learning, and scene understanding. ![Image](images/af8aac729f979f12c169852801bda0140311b79bba9552c4059cdfd1fc50e6e7.jpg) ![Image](images/b1bc1d5bb32217293b7841c0fc57300d71b4dc5fe2b232cc5ccc155332588872.jpg) Jian Shi received the Ph.D. degree from the University of Chinese Academy of Sciences in 2017. He is currently an Assistant Professor with the Institute of Automation, Chinese Academy of Sciences. He works on computer vision and graphics problems, including intrinsic images and illumination estimation. ![Image](images/358ee401dce21c066399e18e12f7e6508d29001ad10eb7ed566817700d48d9f7.jpg) Yuying Zhu received the bachelor's degree from the University of Science and Technology of China in 2016. She is currently pursuing the master's degree with the Institute of Software, Chinese Academy of Sciences. Her main research interest is computer vision and human-computer interaction. ![Image](images/376505e5c2dd43c82b470ae2fdaddf58a0b2808341af94b9eb03116affec4368.jpg) Dachuan Cheng received the master's degree from the Institute of Software, Chinese Academy of Sciences. He is currently working with ByteDance. His main research interest is computer vision and scene illumination estimation. --- ![Image](images/5e6332f0bd473881c47a14b69fafc3a384e257554d108fa701d857b8a9e3a3d7.jpg)

Dexin Zuo received the bachelor's degree from Nankai University in 2017. He is currently pursuing the master's degree with the Institute of Software, Chinese Academy of Sciences. His main research interest is computer vision and human-computer interaction.

![Image](images/11db5ef03c79484c07c1717b4f84b5f93bc591c5f57e25648ed5c954d9474b7f.jpg) ![Image](images/31d9a464b05e883e3d6932a2d4c1d22a92e5619b12c5c380848df9b752c21528.jpg)

Liang Chang received the bachelor's and master's degrees from Wuhan University, in 2003 and 2005, respectively, and the Ph.D. degree from the Institute of Automation, Chinese Academy of Sciences, in 2008. She is currently an Associate Professor with the School of Artificial Intelligence, Beijing Normal University. Her research interests are computer vision and machine learning.

Zhaopeng Cui received the Ph.D. degree from Simon Fraser University in 2017. He was a Senior Researcher at ETH Zurich. He is currently a Research Professor with the College of Computer Science, Zhejiang University. His research interests include 3D mapping and localization, 3D scene understanding, and image and video editing.

![Image](images/235f510d6aa6c9c32bdaa118edc98ae794fcf9627c1e1c65d00ff8fca6db6148.jpg) Ping Tan (Senior Member, IEEE) received the bachelor's and master's degrees from Shanghai Jiao Tong University, China, in 2000 and 2003, respectively, and the Ph.D. degree from the Hong Kong University of Science and Technology in 2007. He was an Associate Professor with the National University of Singapore. He is currently an Associate Professor with the School of Computing Science, Simon Fraser University, Canada. His research interests include computer vision, computer graphics, and robotics. He served as an Area Chair for CVPR. Hongan Wang received the Ph.D. degree in computer science from the Institute of Software, Chinese Academy of Sciences. He is currently a Professor and the Director of the Beijing Key Laboratory of Human-Computer Interactions, Institute of Software, Chinese Academy of Sciences. His research interest is real-time reasoning and intelligent user interaction. ![Image](images/0060618a556b9a30aae37e970a99a70eff04bfcab88097a3349a9949cd67a1df.jpg) SIGGRAPH, SIGGRAPH Asia, and IROS. He has served as an Editorial Board Member of the IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE (PAMI), the International Journal of Computer Vision (IJCV), the Computer Graphics Forum (CGF), and the Machine Vision and Applications (MVA).