# PatientHandNet: 3-D Open-Palm Hand Reconstruction From Sparse Multiview Depth Images Xinxin Dai $^{ID}$, Graduate Student Member, IEEE, Ran Zhao $^{ID}$, Pengpeng Hu $^{ID}$, and Adrian Munteanu $^{ID}$ Abstract—Accurately reconstructing 3-D hand shapes of patients is important for immobilization device customization, artificial limb generation, and hand disease diagnosis. Traditional 3-D hand scanning requires multiple scans taken around the hand with a 3-D scanning device. These methods require the patients to keep an open-palm posture during scanning, which is painful or even impossible for patients with impaired hand functions. Once multi-view partial point clouds are collected, expensive post-processing is necessary to generate a high-fidelity hand shape. To address these limitations, we propose a novel deep-learning method dubbed PatientHandNet to reconstruct high-fidelity hand shapes in a canonical open-palm pose from multiple-depth images acquired with a single-depth camera. The hand poses in the depth images may vary, hand movements are allowed, facilitating the 3-D scanning process in particular for patients with difficult conditions. The proposed method has strong operability since it is insensitive to the input pose, allowing for pose variations in the input depth images. We also proposed two novel datasets: a large-scale synthetic dataset to train our model and a real-world dataset with ground-truth hand biometrics extracted by an experienced anthropometrist. Extensive experimental results on the unseen synthetic data and real-world data demonstrate that the proposed method provides robust and easy-to-use hand shape reconstruction and outperforms the state-of-the-art methods in biometric accuracy terms. Index Terms—3-D hand reconstruction, 3-D hand shape, hand biometrics, multiscale features, multi-view depth processing, point cloud processing. ### I. INTRODUCTION RECONSTRUCTION of the 3-D hand shape plays an important role in hand-centric medical applications, such as hand immobilization device design [^1], [^2], [^3], artificial limb generation [^4], [^5], and osteoarthritis evaluation [^6], [^7], all of which usually require patients to open their hands for accurate anthropometric measurements [^8]. However, it is difficult for patients with impaired hand functions [^9], [^10] to keep an open palm due to the fact that their hands usually exhibit complex hand poses caused by tenosynovitis, fracture, or other types of diseases. Researchers also observed that different hand postures cause substantial changes in hand geometry [^11]. Therefore, there is a demand for reconstructing the patient's hand shape in an open-palm pose while not having access to such a pose during scanning. Researchers have attempted to reconstruct hand shapes from different signals, such as hand anthropometric measurements, RGB images, depth images, and point clouds. The measurement-based methods need to enter dozens of different dimensions into a hand sizing system for estimating hand shape [^1]. However, obtaining these measurement values highly relies on the expertise of the anthropometrist, which is expensive, time-consuming, and prone to errors. The RGB-based methods require 2-D to 3-D representations for predicting 3-D hand shapes from a single RGB image [^12], [^13], [^14]. Such methods may result in perspective distortions and scale ambiguity due to the lack of depth information. With the development of commodity depth cameras, depth images/point clouds can be easily obtained, encouraging researchers to devise low-cost hand scanners. These methods are mainly classified into two categories: nonparametric methods and parametric methods. Nonparametric methods correspond to traditional 3-D hand-scanning technologies, including single-camera and multi-camera scanning. Single-camera scanning [^15], [^16] moves a camera around a hand for data acquisition, which is time-consuming and may fail due to the fast movement of the camera. Multi-camera scanning [^17], [^18], [^19] allows multiple cameras to capture a hand from different views simultaneously but relies on the quality of external calibration and requires a professional operation, which makes it unportable. More importantly, nonparametric methods usually require the hand to be in a static open-palm pose during scanning. However, maintaining an open-palm pose may induce pain or maybe simply not possible for patients with certain conditions. Parametric methods are developed based on the parametric hand models, which usually factor out the hand using parameters that control the shape and the pose [^20]. Such methods mainly aim to fit the parametric hand models (e.g., SMPL-X) to the input data to obtain the optimal model parameters. Then an open-palm hand can be generated by setting the pose parameters to zero. However, --- such methods are not robust since a slight change in a parameter may result in a significant shape/pose variation. To address this issue, some researchers proposed to regress model vertices instead of parametric parameters [^21], [^22]. However, most of these works pay more attention to hand pose estimation while ignoring the accuracy of the shape. To address the issues mentioned above, we aim to devise an algorithm to reconstruct hand shapes in a canonical open-palm pose from sparse multi-view depth images. Specifically, we propose a novel method termed PatientHandNet to reconstruct a high-fidelity open-palm hand shape from four-depth images. Note that we do not restrict the hand pose from the four views to be the same. The patient does not necessarily need to adopt an open-palm pose nor is required to maintain a rigid pose in the four scans. This is illustrated in Fig. 1(a), where we collected four depth images of the hand from palmar, dorsal, ulnar, and radial views using a single commodity depth camera. These depth images cover the whole hand geometry with minimal overlap, which implicitly accounts for the spatial dependency between adjacent views. The main contributions in this article can be summarized as follows. 1) We proposed, to the best of our knowledge, the first deep learning-based method, dubbed PatientHandNet, for reconstructing a high-fidelity open-palm 3-D hand shape from four depth images captured by a single-commodity depth camera. The method allows for various pose variations among the camera shots. 2) We proposed a large-scale multi-view synthetic dataset with a wide variety of hand shapes and hand poses and corresponding ground truth hand shapes in a canonical open-palm pose. 3) We collected a novel real-world dataset by capturing 18 subjects (13 males and five females) via a structure sensor Mark I employed in an iPad and hired a professional anthropometrist to obtain the ground-truth hand biometrics. 4) Extensive experiments conducted on both synthetic and real-world data demonstrate that our method outperforms the state-of-the-art methods both qualitatively and quantitatively. The rest of the article is organized as follows. Section II briefly reviews related works that include non-parametric and neural parametric hand shape reconstruction. Section III describes the proposed method and proposed dataset. Section IV presents the experimental setup and comparison results among different methods on both synthetic data and real-world data. In Section IV-D, we discuss the effectiveness of different modules of our method via detailed ablation studies. The article is concluded in Section V. ### II. RELATED WORK ### A. Non-Parametric Hand Shape Reconstruction using multiple cameras fixed in different positions. The main advantage of multi-camera scanning is that it enables fast data acquisition and reduces the negative effect of involuntary movements by simultaneously acquiring the hand surface from different viewpoints. However, it highly depends on the quality of external calibration and needs a professional operation. In addition, it lacks mobility due to its expensive setup, configuration, and calibration procedures. In contrast, single-camera scanning is a more cost-effective alternative to multi-camera scanning due to its requirement of only one scanner. This scanner held in the hand moved around the hand to acquire the complete shape [^15], [^16]. However, for successful 3-D reconstruction, the acquisition procedure requires successive viewpoints to contain sufficient redundant information, resulting in time-consuming computation and potential failures due to the fast movement of the camera. As aforementioned, non-parametric methods can be classified into multi-camera scanning and single-camera scanning according to the number of employed sensors. Multi-camera scanning [^17], [^18], [^19], [^23] captures the hand surface. Apart from these limitations, nonparametric methods require the hand to be static and in an open-palm pose during scanning to provide accurate hand measurements; as mentioned, maintaining a rigid pose is impractical for regular patients and impossible for patients with impaired hand functions. To address these problems, we proposed a high-efficiency, posture-immune method which is improved as follows: 1) we only employ four depth images captured by a single camera and the hand poses of these depth images can be different and 2) our method outputs the hand shapes in an open-palm pose, which is essential for downstream applications such as hand biometrics extraction. ### B. Neural Parametric Hand Shape Reconstruction With the rapid development of deep learning, neural networks have been introduced to address the problem of 3-D hand reconstruction. By literature review, we observe that most existing methods focus on predicting the hand pose from RGB images [^24], [^25], [^26] or depth images [^27], [^28], [^29], [^30]. A limited number of researchers have attempted to reconstruct the hand shape, but they estimate the pose and shape of the hand simultaneously. Malik et al. [^31] proposed an end-to-end framework for hand shape reconstruction from a depth image, which embedded model-based hand pose and shape layers to generate 3-D joint positions and hand meshes. Although such a method can predict hand mesh directly from the depth image, it suffers from artifacts due to the difficulty in optimizing complex parameters. Chen et al. [^12] is a self-supervised 3-D hand reconstruction network, which not only predicts 3-D mesh but also outputs texture. Kong et al. [^32] proposed an identity-aware hand mesh estimation model by regressing the parameters of MANO, which can incorporate the identity information to calibrate the shape parameters. Yu et al. [^33] can find a balance between a parametric and nonparametric model to improve the accuracy of hand shape and pose. Luan et al. [^34] can predict detailed hand mesh from a single image using the proposed frequency decomposition loss. However, all of them adopt parameter regression of the MANO model [^35] to predict the hand pose and shape. Such techniques are less robust since a slight change in the parameters can significantly change the shape and pose. Park et al. [^36] and Chen et al. [^37] directly regressed the --- ![Image](images/c1ceab59111a0280500f783a2cd1ac6e6d1995b2c2ed82e7cdf0e9d93959dd03.jpg)
Fig. 1. Schematic of PatientHandNet for open-palm hand shape reconstruction. Our method consists of two steps, taking as input four partial point clouds from palmar view, dorsal view, ulnar view, and radial view of the hand and regressing the vertices of an open-palm hand. In the first step, VVTN consumes the four partial hand point clouds and generates four virtual complete hands in the same open-palm pose. The obtained four virtual complete hands are used to calculate transformation parameters between palmar view, ulnar view, radial view, and dorsal view, which enables to roughly align the input four partial hands to form a holistic representation of hand shape. In the second step, HSRN takes as input the four aligned partial point clouds and learns an aggregated feature through MMFA. Then the aggregated feature of MMFA is fed into the HVR module to regress the vertices of the open-palm hand shape in a coarse-to-fine manner.
3-D joint positions and mesh vertices of the hand model from RGB images. Park et al. [^36] can estimate hand mesh from an RGB image with occlusion and [^37] has lightweight stacked structures that can employ mobile to estimate hand shape from an RGB image. However, such RGB-based methods are easy to result in perspective distortions and scale ambiguity in the estimated outputs. To address the problems of perspective distortions and scale ambiguity, Malik et al. [^38] proposed a vertex regression-based method that effectively establishes a one-to-one mapping between the voxelized depth map and --- the voxelized hand model. However, the voxelized depth map includes redundant information, distracting the attention of the network from extracting valuable features [^39]. Unlike these methods, we focus on hand shape reconstruction by outputting the hand shapes in a canonical open-palm pose. Such a strategy will force the neural network to pay more attention to the shape instead of the pose. ### III. PROPOSED METHOD ### A. Problem Statement In this section, we formulate the problem in this study. Given a set of uncalibrated partial point clouds of a hand $\mathcal{X} = \{\mathbf{S}^n\}_{n=1}^N$, where $\mathbf{S}^n = \{s_i^n \in \mathbb{R}^3 | i = 1, 2, \ldots, I^n\}$ denotes a set of points with $I^n$ points captured from the nth view, N is the number of input partial point clouds and is set to 4 in this study (corresponding to four views of the hand: palmar, ulnar, dorsal, and radial views). Note that the poses of these partial hand point clouds can be the same as well different and the point numbers of each $\mathbf{S}^n$ can also be different. The goal is to devise a low-cost and easy-to-use method to reconstruct a high-fidelity open-palm hand mesh with J points $\mathcal{Y} = \{(y_j \in \mathbb{R}^3, e_z \in \mathbb{Z}^2) | j = 1, 2, \ldots, J, z = 1, 2, \ldots, Z\}$ from $\mathcal{X}$, where $y_i$ and $e_m$ are the vertices and edges of $\mathcal{Y}$, respectively. We proposed a two-step deep learning architecture for this task. First, a transformation network is trained to estimate the relative transformation relationship $\mathcal{T}$ between $\mathcal{X}$. $\mathcal{T}$ is used to align $\mathcal{X}$ into $\hat{\mathcal{X}}$, which is represented as $\hat{\mathcal{X}} = \mathcal{X}\mathcal{T}$. Then we use a shape reconstruction network to predict open-palm hand shape $\mathcal{Y}$ from $\hat{\mathcal{X}}$. Therefore, there establish two mappings $M_1 : \mathcal{X} \longmapsto \mathcal{T}$ and $M_2 : \hat{\mathcal{X}} \longmapsto \mathcal{Y}$. $\mathcal{T}$ estimated by $M_1$ can strengthen dependency between adjacent views, enabling $M_2$ to learn the more robust geometric and spatial representation of hand shape. To further improve the performance, we also propose a coarse-to-fine decoder in $M_2$ to utilize local information to refine the vertex coordinates of the hand. Ultimately, this process yields a high-fidelity open-palm hand mesh derived from four independent partial point clouds. As shown in Fig. 1, PatientHandNet we proposed mainly consists of two networks: view-to-view transformation network (VVTN) and hand shape reconstruction network (HSRN). Their procedures are detailed in Algorithm 1. ### B. Feature Extractor Similar to the encoder design of [^40] and [^41], we stack two simplified PointNet (PN) modules to build the feature extractor, which is named the combined PN (CPN). The architecture of CPN is depicted in Fig. 2. The first PN is a shared multilayer perceptron (MLP) consisting of two layers with ReLU activation. It takes as input $I^n$ points represented as a $I^n \times 3$ matrix where each row denotes the 3-D position of a point $s_i^n$ and converts $s_i^n$ into a point feature vector $f_i^l$. The set of $f_i^l$ can be represented as a feature matrix $F^l$ where each row is $f_i^l$. By means of the point-wise maxpooling operation, $F^l$ is aggregated into a global feature $g^l$. Next, we concatenate $g^l$ to each $f_i^l$ to obtain a updated feature matrix $\hat{F^l}$. $\hat{F}^l$ is further fed into the second PN to learn the final global feature. ```pseudo // Algorithm 1: PatientHandNet Input: $\mathcal{X} : \{\mathbf{S}^1, \mathbf{S}^2, \mathbf{S}^3, \mathbf{S}^4\}$ Output: $\mathcal{Y}$ 1: procedure VVTN $\triangleright$ Input: $\mathcal{X} \triangleright$ Output: $\mathcal{T}$ 2: features of $\mathcal{X} : \mathcal{G}^{(0)} \leftarrow \Psi_1(\mathbf{S}_1, \mathbf{S}_2, \mathbf{S}_3, \mathbf{S}_4)$ 3: initialize step $t \leftarrow 0$ 4: $\mathbf{V}_t^1, \mathbf{V}_t^2, \mathbf{V}_t^3, \mathbf{V}_t^4 \leftarrow \Phi_1(\mathcal{G}^{(t)})$ 5: while $\mathcal{L}_{VVTN}((\mathbf{G}_1, \mathbf{G}_2, \mathbf{G}_3, \mathbf{G}_4), (\mathbf{V}_t^1, \mathbf{V}_t^2, \mathbf{V}_t^3, \mathbf{V}_t^4))$ not ``` converged do 6: $t \leftarrow t + 1$ 7: $\mathbf{V}_t^1, \mathbf{V}_t^2, \mathbf{V}_t^3, \mathbf{V}_t^4 \leftarrow$ AdamOptimizer $(\mathbf{V}_{t-1}^1, \mathbf{V}_{t-1}^2, \mathbf{V}_{t-1}^3, \mathbf{V}_{t-1}^4)$, $\mathbf{V}_{t-1}^4$ 8: $\text{end while}$ 9: $\mathbf{V}_t^1, \mathbf{V}_t^2, \mathbf{V}_t^3, \mathbf{V}_t^4 \leftarrow \mathbf{V}_t^1, \mathbf{V}_t^2, \mathbf{V}_t^3, \mathbf{V}_t^4$ 10: $\mathcal{T} \leftarrow \arg\min_{\mathbf{R}_{n3}, \mathbf{t}_{n3}} \sum_{j=1}^J \|\mathbf{R}_{n3} v_j^n + \mathbf{t}_{n3} - \phi(v_j^n, \mathbf{V}^3)\|^2, n = 1, 2, 4$ 11: $\text{return} \mathcal{T}$ 12: procedure HSRN $\triangleright$ Input: $\hat{\mathcal{X}} \triangleright$ Output: $\mathcal{Y}$ 13: features of $\mathcal{X} : \mathcal{A}^{(0)} \leftarrow \hat{\Psi}(\hat{\mathbf{S}}_1, \hat{\mathbf{S}}_2, \hat{\mathbf{S}}_3, \hat{\mathbf{S}}_4)$ 14: initialize step $t \leftarrow 0$ 15: $\hat{\mathcal{Y}}_t \leftarrow \hat{\Phi}_1(\mathcal{A}^{(t)})$ 16: $\Delta \mathcal{Y}_t \leftarrow \hat{\Phi}_2(\hat{\mathcal{Y}}_t)$ 17: $\mathcal{Y}_t \leftarrow \hat{\mathcal{Y}}_t + \Delta \mathcal{Y}_t$ 18: while $\mathcal{L}_{HSRN}(\mathbf{G}_3, \hat{\mathcal{Y}}_t, \mathcal{Y}_t)$ not converged do 19: $t \leftarrow t + 1$ 20: $\mathcal{Y}_t \leftarrow$ AdamOptimizer $(\mathcal{Y}_{t-1})$ 21: $\text{end while}$ 22: $\mathcal{Y} \leftarrow \mathcal{Y}_t$ 23: return $\mathcal{Y}$ ![Image](images/b6583e08e70b33b7a8478ac4beed4f1062977f57ab0180b8a384513fae4f4f97.jpg)
Fig. 2. Architecture of the proposed CPN.
$g^h$ following the similar processing of the first PN. $g^l$ and $g^h$ are then concatenated, forming the combined latent vector C. Such a strategy ensures that C contains both low-level and high-level features extracted from the input point clouds. ### C. View-to-View Transformation Network As the step1 in Fig. 1 shows, we design a four-branch encoder since the proposed method takes four partial point clouds $\mathcal{X} = \{\mathbf{S}^1, \mathbf{S}^2, \mathbf{S}^3, \mathbf{S}^4\}$ as input. An intuitive alternative is to simply leverage four CPNs to consume individual partial point clouds to obtain four global features of partial point --- ![Image](images/3b02b0cf0cc496e124cef69400a7c51a7c70c86419e1ff4f34d43748416461f8.jpg)
Fig. 3. Visual comparisons of overlapping the four partial scans without and with pre-alignment. (a) Raw partial point clouds and the overlapping of them by translating their centroids to the origin. (b) Pre-aligned partial point clouds and the overlapping of them.
clouds and then apply a maxpooling operation to fuse these features to obtain the final global feature. However, such a solution ignores the spatial relationship among these point clouds. The input four partial point clouds are captured from the same hand but uncalibrated [Fig. 3(a)]. Our insight is that the final hand reconstruction can be significantly improved if the four partial point clouds are pre-aligned [Fig. 3(b)] before extracting their features (comparisons can be seen in the following ablation studies). However, it is challenging to pre-align the four partial point clouds because of two main reasons: 1) low overlaps exist between the palmar-view point clouds and ulnar-view/radial-view point clouds, and no overlaps exist between the palmar-view and the dorsal-view point clouds, which makes the popular iterative closest point (ICP) algorithm [^42] and its variants fail and 2) the hand pose of the four partial point clouds are different, which is hard to be addressed using rigid registration. To tackle this problem, we propose the VVTN inspired by the work of [^43], as shown in the step1 of Fig. 1. Given the four partial point clouds $\mathbf{S}^1$, $\mathbf{S}^2$, $\mathbf{S}^3$, $\mathbf{S}^4$, VVTN first uses four feature extractors to learn four combined latent vectors, respectively, $$ \begin{aligned}\mathbf{C}_{1}&=\psi_{1}(\mathbf{S}_{1}|w_{1}),\quad\mathbf{C}_{2}=\psi_{2}(\mathbf{S}_{2}|w_{2})\\\mathbf{C}_{3}&=\psi_{3}(\mathbf{S}_{3}|w_{3}),\quad\mathbf{C}_{4}=\psi_{4}(\mathbf{S}_{4}|w_{4})\end{aligned} $$ where $\psi_{n}(\cdot|w_{n})$ represents the feature extraction function with weight $w_{n}$ learned by the CPN, n is the index of views. Then, it employs a maxpooling operation to fuse these four combined latent vectors into a global feature $$ \mathcal{G}=\mathrm{m a x p o o l i n g}(\mathbf{C}_{1},\mathbf{C}_{2},\mathbf{C}_{3},\mathbf{C}_{4};\mathbf{C}_{3},\mathbf{C}_{4};\mathbf{C}_{3}). $$ Therefore, the feature extractor of VVTN $\Psi$ consists of four $\psi_n(\cdot|w_n)$ and one maxpooling $$ \mathcal{G}=\Psi(\mathbf{S}_{1},\mathbf{S}_{2},\mathbf{S}_{3},\mathbf{S}_{4}|w_{e}). $$ Next, $\mathcal{G}$ is further fed into the decoders $\Phi$ that consists of four MLP with reshape operations to predict virtual correspondences [^43] represented by four complete hands in the same open-palm pose $$ \mathbf{V}^{1},\mathbf{V}^{2},\mathbf{V}^{3},\mathbf{V}^{4}=\Phi(\mathcal{G}|w_{d}) $$ where $\mathbf{V}^n = \{v_j^n \in \mathbb{R}^3 | j = 1, 2, \ldots, J\}$ denotes the set of points with J points corresponding to $\mathbf{S}^n$, n is the index of views. Since $\mathbf{V}^1, \mathbf{V}^2, \mathbf{V}^3$, and $\mathbf{V}^4$ have the same number and order of points, and the same hand shape and pose, the virtual correspondences are built among $\mathbf{V}^1, \mathbf{V}^2, \mathbf{V}^3$, and $\mathbf{V}^4$. Given such virtual correspondences, the transformation parameters among $\mathbf{S}^1, \mathbf{S}^2, \mathbf{S}^3$, and $\mathbf{S}^4$ can be easily calculated and can be used to align $\mathbf{S}^1, \mathbf{S}^2, \mathbf{S}^3, \mathbf{S}^4$ despite they have different hand poses and little overlaps. Specifically, we take $\mathbf{V}^3$ as a reference view (or target view) corresponding to our final reconstructed $\mathcal{Y}$. The transformation $\mathcal{T} = \{\mathbf{R}_{n3}, \hat{\mathbf{t}}_{n3} | n = 1, 2, 4\}$ of $\{\mathbf{V}^3, \mathbf{V}^n | n = 1, 2, 4\}$ can be estimated by solving the following optimization: $$ \hat{\mathbf{R}}_{n3},\hat{\mathbf{t}}_{n3}=\underset{\mathbf{R}_{n3},\mathbf{t}_{n3}}{\arg\min}\sum_{j=1}^{J}\left\|\mathbf{R}_{n3}v_{j}^{n}+\mathbf{t}_{n3}-\phi\big(v_{j}^{n},\mathbf{V}^{3}\big)\right\|^{2} $$ where $\mathbf{R}_{n3} \in \mathrm{SO}(3)$ and $\mathbf{t}_{n3} \in \mathbb{R}^3$ are the estimated rotation matrix and translation vector, respectively from the nth view to the 3th view. Suppose $\hat{\mathbf{R}}_{n3}$ and $\hat{\mathbf{t}}_{n3}$ are the optimal rotation and translation, $\phi(v_j^n, \mathbf{V}^3)$ is a correspondence function that maps the point $v_j^n$ in $\mathbf{V}^n$ to its corresponding point in $\mathbf{V}^3$. Each $\mathbf{S}^n$ in $\mathcal{X}$ can be aligned to $\mathbf{S}^3$, which is denoted as $\hat{\mathbf{S}}^n$. $$ \begin{array}{r}{\hat{\mathbf{S}}^{n}=\mathbf{S}^{n}[\hat{\mathbf{R}}_{n3},\hat{\mathbf{t}}_{n3}].}\end{array} $$ Since $\mathbf{S}^3$ is a reference view, $\hat{\mathbf{S}}^3$ is still $\mathbf{S}^3$. Therefore, $\hat{\mathcal{X}}$ is the composition of $\hat{\mathbf{S}}^1$, $\hat{\mathbf{S}}^2$, $\hat{\mathbf{S}}^3$, $\hat{\mathbf{S}}^4$. ### D. Hand Shape Reconstruction Network The HSRN is an encoder-decoder architecture. The proposed encoder learns the aggregated feature in a multiscale and multi-view manner, named multiscale and multi-view feature aggregation (MMFA). The proposed decoder is able to regress 3-D model vertices using a coarse-to-fine strategy, called hand vertex regression (HVR). 1) Multiscale and Multi-View Feature Aggregation: The pre-aligned partial point clouds $\hat{\mathcal{X}} = \{\hat{\mathbf{S}}^{1}, \hat{\mathbf{S}}^{2}, \hat{\mathbf{S}}^{3}, \hat{\mathbf{S}}^{4}\}$ are further fed into MMFA. As its name shows, the MMFA is designed to learn a feature by aggregating the multiscale and multi-view information extracted from the input partial point clouds. We notice that the palmar-view and dorsal-view point clouds contain more information of hand length and breadth, while the ulnar-view and radial-view point clouds are more informative regarding the hand thickness. Therefore, the MMFA is designed to include three steps. First, a CPN is used to extract the feature of a single-view point cloud $\{\hat{\mathbf{S}}^{3}\}$. Next, a CPN is applied to learn the feature of the concatenation of two point clouds $\{\hat{\mathbf{S}}^{3}, \hat{\mathbf{S}}^{4}\}$. Last, another CPN is leveraged to obtain the feature of the concatenation of all point clouds $\{\hat{\mathbf{S}}^{1}, \hat{\mathbf{S}}^{2}, \hat{\mathbf{S}}^{3}, \hat{\mathbf{S}}^{4}\}$. These features can be represented as follows: $$ \begin{aligned}\hat{\mathbf{C}}^{3}&=\hat{\psi}_{1}(\hat{\mathbf{S}}^{3}|\hat{w}_{1})\\\hat{\mathbf{C}}^{3,4}&=\hat{\psi}_{2}(\mathrm{cat}(\hat{\mathbf{S}}^{3},\hat{\mathbf{S}}^{4})|\hat{w}_{2})\\\hat{\mathbf{C}}^{1,2,3,4}&=\hat{\psi}_{3}(\mathrm{cat}(\hat{\mathbf{S}}^{1},\hat{\mathbf{S}}^{2},\hat{\mathbf{S}}^{3},\hat{\mathbf{S}}^{4})|\hat{w}_{3})\end{aligned} $$ where $\hat{\psi}_n(\cdot|\hat{w}_n)$ represents the feature extraction function with weight $\hat{w}_n$ learned by the CPN and cat( $\cdot$) represents the concatenation operation, and $\hat{\mathbf{C}}^3$, $\hat{\mathbf{C}}^{3,4}$, $\hat{\mathbf{C}}^{1,2,3,4}$ are three --- multiscale multi-view features, which describe the hand shape from geometric and spatial perspectives. $\mathbf{C}^{3}$ contains geometric information on the length and breadth of the hand. $\mathbf{\hat{C}}^{3,4}$ not only improves the hand thickness relative to $\mathbf{\hat{C}}^{3}$, but also establishes a relative spatial relationship between $\mathbf{S}^{3}$ and $\mathbf{\hat{S}}^{4}$. $\mathbf{\hat{C}}^{1,2,3,4}$ fuses the information from all the views, which further enhances the geometric and spatial representation of the 3-D hand. All these features are finally fused by the maxpooling operation into a comprehensive representation called aggregated feature $\mathcal{A}$. $$ \mathcal{A}=\mathrm{m a x p o o l i n g}(\hat{\mathbf{C}}^{3};\hat{\mathbf{C}}^{3,4};\hat{\mathbf{C}}^{1,2,3,4}). $$ Therefore, the feature extractor of HSRN $\hat{\Psi}$ consists of three $\hat{\psi}_n(\cdot|\hat{w}_n)$ and one maxpooling $$ \mathcal{A}=\hat{\Psi}(\hat{\mathbf{S}}_{1},\hat{\mathbf{S}}_{2},\hat{\mathbf{S}}_{3},\hat{\mathbf{S}}_{4}|\hat{w}_{e}). $$ 2) Hand Vertex Regression: The HVR takes the aggregated feature $\mathcal{A}$ as input and aims to output a high-fidelity open-palm hand, as shown in the step2 of Fig. 1. To this end, an MLP is used to directly regress the vertices of the open-palm hand. However, such a global strategy ignores the local information of hand. To overcome this limitation, we proposed a refinement method by embedding a local self-transformer (ST) with the MPL into our HVR module to locally refine the vertex coordinates of the reconstructed hand. We first use one MLP with reshape operation $\hat{\Phi}_1$ to predict the initially reconstructed hand shape, which is represented as $\hat{Y}$. $$ \hat{\mathcal{Y}}=\hat{\Phi}_{1}(\mathcal{A}|\hat{w}_{d1}). $$ Then, each point $\hat{y}$ of $\hat{y}$ will be refined through ST, which is depicted in Fig. 4. Given jth point $\hat{y}_j$ of $\hat{y}$, we first calculate the attention scores $\{\mathbf{a}_{j,k}|k = 1,2,\ldots,K\ k \neq j\}$ between $\hat{y}_j$ and its k-nearest neighboring points $(k\text{-NN})$ $\{\hat{y}_{j,k}|k = 1,2,\ldots,K,k \neq j\}$ as follows: $$ \mathbf{a}_{j,k}=\frac{\exp(M_{Q}(\hat{y}_{j})\ominus M_{K}(\hat{y}_{j,k}))}{\sum_{k=1}^{K}\exp(M_{Q}(\hat{y}_{j})\ominus M_{K}(\hat{y}_{j,k}))} $$ where $M_{Q}, M_{K}$ denote the MLPs with different parameters, and $\ominus$ denotes the element-wise subtraction. Finally, the point spatial feature $\mathbf{p}_{j}$ of $\hat{y}_{j}$ can be obtained by $$ \mathbf{p}_{j}=M_{V}(\hat{\mathbf{y}}_{j})\oplus\sum_{k=1}^{K}a_{j,k}\odot M_{V}(\hat{\mathbf{y}}_{j,k}) $$ where $M_V$ denotes the MPL. $\oplus$ and $\odot$ denote element-wise addition and Hadamard product, respectively. The point spatial feature $\mathbf{p}_j$ is fed to the MLP to produce the displacement feature $\mathbf{d}_j$ of $\hat{y}_j$. $\mathbf{d}_j$ is further used for generating the point displacement $\Delta\hat{y}_j$, which is formulated as follows: $$ \Delta\hat{y}_{j}=\operatorname{t a n h}(\operatorname{M L P}(\mathbf{d}_{j})) $$ where $\tanh$ is the hyper-tangent activation. Therefore, all $\hat{y}_j$ of $\hat{y}$ have displacements, which is formulated as follows: $$ \Delta\mathcal{Y}=\hat{\Phi}_{2}(\hat{\mathcal{Y}}|\hat{w}_{d2}) $$ where $\hat{\Phi}_2(\cdot|\hat{w}_{d2})$ is the ST module with weight $\hat{w}_{d2}$. Finally, $\hat{y}$ is refined by $\Delta\mathcal{Y}$ $$ \mathcal{Y}=\hat{\mathcal{Y}}+\Delta\mathcal{Y} $$ where Y is the refined hand shape. ![Image](images/dc8ce8e9a78d58b62820447269193004b9147f08b2a5a9bbc30e65616cff2f94.jpg)
Fig. 4. Structure of the ST module.
### E. Loss Functions The proposed model was trained in a supervised end-to-end manner. As shown in Fig. 1, the proposed PatientHandNet includes two neural networks. To train our model, we define the following loss functions to evaluate the reconstruction errors. 1) View-to-View Transformation Network: VVTN has four branches to predict virtual correspondences $\{\mathbf{V}^1, \mathbf{V}^2, \mathbf{V}^3, \mathbf{V}^4\}$ corresponding to $\{\mathbf{S}^1, \mathbf{S}^2, \mathbf{S}^3, \mathbf{S}^4\}$. These branches are supervised by four SMPL-X hand models with J points in open palm pose, respectively, which are denoted as $\{\mathbf{G}^1, \mathbf{G}^2, \mathbf{G}^3, \mathbf{G}^4\}$ corresponding to the ground truth of $\{\mathbf{S}^1, \mathbf{S}^2, \mathbf{S}^3, \mathbf{S}^4\}$. Therefore, the loss function of VVTN is defined for these four branches $$ \begin{aligned}\mathcal{L}_{\mathrm{VVTN}}&=\mathcal{L}(\mathbf{V}^{1},\mathbf{G}^{1})+\mathcal{L}(\mathbf{V}^{2},\mathbf{G}^{2})+\mathcal{L}(\mathbf{V}^{3},\mathbf{G}^{3})+\mathcal{L}(\mathbf{V}^{4},\mathbf{G}^{4})\\&=\frac{1}{J}\sum_{j=1}^{J}\left\|v_{j}^{1}-g_{j}^{1}\right\|^{2}+\frac{1}{J}\sum_{i=1}^{J}\left\|v_{j}^{2}-g_{j}^{2}\right\|^{2}\\&\quad+\frac{1}{J}\sum_{i=1}^{J}\left\|v_{j}^{3}-g_{j}^{3}\right\|^{2}+\frac{1}{J}\sum_{i=1}^{J}\left\|v_{j}^{4}-g_{j}^{4}\right\|^{2}\tag{16}\end{aligned} $$ where $v_j \in \mathbf{V}$, $g_j \in \mathbf{G}$ are the jth corresponding points pair in $\mathbf{V}$, $\mathbf{G}$. 2) Hand Shape Reconstruction Network: HSRN is supervised by the open-palm hand G $^{3}$ in a coarse-to-fine manner. We, thus, define the loss as follows: $$ \begin{aligned}\mathcal{L}_{\mathrm{H S R N}}&=\mathcal{L}(\hat{\mathcal{Y}},\mathbf{G}^{3})+\mathcal{L}(\mathcal{Y},\mathbf{G}^{3})\\&=\frac{1}{J}\sum_{j=1}^{J}\left\|\hat{\mathbf{y}}_{j}-g_{j}^{3}\right\|^{2}+\frac{1}{J}\sum_{i=1}^{J}\left\|\mathbf{y}_{j}-g_{j}^{3}\right\|^{2}\end{aligned} $$ where $\hat{y}_j \in \hat{\mathcal{Y}}, y_j \in \mathcal{Y}$ and $g_j^3 \in \mathbf{G}^3$ are the jth corresponding points in $\hat{\mathcal{Y}}, \mathcal{Y}$ and $\mathbf{G}^3$, respectively. 3) Complete Loss: Our complete loss is defined by a weighted sum of $\mathcal{L}_{VVTN}$ and $\mathcal{L}_{HSRN}$ $$ \mathcal{L}=\alpha_{1}\times\mathcal{L}_{\mathrm{VVTN}}+\alpha_{2}\times\mathcal{L}_{\mathrm{HSRN}} $$ where $\alpha_{1}$ and $\alpha_{2}$ are the weights that control the contribution of each term. --- ![Image](images/8039e1e3c1ef48f4e3fd14a8e714f2ea17ce1fce8b23472445f32f09f03634a1.jpg)
Fig. 5. Proposed pipeline of synthetic dataset generation. We first utilize FrankMocap to extract pose parameters ( $\theta$) of the SMPL-X model from RGB images. The shape parameters ( $\beta$) of the SMPL-X model are taken from the SURREAL. Next, we randomly combine $\beta$ and $\theta$ to generate four posed bodies with the same identity, from which we easily obtain the hand meshes. The ground-truth open-palm hand meshes can be obtained via resetting the random $\theta$ to be the predefined $\theta_0$. Finally, we render realistic partial scans from the posed hands via the open-source Blensor.
### F. Proposed Dataset 1) Synthetic Dataset: To train our model, a large-scale dataset consisting of inputs with four partial scans of the same hand and corresponding ground-truth open-palm hand meshes is necessary. Moreover, the four partial scans may have the same or different postures. Unfortunately, none of the existing public hand datasets meets these requirements. The existing hand datasets lack ground-truth open-palm hand shapes and most of them only contain single palmar-facing depth images. An intuitive solution is to collect the dataset by scanning real-world subjects. However, such a procedure is extremely expensive and time-consuming for a large-scale dataset (e.g., 100 K samples). To address this problem, we propose an innovative method to generate the appropriate dataset needed for training the proposed PatientHandNet. Fig. 5 illustrates the proposed pipeline for generating such a dataset. a) Generating 3-D Hand Meshes: First, we collected 12 K images online consisting of female and male subjects performing different motions. Second, a state-of-the-art RGB-based hand and motion capture system, FrankMocap [^44], was used to extract the $\theta$ and $\beta$ values of SMPL-X from these images. SMPL-X [^45] is a generative body model with two key parameters: the pose parameter $\theta$ and the shape parameter $\beta$. Given a pair of $\theta$ and $\beta$, a new human body mesh with specified body posture, hand gesture, and facial expression can be obtained. Therefore, 12 K $\theta$s and 12 K $\beta$s were obtained via FrankMocap. We observed that the estimated poses looked good, while the estimated shapes were less accurate due to the inherent scale ambiguity associated with the 2-D to 3-D conversion via FrankMocap. We, thus, dropped the 12 K $\beta$s and used 2103 $\beta$s from the SURREAL dataset [^46]. Next, a large-scale body dataset was obtained by a random combination of $\beta$s and $\theta$s. Specifically, we randomly selected a $\beta$ and four $\theta$s to create four body models, which had the same body shape but different hand poses. By assigning the pre-defined open-palm pose (denoted by $\theta_0$), another body model with the designed open-palm hand shapes was built. The left and right hands were easily extracted from these body models to build a sample of our dataset (four pairs of posed hands as input and a pair of open-palm hands as the ground ![Image](images/726f90efef43dced8bba15ca0c26a5a7fa420a2e3f208f570c29022195f83fc7.jpg)
Fig. 6. Our synthetic meshes with hand shape and pose variations. Each row is from the same subject with different hand poses, the last column depicts the ground-truth hand meshes.
truth). Using this procedure, we generated 300 K samples for the proposed dataset. Fig. 6 depicts samples of the proposed dataset. b) Rendering Multi-View Partial Scans: The open-source blender sensor simulation plugin Blensor [^47] was used to render the partial scans of hands. In this study, we set the camera as Microsoft Kinect V2 and the scanning distance are randomly selected at intervals from 0.5 to 0.8 m. To mimic real-world scanning, we also added a random rotation set at the intervals from $-10^\circ$ to $10^\circ$ in all x, y, and z directions; the noise was added as well. We rendered the left hand and the right hand separately. As aforementioned, each sample of our hand mesh dataset consists of four posed hands and one open-palm ground-truth hand. By rendering one partial scan for each posed hand mesh from one of the four designed views (palmar, ulnar, dorsal, and radial views) completes the data generation process consisting of multi-view partial hand scans and corresponding ground-truth open-palm hand meshes. c) Statistics of the Synthetic Hand Dataset: To analyze the hand shapes of our proposed dataset, we measured lengths of the 13 dominant dimensions [^48] based on the open-palm hand pose, as shown in Fig. 7. Take hand length --- ![Image](images/9659554d79111192513970c7d8e4e5bbebe95da03f19e5635219ac59bd8b7cf0.jpg)
1-Hand length 2-Palm length 3-Palm breadth 4-Palm thickness 5-Thumb length 6-Index length 7-Middle length 8-Ring length 9-Little length 10-Thumb breadth/girth 11-Index breadth/girth 12-Middle breadth/girth 13-Ring breadth/girth 14-Little breadth/girth
Fig. 7. 13 dimensions of hand accompanying with their IDs on the hand template and the statistics of lengths of 13 dimensions in our synthetic dataset.
as an example (Measurement Type #1), 50% of its length is distributed between 17 and 22 cm, which is consistent with the real-world hand biometrics statistical result in [^49]. Moreover, the minimum length and maximum length of hand length are 13 and 25.2 cm, which widely covers the hand length from a child to an adult. The other dimensions have similar statistical properties. 2) Real-World Dataset: Our method also can generalize well to real-world data, but there exists no publicly available multi-view real-world dataset that includes ground-truth open-palm hands. Therefore, we built a novel real-world dataset, as shown in Fig. 8(a). An iPad with an embedded structure sensor Mark I was used to scanning 18 different hands (13 males and five females) in arbitrary posture from four views of the hand. A professional anthropometrist extracted hand measurements according to the 14 dimensions described in Fig. 8(b); these can be considered as the ground truth values of the real-world dataset. ### IV. EXPERIMENTS We compared the proposed method against the state-of-the-art shape reconstruction methods based on synthetic data as well as real-world data, including the following methods: HandAR [^13], $S^{2}$-HAND [^12], HandOccNet [^36], SnowflakeNet [^50], L2H [^51], and MobRecon [^37]. When we use the unseen synthetic dataset mentioned in Section III-F1, we randomly select 720 samples, which have open palm hand as ground truth and no overlapping with training and validation datasets. These selected data are represented as point clouds, which can be the input of point-based methods (SnowflakeNeta [^50] and L2H [^51]). However, RGB-based methods (HandAR [^13], $S^{2}$-HAND [^12], HandOccNet [^36], and MobRecon [^37]) require RGB images as input. To ensure a fair comparison, we obtain RGB images by rendering the ground-truth open-palm mesh corresponding to the selected data. We also evaluate the performance on real-world data. As described in Section III-F2, the real-world data we ![Image](images/245e42a3c9ba0cbc3a7f87f7b0bb7f1327ef99d75a362e0f2fcbfc17fbc088af.jpg)
(a) Four-view depth images acquisition of real-world data
![Image](images/80f41c8ffadf48082c5b1f4c00a9154c644847c1123151da24a5e26a3330616a.jpg)
(b) Measurement values acquisition of real-world data
Fig. 8. (a) iPad embedded with a structure sensor Mark I is employed to scan a hand and produce four-view depth images. (b) Professional anthropometrist uses tape to extract biometrics measurements according to the 14 dimensions detailed in Fig. 7. The table shows the measurement values.
collected consist of partial point clouds and corresponding ground-truth hand measurement values extracted by a professional anthropometrist. Furthermore, the corresponding RGB images are also captured. Therefore, the performance of point-based methods and RGB-based methods can be evaluated on real-world data with ground-truth hand measurement values. In the end, we conduct ablation experiments based on the 720 samples to understand the effectiveness of key components in our proposed method, including multi-view input, prealigned inputs, multiscale, and multi-view feature learning and point-based refinement strategy. ### A. Experimental Setup The training is operated on a desktop PC with Intel $^{1}$ Core i9-10900 CPU @ 2.80 GHz and GeForce RTX 3090Ti. Our training has 200 epochs, and we set $\alpha_1 = 1$, $\alpha_2 = 0$ for the first 50 and set $\alpha_1 = 1$, $\alpha_2 = 1$ for the rest of the epochs. ### B. Evaluation Metrics 1) Evaluation Metrics on Synthetic Data: Since the synthetic dataset contains the ground-truth open-palm hand, errors can be computed by comparing the predicted hand shape and the ground-truth hand shape. To this end, we employ the widely used reconstruction evaluation metric: chamfer distance (CD) [^52]. The CD error measures the average nearest squared distance between predicted hand $\mathcal{Y}$ and the ground truth hand $\mathbf{G}^{3}$, which is defined as follows: $$ \begin{align*}E_{\mathrm{CD}}(\mathcal{Y},\mathbf{G}^{3})=\frac{1}{2}\Biggl[\frac{1}{|\mathcal{Y}|}\sum_{y\in\mathcal{Y}}\min_{g^{3}\in\mathbf{G}^{3}}\|y-g^{3}\|_{2}^{2}\\+\frac{1}{|\mathbf{G}^{3}|}\sum_{g^{3}\in\mathbf{G}^{3}}\min_{y\in\mathcal{Y}}\|g^{3}-y\|_{2}^{2}\Biggr].\end{align*} $$ $^{1}$Registered trademark. --- ![Image](images/4e0bdfc61ed15ef73cc9d38947b5ce22d13e25eb1a3052bc313620e70b73340b.jpg)
Fig. 9. Visual comparison of hand shape reconstruction error with state-of-the-art methods on synthetic data. From top to bottom: the results of HandAR [^13], $S^{2}$-HAND [^12], HandOccNet [^36], SnowflakeNet [^50], L2H [^51], MobRecon [^37], our results, and the ground truth. The colored points represent the point-to-point errors in millimeters between reconstructed hand shape and GT.
2) Evaluation Metrics on Real-World Data: Unlike the synthetic dataset, the real-world dataset only contains the multi-view partial scans as input but lacks the ground-truth open-palm hand shapes, which makes CD fail. Fortunately, our collected real-world dataset contains the hand measurement values measured by a professional anthropometrist, which are ground-truth measurement values. From the reconstructed hand shape, hand measurements can be easily extracted. Therefore, the absolute error (AE) and relative error (RE) are applied to measure the measurement errors by comparing the predicted measurement values $M_{pred}$ and ground-truth measurement values $M_{gt}$. The AE and RE are defined as follows: $$ E_{\mathrm{AE}}=|M_{\mathrm{pred}}-M_{\mathrm{gt}}| $$ $$ E_{\mathrm{RE}}=\frac{\left|M_{\mathrm{pred}}-M_{\mathrm{gt}}\right|}{M_{\mathrm{gt}}}\times100. $$ 3) Results on Synthetic Data: We first compare our results with different methods on the unseen synthetic data. Fig. 9 depicts quantitative comparisons and each point is colorized by the point-to-point errors in millimeters between the reconstructed hand shape and the ground-truth shape. It can be seen that the results of $S^2$-HAND [^12] and HandOccNet [^36] do not have the open palm posture but have severe bending on
TABLE I
AVERAGE MEAN, STANDARD DEVIATION AND MAXIMUM DISTANCE OF CD ON SYNTHETIC DATA (UNIT: mm)
Methods$\mu$$\sigma$max
HandAR $^{[13]}$12.384.3143.84
$S^{2}HAND$ $^{[12]}$13.343.6321.93
HandOccNet $^{[36]}$13.414.9027.57
SnowflakeNe $^{[50]}$0.340.752.50
L2H $^{[51]}$0.430.812.62
MobRecon $^{[37]}$5.983.1919.18
Ours0.210.632.29
the fingers. The results of HandAR [^13] produced a severe deformity on the little finger and the ninth result of them collapses. The results of SnowflakeNet [^50], L2H [^51], and MobRecon [^37] have good geometric shapes. However, their point-to-point errors are larger than our proposed method, especially MobRecon [^37], which demonstrates RGB-based methods are easy to result in scale ambiguity in the shape. We also calculate the average value $\mu$, average standard derivation $\sigma$, and maximum distance max of (19) to quantitatively evaluate each method and summarize the result in Table I. From this we can have two main observations. First, the performance of the depth-based methods are better compared to RGB-based methods. For example, our method ---
TABLE II
MEAN OF ABSOLUTE AND RELATIVE ANTHROPOMETRIC MEASUREMENTS ERRORS COMPARISON ON REAL-WORLD DATA (UNIT: mm)
MeasurementsFemale (S subjects)Male (S subjects)
HandOCC[^150]$S^{2}H$AND[^12]HandOCC[^36]SnowflakeNo[^50]L2H[^51]MobRecon[^37]OursHandOCC[^150]$S^{2}H$AND[^12]HandOCC[^36]SnowflakeNo[^50]L2H[^51]MobRecon[^37]Ours
E_AEE_BEE_AEE_BEE_AEE_BEE_AEE_BEE_AEE_BEE_AEE_BEE_AEE_AEE_BEE_AEE_BEE_AEE_BEE_AEE_BEE_AEE_BEE_AEE_BE
1- Hand length63.835%10.05.4%7.64.2%5.22.8%4.82.6%15.48.2%2.01.1%43.020.0%34.416.1%23.610.8%6.63.2%4.62.1%25.411.6%1.6
2- Palm length33.88.88.66.65.4%5.43.7%5.05.2%10.610.2%1.4%1.4%1.4%18.818.6%14.1%18.5%14.5%5.83.1%3.9%17.417.4%17.42.24.0
3- Palm breadth29.235.9%8.810.5%8.210.0%3.45.3%3.74.1%11.614.2%1.62.0%21.622.3%21.0%22.2%18.619.5%4.85.1%5.85.9%17.818.6%2.0
4- Thumb length25.642.6%11.018.2%7.812.9%4.26.9%4.47.3%9.615.8%1.83.0%17.224.7%15.6%23.4%11.816.8%4.46.5%4.66.6%14.00.0%2.0
5- Palm length27.436.4%10.414.3%8.211.2%3.24.4%2.65.6%7.55.2%0.81.4%14.615.8%17.414.8%15.0%10.1%3.05.8%5.2%9.09.0%14.81.7
6- Middle length29.636.5%7.88.8%4.25.1%4.45.7%3.64.5%3.28.7%3.24.0%18.820.0%15.2%16.4%12.012.8%3.44.0%4.24.7%9.09.8%2.0
7- Ring length25.434.4%9.213.2%7.610.3%3.44.7%1.83.3%6.68.8%2.43.3%18.421.5%10.624.4%13.415.5%4.65.7%5.06.1%13.014.9%2.8
8- File length25.436.6%4.015.2%7.412.5%3.66.0%3.05.1%2.45.8%0.81.7%14.816.8%12.1%16.3%12.9%10.3%3.45.8%4.7%5.9%7.410.9%2.4
9- Palm thickness7.227.8%17.76.017.7%6.023.0%4.816.8%4.216.6%2.810.6%1.25.0%8.828.6%9.230.2%9.029.7%2.27.1%3.612.1%6.220.9%1.8
10- Thumb girth12.819.7%7.812.0%6.810.5%4.46.8%5.68.7%6.09.2%2.23.4%13.817.8%18.223.5%16.421.0%18.7%4.05.4%4.46.1%13.217.0%2.0
11- Middle girth10.07.011.0%7.011.0%9.5%5.28.9%7.55.2%4.06.7%2.62.5%12.616.2%16.4%12.3%13.4%12.4%10.4%1.85.8%6.8%11.8%1.21.7
12- Middle girth11.218.9%6.610.1%4.88.0%4.46.2%4.87.1%3.58.3%1.44.1%12.817.9%15.815.8%12.212.2%5.27.6%3.46.9%8.211.4%1.8
13- Ring width9.216.2%5.610.1%6.411.6%3.46.2%4.87.8%3.08.9%1.42.5%9.615.0%12.219.2%10.817.0%5.28.3%4.87.6%7.211.1%1.4
14- Little length9.219.8%5.010.9%5.211.3%4.09.1%4.410.0%4.69.9%2.86.4%10.418.2%12.422.0%11.019.5%5.29.3%4.07.1%7.012.3%2.2
Average22.729.6%8.021.7%6.510.4%4.26.9%4.16.8%6.79.2%2.13.4%17.520.5%17.421.7%13.817.5%4.56.0%4.36.0%11.91.9
leads to 12.17, 3.68, and 41.55 mm improvements of $\mu$, $\sigma$, and max compared with HandAR [^13], respectively. This verifies that 3-D information is desired for more accurate hand shape reconstruction. Second, compared with different depth image-based methods, our results achieve the best performance in term of all metrics. Specifically, our method significantly outperforms SnowflakeNet [^50] and L2H [^51] in terms of both $\mu$ and $\sigma$, which implies that our method has a higher accuracy for the global hand shape. The improvement of max brought by our method is also obvious, which indicates that our method achieves the most accurate local hand shape reconstruction. 4) Results on Real-World Data: Once our model is successfully trained, it generalizes well to real-world data. We further qualitatively compare the proposed method against different methods based on our collected real-world dataset, as shown in Fig. 10. Note that the results of $S^{2}$-HAND [^12] and HandOccNet [^36] do not have the open palm posture but have bent fingers, and the results of HandAR [^13] obviously have severe deformation in shape. The results of MobRecon [^37] have various hand postures instead of open palm poses. Despite the results of SnowflakeNet [^50] and L2H [^51] looking similar to ours, our method actually achieves the best results according to Table II that shows the quantitative hand measurement comparisons for five females and five males. Besides, the average AE and RE of RGB-based methods are larger than 5 mm and 9%, respectively, the average AE and RE of depth-based methods of SnowflakeNet [^50] and L2H [^51] are larger than 4 mm and 5%, while our results are less than 3 mm and 4%. The results on real-world data are consistent with those on unseen synthetic data, which further demonstrates the effectiveness of the proposed method. ### C. Robustness In this section, we demonstrate the robustness of our proposed method on real-world data in different formats. We randomly selected five male real-world subjects' scans and introduce different levels of Gaussian noise, specifically $\sigma = 0.1\%$, 0.2%, 0.3%, to study the effect of noise on our method. The error results are presented in Table III. Increasing levels of noise will lead to higher errors. However, the maximum absolute error still less than 5 mm even with the presence of 0.3% noise, which is deemed acceptable, which is deemed acceptable. These results indicate that our method effectively filters out irrelevant or erroneous information, enabling it to
TABLE III
MEAN OF ABSOLUTE AND RELATIVE ANTHROPOMETRIC MEASUREMENTS ERRORS ON FIVE MALE REAL-WORLD DATA FOR VARIOUS NOISE LEVELS (UNIT: mm)
Measurements$\sigma = 0$$\sigma = 0.1\%$$\sigma = 0.2\%$$\sigma = 0.3\%$
$E_{AE}$$E_{RE}$$E_{AE}$$E_{RE}$$E_{AE}$$E_{RE}$$E_{AE}$$E_{RE}$
1-Hand length1.600.74%2.201.13%2.601.21%3.601.60%
2-Palm length2.201.91%3.002.33%3.803.29%3.603.24%
3-Palm width2.202.39%2.402.49%2.602.86%3.403.49%
4-Thumb length2.002.83%2.003.20%2.002.98%2.203.07%
5-Index length1.401.71%1.802.52%2.803.54%3.003.57%
6-Middle length2.002.23%2.402.97%3.604.06%4.204.49%
7-Ring length2.803.44%3.104.11%3.704.57%4.405.23%
8-Little length2.003.23%2.804.78%3.806.00%4.606.90%
9-Palm thickness1.203.90%1.806.57%3.009.87%3.8011.81%
10-Thumb girth2.002.66%2.403.52%3.204.30%3.605.08%
11-Index girth1.201.66%1.401.89%1.402.02%2.202.87%
12-Middle girth1.802.61%2.002.42%2.203.18%3.004.03%
13-Ring girth1.402.19%1.602.89%1.001.63%1.402.58%
14-Little girth2.203.94%2.204.04%2.404.48%2.604.53%
TABLE IV
MEAN OF ABSOLUTE AND RELATIVE ANTHROPOMETRIC MEASUREMENTS ERRORS ON FIVE FEMALE REAL-WORLD DATA IN DIFFERENT DENSITY (UNIT: mm)
MeasurementsNo sampling20481024512
$E_{AE}$$E_{RE}$$E_{AE}$$E_{RE}$$E_{AE}$$E_{RE}$$E_{AE}$$E_{RE}$
1-Hand length2.001.09%2.001.10%2.601.41%3.201.73%
2-Palm length2.202.13%2.602.55%3.203.14%3.503.53%
3-Palm breadth1.601.97%1.001.25%1.401.75%1.802.25%
4-Thumb length1.802.96%2.203.66%2.003.33%1.803.00%
5-Index length0.801.10%1.201.65%1.602.20%2.002.76%
6-Middle length3.203.99%3.203.99%3.604.49%3.904.96%
7-Ring length2.403.29%2.803.84%3.004.15%3.004.11%
8-Little length2.804.74%3.005.06%3.205.43%3.606.13%
9-Palm thickness1.204.99%1.405.76%2.008.08%2.8011.17%
10-Thumb girth2.203.41%2.003.08%2.403.72%3.204.97%
11-Index girth2.604.46%2.604.85%2.704.95%3.005.19%
12-Middle girth2.404.06%2.404.06%2.003.43%2.203.83%
13-Ring girth1.402.53%1.402.53%1.602.95%1.402.59%
14-Little girth2.806.37%3.006.79%3.407.64%3.407.67%
focus on extracting valuable features for reliable predictions. Additionally, we conducted experiments on another five female real-world dataset, which we subsampled into three different point cloud densities: 2048, 1024, and 512. The purpose was to examine how the density of the point cloud affects the performance of our method. The error results are shown in Table IV, from which we can see that the errors with different densities are very close. Therefore, our method can handle sparse or dense data without affecting its ability to extract valuable information. In conclusion, our method is able to adapt well to real-world scenarios where noise and data irregularities are prevalent. ### D. Ablation Studies We take the average value $\mu$, average standard derivation $\sigma$ and maximum value max of CD errors to evaluate the performance of critical components. Details are provided next. --- ![Image](images/39f2dea20427865280497e35fc7b7defe57d5a2ad8ed7081a1edab2bbd0cd2eb.jpg)
Fig. 10. Visual comparison of hand shape reconstruction error with state-of-the-art methods on real-world data. From top to bottom: the results of three HandAR [^13], $S^{2}$-HAND [^12], HandOccNet [^36], SnowflakeNet [^50], L2H [^51], MobRecon [^37], and our results.
TABLE V
COMPARISONS OF RECONSTRUCTION ERRORS WITH DIFFERENT NUMBER OF DEPTH IMAGES (UNIT: mm)
single depth imagetwo depth imagesfour depth images
$\mu$0.4430.4300.401
$\sigma$0.1530.1460.099
max1.1061.0340.598
1) Effectiveness of Multi-View Input: In this section, we explored the performance of different inputs based on the proposed method. PatientHandNet takes four depth images (i.e., palmar, ulnar, dorsal, and radial views of hand) as the input since they are the minimal captures necessary to obtain the complete geometry of the hand. We retrain our network based on a single depth image (dorsal view), two depth images (dorsal view and radial view) and four depth images, respectively. Table V compares the reconstruction errors of them. It can be seen that the results from two depth images are better than the results from a single depth image, and the results from four depth images are the best. 2) Effectiveness of Pre-Aligned Inputs: One of our insights is that the reconstruction accuracy is improved if the raw inputs are pre-aligned. To validate it, we compared the reconstruction accuracy for the proposed method without and with the pre-alignment. As Table VI illustrates, it can be seen
TABLE VI
COMPARISONS OF RECONSTRUCTION ERRORS WITHOUT AND WITH PRE-ALIGNING RAW INPUTS (UNIT: mm)
Wihtout pre-alignmentWith pre-alignment
$\mu$0.4680.401
$\sigma$0.1580.099
max1.1340.598
that the pre-alignment of raw inputs can significantly reduce reconstruction errors. With the help of the proposed VVT module, the input partial point clouds are aligned to obtain more accurate global shape information of the complete hand and also introduce the spatial relationship among inputs. 3) Effectiveness of Multiscale and Multi-View Feature Learning: MMFA was proposed in our model to learn hierarchically geometric and spatial features from multi-view inputs for hand shape reconstruction. To validate its effectiveness, two variants of our method were proposed, which adopt a single CPN and multiscale CPNs to capture geometric and spatial context, respectively. The variant of using a single CPN inputs all of views into a CPN to learn features; the multiscale CPNs are described in Section III-D. The experimental results are reported in Table VII, from which we can see that the multiscale CPNs outperform a single CPN in terms of all metrics. This implies that multiscale CPNs are able to learn ---
TABLE VII
COMPARISONS OF RECONSTRUCTION ERRORS WITH A SINGLE CPN AND MULTISCALE CPNS FEATURE LEARNING METHODS (UNIT: mm)
Without MMFAWith MMFA
$\mu$0.5250.401
$\sigma$0.1790.099
max1.3430.598
TABLE VIII
COMPARISONS OF RECONSTRUCTION ERRORS WITH AND WITHOUT POINT-BASED REFINEMENT (UNIT: mm)
Without refinementWith refinement
$\mu$0.4400.401
$\sigma$0.1430.099
max1.1130.598
more robust geometric and spatial features from different dimensions and different number of views compared to a single CPN. 4) Effectiveness of Point-Based Refinement Strategy: We also proposed a point-based refinement strategy to obtain an accurate reconstructed hand shape. Table VIII compares the reconstruction errors with and without the refinement step. It can be seen that the proposed point-based refinement helps to improve the accuracy of our result significantly. This is mainly because the ST in the refinement concentrates on the local spatial context between each point and its neighbors to further optimize the position of each point. ### V. CONCLUSION In this work, we have proposed a novel deep learning-based framework, PatientHandNet, to reconstruct the open-palm hand shape from sparse multi-view depth images captured by a single-commodity depth camera. Compared to the existing methods, our method has the following advantages: 1) we only require four depth images as input acquired with a single depth camera; 2) the subjects are allowed to change their hand poses during data acquisition; and 3) we output a high-fidelity hand mesh in a canonical open-palm pose. This makes it accurate and convenient for hand biometrics extraction of patients with impaired hand functions. We also synthesized a large-scale dataset for training the proposed model and built a novel real-world dataset that includes multi-view partial hand scans of 18 subjects and their ground-truth hand biometrics. Extensive results based on both synthetic data and real-world scans validated the effectiveness of our method and showed that it outperforms the state-of-the-art methods. ## REFERENCES [^1]: C.-C. Kuo, H.-Y. Kung, H.-C. Wu, and M.-J. Wang, “Developing a hand sizing system for a hand exoskeleton device based on the Kansei engineering method,” J. Ambient Intell. Humanized Comput., vol. 2020, pp. 1–13, Aug. 2020. [^2]: H. K. Yap et al., “A fully fabric-based bidirectional soft robotic glove for assistance and rehabilitation of hand impaired patients,” IEEE Robot. Autom. Lett., vol. 2, no. 3, pp. 1383–1390, Jul. 2017. [^3]: N. P. Oess, J. Wanek, and A. Curt, “Design and evaluation of a low-cost instrumented glove for hand function assessment,” J. Neuroeng. Rehabil., vol. 9, no. 2, pp. 1–11, Jan. 2012. [^4]: N. E. Krausz, R. A. L. Rorrer, and R. F. F. Weir, “Design and fabrication of a six degree-of-freedom open source hand,” IEEE Trans. Neural Syst. Rehabil. Eng., vol. 24, no. 5, pp. 562–572, May 2016. [^5]: Z. Xu and E. Todorov, “Design of a highly biomimetic anthropomorphic robotic hand towards artificial limb regeneration,” in Proc. IEEE Int. Conf. Robot. Autom. (ICRA), May 2016, pp. 3485–3492. [^6]: D. Magu, A. Aggarwal, P. Behera, and A. Khurana, “Use of finger length ratio as a marker for knee osteoarthritis: A case-control study of 2,456 patients,” medRxiv, 2020. [^7]: S. M. Hussain et al., “Association between index-to-ring finger length ratio and risk of severe knee and hip osteoarthritis requiring total joint replacement,” Rheumatology, vol. 53, no. 7, pp. 1200–1207, Jul. 2014. [^8]: X. Li, R. Wen, Z. Shen, Z. Wang, K. D. K. Luk, and Y. Hu, "A wearable detector for simultaneous finger joint motion measurement," IEEE Trans. Biomed. Circuits Syst., vol. 12, no. 3, pp. 644–654, Jun. 2018. [^9]: S. Tang et al., “A model-based approach to investigate the effect of a long bone fracture on ultrasound strain elastography,” IEEE Trans. Med. Imag., vol. 37, no. 12, pp. 2704–2717, Dec. 2018. [^10]: E. B. Brokaw, I. Black, R. J. Holley, and P. S. Lum, “Hand spring operated movement enhancer (HandSOME): A portable, passive hand exoskeleton for stroke rehabilitation,” IEEE Trans. Neural Syst. Rehabil. Eng., vol. 19, no. 4, pp. 391–399, Aug. 2011. [^11]: S. H. Nasir and O. Troynikov, “Influence of hand movement on skin deformation: A therapeutic glove design perspective,” Appl. Ergonom., vol. 60, pp. 154–162, Apr. 2017. [^12]: Y. Chen et al., “Model-based 3D hand reconstruction via self-supervised learning,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2021, pp. 10446–10455. [^13]: X. Tang, T. Wang, and C.-W. Fu, “Towards accurate alignment in real-time 3D hand-mesh reconstruction,” in Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), Oct. 2021, pp. 11678–11687. [^14]: P. Chen et al., "I2UV-HandNet: Image-to-UV prediction network for accurate and high-fidelity 3D hand mesh modeling," in Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), Oct. 2021, pp. 12909–12918. [^15]: P. Volonghi, G. Baronio, and A. Signoroni, "3D scanning and geometry processing techniques for customised hand orthotics: An experimental assessment," Virtual Phys. Prototyping, vol. 13, no. 2, pp. 105–116, Apr. 2018. [^16]: D. Koutny, D. Palousek, T. Koutecky, A. Zatocilova, J. Rosicky, and M. Janda, "3D digitalization of the human body for use in orthotics and prosthetics," Int. J. Biomed. Biol. Eng., vol. 6, no. 12, pp. 690–697, 2012. [^17]: M. Carfagni et al., “Fast and low cost acquisition and reconstruction system for human hand-wrist-arm anatomy,” Proc. Manuf., vol. 11, pp. 1600–1608, Jan. 2017. [^18]: B. Ceulemans, S.-P. Lu, G. Lafruit, and A. Munteanu, “Robust multiview synthesis for wide-baseline camera arrays,” IEEE Trans. Multimedia, vol. 20, no. 9, pp. 2235–2248, Sep. 2018. [^19]: A. Yu, K. L. Yick, S. P. Ng, and J. Yip, "2D and 3D anatomical analyses of hand dimensions for custom-made gloves," Appl. Ergonom., vol. 44, no. 3, pp. 381–392, May 2013. [^20]: X. Deng et al., “Weakly supervised learning for single depth-based hand shape recovery,” IEEE Trans. Image Process., vol. 30, pp. 532–545, 2021. [^21]: P. Hu, N. N. Kaashki, V. Dadarlat, and A. Munteanu, “Learning to estimate the body shape under clothing from a single 3-D scan,” IEEE Trans. Ind. Informat., vol. 17, no. 6, pp. 3793–3802, Jun. 2021. [^22]: R. Zhao, X. Dai, P. Hu, and A. Munteanu, “PoseNormNet: Identity-preserved posture normalization of 3D body scans in arbitrary postures,” IEEE Trans. Ind. Informat., early access, Feb. 16, 2023, doi:10.1109/TII.2023.3245682. [^23]: S. E. Ovur, H. Su, W. Qi, E. De Momi, and G. Ferrigno, “Novel adaptive sensor fusion methodology for hand pose estimation with multileap motion,” IEEE Trans. Instrum. Meas., vol. 70, pp. 1–8, 2021. [^24]: F. Mueller et al., “GANerated hands for real-time 3D hand tracking from monocular RGB,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., Jun. 2018, pp. 49–59. [^25]: A. El-Sawah, N. D. Georganas, and E. M. Petriu, “A prototype for 3-D hand tracking and posture estimation,” IEEE Trans. Instrum. Meas., vol. 57, no. 8, pp. 1627–1636, Aug. 2008. [^26]: Y. Cai, L. Ge, J. Cai, N. M. Thalmann, and J. Yuan, "3D hand pose estimation using synthetic data and weakly labeled RGB images," IEEE Trans. Pattern Anal. Mach. Intell., vol. 43, no. 11, pp. 3739–3753, Nov. 2021. --- [^27]: L. Ge, H. Liang, J. Yuan, and D. Thalmann, “Robust 3D hand pose estimation from single depth images using multi-view CNNs,” IEEE Trans. Image Process., vol. 27, no. 9, pp. 4422–4436, Sep. 2018. [^28]: B. Wang et al., “Gaze tracking 3-D reconstruction of object with large-scale motion,” IEEE Trans. Instrum. Meas., vol. 72, pp. 1–12, 2023. [^29]: L. Ge, H. Liang, J. Yuan, and D. Thalmann, “Real-time 3D hand pose estimation with 3D convolutional neural networks,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 41, no. 4, pp. 956–970, Apr. 2019. [^30]: F. Albiol, A. Corbi, and A. Albiol, “Geometrical calibration of X-ray imaging with RGB cameras for 3D reconstruction,” IEEE Trans. Med. Imag., vol. 35, no. 8, pp. 1952–1961, Aug. 2016. [^31]: J. Malik et al., “DeepHPS: End-to-end estimation of 3D hand pose and shape by learning from synthetic depth,” in Proc. Int. Conf. 3D Vis. (3DV), Sep. 2018, pp. 110–119. [^32]: D. Kong et al., “Identity-aware hand mesh estimation and personalization from RGB images,” in Computer Vision—ECCV 2022. Cham, Switzerland: Springer, 2022, pp. 536–553. [^33]: Z. Yu et al., “Overcoming the trade-off between accuracy and plausibility in 3D hand shape reconstruction,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., Jun. 2023, pp. 544–553. [^34]: T. Luan et al., “High fidelity 3D hand shape reconstruction via scalable graph frequency decomposition,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., Jul. 2023, pp. 16795–16804. [^35]: J. Romero, D. Tzionas, and M. J. Black, “Embodied hands: Modeling and capturing hands and bodies together,” 2022, arXiv:2201.02610. [^36]: J. Park, Y. Oh, G. Moon, H. Choi, and K. M. Lee, “HandOccNet: Occlusion-robust 3D hand mesh estimation network,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2022, pp. 1486–1495. [^37]: X. Chen et al., “MobRecon: Mobile-friendly hand mesh reconstruction from monocular image,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2022, pp. 20512–20522. [^38]: J. Malik et al., “HandVoxNet++: 3D hand shape and pose estimation using voxel-based neural networks,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 44, no. 12, pp. 8962–8974, Dec. 2022. [^39]: C. Liu, D. Kong, S. Wang, J. Li, and B. Yin, “DLGAN: Depth-preserving latent generative adversarial network for 3D reconstruction,” IEEE Trans. Multimedia, vol. 23, pp. 2843–2856, 2021. [^40]: R. Q. Charles, H. Su, M. Kaichun, and L. J. Guibas, “PointNet: Deep learning on point sets for 3D classification and segmentation,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jul. 2017, pp. 77–85. [^41]: W. Yuan, T. Khot, D. Held, C. Mertz, and M. Hebert, “PCN: Point completion network,” in Proc. Int. Conf. 3D Vis. (3DV), Sep. 2018, pp. 728–737. [^42]: P. J. Besl and N. D. McKay, “Method for registration of 3-D shapes,” Proc. SPIE, vol. 1611, pp. 586–606, Apr. 1992. [^43]: P. Hu and A. Munteanu, “Method for registration of 3D shapes without overlap for known 3D priors,” Electron. Lett., vol. 57, no. 9, pp. 357–359, Apr. 2021. [^44]: Y. Rong, T. Shiratori, and H. Joo, "FrankMocap: A monocular 3D whole-body pose estimation system via regression and integration," in Proc. IEEE/CVF Int. Conf. Comput. Vis. Workshops (ICCVW), Oct. 2021, pp. 1749–1759. [^45]: G. Pavlakos et al., “Expressive body capture: 3D hands, face, and body from a single image,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2019, pp. 10967–10977. [^46]: G. Varol et al., “Learning from synthetic humans,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jul. 2017, pp. 4627–4635. [^47]: M. Gschwandtner, R. Kwitt, A. Uhl, and W. Pree, “BlenSor: Blender sensor simulation toolbox,” in Proc. Int. Symp. Vis. Comput. Cham, Switzerland: Springer, 2011, pp. 199–208. [^48]: Y. Yang, H. Zhou, Y. Song, and P. Vink, “Identify dominant dimensions of 3D hand shapes using statistical shape model and deep neural network,” Appl. Ergonom., vol. 96, Oct. 2021, Art. no. 103462. [^49]: F. Arifi, M. S. AmetiV, and Z. Metaj, “Stature and its estimation utilizing length of hand measurements of both gender adolescents from western region of Kosovo,” Sports Injr Med, vol. 5, p. 171, Jan. 2021. [^50]: P. Xiang et al., “SnowflakeNet: Point cloud completion by snowflake point deconvolution with skip-transformer,” in Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), Oct. 2021, pp. 5479–5489. [^51]: P. Hu, R. Zhao, X. Dai, and A. Munteanu, “Predicting high-fidelity human body models from impaired point clouds,” Signal Process., vol. 192, Mar. 2022, Art. no. 108375. [^52]: H. Fan, H. Su, and L. Guibas, “A point set generation network for 3D object reconstruction from a single image,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jul. 2017, pp. 2463–2471. ![Image](images/bb3d2afd03d55767d36e781a57f230d7cdb6d2f887ea30eec79f8671778220fd.jpg) Xinxin Dai (Graduate Student Member, IEEE) received the M.E. degree from the Guangdong University of Technology, Guangzhou, China, in 2019. She is currently pursuing the Ph.D. degree with the Department of Electronics and Informatics, Vrije Universiteit Brussel (VUB), Brussels, Belgium. Her current research interests include point cloud processing, 3-D hand reconstruction, biometrics, and person identification. ![Image](images/a8c0bab2f989c2b4baa485bd90e44121a3bd6a175e7168b9d6b95a1bcaf435c8.jpg) Ran Zhao received the M.S. degree in mathematics from East China Normal University, Shanghai, China, in 2018, and the M.E. degree in applied computer science from Vrije Universiteit Brussel (VUB), Brussels, Belgium, in 2021. She is currently pursuing the Ph.D. degree with the Department of Electronics and Informatics, VUB. Her research focuses on point cloud processing, 3-D hand scanning, and biometrics. ![Image](images/f6c98aee43b3a528229900cc4a7ffa1aa3a144638584363ddecf69a4f4adae03.jpg) Pengpeng Hu was a Senior Researcher with the Department of Electronics and Informatics, Vrije Universiteit Brussel (VUB), Brussels, Belgium. In 2016, he was a Visiting Scholar with the School of Informatics, Edinburgh University, Edinburgh, U.K. In 2017, he was a Post-Doctoral Fellow with the Department of Computer and Information Sciences, Northumbria University, Newcastle upon Tyne, U.K. He is an Assistant Professor with the Center for Computational Science and Mathematical Modeling, Coventry University, Coventry, U.K. Since 2018, he has been with VUB. His current research interests include biometrics, geometric deep learning, 3-D human body reconstruction, point cloud processing, and measurement. Dr. Hu is the Early Career Advisory Board Member for the journals Measurement, and Measurement: Sensors. He is also the Topical Advisory Panel Member for the journals Sensors (MDPI) and Designs (MDPI). He is the Guest Editor of the Sensors (MDPI), the Technical Support Chair of BMVC 2018, and a member of the Program Committee in SKIMA 2017, SKIMA 2018, and SKIMA 2019. He is the outstanding paper winner of the Emerald Literati Award 2019. Adrian Munteanu received the M.Sc. degree in electronics and telecommunications from the Politehnica University of Bucharest, Bucharest, Romania, in 1994, the M.Sc. degree in biomedical engineering from the University of Patras, Patras, Greece, in 1996, and the Ph.D. degree in applied sciences from Vrije Universiteit Brussel, Brussels, Belgium, in 2003. ![Image](images/4334b96e0b9fae7dff585d352f3df862756126c736a4f18a3e0e461875550c51.jpg) From 2004 to 2010, he was a Post-Doctoral Fellow with the Fund for Scientific Research–Flanders (FWO), Vlaanderen, Belgium, and since 2007, he has been a Professor with VUB. He is currently a Professor with the Department of Electronics and Informatics, Vrije Universiteit Brussel. He made contribution to more than 400 publications and holds seven patents. Dr. Munteanu was a recipient of the 2004 BARCO-FWO Prize for the Ph.D. work and the Co-Recipient of the Most Cited Paper Award from Elsevier for 2007. He was an Associate Editor for the IEEE TRANSACTIONS ON MULTIMEDIA and is currently an Associate Editor for the IEEE TRANSACTIONS ON IMAGE PROCESSING.