# MobRecon: Mobile-Friendly Hand Mesh Reconstruction from Monocular Image

Xingyu Chen $^{1*}$ Yufeng Liu $^{3}$ Yajiao Dong $^{1}$ Xiong Zhang $^{2}$ Chongyang Ma $^{1}$

Yanmin Xiong $^{1}$ Yuan Zhang $^{1}$ Xiaoyan Guo $^{1}$

 $^{1}$Y-tech, Kuaishou Technology $^{2}$ YY Live, Baidu Inc.

 $^{3}$SEU-ALLEN Joint Center, Institute for Brain and Intelligence, Southeast University, China.

## Abstract

In this work, we propose a framework for single-view hand mesh reconstruction, which can simultaneously achieve high reconstruction accuracy, fast inference speed, and temporal coherence. Specifically, for 2D encoding, we propose lightweight yet effective stacked structures. Regarding 3D decoding, we provide an efficient graph operator, namely depth-separable spiral convolution. Moreover, we present a novel feature lifting module for bridging the gap between 2D and 3D representations. This module begins with a map-based position regression (MapReg) block to integrate the merits of both heatmap encoding and position regression paradigms for improved 2D accuracy and temporal coherence. Furthermore, MapReg is followed by pose pooling and pose-to-vertex lifting approaches, which transform 2D pose encodings to semantic features of 3D vertices. Overall, our hand reconstruction framework, called MobRecon, comprises affordable computational costs and miniature model size, which reaches a high inference speed of 83FPS on Apple A14 CPU. Extensive experiments on popular datasets such as FreiHAND, RHD, and HO3Dv2 demonstrate that our MobRecon achieves superior performance on reconstruction accuracy and temporal coherence. Our code is publicly available at <https://github.com/SeanChenxy/HandMesh>.

### 1. Introduction

Single-view hand mesh reconstruction has been extensively investigated for years due to its wide range of applications in AR/VR [^24] [^65], behavior understanding [^36] [^56], etc. Tremendous research efforts have been made towards this task, including [^15] [^79] [^42] [^75], to name a few.

The primary focus of typical existing methods is the reconstruction accuracy [^45] [^46], while real-world applications additionally necessitate inference efficiency and temporal consistency. In particular, 3D hand information

![Image](images/ce5609a039795a331b1e8e07c0012289ab8349ff2953a11eeaf8a8076972c4f5.jpg)

<div style="text-align: center;">Figure 1. Accuracy vs. inference speed. The marker size is related to the model size. Besides, our tailored method can run at a fast speed on mobile CPUs.</div>

is a vital component in mobile applications [^65], where the devices comprise limited memory and computational budgets. Thereby, this work aims to explore 3D hand reconstruction for mobile platforms.

A typical pipeline for single-view hand reconstruction includes three phases: 2D encoding, 2D-to-3D mapping, and 3D decoding. In 2D encoding, existing approaches [^42] [^9] [^45] [^46] usually adopt computationally intensive networks [^26] [^67] to handle this highly non-linear task, which are hard to deploy on mobile devices. Instead, if naively leveraging a mature mobile network (e.g., [^28]) which is not tailored for our target task, the reconstruction accuracy dramatically degrades [^18]. Hence, our motivation is to develop a lightweight 2D encoding structure tailored to balance the inference efficiency and accuracy. Besides, the efficiency of 2D-to-3D mapping and 3D decoding remains relatively unexplored. Thus, we intend to explore a lightweight yet effective lifting method to tackle the 2D-to-3D mapping problem and design an efficient graph operator for processing of 3D mesh data.

Although as crucial as accuracy in real-world applications, temporal coherence is usually neglected in the task of 3D hand reconstruction. Previous methods [^11] [^37] [^41] [^49] adopt sequential models to incorporate both past and future

---

semantic information for stable predictions. Since this methodology is offline or computationally expensive, these approaches are difficult to use for mobile applications. Hence, we are inspired to explore the temporal coherence with a non-sequential method.

In this work, we propose Mobile Mesh Reconstruction (MobRecon) for 3D hand to simultaneously explore superior accuracy, efficiency, and temporal coherence. For 2D encoding, we leverage the spirit of the hourglass network [^55] to design efficient stacked encoding structures. As for 3D decoding, we propose a depth-separable spiral convolution (DSConv), which is a novel graph operator based on spiral sampling [^44]. The DSConv is inspired by depth-wise separable convolution [^29], leading to efficient handling of graph-structured mesh data. Regarding the 2D-to-3D mapping, we propose a feature lifting module with map-based position regression (MapReg), pose pooling, and pose-to-vertex lifting (PVL) approaches. In this module, we first investigate the pros and cons of 2D pose representations based on heatmap or position regression, and then propose a hybrid method MapReg to simultaneously improve 2D pose accuracy and temporal consistency. Furthermore, the PVL transforms 2D pose encodings to 3D vertex features based on a learnable lifting matrix, resulting in enhanced 3D accuracy and temporal consistency. Compared to traditional approaches based on fully connected operation in a latent space [^42] [^15] [^9], our feature lifting module also significantly reduces the model size. In addition, we build a synthetic dataset with uniformly distributed hand poses and viewpoints. Referring to Figure 1, we achieve better performance in terms of accuracy, speed, and model size.

Our main contributions are summarized as follows:

- We propose MobRecon as a mobile-friendly pipeline for hand mesh reconstruction, which only involves 123M multiply-add operations (Mult-Adds) and 5M parameters and can run up to 83FPS on Apple A14 CPU.

- We present lightweight stacked structures and DSConv for efficient 2D encoding and 3D decoding.

- We propose a novel feature lifting module with MapReg, pose pooling, and PVL methods to bridge the 2D and 3D representations.

• We demonstrate that our method achieves superior performance in terms of model efficiency, reconstruction accuracy, and temporal coherence via comprehensive evaluations and comparisons with state-of-the-art approaches.

### 2. Related Work

Hand mesh estimation. Popular hand mesh estimation methods can be divided into five types, whose core ideas are based on the parametric model, voxel representation, implicit function, UV map, and vertex position, respectively.

Model-based approaches [^76] [^79] [^71] [^25] [^75] [^4] [^1] [^82] [^10] [^77] [^6] [^2] [^35] [^47] [^73] [^74] typically use MANO [^58] as the parametric model, which factorizes a hand mesh into coefficients of shape and pose. This pipeline, however, is not suitable for usage with a lightweight network because the coefficient estimation is a highly abstract problem that ignores spatial correlations [^18].

Voxel-based approaches [^33] [^52] [^70] [^54] describe the 3D data in a 2.5D manner. Moon et al. [^52] proposed I2L-MeshNet, which divided the voxel space into three l�ixel spaces and used 1D heatmaps to reduce memory consumption. Despite this optimization, I2L-MeshNet still requires massive memory occupation to process the lixel-style 2.5D heatmaps. Hence, the voxel-based approach is not friendly for memory-constrained mobile devices.

The implicit function [^50] has merits of continuity and high resolution, which is recently used for digitizing articulated human [^51] [^34] [^3] [^59] [^32] [^57] [^38] [^39]. However, the implicit methods usually need to compute thousands of 3D points, lacking efficiency in mobile settings.

Chen et al. [^8] treated hand mesh reconstruction as an image-to-image translation task, employing UV map to connect 2D and 3D spaces. This pipeline could be improved by incorporating geometry correlations.

Vertex-based methods [^15] [^42] [^9] [^45] [^46] predict 3D vertex coordinates directly, which usually follow a procedure of 2D encoding, 2D-to-3D mapping, and 3D decoding. For example, Kulon et al. [^42] designed an encoder-decoder based on ResNet [^26], global pooling, and spiral convolution (SpiralConv) [^44] to obtain 3D vertex coordinates. We re-construct the vertex-based pipeline with efficient modules, i.e., lightweight stacked structures for 2D encoding, feature lifting module for 2D-to-3D mapping, and DSConv for 3D decoding. As a result, we achieve high reconstruction accuracy and across-time consistency.

Lightweight networks. For a timely fashion on a computationally limited platform, lightweight networks have been studied for years such as [^29] [^60] [^28] [^23]. We leverage popular efficient ideas to design stacked networks for Euclidean 2D images and a graph network for non-Euclidean 3D meshes. More specifically, we propose a feature lifting module to deal with the problem of 2D-to-3D mapping efficiently. As a highly related work, MobileHand [^18] was able to run at 75FPS on a mobile CPU. In contrast, our MobRecon achieves more powerful performance on accuracy and inference speed (as shown in Table 5).

Temporally coherent human reconstruction. There has been limited research on the temporal coherence of human/hand mesh reconstruction, yet it is as crucial as reconstruction accuracy in real-world applications. Previous research [^11] [^37] [^41] [^49] has concentrated on temporal performance with temporal approaches. Kocabas et al. [^41] used a bi-directional gated recurrent unit [^12] to fuse

---

![Image](images/91ef63ab1286826cad733f9284b5c7337fdf77e25c18600d937d816c3cf08f88.jpg)

<div style="text-align: center;">Figure 2. Overview of our MobRecon framework.</div>

across-time features so that SMPL [^48] parameters could be regressed with temporal cues. This sequential manner could raise computational costs and even requires future information [^11]. In contrast, we design a feature lifting module with MapReg to enhance the temporal coherence for a non-sequential single-view method.

Pixel-aligned representation. Convolutional features are made up of dense and regularly structured 2D cues, but sparse and unordered points represent 3D data. To better extract image features to describe 3D information, recent works usually adopt pixel-aligned representations [^59] [^53] [^19] [^74] [^22] [^78]. Inspired by them, we use the idea of pixel alignment for feature lifting and design PVL to transform pose-aligned encodings to vertex features.

It is noteworthy that the above-mentioned literature used heatmap [^19] [^22] [^74] or position [^59] [^78] as the 2D representations. Li et al. [^43] analyzed the heatmap- and position-based human pose in terms of accuracy and proposed RLE to achieve high-accuracy regression. We consider these two representations from the view of temporal coherence and propose MapReg to integrate the merits of heatmap- and position-based 2D representations. Compared to [^43], we have different insights and study perspectives, so RLE and our MapReg could complement each other.

### 3. Our Method

With a single-view image as the input, we aim to infer 3D hand mesh with predicted vertices $\mathbf{V} = \{\mathbf{v}_i\}_{i=1}^V$ and predefined faces $\mathbf{C} = \{\mathbf{c}_i\}_{i=1}^C$. Figure 2 illustrates the overall architecture of MobRecon that includes three phases. For 2D encoding, we leverage convolutional networks to extract image features. In the feature lifting module, 2D pose is delineated with MapReg, followed by pose pooling to retrieve pose-aligned features. Then, vertex features are obtained with PVL. For 3D decoding, we develop an efficient graph network to predict vertex coordinates.

#### 3.1. Stacked Encoding Network

Inspired by the hourglass network [^55], we develop a stacked encoding network to obtain gradually refined encoding features. As shown in Figure 3, the stacked network consists of two groups of cascaded encoding blocks, the first of which is followed by an upsampling module for feature

![Image](images/f1d4b27eacbdb8f769387e575f820f7f22ed645b5440a6d637f45b77496243e8.jpg)

<div style="text-align: center;">Figure 3. Our tailored stacked encoding structure.</div>

fusion. With a single-view image as the input, the encoding feature $\mathbf{F}^e \in \mathbb{R}^{C^e \times H^e \times W^e}$ is generated, where C, H, W denote tensor channel size, height, and width.

We design two block alternatives. According to DenseNet [^31] and SENet [^30], we present a dense block to form DenseStack (see Figure 3). To further reduce model size, we leverage the ghost operation [^23] to develop GhostStack, where cheap operations can produce ghost features based on primary features. With $128 \times 128$ input resolution, the DenseStack involves $373.0M$ Mult-Adds and $6.6M$ parameters while the GhostStack consists of $96.2M$ Mult-Adds and $5.0M$ parameters. In contrast, a stacked network with ResNet18 has $2391.3M$ Mult-Adds and $25.2M$ parameters, prohibiting it from mobile applications.

#### 3.2. Feature Lifting Module

Lifting means 2D-to-3D mapping [^71]. For feature lifting, two problems should be concerned: (1) how to collect 2D features and (2) how to map them to 3D domain. To this end, previous methods [^42] [^15] [^9] tend to embed $F^e$ as a latent vector via the global average pooling operation. Then, the latent vector is mapped to 3D domain with a fully connected layer (FC), and vertex features are obtained with vector re-arrangement. This manner brings increased model size due to the large dimension of FC, i.e., this layer contains 3.2M parameters when $C^e = 256$.

Recent researches report pixel-aligned feature extraction based on 2D landmarks and pixel-aligned feature pooling [^59] [^19] [^22] [^74] [^78]. Heatmap $H^{p}$ is usually employed to

---

![Image](images/0e6eb2fd3c3c91e46339e63509bcbfa4a62c5a75ef9739a16ef4c75e195a09da.jpg)

<div style="text-align: center;">Figure 4. Comparisons of 2D representations and pose pooling methods. (a) heatmap, (b) heatmap + soft-argmax, (c) regression-based position, (d) MapReg-based position, (e) joint-wise pooling with a heatmap, (f) grid sampling with a 2D position. For better visualization, only 4 landmarks are illustrated.</div>

encode 2D landmarks [^19] [^15] [^9] [^75], which derives more accurate landmarks compared with direct regression of the 2D positions $L^p$ [^43] [^66] [^7] [^63] [^68].

Map-based position regression. Figure 4 comprehensively investigates these 2D representations. It is seen that $H^p$ is a high-resolution representation (Figure 4(a)). As shown in Figure 4(b), soft-argmax [^64] is a differentiable technique that decodes $H^p$ to the corresponding 2D position. In contrast, direct position regression (Figure 4(c)) is a low-resolution representation, which can yield $L^p$ without resorting to $H^p$. It has been proven that the human pose estimation task requires both low- and high-resolution encodings [^67]. Hence, the skip connection that fuses features is the primary cause behind the superior accuracy of $H^p$ [^55]. However, our critical insight is that the global feature (with resolution of $1 \times 1$) used to predict $L^p$ (see Figure 4(c)) shall induce better temporal coherence because of global semantics and receptive field. This global property can better describe articulated relation of hand pose. Conversely, $H^p$ is predicted with convolution and high-resolution features, where limited receptive field leads to the lack of inter-landmark constraints.

As shown in Figure 4(d), we propose a middle-resolution method MapReg to combine the advantages of heatmap-and position-based paradigms by (1) fusing low- and high-resolution features for accuracy and (2) using global receptive field for temporal coherence. To this end, we incorporate the skip connection in the position regression paradigm, producing a spatially small-size (e.g. $16 \times 16$) feature map. Each feature channel is then flattened into a vector, followed by a multi-layer perceptron (MLP) to generate a 2D position. In this manner, we obtain middle-resolution spatial complexity which is superior than heatmap since only two $2 \times$ -upsampling operations are involved.

Pose pooling. After obtaining 2D representations, pixel-aligned features can be retrieved. We call this process pose pooling and capture pose-aligned features with N 2D joint landmarks. If heatmap $\mathbf{H}^{p}$ is employed as the 2D representation, the pose pooling is conducted with joint-wise pooling [^74] (Figure 4(e)), which can be given as

 $$ \mathbf{F}^{p}=[\Psi(\mathbf{F}^{e}\odot\operatorname{i n t e r p o l a t i o n}(\mathbf{H}_{i}^{p}))]_{i=1,2,\ldots,N}, $$

where $[.]$ denotes concatenation. First, the spatial size of $\mathbf{H}^p$ is interpolated to $H^e \times W^e$, and then channel-wise multiplication is adopted between interpolated $\mathbf{H}^p$ and $\mathbf{F}^e$. In this manner, features unrelated to joint landmarks are suppressed. For extracting joint-wise features, feature reduction $\Psi$ is designed. In detail, $\Psi$ denotes global max-pooling or spatial sum reduction, which produces a $C^e$-length feature vector. After concatenation, $\mathbf{F}^p \in \mathbb{R}^{N \times C^e}$ indicates the pose-aligned feature.

If we use $L^p$ instead of $H^p$ to describe 2D pose, pose pooling can be achieved with grid sampling [^59] (Figure 4(f)) as follows,

 $$ \mathbf{F}^{p}=[\mathbf{F}^{e}(\mathbf{L}_{i}^{p})]_{i=1,2,\ldots,N}. $$

As a result, the convolutional encoding $\mathbf{F}^{e}$ is transformed to the pose-aligned representation $\mathbf{F}^{p}$.

Pose-to-vertex lifting. Referring to Figure 2, we design a linear operator for feature mapping towards 3D space with a few learnable parameters, namely PVL. MANO-style hand mesh [^58] comprises V vertices and N joints, where $V = 778$, $N = 21$. Because $V \gg N$, it is hard to transform $\mathbf{F}^p$ to V vertex features. Instead, we downsample the template mesh 4-times by a factor of 2 [^14], and obtain a minimal-size hand mesh with $V^{mini} = 49$ vertices. Then, we design a learnable lifting matrix $\mathbf{M}^l \in \mathbb{R}^{V^{mini} \times N}$ for 2D-to-3D feature mapping. Thereby, PVL is given as

 $$ \mathbf{F}^{m i n i\_{v}}=\mathbf{M}^{l}\cdot\mathbf{F}^{p}, $$

where $\mathbf{F}^{mini\_v}$ denotes minimal-size vertex feature. The PVL reduces the computational complexity of feautre mapping from $\mathcal{O}(V^{mini}C^{e2})$ [^42] [^9] [^15] to $\mathcal{O}(NV^{mini}C^{e})$.

#### 3.3. Depth-Separable SpiralConv

As a graph operator, SpiralConv [^44] is equivalent to the Euclidean convolution, which designs a spiral neighbor as

 $$ \begin{aligned}0-ring(\mathbf{v})&=\{\mathbf{v}\}\ $ k+1)-ring(\mathbf{v})&=\mathbb{N}(k-ring(\mathbf{v}))\setminus k-disk(\mathbf{v})\\k-disk(\mathbf{v})&=\cup_{i=0,\ldots,k}i-ring(\mathbf{v}),\end{aligned} $$

where N extracts the neighbourhood of a vertex v. With k-disk(v), SpiralConv formulates convolution as a sequential problem and leverages LSTM [^27] for feature fusion:

 $$ \mathbf{f}_{\mathbf{v}}^{{o u t}}=\operatorname{L S T M}(\mathbf{f}_{\mathbf{v}^{\prime}}),\quad\mathbf{v}^{\prime}\in k\mathbf{-d i s k}(\mathbf{v}), $$

---

![Image](images/6d80f76f535f29e4f36e2ff14868ddae80f7dd1c3012f1ae20d3b92b984a2542.jpg)

<div style="text-align: center;">Figure 5. Comparison of SpiralConv, SpiralConv++, and DSConv. For better visualization, a case with S = D = 3 is shown.</div>

where $\mathbf{f}_{\mathbf{v}} \in \mathbb{R}^{1 \times D}$ denotes the feature vector at $\mathbf{v}$ with dimension D. SpiralConv with LSTM could be potentially slow because of serial sequence processing.

By explicitly formulating the order of aggregating neighboring vertices, SpiralConv++ [^16] presents an efficient version of SpiralConv. For high efficiency, SpiralConv++ only adopts a fixed-size spiral neighbor with S vertices and leverages FC to fuse these features:

 $$ \mathbf{f}_{\mathbf{v}}^{{o u t}}=\mathbf{W}\cdot[\mathbf{f}_{\mathbf{v}^{\prime}}]+\mathbf{b},\quad\mathbf{v}^{\prime}\in k\mathbf{-d i s k}(\mathbf{v})_{S}, $$

where $[\cdot]$ denotes concatenation; k-disk( $\mathbf{v}$) $_{S}$ contains the first S elements in k-disk( $\mathbf{v}$); $\mathbf{W}$ and $\mathbf{b}$ are learnable parameters. SpiralConv++ significantly increases model size because of the large dimension of FC.

As an efficient graph convolution, we propose DSConv with the spirit of depth-wise separable convolution [^29]. For a vertex v, k-disk(v) $_{S}$ is sampled following Equation 4. DSConv comprises a depth-wise operation and a point-wise operation, the former of which can be formulated as

 $$ \mathbf{f}_{\mathbf{v}}^{d}=[\mathbf{W}_{i}^{d}\cdot[\mathbf{f}_{\mathbf{v}^{\prime},i}]]_{i=1}^{D},\quad\mathbf{v}^{\prime}\in k\mathbf{-d i s k}(\mathbf{v})_{S}. $$

Then, point-wise operation can be formulated as

 $$ \mathbf{f}_{\mathbf{v}}^{{o u t}}=\mathbf{W}^{p}\cdot\mathbf{f}_{\mathbf{v}}^{d}. $$

Figure 5 illustrates the difference among SpiralConv, SpiralConv++, and our DSConv. The SpiralConv++'s computational complexity obeys $\mathcal{O}(SD^2)$, whereas that of DSConv is $\mathcal{O}(SD + D^2)$. Hence, we essentially improve the efficiency with the separable structure.

The 3D decoder is built with four blocks, each of which involves upsampling, DSConv, and ReLU. In each block, vertex features are upsampled by a factor of 2 and then processed by DSConv. Finally, vertex coordinates V are predicted by a DSConv.

#### 3.4. Loss Functions

Accuracy loss. We use $L_{1}$ norm for the 3D mesh loss $\mathcal{L}_{mesh}$ and 2D pose loss $\mathcal{L}_{pose2D}$. Normal loss $\mathcal{L}_{norm}$ and edge length loss $\mathcal{L}_{edge}$ are adopted for mesh smoothness according to [^15]. Formally, we have

 $$ \begin{aligned}&\mathcal{L}_{m e s h}=||\mathbf{V}-\mathbf{V}^{\star}||_{1},\mathcal{L}_{p o s e2D}=||\mathbf{L}^{p}-\mathbf{L}^{p,\star}||_{1}\\&\mathcal{L}_{n o r m}=\sum_{\mathbf{c}\in\mathbf{C}}\sum_{(i,j)\subset\mathbf{c}}|\frac{\mathbf{V}_{i}-\mathbf{V}_{j}}{||\mathbf{V}_{i}-\mathbf{V}_{j}||_{2}}\cdot\mathbf{n}_{\mathbf{c}}^{\star}|\\&\mathcal{L}_{e d g e}=\sum_{\mathbf{c}\in\mathbf{C}}\sum_{(i,j)\subset\mathbf{c}}|||\mathbf{V}_{i}-\mathbf{V}_{j}||_{2}-||\mathbf{V}_{i}^{\star}-\mathbf{V}_{j}^{\star}||_{2}|,\\ \end{aligned} $$

where C, V are face and vertex sets of a mesh; $n_c^*$ indicates unit normal vector of face c; $*$ denotes the ground truth.

Consistency loss. Inspired by the self-supervision task [^61] [^80], we design the consistency supervision based on augmentation without the need of temporal data. That is, two views can be derived from an image sample based on 2D affine transformation and color jitter. We denote relative affine transformation between two views as $T_{1\rightarrow2}$, which contains relative rotation $R_{1\rightarrow2}$. Similar to [^20] [^70], we conduct consistency supervision in both 3D and 2D space:

 $$ \begin{aligned}\mathcal{L}_{con3D}&=||R_{1\rightarrow2}\mathbf{V}_{view1}-\mathbf{V}_{view2}||_{1}\\\mathcal{L}_{con2D}&=||T_{1\rightarrow2}\mathbf{L}^{p}_{view1}-\mathbf{L}^{p}_{view2}||_{1}.\end{aligned} $$

Although T contains the variations of rotation, shift, and scale in 2D space, only R affects $\mathbf{V}$ because it is rooted by the wrist landmark.

 $$ \begin{aligned}Our~overall~loss~function~is~\mathcal{L}_{total}=\mathcal{L}_{mesh}+\mathcal{L}_{pose2D}+\\\mathcal{L}_{norm}+\mathcal{L}_{edge}+\mathcal{L}_{con2D}+\mathcal{L}_{con3D}.\end{aligned} $$

### 4. Experiments

#### 4.1. Implementation Details

We use the Adam optimizer [^40] to train the network with a mini-batch size of 32. All models in our experiments are trained for 38 epochs. The initial learning rate is $10^{-3}$, which is divided by 10 at the 30th epoch. As for hyperparameters, the input resolution is $128 \times 128$, and $S = 9$, $C^e = 256$, $D = \{256, 128, 64, 32\}$.

Existing datasets for 3D hand pose estimation (e.g. [^82] [^42] [^21]) usually suffer from long-tailed distribution of hand pose and viewpoint. Hence, we develop a synthetic dataset with 1520 poses and 216 viewpoints, both of which are uniformly distributed in their respective spaces. Because of this superior property, it can serve as a good complement during training. We redirect the readers to supplementary material for details.

#### 4.2. Evaluation Criterion

We conduct experiments on several commonly-used benchmarks as listed below.

FreiHAND [^82] contains 130,240 training images and 3,960 evaluation samples. The annotations of the evaluation set are not available, so we submit our predictions to the official server for online evaluation.

---

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'>2D encoding</td><td style='text-align: center; word-wrap: break-word;'>Mult-Adds</td><td style='text-align: center; word-wrap: break-word;'>#Param</td><td style='text-align: center; word-wrap: break-word;'>Fine-tuning data</td><td style='text-align: center; word-wrap: break-word;'>PJ $\downarrow$</td></tr><tr><td colspan="5">From-scratch fine-tuning</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>ResNet18-Stack</td><td style='text-align: center; word-wrap: break-word;'>2391.3M</td><td style='text-align: center; word-wrap: break-word;'>25.2M</td><td style='text-align: center; word-wrap: break-word;'>FreiHAND</td><td style='text-align: center; word-wrap: break-word;'>8.86</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>MobileNet-Stack</td><td style='text-align: center; word-wrap: break-word;'>100.2M</td><td style='text-align: center; word-wrap: break-word;'>1.7M</td><td style='text-align: center; word-wrap: break-word;'>FreiHAND</td><td style='text-align: center; word-wrap: break-word;'>14.20</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>GhostStack</td><td style='text-align: center; word-wrap: break-word;'>96.2M</td><td style='text-align: center; word-wrap: break-word;'>5.0M</td><td style='text-align: center; word-wrap: break-word;'>FreiHAND</td><td style='text-align: center; word-wrap: break-word;'>13.09</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>DenseStack</td><td style='text-align: center; word-wrap: break-word;'>373.0M</td><td style='text-align: center; word-wrap: break-word;'>6.6M</td><td style='text-align: center; word-wrap: break-word;'>FreiHAND</td><td style='text-align: center; word-wrap: break-word;'>9.56</td></tr><tr><td colspan="5">Pre-training w/ ImageNet</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>ResNet18-Stack</td><td style='text-align: center; word-wrap: break-word;'>2391.3M</td><td style='text-align: center; word-wrap: break-word;'>25.2M</td><td style='text-align: center; word-wrap: break-word;'>FreiHAND</td><td style='text-align: center; word-wrap: break-word;'>8.21</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>MobileNet-Stack</td><td style='text-align: center; word-wrap: break-word;'>100.2M</td><td style='text-align: center; word-wrap: break-word;'>1.7M</td><td style='text-align: center; word-wrap: break-word;'>FreiHAND</td><td style='text-align: center; word-wrap: break-word;'>13.68</td></tr><tr><td colspan="5">Pre-training w/ ours</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>MobileNet-Stack</td><td style='text-align: center; word-wrap: break-word;'>100.2M</td><td style='text-align: center; word-wrap: break-word;'>1.7M</td><td style='text-align: center; word-wrap: break-word;'>FreiHAND</td><td style='text-align: center; word-wrap: break-word;'>12.45</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>GhostStack</td><td style='text-align: center; word-wrap: break-word;'>96.2M</td><td style='text-align: center; word-wrap: break-word;'>5.0M</td><td style='text-align: center; word-wrap: break-word;'>FreiHAND</td><td style='text-align: center; word-wrap: break-word;'>10.05</td></tr><tr><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'>FreiHAND+ours</td><td style='text-align: center; word-wrap: break-word;'>8.89</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>DenseStack</td><td style='text-align: center; word-wrap: break-word;'>373.0M</td><td style='text-align: center; word-wrap: break-word;'>6.6M</td><td style='text-align: center; word-wrap: break-word;'>FreiHAND</td><td style='text-align: center; word-wrap: break-word;'>7.77</td></tr><tr><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'>FreiHAND+ours</td><td style='text-align: center; word-wrap: break-word;'>7.55</td></tr></table>

<div style="text-align: center;">Table 1. Ablation studies of 2D encoding methods and our complement data on FreiHAND. Mult-Adds and #Param are w.r.t. the 2D encoding network.</div>

Rendered Hand Pose Dataset (RHD) [^81] consists of 41,258 and 2,728 synthetic hand data for training and testing on hand pose estimation, respectively.

HO3Dv2 [^21] is a 3D hand-object dataset that contains 66,034 training samples and 11,524 evaluation samples. The annotations of the evaluation set are not available, so we use the official server for online evaluation. We also use this dataset to evaluate temporal performance.

We use the following metrics in quantitative evaluations.

MPJPE/MPVPE measures the mean per joint/vertex position error by Euclidean distance (mm) between the estimated and ground-truth coordinates.

PA-MPJPE/MPVPE is a modification of MPJPE/MPVPE with Procrustes analysis [^17], ignoring global variation. For conciseness, this metric is abbreviated as PJ/PV.

Acc captures the acceleration of 2D/3D joint landmarks in pixel/s $^{2}$ or mm/s $^{2}$ to reflect temporal coherence.

AUC means the area under the curve of PCK (percentage of correct keypoints) vs. error thresholds of $n \sim 50$mm (for 3D measurement) or 0~30 pixel (for 2D measurement). According to [^82] [^81], n = 0 or 20.

F-Score is the harmonic mean between recall and precision between two meshes w.r.t. a specific distance threshold. F@5/F@15 corresponds to a threshold of 5mm/15mm.

Mult-Adds counts multiply-add operations.

#### Param indicates the number of parameters

#### 4.3. Ablation studies

Our stacked networks and complement data. We use CMR [^9] as the baseline, which adopts FC for 2D-to-3D feature mapping and SpiralConv++ for 3D decoding. With the same hyperparameters, we only change the 2D encoding network and data setting for comparisons. On the one hand, we study different stacked structures with from-scratch training. As shown in Table 1, we use ResNet [^26]

<div style="text-align: center;">H^{p} or H^{p}+soft-argmax w/ heatmap supervision</div>

![Image](images/f0d3607bd860690426e0bddb9b4c505d32e87d605e93d71377497f717b53c83c.jpg)

<div style="text-align: center;"> $H^{p}$+soft-argmax w/o heatmap supervision</div>

![Image](images/6f9d742cb03e8aa14c12a0546cbc69af13e174462fef63447c8d7017a82156da.jpg)

![Image](images/89b5d681ae8f3f6d758ebd3ac95938145499ea2f0b736b0c09489e99fc3b94e2.jpg)

<div style="text-align: center;">Figure 6. Visualization of maps in various 2D pose representations. Different from heatmap that focus on individual landmark, the map in MapReg adaptively describes inter-landmark constraints.</div>

![Image](images/3c8929e76249b97b09a2da327cabbc0216091aaacf0f5d27cb00803b10357a9e.jpg)

<div style="text-align: center;">0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42 44 46 48 vertex</div>

<div style="text-align: center;">Figure 7. Learned lifting matrix.</div>

![Image](images/8095bfe1239095701570f730303c69973dc9d005db9d3b3d8b72597d70528483.jpg)

<div style="text-align: center;">Figure 8. Highly relevant connections for several vertices in the lifting matrix. The numbers are vertex indices.</div>

and MobileNet [^60] to design stacked structures, the former of which contains intensive computation costs. Although MobileNet is computational tractability, it whittles down the performance (i.e., PAMPJPE) by a large margin. On the other hand, ImageNet [^13] is used for classification task, the knowledge from which is hard to transfer towards 2D/3D position regression. Consequently, the ImageNet pre-training brings less than 1mm PAMPJPE improvement.

Referring to Table 1, our complement data can be included in both pre-training and fine-tuning steps, and DenseStack/GhostStack induces 7.55/8.89mm PAMPJPE. Hence, we significantly reduce the computational cost without sacrificing reconstruction accuracy with our ta-

---

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td rowspan="3">2D representation</td><td colspan="4">2D part</td><td colspan="7">2D-to-3D part</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Mult-Adds</td><td style='text-align: center; word-wrap: break-word;'>#Param</td><td style='text-align: center; word-wrap: break-word;'>2D AUC $\uparrow$</td><td style='text-align: center; word-wrap: break-word;'>2D Acc $\downarrow$</td><td colspan="2">Pose pooling</td><td style='text-align: center; word-wrap: break-word;'>Lifting</td><td style='text-align: center; word-wrap: break-word;'>Mult-Adds</td><td style='text-align: center; word-wrap: break-word;'>#Param</td><td style='text-align: center; word-wrap: break-word;'>PJ $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>3D AUC $\uparrow$</td></tr><tr><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'>Global pooling</td><td style='text-align: center; word-wrap: break-word;'>FC</td><td style='text-align: center; word-wrap: break-word;'>3.2M</td><td style='text-align: center; word-wrap: break-word;'>3.2M</td><td style='text-align: center; word-wrap: break-word;'>7.55</td><td style='text-align: center; word-wrap: break-word;'>0.850</td><td style='text-align: center; word-wrap: break-word;'>10.12</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>$H_{{p}}$ w/o skip.</td><td style='text-align: center; word-wrap: break-word;'>57.7M</td><td style='text-align: center; word-wrap: break-word;'>1.4M</td><td style='text-align: center; word-wrap: break-word;'>0.796</td><td style='text-align: center; word-wrap: break-word;'>2.57</td><td rowspan="3">Joint-wise pooling</td><td style='text-align: center; word-wrap: break-word;'>PVL</td><td style='text-align: center; word-wrap: break-word;'>0.4M</td><td style='text-align: center; word-wrap: break-word;'>1029</td><td style='text-align: center; word-wrap: break-word;'>7.41</td><td style='text-align: center; word-wrap: break-word;'>0.852</td><td style='text-align: center; word-wrap: break-word;'>9.11</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>$H_{{p}}$</td><td style='text-align: center; word-wrap: break-word;'>57.7M</td><td style='text-align: center; word-wrap: break-word;'>1.4M</td><td style='text-align: center; word-wrap: break-word;'>0.857</td><td style='text-align: center; word-wrap: break-word;'>2.41</td><td style='text-align: center; word-wrap: break-word;'>PVL</td><td style='text-align: center; word-wrap: break-word;'>0.3M</td><td style='text-align: center; word-wrap: break-word;'>1029</td><td style='text-align: center; word-wrap: break-word;'>7.28</td><td style='text-align: center; word-wrap: break-word;'>0.856</td><td style='text-align: center; word-wrap: break-word;'>8.32</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>$H_{{p+soft-argmax}}$</td><td style='text-align: center; word-wrap: break-word;'>57.7M</td><td style='text-align: center; word-wrap: break-word;'>1.4M</td><td style='text-align: center; word-wrap: break-word;'>0.865</td><td style='text-align: center; word-wrap: break-word;'>2.32</td><td style='text-align: center; word-wrap: break-word;'>PVL</td><td style='text-align: center; word-wrap: break-word;'>0.3M</td><td style='text-align: center; word-wrap: break-word;'>1029</td><td style='text-align: center; word-wrap: break-word;'>7.11</td><td style='text-align: center; word-wrap: break-word;'>0.859</td><td style='text-align: center; word-wrap: break-word;'>6.14</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>$L_{{p}}$ w/ reg.</td><td style='text-align: center; word-wrap: break-word;'>0.1M</td><td style='text-align: center; word-wrap: break-word;'>0.1M</td><td style='text-align: center; word-wrap: break-word;'>0.847</td><td style='text-align: center; word-wrap: break-word;'>2.16</td><td style='text-align: center; word-wrap: break-word;'>Grid sampling</td><td style='text-align: center; word-wrap: break-word;'>PVL</td><td style='text-align: center; word-wrap: break-word;'>0.3M</td><td style='text-align: center; word-wrap: break-word;'>1029</td><td style='text-align: center; word-wrap: break-word;'>6.95</td><td style='text-align: center; word-wrap: break-word;'>0.862</td><td style='text-align: center; word-wrap: break-word;'>5.41</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>$L_{{p}}$ w/ MapReg (ours)</td><td style='text-align: center; word-wrap: break-word;'>47.5M</td><td style='text-align: center; word-wrap: break-word;'>1.4M</td><td style='text-align: center; word-wrap: break-word;'>0.870</td><td style='text-align: center; word-wrap: break-word;'>2.04</td><td style='text-align: center; word-wrap: break-word;'>Grid sampling</td><td style='text-align: center; word-wrap: break-word;'>PVL</td><td style='text-align: center; word-wrap: break-word;'>0.3M</td><td style='text-align: center; word-wrap: break-word;'>1029</td><td style='text-align: center; word-wrap: break-word;'>6.95</td><td style='text-align: center; word-wrap: break-word;'>0.862</td><td style='text-align: center; word-wrap: break-word;'>5.41</td></tr></table>

Table 2. Ablation study of feature lifting module. The first row is the setting of CMR [^9] (i.e. the last row of Table 1); skip. and reg. denote skip connection and direct regression; 2D and 3D accuracy are tested on RHD and FreiHAND, respectively; Acc is w.r.t. HO3Dv2.

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'>$\mathcal{L}_{con3D}$</td><td style='text-align: center; word-wrap: break-word;'>$\mathcal{L}_{con2D}$</td><td style='text-align: center; word-wrap: break-word;'>PJ $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>3D AUC $\uparrow$</td><td style='text-align: center; word-wrap: break-word;'>2D Acc $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>3D Acc $\downarrow$</td></tr><tr><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'>6.95</td><td style='text-align: center; word-wrap: break-word;'>0.862</td><td style='text-align: center; word-wrap: break-word;'>2.04</td><td style='text-align: center; word-wrap: break-word;'>5.41</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'>6.85</td><td style='text-align: center; word-wrap: break-word;'>0.864</td><td style='text-align: center; word-wrap: break-word;'>2.03</td><td style='text-align: center; word-wrap: break-word;'>4.78</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>6.85</td><td style='text-align: center; word-wrap: break-word;'>0.864</td><td style='text-align: center; word-wrap: break-word;'>1.98</td><td style='text-align: center; word-wrap: break-word;'>4.75</td></tr></table>

Table 3. Ablation study of consistency learning. The accuracy is w.r.t. FreiHAND and Acc is w.r.t. HO3Dv2.

lored encoding structures and dataset, making our DenseStack/GhostStack suitable for mobile environments.

Feature lifting module. With DenseStack, we perform a joint study of accuracy and temporal coherence. In Table 2, we do not use sequential module, temporal optimization, or post-processing. First, we explore various 2D pose representations in detail. $\mathbf{H}^{p}$ (Figure 4(a)) is a high-accuracy representation, and the skip connection proves critical to fuse high- and low-resolution features. Hence, when the skip connection is removed, $\mathbf{H}^{p}$ performs poorly in accuracy. As a differentiable form of picking maximum position, soft-argmax can produce smoother 2D positions from $\mathbf{H}^{p}$, hence improving both 2D AUC and Acc. In addition, $\mathbf{L}^{p}$ w/ reg. (Figure 4(c)) has relatively modest accuracy but produces better temporal performance because of global receptive field. Ultimately, our MagReg achieves better 2D accuracy and temporal coherence by integrating the merits of heatmap- and regression-based paradigms.

To clearly reveal details, we illustrate maps in Figure 6. As for $H^{P}$+soft-argmax, we additionally use the same loss setting (Equation 9) for training, i.e., heatmap supervision is not involved. This manner naively induces a smooth version of $H^{P}$ because the heuristic soft-argmax neglects visual semantics. In terms of MapReg, we present the map before it is flattened into a vector. Different from $H^{P}$, the map can adaptively describe joint landmark constraints, and then the 2D positions are predicted using adaptive local-global information. For example, the entire thumb is activated when predicting a landmark on it. Hence, our MapReg can produce more reasonable articulated structures (as shown in the supplemental material) to improve temporal coherence.

Subsequently, we explore 3D performance with 2D pose-aligned features. During pose pooling, joint-wise pooling [^74] (Figure 4(e)) can be used to obtain 2D pose-aligned features based on $H^p$, while grid sampling [^59] (Figure 4(f)) is commonly adopted when soft-argmax or $L^p$ is utilized. It is noteworthy that although $L^p$ w/reg. lags behind in terms of 2D accuracy, it produces better PAMPJPE than $H^p$+soft-argmax. Hence, with the same pose pooling method, 2D consistency is more crucial than accuracy in establishing a stable training process. Ultimately, MagRep-based $L^p$ induces the best PAMPJPE and 3D Acc.

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'>3D decoding</td><td style='text-align: center; word-wrap: break-word;'>Mult-Adds</td><td style='text-align: center; word-wrap: break-word;'>#Param</td><td style='text-align: center; word-wrap: break-word;'>PJ $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>Acc $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>FPS $\uparrow$</td></tr><tr><td colspan="6">w/ GhostStack</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>SpiralConv++</td><td style='text-align: center; word-wrap: break-word;'>159.0/263.1M</td><td style='text-align: center; word-wrap: break-word;'>1.0/6.2M</td><td style='text-align: center; word-wrap: break-word;'>8.63</td><td style='text-align: center; word-wrap: break-word;'>2.31/7.07</td><td style='text-align: center; word-wrap: break-word;'>77</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>DSConv (ours)</td><td style='text-align: center; word-wrap: break-word;'>19.5/123.5M</td><td style='text-align: center; word-wrap: break-word;'>0.1/5.3M</td><td style='text-align: center; word-wrap: break-word;'>8.76</td><td style='text-align: center; word-wrap: break-word;'>2.30/6.98</td><td style='text-align: center; word-wrap: break-word;'>83</td></tr><tr><td colspan="6">w/ DenseStack</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>SpiralConv++</td><td style='text-align: center; word-wrap: break-word;'>159.0/579.4M</td><td style='text-align: center; word-wrap: break-word;'>1.0/9.0M</td><td style='text-align: center; word-wrap: break-word;'>6.85</td><td style='text-align: center; word-wrap: break-word;'>1.98/4.75</td><td style='text-align: center; word-wrap: break-word;'>59</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>DSConv (ours)</td><td style='text-align: center; word-wrap: break-word;'>19.5/439.9M</td><td style='text-align: center; word-wrap: break-word;'>0.1/8.1M</td><td style='text-align: center; word-wrap: break-word;'>6.87</td><td style='text-align: center; word-wrap: break-word;'>1.92/4.73</td><td style='text-align: center; word-wrap: break-word;'>67</td></tr><tr><td colspan="6">Table 4. Ablation study of 3D decoding. Mult-Adds and #Param are w.r.t. the 3D decoder/overall model; 2D/3D Acc is presented; FPS is tested on Apple A14 CPU; Accuracy and temporal performance are tested on FreiHAND and HO3Dv2, respectively.</td></tr></table>

In PVL, we design a linear operation with lifting matrix $\mathbf{M}^l \in \mathbb{R}^{V^{mini} \times N}$ to transform features from 2D pose space to 3D vertex space. Thus, $V^{mini}$ vertex features are produced by a linear combination of N landmark features. Figure 7 depicts a well-trained lifting matrix, where we illustrate abs( $\mathbf{M}^l$) to clearly reveal the joint-vertex relations. As can be observed, the learned $\mathbf{M}^l$ is sparse. A joint landmark (i.e., joint 5 that locates at the root of forefinger) serves as the global information and contributes to the majority of vertices. Besides, some joint landmark traits are propagated to their corresponding vertices. Figure 8 depicts highly relevant connections in $\mathbf{M}^l$, demonstrating that the PVL approach can preserve the semantic consistency.

When compared to CMR [^9] (the first row of Table 2), our feature lifting module results in better PAMPJPE and 3D Acc. Moreover, we significantly reduce the computational expenses in the 2D-to-3D part. Despite the use of extra Mult-Adds in the 2D part, 2D pose prediction has previously proven to be beneficial in 3D hand reconstruction due to multi-task learning [^75] and root recovery task [^9].

Towards balancing model efficiency and performance. We design 3D/2D consistency loss to improve the performance further. From Table 3, we can see that consistency learning improves temporal coherence. Besides, the accuracy and temporal coherence can benefit from each other so that PAMPJPE is also enhanced.

---

![Image](images/b014e403d5d8c3be78c8d4016c203516502fa03ef97b221f64c700aae52d5194.jpg)

![Image](images/a3da21d9e4b73bf4f031fd7b5d7236205499f0cbabb3638049fa40a90f4d63cb.jpg)

![Image](images/e481c4fd5a5f0370ff4cbe5d1c971d3582662e80eeaa363e0c4cb123545678d3.jpg)

<div style="text-align: center;">Figure 9. 3D PCK vs. error thresholds.</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'>Method</td><td style='text-align: center; word-wrap: break-word;'>Backbone</td><td style='text-align: center; word-wrap: break-word;'>PJ $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>PV $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>F@5 $\uparrow$</td><td style='text-align: center; word-wrap: break-word;'>F@15 $\uparrow$</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>MobileHand [^18]</td><td style='text-align: center; word-wrap: break-word;'>MobileNet</td><td style='text-align: center; word-wrap: break-word;'>—</td><td style='text-align: center; word-wrap: break-word;'>13.1</td><td style='text-align: center; word-wrap: break-word;'>0.439</td><td style='text-align: center; word-wrap: break-word;'>0.902</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>FreiHAND [^82]</td><td style='text-align: center; word-wrap: break-word;'>ResNet50</td><td style='text-align: center; word-wrap: break-word;'>11.0</td><td style='text-align: center; word-wrap: break-word;'>10.9</td><td style='text-align: center; word-wrap: break-word;'>0.516</td><td style='text-align: center; word-wrap: break-word;'>0.934</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>YotubeHand [^42]</td><td style='text-align: center; word-wrap: break-word;'>ResNet50</td><td style='text-align: center; word-wrap: break-word;'>8.4</td><td style='text-align: center; word-wrap: break-word;'>8.6</td><td style='text-align: center; word-wrap: break-word;'>0.614</td><td style='text-align: center; word-wrap: break-word;'>0.966</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>I2L-MeshNet [^52]</td><td style='text-align: center; word-wrap: break-word;'>ResNet50 $^{{*}}$</td><td style='text-align: center; word-wrap: break-word;'>7.4</td><td style='text-align: center; word-wrap: break-word;'>7.6</td><td style='text-align: center; word-wrap: break-word;'>0.681</td><td style='text-align: center; word-wrap: break-word;'>0.973</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>HIU-DMTL [^75]</td><td style='text-align: center; word-wrap: break-word;'>Customized $^{{*}}$</td><td style='text-align: center; word-wrap: break-word;'>7.1</td><td style='text-align: center; word-wrap: break-word;'>7.3</td><td style='text-align: center; word-wrap: break-word;'>0.699</td><td style='text-align: center; word-wrap: break-word;'>0.974</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>CMR [^9]</td><td style='text-align: center; word-wrap: break-word;'>ResNet50 $^{{*}}$</td><td style='text-align: center; word-wrap: break-word;'>6.9</td><td style='text-align: center; word-wrap: break-word;'>7.0</td><td style='text-align: center; word-wrap: break-word;'>0.715</td><td style='text-align: center; word-wrap: break-word;'>0.977</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>I2UV-HandNet [^8]</td><td style='text-align: center; word-wrap: break-word;'>ResNet50</td><td style='text-align: center; word-wrap: break-word;'>6.7</td><td style='text-align: center; word-wrap: break-word;'>6.9</td><td style='text-align: center; word-wrap: break-word;'>0.707</td><td style='text-align: center; word-wrap: break-word;'>0.977</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>METRO [^45]</td><td style='text-align: center; word-wrap: break-word;'>HRNet</td><td style='text-align: center; word-wrap: break-word;'>6.7</td><td style='text-align: center; word-wrap: break-word;'>6.8</td><td style='text-align: center; word-wrap: break-word;'>0.717</td><td style='text-align: center; word-wrap: break-word;'>0.981</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Tang et al. [^65]</td><td style='text-align: center; word-wrap: break-word;'>ResNet50</td><td style='text-align: center; word-wrap: break-word;'>6.7</td><td style='text-align: center; word-wrap: break-word;'>6.7</td><td style='text-align: center; word-wrap: break-word;'>0.724</td><td style='text-align: center; word-wrap: break-word;'>0.981</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>MeshGraphormer [^46]</td><td style='text-align: center; word-wrap: break-word;'>HRNet</td><td style='text-align: center; word-wrap: break-word;'>5.9</td><td style='text-align: center; word-wrap: break-word;'>6.0</td><td style='text-align: center; word-wrap: break-word;'>0.765</td><td style='text-align: center; word-wrap: break-word;'>0.987</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>MobRecon (ours)</td><td style='text-align: center; word-wrap: break-word;'>GhostStack $^{{*}}$</td><td style='text-align: center; word-wrap: break-word;'>8.8</td><td style='text-align: center; word-wrap: break-word;'>9.1</td><td style='text-align: center; word-wrap: break-word;'>0.597</td><td style='text-align: center; word-wrap: break-word;'>0.960</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>MobRecon (ours)</td><td style='text-align: center; word-wrap: break-word;'>DenseStack $^{{*}}$</td><td style='text-align: center; word-wrap: break-word;'>6.9</td><td style='text-align: center; word-wrap: break-word;'>7.2</td><td style='text-align: center; word-wrap: break-word;'>0.694</td><td style='text-align: center; word-wrap: break-word;'>0.979</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>ours $^{\dagger}$</td><td style='text-align: center; word-wrap: break-word;'>ResNet18 $^{{*}}$</td><td style='text-align: center; word-wrap: break-word;'>6.7</td><td style='text-align: center; word-wrap: break-word;'>6.8</td><td style='text-align: center; word-wrap: break-word;'>0.727</td><td style='text-align: center; word-wrap: break-word;'>0.979</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>ours $^{\dagger}$</td><td style='text-align: center; word-wrap: break-word;'>ResNet18 $^{{*}}$</td><td style='text-align: center; word-wrap: break-word;'>6.1</td><td style='text-align: center; word-wrap: break-word;'>6.3</td><td style='text-align: center; word-wrap: break-word;'>0.758</td><td style='text-align: center; word-wrap: break-word;'>0.983</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>ours $^{\dagger}$</td><td style='text-align: center; word-wrap: break-word;'>ResNet50 $^{{*}}$</td><td style='text-align: center; word-wrap: break-word;'>6.1</td><td style='text-align: center; word-wrap: break-word;'>6.2</td><td style='text-align: center; word-wrap: break-word;'>0.760</td><td style='text-align: center; word-wrap: break-word;'>0.984</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>ours $^{\dagger}$</td><td style='text-align: center; word-wrap: break-word;'>ResNet50 $^{{*}}$</td><td style='text-align: center; word-wrap: break-word;'>5.7</td><td style='text-align: center; word-wrap: break-word;'>5.8</td><td style='text-align: center; word-wrap: break-word;'>0.784</td><td style='text-align: center; word-wrap: break-word;'>0.986</td></tr></table>

<div style="text-align: center;">Table 5. Results on the FreiHAND dataset. *: stacked structure; †: These models are based on ImageNet pre-trained backbone and mixed fine-tuning data;‡: These models are totally unrelated to our complement data.</div>

As shown in Table 4, DSConv dramatically decreases the Mult-Adds and #Param of the 3D decoder and obtains on par, sometimes even better, performance compared with SpiralConv++. Overall, our MobRecon with DenseStack/GhostStack can reach 67/83 FPS on Apple A14 CPU.

Discussion. MobRecon has a limitation that the DSConv increases memory access cost, so some engineering optimization should be involved for higher inference speed.

### 4.4. Comparisons with Contemporary Methods

On the FreiHAND dataset, we scale up our ResNet-based model with $224 \times 224$ input resolution for a fair comparison. As shown in Table 5, we surpass previous methods with ResNet50, leading to a new state of the art, i.e., 5.7mm PAMPJPE. Based on DenseStack or GhostStack, our MobRecon outmatches some ResNet-based methods. Referring to Figure 9, the proposed MobRecon has superior performance on 3D PCK. Beyond high accuracy, we also

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'>Method</td><td style='text-align: center; word-wrap: break-word;'>Backbone</td><td style='text-align: center; word-wrap: break-word;'>PJ $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>PV $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>F@5 $\uparrow$</td><td style='text-align: center; word-wrap: break-word;'>F@15 $\uparrow$</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>ObMan [^25]</td><td style='text-align: center; word-wrap: break-word;'>ResNet18</td><td style='text-align: center; word-wrap: break-word;'>11.0</td><td style='text-align: center; word-wrap: break-word;'>11.0</td><td style='text-align: center; word-wrap: break-word;'>0.464</td><td style='text-align: center; word-wrap: break-word;'>0.939</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>HO3D [^21]</td><td style='text-align: center; word-wrap: break-word;'>CPM [^69]</td><td style='text-align: center; word-wrap: break-word;'>10.7</td><td style='text-align: center; word-wrap: break-word;'>10.6</td><td style='text-align: center; word-wrap: break-word;'>0.506</td><td style='text-align: center; word-wrap: break-word;'>0.942</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>I2UV-HandNet [^8]</td><td style='text-align: center; word-wrap: break-word;'>ResNet50</td><td style='text-align: center; word-wrap: break-word;'>9.9</td><td style='text-align: center; word-wrap: break-word;'>10.1</td><td style='text-align: center; word-wrap: break-word;'>0.500</td><td style='text-align: center; word-wrap: break-word;'>0.943</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>MobRecon (ours)</td><td style='text-align: center; word-wrap: break-word;'>GhostStack</td><td style='text-align: center; word-wrap: break-word;'>10.0</td><td style='text-align: center; word-wrap: break-word;'>10.2</td><td style='text-align: center; word-wrap: break-word;'>0.488</td><td style='text-align: center; word-wrap: break-word;'>0.948</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>MobRecon (ours)</td><td style='text-align: center; word-wrap: break-word;'>DenseStack</td><td style='text-align: center; word-wrap: break-word;'>9.2</td><td style='text-align: center; word-wrap: break-word;'>9.4</td><td style='text-align: center; word-wrap: break-word;'>0.538</td><td style='text-align: center; word-wrap: break-word;'>0.957</td></tr></table>

<div style="text-align: center;">Table 6. Results on the HO3Dv2 dataset.</div>

achieve superior inference speed, as shown in Figure 1.

In experiments on RHD and HO3Dv2, our complement data are only used to pre-train DenseStack/GhostStack. On the RHD dataset, we compare with several pose estimation methods (such as [^1] [^5] [^62] [^72]) in Figure 9. Our MobRecon with DenseStack/GhostStack induces 3D AUC of 0.955 and 0.940, outperforming most compared approaches.

The HO3Dv2 dataset is employed for evaluation. According to Table 6, our MobRecon outperforms existing methods such as [^25] [^21] [^8]. HO3Dv2 is more challenging than FreiHAND and RHD because of serious object occlusion. In this case, MobRecon outperforms some ResNet-based methods because of better generalization ability. Besides, we also achieve better temporal coherence in this sequential task, as shown in Table 2.

Please refer to the supplementary materials for more qualitative analyses, mobile applications, etc.

## 5. Conclusions and Future Work

In this work, we present a novel hand mesh reconstruction method with superior efficiency, accuracy, and temporal coherence. First, we propose lightweight stacked structures for 2D encoding. Then, a feature lifting module with MapReg, pose pooling, and PVL approaches is designed for 2D-to-3D mapping. Besides, DSConv is developed to handle the 3D decoding task efficiently. Our MoBRecon only involves 123M Mult-Adds and 5M parameters so that it reaches a fast inference speed of 83FPS on Apple A14 CPU. Moreover, we achieve the state-of-the-art performance on FreiHAND, RHD, and HO3Dv2. We plan to investigate efficient methods for interacting hands.

---

### References

[^1]: Seungryul Baek, Kwang In Kim, and Tae-Kyun Kim. Pushing the envelope for RGB-based dense 3D hand pose estimation via neural rendering. In CVPR, 2019. 2, 8

[^2]: Seungryul Baek, Kwang In Kim, and Tae-Kyun Kim. Weakly-supervised domain adaptation via GAN and mesh model for estimating 3D hand poses interacting objects. In CVPR, 2020. 2

[^3]: Bharat Lal Bhatnagar, Cristian Sminchisescu, Christian Theobalt, and Gerard Pons-Moll. LoopReg: Self-supervised learning of implicit surface correspondences, pose and shape for 3d human mesh registration. In NeurIPS, 2020. 2

[^4]: Adnane Boukhayma, Rodrigo de Bem, and Philip HS Torr. 3D hand shape and pose from images in the wild. In CVPR, 2019. 2

[^5]: Yujun Cai, Liuhao Ge, Jianfei Cai, and Junsong Yuan. Weakly-supervised 3D hand pose estimation from monocular RGB images. In ECCV, 2018. 8

[^6]: Zhe Cao, Ilija Radosavovic, Angjoo Kanazawa, and Jitendra Malik. Reconstructing hand-object interactions in the wild. In ICCV, 2021. 2

[^7]: Joao Carreira, Pulkit Agrawal, Katerina Fragkiadaki, and Jitendra Malik. Human pose estimation with iterative error feedback. In CVPR, 2016. 4

[^8]: Ping Chen, Yujin Chen, Dong Yang, Fangyin Wu, Qin Li, Qingpei Xia, and Yong Tan. I2uv-handnet: Image-to-uv prediction network for accurate and high-fidelity 3d hand mesh modeling. In ICCV, pages 12929–12938, 2021. 2, 8

[^9]: Xingyu Chen, Yufeng Liu, Chongyang Ma, Jianlong Chang, Huayan Wang, Tian Chen, Xiaoyan Guo, Pengfei Wan, and Wen Zheng. Camera-space hand mesh recovery via semantic aggregation and adaptive 2D-1D registration. In CVPR, 2021. 1, 2, 3, 4, 6, 7, 8

[^10]: Yujin Chen, Zhigang Tu, Di Kang, Linchao Bao, Ying Zhang, Xuefei Zhe, Ruizhi Chen, and Junsong Yuan. Model-based 3d hand reconstruction via self-supervised learning. In CVPR, 2021. 2

[^11]: Hongsuk Choi, Gyeongsik Moon, Ju Yong Chang, and Kyoung Mu Lee. Beyond static features for temporally consistent 3D human pose and shape from a video. In CVPR, 2021. 1, 2, 3

[^12]: Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. In NeurIPS Workshop, 2014. 2

[^13]: Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In CVPR, 2009. 6

[^14]: Michael Garland and Paul S Heckbert. Surface simplification using quadric error metrics. In Proceedings of the Annual Conference on Computer Graphics and Interactive Techniques, 1997. 4

[^15]: Liuhao Ge, Zhou Ren, Yuncheng Li, Zehao Xue, Yingying Wang, Jianfei Cai, and Junsong Yuan. 3D hand shape and pose estimation from a single RGB image. In CVPR, 2019. 1, 2, 3, 4, 5

[^16]: Shunwang Gong, Lei Chen, Michael Bronstein, and Stefanos Zafeiriou. SpiralNet++: A fast and highly efficient mesh convolution operator. In ICCV Workshops, 2019. 5

[^17]: John C Gower. Generalized procrustes analysis. Psychometrika, 1975. 6

[^18]: Lim Guan Ming, Jatesiktat Prayook, and Ang Wei Tech. MobileHand: Real-time 3D hand shape and pose estimation from color image. In ICONIP, 2020. 1, 2, 8

[^19]: Riza Alp Guler and Iasonas Kokkinos. HoloPose: Holistic 3D human reconstruction in-the-wild. In CVPR, 2019. 3, 4

[^20]: Jia Guo, Jiankang Deng, Niannan Xue, and Stefanos Zafeiriou. Stacked dense U-Nets with dual transformers for robust face alignment. In BMVC, 2018. 5

[^21]: Shreyas Hampali, Mahdi Rad, Markus Oberweger, and Vincent Lepetit. Honnotate: A method for 3d annotation of hand and object poses. In CVPR, 2020. 5, 6, 8

[^22]: Shreyas Hampali, Sayan Deb Sarkar, Mahdi Rad, and Vincent Lepetit. HandsFormer: Keypoint transformer for monocular 3d pose estimation of hands and object in interaction. arXiv:2104.14639, 2021. 3

[^23]: Kai Han, Yunhe Wang, Qi Tian, Jianyuan Guo, Chunjing Xu, and Chang Xu. GhostNet: More features from cheap operations. In CVPR, 2020. 2, 3

[^24]: Shangchen Han, Beibei Liu, Randi Cabezas, Christopher D Twigg, Peizhao Zhang, Jeff Petkau, Tsz-Ho Yu, Chun-Jung Tai, Muzaffer Akbay, Zheng Wang, et al. MEgATrack: monochrome egocentric articulated hand-tracking for virtual reality. TOG, 2020. 1

[^25]: Yana Hasson, Gul Varol, Dimitrios Tzionas, Igor Kalevatykh, Michael J Black, Ivan Laptev, and Cordelia Schmid. Learning joint reconstruction of hands and manipulated objects. In CVPR, 2019. 2, 8

[^26]: Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016. 1, 2, 6

[^27]: Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 1997. 4

[^28]: Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. Searching for MobileNetv3. In ICCV, 2019. 1,2

[^29]: Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. MobileNets: Efficient convolutional neural networks for mobile vision applications. arXiv:1704.04861, 2017. 2, 5

[^30]: Jie Hu, Li Shen, and Gang Sun. Squeeze-and-excitation networks. In CVPR, 2018. 3

[^31]: Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In CVPR, 2017. 3

[^32]: Zeng Huang, Yuanlu Xu, Christoph Lassner, Hao Li, and Tony Tung. ARCH: Animatable reconstruction of clothed humans. In CVPR, 2020. 2

[^33]: Umar Iqbal, Pavlo Molchanov, Thomas Breuel Juergen Gall, and Jan Kautz. Hand pose estimation via latent 2.5D heatmap regression. In ECCV, 2018. 2

---

[^34]: Timothy Jeruzalski, Boyang Deng, Mohammad Norouzi, John P Lewis, Geoffrey Hinton, and Andrea Tagliasacchi. NASA: Neural articulated shape approximation. In ECCV, 2020. 2

[^35]: Hanwen Jiang, Shaowei Liu, Jiashun Wang, and Xiaolong Wang. Hand-object contact consistency reasoning for human grasps generation. In ICCV, 2021. 2

[^36]: Hanbyul Joo, Tomas Simon, Mina Cikara, and Yaser Sheikh. Towards social artificial intelligence: Nonverbal social signal prediction in a triadic interaction. In CVPR, 2019. 1

[^37]: Angjoo Kanazawa, Jason Y Zhang, Panna Felsen, and Jitendra Malik. Learning 3D human dynamics from video. In CVPR, 2019. 1, 2

[^38]: Korrawe Karunratanakul, Adrian Spurr, Zicong Fan, Otmar Hilliges, and Siyu Tang. A skeleton-driven neural occupancy representation for articulated hands. arXiv:2109.11399, 2021. 2

[^39]: Korrawe Karunratanakul, Jinlong Yang, Yan Zhang, Michael J Black, Krikamol Muandet, and Siyu Tang. Grasping field: Learning implicit representations for human grasps. In 3DV, 2020. 2

[^40]: Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2014. 5

[^41]: Muhammed Kocabas, Nikos Athanasiou, and Michael J Black. VIBE: Video inference for human body pose and shape estimation. In CVPR, 2020. 1, 2

[^42]: Dominik Kulon, Riza Alp Guler, Iasonas Kokkinos, Michael M Bronstein, and Stefanos Zafeiriou. Weakly-supervised mesh-convolutional hand reconstruction in the wild. In CVPR, 2020. 1, 2, 3, 4, 5, 8

[^43]: Jiefeng Li, Siyuan Bian, Ailing Zeng, Can Wang, Bo Pang, Wentao Liu, and Cewu Lu. Human pose regression with residual log-likelihood estimation. In ICCV, 2021. 3, 4

[^44]: Isaak Lim, Alexander Dielen, Marcel Campen, and Leif Kobbelt. A simple approach to intrinsic correspondence learning on unstructured 3D meshes. In ECCV, 2018. 2, 4

[^45]: Kevin Lin, Lijuan Wang, and Zicheng Liu. End-to-end human pose and mesh reconstruction with transformers. In CVPR, 2021. 1, 2, 8

[^46]: Kevin Lin, Lijuan Wang, and Zicheng Liu. Mesh graphormer. In ICCV, 2021. 1, 2, 8

[^47]: Shaowei Liu, Hanwen Jiang, Jiarui Xu, Sifei Liu, and Xiaolong Wang. Semi-supervised 3D hand-object poses estimation with interactions in time. In CVPR, 2021. 2

[^48]: Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black. SMPL: A skinned multi-person linear model. TOG, 2015. 3

[^49]: Zhengyi Luo, S Alireza Golestaneh, and Kris M Kitani. 3D human motion estimation via motion compression and refinement. In ACCV, 2020. 1, 2

[^50]: Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3D reconstruction in function space. In CVPR, 2019. 2

[^51]: Marko Mihajlovic, Yan Zhang, Michael J Black, and Siyu Tang. LEAP: Learning articulated occupancy of people. In CVPR, 2021. 2

[^52]: Gyeongsik Moon and Kyoung Mu Lee. I2L-MeshNet: Image-to-pixel prediction network for accurate 3D human pose and mesh estimation from a single RGB image. In ECCV, 2020. 2, 8

[^53]: Gyeongsik Moon and Kyoung Mu Lee. Pose2Pose: 3d positional pose-guided 3d rotational pose prediction for expressive 3D human pose and mesh estimation. arXiv:2011.11534, 2020. 3

[^54]: Gyeongsik Moon, Shoou-I Yu, He Wen, Takaaki Shiratori, and Kyoung Mu Lee. Interhand2.6M: A dataset and baseline for 3d interacting hand pose estimation from a single RGB image. In ECCV, 2020. 2

[^55]: Alejandro Newell, Kaiyu Yang, and Jia Deng. Stacked hourglass networks for human pose estimation. In ECCV, 2016. 2, 3, 4

[^56]: Evonne Ng, Shiry Ginosar, Trevor Darrell, and Hanbyul Joo. Body2Hands: Learning to infer 3D hands from conversational gesture body dynamics. In CVPR, 2021. 1

[^57]: Sida Peng, Yuanqing Zhang, Yinghao Xu, Qianqian Wang, Qing Shuai, Hujun Bao, and Xiaowei Zhou. Neural body: Implicit neural representations with structured latent codes for novel view synthesis of dynamic humans. In CVPR, 2021. 2

[^58]: Javier Romero, Dimitrios Tzionas, and Michael J Black. Embodied hands: Modeling and capturing hands and bodies together. TOG, 2017. 2, 4

[^59]: Shunsuke Saito, Zeng Huang, Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa, and Hao Li. PIFu: Pixel-aligned implicit function for high-resolution clothed human digitization. In ICCV, 2019. 2, 3, 4, 7

[^60]: Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. MobileNetv2: Inverted residuals and linear bottlenecks. In CVPR, 2018. 2, 6

[^61]: Adrian Spurr, Aneesh Dahiya, Xi Wang, Xucong Zhang, and Otmar Hilliges. Self-supervised 3d hand pose estimation from monocular rgb via contrastive learning. In ICCV, 2021. 5

[^62]: Adrian Spurr, Jie Song, Seonwook Park, and Otmar Hilliges. Cross-modal deep variational hand pose estimation. In CVPR, 2018. 8

[^63]: Xiao Sun, Jiaxiang Shang, Shuang Liang, and Yichen Wei. Compositional human pose regression. In ICCV, 2017. 4

[^64]: Xiao Sun, Bin Xiao, Fangyin Wei, Shuang Liang, and Yichen Wei. Integral human pose regression. In ECCV, 2018. 4

[^65]: Xiao Tang, Tianyu Wang, and Chi-Wing Fu. Towards accurate alignment in real-time 3d hand-mesh reconstruction. In ICCV, 2021. 1, 8

[^66]: Alexander Toshev and Christian Szegedy. DeepPose: Human pose estimation via deep neural networks. In CVPR, 2014. 4

[^67]: Jingdong Wang, Ke Sun, Tianheng Cheng, Borui Jiang, Chaorui Deng, Yang Zhao, Dong Liu, Yadong Mu, Mingkui Tan, Xinggang Wang, Wenyu Liu, and Bin Xiao. Deep high-resolution representation learning for visual recognition. TPAMI, 2020. 1, 4

[^68]: Fangyun Wei, Xiao Sun, Hongyang Li, Jingdong Wang, and Stephen Lin. Point-set anchors for object detection, instance segmentation and pose estimation. In ECCV, 2020. 4

---

[^69]: Shih-En Wei, Varun Ramakrishna, Takeo Kanade, and Yaser Sheikh. Convolutional pose machines. In CVPR, 2016. 8

[^70]: Linlin Yang, Shicheng Chen, and Angela Yao. SemiHand: Semi-supervised hand pose estimation with consistency. In ICCV, 2021. 2, 5

[^71]: Lixin Yang, Jiasen Li, Wenqiang Xu, Yiqun Diao, and Cewu Lu. BiHand: Recovering hand mesh with multi-stage bisected hourglass networks. In BMVC, 2020. 2, 3

[^72]: Linlin Yang, Shile Li, Dongheui Lee, and Angela Yao. Aligning latent spaces for 3D hand pose estimation. In ICCV, 2019. 8

[^73]: Lixin Yang, Xinyu Zhan, Kailin Li, Wenqiang Xu, Jiefeng Li, and Cewu Lu. CPF: Learning a contact potential field to model the hand-object interaction. In ICCV, 2021. 2

[^74]: Baowen Zhang, Yangang Wang, Xiaoming Deng, Yinda Zhang, Ping Tan, Cuixia Ma, and Hongan Wang. Interacting two-hand 3D pose and shape reconstruction from single color image. In ICCV, 2021. 2, 3, 4, 7

[^75]: Xiong Zhang, Hongsheng Huang, Jianchao Tan, Hongmin Xu, Cheng Yang, Guozhu Peng, Lei Wang, and Ji Liu. Hand image understanding via deep multi-task learning. In ICCV, 2021. 1, 2, 4, 7, 8

[^76]: Xiong Zhang, Qiang Li, Hong Mo, Wenbo Zhang, and Wen Zheng. End-to-end hand mesh recovery from a monocular RGB image. In ICCV, 2019. 2

[^77]: Zimeng Zhao, Xi Zhao, and Yangang Wang. TravelNet: Self-supervised physically plausible hand motion learning from monocular color images. In ICCV, 2021. 2

[^78]: Zerong Zheng, Tao Yu, Yebin Liu, and Qionghai Dai. PaMIR: Parametric model-conditioned implicit representation for image-based human reconstruction. TPAMI, 2021. 3

[^79]: Yuxiao Zhou, Marc Habermann, Weipeng Xu, Ikhsanul Habibie, Christian Theobalt, and Feng Xu. Monocular real-time hand shape and motion capture using multi-modal data. In CVPR, 2020. 1,2

[^80]: Christian Zimmermann, Max Argus, and Thomas Brox. Contrastive representation learning for hand shape estimation. arXiv:2106.04324, 2021. 5

[^81]: Christian Zimmermann and Thomas Brox. Learning to estimate 3D hand pose from single RGB images. In ICCV, 2017. 6

[^82]: Christian Zimmermann, Duygu Ceylan, Jimei Yang, Bryan Russell, Max Argus, and Thomas Brox. FreiHAND: A dataset for markerless capture of hand pose and shape from single RGB images. In ICCV, 2019. 2, 5, 6, 8