# MobRecon：基于单目图像的移动端友好型手部网格重建

陈星宇 $^{1*}$ 刘宇峰 $^{3}$ 董雅娇 $^{1}$ 张雄 $^{2}$ 马崇阳 $^{1}$

熊彦敏 $^{1}$ 张媛 $^{1}$ 郭晓燕 $^{1}$

 $^{1}$Y-tech，快手科技 $^{2}$ YY直播，百度公司。

 $^{3}$东南大学-艾伦联合中心、脑科学与智能研究院，中国。

## 摘要

在这项工作中，我们提出了一种单视图手部网格重建框架，能够同时实现高重建精度、快速推理速度和时间连贯性。具体而言，在2D编码方面，我们提出了轻量且高效的堆叠结构；在3D解码方面，我们设计了一种高效的图运算算子——深度可分离螺旋卷积。此外，我们还提出了一种新颖的特征提升模块，用于弥合2D与3D表示之间的差距。该模块首先采用基于映射的位置回归（MapReg）模块，结合热图编码和位置回归两种范式的优势，以提升2D精度和時間连贯性。随后，MapReg模块会依次通过姿态池化和姿态到顶点提升方法，将2D姿态编码转换为3D顶点的语义特征。总体而言，我们的手部重建框架MobRecon计算成本较低且模型尺寸小巧，在苹果A14 CPU上的推理速度可达83FPS。我们在FreiHAND、RHD和HO3Dv2等常用数据集上进行的大量实验表明，MobRecon在重建精度和時間连贯性方面均表现出色。我们的代码已公开在<https://github.com/SeanChenxy/HandMesh>。

### 1. 引言

由于在AR/VR [^24] [^65]、行为理解 [^36] [^56] 等领域有广泛应用，单视图手部网格重建多年来一直是研究热点。针对这一任务已有大量研究成果，包括 [^15] [^79] [^42] [^75] 等等。

现有典型方法主要侧重于重建精度 [^45] [^46]，而实际应用还要求具备推理效率和时间连贯性。尤其是3D手部信息

![Image](images/ce5609a039795a331b1e8e07c0012289ab8349ff2953a11eeaf8a8076972c4f5.jpg)

<div style="text-align: center;">Figure 1. Accuracy vs. inference speed. The marker size is related to the model size. Besides, our tailored method can run at a fast speed on mobile CPUs.</div>

是移动应用中的关键组成部分 [^65]，这类设备的内存和计算资源有限。因此，本研究旨在探索面向移动平台的3D手部重建技术。

单视图手部重建的典型流程包含三个阶段：2D编码、2D到3D映射以及3D解码。在2D编码阶段，现有方法[^42] [^9] [^45] [^46]通常采用计算复杂度较高的网络[^26] [^67]来处理这一高度非线性的任务，这类网络很难部署在移动设备上。相反，如果直接采用为其他任务设计的成熟移动网络（例如[^28]）来应对我们的目标任务，重建精度会大幅下降[^18]。因此，我们的目标是开发一种轻量级的2D编码结构，以兼顾推理效率与精度。此外，2D到3D映射以及3D解码的效率问题至今仍鲜有研究。基于此，我们打算探索一种轻量且高效的升维方法来解决2D到3D映射问题，并设计高效的图运算算子来处理3D网格数据。

尽管在实际应用中时间连贯性与精度同样重要，但在3D手部重建任务中它往往被忽视。以往的方法[^11] [^37] [^41] [^49]采用序列模型来整合过去与未来的语义信息以实现稳定的预测。由于这类方法要么需要离线处理，要么计算开销较大，因此难以应用于移动端场景。因此，我们受到启发，打算用非序列方法来探索时间连贯性。

---

这类方法利用语义信息来实现稳定的预测。但由于该方法属于离线处理或计算成本较高，因此难以用于移动应用。因此，我们受到启发，打算用非序列方法来探索时间连贯性。

在这项工作中，我们提出了面向3D手部的Mobile Mesh Reconstruction（MobRecon）算法，旨在同时实现更高的精度、效率与时间连贯性。在2D编码方面，我们借鉴沙漏网络[^55]的设计思路，构建了高效的堆叠式编码结构。在3D解码方面，我们提出了深度可分离螺旋卷积（DSConv），这是一种基于螺旋采样的新型图运算算子[^44]。DSConv的灵感来源于深度可分离卷积[^29]，能够高效处理图结构的网格数据。至于2D到3D映射环节，我们提出了一种结合基于映射的位置回归（MapReg）、姿态池化以及姿态到顶点升维（PVL）方法的特征升维模块。在该模块中，我们首先分析了基于热图或位置回归的2D姿态表示方式的优劣，随后提出了混合方法MapReg，以同时提升2D姿态精度与时间连贯性。此外，PVL基于可学习的升维矩阵将2D姿态编码转换为3D顶点特征，从而进一步提高3D精度与时间连贯性。与传统的基于隐空间全连接运算的方法[^42] [^15] [^9]相比，我们的特征升维模块还能显著减小模型规模。另外，我们构建了一个包含均匀分布的手部姿态与视角的合成数据集。如图1所示，我们在精度、速度以及模型规模方面均取得了更优的性能。

我们的主要贡献总结如下：

- 我们提出了MobRecon，这是一种适用于移动端的手部网格重建流程，其乘法累加运算量仅为123M次，参数数量仅5M个，在Apple A14 CPU上运行帧率可达83FPS。

- 我们提出了轻量化的堆叠结构与DSConv，用于实现高效的2D编码与3D解码。

- 我们提出了一种新颖的特征升维模块，结合了MapReg、姿态池化以及PVL方法，用以衔接2D与3D表示形式。

• 我们通过全面的评估以及与最先进方法的比较，证明我们的方法在模型效率、重建精度和时序一致性方面均取得了更优的性能。

### 2. 相关工作

手部网格估计。主流的手部网格估计方法可分为五类，其核心思想分别基于参数化模型、体素表示、隐式函数、UV映射和顶点位置。

基于模型的方法[^76] [^79] [^71] [^25] [^75] [^4] [^1] [^82] [^10] [^77] [^6] [^2] [^35] [^47] [^73] [^74]通常采用MANO[^58]作为参数化模型，该模型将手部网格分解为形状系数和姿态系数。然而，由于系数估计属于高度抽象的问题且忽略了空间相关性[^18]，这一流程并不适合与轻量级网络配合使用。

基于体素的方法[^33] [^52] [^70] [^54]以2.5D的方式描述3D数据。Moon等人[^52]提出的I2L-MeshNet将体素空间划分为三个像素空间，并利用1D热图来降低内存消耗。尽管有此类优化，I2L-MeshNet在处理此类2.5D热图时仍需要占用大量内存。因此，基于体素的方法对内存受限的移动设备并不友好。

隐式函数[^50]具有连续性和高分辨率的优势，近年来被用于对关节人体[^51] [^34] [^3] [^59] [^32] [^57] [^38] [^39]进行数字化处理。不过，隐式方法通常需要计算数千个3D点，在移动场景下效率较低。

Chen等人[^8]将手部网格重建视为图像到图像的转换任务，利用UV映射来连接2D和3D空间。若能将几何相关性纳入该流程，还可进一步提升性能。

基于顶点的方法[^15] [^42] [^9] [^45] [^46]直接预测3D顶点坐标，其流程通常包括2D编码、2D到3D映射以及3D解码。例如，Kulon等人[^42]设计了基于ResNet[^26]、全局池化和螺旋卷积（SpiralConv）[^44]的编码器-解码器结构来获取3D顶点坐标。我们采用高效的模块重构了基于顶点的流程：即用于2D编码的轻量级堆叠结构、用于2D到3D映射的特征提升模块，以及用于3D解码的DSConv。最终，我们实现了高重建精度和跨时间一致性。

轻量级网络。为了在计算资源受限的平台上实现实时处理，多年来研究人员一直致力于轻量级网络的研究，例如 [^29]、[^60]、[^28]、[^23]。我们借鉴了多种高效的设计思路，为欧几里得二维图像设计了堆叠式网络，并为非欧几里得三维网格设计了图网络。更具体地说，我们提出了一种特征提升模块，以高效解决二维到三维映射的问题。作为高度相关的研究工作，MobileHand [^18] 能够在移动CPU上以75帧/秒的速度运行。相比之下，我们的MobRecon在准确性和推理速度方面表现更为出色（如表5所示）。

时间上连贯的人体重建。关于人体/手部网格重建的时间连贯性研究较少，但在实际应用中它与重建精度同样重要。以往的研究 [^11]、[^37]、[^41]、[^49] 主要侧重于采用时序方法来提升时间性能。Kocabas等人 [^41] 利用双向门控循环单元 [^12] 来融合

---

![Image](images/91ef63ab1286826cad733f9284b5c7337fdf77e25c18600d937d816c3cf08f88.jpg)

<div style="text-align: center;">Figure 2. Overview of our MobRecon framework.</div>

跨时间特征，从而能够借助时序线索回归出SMPL [^48] 参数。这种顺序处理方式会增加计算成本，甚至需要依赖未来信息 [^11]。相比之下，我们设计了结合MapReg的特征提升模块，以提升非顺序单视图方法的时间连贯性。

像素对齐表示。卷积特征由密集且规则排列的二维线索构成，而三维数据则由稀疏且无序的点来表示。为了更好地提取图像特征来描述三维信息，近期的研究通常采用像素对齐表示法 [^59]、[^53]、[^19]、[^74]、[^22]、[^78]。受此启发，我们将像素对齐的思想用于特征提升，并设计了PVL来将姿态对齐编码转换为顶点特征。

值得注意的是，上述文献中采用的是热图 [^19]、[^22]、[^74] 或位置 [^59]、[^78] 作为二维表示方法。Li等人 [^43] 从准确性角度分析了基于热图和基于位置的人体姿态表示方法，并提出了RLE以实现高精度回归。我们从时间连贯性的角度考量这两种表示方法，提出了MapReg来整合基于热图和基于位置的二维表示方法的优点。与 [^43] 相比，我们的研究视角和见解有所不同，因此RLE和我们的MapReg可以互补。

### 3. 我们的方法

我们以单视图图像为输入，旨在推断出具有预测顶点 $\mathbf{V} = \{\mathbf{v}_i\}_{i=1}^V$ 和预定义面片 $\mathbf{C} = \{\mathbf{c}_i\}_{i=1}^C$ 的三维手部网格。图2展示了MobRecon的整体架构，包含三个阶段。在二维编码阶段，我们利用卷积网络提取图像特征。在特征提升模块中，先通过MapReg对二维姿态进行刻画，再通过姿态池化获取姿态对齐特征。随后，利用PVL得到顶点特征。在三维解码阶段，我们开发了高效的图网络来预测顶点坐标。

#### 3.1. 堆叠式编码网络

受沙漏网络[^55]的启发，我们开发了一种堆叠式编码网络，以获取逐步精炼的编码特征。如图3所示，该堆叠网络由两组级联的编码块组成，其中第一组编码块之后会接入一个用于特征融合的上采样模块。

![Image](images/f1d4b27eacbdb8f769387e575f820f7f22ed645b5440a6d637f45b77496243e8.jpg)

<div style="text-align: center;">Figure 3. Our tailored stacked encoding structure.</div>

以单视图图像为输入时，会生成编码特征$\mathbf{F}^e \in \mathbb{R}^{C^e \times H^e \times W^e}$，其中C、H、W分别表示张量的通道数、高度和宽度。

我们设计了两种模块变体。参考DenseNet[^31]和SENet[^30]的思路，我们提出了一种密集块来构成DenseStack（见图3）。为了进一步缩小模型规模，我们利用Ghost操作[^23]来开发GhostStack，这种操作能够基于原始特征生成“幽灵特征”。在输入分辨率为$128 \times 128$的情况下，DenseStack需要$373.0M$次乘法累加运算，包含$6.6M$个参数；而GhostStack则需要$96.2M$次乘法累加运算，包含$5.0M$个参数。相比之下，采用ResNet18的堆叠网络需要$2391.3M$次乘法累加运算，包含$25.2M$个参数，因此无法应用于移动端场景。

#### 3.2. 特征提升模块

特征提升指的是2D到3D的映射[^71]。在特征提升过程中，需要解决两个问题：（1）如何收集2D特征；（2）如何将它们映射到3D域。为此，以往的方法[^42] [^15] [^9]通常通过全局平均池化操作将$F^e$嵌入为潜向量。随后，该潜向量会通过全连接层（FC）映射到3D域，再通过向量重排得到顶点特征。这种方式会导致模型规模增大，因为当$C^e = 256$时，该层就包含了320万个参数。

近期的研究报道了基于2D地标和像素对齐特征池化的像素对齐特征提取方法[^59] [^19] [^22] [^74] [^78]。热图$H^{p}$通常被用来

---

![Image](images/0e6eb2fd3c3c91e46339e63509bcbfa4a62c5a75ef9739a16ef4c75e195a09da.jpg)

<div style="text-align: center;">Figure 4. Comparisons of 2D representations and pose pooling methods. (a) heatmap, (b) heatmap + soft-argmax, (c) regression-based position, (d) MapReg-based position, (e) joint-wise pooling with a heatmap, (f) grid sampling with a 2D position. For better visualization, only 4 landmarks are illustrated.</div>

对2D地标进行编码[^19] [^15] [^9] [^75]，与直接回归2D坐标位置相比，这种方法能得到更精确的地标$L^p$ [^43] [^66] [^7] [^63] [^68]。

Map-based position regression. Figure 4 comprehensively investigates these 2D representations. It is seen that $H^p$ is a high-resolution representation (Figure 4(a)). As shown in Figure 4(b), soft-argmax [^64] is a differentiable technique that decodes $H^p$ to the corresponding 2D position. In contrast, direct position regression (Figure 4(c)) is a low-resolution representation, which can yield $L^p$ without resorting to $H^p$. It has been proven that the human pose estimation task requires both low- and high-resolution encodings [^67]. Hence, the skip connection that fuses features is the primary cause behind the superior accuracy of $H^p$ [^55]. However, our critical insight is that the global feature (with resolution of $1 \times 1$) used to predict $L^p$ (see Figure 4(c)) shall induce better temporal coherence because of global semantics and receptive field. This global property can better describe articulated relation of hand pose. Conversely, $H^p$ is predicted with convolution and high-resolution features, where limited receptive field leads to the lack of inter-landmark constraints.

As shown in Figure 4(d), we propose a middle-resolution method MapReg to combine the advantages of heatmap-and position-based paradigms by (1) fusing low- and high-resolution features for accuracy and (2) using global receptive field for temporal coherence. To this end, we incorporate the skip connection in the position regression paradigm, producing a spatially small-size (e.g. $16 \times 16$) feature map. Each feature channel is then flattened into a vector, followed by a multi-layer perceptron (MLP) to generate a 2D position. In this manner, we obtain middle-resolution spatial complexity which is superior than heatmap since only two $2 \times$ -upsampling operations are involved.

Pose pooling. After obtaining 2D representations, pixel-aligned features can be retrieved. We call this process pose pooling and capture pose-aligned features with N 2D joint landmarks. If heatmap $\mathbf{H}^{p}$ is employed as the 2D representation, the pose pooling is conducted with joint-wise pooling [^74] (Figure 4(e)), which can be given as

 $$ \mathbf{F}^{p}=[\Psi(\mathbf{F}^{e}\odot\operatorname{i n t e r p o l a t i o n}(\mathbf{H}_{i}^{p}))]_{i=1,2,\ldots,N}, $$

where $[.]$ denotes concatenation. First, the spatial size of $\mathbf{H}^p$ is interpolated to $H^e \times W^e$, and then channel-wise multiplication is adopted between interpolated $\mathbf{H}^p$ and $\mathbf{F}^e$. In this manner, features unrelated to joint landmarks are suppressed. For extracting joint-wise features, feature reduction $\Psi$ is designed. In detail, $\Psi$ denotes global max-pooling or spatial sum reduction, which produces a $C^e$-length feature vector. After concatenation, $\mathbf{F}^p \in \mathbb{R}^{N \times C^e}$ indicates the pose-aligned feature.

If we use $L^p$ instead of $H^p$ to describe 2D pose, pose pooling can be achieved with grid sampling [^59] (Figure 4(f)) as follows,

 $$ \mathbf{F}^{p}=[\mathbf{F}^{e}(\mathbf{L}_{i}^{p})]_{i=1,2,\ldots,N}. $$

As a result, the convolutional encoding $\mathbf{F}^{e}$ is transformed to the pose-aligned representation $\mathbf{F}^{p}$.

Pose-to-vertex lifting. Referring to Figure 2, we design a linear operator for feature mapping towards 3D space with a few learnable parameters, namely PVL. MANO-style hand mesh [^58] comprises V vertices and N joints, where $V = 778$, $N = 21$. Because $V \gg N$, it is hard to transform $\mathbf{F}^p$ to V vertex features. Instead, we downsample the template mesh 4-times by a factor of 2 [^14], and obtain a minimal-size hand mesh with $V^{mini} = 49$ vertices. Then, we design a learnable lifting matrix $\mathbf{M}^l \in \mathbb{R}^{V^{mini} \times N}$ for 2D-to-3D feature mapping. Thereby, PVL is given as

 $$ \mathbf{F}^{m i n i\_{v}}=\mathbf{M}^{l}\cdot\mathbf{F}^{p}, $$

where $\mathbf{F}^{mini\_v}$ denotes minimal-size vertex feature. The PVL reduces the computational complexity of feautre mapping from $\mathcal{O}(V^{mini}C^{e2})$ [^42] [^9] [^15] to $\mathcal{O}(NV^{mini}C^{e})$.

#### 3.3. Depth-Separable SpiralConv

As a graph operator, SpiralConv [^44] is equivalent to the Euclidean convolution, which designs a spiral neighbor as

 $$ \begin{aligned}0-ring(\mathbf{v})&=\{\mathbf{v}\}\ $ k+1)-ring(\mathbf{v})&=\mathbb{N}(k-ring(\mathbf{v}))\setminus k-disk(\mathbf{v})\\k-disk(\mathbf{v})&=\cup_{i=0,\ldots,k}i-ring(\mathbf{v}),\end{aligned} $$

where N extracts the neighbourhood of a vertex v. With k-disk(v), SpiralConv formulates convolution as a sequential problem and leverages LSTM [^27] for feature fusion:

 $$ \mathbf{f}_{\mathbf{v}}^{{o u t}}=\operatorname{L S T M}(\mathbf{f}_{\mathbf{v}^{\prime}}),\quad\mathbf{v}^{\prime}\in k\mathbf{-d i s k}(\mathbf{v}), $$

---

![Image](images/6d80f76f535f29e4f36e2ff14868ddae80f7dd1c3012f1ae20d3b92b984a2542.jpg)

<div style="text-align: center;">Figure 5. Comparison of SpiralConv, SpiralConv++, and DSConv. For better visualization, a case with S = D = 3 is shown.</div>

where $\mathbf{f}_{\mathbf{v}} \in \mathbb{R}^{1 \times D}$ denotes the feature vector at $\mathbf{v}$ with dimension D. SpiralConv with LSTM could be potentially slow because of serial sequence processing.

By explicitly formulating the order of aggregating neighboring vertices, SpiralConv++ [^16] presents an efficient version of SpiralConv. For high efficiency, SpiralConv++ only adopts a fixed-size spiral neighbor with S vertices and leverages FC to fuse these features:

 $$ \mathbf{f}_{\mathbf{v}}^{{o u t}}=\mathbf{W}\cdot[\mathbf{f}_{\mathbf{v}^{\prime}}]+\mathbf{b},\quad\mathbf{v}^{\prime}\in k\mathbf{-d i s k}(\mathbf{v})_{S}, $$

where $[\cdot]$ denotes concatenation; k-disk( $\mathbf{v}$) $_{S}$ contains the first S elements in k-disk( $\mathbf{v}$); $\mathbf{W}$ and $\mathbf{b}$ are learnable parameters. SpiralConv++ significantly increases model size because of the large dimension of FC.

As an efficient graph convolution, we propose DSConv with the spirit of depth-wise separable convolution [^29]. For a vertex v, k-disk(v) $_{S}$ is sampled following Equation 4. DSConv comprises a depth-wise operation and a point-wise operation, the former of which can be formulated as

 $$ \mathbf{f}_{\mathbf{v}}^{d}=[\mathbf{W}_{i}^{d}\cdot[\mathbf{f}_{\mathbf{v}^{\prime},i}]]_{i=1}^{D},\quad\mathbf{v}^{\prime}\in k\mathbf{-d i s k}(\mathbf{v})_{S}. $$

Then, point-wise operation can be formulated as

 $$ \mathbf{f}_{\mathbf{v}}^{{o u t}}=\mathbf{W}^{p}\cdot\mathbf{f}_{\mathbf{v}}^{d}. $$

Figure 5 illustrates the difference among SpiralConv, SpiralConv++, and our DSConv. The SpiralConv++'s computational complexity obeys $\mathcal{O}(SD^2)$, whereas that of DSConv is $\mathcal{O}(SD + D^2)$. Hence, we essentially improve the efficiency with the separable structure.

The 3D decoder is built with four blocks, each of which involves upsampling, DSConv, and ReLU. In each block, vertex features are upsampled by a factor of 2 and then processed by DSConv. Finally, vertex coordinates V are predicted by a DSConv.

#### 3.4. Loss Functions

Accuracy loss. We use $L_{1}$ norm for the 3D mesh loss $\mathcal{L}_{mesh}$ and 2D pose loss $\mathcal{L}_{pose2D}$. Normal loss $\mathcal{L}_{norm}$ and edge length loss $\mathcal{L}_{edge}$ are adopted for mesh smoothness according to [^15]. Formally, we have

 $$ \begin{aligned}&\mathcal{L}_{m e s h}=||\mathbf{V}-\mathbf{V}^{\star}||_{1},\mathcal{L}_{p o s e2D}=||\mathbf{L}^{p}-\mathbf{L}^{p,\star}||_{1}\\&\mathcal{L}_{n o r m}=\sum_{\mathbf{c}\in\mathbf{C}}\sum_{(i,j)\subset\mathbf{c}}|\frac{\mathbf{V}_{i}-\mathbf{V}_{j}}{||\mathbf{V}_{i}-\mathbf{V}_{j}||_{2}}\cdot\mathbf{n}_{\mathbf{c}}^{\star}|\\&\mathcal{L}_{e d g e}=\sum_{\mathbf{c}\in\mathbf{C}}\sum_{(i,j)\subset\mathbf{c}}|||\mathbf{V}_{i}-\mathbf{V}_{j}||_{2}-||\mathbf{V}_{i}^{\star}-\mathbf{V}_{j}^{\star}||_{2}|,\\ \end{aligned} $$

where C, V are face and vertex sets of a mesh; $n_c^*$ indicates unit normal vector of face c; $*$ denotes the ground truth.

Consistency loss. Inspired by the self-supervision task [^61] [^80], we design the consistency supervision based on augmentation without the need of temporal data. That is, two views can be derived from an image sample based on 2D affine transformation and color jitter. We denote relative affine transformation between two views as $T_{1\rightarrow2}$, which contains relative rotation $R_{1\rightarrow2}$. Similar to [^20] [^70], we conduct consistency supervision in both 3D and 2D space:

 $$ \begin{aligned}\mathcal{L}_{con3D}&=||R_{1\rightarrow2}\mathbf{V}_{view1}-\mathbf{V}_{view2}||_{1}\\\mathcal{L}_{con2D}&=||T_{1\rightarrow2}\mathbf{L}^{p}_{view1}-\mathbf{L}^{p}_{view2}||_{1}.\end{aligned} $$

Although T contains the variations of rotation, shift, and scale in 2D space, only R affects $\mathbf{V}$ because it is rooted by the wrist landmark.

 $$ \begin{aligned}Our~overall~loss~function~is~\mathcal{L}_{total}=\mathcal{L}_{mesh}+\mathcal{L}_{pose2D}+\\\mathcal{L}_{norm}+\mathcal{L}_{edge}+\mathcal{L}_{con2D}+\mathcal{L}_{con3D}.\end{aligned} $$

### 4. Experiments

#### 4.1. Implementation Details

We use the Adam optimizer [^40] to train the network with a mini-batch size of 32. All models in our experiments are trained for 38 epochs. The initial learning rate is $10^{-3}$, which is divided by 10 at the 30th epoch. As for hyperparameters, the input resolution is $128 \times 128$, and $S = 9$, $C^e = 256$, $D = \{256, 128, 64, 32\}$.

Existing datasets for 3D hand pose estimation (e.g. [^82] [^42] [^21]) usually suffer from long-tailed distribution of hand pose and viewpoint. Hence, we develop a synthetic dataset with 1520 poses and 216 viewpoints, both of which are uniformly distributed in their respective spaces. Because of this superior property, it can serve as a good complement during training. We redirect the readers to supplementary material for details.

#### 4.2. Evaluation Criterion

We conduct experiments on several commonly-used benchmarks as listed below.

FreiHAND [^82] contains 130,240 training images and 3,960 evaluation samples. The annotations of the evaluation set are not available, so we submit our predictions to the official server for online evaluation.

---

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'>2D encoding</td><td style='text-align: center; word-wrap: break-word;'>Mult-Adds</td><td style='text-align: center; word-wrap: break-word;'>#Param</td><td style='text-align: center; word-wrap: break-word;'>Fine-tuning data</td><td style='text-align: center; word-wrap: break-word;'>PJ $\downarrow$</td></tr><tr><td colspan="5">From-scratch fine-tuning</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>ResNet18-Stack</td><td style='text-align: center; word-wrap: break-word;'>2391.3M</td><td style='text-align: center; word-wrap: break-word;'>25.2M</td><td style='text-align: center; word-wrap: break-word;'>FreiHAND</td><td style='text-align: center; word-wrap: break-word;'>8.86</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>MobileNet-Stack</td><td style='text-align: center; word-wrap: break-word;'>100.2M</td><td style='text-align: center; word-wrap: break-word;'>1.7M</td><td style='text-align: center; word-wrap: break-word;'>FreiHAND</td><td style='text-align: center; word-wrap: break-word;'>14.20</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>GhostStack</td><td style='text-align: center; word-wrap: break-word;'>96.2M</td><td style='text-align: center; word-wrap: break-word;'>5.0M</td><td style='text-align: center; word-wrap: break-word;'>FreiHAND</td><td style='text-align: center; word-wrap: break-word;'>13.09</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>DenseStack</td><td style='text-align: center; word-wrap: break-word;'>373.0M</td><td style='text-align: center; word-wrap: break-word;'>6.6M</td><td style='text-align: center; word-wrap: break-word;'>FreiHAND</td><td style='text-align: center; word-wrap: break-word;'>9.56</td></tr><tr><td colspan="5">Pre-training w/ ImageNet</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>ResNet18-Stack</td><td style='text-align: center; word-wrap: break-word;'>2391.3M</td><td style='text-align: center; word-wrap: break-word;'>25.2M</td><td style='text-align: center; word-wrap: break-word;'>FreiHAND</td><td style='text-align: center; word-wrap: break-word;'>8.21</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>MobileNet-Stack</td><td style='text-align: center; word-wrap: break-word;'>100.2M</td><td style='text-align: center; word-wrap: break-word;'>1.7M</td><td style='text-align: center; word-wrap: break-word;'>FreiHAND</td><td style='text-align: center; word-wrap: break-word;'>13.68</td></tr><tr><td colspan="5">Pre-training w/ ours</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>MobileNet-Stack</td><td style='text-align: center; word-wrap: break-word;'>100.2M</td><td style='text-align: center; word-wrap: break-word;'>1.7M</td><td style='text-align: center; word-wrap: break-word;'>FreiHAND</td><td style='text-align: center; word-wrap: break-word;'>12.45</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>GhostStack</td><td style='text-align: center; word-wrap: break-word;'>96.2M</td><td style='text-align: center; word-wrap: break-word;'>5.0M</td><td style='text-align: center; word-wrap: break-word;'>FreiHAND</td><td style='text-align: center; word-wrap: break-word;'>10.05</td></tr><tr><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'>FreiHAND+ours</td><td style='text-align: center; word-wrap: break-word;'>8.89</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>DenseStack</td><td style='text-align: center; word-wrap: break-word;'>373.0M</td><td style='text-align: center; word-wrap: break-word;'>6.6M</td><td style='text-align: center; word-wrap: break-word;'>FreiHAND</td><td style='text-align: center; word-wrap: break-word;'>7.77</td></tr><tr><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'>FreiHAND+ours</td><td style='text-align: center; word-wrap: break-word;'>7.55</td></tr></table>

<div style="text-align: center;">Table 1. Ablation studies of 2D encoding methods and our complement data on FreiHAND. Mult-Adds and #Param are w.r.t. the 2D encoding network.</div>

Rendered Hand Pose Dataset (RHD) [^81] consists of 41,258 and 2,728 synthetic hand data for training and testing on hand pose estimation, respectively.

HO3Dv2 [^21] is a 3D hand-object dataset that contains 66,034 training samples and 11,524 evaluation samples. The annotations of the evaluation set are not available, so we use the official server for online evaluation. We also use this dataset to evaluate temporal performance.

We use the following metrics in quantitative evaluations.

MPJPE/MPVPE measures the mean per joint/vertex position error by Euclidean distance (mm) between the estimated and ground-truth coordinates.

PA-MPJPE/MPVPE is a modification of MPJPE/MPVPE with Procrustes analysis [^17], ignoring global variation. For conciseness, this metric is abbreviated as PJ/PV.

Acc captures the acceleration of 2D/3D joint landmarks in pixel/s $^{2}$ or mm/s $^{2}$ to reflect temporal coherence.

AUC means the area under the curve of PCK (percentage of correct keypoints) vs. error thresholds of $n \sim 50$mm (for 3D measurement) or 0~30 pixel (for 2D measurement). According to [^82] [^81], n = 0 or 20.

F-Score is the harmonic mean between recall and precision between two meshes w.r.t. a specific distance threshold. F@5/F@15 corresponds to a threshold of 5mm/15mm.

Mult-Adds counts multiply-add operations.

#### Param indicates the number of parameters

#### 4.3. Ablation studies

Our stacked networks and complement data. We use CMR [^9] as the baseline, which adopts FC for 2D-to-3D feature mapping and SpiralConv++ for 3D decoding. With the same hyperparameters, we only change the 2D encoding network and data setting for comparisons. On the one hand, we study different stacked structures with from-scratch training. As shown in Table 1, we use ResNet [^26]

<div style="text-align: center;">H^{p} or H^{p}+soft-argmax w/ heatmap supervision</div>

![Image](images/f0d3607bd860690426e0bddb9b4c505d32e87d605e93d71377497f717b53c83c.jpg)

<div style="text-align: center;"> $H^{p}$+soft-argmax w/o heatmap supervision</div>

![Image](images/6f9d742cb03e8aa14c12a0546cbc69af13e174462fef63447c8d7017a82156da.jpg)

![Image](images/89b5d681ae8f3f6d758ebd3ac95938145499ea2f0b736b0c09489e99fc3b94e2.jpg)

<div style="text-align: center;">Figure 6. Visualization of maps in various 2D pose representations. Different from heatmap that focus on individual landmark, the map in MapReg adaptively describes inter-landmark constraints.</div>

![Image](images/3c8929e76249b97b09a2da327cabbc0216091aaacf0f5d27cb00803b10357a9e.jpg)

<div style="text-align: center;">0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42 44 46 48 vertex</div>

<div style="text-align: center;">Figure 7. Learned lifting matrix.</div>

![Image](images/8095bfe1239095701570f730303c69973dc9d005db9d3b3d8b72597d70528483.jpg)

<div style="text-align: center;">Figure 8. Highly relevant connections for several vertices in the lifting matrix. The numbers are vertex indices.</div>

and MobileNet [^60] to design stacked structures, the former of which contains intensive computation costs. Although MobileNet is computational tractability, it whittles down the performance (i.e., PAMPJPE) by a large margin. On the other hand, ImageNet [^13] is used for classification task, the knowledge from which is hard to transfer towards 2D/3D position regression. Consequently, the ImageNet pre-training brings less than 1mm PAMPJPE improvement.

Referring to Table 1, our complement data can be included in both pre-training and fine-tuning steps, and DenseStack/GhostStack induces 7.55/8.89mm PAMPJPE. Hence, we significantly reduce the computational cost without sacrificing reconstruction accuracy with our ta-

---

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td rowspan="3">2D representation</td><td colspan="4">2D part</td><td colspan="7">2D-to-3D part</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Mult-Adds</td><td style='text-align: center; word-wrap: break-word;'>#Param</td><td style='text-align: center; word-wrap: break-word;'>2D AUC $\uparrow$</td><td style='text-align: center; word-wrap: break-word;'>2D Acc $\downarrow$</td><td colspan="2">Pose pooling</td><td style='text-align: center; word-wrap: break-word;'>Lifting</td><td style='text-align: center; word-wrap: break-word;'>Mult-Adds</td><td style='text-align: center; word-wrap: break-word;'>#Param</td><td style='text-align: center; word-wrap: break-word;'>PJ $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>3D AUC $\uparrow$</td></tr><tr><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'>Global pooling</td><td style='text-align: center; word-wrap: break-word;'>FC</td><td style='text-align: center; word-wrap: break-word;'>3.2M</td><td style='text-align: center; word-wrap: break-word;'>3.2M</td><td style='text-align: center; word-wrap: break-word;'>7.55</td><td style='text-align: center; word-wrap: break-word;'>0.850</td><td style='text-align: center; word-wrap: break-word;'>10.12</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>$H_{{p}}$ w/o skip.</td><td style='text-align: center; word-wrap: break-word;'>57.7M</td><td style='text-align: center; word-wrap: break-word;'>1.4M</td><td style='text-align: center; word-wrap: break-word;'>0.796</td><td style='text-align: center; word-wrap: break-word;'>2.57</td><td rowspan="3">Joint-wise pooling</td><td style='text-align: center; word-wrap: break-word;'>PVL</td><td style='text-align: center; word-wrap: break-word;'>0.4M</td><td style='text-align: center; word-wrap: break-word;'>1029</td><td style='text-align: center; word-wrap: break-word;'>7.41</td><td style='text-align: center; word-wrap: break-word;'>0.852</td><td style='text-align: center; word-wrap: break-word;'>9.11</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>$H_{{p}}$</td><td style='text-align: center; word-wrap: break-word;'>57.7M</td><td style='text-align: center; word-wrap: break-word;'>1.4M</td><td style='text-align: center; word-wrap: break-word;'>0.857</td><td style='text-align: center; word-wrap: break-word;'>2.41</td><td style='text-align: center; word-wrap: break-word;'>PVL</td><td style='text-align: center; word-wrap: break-word;'>0.3M</td><td style='text-align: center; word-wrap: break-word;'>1029</td><td style='text-align: center; word-wrap: break-word;'>7.28</td><td style='text-align: center; word-wrap: break-word;'>0.856</td><td style='text-align: center; word-wrap: break-word;'>8.32</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>$H_{{p+soft-argmax}}$</td><td style='text-align: center; word-wrap: break-word;'>57.7M</td><td style='text-align: center; word-wrap: break-word;'>1.4M</td><td style='text-align: center; word-wrap: break-word;'>0.865</td><td style='text-align: center; word-wrap: break-word;'>2.32</td><td style='text-align: center; word-wrap: break-word;'>PVL</td><td style='text-align: center; word-wrap: break-word;'>0.3M</td><td style='text-align: center; word-wrap: break-word;'>1029</td><td style='text-align: center; word-wrap: break-word;'>7.11</td><td style='text-align: center; word-wrap: break-word;'>0.859</td><td style='text-align: center; word-wrap: break-word;'>6.14</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>$L_{{p}}$ w/ reg.</td><td style='text-align: center; word-wrap: break-word;'>0.1M</td><td style='text-align: center; word-wrap: break-word;'>0.1M</td><td style='text-align: center; word-wrap: break-word;'>0.847</td><td style='text-align: center; word-wrap: break-word;'>2.16</td><td style='text-align: center; word-wrap: break-word;'>Grid sampling</td><td style='text-align: center; word-wrap: break-word;'>PVL</td><td style='text-align: center; word-wrap: break-word;'>0.3M</td><td style='text-align: center; word-wrap: break-word;'>1029</td><td style='text-align: center; word-wrap: break-word;'>6.95</td><td style='text-align: center; word-wrap: break-word;'>0.862</td><td style='text-align: center; word-wrap: break-word;'>5.41</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>$L_{{p}}$ w/ MapReg (ours)</td><td style='text-align: center; word-wrap: break-word;'>47.5M</td><td style='text-align: center; word-wrap: break-word;'>1.4M</td><td style='text-align: center; word-wrap: break-word;'>0.870</td><td style='text-align: center; word-wrap: break-word;'>2.04</td><td style='text-align: center; word-wrap: break-word;'>Grid sampling</td><td style='text-align: center; word-wrap: break-word;'>PVL</td><td style='text-align: center; word-wrap: break-word;'>0.3M</td><td style='text-align: center; word-wrap: break-word;'>1029</td><td style='text-align: center; word-wrap: break-word;'>6.95</td><td style='text-align: center; word-wrap: break-word;'>0.862</td><td style='text-align: center; word-wrap: break-word;'>5.41</td></tr></table>

Table 2. Ablation study of feature lifting module. The first row is the setting of CMR [^9] (i.e. the last row of Table 1); skip. and reg. denote skip connection and direct regression; 2D and 3D accuracy are tested on RHD and FreiHAND, respectively; Acc is w.r.t. HO3Dv2.

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'>$\mathcal{L}_{con3D}$</td><td style='text-align: center; word-wrap: break-word;'>$\mathcal{L}_{con2D}$</td><td style='text-align: center; word-wrap: break-word;'>PJ $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>3D AUC $\uparrow$</td><td style='text-align: center; word-wrap: break-word;'>2D Acc $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>3D Acc $\downarrow$</td></tr><tr><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'>6.95</td><td style='text-align: center; word-wrap: break-word;'>0.862</td><td style='text-align: center; word-wrap: break-word;'>2.04</td><td style='text-align: center; word-wrap: break-word;'>5.41</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'>6.85</td><td style='text-align: center; word-wrap: break-word;'>0.864</td><td style='text-align: center; word-wrap: break-word;'>2.03</td><td style='text-align: center; word-wrap: break-word;'>4.78</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>6.85</td><td style='text-align: center; word-wrap: break-word;'>0.864</td><td style='text-align: center; word-wrap: break-word;'>1.98</td><td style='text-align: center; word-wrap: break-word;'>4.75</td></tr></table>

Table 3. Ablation study of consistency learning. The accuracy is w.r.t. FreiHAND and Acc is w.r.t. HO3Dv2.

lored encoding structures and dataset, making our DenseStack/GhostStack suitable for mobile environments.

Feature lifting module. With DenseStack, we perform a joint study of accuracy and temporal coherence. In Table 2, we do not use sequential module, temporal optimization, or post-processing. First, we explore various 2D pose representations in detail. $\mathbf{H}^{p}$ (Figure 4(a)) is a high-accuracy representation, and the skip connection proves critical to fuse high- and low-resolution features. Hence, when the skip connection is removed, $\mathbf{H}^{p}$ performs poorly in accuracy. As a differentiable form of picking maximum position, soft-argmax can produce smoother 2D positions from $\mathbf{H}^{p}$, hence improving both 2D AUC and Acc. In addition, $\mathbf{L}^{p}$ w/ reg. (Figure 4(c)) has relatively modest accuracy but produces better temporal performance because of global receptive field. Ultimately, our MagReg achieves better 2D accuracy and temporal coherence by integrating the merits of heatmap- and regression-based paradigms.

To clearly reveal details, we illustrate maps in Figure 6. As for $H^{P}$+soft-argmax, we additionally use the same loss setting (Equation 9) for training, i.e., heatmap supervision is not involved. This manner naively induces a smooth version of $H^{P}$ because the heuristic soft-argmax neglects visual semantics. In terms of MapReg, we present the map before it is flattened into a vector. Different from $H^{P}$, the map can adaptively describe joint landmark constraints, and then the 2D positions are predicted using adaptive local-global information. For example, the entire thumb is activated when predicting a landmark on it. Hence, our MapReg can produce more reasonable articulated structures (as shown in the supplemental material) to improve temporal coherence.

Subsequently, we explore 3D performance with 2D pose-aligned features. During pose pooling, joint-wise pooling [^74] (Figure 4(e)) can be used to obtain 2D pose-aligned features based on $H^p$, while grid sampling [^59] (Figure 4(f)) is commonly adopted when soft-argmax or $L^p$ is utilized. It is noteworthy that although $L^p$ w/reg. lags behind in terms of 2D accuracy, it produces better PAMPJPE than $H^p$+soft-argmax. Hence, with the same pose pooling method, 2D consistency is more crucial than accuracy in establishing a stable training process. Ultimately, MagRep-based $L^p$ induces the best PAMPJPE and 3D Acc.

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'>3D decoding</td><td style='text-align: center; word-wrap: break-word;'>Mult-Adds</td><td style='text-align: center; word-wrap: break-word;'>#Param</td><td style='text-align: center; word-wrap: break-word;'>PJ $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>Acc $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>FPS $\uparrow$</td></tr><tr><td colspan="6">w/ GhostStack</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>SpiralConv++</td><td style='text-align: center; word-wrap: break-word;'>159.0/263.1M</td><td style='text-align: center; word-wrap: break-word;'>1.0/6.2M</td><td style='text-align: center; word-wrap: break-word;'>8.63</td><td style='text-align: center; word-wrap: break-word;'>2.31/7.07</td><td style='text-align: center; word-wrap: break-word;'>77</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>DSConv (ours)</td><td style='text-align: center; word-wrap: break-word;'>19.5/123.5M</td><td style='text-align: center; word-wrap: break-word;'>0.1/5.3M</td><td style='text-align: center; word-wrap: break-word;'>8.76</td><td style='text-align: center; word-wrap: break-word;'>2.30/6.98</td><td style='text-align: center; word-wrap: break-word;'>83</td></tr><tr><td colspan="6">w/ DenseStack</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>SpiralConv++</td><td style='text-align: center; word-wrap: break-word;'>159.0/579.4M</td><td style='text-align: center; word-wrap: break-word;'>1.0/9.0M</td><td style='text-align: center; word-wrap: break-word;'>6.85</td><td style='text-align: center; word-wrap: break-word;'>1.98/4.75</td><td style='text-align: center; word-wrap: break-word;'>59</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>DSConv (ours)</td><td style='text-align: center; word-wrap: break-word;'>19.5/439.9M</td><td style='text-align: center; word-wrap: break-word;'>0.1/8.1M</td><td style='text-align: center; word-wrap: break-word;'>6.87</td><td style='text-align: center; word-wrap: break-word;'>1.92/4.73</td><td style='text-align: center; word-wrap: break-word;'>67</td></tr><tr><td colspan="6">Table 4. Ablation study of 3D decoding. Mult-Adds and #Param are w.r.t. the 3D decoder/overall model; 2D/3D Acc is presented; FPS is tested on Apple A14 CPU; Accuracy and temporal performance are tested on FreiHAND and HO3Dv2, respectively.</td></tr></table>

In PVL, we design a linear operation with lifting matrix $\mathbf{M}^l \in \mathbb{R}^{V^{mini} \times N}$ to transform features from 2D pose space to 3D vertex space. Thus, $V^{mini}$ vertex features are produced by a linear combination of N landmark features. Figure 7 depicts a well-trained lifting matrix, where we illustrate abs( $\mathbf{M}^l$) to clearly reveal the joint-vertex relations. As can be observed, the learned $\mathbf{M}^l$ is sparse. A joint landmark (i.e., joint 5 that locates at the root of forefinger) serves as the global information and contributes to the majority of vertices. Besides, some joint landmark traits are propagated to their corresponding vertices. Figure 8 depicts highly relevant connections in $\mathbf{M}^l$, demonstrating that the PVL approach can preserve the semantic consistency.

When compared to CMR [^9] (the first row of Table 2), our feature lifting module results in better PAMPJPE and 3D Acc. Moreover, we significantly reduce the computational expenses in the 2D-to-3D part. Despite the use of extra Mult-Adds in the 2D part, 2D pose prediction has previously proven to be beneficial in 3D hand reconstruction due to multi-task learning [^75] and root recovery task [^9].

Towards balancing model efficiency and performance. We design 3D/2D consistency loss to improve the performance further. From Table 3, we can see that consistency learning improves temporal coherence. Besides, the accuracy and temporal coherence can benefit from each other so that PAMPJPE is also enhanced.

---

![Image](images/b014e403d5d8c3be78c8d4016c203516502fa03ef97b221f64c700aae52d5194.jpg)

![Image](images/a3da21d9e4b73bf4f031fd7b5d7236205499f0cbabb3638049fa40a90f4d63cb.jpg)

![Image](images/e481c4fd5a5f0370ff4cbe5d1c971d3582662e80eeaa363e0c4cb123545678d3.jpg)

<div style="text-align: center;">Figure 9. 3D PCK vs. error thresholds.</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'>Method</td><td style='text-align: center; word-wrap: break-word;'>Backbone</td><td style='text-align: center; word-wrap: break-word;'>PJ $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>PV $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>F@5 $\uparrow$</td><td style='text-align: center; word-wrap: break-word;'>F@15 $\uparrow$</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>MobileHand [^18]</td><td style='text-align: center; word-wrap: break-word;'>MobileNet</td><td style='text-align: center; word-wrap: break-word;'>—</td><td style='text-align: center; word-wrap: break-word;'>13.1</td><td style='text-align: center; word-wrap: break-word;'>0.439</td><td style='text-align: center; word-wrap: break-word;'>0.902</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>FreiHAND [^82]</td><td style='text-align: center; word-wrap: break-word;'>ResNet50</td><td style='text-align: center; word-wrap: break-word;'>11.0</td><td style='text-align: center; word-wrap: break-word;'>10.9</td><td style='text-align: center; word-wrap: break-word;'>0.516</td><td style='text-align: center; word-wrap: break-word;'>0.934</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>YotubeHand [^42]</td><td style='text-align: center; word-wrap: break-word;'>ResNet50</td><td style='text-align: center; word-wrap: break-word;'>8.4</td><td style='text-align: center; word-wrap: break-word;'>8.6</td><td style='text-align: center; word-wrap: break-word;'>0.614</td><td style='text-align: center; word-wrap: break-word;'>0.966</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>I2L-MeshNet [^52]</td><td style='text-align: center; word-wrap: break-word;'>ResNet50 $^{{*}}$</td><td style='text-align: center; word-wrap: break-word;'>7.4</td><td style='text-align: center; word-wrap: break-word;'>7.6</td><td style='text-align: center; word-wrap: break-word;'>0.681</td><td style='text-align: center; word-wrap: break-word;'>0.973</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>HIU-DMTL [^75]</td><td style='text-align: center; word-wrap: break-word;'>Customized $^{{*}}$</td><td style='text-align: center; word-wrap: break-word;'>7.1</td><td style='text-align: center; word-wrap: break-word;'>7.3</td><td style='text-align: center; word-wrap: break-word;'>0.699</td><td style='text-align: center; word-wrap: break-word;'>0.974</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>CMR [^9]</td><td style='text-align: center; word-wrap: break-word;'>ResNet50 $^{{*}}$</td><td style='text-align: center; word-wrap: break-word;'>6.9</td><td style='text-align: center; word-wrap: break-word;'>7.0</td><td style='text-align: center; word-wrap: break-word;'>0.715</td><td style='text-align: center; word-wrap: break-word;'>0.977</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>I2UV-HandNet [^8]</td><td style='text-align: center; word-wrap: break-word;'>ResNet50</td><td style='text-align: center; word-wrap: break-word;'>6.7</td><td style='text-align: center; word-wrap: break-word;'>6.9</td><td style='text-align: center; word-wrap: break-word;'>0.707</td><td style='text-align: center; word-wrap: break-word;'>0.977</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>METRO [^45]</td><td style='text-align: center; word-wrap: break-word;'>HRNet</td><td style='text-align: center; word-wrap: break-word;'>6.7</td><td style='text-align: center; word-wrap: break-word;'>6.8</td><td style='text-align: center; word-wrap: break-word;'>0.717</td><td style='text-align: center; word-wrap: break-word;'>0.981</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Tang et al. [^65]</td><td style='text-align: center; word-wrap: break-word;'>ResNet50</td><td style='text-align: center; word-wrap: break-word;'>6.7</td><td style='text-align: center; word-wrap: break-word;'>6.7</td><td style='text-align: center; word-wrap: break-word;'>0.724</td><td style='text-align: center; word-wrap: break-word;'>0.981</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>MeshGraphormer [^46]</td><td style='text-align: center; word-wrap: break-word;'>HRNet</td><td style='text-align: center; word-wrap: break-word;'>5.9</td><td style='text-align: center; word-wrap: break-word;'>6.0</td><td style='text-align: center; word-wrap: break-word;'>0.765</td><td style='text-align: center; word-wrap: break-word;'>0.987</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>MobRecon (ours)</td><td style='text-align: center; word-wrap: break-word;'>GhostStack $^{{*}}$</td><td style='text-align: center; word-wrap: break-word;'>8.8</td><td style='text-align: center; word-wrap: break-word;'>9.1</td><td style='text-align: center; word-wrap: break-word;'>0.597</td><td style='text-align: center; word-wrap: break-word;'>0.960</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>MobRecon (ours)</td><td style='text-align: center; word-wrap: break-word;'>DenseStack $^{{*}}$</td><td style='text-align: center; word-wrap: break-word;'>6.9</td><td style='text-align: center; word-wrap: break-word;'>7.2</td><td style='text-align: center; word-wrap: break-word;'>0.694</td><td style='text-align: center; word-wrap: break-word;'>0.979</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>ours $^{\dagger}$</td><td style='text-align: center; word-wrap: break-word;'>ResNet18 $^{{*}}$</td><td style='text-align: center; word-wrap: break-word;'>6.7</td><td style='text-align: center; word-wrap: break-word;'>6.8</td><td style='text-align: center; word-wrap: break-word;'>0.727</td><td style='text-align: center; word-wrap: break-word;'>0.979</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>ours $^{\dagger}$</td><td style='text-align: center; word-wrap: break-word;'>ResNet18 $^{{*}}$</td><td style='text-align: center; word-wrap: break-word;'>6.1</td><td style='text-align: center; word-wrap: break-word;'>6.3</td><td style='text-align: center; word-wrap: break-word;'>0.758</td><td style='text-align: center; word-wrap: break-word;'>0.983</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>ours $^{\dagger}$</td><td style='text-align: center; word-wrap: break-word;'>ResNet50 $^{{*}}$</td><td style='text-align: center; word-wrap: break-word;'>6.1</td><td style='text-align: center; word-wrap: break-word;'>6.2</td><td style='text-align: center; word-wrap: break-word;'>0.760</td><td style='text-align: center; word-wrap: break-word;'>0.984</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>ours $^{\dagger}$</td><td style='text-align: center; word-wrap: break-word;'>ResNet50 $^{{*}}$</td><td style='text-align: center; word-wrap: break-word;'>5.7</td><td style='text-align: center; word-wrap: break-word;'>5.8</td><td style='text-align: center; word-wrap: break-word;'>0.784</td><td style='text-align: center; word-wrap: break-word;'>0.986</td></tr></table>

<div style="text-align: center;">Table 5. Results on the FreiHAND dataset. *: stacked structure; †: These models are based on ImageNet pre-trained backbone and mixed fine-tuning data;‡: These models are totally unrelated to our complement data.</div>

As shown in Table 4, DSConv dramatically decreases the Mult-Adds and #Param of the 3D decoder and obtains on par, sometimes even better, performance compared with SpiralConv++. Overall, our MobRecon with DenseStack/GhostStack can reach 67/83 FPS on Apple A14 CPU.

Discussion. MobRecon has a limitation that the DSConv increases memory access cost, so some engineering optimization should be involved for higher inference speed.

### 4.4. Comparisons with Contemporary Methods

On the FreiHAND dataset, we scale up our ResNet-based model with $224 \times 224$ input resolution for a fair comparison. As shown in Table 5, we surpass previous methods with ResNet50, leading to a new state of the art, i.e., 5.7mm PAMPJPE. Based on DenseStack or GhostStack, our MobRecon outmatches some ResNet-based methods. Referring to Figure 9, the proposed MobRecon has superior performance on 3D PCK. Beyond high accuracy, we also

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'>Method</td><td style='text-align: center; word-wrap: break-word;'>Backbone</td><td style='text-align: center; word-wrap: break-word;'>PJ $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>PV $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>F@5 $\uparrow$</td><td style='text-align: center; word-wrap: break-word;'>F@15 $\uparrow$</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>ObMan [^25]</td><td style='text-align: center; word-wrap: break-word;'>ResNet18</td><td style='text-align: center; word-wrap: break-word;'>11.0</td><td style='text-align: center; word-wrap: break-word;'>11.0</td><td style='text-align: center; word-wrap: break-word;'>0.464</td><td style='text-align: center; word-wrap: break-word;'>0.939</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>HO3D [^21]</td><td style='text-align: center; word-wrap: break-word;'>CPM [^69]</td><td style='text-align: center; word-wrap: break-word;'>10.7</td><td style='text-align: center; word-wrap: break-word;'>10.6</td><td style='text-align: center; word-wrap: break-word;'>0.506</td><td style='text-align: center; word-wrap: break-word;'>0.942</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>I2UV-HandNet [^8]</td><td style='text-align: center; word-wrap: break-word;'>ResNet50</td><td style='text-align: center; word-wrap: break-word;'>9.9</td><td style='text-align: center; word-wrap: break-word;'>10.1</td><td style='text-align: center; word-wrap: break-word;'>0.500</td><td style='text-align: center; word-wrap: break-word;'>0.943</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>MobRecon (ours)</td><td style='text-align: center; word-wrap: break-word;'>GhostStack</td><td style='text-align: center; word-wrap: break-word;'>10.0</td><td style='text-align: center; word-wrap: break-word;'>10.2</td><td style='text-align: center; word-wrap: break-word;'>0.488</td><td style='text-align: center; word-wrap: break-word;'>0.948</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>MobRecon (ours)</td><td style='text-align: center; word-wrap: break-word;'>DenseStack</td><td style='text-align: center; word-wrap: break-word;'>9.2</td><td style='text-align: center; word-wrap: break-word;'>9.4</td><td style='text-align: center; word-wrap: break-word;'>0.538</td><td style='text-align: center; word-wrap: break-word;'>0.957</td></tr></table>

<div style="text-align: center;">Table 6. Results on the HO3Dv2 dataset.</div>

achieve superior inference speed, as shown in Figure 1.

In experiments on RHD and HO3Dv2, our complement data are only used to pre-train DenseStack/GhostStack. On the RHD dataset, we compare with several pose estimation methods (such as [^1] [^5] [^62] [^72]) in Figure 9. Our MobRecon with DenseStack/GhostStack induces 3D AUC of 0.955 and 0.940, outperforming most compared approaches.

The HO3Dv2 dataset is employed for evaluation. According to Table 6, our MobRecon outperforms existing methods such as [^25] [^21] [^8]. HO3Dv2 is more challenging than FreiHAND and RHD because of serious object occlusion. In this case, MobRecon outperforms some ResNet-based methods because of better generalization ability. Besides, we also achieve better temporal coherence in this sequential task, as shown in Table 2.

Please refer to the supplementary materials for more qualitative analyses, mobile applications, etc.

## 5. Conclusions and Future Work

In this work, we present a novel hand mesh reconstruction method with superior efficiency, accuracy, and temporal coherence. First, we propose lightweight stacked structures for 2D encoding. Then, a feature lifting module with MapReg, pose pooling, and PVL approaches is designed for 2D-to-3D mapping. Besides, DSConv is developed to handle the 3D decoding task efficiently. Our MoBRecon only involves 123M Mult-Adds and 5M parameters so that it reaches a fast inference speed of 83FPS on Apple A14 CPU. Moreover, we achieve the state-of-the-art performance on FreiHAND, RHD, and HO3Dv2. We plan to investigate efficient methods for interacting hands.

---

### References

[^1]: Seungryul Baek, Kwang In Kim, and Tae-Kyun Kim. Pushing the envelope for RGB-based dense 3D hand pose estimation via neural rendering. In CVPR, 2019. 2, 8

[^2]: Seungryul Baek, Kwang In Kim, and Tae-Kyun Kim. Weakly-supervised domain adaptation via GAN and mesh model for estimating 3D hand poses interacting objects. In CVPR, 2020. 2

[^3]: Bharat Lal Bhatnagar, Cristian Sminchisescu, Christian Theobalt, and Gerard Pons-Moll. LoopReg: Self-supervised learning of implicit surface correspondences, pose and shape for 3d human mesh registration. In NeurIPS, 2020. 2

[^4]: Adnane Boukhayma, Rodrigo de Bem, and Philip HS Torr. 3D hand shape and pose from images in the wild. In CVPR, 2019. 2

[^5]: Yujun Cai, Liuhao Ge, Jianfei Cai, and Junsong Yuan. Weakly-supervised 3D hand pose estimation from monocular RGB images. In ECCV, 2018. 8

[^6]: Zhe Cao, Ilija Radosavovic, Angjoo Kanazawa, and Jitendra Malik. Reconstructing hand-object interactions in the wild. In ICCV, 2021. 2

[^7]: Joao Carreira, Pulkit Agrawal, Katerina Fragkiadaki, and Jitendra Malik. Human pose estimation with iterative error feedback. In CVPR, 2016. 4

[^8]: Ping Chen, Yujin Chen, Dong Yang, Fangyin Wu, Qin Li, Qingpei Xia, and Yong Tan. I2uv-handnet: Image-to-uv prediction network for accurate and high-fidelity 3d hand mesh modeling. In ICCV, pages 12929–12938, 2021. 2, 8

[^9]: Xingyu Chen, Yufeng Liu, Chongyang Ma, Jianlong Chang, Huayan Wang, Tian Chen, Xiaoyan Guo, Pengfei Wan, and Wen Zheng. Camera-space hand mesh recovery via semantic aggregation and adaptive 2D-1D registration. In CVPR, 2021. 1, 2, 3, 4, 6, 7, 8

[^10]: Yujin Chen, Zhigang Tu, Di Kang, Linchao Bao, Ying Zhang, Xuefei Zhe, Ruizhi Chen, and Junsong Yuan. Model-based 3d hand reconstruction via self-supervised learning. In CVPR, 2021. 2

[^11]: Hongsuk Choi, Gyeongsik Moon, Ju Yong Chang, and Kyoung Mu Lee. Beyond static features for temporally consistent 3D human pose and shape from a video. In CVPR, 2021. 1, 2, 3

[^12]: Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. In NeurIPS Workshop, 2014. 2

[^13]: Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In CVPR, 2009. 6

[^14]: Michael Garland and Paul S Heckbert. Surface simplification using quadric error metrics. In Proceedings of the Annual Conference on Computer Graphics and Interactive Techniques, 1997. 4

[^15]: Liuhao Ge, Zhou Ren, Yuncheng Li, Zehao Xue, Yingying Wang, Jianfei Cai, and Junsong Yuan. 3D hand shape and pose estimation from a single RGB image. In CVPR, 2019. 1, 2, 3, 4, 5

[^16]: Shunwang Gong, Lei Chen, Michael Bronstein, and Stefanos Zafeiriou. SpiralNet++: A fast and highly efficient mesh convolution operator. In ICCV Workshops, 2019. 5

[^17]: John C Gower. Generalized procrustes analysis. Psychometrika, 1975. 6

[^18]: Lim Guan Ming, Jatesiktat Prayook, and Ang Wei Tech. MobileHand: Real-time 3D hand shape and pose estimation from color image. In ICONIP, 2020. 1, 2, 8

[^19]: Riza Alp Guler and Iasonas Kokkinos. HoloPose: Holistic 3D human reconstruction in-the-wild. In CVPR, 2019. 3, 4

[^20]: Jia Guo, Jiankang Deng, Niannan Xue, and Stefanos Zafeiriou. Stacked dense U-Nets with dual transformers for robust face alignment. In BMVC, 2018. 5

[^21]: Shreyas Hampali, Mahdi Rad, Markus Oberweger, and Vincent Lepetit. Honnotate: A method for 3d annotation of hand and object poses. In CVPR, 2020. 5, 6, 8

[^22]: Shreyas Hampali, Sayan Deb Sarkar, Mahdi Rad, and Vincent Lepetit. HandsFormer: Keypoint transformer for monocular 3d pose estimation of hands and object in interaction. arXiv:2104.14639, 2021. 3

[^23]: Kai Han, Yunhe Wang, Qi Tian, Jianyuan Guo, Chunjing Xu, and Chang Xu. GhostNet: More features from cheap operations. In CVPR, 2020. 2, 3

[^24]: Shangchen Han, Beibei Liu, Randi Cabezas, Christopher D Twigg, Peizhao Zhang, Jeff Petkau, Tsz-Ho Yu, Chun-Jung Tai, Muzaffer Akbay, Zheng Wang, et al. MEgATrack: monochrome egocentric articulated hand-tracking for virtual reality. TOG, 2020. 1

[^25]: Yana Hasson, Gul Varol, Dimitrios Tzionas, Igor Kalevatykh, Michael J Black, Ivan Laptev, and Cordelia Schmid. Learning joint reconstruction of hands and manipulated objects. In CVPR, 2019. 2, 8

[^26]: Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016. 1, 2, 6

[^27]: Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 1997. 4

[^28]: Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. Searching for MobileNetv3. In ICCV, 2019. 1,2

[^29]: Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. MobileNets: Efficient convolutional neural networks for mobile vision applications. arXiv:1704.04861, 2017. 2, 5

[^30]: Jie Hu, Li Shen, and Gang Sun. Squeeze-and-excitation networks. In CVPR, 2018. 3

[^31]: Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In CVPR, 2017. 3

[^32]: Zeng Huang, Yuanlu Xu, Christoph Lassner, Hao Li, and Tony Tung. ARCH: Animatable reconstruction of clothed humans. In CVPR, 2020. 2

## [^33]: Umar Iqbal, Pavlo Molchanov, Thomas Breuel Juergen Gall, and Jan Kautz. Hand pose estimation via latent 2.5D heatmap regression. In ECCV, 2018. 2

---

[^34]: Timothy Jeruzalski, Boyang Deng, Mohammad Norouzi, John P Lewis, Geoffrey Hinton, and Andrea Tagliasacchi. NASA: Neural articulated shape approximation. In ECCV, 2020. 2

[^35]: Hanwen Jiang, Shaowei Liu, Jiashun Wang, and Xiaolong Wang. Hand-object contact consistency reasoning for human grasps generation. In ICCV, 2021. 2

[^36]: Hanbyul Joo, Tomas Simon, Mina Cikara, and Yaser Sheikh. Towards social artificial intelligence: Nonverbal social signal prediction in a triadic interaction. In CVPR, 2019. 1

[^37]: Angjoo Kanazawa, Jason Y Zhang, Panna Felsen, and Jitendra Malik. Learning 3D human dynamics from video. In CVPR, 2019. 1, 2

[^38]: Korrawe Karunratanakul, Adrian Spurr, Zicong Fan, Otmar Hilliges, and Siyu Tang. A skeleton-driven neural occupancy representation for articulated hands. arXiv:2109.11399, 2021. 2

[^39]: Korrawe Karunratanakul, Jinlong Yang, Yan Zhang, Michael J Black, Krikamol Muandet, and Siyu Tang. Grasping field: Learning implicit representations for human grasps. In 3DV, 2020. 2

[^40]: Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2014. 5

[^41]: Muhammed Kocabas, Nikos Athanasiou, and Michael J Black. VIBE: Video inference for human body pose and shape estimation. In CVPR, 2020. 1, 2

[^42]: Dominik Kulon, Riza Alp Guler, Iasonas Kokkinos, Michael M Bronstein, and Stefanos Zafeiriou. Weakly-supervised mesh-convolutional hand reconstruction in the wild. In CVPR, 2020. 1, 2, 3, 4, 5, 8

[^43]: Jiefeng Li, Siyuan Bian, Ailing Zeng, Can Wang, Bo Pang, Wentao Liu, and Cewu Lu. Human pose regression with residual log-likelihood estimation. In ICCV, 2021. 3, 4

[^44]: Isaak Lim, Alexander Dielen, Marcel Campen, and Leif Kobbelt. A simple approach to intrinsic correspondence learning on unstructured 3D meshes. In ECCV, 2018. 2, 4

[^45]: Kevin Lin, Lijuan Wang, and Zicheng Liu. End-to-end human pose and mesh reconstruction with transformers. In CVPR, 2021. 1, 2, 8

[^46]: Kevin Lin, Lijuan Wang, and Zicheng Liu. Mesh graphormer. In ICCV, 2021. 1, 2, 8

[^47]: Shaowei Liu, Hanwen Jiang, Jiarui Xu, Sifei Liu, and Xiaolong Wang. Semi-supervised 3D hand-object poses estimation with interactions in time. In CVPR, 2021. 2

[^48]: Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black. SMPL: A skinned multi-person linear model. TOG, 2015. 3

[^49]: Zhengyi Luo, S Alireza Golestaneh, and Kris M Kitani. 3D human motion estimation via motion compression and refinement. In ACCV, 2020. 1, 2

[^50]: Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3D reconstruction in function space. In CVPR, 2019. 2

[^51]: Marko Mihajlovic, Yan Zhang, Michael J Black, and Siyu Tang. LEAP: Learning articulated occupancy of people. In CVPR, 2021. 2

[^52]: Gyeongsik Moon and Kyoung Mu Lee. I2L-MeshNet: Image-to-pixel prediction network for accurate 3D human pose and mesh estimation from a single RGB image. In ECCV, 2020. 2, 8

[^53]: Gyeongsik Moon and Kyoung Mu Lee. Pose2Pose: 3d positional pose-guided 3d rotational pose prediction for expressive 3D human pose and mesh estimation. arXiv:2011.11534, 2020. 3

[^54]: Gyeongsik Moon, Shoou-I Yu, He Wen, Takaaki Shiratori, and Kyoung Mu Lee. Interhand2.6M: A dataset and baseline for 3d interacting hand pose estimation from a single RGB image. In ECCV, 2020. 2

[^55]: Alejandro Newell, Kaiyu Yang, and Jia Deng. Stacked hourglass networks for human pose estimation. In ECCV, 2016. 2, 3, 4

[^56]: Evonne Ng, Shiry Ginosar, Trevor Darrell, and Hanbyul Joo. Body2Hands: Learning to infer 3D hands from conversational gesture body dynamics. In CVPR, 2021. 1

[^57]: Sida Peng, Yuanqing Zhang, Yinghao Xu, Qianqian Wang, Qing Shuai, Hujun Bao, and Xiaowei Zhou. Neural body: Implicit neural representations with structured latent codes for novel view synthesis of dynamic humans. In CVPR, 2021. 2

[^58]: Javier Romero, Dimitrios Tzionas, and Michael J Black. Embodied hands: Modeling and capturing hands and bodies together. TOG, 2017. 2, 4

[^59]: Shunsuke Saito, Zeng Huang, Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa, and Hao Li. PIFu: Pixel-aligned implicit function for high-resolution clothed human digitization. In ICCV, 2019. 2, 3, 4, 7

[^60]: Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. MobileNetv2: Inverted residuals and linear bottlenecks. In CVPR, 2018. 2, 6

[^61]: Adrian Spurr, Aneesh Dahiya, Xi Wang, Xucong Zhang, and Otmar Hilliges. Self-supervised 3d hand pose estimation from monocular rgb via contrastive learning. In ICCV, 2021. 5

[^62]: Adrian Spurr, Jie Song, Seonwook Park, and Otmar Hilliges. Cross-modal deep variational hand pose estimation. In CVPR, 2018. 8

[^63]: Xiao Sun, Jiaxiang Shang, Shuang Liang, and Yichen Wei. Compositional human pose regression. In ICCV, 2017. 4

[^64]: Xiao Sun, Bin Xiao, Fangyin Wei, Shuang Liang, and Yichen Wei. Integral human pose regression. In ECCV, 2018. 4

[^65]: Xiao Tang, Tianyu Wang, and Chi-Wing Fu. Towards accurate alignment in real-time 3d hand-mesh reconstruction. In ICCV, 2021. 1, 8

[^66]: Alexander Toshev and Christian Szegedy. DeepPose: Human pose estimation via deep neural networks. In CVPR, 2014. 4

[^67]: Jingdong Wang, Ke Sun, Tianheng Cheng, Borui Jiang, Chaorui Deng, Yang Zhao, Dong Liu, Yadong Mu, Mingkui Tan, Xinggang Wang, Wenyu Liu, and Bin Xiao. Deep high-resolution representation learning for visual recognition. TPAMI, 2020. 1, 4

## [^68]: Fangyun Wei, Xiao Sun, Hongyang Li, Jingdong Wang, and Stephen Lin. Point-set anchors for object detection, instance segmentation and pose estimation. In ECCV, 2020. 4

---

[^69]: Shih-En Wei, Varun Ramakrishna, Takeo Kanade, and Yaser Sheikh. Convolutional pose machines. In CVPR, 2016. 8

[^70]: Linlin Yang, Shicheng Chen, and Angela Yao. SemiHand: Semi-supervised hand pose estimation with consistency. In ICCV, 2021. 2, 5

[^71]: Lixin Yang, Jiasen Li, Wenqiang Xu, Yiqun Diao, and Cewu Lu. BiHand: Recovering hand mesh with multi-stage bisected hourglass networks. In BMVC, 2020. 2, 3

[^72]: Linlin Yang, Shile Li, Dongheui Lee, and Angela Yao. Aligning latent spaces for 3D hand pose estimation. In ICCV, 2019. 8

[^73]: Lixin Yang, Xinyu Zhan, Kailin Li, Wenqiang Xu, Jiefeng Li, and Cewu Lu. CPF: Learning a contact potential field to model the hand-object interaction. In ICCV, 2021. 2

[^74]: Baowen Zhang, Yangang Wang, Xiaoming Deng, Yinda Zhang, Ping Tan, Cuixia Ma, and Hongan Wang. Interacting two-hand 3D pose and shape reconstruction from single color image. In ICCV, 2021. 2, 3, 4, 7

[^75]: Xiong Zhang, Hongsheng Huang, Jianchao Tan, Hongmin Xu, Cheng Yang, Guozhu Peng, Lei Wang, and Ji Liu. Hand image understanding via deep multi-task learning. In ICCV, 2021. 1, 2, 4, 7, 8

[^76]: Xiong Zhang, Qiang Li, Hong Mo, Wenbo Zhang, and Wen Zheng. End-to-end hand mesh recovery from a monocular RGB image. In ICCV, 2019. 2

[^77]: Zimeng Zhao, Xi Zhao, and Yangang Wang. TravelNet: Self-supervised physically plausible hand motion learning from monocular color images. In ICCV, 2021. 2

[^78]: Zerong Zheng, Tao Yu, Yebin Liu, and Qionghai Dai. PaMIR: Parametric model-conditioned implicit representation for image-based human reconstruction. TPAMI, 2021. 3

[^79]: Yuxiao Zhou, Marc Habermann, Weipeng Xu, Ikhsanul Habibie, Christian Theobalt, and Feng Xu. Monocular real-time hand shape and motion capture using multi-modal data. In CVPR, 2020. 1,2

[^80]: Christian Zimmermann, Max Argus, and Thomas Brox. Contrastive representation learning for hand shape estimation. arXiv:2106.04324, 2021. 5

[^81]: Christian Zimmermann and Thomas Brox. Learning to estimate 3D hand pose from single RGB images. In ICCV, 2017. 6

[^82]: Christian Zimmermann, Duygu Ceylan, Jimei Yang, Bryan Russell, Max Argus, and Thomas Brox. FreiHAND: A dataset for markerless capture of hand pose and shape from single RGB images. In ICCV, 2019. 2, 5, 6, 8