Figure 1. MaskHand: a novel generative masked model for accurate and robust 3D hand mesh recovery from single RGB images, excelling in diverse scenarios like occlusions, hand-object interactions, and varied appearances. Watch our Supplemental Video to see it in action!
## 摘要
从单张RGB图像中重建3D手部网格是一项极具挑战性的任务，因为手部存在复杂的关节运动、自身遮挡以及深度歧义等问题。传统的判别性方法通过学习从2D图像到单一3D网格的确定性映射关系，往往难以应对2D到3D映射过程中存在的固有歧义。为攻克这一难题，我们提出了MaskHand——一种用于手部网格恢复的新型生成式掩码模型。该模型通过学习并采样2D到3D映射过程的概率分布，从而合成合理的3D手部网格。MaskHand包含两大核心组件：（1）VQ-MANO，它可将3D手部关节运动编码为潜空间中的离散姿态标记；（2）上下文引导的掩码Transformer，它会随机掩盖部分姿态标记，并在受损的标记序列、图像上下文以及2D姿态线索的条件下，学习这些标记的联合分布。这种学习到的分布能够在推理阶段实现基于置信度的采样，进而生成不确定性低且精度高的网格重建结果。在基准数据集和真实世界数据集上的大量实验表明，MaskHand在3D手部网格重建的准确性、鲁棒性和效率方面均达到了最先进水平。
该模型在3D手部网格重建任务中展现出卓越的性能、稳健性及真实感。项目网站：。
### 1. 引言
由于其在机器人、人机交互[^43] [^57]、动画以及AR/VR[^12] [^31]等领域的广泛应用，手部网格恢复在计算机视觉领域受到了广泛关注。尽管以往的方法探索了无标记、基于图像的手部理解技术，但大多数方法依赖深度相机[^3] [^24] [^44] [^50] [^58]或多视角图像[^5] [^25] [^54] [^56]。因此，在仅能获取单目RGB图像的现实应用场景中，这些方法大多无法适用。另一方面，从单张RGB图像中恢复单目手部网格——尤其是没有身体上下文或明确相机参数的情况——极具挑战性，因为3D空间中手部外观存在巨大差异，且经常发生自身遮挡和复杂的关节运动。
近年来，尤其是基于Transformer的方法，在捕捉复杂的手部结构方面展现出在单目手部网格恢复（HMR）领域的巨大潜力。
---
这些模型能够建模手部的几何特征与空间关系。例如，METRO[^15]和MeshGraphormer[^40]利用多层注意力机制来建模顶点与顶点之间、顶点与关节之间的相互作用，从而提升网格的保真度。随后，HaMeR[^48]证明了大型视觉Transformer和大规模数据集对HMR的促进作用，实现了更高的重建精度。然而，这些方法本质上属于判别式方法，针对每张图像都会产生确定性的输出。因此，在遮挡、手物交互以及复杂视角导致的歧义现象普遍存在的复杂现实场景中，它们会受到限制。
Multiple Reconstruction Hypotheses with Explicit Confidence Levels
Figure 2. MaskHand is a quantifiable probabilistic HMR method that can learn 2D-to-3D mapping distributions and explicitly estimate confidence levels or prediction probabilities for all mesh reconstruction hypotheses
为了克服这些局限性，我们提出了MaskHand——一种用于精确恢复3D手部网格的新型生成式掩码模型。通过学习和采样手部关节运动与图像特征之间的潜在分布，该模型能够合成最有可能出现的3D手部网格，从而消除单视图重建中固有的歧义性，如图2所示。MaskHand由两个主要组件构成：VQ-MANO和上下文引导的掩码Transformer，它们分两个阶段依次训练。在第一阶段，利用向量量化变分自编码器（VQ-VAE）[^60]对VQ-MANO进行训练，将MONO参数化手部模型的连续手部姿态（如关节旋转）编码为一系列离散的姿态标记。VQ-MANO中的离散化处理对于生成式掩码建模至关重要，它使模型能够基于输入图像学习每个标记的条件概率。在第二阶段，标记序列会被部分掩码处理，随后训练上下文引导的掩码Transformer，使其基于多种上下文线索——包括受损的标记序列、图像特征以及2D姿态结构——来学习标记的条件分布，进而重构被掩码的标记。所学到的分布能够在每个预测步骤中量化不确定性，从而支持基于置信度的采样过程，逐步优化网格重建效果。通过在推理阶段选择性地保留高置信度的标记，MaskHand能够最大限度地降低2D到3D映射过程中的不确定性，即便在存在遮挡或歧义的情况下也能实现高精度的手部网格恢复。
- MaskHand是首个利用生成式掩码建模来重建稳健3D手部网格的模型。其核心思想是显式地学习2D到3D的概率映射关系，并通过从学习到的分布中采样来生成具有高置信度且合理的3D网格。
- 我们设计的上下文引导掩码Transformer能够有效融合多种上下文线索，包括2D姿态、图像特征以及未被掩码处理的3D姿态标记。
- 我们提出了差分掩码训练方法，以在所有上下文线索的条件下学习手部姿态标记的分布。这种学习到的分布能够在推理阶段支持基于置信度的采样，从而生成不确定性低且精度高的网格重建结果。
- 我们证明了MaskHand是一个通用框架，能够同时实现手部网格估计与手部网格生成。
- 大量实验表明，MaskHand在标准数据集上的表现优于现有最先进的方法。
### 2. 相关工作
#### 2.1. 判别式方法
近年来，人手重建技术取得了显著进展。早期方法[^23] [^29] [^59] [^62] [^71] [^73]利用优化技术，根据二维骨骼检测结果来估计手部姿态。随后，MANO[^52]提出了一种可微分的参数化网格模型，能够捕捉手部形状和关节运动，从而使该模型能够从单视图图像中直接端到端地估计模型参数，并生成合理的网格。Boukhayma等人[^6]提出了首个完全可学习的框架，可直接从RGB图像预测MANO手部模型参数[^52]。此后，多种方法也相继采用热图[^70]和迭代细化技术[^4]来确保二维对齐。Kulon等人[^35] [^36]提出了一种不同的回归方法，直接预测三维顶点而非MANO姿态参数，在性能上较以往方法有了显著提升。近期的方法，如METRO[^49]、MeshGraphormer[^40]、HaMeR[^48]，通过建模顶点间以及顶点与关节间的相互作用，达到了最先进的性能水平。这些现有方法基于判别性回归，学习的是从输入图像到输出网格的确定性映射。这种确定性方法难以捕捉由手部自遮挡、物体交互、极端姿态或拍摄角度所引起的不确定性和模糊性，从而导致重建出的手部网格不够真实。
#### 2.2. 生成式方法
我们的MaskHand采用了一种生成式方法，该方法学习从输入图像到输出网格的概率映射。它利用这一学习到的分布来合成高置信度且合理的三维手部网格。
---
二维视觉上下文。文献中另一种生成式手部网格重建方法是HHMR[^37]。与采用扩散模型的HHMR[^37]不同，MaskHand的灵感来源于掩码图像模型和语言模型在图像及文本生成任务中的成功应用[^8] [^9] [^18] [^19] [^72]。这一根本性差异使得MaskHand能够显式且定量地估算所有网格重建假设的置信度或预测概率，从而能够基于置信度选择最优假设以实现精准重建，如图2所示。相比之下，HHMR的去噪扩散过程会生成多个网格假设，但并未为每个假设赋予相应的置信度。因此，它只能在假定存在真实网格的前提下，通过寻找重构误差最小的假设来给出理论上最佳的网格重建结果。
### 3. 所提出的方法：MaskHand
问题表述。给定单手图像I，我们的目标是学习一个映射函数$f(I) = \{\theta, \beta, \pi\}$，该函数能从输入图像中输出MANO[^52]模型参数。该映射函数包含三个关键组成部分：手部姿态参数$\theta \in \mathbb{R}^{48}$、形状参数$\beta \in \mathbb{R}^{10}$以及相机参数$\pi \in \mathbb{R}^{3}$，从而实现全面的三维手部重建。
所提出方法的概述。如图3所示，MaskHand由两个主要模块组成：VQ-MANO和上下文引导的掩码Transformer。首先，VQ-MANO将3D MANO姿态参数（$\theta$）离散化为离散标记，从而实现生成式掩码建模和不确定性量化。接下来，上下文引导的掩码Transformer利用编码器-解码器结构：编码器提取多尺度图像特征，这些特征与2D姿态线索及未掩码标记结合后，由掩码解码器进行处理。解码器采用基于图的解剖学姿态优化方法来捕捉关节间的依赖关系，并利用上下文注入的掩码合成器来融合视觉特征与标记交互，通过差分掩码建模学习概率性重建。在推理阶段，通过选择性保留高置信度标记并重新掩码不确定标记，迭代地优化姿态预测结果，从而逐步提升准确性。最终，将预测得到的姿态（$\theta$）、形状（$\beta$）以及相机参数（$\pi$）输入MANO手部模型，完成重建过程。
#### 3.1. 手部模型与VQ-MANO
我们的方法采用了MANO模型[^52]，该模型可将姿态$\theta \in \mathbb{R}^{48}$和形状$\beta \in \mathbb{R}^{10}$映射为3D手部网格$M \in \mathbb{R}^{778 \times 3}$及关节位置$X \in \mathbb{R}^{21 \times 3}$，从而同时表征手部表面与姿态信息。
VQ-MANO是一种MANO手部标记器，它利用一维卷积自编码器为3D姿态参数$\theta \in \mathbb{R}^{48}$学习离散潜空间，具体做法是将连续的姿态嵌入量化为编码簿C中的离散代码条目，如图3(a)所示。其关键特性在于上采样机制：能将16种MANO姿态扩展为64个离散姿态标记，从而增强空间细节并提升网格恢复精度（详见补充材料第C节）。为此，我们采用向量量化变分自编码器（VQ-VAE）[^60]来对标记器进行预训练。具体而言，我们将MANO姿态参数$\theta$输入卷积编码器E，将其映射为潜嵌入z。随后，每个嵌入$z_i$会根据欧几里得距离被量化为最接近的编码簿条目$c_k \in C$，计算公式如下：
$$ \hat{z}_{i}=\arg\min_{c_{k}\in C}\|z_{i}-c_{k}\|_{2}. $$
VQ-MANO的总损失函数定义为：
$$ \begin{array}{r}{\mathcal{L}_{\mathrm{v q-m a n o}}=\lambda_{\mathrm{r e}}\mathcal{L}_{\mathrm{r e c o n}}+\lambda_{\mathrm{E}}\|\operatorname{s g}[z]-c\|_{2}+\lambda_{\alpha}\|z-\operatorname{s g}[c]\|_{2},}\end{array} $$
其中，$\mathcal{L}_{\mathrm{vq-mano}}$由MANO重建损失$\mathcal{L}_{\mathrm{recon}}$、潜嵌入损失以及承诺损失组成，它们分别由超参数$\lambda_{\mathrm{re}}$、$\lambda_{\mathrm{E}}$和$\lambda_{\alpha}$加权。此处，$\mathrm{sg}[\cdot]$表示停止梯度运算符，用于阻止梯度在反向传播过程中流经其参数。为了进一步提升重建质量，我们引入了额外的L1损失：
$$ \begin{array}{r}{\mathcal{L}_{\mathrm{r e c o n}}=\lambda_{\theta}\mathcal{L}_{\theta}+\lambda_{\mathrm{V}}\mathcal{L}_{\mathrm{V}}+\lambda_{\mathrm{J}}\mathcal{L}_{\mathrm{J}},}\end{array} $$
该损失旨在最小化预测MANO参数与真实MANO参数之间的差异，包括姿态参数$\theta$、网格顶点V以及手部关节J。标记器的优化借助直通梯度估计器来实现，从而确保梯度能够穿过不可微的量化步骤。此外，编码簿条目C会通过指数移动平均法以及定期的编码簿重置操作来更新，相关细节参见[^21]和[^64]。
#### 3.2. 上下文引导的掩码Transformer
上下文引导的掩码Transformer包含两个主要组件：多尺度图像编码器和掩码图Transformer解码器。
##### 3.2.1. 多尺度图像编码器
我们的编码器采用视觉Transformer（ViT-H/16）[^2]来提取图像特征[^48]，将16×16像素的图块处理为特征令牌。遵循ViTDet [^38]的思路，我们采用多尺度特征方法：对初始特征图进行上采样，以生成不同分辨率的特征图。这种多尺度表示对于处理手部姿态中的复杂关节运动和自遮挡现象至关重要。高分辨率特征图能提供精细的关节细节，而低分辨率特征图则能捕捉手部的整体结构，从而在关节定位精度与整体解剖一致性之间取得平衡。此外，我们在x-Attention头中利用低分辨率特征图进行交叉注意力计算，以实现回归。
---
Figure 3. MaskHand Training Phase. MaskHand consists of two key components: (1) VQ-MANO, which encodes 3D hand poses into a sequence of discrete tokens within a latent space, and (2) a Context-Guided Masked Transformer that models the probabilistic distributions of these tokens, conditioned on the input image, 2D pose cues, and a partially masked token sequence.
Figure 4. Architecture of Graph-based Anatomical Pose Refinement (GAPR) and Context-Infused Masked Synthesizer, illustrating fusion of anatomical pose dependencies and contextual cues for precise mesh reconstruction.
稳定的形状参数（$\beta$）和相机方向（$\pi$）使得该过程在计算上更高效。这种方法将形状估计与姿态建模分离，既保留了形态稳定性与空间对齐性，又提升了3D手部重建的鲁棒性和解剖准确性。
##### 3.2.2. 掩码图Transformer解码器
掩码Transformer解码器由两个关键组件构成（见图3和图4）：基于图的解剖姿态优化模块（GAPR）以及上下文注入式掩码合成模块。
基于图的解剖姿态优化模块（GAPR）。我们的解码器采用了两个轻量级图Transformer模块，用于处理由VQ-MANO生成的姿态令牌，并结合2D姿态引导信息，从而确保如图4所示的解剖学上一致的手部重建结果。为了构建能够反映手部运动学的2D姿态令牌，我们首先通过带有固定手部骨骼邻接矩阵的图卷积网络（GCN）来处理OpenPose 2D关键点[^7]，以此捕捉显式的运动学结构。同时，提取出本身即编码了隐式和非线性关节依赖关系的VQ-MANO姿态令牌。随后，这两种表示被融合并通过两个图Transformer模块进行优化，其中可学习的邻接矩阵能够自适应地建模复杂的关节关系，从而保证稳健的空间和结构一致性。为了进一步提升稳定性和解剖准确性，我们引入了一个带有挤压-激励（SE）模块的Transformer编码器[^30]，该模块着重强调关节方向和角度，同时通过1×1卷积层确保空间对齐。在Transformer编码器中，多头注意力机制（MHA）和点卷积层进一步细化关节间的高分辨率依赖关系，最终生成连贯且解剖对齐的3D姿态表示，记为$Q_{GAPR}$。输出$Q_{GAPR}$的表达式如下：
$$ Q^{\prime}=\mathrm{MHA}(\mathrm{Norm}(Q_{\mathrm{C}}))+\mathrm{Conv}(\mathrm{Norm}(Q_{\mathrm{C}}))+Q_{\mathrm{C}} $$
$$ Q_{\mathrm{G A P R}}=\mathrm{S E}(\mathrm{N o r m}(Q^{\prime})) $$
其中$Q_C \in \mathbb{R}^{K \times D}$代表MANO姿态令牌及2D姿态上下文查询，而$Q_{\text{GAPR}}$则是经过稳定化处理、解剖学上一致的姿态表示。这些优化后的令牌$Q_{\text{GAPR}}$随后会被送入上下文注入式掩码合成模块，以生成精确的3D手部网格。
上下文注入式掩码合成模块。我们利用多层Transformer（如图4所示），其输入为优化后的姿态令牌$Q_{GAPR}$，并与图像编码器生成的多尺度特征图进行交叉注意力计算。为了提高处理高分辨率特征图时的计算效率，我们采用了可变形交叉注意力机制[^75]。该机制允许每个姿态令牌聚焦于围绕可学习参考点的一组选定采样点上，而非整个特征图。通过集中注意力于相关区域，模型能够实现……
---
在计算效率与空间精度之间取得平衡，从而保留用于精确构建3D手部模型的关键信息。可变形交叉注意力机制定义如下：
$$ \mathbf{M C D A}(Q_{\mathrm{G A P R}},\hat{p}_{y},\{x^{l}\})=\sum_{l,k}A_{l y k}\cdot\mathbf{W}x^{l}(\hat{p}_{y}+\Delta p_{l y k}) $$
其中，$Q_{GAPR}$ 是经过优化的ManoPose词元查询向量，$\hat{p}_y$ 是可学习的参考点，$\Delta p_{lyk}$ 是采样偏移量，$\{x^l\}$ 是多尺度特征，$A_{lyk}$ 是注意力权重，而 $\mathbf{W}$ 则是可学习的权重矩阵。通过在掩码Transformer解码器中加入[MASK]词元，该模块在训练阶段能够预测被掩码的姿势词元，同时在推理阶段也能辅助生成词元。这种方法利用[MASK]词元作为姿势预测的占位符，从而实现遮挡或未被观测到的手部部位的稳健合成，最终完成连贯的3D手部重建。
#### 3.3. 训练：差分掩码建模
基于上下文条件的掩码建模。我们采用掩码建模方法来训练模型，使其能在多种上下文线索的条件下学习3D手部姿势的概率分布。给定来自姿势分词器的离散姿势词元序列 $Y = [y_i]_{i=1}^L$，其中L为序列长度，我们会随机掩码掉其中的m个词元，掩码标记即为 $m = \lceil \gamma(\tau) \cdot L \rceil$，此处 $\gamma(\tau)$ 是基于余弦函数的掩码比例函数。这里，$\tau$ 是从均匀分布 $U(0,1)$ 中采样的，我们采用了受生成式文本到图像建模策略启发的掩码函数 $\gamma(\tau) = \cos\left(\frac{\pi\tau}{2}\right)$ [^8]。被掩码的词元会被替换为可学习的[MASK]词元，从而形成模型需要重构的损坏序列 $Y_{\overline{M}}$。每个词元 $y_i$ 的预测都基于概率分布 $p(y_i | Y_{\overline{M}}, X_{2D}, X_{img})$，该分布以损坏的词元序列 $Y_{\overline{M}}$、2D姿势嵌入 $X_{2D}$ 以及图像提示 $X_{img}$ 为条件。这种方法使模型能够明确考虑将2D观测结果映射为连贯3D手部网格时所存在的不确定性。训练目标是最小化正确预测序列中每个姿势词元的负对数似然值，其公式如下：
$$ \mathcal{L}_{\mathrm{m a s k}}=-\mathbb{E}_{Y\in\mathcal{D}}\left[\sum_{\forall i\in[1,L]}\log p(y_{i}|Y_{\overline{{\mathbf{M}}}},X_{2D},X_{i m g})\right]. $$
期望近似差分采样。训练目标 $\mathcal{L}_{mask}$ 能够捕捉手部网格重建中的随机不确定性，从而在结构化的隐空间中精准估计姿势参数 $\theta$。近期研究 [^48] 表明，辅助的3D和2D关节损失能够通过对齐预测的2D/3D关节与真实关节来提升姿势恢复效果。要整合这些损失，需要将隐式姿势词元转换为 $\theta$，这一过程涉及不可微的概率采样操作。为解决这一问题，我们并未直接从分布 $p(y_i | Y_{\overline{M}}, X_{2D}, X_{img})$ 中采样以获取码本中最可能的词元，
Figure 5. MaskHand Inference Phase: Confidence-Guided Iterative Sampling — a step-by-step refinement of pose selection by probabilistically sampling high-confidence tokens.
而是采用了基于期望的差分松弛方法：模型无需直接估计离散的码索引，而是输出每个词元的对数几率L。这些对数几率经过softmax运算后生成词元分布，再将该分布与预训练的码本相乘，最终得到均值量化特征表示 $\bar{z} = [\bar{z}_1, \bar{z}_2, \ldots, \bar{z}_M]$：
$$ \bar{z}=\mathrm{softmax}\left(L_{M\times K}\right)\times\mathrm{CB}_{K\times D} $$
其中，L代表logits矩阵，CB表示码本，M为词元数量，K指定码本大小，D则定义了每个码本条目的维度。这意味着词元嵌入$\bar{z}$会被输入解码器以重构姿态参数$\theta^{\prime}$。结合形状参数$\beta$和相机参数$\pi$，这一过程便可重构出3D手部网格，进而计算出3D关节损失和2D投影损失。由于$\bar{z}$是可微的，因此模型可以利用总损失函数进行端到端训练。
$$ \mathcal{L}_{\mathrm{t o t a l}}=\mathcal{L}_{\mathrm{m a s k}}+\mathcal{L}_{\mathrm{M A N O}}+\mathcal{L}_{\mathrm{3D}}+\mathcal{L}_{\mathrm{2D}} $$
该损失函数结合了掩码词元预测损失($L_{mask}$)、3D关节损失($L_{3D}$)、2D投影损失($L_{2D}$)以及MANO参数损失$L_{MANO}$。这些项共同作用，可最小化MANO空间内的差异，从而实现精确的三维手部重建。
#### 3.4. 推理：基于置信度的采样
我们的模型利用基于置信度的采样技术来实现精确且稳定的3D手部姿态预测，如图5所示。这一过程始于长度为L的全掩码序列$Y_1$，其中每个词元初始值均为[MASK]。在T次解码迭代中，第t次迭代会应用随机采样，根据分布$p(y_i | Y_{\overline{\mathbf{M}}}, X_{2D}, X_{img})$来预测被掩码的词元。每次采样步骤之后，预测置信度最低的词元会被重新掩码，以便在后续迭代中重新预测。被重新掩码的词元数量由
---
Figure 6. Comparison of SOTA methods: MaskHand synthesizes unobserved parts for accurate 3D reconstructions in occluded scenarios.
掩码调度机制$\lceil \gamma \left( \frac{t}{T} \right) \cdot L \rceil$决定，其中$\gamma$是t的衰减函数。该调度机制会根据置信度动态调整掩码强度：在预测置信度较低的早期迭代中采用更高的掩码比例。如此一来，模型便能逐步细化模糊区域，随着上下文信息的积累，预测置信度也会不断提升。每一步的掩码比例都会降低，并由余弦衰减函数$\gamma$加以稳定。
### 4. 实验
| Method | PA-MPJPE\nmm( $\downarrow$) | PA-MPVPE\nmm( $\downarrow$) | F@5\n( $\uparrow$) | F@15\n( $\uparrow$) | AUC $_j$ | AUC $_v$ |
| S $^2$HAND [^13] | 11.5 | 11.1 | 0.448 | 0.932 | 0.769 | 0.778 |
| KPT-Transf. [^28] | 10.9 | - | - | - | 0.785 | - |
| ArtiBoost [^67] | 10.8 | 10.4 | 0.507 | 0.946 | 0.785 | 0.792 |
| Yu et al. [^69] | 10.8 | 10.4 | - | - | - | - |
| HandGCAT [^63] | 9.3 | 9.1 | 0.552 | 0.956 | 0.814 | 0.818 |
| AMVUR [^32] | 8.7 | 8.3 | 0.593 | 0.964 | 0.826 | 0.834 |
| HMP [^20] | 10.1 | - | - | - | - | - |
| SPMHand [^42] | 8.8 | 8.6 | 0.574 | 0.962 | - | - |
| MaskHand | 7.0 | 7.0 | 0.663 | 0.984 | 0.860 | 0.860 |
Table 1. Zero-Shot 3D Mesh Reconstruction Evaluation on HO3Dv3 [^28] Benchmark: SOTA Comparison. "-" indicates unreported metrics.
训练数据集。为了训练手部姿态分词器，我们采用了多种数据集来覆盖广泛的手部姿态与交互情况。这些数据集包括DexYCB [^10]、InterHand2.6M [^45]、MTC [^65]和RHD [^76]。为了训练MaskHand，我们参照[^48]中的类似设置构建了一个综合数据集，以确保对比的公平性。具体而言，在HO3Dv3 [^28]和HInt [^48]上的评估中，MaskHand是在包括FreiHAND [^77]、HODv2 [^27]、MTC [^65]、RHD [^76]、InterHand2.6M [^45]、H2O3D [^27]、DexYCB [^10]、COCO Wholebody [^33]、Halpe [^22]以及MPII NZSL [^54]在内的多种数据集上训练的。不过，对于FreiHAND和DexYCB，MaskHand是在其各自对应的训练数据集上专门训练的。
| Methods | PA-MPJPE\nmm ( $\downarrow$) | PA-MPVPE\nmm ( $\downarrow$) | F@5\n( $\uparrow$) | F@15\n( $\uparrow$) |
| MeshGraphormer $^{\dagger}$ [^40] | 5.9 | 6.0 | 0.764 | 0.986 |
| FastMETRO [^16] | 6.5 | 7.1 | 0.687 | 0.983 |
| FastViT [^61] | 6.6 | 6.7 | 0.722 | 0.981 |
| AMVUR [^32] | 6.2 | 6.1 | 0.767 | 0.987 |
| Deformer [^68] | 6.2 | 6.4 | 0.743 | 0.984 |
| PointHMR [^34] | 6.1 | 6.6 | 0.720 | 0.984 |
| Zhou et al. [^74] | 5.7 | 6.0 | 0.772 | 0.986 |
| HaMeR [^48] | 6.0 | 5.7 | 0.785 | 0.990 |
| HHMR $^{\S}$ [^37] | 5.8 | 5.8 | - | - |
| MaskHand $^{\ddagger}$ | 5.5 | 5.4 | 0.801 | 0.991 |
Table 2. 3D Mesh Reconstruction Accuracy on FreiHAND [^77]: SOTA Comparison. ${}^{\dagger}$ Denotes Test-Time Augmentation. PA-MPJPE/PA-MPVPE in mm. “-” indicates unreported metrics.
MaskHand produces confident mesh reconstruction, guided by the learned 2D-to-3D distribution without assuming GT meshes. HHMR reports best hypothesis mesh reconstruction with minimum MPJPE and MPVPE, with the assumption of available GT meshes.
| Method | PA-MPJPE\n( $\downarrow$) | PA-MPVPE\n( $\downarrow$) | MPJPE\n( $\downarrow$) | MPVPE\n( $\downarrow$) |
| METRO [^70] | 7.0 | - | - | - |
| Spurr et al. [^55] | 6.8 | - | - | - |
| Liu et al. [^41] | 6.6 | - | - | - |
| HandOccNet [^46] | 5.8 | 5.5 | 14.0 | 13.1 |
| MobRecon [^11] | 6.4 | 5.6 | 14.2 | 13.1 |
| H2ONet [^66] | 5.7 | 5.5 | 14.0 | 13.0 |
| Zhou et al. [^74] | 5.5 | 5.5 | 12.4 | 12.1 |
| Ours | 5.0 | 4.9 | 11.7 | 11.2 |
Table 3. 3D Mesh Reconstruction Accuracy on DexYCB [^10]: SOTA Comparison. "-" indicates unreported metrics.
评估指标。我们遵循标准协议[^48]，使用PA-MPJPE和$AUC_{J}$来评估3D关节重建效果。对于3D网格评估，我们使用PA-MPVPE、$AUC_{V}$以及F@5mm/F@15mm来衡量顶点精度。此外，PCK通过将3D关节重投影到2D空间来评估2D重建效果。计算效率则通过每张图像的平均推理时间(AITI)（单位：秒）[^7]来衡量，数值越低代表性能越好。
---
| Method | Venue | NewDays | VISOR | Ego4D |
| @0.05 ( $\uparrow$) | @0.1 ( $\uparrow$) | @0.15 ( $\uparrow$) | @0.05 ( $\uparrow$) | @0.1 ( $\uparrow$) | @0.15 ( $\uparrow$) | @0.05 ( $\uparrow$) | @0.1 ( $\uparrow$) | @0.15 ( $\uparrow$) |
| Occluded Joints Evaluation |
| FrankMocap [^53] | ICCVW 2021 | 9.2 | 28.0 | 46.9 | 11.0 | 33.0 | 55.0 | 8.4 | 26.9 | 45.1 |
| METRO [^39] | CVPR 2021 | 7.0 | 23.6 | 42.4 | 10.2 | 32.4 | 53.9 | 8.0 | 26.2 | 44.7 |
| MeshGraphormer [^40] | ICCV 2021 | 7.9 | 25.7 | 44.3 | 10.9 | 33.3 | 54.1 | 9.3 | 32.6 | 51.7 |
| HandOccNet (param) [^47] | CVPR 2022 | 7.2 | 23.5 | 42.4 | 7.4 | 26.1 | 46.7 | 7.2 | 26.1 | 45.7 |
| HandOccNet (no param) [^47] | CVPR 2022 | 9.8 | 31.2 | 50.8 | 9.9 | 33.7 | 55.4 | 9.6 | 31.1 | 52.7 |
| HaMeR [^48] | CVPR 2024 | 27.2 | 60.8 | 78.9 | 25.9 | 60.8 | 80.7 | 23.0 | 56.9 | 76.3 |
| MaskHand | Ours | 29.4 | 64.1 | 80.3 | 31.4 | 64.2 | 83.6 | 29.4 | 65.3 | 80.1 |
| Occluded Joints Evaluation with 80% Masked Hand |
| HaMeR [^48] | CVPR 2024 | 8.7 | 28.4 | 48.9 | 7.1 | 24.4 | 41.7 | 7.1 | 23.4 | 40.2 |
| MaskHand | Ours | 10.5 | 31.6 | 52.0 | 8.2 | 25.0 | 41.9 | 8.2 | 26.2 | 44.1 |
| Occluded Joints Evaluation with 90% Masked Hand |
| HaMeR [^48] | CVPR 2024 | 8.7 | 27.4 | 48.1 | 7.1 | 24.3 | 42.2 | 7.1 | 23.4 | 40.2 |
| MaskHand | Ours | 10.4 | 31.0 | 51.1 | 9.1 | 27.3 | 44.3 | 8.0 | 25.1 | 43.2 |
Table 4. Zero-shot evaluation on the HInt Benchmark [^48] for occluded joint reconstruction using the PCK metric, comparing MaskHand to SOTA methods. MaskHand demonstrates superior robustness and resilience to missing hand structures under severe occlusions, maintaining strong performance even with 80% and 90% masked hand regions. None of the models were trained on the HInt dataset.
重建效果评估。我们在HO3Dv3 [^28]、DexYCB [^10]、FreiHAND [^77]和HInt [^48]基准测试集上评估了MaskHand的关节与网格重建性能。
零样本泛化能力评估：包含2万张标注图像的HO3Dv3数据集，被用作评估复杂手-物交互中模型泛化能力的基准。我们在未对MaskHand进行预先训练的情况下，在HO3Dv3上测试它，以检验其处理未见数据、遮挡、数据集偏差以及现实场景中多样手部姿态的能力。此外，我们还在HInt基准测试集 [^48] 上评估MaskHand的零样本性能，该测试集涵盖了光照多变、视角极端以及复杂的手-物交互等挑战。借助PCK指标，我们在HInt NewDays [^14]、HInt-EpicKitchensVISOR [^17]和HInt-Ego4D [^26]上对MaskHand进行测试，验证了其在多样化且遮挡严重的环境中运行的能力。
有监督评估：我们在FreiHAND [^77]和DexYCB [^10]数据集上评估了MaskHand的性能。前者包含4千张在受控环境下采集的图像，后者则是一个包含7.8万张手与物体交互图像的大规模数据集。为了公平比较，我们严格按照MeshGraphormer [^40]和HHMR [^37]的评估协议，仅在FreiHAND数据集上训练MaskHand。同样，参照近期相关研究 [^74]，我们在DexYCB数据集上训练MaskHand，并在相同条件下开展评估，以确保对比的直接性与一致性。
#### 4.1. 与最先进方法的对比
我们在HO3Dv3、FreiHAND、DexYCB和HInt数据集上评估了MaskHand（见表1–4），结果显示它优于现有最先进方法，展现出强大的零样本泛化能力和野外场景鲁棒性。与以往的方法不同，MaskHand利用基于置信度的掩码建模技术来有效应对不确定性、遮挡以及复杂的手部姿态问题。在HO3Dv3上的零样本评估中（表1），MaskHand将PA-MPJPE降低了19.5%，PA-MPVPE降低了15.7%，体现了出色的跨数据集重建能力。在FreiHAND数据集上（表2），其PA-MPJPE降低了3.5%，PA-MPVPE降低了5.3%，同时F@5mm（0.801）和F@15mm（0.991）指标均位居第一，确保了高精度、高保真的手部恢复效果。在DexYCB数据集上（表3），MaskHand将MPJPE降低了9.1%，MPVPE降低了10.9%，证实了其在现实世界手-物交互中的鲁棒性。在HInt数据集上的零样本评估中（表4），由于该数据集包含极具挑战性的 Egocentric 视角及严重遮挡的手部姿态，MaskHand在遮挡关节重建方面取得了最先进的成果：在HInt-NewDays上PCK@0.05指标提升了8.1%，在HInt-VISOR上提升了21.2%，在HInt-Ego4D上更是提升了27.8%。从定性结果来看，图6展示了MaskHand相比以往方法在遮挡区域合成方面的优势，而图7则体现了其在极端遮挡情况下的鲁棒性。这些结果凸显了MaskHand在具有挑战性的现实场景中的强大泛化能力与实用价值。
#### 4.2. 消融实验
MaskHand之所以能取得优异效果，关键在于其生成式掩码建模与不确定性量化机制，这使其能够在遮挡区域实现稳健的重建。其迭代解码策略能够逐步优化预测结果，从而在复杂场景中不断提升精度。本次消融实验重点探讨了迭代细化机制以及MaskHand在文本条件驱动的3D手部合成中的多功能性。
补充材料中的进一步分析、实验细节和定性可视化结果探讨了关键方面，包括架构选择（如VQ-MANO分词器、所提出的模块、多尺度特征分辨率、可变形注意力层）、训练策略（掩码比例、通过差分采样实现的正则化损失）以及MaskHand的高级功能，例如无条件网格生成和在严重现实遮挡下的稳健重建能力。
---
Figure 7. SOTA Comparison: Qualitative zero-shot evaluation on HInt Benchmark [^48] for heavily masked hand images. MaskHand reconstructs occluded hand poses, demonstrating robustness to severe occlusions and generalizability to unseen masked regions.
在严重的现实世界遮挡情况下，MaskHand仍能实现稳健的重建。
迭代式置信度引导采样的影响。在置信度引导采样中，我们通过保留高置信度的标记并重新掩码低置信度的标记来迭代优化预测结果，从而逐步提升稳定性和准确性。如表5所示，将迭代次数从1增加到5后，PA-MPVPE值有所降低（HO3Dv3数据集上从7.2降至7.0，FreiHAND数据集上从5.6降至5.4）。这些改进源于解码过程中对不确定区域的置信度引导掩码处理，进而提升了重建精度。
| # of iter. | $HO3Dv3$ | FreiHAND | AITI (s) |
| PA-MPJPE | PA-MPVPE | PA-MPJPE | PA-MPVPE | ( \downarrow ) |
| 1 | 7.2 | 7.2 | 5.6 | 5.6 | 0.04 |
| 3 | 7.1 | 7.1 | 5.6 | 5.5 | 0.08 |
| 5 | 7.0 | 7.0 | 5.5 | 5.4 | 0.12 |
Table 5. Iterations in Iterative Confidence-Guided Sampling. AITI is obtained on a single mid-grade GPU (NVIDIA RTX A5000).
#### 4.3. 应用：文本到网格生成
MaskHand的模块化设计不仅适用于手部网格重建，还可用于基于文本的3D手部合成。通过将图像编码器替换为基于CLIP的文本编码器[^51]，MaskHand能够根据文本输入生成3D手部网格，如图8所示。我们利用ASL数据集[^1]，借助MaskHand生成的伪真实标注对模型进行训练。测试阶段采用前5%概率的采样策略，以确保生成的网格既多样化又合理，评估指标包括豪斯多夫距离、倒角距离和PA-MPVPE。表6的结果表明，基于文本驱动的重建具有很高的保真度和一致性，定性示例见图9。更多细节详见补充材料的B节。
| Metric | Mean | Standard Deviation |
| Hausdorff Distance | 0.0221 | 0.0073 |
| Chamfer Distance | $9.73 \times 10^{-5}$ | $5.47 \times 10^{-5}$ |
| PA-MPVPE (mm) | 12.2 | 3.1 |
Table 6. Quantitative evaluation of text-to-mesh generation using MaskHand's Masked Synthesizer, reporting mean and standard deviation across all generated meshes per text prompt.
Figure 8. Repurposed MaskHand for Text-to-Mesh Generation
Figure 9. Text-to-mesh generation results on the ASL dataset, demonstrating MaskHand's Masked Synthesizer's ability to generate structurally accurate and diverse meshes from text inputs.
### 5. 结论
我们提出了MaskHand，这是一种用于准确且稳健的3D手部网格重建的生成式掩码模型，尤其适用于存在遮挡和模糊情况的场景。通过显式学习并从概率化的2D到3D映射中采样，MaskHand能够有效处理自遮挡、深度模糊以及手部缺失部分的问题。其核心组件VQ-MANO将3D手部动作编码为离散的姿态标记，从而实现结构化的隐式表示。上下文引导的掩码Transformer通过融入图像上下文、2D姿态线索和掩码标记分布来建模标记间的依赖关系。借助置信度引导的迭代采样，MaskHand逐步优化重建结果，选择性地更新不确定区域，最终生成解剖结构合理且精度高的手部网格。在遮挡严重的场景下，MaskHand的性能优于现有最先进方法，为人机交互、增强现实和虚拟现实领域的3D手部建模树立了新标杆。
---
## 参考文献
[^1]: Kaggle asl alphabet. Available online: (accessed on 19 July 2021). 8
[^2]: Dosovitskiy Alexey. An image is worth $16 \times 16$ words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020. 3
[^3]: Seungryul Baek, Kwang In Kim, and Tae-Kyun Kim. Augmented skeleton space transfer for depth-based hand pose estimation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018. 1
[^4]: Seungryul Baek, Kwang In Kim, and Tae-Kyun Kim. Pushing the envelope for rgb-based dense 3d hand pose estimation via neural rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1067–1076, 2019. 2
[^5]: Luca Ballan, Aparna Taneja, Jürgen Gall, Luc Van Gool, and Marc Pollefeys. Motion capture of hands in action using discriminative salient points. In European Conference on Computer Vision (ECCV), 2012. 1
[^6]: Adnane Boukhayma, Rodrigo de Bem, and Philip HS Torr. 3d hand shape and pose from images in the wild. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10843–10852, 2019. 2
[^7]: Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. Realtime multi-person 2d pose estimation using part affinity fields. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7291–7299, 2017. 4, 6
[^8]: Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11315–11325, 2022. 3, 5
[^9]: Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming-Hsuan Yang, Kevin Murphy, William T Freeman, Michael Rubinstein, et al. Muse: Text-to-image generation via masked generative transformers. arXiv preprint arXiv:2301.00704, 2023. 3
[^10]: Yu-Wei Chao, Wei Yang, Yu Xiang, Pavlo Molchanov, Ankur Handa, Jonathan Tremblay, Yashraj S Narang, Karl Van Wyk, Umar Iqbal, Stan Birchfield, et al. Dexycb: A benchmark for capturing hand grasping of objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9044–9053, 2021. 6, 7
[^11]: Xingyu Chen, Yufeng Liu, Yajiao Dong, Xiong Zhang, Chongyang Ma, Yanmin Xiong, Yuan Zhang, and Xiaoyan Guo. Mobrecon: Mobile-friendly hand mesh reconstruction from monocular image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20544–20554, 2022. 6
[^12]: Xingyu Chen, Baoyuan Wang, and Heung-Yeung Shum. Hand avatar: Free-pose hand animation and rendering from monocular video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023. 1
[^13]: Yujin Chen, Zhigang Tu, Di Kang, Linchao Bao, Ying Zhang, Xuefei Zhe, Ruizhi Chen, and Junsong Yuan. Model-
基于自监督学习的3D手部重建。载于IEEE/CVF计算机视觉与模式识别会议论文集，第10451–10460页，2021年。6
[^14]: Tianyi Cheng, Dandan Shan, Ayda Hassen, Richard Higgins, and David Fouhey. Towards a richer 2d understanding of hands at scale. Advances in Neural Information Processing Systems, 36:30453–30465, 2023. 7
[^15]: Junhyeong Cho, Kim Youwang, and Tae-Hyun Oh. Cross-attention of disentangled modalities for 3d human mesh recovery with transformers. In European Conference on Computer Vision (ECCV), 2022. 2
[^16]: Youwang Kim Oh Tae-Hyun Cho, Junhyeong. Cross-attention of disentangled modalities for 3d human mesh recovery with transformers. In European Conference on Computer Vision, pages 342–359. Springer, 2022. 6
[^17]: Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. Scaling egocentric vision: The epic-kitchens dataset. In Proceedings of the European conference on computer vision (ECCV), pages 720–736, 2018. 7
[^18]: Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arxiv. arXiv preprint arXiv:1810.04805, 2019. 3
[^19]: Ming Ding, Wendi Zheng, Wenyi Hong, and Jie Tang. Cogview2: Faster and better text-to-image generation via hierarchical transformers. Advances in Neural Information Processing Systems, 35:16890–16902, 2022. 3
[^20]: Enes Duran, Muhammed Kocabas, Vasileios Choutas, Zicong Fan, and Michael J Black. Hmp: Hand motion priors for pose and shape estimation from video. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 6353–6363, 2024. 6
[^21]: Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12873–12883, 2021. 3
[^22]: Hao-Shu Fang, Jiefeng Li, Hongyang Tang, Chao Xu, Haoyi Zhu, Yuliang Xiu, Yong-Lu Li, and Cewu Lu. Alphapose: Whole-body regional multi-person pose estimation and tracking in real-time. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(6):7157–7173, 2022.
[^23]: Chengying Gao, Yujia Yang, and Wensheng Li. 3d interacting hand pose and shape estimation from a single rgb image. Neurocomputing, 2022. 2
[^24]: Guillermo Garcia-Hernando, Shanxin Yuan, Seungryul Baek, and Tae-Kyun Kim. First-person hand action benchmark with rgb-d videos and 3d hand pose annotations. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018. 1
[^25]: Francisco Gomez-Donoso, Sergio Orts-Escolano, and Miguel Cazorla. Large-scale multiview 3d hand pose dataset. arXiv preprint arXiv:1707.03742, 2017. 1
## [^26]: Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson
---
Hamburger、Hao Jiang、Miao Liu、Xingyu Liu等。Ego4d：历时3000小时的全景式第一人称视频研究。载于IEEE/CVF计算机视觉与模式识别会议论文集，第18995–19012页，2022年。7
[^27]: Shreyas Hampali, Mahdi Rad, Markus Oberweger, and Vincent Lepetit. Honnotate: A method for 3d annotation of hand and object poses. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3196–3206, 2020. 6
[^28]: Shreyas Hampali, Sayan Deb Sarkar, Mahdi Rad, and Vincent Lepetit. Keypoint transformer: Solving joint identification in challenging hands and object interactions for accurate 3d pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11090–11100, 2022. 6, 7
[^29]: Ludovic Hoyet, Kenneth Ryall, Rachel McDonnell, and Carol O'Sullivan. Sleight of hand: perception of finger motion from reduced marker sets. In SIGGRAPH, 2012. 2
[^30]: Jie Hu, Li Shen, and Gang Sun. Squeeze-and-excitation networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7132–7141, 2018. 4
[^31]: Rongtian Huo, Qing Gao, Jing Qi, and Zhaojie Ju. 3d human pose estimation in video for human-computer/robot interaction. In Intelligent Robotics and Applications, pages 176–187, Singapore, 2023. Springer Nature Singapore. 1
[^32]: Zheheng Jiang, Hossein Rahmani, Sue Black, and Bryan M Williams. A probabilistic attention model with occlusion-aware texture regression for 3d hand reconstruction from a single rgb image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 758–767, 2023. 6
[^33]: Sheng Jin, Lumin Xu, Jin Xu, Can Wang, Wentao Liu, Chen Qian, Wanli Ouyang, and Ping Luo. Whole-body human pose estimation in the wild. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part IX 16, pages 196–214. Springer, 2020. 6
[^34]: Jeonghwan Kim, Mi-Gyeong Gwon, Hyunwoo Park, Hyukmin Kwon, Gi-Mun Um, and Wonjun Kim. Sampling is matter: Point-guided 3d human mesh reconstruction. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pages 12880–12889, 2023. 6
[^35]: Dominik Kulon, Haoyang Wang, Riza Alp Güler, Michael Bronstein, and Stefanos Zafeiriou. Single image 3d hand reconstruction with mesh convolutions. arXiv preprint arXiv:1905.01326, 2019. 2
[^36]: Dominik Kulon, Riza Alp Guler, Iasonas Kokkinos, Michael M Bronstein, and Stefanos Zafeiriou. Weakly-supervised mesh-convolutional hand reconstruction in the wild. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4990–5000, 2020. 2
[^37]: Mengcheng Li, Hongwen Zhang, Yuxiang Zhang, Ruizhi Shao, Tao Yu, and Yebin Liu. Hhmr: Holistic hand mesh recovery by enhancing the multimodal controllability of graph diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 645–654, 2024. 3, 6, 7
[^38]: Yanghao Li, Hanzi Mao, Ross Girshick, and Kaiming He. Exploring plain vision transformer backbones for object detection. In European conference on computer vision, pages 280–296. Springer, 2022. 3
[^39]: Kevin Lin, Lijuan Wang, and Zicheng Liu. End-to-end human pose and mesh reconstruction with transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1954–1963, 2021. 7
[^40]: Kevin Lin, Lijuan Wang, and Zicheng Liu. Mesh graphormer. In Proceedings of the IEEE/CVF international conference on computer vision, pages 12939–12948, 2021. 2, 6, 7
[^41]: Shaowei Liu, Hanwen Jiang, Jiarui Xu, Sifei Liu, and Xiaolong Wang. Semi-supervised 3d hand-object poses estimation with interactions in time. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14687–14697, 2021. 6
[^42]: Haofan Lu, Shuiping Gou, and Ruimin Li. Spmhand: Segmentation-guided progressive multi-path 3d hand pose and shape estimation. IEEE Transactions on Multimedia, 2024. 6
[^43]: Anders Markussen, Mikkel Rønne Jakobsen, and Kasper Hornbæk. Vulture: a mid-air word-gesture keyboard. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, page 1073–1082, New York, NY, USA, 2014. Association for Computing Machinery. 1
[^44]: Gyeongsik Moon, Ju Yong Chang, and Kyoung Mu Lee. V2v-posenet: Voxel-to-voxel prediction network for accurate 3d hand and human pose estimation from a single depth map. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018. 1
[^45]: Gyeongsik Moon, Shoou-I Yu, He Wen, Takaaki Shiratori, and Kyoung Mu Lee. Interhand2.6m: A dataset and baseline for 3d interacting hand pose estimation from a single rgb image. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XX 16, pages 548–564. Springer, 2020. 6
[^46]: JoonKyu Park, Yeonguk Oh, Gyeongsik Moon, Hongsuk Choi, and Kyoung Mu Lee. Handoccnet: Occlusion-robust 3d hand mesh estimation network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022. 6
[^47]: JoonKyu Park, Yeonguk Oh, Gyeongsik Moon, Hongsuk Choi, and Kyoung Mu Lee. Handoccnet: Occlusionrobust 3d hand mesh estimation network. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1496–1505, 2022. 7
[^48]: Georgios Pavlakos, Dandan Shan, Ilija Radosavovic, Angjoo Kanazawa, David Fouhey, and Jitendra Malik. Reconstructing hands in 3d with transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9826–9836, 2024. 2, 3, 5, 6, 7, 8
## [^49]: Rolandos Alexandros Potamias, Jinglei Zhang, Jiankang Deng, and Stefanos Zafeiriou. Wilor: End-to-end 3d hand localization and reconstruction in-the-wild. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 12242–12254, 2025. 2
---
[^50]: Chen Qian, Xiao Sun, Yichen Wei, Xiaoou Tang, and Jian Sun. Realtime and robust hand tracking from depth. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014. 1
[^51]: Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PMLR, 2021. 8
[^52]: Javier Romero, Dimitrios Tzionas, and Michael J. Black. Embodied hands: Modeling and capturing hands and bodies together. ACM Transactions on Graphics, (Proc. SIGGRAPH Asia), 36(6), 2017. 2, 3
[^53]: Yu Rong, Takaaki Shiratori, and Hanbyul Joo. Frankmocap: A monocular 3d whole-body pose estimation system via regression and integration. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1749–1759, 2021. 7
[^54]: Tomas Simon, Hanbyul Joo, Iain Matthews, and Yaser Sheikh. Hand keypoint detection in single images using multiview bootstrapping. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 1145–1153, 2017. 1, 6
[^55]: Adrian Spurr, Umar Iqbal, Pavlo Molchanov, Otmar Hilliges, and Jan Kautz. Weakly supervised 3d hand pose estimation via biomechanical constraints. In European conference on computer vision, pages 211–228. Springer, 2020. 6
[^56]: Srinath Sridhar, Antti Oulasvirta, and Christian Theobalt. Interactive markerless articulated hand motion tracking using rgb and depth data. In IEEE International Conference on Computer Vision (ICCV), 2013. 1
[^57]: Srinath Sridhar, Anna Maria Feit, Christian Theobalt, and Antti Oulasvirta. Investigating the dexterity of multi-finger input for mid-air text entry. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, page 3643–3652, New York, NY, USA, 2015. Association for Computing Machinery. 1
[^58]: Srinath Sridhar, Franziska Mueller, Antti Oulasvirta, and Christian Theobalt. Fast and robust hand tracking using detection-guided optimization. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015. 1
[^59]: Jonathan Taylor, Vladimir Tankovich, Danhang Tang, Cem Keskin, David Kim, Philip Davidson, Adarsh Kowdle, and Shahram Izadi. Articulated distance fields for ultra-fast tracking of hands interacting. TOG, 2017. 2
[^60]: Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017. 2, 3
[^61]: Pavan Kumar Anasosalu Vasu, James Gabriel, Jeff Zhu, Oncel Tuzel, and Anurag Ranjan. Fastvit: A fast hybrid vision transformer using structural reparameterization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5785–5795, 2023. 6
[^62]: Jiayi Wang, Franziska Mueller, Florian Bernard, Suzanne Sorli, Oleksandr Sotnychenko, Neng Qian, Miguel A Otaduy, Dan Casas, and Christian Theobalt. Rgb2hands:
基于单目摄像头对三维手部交互的实时追踪
RGB视频。TOG，2020年，第2期。
[^63]: Shuaibing Wang, Shunli Wang, Dingkang Yang, Mingcheng Li, Ziyun Qian, Liuzhen Su, and Lihua Zhang. Handgcat: Occlusion-robust 3d hand mesh reconstruction from monocular images. In 2023 IEEE International Conference on Multimedia and Expo (ICME), pages 2495–2500. IEEE, 2023. 6
[^64]: Will Williams, Sam Ringer, Tom Ash, David MacLeod, Jamie Dougherty, and John Hughes. Hierarchical quantized autoencoders. Advances in Neural Information Processing Systems, 33:4524–4535, 2020. 3
[^65]: Donglai Xiang, Hanbyul Joo, and Yaser Sheikh. Monocular total capture: Posing face, body, and hands in the wild. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10965–10974, 2019. 6
[^66]: Hao Xu, Tianyu Wang, Xiao Tang, and Chi-Wing Fu. H2onet: Hand-occlusion-and-orientation-aware network for real-time 3d hand mesh reconstruction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 17048–17058, 2023. 6
[^67]: Lixin Yang, Kailin Li, Xinyu Zhan, Jun Lv, Wenqiang Xu, Jiefeng Li, and Cewu Lu. Artiboost: Boosting articulated 3d hand-object pose estimation via online exploration and synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2750–2760, 2022. 6
[^68]: Yusuke Yoshiyasu. Deformable mesh transformer for 3d human mesh recovery. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17006–17015, 2023. 6
[^69]: Ziwei Yu, Linlin Yang, You Xie, Ping Chen, and Angela Yao. Uv-based 3d hand-object reconstruction with grasp optimization. arXiv preprint arXiv:2211.13429, 2022. 6
[^70]: Xiong Zhang, Qiang Li, Hong Mo, Wenbo Zhang, and Wen Zheng. End-to-end hand mesh recovery from a monocular rgb image. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2354–2364, 2019. 2, 6
[^71]: Yuxiang Zhang, Zhe Li, Liang An, Mengcheng Li, Tao Yu, and Yebin Liu. Light-weight multi-person total capture using sparse multi-view cameras. In ICCV, 2021. 2
[^72]: Zhu Zhang, Jianxin Ma, Chang Zhou, Rui Men, Zhikang Li, Ming Ding, Jie Tang, Jingren Zhou, and Hongxia Yang. Ufcbert: Unifying multi-modal controls for conditional image synthesis. Advances in Neural Information Processing Systems, 34:27196–27208, 2021. 3
[^73]: Wenping Zhao, Jinxiang Chai, and Ying-Qing Xu. Combining marker-based mocap and rgb-d camera for acquiring high-fidelity hand motion data. In SIGGRAPH, 2012. 2
[^74]: Zhishan Zhou, Shihao Zhou, Zhi Lv, Minqiang Zou, Yao Tang, and Jiajun Liang. A simple baseline for efficient hand mesh reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1367–1376, 2024. 6, 7
## [^75]: Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. ArXiv, abs/2010.04159, 2020. 4
---
[^76]: Christian Zimmermann and Thomas Brox. Learning to estimate 3d hand pose from single rgb images. In Proceedings of the IEEE international conference on computer vision, pages 4903–4911, 2017. 6
[^77]: Christian Zimmermann, Duygu Ceylan, Jimei Yang, Bryan Russell, Max Argus, and Thomas Brox. Freihand: A dataset for markerless capture of hand pose and shape from single rgb images. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 813–822, 2019. 6,7