# WiLoR：野外环境下的端到端手部定位与重建
Rolandos Alexandros Potamias $^{1}$、张景雷 $^{2}$、
邓建康 $^{1}$、Stefanos Zafeiriou $^{1}$
$^{1}$帝国理工学院，$^{2}$上海交通大学
Figure 1. We propose WiLoR, a full-stack in-the-Wild Localization and 3D hand Reconstruction method. WiLoR first localizes and defines the handedness of the detected hands which are then lifted to 3D using a transformer-based hand pose estimation module. To aid high-fidelity reconstructions and facilitate image-alignment, we introduce a refinement module that extracts localized features to correct misaligned poses. WiLoR achieves state-of-the-art performance under different benchmark datasets while boosting the temporal coherence of image-based 3D hand pose estimation methods.
## 摘要
近年来，3D手部姿态估计方法因其在人机交互、虚拟现实和机器人领域的广泛应用而备受关注。相比之下，手部检测流程存在明显短板，这给构建高效的真实场景多手重建系统带来了巨大挑战。在这项工作中，我们提出了一种面向野外环境的高效多手重建数据驱动流程。该流程由两部分组成：实时全卷积手部定位模块和高保真度基于Transformer的3D手部重建模型。为克服现有方法的局限并构建鲁棒且稳定的检测网络，我们构建了一个包含超过200万张野外手部图像的大规模数据集，这些图像涵盖了多样的光照、照明及遮挡条件。在主流的2D和3D基准测试中，我们的方法在效率和精度方面均优于以往的方法。最后，我们
展示了该流程在单目视频中实现流畅3D手部跟踪的有效性，且无需使用任何时序组件。代码、模型和数据集可在我们的项目页面上获取。
### 1. 引言
由于手部检测与重建有诸多应用场景，从虚拟现实 [^22] 到手语识别 [^2] [^30] [^102] 以及人类行为识别 [^65]，它一直是长期研究的课题。考虑到手部外观和关节活动存在较大差异 [^64]，加之手部交互过程中通常会出现严重遮挡和运动模糊现象 [^24] [^26]，手部姿态估计任务极具挑战性。多年来，人们提出了多种方法来解决3D手部姿态估计问题 [^11] [^42] [^49] [^61]。不过，尽管这些方法能产出可信的结果，但它们主要针对包含固定数量手部的图像，因此无法推广到野外图像。
---
由于只能处理固定数量的手部图像，因此无法推广到真实场景中的图像。
在密切相关的3D人体和面部重建领域，最先进的方法[^7] [^17] [^21] [^23]采用基于高性能检测模型的自下而上的流程，这些模型最初能在图像中定位人体和面部，从而使其能推广到自然场景图像中。尽管已有众多方法被提出来解决人体和面部检测任务，但实时手部检测方法却明显不足。考虑到当前3D手部姿态估计框架依赖于手部区域周围的紧凑裁剪区[^10] [^97]，手部检测器的重要性愈发凸显。因此，手部检测器对于此类方法在自然场景中的推广应用至关重要。主流的手部检测与定位方法[^7] [^92]在检测多只手及复杂姿态时表现欠佳，而较新的方法[^39] [^52] [^53]虽能得出合理结果，却难以实现实时运行。鉴于缺乏精准的手部检测框架，我们提出了一种稳健的单状态锚定检测器，其运行帧率可超过100帧/秒。实验表明，稳健的检测结果能够促成更稳定的4D重建，并消除抖动伪影——而这正是当前基于帧的3D姿态估计方法的主要局限之一。
与尚未得到充分探索的手部检测与定位问题相比，3D手部姿态估计受到了更多关注。早期的3D姿态重建方法主要依赖传统的卷积骨干网络来处理和提取图像特征[^5] [^11] [^36] [^49]。随着Transformer的成功及其处理海量数据的能力得到验证[^14] [^37] [^70] [^83] [^85] [^94]，多种方法开始利用Transformer架构来提升3D人体和手部恢复的性能[^41] [^42] [^61]。近期，Pavlakos等人[^61]展示了视觉Transformer（ViT）的有效性，他们采用了一个简单却强大的框架，并在大规模数据集上进行了训练。该方法的成功关键在于其架构规模——包含超过5亿个参数，从而能够有效处理海量数据。然而，正如文献中所揭示的那样[^40] [^55] [^79] [^93]，从单张图像中直接回归手部参数会导致对齐效果不佳且姿态错误。目前，旨在改善图像对齐效果的方法往往采用次优方案，例如中间热图表示[^10] [^34] [^97]。为解决这一问题，我们提出了一种高保真度的3D姿态估计方法，将3D手部重建分为两个阶段：解码器首先预测粗略的手部估计结果，再利用该结果从细化模块中提取多尺度、与图像对齐的特征。借助这种粗略的手部估计，我们能够提取出有意义的空间特征，从而实现更好的图像对齐，并在FreiHand[^101]和HO3D[^24]基准测试集上取得业界领先的性能。此外，与直接回归3D顶点的顶点回归方法[^36] [^41] [^42]不同，我们的方法预测的是MANO参数[^71]，从而确保手部姿态既具有可解释性又符合物理规律。
在本文中，我们提出了一种高保真全栈方法，能够实时重建3D手部模型。具体而言：
鉴于当前手部检测基准数据集的局限性，我们收集了一个包含多只手的自然场景图像的大规模数据集，并推出了一个具有挑战性的手部检测基准测试。我们将该数据集以及对应的2D和3D标注公开提供给大家。
- 我们提出了一种基于所收集的大规模数据集训练的实时手部检测方法，其在准确性和效率方面均大幅优于以往的手部检测方法。
- 我们提出了一种基于Transformer的方法，能够实现高保真的3D重建，并通过新颖的优化模块解决了以往方法的架构局限。该方法不仅效率极高，而且在Freihand和HO3D基准测试集上均取得了最先进的性能。
### 2. 相关工作
手部检测与跟踪。物体检测在文献中已被广泛研究，取得了显著进展[^44] [^47] [^67]，为人体[^17] [^19] [^58] [^63] [^80] [^90]及面部检测[^12] [^99]流程奠定了基础。相比之下，尽管经过二十年的研究努力[^69]，手部检测尚未取得可相提并论的重大突破。早期的研究采用受控条件和深度相机[^77] [^78] [^87] [^88] [^98]来检测和跟踪人手。人们也尝试通过多阶段框架[^48] [^62]来提升不同肤色和背景下手部检测的性能，但这类方法在复杂环境中泛化能力较差。借鉴物体检测领域的成功经验，多种方法采用了全卷积架构来进行手部检测[^13] [^29] [^72] [^73] [^92]。Simon等人[^75]提出了一种多视角引导标注流程，用于标注自然场景数据并训练实时卷积检测网络。近期，Narasimhaswamy等人[^52]对MaskRCNN[^28]网络进行了扩展，用于检测自然场景中的手部并识别其对应接触点[^53]及身体关联[^54]。然而，尽管文献中已有大量相关研究，大多数方法仍依赖运算速度较慢的骨干网络，且在处理复杂图像时表现不佳。主要问题在于缺乏包含多种遮挡情况和运动模糊现象的大规模自然场景训练数据。为解决这些局限，我们提出了一种更轻量化的手部检测器，其运行速度$45 \times$更快。
---
与以往的最先进检测器相比，该检测器是在包含多样化环境和遮挡情况的200万张自然场景图像上训练而成的。
3D手部姿态估计。与手部检测类似，早期的手部姿态估计方法依赖深度相机[^18] [^57] [^81]来重建3D手部模型。Boukhayma等人[^5]提出了首个完全可学习的流程，能够直接从RGB图像中回归MANO手部模型的参数[^71]。类似地，后续的一些研究采用热图[^95]和迭代优化[^1]来强制实现2D对齐。Kulon等人[^35] [^36]提出了另一种回归方法，直接回归3D顶点而非MANO姿态参数，该方法的表现显著优于以往的方法。为了应对3D姿态估计中的特定挑战，人们提出了多种改进方案，包括提升对遮挡[^59]和运动模糊[^56]的鲁棒性，以及加快推理速度[^10] [^97]。最近，Pavlakos等人[^61]强调了扩大训练数据量和模型容量的重要性。具体而言，他们借鉴了Vision Transformer（ViT）骨干网络在身体姿态估计中的成功经验[^6] [^20] [^40]，证明使用一种简单却高效的大规模Transformer架构，在多种数据集上训练后能够达到最先进的性能。然而，直接一次性从图像中回归MANO参数可能会导致对齐错误和姿态不准确。为解决这一问题，我们提出了一种新颖的优化层，利用网格对齐的多尺度特征来优化手部姿态。
### 3. WHIM数据集
缺乏高保真手部检测系统的一个重要原因在于具备多手部标注的自然场景数据集数量有限。为了构建稳健的手部检测框架，我们收集了一个包含数百万个自然场景下手部样本的大规模数据集（WHIM），这些样本涵盖了多样的姿态、光照条件、遮挡情况和肤色。
为了收集所提出的数据集，我们设计了一套自动标注流程，用于处理来自多样且具有挑战性的自然场景的YouTube视频。具体来说，我们选取了1400多个包含手部活动的YouTube视频，内容涉及手语、烹饪、日常活动、体育运动和游戏等，这些视频包含了自拍和外界视角、运动模糊、不同手部尺寸以及手部交互等情况。为了准确检测并标注每一帧中的手部，我们采用了集成网络组合。首先，我们使用VitPose [^90]和AlphaPose [^17]来检测帧中的所有人体，并选取置信度大于0.65的边界框。随后，我们对这些边界框进行裁剪，并将其输入到由MediaPipe [^92]、OpenPose [^7]和ContactHands [^52]模型组成的集成手部检测流程中。为了确定手部的位置，我们取三个检测器$\mathbf{b}_i$所得到的边界框位置$d_i$的加权平均值，该加权值根据其对应的置信度$P(\mathbf{b}_i | d_i)$来确定：
$$ \hat{y}=\frac{\sum_{i}P(\mathbf{b}_{i}|d_{i})\mathbf{b}_{i}}{\sum_{i}P(\mathbf{b}_{i}|d_{i})} $$
其中$b_{i}$表示估算出的手部边界框。
除了边界框外，我们还利用估算出的2D关键点[^7] [^92]来拟合3D参数化手部模型$\mathcal{M}$ [^71]。更具体地说，我们优化形状$\beta$和姿态$\theta$参数，以最小化回归结果$\mathcal{L}_{proj}$与估算出的关键点$\mathbf{J}_{\mathcal{M}}$之间的投影误差$\hat{\mathbf{J}}_{\mathbf{s}}$：
$$ \mathcal{L}_{p r o j}=||\mathbf{J}_{\mathcal{M}}-\pi(\mathbf{\hat{J}_{s}},K)||_{1}, $$
其中，$\pi(\cdot)$ 表示弱透视投影变换，K 表示估计得到的相机内参矩阵。
考虑到人手的运动自由度，仅使用关节项来优化手部模型往往会导致不自然的姿态。为了解决优化过程中的歧义问题，我们参考了 [^76] 的做法，引入了生物力学损失函数来约束优化过程。具体而言，除了重投影误差外，我们还利用损失函数对骨骼长度和关节旋转角度进行约束，使其落在可行范围内，具体定义见 [^76]：
$$ \mathcal{L}_{B M C}=\mathcal{L}_{B L}+\mathcal{L}_{A} $$
其中，$\mathcal{L}_{BL}$ 和 $\mathcal{L}_{A}$ 分别表示骨骼长度损失项和关节角度损失项。关于生物力学约束的更多细节，请参阅 [^76]。
最后，由于生物力学先验主要作用于关节空间，我们遵循 [^2] 的思路，在 ARCTIC 数据集 [^16] 上训练了一个 PCA 模型作为三维先验，用于建模可行手部姿态的分布。我们将先验损失定义为从 PCA 空间 U 投影并重建出的三维网格 X 的重建误差，公式如下：
$$ \mathcal{L}_{p r i o r}=||\mathbf{X}-[(\mathbf{X}-\boldsymbol{\mu})\mathbf{U}^{T}]\mathbf{U}+\boldsymbol{\mu}||_{2}, $$
其中，$\mathbf{U} \in \mathbb{R}^{d \times N \cdot 3}$ 表示 d 个分量的特征向量基，$\mu$ 表示平均网格。图 2 展示了所提出的 WHIM 数据集的几个示例。
### 4. 方法
#### 4.1. 手部检测与定位
近年来，全卷积网络（FCN）在人体检测 [^86] 和物体检测 [^68] 领域展现出了卓越的效率。基于这一成果，我们采用 FCN 架构来实现精准且实时的手部定位。与物体检测框架类似，给定图像 $\mathbf{I} \in \mathbb{R}^{H \times W \times 3}$，我们的目标是检测出图像中 n 只手的边界框 $\mathbf{B} = \{\mathbf{b}_j \in \mathbb{R}^4 : 0 \leq j \leq n\}$ 以及它们的手侧标签 $\mathbf{y}_j$。我们采用了常用的单阶段骨干-颈部-头部结构，并在此基础上进行构建。
---
Figure 2. Example of the proposed WHIM in-the-wild dataset.
我们选用强大且高效的 DarkNet 骨干网络 [^67]。我们提取该骨干网络的最后三层特征图 $\{C3, C4, C5\}$，在颈部模块中生成多尺度特征金字塔。为了使模型能够通过不同特征图中的自上而下和自下而上的路径有效捕捉多尺度特征，我们采用了路径聚合网络（PANet）[^45]，它是特征金字塔网络 [^43] 的扩展，能够通过自下而上的路径增强来实现细粒度信息流。最后，我们使用三个检测头来预测不同锚点分辨率下的边界框 $b_j$ 和手侧标签 $y_j$。按照 [^15] 的做法，我们采用了无锚点设计来提高定位方法的灵活性，无需依赖预定义的锚框即可直接预测边界框坐标。所提出的检测网络概览如图 3 所示。与 [^12] 的研究结果一致，我们发现关节关键点监督显著提升了检测器的性能和鲁棒性。完整的训练目标可定义为：
$$ \mathcal{L}=\lambda_{0}\mathcal{L}_{B C E}+\lambda_{1}\mathcal{L}_{D F L}+\lambda_{2}\mathcal{L}_{C I o U}+\lambda_{3}\mathcal{L}_{k p t s} $$
其中 $\mathcal{L}_{BCE}$ 表示预测标签与真实标签之间的二元交叉熵损失，$\mathcal{L}_{DFL}$ 代表分布式聚焦损失 [^38]，用于衡量预测边界框分布与真实边界框分布之间的差异；$\mathcal{L}_{CIoU}$ 用于计算预测边界框与真实边界框之间的偏差 [^96]；$\mathcal{L}_{kpts}$ 表示对和 $\lambda_0$ 计算的L2损失，而 $\lambda_1$、$\lambda_2$、$\lambda_3$ 则是用于平衡各项损失的权重。
#### 4.2. 手部重建
给定包含人手的图像 $\mathbf{I}_h \in \mathbb{R}^{H \times W \times 3}$，该图像是围绕手部检测器的边界框进行裁剪得到的，所提出的3D手部重建方法会估计对应的手部姿态 $\theta \in \mathbb{R}^{48}$ 和形状 $\beta \in \mathbb{R}^{10}$ MANO参数 [^71]，同时确定相机参数 $\mathbf{K}_{cam} = \{\mathbf{t}_{cam}, \mathbf{s}_{cam}\}$，最终生成3D手部模型。
为了构建一个能够利用海量数据扩展的强大3D姿态估计网络，我们遵循 [^6]、[^40]、[^61] 的方法，
Figure 3. Detection overview: The proposed fully convolutional one-stage hand detection method receives an image and extracts multi-resolution feature maps that are then processed by the Path Aggregation Network (PANet). The corresponding features are then fed to three detection heads that predict the hand side, bounding box, and hand joints at different resolutions. We train the network with a multi-task loss for each anchor.
我们使用预训练的ViT编码器 [^85]、[^90] 作为骨干网络。图像 $\mathbf{I}_h$ 首先被分割成大小为M的块 $\mathbf{P} \in \mathbb{R}^{HW/M^2 \times (M^2 \times 3)}$，然后嵌入为高维特征向量 $\mathbf{T}_{img} \in \mathbb{R}^{HW/M^2 \times C}$。为了唯一编码它们的空间位置，会将位置嵌入 $\mathbf{P}_e$ 添加到图像特征向量 $\mathbf{T}_{img}$ 中 [^85]。除了图像特征向量外，我们还用三个不同的特征向量 $\mathbf{T}_{pose}$、$\mathbf{T}_{shape}$、$\mathbf{T}_{cam}$ 来显式建模手部姿态、形状和相机参数。随后，我们将拼接后的特征向量输入ViTTransformer编码器，得到一组更新后的特征向量 $\mathbf{T}'_{img}$、$\mathbf{T}'_{pose}$、$\mathbf{T}'_{shape}$、$\mathbf{T}'_{cam}$。通过多层感知机层，我们回归出MANO模型的姿态 $\theta^c$ 和形状 $\beta^c$ 参数的粗略估计值，这些值将作为细化网络的先验；同样，我们也会从相机特征向量中回归出相机的平移和缩放参数 $\mathbf{K}_{cam} = \{\mathbf{t}_{cam}, \mathbf{s}_{cam}\}$。
多尺度姿态细化模块。为了获得更好的图像对齐效果和更精确的手部姿态，我们引入了一个完全可微的细化模块，用于预测粗略手部估计结果的姿态和形状残差。为此，我们利用从ViT骨干网络提取的图像特征作为细化模块中的2D特征线索。具体来说，我们将图像特征向量 $\mathbf{T}'_{img}$ 重塑为低分辨率特征图 $\mathbf{F}_0 \in \mathbb{R}^{H \times \frac{W}{M} \times C}$，并利用估计出的 $\mathcal{M}_l$、$\mathbf{t}_{cam}$ 相机参数，将粗略手部估计结果 $\mathbf{s}_{cam}$ 投影到该特征图上。接着，通过双线性插值，我们从 $\mathbf{F}_0$ 中为每个投影顶点v采样得到特征向量 $\mathbf{f}_0^\mathbf{v}$：
$$ \mathbf{f}_{0}^{\mathbf{v}}=\pi(\mathbf{v},\mathbf{K}_{c a m}) $$
其中，$\pi(\cdot)$ 表示弱透视投影。
需要注意的是，我们将整个手部网格 $\mathcal{M}_{l}$ 投影到特征图上，而非仅投影手部关节，目的是实现更好的形状和姿态图像对齐。随后，将这些与图像对齐的顶点特征聚合起来，形成用于回归姿态和形状残差的全局特征向量：
$$ \begin{aligned}&\Delta\boldsymbol{\beta}=MLP_{\boldsymbol{\beta}}\big(\triangleq_{\mathbf{v}\in\mathcal{M}_{l}}\mathbf{f}_{0}^{\mathbf{v}}\big)\\&\Delta\boldsymbol{\theta}=MLP_{\boldsymbol{\theta}}\big(\triangleq_{\mathbf{v}\in\mathcal{M}_{l}}\mathbf{f}_{0}^{\mathbf{v}}\big)\\ \end{aligned} $$
---
Figure 4. Overview of the proposed 3D hand pose estimation method: Given an image $I_h$ represented as a series of feature tokens $T_{img}$ along with a set of learnable camera $T_{cam}$, pose $T_{pose}$ and shape $T_{shape}$ tokens, we initially predict a rough estimation of the MANO [^71] and camera $K_{cam}$ parameters using a ViT backbone (light blue). The updated image tokens are then reshaped and upsampled through a series of deconvolutional layers to form a set of multi-resolution feature maps $\{F_0, ..., F_0\}$. We then project the estimated 3D hand to the generated feature maps and sample image-aligned multi-scale features through a novel refinement module (purple). The sampled features are used to predict pose and shape residuals $\Delta\theta$, $\Delta\beta$ that refine the coarse hand estimation. Using this coarse-to-fine pose estimation strategy we facilitate image alignment and achieve better reconstruction performance.
其中，$\Box$ 表示聚合函数，例如均值、最大值、求和等。
由于初始特征图的维度很低，我们采用一组反卷积层将 $\mathbf{F}_0$ 上采样为多个更高分辨率的特征图 $\mathbf{F}_0, \mathbf{F}_1, ..., \mathbf{F}_n$，这些特征图将作为所提出的细化模块的多尺度特征。直观来说，低维特征图能够提供手部形状的全局和结构残差，而高分辨率特征则能提供更精细的手部姿态细节。
损失函数。所提出的模型在有监督的情况下进行训练，监督信号包括3D顶点 $\hat{V}_{3D}$、2D关节 $\hat{J}_{2D}$ 以及可用的MANO参数 $\hat{\theta}$、$\hat{\beta}$。此外，按照 [^33] 和 [^61] 的做法，我们利用判别器网络D来强制生成合理的手部姿态和形状，并对不规则的关节运动进行惩罚。完整的损失函数可定义为：
$$ \begin{aligned}\mathcal{L}&=\mathcal{L}_{3D}+\mathcal{L}_{2D}+\mathcal{L}_{m a n o}+\mathcal{L}_{a d v},\\\mathcal{L}_{3D}&=||\mathbf{V}_{3D}-\hat{\mathbf{V}}_{3D}||_{1},\\\mathcal{L}_{2D}&=||\pi(\mathbf{J}_{3D},\mathbf{K}_{c a m})-\hat{\mathbf{J}}_{2D}||_{1}\\\mathcal{L}_{m a n o}&=||\theta-\hat{\theta}||_{2}^{2}+||\beta-\hat{\beta}||_{2}^{2},\\\mathcal{L}_{a d v}&=||D(\theta,\beta)-1||_{2}.\end{aligned} $$
### 5. 实验
在本节中，我们首先利用已有的基准测试集评估所提出手部检测网络的性能；接着对手部姿态估计方法进行全面定性和定量分析；最后展示精确的手部定位对4D手部重建准确性的关键作用。
#### 5.1 手部检测与定位的评估
训练。我们使用经过精心整理的WHIM数据集来训练所提出手部检测网络，该数据集包含超过200万张包含多种手部形态和尺度的自然场景图像。为了进一步提升网络的泛化能力和鲁棒性，我们在训练过程中采用了多种数据增强手段。具体而言，我们引入了范围为 $[-60^{\circ}, 60^{\circ}]$ 的随机旋转以及范围为 $[-0.1, 0.1]$ 的平移操作，同时还会对图像进行随机掩码处理和裁剪。此外，在每个训练批次中，我们还采用了马赛克增强和MixUp增强技术，这些技术能显著增强网络对不同手部尺度的适应能力。
评估。为了对比我们的网络，我们选取了OpenPose [^7] 和MediaPipe [^92] 等社区广泛使用的经典基线模型 [^64] [^66]，以及ContactHands [^52] 和ViTDet [^39] 等较新的手部检测算法。所有方法均从三个维度进行评估：i) 以帧每秒（FPS）为单位的推理速度；ii) 以IoU=0.5时的平均精度（AP）以及不同IoU阈值下（0.5:0.05:0.95）的平均AP为指标的检测性能；iii) 以MB为单位的模型大小。理想的手部检测系统应当体积轻巧，以便适配移动设备；能够实时运行，避免影响3D姿态估计流程的运行；同时还能实现精准的检测结果。在表1中，我们在三个数据集上对所提方法和基线方法进行了评估：包括我们提出的WHIM数据集，以及基准数据集Coco-WholeBody [^32] 和OxfordHands [^48]。所有实验均在NVIDIA RTX 4090 GPU上完成。不难看出，所提——
---
Figure 5. Qualitative Evaluation of the proposed hand detection network on in-the-wild images. The proposed model demonstrates robustness across various lighting conditions, resolutions, hand scales, and even in the presence of motion blur.
所提出的中等尺寸检测器Proposed-M能够以超过130 FPS的速度运行，而其小型版本Proposed-S则能达到高达175 FPS的帧率。与以往的最先进模型相比，该检测器的mAP指标平均提升了26%。此外，与以往最先进的ContactHands模型[^52]相比，所提出的检测器运行速度更快（$45 \times$），且模型尺寸更小（$32 \times$），因此能够应用于移动端及高性能处理流程中，而不会带来显著的开销。值得注意的是，尽管三个数据集中的图像分辨率和手部尺寸存在差异，所提出的模型仍能稳定优于基线方法。这一点在COCO-WholeBody数据集[^32]上表现得尤为明显——该数据集是COCO数据集的扩展，包含全身图像，其中手部相对于整个图像尺寸较小。
消融实验。所提出的手部检测方法的效率与准确性主要得益于骨干网络的选择以及大规模训练数据集的应用。我们通过在OxfordHands [^48]和WHIM数据集上进行消融研究，进一步评估了各组件的贡献；研究中采用了不同的骨干网络和训练数据集。从表2可以看出，使用不同骨干网络训练检测网络时，虽然检测性能相近，但网络的推理速度却显著下降。所提出的大规模训练数据的重要性显而易见——
| | | Coco-Whole | Oxford-Hands | WHIM |
| Method | Size (Mb) $\downarrow$ | FPS $\uparrow$ | AP0.5 $\uparrow$ | mAP $\uparrow$ | AP0.5 $\uparrow$ | mAP $\uparrow$ | AP0.5 $\uparrow$ | mAP $\uparrow$ |
| MediaPipe [^92] | 25 | 25 | 15.43 | 3.72 | 8.72 | 1.80 | 53.09 | 12.01 |
| OpenPose [^7] | 141 | 29 | 37.05 | 9.06 | 20.74 | 4.41 | 76.8 | 34.25 |
| ContactHands [^53] | 819 | 3 | 50.29 | 16.67 | 70.02 | 36.41 | 93.42 | 49.44 |
| ViTDet [^39] | 1400 | 1 | 41.64 | 13.21 | 67.56 | 29.77 | 84.76 | 35.42 |
| Proposed-S | 7(×3.5 $\downarrow$) | 175(×6 $\uparrow$) | 46.96 | 18.56 | 75.21 | 38.16 | 91.80 | 46.50 |
| Proposed-M | 25 | 138 | 62.48 | 25.97 | 82.64 | 48.98 | 96.06 | 53.79 |
Table 1. Comparison with the state-of-the-art hand detection methods on COCO-Whole [^32], Oxford-Hands [^48] and the proposed WHIM dataset. For each method we report the average precision (AP) at IoU=0.5 along with the mean average precision (mAP). We also compare the performance of each method in terms of model size, measured in Mb, and speed, measured in frames per second (FPS).
| Method | Size (Mb) $\downarrow$ | FPS $\uparrow$ | OxfordHands | WHIM |
| AP0.5 $\uparrow$ | mAP $\uparrow$ | AP0.5 $\uparrow$ | mAP $\uparrow$ |
| Proposed-w. 0.25M | - | - | 49.15 | 26.69 | 75.75 | 38.48 |
| Proposed-w. 0.5M | - | - | 58.32 | 35.15 | 83.03 | 42.11 |
| Proposed-w. 1M | - | - | 69.21 | 43.04 | 88.37 | 47.92 |
| Proposed-w.OxfordHands | - | - | 68.15 | 40.34 | 70.14 | 35.29 |
| Proposed-w. ResNet50 | 118 | 34 | 74.13 | 47.34 | 95.43 | 51.85 |
| Proposed-w. HRNet | 132 | 30 | 84.82 | 49.83 | 97.23 | 54.12 |
| Proposed-w/o Augmentations | - | - | 70.76 | 42.17 | 91.13 | 49.93 |
| Proposed-w/o Landmark Loss | - | - | 72.57 | 45.96 | 92.43 | 51.44 |
| Proposed-M | 25 ( $\downarrow 5 \times$) | 138 ( $\uparrow 4 \times$) | 82.64 | 48.98 | 96.06 | 53.79 |
Table 2. Ablation study: Evaluation of individual components in the proposed detection pipeline on OxfordHands and WHIM datasets. We use – to denote identical network architecture and performance.
野外场景下的WHIM数据集也在表2中得到了验证：当模型使用较少的数据进行训练时，例如Proposed w. 0.25M、Proposed w. 0.5M、Proposed w. 1M，其性能会出现显著下降。值得注意的一点是，WHIM数据集具有很好的通用性：与在Oxford-Hands数据集上训练相比，所提出的模型在WHIM数据集上表现更佳。最后，我们评估了所提出的数据增强策略以及关键点回归损失函数的贡献。数据增强策略对跨数据集泛化能力有显著促进作用，使mAP提升了14%。同样，引入关键点回归损失后，检测器的精确度得到提高，从而能生成更稳健的检测结果。
#### 5.2. 三维手部姿态估计评估
训练过程。遵循[^6]、[^40]、[^61]的方法，我们使用多种数据集组合来训练所提出的手部回归器，以增强其对不同姿态、光照条件和遮挡情况的鲁棒性。具体而言，我们使用了包含2D和3D标注的数据集，包括FreiHAND [^101]、HO3D [^24]、MTC [^89]、RHD [^100]、InterHand2.6M [^50]、H2O3D [^24]、DEX YCB [^8]、COCO WholeBody [^32]、Halpe [^17]、MPII NZSL [^75]、BEDLAM [^4]、ARCTIC [^16]、Re:InterHand [^51]以及Hot3D [^3]。总共我们使用了420万张图像，这一数量比以往最先进的方法多出55%。
---
| Method | PA-MPJPE $\downarrow$ | PA-MPVPE $\downarrow$ | F@5 $\uparrow$ | F@15 $\uparrow$ |
| I2L-MeshNet [^49] | 7.4 | 7.6 | 0.681 | 0.973 |
| Pose2Mesh [^11] | 7.7 | 7.8 | 0.674 | 0.969 |
| I2UV-HandNet [^9] | 6.7 | 6.9 | 0.707 | 0.977 |
| METRO [^41] | 6.5 | 6.3 | 0.731 | 0.984 |
| Tang et al. [^82] | 6.7 | 6.7 | 0.724 | 0.981 |
| Mesh Graphormer [^42] | 5.9 | 6.0 | 0.764 | 0.986 |
| MobRecon [^10] | 5.7 | 5.8 | 0.784 | 0.986 |
| AMVUR [^31] | 6.2 | 6.1 | 0.767 | 0.987 |
| HaMeR [^61] | 6.0 | 5.7 | 0.785 | 0.990 |
| Proposed | 5.5 | 5.1 | 0.825 | 0.993 |
Table 3. Comparison with the state-of-the-art on the FreiHAND dataset [^101]. We use the standard protocol and report metrics for evaluation of 3D joint and 3D mesh accuracy. PA-MPVPE and PA-MPJPE numbers are in mm.
| Method | AUC $_J$ $\uparrow$ | PA-MPIPE $\downarrow$ | AUC $_V$ $\uparrow$ | PA-MPVPE $\downarrow$ | F@5 $\uparrow$ | F@15 $\uparrow$ |
| Liu et al. [^46] | 0.803 | 9.9 | 0.810 | 9.5 | 0.528 | 0.956 |
| HandOccNet [^60] | 0.819 | 9.1 | 0.819 | 8.8 | 0.564 | 0.963 |
| I2UV-HandNet [^9] | 0.804 | 9.9 | 0.799 | 10.1 | 0.500 | 0.943 |
| Hampali et al. [^24] | 0.788 | 10.7 | 0.790 | 10.6 | 0.506 | 0.942 |
| Hasson et al. [^27] | 0.780 | 11.0 | 0.777 | 11.2 | 0.464 | 0.939 |
| ArtiBoost [^91] | 0.773 | 11.4 | 0.782 | 10.9 | 0.488 | 0.944 |
| Pose2Mesh [^11] | 0.754 | 12.5 | 0.749 | 12.7 | 0.441 | 0.909 |
| I2L-MeshNet [^49] | 0.775 | 11.2 | 0.722 | 13.9 | 0.409 | 0.932 |
| METRO [^41] | 0.792 | 10.4 | 0.779 | 11.1 | 0.484 | 0.946 |
| MobRecon [^10] | - | 9.2 | - | 9.4 | 0.538 | 0.957 |
| Keypoint Trans [^25] | 0.786 | 10.8 | - | - | - | - |
| AMVUR [^31] | 0.835 | 8.3 | 0.836 | 8.2 | 0.608 | 0.965 |
| HaMeR | 0.846 | 7.7 | 0.841 | 7.9 | 0.635 | 0.980 |
| Proposed | 0.851 | 7.5 | 0.846 | 7.7 | 0.646 | 0.983 |
Table 4. Comparison with the state-of-the-art on the HO3D dataset [^24]. We use the HO3Dv2 protocol and report metrics that evaluate accuracy of the estimated 3D joints and 3D mesh PA-MPVPE and PA-MPJPE numbers are in mm.
评估。为了将我们提出的方法与包括METRO [^41]、Mesh Graphormer [^42]、AMVUR [^31]、MobRecon [^10]、HaMeR [^61]以及SimpleHand [^97]在内的最先进方法进行比较，我们在表3和表4中报告了在常用基准数据集FreiHAND [^101]和HO3Dv2 [^24]上的重建结果。按照通用协议 [^101]，我们以Procrustes对齐的平均关节误差和顶点误差（PA-MPJPE、PA-MPVPE）来衡量重建性能，同时统计误差小于5毫米和15毫米的姿态所占比例（F@5、F@15）。此外，我们还报告了HO3D数据集中3D关节和顶点的曲线下面积（AUC $_j$、AUC $_v$）。我们提出的方法在所有指标上均取得了最先进的性能，在两个基准数据集上的表现都优于以往的方法，这一结果也可从图6中的定性分析得到进一步验证。借助细化模块的图像对齐特征，WiLoR即使在复杂的关节运动情况下也能实现高保真的重建。
消融实验。为了进一步探究我们提出的方法中各个组件的贡献，我们开展了消融研究。
| Method | PA-MPJPE $\downarrow$ | PA-MPVPE $\downarrow$ | F@5 $\uparrow$ | F@15 $\uparrow$ |
| Proposed w. FastViT | 6.5 | 6.3 | 0.741 | 0.967 |
| Proposed w/o ViTPose | 5.9 | 5.7 | 0.795 | 0.989 |
| Proposed w. Single-Scale | 6.0 | 5.9 | 0.793 | 0.991 |
| Proposed w/o Refinement | 6.1 | 5.8 | 0.795 | 0.991 |
| Proposed w. FreiHAND [^101] | 6.1 | 5.8 | 0.793 | 0.990 |
| Proposed w. Datasets [^61] | 5.9 | 5.7 | 0.805 | 0.992 |
| Proposed Full | 5.5 | 5.1 | 0.825 | 0.993 |
Table 5. Ablation study on the FreiHAND dataset [^101]. We report ablations on the backbone and the training data used along with the novel refinement module.
在表5中，我们评估了骨干网络架构、所使用的训练数据集以及细化模块的贡献。可以看出，尽管FastViT [^84]架构具有更高的运行效率，但将其替换原有的ViT骨干网络后（即“ Proposed w. FastViT”组），模型性能会出现显著下降。同样，不使用ViTPose [^90]的预训练权重从头训练骨干网络（“ Proposed w/o ViTPose”组）也会导致性能下滑。为了评估所提出的细化模块的效果，我们训练了一个模型，该模型直接从ViT输出的特征向量回归MANO参数和相机参数，而不使用任何细化模块（“ Proposed w/o Refinement”组）。此外，我们还训练了一个采用单尺度细化模块的模型，该模块仅从单个特征图中采样特征（“ Proposed w. Single-Scale”组）。这两种架构选择都会使所提出模型的重建性能变差，这凸显了所提出的多尺度细化模块的重要性。最后，我们通过仅使用FreiHAND数据集 [^101]训练模型来检验大规模训练集的影响（即“ Proposed w. FreiHAND [^101]”组），这一做法与 [^10]、[^41]、[^42]、[^97]的做法类似；我们还训练了一个使用 [^61] 所用数据集的模型（“ Proposed w. Datasets [^61]”组）。
#### 5.3. 动态重建评估
对于3D姿态估计方法而言，一个关键挑战是在未经动态场景训练的情况下实现稳定且鲁棒的4D重建 [^74]。传统上，基于单张图像的3D姿态估计方法存在时间连贯性差、帧间抖动明显的问题，这给它们在真实视频重建中的泛化能力带来了巨大挑战。为了有效评估所提出方法的时间连贯性，我们对4D序列逐帧进行重建，并测量帧间的抖动情况。具体而言，我们计算连续帧之间3D顶点（MPFVE）和关节（MPFJE）的平均欧几里得距离。此外，与 [^74] 中的做法类似，我们还测量3D手部关节运动的抖动程度（Jitter），以及衡量手腕位移的全局根平移误差（RTE）。
---
Figure 6. Qualitative Evaluation of proposed and the baseline methods on FreiHAND dataset [^101]. WiLoR demonstrates robustness across challenging poses with heavy occlusions, while maintaining precise image alignment.
| Method | MPFVE $\times 100$ | $\downarrow$ MPFJE $\times 100$ | $\downarrow$ Jitter | $\downarrow$ RTE | $\downarrow$ |
| MeshGraphormer [^42] | 21.86 | 4.99 | 41.16 | 7.92 | |
| MobRecon [^10] | 22.18 | 6.09 | 40.25 | 8.03 | |
| SimpleHand [^97] | 19.72 | 5.12 | 38.53 | 6.04 | |
| HaMeR [^61] | 10.60 | 1.768 | 20.43 | 2.92 | |
| Proposed | 4.43 | 0.762 | 5.92 | 0.07 | |
Table 6. Reconstruction of dynamic 3D Hands. We evaluate the temporal coherence and the jittering of the reconstruction for the proposed and the baseline methods on the HO3D dataset.
在表6中，我们报告了在HO3D [^24] 数据集上表现最佳的方法的重建结果。WiLoR凭借其检测结果的稳健稳定性，无需依赖任何时序模块即可在时序连贯性方面超越基线方法。关于定性视频结果，请参阅补充材料。
### 6. 结论
在这项工作中，我们提出了WiLoR——首个全栈式手部检测与3D姿态估计框架。我们利用大规模的自然场景数据集训练出一个轻量但精度极高的手部检测器模型，该模型能够在不同遮挡和光照条件下以超过130 FPS的速度稳健地检测手部。此外，我们基于所提出的新型优化模块，构建了一个高保真度的3D手部姿态估计模型，该模型克服了以往方法的局限性，并缓解了以往方法存在的对齐问题。通过一系列实验，我们证明WiLoR在两个基准数据集上的表现均优于以往的最先进方法，且在具有挑战性的案例中也能展现出稳健的性能。WiLoR为多手检测、定位及3D重建提供了全面的解决方案。
致谢：S. Zafeiriou的研究得到了Turing AI Fellowship（EP/Z534699/1）和GNOMON项目（EP/X011364）的支持。R.A. Potamias的研究得到了EPSRC项目GNOMON（EP/X011364）的支持。
---
## 参考文献
[^1]: Seungryul Baek, Kwang In Kim, and Tae-Kyun Kim. Pushing the envelope for rgb-based dense 3d hand pose estimation via neural rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1067–1076, 2019. 3
[^2]: Vasileios Baltatzis, Rolandos Alexandros Potamias, Evangelos Ververas, Guanxiong Sun, Jiankang Deng, and Stefanos Zafeiriou. Neural sign actors: A diffusion model for 3d sign language production from text. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1985–1995, 2024. 1, 3
[^3]: Prithviraj Banerjee, Sindi Shkodrani, Pierre Moulon, Shreyas Hampali, Fan Zhang, Jade Fountain, Edward Miller, Selen Basol, Richard Newcombe, Robert Wang, et al. Introducing hot3d: An egocentric dataset for 3d hand and object tracking. arXiv preprint arXiv:2406.09598, 2024. 6
[^4]: Michael J Black, Priyanka Patel, Joachim Tesch, and Jinlong Yang. Bedlam: A synthetic dataset of bodies exhibiting detailed lifelike animated motion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8726–8737, 2023. 6
[^5]: Adnane Boukhayma, Rodrigo de Bem, and Philip HS Torr. 3d hand shape and pose from images in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10843–10852, 2019. 2, 3
[^6]: Zhongang Cai, Wanqi Yin, Ailing Zeng, Chen Wei, Qingping Sun, Wang Yanjun, Hui En Pang, Haiyi Mei, Mingyuan Zhang, Lei Zhang, et al. Smpler-x: Scaling up expressive human pose and shape estimation. Advances in Neural Information Processing Systems, 36, 2024. 3, 4, 6
[^7]: Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. Realtime multi-person 2d pose estimation using part affinity fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7291–7299, 2017. 2, 3, 5, 6
[^8]: Yu-Wei Chao, Wei Yang, Yu Xiang, Pavlo Molchanov, Ankur Handa, Jonathan Tremblay, Yashraj S Narang, Karl Van Wyk, Umar Iqbal, Stan Birchfield, et al. Dexycb: A benchmark for capturing hand grasping of objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9044–9053, 2021. 6
[^9]: Ping Chen, Yujin Chen, Dong Yang, Fangyin Wu, Qin Li, Qingpei Xia, and Yong Tan. I2uv-handnet: Image-to-uv prediction network for accurate and high-fidelity 3d hand mesh modeling. In Proceedings of the IEEE/CVF international conference on computer vision, pages 12929–12938, 2021. 7
[^10]: Xingyu Chen, Yufeng Liu, Yajiao Dong, Xiong Zhang, Chongyang Ma, Yanmin Xiong, Yuan Zhang, and Xiaoyan Guo. Mobrecon: Mobile-friendly hand mesh reconstruction from monocular image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20544–20554, 2022. 2, 3, 7, 8
[^11]: Hongsuk Choi, Gyeongsik Moon, and Kyoung Mu Lee. Pose2mesh: Graph convolutional network for 3d human
基于2D人体姿态的姿态与网格恢复。载于《计算机视觉–ECCV 2020：第16届欧洲会议论文集，英国格拉斯哥，2020年8月23–28日》，第VII部分，第769–787页。Springer出版社，2020年。1, 2, 7
[^12]: Jiankang Deng, Jia Guo, Evangelos Ververas, Irene Kotsia, and Stefanos Zafeiriou. Retinaface: Single-shot multilevel face localisation in the wild. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5203–5212, 2020. 2, 4
[^13]: Xiaoming Deng, Yinda Zhang, Shuo Yang, Ping Tan, Liang Chang, Ye Yuan, and Hongan Wang. Joint hand detection and rotation estimation using cnn. IEEE transactions on image processing, 27(4):1888–1900, 2017. 2
[^14]: Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In North American Chapter of the Association for Computational Linguistics, 2019. 2
[^15]: Kaiwen Duan, Song Bai, Lingxi Xie, Honggang Qi, Qingming Huang, and Qi Tian. Centernet: Keypoint triplets for object detection. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6569–6578, 2019. 4
[^16]: Zicong Fan, Omid Taheri, Dimitrios Tzionas, Muhammed Kocabas, Manuel Kaufmann, Michael J. Black, and Otmar Hilliges. ARCTIC: A dataset for dexterous bimanual hand-object manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023. 3, 6
[^17]: Hao-Shu Fang, Jiefeng Li, Hongyang Tang, Chao Xu, Haoyi Zhu, Yuliang Xiu, Yong-Lu Li, and Cewu Lu. Alphapose: Whole-body regional multi-person pose estimation and tracking in real-time. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022. 2, 3, 6
[^18]: Liuhao Ge, Hui Liang, Junsong Yuan, and Daniel Thalmann. Robust 3d hand pose estimation in single depth images: from single-view cnn to multi-view cnns. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3593–3601, 2016. 3
[^19]: Georgia Gkioxari, Bharath Hariharan, Ross Girshick, and Jitendra Malik. Using k-poselets for detecting people and localizing their keypoints. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3582–3589, 2014. 2
[^20]: Shubham Goel, Georgios Pavlakos, Jathushan Rajasegaran, Angjoo Kanazawa $^{*}$, and Jitendra Malik $^{*}$. Humans in 4D: Reconstructing and tracking humans with transformers. In International Conference on Computer Vision (ICCV), 2023. 3
[^21]: Shubham Goel, Georgios Pavlakos, Jathushan Rajasegaran, Angjoo Kanazawa $^{*}$, and Jitendra Malik $^{*}$. Humans in 4D: Reconstructing and tracking humans with transformers. In International Conference on Computer Vision (ICCV), 2023. 2
## [^22]: Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d
---
3000小时自我中心视频的全球探索。载于《IEEE/CVF计算机视觉与模式识别会议论文集》，第18995–19012页，2022年。
[^23]: Jianzhu Guo, Xiangyu Zhu, Yang Yang, Fan Yang, Zhen Lei, and Stan Z Li. Towards fast, accurate and stable 3d dense face alignment. In European Conference on Computer Vision, pages 152–168. Springer, 2020. 2
[^24]: Shreyas Hampali, Mahdi Rad, Markus Oberweger, and Vincent Lepetit. Honnotate: A method for 3d annotation of hand and object poses. In CVPR, 2020. 1, 2, 6, 7, 8
[^25]: Shreyas Hampali, Sayan Deb Sarkar, Mahdi Rad, and Vincent Lepetit. Keypoint transformer: Solving joint identification in challenging hands and object interactions for accurate 3d pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11090–11100, 2022. 7
[^26]: Shreyas Hampali, Sayan Deb Sarkar, Mahdi Rad, and Vincent Lepetit. Keypoint transformer: Solving joint identification in challenging hands and object interactions for accurate 3d pose estimation. In CVPR, 2022. 1
[^27]: Yana Hasson, Gul Varol, Dimitrios Tzionas, Igor Kalevatykh, Michael J Black, Ivan Laptev, and Cordelia Schmid. Learning joint reconstruction of hands and manipulated objects. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11807–11816, 2019. 7
[^28]: Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 2961–2969, 2017. 2
[^29]: T Hoang Ngan Le, Yutong Zheng, Chenchen Zhu, Khoa Luu, and Marios Savvides. Multiple scale faster-rcnn approach to driver's cell-phone usage and hands on steering wheel detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition workshops, pages 46–53, 2016. 2
[^30]: Hezhen Hu, Weichao Zhao, Wengang Zhou, Yuechen Wang, and Houqiang Li. Signbert: pre-training of hand-model-aware representation for sign language recognition. In Proceedings of the IEEE/CVF international conference on computer vision, pages 11087–11096, 2021. 1
[^31]: Zheheng Jiang, Hossein Rahmani, Sue Black, and Bryan M Williams. A probabilistic attention model with occlusion-aware texture regression for 3d hand reconstruction from a single rgb image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 758–767, 2023. 7
[^32]: Sheng Jin, Lumin Xu, Jin Xu, Can Wang, Wentao Liu, Chen Qian, Wanli Ouyang, and Ping Luo. Whole-body human pose estimation in the wild. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part IX 16, pages 196–214. Springer, 2020. 5, 6
[^33]: Angjoo Kanazawa, Michael J Black, David W Jacobs, and Jitendra Malik. End-to-end recovery of human shape and pose. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7122–7131, 2018. 5
[^34]: Jeonghwan Kim, Mi-Gyeong Gwon, Hyunwoo Park, Hyukmin Kwon, Gi-Mun Um, and Wonjun Kim. Sampling is Matter: Point-guided 3d human mesh reconstruction. In CVPR, 2023. 2
[^35]: Dominik Kulon, Haoyang Wang, Riza Alp Güler, Michael M. Bronstein, and Stefanos Zafeiriou. Single image 3d hand reconstruction with mesh convolutions. In Proceedings of the British Machine Vision Conference (BMVC), 2019. 3
[^36]: Dominik Kulon, Riza Alp Guler, Iasonas Kokkinos, Michael M Bronstein, and Stefanos Zafeiriou. Weakly-supervised mesh-convolutional hand reconstruction in the wild. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4990–5000, 2020. 2, 3
[^37]: Kwonjoon Lee, Huiwen Chang, Lu Jiang, Han Zhang, Zhuowen Tu, and Ce Liu. Vitgan: Training gans with vision transformers. In ICLR, 2022. 2
[^38]: Xiang Li, Wenhai Wang, Lijun Wu, Shuo Chen, Xiaolin Hu, Jun Li, Jinhui Tang, and Jian Yang. Generalized focal loss: Learning qualified and distributed bounding boxes for dense object detection. Advances in Neural Information Processing Systems, 33:21002–21012, 2020. 4
[^39]: Yanghao Li, Hanzi Mao, Ross Girshick, and Kaiming He. Exploring plain vision transformer backbones for object detection. In European conference on computer vision, pages 280–296. Springer, 2022. 2, 5, 6
[^40]: Jing Lin, Ailing Zeng, Haoqian Wang, Lei Zhang, and Yu Li. One-stage 3d whole-body mesh recovery with component aware transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21159–21168, 2023. 2, 3, 4, 6
[^41]: Kevin Lin, Lijuan Wang, and Zicheng Liu. End-to-end human pose and mesh reconstruction with transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1954–1963, 2021. 2, 7
[^42]: Kevin Lin, Lijuan Wang, and Zicheng Liu. Mesh graphormer. In Proceedings of the IEEE/CVF international conference on computer vision, pages 12939–12948, 2021. 1, 2, 7, 8
[^43]: Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2117–2125, 2017. 4
[^44]: Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pages 2980–2988, 2017. 2
[^45]: Shu Liu, Lu Qi, Haifang Qin, Jianping Shi, and Jiaya Jia. Path aggregation network for instance segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8759–8768, 2018. 4
## [^46]: Shaowei Liu, Hanwen Jiang, Jiarui Xu, Sifei Liu, and Xiaolong Wang. Semi-supervised 3d hand-object poses estimation with interactions in time. In Proceedings of the
---
IEEE/CVF计算机视觉与模式识别会议，2021年。7
[^47]: Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, and Alexander C Berg. Ssd: Single shot multibox detector. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part I 14, pages 21–37. Springer, 2016. 2
[^48]: Arpit Mittal, Andrew Zisserman, and Philip HS Torr. Hand detection using multiple proposals. In Bmvc, page 5. Cite-seer, 2011. 2, 5, 6
[^49]: Gyeongsik Moon and Kyoung Mu Lee. I2l-meshnet: Image-to-lixel prediction network for accurate 3d human pose and mesh estimation from a single rgb image. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part VII 16, pages 752–768. Springer, 2020. 1, 2, 7
[^50]: Gyeongsik Moon, Shoou-I Yu, He Wen, Takaaki Shiratori, and Kyoung Mu Lee. Interhand2.6m: A dataset and baseline for 3d interacting hand pose estimation from a single rgb image. In European Conference on Computer Vision (ECCV), 2020. 6
[^51]: Gyeongsik Moon, Shunsuke Saito, Weipeng Xu, Rohan Joshi, Julia Buffaloini, Harley Bellan, Nicholas Rosen, Jesse Richardson, Mize Mallorie, Philippe Bree, Tomas Simon, Bo Peng, Shubham Garg, Kevyn McPhail, and Takaaki Shiratori. A dataset of relighted 3D interacting hands. In NeurIPS Track on Datasets and Benchmarks, 2023. 6
[^52]: Supreeth Narasimhaswamy, Zhengwei Wei, Yang Wang, Justin Zhang, and Minh Hoai. Contextual attention for hand detection in the wild. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9567–9576, 2019. 2, 3, 5, 6
[^53]: Supreeth Narasimhaswamy, Trung Nguyen, and Minh Hoai Nguyen. Detecting hands and recognizing physical contact in the wild. Advances in neural information processing systems, 33:7841–7851, 2020. 2, 6
[^54]: Supreeth Narasimhaswamy, Thanh Nguyen, Mingzhen Huang, and Minh Hoai. Whose hands are these? hand detection and hand-body association in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4889–4899, 2022. 2
[^55]: Alejandro Newell, Kaiyu Yang, and Jia Deng. Stacked hourglass networks for human pose estimation. In Computer Vision – ECCV 2016, pages 483–499, Cham, 2016. Springer International Publishing. 2
[^56]: Yeounguk Oh, JoonKyu Park, Jaeha Kim, Gyeongsik Moon, and Kyoung Mu Lee. Recovering 3d hand mesh sequence from a single blurry image: A new dataset and temporal unfolding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023. 3
[^57]: Iason Oikonomidis, Nikolaos Kyriazis, Antonis A Argyros, et al. Efficient model-based 3d tracking of hand articulation using kinect. In BmVC, page 3, 2011. 3
[^58]: George Papandreou, Tyler Zhu, Nori Kanazawa, Alexander Toshev, Jonathan Tompson, Chris Bregler, and Kevin
Murphy。面向野外环境的精准多人姿态估计。载于《IEEE/CVF计算机视觉与模式识别会议论文集》，第4903–4911页，2017年。2
[^59]: JoonKyu Park, Yeonguk Oh, Gyeongsik Moon, Hongsuk Choi, and Kyoung Mu Lee. Handoccnet: Occlusion-robust 3d hand mesh estimation network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022. 3
[^60]: JoonKyu Park, Yeonguk Oh, Gyeongsik Moon, Hongsuk Choi, and Kyoung Mu Lee. Handoccnet: Occlusionrobust 3d hand mesh estimation network. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1496–1505, 2022. 7
[^61]: Georgios Pavlakos, Dandan Shan, Ilija Radosavovic, Angjoo Kanazawa, David Fouhey, and Jitendra Malik. Reconstructing hands in 3D with transformers. In CVPR, 2024. 1, 2, 3, 4, 5, 6, 7, 8
[^62]: Pramod Kumar Pisharady, Prahlad Vadakkepat, and Ai Poh Loh. Attention based detection and recognition of hand postures against complex backgrounds. International Journal of Computer Vision, 101:403–419, 2013. 2
[^63]: Leonid Pishchulin, Arjun Jain, Mykhaylo Andriluka, Thorsten Thormählen, and Bernt Schiele. Articulated people detection and pose estimation: Reshaping the future. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3178–3185. IEEE, 2012. 2
[^64]: Rolandos Alexandros Potamias, Stylianos Ploumpis, Stylianos Moschoglou, Vasileios Triantafyllou, and Stefanos Zafeiriou. Handy: Towards a high fidelity 3d hand shape and appearance model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4670–4680, 2023. 1, 5
[^65]: Jing Qi, Li Ma, Zhenchao Cui, and Yushu Yu. Computer vision-based hand gesture recognition for human-robot interaction: a review. Complex & Intelligent Systems, 10(1): 1581–1606, 2024. 1
[^66]: Neng Qian, Jiayi Wang, Franziska Mueller, Florian Bernard, Vladislav Golyanik, and Christian Theobalt. HTML: A parametric hand texture model for 3d hand reconstruction and personalization. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XI 16, pages 54–71. Springer, 2020. 5
[^67]: Joseph Redmon and Ali Farhadi. Yolov3: An incremental improvement. arXiv preprint arXiv:1804.02767, 2018. 2, 4
[^68]: Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 779–788, 2016. 3
[^69]: James M Rehg and Takeo Kanade. Digiteyes: Vision-based hand tracking for human-computer interaction. In Proceedings of 1994 IEEE workshop on motion of non-rigid and articulated objects, pages 16–22. IEEE, 1994. 2
## [^70]: Robin Rombach, Andreas Blattmann, Dominik Lorenz
---
Patrick Esser和Björn Ommer。基于潜变量扩散模型的高分辨率图像合成，2021年。2
[^71]: Javier Romero, Dimitrios Tzionas, and Michael J Black. Embodied hands: Modeling and capturing hands and bodies together. arXiv preprint arXiv:2201.02610, 2022. 2, 3, 4, 5
[^72]: Kankana Roy, Aparna Mohanty, and Rajiv R Sahay. Deep learning based hand detection in cluttered environment using skin segmentation. In Proceedings of the IEEE international conference on computer vision workshops, pages 640–649, 2017. 2
[^73]: Dandan Shan, Jiaqi Geng, Michelle Shu, and David F Fouhey. Understanding human hands in contact at internet scale. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9869–9878, 2020. 2
[^74]: Soyong Shin, Juyong Kim, Eni Halilaj, and Michael J Black. Wham: Reconstructing world-grounded humans with accurate 3d motion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2070–2080, 2024. 7
[^75]: Tomas Simon, Hanbyul Joo, Iain Matthews, and Yaser Sheikh. Hand keypoint detection in single images using multiview bootstrapping. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1145–1153, 2017. 2, 6
[^76]: Adrian Spurr, Umar Iqbal, Pavlo Molchanov, Otmar Hilliges, and Jan Kautz. Weakly supervised 3d hand pose estimation via biomechanical constraints. In European conference on computer vision, pages 211–228. Springer, 2020. 3
[^77]: Srinath Sridhar, Antti Oulasvirta, and Christian Theobalt. Interactive markerless articulated hand motion tracking using rgb and depth data. In Proceedings of the IEEE international conference on computer vision, pages 2456–2463, 2013. 2
[^78]: Björn Stenger, Arasanathan Thayananthan, Philip HS Torr, and Roberto Cipolla. Model-based hand tracking using a hierarchical bayesian filter. IEEE transactions on pattern analysis and machine intelligence, 28(9):1372–1384, 2006. 2
[^79]: Ke Sun, Bin Xiao, Dong Liu, and Jingdong Wang. Deep high-resolution representation learning for human pose estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5693–5703, 2019. 2
[^80]: Min Sun and Silvio Savarese. Articulated part-based model for joint object detection and pose estimation. In 2011 International Conference on Computer Vision, pages 723–730. IEEE, 2011. 2
[^81]: Andrea Tagliasacchi, Matthias Schröder, Anastasia Tkach, Sofien Bouaziz, Mario Botsch, and Mark Pauly. Robust articulated-icp for real-time hand tracking. In Computer graphics forum, pages 101–114. Wiley Online Library, 2015. 3
[^82]: Xiao Tang, Tianyu Wang, and Chi-Wing Fu. Towards accurate alignment in real-time 3d hand-mesh reconstruction.
载于《IEEE/CVF国际计算机视觉会议论文集》，第11698–11707页，2021年。7
[^83]: Michail Tarasiou, Rolandos Alexandros Potamias, Eimear O'Sullivan, Stylianos Ploumpis, and Stefanos Zafeiriou. Locally adaptive neural 3d morphable models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1867–1876, 2024. 2
[^84]: Pavan Kumar Anasosalu Vasu, James Gabriel, Jeff Zhu, Oncel Tuzel, and Anurag Ranjan. Fastvit: A fast hybrid vision transformer using structural reparameterization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5785–5795, 2023. 7
[^85]: Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017. 2,4
[^86]: Jingdong Wang, Ke Sun, Tianheng Cheng, Borui Jiang, Chaorui Deng, Yang Zhao, Dong Liu, Yadong Mu, Mingkui Tan, Xinggang Wang, et al. Deep high-resolution representation learning for visual recognition. IEEE transactions on pattern analysis and machine intelligence, 43(10):3349–3364, 2020. 3
[^87]: Robert Y Wang and Jovan Popović. Real-time hand-tracking with a color glove. ACM transactions on graphics (TOG), 28(3):1–8, 2009. 2
[^88]: Ying Wu, Qiong Liu, and Thomas S Huang. An adaptive self-organizing color segmentation algorithm with application to robust real-time human hand localization. In Proceedings of Asian Conference on Computer Vision, pages 1106–1111, 2000. 2
[^89]: Donglai Xiang, Hanbyul Joo, and Yaser Sheikh. Monocular total capture: Posing face, body, and hands in the wild. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10965–10974, 2019. 6
[^90]: Yufei Xu, Jing Zhang, Qiming Zhang, and Dacheng Tao. ViTPose: Simple vision transformer baselines for human pose estimation. In Advances in Neural Information Processing Systems, 2022. 2, 3, 4, 7
[^91]: Lixin Yang, Kailin Li, Xinyu Zhan, Jun Lv, Wenqiang Xu, Jiefeng Li, and Cewu Lu. Artiboost: Boosting articulated 3d hand-object pose estimation via online exploration and synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2750–2760, 2022. 7
[^92]: Fan Zhang, Valentin Bazarevsky, Andrey Vakunov, Andrei Tkachenka, George Sung, Chuo-Ling Chang, and Matthias Grundmann. Mediapipe hands: On-device real-time hand tracking. arXiv preprint arXiv:2006.10214, 2020. 2, 3, 5, 6
[^93]: Hongwen Zhang, Yating Tian, Yuxiang Zhang, Mengcheng Li, Liang An, Zhenan Sun, and Yebin Liu. Pymafx: Towards well-aligned full-body model regression from monocular images. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(10):12287–12303, 2023. 2
## [^94]: Jinglei Zhang, Jiankang Deng, Chao Ma, and Rolandos Alexandros Potamias. Hawor: World-space hand motion reconstruction from egocentric videos. arXiv preprint arXiv:2501.02973, 2025. 2
---
[^95]: Xiong Zhang, Qiang Li, Hong Mo, Wenbo Zhang, and Wen Zheng. End-to-end hand mesh recovery from a monocular rgb image. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2354–2364, 2019. 3
[^96]: Zhaohui Zheng, Ping Wang, Dongwei Ren, Wei Liu, Rongguang Ye, Qinghua Hu, and Wangmeng Zuo. Enhancing geometric factors in model learning and inference for object detection and instance segmentation. IEEE transactions on cybernetics, 52(8):8574–8586, 2021. 4
[^97]: Zhishan Zhou, Shihao Zhou, Zhi Lv, Minqiang Zou, Yao Tang, and Jiajun Liang. A simple baseline for efficient hand mesh reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1367–1376, 2024. 2, 3, 7, 8
[^98]: Xiaojin Zhu, Jie Yang, and Alex Waibel. Segmenting hands of arbitrary color. In Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition, pages 446–453. IEEE, 2000. 2
[^99]: Yanjia Zhu, Hongxiang Cai, Shuhan Zhang, Chenhao Wang, and Yichao Xiong. Tinaface: Strong but simple baseline for face detection. arXiv preprint arXiv:2011.13183, 2020. 2
[^100]: Christian Zimmermann and Thomas Brox. Learning to estimate 3d hand pose from single rgb images. In Proceedings of the IEEE international conference on computer vision, pages 4903–4911, 2017. 6
[^101]: Christian Zimmermann, Duygu Ceylan, Jimei Yang, Bryan Russell, Max Argus, and Thomas Brox. Freihand: A dataset for markerless capture of hand pose and shape from single rgb images. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 813–822, 2019. 2, 6, 7, 8
[^102]: Ronglai Zuo, Rolandos Alexandros Potamias, Evangelos Ververas, Jiankang Deng, and Stefanos Zafeiriou. Signs as tokens: An autoregressive multilingual sign language generator. arXiv preprint arXiv:2411.17799, 2024. 1