# 利用Transformer重建三维手部模型
Georgios Pavlakos$^{1}$、单丹丹$^{2}$、Ilija Radosavovic$^{3}$、Angjoo Kanazawa$^{3}$、David Fouhey$^{4}$、Jitendra Malik$^{3}$$^{1}$UT Austin,$^{2}$University of Michigan,$^{3}$UC Berkeley,$^{4}$New York University
Figure 1. Monocular 3D hand mesh reconstruction. We propose HaMeR, a fully transformer-based approach for Hand Mesh Recovery. HaMeR achieves consistent improvements upon the state-of-the-art for 3D hand reconstruction. We can faithfully reconstruct hands in a wide variety of scenarios, including captures from different viewpoints (third person or egocentric), under occlusion, hands that interact with objects or other hands, hands with different skin tones, with gloves, from art paintings or mechanical hands. We encourage the reader to watch our reconstructions in the Supplemental Video to appreciate the temporal stability.
## Abstract
We present an approach that can reconstruct hands in 3D from monocular input. Our approach for Hand Mesh Recovery, HaMeR, follows a fully transformer-based architecture and can analyze hands with significantly increased accuracy and robustness compared to previous work. The key to HaMeR's success lies in scaling up both the data used for training and the capacity of the deep network for hand reconstruction. For training data, we combine multiple datasets that contain 2D or 3D hand annotations. For the deep model, we use a large scale Vision Transformer architecture. Our final model consistently outperforms the previous baselines on popular 3D hand pose benchmarks. To further evaluate the effect of our design in non-controlled
settings, we annotate existing in-the-wild datasets with 2D hand keypoint annotations. On this newly collected dataset of annotations, HInt, we demonstrate significant improvements over existing baselines. We will make our code, data and models publicly available upon publication. We make our code, data and models available on the project website: .
“It is because of his being armed with hands that man is the most intelligent animal.”
Anaxagoras
### 1. Introduction
Consider the images of hands interacting with the world in Figure 1. These interactions are happening in 3D, so to interpret them, we also need a system that can automatically perceive hands in 3D from visual input.
---
Recent developments in computer vision and NLP point to the direction where advances are achieved by simple, high capacity models, powered by huge amounts of data. This emerging insight has been demonstrated in the context of NLP by Large Language Models, like GPT-3 [3] and GPT-4 [43]. In the context of computer vision, we observe this with models like CLIP [45], Stable Diffusion [47] and SAM [29]. In the area of 3D human mesh recovery, a similar trend has been observed, where the simple, large scale HMR2.0 architecture [17] achieves state-of-the-art results.
In this paper, we take this philosophy and apply it to the problem of 3D hand pose estimation. We propose HaMeR, a robust and accurate approach for Hand Mesh Recovery from images and video frames. HaMeR captures faithful 3D reconstructions of hands in a variety of poses, viewpoints and visual conditions, as shown in Figure 1. This translates to improvements over existing baselines in the standard 3D hand pose benchmarks. More importantly, HaMeR shines when evaluated on challenging in-the-wild images, where we outperform the state-of-the-art by significant margins. Even though HaMeR is a single-frame approach, it recovers temporally smooth and consistent reconstructions when applied on video frames (please see the Supplemental Video for video results).
The key to HaMeR’s success lies in scaling up the techniques for hand mesh recovery. More specifically, we scale both the training data and the deep network architecture used for 3D hand reconstruction. For training data, we use multiple available sources of data with hand annotations, including both studio/controlled datasets with 3D ground truth [6, 19, 40, 56, 63, 64], and in-the-wild datasets annotated with 2D keypoint locations [15, 25, 52]. For our network, we use a large-scale transformer architecture [14, 57] which can successfully consume data of this scale. The combination of these two ingredients leads to significant improvements compared to previous work.
Benchmarking progress of these models is challenging and is often constrained on datasets captured in controlled conditions. To encourage evaluation on in-the-wild images, we introduce a new dataset of annotations, HInt, by annotating hands from diverse image sources, including videos from YouTube [9, 51] and egocentric captures [12, 18]. The annotations consist of 2D keypoints for the hand joints, as well as labels of the visibility (occluded or not) for each joint. We provide 2D hand keypoints annotations for 40.4K hands, where 86.7% of them are hands in natural contact. Even though with HInt we can only benchmark the 2D aspect of our 3D reconstruction, this evaluation is complementary to the existing benchmarks due to its diversity of data, and together provide a more holistic picture on the performance of different systems.
We contribute HaMeR, an approach for 3D hand mesh reconstruction from images and video frames. We demonstrate the key effect of scaling up to large scale training data and high capacity deep architectures for the problem of hand mesh recovery. We achieve state-of-the-art results where we obtain $ 2-3\times $ improvement in PCK@0.05 on in-the-wild datasets compared to previous works. We also contribute HInt, a dataset of annotations that complements training and evaluation of 3D hand reconstruction approaches. We make our model, code and data available to support future work.
### 2. Related work
3D hand pose and shape estimation. In this section we focus specifically on the works that estimate 3D hand pose and shape from a single RGB image. The earlier efforts [1, 2, 62] take inspiration from related work on human mesh recovery [27] - they use the MANO parametric hand model [48] and regress the hand pose and shape parameters given an RGB image as input. FrankMocap [49] is a good representative of this line of works which adopts a simple design, similar to HMR [27]. Followup work [10, 16, 31, 39] follows a non-parametric approach and directly regresses the vertices of the MANO mesh. This strategy often leads to results that align better with the image evidence, but it is more prone to failure in cases of occlusions and truncations. The improvements in 3D hand pose estimation have also led to progress in related problems, including joint hand pose and object reconstruction [21, 22, 54, 58] and reconstruction of two interacting hands [28, 32, 37, 38, 46, 55, 60, 61, 65]. More recently, there have been works that address other aspects of the problem. MobRecon [8] focuses on high inference speed, that could potentially be supported on a mobile device. HandOccNet [44] designs an architecture that could offer increased robustness to occlusions. AMVUR [24] proposes a probabilistic approach for hand pose and shape estimation. BlurHand [41] focuses on the problem of motion blur that often exists in footage that captures hand motion. Our work is orthogonal to these approaches. We adopt a simple design and we investigate the effect of scaling up the training data and the capacity of our architecture. Given that our main design is simple, the different choices of previous work could be combined with our HaMeR architecture which could potentially lead to further improvements.
Hand datasets. Many of the datasets used to train and evaluate 3D hand pose estimation systems are captured in indoor/studio settings and provide 3D ground truth. FreiHAND [64] is captured in a multi-camera setting and focuses on different hand poses as well as hands interacting with objects. HO-3D [19] and DexYCB [6] are also captured in a controlled setting with multiple cameras but focuses more specifically on cases where hands interact with objects. InterHand2.6M [40] is captured in a studio with a focus on two interacting hands. Hand pose datasets [52, 56]
---
Figure 2. Dataset and Architecture. (Top) Hand crops with keypoint annotations from our HInt dataset of annotations for different image sources, Hands23 [9], Epic-Kitchens [12, 13], and Ego4D [18]. We provide location annotations for 21 hand keypoints as well as the “occlusion” label for each joint. Occluded keypoints are marked using solid dot filled with black ● while non-occluded ones are filled with white ○. The pie chart shows the distribution and statistics of our dataset. (Bottom) The architecture for HaMeR follows a fully transformer-based design. We use a large scale ViT backbone [14] followed by a transformer decoder to regress the parameters of the hand.
captured in the Panoptic studio [26] also offer 3D hand annotations. AssemblyHands [42] annotated 3D hand poses for synchronized images from Assembly101 [50] which participants assemble and disassemble take-apart toys in a multi-camera setting. In this work, we use these datasets for training and evaluation. However, we also argue that to get a more holistic picture about the accuracy and the robustness of 3D hand pose estimation systems, it is important to evaluate performance on in-the-wild images as well.
While we cannot annotate 3D ground truth poses for in-the-wild images, there is work that annotates 2D keypoint positions. Among the larger scale efforts, COCO-WholeBody [25] provides hand annotations for the people in the COCO dataset [35] and Halpe [15] annotates hands in the HICO-DET dataset [4, 5]. Both of them source images from image datasets that contain very few egocentric images or transitionality moments. In our dataset, HInt, we sourced images from both egocentric and third-person video datasets. Since our annotated hands come from video, they depict more natural interactions with the world.
### 3. Technical approach
In this section, we describe HaMeR, our approach for hand mesh recovery from monocular input. We follow a simple, fully “transformerized” design that focuses on scaling up the training data and the deep model architecture.
#### 3.1. MANO parametric hand model
We adopt the MANO parametric model of the human hand [48]. MANO takes as input the pose parameters $ \theta \in \mathbb{R}^{48} $ and shape parameters $ \beta \in \mathbb{R}^{10} $ and defines a function $ \mathcal{M}(\theta, \beta) $ that returns the mesh of the hand $ M \in \mathbb{R}^{V \times 3} $, with $ V = 778 $ vertices. MANO additionally returns the joints $ X \in \mathbb{R}^{K \times 3} $ of the hand, for a total of $ K = 21 $ joints.
#### 3.2. Hand mesh recovery
Given an RGB image of a hand, $I$, our goal is to reconstruct the 3D hand surface. We approach this problem by estimating the MANO pose and shape parameters for the hand in the image. Similar to previous work in the parametric human [17, 27] and hand [49, 62] reconstruction, we use a network to learn the mapping $f$ from image pixels to MANO parameters. Our regressor also estimates camera parameters $\pi$. The camera $\pi$ corresponds to a translation $t \in \mathbb{R}^3$ that allows us to project the 3D mesh and the 3D joints to the image. Given fixed camera intrinsics $K$, the projection of the 3D joints $X$ is: $x = \pi(X) = \Pi_K(X + t)$. Eventually, we learn the mapping $f(I) = \Theta$, where the regressed parameters are $\Theta = \{\theta, \beta, \pi\}$.
#### 3.3. Architecture
HaMeR adopts a simple architecture with a fully transformer-based design (Figure 2, bottom), similar to [17]. We use a Vision Transformer (ViT) [14] as the backbone, followed by a transformer head that regresses the hand and camera parameters. We first convert the input RGB image to patches, which are fed as input tokens to ViT which follows the “huge” design, i.e., ViT-H. The ViT backbone processes the image patches and returns a series of output tokens. The transformer head is a transformer decoder that processes a single token while cross-attending to the ViT output tokens. The output of the head returns the parameters $ \Theta $ for the input image.
---
#### 3.4. Losses
For our training losses, we follow best practices for parametric human and hand reconstruction [17, 27, 30, 49] and supervise our model with a combination of 2D and 3D losses. For the images that provide 3D ground truth, we can directly apply a loss on the model parameters, $ \theta $ and $ \beta $. Simultaneously, we can encourage consistency in the actual 3D space, and supervise on the level of the 3D joints $ X^{*} $:
$$ \mathcal{L}_{\mathrm{3D}}=||\theta-\theta^{*}||_{2}^{2}+||\beta-\beta^{*}||_{2}^{2}+||X-X^{*}||_{1}。$$
To enable training with 2D annotations, we also apply a reprojection loss between the projection x of the 3D joints X and the ground truth 2D keypoint annotations $ x^{*} $:
$$ \mathcal{L}_{2\mathrm{D}}=||x-x^{*}||_{1}。$$
We apply this loss, even when 3D ground truth is available, since it promotes consistency on the output image space.
Finally, if only 2D keypoints are available, it is possible to recover an unnatural pose that still reprojects well to the image. To encourage the reconstruction of natural hands, we train discriminators $ D_k $ for a) the hand shape $ \beta $, b) the hand pose $ \theta $, and c) each hand joint angle separately [27]. Then, we can apply an adversarial loss:
$$ \mathcal{L}_{\mathrm{a d v}}=\sum_{k}(D_{k}(\Theta)-1)^{2}。$$
## 3.5. 训练数据
为了训练我们的模型，我们整合了多个提供2D或3D手部标注的数据集。具体来说，我们使用 FreiHAND [^64]、HO3D [^19]、MTC [^56]、RHD [^63]、InterHand2.6M [^40]、H2O3D [^19]、DEX YCB [^6]、COCO WholeBody [^25]、Halpe [^15] 和 MPII NZSL [^52]。这产生了 270 万个训练样本，比流行的 FrankMocap 系统 $4\times$ 的训练集大 [^49]。这些数据大部分是在受控环境中收集的（例如，摄影棚或多相机设置），而只有 5% 的训练样本（COCO WholeBody、Halpe 和 MPII NZSL）包含野外数据集中的图像。
### 4. 野外环境中的HInt：$\underline{H}$与$\underline{Int}$交互
在本节中，我们将介绍我们所贡献的数据集，旨在补充现有的用于训练和评估的数据集。由于我们专注于自然场景下的手部交互，因此将该数据集命名为HInt。HInt为每个关键点标注了二维手部关键点位置以及遮挡标签。我们基于Hands23 [^9]（使用了从作者处获取的早期版本）、Epic-Kitchens [^12]和Ego4D [^18]进行了构建。
与从图像数据中选取样本相比，从视频数据集中选取能够获取更多过渡时刻和自然姿态。对于HInt数据集，我们从三个视频数据集中提取帧。在Hands23中，我们选择了New Days子集[^9]，该子集包含人类进行日常活动的YouTube视频帧。在Epic-Kitchens中，我们选取了VISOR [^13]中的帧，这些帧提取自烹饪动作场景。在Ego4D [^18]中，我们选取了FHO（手部与物体预测）任务中的关键帧（pre45、pre30、pre15、preframe、contact-frame、point-of-no-return frame和postframe）。
对于验证集和测试集，我们随机采样帧以保持与源数据集相同的数据分布。对于训练集，我们的目标是纳入更具挑战性的样本，以弥补其他现有二维关键点数据集的不足。因此，对于New Days和VISOR数据集，我们采用随机抽样的方式选取一半样本，另一半则强制包含手-物体或手-手交互场景。对于Ego4D数据集，我们仍然采用随机抽样的方式，因为其中的关键帧本身通常就聚焦于交互场景。
从头标注手部关键点可能十分耗时。与[^25]类似，我们借助现有的关键点检测模型[^11]来初始化标注流程，从而获取关键点的粗略位置。根据标注说明（详见补充材料），标注人员需修正关键点的位置（见图2上方的标注示例）。此外，每个关键点还会被标注上“存在性”和“遮挡”标签。“存在性”用于指示该关键点是否出现在图像帧中，“遮挡”则用于指示该关键点是否被遮挡。据我们所知，HInt是首个为2D手部关键点提供“遮挡”标注的数据集。我们认为这将有助于对姿态估计系统进行更细致的分析。
总体而言，我们标注了40.4K个带关键点的手部样本，其中New Days数据集有12.0K个，VISOR数据集有5.3K个，Ego4D数据集有23.2K个。在我们的Ego4D子集中，我们标注了来自视频序列的9.3K个手部样本，这有助于未来对时序任务的评估。
最后，我们进行了标注一致性检查，选取了90张有效图像进行两次标注。在这一子集中，90.5%的遮挡标签和100%的“存在性”标签保持一致。就关键点位置而言，94.6%的可见关键点偏移距离在手掌长度的$0.25\times$以内（关于数据标注过程和分析的详情见补充材料）。
### 5. 实验
在本节中，我们将对系统性能进行定量与定性评估。首先，我们评估HaMeR的3D姿态精度（5.1小节）和2D姿态精度（5.2小节）；接着，我们对系统的某些特性进行消融分析（5.3小节），并展示定性结果与对比情况（5.4小节）。
---
### 5.1.3 3D姿态精度
为了评估HaMeR的3D精度，我们使用了两个标准的3D手部姿态估计基准数据集：FreiHAND [^64]和HO3Dv2 [^19]。这两个数据集均在受控的多摄像头环境中采集，并以3D手部网格的形式提供3D真实标注（采用MANO模型）。为了与以往的研究结果可比，我们遵循典型的协议[^34] [^44]，并报告用于评估3D关节和3D网格精度的指标。这些指标包括PA-MPJPE和AUC_J（3D关节评估），以及PA-MPVPE、AUC_V、F@5mm和F@15mm（3D网格评估）。
我们在表1中展示了FreiHAND的完整结果，在表2中展示了HO3Dv2的结果。我们将其与众多从单张图像中估计3D手部网格的基线方法进行比较，这些方法要么以参数化形式输出手部模型参数，要么以非参数化形式直接回归手部模型顶点。结果显示，我们的HaMeR方法取得了最先进的结果，在大多数指标上均优于以往的研究。
#### 5.2. 2D姿态精度
尽管3D手部姿态数据集为评估提供了精确的3D真实值，但它们通常是在受控环境下收集的，这就限制了受试者、视角、交互物体以及环境等方面的多样性。为了更好地分析不同手部姿态估计系统的特性，我们提议在HInt基准测试上进行评估，该基准测试相比以往的3D基准测试更贴近真实的野外条件。HInt的标注形式为2D关键点。仅基于2D的指标只能评估3D方法的投影精度，但由于图像本身来自野外环境，我们能够获得关于方法性能的补充证据。在评估时，我们采用常用的PCK指标[^59]来报告不同阈值下的结果。鉴于HInt的数据形式，我们提供了更为详细的分析，分别报告来自New Days [^9]、VISOR [^12]和Ego4D [^18]的图像所对应的结果。此外，我们还给出了更细致的结果：既考虑了所有关节，也分别考虑了被标注为可见（非遮挡）的关节以及被标注为遮挡的关节。
完整结果如表3所示。在此，我们与若干提供公开代码的近期3D手部网格估计方法进行了比较。与在FreiHAND和HO3D上的结果类似，我们发现我们的方法优于以往的基础模型。不过，在这些数据集上我们观察到的提升幅度更大。这凸显出我们方法在鲁棒性方面的显著改进——该方法能在多种基准测试中保持稳定的性能。FreiHAND和HO3D上的性能往往趋于饱和，因此我们的方法在这些数据集上的提升幅度较小也就不足为奇了。
| Method | PA-MPJPE $\downarrow$ | PA-MPVPE $\downarrow$ | F@5 $\uparrow$ | F@15 $\uparrow$ |
| I2L-MeshNet [^39] | 7.4 | 7.6 | 0.681 | 0.973 |
| Pose2Mesh [^10] | 7.7 | 7.8 | 0.674 | 0.969 |
| I2UV-HandNet [^7] | 6.7 | 6.9 | 0.707 | 0.977 |
| METRO [^33] | 6.5 | 6.3 | 0.731 | 0.984 |
| Tang et al. [^53] | 6.7 | 6.7 | 0.724 | 0.981 |
| Mesh Graphormer [^34] | 5.9 | 6.0 | 0.764 | 0.986 |
| MobRecon [^8] | 5.7 | 5.8 | 0.784 | 0.986 |
| AMVUR [^24] | 6.2 | 6.1 | 0.767 | 0.987 |
| Ours | 6.0 | 5.7 | 0.785 | 0.990 |
Table 1. Comparison with the state-of-the-art on the FreiHAND dataset [^64]. We use the standard protocol and report metrics for evaluation of 3D joint and 3D mesh accuracy. PA-MPVPE and PA-MPJPE numbers are in mm.
| Method | AUC $_J$ $\uparrow$ | PA-MPIPE $\downarrow$ | AUC $_V$ $\uparrow$ | PA-MPVPE $\downarrow$ | F@5 $\uparrow$ | F@15 $\uparrow$ |
| Liu et al. [^36] | 0.803 | 9.9 | 0.810 | 9.5 | 0.528 | 0.956 |
| HandOccNet [^44] | 0.819 | 9.1 | 0.819 | 8.8 | 0.564 | 0.963 |
| I2UV-HandNet [^7] | 0.804 | 9.9 | 0.799 | 10.1 | 0.500 | 0.943 |
| Hampali et al. [^19] | 0.788 | 10.7 | 0.790 | 10.6 | 0.506 | 0.942 |
| Hasson et al. [^21] | 0.780 | 11.0 | 0.777 | 11.2 | 0.464 | 0.939 |
| ArtiBoost [^58] | 0.773 | 11.4 | 0.782 | 10.9 | 0.488 | 0.944 |
| Pose2Mesh [^10] | 0.754 | 12.5 | 0.749 | 12.7 | 0.441 | 0.909 |
| I2L-MeshNet [^39] | 0.775 | 11.2 | 0.722 | 13.9 | 0.409 | 0.932 |
| METRO [^33] | 0.792 | 10.4 | 0.779 | 11.1 | 0.484 | 0.946 |
| MobRecon[^8] | - | 9.2 | - | 9.4 | 0.538 | 0.957 |
| Keypoint Trans [^20] | 0.786 | 10.8 | - | - | - | - |
| AMVUR [^24] | 0.835 | 8.3 | 0.836 | 8.2 | 0.608 | 0.965 |
| Ours | 0.846 | 7.7 | 0.841 | 7.9 | 0.635 | 0.980 |
Table 2. Comparison with the state-of-the-art on the HO3D dataset [^19]. We use the HO3Dv2 protocol and report metrics that evaluate accuracy of the estimated 3D joints and 3D mesh PA-MPVPE and PA-MPJPE numbers are in mm.
相比之下，野外数据集上的性能更能反映方法在不同视觉条件、不同视角以及不同交互场景（例如与周围物体的接触）下的鲁棒性。
#### 5.3. 消融分析
在证明了HaMeR的有效性之后，我们进一步对系统的不同选项进行了消融分析。
大规模数据与深度模型的影响。HaMeR的关键特点之一是：只要扩大训练数据规模并使用大规模模型进行手部重建，即便设计简单也能取得出色的性能。我们在HInt数据集上使用不同模型对这些选择进行了评估，完整结果见表5。具体而言，我们从基础设计开始（表5第2行），该设计遵循[^49]的选择（表5第1行），采用ResNet50架构[^23]，且训练集规模相对较小（仅为用于训练HaMeR的数据量的四分之一）。从定量结果来看，这种基础设计确实与[^49]非常接近。随后，在保持架构不变的情况下，我们增加了训练样本的数量，使用完整的训练集。这一模型（表5第3行）
---
| Method | New Days | VISOR | Ego4D |
| @0.05 | @0.1 | @0.15 | @0.05 | @0.1 | @0.15 | @0.05 | @0.1 | @0.15 |
| All Joints | FrankMocap [^49] | 16.1 | 41.5 | 60.3 | 16.8 | 45.6 | 66.3 | 13.2 | 37.1 | 56.0 |
| METRO [^33] | 14.7 | 38.9 | 57.4 | 16.8 | 45.4 | 65.7 | 13.2 | 36.0 | 54.5 |
| MeshGraphormer [^34] | 16.8 | 42.0 | 59.7 | 19.1 | 48.5 | 67.4 | 14.7 | 38.4 | 56.3 |
| HandOccNet (param) [^44] | 9.1 | 28.5 | 47.9 | 8.1 | 27.8 | 49.5 | 7.7 | 26.6 | 48.0 |
| HandOccNet (no param) [^44] | 13.7 | 39.2 | 59.4 | 12.4 | 38.7 | 61.9 | 11.0 | 35.3 | 59.2 |
| Ours | 49.4 | 79.3 | 89.8 | 44.4 | 77.5 | 89.7 | 40.3 | 72.4 | 85.2 |
| Visible Joints | FrankMocap [^49] | 20.2 | 49.2 | 67.6 | 20.5 | 52.3 | 71.7 | 16.4 | 43.5 | 62.4 |
| METRO [^33] | 19.3 | 47.7 | 66.0 | 19.7 | 52.0 | 72.1 | 15.9 | 42.0 | 60.6 |
| Mesh Graphormer [^34] | 22.3 | 51.6 | 68.9 | 23.6 | 56.4 | 74.7 | 18.5 | 45.9 | 63.5 |
| HandOccNet (param) [^44] | 10.2 | 31.4 | 51.3 | 8.6 | 28.0 | 50.0 | 7.4 | 26.3 | 48.3 |
| HandOccNet (no param) [^44] | 15.8 | 43.4 | 64.1 | 13.1 | 40.0 | 63.2 | 11.3 | 36.5 | 60.8 |
| Ours | 62.2 | 89.0 | 95.1 | 58.5 | 88.4 | 95.0 | 53.9 | 84.2 | 91.8 |
| Occluded Joints | FrankMocap [^49] | 9.3 | 28.1 | 47.0 | 11.0 | 33.0 | 55.0 | 8.4 | 27.1 | 45.4 |
| METRO [^33] | 7.0 | 23.7 | 42.5 | 10.2 | 32.4 | 53.9 | 8.2 | 26.4 | 45.1 |
| MeshGraphormer [^34] | 7.8 | 25.7 | 44.3 | 10.9 | 33.4 | 54.1 | 8.4 | 27.1 | 45.0 |
| HandOccNet (param) [^44] | 7.2 | 23.6 | 42.5 | 7.3 | 26.2 | 46.9 | 8.0 | 26.2 | 45.8 |
| HandOccNet (no param) [^44] | 9.8 | 31.3 | 50.9 | 9.9 | 33.7 | 55.4 | 9.7 | 31.2 | 52.9 |
| Ours | 28.4 | 62.4 | 80.1 | 26.9 | 61.8 | 81.2 | 24.3 | 58.7 | 77.3 |
Table 3. Evaluation on our HInt benchmark. We report results using PCK scores at three different thresholds. All methods are 3D and we evaluate the scores through the 2D projection of 3D joints. We report separate results for the three subsets of HInt, i.e., New Days of Hands [^9], Epic- Kitchens VISOR [^13] and Ego4D [^18]. We also report separate results considering all joints (first set of rows), considering only the joints annotated as visible (second set of rows), or considering only the joints annotated as occluded (third set of rows).
| Method | New Days | VISOR | Ego4D |
| @0.05 | @0.1 | @0.15 | @0.05 | @0.1 | @0.15 | @0.05 | @0.1 | @0.15 |
| All | Ours | 49.4 | 79.3 | 89.8 | 44.4 | 77.5 | 89.7 | 40.3 | 72.4 | 85.2 |
| Ours $^{{*}}$ | 51.7 | 81.9 | 91.9 | 56.5 | 88.1 | 95.7 | 47.2 | 79.6 | 90.6 |
| Vis. | Ours | 62.2 | 89.0 | 95.1 | 58.5 | 88.4 | 95.0 | 53.9 | 84.2 | 91.8 |
| Ours $^{{*}}$ | 62.9 | 89.4 | 95.8 | 66.6 | 92.8 | 97.5 | 59.5 | 87.3 | 94.4 |
| Occl. | Ours | 28.4 | 62.4 | 80.1 | 26.9 | 61.8 | 81.2 | 24.3 | 58.7 | 77.3 |
| Ours $^{{*}}$ | 33.1 | 68.3 | 84.7 | 42.6 | 79.0 | 91.3 | 33.4 | 70.2 | 85.4 |
Table 4. Effect of training with HInt. We compare our general model (Ours) with the model trained on HInt as well (Ours $^{*}$). We report PCK scores on the test set of HInt. Using the training set of HInt can be helpful particularly to improve performance on egocentric data (VISOR and Ego4D).
该方法已经取得了相较于先前基准模型的持续改进。同样，如果我们使用基础设计的小规模训练集，但采用大规模架构——即ViT-H [^14]（表5的第4行）——我们也会发现其相较于基础设计有改进。最后，我们可以将这两种独立优化结合起来：增加训练样本数量的同时采用高容量架构，这实际上就是HaMeR的设计思路。这一版本（表5的第5行）在性能上大幅超越其他版本，充分证明了大规模数据与大型深度模型在我们设计中的重要作用。
基于HInt数据的训练。与以往研究不同，我们此前并未使用HInt的训练集进行训练；不过在此我们提供了使用该数据训练的直接对比结果。表4展示了在HInt数据集上的详细实验结果。我们发现，在VISOR和Ego4D这两个以第一人称视角为主的HInt子数据集上，模型性能有明显提升。这是因为此前野外环境中几乎没有带有手部标注的第一人称视角数据，因此利用这类标注数据进行训练有助于提升模型性能。除此之外，我们在New Days数据集上也观察到了性能提升；由于New Days主要包含第三人称视角视频，因此提升幅度较小，但在各项指标上均表现出一致性。
#### 5.4. 定性结果
我们在图1中展示了该方法的定性结果，而在图4中则进行了更详细的分析，图中呈现了我们重建的3D手部的侧视图和顶视图。我们的方法对不同视角、不同肤色或手部外观（例如佩戴不同类型手套的情况）以及会造成不同程度遮挡的各种交互物体都具有较强的适应性。此外，在图3中，我们展示了与以往基准模型的更详细对比结果。我们将其与METRO [^33]、MeshGraphormer [^34]以及FrankMocap [^49]进行了比较。与定量对比的趋势一致，HaMeR始终比以往的方法更具鲁棒性和精确度。
#### 5.5. 实现细节
与以往的研究类似，例如[^33]、[^34]、[^49]，HaMeR在测试时以手部边界框作为输入，输出3D手部网格。我们首先检测[^9]、[^51]手部边界框以及手部所属侧别（左手或右手），随后进行处理。
---
| Method | Large Data | Large Model | New Days | VISOR | Ego4D |
| @0.05 | @0.1 | @0.15 | @0.05 | @0.1 | @0.15 | @0.05 | @0.1 | @0.15 |
| All | FrankMocap [^49] | ✗ | ✗ | 16.1 | 41.5 | 60.3 | 16.8 | 45.6 | 66.3 | 13.2 | 37.1 | 56.0 |
| Base design | ✗ | ✗ | 16.9 | 43.6 | 62.7 | 17.5 | 47.5 | 67.3 | 14.0 | 38.0 | 56.2 |
| + large data | ✓ | ✗ | 31.3 | 65.8 | 81.9 | 29.9 | 65.0 | 81.7 | 24.8 | 56.4 | 74.2 |
| + large model | ✗ | ✓ | 25.9 | 58.9 | 77.0 | 24.1 | 62.5 | 81.3 | 19.5 | 51.8 | 71.3 |
| HaMeR | ✓ | ✓ | 49.4 | 79.3 | 89.8 | 44.4 | 77.5 | 89.7 | 40.3 | 72.4 | 85.2 |
| Visible | FrankMocap [^49] | ✗ | ✗ | 20.2 | 49.2 | 67.6 | 20.5 | 52.3 | 71.7 | 16.4 | 43.5 | 62.4 |
| Base design | ✗ | ✗ | 21.2 | 51.5 | 70.4 | 21.4 | 54.5 | 73.5 | 17.5 | 45.2 | 64.0 |
| + large data | ✓ | ✗ | 38.5 | 75.0 | 88.0 | 36.6 | 73.2 | 87.0 | 30.6 | 65.2 | 81.3 |
| + large model | ✗ | ✓ | 33.1 | 69.4 | 85.1 | 29.2 | 72.9 | 88.9 | 24.4 | 62.6 | 81.5 |
| HaMeR | ✓ | ✓ | 62.2 | 89.0 | 95.1 | 58.5 | 88.4 | 95.0 | 53.9 | 84.2 | 91.8 |
| Occluded | FrankMocap [^49] | ✗ | ✗ | 9.3 | 28.1 | 47.0 | 11.0 | 33.0 | 55.0 | 8.4 | 27.1 | 45.4 |
| Base design | ✗ | ✗ | 9.4 | 29.8 | 48.9 | 11.8 | 35.6 | 57.4 | 9.3 | 27.6 | 45.2 |
| + large data | ✓ | ✗ | 19.0 | 49.5 | 70.8 | 19.1 | 51.6 | 72.5 | 17.1 | 44.9 | 64.6 |
| + large model | ✗ | ✓ | 14.7 | 41.7 | 63.3 | 16.3 | 47.9 | 69.4 | 14.6 | 40.5 | 60.4 |
| HaMeR | ✓ | ✓ | 28.4 | 62.4 | 80.1 | 26.9 | 61.8 | 81.2 | 24.3 | 58.7 | 77.3 |
Table 5. Effect of large scale data and deep model. We evaluate the effect of different design choices when evaluating on HInt. We start from a basic design similar to FrankMocap [^49], using a ResNet50 backbone and a small training set (2nd row). Increasing the training examples by $4\times$ (3rd row) or adopting a high capacity ViT-H architecture (4th row) results in consistent improvements in 2D accuracy over the base model. Combining the data scale and high capacity architecture (HaMeR-5th row), obtains the best results by large margins.
Figure 3. Qualitative comparison. We compare our approach qualitatively with state-of-the-art methods for hand mesh reconstruction. The baselines include METRO [^33], Mesh Graphormer [^34] and FrankMocap [^49]. METRO and Mesh Graphormer are non-parametric methods (regressing MANO vertices directly), while FrankMocap and HaMeR (ours) are parametric methods (regressing MANO parameters). The reconstructions from HaMeR are consistently better, particularly on more challenging examples, e.g., cases with motion blur, or images with hand-hand or hand-object interaction. We encourage the reader to watch the Supplemental Video for more comparisons.
我们独立处理每只手部。我们仅训练针对右手的网络；如果输入的是左手图像，我们会将输入图像和输出网格进行镜像处理。
#### 6. 结论
我们提出了HaMeR，一种基于单目输入的3D手部网格重建方法。HaMeR结构简单，没有多余的复杂设计，它体现了两项关键设计选择的重要性：一是扩大手部网格恢复模型的训练数据量，二是采用适合3D手部重建的大容量架构。通过整合多个带有手部标注（2D或3D）的数据集，并采用高容量的深度模型（ViT-H $^{[14]}$），我们能够在传统的3D手部姿态基准测试中超越以往的研究。此外，我们还为包含多样化手部样本的数据集提供了2D关键点标注，这些样本来自第一人称视角$^{[12,18]}$或YouTube视频$^{[9]}$。评估…
---
Figure 4. Qualitative results. We present qualitative results of our approach on the test set of HInt. We include images from New Days (row 1-2), VISOR (row 3-4), Ego4D (row 5-6), as well as various Internet images (row 7-8). HaMeR is particularly robust and can gracefully handle cases with heavy occlusion and interactions with objects or other hands.
在这一具有挑战性的新HInt基准测试中，我们的方法相较于以往基线模型展现出了更为显著的改进。我们希望该方法的鲁棒性和精确度能够激发人们进一步将其应用于3D手部估计至关重要的领域，包括但不仅限于机器人技术、动作识别以及手语理解等。
致谢：我们感谢BAIR社区成员的有益讨论，也感谢StabilityAI通过计算资源资助对我们的支持。本研究得到了BAIR/BDD赞助方、ONR MURI项目（N00014-211-2801）以及DARPA MCS项目的资助。DF和DS的研究得到了美国国家科学基金会授予的2006619号基金的支持。
---
### 参考文献
[^1]: Seungryul Baek, Kwang In Kim, and Tae-Kyun Kim. Pushing the envelope for RGB-based dense 3D hand pose estimation via neural rendering. In CVPR, 2019.
[^2]: Adnane Boukhayma, Rodrigo de Bem, and Philip HS Torr. 3D hand shape and pose from images in the wild. In CVPR, 2019.
[^3]: Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In NeurIPS, 2020.
[^4]: Yu-Wei Chao, Zhan Wang, Yugeng He, Jiaxuan Wang, and Jia Deng. HICO: A benchmark for recognizing human-object interactions in images. In ICCV, 2015.
[^5]: Yu-Wei Chao, Yunfan Liu, Xieyang Liu, Huayi Zeng, and Jia Deng. Learning to detect human-object interactions. In WACV, 2018.
[^6]: Yu-Wei Chao, Wei Yang, Yu Xiang, Pavlo Molchanov, Ankur Handa, Jonathan Tremblay, Yashraj S Narang, Karl Van Wyk, Umar Iqbal, Stan Birchfield, Jan Kautz, and Dieter Fox. DexYCB: A benchmark for capturing hand grasping of objects. In CVPR, 2021.
[^7]: Ping Chen, Yujin Chen, Dong Yang, Fangyin Wu, Qin Li, Qingpei Xia, and Yong Tan. I2UV-HandNet: Image-to-UV prediction network for accurate and high-fidelity 3D hand mesh modeling. In ICCV, 2021.
[^8]: Xingyu Chen, Yufeng Liu, Yajiao Dong, Xiong Zhang, Chongyang Ma, Yanmin Xiong, Yuan Zhang, and Xiaoyan Guo. Mobrecon: Mobile-friendly hand mesh reconstruction from monocular image. In CVPR, 2022.
[^9]: Tianyi Cheng, Dandan Shan, Ayda Sultan, Jiaqi Geng, Richard EL Higgins, and David F Fouhey. Towards a richer 2D understanding of hands at scale. In NeurIPS, 2023.
[^10]: Hongsuk Choi, Gyeongsik Moon, and Kyoung Mu Lee. Pose2Mesh: Graph convolutional network for 3D human pose and mesh recovery from a 2D human pose. In ECCV, 2020.
[^11]: MMPose Contributors. OpenMMLab pose estimation toolbox and benchmark. , 2020.
[^12]: Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. Scaling egocentric vision: The EPICKITCHENS dataset. In ECCV, 2018.
[^13]: Ahmad Darkhalil, Dandan Shan, Bin Zhu, Jian Ma, Amlan Kar, Richard Higgins, Sanja Fidler, David Fouhey, and Dima Damen. EPIC-KITCHENS VISOR benchmark: Video Segmentations and Object Relations. In NeurIPS Track on Datasets and Benchmarks, 2022.
[^14]: Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2020.
[^15]: Hao-Shu Fang, Jiefeng Li, Hongyang Tang, Chao Xu, Haoyi Zhu, Yuliang Xiu, Yong-Lu Li, and Cewu Lu. Alpha-Pose: Whole-body regional multi-person pose estimation and tracking in real-time. PAMI, 2022.
[^16]: Liuhao Ge, Zhou Ren, Yuncheng Li, Zehao Xue, Yingying Wang, Jianfei Cai, and Junsong Yuan. 3D hand shape and pose estimation from a single RGB image. In CVPR, 2019.
[^17]: Shubham Goel, Georgios Pavlakos, Jathushan Rajasegaran, Angjoo Kanazawa, and Jitendra Malik. Humans in 4D: Reconstructing and tracking humans with transformers. In ICCV, 2023.
[^18]: Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4D: Around the world in 3,000 hours of egocentric video. In CVPR, 2022.
[^19]: Shreyas Hampali, Mahdi Rad, Markus Oberweger, and Vincent Lepetit. HOnnotate: A method for 3D annotation of hand and object poses. In CVPR, 2020.
[^20]: Shreyas Hampali, Sayan Deb Sarkar, Mahdi Rad, and Vincent Lepetit. Keypoint transformer: Solving joint identification in challenging hands and object interactions for accurate 3D pose estimation. In CVPR, 2022.
[^21]: Yana Hasson, Gul Varol, Dimitrios Tzionas, Igor Kalevatykh, Michael J Black, Ivan Laptev, and Cordelia Schmid. Learning joint reconstruction of hands and manipulated objects. In CVPR, 2019.
[^22]: Yana Hasson, Bugra Tekin, Federica Bogo, Ivan Laptev, Marc Pollefeys, and Cordelia Schmid. Leveraging photometric consistency over time for sparsely supervised hand-object reconstruction. In CVPR, 2020.
[^23]: Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[^24]: Zheheng Jiang, Hossein Rahmani, Sue Black, and Bryan M Williams. A probabilistic attention model with occlusion-aware texture regression for 3D hand reconstruction from a single RGB image. In CVPR, 2023.
[^25]: Sheng Jin, Lumin Xu, Jin Xu, Can Wang, Wentao Liu, Chen Qian, Wanli Ouyang, and Ping Luo. Whole-body human pose estimation in the wild. In ECCV, 2020.
[^26]: Hanbyul Joo, Hao Liu, Lei Tan, Lin Gui, Bart Nabbe, Iain Matthews, Takeo Kanade, Shohei Nobuhara, and Yaser Sheikh. Panoptic studio: A massively multiview system for social motion capture. In ICCV, 2015.
[^27]: Angjoo Kanazawa, Michael J Black, David W Jacobs, and Jitendra Malik. End-to-end recovery of human shape and pose. In CVPR, 2018.
[^28]: Dong Uk Kim, Kwang In Kim, and Seungryul Baek. End-to-end detection and pose estimation of two interacting hands. In ICCV, 2021.
[^29]: Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything. In ICCV, 2023.
### [^30]: Nikos Kolotouros, Georgios Pavlakos, Michael J Black, and Kostas Daniilidis. Learning to reconstruct 3D human pose and shape via model-fitting in the loop. In ICCV, 2019
---
[^31]: Dominik Kulon, Riza Alp Guler, Iasonas Kokkinos, Michael M Bronstein, and Stefanos Zafeiriou. Weakly-supervised mesh-convolutional hand reconstruction in the wild. In CVPR, 2020.
[^32]: Mengcheng Li, Liang An, Hongwen Zhang, Lianpeng Wu, Feng Chen, Tao Yu, and Yebin Liu. Interacting attention graph for single image two-hand reconstruction. In CVPR, 2022.
[^33]: Kevin Lin, Lijuan Wang, and Zicheng Liu. End-to-end human pose and mesh reconstruction with transformers. In CVPR, 2021.
[^34]: Kevin Lin, Lijuan Wang, and Zicheng Liu. Mesh graphormer. In ICCV, 2021.
[^35]: Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Common objects in context. In ECCV, 2014.
[^36]: Shaowei Liu, Hanwen Jiang, Jiarui Xu, Sifei Liu, and Xiaolong Wang. Semi-supervised 3D hand-object poses estimation with interactions in time. In CVPR, 2021.
[^37]: Hao Meng, Sheng Jin, Wentao Liu, Chen Qian, Mengxiang Lin, Wanli Ouyang, and Ping Luo. 3D interacting hand pose estimation by hand de-occlusion and removal. In ECCV, 2022.
[^38]: Gyeongsik Moon. Bringing inputs to shared domains for 3D interacting hands recovery in the wild. In CVPR, 2023.
[^39]: Gyeongsik Moon and Kyoung Mu Lee. I2L-MeshNet: Image-to-pixel prediction network for accurate 3D human pose and mesh estimation from a single RGB image. In ECCV, 2020.
[^40]: Gyeongsik Moon, Shoou-I Yu, He Wen, Takaaki Shiratori, and Kyoung Mu Lee. InterHand2.6M: A dataset and baseline for 3D interacting hand pose estimation from a single RGB image. In ECCV, 2020.
[^41]: Yeonguk Oh, JoonKyu Park, Jaeha Kim, Gyeongsik Moon, and Kyoung Mu Lee. Recovering 3D hand mesh sequence from a single blurry image: A new dataset and temporal unfolding. In CVPR, 2023.
[^42]: Takehiko Ohkawa, Kun He, Fadime Sener, Tomas Hodan, Luan Tran, and Cem Keskin. Assembly Hands: Towards egocentric activity understanding via 3D hand pose estimation. In CVPR, 2023.
[^43]: OpenAI. GPT-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
[^44]: JoonKyu Park, Yeonguk Oh, Gyeongsik Moon, Hongsuk Choi, and Kyoung Mu Lee. HandOccNet: Occlusion-robust 3D hand mesh estimation network. In CVPR, 2022.
[^45]: Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021.
[^46]: Pengfei Ren, Chao Wen, Xiaozheng Zheng, Zhou Xue, Haifeng Sun, Qi Qi, Jingyu Wang, and Jianxin Liao. Decoupled iterative refinement framework for interacting hands reconstruction from a single RGB image. In ICCV, 2023.
[^47]: Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
[^48]: Javier Romero, Dimitris Tzionas, and Michael J Black. Embodied hands: Modeling and capturing hands and bodies together. ACM Transactions on Graphics, 36(6), 2017.
[^49]: Yu Rong, Takaaki Shiratori, and Hanbyul Joo. FrankMocap: A monocular 3D whole-body pose estimation system via regression and integration. In ICCV, 2021.
[^50]: Fadime Sener, Dibyadip Chatterjee, Daniel Shelepov, Kun He, Dipika Singhania, Robert Wang, and Angela Yao. Assembly101: A large-scale multi-view video dataset for understanding procedural activities. In CVPR, 2022.
[^51]: Dandan Shan, Jiaqi Geng, Michelle Shu, and David F Fouhey. Understanding human hands in contact at internet scale. In CVPR, 2020.
[^52]: Tomas Simon, Hanbyul Joo, Iain Matthews, and Yaser Sheikh. Hand keypoint detection in single images using multiview bootstrapping. In CVPR, 2017.
[^53]: Xiao Tang, Tianyu Wang, and Chi-Wing Fu. Towards accurate alignment in real-time 3dD hand-mesh reconstruction. In ICCV, 2021.
[^54]: Tze Ho Elden Tse, Kwang In Kim, Ales Leonardis, and Hyung Jin Chang. Collaborative learning for hand and object reconstruction with attention-guided graph convolution. In CVPR, 2022.
[^55]: Congyi Wang, Feida Zhu, and Shilei Wen. MeMaHand: Exploiting mesh-mano interaction for single image two-hand reconstruction. In CVPR, 2023.
[^56]: Donglai Xiang, Hanbyul Joo, and Yaser Sheikh. Monocular total capture: Posing face, body, and hands in the wild. In CVPR, 2019.
[^57]: Yufei Xu, Jing Zhang, Qiming Zhang, and Dacheng Tao. ViTPose: Simple vision transformer baselines for human pose estimation. NeurIPS, 2022.
[^58]: Lixin Yang, Kailin Li, Xinyu Zhan, Jun Lv, Wenqiang Xu, Jiefeng Li, and Cewu Lu. ArtiBoost: Boosting articulated 3D hand-object pose estimation via online exploration and synthesis. In CVPR, 2022.
[^59]: Yi Yang and Deva Ramanan. Articulated human detection with flexible mixtures of parts. PAMI, 2012.
[^60]: Zhengdi Yu, Shaoli Huang, Chen Fang, Toby P Breckon, and Jue Wang. ACR: Attention collaboration-based regressor for arbitrary two-hand reconstruction. In CVPR, 2023.
[^61]: Baowen Zhang, Yangang Wang, Xiaoming Deng, Yinda Zhang, Ping Tan, Cuixia Ma, and Hongan Wang. Interacting two-hand 3D pose and shape reconstruction from single color image. In ICCV, 2021.
[^62]: Xiong Zhang, Qiang Li, Hong Mo, Wenbo Zhang, and Wen Zheng. End-to-end hand mesh recovery from a monocular RGB image. In ICCV, 2019.
[^63]: Christian Zimmermann and Thomas Brox. Learning to estimate 3D hand pose from single RGB images. In ICCV, 2017.
#### [^64]: Christian Zimmermann, Duygu Ceylan, Jimei Yang, Bryan Russell, Max Argus, and Thomas Brox. FreiHAND: A dataset for markerless capture of hand pose and shape from single RGB images. In ICCV, 2019
---
[^65]: Binghui Zuo, Zimeng Zhao, Wenqian Sun, Wei Xie, Zhou Xue, and Yangang Wang. Reconstructing interacting hands with interaction prior from monocular images. In ICCV, 2023.