# MVHM：用于精确三维手部姿态估计的大规模多视角手部网格基准数据集 陈良健 $^{1}$、林世耀 $^{2}$、谢宇生 $^{*3}$、林彦宇 $^{4}$、谢晓辉 $^{1}$ $^{1}$加州大学欧文分校，$^{2}$腾讯美国公司，$^{3}$亚马逊公司，$^{4}$国立交通大学，{liangjc2,xhx}@ics.uci.edu，，， ## 摘要 从单张RGB图像中估计三维手部姿态极具挑战性，因为深度模糊会导致该问题成为不适定问题。利用三维手部网格标注和多视角图像训练手部姿态估计器，往往能带来显著的性能提升。然而，现有的多视角数据集规模相对较小，其中的手部关节标注要么由现成的追踪器生成，要么通过模型预测得到，这两种方式都可能存在误差并引入偏差。收集带有精准网格和关节标注的大规模多视角三维手部姿态图像数据集虽具有重要价值，但工作量极大。在本文中，我们设计了一种旋转匹配算法，该算法能够让刚性网格模型与任意目标网格真值进行匹配。基于该匹配算法，我们提出了一种高效的流程，用于生成大规模多视角手部网格（MVHM）数据集，该数据集包含精准的三维手部网格和关节标签。我们进一步提出了一种多视角手部姿态估计方法，验证了使用我们生成的数据集训练手部姿态估计器能够大幅提升性能。实验结果表明，在MHP数据集上，我们的方法达到了0.990的$AUC_{20-50}$指标，而此前该数据集上的最佳性能仅为0.939。我们的数据集可在获取。 ### 1. 引言 从图像中估计三维手部姿态受到了越来越多的关注，因为它对人机交互（HCI）[^1] [^27] [^26]、虚拟现实（VR）[^12] [^18]、增强现实（AR）[^7]、医疗诊断[^20]以及手语理解[^43]等广泛应用而言至关重要。尽管数十年来研究人员在该领域投入了大量精力，但仍有若干难题尚未解决。其中最为关键的挑战之一是如何应对单视角三维手部姿态估计中存在的深度模糊问题。 传统研究主要侧重于推断三维手部…… ![Image](images/ee827024def21fab0e25fe10a0d238e10b263839c8d6532929c814b3fad6b848.jpg)
Figure 1. Our core idea. We build a synthetic dataset from a multiview perspective, e.g., rendering hand images from different angles. With the aid of this dataset, a single-view method takes the image from each view and generates a possible hand pose candidate. We proposed a multi-view method takes different single-view predictions as input and predicts the final result.
poses from either depth or RGB images directly. To address the problems caused by depth ambiguity, Some previous studies [^5] [^19] [^8] [^9] want to address this problem by leveraging depthmap information. These works come up with several ways to introduce depthmap into the training procedure, such as making depthmap as intermediate supervision [^19] or using depth regularizer [^5]. On the other hand, recent studies [^22] [^50] point out that imposing 3D hand shape (mesh) supervision can boost both the performance of 3D hand pose and shape estimators. It is clear that 3D hand shape brings richer hand structure information than hand keypoints. Furthermore, a preset mesh serves as a strong prior to reduce the freedom of the hand, therefore mitigating depth ambiguity. Along this line, several methods such as [^3] [^2] [^45] [^25] [^40] [^39] [^48] [^47] [^46] are proposed. Despite the potential, the aforementioned methods highly rely on a preset hand model learned with a large number of accurate 3D mesh annotations. Hence, a large-scale dataset with accurate annotations of mesh vertices is in great demand. Accurate mesh ground truth is hard to be manually an- --- notated in general. The hand mesh annotations in most existing datasets are often annotated by hand shape estimator which can be inaccurate because hand mesh estimation itself is an even more challenging task. Most existing methods leveraging mesh information for 3D hand pose estimation are derived based on a single view. However, more mesh information is insufficient to address depth ambiguity. Thus, 3D pose estimation still remains ill-posed in these methods. The issue of depth ambiguity can be tackled by multi-view vision according to epipolar geometry. Multi-view sensing systems can capture hand images from cameras in different angles and therefore depth information of hands can be accurately inferred as long as camera parameters are known. Inspired by the above observations, we aim to build a large-scale multi-view hand mesh dataset that provides hand meshes and multi-view hand images simultaneously for training pose estimators. In this work, we present an effective mechanism to synthesize 3D hand joint and mesh annotations, and establish a large-scale multi-view hand mesh (MVHM) dataset. We acquired a hand mesh model with a rigging system, and 3D hand ground truth from existing datasets, and a rigged hand model to match the given ground truth to perform various gestures. We render the hand model from different angles to collect multi-view images, as well as the 3D keypoints and mesh annotations to built MVHM. Then, we determine if the generated MVHM dataset can be used to improve 3D hand pose estimators. To this end, a multi-view based approach is developed for inferring 3D hand poses. The experimental results show that the resultant pose estimator can be greatly boosted by leveraging the generated MVHM dataset, and performs favorably against existing methods. This work makes three major contributions, which are summarized as follows: 1. We propose an effective mechanism for compiling a large-scale multi-view hand mesh (MVHM) dataset for 3D hand pose estimator training. To the best of our knowledge, this is the first large-scale hand dataset with multi-view hand images, accurate mesh annotations, hand joint keypoints labels. 2. We present a multi-view hand pose estimation approach based on an end-to-end trainable graph convolutional neural network where information from multi-view images is utilized to predict 3D hand poses. 3. Our proposed approach achieves the state-of-the-art performance on the benchmark, the MHP dataset, in both single-view and multi-view settings. ### 2. Related Work #### 2.1. RGB based 3D Hand Pose Estimation RGB cameras are much more widely used than depth sensors. Estimating 3D hand poses merely from monocular RGB images are more practical and active in the literature [^5] [^10] [^19] [^29] [^37] [^41] [^49] [^24] [^23] [^11]. The pioneering work by Zimmermann and Brox [^49] utilizes convolutional neural networks (CNN) to extract image feature, and feed camera parameters with these features to a 3D lift network where depth information is then estimated. Based on [^49], Iqbal et al. [^19] leverage depth maps as intermediate supervision. Meanwhile, Cai et al. [^5] propose a weakly supervised approach that reconstructs the depth map and uses it as a regularizer during model training. #### 2.2. 3D Hand Mesh Estimation 3D hand pose estimation provides sparse joint locations. However, many computer vision applications would benefit more from hand shape information than sparse joints. Therefore, 3D hand mesh estimation, an effective shape representation, has emerged as an increasingly popular topic [^16] [^3] [^2] [^21] [^45]. Most methods [^3] [^2] [^45] [^25] [^40] are developed around a pre-defined deformable hand mesh model called MANO [^32]. Because of the high degree of freedom and complexity of the hand gesture, searching for the right hand mesh in such a high dimensional space is quite challenging. Using this MANO model often relies on strong prior to constrain the model to only regress low-dimensional model parameters, and may ignore the high-dimensional information. Ge et al. [^16] argue that mesh is a graph-structure data, and propose to directly regress 3D mesh vertices through graph convolutional neural network (GCN) with a pre-defined mesh graph. #### 2.3. Multi-View Hand Pose Estimation Unlike single-view pose estimation, few research efforts focus on 3D hand pose estimation from multi-view data. Ge et al. [^15] first introduce multi-view CNN to formulate it as an estimation problem. Their method assumes that hand joint locations independently follow 3D Gaussian distributions, and uses CNN to estimate the mean and variance of the location distribution of each joint. The main drawbacks of their method include 1) its inability to train in an end-to-end manner and 2) its impractical assumption about the independence among different joint locations. Simon et al. [^33] propose a multi-view system which is trained to progressively improve hand keypoints detection. Their method would work well on fine-tuning a well pre-trained estimator but could not train a 3D hand pose estimator from scratch. #### 2.4. 3D Hand Pose Benchmark 目前已有大量研究工作致力于构建用于3D手部姿态估计的手部数据集，例如 [^36] [^38] [^35] [^42] [^44] [^49] [^34] [^30] [^28] [^25] [^50]。我们在表1中总结了公开可用的手部数据集以及我们的数据集。大多数现有数据集并不包含网格信息，因为人工标注手部网格几乎是不可行的。 ---
Table 1. Comparison between our dataset with publicly available datasets. Auto in field Annotation represents that the annotation is made by some algorithms and therefore may not be accurate. Mano means the emsh annotation is fitted by Mano Model
DatasetRGBDepthImage TypeResolutionAnnotationDataset SizeMulti-ViewMesh
ICVL [^36]✗✓real$320 \times 320$tracking18K✗✗
NYU [^38]✗✓real$648 \times 480$tracking243K✗✗
MSRA [^35]✗✓real$1920 \times 1080$tracking76K✗✗
BigHand2.2M [^42]✗✓real$640 \times 480$marker2.2M✗✗
STB [^44]✓✓real$640 \times 480$manual36K✗✗
RHP [^49]✓✓synthetic$640 \times 480$synthetic44K✗✗
Dexter+Object [^34]✓✓real$640 \times 480$manual3K✗✗
EgoDexter [^30]✓✓real$640 \times 480$manual3K✗✗
MHP [^17]✓✗real$480 \times 480$auto80K✓✗
FreiHand [^50]✓✗real$224 \times 224$auto134K✗Mano
InterHand [^28]✓✗real$512 \times 334$auto2.2M✓Mano
Youtube Hand [^25]✓✗real$256 \times 256$auto47K✗Mano
Ours✓✓synthetic$256 \times 256$synthetic320K✓✓
人工标注手部网格对标注员而言几乎是不可能完成的任务。 为了解决这种劳动密集型标注的问题，近期的研究 [^50] [^25] [^9] 提出了半自动的RGB图像标注方法。FreiHand（Zimmermann等人 [^50]）采用迭代流程：先由训练好的模型对图像进行预测，再由标注员进行必要的调整。YoutubeHand（Kulon等人 [^25]）则利用OpenPose [^6] 获取2D标注信息，进而回归出MANO模型的参数。系统会根据置信度分数设置阈值，剔除置信度低的数据，以此保证标注质量。尽管在RGB图像标注的效率与效果上取得了进展，但标注的准确性仍严重依赖于过程中使用的预训练模型。此外，这些方法都以MANO模型作为真实网格生成器，正如2.2节所述，这种方式可能会丢失手部的高维信息。与现有数据集相比，我们的数据集包含大规模RGB图像且涵盖多种序列；同时，由于是合成数据，手部关节和网格的标注准确率可达100%。我们首次尝试收集具备大规模多视角训练图像的数据集，从而从多视角维度优化姿态估计器的训练效果。 ### 3. 多视角数据集的生成 目前尚无同时提供大规模网格标注和多视角3D手部标注的数据集，尽管许多潜在应用都能从中受益。因此，我们创建了一种名为MultiView Hand Mesh（MVHM）的新数据集，用于训练多视角手部姿态估计器。为了获得精确的网格和关节标注，我们使用了来自TurboSquid的优质手部模型 $^{1}$。 ![Image](images/7ec7cad98829e1c27935f0081da58d8acc7db0bc7dff91daca8077398c5ea383.jpg)
(a)
![Image](images/b7b3d57058705c00b8e7f5ec1d60a60f72255258a1c898505a17083da53454d1.jpg)
(b)
![Image](images/20fb01c18509d128ce10a5bd8a73f4abd2be6f3fd59dbf733b02b48504fb7e38.jpg)
(c)
Figure 2. An example of hand joint and bone labels and their orders adopted in this work. (a) Joint labels. (b) Bone labels, which are used by Algorithm 1 during spin matching. (c) A failure case when directly rigging the hand mesh based on the bone coordinates without using Algorithm 1.
该模型提供了约2000个网格顶点以及用于构成各种手势的骨骼系统。我们使用开源软件Blender $^{2}$ 来渲染图像。 为了生成不同手势的图像和网格，我们采用了NYU数据集 [^38]，该数据集提供了各种手部姿态、精确的关键点标注，以及与我们模型中骨骼相匹配的绑定手部骨骼。绑定系统中的手部骨骼共有7个自由度：其中3个用于骨骼头部，3个用于骨骼尾部，1个用于旋转。前6个自由度决定了骨骼的位置，最后一个自由度代表其方向。为了正确绑定手部关节和网格表面，我们必须同时考虑位置和方向。图2(c)展示了若不进行方向匹配、仅移动骨骼位置时所得到的扭曲网格示例。 为此，我们将骨骼向量定义为骨骼尾部与头部之间的差值。假设u为当前处理的骨骼的骨骼向量，我们取其 --- ![Image](images/d1c2341ad8e2c67695a6a4e88e4ddb46ca0a25519799af2cec6b992e6c0d643e.jpg)
Figure 3. Some examples of the MVHM dataset. Column 1 to Column 8 shows the image with 2D annotation from view 1 to view 8, Column 9 shows the mesh. Column 10 shows the 3D annotation
将相邻骨骼的向量作为旋转参考基准。我们将旋转符号向量定义为$u \times ref \times u$，并确保该向量在与真实数据匹配骨骼后保持不变。详细算法总结在算法1中。 对于每个真实手势，我们设置8个不同的相机位置，这些位置均匀分布在垂直于手掌的平面上形成的圆周上。所有8个相机都指向手掌中心，以确保手部位于每张渲染图像的中心。图4展示了我们渲染手部图像时的场景。 为了增加收集到的MVHM数据集的多样性，我们在Blender中随机调整光线强度和全局光照。此外，我们从网络资源中选择一些背景场景，并在渲染时随机将其用作背景。我们为构建MVHM数据集共渲染了32万张分辨率为$256 \times 256$的图像。 我们强调，MVHM中的每个样本都配有21个手部关节和2651个网格顶点的完整标注。按照[^49]中的设定，每个手指由4个关键点（掌指关节、近节指骨间关节、远节指骨间关节和指尖）完整表示。MVHM中还标注了腕掌关节。图2(a)展示了一个手部关节标签的样本。此外，我们还提供了每个样本的手部分割掩码、相机内参矩阵以及光流信息。不过，本文中我们仅使用从收集的MVHM数据集中获取的多视角和网格信息来训练手部姿态估计器。 ### 4. 方法论 #### 4.1. 概述 给定一张手部的RGB图像$\mathbf{I} \in \mathbb{R}^{W \times H \times 3}$，我们的目标是估计手部关节的3D位置$\mathbf{P}_j \in \mathbb{R}^{k \times 3}$，其中W和H分别表示图像的高度和宽度，K为手部关节的数量。近期的研究[^16] [^3]表明，在训练期间使用网格距离损失作为中间监督可以提升所学手部姿态估计器的性能。 ![Image](images/196df45b2373a86a68e6a6c6d1fadfec923b28023dfc42f72fec8b3066da02f9.jpg)
Figure 4. Synthetic scene when rendering hands in MVHM.
受[^16]中方法的启发，我们将手部网格定义为一个双向图$\mathbf{G}(\mathbf{V}, \boldsymbol{\Lambda})$，其中$\mathbf{V}$是顶点集，$\boldsymbol{\Lambda}$是邻接矩阵。我们还假设$\mathbf{V}$包含N个不同的元素$(i.e., \text{points on the mesh})$，而我们的网格估计器会预测$\mathbf{P}_m \in \mathbb{R}^{N \times 3}$中所有顶点的3D位置$\mathbf{V}$。 在我们的单视角方法中，我们使用堆叠式沙漏网络[^31]作为CNN骨干网络，从图像中提取手部特征。随后应用图卷积网络（GCN）来估计3D姿态和网格。图5展示了我们单视角网络的架构，它由三个主要组件构成：2D证据网络、网格证据网络和3D姿态估计器。这些组件将在后续小节中详细介绍。 #### 4.2. 2D证据网络 2D证据网络具有两项主要功能。首先，它估计手部关键点热图以获取2D手部关节位置；其次，它提取图像特征，并将其作为网格证据网络的输入。我们将估计出的热图记为$\mathbf{H} \in \mathbb{R}^{K \times H \times W}$。如图5所示，沙漏型骨干网络会输出两个结果，即估计出的手部关节热图和…… --- // Algorithm 1: Spin matching Algorithm for rigging hand mesh base on 3D hand pose ground truth Input: C是我们希望网格与之匹配的3D关键点真实值数组。 B是绑定系统中的手部骨骼数组。每根骨骼有两个属性：head和tail，分别表示骨骼的起始位置和结束位置。 B和C存储在一个数组中，其顺序如图2(a)和2(b)所示。 ## 开始 开始 将B[^0].tail移动到C[^0]的位置； // 旋转手掌内的骨骼 for i ∈ {1, 5, 9, 13, 17} do // bone vector u ← B[i].tail - B[i].head; v ← C[i] - C[^0]; adj ← i + 4; if i = 17 then adj ← i - 4; end // reference vector ref_ori ← B[adj].tail - B[adj].head; ref_aft ← C[adj] - C[^0]; Move B[i] to match groundturth; // sign vector e_1 = u × ref_ori × u; e_2 = v × ref_aft × v; Spin B[i] with the angle between $e_1$ & $e_2$; end for i ← 1 to 21 do if i mod 4 ≠ 1 then Move B[i] to match groundturth; B[i] performs the same spin as B[i-1]; end end 结束 提取的特征。真实值热图$\tilde{H}_s$是通过对关键点位置$k^{th}$进行高斯模糊处理得到的。为了训练2D证据网络，我们将热图损失$\mathcal{L}_h$作为监督信号应用于每个沙漏模块。热图损失的定义如下： $$ \mathcal{L}_{h}=\frac{1}{S*K}\sum_{s=1}^{S}\sum_{k=1}^{K}||\boldsymbol{H}_{k}^{s}-\bar{\boldsymbol{H}}_{k}||_{F}^{2}, $$ 其中S和K分别表示沙漏模块的数量和关键点的数量。 ![Image](images/2c7e6422dce961384e8b66557f47a20909c352c336fa917b897ed8a05f660c55.jpg)
Figure 5. Overview of our single-view method. Given a single-view RGB image, the 2D evidence network predicts its heatmap and outputs the encoded image features. The mesh evidence network takes image features as input and outputs the hand mesh. Based on the estimated mesh, the 3D pose estimator gives final hand pose prediction.
### 4.2.1 网格证据网络 我们的网格证据网络基于谱GCN[^4]构建。给定由2D证据网络提取的图像特征，网格证据网络会估算出3D手部网格。3D手部网格由一组顶点坐标$\mathbf{P}_m \in \mathbb{R}^{N \times 3}$表示，其中N为手部网格中顶点的数量。我们将手部网格表示为图$\mathbf{G}(\mathbf{V}, \mathbf{\Lambda})$，其中$\mathbf{V}$为顶点集，$\mathbf{\Lambda}$为邻接矩阵。如果顶点i和顶点j之间存在边，则$\mathbf{\Lambda}_{i,j}$为1，否则为0。 具体来说，我们首先通过图拉普拉斯运算对邻接矩阵$\Lambda$进行归一化，得到归一化的邻接矩阵$L = I - D^{-\frac{1}{2}} \Lambda D^{-\frac{1}{2}}$，其中D为图G的度矩阵，I为单位矩阵。随后利用图谱分解将归一化的邻接矩阵L分解为$U A U^{T}$，其中$A = \text{diag}(\lambda_1, \lambda_2, \ldots, \lambda_N)$包含L的特征值，$\lambda_{max}$为L的最大特征值。 遵循[^13]的方法，我们定义GCN中的卷积核$\hat{A}$如下： $$ \hat{\boldsymbol{A}}=diag(\sum_{i=0}^{S}\alpha_{i}\lambda_{1}^{i},...,\sum_{i=0}^{S}\alpha_{i}\lambda_{C}^{i}), $$ 其中$\alpha$为核参数，S为预设的超参数，用于控制感受野大小。 因此，GCN卷积运算定义为： $$ \boldsymbol{F}^{\prime}=\boldsymbol{U}\hat{\boldsymbol{\Lambda}}\boldsymbol{U}^{T}\boldsymbol{F}\boldsymbol{\theta}_{i}=\sum_{i=0}^{S}\alpha_{i}\boldsymbol{L}^{i}\boldsymbol{F}\boldsymbol{\theta}_{i}, $$ 其中$\boldsymbol{F} \in \mathbb{R}^{N \times F_{\text{in}}}$和$\boldsymbol{F}' \in \mathbb{R}^{N \times F_{\text{out}}}$分别表示输入特征和输出特征，$\boldsymbol{\theta}_i \in \mathbb{R}^{F_{\text{in}} \times F_{\text{out}}}$为可训练参数，用于优化输入特征并控制输出通道大小。 --- ![Image](images/cd3914f964a410906810beb4391a3a111d05cac5c55355360e7496b0e8aad512.jpg)
Figure 6. Overview of the multi-view method. The single-view method first predicts the hand pose for each view independently. A graph U-Net takes the concatenation of these single-view predictions as input, and estimates the final pose estimation. $N(\cdot)$ and $C(\cdot)$ represent the number of nodes in the graph and feature size of each node, respectively.
able parameter used to refine the input feature and control the output channel size. 由于我们的手部网格表面由2561个顶点组成，若对每个顶点都执行上述运算，计算成本将非常高，因为$L^i$的矩阵乘法时间复杂度为$O(N^3)$。因此，我们利用切比雪夫多项式近似来降低复杂度。此时卷积运算定义为： $$ \mathbf{F}^{\prime}=\sum_{i=0}^{S}\alpha_{i}T_{i}(\hat{\mathbf{L}})\boldsymbol{\theta}_{i}, $$ 其中$T_k(x)$为$k^{th}$切比雪夫多项式，$\hat{L} = 2L/\lambda_{max} - I$用于对输入特征进行归一化。 为了让模型能够学习局部和全局特征，我们采用了一种在 [^13] 和 [^16] 中用于从粗到细生成手部网格的方案。我们利用重边匹配算法通过三种不同的粗化层级来对图进行粗化，并记录每两个相邻层级之间图节点的映射关系。在前向传播过程中，模型首先构建最粗糙的手部网格，随后根据存储的映射关系，从粗糙图向上采样更多节点到精细图中。 在GCN的最后一层，我们将 $F_{out}$ 设为3，用以表示三维坐标顶点。此外，我们采用真实网格与预测图之间的 $l_2$ 损失作为网格损失函数： $$ \mathcal{L}_{m}=\frac{1}{N}||P_{m}-\bar{P}_{m}||_{F}^{2}. $$ #### 4.3. 三维深度证据网络 所提出的三维证据网络利用网格证据网络预测的手部网格 $P_d$，来推断三维手部关键点的深度 $P_m$。以 $P_m$ 为输入，我们采用结构与网格证据网络类似的两层GCN来预测姿态特征。这些姿态特征随后被送入两个全连接层，以回归三维手部关键点的位置深度。相应的损失函数定义为： $$ \mathcal{L}_{d}=\frac{1}{K}||\boldsymbol{D}-\boldsymbol{\bar{D}}||_{F}^{2}, $$ 其中 $D \in \mathbb{R}^K$ 和 $\bar{D} \in \mathbb{R}^K$ 分别表示预测的深度和真实的关节深度。 为了推断三维手部关键点，我们使用非极大值抑制方法从估计的热图中获取二维坐标。借助估计出的二维坐标以及三维深度证据网络计算出的深度图，我们便能得到相机坐标系下的三维坐标。由于相机参数已知，我们进而可以推断出世界坐标系中的手部关键点。 #### 4.4. 多视角方法 基于我们的单视角方法，我们提出了一种简单却有效的多视角手部姿态估计方案。图6展示了该方法的核心思想。 我们的单视角方法会独立地为每个视角预测三维手部姿态。我们将这些特定于视角的预测结果在其坐标通道上进行拼接。拼接后的预测结果作为图U-Nets[^14]的输入特征，用于预测最终的三维手部关键点。我们在多视角网络中采用 $L_{2}$ 距离作为损失函数，即： $$ \mathcal{L}_{s}=\frac{1}{K}||P_{j}-\bar{P}_{j}||_{F}^{2}, $$ 其中 $P_{j}$ 和 $\bar{P}_{j}$ 分别表示预测的深度和真实的关节深度。 ### 5. 实验设置 #### 5.1. 评估用数据集 我们在两个基准手部姿态数据集上评估我们的单视角方法，包括立体跟踪基准（STB）数据集 [^44] 和多视角三维手部姿态数据集。 --- ![Image](images/9c8eec611d80df5b77d8189f4beb0738f63eba059f5f7cf1fc44f5452825d9ab.jpg)
Figure 7. Examples of the two hand pose datasets used for evaluation. The first row shows images with the annotated hand poses from the STB dataset [^44] while the second row shows those from the MHP dataset [^17].
对于STB数据集，我们使用其中的SK子集进行评估，该子集包含6段不同的手部视频。按照 [^16] 中的训练-验证划分方式，我们将第一段视频作为验证集，其余视频则作为训练集。 (MHP) 数据集 [^17]。所提出的多视角方法在MHP数据集上进行了评估。MHP和STB数据集均提供了由不同人员在各种背景下拍摄的真实手部视频序列。STB数据集中的手部关节标注是人工标记的，而MHP数据集的标注则是通过Leap Motion传感器获取的。我们构建的MVHM数据集被用于所有的实验。我们的目标是确定使用MVHM数据集训练手部姿态估计器是否能在不同的实验设置中有效提升性能。 MHP数据集包含21个不同的手部运动视频。每个手部运动视频都提供了手部RGB图像以及每个样本中的多种标注，包括边界框和2D/3D手部关节位置。图7展示了STB和MHP数据集的一些示例。我们遵循 [^5] [^49] 的方法，对STB和MHP数据集均采用标准的数据预处理流程。在数据预处理阶段，我们首先裁剪图像以去除无关的背景，并确保手部位于图像中心。所有裁剪后的图像随后会被调整分辨率至 $256 \times 256$。其次，我们按照 [^5] 中采用的机制，将STB和MHP数据集中的手部中心点从掌骨中心改为腕关节处。 #### 5.2. 评价指标 我们参照以往研究中的设置 [^49] [^16]，采用平均端点误差（EPE $_{m}$）以及在特定阈值范围内正确关键点百分比（PCK）下的曲线下面积（AUC）作为评估模型有效性的指标。我们分别报告了PCK在0mm至50mm范围内以及20mm至50mm范围内的AUC性能。
Table 2. Ablation studies of 3D hand pose estimation on the STB and MHP datasets. $\uparrow$: higher is better; $\downarrow$: lower is better; The measuring unit of EPE is millimeter(mm). SV stands for the single-view method and MV represents the multi-view method.
AUC $_{0-50}$ $\uparrow$AUC $_{20-50}$ $\uparrow$EP $E_{m}$ $\downarrow$
MHP Dataset
SV w/o MVHM0.6040.80222.13
SV w/ MVHM0.6600.85718.09
MV w/o MVHM0.8320.9858.43
MV w/ MVHM0.8950.9905.20
STB Dataset
SV w/o MVHM0.8200.9878.95
SV w/ MVHM0.8320.9918.38
Table 3. Results on the MHP dataset. ↑: higher is better.
AUC $_{20-50}$ $\uparrow$
Zimmermann et al. [^50]0.717
Cai et al. [^5]0.928
Chen et al. [^8]0.939
Our multi-view method0.991
#### 5.3. 实现细节 我们使用Python和PyTorch实现了单视角与多视角方法。在训练阶段，我们将批量大小设为8，并采用初始学习率为0.01的Adam优化器。两种模型均在配备四块GeForce GTX 1080-Ti GPU的服务器上进行训练。 在训练单视角网络时，我们采用了多阶段训练策略。第一阶段，我们利用热图损失 $\mathcal{L}_{h}$ 训练2D证据网络。第二阶段，我们固定2D证据网络的权重，并利用网格损失 $\mathcal{L}_{m}$ 训练网格网络。第三阶段，我们固定2D证据网络和网格网络的权重，重点利用损失 $\mathcal{L}_{d}$ 训练关节深度网络。最后阶段，整个网络会进行端到端的优化。 对于多视角网络的训练，我们采用了相同的多阶段训练策略。第一阶段，我们使用单视角网络预训练的权重来初始化3D手部单视角网络，并在训练3D手部融合网络时保持该部分权重不变。第二阶段，我们激活两个网络，并以端到端的方式对整个网络架构进行微调。 ### 6. 实验结果 #### 6.1. 多视角任务 为了评估所提出多视角方法的有效性，我们在MHP数据集上比较了单视角方法与多视角方法，分别考察使用或不使用MVHM数据集的情况。表2和图8(a)表明，利用MHP数据集本身的多视角信息能够显著提升测试性能。 --- ![Image](images/bf46d4c45451865954d80f6f6f76eabba0c7c73e0afa121810c52025544e48d8.jpg) ![Image](images/d735de80752a42c7ca6fa49f752b4eedc1cebba5e02abe9a53eed46cbd773010.jpg)
(a)
![Image](images/012fc2474e9bd6375592117d7f430f57aa93eb801af29fca21a056585de70b75.jpg)
(b)
![Image](images/ff557cd45a3a52242fde69f6f2be08e9c3fe92b62dcff370999ae0c80084f74e.jpg)
(c)
(d)
Figure 8. Ablation studies and comparison of the state-of-the-art methods for single-view pose estimation. (a) PCK results of different settings on the STB dataset. (b) Comparison results in PCK for the state-of-the-art methods on the STB dataset. (c) PCK results under different settings on the MHP dataset. (d) Comparison results in PCK for the state-of-the-art methods on the MHP dataset.
该模型在$AUC_{0-50}$、$AUC_{20-50}$和$EPE_{m}$上的表现远超其他模型，具体数值分别为0.218、0.183和13.80毫米。当使用MVHM数据集中的额外数据进行训练时，模型性能进一步提升，这充分体现了利用收集到的MVHM数据集进行训练的有效性。 为了与我们的方法在MHP数据集上进行比较，我们选取了三种当前最先进的方法，包括Zimmermann等人[^50]的方法（在$AUC_{20-50}$指标下得分为0.717）、Cai等人[^5]的方法（在$AUC_{20-50}$指标下得分为0.928）$^{3}$，以及Chen等人[^8]的方法（在$AUC_{20-50}$指标下得分为0.939）。由于Zimmermann等人[^50]仅报告了数值结果，因此我们将他们的结果列入表3，而未在图8(b)中展示。我们的多视角方法在$AUC_{20-50}$指标下的得分达到了0.990，大幅超越了这些竞争方法。该实验表明，所提出的多视角方法以及已构建的MVHM数据集均具有实用价值，二者结合能够在MHP数据集上取得新的最先进性能。 #### 6.2. 单视角任务 为了进一步验证生成的网格数据集MVHM以及多视角方法的有效性，我们还开展了针对单视角方法的对比实验。我们比较了仅在MHP/STB数据集上训练的模型，以及在MHP/STB数据集基础上加入MVHM数据集训练的模型的表现。表2、图8(a)和图8(c)显示，无论是在MHP还是STB数据集上，引入网格数据都能显著提升模型性能，因为这样能使模型具备捕捉网格级特征的能力，从而获得更好的结果。 我们选取了七种性能强劲且近期发表的方法与所提方法进行比较，包括PSO [^3]、ICPPSO [^10]、CHPR [^44]、Iqbal等人[^19]的方法、Cai等人[^5]的方法、Zimmermann和Brox [^49]的方法，以及Ge等人[^16]的方法。AUC曲线如图8(d)所示。Ge等人[^16]同样利用额外的数据集训练模型并取得了最先进的结果，这证明了他们所用网格数据集的有效性。此外，他们还引入了更复杂的网格度量指标，比如表面法向损失。Iqbal等人[^19]和Cai等人[^5]则利用额外的深度图信息来构建模型，也取得了不错的效果。作为一种无需复杂组件的多视角方法，我们的方法在单视角任务中的表现与Ge等人[^16]和Iqbal等人[^19]的方法相当，且优于其中大多数方法。 ### 7. 结论 由于深度模糊性，从单目图像估计3D手部姿态是一个不适定问题。不过，多视角图像可以弥补这一缺陷。为此，我们构建了多视角网格手部数据集MVHM，以便能够利用网格监督来训练3D姿态估计器。我们提出了一种能有效融合单视角预测结果的多视角方法。在真实的多视角数据集MHP上进行测试时，借助MVHM数据集，我们的多视角方法取得了最先进的性能。 致谢。作者感谢Kun Han在生成本工作所用数据集图像方面提供的帮助。本研究部分得到了科技部（MOST）资助，项目编号为MOST 107-2628-E-009-007MY3、MOST 109-2634-F-007-013和MOST 109-2221-E-009-113-MY3，同时也得到了高通公司通过台湾大学研究合作项目的支持。 --- ## 参考文献 [^1]: Shamama Anwar, Subham Kumar Sinha, Snehanshu Vivek, and Vishal Ashank. Hand gesture recognition: a survey. In Nanoelectronics, Circuits and Communication Systems. 2019. [^2]: Seungryul Baek, Kwang In Kim, and Tae-Kyun Kim. Pushing the envelope for rgb-based dense 3d hand pose estimation via neural rendering. In CVPR, 2019. [^3]: Adnane Boukhayma, Rodrigo de Bem, and Philip HS Torr. 3d hand shape and pose from images in the wild. In CVPR, 2019. [^4]: Joan Bruna, Wojciech Zaremba, Arthur Szlam, and Yann Le-Cun. Spectral networks and locally connected networks on graphs. arXiv preprint arXiv:1312.6203, 2013. [^5]: Yujun Cai, Liuhao Ge, Jianfei Cai, and Junsong Yuan. Weakly-supervised 3d hand pose estimation from monocular rgb images. In ECCV, 2018. [^6]: Zhe Cao, Gines Hidalgo, Tomas Simon, Shih-En Wei, and Yaser Sheikh. OpenPose: realtime multi-person 2D pose estimation using Part Affinity Fields. In TPAMI, 2018. [^7]: Tejo Chalasani and Aljosa Smolic. Simultaneous segmentation and recognition: Towards more accurate ego gesture recognition. In ICCVW, 2019. [^8]: Liangjian Chen, Shih-Yao Lin, Yusheng Xie, Yen-Yu Lin, Wei Fan, and Xiaohui Xie. Dggan: Depth-image guided generative adversarial networks for disentangling rgb and depth images in 3d hand pose estimation. In WACV, 2020. [^9]: Liangjian Chen, Shih-Yao Lin, Yusheng Xie, Yen-Yu Lin, and Xiaohui Xie. Temporal-aware self-supervised learning for 3d hand pose and mesh estimation in videos. In WACV, 2021. [^10]: Liangjian Chen, Shih-Yao Lin, Yusheng Xie, Hui Tang, Yufan Xue, Yen-Yu Lin, Xiaohui Xie, and Wei Fan. Tagan: Tonality-alignment generative adversarial networks for realistic hand pose synthesis. In BMVC, 2019. [^11]: Liangjian Chen, Shih-Yao Lin, Yusheng Xie, Hui Tang, Yufan Xue, Xiaohui Xie, Yen-Yu Lin, and Wei Fan. Generating realistic training images based on tonality-alignment generative adversarial networks for hand pose estimation. arXiv preprint arXiv:1811.09916, 2018. [^12]: Ulysse Côté-Allard, Cheikh Latyr Fall, Alexandre Drouin, Alexandre Campeau-Lecours, Clément Gosselin, Kyrre Glette, François Laviolette, and Benoit Gosselin. Deep learning for electromyographic hand gesture signal classification using transfer learning. IEEE Trans Neural Systems and Rehabilitation Engineering, 27(4):760–771, 2019. [^13]: Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional neural networks on graphs with fast localized spectral filtering. In NeurIPS, 2016. [^14]: Hongyang Gao and Shuiwang Ji. Graph u-nets. arXiv preprint arXiv:1905.05178, 2019. [^15]: Liuhao Ge, Hui Liang, Junsong Yuan, and Daniel Thalmann. Robust 3d hand pose estimation in single depth images: from single-view cnn to multi-view cnns. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3593–3601, 2016. [^16]: Liuhao Ge, Zhou Ren, Yuncheng Li, Zehao Xue, Yingying Wang, Jianfei Cai, and Junsong Yuan. 3d hand shape and pose estimation from a single rgb image. In CVPR, 2019. [^17]: Francisco Gomez-Donoso, Sergio Orts-Escolano, and Miguel Cazorla. Large-scale multiview 3d hand pose dataset. arXiv preprint arXiv:1707.03742, 2017. [^18]: Yi-Ping Hung and Shih-Yao Lin. Re-anchorable virtual panel in three-dimensional space, Dec. 27 2016. US Patent 9,529,446. [^19]: Umar Iqbal, Pavlo Molchanov, Thomas Breuel Juergen Gall, and Jan Kautz. Hand pose estimation via latent 2.5 d heatmap regression. In ECCV, 2018. [^20]: MR Mohamad Ismail, CK Lam, K Sundaraj, and MHF Rahiman. Hand motion pattern recognition analysis of forearm muscle using mmg signals. Bulletin of Electrical Engineering and Informatics, 8(2):533–540, 2019. [^21]: Hanbyul Joo, Tomas Simon, and Yaser Sheikh. Total capture: A 3d deformation model for tracking faces, hands, and bodies. In CVPR, 2018. [^22]: Mia Kokic, Danica Kragic, and Jeannette Bohg. Learning to estimate pose and shape of hand-held objects from RGB images. arXiv preprint arXiv:1903.03340, 2019. [^23]: Deying Kong, Yifei Chen, Haoyu Ma, Xiangyi Yan, and Xiaohui Xie. Adaptive graphical model network for 2d hand-pose estimation. arXiv preprint arXiv:1909.08205, 2019. [^24]: Deying Kong, Haoyu Ma, and Xiaohui Xie. Sia-gcn: A spatial information aware graph neural network with 2d convolutions for hand pose estimation. arXiv preprint arXiv:2009.12473, 2020. [^25]: Dominik Kulon, Riza Alp Guler, Iasonas Kokkinos, Michael M Bronstein, and Stefanos Zafeiriou. Weakly-supervised mesh-convolutional hand reconstruction in the wild. In CVPR, pages 4990–5000, 2020. [^26]: Shih-Yao Lin, Yun-Chien Lai, Li-Wei Chan, and Yi-Ping Hung. Real-time 3d model-based gesture tracking for multimedia control. In ICPR, 2010. [^27]: Shih-Yao Lin, Chuen-Kai Shie, Shen-Chi Chen, and Yi-Ping Hung. Airtouch panel: a re-anchorable virtual touch panel. In MM, 2013. [^28]: Gyeongsik Moon, Shoou-I Yu, He Wen, Takaaki Shiratori, and Kyoung Mu Lee. Interhand2.6m: A dataset and baseline for 3d interacting hand pose estimation from a single rgb image. [^29]: Franziska Mueller, Florian Bernard, Oleksandr Sotny-chenko, Dushyant Mehta, Srinath Sridhar, Dan Casas, and Christian Theobalt. Generated hands for real-time 3d hand tracking from monocular rgb. In CVPR, 2018. [^30]: Franziska Mueller, Dushyant Mehta, Oleksandr Sotnychenko, Srinath Sridhar, Dan Casas, and Christian Theobalt. Real-time hand tracking under occlusion from an egocentric rgb-d sensor. In ICCVW, 2017. [^31]: Alejandro Newell, Kaiyu Yang, and Jia Deng. Stacked hour-glass networks for human pose estimation. In ECCV, pages 483–499, 2016. ## [^32]: Javier Romero, Dimitrios Tzionas, and Michael J Black. Embodied hands: Modeling and capturing hands and bodies together. ACM Transactions on Graphics, 36(6):245, 2017 --- [^33]: Tomas Simon, Hanbyul Joo, Iain Matthews, and Yaser Sheikh. Hand keypoint detection in single images using multiview bootstrapping. In CVPR, 2017. [^34]: Srinath Sridhar, Franziska Mueller, Michael Zollhöfer, Dan Casas, Antti Oulasvirta, and Christian Theobalt. Real-time joint tracking of a hand manipulating an object from rgb-d input. In ECCV, 2016. [^35]: Xiao Sun, Yichen Wei, Shuang Liang, Xiaoou Tang, and Jian Sun. Cascaded hand pose regression. In CVPR, 2015. [^36]: Danhang Tang, Hyung Jin Chang, Alykhan Tejani, and Tae-Kyun Kim. Latent regression forest: Structured estimation of 3d articulated hand posture. In CVPR, 2014. [^37]: Bugra Tekin, Federica Bogo, and Marc Pollefeys. H+o: Unified egocentric recognition of 3d hand-object poses and interactions. CVPR, 2019. [^38]: Jonathan Tompson, Murphy Stein, Yann Lecun, and Ken Perlin. Real-time continuous pose recovery of human hands using convolutional networks. ACM Transactions on Graphics, 33, 2014. [^39]: Zhenyu Wu, Duc Hoang, Shih-Yao Lin, Yusheng Xie, Liangjian Chen, Yen-Yu Lin, Zhangyang Wang, and Wei Fan. Mm-hand: 3d-aware multi-modal guided hand generative network for 3d hand pose synthesis. In MM, 2020. [^40]: John Yang, Hyung Jin Chang, Seungeui Lee, and Nojun Kwak. Seqhand: RGB-sequence-based 3d hand pose and shape estimation. arXiv preprint arXiv:2007.05168, 2020. [^41]: Linlin Yang and Angela Yao. Disentangling latent hands for image synthesis and pose estimation. CVPR, 2019. [^42]: Shanxin Yuan, Qi Ye, Bjorn Stenger, Siddhant Jain, and Tae-Kyun Kim. Bighand2.2m benchmark: Hand pose dataset and state of the art analysis. In CVPR, 2017. [^43]: Zahooor Zafrulla, Helene Brashear, Thad Starner, Harley Hamilton, and Peter Presti. American sign language recognition with the kinect. In ICMI, 2011. [^44]: Jiawei Zhang, Jianbo Jiao, Mingliang Chen, Liangqiong Qu, Xiaobin Xu, and Qingxiong Yang. 3d hand pose tracking and estimation using stereo matching. arXiv preprint arXiv:1610.07214, 2016. [^45]: Xiong Zhang, Qiang Li, Wenbo Zhang, and Wen Zheng. End-to-end hand mesh recovery from a monocular rgb image. arXiv preprint arXiv:1902.09305, 2019. [^46]: Zhengli Zhao, Sameer Singh, Honglak Lee, Zizhao Zhang, Augustus Odena, and Han Zhang. Improved consistency regularization for gans. arXiv preprint arXiv:2002.04724, 2020. [^47]: Zhengli Zhao, Samarth Sinha, Anirudh Goyal, Colin Raffel, and Augustus Odena. Top-k training of gans: Improving gan performance by throwing away bad samples. arXiv preprint arXiv:2002.06224, 2020. [^48]: Zhengli Zhao, Zizhao Zhang, Ting Chen, Sameer Singh, and Han Zhang. Image augmentations for gan training. arXiv preprint arXiv:2006.02595, 2020. [^49]: Christian Zimmermann and Thomas Brox. Learning to estimate 3d hand pose from single rgb images. In CVPR, 2017. [^50]: Christian Zimmermann, Duygu Ceylan, Jimei Yang, Bryan Russell, Max Argus, and Thomas Brox. Freihand: A dataset for markerless capture of hand pose and shape from single rgb images. In ICCV, 2019.