# I2UV-HandNet：用于精确且高保真3D手部网格建模的图像到UV预测网络 陈平 $^{1}$ 陈宇金 $^{2,3}$ 杨东 $^{1}$ 吴方银 $^{1}$ 李琴 $^{1}$ 夏庆佩 $^{1}$ 谭勇 $^{1}$ $^{1}$爱奇艺公司 $^{2}$武汉大学 $^{3}$慕尼黑工业大学 {redcping, terencecyj}@gmail.com {yangdong01, wufangying, liqin01, xiaqingpei, tanyong}@qiyi.com ![Image](images/0db03472b05626e3e2508f3502135e63f2038745afd667148c528b321d5360a6.jpg) 输入 粗糙网格 高保真网格 输入 图1：我们提出了一种新颖的图像到UV预测网络（I2UV-HandNet），能够从单张RGB图像中估算出精确且高保真的手部网格。此处展示了在HO-3D数据集（左）和FreiHAND数据集（右）上的示例结果。从左到右依次为：输入图像、估算出的粗糙网格、估算出的高保真网格（两个视角）。 粗糙网格 高保真网格 ## 摘要 从彩色图像中重建高精度、高保真的3D人手模型，在人机交互和虚拟现实应用中实现逼真的虚拟手方面发挥着核心作用。由于手部姿态多样且存在严重遮挡现象，现有方法的结果在精度和保真度上仍有不足。在本研究中，我们提出了一种I2UV-HandNet模型，用于精确的手部姿态与形状估计以及3D手部超分辨率重建。具体而言，我们首次提出了基于UV的3D手部形状表示方法。为了从RGB图像中恢复3D手部网格，我们设计了AffineNet，以图像到图像转换的方式从输入图像中预测UV位置图。为了获得更高保真度的形状，我们利用额外的SRNet将AffineNet输出的低分辨率UV图转换为高分辨率UV图。我们首次验证了基于UV的手部形状表示所具有的特征刻画能力。我们的实验表明，所提出的方法在多个具有挑战性的基准测试中均取得了最先进的性能。 ments show that the proposed method achieves state-of-the-art performance on several challenging benchmarks. ### 1. 引言 观察和理解人手一直是计算机视觉与人机交互领域的重要任务，其应用涵盖手势识别到增强现实（AR）和虚拟现实（VR）等多个领域。近年来，得益于大规模数据收集与标注工作的推进[^5] [^7] [^8]，以及3D表示与学习方法的发展[^10] [^14]，3D手部姿态与形状估计领域取得了显著进展[^36] [^52] [^49] [^46] [^52] [^53] [^6] [^33]。这使得从单视图彩色图像中理解3D手部信息取得了令人瞩目的进步。 由于缺少手部表面数据，早期的大多数研究通过从单张图像中估算3D关节位置来开展3D姿态估计工作[^3] [^7] [^20] [^37] [^52]。然而，这种稀疏的关节表示无法满足相关需求。 --- 在许多应用场景中，比如在沉浸式虚拟现实场景中让虚拟手与物体交互[^19]。为了更好地显示手部表面，以往的方法会回归参数化手部模型（MANO）[^34]，该模型具有关节运动和非刚性变形特性[^5] [^15] [^26] [^31] [^48]。虽然利用CNN从RGB输入预测MANO参数，并通过3D标注来监督这一回归过程较为简单[^17] [^53]，但这种高维非线性回归限制了重建手部的精度。随后，基于回归的方法引入了各种中间表示来指导训练过程。在这些方法中，3D手部重建分为两个阶段：首先回归一组中间表示，如2D关键点、掩码或3D关键点，然后根据这些中间表示预测模型参数[^48] [^50]。这些方法的性能很大程度上取决于这些中间表示的设计以及合理监督项的使用。最近，[^14] [^27] 消除了对参数化模型先验的依赖，直接回归网格顶点的3D坐标。尽管这些方法表现良好，但上述那些从高维编码特征估计模型参数或顶点坐标的方法，破坏了原始像素空间中包含的空间关系。[^29] 提出为每个网格顶点坐标预测一个1D热图，从而取得了最先进的性能，但它仅在特征变换过程中保留了空间信息，而其逐顶点输出仍是离散的3D表示。与上述3D表示和学习方法不同，我们在本工作中提出采用UV位置图[^13]作为手部表示。 受近期将人体3D网格映射为UV图表示的3D人体恢复方法的启发[^1] [^43]，我们提出在UV空间中表示3D手部表面，并训练神经网络从单个RGB输入预测3D手部形状。采用UV表示能使高效网络直接回归手部表面，无需依赖任何模型先验或中间表示。为了能从RGB输入正确预测UV位置图，我们提出了AffineNet，它解决了图像到图像转换任务中的单视图3D重建问题。传统的图像到图像转换流程是为输入与输出之间具有良好空间对齐的任务（如外观转换或语义分割）设计的[^41] [^51]。然而在我们的设置中，UV位置图所显示的手部形状与输入RGB图像中的形状不同。为解决这一问题，我们提出了一种新颖的仿射连接模块，用于将编码后的特征图与UV图对齐，再将对齐后的特征图与解码后的特征图相连。在AffineNet中，采用了分层的UV位置图和多级特征图，且在训练阶段可对多个UV图进行监督。对于3D姿态估计，我们通过预定义的映射从输出的手部网格中获取一组3D关键点。 基于UV表示的另一个优势在于，密集的UV位置图能够在UV位置图的有效区域内进行采样，从而重建出包含更多顶点的3D表面。受此启发，我们提出了一种基于UV的3D手部超分辨率重建模块SRNet，用于从粗糙的3D手部形状中实现高保真度的3D手部重建。为了充分利用所提出的手部UV位置图表示法，我们采用卷积神经网络将低分辨率的UV位置图映射为高分辨率图像，进而恢复出高保真度的手部形状。然而，目前缺乏高保真度的手部表面数据来指导SRNet的学习过程。因此，我们构建了一个名为SuperHandScan的扫描数据集以训练SRNet。我们将高质量的3D手部扫描数据与配准后的粗糙MANO模型转换为高/低分辨率的UV位置图，再利用这些UV图来训练SRNet。由于SRNet的输入是基于UV表示的粗糙手部网格，因此SRNet具有广泛的应用范围；换句话说，训练有素的SRNet可用于任意粗糙手部网格的网格超分辨率重建。 综上所述，我们提出了I2UV-HandNet模型，该模型由用于3D手部姿态和形状估计的AffineNet以及用于手部网格超分辨率重建的SRNet组成。总体而言，本文的主要贡献可总结如下： - 据我们所知，我们是首个在3D手部姿态和形状估计中引入UV图表示法的研究团队。基于这一新颖的表示方法，我们提出了一种名为AffineNet的端到端网络，能够从单张彩色图像中预测手部网格。 在手部重建领域，我们首次提出了SRNet——一种手部网格超分辨率重建网络，能够从粗糙的手部网格预测出高保真度的手部网格。 - 我们的方法能够从RGB输入中预测出精确且高保真度的手部网格。实验结果表明，在多个具有挑战性的数据集上，我们的方法均优于其他最先进的方法。 ### 2. 相关工作 本节将介绍有关3D手部姿态估计、手部姿态与形状恢复以及密集形状表示的相关研究。下文我们将把我们的成果与以往的工作进行对比。 #### 2.1. 3D手部姿态估计 3D手部姿态估计的任务旨在预测手部关节的3D位置。近年来，基于深度图像或RGB图像的3D手部姿态估计已成为研究热点。关于基于深度的手部姿态估计的相关工作，请参阅 --- ![Image](images/1c664ae6a21f7bfff6dde8df60f778fb4dc45dbea27f7d01369bb5e1962fafff.jpg)
Figure 2: Overview of the proposed framework. The proposed I2UV-HandNet model, which consists of the AffineNet and the SRNet, enables to predict high-precision and high-fidelity hand meshes from a single-view color image.
[^2]: [^44]. Here, we mainly compare RGB-based hand pose estimation methods. 由于从2D图像中直接获取3D关节标注较为困难，许多方法利用3D关节与其2D投影之间的对应关系来提升3D姿态估计的效果。[^36]提出先检测多视角彩色图像中的2D手部关键点，再利用多视角几何原理将这些2D检测结果三角化得到3D坐标。随着众多带有3D标注的数据集的出现[^46] [^52]，研究人员开始探索利用3D标注来监督网络训练，从而直接从RGB图像中回归3D关节坐标的方法[^52]。此后，许多方法沿用这一策略，并通过引入中间表示[^20]或增加更多的监督项[^7] [^37]来进一步提升性能。最近，[^29]中将3D关节表示为三个一维热图，其效果同样令人信服。在本文中，我们仅利用输出的3D关节数据来评估手部网格建模的性能。 #### 2.2. 手部姿态与形状估计 由于稀疏关键点对三维形状的表征能力有限，近期的研究将稀疏姿态估计与密集形状重建相结合，以提供更完整的形状表征。该领域的方法可分为两类：一类使用形状先验模型，另一类则不使用。 为了解决二维图像缺乏足够的深度信息和形状知识从而导致无法恢复三维形状的问题，基于三维扫描数据构建的参数化形状模型（如用于人脸的3DMM [^4]、用于人体的SMPL [^28]以及用于手部的MANO [^34]）利用低维参数来表征复杂的三维表面。近期的研究 [^5] [^15] [^25] [^39] [^48] 将基于参数的手部模型（MANO）与端到端深度网络相结合，用于手部姿态和形状估计。其基本思路是从输入图像中回归出MANO参数，再根据这些回归出的参数恢复三维关节和形状；其中三维关节、网格或拟合的MANO参数被用于指导网络训练 [^5] [^17] [^48] [^50]。还有一些方法试图去除三维监督信息 [^9] [^37]，或从三维姿态中恢复三维形状 [^11]。 尽管参数化模型能提供三维形状先验，但从RGB图像中估计模型参数会破坏二维像素之间的空间关系。为解决这一问题，I2L-MeshNet [^29] 预测每个网格顶点坐标的一维热图，而非直接回归参数。[^14] 和 [^27] 通过图卷积网络（GCN）回归每个顶点的位置。与他们用三维空间表示三维手部的方式不同，我们通过二维UV贴图来表示三维手部的表面，这种贴图可以像 [^21] 那样通过图像到图像的转换方式从输入图像中映射得到。 关于输出网格的顶点数量，[^27] 和 [^29] 使用了778个顶点，且采用与MANO相同的网格拓扑结构；而 [^14] 通过GCN输出包含1,280个顶点的网格。DeepHandMesh [^30] 能回归出包含12,553个顶点的高保真网格，但它需要在受控环境下采集的数据上进行训练（或预训练），并且由于它是通过参数来表征手部，仍面临高度非线性的回归问题。在这项工作中，我们提出了一种更通用的高保真手部重建方案：输入低分辨率的UV位置图（MANO级别的手部网格），输出高分辨率的UV位置图（高保真手部网格）。 #### 2.3. 密集形状表征 尽管通过参数化模型或三维三角网格来表示三维形状既简单又便于进行监督学习，但有一些研究[^1] [^13] [^43] [^45] [^47]提出以更密集的方式表示三维表面，即引入UV表示来刻画图像到表面的对应关系，进而可以直接利用强大的二维CNN来学习图像到UV的映射。UV表示可分为IUV和UV位置图：在IUV与RGB图像中相同位置具有空间一致性，而UV位置图则与RGB图像不一致。我们将这种不一致性称为坐标歧义。IUV表示已被应用于单视图三维人脸重建[^13]以及人体重建[^42] [^43]。近期，[^1] [^45]结合IUV、UV位置图和SMPL模型[^28]来重建三维人体。尽管这些方法取得了不错的效果，但 --- ![Image](images/69a6a11e523a98d24c144e0355578a9237eedab33fd43c5dbe6a008002fc1e2a.jpg)
Figure 3: Three UV unfolding forms in terms of different cutting and combination strategies.
RGB图像与UV位置图之间的坐标歧义问题并未得到妥善解决。在本文中，我们首次将UV位置图引入手部重建任务，并提出通过仿射连接模块（第3.2.1节）来消除这种坐标歧义。 ### 3. 方法 所提出的I2UV-HandNet模型能够通过UV位置图来表示三维手部表面（第3.1节），并以图像到图像的转换方式从单视图图像中学习估计精确且高保真度的手部形状（第3.2节和第3.3节）。接下来，我们将详细介绍所提出的方法。 #### 3.1. 三维手部表示 参数化手部模型。MANO是一种基于手部扫描数据训练得到的参数化手部模型。它定义了姿态参数和形状参数到包含778个顶点和1538个面的网格的映射关系，其中网格拓扑是固定的，用于指示手部表面顶点的连接方式。根据给定的网格拓扑，可以直接从手部网格中推导出一组共21个关节。在此，我们使用MANO模型来推断16个关节，并根据预定义的顶点索引获取5个指尖位置[^17]。 将手部表面表示为UV位置图。给定手部表面（如MANO手部网格），我们可以将其展开为一张UV图$^{1}$，从而将三维表面表示为图像。此处，U和V分别表示图像的两个坐标轴。UV映射定义了网格顶点与图像像素之间的对应关系。图3展示了三种不同的切割与组合策略所对应的UV映射形式，第4.6节对其进行了消融对比实验。在我们的流程中，AffineNet直接从输入图像输出UV位置图，SRNet则从输入的UV位置图生成另一张UV位置图，随后通过上述定义的UV映射从UV位置图中恢复出三维手部网格。为了训练模型，手部网格标注会被转换为UV图，用以监督UV图的预测。具体而言，使用正交投影将手部网格与对应的RGB图像进行空间对齐，使三维手部网格与图像平面上的二维手部相匹配。对于网格上的每个三维顶点，其三维坐标会被映射为UV位置图中某个点的RGB通道值[^43]。通过插值处理可生成连续的图像。 #### 3.2. I2UV-HandNet 如图2所示，所提出的I2UV-HandNet模型通过AffineNet预测UV位置图，再利用SRNet恢复高分辨率的UV位置图，从而实现精确且高保真度的手部形状估计。 #### 3.2.1 AffineNet 为了预测手部的UV位置图，采用编码器-解码器机制将输入图像映射为UV图像。与U-net[^35]类似，AffineNet由收缩路径和扩张路径组成（如图2所示）。给定包含手部的彩色图像I，使用ResNet-50骨干网络[^18]将图像编码为一系列不同分辨率的编码特征$\{E^i | i = 1, 2, 3, 4, 5\}$。在扩张路径中，每一步都会对特征图和UV图预测进行上采样，同时利用对应的编码特征图，最终得到一系列解码特征$\{D^i | i = 0, 1, 2, 3, 4\}$以及预测的UV位置图$\{I_{UV}^i | i = 0, 1, 2, 3, 4\}$： $$ \left\{\begin{array}{l}D^{4}=f_{u p}(E^{5})\\ I_{U V}^{4}=f_{c o n}(D^{4})\end{array}\right. $$ 和 $$ \left\{\begin{array}{l}A^{3}=f_{u p}(f_{a c}(\pi(I_{U V}^{4}),E^{4}))\\ D^{3}=f_{u p}(D^{4}),\\ I_{U V}^{3}=f_{c o n}(A^{3},D^{3},f_{u p}(I_{U V}^{4})),\end{array}\right. $$ 和 $$ \left\{\begin{array}{l}A^{i}=f_{u p}(f_{a c}(\pi(I_{U V}^{i+1}),E^{i+1})),\\ D^{i}=f_{u p}(f_{c o n}(D^{i+1},A^{i+1},I_{U V}^{i+1})),\quad i=0,1,2\\ I_{U V}^{i}=f_{c o n}(A^{i},D^{i},f_{u p}(I_{U V}^{i+1})),\end{array}\right. $$ 此处，$E^i$表示第i个金字塔层级编码的特征图，$A^i$表示经仿射变换对齐后的特征，$D^i$表示特征图，$f_{up}$表示$2 \times$上采样操作，$f_{ac}$表示仿射连接操作，$f_{con}$表示卷积层，$\pi$表示从UV位置图到图像坐标系的投影。i值越小，分辨率越高。我们注意到，仿射连接$f_{ac}$会在连接编码特征和解码特征之前，通过仿射操作将它们对齐；这种仿射操作类似于STN[^22]，是基于当前预测的UV图中每个顶点坐标的二维投影实现的。更多细节详见附录。 --- #### 3.2.2 SRNet 由于我们的3D手部表面是由UV位置图表示的，因此我们提出通过在UV图像空间中进行超分辨率处理来获得更精细的手部表面。我们提出了SRNet，用于将低分辨率的UV位置图转换为高分辨率图。SRNet的网络架构与超分辨率卷积神经网络（SRCNN）[^12]类似，但输入和输出均为UV位置图而非RGB图像。通过SRNet回归出高分辨率的UV位置图后，便可重建出保真度更高的手部网格（如图2右侧所示）。关于SRNet架构的更多细节详见附录。 #### 3.3. 训练目标 #### 3.3.1 AffineNet的损失函数 为了训练AffineNet，我们引入了UV对齐$E_{UV}$、UV梯度对齐$E_{grad}$以及网格对齐$E_{verts}$约束： $$ E_{a f f i n e}=\lambda_{1}E_{U V}+\lambda_{2}E_{g r a d}+\lambda_{3}E_{v e r t s} $$ UV对齐。我们提出了基于真实UV位置图$E_{UV}$与输出UV位置图$\hat{I}_{UV}$之间的L1距离的UV对齐损失$I_{UV}$： $$ E_{U V}=\left|\left(I_{U V}-\hat{I}_{U V}\right)\cdot M\right| $$ 此处，M是UV图掩码，因为只有UV位置图的有效区域才对应3D手部表面的相应区域。 UV梯度对齐。理想的手部表面应当是连续的，UV位置图也是如此。为此，我们引入UV梯度对齐机制，以促使预测的UV位置图与真实UV位置图具有相同的梯度： $$ \begin{aligned}\boldsymbol{E}_{grad}=&\left|\partial_{u}(I_{UV}\cdot M)-\partial_{u}(I_{UV}^{*}\cdot M)\right|\\+&\left|\partial_{v}(I_{UV}\cdot M)-\partial_{v}(I_{UV}^{*}\cdot M)\right|\end{aligned} $$ 其中，$\partial_{u}$和$\partial_{v}$分别是沿U轴和V轴的梯度。 网格对齐。除了用于衡量二维UV位置图空间中形状重建效果的$E_{UV}$和$E_{grad}$之外，我们还引入了网格对齐损失$E_{verts}$，以强制使预测出的3D手部网格与真实网格保持一致： $$ E_{verts}=\frac{1}{N_{vert}}\sum_{i=1}^{N_{vert}}\left|v_{i}-\hat{v}_{i}\right| $$ 此处，v_{i}和\hat{v}_{i}分别是输出网格与真实网格中第i个顶点的3D坐标，N_{vert}表示网格的顶点数量。 由于存在多个预测的UV位置图，我们会对多尺度UV图应用$E_{affine}$。在训练AffineNet时，会采用最后四个UV图，且赋予它们相同的权重。 #### 3.3.2 SRNet的损失函数 SRNet输出的UV位置图与AffineNet的输出类似，只不过SRNet能生成分辨率更高的UV图。在此，我们采用了与AffineNet类似的损失函数。 $$ E_{SR}=E_{UV\_{S}R}+E_{verts\_{S}R} $$ 在此，我们将$E_{UV}$中的对应分量替换为SRNet中的相应分量，从而构建出$E_{UV\_SR}$；例如，用分辨率更高的UV位置图替换原有的UV位置图。此外，$E_{verts\_SR}$也是按照相同方式构建的。 ### 4. 实验 在本节中，我们首先介绍数据集（4.1节）、评估指标（4.2节）以及实现细节（4.3节）。随后，将展示所提方法的整体性能并进行全面分析（4.4节、4.5节和4.6节）。 #### 4.1. 数据集 FreiHAND。FreiHAND数据集[^53]包含各种姿态、物体交互以及不同光照条件下的真实手部数据。该数据集包含130,240个训练样本和3,960个测试样本。每个训练样本包含单视角RGB图像、基于MANO模型的3D手部关节与网格标注，以及相机位姿参数。测试集的结果需通过在线提交系统$^{2}$进行评估。 HO3D。HO3D数据集[^15]是近期发布的数据集，收录了手部与物体交互的彩色图像。该数据集包含66,034个训练样本，涵盖单视角RGB图像、基于MANO模型的手部关节与网格，以及相机位姿信息。测试集则提供了11,524张RGB图像及检测边界框的标注。该数据集中的物体主要源自TCB-Video数据集[^11]。测试集的结果需通过其在线提交系统$^{3}$进行评估。 ObMan。ObMan数据集[^17]是一个大规模的合成数据集，包含手部与物体交互的图像。它包含141,550个训练样本、6,463个验证样本以及6,285个测试样本。每个样本都包含RGB-D --- 图像、3D手部关节、3D手部网格、物体网格以及相机位姿参数。 YT-3D。YouTube-3D-Hands（YT-3D）数据集收录了来自YouTube的各种真实手部图像，并通过自动化采集系统[^27]对这些图像进行标注。训练集由选定的102个视频生成，包含47,125张带有3D关节与网格标注的手部图像。验证集和测试集分别涵盖7个视频，各包含1,525个样本。 SHS。我们通过激光扫描仪收集高质量的三维手部扫描数据，以此构建SuperHandScan（SHS）数据集。之所以开发SHS数据集，是因为MANO手部网格仅通过778个顶点（含1538个面）来表征手部表面，所能呈现的表面信息较为粗糙；而基于UV的方法则能生成细节更丰富的手部表面。因此，借助一系列手部扫描数据以及给定的MANO模型，我们获得了分辨率高出三倍的手部网格。具体而言，我们首先采用基于边的反池化方法，将原始的MANO手部网格从778个顶点（含1538个面）上采样至3093个顶点（含6152个面），详见[^40]。随后，利用迭代最近点（ICP）算法将上采样后的三维网格与扫描仪生成的三维点云进行配准。我们的SHS数据集包含6000组带有密集三维点云及对应手部网格的扫描数据。这种拥有3093个顶点和6152个面的手部网格比MANO手部网格更密集，因此能够辅助SRNet学习到质量更高的手部网格。HIC。Hands in Action数据集（HIC）[^38]收录了手部与物体交互的图像。每个样本都包含RGB-D图像、三维物体形状以及贴合MANO模型的手部形状。我们利用所有样本来评估SRNet的性能。 #### 4.2. 评估指标 为了评估所提方法的性能，我们采用了多种指标来衡量手部姿态估计与网格重建效果。姿态误差用于衡量预测结果与真实三维关节之间的平均欧几里得距离；网格误差则用于衡量预测网格顶点与真实网格顶点之间的平均欧几里得距离。姿态AUC表示正确关键点百分比（PCK）曲线下的面积，网格AUC则表示正确顶点百分比（PCV）曲线下的面积。我们还对比了F分数[^24]，它是在给定距离阈值下两个网格之间召回率与精度的调和平均值。我们分别通过F@5mm和F@15mm来报告5毫米和15毫米距离阈值下的网格顶点F分数。遵循近期相关研究的方法，我们将对齐后的预测结果与Procrustes对齐结果进行对比。 我们采用PSNR和RMSE来评估手部超分辨率重建的性能。PSNR用于计算两幅图像之间的峰值信噪比（单位为分贝），作为原始图像与重建图像之间的质量衡量标准；PSNR值越高，说明重建表面的质量越好。RMSE（均方根误差）则是残差的标准差。在本研究中，我们利用PSNR和RMSE来评估渲染深度图与对应真实值之间的差异。 #### 4.3. 实现细节 我们在四块NVIDIA Tesla V100 GPU上训练模型。我们使用Adam优化器[^23]来优化网络，并采用PyTorch框架[^32]进行实现。所提出的模型由两部分组成，即AffineNet和SRNet：其中AffineNet旨在MANO模型层面重建手部网格，SRNet则用于预测细节更丰富的手部网格。没有哪套图像集能同时提供MANO级别和超MANO级别的网格标注，因此我们采用分阶段训练策略，针对不同模块使用不同的数据进行监督优化——即利用图像数据集（如FreiHAND）训练AffineNet，同时利用扫描数据（如SHS）训练SRNet。 The AffineNet is trained for 200 epochs with the batch size of 128 and the learning rate initialized to $1 \times 10^{-4}$ and changed according to a Cosine Learning rate decay. The input image is cropped to $3 \times 256 \times 256$. During its training, the input image is augmented by scaling, rotation and, color channel permutation. The SRNet is trained for 100 epochs with the batch size of 512 and the learning rate is set to $1 \times 10^{-3}$. The input and output UV position map of SRNet is $3 \times 256 \times 256$. For each sample in the SHS dataset, there are two sources of SRNet's input, one is the UV of the corresponding MANO mesh, and the other is the UV map after Gaussian smoothing. #### 4.4. Comparison with State-of-the-art Methods Since our SRNet part is trained by scanning data, we only compare the results of AffineNet with other methods to ensure the fairness of the comparison. We compare the proposed method with several state-of-the-art approaches [^5] [^11] [^16] [^27] [^29] [^53] for hand pose and shape estimation on the FreiHAND dataset. The results are shown in Table 1. In general, methods without directly regressing MANO parameters ([^11] [^27] [^29] and our method) perform better than methods using MANO parameters regression ([^5] [^17] [^53]). When no extra training data is used, our method surpasses all previous methods whether or not they use MANO model prior. When the additional training data (the training set of ObMan and YT-3D) is used, our method achieves better performance (see “Ours* (AffineNet)” in the table). We further plot the 3D PCK and PCV of the FreiHAND test set compared with some state-of-the-art methods [^27] [^29] [^53], where our method shows better performance. In the hand-object interaction scenario, we compare with ---
MethodPose Error $\downarrow$Pose AUC $\uparrow$Mesh Error $\downarrow$Mesh AUC $\uparrow$F@5 mm $\uparrow$F@15 mm $\uparrow$
Boukhayma et al. [^5]3.500.3511.320.7380.4270.894
Zimmermann et al. [^53] (Mean Shape)1.710.6621.640.6740.3360.837
Zimmermann et al. [^53] (Mano Fit)1.370.7301.370.7290.4390.892
Hasson et al. [^16]1.330.7371.330.7360.4290.907
Spurr et al. [^37]1.130.78----
Zimmermann et al. [^53] (MANO CNN)1.100.7831.090.7830.5160.934
Kulon et al. [^27]0.840.8340.860.8300.6140.966
Choi et al. [^11]0.77-0.78-0.6740.969
Moon et al. [^29]0.740.8540.760.8500.6810.973
Ours (AffineNet)0.720.8560.740.8520.6820.973
Ours $^{{*}}$ (AffineNet)0.680.8650.690.8620.7060.977
Table 1: Comparison of main results on the FreiHAND test set. $*$ indicates the system is trained on a combination of datasets.
MethodPose Error $\downarrow$Pose AUC $\uparrow$Mesh Error $\downarrow$Mesh AUC $\uparrow$F@5 mm $\uparrow$F@15 mm $\uparrow$
Hasson et al. [^16]1.140.7731.140.7730.4280.932
Hasson et al. [^17]1.10---0.460.93
Hampali et al. [^15]1.070.7881.060.7900.5060.942
Ours (AffineNet)0.990.8041.010.7990.5000.943
Ours $^{\dagger}$ (AffineNet)1.040.7931.090.7820.4840.935
Ours $^{*}$ (AffineNet)0.810.8380.840.8310.5770.970
LossesPose Error $\downarrow$Pose AUC $\uparrow$Mesh Error $\downarrow$Mesh AUC $\uparrow$
$E_{UV}$$E_{grad}$$E_{verts}$
✓0.750.8500.770.846
✓✓0.730.8540.750.850
✓✓✓0.720.8560.740.852
Table 2: Comparison of main results on the HO3D test set. † indicates cross-dataset clouds in world coordinates, and then the dist evaluation. * indicates trained use extra data. is computed between corresponding points.
Table 4: Ablation studies for different losses used in our method on the FreiHAND testing set.
$f_{ac}$Crop RatioPose Error $\downarrow$Pose AUC $\uparrow$Mesh Error $\downarrow$Mesh AUC $\uparrow$
w/o10.850.8310.870.827
3/40.810.8390.820.837
1/20.760.8480.780.845
w/10.770.8470.790.843
3/40.720.8560.740.852
1/20.740.8520.760.849
Table 5: Comparison of the affine connection module $f_{ac}$ used or not, where three crop ratios are compared in each case.
state-of-the-art methods [^15] [^16] [^17] on the HO3D dataset in Table 2. In the condition that no extra training data is used, our method outperforms all previous methods. In addition, when the additional training data (the training set of FreiHAND, ObMan, and YT-3D) is used, our method achieves better performance (see “Ours* (AffineNet)” in the table). We also notice that our cross-dataset evaluation results (“Ours† (AffineNet)” in the table) surpass other results, where different from their training on the HO3D training set, we train the model on a combination of data from other datasets (the training data of FreiHAND, ObMan, and YT-3D). Even though most of the samples in the training set (FreiHAND, ObMan, and YT-3D) represent bare hands while the samples in the HO3D test set are hand-object interaction images, this cross-dataset evaluation still has good performance, showing the robustness and effectiveness of the proposed AffineNet.
SetRMSE $\downarrow$PSNR $\uparrow$
Input28.1227.58
Output (sampled)12.0637.74
Output7.6839.18
Table 3: Comparison of main results on HIC. Note that the depth maps are transferred into point clouds in world coordinates, and then the distance is computed between corresponding points.
#### 4.5. Evaluation of Hand Super-resolution Reconstruction The SRNet is trained on the SHS dataset, and the training detail is shown in Section 4.3. Once trained, our SRNet can be directly used for hand super-resolution reconstruction without any fine-tuning. Here, we evaluate the SRNet on FreiHAND, HO3D and HIC. The FreiHAND and HO3D are annotated using MANO fitting, thus their ground truth meshes are as coarse as MANO hand mesh. We use the output UV position maps of AffineNet as the input of the SRNet, and the SRNet can output hand meshes with higher fidelity. The qualitative results on HO3D and FreiHAND are shown in Figure 1. For each input RGB image, we visualize the output of the AffineNet, the output of the SRNet in two viewpoints. We find that the SRNet outputs higher-resolution hand meshes than the output of the AffineNet while preserving the same pose information. The high-resolution meshes of the SRNet show smoother and more realistic skin surface. To quantitatively evaluate the SRNet, experiments are conducted on the HIC dataset. The input of the SRNet is the UV position map converted from its MANO-fitted mesh with 778 vertices (“Input” in Table 3), and the output is the high-resolution UV position map that can be converted into a high-fidelity mesh with 3093 vertices (“Output” in the table). Besides, we also down-sample the high-fidelity mesh to 778 vertices (“Output (sampled)” in the table). In order to compare these three sets of hand surfaces, we render them as depth maps based on the viewpoint of the input image and use the depth map observation to calculate the RMSE and PSNR of the rendered depth image. Note that the background is erased by the intersection of these depth maps. As shown in Table 3, the output of SRNet (“Output” in the table) shows a better reconstruction quality to the original ---
Variants of our MethodAUC of PCK $\uparrow$AUC of PCV $\uparrow$
UV10.8600.856
UV20.8620.860
UV30.8650.862
Table 6: The results comparison of different UV forms in different stages.
Variants of our MethodAUC of PCK $\uparrow$AUC of PCV $\uparrow$
S00.8650.862
S10.8620.858
S20.8390.834
Table 7: The results comparison of different UV forms in different stages, $S_i$ means the i-th pyramid level output $I_{UV}^i$.
![Image](images/4f72724098441cf57c1aea9fd826aad68d70450de5fd304d0b8068c088c94b36.jpg) ![Image](images/509bedbe66b743ee4702d6fe80766a991a76fa76117e30cb86f624c73830d7c5.jpg) depth map. We also notice that the down-sampled output ("Output (sampled)" in the table) obtains higher accuracy than the mesh fitted by MANO ("Input" in the table), even though they have the same mesh resolution. #### 4.6. Ablation Study Effect of Each Loss Function. As presented in Table 4, we give evaluation results of AffineNet on the FreiHAND test set of settings with different losses used during the training. From the table, we can see that the best performance is achieved when all proposed loss functions are used. More results are shown in the Appendix. Effect of the Affine Connection. As presented in Table 5, we give evaluation results on FreiHAND of settings with the affine connection module $f_{ac}$ used or not, where three crop ratios are compared in each case. Here, the cropping operation is designed to compare the effect of different foreground and background ratios on the reconstruction result. For example, when the crop ratio is 1/2, the image is center cropped by 1/2 of the width and height and then resized into the original size. In Table 5, for each crop ratio, we find the “w/ $f_{ac}$” gets better performance than “w/o $f_{ac}$” and it gets the best performance when the crop ratio is 3/4. Therefore, we choose 3/4 as the crop ratio in other experiments. In this case, $f_{ac}$ results in an 11.1% reduction in pose error and a 9.8% reduction in mesh error. Effect of UV unfolding forms. The UV map is obtained by unfolding the hand mesh. Thus, the valid part of the UV map is dependent on the crop trajectory of the surface (i.e., the black lines on the 3D surface in Figure 3). We compare three different UV position maps using different crop and combine schemes, and indicate UV1, UV2 and, UV2 from left to right in Figure 3. UV1 separates the front and back of the hand and the area on the UV position map of each piece is proportional to the area of the mesh surface. UV2 and UV3 don't separate each finger and the area on the UV position map of each piece is proportional to the vertex number of the mesh surface, where each piece has a different position in UV space. In Table 6, we give evaluation results on the FreiHAND test set while using a combined training set (refer to the train data of FreiHAND, ObMan,
Figure 4: Comparison of 3D PCK and 3D PCV on the FreiHAND dataset. The proposed method is superior to [^27], [^29] and [^53].
and YT-3D). Despite different UV position maps are used, the performance of hand pose and shape estimation is similar, where less than 0.7% difference on AUC of PCK/PCV is shown. The results show that our method is robust to the UV position map designing. In this paper, we use UV3 as the template UV position map. Comparison of results from UV maps of different stages. As illustrated in Section 3.3.1, the supervision is used on multiple-scale UV maps. Thus, the mesh can be recovered from each UV map. Here, we compare the mesh prediction results of the last three UV maps (S0, S1, and S2) on the FreiHAND test set while using a combined training set. S0 indicates the full resolution UV map prediction, while S1 refers to the second last output with 1/2 of the full resolution and S2 refers to the third last output with 1/4 of the full resolution. As shown in Table 7, from low-resolution S2 to high-resolution S0, the AUC of PCK and PCV show obvious improvement. ### 5. Conclusion We have presented a novel I2UV-HandNet approach for accurate and high-fidelity 3D hand reconstruction from a single color image. The proposed UV position map enables representing a 3D hand surface in an image style. For accurate hand pose and shape estimation from the monocular images, we present an AffineNet to predict the UV position map from the RGB input. In AffineNet, a hierarchical coarse-to-fine regressing architecture is designed with a novel affine connection module that can resolve the coordinates ambiguity between the RGB image and the UV map. The proposed AffineNet achieves state-of-the-art performances on multiple challenging datasets. For high-fidelity hand shape reconstruction, we present an SRNet to restore a high-resolution UV position map from a low-resolution one. The proposed SRNet is not likely to be affected by the reconstruction method, and can robustly restore a high-fidelity hand from the inputed coarse shape. As for the future study, the UV-based hand representation can be extended to more complex joint hand-hand or hand-object reconstruction tasks, or the architecture can be modified for enabling sparse/weak supervision. --- ## References [^1]: Thiemo Alldieck, Gerard Pons-Moll, Christian Theobalt, and Marcus Magnor. Tex2shape: Detailed full human body geometry from a single image. In International Conference on Computer Vision, 2019. [^2]: Anil Armagan, Guillermo Garcia-Hernando, Seungryul Baek, Shreyas Hampali, Mahdi Rad, Zhaohui Zhang, Shipeng Xie, MingXiu Chen, Boshen Zhang, Fu Xiong, et al. Measuring generalisation to unseen viewpoints, articulations, shapes and objects for 3d hand pose estimation under hand-object interaction. In European Conference on Computer Vision, 2020. [^3]: Vassilis Athitsos and Stan Sclaroff. Estimating 3d hand pose from a cluttered image. In Conference on Computer Vision and Pattern Recognition, 2003. [^4]: Volker Blanz and Thomas Vetter. A morphable model for the synthesis of 3d faces. In Annual Conference on Computer Graphics and Interactive Techniques, 1999. [^5]: Adnane Boukhayma, Rodrigo de Bem, and Philip HS Torr. 3d hand shape and pose from images in the wild. In Conference on Computer Vision and Pattern Recognition, 2019. [^6]: Michael M Bronstein, Joan Bruna, Yann LeCun, Arthur Szlam, and Pierre Vandergheynst. Geometric deep learning: going beyond euclidean data. IEEE Signal Processing Magazine, 34(4):18–42, 2017. [^7]: Yujun Cai, Liuhao Ge, Jianfei Cai, and Junsong Yuan. Weakly-supervised 3d hand pose estimation from monocular rgb images. In European Conference on Computer Vision, 2018. [^8]: Yujin Chen, Zhigang Tu, Liuhao Ge, Dejun Zhang, Ruizhi Chen, and Junsong Yuan. So-handnet: Self-organizing network for 3d hand pose estimation with semi-supervised learning. In International Conference on Computer Vision, 2019. [^9]: Yujin Chen, Zhigang Tu, Di Kang, Linchao Bao, Ying Zhang, Xuefei Zhe, Ruizhi Chen, and Junsong Yuan. Model-based 3d hand reconstruction via self-supervised learning. In Conference on Computer Vision and Pattern Recognition, 2021. [^10]: Yujin Chen, Zhigang Tu, Di Kang, Ruizhi Chen, Linchao Bao, Zhengyou Zhang, and Junsong Yuan. Joint hand-object 3d reconstruction from a single image with cross-branch feature fusion. IEEE Transactions on Image Processing, 30:4008–4021, 2021. [^11]: Hongsuk Choi, Gyeongsik Moon, and Kyoung Mu Lee. Pose2mesh: Graph convolutional network for 3d human pose and mesh recovery from a 2d human pose. In European Conference on Computer Vision, 2020. [^12]: Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. Image super-resolution using deep convolutional networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015. [^13]: Yao Feng, Fan Wu, Xiaohu Shao, Yanfeng Wang, and Xi Zhou. Joint 3d face reconstruction and dense alignment with position map regression network. In European Conference on Computer Vision, 2018. [^14]: Liuhao Ge, Zhou Ren, Yuncheng Li, Zehao Xue, Yingying Wang, Jianfei Cai, and Junsong Yuan. 3d hand shape and pose estimation from a single rgb image. In Conference on Computer Vision and Pattern Recognition, 2019. [^15]: Shreyas Hampali, Mahdi Rad, Markus Oberweger, and Vincent Lepetit. Honnotate: A method for 3d annotation of hand and object poses. In Conference on Computer Vision and Pattern Recognition, 2020. [^16]: Yana Hasson, Bugra Tekin, Federica Bogo, Ivan Laptev, Marc Pollefeys, and Cordelia Schmid. Leveraging photometric consistency over time for sparsely supervised hand-object reconstruction. In Conference on Computer Vision and Pattern Recognition, 2020. [^17]: Yana Hasson, Gul Varol, Dimitrios Tzionas, Igor Kalevatykh, Michael J Black, Ivan Laptev, and Cordelia Schmid. Learning joint reconstruction of hands and manipulated objects. In Conference on Computer Vision and Pattern Recognition, 2019. [^18]: Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Conference on Computer Vision and Pattern Recognition, 2016. [^19]: Markus Höll, Markus Oberweger, Clemens Arth, and Vincent Lepetit. Efficient physics-based implementation for realistic hand-object interaction in virtual reality. In IEEE Conference on Virtual Reality and 3D User Interfaces, 2018. [^20]: Umar Iqbal, Pavlo Molchanov, Thomas Breuel Juergen Gall, and Jan Kautz. Hand pose estimation via latent 2.5 d heatmap regression. In European Conference on Computer Vision, 2018. [^21]: Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A. Efros. Image-to-image translation with conditional adversarial networks. In Conference on Computer Vision and Pattern Recognition, 2017. [^22]: Max Jaderberg, Karen Simonyan, Andrew Zisserman, and Koray Kavukcuoglu. Spatial transformer networks. In Proceedings of the 28th International Conference on Neural Information Processing Systems-Volume 2, pages 2017–2025, 2015. [^23]: Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference for Learning Representations, 2014. [^24]: Arno Knapitsch, Jaesik Park, Qian-Yi Zhou, and Vladlen Koltun. Tanks and temples: Benchmarking large-scale scene reconstruction. ACM Transactions on Graphics, 2017. [^25]: Nikos Kolotouros, Georgios Pavlakos, Michael J Black, and Kostas Daniilidis. Learning to reconstruct 3d human pose and shape via model-fitting in the loop. In International Conference on Computer Vision, 2019. [^26]: Nikos Kolotouros, Georgios Pavlakos, and Kostas Daniilidis. Convolutional mesh regression for single-image human shape reconstruction. In Conference on Computer Vision and Pattern Recognition, 2019. ## [^27]: Dominik Kulon, Riza Alp Guler, Iasonas Kokkinos, Michael M. Bronstein, and Stefanos Zafeiriou. Weakly-supervised mesh-convolutional hand reconstruction in the wild. In Conference on Computer Vision and Pattern Recognition, 2020 --- [^28]: Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black. Smpl: A skinned multi-person linear model. ACM Transactions on Graphics, 2015. [^29]: Gyeongsik Moon and Kyoung Mu Lee. I2l-meshnet: Image-to-pixel prediction network for accurate 3d human pose and mesh estimation from a single RGB image. In European Conference on Computer Vision, 2020. [^30]: Gyeongsik Moon, Takaaki Shiratori, and Kyoung Mu Lee. Deephandmesh: A weakly-supervised deep encoder-decoder framework for high-fidelity hand mesh modeling. In European Conference on Computer Vision, 2020. [^31]: Franziska Mueller, Micah Davis, Florian Bernard, Oleksandr Sotnychenko, Mickeal Verschoor, Miguel A Otaduy, Dan Casas, and Christian Theobalt. Real-time pose and shape reconstruction of two interacting hands with a single depth camera. ACM Transactions on Graphics (TOG), 38(4):1–13, 2019. [^32]: Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. PyTorch: An imperative style, high-performance deep learning library. In Neural Information Processing Systems, 2019. [^33]: Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Conference on Computer Vision and Pattern Recognition, 2017. [^34]: Javier Romero, Dimitrios Tzionas, and Michael J Black. Embodied hands: Modeling and capturing hands and bodies together. ACM Transactions on Graphics, 2017. [^35]: Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention. Springer, 2015. [^36]: Tomas Simon, Hanbyul Joo, Iain Matthews, and Yaser Sheikh. Hand keypoint detection in single images using multiview bootstrapping. In Conference on Computer Vision and Pattern Recognition, 2017. [^37]: Adrian Spurr, Umar Iqbal, Pavlo Molchanov, Otmar Hilliges, and Jan Kautz. Weakly supervised 3d hand pose estimation via biomechanical constraints. In European Conference on Computer Vision, 2020. [^38]: Dimitrios Tzionas, Luca Ballan, Abhilash Srikantha, Pablo Aponte, Marc Pollefeys, and Juergen Gall. Capturing hands in action using discriminative salient points and physics simulation. International Journal of Computer Vision, 118(2):172–193, 2016. [^39]: Jiayi Wang, Franziska Mueller, Florian Bernard, Suzanne Sorli, Oleksandr Sotnychenko, Neng Qian, Miguel A Otaduy, Dan Casas, and Christian Theobalt. RGB2hands: real-time tracking of 3d hand interactions from monocular rgb video. ACM Transactions on Graphics (TOG), 39(6):1–16, 2020. [^40]: Nanyang Wang, Yinda Zhang, Zhuwen Li, Yanwei Fu, Wei Liu, and Yu-Gang Jiang. Pixel2mesh: Generating 3d mesh models from single rgb images. In European Conference on Computer Vision, 2018. [^41]: Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High-resolution image synthesis and semantic manipulation with conditional gains. In Conference on Computer Vision and Pattern Recognition, 2018. [^42]: Yuanlu Xu, Song-Chun Zhu, and Tony Tung. Denserac: Joint 3d pose and shape estimation by dense render-and-compare. In International Conference on Computer Vision, 2019. [^43]: Pengfei Yao, Zheng Fang, Fan Wu, Yao Feng, and Jiwei Li. Densebody: Directly regressing dense 3d human pose and shape from a single color image. arXiv preprint arXiv:1903.10153, 2019. [^44]: Shanxin Yuan, Guillermo Garcia-Hernando, Björn Stenger, Gyeongsik Moon, Ju Yong Chang, Kyoung Mu Lee, Pavlo Molchanov, Jan Kautz, Sina Honari, Liuhao Ge, et al. Depth-based 3d hand pose estimation: From current achievements to future goals. In Conference on Computer Vision and Pattern Recognition, 2018. [^45]: Wang Zeng, Wanli Ouyang, Ping Luo, Wentao Liu, and Xiaogang Wang. 3d human mesh regression with dense correspondence. In Conference on Computer Vision and Pattern Recognition, 2020. [^46]: Jiawei Zhang, Jianbo Jiao, Mingliang Chen, Liangqiong Qu, Xiaobin Xu, and Qingxiong Yang. 3d hand pose tracking and estimation using stereo matching. arXiv preprint arXiv:1610.07214, 2016. [^47]: Tianshu Zhang, Buzhen Huang, and Yangang Wang. Object-occluded human shape and pose estimation from a single color image. In Conference on Computer Vision and Pattern Recognition, 2020. [^48]: Xiong Zhang, Qiang Li, Hong Mo, Wenbo Zhang, and Wen Zheng. End-to-end hand mesh recovery from a monocular rgb image. In International Conference on Computer Vision, 2019. [^49]: Xingyi Zhou, Qixing Huang, Xiao Sun, Xiangyang Xue, and Yichen Wei. Towards 3d human pose estimation in the wild: a weakly-supervised approach. In International Conference on Computer Vision, 2017. [^50]: Yuxiao Zhou, Marc Habermann, Weipeng Xu, Ikhsanul Habibie, Christian Theobalt, and Feng Xu. Monocular real-time hand shape and motion capture using multi-modal data. In Conference on Computer Vision and Pattern Recognition, 2020. [^51]: Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In International Conference on Computer Vision, 2017. [^52]: Christian Zimmermann and Thomas Brox. Learning to estimate 3d hand pose from single rgb images. In International Conference on Computer Vision, 2017. [^53]: Christian Zimmermann, Duygu Ceylan, Jimei Yang, Bryan Russell, Max Argus, and Thomas Brox. Freihand: A dataset for markerless capture of hand pose and shape from single rgb images. In International Conference on Computer Vision, 2019.