# 学习用于3D手部姿态估计的相关性感知随机不确定性建模 李彩妍 $^{1}$ 南贤宇² 吴泰贤 $^{3}$ $^{1}$ 韩国浦项科技大学人工智能研究生院 $^{2}$韩国浦项科技大学电气工程系 $^{3}$ 韩国科学技术院计算学院 ## 摘要 3D手部姿态估计是理解人类手部的基础任务。然而，由于手部运动复杂、具有自相似性且经常被遮挡，准确估计3D手部姿态仍颇具挑战性。在本研究中，我们解决了两个局限性：现有3D手部姿态估计方法无法估计随机（数据）不确定性，以及缺乏融入关节相关性知识的不确定性建模，而后者尚未得到充分研究。为此，我们将随机不确定性建模引入3D手部姿态估计框架，旨在实现关节相关性建模与计算效率之间的更好平衡。我们提出了一种新颖的参数化方式，利用单个线性层来捕捉手部关节间的内在相关性。这一实现的前提是将手部关节输出空间建模为概率分布，从而使线性层能够捕获关节相关性。我们提出的参数化方式被用作任务头层，可作为附加模块应用于现有模型之上。实验表明，我们用于不确定性建模的参数化方式优于现有方法。此外，搭载了我们提出的不确定性头的3D手部姿态估计模型在保持较高精度的同时，还具备新的不确定性建模能力。项目页面可在 查看。 ## 引言 理解人类手部对于从机器人技术到AR/VR等各类应用而言都至关重要 [^1]。感知并解读人类手部的能力能够显著提升机器人辅助任务的灵巧性，确保机器人与人之间能进行无缝交互。在从人类演示中学习的过程中，机器人可通过观察这些演示来获取人类手部行为模式 [^1] [^19] [^19]。在本研究中，我们针对最先进的3D手部姿态估计方法中存在的两个关键局限性展开探讨。 (1) 无法估计随机（数据）不确定性。尽管基于学习的3D手部姿态估计领域的最新研究 [^20] [^28] [^30] [^33] [^42] [^53] [^54] [^55] [^60] 已取得显著进展，但准确的3D手部姿态估计本质上会受到真实场景视频中不确定性的影响。这些不确定性源于手部所具有的大量自由度。 --- 手部[^0]、频繁出现的自相似性和遮挡现象[^37] [^49]，以及因其动态特性而产生的运动模糊[^39] [^41]。面对这些观测噪声，量化随机不确定性能够提升手部估计模型的置信度，从而使其得以应用于实际场景。(2) 在缺乏关节相关性知识的情况下的不确定性建模。人类手部拥有众多独立的自由度，但各关节的运动是相互关联的[^42] [^43] [^44] [^45] [^46]。尽管一个手关节的不确定性会影响另一个关节的不确定性，但以往的研究[^47] [^48]出于计算和参数效率的考虑，在独立假设下逐行独立地对不确定性进行建模。 在本研究中，我们将随机不确定性建模引入3D手部姿态估计框架，实现了相关性建模与效率之间的更好平衡。我们提出了一种新颖的参数化方法，利用单个线性层来捕捉手部关节间的内在相关性。为此，我们设计了一个概率性的手部关节输出空间，以便在进行不确定性建模时考虑关节间的依赖关系。具体而言，我们首先训练一个预训练的大型模型[^2]来进行随机不确定性估计，方法是添加一个能够回归各关节方差的Transformer头。我们采用具有对角协方差矩阵的高斯负对数似然损失函数，这使得网络能够在独立假设下预测代表手部关节不确定性的方差。所估计的不确定性定义了一个概率性输出空间，我们从中抽取样本并输入到单个线性层中，从而将输出空间转换为具有相关性感知能力的空间。我们的参数化方法介于对角协方差矩阵参数化和全协方差矩阵参数化之间。与对角形式相比，它能以更高的表达能力捕捉关节相关性，同时所需的参数数量又远少于全协方差参数化方法。 我们在两个标准的3D手部姿态估计基准测试集FreiHAND[^59]和HO3Dv2[^63]上验证了该方法的有效性。在不确定性估计方面，我们的方法优于现有的随机不确定性建模方法。该方法有效性的关键在于将手部关节输出空间表述为概率分布，从而使线性层能够有效学习手部关节的相关性。这种方法能够实现结构化协方差矩阵的解析表示，便于直接估计不确定性。此外，我们的方法在3D手部姿态估计中仍保持了具有竞争力的准确性，这表明建模不确定性并不会损害姿态估计的性能。我们的主要贡献总结如下： • 我们将随机不确定性建模引入3D手部姿态估计框架。 - 我们提出了一种新颖的参数化方法，利用单个线性层有效建模固有的手部关节相关性，而这得益于将手部关节输出空间表述为概率分布。 - 全面的实验表明，所提出的方法在不确定性估计方面显著优于现有的随机不确定性建模方法，同时还能保持准确的3D手部姿态估计性能。 ## 2 相关工作 3D手部姿态估计。从单张RGB图像中进行3D手部姿态估计，在理解复杂的手部交互方面受到了广泛关注。深度学习的出现尤其提升了3D手部姿态估计的性能，优于传统的手工设计几何特征方法。大多数现有研究[^1] [^2] [^4] [^5]都采用了MANO参数化模型。 --- 手部模型[^15]，并直接从RGB图像中回归手部姿态和形状参数。另一些研究[^11] [^12] [^13] [^14] [^15]采用非参数化方法，直接从图像中回归网格顶点坐标，以实现更精细的手部表面重建。还有一类研究[^1]、[^2]、[^3]、[^4]用于推断x、y、z三个维度的3D手部关节位置，这些位置可作为手部姿态的骨骼表示。最近，HaMeR[^16]利用Transformer网络[^17]并在大型数据集上训练，实现了鲁棒的3D手部重建，对自然场景图像也表现出强大的泛化能力。我们的工作旨在解决3D手部姿态估计中的偶然不确定性问题，这一问题此前尚未得到充分探索。 深度学习中的不确定性。深度学习中的不确定性可分为偶然不确定性和认知不确定性[^1]。偶然不确定性源于数据的不确定性这一非确定性本质；认知不确定性则源于模型因训练数据知识不足而产生的不确定性。在本研究中，我们重点关注3D手部关节位置的偶然不确定性。我们假设这种不确定性具有异方差性[^27]，即它依赖于模型的输入，因为某些输入本身可能就具有更高的不确定性。常用的方法是估计输出的概率分布，并通过最小化真实值的负对数似然（NLL）来训练网络[^3] [^57] [^67]。Caramalau等人[^8]证明，在应用主动学习框架从单张深度图像估计3D手部姿态时，同时建模偶然不确定性和认知不确定性效果显著。然而，现有的从单张RGB图像估计3D手部姿态的方法要么忽略了输入图像中的异方差不确定性，要么依赖隐式分布通过采样来捕捉偶然不确定性[^61] [^62]，且这些方法并未公开。Zhang等人[^63]将NLL损失应用于3D手部重建以建模输出空间的不确定性；不过他们的方法仅限于建模2D手部关节位置的不确定性，可能无法准确反映3D关节空间的不确定性。AMVUR[^24]采用了一种用于3D手部姿态和形状估计的概率框架，天然支持对3D手部关节位置的不确定性量化。在本研究中，我们通过学习3D手部关节位置上的高斯分布，显式建模输出空间中的逐关节不确定性，并将其通过简单而有效的NLL损失纳入训练目标。此外，我们利用单一线性变换来建模关节间的依赖关系，从而无需依赖基于采样的近似方法，就能解析地表述这种结构化不确定性。 ## 3 方法 ### 3.1 预备知识 符号说明。我们用$\mathbf{x} \in \mathbb{R}^d_i$表示输入图像，用$f: \mathbb{R}^d_i \to \mathbb{R}^d_f$表示特征提取器，用$g: \mathbb{R}^d_f \to \mathbb{R}^d_o$表示参数回归器，而$\text{diag}(\cdot)$则用于根据向量输出对角矩阵。如图1a所示，未采用不确定性建模的模型记为$g(f(\mathbf{x}))$。采用不确定性建模[LJ]时，回归器g会被修改为$g(f(\mathbf{x})) \sim \mathcal{N}(\mu, \Sigma)$，其中$\mathcal{N}$是均值为$\mu \in \mathbb{R}^d_o$、协方差为$\Sigma \in \mathbb{R}^{d_o \times d_o}$的高斯分布。后续段落将介绍两种基于协方差矩阵$\Sigma$建模的方法：对角协方差矩阵和全协方差矩阵。 对角协方差矩阵。最常用的不确定性建模方式是采用对角协方差矩阵。这种方式参数效率较高，因为回归器仅需输出均值和对角方差向量即可。对角高斯建模所需的输出维度是原来的两倍，即$g_{diag} : \mathbb{R}^{d_f} \to \mathbb{R}^{2 \cdot d_o}$（见图1b）。其中一半维度用于 --- ![Image](images/fe65ff1e7fc0aa31b671a2d354da689a6757076db27dc72a9e8860d171b94913.jpg)
(a) Deterministic
(b) Diagonal
(c) Full
![Image](images/d21e1e7ac6c132dec6bff75ab4d271fac3bdbeb11eaa5bc92c5914a8636d4984.jpg) ![Image](images/fae03a668ecb4754ea1f76c9e1de37743d7db413e5b2b04a43ea02658607730b.jpg)
(d) Ours
Figure 1: Illustration of deterministic modeling and correlation modelings of uncertainty. (a) Deterministic modeling produces a single deterministic output, represented as point embedding in the output space; (b) Diagonal and (c) Full correlation modeling of output produce means and covariances of a Gaussian distribution, where the uncertainty of the prediction is modeled by its variance. (d) Ours learns the variation of each dimension in the output space and shared weight $\mathbf{W}$ which captures intrinsic hand joint dependencies, so that our covariance matrix is represented by $\Sigma = \mathbf{W}\sigma^2\mathbf{W}^\top$.
存储均值向量$\mu \in \mathbb{R}^{d_o}$，另一半维度则用于存储方差向量$\sigma^2 \in \mathbb{R}^{d_o}$，具体形式如下： $$ g_{d i a g}(f(\mathbf{x}))\sim\mathcal{N}(\mu,\mathrm{d i a g}(\boldsymbol{\sigma}^{2})). $$ 在实现中，并非直接使用 $\sigma^2$，而是使用 $\sigma^2$ 的对数，并通过指数函数映射以确保取值为正。模型通过以真实值 $\mathbf{y}$ 最小化负对数似然（NLL）损失来训练，即 $$ \mathcal{L}_{N L L}=\operatorname{l o g}p(\mathbf{y}|\mathbf{\mu},\mathbf{\sigma}^{2})=\frac{||\mathbf{y}-\mathbf{\mu}||^{2}}{2\mathbf{\sigma}^{2}}+0.5\operatorname{l o g}\mathbf{\sigma}^{2}. $$ 全协方差矩阵。在全协方差矩阵中，回归器会输出所有的协方差矩阵元素，具体形式为 $$ g_{f u l l}(f(\mathbf{x}))\sim\mathcal{N}(\mu,\Sigma), $$ 其中 $\mu \in \mathbb{R}^{d_o}$ 和 $\Sigma \in \mathbb{R}^{d_o \times d_o}$（见 Fig. 1c）。由于 $\Sigma$ 应为正定矩阵，我们将 $\Sigma$ 构造为 $\mathbf{AA}^\top$，从而确保 $\Sigma$ 为正定矩阵。尽管与对角协方差相比，其参数效率较低
ParametrizationUncertainty modeling# Params.Example [# Params]
Deterministic✗$d_{f}do$0.065M
Diagonal covariance✓(independent)$2d_{f}do$0.129M
Full covariance✓$d_{f}(do + d_{o}^{2})$4.129M
Ours✓$2d_{f}do + d_{o}^{2}$0.133M
Table 1: The number of parameters. The dimensions of feature and output are $d_{f}$ and $d_{o}$, respectively. We list the required number of parameters for each parameterization. We set $d_{f}=1024$, $d_{o}=63$, and k=21 for example.
但它具备更强的捕捉相关性的能力。该模型的训练同样是通过最小化NLL损失来实现的，具体形式如下： $$ \begin{array}{r}{\mathcal{L}_{N L L}=\log p(\mathbf{y}|\boldsymbol{\mu},\boldsymbol{\Sigma})=\frac{1}{2}(\mathbf{y}-\boldsymbol{\mu})^{\top}\boldsymbol{\Sigma}^{-1}(\mathbf{y}-\boldsymbol{\mu})+\frac{1}{2}\log\operatorname*{d e t}\boldsymbol{\Sigma}.}\end{array} $$ 目标函数中涉及逆协方差矩阵，这会引入优化不稳定性。因此，直接估计精度矩阵$\Psi = \Sigma^{-1}$会更加稳定。 综上所述，对角协方差矩阵虽然参数效率高，但无法捕捉相关性；全协方差矩阵的特性则恰好相反。我们提出了一种新的中间表示方式，它利用单个线性层，既具备参数高效性，又能以概率方式有效捕捉手部关节的相关性。 --- ![Image](images/3dde3aaeece2b1fcdb004ae02d940b6047ec8f837fe195c260eeb7c54edc2d58.jpg)
Figure 2: Pipeline of our proposed method. We train a pre-trained large model [L] by introducing an additional transformer head that estimates the variance for each joint under the independent assumption. This estimated uncertainty then defines a probabilistic hand joint output space, from which we sample and pass the samples through a single linear layer to model the correlations between hand joints.
### 3.2 手部关节中的相关性感知随机不确定性估计 我们的目标是以高效且富有表现力的方式，结合关节相关性知识来估计手部关节的随机不确定性。核心思路是先在独立性假设下定义基于各关节不确定性的概率输出空间，再通过单个线性层将其转换为能够反映相关性的空间。 基于对角协方差矩阵的随机不确定性估计。我们采用HaMeR [☐]——一种最近提出的、基于Transformer且已在大规模数据集上预训练的手部姿态估计模型，以实现不确定性建模。在HaMeR中，视觉Transformer（ViT）[☐]骨干网络充当特征提取器，而均值3D手部关节位置$\mu_{3D}$则由Transformer头层回归，随后通过MANO [☐]模型处理。为了实现不确定性建模，我们引入了一个额外的Transformer头层，即不确定性回归器，用于回归每个关节的方差$\sigma_{3D}^2$。该模型通过最小化NLL损失（公式(2)）进行训练，训练数据为真值3D手部关节位置$\mathbf{y}_{3D}$。至此，不确定性建模与对角协方差矩阵的处理方式一致。 概率性手部关节输出空间。我们在零均值高斯假设下定义了概率性手部关节输出空间，其中在独立性假设下学习得到的各关节方差被用作协方差矩阵的对角元素，即$p(\mathbf{z}|\mathbf{x}) \sim \mathcal{N}(\mathbf{0}, \mathrm{diag}(\sigma_{\mathbf{3D}}^2))$。我们从$p(\mathbf{z}|\mathbf{x})$中抽取N个样本，并将这些样本输入到权重为$\mathbf{W}$的单个线性层中，利用高斯分布的线性特性确保最终输出遵循$p(\hat{\mathbf{y}}|\mathbf{z}) \sim \mathcal{N}(\mathbf{0}, \mathbf{W} \text{diag}(\sigma_{\mathbf{3D}}^2) \mathbf{W}^\top)$。通过采用零均值假设，我们消除了权重$\mathbf{W}$与均值$\mu_{\mathbf{3D}}$之间的依赖关系，从而使模型能够专注于捕捉关节间的相关性。之后，将均值3D手部关节位置$\mu_{\mathbf{3D}}$添加到该输出空间中。最终我们得到： $$ p(\hat{\mathbf{y}}|\mathbf{z})\sim\mathcal{N}(\mu_{\mathbf{3D}},\mathbf{W}\mathrm{d i a g}(\sigma_{\mathbf{3D}}^{2})\mathbf{W}^{\top}). $$ 我们最小化预测值与真值之间的均方误差（MSE）： $$ \mathcal{L}_{M S E}=\mathbb{E}||\mathbf{y}-\widehat{\mathbf{y}}||_{2}^{2}, $$ --- 其中E表示期望值。整体训练损失定义如下： $$ \mathcal{L}=\mathcal{L}_{D E T E R}+\lambda_{N L L}\mathcal{L}_{N L L}+\lambda_{M S E}\mathcal{L}_{M S E}, $$ 其中$\mathcal{L}_{DETER}$表示HaMeR中用于确定性3D手部姿态估计的损失函数[^10]，而$\lambda_{NLL}$和$\lambda_{MSE}$则是不确定性建模项的权重系数。 通过从概率性手部关节输出空间中采样并应用单个线性变换，我们能够自然地捕捉手部关节之间的相关性，这一关系也可通过解析方式表述。我们将所提方法定位为对角协方差建模与完全协方差建模之间的中间表示形式。首先，相较于对角协方差建模，它在捕捉手部关节间固有相关性方面具有更高的表达能力；通过采用概率性空间，我们的输出分布能够像完全协方差建模一样刻画手部关节的相关结构。其次，与完全协方差建模相比，该方法所需的参数数量更少——完全协方差建模涉及的参数数量与输出维度的平方成正比，这使其在高维输出场景下不具实用性且容易引发优化不稳定问题。相比之下，我们的方法仅在对角协方差建模基础上引入一个线性变换矩阵，既保证了计算效率（见表1），又提升了优化稳定性。 ## 4 实验 ### 4.1 实验设置 数据集。我们在来自多个数据集的270万个训练样本上训练我们的方法，这些数据集提供了如HaMeR中所述的2D或3D手部标注[^72]。这些数据集包括FreiHAND [^79]、HO3D [^73]、MTC [^77]、RHD [^78]、InterHand2.6M [^77]、H2O3D [^73]、DEX YCB [^71]、COCO WholeBody [^75]、Halpe [^75]以及MPII NZSL [^73]。为了评估我们方法所估计的不确定性以及3D手部姿态估计的准确性，我们使用了两个标准的3D手部姿态估计基准测试集：FreiHAND [^79]和HO3Dv2 [^73]，它们提供了真实的3D手部标注。 3D手部姿态估计指标。我们遵循以往研究中常用的协议[^77] [^77]，报告用于评估估计的3D手部关节的PA-MPJPE和AUC $_J$，以及用于评估估计的3D手部网格的PA-MPVPE、AUC $_V$、F@5mm和F@15mm。PA-MPJPE和PA-MPVPE的单位为毫米。 不确定性估计指标。为了评估估计的不确定性质量，我们使用稀疏化曲线[^2]、[^43]。预测的手部关节会根据估计的不确定性进行排序。给定一个误差度量$\varepsilon$，我们评估确定性最高的前$x\%$个关节。理想情况下，如果不确定性估计是校准良好的，那么随着不确定性较高的预测被移除，误差应该会减小。我们将x的值从2变化到100，每次递增2，并像以往研究那样报告稀疏化曲线下的面积（AUSC）[^21]。这一指标既受预测准确性的影响，也受基于不确定性的排序与实际基于误差的排序相似度的影响。 ![Image](images/9f7296d8333bc486916965227cc8c3b040e7c92a9444e44f0c6769727c1b398b.jpg)
(b) HO3Dv2
![Image](images/fe362c794fcafa856477ed71504b85aa82ce8c75e96301555810421c1127e94a.jpg)
Figure 3: Sparsification curves. We compare sparsification curves obtained by different methods of estimating the uncertainty of 3D hand joints.
---
MethodFreiHAND [^63]HO3Dv2 [^83]
AUSC $\downarrow$AUSE $\downarrow$Pearson’s $\rho$ $\uparrow$AUSC $\downarrow$AUSE $\downarrow$Pearson’s $\rho$ $\uparrow$
Diagonal65554.60.39351163.30.448
Full64847.60.45351264.30.493
Ours64242.20.56950557.60.600
Table 2: Quantitative evaluation on uncertainty estimation. We demonstrate the effectiveness of our parametrization in consistently enhancing all three metrics.
![Image](images/3aeaf8d2a8e7cb2131294a540a826861c34f07b1512cda351f98dcbffedde6b3.jpg)
Figure 4: Qualitative results of uncertainty estimation. We evaluate the quality of uncertainty estimates by comparing our method with existing uncertainty modeling methods. Specifically, we visualize the prediction errors alongside the corresponding uncertainty values. A desirable property of an uncertainty estimator is its proportionality to the actual prediction error—i.e., higher uncertainty values should correspond to higher prediction errors. The uncertainty estimated by our method shows stronger correlation with the prediction error, indicating its effectiveness in capturing model confidence.
排序。为了仅评估后者，我们还报告稀疏化误差下的面积（AUSE）[☐]，其计算方法是从估计的稀疏化曲线中减去基于$\varepsilon$大小排序关节所得的理想稀疏化曲线。我们假设MPJPE即为$\varepsilon$。每个关节的不确定性通过其估计的协方差矩阵的迹来衡量。此外，我们报告皮尔逊相关系数$\rho$，以量化预测的不确定性与实际误差之间的相关程度。皮尔逊相关系数$\rho$衡量两个连续变量之间线性关系的强度和方向，其值介于-1和1之间。 基线方法。我们实现了两种基于高斯负对数似然（NLL）的传统不确定性建模方法作为基线，并将它们与我们的方法进行比较。（1）对角矩阵：不确定性回归器在独立假设下输出每个关节的方差；不确定性通过对角协方差矩阵来建模。（2）完整矩阵：不确定性回归器输出协方差矩阵的所有元素，用完整的协方差结构来建模不确定性（详见第3.1节）。 --- ![Image](images/cbc857d41f4af7c2ea5fc397b7abe40417d95933f6ffc96ac56b704e79114c00.jpg) ![Image](images/7503105f239eaf78e46e5d22b43e3cb784580cf7e1a473f29c61db8fe511cc04.jpg) ![Image](images/45cb970bf5720c6033841aec7f4938c3fa727d015aa83c800259f01323a125bd.jpg)
(b) HO3Dv2
![Image](images/b3dd67506f2ae3a100255501814eeb5e85b1f4b30bbd8b2d27f57158006f94ba.jpg) ![Image](images/711994b0e21f1fb9b7005734d76d4831925f958bda8af20797c5a661bb23b212.jpg) ![Image](images/1b9da706f21a6182f90c74b58fd4dcde161c286bc941cfa71a9f3da7a75b6b8e.jpg)
Figure 5: 2D histograms of model error and estimated uncertainty. We show 2D histograms of the x-axis representing the model's MPJPE error and the y-axis representing the estimated uncertainty on (a) FreiHAND [☑] and (b) HO3Dv2 [☒] datasets.
实现细节。在所有实验中，包括基线和我们的方法，我们都设置$N = 25$、$\lambda_{NLL} = 5e-4$。在我们的方法中，我们还设置了$\lambda_{MSE} = 5e-4$。我们在单张NVIDIA A100 GPU上训练模型，迭代次数为55万次，使用AdamW优化器，权重衰减参数为$1e-4$、$\beta_1 = 0.9$和$\beta_2 = 0.999$。学习率初始化为$1e-6$，小批量大小设为64。其他所有实验设置均与HaMeR中的设置一致[^12]。 ### 4.2 不确定性估计 我们开展评估以检验所估计不确定性的质量。表2中展示了与现有不确定性建模方法的定量比较结果。结果表明，我们的方法在所有指标上均优于基线模型。图3中的稀疏化曲线显示，尽管在针对所有3D关节进行评估时各方法表现相近，但当我们剔除不确定性较高的3D关节后，我们的方法准确率明显高于其他方法。这表明我们所建模的不确定性能更好地反映真实的3D关节预测误差。我们通过图4中的定性比较进一步验证了这一点，其中我们的不确定性与预测误差的相关性更强。此外，我们通过可视化模型的MPJPE误差与其估计不确定性之间的二维直方图，来展示皮尔逊相关系数的定性结果。如图5所示，我们提出的方法所得到的不确定性估计与模型误差的相关性，比Diagonal和Full基线模型更好，同时异常值也更少。 ### 4.3 3D手部姿态估计 我们评估了该方法在3D手部姿态估计方面的能力。表3将我们的方法与现有的不支持不确定性估计的3D手部姿态估计方法，以及不确定性建模基线模型进行了比较。为了公平比较，我们还 ---
CHAE-YEON ET AL.: LEARNING CORRELATION-AWARE ALEATORIC UNCERTAINTY
MethodFreiHAND [☐]HO3Dv2 [☐]
PA-MPIPE $\downarrow$PA-MPVPE $\downarrow$F@5 $\uparrow$F@15 $\uparrow$AUC $_j$ $\uparrow$PA-MPIPE $\downarrow$AUC $_v$ $\uparrow$PA-MPVPE $\downarrow$F@5 $\uparrow$F@15 $\uparrow$
I2L-MeshNet [☐]7.47.60.6810.9730.77511.20.72213.90.4090.932
Pose2Mesh [☐]7.77.80.6740.9690.75412.50.74912.70.4410.909
I2UV-HandNet [☐]6.76.90.7070.9770.8049.90.79910.10.5000.943
METRO [☐]6.56.30.7310.9840.79210.40.77911.10.4840.946
MobRecon [☐]5.75.80.7840.986-9.2-9.40.5380.957
AMVUR [☐]6.26.10.7670.9870.8358.30.8368.20.6080.965
HaMeR [☐]6.05.70.7850.9900.8467.70.8417.90.6350.980
Deterministic6.15.70.7850.9900.8457.80.8408.00.6290.980
Diagonal6.15.80.7820.9900.8477.70.8427.90.6380.981
Full6.15.90.7730.9890.8497.60.8447.80.6440.981
Ours6.05.70.7840.9900.8477.60.8437.90.6400.981
Table 3: Quantitative evaluation on 3D hand pose estimation. Our method maintains comparable performance compared to 3D hand pose estimation methods and uncertainty modeling baselines.
MethodFreiHAND [^69]HO3Dv2 [^83]
AUSC $\downarrow$AUSE $\downarrow$Pearson's $\rho$ $\uparrow$AUSC $\downarrow$AUSE $\downarrow$Pearson's $\rho$ $\uparrow$
Ours w/o linear layer64847.90.54450961.40.524
Ours64242.20.56950557.60.600
Table 4: Ablation study on joint correlation in uncertainty estimation. We show that our joint correlation modeling consistently improves uncertainty estimation performance across all three metrics.
MethodFreiHAND [☐]HO3Dv2 [☐]
PA-MPJPE $\downarrow$PA-MPVPE $\downarrow$F@5 $\uparrow$F@15 $\uparrow$AUC $_j$ $\uparrow$PA-MPJPE $\downarrow$AUC $_v$ $\uparrow$PA-MPVPE $\downarrow$F@5 $\uparrow$F@15 $\uparrow$
Ours w/o linear layer6.15.80.7820.9900.8477.70.8437.90.6380.981
Ours6.05.70.7840.9900.8477.60.8437.90.6400.981
Table 5: Ablation study on joint correlation in 3D hand pose estimation. We demonstrate that our joint correlation modeling shows improvement in 3D hand pose estimation accuracy.
将我们的方法与一个名为Deterministic的确定性基线模型进行对比，该基线模型在不引入不确定性建模的情况下对预训练网络进行微调。与3D 手部姿态估计方法和确定性基线模型相比，我们的模型在两个基准测试集上均取得了具有竞争力的性能，同时还能够提供不确定性估计。在 相较于手部姿态估计方法和确定性基线模型，我们的模型取得了具有竞争力的结果。 方法通过高效的关节相关性建模，提供了可靠得多的不确定性估计（见表1和表2）。 在不确定性建模基线的比较中，尽管完整基线模型的性能略好一些…… 虽然在 HO3Dv2 [^13] 评估数据集上，完整基线模型在3D手部姿态估计方面的表现更优，但我们的模型…… 方法通过高效的关节相关性建模，提供了可靠得多的不确定性估计（见表1和表2）。 方法通过高效的关节相关性建模，提供了可靠得多的不确定性估计（见表1和表2）。 ### 4.4 消融研究 关于关节相关性的消融研究。我们开展了一项消融研究，旨在探究使用线性层建模关节相关性是否能提升不确定性估计效果。具体而言，我们移除了该线性层，从而得到一个基于独立性假设构建的概率性手部关节输出空间，其中的不确定性是通过每个关节的方差来建模的。表4和表5中的定量结果支持了我们的结论：引入线性层以捕捉关节相关性，能够提升不确定性估计及3D手部姿态估计的性能。 --- ![Image](images/9a789d4c64facea98a442ebafa83b2e07e3f989f8eac1a93066488e9e12727cd.jpg)
Figure 6: Sparsification curve of AMVUR [^24]. The competing method exhibits a large discrepancy between uncertainty-based and error-based sortings.
MethodAUSC $\downarrow$AUSE $\downarrow$Pearson's $\rho$ $\uparrow$
AMVUR [^2]5861130.127
Ours50557.60.600
Table 6: Quantitative comparison with AMVUR [^24]. Our method demonstrates consistently better performance than the competing method across all uncertainty evaluation metrics.
# SamplesFreiHAND [^63]HO3Dv2 [^13]
AUSC $\downarrow$AUSE $\downarrow$Pearson's $\rho$ $\uparrow$AUSC $\downarrow$AUSE $\downarrow$Pearson's $\rho$ $\uparrow$
165352.70.39950758.60.557
565554.30.41451062.40.484
1064847.10.55150759.10.536
2564242.20.56950557.60.600
Table 7: Effect of the number of samples on uncertainty estimation. We change the number of samples in output space from 1 to 25 and evaluate the quality of estimated uncertainty.
与相关工作的比较。我们将我们的方法在HO3Dv2 [^13]数据集上的不确定性估计质量，与一种开源的竞争方法[^2]进行了比较，该方法将3D手部关节输出空间建模为概率分布，本身支持不确定性测量。如表6所示，我们的方法在不确定性估计性能方面优于该竞争方法。此外，图6中AMVUR [^2]的稀疏化曲线表明，它难以让基于不确定性的排序与实际基于误差的排序相匹配。 关于样本数量的消融实验。我们的方法在输出空间中抽取样本，并将这些样本输入回归器。我们通过改变样本数量来进行消融实验，以探究其对不确定性估计的影响。表7展示了实验结果。总体而言，样本数量越多，性能越好。综合考虑性能与计算复杂度之间的权衡，我们将默认样本数量设定为25。 ## 5 结论 本文针对现有3D手部姿态估计模型中的两个关键挑战展开研究：无法估计随机不确定性，以及缺乏融入关节相关性的不确定性建模。我们对手部3D姿态估计中的随机不确定性进行了估计与评估。我们提出了一种新的参数化方式，利用单个线性层有效捕捉手部关节的固有相关性，从而在建模关节相关性与计算效率之间取得更好的平衡。为实现这一点，我们构建了概率性手部关节输出空间，再通过该线性层将其转换为考虑相关性的空间。实验结果表明，我们方法所估计的随机不确定性与预测误差具有良好相关性，同时能保持3D手部姿态估计的性能。 --- 致谢 本研究得到了韩国政府（MSIT）资助的信息通信技术规划评估研究所（IITP）的资助（项目编号：RS-2025-25443318，物理基础智能：通过现实世界的构建与推理实现具身AGI的双重能力路径；项目编号：RS-2019-II191906，人工智能研究生院项目（POSTECH））；2024年还得到了文化体育观光部资助的韩国创意内容署文化、体育及旅游研发项目的支持（项目名称：开发无障碍体验型XR内容技术以提升肢体残疾人士参与在线活动的能力，项目编号：RS-2024-00396700，贡献率：20%）。本研究还得到了KAIST跨代协作实验室项目以及“科学和信息通信技术部”与NIPA（“高性能计算支持”项目）的支持。 ## 参考文献 [^1]: Seungryul Baek, Kwang In Kim, and Tae-Kyun Kim. Pushing the envelope for rgb-based dense 3d hand pose estimation via neural rendering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1067–1076, 2019. [^2]: Sara Bilal, Rini Akmeliawati, Momoh Jimoh El Salami, and Amir A Shafie. Vision-based hand posture detection and recognition for sign language—a study. In 2011 4th International Conference on Mechatronics (ICOM), pages 1–6. IEEE, 2011. [^3]: Lennart Bramlage, Michelle Karg, and Cristóbal Curio. Plausible uncertainties for human pose regression. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15133–15142, 2023. [^4]: Yujun Cai, Liuhao Ge, Jianfei Cai, and Junsong Yuan. Weakly-supervised 3d hand pose estimation from monocular rgb images. In Proceedings of the European conference on computer vision (ECCV), pages 666–682, 2018. [^5]: Razvan Caramalau, Binod Bhattarai, and Tae-Kyun Kim. Active learning for bayesian 3d hand pose estimation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 3419–3428, 2021. [^6]: Yu-Wei Chao, Wei Yang, Yu Xiang, Pavlo Molchanov, Ankur Handa, Jonathan Tremblay, Yashraj S Narang, Karl Van Wyk, Umar Iqbal, Stan Birchfield, et al. Dexycb: A benchmark for capturing hand grasping of objects. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9044–9053, 2021. [^7]: Ping Chen, Yujin Chen, Dong Yang, Fangyin Wu, Qin Li, Qingpei Xia, and Yong Tan. I2uv-handnet: Image-to-uv prediction network for accurate and high-fidelity 3d hand mesh modeling. In Proceedings of the IEEE/CVF international conference on computer vision, pages 12929–12938, 2021. ## [^8]: Xingyu Chen, Yufeng Liu, Yajiao Dong, Xiong Zhang, Chongyang Ma, Yanmin Xiong, Yuan Zhang, and Xiaoyan Guo. Mobrecon: Mobile-friendly hand mesh reconstruction from monocular image. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 20544–20554, 2022 --- [^9]: Seunggeun Chi, Pin-Hao Huang, Enna Sachdeva, Hengbo Ma, Karthik Ramani, and Kwonjoon Lee. Estimating ego-body pose from doubly sparse egocentric video data. arXiv preprint arXiv:2411.03561, 2024. [^10]: Hongsuk Choi, Gyeongsik Moon, and Kyoung Mu Lee. Pose2mesh: Graph convolutional network for 3d human pose and mesh recovery from a 2d human pose. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part VII 16, pages 769–787. Springer, 2020. [^11]: Armen Der Kiureghian and Ove Ditlevsen. Aleatory or epistemic? does it matter? Structural safety, 31(2):105–112, 2009. [^12]: Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020. [^13]: Yuming Du, Philippe Weinzaepfel, Vincent Lepetit, and Romain Brégier. Multi-finger grasping like humans. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1564–1570. IEEE, 2022. [^14]: Zicong Fan, Omid Taheri, Dimitrios Tzionas, Muhammed Kocabas, Manuel Kaufmann, Michael J Black, and Otmar Hilliges. Arctic: A dataset for dexterous bimanual hand-object manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12943–12954, 2023. [^15]: Hao-Shu Fang, Jiefeng Li, Hongyang Tang, Chao Xu, Haoyi Zhu, Yuliang Xiu, Yong-Lu Li, and Cewu Lu. Alphapose: Whole-body regional multi-person pose estimation and tracking in real-time. IEEE transactions on pattern analysis and machine intelligence, 45(6):7157–7173, 2022. [^16]: Liuhao Ge, Zhou Ren, Yuncheng Li, Zehao Xue, Yingying Wang, Jianfei Cai, and Junsong Yuan. 3d hand shape and pose estimation from a single rgb image. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10833–10842, 2019. [^17]: Charlotte Häger-Ross and Marc H Schieber. Quantifying the independence of human finger movements: comparisons of digits, hands, and movement frequencies. Journal of Neuroscience, 20(22):8542–8550, 2000. [^18]: Shreyas Hampali, Mahdi Rad, Markus Oberweger, and Vincent Lepetit. Honnotate: A method for 3d annotation of hand and object poses. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3196–3206, 2020. [^19]: Ankur Handa, Karl Van Wyk, Wei Yang, Jacky Liang, Yu-Wei Chao, Qian Wan, Stan Birchfield, Nathan Ratliff, and Dieter Fox. Dexpilot: Vision-based teleoperation of dexterous robotic hand-arm system. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 9164–9170. IEEE, 2020. ## [^20]: Yana Hasson, Gul Varol, Dimitrios Tzionas, Igor Kalevatykh, Michael J Black, Ivan Laptev, and Cordelia Schmid. Learning joint reconstruction of hands and manipulated objects. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11807–11816, 2019 --- [^21]: Xiaoyan Hu and Philippos Mordohai. A quantitative evaluation of confidence measures for stereo vision. IEEE transactions on pattern analysis and machine intelligence, 34(11):2121–2133, 2012. [^22]: Eddy Ilg, Ozgun Cicek, Silvio Galesso, Aaron Klein, Osama Makansi, Frank Hutter, and Thomas Brox. Uncertainty estimates and multi-hypotheses networks for optical flow. In Proceedings of the European Conference on Computer Vision (ECCV), pages 652–667, 2018. [^23]: James N Ingram, Konrad P Körding, Ian S Howard, and Daniel M Wolpert. The statistics of natural hand movements. Experimental brain research, 188:223–236, 2008. [^24]: Zheheng Jiang, Hossein Rahmani, Sue Black, and Bryan M Williams. A probabilistic attention model with occlusion-aware texture regression for 3d hand reconstruction from a single rgb image. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 758–767, 2023. [^25]: Sheng Jin, Lumin Xu, Jin Xu, Can Wang, Wentao Liu, Chen Qian, Wanli Ouyang, and Ping Luo. Whole-body human pose estimation in the wild. In European Conference on Computer Vision, pages 196–214. Springer, 2020. [^26]: Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for computer vision? Advances in neural information processing systems, 30, 2017. [^27]: Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for computer vision? In NIPS, 2017. [^28]: Dong Uk Kim, Kwang In Kim, and Seungryul Baek. End-to-end detection and pose estimation of two interacting hands. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11189–11198, 2021. [^29]: Dominik Kulon, Riza Alp Guler, Iasonas Kokkinos, Michael M Bronstein, and Stefanos Zafeiriou. Weakly-supervised mesh-convolutional hand reconstruction in the wild. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4990–5000, 2020. [^30]: Mengcheng Li, Liang An, Hongwen Zhang, Lianpeng Wu, Feng Chen, Tao Yu, and Yebin Liu. Interacting attention graph for single image two-hand reconstruction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2761–2770, 2022. [^31]: Kevin Lin, Lijuan Wang, and Zicheng Liu. End-to-end human pose and mesh reconstruction with transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1954–1963, 2021. [^32]: Kevin Lin, Lijuan Wang, and Zicheng Liu. Mesh graphormer. In Proceedings of the IEEE/CVF international conference on computer vision, pages 12939–12948, 2021. ## [^33]: Yuan Liu, Li Jiang, Dapeng Yang, and Hong Liu. Analysis of hand and wrist postural synergies in tolerance grasping of various objects. PloS one, 11(8):e0161772, 2016 --- [^34]: Yuan Liu, Bo Zeng, Ting Zhang, Li Jiang, Hong Liu, and Dong Ming. Quantitative investigation of hand grasp functionality: hand joint motion correlation, independence, and grasping behavior. Applied Bionics and Biomechanics, 2021(1):2787832, 2021. [^35]: Hao Meng, Sheng Jin, Wentao Liu, Chen Qian, Mengxiang Lin, Wanli Ouyang, and Ping Luo. 3d interacting hand pose estimation by hand de-occlusion and removal. In European Conference on Computer Vision, pages 380–397. Springer, 2022. [^36]: Gyeongsik Moon and Kyoung Mu Lee. I2l-meshnet: Image-to-lixel prediction network for accurate 3d human pose and mesh estimation from a single rgb image. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part VII 16, pages 752–768. Springer, 2020. [^37]: Gyeongsik Moon, Shoou-I Yu, He Wen, Takaaki Shiratori, and Kyoung Mu Lee. Interhand2.6m: A dataset and baseline for 3d interacting hand pose estimation from a single rgb image. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XX 16, pages 548–564. Springer, 2020. [^38]: Seong Joon Oh, Andrew C. Gallagher, Kevin P. Murphy, Florian Schroff, Jiyan Pan, and Joseph Roth. Modeling uncertainty with hedged instance embeddings. In ICLR, 2019. [^39]: Yeonguk Oh, JoonKyu Park, Jaeha Kim, Gyeongsik Moon, and Kyoung Mu Lee. Recovering 3d hand mesh sequence from a single blurry image: A new dataset and temporal unfolding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 554–563, 2023. [^40]: JoonKyu Park, Yeonguk Oh, Gyeongsik Moon, Hongsuk Choi, and Kyoung Mu Lee. Handoccnet: Occlusion-robust 3d hand mesh estimation network. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1496–1505, 2022. [^41]: Joonkyu Park, Gyeongsik Moon, Weipeng Xu, Evan Kaseman, Takaaki Shiratori, and Kyoung Mu Lee. 3d hand sequence recovery from real blurry images and event stream. In European Conference on Computer Vision, pages 343–359. Springer, 2024. [^42]: Georgios Pavlakos, Dandan Shan, Ilija Radosavovic, Angjoo Kanazawa, David Fouhey, and Jitendra Malik. Reconstructing hands in 3d with transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9826–9836, 2024. [^43]: Matteo Poggi, Filippo Aleotti, Fabio Tosi, and Stefano Mattoccia. On the uncertainty of self-supervised monocular depth estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3227–3237, 2020. [^44]: Yuzhe Qin, Yueh-Hua Wu, Shaowei Liu, Hanwen Jiang, Ruihan Yang, Yang Fu, and Xiaolong Wang. Dexmv: Imitation learning for dexterous manipulation from human videos. In European Conference on Computer Vision, pages 570–587. Springer, 2022. ## [^45]: Javier Romero, Dimitrios Tzionas, and Michael J Black. Embodied hands: Modeling and capturing hands and bodies together. arXiv preprint arXiv:2201.02610, 2022 --- [^46]: Yu Rong, Takaaki Shiratori, and Hanbyul Joo. Frankmocap: A monocular 3d whole-body pose estimation system via regression and integration. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1749–1759, 2021. [^47]: Marc H Schieber. Muscular production of individuated finger movements: the roles of extrinsic finger muscles. Journal of Neuroscience, 15(1):284–297, 1995. [^48]: Tomas Simon, Hanbyul Joo, Iain Matthews, and Yaser Sheikh. Hand keypoint detection in single images using multiview bootstrapping. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 1145–1153, 2017. [^49]: Breannan Smith, Chenglei Wu, He Wen, Patrick Peluse, Yaser Sheikh, Jessica K Hodgins, and Takaaki Shiratori. Constraining dense hand surface tracking with elasticity. ACM Transactions on Graphics (ToG), 39(6):1–14, 2020. [^50]: Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017. [^51]: Jiayi Wang, Diogo Luvizon, Franziska Mueller, Florian Bernard, Adam Kortylewski, Dan Casas, and Christian Theobalt. Handflow: Quantifying view-dependent 3d ambiguity in two-hand reconstruction with normalizing flow. arXiv preprint arXiv:2210.01692, 2022. [^52]: Donglai Xiang, Hanbyul Joo, and Yaser Sheikh. Monocular total capture: Posing face, body, and hands in the wild. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10965–10974, 2019. [^53]: Lixin Yang, Kailin Li, Xinyu Zhan, Jun Lv, Wenqiang Xu, Jiefeng Li, and Cewu Lu. Artiboost: Boosting articulated 3d hand-object pose estimation via online exploration and synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2750–2760, 2022. [^54]: Zhengdi Yu, Shaoli Huang, Chen Fang, Toby P Breckon, and Jue Wang. Acr: Attention collaboration-based regressor for arbitrary two-hand reconstruction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12955–12964, 2023. [^55]: Baowen Zhang, Yangang Wang, Xiaoming Deng, Yinda Zhang, Ping Tan, Cuixia Ma, and Hongan Wang. Interacting two-hand 3d pose and shape reconstruction from single color image. In Proceedings of the IEEE/CVF international conference on computer vision, pages 11354–11363, 2021. [^56]: Xiaoran Zhang, Daniel H Pak, Shawn S Ahn, Xiaoxiao Li, Chenyu You, Lawrence H Staib, Albert J Sinusas, Alex Wong, and James S Duncan. Heteroscedastic uncertainty estimation framework for unsupervised registration. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 651–661. Springer, 2024. ## [^57]: Yufei Zhang, Jeffrey O Kephart, and Qiang Ji. Weakly-supervised 3d hand reconstruction with knowledge prior and uncertainty guidance. In European Conference on Computer Vision, pages 106–125. Springer, 2024 --- [^58]: Christian Zimmermann and Thomas Brox. Learning to estimate 3d hand pose from single rgb images. In Proceedings of the IEEE international conference on computer vision, pages 4903–4911, 2017. [^59]: Christian Zimmermann, Duygu Ceylan, Jimei Yang, Bryan Russell, Max Argus, and Thomas Brox. Freihand: A dataset for markerless capture of hand pose and shape from single rgb images. In Proceedings of the IEEE/CVF international conference on computer vision, pages 813–822, 2019. [^60]: Binghui Zuo, Zimeng Zhao, Wenqian Sun, Wei Xie, Zhou Xue, and Yangang Wang. Reconstructing interacting hands with interaction prior from monocular images. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9054–9064, 2023.