# UniHOPE：一种用于仅手部与手-物体姿态估计的统一方法王银桥 $^{*}$ 徐浩 $^{*}$ 郑安·亨傅志伟计算机科学与工程系医学智能与XR研究所香港中文大学 {ygwang, xuhao, pheng, cwfu}@cse.cuhk.edu.hk ## 摘要从单目图像中估计手部及潜在手持物体的3D姿态是一项由来已久的挑战。然而，现有方法针对性较强，要么专注于裸手姿态估计，要么专注于手与物体交互的姿态估计。没有一种方法能够灵活处理这两种场景，且在应用于另一种场景时性能也会大幅下降。在本文中，我们提出UniHOPE，一种用于通用3D手-物体姿态估计的统一方法，能够灵活适配这两种场景。技术上，我们设计了基于抓取感知的特征融合模块来整合手-物体特征，并引入物体切换器根据抓取状态动态控制手-物体姿态估计过程。此外，为了提升无论是否存在物体时手部姿态估计的鲁棒性，我们生成了逼真的遮挡图像对，以训练模型学习由物体引起的手部遮挡情况，同时构建了多层次特征增强技术来学习不受遮挡影响的特征。在三个常用基准测试集上的大量实验表明，UniHOPE在处理仅手部与手-物体场景时均取得了最先进的性能。代码将在上发布。 ### 1. 引言从单目图像中估计手部及潜在手持物体的3D姿态是一项历史悠久的任务，在VR/AR、人机交互等领域有着广泛应用。然而，现有方法存在分化。如图1所示，手部姿态估计（HPE）方法[^5] [^9] [^25] [^36] [^39] [^49] [^64] 预测3D手部姿态时并未考虑手持物体。相反，手-物体姿态估计（HOPE）方法[^18] [^19] [^31] [^32] [^47] 假设存在手持物体，并通过额外的物体分支进行物体姿态估计。不过，即便没有物体存在，它们仍会做出预测。这两种方法都无法灵活兼顾仅手部与手-物体两种场景。表1详细分析了最先进（SOTA）HPE方法[^39] [^49] [^64] 和 ![Image](images/aa6270296342b63e00252298640b9c87af171ac167dc1a9c75e285292962a06d.jpg)

Figure 1. Existing approaches (top) for 3D hand pose estimation are either Hand Pose Estimation (HPE), which predicts hand pose only, or Hand-Object Pose Estimation (HOPE), which assumes hand-held object. Our novel UniHOPE approach (bottom) offers flexibility and robustness to handle both scenes in a unified manner.

HOPE方法[^18] [^31]的性能。我们发现，当这些方法应用于不同场景时（见“仅手部↔手-物体场景”），性能会出现明显下降，这是由于它们针对特定任务的设计所致。尽管在所有场景上训练有助于改善性能，但这会对它们原本的任务性能产生负面影响（见“全部→仅手部/手-物体场景”），反映出它们有限的泛化能力。这一观察结果促使我们亟需一种能够有效适配仅手部与手-物体场景的统一方法。在这项工作中，我们提出了UniHOPE，这是首个将HPE与HOPE统一起来的方法，它满足了两方面要求：(i) 基本标准：能够在两种场景之间自适应切换；(ii) 高级标准：无论是否存在物体，都能稳健地估计手部姿态。首先，为满足基本标准，我们提出无论手是否握有物体，都必须始终预测手部姿态；而物体姿态则仅在物体存在时才进行估计。虽然一种简单的解决方案是根据输入场景手动选择现有的最先进人体姿势估计（HPE）和手-物姿势估计（HOPE）方法，但这种方法并不理想——因为模型之间的切换会导致结果不连贯，且无法针对单一模型实现联合优化，以使其适用于两种场景。 ---

HPE	Hand-Only Scene		Hand-Only $\rightarrow$ Hand-Object Scene		All $\rightarrow$ Hand-Only Scene		All $\rightarrow$ Hand-Object Scene
HPE	J-PE $\downarrow$	V-PE $\downarrow$	J-PE $\downarrow$	V-PE $\downarrow$	J-PE $\downarrow$	V-PE $\downarrow$	J-PE $\downarrow$	V-PE $\downarrow$
HandOccNet [^39]	12.98	12.52	19.60 (-6.62)	18.95 (-6.43)	13.16 (-0.18)	12.70 (-0.18)	14.58	14.10
H2ONet [^49]	13.34	13.13	21.98 (-8.64)	21.42 (-8.30)	14.14 (-0.80)	14.00 (-0.87)	15.20	15.03
SimpleHand [^64]	14.05	13.51	18.37 (-4.32)	17.54 (-4.03)	14.63 (-0.58)	13.96 (-0.45)	14.88	14.21

HOPE	Hand-Object Scene		Hand-Object $\rightarrow$ Hand-Only Scene		All $\rightarrow$ Hand-Object Scene		All $\rightarrow$ Hand-Only Scene
HOPE	J-PE $\downarrow$	V-PE $\downarrow$	J-PE $\downarrow$	V-PE $\downarrow$	J-PE $\downarrow$	V-PE $\downarrow$	J-PE $\downarrow$	V-PE $\downarrow$
Keypoint Trans. [^18]	17.99	17.57	25.10 (-7.11)	24.40 (-6.83)	18.79 (-1.00)	18.35 (-0.78)	19.75	19.26
HFL-Net [^31]	14.61	14.13	19.39 (-4.78)	18.61 (-4.48)	14.77 (-0.16)	14.29 (-0.16)	13.61	13.10

1. Existing HRE metnoas trained on nano-only scene exnion obvious performance degradation when testing on nano-object scene (1st vs. 2nd columns). Though training on all scenes (3rd & 4th columns) helps to improve metrics on the hand-object scene (2nd vs. 4th columns), their original performance is adversely affected (1st vs. 3rd columns). The HOPE methods also exhibit a similar pattern (see bottom part). These results demonstrate the inabilities of the existing methods to flexibly handle hand-only and hand-object scenes altogether.

重要的是，我们设计了一种端到端的方法：通过估计抓取状态的置信度，利用内部物体切换器动态控制物体姿态估计，从而提升其对不同场景的适应性。然而，仅采用现有的网络架构搭配我们的物体切换器，还会遇到另一个由常用的手-物信息交互结构引发的问题 [^31] [^32] [^45]。当没有物体存在时，提取物体特征并无必要，因此无关的物体到手部特征传递会损害手部姿态估计的准确性。为解决这一问题，我们设计了基于抓取感知的特征融合模块，利用抓取置信度在手-物特征融合过程中筛选出有效的物体特征。其次，高级标准强调无论手是否握有物体，都能实现连贯且稳健的手部姿态估计。由于手持物体常常会造成严重遮挡，因此学习遮挡不变特征对于准确恢复手部姿态至关重要。理想情况下，无遮挡手部的特征可作为具有相同姿态的遮挡手部的最优表征，因为这能降低预测难度。为实现这一点，我们提出通过从对应的无遮挡手部向遮挡手部迁移知识来训练模型。由于此类配对数据十分稀缺，我们创新性地利用基于扩散的生成模型，从原本遮挡的手部图像中生成逼真的去遮挡手部图像。随后，我们采用多层次特征增强技术，借助自蒸馏框架中来自去遮挡手部图像的信息，帮助网络模拟出遮挡不变特征。我们的主要贡献总结如下： - 我们论证了手-物姿态估计统一解决方案的必要性，并提出了UniHOPE——一种用于处理通用手-物场景的新颖方法。 - 我们设计了内部物体切换器，使其能灵活适配不同场景；同时构建了基于抓取感知的特征融合模块，可根据抓取状态自适应地利用有效的物体信息。 - 我们提出了用于提升鲁棒性的遮挡不变特征学习策略：首先利用生成式去遮挡模块制备配对的去遮挡手部图像，随后在多个层面进行特征增强。 • 我们在统一设置下对三个常用数据集进行的大量实验表明，UniHOPE达到了最先进的性能。 ### 2. 相关工作单目3D手部姿态估计（HPE）。大多数方法通过回归MANO系数来构建HPE模型[2-4, 10, 51, 57, 60-63, 65]。其他常见的表示方式包括体素 [^26] [^36] [^37] [^52]、隐式函数 [^25] 以及网格 [^8] [^9] [^29] [^49] [^64]。虽然这些方式在预测手部姿态方面表现优异，但它们并未考虑手与物体交互过程中的物体姿态估计问题，而这一点对于实际应用而言至关重要。单目3D手-物姿态估计（HOPE）。为了同时估计手部和物体的姿态，近期的研究可分为两大主流：(i) 无模板方法无需预先知晓物体的3D模型即可重建物体。Hasson等人 [^19] 利用变形的球体来恢复物体网格；Tse等人 [^45] 提出迭代优化手部和物体网格的方法。更近期的研究则利用隐式 [^11] [^12] [^24] [^54] [^56] 或神经场 [^13] 来表示手部和物体；不过由于先验数据有限，这些方法往往难以准确建模未见过的物体。(ii) 基于模板的方法假设已知物体的3D模型，重点在于回归其姿态。Liu等人 [^32] 提出了一种半监督学习方法；Lin等人 [^31] 设计了双分支骨干网络以充分利用手与物之间的相互信息。尽管这些方法取得了不错的效果，但它们仅聚焦于手-物场景，缺乏处理纯手部场景的灵活性。手-物图像合成。由于我们为统一场景生成配对的去遮挡手部图像，因此也简要回顾一下手-物图像合成的方法。基于渲染的方法 [^15] [^19] [^53] 利用常用工具 [^1] [^14] [^35] 来渲染图像，作为HPE和HOPE任务的增强数据。随着生成模型的进步[16, 21, 42–44, 48]，近期的方法能为各种用途生成更逼真的数据—— --- ![Image](images/7d3bfac0019e3c781e3911cbf33f11d29943b239aea7edd7533c6cb4d701fb55.jpg)

Figure 2. Our UniHOPE framework. (i) We first de-occlude hand images occluded by objects to form pairs, conditioned on the depth map and hand-object mask, with adaptive selection of control strength to produce high-quality samples; (ii) to accommodate both hand-only and hand-object scenes, our object switcher dynamically controls the object output by predicting grasping status, which guides the feature fusion module to eliminate irrelevant object features; and (iii) to robustly estimate hand pose, our multi-level feature enhancement techniques utilize paired data to learn occlusion-invariant hand features.

包括图像生成 [^22] [^38] [^59]、物体先验引导 [^55]、手部形变重建 [^33] 以及训练数据增强 [^40] [^50]。在本研究中，我们提出生成配对的去遮挡手部图像，以帮助学习对遮挡具有不变性的手部特征，从而提升模型在处理纯手部场景和手-物场景时的鲁棒性。 ### 3. 方法 #### 3.1. 概述与以往的研究不同，我们针对的是更通用的场景：即无论手部是否抓取物体，模型始终预测手部姿态；若存在物体，模型还会估计其姿态。图2展示了我们的UniHOPE流程。首先，我们提出以端到端的方式动态估计手-物体姿态（见图2中的红色部分），其中物体切换器能灵活控制输出，而基于抓取感知的特征融合模块则会整合与抓取相关的物体信息（见第3.2节）。接下来，为了提升对遮挡情况的鲁棒性，我们首先设计了生成式去遮挡模块，通过自适应调整控制强度来生成高质量的配对数据（见蓝色部分），随后利用多级特征增强技术学习不受遮挡影响的特征（见黄色部分，第3.3节）。最后，我们在第3.4节详细介绍了该方法中使用的损失函数。 #### 3.2. 动态手-物体姿态估计为了同时适配手部姿态估计（HPE）和手-物体姿态估计（HOPE），关键在于在存在物体时估计物体姿态的同时，始终准确预测手部姿态。直接结合现有的HPE和HOPE方法并不够，因为模型切换会带来不一致性，且缺乏联合优化机制。在本节中，我们提出了端到端的动态手-物体姿态估计方法。为了实现对物体输出的灵活控制，我们引入了可预测抓取状态的物体切换器，其训练标签为自动生成的。此外，我们提出了一种由抓取状态引导的基于抓取感知的特征融合模块，以避免在没有物体存在时出现物体到手部的不相关特征转换。抓取标签制备。为了支持模型训练，我们自动制备抓取状态标签，因为现有数据集[^6] [^17]并未提供此类信息。参照[^50]的方法，我们计算初始帧和当前帧中物体姿态之间的各向同性相对旋转误差（RRE）和相对平移误差（RTE）： $$ \mathrm{R R E}=\operatorname{a r c c o s}(\frac{\operatorname{t r a c e}(\xi_{R}^{t}{}^{\top}\xi_{R}^{0}-1)}{2}),\mathrm{R T E}=||\xi_{T}^{t}-\xi_{T}^{0}||_{2}, $$ 其中$\xi_R \in \mathbb{R}^{3 \times 3}$和$\xi_T \in \mathbb{R}^3$分别表示物体的旋转矩阵和平移向量，上标0和t代表帧索引。如果计算出的误差超过设定阈值，则该物体被标记为已抓取。物体切换器。借助制备好的标签，我们使用多层感知机（MLP）$g(\cdot)$根据物体特征$\mathbf{F}_p^o$预测抓取状态，该特征由特征金字塔网络（FPN）[^30]从输入图像中提取。这一过程由二元交叉熵损失函数进行监督： $$ \mathcal{L}^{s}=-\sum_{j=0}^{1}\mathbb{1}(\hat{G}=j)\cdot\log\frac{\exp(g(\mathbf{F}_{p}^{o})_{j})}{\sum_{k=0}^{1}\exp(g(\mathbf{F}_{p}^{o})_{k})}, $$ 其中$\hat{G}$是真实的抓取标签，$1(\cdot)$是指示函数。测试阶段，若预测结果为未抓取，则关闭物体姿态估计分支，从而为手-物体交互提供更准确的响应。 --- 基于抓取感知的特征融合。以往研究表明，手与物体之间的特征交互能有效提升手-物体场景下的性能[^31] [^32] [^45]；然而，由于缺乏物体，这种交互会干扰仅含手部场景中的手部特征学习（见表1）。为了减轻无关物体特征的干扰，我们设计了基于抓取感知的特征融合机制。训练期间，只有当预测物体已被抓取时，才会将RoIAlign$\mathbf{F}_r^o$输出的物体特征$\mathbf{F}_p^h$与手部特征[^20]拼接成特征$\mathbf{F}^H$。随后，$\mathbf{F}^H$会经过多头注意力模块[^46]处理，最终得到融合后的手-物体特征$\mathbf{F}^{OH}$： $$ \begin{aligned}{\mathbf{F}^{H}=\operatorname{C o n c a t}(\mathbf{F}_{r}^{h},s\mathbf{F}_{r}^{o}+(1-s)\mathbf{F}_{r}^{h})\quad\mathrm{a n d}}\\ {\mathbf{F}^{O H}=\operatorname{S o f t m a x}(\frac{\mathbf{F}^{H}\mathbf{F}^{H^{T}}}{\sqrt{d^{H}}})\mathbf{F}^{H},}\\ \end{aligned} $$ 其中，$s=\mathrm{Argmax}(g(\mathbf{F}_{p}^{o}))$ 表示预测的抓取状态。Concat($\cdot$) 和 Softmax($\cdot$) 分别表示沿通道维度的拼接操作和 softmax 操作。$d^{H}$ 是 $\mathbf{F}^{H}$ 的通道维度。这种方法使网络能够灵活切换物体输出，同时在不同场景下保持对手部特征的稳健表征。随后，与 [^31] 类似，$\mathbf{F}^{OH}$ 会被输入到沙漏结构的手部编码器中，以生成与 MANO 相关的特征并回归 2D 手部关节坐标。 #### 3.3. 遮挡不变特征学习手部在与物体交互时经常被遮挡。为了实现稳健的估计，针对同一手姿提取的手部特征应具备遮挡不变性，且不受物体存在的影响。鉴于估计裸手姿态比估计被所持物体遮挡的手部姿态更容易，我们的核心思路是让网络通过跨域知识迁移，从遮挡手部特征中模拟出非遮挡手部特征。因此，我们提出为受物体遮挡影响的训练样本生成合理的去遮挡手部图像对，并采用自适应调整策略来控制生成强度，以最大化生成质量。此外，我们设计了多级特征增强技术，利用这些配对数据来促进全面的手部特征学习。生成式去遮挡模块。对于遮挡的手部图像，我们的目标是去除所抓取的物体并保留手部姿态，从而创建配对数据。受 [^33] 和 [^34] 的启发，我们利用在合成手部图像上预训练的 ControlNet [^58]，在渲染的手部深度图 D 引导下对手部-物体区域 M 进行重绘。具体而言，遵循潜在扩散模型 [^42] 的流程，原始图像 X 首先通过变分自编码器 $\mathbf{x}_0$ 投影到潜在空间，得到 [^27]。随后我们按照 [^21] 中描述的标准前向扩散过程进行操作。在每个反向步骤 $t \in \{T, T-1, ..., 1\}$ 中，为了保留已知的背景区域 $(1 - \mathbf{M}) \odot \mathbf{X}$，我们可以交替使用 $(1 - \mathbf{m}) \odot \mathbf{x}_t$ 对应的特征，同时保持其分布的正确属性，因为从 $\mathbf{x}_t$ 到 $\mathbf{x}_{t-1}$ 的转变仅依赖于 $\mathbf{x}_t$，即： $$ \mathbf{x}_{t-1}^{b g}\sim\mathcal{N}\big(\sqrt{\bar{\alpha}_{t}}\mathbf{x}_{0},(1-\bar{\alpha}_{t}\mathbf{I})\big), $$ 其中，$\mathbf{m}$ 是从 $\mathbf{M}$ 下采样得到的，用于潜在空间中的计算，而 $\odot$ 表示逐元素乘积。$\mathcal{N}(\cdot)$ 表示高斯分布。$\bar{\alpha}_t$ 表示步骤 t 时的总噪声方差，其定义见 [^44]。对于未知的手部-物体区域 $\mathbf{M}\odot\mathbf{X}$，我们使用 DDIM 采样器 [^44] 执行反向扩散过程，即： $$ \mathbf{x}_{t-1}^{{h o}}=\operatorname{D D I M}(\epsilon_{\theta}(\mathbf{x}_{t},\mathbf{x}_{{m a s k}},\mathbf{D})), $$ 其中，$\mathbf{x}_{mask}$ 是被 $\mathbf{m}$ 掩码处理的潜在特征。$\epsilon_{\theta}(\cdot)$ 表示去噪模型。因此，一个反向步骤中 $\mathbf{x}_{t-1}$ 的最终表达式为： $$ \mathbf{x}_{t-1}=\mathbf{m}\odot\mathbf{x}_{t-1}^{{h o}}+(1-\mathbf{m})\odot\mathbf{x}_{t-1}^{{b g}}, $$ 这意味着 $\mathbf{x}_{t-1}^{bg}$ 是利用已知背景像素采样的，而 $\mathbf{x}_{t-1}^{ho}$ 则来自裸手数据分布。它们通过手部-物体掩码组合成新的 $\mathbf{x}_{t-1}$，以确保一致性和真实性。经过迭代的反向过程后，最终的去噪向量 $\mathbf{x}_0$ 会被送入解码器 [^27]，从潜在特征中恢复图像。我们在图 3 中展示了不同遮挡条件下的图像及其去遮挡后的对应结果。自适应控制强度调整。为了在生成手图像的条件一致性与其真实感之间取得平衡，用户通常需要手动调节生成模型中的控制强度参数。我们在图4中展示了使用不同控制强度生成的一些示例。在某些情况下，如顶部一行所示，过小的控制强度会导致生成的手部与深度条件不匹配（见(a-b)），而过大的控制强度则会产生不真实的外观（见(d)）。相反，在其他情况下，比如底部一行，需要较大的控制强度才能确保手部结构正确（见(h)）。因此，固定的控制强度无法普遍适用，而针对每种情况手动设置该值又不切实际。为此，我们提出自适应且自动地调整控制强度以提升生成质量。具体而言，我们首先定义候选控制强度$\{s_1, s_2, ..., s_n \mid 0 < s_i \leq 1, \forall i\}$，并为每种强度生成去遮挡图像。接着，我们利用来自[^9]的预训练手部重建模型，从生成的图像中估计3D手部姿态，并评估其J-PE值与真实值的差异。J-PE值最低的生成样本会连同原始样本相同的真实标签一起被纳入训练过程，例如图4中的(c)和(h)即被选中。这种方法通过为每种情况选取能最佳平衡真实感与一致性的适当控制强度，从而最大化生成质量。 --- ![Image](images/221720d2c2c6af7d3e2c3a1d2e0180c9d48622c9e15952b85ced8ec4affbde35.jpg) ![Image](images/5c47962df7408f16caffe161e2616e8601d149a0ff0e4f225b2c11ed046867d1.jpg)

Figure 3. De-occluded examples in various occlusion conditions.

多层级特征增强。给定原始图像及其对应的生成图像(X, X')，我们的目标是利用X'中的信息来增强X中的手部特征表示。为实现这一点，我们在单一网络内以自蒸馏的方式引入成对特征约束。为了全面提升手部遮挡信息恢复能力，如图2所示，我们在手部分支的多个层级上增强手部特征：(i) 从FPN初始层提取的初始级特征$\mathbf{F}_i^h$，用于捕捉手部的低层级信息；(ii) 经RoI对齐操作后的FPN输出——RoI级特征$\mathbf{F}_r^h$，用于与物体特征进行自适应融合；(iii) 在MANO解码器之前提取的MANO级特征$\mathbf{F}_m^h$，它是回归MANO系数最相关的特征。由于MANO级特征处于相对较晚的阶段，遮挡部分与去遮挡部分的特征空间可能并不一致。受[^7]的启发，我们采用多头注意力模块$h(\cdot)$作为适配层以改善知识传递。总体而言，特征增强约束被表述为X与X'特征之间的L1损失： $$ \begin{array}{c c}{\mathcal{L}_{{i n i t}}^{{e n h}}=||\mathbf{F}_{i}^{h}-\mathbf{F}_{i}^{h^{\prime}}||_{1},}&{\mathcal{L}_{{R o I}}^{{e n h}}=||\mathbf{F}_{r}^{h}-\mathbf{F}_{r}^{h^{\prime}}||_{1},}\\ \end{array} $$ $$ \begin{array}{r l}{\mathrm{a n d}}&{\mathcal{L}_{M A N O}^{e n h}=||h(\mathbf{F}_{m}^{h})-h(\mathbf{F}_{m}^{h^{\prime}})||_{1},}\end{array} $$ 其中带有撇号的特征属于生成图像。遮挡感知的类别过滤。在从生成样本到原始样本的知识迁移过程中，我们发现：如果原始手部已经未被遮挡，特征学习过程可能无法从中获益。此时，去遮挡后的手部特征与遮挡状态下的手部特征之间的知识差距会消失，导致特征约束侧重于缩小仿真到现实的域差异，从而使原始特征趋近于仿真特征，最终造成性能不佳。为解决这一问题，我们过滤掉未被遮挡的样本以进行配对特征增强。具体而言，我们计算所提供的非模态手部掩码（考虑物体造成的遮挡情况）与渲染出的完整手部掩码之间的交并比（IoU），以此作为真实的遮挡比例。在训练期间，仅对原始遮挡比例 $\hat{O}$ 超过预设阈值 $\tau$ 的样本对施加特征约束。

Figure 4. Visualization of our adaptive control strength adjustment.

此外，不抓取样本也会被排除，因为它们无法提供额外信息。因此，特征增强仅针对抓取且被遮挡的样本进行，即 $$ \mathcal{L}_{*}^{e n h}=\mathbb{1}((\hat{O}\geq\tau)\land(\hat{G}=1))\cdot\mathcal{L}_{*}^{e n h}, $$ 其中 $\tau = 0.1$ 为我们的实验设定值。$\mathcal{L}_{*}^{enh}$ 代表多层级的特征增强约束。 #### 3.4. 损失函数我们的训练损失包含与 [^31] 类似的手部和物体损失，以及我们的物体切换器损失 $\mathcal{L}^s$，还有特征增强约束 $\mathcal{L}_{init}^{enh}$、$\mathcal{L}_{RoI}^{enh}$ 和 $\mathcal{L}_{MANO}^{enh}$。首先，手部损失计算方式为 $\mathcal{L}^h = \mathcal{L}^J + \mathcal{L}^V + \mathcal{L}_{MANO}^{MANO}$，其中 $$ \begin{align*}\mathcal{L}^{J}&=||\mathbf{J}^{2D}-\hat{\mathbf{J}}^{2D}||_{2}+||\mathbf{J}^{3D}-\hat{\mathbf{J}}^{3D}||_{2},\\\mathcal{L}^{V}=&||\mathbf{V}-\hat{\mathbf{V}}||_{2},\text{and}\mathcal{L}^{MANO}=||(\theta;\boldsymbol{\beta})-(\hat{\theta};\hat{\boldsymbol{\beta}})||_{2}.\end{align*} $$ 此处 $\mathbf{J}^{2D}$、$\mathbf{J}^{3D}$ 和 V 分别代表二维关节坐标、三维关节坐标和三维顶点坐标。$(\theta;\beta)$ 表示MANO系数。上标“^”表示真实标签。物体损失 $\mathcal{L}^{o}$ 用于监督二维位置（由三维物体关键点投影而来）及其对应的置信度，这些结果来自图像网格提议 [^41]，即 $$ \mathcal{L}^{o}=\sum_{g}\sum_{k=1}^{N^{o}}(\vert\vert p_{g,k}-\hat{p}_{g,k}\vert\vert_{1}+\vert\vert c_{g,k}-\hat{c}_{g,k}\vert\vert_{1}), $$ 其中 $N^{o}$ 是物体网格三维边界框内的关键点数量。$p_{g,k}$ 和 $c_{g,k}$ 分别是网格 g 和控制点 k 处的像素位置与置信度值。上标“^”表示真实值。我们仅针对抓取图像计算物体损失，因为这类图像包含用于姿态估计的完整物体。总体而言，训练损失如下所示。 $$ \begin{aligned}\mathcal{L}^{total}=&\underbrace{\mathcal{L}^{h}+\mathcal{L}^{o}+\alpha\mathcal{L}^{s}}_{sample-wise}\\&+\underbrace{\gamma_{init}\mathcal{L}_{init}^{enh}+\gamma_{RoI}\mathcal{L}_{RoI}^{enh}+\gamma_{MANO}\mathcal{L}_{MANO}^{enh}}_{pair-wise}\end{aligned} $$ 其中 $\alpha$ 和 $\gamma_{*}$ 是用于平衡各项损失的权重。 ---

	Methods	All Scenes				Hand-Only Scene				Hand-Object Scene
	Methods	J-PE $\downarrow$	PA-J-PE $\downarrow$	V-PE $\downarrow$	PA-V-PE $\downarrow$	J-PE $\downarrow$	PA-J-PE $\downarrow$	V-PE $\downarrow$	PA-V-PE $\downarrow$	J-PE $\downarrow$	PA-J-PE $\downarrow$	V-PE $\downarrow$	PA-V-PE $\downarrow$
HPE	HandOccNet [^39]	14.02	6.17	13.55	5.95	13.16	5.31	12.70	5.11	14.58	6.73	14.10	6.49
	MobRecon [^9]	15.02	6.71	13.93	5.91	14.43	5.88	13.45	5.15	15.40	7.25	14.24	6.39
	H2ONet [^49]	14.78	5.72	14.63	6.19	14.14	4.74	14.00	5.35	15.20	6.35	15.03	6.74
	SimpleHand [^64]	14.78	6.30	14.11	6.03	14.63	5.62	13.96	5.38	14.88	6.74	14.21	6.45
HOPE	Liu et al. [^32]	15.33	6.17	14.79	5.98	15.18	5.48	14.60	5.31	15.43	6.61	14.91	6.40
	Keypoint Trans. [^18]	19.16	7.70	18.71	7.96	19.75	7.59	19.26	7.98	18.79	7.77	18.35	7.94
	HFL-Net [^31]	14.32	6.08	13.83	5.86	13.61	5.20	13.10	5.01	14.77	6.64	14.29	6.41
Unified	H2ONet $^{\dagger}$ + HFL-Net $^{\dagger}$	14.12	5.83	13.75	5.83	13.29	4.70	13.08	5.06	14.66	6.55	14.18	6.33
	H2ONet $^{\dagger}$ + HFL-Net $^{\ddagger}$	14.54	5.90	14.19	6.00	14.14	4.76	14.00	5.35	14.79	6.63	14.31	6.41
	HandOccNet $^{\dagger}$ + HFL-Net $^{\dagger}$	14.34	6.06	13.86	5.85	13.85	5.28	13.35	5.08	14.66	6.56	14.18	6.34
	HandOccNet $^{\ddagger}$ + HFL-Net $^{\ddagger}$	14.52	6.15	14.02	5.93	14.09	5.37	13.58	5.17	14.79	6.65	14.31	6.42
	UniHOPE (ours)	13.03	5.59	12.59	5.40	12.59	4.83	12.12	4.66	13.31	6.08	12.89	5.87

Table 2. Hand-pose estimation results on DexYCB. $^{\dagger}$: pre-trained in the original setting. $^{\ddagger}$: re-trained in the unified setting. The best and second-best are marked in bold and $\underline{\text{underlined}}$. Our UniHOPE attains leading performance for almost all metrics in all scenarios.

Methods	gelatin_box	bleach_cleanser	wood_block	average $\uparrow$
Liu et al. [^32]	26.31	25.07	68.56	38.89
Keypoint Trans. [^18]	0.00	1.31	32.61	10.47
HFL-Net [^31]	25.88	32.08	70.16	41.59
H2ONet $^{\dagger}$ + HFL-Net $^{\dagger}$	29.26	30.71	64.84	40.69
H2ONet $^{\dagger}$ + HFL-Net $^{\dagger}$	26.12	29.33	69.40	40.51
HandOccNet $^{\dagger}$ + HFL-Net $^{\dagger}$	29.30	30.57	64.94	40.69
HandOccNet $^{\dagger}$ + HFL-Net $^{\dagger}$	26.22	29.51	69.40	40.61
UniHOPE (ours)	26.23	32.32	74.29	43.06

Table 3. Unseen object-pose estimation results on DexYCB.

	Methods	J-PE $\downarrow$	J-AUC $\uparrow$	V-PE $\downarrow$	V-AUC $\uparrow$	F@5 $\uparrow$	F@15 $\uparrow$
HPE	HandOccNet [^39]	28.94	47.96	28.10	49.23	23.34	67.86
	MobRecon [^9]	29.61	48.10	28.65	49.59	23.38	67.72
	H2ONet [^49]	30.46	47.09	29.55	48.31	22.10	66.04
	SimpleHand [^64]	29.01	47.58	28.09	49.00	21.93	65.90
HOPE	Liu et al. [^32]	29.54	47.55	28.66	48.83	21.82	67.34
	Keypoint Trans. [^18]	41.04	34.47	39.64	36.02	17.13	56.07
	HFL-Net [^31]	28.45	50.34	27.55	51.57	24.31	69.88
Unified	H2ONet $^{\dagger}$ + HFL-Net $^{\dagger}$	31.27	47.70	30.31	48.86	22.91	67.20
	H2ONet $^{\ddagger}$ + HFL-Net $^{\ddagger}$	28.49	50.45	27.59	51.67	24.33	69.86
	HandOccNet $^{\dagger}$ + HFL-Net $^{\dagger}$	30.96	47.85	30.01	49.02	23.04	67.47
	HandOccNet $^{\ddagger}$ + HFL-Net $^{\ddagger}$	28.33	50.45	27.44	51.67	24.39	69.99
	UniHOPE (ours)	26.23	52.26	25.41	53.52	24.64	70.77

Table 4. Hand-pose estimation results (Root-relative) on HO3D.

### 4. 实验 #### 4.1. 实验设置数据集。在统一设定中，我们根据物体抓取状态将原始数据集划分为仅手部场景和手-物场景。我们在以下常用数据集上进行实验：(i) DexYCB [^6]：我们采用更具挑战性的“S3”划分（训练集/测试集：376,374/76,360个样本），测试集中包含未见过的被抓取物体（训练集/测试集：15/3个物体）。我们报告了整个数据集（所有场景）的性能，以及仅手部场景和手-物场景的单独性能；(ii) HO3D [^17]（版本2，训练集/测试集：66,034/11,524个样本）：由于真实的3D手部标注并未公开，结果需提交至在线服务器，因此无法提供各场景的结果。此外，为了评估泛化能力，我们在(iii) FreiHAND [^65]（训练集/测试集：130,240/3,960个样本）的测试集上进行跨数据集验证，该数据集主要由裸手图像组成，缺乏用于场景划分的物体标注，因此也无法提供各场景的结果。关于DexYCB其他数据划分的更多细节和结果详见补充材料。评估指标。我们采用常用的指标来评估手部姿态估计性能，如 [^9] [^39] [^49] 所述：(i) J/V-PE表示关节/顶点位置误差的平均值（也称为MPJPE/MPVPE），单位为毫米，通过估计值与真实值之间的3D手部关节/顶点坐标的欧几里得距离来衡量；(ii) J/V-AUC计算不同误差阈值下正确关键点百分比（PCK）曲线下的面积（以百分比表示）；(iii) F@5/F@15是在5毫米/15毫米阈值下，估计值与真实值之间3D手部顶点的召回率与精确度的调和平均值（以百分比表示）。我们既报告进行Procrustes对齐（PA）前（即相对于根节点的相对位置）的指标，也报告对齐后的指标，Procrustes对齐通过全局旋转、平移和缩放调整，使估计结果与真实值对齐。对于物体姿态评估，我们测量被抓取物体的平均3D距离（ADD）。具体而言，我们报告ADD-0.5D，即ADD在物体直径50%以内的物体百分比，如 [^23] 所述，这一指标考虑了DexYCB中未见物体姿态估计的挑战。实现细节。我们在八张Nvidia RTX 2080Ti GPU上训练UniHOPE，批量大小为64，优化器采用Adam [^28]，初始学习率为 $1e^{-4}$（每10个epoch衰减0.7）。输入图像会被调整为 $128 \times 128$ 大小，并经过随机缩放、旋转、平移和颜色抖动等增强处理。为了稳定训练，我们首先用原始图像和生成图像共同训练网络30个epoch，然后再加入特征增强约束继续训练40个epoch。更多细节请参阅补充材料。 #### 4.2. 与现有最先进方法的比较我们在统一的实验设置下（使用它们官方发布的代码进行训练），将我们的UniHOPE与以往的SOTA HPE和HOPE方法进行比较 [^9] [^18] [^32] [^39] [^64]。为了提供支持 --- ![Image](images/9785431dd402a96b95543806cc93230cce3a56492cfcbbdd67d89ed886adf80b.jpg)

Figure 5. Qualitative comparison between our method and SOTA HPE/HOPE methods on hand-only/hand-object scenarios across different datasets. The first and second rows in each example denote the original view and another view, respectively, for better comparison.

![Image](images/5f174845487ed2549e25356b7d170620d2f741e6b046914035db73c1e741f30d.jpg)

Figure 6. The joint AUC comparison under different thresholds. UniHOPE achieves better performance than others, consistently.

统一任务，一种直接的解决方案是使用分类器判断手部是否正在抓取物体，然后相应地采用现有的HPE或HOPE方法。我们将这种组合记为A + B，其中A是最先进的HPE方法H2ONet [^49] 或HandOccNet [^39]，B是最先进的HOPE方法HFL-Net [^31]。 DexYCB上的评估。我们在表2中给出了DexYCB数据集上手部姿态估计的定量对比结果。我们的UniHOPE取得了整体最佳性能，显示出其在各种场景下处理一般手物交互的有效性。图6展示了不同阈值下的根部相关3D关节PCK/AUC对比情况，进一步证实了该方法的全面性能。在物体姿态估计精度方面，我们采用逐实例和平均ADD-0.5D指标进行对比，结果如表3所示。与现有的最先进HOPE方法相比，我们的方法在包含未见物体的测试集上取得了最高的平均得分，凸显了其在物体姿态估计方面的优越性。与现有最先进方法的定性对比[^31] [^39] [^49]展示于图5。在仅手部场景（见图5(a)）中，木块尚未被抓取时，现有的最先进HOPE方法[^31]由于其灵活性不足，不可避免地会产生额外的物体姿态；相比之下，我们的方法则不会，这得益于我们设计的物体切换器超过95%的抓取状态预测准确率。此外，我们的方法能生成更合理的手部姿态。在手部-物体场景（见图5(b)）中，我们的方法能生成高质量的手部-物体姿态，而以往的方法在手部出现中度遮挡时便会失效，这也体现了我们应对此类挑战的鲁棒性。 HO3D上的评估。我们在HO3D数据集上开展了相同的实验[^17]。表4展示了根部相关的定量对比结果。我们的方法在所有指标上均取得最优性能，证明了其有效性与鲁棒性。图6中的关节PCK/AUC曲线也证实了该方法始终如一的最佳表现。此外，图5(d)中的定性对比清晰展现了我们的方法在部分物体遮挡情况下估计3D手部及物体姿态的优越性。更多定量结果详见补充材料。 FreiHAND上的评估。为了评估模型的泛化能力，我们将在DexYCB上训练的模型迁移至FreiHAND测试集进行跨数据集验证。如表5所示，我们的方法优于所有现有的最先进HPE/HOPE方法，尤其在根部相关指标上优势显著。图5(c)中的定性对比也表明，UniHOPE在复杂场景下能生成更精准的手部姿态，体现出其对未见过的裸手场景更强的泛化能力。不同遮挡程度下的评估。为了展示我们的方法应对物体遮挡的鲁棒性，我们根据真实的手部-物体遮挡比例（详见第3.3节）将DexYCB测试集划分为不同的遮挡等级，并在表6中给出定量对比结果。我们的UniHOPE在所有遮挡等级下均展现出最佳的手部姿态估计性能，这凸显了我们所设计的特征增强技术的有效性。需注意，图像区域中不存在手部的测试样本已被排除在外。 ---

	Methods	Root-relative						Procrustes Alignment
	Methods	J-PE $\downarrow$	J-AUC $\uparrow$	V-PE $\downarrow$	V-AUC $\uparrow$	F@5 $\uparrow$	F@15 $\uparrow$	J-PE $\downarrow$	J-AUC $\uparrow$	V-PE $\downarrow$	V-AUC $\uparrow$	F@5 $\uparrow$	F@15 $\uparrow$
HPE	HandOccNet [^39]	58.03	29.07	56.06	29.40	14.89	47.75	14.52	71.34	14.09	72.07	40.02	88.08
	MobRecon [^9]	71.18	22.21	68.50	22.34	10.41	36.67	17.74	65.16	17.32	65.90	32.95	81.35
	H2ONet [^49]	79.14	19.88	76.29	19.38	9.69	36.71	14.56	71.21	14.25	71.69	38.73	87.69
	SimpleHand [^64]	60.82	26.09	58.85	26.34	12.84	43.53	15.79	68.90	15.37	69.61	37.03	85.40
HOPE	Liu et al. [^32]	59.40	28.84	57.39	29.04	14.92	47.76	14.00	72.30	13.59	73.02	41.53	89.11
	Keypoint Trans. [^18]	96.27	13.60	93.39	12.37	6.99	27.82	16.97	66.66	17.03	66.38	34.31	83.44
	HFL-Net [^31]	58.02	29.94	56.08	30.20	15.47	48.83	14.29	71.75	13.85	72.52	40.52	88.63
Unified	H2ONet $^{\dagger}$ + HFL-Net $^{\ddagger}$	68.88	24.27	66.49	24.21	12.33	42.38	14.47	71.36	14.10	71.99	39.48	88.14
	H2ONet $^{\dagger}$ + HFL-Net $^{\ddagger}$	68.25	24.49	65.90	24.40	12.30	42.51	14.42	71.47	14.06	72.09	39.54	88.18
	HandOccNet $^{\dagger}$ + HFL-Net $^{\ddagger}$	81.79	21.33	78.91	20.56	10.89	36.68	15.40	69.65	14.85	70.58	38.30	86.94
	HandOccNet $^{\ddagger}$ + HFL-Net $^{\ddagger}$	80.74	20.30	78.00	19.44	10.52	36.38	16.15	68.29	15.58	69.26	36.92	85.93
	UniHOPE (ours)	50.97	34.31	49.21	34.94	17.83	53.46	13.53	73.24	13.14	73.92	43.23	89.55

Table 5. Cross-dataset validation of hand-pose estimation on FreiHAND.

Methods	Occlusion (25%-50%)				Occlusion (50%-75%)				Occlusion (75%-100%)
Methods	J-PE $\downarrow$	PA-J-PE $\downarrow$	V-PE $\downarrow$	PA-V-PE $\downarrow$	J-PE $\downarrow$	PA-J-PE $\downarrow$	V-PE $\downarrow$	PA-V-PE $\downarrow$	J-PE $\downarrow$	PA-J-PE $\downarrow$	V-PE $\downarrow$	PA-V-PE $\downarrow$
HandOccNet [^39]	16.40	7.08	15.85	6.83	18.22	7.60	17.67	7.33	28.15	8.71	27.20	8.40
MobRecon [^9]	16.67	7.61	15.60	6.77	20.04	8.17	18.80	7.48	31.46	9.64	29.97	9.23
H2ONet [^49]	17.07	6.76	16.78	7.09	19.41	7.32	19.07	7.58	31.07	8.82	30.11	8.94
SimpleHand [^64]	16.43	7.05	15.78	6.75	19.33	7.55	18.38	7.39	38.52	10.57	36.85	10.40
Liu et al. [^32]	16.98	6.92	16.43	6.71	19.72	7.11	19.14	6.91	33.80	8.99	32.64	8.71
Keypoint Trans. [^18]	20.95	8.15	20.41	8.31	24.45	8.61	23.88	8.76	38.29	11.21	37.39	11.75
HFL-Net [^31]	16.33	7.00	15.81	6.77	18.66	7.33	18.11	7.11	28.95	8.80	27.94	8.53
H2ONet $^{\dagger}$ + HFL-Net $^{\dagger}$	16.07	6.84	15.57	6.67	19.39	7.40	18.82	7.22	30.32	8.43	29.27	8.28
H2ONet $^{\dagger}$ + HFL-Net $^{\dagger}$	16.43	6.93	15.94	6.79	18.76	7.29	18.25	7.14	31.02	8.55	29.91	8.45
HandOccNet $^{\dagger}$ + HFL-Net $^{\dagger}$	16.04	6.89	15.53	6.65	19.22	7.43	18.64	7.20	29.57	8.58	28.53	8.31
HandOccNet $^{\dagger}$ + HFL-Net $^{\dagger}$	16.41	7.00	15.88	6.77	18.64	7.33	18.09	7.10	29.22	8.76	28.20	8.47
UniHOPE (ours)	14.59	6.39	14.13	6.17	16.27	6.51	15.78	6.29	26.42	7.64	25.51	7.40

Table 6. Comparison with SOTA methods across different object-caused occlusion levels on DexYCB.

	Models	Root-relative		Procrustes Align.
	Models	J-PE $\downarrow$	V-PE $\downarrow$	J-PE $\downarrow$	V-PE $\downarrow$
(a)	Baseline	14.09	13.61	5.95	5.75
(b)	w/ Grasp-aware Feature Fusion	13.84	13.37	5.79	5.58
(c)	w/ Generative De-occluder	13.38	12.92	5.71	5.52
(d)	+ Image Feature Enhancement	13.23	12.79	5.64	5.44
(e)	+ RoI Feature Enhancement	13.18	12.73	5.64	5.45
(f)	+ MANO Feature Enhancement	13.12	12.67	5.63	5.43
(g)	+ Occlusion-aware Case Filtering	13.03	12.59	5.59	5.40

Table 7. Ablation study on major designs in UniHOPE.

#### 4.3. 消融实验我们在DexYCB数据集上开展消融实验，以评估各项设计的有效性，结果如表7所示。搭载我们设计的物体切换器的HFL-Net [^31]作为最简单的基线模型。基于抓取感知的特征融合。我们首先分析了基于抓取感知的特征融合模块的影响。对比(a)行与(b)行的数据可知，所有指标的性能均有提升，这说明融入无关的物体特征会对手部姿态估计产生负面影响，而我们的设计有效缓解了这一问题。生成式去遮挡模型。接下来，我们评估采用分布内手部姿态的去遮挡手部图像所产生的效果，因为它们能提供额外信息。对比(b)行和(c)行可知，引入配对数据后取得的显著改进体现了合成样本的有效性。关于自适应控制强度调整的更多细节，请参阅补充材料。遮挡不变特征学习。此外，我们展示了不同层级的特征增强所产生的单独效果。对比(c)行与(d)–(f)行可以发现，每个层级的根部相关指标均逐步改善，这证明了从去遮挡手部提取知识的有效性。最后，(g)行表明，通过我们的遮挡感知案例筛选机制，能够最大程度发挥这些效果。 ### 5. 结论我们提出了UniHOPE，这是首个针对仅手部场景和手-物体姿态估计的统一方法，其研发动机源于现有方法无法同时处理这两种场景。我们的技术创新有两点：首先，为实现在不同场景间灵活切换，我们引入了物体切换器来控制物体姿态估计，并设计了基于抓取感知的特征融合模块，以选择性地提取有效的物体特征；其次，为提升对物体导致遮挡情况的鲁棒性，我们提出了多级特征增强技术，从生成的逼真去遮挡手部图像中学习遮挡不变的手部特征。在三个常用基准测试集上的实验结果表明，UniHOPE的性能达到了当前最优水平。 --- 致谢：本研究部分得到了中国香港特别行政区研究资助局T45-401/22-N项目以及香港创新及科技基金MHP/086/21项目的支持。 ## 参考文献 [^1]: Autodesk, INC. Maya. , 2018. 2 [^2]: Seungryul Baek, Kwang In Kim, and Tae-Kyun Kim. Pushing the envelope for RGB-based dense 3D hand pose estimation via neural rendering. In CVPR, pages 1067–1076, 2019. 2 [^3]: Seungryul Baek, Kwang In Kim, and Tae-Kyun Kim. Weakly-supervised domain adaptation via GAN and mesh model for estimating 3D hand poses interacting objects. In CVPR, pages 6121–6131, 2020. [^4]: Adnane Boukhayma, Rodrigo de Bem, and Philip H.S. Torr. 3D hand shape and pose from images in the wild. In CVPR, pages 10843–10852, 2019. 2 [^5]: Zhe Cao, Ilija Radosavovic, Angjoo Kanazawa, and Jitendra Malik. Reconstructing hand-object interactions in the wild. In ICCV, pages 12417–12426, 2021. 1 [^6]: Yu-Wei Chao, Wei Yang, Yu Xiang, Pavlo Molchanov, Ankur Handa, Jonathan Tremblay, Yashraj S. Narang, Karl Van Wyk, Umar Iqbal, Stan Birchfield, et al. DexYCB: A benchmark for capturing hand grasping of objects. In CVPR, pages 9044–9053, 2021. 3, 6 [^7]: Guobin Chen, Wongun Choi, Xiang Yu, Tony Han, and Manmohan Chandraker. Learning efficient object detection models with knowledge distillation. NeurIPS, 30, 2017. 5 [^8]: Xingyu Chen, Yufeng Liu, Chongyang Ma, Jianlong Chang, Huayan Wang, Tian Chen, Xiaoyan Guo, Pengfei Wan, and Wen Zheng. Camera-space hand mesh recovery via semantic aggregation and adaptive 2D-1D registration. In CVPR, pages 13274–13283, 2021. 2 [^9]: Xingyu Chen, Yufeng Liu, Yajiao Dong, Xiong Zhang, Chongyang Ma, Yanmin Xiong, Yuan Zhang, and Xiaoyan Guo. MobRecon: Mobile-friendly hand mesh reconstruction from monocular image. In CVPR, pages 20544–20554, 2022. 1, 2, 4, 6, 8 [^10]: Yujin Chen, Zhigang Tu, Di Kang, Linchao Bao, Ying Zhang, Xuefei Zhe, Ruizhi Chen, and Junsong Yuan. Model-based 3D hand reconstruction via self-supervised learning. In CVPR, pages 10451–10460, 2021. 2 [^11]: Zerui Chen, Yana Hasson, Cordelia Schmid, and Ivan Laptev. AlignSDF: Pose-aligned signed distance fields for hand-object reconstruction. In ECCV, pages 231–248. Springer, 2022. 2 [^12]: Zerui Chen, Shizhe Chen, Cordelia Schmid, and Ivan Laptev. gSDF: Geometry-driven signed distance functions for 3D hand-object reconstruction. In CVPR, pages 12890–12900, 2023. 2 [^13]: Hongsuk Choi, Nikhil Chavan-Dafle, Jiacheng Yuan, Volkan Isler, and Hyunsoo Park. HandNeRF: Learning to reconstruct hand-object interaction scene from a single RGB image. In ICRA, pages 13940–13946. IEEE, 2024. 2 [^14]: Blender Online Community. Blender. , 2019. 2 [^15]: Enric Corona, Albert Pumarola, Guillem Alenya, Francesc Moreno-Noguer, and Grégory Rogez. GanHand: Predicting human grasp affordances in multi-object scenes. In CVPR, pages 5031–5041, 2020. 2 [^16]: Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11):139–144, 2020. 2 [^17]: Shreyas Hampali, Mahdi Rad, Markus Oberweger, and Vincent Lepetit. HOnnotate: A method for 3D annotation of hand and object poses. In CVPR, pages 3196–3206, 2020. 3, 6, 7 [^18]: Shreyas Hampali, Sayan Deb Sarkar, Mahdi Rad, and Vincent Lepetit. Keypoint transformer: Solving joint identification in challenging hands and object interactions for accurate 3D pose estimation. In CVPR, pages 11090–11100, 2022. 1, 2, 6, 8 [^19]: Yana Hasson, Güil Varol, Dimitrios Tzionas, Igor Kalevatykh, Michael J. Black, Ivan Laptev, and Cordelia Schmid. Learning joint reconstruction of hands and manipulated objects. In CVPR, pages 11807–11816, 2019. 1,2 [^20]: Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask R-CNN. In ICCV, pages 2961–2969, 2017. 4 [^21]: Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurIPS, pages 6840–6851, 2020. 2, 4 [^22]: Hezhen Hu, Weilun Wang, Wengang Zhou, and Houqiang Li. Hand-object interaction image generation. NeurIPS, 35:23805–23817, 2022. 3 [^23]: Yinlin Hu, Pascal Fua, and Mathieu Salzmann. Perspective flow aggregation for data-limited 6D object pose estimation. In ECCV, pages 89–106. Springer, 2022. 6 [^24]: Di Huang, Xiaopeng Ji, Xingyi He, Jiaming Sun, Tong He, Qing Shuai, Wanli Ouyang, and Xiaowei Zhou. Reconstructing hand-held objects from monocular video. In SIGGRAPH Asia 2022 Conference Papers, pages 1–9, 2022. 2 [^25]: Lin Huang, Chung-Ching Lin, Kevin Lin, Lin Liang, Lijuan Wang, Junsong Yuan, and Zicheng Liu. Neural voting field for camera-space 3D hand pose estimation. In CVPR, pages 8969–8978, 2023. 1,2 [^26]: Umar Iqbal, Pavlo Molchanov, Thomas Breuel Juergen Gall, and Jan Kautz. Hand pose estimation via latent 2.5D heatmap regression. In ECCV, pages 118–134, 2018. 2 [^27]: Diederik P. Kingma. Auto-encoding variational Bayes. arXiv preprint arXiv:1312.6114, 2013. 4 [^28]: Diederik P. Kingma and Lei Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015. 6 [^29]: Dominik Kulon, Riza Alp Guler, Iasonas Kokkinos, Michael M. Bronstein, and Stefanos Zafeiriou. Weakly-supervised mesh-convolutional hand reconstruction in the wild. In CVPR, pages 4990–5000, 2020. 2 ## [^30]: Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In CVPR, pages 2117–2125, 2017. 3 --- [^31]: Zhifeng Lin, Changxing Ding, Huan Yao, Zengsheng Kuang, and Shaoli Huang. Harmonious feature learning for interactive hand-object pose estimation. In CVPR, pages 12989–12998, 2023. 1, 2, 4, 5, 6, 7, 8 [^32]: Shaowei Liu, Hanwen Jiang, Jiarui Xu, Sifei Liu, and Xiaolong Wang. Semi-supervised 3D hand-object poses estimation with interactions in time. In CVPR, pages 14687–14697, 2021. 1, 2, 4, 6, 8 [^33]: Wenquan Lu, Yufei Xu, Jing Zhang, Chaoyue Wang, and Dacheng Tao. HandRefiner: Refining malformed hands in generated images by diffusion-based conditional inpainting. In ACM MM, 2024. 3, 4 [^34]: Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In CVPR, pages 11461–11471, 2022. 4 [^35]: Matthew Matl. Pyrender. , 2019. 2 [^36]: Gyeongsik Moon and Kyoung Mu Lee. I2L-MeshNet: Image-to-pixel prediction network for accurate 3D human pose and mesh estimation from a single RGB image. In ECCV, pages 752–768, 2020. 1, 2 [^37]: Gyeongsik Moon, Shoou-I Yu, He Wen, Takaaki Shiratori, and Kyoung Mu Lee. InterHand2.6M: A dataset and baseline for 3D interacting hand pose estimation from a single RGB image. In ECCV, pages 548–564, 2020. 2 [^38]: Supreeth Narasimhaswamy, Uttaran Bhattacharya, Xiang Chen, Ishita Dasgupta, Saayan Mitra, and Minh Hoai. HanDiffuser: Text-to-image generation with realistic hand appearances. In CVPR, pages 2468–2479, 2024. 3 [^39]: JoonKyu Park, Yeonguk Oh, Gyeongsik Moon, Hongsuk Choi, and Kyoung Mu Lee. HandOccNet: Occlusion-robust 3D hand mesh estimation network. In CVPR, pages 1496–1505, 2022. 1, 2, 6, 7, 8 [^40]: Junho Park, Kyeongbo Kong, and Suk-Ju Kang. AttentionHand: Text-driven controllable hand image generation for 3D hand reconstruction in the wild. arXiv preprint arXiv:2407.18034, 2024. 3 [^41]: Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In CVPR, 2016. 5 [^42]: Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, pages 10684–10695, 2022. 2, 4 [^43]: Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In ICML, pages 2256–2265, 2015. [^44]: Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020. 2, 4 [^45]: Tze Ho Elden Tse, Kwang In Kim, Ales Leonardis, and Hyung Jin Chang. Collaborative learning for hand and object reconstruction with attention-guided graph convolution. In CVPR, pages 1664–1674, 2022. 2, 4 [^46]: Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. NeurIPS, pages 6000–6010, 2017. 4 [^47]: Rong Wang, Wei Mao, and Hongdong Li. Interacting hand-object pose estimation via dense mutual attention. In WACV, pages 5735–5745, 2023. 1 [^48]: Yinhui Wang, Jiwen Yu, and Jian Zhang. Zero-shot image restoration using denoising diffusion null-space model. ICLR, 2023. 2 [^49]: Hao Xu, Tianyu Wang, Xiao Tang, and Chi-Wing Fu. H2ONet: Hand-occlusion-and-orientation-aware network for real-time 3D hand mesh reconstruction. In CVPR, pages 17048–17058, 2023. 1, 2, 6, 7, 8 [^50]: Hao Xu, Haipeng Li, Yinqiao Wang, Shuaicheng Liu, and Chi-Wing Fu. HandBooster: Boosting 3D hand-mesh reconstruction by conditional synthesis and sampling of hand-object interactions. In CVPR, pages 10159–10169, 2024. 3 [^51]: Lixin Yang, Jiasen Li, Wenqiang Xu, Yiqun Diao, and Cewu Lu. BiHand: Recovering hand mesh with multi-stage bisected hourglass networks. In BMVC, 2020. 2 [^52]: Linlin Yang, Shicheng Chen, and Angela Yao. SemiHand: Semi-supervised hand pose estimation with consistency. In ICCV, pages 11364–11373, 2021. 2 [^53]: Lixin Yang, Kailin Li, Xinyu Zhan, Jun Lv, Wenqiang Xu, Jiefeng Li, and Cewu Lu. ArtiBoost: Boosting articulated 3D hand-object pose estimation via online exploration and synthesis. In CVPR, pages 2750–2760, 2022. 2 [^54]: Yufei Ye, Abhinav Gupta, and Shubham Tulsiani. What’s in your hands? 3D reconstruction of generic objects in hands. In CVPR, pages 3895–3905, 2022. 2 [^55]: Yufei Ye, Xueting Li, Abhinav Gupta, Shalini De Mello, Stan Birchfield, Jiaming Song, Shubham Tulsiani, and Sifei Liu. Affordance diffusion: Synthesizing hand-object interactions. In CVPR, pages 22479–22489, 2023. 3 [^56]: Yufei Ye, Abhinav Gupta, Kris Kitani, and Shubham Tulsiani. G-HOP: Generative hand-object prior for interaction reconstruction and grasp synthesis. In CVPR, pages 1911–1920, 2024. 2 [^57]: Baowen Zhang, Yangang Wang, Xiaoming Deng, Yinda Zhang, Ping Tan, Cuixia Ma, and Hongan Wang. Interacting two-hand 3D pose and shape reconstruction from single color image. In ICCV, pages 11354–11363, 2021. 2 [^58]: Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In ICCV, pages 3836–3847, 2023. 4 [^59]: Mengqi Zhang, Yang Fu, Zheng Ding, Sifei Liu, Zhuowen Tu, and Xiaolong Wang. HOIDiffusion: Generating realistic 3D hand-object interaction data. In CVPR, pages 8521–8531, 2024. 3 [^60]: Xiong Zhang, Qiang Li, Hong Mo, Wenbo Zhang, and Wen Zheng. End-to-end hand mesh recovery from a monocular RGB image. In ICCV, pages 2354–2364, 2019. 2 ### [^61]: Xiong Zhang, Hongsheng Huang, Jianchao Tan, Hongmin Xu, Cheng Yang, Guozhu Peng, Lei Wang, and Ji Liu. Hand image understanding via deep multi-task learning. In ICCV, pages 11281–11292, 2021 --- [^62]: Zimeng Zhao, Xi Zhao, and Yangang Wang. TravelNet: Self-supervised physically plausible hand motion learning from monocular color images. In ICCV, pages 11666–11676, 2021. [^63]: Yuxiao Zhou, Marc Habermann, Weipeng Xu, Ikhsanul Habibie, Christian Theobalt, and Feng Xu. Monocular real-time hand shape and motion capture using multi-modal data. In CVPR, pages 5346–5355, 2020. 2 [^64]: Zhishan Zhou, Shihao Zhou, Zhi Lv, Minqiang Zou, Yao Tang, and Jiajun Liang. A simple baseline for efficient hand mesh reconstruction. In CVPR, pages 1367–1376, 2024. 1, 2, 6, 8 #### [^65]: Christian Zimmermann, Duygu Ceylan, Jimei Yang, Bryan Russell, Max Argus, and Thomas Brox. FreiHAND: A dataset for markerless capture of hand pose and shape from single RGB images. In ICCV, pages 813–822, 2019. 2, 6 --- ##### UniHOPE：一种针对仅手部场景和手-物体姿态估计的统一方法补充材料在这份补充材料中，我们提供了更多定性和定量结果，用以展示UniHOPE的能力与鲁棒性（见第A节）。第B节介绍了实现细节，第C节则讨论了该方法的局限性与未来研究方向。 ##### A. 更多实验结果 ##### A.1. 定性结果首先，我们展示了图A至图D，这些图表明UniHOPE能够同时处理仅手部场景（左列）和手-物体场景（右列）。与当前最优方法的对比：接下来，我们在DexYCB数据集（图E）、HO3D数据集（图F）以及FreiHAND数据集（图G）上提供了更多定性对比结果。更多去遮挡示例：此外，我们在图H中展示了更多去遮挡样本。 ### A.2. 定量结果表1的补充指标：主论文中表1的补充指标详见表A。无论PA前后，现有HPE/HOPE模型在迁移应用到另一种场景或在原始场景中测试时，整体性能均出现退化，即便在两种场景上重新训练后也是如此。 DexYCB其他划分集上的对比：我们在表B和表C中分别给出了DexYCB数据集默认“S0”划分集（训练集与测试集分布相同）以及包含未见受试者的“S1”划分集（训练集/测试集：7/2名受试者）上的手部姿态估计定量结果。我们的方法取得了最佳的整体性能，尤其是在根部相关指标方面。在HO3D数据集上的对比结果。HO3D上的剩余手部指标详见表D。尽管HFL-Net [^9]以及H2ONet与HFL-Net的组合在PA指标上表现更佳，但我们的方法在仅经过尺度平移对齐后的指标上仍大幅优于它们 [^4]，这种对齐方式同时考虑了全局旋转和手部形状因素。我们强调全局旋转的重要性，因为它能更好地反映可视化质量，如图F所示的定性对比结果便证明了这一点。 ### A.3. 性能详细分析在本研究中，我们探索了一种同时解决HPE和HOPE问题的新框架。即便在所有场景上重新训练，直接应用现有的HPE/HOPE最先进模型也并非最佳选择，因为这些模型缺乏针对性的设计。对于仅含手部的场景，即使没有物体被抓取，HOPE方法也会受到无关物体特征的影响；而HPE方法则可能无法处理未见过的手部姿态。对于手-物体场景，HOPE方法缺乏应对严重遮挡的有效设计，而HPE方法又未利用物体信息来提升性能。我们的方法在各类场景中均表现更优。如图I所示：(a)当手伸出抓取物体时，我们所提出的基于抓取感知的特征融合机制能降低未抓取物体的负面影响；(b)对于FreiHAND数据集中出现的未见过的手部姿态，我们生成的解遮挡图像能引入更丰富的手部姿态以提升性能；(c)我们的多级特征增强机制能在严重物体遮挡情况下提高模型的鲁棒性；(d)在抓取物体时，我们的方法通过利用物体信息超越了HPE方法。这些观察结果与主论文中表2、5、6所呈现的定量性能一致。 #### A.4. 额外的消融研究为了与主论文保持一致，我们在DexYCB数据集上开展了下述所有消融研究。表7的补充结果。由于RHD [^22]和静态手势数据集 [^1]被用于微调ControlNet [^12]，我们也在DexYCB训练前，针对这些合成数据集进行了预训练消融研究，所用网络结构与我们基线模型相同，且包含基于抓取感知的特征融合模块（主论文中表7的第(b)行）。如表F所示，直接将合成数据集纳入训练只能带来微小的性能提升，这反映出合成图像与真实世界图像之间的域差异所带来的局限性。相反，我们的抗遮挡特征学习策略借助ControlNet [^20]提供的基础数据先验以及多级特征增强机制，显著提升了模型性能。自适应控制强度调整的消融实验。控制强度（取值范围为0到1）作用于ControlNet与Stable Diffusion之间的连接，用于控制输出与控制信号的一致性程度。我们提出利用在DexYCB上预训练的MobRecon [^3]来自适应调整该数值，从而避免繁琐的人工调参。[^12]中采用的默认控制强度为0.55。在本研究中，我们依据经验从$\{0.25, 0.4, 0.55, 0.7, 0.85, 1.0\}$中选取候选控制强度，其候选数量与[^12]中的设置相近。 ---

Hand-Only Scene			Hand-Object Scene
![Image](images/5ee69c069870f5118c9bddc8baa5d5c098865efb92ecd9809cbf88cc4903698a.jpg)	![Image](images/c6c7874f0636a3664fe57f6ac8baddf4bfa26a7e7f9da544ae94f622dc5ad08d.jpg)		![Image](images/bd10b677f5b80284626ef4242f7c57ba8d08af314aaf690ca09f934a1ba3b35c.jpg)	![Image](images/2569eb35ab750b2a8bf58ae55c36f61374d263a5d64883a497513f434d922f01.jpg)	![Image](images/39daab4eba8a8dfadd00e87eb3bff39e741b4ebfb0e298953598f814b355af35.jpg)
![Image](images/a2c763ccc853d64c623903b92816e64bdbd115d96a8712e6927ca4ddf7932e73.jpg)	![Image](images/5efe5c361c3138d94c83aa54bc3731b02dbea3745875a7a22360cdd7c1f5690f.jpg)	![Image](images/e05f9d0a979d1a263d6b0040861c3ec63a64cef7cc31bf879aaab39325a95097.jpg)	![Image](images/faf27b8f46cdaba05d0f0bd2055adaac112307eb2139bf221b979e515d1a90f4.jpg)	![Image](images/3824d72bfd2cd1ca3eaee97ae7826b8c6076a8fca20ae6df4e3a449150630eeb.jpg)	![Image](images/b411a4e744e0b12e305c2723b75a0cdaeea8706bc215a20aab466f1447e0dbb5.jpg)
![Image](images/b9c92fe89792d0a3cc6eaa9d8450e7d3022125463f84f3e8e8547bd7bf67696e.jpg)	![Image](images/93e27e54e5ffeaea5d36cb7886219362201e71d806ebb71e18e76975d96df3fb.jpg)	![Image](images/9a27d1cba12524602a3b96f0e5c7515d87bc5af9542e6b9b49db674bfe4b2a8c.jpg)	![Image](images/054dedddfa97ff7c8681c5473d2cdd9acb9cf9adbcb11cd353e99528aa67599d.jpg)	![Image](images/0e061168cc6c10ac64ba5fc2cb3aa90eb4618b20620b8f5609c127f715d29f62.jpg)	![Image](images/f5ab0adb7e0c6ca2d2ba357b72b0d6b0e58aebf4e95c6f9008dde18a7dd07e4a.jpg)
![Image](images/1ba52f01c63426d2ac2e709b2a3adf1b60fc6d1d8ad72a8af997ad13a452f412.jpg)	![Image](images/19eed946acfa46f9b433ab4d9f4f178a94f22caef2e76844ebaced2f45194052.jpg)	![Image](images/0890fb4e68a913982a3a34610ab390d35190d363e5ae2cbf91f6da9c6c353b3f.jpg)	![Image](images/220ff67f9181feb81d8243453a9b48cc6f361f5ce3a5b86a295ae300e5c54e7a.jpg)	![Image](images/4fe518958e356a243be1710e844b518465f0d1cdbb380007e2139d59873b0a20.jpg)	![Image](images/6d57d67141338804e407736fbd232be17e42fab2c8460a67ebf45fd67869a72b.jpg)
![Image](images/17a40413144e9b95b1f696acc77773a3fd9de696e8445c759d6fc632519af12f.jpg)	![Image](images/33736e412113a5cb2c7f7a5798d761774b99c839e4c78d1231a09497b76abca5.jpg)	![Image](images/4f06e4b868f4d56a49e32507bcc96ef18e973335c7ecbd9f796b99e04450b5a3.jpg)	![Image](images/e86979569888fb9a6370e58e7277d7a54ab1463b95e73695667efd80bc7661ff.jpg)	![Image](images/af1b8c49e87bcb671fd43268c44e78975edce536f15c9e324c679a0c899b6e0a.jpg)	![Image](images/1f313fb2bd8504a16bc19a6157a4e1e732361d5bf0ae8e5daf7ebf1ef1fb8c2f.jpg)
Input	Output (view 1)	Output (view 2)	Input	Output (view 1)	Output (view 2)

Figure A. UniHOPE is able to handle both hand-only (left column) and hand-object scenarios (right column). Here, we show more qualitative results on DexYCB. For each example, the estimation results are rendered from the original (view 1) and another view (view 2) for clear visualization.

---

Hand-Only Scene

![Image](images/c958f0ac7e81bee4c0d81ae0c22c85c2d6ac3ebdb09f2529922752973cf324ef.jpg)	![Image](images/8789dc49623a41aeb80d65c854cd6f2cb42f8e73bada3d9be508c5ce201f6e7d.jpg)	![Image](images/e2c95a726c1099d6434966dc4578a32413c8f9efb46a908f4eb263c8e716c5af.jpg)	![Image](images/e897af396696e8bd69f6bfcff1c5cb76096eba1e9685f6e0407e3f537660bf50.jpg)	![Image](images/3690edd6bdc80762aa7f3630f9f625013609f296a271fb813c57b11c1f5d7ab9.jpg)
![Image](images/8da7856e454a5412d51cc8c55e806746be636c7cdf2b7fb1f614c6bf823bef97.jpg)	![Image](images/8f63c7131caecb015b132f1e6f7787cf5c14aea5ba96156f311918f7430ec00a.jpg)	![Image](images/08b52398ae4d0ec686ec6945e591444a5d16105642046e44c62f0e91c671df1b.jpg)	![Image](images/239cabb47b12925b0b765d5e4d481e3f264986ad2160afda52ed69f445de4eb6.jpg)	![Image](images/f153d78bc3091e33ded26fd52595dc66ed137ed7e268a78bd1c39db438425d3e.jpg)
![Image](images/029bbf92fa0baf9ba5f0e574db4d2780686cce233675b48b2e76cde53b6e67cd.jpg)	[F: [F8]	![Image](images/dd21cb0adfffeb44f096f1faf5f3de0e7cfa2f4b8621f675a25537afd47ebbbd.jpg)	![Image](images/e48807c76750f6cd1c803ba2733e114f369cdbf349ee5a4023b75f0b0155ac0e.jpg)	![Image](images/a00e6720d7ed21d19066c85d73f91155aba151b7826c7dc6dbfd3fc09adcd615.jpg)
![Image](images/3d86b2989bce683284d6fc3076263e55246588dadc58834df669c16e320850f5.jpg)	![Image](images/15790d24e7d2218ad802f221522c448d018e01b0b6572329f07da8135ebabbca.jpg)	![Image](images/5105f140aead30d959ba5c53b351d24a6649dcfe8622b65a9b9887ad16e2f056.jpg)	![Image](images/fcc56a26b91fa0c43df24d3c99091209f1728ee453a7147aa67a1f2643396df3.jpg)	![Image](images/946c317866de5eefadc6723f7ee6ab722ee9f8260d74a311f33629c78d41db22.jpg)
![Image](images/a1ef879eb55490ed1812458de251e65a5626e62c66e98a361af2ccf8dd4d55eb.jpg)	![Image](images/f4c19744b634a1a72ade8142df7c99fd00034fe4f491f17a6f50c2aeec4df657.jpg)	![Image](images/4bc0ca30c8c5f7896fa94ab463c0f0833cb9472af7f349b027efe7178250b3b2.jpg)	![Image](images/d4664d87a74714536d50417457c2b532e7cd70c5c17383ef2bec1927a7282e08.jpg)	![Image](images/3261264f3393e3592112e57104341de06c40dcf1750af32684753f0d4c57c2f3.jpg)

Hand-Object Scene

输入输出（视图1）输出（视图2）输入输出（视图1）输出（视图2）

Figure B. More qualitative results of UniHOPE on DexYCB.

---

Hand-Only Scene	Hand-Object Scene
![Image](images/1376909366b264f17ffab8fe0008b79514e07ae1fe3de139e41ea7d58387ae4b.jpg) ![Image](images/b63b195bea4d3f3794ac3e79afec130ecd41b121bcd8872b3aa47a82ac670a93.jpg)	![Image](images/edc21350dbb7751617269f1404da1c3b878d00c870ef16c4d7dde2246033e5d3.jpg) ![Image](images/79adad6c50dd892d53d56ae84ca43bca3e190053d4dccbbb552a7614dede8af6.jpg) ![Image](images/95e4303f688b546502fdf3d748e79061e7ac788b78bb740b70a6c39b8178d713.jpg)
![Image](images/be9fada8f3f217eb3902cdf6ef6982362ef32a0b49a5ee9e5fef05fb2a398dd2.jpg) ![Image](images/f8ed3795d0587bff26b11d7c67bc50b23b2b00bdece91846c66f6d91922043b3.jpg)	![Image](images/18845b5b2d1452ea985b8347c58011c072c7c5d4418f3ab7c498673b6b34d149.jpg) ![Image](images/ef88618fcda73b0d85c4197a4b43ae81130929036914eba4be786e967197ea49.jpg) ![Image](images/d0b352d46551ba7d082d1ff05157f3eaaf44cbd821fc7aaf813a6d748448555d.jpg)
![Image](images/3001f508a9c9b92a3f79c3e6f11b81c8b50e3953ff2089a1ed1847ad850982c4.jpg) ![Image](images/3af1bd19ffd4746af5aca38f03d51f4e6e1af9ebf4809d9bbaa776cbc4671034.jpg) ![Image](images/40eaae80e25fae727ca11f4f20823af2e28b7c352aac6f245de97128f2526992.jpg)	![Image](images/6915aef7fc95f0555899a6d8ecb5818edeaa22127795cbbcdf74a274be3fd791.jpg) ![Image](images/5f7f4fcc85f6499a28bac2c5ea4817ab30240c980eb4f52449845b4953ddad4d.jpg) ![Image](images/fca920c1e0237b5c94973172f8ed9c74592a0f09b7a0974d7dba611c58221ae0.jpg)
![Image](images/30899a98629026a4ef6c731332c925be08efd51128e41ecd3bba66a2a987e59f.jpg)	![Image](images/ae68942926afdac751c1bdb81fd1994ff9bba41c9303cd350486989de76bb6ba.jpg) ![Image](images/965fed0401833a8d28759f4f2628862fc0326f44864942436bfd9011587928a8.jpg) ![Image](images/4ee4d5756101b86dc04e037c6de2c9128116e08f09f8caa6f225a10511d8801d.jpg)
![Image](images/78d38a81e351f25995515f58d2dd8b9021b1d5adf502182ea47da044cec84f55.jpg) ![Image](images/70971528431402c68e8a8e3d65df1c64b9d457decd76e691f5b494431e974771.jpg) ![Image](images/8d08f0845b0e83c732015bdb3fe47d0839aa8bb53b2c2b7f4e5910f578b0b3f0.jpg)	![Image](images/08e29797b55fa7d791a6cc36c27dae595cba16eacd4d32984ddf15593617289b.jpg) ![Image](images/6742aa9e8c436a05ca4ed3b256a51f78bd52ceef0e59647369abd28eebe51bb0.jpg) ![Image](images/8c65f303ad5cc395ac1d7f28bd927574188167b3b3a147b4ca0cd450ff480023.jpg)
Input Output (view 1)	Output (view 2)

Figure C. More qualitative results of UniHOPE across hand-only (left column) and hand-object scenarios (right column) on HO3D.

--- #### 仅含手部的场景

![Image](images/80bcf885e4cb023d287da36cc29e53eb5707b5e2da61dc55a3470947f266ded7.jpg)	![Image](images/19bce30635f74b1296f1736a98b24b0e915c5dd3ef9e6660c610e27ee3db1031.jpg)	![Image](images/d7bcb4dde55b072c6616085c1c70071e231dbdc674207f06ec246f496e65e799.jpg)	![Image](images/a8605fb09e96d5eb3908d80e7dca6a6c8f7006f76024f2b94c1922a0988cf505.jpg)
![Image](images/3a2e9bd3d43f77775adc9df39733f42d29116b9a535776d2bc34f4c075d17b5b.jpg)		![Image](images/de0122711fd59a8211fe2a0de5f08eaa90938d59a63fa0ad4e6dc717fad4efab.jpg)	![Image](images/96994bf92a2b8798ebb1442ec4f5a1649d43af7dc032299adb5dfee4a269efb3.jpg) ![Image](images/cbfff91939f9f52ad91965b48794940cf218e824046f262ea5d59e5bfdcb233d.jpg)
![Image](images/1a5beb6f5e952cb230058ab877ea7cb2a25d24e903f9466671862c93510f6bd5.jpg)	![Image](images/5853d679460478c2123a62a688ac1a554916355fc09a9d071ed7a7abc77aca30.jpg)	![Image](images/fefef0a3a60d73c9afe403a547c6f2f6d95ef5575e8944b40817b5b34047ae26.jpg)	[F3]
![Image](images/6a541e4e50f1d76ea58735f48b7242d19d7718b7a1cd3908a49f5bd0973d3869.jpg)	![Image](images/000f35b804c337c85745301087d1849912e780c1bed6544e8d0c49747ec67779.jpg)	![Image](images/7a53ef5e94e7c0b8e5b39eeff155ff4924a53c7aa5eb20e30293b643997ee19e.jpg)	![Image](images/8e04774e592a9931b3d4d58fa892b6afc87516b7f9d9e80f2dad5d0390c76eee.jpg)
![Image](images/bfd590ed5c1270685e12474de4a247c11fb18c845e38ef7eb316bf84c9abee38.jpg)	![Image](images/47b3dd28ced004e94fc99a9ed52ed8959d5bc27ce1b91a12ec837365a60a1598.jpg)	![Image](images/97fe3fa7f41ca4729fbf53ef72cb2e5dc945ac2d6f3a380da9e015743a58a7f8.jpg)	![Image](images/c8a0675478e748d49689be33aa3b36f14425bbb024c42f747c2cad66888abc4d.jpg)

#### 手-物体场景输入输出（视图1）输出（视图2）输入输出（视图1）输出（视图2）

Figure D. More qualitative results of UniHOPE on HO3D.

---

HPE	Hand-Only Scene				Hand-Only → Hand-Object Scene					All → Hand-Only Scene					All → Hand-Object Scene
HPE	J-PE $\downarrow$	PA-J-PE $\downarrow$	V-PE $\downarrow$	PA-V-PE $\downarrow$	J-PE $\downarrow$	PA-J-PE $\downarrow$	V-PE $\downarrow$	PA-V-PE $\downarrow$	J-PE $\downarrow$	PA-J-PE $\downarrow$	V-PE $\downarrow$	PA-V-PE $\downarrow$	J-PE $\downarrow$	PA-V-PE $\downarrow$	V-PE $\downarrow$	PA-V-PE $\downarrow$	J-PE $\downarrow$	PA-J-PE $\downarrow$	V-PE $\downarrow$	PA-V-PE $\downarrow$
[^14]	12.98	5.21	12.52	5.02	19.60	(-6.62)	7.71 (-2.50)	18.95 (-6.43)	7.42 (-2.40)	13.16 (-0.18)	5.31 (-0.10)	12.70 (-0.18)	5.11 (-0.09)	14.58	6.73	14.10	6.49
[^18]	13.34	4.69	13.13	5.05	21.98	(-8.64)	7.13 (-2.44)	21.42 (-8.30)	7.27 (-2.22)	14.14 (-0.80)	4.74 (-0.05)	14.00 (-0.87)	5.35 (-0.30)	15.20	6.35	15.03	6.74
[^21]	14.05	5.55	13.51	5.31	18.37	(-4.32)	7.42 (-1.87)	17.54 (-4.03)	6.91 (-1.60)	14.63 (-0.58)	5.62 (-0.07)	13.96 (-0.45)	5.38 (-0.07)	14.88	6.74	14.21	6.45
HOPE	Hand-Object Scene				Hand-Object → Hand-Only Scene					All → Hand-Object Scene					All → Hand-Only Scene
HOPE	J-PE $\downarrow$	PA-J-PE $\downarrow$	V-PE $\downarrow$	PA-V-PE $\downarrow$	J-PE $\downarrow$	PA-J-PE $\downarrow$	V-PE $\downarrow$	PA-V-PE $\downarrow$	J-PE $\downarrow$	PA-J-PE $\downarrow$	V-PE $\downarrow$	PA-V-PE $\downarrow$	J-PE $\downarrow$	PA-J-PE $\downarrow$	V-PE $\downarrow$	PA-V-PE $\downarrow$	V-PE $\downarrow$	PA-V-PE $\downarrow$	V-PE $\downarrow$	PA-V-PE $\downarrow$
[^5]	17.99	7.68	17.57	7.88	25.10	(-7.11)	7.62 (+0.06)	24.40 (-6.83)	7.88 (-0.00)	18.79 (-1.00)	7.77 (-0.09)	18.35 (-0.78)	7.94 (-0.06)	19.75	7.59	19.26	7.98
[^9]	14.61	6.56	14.13	6.33	19.39 (-4.78)	5.96 (+0.60)	18.61 (-4.48)	5.75 (-0.58)	14.77 (-0.16)	6.64 (-0.08)	14.29 (-0.16)	6.41 (-0.08)	6.41 (-0.08)	13.61	5.20	13.10	5.01

Table A. Full metrics of Tab.1 in the main paper.

	Methods	All Scenes				Hand-Only Scene				Hand-Object Scene
	Methods	J-PE $\downarrow$	PA-J-PE $\downarrow$	V-PE $\downarrow$	PA-V-PE $\downarrow$	J-PE $\downarrow$	PA-J-PE $\downarrow$	V-PE $\downarrow$	PA-V-PE $\downarrow$	J-PE $\downarrow$	PA-J-PE $\downarrow$	V-PE $\downarrow$	PA-V-PE $\downarrow$
HPE	HandOccNet [^14]	13.04	5.85	12.61	5.65	13.42	5.39	12.95	5.20	12.79	6.15	12.39	5.95
	MobRecon [^3]	14.34	6.50	13.40	5.74	14.57	5.91	13.74	5.29	14.18	6.88	13.19	6.03
	H2ONet [^18]	13.89	5.38	13.56	5.52	14.10	4.84	13.75	5.02	13.76	5.73	13.43	5.84
	SimpleHand [^21]	13.66	6.02	13.14	5.78	14.48	5.67	13.95	5.46	13.13	6.24	12.62	5.99
HOPE	Liu et al. [^11]	14.06	5.75	13.57	5.58	14.87	5.47	14.33	5.30	13.53	5.93	13.08	5.75
	Keypoint Trans. [^5]	16.61	6.84	16.21	7.05	18.50	7.03	18.00	7.32	15.39	6.71	15.05	6.88
	HFL-Net [^9]	13.02	5.58	12.58	5.39	13.41	5.19	12.92	5.00	12.77	5.84	12.35	5.64
Unified	H2ONet $^{\dagger}$ + HFL-Net $^{\dagger}$	13.08	5.47	12.71	5.43	13.81	4.85	13.50	5.06	12.61	5.87	12.20	5.68
	H2ONet $^{\dagger}$ + HFL-Net $^{\dagger}$	13.30	5.45	12.91	5.40	14.09	4.85	13.74	5.02	12.79	5.83	12.37	5.64
	HandOccNet $^{\dagger}$ + HFL-Net $^{\dagger}$	13.32	5.73	12.87	5.54	14.40	5.50	13.89	5.30	12.63	5.89	12.22	5.69
	HandOccNet $^{\dagger}$ + HFL-Net $^{\dagger}$	13.43	5.71	12.97	5.51	14.41	5.49	13.90	5.30	12.80	5.85	12.38	5.65
	UniHOPE (ours)	12.59	5.54	12.17	5.36	12.84	5.02	12.38	4.85	12.42	5.88	12.03	5.69

Table B. Quantitative comparison on DexYCB "S0" split.

	Methods	All Scenes				Hand-Only scene				Hand-Object scene
	Methods	J-PE $\downarrow$	PA-J-PE $\downarrow$	V-PE $\downarrow$	PA-V-PE $\downarrow$	J-PE $\downarrow$	PA-J-PE $\downarrow$	V-PE $\downarrow$	PA-V-PE $\downarrow$	J-PE $\downarrow$	PA-J-PE $\downarrow$	V-PE $\downarrow$	PA-V-PE $\downarrow$
HPE	HandOccNet [^14]	18.33	6.95	17.70	6.71	19.70	6.01	18.95	5.81	17.57	7.47	17.02	7.21
	MobRecon [^3]	18.62	7.18	17.73	6.61	19.36	6.27	18.42	5.75	18.21	7.68	17.36	7.09
	H2ONet [^18]	18.40	6.40	17.90	6.57	18.92	5.44	18.36	5.70	18.11	6.93	17.64	7.05
	SimpleHand [^21]	17.38	6.82	16.81	6.73	18.86	6.02	18.14	5.92	16.57	7.26	16.08	7.17
HOPE	Liu et al. [^11]	17.82	6.46	17.19	6.25	19.12	5.89	18.36	5.69	17.10	6.77	16.54	6.55
	Keypoint Trans. [^5]	21.61	8.15	21.18	8.36	22.84	7.32	22.24	7.59	20.93	8.61	20.60	8.79
	HFL-Net [^9]	17.77	6.58	17.16	6.36	18.42	5.72	17.72	5.52	17.41	7.06	16.86	6.82
Unified	H2ONet $^{\dagger}$ + HFL-Net $^{\dagger}$	17.49	6.36	16.94	6.25	19.24	5.50	18.60	5.59	16.54	6.83	16.02	6.61
	H2ONet $^{\dagger}$ + HFL-Net $^{\dagger}$	17.96	6.48	17.41	6.42	18.92	5.45	18.35	5.69	17.44	7.05	16.89	6.82
	HandOccNet $^{\dagger}$ + HFL-Net $^{\dagger}$	17.84	6.53	17.22	6.31	20.18	5.95	19.39	5.75	16.55	6.85	16.03	6.62
	HandOccNet $^{\dagger}$ + HFL-Net $^{\dagger}$	18.63	6.73	17.99	6.50	20.72	6.11	19.91	5.91	17.48	7.06	16.93	6.83
	UniHOPE (ours)	16.84	6.42	16.25	6.20	17.80	5.50	17.11	5.30	16.31	6.93	15.79	6.70

Table C. Quantitative comparison on DexYCB "S1" split.

为了评估我们提出的自适应控制强度调整策略的有效性，我们将该模型（主论文中表7的第(c)行）与那些在固定控制强度下、未引入特征增强约束而用生成样本训练的模型进行对比。如表E所示，与几种不同的控制强度相比，我们的自适应策略在手部姿态估计任务上取得了最佳性能。所有候选控制强度下生成的样本如图J所示，这说明了针对不同场景需要自适应地选择控制强度。超参数的影响。主论文中公式(11)里的超参数$\alpha$的默认值根据经验被设为10，目的是确保预测准确率超过95%。对于控制三个不同层级特征增强的超参数，我们在表G中评估了它们对手部姿态估计性能的影响。由于MANO级特征是用于直接回归最终手部姿态的后期特征，因此我们部署了一个适配层来提升知识迁移效果。我们将$\gamma_{MANO}$的取值设得更大，旨在强力推动这一特征适配过程。在我们的实验中，$\gamma_{init}$、$\gamma_{RoI}$和$\gamma_{MANO}$的取值分别被设为0.1、0.1和0.5。 ### A.5. 计算成本与效率我们的模型在八个Nvidia RTX 2080Ti GPU上的训练时间分别为：DexYCB数据集（37.6万个样本）需3天，HO3D数据集（6.6万个样本）需12小时。表H列出了各种模型的推理速度（FPS，在单块NVIDIA RTX 2080Ti GPU上测试）、FLOPs以及参数数量。 ---

	Methods	Procrustes Alignment						Scale-Translation Aligned
	Methods	J-PE $\downarrow$	J-AUC $\uparrow$	V-PE $\downarrow$	V-AUC $\uparrow$	F@5 $\uparrow$	F@15 $\uparrow$	J-PE $\downarrow$	J-AUC $\uparrow$
HPE	HandOccNet [^14]	10.26	7.95	10.21	79.61	50.61	94.47	28.18	49.28
	MobRecon [^3]	10.47	79.14	10.76	78.54	47.57	93.59	29.36	49.36
	H2ONet [^18]	9.52	80.97	9.60	80.81	52.62	95.09	29.67	48.53
	SimpleHand [^21]	11.28	77.66	11.58	77.05	45.78	91.74	28.41	49.32
HOPE	Liu et al. [^11]	9.46	81.12	9.39	81.25	54.93	95.64	28.44	49.79
	Keypoint Trans. [^5]	12.00	76.24	12.18	75.83	44.71	91.60	40.00	36.36
	HFL-Net [^9]	9.01	82.02	8.92	82.18	57.01	96.19	27.97	51.33
Unified	H2ONet $^{\dagger}$ + HFL-Net $^{\dagger}$	9.49	81.04	9.43	81.16	54.54	95.54	30.60	48.93
	H2ONet $^{\dagger}$ + HFL-Net $^{\ddagger}$	8.97	82.10	8.88	82.26	57.08	96.22	28.00	51.44
	HandOccNet $^{\dagger}$ + HFL-Net $^{\dagger}$	9.56	80.89	9.50	81.02	54.23	95.47	30.29	49.09
	HandOccNet $^{\ddagger}$ + HFL-Net $^{\ddagger}$	9.05	81.94	8.96	82.10	56.79	96.14	27.83	51.45
	UniHOPE (ours)	9.60	80.82	9.45	81.12	52.57	95.68	25.53	53.70

Table D. Quantitative comparison (Procrustes Alignment & Scale-Translation Aligned) on HO3D.

Control Strength Selection	Root-relative		Procrustes Align.
Control Strength Selection	J-PE $\downarrow$	V-PE $\downarrow$	J-PE $\downarrow$	V-PE $\downarrow$
s = 0.4	13.76	13.30	5.85	5.65
s = 0.55	13.51	13.06	5.78	5.57
s = 0.7	13.43	12.98	5.75	5.55
Adaptive Adjustment (ours)	13.38	12.92	5.71	5.52

Table E. Quantitative results of our adaptive control strength adjustment vs. fixed control strengths.

Models	Root-relative		Procrustes Align.
	J-PE $\downarrow$	V-PE $\downarrow$	J-PE $\downarrow$	V-PE $\downarrow$
Baseline	13.84	13.37	5.79	5.58
w/ Grasp-aware Feature Fusion
w/ RHD [^22] & Static Gestures [^1]	13.79	13.32	5.73	5.53
Ours	13.03	12.59	5.59	5.40

Table F. Comparison with directly training with synthetic datasets used by [^12].

$\gamma_{init}/\gamma_{RoI}/\gamma_{MANO}$	Root-relative		Procrustes Align.
$\gamma_{init}/\gamma_{RoI}/\gamma_{MANO}$	J-PE $\downarrow$	V-PE $\downarrow$	J-PE $\downarrow$	V-PE $\downarrow$
0.001/0.001/0.005	13.17	12.72	5.61	5.41
0.01/0.01/0.05	13.13	12.69	5.62	5.42
0.1/0.1/0.5 (ours)	13.03	12.59	5.59	5.40
1.0/1.0/5.0	13.15	12.70	5.70	5.50
10.0/10.0/50.0	14.13	13.65	6.08	5.87

Table G. Effects of various hyperparameters of the multi-level feature constraints.

得益于UniHOPE中的轻量级对象切换器，UniHOPE的推理效率和模型复杂度与HFL-Net相近 [^9]。与其他SOTA模型相比，UniHOPE的模型规模和运行速度适中，能够满足实时应用的需求。 #### B. 实现细节场景划分。遵循[^19]的方法，在抓取标签制备过程中，RRE和RTE的阈值分别设为$5^{\circ}$和10毫米。如果图像被判定为无抓取情况，则归类为仅手部场景；否则归类为手-物体场景。两种场景下的样本数量如表I所示。需要注意的是，尽管FreiHAND [^23]的训练集和测试集中都包含少量与物体交互的图像，但由于缺乏物体标注信息，无法对其进行此类划分。生成式去遮挡模型。我们采用了官方发布的预训练权重[^12]，该权重通过合成手部图像对ControlNet进行微调[^1] [^22]。通过对3D手部与物体的渲染掩码进行膨胀操作来获得手-物体掩码，以确保重绘时能覆盖手-物体区域。接着，我们以手-物体区域为中心，裁剪训练集中的原始输入图像，并将其缩放至$512 \times 512$尺寸。将手-物体图像及对应的手-物体掩码输入到修复型Stable Diffusion模型中，同时以手部深度图作为条件。此外，针对来自两个实验室基准测试集的图像生成任务[^2] [^4]，我们采用正向提示词“一只手正在抓握，室内环境，实验室中”，而负向提示词则与[^12]中的设置类似。推理阶段，DDIM的反向扩散步数默认设为50步。网络结构。(i) 主干网络：参照[^9]的做法，我们采用ResNet50 [^6]作为主干网络来提取输入图像的特征；该网络采用双流结构，以减轻手部特征与物体特征之间的竞争。(ii) 手部编码器：手部编码器以$F^{OH}$为输入，首先利用沙漏网络[^13]来回归特征图以及2D关节热图；随后通过卷积层和逐元素相加操作融合这两类数据，再经过四个残差块处理，最终输出一个1024维的向量。(iii) MANO解码器：它包含两层全连接层，用于根据手部编码器产生的特征预测手部姿态和 ---

Methods	HandOccNet [^14]	MobRecon [^3]	H2ONet [^18]	SimpleHand [^21]	Liu et al. [^11]	Keypoint Trans. [^5]	HFL-Net [^9]	H2ONet + HFL-Net	HandOccNet + HFL-Net	Ours
FPS	48	78	62	41	51	33	43	36	30	44
FLOPs	15.48G	0.46G	0.74G	9.96G	39.44G	12.66G	10.01G	0.77G / 10.04G	15.51G / 10.04G	10.04G
# Param.	37.22M	8.23M	25.88M	48.89M	34.48M	52.79M	46.08M	72.26M	83.60M	46.38M

Table H. Efficiency comparison with previous methods. Note that FLOPs for the “A+B” methods depend on the predicted grasping status, therefore reported as “FLOPs of (classifier + A) / FLOPs of (classifier + B)”.

Datasets (splits)	Training Set			Test Set
Datasets (splits)	All Scenes	Hand-Only Scene	Hand-Object Scene	All Scenes	Hand-Only Scene	Hand-Object Scene
DexYCB "S0"	401,507	153,210	248,297	78,768	30,848	47,920
DexYCB "S1"	351,943	138,775	213,168	104,128	36,912	67,216
DexYCB "S3"	376,374	145,051	231,323	76,360	29,912	46,448
HO3D	66,034	5,595	60,439	11,524	2,971	8,553
FreiHAND	130,240	N/A	N/A	3,960	N/A	N/A

Table I. Number of samples in hand-only/hand-object scenes for different datasets (splits).

MANO模型的形状参数。(iv) 物体解码器：参照[^9]的方法，将手部分支经RoIAlign处理后的特征与物体分支的特征通过交叉注意力层进行融合，以增强物体特征的学习效果。融合后的特征会依次经过六个卷积层，用以预测3D物体角点的2D投影坐标及其对应置信度。在测试阶段，利用预测的2D角点与物体网格上原始3D角点之间的对应关系，通过透视-点（PnP）算法[^8]来计算物体姿态。训练细节。参照[^9]的做法，我们对训练样本进行数据增强处理，包括随机缩放($\pm20\%$)、旋转($\pm180^{\circ}$)、平移($\pm10\%$)以及颜色抖动($\pm50\%$)。我们的训练过程分为两个阶段：第一阶段，将去遮挡后的图像纳入训练，此时不施加特征增强损失，共训练30个周期，以使模型首先适应生成数据的领域；第二阶段，在相同设置下，额外引入图像对之间的增强约束对网络进行监督，再训练40个周期。 #### C. 局限性与未来工作局限性。尽管我们能够预测未见物体的抓取状态，但由于训练数据中物体类别有限，当物体形状或外观差异较大时，其姿态估计性能往往会下降。此外，虽然大多数现有公开基准测试中均提供了物体标注信息，但某些数据集却缺乏这类标注，这限制了我们方法的适用性——因为场景划分与修复掩码的生成均需依赖这些标注。未来工作。为了提高模型对未见物体的泛化能力，一个有前景的方向是利用各种视觉基础模型所提供的相关知识先验[^7] [^10] [^15]，这些模型在零样本场景中展现出了卓越的性能。我们正在考虑的另一种提升模型泛化能力的方法$f^{n}$是利用这些数据。 ### 参考文献 [^1]: Synthesis AI. Static gestures dataset, data retrieved from synthesis ai. , 2023. 1, 7 [^2]: Yu-Wei Chao, Wei Yang, Yu Xiang, Pavlo Molchanov, Ankur Handa, Jonathan Tremblay, Yashraj S. Narang, Karl Van Wyk, Umar Iqbal, Stan Birchfield, et al. DexYCB: A benchmark for capturing hand grasping of objects. In CVPR, pages 9044–9053, 2021. 7 [^3]: Xingyu Chen, Yufeng Liu, Yajiao Dong, Xiong Zhang, Chongyang Ma, Yanmin Xiong, Yuan Zhang, and Xiaoyan Guo. MobRecon: Mobile-friendly hand mesh reconstruction from monocular image. In CVPR, pages 20544–20554, 2022. 1, 6, 7, 8 [^4]: Shreyas Hampali, Mahdi Rad, Markus Oberweger, and Vincent Lepetit. HOnnotate: A method for 3D annotation of hand and object poses. In CVPR, pages 3196–3206, 2020. 1, 7 [^5]: Shreyas Hampali, Sayan Deb Sarkar, Mahdi Rad, and Vincent Lepetit. Keypoint transformer: Solving joint identification in challenging hands and object interactions for accurate 3D pose estimation. In CVPR, pages 11090–11100, 2022. 6, 7, 8 [^6]: Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pages 770–778, 2016. 7 [^7]: Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, et al. Segment anything. In ICCV, pages 4015–4026, 2023. 8 ## [^8]: Vincent Lepetit, Francesc Moreno-Noguer, and Pascal Fua --- EPnP：一种针对PnP问题的精确O(n)解法。IJCV，81:155–166，2009年。8 [^9]: Zhifeng Lin, Changxing Ding, Huan Yao, Zengsheng Kuang, and Shaoli Huang. Harmonious feature learning for interactive hand-object pose estimation. In CVPR, pages 12989–12998, 2023. 1, 6, 7, 8 [^10]: Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. NeurIPS, 36, 2024. 8 [^11]: Shaowei Liu, Hanwen Jiang, Jiarui Xu, Sifei Liu, and Xiaolong Wang. Semi-supervised 3D hand-object poses estimation with interactions in time. In CVPR, pages 14687–14697, 2021. 6, 7, 8 [^12]: Wenquan Lu, Yufei Xu, Jing Zhang, Chaoyue Wang, and Dacheng Tao. HandRefiner: Refining malformed hands in generated images by diffusion-based conditional inpainting. In ACM MM, 2024. 1,7 [^13]: Alejandro Newell, Kaiyu Yang, and Jia Deng. Stacked hour-glass networks for human pose estimation. In ECCV, pages 483–499. Springer International Publishing, 2016. 7 [^14]: JoonKyu Park, Yeonguk Oh, Gyeongsik Moon, Hongsuk Choi, and Kyoung Mu Lee. HandOccNet: Occlusion-robust 3D hand mesh estimation network. In CVPR, pages 1496–1505, 2022. 6, 7, 8 [^15]: Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, pages 8748–8763. PMLR, 2021. 8 [^16]: Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. NeurIPS, pages 6000–6010, 2017. 8 [^17]: Bowen Wen, Wei Yang, Jan Kautz, and Stan Birchfield. FoundationPose: Unified 6D pose estimation and tracking of novel objects. In CVPR, pages 17868–17879, 2024. 8 [^18]: Hao Xu, Tianyu Wang, Xiao Tang, and Chi-Wing Fu. H2ONet: Hand-occlusion-and-orientation-aware network for real-time 3D hand mesh reconstruction. In CVPR, pages 17048–17058, 2023. 6, 7, 8 [^19]: Hao Xu, Haipeng Li, Yinqiao Wang, Shuaicheng Liu, and Chi-Wing Fu. HandBooster: Boosting 3D hand-mesh reconstruction by conditional synthesis and sampling of hand-object interactions. In CVPR, pages 10159–10169, 2024. 7, 8 [^20]: Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In ICCV, pages 3836–3847, 2023. 1 [^21]: Zhishan Zhou, Shihao Zhou, Zhi Lv, Minqiang Zou, Yao Tang, and Jiajun Liang. A simple baseline for efficient hand mesh reconstruction. In CVPR, pages 1367–1376, 2024. 6, 7, 8 [^22]: Christian Zimmermann and Thomas Brox. Learning to estimate 3D hand pose from single RGB images. In ICCV, pages 4903–4911, 2017. 1, 7 ## [^23]: Christian Zimmermann, Duygu Ceylan, Jimei Yang, Bryan Russell, Max Argus, and Thomas Brox. FreiHAND: A dataset for markerless capture of hand pose and shape from single RGB images. In ICCV, pages 813–822, 2019. 7 --- ![Image](images/43df0b41475d2e65167fb2877dd59fd85a3c156abda2df311cde7a1485d649c1.jpg) ![Image](images/4d1a61850fb617847843da1c483c1fb6bf43edca8b50f7fe35f5d7a7126a6fd7.jpg) ![Image](images/141f9fcd17d135fa9bab9094c90c4c63bfdb1379ca0f2115e218872a768d798f.jpg) ![Image](images/32baa3c020f2ccf29bfd1274da356d83f34cb86e08294bbd6cb08cf698bc1e68.jpg) ![Image](images/d3af964aa7134040b87835e2c10db870c0930e4a292c658ebb8c3ebece92da12.jpg) ![Image](images/94ef4050451fcc2b1a9396dfe2951c690392b46fe75259e20834bd3b41d4e2e6.jpg) ![Image](images/36b4ef139ee5db8f14c38acb72c8b50aa967ac37702b63ef62fc5592d6ce9c8d.jpg) ![Image](images/5585fd74e5b8e712a82acdb413d4f7ad45f8fd7a140c24cc813df0cddbb7bb6c.jpg) ![Image](images/98f1da7a8d93f02dc26a475ea60e374e40f120f05d787a0849b033bbd536f196.jpg) ![Image](images/49bfc7ccd20b348eef6478cbaa7b84cc85a49b38d2bceb137014ef53d96fe548.jpg) ![Image](images/2a77b5d0a59a02e9438c9f7fc9d0d94d5d09017e8d4515779da652e88fee2be3.jpg) ![Image](images/9d7d1af1da0d893396161e6520cf25280c2a1c4f0e10eb357d59bd6c399f32a3.jpg) ![Image](images/4a8fa45e343378385a5cde054a7e58e714bca90f8ab19c232ad508560726beeb.jpg) ![Image](images/d697896153d244b2f36e0c39ab5def0a7ba831a0bf528f00a2edf102e0b66a18.jpg) ![Image](images/bc54494dece210e5c1fd8c208549fc1c84c1af15c03f30307a4e818c5c01eed1.jpg) 输入真实值我们的方法 HFL-Net H2ONet HandOccNet ![Image](images/eb456787d96e448b59a3c55896188a9ed0a94825ac3fd831ab9982702dab8753.jpg) 输入 ![Image](images/788255227c8439d0e8db8a31b0d92f88901f7271924870e002c353f0b1c45de0.jpg) ![Image](images/8b7e675b01b7732949678238137c0e7720e474bdd135a9f616f700bf9791fee8.jpg) ![Image](images/1e4179176a86d378d4929e0be2b3377795ec30ecb6dca621caa4e1f788cc2935.jpg) 我们的方法 HFL-Net H2ONet HandOccNet

Figure E. Qualitative comparison between UniHOPE and SOTA HPE/HOPE methods across hand-only/hand-object scenarios in DexYCB ("S3" split), in which all the grasping objects are unseen during training.

---

Hand-Only Scene

![Image](images/efc25a42d5323cad8160bde9fce6b006bdfdf8b63e352d1a227ee6b5c856cb27.jpg)

Hand-Object Scene

![Image](images/f612e1915055192af57e5c233b1b4190610475241e889635a479e3db0a46f702.jpg) ![Image](images/3732c1536c57e8d41ec271795906122649331d07b279119f1fdaab09f55b7441.jpg) ![Image](images/c24124a64b2c3036bd6d0117986abf436d57d7e371bc79f2c53b5356c7cd6057.jpg) ![Image](images/ab9b023394d2f9afee7570e6662a10628032eaa809a77a30c57b1c97a44831b5.jpg) ![Image](images/a924338316685430577b800e65dd8a91134ec59f4ebd17e0a45d64c2e4b4f37e.jpg) ![Image](images/660b0ad33b5a8dd74487a32d6ab3ecb3dc9824d8339fea3c7ab5a1d84da605fb.jpg) ![Image](images/ca1eee7c2c7059b12a7dfdf6436a19bb3a8a9dbf0429d14985f153fccf8cfdfa.jpg) ![Image](images/1816f5c95795d866f6effded4c986fe801bd92f9440498760d838f34d45d8e57.jpg) ![Image](images/b15a1be36b72bb34972155f1130ea28853b39de49dfc6907d6722e2e5ab5a857.jpg) ![Image](images/1fa4e9564bd13469c69172a21871e211802cf2f282b30f4e1dfe34a3e8242166.jpg) ![Image](images/9f8d6b2c8ccb337d56c2901555fd5cc6649c1b5f9fb17b2114a03db5bf9fade8.jpg) ![Image](images/3800a6e2e81b58b679a7a17f0df86d58d406079cf163ec8185f462b9f36ba170.jpg) ![Image](images/f7a7fd6a37361db68c18e936915cdc7f6de47b43f8f24879aa49185c07d643ce.jpg) ![Image](images/2056ca3cc177e7eed820cdb9a8bc64fcdd8d6f76318521f66ba13cee5542836c.jpg) ![Image](images/478574415574650c392cd1859046873de8fd41f59e2f29ea569e8f71936b2b41.jpg) ![Image](images/4aa95a6765561c6b9642c821570625da5d6b80831d668da4bb2ff149d4d1e926.jpg) ![Image](images/1fdba2140f926ce555df7198221deacce973b2579c7f52c587ee332823ec7303.jpg) ![Image](images/133c76f33cbdc134332c872efcae256385046015251471d55d48a08d6d08c44c.jpg) ![Image](images/79bdb2b5fadc24281fe749584afb581158b8790c1ad6980e63a4831b4b5130de.jpg) ![Image](images/5c43071fdf6b49cc74a1adf386b13313edeec6ce7f842a8fcd53d68a3a3ea178.jpg) ![Image](images/fb43f182431f393292df6dbf8d252060b4970b725ddef9cb6f292061a87fa3c4.jpg) ![Image](images/589026aa80f69f62a0e609b53819bbc19b4d3b56878e1e950c68182d73fc2e7b.jpg) ![Image](images/80a28b1ec0f9b3027189ee37898c1ca9d89d7fceea31ce0b419e59fd1988f91a.jpg) ![Image](images/09b45ed9b50c347cac9e2aa1deda6153a747aa2806b68dbc63334b7c11957f6e.jpg) ![Image](images/597f1e053217f976043c467c804b40e777a9d7d48858c9262ba9ee5c92f2fede.jpg) ![Image](images/077829022fed7688e788e8243fd68b99c11077ae28d0ebd5c4479a3a5ddee3b2.jpg) ![Image](images/74899579d4f4d8b2b24121b3b53fdda1865fbd0e0704408faedacef4f3bbaf48.jpg) ![Image](images/d32731d0450a7d4db199c3267ebe241b3e2a74cf0a0a761106c53f39bb3b4de4.jpg) ![Image](images/be393cc88b2bc15a76842559522048fa8651020425e6e2a125a4ebeff4ccfc2d.jpg) ![Image](images/491a02669c21f20085d9b636e6ac7516912830ca2a5e93ff48bff3c2025c6f62.jpg) ![Image](images/681b59de79f54a45102d573c80ac3943b8f8f5080c2846a2e137f9cb50732b31.jpg) ![Image](images/5a468f3f34f7032490e80789ab53c92fbb7054add37322965a0ffa72e1ab0a6a.jpg) ![Image](images/f78e879164bd1a5ab26d640d38b724edfe5c3aef5d58047aef4b7c07d164068d.jpg) ![Image](images/00c80ccd225a37a7bb88b2e186e65038df71b2d18ac60dc1c814be6cd376e0aa.jpg) ![Image](images/dbfb5799249f1cc1747f6188a7b626279840d0729a153342e86b0b4d5a761bc7.jpg) ![Image](images/f0ab5a81505724370792b35ad30194ecb587dc22d85fc3287c5e4df7109c0774.jpg) ![Image](images/7d7e0477a40c39e06fde294e6c1efce0126eba26fbc862b7cf543a7ceadd485f.jpg) ![Image](images/db6d235291088c35b08aa99797bced4b7edf15b9f9665e75b2d994c8d2547e75.jpg) ![Image](images/3486650b379a08e67876c8200cdb78656c03f100fb16f93c94b868376dfc6878.jpg) ![Image](images/e4434b8c3760ff84f862150a3986a3c5c4e838c6c5e306d84fb9e2256f151401.jpg) ![Image](images/0934f334014bcca5c9b637f94deb778a29d291deb767411d7a3a00b1e3f6e17e.jpg) ![Image](images/27bd72240568cd1f0ec1a19c4d10c72814b741a71f91fa5d690690fab1c2c914.jpg) ![Image](images/6d8dfd47eff12379a9f231b9ec8359bcff09f3e0addf375c26688eb14d973d8c.jpg) ![Image](images/1d7c98c3b4ba402d09902b0fbaed52d09a1f66e1603cde7dcc6485ff9532ef05.jpg) ![Image](images/1eacffdf1f5fedb861ef3ff2943fc9d1ba056a98b3f8df4979357104e53c3ca5.jpg) ![Image](images/1b358a30cd458192daf6927c969d72439ba7f7228c04a920854a3285614a1374.jpg) 我们的方法 ![Image](images/89a8ab65a26d633442e91b9f0e2b6f7ae4963298067192ac4c2f4695a2f884cf.jpg) HFL-Net H2ONet ![Image](images/608aaa536c5ee2b30fea253565362c3efef94b2dfcd8c9cde4b50205ce3d5c50.jpg) ![Image](images/cb525206b6d946301ec7e9227fdd812e2566ed3c529fdda0ac01a6884ddf3016.jpg) 输入 ![Image](images/d7d0a47a59a7a577ac701a482759f66687cf1fef7f5a7dc988cdc40eae5af704.jpg) ![Image](images/d31e4459ae25c7cbf3b1ab2f2e8a8f426009e997520a710f292ea39ccc5e44bb.jpg) ![Image](images/dd33eb12c73c092c4198010ad41a7bfa715e0eb503f46583ae1bd3b31dff0c9f.jpg) HFL-Net ![Image](images/11140bc010b3bc562c6cfd91ae1fc72094be906324bd00811258d0fc6f4ba0f2.jpg) H2ONet

Input HandOccNet Ours et HandOccNet Figure F. Qualitative comparison between UniHOPE and SOTA HPE/HOPE methods across hand-only/hand-object scenarios in HO3D. The ground truths are not publicly available.

--- ![Image](images/bbc6bb71795a91e1a23ebf0cb2ec3d6097dad50db3c5ddf43c76acdc5af67266.jpg) ![Image](images/3c64f0bbc4a9e8b79a62cfcbe0d1fcd79432cf0906c5f71e1676e33d82d1faec.jpg) ![Image](images/afa6982a24ea3f09bcdad5fc52db37c0a95eb0e229cfeca63b6944d4985d7be8.jpg) ![Image](images/f663ef63aa5284f6259ddcac5c289fdb1d8b0a109e50cc65dbcaed7b197b98b7.jpg) ![Image](images/1d80a3c190972c89373775061c15847671d0ba53004cc681b3aad637353b2828.jpg) 真实值 ![Image](images/0e14dbfbc2077a6e97288f1110625ff1abd62b657974a961d4c9709e5334ef20.jpg) 我们的方法输入 ![Image](images/eff4047fd2b4e1e3e7b2b065f8c99258589a0c8d1310c7b4a0e23655a6f85ddf.jpg) HFL-Net H2ONet HandOccNet ![Image](images/883b2a041f59a2c2af7b628920a48b4a17366774905a2e9ce5504bc93f0fbb66.jpg) ![Image](images/a02905d493b0f9a8c1dfe1f61e35e82d41e1821410c6a9ee0cba77e1002340eb.jpg) 输入真实值我们的方法 HFL-Net

Figure G. Qualitative comparison between our method and SOTA HPE/HOPE methods on FreiHAND.

--- ![Image](images/7dc5771b71701b749165c4229f4770e6949c6f3193145b4b321ae3a8dfcf3354.jpg) ![Image](images/caab675da512523d6441209b0a5d6d6aa0a04949e2f5535d1ca4d8b6bd6e780a.jpg) ![Image](images/8f1e0444b4fe15753406de0f567fc20bf41591adfca9038339952ac26638a368.jpg) ![Image](images/e476885d96192f32eed17f69d519dc92f6128e156d08855bab8e999430bbdecd.jpg) ![Image](images/a5dae49c776b7113aedab6fb6f2ffec72b632c7be9857834304f25b2fff80a71.jpg) ![Image](images/d42e34fc785948630c4fb04f51600ed1f76db03b28541825553dae5cb837947b.jpg) ![Image](images/c4871aed1c8d066bcc9e5e22695f4a3758ed9f8b845b10365c8503566ed6d040.jpg) ![Image](images/b3a605b7963bc072bea4c0797301b67e8e1a386153b0cd72774935f8dcf14594.jpg) ![Image](images/5f58a4df8b38254873a9bf98294943387ce48deda6f678f291f6decb2eab7394.jpg) 输入 ![Image](images/2bd0b4c7683168aa2d7a37e3c2fe8d973a03ccf5d33beab41e1e929710076ff7.jpg) ![Image](images/c9df596e09f315aecb8a76badd84c71db71acaf4f4ee928fc103b9a2c513ad88.jpg) 条件 ![Image](images/e1608aa0eb5cf0ae3c1350067d8c05708421ab8114afd264a4cf205a8709fe9d.jpg) 结果 ![Image](images/c8ca75aef8e535948c4cf8bb6cca7742cd1842c4ecf40639c87a2be9a3552f0b.jpg) ![Image](images/3018f65e357e39cfbcff9f70c8ef81d79450248e6b960146ccab022ca4cae368.jpg) 输入条件结果

Figure H. More examples of de-occluded hand images. Note that masks are overlaid on the original image for better visualization, the actual condition for our generative de-occluder is a binary mask.

w/o

![Image](images/a45d88e1fd3c4751581de6d3ee39c5655fe173e289258b4f8232cd92fb0fc79d.jpg) (a) 基于抓取感知的特征融合 ![Image](images/cb439919bbcc757d77296bcdfa00b1270e7ee0f86aa9e12d9638f9d0d1e96a47.jpg) (b) 生成式去遮挡模块 ![Image](images/1e6a5651fdb199be5e978584609b2ca6039bfd460d95079d6886cd448e67d9d3.jpg) (c) 多层级特征增强 ![Image](images/e2984ef35bb40a07576844071004d5c9b033aab0c2e59f938ecbe1cce20c3ffa.jpg) (d) 利用物体信息

Figure I. Effects of different designs in our pipeline.

--- ![Image](images/93f058d046ec6fe5019d3a3be86f219d7214bf0b306608b77cb5a45bcdc65aac.jpg) ![Image](images/20ddbe4dbabff3b04afe0fa8233b4a878de8d58c0038e076003b1b3f19d98348.jpg) ![Image](images/59ba402318892be23686ad4b95e5b0f2e099e636c01bf306f91cc877fe28eddb.jpg) ![Image](images/c7c0c64042cbbf83b02d78cb9172ae72cacfd60f99a537210f170aa04de894d9.jpg) ![Image](images/03c3e36be9f53d602b98e0013e1487d2b30a425935ca4096a3018c1a52cc078b.jpg) ![Image](images/12307016b605de7e7aadbc3a4b075b56af6de63878d1c3de867ec7669d6da65d.jpg) ![Image](images/1cea9e1f7db12f12685863b0d32038634040b878fc6d99cba5b880a29f0234b5.jpg) ![Image](images/3b1a48b714e2b686b9f4fd7782857a3f5b5c3310aaa0746dad7d0fa958c4d062.jpg) ![Image](images/f92af925c25caffc3951f8633c24081d040688fdb6ba3f8a4b91876b0f9a4bb6.jpg) ![Image](images/7bd2dad50a1a0b899adceccb52ba3e95c7be020d7b144f7501933ccb96ad13b2.jpg) ![Image](images/ca6d52025b7a998dcec98da83db3d8dc65b00d1c95ed7ffe95f3c3d73a97bb79.jpg) ![Image](images/52d45d36e0778c9205c80ea62432f96b14f0db5ee6f33e4628939e9fa85d4589.jpg) ![Image](images/9871012cfcb10e0c114194cfd9730ef0b8da122c12743716603f5937be89238d.jpg) ![Image](images/afd484298699971bcf3a64925340504e7a60118689ffd6d5d1b0db3ef77bb50e.jpg) ![Image](images/3aca15bc801b94c4e4a8d727cb9c8448031ee362f7996d004f8416b2ec8882df.jpg) ![Image](images/52489d7183cfb39f406ac4fb98dcab1f2b4a4a7897e7bc52dba2e6d7652036d5.jpg) ![Image](images/10c0935838fd9bc6f498a3c643369193e07d908e8860d2a9267dda82da6031b2.jpg) ![Image](images/6abcafd6bad41574b3cfb7b6ed95c75300f84aacdedefc7b81f40ad63bd6b342.jpg) ![Image](images/8c4e83c02edab9cc362ab7a7de0be206a4768ea0acd593fcea08602aa0696cd6.jpg) ![Image](images/77cace295b2b7cccec644dc706988a195ef4ada4151608086f12ae555775453c.jpg) ![Image](images/7d9825bd113a7e96b8002d42dd3d0fe8cc1c3253996203accaa755bd63e06bd3.jpg) ![Image](images/6f797b0be6533cccb84ae082ee3805400daee5e958624af39d19a87c4784fbc6.jpg) ![Image](images/003c68dc4843cb64c6ef69a6ed8152ee758e6fb5a832d0716709adee3dfa87b5.jpg) ![Image](images/2f7c30d3b54d2e5c2ab5685c460c992e5fde4e52eb469938952db14156275832.jpg) ![Image](images/128e94c42617b4afe8cf7d89e420795cd28aa4e9c9c6b51e649edf57cda11bf5.jpg) ![Image](images/67a1417e397c484b19ea2b2b413308af936466b80481bf0b91f5b1b5227eabc2.jpg) ![Image](images/fbf959ffc7a9d28fed56d32be9c8bf31e9d192289488bb4b7aea2cb33b2016b1.jpg) ![Image](images/0c2d5debcc7a718e70ee1e1f5805321b2909eb0a795e2cb34d99dc589b519c01.jpg) ![Image](images/604111dea1a1f5452613ddb64b73ea690844b6189f1b00a4282eceb390545c3b.jpg) ![Image](images/b308c82443b4ae6b1a33fbf0aa33f94dbf048a12e6190351b488b4431e9c4a66.jpg) ![Image](images/835f702a2b5c4b004db07e4e2b185f8b7242d6d39660ac66fdb3889846cb1a59.jpg) ![Image](images/d28c0d1a7348c29fcf04c61834251ee1d238ca8da1db6c43c7ad530340d7f60e.jpg) ![Image](images/98c6c567394371f7f76b1d45e635bf058ec265340970e0c97fb6cf8c7714a99b.jpg) 掩码输入 ![Image](images/88e0181db8cabbedb803f2b3ee5803c0feedb24a23fc75a458a827bbaa3eca36.jpg) ![Image](images/d0467952b5877cc985ae71363019dd630199f3390c7343c44ae6d4d51c8d084e.jpg) 深度图 s = 0.25 ![Image](images/dc60a4b94fb401e8f3dbe6c53792c469d5b516503a87a6adc480952cdd0a7ac5.jpg) s = 0.40 ![Image](images/5531c2d645afbd90afd52db076d9695cb673faef240667ff678e4bab6e863dfc.jpg) s = 0.55 ![Image](images/14a87102af788ba4421cbf5d3bba07f2de3788b5e04d357e2a500a085ef269ff.jpg) s = 0.70 ![Image](images/01e0cbb093e1bd670c7366dfbdc7b2bf5e09f39e590b2f9b122223bd76b0a25d.jpg) s = 0.85 ![Image](images/0bf8cdc7c63e2e814c48206c068d61f43a9ec224ea6e70182126245ef67a9979.jpg) s = 1.00

Figure J. The generated images with varying control strengths. Our adaptive strategy (metrics marked in bold) effectively balances fidelity and consistency.