# Follow My Hold: Hand-Object Interaction Reconstruction through Geometric Guidance

Ayce Idil Aytekin $^{1*}$ Helge Rhodin $^{1,2}$ Rishabh Dabral $^{1}$ Christian Theobalt $^{1}$

 $^{1}$ Max Planck Institute for Informatics and Saarland University

 $^{2}$ Bielefeld University

![Image](images/d3965800e2eb657497ad15d0a5909419b2765cf9ac1198d03298455e73177ecf.jpg)

<div style="text-align: center;">Figure 1. In-the-wild results. From a single RGB frame from in-the-wild (top row), our method reconstructs detailed 3D hand–object interactions (bottom row). It does so by guiding a latent diffusion model with multi-modal cues derived from the input image and several foundation models.</div>

## Abstract

We propose a novel diffusion-based framework for reconstructing 3D geometry of hand-held objects from monocular RGB images by leveraging hand-object interaction as geometric guidance. Our method conditions a latent diffusion model on an inpainted object appearance and uses inference-time guidance to optimize the object reconstruction, while simultaneously ensuring plausible hand-object interactions. Unlike prior methods that rely on extensive post-processing or produce low-quality reconstructions, our approach directly generates high-quality object geometry during the diffusion process by introducing guidance with an optimization-in-the-loop design. Specifically, we guide the diffusion model by applying supervision to the velocity field while simultaneously optimizing the transformations of both the hand and the object being reconstructed. This optimization is driven by multi-modal geometric cues, including normal and depth alignment, silhouette consistency, and 2D keypoint reprojection. We further incorporate

signed distance field supervision and enforce contact and non-intersection constraints to ensure physical plausibility of hand-object interaction. Our method yields accurate, robust and coherent reconstructions under occlusion while generalizing well to in-the-wild scenarios. The project page is at <https://aidilayce.github.io/FollowMyHold-page/>.

### 1. Introduction

Our hands are how we shape the world; by picking, pushing, holding, slicing, or sculpting. From watering flowers to assembling furniture, hand-object interactions are ubiquitous. However, from a 3D reconstruction perspective, these interactions are inherently ambiguous: hands occlude objects and shapes overlap. This makes an accurate 3D reconstruction of hand-object interaction (HOI) difficult as ambiguity is even greater when working with a single image, where depth and contact cues are limited. Tackling this practical setting is crucial to enable robust and scalable 3D understanding in AR/VR, embodied AI, and robotics.

It is noteworthy that hand and object reconstruction have been long-standing problems [^12] [^21] [^24] [^25] [^31] [^45] [^46]

---

with several challenges arising out of the ill-posedness of the task. Unlike clothed bodies or textured objects, hands are of rather homogeneous colors, making it challenging to reconstruct them from a single image. Likewise, reconstructing objects in 3D, despite the impressive progress in the recent years [^29] [^40] [^43] [^50] [^56], remains a challenging task since objects come in such diverse shapes that building a strong prior is difficult. The reconstruction problem becomes significantly more complex when we combine the two tasks into one: reconstructing the 3D objects when they are undergoing hand-held interaction.

无需多言，若不合理建模手部造成的遮挡现象，便无法解决HOI重建问题。在此领域，现有研究尝试了多种方法，例如基于模板的优化[20–23, 54]、利用3D手-物体数据进行训练[^24] [^27] [^53]、数据驱动的法向预测[^18]或6DoF预测[^4] [^45]。然而，这类模型要么需要以模板形式存在的显式几何先验，要么更适用于多视角/多帧的重建任务。此外，直接回归部分点云的方法[^48]无法补全被遮挡的物体部分。

近期，诸如EasyHOI [^32]和Gen3DSR [^13]之类的方法试图借助大规模2D及3D基础模型，构建能将不同基础模型中的隐式先验融合为一体的流程。尽管这些方法在域外泛化能力方面表现优异，但它们往往稳定性较差；优化过程中一旦某个组件出现故障，便会导致重建结果彻底失败。我们并未直接采用生成模型的输出作为最终重建结果，而是提出将2D基础模型的监督信息直接嵌入到3D生成采样过程中，以此提升鲁棒性。

基于这一核心理念，我们推出了FollowMyHold——它能在推理阶段借助几何监督来引导预训练的基于流的图像到3D生成器。具体来说，我们：(i)利用2D/3D基础模型提取互补的线索：交互掩码[^33]和手部检测结果[^38]；通过Gemini提示生成的填充后物体外观[^28] [^44]；初始手部网格[^37]；由Hunyuan3D-2生成的初始粗糙的HOI网格[^56]；以及部分HOI点云[^48]；(ii)将所有输出结果对齐到一个共享的、与图像对齐的坐标系中；(iii)采用循环优化设计来引导修正流3D生成器（Hunyuan3D-2 [^56]）进行采样。我们的分阶段优化设计（先优化手部，再优化物体，最后优化两者结合部分）会同时应用像素级对齐的2D损失（法向、深度、轮廓、关键点）以及3D交互约束（相交与邻近关系），从而生成物理上合理且全局一致的HOI重建结果。

我们在OakInk [^52]、Arctic [^17]和DexYCB [^5]等成熟的HOI图像到3D重建基准测试集上进行了评估。FollowMyHold在生成式方法中取得了最先进的性能，其物体重建精度超过了EasyHOI，重建率更是提升了近两倍，同时具备强大的鲁棒性和野外场景泛化能力。大量的实验验证了该方法的有效性与鲁棒性。相关代码将会公开发布。

### 2. 相关工作

#### 2.1. 手部与物体交互的重建

从单张图像中恢复交互中的手部与物体的3D结构颇具挑战性，原因在于遮挡现象以及3D监督信息的匮乏。许多现有方法为了简化问题，会假设能够获取3D物体模板[3, 4, 7, 12, 19–23, 54]，仅从视频序列[^6] [^36]或多视角输入[^39]中估计手部和物体的姿态。无需模板的方法则利用3D交互数据集进行学习[^9] [^10]，但由于训练集中物体种类有限，这类方法往往表现不佳。近期的研究[^32]利用基础模型分别重建手部和物体，但会提前固定物体的几何形状，导致该流程对初始误差极为敏感。相比之下，我们的方法以类别无关的方式联合优化物体形状、姿态以及手部姿态，在利用基础模型先验的同时，通过几何信号引导模型。

#### 2.2. 单目3D物体重建

从单张图像中恢复三维形状一直是三维视觉领域最基础的问题之一。从早期的基于CNN的模型[^29] [^40]，到基于检索的系统[^43]以及体素化方法[^34]，人们提出了各种各样的策略。近期的趋势转向生成式重建，其中大型重建模型（LRMs）[^26] [^42] [^47] [^49] [^50] [^56]能够从稀疏输入中生成高保真的几何结构。然而，这些模型通常是在以物体为中心、无遮挡的输入假设下训练的。在实际应用中，一旦出现遮挡情况——比如人与物体的交互过程——重建效果就会显著下降。我们的方法通过将源自手与物体接触及可见性信息的额外线索整合进来，使大规模物体重建模型能够适应这种更复杂的场景，即便在部分观测条件下也能生成更合理的补全结果。

#### 2.3. 手部网格恢复

借助MANO等参数化模型的引入，从图像中估计手部姿态和形状技术取得了重大进展，这些模型能够基于单目RGB图像[^14] [^37] [^38]进行密集预测。近期的方法要么直接回归MANO参数[^1] [^2]，要么通过图像级约束来优化这些参数[^55] [^57]。虽然这些技术在单独使用时表现良好，但往往无法兼顾物理合理性。

---

在存在相互作用物体的情况下，物理合理性尤为重要。在我们的方法中，采用了HaMeR[^37]这一最先进的手部网格恢复方法，并利用该网格作为信号来指导物体重建过程。

### 3. 预备知识：修正流与符号说明

我们回顾一下修正流模型，该模型是我们3D生成器（Hunyuan3D-2）的核心，借此阐明推理时的引导机制是如何运作的。

修正流 $\hat{x}_1$ 的公式表述。修正流模型是一类采用流匹配目标函数 [^16] 进行训练的生成模型，它将生成过程建模为求解常微分方程（ODE）。与DDPM（通过近似分数函数来预测噪声）不同，这类模型学习的是一种速度场，该速度场会随时间将含噪样本推向干净的目标样本。给定时间t下的含噪样本 $\mathbf{x}_t$ 以及条件信息 $\mathbf{c}$，模型会预测出 $\mathbf{v}_\theta(x_t, t, \mathbf{c})$，更新规则如下：

 $$ \mathbf{x}_{t+\Delta t}\;=\;\mathbf{x}_{t}\;+\;\left(\sigma_{t+\Delta t}-\sigma_{t}\right)\mathbf{v}_{\theta}(x_{t},t,\mathbf{c}), $$

其中 $\sigma_t \in [0,1]$ 编码了步骤 $t \in [0,1]$ 对应的流时间。在推理阶段，我们无需更改模型权重，就能通过调整 $\mathbf{v}_\theta(x_t, t, \mathbf{c})$ 来引导生成过程——具体是利用任务目标G的梯度（详见第4节），比如几何一致性损失，从而推动生成轨迹向G值更低的方向发展。为了评估G，我们通常需要先估算干净样本 $\hat{x}_1$（遵循Hunyuan3D-2采用的 $\mathbf{x}_1$ 目标表述方式），因为它是对潜在目标样本的当前近似结果。该干净样本可通过公式4.54，利用流 $\mathbf{v}_\theta(x_t, t, \mathbf{c})$ 推导得出，相关内容参见 [^30]。

 $$ \hat{\mathbf{x}}_{1}\;=\;\mathbf{x}_{t}\;+\;\left(1-\sigma_{t}\right)\mathbf{v}_{\theta}(x_{t},t,\mathbf{c}). $$

推导细节详见补充材料。

记号说明。我们用$\mathcal{M}$来表示网格，其上标用于指示对应的标准空间：$\mathcal{M}^{\mathrm{U}}$对应Hunyuan空间，$\mathcal{M}^{\mathrm{H}}$对应HaMeR空间，而$\mathcal{M}^{\mathrm{I}}$则对应图像对齐空间。不同空间之间的变换由相似变换$\mathcal{T}$来描述，该变换由缩放参数$s \in \mathbb{R}$、旋转参数$R \in \mathrm{SO}(3)$以及平移参数$\mathbf{t} \in \mathbb{R}^3$共同确定。例如，HaMeR输出空间中的网格可以通过$\mathcal{M}^{\mathrm{I}} = \mathcal{T}_{\mathrm{H} \to \mathrm{I}} \cdot \mathcal{M}^{\mathrm{H}}$变换到图像对齐空间。

### 4. 方法

从单张RGB图像中重建物理上合理的三维手-物体交互场景本质上是一个不适定问题。遮挡、几何交叠以及有限深度线索的存在，使得准确恢复手部与物体的形状及其空间排布变得十分困难。近期出现的基础模型为手部建模（如HaMeR）、三维几何生成（如Hunyuan3D-2）以及带深度线索的像素对齐局部几何重建（如MoGe-2）提供了强大的先验知识。每种模型都各有优劣，我们的目标就是提出一种能够有效协调这些相互冲突的特性之法。然而，这些模型之间并不兼容，比如它们各自工作在独立的标准坐标空间中，且在缩放、朝向和对齐方面的假设也存在差异。

仅仅简单地将这些模型的输出结果拼接起来，正如EasyHOI [^32]和Gen3DSR [^13]等近期方法所尝试的那样，效果并不理想——因为规范空间的不对齐阻碍了模型输出的直接比较或联合优化。我们的方法直接应对这一协调难题：通过将模型输出显式对齐到一个共享的、与图像对齐的参考框架中。这种对齐解锁了一项关键能力：我们现在能够将3D预测生成的法线图、视差图和轮廓图，直接与由MoGe-2 [^48]（一种先进的点云预测方法）渲染出的2D监督图进行对比。这使得在优化过程中能够实现像素级精确且可微分的监督。

从宏观层面来看，我们的方法：(1)利用基础模型对图像进行分割和修复，生成初始的3D预测结果，包括手部网格、交互物体的部分点云以及粗略的交互物体网格（详见第4.1.1节和第4.1.2节）；(2)通过两阶段变换链将所有输出结果对齐到同一个坐标框架中（详见第4.1.3节）；(3)从该统一框架中渲染图像，并在扩散过程中利用基于梯度的引导机制，同时结合2D监督信号（如法线、深度、轮廓）和3D交互约束（如相交损失和邻近损失）（详见第4.2.1节）。有关我们方法的概览，请参见图2。

#### 4.1. 基于基础模型的初始化

#### 4.1.1 2D信号提取

输入的为单张RGB图像$I_{\text{full}} \in \mathbb{R}^{H \times W \times 3}$，我们首先利用基础模型定位交互区域。具体而言，使用LangSAM [^33]来提取手部$(M_h)$和物体$(M_o)$的掩码，同时借助WiLoR的手部检测器[^38]来检测手部边界框。二者共同定义了一个裁剪区域，从而得到$I_{\text{crop}} \in \mathbb{R}^{H \times W \times 3}$。接着，我们使用基于扩散的图像修复模型（由Gemini [^28]文本提示引导的FLUX.1 Context[dev] [^44]）来移除手部并补全被遮挡物体的外观，生成修复后的物体图像$I_{\text{obj}}$。此外，我们还将$I_{\text{crop}}$用$M_h$与$M_o$的并集进行掩码处理，得到$I_{hoi}$，将其作为MoGe-2和Hunyuan3D-2的输入。

#### 4.1.2 三维几何初始化

我们预先计算三维线索以指导基于扩散的手部-物体重建。首先，采用先进的手部重建模型HaMeR [^37]，它根据$\mathcal{M}_h^H$提供初始手部网格$\tilde{K}_{2D}$及其二维关键点$I_{\text{crop}}$。随后，使用MoGe-2 [^48]从$P_{\text{hoi}} \in \mathbb{R}^{N_m \times 3}$中估计部分点云$\phi$和相机参数$I_{\text{hoi}}$。

---

![Image](images/95eb990d2e9d7ddea3906e0e2a0819ec9839de6234fbe4d5938c5bdc4bec428e.jpg)

<div style="text-align: center;">Figure 2. Overview of FollowMyHold. Given a single RGB frame, we (1) isolate the interaction region and derive binary hand/object masks with LangSAM and WiLoR’s hand detector; (2) inpaint the occluded object appearance using FLUX.1 Kontext + Gemini (§4.1). Next, we obtain three complementary 3D cues: a HaMeR hand mesh, a MoGe-2 partial point cloud (with camera pose  $\phi$), and a coarse Hunyuan3D-2 HOI mesh. A two-step ICP registers all cues into a common image-aligned frame. Finally, we perform inference-time guidance with a staged optimization (§4.2): Phase 1 optimizes the hand transform  $T_h$; Phase 2 optimizes the object transform  $T_o$ and guides the velocity field; Phase 3 jointly refines  $(T_h, T_o)$ while guiding with pixel-aligned 2D losses ( $G_{2D}$: normals, disparity, silhouette) and 3D constraints ( $G_{3D}$: intersection, proximity). The right bottom row shows progressive object refinement over diffusion steps.</div>

如Fig. 2所示，MoGe-2输出的部分点云$P_{hoi}$仅包含输入物体图像$I_{hoi}$中未被遮挡的部分所对应的点。接着，我们利用$P_{hoi}$从$\phi$中渲染法向图、视差图和轮廓图，将其作为采样过程中几何引导目标的监督图。

我们的框架核心是Hunyuan3D-2 [^56]，这是一款用于三维形状生成的大规模潜变量扩散模型。扩散变换器$\mathcal{E}_{\text{hoi}}$会生成HOI潜变量$\mathbf{x}_{\text{hoi}} = \mathcal{E}_{\text{hoi}}(\mathbf{x}_t; t, \mathbf{c}_{\text{hoi}})$，其中$\mathbf{c}_{\text{hoi}}$是[^35]的DINOv2 $I_{\text{crop}}$特征。生成的潜变量$\mathbf{x}_{\text{hoi}}$会被Hunyuan3D-2的解码器$\mathcal{D}$在查询网格$\mathbf{q} \in [-1, 1]^3$上解码为有符号距离场（SDF），再通过FlexiCubes [^41]进行网格化处理：得到$\text{SDF}_{\text{hoi}} = \mathcal{D}(\mathbf{x}_{\text{hoi}}, q)$和$\mathcal{M}_{\text{hoi}}^U = \text{FlexiCubes}(\text{SDF}_{\text{hoi}})$。尽管Hunyuan3D-2是在单个物体数据上训练的，但如“利用基础模型进行初始化”部分中的Fig. 2所示，它在HOI输入下仍能保留粗略的手部-物体排列关系。

在我们的研究中，我们提出利用这种粗略对齐作为重要线索，来实现网格在规范空间中的变换与对齐。

#### 4.1.3 利用ICP在规范空间中进行变换

虽然对我们的流程而言至关重要，但HaMeR、Hunyuan3D-2和MoGe-2的输出却位于不同的坐标空间中。为了实现联合推理与渲染，我们采用迭代最近点（ICP）对齐算法来估算这些坐标空间之间的刚性变换。

我们首先将粗略的HOI网格与部分点云对齐（得到$T_{U\to I}$），再将手部网格与粗略的HOI网格对齐（得到$T_{H\to U}$），进而组合得到$T_{H\to I}$；这种两步ICP流程避免了直接将手部与不完整点云对齐所带来的问题。这些变换将所有几何实体统一到同一个坐标框架中（即MoGe-2部分点云的图像帧），以便进行一致的建模与推理时的引导。需要注意的是，我们采用相似性变换来初始化ICP，该变换会使源和目标物体的质心及全局尺度对齐。

#### 4.2 推理时的引导与分阶段优化

我们将推理时的引导机制与变换参数的分阶段优化相结合。我们首先阐述在修正流中的引导策略，然后说明各个优化阶段的作用。

#### 4.2.1 推理时引导

我们的基础图像转3D生成模型Hunyuan3D-2采用了专为高效性设计的修正流模型（见第3节）

---

用于图像到形状的生成。我们通过在推理时将基于梯度的监督机制直接整合到流轨迹中，对该模型进行修改，从而能够在特定损失约束下对物体几何形状和姿态进行端到端的精细化调整。我们将重建过程划分为三个阶段，每个阶段分别应对HOI恢复中的特定挑战。

在引入几何引导机制之前，我们先让去噪过程进展到时间步$t \geq \tau_1$，此时去噪后的潜变量已达到粗略但足够的质量。对于早期的扩散步骤$t < \tau_1$，我们采用公式1中的标准修正流更新方式，不施加引导。

```pseudo
// Algorithm 1: Inference-time Multi-step Optimization

Require: $\mathcal{R}_S$: Supervision maps

Require: $\mathbf{w}_{\text{opt}}$: geometric optimization guidance strengths

Require: Differentiable Renderer

1: $x_T \sim \mathcal{N}(0, I)$ ▷ Sample noisy latent vector
2: for all t from $0$ to $1$ do
3: $\mathbf{v}_t \leftarrow \mathcal{E}(x_t, t, c_{\text{obj}})$
4: if $t > t_2$ then ▷ Geometric guidance in Phase 3
5: $\mathbf{Z}_t = [\mathbf{v}_t, T_{o,t}, T_{h,t}]$ ▷ Define optimization variables
6: for $k = 1$ to K do ▷ Multiple gradient descent steps
7: $\hat{\mathbf{x}}_1^k = \mathbf{x}_t + (1 - \sigma_t) \cdot \mathbf{v}_t^k$
8: $\mathcal{M}_o^U = \text{FlexiCubes}(\mathcal{D}(\hat{\mathbf{x}}_1^k, \mathbf{q}))$
9: $\mathcal{M}_o^I = T_{o,t}^k T_{U \to I} \mathcal{M}_o^U, \mathcal{M}_h^I = T_{h,t}^k T_{H \to I} \mathcal{M}_h^H$
10: $\mathcal{R}_t^k \leftarrow \text{Differentiable Renderer}(\mathcal{M}_o^I \cup \mathcal{M}_h^I)$
11: $\mathcal{G}_{2D} \leftarrow \mathcal{L}_{2D}(\mathcal{R}_t^k, \mathcal{R}_S), \mathcal{G}_{3D} \leftarrow \mathcal{L}_{3D}(\mathcal{M}_o^I, \mathcal{M}_h^I)$
12: $\mathbf{Z}_t^k \leftarrow \mathbf{Z}_t^k - \mathbf{w}_{\text{opt}} \nabla_{\mathbf{Z}_t^k}(G_{2D}(\mathbf{Z}_t^k) + G_{3D}(\mathbf{Z}_t^k))$
13: end for
14: $\mathbf{v}_t^* = \mathbf{v}_t^K$ ▷ Update v with the steered velocity field
15: end if
16: $\mathbf{x}_t \leftarrow \mathbf{x}_t + \mathbf{v}_t^*$ ▷ Rectified Flow step
17: end for
```

▷ 获取最终的手部和物体网格
18: $\mathcal{M}_o^I = T_{o,1} T_{U \to I}$ FlexiCubes($\mathcal{D}(\mathbf{x}_1, q)$)
19: $\mathcal{M}_h^I = T_{h,1} T_{H \to I} \mathcal{M}_{h,1}^I$

```pseudo
// Algorithm 1: describes our full inference-time multi-step optimization used in Phase 3, where both the object and hand transformations  $T_o$ and  $T_h$ are jointly optimized and the velocity field  $\mathbf{v}_\theta(x_t, t, \mathbf{c}_{obj})$ is guided using both 2D and 3D objectives. We denote the flow field  $\mathbf{v}_\theta(x_t, t, \mathbf{c})$ as  $\mathbf{v}_t$ for simplicity.  $\mathcal{R}_s$ represents supervision maps rendered from  $P_{hoi}$: normal, disparity, and silhouette maps. Unlike traditional inference-time guidance, we follow an optimization-in-the-loop strategy, jointly optimizing the transformations  $(T_h$ and  $T_o)$ along with the flow field estimated by the model. In particular, for each diffusion time step  $t \in [\tau_2, 1]$, we perform  K inner gradient descent iterations to jointly optimize the transformations and steer  $\mathbf{v}_\theta(x_t, t, \mathbf{c})$ using 2D and 3D geometric objectives  $G_{2D}$ and  $G_{3D}$.

```

在各个阶段中，2D损失会将渲染图与$\mathcal{R}_s$进行比较：法线对齐$L_{norm,(\cdot)}$、L1视差$L_{disp,(\cdot)}$以及二值交叉熵轮廓$L_{sil,(\cdot)}$，分别对应手部、物体及其并集的$(\cdot) \in \{h, o, hoi\}$。需要注意的是，在物体重建期间我们将手部几何形状视为固定不变的，因为即便在遮挡情况下HaMeR也能提供可靠的手部估计结果。相比之下，由于部分可见性以及可能的形状范围广泛，物体所受的约束较少。我们利用手部作为稳定的几何锚点来约束物体姿态的优化。总体而言，这种分阶段策略能够仅凭单张RGB图像就实现稳定且符合物理规律的3D交互重建。各阶段的具体说明如下。

#### 4.2.2 第一阶段：仅手部优化

第一阶段仅专注于优化手部，以便在引入物体引导机制前建立一个具有空间意义的锚点。在扩散步骤$t = \tau_1$时，我们优化手部变换$T_h$，其初始值为单位缩放、恒等旋转和零平移，同时保持物体变换参数$T_o$以及速度场$\mathbf{v}_\theta(x_t, t, \mathbf{c}_{obj})$不变。我们首先将MANO手部网格$\mathcal{M}_h^H$转换到与图像对齐的空间中，得到$\mathcal{M}_h^I = \mathcal{T}_{H \to I} \mathcal{M}_h^H$，再将其按照$\widetilde{H} = T_h \cdot \mathcal{M}_h^I$进行姿态设定。渲染结果$\widetilde{H}$会生成$L_{norm.h}$、$L_{disp.h}$、$L_{sil.h}$。我们还会计算投影后的3D手部关键点与HaMeR检测到的对应2D关键点之间的$\ell_2$损失，并通过$L_{reg.h} = \|\mathbf{t}_h\|^2$对过度平移进行正则化。第一阶段的目标函数$L_{phase1}$是这些项的总和（权重详见补充材料）。完成后，我们便能得到姿态合理的手部网格，进而继续去噪过程。

#### 4.2.3 第二阶段：仅物体优化

这一阶段解决了一个关键的鲁棒性问题：如果在 $t = \tau_1$ 处重建的物体存在严重对齐偏差或缩放不当，它可能会在渲染视图中遮挡手部或被手部遮挡，从而降低二维损失的效用。为缓解这一问题，我们在第三阶段启用手-物联合推理之前，先对物体进行粗略对齐。在 $t = \tau_2$ 处，我们优化 $T_o$ 并对 $\mathbf{v}_\theta(x_t, t, \mathbf{c}_{obj})$ 施加基于梯度的引导，同时保持手部变换 $T_h$ 固定不变。这一阶段的操作与算法1类似，只是不包含手部网格。$T_o$ 初始化为单位矩阵，按照第4.2.1节的描述，我们提取物体的网格，该网格定义在Hunyuan3D-2的规范空间中。随后通过 $\mathcal{M}_o^I = \mathcal{T}_{U \to I} \mathcal{M}_o^I$ 将其转换到与图像对齐的空间，并由 $\tilde{O} = T_o \cdot \mathcal{M}_o^I$ 确定其姿态。从 $\tilde{O}$ 开始，我们渲染几何图并计算监督损失 $L_{norm,o}$、$L_{disp,o}$、$L_{sil,o}$。我们还计算了正则化损失 $L_{reg,o}$，用于惩罚过度的平移、缩放漂移和网格噪声。第二阶段损失 $L_{phase2}$（这些项的总和；权重见补充材料）作为 $G_{2D}$，用于引导 $\mathbf{v}_\theta(x_t, t, \mathbf{c}_{obj})$，同时更新 $T_o$ 共 $k_2$ 步。一旦

---

完成，我们便能得到粗略定位的物体网格以及受监督引导的速度场 $\mathbf{v}_{\theta}^{*}(x_t, t, \mathbf{c})$，接着通过公式1更新物体的隐变量。

#### 4.2.4 第三阶段：联合优化

在这一最终阶段（在 $t \in [\tau_2,1]$ 处），我们联合优化 $\mathbf{v}_\theta(x_t, t, \mathbf{c}_{obj})$、$T_o$ 和 $T_h$。这一阶段引入了三维几何监督以及二维目标，以确保空间合理性和物理交互性。

二维几何监督：当 $\widetilde{H}$ 和 $\widetilde{O}$ 确定姿态后，我们定义 $\widetilde{HOI} = \widetilde{H} \cup \widetilde{O}$ 并计算 $L_{\text{norm,hoi}}$、$L_{\text{disp,hoi}}$、$L_{\text{sil,hoi}}$。

三维几何监督：为确保手-物交互符合物理规律，我们引入了相交损失和邻近损失。

相交损失。我们添加相交惩罚项，旨在引入一种引导目标，促使采样过程得到不会与手部相交的三维隐变量。具体来说，我们将已确定姿态的手部网格和物体网格转换为共享体素网格上的有符号距离场（SDF），并将目标定义为

 $$ \begin{array}{r}{L_{\mathrm{i n t e r s e c t i o n}}=\frac{1}{K}\displaystyle\sum_{i=1}^{K}\mathbb{1}\left[\mathbf{S D F}_{h}(f_{i})<0\land\mathbf{S D F}_{o}(f_{i})<0\right],}\end{array} $$

其中 $\mathbb{1}[\cdot]$ 表示指示函数。该损失会惩罚两个SDF均为负值、即存在体积重叠的区域。

邻近损失。虽然相交项能确保得到的SDF $_{0}$ 不会穿透手部，但无法保证物体始终靠近手部。因此，我们使用带有边界 $\delta_{contact}$ 的吸引项来定义邻近损失，

 $$ L_{\mathrm{p r o x i m i t y}}=\frac{1}{|V_{h}^{I}|}\sum_{v_{h}\in V_{h}^{I}}\max(0,d_{\mathrm{h o}}(v_{h})-\delta_{\mathrm{c o n t a c t}}). $$

其中 $d_{\mathrm{ho}}(v_h)$ 表示从 $V_h^I$ 到 $V_o^I$ 的单侧距离。我们之所以从手部到物体方向计算单侧距离，是因为手部网格的几何结构通常更可靠。这样既能避免来自物体预测噪声的不稳定梯度，又能确保在优化过程中形成稳定且合理的接触。这两种损失提供了互补的引导：相交损失强制两者分离，而邻近损失则促进紧密接触。

第三阶段的最终损失。总损失 $L_{\text{phase3}}$ 汇总了HOI 2D项、交集与邻近性损失，同时以较低权重温和地复用 $L_{\text{phase1}}$ 和 $L_{\text{phase2}}$，以避免覆盖新信号。在每一步 $(\tau_2 < t \leq 1)$ 中，我们优化 $(T_h, T_o)$ 并借助 $\mathbf{v}_t$ 引导 $k_3$ 进行 $L_{\text{phase3}}$ 次迭代，随后通过公式1更新隐变量。在此阶段末尾，我们利用引导后的速度场 $\mathbf{v}_\theta^*(x_t, t, \mathbf{c})$ 解码最终隐变量，提取物体网格，并利用优化后的 $T_h$ 和 $T_o$ 确定手与物体的姿态，从而得到最终的HOI重建结果 $\widetilde{\text{HOI}}$。

### 5. 实验结果

我们在三个公开可用的数据集上评估我们的方法：OakInk [^52]、Arctic [^17] 和 DexYCB [^5]。所有数据集均包含标注有3D手-物体姿态、形状和网格的人类抓取数据。OakInk包含100个刚性物体，Arctic包含11个铰接物体，DexYCB包含20个YCB物体。我们从每个数据集中随机选取1000张图像作为测试集。由于我们使用了预训练的基础模型，因此无需对模型进行额外训练。

Comparison Baselines. We compare against the SOTA HOI methods in two groups: deterministic feed-forward (FF) and generative (Gen). For FF methods, following [^32], we include IHOI [^53], AlignSDF [^8] and gSDF [^10]. We also include HORT [^11], which predicts point clouds; for fair comparison, we convert them to meshes using alpha shapes [^15] as in their Supplementary Sec. B.1. Our primary focus, though, is on Gen models, where EasyHOI [^32] is the current state-of-the-art leveraging 2D and 3D foundation models for hand–object reconstruction. However, we deviate from EasyHOI's evaluation scheme wherein new test inputs are sampled until a set of 500 successful generations are achieved. Instead, we randomly sample 1000 test images and evaluate all the baseline methods on these test images. Since AlignSDF, gSDF, and HORT were trained on DexYCB, we exclude them from DexYCB evaluation for fairness. Video-based methods are excluded since our approach is single-frame. All the results for all the methods are re-computed on our randomly sampled testset.

性能指标。我们从三个方面评估手-物体交互重建任务的表现。参照 [^9]，我们通过基于3万个点样本的倒角距离（CD）以及5毫米/10毫米处的F分数（F5/F10）来报告物体精度。所有物体指标（CD、F5、F10）均能反映物体形状以及手-物体相对姿态的误差。对于抓取合理性，我们测量手-物体交集体积（I.V.）[^32]。这些精度指标仅在成功重建的样本上计算，那些未输出结果的样本（例如由于分割缺失或优化失败导致的）不予计入。为了评估一种方法的鲁棒性，我们计算重建率（R.R），即该方法在整个测试集中能产生输出的样本所占比例，无论其质量如何。有关评估的更多细节见补充材料。

实现细节。所有实验均在NVIDIA Tesla H100 NVL GPU上进行。我们的框架基于PyTorch实现，并与Hunyuan3D-2预训练扩散骨干网络集成。我们使用Adam优化器来更新手与物体的变换参数，并引导速度场。由于我们是针对单个输入视图进行优化，因此批量大小为1，总共使用了

---

<div style="text-align: center;">Table 1. Quantitative evaluation for HOI reconstruction. We do not evaluate AlignSDF, gSDF, and HORT on DexYCB as they are trained on it. F5 and F10 measure the F-scores of reconstructed object points within 5mm and 10mm of the ground-truth (GT) object, respectively. C.D. refers to the Chamfer Distance (in cm²) between the reconstructed object and GT object. I.V. refers to Intersection Volume (in cm³) between the hand mesh and the object mesh. R.R. refers to the Reconstruction Rate, defined as the fraction of objects for which a method produces an output, regardless of quality. Methods are also categorized as feed-forward (FF) or generative (Gen) approaches.</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td rowspan="2" colspan="2">Method</td><td colspan="5">OakInk</td><td colspan="5">Arctic</td><td colspan="5">DexYCB</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>F5 $\uparrow$</td><td style='text-align: center; word-wrap: break-word;'>F10 $\uparrow$</td><td style='text-align: center; word-wrap: break-word;'>C.D. $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>I.V. $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>R.R. $\uparrow$</td><td style='text-align: center; word-wrap: break-word;'>F5 $\uparrow$</td><td style='text-align: center; word-wrap: break-word;'>F10 $\uparrow$</td><td style='text-align: center; word-wrap: break-word;'>C.D. $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>I.V. $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>R.R. $\uparrow$</td><td style='text-align: center; word-wrap: break-word;'>F5 $\uparrow$</td><td style='text-align: center; word-wrap: break-word;'>F10 $\uparrow$</td><td style='text-align: center; word-wrap: break-word;'>C.D. $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>I.V. $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>R.R. $\uparrow$</td></tr><tr><td rowspan="4">FF</td><td style='text-align: center; word-wrap: break-word;'>IHOI</td><td style='text-align: center; word-wrap: break-word;'>0.132</td><td style='text-align: center; word-wrap: break-word;'>0.252</td><td style='text-align: center; word-wrap: break-word;'>3.97</td><td style='text-align: center; word-wrap: break-word;'>6.20</td><td style='text-align: center; word-wrap: break-word;'>0.82</td><td style='text-align: center; word-wrap: break-word;'>0.079</td><td style='text-align: center; word-wrap: break-word;'>0.148</td><td style='text-align: center; word-wrap: break-word;'>11.9</td><td style='text-align: center; word-wrap: break-word;'>4.27</td><td style='text-align: center; word-wrap: break-word;'>0.63</td><td style='text-align: center; word-wrap: break-word;'>0.108</td><td style='text-align: center; word-wrap: break-word;'>0.211</td><td style='text-align: center; word-wrap: break-word;'>4.87</td><td style='text-align: center; word-wrap: break-word;'>6.41</td><td style='text-align: center; word-wrap: break-word;'>0.49</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>AlignSDF</td><td style='text-align: center; word-wrap: break-word;'>0.060</td><td style='text-align: center; word-wrap: break-word;'>0.119</td><td style='text-align: center; word-wrap: break-word;'>7.75</td><td style='text-align: center; word-wrap: break-word;'>12.5</td><td style='text-align: center; word-wrap: break-word;'>0.94</td><td style='text-align: center; word-wrap: break-word;'>0.042</td><td style='text-align: center; word-wrap: break-word;'>0.083</td><td style='text-align: center; word-wrap: break-word;'>17.7</td><td style='text-align: center; word-wrap: break-word;'>15.6</td><td style='text-align: center; word-wrap: break-word;'>0.86</td><td style='text-align: center; word-wrap: break-word;'>-</td><td style='text-align: center; word-wrap: break-word;'>-</td><td style='text-align: center; word-wrap: break-word;'>-</td><td style='text-align: center; word-wrap: break-word;'>-</td><td style='text-align: center; word-wrap: break-word;'>-</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>gSDF</td><td style='text-align: center; word-wrap: break-word;'>0.047</td><td style='text-align: center; word-wrap: break-word;'>0.093</td><td style='text-align: center; word-wrap: break-word;'>8.17</td><td style='text-align: center; word-wrap: break-word;'>0.62</td><td style='text-align: center; word-wrap: break-word;'>0.87</td><td style='text-align: center; word-wrap: break-word;'>0.036</td><td style='text-align: center; word-wrap: break-word;'>0.070</td><td style='text-align: center; word-wrap: break-word;'>18.9</td><td style='text-align: center; word-wrap: break-word;'>1.23</td><td style='text-align: center; word-wrap: break-word;'>0.79</td><td style='text-align: center; word-wrap: break-word;'>-</td><td style='text-align: center; word-wrap: break-word;'>-</td><td style='text-align: center; word-wrap: break-word;'>-</td><td style='text-align: center; word-wrap: break-word;'>-</td><td style='text-align: center; word-wrap: break-word;'>-</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>HORT</td><td style='text-align: center; word-wrap: break-word;'>0.319</td><td style='text-align: center; word-wrap: break-word;'>0.508</td><td style='text-align: center; word-wrap: break-word;'>2.22</td><td style='text-align: center; word-wrap: break-word;'>16.6</td><td style='text-align: center; word-wrap: break-word;'>0.99</td><td style='text-align: center; word-wrap: break-word;'>0.101</td><td style='text-align: center; word-wrap: break-word;'>0.190</td><td style='text-align: center; word-wrap: break-word;'>11.0</td><td style='text-align: center; word-wrap: break-word;'>30.8</td><td style='text-align: center; word-wrap: break-word;'>0.88</td><td style='text-align: center; word-wrap: break-word;'>-</td><td style='text-align: center; word-wrap: break-word;'>-</td><td style='text-align: center; word-wrap: break-word;'>-</td><td style='text-align: center; word-wrap: break-word;'>-</td><td style='text-align: center; word-wrap: break-word;'>-</td></tr><tr><td rowspan="2">Gen</td><td style='text-align: center; word-wrap: break-word;'>EasyHOI</td><td style='text-align: center; word-wrap: break-word;'>0.109</td><td style='text-align: center; word-wrap: break-word;'>0.210</td><td style='text-align: center; word-wrap: break-word;'>4.62</td><td style='text-align: center; word-wrap: break-word;'>21.31</td><td style='text-align: center; word-wrap: break-word;'>0.39</td><td style='text-align: center; word-wrap: break-word;'>0.079</td><td style='text-align: center; word-wrap: break-word;'>0.145</td><td style='text-align: center; word-wrap: break-word;'>10.9</td><td style='text-align: center; word-wrap: break-word;'>18.3</td><td style='text-align: center; word-wrap: break-word;'>0.36</td><td style='text-align: center; word-wrap: break-word;'>0.090</td><td style='text-align: center; word-wrap: break-word;'>0.176</td><td style='text-align: center; word-wrap: break-word;'>6.26</td><td style='text-align: center; word-wrap: break-word;'>19.13</td><td style='text-align: center; word-wrap: break-word;'>0.30</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Ours</td><td style='text-align: center; word-wrap: break-word;'>0.179</td><td style='text-align: center; word-wrap: break-word;'>0.322</td><td style='text-align: center; word-wrap: break-word;'>1.80</td><td style='text-align: center; word-wrap: break-word;'>5.96</td><td style='text-align: center; word-wrap: break-word;'>0.87</td><td style='text-align: center; word-wrap: break-word;'>0.160</td><td style='text-align: center; word-wrap: break-word;'>0.288</td><td style='text-align: center; word-wrap: break-word;'>2.57</td><td style='text-align: center; word-wrap: break-word;'>5.08</td><td style='text-align: center; word-wrap: break-word;'>0.92</td><td style='text-align: center; word-wrap: break-word;'>0.158</td><td style='text-align: center; word-wrap: break-word;'>0.300</td><td style='text-align: center; word-wrap: break-word;'>2.04</td><td style='text-align: center; word-wrap: break-word;'>7.02</td><td style='text-align: center; word-wrap: break-word;'>0.58</td></tr></table>

![Image](images/109b45d5717bc1f01db75ee778608588a828d0d7a4194aec2836abc3d5d25020.jpg)

<div style="text-align: center;">Figure 3. Qualitative comparison of 3D HOI reconstructions on OakInk and Arctic datasets. From left to right: input RGB frame; reconstructions produced by our method, by EasyHOI, by HORT, and by IHOI. We evaluate and visualize HORT results after converting the point clouds into meshes following the procedure mentioned in the HORT paper, Supplementary Section B.1. Our approach yields more accurate object geometry with more plausible hand-object interactions.</div>

![Image](images/24ce6b793e09148e123a0e683c2a48ba4cc2d3f16cc6fcb003071056b9c0a1ae.jpg)

<div style="text-align: center;">Figure 4. Qualitative comparison of 3D HOI reconstructions on DexYCB dataset. From left to right: input RGB frame; reconstructions produced by our method, by EasyHOI, and by IHOI. Since HORT is trained on DexYCB, we do not include it to the comparison. Our method yields more accurate geometry and plausible interactions.</div>

采用20个扩散推理步骤，其中在第9步进行仅针对手部的优化，从第10步开始应用基于梯度的引导。物体引导系数固定为5.0。各阶段的其余超参数可在补充材料中找到。我们的方法处理每个样本大约需要6.03分钟。

#### 5.1. 对比实验

我们在表1中报告了该方法与基线模型之间的定量对比结果。准确性方面：FollowMyHold在所有数据集上均取得了最低的倒角距离，这表明其重建物体的全局对齐效果更优。在OakInk数据集上，尽管HORT的倒角距离更大，却获得了更高的F分数。这说明HORT的重建结果比我们的更粗糙（倒角距离更大），但在棘手案例上的表现并未出现灾难性失败（F5/F10分数更高），这一点从其较高的重建率也能看出，这得益于其前馈式设计。在Arctic数据集上，FollowMyHold的泛化能力更强。HORT在Arctic数据集中对细长、细节丰富或较大的工具物体更容易出现重建失败的情况，而FollowMyHold则能保持连贯的几何形状且倒角距离更低。此外，我们的方法保持了相对较低的交叠体积，显著优于EasyHOI和HORT。虽然gSDF的交叠体积更小，但它在物体重建的准确性和鲁棒性方面存在明显不足。鲁棒性方面：FollowMyHold的成功重建率远高于EasyHOI（在OakInk数据集上分别为87%和39%），也就是说它能在更具挑战性的案例上取得成功，同时整体准确性也更高。实际上，我们的重建成功率与那些除检测/分割效果不佳外很少失败的前馈式方法相当。这充分体现了我们这种“优化循环引导”策略的优势——它将准确性与高成功率结合，在像EasyHOI这样的生成式方法中表现突出。

图3和图4显示，我们的方法能够准确重建物体几何形状以及合理的手-物交互关系，表现优于EasyHOI、HORT和IHOI。EasyHOI在尺度一致性和物体完整性方面存在不足，

---

![Image](images/7022a10d8d603a017e9e9dbcc766cfea07c85e945a5e275e918fe7a512b4409e.jpg)

<div style="text-align: center;">Figure 5. Ablation of the phased optimization. We visualize the ablation study on our staged optimization design. We compare HOI reconstruction quality when removing inference-time guidance, dropping Phase 1 or Phase 2, swapping their order, or performing joint optimization. The full method yields the most accurate grasp and object. The bottom row shows the camera viewpoint, while the top row provides an alternative view to highlight differences.</div>

而IHOI虽然在抓取准确性上优于EasyHOI，但在物体重建方面却经常失败。HORT在处理罕见的细节丰富的几何形状时表现不佳（例如图3第二行第四列所示）。通过AlphaShapes算法进行的点云网格化处理（详见HORT补充材料第B.1节）也会破坏细长的部件。更多的定性结果可在补充材料中查看。我们还在图1中展示了在野外图像上的实验结果，这些图像来自互联网及日常生活场景，体现了FollowMyHold在真实世界示例中的鲁棒性和准确性。

#### 5.2. 消融实验

我们分析了分阶段设计的效果以及引导机制的影响（见图5和表2）。我们的消融实验集包含来自OakInk数据集的100个随机样本。完整模型能达到$1.70cm^2$的CD值。若不使用引导机制（先生成物体，再优化变换参数），CD值会升至$3.70cm^2$，这证实了引导机制对物体精细化处理的重要性。若跳过第一阶段（不进行仅针对手部的优化），CD值会降至$5.67cm^2$，说明手部可作为空间锚点。若跳过第二阶段（不进行仅针对物体的优化），CD值会增至$3.53cm^2$，表明物体先于关节的精细化处理是必要的。若调换第一阶段和第二阶段的顺序，CD值会变为$3.18cm^2$，这体现了阶段顺序的重要性。若从一开始便进行手-物体联合优化，CD值会升至$5.5cm^2$，凸显出分阶段优化的必要性。此外，为证明我们相比EasyHOI的主要性能提升源于方法设计而非Hunyuan3D-2，我们将EasyHOI的网格生成器替换为Hunyuan3D-2，其仅能略微改善CD值（$5.23 \rightarrow 5.13cm^2$）。而我们的方法仍能使CD值降低近$3\times$。补充材料包含了相关细节以及我们对损失项的消融实验结果。

### 6. 局限性与未来工作

我们在推理阶段采用的引导机制以牺牲精度和鲁棒性为代价来换取计算效率：每一步都包含内循环梯度更新

<div style="text-align: center;">Table 2. Ablation study for each method phase on the ablation set.</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'>Method</td><td style='text-align: center; word-wrap: break-word;'>C.D. $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>I.V. $\downarrow$</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>w/o inference-time guidance</td><td style='text-align: center; word-wrap: break-word;'>3.70</td><td style='text-align: center; word-wrap: break-word;'>5.83</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>w/o Phase 1</td><td style='text-align: center; word-wrap: break-word;'>5.67</td><td style='text-align: center; word-wrap: break-word;'>7.66</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>w/o Phase 2</td><td style='text-align: center; word-wrap: break-word;'>3.53</td><td style='text-align: center; word-wrap: break-word;'>4.82</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>swapping Phases</td><td style='text-align: center; word-wrap: break-word;'>3.18</td><td style='text-align: center; word-wrap: break-word;'>5.19</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>joint optimization</td><td style='text-align: center; word-wrap: break-word;'>5.51</td><td style='text-align: center; word-wrap: break-word;'>11.7</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Full Method</td><td style='text-align: center; word-wrap: break-word;'>1.70</td><td style='text-align: center; word-wrap: break-word;'>4.03</td></tr></table>

并通过潜码解码器进行反向传播，从而增加了运行时间。我们的方法还依赖于可靠的上游分割与修复结果，但此阶段的伪影可能会传递到重建过程中。另一个局限性在于对薄型物体的重建，此类物体的监督信号较弱。相关示例见补充材料。

未来的工作包括开发更稳健的2D与3D信号融合方式，结合针对不确定区域的习得交互先验；将我们的几何目标融入扩散模型训练过程以消除引导机制的内循环；以及将该方法拓展到视频领域，利用时间一致性（如光流对齐）来重建适用于AR/VR和远程呈现的稳定逐帧序列。

### 7. 结论

我们提出了FollowMyHold，这是一种单图像HOI重建方法，它在推理阶段利用“优化内循环”设计来引导预训练的3D修正流模型。研究表明，所提出的基于像素对齐的2D线索（法向与视差对齐、轮廓一致性）以及3D交互约束（相交、邻近关系）的推理阶段引导机制，结合分阶段优化（手部$\rightarrow$物体$\rightarrow$联合优化），能够生成物理上合理且全局一致的重建结果。有趣的是，这一策略不仅提升了物体重建质量，还显著增强了重建的鲁棒性。

---

## 参考文献

[^1]: Seungryul Baek, Kwang In Kim, and Tae-Kyun Kim. Pushing the envelope for rgb-based dense 3d hand pose estimation via neural rendering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1067–1076, 2019. 2

[^2]: Adnane Boukhayma, Rodrigo de Bem, and Philip HS Torr. 3d hand shape and pose from images in the wild. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10843–10852, 2019. 2

[^3]: Samarth Brahmbhatt, Chengcheng Tang, Christopher D Twigg, Charles C Kemp, and James Hays. Contactpose: A dataset of grasps with object contact and hand pose. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XIII 16, pages 361–378. Springer, 2020. 2

[^4]: Zhe Cao, Ilija Radosavovic, Angjoo Kanazawa, and Jitendra Malik. Reconstructing hand-object interactions in the wild. In Proceedings of the IEEE/CVF international conference on computer vision, pages 12417–12426, 2021. 2

[^5]: Yu-Wei Chao, Wei Yang, Yu Xiang, Pavlo Molchanov, Ankur Handa, Jonathan Tremblay, Yashraj S Narang, Karl Van Wyk, Umar Iqbal, Stan Birchfield, et al. Dexycb: A benchmark for capturing hand grasping of objects. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9044–9053, 2021. 2, 6, 8

[^6]: Jiayi Chen, Mi Yan, Jiazhao Zhang, Yinzhen Xu, Xiaolong Li, Yijia Weng, Li Yi, Shuran Song, and He Wang. Tracking and reconstructing hand object interactions from point cloud sequences in the wild. arXiv preprint arXiv:2209.12009, 2022. 2

[^7]: Yujin Chen, Zhigang Tu, Di Kang, Ruizhi Chen, Linchao Bao, Zhengyou Zhang, and Junsong Yuan. Joint hand-object 3d reconstruction from a single image with cross-branch feature fusion. IEEE Transactions on Image Processing, 30:4008–4021, 2021. 2

[^8]: Zerui Chen, Yana Hasson, Cordelia Schmid, and Ivan Laptev. Alignsdf: Pose-aligned signed distance fields for hand-object reconstruction. In Computer Vision – ECCV 2022, pages 231–248, Cham, 2022. Springer Nature Switzerland. 6

[^9]: Zerui Chen, Yana Hasson, Cordelia Schmid, and Ivan Laptev. Alignsdf: Pose-aligned signed distance fields for hand-object reconstruction. In European conference on computer vision, pages 231–248. Springer, 2022. 2, 6, 1

[^10]: Zerui Chen, Shizhe Chen, Cordelia Schmid, and Ivan Laptev. gsdf: Geometry-driven signed distance functions for 3d hand-object reconstruction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12890–12900, 2023. 2, 6

[^11]: Zerui Chen, Rolandos Alexandros Potamias, Shizhe Chen, and Cordelia Schmid. HORT: Monocular hand-held objects reconstruction with transformers. In ICCV, 2025. 6

[^12]: Enric Corona, Albert Pumarola, Guillem Alenya, Francesc Moreno-Noguer, and Grégory Rogez. Ganhand: Predicting human grasp affordances in multi-object scenes. In Proceedings of the 2014 International Conference on Computer Vision and Networking in the 2014 International Conference on Computer Vision and Networking, 2014, pp. 103-108. ACM, New York, USA, 2014.

IEEE/CVF计算机视觉与模式识别会议论文集，第5031–5041页，2020年。1, 2

[^13]: Andreea Dogaru, Mert Özer, and Bernhard Egger. Generalizable 3d scene reconstruction via divide and conquer from a single view. arXiv preprint arXiv:2404.03421, 2024. 2, 3

[^14]: Haoye Dong, Aviral Chharia, Wenbo Gou, Francisco Vicente Carrasco, and Fernando D De la Torre. Hamba: Single-view 3d hand reconstruction with graph-guided bi-scanning mamba. Advances in Neural Information Processing Systems, 37:2127–2160, 2024. 2

[^15]: H. Edelsbrunner, D. Kirkpatrick, and R. Seidel. On the shape of a set of points in the plane. IEEE Transactions on Information Theory, 29(4):551–559, 1983. 6

[^16]: Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first international conference on machine learning, 2024. 3

[^17]: Zicong Fan, Omid Taheri, Dimitrios Tzionas, Muhammed Kocabas, Manuel Kaufmann, Michael J Black, and Otmar Hilliges. Arctic: A dataset for dexterous bimanual hand-object manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12943–12954, 2023. 2, 6, 1, 7, 8

[^18]: Zicong Fan, Maria Parelli, Maria Eleni Kadoglou, Xu Chen, Muhammed Kocabas, Michael J Black, and Otmar Hilliges. Hold: Category-agnostic 3d reconstruction of interacting hands and objects from video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 494–504, 2024. 2

[^19]: Guillermo Garcia-Hernando, Shanxin Yuan, Seungryul Baek, and Tae-Kyun Kim. First-person hand action benchmark with rgb-d videos and 3d hand pose annotations. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 409–419, 2018. 2

[^20]: Georgia Gkioxari, Ross Girshick, Piotr Dollár, and Kaiming He. Detecting and recognizing human-object interactions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8359–8367, 2018. 2

[^21]: Patrick Grady, Chengcheng Tang, Christopher D Twigg, Minh Vo, Samarth Brahmbhatt, and Charles C Kemp. Contactopt: Optimizing contact to improve grasps. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1471–1481, 2021. 1

[^22]: Henning Hamer, Juergen Gall, Thibaut Weise, and Luc Van Gool. An object-dependent hand pose prior from sparse training data. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 671–678. IEEE, 2010.

[^23]: Shreyas Hampali, Mahdi Rad, Markus Oberweger, and Vincent Lepetit. Honnotate: A method for 3d annotation of hand and object poses. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3196–3206, 2020. 2

## [^24]: Yana Hasson, Gul Varol, Dimitrios Tzionas, Igor Kalevatykh, Michael J Black, Ivan Laptev, and Cordelia Schmid

---

学习手部与被操纵物体的联合重建。收录于IEEE/CVF计算机视觉与模式识别会议论文集，第11807–11816页，2019年。1,2

[^25]: Yana Hasson, Gül Varol, Cordelia Schmid, and Ivan Laptev. Towards unconstrained joint hand-object reconstruction from rgb videos. In 2021 International Conference on 3D Vision (3DV), pages 659–668. IEEE, 2021. 1

[^26]: Yicong Hong, Kai Zhang, Jiuxiang Gu, Sai Bi, Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli, Trung Bui, and Hao Tan. Lrm: Large reconstruction model for single image to 3d. arXiv preprint arXiv:2311.04400, 2023. 2

[^27]: Korrawe Karunratanakul, Jinlong Yang, Yan Zhang, Michael J Black, Krikamol Muandet, and Siyu Tang. Grasping field: Learning implicit representations for human grasps. In 2020 International Conference on 3D Vision (3DV), pages 333–344. IEEE, 2020. 2

[^28]: Black Forest Labs, Stephen Batifol, Andreas Blattmann, Frederic Boesel, Saksham Consul, Cyril Diagne, Tim Dockhorn, Jack English, Zion English, Patrick Esser, Sumith Kulal, Kyle Lacey, Yam Levi, Cheng Li, Dominik Lorenz, Jonas Müller, Dustin Podell, Robin Rombach, Harry Saini, Axel Sauer, and Luke Smith. Flux.1 context: Flow matching for in-context image generation and editing in latent space, 2025. 2, 3

[^29]: Xueting Li, Sifei Liu, Kihwan Kim, Shalini De Mello, Varun Jampani, Ming-Hsuan Yang, and Jan Kautz. Self-supervised single-view 3d reconstruction via semantic consistency. In ECCV, 2020. 2

[^30]: Yaron Lipman, Marton Havasi, Peter Holderrieth, Neta Shaul, Matt Le, Brian Karrer, Ricky TQ Chen, David Lopez-Paz, Heli Ben-Hamu, and Itai Gat. Flow matching guide and code. arXiv preprint arXiv:2412.06264, 2024. 3, 1

[^31]: Shaowei Liu, Hanwen Jiang, Jiarui Xu, Sifei Liu, and Xiaolong Wang. Semi-supervised 3d hand-object poses estimation with interactions in time. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14687–14697, 2021. 1

[^32]: Yumeng Liu, Xiaoxiao Long, Zemin Yang, Yuan Liu, Marc Habermann, Christian Theobalt, Yuexin Ma, and Wenping Wang. Easyhoi: Unleashing the power of large models for reconstructing hand-object interactions in the wild. arXiv preprint arXiv:2411.14280, 2024. 2, 3, 6, 1

[^33]: Luca Medeiros. Language segment anything. <https://github.com/luca-medeiros/lang-segment-anything>, 2023. Accessed: 2025-05-23. 2, 3

[^34]: Michael Niemeyer and Andreas Geiger. Giraffe: Representing scenes as compositional generative neural feature fields. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2021. 2

[^35]: Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023. 4

[^36]: Austin Patel, Andrew Wang, Ilija Radosavovic, and Jitendra Malik. Learning to imitate object interactions from internet videos, 2022. 2

[^37]: Georgios Pavlakos, Dandan Shan, Ilija Radosavovic, Angjoo Kanazawa, David Fouhey, and Jitendra Malik. Reconstructing hands in 3d with transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9826–9836, 2024. 2, 3

[^38]: Rolandos Alexandros Potamias, Jinglei Zhang, Jiankang Deng, and Stefanos Zafeiriou. Wilor: End-to-end 3d hand localization and reconstruction in-the-wild, 2024. 2, 3

[^39]: Wentian Qu, Zhaopeng Cui, Yinda Zhang, Chenyu Meng, Cuixia Ma, Xiaoming Deng, and Hongan Wang. Novel-view synthesis and pose estimation for hand-object interaction from sparse views. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 15100–15111, 2023. 2

[^40]: Katja Schwarz, Yiyi Liao, Michael Niemeyer, and Andreas Geiger. Graf: Generative radiance fields for 3d-aware image synthesis. In Advances in Neural Information Processing Systems, pages 20154–20166. Curran Associates, Inc., 2020.

[^41]: Tianchang Shen, Jacob Munkberg, Jon Hasselgren, Kangxue Yin, Zian Wang, Wenzheng Chen, Zan Gojcic, Sanja Fidler, Nicholas Sharp, and Jun Gao. Flexible isosurface extraction for gradient-based mesh optimization. ACM Transactions on Graphics (TOG), 42(4):1–16, 2023. 4

[^42]: Jiaxiang Tang, Zhaoxi Chen, Xiaokang Chen, Tengfei Wang, Gang Zeng, and Ziwei Liu. Lgm: Large multi-view gaussian model for high-resolution 3d content creation. In European Conference on Computer Vision, pages 1–18. Springer, 2024. 2

[^43]: Maxim Tatarchenko, Stephan R Richter, René Ranftl, Zhuwen Li, Vladlen Koltun, and Thomas Brox. What do single-view 3d reconstruction networks learn? In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3405–3414, 2019. 2

[^44]: Gemini Team. Gemini: A family of highly capable multi-modal models, 2024. 2, 3

[^45]: Bugra Tekin, Federica Bogo, and Marc Pollefeys. H+o: Unified egocentric recognition of 3d hand-object poses and interactions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4511–4520, 2019. 1, 2

[^46]: Tze Ho Elden Tse, Kwang In Kim, Ales Leonardis, and Hyung Jin Chang. Collaborative learning for hand and object reconstruction with attention-guided graph convolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1664–1674, 2022. 1

[^47]: Peng Wang, Hao Tan, Sai Bi, Yinghao Xu, Fujun Luan, Kalyan Sunkavalli, Wenping Wang, Zexiang Xu, and Kai Zhang. Pf-lrm: Pose-free large reconstruction model for joint pose and shape prediction. arXiv preprint arXiv:2311.12024, 2023. 2

[^48]: Ruicheng Wang, Sicheng Xu, Cassie Dai, Jianfeng Xiang, Yu Deng, Xin Tong, and Jiaolong Yang. Moge: Unlocking accurate monocular geometry estimation for open-domain images with optimal training supervision. arXiv preprint arXiv:2410.19115, 2024. 2, 3, 1

## [^49]: Zhengyi Wang, Yikai Wang, Yifei Chen, Chendong Xiang, Shuo Chen, Dajiang Yu, Chongxuan Li, Hang Su, and Jun

---

朱等人。Crm：基于卷积重建模型的单图像到3D纹理网格生成。载于《欧洲计算机视觉会议》，第57–74页。施普林格出版社，2024年。2

[^50]: Jianfeng Xiang, Zelong Lv, Sicheng Xu, Yu Deng, Ruicheng Wang, Bowen Zhang, Dong Chen, Xin Tong, and Jiaolong Yang. Structured 3d latents for scalable and versatile 3d generation. arXiv preprint arXiv:2412.01506, 2024. 2

[^51]: Jiale Xu, Weihao Cheng, Yiming Gao, Xintao Wang, Shenghua Gao, and Ying Shan. Instantmesh: Efficient 3d mesh generation from a single image with sparse-view large reconstruction models. arXiv preprint arXiv:2404.07191, 2024. 3, 4

[^52]: Lixin Yang, Kailin Li, Xinyu Zhan, Fei Wu, Anran Xu, Liu Liu, and Cewu Lu. Oakink: A large-scale knowledge repository for understanding hand-object interaction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 20953–20962, 2022. 2, 6, 5, 7

[^53]: Yufei Ye, Abhinav Gupta, and Shubham Tulsiani. What's in your hands? 3d reconstruction of generic objects in hands. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3895–3905, 2022. 2, 6

[^54]: Jason Y Zhang, Sam Pepose, Hanbyul Joo, Deva Ramanan, Jitendra Malik, and Angjoo Kanazawa. Perceiving 3d human-object spatial arrangements from a single image in the wild. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XII 16, pages 34–51. Springer, 2020. 2

[^55]: Xiong Zhang, Qiang Li, Hong Mo, Wenbo Zhang, and Wen Zheng. End-to-end hand mesh recovery from a monocular rgb image. In Proceedings of the IEEE/CVF international conference on computer vision, pages 2354–2364, 2019. 2

[^56]: Zibo Zhao, Zeqiang Lai, Qingxiang Lin, Yunfei Zhao, Haolin Liu, Shuhui Yang, Yifei Feng, Mingxin Yang, Sheng Zhang, Xianghui Yang, et al. Hunyuan3d 2.0: Scaling diffusion models for high resolution textured 3d assets generation. arXiv preprint arXiv:2501.12202, 2025. 2, 4, 1

[^57]: Yuxiao Zhou, Marc Habermann, Weipeng Xu, Ikhsanul Habibie, Christian Theobalt, and Feng Xu. Monocular real-time hand shape and motion capture using multi-modal data. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5346–5355, 2020. 2