# PAM：一种用于仿真到现实HOI视频生成的姿态-外观-运动引擎
高明举 $^{*1,2}$、杨凯森 $^{*2}$、高焕昂 $^{2}$、李博涵 $^{4,5}$、丁翱 $^{2}$、李文毅 $^{2}$、于阳成 $^{2}$、刘金坤 $^{2}$、徐少聪 $^{3}$、牛一可 $^{2}$、迟浩翰 $^{2}$、陈浩 $^{6}$、唐浩 $^{1}$、张宇 $^{\dagger4}$、易立 $^{2}$、赵浩 $^{\dagger2,3}$
$^{1}$北京大学 $^{2}$清华大学 $^{3}$北京人工智能研究院 $^{4}$上海交通大学
$^{5}$东方理工大学 $^{6}$剑桥大学
项目页面：
Figure 1. (a) Pose Generation. Given an initial and target hand–object pose, our pose generation module synthesizes a full MANO-based HOI trajectory. (b) Appearance Generation. Conditioned on rendered hand–object geometry, the appearance model produces diverse and realistic reference frame. (c) Motion Generation. Using the same rendered conditions, the motion model generates the full HOI video with coherent dynamics and hand–object interactions. (d) Quality and Downstream Validation. Our unified engine substantially improves video fidelity and geometric accuracy over prior methods.
## 摘要
手-物体交互（HOI）的重建与合成正逐渐成为具身人工智能和AR/VR领域的核心研究方向。然而，尽管相关研究进展迅速，现有的HOI生成研究仍分散在三个互不相关的方向：（1）仅姿态合成，只能预测MANO轨迹而无法生成像素级图像；（2）单图像HOI生成，能根据掩码或2D线索生成外观但缺乏动态效果；（3）视频生成方法需要完整的姿态序列以及真实的第一帧作为输入，因此无法真正应用于仿真到现实的场景。受[^33]理念的启发，我们认为HOI生成需要一个能将姿态、外观和运动整合到一个统一框架中的引擎。据此我们推出了
PAM：一种用于可控HOI视频生成的姿态-外观-运动引擎。我们的引擎性能通过以下实验得到验证：（1）在DexYCB数据集上，我们的FVD值达到29.13（而InterDyn的FVD值为38.83），MPJPE值为19.37毫米（而CosHand的MPJPE值为30.05毫米），同时还能生成分辨率为480×720的视频，优于基线模型的256×256/256×384分辨率。（2）在OAKINK2数据集上，我们的全多条件模型的FVD值从68.76降至46.31。（3）在DexYCB数据集上对输入条件进行的消融实验表明，结合深度、分割和关键点信息能始终获得最佳结果。（4）在下游的手部姿态估计任务中，使用SimpleHand模型时，若用3400段合成视频（共20.7万帧）扩充训练数据，那么仅用50%的真实数据加上这些合成数据训练的模型，其性能就能与用100%真实数据训练的基线模型相媲美。
---
Figure 2. Overview of Four Approaches to HOI Synthesis. (a) Pose-Only Synthesis [^79]: This method predicts the MANO trajectories without generating pixel data; (b) Appearance Generation [^83]: This approach generates appearance based on masks or 2D cues but lacks dynamic motion; (c) Motion Generation [^3] [^48]: These methods require both the full pose sequence and the ground-truth first frame as inputs, limiting their application for true sim-to-real transfer. (d) Our Pipeline PAM: In this approach, video generation does not rely on the first frame or the whole HOI pose sequence, allowing for the transfer of HOI pose sequences from the simulator to real-world videos.
### 1. 引言
用双手操控物体是人类的一项基本技能，而对这种能力的计算理解——即手-物体交互（HOI）理解——在计算机视觉和具身人工智能领域的重要性日益凸显。该领域已逐渐转向数据驱动的范式，大规模HOI数据集对于精准的手部姿态估计至关重要 [^32] [^56] [^65] [^75] [^76] [^80] [^93]，进而能够实现逼真的人机运动迁移 [^38] [^40] [^45] 以及3D建模[9–11, 18, 19, 44, 52, 58, 66, 76, 85, 87]。然而，关键挑战在于数据本身。尽管人们在收集带有详细标注的真实世界HOI序列上投入了大量资源 [^20] [^46] [^70]，但依赖成本高昂且劳动密集型的手动标注方式，仍是制约其规模化发展的根本瓶颈。
深度学习与扩散模型的出现为HOI视频的可扩展生成开辟了充满希望的道路。然而，正如图2所示，最先进的方法[^3] [^48] [^61] [^79] [^83] [^90]仍面临重大挑战：(1)仅基于姿态的合成方法在生成逼真外观方面存在局限，因为它们仅预测MANO轨迹而无法产生像素级细节，这降低了其实际应用价值。(2)单图像HOI生成（外观生成）方法依赖从掩码或2D线索中“幻想”出外观，无法捕捉时间动态，应用于真实场景时会导致动作不连贯。(3)视频生成（动作生成）方法需要完整的姿态序列和真实的第一帧，不适合实际部署，因为模拟器中无法提供真实的第一帧。这些局限性凸显了开发创新方法的必要性：这类方法需能在不依赖固定姿态序列或真实第一帧输入的情况下，生成高质量且时间上一致的HOI视频，从而为从模拟到现实的迁移提供更灵活、更稳健的解决方案。
在本文中，我们提出了PAM：一款开创性的用于从模拟到现实的HOI视频生成的姿态-外观-动作引擎，它仅需初始姿态、目标姿态以及物体几何信息作为输入。通过整合动作-外观扩散过程，我们的方法无需依赖条件化的第一帧，从而最大限度地提升了动作与外观的多样性——这是以往研究无法实现的能力。为确保高度逼真的效果，我们引入了多种约束条件，有效保留了精细的手部细节。
我们的框架包含三个核心阶段。姿态序列生成阶段利用预训练模型生成合理的手部运动轨迹。接下来，外观生成阶段采用可控图像扩散模型Flux [^5]来合成逼真的初始帧；该模型以深度图、语义掩码和手部关键点图的融合结果为条件，确保了几何精度、语义一致性以及对精细手部细节的保留。动作生成阶段首先将手部运动轨迹渲染为深度、语义和手部关键点序列，随后基于CogVideoX [^71]的可控视频扩散模型生成视频。重要的是，视频模型以与外观生成阶段相同的多模态输入为条件，以确保与生成的HOI姿态序列保持一致。
我们在DexYCB [^7]和OAKINK2 [^77]基准测试中评估了我们的方法，结果显示它在视频生成质量、动作合理性和手部姿态保真度方面均全面超越了现有方法。更重要的是，如图1所示，我们的方法生成的合成视频作为合成数据具有重大价值：用于训练时，能显著提升下游手部姿态估计模型的性能，证明了其作为数据增强工具的有效性。
我们将我们的贡献总结如下：
- 低条件约束的生成：我们开创了用于从模拟到现实的HOI视频生成的姿态-外观-动作引擎，它仅需稀疏的姿态关键帧和物体几何信息作为输入，从而克服了以往方法中的第一帧瓶颈。
- 解耦的生成架构：我们设计了一种新颖的流程，将姿态、外观和动作解耦……
---
该论文利用多模态条件来实现更出色的真实感、可控性和多样性。
• 顶尖的性能与实用性：我们的方法在现有基准测试中取得了优异成果，并且通过有效的数据增强在下游任务性能上实现了显著提升，从而证明了其实际应用价值。
### 2. 相关工作
#### 2.1. 手-物体运动合成
在计算机动画和机器人抓取领域，合成高保真度的手-物体运动是一项根本性挑战[^1] [^13] [^21] [^41] [^88]。目前主流的数据驱动方法依赖于从大规模、标注完善的数据集中进行监督学习[12, 15, 30, 31, 35, 40, 43, 49, 53, 86, 91–93]。然而，这些方法的可扩展性受到其对成本高昂且难以获取的数据的依赖限制[^7] [^17] [^20] [^26] [^46] [^47] [^70] [^77]。为克服这一局限，强化学习（RL）应运而生，成为一种颇具前景的替代方案。诸如[^13] [^69]之类的方法会在合成运动前先生成参考抓取姿态，而GraspXL[^79]则能在仿真环境中直接学习可泛化的抓取策略，无需预先定义参考姿态。这些基于强化学习的技术能够生成高质量的交互数据，为从仿真到现实的迁移奠定坚实基础。不过，这些方法缺乏外观建模能力。
#### 2.2. 可控视频生成
视频生成基础模型的最新突破[^2] [^6] [^36] [^63] [^71]，激发了人们对能与用户意图精准匹配的可控生成技术的浓厚兴趣。尽管文本转视频和图像转视频模型[^2] [^24] [^39] [^54] [^59] [^63] [^68] [^71] [^81]已展现出令人惊叹的能力，但它们往往缺乏应对专业任务的精细度。这推动了将语义图、深度信息和相机运动等更精确的控制信号融入研究的热情。ControlNet[^82]及其变体[^22] [^25] [^42] [^78]能够实现基于密集输入的条件约束，而VideoComposer[^67]等工作则通过融合多种条件来增强控制能力。通过将参数嵌入扩散模型，相机运动已被明确建模[^4] [^27]。然而，生成手-物体交互（HOI）视频面临独特挑战，因为手部运动具有极高的自由度。这就需要更为丰富且专门的控制机制——结合语义、几何和精确的姿态线索——才能达成所需的保真度和精确度。
#### 2.3. 手-物体交互图像与视频生成
生成手-物体交互（HOI）内容对于理解人类活动至关重要。以往关于HOI图像生成的研究[^8] [^28] [^37] [^51] [^64] [^74] [^83]通常以分割掩码和关键点等二维信号为条件。然而，这些静态方法无法捕捉交互的动态特性。近期，有几项研究[^3] [^14] [^16] [^48] [^61] [^73] [^89]探索了HOI视频生成。InterDyn[^3]通过ControlNet[^82]以手部掩码序列为条件，但未能充分利用模拟器提供的丰富条件。ManiVideo[^48]引入了遮挡感知表示，但需要人体外观数据，而这类数据无法从GraspXL[^79]等模拟器中获取。更重要的是，这些方法主要侧重于生成质量，并未充分研究其合成数据的下游实用性，而这种实用性对于验证感知指标之外的实际影响至关重要。
### 3. 所提出的方法
#### 3.1. 概述
图3展示了我们的方法。在初始MANO[^57]手部姿态$\mathbf{h}_0 \in \mathbb{R}^{51 \times 3}$、无外观的物体网格$\mathbf{m}$、初始6自由度物体姿态$\mathbf{o}_0 \in \mathbb{R}^6$以及目标手部姿态$\mathbf{h}_T \in \mathbb{R}^{51 \times 3}$的条件下，我们的生成模型
$$ f_{\boldsymbol{\theta}}:\left(\mathbf{h}_{0},\mathbf{m},\mathbf{o}_{0},\mathbf{h}_{T}\right)\to\left\{I_{t}\right\}_{t=0}^{T} $$
能够生成照片级真实的视频，该视频：(i) 以$\mathbf{h}_{0}$开始，(ii) 以$\mathbf{h}_{T}$结束，(iii) 呈现时间上连贯的抓取-放置动作。所有手部姿态均由全局平移+旋转以及关节角度参数化；帧$I_{t}$为RGB图像。
由于高维时空流形[^23]的存在，同时建模姿态、外观和运动本身极具挑战性。为解决这一问题，我们将生成过程分解为三个独立阶段：
1. 姿态生成——预训练的姿态生成模型根据初始和目标手-物姿态以及物体网格$\{\mathbf{h}_t, \mathbf{o}_t\}_{t=0}^T$，合成对齐的手-物姿态序列$(h_0, o_0, \mathbf{m})$（见第3.2节）。
2. 外观生成——可控图像扩散模型以初始HOI姿态和物体网格为条件，生成第一帧$I_{0}$（见第3.3节）。
3. 运动生成——将合成的HOI姿态序列和第一帧输入视频扩散模型，从而将$I_0$动画化为照片级真实的视频片段$\{I_t\}_{t=0}^T$（见第3.4节）。
#### 3.2. 第一阶段：姿态生成阶段
在第一阶段，我们的目标是生成插值后的手-物体交互（HOI）姿态序列，该序列从初始姿态过渡到目标姿态，同时纳入物体网格信息。如图3所示，我们采用了GraspXL[^79]，这是一款用于手-物体交互任务的预训练模型。GraspXL的输入为初始和目标MANO手部姿态$h_0$、6自由度物体姿态$o_0$以及物体网格m，输出结果为
---
Figure 3. Overview of our three-stage generation pipeline. (1) Pose Generation: A pretrained pose generation model generates the intermediate hand-object interaction (HOI) poses based on the initial and target poses, along with the object mesh. (2) Appearance Generation: A controllable image diffusion model synthesizes the first frame of the video, conditioned on multi-modal inputs (depth maps, semantic masks, and keypoint annotations). (3) Motion Generation: The generated HOI sequence and the first frame are rendered into a full video sequence by a video diffusion model, conditioned on the same multi-modal inputs used in the appearance generation stage.
手部和物体姿态的时间上连贯的轨迹$\{h_t, \mathbf{o}t\}_{t=0}^T$。这既确保了平滑的插值效果，又保持了手-物体交互的物理一致性，为后续的视频生成阶段奠定了基础。
#### 3.3. 第二阶段：外观生成阶段
模拟到现实HOI视频合成的桥接条件：本研究的主要目标是提升模拟视频的视觉质量，同时保留其他条件，从而缩小模拟场景与现实世界之间的差距。通过整合模拟器提供的几何信息（如深度图）和语义数据（如分割掩码），我们力求准确重建场景和物体的视觉表现，并确保所有其他条件的一致性。然而，仅依赖这两种数据类型不足以生成精确的手物交互（HOI）图像或视频。这一局限源于手部运动的复杂性和高自由度，仅靠几何与语义数据无法完全捕捉。具体而言，这些条件未能考虑手指数量及其各自姿态等关键细节。为解决这一问题，我们引入了额外条件——手关键点序列，正如[^83]所提出的那样——以实现更精确、更准确的手部姿态生成。该方法能够生成逼真的手部姿态，进而增强交互的整体真实感。在第4.1节和4.2节中，我们将探讨各条件的影响。
我们使用一个接受深度图[^5]、分割掩码[^82]以及手关键点图像$D_0$（每种均为$S_0$）的ControlNet$K_0$分支对Flux$H \times W \times 3$进行微调。所有提示信息均通过VAE编码为$\frac{H}{8} \times \frac{W}{8} \times 16$潜变量，再按通道维度拼接，并注入到两层DiT[^50]模块中，其权重初始化自原始Flux的前两层。
$$ f_{l}=f_{l}+\mathcal{Z}(f_{l}^{\prime}), $$
其中，$f_l$是原始DiT[^50]模块第l层的输出，而$f'_l$则是输入为拼接条件的复制版DiT模块第l层的输出。此处，$l \in \{0,1\}$和$\mathcal{Z}$代表零卷积层，即所有参数初始化为零的$1 \times 1$卷积。训练期间，仅有ControlNet的参数会被更新。
#### 3.4. 第三阶段：运动生成阶段
为了生成目标视频序列，我们将生成的HOI姿态序列与一个可控的视频扩散模型相结合。如图3所示，深度图、实例级
---
分割掩码以及二维手关键点图像会在第一阶段于每一帧被栅格化。随后，这些条件通过预训练的视频VAE编码为形状为$\mathbb{R}^{\frac{T+1}{4} \times \frac{H}{8} \times \frac{W}{8} \times 16}$的潜变量张量。接着，这些潜变量表示沿通道维度被集中，并通过12个复制的DiT模块注入到CogVideo-X中，如公式2所述。为防止训练时过度依赖某单一模态，每种提示信息会以0.2的概率被随机掩码处理。
### 4. 实验
#### 4.1. 实验设置
数据集与数据处理。我们在两个标准的HOI视频生成基准测试集上评估我们的方法：DexYCB[^7]和OAKINK2[^77]。对于DexYCB，我们采用s0-split划分，包含6,400个训练视频和1,600个验证视频。由于OAKINK2的规模较大，我们选用了8,000个视频片段（每个片段含49帧）作为子集，其中6,400个用于训练，1,600个用于验证。我们模型所需的条件——深度图、语义掩码和手关键点——获取方式如下：深度图由DepthCrafter[^29]估算得出，而语义和关键点信息则直接取自数据集标注。
评估指标。我们采用一套全面的指标，从四个维度来评估我们的方法：
- 图像质量：我们使用结构相似性指数（SSIM）、学习到的感知图像块相似度（LPIPS）[^84]以及峰值信噪比（PSNR）来评估感知质量。
• 时空一致性：我们采用Fréchet视频距离（FVD）[^62]来评估视频的整体真实感，该指标的实现来自[^60]。
- 运动保真度：我们使用运动保真度（MF）指标[^72]来量化动态准确性。对于每段视频，我们会选取100个前景点（位于手部或物体上），利用CoTracker3 [^34]对其进行跟踪，然后比较生成视频与真实视频中的轨迹。对于真实轨迹$\mathcal{T} = \{\tau_1, \ldots, \tau_T\}$和生成轨迹$\tilde{\mathcal{T}} = \{\tilde{\tau}_1, \ldots, \tilde{\tau}_T\}$，其中$\tau_t \in \mathbb{R}^2$，MF的定义如下：
$$ \mathbf{M}\mathbf{F}=\frac{1}{|\tilde{\mathcal{T}}|}\sum_{\tilde{\tau}\in\tilde{\mathcal{T}}}\max_{\tau\in\mathcal{T}}\mathbf{c o r r}(\tau,\tilde{\tau})+\frac{1}{|\mathcal{T}|}\sum_{\tau\in\mathcal{T}}\max_{\tilde{\tau}\in\tilde{\mathcal{T}}}\mathbf{c o r r}(\tau,\tilde{\tau}) $$
两条轨迹之间的相关性计算如下：
$$ \mathbf{c o r r}(\tau,\tilde{\tau})=\frac{1}{F}\sum_{k=1}^{F}\frac{\mathbf{v}_{k}\cdot\mathbf{\tilde{v}}_{k}}{\|\mathbf{v}_{k}\|\|\mathbf{\tilde{v}}_{k}\|}, $$
其中$\mathbf{v}_k = (v_k^x, v_k^y)$和$\tilde{\mathbf{v}}_k = (\tilde{v}_k^x, \tilde{v}_k^y)$分别是轨迹$\tau$和$\tilde{\tau}$在第k帧处的位移向量。
• 手部姿态精度：我们报告以毫米为单位的平均关节位置误差（MPJPE）[^17]，该数值衡量的是根节点对齐后，21个预测手部关节与真实关节之间的平均欧几里得距离。MPJPE越小，说明姿态估计精度越高。我们利用Hamer [^49]来估算生成视频中的手部关节。
#### 4.2. 主要结果
基线模型。我们将我们的方法与最先进的HOI视频生成方法进行对比，这些基线模型包括：ManiVideo [^48]、InterDyn [^3]和CosHand [^61]，测试数据集为DexYCB [^7]。对于ManiVideo和InterDyn，我们直接引用其原始论文中的结果（由于这些方法并非开源，部分无法获取结果的指标未予列出）。对于CosHand，我们使用官方实现，并在DexYCB的s0-split训练集上对它进行微调以实现公平对比。我们也在OAKINK2数据集上进行评估，对比对象为同样经过微调的CosHand模型。所有基线模型均在图像到视频的设置下进行评估，即提供真实的第一帧作为输入，因为这是这些方法的要求。
定量对比。我们的定量评估结果（表1、表2）表明，我们的方法在大多数指标上均达到了最先进的水平。我们将这种优越性归因于我们的多条件约束策略，该策略为扩散模型提供了丰富的几何与语义线索（深度图、掩码、关键点），从而能够同时优化视觉真实感与姿态精度。相比之下，基线模型存在局限性：InterDyn、ManiVideo和CosHand所依赖的条件信号较为有限，或者它们基于的基础模型难以捕捉手物交互的细微特征，因此性能表现欠佳。
定性比较。如图4所示，即便CosHand在相同的训练数据上进行了微调，我们的方法仍能生成视觉质量更优的结果。我们发现CosHand存在两大主要局限：(1)它仅依赖手部掩码作为条件信号，为重建精确的手部姿态提供的几何引导不足；(2)它缺乏显式的时间建模机制，导致帧与帧之间的输出不一致。相比之下，我们的方法通过利用配备时间注意力机制的视频扩散基础模型来解决这些问题，从而确保帧间的一致性。此外，将手部关键点图用作条件信号，能够明确保留手部结构细节，进而生成更准确、更流畅的视频序列。
#### 4.3. 输入条件的消融研究
我们在DexYCB数据集上开展消融研究，以评估不同输入条件的贡献。研究结果（表1、表2和表3）给出了三个关键发现：
随着条件数量的增加，性能也得到提升，这验证了我们的多条件设计的有效性。
---
| Method | Venue | FVD ( $\downarrow$) | MF ( $\uparrow$) | LPIPS ( $\downarrow$) | SSIM ( $\uparrow$) | PSNR ( $\uparrow$) | MPJPE ( $\downarrow$) | Resolution |
| CosHand [^61] | ECCV'24 | 58.51 | 0.591 | 0.139 | 0.767 | 23.20 | 30.05 | 256 x 256 |
| InterDyn [^3] | CVPR'25 | 38.83 | 0.680 | 0.119 | 0.848 | 24.86 | - | 256 x 384 |
| ManiVideo [^48] | CVPR'25 | - | - | 0.079 | 0.913 | 30.10 | 57.30 | - |
| Ours w/ all | - | 29.13 | 0.712 | 0.069 | 0.914 | 30.17 | 19.37 | 480 x 720 |
Table 1. Quantitative comparison on DexYCB dataset. Our method is evaluated against CosHand, InterDyn, and ManiVideo. Results for InterDyn and ManiVideo are taken from their original papers. For fair comparison, CosHand was fine-tuned on the s0-split training set identical to ours. Our approach achieves state-of-the-art performance across all metrics (FVD, LPIPS, MF, MPJPE) while generating high-resolution 480x720 videos.
| Method | FVD ( $\downarrow$) | MF ( $\uparrow$) | LPIPS ( $\downarrow$) | SSIM ( $\uparrow$) | PSNR ( $\uparrow$) | MPJPE ( $\downarrow$) |
| CosHand [^61] | 68.76 | 0.651 | 0.156 | 0.765 | 23.84 | 14.49 |
| Ours w/ seg | 48.97 | 0.708 | 0.084 | 0.831 | 25.76 | 9.61 |
| Ours w/ depth | 50.85 | 0.702 | 0.086 | 0.845 | 26.98 | 10.07 |
| Ours w/ hand | 52.41 | 0.671 | 0.113 | 0.838 | 25.66 | 8.01 |
| Ours w/ all | 46.31 | 0.777 | 0.081 | 0.851 | 28.36 | 7.01 |
Table 2. Quantitative results on the OAKINK2 dataset. Comparison of our method with CosHand. For a fair evaluation, both models are trained on the same dataset. Our approach achieves state-of-the-art performance, outperforming CosHand across all evaluated metrics.
| Conditions | FVD ( $\downarrow$) | MF ( $\uparrow$) | LPIPS ( $\downarrow$) | SSIM ( $\uparrow$) | PSNR ( $\uparrow$) | MPJPE ( $\downarrow$) |
| Seg | 33.23 | 0.695 | 0.077 | 0.900 | 29.27 | 21.14 |
| Depth | 30.00 | 0.703 | 0.070 | 0.906 | 29.15 | 23.16 |
| Hand | 33.41 | 0.713 | 0.086 | 0.901 | 29.07 | 20.70 |
| Depth, Hand | 29.62 | 0.711 | 0.071 | 0.899 | 29.95 | 20.46 |
| Seg, Hand | 29.53 | 0.711 | 0.073 | 0.902 | 29.57 | 19.92 |
| Depth, Seg | 29.32 | 0.712 | 0.071 | 0.906 | 30.60 | 22.51 |
| Depth, Hand, Seg | 29.13 | 0.712 | 0.069 | 0.914 | 30.17 | 19.37 |
Table 3. Ablation study on input conditions on DexYCB dataset.
条件设计。
- 即便使用与CosHand和InterDyn相同的分割掩码条件，我们的方法仍能取得更优结果，这充分体现了我们流程的优势。
- 虽然仅使用手部关键点时MPJPE数值较低（得益于显式的姿态监督），但在其他指标上的表现却较差，这是因为缺乏更广泛的几何与语义上下文信息。这一结果凸显出：要实现最佳性能，必须将详细的局部线索（关键点）与全局场景理解（深度、语义）相结合。
图5中的可视化结果进一步支持了上述发现：使用所有条件时能生成准确的姿态；仅使用语义掩码或深度图会导致姿态不准确；而仅使用关键点则会降低外观质量。
#### 4.4. 多样化的仿真到现实迁移
我们开展了仿真到现实的迁移实验，以验证完整流程的有效性。为此，我们选用GraspXL [^79]作为手部运动生成器，因为它性能优异且泛化能力强大。我们使用DexYCB数据集中的物体，随机初始化手部和物体的姿态以及目标手部姿态。随后由GraspXL生成中间运动序列，用于为我们的视频生成模型渲染所需的条件——深度、语义掩码和关键点。如图1和图6所示，我们的方法仅利用初始姿态、目标姿态以及物体几何信息这些极简的输入，便成功合成出主体和背景各不相同的多样化、逼真的视频。这种能力源于我们的解耦生成架构，它能有效结合GraspXL的运动先验与我们扩散模型的外观建模能力。第4.5节将探讨这些合成视频在下游任务中的应用价值。
#### 4.5. 下游任务验证
为了评估我们生成的视频的实用性，我们在手部姿态估计任务中将其用于数据增强。我们使用SimpleHand [^93]作为姿态估计模型，该模型能从单张图像中回归出MANO参数 [^57]。我们的流程在DexYCB数据集上训练，可生成3,400个视频序列（共207,400帧）用于增强数据。我们将这些合成数据与不同比例（25%、50%、75%、100%）的原始DexYCB s0-split训练集（406,888帧）相结合。所有模型均在DexYCB验证集上通过四项指标进行评估：Procrustes对齐的平均关节位置误差（PA-MPJPE）、Procrustes对齐的平均顶点位置误差（PA-MPVPE）以及F分数。
---
Figure 4. Qualitative comparison against CosHand. Example results on DexYCB and OAKINK2 highlight the strengths of our method in two key areas: (1) higher visual fidelity in both foreground and background generation, and (2) improved geometric accuracy of the synthesized hand poses.
Figure 5. Ablation study on input conditions on DexYCB dataset.
PA-MPJPE和PA-MPVPE分别衡量预测关节/顶点与真实关节/顶点在Procrustes对齐后的平均欧几里得距离（单位：毫米）。此处使用的PA-MPJPE指标与表1中的有所不同。在此语境下，MPJPE用于衡量下游任务的数据效率，而表1中的MPJPE则指生成视频的手部精度。
定量结果（表4）表明，引入我们生成的数据后，所有指标下的手部姿态估计精度均得到持续提升。图7揭示了两个关键趋势：（1）随着真实数据量的增加，模型性能单调提升；（2）最值得注意的一点是，仅使用50%的真实数据，再辅以我们的合成样本，就能达到与100%真实数据基线相媲美的性能。这表明我们的合成数据能够有效弥补真实数据量不足的问题。此外，使用多条件生成的视频所展现出的更优性能，也验证了我们的多条件生成方法在产生多样化且实用的训练数据方面的重要性。
---
Figure 6. Sim-to-real transfer results. Our pipeline can generate realistic videos given initial and target states with diversity.
| Setting | PA-MPJPE ( $\downarrow$) | PA-MPVPE ( $\downarrow$) | F-Score@05 ( $\uparrow$) | F-Score@15( $\uparrow$) |
| All real data | 5.5 | 5.5 | 0.7953 | 0.9899 |
| All gen. data | 8.2 | 8.1 | 0.6274 | 0.9626 |
| All gen. + 25% real data | 6.1 | 6.0 | 0.7512 | 0.9851 |
| All gen. + 50% real data | 5.5 | 5.5 | 0.8001 | 0.9879 |
| All gen. + 75% real data | 5.4 | 5.3 | 0.7984 | 0.9899 |
| All gen. + 100% real data | 5.3 | 5.3 | 0.8025 | 0.9904 |
Table 4. Downstream task evaluation on SimpleHand [^93].
Figure 7. Data augmentation analysis with varying ratios of real data. We augment different portions of the DexYCB training set (25%, 50%, 75%, 100%) with our generated synthetic data. The baseline (dashed line) indicates performance when training solely on 100% of the real DexYCB data without synthetic augmentation.
Figure 8. Zero-shot result on OAKINK2 dataset for i2v task. We use the weight trained on DexYCB dataset.
#### 4.6. 零样本实验结果
为了评估我们方法的泛化能力，我们将在DexYCB数据集（单手交互）上训练的视频扩散模型直接在OAKINK2数据集（双手交互）上进行零样本图像到视频生成测试。如图8所示，尽管存在显著的领域差异，我们的方法仍能生成合理的视频，且这些视频能与真实的手部姿态及视觉细节保持较好的一致性。这种跨数据集的泛化能力得益于我们将预训练的视频扩散模型权重作为坚实基础，同时结合ControlNet机制 [^82]，该机制有助于在适应新条件信号的同时保留模型原有的生成质量。
### 5. 结论
本文提出了一种框架，旨在解决从极少量姿态输入生成逼真的人体-物体交互视频这一难题。我们提出的解耦式多条件架构在感知质量和几何精度方面均取得了优异成果，并通过提升下游任务性能展现了其实际应用价值。虽然我们的方法展现出强大的泛化能力，但未来的工作可以探索将其扩展到更复杂的物体交互场景，或将运动与外观生成阶段整合为端到端模型。我们相信，我们的研究成果能为具身人工智能领域的生成模型研究奠定坚实基础。
---
## 致谢
本研究部分得到了北京市自然科学基金QY25047项目的资助。
## 参考文献
[^1]: Ananye Agarwal, Shagun Uppal, Kenneth Shaw, and Deepak Pathak. Dexterous functional grasping. arXiv preprint arXiv:2312.02975, 2023. 3
[^2]: Niket Agarwal, Arslan Ali, Maciej Bala, Yogesh Balaji, Erik Barker, Tiffany Cai, Prithvijit Chattopadhyay, Yongxin Chen, Yin Cui, Yifan Ding, et al. Cosmos world foundation model platform for physical ai. arXiv preprint arXiv:2501.03575, 2025. 3
[^3]: Rick Akkerman, Haiwen Feng, Michael J Black, Dimitrios Tzionas, and Victoria Fernández Abrevaya. Interdyn: Controllable interactive dynamics with video diffusion models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 12467–12479, 2025. 2, 3, 5, 6
[^4]: Jianhong Bai, Menghan Xia, Xiao Fu, Xintao Wang, Lianrui Mu, Jinwen Cao, Zuozhu Liu, Haoji Hu, Xiang Bai, Pengfei Wan, et al. Recammaster: Camera-controlled generative rendering from a single video. arXiv preprint arXiv:2503.11647, 2025. 3
[^5]: black-forest labs. , 2024. 2, 4
[^6]: Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127, 2023. 3, 14
[^7]: Yu-Wei Chao, Wei Yang, Yu Xiang, Pavlo Molchanov, Ankur Handa, Jonathan Tremblay, Yashraj S Narang, Karl Van Wyk, Umar Iqbal, Stan Birchfield, et al. Dexycb: A benchmark for capturing hand grasping of objects. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9044–9053, 2021. 2, 3, 5
[^8]: Kefan Chen, Chaerin Min, Linguang Zhang, Shreyas Hampali, Cem Keskin, and Srinath Sridhar. Foundhand: Large-scale domain-specific learning for controllable hand image generation. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 17448–17460, 2025. 3
[^9]: Xiaoxue Chen, Tianyu Liu, Hao Zhao, Guyue Zhou, and Ya-Qin Zhang. Cerberus transformer: Joint semantic, affordance and attribute parsing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19649–19658, 2022. 2
[^10]: Xingyu Chen, Zhuheng Song, Xiaoke Jiang, Yaoqing Hu, Junzhi Yu, and Lei Zhang. Handos: 3d hand reconstruction in one stage. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 17304–17314, 2025.
[^11]: Zerui Chen, Rolandos Alexandros Potamias, Shizhe Chen, and Cordelia Schmid. Hort: Monocular hand-held objects reconstruction with transformers. arXiv preprint arXiv:2503.21313, 2025. 2
[^12]: Sammy Christen, Muhammed Kocabas, Emre Aksan, Jemin Hwangbo, Jie Song, and Otmar Hilliges. D-grasp: Physically
用于手物交互的可行动态抓取合成技术。载于IEEE/CVF计算机视觉与模式识别会议论文集，第20577–20586页，2022年。3
[^13]: Sammy Christen, Lan Feng, Wei Yang, Yu-Wei Chao, Otmar Hilliges, and Jie Song. Synh2r: Synthesizing hand-object motions for learning human-to-robot handovers. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 3168–3175. IEEE, 2024. 3
[^14]: Lingwei Dang, Ruizhi Shao, Hongwen Zhang, Wei Min, Yebin Liu, and Qingyao Wu. Svimo: Synchronized diffusion for video and motion generation in hand-object interaction scenarios. arXiv preprint arXiv:2506.02444, 2025. 3
[^15]: Haoye Dong, Aviral Chharia, Wenbo Gou, Francisco Vicente Carrasco, and Fernando D De la Torre. Hamba: Single-view 3d hand reconstruction with graph-guided bi-scanning mamba. Advances in Neural Information Processing Systems, 37:2127–2160, 2024. 3
[^16]: Yingying Fan, Quanwei Yang, Kaisiyuan Wang, Hang Zhou, Yingying Li, Haocheng Feng, Errui Ding, Yu Wu, and Jingdong Wang. Re-hold: Video hand object interaction reenactment via adaptive layout-instructed diffusion model. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 17550–17560, 2025. 3
[^17]: Zicong Fan, Omid Taheri, Dimitrios Tzionas, Muhammed Kocabas, Manuel Kaufmann, Michael J Black, and Otmar Hilliges. Arctic: A dataset for dexterous bimanual hand-object manipulation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12943–12954, 2023. 3, 5
[^18]: Yao Feng, Weiyang Liu, Timo Bolkart, Jinlong Yang, Marc Pollefeys, and Michael J Black. Learning disentangled avatars with hybrid 3d representations. arXiv preprint arXiv:2309.06441, 2023. 2
[^19]: Yao Feng, Jing Lin, Sai Kumar Dwivedi, Yu Sun, Priyanka Patel, and Michael J Black. Chatpose: Chatting about 3d human pose. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2093–2103, 2024. 2
[^20]: Rao Fu, Dingxi Zhang, Alex Jiang, Wanjia Fu, Austin Funk, Daniel Ritchie, and Srinath Sridhar. Gigahands: A massive annotated dataset of bimanual hand activities. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 17461–17474, 2025. 2, 3
[^21]: Anindita Ghosh, Rishabh Dabral, Vladislav Golyanik, Christian Theobalt, and Philipp Slusallek. Imos: Intent-driven full-body motion synthesis for human-object interactions. In Computer Graphics Forum, pages 1–12. Wiley Online Library, 2023. 3
[^22]: Zekai Gu, Rui Yan, Jiahao Lu, Peng Li, Zhiyang Dou, Chenyang Si, Zhen Dong, Qifeng Liu, Cheng Lin, Ziwei Liu, et al. Diffusion as shader: 3d-aware video diffusion for versatile video generation control. In Proceedings of the Special Interest Group on Computer Graphics and Interactive Techniques Conference Conference Papers, pages 1–12, 2025. 3
### [^23]: Xiefan Guo, Jinlin Liu, Miaomiao Cui, Liefeng Bo, and Di
---
黄。I4vgen：作为文本转视频生成自由基石的图像。arXiv预印本arXiv:2406.02230，2024年。3
[^24]: Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. arXiv preprint arXiv:2307.04725, 2023. 3
[^25]: Yuwei Guo, Ceyuan Yang, Anyi Rao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Sparsectrl: Adding sparse controls to text-to-video diffusion models. In European Conference on Computer Vision, pages 330–348. Springer, 2024. 3
[^26]: Shreyas Hampali, Mahdi Rad, Markus Oberweger, and Vincent Lepetit. Honnotate: A method for 3d annotation of hand and object poses. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3196–3206, 2020. 3
[^27]: Hao He, Yinghao Xu, Yuwei Guo, Gordon Wetzstein, Bo Dai, Hongsheng Li, and Ceyuan Yang. Cameractrl: Enabling camera control for text-to-video generation. arXiv preprint arXiv:2404.02101, 2024. 3
[^28]: Hezhen Hu, Weilun Wang, Wengang Zhou, and Houqiang Li. Hand-object interaction image generation. Advances in Neural Information Processing Systems, 35:23805–23817, 2022. 3
[^29]: Wenbo Hu, Xiangjun Gao, Xiaoyu Li, Sijie Zhao, Xiaodong Cun, Yong Zhang, Long Quan, and Ying Shan. Depthcrafter: Generating consistent long depth sequences for open-world videos. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 2005–2015, 2025. 5
[^30]: Mingzhen Huang, Fu-Jen Chu, Bugra Tekin, Kevin J Liang, Haoyu Ma, Weiyao Wang, Xingyu Chen, Pierre Gleize, Hongfei Xue, Siwei Lyu, et al. Hoigpt: Learning long-sequence hand-object interaction with language models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 7136–7146, 2025. 3
[^31]: Hanwen Jiang, Shaowei Liu, Jiashun Wang, and Xiaolong Wang. Hand-object contact consistency reasoning for human grasps generation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 11107–11116, 2021. 3
[^32]: Shijian Jiang, Qi Ye, Rengan Xie, Yuchi Huo, and Jiming Chen. Hand-held object reconstruction from RGB video with dynamic interaction. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 12220–12230, 2025. 2
[^33]: Hanbyul Joo, Tomas Simon, and Yaser Sheikh. Total capture: A 3d deformation model for tracking faces, hands, and bodies. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8320–8329, 2018. 1
[^34]: Nikita Karaev, Iurii Makarov, Jianyuan Wang, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. C-tracker3: Simpler and better point tracking by pseudolabelling real videos. arXiv preprint arXiv:2410.11831, 2024.5
[^35]: Korrawe Karunratanakul, Jinlong Yang, Yan Zhang, Michael J Black, Krikamol Muandet, and Siyu Tang. Grasping field: Learning implicit representations for human grasps. In 2020
国际3D视觉会议（3DV），第333–344页。IEEE出版社，2020年。3
[^36]: Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models. arXiv preprint arXiv:2412.03603, 2024. 3
[^37]: Patrick Kwon, Chen Chen, and Hanbyul Joo. Graspdiffusion: Synthesizing realistic whole-body hand-object interaction. arXiv preprint arXiv:2410.13911, 2024. 3
[^38]: Marion Lepert, Jiaying Fang, and Jeannette Bohg. Phantom: Training robots without robots using only human videos. arXiv preprint arXiv:2503.00779, 2025. 2
[^39]: Bohan Li, Jiazhe Guo, Hongsi Liu, Yingshuang Zou, Yikang Ding, Xiwu Chen, Hu Zhu, Feiyang Tan, Chi Zhang, Tiancai Wang, et al. Uniscene: Unified occupancy-centric driving scene generation. In Proceedings of the computer vision and pattern recognition conference, pages 11971–11981, 2025. 3
[^40]: Kailin Li, Puhao Li, Tengyu Liu, Yuyang Li, and Siyuan Huang. Maniptrans: Efficient dexterous bimanual manipulation transfer via residual learning. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 6991–7003, 2025. 2, 3
[^41]: Muchen Li, Sammy Christen, Chengde Wan, Yujun Cai, Renjie Liao, Leonid Sigal, and Shugao Ma. Latenthoi: On the generalizable hand object motion generation with latent hand diffusion. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 17416–17425, 2025. 3
[^42]: Wenyi Li, Haoran Xu, Guiyu Zhang, Huan-ang Gao, Mingju Gao, Mengyu Wang, and Hao Zhao. Fairdiff: Fair segmentation with point-image diffusion. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 617–628. Springer, 2024. 3
[^43]: Xueyi Liu and Li Yi. Geneoh diffusion: Towards generalizable hand-object interaction denoising via denoising diffusion. arXiv preprint arXiv:2402.14810, 2024. 3
[^44]: Xingyu Liu, Pengfei Ren, Qi Qi, Haifeng Sun, Zirui Zhuang, Jing Wang, Jianxin Liao, and Jingyu Wang. Generalizable hand-object modeling from monocular rgb images via 3d gaussians. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. 2
[^45]: Xueyi Liu, Jianibieke Adalibieke, Qianwei Han, Yuzhe Qin, and Li Yi. Dextrack: Towards generalizable neural tracking control for dexterous manipulation from human references. arXiv preprint arXiv:2502.09614, 2025. 2
[^46]: Yunze Liu, Yun Liu, Che Jiang, Kangbo Lyu, Weikang Wan, Hao Shen, Boqiang Liang, Zhoujie Fu, He Wang, and Li Yi. Hoi4d: A 4d egocentric dataset for category-level human-object interaction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21013–21022, 2022. 2, 3
[^47]: Yun Liu, Haolin Yang, Xu Si, Ling Liu, Zipeng Li, Yuxiang Zhang, Yebin Liu, and Li Yi. Taco: Benchmarking generalizable bimanual tool-action-object understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21740–21751, 2024. 3
### [^48]: Youxin Pang, Ruizhi Shao, Jiajun Zhang, Hanzhang Tu, Yun Liu, Boyao Zhou, Hongwen Zhang, and Yebin Liu
---
Manivideo：生成具有灵巧且可泛化抓取能力的手物操作视频。载于计算机视觉与模式识别会议论文集，第12209–12219页，2025年。2、3、5、6
[^49]: Georgios Pavlakos, Dandan Shan, Ilija Radosavovic, Angjoo Kanazawa, David Fouhey, and Jitendra Malik. Reconstructing hands in 3d with transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9826–9836, 2024. 3, 5, 13
[^50]: William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4195–4205, 2023. 4
[^51]: Anton Pelykh, Ozge Mercanoglu Sincan, and Richard Bowden. Giving a hand to diffusion models: a two-stage approach to improving conditional human image generation. In 2024 IEEE 18th International Conference on Automatic Face and Gesture Recognition (FG), pages 1–10. IEEE, 2024. 3
[^52]: Rolandos Alexandros Potamias, Jinglei Zhang, Jiankang Deng, and Stefanos Zafeiriou. Wilor: End-to-end 3d hand localization and reconstruction in-the-wild. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 12242–12254, 2025. 2
[^53]: Aditya Prakash, Benjamin Lundell, Dmitry Andreychuk, David Forsyth, Saurabh Gupta, and Harpreet Sawhney. How do I do that? synthesizing 3d hand motion and contacts for everyday interactions. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 7026–7036, 2025. 3
[^54]: Zhiwu Qing, Shiwei Zhang, Jiayu Wang, Xiang Wang, Yujie Wei, Yingya Zhang, Changxin Gao, and Nong Sang. Hierarchical spatio-temporal decoupling for text-to-video generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6635–6645, 2024. 3
[^55]: Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–16. IEEE, 2020. 13
[^56]: Pengfei Ren, Jingyu Wang, Haifeng Sun, Qi Qi, Xingyu Liu, Menghao Zhang, Lei Zhang, Jing Wang, and Jianxin Liao. Prior-aware dynamic temporal modeling framework for sequential 3d hand pose estimation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6476–6487, 2025. 2
[^57]: Javier Romero, Dimitrios Tzionas, and Michael J Black. Embodied hands: Modeling and capturing hands and bodies together. arXiv preprint arXiv:2201.02610, 2022. 3, 6
[^58]: Muhammad Usama Saleem, Ekkasit Pinyoanuntapong, Mayur Jagdishbhai Patel, Hongfei Xue, Ahmed Helmy, Srijan Das, and Pu Wang. Maskhand: Generative masked modeling for robust hand mesh reconstruction in the wild. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8372–8383, 2025. 2
[^59]: Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual,
奥兰·加夫尼等。Make-a-video：无需文本-视频数据即可实现文本转视频生成。arXiv预印本arXiv:2209.14792，2022年。3
[^60]: Ivan Skorokhodov, Sergey Tulyakov, and Mohamed Elhoseiny. Stylegan-v: A continuous video generator with the price, image quality and perks of stylegan2. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3626–3636, 2022. 5
[^61]: Sruthi Sudhakar, Ruoshi Liu, Basile Van Hoorick, Carl Vondrick, and Richard Zemel. Controlling the world by sleight of hand. In European Conference on Computer Vision, pages 414–430. Springer, 2024. 2, 3, 5, 6
[^62]: Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: A new metric & challenges. ArXiv, abs/1812.01717, 2018. 5
[^63]: Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, et al. Wan: Open and advanced large-scale video generative models. arXiv preprint arXiv:2503.20314, 2025. 3
[^64]: Benzhi Wang, Jingkai Zhou, Jingqi Bai, Yang Yang, Weihua Chen, Fan Wang, and Zhen Lei. Realishuman: A two-stage approach for refining malformed human parts in generated images. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 7509–7517, 2025. 3
[^65]: Jikai Wang, Qifan Zhang, Yu-Wei Chao, Bowen Wen, Xiaohu Guo, and Yu Xiang. Ho-cap: A capture system and dataset for 3d reconstruction and pose tracking of hand-object interaction. arXiv preprint arXiv:2406.06843, 2024. 2
[^66]: Shibo Wang, Haonan He, Maria Parelli, Christoph Gebhardt, Zicong Fan, and Jie Song. Magichoi: Leveraging 3d priors for accurate hand-object reconstruction from short monocular video clips. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5957–5968, 2025. 2
[^67]: Xiang Wang, Hangjie Yuan, Shiwei Zhang, Dayou Chen, Jiuniu Wang, Yingya Zhang, Yujun Shen, Deli Zhao, and Jingren Zhou. Videocomposer: Compositional video synthesis with motion controllability. Advances in Neural Information Processing Systems, 36:7594–7611, 2023. 3
[^68]: Ruben Wiersma, Julien Philip, Miloš Hašan, Krishna Mullia, Fujun Luan, Elmar Eisemann, and Valentin Deschaintre. Uncertainty for svbrdf acquisition using frequency analysis. In Proceedings of the Special Interest Group on Computer Graphics and Interactive Techniques Conference Conference Papers, pages 1–12, 2025. 3
[^69]: Yinzhen Xu, Weikang Wan, Jialiang Zhang, Haoran Liu, Zikang Shan, Hao Shen, Ruicheng Wang, Haoran Geng, Yijia Weng, Jiayi Chen, et al. Unidexgrasp: Universal robotic dexterous grasping via learning diverse proposal generation and goal-conditioned policy. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4737–4746, 2023. 3
#### [^70]: Lixin Yang, Kailin Li, Xinyu Zhan, Fei Wu, Anran Xu, Liu Liu, and Cewu Lu. Oakink: A large-scale knowledge repository for understanding hand-object interaction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 20953–20962, 2022. 2, 3
---
[^71]: Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint arXiv:2408.06072, 2024. 2, 3
[^72]: Danah Yatim, Rafail Fridman, Omer Bar-Tal, Yoni Kasten, and Tali Dekel. Space-time diffusion features for zero-shot text-driven motion transfer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8466–8476, 2024. 5
[^73]: Kai Ye, Yuhang Wu, Shuyuan Hu, Junliang Li, Meng Liu, Yongquan Chen, and Rui Huang. \textsc {Gen2Real}: Towards demo-free dexterous manipulation by harnessing generated video. arXiv preprint arXiv:2509.14178, 2025. 3
[^74]: Yufei Ye, Xueting Li, Abhinav Gupta, Shalini De Mello, Stan Birchfield, Jiaming Song, Shubham Tulsiani, and Sifei Liu. Affordance diffusion: Synthesizing hand-object interactions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22479–22489, 2023. 3
[^75]: Brent Yi, Vickie Ye, Maya Zheng, Yunqi Li, Lea Müller, Georgios Pavlakos, Yi Ma, Jitendra Malik, and Angjoo Kanazawa. Estimating body and hand motion in an ego-sensed world. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 7072–7084, 2025. 2
[^76]: Zhenjun Yu, Wenqiang Xu, Pengfei Xie, Yutong Li, Brian W Anthony, Zhuorui Zhang, and Cewu Lu. Dynamic reconstruction of hand-object interaction with distributed force-aware contact representation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8590–8599, 2025. 2
[^77]: Xinyu Zhan, Lixin Yang, Yifei Zhao, Kangrui Mao, Hanlin Xu, Zenan Lin, Kailin Li, and Cewu Lu. Oakink2: A dataset of bimanual hands-object manipulation in complex task completion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 445–456, 2024. 2, 3, 5
[^78]: Guiyu Zhang, Huan-ang Gao, Zijian Jiang, Hao Zhao, and Zhedong Zheng. Ctrl-u: Robust conditional image generation via uncertainty-aware reward modeling. arXiv preprint arXiv:2410.11236, 2024. 3
[^79]: Hui Zhang, Sammy Christen, Zicong Fan, Otmar Hilliges, and Jie Song. Graspxl: Generating grasping motions for diverse objects at scale. In European Conference on Computer Vision, pages 386–403. Springer, 2024. 2, 3, 6
[^80]: Jinglei Zhang, Jiankang Deng, Chao Ma, and Rolandos Alexandros Potamias. Hawor: World-space hand motion reconstruction from egocentric videos. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 1805–1815, 2025. 2
[^81]: Lvmin Zhang and Maneesh Agrawala. Packing input frame context in next-frame prediction models for video generation. arXiv preprint arXiv:2504.12626, 2025. 3
[^82]: Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF international conference on computer vision, pages 3836–3847, 2023. 3, 4, 8
[^83]: Mengqi Zhang, Yang Fu, Zheng Ding, Sifei Liu, Zhuowen Tu, and Xiaolong Wang. Hoidiffusion: Generating realistic 3d
手物交互数据。载于IEEE/CVF计算机视觉与模式识别会议论文集，第8521–8531页，2024年。2、3、4
[^84]: Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586–595, 2018. 5
[^85]: Yufei Zhang, Zijun Cui, Jeffrey O Kephart, and Qiang Ji. Diffusion-based 3d hand motion recovery with intuitive physics. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7306–7317, 2025. 2
[^86]: Zhenhao Zhang, Ye Shi, Lingxiao Yang, Suting Ni, Qi Ye, and Jingya Wang. Openhoi: Open-world hand-object interaction synthesis with multimodal large language model. arXiv preprint arXiv:2505.18947, 2025. 3
[^87]: Hao Zhao, Ming Lu, Anbang Yao, Yiwen Guo, Yurong Chen, and Li Zhang. Physics inspired optimization on semantic transfer features: An alternative method for room layout estimation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 10–18, 2017. 2
[^88]: Hao Zhao, Ming Lu, Anbang Yao, Yurong Chen, and Li Zhang. Learning to draw sight lines. International Journal of Computer Vision, 128(5):1076–1100, 2020. 3
[^89]: Hongxiang Zhao, Xingchen Liu, Mutian Xu, Yiming Hao, Weikai Chen, and Xiaoguang Han. Taste-rob: Advancing video generation of task-oriented hand-object interaction for generalizable robotic manipulation. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 27683–27693, 2025. 3
[^90]: Zhuoran Zhao, Linlin Yang, Pengzhan Sun, Pan Hui, and Angela Yao. Analyzing the synthetic-to-real domain gap in 3d hand pose estimation. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 12255–12265, 2025. 2
[^91]: Yiming Zhong, Qi Jiang, Jingyi Yu, and Yuexin Ma. Dexgrasp anything: Towards universal robotic dexterous grasping with physics awareness. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 22584–22594, 2025. 3
[^92]: Bohan Zhou, Yi Zhan, Zhongbin Zhang, and Zongqing Lu. Megohand: Multimodal egocentric hand-object interaction motion generation. arXiv preprint arXiv:2505.16602, 2025.
#### [^93]: Zhishan Zhou, Shihao Zhou, Zhi Lv, Minqiang Zou, Yao Tang, and Jiajun Liang. A simple baseline for efficient hand mesh reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1367–1376, 2024. 2, 3, 6, 8
---
#### 6. 实现细节
| Resource Usage | Stage-I | Stage-II | Stage-III |
| Memory (GB) | 0.03 | 41.4 | 30.3 |
| Time (s) | 19.3 | 36.1 | 245.7 |
Table 5. Resource Usage across Different Stages
训练细节：我们的模型在由8块NVIDIA 800 GPU组成的设备上训练，批量大小为4×8，学习率为$1 \times 10^{-4}$。训练过程共包含8000个训练步骤，采用AdamW优化器和DeepSpeed训练架构[^55]。
评估细节：在视频生成评估中，我们从测试集中采样1600个视频，每个视频包含49帧。在评估平均关节位置误差（MPJPE）时，我们利用Hamer [^49]来估计生成视频中的手部关节，并通过比较估计的关节位置与真实手部关节来计算损失值。为了评估下游任务的表现，我们使用SimpleHand模型的官方实现对其训练200个周期。
下游验证细节：对于下游任务，给定输入后，我们首先利用外观生成器（可控图像扩散模型）随机采样30个候选结果。随后，我们使用Hamer筛选出手部姿态精度较低的样本。具体而言，我们对每一帧使用Hamer预测手部关键点，将其与真实值进行比较，并丢弃姿态精度排名靠后的25%的生成视频。
#### 7. 额外结果
#### 7.1. 计算与内存基准测试
如表5所示，我们报告了在NVIDIA H20 GPU上各阶段的资源使用情况。第一阶段计算量较轻，第二阶段的内存峰值最高（41.4 GB），第三阶段由于涉及扩散计算，耗时最长（245.7秒）。总体而言，生成40帧视频的完整流程耗时301.1秒。
#### 7.2. 掩码概率与输入条件质量的消融实验
我们通过向输入条件中加入随机高斯噪声来评估模型的鲁棒性。加入这种噪声是为了模拟现实世界中的扰动，从而检验模型在此类干扰下保持性能的能力。如表6所示，当不应用掩码时，模型性能会出现明显退化——模型在受到噪声影响时，各项性能指标显著下降。
然而，当对输入条件应用掩码后，性能下降幅度大幅减小。这一
Table 6. Ablation Study on Masking Probability and Input Condition Quality
| Settings | FVD ( $\downarrow$) | PSNR ( $\uparrow$) | MPJPE ( $\downarrow$) |
| 0 Mask Prob + Clean Cond | 28.56 | 30.99 | 19.01 |
| 0 Mask Prob + Noisy Cond | 34.58 | 27.11 | 23.67 |
| 0.2 Mask Prob + Clean Cond (Ours) | 29.13 | 30.17 | 19.37 |
| 0.2 Mask Prob + Noisy Cond | 30.45 | 29.67 | 20.31 |
观察结果强烈表明，随机掩码技术能够提升模型的鲁棒性。通过对输入数据的一部分进行掩码处理，模型对噪声的敏感度似乎有所降低，这可能是因为模型会更多地关注数据中更稳定、噪声更少的特征。因此，使用随机掩码能够提升模型的泛化能力和抵御外部噪声的能力，从而支持了“掩码是改善复杂条件下模型鲁棒性的有效策略”这一假设。
#### 7.3. 外观生成方法的消融实验
为了评估我们的方法对第一阶段轨迹的敏感性，我们用D-Grasp替换了GraspXL。如表7所示，GraspXL的性能更优，这表明最终的生成质量依赖于第一阶段姿态序列的质量。
#### 7.4. 手部表示方式的消融实验
我们对比了手部关键点与2D手部网格投影作为条件时的效果。表8显示，关键点在大多数指标上表现更好，尤其是在MPJPE指标上。这可能是因为网格投影更容易出现自遮挡现象。
#### 7.5. 手部编码器的消融实验
为了验证基于VAE的条件编码器，我们在1000张采样的关键点图像上对其进行了评估，得到的PSNR值为
Table 7. Ablation on Stage-I Method
| Method (Stage-I) | FVD ( $\downarrow$) | MF ( $\uparrow$) | MPJPE ( $\downarrow$) |
| D-Grasp | 58.17 | 0.599 | 36.18 |
| GraspXL (Ours) | 49.98 | 0.645 | 30.96 |
Table 8. Ablation on Hand Representation
| Hand Representation | FVD ( $\downarrow$) | PSNR ( $\uparrow$) | MPJPE ( $\downarrow$) |
| Mesh Projection | 29.33 | 30.17 | 36.18 |
| Keypoints (Ours) | 29.13 | 30.05 | 30.96 |
Table 9. Ablation on Hand Encoder
| Encoder Type | FVD ( $\downarrow$) | PSNR ( $\uparrow$) | MPJPE ( $\downarrow$) |
| MLP | 31.59 | 30.07 | 21.96 |
| VAE (Ours) | 29.13 | 30.17 | 19.37 |
---
40.58，说明该编码器对条件图像的重建误差较低。如表9所示，VAE在所有指标上的表现都优于基于MLP的2D坐标编码器。这种优异的性能主要归功于VAE在保留局部空间信息方面的更强能力。
#### 7.6. I2V骨干网络的消融实验
Table 10. Ablation on Backbone
| Encoder Type | FVD ( $\downarrow$) | PSNR ( $\uparrow$) | MPJPE ( $\downarrow$) |
| InterDyn (SVD w/ single cond) | 38.83 | 24.86 | 28.15 |
| SVD w/ multi conds | 34.91 | 25.84 | 25.11 |
| Ours (CogVideo w/ multi conds) | 29.13 | 30.17 | 19.37 |
为了分离出我们的多模态条件机制的影响，我们在表10中展示了对照消融实验结果。值得注意的是，即使采用相对较弱的骨干网络，多条件约束的SVD [^6]基线在所有指标上的表现也始终优于单条件约束的InterDyn。这一对比表明，整合多种条件能够显著提升整体生成性能。
#### 7.7. 误差传播现象的研究
图9中的实验表明，第一阶段的几何误差（如物体相互穿透或接触缺失）可能会传播下去，即便生成的视频看起来逼真，也会导致物理上不合理的交互。此外，我们发现第三阶段的质量严重依赖第二阶段提供的外观引导：质量较差的初始参考帧会降低最终纹理质量，并加剧时间上的闪烁现象。
#### 7.8. 额外的定性结果
我们在图10和图11中针对DexYCB数据集提供了更优质的结果，在图12和图13中针对OANINK2数据集也展示了类似结果。
---
Figure 10. More qualitative results on DexYCB dataset (a).
---
Figure 11. More qualitative results on DexYCB dataset (b).
---
Figure 12. More qualitative results on OAKINK2 dataset (a).
---
Figure 13. More qualitative results on OAKINK2 dataset (b).