# 大型基础模型时代的手物交互：重建、生成与具身迁移林伟权 $^{1,2}$、邓宇 $^{3}$、刘世阳 $^{2}$、肖鲁平 $^{2}$、唐旭 $^{1}$、虞俊志 $^{4}$、杨娇龙 $^{3}$、张雷 $^{5}$、陈星宇 $^{2}$ $^{1}$西安电子科技大学 $^{2}$中关村研究院 $^{3}$微软亚洲研究院 $^{4}$北京大学 $^{5}$Visincept ![Image](images/cce01113a61e6a788d726755d25ac6b9e852bb6636fc18d632b01295535a86d5.jpg)

Figure 1: Overview of this survey. The center organizes six HOI tasks into reconstruction (pose estimation, object reconstruction, and dynamic reconstruction) and generation (grasp, motion, and image/video generation). These tasks are supported by eight foundation-model sub-priors grouped into three families. Geometric priors cover shape retrieval, shape reconstruction, and spatial reconstruction. Semantic priors cover semantic grounding and language reasoning. Visual priors cover visual representation, image generation, and video generation. The resulting HOI knowledge further enables embodied transfer through human-data pretraining, human-to-robot skill transfer, and HOI-to-robot data generation.

--- ## 目录摘要 ..... 3 1 引言 ..... 3 2 手物交互与基础模型先验的预备知识 ..... 4 2.1 手物交互的任务分类 ..... 4 2.2 手物交互的表征方式 ..... 5 2.3 非基础模型先验的手物交互重建方法 ..... 6 2.4 非基础模型先验的手物交互生成方法 ..... 6 2.5 常见的手物交互流程抽象 ..... 7 2.6 手物交互任务中采用的先验家族 ..... 8 3 手物交互的几何先验 ..... 8 3.1 几何先验的范围 ..... 8 3.2 形状检索先验 ..... 10 3.3 形状重建先验 ..... 10 3.4 空间重建先验 ..... 11 4 手物交互的语义先验 ..... 12 4.1 语义先验的范围 ..... 12 4.2 语义定位先验 ..... 12 4.3 语言推理先验 ..... 13 5 手物交互的视觉先验 ..... 14 5.1 视觉先验的范围 ..... 14 5.2 视觉表征先验 ..... 14 5.3 图像生成先验 ..... 15 5.4 视频生成先验 ..... 15 6 基于手物交互的具身迁移 ..... 16 6.1 具身迁移的范围 ..... 16 6.2 人类数据预训练 ..... 17 6.3 从人类到机器人的技能迁移 ..... 17 6.4 手物交互到机器人的数据引擎 ..... 18 7 数据集与评估协议 ..... 19 7.1 数据集与预训练来源 ..... 19 7.2 指标家族及其盲点 ..... 19 8 开放挑战与未来方向 ..... 21 8.1 迈向集成化且可验证的手物交互基础系统 ..... 21 8.2 从几何正确性到交互正确性 ..... 21 8.3 长时序、动态相机及世界空间下的手物交互 ..... 21 8.4 先验的可靠性、路由与冲突解决 ..... 21 8.5 从手物交互分析到动态具身记忆 ..... 21 8.6 以机器人为中心的人类手物交互理解 ..... 21 9 结论 ..... 21 --- ## 摘要手-物体交互（HOI）建模仍然颇具挑战性，因为它需要对手部姿态、物体几何形状、接触关系、语义信息以及动力学进行联合推理，且这一过程需在严重的视觉不确定性条件下完成。基础模型能够利用从大规模跨领域数据中学习到的可迁移先验知识，为解决这些难题提供了新途径，而无需依赖特定任务的数据和模型。然而，相关文献数量虽迅速增长却较为零散；现有研究往往只是简单地将这类方法描述为“使用大模型”，却并未系统阐明引入了何种知识、这些知识在HOI流程中的具体作用位置，以及它们有助于降低哪类HOI不确定性。本综述首次对用于HOI的基础模型先验进行了系统性梳理。我们将相关文献划分为涵盖重建与生成在内的六类HOI任务。更重要的是，我们提出了由八种基础模型子先验构成的分类体系，这些子先验可分为几何、语义和视觉三大类。几何先验包括形状检索、形状重建和空间重建；语义先验涵盖语义定位与语言推理；视觉先验则涉及视觉表征、图像生成及视频生成。基于这一分类体系，我们系统分析了不同先验在HOI流程及各类任务中的表达方式、注入机制与适配方式。除了探讨基础模型如何赋能HOI外，我们还进一步研究了如何将HOI衍生的知识应用于机器人学习领域，包括人类数据预训练、从人类到机器人的技能迁移，以及HOI到机器人的数据生成。最后，我们总结了相关数据集与评估协议，并讨论了当前研究的局限性及迈向更具通用性的HOI系统的未来方向。为推动该领域的长期发展，我们维护了一个实时更新的资源库，用以持续汇总新兴方法与基准测试。关键词：手-物体交互、HOI重建、HOI生成、基础模型先验、具身智能 ## 1 引言手-物体交互（HOI）对于理解人类如何操控物理世界至关重要，它也是计算机视觉、增强/虚拟现实、数字人以及具身人工智能领域的重要基础 [^1] [^2] [^3] [^4]。HOI重建与生成需同时建模手部姿态、物体几何形状、接触关系、交互意图以及时间演变过程。由于严重的相互遮挡现象，加之姿态、形状、深度、运动、语义及物理约束之间的耦合作用，这些变量往往只能被部分观测到。因此，合理的HOI结果应当在视觉、几何、语义、时间和物理层面均保持一致 [^5] [^6] [^7] [^8]。这些挑战表现为以下五种相互关联的不确定性。当手部遮挡物体区域时，就会产生形状不确定性，即便在多个视角下也无法确定完整的几何结构 [^5] [^9]。空间不确定性源于单目投影中姿态、尺度、深度和视角的相互交织，导致同一图像可能存在多种三维解释 [^1] [^2]。物理不确定性涉及接触关系的模糊性以及物理有效性问题，也就是说仅凭视觉上的邻近性无法判定接触情况、作用力或稳定性 [^6] [^10] [^11]。语义不确定性涵盖开放世界中的物体身份、功能部件、可供性和任务意图 [^7] [^12] [^13]。动态不确定性则出现在相机、手部和物体的运动在快速视角变化和运动模糊作用下相互纠缠时 [^8] [^4]。由于这些不确定性是相互关联的，单独减少其中一种并不能保证得到在几何、接触、语义和运动方面都一致的人体-物体交互结果；因此，可靠的人体-物体交互重建与生成需要能够约束多个变量及其相互作用的补充证据。基础模型提供了这种超越特定任务监督的补充性跨领域证据。它们并非取代专门针对人体-物体交互的模型，而是提供可在人体-物体交互流程的不同阶段引入的几何、语义和视觉知识。几何先验有助于资产检索、形状恢复以及考虑相机因素的空间重建；语义先验可提供开放词汇的语义定位与语言推理能力；而视觉先验则能传递可复用的表征，或生成外观与运动分布 [^14] [^15] [^7] [^16] [^17] [^18] [^19]。这种视角将核心问题从“基础模型是否有助于人体-物体交互”转变为“引入了何种知识、它如何融入人体-物体交互流程，以及它能帮助减少哪些相互关联的不确定性”。我们设定了一个刻意狭窄的操作边界。只有当一个明确识别出的大规模通用预训练模型通过预测、表征、参数迁移、适配或蒸馏等方式贡献跨领域知识时，该方法才被归类为“基础模型先验”方法。特定任务的预训练以及人体-物体交互领域的资源（如三维资产、接触图或隐式场）本身并不符合这一定义；只有当基础模型为目标任务检索、生成、对齐、过滤或评分这些资源时，它们才被视为注入的基础模型先验 [^20] [^21] [^22]。所有其他方法均被归为“非基础模型先验”方法，但这并不意味着它们缺乏预训练或领域知识。如图1所示，我们的分类体系包含两个维度。任务维度包括三项重建任务（即手-物体姿态估计、手持物体重建以及动态人体-物体交互重建）和三项生成任务（即抓取合成、人体-物体交互运动生成以及人体-物体交互图像/视频生成）。先验维度则包含八个子先验，它们被归类为几何类（即形状检索、形状重建和空间重建）。 --- 语义类（即语义基础与语言推理）和视觉类（即视觉表示、图像生成及视频生成）。我们进一步分析了这些先验如何通过初始化、对齐、融合、条件化、适配或分数引导等方式融入HOI系统。除了重建与生成外，我们还探讨了HOI视频、重建后的状态、抓取动作、轨迹、可供性以及生成的计划如何通过人类数据预训练、从人类到机器人的技能迁移以及HOI到机器人的数据引擎，被转移到机器人学习中 [^23] [^24] [^25] [^26]。现有的综述涵盖了手部姿态估计、手-物体姿态估计、交互手部或灵巧抓取等内容 [^27] [^28] [^29] [^30]，但并未像表1所示那样对HOI重建、生成、接触与可供性推理、基础模型先验以及具身迁移进行联合分析。随着近期各类流程开始整合异构的基础模型先验，这一差距愈发显著 [^31] [^32]。据我们所知，这是首篇从基础模型先验注入的角度出发，对HOI技术及其具身迁移进行的系统性且具批判性的综述。本综述的结构如图2所示。主要贡献如下： • 具身迁移（第6节）。我们探讨了HOI衍生的知识如何支持人类数据预训练、从人类到机器人的技能迁移以及机器人数据生成。 • 任务与知识边界（第2节）。我们构建了六类任务的HOI分类体系，并区分了跨领域基础模型先验与特定任务预训练及HOI领域知识。 - 先验分类与注入机制（第3–5节）。我们对几何、语义及视觉先验进行了归类，并比较了它们的来源、表示方式、注入机制、支持的任务、缓解的不确定性以及局限性。数据集与评估（第7节）。我们汇总了任务基准、大规模预训练数据源及评估指标，同时指出了当前协议中的盲点。 • 开放挑战（第8节）。我们明确了实现通用化、交互感知且物理基础扎实的HOI及具身操控的研究方向。 ## 2 手-物体交互与基础模型先验概述本节定义了本综述中使用的任务与表示规范，回顾了未利用基础模型跨领域知识的方法，总结了HOI任务中所采用的基础模型先验类别，并针对六类HOI任务提出了通用的流程抽象。这些非基础模型先验方法可能会使用领域内先验，例如手部模型（MANO [^33]、NIMBLE [^34]）、物体模板 [^1] [^35]、隐式场 [^36] [^37]、接触统计 [^10]、时间平滑性 [^8] [^4]、场景估计（SLAM [^38]、SfM [^39]）、物理约束或模拟器 [^40] [^11]，以及HOI生成模型 [^9] [^41]。其中许多组件也会出现在基础模型先验方法中，作为表示或细化约束。二者的区别在于是否有大规模预训练模型提供额外的跨领域知识。 ### 2.1 手-物体交互的任务分类本小节定义了六种HOI任务以及本综述中使用的术语。接触与可操作性是作为输出属性而非独立的任务类别来处理的。为在表4和表5中保持简洁，R1–R3和G1–G3分别表示如下定义的重建任务与生成任务。R1表示手-物体姿态估计；R2表示手持物体重建；R3表示动态HOI重建。G1表示手-物体抓取合成；G2表示HOI运动生成；G3表示HOI图像/视频生成。ET表示第6节中讨论的下游具身转移场景。 #### 2.1.1 HOI重建 HOI重建旨在从视觉观测中恢复手和/或物体的空间状态与结构。我们根据输出范围和时间跨度来区分三类任务：手-物体姿态估计（R1）用于估算手与被操纵物体的关节空间状态，包括手部关节活动与物体姿态。根据可用的观测数据与标注信息，输出形式涵盖手部关键点、MANO参数、手部网格，以及物体的6D姿态和手-物体关节姿态。之所以包含仅涉及手部的研究方法，是因为它们常作为HOI流程的前端模块；在此流程中，物体拟合、接触估计与时间优化均依赖于可靠的手部状态估计 [^42] [^1]。手持物体重建（R2）旨在恢复手中所持物体的完整3D几何形状，可采用点云或网格等显式表示方法，也可采用SDF、DDF或占据场等隐式表示方法。其核心挑战在于推断被手部遮挡或无法观测到的物体区域，尤其是针对没有已知CAD模板的未知物体或开放世界物体 [^5] [^36] [^43]。动态HOI重建（R3）则从视频中恢复随时间变化的手、物体、相机、接触及交互状态。可渲染的4D HOI重建 [^4] [^44] [^45] 被视为此类重建任务中的不同表示形式。 #### 2.1.2 HOI生成 HOI生成是在指定条件下创建手-物体状态、运动或视觉内容。我们根据输出模态定义了三类生成任务，而将编辑、修复、物体替换与重演视为条件生成而非独立任务。 --- ![Image](images/d47d6df3873a911742ded4c874f8950f9def8d54f120122fd558c76675214674.jpg)

Figure 2: Taxonomy roadmap of this survey. Three foundation-model prior families are decomposed into section-level sub-priors and linked to the HOI reconstruction or generation tasks reviewed under each subsection.

Table 1: Comparison of existing HOI-related surveys. ✓ = systematic coverage; × = no systematic coverage.

Existing survey	Hand-only Hand-object pose	HOI		Contact and affordance	Foundation model	Embodied priors	transfer
Efficient annotation and learning for 3D hand pose estimation: A survey [^27]	✓	✗	✗	✗	✗	✗	✗
A Survey of Deep Learning Methods and Datasets for Hand Pose Estimation from Hand-Object Interaction Images [^28]	✓	✗	✗	✗	✗	✗	✗
Advances in Vision-Based Deep Learning Methods for Interacting Hands Reconstruction: A Survey [^29]	✓	✗	✗	✗	✗	✗	✗
An Overview of Learning-Based Dexterous Grasping: Recent Advances and Future Directions [^30]	✗	✗	✓	✓	✗	✓	✓
Ours	✓	✓	✓	✓	✓	✓	✓

手-物体抓取合成（G1）根据给定的物体、语言、接触或功能目标，生成合理的静态手部抓取姿态、手部网格或接触配置 [^46] [^9] [^10]。HOI运动生成（G2）则生成包含随时间变化的手部运动、物体运动、交互阶段及状态转换的操作序列 [^11] [^47]。HOI图像/视频生成（G3）用于合成包含手-物体交互的图像或视频，或在文本、图像、掩码、姿态、深度或接触条件下来执行条件编辑、修复或重演 [^48] [^49]。几何表示用于描述手、物体及场景的空间状态与形状。2D/3D关键点和骨架等稀疏表示在姿态描述方面效率较高。 #### 2.2.1 几何表示 ### 2.2 手-物体交互的表示方法估计与跟踪。像MANO [^33]和NIMBLE [^34]这样的参数化手部模型，能够将手部的关节活动和形状编码到低维空间中，因此被广泛用于手部网格回归和优化。网格和点云等显式表示是重建的常见输出形式。有符号距离场（SDF）、占据场以及有向距离场（DDF）[^50] [^43]等隐式表示则用于建模连续表面。例如，HALO [^51]就能将三维手部骨骼映射为可微分的关节手部占据场。NeRF [^52]和3D高斯泼溅 [^53]这类可渲染的表示方法，还能进一步支持新视角渲染和4D HOI重建。 #### 2.2.2 交互表示交互表征描述了手、物体与任务之间的关系。接触图则将接触标签或概率分配给图像像素、点云样本或网格顶点。 --- 接触图将接触标签或概率分配给图像像素、点云样本或网格顶点 [^3] [^54]，而可供性图则用于识别适合特定操作的物体区域 [^55]。基于场的表征在三维空间中连续定义这些关系：接触场可在任意查询位置建模接触或邻近情况，而交互场则编码手与物体之间更广泛的空间关系 [^9] [^56]。密集对应关系指定了点对级别的手-物关联 [^57]，而物体状态转换则描述了物体在操作过程中的运动或形变方式 [^58] [^59]。 ### 2.3 非基础先验的HOI重建方法 #### 2.3.1 手部-物体姿态估计手部-物体姿态估计包括仅手部的估计以及手部-物体联合状态的恢复。HandOccNet [^60]、MobRecon [^61]、HaMeR [^62]、WiLoR [^63]、simpleHand [^64]以及S $^{2}$HAND [^65]等仅手部的方法，能从单张RGB图像中估计手部关键点、MANO参数或网格；而KeypointFusion [^66]还会融合RGB和深度信息来进行三维手部姿态估计。H+O [^1]、Hasson等人 [^2]、HOPE-Net [^67]、Keypoint Transformer [^68]、THOR-Net [^69]以及HOISDF [^70]等联合方法，会结合MANO模型、已知的物体几何信息、关键点推理、SDF约束、可微分渲染或互斥性约束来进行估计。CPF [^3]、S $^{2}$Contact [^54]、ContactArt [^71]以及CHOIR [^56]等考虑接触信息的变体，还会额外预测接触场或接触图，但最终仍是为了估计或优化手部-物体状态。这些方法的局限性并不在于缺乏结构，而在于其估计结果仍受任务监督、类似MANO的关节模型、有限物体覆盖范围、薄弱的接触标签以及数据集特定的接触统计信息的限制。在单目遮挡和开放式操作目标下，姿态模糊、基于邻近度的接触误差、物体几何信息缺失以及功能意图不明确仍是常见的失败模式。对于手-物联合估计而言，形状检索与形状重建先验用于解决未知物体的几何问题，空间重建先验用于消除3D姿态与摄像机的模糊性，而语义定位与语言推理先验则用于处理观测与意图方面的不确定性。这些先验类别将在第3.2–3.4节、第4.2节和第4.3节中详细阐述。 #### 2.3.2 手持物体重建手持物体重建旨在在严重的手部遮挡情况下恢复被操纵物体的3D形状。代表性方法包括IHOI [^5]、AlignSDF [^36]、gSDF [^72]、DDF-HO [^43]、CHORD [^73]、HandNeRF [^37]、HHOR [^74]、HOLD [^4]、MOHO [^75]以及TexHOI [^76]。HandNeRF能够从单张RGB图像中重建手-物场景，其余方法则适用于单帧或多帧场景。这些方法通常用SDF、占用场或DDF场来表示物体几何形状；它们通过轮廓一致性、可微渲染或多帧观测来优化几何结构，再通过互斥约束与接触约束来确保手-物之间的兼容性。其中一些方法还会从HOI数据中学习类别级的形状先验。这些工具虽然能够实现重建，但无法应对开放式场景。持续的遮挡可能导致对交互至关重要的物体区域无法被观测到，类别级先验也无法覆盖开放世界中的任意物体，而且更优的表面度量指标并不能保证接触的一致性或物理合理性。这些局限性正是第3.2节、第3.3节和第4.2节中所阐述的形状检索、形状重建及语义定位先验的研究动机。 #### 2.3.3 动态HOI重建动态HOI重建将重建范围扩展到视频领域，此时手部运动、物体运动、接触演变及摄像机运动相互关联。代表性方法包括仅针对手部的时序重建方法，如SeqHAND [^77]和Dyn-HaMR [^78]；手-物联合姿态估计方法包括HOMAN [^8]、InteractionFusion [^79]、HOLD [^4]，以及Tian等人提出的交互感知型4D高斯泼溅技术 [^80]；可渲染或具备物理特性的重建方法则包括NCRF [^44]和DeepSimHO [^40]。这些方法通常结合了时序正则化、几何估计与物理约束，包括运动平滑性、MANO一致性、刚体约束、SLAM [^38]/SfM [^39]摄像机估计、多帧光度优化、接触连续性、稳定抓取假设以及仿真模拟。然而，以自我为中心的摄像机运动、模糊现象及长时间段的漂移会削弱这些约束的效果；即便可渲染的轨迹也仍可能出现穿透、脱离接触或产生不符合物理规律的物体响应的情况。这些局限性正是第3.4节和第5.4节中所阐述的空间重建与视频生成先验，以及第7.2.3节中讨论的接触与物理评估的研究动机。 ### 2.4 非基础先验类的HOI生成方法 #### 2.4.1 手-物体抓取合成手-物体抓取合成用于生成物体的静态手部姿态或接触配置。代表性方法包括GanHand [^46]、GrabNet [^81]、GraspTTA [^82]、ContactOpt [^10]、ContactGen [^83]、Contact2Grasp [^84]、G-HOP [^85]、ClickDiff [^86]以及FastGrasp [^87]。G-HOP学习了手部和物体场上的联合空间扩散先验，并将其应用于交互重建和静态抓取合成。不同的抓取合成方法会使用不同的HOI特定组件，包括基于MANO模型的手部姿态的条件分布、 --- 以及手部形状参数、接触图、隐式交互场、穿透惩罚项、局部几何约束和受力闭合启发的目标函数。其主要局限性在于功能模糊性：看似稳定的抓取仍可能无法明确物体应被握住、按压、转动、传递还是打开。接触覆盖度和穿透惩罚项也仍只是执行稳定性的不完美替代指标，这也促使我们在第4.3节中探讨语言推理先验，并在第6节中讨论具身验证。 #### 2.4.2 HOI运动生成 HOI运动生成根据动作捕捉数据、GRAB/OakInk风格数据集或特定任务演示来生成动态操作序列。D-Grasp [^11]、ArtiGrasp [^47]、GEARS [^93]、LatentHOI [^41]、Gaze-guided HOI Synthesis [^94]、How Do I Do That? [^95]以及BimArt [^89]从HOI数据、交互码本、局部几何特征、视线/接触约束、强化学习或物理仿真中学习运动或轨迹分布，而非依赖外部运动或视频基础模型。GEARS则根据手部和物体的轨迹来合成手部运动序列。 #### 2.4.3 HOI图像/视频生成这些方法提供了重要的对比参考，但它们所学到的分布仍受限于可用HOI运动数据的规模和多样性。长时程物体转移、可执行的物理接触，以及针对新物体或意图的任务条件泛化仍然难以实现，这也促使我们在第4节和第5节中探讨语言推理和视频生成先验。在基础模型规模的生成模型出现之前，手-物体图像和视频生成相关研究相对较少。HOGAN [^48]可直接生成手-物体图像，而HOGSA [^92]则利用基于网格的3D高斯泼溅技术来渲染姿势和视角多样的双手法HOI图像，用于数据增强。成熟的任务特定视频生成技术则十分匮乏：早期的GAN基模型难以保留手部拓扑结构、物体身份、接触关系、时间一致性以及开放词汇可控性。这一缺口直接推动了第5节中图像和视频生成先验的研究。 ### 2.5 常见的HOI流程抽象这六类HOI任务共享一个功能流程：输入数据经过HOI骨干网络处理后，由特定任务的头模块将其表征映射为所需输出。对于生成任务而言，提示词、初始状态及其他条件均属于输入。HOI骨干网络负责主要的特征提取、几何推理、时序建模、优化或去噪过程；任务头则负责生成特定于输出的结果。 ![Image](images/c53dfd0c9e98b53a29dcd37f022a21e77a72e3b8a48840e26f098cfb19fd9007.jpg)

Figure 3: Residual HOI uncertainties and corresponding foundation-prior interventions.

回归器、解码器、评分器或渲染器。表2总结了这六项任务中的具体实现方式。 “主干网络”和“任务头”指的是功能角色，并不一定对应独立的神经模块。基于优化的重建方法、隐式场方法以及扩散模型可以通过迭代优化、神经场、去噪器或渲染解码器来实现这些角色。基础模型先验在保留HOI任务定义的同时，还能对主干网络或任务头进行初始化、增强、条件化或实例化。它们可以以观测数据、预训练参数、特征、约束条件、训练数据或推理与解码模块的形式参与进来。即便采用了传统的HOI组件，在严重遮挡、开放世界语义以及复杂动态场景下仍会存在五种残余不确定性。第3至5节中介绍的三类基础模型先验能够通过针对性干预来缓解这些不确定性，如图3所总结的那样。模板和隐式几何表示会导致被遮挡或不可见物体的表面信息模糊不清，而二维证据和相机估计又无法充分约束深度、尺度、相机运动以及世界空间对齐关系。几何先验能够缓解由此产生的形状与空间不确定性。视觉先验在这些情况下能提供额外依据：生成式去噪引导可以规范观测较弱的物体几何结构，而通用的视觉表征则能提升手-物体姿态及空间状态的估计精度。接触与物理约束对不准确的几何信息以及错误推断的功能性接触十分敏感，因此几何与语义先验共同作用，能为减少物理不确定性提供更强的结构与交互依据。封闭集监督依赖于预定义的对象类别与可供性标签，这限制了其对未见过的物体、功能部件以及交互意图的泛化能力。 ---

Table 2: Functional abstraction of typical HOI backbones and task heads across the six tasks.

Task	Typical HOI backbone	Typical task head	Canonical output
R1: Hand-Object Pose Estimation	CNN encoder [^1]; ViT encoder [^62]; geometry-aware transformer [^68]	Keypoint/object-pose regressors [^1]; MANO/mesh regressor [^2] [^62]	Hand pose; object 6D pose
R2: Hand-Held Object Reconstruction	Image encoder or neural renderer [^37]; multi-frame reconstruction optimization [^4]; implicit-field model [^36] [^43]	SDF/DDF/occupancy decoder [^36] [^43] [^73]; mesh/radiance decoder [^74] [^37]	Held-object geometry
R3: Dynamic HOI Reconstruction	Temporal tracking [^8]; 4D optimization [^4]; 4D neural renderer [^44] [^80]	Trajectory/camera/object-state head [^8] [^4]; deformation/rendering head [^44] [^80]	Dynamic hand-object state
G1: Hand-Object Grasp Synthesis	Object encoder/conditional VAE [^81]; diffusion model [^87] [^85]	Grasp/MANO head [^81]; contact/score head [^82] [^10]	Static grasp pose or contact map
G2: HOI Motion Generation	Temporal transformer [^88] [^89]; diffusion motion model [^11] [^41]	Hand-object trajectory/sequence decoder [^11] [^88] [^89]	HOI motion sequence
G3: HOI Image/Video Generation	Image/video U-Net [^49] [^90]; DiT/video transformer [^91]	Image/video decoder [^49] [^91]; rendering head [^92]	HOI image/video

语义先验通过开放词汇接地与语言推理来弥补这一局限。最后，针对移动相机下的长时序交互场景，特定于任务的时间模型与局部一致性约束——包括运动平滑性、时序姿态一致性、跨帧外观一致性以及时序接触一致性——仍然显得不足。几何先验能够稳定以相机为参照的世界空间结构，而视觉先验则能提供更广泛的时间规律性以及交互展开模型，从而缓解动态不确定性。 ### 2.6 HOI任务中采用先验家族我们总结了本次调研中涉及的三类基础模型先验及其八种子先验。 #### 2.6.1 几何先验几何先验从预训练的几何模型中提取三维形状与空间结构。几何形状检索（G-Ret）会从形状数据库中查找候选物体资源，以确定物体的几何形态 [^15]。几何形状重建（G-Rec）则能从图像或文本中生成完整的物体形状 [^98] [^102]。几何空间重建（G-Spa）可提供度量深度、点云图以及相机参数 [^14] [^97]。 #### 2.6.2 语义先验语义先验传递视觉与语言理解能力。语义接地（S-Gnd）能将文本或多模态提示转化为掩码、框以及轨迹等局部化区域 [^7] [^12]。语言推理（S-Lng）则利用预训练的语言模型来推断意图、功能部件以及可供性 [^139] [^125]。 #### 2.6.3 视觉先验视觉先验能够跨领域传递视觉知识。视觉表征（V-Rep）提供通用的图像特征或标记[^17]。图像生成（V-Img）则利用预训练的扩散模型骨干来处理外观相关任务。 ![Image](images/745b1e433bfec079ce69f66af74bd76ad1e471d26dad7546cd95b8311f273ccf.jpg)

Figure 4: Injection mechanisms of geometric priors for HOI.

布局[^18] [^138]。视频生成（V-Vid）还会对时间和交互动态进行建模[^19] [^133]。图3总结了残余不确定性及相应的先验干预措施，而表3则列出了采用第3至5节所构建分类法的代表性基础模型先验方法。 ## 3 用于HOI的几何先验 ### 3.1 几何先验的范围本节将综述源自基础模型的3D知识，这类知识能够降低HOI中的形状与空间不确定性。文中区分了三种子先验：形状检索先验通过基础模型嵌入将视觉观测结果与外部3D资产库中的资产进行匹配，从而选出拓扑结构稳定的形状候选；形状重建先验借助单图像转3D或多视图生成模型来提供物体被遮挡或不可见的几何信息，这些模型要么补全部分证据，要么从零开始合成初始形状；空间重建先验则迁移预训练参数，或提供深度、相机参数、 ---

Table 3: Representative foundation-model-prior methods for HOI. The tag before P identifies the sub-prior defined in Sec. 2.2; P and A denote primary and auxiliary priors, respectively. The representation and operator columns describe P. Uncertainty↓ lists the HOI uncertainties mitigated by P: Sh = shape, Sp = spatial, Ph = physical, Se = semantic, and Dy = dynamic.

Method	Status	Task	Foundation-model prior source	Injected representation	Injection operator	Unc. $\downarrow$
Primary family: Geometric priors (Sec. 3)
GeoHand [^96]	ACMMM 2026	R1: Hand-Object Pose Estimation	G-Spa; P: MoGe-2 [^97]	spatial geometry tokens	token fusion	Sp
EasyHOI [^31]	CVPR 2025	R2: Hand-Held Object Reconstruction	G-Rec; P: InstantMesh [^98]; A: LISA [^99], SAM [^12] (S-Gnd), Affordance Diffusion [^100] (V-Img)	reconstructed shape	initialize	Sh
Follow My Hold [^101]	3DV 2026	R2: Hand-Held Object Reconstruction	G-Rec; P: Hunyuan3D [^102]; A: MoGe-2 [^97]	reconstructed shape	regularize	Sh
HOST3R [^103]	ICCV Workshop 2025	R2: Hand-Held Object Reconstruction	G-Spa; P: DUST3R [^14]	pretrained parameters; point maps	weight initialization/fine-tuning	Sp
ArtHOI [^32]	CVPR 2026	R3: Dynamic HOI Reconstruction	G-Spa; P: Video Depth Anything [^104], UniDepth V2 [^105]; A: Hunyuan3D [^102] (G-Rec), SAM 2 [^106], Qwen-VL-Max [^16]	metric depth / camera parameters	scale alignment / registration	Sp+Dy
GHOST [^107]	CVPR 2026	R3: Dynamic HOI Reconstruction	G-Ret; P: OpenShape [^15]; A: SAM 2 [^106], InternVL [^108]	retrieved shape	retrieve / align / regularize	Sh
HaWoR [^109]	CVPR 2025	R3: Dynamic HOI Reconstruction	G-Spa; P: Metric3D [^110]	metric depth	scale alignment	Sp+Dy
GraG [^45]	arXiv 2026	R3: Dynamic HOI Reconstruction	G-Rec; P: SAM 3D [^111], MV-SAM3D [^112]; A: SAM 3 [^113], Depth Anything 3 [^114]	reconstructed shape	initialize / regularize	Sh+Dy
Primary family: Semantic priors (Sec. 4)
HandOS [^115]	CVPR 2025	R1: Hand-Object Pose Estimation	S-Gnd; P: Grounding DINO 1.5 [^7]	region-aligned representation	region conditioning	Se+Sp
CHOIR [^116]	arXiv 2026	R3: Dynamic HOI Reconstruction	S-Gnd; P: SAM 2 [^106], Amodal Video Segments [^117]; A: SAM 3D [^111] (G-Rec), MoGe-2 [^97] (G-Spa)	modal / amodal masks	region conditioning	Se+Dy
AffordGrasp [^118]	CVPR 2026	G1: Hand-Object Grasp Synthesis	S-Lng; P: RoBERTa [^119], Qwen2 [^120]	interaction semantics	condition / fuse	Se
NL2Contact [^121]	ECCV 2024	G1: Hand-Object Grasp Synthesis	S-Lng; P: ChatGPT [^122], BERT [^123]	interaction semantics	condition / fuse	Ph
HOIGPT [^124]	CVPR 2025	G2: HOI Motion Generation	S-Lng; P: LLaMA-13B [^125]	interaction semantics	condition / fuse	Se
OpenHOI [^126]	NeurIPS 2025	G2: HOI Motion Generation	S-Lng; P: ShapeLLM [^127]	interaction semantics	condition / fuse	Se+Ph
Primary family: Visual priors (Sec. 5)
HOPFormer [^57]	ECCV 2026	R1: Hand-Object Pose Estimation	V-Rep; P: DINOv2 [^17]	visual feature tokens	token fusion	Sp
ForeHOI [^128]	CVPR 2026	R2: Hand-Held Object Reconstruction	V-Rep; P: DINOv2 [^17]	visual feature tokens	token fusion	Sh
MagicHOI [^129]	ICCV 2025	R2: Hand-Held Object Reconstruction	V-Img; P: Zero-1-to-3 [^130]	denoising score estimates	score-guided regularization	Sh
BIGS [^131]	CVPR 2025	R3: Dynamic HOI Reconstruction	V-Img; P: Stable Diffusion [^18]	denoising score estimates; generative latent features	score-guided regularization; ControlNet conditioning	Sh+Dy
DreamHand [^132]	arXiv 2026	R3: Dynamic HOI Reconstruction	V-Vid; P: Wan [^133]	generative latent features	weight initialization/fine-tuning	Dy
HUG [^134]	arXiv 2026	G1: Hand-Object Grasp Synthesis	V-Rep; P: DINOv2 [^17]	visual feature tokens	token fusion	Se+Sp
SIGHT [^135]	arXiv 2025	G2: HOI Motion Generation	V-Img; P: Stable Diffusion [^18]; A: CLIP [^21]	denoising score estimates	score-guided guidance	Dy
SViMo [^91]	NeurIPS 2025	G3: HOI Image/Video Generation	V-Vid; P: CogVideoX-5B [^19]; A: Google T5 [^136]	pretrained parameters; generative latent features	weight initialization/fine-tuning; adapter conditioning	Dy
Re-HOLD [^137]	CVPR 2025	G3: HOI Image/Video Generation	V-Img; P: Stable Diffusion [^18]; A: LISA [^99]	pretrained parameters; generative latent features	weight initialization/fine-tuning; ControlNet conditioning	Dy
Affordance Diffusion [^100]	CVPR 2023	G3: HOI Image/Video Generation	V-Img; P: GLIDE [^138], Latent Diffusion [^18]	pretrained parameters; generative latent features	weight initialization/fine-tuning	Se+Sh

--- 点云图、跨视图对应关系以及来自预训练空间重建模型的空间特征标记。图4总结了相应的注入机制。检索到的形状通过检索、对齐和几何正则化来加以利用，而重建后的形状则用于初始化或正则化HOI解决方案。空间重建先验既可以初始化并微调模型参数，也能为尺度对齐、配准和标记融合提供空间几何依据。 ### 3.2 形状检索先验形状检索先验通过基础模型嵌入将视觉或语言证据与资产表示进行匹配，从而从外部3D资产库中选取候选物体形状。在本文调研的方法中，InternVL[^108]负责识别被操纵的物体，而OpenShape[^15]则从Objaverse[^20]中检索其3D假设模型。 #### 3.2.1 基于形状检索的动态HOI重建 #### 3.2.2 分析与讨论作为HOI的主要基础模型先验，形状检索方面的研究仍较为有限。典型的例子是GHOST[^107]，在该方法中，InternVL[^108]负责识别被操纵的物体，OpenShape[^15]则从Objaverse[^20]中检索对应的资产。检索到的几何信息会与视频证据对齐，进而用于初始化和正则化后续的高斯重建过程。在基于基础模型的手物交互（HOI）任务中，这一路径尚未形成广泛的方法集群。它要求库中存在合适的资产、能够在手部遮挡条件下可靠地识别物体，以及能在多帧间实现稳健的几何对齐。对于特定实例、可活动、可变形或此前未见过的物体而言，这些要求很难满足。因此，检索往往仅作为辅助初始化或备用假设，而近期的研究系统更倾向于直接从观测数据重建或生成物体几何结构。当存在合适资产时，检索能提供一个拓扑稳定的物体假设，这种假设比生成的形状更易于对齐和验证，不过其在几何、尺度或活动方式上仍可能与目标物体存在差异。因此，基于检索的HOI重建应将检索失败情况与姿态拟合或接触优化错误区分开来处理，并将检索到的资产视为带置信度加权的提议，而非绝对可靠的锚点。 ### 3.3 形状重建先验形状重建先验利用通用的3D重建与生成模型，从部分观测数据中恢复完整的物体几何结构，或为后续对齐合成初始形状。本文综述的方法包括InstantMesh [^98]、Luma AI Genie [^140]、Hunyuan3D [^102]、SAM 3D [^111]以及MV-SAM3D [^112]。它们重建出的形状为遮挡或不可见区域的几何结构提供了假设，但并未明确真实的手物接触状态，因此需要对照HOI观测结果进行验证。 #### 3.3.1 通过形状重建实现手持物体重建形状重建先验的主要应用场景是手持物体重建，此时手部会遮挡物体表面的大部分区域。形状重建方法利用基于基础模型生成的几何结构来初始化或约束HOI主干网络，帮助其推断出视觉观测中缺失的物体表面信息。 EasyHOI [^31]利用InstantMesh [^98]重建由其上游定位与修复模块生成的完整物体网格。该生成的网格用于初始化后续的手物对齐与物理优化过程。MCC-HO [^22]将检索增强重建（RAR）作为内部组件。RAR借助GPT-4V [^141]对物体的描述来引导Luma AI Genie [^140]生成3D物体模型。生成的网格用于初始化物体重建，随后在多帧间进行对齐。由于物体假设是生成而非从外部资产库中选择，因此MCC-HO主要被归类为形状重建范畴。Jiang等人 [^142]利用ChatGPT [^122]描述所握物体，再借助Luma AI Genie [^140]生成多个带纹理的网格假设。OpenShape [^15]会选择与视频内容最一致的生成候选结果，再利用DINOv2 [^17]的特征将其渲染视图与各帧对齐，从而在联合姿态与隐式形状优化前初始化物体姿态。Follow My Hold [^101]在推理阶段会对Hunyuan3D [^102]的潜空间形状样本进行引导。HaMeR [^62]提供手部网格，MoGe-2 [^97]则提供部分点云地图与相机估计结果。在这些线索对齐后，法向、视差、轮廓、关键点、交集及邻近性损失会共同引导扩散速度场以及手物变换参数。 #### 3.3.2 基于形状重建的动态HOI重建形状重建先验同样可用于视频和4D HOI重建，它们能够初始化物体几何结构，随后随时间对其进行跟踪。 AGILE [^143]利用Hunyuan3D [^102]从选定的多视角观测数据中重建物体网格，以此初始化其时序优化所使用的物体表示。CHOIR [^116]在接触感知型4D优化之前，会先使用SAM 3D [^111]重建具有度量尺度的标准物体网格以及初始的6D位姿。Grasp in Gaussians (GraG) [^45]则利用MV-SAM3D [^112]从选定的关键帧中重建标准物体几何结构。 --- 该方法对SAM 3D [^111]进行调整，使其能够估计每帧的物体位姿和尺度，同时保持标准形状的固定性；随后将密集的高斯模型转换为轻量级的“高斯之和”表示形式，以实现高效跟踪。在这些流程中，由基础模型生成的物体形状会初始化后续位姿估计和跟踪所使用的几何结构。 #### 3.3.3 分析与讨论当手部遮挡隐藏了交互关键的物体几何结构时，形状重建先验的作用最为显著。它们拓展了超出类别特定HOI数据的假设空间，但重建出的表面可能会生成与手-物体位姿、关节运动或物理可行性相冲突的接触区域。因此，这些先验最适合用于生成候选结果或作为初始化依据，随后再进行位姿拟合、接触检测与时序验证，以及物理合理性过滤。评估时应结合倒角距离和F分数，同时考量接触正确性、穿透情况以及后续交互的稳定性。 ### 3.4 空间重建先验空间重建先验利用预训练的空间重建模型，从输入的图像或视频中恢复3D结构和相机几何信息。它们能提供深度数据、相机参数、点云图、跨视图对应关系以及相机轨迹，从而支持在共享的世界坐标系中对手-物体状态进行配准。本文综述的方法包括DUSt3R [^14]、VGGT [^144]、CUT3R [^145]、Metric3D [^110]、MoGe-2 [^97]、Depth Anything 3 [^114]、Video Depth Anything [^104]以及UniDepth V2 [^105]。 #### 3.4.1 基于空间重建先验的手持物体重建 HOSt3R [^103]利用DUSt3R [^14]的预训练权重来初始化成对点云图估计网络，随后在合成的手-物体数据上进行微调。在估计手-物体3D变换以及从多视角重建物体时，其所估计的点云图可替代传统的SfM和点匹配方法。 #### 3.4.2 基于空间重建的手-物体位姿估计 GeoHand [^96] 对基础单目几何估计器 MoGe-2 [^97] 进行冻结处理，并通过带有门控跨模态令牌融合功能的 GeoAdapter 将其适配于三维手部重建任务。研究表明，通用的空间重建知识能够提升在具有挑战性的单目观测条件下的手部姿态估计精度。HGGT [^146] 以 VGGT [^144] 为基础，能够从未经校准的图像中同时推断手部网格与相机姿态。空间先验虽然无法还原完整的手-物状态，但其估计出的手部几何结构与相机参数能为下游的HOI模块提供空间线索，进而用于物体重建、接触点估计以及运动恢复。 #### 3.4.3 基于空间重建的动态HOI重建在动态相机设置、长时序序列以及世界空间重建场景中，传统的跟踪与SfM方法容易出现漂移或失效问题，此时空间重建先验显得尤为宝贵。 HaWoR [^109] 将自适应自我中心SLAM与基于Metric3D [^110] 的尺度恢复技术相结合，以实现世界坐标系下的动态手部单独重建。EgoGrasp [^147] 利用Depth Anything 3 [^114] 提供的深度数据与相机估计结果，再结合SAM 3D [^111] 生成的物体几何信息，来进行世界空间中的自我中心HOI估计。ArtHOI [^32] 将来自Video Depth Anything [^104] 和UniDepth V2 [^105] 的度量深度与相机线索，以及Hunyuan3D [^102] 生成的物体几何信息，注入到单目4D优化过程中，从而实现对手部与关节物体间交互作用的重建。在GraG [^45] 中，Depth Anything 3 [^114] 负责提供点云地图与相机估计数据，用以确定物体的基准形态及其逐帧运动情况。CHOIR [^116] 则利用MoGe-2 [^97] 生成的度量深度来确定其由SAM 3D推导出的物体初始化的尺度。Hand3R [^148] 将经过冻结处理的HaMeR手部模型与CUT3R [^145] 相结合，借助场景感知的视觉提示来融合手部特征与局部场景令牌，从而实现基于视频的在线度量尺度手部-场景重建。 #### 3.4.4 分析与讨论空间重建先验主要解决的是手部、物体、相机及场景在三维空间中的位置问题。密集的点云地图、深度估计、相机参数以及多视图对应关系能够显著减少单目环境下的尺度-深度歧义、自我中心相机的漂移现象以及动态视图重建误差。其优势并不在于物体语义或交互意图的理解，而在于几何结构的稳定性：它们为HOI专用模块提供了世界空间的框架，便于估计接触点、物体运动及交互状态。然而，当任务要求确定应使用哪个物体部位、判断表面上的邻近关系是否对应真实接触、或是判断重建出的运动是否具有物理可行性时，空间重建先验仍显不足。因此，应将其视为HOI流程中提供相机与几何相关证据的来源，而非完整的HOI推理系统。未来的评估工作应涵盖相机姿态误差、深度或点云地图的重建质量、手-物对齐程度、时间漂移情况，以及下游的接触点或物体状态一致性等指标。 --- ![Image](images/6c2927b27f33005cfea01379959a4a31908daf2b1dca138e3ca7019580eb144e.jpg)

Figure 5: Injection mechanisms of semantic priors for HOI.

## 4 HOI的语义先验 ### 4.1 语义先验的范围本节重点在于确定所观测区域所代表的对象、涉及的功能部件以及它们所暗示的交互意图。语义先验将基础模型生成的视觉与语言理解信息注入到HOI流程中。我们根据它们在HOI流程中的功能，区分出两种互补的子先验。语义定位先验能够将开放词汇语义、文本提示或空间查询转化为局部的视觉证据，包括边界框、掩码、区域对齐表示、区域对应关系以及时序区域轨迹。语言推理先验则将自然语言、功能知识与交互意图转化为对抓取目标、接触区域、运动阶段及任务语义的约束。当同一VLM用于区域定位时，它充当语义定位先验；而当用于推断交互意图时，则充当语言推理先验。图5展示了两种语义干预机制：语义定位通过区域条件注入包括掩码、边界框和区域对齐表示在内的区域证据；而语言推理则把可操作性、意图及接触线索作为交互约束或评分注入。 ### 4.2 语义定位先验此处综述的语义定位方法采用Grounding DINO [^7]进行文本提示下的定位，利用SAM [^12]、SAM 2 [^106]和SAM 3 [^113]实现可提示的图像与视频分割，借助LISA [^99]完成基于推理的分割，并通过Amodal Video Segmenter [^117]恢复视频中被遮挡的物体区域。 #### 4.2.1 基于语义定位的手部-物体姿态估计语义定位可直接作为姿态估计的重建骨干，而非仅生成外部的裁剪结果或掩码。HandOS [^115]使用预训练的Grounding DINO 1.5 [^7]，以文本提示“Hand”进行查询，并通过针对手部设计的一体化解码器，利用其定位检测特征与实例查询来预测二维关节点、三维手部顶点以及相机平移参数。ScaleHP [^149]同样采用预训练的Grounding DINO，以通用提示“Hand”获取多模态特征、定位查询及参考点；其度量二维-三维解码器将这些输出与二维、三维关节查询及尺度标记相结合，再通过受透视约束的最小二乘模块，计算出相机坐标系下的全局平移参数与度量手部姿态。 #### 4.2.2 基于语义定位的手持物体重建 EasyHOI [^31]利用LISA [^99]和SAM [^12]这两种分割模型生成手部与物体的掩码，以此界定后续修复与三维生成的空间范围。 #### 4.2.3 基于语义定位的动态HOI重建在视频和4D HOI重建中，语义定位能够提供物体定位、掩码传播以及动态区域关联。ArtHOI [^32]利用SAM 2 [^106]生成的手部和物体掩码来定位相关区域，再将其传递给4D优化器。CHOIR [^116]将SAM 2与基于扩散的无模态视频分割器 [^117]相结合，以生成用于接触感知优化的时间上完整物体掩码。GraG [^45]借助SAM 3 [^113]的概念定位功能，从输入视频中提取手部和物体掩码。EgoGrasp [^147]和AGILE [^143]先利用SAM 2掩码分离出手部和物体区域，然后再应用各自的几何重建模块。GHOST [^107]则使用SAM 2在交互视频中传播物体掩码，随后进行资产检索和高斯重建。 #### 4.2.4 分析与讨论语义定位先验作为高召回率的观测提取器最为有效。它们能将开放词汇文本或多模态提示转化为掩码、边界框、局部区域、区域标记或时间轨迹，从而让手物处理流程对封闭集检测器和特定任务分割标签的依赖程度降低。这一点对于现实场景中的HOI尤为重要，因为被操控的物体、功能部件或交互区域可能并不属于预定义的类别。其主要弱点在于误差传播：略微不准确的掩码就可能导致修复、形状生成、资产检索、接触估计以及4D优化出现偏差；尤其在指尖、透明物体、反射表面以及严重遮挡的接触区域，这种误差会更加明显。因此，评估语义定位时不仅要看检测或分割分数，还要考量其对重建、接触及生成质量产生的后续影响。 --- 优化过程中，指尖、透明物体、反射表面以及严重遮挡的接触区域容易出现严重的误差。因此，语义定位的效果不应仅通过检测或分割分数来评估，还需考量其对重建、接触及生成质量的后续影响。 ### 4.3 语言推理先验这类先验包含两种不同的语义机制。基于推理的约束利用LLM或MLLM来推断功能部件、接触意图、任务阶段或可供性；而语言对齐的条件化则是将可复用的文本和视觉嵌入或标记直接传入抓取或运动生成器，无需显式的推理过程。典型的来源包括GPT-4/4o [^139]、LLaMA [^125]、Qwen [^150]、CLIP [^21]，以及多模态模型如LLaVA [^151]、GPT-4V [^141]和Qwen-VL [^16]。在这两种情况下，都需要预训练好的跨域模型来改变语义条件、交互约束或生成结果；仅靠文本标签或针对任务训练的语言编码器是不够的。 #### 4.3.1 基于语言推理的手物抓取合成语言推理先验将功能语义注入手物抓取合成过程，不仅能回答“能否抓取”，还能确定“抓取的目的”。 Text2Grasp [^152]利用GPT-3 [^153]和CLIP [^21]将任务级或个性化描述转化为零件级的抓取方案。SemGrasp [^13]结合了Vicuna [^154]、GPT-4/4V [^139]以及CLIP [^21]，用于将自然语言描述映射为离散的抓取表示。Multi-GraspLLM [^155]将物体点云特征与基于Vicuna的语言骨干网络 [^154]对齐，并自回归地预测多个机械手的抓取标记。NL2Contact [^121]将ChatGPT生成的接触描述 [^122]转化为BERT嵌入 [^123]，以此来约束手-物体接触图的生成。AffordGrasp [^118]结合了RoBERTa [^119]和Qwen2 [^120]的条件约束，并利用可操作性感知扩散模型来限制人类抓取合成过程。G-DexGrasp [^156]利用GPT-4o [^157]推断任务指令指定的可操作类型及接触部位，再借助GLIP [^158]在物体上定位该部位。由此产生的接触-部位证据以及检索到的姿态-接触分布，会指导针对未见过的物体类别进行基于MANO的抓取生成与优化。AffordDexGrasp [^159]使用GPT-4o [^157]提取意图、物体部位及抓取方向等线索，以此约束开放集下的灵巧抓取合成过程。 #### 4.3.2 基于语言推理的HOI运动生成语义先验同样会影响HOI运动生成，其中语言用于指定操控目标、阶段结构、交互语义，以及文本与运动之间的对应关系。此处分别单独介绍显式的LLM/VLM推理和预训练的语言对齐条件作用，而非将其视为同一种操作。有一类方法会将预训练的语言对齐嵌入注入到特定任务的运动生成器中。Text2HOI [^88]利用CLIP [^21]的文本特征来约束其3D HOI运动生成器，同时辅以辅助接触模块来预测手-物体表面接触概率。DiffH2O [^160]将CLIP文本嵌入与物体形状表征一同输入给扩散去噪器，用于生成抓取与操控运动；因此语义先验是通过去噪器条件作用来引入的。JointHOI [^161]将CLIP文本嵌入与PointNet物体特征结合，转化为Transformer去噪器的前缀条件标记，从而共同生成双手法运动、物体运动及动态接触图。StructBiHOI [^162]则将CLIP文本嵌入与物体特征结合，用来约束其基于Mambas的扩散去噪器，以生成长时程的双手法运动。另一组方法则对预训练语言模型进行调整，以表示交互序列或生成显式的中间约束。HOIGPT [^124] 首先将手部和物体的运动编码为分解后的 VQ-VAE 标记，随后调整 LLaMA-13B [^125] 模型，使其能基于混合文本与 HOI 标记进行自回归生成。这样一来，语言模型的先验知识便在序列生成阶段发挥作用。OpenHOI [^126] 则以 ShapeLLM [^127] 为基础初始化其 3D 多模态大语言模型，并通过微调使其能够预测空间可供性图谱与语义任务分解结果，这些结果会进一步作为后续可供性驱动扩散模块及物理优化模块的输入条件。SynHLMA [^163] 对预训练的 Vicuna-7B 模型 [^154] 进行调整，将语言嵌入与离散的操作标记对齐到共享的自回归序列中，之后再解码出与关节物体之间的交互行为。TOUCH [^164] 利用 Qwen-7B [^150] 将文本转化为分层式的粗到细粒度条件，这些条件会输入到其可控的 HOI 生成器以及显式的接触图预测分支中。MEgoHand [^165] 借助 Eagle-2 [^166] 对任务指令与以自我为中心的 RGB 图像上下文进行编码，同时利用 UniDepth V2 [^105] 添加度量化的空间线索。随后通过跨模态注意力机制融合得到的语义与深度表征，再由经过任务训练的 DiT 流匹配解码器生成精细的手-物轨迹。在这些方法中，通用预训练语言模型均能为 HOI 生成提供序列级的指导，或是中间的可供性、任务及接触表征。 #### 4.3.3 分析与讨论语言推理先验通过将任务描述、物体可供性、功能知识与意图转化为抓取目标、接触区域、运动阶段或交互约束，从而降低了语义不确定性。其价值在于能够区分视觉上相似但功能各异的物体。 --- 这类交互包括握住、传递、打开、按压或转动同一物体等。不过，语义上的合理性并不等同于几何或物理上的有效性。语言模型或许能正确推断出应该抓住杯子的把手，但它自身却无法确定无碰撞的手指放置位置、稳定的力闭合状态、摩擦可行性，或是符合动力学规律的物体运动方式。因此，语言推理必须被落实到具体的空间区域、接触候选点、轨迹，或是模拟器/机器人验证循环中。未来的研究应当明确评估语言先验是否能提升功能性接触效果与任务成功率，而不只是改善文本输出的对齐程度或定性可解释性。ContactPrompt [^167] 以及 Suzuki 等人提出的可供性引导扩散先验 [^168] 进一步表明，多模态大语言模型/视觉语言模型的推理能够为模块化 HOI 系统生成仅涉及接触或手部动作的辅助信号。 ## 5 HOI 的视觉先验 ### 5.1 视觉先验的范围本节综述了来自通用预训练模型的可迁移视觉知识：其中视觉表征先验提供通用的图像特征或标记；图像生成先验用于建模单帧的外观与布局；而视频生成先验还能进一步建模时间维度的外观变化、身份持续性以及交互演化过程。图6总结了视觉先验的三种常见注入路径。通用视觉编码器提供的特征令牌会与特定于HOI的表示相融合；预训练的图像和视频生成器则通过权重初始化、任务特定的微调以及基于适配器或ControlNet的条件化方式，贡献参数或潜在特征，其去噪分数估计能够在优化过程中对观测较弱的HOI变量起到正则化作用。在动作条件化的世界模型中，预测得到的观测结果还会被反馈作为输入，用于交互式展开。 ### 5.2 视觉表示先验视觉表示先验将通用的图像特征或令牌传入HOI模型，而非直接生成掩码、3D几何结构或图像。DINOv2 [^17]是这类先验的主要来源。其视觉特征会通过令牌融合方式与手部、物体或点云表示相结合，这种融合可通过交叉注意力机制来实现。 #### 5.2.1 基于视觉表示先验的手部-物体姿态估计 HOPFormer [^57]利用DINOv2 ViT-G [^17]提取物体和场景令牌，再通过交叉注意力机制与专为姿态估计设计的WiLoR手部令牌 [^63]进行融合。融合后的表示能够同时估计两只MANO手以及被操纵物体的6D或关节化姿态。 ![Image](images/c73cb9577c50218db8bd190d773366e565d0ea2bf98c7b7fd23401ab84f5824b.jpg)

Figure 6: Injection mechanisms of visual priors for HOI.

#### 5.2.2 基于视觉表示先验的手持物体重建 HORT [^169]将DINOv2 [^17]的视觉令牌用于手持物体重建，在预测由粗到细的物体点云及其相对于手部的姿态之前，先将其与手部几何结构融合。ForeHOI [^128]会聚合交互视频中的DINOv2补丁特征，再将其与手部特征结合，通过前馈式掩码补全和3D形状预测来重建被遮挡的物体。MCC-HO [^22]利用DINOv2特征作为辅助视觉先验，以在帧间对齐其生成的对象几何结构。Jiang等人 [^142]则在物体姿态初始化阶段，利用DINOv2特征将所选生成网格的渲染视图与视频帧对齐。 #### 5.2.3 基于视觉表示先验的手部-物体抓取合成用于抓取合成的视觉表示先验相关研究还较少。典型的例子是人类通用抓取（HUG）[^134]：该方法会固定DINOv2 [^17]的图像特征，将其与度量点云令牌融合，再根据RGB-D输入和查询点生成开放世界中的人类抓取姿势。生成的MANO抓取姿势随后可迁移到各类机器人手上。 #### 5.2.4 分析与讨论这种方法的主要价值在于其对开放类别的鲁棒性。DINOv2 [^17]的特征源自大规模图像预训练，因此对于HOI训练集中很少涵盖的物体类别仍能提供有效信息，且在手部遮挡情况下也能保持较好的性能。在不同任务中，这些特征会以两种方式进入HOI模型：一是通过交叉注意力机制与特定任务的表示（如专为姿态估计设计的手部令牌、手部几何结构以及度量点云令牌）融合；二是将生成或检索到的物体几何结构与视频对齐。 --- 观测结果如此。然而，外观特征并不具备度量尺度、关节信息或接触信息，因此它们虽有助于识别与对齐，却无法确定三维结构或物理有效性；而缺乏纹理或透明物体等模糊的外观特征，可能会误导融合阶段。因此，评估时应侧重未见过的物体类别，而非仅限于基准测试中的分类，同时还需考虑其对姿态估计、重建效果及接触质量所产生的后续影响。 ### 5.3 图像生成先验此处综述的图像生成方法以GLIDE [^138]、Stable Diffusion [^18]、SDXL [^170]以及Zero-1-to-3 [^130]作为图像合成与新视角生成的预训练扩散模型骨干。ControlNet [^171]也参与其中，但其作用有所不同：它仅为这些骨干模型添加空间条件约束，并非作为独立的基础模型使用。 #### 5.3.1 基于图像生成先验的手持物体重建在几何主导的流水线中，图像生成也可作为辅助先验出现。EasyHOI [^31]利用Affordance Diffusion [^100]来重建被手部遮挡的物体外观，随后由InstantMesh生成物体网格，供几何重建阶段使用。 MagicHOI [^129]则采用已训练的Zero-1-to-3 [^130]模型，在从短单目视频中进行手-物联合重建时提供新视角的去噪引导。由此产生的去噪分数估计能够对输入视图中观察较弱的物体几何进行约束。Diffusion-Guided Reconstruction [^172]则以GLIDE [^138]为基础初始化图像条件扩散模型，将其适配为手条件下的几何渲染模型，再利用得到的去噪分数估计来优化每个交互片段中的完整物体形状。在这两种方法中，所迁移的先验均通过去噪分数估计来约束重建过程，而非直接提供来自三维基础模型的重建形状。 #### 5.3.2 基于图像生成先验的动态HOI重建针对动态HOI重建的图像生成先验相关研究仍较为有限。典型的例子是BIGS [^131]，它利用Stable Diffusion [^18]产生的去噪分数估计来约束随时间变化的物体高斯分布中手部遮挡的区域。在此过程中，ControlNet [^171]负责提供空间条件约束。这种方法必须在保留物体外观及交互序列中的时间一致性的同时，对观察较弱的几何信息进行约束，这也导致在动态重建中直接运用图像生成先验的情况并不常见。 #### 5.3.3 基于图像生成先验的HOI图像/视频生成 HOIDiffusion [^49] 通过结合CLIP [^18]、ChatGPT [^21] 和 LLaVA [^122] 条件对Stable Diffusion [^151] 进行改进，以生成3D手-物体交互数据，证明了图像先验能够合成用于HOI重建的训练数据。Affordance Diffusion [^100] 从GLIDE [^138] 初始化其LayoutNet，用以采样合理的手-物体接触布局；随后，它将大规模预训练的潜扩散图像修复模型 [^18] 作为ContentNet进行微调，以便根据物体和采样到的布局来渲染手-物体图像。单视图到新视图生成 [^173] 从Zero123XL [^174] 初始化其新视图扩散骨干网络，该模型是在Objaverse-XL上训练的Zero-1-to-3 [^130] 的扩展版本；同时，它通过ControlNet风格的条件分支 [^171] 注入手部深度和骨骼线索。Prompt-Propose-Verify [^175] 利用GPT-4 [^139] 丰富交互提示词，并借助经过DreamBooth适配的SDXL生成器 [^176] [^170] 构建HOI图文数据集，随后对SDXL进行微调以实现HOI图像生成。除了合成完整的交互数据外，这些先验还支持以手为中心的图像生成与编辑。Hand1000 [^177] 利用手势感知的文本嵌入对Stable Diffusion [^18] 进行微调，从而实现可控的手部图像生成。RHanDS [^178] 从Stable Diffusion Inpainting v1.5 [^18] 初始化其VAE和去噪U-Net，并从预训练的深度ControlNet [^171] 获取结构编码器，以便在保持图像风格的同时修复畸形的手部。AttentionHand [^179] 结合文本和渲染网格条件对Stable Diffusion [^18] 进行改进，以生成用于手部重建的训练图像。 #### 5.3.4 基于图像生成先验的HOI运动生成 SIGHT [^135] 通过结合CLIP [^18] 条件对Stable Diffusion [^21] 进行扩展，借助推理时的扩散引导来生成3D手-物体轨迹。 #### 5.3.5 分析与讨论图像生成先验拓宽了外观、物体类别、背景以及手-物体布局的分布范围，但图像保真度并不等同于有效的解剖结构、接触关系、遮挡情况或抓取功能。因此，应依据交互相关的标准以及其在姿态估计、接触预测、抓取合成或开放世界泛化方面的下游收益来评估它们的价值。 ### 5.4 视频生成先验本文综述的视频生成方法采用了DynamiCrafter [^180]、CogVideoX [^19]和Wan [^133]，其中FLUX.1 [^181]为这些视频生成流程提供了基于图像先验的外观初始化与修复条件。 --- 从而将时间维度的外观、物体身份以及交互动态信息迁移到有或无交互展开过程的HOI任务中。 #### 5.4.1 基于视频生成先验的动态HOI重建 DreamHand [^132] 重新利用Wan [^133] 从第一人称视角视频中恢复抗遮挡的3D手部运动，表明当与几何约束相结合时，视频生成先验能够支持结构化重建。 #### 5.4.2 基于视频生成先验的HOI图像/视频生成视频生成先验技术推动了人机交互（HOI）视频生成、编辑和重演能力的快速发展。 Re-HOLD [^137] 以Stable Diffusion v1.5 [^18]为基础初始化其去噪分支与参考分支，并通过ControlNet风格的分支 [^171] 为人机交互视频重演添加布局约束。TASTE-Rob [^182] 在面向任务的人机交互视频上对DynamiCrafter [^180] 进行微调，以合成具有任务对齐的手部-物体动态的交互序列。SViMo [^91] 对CogVideoX-5B [^19]进行调整，结合T5文本特征 [^136]，共同生成人机交互视频及显式的3D交互序列，并将生成的运动反馈到扩散过程中，以提升视频与运动的一致性。iDiT-HOI [^183] 将Wan-14B图像转视频先验 [^133] 与FLUX.1-dev [^181] 的修复条件相结合，实现两阶段人机交互重演。Yan等人 [^184] 对CogVideoX-12V-5B [^19] 和Wan2.1-12V-14B [^133] 进行调整，引入结构感知与接触感知约束，用于开放世界的人机交互视频生成。HVG-3D [^185] 冻结预训练的CogVideoX-5B-12V骨干网络 [^19]，通过可训练的、初始权重为零的3D ControlNet注入3D点云与跟踪线索，从而实现几何可控的人机交互视频合成。PAM [^186] 以FLUX.1 [^181] 和CogVideoX [^19] 为基础初始化其外观与运动阶段，通过ControlNet分支 [^171] 注入渲染的深度图、实例掩码和手部关键点，用于从仿真到现实的人机交互视频生成。 #### 5.4.3 从人机交互视频生成到人机交互世界模型近期的研究进一步将人机交互视频生成拓展为动作条件化的交互合成。给定当前观测数据与动作信号，这些模型无需将未来物体轨迹作为输入即可预测未来的人机交互帧。EgoHOI [^187] 对Wan 2.1-14B [^133] 进行微调，融入物理感知的3D几何与运动学嵌入，从而根据首帧与动作信号预测以人眼为中心的互动过程。Hand2World [^188] 对Wan2.1-1.3B Control [^133] 进行调整，加入相机适配器与手势约束，以实现自回归的以人眼为中心的交互过程生成。Generated Reality [^189] 以Wan2.2 [^133] 为基础，结合手部与相机的联合控制条件，生成更长的人类中心交互视频。Dexterous World Models [^190] 以CogVideoX-Fun（CogVideoX的修复变体） [^19] 为基础进行初始化，并以以人眼为中心的场景渲染结果及手部网格运动作为模型的条件输入。HandsOnWorld [^191] 对Wan [^133] 进行微调，使用仅包含主角的3D手部轨迹与Plücker手图，在日常交互视频中分离相机自身运动与手部控制。 Wh0 [^192] 对Wan-I2V-A14B [^133] 进行微调，融入语言、物体与场景条件，以合成大规模以人眼为中心的手部-物体操作视频。随后它提取手部运动数据并进行视觉编辑，将生成的视频转化为机器人策略训练的监督数据。 #### 5.4.4 分析与讨论视频生成先验模型能够刻画交互各阶段的时间外观与身份特征，从而支持增强、重演、规划以及交互式展开等操作。除了用于交互合成外，它们还可充当可扩展的HOI数据生成器，而非直接的交互模拟器，正如Wh0中所展示的那样。这种数据生成角色不同于具身迁移中的生成过程（第6节）：后者由HOI重建或生成模型产出与任务对齐的交互结果，直接服务于机器人学习；而此处，通用视频先验仅提供视觉HOI数据，将其转化为机器人监督信息属于后续应用。不过，时间连贯性并不等同于因果或物理上的有效性：接触漂移、不合理的物体运动、不一致的关节活动以及深度排序错误等问题仍可能存在。因此，评估时应将视频质量指标与接触持续性、物体状态转换准确性、遮挡一致性以及重建或机器人执行性能等指标结合起来考量。 ## 6 基于HOI的具身迁移 ### 6.1 具身迁移的范围本节探讨HOI证据及HOI重建或生成输出如何支撑机器人学习。此处所说的HOI证据，是指原始手-物视频中存在的、无需显式三维恢复即可利用的交互信号，例如手-物运动、接触情况以及物体状态变化等。输出则是指HOI重建或生成模型基于这些证据所产出的结构化变量，比如手和物体的轨迹、接触图以及生成的交互视频等。无论是哪种形式，均可直接用于指导机器人策略，也可在不同实体间适配，或编码为从视觉变化中推断出的潜在动作，其中实体映射会将人类产生的信号与机器人的形态或动作空间相匹配。我们涵盖的方法包括：利用这类知识提供视频级监督、显式的HOI变量，或是用于约束机器人决策的交互引导信息。 --- ![Image](images/071b9a33403f0ae9d7fac868cc39a42bd24dcea856527de65829d63eedbc5eb7.jpg)

Figure 7: Five routes by which HOI evidence becomes robot-learning supervision, target-skill guidance, or reusable robot training data.

图7梳理了五种路径：基于视频和结构化的人类数据预训练、针对目标技能的演示重定向与交互引导迁移，以及能产出可复用且适配机器人的训练数据的HOI转机器人数据引擎。 ### 6.2 人类数据预训练 #### 6.2.1 基于视频的预训练基于视频的预训练是此处讨论的结构化程度最低的迁移路径。它无需显式重建手部、物体或接触情况，而是利用以操作任务为中心的人类视频来提供视觉状态转换信息，策略据此学习潜在动作、图像目标或运动标记。当视频片段清晰捕捉到手部接近、接触、物体响应以及释放等过程时，这些表征就能提供可与机器人动作对齐的弱HOI证据。LAPA [^23] 能从无标签的视频转换中学习潜在动作，并在后续适配过程中将其解码为机器人动作。Moto [^193] 将潜在运动标记作为视频预测与控制之间的桥梁。Video Prediction Policy [^194] 通过视频预测在人类操作视频上预训练预测性视觉表征，随后将这些学到的动态特性迁移到通用机器人策略中。IGOR [^195] 研究了图像目标表征，而In-N-On [^196] 和Emergence of Human to Robot Transfer [^197] 则探讨了从人类视频到机器人策略的迁移问题。当机器人数据、动作词汇表或特定形态的适配器可用时，人类视频预训练会与机器人动作空间明确对齐。CLAP [^198] 将人类视频过渡与机器人的本体感知动作编码本对齐。UniVLA [^199] 学习以任务为中心的潜在动作，并能将其适配到不同形态的机器人上；而villa-X [^200] 则改进了潜在动作的建模及其在VLA预训练中的整合。大型视频预训练机器人模型提供了更多佐证：GR-1 [^201] 和 GR-2 [^202] 将通过视频学到的视觉动态特性迁移到策略微调中；GR00T N1 [^203] 通过潜在动作或逆动力学，将无动作的操作视频转化为策略监督信号；Motus [^204] 能推导出像素级的潜在动作；EgoScale [^205] 则在人机协同训练前先进行以自我为中心的预训练。在这些视频预训练方法中，规模越大覆盖范围越广，但潜在压缩过程可能会丢弃指尖接触信息、物体的6D状态以及因果物理效应。因此，它们的迁移效果应通过机器人数据效率和对未见过的任务、物体及形态的泛化能力来评估。 #### 6.2.2 结构化HOI监督视频预训练通常学习的是视觉和时间变化的隐式表征，而非显式的手-物变量。结构化HOI监督则将这些变量以手部和腕部运动、3D手-物状态、接触与可供性线索、帧对齐的动作片段以及6自由度物体轨迹的形式呈现出来，从而为HOI分析与人类数据预训练之间提供直接接口。有一类方法会直接将这些变量用作预训练的监督信号。Being-H0 [^206] 构建了具有3D物理对齐特性的部件级手部运动标记。VITRA [^207] 将人类视频转换为帧对齐的3D手部运动及动作片段。EgoScaler [^208] 从以自我为中心的视频中提取6自由度物体轨迹，并自动优化那些噪声较大或不完整的轨迹，从而使交互信号变得显式化。其他方法则将结构化HOI变量与机器人端的对齐或策略训练相结合。EgoVLA [^24] 从以自我为中心的HOI视频中推导出腕部和手部动作目标，再通过逆运动学和重定向将其映射到机器人动作上。H-RDT [^209] 在跨形态微调前，将3D手部姿态作为特权监督信号。UniHM [^210] 利用语言条件化的HOI状态与动作来实现跨手部及跨任务的迁移。相较于潜在的视频标记，这些映射方式更具可解释性。 ### 6.3 人类技能向机器人的迁移 #### 6.3.1 演示对齐与重定向这一路径会将人类演示转化为针对特定技能的、与机器人对齐的动作序列。与宽泛的人类数据预训练不同，该路径会利用经过重构、推断或编辑的中间HOI状态，例如抓取动作、手-物轨迹、物体状态变化或机器人化的视觉观测结果，并在不同形态间实现显式对齐。 --- DexMV [^25]和Web2Grasp [^211]利用运动学重定向技术，将推断出的人类演示或功能性抓取动作转化为灵巧机器人所需的监督信号。HUG [^134]则基于冻结的DINOv2 [^17]特征与度量点云令牌来合成MANO抓取动作，无需针对特定机器人本体进行训练即可将其重定向到不同的机械手上。GraspDreamer [^212]借助Veo 3.1、Kling Video 2.1或Gemini 2.5 Image来合成功能性抓取演示，并对恢复出的手部轨迹进行优化以供机器人执行。ManipTrans [^213]学习从人类双手动作到机器人化演示的残差映射关系，而DexMachina [^214]则通过以物体状态为中心的重定向机制来保留任务功能。CHORD [^215]则在以物体为中心的接触力矩空间中对齐人类与机器人的执行过程，利用产生的力-扭矩效应作为强化学习的引导信号。EgoMimic [^216]在经历3D手部追踪、动作归一化和视觉掩码处理后，同时利用人类与机器人的演示数据进行训练，从而省去了单独的重定向步骤；DexUMI [^217]则将可穿戴设备采集的人类手部动作映射到机械手上。Object-Centric Dexterous Manipulation [^218]将基于人类手部动作捕捉数据训练的腕部轨迹生成器与强化学习得到的手指控制器相结合，即便存在本体差异也能恢复出以物体为中心的操控技能。VideoManip [^219]能从单目RGB视频中重建3D手-物轨迹，并通过接触优化对其进行重定向，无需可穿戴设备即可训练出灵巧操控策略。有些方法并不采用运动学重定向技术，而是直接从重构或编辑后的HOI状态中构建演示内容。DexInit [^220]能从单目人类视频中重构双手演示；Masquerade [^221]能将人类观测结果编辑为机器人化的视觉演示；YOTO [^222]则从一次人类演示中提取交互状态与关键帧轨迹，再将其扩展为机器人训练数据。在这些变体中，技能迁移的质量需同时考量跨本体映射的精确度以及下游任务成果（如预期的物体状态变化）；仅依靠关节角度相似性并不足以评估其效果。 #### 6.3.2 交互引导的机器人操控交互引导式操控无需逐帧进行人类到机器人的对齐操作。它传递的是一种与本体无关的交互描述，比如可供性、接触目标、视觉轨迹或生成的操控视频，这些描述可作为机器人决策的依据。多种方法会依据从HOI视频中推断出的可供性或轨迹来指导决策过程。GAT-Grasp [^223]从HOI视频中推断可供性，并将得出的抓取目标传递给机器人抓取策略。ATM [^224]能预测视频中任意点的未来轨迹，随后将这些轨迹作为控制指引提供给视觉-运动策略，从而减少了对带标签的动作演示的需求。VidBot [^225]通过单目深度信息与运动恢复结构技术，从自然场景视频中恢复出度量级的3D手部轨迹，随后学习语言条件化的3D可供性，以此指导跨本体的操控行为。FlowHOI [^226]生成语义对齐的手-物运动，并将该运动作为灵巧操控的引导信号。基于流的传输方法使中间物体的运动更加直观。Im2Flow2Act [^227] 从人类演示视频中提取物体点，预测它们的未来轨迹，并将生成的物体流用作生成机器人动作的接口。3DFlowAction [^228] 从人类和机器人的操作数据中学习语言条件化的3D流世界模型，预测未来的物体流，并利用该预测来约束闭环机器人动作优化。NovaFlow [^229] 和 Dream2Flow [^230] 同样会将生成的视频提炼为3D物体流或运动，再将其转化为针对刚体和变形体的抓取方案、轨迹优化结果或底层机器人指令。基于点追踪的方法采用类似的接口，但无需密集的物体流场。3PointTr [^231] 从普通的人类视频中预测与实体无关的3D点轨迹，将其压缩为点追踪标记，再通过少量机器人动作数据训练机器人策略来遵循这些标记。Dex4D [^232] 从人类视频或生成的点预测中获取以物体为中心的点轨迹，在仿真环境中训练点到动作的策略，再通过实时追踪器执行该策略以实现闭环的仿真到现实操作。另一条路径则利用生成的视觉交互规划。Gen2Act [^233] 直接利用预训练的VideoPoet模型 [^234] 生成语言条件化的人类操作视频，再将该视频作为视觉任务规划输入到机器人策略中。TASTE-Rob [^182]（参见第5.4节）会优化其生成的手部-物体运动，并将生成的视频用作模仿学习的演示样本。 ### 6.4 HOI到机器人的数据引擎上述方法要么从人类HOI证据中预训练策略，要么将这类证据转化为单一的目标技能。另一类互补的方法则将真实的人类HOI证据转化为可复用的跨实体数据集、机器人化视频或可执行的演示样本，用于训练不同学习算法下的机器人策略。近期的研究系统也聚焦于具身迁移中的数据获取环节。ACE-Data-0 [^235] 推出了一种环境捕获引擎，可在家庭环境中同步记录第一视角和外视角视频、身体和手部运动、物体几何形状及6自由度轨迹、音频和触觉信号，还设置了分层基准测试来关联感官信号与HOI。除了数据捕获之外，其他系统会将记录的HOI证据转化为机器人端的训练数据。 --- RoboWheel [^236] 通过轨迹重建和跨实体重定向，将单目人类HOI视频转化为机器人数据。EgoEngine [^26] 可生成灵巧的机器人演示，而Ego2Robot [^237] 则侧重于实现不同机器人形态下的可扩展动作与视觉对齐。HandEdit [^238] 针对26种实体形态开展人类到机器人的视觉编辑基准测试。RoboEdit [^239] 通过视频编辑将人类视频转化为机器人经验。Do as I Do [^240] 重构手部-物体轨迹并将其重定向用于机器人操作。EgoInfinity [^241] 可处理网络规模级的HOI证据，并支持任意视角的重定向。Human2Robot [^242] 将人类操作转化为与机器人对齐的视觉演示。TraceGen [^243] 将视频转化为3D轨迹空间的监督信号。H2R-Grounder [^244] 将人类视频转化为物理上合理的机器人视频。Qwen-RobotManip [^245] 将手部轨迹重定向用于跨平台机器人演示。这些引擎在生成的机器人端数据上存在差异，包括状态-动作轨迹、配对演示以及机器人化视频，但它们都面临着在弥合视觉、运动学和物理实体化差距的同时，保持与任务相关的HOI状态的挑战。评估时应区分HOI重建的保真度、视觉真实感、动作可执行性、跨实体化覆盖范围以及下游数据效率。 ## 7 数据集与评估协议我们梳理了HOI任务分类和实体化迁移中使用的基准数据集、预训练数据源以及评估协议，总结了各任务常用的指标类别及其盲点，并沿用第2.1节中介绍的缩写R1–R3、G1–G3和ET。 ### 7.1 数据集与预训练数据源表4展示了两组并列的内容：用于评估重建与生成方法的任务级HOI基准，以及用于模型开发和实体化迁移的大规模以人为中心的数据源。对于大规模数据源，“用途”列区分了视频模型预训练与人数据策略预训练。前者从以人为中心的视频中学习通用的视觉和时间表征，后者则利用人类交互数据来预训练机器人策略或动作表征。这些标签仅描述数据的用途；方法层面的分类遵循第6节的规定。 ### 7.2 指标类别及其盲点 #### 7.2.1 几何指标 MPJPE和MPVPE等关节与顶点误差需要配对的3D真实值，用于衡量评估坐标框架内的手部姿态或网格精度。它们的Procrustes对齐变体PA-MPJPE和PA-MPVPE会先通过相似变换对预测结果与真实值进行对齐，消除全局平移、旋转和缩放的影响，从而突出关节姿态/形状的吻合程度，同时掩盖绝对位置、方向和度量尺度上的误差 [^289]。物体指标则有不同的数据要求：旋转与平移误差需要真实的物体姿态；ADD和ADD-S还需要物体模型点；而Chamfer距离、F分数和IoU等表面指标则需要真实的表面或占据几何信息，但不一定需要带姿态的CAD模型 [^290] [^291] [^292]。这些指标量化了几何一致性，但较低的误差并不等同于正确的接触、可抓取性或功能性交互。 #### 7.2.2 视觉与运动指标 FID和FVD分别用于比较非配对的图像和视频分布，而PSNR、SSIM和LPIPS则需要配对的参考样本，更适合用于重建、编辑或重演 [^293] [^294] [^295] [^296]。运动和轨迹误差同样需要配对的序列，而多样性和覆盖范围则用于描述生成样本的分布情况而非其正确性。对于条件生成而言，条件遵循率用于衡量输出是否满足指定的文本、姿态、接触或运动条件，但无法验证其几何/物理有效性。这些指标均无法验证手部解剖结构、3D一致性、接触持续性、任务完成度或物理动力学特性。 #### 7.2.3 接触与物理指标接触精度、召回率、F1值、IoU以及距离等指标需要依靠接触标注或已配准的手-物体表面数据，来衡量预测接触区域与真实接触区域之间的空间重叠程度 [^2] [^82]。穿透度、模拟位移以及力闭合等指标则用于评估物理合理性的互补方面，不过它们严重依赖于网格质量、摩擦力、接触建模以及仿真器。高精度的接触预测并不等同于能保证力闭合、执行稳定性或功能正确性，而良好的物理评分本身也不足以证明任务能够成功完成。具身迁移评估涵盖两个互补的层面：重定位误差用于衡量将人类产生的抓取动作或轨迹映射到机器人实体上的保真度；而任务成功率与物体状态成功率则用于检验所迁移的信号是否能支撑预期的机器人行为。因此，映射精度虽具有参考价值，但不应与下游任务性能混为一谈。 ---

Table 4: Benchmark datasets and pretraining sources used in HOI research. Panel A summarizes released observations, camera view, acquisition setting, ground-truth (GT) annotations, and use; Panel B summarizes supervision and use for large-scale pretraining corpora. Observation denotes raw sensor streams or released input assets rather than 3D annotations. View — ego: first-person; exo: external third-person; multi-view: synchronized camera rig; object: object-centric; N/A: no camera view. Setting — controlled, in-the-wild, synthetic, web, or mixed. “-” indicates unavailable annotations.

Dataset	Year	Observation	View	Setting	Hand GT	Object GT	HOI GT	Task
Dexter+Object [^246]	2016	RGB-D	exo	controlled	fingertips	cuboid corners	-	R1, R3
EgoDexter [^247]	2017	RGB-D	ego	in-the-wild	fingertips	-	-	R1, R3 (hand-only)
FPHA [^248]	2018	RGB-D	ego	controlled	3D joints	6D pose (subset)	action	R1, R3, ET
FreiHAND [^249]	2019	RGB	exo	mixed	MANO	-	-	R1 (hand-only)
ObMan [^2]	2019	RGB-D	exo	synthetic	MANO	shape+pose	contact	R1, R2, G1
ContactDB [^250]	2019	thermal	object	controlled	-	object mesh	measured contact	G1
HO3D [^35] [^251]	2020	RGB	exo	controlled	MANO	6D pose+CAD	-	R1, R3
InterHand2.6M [^252]	2020	RGB	exo	controlled	MANO (2H)	-	-	R1 (hand-only)
ContactPose [^6]	2020	RGB-D	multi-view	controlled	MANO	6D pose	contact	R1, G1
GRAB [^81]	2020	MoCap	N/A	controlled	SMPL-X	mesh+motion	contact/action	G1, G2
MOW [^253]	2021	RGB	exo	in-the-wild	3D joints	shape+pose	pseudo 3D	R2
DexYCB [^254]	2021	RGB-D	multi-view	controlled	MANO	6D pose+CAD	-	R1, R3
H2O [^255]	2021	RGB-D	ego+multi-view	controlled	MANO (2H)	6D pose+mesh	action/interaction	R1, R3, G2
H2O-3D [^68]	2022	RGB-D	multi-view	controlled	3D joints (2H)	6D pose	-	R1, R3
HOI4D [^58]	2022	RGB-D	ego	controlled	MANO	6D pose+state	action/interaction	R3, ET
OakInk [^256]	2022	RGB	multi-view	controlled	MANO	shape+pose	intent/text	R1, G1
ARCTIC [^59]	2023	RGB	multi-view	controlled	MANO (2H)	art. pose+mesh	contact/action	R1, R3, G2
AssemblyHands [^257]	2023	RGB	ego	controlled	3D joints	-	-	R1 (hand-only)
SHOWMe [^258]	2023	RGB-D	exo	controlled	hand mesh	object mesh	-	R2, R3
AffordPose [^55]	2023	3D assets	N/A	synthetic	MANO	shape+pose	affordance	G1
GazeHOI [^94]	2024	MoCap+gaze	N/A	controlled	hand motion	object motion	gaze/contact	G2
HOGraspNet [^259]	2024	RGB-D	multi-view	controlled	MANO mesh/joints	object mesh	contact/grasp	R1, G1
OakInk2 [^260]	2024	RGB	multi-view	controlled	MANO (2H)	shape+pose	intent/text	R1, R3, G2
TACO [^261]	2024	RGB	ego+exo	controlled	hand motion	object motion	tool/action	G1, G2
GigaHands [^262]	2025	RGB	multi-view	controlled	MANO (2H)	shape+pose	masks/text	R1, R3, G2
HO-Cap [^263]	2025	RGB-D	ego+multi-view	controlled	MANO	6D pose+mesh	tracks	R1, R3, ET
HOT [^264]	2025	RGB-D+tactile	multi-view	synthetic	3D joints	deformable state	force/contact	R3
HOT3D [^265]	2025	RGB	ego+multi-view	mixed	MANO (2H)	6D pose	gaze/scene points	R1, R3
ArtHOI-RGBD [^32]	2026	RGB-D	exo	controlled	-	mesh+part motion	contact	R3
EPIC-Contact [^57]	2026	RGB	ego	in-the-wild	MANO (2H)	6D pose+mesh	dense contact	R1
DexGloveHOI [^266]	2026	RGB+IMU	ego	controlled	3D hand pose	-	-	R1 (hand-only)
SHOW3D [^267]	2026	RGB	ego+exo	in-the-wild	3D joints (2H)	6D pose	captions	R1, R3
HandX [^268]	2026	MoCap	N/A	controlled	MANO (2H)	mesh+motion	contact/text	G2
HUG-Bench [^134]	2026	RGB-D	object	controlled	MANO	metric mesh	-	G1, ET

B. Large-scale pretraining sources and embodied data engines

Dataset	Year	Observation	View	Setting	Supervision	Use
ActivityNet [^269]	2015	RGB	mixed	web	action labels	video-model pretraining
Charades [^270]	2016	RGB	exo	controlled	action labels	video-model pretraining
Kinetics [^271]	2017	RGB	mixed	web	action labels	video-model pretraining
Something-Something [^272]	2017	RGB	exo	controlled	action labels	video-model pretraining
AVA [^273]	2018	RGB	exo	web	spatiotemporal action labels	video-model pretraining
HACS [^274]	2019	RGB	mixed	web	action segments	video-model pretraining
HowTo100M [^275]	2019	RGB	mixed	web	narration/ASR	video-model pretraining
FineGym [^276]	2020	RGB	exo	web	fine-grained actions	video-model pretraining
EPIC-KITCHENS [^277]	2021	RGB	ego	in-the-wild	narration/action labels	video-model pretraining
Assembly101 [^278]	2022	RGB	multi-view	controlled	action/procedure labels	video-model pretraining
Ego4D [^279]	2022	RGB	ego	in-the-wild	narration	video-model pretraining
HoloAssist [^280]	2023	RGB-D+IMU	ego	controlled	instruction/gaze/audio/hand pose	video-model pretraining
Ego-Exo4D [^281]	2024	RGB	ego+exo	mixed	narration/body+hand pose	video-model pretraining
EgoDex [^282]	2025	RGB	ego	in-the-wild	language/camera/3D hand pose	human-data policy pretraining
OpenEgo [^283]	2025	multimodal	ego	in-the-wild	standardized signals/3D hand pose	human-data policy pretraining
VITRA [^207]	2025	RGB	ego	web	action chunks/hand motion	human-data policy pretraining
EgoLive [^284]	2026	stereo RGB+IMU	ego	in-the-wild	depth/3D pose/masks/tracks/language	human-data policy pretraining
EgoScale [^205]	2026	RGB	ego	web	hand trajectories/camera/action labels	human-data policy pretraining
EgoVerse [^285]	2026	RGB	ego	web	camera/head motion/text	human-data policy pretraining
HumanNet [^286]	2026	RGB	ego+exo	web	captions/motion	video-model pretraining
FEEL [^287]	2026	RGB+force	ego	in-the-wild	force/contact	video-model pretraining
Open-AoE [^288]	2026	RGB	ego	in-the-wild	text/MANO/camera/action segments	human-data policy pretraining
ACE-Data-0 [^235]	2026	multimodal RGB/video+motion+audio+tactile	ego+exo	in-the-wild	body/hand/object trajectories; contact and language labels	human-data policy pretraining
Ego2Robot [^237]	2026	RGB video+motion	ego	in-the-wild	robot actions/video; quality labels	human-data policy pretraining
HandEdit [^238]	2026	RGB (edit pairs)	ego	mixed	human-to-robot edited pairs; 26 URDF embodiments	human-data policy pretraining
RoboWheel [^236]	2026	RGB	mixed	in-the-wild	robot trajectories/retargeted demonstrations	human-data policy pretraining
EgoEngine [^261]	2026	RGB	ego	in-the-wild	hand-object trajectories/dexterous robot demonstrations	human-data policy pretraining
EgoInfinity [^241]	2026	RGB	ego+exo	web	4D HOI/robot-aligned action data	human-data policy pretraining
Do as I Do [^240]	2026	RGB	ego	in-the-wild	hand-object trajectories/robot demonstrations	human-data policy pretraining
RoboEdit [^239]	2026	RGB	mixed	in-the-wild	robotized visual experience	human-data policy pretraining
Human2Robot [^242]	2026	RGB	mixed	mixed	paired human-robot videos/actions	human-data policy pretraining

---

Table 5: Common metric families in HOI evaluation.

Metric family	Common metrics	Applicable task(s)
Hand pose/mesh accuracy	MPJPE, PA-MPJPE, MPVPE, PA-MPVPE, PCK	R1, R3; G1–G3 when ground-truth hand state exists
Object pose/shape accuracy	ADD/ADD-S, rotation/translation error, Chamfer distance, F-score, IoU	R1–R3; G2 when ground-truth object state exists
Contact agreement	Contact precision/recall/F1/IoU, contact distance	R1, R3; G1–G3 when contact annotations exist
Physical plausibility	Penetration depth/volume, simulation displacement, force closure	R1, R3, G1, G2
Image/video distribution fidelity	FID, FVD	G3
Paired visual similarity	PSNR, SSIM, LPIPS	R2/R3 rendered outputs; paired G3 editing/reenactment
Generative diversity/coverage	Diversity, coverage/multimodality, condition-adherence rate	G1–G3, when multiple samples or conditions are evaluated
Motion/trajectory accuracy	Sequence MPJPE/MPVPE, hand/object trajectory error	R3, G2
Embodied-transfer evaluation	Retargeting error, task success rate, object-state success	ET

## 8 开放性挑战与未来发展方向 ### 8.1 迈向集成化且可验证的HOI基础系统没有单一的先验族能够完全解决HOI问题：几何先验能改善形状与对齐效果，语义先验可定位功能与意图，视觉先验则能建模外观与动态特性，但每种先验都可能存在缺陷。集成化系统应围绕共享的手-物体状态、接触情况以及轨迹，根据主导的不确定性来组合这些先验，随后验证接触准确性、物理合理性，以及在适用情况下还要验证机器人执行效果。 ### 8.2 从几何正确性到交互正确性 HOI评估仍主要依赖MPJPE、Chamfer距离和ADD等指标。尽管ContactDB [^250]、ContactPose [^6]、GRAB [^81]、ARCTIC [^59]以及EPIC-Contact [^57]提供了接触标注，但相关评估协议很少将接触情况与功能、物理合理性以及物体状态转换结合起来考量。未来的基准测试应统一这些维度，并区分仅靠近接触与能产生作用力并引发预期响应的有效接触。 ### 8.3 长时序、动态相机以及世界坐标系下的HOI 以自我为中心的HOI在长时间操作过程中会将相机运动与手-物体运动交织在一起，导致基于相机坐标系的估计结果出现偏差。HOSt3R [^103]用DUSt3R [^14]生成的点云图取代了SfM技术，EgoGrasp [^147]致力于实现世界坐标系下的HOI，ArtHOI [^32]则能处理关节物体的4D重建问题。如何在同一个世界坐标系中，同时恢复自我运动、接触情况以及长时序的手-物体状态，涵盖抓取、操作、释放和重新抓取全过程，仍是尚未解决的问题。 ### 8.4 先验的可靠性、路由与冲突解决组合后的先验可能会出现不一致：检索过程可能返回不匹配的素材，重建过程可能生成与接触情况不符的几何结构，语义定位可能选错功能部件，生成的视频可能隐藏无效的物理现象。系统应估算各先验的置信度，依据可见性、相机运动、遮挡情况以及意图来路由先验，并直接暴露冲突，而非对不兼容的证据取平均值。 ### 8.5 从HOI分析到动态具身记忆抓取之后，夹持器往往会遮挡物体间关系的演变过程。接触前的重建与定位能够在这类信息消失之前，捕捉到物体的身份、几何形状、姿态、功能区域以及预期接触点。动态的具身记忆应当保存这些变量及其不确定性，通过视觉、本体感知、接触事件和物体响应来更新它们，并在抓取、操作、释放和重新抓取过程中维持接触历史与状态转换。难点在于如何在自身遮挡和视角变化的情况下，将抓取前的估计与后续证据相协调。 ### 8.6 以机器人为中心的人类交互理解具身迁移往往会将人类交互转化为没有意图、物体/部件身份、接触信息以及由此产生的场景状态变化的运动轨迹 [^23] [^223] [^224] [^225]。要实现稳健的跨具身迁移，就需要机器人可用的描述方式，涵盖这些变量及交互阶段，从而使策略能够在受到干扰时重新规划，而非单纯重复记忆中的动作。因此，评估时应衡量意图、接触时机、物体状态结果以及执行过程中的安全性，而不只是轨迹的相似性。 ## 9 结论本综述从基础模型先验的角度探讨了交互重建与生成问题。我们将现有方法分为几何、语义和视觉先验三类，分析了它们的知识如何融入交互流程。我们还回顾了由交互衍生的信号如何支持具身迁移。尽管进展迅速，但在整合异质先验、确保交互与物理正确性，以及处理长时序开放世界交互方面仍存在挑战。希望本综述能为理解当前进展提供指引， --- 助力开发具备通用性、交互感知能力且物理上合理的交互系统。 ## 参考文献 [^1]: Bugra Tekin, Federica Bogo, and Marc Pollefeys. H+O: unified egocentric recognition of 3D hand-object poses and interactions. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2019. doi: 10.1109/CVPR.2019.00464. [^2]: Yana Hasson, Gul Varol, Dimitrios Tzionas, Igor Kalevatykh, Michael J Black, Ivan Laptev, and Cordelia Schmid. Learning joint reconstruction of hands and manipulated objects. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2019. [^3]: Lixin Yang, Xinyu Zhan, Kailin Li, Wenqiang Xu, Jiefeng Li, and Cewu Lu. CPF: Learning a contact potential field to model the hand-object interaction. In Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), 2021. [^4]: Zicong Fan, Maria Parelli, Maria Eleni Kadoglou, Xu Chen, Muhammed Kocabas, Michael J Black, and Otmar Hilliges. HOLD: Category-agnostic 3D reconstruction of interacting hands and objects from video. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2024. [^5]: Yufei Ye, Abhinav Gupta, and Shubham Tulsiani. What's in your hands? 3D reconstruction of generic objects in hands. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2022. [^6]: Samarth Brahmbhatt, Chengcheng Tang, Christopher D Twigg, Charles C Kemp, and James Hays. Contactpose: A dataset of grasps with object contact and hand pose. In Proc. Eur. Conf. Comput. Vis. (ECCV), 2020. [^7]: Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, et al. Grounding DINO: Marrying DINO with grounded pre-training for open-set object detection. In Proc. Eur. Conf. Comput. Vis. (ECCV), 2024. [^8]: Yana Hasson, Güil Varol, Cordelia Schmid, and Ivan Laptev. Towards unconstrained joint hand-object reconstruction from RGB videos. In Proc. Int. Conf. 3D Vis. (3DV), 2021. [^9]: Korrawe Karunratanakul, Jinlong Yang, Yan Zhang, Michael J Black, Krikamol Muandet, and Siyu Tang. Grasping field: Learning implicit representations for 人类抓取行为。载于国际3D视觉会议（3DV）论文集，2020年。 [^10]: Patrick Grady, Chengcheng Tang, Christopher D Twigg, Minh Vo, Samarth Brahmbhatt, and Charles C Kemp. Contactopt: Optimizing contact to improve grasps. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2021. [^11]: Sammy Christen, Muhammed Kocabas, Emre Aksan, Jemin Hwangbo, Jie Song, and Otmar Hilliges. D-grasp: Physically plausible dynamic grasp synthesis for hand-object interactions. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2022. [^12]: Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), 2023. [^13]: Kailin Li, Jingbo Wang, Lixin Yang, Cewu Lu, and Bo Dai. SemGrasp: Semantic grasp generation via language aligned discretization. In Proc. Eur. Conf. Comput. Vis. (ECCV), 2024. [^14]: Shuzhe Wang, Vincent Leroy, Yohann Cabon, Boris Chidlovskii, and Jerome Revaud. DUSt3R: Geometric 3D vision made easy. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2024. [^15]: Minghua Liu, Ruoxi Shi, Kaiming Kuang, Yinhao Zhu, Xuanlin Li, Shizhong Han, Hong Cai, Fatih Porikli, and Hao Su. OpenShape: Scaling up 3D shape representation towards open-world understanding. In Adv. Neural Inf. Process. Syst. (NeurIPS), volume 36, 2023. [^16]: Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-VL: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966, 2023. [^17]: Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. DINOv2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023. ## [^18]: Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2022 --- [^19]: Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer. In Proc. Int. Conf. Learn. Represent. (ICLR), 2025. [^20]: Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3D objects. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2023. [^21]: Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Proc. Int. Conf. Mach. Learn. (ICML), 2021. [^22]: Jane Wu, Georgios Pavlakos, Georgia Gkioxari, and Jitendra Malik. Reconstructing hand-held objects in 3D from images and videos. In Proc. Int. Conf. 3D Vis. (3DV), 2026. [^23]: Seonghyeon Ye, Joel Jang, Byeongguk Jeon, Se June Joo, Jianwei Yang, Baolin Peng, Ajay Mandlekar, Reuben Tan, Yu-Wei Chao, Bill Yuchen Lin, et al. Latent action pretraining from videos. In Proc. Int. Conf. Learn. Represent. (ICLR), 2025. [^24]: Ruihan Yang, Qinxi Yu, Yecheng Wu, Rui Yan, Borui Li, An-Chieh Cheng, Xueyan Zou, Yunhao Fang, Xuxin Cheng, Ri-Zhao Qiu, et al. Egovla: Learning vision-language-action models from egocentric human videos. arXiv preprint arXiv:2507.12440, 2025. [^25]: Yuzhe Qin, Yueh-Hua Wu, Shaowei Liu, Hanwen Jiang, Ruihan Yang, Yang Fu, and Xiaolong Wang. Dexmv: Imitation learning for dexterous manipulation from human videos. In Proc. Eur. Conf. Comput. Vis. (ECCV), 2022. [^26]: Yangcen Liu, Shuo Cheng, Xinchen Yin, Woo Chul Shin, Alfred Cueva, Yiran Yang, Zhenyang Chen, Chuye Zhang, and Danfei Xu. Egoengine: From egocentric human videos to high-fidelity dexterous robot demonstrations. arXiv preprint arXiv:2606.12604, 2026. [^27]: Takehiko Ohkawa, Ryosuke Furuta, and Yoichi Sato. Efficient annotation and learning for 3D hand pose estimation: A survey. Int. J. Comput. Vis., 131(12), 2023. [^28]: Taeyun Woo, Wonjung Park, Woohyun Jeong, and Jinah Park. A survey of deep learning methods and datasets for hand pose estimation from hand-object interaction images. Comput. Graph., 116, 2023. [^29]: Yu Miao and Yue Liu. Advances in vision-based deep learning methods for interacting hands reconstruction: A survey. Comput. Graph., 124, 2024. [^30]: Xu Song, Yongyao Li, Yunfan Zhang, Yufei Liu, and Lei Jiang. An overview of learning-based dexterous grasping: recent advances and future directions. Artif. Intell. Rev., 58(10), 2025. [^31]: Yumeng Liu, Xiaoxiao Long, Zemin Yang, Yuan Liu, Marc Habermann, Christian Theobalt, Yuexin Ma, and Wenping Wang. EasyHOI: Unleashing the power of large models for reconstructing hand-object interactions in the wild. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2025. [^32]: Zikai Wang, Zhilu Zhang, Yiqing Wang, Hui Li, and Wangmeng Zuo. Arthoi: Taming foundation models for monocular 4D reconstruction of hand-articulated-object interactions. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2026. [^33]: Javier Romero, Dimitrios Tzionas, and Michael J. Black. Embodied hands: modeling and capturing hands and bodies together. ACM Trans. Graph., 36(6), 2017. doi: 10.1145/3130800.3130883. [^34]: Yuwei Li, Longwen Zhang, Zesong Qiu, Yingwenqi Jiang, Nianyi Li, Yuexin Ma, Yuyao Zhang, Lan Xu, and Jingyi Yu. Nimble: a non-rigid hand model with bones and muscles. ACM Trans. Graph., 41(4), 2022. [^35]: Shreyas Hampali, Mahdi Rad, Markus Oberweger, and Vincent Lepetit. HOnnotate: A method for 3D annotation of hand and object poses. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2020. [^36]: Zerui Chen, Yana Hasson, Cordelia Schmid, and Ivan Laptev. AlignSDF: Pose-aligned signed distance fields for hand-object reconstruction. In Proc. Eur. Conf. Comput. Vis. (ECCV), 2022. [^37]: Hongsuk Choi, Nikhil Chavan-Dafle, Jiacheng Yuan, Volkan Isler, and Hyunsoo Park. HandNeRF: Learning to reconstruct hand-object interaction scene from a single RGB image. In Proc. IEEE Int. Conf. Robot. Autom. (ICRA), 2024. ## [^38]: Hugh Durrant-Whyte and Tim Bailey. Simultaneous localization and mapping: part i. IEEE Robot. Autom. Mag., 13(2), 2006 --- [^39]: Noah Snavely, Steven M. Seitz, and Richard Szeliski. Photo tourism: Exploring photo collections in 3D. In ACM SIGGRAPH 2006 Papers. ACM, 2006. [^40]: Rong Wang, Wei Mao, and Hongdong Li. DeepSimHO: Stable pose estimation for hand-object interaction via physics simulation. In Adv. Neural Inf. Process. Syst. (NeurIPS), volume 36, 2023. [^41]: Muchen Li, Sammy Christen, Chengde Wan, Yujun Cai, Renjie Liao, Leonid Sigal, and Shugao Ma. LatentHOI: On the generalizable hand object motion generation with latent hand diffusion. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2025. [^42]: Aditya Prakash, Ruisen Tu, Matthew Chang, and Saurabh Gupta. 3D hand pose estimation in everyday egocentric images. In Proc. Eur. Conf. Comput. Vis. (ECCV), 2024. [^43]: Chenyangguang Zhang, Yan Di, Ruida Zhang, Guangyao Zhai, Fabian Manhardt, Federico Tombari, and Xiangyang Ji. DDF-HO: Hand-held object reconstruction via conditional directed distance field. In Adv. Neural Inf. Process. Syst. (NeurIPS), volume 36, 2023. [^44]: Zhongqun Zhang, Jifei Song, Eduardo Pérez-Pellitero, Yiren Zhou, Hyung Jin Chang, and Aleš Leonardis. Ncrf: neural contact radiance fields for free-viewpoint rendering of hand-object interaction. In Proc. Int. Conf. 3D Vis. (3DV), 2024. [^45]: Ayce Idil Aytekin, Xu Chen, Zhengyang Shen, Thabo Beeler, Helge Rhodin, Rishabh Dabral, and Christian Theobalt. Grasp in gaussians: Fast monocular reconstruction of dynamic hand-object interactions. arXiv preprint arXiv:2604.12929, 2026. [^46]: Enric Corona, Albert Pumarola, Guillem Alenya, Francesc Moreno-Noguer, and Grégory Rogez. Ganhand: Predicting human grasp affordances in multi-object scenes. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2020. [^47]: Hui Zhang, Sammy Christen, Zicong Fan, Luocheng Zheng, Jemin Hwangbo, Jie Song, and Otmar Hilliges. Artigrasp: Physically plausible synthesis of bi-manual dexterous grasping and articulation. In Proc. Int. Conf. 3D Vis. (3DV), 2024. [^48]: Hezhen Hu, Weilun Wang, Wengang Zhou, and Houqiang Li. Hand-object interaction image generation. In Adv. Neural Inf. Process. Syst. (NeurIPS), volume 35, 2022. [^49]: Mengqi Zhang, Yang Fu, Zheng Ding, Sifei Liu, Zhuowen Tu, and Xiaolong Wang. HOIDiffusion: Generating realistic 3D hand-object interaction data. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognition. (CVPR), 2024. [^50]: Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2019. [^51]: Korrawe Karunratanakul, Adrian Spurr, Zicong Fan, Otmar Hilliges, and Siyu Tang. A skeleton-driven neural occupancy representation for articulated hands. In Proc. Int. Conf. 3D Vis. (3DV), 2021. [^52]: Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing scenes as neural radiance fields for view synthesis. In Proc. Eur. Conf. Comput. Vis. (ECCV), 2020. doi: 10.1007/978-3-030-58452-8\24. [^53]: Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3D gaussian splatting for real-time radiance field rendering. ACM Trans. Graph., 42(4), 2023. [^54]: Tze Ho Elden Tse, Zhongqun Zhang, Kwang In Kim, Ales Leonardis, Feng Zheng, and Hyung Jin Chang. S $^{2}$contact: Graph-based network for 3D hand-object contact estimation with semi-supervised learning. In Proc. Eur. Conf. Comput. Vis. (ECCV), 2022. doi: 10.1007/978-3-031-19769-7\_33. [^55]: Juntao Jian, Xiuping Liu, Manyi Li, Ruizhen Hu, and Jian Liu. Affordpose: A large-scale dataset of hand-object interactions with affordance-driven hand pose. In Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), 2023. [^56]: Théo Morales, Omid Taheri, and Gerard Lacey. A versatile and differentiable hand-object interaction representation. In Proc. IEEE/CVF Winter Conf. Appl. Comput. Vis. (WACV), 2025. [^57]: Siddhant Bansal, Zhifan Zhu, Shashank Tripathi, Jiahe Zhao, Michael J. Black, and Dima Damen. Towards in-the-wild egocentric 3D hand-object pose estimation. In Proc. Eur. Conf. Comput. Vis. (ECCV), 2026. ## [^58]: Yunze Liu, Yun Liu, Che Jiang, Kangbo Lyu, Weikang Wan, Hao Shen, Boqiang Liang, Zhoujie Fu, He Wang, and Li Yi. HOI4D: A 4D egocentric --- 用于类别级人机交互的数据集。载于IEEE/CVF计算机视觉与模式识别会议（CVPR）论文集，2022年。 [^59]: Zicong Fan, Omid Taheri, Dimitrios Tzionas, Muhammed Kocabas, Manuel Kaufmann, Michael J Black, and Otmar Hilliges. Arctic: A dataset for dexterous bimanual hand-object manipulation. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2023. [^60]: JoonKyu Park, Yeonguk Oh, Gyeongsik Moon, Hong-suk Choi, and Kyoung Mu Lee. Handoccnet: Occlusion-robust 3D hand mesh estimation network. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2022. [^61]: Xingyu Chen, Yufeng Liu, Yajiao Dong, Xiong Zhang, Chongyang Ma, Yanmin Xiong, Yuan Zhang, and Xiaoyan Guo. Mobrecon: Mobile-friendly hand mesh reconstruction from monocular image. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2022. [^62]: Georgios Pavlakos, Dandan Shan, Ilija Radosavovic, Angjoo Kanazawa, David Fouhey, and Jitendra Malik. Reconstructing hands in 3D with transformers. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2024. [^63]: Rolandos Alexandros Potamias, Jinglei Zhang, Jiankang Deng, and Stefanos Zafeiriou. Wilor: End-to-end 3D hand localization and reconstruction in-the-wild. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2025. [^64]: Zhishan Zhou, Shihao Zhou, Zhi Lv, Minqiang Zou, Yao Tang, and Jiajun Liang. A simple baseline for efficient hand mesh reconstruction. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2024. [^65]: Yujin Chen, Zhigang Tu, Di Kang, Linchao Bao, Ying Zhang, Xuefei Zhe, Ruizhi Chen, and Junsong Yuan. Model-based 3D hand reconstruction via self-supervised learning. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2021. [^66]: Xingyu Liu, Pengfei Ren, Yuanyuan Gao, Jingyu Wang, Haifeng Sun, Qi Qi, Zirui Zhuang, and Jianxin Liao. Keypoint fusion for RGB-D based 3D hand pose estimation. In Proc. AAAI Conf. Artif. Intell., 2024. [^67]: Bardia Doosti, Shujon Naha, Majid Mirbagheri, and David J Crandall. Hope-net: A graph-based model for hand-object pose estimation. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2020. [^68]: Shreyas Hampali, Sayan Deb Sarkar, Mahdi Rad, and Vincent Lepetit. Keypoint transformer: Solving joint identification in challenging hands and object interactions for accurate 3D pose estimation. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2022. [^69]: Ahmed Tawfik Aboukhadra, Jameel Malik, Ahmed Elhayek, Nadia Robertini, and Didier Stricker. Thor-net: End-to-end graformer-based realistic two hands and object reconstruction with self-supervision. In Proc. IEEE/CVF Winter Conf. Appl. Comput. Vis. (WACV), 2023. [^70]: Haozhe Qi, Chen Zhao, Mathieu Salzmann, and Alexander Mathis. HOISDF: Constraining 3D hand-object pose estimation with global signed distance fields. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2024. [^71]: Zehao Zhu, Jiashun Wang, Yuzhe Qin, Deqing Sun, Varun Jampani, and Xiaolong Wang. Contactart: Learning 3D interaction priors for category-level articulated object and hand poses estimation. In Proc. Int. Conf. 3D Vis. (3DV), 2024. [^72]: Zerui Chen, Shizhe Chen, Cordelia Schmid, and Ivan Laptev. gSDF: Geometry-driven signed distance functions for 3D hand-object reconstruction. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2023. [^73]: Kailin Li, Lixin Yang, Haoyu Zhen, Zenan Lin, Xinyu Zhan, Licheng Zhong, Jian Xu, Kejian Wu, and Cewu Lu. Chord: Category-level hand-held object reconstruction via shape deformation. In Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), 2023. [^74]: Di Huang, Xiaopeng Ji, Xingyi He, Jiaming Sun, Tong He, Qing Shuai, Wanli Ouyang, and Xiaowei Zhou. Reconstructing hand-held objects from monocular video. In Proc. ACM SIGGRAPH Asia, 2022. [^75]: Chenyangguang Zhang, Guanlong Jiao, Yan Di, Gu Wang, Ziqin Huang, Ruida Zhang, Fabian Manhardt, Bowen Fu, Federico Tombari, and Xiangyang Ji. Moho: Learning single-view hand-held object reconstruction with multi-view occlusion-aware supervision. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2024. ## [^76]: Alakh Aggarwal, Ningna Wang, and Xiaohu Guo. Texhoi: Reconstructing textures of 3D unknown objects in monocular hand-object interaction scenes. IEEE Trans. Vis. Comput. Graph., 2025 --- [^77]: John Yang, Hyung Jin Chang, Seungeui Lee, and Nojun Kwak. SeqHAND: RGB-sequence-based 3D hand pose and shape estimation. In Proc. Eur. Conf. Comput. Vis. (ECCV), 2020. [^78]: Zhengdi Yu, Stefanos Zafeiriou, and Tolga Birdal. Dyn-hamr: Recovering 4D interacting hand motion from a dynamic camera. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2025. [^79]: Hao Zhang, Zi-Hao Bo, Jun-Hai Yong, and Feng Xu. Interaction fusion: real-time reconstruction of hand poses and deformable objects in hand-object interactions. ACM Trans. Graph., 38(4), 2019. [^80]: Hao Tian, Chenyangguang Zhang, Rui Liu, Wen Shen, and Xiaolin Qin. Interaction-aware 4D gaussian splatting for dynamic hand-object interaction reconstruction. arXiv preprint arXiv:2511.14540, 2025. [^81]: Omid Taheri, Nima Ghorbani, Michael J Black, and Dimitrios Tzionas. Grab: A dataset of whole-body human grasping of objects. In Proc. Eur. Conf. Comput. Vis. (ECCV), 2020. [^82]: Hanwen Jiang, Shaowei Liu, Jiashun Wang, and Xiaolong Wang. Hand-object contact consistency reasoning for human grasps generation. In Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), 2021. [^83]: Shaowei Liu, Yang Zhou, Jimei Yang, Saurabh Gupta, and Shenlong Wang. Contactgen: Generative contact modeling for grasp generation. In Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), 2023. [^84]: Haoming Li, Xinzhuo Lin, Yang Zhou, Xiang Li, Yuchi Huo, Jiming Chen, and Qi Ye. Contact2grasp: 3D grasp synthesis via hand-object contact constraint. In Proc. Int. Joint Conf. Artif. Intell. (IJCAI), 2023. doi: 10.24963/IJCAI.2023/117. [^85]: Yufei Ye, Abhinav Gupta, Kris Kitani, and Shubham Tulsiani. G-hop: Generative hand-object prior for interaction reconstruction and grasp synthesis. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2024. [^86]: Peiming Li, Ziyi Wang, Mengyuan Liu, Hong Liu, and Chen Chen. Clickdiff: click to induce semantic contact map for controllable grasp generation with diffusion models. In Proc. ACM Int. Conf. Multimedia (ACM MM), 2024. [^87]: Xiaofei Wu, Tao Liu, Caoji Li, Yuexin Ma, Yujiao Shi, and Xuming He. Fastgrasp: Efficient grasp synthesis with diffusion. In Proc. Int. Conf. 3D Vis. (3DV), 2025. [^88]: Junuk Cha, Jihyeon Kim, Jae Shin Yoon, and Seungryul Baek. Text2HOI: Text-guided 3D motion generation for hand-object interaction. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2024. [^89]: Wanyue Zhang, Rishabh Dabral, Vladislav Golyanik, Vasileios Choutas, Eduardo Alvarado, Thabo Beeler, Marc Habermann, and Christian Theobalt. Bimart: A unified approach for the synthesis of 3D bimanual interaction with articulated objects. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2025. [^90]: Youxin Pang, Ruizhi Shao, Jiajun Zhang, Hanzhang Tu, Yun Liu, Boyao Zhou, Hongwen Zhang, and Yebin Liu. Manivideo: Generating hand-object manipulation video with dexterous and generalizable grasping. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2025. [^91]: Lingwei Dang, Ruizhi Shao, Hongwen Zhang, Wei Min, Yebin Liu, and Qingyao Wu. Svimo: Synchronized diffusion for video and motion generation in hand-object interaction scenarios. In Adv. Neural Inf. Process. Syst. (NeurIPS), volume 38, 2025. [^92]: Wentian Qu, Jiahe Li, Jian Cheng, Jian Shi, Chenyu Meng, Cuixia Ma, Hongan Wang, Xiaoming Deng, and Yinda Zhang. Hogsa: Bimanual hand-object interaction understanding with 3D gaussian splatting based data augmentation. In Proc. AAAI Conf. Artif. Intell., 2025. [^93]: Keyang Zhou, Bharat Lal Bhatnagar, Jan Eric Lenssen, and Gerard Pons-Moll. Gears: Local geometry-aware hand-object interaction synthesis. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2024. [^94]: Jie Tian, Ran Ji, Lingxiao Yang, Suting Ni, Yuexin Ma, Lan Xu, Jingyi Yu, Ye Shi, and Jingya Wang. Gaze-guided hand-object interaction synthesis: Dataset and method. arXiv preprint arXiv:2403.16169, 2024. [^95]: Aditya Prakash, Benjamin Lundell, Dmitry Andreychuk, David Forsyth, Saurabh Gupta, and Harpreet Sawhney. How do I do that? synthesizing 3D hand motion and contacts for everyday interactions. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2025. ## [^96]: Weiquan Lin, Yaoqing Hu, Liangchen Dai, Xu Tang, and Xingyu Chen. Geohand: Unlocking prior geometry knowledge for monocular 3D hand reconstruction. arXiv preprint arXiv:2605.17354, 2026 --- [^97]: Ruicheng Wang, Sicheng Xu, Yue Dong, Yu Deng, Jianfeng Xiang, Zelong Lv, Guangzhong Sun, Xin Tong, and Jiaolong Yang. MoGe-2: Accurate monocular geometry with metric scale and sharp details. In Adv. Neural Inf. Process. Syst. (NeurIPS), volume 38, 2025. [^98]: Jiale Xu, Weihao Cheng, Yiming Gao, Xintao Wang, Shenghua Gao, and Ying Shan. Instantmesh: Efficient 3D mesh generation from a single image with sparse-view large reconstruction models. arXiv preprint arXiv:2404.07191, 2024. [^99]: Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2024. [^100]: Yufei Ye, Xueting Li, Abhinav Gupta, Shalini De Mello, Stan Birchfield, Jiaming Song, Shubham Tulsiani, and Sifei Liu. Affordance diffusion: Synthesizing hand-object interactions. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2023. [^101]: Ayce Idil Aytekin, Helge Rhodin, Rishabh Dabral, and Christian Theobalt. Follow my hold: Hand-object interaction reconstruction through geometric guidance. In Proc. Int. Conf. 3D Vis. (3DV), 2026. [^102]: Zeqiang Lai, Yunfei Zhao, Haolin Liu, Zibo Zhao, Qingxiang Lin, Huiwen Shi, Xianghui Yang, Mingxin Yang, Shuhui Yang, Yifei Feng, et al. Hunyuan3d 2.5: Towards high-fidelity 3D assets generation with ultimate details. arXiv preprint arXiv:2506.16504, 2025. [^103]: Anilkumar Swamy, Vincent Leroy, Philippe Weinzaepfel, Jean-Sébastien Franco, and Grégory Rogez. Host3r: Keypoint-free hand-object 3D reconstruction from RGB images. In Proc. IEEE/CVF Int. Conf. Comput. Vis. Workshops (ICCVW), 2025. [^104]: Sili Chen, Hengkai Guo, Shengnan Zhu, Feihu Zhang, Zilong Huang, Jiashi Feng, and Bingyi Kang. Video depth anything: Consistent depth estimation for super-long videos. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2025. [^105]: Luigi Piccinelli, Christos Sakaridis, Yung-Hsu Yang, Mattia Segu, Siyuan Li, Wim Abbeloos, and Luc Van Gool. Unidepthv2: Universal monocular metric depth estimation made simpler. IEEE Trans. Pattern Anal. Mach. Intell., 2025. [^106]: Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Rong-hang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr、Roman Rädle、Chloe Rolland、Laura Gustafson等。SAM 2：图像与视频中的任意物体分割。载于国际学习表征会议（ICLR）论文集，2025年。 [^107]: Ahmed Tawfik Aboukhadra, Marcel Rogge, Nadia Robertini, Abdalla Arafa, Jameel Malik, Ahmed Elhayek, and Didier Stricker. Ghost: Fast category-agnostic hand-object interaction reconstruction from RGB videos using gaussian splatting. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2026. [^108]: Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2024. [^109]: Jinglei Zhang, Jiankang Deng, Chao Ma, and Rolandos Alexandros Potamias. Hawor: World-space hand motion reconstruction from egocentric videos. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2025. [^110]: Wei Yin, Chi Zhang, Hao Chen, Zhipeng Cai, Gang Yu, Kaixuan Wang, Xiaozhi Chen, and Chun Shen. Metric3d: Towards zero-shot metric 3D prediction from a single image. In Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), 2023. [^111]: Xingyu Chen, Fu-Jen Chu, Pierre Gleize, Kevin J Liang, Alexander Sax, Hao Tang, Weiyao Wang, Michelle Guo, Thibaut Hardin, Xiang Li, et al. SAM 3D: 3Dfy anything in images. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2026. [^112]: Baicheng Li, Dong Wu, Jun Li, Shunkai Zhou, Zecui Zeng, Lusong Li, and Hongbin Zha. MV-SAM3D: Adaptive multi-view fusion for layout-aware 3D generation. arXiv preprint arXiv:2603.11633, 2026. [^113]: Nicolas Carion, Laura Gustafson, Yuan-Ting Hu, Shoubhik Debnath, Ronghang Hu, Didac Suris, Chaitanya Ryali, Kalyan Vasudev Alwala, Haitham Khedr, Andrew Huang, et al. SAM 3: Segment anything with concepts. arXiv preprint arXiv:2511.16719, 2025. [^114]: Haotong Lin, Sili Chen, Junhao Liew, Donny Y Chen, Zhenyu Li, Guang Shi, Jiashi Feng, and Bingyi Kang. Depth anything 3: Recovering the visual space from any views. arXiv preprint arXiv:2511.10647, 2025. ## [^115]: Xingyu Chen, Zhuheng Song, Xiaoke Jiang, Yaoqing Hu, Junzhi Yu, and Lei Zhang. Handos: 3D hand --- 单阶段重建技术。载于IEEE/CVF计算机视觉与模式识别会议（CVPR）论文集，2025年。 [^116]: Hao Xu, Yilin Liu, Yinqiao Wang, Chi-Wing Fu, and Niloy J Mitra. Choir: Contact-aware 4D hand-object interaction reconstruction. arXiv preprint arXiv:2605.20992, 2026. [^117]: Kaihua Chen, Deva Ramanan, and Tarasha Khurana. Using diffusion priors for video amodal segmentation. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2025. [^118]: Xiaofei Wu, Yi Zhang, Yumeng Liu, Yuexin Ma, Yujiao Shi, and Xuming He. Affordgrasp: Cross-modal diffusion for affordance-aware grasp synthesis. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2026. [^119]: Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019. [^120]: An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Xuejing Liu, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, and Zhihao Fan. Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2024. [^121]: Zhongqun Zhang, Hengfei Wang, Ziwei Yu, Yihua Cheng, Angela Yao, and Hyung Jin Chang. NL2Contact: Natural language guided 3D hand-object contact modeling with diffusion model. In Proc. Eur. Conf. Comput. Vis. (ECCV), 2024. [^122]: OpenAI. ChatGPT, 2022. URL . Accessed: 2026-03-24. [^123]: Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proc. Conf. North Amer. Chapter Assoc. Comput. Linguistics: Human Lang. Technol. (NAACL-HLT), 2019. [^124]: Mingzhen Huang, Fu-Jen Chu, Bugra Tekin, Kevin J Liang, Haoyu Ma, Weiyao Wang, Xingyu Chen, Pierre Gleize, Hongfei Xue, Siwei Lyu, et al. HOIGPT: Learning long-sequence hand-object interaction with language models. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2025. [^125]: Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. [^126]: Zhenhao Zhang, Ye Shi, Lingxiao Yang, Suting Ni, Qi Ye, and Jingya Wang. OpenHOI: Open-world hand-object interaction synthesis with multimodal large language model. In Adv. Neural Inf. Process. Syst. (NeurIPS), volume 38, 2025. [^127]: Zekun Qi, Runpei Dong, Shaochen Zhang, Haoran Geng, Chunrui Han, Zheng Ge, Li Yi, and Kaisheng Ma. ShapeLLM: Universal 3D object understanding for embodied interaction. In Proc. Eur. Conf. Comput. Vis. (ECCV), 2024. [^128]: Yuantao Chen, Jiahao Chang, Chongjie Ye, Chaoran Zhang, Zhaojie Fang, Chenghong Li, and Xiaoguang Han. Forehoi: Feed-forward 3D object reconstruction from daily hand-object interaction videos. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2026. [^129]: Shibo Wang, Haonan He, Maria Parelli, Christoph Gebhardt, Zicong Fan, and Jie Song. MagicHOI: Leveraging 3D priors for accurate hand-object reconstruction from short monocular video clips. In Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), 2025. [^130]: Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero-1-to-3: Zero-shot one image to 3D object. In Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), 2023. [^131]: Jeongwan On, Kyeonghwan Gwak, Gunyoung Kang, Junuk Cha, Soohyun Hwang, Hyein Hwang, and Seungryul Baek. Bigs: Bimanual category-agnostic interaction reconstruction from monocular videos via 3D gaussian splatting. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2025. ## [^132]: Haixin He et al. DreamHand: Repurposing video diffusion models for occlusion-robust egocentric 3D hand motion recovery. arXiv preprint arXiv:2608.20308, 2026 --- [^133]: Wan Team, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, et al. Wan: Open and advanced large-scale video generative models. arXiv preprint arXiv:2503.20314, 2025. [^134]: Kevin Yuanbo Wu, Tianxing Zhou, Isaac Tu, Billy Yan, Irmak Guzey, David Fouhey, Dandan Shan, and Lerrel Pinto. Human universal grasping. arXiv preprint arXiv:2606.17054, 2026. [^135]: Alexey Gavryushin, Alexandros Delitzas, Luc Van Gool, Marc Pollefeys, Kaichun Mo, and Xi Wang. SIGHT: synthesizing image-text conditioned and geometry-guided 3d hand-object trajectories. arXiv preprint arXiv:2503.22869, 2025. [^136]: Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21(140), 2020. [^137]: Yingying Fan, Quanwei Yang, Kaisiyuan Wang, Hang Zhou, Yingying Li, Haocheng Feng, Errui Ding, Yu Wu, and Jingdong Wang. Re-hold: Video hand object interaction reenactment via adaptive layout-instructed diffusion model. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2025. [^138]: Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models. In Proc. Int. Conf. Mach. Learn. (ICML), 2022. [^139]: Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. GPT-4 technical report. arXiv preprint arXiv:2303.08774, 2023. [^140]: Luma AI. Genie: Text-to-3D generation. , 2024. Accessed: Dec. 15, 2024. [^141]: OpenAI. GPT-4V(ision) system card. Technical report, OpenAI, 2023. URL . Accessed: Dec. 15, 2024. [^142]: Shijian Jiang, Qi Ye, Rengan Xie, Yuchi Huo, and Jiming Chen. Hand-held object reconstruction from RGB video with dynamic interaction. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2025. [^143]: Jin-Chuan Shi, Binhong Ye, Tao Liu, Junzhe He, Yangjinhui Xu, Xiaoyang Liu, Zeju Li, Hao Chen, and Chunhua Shen. Agile: Hand-object interaction reconstruction from video via agentic generation. arXiv preprint arXiv:2602.04672, 2026. [^144]: Jianyuan Wang, Minghao Chen, Nikita Karaev, Andrea Vedaldi, Christian Rupprecht, and David Novotny. VGGT: Visual geometry grounded transformer. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2025. [^145]: Qianqian Wang, Yifei Zhang, Aleksander Holynski, Alexei A. Efros, and Angjoo Kanazawa. Continuous 3D perception model with persistent state. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2025. [^146]: Yumeng Liu, Xiao-Xiao Long, Marc Habermann, Xuanze Yang, Cheng Lin, Yuan Liu, Yuexin Ma, Wenping Wang, and Ligang Liu. Hggt: Robust and flexible 3D hand mesh reconstruction from uncalibrated images. arXiv preprint arXiv:2603.23997, 2026. [^147]: Hongming Fu, Wenjia Wang, Xiaozhen Qiao, Rollandos Alexandros Potamias, Taku Komura, Shuo Yang, Zheng Liu, and Bo Zhao. Eggrasp: World-space hand-object interaction estimation from egocentric videos. arXiv preprint arXiv:2601.01050, 2026. [^148]: Wendi Hu, Haonan Zhou, Wenhao Hu, and Gaoang Wang. Hand3r: Online 4D hand-scene reconstruction in the wild. arXiv preprint arXiv:2602.03200, 2026. [^149]: Ruitao Jing, Xingyu Chen, Hongyang Li, Qing Jiang, Yukai Shi, and Lei Zhang. ScaleHP: Estimating hand pose in metric space. arXiv preprint arXiv:2606.25619, 2026. [^150]: Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023. [^151]: Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Adv. Neural Inf. Process. Syst. (NeurIPS), volume 36, 2023. [^152]: Xiaoyun Chang and Yi Sun. Text2grasp: Synthesis of grasps by text prompts for object grasping parts. In Proc. Int. Symp. Neural Netw. (ISNN), 2025. ## [^153]: Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In Adv. Neural Inf. Process. Syst. (NeurIPS), volume 33, 2020 --- [^154]: Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing GPT-4 with 90%* ChatGPT quality, March 2023. URL . [^155]: Haosheng Li, Weixin Mao, Weipeng Deng, Chenyu Meng, Haoqiang Fan, Tiancai Wang, Yoshie Osamu, Ping Tan, Hongan Wang, and Xiaoming Deng. Multi-graspllm: A multimodal LLM for multi-hand semantic guided grasp generation. arXiv preprint arXiv:2412.08468, 2024. [^156]: Juntao Jian, Xiuping Liu, Zixuan Chen, Manyi Li, Jian Liu, and Ruizhen Hu. G-dexgrasp: Generalizable dexterous grasping synthesis via part-aware prior retrieval and prior-assisted generation. In Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), 2025. [^157]: OpenAI. GPT-4o system card. Technical report, OpenAI, 2024. URL . [^158]: Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. Grounded language-image pre-training. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2022. [^159]: Yi-Lin Wei, Mu Lin, Yuhao Lin, Jian-Jian Jiang, Xiao-Ming Wu, Ling-An Zeng, and Wei-Shi Zheng. Afford-dexgrasp: Open-set language-guided dexterous grasp with generalizable-instructive affordance. In Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), 2025. [^160]: Sammy Christen, Shreyas Hampali, Fadime Sener, Edoardo Remelli, Tomas Hodan, Eric Sauser, Shugao Ma, and Bugra Tekin. Diff $^{2}$o: Diffusion-based synthesis of hand-object interactions from textual descriptions. In Proc. ACM SIGGRAPH Asia, 2024. [^161]: Mingyeong Song, Jungbin Cho, Jisoo Kim, Ananya Bal, Kartik Sharma, Youngjae Yu, Laszlo A. Jeni, and Junhyug Noh. Jointly generating contact maps enhances hand object interaction generation. arXiv preprint arXiv:2607.01768, 2026. [^162]: Zhi Wang, Liu Liu, Ruonan Liu, Dan Guo, and Meng Wang. Structibhoi: Structured articulation modeling for long-horizon bimanual hand-object interaction generation. arXiv preprint arXiv:2603.08390, 2026. [^163]: Zhi Wang, Yuyan Liu, Liu Liu, Li Zhang, Ruixuan Lu, and Dan Guo. Synhlma: Synthesizing hand language 针对关节物体的离散式人机交互表示下的操控技术。arXiv预印本arXiv:2510.25268，2025年。 [^164]: Guangyi Han, Wei Zhai, Yuhang Yang, Yang Cao, and Zheng-Jun Zha. Touch: Text-guided controllable generation of free-form hand-object interactions. arXiv preprint arXiv:2510.14874, 2025. [^165]: Bohan Zhou, Yi Zhan, Zhongbin Zhang, and Zongqing Lu. Megohand: Multimodal egocentric hand-object interaction motion generation. In Adv. Neural Inf. Process. Syst. (NeurIPS), volume 38, 2025. [^166]: Zhiqi Li, Guo Chen, Shilong Liu, Shihao Wang, VS Vibashan, Yishen Ji, Shiyi Lan, Hao Zhang, Yilin Zhao, Subhashree Radhakrishnan, et al. Eagle 2: Building post-training data strategies from scratch for frontier vision-language models. arXiv preprint arXiv:2501.14818, 2025. [^167]: Daniel Sungho Jung and Kyoung Mu Lee. Training-free dense hand contact estimation with multi-modal large language models. arXiv preprint arXiv:2605.05886, 2026. [^168]: Naru Suzuki, Takehiko Ohkawa, Tatsuro Banno, Jihyun Lee, Ryosuke Furuta, and Yoichi Sato. Affordance-guided diffusion prior for 3D hand reconstruction. arXiv preprint arXiv:2510.00506, 2025. [^169]: Zerui Chen, Rolandos Alexandros Potamias, Shizhe Chen, and Cordelia Schmid. Hort: Monocular handheld objects reconstruction with transformers. In Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), 2025. [^170]: Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. In Proc. Int. Conf. Learn. Represent. (ICLR), 2024. [^171]: Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), 2023. [^172]: Yufei Ye, Poorvi Hebbar, Abhinav Gupta, and Shubham Tulsiani. Diffusion-guided reconstruction of everyday hand-object interaction clips. In Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), 2023. ## [^173]: Zhongqun Zhang, Yihua Cheng, Eduardo Pérez-Pellitero, Yiren Zhou, Jiankang Deng, Hyung Jin Chang, and Jifei Song. Single-view image to novel-view generation for hand-object interactions. In Proc. AAAI Conf. Artif. Intell., 2025 --- [^174]: Matt Deitke, Ruoshi Liu, Matthew Wallingford, Huong Ngo, Oscar Michel, Aditya Kusupati, Alan Fan, Christian Laforte, Vikram Voleti, Samir Yitzhak Gadre, et al. Objaverse-XL: A universe of 10m+ 3D objects. In Adv. Neural Inf. Process. Syst. (NeurIPS), Datasets and Benchmarks Track, 2023. [^175]: Gurusha Juneja and Sukrit Kumar. Prompt-propose-verify: A reliable hand-object-interaction data gener-ation framework using foundational models. arXiv preprint arXiv:2312.15247, 2023. [^176]: Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2023. [^177]: Haozhuo Zhang, Bin Zhu, Yu Cao, and Yanbin Hao. Hand1000: Generating realistic hands from text with only 1,000 images. In Proc. AAAI Conf. Artif. Intell., 2025. [^178]: Chengrui Wang, Pengfei Liu, Min Zhou, Ming Zeng, Xubin Li, Tiezheng Ge, and Bo Zheng. Rhands: Refining malformed hands for generated images with decoupled structure and style guidance. In Proc. AAAI Conf. Artif. Intell., 2025. [^179]: Junho Park, Kyeongbo Kong, and Suk-Ju Kang. Attentionhand: Text-driven controllable hand image generation for 3D hand reconstruction in the wild. In Proc. Eur. Conf. Comput. Vis. (ECCV), 2024. [^180]: Jinbo Xing, Menghan Xia, Yong Zhang, Haoxin Chen, Wangbo Yu, Hanyuan Liu, Gongye Liu, Xintao Wang, Ying Shan, and Tien-Tsin Wong. Dynamicrafter: Animating open-domain images with video diffusion priors. In Proc. Eur. Conf. Comput. Vis. (ECCV), 2024. [^181]: Black Forest Labs, Stephen Batifol, Andreas Blattmann, Frederic Boesel, Saksham Consul, Cyril Diagne, Tim Dockhorn, Jack English, Zion English, Patrick Esser, Sumith Kulal, Kyle Lacey, Yam Levi, Cheng Li, Dominik Lorenz, Jonas Müller, Dustin Podell, Robin Rombach, Harry Saini, Axel Sauer, and Luke Smith. FLUX.1 context: Flow matching for in-context image generation and editing in latent space. arXiv preprint arXiv:2506.15742, 2025. [^182]: Hongxiang Zhao, Xingchen Liu, Mutian Xu, Yiming Hao, Weikai Chen, and Xiaoguang Han. Taste-rob: 推进面向任务的手-物交互视频生成，以实现通用的机器人操控。载于IEEE/CVF计算机视觉与模式识别会议（CVPR）论文集，2025年。 [^183]: Zhelun Shen, Chenming Wu, Junsheng Zhou, Chen Zhao, Kaisiyuan Wang, Hang Zhou, Yingying Li, Haocheng Feng, Wei He, and Jingdong Wang. iditHOI: Inpainting-based hand object interaction reenactment via video diffusion transformer. arXiv preprint arXiv:2506.12847, 2025. [^184]: Haodong Yan, Hang Yu, Zhide Zhong, Weilin Yuan, Xin Gong, Zehang Luo, Chengxi Heyu, Junfeng Li, Wenxuan Song, Shunbo Zhou, et al. Open-world hand-object interaction video generation based on structure and contact-aware representation. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2026. [^185]: Mingjin Chen, Junhao Chen, Zhaoxin Fan, Yujian Lee, Zichen Dang, Lili Wang, Yawen Cui, Lap-Pui Chau, and Yi Wang. Hvg-3D: Bridging real and simulation domains for 3D-conditional hand-object interaction video synthesis. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2026. [^186]: Mingze Gao, Kai Yang, Hongbo Gao, Bo Li, Aoxiang Ding, Wenxiang Li, Yujie Yu, Jian Liu, Shugong Xu, Yi Niu, Haoyu Chi, He Chen, Hui Tang, Li Yi, and Hao Zhao. PAM: A pose-appearance-motion engine for sim-to-real HOI video generation. arXiv preprint arXiv:2603.22193, 2026. [^187]: Dayou Li, Lulin Liu, Bangya Liu, Shijie Zhou, Jiu Feng, Ziqi Lu, Minghui Zheng, Chenyu You, and Zhiwen Fan. Egocentric world model for photorealistic hand-object interaction synthesis. arXiv preprint arXiv:2603.13615, 2026. [^188]: Yuxi Wang, Wenqi Ouyang, Tianyi Wei, Yi Dong, Zhiqi Shen, and Xingang Pan. Hand2world: Autoregressive egocentric interaction generation via free-space hand gestures. arXiv preprint arXiv:2602.09600, 2026. [^189]: Linxi Xie, Lisong C Sun, Ashley Neall, Tong Wu, Shengqu Cai, and Gordon Wetzstein. Generated reality: Human-centric world simulation using interactive video generation with hand and camera control. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2026. ## [^190]: Byungjun Kim, Taeksoo Kim, Junyoung Lee, and Hanbyul Joo. Dexterous world models. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2026 --- [^191]: Yushuo Chen, Xiaoyu Shi, Xiaoshi Wu, Xintao Wang, Pengfei Wan, and Yebin Liu. Handsonworld: Unconstrained egocentric video generation with camera-disentangled hand control. arXiv preprint arXiv:2607.02075, 2026. [^192]: Yangtao Chen, Zixuan Chen, Peiyang Wang, YongLu Li, Jing Huo, Jieqi Shi, and Yang Gao. Wh0: Generative world models as scalable sources of egocentric human hand manipulation data. arXiv preprint arXiv:2606.22136, 2026. [^193]: Yi Chen, Yuying Ge, Weiliang Tang, Yizhuo Li, Yixiao Ge, Mingyu Ding, Ying Shan, and Xihui Liu. Moto: Latent motion token as the bridging language for learning robot manipulation from videos. In Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), 2025. [^194]: Yucheng Hu, Yanjiang Guo, Pengchao Wang, Xiaoyu Chen, Yen-Jen Wang, Jianke Zhang, Koushil Sreenath, Chaochao Lu, and Jianyu Chen. Video prediction policy: A generalist robot policy with predictive visual representations. arXiv preprint arXiv:2412.14803, 2024. [^195]: Xiaoyu Chen, Junliang Guo, Tianyu He, Chuheng Zhang, Pushi Zhang, Derek Cathera Yang, Li Zhao, and Jiang Bian. Igor: Image-goal representations are the atomic control units for foundation models in embodied ai. arXiv preprint arXiv:2411.00785, 2024. [^196]: Xiongyi Cai, Ri-Zhao Qiu, Geng Chen, Lai Wei, Isabella Liu, Tianshu Huang, Xuxin Cheng, and Xiaolong Wang. In-n-on: Scaling egocentric manipulation with in-the-wild and on-task data. arXiv preprint arXiv:2511.15704, 2025. [^197]: Simar Kareer, Karl Pertsch, James Darpinian, Judy Hoffman, Danfei Xu, Sergey Levine, Chelsea Finn, and Suraj Nair. Emergence of human to robot transfer in vision-language-action models. arXiv preprint arXiv:2512.22414, 2025. [^198]: Chubin Zhang, Jianan Wang, Zifeng Gao, Yue Su, Tianru Dai, Cai Zhou, Jiwen Lu, and Yansong Tang. Clap: Contrastive latent action pretraining for learning vision-language-action models from human videos. arXiv preprint arXiv:2601.04061, 2026. [^199]: Qingwen Bu, Yanting Yang, Jisong Cai, Shenyuan Gao, Guanghui Ren, Maoqing Yao, Ping Luo, and Hongyang Li. Univla: Learning to act anywhere with task-centric latent actions. arXiv preprint arXiv:2505.06111, 2025. [^200]: Xiaoyu Chen, Hangxing Wei, Pushi Zhang, Chuheng Zhang, Kaixin Wang, Yanjiang Guo, Rushuai Yang, 王宇辰、肖新全、赵丽等。Villa-x：增强视觉-语言-动作模型中的潜在动作建模。arXiv预印本arXiv:2507.23682，2025年。 [^201]: Hongtao Wu, Ya Jing, Chilam Cheang, Guangzeng Chen, Jiafeng Xu, Xinghang Li, Minghuan Liu, Hang Li, and Tao Kong. Unleashing large-scale video generative pre-training for visual robot manipulation. In Proc. Int. Conf. Learn. Represent. (ICLR), 2024. [^202]: Chi-Lam Cheang, Guangzeng Chen, Ya Jing, Tao Kong, Hang Li, Yifeng Li, Yuxiao Liu, Hongtao Wu, Jiafeng Xu, Yichu Yang, et al. Gr-2: A generative video-language-action model with web-scale knowledge for robot manipulation. arXiv preprint arXiv:2410.06158, 2024. [^203]: Johan Bjorck, Fernando Castañeda, Nikita Cherniadev, Xingye Da, Runyu Ding, Linxi Fan, Yu Fang, Dieter Fox, Fengyuan Hu, Spencer Huang, et al. GR00T n1: An open foundation model for generalist humanoid robots. arXiv preprint arXiv:2503.14734, 2025. [^204]: Hongzhe Bi, Hengkai Tan, Shenghao Xie, Zeyuan Wang, Shuhe Huang, Haitian Liu, Ruowen Zhao, Yao Feng, Chendong Xiang, Yinze Rong, et al. Motus: A unified latent action world model. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2026. [^205]: Ruijie Zheng, Dantong Niu, Yuqi Xie, Jing Wang, Mengda Xu, Yunfan Jiang, Fernando Castañeda, Fengyuan Hu, You Liang Tan, Letian Fu, et al. Egoscale: Scaling dexterous manipulation with diverse egocentric human data. arXiv preprint arXiv:2602.16710, 2026. [^206]: Hao Luo, Yicheng Feng, Wanpeng Zhang, Sipeng Zheng, Ye Wang, Haoqi Yuan, Jiazheng Liu, Chaoyi Xu, Qin Jin, and Zongqing Lu. Being-h0: vision-language-action pretraining from large-scale human videos. arXiv preprint arXiv:2507.15597, 2025. [^207]: Qixiu Li, Yu Deng, Yaobo Liang, Lin Luo, Lei Zhou, Chengtang Yao, Lingqi Zeng, Zhiyuan Feng, Huizhi Liang, Sicheng Xu, et al. Scalable vision-language-action model pretraining for robotic manipulation with real-life human activity videos. arXiv preprint arXiv:2510.21571, 2025. ## [^208]: Tomoya Yoshida, Shuhei Kurita, Taichi Nishimura, and Shinsuke Mori. Developing vision-language-action model from egocentric videos. arXiv preprint arXiv:2509.21986, 2025 --- [^209]: Hongzhe Bi, Lingxuan Wu, Tianwei Lin, Hengkai Tan, Zhizhong Su, Hang Su, and Jun Zhu. H-rdt: Human manipulation enhanced bimanual robotic manipulation. In Proc. AAAI Conf. Artif. Intell., 2026. [^210]: Zhenhao Zhang, Jiaxin Liu, Ye Shi, and Jingya Wang. Unihm: Unified dexterous hand manipulation with vision language model. arXiv preprint arXiv:2603.00732, 2026. [^211]: Hongyi Chen, Yunchao Yao, Yufei Ye, Zhixuan Xu, Homanga Bharadhwaj, Jiashun Wang, Shubham Tulsiani, Zackory Erickson, and Jeffrey Ichnowski. Web2grasp: Learning functional grasps from web images of hand-object interactions. arXiv preprint arXiv:2505.05517, 2025. [^212]: Chao Tang, Jiacheng Xu, Haofei Lu, Bolin Zou, Wenlong Dong, Hong Zhang, and Danica Kragic. Grasp as you dream: Imitating functional grasping from generated human demonstrations. arXiv preprint arXiv:2604.07517, 2026. [^213]: Kailin Li, Puhao Li, Tengyu Liu, Yuyang Li, and Siyuan Huang. Maniptrans: Efficient dexterous bimanual manipulation transfer via residual learning. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2025. [^214]: Zhao Mandi, Yifan Hou, Dieter Fox, Yashraj Narang, Ajay Mandlekar, and Shuran Song. Dexmachina: Functional retargeting for bimanual dexterous manipulation. arXiv preprint arXiv:2505.24853, 2025. [^215]: Xinghao Zhu, Zixi Liu, Shalin Jain, Chenran Li, Milad Noori, Huihua Zhao, John Welsh, Michael Andres Lin, Wei Liu, Tingwu Wang, et al. Learning dexterous manipulation using contact wrench guidance from human demonstration. arXiv preprint arXiv:2607.00033, 2026. [^216]: Simar Kareer, Dhruv Patel, Ryan Punamiya, Pranay Mathur, Shuo Cheng, Chen Wang, Judy Hoffman, and Danfei Xu. Egomimic: Scaling imitation learning via egocentric video. In Proc. IEEE Int. Conf. Robot. Autom. (ICRA), 2025. [^217]: Mengda Xu, Han Zhang, Yifan Hou, Zhenjia Xu, Linxi Fan, Manuela Veloso, and Shuran Song. Dexumi: Using human hand as the universal manipulation interface for dexterous manipulation. arXiv preprint arXiv:2505.21864, 2025. [^218]: Yuanpei Chen, Chen Wang, Yaodong Yang, and C Karen Liu. Object-centric dexterous manipulation from human motion data. arXiv preprint arXiv:2411.04005, 2024. [^219]: Hongyi Chen, Tony Dong, Tiancheng Wu, Liquan Wang, Yash Jangir, Yaru Niu, Yufei Ye, Homanga Bharadhwaj, Zackory Erickson, and Jeffrey Ichnowski. Dexterous manipulation policies from RGB human videos via 3D hand-object trajectory reconstruction. arXiv preprint arXiv:2602.09013, 2026. [^220]: Juncheng Mu, Sizhe Yang, Yiming Bao, Hojin Bae, Tianming Wei, Linning Xu, Boyi Li, Huazhe Xu, and Jiangmiao Pang. Deximit: Learning bimanual dexterous manipulation from monocular human videos. arXiv preprint arXiv:2602.10105, 2026. [^221]: Marion Lepert, Jiaying Fang, and Jeannette Bohg. Masquerade: Learning from in-the-wild human videos using data-editing. arXiv preprint arXiv:2508.09976, 2025. [^222]: Huayi Zhou, Ruixiang Wang, Yunxin Tai, Yueci Deng, Guiliang Liu, and Kui Jia. You only teach once: Learn one-shot bimanual robotic manipulation from video demonstrations. arXiv preprint arXiv:2501.14208, 2025. [^223]: Ruixiang Wang, Huayi Zhou, Xinyue Yao, Guiliang Liu, and Kui Jia. Gat-grasp: Gesture-driven affordance transfer for task-aware robotic grasping. In Proc. IEEE/RSJ Int. Conf. Intell. Robots Syst. (IROS), 2025. [^224]: Chuan Wen, Xingyu Lin, John Ian Reyes So, Kai Chen, Qi Dou, Yang Gao, and Pieter Abbeel. Any-point trajectory modeling for policy learning. In Proc. Robot.: Sci. Syst. (RSS), 2024. doi: 10.15607/RSS.2024.XX.092. [^225]: Hanzhi Chen, Boyang Sun, Anran Zhang, Marc Pollefeys, and Stefan Leutenegger. Vidbot: Learning generalizable 3D actions from in-the-wild 2D human videos for zero-shot robotic manipulation. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2025. [^226]: Huajian Zeng, Lingyun Chen, Jiaqi Yang, Yuantai Zhang, Fan Shi, Peidong Liu, and Xingxing Zuo. Flowhoi: Flow-based semantics-grounded generation of hand-object interactions for dexterous robot manipulation. arXiv preprint arXiv:2602.13444, 2026. [^227]: Mengda Xu, Zhenjia Xu, Yinghao Xu, Cheng Chi, Gordon Wetzstein, Manuela Veloso, and Shuran Song. Flow as the cross-domain manipulation interface. arXiv preprint arXiv:2407.15208, 2024. ## [^228]: Hongyan Zhi, Peihao Chen, Siyuan Zhou, Yubo Dong, Quanxi Wu, Lei Han, and Mingkui Tan --- 3DFlowAction：基于3D流世界模型学习跨本体操控能力。arXiv预印本arXiv:2506.06199，2025年。 [^229]: Hongyu Li, Lingfeng Sun, Yafei Hu, Duy Ta, Jennifer Barry, George Konidaris, and Jiahui Fu. NovaFlow: Zero-shot manipulation via actionable flow from generated videos. arXiv preprint arXiv:2510.08568, 2025. [^230]: Karthik Dharmarajan, Wenlong Huang, Jiajun Wu, Li Fei-Fei, and Ruohan Zhang. Dream2flow: Bridging video generation and open-world manipulation with 3D object flow. arXiv preprint arXiv:2512.24766, 2025. [^231]: Adam Hung, Bardienus Pieter Duisterhof, and Jeffrey Ichnowski. 3PoinTr: 3D point tracks for learning manipulation from unconstrained human videos. arXiv preprint arXiv:2603.08485, 2026. [^232]: Yuxuan Kuang, Sungjae Park, Katerina Fragkiadaki, and Shubham Tulsiani. Dex4D: Task-agnostic point track policy for sim-to-real dexterous manipulation. arXiv preprint arXiv:2602.15828, 2026. [^233]: Homanga Bharadhwaj, Debidatta Dwibedi, Abhinav Gupta, Shubham Tulsiani, Carl Doersch, Ted Xiao, Dhruv Shah, Fei Xia, Dorsa Sadigh, and Sean Kirmani. Gen2act: Human video generation in novel scenarios enables generalizable robot manipulation. arXiv preprint arXiv:2409.16283, 2024. [^234]: Dan Kondratyuk, Lijun Yu, Xiuye Gu, José Lezama, Jonathan Huang, Grant Schindler, Rachel Hornung, Vighnesh Birodkar, Jimmy Yan, Ming-Chang Chiu, et al. Videopoet: A large language model for zero-shot video generation. arXiv preprint arXiv:2312.14125, 2023. [^235]: Yukang Cao, Haozhe Xie, Beichen Wen, Runmao Yao, Yinghao Liu, Yue Huang, Zhichao Liao, Yunxiang Wang, Haiheng Liu, Xingshun Tian, Dawei Su, Long Zhuo, Dacheng Tao, Xiaogang Wang, Liang Pan, and Ziwei Liu. ACE-Data-0: Human-centric ambient capture as embodied data engine. arXiv preprint arXiv:2607.28625, 2026. [^236]: Yuhong Zhang, Zihan Gao, Shengpeng Li, Ling-Hao Chen, Kaisheng Liu, Runqing Cheng, Xiao Lin, Junjia Liu, Zhuoheng Li, Jingyi Feng, et al. Robowheel: A data engine from real-world human demonstrations for cross-embodiment robotic learning. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2026. [^237]: Ye Wang, Pei Lin, Xiong-Hui Chen, Haoqi Yuan, Zhixuan Liang, Yiyang Huang, Anzhe Chen, Zixing Lei, Tao Zhang, Huaguang Wang, Long Zhang, Chenxi Xiao, Ziyuan Jiao, and Qin Jin. Ego2Robot: Scalable robot data synthesis from egocentric human data. arXiv preprint arXiv:2608.02580, 2026. [^238]: Zhenjie Yang, Xingyu Jiao, Guopeng Zhong, Shuzhe Yang, Shi Che, Chao Wu, Chenyu Jiang, Dongjie Zhang, Yideng Zhang, Zheng Zhang, et al. HandEdit: A unified benchmark for egocentric human-to-robot dexterous hand image editing. arXiv preprint arXiv:2608.12122, 2026. [^239]: Zeyi Liu et al. RoboEdit: Turning human manipulation videos into scalable robot experience. arXiv preprint arXiv:2608.18948, 2026. [^240]: Bhawna Paliwal, Haritheja Etukuru, William Liang, Pieter Abbeel, Nur Muhammad Mahi Shafiullah, and Jitendra Malik. Do as i do: Dexterous manipulation data from everyday human videos. arXiv preprint arXiv:2606.19333, 2026. [^241]: Gaotian Wang, Kejia Ren, Andrew Morgan, Yiting Chen, Howard H Qian, Podshara Chanrungmaneekul, and Kaiyu Hang. Egoinfinity: A web-scale 4D hand-object interaction data engine for any-view robot re-targeting and video-to-action robot learning. arXiv preprint arXiv:2606.17385, 2026. [^242]: Sicheng Xie, Haidong Cao, Zejia Weng, Zhen Xing, Haoran Chen, Shiwei Shen, Jiaqi Leng, Zuxuan Wu, and Yu-Gang Jiang. Human2robot: Learning robot actions from paired human-robot videos. In Proc. AAAI Conf. Artif. Intell., 2026. [^243]: Seungjae Lee, Yoonkyo Jung, Inkook Chun, Yao-Chih Lee, Zikui Cai, Hongjia Huang, Aayush Talreja, Tan Dat Dao, Yongyuan Liang, Jia-Bin Huang, and Furong Huang. Tracegen: World modeling in 3D trace-space enables learning from cross-embodiment videos. arXiv preprint arXiv:2511.21690, 2025. [^244]: Hai Ci, Xiaokang Liu, Pei Yang, Yiren Song, and Mike Zheng Shou. H2r-grounder: A paired-data-free paradigm for translating human interaction videos into physically grounded robot videos. arXiv preprint arXiv:2512.09406, 2025. ## [^245]: Haoqi Yuan, Zhixuan Liang, Anzhe Chen, Ye Wang, Haoyang Li, Pei Lin, Yiyang Huang, Zixing Lei, Tong Zhang, JiaZhao Zhang, et al. Qwen-robotmanip technical report: Alignment unlocks scale for robotic manipulation foundation models. arXiv preprint arXiv:2606.17846, 2026 --- [^246]: Srinath Sridhar, Franziska Mueller, Michael Zollhoefer, Dan Casas, Antti Oulasvirta, and Christian Theobalt. Real-time joint tracking of a hand manipulating an object from RGB-D input. In Proc. Eur. Conf. Comput. Vis. (ECCV), 2016. [^247]: Franziska Mueller, Dushyant Mehta, Oleksandr Sotnychenko, Srinath Sridhar, Dan Casas, and Christian Theobalt. Real-time hand tracking under occlusion from an egocentric RGB-D sensor. In Proc. IEEE Int. Conf. Comput. Vis. (ICCV), 2017. [^248]: Guillermo Garcia-Hernando, Shanxin Yuan, Seungryul Baek, and Tae-Kyun Kim. First-person hand action benchmark with RGB-D videos and 3D hand pose annotations. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2018. [^249]: Christian Zimmermann, Duygu Ceylan, Jimei Yang, Bryan Russell, Max Argus, and Thomas Brox. Freihand: A dataset for markerless capture of hand pose and shape from single RGB images. In Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), 2019. [^250]: Samarth Brahmbhatt, Cush Ham, Charles C Kemp, and James Hays. Contactdb: Analyzing and predicting grasp contact via thermal imaging. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2019. [^251]: Shreyas Hampali, Sayan Deb Sarkar, and Vincent Lepetit. Ho-3D_v3: Improving the accuracy of hand-object annotations of the HO-3D dataset. arXiv preprint arXiv:2107.00887, 2021. [^252]: Gyeongsik Moon, Shoou-I Yu, He Wen, Takaaki Shiratori, and Kyoung Mu Lee. Interhand2.6m: A dataset and baseline for 3D interacting hand pose estimation from a single RGB image. In Proc. Eur. Conf. Comput. Vis. (ECCV), 2020. doi: 10.1007/978-3-030-58565-5\_33. [^253]: Zhe Cao, Ilija Radosavovic, Angjoo Kanazawa, and Jitendra Malik. Reconstructing hand-object interactions in the wild. In Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), 2021. [^254]: Yu-Wei Chao, Wei Yang, Yu Xiang, Pavlo Molchanov, Ankur Handa, Jonathan Tremblay, Yashraj S Narang, Karl Van Wyk, Umar Iqbal, Stan Birchfield, et al. Dexycb: A benchmark for capturing hand grasping of objects. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2021. [^255]: Taein Kwon, Bugra Tekin, Jan Stühmer, Federica Bogo, and Marc Pollefeys. H2o: Two hands manipulating objects for first person interaction recognition. 发表于IEEE/CVF国际计算机视觉会议（ICCV），2021年。 [^256]: Lixin Yang, Kailin Li, Xinyu Zhan, Fei Wu, Anran Xu, Liu Liu, and Cewu Lu. Oakink: A large-scale knowledge repository for understanding hand-object interaction. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2022. [^257]: Takehiko Ohkawa, Kun He, Fadime Sener, Tomas Hodan, Luan Tran, and Cem Keskin. Assembly-hands: Towards egocentric activity understanding via 3D hand pose estimation. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2023. [^258]: Anilkumar Swamy, Vincent Leroy, Philippe Weinzaepfel, Fabien Baradel, Salma Galaaoui, Romain Brégier, Matthieu Armando, Jean-Sebastien Franco, and Grégory Rogez. Showme: Benchmarking object-agnostic hand-object 3D reconstruction. In Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), 2023. [^259]: Woojin Cho, Jihyun Lee, Minjae Yi, Minje Kim, Taeyun Woo, Donghwan Kim, Taewook Ha, Hyokeun Lee, Je-Hwan Ryu, Woontack Woo, et al. Dense hand-object (ho) graspnet with full grasping taxonomy and dynamics. In Proc. Eur. Conf. Comput. Vis. (ECCV), 2024. [^260]: Xinyu Zhan, Lixin Yang, Yifei Zhao, Kangrui Mao, Hanlin Xu, Zenan Lin, Kailin Li, and Cewu Lu. Oakink2: A dataset of bimanual hands-object manipulation in complex task completion. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2024. [^261]: Yun Liu, Haolin Yang, Xu Si, Ling Liu, Zipeng Li, Yuxiang Zhang, Yebin Liu, and Li Yi. Taco: Benchmarking generalizable bimanual tool-action-object understanding. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2024. [^262]: Rao Fu, Dingxi Zhang, Alex Jiang, Wanjia Fu, Austin Funk, Daniel Ritchie, and Srinath Sridhar. Gigahands: A massive annotated dataset of bimanual hand activities. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2025. [^263]: Jikai Wang, Qifan Zhang, Yu-Wei Chao, Bowen Wen, Xiaohu Guo, and Yu Xiang. Ho-cap: A capture system and dataset for 3D reconstruction and pose tracking of hand-object interaction. In Adv. Neural Inf. Process. Syst. (NeurIPS), volume 38, 2025. ## [^264]: Zhenjun Yu, Wenqiang Xu, Pengfei Xie, Yutong Li, Brian W. Anthony, Zhuorui Zhang, and Cewu Lu. Dynamic reconstruction of hand-object interaction --- 采用分布式力感知接触表征技术。发表于IEEE/CVF国际计算机视觉会议（ICCV），2025年。 [^265]: Prithviraj Banerjee, Sindi Shkodrani, Pierre Moulon, Shreyas Hampali, Shangchen Han, Fan Zhang, Linguang Zhang, Jade Fountain, Edward Miller, Selen Basol, et al. Hot3d: Hand and object tracking in 3D from egocentric multi-view videos. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2025. [^266]: Ziyi Kou, Ankit Kumar, Mia Huang, Taylor Niehues, Vatsal Mehta, Ergys Ristani, and Li Guan. AVI-HT: Adaptive vision-IMU fusion for 3D hand tracking. arXiv preprint arXiv:2605.21714, 2026. [^267]: Patrick Rim, Kevin Harris, Braden Copple, Shangchen Han, Xu Xie, Ivan Shugurov, Sizhe An, He Wen, Alex Wong, Tomas Hodan, et al. Show3d: Capturing scenes of 3D hands and objects in the wild. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2026. [^268]: Zimu Zhang, Yucheng Zhang, Xiyan Xu, Ziyin Wang, Sirui Xu, Kai Zhou, Bing Zhou, Chuan Guo, Jian Wang, Yu-Xiong Wang, et al. Handx: Scaling bimanual motion and interaction generation. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2026. [^269]: Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), 2015. doi: 10.1109/CVPR.2015.7298698. [^270]: Gunnar A Sigurdsson, Gül Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev, and Abhinav Gupta. Hollywood in homes: Crowdsourcing data collection for activity understanding. In Proc. Eur. Conf. Comput. Vis. (ECCV), 2016. [^271]: Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint arXiv:1705.06950, 2017. [^272]: Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fründ, Peter Yianilos, Moritz Mueller-Freitag, Florian Hoppe, Christian Thurau, Ingo Bax, and Roland Memisevic. The 用于学习和评估视觉常识的“某物某物”视频数据库。发表于IEEE国际计算机视觉会议（ICCV），2017年。doi:10.1109/ICCV.2017.622。 [^273]: Chunhui Gu, Chen Sun, David A Ross, Carl Vondrick, Caroline Pantofaru, Yeqing Li, Sudheendra Vijayanarasimhan, George Toderici, Susanna Ricco, Rahul Sukthankar, et al. Ava: A video dataset of spatio-temporally localized atomic visual actions. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2018. [^274]: Hang Zhao, Antonio Torralba, Lorenzo Torresani, and Zhicheng Yan. Hacs: Human action clips and segments dataset for recognition and temporal localization. In Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), 2019. [^275]: Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. How to 100m: Learning a text-video embedding by watching hundred million narrated video clips. In Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), 2019. [^276]: Dian Shao, Yue Zhao, Bo Dai, and Dahua Lin. Fine-gym: A hierarchical video dataset for fine-grained action understanding. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2020. [^277]: Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. The epic-kitchens dataset: Collection, challenges and baselines. IEEE Trans. Pattern Anal. Mach. Intell., 43(11), 2021. [^278]: Fadime Sener, Dibyadip Chatterjee, Daniel Shelepov, Kun He, Dipika Singhania, Robert Wang, and Angela Yao. Assembly101: A large-scale multi-view video dataset for understanding procedural activities. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2022. [^279]: Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2022. ## [^280]: Xin Wang, Taein Kwon, Mahdi Rad, Bowen Pan, Ishani Chakraborty, Sean Andrist, Dan Bohus, Ashley Feniello, Bugra Tekin, Felipe Vieira Frujeri, et al. Holoassist: an egocentric human interaction dataset --- 面向现实世界中交互式AI助手的应用。发表于IEEE/CVF国际计算机视觉会议（ICCV），2023年。 [^281]: Kristen Grauman, Andrew Westbury, Lorenzo Torresani, Kris Kitani, Jitendra Malik, Triantafyllos Afouras, Kumar Ashutosh, Vijay Baiyya, Siddhant Bansal, Bikram Boote, et al. Ego-exo4d: Understanding skilled human activity from first-and third-person perspectives. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2024. [^282]: Ryan Hoque, Peide Huang, David J Yoon, Mouli Sivapurapu, and Jian Zhang. Egodex: Learning dexterous manipulation from large-scale egocentric video. arXiv preprint arXiv:2505.11709, 2025. [^283]: Ahad Jawaid and Yu Xiang. Openego: A large-scale multimodal egocentric dataset for dexterous manipulation. arXiv preprint arXiv:2509.05513, 2025. [^284]: Yihang Li, Xuelong Wei, Jingzhou Luo, Yingjing Xiao, Yibo Bai, Guangyuan Zhou, Teng Zou, Chenguang Gui, Jiajun Wen, He Zhang, et al. Egolive: A large-scale egocentric dataset from real-world human tasks. arXiv preprint arXiv:2604.23570, 2026. [^285]: Ryan Punamiya, Simar Kareer, Zeyi Liu, Josh Citron, Ri-Zhao Qiu, Xiongyi Cai, Alexey Gavryushin, Jiaqi Chen, Davide Liconti, Lawrence Y Zhu, et al. Egoverse: An egocentric human dataset for robot learning from around the world. arXiv preprint arXiv:2604.07607, 2026. [^286]: Yufan Deng and Daquan Zhou. Humannet: Scaling human-centric video learning to one million hours. arXiv preprint arXiv:2605.06747, 2026. [^287]: Eadom Dessalene, Botao He, Michael Maynord, Yonatan Tussa, Pavan Mantripragada, Yianni Karabati, Nirupam Roy, and Yiannis Aloimonos. FEEL (force-enhanced egocentric learning): A dataset for physical action understanding. arXiv preprint arXiv:2603.15847, 2026. [^288]: Zishuo Li, Bowen Yang, Changtao Miao, Kai Zhu, Hao Chen, Qingze Guan, Zhengxing Wu, Wanke Zhan, Yang Sun, Zhiyi Huang, et al. OpenAoE: An open egocentric manipulation dataset and toolchain for embodied learning. arXiv preprint arXiv:2607.14183, 2026. [^289]: Catalin Ionescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu. Human3.6m: Large scale datasets and predictive methods for 3D human sensing in natural environments. IEEE Trans. Pattern Anal. Mach. Intell., 36(7), 2014. doi: 10.1109/TPAMI.2013.248. [^290]: Stefan Hinterstoisser, Vincent Lepetit, Slobodan Ilic, Stefan Holzer, Gary Bradski, Kurt Konolige, and Nassir Navab. Model based training, detection and pose estimation of texture-less 3D objects in heavily cluttered scenes. In Proc. Asian Conf. Comput. Vis. (ACCV), 2012. [^291]: Yu Xiang, Tanner Schmidt, Venkatraman Narayanan, and Dieter Fox. PoseCNN: A convolutional neural network for 6D object pose estimation in cluttered scenes. arXiv preprint arXiv:1711.00199, 2017. [^292]: Haoqiang Fan, Hao Su, and Leonidas J Guibas. A point set generation network for 3D object reconstruction from a single image. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2017. [^293]: Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by a two time-scale update rule converge to a local nash equilibrium. In Adv. Neural Inf. Process. Syst. (NeurIPS), 2017. [^294]: Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: A new metric & challenges. arXiv preprint arXiv:1812.01717, 2018. [^295]: Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process., 13(4), 2004. [^296]: Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2018.