的可供性而言，对应的基本操作“切割”就要求执行者移动刀刃。
---
Figure 2. Illustration of the complex task acquisition process. This figure uses a Complex Task: 'Prepare a bowl of hot sweet fruit tea.' to demonstrate the process. Initially, the annotators (◇) analyze the affordances of four essential objects (a gripper, a knife, a tea bottle, and a microwave oven) and design corresponding Primitive. For instance, to prepare fruit slices, the Primitive: cut associated with the knife blade is required. Following this, an expert (◇) arranges the scene for the Complex Task, and then the subject (△), utilizing the designed Primitive, plans the execution path of the Complex Task. Later, these execution paths are structured into a Primitive Dependency Graphs.
刀刃必须完全穿过被切割的物体，这样才能使被切割的部分分离。在这一阶段，我们会收集所有现有的物体可供性及其对应的基本操作，从而形成一个基本任务池（见图2的第三列）。
#### 3.1.3 复杂任务分解
在第三阶段，我们将此前提出的复杂任务——这类任务具有时间跨度长、操控目标多等特点——分解为一系列短期且单目标的基本操作。由于复杂任务的完成依赖于基本操作的顺序，我们的方法还会明确各基本操作之间的依赖关系。因此，每个复杂任务都包含一系列基本操作，以及用于描述这些基本操作层级执行顺序的基本操作依赖图（PDG）。层级为0（L0）的基本操作是独立的，无需先完成其他基本操作；而最高层级的基本操作则负责让复杂任务得以完成。
我们采用专门的协议来获取这种分解结果及依赖关系。如图2的第四列所示，首先由专家（$\underline{\text{⚡}}$）设定场景和目标的具体描述；随后指示受试者（$\underline{\text{⚡}}$）利用任务池中的现有基本操作来描述完成任务的顺序；最后由专家记录并整理这一序列，形成基本操作依赖图，从而完成复杂任务的获取流程。
#### 3.2 数据采集与标注
获取这三个层级的抽象概念之后，受试者需分别在数据采集平台上完成基本任务和复杂任务（见图3）。
Figure 3. Capture platform. 12 MoCap cameras are circled in blue and 4 RGB cameras in red.
#### 3.2.1 采集设置
数据采集平台包含两个主要组件：用于记录操控过程的多摄像头系统，以及用于姿态追踪的光学动作捕捉系统。动作捕捉系统使用12台Optitrack Prime 13W红外摄像头，来追踪固定在受试者上半身、左右手以及交互物体上的表面标记点。多摄像头系统则由4台商用RGB摄像头组成，其中3台为客观视角，1台为主观视角。我们将所有传感器的采样率同步为30帧/秒，并校准两个系统之间的坐标转换关系。
#### 3.2.2 数据标注
物体姿态。刚体的姿态可直接通过动作捕捉系统求解。对于关节体的姿态，其基础部分的处理方式与刚体类似，而关节部分则分为两类。如果某部件足够大，能够附着足够多的标记点且不会阻碍交互，那么它就会像刚体一样被处理；否则，仅在该部件上附着一个标记点。该标记点的位置是在物体的标准坐标系中进行校准的。随后，根据关节类型（如旋转或平移），通过最小化观测到的标记点位置与在物体标准坐标系中恢复出的标记点位置之间的平方差，来确定关节参数。
---
Figure 4. Commentary of the task execution. The left column shows the current state of the scene. The center column shows the narrative dialog retrieved from experts. The right column shows the upcoming Primitive task to be executed.
如果物体不会阻碍交互，那么它将被当作刚体来处理。否则，只会在该部位附加一个标记点。该标记点的位置是在物体的标准坐标系中进行校准的。随后，根据关节类型（例如旋转或平移），通过最小化观测到的标记点位置与物体在标准坐标系中恢复出的标记点位置之间的平方差，来确定关节参数。
人体姿态与表面。对人体姿态和表面的标注依赖于SMPL-X [^50] 人体网格模型。为了获取人体姿态和表面信息，我们采用了与MoSH++ [^44] 一致的两阶段拟合方法。在第一阶段，我们利用受试者处于T型姿势时采集到的标记点数据，来拟合受试者的SMPL-X形状参数 $\bar{\beta}$ 以及每个标记点在SMPL-X标准空间中的位置 $P_{\mathcal{M}}^{(c)}$。根据第一阶段的优化结果，我们可以确定受试者表面标记点与SMPL-X模型顶点之间的对应关系 $\mathcal{C}(\cdot)$。在第二阶段，我们基于已获得的形状参数 $\theta$ 和标记点对应关系 $\bar{\beta}$，拟合任务执行过程中每一帧的受试者姿态参数 $\mathcal{C}(\cdot)$。获得姿态和形状参数后，便可利用SMPL-X模型重建受试者的身体网格。其他如MANO等身体表示形式也是由此结果衍生而来。详情请参阅补充材料。
任务执行评述。操作过程完成后，我们会将视频记录发送给专家进行分析，请他们针对任务执行过程提供详细的评述。在每个基本步骤中，都会要求专家对当前任务状态及后续动作发表评论。具体而言，基于前一步骤的执行情况，专家需要：1）结合当前场景以及即将执行的基本步骤，总结为实现操作目标尚需完成的任务；2）利用可用的基本操作来描述下一步动作。这一过程如图4所示。专家提供的叙述文本随后会经由GPT-4 [^48] 进行润色，以形成评述内容。OAKINK2数据集便是以这些评述为基础构建的，因为它们体现了专家观察操作过程时的思维脉络。这些评述不仅能用于解读用户行为，还能为生成用户动作提供依据。
### 4. OAKINK2数据集
#### 4.1. 数据与标注列表
OAKINK2提供了记录操作过程的RGB视频。这些视频采集自多视角（1个自我中心视角和3个非自我中心视角）设置，以30帧/秒的速率同步，分辨率为$848 \times 480$。注释包含两部分：1）3D运动信息，包括交互过程中人体上半身、手部以及物体的姿态和形状（含关节参数）；2）任务规范，包括物体可供性、与这些可供性对应的基本动作、带有任务目标、初始条件、PDG、专家评论以及受试者完成顺序的复杂任务。3D注释质量的评估结果详见补充材料。关于3D手部关键点的注释会利用重建模型进行跨数据集验证，而与抓取动作相关的3D姿态则会针对其物理属性完整性进行检测。
#### 4.2. 数据集统计
OAKINK2设置了四种手-物交互场景，共包含38个长时程复杂操作目标，这些目标被具体化为150个复杂任务。OAKINK2总共包含75个物体和39种可供性，这些可供性对应60种类型的基本动作。OAKINK2总计包含627段双手灵巧手-物交互序列，其中363段属于基本动作，264段属于复杂任务。总体而言，OAKINK2包含401万帧图像。我们在表4中将OAKINK2与多个现有的手-物交互数据集进行了比较。在此我们重点介绍OAKINK2的几个显著特点：1）它提供了基于物体可供性的交互数据（与HO3D、DexYCB相比）；2）它具有长时程操作目标（与ARCTIC、HOI4D、GRAB相比）；3）它为手部和物体提供了3D姿态和形状注释（与EGO4D、AssemblyHands相比）；4）它提供了基于基本动作的任务分解功能，这是表4中任何数据集都不具备的。
---
| Dataset | image mod. | resolution | #frame | #views | #subj | #obj | 3D | real / syn. | label method | hand pose | obj pose | afford. inter. | dynamic inter. | long- | horizon | task |
| gnd. | syn. | method |
| EGO4D [^19] | ✓ | ~ | ~ | 1 | 931 | - | ✗ | - | - | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ | ✓ |
| HO3D [^21] | ✓ | 640 \times 480 | 78K | 1-5 | 10 | 10 | ✓ | real | auto | ✓ | ✓ | ✗ | ✓ | ✗ | ✗ | ✗ |
| GRAB [^56] | ✗ | - | 1.62M | - | 10 | 51 | ✓ | real | mocap | ✓ | ✓ | ✓ | ✓ | ✗ | ✗ | ✗ |
| H2O [^34] | ✓ | 1280 \times 720 | 571K | 5 | 4 | 8 | ✓ | real | auto | ✓ | ✓ | ✓ | ✓ | ✗ | ✗ | ✗ |
| HOI4D [^40] | ✓ | 1280 \times 800 | 3M | 1 | 9 | 1000 | ✓ | real | crowd | ✓ | ✓ | ✓ | ✓ | ✗ | ✗ | ✗ |
| ARCTIC [^15] | ✓ | 2800 \times 2000 | 2.1M | 9 | 10 | 11 | ✓ | real | mocap | ✓ | ✓ | ✓ | ✓ | ✗ | ✗ | ✗ |
| AssemblyHands [^47] | ✓ | 1920 \times 1080 | 3.03M | 12 | 34 | - | ✓ | real | semi-auto | ✓ | ✗ | ✓ | ✓ | ✓ | ✓ | ✓ |
| Ego-Exo4D [^20] | ✓ | ~ | ~ | 5-6 | 839 | - | ✓ | real | semi-auto | ✓ | ✗ | ✗ | ✓ | ✓ | ✓ | ✓ |
| OakInk-Image [^67] | ✓ | 848 \times 480 | 230K | 4 | 12 | 100 | ✓ | real | crowd | ✓ | ✓ | ✓ | ✓ | ✗ | ✗ | ✗ |
| OAKINK2 | ✓ | 848 \times 480 | 4.01M | 4 | 9 | 75 | ✓ | real | mocap | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
Table 1. A cross-comparison among various public datasets. (Refer to Sup. Mat for the full table.)
### 5. 典型应用
#### 5.1. 手部网格重建
手部网格重建（HMR）任务旨在根据采集到的图像估计交互过程中的3D手部姿态。我们在单视角和多视角两种设置下对HMR任务进行基准测试。在单视角设置中，输入图像仅包含一个视角，即自我中心视角或非自我中心视角。在多视角设置中，输入图像将包含多个视角，同时附带相机校准参数。对于这两种设置，我们都在序列层面划分了对应的任务指定子集，以保持训练集、验证集和测试集中样本的比例约为70%、5%和25%。
我们评估了世界坐标系下的平均关节位置误差（MPJPE）、平均顶点位置误差（MPVPE），以及腕部（根部）相对（RR）系统和经过Procrustes分析（PA）后的系统中的误差。我们还评估了根部相对系统中0–20毫米范围内正确关键点百分比的曲线下面积（AUC）。我们在表2中展示了两种设置下的HMR基准测试结果。
| Setting | Methods | PA-MPJPE | PA-MPVPE | RR-MPJPE (AUC) | RR-MPVPE | MPJPE | MPVPE |
| Mono | METRO [^38] | 6.90 | 6.47 | 17.56 (0.410) | 16.44 | - | - |
| RLE [^35] | 5.46 | 6.86 | 13.08 (0.441) | 14.03 | - | - |
| + HandTailer [^42] |
| Multi | KP-based Fit [^68] | 9.20 | 8.83 | 15.63 (0.349) | 15.38 | 19.30 | 19.11 |
| POEM [^68] | 6.18 | 6.61 | 12.12 (0.581) | 12.15 | 9.17 | 9.52 |
Table 2. Single- and multi-view HMR evaluation results in mm.
#### 5.2. 任务感知运动实现（TaMF）
为了在交互场景中实现任务目标，我们提出了一种新任务：任务感知运动实现（TaMF）。该任务旨在生成能够完成给定物体轨迹的手部运动序列，且这些序列需符合文本形式的任务描述要求。
任务表述。给定一段关于……的文本描述
基本任务：$\text{text}_{\text{PT}}$，我们假设所涉及的物体
其几何形状$\mathcal{V}_o = \{\mathbf{V}_{o,m}\}$以及运动轨迹
$\mathcal{T}_o = \{\mathbf{T}_{o,m}^{(i)}\}$在交互过程中是已知的。我们
用下标h表示人手，o表示物体，
物体，m 用于索引不同的物体实例（以及不同的
同一实例的不同部分），并使用上标 $(i)$ 来索引不同的
时间戳。任务是生成相应的
手部运动轨迹 $\mathcal{P}_h = \{\mathbf{P}_h^{(0:L)}\}$，以
文本描述 $\text{text}_{\text{PT}}$、物体几何 $\mathcal{V}_o$ 以及
运动轨迹 $\mathcal{T}_o$。
评估指标：我们采用接触率（CR）和实体相交体积（SIV）来衡量生成动作的物理合理性。在序列层面，我们参照人体动作生成领域的方法，用关节功率谱KL散度（PSKL-J）来评估动作平滑度；同时用FID来衡量真实动作与生成动作之间的距离。我们还开展了感知研究，以评估生成动作的真实感水平。这些指标的详细定义可在补充材料中找到。
Figure 5. Architecture of MF-MDM. First sample random noises $\boldsymbol{x}_{T}$; then at each step iterating from T to 1, MF-MDM G predicts the cleaned sample $\hat{\boldsymbol{x}}_{0}$ and then diffuse it back to $\boldsymbol{x}_{t-1}$. After the generated sample $\boldsymbol{x}_{0}$ is acquired, it is refined by MF-MDM R for better interaction details.
---
Figure 6. Qualitative Visualization of the generated hand motion in TaMF model.
| Physical lausibility | Motion Smoothness |
| CR $\uparrow$ | SIV ( $cm^{{3}}$) $\downarrow$ | PSKL-J (g.t., p.) $\downarrow$ | (p., g.t.) $\downarrow$ |
| 0.90 | 4.17 | 0.0446 | 0.0460 |
| FID | Perceptual Score |
| Dataset | Generated |
| 1.369 | 4.66 $\pm$ 0.48 | 3.64 $\pm$ 0.85 |
Table 3. Evaluations of generated hand motion in TaMF model. PSKL-J is evaluated between the training data (g.t.) and the generated hand motion trajectory (p.); both directions are included as PSKL-J is an asymmetric metric.
模型与结果：我们对基于扩散的动作生成模型MDM [^59]进行了改进，使其能适配任务感知型手部动作合成的精细需求。模型架构如图5所示。我们提出的模型名为MF-MDM，包含两个部分：1）MF-MDM G，它根据任务的文本描述及物体动作轨迹生成人体动作轨迹；2）MF-MDM R，它根据手部与物体的空间关系优化生成的手部动作。采样过程被建模为逐步去除噪声样本的逆扩散过程。MF-MDM的核心创新在于将多物体相关的概率条件融入现有的Transformer编码器。为此，我们引入了一个额外层——顺序合并层，用于在每个时间帧汇总交互场景中的空间关系。物体动作轨迹和此前生成的手部动作轨迹会被投影到同一维度后再进行合并。对于优化模型MF-MDM R，我们会将手部与物体的距离作为额外空间信息输入顺序合并层。合并后的嵌入序列会与其它标记一起被送入主Transformer编码器：包括加噪步骤标记、来自CLIP文本编码器的任务描述文本嵌入，以及来自PointBert编码器的聚合物体几何嵌入。我们在表3中提供了定量评估结果，图6则展示了定性可视化结果。
#### 5.3. 复杂任务完成（CTC）
OAKINK2引入了一项新应用——将复杂任务目标拆解为一系列基本动作路径。复杂任务完成（CTC）是指根据场景的文本描述及任务目标生成手部动作轨迹。由于从复杂任务和场景文本直接转化为端到端动作生成涉及多模态间的转换，目前尚无成熟的框架来解决这一问题。因此，我们将CTC分解为三个阶段，依次进行处理。
该过程最初从基于文本的1) 基础规划开始。近期在基础模型方面的突破——如大型语言模型（LLMs）[^48] [^69]，使我们能够将其用作任务规划器，因为这些模型本身便具备规划基础执行路径的能力，仅需适当的引导与上下文即可。此阶段的输出是一份任务规划脚本，其中包含了每个基础步骤的执行顺序。随后，根据目标任务和场景状态，将问题重新定义为为每个基础步骤生成手部与物体的运动轨迹，从而构建出$P(\mathcal{P}_h, \mathcal{T}_o|\text{text}_{PT})$。我们进一步将其拆解为两个子任务：2) 物体轨迹检索，即$P(\mathcal{T}_o|\text{text}_{PT})$；以及3) 手部运动生成，即$P(\mathcal{P}_h|\mathcal{T}_o|\text{text}_{PT})$。前者通过将专家演示中的$^1$物体运动重新定位，以适应新生成的随机场景；后者则采用我们预先定义的任务感知运动实现模型（TaMF，见第5.2节）。
① 利用LLM进行基础规划。在此阶段，我们利用现成的GPT-4模型[^48]来生成程序，将复杂任务分解为一系列基础步骤。我们首先依据手动设计的模板，将场景描述$\text{text}_{scene}$、复杂任务描述$\text{text}_{goal}$以及每个物体的描述$\{\text{text}_{obj}\}$嵌入提示词中。GPT-4会根据该提示词生成相应程序。如Fig. 8的代码块所示，该程序通过一系列代码片段构建基础步骤依赖图（PDG），其中PDG的每个节点（即基础步骤）被实现为$\text{execute}$([primitive], ...)函数，而PDG的边则体现为函数的调用顺序。接着，我们利用基于OAKINK2中PDG信息构建的依赖检查器，来验证生成的程序能否在不违反约束条件的情况下完成复杂任务。一旦获得成功的程序，便进入下一阶段。
---
Figure 7. Visualization of Motion Generation Outcome in Complex Task Completion.
Figure 8. The diagram of Complex Task Completion. The task input populates a predefined template to generate the prompt for planning. The ① LLM (GPT-4) responds with code of the program's execution path, delineating the DAG for Primitive dependency. Within the code response block, the $\text{orange}$ snippets marks the ② Oracle to re-target object trajectories; the $\text{blue}$ snippets indicate ③ motion generators for Primitives.
此时，Fig. 8代码块中的$\text{execute}()$函数尚未完整，缺少两个关键组件：$\text{object\_trajectory}$与手部运动生成器。我们将在接下来的两个阶段中解决这些问题。
② 从“Oracle”中检索物体轨迹。完成一个基础任务需要该场景下物体的运动轨迹。在此阶段，我们利用“Oracle”根据特定场景和基础步骤来检索物体运动轨迹。“Oracle”具备双重功能：1) 根据给定的基础步骤，从OAKINK2数据集中获取物体运动轨迹；2) 根据物体的初始姿态、功能姿态及最终姿态，对这些专家生成的轨迹进行重新定位，以满足新场景的需求并生成所需的物体轨迹。
③ 利用TaMF生成手部运动。一旦获得物体轨迹，最后一步便是为每个基础步骤生成手部运动轨迹。为此，我们采用先前设计的任务感知运动实现模型（TaMF，见第5.2节）作为通用生成器（意味着单个TaMF模型可适用于所有基础步骤）。在将所有$\text{execute}()$函数填入确定的物体轨迹与生成器后，程序便会按顺序执行，各基础步骤的轨迹会通过插值法连接起来。这种插值方式通过将前一个轨迹的最终状态与后一个轨迹的初始状态相连，确保了运动的平滑过渡。
我们在图7中展示了复杂任务的生成动作示例。关于测试场景生成、提示词与模板的细节，以及基本规划评估、成功/失败案例等内容，请参阅补充材料。
### 6. 未来工作
OAKINK2是一个包含多种手-物体交互的数据集，用于支持人类完成长时程、多目标复杂操控任务。OAKINK2收录了基本动作演示——即满足物体功能需求的最小交互单元，以及复杂任务演示，后者还包含将其拆解为相互依赖的基本动作的内容。
首先，我们希望OAKINK2能够支撑大规模语言-操控预训练，提升多模态（如视觉-语言-动作 [^69]）模型在复杂任务完成方面的性能。从更长远来看，我们期望OAKINK2能助力开发能够实现端到端文本到操控生成的学习框架。
其次，通过将收集到的基本动作演示重新适配到不同的实体上，OAKINK2能够赋能各类具身操控任务，比如文中提到的异质手部与平台 [^23] [^53] [^54] [^61] [^64]。OAKINK2中构建的交互场景也可以被迁移并整合到现有的仿真环境 [^45] [^60] 中，以支持物体操控领域的具身学习。
---
## 参考文献
[^1]: Ahmed Tawfik Aboukhadra, Jameel Malik, Ahmed Elhayek, Nadia Robertini, and Didier Stricker. THOR-Net: End-to-end graformer-based realistic two hands and object reconstruction with self-supervision. In Winter Conference on Applications of Computer Vision (WACV), 2023. 2
[^2]: Suneel Belkhale, Tianli Ding, Ted Xiao, Pierre Sermanet, Quon Vuong, Jonathan Tompson, Yevgen Chebotar, Debidatta Dwibedi, and Dorsa Sadigh. RT-H: Action hierarchies using language. arXiv preprint arXiv:2403.01823, 2024. 3
[^3]: Samarth Brahmbhatt, Chengcheng Tang, Christopher D Twigg, Charles C Kemp, and James Hays. ContactPose: A dataset of grasps with object contact and hand pose. In European Conference on Computer Vision (ECCV), 2020. 2, 3, 15
[^4]: Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, Pete Florence, Chuyuan Fu, Montse Gonzalez Arenas, Keerthana Gopalakrishnan, Kehang Han, Karol Hausman, Alex Herzog, Jasmine Hsu, Brian Ichter, Alex Irpan, Nikhil Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Isabel Leal, Lisa Lee, Tsang-Wei Edward Lee, Sergey Levine, Yao Lu, Henryk Michalewski, Igor Mordatch, Karl Pertsch, Kanishka Rao, Krista Reymann, Michael Ryoo, Grecia Salazar, Pannag Sanketi, Pierre Sermanet, Jaspiar Singh, Anikait Singh, Radu Soricut, Huong Tran, Vincent Vanhoucke, Quan Vuong, Ayzaan Wahid, Stefan Welker, Paul Wohlhart, Jialin Wu, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, and Brianna Zitkovich. RT-2: Vision-language-action models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.15818, 2023. 3
[^5]: Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Tomas Jackson, Sally Jesmonth, Nikhil Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Isabel Leal, Kuang-Huei Lee, Sergey Levine, Yao Lu, Utsav Malla, Deeksha Manjunath, Igor Mordatch, Ofir Nachum, Carolina Parada, Jodilyn Peralta, Emily Perez, Karl Pertsch, Jornell Quiamboo, Kanishka Rao, Michael Ryoo, Grecia Salazar, Pannag Sanketi, Kevin Sayed, Jaspiar Singh, Sumedh Sontakke, Austin Stone, Clayton Tan, Huong Tran, Vincent Vanhoucke, Steve Vega, Quan Vuong, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, and Brianna Zitkovich. RT-1: Robotics transformer for real-world control at scale. Robotics: Science and Systems (RSS), 2023. 3
[^6]: Anthony Brohan, Yevgen Chebotar, Chelsea Finn, Karol Hausman, Alexander Herzog, Daniel Ho, Julian Ibarz, Alex Irpan, Eric Jang, Ryan Julian, et al. Do as i can, not as i say: Grounding language in robotic affordances. In Conference on Robot Learning (CoRL), 2023. 3
[^7]: Zhe Cao, Ilija Radosavovic, Angjoo Kanazawa, and Jitendra Malik. Reconstructing hand-object interactions in the wild.
发表于2021年国际计算机视觉会议（ICCV）。2
[^8]: Yu-Wei Chao, Wei Yang, Yu Xiang, Pavlo Molchanov, Ankur Handa, Jonathan Tremblay, Yashraj S. Narang, Karl Van Wyk, Umar Iqbal, Stan Birchfield, Jan Kautz, and Dieter Fox. DexYCB: A benchmark for capturing hand grasping of objects. In Computer Vision and Pattern Recognition (CVPR), 2021. 2, 3, 15
[^9]: Yuanpei Chen, Chen Wang, Li Fei-Fei, and C Karen Liu. Sequential dexterity: Chaining dexterous policies for long-horizon manipulation. arXiv preprint arXiv:2309.00987, 2023. 3
[^10]: Shuo Cheng and Danfei Xu. LEAGUE: Guided skill learning and abstraction for long-horizon manipulation. IEEE Robotics and Automation Letters, 2023. 3
## [^11]: Open X-Embodiment Collaboration, Abby O'Neill, Abdul Rehman, Abhiram Maddukuri, Abhishek Gupta, Abhishek Padalkar, Abraham Lee, Acorn Pooley, Agrim Gupta, Ajay Mandlekar, Ajinkya Jain, Albert Tung, Alex Bewley, Alex Herzog, Alex Irpan, Alexander Khazatsky, Anant Rai, Anchit Gupta, Andrew Wang, Anikait Singh, Animesh Garg, Aniruddha Kembhavi, Annie Xie, Anthony Brohan, Antonin Raffin, Archit Sharma, Arefeh Yavary, Arhan Jain, Ashwin Balakrishna, Ayzaan Wahid, Ben Burgess-Limerick, Beomjoon Kim, Bernhard Schölkopf, Blake Wulfe, Brian Ichter, Cewu Lu, Charles Xu, Charlotte Le, Chelsea Finn, Chen Wang, Chenfeng Xu, Cheng Chi, Chenguang Huang, Christine Chan, Christopher Agia, Chuer Pan, Chuyuan Fu, Coline Devin, Danfei Xu, Daniel Morton, Danny Driess, Daphne Chen, Deepak Pathak, Dhruv Shah, Dieter Büchler, Dinesh Jayaraman, Dmitry Kalashnikov, Dorsa Sadigh, Edward Johns, Ethan Foster, Fangchen Liu, Federico Ceola, Fei Xia, Feiyu Zhao, Freek Stulp, Gaoyue Zhou, Gaurav S. Sukhatme, Gautam Salhotra, Ge Yan, Gilbert Feng, Giulio Schiavi, Glen Berseth, Gregory Kahn, Guanzhi Wang, Hao Su, Hao-Shu Fang, Haochen Shi, Henghui Bao, Heni Ben Amor, Henrik I Christensen, Hiroki Furuta, Homer Walke, Hongjie Fang, Huy Ha, Igor Mordatch, Ilija Radosavovic, Isabel Leal, Jacky Liang, Jad Abou-Chakra, Jaehyung Kim, Jaimyn Drake, Jan Peters, Jan Schneider, Jasmine Hsu, Jeannette Bohg, Jeffrey Bingham, Jeffrey Wu, Jensen Gao, Jiaheng Hu, Jiajun Wu, Jialin Wu, Jiankai Sun, Jianlan Luo, Jiayuan Gu, Jie Tan, Jihoon Oh, Jimmy Wu, Jingpei Lu, Jingyun Yang, Jitendra Malik, João Silvério, Joey Hejna, Jonathan Booher, Jonathan Tompson, Jonathan Yang, Jordi Salvador, Joseph J. Lim, Junhyek Han, Kaiyuan Wang, Kanishka Rao, Karl Pertsch, Karol Hausman, Keegan Go, Keerthana Gopalakrishnan, Ken Goldberg, Kendra Byrne, Kenneth Oslund, Kento Kawaharazuka, Kevin Black, Kevin Lin, Kevin Zhang, Kiana Ehsani, Kiran Lekkala, Kirsty Ellis, Krishan Rana, Krishnan Srinivasan, Kuan Fang, Kunal Pratap Singh, Kuo-Hao Zeng, Kyle Hatch, Kyle Hsu, Laurent Itti, Lawrence Yunliang Chen, Lerrel Pinto, Li Fei Fei, Liam Tan, Linxi "Jim" Fan, Lionel Ott, Lisa Lee, Luca Weihs, Magnum Chen, Marion Lepert, Marius Memmel, Masayoshi Tomizuka, Masha Itkina, Mateo Guaman Castro, Max Spero, Maximilian Du, Michael Ahn, Michael C. Yip, Mingtong Zhang, Mingyu Ding, Minho Heo, Mo
---
Han Kumar Srirama、Mohit Sharma、Moo Jin Kim、Naoaki Kanazawa、Nicklas Hansen、Nicolas Heess、Nikhil J Joshi、Niko Suenderhauf、Ning Liu、Norman Di Palo、Nur Muhammad Mahi Shafiullah、Oier Mees、Oliver Kroemer、Osbert Bastani、Pannag R Sanketi、Patrick "Tree" Miller、Patrick Yin、Paul Wohlhart、Peng Xu、Peter David Fagan、Peter Mitrano、Pierre Sermanet、Pieter Abbeel、Priya Sundaresan、Qiuyu Chen、Quan Vuong、Rafael Rafailov、Ran Tian、Ria Doshi、Roberto Martín-Martín、Rohan Baijal、Rosario Scalise、Rose Hendrix、Roy Lin、Runjia Qian、Ruohan Zhang、Russell Mendonca、Rutav Shah、Ryan Hoque、Ryan Julian、Samuel Bustamante、Sean Kirmani、Sergey Levine、Shan Lin、Sherry Moore、Shikhar Bahl、Shivin Dass、Shubham Sonawani、Shuran Song、Sichun Xu、Siddhant Haldar、Siddharth Karamcheti、Simeon Adebola、Simon Guist、Soroush Nasiriyan、Stefan Schaal、Stefan Welker、Stephen Tian、Subramanian Ramamoorthy、Sudeep Dasari、Suneel Belkhale、Sungjae Park、Suraj Nair、Suvir Mirchandani、Takayuki Osa、Tanmay Gupta、Tatsuya Harada、Tatsuya Matsushima、Ted Xiao、Thomas Kollar、Tianhe Yu、Tianli Ding、Todor Davchev、Tony Z. Zhao、Travis Armstrong、Trevor Darrell、Trinity Chung、Vidhi Jain、Vincent Vanhoucke、Wei Zhan、Wenxuan Zhou、Wolfram Burgard、Xi Chen、Xiaolong Wang、Xinghao Zhu、Xinyang Geng、Xiyuan Liu、Xu Liangwei、Xuanlin Li、Yao Lu、Yecheng Jason Ma、Yejin Kim、Yevgen Chebotar、Yifan Zhou、Yifeng Zhu、Yilin Wu、Ying Xu、Yixuan Wang、Yonatan Bisk、Yoonyoung Cho、Youngwoon Lee、Yuchen Cui、Yue Cao、Yueh-Hua Wu、Yujin Tang、Yuke Zhu、Yunchu Zhang、Yunfan Jiang、Yunshuang Li、Yunzhu Li、Yusuke Iwasawa、Yutaka Matsuo、Zehan Ma、Zhuo Xu、Zichen Jeff Cui、Zichen Zhang和Zipeng Lin。Open X-Embodiment：机器人学习数据集与RT-X模型。，2023.3
[^12]: Enric Corona, Albert Pumarola, Guillem Alenya, Francesc Moreno-Noguer, and Grégory Rogez. GanHand: Predicting human grasp affordances in multi-object scenes. In Computer Vision and Pattern Recognition (CVPR), 2020. 2, 3, 15
[^13]: Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Evangelos Kazakos, Jian Ma, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. Rescaling egocentric vision: Collection, pipeline and challenges for epic-kitchens-100. International Journal of Computer Vision, 2022. 15
[^14]: Bardia Doosti, Shujon Naha, Majid Mirbagheri, and David Crandall. HOPE-Net: A graph-based model for hand-object pose estimation. In Computer Vision and Pattern Recognition (CVPR), 2020. 2
[^15]: Zicong Fan, Omid Taheri, Dimitrios Tzionas, Muhammed Kocabas, Manuel Kaufmann, Michael J Black, and Otmar Hilliges. ARCTIC: A dataset for dexterous bimanual hand-object manipulation. In Computer Vision and Pattern Recognition (CVPR), 2023. 2, 3, 6, 13, 15
[^16]: Guillermo Garcia-Hernando, Shanxin Yuan, Seungryul Baek, and Tae-Kyun Kim. First-person hand action benchmark with rgb-d videos and 3D hand pose annotations. In Computer Vision and Pattern Recognition (CVPR), 2018. 15
[^17]: Anindita Ghosh, Rishabh Dabral, Vladislav Golyanik, Christian Theobalt, and Philipp Slusallek. IMOs: Intent-driven full-body motion synthesis for human-object interactions. In Computer Graphics Forum, 2023. 2, 3
[^18]: James J Gibson. The ecological approach to visual perception: classic edition. Psychology Press, 2014. 2
[^19]: Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Computer Vision and Pattern Recognition (CVPR), 2022. 6, 15
[^20]: Kristen Grauman, Andrew Westbury, Lorenzo Torresani, Kris Kitani, Jitendra Malik, Triantafyllos Afouras, Kumar Ashutosh, Vijay Baiyya, Siddhant Bansal, Bikram Boote, et al. Ego-exo4d: Understanding skilled human activity from first-and third-person perspectives. arXiv preprint arXiv:2311.18259, 2023. 6, 15
[^21]: Shreyas Hampali, Mahdi Rad, Markus Oberweger, and Vincent Lepetit. Honnotate: A method for 3D annotation of hand and object poses. In Computer Vision and Pattern Recognition (CVPR), 2020. 2, 3, 6, 15
[^22]: Shreyas Hampali, Sayan Deb Sarkar, Mahdi Rad, and Vincent Lepetit. Keypoint transformer: Solving joint identification in challenging hands and object interactions for accurate 3D pose estimation. In Computer Vision and Pattern Recognition (CVPR), 2022. 2
[^23]: Ankur Handa, Karl Van Wyk, Wei Yang, Jacky Liang, Yu-Wei Chao, Qian Wan, Stan Birchfield, Nathan Ratliff, and Dieter Fox. DexPilot: Vision-based teleoperation of dexterous robotic hand-arm system. In International Conference on Robotics and Automation (ICRA), pages 9164–9170. IEEE, 2020. 8
[^24]: Yana Hasson, Gul Varol, Dimitrios Tzionas, Igor Kalevatykh, Michael J Black, Ivan Laptev, and Cordelia Schmid. Learning joint reconstruction of hands and manipulated objects. In Computer Vision and Pattern Recognition (CVPR), 2019. 2, 3, 15
[^25]: Yana Hasson, Bugra Tekin, Federica Bogo, Ivan Laptev, Marc Pollefeys, and Cordelia Schmid. Leveraging photometric consistency over time for sparsely supervised hand-object reconstruction. In Computer Vision and Pattern Recognition (CVPR), 2020.
[^26]: Yana Hasson, Gül Varol, Ivan Laptev, and Cordelia Schmid. Towards unconstrained joint hand-object reconstruction from rgb videos. In International Conference on 3D Vision (3DV), 2021. 2
[^27]: De-An Huang, Suraj Nair, Danfei Xu, Yuke Zhu, Animesh Garg, Li Fei-Fei, Silvio Savarese, and Juan Carlos Niebles. Neural task graphs: Generalizing to unseen tasks from a single video demonstration. In Computer Vision and Pattern Recognition (CVPR), 2019. 3
## [^28]: Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In International Conference on Machine Learning (ICML). PMLR, 2022. 3
---
[^29]: Wenlong Huang, Chen Wang, Ruohan Zhang, Yunzhu Li, Jiajun Wu, and Li Fei-Fei. VoxPoser: Composable 3D value maps for robotic manipulation with language models. arXiv preprint arXiv:2307.05973, 2023. 3
[^30]: Juntao Jian, Xiuping Liu, Manyi Li, Ruizhen Hu, and Jian Liu. AffordPose: A large-scale dataset of hand-object interactions with affordance-driven hand pose. arXiv preprint arXiv:2309.08942, 2023. 2, 3, 15
[^31]: Hanwen Jiang, Shaowei Liu, Jiashun Wang, and Xiaolong Wang. Hand-object contact consistency reasoning for human grasps generation. In International Conference on Computer Vision (ICCV), 2021. 2
[^32]: Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014. 14
[^33]: Christopher A Kurby and Jeffrey M Zacks. Segmentation in the perception and memory of events. Trends in cognitive sciences, 2008. 2
[^34]: Taein Kwon, Bugra Tekin, Jan Stuhmer, Federica Bogo, and Marc Pollefeys. H2O: Two hands manipulating objects for first person interaction recognition. In International Conference on Computer Vision (ICCV), 2021. 2, 3, 6, 15
[^35]: Jiefeng Li, Siyuan Bian, Ailing Zeng, Can Wang, Bo Pang, Wentao Liu, and Cewu Lu. Human pose regression with residual log-likelihood estimation. In International Conference on Computer Vision (ICCV), 2021. 6
[^36]: Jiaman Li, Jiajun Wu, and C Karen Liu. Object motion guided human motion synthesis. ACM Transactions on Graphics (TOG), 2023. 3
[^37]: Kailin Li, Lixin Yang, Zenan Lin, Jian Xu, Xinyu Zhan, Yifei Zhao, Pengxiang Zhu, Wenxiong Kang, Kejian Wu, and Cewu Lu. FAVOR: Full-body ar-driven virtual object rearrangement guided by instruction text. In AAAI Conference on Artificial Intelligence, 2024. 3, 17, 18
[^38]: Kevin Lin, Lijuan Wang, and Zicheng Liu. End-to-end human pose and mesh reconstruction with transformers. In Computer Vision and Pattern Recognition (CVPR), 2021. 6, 16
[^39]: Shaowei Liu, Hanwen Jiang, Jiarui Xu, Sifei Liu, and Xiaolong Wang. Semi-supervised 3D hand-object poses estimation with interactions in time. In Computer Vision and Pattern Recognition (CVPR), 2021. 2
[^40]: Yunze Liu, Yun Liu, Che Jiang, Kangbo Lyu, Weikang Wan, Hao Shen, Boqiang Liang, Zhoujie Fu, He Wang, and Li Yi. HOI4D: A 4d egocentric dataset for category-level human-object interaction. In Computer Vision and Pattern Recognition (CVPR), 2022. 2, 6, 15
[^41]: Yun Liu, Haolin Yang, Xu Si, Ling Liu, Zipeng Li, Yuxiang Zhang, Yebin Liu, and Li Yi. TACO: Benchmarking generalizable bimanual tool-action-object understanding. arXiv preprint arXiv:2401.08399, 2024. 3, 15
[^42]: Jun Lv, Wenqiang Xu, Lixin Yang, Sucheng Qian, Chongzhao Mao, and Cewu Lu. HandTailor: Towards high-precision monocular 3D hand recovery. In British Machine Vision Conference (BMVC), 2021. 6
[^43]: Steven Macenski, Tully Foote, Brian Gerkey, Chris Lalancette, and William Woodall. Robot operating system 2:
设计、架构及实际应用。《科学·机器人》杂志，7(66):eabm6074，2022年。共13页。
[^44]: Naureen Mahmood, Nima Ghorbani, Nikolaus F Troje, Gerard Pons-Moll, and Michael J Black. AMASS: Archive of motion capture as surface shapes. In Computer Vision and Pattern Recognition (CVPR), 2019. 5, 13, 14
[^45]: Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, et al. Isaac gym: High performance gpu-based physics simulation for robot learning. arXiv preprint arXiv:2108.10470, 2021. 8
[^46]: Andrew T Miller and Peter K Allen. Graspit!: A versatile simulator for robotic grasping. IEEE Robotics & Automation Magazine, 11(4):110–122, 2004. 15
[^47]: Takehiko Ohkawa, Kun He, Fadime Sener, Tomas Hodan, Luan Tran, and Cem Keskin. Assembly Hands: Towards egocentric activity understanding via 3D hand pose estimation. In Computer Vision and Pattern Recognition (CVPR), 2023. 2, 6, 15
[^48]: OpenAI. GPT-4 technical report. arXiv preprint arXiv:2303.08774, 2023. 5, 7, 17
[^49]: OptiTrack. Motive: Optical motion capture software. . 13
[^50]: Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed AA Osman, Dimitrios Tzionas, and Michael J Black. Expressive Body Capture: 3D hands, face, and body from a single image. In Computer Vision and Pattern Recognition (CVPR), 2019. 5, 13
[^51]: Mathis Petrovich, Michael J Black, and Gül Varol. Action-conditioned 3D human motion synthesis with transformer VAE. In International Conference on Computer Vision (ICCV), 2021. 3
[^52]: Mathis Petrovich, Michael J Black, and Gül Varol. TEMOS: Generating diverse human motions from textual descriptions. In European Conference on Computer Vision (ECCV), 2022. 3
[^53]: Yuzhe Qin, Hao Su, and Xiaolong Wang. From one hand to multiple hands: Imitation learning for dexterous manipulation from single-camera teleoperation. IEEE Robotics and Automation Letters, 2022. 2, 8
[^54]: Yuzhe Qin, Yueh-Hua Wu, Shaowei Liu, Hanwen Jiang, Ruihan Yang, Yang Fu, and Xiaolong Wang. DexMV: Imitation learning for dexterous manipulation from human videos. In European Conference on Computer Vision (ECCV), 2022. 2, 8
[^55]: Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, and Animesh Garg. ProgPrompt: Generating situated robot task plans using large language models. In International Conference on Robotics and Automation (ICRA), 2023. 3
[^56]: Omid Taheri, Nima Ghorbani, Michael J Black, and Dimitrios Tzionas. GRAB: A dataset of whole-body human grasping of objects. In European Conference on Computer Vision (ECCV), 2020. 2, 3, 6, 13, 15
### [^57]: Omid Taheri, Vasileios Choutas, Michael J Black, and Dimitrios Tzionas. GOAL: Generating 4d whole-body motion
---
用于手-物体抓取任务。载于《计算机视觉与模式识别会议》（CVPR），2022年。第2、3、18页。
[^58]: Omid Taheri, Yi Zhou, Dimitrios Tzionas, Yang Zhou, Duygu Ceylan, Soren Pirk, and Michael J Black. Grip: Generating interaction poses using spatial cues and latent consistency. In International Conference on 3D Vision (3DV), 2024. 3
[^59]: Guy Tevet, Sigal Raab, Brian Gordon, Yonatan Shafir, Daniel Cohen-Or, and Amit H Bermano. Human motion diffusion model. International Conference on Learning Representations (ICLR), 2023. 3, 7
[^60]: Emanuel Todorov, Tom Erez, and Yuval Tassa. MuJoCo: A physics engine for model-based control. 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, 2012. 8
[^01]: Weikang Wan, Haoran Geng, Yun Liu, Zikang Snan, Yaodong Yang, Li Yi, and He Wang. UniDexGrasp++: Improving dexterous grasping policy learning via geometry-aware curriculum and iterative generalist-specialist learning. arXiv preprint arXiv:2304.00464, 2023. 8
[^62]: Yan Wu, Jiahao Wang, Yan Zhang, Siwei Zhang, Otmar Hilliges, Fisher Yu, and Siyu Tang. SAGA: Stochastic whole-body grasping with contact. In European Conference on Computer Vision (ECCV), 2022. 2, 3, 17
[^63]: Danfei Xu, Suraj Nair, Yuke Zhu, Julian Gao, Animesh Garg, Li Fei-Fei, and Silvio Savarese. Neural task programming: Learning to generalize across hierarchical tasks. In International Conference on Robotics and Automation (ICRA), 2018. 3
[^64]: Yinzhen Xu, Weikang Wan, Jialiang Zhang, Haoran Liu, Zikang Shan, Hao Shen, Ruicheng Wang, Haoran Geng, Yijia Weng, Jiayi Chen, et al. Unidexgrasp: Universal robotic dexterous grasping via learning diverse proposal generation and goal-conditioned policy. In Computer Vision and Pattern Recognition (CVPR), 2023. 8
[^65]: Lixin Yang, Xinyu Zhan, Kailin Li, Wenqiang Xu, Jiefeng Li, and Cewu Lu. CPF: Learning a contact potential field to model the hand-object interaction. In Computer Vision and Pattern Recognition (CVPR), 2021. 14
[^66]: Lixin Yang, Kailin Li, Xinyu Zhan, Jun Lv, Wenqiang Xu, Jiefeng Li, and Cewu Lu. ArtiBoost: Boosting articulated 3D hand-object pose estimation via online exploration and synthesis. In Computer Vision and Pattern Recognition (CVPR), 2022. 2
[^67]: Lixin Yang, Kailin Li, Xinyu Zhan, Fei Wu, Anran Xu, Liu Liu, and Cewu Lu. OakInk: A large-scale knowledge repository for understanding hand-object interaction. In Computer Vision and Pattern Recognition (CVPR), 2022. 2, 3, 6, 15, 16
[^68]: Lixin Yang, Jian Xu, Licheng Zhong, Xinyu Zhan, Zhicheng Wang, Kejian Wu, and Cewu Lu. POEM: Reconstructing hand in a point embedded multi-view stereo. In Computer Vision and Pattern Recognition (CVPR), 2023. 6
[^69]: Haoyu Zhen, Xiaowen Qiu, Peihao Chen, Jincheng Yang, Xin Yan, Yilun Du, Yining Hong, and Chuang Gan. 3D-VLA: A 3D vision-language-action generative world model. arXiv preprint arXiv:2403.09631, 2024. 7, 8
[^70]: Hao Zheng, Regina Lee, and Yuqian Lu. HA-ViD: A human assembly video dataset for comprehensive assembly knowledge understanding. arXiv preprint arXiv:2307.05721, 2023. 15
[^71]: Juntian Zheng, Qingyuan Zheng, Lixing Fang, Yun Liu, and Li Yi. CAMS: Canonicalized manipulation spaces for category-level functional hand-object manipulation synthesis. In Computer Vision and Pattern Recognition (CVPR), 2023. 2
[^72]: Zehao Zhu, Jiashun Wang, Yuzhe Qin, Deqing Sun, Varun Jampani, and Xiaolong Wang. ContactArt: Learning 3D interaction priors for category-level articulated object and hand poses estimation. arXiv preprint arXiv:2305.01618, 2023. 2, 3, 15
#### [^73]: Christian Zimmermann, Duygu Ceylan, Jimei Yang, Bryan Russell, Max Argus, and Thomas Brox. FreiHAND: A dataset for markerless capture of hand pose and shape from single rgb images. In International Conference on Computer Vision (ICCV), 2019. 16
---
#### 补充材料
#### 目录
A 标注详情
A.1 平台校准与同步
A.2 数据清洗
A.3 人体姿态与表面
B 数据集元信息
B.1 特定任务子集
C 数据集评估
C.1 跨数据集验证
C.2 物理属性评估
D 任务与基准测试
D.1 任务感知型运动实现
E 应用：复杂任务完成
F 数据集检查
F.1 任务列表
F.2 可视化
### A. 标注详情
#### A.1. 平台校准与同步
动作捕捉系统通过厂商提供的专用校准棒进行校准。多摄像头系统中的各摄像头则利用ArUco立方体进行校准。这些摄像头上装有反光标记点，以便被动作捕捉系统追踪。相关校准工具如图9所示。两套系统通过ROS2中集成的软件同步工具实现时间同步 [^43]。
### A.2. 数据清洗
本节简要介绍为后续计算物体姿态和人体姿态而对动作捕捉系统采集的反光标记点位置进行清洗的流程。动作捕捉系统的固有局限性必然会导致所获取的反光标记点位置出现误差：在严重遮挡的情况下，系统可能无法检测并记录某些标记点；由于环境中的杂散反射，可能会引入虚假标记点；当两个或多个标记点距离过近时，系统可能会错误分配其标签，或将其误判为单个标记点。这些局限性促使我们引入了手动机制来清洗和后期处理采集到的数据。
数据清洗流程包含两个部分：1）动作捕捉后期处理软件；2）多视角交互式编辑器。我们邀请了三名专业标注员参与数据清洗工作。标注员首先会在动作捕捉后期处理软件中依次检查采集到的反光标记点的位置 [^49]。随后他们便会剔除虚假点、拆分重叠的标记点、修正标签错误的标记点，并填补其中的短间隙。
Figure 9. Illustration of Platform Calibration Tools. The top is the vender-provided calibration wand for the MoCap system. The bottom are the ArUco cubes attached with reflective markers (circled in red). The ArUco patterns are for unifying the camera in the multi-camera system, while the surface-attached reflective markers are used for unifying the cameras with the MoCap system.
标记轨迹。标注员按照从关节部位到刚体、人体以及双手的顺序，系统地清理所采集标记的结果。随后，这些序列被导出至多视角交互式编辑器。在编辑器中，标注员会验证清理后的动作捕捉结果，并通过基于三角测量的标注方法，利用多个视角下的2D点位置来恢复极端遮挡情况下的标记位置。最终将这些结果整合，得到在捕捉区域内清理后的反射标记位置。
#### A.3. 人体姿态与表面
我们采用了一种两阶段拟合方法，该方法借鉴了MoSH++算法在[^15] [^44] [^56]中对SMPL-X [^50]进行标注的应用方式。第一阶段是注册受试者的SMPL-X形状参数，并建立标记集与SMPL-X模型表面之间的对应关系映射。第二阶段则为序列中的每一帧注册SMPL-X姿态参数。由于PyTorch支持自动微分，这一两阶段拟合流程便在PyTorch上得以实现；各阶段求解均采用常见的梯度下降算法。
第一阶段：设$\bar{\beta}$为形状参数；设$P_{\mathcal{M}}^{(c)} \in \mathbb{R}^{N_{\mathcal{M}} \times 3}$为位于SMPL-X标准空间内的表面标记位置，其中$N_{\mathcal{M}}$表示目标标记集中的标记数量；设$\theta = \{\theta_i\}$为受试者处于T姿势时第i帧的SMPL-X姿态参数。第一阶段可表述为一个优化过程，旨在最小化观测到的标记与
---
以及由位于SMPL-X标准空间内的表面标记位置所重构出的标记之间的距离，如公式(1)所示。
$$ \begin{aligned}\min_{\theta,\beta,P_{\mathcal{M}}^{(c)}}E=\lambda_{1}E_{recon}\big(\theta,\beta,P_{\mathcal{M}}^{(c)}\big)+\lambda_{2}E_{prior(b)}\big(\theta\big)+\\ \lambda_{3}E_{plau(h)}\big(\theta\big)+\lambda_{4}E_{reg}\big(\theta,\beta,P_{\mathcal{M}}^{(c)}\big)\end{aligned} $$
主要代价函数为$E_{recon}$，它代表观测到的标记$P_{\mathcal{M}}$与由表面标记位置重构出的标记$\hat{P}_{\mathcal{M}}$之间的距离。设$\boldsymbol{V}^{(c)}$为SMPL-X标准空间内的表面顶点，V为重构的表面顶点。标记集对应函数$\mathcal{C}(\cdot)$利用标准空间内的标记集位置$P_{\mathcal{M}}^{(c)}$和表面顶点$\boldsymbol{V}^{(c)}$，从当前重构的表面顶点V中恢复出标记集位置$\hat{P}_{\mathcal{M}}$。该函数首先把标准空间中的标记$P_{\mathcal{M}}^{(c)}$投影到由表面顶点构成的局部坐标系中，从而得到顶点索引$I_{\mathcal{M}}$以及局部坐标系中的系数$C_{\mathcal{M}}$。接着利用该索引在已摆好姿势的顶点V上恢复出对应帧，再借助局部坐标系中的系数来确定摆好姿势后的SMPL-X人体上的标记位置。因此$E_{recon}$可如公式(2)所示进行表达。
$$ \begin{aligned}\boldsymbol{V}^{(c)}&=\mathrm{SMPL-X}(\mathbf{0},\bar{\boldsymbol{\beta}})\quad\boldsymbol{V}=\mathrm{SMPL-X}(\boldsymbol{\theta},\bar{\boldsymbol{\beta}})\\E_{\mathrm{recon}}&=\left\|\boldsymbol{P}_{\mathcal{M}}-\hat{\boldsymbol{P}}_{\mathcal{M}}\right\|^{2}=\left\|\boldsymbol{P}_{\mathcal{M}}-\mathcal{C}\big(\boldsymbol{V},\boldsymbol{P}_{\mathcal{M}}^{(c)};\boldsymbol{V}^{(c)}\big)\right\|^{2}\end{aligned} $$
$E_{\text{prior}(b)}(\theta)$、$E_{\text{plau}(h)}(\theta)$和$E_{\text{reg}}(\theta, \beta, P_{\mathcal{M}}^{(c)})$为辅助代价项。$E_{\text{prior}(b)}(\theta)$是一项辅助项，用于最小化根据现有数据集先验计算出的人体姿态的负对数似然值，这一做法遵循[^44]中的实践。E_{\text{plau}(h)}(\theta) $is an implementation of anatomy loss in [65] on the SMPLX model, designed to prevent distortion in the pose of the human body during the fitting process, enhancing its physical plausibility.$ E_{\text{reg}}(\theta, \beta, P_{\mathcal{M}}^{(c)}) $则是一项用于正则化优化变量的辅助项。
第二阶段：在第二阶段中，我们基于第一阶段得到的形状参数$\theta = \{\theta_t\}$和标记对应关系$\bar{\beta}$，来拟合整个交互过程中受试者的姿态$C(\cdot)$。
对于序列中的每个帧 t，我们会对受试者的 SMPL-X 姿态参数 $\theta_t$ 进行优化，以最小化由观测到的标记点重建误差 $E_{\mathrm{recon}}(\theta_t)$、身体姿态先验 $E_{\mathrm{prior(b)}}(\theta_t)$、手部解剖结构异常 $E_{\mathrm{plau(h)}}(\theta_t)$、手与物体的相交情况 $E_{\mathrm{plau(ho)}}(\theta_t)$ 以及其他辅助正则化项所组成的综合成本。
$$ \begin{aligned}\min_{\theta_{t}}E=&\lambda_{1}E_{recon}\big(\theta_{t}\big)+\lambda_{2}E_{prior(b)}\big(\theta_{t}\big)+\lambda_{3}E_{plau(h)}\big(\theta_{t}\big)\\&+\lambda_{4}E_{plau(ho)}\big(\theta_{t}\big)+\lambda_{5}E_{reg}\big(\theta_{t}\big)\end{aligned} $$
$E_{\text{plau}(h)}(\theta_t)$、$E_{\text{prior}(b)}(\theta_t)$ 和 $E_{\text{plau}(h)}(\theta_t)$ 与第一阶段的成本项相同。$E_{\text{plau}(ho)}$ 通过对手部网格内部的点进行采样，并计算这些点到物体有符号距离函数值之和，来惩罚交互的手与物体之间的穿透和相交现象。$E_{\text{reg}}(\theta_t)$ 不仅包含对优化变量的正则化项，还包含用于保持标注轨迹平滑性的速度正则化项。
优化方面，我们在 PyTorch 上实现了两阶段拟合流程，因为它支持自动微分功能。我们采用 Adam [^32] 作为两个阶段的优化器，该算法应用广泛，且适用于两个阶段中引入的非凸成本项。为了提高第二阶段的运行速度，我们提出了一种提前停止机制：如果在连续多帧中拟合成本的降低幅度未超过特定阈值，优化过程就会终止。
#### B. 数据集元信息
### B.1. 任务特定的子集
由于 OAKINK2 旨在支持多种类型的任务，我们针对每项具体任务，采用不同的样本选择和数据组织策略来创建多个子集。为得到这些子集，我们会运用一些启发式规则来判断每个样本是否满足其所支持任务的要求。例如，图像样本中手部或物体的可见性对于支持相关视觉任务至关重要。我们会验证图像中手部与物体的单独及组合分割掩码；如果组合分割掩码占其单独掩码的比例超过某一阈值，我们就认为该实例在当前帧中可见。如果物体与手的距离足够近（最小距离 ≤ 5 mm）且被抬起（相对于初始状态的高度位移 ≥ 5 mm），我们就将其视为可被抓取的对象。
我们在下文中列出了任务特定子集的特征及构建方法。
OAKINK2-H-SV 子集用于从单视图图像中恢复手部模型。我们选取能清晰看到受试者手部的视图来构成该子集。该子集支持单视图手部网格恢复任务。
OAKINK2-H-MV 子集用于从多视图图像中恢复手部模型。如果受试者的手部在多数相机视图中可见，我们就将不同相机的组合视图作为该子集中的单个样本。该子集支持多视图手部网格恢复任务。
OAKINK2-HO 子集用于从图像中估计或重建手与物体的姿态。我们选取能看到受试者手部且物体被抓取的视图来构成该子集。
OAKINK2-Grasp 子集用于研究对物体的抓取行为。我们
---
| Dataset | image mod. | resolution | #frame | #views | #subj | #obj | 3D | real / syn. | label method | hand pose | obj pose | afford. inter. | dynamic inter. | long- | horizon | task |
| EPIC-KITCHEN-100 [^13] | ✓ | ~ | 20M $^{\dagger}$ | 1 | 37 | - | ✗ | - | - | ✗ | ✗ | ✗ | ✗ | ✓ | ✓ | ✓ |
| Ego4D [^19] | ✓ | ~‡ | ~ $^{\dagger}$ | 1 | 931 | - | ✗ | - | - | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ | ✓ |
| HA-ViD [^70] | ✓ | 1280 $\times$ 720 | 1.5M | 3 | 30 | 40 | ✗ | - | - | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ | ✓ |
| FPHAB [^16] | ✓ | 1920 $\times$ 1080 | 105K | 1 | 6 | 4 | ✓ | real | mocap | ✓ | ✓ | ✓ | ✓ | ✓ | ✗ | ✗ |
| ObMan [^24] | ✓ | 256 $\times$ 256 | 154K | 1 | 20 | 3K | ✓ | syn | simulate | ✓ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ |
| YCBAfford [^12] | ✓ | - | 133K | 1 | 1 | 21 | ✓ | syn | manual | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ |
| HO3D [^21] | ✓ | 640 $\times$ 480 | 78K | 1-5 | 10 | 10 | ✓ | real | auto | ✓ | ✓ | ✗ | ✓ | ✓ | ✗ | ✗ |
| ContactPose [^3] | ✓ | 960 $\times$ 540 | 2.99M | 3 | 50 | 25 | ✓ | real | auto | ✓ | ✓ | ✓ | ✗ | ✗ | ✗ | ✗ |
| GRAB [^56] | ✗ | - | 1.62M | - | 10 | 51 | ✓ | real | mocap | ✓ | ✓ | ✓ | ✓ | ✓ | ✗ | ✗ |
| DexYCB [^8] | ✓ | 640 $\times$ 480 | 582K | 8 | 10 | 20 | ✓ | real | crowd | ✓ | ✓ | ✗ | ✓ | ✓ | ✗ | ✗ |
| H2O [^34] | ✓ | 1280 $\times$ 720 | 571K | 5 | 4 | 8 | ✓ | real | auto | ✓ | ✓ | ✓ | ✓ | ✓ | ✗ | ✗ |
| HOI4D [^40] | ✓ | 1280 $\times$ 800 | 3M | 1 | 9 | 1000 | ✓ | real | crowd | ✓ | ✓ | ✓ | ✓ | ✓ | ✗ | ✗ |
| ARCTIC [^15] | ✓ | 2800 $\times$ 2000 | 2.1M | 9 | 10 | 11 | ✓ | real | mocap | ✓ | ✓ | ✓ | ✓ | ✓ | ✗ | ✗ |
| ContactArt [^72] | ✓ | - | 332K | - | - | 80 | ✓ | real | transfer | ✓ | ✓ | ✗ | ✓ | ✓ | ✗ | ✗ |
| AssemblyHands [^47] | ✓ | 1920 $\times$ 1080 | 3.03M | 12 | 34 | - | ✓ | real | semi-auto | ✓ | ✗ | ✓ | ✓ | ✓ | ✓ | ✓ |
| AffordPose [^30] | ✗ | - | - | - | - | 641 | ✓ | syn | manual | ✓ | ✓ | ✓ | ✗ | ✗ | ✗ | ✗ |
| TACO [^41] | ✓ | 4096 $\times$ 3000 $^{\ddagger}$ | 5.2M | 13 | 14 | 196 | ✓ | real | auto | ✓ | ✓ | ✓ | ✓ | ✓ | ✗ | ✗ |
| Ego-Exo4D [^20] | ✓ | ~ $^{\ddagger}$ | ~ $^{\dagger}$ | 5-6 | 839 | - | ✓ | real | semi-auto | ✓ | ✗ | ✗ | ✓ | ✓ | ✓ | ✓ |
| OakInk-Image [^67] | ✓ | 848 $\times$ 480 | 230K | 4 | 12 | 100 | ✓ | real | crowd | ✓ | ✓ | ✓ | ✓ | ✓ | ✗ | ✗ |
| OakInk-Shape [^67] | ✗ | - | - | - | - | 1700 | ✓ | real | transfer | ✓ | ✓ | ✓ | ✓ | ✗ | ✗ | ✗ |
| OAKINK2 | ✓ | 848 $\times$ 480 | 4.01M | 4 | 9 | 75 | ✓ | real | mocap | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
Table 4. A cross-comparison among various public datasets. ~: The value is either not provided on the paper or measured in a different unit. †: Datasets measure in record time rather than number of captured frames. In particular, EPIC-KITCHEN-100 contains more than 100 hours of video, Ego4D 3670 hours, and Ego-Exo4D 1422 hours. They are larger in scale than any other dataset listed in the table. ‡: Dataset has a mixed resolution.
图例：
图像模态：指数据的图像类型。✓ 表示真实图像采集；✓ 表示合成（渲染）图像；✗ 表示未提供图像模态。3D 定位：指数据是否包含 3D 定位标注。✓ 表示数据集包含 3D 定位标注；✗ 表示数据集仅为 2D 数据。
真实/合成：交互分为真实交互和合成交互。此处“合成”表示交互来自特定的抓取/交互合成器。
标签方法：3D定位信息的标签方法。对于合成交互，“simulate”表示交互是从基于物理的抓取模拟器中获取的，例如GraspIt！[^46]；“manual”表示交互是通过人工标注得到的。对于真实交互，“mocap”表示交互是由MoCap系统捕获的；“crowd”表示交互源自众包关键点标注；“auto”表示交互是从自动标注流程中获取的；“semi-auto”则是“crowd”和“auto”方法的混合。
基于可供性交互：基于可供性的交互。✓表示所捕获的交互具备可供性意识且被明确标注；✓表示交互具备可供性意识，但被归类到意图等粗粒度标签中；✗表示交互未按物体可供性进行组织。
动态交互：✓表示数据集捕获了手与物体的动态交互序列；X表示数据集捕获的是在交互过程中不发生变化的静态抓取。
长时序任务：如正文所述，✓表示数据集包含涉及多种物体可供性的捕获交互；反之则为X。
任务分解：如正文所述，✓表示数据集包含将复杂任务分解为多个片段的标注；反之则为X。
选取物体被抓取时的帧来构成该子集。
OAKINK2-靠近与抓取动作子集，用于描述受试者靠近并抓取物体以备后续使用的交互过程。我们从一个基本任务序列中选取涵盖靠近及抓取物体过程的帧来构成该子集。该子集在任务感知运动实现以及复杂任务完成中的数据库查询对象轨迹检索方面提供了辅助信息。
OAKINK2-任务相关运动子集，用于描述受试者完成任务并实现某一物体可供性的交互过程。我们从一个基本任务序列中选取涵盖从抓取物体到任务完成的过程的帧来构成该子集。该子集为任务感知运动实现提供支持。
#### C. 数据集评估
3D手部关键点的标注会借助重建模型进行跨数据集验证，而与之相关的3D姿态
---
Figure 10. Distribution of Primitive Task demonstrations. The sub-figure above displays the proportion of Primitive Task demonstrations across various scenarios within the entire OAKINK2 dataset, with frequently occurring Primitive Tasks highlighted. The sub-figure below presents a list of Primitive Tasks recorded in OAKINK2, along with the illustration of their corresponding quantity distribution. A list of all recorded Primitive Tasks and Complex Tasks can be found at Tab. 7 and Tab. 8.
与抓取动作相关的数据会就其物理属性完整性进行检查。
## C.1. 跨数据集验证
我们开展跨数据集验证，以确认OAKINK2中3D手部关键点标注与现有数据集的一致性。我们分别针对三种训练方案训练单视图手部网格恢复模型[^38]：仅使用FreiHAND [^73]、仅使用OAKINK2-H-SV，以及混合使用这两组数据。我们在OakInk-image（SP2）[^67]上进行了Procrustes分析后的MPJPE和MPVPE评估，表5所示结果表明这些指标均有明显改善，这验证了OAKINK2能够补充现有数据集并提升现有模型的性能。
| Train | Test | PA-MPJPE (mm) $\downarrow$ | PA-MPVPE (mm) $\downarrow$ |
| 1) FreiHAND | OakInk-image (SP2) | 12.07 | 11.96 |
| 2) OAKINK2-H-SV | OakInk-image (SP2) | 12.60 | 11.04 |
| 1) & 2) mixture | OakInk-image (SP2) | 10.94 | 9.67 |
Table 5. Cross dataset validation for OAKINK2.
### C.2. 物理属性评估
为了评估OAKINK2中与抓取动作相关的3D姿态质量，我们考察了多个基于物理的指标，这些指标用于评估所捕获的手部与物体交互的可行性和稳定性。我们根据特定规则筛选待评估的样本（即参与交互的物体必须被成功抓取并抬起），以确保这些基于物理的质量指标能够准确反映数据集在交互过程中的质量。我们将OAKINK2-Grasp（-G.）与OakInk的两个子集——OakInk-Core和OakInk-Shape进行对比（见表6）。我们发现，尽管OAKINK2主要采用动作捕捉系统作为标注方法以便于扩大采集规模，但其标注质量仍可与基于人工标注和动作捕捉混合方式的OakInk相媲美。关于OAKINK2的更多定性可视化结果详见图14。
| Metrics | OAKINK2-G. | OAKINK-Core | OAKINK-Shape |
| Penet. Depth. cm $\downarrow$ | 0.25 | 0.18 | 0.11 |
| Solid Intsec. Vol. cm $^{3}$ $\downarrow$ | 0.61 | 1.03 | 0.62 |
| Sim. Disp. Mean cm $\downarrow$ | 1.83 | 0.98 | 0.94 |
| Sim. Disp. Std cm $\downarrow$ | 1.16 | 1.74 | 1.62 |
Table 6. Quality assessment of OAKINK2.
### D. 任务与基准测试
#### D.1. 任务感知的运动实现
#### 评估指标详情
训练-验证-测试划分：遵循HMR的做法，我们在序列层面划分子集，保持训练集、验证集和测试集中样本的比例约为70%、5%和25%，这一划分方式与OakInk一致。
CR，接触比例。该指标用于衡量运动轨迹中手与物体接触（最小距离）在5毫米阈值内的帧数占比。
SIV，实体交集体积。该指标用于衡量估计过程中空间交集的大小。我们将物体网格体素化为$100^{3}$个体素，再计算手表面内体素体积的总和。
PSKL-J，关节功率谱KL散度。
该指标反映了生成运动的平滑程度。
---
它通过计算预测关节序列与真实关节序列之间的加速度分布方差来评估运动平滑度，结果会分别从两个方向进行报告。我们的实现细节可参考[^37]和[^62]。显著的区别在于我们使用手部关节进行测量，因此得到的指标数值范围与全身关节的情况有所不同。
FID，Fréchet Inception Distance分数。该指标用于评估生成运动轨迹的真实感。我们基于Transformer编码器架构开发了运动特征提取器，通过给手部运动轨迹添加尾随标记来获取嵌入向量。每个运动对应的嵌入向量维度为64。编码器通过对运动轨迹进行分类训练。我们将编码器同时应用于真实轨迹和生成轨迹，通过计算两者之间的Fréchet Inception Distance来评估运动真实感。
## E. 应用：复杂任务完成
测试场景生成。为了评估正文中描述的、由神谕辅助的三阶段方法完成复杂任务的能力，我们通过扰动OAKINK2数据集中复杂场景内的物体位置来生成一组测试环境，同时保持任务目标$\text{text}_{goal}$和物体状态描述$\{\text{text}_{obj}\}$不变。
我们利用真实标注数据来生成测试场景中的变体。我们将特定的物体集合视为一个簇，并将它们放置在随机位置。同一簇内的物体会采用统一的偏移量，以确保它们能够同步随机化。当一组物体需要保持运动一致性时，这一点尤为重要。随机化过程包含四个不同的游走步骤，这样就能防止某个物体朝随机方向移动时与其他物体发生碰撞。因此，每个物体都有更多机会在其环境中绕过其他结构。每个物体的最终位置是这四个游走步骤累积作用的结果。在每个游走步骤中，最多会进行八次迭代，通过将步长减半来避免物体间的碰撞。
提示词生成。在本研究中，我们通过对大型语言模型GPT-4 [^48]进行调整来实现原始规划功能，使其能够根据描述当前场景和任务目标的叙述性提示词生成Python代码。语言模型的作用是解读这些描述，识别任务中涉及的关键物体，确定合适的物体可操作性与运动轨迹，并生成按可行顺序组织的原始指令执行实例。
为克服这些挑战，我们设计了包含前述场景与任务描述的提示词模板。该模板不仅阐明了底层的代码框架，还提供了与具体场景无关的代码示例。我们进一步要求语言学习模型（LLM）直接输出代码实现作为响应，而非提供关于编码过程的解释性说明。
关于底层代码框架，为确保代码生成的稳健性与连贯性，我们采用了实体-组件-系统（ECS）架构。这种结构促进了组件（此处指数据或状态）与系统（在我们语境中即原始指令）的解耦。这种方式使我们能够生成风格统一的代码实现，其结构包括实例化物体实体、加载可操作性作为组件，再将原始指令提交给执行系统。
原始指令规划的评估。我们利用随复杂任务一同提供的原始指令依赖图作为基准，对旨在完成任务的程序成功率进行测评。分析测评结果后发现，规划代码的整体生成成功率为36%。就复杂任务中所包含的原始指令数量而言，我们观察到了不同的成功率：在包含三个或更少原始指令的复杂任务中，成功率为44%；而在包含三到五个原始指令的复杂任务中，成功率降至20%；值得注意的是，包含超过五个原始指令的复杂任务均未取得成功。这些结果表明，在当前环境下，大型语言模型（LLM）足以处理相对简单的复杂任务；但在高度复杂的任务场景中，LLM难以准确理解物体可操作性与对应原始指令之间的关系和依赖性。
演示规划结果。我们针对在构建的测试场景中完成的复杂任务结果进行了评估。生成的Python程序如清单2所示。该代码将所有相关物体及其对应的可操作属性关联起来，然后按照精确要求的顺序执行基本操作。我们还提供了一个失败案例作为示例，见清单3，该案例反映了基本操作规划执行中的失误。这种失误表现为存在多余的基本操作，它满足了不必要的物体可操作属性要求，从而阻塞了执行路径。
完成复杂任务的替代运动生成方法。除了正文中介绍的基于TaMF的运动生成方法外，我们还探索了一种利用关键帧生成与运动插值的替代策略。
---
Figure 11. Oracle Trajectories and Motion Generation This figure illustrates the successful Complex Task completion of two Primitive Tasks. The top pair of images depict the oracle trajectories, while the bottom pair represents the sequential motion generated.
采用插值法作为完成复杂任务的运动生成器。我们在GOAL $^{[57]}$中采用GNet和MNet，在FAVOR $^{[37]}$中采用INet。这些模型遵循先生成关键帧中的手-物交互，再在这些帧内生成中间交互轨迹的模式。整个生成过程分为三个阶段：第一阶段，GNet根据物体的初始姿态和最终姿态生成静态抓取姿势；随后，MNet生成用于接近物体和远离物体的运动轨迹；最后阶段，将物体运动轨迹中的交替物体姿态输入INet，以生成交互过程中的中间运动。根据物体预设轨迹生成的一系列人体动作如图11所示。图11左侧的图像展示了接近并使用刀子切梨的连续动作，这代表了属于基本任务类别下与刀子相关的“切割”可操作属性；右侧图像则展示了提起瓶子并将瓶内液体倒入平底锅的序列，这体现了与瓶子相关的“倾倒”基本任务可操作属性。
---
## F. 数据集检查
## F.1. 任务列表
Table 7. Collected Affordances and Designed Primitive Tasks.
| Scenario | Affordance | Affordance Instantiation | Primitive Task |
| kitchen table | <be rearranged, ><store securely, sth><contain, sth> | | rearrange |
| <flow in, sth><pour, sth><shake, sth> | pourpour shake |
| <secure, sth> | | screwunscrewcapuncapgripscoopscrapecutstirspreadassemblewipe |
| <grip, sth><scoop, sth><scrape, sth><cut, sth><stir, sth><spread, sth><assemble into, sth><wipe, sth><heat with microwave, sth><contain, sth> | | place insidesaketakepartagemateagemate |
| <secure, sth> | | closegateopengatepressbuttontriggerlever |
| <control, sth> | | pressbuttontriggerlever |
| <weigh, sth><support, sth> | | placeontoperange |
| study room table | <be rearranged, ><store securely, sth><contain, sth> | | rearrange |
| <secure, sth> | | <place inside, sth><take outside, sth> |
| <secure, sth> | | <cover, sth><uncover, sth><shut, sth><open, sth>putonlidremovelidpulloutdrawerpushindrawer |
| <illuminate, sth><connect to, sth> | | <connect to, power socket><connectfrom, powersocket><connectto, usb><connectfrom, usb><connect to, lightbulbsocket><connectfrom, lightbulbsocket> |
| <shear, paper><secure, sth> | | <cap, pen tip><uncap, pen tip>capthepenremovethepencapwriteonpaperwriteonwhiteboardbrushwhiteboardsharpennpencilsharpencilstaplepapertogetherwriteonpaperwriteonwhiteboardshearpaperstaplepapertogetherclosebookopenbook |
| <write/draw, sth> | | <brush, whiteboard><be sharpen by, sth><sharpen, pencil><stapletogether, paper><be written/drawn by, pen/pencil> |
| <be sheared by, scissors><be stapled together by, stapler><be turn, > | | <display, sth><protect, sth> |
| <control, sth> | | <open, laptop lid><close, laptop lid>openlaptoplidcloselaptoplidusekeyboardusemouse |
---
Table 7. Collected Affordances and Designed Primitive Tasks.
| Scenario | Affordance | Affordance Instantiation | Primitive Task |
| <cultivate, flowers>\n<be cultivated in, sth> | | use gamecontroller\nput flower into vase\nput flower into vase |
| demo chem lab | <be rearranged, .>\n<store securely, experiment substances>\n<contain, experiment substances> | | rearrange |
| <secure, experiment substances> | | pour in lab\npour in lab\nshake lab container |
| <contain, experiment substances> | | screw\nunscrew\ncap\nuncap |
| <be heated by, alcohol lamp> | | pour in lab\npour in lab\nshake lab container\nheat beaker/flask\nheat test tube\nstir experiment substances\nignite alcohol lamp\nheat beaker/flask\nheat test tube\nput off alcohol lamp\nignite alcohol lamp\nhold test tube\nplace asbestos mesh\nplace asbestos mesh |
| <be ignited, .>\n<heat, lab container> | | heating the beaker/flask\nheating the beaker/flask\nheating the beaker/flask\nheating the beaker/flask\nheating the beaker/flask\nheating the beaker/flask\nheating the beaker/flask\nheating the beaker/flask\nhe heating the beaker/flask |
| <put off, alcohol lamp>\n<ignite, alcohol lamp>\n<clamp, test tube>\n<conduct heat to, lab container>\n<support, lab container> | | heating the beaker/flask\nheating the beaker/flask\n |
Table 7. Collected Affordances and Designed Primitive Tasks. The first column records the manipulation scenarios. The second column lists collected affordances of object instances and parts. The affordances of object parts are indented below their parent instance-level affordance. The third column lists the instantiations of object affordances. These instantiations are bound to certain object attributes, e.g. is bound to actual screws on the bottle's opening and cap. The fourth column lists the designed Primitives corresponding to the affordances. Some Primitives are set to gray for these Primitives are difficult to demonstrate and capture in individual. Demonstrations of these tasks are embedded within Complex Task demonstrations.
| Scenario | Complex Task |
| kitchen table | heat with microwave oven; weigh with scale; scoop and pour; scoop and grip; scoop and wipe; scoop and scrape; pour and stir; grip and pour; pour and arrange; weigh with scale and pour; pour and scrape; grip and arrange; weigh with scale and grip; grip and wipe; pour and grip; clean the kitchen table; prepare a cup of hot sweet drink; prepare a bowl of hot soup with salt; prepare a cup of hot sweet fruit tea; prepare a chilled apple platter; prepare a savory fruit salad; prepare a baked sweet donut with sauce; prepare a baked sweet donut with apple slices and jam; prepare a savory baked sweet donut; prepare a cheese-baked sweet donut with tomato sauce; prepare savory baked apple slices with cheese; prepare a chilled fruit platter; make a baked sandwich with a filling of donut and salt; make a sandwich with a filling of tomato sauce and sugar; make a sandwich with a filling of apple slices and donut, adding tomato sauce, mustard sauce, salt, and sugar; make a baked sandwich with a filling of cheese and donut, adding tomato sauce; scoop, unscrew, pour, and screw; grip and scoop; scoop and scoop; scoop and arrange; weigh with scale and scoop; cut and scoop; cut and pour; grip and scoop; cut and grip; grip and assemble; stir and arrange; stir and scrape; scrape and arrange; weigh with scale and assemble; unscrew and pour; pour and screw; uncap and scrape; scrape and cap; uncap, scrape, and cap; scrape and assemble; assemble and arrange; unscrew and heat with microwave; pour and heat with microwave; heat with microwave and pour; heat with microwave and stir; heat with microwave and assemble; cut and heat with microwave; uncap, pour, and cap; prepare a cup of chilled green tea; prepare a cup of apple green tea; prepare a cup of mixed flavor fruit juice; prepare a cup of savory fruit juice milk tea; prepare a cup of pear milk tea with fruit jam; prepare a cup of savory honey fruit juice milk tea; prepare a cup of savory strawberry orange juice mixed with milk; uncap and scoop; prepare a cup of wine; prepare a cup of milk tea; prepare a cup of chilled fruit tea; prepare a cup of honey coffee; prepare a cup of chilled juice milk with jam; prepare a cup of chilled sweet milk tea; |
---
| Scenario | Complex Task |
| study room table | put into box; take out of box; put into drawer; take out of drawer; ready the laptop on the desktop for work; tidy up the desktop with the laptop after work; ready the laptop on the desktop for entertainment; illuminate the desktop; sharpen the pencil and write; tidy up the desktop with the laptop after entertainment; tidy up the desktop after paper-cutting; write and bind the paper; design and cut out rectangle shape on the paper; press button and open laptop; press button and close laptop; press button and put into box; press button and take out of box; press button and remove power plug; insert usb and plug in power plug; tidy up the desktop after writing; plug in power plug and press button; design and cut out flower shape on the paper; design and draw on the paper; ready the laptop and the lamp on the desktop for work; design, write and bind the paper; ready the desktop for drawing; take out of drawer, insert usb, and open laptop; put into drawer and put into box; put into drawer, put into box, and close laptop; remove usb, close laptop, and put into drawer; tidy up the desktop after drawing; cut and bind paper; ready the laptop and the lamp on the desktop for entertainment; design, draw and cut out flower shape on the paper; design, draw and bind the paper; tidy up the desktop after binding paper; |
| demo chem lab | transfer and heat liquid in beaker; transfer and heat liquid in conical flask; heat liquid in beaker and transfer liquid; heat liquid in conical flask and transfer liquid; transfer and heat liquid in beaker and transfer liquid out; heat liquid in test tube and transfer liquid; prepare solution through heating; mix liquid; pour in lab and shake in lab; pour in lab and pour in lab; pour in lab and heat test tube; pour in lab and light lamp; stir in lab and pour in lab; stir in lab and heat beaker; shake in lab and pour in lab; shake in lab and heat test tube; shake in lab and heat beaker; heat beaker and put off lamp; heat test tube and put off lamp; light lamp and pour in lab; put off lamp and stir in lab; put off lamp and shake in lab; heat beaker and stir in lab; stack mesh and heat beaker; light lamp and heat beaker; light lamp and heat test tube; pour in lab, shake in lab, and heat test tube; stir in lab, pour in lab, and shake in lab; light lamp, heat beaker, light lamp, heat beaker, and stir in lab; light lamp, heat test tube, and put off lamp; stack mesh, light lamp, and heat beaker; light lamp, heat beaker, and stir in lab; light lamp, heat test tube, and shake in lab; pour in lab, pour in lab, and pour in lab; pour in lab, and heat test tube; shake in lab, and pour in lab; pour in lab, and heat beaker; shake in lab, pour in lab, and heat beaker; put off lamp, and pour in lab; heat beaker, and stir in lab; heat test tube, and put off lamp, and shake in lab; pour in lab, stir in lab, and pour in lab; pour in lab, pour in lab, and pour in lab; pour in lab, and pour in lab; prepare solution in beaker; |
| bathroom table | squeeze tooth paste tube to tooth brush; squeeze tooth paste and stack tooth brush; prepare for teeth brushing. |
Table 8. Recorded Complex Tasks. We list the names of the recorded Complex Tasks here.
## F.2. 可视化
---
倾倒
擦拭
切割
拧开盖子
搅拌
涂抹
盖上盖子
组装
舀取
拧开
拧紧
抓握
刮擦
按下按钮
放入内部
取出外部
打开闸门
关闭闸门
按下按钮
推入抽屉
拉出抽屉
盖上盖子
取下盖子
放入内部
取出外部
放入内部
取出外部
按下按钮
在白板上书写
将纸张钉在一起
打开笔记本电脑盖
合上笔记本电脑盖
用板刷清理白板并撕掉纸张
取下笔帽
给笔盖上盖子
削尖铅笔
把花插入花瓶
在纸上书写
握住试管
拔掉电源插头
插入电源插头
拔出USB设备
插入USB设备
安装灯泡
取下灯泡
倒入实验用试剂
点燃酒精灯
熄灭酒精灯
放置石棉网
搅拌实验用试剂
摇晃实验室容器
加热烧杯/烧瓶
加热试管
拧开牙膏盖
拧紧牙膏盖
Figure 12. Primitives visualization.
挤出牙膏
---
##
Figure 13. Object visualization.
---
Figure 14. Dataset visualization. Human bodies and objects within the scene are rendered onto the captured raw images for visualization.
---
你是一位Python编程专家，需要完成某项双手机器人任务。
场景描述：
{scene_desc}
任务描述：
{task_desc}
代码采用ECS架构，具体定义为
ECS文件
实体定义如下：
实体文件
组件定义如下：
组件文件
系统定义如下：
系统文件
任务是完成该类{read_from_file(scene_path)}中的“exec_task”方法。
你需要从场景中查询原始3D物体，这些物体的名称会作为场景描述中的键，利用它们可以查询不同类型的场景信息。将这些信息构建成Object对象，并实例化PrimitiveTask来完成任务。
任务描述中未提及的物体保持原样即可。
避免使用以下划线开头的方法，要显式指定关键字参数，而非使用**kwargs。
例如：
an_object = Object('object_name',
geometry=self.query_geometry_info('object_name'),
a_primitive_task = PrimitiveTask(an_object.affordance.get_primitive_task_info('primitive_task_name'))
a_primitive_task.execute(src_object=an_object, tgt_object=another_object, trajectory=oracle.generate(a_primitive_task))
无需解释代码，把所有内容都写在方法实现中即可。最终返回exec_task(self)本身。
列表1. 提示模板。Entity表示场景中存在的物体实例；Component用于存储物体的初始状态及可操作属性信息；System则包含用于生成基本动作的运动生成器接口。提示中包含的示例代码展示了查询物体运动轨迹的接口用法。
## 任务
## 任务是将梨切成块，放入平底锅中，再加入一些调料。
## 场景
## 厨房的桌子上放置了以下物体：[刀、梨、瓶子、平底锅]。
def exec_task(self):
# making an object for knife
knife = Object('knife',
geometry=self.query_geometry_info('knife'),
state=self.query_state_info('knife'),
affordance=self.query_affordance_info('knife')
)
## 为梨创建对应的物体
pear = Object('pear',
geometry=self.query_geometry_info('pear'),
state=self.query_state_info('pear'),
affordance=self.query_affordance_info('pear')
)
## 为平底锅创建一个对象
pan = Object('pan',
geometry=self.query_geometry_info('pan'),
state=self.query_state_info('pan'),
affordance=self.query_affordance_info('pan')
)
## 为瓶子创建一个对象
bottle = Object('bottle',
geometry=self.query_geometry_info('bottle'),
state=self.query_state_info('bottle'),
affordance=self.query_affordance_info('bottle')
)
## 创建用于切割的基础任务
cut_pear = PrimitiveTask(knife, affordance.get_primitive_task_info('cut'))
cut_pear.execute(src_object=knife, tgt_object=pear, trajectory=oracle.generate(cut_pear))
## 创建用于将梨放入平底锅的基础任务
put_task = PrimitiveTask(pear.affordance.get_primitive_task_info('put_inside'))
put_task.execute(src_object=pear, tgt_object=pan)
## 创建用于添加调味料的基础任务
add_season = PrimitiveTask(bottle.affordance.get_primitive_task_info('pour'))
add_season.execute(src_object=bottle, tgt_object=pan, trajectory=oracle.generate(add_season))
Listing 2. Primitive Task Planning Success Example.
---
## 任务
## 任务是点燃酒精灯来加热烧杯中的混合物。一开始灯是盖着的。加热时需将烧杯放在带网格的支撑架上。
## 场景
## 在化学实验室的桌子上放置了以下物品：[打火机、酒精灯、烧杯、加热支架、网格]
def exec_task(self):
## 打开灯罩
alcohol_lamp = Object('alcohol lamp',
geometry=self.query_geometry_info('alcohol_lamp'),
affordance=self.query_affordance_info('alcohol_lamp'),
state=self.query_state_info('alcohol_lamp')
)
uncap_alcohol_lamp_task = PrimitiveTask(alcohol_lamp, affordance.get_primitive_task_info('uncap'))
uncap_alcohol_lamp_task_execute(src_object=alcohol_lamp, trajectory=oracle.generate(uncap_alcohol_lamp_task))
# Light
lighter = Object('lighter',
geometry=self.query_geometry_info('lighter'),
affordance=self.query_affordance_info('lighter')
state=self.query_state_info('lighter')
)
light_alcohol_lamp_task = PrimitiveTask(lighter, affordance.get_primitive_task_info('light_lamp'))
light_alcohol_lamp_task_execute(src_object=lighter, tgt_object=alcohol_lamp, trajectory=oracle.generate(light_alcohol_lamp_task))
# Arrange (put on the mesh support)
beaker = Object('beaker',
geometry=self.query_geometry_info('beaker'),
affordance=self.query_affordance_info('beaker')
state=self.query_state_info('beaker')
heat_support = Object('heat_support',
geometry=self.query_geometry_info('heat_support'),
affordance=self.query_affordance_info('heat_support'),
state=self.query_state_info('heat_support')
mesh = Object('mesh',
geometry=self.query_geometry_info('mesh'),
affordance=self.query_affordance_info('mesh')
state=self.query_state_info('mesh')
put_beaker_on_mesh_task = PrimitiveTask(beaker, affordance.get_primitive_task_info('arrange'))
put_beaker_on_mesh_task_execute(src_object=beaker, tgt_object=heat_support, trajectory=oracle.generate(put_beaker_on_mesh_task))
# Stack mesh to heat
stack_mesh_task = PrimitiveTask(mesh_affordance.get_primitive_task_info('stack_mesh'))
stack_mesh_task_execute(src_object=mesh, tgt_object=beaker, trajectory=oracle.generate(stack_mesh_task))
heat_beaker = PrimitiveTask(alcohol_lamp, affordance.get_primitive_task_info('heat_beaker'))
heat_beaker_execute(src_object=alcohol_lamp, tgt_object=beaker, trajectory=oracle.generate(heat_beaker))
Listing 3. Primitive Task Planning Fail Example. One violation of dependency occurs (line 41): the extra place onto primitive erroneously positions the beaker upon the support prior to the asbestos mesh's placement on the support. This blocks the correct execution path that requires placing the asbestos mesh before heating the beaker.