# GigaHands：一个大规模的双手动作标注数据集
Figure 1. GigaHands is a massive dataset of human bimanual activities with paired text annotations. Each column above shows an activity sequence from the dataset. The dataset covers diverse 3D hand activities, including hand-object interactions (blue) across object scales, gestures (orange), and self-interactions (red). Each clip is paired with descriptive text and 51 camera views, enabling radiance field reconstruction. The bottom row show other annotations in the dataset including hand shape, object shape and pose (left half images). The right half images show novel views from dynamic radiance field fitting.
## 摘要
理解人类双手的动作是人工智能与机器人领域的一个关键问题。由于现有数据集在规模、多样化手部动作的覆盖范围以及详细标注方面存在不足，我们无法构建大规模的双手动作模型。我们推出的GigaHands是一个大规模标注数据集，它记录了来自56名受试者的34小时双手动作数据，涉及417个物体，共生成14000个运动片段，这些片段源自1.83亿帧图像，并配有84000条文本标注。我们的无标记捕捉方案与数据采集协议能够实现三维手部及物体的全自动估计，同时大幅降低文本标注所需的工作量。GigaHands的庞大规模与多样性可支撑广泛的应用，包括基于文本的动作合成、手部动作描述生成以及动态辐射场重建。
### 1. 引言
“人类的手是进化的奇迹，其复杂的结构与能力使我们能够以其他物种无法企及的方式操控环境。”
---
“它以其他物种无法比拟的方式塑造着环境。”
F.R. Wilson [^108]
从烹饪美食所需的精细操作，到敲击这段文字时手指的快速移动，双手始终在塑造我们的环境并与他人交流。让机器能够理解并复制人类双手的非凡能力，是人工智能与机器人领域面临的重大挑战之一。实现这一目标的一种途径是在大规模数据集上训练模型，近期的一些大模型正是采用这种策略：它们分别从万亿级文本词元[^27]、数十亿对文本-图像数据[^89]、数千万个三维形状[^25]或数百万条机器人运动轨迹[^12] [^13] [^46] [^73]中学习知识。然而，目前尚不存在针对自然手部动作的大规模数据集。
获取大规模的三维手部动作数据颇具挑战性。最常见的两种数据采集方法分别是利用摄像头在野外或受控摄影棚环境中捕捉手部操作过程。野外数据包括单目互联网视频[^91]、通过可穿戴摄像头采集的第一人称视角视频[^22] [^23] [^32] [^52] [^57] [^79]，或是第三人称摄像头的多视角视频[^30] [^62] [^91]。不过这类数据较为稀疏，校准难度大且噪声多，导致三维运动重建的精度有限，尤其是针对物体而言。另一种方法是在摄影棚环境中让受试者处于摄像头密集的区域，并可借助标记点来实现精准重建。但这种人为设定的场景缺乏真实世界背景，限制了数据的多样性；而且基于标记点的追踪方式[^8] [^28] [^29] [^55] [^58] [^59] [^101] [^111] [^116]会妨碍自然的交互过程。
我们通过推出GigaHands来解决阻碍手部数据集扩展的数据来源问题。GigaHands是一个多样化、大规模且标注完整的3D双手活动数据集（见图1）。据我们所知，GigaHands是规模最大的双手活动数据集，包含超过1.83亿个包含两只手的独特图像帧（$0.37 \times 10^9$种独特的手部姿态，因此得名GigaHands）。我们采用多摄像头无标记捕捉系统来获取精确的双手与物体交互活动、手势以及自我接触数据。为了克服摄影棚捕捉的限制，我们设计了程序化活动诱发协议，以纳入尽可能多的现实世界中的自然场景动作。为确保数据的多样性能够反映真实世界情况，我们采集了56名受试者与417个现实物体互动的数据。我们数据集中的所有图像都经过了完整标注：详细的活动文本描述；3D手部形状和姿态；MANO [^86] 手部网格；3D物体形状、姿态和外观；手部/物体分割掩码；2D/3D手部关键点；相机姿态。我们通过程序化的“指令引导标注”策略来减少人工标注工作量，即给受试者提供详细指令，从而降低采集后的标注难度。总共而言，我们收集了34小时带文本标注的双手活动数据、1.3万个3D运动序列，以及超过1.4万个经过处理后的3D运动片段，这些片段配有8.4万个详细的原子级文本描述标注。我们的3D手部运动片段规模超过了所有现有3D双手活动数据集的总和 [^57] [^58] [^116]，而且我们文本标注中所包含的动词范围也比任何其他手部数据集都要广，即便是那些在野外采集的数据集也不例外 [^32] [^33]。
GigaHands能够在运动生成、机器人技术以及动态3D重建等应用中释放新能力 [^81]。为证明这一点，我们展示了该数据集在文本驱动的动作合成和手部运动描述方面带来的改进，以及动态辐射场重建的示例。总结来说，我们的贡献如下：
• 我们推出了GigaHands，这是一个大规模、多样化且标注完整的3D双手活动数据集。它包含34小时的活动数据、1.4万个配有8.4万个文本标注的手部运动片段，以及超过1.83亿张独特的手部图像。
- 我们采用了一种程序化的数据采集策略，该策略能确保活动的多样性和详细的原子级文本描述，同时最大限度地减少人工工作量。我们还构建了一个精确且全自动的流程，用于估算3D手部/物体形状与姿态、分割结果以及相机姿态。
- 我们展示了该数据集的规模所带来的应用潜力，包括3D手部运动生成、手部运动描述以及动态语义场景重建。
### 2. 相关工作
#### 2.1. 手部运动数据的来源
手部运动数据主要有四种来源，每种来源在真实性、多样性、准确性和实用性方面各有优劣。
静态姿态涉及从单个静态位置进行的运动规划 [^5] [^7] [^10] [^93]、策略学习 [^19] [^60]，或是运动合成 [^20] [^117] [^118]。虽然这些方法能提供可靠的3D数据，但缺乏语义丰富性，导致产生的运动与现实世界活动脱节，无法完全体现人类多样化的运动方式。
来自游戏引擎[^126]、增强现实[^66]、虚拟现实[^38]或模拟场景[^39] [^63] [^105]的合成数据，能够生成可控且精确的3D运动序列。尽管这些数据可靠性较高，但往往难以捕捉真实人类动作的细微之处，从而限制了其在建模自然手部行为方面的应用。
在现实世界的自然场景中，需借助可穿戴或便携式传感器来采集手部交互数据，这些数据可能来自第一视角[^22] [^23] [^32] [^52] [^57] [^79]、第三人称视角[^30] [^62] [^91]，或是这两种视角的混合[^33] [^95]，进而得到逼真且自然的运动数据。然而，由于传感器部署有限且精度受限，要获取双手及物体的可靠3D运动数据颇具挑战性。此外，标注这类数据需要大量人力，某些自接触类动作也难以被精准记录。
实验室环境会将参与者置于配备大量传感器的受控环境中，从而能够可靠地采集诸如3D手部运动、物体运动、接触区域、音频以及触觉数据等详细信息。不过，在实验室中采集数据过程繁琐，且可能无法反映现实场景的多样性。参与者在不熟悉的环境中，尤其是身上佩戴着可穿戴传感器时，很难做出自然的动作。例如，动作捕捉系统需要在人体上粘贴标记点，这既降低了视觉上的真实感，又限制了双手之间的自接触等动作，还需额外的后期处理才能恢复逼真的外观。另一方面，无标记点的动作捕捉系统能提供更逼真的视觉效果并促进自然动作，但可能会牺牲捕捉精度。GigaHands通过在活动诱发过程中模拟自然场景，并利用无标记点动作捕捉技术估算精确的3D运动，从而在精度、多样性、真实感和实用性之间取得了平衡。
---
配备大量传感器的设备能够可靠地采集详细信息，例如三维手部动作 [^45] [^54] [^55] [^72] [^120] [^122]、物体运动 [^8] [^18] [^29] [^36] [^37] [^48] [^58] [^59] [^111] [^116]、接触区域 [^28] [^80] [^101]、音频 [^44] [^49] [^71] [^113] 以及触觉数据 [^14] [^31] [^78] [^100]。不过，在摄影棚中采集数据十分繁琐，且可能无法反映现实场景的多样性。参与者在不熟悉的环境中很难做出自然动作，尤其是当他们需要佩戴可穿戴传感器时。例如，动作捕捉（MoCap）系统 [^8] [^28] [^29] [^55] [^58] [^59] [^101] [^111] [^116] 需要在人体上粘贴标记点，这会降低视觉上的真实感，还会妨碍双手相互接触等动作，因此还需要额外的后期处理才能恢复逼真的外观。另一方面，无标记动作捕捉系统 [^18] [^36] [^37] [^45] [^48] [^54] [^72] [^80] [^120] [^122] 能提供更真实的视觉效果并鼓励自然动作，但可能会牺牲捕捉精度。GigaHands通过在活动诱发过程中模拟真实环境，并从无标记动作捕捉数据中估算出精确的三维动作，从而兼顾了精度、多样性、真实感和实用性。
#### 2.2. 手部数据集与标注
手部数据集在来源和规模上各不相同，每种数据集均为特定目的而构建，并配有各种注释信息。手部动作是最常见的注释类型，表现为边界框 [^30] [^79] [^91]、分割掩码 [^24] [^52] [^70]、2D关键点 [^6] [^43] [^61] [^90]、3D关键点 [^82] [^99] [^102] [^106]，或是参数化模型的参数 [^55] [^77] [^97]。根据数据源和所需细节的不同，这些注释可通过人工标注 [^106]、合成 [^66] [^67] [^92] [^126]、标记物和传感器 [^29] [^45] [^54] [^104] [^115] [^120] [^122]、多视角自举法 [^65] [^96]、迭代训练的关键点提取网络 [^72] [^127]、多视角注释 [^72]，或音频优化 [^44] 来获取。半自动和自动标注技术显著提升了可扩展性。
由于手部与物体之间频繁发生交互，因此提供了物体注释信息。这些注释可通过人工标注 [^18] [^57] [^98]、合成 [^21] [^39]、动作捕捉技术 [^8] [^28] [^58] [^59] [^111] [^112] [^116] [^124]、手-物重建或检索 [^15] [^64] [^109]，或多视角RGB-D数据 [^36] [^37] [^48] 获得。我们的研究表明，多视角RGB数据足以实现自动注释，从而提升可扩展性。
文本注释。为了捕捉人类手部动作中丰富的语义信息，某些数据集提供了文本注释。这些注释可能包括动作类型 [^26] [^30] [^32] [^47] [^79]、原子动作描述 [^22] [^23] [^33] [^52] [^90]、活动叙述 [^22] [^23] [^33] [^62]、活动评论 [^17] [^32] [^33] [^116]、物体可供性 [^42] [^58] [^111] [^116]，或身体动态 [^114] [^122] [^123]。通常，这些注释由人工提供，尤其是在无控制、非脚本化的环境中收集数据时 [^22] [^23] [^32] [^33]。
Table 1. Comparisons of 3D bimanual motion datasets. Dataset names are highlighted with different colors if it has no text annotations (gray), action type (green), sparse description (red), and dense description (blue). See the supp. document for the full table.
| Name | mins | motions | poses | views | frames | sub. | obj. |
| w/o Object Annotation |
| AssemblyHands [^72] | 630 | 62 | 203k | 12 | 3.03M | 34 | / |
| Ego-Exo4D [^33] | / | / | 4.4M | 5-6 | / | 740 | / |
| w/ Object Annotation |
| HOI4D [^57] | 1,333 | 4k | 1.2M | 1 | 2.4M | 4 | 800 |
| ARCTIC [^28] | 121 | 339 | 218k | 9 | 2.1M | 10 | 11 |
| TACO [^58] | 202 | 2.3k | 363k | 13 | 4.7M | 14 | 196 |
| OakInk2 [^116] | 557 | 2.8k | 993k | 4 | 4.01M | 9 | 75 |
| HOT3D [^8] | 833 | 4.1k | 1.7M | 2-3 | 3.7M | 19 | 33 |
| GigaHands (Ours) | 2,034 | 13.9k | 3.7M | 51 | 183M | 56 | 417 |
即便是在摄影棚环境中，也常常需要后期处理才能提取出有意义的动作片段 [^52] [^116]。我们提出了一种流程化的“指令-注释”流水线，旨在最大化语义可解释性并最小化注释工作量。
接触点来源于人工标注 [^91]、视频定位 [^69]、热传感器 [^9] [^11] 或基于采集数据的几何分析 [^80] [^101] [^125]。借助我们的数据集，我们能够通过对表面网格和密度场进行几何分析来提取接触区域。
### 3. GigaHands 数据集
数据集特征。GigaHands 是一个规模庞大且种类丰富的双手手部活动数据集，配有详细的标注信息。它涵盖了多种场景，包括手部与物体的交互、手势动作以及 56 名受试者进行的自我接触，这些受试者共使用了 417 个物体（见表 1）。该数据集包含 2,034 分钟的双手手部活动视频，其时长超过了现有的任何 3D 手部运动数据集。与其他未标注的数据集 [^8] [^28]、仅包含运动类型标注的数据集 [^57] [^58] [^72]，或仅提供稀疏运动描述的数据集 [^33] [^116] 相比，GigaHands 为所有采集到的活动提供了详细的文本标注。该数据集包含总计 13,000 条指令驱动的运动序列、14,000 个标注的运动片段以及 84,000 条增强型文本描述，规模远超其他任何数据集。此外，GigaHands 还收录了 370 万个双手 3D 手部姿态，这些姿态既以 3D 关键点形式呈现，也以 MANO [^86] 手部网格形式呈现，其规模可与目前最大的标注 3D 手部姿态数据集 Ego-Exo4D [^33] 相媲美。不过，与仅对选定的非连续帧进行标注的 Ego-Exo4D 不同，GigaHands 提供的是连续的双手 3D 手部姿态和形状数据。每个运动片段均通过 51 个摄像头视角采集，共生成 1.83 亿帧 RGB 图像（以及 3.66 亿张独特的手部图像）。这种多视角设置为动态 3D 辐射场重建等新应用创造了条件。此外，
---
Figure 2. Dataset Diversity. The left and middle figures illustrate the diversity of pose and motion variations in GigaHands, visualized using t-SNE embeddings. Some points along the convex hull are highlighted with their corresponding text instructions, showcasing unique motions captured in our dataset. The right figure compares the verb sets among different datasets using an UpSet visualization [^50]. Each column represents the number of verbs exclusive to specific subsets of datasets, indicated by the connected dots below the columns. The rows indicate the total verb count in each dataset. GigaHands contains more verbs and more exclusive verbs compared to other datasets.
该数据集还包含手部/物体分割掩码、3D 物体形状、姿态和外观、2D/3D 关键点以及摄像头姿态等标注信息。
多样化的手部姿态和运动。图 2（左）通过双手 3D 关键点的 t-SNE [^107] 聚类，对比了各数据集中的手部姿态多样性。同样，图 2（中）通过用于运动重建的变分自编码器隐码的 t-SNE 聚类，展示了手部运动的多样性。在双手手部姿态和运动方面，GigaHands 展现出的多样性明显优于现有数据集。此外，数据集边界附近的数据点似乎对应着数据集中独有的文本标注，这表明多样的动词能够带来多样的手部姿态和运动。
多样化的文本标注。如图 2（右）所示，通过动词数量来衡量，GigaHands 包含的活动种类比现有数据集更加丰富 [^32] [^33] [^116]。由于每个动词对应一种活动，因此动词种类越丰富，活动多样性就越高。GigaHands 包含的动词数量最多，达 1,467 个，其中 580 个为数据集独有。实现这种多样性的关键在于我们采用的“指令驱动标注”策略（见第 4.3 节），该策略用于自动化获取带标签的活动数据。有关动词来源及详细的动词对比信息，请参阅补充文档。
多样化的物体。图3展示了GigaHands中包含的一些桌面物体。我们为刚性物体提供了带有相关纹理的3D网格[^2]，为非刚性物体提供了多视角分割掩码。与大多数数据集不同，我们还纳入了难以捕捉的小尺寸物体（如笔）。我们采用单视角重建技术[^3]来获取这些小物体的网格。总体而言，该数据集包含417个物体，其中310个为多视角扫描网格，31个为单视角生成的网格。
Figure 3. Diverse Objects and Frequent Hand Contact Regions. GigaHands provide objects (left) spanning diverse scenarios, including cooking, office working, crafting, entertainment, and housework. The diverse activities result in contact regions (right) spanning both the front and back of both hands.
多样化的接触情况。由于我们有双手和物体的3D网格，因此可以依据[^101]中的方法，估算双手上的接触区域。图3展示了从数据集中随机采样的帧所累积的接触区域。结果揭示了多样的接触区域，包括手指之间以及手背上的区域（例如出拳时会用到手背）。由于大多数受试者均为右利手，右手与物体的交互频率高于左手。
### 4. 数据集获取
我们介绍了数据采集流程，该流程在兼顾真实性、多样性和准确性的同时，减少了标注工作量。我们的流程名为“指令引导标注”（详见图4），包含几个关键组成部分：
• 程序化指令生成（第4.1节）：用于生成指导参与者采集数据的指令。
• 拍摄（第4.2节）：使用无标记多摄像头系统记录高质量的手部活动序列。
---
Figure 4. Instruct-to-Annotate Pipeline. The instruction elicitation process (left yellow block) creates atomic action-level instruction scripts in a temporally smooth order, structured within scenes. This is achieved by parsing action datasets, grouping verbs into a pool, structuring scenarios, and generating scene scripts. During filming, subjects act according to these scripts, producing recorded motion sequences. Annotators then process these sequences (right blue block) by segmenting them into clips and annotating unscripted motions.
- 动作标注与文本增强（第4.3节）：通过将序列拆分为更短的片段并添加精准标注来优化序列。
- 手部动作（第4.4节）与物体运动估计（第4.5节）：根据采集的多视角RGB视频获取详细的3D手部及物体运动信息。
#### 4.1. 指令生成
为确保手部活动的多样性并减少标注工作量，“指令引导标注”流程首先采用程序化指令生成协议。我们首先从多个数据集中提取原子动作[^32] [^33] [^58] [^116]，从中整理出对应这些动作的动词集合。为了引导受试者执行这些动词对应的动作，我们手动将每个动词与多个物体关联起来，并在大语言模型[^4]的辅助下，把动词和物体归类为烹饪与进食、办公、手工制作、娱乐、家务等不同场景。随后我们将这些场景组织成物体可共存的场景。利用大语言模型，我们把这些场景整理成按时间顺序流畅排列、涉及相关物体和动词的活动列表，并自动生成详细的指令脚本。这些脚本包含5个场景、25个场景片段、191项活动以及1370条指令，共涉及533个动词。详情请参阅补充文档。
#### 4.2. 拍摄
硬件设备。为了采集数据集，我们采用了一种定制设计的多摄像头桌面采集系统，该系统由均匀排列在一个立方体采集区域内的51个RGB摄像头组成，立方体的每个面上都布设了$3 \times 3$网格状的摄像头，且这些摄像头均由LED灯均匀照明。立方体内部有一块透明玻璃表面，用作支撑物体的平台。每台摄像头以30帧/秒的速率拍摄，分辨率为$1280 \times 720$。各摄像头通过软件同步，其时间相位偏差小于3毫秒。摄像头的内部参数和外部参数则是借助COLMAP [^87] [^88]以及基准标记来获取。
拍摄过程。拍摄期间，受试者按照给定的指令执行动作，双手始终保持在拍摄区域内。指令会被转换为音频并依次播放以引导参与者，同时操作员负责播放每条音频指令并记录对应的动作序列。如果某个动作的结束状态与下一条指令不符，就会给出纠正指令并重新录制，以确保动作过渡顺畅并减少标注工作量。这种方式能确保所有录制的序列都与预先编写或录制的指令相对应。
#### 4.3. 动作标注与增强
尽管每段拍摄的动作序列都配有相应的指令，但由于两个原因仍需进行人工标注。首先，大语言模型生成的指令偶尔会出现不一致或幻觉现象；其次，受试者可能会误解指令，或在拍摄过程中添加非故意的动作。为了鼓励受试者自然、自由地表演，我们特意保留了这些录像。随后，标注员会将这类序列拆分为单个片段，并对原始指令中未包含的动作进行标注。这一过程最终生成了描述与片段相匹配的更精准的数据集。总共而言，我们将13k个动作序列优化为14k个动作片段。
为了进一步优化文本描述，我们利用大语言模型对每个描述进行5次改写，从而为每个动作片段提供多种文本变体。这样一来，原本的14k个动作片段便扩展为84k组动作-文本对，其中包含1,467个独特的动词。增强所用的提示词及示例详见补充文档。
#### 4.4. 手部动作估计
GigaHands提供了详细的手部动作数据，包括二维和三维手部关键点，以及双手的MANO [^86]网格模型。由于现有的手部形状和姿态估计方法效果欠佳，我们构建了属于自己的混合方法。
---
我们首先利用YOLOv8 [^85]从视频中获取手部的边界框。接着，从通过HaMeR [^77]估计出的MANO网格中提取二维关键点，并利用ViTPose [^110]确定手的左右属性。由于HaMeR网格缺乏精确的深度信息，无法直接投入使用。借助相机参数，我们跨视角对二维关键点进行三角测量 [^97]，从而得到精确的三维位置。为确保序列的时间连续性，我们对二维和三维关键点均应用了一欧元滤波器 [^16]。有了边界框以及二维和三维关键点后，我们便按照EasyMoCap流程 [^86]来拟合MANO参数化手部模型 [^1]。最终，我们实现了提取连贯且精确的手部动作的完全自动化流程。各步骤的详细信息和评估结果详见补充文档。
#### 4.5. 物体动作估计
GigaHands还提供了以3D形状和6D姿态表示的3D物体运动。利用预扫描或单视图重建得到的目标网格，我们借助多视图约束来追踪这些物体。首先，我们在每个视图中将目标物体从背景中分割出来。我们使用DINOv2 [^74] 在每秒1帧的采样帧上检测视频中的显著物体。结合物体文本描述以及来自多个视图的渲染模板网格，我们利用Grounding DINO [^56] 选出整个视频中与模板网格最匹配的Top-k个边界框。为了排除误检结果，我们使用OpenCLIP [^41] 过滤掉与负向提示词对齐的边界框。以这些正向边界框作为提示，我们使用SAM2 [^84] 在整个视频中分割物体掩码。
由于现有的物体姿态估计方法效果都不理想 [^75]，我们决定构建一种利用密集多视图环境的鲁棒方法。我们采用可微渲染方法 [^83]，并借助多视图掩码进行监督训练。为了初始化平移和旋转参数，我们首先使用Instant-NGP [^68] 构建辐射场。我们以辐射场的中心作为物体平移的初始值。为解决物体对称性带来的问题，我们利用物体外观来初始化旋转参数。借鉴FoundPose [^75] 的思路，我们用多种初始旋转角度渲染模板网格，再利用DINOv2特征找到最佳匹配项，从而聚合跨视图信息。即便在复杂、杂乱的场景中，这一过程也能给出精确的物体运动估计结果。更多细节以及各步骤的评估结果详见补充文档。
### 5. 应用与实验
我们展示了GigaHands在多个需要大规模数据的应用场景中的实用性，包括文本驱动的手部运动合成（第5.1节）、针对本数据集的运动描述生成（第5.2节）以及针对自然场景数据集的运动描述生成（第5.2节），还有动态辐射场重建（第5.2节）。
Table 2. Quantitative results for text-driven motion synthesis with models trained on different datasets. upper bound indicates performance calculated with the ground truth. We report the mean of 20 evaluations, and → means the closer to the upper bound the better. The model trained on GigaHands performs best on most metrics.
| Dataset | R Precision(%) $\uparrow$ | MM Dist. $\downarrow$ | FID $\downarrow$ | Div. $\rightarrow$ | MM. $\uparrow$ |
| @1 | @2 | @3 |
| upper bound | 64.4 | 86.2 | 89.4 | 2.86 | 0.045 | 14.2 | - |
| TACO [^58] | 18.9 | 37.7 | 52.9 | 7.39 | 11.0 | 11.1 | 6.83 |
| upper bound | 50.4 | 71.2 | 81.1 | 3.67 | 0.022 | 9.30 | - |
| OakInk2 [^116] | 17.9 | 31.7 | 47.9 | 7.75 | 19.6 | 6.88 | 3.45 |
| upper bound | 77.4 | 88.8 | 91.3 | 2.96 | 0.002 | 11.9 | - |
| GigaHands | 31.2 | 44.7 | 53.1 | 6.68 | 4.70 | 10.5 | 9.11 |
tion 5.3). More results can be found in the supp. document.
#### 5.1. 文本驱动的手部运动合成
生成多样且复杂的运动对于训练虚拟智能体和机器人操控系统至关重要。我们利用GigaHands训练文本驱动的手部运动合成模型，从而克服那些依赖严格条件 [^116] 或由于数据受限只能生成简单动作 [^17] 的方法的局限性。事实证明，与其他数据集 [^58] [^116] 以及GigaHands的子集相比，GigaHands的规模优势能带来更好的性能。
训练与评估流程。我们将GigaHands按16:3:1的比例划分为训练集、测试集和验证集。在各类标注数据中，我们选取了双手共42个3D关键点作为模型训练的表示形式（关于其他表示形式的分析详见补充文档）。为了评估生成运动的自然度、多样性以及与文本描述的对齐程度，我们采用了 [^34] 中提出的指标，包括R-Precision、多模态距离（MM Dist）、弗里谢特 inception 距离（FID）、多样性（Div.）和多模态性（MM.）。计算FID和Div.时，我们训练了一个运动自编码器来定义压缩后的运动空间。对于R-Precision、MM Dist和MM.，我们遵循 [^34] 的方法，利用对比学习构建联合运动-文本嵌入空间。针对每个数据集和子集，特征提取器都是独立训练的。
实验结果。我们在表2中报告了使用T2M-GPT [^119] 骨干网络进行文本到手部运动合成的性能（更多关于基于GRU的[^35] 和扩散模型[^103] 架构的结果详见补充文档）。由于评估所用的嵌入空间是在不同测试集上训练的，我们也在真实测试数据上评估各项指标，以反映每个测试集的上限性能。在除MM Dist.之外的所有指标上，基于GigaHands训练的模型表现均优于其他模型；其R-Precision值更高，这得益于更好的文本-运动对齐效果。与同样包含手部运动文本描述的OakInk2相比，GigaHands在FID、多样性以及多模态得分方面均表现出显著优势。
---
Figure 5. Generated motions from models trained on different datasets. Texts highlighted in green, orange, and blue come from the OakInk2, TACO, and GigaHands datasets. In the bottom two rows, hand meshes highlighted in these colors are generated by models trained on the corresponding datasets. The model trained on GigaHands can generate diverse motions from a single text (right four columns) and accurate motion with text from other datasets (left two columns). Darker color indicates later frame in the sequence.
Figure 6. Effect of dataset size on motion reconstruction and text-to-motion generation performance. The x-axis shows the percentage of training data used (10%, 20%, 50%, 80%, and 100%), and the y-axis displays performance metrics: FID, MM Dist., Top-1, and Top-3 accuracy. Larger datasets consistently improve performance across all metrics, highlighting the benefits of increased data scale.
这很可能是由于其数据集多样性更丰富所致。图5展示了使用GigaHands训练的T2M-GPT模型，根据来自GigaHands、OakInk和TACO测试集的文本输入生成的手部运动。即便没有物体几何信息输入，我们的模型仍能生成合理的手部形状和姿势以实现物体操控，这得益于数据集中物体的多样性。我们的模型也能利用其他数据集中的文本生成合理的运动，这体现了GigaHands的全面性。
数据规模的影响。图6展示了数据集大小对运动重建和文本到运动生成任务的影响。我们基于T2M-GPT架构，分别使用训练集的10%、20%、50%、80%和100%来训练用于运动重建的运动VQ-VAE模型，以及用于文本到运动生成的生成式预训练Transformer模型，并在同一测试集上进行评估。我们评估的指标包括FID、MM Dist.以及Top 1和Top 3准确率。随着数据集规模的扩大，大多数指标均持续改善，这证明了大规模数据的价值。
#### 5.2. 手部运动描述生成
为手部运动生成描述有助于解读动作意图。我们在GigaHands数据集上训练了运动描述模型，并为其测试集运动生成描述；我们也展示了在野外数据集中生成描述示例。
训练与评估方案。我们采用TM2T [^35] 作为模型骨干。在文本-运动对齐评估方面，我们利用第5.1节所述的相同嵌入空间来计算R-Precision和MM Dist.。为了评估与真实文本标注的对齐程度，我们使用BLEU [^76]、ROUGE [^53] 和BERTScore [^121]。由于我们的数据集包含多种动词，我们采用distinct-n [^51] 和Pairwise BLEU [^94] 来评估文本多样性。由于TACO缺乏文本标注，我们将其三元组标签（<动作、工具、物体>）视为脚本，从而将TACO中的运动描述任务转化为分类任务。
Table 3. Quantitative evaluation for motion captioning with models trained on different datasets. GigaHands performs best on most metrics.
| Datasets | RP@1(%) $\uparrow$ | RP@2(%) $\uparrow$ | RP@3(%) $\uparrow$ | MM Dist. $\downarrow$ |
| upper bound | 64.4 | 86.2 | 94.2 | 2.86 |
| TACO [^58] | 43.5 | 59.6 | 68.5 | 5.84 |
| upper bound | 51.1 | 69.5 | 79.6 | 3.67 |
| Oakink2 [^116] | 40.4 | 58.8 | 68.2 | 4.55 |
| upper bound | 75.3 | 89.1 | 93.9 | 2.87 |
| GigaHands | 57.0 | 66.1 | 69.8 | 5.37 |
Table 4. Pairewise-BLEU, BLEU@4, ROUGE, distinct-n, and BERTScore for motion captioning with models trained on different datasets. GigaHands performs best on most metrics.
| Datasets | P-B $\downarrow$ | B@4 $\uparrow$ | R $\uparrow$ | dist.-1(%) $\uparrow$ | dist.-2 $\uparrow$(%) | BScore $\uparrow$ |
| TACO [^58] | 4.59 | 39.4 | 61.2 | 2.03 | 6.35 | 57.1 |
| Oakink2 [^116] | 8.21 | 39.9 | 56.3 | 2.76 | 6.82 | 35.3 |
| GigaHands | 0.916 | 43.1 | 57.7 | 15.3 | 36.9 | 55.4 |
---
Figure 7. Motion captioning results with different datasets. Each column shows a motion sequence, its ground truth text description, and two generated texts. Hand motions highlighted in green, orange, and blue come from OakInk2, TACO, and GigaHands, respectively. Texts highlighted in these colors are generated by models trained on the corresponding datasets. The model trained on GigaHands generates diverse captions from a single motion (right three columns) and accurately captions motions from other datasets (left two columns).
Figure 8. Synthesized test views using 2DGS for the motion ‘zip up the pants,’ displayed at two timesteps and from two viewpoints. The synthesized views faithfully reconstruct the scene by leveraging the ample camera views provided by GigaHands.
GigaHands的手部运动描述生成。图7展示了生成的3D运动描述示例。在GigaHands上训练的模型能生成多样且准确的描述。尽管输入中并未包含物体信息，该模型仍会生成包含合理物体描述的内容。表3和表4报告了在不同数据集上训练的模型的定量性能[^58] [^116]。在GigaHands上训练的模型在运动-文本对齐方面表现更优，尤其是在反映描述多样性的Pairwise BLEU和distinct-n指标上。虽然我们的准确率略低于TACO中简单的三元组描述方式，但我们的方法能生成更丰富、更多样的描述。
无标注数据集的3D手部运动描述生成
GigaHands中包含的丰富动作类型及相关注释，使得基于该数据集训练的模型能够为其他3D手部动作数据集生成描述。图7展示了使用在GigaHands上训练的模型，对TACO和OakInk2数据集中的动作进行描述的效果——这些数据集的动作范围事先已与GigaHands对齐。用GigaHands训练的该模型能为这些其他数据集生成更细粒度的描述。
#### 5.3. 动态辐射场重建
与现有的手部活动数据集不同，GigaHands提供了密集的摄像机视角。这为构建动态辐射场等新应用创造了条件。在提供的51个视角中，我们因光照问题剔除了12个摄像机视角，并随机选取一个作为测试视角，最终用于训练的有38个视角。按照第4.5节的描述，我们对视频中的物体和手部 mask 进行一致性分割。随后，我们采用2DGS [^40] 来进行逐帧辐射场重建，为保持时间一致性，每一帧的初始化都以上一帧为基础。图8展示了一个动作片段的合成测试视角示例。值得注意的是，裤子属于非刚性物体，难以被精准追踪，但GigaHands提供了充足的摄像机视角来捕捉这类非刚性物体（更多结果见补充文档）。
### 6. 结论
我们提出了GigaHands，这是一个规模庞大的双手法部活动标注数据集。该数据集包含来自56名受试者的14k个动作片段，涵盖了他们与417个真实物体互动时产生的3D手部和物体动作，所有数据均从51个摄像机视角采集。它提供了1.83亿张独特的图像帧和8.4万条文本描述，可支持广泛的应用。我们证明了该数据集的规模和多样性如何助力文本驱动的动作合成与动作描述任务，无论是在数据集内部还是其他数据上均是如此。此外，我们还证实GigaHands能够实现动态辐射场重建，为下游任务开辟了新可能。
局限性与未来方向。尽管GigaHands规模庞大且种类多样，但仍存在局限性。由于是在演播室环境中采集数据，数据采集范围受限，因此难以精准捕捉需要更大空间才能完成的动作。虽然我们能够追踪刚性物体及其部件，但要全自动追踪关节型和非刚性物体仍颇具挑战。此外，尽管我们已经展示了该数据集在动作合成和描述方面的应用，但后续研究还可以探索如何利用其规模和多样性来提升机器人操控和人机交互任务的表现。
---
## 致谢
本研究得到了AFOSR基金FA9550-21-1-0214、NSF CAREER基金#2143576以及ONR DURIP基金N00014-23-1-2804的支持。我们感谢OpenAI研究访问计划提供的API支持，并向Ellie Pavlick、Tianran Zhang、Carmen Yu、Angela Xing、Chandradeep Pokhariya、Sudarshan Harithas、Hongyu Li、Chaerin Min、Xindi Qu、Xiaoquan Liu、Hao Sun、Melvin He和Brandon Woodard表达诚挚谢意。
## 参考文献
[^1]: Easymocap - make human motion capture easier. Github, 2021. 6
[^2]: AR Code, Augmented Reality QR Codes — ar-code.com. , 2024. [Accessed 05-11-2024]. 4
[^3]: Meshy Docs — docs.meshy.ai. , 2024. [Accessed 13-11-2024]. 4
[^4]: Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. 5
[^5]: Heni Ben Amor, Oliver Kroemer, Ulrich Hillenbrand, Gerhard Neumann, and Jan Peters. Generalization of human grasping for multi-fingered robot hands. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 2043–2050. IEEE, 2012. 2
[^6]: Mykhaylo Andriluka, Leonid Pishchulin, Peter Gehler, and Bernt Schiele. 2d human pose estimation: New benchmark and state of the art analysis. In Proceedings of the IEEE Conference on computer Vision and Pattern Recognition, pages 3686–3693, 2014. 3
[^7]: Yunfei Bai and C Karen Liu. Dexterous manipulation using both palm and fingers. In 2014 IEEE International Conference on Robotics and Automation (ICRA), pages 1560–1565. IEEE, 2014. 2
[^8]: Prithviraj Banerjee, Sindi Shkodrani, Pierre Moulon, Shreyas Hampali, Fan Zhang, Jade Fountain, Edward Miller, Selen Basol, Richard Newcombe, Robert Wang, et al. Introducing hot3d: An egocentric dataset for 3d hand and object tracking. arXiv preprint arXiv:2406.09598, 2024. 2, 3
[^9]: Samarth Brahmbhatt, Cush Ham, Charles C Kemp, and James Hays. Contactdb: Analyzing and predicting grasp contact via thermal imaging. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8709–8719, 2019. 3
[^10]: Samarth Brahmbhatt, Ankur Handa, James Hays, and Dieter Fox. Contactgrasp: Functional multi-finger grasp synthesis from contact. In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 2386–2393. IEEE, 2019. 2
[^11]: Samarth Brahmbhatt, Chengcheng Tang, Christopher D Twigg, Charles C Kemp, and James Hays. Contactpose:
包含物体接触和手部姿态的抓取数据集。载于《计算机视觉–ECCV 2020：第16届欧洲会议论文集，英国格拉斯哥，2020年8月23–28日》，第13部分，第361–378页。Springer出版社，2020年。3
[^12]: Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06817, 2022. 2
[^13]: Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.15818, 2023. 2
[^14]: Gereon H Büischer, Risto Kõiva, Carsten Schürmann, Robert Haschke, and Helge J Ritter. Flexible and stretchable fabric-based tactile sensor. Robotics and Autonomous Systems, 63:244–252, 2015. 3
[^15]: Zhe Cao, Ilija Radosavovic, Angjoo Kanazawa, and Jitendra Malik. Reconstructing hand-object interactions in the wild. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12417–12426, 2021. 3
[^16]: Géry Casiez, Nicolas Roussel, and Daniel Vogel. 1€ filter: a simple speed-based low-pass filter for noisy input in interactive systems. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pages 2527–2530, 2012. 6
[^17]: Junuk Cha, Jihyeon Kim, Jae Shin Yoon, and Seungryul Baek. Text2hoi: Text-guided 3d motion generation for hand-object interaction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1577–1585, 2024. 3, 6
[^18]: Yu-Wei Chao, Wei Yang, Yu Xiang, Pavlo Molchanov, Ankur Handa, Jonathan Tremblay, Yashraj S Narang, Karl Van Wyk, Umar Iqbal, Stan Birchfield, et al. Dexycb: A benchmark for capturing hand grasping of objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9044–9053, 2021. 3
[^19]: Sammy Christen, Muhammed Kocabas, Emre Aksan, Jemin Hwangbo, Jie Song, and Otmar Hilliges. D-grasp: Physically plausible dynamic grasp synthesis for hand-object interactions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20577–20586, 2022. 2
[^20]: Sammy Christen, Shreyas Hampali, Fadime Sener, Edoardo Remelli, Tomas Hodan, Eric Sauser, Shugao Ma, and Bugra Tekin. Diff $^{2}$o: Diffusion-based synthesis of hand-object interactions from textual descriptions. arXiv preprint arXiv:2403.17827, 2024. 2
[^21]: Enric Corona, Albert Pumarola, Guillem Alenya, Francesc Moreno-Noguer, and Grégory Rogez. Ganhand: Predicting human grasp affordances in multi-object scenes. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5031–5041, 2020. 3
## [^22]: Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price
---
等人。扩展以自我为中心的视觉：EPIC-Kitchens数据集。载于《欧洲计算机视觉会议（ECCV）论文集》，第720–736页，2018年。2, 3
[^23]: Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Evangelos Kazakos, Jian Ma, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. Rescaling egocentric vision: Collection, pipeline and challenges for epic-kitchens-100. International Journal of Computer Vision, pages 1–23, 2022. 2, 3
[^24]: Ahmad Darkhalil, Dandan Shan, Bin Zhu, Jian Ma, Amlan Kar, Richard Higgins, Sanja Fidler, David Fouhey, and Dima Damen. Epic-kitchens visor benchmark: Video segmentations and object relations. Advances in Neural Information Processing Systems, 35:13745–13758, 2022. 3
[^25]: Matt Deitke, Ruoshi Liu, Matthew Wallingford, Huong Ngo, Oscar Michel, Aditya Kusupati, Alan Fan, Christian Laforte, Vikram Voleti, Samir Yitzhak Gadre, et al. Objaverse-xl: A universe of 10m+ 3d objects. Advances in Neural Information Processing Systems, 36, 2024. 2
[^26]: Christian RG Dreher, Mirko Wächter, and Tamim Asfour. Learning object-action relations from bimanual human demonstration using graph networks. IEEE Robotics and Automation Letters, 5(1):187–194, 2019. 3
[^27]: Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. 2
[^28]: Zicong Fan, Omid Taheri, Dimitrios Tzionas, Muhammed Kocabas, Manuel Kaufmann, Michael J Black, and Otmar Hilliges. Arctic: A dataset for dexterous bimanual hand-object manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12943–12954, 2023. 2, 3
[^29]: Guillermo Garcia-Hernando, Shanxin Yuan, Seungryul Baek, and Tae-Kyun Kim. First-person hand action benchmark with rgb-d videos and 3d hand pose annotations. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 409–419, 2018. 2, 3
[^30]: Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, et al. The something something video database for learning and evaluating visual common sense. In Proceedings of the IEEE international conference on computer vision, pages 5842–5850, 2017. 2, 3
[^31]: Patrick Grady, Chengcheng Tang, Samarth Brahmbhatt, Christopher D Twigg, Chengde Wan, James Hays, and Charles C Kemp. Pressurevision: estimating hand pressure from a single rgb image. In European Conference on Computer Vision, pages 328–345. Springer, 2022. 3
[^32]: Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18995–19012, 2022. 2, 3, 4, 5
[^33]: Kristen Grauman, Andrew Westbury, Lorenzo Torresani, Kris Kitani, Jitendra Malik, Triantafyllos Afouras, Kumar Ashutosh, Vijay Baiyya, Siddhant Bansal, Bikram Boote, et al. Ego-exo4d: Understanding skilled human activity from first-and third-person perspectives. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19383–19400, 2024. 2, 3, 4, 5
[^34]: Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang, Wei Ji, Xingyu Li, and Li Cheng. Generating diverse and natural 3d human motions from text. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5152–5161, 2022. 6
[^35]: Chuan Guo, Xinxin Zuo, Sen Wang, and Li Cheng. Tm2t: Stochastic and tokenized modeling for the reciprocal generation of 3d human motions and texts. In ECCV, 2022. 6, 7
[^36]: Shreyas Hampali, Mahdi Rad, Markus Oberweger, and Vincent Lepetit. Honnotate: A method for 3d annotation of hand and object poses. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3196–3206, 2020. 3
[^37]: Shreyas Hampali, Sayan Deb Sarkar, Mahdi Rad, and Vincent Lepetit. Keypoint transformer: Solving joint identification in challenging hands and object interactions for accurate 3d pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11090–11100, 2022. 3
[^38]: Shangchen Han, Po-chen Wu, Yubo Zhang, Beibei Liu, Linguang Zhang, Zheng Wang, Weiguang Si, Peizhao Zhang, Yujun Cai, Tomas Hodan, et al. Umetrack: Unified multi-view end-to-end hand tracking for vr. In SIGGRAPH Asia 2022 conference papers, pages 1–9, 2022. 2
[^39]: Yana Hasson, Gul Varol, Dimitrios Tzionas, Igor Kalevatykh, Michael J Black, Ivan Laptev, and Cordelia Schmid. Learning joint reconstruction of hands and manipulated objects. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11807–11816, 2019. 2,3
[^40]: Binbin Huang, Zehao Yu, Anpei Chen, Andreas Geiger, and Shenghua Gao. 2d gaussian splatting for geometrically accurate radiance fields. In ACM SIGGRAPH 2024 Conference Papers, pages 1–11, 2024. 8
[^41]: Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, 2021. If you use this software, please cite it as below.
[^42]: Juntao Jian, Xiuping Liu, Manyi Li, Ruizhen Hu, and Jian Liu. Affordpose: A large-scale dataset of hand-object interactions with affordance-driven hand pose. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14713–14724, 2023. 3
## [^43]: Sheng Jin, Lumin Xu, Jin Xu, Can Wang, Wentao Liu, Chen Qian, Wanli Ouyang, and Ping Luo. Whole-body human pose estimation in the wild. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, Au
---
2020年8月23日至28日，会议论文集，第IX部分16卷，第196–214页。Springer出版社，2020年。3
[^44]: Yitong Jin, Zhiping Qiu, Yi Shi, Shuangpeng Sun, Chongwu Wang, Donghao Pan, Jiachen Zhao, Zhenghao Liang, Yuan Wang, Xiaobing Li, et al. Audio matters too! enhancing markerless motion capture with audio signals for string performance capture. ACM Transactions on Graphics (TOG), 43(4):1–10, 2024. 3
[^45]: Hanbyul Joo, Tomas Simon, and Yaser Sheikh. Total capture: A 3d deformation model for tracking faces, hands, and bodies. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8320–8329, 2018. 3
[^46]: Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Open-vla: An open-source vision-language-action model. arXiv preprint arXiv:2406.09246, 2024. 2
[^47]: Franziska Krebs, Andre Meixner, Isabel Patzer, and Tamim Asfour. The kit bimanual manipulation dataset. In 2020 IEEE-RAS 20th International Conference on Humanoid Robots (Humanoids), pages 499–506. IEEE, 2021. 3
[^48]: Taein Kwon, Bugra Tekin, Jan Stühmer, Federica Bogo, and Marc Pollefeys. H2o: Two hands manipulating objects for first person interaction recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10138–10148, 2021. 3
[^49]: Gilwoo Lee, Zhiwei Deng, Shugao Ma, Takaaki Shiratori, Siddhartha S Srinivasa, and Yaser Sheikh. Talking with hands 16.2 m: A large-scale dataset of synchronized body-finger motion and audio for conversational motion analysis and synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 763–772, 2019. 3
[^50]: Alexander Lex, Nils Gehlenborg, Hendrik Strobelt, Romain Vuillemot, and Hanspeter Pfister. Upset: visualization of intersecting sets. IEEE transactions on visualization and computer graphics, 20(12):1983–1992, 2014. 4
[^51]: Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. A diversity-promoting objective function for neural conversation models. arXiv preprint arXiv:1510.03055, 2015. 7
[^52]: Yin Li, Miao Liu, and James M Rehg. In the eye of the beholder: Gaze and actions in first person video. IEEE transactions on pattern analysis and machine intelligence, 45(6):6731–6747, 2021. 2, 3
[^53]: Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74–81, 2004. 7
[^54]: Jing Lin, Ailing Zeng, Shunlin Lu, Yuanhao Cai, Ruimao Zhang, Haoqian Wang, and Lei Zhang. Motion-x: A large-scale 3d expressive whole-body human motion dataset. Advances in Neural Information Processing Systems, 36, 2024. 3
[^55]: Haiyang Liu, Zihao Zhu, Naoya Iwamoto, Yichen Peng, Zhengqing Li, You Zhou, Elif Bozkurt, and Bo Zheng. Beat: A large-scale semantic and emotional multi-modal dataset for conversational gestures synthesis. In European
计算机视觉会议论文集，第612–630页。Springer出版社，2022年。2, 3
[^56]: Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499, 2023. 6
[^57]: Yunze Liu, Yun Liu, Che Jiang, Kangbo Lyu, Weikang Wan, Hao Shen, Boqiang Liang, Zhoujie Fu, He Wang, and Li Yi. Hoi4d: A 4d egocentric dataset for category-level human-object interaction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21013–21022, 2022. 2, 3
[^58]: Yun Liu, Haolin Yang, Xu Si, Ling Liu, Zipeng Li, Yuxiang Zhang, Yebin Liu, and Li Yi. Taco: Benchmarking generalizable bimanual tool-action-object understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21740–21751, 2024. 2, 3, 5, 6, 7, 8
[^59]: Xintao Lv, Liang Xu, Yichao Yan, Xin Jin, Congsheng Xu, Shuwen Wu, Yifan Liu, Lincheng Li, Mengxiao Bi, Wenjun Zeng, et al. Himo: A new benchmark for full-body human interacting with multiple objects. In European Conference on Computer Vision, pages 300–318. Springer, 2025. 2, 3
[^60]: Priyanka Mandikal and Kristen Grauman. Learning dexterous grasping with object-centric visual affordances. In 2021 IEEE international conference on robotics and automation (ICRA), pages 6169–6176. IEEE, 2021. 2
[^61]: R McKee, D McKee, D Alexander, and E Paillat. Nz sign language exercises. deaf studies department of victoria university of wellington, 2024. 3
[^62]: Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. How to 100m: Learning a text-video embedding by watching hundred million narrated video clips. In Proceedings of the IEEE/CVF international conference on computer vision, pages 2630–2640, 2019. 2, 3
[^63]: Andrew T Miller and Peter K Allen. Graspit! a versatile simulator for robotic grasping. IEEE Robotics & Automation Magazine, 11(4):110–122, 2004. 2
[^64]: Chaerin Min and Srinath Sridhar. Genheld: Generating and editing handheld objects. arXiv preprint arXiv:2406.05059, 2024. 3
[^65]: Gyeongsik Moon, Shoou-I Yu, He Wen, Takaaki Shiratori, and Kyoung Mu Lee. Interhand2.6m: A dataset and baseline for 3d interacting hand pose estimation from a single rgb image. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XX 16, pages 548–564. Springer, 2020. 3
[^66]: Franziska Mueller, Dushyant Mehta, Oleksandr Sotnychenko, Srinath Sridhar, Dan Casas, and Christian Theobalt. Real-time hand tracking under occlusion from an egocentric rgb-d sensor. In Proceedings of the IEEE international conference on computer vision, pages 1154–1163, 2017. 2, 3
## [^67]: Franziska Mueller, Florian Bernard, Oleksandr Sotnychenko, Dushyant Mehta, Srinath Sridhar, Dan Casas, and
---
Christian Theobalt。基于单目RGB图像实现实时3D手部追踪的生成式手部模型。载于IEEE计算机视觉与模式识别会议论文集，第49–59页，2018年。3
[^68]: Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM transactions on graphics (TOG), 41(4):1–15, 2022. 6
[^69]: Tushar Nagarajan, Christoph Feichtenhofer, and Kristen Grauman. Grounded human-object interaction hotspots from video. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8688–8697, 2019. 3
[^70]: Supreeth Narasimhaswamy, Zhengwei Wei, Yang Wang, Justin Zhang, and Minh Hoai. Contextual attention for hand detection in the wild. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9567–9576, 2019. 3
[^71]: Evonne Ng, Javier Romero, Timur Bagautdinov, Shaojie Bai, Trevor Darrell, Angjoo Kanazawa, and Alexander Richard. From audio to photoreal embodiment: Synthesizing humans in conversations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1001–1010, 2024. 3
[^72]: Takehiko Ohkawa, Kun He, Fadime Sener, Tomas Hodan, Luan Tran, and Cem Keskin. Assemblyhands: Towards egocentric activity understanding via 3d hand pose estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12999–13008, 2023. 3
[^73]: Abby O’Neill, Abdul Rehman, Abhinav Gupta, Abhiram Maddukuri, Abhishek Gupta, Abhishek Padalkar, Abraham Lee, Acorn Pooley, Agrim Gupta, Ajay Mandlekar, et al. Open x-embodiment: Robotic learning datasets and rt-x models. arXiv preprint arXiv:2310.08864, 2023. 2
[^74]: Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023. 6
[^75]: Evin Pınar Örnek, Yann Labbé, Bugra Tekin, Lingni Ma, Cem Keskin, Christian Forster, and Tomas Hodan. Found-pose: Unseen object pose estimation with foundation fea-tures. arXiv preprint arXiv:2311.18809, 2023. 6
[^76]: Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318, 2002. 7
[^77]: Georgios Pavlakos, Dandan Shan, Ilija Radosavovic, Angjoo Kanazawa, David Fouhey, and Jitendra Malik. Reconstructing hands in 3d with transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9826–9836, 2024. 3, 6
[^78]: Tu-Hoa Pham, Nikolaos Kyriazis, Antonis A Argyros, and Abderrahmane Kheddar. Hand-object contact force estimation from markerless visual tracking. IEEE transactions
模式分析与机器智能期刊，第40卷第12期：2883–2896页，2017年。3
[^79]: Hamed Pirsiavash and Deva Ramanan. Detecting activities of daily living in first-person camera views. In 2012 IEEE conference on computer vision and pattern recognition, pages 2847–2854. IEEE, 2012. 2, 3
[^80]: Chandradeep Pokhariya, Ishaan Nikhil Shah, Angela Xing, Zekun Li, Kefan Chen, Avinash Sharma, and Srinath Sridhar. Manus: Markerless grasp capture using articulated 3d gaussians. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2197–2208, 2024. 3
[^81]: Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer. D-nerf: Neural radiance fields for dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10318–10327, 2021. 2
[^82]: Chen Qian, Xiao Sun, Yichen Wei, Xiaoou Tang, and Jian Sun. Realtime and robust hand tracking from depth. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1106–1113, 2014. 3
[^83]: Nikhila Ravi, Jeremy Reizenstein, David Novotny, Taylor Gordon, Wan-Yen Lo, Justin Johnson, and Georgia Gkioxari. Accelerating 3d deep learning with pytorch3d. arXiv:2007.08501, 2020. 6
[^84]: Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00714, 2024. 6
[^85]: Dillon Reis, Jordan Kuepc, Jacqueline Hong, and Ahmad Daoudi. Real-time flying object detection with yolov8. arXiv preprint arXiv:2305.09972, 2023. 6
[^86]: Javier Romero, Dimitrios Tzionas, and Michael J Black. Embodied hands: Modeling and capturing hands and bodies together. arXiv preprint arXiv:2201.02610, 2022. 2, 3, 5, 6
[^87]: Johannes Lutz Schönberger and Jan-Michael Frahm. Structure-from-motion revisited. In Conference on Computer Vision and Pattern Recognition (CVPR), 2016. 5
[^88]: Johannes Lutz Schönberger, Enliang Zheng, Marc Pollefeys, and Jan-Michael Frahm. Pixelwise view selection for unstructured multi-view stereo. In European Conference on Computer Vision (ECCV), 2016. 5
[^89]: Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35:25278–25294, 2022. 2
## [^90]: Fadime Sener, Dibyadip Chatterjee, Daniel Shelepov, Kun He, Dipika Singhania, Robert Wang, and Angela Yao. Assembly101: A large-scale multi-view video dataset for understanding procedural activities. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21096–21106, 2022. 3
---
[^91]: Dandan Shan, Jiaqi Geng, Michelle Shu, and David F Fouhey. Understanding human hands in contact at internet scale. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9869–9878, 2020. 2, 3
[^92]: Toby Sharp, Cem Keskin, Duncan Robertson, Jonathan Taylor, Jamie Shotton, David Kim, Christoph Rhemann, Ido Leichter, Alon Vinnikov, Yichen Wei, et al. Accurate, robust, and flexible real-time hand tracking. In Proceedings of the 33rd annual ACM conference on human factors in computing systems, pages 3633–3642, 2015. 3
[^93]: Qijin She, Ruizhen Hu, Juzhan Xu, Min Liu, Kai Xu, and Hui Huang. Learning high-dof reaching-and-grasping via dynamic representation of gripper-object interaction. arXiv preprint arXiv:2204.13998, 2022. 2
[^94]: Tianxiao Shen, Myle Ott, Michael Auli, and Marc’Aurelio Ranzato. Mixture models for diverse machine translation: Tricks of the trade. In International conference on machine learning, pages 5719–5728. PMLR, 2019. 7
[^96]: Tomas Simon, Hanbyul Joo, Iain Matthews, and Yaser Sheikh. Hand keypoint detection in single images using multiview bootstrapping. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 1145–1153, 2017. 3
[^95]: Gunnar A Sigurdsson, Abhinav Gupta, Cordelia Schmid, Ali Farhadi, and Karteek Alahari. Charades-ego: A large-scale dataset of paired third and first person videos. arXiv preprint arXiv:1804.09626, 2018. 2
[^97]: Srinath Sridhar, Antti Oulasvirta, and Christian Theobalt. Interactive markerless articulated hand motion tracking using rgb and depth data. In Proceedings of the IEEE international conference on computer vision, pages 2456–2463, 2013. 3, 6
[^98]: Srinath Sridhar, Franziska Mueller, Michael Zollhöfer, Dan Casas, Antti Oulasvirta, and Christian Theobalt. Real-time joint tracking of a hand manipulating an object from rgb-d input. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14, pages 294–310. Springer, 2016. 3
[^99]: Xiao Sun, Yichen Wei, Shuang Liang, Xiaoou Tang, and Jian Sun. Cascaded hand pose regression. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 824–832, 2015. 3
[^100]: Subramanian Sundaram, Petr Kellnhofer, Yunzhu Li, Jun-Yan Zhu, Antonio Torralba, and Wojciech Matusik. Learning the signatures of the human grasp using a scalable tactile glove. Nature, 569(7758):698–702, 2019. 3
[^101]: Omid Taheri, Nima Ghorbani, Michael J Black, and Dimitrios Tzionas. Grab: A dataset of whole-body human grasping of objects. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part IV 16, pages 581–600. Springer, 2020. 2, 3, 4
[^102]: Danhang Tang, Hyung Jin Chang, Alykhan Tejani, and Tae-Kyun Kim. Latent regression forest: Structured estimation of 3d articulated hand posture. In Proceedings of the
IEEE计算机视觉与模式识别会议论文集，第3786–3793页，2014年。3
[^103]: Guy Tevet, Sigal Raab, Brian Gordon, Yoni Shafir, Daniel Cohen-or, and Amit Haim Bermano. Human motion diffusion model. In The Eleventh International Conference on Learning Representations, 2023. 6
[^104]: Jonathan Tompson, Murphy Stein, Yann Lecun, and Ken Perlin. Real-time continuous pose recovery of human hands using convolutional networks. ACM Transactions on Graphics (ToG), 33(5):1–10, 2014. 3
[^105]: Dylan Turpin, Liquan Wang, Eric Heiden, Yun-Chun Chen, Miles Macklin, Stavros Tsogkas, Sven Dickinson, and Animesh Garg. Grasp'd: Differentiable contact-rich grasp synthesis for multi-fingered hands. In European Conference on Computer Vision, pages 201–221. Springer, 2022. 2
[^106]: Dimitrios Tzionas, Luca Ballan, Abhilash Srikantha, Pablo Aponte, Marc Pollefeys, and Juergen Gall. Capturing hands in action using discriminative salient points and physics simulation. International Journal of Computer Vision, 118:172–193, 2016. 3
[^107]: Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9 (11), 2008. 4
[^108]: Frank R Wilson. The hand: How its use shapes the brain, language, and human culture. Vintage, 1999. 2
[^109]: Wei Xie, Zhipeng Yu, Zimeng Zhao, Binghui Zuo, and Yanggang Wang. Hmdo: Markerless multi-view hand manipulation capture with deformable objects. Graphical Models, 127:101178, 2023. 3
[^110]: Yufei Xu, Jing Zhang, Qiming Zhang, and Dacheng Tao. Vitpose: Simple vision transformer baselines for human pose estimation. Advances in Neural Information Processing Systems, 35:38571–38584, 2022. 6
[^111]: Lixin Yang, Kailin Li, Xinyu Zhan, Fei Wu, Anran Xu, Liu Liu, and Cewu Lu. Oakink: A large-scale knowledge repository for understanding hand-object interaction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pages 20953–20962. 2022. 2.3
[^112]: Ruolin Ye, Wenqiang Xu, Zhendong Xue, Tutian Tang, Yanfeng Wang, and Cewu Lu. H2o: A benchmark for visual human-human object handover analysis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15762–15771, 2021. 3
[^113]: Youngwoo Yoon, Pieter Wolfert, Taras Kucherenko, Carla Viegas, Teodor Nikolov, Mihail Tsakov, and Gustav Eje Henter. The genea challenge 2022: A large evaluation of data-driven co-speech gesture generation. In Proceedings of the 2022 International Conference on Multimodal Interaction, pages 736–747, 2022. 3
[^114]: Zhengdi Yu, Shaoli Huang, Yongkang Cheng, and Tolga Birdal. Signavatars: A large-scale 3d sign language holistic motion dataset and benchmark, 2024. 3
## [^115]: Shanxin Yuan, Qi Ye, Bjorn Stenger, Siddhant Jain, and Tae-Kyun Kim. Bighand2.2m benchmark: Hand pose dataset and state of the art analysis. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4866–4874, 2017. 3
---
[^116]: Xinyu Zhan, Lixin Yang, Yifei Zhao, Kangrui Mao, Hanlin Xu, Zenan Lin, Kailin Li, and Cewu Lu. Oakink2: A dataset of bimanual hands-object manipulation in complex task completion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 445–456, 2024. 2, 3, 4, 5, 6, 7, 8
[^117]: Hui Zhang, Sammy Christen, Zicong Fan, Otmar Hilliges, and Jie Song. Graspxl: Generating grasping motions for diverse objects at scale. arXiv preprint arXiv:2403.19649, 2024. 2
[^118]: Hui Zhang, Sammy Christen, Zicong Fan, Luocheng Zheng, Jemin Hwangbo, Jie Song, and Otmar Hilliges. Artigrasp: Physically plausible synthesis of bi-manual dexterous grasping and articulation. In 2024 International Conference on 3D Vision (3DV), pages 235–246. IEEE, 2024. 2
[^119]: Jianrong Zhang, Yangsong Zhang, Xiaodong Cun, Yong Zhang, Hongwei Zhao, Hongtao Lu, Xi Shen, and Ying Shan. Generating human motion from textual descriptions with discrete representations. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14730–14740, 2023. 6
[^120]: Siwei Zhang, Qianli Ma, Yan Zhang, Zhiyin Qian, Taein Kwon, Marc Pollefeys, Federica Bogo, and Siyu Tang. Egobody: Human body shape and motion of interacting people from head-mounted devices. In European conference on computer vision, pages 180–200. Springer, 2022. 3
[^121]: Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675, 2019.7
[^122]: Wenqian Zhang, Molin Huang, Yuxuan Zhou, Juze Zhang, Jingyi Yu, Jingya Wang, and Lan Xu. Both2hands: Inferring 3d hands from both text prompts and body dynamics. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2393–2404, 2024. 3
[^123]: Zhongqun Zhang, Hengfei Wang, Ziwei Yu, Yihua Cheng, Angela Yao, and Hyung Jin Chang. Nl2contact: Natural language guided 3d hand-object contact modeling with diffusion model. arXiv preprint arXiv:2407.12727, 2024. 3
[^124]: Chengfeng Zhao, Juze Zhang, Jiashen Du, Ziwei Shan, Junye Wang, Jingyi Yu, Jingya Wang, and Lan Xu. I'm hoi: Inertia-aware monocular capture of 3d human-object interactions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 729–741, 2024. 3
[^125]: Zehao Zhu, Jiashun Wang, Yuzhe Qin, Deqing Sun, Varun Jampani, and Xiaolong Wang. Contactart: Learning 3d interaction priors for category-level articulated object and hand poses estimation. In 2024 International Conference on 3D Vision (3DV), pages 201–212. IEEE, 2024. 3
[^126]: Christian Zimmermann and Thomas Brox. Learning to estimate 3d hand pose from single rgb images. In Proceedings of the IEEE international conference on computer vision, pages 4903–4911, 2017. 2, 3
[^127]: Christian Zimmermann, Duygu Ceylan, Jimei Yang, Bryan Russell, Max Argus, and Thomas Brox. Freihand: A dataset
用于从单张图像中无标记地捕捉手部姿态与形状。
基于RGB图像。载于IEEE/CVF国际计算机视觉会议论文集，
第813–822页，2019年。3