# 第一人称视角下的感知与行动：以人为中心的人-物-人交互数据集与基准测试
徐亮$^{1,2,3}$ 杨承群$^{1}$ 林子力$^{1,2,3}$ 许飞$^{1}$ 刘一凡$^{1}$ 徐从胜$^{1}$ 张艺艺$^{4}$ 秦杰$^{5}$ 盛星东$^{6}$ 刘云辉$^{6}$ 金鑫$^{2,3}$ 严一超$^{1*}$ 曾文军$^{2,3}$ 杨晓康$^{1}$
$^{1}$上海交通大学人工智能研究院人工智能教育部关键实验室
$^{2}$中国宁波东方理工大学数字孪生研究所
$^{3}$中国宁波空间智能与数字衍生技术宁波市重点实验室
$^{4}$上海交通大学计算机学院人工智能教育部关键实验室
$^{5}$南京航空航天大学 $^{6}$联想集团
https://liangxuy.github.io/InterVLA/
Figure 1. InterVLA features a large-scale human-object-human interaction dataset in a vision-language-action scheme, where an assistant provides services to an instructor based on egocentric perception and verbal commands. This comprehensive dataset comprises 3.9K sequences, totaling 11.4 hours and 1.2M frames of multimodal interaction data, including egocentric and exocentric RGB videos, language commands and high-precision human/object motions, promoting the development of general-purpose intelligent AI assistants.
## 摘要
从真实世界中以人为中心的交互数据集中学习动作模型，对于构建高效、通用的智能助手而言至关重要。然而，现有大多数数据集仅提供特定类别的交互数据，却忽略了人工智能助手是基于第一人称视角来获取信息并开展行动的这一事实。我们认为，通用交互知识与以人为中心的模态二者缺一不可。在本文中，我们将人工辅助任务嵌入到视觉-语言-动作框架中，该框架下助手会依据第一人称视觉信息与指令为指导者提供服务。借助我们的混合RGB-MoCap系统，助手与指导者配对，按照GPT生成的脚本与多种物体及场景进行互动。
在这种设定下，我们构建了InterVLA——首个大规模人-物-人交互数据集，该数据集包含11.4小时、120万帧的多模态数据，涵盖2段第一人称视角视频和5段第三人称视角视频，同时记录了精确的人体/物体运动轨迹以及语音指令。此外，我们针对以人为中心的人体运动估计、交互合成和交互预测设立了全新的基准测试，并给出了全面的分析结果。我们相信，我们的InterVLA测试平台及相关基准测试将推动未来物理世界中人工智能智能体的研究工作。
### 1. 引言
为了开发出能够在物理世界中协助人类的通用智能体，掌握通用的交互知识是必不可少的。为避免收集成本高昂的机器人数据，从以人为中心的交互数据集中学习则更为高效。
---
现有的人与人之间的交互数据集有助于促进人机交互 [^8] [^77] [^80] [^97]；遥操作技术 [^14] [^56] [^81] [^93]；人与物体的交互则能促进人机物品交接 [^44] [^45] [^100]，人机协作 [^17] [^18] [^115] [^116]，而人与场景的交互则能推动导航技术的发展 [^20] [^98] [^134] [^118] [^136] [^139]。
尽管以人为中心的数据集和基准测试在规模和丰富度上发展迅速，但在构建智能助手方面仍存在一些局限。设想一下家用机器人最基本的能力：感知并理解指令、顺畅导航以及操作物体。然而，大多数数据集仅提供特定类别的交互数据[^37] [^66] [^76] [^86] [^116] [^122] [^130] [^134]，而非由多样化的人与人、物体及场景交互构成的通用场景。此外，现有数据集[^37] [^66] [^76] [^116] [^122] [^130] [^137]忽略了这样一个事实：AI助手总是先通过第一人称视角获取信息，再据此做出反应[^112] [^134]。缺乏以自我为中心的视角可能会阻碍AI助手的物理部署。
为克服这些局限并促进通用型以人为中心的交互及多功能AI助手的发展，一个涵盖多样化交互模式且具备稳定以自我为中心感知能力的数据集至关重要。在本文中，我们聚焦于日常手动辅助任务中的常见场景，其中大部分属于人与物之间的交互，即助手根据以自我为中心的视觉和语音指令为指令者提供服务，例如“把桌上那个杯子递给我”。这类场景中，人与人、物体及场景的交互自然融合在一起。为了模拟真实的机器人辅助场景，我们随机布置了各种家具和可操作的物体来构建场景。指令者会发出带有肢体动作的语音指令，而助手则理解其意图并作出相应响应。
受旨在实现指令跟随机器人的视觉-语言-动作（VLA）范式的启发，我们在VLA框架内设计了数据采集方案，并推出了InterVLA——首个大规模以自我为中心的人与物交互数据集，包含多种交互类别，如图1所示。在视觉模态方面，我们从指令者的视角拍摄两段以自我为中心的视频，同时拍摄五段涵盖整个场景的第三人称视角视频。语言部分则由100个精心编写的脚本组成，涵盖了各种场景布置、多样的交互类型、多物体交互以及导航任务。为了获取动作数据，我们在人体和物体表面贴上反光标记，从而能在保持RGB数据完整性的同时进行高精度运动追踪。我们招募了47名参与者，组成27对独特的指令者-助手组合……
| Dataset | Modality | Scale |
| HHI | HOI | HSI | Multi-Obj | Ego | Exo | #Seqs | #Objs | #Hours |
| You2Me [^86] | ✓ | ✗ | ✗ | ✗ | ✓ | ✗ | 42 | - | 1.4 |
| ExPI [^37] | ✓ | ✗ | ✗ | ✗ | ✗ | ✓ | 115 | - | 0.3 |
| Hi4D [^130] | ✓ | ✗ | ✗ | ✗ | ✗ | ✓ | 100 | - | - |
| InterHuman [^66] | ✓ | ✗ | ✗ | ✗ | ✗ | ✓ | 6.0K | - | 6.6 |
| Inter-X [^122] | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | 11.4K | - | 18.8 |
| HIMO [^76] | ✗ | ✓ | ✗ | ✓ | ✗ | ✗ | 3.4K | 53 | 9.4 |
| HOH [^116] | ✗ | ✓ | ✗ | ✗ | ✗ | ✓ | 2.7K | 136 | - |
| CORE4D [^134] | ✓ | ✓ | ✗ | ✗ | ✓ | ✓ | 1.0K | 37 | - |
| HOI-M $^{3}$ [^137] | ✓ | ✓ | ✓ | ✓ | ✗ | ✓ | 199 | 90 | 20 |
| InterVLA | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | 3.9K | 50 | 11.2 |
Table 1. Dataset comparison. We compare InterVLA with existing human-centric interactive datasets. Modality measures the human-human, human-object, human-scene interactions, multi-object interactions, egocentric and exocentric views. Scale measures the number of sequences, objects and hours.
他们将与50件家用物品互动，最终生成3.9千个序列，总时长达11.4小时，共产生120万帧交互数据。所有采集到的数据都经过精心校准且时间上同步。表1总结了该数据集与现有以人为中心的交互数据集之间的对比情况。
借助我们提出的InterVLA数据集，我们引入了关于AI助手如何更好地感知周围环境并生成恰当响应的新任务与基准测试。我们定义了四类下游任务：1）以第一人称视角为基础，从助手的视角提取指导者全身运动信息的以自我为中心的人体运动估计；2）交互合成，即根据文本描述以及人体和物体的初始状态，生成合理的人-物-人序列；3）基于运动的交互预测，即根据之前的运动帧来预测未来的人体/物体运动；4）基于视觉语言的交互预测，即根据历史的第一人称视频和语音指令来预测未来的人体运动。我们为这些任务建立了全面的基准测试，提供了基线模型、定量与定性评估结果以及深入分析。研究结果凸显了InterVLA场景中快速相机移动、视野受限、遮挡以及多物体交互所带来的挑战。此外，我们还强调了该数据集在稀疏视图4D场景重建、手-物交互以及从稀疏信号中恢复运动等任务中的潜在应用价值。我们的贡献可总结如下：
- 我们收集了首个大规模的人-物-人交互数据集InterVLA，该数据集涵盖了多样化的通用交互类别，且采用以自我为中心的视角。
- 我们提出的包含对以自我为中心的人体运动估计、交互合成及交互预测的详尽分析的基准测试，将推动未来智能AI助手相关研究的发展。我们将发布全部数据集、代码和模型以供进一步探索。
### 2. 相关工作
以自我为中心的视觉。以自我为中心的视觉正受到越来越多的关注，相关数据集有[6, 22, 23, 32, 34, 35, 41, 62, 64,
---
。
(a) Human Setup
(b) Object-Scene Setup
(c) Hybrid RGB-MoCap System
Figure 2. InterVLA capturing system. (a). We attach 41 reflective markers to the subject's body with strong medical glue to track the human body motion. The assistant wears two GoPro cameras to capture the egocentric data. (b). Reflective markers are positioned on the surface of real objects to track the precise object trajectories. We also collect the precise 3D object scans with a KSCAN Magic Scanner. (c). Our hybrid RGB-MoCap system with two egocentric RGB cameras, five exocentric RGB cameras, and an OptiTrack MoCap system.
[^85]: [^112]，这一领域的发展受到了机器人、AR和VR等应用的推动。以自我为中心的视觉独特的动态特性和视角为各类任务带来了新的挑战与机遇，包括佩戴者姿态估计[^3] [^53] [^62] [^75] [^82] [^110] [^111]、活动识别[^35] [^57] [^65] [^141]、人-物交互（HOI）[^15] [^21] [^61] [^77] [^84]、机器人主动感知[^4] [^58] [^107] [^126]以及交互式助手[^112]。InterVLA通过图2所示的两台GoPro相机采集以自我为中心的数据，同时借助OptiTrack动作捕捉系统获取精确的人体和物体运动信息。与Ego-Exo4D [^35]等大多数视频数据集相比，InterVLA提供了由光学动作捕捉系统采集的真实4D人体和物体运动数据。
人机交互。除了众多单人动作数据集[^36] [^50] [^67] [^69] [^91] [^96] [^113] [^121] [^143]外，也有几个用于交互合成[^33] [^37] [^66] [^69] [^86] [^108] [^122] [^130] [^133]的人机交互数据集[^28] [^66] [^120] [^122]、人类反应生成[^122] [^123]。[^14] [^44] [^56] [^92] [^93]也验证了所学的交互知识可应用于人机交互。InterVLA本质上是一个由指导者与助手共同与物体互动所构成的人机交互数据集。
人与物体的交互。近期的研究拓展了人与物体交互数据集的范畴，从单物体的手部交互[^11] [^39] [^40] [^51] [^72] [^73]和全身交互[^29] [^48] [^105] [^125]，扩展到多物体的全身交互[^76] [^79]。与以往仅关注无上下文的单个人与物体交互片段的数据集不同，InterVLA能够捕捉每个GPT生成的脚本所对应的系列连贯且连续的人与物体交互片段。
人与场景的交互。人与场景的交互涵盖了从导航、避碰到与场景中物体互动等多个层面，在具身智能和虚拟现实等领域有着广泛应用。现实世界中的数据集[^9] [^38] [^42] [^43] [^54] [^83] [^99] [^138]存在诸多局限，比如场景固定[^38] [^43]、物体单一[^9] [^54] [^138]，或是基于图像估计的3D姿态存在噪声[^42] [^83] [^99]。合成数据集[^5] [^10] [^16] [^55]虽解决了这些问题，但在外观真实性和物理合理性方面仍有不足。我们构建了带有随机家具布置的简单场景，可提供现实世界动态场景中的全面导航与交互数据。
人与物体与人之间的交互。HOH交互在协作任务和场景中十分常见。一些专注于物品传递的数据集[^17] [^18] [^27] [^60] [^116]受限于固定的场景和身体位置设置、缺乏以自我为中心的RGB数据，且缺少自然的人类外观或3D人体姿态数据。虽然CORE4D[^134]提供了以自我为中心的HOH交互数据，但它仅聚焦于两人协作重新摆放物品的单一交互类型。InterVLA弥补了这些不足，涵盖了多样化的HOH交互类型、灵活的场景设置以及多物体交互。
视觉-语言-动作模型。VLA模型将视觉和语言作为多模态输入，并生成动作以帮助智能体完成任务。过去几年中，在大型预训练模型的支持下，涌现出了许多优秀成果[^12] [^13] [^25] [^47] [^59] [^78] [^87] [^140]，推动了VLA模型的发展。HumanVLA[^126]率先利用模拟器风格的图像作为视觉输入来模拟类人型VLA。我们期待InterVLA能够借助真实的以自我为中心的RGB数据、语言指令、人类响应动作以及物体轨迹，加速类人型VLA在现实世界应用中的落地。
### 3. InterVLA数据集
#### 3.1. 概述
InterVLA是一个按照视觉-语言-动作框架收集的大规模人机交互数据集，它
---
1. 打开啤酒瓶，将一些啤酒倒入茶壶中。
2. 把啤酒瓶放回桌边。
(a) Egocentric Videos (b) Exocentric Videos
(c) Instructions
(d) Human & Object Motions
Figure 3. Components of the InterVLA dataset. For the vision modality, we capture (a) two egocentric and (b) five exocentric RGB videos; For the language modality, we comprehensively supply the (c) GPT-generated commands; For the action modality, we provide the high-precision (d) human and object motions during the interactions.
包含助手在日常场景中为指导者提供多样化服务的内容。我们认为，这种双人多物体的设定整合了多种以人为中心的互动模式，将有助于推动以机器人为中心的互动研究。此外，我们的InterVLA数据集还着重体现了以自我为中心的感知及其指导下助手动作的实用性。如图1所示，每个场景都由随机摆放的家具组成，两人会与若干日常物品互动。助手会按照指导者的语言指令执行一系列连续动作，比如“把桌上的杯子递给我”。我们提供了多视角的客观（第三人称）视角，以及助手的两种以自我为中心（第一人称）视角。人体和物体的运动数据由光学动作捕捉（MoCap）系统获取。接下来，我们将在第3.2节描述视觉-语言-动作数据的采集方案与流程，在第3.3节介绍数据集的后处理、组成部分及统计数据。
#### 3.2. 视觉-语言-动作数据采集
视觉-语言-动作（VLA）的概念随着指令跟随机器人的兴起而出现。受其启发，我们将人工辅助任务纳入VLA框架，让助手为指导者提供各类服务，如拾取、取回、交接以及重新摆放多个物体。InterVLA的混合RGB-MoCap采集系统在图2中有详细展示。
数据采集流程。我们招募了46名参与者，组成27对独特的指导者-助手组合来进行数据采集。在物体和场景设置方面，我们主要选取了50种常见家用物品，尺寸各异，包括水果、杯子、刀具等小型物品，以及行李箱、衣架、扫帚等大型物品。需要注意的是，为了保证RGB模态下的真实性，我们选用真实物品而非3D打印物品，如[^76] [^105]所示。物品清单的详细信息见补充材料。为了贴合现实世界的智能机器人辅助场景并丰富交互类别，我们精心设计了以下数据采集流程。首先，我们在动作捕捉场地随机摆放一些桌椅类家具，并在场景中放置若干可操作的物品。采集过程中，指导者会先通过口头指令配合身体手势与助手沟通，而助手则需理解指导者的意图并做出相应反应。每次录制都会执行一系列原子级交互，以保留长时间的交互过程并确保连贯性。
视觉系统。我们在助手的额头和胸部分别紧密安装两台GoPro摄像头，用于拍摄分辨率为$5312 \times 2988$、帧率为30fps的第一人称RGB视频。摄像头内参是依据[^1]获取的。根据参与者的身高，我们仔细调整摄像头的位置和方向，以最大化场景、另一人以及交互行为的有效拍摄范围。此外，我们还集成了五台经过校准的RGB摄像头，以提供分辨率为$1920 \times 1080$、帧率为30fps的多视角外摄画面。第一人称和外摄摄像头均通过毫秒级时间戳实现时间同步。为保护参与者隐私，我们用[^2]对所有外摄及第一人称RGB视频中的人脸进行遮挡处理。
语言系统。指导者的指令是后续交互的起点、触发点和桥梁。基于家用物品和家具库，我们利用大型语言模型，即ChatGPT [^88]，来选择场景和物品布局、确定其摆放位置，再根据物品的可用性生成场景中指导者-助手交互序列的脚本。由于大多数交互集中在人-物-人交互，比如物品传递和协同整理，我们也纳入了一些纯人-人交互，比如扶持或按摩。此外，我们鼓励多物品间的交互以及物品间的互动，比如“用刀切苹果”，还有同时操作多个物品的情况，比如“整理桌上的物品”。
---
此外，也支持在场景内移动和导航。最终，我们共生成了100个脚本，每个脚本平均涉及2-3件家具、5个家用物品以及8条连续的原子级指令。每个脚本都经过人工审核以确保其有效性。
动作。对于机械臂而言，动作可以定义为机器人关节的旋转和平移。类似地，运动是一种用于建模人类动作和物体移动的紧凑表示方式。为了获取高质量的人体和物体运动数据，我们搭建了一个边长为$8.5 \times 5.4$米的动作捕捉系统，该系统配备了20台红外摄像头。针对人体运动数据，我们摒弃了紧身的动作捕捉服，而是用强力医用胶直接将反光标记点贴在皮肤或衣物表面，以此保证RGB模态下的数据保真度[^29] [^49]。通过胶水固定可消除皮肤与衣物间的相对位移，从而确保动作捕捉结果的精确性与稳定性。物体则被视为刚体，其表面至少会放置四个反光标记点以实现光学追踪。在我们的实验中，无论对人体还是物体，均采用直径为12.5毫米的反光球以获得最佳追踪效果。所有相关设备均在录制前经过精心制作与校准。此外，该动作捕捉系统还配备了时间码功能，以便与自拍/外摄RGB视频进行时间对齐。
#### 3.3. 数据集组成部分
人体参数化模型。SMPL参数化模型[^74]在人体交互类数据集中被广泛采用，该模型将人体网格表示为身体姿态$\theta \in \mathbb{R}^{23 \times 3}$、全局朝向$q_i \in \mathbb{R}^3$、根部平移$\gamma_i \in \mathbb{R}^3$以及身体形状参数$\beta \in \mathbb{R}^{10}$，这些参数会根据参与者的身高、体重和性别来确定[^76] [^94] [^122]。我们将从动作捕捉数据中获取的BVH格式人体骨骼数据拟合到SMPL参数中，具体的优化目标如下：
$$ \mathcal{L}=\lambda_{j}\mathcal{L}_{j}+\lambda_{s}\mathcal{L}_{s}+\lambda_{r e g}\mathcal{L}_{r e g}, $$
其中，$\mathcal{L}_j$用于衡量原始动作捕捉关节位置与优化结果之间的差异，$\mathcal{L}_s$用于平滑帧间运动过渡并减轻姿态抖动现象，$\mathcal{L}_{reg}$用于防止优化后的姿态发生偏离，而$\lambda_j = 1$、$\lambda_s = 0.1$、$\lambda_{reg} = 0.01$则为各项损失的权重。关于各优化项的更多细节，我们在补充材料中进行了说明。
**物体网格与追踪。**为了实现精准的人体-物体交互渲染，我们对全部50个物体进行了扫描，利用KSCAN Magic扫描仪获取精确的物体表面几何结构。每个物体上会粘贴超过3个反光标记点，以便通过光学动作捕捉系统进行追踪。我们还会将粘贴好标记点的物体再次扫描，并将新扫描结果与先前的结果对齐，从而消除两个质心之间的偏移。物体运动可以用平移$t^o \in \mathbb{R}^3$以及6D旋转表示下的旋转$r^o \in \mathbb{R}^6$来描述[^142]。
**对齐与分割。**我们将所有视频数据的分辨率统一调整为$1920 \times 1080$，帧率为30帧/秒，同时将动作捕捉数据的采样率也下调至相同帧率以保证一致性。外摄视频、自拍视频和运动数据均通过毫秒级时间戳实现精准同步。为了便于下游任务的训练，我们会先渲染交互结果，再按照原子指令手动将长时长的脚本分割成短片段，同时保留片段间的时间连续性。RGB视频也采用同样的方式进行分割。
InterVLA统计数据。如前所述，我们招募了47名参与者，组成27对独特的教师-助手组合来进行数据收集，共包含519段有效的长时间交互录像，这些录像采用了不同的GPT生成的脚本和场景安排。我们确保每个物体至少出现在五种场景中。经过对齐、分割以及对运动数据的细致手动优化后，我们获得了3,906个交互序列，这些数据由以自我为中心的视频、以物体为中心的视频、指令以及人类/物体运动组成，总时长达11.4小时，包含120万帧四元组数据。我们在图3中展示了该数据集的一个示例以便更直观地说明，更多视频示例将在补充材料中提供。
### 4. 任务、基准测试与实验
我们定义了以自我为中心的人体运动估计、交互合成与交互预测的新任务和基准测试，如图4所示；同时基于InterVLA还定义了面向通用人工智能智能体的潜在下游任务。我们还提供了所提出基准测试的评估结果，并附上详细分析。
#### 4.1. 初步公式化
以自我为中心的视频展现了主动视角，在捕捉近距离交互细节及第一人称意图方面具有优势。不过，这类视频也存在视野受限、缺乏全局感知的问题，且物体会快速进出画面，导致视角频繁变化。
我们将每个序列的多模态数据表示为[V, T, H, O, l]，其中$V = \{v_h, v_c\}$分别是头戴式和胸戴式以自我为中心的视频，$T = \{t_i\}_{i=0}^{N_t}$是以物体为中心的视频，$N_t = 5$表示视频编号。人类运动分别用$H = \{h_I, h_A\}$表示教师和助手的运动，其中$h_I / h_A$可细分为分别表示身体姿态、全局朝向、根部平移以及身体形状参数的$\{\theta \in \mathbb{R}^{23 \times 6}, q \in \mathbb{R}^6, \gamma \in \mathbb{R}^3, \beta \in \mathbb{R}^{10}\}$。物体运动可用$O = \{o_i\}_{i=0}^{N_0}$表示，其中$o_i = \{r_i^o \in \mathbb{R}^6, t_i^o \in \mathbb{R}^3\}$分别对应物体的旋转和平移；$N_o$表示物体的数量。所有相关物体的几何形状均经过精确扫描，记为$G = \{g_i\}_{i=0}^{N_0}$。此处
---
(A) Egocentric Human Motion Estimation
(C-1) Motion-based Interaction Prediction
C-2) Vision-Language Based Interaction Prediction
Figure 4. Task Formulation of InterVLA. We establish multiple downstream tasks on egocentric human motion estimation, text-driven interaction synthesis, motion-based interaction prediction and vision-language based interaction prediction. All these benchmarks showcase the great challenges of our InterVLA dataset and will benefit practical, intelligent AI assistants.
我们采用6D旋转表示方法[^142]来描述人类和物体的运动，这一方法在以往的研究中已有应用[^36] [^90]。l代表语言指令。我们将数据集划分为训练集、测试集和验证集，用于所有下游任务的比例分别为0.8、0.15和0.05。需要注意的是，所有基于视频的实验仅采用头戴式摄像头$v_h$采集的数据。
#### 4.2. 以自我为中心的人体运动估计
任务定义。对教师的意图进行以自我为中心的感知与理解，是人工智能助手发挥作用的首要步骤。如图4（A）所示，我们的目标是根据以自我为中心的RGB视频重建基于真实世界的人类运动序列。
$$ F_{M E}(\boldsymbol{v}_{h})\mapsto\hat{h}_{I}, $$
其中$\hat{h}_{I}$表示估算出的教师运动。需要注意的是，要保持准确的人体姿态估计并维持一致的全局坐标系难度极大。以自我为中心的摄像头的快速移动、遮挡以及有限的可见性，给这项任务带来了巨大挑战。
实验设置。我们选取了四种最先进的全局人体运动估计方法——TRACE [^104]、GLAMR [^132]、TRAM [^114]和WHAM [^103]，在InterVLA数据集的所有头戴式摄像头序列上进行评估，以便直观地理解其性能。
评估指标。遵循以往的研究 [^103] [^114]，我们计算平均关节位置误差（MPJPE）、经过Procrustes对齐的MPJPE（PA-MPJPE）以及顶点误差（PVE），用以评估3D人体姿态与体型估计的性能；同时计算加速度误差（Accel）来衡量帧间运动的平滑度。具体而言，MPJPE计算预测关节位置与真实关节位置之间的欧几里得距离平均值；PA-MPJPE是在通过平移、旋转和缩放将预测姿态与真实姿态对齐后计算得到的MPJPE；PVE衡量的是基于SMPL参数模型生成的6,890个顶点的预测位置与真实位置之间的平均距离。加速过程中的A
结果与分析。我们在图5中展示了可视化结果对比，包括原始以自我为中心的画面、由MoCap系统捕捉到的真实运动数据以及基线结果。需要注意的是，我们手动添加了水平偏移以避免运动轨迹相互缠绕，同时将锚点置于地面，以便参考运动的全局平移情况。结果显示，即便性能最优的算法得到的结果也不尽如人意。对于这三个样本而言，助手会轻微（第一行）或快速（第三行）转头寻找场景中的物体，导致指导者移出摄像头的视野范围。红色虚线框显示GLAMR和WHAM无法追踪正确的全局方向并保持正确的坐标系；绿色虚线框则表明TRACE遗漏了许多帧。此外，我们还在表2中提供了定量结果对比，其中WHAM [^103]除PA-MPJPE外，在所有指标上均展现出显著的优势。不过，其性能与真实结果之间仍存在明显差距。这种差异源于数据集本身存在的若干挑战，比如遮挡、相机快速运动、人体轮廓捕捉不完整，以及受试者频繁进出画面（重新进入画面）等情况。
#### 4.3. 交互合成
任务定义。遵循现有的文本驱动HOI合成方法 [^24] [^76] [^89] [^137]，我们将该任务定义为多个人体和物体的运动生成，如图4（B）所示。我们将此任务表述为
$$ F_{I S}(\boldsymbol{H}^{0},\boldsymbol{O}^{0},\boldsymbol{G},l)\mapsto\{\hat{\boldsymbol{H}},\hat{\boldsymbol{O}}\}, $$
其中 $H^{0}$ 和 $O^{0}$ 分别代表人体和物体的初始姿态，而 $\hat{H}$ 和 $\hat{O}$ 则是生成的人体和物体运动。与BEHAVE [^9] 等以往的HOI数据集相比，我们的InterVLA数据集包含
---
Figure 5. Visualization result comparison of the egocentric world-grounded human motion estimation results on the InterVLA dataset. For each sequence, we provide the original RGB frames along with the ground truth motion obtained by the MoCap system and four baseline models. Higher opacity indicates later frames of the sequence. Please zoom in for a more detailed view.
| Method | PA-MPJPE $\downarrow$ | MPJPE $\downarrow$ | PVE $\downarrow$ | Accel $\downarrow$ |
| TRACE [^104] | 91.2 | 720.1 | 761.3 | 31.0 |
| GLAMR [^132] | 134.8 | 589.9 | 596.7 | 24.9 |
| TRAM [^114] | 102.7 | 684.1 | 718.7 | 27.5 |
| WHAM [^103] | 103.2 | 333.6 | 359.7 | 8.7 |
Table 2. Quantitative results of egocentric global human motion estimation on the InterVLA dataset. Bold for the best results.
多个人体和物体以及多物体操控场景，这带来了更大的挑战。
实验设置。我们重新实现了文本驱动的人体运动生成方法MDM [^106]、priorMDM [^101]和HIMO [^76]，以支持物体网格以及两名人员和多个物体的初始状态作为条件输入。关于这些方法的更多细节可在补充材料中找到。
评估指标。我们首先通过对比学习来训练文本特征提取器和人体-物体运动特征提取器，如 [^36] 和 [^76] 所示。生成质量通过以下指标进行评估：R Precision用于衡量检索真实描述时的前3位准确率；Frechet Inception Distance (FID) [^46] 用于度量真实样本与合成样本之间的潜在空间差异；MultiModal距离（MM Dist）用于确定生成的运动与输入文本之间的潜在空间距离；Diversity用于衡量潜在空间内的变异程度；multi-modality（MModality）则用于量化输出的多样性。
| Methods | R Precision (Top 3) † | FID ↓ | MM Dist ↓ | Diversity → | MModality ↑ |
| Real | 0.7592 \pm 0.0026 | 0.0203 \pm 0.0018 | 3.8718 \pm 0.0064 | 9.0164 \pm 0.0831 | — |
| MDM [^106] | 0.4897 \pm 0.0067 | 2.8039 \pm 0.0727 | 5.4879 \pm 0.0212 | 7.7260 \pm 0.0633 | 1.9888 \pm 0.0694 |
| priorMDM [^101] | 0.5250 \pm 0.0068 | 6.2766 \pm 0.0777 | 5.5129 \pm 0.0188 | 8.9414 \pm 0.0888 | 2.1227 \pm 0.0993 |
| HIMO [^76] | 0.5707 \pm 0.0029 | 0.6805 \pm 0.0136 | 4.9609 \pm 0.0162 | 8.9849 \pm 0.0554 | 1.1478 \pm 0.0658 |
Table 3. Quantitative results of human-object-human interaction synthesis on the InterVLA dataset, where $\pm$ indicates 95% confidence interval and $\rightarrow$ means the closer the better. Bold highlights the best results.
由相同的文本输入生成。
结果与分析。表3中呈现的定量结果表明，除MModality外，HIMO [^76] 在所有指标上均优于其他方法，尤其在FID方面表现突出。所有方法的FID值均高于真实交互数据，这表明未来仍有很大空间来提升生成交互结果的自然度。
#### 4.4. 交互预测
任务定义。我们提出了两种基于运动的和基于视觉-语言的交互预测任务，如图4中的(C1)和(C2)所示。对于基于运动的交互预测，模型根据相邻的过去几帧HOI序列来预测后续帧的HOI序列。
$$ F_{M-I P}(\boldsymbol{H}^{t_{I}:t-1},\boldsymbol{O}^{t_{I}:t-1},\boldsymbol{G})\mapsto\{\hat{\boldsymbol{H}}^{t:t_{E}},\hat{\boldsymbol{O}}^{t:t_{E}}\}, $$
---
| Methods | Human | Object | Contact |
| $J_{e}(mm,\downarrow)$ | $T_{e}(mm,\downarrow)$ | $R_{e}(°,\downarrow)$ | $C_{acc}(%, \uparrow)$ | $P_{r}(%, \downarrow)$ |
| MDM [^106] | 175.3 ( $\pm$ 0.8) | 140.2 ( $\pm$0.7) | 11.0 ( $\pm$0.2) | 85.5 ( $\pm$0.3) | 0.4 ( $\pm$0.0) |
| InterDiff [^124] | 175.3 ( $\pm$ 0.8) | 138.7 ( $\pm$0.6) | 10.8 ( $\pm$0.1) | 86.0 ( $\pm$0.2) | 0.4 ( $\pm$0.0) |
| CAHMP [^19] | 172.5 ( $\pm$ 0.4) | 115.6 ( $\pm$0.5) | 9.5 ( $\pm$0.1) | - | - |
Table 4. Quantitative results of motion-based interaction prediction on the InterVLA dataset.
其中 $t_I$（$t_E$）表示序列的起始（结束）帧，$\hat{H}^{t:t_E}$ 和 $\hat{O}^{t:t_E}$ 为预测的未来运动。在实验中，我们设定 $t_I = 0$、$t = 15$ 和 $t_E = 30$，以根据前15帧预测后续15帧中的姿态。而对于基于视觉-语言的交互预测，模型能够根据历史第一人称视频和语言指令来预测未来的人体运动。
$$ F_{{V L-I P}}(\mathbf{v}_{h}^{t_{I}:t-1},\mathbf{h}_{A}^{t_{I}:t-1},\boldsymbol{l})\mapsto\{\hat{\mathbf{h}_{A}}^{t:t_{E}}\}, $$
其中 $\hat{h}_{A}^{t:t_{E}}$ 表示助手的预测运动。实验设置。我们评估了三种最先进的方法：CAHMP [^19]、MDM [^106] 和 InterDiff [^124]，用于基于运动的交互预测。对于视觉-语言引导的交互预测，我们重新实现了现有的手部轨迹预测模型FHOI [^70]、OCT [^71] 和 USST [^7]，并集成语言嵌入以预测助手的运动。
评估指标。对于基于运动的交互预测，我们遵循[^134]的方法，采用人体关节位置误差（$J_{e}$）来衡量两个人的平均关节位置误差（MPJPE）；采用物体平移误差（$T_{e}$）来表示预测物体平移与实际物体平移之间的平均L2差异；采用物体旋转误差（$R_{e}$）来表示预测物体旋转与实际物体旋转之间的平均测地距离差异；采用人机接触准确率（$C_{acc}$）来评估接触检测的平均误差率，其中设定5厘米为检测手部接触的阈值；采用穿透率（$P_{r}$）来计算物体顶点穿透人体网格的百分比。对于视觉-语言引导的交互预测，我们采用平均位移误差作为预测轨迹与真实轨迹之间的平均L2距离，而最终位移误差则按照FHOI [^70]的定义，指最终预测位置与真实位置之间的L2距离。结果与分析。我们在表4中给出了三种最先进的基于运动的交互预测模型的定量结果。从结果可以看出，CAHMP [^19]在人体关节位置误差、物体平移及旋转误差这两项指标上均优于其他基线方法，这得益于其用于学习人与环境物体之间关系的语义图模型。从基于视觉-语言的交互预测的定量对比来看——
| Method | Avg. Disp. Error $\downarrow$ | Final Disp. Error $\downarrow$ |
| FHOI [^70] | 0.29 | 0.38 |
| OCT [^71] | 0.28 | 0.36 |
| USST [^7] | 0.24 | 0.32 |
Table 5. Quantitative results of vision-language based interaction prediction on the InterVLA dataset.
由于所提出的不确定性感知状态空间Transformer，该模型在这两项指标上取得了最佳性能。然而我们发现，现有最先进的方法在这两项任务上的表现并不尽如人意。在这两项任务中，预测的人体运动、物体平移与旋转以及人机接触方面仍存在显著误差，这表明我们的数据集对后续的优化工作提出了重大挑战。
#### 4.5. 潜在的下游任务
稀疏视角4D场景重建。多视角外摄视频与第一人称视角视频在提供场景全局信息方面具有协同作用。现有研究[^63] [^109] [^119] [^131] [^135]致力于利用网格、神经辐射场、3D高斯泼溅等3D表示形式，从稀疏视角的图像中重建静态场景；而现有的大多数4D重建工作[^68] [^95] [^102] [^117]仅局限于密集的透视输入数据。我们包含五段外摄视角视频的数据集，有望通过提供人物和物体的姿态、网格等额外监督信号，助力4D场景重建这类下游任务的开展。
手-物交互重建。与[^30] [^31] [^127] [^128]类似，这些研究旨在共同估计双手及相互作用物体的姿态，而我们的InterVLA数据集包含第一人称视角及多物体操控场景，也能为这一具有重大挑战性的任务提供有力支持。
基于稀疏信号的运动重建。该任务旨在根据以往研究[^26] [^52] [^129]中所采用的、由第一人称视角捕捉到的下肢或手臂的稀疏信号，重建助手的全身运动。
### 5. 结论
在本文中，我们介绍了一个大规模以自我为中心的“人-物体-人”交互数据集，名为InterVLA。通过将人工辅助任务嵌入到视觉-语言-动作框架中，我们将视觉定义为以自我为中心的视角，语言定义为指导者的口头指令，动作则定义为人与物体的运动，从而证明了构建物理世界AI助手时，通用交互知识与以自我为中心的感知能力二者缺一不可。借助我们构建的这一大型数据集以及用于以自我为中心的运动估计、交互合成和交互预测的新颖基准测试，我们提供了宝贵的工具，将推动现实世界AI辅助应用领域的进一步研究与开发。
---
## 致谢
本研究部分得到了国家自然科学基金（62201342）、上海市重大科技项目（2021SHZDZX0102）、国家自然科学基金项目（62302246）、浙江省自然科学基金项目（LQ23F010008）、宁波市科研项目（2023Z237、2024Z284、2024Z289、2023CX050011、2025Z038）的支持，同时得到了东方理工大学高性能计算中心和宁波数字孪生研究院的支持。作者们感谢上海交通大学学生创新中心提供的GPU设备。
## 参考文献
[^1]: Easymocap - make human motion capture easier. . 4
[^2]: Opencv: opencv. . 4
[^3]: Hiroyasu Akada, Jian Wang, Vladislav Golyanik, and Christian Theobalt. 3d human pose perception from egocentric stereo videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 767–776, 2024. 3
[^4]: Boshi An, Yiran Geng, Kai Chen, Xiaoqi Li, Qi Dou, and Hao Dong. Rgbmanip: Monocular image-based robotic manipulation through active object pose estimation. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 7748–7755. IEEE, 2024. 3
[^5]: Joao Pedro Araújo, Jiaman Li, Karthik Vetrivel, Rishi Agarwal, Jiajun Wu, Deepak Gopinath, Alexander William Clegg, and Karen Liu. Circle: Capture in rich contextual environments. In CVPR, pages 21211–21221, 2023. 3
[^6]: Prithviraj Banerjee, Sindi Shkodrani, Pierre Moulon, Shreyas Hampali, Fan Zhang, Jade Fountain, Edward Miller, Selen Basol, Richard Newcombe, Robert Wang, et al. Introducing hot3d: An egocentric dataset for 3d hand and object tracking. arXiv preprint arXiv:2406.09598, 2024. 2
[^7]: Wentao Bao, Lele Chen, Libing Zeng, Zhong Li, Yi Xu, Junsong Yuan, and Yu Kong. Uncertainty-aware state space transformer for egocentric 3d hand trajectory forecasting. In Proceedings of the IEEE/CVF international conference on computer vision, pages 13702–13711, 2023. 8
[^8]: Homanga Bharadhwaj, Debidatta Dwibedi, Abhinav Gupta, Shubham Tulsiani, Carl Doersch, Ted Xiao, Dhruv Shah, Fei Xia, Dorsa Sadigh, and Sean Kirmani. Gen2act: Human video generation in novel scenarios enables generalizable robot manipulation, 2024. 2
[^9]: Bharat Lal Bhatnagar, Xianghui Xie, Ilya A Petrov, Cristian Sminchisescu, Christian Theobalt, and Gerard Pons-Moll. Behave: Dataset and method for tracking human object interactions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15935–15946, 2022. 3, 6
[^10]: Michael J Black, Priyanka Patel, Joachim Tesch, and Jinlong Yang. Bedlam: A synthetic dataset of bodies ex-
能够呈现细节丰富的逼真动画运动。载于CVPR会议论文集，第8726–8737页，2023年。3
[^11]: Samarth Brahmbhatt, Chengcheng Tang, Christopher D Twigg, Charles C Kemp, and James Hays. Contactpose: A dataset of grasps with object contact and hand pose. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XIII 16, pages 361–378. Springer, 2020. 3
[^12]: Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06817, 2022. 3
[^13]: Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.15818, 2023. 3
[^14]: Judith Bütepage, Ali Ghadirzadeh, Özge Öztimur Karadag, Märten Björkman, and Danica Kragic. Imitating by generating: Deep generative models for imitation of interactive tasks. Frontiers in Robotics and AI, 7:47, 2020. 2, 3
[^15]: Minjie Cai, Kris M Kitani, and Yoichi Sato. Understanding hand-object manipulation with grasp types and object attributes. In Robotics: Science and Systems, 2016. 3
[^16]: Zhe Cao, Hang Gao, Karttikeya Mangalam, Qi-Zhi Cai, Minh Vo, and Jitendra Malik. Long-term human motion prediction with scene context. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part I 16, pages 387–404. Springer, 2020. 3
[^17]: Alessandro Carfi, Francesco Foglino, Barbara Bruno, and Fulvio Mastrogiovanni. A multi-sensor dataset of human-human handover. Data in brief, 22:109–117, 2019. 2, 3
[^18]: Francesca Cini, V Ortenzi, P Corke, and MJSR Controzzi. On the choice of grasp type and location when handing over an object. Science Robotics, 4(27):eaau9757, 2019. 2, 3
[^19]: Enric Corona, Albert Pumarola, Guillem Alenyà, and Francesc Moreno-Noguer. Context-aware human motion prediction, 2020. 8
[^20]: Ana Cunha, Flora Ferreira, Emanuel Sousa, Luis Louro, Paulo Vicente, Sergio Monteiro, Wolfram Erlhagen, and Estela Bicho. Towards collaborative robots as intelligent co-workers in human-robot joint tasks: what to do and who does it? In ISR 2020; 52th International Symposium on Robotics, pages 1–8. VDE, 2020. 2
[^21]: Dima Damen, Teesid Leelasawassuk, and Walterio Mayol-Cuevas. You-do, i-learn: Egocentric unsupervised discovery of objects and their modes of interaction towards video-based guidance. Computer Vision and Image Understanding, 149:98–112, 2016. 3
## [^22]: Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. Scaling egocentric vision: The epic-kitchens dataset. In Proceedings of the European conference on computer vision (ECCV), pages 720–736, 2018. 2
---
[^23]: Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Evangelos Kazakos, Jian Ma, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. Rescaling egocentric vision: Collection, pipeline and challenges for epic-kitchens-100. International Journal of Computer Vision, pages 1–23, 2022. 2
[^24]: Christian Diller and Angela Dai. Cg-hoi: Contact-guided 3d human-object interaction generation. arXiv preprint arXiv:2311.16097, 2023. 6
[^25]: Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378, 2023. 3
[^26]: Yuming Du, Robin Kips, Albert Pumarola, Sebastian Starke, Ali Thabet, and Artsiom Sanakoyeu. Avatars grow legs: Generating smooth human motion from sparse tracking inputs with diffusion model. In CVPR, 2023. 8
[^27]: Tair Faibish, Alap Kshirsagar, Guy Hoffman, and Yael Edan. Human preferences for robot eye gaze in human-to-robot handovers. International Journal of Social Robotics, 14(4):995–1012, 2022. 3
[^28]: Ke Fan, Shunlin Lu, Minyue Dai, Runyi Yu, Lixing Xiao, Zhiyang Dou, Junting Dong, Lizhuang Ma, and Jingbo Wang. Go to zero: Towards zero-shot motion generation with million-scale data. arXiv preprint arXiv:2507.07095, 2025. 3
[^29]: Zicong Fan, Omid Taheri, Dimitrios Tzionas, Muhammed Kocabas, Manuel Kaufmann, Michael J Black, and Otmar Hilliges. Arctic: A dataset for dexterous bimanual hand-object manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12943–12954, 2023. 3, 5
[^30]: Zicong Fan, Maria Parelli, Maria Eleni Kadoglou, Xu Chen, Muhammed Kocabas, Michael J Black, and Otmar Hilliges. Hold: Category-agnostic 3d reconstruction of interacting hands and objects from video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 494–504, 2024. 8
[^31]: Zicong Fan, Takehiko Ohkawa, Linlin Yang, Nie Lin, Zhishan Zhou, Shihao Zhou, Jiajun Liang, Zhong Gao, Xu-anyang Zhang, Xue Zhang, et al. Benchmarks and challenges in pose estimation for egocentric hand interactions with objects. In European Conference on Computer Vision, pages 428–448. Springer, 2025. 8
[^32]: Alircza Fathi, Jessica K Hodgins, and James M Rehg. Social interactions: A first-person perspective. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pages 1226–1233. IEEE, 2012. 2
[^33]: Mihai Fieraru, Mihai Zanfir, Elisabeta Oneata, Alin-Ionut Popa, Vlad Olaru, and Cristian Sminchisescu. Three-dimensional reconstruction of human interactions. In CVPR, pages 7214–7223, 2020. 3
[^34]: Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In
载于IEEE/CVF计算机视觉与模式识别会议论文集，第18995–19012页，2022年。2
[^35]: Kristen Grauman, Andrew Westbury, Lorenzo Torresani, Kris Kitani, Jitendra Malik, Triantafyllos Afouras, Kumar Ashutosh, Vijay Baiyya, Siddhant Bansal, Bikram Boote, et al. Ego-exo4d: Understanding skilled human activity from first-and third-person perspectives. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19383–19400, 2024. 2, 3
[^36]: Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang, Wei Ji, Xingyu Li, and Li Cheng. Generating diverse and natural 3d human motions from text. In CVPR, pages 5152–5161, 2022. 3, 6, 7
[^37]: Wen Guo, Xiaoyu Bie, Xavier Alameda-Pineda, and Francesc Moreno-Noguer. Multi-person extreme motion prediction. In CVPR, pages 13053–13064, 2022. 2, 3
[^38]: Vladimir Guzov, Julian Chibane, Riccardo Marin, Yannan He, Yunus Saracoglu, Torsten Sattler, and Gerard Pons-Moll. Interaction replica: Tracking human-object interaction and scene changes from human motion. In 2024 International Conference on 3D Vision (3DV), pages 1006–1016. IEEE, 2024. 3
[^39]: Shreyas Hampali, Mahdi Rad, Markus Oberweger, and Vincent Lepetit. Honnotate: A method for 3d annotation of hand and object poses. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3196–3206, 2020. 3
[^40]: Shreyas Hampali, Sayan Deb Sarkar, Mahdi Rad, and Vincent Lepetit. Keypoint transformer: Solving joint identification in challenging hands and object interactions for accurate 3d pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11090–11100, 2022. 3
[^41]: Shengyu Hao, Wenhao Chai, Zhonghan Zhao, Meiqi Sun, Wendi Hu, Jieyang Zhou, Yixian Zhao, Qi Li, Yizhou Wang, Xi Li, et al. Ego3dt: Tracking every 3d object in ego-centric videos. arXiv preprint arXiv:2410.08530, 2024.
[^42]: Mohamed Hassan, Vasileios Choutas, Dimitrios Tzionas, and Michael J Black. Resolving 3d human pose ambiguities with 3d scene constraints. In ICCV, pages 2282–2292, 2019. 3
[^43]: Mohamed Hassan, Duygu Ceylan, Ruben Villegas, Jun Saito, Jimei Yang, Yi Zhou, and Michael J Black. Stochastic scene-aware motion prediction. In ICCV, pages 11374–11384, 2021. 3
[^44]: Tairan He, Zhengyi Luo, Xialin He, Wenli Xiao, Chong Zhang, Weinan Zhang, Kris Kitani, Changliu Liu, and Guanya Shi. Omnih2o: Universal and dexterous human-to-humanoid whole-body teleoperation and learning. arXiv preprint arXiv:2406.08858, 2024. 2, 3
[^45]: Tairan He, Zhengyi Luo, Wenli Xiao, Chong Zhang, Kris Kitani, Changliu Liu, and Guanya Shi. Learning human-to-humanoid real-time whole-body teleoperation, 2024. 2
## [^46]: Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a
---
双时间尺度更新规则会收敛到局部纳什均衡。载于《神经信息处理系统进展》第30卷，2017年。7
[^47]: Wenlong Huang, Chen Wang, Ruohan Zhang, Yunzhu Li, Jiajun Wu, and Li Fei-Fei. Voxposer: Composable 3d value maps for robotic manipulation with language models. arXiv preprint arXiv:2307.05973, 2023. 3
[^48]: Yinghao Huang, Omid Taheri, Michael J Black, and Dimitrios Tzionas. Intercap: Joint markerless 3d tracking of humans and objects in interaction. In DAGM German Conference on Pattern Recognition, pages 281–299. Springer, 2022. 3
[^49]: Catalin Ionescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu. Human3.6m: Large scale datasets and predictive methods for 3d human sensing in natural environments. IEEE transactions on pattern analysis and machine intelligence, 36(7):1325–1339, 2013. 5
[^50]: Yanli Ji, Feixiang Xu, Yang Yang, Fumin Shen, Heng Tao Shen, and Wei-Shi Zheng. A large-scale rgb-d database for arbitrary-view human action recognition. In ACMMM, page 1510–1518, 2018. 3
[^51]: Juntao Jian, Xiuping Liu, Manyi Li, Ruizhen Hu, and Jian Liu. Affordpose: A large-scale dataset of hand-object interactions with affordance-driven hand pose. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14713–14724, 2023. 3
[^52]: Jiaxi Jiang, Paul Streli, Huajian Qiu, Andreas Fender, Larissa Laich, Patrick Snape, and Christian Holz. Avatarposer: Articulated full-body pose tracking from sparse motion sensing. In ECCV, pages 443–460. Springer, 2022. 8
[^53]: Jiaxi Jiang, Paul Streli, Manuel Meier, and Christian Holz. Egoposer: Robust real-time ego-body pose estimation in large scenes. arXiv preprint arXiv:2308.06493, 2023. 3
[^54]: Nan Jiang, Tengyu Liu, Zhexuan Cao, Jieming Cui, Zhiyuan Zhang, Yixin Chen, He Wang, Yixin Zhu, and Siyuan Huang. Full-body articulated human-object interaction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9365–9376, 2023. 3
[^55]: Nan Jiang, Zhiyuan Zhang, Hongjie Li, Xiaoxuan Ma, Zan Wang, Yixin Chen, Tengyu Liu, Yixin Zhu, and Siyuan Huang. Scaling up dynamic human-scene interaction modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1737–1747, 2024. 3
[^56]: Zhenyu Jiang, Yuqi Xie, Jinhan Li, Ye Yuan, Yifeng Zhu, and Yuke Zhu. Harmon: Whole-body motion generation of humanoid robots from language descriptions. arXiv preprint arXiv:2410.12773, 2024. 2, 3
[^57]: Evangelos Kazakos, Arsha Nagrani, Andrew Zisserman, and Dima Damen. Epic-fusion: Audio-visual temporal binding for egocentric action recognition. In Proceedings of the IEEE/CVF international conference on computer vision, pages 5492–5501, 2019. 3
[^58]: Soheil Khatibi, Meisam Teimouri, and Mahdi Rezaei. Real-time active vision for a humanoid soccer robot using deep reinforcement learning. arXiv preprint arXiv:2011.13851, 2020. 3
[^59]: Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source vision-language-action model. arXiv preprint arXiv:2406.09246, 2024. 3
[^60]: Alap Kshirsagar, Raphael Fortuna, Zhiming Xie, and Guy Hoffman. Dataset of bimanual human-to-human object handovers. Data in Brief, 48:109277, 2023. 3
[^61]: Taein Kwon, Bugra Tekin, Jan Stühmer, Federica Bogo, and Marc Pollefeys. H2o: Two hands manipulating objects for first person interaction recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10138–10148, 2021. 3
[^62]: Jiaman Li, Karen Liu, and Jiajun Wu. Ego-body pose estimation via ego-head pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17142–17151, 2023. 2, 3
[^63]: Jiahe Li, Jiawei Zhang, Xiao Bai, Jin Zheng, Xin Ning, Jun Zhou, and Lin Gu. Dngaussian: Optimizing sparse-view 3d gaussian radiance fields with global-local depth normalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20775–20785, 2024. 8
[^64]: Yin Li, Miao Liu, and James M Rehg. In the eye of beholder: Joint learning of gaze and actions in first person video. In Proceedings of the European conference on computer vision (ECCV), pages 619–635, 2018. 2
[^65]: Yanghao Li, Tushar Nagarajan, Bo Xiong, and Kristen Grauman. Ego-exo: Transferring visual representations from third-person to first-person videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6943–6953, 2021. 3
[^66]: Han Liang, Wenqian Zhang, Wenxuan Li, Jingyi Yu, and Lan Xu. Intergen: Diffusion-based multi-human motion generation under complex interactions. arXiv preprint arXiv:2304.05684, 2023. 2, 3
[^67]: Jing Lin, Ailing Zeng, Shunlin Lu, Yuanhao Cai, Ruimao Zhang, Haoqian Wang, and Lei Zhang. Motion-x: A large-scale 3d expressive whole-body human motion dataset. arXiv preprint arXiv:2307.00818, 2023. 3
[^68]: Youtian Lin, Zuozhuo Dai, Siyu Zhu, and Yao Yao. Gaussian-flow: 4d reconstruction with dynamic 3d gaussian particle. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21136–21145, 2024. 8
[^69]: Jun Liu, Amir Shahroudy, Mauricio Perez, Gang Wang, Ling-Yu Duan, and Alex C Kot. Ntu rgb+ d 120: A large-scale benchmark for 3d human activity understanding. T-PAMI, 42(10):2684–2701, 2019. 3
[^70]: Miao Liu, Siyu Tang, Yin Li, and James M Rehg. Forecasting human-object interaction: joint prediction of motor attention and actions in first person video. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part I 16, pages 704–721. Springer, 2020. 8
## [^71]: Shaowei Liu, Subarna Tripathi, Somdeb Majumdar, and Xiaolong Wang. Joint hand motion and interaction hotspots
---
基于以自我为中心的视频进行预测。载于IEEE/CVF计算机视觉与模式识别会议论文集，第3282–3292页，2022年。8
[^72]: Yunze Liu, Yun Liu, Che Jiang, Kangbo Lyu, Weikang Wan, Hao Shen, Boqiang Liang, Zhoujie Fu, He Wang, and Li Yi. Hoi4d: A 4d egocentric dataset for category-level human-object interaction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21013–21022, 2022. 3
[^73]: Yun Liu, Haolin Yang, Xu Si, Ling Liu, Zipeng Li, Yuxiang Zhang, Yebin Liu, and Li Yi. Taco: Benchmarking generalizable bimanual tool-action-object understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21740–21751, 2024. 3
[^74]: Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J. Black. SMPL: a skinned multi-person linear model. ACM Trans. Graph., 34(6): 248:1–248:16, 2015. 5
[^75]: Zhengyi Luo, Ryo Hachiuma, Ye Yuan, and Kris Kitani. Dynamics-regulated kinematic policy for egocentric pose estimation. Advances in Neural Information Processing Systems, 34:25019–25032, 2021. 3
[^76]: Xintao Lv, Liang Xu, Yichao Yan, Xin Jin, Congsheng Xu, Shuwen Wu, Yifan Liu, Lincheng Li, Mengxiao Bi, Wenjun Zeng, et al. Himo: A new benchmark for full-body human interacting with multiple objects. In European Conference on Computer Vision, pages 300–318. Springer, 2025. 2, 3, 4, 5, 6, 7
[^77]: Junyi Ma, Jingyi Xu, Xieyuanli Chen, and Hesheng Wang. Diff-ip2d: Diffusion-based hand-object interaction prediction on egocentric videos. arXiv preprint arXiv:2405.04370, 2024. 2, 3
[^78]: Yueen Ma, Zixing Song, Yuzheng Zhuang, Jianye Hao, and Irwin King. A survey on vision-language-action models for embodied ai. arXiv preprint arXiv:2405.14093, 2024. 3
[^79]: Christian Mandery, Ömer Terlemez, Martin Do, Nikolaus Vahrenkamp, and Tamim Asfour. Unifying representations and large-scale whole-body motion databases for studying human motion. IEEE Transactions on Robotics, 32(4):796–809, 2016. 3
[^80]: Priyanka Mandikal and Kristen Grauman. Dexvip: Learning dexterous grasping with human hand pose priors from video. In Conference on Robot Learning, pages 651–661. PMLR, 2022. 2
[^81]: Esteve Valls Mascaro, Daniel Sliwowski, and Dongheui Lee. Hoi4abot: Human-object interaction anticipation for human intention reading collaborative robots, 2024. 2
[^82]: Christen Millerdurai, Hiroyasu Akada, Jian Wang, Diogo Luvizon, Christian Theobalt, and Vladislav Golyanik. Eventego3d: 3d human motion capture from egocentric event streams. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1186–1195, 2024. 3
[^83]: Aron Monszpart, Paul Guerrero, Duygu Ceylan, Ersin Yumer, and Niloy J Mitra. imapper: interaction-guided scene mapping from monocular videos. ACM Transactions On Graphics (TOG), 38(4):1–15, 2019. 3
[^84]: Tushar Nagarajan, Christoph Feichtenhofer, and Kristen Grauman. Grounded human-object interaction hotspots from video. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8688–8697, 2019.
[^85]: Katsuyuki Nakamura, Serena Yeung, Alexandre Alahi, and Li Fei-Fei. Jointly learning energy expenditures and activities using egocentric multimodal signals. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1868–1877, 2017. 3
[^86]: Evonne Ng, Donglai Xiang, Hanbyul Joo, and Kristen Grauman. You2me: Inferring body pose in egocentric video via first and second person interactions. In CVPR, pages 9890–9900, 2020. 2, 3
[^87]: Abby O’Neill, Abdul Rehman, Abhinav Gupta, Abhiram Maddukuri, Abhishek Gupta, Abhishek Padalkar, Abraham Lee, Acorn Pooley, Agrim Gupta, Ajay Mandlekar, et al. Open x-embodiment: Robotic learning datasets and rt-x models. arXiv preprint arXiv:2310.08864, 2023. 3
[^88]: OpenAI. GPT-3.5 turbo fine-tuning and api updates. , 2023.4
[^89]: Xiaogang Peng, Yiming Xie, Zizhao Wu, Varun Jampani, Deqing Sun, and Huaizu Jiang. Hoi-diff: Text-driven synthesis of 3d human-object interactions using diffusion models. arXiv preprint arXiv:2312.06553, 2023. 6
[^90]: Mathis Petrovich, Michael J Black, and Gül Varol. Action-conditioned 3d human motion synthesis with transformer vae. In CVPR, pages 10985–10995, 2021. 6
[^91]: Matthias Plappert, Christian Mandery, and Tamim Asfour. The kit motion-language dataset. Big data, 4(4):236–252, 2016. 3
[^92]: Vignesh Prasad, Dorothea Koert, Ruth Stock-Homburg, Jan Peters, and Georgia Chalvatzaki. Mild: multimodal interactive latent dynamics for learning human-robot interaction. In 2022 IEEE-RAS 21st International Conference on Humanoid Robots (Humanoids), pages 472–479. IEEE, 2022. 3
[^93]: Vignesh Prasad, Alap Kshirsagar, Dorothea Koert Ruth Stock-Homburg, Jan Peters, and Georgia Chalvatzaki. Moveint: Mixture of variational experts for learning human-robot interactions from demonstrations. IEEE Robotics and Automation Letters, 2024. 2, 3
[^94]: Sergi Pujades, Betty Mohler, Anne Thaler, Joachim Tesch, Naureen Mahmood, Nikolas Hesse, Heinrich H. Bülthoff, and Michael J. Black. The virtual caliper: Rapid creation of metrically accurate avatars from 3d measurements. IEEE transactions on visualization and computer graphics, 25(5):1887–1897, 2019.
[^95]: Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer. D-nerf: Neural radiance fields for dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10318–10327, 2021. 8
## [^96]: Abhinanda R Punnakkal, Arjun Chandrasekaran, Nikos Athanasiou, Alejandra Quiros-Ramirez, and Michael J Black. Babel: bodies, action and behavior with english labels. In CVPR, pages 722–731, 2021. 3
---
[^97]: Yuzhe Qin, Yueh-Hua Wu, Shaowei Liu, Hanwen Jiang, Ruihan Yang, Yang Fu, and Xiaolong Wang. Dexmv: Imitation learning for dexterous manipulation from human videos. In European Conference on Computer Vision, pages 570–587. Springer, 2022. 2
[^98]: Ayumu Sasagawa, Kazuki Fujimoto, Sho Sakaino, and Toshiaki Tsuji. Imitation learning based on bilateral control for human-robot cooperation. IEEE Robotics and Automation Letters, 5(4):6169–6176, 2020. 2
[^99]: Manolis Savva, Angel X Chang, Pat Hanrahan, Matthew Fisher, and Matthias Nießner. Pigraphs: learning interaction snapshots from observations. ACM Transactions On Graphics (TOG), 35(4):1–12, 2016. 3
[^100]: Mingyo Seo, Steve Han, Kyutae Sim, Seung Hyeon Bang, Carlos Gonzalez, Luis Sentis, and Yuke Zhu. Deep imitation learning for humanoid loco-manipulation through human teleoperation, 2023. 2
[^101]: Yonatan Shafir, Guy Tevet, Roy Kapon, and Amit H Bermano. Human motion diffusion as a generative prior. arXiv preprint arXiv:2303.01418, 2023. 7
[^102]: Meng-Li Shih, Jia-Bin Huang, Changil Kim, Rajvi Shah, Johannes Kopf, and Chen Gao. Modeling ambient scene dynamics for free-view synthesis. In ACM SIGGRAPH 2024 Conference Papers, pages 1–11, 2024. 8
[^103]: Soyong Shin, Juyong Kim, Eni Halilaj, and Michael J Black. Wham: Reconstructing world-grounded humans with accurate 3d motion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2070–2080, 2024. 6, 7
[^104]: Yu Sun, Qian Bao, Wu Liu, Tao Mei, and Michael J Black. Trace: 5d temporal regression of avatars with dynamic cameras in 3d environments. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8856–8866, 2023. 6, 7
[^105]: Omid Taheri, Nima Ghorbani, Michael J Black, and Dimitrios Tzionas. Grab: A dataset of whole-body human grasping of objects. In ECCV, pages 581–600, 2020. 3, 4
[^106]: Guy Tevet, Sigal Raab, Brian Gordon, Yonatan Shafir, Daniel Cohen-Or, and Amit H Bermano. Human motion diffusion model. arXiv preprint arXiv:2209.14916, 2022. 7, 8
[^107]: Dhruva Tirumala, Markus Wulfmeier, Ben Moran, Sandy Huang, Jan Humplik, Guy Lever, Tuomas Haarnoja, Leonard Hasenclever, Arunkumar Byravan, Nathan Batchelor, et al. Learning robot soccer from egocentric vision with deep reinforcement learning. arXiv preprint arXiv:2405.02425, 2024. 3
[^108]: NP Van der Aa, Xinghan Luo, Geert-Jan Giezeman, Robby T Tan, and Remco C Veltkamp. Umpm benchmark: A multi-person dataset with synchronized video and motion capture data for evaluation of articulated human motion and interaction. In ICCV Workshops, pages 1264–1269. IEEE, 2011. 3
[^109]: Guangcong Wang, Zhaoxi Chen, Chen Change Loy, and Ziwei Liu. Sparsenerf: Distilling depth ranking for few-shot novel view synthesis. In Proceedings of the IEEE/CVF
载于国际计算机视觉会议论文集，第9065–9076页，2023年。8
[^110]: Jian Wang, Diogo Luvizon, Weipeng Xu, Lingjie Liu, Kripasindhu Sarkar, and Christian Theobalt. Scene-aware egocentric 3d human pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13031–13040, 2023. 3
[^111]: Jian Wang, Zhe Cao, Diogo Luvizon, Lingjie Liu, Kripasindhu Sarkar, Danhang Tang, Thabo Beeler, and Christian Theobalt. Egocentric whole-body motion capture with fisheyevit and diffusion-based motion refinement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 777–787, 2024. 3
[^112]: Xin Wang, Taein Kwon, Mahdi Rad, Bowen Pan, Ishani Chakraborty, Sean Andrist, Dan Bohus, Ashley Feniello, Bugra Tekin, Felipe Vieira Frujeri, et al. Holoassist: an egocentric human interaction dataset for interactive ai assistants in the real world. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 20270–20281, 2023. 2, 3
[^113]: Ye Wang, Sipeng Zheng, Bin Cao, Qianshan Wei, Qin Jin, and Zongqing Lu. Quo vadis, motion generation? from large language models to large motion models, 2024. 3
[^114]: Yufu Wang, Ziyun Wang, Lingjie Liu, and Kostas Daniilidis. Tram: Global trajectory and motion of 3d humans from in-the-wild videos. In European Conference on Computer Vision, pages 467–487. Springer, 2025. 6,7
[^115]: Zifan Wang, Junyu Chen, Ziqing Chen, Pengwei Xie, Rui Chen, and Li Yi. Genh2r: Learning generalizable human-to-robot handover via scalable simulation demonstration and imitation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16362–16372, 2024. 2
[^116]: Noah Wiederhold, Ava Megyeri, DiMaggio Paris, Sean Banerjee, and Natasha Banerjee. Hoh: Markerless multimodal human-object-human handover dataset with large object count. Advances in Neural Information Processing Systems, 36, 2024. 2, 3
[^117]: Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Xinggang Wang. 4d gaussian splatting for real-time dynamic scene rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20310–20320, 2024. 8
[^118]: Zeqi Xiao, Tai Wang, Jingbo Wang, Jinkun Cao, Wenwei Zhang, Bo Dai, Dahua Lin, and Jiangmiao Pang. Unified human-scene interaction via prompted chain-of-contacts. arXiv preprint arXiv:2309.07918, 2023. 2
[^119]: Haolin Xiong, Sairisheek Muttukuru, Rishi Upadhyay, Pradyumna Chari, and Achuta Kadambi. Sparsegs: Real-time 360 $\{\backslash deg\}$ sparse view synthesis using gaussian splatting. arXiv preprint arXiv:2312.00206, 2023. 8
## [^120]: Liang Xu, Ziyang Song, Dongliang Wang, Jing Su, Zhicheng Fang, Chenjing Ding, Weihao Gan, Yichao Yan, Xin Jin, Xiaokang Yang, et al. Actformer: A gan-based transformer towards general action-conditioned 3d human motion generation. In ICCV, pages 2228–2238, 2023. 3
---
[^121]: Liang Xu, Shaoyang Hua, Zili Lin, Yifan Liu, Feipeng Ma, Yichao Yan, Xin Jin, Xiaokang Yang, and Wenjun Zeng. Motionbank: A large-scale video motion benchmark with disentangled rule-based annotations, 2024. 3
[^122]: Liang Xu, Xintao Lv, Yichao Yan, Xin Jin, Shuwen Wu, Congsheng Xu, Yifan Liu, Yizhou Zhou, Fengyun Rao, Xingdong Sheng, et al. Inter-x: Towards versatile human-human interaction analysis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22260–22271, 2024. 2, 3, 5
[^123]: Liang Xu, Yizhou Zhou, Yichao Yan, Xin Jin, Wenhan Zhu, Fengyun Rao, Xiaokang Yang, and Wenjun Zeng. Regenet: Towards human action-reaction synthesis. In CVPR, pages 1759–1769, 2024. 3
[^124]: Sirui Xu, Zhengyuan Li, Yu-Xiong Wang, and Liang-Yan Gui. InterDiff: Generating 3d human-object interactions with physics-informed diffusion. In ICCV, 2023. 8
[^125]: Xiang Xu, Hanbyul Joo, Greg Mori, and Manolis Savva. D3d-hoi: Dynamic 3d human-object interactions from videos. arXiv preprint arXiv:2108.08420, 2021. 3
[^126]: Xinyu Xu, Yizheng Zhang, Yong-Lu Li, Lei Han, and Cewu Lu. Humanvla: Towards vision-language directed object rearrangement by physical humanoid. arXiv preprint arXiv:2406.19972, 2024. 3
[^127]: Yufei Ye, Abhinav Gupta, and Shubham Tulsiani. What's in your hands? 3d reconstruction of generic objects in hands. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3895–3905, 2022. 8
[^128]: Yufei Ye, Poorvi Hebbar, Abhinav Gupta, and Shubham Tulsiani. Diffusion-guided reconstruction of everyday hand-object interaction clips. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 19717–19728, 2023. 8
[^129]: Brent Yi, Vickie Ye, Maya Zheng, Lea Müller, Georgios Pavlakos, Yi Ma, Jitendra Malik, and Angjoo Kanazawa. Estimating body and hand motion in an ego-sensed world, 2024. 8
[^130]: Yifei Yin, Chen Guo, Manuel Kaufmann, Juan Jose Zarate, Jie Song, and Otmar Hilliges. Hi4d: 4d instance segmentation of close human interaction. In CVPR, pages 17016–17027, 2023. 2, 3
[^131]: Hanyang Yu, Xiaoxiao Long, and Ping Tan. Lm-gaussian: Boost sparse-view 3d gaussian splatting with large model priors. arXiv preprint arXiv:2409.03456, 2024. 8
[^132]: Ye Yuan, Umar Iqbal, Pavlo Molchanov, Kris Kitani, and Jan Kautz. Glamr: Global occlusion-aware human mesh recovery with dynamic cameras. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11038–11049, 2022. 6, 7
[^133]: Kiwon Yun, Jean Honorio, Debaleena Chattopadhyay, Tamara L Berg, and Dimitris Samaras. Two-person interaction detection using body-pose features and multiple instance learning. In CVPR, pages 28–35. IEEE, 2012. 3
[^134]: Chengwen Zhang, Yun Liu, Ruofan Xing, Bingda Tang, and Li Yi. Core4d: A 4d human-object-human interaction dataset for collaborative object rearrangement. arXiv preprint arXiv:2406.19353, 2024. 2, 3, 8
[^135]: Chuanrui Zhang, Yingshuang Zou, Zhuoling Li, Minmin Yi, and Haoqian Wang. Transplat: Generalizable 3d gaussian splatting from sparse multi-view images with transformers. arXiv preprint arXiv:2408.13770, 2024. 8
[^136]: Hang Zhang, Wenxiao Zhang, Haoxuan Qu, and Jun Liu. Enhancing human-centered dynamic scene understanding via multiple llms collaborated reasoning. Visual Intelligence, 3(1):3, 2025. 2
[^137]: Juze Zhang, Jingyan Zhang, Zining Song, Zhanhe Shi, Chengfeng Zhao, Ye Shi, Jingyi Yu, Lan Xu, and Jingya Wang. Hoi-m $^{3}$: Capture multiple humans and objects interaction within contextual environment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 516–526, 2024. 2, 6
[^138]: Xiaohan Zhang, Bharat Lal Bhatnagar, Sebastian Starke, Vladimir Guzov, and Gerard Pons-Moll. Couch: towards controllable human-chair interactions. In ECCV, pages 518–535, 2022. 3
[^139]: Xiaohan Zhang, Sebastian Starke, Vladimir Guzov, Zhensong Zhang, Eduardo Pérez Pellitero, and Gerard Pons-Moll. Scenic: Scene-aware semantic navigation with instruction-guided control. arXiv preprint arXiv:2412.15664, 2024. 2
[^140]: Haoyu Zhen, Xiaowen Qiu, Peihao Chen, Jincheng Yang, Xin Yan, Yilun Du, Yining Hong, and Chuang Gan. 3d-vla: A 3d vision-language-action generative world model. arXiv preprint arXiv:2403.09631, 2024. 3
[^141]: Yipin Zhou and Tamara L Berg. Temporal perception and prediction in ego-centric video. In Proceedings of the IEEE International Conference on Computer Vision, pages 4498–4506, 2015. 3
[^142]: Yi Zhou, Connelly Barnes, Jingwan Lu, Jimei Yang, and Hao Li. On the continuity of rotation representations in neural networks. In CVPR, pages 5745–5753. Computer Vision Foundation / IEEE, 2019. 5, 6
[^143]: Shihao Zou, Xinxin Zuo, Yiming Qian, Sen Wang, Chi Xu, Minglun Gong, and Li Cheng. 3d human shape reconstruction from a polarization image. In ECCV, pages 351–368, 2020. 3