# EgoM2P：以自我为中心的多模态多任务预训练 李根 $^{1}$ 陈宇彤 $^{*1}$ 吴一谦 $^{*1,2}$ 赵凯峰 $^{*1}$ 马克·波勒费斯 $^{1,3}$ 唐思宇 $^{1}$ $^{1}$苏黎世联邦理工学院 $^{2}$浙江大学 $^{3}$微软 ## 摘要 理解以自我为中心的视觉中的多模态信号——比如RGB视频、深度数据、相机位姿和视线信息——对于增强现实、机器人技术以及人机交互领域的应用至关重要，这些应用能让系统更好地解读佩戴相机者的行为、意图以及周围环境。然而，构建大规模以自我为中心的多模态多任务模型面临着独特的挑战。以自我为中心的数据本身具有异质性，不同设备和场景下的模态覆盖情况差异很大。为缺失的模态（比如视线或头戴式相机轨迹）生成伪标签往往不可行，这使得传统的监督学习方法难以扩展。此外，动态的相机运动以及第一人称视频复杂的时间和空间结构，也给现有多模态基础模型的直接应用带来了额外挑战。 为解决这些挑战，我们提出了一组高效的时间令牌化器，并提出了EgoM2P——一种掩码建模框架，它利用时间感知的多模态令牌来训练用于以自我为中心的4D理解的大规模通用模型。这种统一的设计支持多种以自我为中心的感知与合成任务，包括视线预测、以自我为中心的相机跟踪，以及从以自我为中心的视频中估计单目深度；同时它也可作为条件以自我为中心视频合成的生成模型。在这些任务中，EgoM2P的性能与专业模型相当甚至更优，而运行速度却快一个数量级。我们将完全开源EgoM2P，以支持社区发展并推动以自我为中心的视觉研究进步。 ### 1. 引言 随着多模态数据的融入，以自我为中心的视频采集技术取得了显著进展，这些数据包括RGB图像、深度数据、视线信息和相机轨迹。这些模态之间相互作用，为理解人的头部运动、意图以及场景几何结构提供了最关键的信息。现实世界中多模态以自我为中心视频采集方法的日益普及，也提升了图像检测的效率与准确性。 ![Image](images/0701b1d32648b089e206583859296ec87ec56012a2a4f50667c9691bac8ec586.jpg)
Figure 1. EgoM2P: A large-scale egocentric multimodal and multitask model, pretrained on eight extensive egocentric datasets. It incorporates four modalities—RGB and depth video, gaze dynamics, and camera trajectories—to handle challenging tasks like monocular egocentric depth estimation, camera tracking, gaze estimation, and conditional egocentric video synthesis. For simplicity, we only visualize four frames here.
EgoExo4D、HoloAssist和HOT3D等丰富的、多样化的且具备语义意义的数据集[^5] [^10] [^20] [^23] [^26] [^29] [^30] [^49] [^60] [^61] [^69] [^70] [^77] [^87] [^106] [^122] [^125]，为相关研究提供了宝贵资源。此外，像EgoGen[^52]这样的模拟器生成的大规模合成数据集，能提供精确的真实标注，而这些标注在现实中获取往往成本高昂且耗时漫长。通过整合这些互补的数据源，训练用于以自我为中心的视觉的大规模多模态多任务模型变得越来越可行。这类模型有望进一步加深我们对人类行为和现实世界互动的理解，为增强现实、虚拟现实和机器人技术领域的应用开辟新可能。 当前的大规模视频模型主要侧重于理解和生成第三人称视角的视频：视频理解任务涵盖视频字幕生成、问答、检索和分割 [^39] [^51] [^59] [^71] [^112] [^116] [^121] [^123] [^127]；视频生成模型主要根据文本输入生成第三人称视角的视频 [^11] [^12] [^17] [^22] [^31] [^35] [^37] [^45] [^67] [^68] [^78] [^85] [^88] [^100] [^107] [^111] [^117]。近期的研究 [^48] [^81] 将模态范围扩展到了音频，但这些模型仍只能在有限的模态范围内运作。尽管第三人称视频模型取得了显著进展，但以人为中心的模型发展却相对缓慢。 --- 以人为中心的基础模型发展速度要慢得多。第一人称视角带来了独特的挑战，包括动态摄像头运动、复杂的人物-物体交互以及人类意图的影响，而这些正是第三人称模型未能充分解决的。此外，由于可用的第一人称数据集规模有限，大型以人为中心模型的开发也受到了阻碍。随着第一人称数据集规模的扩大，近期在以人为中心的基础模型方面的研究 [^90] [^118] 主要集中在以人为中心的视频问答上。然而，这些模型往往忽略了关键的人类相关模态，而这些模态恰好能被头戴式摄像头捕捉到，对于理解人类意图至关重要。另外，这些模型缺乏3D或4D重建能力，因此无法完全捕捉人类动作和意图的时空特征。 多模态和多任务视觉基础模型的最新进展主要集中在图像领域，它们在跨模态预测方面展现出了卓越的能力 [^8] [^18] [^47] [^64] [^65] [^76] [^103] [^105]。这类基于Transformer架构 [^99] 的模型能够实现灵活的多对多预测，从而支持深度、表面法线、分割掩码等多种模态下的多任务处理。不过，这些模型依赖伪标签网络来生成跨模态的对齐数据，但由于与第一人称视图存在领域差异，适用于第一人称视频的有效伪标签器仍然十分有限。此外，这些模型侧重于单图像预测，当应用于摄像头姿态快速变化的第一人称视频序列时，难以保持时间一致性。 在本文中，我们提出了EgoM2P，这是首个面向4D第一人称数据的多模态多任务模型。我们的方法涵盖了四种模态：RGB视频、深度视频、视线动态以及摄像头轨迹。EgoM2P支持多对多模态预测，并在多种任务中展现了其多任务处理能力：视线估计、第一人称摄像头跟踪、从单目第一人称视频中估计深度，以及条件性第一人称视频生成。 具体来说，我们通过整理来自真实世界和合成数据的八个大规模以眼为中心的数据集，构建了一个包含四十亿个训练词元的多模态词元数据库。为了解决模态标注缺失的问题，我们将原本为图像基础模型设计、且假设所有模态均可用时的多模态掩码预训练技术，有效扩展到了以眼为中心的视频领域。即便某些模态被掩码掉，EgoM2P仍能有效地预测它们。我们设计了统一的时间令牌化器架构，用于将多模态数据转化为具有时间感知能力的词元。通过在EgoM2P的训练过程中采用可变的掩码率来掩盖输入词元和目标词元，我们得以利用其并行推理能力，并在各类下游应用中展现出出色的多任务处理性能，同时大幅提升了运算速度。总而言之，本研究的贡献如下： 1. 我们提出了EgoM2P，这是首个面向RGB、深度视频、眼动动态以及相机轨迹的多模态多任务大型以眼为中心模型。 2. 我们通过应对更复杂的时空动态特性以及某些固有缺失模态的标注不足等挑战，将多模态掩码预训练技术从图像领域扩展到了以眼为中心的视频领域。 3. 在以眼为中心的相机跟踪、眼动动态估计、单目以眼为中心深度估计以及条件化以眼为中心视频合成等任务中，EgoM2P的性能可与最先进的算法相媲美甚至更优，同时效率也显著更高。 ### 2. 相关工作 图像基础模型。图像基础模型是经过预训练的通用神经网络，可作为各类下游视觉任务的基础，例如图像分类、检测和分割 [^15] [^16] [^21] [^46] [^62] [^79]。CLIP [^82] 通过对比学习来对齐图像和文本嵌入，从而实现了零样本和开放词汇分类。ImageBind [^27] 将多模态对齐的范围拓展到了文本之外，将图像与音频、深度及热成像等模态连接起来。多模态语言模型 [^1] [^2] [^41] [^57] [^72] [^92] [^113] 能够实现跨文本、图像和音频的统一推理。近期的研究工作4M [^8] [^76] 利用Transformer训练了多模态图像基础模型，从而能够支持任意输入-输出模态对之间的预测。我们的工作通过引入时序建模，对4M进行了拓展，训练出了面向以眼为中心视觉的统一多模态多任务模型。 视频基础模型。近期视频基础模型的进展建立在视觉语言模型（VLM）的成功之上，主要通过多种方法实现视频理解与生成，例如视频-语言对比学习 [^109] [^110] [^114]、掩码建模 [^25] [^95] [^102]，以及自回归序列预测 [^4] [^53] [^55] [^71] [^91]。基于扩散的视频生成模型 [^3] [^11] [^78] [^81] [^93]能够通过文本提示和参考图像等条件信号，实现具有精细内容控制的照片级真实感视频生成。这些能力使视频模型有望成为世界模型的候选者 [^3] [^14] [^32] [^74] [^97]，因为其生成过程本质上能够从互联网规模的数据中捕捉现实世界的时序动态。 第一人称视角视频理解与生成。通过第一人称视角视频理解世界，对于增强现实、虚拟现实和机器人等领域的应用至关重要。目前已收集了多个第一人称视角视频数据集 [^10] [^20] [^29] [^30] [^106]，以捕捉日常生活场景的多样性和复杂性。这些第一人称视角视频带来了重大的技术挑战，包括：活动 ---
Datasets\nModalitiesRGBDepthGazeCamera
EgoExo4D [^30]✓✗✓✓
HoloAssist [^106]✓✓ $^{{*}}$✓✓
HOT3D (Aria) [^10]✓✓ $^{{*}}$✓✓
HOT3D (Quest) [^10]gray✓ $^{{*}}$✗✓
ARCTIC [^23]✓✓ $^{{*}}$✗✓
TACO [^61]✓✓ $^{{*}}$✗✓
H2O [^49]✓✓✗✓
EgoGen [^52]✓✓✗✓
Table 1. Datasets used in our method. A green checkmark (✓) signifies availability, a red cross (×) denotes unavailability or exclusion due to low quality, and a blue checkmark with a star (✓*) indicates the use of pseudo labels.
识别 [^42] [^101] [^129]、手部动作与物体交互估计 [^9] [^24] [^119]、第一人称视角视频预测与生成 [^56] [^58] [^108] [^115]、第一人称视角相机定位 [^86] [^94] 等。尽管针对这些挑战已经开发了特定任务的方法，但目前仍缺乏统一的第一人称视角视频基础模型。我们的工作填补了这一空白，实现了RGB、深度、相机姿态和视线信号之间的跨模态序列预测，这代表了迈向用于统一理解场景与人类行为的多任务基础模型的初步尝试。 ### 3. 方法 本节概述数据整理流程与训练范式。3.1节将异构的第一人称视角数据集转换为统一格式。3.2节描述了将高维多模态数据压缩为紧凑离散标记的过程，从而实现高效的训练与推理。接下来，3.3节介绍了多模态离散标记的嵌入以及EgoM2P的掩码预训练。最后，3.4节解释了如何通过从预训练模型中采样来预测最终的目标标记。 #### 3.1. 数据整理 如表1所示，不同第一人称视角数据集在数据标注覆盖范围上存在差异。由于硬件和处理限制，采集的深度数据中常出现帧丢失现象，这使得可穿戴AR眼镜 [^73] [^75] 难以实现像素对齐的深度流。此外，安装在头盔上的Kinect相机无法捕捉视线信息。这种非结构化的第一人称视角数据特性给数据整理和模型训练都带来了挑战。 我们的数据整理流程包含3个步骤：1）拆分，2）标注，3）标准化： 拆分。原始的多模态数据被分割成长度为T帧的片段，这些视频片段会被重新编码为分辨率不变的高质量mp4格式。 标注。现实世界中的第一人称深度数据十分稀缺，且往往包含传感器噪声、存在丢帧现象。我们利用RollingDepth [^44]生成深度标注用的伪标签，从而得到像素对齐的深度视频。为了进一步扩大精准深度训练数据的规模，我们采用了EgoGen [^52]——一种新型的第一人称合成数据生成器，来生成约30小时、帧率为30 FPS的视频数据。具体做法是让虚拟人物在Replica [^89]场景以及GIMO [^128]场景扫描结果中行走，渲染出他们的第一人称视角，进而获得精准的深度和相机轨迹标注。如补充材料第B.3节所分析，EgoGen能够提升EgoM2P的性能。对于没有注视标注的数据集，由于缺少针对注视动态的有效伪标签生成器，我们暂不对其进行标注。 标准化。这些数据集的视频分辨率各异，定义的世界坐标系也不尽相同，包括原点、坐标轴约定、左右手规则和缩放比例都存在差异。我们对多模态数据的标准化方式如下：所有数据流均为30 FPS；深度视频采用逆深度表示进行编码，且每个序列会单独做归一化处理；带噪的Kinect深度标签会先通过形态学操作和图像修复填补空洞，再经中值滤波和双边滤波去噪。原本以带深度的3D射线形式表示的视线数据会被重新投影到2D图像平面上，转化为该平面上的移动点；相机轨迹则统一为OpenCV约定下的相机到世界变换矩阵，以第一帧为参考帧，从而实现不同数据集间世界坐标的标准化。 #### 3.2. 分词器 给定一个包含T帧的多模态片段，我们对每种模态的表示方法如下： - RGB视频：$\mathbf{X}^{\mathrm{rgb}} \in \mathbb{R}^{T \times H \times W \times 3}$，其中H和W分别表示空间分辨率。 • 深度视频：$X^{\text{depth}} \in \mathbb{R}^{T \times H \times W \times 1}$。 • 注视动态：$\mathbf{X}^{\text{gaze}} \in \mathbb{R}^{T \times 2}$，其中每个条目对应2D注视坐标。 • 相机轨迹：$\mathbf{X}^{\mathrm{cam}} \in \mathbb{R}^{T \times 9}$，其中每个姿态由6D旋转表示[^130]和平移参数描述，参考帧为第一帧。 我们的多模态数据集表示为$\mathbf{X} = \{\mathbf{X}^{\mathrm{rgb}}, \mathbf{X}^{\mathrm{depth}}, \mathbf{X}^{\mathrm{gaze}}, \mathbf{X}^{\mathrm{cam}}\}$，根据表1所示，每个子数据集可能缺失部分模态。 我们利用最先进的Cosmos分词器[^3]对视频模态进行分词，采用4倍时间压缩率和8倍空间压缩率将视频转换为离散词元。对于其他模态，我们训练特定于该模态的分词器。为确保引入新模态时的适应性，我们采用了基于Transformer的矢量量化变分自编码器（VQ-VAE）[^98]作为统一的分词器架构。网络架构展示在上方。 --- ![Image](images/94075a9a5d986b17e849e1938ac13edfd74161c094fafb5d6044a0a191f33b9b.jpg)
Figure 2. Network Architecture: (1) We train VQ-VAE [^98] tokenizers for camera trajectories and gaze dynamics (Sec. 3.2), and adopt Cosmos tokenizers [^3] to tokenize RGB and depth streams. High-dimensional input modalities, including videos, gaze dynamics, and camera trajectories, are compressed into discrete tokens to serve as our training database. (2) Our EgoM2P follows the architecture of T5-Base [^83]. We perform multimodal masked pretraining (Sec. 3.3), where we randomly sample a fixed number of input and target tokens from our token database without overlap. For simplicity, we only visualize four frames here.
图2. 对于每种模态 $mod \in \{\text{gaze}, \text{cam}\}$，设 $N = \dim(\mathbf{X}^{mod}) - 1$。编码器 $\mathcal{E}$ 首先执行N维卷积操作，表示为 $\text{Conv}_N \mathrm{d}(\mathbf{X}^{mod})$。该操作会沿时间轴将数据下采样2倍。接着，它会在嵌入数据中加入N维位置编码，随后将其传入12个Transformer块。每个Transformer块会关注所有标记之间的两两交互关系。之后，我们使用量化器 $\mathcal{Q}$ 对嵌入向量进行量化，以学习特定于模态的码本，其中采用余弦相似度作为距离度量，具体可参考 [^76] 和 [^120]。最后，离散码通过解码器 $\mathcal{D}$ 进行解码，该解码器的架构与编码器类似： $$ \mathbf{z}=\mathcal{E}(\mathbf{X}^{mod}) $$ $$ \mathbf{q}=\mathcal{Q}(\mathbf{z}) $$ $$ \hat{\mathbf{X}}^{m o d}=\mathcal{D}(\mathbf{q}) $$ 总体训练损失为： $$ \begin{array}{r}{\mathcal{L}_{t o k}=\|\mathbf{X}^{m o d}-\hat{\mathbf{X}}^{m o d}\|_{2}^{2}+\|\operatorname{s g}[\mathbf{z}]-\mathbf{q}\|_{2}^{2}+\beta\|\mathbf{z}-\operatorname{s g}[\mathbf{q}]\|_{2}^{2},}\end{array} $$ 其中 $sg[\cdot]$ 是停止梯度算子，$\beta$ 用于平衡承诺损失。由于硬件限制，某些模态（如视线数据）在追踪丢失时可能包含无效数据。我们会屏蔽这些无效数据，并改用掩码L2损失。详情请参阅补充材料第A.1节和C节。 #### 3.3. 多模态掩码预训练 4M [^8] [^76] 提出了针对静态图像模态的大规模多模态掩码建模训练方案：利用另一批采样的多模态输入标记来预测一小批采样的多模态目标标记。然而，当将掩码建模应用于第一人称视角视频时，4M面临几个挑战：1）缺乏确保时间一致性的机制；2）每个视频样本中的标记数量远多于图像模态，阻碍了高效训练和可扩展性；3）不同模态间的标记比例严重失衡，例如每个样本中视频标记的数量是视线标记的170倍，这可能导致较少被表示的模态的关键信息被忽略；4）第一人称视角数据集可能存在标注缺失问题，而4M会对所有模态进行伪标注。 多模态标记与数据集平衡。在第3.2节中，我们利用基于Transformer的VQ-VAE和Cosmos Tokenizer [^3] 将每种模态转化为具有时间感知能力的离散标记。大量视频标记给多模态标记预训练带来了挑战。虽然在标记化过程中激进地压缩视频标记可能有益，但这会对视频质量产生负面影响。为解决这一问题，我们将视频下采样至8帧/秒，使每个视频的标记数量降至原来的1/3。 --- 标记化完成后，训练集包含约40亿个多模态标记，而视线标记仅有1300万个。此外，不同数据集的规模也严重失衡，例如EgoExo4D [^30] 的样本量是H2O [^49] 的160倍。直接使用这些不平衡的数据集训练会导致模型忽略标记数量较少的模态。此外，有些数据集可能出现过拟合，而另一些则可能出现欠拟合。为缓解这一问题，我们针对数据集采样以及数据集内跨模态的标记采样尝试了不同的采样权重。我们发现，最初按照数据集大小的比例采样数据集，再按照4M的方法用均匀浓度参数从各模态中采样标记，能够实现最稳定的训练和最佳性能。详情请参阅补充材料第A.2节。 时序多模态词元嵌入。如图2下部所示，针对每种模态，我们使用特定于该模态的嵌入层将输入词元映射到一个高维统一空间，从而促进多模态信息的对齐与整合。随后我们添加正弦余弦位置嵌入：注视和摄像头词元采用一维嵌入，视频词元则采用三维嵌入。目标词元也采用相同处理方式。与4M [^8] [^76]类似，我们引入了可学习的模态类别嵌入，该嵌入在输入词元嵌入模块和目标词元嵌入模块中共享使用。 掩码处理。先前的研究已证实掩码建模的有效性 [^7] [^8] [^33] [^76] [^95]。4M需要对齐的多模态标注数据，并依赖伪标签技术；但对于以自我为中心的数据集而言，为所有模态生成伪标签并不现实。我们将缺失的模态表示为占位符并将其掩码掉。关于缺失模态的处理细节，请参阅补充材料第A.4节。与4M类似，我们通过应用输入和目标掩码来降低计算成本，仅对固定数量的可见词元进行编码和解码。虽然4M将该数值上限设为256，但考虑到视频的词元数量超过5000，我们将这一数值提高到2048以容纳更多信息。对于每个多模态数据样本 $\mathbf{X}_i$，我们会先确定每种可用模态作为输入和目标的可见词元数量 $mod \in \{\text{rgb}, \text{depth}, \text{gaze}, \text{cam}\}$，随后在这些限制范围内为每个视频片段中的词元采样 $\mathbf{X}_i^{\text{mod}}$（详见补充材料第A.2节）。可见的输入词元与目标词元互不重叠。 模型架构。除了用于输入词元和目标词元的特定模态嵌入层外，主架构遵循T5-Base [^83]。编码器对所有采样的可见输入词元应用自注意力机制，从而整合来自多种模态的时空信息。解码器输入由掩码处理后的采样可见目标词元构成。解码器以编码器输出作为上下文，应用交叉注意力机制，同时执行受限在同一模态内的掩码自注意力操作，以此预测那些被掩码的目标词元——这一设计确保了解码器生成的同一模态内词元具有连贯性。训练损失采用交叉熵。对于每种模态m： $$ \begin{aligned}\mathcal{L}_{m}&=-\frac{1}{T_{m}}\sum_{t=1}^{T_{m}}\sum_{c=1}^{C_{m}}y_{t,c}^{(m)}\log\left(\hat{y}_{t,c}^{(m)}\right)\\&\quad\mathcal{L}_{total}=\frac{1}{|\mathcal{M}|}\sum_{m\in\mathcal{M}}\mathcal{L}_{m}\end{aligned} $$ 其中 $\mathcal{M}$ 表示有可用词元的模态集合，$T_m$ 是每种模态的词元数量，$C_m$ 是码本大小，$\hat{y}_{t,c}$ 是每个词元的预测概率分布。详情请参阅补充材料第A.3节。 #### 3.4. 推理 训练期间，我们采用可变掩码率随机掩码掉包含时序信息的多模态词元。先前的研究 [^76] 表明，采用此种方式训练的掩码图像模型可作为顺序无关的自回归模型 [^38]，能够按随机顺序迭代解码词元以实现并行推理。同样，我们的实验证明EgoM2P能够同时预测被掩码词元的分布，这有望满足实时应用对处理速度的要求。如表2所示，我们的方法可在0.18秒内预测出长度为60帧的视频的摄像头轨迹（即每秒处理300帧以上）。 并行推理过程可以被表述为一个多步解码流程。我们采用线性调度方法，在s个解码步骤中预测n个目标词元。在每一步中，我们会随机选取$n/s$个目标词元，首先利用所有可见的输入词元让预训练模型进行一次前向传播，从而并行预测所选目标词元的条件分布$\hat{y}_{\mathrm{cond}}$。接着，我们将所有输入词元掩码化，再进行第二次前向传播，通过针对掩码后的输入词元的交叉注意力机制来预测所选目标词元的无条件概率$\hat{y}_{\mathrm{uncond}}$。最终预测得到的目标词元分布$\hat{y}$是通过无分类器引导技术[^34]来估算的，其权重为$\omega$： $$ \hat{y}=(1+\omega)\hat{y}_{\mathrm{c o n d}}-\omega\hat{y}_{\mathrm{u n c o n d}} $$ 最终的目标词元预测是从预测分布$\hat{y}$中通过核采样方法[^36]得到的。我们发现，对于词元数量较少的模态而言，单个解码步骤就足够了；而对于视频模态，增加解码步骤并在每一步预测一部分词元会更有益处。伪代码详见补充材料第A.5节。 #### 3.5. 实现细节 对于每个多模态剪辑，我们将其长度设定为2秒。非视频模态包含$T = 60$帧，而由于帧率较低，视频模态则包含$T = 16$帧。视频分辨率为$256 \times 256$。分词处理后，每个样本的RGB视频和深度视频分别包含5120个词元，视线与相机轨迹每个样本则包含30个词元。更多细节详见补充材料第A节。 ---
MethodEgoExo4D [^30]ADT [^80] (unseen)Time $\downarrow$
ATE $\downarrow$RTE $\downarrow$RRE $\downarrow$ATE $\downarrow$RTE $\downarrow$RRE $\downarrow$
DROID-SLAM [^94]0.0180.0050.5060.0340.0100.3162.7s
ACE-Zero [^13]0.0280.0070.6720.0490.0110.333426s
Align3R [^66]0.0190.0060.7620.0280.0100.276372s
EgoM2P0.0170.0040.4290.0320.0060.4900.18s
0.0260.0050.480
Table 2. Evaluation on camera tracking. Compared to specialist SOTAs that require geometry test-time optimization, EgoM2P's feed-forward tracking results achieve comparable performance yet with significantly higher efficiency. We report the average runtime per sequence. $\underline{\text{Underlined}}$ denotes post-training results (Sec. 4.5).
### 4. 实验 我们在下游任务中，包括以自我为中心的感知与合成任务上，将EgoM2P的多任务能力与当前最先进的模型进行了基准测试。我们还在未经任何微调的情况下，在未见过的数据集上对其进行了测试，以展示其预训练特征的强大泛化能力。此外，我们还证明EgoM2P可以通过训练后微调轻松适配新任务。 #### 4.1. 以自我为中心的相机跟踪 评估协议。我们从EgoExo4D数据集的验证集中抽取200个视频剪辑[^30]用于评估。为了检验EgoM2P对未见数据集的泛化能力，我们还在Aria Digital Twin（ADT）数据集的200个视频剪辑上进行了评估[^80]，该数据集完全未包含在我们的训练集中。我们将所有输入帧处理为$256 \times 256$分辨率且帧率为8 FPS。我们采用标准的误差指标：绝对平移误差（ATE）、相对平移误差（RTE）以及相对旋转误差（RRE）。 基线模型。我们将其与专门的相机跟踪方法进行比较，包括DROID-SLAM [^94]、ACE-Zero [^13]以及Align3R [^66]。需要注意的是，所有基线方法都利用了显式的几何约束，并在测试时优化阶段进行束调整。ACE-Zero和Align3R [^66]还依赖现成的单目深度估计与光流预测结果作为额外输入。相比之下，EgoM2P在训练时并未引入任何几何建模或3D归纳偏置，仅需一次前馈传播就能直接从RGB输入预测相机位姿。 结果。相关指标详见表2。与那些需要显式几何建模的专业SOTA模型相比，我们无需任何3D归纳偏置即可训练出的通用模型，其性能可与前者相媲美。值得注意的是，所有基线模型都需要耗时的测试时优化，而得益于我们的并行解码方法，EgoM2P的推理速度可达300帧/秒以上。由于运动速度快且相机视差小，以自我为中心的相机跟踪有时颇具挑战性。如图3所示，虽然某些情况下基线模型会出现时间抖动和误差累积问题，但EgoM2P却能预测出平滑且合理的相机轨迹，因为它学会了捕捉以自我为中心运动的独特性。 ![Image](images/624b44b501d8156cb8a6bf104f8101a611c1e38f235f6aa8f79bae995b74ab91.jpg)
(b) Comparison of camera tracking on ADT [^80]. Ground truth and predictions are represented by black and colored wireframes, respectively.
Figure 3. Egocentric capture often involves rapid head rotations, which challenges baseline tracking methods. However, EgoM2P effectively predicts smooth and plausible camera poses in the shown examples. This capability also generalizes to the unseen ADT [^80] dataset without post-training.
这种能力甚至能泛化到域外的ADT测试集上。 #### 4.2. 以自我为中心的注视动态估计 评估协议。我们从EgoExo4D数据集的验证集中抽取了1,000个视频[^30]。我们将所有方法的真实标签和预测标签都归一化到[^0] [^1]范围内，并使用均方误差（MSE）来评估注视估计的准确性。 基线模型。我们将EgoM2P与两种能从以自我为中心的视频中预测2D注视位置的SOTA方法进行比较：Huang等人[^40]的方法与Lai等人[^50]的方法，评估时采用它们的官方实现。 结果。基线模型使用30帧/秒的输入视频；按照各基线模型的设定，输入视频帧分别被缩放为$224 \times 224$（针对[^40]）和$256 \times 256$（针对[^50]）。我们的方法取得了最低的MSE值（$0.0162$），表现优于Huang等人[^40]的方法（0.0255）以及Lai等人[^50]的方法（0.0175）。关于定性比较结果，可参见图4和补充视频。EgoM2P能生成更一致的注视预测结果，这凸显了其理解人类意图的多任务处理能力。 --- ![Image](images/76f0004ddb8442755bc0a22e460c8b4b911cae3f6c692853a3b5c1c312edd8ac.jpg)
Figure 4. Gaze dynamics estimation. EgoM2P can predict results that are more aligned with human intentions.
MethodH2O [^49]HOI4D [^60] (unseen)Time $\downarrow$
Abs Rel $\downarrow$$\delta_{{1.25}}\uparrow$Abs Rel $\downarrow$$\delta_{{1.25}}\uparrow$
RollingDepth [^44]0.08790.50.05797.637s
Align3R [^66]0.07491.80.04598.190s
EgoM2P0.05596.00.06198.00.8s
0.04199.0
Table 3. Evaluation on egocentric video depth estimation. Compared to specialist SOTAs requiring geometry-based test-time optimization, the versatile EgoM2P achieves comparable performance while being significantly more efficient. With post-training described in Sec. 4.5, EgoM2P excels (see $\underline{\text{underlined results}}$).
#### 4.3. 以自我为中心的单目视频深度估计 评估协议。我们在H2O的测试集上进行评估[^49]，该集包含236个时长两秒的视频片段。为了验证我们方法的跨域泛化能力，我们还在HOI4D的100个时长两秒的视频片段上进行了评估[^60]，这些视频在EgoM2P训练期间完全未见。我们将每个视频片段的时间分辨率降至8帧/秒，并将其缩放为$256 \times 256$作为输入。我们通过序列级的缩放和平移因子将估计的相对深度与真实深度对齐，随后用绝对相对误差（Abs Rel）以及预测深度在真实深度的1.25倍范围内的百分比（$\delta_{1.25}$）来评估深度估计的准确性。 基线模型。我们将EgoM2P与两种专门的视频深度估计器进行比较：RollingDepth[^44]和Align3R[^66]。这两种方法都利用预训练网络[^43] [^104]来估计单目或成对深度图，随后通过分层的序列级优化来实现逐帧深度的时间对齐。 实验结果。我们在表3中报告了实验结果，并在图5中提供了可视化对比。RollingDepth和Align3R在逐帧预测后需要进行序列级优化，处理一段两秒长的序列耗时可达一分钟。相比之下，EgoM2P只需单次端到端前馈计算就能预测整段深度视频，推理速度至少快30倍。从定量结果来看，EgoM2P在两个测试集上的表现与基线方法相当，甚至优于最优的$\delta_{1.25}$；尤其是在未见过的HOI4D数据集上，这验证了EgoM2P对域外中心视觉数据的泛化能力。若能降低Cosmos分词器的量化误差[^3]，我们的深度预测结果还会进一步提升。更多可视化内容请参见我们的补充视频。 ![Image](images/2ae7fa79d4aebef1456a9c906ca018be52eaa03334773d35bbe8a36be412b401.jpg)
Figure 5. Egocentric video depth estimation. EgoM2P achieves comparable performance with specialist SOTA methods. Rolling Depth [^44] struggles in estimating the depth of hands, an important component in egocentric view, while our method can capture hand movement even in the out-of-domain HOI4D [^60] dataset, without any post-training or fine-tuning.
#### 4.4. 条件式中心视觉视频合成 EgoM2P能够从其他模态合成RGB视频。本节重点探讨深度到RGB视频的合成。 评估协议。我们从HoloAssist [^106]测试集中随机选取了142段深度视频，从ASE [^6]数据集中选取了100段深度视频。需要注意的是，整个ASE数据集并未出现在模型的训练集中，因此可用于评估我们方法的泛化能力。对于ASE数据集，我们将其标注的鱼眼深度视频作为输入；对于HoloAssist数据集，由于其深度标签与RGB帧未对齐，我们采用RollingDepth [^44]为每帧RGB图像生成伪深度标签作为输入。在定量指标方面，我们采用了Fréchet视频距离（FVD）[^96]、结构相似性指数（SSIM）、峰值信噪比（PSNR）以及感知指标LPIPS [^124]。 基线方法。我们将我们的方法与两种基于SOTA ControlNet的方法进行对比：Control-A-Video [^19]和ControlVideo [^126]，二者均支持将深度作为条件输入。 实验结果。由于这两种基线方法需要额外的文本输入，我们针对每个数据集采用了不同的模板。在HoloAssist数据集中，我们使用模板“两只手拿着{object}”，其中{object}代表RGB视频中被拿取的物品。 ---
MethodHoloAssist [^106]ASE [^6] (unseen)
$FVD^{{*}} \downarrow$SSIM $\uparrow$PSNR $\uparrow$LPIPS $\downarrow$$FVD^{{*}} \downarrow$SSIM $\uparrow$PSNR $\uparrow$LPIPS $\downarrow$
Control-A-Video [^19]2.3090.1859.250.6772.2260.28911.110.817
ControlVideo [^126]1.3630.2358.180.6531.3920.27510.460.676
EgoM2P0.7590.59215.1630.3361.3360.3086.9230.715
0.5250.59416.9240.520
Table 4. Evaluation on depth-to-RGB video synthesis. EgoM2P outperforms baselines on the HoloAssist test set, producing higher-quality egocentric videos. On the unseen ASE dataset, it generates videos that more closely resemble real ones with a lower FVD* (FVD/10³). With post-training (Sec. 4.5), EgoM2P excels on unseen datasets indicated by underlined results.
在ASE数据集中，由于视频场景为室内场景，我们使用提示词“室内场景”。在图6中，我们展示了定性对比结果，凸显了两种基线方法在生成与输入深度图准确对应的RGB帧时面临的挑战。这些基线方法生成的输出在语义和几何对齐方面往往存在明显差异，尤其是在ASE [^6]数据集上。相比之下，我们的方法EgoM2P表现更优，能够保持深度图与生成的RGB帧之间的对齐。定量结果见表4。在HoloAssist数据集 [^106]上，我们的模型性能优于基线方法；在未见过的、风格与训练数据差异较大的ASE数据集 [^6]上，我们的方法展现出强大的泛化能力，生成的中心视觉视频更接近真实视频，这从较低的FVD分数中可看出。相比之下，以强大的Stable Diffusion [^84]为初始模型的基线模型则容易出现幻觉现象。虽然它们的PSNR分数更高，但输出结果往往偏离真实的中心视觉视频分布，这从它们较高的FVD分数中可看出。 #### 4.5. 训练后处理 预训练的EgoM2P在跨数据集泛化测试中表现出强大的泛化能力。此外，在表2、表3和表4中，我们发现通过在未见过的数据集的训练集上进行后续训练，可以进一步增强EgoM2P的性能，从而能够以极少的数据快速适应新领域。更多细节请参阅补充材料第D节。 ### 5. 结论 我们提出了EgoM2P，这是首个整合了中心视觉中四种常见模态的多模态多任务大型中心视角模型。为了处理多模态中的复杂时空动态，我们提出了一种统一的时间分词器架构，将视线与摄像头轨迹转化为携带时间信息的离散词元。为解决中心视觉数据集的异质性问题，我们将多模态掩码建模扩展到视频领域，并利用从40亿个多模态词元数据库中采样的4000亿个词元对EgoM2P进行预训练。EgoM2P的性能与 ![Image](images/d4f40bc5631a2fe0dbc10fa084c8db374c52ab8619515d5ed549399ddcece29c.jpg)
Figure 6. Comparison of depth-to-RGB video synthesis. Red boxes highlight incorrectly generated fingers in baselines, while ours generate meaningful hand motion. Our results show improved alignment with the input depth, minimizing hallucinations. No post-training was applied for ASE results in this figure.
或超越现有的最先进的专用模型，且在多种下游应用中展现出高效性，包括中心视角摄像头跟踪、中心视角视频中的视线估计、中心视角单目深度估计以及条件化中心视角视频合成。 局限性：合成视频的视觉质量本质上受到当前最先进视频分词器的限制[^3]。尽管更好的分词器可以减少量化过程中的视频质量损失，但这并非我们的主要研究方向。利用基于离散视频词元的条件化扩散解码器来提升视觉质量可能会是一种有效方案。未来工作：可穿戴设备的计算资源有限，且为了实现无缝的人机交互需要实时处理能力。我们计划探索在嵌入式GPU上对EgoM2P进行性能优化。在本研究中，我们考虑了中心视觉中最常见的模态。将手部动作、音频、文本等融入模型是未来研究的一个有前景的方向。 --- 致谢：李根获得了微软苏黎世空间人工智能实验室的博士奖学金支持，陈雨桐获得了瑞士创新署Innosuisse的支持，赵凯峰获得了SDSC博士奖学金的支持。本研究还得到了瑞士国家超级计算中心（CSCS）在Alps项目ID为a03的项目下提供的瑞士人工智能倡议基金资助。我们衷心感谢张思伟的有益讨论。 ## 参考文献 [^1]: Gpt-4v(ision) system card. 2023. 2 [^2]: Marah I Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. Technical Report MSR-TR-2024-12, Microsoft, 2024. 2 [^3]: Niket Agarwal, Arslan Ali, Maciej Bala, Yogesh Balaji, Erik Barker, Tiffany Cai, Prithvijit Chattopadhyay, Yongxin Chen, Yin Cui, Yifan Ding, et al. Cosmos world foundation model platform for physical ai. arXiv preprint arXiv:2501.03575, 2025. 2, 3, 4, 7, 8, 1 [^4]: Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716–23736, 2022. 2 [^5]: Armen Avetisyan, Christopher Xie, Henry Howard-Jenkins, Tsun-Yi Yang, Samir Aroudj, Suvam Patra, Fuyang Zhang, Duncan Frost, Luke Holland, Campbell Orme, Jakob Engel, Edward Miller, Richard Newcombe, and Vasileios Balntas. Scenescript: Reconstructing scenes with an autoregressive structured language model. In European Conference on Computer Vision (ECCV), 2024. 1 [^6]: Armen Avetisyan, Christopher Xie, Henry Howard-Jenkins, Tsun-Yi Yang, Samir Aroudj, Suvam Patra, Fuyang Zhang, Duncan Frost, Luke Holland, Campbell Orme, Jakob Engel, Edward Miller, Richard A. Newcombe, and Vasileios Balntas. Scenescript: Reconstructing scenes with an autoregressive structured language model. In ECCV, 2024. 7, 8, 4 [^7]: Roman Bachmann, David Mizrahi, Andrei Atanov, and Amir Zamir. MultiMAE: Multi-modal multi-task masked autoencoders. In European Conference on Computer Vision, 2022. 5 [^8]: Roman Bachmann, Oğuzhan Fatih Kar, David Mizrahi, Ali Garjani, Mingfei Gao, David Griffiths, Jiaming Hu, Afshin Dehghan, and Amir Zamir. 4M-21: An any-to-any vision model for tens of tasks and modalities. In Thirty-eighth Conference on Neural Information Processing Systems, 2024. 2, 4, 5, 3 [^9]: Sven Bambach, Stefan Lee, David J Crandall, and Chen Yu. Lending a hand: Detecting hands and recognizing activities in complex egocentric interactions. In Proceedings of the IEEE international conference on computer vision, pages 1949–1957, 2015. 3 [^10]: Prithviraj Banerjee, Sindi Shkodrani, Pierre Moulon, Shreyas Hampali, Fan Zhang, Jade Fountain, Edward Miller, Selen Basol, Richard Newcombe, Robert Wang, et al. Introducing hot3d: An egocentric dataset for 3d hand and object tracking. arXiv preprint arXiv:2406.09598, 2024. 1, 2, 3 [^11]: Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, Varun Jampani, and Robin Rombach. Stable video diffusion: Scaling latent video diffusion models to large datasets, 2023. 1, 2 [^12]: Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023. 1 [^13]: Eric Brachmann, Jamie Wynn, Shuai Chen, Tommaso Cavallari, Áron Monszpart, Daniyar Turmukhambetov, and Victor Adrian Prisacariu. Scene coordinate reconstruction: Posing of image collections via incremental learning of a relocalizer. In ECCV, 2024. 6 [^14]: Jake Bruce, Michael D Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, et al. Genie: Generative interactive environments. In Forty-first International Conference on Machine Learning, 2024. 2 [^15]: Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pages 213–229. Springer, 2020. 2 [^16]: Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9650–9660, 2021. 2 [^17]: Haoxin Chen, Yong Zhang, Xiaodong Cun, Menghan Xia, Xintao Wang, Chao Weng, and Ying Shan. Videocrafter2: Overcoming data limitations for high-quality video diffusion models, 2024. 1 [^18]: Ting Chen, Saurabh Saxena, Lala Li, Tsung-Yi Lin, David J. Fleet, and Geoffrey Hinton. A unified sequence interface for vision tasks. In Advances in Neural Information Processing Systems, 2022. 2 [^19]: Weifeng Chen, Yatai Ji, Jie Wu, Hefeng Wu, Pan Xie, Jiashi Li, Xin Xia, Xuefeng Xiao, and Liang Lin. Control-a-video: Controllable text-to-video diffusion models with motion prior and reward feedback learning, 2024. 7, 8 [^20]: Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. Scaling egocentric vision: The epic-kitchens dataset. In European Conference on Computer Vision (ECCV), 2018. 1, 2 ## [^21]: Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold --- Sylvain Gelly等人。一张图像相当于16×16个词：用于大规模图像识别的Transformer。arXiv预印本arXiv:2010.11929，2020年。2 [^22]: Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Germanidis. Structure and Content-Guided Video Synthesis with Diffusion Models. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 7312–7322, Los Alamitos, CA, USA, 2023. IEEE Computer Society. 1 [^23]: Zicong Fan, Omid Taheri, Dimitrios Tzionas, Muhammed Kocabas, Manuel Kaufmann, Michael J. Black, and Otmar Hilliges. ARCTIC: A dataset for dexterous bimanual hand-object manipulation. In Proceedings IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023. 1, 3 [^25]: Tsu-Jui Fu, Linjie Li, Zhe Gan, Kevin Lin, William Yang Wang, Lijuan Wang, and Zicheng Liu. Violet: End-to-end video-language transformers with masked visual-token modeling. arXiv preprint arXiv:2111.12681, 2021. 2 [^24]: Zicong Fan, Maria Parelli, Maria Eleni Kadoglou, Muhammed Kocabas, Xu Chen, Michael J Black, and Otmar Hilliges. HOLD: Category-agnostic 3d reconstruction of interacting hands and objects from video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 494–504, 2024. 3 [^26]: Guillermo Garcia-Hernando, Shanxin Yuan, Seungryul Baek, and Tae-Kyun Kim. First-person hand action benchmark with rgb-d videos and 3d hand pose annotations. In Proceedings of Computer Vision and Pattern Recognition (CVPR), 2018. 1 [^27]: Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15180–15190, 2023. 2 [^28]: Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in 1 hour, 2018. 1 [^29]: Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, et al. Ego4d: Around the World in 3,000 Hours of Egocentric Video. In IEEE/CVF Computer Vision and Pattern Recognition (CVPR), 2022. 1,2 [^30]: Kristen Grauman, Andrew Westbury, Lorenzo Torresani, Kris Kitani, Jitendra Malik, Triantafyllos Afouras, Kumar Ashutosh, Vijay Baiyya, et al. Ego-exo4d: Understanding skilled human activity from first- and third-person perspectives. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 19383–19400, 2023. 1, 2, 3, 5, 6 [^31]: Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. In The Twelfth International Conference on Learning Representations, 2024. 1 [^32]: David Ha and Jürgen Schmidhuber. World models. arXiv preprint arXiv:1803.10122, 2018. 2 [^33]: Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll'ar, and Ross B. Girshick. Masked autoencoders are scalable vision learners. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15979–15988, 2021. 5 [^34]: Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021. 5 [^35]: Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. In Advances in Neural Information Processing Systems, pages 8633–8646. Curran Associates, Inc., 2022. [^36]: Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In International Conference on Learning Representations, 2020. 5 [^37]: Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pretraining for text-to-video generation via transformers. In The Eleventh International Conference on Learning Representations, 2023. 1 [^38]: Emiel Hoogeboom, Alexey A. Gritsenko, Jasmijn Bastings, Ben Poole, Rianne van den Berg, and Tim Salimans. Autoregressive diffusion models. In International Conference on Learning Representations, 2022. 5 [^39]: Ziniu Hu, Ahmet Iscen, Chen Sun, Zirui Wang, Kai-Wei Chang, Yizhou Sun, Cordelia Schmid, David A. Ross, and Alireza Fathi. Reveal: Retrieval-augmented visual-language pre-training with multi-source multimodal knowledge memory. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 23369–23379, 2022. 1 [^40]: Yifei Huang, Minjie Cai, Zhenqiang Li, and Yoichi Sato. Predicting gaze in egocentric video by learning task-dependent attention transition. In Computer Vision - ECCV 2018 - 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part IV, pages 789–804. Springer, 2018. 6 [^41]: Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024. 2 [^42]: Evangelos Kazakos, Arsha Nagrani, Andrew Zisserman, and Dima Damen. Epic-fusion: Audio-visual temporal binding for egocentric action recognition. In Proceedings of the IEEE/CVF international conference on computer vision, pages 5492–5501, 2019. 3 [^43]: Bingxin Ke, Anton Obukhov, Shengyu Huang, Nando Metzger, Rodrigo Caye Daudt, and Konrad Schindler. Repurposing diffusion-based image generators for monocular depth estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 7 ## [^44]: Bingxin Ke, Dominik Narnhofer, Shengyu Huang, Lei Ke, Torben Peters, Katerina Fragkiadaki, Anton Obukhov, and --- Konrad Schindler。无需视频模型的视频深度估计。收录于CVPR 2025会议论文集。3, 7 [^45]: Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 15908–15918, Los Alamitos, CA, USA, 2023. IEEE Computer Society. 1 [^46]: Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4015–4026, 2023. 2 [^47]: Alexander Kolesnikov, André Susano Pinto, Lucas Beyer, Xiaohua Zhai, Jeremiah J. Harmsen, and Neil Houlsby. UVim: A unified modeling approach for vision with learned guiding codes. In Advances in Neural Information Processing Systems, 2022. 2 [^48]: Dan Kondratyuk, Lijun Yu, Xiuye Gu, Jose Lezama, Jonathan Huang, Grant Schindler, Rachel Hornung, et al. VideoPoet: A large language model for zero-shot video generation. In Proceedings of the 41st International Conference on Machine Learning, pages 25105–25124. PMLR, 2024. [^49]: Taein Kwon, Bugra Tekin, Jan Stühmer, Federica Bogo, and Marc Pollefeys. H2o: Two hands manipulating objects for first person interaction recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 10138–10148, 2021. 1, 3, 5, 7, 4 [^50]: Bolin Lai, Miao Liu, Fiona Ryan, and James M. Rehg. In the eye of transformer: Global-local correlation for egocentric gaze estimation. In 33rd British Machine Vision Conference 2022, BMVC 2022, London, UK, November 21-24, 2022, page 227. BMVA Press, 2022. 6 [^51]: Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9579–9589, 2024. 1 [^52]: Gen Li, Kaifeng Zhao, Siwei Zhang, Xiaozhong Lyu, Mihai Dusmanu, Yan Zhang, Marc Pollefeys, and Siyu Tang. Egogen: An egocentric synthetic data generator. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14497–14509, 2024. 1, 3, 4 [^53]: Kunchang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305.06355, 2023. 2 [^54]: Zhengqi Li, Richard Tucker, Forrester Cole, Qianqian Wang, Linyi Jin, Vickie Ye, Angjoo Kanazawa, Aleksander Holynski, and Noah Snavely. Megasam: Accurate, fast and robust structure and motion from casual dynamic videos. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 10486–10496, 2025. 4 [^55]: Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-LLaVA: Learning united visual 通过对齐后再投影的方式进行表示。收录于2024年自然语言处理实证方法会议论文集，第5971–5984页，美国佛罗里达州迈阿密。计算机协会。2 [^56]: Gaowen Liu, Hao Tang, Hugo M Latapie, Jason J Corso, and Yan Yan. Cross-view exocentric to egocentric video synthesis. In Proceedings of the 29th ACM International Conference on Multimedia, pages 974–982, 2021. 3 [^57]: Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, 2024. 2 [^58]: Jia-Wei Liu, Weijia Mao, Zhongcong Xu, Jussi Keppo, and Mike Zheng Shou. Exocentric-to-egocentric video generation. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. 3 [^59]: Ruyang Liu, Chen Li, Yixiao Ge, Thomas H. Li, Ying Shan, and Ge Li. BT-Adapter: Video Conversation is Feasible Without Video Instruction Tuning. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13658–13667, Los Alamitos, CA, USA, 2024. IEEE Computer Society. 1 [^60]: Yunze Liu, Yun Liu, Che Jiang, Kangbo Lyu, Weikang Wan, Hao Shen, Boqiang Liang, Zhoujie Fu, He Wang, and Li Yi. Hoi4d: A 4d egocentric dataset for category-level human-object interaction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 21013–21022, 2022. 1, 7, 4 [^61]: Y. Liu, H. Yang, X. Si, L. Liu, Z. Li, Y. Zhang, Y. Liu, and L. Yi. Taco: Benchmarking generalizable bimanual tool-action-object understanding. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 21740–21751, Los Alamitos, CA, USA, 2024. IEEE Computer Society. 1, 3 [^62]: Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11976–11986, 2022. 2 [^63]: Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. 1, 2 [^64]: Jiasen Lu, Christopher Clark, Sangho Lee, Zichen Zhang, Savya Khosla, Ryan Marten, Derek Hoiem, and Aniruddha Kembhavi. Unified-io 2: Scaling autoregressive multimodal models with vision, language, audio, and action. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 26429–26445, 2023. 2 [^65]: Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. UNIFIED-IO: A unified model for vision, language, and multi-modal tasks. In The Eleventh International Conference on Learning Representations, 2023. 2 ## [^66]: Jiahao Lu, Tianyu Huang, Peng Li, Zhiyang Dou, Cheng Lin, Zhiming Cui, Zhen Dong, Sai-Kit Yeung, Wenping Wang, and Yuan Liu. Align3r: Aligned monocular depth estimation for dynamic videos. In CVPR, 2025. 6,7 --- [^67]: LumaLabs. Dream Machine. , 2024. 1 [^68]: Zhengxiong Luo, Dayou Chen, Yingya Zhang, Yan Huang, Liangsheng Wang, Yujun Shen, Deli Zhao, Jinren Zhou, and Tien-Ping Tan. Videofusion: Decomposed diffusion models for high-quality video generation. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10209–10218, 2023. 1 [^69]: Zhaoyang Lv, Nicholas Charron, Pierre Moulon, Alexander Gamino, Cheng Peng, Chris Sweeney, Edward Miller, Huixuan Tang, Jeff Meissner, Jing Dong, et al. Aria everyday activities dataset. arXiv preprint arXiv:2402.13349, 2024. 1 [^70]: Lingni Ma, Yuting Ye, Fangzhou Hong, Vladimir Guzov, Yifeng Jiang, Rowan Postyeni, Luis Pesqueira, Alexander Gamino, Vijay Baiyya, Hyo Jin Kim, et al. Nymeria: A massive collection of multimodal egocentric daily motion in the wild. arXiv preprint arXiv:2406.09905, 2024. 1 [^71]: Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024), 2024. 1, 2 [^72]: Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier Biard, Sam Dodge, Philipp Dufter, Bowen Zhang, Dhruti Shah, Xianzhi Du, Futang Peng, Haotian Zhang, et al. Mml: Methods, analysis & insights from multimodal llm pre-training, 2024. 2 [^73]: Meta. Project Aria Glasses. , 2023.3 [^74]: Vincent Micheli, Eloi Alonso, and François Fleuret. Transformers are sample-efficient world models. arXiv preprint arXiv:2209.00588, 2022. 2 [^75]: Microsoft. HoloLens 2. , 2019. 3.1 [^76]: David Mizrahi, Roman Bachmann, Oğuzhan Fatih Kar, Teresa Yeo, Mingfei Gao, Afshin Dehghan, and Amir Zamir. 4M: Massively multimodal masked modeling. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. 2, 4, 5, 3 [^77]: Takehiko Ohkawa, Kun He, Fadime Sener, Tomas Hodan, Luan Tran, and Cem Keskin. AssemblyHands: towards egocentric activity understanding via 3d hand pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12999–13008, 2023. 1 [^78]: OpenAI. Video generation models as world simulators. , 2024.1,2 [^79]: Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V. Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaaeldin El Nouby, Mido Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu、Herve Jegou、Julien Mairal、Patrick Labatut、Armand Joulin和Piotr Bojanowski。DINOv2：无监督学习鲁棒的视觉特征。《机器学习研究汇刊》，2024年。2 [^80]: Xiaqing Pan, Nicholas Charron, Yongqian Yang, Scott Peters, Thomas Whelan, Chen Kong, Omkar Parkhi, Richard Newcombe, and Carl Yuheng Ren. Aria digital twin: A new benchmark dataset for egocentric 3d machine perception. In ICCV, 2023. 6, 3, 4 [^81]: Adam Polyak, Amit Zohar, Andrew Brown, Andros Tjandra, Animesh Sinha, Ann Lee, Apoorv Vyas, Bowen Shi, Chih-Yao Ma, Ching-Yao Chuang, et al. Movie gen: A cast of media foundation models. arXiv preprint arXiv:2410.13720, 2024. 1, 2 [^82]: Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PmLR, 2021. 2 [^83]: Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21(1), 2020. 4, 5 [^84]: Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjorn Ommer. High-Resolution Image Synthesis with Latent Diffusion Models. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10674–10685, Los Alamitos, CA, USA, 2022. IEEE Computer Society. 8 [^85]: RunwayML. Gen-3 Alpha. , 2024. 1 [^86]: Paul-Edouard Sarlin, Mihai Dusmanu, Johannes L Schönberger, Pablo Speciale, Lukas Gruber, Viktor Larsson, Ondrej Miksik, and Marc Pollefeys. Lamar: Benchmarking localization and mapping for augmented reality. In European Conference on Computer Vision, pages 686–704. Springer, 2022. 3 [^87]: F. Sener, D. Chatterjee, D. Shelepov, K. He, D. Singhania, R. Wang, and A. Yao. Assembly101: A large-scale multi-view video dataset for understanding procedural activities. CVPR 2022. 1 [^88]: Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, Devi Parikh, Sonal Gupta, and Yaniv Taigman. Make-a-video: Text-to-video generation without text-video data. In The Eleventh International Conference on Learning Representations, 2023. 1 ## [^89]: Julian Straub, Thomas Whelan, Lingni Ma, Yufan Chen, Erik Wijmans, Simon Green, Jakob J. Engel, Raul Mur-Artal, Carl Ren, Shobhit Verma, Anton Clarkson, Mingfei Yan, Brian Budge, Yajie Yan, Xiaqing Pan, June Yon, Yuyang Zou, Kimberly Leon, Nigel Carter, Jesus Briales, Tyler Gillingham, Elias Mueggler, Luis Pesqueira, Manolis Savva, Dhruv Batra, Hauke M. Strasdat, Renzo De Nardi, Michael Goesele, Steven Lovegrove, and Richard New- --- Combe。Replica数据集：室内空间的数字副本。arXiv预印本arXiv:1906.05797，2019年。3 [^90]: Alessandro Suglia, Claudio Greco, Katie Baker, Jose L. Part, Ioannis Papaioannou, Arash Eshghi, Ioannis Konstas, and Oliver Lemon. AlanaVLM: A multimodal embodied AI foundation model for egocentric video understanding. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 11101–11122, Miami, Florida, USA, 2024. Association for Computational Linguistics. 2 [^91]: Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Emu: Generative pretraining in multimodality. In The Twelfth International Conference on Learning Representations, 2024. 2 [^92]: Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023. 2 [^93]: Kuaishou Technology. Kling ai video generator. , 2024. Accessed: 2025-03-03. 2 [^94]: Zachary Teed and Jia Deng. DROID-SLAM: Deep visual SLAM for monocular, stereo, and RGB-d cameras. In NeuRIPS, 2021. 3, 6 [^95]: Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. VideoMAE: Masked autoencoders are data-efficient learners for self-supervised video pre-training. In Advances in Neural Information Processing Systems, 2022. 2, 5 [^96]: Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: A new metric & challenges, 2019. 7 [^97]: Dani Valevski, Yaniv Leviathan, Moab Arar, and Shlomi Fruchter. Diffusion models are real-time game engines. arXiv preprint arXiv:2408.14837, 2024. 2 [^98]: Aäron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning. In Neural Information Processing Systems, 2017. 3, 4 [^99]: Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems. Curran Associates, Inc., 2017. 2 [^100]: Ruben Villegas, Mohammad Babaeizadeh, Pieter-Jan Kindermans, Hernan Moraldo, Han Zhang, Mohammad Taghi Saffar, Santiago Castro, Julius Kunze, and Dumitru Erhan. Phenaki: Variable length video generation from open domain textual descriptions. In International Conference on Learning Representations, 2023. 1 [^101]: Huiyu Wang, Mitesh Kumar Singh, and Lorenzo Torresani. Ego-only: Egocentric action detection without exocentric transferring. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5250–5261, 2023. 3 [^102]: Limin Wang, Bingkun Huang, Zhiyu Zhao, Zhan Tong, Yinan He, Yi Wang, Yali Wang, and Yu Qiao. Videomae v2: 采用双重掩码机制扩展视频掩码自编码器。收录于IEEE/CVF计算机视觉与模式识别会议（CVPR）论文集，第14549–14560页，2023年。2 [^103]: Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. OFA: unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, pages 23318–23340. PMLR, 2022. 2 [^104]: Shuzhe Wang, Vincent Leroy, Yohann Cabon, Boris Chidlovskii, and Jerome Revaud. Dust3r: Geometric 3d vision made easy. In CVPR, 2024. 7 [^105]: Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, and Furu Wei. Image as a foreign language: BEiT pretraining for vision and vision-language tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023. 2 [^106]: Xin Wang, Taein Kwon, Mahdi Rad, Bowen Pan, Ishani Chakraborty, Sean Andrist, Dan Bohus, Ashley Feniello, Bugra Tekin, Felipe Vieira Frujeri, Neel Joshi, and Marc Pollefeys. Holoassist: an egocentric human interaction dataset for interactive ai assistants in the real world. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 20270–20281, 2023. 1, 2, 3, 7, 8 [^107]: Xiang Wang, Hangjie Yuan, Shiwei Zhang, Dayou Chen, Jiuniu Wang, Yingya Zhang, Yujun Shen, Deli Zhao, and Jingren Zhou. Videocomposer: Compositional video synthesis with motion controllability. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. 1 [^108]: Xiaofeng Wang, Kang Zhao, Feng Liu, Jiayu Wang, Guosheng Zhao, Xiaoyi Bao, Zheng Zhu, Yingya Zhang, and Xingang Wang. Egovid-5m: A large-scale video-action dataset for egocentric video generation. arXiv preprint arXiv:2411.08380, 2024. 3 [^109]: Yi Wang, Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang, Sen Xing, Guo Chen, Junting Pan, Jiashuo Yu, Yali Wang, Limin Wang, and Yu Qiao. Internvideo: General video foundation models via generative and discriminative learning. arXiv preprint arXiv:2212.03191, 2022. 2 [^110]: Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Guo Chen, Baoqi Pei, Rongkun Zheng, Zun Wang, Yansong Shi, Tianxiang Jiang, Songze Li, Jilan Xu, Hongjie Zhang, Yifei Huang, Yu Qiao, Yali Wang, and Limin Wang. InternVideo2：为多模态视频理解领域扩展基础模型。收录于2024年欧洲计算机视觉会议（ECCV 2024）论文集第LXXXV卷，第396–416页，意大利米兰，2024年9月29日至10月4日。Springer出版社，2024年。2 ## [^111]: Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning --- 图像扩散模型在文本生成视频领域的应用。收录于IEEE/CVF国际计算机视觉会议论文集，第7623–7633页，2023年。1 [^112]: Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. Next-gpt: Any-to-any multimodal llm. In Proceedings of the International Conference on Machine Learning, pages 53366–53397, 2024. 1 [^113]: Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai Dai, Huazuo Gao, Yiyang Ma, Chengyue Wu, Bingxuan Wang, Zhenda Xie, Yu Wu, Kai Hu, Jiawei Wang, Yaofeng Sun, Yukun Li, Yishi Piao, Kang Guan, Aixin Liu, Xin Xie, Yuxiang You, Kai Dong, Xingkai Yu, Haowei Zhang, Liang Zhao, Yisong Wang, and Chong Ruan. Deepseek-v12: Mixture-of-experts vision-language models for advanced multimodal understanding, 2024. 2 [^114]: Hu Xu, Gargi Ghosh, Po-Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer. VideoCLIP: Contrastive pretraining for zero-shot video-text understanding. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online, 2021. Association for Computational Linguistics. 2 [^115]: Jilan Xu, Yifei Huang, Baoqi Pei, Junlin Hou, Qingqiu Li, Guo Chen, Yuejie Zhang, Rui Feng, and Weidi Xie. X-gen: Ego-centric video prediction by watching exo-centric videos. In The Thirteenth International Conference on Learning Representations, 2025. 3 [^116]: Antoine Yang, Arsha Nagrani, Paul Hongsuck Seo, Antoine Miech, Jordi Pont-Tuset, Ivan Laptev, Josef Sivic, and Cordelia Schmid. Vid2seq: Large-scale pretraining of a visual language model for dense video captioning. In CVPR, 2023. 1 [^117]: Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, Da Yin, Xiaotao Gu, Yuxuan Zhang, Weihan Wang, Yean Cheng, Bin Xu, Yuxiao Dong, and Jie Tang. Cogvideox: Text-to-video diffusion models with an expert transformer. In The Thirteenth International Conference on Learning Representations, 2025. 1 [^118]: Hanrong Ye, Haotian Zhang, Erik Daxberger, Lin Chen, Zongyu Lin, Yanghao Li, Haoxuan You, Dan Xu, Zhe Gan, Jiasen Lu, Yinfei Yang, and Bowen Zhang. MMEgo: Towards building egocentric multimodal LLMs. In The Thirteenth International Conference on Learning Representations, 2025. 2 [^119]: Yufei Ye, Poorvi Hebbar, Abhinav Gupta, and Shubham Tulsiani. Diffusion-guided reconstruction of everyday hand-object interaction clips. In Proceedings of the IEEE/CVF international conference on computer vision, pages 19717–19728, 2023. 3 [^120]: Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved VQGAN. In International Conference on Learning Representations, 2022. 4, 1 [^121]: Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision- 语言模型的行为类似于词袋模型，该如何应对这一问题？收录于第十一届学习表征国际会议论文集，2023年。1 [^122]: Xinyu Zhan, Lixin Yang, Yifei Zhao, Kangrui Mao, Hanlin Xu, Zenan Lin, Kailin Li, and Cewu Lu. OakInk2: A Dataset of Bimanual Hands-Object Manipulation in Complex Task Completion. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 445–456, Los Alamitos, CA, USA, 2024. IEEE Computer Society. 1 [^123]: Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video understanding. In Conference on Empirical Methods in Natural Language Processing, 2023. 1 [^124]: Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018. 7 [^125]: Siwei Zhang, Qianli Ma, Yan Zhang, Zhiyin Qian, Taein Kwon, Marc Pollefeys, Federica Bogo, and Siyu Tang. Egobody: Human body shape and motion of interacting people from head-mounted devices. In European Conference on Computer Vision, pages 180–200. Springer, 2022. [^126]: Yabo Zhang, Yuxiang Wei, Dongsheng Jiang, Xiaopeng Zhang, Wangmeng Zuo, and Qi Tian. Controlvideo: Training-free controllable text-to-video generation. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. 7, 8 [^127]: Yue Zhao, Ishan Misra, Philipp Krähenbühl, and Rohit Girdhar. Learning video representations from large language models. In CVPR, 2023. 1 [^128]: Yang Zheng, Yanchao Yang, Kaichun Mo, Jiaman Li, Tao Yu, Yebin Liu, C Karen Liu, and Leonidas J Guibas. Gimo: Gaze-informed human motion prediction in context. In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XIII, pages 676–694. Springer, 2022. 3 [^129]: Yipin Zhou and Tamara L Berg. Temporal perception and prediction in ego-centric video. In Proceedings of the IEEE International Conference on Computer Vision, pages 4498–4506, 2015. 3 [^130]: Yi Zhou, Connelly Barnes, Lu Jingwan, Yang Jimei, and Li Hao. On the continuity of rotation representations in neural networks. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019. 3