# HD-EPIC：一个高度精细的以自我为中心的视频数据集

托比·佩雷特 $^{**}$ 艾哈迈德·达尔卡希尔 $^{**}$ 萨普塔尔希·辛哈 $^{**}$ 奥马尔·埃马拉 $^{**}$ 萨姆·波拉德 $^{**}$ 克兰蒂·库马尔·帕里达 $^{**}$ 刘凯婷 $^{**}$ 普拉杰瓦尔·加蒂 $^{**}$ 西德汉特·班萨尔 $^{**}$ 凯文·弗拉纳根 $^{**}$ 雅各布·查尔克 $^{**}$ 朱志凡 $^{**}$ 罗德里·格里尔 $^{**}$ 法赫德·阿卜杜勒阿齐姆 $^{**}$ 朱斌 $^{**}$

Davide Moltisanti Michael Wray Hazel Doughty Dima Damen

 $^{*}$布里斯托大学 $^{*}$莱顿大学 新加坡管理大学 巴斯大学 $^{*}$：贡献同等

<http://hd-epic.github.io>

<http://hd-epic.github.io>

## 摘要

我们提供了一个新采集的、基于厨房场景的以自我为中心视频的验证数据集，这些视频经过人工标注，包含高度精细且相互关联的实况标签，涵盖：食谱步骤、细粒度动作、带有营养值的食材、移动物体以及音频标注。重要的是，所有标注都通过场景的数字孪生技术实现三维定位，同时结合了视线信息。视频采集自不同家庭环境中的非脚本化录制内容，这使得HD EPIC成为首个在真实场景中采集、但标注精细度可与受控实验室环境相媲美的数据集。

我们通过一个包含2.6万条问题的具有挑战性的VQA基准测试，展示了我们高度精细的标注的潜力；该测试用于评估模型识别食谱、食材、营养成分、细粒度动作、三维感知、物体运动以及视线方向的能力。功能强大的长上下文Gemini Pro在该基准测试中仅取得37.6%的准确率，这一结果既体现了测试的难度，也凸显了当前视觉语言模型的不足。我们还基于HD-EPIC对动作识别、声音识别以及长期视频-物体分割任务进行了评估。

HD-EPIC包含来自9个厨房的共41小时的视频数据，配有413个厨房设施的数字孪生模型，记录了69种食谱、5.9万个细粒度动作、5.1万个音频事件、2万个物体运动轨迹以及3.7万个三维物体掩码。平均而言，我们非脚本化视频中每分钟就有263个标注。

![Image](images/aee6da957ed68e051c7414c79a9ba496430b69d8f44e530bb2e9de4d0f26caa3.jpg)

<div style="text-align: center;">Figure 1. Annotation Highlights. We capture multi-day recordings of unscripted activities. Centre-Top: Recipes are recorded with steps and their preparation temporarily annotated, along with ingredient addition. Ingredients are weighed and nutrition recorded. Centre-Middle: Dense fine-grained narrations detailing what, how, and why are parsed and clustered. Audio events are also annotated. Centre-Bottom: Object movements are temporally annotated with bounding boxes and hands and object masks. Right-Top: All annotations are temporally grounded in a 3D digital twin. We show trajectories of 3 (masked) objects: Sweet potato, Food processor and Spoon, highlighting relevant kitchen fixtures. Right-Bottom: Gaze captures when objects are primed (i.e. looked at) before being taken/placed.</div>

---

### 1. 引言

对于人类而言，从短暂的细粒度动作到持续数小时的整体活动，对视频的细致理解轻而易举，但目前无论是基础模型还是专用模型都尚无法实现这一点。尤其是以自我为中心的视频，给通用视频理解带来了额外挑战：包括显著的摄像头运动、细微的动作变化、操作过程中被遮挡或频繁移出视野的物体等。理解这类视频需要分离头部运动、手部交互以及动态场景的整体理解所发出的综合信号。因此，以自我为中心的视频是全面评估视频感知模型的绝佳测试平台。

近年来，以自我为中心的视觉研究领域涌现了大量数据集 [^18] [^28] [^29] [^80]。虽然这些数据集规模庞大，适合用于训练，但标注较为稀疏，尤其对于涉及长视频各部分的关联任务或需要三维定位的任务更是如此。相比之下，标注丰富的数据集往往属于合成数据，或在受控环境中采集 [^7] [^50] [^68]，因此其真实性有限。为填补这一缺口，我们推出了这个标注最密集的非脚本化录制数据集，非常适合用于对纯视频模型及视频-语言模型进行全面验证。

★ 菜谱步骤都带有时间标注，并与与该步骤相关的所有准备动作标注相链接。

我们收集新的视频，以便获取更多元数据，并确保这些视频未被用于训练现有模型。参照EPIC-Kitchens [^17] 的做法，参与者会记录三天内的所有厨房活动。因此我们将我们的数据集命名为“高度详细的EPIC（HD-EPIC）”。图1展示了多层标注的概览，其中不少标注都是首创：

★ 每个动作都有详细的描述，涵盖了动作在开始和结束时间点的内容、方式及原因。

★ 视频中会称量食材的重量并标注其营养成分。随着食材的加入，我们会实时追踪菜肴的营养成分。

★ 针对每个厨房，我们都会整理出带有标注的数字化孪生模型。这些模型会与相关动作（如打开/关闭）以及物体的拿取/放置操作相关联。

★ 所有被移动的物体都会被追踪，手动绘制的掩码会被升级为3D边界框。

 $^{*}$ 我们将视线与物体运动关联起来，标注出在拿取/放置动作发生前物体被看到的情况。

借助这些详细的标注，我们设计了一个包含2.6万个问题的具有挑战性的视觉问答（VQA）基准测试。我们有意不采用大语言模型来生成负样本，而是使用类似的标注数据。下面介绍几种新颖的问题类型：

★ 菜谱营养：我们询问添加一种或多种食材后菜谱营养成分的变化情况。

 $^{*}$ 多视频场景：我们询问跨多个录像片段制作的菜谱相关信息，这类VQA问题会涉及多段长视频。

★ 物体移动轨迹：我们询问在长视频中物体相对于厨房固定装置的多次移动情况。

★ 固定装置交互：我们询问某个特定的橱柜/抽屉被打开/关闭的次数。

★ 动作的方式/原因：我们询问某个动作是如何/为何执行的，答案来自参与者的叙述性说明。

★ 基于视线的预判：借助视线引导，我们可以查询下一个物体的移动方向，从而实现有依据的预判。此外，我们还报告了动作识别、声音识别和长视频物体分割任务的结果。

因此本文的贡献包括：(i) 41小时的多日无脚本第一视角录像；(ii) 高度详细的标注，包括新颖的标签（如食材营养、数字化孪生、视线引导）；(iii) 具有挑战性的VQA基准测试，涵盖新颖的问题类型（如物体移动轨迹、菜谱营养变化），同时还包括3个标准的视频基准测试。

### 2. 相关工作

随着基础模型的兴起[^5] [^6] [^11] [^13] [^21] [^59] [^70] [^79] [^81] [^88]，近期涌现了大量旨在测试视频理解能力的基准测试[8, 10, 15, 23, 25, 35, 42–44, 49, 58, 78, 92]。这些基准测试评估了多种能力，例如物理规律[^58]、计数[^23]、时序推理[^8] [^10]以及长视频处理[^23] [^25] [^49]。

有几项基准测试旨在检验具身认知或自我中心理解能力。[^28]发布了一项自然语言查询（NLQ）基准测试，包含1.92万条围绕物体情景记忆的查询。[^48]收集了1.6千条由人类编写的问题与答案，涉及物体相对位置、情景记忆和空间推理等主题。不过，该数据集采用了HM3D [^63]和ScanNet [^16]数据集中的视图，因此这些问题是基于静态环境的被动视图提出的。[^49]和[^86]根据Ego4D叙述内容自动生成了5千条和7千条问题。虽然这种方法效率较高，但仅限于这些简短的叙述。[^8]为包括Ego4D在内的多个数据集中的视频自行标注了相关内容；其基准测试仅聚焦于与排序、计数、因果关系和方向相关的时间类问题。

为了评估更广泛的能力，需要更多种类的标注数据。其中尤为重要的是3D定位标注。Ego4D [^28]包含了一些环境扫描数据和静态3D物体位置信息。配备SLAM设备的[^91]构建了一个3D物体追踪基准测试；[^50]包含了办公室和客厅的数字孪生模型；而[^29]则收录了专家任务中的自我视角和外部视角数据。

与那些仅关注少数几种标注类型的研究不同，我们在一个数据集中收集了最全面的标注内容，包括高度详细的描述、物体和手部分割信息，以及场景和物体的完整3D数字孪生模型，所有数据均来自参与者家中未经脚本安排的第一人称视角视频。

### 3. 数据收集

招募与设备。每位参与者需投入大量时间（约50小时）参与数据记录——

---

![Image](images/3714815889c843526b65483dadad3ea4e20c97aaf8ae705028cfd918acfdd919.jpg)

<div style="text-align: center;">Figure 2. Diversity in HD-EPIC, which is filmed over 3 days in-the-wild, resulting in many objects, activities and recipes.</div>

![Image](images/321a5f79d554478c648b5c4ea250db2f9a2bfe3daa0bd5c93c2b9ddd4937e1f0.jpg)

<div style="text-align: center;">Figure 3. Recipe modification in ingredients and steps.</div>

同时提供详细的描述、食谱和营养信息。数据是通过Project Ariaglasses [^72]采集的——这是一款多传感器平台，配备3个前置摄像头（1个RGB摄像头和2个SLAM摄像头）、7个麦克风以及用于视线估计的内置摄像头。我们以$1408 \times 1408$分辨率采集30帧/秒的RGB视频，以60帧/秒的频率采集眼动数据，以30帧/秒的频率采集SLAM数据。我们为参与者提供了多种设备，包括用于营养追踪的电子秤（见图A1）。

指导说明与采集的数据。参与者需连续至少3天记录他们所有的日常厨房活动。9位参与者每次进入厨房时都被要求佩戴眼镜，进入时按下录制键，离开厨房时停止录制。参与者的录制时长在3.5到7.2小时之间（平均4.6小时）。总体而言，我们共采集了156段视频，平均长度为15.9（$\pm$ 14.5）分钟，总时长达到41.3小时（446万帧）。图2展示了所采集数据的多样性。

数据收集完成后，参与者提供了他们自行制作的食谱，并注明来源（如网站）及任何修改内容（见图3）。我们共收集了69份涵盖各类菜系的食谱。平均而言，每份食谱包含6.6个步骤、8.1种食材，从准备到上桌耗时4小时48分钟，涉及2.1段视频。耗时最长的食谱花费了2天6小时才完成。

为了追踪食谱的营养成分，参与者使用MyFitnessPal [^3]对食材进行称重并手动记录，从而为我们提供了详细的营养信息，也为数据集增添了额外的维度。总体而言，参与者使用了558种食材，包括高蛋白食材（如金枪鱼和芸豆）、碳水化合物类食材（如枣和面粉）以及脂肪类食材（如酸奶油和松子）。参与者制作的菜肴既有高热量菜品（如“懒人蛋糕”，含4.8千卡热量），也有低热量菜品（如“脆黄瓜沙拉”，含274千卡热量）。

叙述内容。我们遵循先前的数据集 [^17] [^18] [^28]的方法，要求参与者观看自己的录像，并通过基于网页的叙述工具 [^28]进行叙述。在此基础上，我们进一步要求参与者描述他们正在做的事情、具体做法以及原因。由此产生的叙述内容比以往的数据集更为丰富、详细（例如，$3.8\times$每分钟的字数多于Ego4D）。相关统计数据见补充材料B。

后期处理——多视频拼接与视线追踪。我们使用Aria MPS [^1]处理视频，从而得到每个厨房的单一天数点云、1kHz采样率的6自由度相机轨迹以及眼睛注视方向。我们对VRS文件进行后期处理，将视频转换为mp4格式，并移除用于匿名处理的视线摄像头输入。更多细节见补充材料B。

### 4. 标注流程

为了获得使HD-EPIC区别于其他视频理解数据集的细节水平，我们收集了广泛的多层级标注数据。下文将详细介绍我们的流程。

#### 4.1. 食谱步骤与食材的标注

我们的视频与网上常见的简短食谱视频有所不同，后者通常只截取关键步骤，还经常经过进一步编辑或加速处理。HD-EPIC中的视频涵盖了更广泛的与食谱相关的活动，比如取用或准备食材。为了全面标注这些视频，我们引入了“准备工作”与“步骤”配对机制。

某个步骤对应的准备工作，是指参与者为执行该步骤所采取的所有必要行动。例如，“切番茄”这一步骤的准备工作包括从储物处取出番茄、清洗番茄，以及拿取刀和砧板。但如果步骤是“加入切碎的洋葱并搅拌”，那么切洋葱的过程就属于该步骤的准备工作。这种标注方式能更细致地理解所有步骤，这是以往的数据集中未曾涉及的 [^29] [^39] [^73]。图4展示了示例性的准备工作-步骤标注。

---

![Image](images/8a426389e029a2387bc09bc091f5ceb21a5c8d0df15fc7bfe1677cfd537e8a68.jpg)

<div style="text-align: center;">Figure 4. For the 'Carbonara' recipe, we visualise the prep and step time segments for three consecutive steps (left), along with sample frames with corresponding action narrations (top). The interleaving of different preps/steps is evident in the annotations.</div>

![Image](images/13f0bfed169997cd1e1bf2bc3a3cbc3590589cd34f773b2ba81ed9baa442a4fe.jpg)

<div style="text-align: center;">Figure 5. Frequency of verb clusters (top) and noun clusters (bottom) in narrated sentences by category, shown on a logarithmic scale.</div>

![Image](images/9862df2132efd6f2108c23ed93c72ebcf5ad279cc0aa114716778927e3b348d9.jpg)

<div style="text-align: center;">Figure 6. Nutrition is monitored throughout recipes as ingredients are incorporated into dishes. Here we show Banana Bread Chocolate Chip Cookies. We annotate when ingredients are weighed, document their nutrition, and locate their adding time, giving us overall dish nutrition at each stage.</div>

共有3个步骤的标注结果。几乎所有步骤（93.1%）都有配对的准备工作标注。通常，准备工作的时间短于步骤本身：平均准备工作时长为54.5秒（$\pm$95.3秒），平均步骤时长为78.2秒（$\pm$100.7秒）。

我们还标注了称重及添加食材的时间段，以便监测随着食材加入整个菜肴的营养成分变化（见图6）。总共我们标注了283个可视范围内的称重序列（平均时长18.9秒）和501个添加食材序列（平均时长31.6秒），香料除外。标注过程的详情见补充材料C。

#### 4.2. 细粒度动作标注

转录工作。我们自动转录参与者提供的所有音频叙述内容，并手动检查校正，从而获得详细的动作描述。

动作边界标注。对于所有叙述内容，我们都会标注精确的开始和结束时间。总共我们获得了59,454个动作的片段，平均持续时间为2.0秒（$\pm$3.4秒）。

解析。我们从开放词汇叙述中解析动词、名词和手部动作，以便将其用于封闭词汇任务，例如动作识别。我们还分别从16,004条和11,540条叙述中提取“如何”和“为何”类从句。例如：“用左手顺时针转动盐罐，使罐盖与罐口对齐。”

聚类。图5展示了HD-EPIC数据集中所有视频的聚类（即类别）分布，以及层次化聚类[^18]。与以往的数据集[^18] [^28]类似，我们的动作和物体类别具有长尾特征，多样性极高。

声音标注。我们遵循[^31]的方法收集音频标注。这些标注记录了音频事件的起止时间以及类别名称（如“咔嗒声”、“沙沙声”、“金属与塑料碰撞声”、“流水声”）。总体而言，我们共收集了来自44个类别的50,968条音频标注。

转录、边界标记、解析、聚类和声音标注的完整细节详见补充材料C。

---

![Image](images/d6a83e877fb791332de355ae4b09c427f44a18ffe772913abbda36b0b403e836.jpg)

<div style="text-align: center;">Figure 7. Digital Twin: from point cloud (left), to surfaces (middle) and labelled fixtures (right). We show two moved objects (masks on top) at fixtures: cheese and pan. Body poses from [^87].</div>

#### 4.3. 数字孪生：场景与物体运动

场景。我们通过重建表面并手动整理每个固定装置（如橱柜、抽屉）、储物空间（如架子、挂钩）以及大型电器（如冰箱、微波炉），来创建参与者厨房的数字副本。这与依赖已知环境复制品的数字孪生不同。我们的数字孪生是在Blender[^2]中，基于录制视频的多视角SLAM点云创建的。

每个厨房平均包含45.9个已标注的固定装置（最少31个，最多62个），其中包括14.2个台面/表面、12.2个橱柜、7.8个抽屉和5.2件电器（示例如图7所示）。我们将这些标注称为固定装置F。

接着，我们将描述场景交互的旁白与F关联起来。我们会找出那些名词表示固定物体的动作，比如“打开抽屉”，然后在数字孪生模型中定位到具体的“抽屉”（例如drawer.001）并更新其状态。根据以往研究表明，人类在互动前最多会注视1秒 [^40]，因此我们会选取旁白出现前1秒内累计注视交集最大的固定物体 $f \in F$。

\textbf{手部掩码。}我们为每段视频中的双手标注少量帧。所选帧需覆盖各种动作和厨房位置。我们利用这些帧自动分割手部区域，并对部分结果进行人工校正。总体而言，我们的数据集包含770万个手部掩码：其中390万个为右手掩码，380万个为左手掩码，其中1.1万个为人工标注（详情见补充材料C）。

二维物体运动。为了生成三维物体运动标注，我们首先标注物体移动的时间段。标注员会在物体被移动直至放置好的整个时间段内标注相应的二维物体边界框，分别位于运动起始和结束位置。例如，如果有人将杯子从台面移到水槽，一个边界框对应台面上的杯子，另一个对应水槽中的杯子。即使是微小的位移或推动也会被标注为轨迹，

![Image](images/913b85b351097a9d16fbc8049e1f9b96846556bdeb11b7954a2321a27987ce2f.jpg)

<div style="text-align: center;">Figure 8. Priming Object Interaction Through Gaze. Top: Camera position with projected eye-gaze and object positions in 3D. Middle: 2D gaze location. Bottom: Timeline for priming object movement e.g. the glass is primed 8.3s before taking.</div>

从而提供所有物体运动的完整标注。

总体而言，我们共收集了19,900条物体运动轨迹和36,900个边界框。平均而言，每分钟有9.2个物体被拿取，9.0个物体被放置。轨迹的平均长度为9.0秒，最长可达461.5秒。

物体掩码。尽管分割技术取得了进展 [^36] [^64]，也有现成的标注数据可用 [^19] [^29]，但模型在 Egocentric 视频中的表现仍较差，尤其是在有遮挡的情况下。我们先用迭代式SAM2算法对每个边界框生成像素级分割结果 [^64]，再进行人工修正。标注员修正了74%的掩码；SAM2生成的掩码与人工修正后的掩码之间的IoU值达到0.82。

掩码转3D。我们利用密集深度估计以及MPS提供的2D到3D稀疏对应关系，将物体掩码转换为3D形式。根据[^85]提供的度量深度信息，我们确定集合S，即物体内部或周围具有3D对应关系的像素集。随后我们计算线性变换系数：$\alpha, \beta = \arg\min_{\alpha, \beta} \|\left(\alpha\hat{D}_S + \beta\right) - D_S\|^2$，其中$\hat{D}_S$为估算的深度值，$D_S$为已有的深度值，最后通过RANSAC算法剔除异常值。

3D物体运动。物体平均移动距离为61.4厘米（±84.5厘米），其中27.6%的物体移动距离不超过10厘米，而7.6%的物体移动距离则达到2米以上。

物体与场景的交互。根据3D物体位置，我们将它们与最近的固定物$f \in F$关联起来，同时遵循特定于固定物的规则（例如物体必须位于台面的x-y平面内）。我们会手动验证所有关联结果，并纠正任何错误。平均而言，每个视频中物体会在1.8个不同的固定物之间移动（详细统计数据见补充材料C）。

物体移动的预示现象。关于拾取和放置物体时的视线行为已有大量研究[^32] [^40]。我们结合眼动数据与3D物体位置，来识别物体被“预示”的时刻——即拾取物体前视线聚焦于该物体位置的时刻（拾取预示），或是放置物体前视线聚焦于其未来位置的时刻（放置预示）。

我们计算了所有物体的预示时间，排除了那些在屏幕外被拾取或放置的物体。此外，有时人们在真正拾取物体之前就已经开始操控它，因此我们也排除了那些有拾取位置的物体。

---

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'>Dataset</td><td style='text-align: center; word-wrap: break-word;'>Val&amp;Test Hours</td><td style='text-align: center; word-wrap: break-word;'>Action Segments</td><td style='text-align: center; word-wrap: break-word;'>Unscripted</td><td style='text-align: center; word-wrap: break-word;'>Free Setting</td><td style='text-align: center; word-wrap: break-word;'>Recipe</td><td style='text-align: center; word-wrap: break-word;'>Nutrition</td><td style='text-align: center; word-wrap: break-word;'>Gaze</td><td style='text-align: center; word-wrap: break-word;'>Audio Labels</td><td style='text-align: center; word-wrap: break-word;'>Object</td><td style='text-align: center; word-wrap: break-word;'>Hand</td><td style='text-align: center; word-wrap: break-word;'>3D object over time</td><td style='text-align: center; word-wrap: break-word;'>Labelled 3D environment</td><td style='text-align: center; word-wrap: break-word;'>Camera pose</td><td style='text-align: center; word-wrap: break-word;'>Fully annotated</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>HOI4D [^46]</td><td style='text-align: center; word-wrap: break-word;'>11.4</td><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>✗</td><td style='text-align: center; word-wrap: break-word;'>✗</td><td style='text-align: center; word-wrap: break-word;'>✗</td><td style='text-align: center; word-wrap: break-word;'>✗</td><td style='text-align: center; word-wrap: break-word;'>✗</td><td style='text-align: center; word-wrap: break-word;'>✗</td><td style='text-align: center; word-wrap: break-word;'>Mask</td><td style='text-align: center; word-wrap: break-word;'>Mask</td><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>✗</td><td style='text-align: center; word-wrap: break-word;'>✓</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Assembly101 [^68]</td><td style='text-align: center; word-wrap: break-word;'>66.8</td><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>✗</td><td style='text-align: center; word-wrap: break-word;'>✗</td><td style='text-align: center; word-wrap: break-word;'>✗</td><td style='text-align: center; word-wrap: break-word;'>✗</td><td style='text-align: center; word-wrap: break-word;'>✗</td><td style='text-align: center; word-wrap: break-word;'>✗</td><td style='text-align: center; word-wrap: break-word;'>✗</td><td style='text-align: center; word-wrap: break-word;'>3D pose</td><td style='text-align: center; word-wrap: break-word;'>✗</td><td style='text-align: center; word-wrap: break-word;'>✗</td><td style='text-align: center; word-wrap: break-word;'>✗</td><td style='text-align: center; word-wrap: break-word;'>✓</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>EPIC-KITCHENS-100 [^18]</td><td style='text-align: center; word-wrap: break-word;'>25.3</td><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>✗</td><td style='text-align: center; word-wrap: break-word;'>✗</td><td style='text-align: center; word-wrap: break-word;'>✗</td><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>Mask</td><td style='text-align: center; word-wrap: break-word;'>Mask</td><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>✗</td><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>✗</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Ego4D [^28]</td><td style='text-align: center; word-wrap: break-word;'>288.7</td><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>✗</td><td style='text-align: center; word-wrap: break-word;'>✗</td><td style='text-align: center; word-wrap: break-word;'>✗</td><td style='text-align: center; word-wrap: break-word;'>✗</td><td style='text-align: center; word-wrap: break-word;'>B-Box</td><td style='text-align: center; word-wrap: break-word;'>B-Box</td><td style='text-align: center; word-wrap: break-word;'>✗</td><td style='text-align: center; word-wrap: break-word;'>✗</td><td style='text-align: center; word-wrap: break-word;'>✗</td><td style='text-align: center; word-wrap: break-word;'>✗</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>HoloAssist [^80]</td><td style='text-align: center; word-wrap: break-word;'>49.8</td><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>✗</td><td style='text-align: center; word-wrap: break-word;'>✗</td><td style='text-align: center; word-wrap: break-word;'>✗</td><td style='text-align: center; word-wrap: break-word;'>✗</td><td style='text-align: center; word-wrap: break-word;'>✗</td><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>✗</td><td style='text-align: center; word-wrap: break-word;'>3D pose</td><td style='text-align: center; word-wrap: break-word;'>✗</td><td style='text-align: center; word-wrap: break-word;'>✗</td><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>✓</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Aria Digital Twin [^50]</td><td style='text-align: center; word-wrap: break-word;'>8.1</td><td style='text-align: center; word-wrap: break-word;'>✗</td><td style='text-align: center; word-wrap: break-word;'>✗</td><td style='text-align: center; word-wrap: break-word;'>✗</td><td style='text-align: center; word-wrap: break-word;'>✗</td><td style='text-align: center; word-wrap: break-word;'>✗</td><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>✗</td><td style='text-align: center; word-wrap: break-word;'>Mask</td><td style='text-align: center; word-wrap: break-word;'>✗</td><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>✓</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Aria Everyday Activities [^47]</td><td style='text-align: center; word-wrap: break-word;'>7.3</td><td style='text-align: center; word-wrap: break-word;'>✗</td><td style='text-align: center; word-wrap: break-word;'>✗</td><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>✗</td><td style='text-align: center; word-wrap: break-word;'>✗</td><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>✗</td><td style='text-align: center; word-wrap: break-word;'>✗</td><td style='text-align: center; word-wrap: break-word;'>✗</td><td style='text-align: center; word-wrap: break-word;'>✗</td><td style='text-align: center; word-wrap: break-word;'>✗</td><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>✗</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Aria Everyday Objects [^74]</td><td style='text-align: center; word-wrap: break-word;'>0.4</td><td style='text-align: center; word-wrap: break-word;'>✗</td><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>✗</td><td style='text-align: center; word-wrap: break-word;'>✗</td><td style='text-align: center; word-wrap: break-word;'>✗</td><td style='text-align: center; word-wrap: break-word;'>✗</td><td style='text-align: center; word-wrap: break-word;'>B-Box</td><td style='text-align: center; word-wrap: break-word;'>✗</td><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>✗</td><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>✓</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Ego-Exo4D [^29]</td><td style='text-align: center; word-wrap: break-word;'>85.1</td><td style='text-align: center; word-wrap: break-word;'>✗</td><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>✗</td><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>✗</td><td style='text-align: center; word-wrap: break-word;'>Mask</td><td style='text-align: center; word-wrap: break-word;'>3D pose</td><td style='text-align: center; word-wrap: break-word;'>✗</td><td style='text-align: center; word-wrap: break-word;'>✗</td><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>✗</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>HD-EPIC</td><td style='text-align: center; word-wrap: break-word;'>41.3</td><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>Mask</td><td style='text-align: center; word-wrap: break-word;'>Mask</td><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>✓</td></tr></table>

<div style="text-align: center;">Table 1. Comparison of Egocentric Video Datasets (see full table A1 in Supp.).</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'>Location</td><td style='text-align: center; word-wrap: break-word;'>Filtered (% Total)</td><td style='text-align: center; word-wrap: break-word;'>Feasible (% Total)</td><td style='text-align: center; word-wrap: break-word;'>Primed (% Feasible)</td><td style='text-align: center; word-wrap: break-word;'>Not Primed (% Feasible)</td><td style='text-align: center; word-wrap: break-word;'>Avg. Time (s)</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Start</td><td style='text-align: center; word-wrap: break-word;'>29.40</td><td style='text-align: center; word-wrap: break-word;'>70.60</td><td style='text-align: center; word-wrap: break-word;'>94.82</td><td style='text-align: center; word-wrap: break-word;'>5.18</td><td style='text-align: center; word-wrap: break-word;'>3.99( $\pm$2.94)</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>End</td><td style='text-align: center; word-wrap: break-word;'>66.92</td><td style='text-align: center; word-wrap: break-word;'>33.08</td><td style='text-align: center; word-wrap: break-word;'>88.46</td><td style='text-align: center; word-wrap: break-word;'>11.54</td><td style='text-align: center; word-wrap: break-word;'>2.62( $\pm$2.05)</td></tr></table>

![Image](images/6af34033266051fab972eaea598530a0f86df0f93c2b3eb156a703cbe32a02a7.jpg)

<div style="text-align: center;">Figure 9. (Top) Priming Statistics for both start and end locations (Bottom) Histogram showing the difference in time when an object is primed before it is picked up (blue) or placed (red).</div>

这些物体早在10秒前就已被视线锁定。图8展示了两个物体的视线预示情况：牛奶瓶和玻璃杯。玻璃杯的最终放置位置是橱柜，在玻璃杯被收起前3秒，其放置位置就已被视线锁定。图9显示了预示现象的统计数据：在可预示的物体中，94.8%的物体会被预示，平均在拾取前4.0秒被预示；而在放置前被预示的物体占比88.5%，平均预示时间为2.6秒。

长期物体跟踪。我们将物体运动连接起来，形成更长的轨迹，也就是物体移动路径，以捕捉物体的运动序列。我们的高效流程利用了转换后的3D位置信息，仅需几分钟就能标注完时长1小时的视频（详情见补充材料C）。

#### 4.4. HD-EPIC与以往的中心视角数据集对比

表1将HD-EPIC与其他中心视角数据集进行了比较（完整表格见补充材料）。与标注有3D环境的最大数据集Aria Digital Twin [^50]相比，HD-EPIC的视频素材量是其5倍，标注也更丰富；更重要的是，HD-EPIC是在参与者家中以非脚本化方式采集的数据。尤其值得一提的是，HD-EPIC是首个在同一组视频上标注食谱、营养值、详细动作片段、视线及音频标签的数据集。凭借这些多样且密集的标注，HD-EPIC堪称视频理解领域的真正零样本基准。

### 5. 基准测试与结果

我们展示了HD-EPIC作为验证数据集的潜力，具体通过通用视频问答（VQA）（第5.1节）、动作与声音识别（第5.2节）以及长视频物体分割（第5.3节）方面的基准测试来验证。

![Image](images/d976114fa4252ca0ee33a3e9dc530129456a0b63e613d92a7606dfd6fd48e6e4.jpg)

<div style="text-align: center;">Figure 10. VQA Question Prototypes. We show our 30 question prototypes by category alongside the number of questions. Outer bars indicate the distribution over input lengths for each question.</div>

#### 5.1. HD-EPIC VQA基准测试与分析

基准测试的构建。我们利用标注流程产生的密集输出，围绕7种类型的标注构建了全面的VQA基准测试：

1. 食谱类。涉及对食谱及其步骤进行时间定位、检索或识别的问题。

2. 食材类。涉及所用食材、其重量、添加时间及顺序的问题。

3. 营养类。涉及食材的营养成分以及加入食谱后营养变化的问题。

4. 细粒度动作类。涉及动作的内容、方式及原因，以及其时间定位的问题。

5. 三维感知类。需要理解三维场景中物体相对位置的问题。

6. 物体运动类。涉及物体在长视频中移动的位置、时间及次数等问题。

7. 视线类。涉及估算对大型标志物的注视点以及预测未来物体交互的问题。

对于每种问题类型，我们定义了原型以从标注数据中采样问题、正确答案及强负样本。例如，“物体运动计数”类问题会问：“视频中在<time>时刻出现的<bbox>物体移动了多少次？”这类问题使用了长视频，需要多次跳转才能正确回答。相比之下，

---

![Image](images/d94607670762361982116fe941a3c6670b6d34bc630b0568ee049ee50507ebf9.jpg)

<div style="text-align: center;">Figure 11. VQA Results per Question Prototype. Our benchmark contains many challenging questions for current models.</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'>Model</td><td style='text-align: center; word-wrap: break-word;'>Recipe</td><td style='text-align: center; word-wrap: break-word;'>Ingredient</td><td style='text-align: center; word-wrap: break-word;'>Nutrition</td><td style='text-align: center; word-wrap: break-word;'>Action</td><td style='text-align: center; word-wrap: break-word;'>3D</td><td style='text-align: center; word-wrap: break-word;'>Motion</td><td style='text-align: center; word-wrap: break-word;'>Gaze</td><td style='text-align: center; word-wrap: break-word;'>Avg.</td></tr><tr><td colspan="9">Blind - Language Only</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Llama 3.2</td><td style='text-align: center; word-wrap: break-word;'>33.5</td><td style='text-align: center; word-wrap: break-word;'>25.0</td><td style='text-align: center; word-wrap: break-word;'>36.7</td><td style='text-align: center; word-wrap: break-word;'>23.3</td><td style='text-align: center; word-wrap: break-word;'>22.3</td><td style='text-align: center; word-wrap: break-word;'>25.5</td><td style='text-align: center; word-wrap: break-word;'>19.5</td><td style='text-align: center; word-wrap: break-word;'>26.5</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Gemini Pro</td><td style='text-align: center; word-wrap: break-word;'>38.0</td><td style='text-align: center; word-wrap: break-word;'>26.8</td><td style='text-align: center; word-wrap: break-word;'>30.0</td><td style='text-align: center; word-wrap: break-word;'>22.1</td><td style='text-align: center; word-wrap: break-word;'>21.5</td><td style='text-align: center; word-wrap: break-word;'>27.7</td><td style='text-align: center; word-wrap: break-word;'>20.5</td><td style='text-align: center; word-wrap: break-word;'>26.7</td></tr><tr><td colspan="9">Video-Language</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>VideoLlama 2</td><td style='text-align: center; word-wrap: break-word;'>30.8</td><td style='text-align: center; word-wrap: break-word;'>25.7</td><td style='text-align: center; word-wrap: break-word;'>32.7</td><td style='text-align: center; word-wrap: break-word;'>27.2</td><td style='text-align: center; word-wrap: break-word;'>25.7</td><td style='text-align: center; word-wrap: break-word;'>28.5</td><td style='text-align: center; word-wrap: break-word;'>21.2</td><td style='text-align: center; word-wrap: break-word;'>27.4</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>LongVA</td><td style='text-align: center; word-wrap: break-word;'>29.6</td><td style='text-align: center; word-wrap: break-word;'>30.8</td><td style='text-align: center; word-wrap: break-word;'>33.7</td><td style='text-align: center; word-wrap: break-word;'>30.7</td><td style='text-align: center; word-wrap: break-word;'>32.9</td><td style='text-align: center; word-wrap: break-word;'>22.7</td><td style='text-align: center; word-wrap: break-word;'>24.5</td><td style='text-align: center; word-wrap: break-word;'>29.3</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>LLaVA-Video</td><td style='text-align: center; word-wrap: break-word;'>36.3</td><td style='text-align: center; word-wrap: break-word;'>33.5</td><td style='text-align: center; word-wrap: break-word;'>38.7</td><td style='text-align: center; word-wrap: break-word;'>43.0</td><td style='text-align: center; word-wrap: break-word;'>27.3</td><td style='text-align: center; word-wrap: break-word;'>18.9</td><td style='text-align: center; word-wrap: break-word;'>29.3</td><td style='text-align: center; word-wrap: break-word;'>32.4</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Gemini Pro</td><td style='text-align: center; word-wrap: break-word;'>60.5</td><td style='text-align: center; word-wrap: break-word;'>46.2</td><td style='text-align: center; word-wrap: break-word;'>34.7</td><td style='text-align: center; word-wrap: break-word;'>39.6</td><td style='text-align: center; word-wrap: break-word;'>32.5</td><td style='text-align: center; word-wrap: break-word;'>20.8</td><td style='text-align: center; word-wrap: break-word;'>28.7</td><td style='text-align: center; word-wrap: break-word;'>37.6</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Sample Human Baseline</td><td style='text-align: center; word-wrap: break-word;'>96.7</td><td style='text-align: center; word-wrap: break-word;'>96.7</td><td style='text-align: center; word-wrap: break-word;'>85.0</td><td style='text-align: center; word-wrap: break-word;'>92.5</td><td style='text-align: center; word-wrap: break-word;'>93.8</td><td style='text-align: center; word-wrap: break-word;'>92.7</td><td style='text-align: center; word-wrap: break-word;'>75.0</td><td style='text-align: center; word-wrap: break-word;'>90.3</td></tr></table>

<div style="text-align: center;">Table 2. VQA Results per Category (% Acc.). Our VQA benchmark cannot be solved blind or by external knowledge and is a challenge for state-of-the-art video VLM models.</div>

“动作识别”类问题会问：“对于动作<verb, noun>，最恰当的描述是什么？”以此测试模型捕捉动作细节的能力。

图10展示了各类别问题的分布以及输入长度的分布情况，输入长度范围从单帧到7小时以上的视频不等。每种原型的采样细节可在补充材料D中找到。我们的问题与答案示例可见图13。

每个问题原型均为五选一的多项选择题。我们通过从数据集中采样难答答案来为原型生成难负样本。例如，我们会选取参与者执行同一动作的4种不同答案。这确保了负样本具有真实性且问题具有挑战性。总共我们有30个原型，生成了26,650道多项选择题。这使得它成为规模最大的视频VQA基准测试之一，同时仍便于评估闭源视觉语言模型。凭借我们标注数据的密集性，我们估计利用这套原型可生成多达100,000个唯一的问题。

- Llama 3.2 90B [^21]。我们将其作为强大的开源文本基线，因为LLM即便没有视觉输入也能在视觉问答基准测试中表现优异 [^82]。

视觉语言模型。由于基准测试中许多问题原型规模较大且涉及较长时间跨度，我们选取了5个具有代表性的模型作为基线（更多细节见补充材料D）：

• VideoLlama 2 7B [^13]。开源短上下文模型。

• LLaVA-Video [^90]。同样基于自我中心数据训练的开源模型。

• Gemini Pro [^75]。闭源模型，拥有所有模型中最长的上下文窗口，且在长视频处理方面处于领先地位 [^25]。

• LongVA [^89]。上下文窗口最大的开源模型。

按类别和原型划分的VQA结果。表2给出了整体及各类别的准确率，这些数值是对图11所示原型结果取平均值得出的。仅使用语言模型的准确率仅为26.5%和26.7%，仅比随机水平高出6.7%。开源视频多模态模型（VideoLlama、LongVA、LlaVA-Video）的表现相近（分别为27.4%、29.3%、32.4%），但如图11所示各有优势。例如，Llama在营养估算方面表现更好，而要超越随机水平实现动作识别和视线估计，则需要视频数据的辅助。Gemini的表现最佳，尤其是在食谱和食材类别中，外部知识能发挥重要作用。不过，其平均准确率（37.6%）与我们的样本人类基准（90.3%）之间的差距，凸显了本VQA基准所带来的挑战。

![Image](images/0ff7b7a43deca19ad7f45a7ef54bf84522a9f2eb9f0782c35825d940c3e593e6.jpg)

<div style="text-align: center;">Figure 12. Effect of Input Length. Models struggle with questions of all video input lengths. s=second, m=minute, h=hour.</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'>Model</td><td style='text-align: center; word-wrap: break-word;'>Modality</td><td style='text-align: center; word-wrap: break-word;'>Verb</td><td style='text-align: center; word-wrap: break-word;'>Noun</td><td style='text-align: center; word-wrap: break-word;'>Action</td><td style='text-align: center; word-wrap: break-word;'>Unseen EPIC-100 Action</td></tr><tr><td colspan="5">EPIC-KITCHENS-100 SOTA</td><td style='text-align: center; word-wrap: break-word;'></td></tr><tr><td style='text-align: center; word-wrap: break-word;'>TIM [^9]</td><td style='text-align: center; word-wrap: break-word;'>A+V</td><td style='text-align: center; word-wrap: break-word;'>77.1</td><td style='text-align: center; word-wrap: break-word;'>67.2</td><td style='text-align: center; word-wrap: break-word;'>57.5</td><td style='text-align: center; word-wrap: break-word;'>44.6</td></tr><tr><td colspan="5">HD-EPIC</td><td style='text-align: center; word-wrap: break-word;'></td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Chance</td><td style='text-align: center; word-wrap: break-word;'>-</td><td style='text-align: center; word-wrap: break-word;'>10.9</td><td style='text-align: center; word-wrap: break-word;'>1.8</td><td style='text-align: center; word-wrap: break-word;'>0.0</td><td style='text-align: center; word-wrap: break-word;'>-</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>SlowFast [^24]</td><td style='text-align: center; word-wrap: break-word;'>V</td><td style='text-align: center; word-wrap: break-word;'>29.2</td><td style='text-align: center; word-wrap: break-word;'>10.6</td><td style='text-align: center; word-wrap: break-word;'>5.3</td><td style='text-align: center; word-wrap: break-word;'>29.0</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Omnivore [^26]</td><td style='text-align: center; word-wrap: break-word;'>V</td><td style='text-align: center; word-wrap: break-word;'>19.5</td><td style='text-align: center; word-wrap: break-word;'>17.1</td><td style='text-align: center; word-wrap: break-word;'>8.7</td><td style='text-align: center; word-wrap: break-word;'>28.7</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>MotionFormer-HR [^51]</td><td style='text-align: center; word-wrap: break-word;'>V</td><td style='text-align: center; word-wrap: break-word;'>35.7</td><td style='text-align: center; word-wrap: break-word;'>20.0</td><td style='text-align: center; word-wrap: break-word;'>10.2</td><td style='text-align: center; word-wrap: break-word;'>32.2</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>VideoMAE-L [^76]</td><td style='text-align: center; word-wrap: break-word;'>V</td><td style='text-align: center; word-wrap: break-word;'>47.5</td><td style='text-align: center; word-wrap: break-word;'>29.4</td><td style='text-align: center; word-wrap: break-word;'>17.9</td><td style='text-align: center; word-wrap: break-word;'>29.3</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>TIM [^9]</td><td style='text-align: center; word-wrap: break-word;'>A+V</td><td style='text-align: center; word-wrap: break-word;'>51.3</td><td style='text-align: center; word-wrap: break-word;'>36.1</td><td style='text-align: center; word-wrap: break-word;'>23.4</td><td style='text-align: center; word-wrap: break-word;'>44.6</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>TIM [^9]</td><td style='text-align: center; word-wrap: break-word;'>V</td><td style='text-align: center; word-wrap: break-word;'>51.2</td><td style='text-align: center; word-wrap: break-word;'>36.5</td><td style='text-align: center; word-wrap: break-word;'>23.9</td><td style='text-align: center; word-wrap: break-word;'>44.4</td></tr></table>

<div style="text-align: center;">Table 3. Action Recognition Benchmark (% Acc.). HD-EPIC provides a significant challenge for state-of-the-art models.</div>

LlaVA-Video的表现相近（分别为27.4%、29.3%、32.4%），但如图11所示，它们各有优势。例如，Llama在营养估算方面表现更佳，而要超越随机水平实现动作识别和视线估计，视频数据则是必不可少的。Gemini的性能最优，尤其是在需要外部知识辅助的食谱和食材相关任务中。不过，其平均性能仅为37.6%，且与人类样本基准水平（90.3%）之间的差距，凸显出本VQA基准测试所面临的挑战。

视频长度。图12显示，各模型在所有视频长度上的表现均不佳，而在处理时长为$\geq 1$分钟的输入时表现最差。

#### 5.2. 识别基准测试

#### 5.2. 识别基准测试

动作识别。我们评估了5种动作识别方法[^9] [^24] [^26] [^51] [^76]，这些方法使用的是在EPIC-KITCHENS-100上微调后的公开可用检查点。结果见表3。为便于参考，表中还列出了EPIC-KITCHENS-100的原始结果（首行）以及在该数据集的未见过的厨房子集上的测试结果（最后一列）。最佳表现

---

![Image](images/a35a087b6dbf3fece81cd592256b7d068352bd3ac7f55e2f434712ecbb01bd5d.jpg)

<div style="text-align: center;">Figure 13. VQA Qualitative Results. We mark GT answers with a green background, and predictions from different models, i.e., LLaMA</div>

<div style="text-align: center;">3.2, VideoLLaMA 2, LongVA, Gemini Pro with coloured dots. Note: Under Nutrition, [fat] values are not provided to the model.</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'>Model</td><td style='text-align: center; word-wrap: break-word;'>Modality</td><td style='text-align: center; word-wrap: break-word;'>Top-1</td><td style='text-align: center; word-wrap: break-word;'>Top-5</td><td style='text-align: center; word-wrap: break-word;'>mCA</td><td style='text-align: center; word-wrap: break-word;'>mAP</td><td style='text-align: center; word-wrap: break-word;'>mAUC</td></tr><tr><td colspan="7">EPIC-Sounds SOTA</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>TIM [^9]</td><td style='text-align: center; word-wrap: break-word;'>A+V</td><td style='text-align: center; word-wrap: break-word;'>58.3</td><td style='text-align: center; word-wrap: break-word;'>86.0</td><td style='text-align: center; word-wrap: break-word;'>25.8</td><td style='text-align: center; word-wrap: break-word;'>30.6</td><td style='text-align: center; word-wrap: break-word;'>0.879</td></tr><tr><td colspan="7">HD-EPIC</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Chance</td><td style='text-align: center; word-wrap: break-word;'>-</td><td style='text-align: center; word-wrap: break-word;'>6.9</td><td style='text-align: center; word-wrap: break-word;'>29.4</td><td style='text-align: center; word-wrap: break-word;'>2.2</td><td style='text-align: center; word-wrap: break-word;'>2.3</td><td style='text-align: center; word-wrap: break-word;'>0.500</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>SSAST [^27]</td><td style='text-align: center; word-wrap: break-word;'>A</td><td style='text-align: center; word-wrap: break-word;'>25.1</td><td style='text-align: center; word-wrap: break-word;'>59.8</td><td style='text-align: center; word-wrap: break-word;'>10.8</td><td style='text-align: center; word-wrap: break-word;'>13.5</td><td style='text-align: center; word-wrap: break-word;'>0.748</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>TIM [^9]</td><td style='text-align: center; word-wrap: break-word;'>A</td><td style='text-align: center; word-wrap: break-word;'>26.9</td><td style='text-align: center; word-wrap: break-word;'>56.9</td><td style='text-align: center; word-wrap: break-word;'>12.4</td><td style='text-align: center; word-wrap: break-word;'>11.4</td><td style='text-align: center; word-wrap: break-word;'>0.689</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>ASF [^34]</td><td style='text-align: center; word-wrap: break-word;'>A</td><td style='text-align: center; word-wrap: break-word;'>27.9</td><td style='text-align: center; word-wrap: break-word;'>64.0</td><td style='text-align: center; word-wrap: break-word;'>11.9</td><td style='text-align: center; word-wrap: break-word;'>14.0</td><td style='text-align: center; word-wrap: break-word;'>0.741</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>TIM [^9]</td><td style='text-align: center; word-wrap: break-word;'>A+V</td><td style='text-align: center; word-wrap: break-word;'>31.9</td><td style='text-align: center; word-wrap: break-word;'>61.0</td><td style='text-align: center; word-wrap: break-word;'>14.4</td><td style='text-align: center; word-wrap: break-word;'>15.7</td><td style='text-align: center; word-wrap: break-word;'>0.765</td></tr></table>

<div style="text-align: center;">Table 4. Sound Recognition Benchmark. Current models struggle on HD-EPIC compared to the EPIC-Sounds state-of-the-art.</div>

在HD-EPIC数据集上，动词识别准确率仅为51%，名词识别准确率为37%，动作识别准确率更是只有24%，仍有很大的提升空间。

声音识别。我们评估了3种音频模型[^9] [^27] [^34]，它们均在EPIC-Sounds数据集上训练。表4显示，与EPIC-Sounds数据集相比，HD-EPIC数据集上SSAST、ASF和TIM模型的性能差距分别达到-28.4、-25.9和-26.4。这表明音频模型对新场景或设备的适应性不足。

### 5.3. 长期视频对象分割基准测试

我们利用自己生成的分割结果和跟踪关联数据（见第4.3节）构建了长期视频对象分割基准测试。该基准包含1000个序列，每个序列有1-5个物体以及2个手部掩码（详情见补充材料D）。虽然我们拥有的跟踪数据更多，但为保证可比性，基准规模仍与现有基准相当。我们评估了两种模型[^12] [^64]，同时设了一个将物体掩码保持静态的简单基线作为对比。图14展示了测试结果。SAM2 [^64]在手部分割任务上的表现优于Cutie [^12]，但在物体分割任务上表现较差。总体而言，物体在视角、光照、位置及遮挡方面的多样性给分割带来了更大挑战。

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'></td><td colspan="3">Total</td><td colspan="3">Hands</td><td colspan="3">Objects</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Model</td><td style='text-align: center; word-wrap: break-word;'>J</td><td style='text-align: center; word-wrap: break-word;'>F</td><td style='text-align: center; word-wrap: break-word;'>J&amp;F</td><td style='text-align: center; word-wrap: break-word;'>J</td><td style='text-align: center; word-wrap: break-word;'>F</td><td style='text-align: center; word-wrap: break-word;'>J&amp;F</td><td style='text-align: center; word-wrap: break-word;'>J</td><td style='text-align: center; word-wrap: break-word;'>F</td><td style='text-align: center; word-wrap: break-word;'>J&amp;F</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Static</td><td style='text-align: center; word-wrap: break-word;'>8.0</td><td style='text-align: center; word-wrap: break-word;'>10.3</td><td style='text-align: center; word-wrap: break-word;'>9.2</td><td style='text-align: center; word-wrap: break-word;'>14.6</td><td style='text-align: center; word-wrap: break-word;'>14.4</td><td style='text-align: center; word-wrap: break-word;'>14.5</td><td style='text-align: center; word-wrap: break-word;'>4.8</td><td style='text-align: center; word-wrap: break-word;'>8.4</td><td style='text-align: center; word-wrap: break-word;'>6.6</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Cutie [^12]</td><td style='text-align: center; word-wrap: break-word;'>44.8</td><td style='text-align: center; word-wrap: break-word;'>52.3</td><td style='text-align: center; word-wrap: break-word;'>48.6</td><td style='text-align: center; word-wrap: break-word;'>74.8</td><td style='text-align: center; word-wrap: break-word;'>79.5</td><td style='text-align: center; word-wrap: break-word;'>77.2</td><td style='text-align: center; word-wrap: break-word;'>30.1</td><td style='text-align: center; word-wrap: break-word;'>39.0</td><td style='text-align: center; word-wrap: break-word;'>34.6</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>SAM2 [^64]</td><td style='text-align: center; word-wrap: break-word;'>45.2</td><td style='text-align: center; word-wrap: break-word;'>49.6</td><td style='text-align: center; word-wrap: break-word;'>47.4</td><td style='text-align: center; word-wrap: break-word;'>87.5</td><td style='text-align: center; word-wrap: break-word;'>90.8</td><td style='text-align: center; word-wrap: break-word;'>89.1</td><td style='text-align: center; word-wrap: break-word;'>24.5</td><td style='text-align: center; word-wrap: break-word;'>29.5</td><td style='text-align: center; word-wrap: break-word;'>27.0</td></tr></table>

![Image](images/09c79a118d3a8bfd86c59a21981ef645ab0afb1c37f8dd2c0f43f475e9ec9ef2.jpg)

<div style="text-align: center;">Figure 14. Long-Term VOS. jaccard index  $\mathcal{I}$ & contour accuracy  $\mathcal{F}$ show Cutie and SAM2 struggle with segmenting objects.</div>

遮挡与遮蔽问题。

## 6. 展望未来

HD-EPIC的相关资源可从<http://dx.doi.org/10.5523/bris.3cqb5b81wk2dc2379fx1mrxh47-i.e>获取：包括视频、音频、视线数据、Blender数字孪生模型及相机位姿估计结果。注释数据可在<http://hd-epic.github.io-i.e>下载：涵盖物体运动轨迹、物体掩码及三维位置信息、长时序物体跟踪数据，以及物体-动作-固定装置标签。我们希望HD-EPIC能引导未来研究朝着更全面地理解第一人称视频的方向发展。

---

## 致谢

布里斯托尔大学的研究得到了EPSRC Fellowship UM-PIRE（EP/T004991/1）、EPSRC项目资助Visual AI（EP/T028572/1）以及EPSRC博士培养计划的支持。O Emara、K Flanagan和F Abdelazim的研究得到了UKRI CD in Interactive AI（EP/S022937/1）项目的资助。该项目还获得了Meta（Aria Project Partnership）向布里斯托尔大学提供的无限制慈善捐赠。Gemini Pro的研究成果则得益于Google DeepMind提供的研究经费支持。

莱顿大学的研究得到了荷兰科学研究理事会（NWO）颁发的Veni基金（VI.Veni.222.160）的支持。新加坡的研究得到了新加坡教育部（MOE）学术研究基金（AcRF）一级项目（编号MSS23C018）的资助。

我们感谢David Fouhey和Evangelos Kazakos对项目早期提出的宝贵意见。我们也感谢Aria团队的Pierre Moulon、Vijay Baiyya和Cheng Peng在使用MPS代码和服务方面提供的技术协助。

我们感谢Elancer公司的Rajan及其团队在时间标注和音频标注方面给予的大力帮助。我们感谢Srdjan Delic及其团队在掩码标注方面的协助。我们感谢Damian Steer以及布里斯托尔大学的RDSF团队对数据集的托管和维护工作。我们还要感谢Owen Tyley使用Blender制作的厨房环境3D数字孪生模型。

我们感谢EPSRC Tier-2 Jade集群的使用。作者们也感谢Isambard-AI国家人工智能研究资源（AIRR）的使用。Isambard-AI由布里斯托尔大学运营，其资金来源于英国政府科学、创新和技术部（DSIT）通过英国研究与创新署提供的资助；以及科学和技术设施委员会[ST/AIRR/I-A-I/1023]的资助。

## 参考文献

[^1]: Project aria machine perception services. <https://facebookresearch.github.io/projectaria_tools/docs/ARK/mps.3.2>

[^2]: Blender. <https://www.blender.org/>. 5

[^3]: My fitness app. <https://www.myfitnesspal.com/>. 3,1

[^4]: VGG List Annotator (LISA), 2022. <https://www.robots.ox.ac.uk/vgg/software/lisa/>. 5

[^5]: Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florence Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyam Anadkat, et al. GPT-4 technical report. arXiv preprint arXiv:2303.08774, 2023. 2, 9

[^6]: Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a

用于小样本学习的视觉语言模型。《神经信息处理系统进展》（NeurIPS），2022年。2

[^7]: Prithviraj Banerjee, Sindi Shkodrani, Pierre Moulon, Shreyas Hampali, Fan Zhang, Jade Fountain, Edward Miller, Selen Basol, Richard Newcombe, Robert Wang, Jakob Julian Engel, and Tomas Hodan. HOT3D: Hand and Object Tracking in 3D from Egocentric Multi-View Videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025. 2

[^8]: Mu Cai, Reuben Tan, Jianrui Zhang, Bocheng Zou, Kai Zhang, Feng Yao, Fangrui Zhu, Jing Gu, Yiwu Zhong, Yuzhang Shang, Yao Dou, Jaden Park, Jianfeng Gao, Yong Jae Lee, and Jianwei Yang. TemporalBench: Towards fine-grained temporal understanding for multimodal video models. arXiv preprint arXiv:2410.10818, 2024. 2

[^9]: Jacob Chalk, Jaesung Huh, Evangelos Kazakos, Andrew Zisserman, and Dima Damen. TIM: A Time Interval Machine for Audio-Visual Action Recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 7, 8, 15

[^10]: Jr-Jen Chen, Yu-Chien Liao, Hsi-Che Lin, Yu-Chu Yu, Yen-Chun Chen, and Yu-Chiang Frank Wang. ReXTime: A benchmark suite for reasoning-across-time in videos. In Advances in Neural Information Processing Systems (NeurIPS), 2024. 2

[^11]: Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. InternVL: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 2

[^12]: Ho Kei Cheng, Seoung Wug Oh, Brian Price, Joon-Young Lee, and Alexander Schwing. Putting the object back into video object segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 8, 15

[^13]: Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, et al. VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs. arXiv preprint arXiv:2406.07476, 2024. 2, 7, 14

[^14]: Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. Qwen-Audio: Advancing universal audio understanding via unified large-scale audio-language models. arXiv preprint arXiv:2311.07919, 2023. 3

[^15]: Daniel Cores, Michael Dorkenwald, Manuel Mucientes, Cees GM Snoek, and Yuki M Asano. TVBench: Redesigning video-language evaluation. arXiv preprint arXiv:2410.07752, 2024. 2

[^16]: Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. ScanNet: Richly-annotated 3D reconstructions of indoor scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2017. 2

## [^17]: Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide

---

Moltisanti、Jonathan Munro、Toby Perrett、Will Price等。扩展以自我为中心的视觉：EPIC-KITCHENS数据集。载于《欧洲计算机视觉会议论文集》（ECCV），2018年。2、3、4

[^18]: Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Jian Ma, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. Rescaling egocentric vision: Collection, pipeline and challenges for EPIC-KITCHENS-100. International Journal of Computer Vision (IJCV), 2022. 2, 3, 4, 6, 5

[^19]: Ahmad Darkhalil, Dandan Shan, Bin Zhu, Jian Ma, Amlan Kar, Richard Higgins, Sanja Fidler, David Fouhey, and Dima Damen. EPIC-KITCHENS VISOR benchmark: Video segmentations and object relations. In Proceedings of the Neural Information Processing Systems (NeurIPS) Track on Datasets and Benchmarks, 2022. 5, 7

[^20]: Fernando De la Torre, Jessica Hodgins, Adam Bargteil, Xavier Martin, Justin Macey, Alex Collado, and Pep Beltran. Guide to the Carnegie Mellon University Multimodal activity (CMU-MMAC) database. 2009. 3

[^21]: Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The Llama 3 Herd of Models. arXiv preprint arXiv:2407.21783, 2024. 2, 7

[^22]: Abhishek Dutta and Andrew Zisserman. The VIA annotation software for images, audio and video. In Proceedings of the 27th ACM International Conference on Multimedia (ACMMM), 2019. 5, 6

[^23]: Xinyu Fang, Kangrui Mao, Haodong Duan, Xiangyu Zhao, Yining Li, Dahua Lin, and Kai Chen. MMBench-Video: A long-form multi-shot benchmark for holistic video understanding. In Advances in Neural Information Processing Systems (NeurIPS), 2024. 2

[^24]: Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. SlowFast networks for video recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019. 7, 15

[^25]: Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025. 2, 7

[^26]: Rohit Girdhar, Mannat Singh, Nikhila Ravi, Laurens van der Maaten, Armand Joulin, and Ishan Misra. Omnivore: A Single Model for Many Visual Modalities. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 7, 15

[^27]: Yuan Gong, Cheng-I Lai, Yu-An Chung, and James Glass. SSAST: Self-Supervised Audio Spectrogram Transformer. In Proceedings of the AAAI Conference on Artificial Intelligence, 2022. 8

[^28]: Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, Miguel

Martin、Tushar Nagarajan、Ilija Radosavovic、Santhosh Kumar Ramakrishnan、Fiona Ryan、Jayant Sharma、Michael Wray、Mengmeng Xu、Eric Zhongcong Xu、Chen Zhao、Siddhant Bansal、Dhruv Batra、Vincent Cartillier、Sean Crane、Tien Do、Morrie Doulaty、Akshay Erapalli、Christoph Feichtenhofer、Adriano Fragomeni、Qichen Fu、Christian Fuegen、Abraham Gebreselasie、Cristina Gonzalez、James Hillis、Xuhua Huang、Yifei Huang、Wenqi Jia、Weslie Khoo、Jachym Kolar、Satwik Kottur、Anurag Kumar、Federico Landini、Chao Li、Yanghao Li、Zhenqiang Li、Karttikeya Mangalam、Raghava Modhugu、Jonathan Munro、Tullie Murrell、Takumi Nishiyasu、Will Price、Paola Ruiz Puentes、Merey Ramazanova、Leda Sari、Kiran Somasundaram、Audrey Southerland、Yusuke Sugano、Ruijie Tao、Minh Vo、Yuchen Wang、Xindi Wu、Takuma Yagi、Yunyi Zhu、Pablo Arbelaez、David Crandall、Dima Damen、Giovanni Maria Farinella、Bernard Ghanem、Vamisi Krishna Ithapu、C. V. Jawahar、Hanbyul Joo、Kris Kitani、Haizhou Li、Richard Newcombe、Aude Oliva、Hyun Soo Park、James M. Rehg、Yoichi Sato、Jianbo Shi、Mike Zheng Shou、Antonio Toralba、Lorenzo Torresani、Mingfei Yan以及Jitendra Malik。Ego4D：基于3000小时自我中心视频的全球场景研究。发表于IEEE/CVF计算机视觉与模式识别会议（CVPR），2022年。2, 3, 4, 6

[^29]: Kristen Grauman, Andrew Westbury, Lorenzo Torresani, Kris Kitani, Jitendra Malik, Triantafyllos Afouras, Kumar Ashutosh, Vijay Baiyya, Siddhant Bansal, Bikram Boote, Eugene Byrne, Zach Chavis, Joya Chen, Feng Cheng, Fu-Jen Chu, Sean Crane, Avijit Dasgupta, Jing Dong, Maria Escobar, Cristhian Forigua, Abraham Gebreselasie, Sanjay Haresh, Jing Huang, Md Mohaiminul Islam, Suyog Jain, Rawal Khirodkar, Devansh Kukreja, Kevin J Liang, Jia-Wei Liu, Sagnik Majumder, Yongsen Mao, Miguel Martin, Effrosyni Mavroudi, Tushar Nagarajan, Francesco Ragusa, Santhosh Kumar Ramakrishnan, Luigi Seminara, Arjun Somayazulu, Yale Song, Shan Su, Zihui Xue, Edward Zhang, Jinxu Zhang, Angela Castillo, Changan Chen, Xinzhu Fu, Ryosuke Furuta, Cristina Gonzalez, Prince Gupta, Jiabo Hu, Yifei Huang, Yiming Huang, Weslie Khoo, Anush Kumar, Robert Kuo, Sach Lakhavani, Miao Liu, Mi Luo, Zhengyi Luo, Brighid Meredith, Austin Miller, Oluwatuminu Oguntola, Xiaqing Pan, Penny Peng, Shraman Pramanick, Merey Ramazanova, Fiona Ryan, Wei Shan, Kiran Somasundaram, Chenan Song, Audrey Southerland, Masatoshi Tateno, Huiyu Wang, Yuchen Wang, Takuma Yagi, Mingfei Yan, Xitong Yang, Zecheng Yu, Shengxin Cindy Zha, Chen Zhao, Ziwei Zhao, Zhifan Zhu, Jeff Zhuo, Pablo Arbelaez, Gedas Bertasius, Dima Damen, Jakob Engel, Giovanni Maria Farinella, Antonino Furnari, Bernard Ghanem, Judy Hoffman, C.V. Jawahar, Richard Newcombe, Hyun Soo Park, James M. Rehg, Yoichi Sato, Manolis Savva, Jianbo Shi, Mike Zheng Shou, and Michael Wray. Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 2, 3, 5, 6

## [^30]: Matthew Honnibal and Ines Montani. spaCy 2: Natural language understanding with Bloom embeddings, convolutional

---

神经网络与增量解析。2020年。5, 10

[^31]: Jaesung Huh, Jacob Chalk, Evangelos Kazakos, Dima Damen, and Andrew Zisserman. EPIC-SOUNDS: A Large-Scale Dataset of Actions that Sound. In IEEE International Conference on Acoustics, Speech, & Signal Processing (ICASSP), 2023. 4, 5

[^32]: Roland Johansson, Göran Westling, Anders Bäckström, and John Flanagan. Eye-Hand Coordination in Object Manipulation. The Journal of neuroscience: the official journal of the Society for Neuroscience, 2001. 5

[^33]: Amlan Kar, Seung Wook Kim, Marko Boben, Jun Gao, Tianxing Li, Huan Ling, Zian Wang, and Sanja Fidler. Toronto annotation suite. <https://aidemos.cs.toronto.edu/toras>, 2021.7

[^34]: Evangelos Kazakos, Arsha Nagrani, Andrew Zisserman, and Dima Damen. Slow-Fast Auditory Streams For Audio Recognition. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2021. 8

[^35]: Ilker Kesen, Andrea Pedrotti, Mustafa Dogan, Michele Cafagna, Emre Can Acikgoz, Letitia Parcalabescu, Iacer Calixto, Anette Frank, Albert Gatt, Aykut Erdem, and Erkut Erdem. ViLMA: A Zero-Shot Benchmark for Linguistic and Temporal Grounding in Video-Language Models. In International Conference on Learning Representations (ICLR), 2024. 2

[^36]: Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023. 5

[^37]: Nikita Kitaev and Dan Klein. Constituency parsing with a self-attentive encoder. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2018. 5

[^38]: Oleksii Kuchaiev, Jason Li, Huyen Nguyen, Oleksii Hrinchuk, Ryan Leary, Boris Ginsburg, Samuel Kriman, Stanislav Beliaev, Vitaly Lavrukhin, Jack Cook, Patrice Castonguay, Mariya Popova, Jocelyn Huang, and Jonathan M. Cohen. NeMo: a toolkit for building AI applications using Neural Modules. arXiv preprint arXiv:1909.09577, 2019. 3

[^39]: Bolin Lai, Xiaoliang Dai, Lawrence Chen, Guan Pang, James M Rehg, and Miao Liu. LEGO: Learning EGOcentric Action Frame Generation via Visual Instruction Tuning. In Proceedings of the European Conference on Computer Vision (ECCV), 2023. 3

[^40]: Michael Land, Neil Mennie, and Jennifer Rusted. The roles of vision and eye movements in the control of activities of daily living. Perception, 1999. 5

[^41]: Yong Jae Lee, Joydeep Ghosh, and Kristen Grauman. Discovering important people and objects for egocentric video summarization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2012. 3

[^42]: Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. SEED-Bench: Benchmarking Multimodal Large Language Models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 2

[^43]: Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. MVBench: A Comprehensive Multi-modal Video Understanding Benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.

[^44]: Shicheng Li, Lei Li, Shuhuai Ren, Yuanxin Liu, Yi Liu, Rundong Gao, Xu Sun, and Lu Hou. VITATECS: A Diagnostic Dataset for Temporal Concept Understanding of Video-Language Models. In Proceedings of the European Conference on Computer Vision (ECCV), 2024. 2

[^45]: Yin Li, Miao Liu, and James M Rehg. In the eye of the beholder: Gaze and actions in first person video. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2021. 3

[^46]: Yunze Liu, Yun Liu, Che Jiang, Kangbo Lyu, Weikang Wan, Hao Shen, Boqiang Liang, Zhoujie Fu, He Wang, and Li Yi. HOI4D: A 4D Egocentric Dataset for Category-Level Human-Object Interaction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 6, 3

[^47]: Zhaoyang Lv, Nickolas Charron, Pierre Moulon, Alexander Gamino, Cheng Peng, Chris Sweeney, Edward Miller, Huixuan Tang, Jeff Meissner, Jing Dong, et al. Aria Everyday Activities Dataset. arXiv preprint arXiv:2402.13349, 2024. 6, 3

[^48]: Arjun Majumdar, Anurag Ajay, Xiaohan Zhang, Pranav Putta, Sriram Yenamandra, Mikael Henaff, Sneha Silwal, Paul Mcvay, Oleksandr Maksymets, Sergio Arnaud, Karmesh Yadav, Qiyang Li, Ben Newman, Mohit Sharma, Vincent Berges, Shiqi Zhang, Pulkit Agrawal, Yonatan Bisk, Dhruv Batra, Mrinal Kalakrishnan, Franziska Meier, Chris Paxton, Sasha Sax, and Aravind Rajeswaran. OpenEQA: Embodied Question Answering in the Era of Foundation Models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 2

[^49]: Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding. Advances in Neural Information Processing Systems (NeurIPS), 2023. 2

[^50]: Xiaqing Pan, Nicholas Charron, Yongqian Yang, Scott Peters, Thomas Whelan, Chen Kong, Omkar Parkhi, Richard Newcombe, and Yuheng Carl Ren. Aria Digital Twin: A New Benchmark Dataset for Egocentric 3D Machine Perception. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023. 2, 6, 3

[^51]: Mandela Patrick, Dylan Campbell, Yuki M. Asano, Ishan Misra Florian Metze, Christoph Feichtenhofer, Andrea Vedaldi, and João F. Henriques. Keeping your eye on the ball: Trajectory attention in video transformers. In Advances in Neural Information Processing Systems (NeurIPS), 2021. 7, 15

## [^52]: Rohith Peddi, Shivvrat Arya, Bharath Challa, Likhitha Pallapothula, Akshay Vyas, Bhavya Gouripeddi, Jikai Wang, Qifan Zhang, Vasundhara Komaragiri, Eric Ragan, Nicholas Ruozzi, Yu Xiang, and Vibhav Gogate. CaptainCook4D: A Dataset for Understanding Errors in Procedural Activities. In

---

神经信息处理系统会议（NeurIPS）数据集与基准测试专题论文集，2024年。3

[^53]: Federico Perazzi, Jordi Pont-Tuset, Brian McWilliams, Luc Van Gool, Markus Gross, and Alexander Sorkine-Hornung. A benchmark dataset and evaluation methodology for video object segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2016. 15

[^54]: Toby Perrett, Tengda Han, Dima Damen, and Andrew Zisserman. It's just another day: Unique video captioning by discriminative prompting. In Proceedings of the Asian Conference on Computer Vision (ACCV), 2024. 2

[^55]: Hamed Pirsiavash and Deva Ramanan. Detecting activities of daily living in first-person camera views. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2012. 3

[^56]: Chiara Plizzari, Shubham Goel, Toby Perrett, Jacob Chalk, Angjoo Kanazawa, and Dima Damen. Spatial Cognition from Egocentric Video: Out of Sight, Not Out of Mind. In Proceedings of the IEEE International Conference on 3D Vision (3DV), 2025. 7

[^57]: Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alexander Sorkine-Hornung, and Luc Van Gool. The 2017 davis challenge on video object segmentation. arXiv:1704.00675, 2017. 15

[^58]: Viorica Pătrăucean, Lucas Smaira, Ankush Gupta, Adrià Recasens Continente, Larisa Markeeva, Dylan Banarse, Skanda Koppula, Joseph Heyward, Mateusz Malinowski, Yi Yang, Carl Doersch, Tatiana Matejovicova, Yury Sulsky, Antoine Miech, Alex Frechette, Hanna Klimczak, Raphael Koster, Junlin Zhang, Stephanie Winkler, Yusuf Aytar, Simon Osindero, Dima Damen, Andrew Zisserman, and João Carreira. Perception test: A diagnostic benchmark for multimodal video models. In Advances in Neural Information Processing Systems (NeurIPS), 2023. 2

[^59]: Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML), 2021. 2

[^60]: Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In International Conference on Machine Learning (ICML), 2023. 3

[^61]: Francesco Ragusa, Antonino Furnari, Sebastiano Battiato, Giovanni Signorello, and Giovanni Maria Farinella. EGOCH: Dataset and fundamental tasks for visitors behavioral understanding using egocentric vision. Pattern Recognition Letters, 2020. 3

[^62]: Francesco Ragusa, Antonino Furnari, Salvatore Livatino, and Giovanni Maria Farinella. The MECCANO Dataset: Understanding Human-Object Interactions from Egocentric Videos in an Industrial-like Domain. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2021. 3

[^63]: Santhosh K Ramakrishnan, Aaron Gokaslan, Erik Wijmans, Oleksandr Maksymets, Alex Clegg, John Turner, Eric Un-

Dersander、Wojciech Galuba、Andrew Westbury、Angel X Chang等人。Habitat-Matterport 3D数据集（HM3D）：面向具身人工智能的1000个大规模3D环境。发表于神经信息处理系统会议（NeurIPS）数据集与基准测试专题，2021年。2

[^64]: Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et al. SAM 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00714, 2024. 5, 8, 7, 15

[^65]: Johannes Lutz Schönberger and Jan-Michael Frahm. Structure-from-motion revisited. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2016. 2

[^66]: Johannes Lutz Schönberger, Enliang Zheng, Marc Pollefeys, and Jan-Michael Frahm. Pixelwise view selection for unstructured multi-view stereo. In Proceedings of the European Conference on Computer Vision (ECCV), 2016. 2

[^67]: Tim J Schoonbeek, Tim Houben, Hans Onvlee, Fons van der Sommen, et al. IndustReal: A Dataset for Procedure Step Recognition Handling Execution Errors in Egocentric Videos in an Industrial-Like Setting. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2024. 3

[^68]: Fadime Sener, Dibyadip Chatterjee, Daniel Shelepov, Kun He, Dipika Singhania, Robert Wang, and Angela Yao. Assembly101: A large-scale multi-view video dataset for understanding procedural activities. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 2, 6, 3

[^69]: Gunnar A Sigurdsson, Abhinav Gupta, Cordelia Schmid, Ali Farhadi, and Karteek Alahari. Actor and observer: Joint modeling of first and third-person videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018. 3

[^70]: Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela. FLAVA: A foundational language and vision alignment model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 2

[^71]: Krishna Kumar Singh, Kayvon Fatahalian, and Alexei Afros. Krishnacam: Using a longitudinal, single-person, egocentric dataset for scene understanding tasks. In Proceedings of the IEEE Winter Conference on Applications of Computer Vision (WACV), 2016. 3

[^72]: Kiran K. Somasundaram, Jing Dong, Huixuan Tang, Julian Straub, Mingfei Yan, Michael Goesele, Jakob J. Engel, Renzo De Nardi, and Richard A. Newcombe. Project Aria: A New Tool for Egocentric Multi-Modal AI Research. arXiv preprint arXiv:2308.13561, 2023. 3, 1

[^73]: Yale Song, Eugene Byrne, Tushar Nagarajan, Huiyu Wang, Miguel Martin, and Lorenzo Torresani. Ego4D Goal-Step: Toward Hierarchical Understanding of Procedural Activities. Advances in Neural Information Processing Systems (NeurIPS), 2024. 3

## [^74]: Julian Straub, Daniel DeTone, Tianwei Shen, Nan Yang, Chris Sweeney, and Richard Newcombe. EFM3D: A Bench-

---

用于衡量3D自我中心基础模型进展的基准。arXiv预印本arXiv:2406.10224，2024年。6, 3

[^75]: Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024. 7, 14

[^76]: Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. VideoMAE: Masked autoencoders are data-efficient learners for self-supervised video pre-training. In Advances in Neural Information Processing Systems (NeurIPS), 2022. 7, 15

[^77]: Vadim Tschernezki, Ahmad Darkhalil, Zhifan Zhu, David Fouhey, Iro Larina, Diane Larlus, Dima Damen, and Andrea Vedaldi. EPIC Fields: Marrying 3D Geometry and Video Understanding. In Advances in Neural Information Processing Systems (NeurIPS), 2023. 2

[^78]: Andong Wang, Bo Wu, Sunli Chen, Zhenfang Chen, Haotian Guan, Wei-Ning Lee, Li Erran Li, and Chuang Gan. SOK-Bench: A Situated Video Reasoning Benchmark with Aligned Open-World Knowledge. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 2

[^79]: Wenhai Wang, Jifeng Dai, Zhe Chen, Zhenhang Huang, Zhiqi Li, Xizhou Zhu, Xiaowei Hu, Tong Lu, Lewei Lu, Hongsheng Li, et al. InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023. 2

[^80]: Xin Wang, Taein Kwon, Mahdi Rad, Bowen Pan, Ishani Chakraborty, Sean Andrist, Dan Bohus, Ashley Feniello, Bugra Tekin, Felipe Vieira Frujeri, Neel Joshi, and Marc Pollefeys. HoloAssist: an Egocentric Human Interaction Dataset for Interactive AI Assistants in the Real World. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023. 2, 6, 3

[^81]: Bin Xiao, Haiping Wu, Weijian Xu, Xiyang Dai, Houdong Hu, Yumao Lu, Michael Zeng, Ce Liu, and Lu Yuan. Florence-2: Advancing a unified representation for a variety of vision tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 2

[^82]: Junbin Xiao, Angela Yao, Yicong Li, and Tat-Seng Chua. Can I trust your answer? Visually grounded video question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 7

[^83]: Ning Xu, Linjie Yang, Yuchen Fan, Dingcheng Yue, Yuchen Liang, Jianchao Yang, and Thomas Huang. YouTube-VOS: A large-scale video object segmentation benchmark. arXiv preprint arXiv:1809.03327, 2018. 15

[^84]: Linjie Yang, Yuchen Fan, and Ning Xu. The 2nd large-scale video object segmentation challenge - video object segmentation track, 2019. 15

[^85]: Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 5

[^86]: Hanrong Ye, Haotian Zhang, Erik Daxberger, Lin Chen, Zongyu Lin, Yanghao Li, Bowen Zhang, Haoxuan You, Dan Xu, Zhe Gan, et al. MM-Ego: Towards Building Egocentric Multimodal LLMs. In International Conference on Learning Representations (ICLR), 2025. 2

[^87]: Brent Yi, Vickie Ye, Maya Zheng, Lea Müller, Georgios Pavlakos, Yi Ma, Jitendra Malik, and Angjoo Kanazawa. Estimating body and hand motion in an ego-sensed world. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025. 5

[^88]: Hang Zhang, Xin Li, and Lidong Bing. Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding. Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023. 2

[^89]: Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. Long context transfer from language to vision. arXiv preprint arXiv:2406.16852, 2024. 7, 14

[^90]: Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziewei Liu, and Chunyuan Li. Video instruction tuning with synthetic data. arXiv preprint arXiv:2410.02713, 2024. 7, 14

[^91]: Yunhan Zhao, Haoyu Ma, Shu Kong, and Charless Fowlkes. Instance tracking in 3D scenes from egocentric videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 2, 3

[^92]: Zijia Zhao, Haoyu Lu, Yuqi Huo, Yifan Du, Tongtian Yue, Longteng Guo, Bingning Wang, Weipeng Chen, and Jing Liu. Needle In A Video Haystack: A Scalable Synthetic Framework for Benchmarking Video MLLMs. In International Conference on Learning Representations (ICLR), 2025. 2

[^93]: Luowei Zhou, Chenliang Xu, and Jason Corso. Towards automatic learning of procedures from web instructional videos. In Proceedings of the AAAI Conference on Artificial Intelligence, 2018. 2