{ "paper_id": "e465fdbfa4de1f958b1b8804edcf4b9c", "paper_title": "Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives", "paper_authors": [ "Kristen Grauman", "Andrew Westbury", "Lorenzo Torresani", "Kris Kitani", "Jitendra Malik", "Triantafyllos Afouras", "Kumar Ashutosh", "Vijay Baiyya", "Siddhant Bansal", "Bikram Boote", "Eugene Byrne", "Zach Chavis", "Joya Chen", "Feng Cheng", "Fu-Jen Chu", "Sean Crane", "Avijit Dasgupta", "Jing Dong", "Maria Escobar", "Cristhian Forigua", "Abrham Gebreselasie", "Sanjay Haresh", "Jing Huang", "Md Mohaiminul Islam", "Suyog Jain", "Rawal Khirodkar", "Devansh Kukreja", "Kevin J. Liang", "Jia-Wei Liu", "Sagnik Majumder", "Yongsen Mao", "Miguel Martin", "Effrosyni Mavroudi", "Tushar Nagarajan", "Francesco Ragusa", "Santhosh Kumar Ramakrishnan", "Luigi Seminara", "Arjun Somayazulu", "Yale Song", "Shan Su", "Zihui Xue", "Edward Zhang", "Jinxu Zhang", "Angela Castillo", "Changan Chen", "Xinzhu Fu", "Ryosuke Furuta", "Cristina González", "Prince Gupta", "Jiabo Hu", "Yifei Huang", "Yiming Huang", "Weslie Khoo", "Anush Kumar", "Robert Kuo", "Sach Lakhavani", "Miao Liu", "Mi Luo", "Zhengyi Luo", "Brighid Meredith", "Austin Miller", "Oluwatumininu Oguntola", "Xiaqing Pan", "Penny Peng", "Shraman Pramanick", "Merey Ramazanova", "Fiona Ryan", "Wei Shan", "Kiran Somasundaram", "Chenan Song", "Audrey Southerland", "Masatoshi Tateno", "Huiyu Wang", "Yuchen Wang", "Takuma Yagi", "Mingfei Yan", "Xitong Yang", "Zecheng Yu", "Shengxin Cindy Zha", "Chen Zhao", "Ziwei Zhao", "Zhifan Zhu", "Jeff Zhuo", "Pablo Arbeláez", "Gedas Bertasius", "David Crandall", "Dima Damen", "Jakob Engel", "Giovanni Maria Farinella", "Antonino Furnari", "Bernard Ghanem", "Judy Hoffman", "C. V. Jawahar", "Richard Newcombe", "Hyun Soo Park", "James M. Rehg", "Yoichi Sato", "Manolis Savva", "Jianbo Shi", "Mike Zheng Shou", "Michael Wray" ], "publication_date": "2025-11-24", "publication_venue": "International Journal of Computer Vision", "abstract": "We present Ego-Exo4D, a diverse, large-scale multimodal multiview video dataset and benchmark challenge. Ego-Exo4D centers around simultaneously-captured egocentric and exocentric video of skilled human activities (e.g., sports, music, dance, bike repair). 740 participants from 13 cities worldwide performed these activities in 123 different natural scene contexts, yielding long-form captures from 1 to 42 minutes each and 1,286 hours of video combined. The multimodal nature of the dataset is unprecedented: the video is accompanied by multichannel audio, eye gaze, 3D point clouds, camera poses, IMU, and multiple paired language descriptions—including a novel “expert commentary” done by coaches and teachers and tailored to the skilled-activity domain. To push the frontier of first-person video understanding of skilled human activity, we also present a suite of benchmark tasks and their annotations, including fine-grained activity understanding, proficiency estimation, cross-view translation, and 3D hand/body pose. All resources are open sourced to fuel new research in the community. \n", "keywords": [ "Video understanding", "First-person video", "Egocentric", "Video-language", "3D", "Body pose" ], "paper_institutions": [ "IIIT, Hyderabad, Hyderabad, India", "University of Illinois, Urbana Champaign, Urbana and Champaign, United States", "Carnegie Mellon University, Pittsburgh, United States", "University of Minnesota, Minneapolis, United States", "National University of Singapore, Singapore, Singapore", "Universidad de los Andes, Bogotá, Colombia", "Simon Fraser University, Burnaby, Canada", "University of North Carolina, Chapel Hill, United States", "University of Catania, Catania, Italy", "University of Pennsylvania, Philadelphia, United States", "University of Tokyo, Bunkyo City, Japan", "Meta, Menlo Park, California, United States", "Indiana University, Bloomington, United States", "Johns Hopkins University, Baltimore, United States", "King Abdullah University of Science and Technology, Thuwal, Kingdom of Saudi Arabia", "Georgia Tech, Atlanta, United States", "UT Austin, United States", "University of Bristol, United Kingdom" ], "output_language": "zh", "provider": "doubao", "model": "kimi-k2.8-preview", "prompt_template": "simple", "extracted_at": "2026-10-20T05:20:53.668647Z", "template_tag": "simple", "summary": "### 摘要\n\nEgo-Exo4D 要解决的核心问题是让 AI 真正理解人类的技能性活动（如体育、音乐、舞蹈、烹饪、自行车维修），以支撑 AR 实时虚拟教练、机器人技能习得、社交网络等新应用；认知科学表明观察者（第三人称 exo）与执行者（第一人称 ego）之间的视角转换是视觉学习的基础，但现有 ego-exo 数据集数量少、规模小、相机不同步或过于摆拍，致使研究只能分别处理单一视角，无法流畅地在第一与第三人称之间迁移。为此，15 家研究机构历时近两年构建了首个大规模多模态多视角视频数据集与基准：740 名来自全球 13 个城市 123 个真实场景的参与者，在厨房、球场、攀岩馆、车店等自然环境中非剧本化地完成 8 大领域 43 项活动（程序性与身体性技能），产出 5,035 个实例、1,286 小时时间同步视频；采集采用低成本可复制的 1 副 Aria 眼镜（RGB、双目 SLAM、眼动追踪、7 麦克风、双 IMU、气压计、磁力计）加 4–5 台 GoPro 的机位方案，所有视图经 QR 码同步（误差 1 帧内）并在度量、重力对齐坐标系中实现 6DoF 定位。数据集还包含三种时间对齐的配对语言资源——由 52 位持证教练/教师给出的 117,812 条“专家点评”（关注动作“如何做”而非“做什么”）、参与者自述的 narrate-and-act、以及 432K 句原子动作描述；并定义了关系（对应、跨视角翻译）、识别（细粒度 keystep、能效多模态 keystep、程序理解）、熟练度估计、ego 姿态四大类基准，标注总投入超过 20 万小时。其核心机制之一是能耗受限在线识别任务中的统一能耗模型 E(O^t)=α·C(O^t)+β·M(O^t)+Σ_j γ_j·1(S_j∈O^t)，将前向计算、内存搬运与传感器触发成本折算为功率预算（如 20mW 高能效档）。基线结果验证了数据集的挑战性：对应任务最佳 IoU 约 34.9%（ego→exo），时序模型显著优于单帧空间模型；keystep 识别中视图不变编码器取得 41.53% 测试 top-1，优于直接融合；高能效档最佳音视模型 mcAP 77.89%（19.70mW），高性能档 LaViLa 达 93.24%；熟练度估计单视图最高约 50.4%，且 ego 视角利于手-物交互场景、exo 视角利于全身姿态场景；身体姿态定位基线 MPJPE 18.51cm，手部 POTTER 的 PA-MPJPE 为 11.07mm，而自动生成的伪标签（身体 3.33cm、手 1.87cm）已优于全部基线。局限与真实约束包括：仅单人 focal participant 无法研究多人交互、活动时长呈长尾分布、固定 exo 机位常被人体遮挡；未来方向涵盖视图不变/蒸馏表征学习、AI 教练与个性化反馈、机器人从演示学习、通用 3D 视觉（环境重建、相机重定位）与视频-语言预训练。\n\n### 关键要点\n\n- 数据集规模为 740 名参与者、123 个场景、13 个城市、5,035 个 take、1,286 小时 ego+exo 同步视频，覆盖 8 大领域 43 项活动与 689 个 keystep，参与者年龄 18–74 岁、技能从新手到专家，采集遵循严格的隐私伦理审查与知情同意。\n- 采集装置为 1 副 Aria 眼镜加 4–5 台 GoPro 的低成本机位（不含 Aria/手机/电脑低于 3,000 美元），通过 29fps QR 码视频实现亚帧级时间同步（GoPro 约 ±0.558ms），并利用 Aria MPS 获得毫米级 6DoF 轨迹、眼动、3D 点云及 GoPro 定位。\n- 三种时间对齐语言资源互补：专家点评（52 位专家、117,812 条、含空间标注与 1–10 分熟练度评分）、narrate-and-act（约 10% 的 take、参与者边做边解释原因）、原子动作描述（432K 句、每秒约 31 句并带可见性/最佳 exo 视角标签），可支撑动作定位、预训练与自动字幕研究。\n- ego-exo 对应任务在 1,335 对视频上标注了 5,566 个物体、180 万对分割掩码；基线 XSegTx（空间）与 XView-XMem（时空）显示时序信息将 Ego→Exo 的 IoU 从 13.88% 提升至 22.14%，最佳组合达 34.90%，说明任务因小物体与严重遮挡而极具难度。\n- 跨视角翻译被拆分为 ego 轨迹预测与 ego 片段生成：pix2pix-mask/GNT-mask 的可见性预测仅约随机水平（50%），而扩散模型 DiT-pix 在片段生成上全面优于 GAN（SSIM 0.59、LPIPS 0.46、CLIP 81.9），且裁剪输入与 clip-to-clip 变体对一致性至关重要。\n- 细粒度 keystep 识别包含 143,442 个片段、17 项活动、664 个 keystep；视图不变（VI）编码器取得最佳 41.53% 测试 top-1，视点蒸馏 39.49%、EgoVLPv2 预训练 38.76%，而简单多视角融合反而退化，凸显显式跨视角学习机制的价值。\n- 能效多模态 keystep 识别是在线流式任务，能耗模型 E(O^t)=α·C(O^t)+β·M(O^t)+Σ_j γ_j·1(S_j∈O^t) 统一计算、内存与传感器成本；20mW 档最佳 AV-LF 达 77.89 mcAP（19.70mW），2.8W 档 LaViLa 达 93.24 mcAP，视听互补且视觉最关键。\n- 程序理解以含 OR/XOR/可选/可重复节点的任务图编码 keystep 依赖，在弱监督下预测前序、可选、错误、缺失与后续 keystep；图方法在实例级监督下前序 keystep cAP 达 82.49%，但未来 keystep 预测仍接近均匀基线，是开放难题。\n- 熟练度估计含演示者四级分类（TimeSFormer+EgoVLPv2 ego 测试 50.4%，多次演示融合 49.1–51.0%）与演示级时序定位（ActionFormer 平均 mAP 约 4.04）；ego 视角利于烹饪等手-物交互，exo 视角利于攀岩等全身姿态场景。\n- ego 姿态基准提供文献中最大规模人工标注（身体 376K 3D/2M 2D、手 68K 3D/340K 2D）加自动伪标签（约 1,400 万帧，身体 MPJPE 3.33cm、手 1.87cm）；身体姿态最佳基线 MPJPE 18.51cm，手部 POTTER 为 28.94mm MPJPE / 11.07mm PA-MPJPE。\n- 统一数据划分按活动类型与熟练度分层（3,082 训练 / 842 验证 / 1,121 测试），同一参与者的所有 take 只归入一个 split 以防泄漏；所有数据、标注、采集协议与基线均开源，并以许可门禁管理使用。" }