{
  "paper_id": "13d2c8ca5b88e638a33531f2b8b223e4",
  "paper_title": "EgoInfinity: A Web-Scale 4D Hand-Object Interaction Data Engine for Any-View Robot Retargeting and Video-to-Action Robot Learning",
  "paper_authors": [
    "Gaotian Wang",
    "Kejia Ren",
    "Yiting Chen",
    "Andrew Morgan",
    "Howard H. Qian",
    "Podshara Chanrungmaneekul",
    "Kaiyu Hang"
  ],
  "publication_date": "",
  "publication_venue": "unknown",
  "abstract": "Internet videos constitute the largest reservoir of embodied human manipulation knowledge, yet converting arbitrary RGB footage into actionable robot training data remains a major bottleneck. Existing lab- or factory-collected datasets are narrow in scale and diversity, limiting their ability to support open-world robot learning. Instead of proposing a static dataset, we introduce EgoInfinity, a universal 4D hand-object interaction data engine that enables web-scale data generation for robot retargeting and learning. EgoInfinity is built as a modular engine that integrates perception, segmentation, reconstruction, interaction-aware refinement, and retargeting components to automate this traditionally unscalable video-to-action problem without human-in-the-loop annotation. Its modular design allows the engine to continuously benefit from advances in any incorporated component. With EgoInfinity, in-the-wild human manipulation videos are lifted into agent-agnostic, metric 4D hand-object interaction representations, including hand trajectories, 6-DoF object poses, and contact-relevant states. Rather than naively connecting standalone components, EgoInfinity combines cross-module metric calibration with interaction-aware refinement to improve physical reliability, reducing drift and contact inconsistencies common in pure visual reconstruction. In addition, we propose a novel motion retargeter to compile the recovered 3D hand motions into executable joint trajectories for diverse robot morphologies, enabling video-to-action retargeting on any robot from arbitrary viewpoints and shot sizes (e.g., the human body is only partially visible). We validate EgoInfinity across perception fidelity, kinematic feasibility, contact consistency, cross-embodiment generalization, and real-robot skill acquisition (e.g., grasping, cutting, wiping, and pouring), demonstrating a scalable bridge from internet videos to executable robot behavior for robust open-world robot learning.",
  "keywords": [
    "Automatic Data Generation",
    "Benchmarks and Datasets for Robot Learning",
    "Learning from Human Videos",
    "Cross-Embodiment Retargeting"
  ],
  "paper_institutions": [
    "Rice University",
    "Robotics and AI Institute"
  ],
  "output_language": "zh",
  "provider": "doubao",
  "model": "glm-5.3",
  "prompt_template": "simple",
  "extracted_at": "2026-09-17T09:37:41.217058Z",
  "template_tag": "simple",
  "summary": "### 摘要\n\n互联网视频是具身人类操作知识的最大储备库，但将任意RGB视频转化为可执行的机器人训练数据仍是重大瓶颈；现有实验室或工厂采集的数据集（如Ego4D、EgoDex、HOT3D、OakInk2、DROID、Open X-Embodiment）受限于规模、任务多样性与采集成本，且依赖穿戴设备、动捕或人工标注，而现有从人类视频学习的方法多停留在2D伪动作层面，缺乏度量3D几何与6-DoF物体状态。为此，论文提出EgoInfinity，一个完全自动化的通用4D手-物交互数据引擎，通过模块化流水线（WILOR手部网格估计、MOGE-2度量几何、FLOW3R稠密深度、GEOCALIB重力估计、SAM-3/SAM-2目标物体发现与分割跟踪、SAM-3D物体重建、FOUNDATIONPOSE++位姿跟踪）将野外RGB视频提升为与智能体无关的度量4D表示（手轨迹、6-DoF物体位姿、接触相关状态），无需人工标注，且各模块可随基础模型进步独立升级。其核心机制是交互感知精化：基于MEMFOF光流与手部关键点将每帧分类为static/grasped/moving状态，被抓取物体通过刚性手绑定 $^{c}\\hat{\\mathbf{p}}_t^o = {}^{c}\\mathbf{p}_t^h \\cdot \\mathbf{T}^{\\text{cano}}$ 刚性附着于手掌对齐的手坐标系，静态物体锁定于鲁棒点云质心，从而抑制纯视觉重建中的漂移与接触不一致。数据方面，引擎在Action100M（147M动作片段、14.6年视频）上以两遍策略（轻量扫描筛选含手片段后仅对活跃片段运行完整重建栈）进行网络规模处理，当前策划子集含106个已处理操作视频（规模上限约127K小时），并通过浏览器数据服务器提供交互式检索与可视化。跨本体重定向采用SE(3)等变Vector-Neuron流匹配网络在MuJoCo仿真中程序化生成配对轨迹训练（单卡RTX 3060约1.5-2小时/机器人，无需真实世界监督），在Unitree G1、Robonaut2与双臂Franka FR3三种差异显著本体上评估：IK成功率分别为0.821/0.774/0.706，位置误差2.86/6.67/10.27 cm，并报告姿态误差、关节限位裕度、可操作性与平滑度，验证了度量校准与交互感知精化对物理可靠性的贡献；真机实验中，提取的手部运动作为先验在LEAP灵巧手上训练抓取策略（泛化到苹果、香蕉、番茄罐等不同形状物体），并直接重定向到双臂Franka FR3执行切、倒、擦等技能，完整打通视频到动作链路。未来方向包括引入SLAM感知深度模型以放宽近似静态相机假设、实现接触级精确对齐与引入触觉观测；现实约束包括假设近似静态相机、不保证指尖级接触精度与力一致性、无触觉反馈、重定向器需针对新机器人重新训练或校准。\n\n### 关键要点\n\n- EgoInfinity是完全自动化的模块化4D操作数据引擎，将野外互联网RGB视频转换为与智能体无关的度量手-物表示，无需穿戴设备、深度传感器、CAD模型或人工指定的物体标注。\n- 模块化流水线集成WILOR手部重建、MOGE-2度量几何、FLOW3R深度、GEOCALIB重力估计、SAM-3/SAM-2分割跟踪、SAM-3D物体重建与FOUNDATIONPOSE++位姿跟踪，各模块可独立升级，引擎持续受益于组件进步。\n- 交互感知精化机制基于MEMFOF光流与手部关键点将每帧分类为static/grasped/moving状态：被抓取物体通过刚性手绑定 $^{c}\\hat{\\mathbf{p}}_t^o = {}^{c}\\mathbf{p}_t^h \\cdot \\mathbf{T}^{\\text{cano}}$ 附着于手，静态物体锁定鲁棒质心，抑制纯视觉跟踪的漂移与遮挡失败。\n- 采用两遍可扩展策略：第一遍轻量时序扫描利用手部运动统计与相机运动线索筛选含手片段，第二遍仅在活跃片段上运行完整重建栈，使网络规模处理可行。\n- 数据来源为Action100M（147M动作片段、14.6年视频），当前策划子集含106个已处理操作视频，规模上限约127K小时，引擎本身与语料库无关；语义提示直接取自Action100M的层级标注，处理时无需额外VLM。\n- 外观到自我视角转换通过在恢复的3D空间中刚体重构帧实现：虚拟相机置于双手中点上方、上轴对齐GeoCalib重力向量并指向交互区域，避免生成式视频翻译的像素幻觉并保持度量几何与接触一致。\n- 跨本体重定向器采用SE(3)等变Vector-Neuron流匹配网络估计机器人特定根坐标系，在MuJoCo仿真中程序化生成配对轨迹训练（RTX 3060、500 epochs、约1.5-2小时/机器人），推理时以k-means聚类根帧假设并通过IK收敛率、残差误差、可操作性、关节限位裕度与平滑度评分选优。\n- 训练采用多种增强模拟野外重建误差：位置噪声σ=0.01 m、跟踪跳变（概率0.20、最大0.15 m）、手部遮挡（概率0.20）、重力噪声（最大0.10 rad）与30%重力丢弃，使网络在重力缺失时优雅退化。\n- 重定向评估在三种差异显著本体上进行：Unitree G1（IK成功率0.821、位置误差2.86 cm、姿态误差6.73°）、Robonaut2（0.774、6.67 cm、8.25°）、双臂Franka FR3（0.706、10.27 cm、12.17°），并报告关节限位裕度、可操作性与平滑度等数据集级统计。\n- 真机验证包括：以提取的手部运动为先验在LEAP灵巧手上训练抓取策略并泛化到苹果、香蕉、番茄罐等不同形状物体；将提取运动直接重定向到双臂Franka FR3执行切、倒、擦等技能，验证完整视频到动作管线。\n- 提供浏览器数据服务器，支持自然语言关键词搜索、在线处理与可视化（MANO手网格轨迹、6-DoF物体轨迹、重建几何、点云、深度图、光流、抓取事件），使输出可直接检查用于失败诊断并作为交互式策划工具。\n- 局限性与约束：假设近似静态相机视频（排除身体佩戴或手持拍摄）、不保证接触级精度（如精确指尖放置、力一致性、无滑移约束）、不提供触觉观测、重定向器针对特定机器人需重训练或校准，且面向功能性运动迁移而非精细运动学模仿。"
}