{
  "paper_id": "67cfac2630fa42cc766027d63e9d3466",
  "paper_title": "MINT: A Unified Model for World-Space Camera and Hand Motion Estimation from Scalable Egocentric Pipeline Supervision",
  "paper_authors": [
    "Zijie Zhu",
    "Weiren Cai",
    "Yizhou Wang",
    "Zhenjie Yang",
    "Yide Liu",
    "Jiahao Chen",
    "Guanqi He"
  ],
  "publication_date": "",
  "publication_venue": "unknown",
  "abstract": "Recovering camera and hand motion in world coordinates from egocentric video is a key capability for activity understanding, robot learning, and augmented reality. Existing systems typically decompose this problem into separate stages for camera motion, depth estimation, hand reconstruction, and trajectory refinement, resulting in substantial computational overhead and preventing the joint modeling of camera and hand motion. We introduce MINT (Minting IN-the-Wild Trajectories), a foundation model for world-space hand motion reconstruction from ego-centric RGB video. From a single shared spatiotemporal video representation, MINT jointly predicts the camera trajectory, field of view (FoV), camera-frame hand states, and per-frame hand observability, and then produces world-space hand motion via explicit coordinate transformations. Training such a model at scale is challenging, since paired world-space camera and hand annotations are scarce. We therefore develop an open-source labeling EgoPipeline that converts large collections of public egocentric videos into structured camera-and-hand trajectory supervision. MINT is first pretrained on these large-scale pseudo-labels and then fine-tuned on a small set of high-quality camera-and-hand annotations. Across public benchmarks MINT approaches state-of-the-art accuracy without seeing either benchmark in training, reaching 0.945 frame accuracy, 13.646 mm PA-MPJPE-p and 55.058 px EPE-p for camera-frame bimanual reconstruction on HOT3D, 4.690 mm RPE-T and 0.284° RPE-R for camera trajectory, and a 3.67× end-to-end speedup over the labeling pipeline that supervises it. We release the model, training and inference code, labeling pipeline, and a curated 1,021-hour egocentric trajectory dataset.",
  "keywords": [
    "egocentric video",
    "world-space hand reconstruction",
    "camera trajectory",
    "pipeline amortization",
    "cross-domain generalization"
  ],
  "paper_institutions": [],
  "output_language": "zh",
  "provider": "doubao",
  "model": "glm-5.3",
  "prompt_template": "simple",
  "extracted_at": "2026-09-17T09:25:57.786656Z",
  "template_tag": "simple",
  "summary": "### 摘要\n\n从第一人称视频中恢复世界坐标系下的相机与手部运动，对活动理解、机器人学习和增强现实至关重要，但现有方法（如HaWoR、Dyn-HaMR等）通常将问题分解为相机运动估计、深度估计、手部重建和轨迹精化等多个独立阶段，导致计算开销巨大且无法联合建模相机与手部运动。MINT提出一种统一的前馈基础模型，基于预训练的几何上下文变换器GCT（来自LingBot-Map），从单一共享时空视频表示中联合预测相机轨迹、视场角、双手MANO状态和逐帧手部可观测性，并通过可微的相机到世界坐标变换 p^w_{t,s} = R_t^T (p^c_{t,s} - t_t) 生成世界空间手部运动。为解决配对的世界空间相机-手部标注稀缺问题，作者构建了开源EgoPipeline，将Ego4D、EPIC-KITCHENS和EgoDex共1,729小时公开视频处理为1,021小时结构化运动监督伪标签，其多样性覆盖复杂背景、动态物体交互、严重手部遮挡和光照变化等难以在受控采集中重现的真实条件；两阶段训练先用大规模伪标签学习手-相机耦合几何，再冻结其他模块、仅用高精度设备采集的小型内部数据集校正相机外参头的绝对尺度。在HOT3D和ARCTIC零样本评测中，MINT在HOT3D上达到0.945帧准确率、13.646 mm PA-MPJPE-p、55.058 px EPE-p，相机轨迹RPE-T为4.690 mm、RPE-R为0.284°，ARCTIC上RPE-T降至3.390 mm，且相比监督管线实现3.67倍端到端加速，验证了统一建模与管线摊销的有效性。未来工作将探索更精确多样的度量监督以减少尺度误差并改进长程轨迹重建；局限包括单目重建继承的尺度漂移和32帧训练窗口导致的长时程漂移。该模型适用于具身AI训练数据生成、机器人模仿学习等真实场景，但受限于训练窗口长度和伪标签的尺度模糊性。\n\n### 关键要点\n\n- MINT是一个统一的前馈模型，从第一人称RGB视频联合恢复世界空间相机轨迹、视场角、双手MANO状态和手部可观测性，无需外部手部检测器、运动补全器或测试时优化。\n- 模型采用LingBot-Map的预训练几何上下文变换器（GCT）作为共享时空几何编码器，每个32帧窗口仅需单次编码器前向传播，通过交替帧注意力和全局注意力建模双向时序上下文，长视频通过重叠滑动窗口链式处理。\n- EgoPipeline将1,729小时公开视频（Ego4D、EPIC-KITCHENS、EgoDex）转换为1,021小时世界空间相机-手部运动监督，流程包括YOLO手部检测过滤、GeoCalib相机内参估计、MoGe-2单目深度注入MegaSaM/DROID-SLAM恢复度量相机轨迹、HaWoR双手重建及UKF后处理稳定轨迹。\n- 两阶段训练策略：第一阶段在多样化伪标签上联合优化相机运动、MANO、可观测性和世界帧一致性四个损失；第二阶段冻结几何编码器和手部相关头，仅用高精度设备采集的相机轨迹数据校正相机外参头的绝对尺度。\n- 世界帧一致性通过可微坐标变换 p^w = R^T(p^c - t) 将相机帧手部状态映射到世界空间，使世界帧损失能同时约束相机和手部预测，显式耦合二者几何。\n- 在HOT3D零样本评测中，MINT达到0.945帧准确率（最强基线WiLoR为0.827）、0.983召回率、13.646 mm PA-MPJPE-p和55.058 px EPE-p，在所有姿态指标上均为最佳方法。\n- 相机轨迹方面，MINT在ARCTIC上将相对平移误差从最强基线的8.730 mm降至3.390 mm，HOT3D上达4.690 mm RPE-T和0.284° RPE-R，弧长比从0.466提升至1.094，验证了第二阶段尺度校正的作用。\n- 推理效率上，MINT单GPU每帧延迟72.4 ms（HaWoR为105.1 ms，VITRA为1260.0 ms），四GPU并行降至22.7 ms，相比EgoPipeline实现3.67倍端到端加速，相比VITRA复现实现12.5倍加速。\n- 消融实验表明：LingBot-Map预训练初始化相比随机初始化将EPE-p从82.356降至55.038 px、RPE-T从6.130降至3.390 mm；混合数据源训练在六项指标中五项优于单一数据源，MPJPE-p从约48 mm降至26.527 mm。\n- UKF时序滤波在推理时应用可将Jitter从12.959降至2.621（ARCTIC）和2.373（HOT3D），接近五倍平滑度提升，且不破坏恢复的几何结构。\n- 局限性包括单目重建监督管线继承的尺度漂移、32帧训练窗口导致的长时程漂移，以及ARCTIC上第二阶段尺度校正的过冲现象（弧长比从0.755升至1.412）。\n- 模型、训练与推理代码、标注管线及1,021小时精选第一人称轨迹数据集均已开源发布，训练数据中无任何人工标注的手部标签。"
}