{
  "paper_id": "2d185ed77af9bc227d119428d88147fd",
  "paper_title": "ACE-Ego-Hand: Repurposing Video Diffusion Models for Occlusion-Robust Egocentric 3D Hand Motion Recovery",
  "paper_authors": [
    "Yufei Liu",
    "Xixi Wang",
    "Hao Li",
    "Ganlong Zhao",
    "Kaitong Cai",
    "Chengkai Jin",
    "Chunxiao Liu",
    "Jianbo Liu",
    "Siyuan Huang",
    "Xingang Pan",
    "Hongsheng Li"
  ],
  "publication_date": "2026",
  "publication_venue": "unknown",
  "abstract": "Egocentric video offers scalable manipulation data for embodied AI, yet recovering metric 3D hand trajectories remains challenging due to severe object occlusion and frequent out-of-sight gaps. Existing single-frame and windowed temporal regressors fail when a hand briefly leaves the frame, while recent video diffusion models (VDMs) rely on heavy, stochastic multi-step sampling as pixel-space renderers. We instead repurpose a VDM into a deterministic geometry encoder. A single forward pass over the clean latent exposes scene content beyond current observations, including occluded and out-of-sight hands. We introduce ACE-Ego-Hand, an offline clip-level framework that extracts features via a Deterministic Clean-Latent Encoder and decodes them with a Bidirectional Spatiotemporal Decoder. ACE-Ego-Hand recovers continuous bimanual trajectories with metric placement and no external detector, while a Ray-Based Camera Solver supports a second configuration that needs no test-time camera intrinsics. Across five egocentric benchmarks, ACE-Ego-Hand sets a new state of the art, cutting MPJPE-p by 30% on occlusion-heavy ARCTIC and 40% on HOT3D. These gains reach 46%–61% once out-of-sight hands are included in the evaluation, offering a scalable path from everyday human video to robot manipulation data. The code is publicly available at github.com/ggxxii/ACE-Ego-Hand.",
  "keywords": [
    "第一人称视频",
    "3D手部运动恢复",
    "视频扩散模型",
    "遮挡鲁棒性",
    "出视野手部重建",
    "MANO",
    "免内参推理",
    "机器人操作数据"
  ],
  "paper_institutions": [
    "Shanghai Jiao Tong University",
    "Nanyang Technological University",
    "The Chinese University of Hong Kong",
    "ACE Robotics"
  ],
  "output_language": "zh",
  "provider": "doubao",
  "model": "glm-5.3",
  "prompt_template": "simple",
  "extracted_at": "2026-09-17T09:33:46.191090Z",
  "template_tag": "simple",
  "summary": "### 摘要\n\n本文研究从第一人称视频中恢复度量级双手3D运动轨迹的问题,该任务因严重的手-物交互遮挡与头部快速运动导致手部完全离开视野(OOS)而极具挑战:现有单帧\"检测-裁剪-回归\"方法与窗口化时序模型在手部出画后丢失历史上下文,而近期利用视频扩散模型(VDM)先验的生成式方法(如ViDiHand)依赖多步随机采样作为像素空间渲染器,产生采样延迟并可能将生成伪影传入姿态估计。为此,作者提出离线片段级框架ACE-Ego-Hand,核心洞察是将预训练VDM(Wan2.2 DiT)重新用作确定性几何编码器而非渲染器:冻结VAE将片段压缩为干净潜变量z=E(V),在零噪声下执行单次确定性前向传播F=Φ_{0:L*}(z; σ=0)(在30块DiT栈的第15块截断,计算约减半且保留近全部精度),并通过LoRA与可训练patch嵌入进行端到端3D监督适配,使通用特征变为几何感知特征;双向时空解码器以空间锚定查询(每潜帧48个:2手token、42关节token、4寄存器token)、片段级形状先验和无因果掩码的双向注意力重建遮挡与出视野手部;基于射线的相机求解器从扩散潜变量预测连续视角射线场并以混合PnP闭式求解度量平移,支持测试时无需内参的K-free推理。训练采用七源混合数据——MANO标注的ARCTIC、HOT3D、H2O、OakInk2自我中心视频(分别验证严重双目遮挡、复杂自我视角、双臂手-物姿态与长时程多物体操作能力)、Re:InterHand渲染数据及FreiHAND/RHD图像(增加外观与姿态多样性),标准配置在16块A100上训练20k步、K-free在8块上训练,HOI4D完全保留作零样本测试以验证类别级4D交互泛化。评估指标涵盖检测(FAcc/Recall/F1)、姿态(MPJPE-p、PA-p)、定位(EPE2D-p、GO-p、CT-p)、时序平滑及出视野指标(MPPE+OOS等)。结果显示ACE-Ego-Hand在五个基准上创造新纪录、48项主要指标比较中45项领先:MPJPE-p在ARCTIC降低30%(21.67→15.26mm)、HOT3D降低40%(21.51→12.89mm)、零样本HOI4D降低23%,出视野纳入后增益达46%–61%(ARCTIC 16.8mm、HOT3D 17.3mm MPJPE+OOS),单次前向传播达63.1fps、较ViDiHand快33倍——这些结果共同验证了确定性干净潜空间编码、双向解码与射线求解器三项创新的有效性。未来方向包括将编码器蒸馏为流式因果变体以支持在线部署、扩展射线场的相机族泛化、以及获取更高精度标注数据。实际应用场景为从日常人类视频规模化策展机器人操作数据(如将恢复轨迹重定向至灵巧Inspire Hand),约束在于方法本质离线(延迟受片段长度下界约束,适合数据策展而非闭环控制)、出视野绝对放置依赖时序上下文回归的深度与方位而缺乏图像证据、K-free拟合假设针孔模型(鱼眼需回退至直接读取射线场)。\n\n### 关键要点\n\n- **核心机制**:将预训练视频扩散模型(Wan2.2 DiT)重新用作确定性干净潜空间编码器,在σ=0零噪声下对完整片段执行单次前向传播并在第15块截断30块DiT栈,计算量约减半且保留近全部精度,避免了多步生成采样的延迟与伪影传播。\n- **端到端适配**:LoRA(rank 64)注入全部30块的注意力与前馈线性层、patch嵌入层可训练,使3D监督反向传播将通用VDM特征重塑为几何感知特征;消融显示移除LoRA使MPJPE-p从16.95升至23.10mm。\n- **双向时空解码器**:每潜帧48个空间锚定查询(2手token+42关节token+4寄存器token),四层交替空间交叉注意力与时序自注意力且无因果掩码,配合旋转相对位置编码支持任意长度单次解码;片段级形状先验每手每片段预测单一β参数避免尺寸闪烁。\n- **射线相机求解器**:Ray Head以1×1卷积预测逐单元单位视角射线并由余弦距离损失监督,K-free配置通过闭式逐轴回归拟合有效针孔相机,混合PnP以闭式加权最小二乘求解面内平移,有效关节不足或残差过大时回退至腕部逆投影射线。\n- **训练设置**:七源混合数据(ARCTIC、HOT3D、H2O、OakInk2的MANO标注自我中心视频,Re:InterHand渲染数据,FreiHAND与RHD图像),RHD无MANO标注仅监督2D锚点/3D关节/存在性头;标准配置16块A100训练20k步、K-free为8块;HOI4D完全保留作零样本测试。\n- **主要结果**:五个基准上48项主要指标中45项领先;MPJPE-p在ARCTIC降低30%(21.67→15.26mm)、HOT3D降低40%(21.51→12.89mm)、零样本HOI4D降低23%(30.09→23.03mm);检测近饱和(ARCTIC上FAcc/Recall/F1均达1.000)。\n- **出视野恢复**:MPJPE+OOS在ARCTIC与HOT3D分别达16.8与17.3mm,较ViDiHand(31.05/44.44mm)降低45.9%与61.1%;仅出视野层达35.2与38.6mm,对比ViDiHand的149.1/151.7mm与最强基线WildHands的135.4/126.4mm。\n- **效率**:单次确定性前向传播在单块A100上达63.1fps(K-free为63.3fps),对比ViDiHand的1.91fps,快33倍;运行时72%消耗于VAE编码。\n- **消融证据**:特征源对比显示VDM特征(MPJPE-p 16.95mm)优于V-JEPA 2(17.81mm)、VideoMAE(22.53mm)、VAE潜变量(32.50mm)与原始RGB(46.50mm);V-JEPA 2出视野误差跳升至66.1mm,证明生成先验在视觉线索缺失时的预测优势。\n- **K-free代价**:检测与度量放置与标准配置持平(HOT3D上F1 0.995 vs 0.996、EPE2D-p 6.07 vs 6.42px),残余成本为窄视场基准上小幅腕对齐姿态差距(ARCTIC +1.36mm MPJPE-p),零样本HOI4D反而改善(22.15 vs 23.03mm)。\n- **约束与局限**:方法本质离线,延迟受片段长度下界约束,适合数据策展而非闭环控制;出视野增益为腕对齐指标,不认证绝对放置;射线场在训练所见相机族内泛化最佳;HOI4D与H2O部分依赖伪真值标注。\n- **应用价值**:恢复的双臂轨迹可重定向至灵巧Inspire Hand产生自然协调运动,为从大规模人类视频到机器人操作数据提供可扩展路径;代码公开于github.com/ggxxii/ACE-Ego-Hand。",
  "available_templates": [
    "deep_read",
    "simple"
  ]
}