{
  "paper_id": "ba71936dae8aa3cd8d76d87af10d82c3",
  "paper_title": "Controllable Egocentric Video Generation via Occlusion-Aware Sparse 3D Hand Joints",
  "paper_authors": [
    "Chenyangguang Zhang",
    "Botao Ye",
    "Boqi Chen",
    "Alexandros Delitzas",
    "Fangjinhua Wang",
    "Marc Pollefeys",
    "Xi Wang"
  ],
  "publication_date": "",
  "publication_venue": "unknown",
  "abstract": "Controllable video generation for complex hand-object interactions is a critical step toward building visual world models. However, existing methods often struggle to achieve fine-grained, 3D-consistent hand articulation in generated videos. By relying on dense 2D trajectories or implicit pose representations, they collapse crucial geometric structures into spatially ambiguous signals, leading to severe motion inconsistencies and hallucinated artifacts under egocentric occlusions. To address this, we propose leveraging sparse 3D hand joints as explicit control signals with three key advantages: explicit geometry to resolve occlusions, an intuitive interface for interactive editing, and cross-embodiment generalization to robotic hands. Built upon this, our efficient control module extracts occlusion-aware features from the source reference frame by penalizing unreliable visual features from hidden joints, and employs a 3D-based weighting mechanism to handle dynamically occluded target joints during motion propagation. Meanwhile, it directly injects 3D geometric embeddings into the latent space to enforce structural consistency. To facilitate robust training and evaluation, we develop an automated annotation pipeline, yielding 1M high-quality egocentric video clips paired with precise hand trajectories. Experiments demonstrate that our approach outperforms state-of-the-art baselines, generating high-fidelity egocentric videos with realistic hand-object interactions.",
  "keywords": [
    "Egocentric Vision",
    "World Model",
    "Video Generation"
  ],
  "paper_institutions": [
    "ETH Zurich",
    "ETH AI Center",
    "MPI for Informatics",
    "Microsoft"
  ],
  "output_language": "zh",
  "provider": "doubao",
  "model": "glm-5.3",
  "prompt_template": "simple",
  "extracted_at": "2026-09-17T09:41:08.536542Z",
  "template_tag": "simple",
  "summary": "### 摘要\n复杂手-物交互的可控视频生成是构建视觉世界模型的关键步骤，但现有基于稠密2D轨迹或隐式姿态表示的方法会将关键几何结构压缩为空间歧义信号，在自我中心视角的严重遮挡下产生运动不一致与幻觉伪影。针对该问题，本文提出以稀疏3D手部关节点作为显式控制信号，构建遮挡感知且几何保持的图像到视频生成框架：其核心机制是成对遮挡惩罚 P_{i←j}=exp(-||u_i-u_j||²/2τ²)·σ(γ(d_j-d_i))，该公式结合空间重叠与深度排序估计关节j遮挡关节i的概率，并通过门控平均池化抑制被遮挡关节的不可靠视觉特征；在运动传播阶段，采用基于视差的softmax加权机制（可微Z缓冲）处理动态目标遮挡，同时将融合3D坐标正弦编码与可学习语义身份的几何嵌入经因果Conv3D直接注入潜空间以强化结构一致性，整个控制模块仅约2万参数、轻量且最小扰动预训练潜空间。为支撑训练与评估，作者基于Ego4D构建自动化标注管线（YOLO手部检测、WiLoR逐帧MANO重建、二分图匹配时空跟踪与质量过滤），产出100万条带精确手部轨迹的高质量自我中心视频片段，并选用Ego4D（域内日常活动）与EgoDex（域外灵巧桌面操作）作为评估集以分别验证生成能力与泛化性；模型基于WAN 2.1骨干、秩64 LoRA微调，在16块NVIDIA GH200 GPU上训练约48小时，训练中随机掩码5%关节以增强鲁棒性。实验采用FID、FVD、PSNR、SSIM衡量视觉质量，MPJPE、MPVPE衡量3D手部控制精度，结果显示方法全面超越Mask2IV、WAN-Fun、MotionStream与WAN-Move等基线（如Ego4D上FVD较MotionStream至少降低16%、EgoDex上MPJPE较WAN-Move*降低68%），消融实验分别验证了遮挡去除上下文聚合、遮挡感知传播与3D几何嵌入三项创新的贡献；进一步探索包括单关节级交互式微操控编辑与经少量微调即可跨具身泛化至Unitree G1-Dex3-1、H1-Inspire等机器人灵巧手，适用于VR渲染与机器人学习等真实场景，其约束在于依赖上游手部跟踪器的轨迹质量与大规模标注数据。\n\n### 关键要点\n- 方法核心：以稀疏3D手部关节点（每手21个、双手至多42个）作为显式控制信号，兼具显式3D几何以消解遮挡歧义、直观交互编辑接口与跨具身（机器人灵巧手）泛化三重优势。\n- 遮挡去除上下文聚合（OCA）：在源参考帧以高斯热图聚合各关节周围局部VAE潜特征，并用成对遮挡惩罚进行门控，抑制被遮挡关节的不可靠纹理，防止后续帧出现幻觉。\n- 遮挡感知特征传播（OP）：引入基于视差的softmax加权机制充当可微Z缓冲，在前景手指越过背景手指时将大部分权重分配给更靠近相机的关节，避免特征混淆与手指融合等解剖学伪影。\n- 3D几何嵌入（3DGE）：将3D坐标的正弦位置编码与可学习关节身份嵌入经浅层MLP融合，用高斯热图泼溅到空间网格后经因果Conv3D与LayerNorm对齐视频潜空间，保证时间因果性与结构一致性。\n- 轻量控制设计：整个控制模块仅约2万参数，通过通道维拼接注入噪声潜变量与条件张量，避免ControlNet类重型分支的计算开销及对预训练潜分布的破坏。\n- 数据构建：基于Ego4D的自动化管线采用YOLO检测手部框、WiLoR逐帧重建MANO参数、含迟滞阈值与间隙重置的二分图匹配跟踪，再经质量过滤（有效手实例帧占比、MANO密度、多手检测等阈值）得到约100万条高精度轨迹片段。\n- 训练设置：基于WAN 2.1图像到视频模型，对transformer层施加秩64的LoRA微调，控制模块参数从零训练，使用16块NVIDIA GH200 GPU训练约48小时，并随机掩码5%输入关节以容忍跟踪器间歇失效与稀疏用户输入。\n- 评估协议：在Ego4D（40条未见序列，域内）与EgoDex（30条序列，域外灵巧桌面操作）上评估，指标涵盖FID、FVD、PSNR、SSIM视觉质量与经WiLoR提取的Procrustes对齐MPJPE、MPVPE三维手部精度。\n- 主要结果：方法在Ego4D上取得FID 67.70、FVD 259.99、MPJPE 1.42，较WAN-Fun的FVD优14%、MPJPE低19%，较MotionStream的FVD至少降16%，较WAN-Move*在EgoDex上MPJPE降68%，30人2AFC用户研究亦确认感知质量与运动准确度优势。\n- 消融证据：仅OCA即超越2D轨迹方法；加入OP后Ego4D上FID再降5%、MPJPE降23%；加入3DGE后FID提升11%、MPJPE改善32%，证明各组件对遮挡处理与3D感知控制的必要性。\n- 交互与跨具身：支持序列驱动与交互编辑两种模式，用户可拖动单个或多个关节实现从全局平移、腕部旋转到单指微动的细粒度控制；经小规模机器人数据微调后，单一统一模型可控制Unitree G1-Dex3-1与H1-Inspire等不同结构机器人手。\n- 适用约束：性能依赖上游3D手部跟踪器的轨迹质量与大规模自我中心标注数据，跨具身能力需额外微调，评估基于有限的测试序列。"
}