{
  "paper_id": "e894cb9eac946721bbfe850375cf9c28",
  "paper_title": "Deformer: Dynamic Fusion Transformer for Robust Hand Pose Estimation",
  "paper_authors": [
    "Qichen Fu",
    "Xingyu Liu",
    "Ran Xu",
    "Juan Carlos Niebles",
    "Kris M. Kitani"
  ],
  "publication_date": "",
  "publication_venue": "2023 IEEE/CVF International Conference on Computer Vision (ICCV)",
  "abstract": "Accurately estimating 3D hand pose is crucial for understanding how humans interact with the world. Despite remarkable progress, existing methods often struggle to generate plausible hand poses when the hand is heavily occluded or blurred. In videos, the movements of the hand allow us to observe various parts of the hand that may be occluded or blurred in a single frame. To adaptively leverage the visual clue before and after the occlusion or blurring for robust hand pose estimation, we propose the Deformer: a framework that implicitly reasons about the relationship between hand parts within the same image (spatial dimension) and different timesteps (temporal dimension). We show that a naive application of the transformer self-attention mechanism is not sufficient because motion blur or occlusions in certain frames can lead to heavily distorted hand features and generate imprecise keys and queries. To address this challenge, we incorporate a Dynamic Fusion Module into Deformer, which predicts the deformation of the hand and warps the hand mesh predictions from nearby frames to explicitly support the current frame estimation. Furthermore, we have observed that errors are unevenly distributed across different hand parts, with vertices around fingertips having disproportionately higher errors than those around the palm. We mitigate this issue by introducing a new loss function called maxMSE that automatically adjusts the weight of every vertex to focus the model on critical hand parts. Extensive experiments show that our method significantly outperforms state-of-the-art methods by 10%, and is more robust to occlusions (over 14%).",
  "keywords": [
    "3D手部姿态估计",
    "Transformer",
    "动态融合模块",
    "遮挡鲁棒性",
    "运动模糊",
    "视频手部姿态估计",
    "MANO手部模型",
    "maxMSE损失函数",
    "自注意力机制",
    "时序一致性"
  ],
  "paper_institutions": [
    "Carnegie Mellon University",
    "Salesforce Research"
  ],
  "output_language": "zh",
  "provider": "doubao",
  "model": "glm-5.3",
  "prompt_template": "simple",
  "extracted_at": "2026-09-17T09:29:59.574771Z",
  "template_tag": "simple",
  "summary": "### 摘要\n本文研究视频中3D手部姿态估计在严重遮挡与运动模糊条件下难以生成合理手部姿态的问题。现有单帧方法在静态图像上预测不稳定，基于循环神经网络的时序方法（如VIBE、TCMR）依赖局部关系、难以建模长程依赖，且朴素的自注意力机制会因遮挡帧特征被污染而产生不精确的键和查询。为此，作者提出Deformer动态融合Transformer框架：先用共享CNN提取逐帧手部特征，再由空间Transformer推理单帧内手部部件间的非局部关系，时序Transformer建模不同时间步特征的相关性并回归全局手部形状，最后由动态融合模块预测每帧的前向/后向运动及置信度分数，将邻近帧的MANO手部网格预测显式变形到当前帧并融合，其核心机制为置信度加权的姿态融合公式 $\\hat{\\theta}_{t}^{+}=\\frac{\\sum_{t^{\\prime}=1}^{T}e^{\\hat{c}_{t^{\\prime}}}\\hat{\\theta}_{t^{\\prime}\\to t}}{\\sum_{t^{\\prime}=1}^{T}e^{\\hat{c}_{t^{\\prime}}}}$。针对指尖顶点误差显著高于手掌的误差不均衡问题，提出受focal loss启发的maxMSE损失，按 $w_i=\\frac{(\\mathcal{P}_i-\\mathcal{P}_i')^2}{\\sum_{j=1}^{N}(\\mathcal{P}_j-\\mathcal{P}_j')^2}$ 自动为每个顶点分配权重以聚焦关键部位，并引入基于GRU的运动判别器进行对抗训练以保证时序合理性。实验选用两个大规模视频数据集：DexYCB（1000序列、582K帧、8视角抓取20种YCB物体，手物交互中频繁遮挡，用于验证遮挡鲁棒性，测试集按遮挡程度分三档）和HO3D（68序列、77,558帧、10名受试者操作10种物体，测试标注未公开需经官方在线系统评估泛化能力）。实现上采用ImageNet预训练的ResNet-50加ROIAlign提取32×32、256维特征，空间/时序Transformer各3层8头自注意力，序列长度T=7、采样间隔10帧以覆盖约2-3秒的手部运动并平衡内存与性能，Adam优化器端到端训练60个epoch，使用8块RTX 2080Ti。评估指标包括Procrustes对齐后的MPJPE、AUC、F-score及加速度误差，结果显示方法整体比现有最优方法提升约10%、遮挡鲁棒性提升超14%（DexYCB上MPJPE 5.22mm/AUC 89.6，重度遮挡下6.34mm，优于TCMR的6.28mm/6.95mm），且31M参数量比HandOccNet（39M）少20%。未来工作将探索从视频中学习细粒度表示并减少对标注的依赖。该技术面向人机交互、AR/VR和模仿学习等现实场景，但需应对物体遮挡、自遮挡、运动模糊和低相机曝光等约束。\n\n### 关键要点\n- 方法架构：Deformer由空间Transformer、时序Transformer和动态融合模块顺序组成；空间Transformer通过多头自注意力捕获单帧内手部部件的非局部交互以增强遮挡鲁棒性，时序Transformer使每帧直接受益于其他帧而不受时间距离限制。\n- 动态融合模块从时序增强特征中额外预测置信度分数与前向/后向运动元组（运动真值为MANO参数到相邻帧的变形），通过累加运动可将任意帧i的手部姿态变形到帧j，再按置信度软融合为最终输出。\n- maxMSE损失通过权重 w_i=(P_i−P_i′)²/Σ_j(P_j−P_j′)² 最大化MSE（等价于 Σ‖P_i−P_i′‖⁴/Σ‖P_j−P_j′‖²），为误差大的顶点（如指尖）分配更大权重，缓解手掌误差低、指尖误差高的不均衡问题。\n- 采用MANO参数化手部模型，通过可微分MANO层从预测的姿态与形状参数恢复21个3D关节和778个网格顶点坐标，以减少对标注的依赖并防止过拟合。\n- 训练目标为总损失 L_hand = L_mesh + L_adv + L_aux 的线性组合，辅助损失包含2D关节热图中间监督、单目MANO预测监督、变形运动一致性约束以及鼓励一阶/二阶导数接近零的平滑约束。\n- DexYCB数据集包含1000个序列、582K帧，8个相机视角下抓取20种YCB物体，手物交互中手部频繁被遮挡；测试集按重投影遮挡比例分为25%-50%、50%-75%、75%-100%三档以专门评估遮挡鲁棒性。\n- HO3D数据集包含68个序列、77,558帧（训练66,034帧、测试11,524帧），10名受试者操作10种YCB物体；因测试集标注未公开，预测需提交官方在线系统评估。\n- 实现细节：ResNet-50（ImageNet预训练）+ROIAlign提取32×32、256维手部特征；空间/时序Transformer各3层8头自注意力、前馈维度256；序列长度T=7、采样间隔10帧；Adam优化器（Transformer学习率1e-5、判别器1e-3），训练60个epoch且每10个epoch学习率乘0.7，使用8块RTX 2080Ti。\n- 主要结果：DexYCB上MPJPE为5.22mm（AUC 89.6），各遮挡级别均最优，重度遮挡（75%-100%）下6.34mm，整体比现有方法提升约10%、遮挡鲁棒性提升超14%；root对齐MPJPE 13.64mm与加速度误差31.7 mm/s²亦优于VIBE、TCMR等。\n- HO3D上关节误差9.4mm、网格误差9.1mm、F@5为54.6、F@15为96.3，与参数量更大的HandOccNet（39M对31M）性能相当，而模型参数量少20%。\n- 消融实验：动态融合（5.22mm）优于中心帧（5.72mm）、简单平均（5.40mm）和基于遮挡级别的加权（5.34mm）聚合；maxMSE与运动判别器各自独立带来增益，二者结合达到最佳性能。\n- 现实约束与应用：方法受自遮挡、物体遮挡、运动模糊和低相机曝光影响，适用于人机交互、AR/VR、模仿学习等场景；单卡推理速度为105 FPS或15 FPS（取决于所选步长），未来将探索细粒度视频表示并减少对标注的依赖。"
}