{
  "paper_id": "fc312182743496e01400dd5ef1256ac4",
  "paper_title": "Bringing Inputs to Shared Domains for 3D Interacting Hands Recovery in the Wild",
  "paper_authors": [
    "Gyeongsik Moon"
  ],
  "publication_date": "",
  "publication_venue": "2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)",
  "abstract": "Despite recent achievements, existing 3D interacting hands recovery methods have shown results mainly on motion capture (MoCap) environments, not on in-the-wild (ITW) ones. This is because collecting 3D interacting hands data in the wild is extremely challenging, even for the 2D data. We present InterWild, which brings MoCap and ITW samples to shared domains for robust 3D interacting hands recovery in the wild with a limited amount of ITW 2D/3D interacting hands data. 3D interacting hands recovery consists of two sub-problems: 1) 3D recovery of each hand and 2) 3D relative translation recovery between two hands. For the first sub-problem, we bring MoCap and ITW samples to a shared 2D scale space. Although ITW datasets provide a limited amount of 2D/3D interacting hands, they contain large-scale 2D single hand data. Motivated by this, we use a single hand image as an input for the first sub-problem regardless of whether two hands are interacting. Hence, interacting hands of MoCap datasets are brought to the 2D scale space of single hands of ITW datasets. For the second sub-problem, we bring MoCap and ITW samples to a shared appearance-invariant space. Unlike the first sub-problem, 2D labels of ITW datasets are not helpful for the second sub-problem due to the 3D translation's ambiguity. Hence, instead of relying on ITW samples, we amplify the generalizability of MoCap samples by taking only a geometric feature without an image as an input for the second sub-problem. As the geometric feature is invariant to appearances, MoCap and ITW samples do not suffer from a huge appearance gap between the two datasets. The code is publicly available.",
  "keywords": [
    "3D interacting hands recovery",
    "in-the-wild",
    "domain gap",
    "hand mesh recovery",
    "MANO",
    "2.5D pose",
    "geometric features",
    "mixed-batch training"
  ],
  "paper_institutions": [
    "Meta Reality Labs"
  ],
  "output_language": "zh",
  "provider": "doubao",
  "model": "glm-5.3",
  "prompt_template": "simple",
  "extracted_at": "2026-09-17T09:34:01.307487Z",
  "template_tag": "simple",
  "summary": "### 摘要\n\n现有3D交互双手恢复方法主要在动作捕捉环境上表现良好，而在真实场景中因MoCap与ITW数据集间巨大的外观差异（如IH2.6M的黑色背景与人工光照）而失效，且在野外采集大规模3D甚至2D交互手部数据因严重遮挡与自相似性而极其困难。相关研究包括基于IH2.6M数据集的InterNet、IntagHand等回归方法，以及利用ITW 2D数据的混合批次训练范式，但后者因双手图像与单手图像的2D尺度分布差异及3D相对平移的尺度歧义而难以奏效。本文提出InterWild框架，将问题分解为两个子问题：针对每只手的3D恢复，SHNet无论双手是否交互均以裁剪的单手图像为输入，将MoCap交互手与ITW单手数据统一到共享2D尺度空间；针对双手间3D相对平移，TransNet仅以外观不变的几何特征（2.5D高斯热图）为输入，且3D相对平移仅在MoCap样本上监督，避免ITW 2D弱监督的负面影响。训练采用IH2.6M与MSCOCO全身版各占一半的混合批次，以HIC（RGBD相机采集、含3D真值网格且未参与训练）为主要测试集以验证对未见外观的泛化能力，并在IH2.6M上报告误差，MSCOCO用于定性结果。评估指标为MPJPE、MPVPE（3D关节/顶点误差）与MRRPE（3D相对平移误差），InterWild在HIC上取得MPVPE 15.65/15.70 mm、MRRPE 31.35 mm，在IH2.6M上取得MPVPE 11.12/13.01 mm、MRRPE 29.29 mm，全面超越IHMR、Zhang等与IntagHand，验证了共享域设计（单手输入归一化2D尺度、几何特征消除外观差异、手腕特征提取替代GAP）的有效性。未来方向包括与全身3D人体网格估计方法集成。该方法适用于AR/VR等真实场景中多样外观图像的双手重建，但受限于ITW数据稀缺、单图3D全局平移估计的固有歧义以及相对平移缺乏强尺度约束。核心机制为：最终3D交互双手网格由右手网格Mr与左手网格Ml加3D相对平移t构成，即左手最终网格 = Ml + t，其中t由TransNet从双手2.5D热图H ∈ R^{2J×D×H×W}经ResNet-18提取手腕特征后回归得到，整个网络以预测与真值的边界框、MANO参数、3D关节坐标及3D相对平移的L1距离端到端训练。\n\n### 关键要点\n\n- InterWild由DetectNet（检测手部）、SHNet（单手3D网格与2.5D姿态估计）和TransNet（双手3D相对平移估计）三个网络组成，并以L1损失端到端训练。\n- SHNet无论双手是否交互均以裁剪的单手图像为输入，将MoCap交互手数据的2D尺度归一化到ITW单手数据的2D尺度空间，使大规模ITW 2D单手数据得以有效利用。\n- 左手图像被水平翻转为右手输入，将输入手的左右属性归一化以减轻学习负担，并在双手严重重叠时实现被遮挡左手的隐式去遮挡。\n- TransNet仅以2.5D姿态热图（几何特征）为输入，该特征对颜色与光照等外观因素不变，从而消除MoCap与ITW数据集间的外观差异，即使仅在IH2.6M上监督也能泛化到ITW图像。\n- 3D相对平移仅在MoCap样本上监督：实验表明ITW 2D弱监督因3D全局平移估计错误与相对平移尺度歧义而恶化结果（HIC上MRRPE从31.35升至61.05 mm）。\n- TransNet通过在特征图上对手腕2D位置进行双线性插值提取手腕特征，优于广泛使用的全局平均池化（HIC上MRRPE 31.35 vs 39.85 mm）。\n- 训练集为IH2.6M与MSCOCO全身版，mini批次各占一半；MoCap样本以3D真值全监督，ITW样本以2D真值弱监督。\n- 主测试集HIC为RGBD相机采集、含拟合3D点云的真值网格且未参与训练，其多样真实外观可检验网络对未见外观的泛化能力。\n- 在HIC上InterWild取得MPVPE 15.65/15.70 mm、MRRPE 31.35 mm，在IH2.6M上取得MPVPE 11.12/13.01 mm、MRRPE 29.29 mm，全面优于IHMR、Zhang等和IntagHand，且在HIC上的优势尤为显著。\n- 消融实验显示：双手交互时取单手图像优于取双手图像（HIC交互序列MPVPE 15.70 vs 35.86 mm），翻转左手并共享权重的组合取得最优结果。\n- 与仅用图像或图像加2.5D热图的TransNet相比，纯2.5D热图输入在HIC上MRRPE最低（31.35 vs 206.83/54.36 mm），验证了几何输入对外观不变性的关键作用。\n- 未来方向为将InterWild与全身3D人体网格估计方法集成；代码与训练模型已公开，且推理时使用DetectNet预测的手部框而非真值框。"
}