{
  "paper_id": "0b4954abfd587bd7a35e7f408784e606",
  "paper_title": "DGGAN: Depth-image Guided Generative Adversarial Networks for Disentangling RGB and Depth Images in 3D Hand Pose Estimation",
  "paper_authors": [
    "Liangjian Chen",
    "Shih-Yao Lin",
    "Yusheng Xie",
    "Yen-Yu Lin",
    "Wei Fan",
    "Xiaohui Xie"
  ],
  "publication_date": "",
  "publication_venue": "2020 IEEE Winter Conference on Applications of Computer Vision (WACV)",
  "abstract": "Estimating 3D hand poses from RGB images is essential to a wide range of potential applications, but is challenging owing to substantial ambiguity in the inference of depth information from RGB images. State-of-the-art estimators address this problem by regularizing 3D hand pose estimation models during training to enforce the consistency between the predicted 3D poses and the ground-truth depth maps. However, these estimators rely on both RGB images and the paired depth maps during training. In this study, we propose a conditional generative adversarial network (GAN) model, called Depth-image Guided GAN (DGGAN), to generate realistic depth maps conditioned on the input RGB image, and use the synthesized depth maps to regularize the 3D hand pose estimation model, therefore eliminating the need for ground-truth depth maps. Experimental results on multiple benchmark datasets show that the synthesized depth maps produced by DGGAN are quite effective in regularizing the pose estimation model, yielding new state-of-the-art results in estimation accuracy, notably reducing the mean 3D endpoint errors (EPE) by 4.7%, 16.5%, and 6.8% on the RHD, STB and MHP datasets, respectively.\n",
  "keywords": [
    "三维手部姿态估计",
    "深度图重建",
    "生成对抗网络",
    "非成对RGB-深度数据",
    "深度正则化",
    "单目RGB图像"
  ],
  "paper_institutions": [
    "University of California, Irvine",
    "Tencent America",
    "Amazon",
    "National Chiao Tung University"
  ],
  "output_language": "zh",
  "provider": "doubao",
  "model": "doubao-seed-2.1-pro",
  "prompt_template": "simple",
  "extracted_at": "2026-10-20T03:43:18.005417Z",
  "template_tag": "simple",
  "summary": "### 摘要\n\n从单目RGB图像估计三维手部姿态可服务于人机交互、手语理解、AR/VR和机器人等场景，但由RGB推断深度存在固有歧义；已有深度正则化方法虽能在推理阶段仅使用RGB，却依赖成对RGB-深度训练数据，而深度传感器昂贵、多数数据集只包含RGB或深度单一模态且无法配对。为此，论文提出条件生成对抗网络DGGAN，由深度图重建模块和手部姿态估计模块组成，并以端到端方式联合训练。其核心机制是将RGB到深度的转换视为非成对模态适应，优化目标为 G^*=\\arg\\min_G\\max_D(\\lambda_t\\mathcal{L}_{task}+\\lambda_g\\mathcal{L}_{GAN})：生成器根据RGB合成手部深度图，判别器区分真实深度图与生成深度图；姿态模块先用六阶段CPM从RGB预测2D关节热图，再由mini-CPM和全连接层回归关节相对深度，深度正则器则根据关节深度重建深度图，从而约束三维姿态。实验选用合成数据集RHD、真实数据集STB和真实多视角数据集MHP，RHD用于检验复杂视角和手形，STB使用BB子集训练、SK子集做非成对测试，MHP因没有稠密深度图而采用RHD深度图作为深度源，这些设置共同验证方法可跨数据集利用非成对数据。评价指标包括PCK阈值20–50mm的AUC以及三维关节端点误差EPE的均值和中位数；消融实验中，生成深度图使RHD、STB、MHP的AUC分别提升0.043、0.024、0.011，EPE均值分别降低13.2%、19.7%、7.3%，整体相对已有方法在三个数据集上的EPE均值分别降低4.7%、16.5%和6.8%，取得新的最优结果，支持非成对合成深度可有效替代成对真实深度正则化。在STB上，使用生成深度图的结果甚至优于使用真实传感器深度图，说明高质量仿真深度可降低传感器噪声。该方法的约束是训练阶段仍需RGB图像集合和非成对深度图集合，深度正则器初始化会使用真实深度图，MHP还需借助外部RHD深度；未来可进一步探索三维形状估计等下游应用以及减少对外部深度数据的依赖。\n\n### 关键要点\n\n- DGGAN包含深度图重建模块和手部姿态估计模块，两模块可联合进行端到端训练。\n- 深度图重建模块采用编码器—解码器生成器，将输入RGB手部图像转换为深度图，并用判别器判断深度图来自真实集合还是生成器。\n- 该模块把RGB与深度看作两种模态，只要求提供非成对RGB图像集合和真实深度图集合，不需要同一手部实例的配对数据。\n- 手部姿态估计模块首先使用六阶段CPM预测K个手部关节的2D热图，并对每个阶段施加MSE中间监督。\n- 2D热图随后输入mini-CPM和三个全连接层以回归各关节的相对深度，相对深度训练采用smooth L1损失。\n- 深度正则器将关节相对深度重塑为多通道张量，再经六层转置卷积上采样为单通道相对深度图，并使用L1损失进行约束。\n- 深度正则器在初始化阶段可使用真实深度图，初始化完成后改用DGGAN生成的深度图提供正则化。\n- RHD是包含RGB、深度及2D/3D标注的合成数据集，STB是真实手部数据集，MHP是包含四个视角的真实多视角手部数据集。\n- 在STB实验中，BB子集用于DGGAN训练，SK子集用于非成对测试；MHP因不含稠密深度图，训练时随机使用RHD深度图作为真实深度样本。\n- 消融结果显示，加入DGGAN生成深度图后，RHD、STB和MHP的AUC 20–50分别提升至0.839、0.990和0.939，EPE均值分别降至19.0mm、9.11mm和13.12mm。\n- 在STB上，使用生成深度图比使用真实配对深度图的EPE均值低0.94mm，论文认为原因可能是生成器利用高质量仿真深度减轻了真实深度传感器的噪声。\n- 该方法适用于仅有非成对RGB和深度数据的训练场景，并可用于人机交互、AR/VR、手语理解和机器人等任务，但仍依赖一定数量的真实深度图用于对抗训练和模块初始化。"
}