{
  "paper_id": "e49bd20d13884d4bdaccd03d30cd9fcc",
  "paper_title": "HandVoxNet: Deep Voxel-Based Network for 3D Hand Shape and Pose Estimation from a Single Depth Map",
  "paper_authors": [
    "Jameel Malik",
    "Sk Aziz Ali",
    "Ibrahim Abdelaziz",
    "Ahmed Elhayek",
    "Soshi Shimada",
    "Vladislav Golyanik",
    "Christian Theobalt",
    "Didier Stricker"
  ],
  "publication_date": "2020",
  "publication_venue": "CVPR 2020",
  "abstract": "3D hand shape and pose estimation from a single depth map is a new and challenging computer vision problem with many applications. The state-of-the-art methods directly regress 3D hand meshes from 2D depth images via 2D convolutional neural networks, which leads to artefacts in the estimations due to perspective distortions in the images. In contrast, we propose a novel architecture with 3D convolutions trained in a weakly-supervised manner. The input to our method is a 3D voxelized depth map, and we rely on two hand shape representations. The first one is the 3D voxelized grid of the shape which is accurate but does not preserve the mesh topology and the number of mesh vertices. The second representation is the 3D hand surface which is less accurate but does not suffer from the limitations of the first representation. We combine the advantages of these two representations by registering the hand surface to the voxelized hand shape. In the extensive experiments, the proposed approach improves over the state of the art by 47.8% on the SynHand5M dataset. Moreover, our augmentation policy for voxelized depth maps further enhances the accuracy of 3D hand pose estimation on real data. Our method produces visually more reasonable and realistic hand shapes on NYU and BigHand2.2M datasets compared to the existing approaches.",
  "keywords": [
    "3D hand pose estimation",
    "3D hand shape estimation",
    "depth map",
    "voxel-based representation",
    "3D CNN",
    "weakly-supervised learning",
    "3D data augmentation",
    "shape registration"
  ],
  "paper_institutions": [
    "TU Kaiserslautern",
    "DFKI Kaiserslautern",
    "NUST Pakistan",
    "UPM Saudi Arabia",
    "MPII Saarland"
  ],
  "output_language": "zh",
  "provider": "doubao",
  "model": "glm-5.3",
  "prompt_template": "simple",
  "extracted_at": "2026-09-17T09:34:24.917126Z",
  "template_tag": "simple",
  "summary": "### 摘要\n本文研究从单张深度图同时估计3D手部形状与姿态这一新兴且具有挑战性的计算机视觉问题。现有最先进方法（如DeepHPS、WHSP-Net等）通过2D卷积神经网络直接从2D深度图像回归3D手部网格，由于图像中的透视畸变导致估计结果出现伪影。为此，作者提出首个基于3D卷积的体素化架构HandVoxNet：以88×88×88的3D体素化深度图为输入，先由V2V-PoseNet估计3D关节热图，再通过V2V-ShapeNet（体素到体素网络）与V2S-Net（体素到表面网络）分别估计两种手部形状表示——前者精确但不保持网格拓扑与顶点数，后者精度较低但保持拓扑——最后利用DispVoxNet（合成数据）与NRGA（真实数据）将形状表面配准到体素化形状以融合两者优势。由于真实手部形状标注难以获取，训练采用弱监督方式，由V2V-SynNet与S2V-SynNet两个3D CNN合成器从形状表示重建体素深度图提供监督信号，其核心机制为总损失 L_T = L_H + 1·L_VS + 1·L_VT + L_VD^v + L_VD^s，其中指示函数层仅对合成数据启用形状损失项。实验选用SynHand5M（500万张全标注合成图像，因提供形状真值用于定量评估）、NYU（72757训练/8252测试RGBD图像）与BigHand2.2M（956k训练/296k测试深度图像，百万级真实基准）三个数据集，采用3D关节误差、3D顶点误差与体素化形状误差（逐体素二元交叉熵）作为指标；在SynHand5M上形状估计精度较WHSP-Net提升47.8%（顶点误差2.67mm对5.12mm），DispVoxNet配准使表面重建精度提升20.5%（3.36mm至2.67mm），姿态估计达3.75mm（较WHSP-Net提升13.19%），验证了一一映射与弱监督合成器的有效性；新的3D数据增强策略（三轴随机欧拉旋转及缩放平移）使NYU与BigHand2.2M上姿态精度分别提升5.42%与6.83%，验证了体素表示适合3D增强。未来方向包括生成更逼真的合成数据集以增强真实图像形状重建，以及通过GPU并行化改进配准方法运行时间。实际应用中，方法在严重遮挡与深度信息缺失时无法产生合理形状，且NRGA配准在CPU上需59-70秒，限制了实时部署场景。\n\n### 关键要点\n- 提出首个基于体素的手部形状与姿态联合估计方法HandVoxNet，以3D体素化深度图为输入，通过3D卷积建立体素深度图、体素化手形与3D关节热图之间的一一映射，避免了2D CNN处理深度图固有的透视畸变问题。\n- 方法包含四个新颖组件：V2V-ShapeNet（体素到体素3D CNN形状回归网络）、V2S-Net（体素到表面3D CNN回归网络）、V2V-SynNet与S2V-SynNet（3D CNN体素深度图合成器）以及手部形状配准组件。\n- 采用两种互补的形状表示：体素化形状精确但不保持网格拓扑与顶点数，手部表面保持拓扑但精度较低；通过配准（合成数据用DispVoxNet、真实数据用NRGA）将表面拟合到体素化形状以结合两者优势。\n- 由于真实手部形状标注难以获取，采用弱监督训练：合成器从估计的形状表示重建体素深度图提供监督信号（测试时移除），指示函数层仅对合成数据启用形状损失以阻断真实数据的梯度流。\n- 提出新的3D数据增强策略：对体素化深度图与关节热图施加三轴随机欧拉旋转（x、y轴范围[-40°,+40°]，z轴范围[-120°,+120°]）以及缩放[+0.8,+1.2]与平移[-8,+8]，提升姿态估计的鲁棒性与精度。\n- 实验数据集包括SynHand5M（500万张全标注合成深度图，训练4.5M/测试500k）、NYU（72757训练/8252测试RGBD图像，42 DOF手部模型）与BigHand2.2M（956k训练/296k测试深度图像，10名受试者，21关节标注）。\n- 评估指标为3D关节平均位置误差、3D顶点平均误差与体素化形状误差（逐体素二元交叉熵）；训练使用RMSProp（批大小8，学习率2.5×10⁻⁴）与Adam（DispVoxNet，学习率3.0×10⁻⁴，批大小12），在配备Nvidia Titan X GPU的工作站上训练至收敛。\n- 关键结果：SynHand5M上形状估计较WHSP-Net提升47.8%（顶点误差2.67mm对5.12mm，DeepHPS为11.8mm）；DispVoxNet配准使表面重建精度提升20.5%（3.36mm至2.67mm）；姿态估计3.75mm，较WHSP-Net提升13.19%。\n- 3D数据增强使V2V-PoseNet在NYU上误差从9.22mm降至8.72mm（提升5.42%），在BigHand2.2M上从9.95mm降至9.27mm（提升6.83%），指尖关节精度改善尤为明显。\n- 消融研究表明，将3D关节热图与体素化深度图拼接作为输入优于任一单独输入（V2S-Net误差3.36mm对比仅用热图8.78mm或仅用深度图3.54mm）；加入合成器使误差从2.92mm进一步降至2.67mm。\n- 局限与约束：严重遮挡与深度信息缺失时方法无法产生合理形状；NRGA配准在CPU上需59-70秒、DispVoxNet需0.162秒，影响实时应用；未来可通过GPU并行化与更逼真的合成数据集改进。"
}