{
  "paper_id": "b1dd509d6ffeb7a4696295a74d9d2aa4",
  "paper_title": "Hand Pose Understanding With Large-Scale Photo-Realistic Rendering Dataset",
  "paper_authors": [
    "Xiaoming Deng",
    "Yinda Zhang",
    "Jian Shi",
    "Yuying Zhu",
    "Dachuan Cheng",
    "Dexin Zuo",
    "Zhaopeng Cui",
    "Ping Tan",
    "Liang Chang",
    "Hongan Wang"
  ],
  "publication_date": "2021-04-07",
  "publication_venue": "IEEE Transactions on Image Processing",
  "abstract": "Hand pose understanding is essential to applications such as human computer interaction and augmented reality. Recently, deep learning based methods achieve great progress in this problem. However, the lack of high-quality and large-scale dataset prevents the further improvement of hand pose related tasks such as 2D/3D hand pose from color and depth from color. In this paper, we develop a large-scale and high-quality synthetic dataset, PBRHand. The dataset contains millions of photo-realistic rendered hand images and various ground truths including pose, semantic segmentation, and depth. Based on the dataset, we firstly investigate the effect of rendering methods and used databases on the performance of three hand pose related tasks: 2D/3D hand pose from color, depth from color and 3D hand pose from depth. This study provides insights that photo-realistic rendering dataset is worthy of synthesizing and shows that our new dataset can improve the performance of the state-of-the-art on these tasks. This synthetic data also enables us to explore multi-task learning, while it is expensive to have all the ground truth available on real data. Evaluations show that our approach can achieve state-of-the-art or competitive performance on several public datasets.",
  "keywords": [
    "Hand pose estimation",
    "photo-realistic synthetic dataset",
    "physical-based rendering",
    "multi-task CNN"
  ],
  "paper_institutions": [
    "Beijing Key Laboratory of Human-Computer Interactions, Institute of Software, Chinese Academy of Sciences",
    "Google",
    "Institute of Automation, Chinese Academy of Sciences",
    "College of Computer Science, Zhejiang University",
    "School of Computing Science, Simon Fraser University",
    "School of Artificial Intelligence, Beijing Normal University"
  ],
  "output_language": "zh",
  "provider": "doubao",
  "model": "glm-5.3-flash",
  "prompt_template": "simple",
  "extracted_at": "2026-09-17T10:44:17.414391Z",
  "template_tag": "simple",
  "summary": "### 摘要\n\n手部姿态理解是人机交互、增强现实和机器人等应用的关键技术，但现有数据集在规模、受试者数量、标注多样性与精度上的不足制约了基于深度学习的手部姿态相关任务（彩色图像2D/3D姿态估计、彩色图像深度估计、深度图像3D姿态估计）的进一步提升。针对真实数据采集与标注成本高昂、已有合成数据（如RHD仅约4.4万张且渲染不够真实）存在域差距的问题，本文构建了大规模照片级真实感合成数据集PBRHand：使用Artec Eva扫描仪采集50名受试者的高保真手部网格、用ShapeHand数据手套采集1840种手势，经绑定与变形生成92K个手部网格，再用Mitsuba物理渲染器在随机视点与20张HDR环境光照下渲染出约552万张图像，并同时提供3D关节、逐像素语义分割与深度真值。基于该数据集，作者在三个手部相关任务上验证了预训练收益，并提出了级联多任务卷积网络，其核心机制为总损失 L_total = L_pose + λ_d·L_depth（λ_d=1.0），其中姿态损失为3D关节的欧氏距离，深度损失为手部前景区域内预测深度与真值的L1范数；网络第一阶段用U-net恢复深度特征并预测深度图，第二阶段融合图像特征与深度特征回归归一化3D关节。实验采用EPE、AUC、PCK、MAE等指标，在Stereo、RHD、MPII+NZSL、NYU、Dexter+Object和Hands 2019等公开数据集上评估，结果显示PBRHand预训练优于RHD与OpenGL渲染版本（如Stereo上3D EPE从8.626mm降至7.716mm），级联网络达到6.39mm EPE、99.79% AUC的最优或竞争性能；未来方向包括加入手臂等更具挑战性的部位建模。该方法面向需要实时手部交互的真实场景，网络约26.7M参数、在Titan X GPU上达180fps，具备嵌入式实时部署潜力。\n\n### 关键要点\n\n- PBRHand数据集包含约552万张照片级真实感合成彩色图像，由50名受试者（30男20女）的手型与1840种手势组合生成92K个手部网格，图像数量约为RHD数据集的125倍。\n- 数据采集流程为：Artec Eva手持扫描仪采集手部点云并融合成网格，经非刚性ICP重拓扑到标准模板，纹理由专业设计师精修，每名受试者的手工处理约需85分钟（采集10分钟、纹理60分钟、绑定变形15分钟）。\n- 手势通过Measurand ShapeHand数据手套在不到两小时内采集，覆盖抓取、捏合、手指屈伸/外展等典型动作，采集后由设计师精修关节位置与朝向以消除噪声和网格穿透等瑕疵。\n- 渲染使用Mitsuba物理渲染器与路径追踪，相机距离在400–800mm随机采样，天顶角取[π/6, 2π/3]、方位角取[0, 2π)，并使用10张室内与10张室外HDR环境贴图模拟真实光照，图像分辨率640×480、水平视场角54度。\n- 每个样本同时提供彩色图像、深度图、15个手部部件的逐像素语义分割和3D关节坐标真值，这些多样化标注在真实数据上获取代价高昂，是合成数据的核心优势。\n- 渲染真实性消融证明其重要性：在Stereo数据集上，PBRHand预训练的PoseNet 2D AUC(0-30像素)为80.8、EPE 6.467像素，而OpenGL非真实感渲染版本（PBRHand-OpenGL）仅为76.9、8.203像素。\n- 在彩色图像3D姿态任务上，PBRHand预训练+Stereo微调达到AUC(20-50mm) 99.3%、EPE 7.716mm，优于RHD预训练（98.7%、8.626mm）与不预训练（97.9%、9.142mm）。\n- 深度估计任务中，PBRHand+Stereo训练的堆叠沙漏网络在Stereo数据集上手部前景MAE为4.541mm，优于仅用Stereo（5.877mm）和RHD+Stereo（4.571mm）。\n- 深度图像3D姿态任务中，基于扫描手型的PBRHand-Sub+NYU训练DenseReg平均误差11.75mm、AUC 66.5%，优于参数化MANO模型+NYU的12.55mm、63.4%，说明扫描手型比MANO更真实。\n- 级联多任务网络以128×128彩色图像为输入，第一阶段用U-net生成深度特征并预测深度图，第二阶段将图像特征与深度特征拼接后再经U-net融合，最终回归归一化3D关节位置。\n- 训练损失为 L_total = L_pose + λ_d·L_depth（λ_d=1.0），姿态损失为关节欧氏距离，深度损失为手部前景二值掩码内预测与真值深度的L1范数；消融显示特征级联（RGB_feature + D_feature）优于输入/输出级联。\n- 级联网络在Stereo上达到EPE 6.39mm、AUC 99.79%，比单任务预训练网络低约2.72mm；在Hands 2019挑战赛上PBRHand预训练模型（User:Fractality）在EXTRAP（28.81mm）、INTERP、OBJECT三个划分上优于对比方法，网络约26.7M参数、Titan X GPU上测试速度达180fps。"
}