{
  "paper_id": "086ccfb86d8b34030452d8696a134546",
  "paper_title": "MVHM: A Large-Scale Multi-View Hand Mesh Benchmark for Accurate 3D Hand Pose Estimation",
  "paper_authors": [
    "Liangjian Chen",
    "Shih-Yao Lin",
    "Yusheng Xie",
    "Yen-Yu Lin",
    "Xiaohui Xie"
  ],
  "publication_date": "2021",
  "publication_venue": "2021 IEEE Winter Conference on Applications of Computer Vision (WACV)",
  "abstract": "Estimating 3D hand poses from a single RGB image is challenging because depth ambiguity leads the problem illposed. Training hand pose estimators with 3D hand mesh annotations and multi-view images often results in significant performance gains. However, existing multi-view datasets are relatively small with hand joints annotated by off-the-shelf trackers or automated through model predictions, both of which may be inaccurate and can introduce biases. Collecting a large-scale multi-view 3D hand pose images with accurate mesh and joint annotations is valuable but strenuous. In this paper, we design a spin match algorithm that enables a rigid mesh model matching with any target mesh ground truth. Based on the match algorithm, we propose an efficient pipeline to generate a large-scale multi-view hand mesh (MVHM) dataset with accurate 3D hand mesh and joint labels. We further present a multi-view hand pose estimation approach to verify that training a hand pose estimator with our generated dataset greatly enhances the performance. Experimental results show that our approach achieves the performance of 0.990 in AUC_{20-50} on the MHP dataset compared to the previous state-of-the-art of 0.939 on this dataset. Our dataset is available at https://github.com/Kuzphi/MVHM.",
  "keywords": [
    "3D hand pose estimation",
    "multi-view",
    "hand mesh",
    "MVHM dataset",
    "graph convolutional network",
    "spin matching",
    "synthetic data",
    "depth ambiguity"
  ],
  "paper_institutions": [
    "University of California, Irvine",
    "Tencent America",
    "Amazon",
    "National Chiao Tung University"
  ],
  "output_language": "zh",
  "provider": "doubao",
  "model": "glm-5.3",
  "prompt_template": "simple",
  "extracted_at": "2026-09-17T09:26:38.238162Z",
  "template_tag": "simple",
  "summary": "### 摘要\n\n从单张RGB图像估计3D手部姿态因深度歧义而本质上是病态问题，尽管已有研究通过深度图中间监督（Iqbal等）、弱监督深度正则化（Cai等）或基于MANO模型的网格监督（Boukhayma、Ge等）加以缓解，但现有多视图数据集规模较小、其标注依赖现成跟踪器或模型预测而不准确且会引入偏差。为此，本文设计了自旋匹配算法，使带骨架的刚性网格模型能与任意目标网格真值精确对齐（每根骨骼含7个自由度：骨头3维、骨尾3维、自旋1维，通过保持自旋符号向量 e = u × ref × u 在匹配前后不变来旋转骨骼，避免仅移动骨骼位置导致网格扭曲），并基于该算法构建了大规模多视图手部网格数据集MVHM：利用TurboSquid手部模型匹配NYU数据集的精确关键点真值，在Blender中以8个均匀分布于垂直手掌平面圆上且指向掌心的相机渲染出32万张256×256图像，每样本附带21个关节与2651个网格顶点的100%准确标注（另含分割掩码、相机内参与光流）。为验证数据集价值，本文提出端到端可训练的多视图手部姿态估计方法：单视图分支以stacked hourglass为骨干提取特征并预测关节热图，谱图卷积网络采用Chebyshev多项式近似（核心卷积公式 F' = Σ_{i=0}^{S} α_i T_i(L̂) θ_i，将复杂度从O(N³)降低）并经由heavy-edge matching三级粗化实现由粗到细的网格估计，深度证据网络回归关节深度；多视图分支将各视图预测在坐标通道拼接后输入graph U-Net融合出最终3D关键点。实验选择真实多视图基准MHP（21段视频、Leap Motion标注，用于验证多视图融合与真实场景泛化）与人工标注的STB SK子集（6段视频、首段作验证集）进行评估，采用EPE_m、AUC_{0-50}与AUC_{20-50}指标，训练采用batch size 8、Adam初始学习率0.01、4块GTX 1080-Ti GPU及多阶段训练策略；结果显示多视图方法配合MVHM在MHP上取得AUC_{20-50}=0.990、EPE=5.20mm，大幅超越此前最优的0.939（Chen等），消融实验进一步证明MVHM使单视图EPE从22.13mm降至18.09mm、多视图从8.43mm降至5.20mm，同时支撑了多视角信息融合与精确网格监督两项创新的有效性。未来可探索更精细的网格度量（如表面法向损失）、时序信息利用及缩小合成与真实的域差距；该方法可服务于人机交互、VR/AR、医疗诊断与手语理解等现实场景，但受限于多视图部署需已知相机参数、合成数据存在域差距、MANO低维参数化会丢失高维手部信息等约束。\n\n### 关键要点\n\n- 问题定义：单视图RGB 3D手部姿态估计因深度歧义而病态，而依据对极几何的多视图成像在相机参数已知时可准确推断深度，这是构建多视图数据集与方法的核心动机。\n- 自旋匹配算法：骨骼含7个自由度（骨头3维、骨尾3维、自旋1维），定义骨骼向量 u = B[i].tail − B[i].head 与自旋符号向量 u × ref × u，通过匹配前后符号向量之间的夹角旋转骨骼，避免仅移动骨骼位置导致的网格扭曲。\n- MVHM数据集规模：包含32万张256×256合成图像，8个相机均匀分布在垂直于手掌的平面上并指向掌心以保证手部位于图像中心，每样本附带21个手部关节与2651个网格顶点的精确标注。\n- 数据生成管线：采用TurboSquid带骨架手部模型匹配NYU数据集的关键点真值，在开源软件Blender中渲染，并随机改变光照强度、全局光照与背景场景以提升数据多样性。\n- 附加标注资源：MVHM还提供手部分割掩码、相机内参矩阵与光流，但论文本身仅使用其中的多视图图像与网格信息进行姿态估计器训练。\n- 单视图网络结构：stacked hourglass骨干提取特征并输出关节热图（热图损失 L_h），谱GCN基于图拉普拉斯归一化与Chebyshev多项式近似估计3D网格（网格损失 L_m），深度证据网络以两层GCN加全连接层回归关节深度（损失 L_d）。\n- 由粗到细网格生成：利用heavy-edge matching算法将网格图三级粗化并记录相邻层节点映射，前向传播时先构建最粗网格再逐级上采样，使模型同时学习局部与全局特征。\n- 多视图融合机制：各视图的单视图3D预测在坐标通道上拼接后输入graph U-Net，以L2距离损失预测最终3D手部关键点，整个架构可端到端训练。\n- 训练设置：采用多阶段训练策略（先训2D证据网络、固定其权重训网格网络、再固定两者训深度网络、最后整体端到端优化），batch size为8，Adam初始学习率0.01，在4块GeForce GTX 1080-Ti GPU上实现。\n- 评估协议：在MHP（21段真实多视图视频、Leap Motion自动标注）与STB SK子集（6段视频、人工标注、首段作验证集）上评估，图像经裁剪去背景、居中并缩放至256×256，手部中心从掌心改为手腕关节。\n- 关键结果：多视图方法配合MVHM在MHP上达AUC_{20-50}=0.990、EPE=5.20mm，超越Zimmermann等(0.717)、Cai等(0.928)与Chen等(0.939)；消融显示MVHM使单视图在MHP上的EPE从22.13mm降至18.09mm、AUC_{20-50}从0.802升至0.857。\n- 适用约束：多视图部署依赖已知相机参数，合成数据与真实图像存在域差距，MANO低维参数化可能丢失高维手部信息；潜在应用涵盖人机交互、VR、AR、医疗诊断与手语理解。",
  "available_templates": [
    "deep_read",
    "simple"
  ]
}