{
  "paper_id": "c392d04ca72678b74db9fd81add0247e",
  "paper_title": "HHMR: Holistic Hand Mesh Recovery by Enhancing the Multimodal Controllability of Graph Diffusion Models",
  "paper_authors": [
    "Mengcheng Li",
    "Hongwen Zhang",
    "Yuxiang Zhang",
    "Ruizhi Shao",
    "Tao Yu",
    "Yebin Liu"
  ],
  "publication_date": "",
  "publication_venue": "2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)",
  "abstract": "Recent years have witnessed a trend of the deep integration of the generation and reconstruction paradigms. In this paper, we extend the ability of controllable generative models for a more comprehensive hand mesh recovery task: direct hand mesh generation, inpainting, reconstruction, and fitting in a single framework, which we name as Holistic Hand Mesh Recovery (HHMR). Our key observation is that different kinds of hand mesh recovery tasks can be achieved by a single generative model with strong multimodal controllability, and in such a framework, realizing different tasks only requires giving different signals as conditions. To achieve this goal, we propose an all-in-one diffusion framework based on graph convolution and attention mechanisms for holistic hand mesh recovery. In order to achieve strong control generation capability while ensuring the decoupling of multimodal control signals, we map different modalities to a shared feature space and apply cross-scale random masking in both modality and feature levels. In this way, the correlation between different modalities can be fully exploited during the learning of hand priors. Furthermore, we propose Condition-aligned Gradient Guidance to enhance the alignment of the generated model with the control signals, which significantly improves the accuracy of the hand mesh reconstruction and fitting. Experiments show that our novel framework can realize multiple hand mesh recovery tasks simultaneously and outperform the existing methods in different tasks, which provides more possibilities for subsequent downstream applications including gesture recognition, pose generation, mesh editing, and so on.",
  "keywords": [
    "手部网格恢复",
    "扩散模型",
    "图卷积网络",
    "多模态可控生成",
    "条件对齐梯度引导",
    "随机掩码策略",
    "手部姿态估计",
    "网格补全"
  ],
  "paper_institutions": [
    "Department of Automation, Tsinghua University",
    "School of Artificial Intelligence, Beijing Normal University",
    "Beijing National Research Center for Information Science and Technology, Tsinghua University"
  ],
  "output_language": "zh",
  "provider": "doubao",
  "model": "glm-5.3",
  "prompt_template": "simple",
  "extracted_at": "2026-09-17T09:31:55.378003Z",
  "template_tag": "simple",
  "summary": "### 摘要\n\n手部网格恢复旨在从图像中恢复手部姿态与网格，广泛应用于AR/VR、人机交互与机器人领域。传统研究经历了从基于关键点的网格与模型拟合到基于学习的回归范式的转变，但单向回归策略无法建立或充分利用自然手部姿态与网格的底层分布，难以支持网格补全与生成等高级任务；而基于VAE/GAN的隐式手部模型多构建于参数空间或3D骨架空间，难以直接生成网格且需要耗时的反演或拟合过程。为此，本文提出HHMR（Holistic Hand Mesh Recovery）一体化框架，基于图卷积（GCN）与注意力机制的图扩散模型，将不同模态条件（RGB图像、2D/3D骨架）映射到共享特征空间，并在模态级与特征级施加跨尺度随机掩码策略，实现多模态控制信号的解耦与相关性学习；同时提出条件对齐梯度引导机制，在反向扩散过程的高斯分布均值上添加梯度偏置：$\\bar{\\mu}_{t}=\\mu_{t}-s\\Sigma_{t}\\triangledown_{x_{t}}||P f_{\\theta}(x_{t},\\mathbf{c},t)-P x_{0}||$，其中P为任务特定算子（如关节回归矩阵或顶点二值掩码），无需额外分类器网络即可增强生成结果与控制信号的对齐。训练混合使用Two-hand 500K、InterHand2.6M姿态数据集以及FreiHand、HO3D_V3、CompHand重建数据集，以覆盖不同条件类型并验证模型在纯姿态与图像-网格配对数据上的泛化能力；实现上采用PyTorch在单张NVIDIA RTX 4090 GPU上以AdamW优化器、批大小64、初始学习率5e-4（余弦退火至1e-5）训练100万批次，训练时去噪步数为1000，推理采用DDIM加速至T=10步。评估指标包括衡量生成多样性与真实性的APD（平均成对距离）与SI（自相交面比例），以及衡量重建与拟合精度的MPJPE与MPVPE；结果显示生成任务APD为17.30mm、SI为0.05%优于PCA采样（15.00mm、0.39%），单假设重建在FreiHand上达到MPJPE/MPVPE均为6.0mm与SOTA相当，多假设重建（n=8）达5.9mm超越SOTA，2D拟合结合图像与骨架条件及梯度引导时MPVPE降至5.05mm，验证了共享特征空间映射、两级随机掩码与梯度引导等创新的有效性。未来可进一步探索手势识别、姿态生成、网格编辑等下游应用；实际约束方面，该方法在极端缺失或过度噪声的输入条件下仍可能失败，且增加去噪步数以获得更精细的生成结果会带来显著的时间开销。\n\n### 关键要点\n\n- 提出基于图扩散模型的All-In-One框架HHMR，可在单一框架内同时实现手部网格直接生成、补全、重建与拟合，无需任何额外微调或反演步骤。\n- 网络主体为U形结构，由下采样编码器与上采样解码器及跳跃连接组成，每个块包含GCN层、自注意力层、交叉注意力层及可选的上/下采样层，直接在3D网格空间而非参数空间上处理。\n- 将不同模态条件映射到共享特征空间：RGB图像经ResNet50编码并裁剪为8×8 patch后由ViT处理（含全局token），2D/3D骨架由三层MLP编码，拼接后输入交叉注意力层。\n- 设计两级随机掩码策略：模态级以概率独立置空各条件并以概率全局掩码训练无条件生成能力；特征级对图像patch、骨架关节加噪与手指随机掩码，以处理复杂遮挡与不完整骨架输入。\n- 提出条件对齐梯度引导，在反向过程均值上添加梯度偏置，通过任务特定算子P（关节回归矩阵、顶点掩码、投影算子组合等）实现神经拟合式几何控制，无需额外分类器网络与额外训练，即插即用。\n- 训练损失由三部分组成：扩散数据损失（L1）、顶点与关节损失（基于预训练关节回归矩阵的L1监督）、网格平滑损失（法向一致性与边长L1距离）。\n- 训练数据混合使用Two-hand 500K与InterHand2.6M姿态数据集以及FreiHand、HO3D_V3、CompHand重建数据集，在单张RTX 4090上以AdamW优化器、批大小64、学习率5e-4余弦退火至1e-5训练100万批次。\n- 生成任务中随机生成500个手部网格，APD为17.30mm、SI为0.05%，在多样性（APD 15.00mm）与真实性（SI 0.39%）上均优于MANO PCA空间随机采样。\n- 在FreiHand数据集上，单假设重建（DDIM T=10步，n=1）MPJPE/MPVPE均为6.0mm，与MeshGraphormer等SOTA相当；多假设重建n=8时达5.9mm、n=16时达5.8mm，超越现有方法。\n- 消融实验验证GCN模块有效性：使用GCN时MPJPE为5.99mm，将GCN全部替换为自注意力层（Only SA）则退化至6.39mm。\n- 2D骨架拟合任务中，结合图像条件、2D骨架条件与梯度引导时MPVPE降至5.05mm、MPJPE为4.94mm，显著优于传统基于PCA先验的拟合算法（MPVPE 7.00mm）。\n- 局限性在于：面对极端缺失或过度噪声的输入条件模型仍可能失败；虽然T=10步去噪即可获得较好结果，但增加去噪步数以提升生成精度较为耗时。"
}