{
  "paper_id": "73bc99c744cf147fe97d619e4806973e",
  "paper_title": "Fast-HaMeR: Boosting Hand Mesh Reconstruction using Knowledge Distillation",
  "paper_authors": [
    "Hunain Ahmed Jillani",
    "Ahmed Tawfik Aboukhadra",
    "Ahmed Elhayek",
    "Jameel Malik",
    "Nadia Robertini",
    "Didier Stricker"
  ],
  "publication_date": "",
  "publication_venue": "Proceedings of the 21st International Conference on Computer Vision Theory and Applications",
  "abstract": "Fast and accurate 3D hand reconstruction is essential for real-time applications in VR/AR, human-computer interaction, robotics, and healthcare. Most state-of-the-art methods rely on heavy models, limiting their use on resource-constrained devices like headsets, smartphones, and embedded systems. In this paper, we investigate how the use of lightweight neural networks, combined with Knowledge Distillation, can accelerate complex 3D hand reconstruction models by making them faster and lighter, while maintaining comparable reconstruction accuracy. While our approach is suited for various hand reconstruction frameworks, we focus primarily on boosting the HaMeR model, currently the leading method in terms of reconstruction accuracy. We replace its original ViT-H backbone with lighter alternatives, including MobileNet, MobileViT, ConvNeXt, and ResNet, and evaluate three knowledge distillation strategies: output-level, feature-level, and a hybrid of both. Our experiments show that using lightweight backbones that are only 35% the size of the original achieves 1.5x faster inference speed while preserving similar performance quality with only a minimal accuracy difference of 0.4mm. More specifically, we show how output-level distillation notably improves student performance, while feature-level distillation proves more effective for higher-capacity students. Overall, the findings pave the way for efficient real-world applications on low-power devices. The code and models are publicly available under https://github.com/hunainahmedj/Fast-HaMeR.",
  "keywords": [
    "3D Hand Reconstruction",
    "Pose Estimation",
    "Knowledge Distillation",
    "Real-Time Inference"
  ],
  "paper_institutions": [
    "RPTU, Kaiserslautern, Germany",
    "DFKI-AV, Kaiserslautern, Germany",
    "UPM, Saudi Arabia",
    "NUST-SEECS, Islamabad, Pakistan"
  ],
  "output_language": "zh",
  "provider": "doubao",
  "model": "glm-5.3",
  "prompt_template": "simple",
  "extracted_at": "2026-09-17T09:30:26.131717Z",
  "template_tag": "simple",
  "summary": "### 摘要\n\n快速且准确的3D手部重建对VR/AR、人机交互、机器人与医疗等实时应用至关重要，但以HaMeR为代表的最先进方法依赖重型ViT-H骨干网络（671M参数、仅27 FPS），难以部署到头显、智能手机和嵌入式系统等资源受限设备。以往单目手部重建研究分为参数化模型驱动（借助MANO等解剖先验）与直接网格恢复（如METRO、Mesh Graphormer、THOR-Net、Hamba）两类，而知识蒸馏虽已广泛用于大语言模型压缩，在3D重建与姿态估计中却鲜有系统探索。为此，本文提出Fast-HaMeR：以冻结权重的HaMeR为教师网络，将其ViT-H骨干替换为MobileNet、MobileViT、ConvNeXt、ResNet等轻量替代，并系统评估输出级、特征级及二者混合三种蒸馏策略，其核心机制为总损失 L_total = L_GT + λ_KD·L_KD（输出级蒸馏损失 L_KD,out = ||K_S^3D − K_T^3D||² + ||K_S^2D − K_T^2D||² + ||Θ_S − Θ_T||²，特征级蒸馏则通过可学习1×1卷积φ对齐教师特征通道维度并以双线性插值对齐空间尺寸后最小化 ||F_S − φ(F_T)||²）。训练沿用HaMeR约270万标注样本的组合数据集（FreiHAND、HO3D、InterHand2.6M、DexYCB、COCO WholeBody等）及其采样分布，以支撑大规模监督训练并保持与教师一致的数据基础；评估在具有挑战性手-物交互场景的HO3D-v2上进行，便于与HaMeR及其他基线直接对比，所有实验与FPS测算均在消费级RTX 4060 Ti GPU上完成。采用PA-MPJPE（J_err）、PA-MPVPE（V_err）、F@5mm与F@15mm指标，最佳配置ConvNeXt-L配合特征级蒸馏（λ_KD=0.8、γ_FD=12）取得J_err 8.1mm、V_err 8.3mm，模型仅为原规模35%、推理提速1.5倍、精度差仅0.4mm，验证了轻量骨干与蒸馏结合的有效性；实验进一步表明输出级蒸馏对小型或结构对齐的学生更有效，特征级蒸馏更利于高容量学生，混合蒸馏通常不优于单一策略。未来可探索注意力蒸馏、空间/通道级蒸馏及关系知识蒸馏等更具表达力的特征级增强方式。该工作为低功耗设备上的实时手部重建提供了实用方案，适用于VR/AR、人机交互、手语识别等场景，但需在精度与效率间权衡，学生模型仍存在轻微精度损失。\n\n### 关键要点\n\n- **目标问题**：最先进3D手部重建模型（如HaMeR，671M参数、27 FPS）计算开销大，难以在头显、智能手机等资源受限设备上实时运行；本文通过轻量网络与知识蒸馏在保持相近精度的同时加速推理。\n- **教师模型**：HaMeR采用ViT-H骨干加transformer解码器回归MANO参数（θ∈R^48、β∈R^10）与相机参数，经MANO头输出3D网格与21个关键点，训练使用重投影损失L_2D、3D关键点损失L_3D与MANO参数损失L_MANO（合称L_GT）。\n- **学生骨干选择**：以MobileNet、MobileViT、ConvNeXt、ResNet替换ViT-H，依据为ImageNet-1K相对性能、torchvision中的参数量与运行效率，分别代表低参数量/深度可分离卷积、轻量混合transformer、现代分层卷积与残差连接等设计理念。\n- **三种蒸馏策略**：输出级蒸馏在教师与学生的预测（3D/2D关键点、MANO参数）间施加L2损失；特征级蒸馏经可学习1×1卷积φ投影教师特征通道并用双线性插值对齐空间尺寸后最小化特征差异；混合蒸馏同时施加两者。\n- **核心损失公式**：L_total = L_GT + λ_KD·L_KD，其中λ_KD控制蒸馏贡献，特征级蒸馏额外用γ_feat缩放特征损失；输出级蒸馏损失为 ||K_S^3D−K_T^3D||² + ||K_S^2D−K_T^2D||² + ||Θ_S−Θ_T||²。\n- **训练数据**：沿用HaMeR约270万标注样本的组合数据集与采样分布，涵盖FreiHAND、HO3D、H2O3D、InterHand2.6M、MTC、DexYCB、RHD（3D标注）及COCO WholeBody、Haple-FullBody、MPII+NZSL（2D标注）。\n- **评估设置**：在HO3D-v2上定量评估，因其提供真实且具挑战性的手-物交互场景并便于与HaMeR等基线直接比较；指标为PA-MPJPE（J_err）、PA-MPVPE（V_err）、F@5mm与F@15mm；所有实验在消费级RTX 4060 Ti GPU上进行。\n- **无蒸馏基线**：轻量骨干呈现明确的精度-复杂度权衡——ConvNeXt-L以约64%参数削减和1.48倍提速仅损失0.6mm精度，ResNet-50以10%参数量实现1.85倍提速、精度差约1mm，表明数据扩展可能比单纯增大模型更有益。\n- **蒸馏策略发现**：输出级蒸馏对MobileNet-L无改善甚至最多退化0.3mm，但对与教师结构相似的MobileViT-S（λ_KD=0.5）带来0.4mm的PA-MPVPE提升；特征级蒸馏仅令高容量的ConvNeXt-L受益（λ_KD=0.8、γ_FD=12时PA-MPVPE改善0.2mm）；混合蒸馏普遍介于两种单一策略之间且不超越各自最佳。\n- **最终性能**：最佳配置ConvNeXt-L+特征级蒸馏取得J_err 8.1mm、V_err 8.3mm、F@5mm 0.599、F@15mm 0.979，优于HandOccNet、METRO、ArtiBoost、I2L-MeshNet等方法，模型规模为HaMeR的35%、推理快1.5倍、精度差仅0.4mm。\n- **约束与展望**：学生模型仍存在轻微精度损失（定性结果显示HaMeR更贴合图像中的手部）；方法适用于VR/AR、人机交互、手语识别等低功耗实时场景；未来可探索注意力蒸馏、空间/通道级蒸馏与关系知识蒸馏以增强特征级蒸馏效果。"
}