{
  "paper_id": "983fddaa0f9a7f49e34a72fa29ab31f3",
  "paper_title": "Hamba: Single-view 3D Hand Reconstruction with Graph-guided Bi-Scanning Mamba",
  "paper_authors": [
    "Haoye Dong",
    "Aviral Chharia",
    "Wenbo Gou",
    "Francisco Vicente Carrasco",
    "Fernando De la Torre"
  ],
  "publication_date": "2024",
  "publication_venue": "NeurIPS",
  "abstract": "3D Hand reconstruction from a single RGB image is challenging due to the articulated motion, self-occlusion, and interaction with objects. Existing SOTA methods employ attention-based transformers to learn the 3D hand pose and shape, yet they do not fully achieve robust and accurate performance, primarily due to inefficiently modeling spatial relations between joints. To address this problem, we propose a novel graph-guided Mamba framework, named Hamba, which bridges graph learning and state space modeling. Our core idea is to reformulate Mamba's scanning into graph-guided bidirectional scanning for 3D reconstruction using a few effective tokens. This enables us to efficiently learn the spatial relationships between joints for improving reconstruction performance. Specifically, we design a Graph-guided State Space (GSS) block that learns the graph-structured relations and spatial sequences of joints and uses 88.5% fewer tokens than attention-based methods. Additionally, we integrate the state space features and the global features using a fusion module. By utilizing the GSS block and the fusion module, Hamba effectively leverages the graph-guided state space features and jointly considers global and local features to improve performance. Experiments on several benchmarks and in-the-wild tests demonstrate that Hamba significantly outperforms existing SOTA, achieving the PA-MPVPE of 5.3mm and F@15mm of 0.992 on FreiHAND. At the time of this paper's acceptance, Hamba holds the top position, Rank 1, in two competition leaderboards on 3D hand reconstruction.",
  "keywords": [
    "3D手部重建",
    "Mamba",
    "状态空间模型",
    "图学习",
    "图引导双向扫描",
    "MANO",
    "单视角RGB图像"
  ],
  "paper_institutions": [
    "Carnegie Mellon University"
  ],
  "output_language": "zh",
  "provider": "doubao",
  "model": "glm-5.3",
  "prompt_template": "simple",
  "extracted_at": "2026-09-17T09:28:33.171974Z",
  "template_tag": "simple",
  "summary": "### 摘要\n\n从单张RGB图像重建3D手部网格因关节化运动、自遮挡以及手-物/手-手交互而极具挑战，现有SOTA方法（METRO、MeshGraphormer、HaMeR等基于注意力的Transformer）因对关节间空间关系建模低效且需处理大量token，难以在真实场景中实现鲁棒精确的重建。为此，论文提出Hamba——首个融合图学习与状态空间建模的Mamba框架，核心思想是将Mamba的单向扫描重构为图引导双向扫描（GBS），仅用少量有效token（较注意力方法减少88.5%）高效学习关节间空间关系；具体设计了图引导状态空间（GSS）模块，通过图卷积层与SS2D Mamba块学习关节的图结构关系与空间序列，并配合Token Sampler（TS）与融合模块联合利用全局与局部特征，最终由MLP回归MANO参数生成778×3的手部网格。模型在与HaMeR相同的270万样本多数据集（FreiHAND、HO3D、MTC、RHD、InterHand2.6M、H2O3D、DexYCB、COCO-Wholebody、Halpe、MPII NZSL）上训练以实现公平比较，采用ViT-H骨干（630M参数）、AdamW优化器（学习率1e-5、权重衰减1e-4）、2块A6000 GPU、batch size 56训练170k步，该设置既保证与SOTA的可比性，又验证了方法在室内基准与野外场景下的泛化能力。评估采用PA-MPJPE、PA-MPVPE、F@5mm、F@15mm、AUC_J/AUC_V及PCK等指标，Hamba在FreiHAND上取得PA-MPVPE 5.3mm与F@15mm 0.992（含TTA），在HO3Dv2/v3及未训练过的HInt野外基准上全面超越现有SOTA，并在论文接收时占据两个3D手部重建竞赛排行榜第一，这些结果支撑了图引导状态空间建模与token高效设计两项核心创新的有效性。核心机制为GSS块的计算：T_GSS_l = FFN(LN(SS2D(T_GCN_l^c)+T_GCN_l^c))+SS2D(T_GCN_l^c)+T_GCN_l^c，其中GCN输出与全局均值token拼接后经双向SS2D扫描并辅以残差连接。未来方向包括探索视频时序特征（当前因视频3D标注采集成本高而受限）；应用场景涵盖机器人、动画、人机交互与AR/VR，但需注意模型可能被滥用于未经授权的监控或隐私侵犯，且在严重运动模糊、掌心朝向歧义等约束下仍存在失败案例。\n\n### 关键要点\n\n- Hamba首次将图学习与状态空间建模（SSM）结合用于单视角3D手部网格重建，将Mamba的单向扫描重构为图引导双向扫描（GBS），以少量有效token捕获关节间的语义与空间关系。\n- GSS（图引导状态空间）模块由图卷积层（PGraphConv+BatchNorm+ReLU，邻接矩阵基于MANO手部骨架定义）与Mamba块（SS2D+LayerNorm+FFN）构成，GCN输出与全局均值token拼接后输入SS2D进行双向扫描。\n- Token Sampler利用Joints Regressor（4个VSS块+MLP）预测的MANO参数经焦距5000mm透视投影得到2D关节位置，再通过双线性插值采样有效token以避免背景特征干扰；融合模块整合状态空间token、全局均值特征、采样token与2D关节特征后回归最终MANO参数。\n- 训练使用270万样本，混合FreiHAND、HO3D、MTC、RHD、InterHand2.6M、H2O3D、DexYCB、COCO-Wholebody、Halpe与MPII NZSL共10个数据集，采样权重与HaMeR一致（FreiHAND与InterHand2.6M为0.25，MTC与COCO Wholebody为0.1，其余为0.05）。\n- 实现细节：ViT-H骨干（50层Transformer、630M参数）输入256×192×3图像，token经Conv2D从1280维降至512维；最佳模型使用4个GSS块；损失为2D/3D关节损失（L1）、姿态/形状损失（L2）与对抗损失的加权和，权重分别为λ_3D=0.05、λ_2D=0.01、λ_θ=0.001、λ_β=λ_adv=0.0005。\n- FreiHAND基准上Hamba（含TTA）达PA-MPJPE 5.7mm、PA-MPVPE 5.3mm、F@5mm 0.806、F@15mm 0.992，均优于HaMeR（6.0/5.7/0.785/0.990）等SOTA；论文接收时在HO3Dv2与HO3Dv3两个竞赛排行榜均排名第一。\n- HO3Dv2上取得PA-MPJPE 7.5mm、PA-MPVPE 7.7mm、AUC_J 0.850、AUC_V 0.846；HO3Dv3上PA-MPJPE 6.9mm、PA-MPVPE 6.8mm、AUC_J 0.861、AUC_V 0.864，全面超越HandBooster、HaMeR等对比方法。\n- 在未参与训练的HInt野外基准上，Hamba在NewDays、VISOR、Ego4D三个子集的全关节、可见关节与遮挡关节PCK指标上均超过HaMeR（如VISOR可见关节@0.05为61.4对56.6），证明跨数据集泛化与遮挡鲁棒性。\n- 消融实验：移除GSS token分支使F@5mm从0.738降至0.717；移除GCN或Mamba块均使PA-MPJPE从6.6升至7.3、F@5mm分别降至0.673/0.675，证明图学习与状态空间建模同等重要；将双向扫描替换为单向扫描或移除图引导扫描亦导致性能下降。\n- 效率对比：相比GCN+Transformer方案，GCN+SS2D将扫描token从192减至22（减少88.5%），解码器参数减少51.8%、解码器FLOPs减少21.8%、解码器运行时间减少46.1%、GPU显存占用减少83.7%，且各项精度指标更优。\n- 局限与约束：约95%训练数据来自受控室内环境；模型在严重运动模糊、掌心朝向歧义（可能旋转180°）、检测器误检或复杂手指手势下仍会失败；当前方法未探索视频时序特征，且存在被滥用于监控或侵犯隐私的潜在风险。\n- 迁移能力：GSS块可作为即插即用模块迁移至3D人体网格恢复（HMR）任务，在LSP-Extended与COCO上PCK优于HMR2.0b，3DPW上MPJPE 81.7mm与之相当（81.3mm），尽管仅用单GPU训练300k步。"
}