{
  "paper_id": "c240ec81afbb5f6e4043efb96d8bf1ad",
  "paper_title": "Hand-Transformer: Non-Autoregressive Structured Modeling for 3D Hand Pose Estimation",
  "paper_authors": [
    "Lin Huang",
    "Jianchao Tan",
    "Ji Liu",
    "Junsong Yuan"
  ],
  "publication_date": "",
  "publication_venue": "ECCV 2020 (Lecture Notes in Computer Science)",
  "abstract": "3D hand pose estimation is still far from a well-solved problem mainly due to the highly nonlinear dynamics of hand pose and the difficulties of modeling its inherent structural dependencies. To address this issue, we connect this structured output learning problem with the structured modeling framework in sequence transduction field. Standard transduction models like Transformer adopt an autoregressive connection to capture dependencies from previously generated tokens and further correlate this information with the input sequence in order to prioritize the set of relevant input tokens for current token generation. To borrow wisdom from this structured learning framework while avoiding the sequential modeling for hand pose, taking a 3D point set as input, we propose to leverage the Transformer architecture with a novel non-autoregressive structured decoding mechanism. Specifically, instead of using previously generated results, our decoder utilizes a reference hand pose to provide equivalent dependencies among hand joints for each output joint generation. By imposing the reference structural dependencies, we can correlate the information with the input 3D points through a multi-head attention mechanism, aiming to discover informative points from different perspectives, towards each hand joint localization. We demonstrate our model's effectiveness over multiple challenging hand pose datasets, comparing with several state-of-the-art methods.\n",
  "keywords": [
    "3D hand pose estimation",
    "Structured learning",
    "Attention",
    "Non-autoregressive transformer"
  ],
  "paper_institutions": [
    "State University of New York, Buffalo, USA",
    "Y-tech, Kwai Inc., Beijing, China"
  ],
  "output_language": "zh",
  "provider": "doubao",
  "model": "doubao-seed-2.1-pro",
  "prompt_template": "simple",
  "extracted_at": "2026-10-20T04:22:52.655055Z",
  "template_tag": "simple",
  "summary": "### 摘要\n\n本文针对从深度图转换的三维点云进行手部姿态估计的问题，重点解决手部运动高度非线性、全局取向变化大、手指自相似与严重自遮挡导致关节结构依赖难以建模的困难。现有方法常把各关节视为独立三维坐标，或依赖预定义运动学模型与手工先验，难以覆盖复杂多样的手部关联；标准Transformer虽可用自回归解码建模输出token依赖，却带来逐关节串行延迟，并把关节关联偏置为固定先后顺序。为此，作者首次将结构化手部姿态估计与自然语言处理中的序列转导框架连接，提出非自回归Hand Transformer（NARHT）。点云是深度数据的简单有效表示，可直接利用三维几何，输入点经OBB坐标系统归一化以增强对手部取向的鲁棒性；PointNet与自注意力编码器生成增强的点特征，PointNet++结构化参考提取器生成J个关节级参考特征，解码器通过非因果自注意力建模全部参考关节之间的依赖，再以这些依赖为查询、经多头编码器-解码器注意力关注输入点云，自适应聚合逐点投票信息并并行回归所有关节坐标，其核心机制可写为 p_narht(Φ|P;θ)=∏_{j=1}^J p(φ_j|φ̃_{1:J},p_{1:N};θ)。模型在ICVL、MSRA、NYU与HANDS 2017上评估，这些数据集涵盖不同受试者、手势、视角、已见/未见主体测试和16、14、21关节标注，可检验精度、鲁棒性与跨主体泛化；实现中采样1024个点并拼接估计的三维法向，编码器为PointNet加3层Transformer编码器、8头、d_model=256，参考提取器为PointNet++，解码器为6层非因果Transformer，使用L=10L1+L2的平方L2联合损失、Adam初始学习率1e-3、批大小16并在单张NVIDIA TITAN Xp上训练。评价采用平均每关节3D欧氏误差（mm）和所有关节均落在指定阈值内的好帧比例；NARHT在ICVL、MSRA、NYU的平均误差分别为6.47、7.55、9.80 mm，单GPU推理速度为43.2 FPS，虽在部分数据集平均误差上并非绝对最低，但在ICVL阈值大于20 mm、MSRA大于5 mm、NYU大于30 mm时好帧比例优于已有方法，并在HANDS 2017未见主体上表现出较强泛化。消融显示自回归Transformer误差为8.57 mm、坐标型参考姿态为6.67 mm、Point-to-Point编码器为6.70 mm，而完整NARHT为6.47 mm，且注意力编码器平均每帧3.80 ms，快于Point-to-Point编码器的35.90 ms，支持并行结构化解码、特征型参考姿态和自适应点投票的有效性。该方法适用于人机交互、机器人、虚拟现实/增强现实等需要实时深度手部姿态估计的场景，但当前约束是依赖可转换为点云的深度输入，并受遮挡、噪声和参考姿态近似质量影响；未来可探索RGB手部姿态估计、图像描述以及进一步连接姿态估计与自然语言处理结构化输出学习。\n\n### 关键要点\n\n- NARHT以从手部深度图转换并归一化到OBB坐标系的无序三维点集为输入，输出一组三维手部关节坐标。\n- 编码器先用基础PointNet提取逐点特征，再用置换不变的自注意力增强点间长程依赖，以更直接地保留关节定位所需信息。\n- 结构化参考提取器采用PointNet++将输入点映射为J个关节级特征，并通过额外全连接层和L2损失对参考姿态进行中间监督。\n- 解码器将传统Transformer的掩码自注意力替换为非因果自注意力，使每个参考关节可以关注其他参考关节而不受串行生成顺序限制。\n- 编码器-解码器多头注意力以参考关节依赖为查询、以输入点特征为键和值，从多个表示子空间选择对特定关节有贡献的点。\n- 该注意力机制构成完全自适应的逐点投票方案，不局限于基于欧氏偏移的固定投票，可由模型自主判断应关注的方面。\n- 训练目标为最终关节预测损失L1与参考姿态辅助损失L2的加权组合L=10L1+L2，两项均采用预测坐标与真值之间的平方L2误差。\n- ICVL包含22k训练帧、1.5k测试帧和16个关节；MSRA包含9个受试者、76.5k深度图和21个关节；NYU包含72k训练帧、8.2k测试帧并使用正面视角的14个关节。\n- HANDS 2017包含957k训练帧和295k测试帧，提供21个三维关节标注，测试包含已见和未见受试者情形。\n- 主要量化结果包括ICVL 6.47 mm、MSRA 7.55 mm、NYU 9.80 mm的平均关节误差，以及单GPU 43.2 FPS的推理速度。\n- 在好帧比例指标上，论文称阈值分别大于ICVL的20 mm、MSRA的5 mm、NYU的30 mm时，NARHT相较此前方法具有一定优势。\n- 方法适用于具有深度传感器的实时场景，但在严重自遮挡、点云噪声或参考姿态质量受限时，估计精度仍可能受到影响。"
}