{
  "paper_id": "96192396226b12bad2af58129ceb3f85",
  "paper_title": "Embodied Hands: Modeling and Capturing Hands and Bodies Together",
  "paper_authors": [
    "Javier Romero",
    "Dimitrios Tzionas",
    "Michael J. Black"
  ],
  "publication_date": "2017-11",
  "publication_venue": "ACM Transactions on Graphics (TOG), Vol. 36, No. 6, Article 245",
  "abstract": "Humans move their hands and bodies together to communicate and solve tasks. Capturing and replicating such coordinated activity is critical for virtual characters that behave realistically. Surprisingly, most methods treat the 3D modeling and tracking of bodies and hands separately. Here we formulate a model of hands and bodies interacting together and fit it to full-body 4D sequences. When scanning or capturing the full body in 3D, hands are small and often partially occluded, making their shape and pose hard to recover. To cope with low-resolution, occlusion, and noise, we develop a new model called MANO (hand Model with Articulated and Non-rigid deformations). MANO is learned from around 1000 high-resolution 3D scans of hands of 31 subjects in a wide variety of hand poses. The model is realistic, low-dimensional, captures non-rigid shape changes with pose, is compatible with standard graphics packages, and can fit any human hand. MANO provides a compact mapping from hand poses to pose blend shape corrections and a linear manifold of pose synergies. We attach MANO to a standard parameterized 3D body shape model (SMPL), resulting in a fully articulated body and hand model (SMPL+H). We illustrate SMPL+H by fitting complex, natural, activities of subjects captured with a 4D scanner. The fitting is fully automatic and results in full body models that move naturally with detailed hand motions and a realism not seen before in full body performance capture. The models and data are freely available for research purposes.",
  "keywords": [
    "Hands",
    "Human body shape",
    "3D shape",
    "Learning",
    "Performance capture",
    "Motion capture"
  ],
  "paper_institutions": [
    "Body Labs Inc.",
    "Max Planck Institute for Intelligent Systems"
  ],
  "output_language": "zh",
  "provider": "doubao",
  "model": "glm-5.3",
  "prompt_template": "simple",
  "extracted_at": "2026-09-17T09:35:47.267581Z",
  "template_tag": "simple",
  "summary": "### 摘要\n\n本文解决的核心问题是:手部与身体在运动中密不可分,但以往研究将二者的3D建模与跟踪分开处理——SCAPE、SMPL等统计身体模型的手部关节有限或缺失,而手部跟踪方法(基于RGB-D、标记、数据手套等)要么只捕获孤立手部、要么具有侵入性,导致全身扫描中小尺寸、常被遮挡的手部形状与姿态难以恢复。为此作者提出MANO(具有关节与非刚性变形的手部模型),基于SMPL顶点建模框架,核心机制为 M(β,θ)=W(T_P(β,θ), J(β), θ, 𝒲),其中 T_P=T̄+B_S(β)+B_P(θ),即用形状混合形状B_S刻画身份差异、用姿态相关修正混合形状B_P(以旋转矩阵元素为输入的线性组合)捕捉非刚性姿态变形,并通过基于测地距离的顶点-关节依赖正则化促使修正局部化,再以PCA得到每只手6维姿态线性流形。数据方面,使用3dMDhand系统(约5万顶点分辨率、0.2mm RMS精度)采集31名受试者左右手共2018个高分辨率扫描(最多51种姿态,含Feix抓取分类),左手镜像为右手以加倍数据并消除利手偏差,经两轮\"配准-建模\"迭代后1554个配准成功;选择高分辨率专用手部扫描器是为了在隔离条件下捕获手部变形细节,验证模型能从数据中学习真实、低维且可泛化的表示。将MANO与SMPL结合得到SMPL+H(78自由度,手部形状空间取自全身模型以捕获手-身体形态相关性),并修改4Cap时序配准算法,加入GMM手部姿态先验与零速度正则项,在3dMDbody.u系统(60fps)采集的41段4D序列(共27156帧)上拟合。评估指标包括紧致性、泛化性与消融实验:6/10/15个PCA分量分别解释81%/90%/95%姿态方差;全空间scan-to-mesh误差0.93mm,镜像增强数据集(0.93mm)优于仅右手数据集(1.05mm),启用姿态混合形状(0.93mm)优于禁用(1.3mm),学习参数使误差从2.90mm降至1.01mm(提升超60%),这些结果支持了镜像增强、姿态混合形状与参数学习等创新;SMPL+H相比SMPL在视觉真实感上显著提升,在手部数据完全缺失、快速运动或严重噪声下仍能自然捕获。未来方向包括手-物交互与自接触建模、用MANO/SMPL+H生成深度学习训练数据、学习身体-手部运动关联映射。实际应用场景涵盖高端无标记表演捕获、兼容现有动画引擎的角色制作及实时应用(线性混合形状的内存与运行时与LBS相当);约束包括低维姿态空间无法表示异常手部姿态、手-物交互未显式建模、捕获环境复杂昂贵等。\n\n### 关键要点\n\n- MANO基于SMPL框架构建,包含模板网格、运动树(15个关节加全局朝向)、形状与姿态混合形状、蒙皮权重和关节回归器,与标准图形软件兼容,可单独使用或与身体模型结合。\n- 手部数据使用3dMDhand主动立体系统采集(5个扫描单元、约5万顶点分辨率、0.2mm RMS精度),共31名受试者的2018个扫描,涵盖关节探索姿态、Feix抓取分类的31种姿态及混合姿态,并包含手与物体交互的抓取扫描。\n- 左手扫描镜像为右手以虚拟加倍训练数据并消除利手偏差,训练单一右手模型后再镜像生成左手模型;实验显示增强数据集误差(0.93mm)低于仅右手数据集(1.05mm)。\n- 姿态混合形状通过测地距离依赖的正则化 Λ_P(i,j)=J_j·D_geo(t_i,T̄) 保持局部性,修正线性混合蒙皮的关节塌陷伪影;启用时误差0.93mm,禁用(纯LBS)时为1.3mm。\n- 姿态空间经PCA降维,6个分量解释约81%训练姿态方差,在精度与鲁棒性间权衡:干净的手部数据可用更多分量,噪声全身数据则受益于少分量带来的额外正则化。\n- SMPL+H将MANO的手指关节、蒙皮权重与姿态变形并入SMPL身体模型,姿态自由度从66增至78,手部形状空间取自全身模型以捕获手与身体形态间的相关性。\n- 4D配准采用两阶段方法:先用前50帧(受试者近似静止)构建受试者特定模板,再以GMM手部姿态先验(N=10)和零速度正则项进行序列配准,防止数据完全缺失时的突然手部运动;优化采用dogleg拟牛顿法,每帧约4分钟。\n- 定量评估基于6名未参与训练受试者的50个右手扫描测试集:全姿态空间s2m误差0.93mm;未训练模型误差2.90mm,学习参数后降至1.01mm(提升超60%)。\n- SMPL+H在41段4D序列(共27156帧,含即兴动作、篮球、手指计数、键盘打字及专业演员的恐惧表达)上评估,相比SMPL始终平坦张开的手,能准确捕获富有表现力的手部姿态,在严重缺失数据、快速运动和测量噪声下依然鲁棒。\n- 典型失败案例包括手-物交互(未显式建模导致错误对应)和超出低维姿态空间的异常手部姿态;所有失败案例均包含在公开数据集中以促进后续研究。\n- 模型与数据免费供研究使用;线性混合形状的内存占用与运行时与LBS相当,适合实时应用;未来计划用模型生成带真实纹理的深度学习训练数据,并基于真实无标记数据学习身体-手部运动关联映射。"
}