{
  "paper_id": "fb6c32df8e9b9afc38cf7c3e82ce0228",
  "paper_title": "Capturing Hands in Action Using Discriminative Salient Points and Physics Simulation",
  "paper_authors": [
    "Dimitrios Tzionas",
    "Luca Ballan",
    "Abhilash Srikantha",
    "Pablo Aponte",
    "Marc Pollefeys",
    "Juergen Gall"
  ],
  "publication_date": "2016-03-18",
  "publication_venue": "International Journal of Computer Vision",
  "abstract": "Hand motion capture is a popular research field, recently gaining more attention due to the ubiquity of RGB-D sensors. However, even most recent approaches focus on the case of a single isolated hand. In this work, we focus on hands that interact with other hands or objects and present a framework that successfully captures motion in such interaction scenarios for both rigid and articulated objects. Our framework combines a generative model with discriminatively trained salient points to achieve a low tracking error and with collision detection and physics simulation to achieve physically plausible estimates even in case of occlusions and missing visual data. Since all components are unified in a single objective function which is almost everywhere differentiable, it can be optimized with standard optimization techniques. Our approach works for monocular RGB-D sequences as well as setups with multiple synchronized RGB cameras. For a qualitative and quantitative evaluation, we captured 29 sequences with a large variety of interactions and up to 150 degrees of freedom.\n",
  "keywords": [
    "Hand motion capture",
    "Hand–object interaction",
    "Fingertip detection",
    "Physics simulation"
  ],
  "paper_institutions": [],
  "output_language": "zh",
  "provider": "doubao",
  "model": "doubao-seed-2.1-pro",
  "prompt_template": "simple",
  "extracted_at": "2026-10-20T04:36:39.731081Z",
  "template_tag": "simple",
  "summary": "### 摘要\n\n针对手部动作捕捉中多数方法只处理孤立单手，而手手交互或手物交互存在严重自遮挡、部位外观相似、姿态高维和视觉数据缺失等问题，论文在生成式跟踪脉络下统一刚体与关节物体建模，提出由线性混合蒙皮网格、Hough森林显著点、碰撞检测、物理模拟、解剖学关节限制和时序正则组成的单一目标函数 $E(θ,D)=E_{model→data}+E_{data→model}+γ_cE_{collision}+E_{salient}+γ_{ph}E_{physics}+γ_aE_{anatomy}+γ_rE_{regularization}$；在固定对应关系时目标函数可微并用高斯-牛顿局部优化，双向数据项分别防止可见部分错误拟合和姿态退化，显著点通过整数规划联合解决检测点与手指关联以跳出局部极小，碰撞项用层次包围盒和局部锥形距离场消除穿透，物理项则在重力、摩擦、恢复系数和质量设定下评估物体稳定性并引导手指形成支撑。实验采集29段序列，包括Primesense单目RGB-D序列和8台同步FullHD RGB相机序列，覆盖手手紧密交互以及球、立方体等6自由度刚体和管道、绳索等关节物体，最高达150自由度；选择这些设置是为了在真实遮挡、缺失深度和多部位接触条件下验证框架，而不仅依赖合成数据。定量指标主要采用三维关节投影到二维标注的像素误差，多相机实验另用三维毫米距离和骨骼角度误差；手手交互中LO+CS平均误差5.12 px、低于LO的5.53 px且11段无失败，所提方法平均3.76 px而FORTH设置为8.09–8.58 px，手物序列完整LO+SCP为6.19 px并将物体质心物理位移从9.26 mm降至9.05 mm，Dexter多数序列误差约1 cm，多相机合成序列ECOS为1.49 mm、真实标注点中位误差不超过1 cm，这些结果支持混合生成-判别约束和物理合理性机制的价值。后续可改进高维能量景观与局部极小、快速运动、缺失深度和误检测，并研究自动手形建模与加速；方法适用于图形动画、人机交互、康复和机器人等场景，但依赖给定手部网格模型、时间连续性和较小姿态调整，CPU运行约数秒每帧，复杂管道或绳索交互中的物理搜索可能超过1分钟，且物理项需以无严重穿透的碰撞项为前提。\n\n### 关键要点\n\n- 方法同时适用于单目RGB-D和多视角同步RGB，可处理两只手交互以及手与刚体或关节物体的交互。\n- 手部和物体采用三角网格加运动学骨架的线性混合蒙皮表示，单手37自由度，双手74自由度，双手加绳索可达150自由度。\n- 目标函数包含双向数据拟合、碰撞、显著点、物理、解剖限制和正则七项，固定对应后用高斯-牛顿局部优化。\n- Hough森林在RGB-D中检测指尖、在多相机RGB中检测指甲；手指与检测的关联通过整数规划联合求解，并允许误检或漏检。\n- 碰撞检测先用层次包围盒找到相交三角形，再用局部锥形三维距离场生成排斥约束，比简单的碰撞顶点对应更稳定。\n- 物理模拟将场景表示为低分辨率凸包，运行35个0.1秒步长，物体质心移动小于3 mm即判为稳定，并从2至4个候选手指部位组合中选择最稳定支撑。\n- RGB-D手手实验使用14段人工标注序列，分辨率为640×480、30 fps，4名标注者标注结果的标准差为1.46 px。\n- 手手组件实验中LO+CS平均误差为5.12 px，LO为5.53 px；与FORTH比较时，所提方法为3.76 px，对方多组设置约为8.09–8.58 px。\n- 手物实验新增7段180至400帧序列，完整LO+SCP平均误差为6.19 px，并使跟踪后物体质心平均位移由9.26 mm降到9.05 mm。\n- Dexter数据上多数序列误差约1 cm，显著点因检测器更准确而贡献更大，但“random”序列因运动过快而失败。\n- 多相机RGB实验使用8台50 fps FullHD相机；合成序列中联合边缘、碰撞、光流和显著点的平均关节位置误差为1.49 mm，真实三点标注的中位误差最高约1 cm。\n- 方法限制包括需要给定手部模型、依赖时序连续性和局部优化，缺失深度或误检测会产生较差姿态，复杂交互运行较慢，物理项甚至可能使每帧增加超过1分钟。"
}