{
  "paper_id": "68c90b469354bce005d807b04d7e8fe4",
  "paper_title": "VICON: Visual-Inertial-Contact based Hand-Object Tracking for Manipulation Datasets",
  "paper_authors": [
    "Youbeen Jeon",
    "Uiseong Shin",
    "Hwanchul La",
    "Jaeseong Kang",
    "Hyelim Choi",
    "Yongseok Lee"
  ],
  "publication_date": "",
  "publication_venue": "unknown",
  "abstract": "Learning dexterous manipulation benefits from human demonstration datasets that capture diverse and natural hand-object interactions. In particular, contact points and forces provide supervision on where and how strongly to interact, which cannot be fully captured by motion trajectories alone. However, methods for jointly capturing hand and object motion, contact points, and forces remain limited. Moreover, severe occlusion from hand-object interaction challenges accurate tracking of both hands and objects. To address these limitations, we present a Visual-Inertial-CONtact based hand-object tracking (VICON) framework. It holistically captures both hand and object motion along with contact information during manipulation, even under severe occlusion. First, we adopt a visual-inertial glove and an RGB-D camera for accurate hand tracking, and redesign the glove to incorporate contact sensing. Specifically, force-sensitive resistors (FSRs) are placed on the glove based on human grasp frequency to synchronously record contact states and calibrated normal forces. Second, without requiring pre-existing CAD models, we estimate object poses using RGB-D images and a mesh reconstructed from a monocular video. We propose factor-graph-based object trajectory estimation that fuses object-pose estimates weighted by visibility under hand-object occlusion, FSR measurements, and a hand-motion prior. Across 40 motion-capture sessions with five objects, VICON achieves a 2.5% failed-frame rate compared with 50.9–64.6% for the baselines, with median errors of 3.9 mm and 3.0° under occlusion. Using VICON, we construct a dataset containing synchronized hand-object motion, contact points, and normal forces, and will publicly release an expanded dataset covering 10 object categories at <https://github.com/VICON-dataset/dataset>.\n",
  "keywords": [
    "hand-object tracking",
    "visual-inertial sensing",
    "contact force sensing",
    "force-sensitive resistors",
    "factor graph optimization",
    "manipulation dataset",
    "occlusion-robust tracking",
    "dexterous manipulation"
  ],
  "paper_institutions": [],
  "output_language": "zh",
  "provider": "doubao",
  "model": "doubao-seed-2.1-pro",
  "prompt_template": "simple",
  "extracted_at": "2026-10-20T04:23:28.382350Z",
  "template_tag": "simple",
  "summary": "### 摘要\n\n针对灵巧操作学习需要同时包含手部-物体运动、接触点和接触力的丰富人类演示数据，而现有方法要么缺乏直接力测量、要么依赖多相机动作捕捉工作室，且手物交互造成的严重遮挡使视觉跟踪容易失效这一问题，本文提出VICON视觉-惯性-接触手物跟踪框架。该框架基于集成8个力敏电阻（FSR）的VIST视觉惯性手套和单个RGB-D相机，其中FSR依据人类抓握接触频率布置在五指指尖、鱼际和远端手掌，同步记录接触状态和经标定的法向力；物体网格从单目视频通过COLMAP、3DGS和SuGaR重建，无需预存CAD模型。核心机制是离线因子图优化，将FoundationPose候选姿态按物体可见性加权，融合接触门控的手部运动先验——当至少3个FSR被识别为接触时（$c_k=\\mathbf{1}[\\sum_{i=1}^{8}\\rho_{i,k}\\geq3]$）判定抓握并施加手物刚性约束，两点抓握时则绕接触线松弛旋转自由度——以及静止约束和长程抓握链接。在4名受试者、5种刚性物体、40个动作捕捉会话共47,222帧（其中24,285帧为遮挡帧）的实验中，VICON失败帧率仅2.5%，而FoundationPose、MegaPose和ICG+分别为50.9%、62.6%和64.6%；遮挡下中位平移和旋转误差分别为3.9 mm和3.0°，消融实验表明去除任一组件失败率升至18.1–28.9%。基于该框架构建的数据集含71个会话、81,307帧同步运动与接触力记录，法向力中位数10.9 N。该系统成本低、可穿戴、不受专用捕捉空间限制，适用于可扩展的接触丰富演示采集；当前局限于单手刚性物体序列和固定抓握先验，未来将扩展受试者、物体类别与快速运动，支持完整手中操作并验证于策略学习。\n\n### 关键要点\n\n- VICON硬件由集成8个FSR的VIST视觉惯性手套和单台Intel RealSense D435i RGB-D相机构成，手套含14个IMU和80个视觉标记，深度采集约30 Hz。\n- 8个FSR依据人类抓握接触频率图布置：5个指尖各1个、鱼际1个、远端手掌第二和第五掌骨区域各1个。\n- FSR信号以200 Hz采样、20 Hz低通滤波，经施密特触发器转为二值激活，并逐个在数字秤上标定至61 N，标定拟合$R^2$为0.83–0.99。\n- 物体网格从单目视频经COLMAP、3DGS和SuGaR重建，以ChArUco板设定米制尺度，在YCB物体上相对扫描平均偏差约1 mm，无需预存CAD模型。\n- FoundationPose候选姿态通过渲染深度与观测深度比较计算可见性$v_d$和悬空比例$v_{far}$，仅在像素数、可见性、桌面穿透和时序连续性等条件满足时接受，并按可见性调整平移与旋转协方差。\n- 接触判定要求FSR激活且其中心距物体网格不超过30 mm，至少3个FSR满足时才判定抓握帧，接触状态经两轮提取与细化以克服初始姿态误差。\n- 因子图最小化视觉、合成姿态、手部运动、静止和抓握链接残差，使用GTSAM中Levenberg–Marquardt分两阶段求解；抓握时手物刚性链接，无锚点的遮挡段使用80 mm和45°大偏差合成姿态。\n- 对连续至少5帧的两点指尖接触，手部运动先验通过秩一协方差更新绕两接触点连线松弛旋转（标称每帧15°），仅在时间间隔不超过0.25 s且轴有效时生效。\n- 在40个动作捕捉会话上，VICON总失败帧率为2.5%，遮挡帧成功率$S_{T20}$和$S_{R15}$分别为97.1%和98.4%，遮挡中位误差3.9 mm和3.0°，40个会话中36个会话遮挡平移中位误差最低。\n- 消融显示去除可见性加权、手部运动先验或接触时序后失败帧率分别升至28.9%、18.1%和23.9%，配对符号检验均支持完整模型（校正后p<0.0005）。\n- 当前数据集含71个会话、81,307帧，记录手骨架、物体轨迹、物体坐标系3D接触点及法向力；力中位数10.9 N，90百分位27.0 N，83%位于0–20 N。\n- 系统适用于低成本可穿戴的接触丰富演示采集，无需多相机工作室；当前约束为单手、5种刚性物体、离线细化且固定抓握先验，不支持完整手中操作。"
}