{
  "paper_id": "0a348b63e2f0c92c57ca7e88453a830f",
  "paper_title": "VIHand: Enhancing 3D Hand Pose Estimation with Visual-Inertial Benchmark",
  "paper_authors": [
    "Xinyi Wang",
    "Pengfei Ren",
    "Haoyang Zhang",
    "Xin Sheng",
    "Da Li",
    "Liang Xie",
    "Yue Gao",
    "Erwei Yin"
  ],
  "publication_date": "October 27–31, 2025",
  "publication_venue": "Proceedings of the 33rd ACM International Conference on Multimedia (MM '25), Dublin, Ireland",
  "abstract": "Accurate and robust 3D hand pose estimation (HPE) plays a crucial role in human-computer interaction. Existing 3D HPE solutions predominantly rely on vision-based or inertial measurement units (IMUs)-based methods. Vision-based methods benefit from rich appearance information for high-accuracy HPE but are sensitive to field of view (FoV), occlusion, motion blur and lighting. IMU-based methods can operate immune to optical sensitivity and FoV constraints but remain vulnerable to cumulative integration errors and drift. Given their complementary strengths, combining dual modalities offers a promising direction for HPE in complex environments. However, the lack of large-scale visual-inertial datasets has limited progress in this area. In this paper, we construct VIHand, the first large-scale glove-worn dataset for visual-inertial HPE, comprising over 1.4 million synchronized RGB-D and IMU frames from 15 subjects. It enables comprehensive research in HPE tasks, such as multimodal fusion and cross-modal knowledge transfer. Building on VIHand, we propose visual-inertial fusion network (VIFNet) for dual modalities estimation, and its distilled student model (VIFNet-S) for IMU-only evaluation. Experimental results reveal that integrating visual and inertial modalities significantly improves the accuracy and robustness of 3D HPE, particularly under occlusion and motion blur. In IMU-only inference even sparse IMU configurations, models distilled from visual-inertial supervision achieve substantial performance gains, enabling robust HPE for challenging optical sensitive scenarios. Our dataset and supplementary materials are available on the project website: https://shirley0118.github.io/VIHand.",
  "keywords": [
    "Multi-modality",
    "Hand Pose Estimation",
    "IMU",
    "Knowledge Distillation"
  ],
  "paper_institutions": [
    "Shanghai Jiao Tong University",
    "Tianjin University",
    "Beijing University of Posts and Telecommunications",
    "Nankai University",
    "Defense Innovation Institute, Academy of Military Sciences",
    "MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University"
  ],
  "output_language": "zh",
  "provider": "doubao",
  "model": "glm-5.3",
  "prompt_template": "simple",
  "extracted_at": "2026-09-17T09:37:43.611552Z",
  "template_tag": "simple",
  "summary": "### 摘要\n\n3D手部姿态估计（HPE）在人机交互、增强现实（AR）与虚拟现实（VR）中至关重要，但现有方法主要依赖单一模态：视觉方法虽能利用丰富外观信息实现高精度估计，却受视场角（FoV）、遮挡、运动模糊和光照变化限制，并存在隐私顾虑；IMU方法虽不受光学敏感性与FoV约束，却面临累积积分误差、磁干扰与漂移问题，且多数仅估计腕部或指尖位置。针对大规模视觉-惯性数据集缺失这一研究瓶颈，本文构建了首个大规模戴手套视觉-惯性HPE数据集VIHand，包含来自15名受试者的超过140万帧时间同步的RGB-D与IMU数据（5台Intel RealSense D415i相机以1280×720分辨率、30FPS采集，数据手套嵌入7个9轴ICM-20948 IMU），并采用多视角深度图像自监督标注加人工精修的两阶段流程提供3D关节坐标与MANO参数标注（自动标注在保留评估集上的平均关节位置误差为3.86mm）。基于该数据集，作者提出视觉-惯性融合网络VIFNet：ResNet-18骨干提取经空间注意力精炼并分为7组的RGB特征，与经嵌入层和Transformer编码器处理的7组IMU特征，通过跨模态注意力机制（CMAM）融合，再由MLP回归MANO参数并解码出3D手部网格与21个关节位置，其训练目标为复合损失 L_VIFNet = λ_3D·L_3D + λ_2D·L_2D + λ_MANO·L_MANO + λ_hp·L_hp；同时提出经跨模态知识蒸馏的学生模型VIFNet-S，利用对比蒸馏损失L_ts使IMU特征对齐教师模型的融合特征表示。实验在PyTorch与NVIDIA RTX 4090上实现，数据集按受试者划分（15个序列中12个训练、3个测试），以MPJPE和MPVPE为指标：VIFNet融合RGB与IMU取得最优的7.86mm MPJPE与8.94mm MPVPE，优于HaMeR（11.98/13.67mm）、WiLoR（10.49/11.74mm）等视觉方法及TransPose、FusePose等IMU方法，验证了多模态融合在遮挡与运动模糊场景下的有效性；VIFNet-S在仅IMU推理下使用7/3/2个IMU分别达到13.69/18.26/20.72mm MPJPE，显著优于无蒸馏方法，证明跨模态蒸馏能有效迁移语义表示。未来方向包括跨模态数据生成与单模态增强等研究；实际应用中需权衡隐私问题、传感器佩戴舒适性与稀疏配置下的精度，该工作为可穿戴设备在光学敏感场景下的鲁棒HPE提供了实用且轻量的方案。\n\n### 关键要点\n\n- VIHand是首个大规模戴手套视觉-惯性3D手部姿态估计数据集，包含15名受试者超过140万帧时间同步的RGB-D与IMU数据，并提供精确的3D关节坐标与MANO参数标注，支持多模态融合、跨模态知识迁移与跨模态数据生成等研究。\n- 数据采集平台由5台Intel RealSense D415i相机（1280×720分辨率、30FPS）和嵌入7个9轴ICM-20948 IMU的数据手套组成，多相机系统经3D标定靶标标定，像素级RMS重投影误差在0.42至0.48像素之间。\n- 每名受试者执行15个手势序列（14个预定义加1个约一分钟的随机自发动作），涵盖手指交叉、互锁、摆动、敲击、捏合、弯曲、打响指和滑动等复杂手势，t-SNE可视化显示其3D姿态多样性显著高于FreiHAND。\n- 标注采用两阶段流程：先使用多视角深度图像自监督标注（CrossView Fusion Network策略加RANSAC三角化，保留评估集上平均关节位置误差3.86mm），再由专业标注员人工审核精修低置信度结果，兼顾精度与可扩展性。\n- VIFNet通过ResNet-18提取经空间注意力精炼的RGB特征并分为7组，IMU信号（3轴加速度与9轴姿态）经嵌入层与Transformer编码器生成7组特征，再由跨模态注意力机制（CMAM）与多头自注意力融合，最终回归MANO参数解码出778×3网格顶点与21×3关节位置。\n- VIFNet训练采用复合损失 L_VIFNet = λ_3D·L_3D + λ_2D·L_2D + λ_MANO·L_MANO + λ_hp·L_hp；VIFNet-S则在此基础上引入对比蒸馏损失L_ts，使学生的IMU特征对齐教师模型的融合特征表示，实现跨模态知识迁移。\n- 实验基于PyTorch与NVIDIA RTX 4090实现，RGB图像先缩放至256×256再中心裁剪为224×224，批大小32、训练80轮、初始学习率0.0001（20轮后衰减0.1倍）、Adam优化器，数据集按受试者固定划分（12个序列训练、3个测试）以保证公平与可复现。\n- 评测采用MPJPE与MPVPE指标（计算前对齐腕关节根平移），VIFNet融合RGB与IMU取得最优的7.86mm MPJPE与8.94mm MPVPE，优于HaMeR（11.98/13.67mm）与WiLoR（10.49/11.74mm）等视觉方法，尤其在复杂自遮挡手势下优势明显。\n- 在仅IMU推理设置下，VIFNet-S使用7、3、2个IMU分别达到13.69、18.26、20.72mm MPJPE，显著优于TransPose（21.84/28.01/30.47mm）与FusePose（18.56/24.91/27.38mm）等无蒸馏方法，验证了跨模态蒸馏在稀疏IMU配置下的有效性。\n- 视觉方法受视场角、遮挡、运动模糊与光照限制并存在隐私顾虑；IMU方法存在累积积分误差、磁干扰与漂移，且多数仅估计腕部或指尖位置而缺乏精确的全手关节，双模态融合可互补增强鲁棒性与泛化性。\n- 密集IMU配置（10–18个）虽精度较高但佩戴不便且硬件笨重，阻碍日常使用；VIHand同时覆盖裸手与戴手套交互，更贴近真实使用场景，数据收集经当地机构审查委员会（IRB）批准并经匿名化保护隐私。\n- 该数据集与基准为AR/VR、康复与人机交互等实际场景提供了基础，未来可进一步探索模态补偿、单模态增强与跨模态数据生成方向，项目网站为 https://shirley0118.github.io/VIHand。"
}