{
  "paper_id": "6fca2c0346ab5522d834cb6ecc136535",
  "paper_title": "EgoEV-HandPose: Egocentric 3D Hand Pose Estimation and Gesture Recognition with Stereo Event Cameras",
  "paper_authors": [
    "Luming Wang",
    "Hao Shi",
    "Jiajun Zhai",
    "Kailun Yang",
    "Kaiwei Wang"
  ],
  "publication_date": "",
  "publication_venue": "unknown",
  "abstract": "Egocentric 3D hand pose estimation and gesture recognition are essential for immersive augmented/virtual reality, human-computer interaction, and robotics. However, conventional frame-based cameras suffer from motion blur and limited dynamic range, while existing event-based methods are hindered by ego-motion interference, monocular depth ambiguity, and the lack of large-scale real-world stereo datasets. To overcome these limitations, we propose EgoEV-HandPose, an end-to-end framework for joint 3D bimanual pose estimation and gesture recognition from stereo event streams. The final to our approach is KeypointBEV, a flexible stereo fusion module that lifts features into a canonical bird's-eye-view space and employs an iterative reprojection-guided refinement loop to progressively resolve depth uncertainty and enforce kinematic consistency. In addition, we introduce EgoEV-Hands, the first large-scale real-world stereo event-camera dataset for egocentric hand perception, containing 5,419 annotated sequences with dense 3D/2D keypoints across 38 gesture classes under varying illumination. Extensive experiments demonstrate that EgoEV-HandPose achieves state-of-the-art performance with an MPJPE of 30.54 mm and 86.87% Top-1 gesture recognition accuracy, significantly outperforming RGB-based stereo and prior event-camera methods, particularly in low-light and bimanual occlusion scenarios, thereby setting a new benchmark for event-based egocentric perception. The established dataset and source code will be publicly released at <https://github.com/ZJUWang01/EgoEV-HandPose>.\n",
  "keywords": [
    "Event Camera",
    "3D Hand Pose Estimation",
    "Stereo Camera",
    "Egocentric Vision",
    "Gesture Recognition"
  ],
  "paper_institutions": [],
  "output_language": "zh",
  "provider": "doubao",
  "model": "doubao-seed-2.1-pro",
  "prompt_template": "simple",
  "extracted_at": "2026-10-20T04:37:29.134146Z",
  "template_tag": "simple",
  "summary": "### 摘要\n\n本文针对AR/VR、人机交互和机器人等场景中的自我中心三维双手姿态估计与手势识别问题：传统帧式相机存在运动模糊、动态范围有限和功耗较高等缺陷，而已有事件相机方法又受到头部自运动噪声、单目深度歧义、双手遮挡以及缺少大规模真实立体事件数据集的制约。为此，作者提出端到端的EgoEV-HandPose框架，将同步左右事件流转换为局部归一化事件表面，通过双视角权重共享的EgoBlaze骨干网络提取二维热图、分割掩膜和局部特征，并用Confidence Gate抑制头部运动造成的背景事件；其核心KeypointBEV模块先由标定内外参和初始二维点三角化得到粗略三维姿态，再将立体特征提升到规范鸟瞰视图空间，在N=3次迭代中执行重投影u=K[R|T]P、双线性采样和Transformer解码，核心更新机制为P^{(k+1)}=P^{(k)}+η^{(k)}ΔP^{(k)}，从而以可微的重投影反馈逐步消除深度不确定性并强化运动学一致性，随后基于腕部中心化和掌长尺度归一化的三维关节序列，由3层Transformer完成38类手势识别。为支撑训练与评测，论文建立EgoEVHands数据集，包含5419个完整标注样本、10名参与者、38类手势、每只手21个关键点的稠密2D/3D标注、分割掩膜、红外图、深度图以及正常光和低光条件，并覆盖单手、双手无遮挡、自遮挡和互遮挡等交互类型；该数据基于RealSense D435、Grounded-SAM、MediaPipe和人工校正构建，适合验证真实世界、跨身份和复杂遮挡条件下的立体事件感知能力。实验基于PyTorch和单张NVIDIA A800，采用同分布与异分布两种划分，指标包括二维像素误差、MPJPE、PA-MPJPE、PCK AUC和Top-1准确率；框架在异分布设置下达到30.54 mm MPJPE和86.87% Top-1准确率，在同分布设置下达到22.03 mm MPJPE，相对立体RGB基线HandMvNet和事件基线Ev2Hands分别降低约57.5%和73.5%的三维误差，低光下将HandMvNet的82.43 mm降至29.50 mm，PCK AUC从初始三角化的0.436提升至0.589，整体仅用8.44M参数、19.86 GFLOPs并实现13.6 FPS，KeypointBEV插入DEV-Pose后还在DHP19上将MPJPE从55.53 mm降至47.88 mm，证明立体几何融合和迭代BEV优化的有效性与可迁移性。该技术适用于头戴式AR/VR、可穿戴人机交互和功耗敏感的移动视觉系统，但19.86 GFLOPs对超低功耗设备仍有压力，对称双手结构在纯单手场景可能产生冗余，未来可通过模型压缩、量化、动态结构门控、跟踪触发的稀疏三维细化、自监督学习以及面向实时扩展现实和人形机器人平台的部署进一步改进。\n\n### 关键要点\n\n- EgoEVHands是论文宣称的首个面向自我中心手部感知的大规模真实世界立体事件相机数据集，可弥补单目事件数据和真实立体标注缺失的问题。\n- 数据集包含5419个完整标注样本、10名参与者、38类手势，并提供同步左右事件流、RealSense红外与深度图像、手部分割掩膜以及稠密2D/3D关键点。\n- 标注流程使用Grounded-SAM生成手部掩膜，用MediaPipe在红外图上检测每只手21个关键点，经人工筛选后结合深度和相机参数反投影为三维真值，再投影到两个事件相机视角。\n- 数据按交互复杂度组织为单手、双手无遮挡、自遮挡和互遮挡等语义组，并同时覆盖正常光与低光环境，可用于评测光照和遮挡鲁棒性。\n- EgoEV-HandPose先将事件流编码为局部归一化事件表面，再由双视角完全权重共享的EgoBlaze骨干网络输出二维热图、分割掩膜和局部特征。\n- EgoBlaze中的Confidence Gate能够预测运动感知注意力图，抑制头部移动诱发的背景事件，同时保留手部高对比度事件信号。\n- KeypointBEV利用相机内参、外参和初始三角化结果，在规范鸟瞰视图中通过重投影采样、门控位置编码和Transformer残差预测完成3次迭代细化。\n- 手势分类分支对关节坐标做腕部中心化和掌长尺度归一化，将每帧168维关节特征投影到512维嵌入，并用3层Transformer和MLP输出38类概率。\n- 训练采用2D骨干预训练、KeypointBEV训练、动作分类器预训练和最终端到端微调的多阶段课程，损失包含热图、分割、三维、中间重投影和交叉熵分类项。\n- 在异分布测试中，完整框架取得30.54 mm MPJPE和86.87% Top-1手势准确率；在同分布测试中取得22.03 mm MPJPE。\n- 相比HandMvNet和Ev2Hands，所提方法在低光、快速运动、双手自遮挡和互遮挡条件下均表现出更稳定的关节定位能力。\n- 完整模型包含8.44M参数和19.86 GFLOPs，推理速度为13.6 FPS；其计算量仍需压缩或量化才能更好适配超低功耗头戴设备，且单手场景可能存在双手结构冗余。"
}