{
  "paper_id": "f8b955a035808ef3bf1daa091fa076c4",
  "paper_title": "HOT3D: Hand and Object Tracking in 3D from Egocentric Multi-View Videos",
  "paper_authors": [
    "Prithviraj Banerjee",
    "Sindi Shkodrani",
    "Pierre Moulon",
    "Shreyas Hampali",
    "Shangchen Han",
    "Fan Zhang",
    "Linguang Zhang",
    "Jade Fountain",
    "Edward Miller",
    "Selen Basol",
    "Richard Newcombe",
    "Robert Wang",
    "Jakob Julian Engel",
    "Tomas Hodan"
  ],
  "publication_date": "2024",
  "publication_venue": "2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)",
  "abstract": "We introduce HOT3D, a publicly available dataset for egocentric hand and object tracking in 3D. The dataset offers over 833 minutes (3.7M+ images) of recordings that feature 19 subjects interacting with 33 diverse rigid objects. In addition to simple pickup, observe, and put-down actions, the subjects perform actions typical for a kitchen, office, and living room environment. The recordings include multiple synchronized data streams containing egocentric multi-view RGB/monochrome images, eye gaze signal, scene point clouds, and 3D poses of cameras, hands, and objects. The dataset is recorded with two headsets from Meta: Project Aria, which is a research prototype of AI glasses, and Quest 3, a virtual-reality headset that has shipped millions of units. Ground-truth poses were obtained by a motion-capture system using small optical markers attached to hands and objects. Hand annotations are provided in the UmeTrack and MANO formats, and objects are represented by 3D meshes with PBR materials obtained by an in-house scanner. In our experiments, we demonstrate the effectiveness of multi-view egocentric data for three popular tasks: 3D hand tracking, model-based 6DoF object pose estimation, and 3D lifting of unknown in-hand objects. The evaluated multi-view methods, whose benchmarking is uniquely enabled by HOT3D, significantly outperform their single-view counterparts.",
  "keywords": [
    "egocentric vision",
    "hand-object interaction",
    "3D hand tracking",
    "6DoF object pose estimation",
    "multi-view",
    "dataset",
    "AR/VR",
    "motion capture",
    "DINOv2"
  ],
  "paper_institutions": [
    "Meta Reality Labs"
  ],
  "output_language": "zh",
  "provider": "doubao",
  "model": "glm-5.3",
  "prompt_template": "simple",
  "extracted_at": "2026-09-17T09:26:03.193603Z",
  "template_tag": "simple",
  "summary": "### 摘要\n\n现有手-物交互理解方法在精度与速度上不足以可靠支撑AR/VR技能迁移、情境AI等应用，而此前数据集（如HO-3D、DexYCB、HOI4D、ARCTIC、HO-Cap等）多为单视角或外置相机采集，缺乏真实头戴设备上的多视角自我中心数据。为此，本文发布HOT3D数据集：使用Meta Project Aria（轻量AI眼镜研究原型）与Quest 3（已量产数百万台的VR头显）录制833分钟（超过370万张图像、150万+多视角帧）的自我中心多视角视频，涵盖19名受试者与33个刚性物体，场景包括简单拾取-观察-放下以及厨房、办公室、客厅等日常动作；数据包含硬件触发同步的RGB/单色图像、眼动信号、SLAM三维点云，以及相机、双手（UmeTrack与MANO格式）和物体的逐帧真值位姿（由被动光学标记动作捕捉系统获得），物体提供带PBR材质的高分辨率三维网格模型。核心机制在于利用硬件同步的多视角流：例如StereoMatch方法在两视角的420×420裁剪图上提取DINOv2 ViT-S块特征，按行建立2D-2D对应（保留循环距离最小的至多500对），三角化为三维点后以鲁棒均值估计物体三维位置。实验以MKPE（手部关键点平均位置误差）、召回率（平移/旋转误差阈值下的正确位姿比例）和mIoU为指标：UmeTrack在两数据集联合训练下双视角模式将MKPE从15.4降至10.9 mm（提升41%）；FoundPose多视角扩展在两台头显上召回率提高8–12个百分点（相对提升13–34%）；StereoMatch在Quest 3上10cm阈值召回率达96.8%，显著优于单目深度方法。这些结果验证了多视角自我中心数据对功耗敏感的AR/VR视觉系统（可负担多相机但难以使用主动深度传感器）的价值。未来方向包括利用眼动信息预测用户意图、注视点感知的计算预算分配、无CAD物体跟踪与三维重建等。实际约束包括：单色低分辨率相机、部分帧标注缺失（150万帧中116万帧完整标注）、测试集真值仅通过评测服务器提供、MonoDepth依赖Aria的SLAM观测而无法用于Quest 3等。\n\n### 关键要点\n\n- HOT3D是首个大规模、多视角、硬件时间同步、由真实头戴设备（Aria与Quest 3）采集的自我中心手-物交互数据集，包含833分钟录制、3.7M+图像、19名受试者与33个刚性物体。\n- 数据由Project Aria（1路RGB 1408×1408 + 2路单色640×480）和Quest 3（2路单色1280×1024）以30fps采集，Aria还额外提供SLAM三维场景点云与眼动信号。\n- 真值位姿通过配备数十台红外OptiTrack相机的被动标记动作捕捉实验室获得，手部标注提供UmeTrack（更精确）与MANO（更标准）两种格式，物体与手腕位姿表示为模型空间到场景空间的三维刚体变换。\n- 33个物体的三维网格模型由内部扫描重建流水线生成，含金属度、粗糙度、法线贴图等PBR材质，可渲染照片级真实训练图像，物体涵盖多样的外观、尺寸与可供性。\n- 场景涵盖简单检查（拾取-观察-放下）及厨房、办公室、客厅四类日常情境，每段录制交互至多6个物体，并定期随机化光照、家具摆放与装饰元素以增加多样性，共425段约2分钟的录制。\n- 训练/测试划分为13名受试者（100万帧）与6名受试者（50万帧），测试集真值仅通过专用评测服务器访问；另发布3832个经质量检查的150帧（5秒）精选片段（HOT3D-Clips）用于基准评测。\n- 3D手部跟踪实验中，UmeTrack在UmeTrack与HOT3D联合训练后双视角模式达到9.5/10.9 mm MKPE，较单视角提升41%，证明多视角输入与跨数据集训练可有效弥合域差距。\n- FoundPose的多视角扩展通过跨视角模板检索、多视角2D-3D对应建立与广义PnP求解，在两台头显上召回率提升8–12个百分点，并能估计单视角下被严重遮挡物体的位姿。\n- 2D在手物体分割任务中，结合Depth Anything V2预测深度的MRCNN-DA在Aria与Quest 3上分别比EgoHOS高出30%和65%的mIoU，预测深度有助于区分前景在手物体与背景。\n- 3D提升任务中，基于DINOv2特征立体匹配的StereoMatch在Quest 3上10cm阈值召回率达96.8%，明显优于单目深度方法MonoDepth，后者的定位误差主要沿光轴方向、源于深度预测不准。\n- HandProxy基线（以手掌中心作为物体位置代理）表明仅靠手部跟踪即可在30cm不确定半径内定位物体，但10cm内的精细定位需要专门的多视角3D提升方法。\n- 数据集还提供物体上线序列（静态桌面正放/倒放与手部操作两种），支持无CAD模型跟踪与NeRF式重建；静态序列提供全部帧真值而动态序列仅首帧，以模拟真实场景中SfM可得与不可得的差异。"
}