# HOT3D：基于第一人称多视角视频的3D手部与物体追踪
Prithviraj Banerjee、Sindi Shkodrani、Pierre Moulon、Shreyas Hampali、Shangchen Han、Fan Zhang、Linguang Zhang、Jade Fountain、Edward Miller、Selen Basol、Richard Newcombe、Robert Wang、Jakob Julian Engel、Tomas Hodan
Meta Reality Labs
facebookresearch.github.io/hot3d
Figure 1. HOT3D overview. The dataset includes multi-view egocentric image streams from Aria [^13] and Quest 3 [^40] annotated with high-quality ground-truth 3D poses and models of hands and objects. Three multi-view frames from Aria are shown on the left, with contours of 3D models of hands and objects in the ground-truth poses in white and green, respectively. Aria also provides 3D point clouds from SLAM and eye gaze information (right).
## 摘要
我们推出HOT3D，这是一套面向第一人称3D手部与物体追踪的公开数据集。该数据集包含超过833分钟（370万张以上图像）的录制内容，记录了19名受试者与33种不同类型的刚性物体互动的场景。除了简单的拾取、观察和下放动作外，受试者还会执行厨房、办公室和客厅环境中常见的动作。录制内容包含多个同步数据流，涵盖第一人称多视角RGB/单色图像、眼动信号、场景点云，以及摄像头、手部和物体的3D姿态信息。该数据集是使用Meta的两款头戴设备录制的：Project Aria是AI眼镜的研究原型，Quest 3则是已售出数百万台虚拟现实头戴设备。真实姿态数据是通过在手部和物体上粘贴小型光学标记点，利用动作捕捉系统获取的。手部标注采用UmeTrack和MANO格式，物体则通过内部扫描仪获取带有PBR材质的3D网格模型。我们的实验证明了多视角第一人称数据在三项热门任务中的有效性：3D手部追踪、基于模型的6自由度物体姿态估计，以及未知手持物体的3D姿态还原。借助HOT3D才能开展的这些多视角方法的基准测试表明，它们显著优于单视角方法。
### 1. 引言
我们会用手来交流、与物体互动，或把物体当作工具来改造周围环境。人类操控物体的灵巧程度是其他物种无法比拟的，这也是我们进化的关键因素[^2]。因此，手与物体的交互自然受到了计算机视觉等多个研究领域的广泛关注[^47]。
一套能够自动理解手与物体交互的基于视觉的系统，若能捕捉到手与物体的3D运动、形状和接触信息，将在诸多领域发挥重要作用。例如，这类系统可以先记录熟练用户进行一系列手与物体交互的过程（比如组装家具、打网球发球等），再将所捕获的信息用于指导经验较少的用户——比如通过AR眼镜实现这一功能；类似地，这些技能也可以从人类传递给机器人。这类系统还能帮助人工智能助手更好地理解用户行为的上下文，或为AR/VR用户提供全新的输入方式。比如，它能将任何物理表面变成虚拟键盘，或将任何铅笔转化为多功能魔法棒。不过，现有手与物体交互理解方法的精度和速度还不足以可靠地支撑这类应用。
---
Figure 2. Sample images from Aria (top) and Quest 3 (bottom). Aria recordings include one RGB and two monochrome image streams, while Quest 3 recordings include two monochrome streams (only images from one of the multi-view streams are shown). Contours of 3D models of hands and objects in the ground-truth poses are shown in white and green respectively. In addition to simple pick-up/observe/put-down actions, the subjects perform actions that are common in a kitchen, office, and living room. To increase diversity, the lighting, furniture, and decorations in the capture lab were regularly randomized.
为了加速计算机视觉领域对手物交互的研究，我们公开发布HOT3D数据集。该数据集采用Meta公司的两款最新头戴式设备录制而成：Project Aria [^13]，这是一款轻量级人工智能眼镜的研究原型；以及Quest 3 [^40]，这款虚拟现实头盔的销量已突破数百万台。该数据集包含超过833分钟的第一人称视频流，其中有多达150万帧多视角画面（370多万张图像），记录了19名受试者与33种不同类型的刚性物体互动的场景。除了简单的取放、观察物体的测试场景外，这些录像还呈现了厨房、办公室和客厅中的典型动作场景。借助基于被动标记的动作捕捉系统，我们对手部及物体进行了精确的三维姿态标注。该数据集还包含了通过内部扫描式三维物体重建流程获得的三维物体模型，这些模型具备高分辨率的几何细节及基于物理的渲染材质 [^39]。Aria设备录制的视频还包含了通过SLAM技术生成的三维场景点云以及眼动信号数据。示例图片见图2。
HOT3D数据集主要用于训练和评估从局部化的第一人称多视角视频流中提取三维空间中手部及物体轨迹的方法，而非基于单目视图或单独图像的方法。由于所有视频流中的图像均通过硬件触发器同步采集（即在同一时间戳下捕获），因此该数据集有助于开发能够利用多视角和/或时间信息的技术。该数据集不仅为基于CAD模型的物体追踪提供了测试平台，还通过提供展示物体不同视角的参考序列，支持无需CAD模型的追踪设置，从而能够以少量样本快速完成物体注册。此外，该数据集还可用于三维物体重建、手物交互的二维检测或分割等任务。我们也鼓励利用Aria设备提供的眼动信息展开相关研究，这有助于预测用户意图，或通过注视点感知技术高效分配计算资源。
在实验中，我们主要致力于验证多视角第一人称数据在多项热门任务中的有效性。尽管当前AR/VR设备上普遍能获取这类数据，但此前对其的研究却相对匮乏。实验结果表明，用于三维手部追踪、基于模型的6自由度物体姿态估计以及未知手持物体的三维坐标还原的多视角方法，其性能明显优于单视角方法。
## 总结来说，我们的贡献如下：
1. 公开发布的HOT3D数据集：我们收集并发布了首个大规模数据集，该数据集具备以下特点：(1) 用真实头戴设备录制的多视角第一人称视频流；(2) 每个场景中头戴设备及多个物体的高质量姿态标注，以及双手的姿态与形状标注；(3) 包含动态抓取动作的非平凡手物交互场景；(4) 用于基于物理渲染的带材质的三维物体模型。HOT3D为评估各类二维/三维手物交互理解任务的方法提供了基准测试平台。
2. 基于HOT3D的任务基线模型：我们针对AR/VR及情境人工智能应用相关的两项任务，开发了简单却强大的多视角基线方法：基于模型的6自由度物体姿态估计（对FoundPose [^49]进行了扩展），以及未知手持物体的三维坐标还原（基于DINOv2 [^48]特征的双目匹配技术）。
3. 证明了多视角中心视觉数据的有效性：我们的实验表明，用于3D手部追踪、6自由度物体姿态估计以及手持物体3D重建的多视角方法，明显优于单视角方法。这对于研究低功耗中心视觉系统而言是一项重要成果——这类系统通常能够支持多视角摄像头配置[^13]，但却无法搭载主动式深度传感器[^66]。
---
| Dataset | Images | Channels | Ego / exo | HW trigger | Headsets | Subjects | Hands | Objects | ...per scene | Gaze | Annotation |
| HOT3D (ours) | 3.7M | RGB/gray | 2-3/0 | ✓ | Aria, Quest 3 | 19 | Both | 33 | ~6 | ✓ | Mocap |
| ARTIC [^15] | 2.1M | RGB | 1/8 | ✓ | Helmet | 10 | Both | 11 | 1 | ✗ | Mocap |
| HOI4D [^38] | 2.4M | RGB-D | 2/0 | ✗ | Helmet | 4 | Both | 800 | 1-12 | ✗ | RGB-D+man. |
| HO-Cap [^67] | 699K | RGB-D | 1/9 | ✗ | HoloLens | 9 | Both | 64 | 4 | ✗ | RGB-D+optim. |
| DexYCB [^8] | 582K | RGB-D | 0/8 | ✓ | - | 10 | Single | 20 | 2-4 | ✗ | Manual |
| HOGraspNet [^9] | 1.5M | RGB-D | 0/4 | ✓ | - | 99 | Single | 30 | 1 | ✗ | RGB-D+optim. |
| $H_{{2O}}$-3D [^24] | 75K | RGB-D | 0/5 | ✗ | - | 6 | Both | 10 | 1 | ✗ | RGB-D+optim. |
| HO-3D [^23] | 78K | RGB | 0/1 | - | - | 10 | Single | 10 | 1 | ✗ | RGB-D+optim. |
| $H_{{2O}}$ [^37] | 572K | RGB-D | 1/4 | ✓ | Helmet | 4 | Both | 8 | 1 | ✗ | RGB-D+optim. |
| ContactPose [^5] | 2.9M | RGB-D | 0/3 | ✗ | - | 50 | Both | 25 | 1 | ✗ | RGB-D+mocap |
| ObMan [^26] | 147K | Synth. RGB | 0/1 | - | - | 20 | Single | 2772 | 1 | ✗ | Synthetic |
Table 1. Datasets with 3D hands and object annotations. HOT3D is the first dataset to provide multi-view, hardware time-synced, egocentric videos captured with real headsets. It is the largest dataset in terms of total image count and provides high quality ground-truth annotations (comparable to [^15]).
### 2. 相关工作
计算机视觉领域的研究进展深受基准数据集的影响[^14] [^19] [^31] [^55] [^56]，这些数据集能够用于比较不同方法的性能并明确其局限性。在本节中，我们首先回顾包含手部或物体姿态标注的现有数据集，随后重点介绍提供手部及手部操控物体标注的数据集。
仅包含手部的数据集：基于视觉的3D手部姿态估计与追踪研究已开展多年，早期方法主要使用包含单色图像的定制数据集[^28] [^52]。后来，在NYU[^64]、ICVL[^63]、MSRA[^61]、Tzionas等人[^65]、EgoDexter[^43]或HANDS17[^71]等数据集提供的RGB-D图像上取得了显著进展。近期，受AR/VR应用场景的推动——该场景下由于功耗较高，深度传感器往往无法使用——研究界已大规模转向使用RGB或单色图像，相关数据集包括Stereo[^72]、InterHand2.6M[^42]、FreiHAND[^74]、UmeTrack[^25]、AssemblyHands[^46]，以及下文将介绍的同时包含双手和物体姿态标注的数据集。
仅包含物体的数据集：6自由度物体姿态估计与追踪的研究路径与之类似，最初使用定制的单色数据集[^44] [^53]，后来则主要转向LM[^29]、YCB-V[^69]和T-LESS[^30]等RGB-D数据集，这些数据集被纳入BOP基准测试[31–33,62]。该基准测试目前包含十二个格式统一的数据集，提供3D物体模型以及标注了6自由度物体姿态的训练和测试用RGB-D图像。3D物体模型是通过人工制作或使用类似KinectFusion的系统[^45]进行3D表面重建得到的。训练图像为真实图像或合成图像（使用BlenderProc[^11] [^12]逼真渲染而成），所有测试图像均为真实图像。除了这些实例级数据集外，研究界还会使用Wild6D[^17]、HouseCat6D[^35]和PhoCal[^68]等类别级RGB-D数据集。近期，一些方法又开始聚焦于从仅含RGB的图像中估计物体姿态，相关数据集包括OnePose[^60]和HANDAL[^21]。
同时包含手部和物体的数据集：许多现有数据集包含了手部与物体的图像（例如[^1] [^6] [^10] [^16] [^51] [^58] [^73]），但仅提供2D边界框、分割掩码或动作标签形式的标注。部分3D手部姿态估计数据集（例如[^43] [^46] [^74]）虽包含手部与物体交互的图像，却未提供6自由度物体姿态标注。
Sridhar等人创建了首个包含双手和物体真实姿态的数据集，该数据集提供了3014张手部操控立方体的外视角RGB-D图像，这些图像经过人工标注，标出了指尖位置和立方体的6自由度姿态。为避免繁琐且难以扩展的人工标注过程，FHPA数据集采用了附着在一只手和物体上的磁传感器，不过这明显影响了物体的外观。该数据集包含105K张单眼视角RGB-D图像，标注了单手和4个物体的真实姿态。ObMan数据集则通过合成手部抓取物体的图像来获取数据，其中的抓取动作由机器人学算法生成。
[^59]: [^18]: [^26]:
HO-3D [^23] was the first dataset with real images annotated by an optimization procedure that leverages multi-view RGB-D image streams and is almost fully automatic. The dataset offers 78K images from several exocentric cameras, showing 10 subjects and 10 objects. A similar annotation procedure was used for several subsequent datasets [^3] [^8] [^24] [^37] [^38]. H2O [^37] includes 572K egocentric multi-view RGB-D images of 4 subjects manipulating 8 objects. H2O-3D [^24] provides 75K exocentric RGB images of 6 subjects manipulating 10 YCB objects [^7]. DexYCB [^8] consists of 1000 clips of 3 seconds with the total of 582K RGB-D images, recorded from 8 exocentric views and showing 10 subjects picking up 20 YCB objects with near-static grasps. HOI4D [^38] includes 2.4M egocentric RGB-D images from over 4000 video sequences showing 9 subjects interacting with 800 different objects from 16 categories in 610 different indoor environments. Besides rigid objects, this dataset contains articulated objects, but focuses on simpler scenarios with a single hand and a single object, and only includes single-view video sequences. An RGB-D optimization procedure was also used in ContactPose [^5] along with the information from thermal cameras for accurately annotating hand poses, while the object poses were annotated using optical markers. ContactPose includes 2.9M RGB-D images of 50 subjects grasping 25 household objects, however, the grasps are static, background green and all objects are blue (3D printed), which makes the images less realistic.
---
与HOT3D类似，最近的ARCTIC数据集也采用了基于标记的动作捕捉系统来收集手和物体的真实姿态。该数据集包含210万张RGB图像，展示了10名受试者与11个关节物体的互动场景。这些图像在233K个时间戳内从9个视角拍摄，其中仅有一个是单眼视角——该视角是通过模拟单眼设备（安装在头盔上的摄像头）录制的。
[^15]:
除了HOT3D外，HO-Cap是唯一一个使用真实头戴设备（Microsoft HoloLens）录制的数据集，它提供了双手和多个物体的3D标注信息，不过这些标注是利用RGB-D相机获得的（而非精密的动作捕捉实验室），因此精度较低。
[^67]:
### 3. HOT3D数据集
共833分钟的录音资料。HOT3D包含以Aria [^13]和Quest 3 [^40]设备录制的以自我为中心的、多视角同步数据流。图像流以30帧/秒的速度录制，包含超过150万帧多视角画面，相当于370多万张图像。每张Aria画面由一张RGB $1408 \times 1408$图像和两张单色 $640 \times 480$图像组成。每张Quest 3画面则由两张单色 $1280 \times 1024$图像构成。所有图像均配有内参参数以及相机到世界的变换信息。Aria录音还包含3D场景点云（来自SLAM技术）以及眼动信号。详情请参阅补充材料。
33个物体的3D网格模型。这些模型是通过自主研发的基于扫描的3D物体重建流程获得的，可生成高分辨率的几何结构以及PBR [^39]材质。这些材质包括金属度贴图、粗糙度贴图和法线贴图，能够用于渲染照片级逼真的训练图像 [^32] [^34]。该物体集合涵盖了外观、尺寸及功能各异的家居和办公用品（见图3）。
19名特征各异的受试者。为确保多样性，我们招募了19名手部外观和形状各不相同的参与者。每位参与者的手部均通过定制的3D手部扫描仪进行扫描，数据以UmeTrack [^25]和MANO [^54]格式提供。
4种日常场景。除了简单的物品拾取、观察和放置场景外，还要求受试者分别在厨房、办公室和客厅执行典型动作。所有场景均在配备相应家具的同一实验室中拍摄。每次录音中，受试者需与最多6个物体互动。为增加数据集的多样性，我们定期随机调整光照、家具摆放和装饰元素等参数。最终生成的数据集包含425段录音，其中198段由Aria录制，226段由Quest 3录制。每段录音时长约为2分钟。
真实标注数据。录音资料中包含了手部和物体的逐帧真实姿态标注，这些标注是在运动捕捉实验室中获得的，具体可见图5并详见补充材料。物体和手腕的姿态以从3D模型空间到场景空间的3D刚体变换来表示，而手部姿态则以UmeTrack [^25]和MANO [^54]格式呈现（UmeTrack的精度更高，而MANO则更为通用——
Figure 3. High-quality 3D mesh models. This image shows a rendering of the 33 object models, demonstrating their quality. The models were obtained by an in-house scanning-based 3D object reconstruction pipeline and include PBR materials, which enable rendering of photo-realistic training images for methods that require it. The collection includes household and office objects of diverse appearance, size, and affordances.
Figure 4. Distances traveled by HOT3D objects. In total, subjects moved the 33 objects over 13 km. While objects like the keyboard and waffles were mostly resting, the white mug is a true explorer.
）。部分帧中的标注可能缺失或质量较低。在数据集包含的150万帧中，有116万帧完成了完整标注（即所有手部和物体的真实姿态均可获取），且通过了我们的视觉检查（手动标记出那些真实姿态下的手部和物体模型与实际图像对齐不佳的帧）。我们发布了全部150万帧数据，这些数据可用于无监督训练，同时提供了有效的116万帧数据的掩码。关于真实物体姿态的统计信息，请参见图4及补充材料。
训练集和测试集划分。HOT3D的训练集包含13名受试者的录音资料（100万帧多视角画面），测试集则包含剩余6名受试者的录音资料（50万帧多视角画面）。真实姿态标注仅对训练集公开发布。测试集的真实标注仅能通过专门的评估服务器访问。
精选视频片段（HOT3D-Clips）。为了便于对各类跟踪和姿态估计方法进行基准测试，我们还发布了从完整录像中提取的3832个精选片段（其中2804个来自训练集，1028个来自测试集；1983个来自Aria设备，1849个来自Quest 3设备）。每个片段包含150帧（5秒），且均通过了多项质量检验（确认所有手部和物体标注齐全、至少有一个手部和物体可见、排除曝光过度的帧），以及前文提到的视觉检查。
---
Figure 5. Motion-capture lab. The HOT3D dataset was collected using a motion-capture rig equipped with a few dozens of infrared exocentric OptiTrack cameras and light diffuser panels for illumination variability.
源自Aria和Quest 3的1849个样本）。每个视频片段包含150帧（5秒），均通过了多项质量测试（确认所有手部及物体标注完整、至少有一个可见的手部和物体、剔除曝光过度的帧），并符合我们前文提到的视觉检查标准。
物体导入序列。为了能够对无需模型的物体跟踪方法[^60]（这类方法能从参考图像中学习新物体）以及3D物体重建方法[^41]进行基准测试，HOT3D包含了两种展示物体所有可能视角的导入序列：（1）展示桌面上静止物体的序列，包括物体直立和倒置的情况；（2）展示手部操控物体的序列。静态导入设置适用于类似NeRF的重建方法[^41]，而后者则更适用于AR/VR应用场景，不过挑战性也更大[^22]。静态序列的所有帧都提供了物体真实姿态数据，但动态序列仅在第一帧提供相关数据。这是为了模拟现实场景：在静态设置中，可通过SfM[^57]轻松获取物体姿态，而在动态设置中获取则颇具挑战性。动态序列第一帧的真实姿态数据用于定义标准物体空间，这是评估6自由度物体跟踪所必需的。
### 4. 实验
可穿戴式头戴设备通常配备多个摄像头，这使其天然适合开发多视角3D视觉方法。在本节中，我们将证明对于几种常见的第一人称视角任务而言，多视角方法的表现优于单视角方法。首先，我们比较了UmeTrack[^25]方法用于3D手部跟踪的单视角版本和多视角版本。其次，我们将基于模型的6自由度物体姿态估计方法FoundPose[^49]扩展到多视角，并与原始版本进行对比评估。第三，我们开发了一种通过DINOv2[^48]特征立体匹配来实现未知手持物体3D重建的方法，并将其与类似OSNOM[^50]的单视角方法进行比较。需要注意的是，实验中所有的3D预测结果均以头戴设备坐标系表示，但由于头戴设备上配备了SLAM功能，可获取摄像头到世界坐标系的变换关系，因此也能将其转换为世界坐标系。
#### 4.1. 3D手部姿态跟踪
实验设置。给定手部形状（处于标准姿态的3D手部骨骼）以及可见手部的真实2D边界框，任务是在输入序列的每一帧中估计手部姿态（骨骼关节的3D位置）。我们在三种训练数据变体上训练UmeTrack [^25]手部追踪器：（1）来自UmeTrack数据集的训练序列；（2）使用Quest 3录制的HOT3D训练序列；（3）上述两种数据的组合。UmeTrack数据集是用Quest 2头显录制的，其摄像头与Quest 3相同但排列方式不同，该数据集包含1397个真实序列和1397个合成序列，每个序列以30帧/秒的速度录制15秒。这些序列展示了53名参与者进行的单手动作及双手交互，但不包含任何手-物体交互。所有三个UmeTrack模型均在双视图图像流上训练，其中其中一个视图会被随机遮蔽（如[^25]所示）。这种遮蔽操作能提升追踪的鲁棒性，并促使模型不依赖两个视图，从而能够公平比较它们的单视图与双视图模式。我们会在测试用UmeTrack序列的所有帧以及Quest 3拍摄的测试用HOT3D剪辑的所有帧上评估这些模型。预测出的3D位置的准确性通过平均关键点位置误差（MPKE）[^25]来衡量。
结果（表2）。当在UmeTrack数据集上训练时，单视图模式下的手部追踪器在HOT3D上的表现较差（HOT3D上的MKPE为24.2，而UmeTrack上为13.6）。同样，当在HOT3D上训练时，单视图追踪器在UmeTrack上的表现也较差（UmeTrack上的MKPE为23.7，而HOT3D上为18.0）。造成这些精度差异的主要原因是：手-物体交互仅出现在HOT3D中，而双手交互仅存在于UmeTrack数据集中。当仍在单一数据集上训练的追踪器在双视图模式下进行评估时，精度下降幅度更大。这是因为这些数据集是用不同头显（Quest 2与Quest 3）录制的，追踪器会过度拟合训练时所见的摄像头配置。当追踪器同时在两种数据集上训练时，两个数据集之间的域差异便被有效消除：在单视图模式下，UmeTrack上的MKPE降至13.4，HOT3D上降至15.4；而在双视图模式下，精度更是提升了41%（UmeTrack上的MKPE为9.5，HOT3D上为10.9）。
#### 4.2. 6自由度物体姿态估计
实验设置。在本实验中，我们专注于无需训练的6自由度物体姿态估计，即仅利用物体的CAD模型[^33]，在短暂的阶段内完成物体录入。我们评估了FoundPose [^49]——这是一种最新的开源方法，能在无需精修的情况下从单张RGB图像中得出最先进的姿态估计结果；同时我们也评估了我们提出的该方法在多视图输入下的扩展版本。
---
| Training dataset | Views | MKPE on UmeTrack $\downarrow$ | MKPE on HOT3D $\downarrow$ |
| UmeTrack [^25] | 1 | 13.6 | 24.2 |
| UmeTrack [^25] | 2 | 9.7 | 25.6 |
| HOT3D-Quest3 | 1 | 23.7 | 18.0 |
| HOT3D-Quest3 | 2 | 30.3 | 13.1 |
| UmeTrack + HOT3D-Quest3 | 1 | 13.4 | 15.4 |
| UmeTrack + HOT3D-Quest3 | 2 | 9.5 | 10.9 |
Table 2. 3D hand pose tracking by UmeTrack [^25]. Reported is the Mean Keypoint Position Error (MKPE, in mm) achieved on the UmeTrack and HOT3D-Quest3 datasets by single-view and two-view variants of the UmeTrack hand tracker, which was trained on training splits of either of the datasets or on their combination.
Figure 6. Example 3D hand pose tracking results by UmeTrack [^25]. Shown are hand skeletons and meshes of the UmeTrack hand model in the ground-truth and estimated poses. The UmeTrack hand tracker was provided the ground-truth hand skeleton and was tasked to estimate 3D locations of the skeleton joints.
详见下文。我们会在Aria和Quest 3拍摄的测试用HOT3D剪辑中，每隔30帧对这些方法的无精修版本进行评估。这些方法的输入是带有可见物体真实2D分割掩码的单视图/多视图帧。估计姿态的准确性通过召回率来衡量，即能估算出正确姿态的样本所占比例。如果姿态估计结果的平移误差与旋转误差均低于阈值，则该估计被视为正确。
FoundPose [^49]及其多视角扩展版本。在简短的初始化阶段，FoundPose会渲染展示不同方位下物体模型的RGB-D模板，同时提取DINOv2补丁特征。
| Test dataset | Views | Recall [%] $\uparrow$ |
| 5 cm, 5° | 10 cm, 10° | 20 cm, 20° |
| HOT3D-Aria | 1 | 25.2 | 41.7 | 54.5 |
| HOT3D-Aria | 3 | 33.8 | 52.9 | 66.2 |
| HOT3D-Quest3 | 1 | 28.9 | 46.6 | 58.9 |
| HOT3D-Quest3 | 2 | 36.9 | 55.9 | 66.4 |
Table 3. 6DoF object pose estimation by FoundPose [^49]. The proposed multi-view extension of FoundPose is compared with the original single-view version in recall rates on test images from both headsets. The recall rate is defined as the fraction of pose estimates whose translational and rotational errors are below a specified threshold.
RGB
Monochrome 1
Monochrome 2
Figure 7. Example 6DoF pose estimation results by FoundPose [^49]. Each row shows synchronized views of the same object from three Aria cameras. Our multi-view extension of FoundPose estimates the object pose from 2D-3D correspondences established between all available views and retrieved RGB-D templates. Thanks to the multi-view input, the method is able to estimate poses of heavily occluded objects (bottom row). Contours of 3D object models in the ground-truth and the estimated poses are shown in green and red respectively (top right corners). FoundPose was provided the ground-truth object mask as input (top left corners).
该方法提取RGB通道中的特征，并利用深度通道将这些特征在三维空间中注册。在推理阶段，FoundPose会围绕物体掩码对RGB查询图像进行裁剪，然后采用基于DINOv2的词袋方法检索出一组最相似的模板。对于每一个检索到的模板，系统会通过匹配图像裁剪区域与模板之间的DINOv2补丁特征来建立二维-三维对应关系，进而利用PnP-RANSAC生成姿态假设。最终，内点对应数量最多的姿态假设会被选为最终的姿态估计结果。
为了研究多视角输入与单视角输入在该任务中的效果差异，我们提出了一种对原始FoundPose方法的简单多视角扩展方案。该扩展版本沿用了相同的初始化阶段，但在推理时会裁剪所有视角下的物体图像。
---
接着，系统会选取各视角下得分总和最高的少量模板，建立每个模板与所有视角之间的多视角二维-三维对应关系，最后通过求解广义PnP问题来确定物体的姿态[^36]。
实验结果（见表3）：我们对FoundPose进行的简单多视角扩展显著优于原始的单视角版本，在两款头显设备的数据集上召回率提升了8%到12%（相对改进幅度为13%到34%）。除了能为三维推理提供额外约束外，多视角输入还能提供更多观测机会，从而识别在单视角下被严重遮挡或完全遮挡的物体。值得注意的是，FoundPose在Quest3的双视角单色设置以及Aria的三视角设置下均表现良好——其中一视角由高分辨率RGB相机采集，另外两视角则由低分辨率单色相机采集。我们将这种跨不同传感器的泛化能力主要归功于作为FoundPose核心组件的DINOv2。
#### 4.3. 手持物体的二维分割
实验设置：给定输入图像，任务目标是预测手部操控物体的二进制二维掩码。该任务不仅是手持物体三维重建的前提（见4.4节），对于手部状态分类或视频活动识别等下游应用也具有重要意义[^73]。我们评估了三种方法，包括现成的EgoHOS [^73]，以及两种Mask R-CNN的变体[^27]。这些模型均在包含52K个手持物体掩码的40万张Aria RGB图像组成的专有数据集上训练而成。其中一种模型直接基于RGB图像通道训练（表4中记为MRCNN），另一种则利用Depth Anything V2预测的深度通道数据训练[^70]（记为MRCNN-DA）。我们在Aria和Quest3设备上采集的训练及测试HOT3D视频中，每第30帧用于评估这些方法（共计约19K帧）。若物体与其在真实姿态下的手部网格顶点之间的最小距离小于1厘米，且物体的移动速度超过1厘米/秒，则该物体被视为手持物体。此类物体的掩码被用作该任务的真实标签。预测掩码的准确性通过平均交并比（mIoU）来衡量，具体计算方法如[^73]所述。
结果（表4）。与在EgoHOS数据集上的表现相比，EgoHOS [^73]模型在HOT3D上的准确率明显下降，我们认为这是由于数据集之间的领域差异所致。这一点在Quest 3单色图像上表现得尤为明显，其准确率降低了$\sim$50%。MRCNN-DA是性能最优的方法，在Aria和Quest 3帧上的表现分别比EgoHOS高出30%和65%。引入预测的深度图能够更精准地分辨手持前景物体与背景，从而生成更优的分割掩码（见图8）。
#### 4.4. 手持物体的三维重建
实验设置。最后，我们聚焦于未知手持物体的三维重建任务，该任务对于物体索引和长期跟踪具有重要意义[^50]。给定每个视角的手持物体二维分割掩码，
| Method | Test dataset | Object in hand (mIoU $\uparrow$): |
| Either | Left | Right | Both |
| EgoHOS [^73] | EgoHOS | - | 62.2 | 44.4 | 52.8 |
| EgoHOS [^73] | HOT3D-Aria | 42.6 | 21.0 | 26.3 | 32.5 |
| MRCNN | HOT3D-Aria | 47.1 | - | - | - |
| MRCNN-DA | HOT3D-Aria | 55.2 | - | - | - |
| EgoHOS [^73] | HOT3D-Quest3 | 33.1 | 13.5 | 14.4 | 24.8 |
| MRCNN | HOT3D-Quest3 | 37.8 | - | - | - |
| MRCNN-DA | HOT3D-Quest3 | 54.7 | - | - | - |
Table 4. 2D segmentation of in-hand objects. EgoHOS [^73] trained on the EgoHOS dataset is compared with our baselines based on Mask R-CNN [^27] and trained on our in-house dataset of images from Aria. We observe a large accuracy drop of the EgoHOS model on HOT3D.
Figure 8. Example results of 2D segmentation of in-hand objects. Masks predicted by EgoHOS [^73] (1st and 3rd row) are compared with masks predicted by our MRCNN-DA (2nd and 4th row). Predicted masks are shown in blue, and the contour of ground-truth masks in green.
我们的目标是估算物体的三维位置。我们开发了三种方法并对比了它们的性能，具体如下。估算位置的准确率通过召回率来衡量，即估算出正确位置的样本所占比例。如果某位置与真实位置（由三维物体边界框的中心定义）的偏移量低于阈值，则该位置被视为正确。
将手作为物体代理（HandProxy）。作为一个简单的基准方法，我们使用真实的掌心中心作为三维物体位置。我们将掌心中心定义为腕关节与中指第一节之间的中点。
---
| Method | Test dataset | Views | Recall [%] $\uparrow$ |
| 5 cm | 10 cm | 20 cm | 30 cm |
| HandProxy | HOT3D-Aria | - | 0.5 | 13.5 | 90.6 | 98.4 |
| Using ground-truth 2D segmentation masks: |
| MonoDepth | HOT3D-Aria | 1 | 14.3 | 30.2 | 53.6 | 69.9 |
| StereoMatch | HOT3D-Aria | 3 | 64.4 | 86.2 | 95.5 | 96.9 |
| StereoMatch | HOT3D-Quest3 | 2 | 76.4 | 96.8 | 99.1 | 99.2 |
| Using 2D segmentation masks predicted by MRCNN-DA: |
| MonoDepth | HOT3D-Aria | 1 | 11.1 | 23.3 | 43.7 | 58.2 |
| StereoMatch | HOT3D-Aria | 3 | 42.6 | 56.4 | 63.6 | 66.0 |
| StereoMatch | HOT3D-Quest3 | 2 | 59.1 | 75.3 | 80.4 | 81.3 |
Table 5. 3D lifting of in-hand objects. Shown are recall rates of our three baseline methods for several thresholds of correctness (a predicted 3D location is considered correct if its distance from the ground-truth location is below a threshold). The multi-view method (StereoMatch) clearly outperforms the other two methods at stricter threshold levels.
Figure 9. Example results of 3D lifting of in-hand objects. Ground-truth 3D object locations are shown in green, predictions from StereoMatch in red, MonoDepth in blue, and HandProxy in orange. The locations are projected to the three Aria views. In each example, the RGB image is on the left, monochrome images on the right, and DINOv2 stereo matching between the RGB and a monochrome image in the top left.
如果两只手都可见，我们就选择二维投影更接近二维物体掩码质心的那个掌心位置。这一基准方法的目的是评估该任务是否需要专门的解决方案，或者精准的三维手部追踪器是否就能提供足够的估算结果。
基于单目深度估计的重建（MonoDepth）。我们的第二种方法受到[^50]的启发，依赖于单目深度预测和稀疏SLAM观测数据。具体来说，我们先将鱼眼相机拍摄的图像校正为针孔相机视角，再将其输入Depth Anything V2 [^70]以生成单目深度图；接着通过尺度平移变换[^50]将深度图与SLAM生成的三维点云对齐；最后计算对齐后来自物体掩码范围内的三维点的中位数，以此作为三维物体位置。由于Quest 3无法提供SLAM观测数据，我们仅在Aria的记录数据（使用RGB图像）上对该方法进行评估。
基于立体匹配的重建（StereoMatch）。我们的第三种手持物体三维重建方法基于立体图像中DINOv2特征的匹配。给定两个视角下的二维物体分割掩码后，我们首先构建一对立体裁剪区域：这些区域紧密围绕给定的掩码，分辨率均为$420 \times 420$像素，且一个裁剪区域中的像素在另一个裁剪区域的同一行中必有对应像素[^4]。接着，我们使用DINOv2 ViT-S [^48]从每个裁剪区域中提取补丁特征；通过将一个裁剪区域中的每个补丁与另一个裁剪区域同一行中距离最近的补丁（依据DINOv2特征向量的L2距离判定）关联起来，建立裁剪区域间的二维对应关系。我们保留循环距离最小的至多500个对应关系[^20]，再通过三角测量得到一组三维点。最终以这组三维点的稳健均值来估算三维物体位置。
结果（表5）。无论使用真实分割掩码还是由MRCNN-DA预测的掩码作为输入，StereoMatch方法的性能都优于MonoDepth。MonoDepth的3D定位误差主要出现在光轴方向上，这是由深度预测不准确导致的。这一点在图9中表现得尤为明显：在相对于输入RGB相机基线较大的摄像头视角下，定位误差最为显著。最后，HandProxy基准测试表明，虽然仅依靠手部追踪就足以将物体定位在30厘米的不确定性半径内（与[^50]中的评估结果类似），但针对手持物体的多视图3D重建专用方法能为10厘米范围内的更精细定位提供额外价值。
### 5. 结论
我们推出了HOT3D——一个大规模数据集，旨在为各类与手物交互相关的2D及3D第一人称视觉任务的训练与评估提供支持。实验结果表明，HOT3D独有的多视图方法基准测试能力使得多视图方法在多个常见任务上的表现显著优于单视图方法。除了多视图3D物体追踪外，我们还研究了多视图6DoF物体姿态估计以及手持物体的3D重建任务，并为此开发了可靠的基准模型。通过公开发布该数据集并围绕它举办公开挑战赛$^{12}$，我们希望能推动第一人称视觉与情境人工智能领域的研究进展。
---
## 致谢
特别感谢Lingni Ma在初期讨论中提供的宝贵思路，以及她在动作捕捉领域分享的专业知识，这些都为HOT3D项目的开展奠定了坚实基础。感谢Kevin Harris和Steve Olsen在搭建动作捕捉实验室时给予的无价帮助与专业指导。感谢Ben Schneider、Mateja Kodila、Elizabeth Bères、Pablo Gomez Ponce和Nicolas Greiner在数据采集组织及数据处理方面提供的协助。感谢Fadime Sener、Bugra Tekin、Yann Labbé、Christian Forster、Mariano Jaimez、Rainer Stal、Timo Stoffregen、Pierluigi Taddei、Eric Sauser、Shitai Li、Lukas Bode和Ben Schneider参与数据采集工作。感谢Evgeniy Oleinik和Maien Hamed在Meta基础设施上托管数据集时给予的协助。感谢Xiaqing Pan在开源HOT3D工具包方面提供的指导。感谢Mark Schwesinger、Christopher Pistritto、Elizabeth Argall和Josiah Zacharias为确保HOT3D按时发布所提供的法律与写作支持。感谢Nick Charron、Alexander Gamino和David Caruso在校准及SLAM-OptiTrack对齐方面给予的帮助。感谢Ka Chen、Allison Tilp和Thibaud Gayraud在创建HOT3D物体的高精度CAD模型时提供的支持。感谢Abha Arora、Luis Pesqueira和Yuyang Zou在组织Aria数据采集活动及搭建Aria数据采集基础设施方面给予的协助。感谢Tomasz Malisiewicz在训练2D手持物体分割模型时提供的帮助。感谢Weiguang Si在为ECCV 2024的手部追踪挑战赛准备HOT3D数据时所做的贡献。
## 参考文献
[^1]: Sven Bambach, Stefan Lee, David J. Crandall, and Chen Yu. Lending a hand: Detecting hands and recognizing activities in complex egocentric interactions. In ICCV, 2015. 3
[^2]: Ameline Bardo, Katie Town, Tracy L Kivell, Georgina Donati, Haiko Ballieux, Cosmin Stamate, Trudi Edginton, and Gillian S Forrester. The precision of the human hand: variability in pinch strength and manual dexterity. Symmetry, 2022. 1
[^3]: Bharat Lal Bhatnagar, Xianghui Xie, Ilya Petrov, Cristian Sminchisescu, Christian Theobalt, and Gerard Pons-Moll. BEHAVE: Dataset and method for tracking human object interactions. In CVPR, 2022. 3
[^4]: Gary Bradski. Learning OpenCV: Computer vision with the OpenCV library. O'REILLY, 2008. 8
[^5]: Samarth Brahmbhatt, Catherine Ham, Charles C Kemp, and James Hays. ContactPose: A dataset of grasps with object contact and hand pose. In ECCV, 2020. 3
[^6]: Ian M. Bullock, Thomas Feix, and Aaron M. Dollar. The Yale human grasping dataset: Grasp, object, and task data in household and machine shop environments. IJRR, 2015. 3
[^7]: Berk Calli, Arjun Singh, Aaron Walsman, Siddhartha Srinivasa, Pieter Abbeel, and Aaron M Dollar. The YCV object and model set: Towards common benchmarks for manipulation research. In ICAR, 2015. 3
[^8]: Yu-Wei Chao, Wei Yang, Yu Xiang, Pavlo Molchanov, Ankur Handa, Jonathan Tremblay, Yashraj S. Narang, Karl Van Wyk, Umar Iqbal, Stan Birchfield, Jan Kautz, and Dieter Fox. DexYCB:
用于捕捉物体抓取动作的基准测试。载于CVPR，2021年。第3页。
[^9]: Woojin Cho, Jihyun Lee, Minjae Yi, Minje Kim, Taeyun Woo, Donghwan Kim, Taewook Ha, Hyokeun Lee, Je-Hwan Ryu, Woon-tack Woo, and Tae-Kyun Kim. Dense hand-object(HO) GraspNet with full grasping taxonomy and dynamics. In ECCV, 2024. 3
[^10]: Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Jian Ma, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. Rescaling egocentric vision: Collection, pipeline and challenges for EPIC-KITCHENS-100. IJCV, 2022. 3
[^11]: Maximilian Denninger, Martin Sundermeyer, Dominik Winkelbauer, Dmitry Olefir, Tomáš Hodaň, Youssef Zidan, Mohamad Elbadrawy, Markus Knauer, Harinandan Katam, and Ahsan Lodhi. BlenderProc: Reducing the reality gap with photorealistic rendering. RSS Workshops, 2020. 3
[^12]: Maximilian Denninger, Martin Sundermeyer, Dominik Winkelbauer, Youssef Zidan, Dmitry Olefir, Mohamad Elbadrawy, Ahsan Lodhi, and Harinandan Katam. Blenderproc. arXiv preprint arXiv:1911.01911, 2019. 3
[^13]: Jakob Engel, Kiran Somasundaram, Michael Goesele, Albert Sun, Alexander Gamino, Andrew Turner, Arjang Talatoff, Arnie Yuan, Bilal Souti, Brighid Meredith, Cheng Peng, Chris Sweeney, Cole Wilson, Dan Barnes, Daniel DeTone, David Caruso, Derek Valleroy, Dinesh Ginjupalli, Duncan Frost, Edward Miller, Elias Mueggler, Evgenio Oleinik, Fan Zhang, Guruprasad Somasundaram, Gustavo Solaira, Harry Lanaras, Henry Howard-Jenkins, Huixuan Tang, Hyo Jin Kim, Jaime Rivera, Ji Luo, Jing Dong, Julian Straub, Kevin Bailey, Kevin Eckenhoff, Lingni Ma, Luis Pesqueira, Mark Schwesinger, Maurizio Monge, Nan Yang, Nick Charron, Nikhil Raina, Omkar Parkhi, Peter Borschowa, Pierre Moulon, Prince Gupta, Raul Mur-Artal, Robbie Pennington, Sachin Kulkarni, Sagar Miglani, Santosh Gondi, Saransh Solanki, Sean Diener, Shangyi Cheng, Simon Green, Steve Saarinen, Suvam Patra, Tassos Mourikis, Thomas Whelan, Tripti Singh, Vasileios Balntas, Vijay Baiyya, Wilson Dreewes, Xiaqing Pan, Yang Lou, Yipu Zhao, Yusuf Mansour, Yuyang Zou, Zhaoyang Lv, Zijian Wang, Mingfei Yan, Carl Ren, Renzo De Nardi, and Richard Newcombe. Project Aria: A new tool for egocentric multi-modal AI research, 2023. 1, 2, 4
[^14]: Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The PASCAL visual object classes (VOC) challenge. IJCV, 2010. 3
[^15]: Zicong Fan, Omid Taheri, Dimitrios Tzionas, Muhammed Kocabas, Manuel Kaufmann, Michael J Black, and Otmar Hilliges. ARCTIC: A dataset for dexterous bimanual hand-object manipulation. In CVPR, 2023. 3,4
[^16]: Alireza Fathi, Xiaofeng Ren, and James M. Rehg. Learning to recognize objects in egocentric activities. In CVPR, 2011. 3
[^17]: Yang Fu and Xiaolong Wang. Category-level 6D object pose estimation in the wild: A semi-supervised learning approach and a new dataset. NeurIPS, 2022. 3
[^18]: Guillermo Garcia-Hernando, Shanxin Yuan, Seungryul Baek, and Tae-Kyun Kim. First-person hand action benchmark with RGB-D videos and 3D hand pose annotations. In CVPR, 2018. 3
[^19]: Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? The KITTI vision benchmark suite. In CVPR, 2012. 3
## [^20]: Walter Goodwin, Sagar Vaze, Ioannis Havoutis, and Ingmar Posner. Zero-shot category-level object pose estimation. In ECCV, 2022. 8
---
[^21]: Andrew Guo, Bowen Wen, Jianhe Yuan, Jonathan Tremblay, Stephen Tyree, Jeffrey Smith, and Stan Birchfield. HANDAL: A dataset of real-world manipulable object categories with pose annotations, affordances, and reconstructions. In IROS, 2023. 3
[^22]: Shreyas Hampali, Tomas Hodan, Luan Tran, Lingni Ma, Cem Keskin, and Vincent Lepetit. In-hand 3D object scanning from an RGB sequence. In CVPR, 2023. 5
[^23]: Shreyas Hampali, Mahdi Rad, Markus Oberweger, and Vincent Lepetit. HOnnotate: A method for 3D annotation of hand and object poses. In CVPR, pages 3196–3206, 2020. 3
[^24]: Shreyas Hampali, Sayan Deb Sarkar, Mahdi Rad, and Vincent Lepetit. Keypoint transformer: Solving joint identification in challenging hands and object interactions for accurate 3D pose estimation. In CVPR, 2022. 3
[^25]: Shangchen Han, Po-chen Wu, Yubo Zhang, Beibei Liu, Linguang Zhang, Zheng Wang, Weiguang Si, Peizhao Zhang, Yujun Cai, Tomas Hodan, et al. UmeTrack: Unified multi-view end-to-end hand tracking for VR. In SIGGRAPH Asia 2022, 2022. 3, 4, 5, 6
[^26]: Yana Hasson, Gül Varol, Dimitrios Tzionas, Igor Kalevatykh, Michael J Black, Ivan Laptev, and Cordelia Schmid. Learning joint reconstruction of hands and manipulated objects. In CVPR, 2019. 3
[^27]: Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask R-CNN. ICCV, 2017. 7
[^28]: Tony Heap and David Hogg. Towards 3D hand tracking using a deformable model. In Proceedings of the Second International Conference on Automatic Face and Gesture Recognition, 1996. 3
[^29]: Stefan Hinterstoisser, Vincent Lepetit, Slobodan Ilic, Stefan Holzer, Gary Bradski, Kurt Konolige, and Nassir Navab. Model based training, detection and pose estimation of texture-less 3D objects in heavily cluttered scenes. In ACCV, 2013. 3
[^30]: Tomáš Hodaň, Pavel Haluza, Štěpán Obdržálek, Jiří Matas, Manolis Lourakis, and Xenophon Zabulis. T-LESS: An RGB-D dataset for 6D pose estimation of texture-less objects. WACV, 2017. 3
[^32]: Tomáš Hodaň, Martin Sundermeyer, Bertram Drost, Yann Labbé, Eric Brachmann, Frank Michel, Carsten Rother, and Jiří Matas. BOP Challenge 2020 on 6D object localization. In ECCV, 2020. 3,4
[^31]: Tomáš Hodaň, Frank Michel, Eric Brachmann, Wadim Kehl, Anders Glent Buch, Dirk Kraft, Bertram Drost, Joel Vidal, Stephan Ihrke, Xenophon Zabulis, Caner Sahin, Fabian Manhardt, Federico Tombari, Tae-Kyun Kim, Jiří Matas, and Carsten Rother. BOP: Benchmark for 6D object pose estimation. ECCV, 2018. 3
[^33]: Tomáš Hodaň, Martin Sundermeyer, Yann Labbé, Van Nguyen Nguyen, Gu Wang, Eric Brachmann, Bertram Drost, Vincent Lepetit, Carsten Rother, and Jiří Matas. BOP challenge 2023 on detection, segmentation and pose estimation of seen and unseen rigid objects. CVPRW, 2024. 3, 5
[^34]: Tomáš Hodaň, Vibhav Vineet, Ran Gal, Emanuel Shalev, Jon Hanzelka, Treb Connell, Pedro Urbina, Sudipta Sinha, and Brian Guenter. Photorealistic image synthesis for object instance detection. ICIP, 2019. 4
[^35]: HyunJun Jung, Guangyao Zhai, Shun-Cheng Wu, Patrick Ruhkamp, Hannah Schieber, Giulia Rizzoli, Pengyuan Wang, Hongcheng Zhao, Lorenzo Garattoni, Sven Meier, et al. HouseCat6D – a large-scale multi-modal category level 6D object perception dataset with household objects in realistic scenarios. arXiv preprint arXiv:2212.10428, 2022. 3
[^36]: Zuzana Kukelova, Jan Heller, and Andrew Fitzgibbon. Efficient intersection of three quadrics and applications in computer vision. In CVPR, 2016. 7
[^37]: Taein Kwon, Bugra Tekin, Jan Stuhmer, Federica Bogo, and Marc Pollefeys. H2O: A benchmark for egocentric hand-object interaction recognition. IEEE Transactions on Multimedia, 2020. 3
[^38]: Yunze Liu, Yun Liu, Che Jiang, Kangbo Lyu, Weikang Wan, Hao Shen, Boqiang Liang, Zhoujie Fu, He Wang, and Li Yi. HOI4D: A 4D egocentric dataset for category-level human-object interaction. In CVPR, 2022. 3
[^39]: Wes McDermott. The PBR guide. Allegorithmic, 2018. 2,4
[^40]: Meta.Quest3.
[^41]: Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing scenes as neural radiance fields for view synthesis. ECCV, 2020. 5
[^42]: Gyeongsik Moon, Shoou-I Yu, He Wen, Takaaki Shiratori, and Kyoung Mu Lee. Interhand2.6M: A dataset and baseline for 3D interacting hand pose estimation from a single RGB image. In ECCV, 2020. 3
[^43]: Franziska Mueller, Dushyant Mehta, Oleksandr Sotnychenko, Srinath Sridhar, Dan Casas, and Christian Theobalt. Real-time hand tracking under occlusion from an egocentric RGB-D sensor. In ICCV, 2017. 3
[^44]: Hiroshi Murase and Shree K Nayar. Visual learning and recognition of 3-D objects from appearance. IJCV, 1995. 3
[^45]: Richard A Newcombe, Shahram Izadi, Otmar Hilliges, David Molyneaux, David Kim, Andrew J Davison, Pushmeet Kohi, Jamie Shotton, Steve Hodges, and Andrew Fitzgibbon. KinectFusion: Real-time dense surface mapping and tracking. ISMAR, 2011. 3
[^46]: Takehi - Tomas Hodan, Luan -entric activ-3.3
[^47]: Iason Oikonomidis, Guillermo Garcia-Hernando, Angela Yao, Antonis Argyros, Vincent Lepetit, and Tae-Kyun Kim. HANDS18: Methods, techniques and applications for hand observation. In ECCVW, 2018. 1
[^48]: Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. DINOv2: Learning robust visual features without supervision. TMLR, 2024. 2, 5, 8
[^49]: Evin Pınar Örnek, Yann Labbé, Bugra Tekin, Lingni Ma, Cem Keskin, Christian Forster, and Tomas Hodan. FoundPose: Unseen object pose estimation with foundation features. ECCV, 2024. 2, 5, 6
[^50]: Chiara Plizzari, Shubham Goel, Toby Perrett, Jacob Chalk, Angjoo Kanazawa, and Dima Damen. Spatial cognition from egocentric video: Out of sight, not out of mind. arXiv preprint arXiv:2404.05072, 2024. 5, 7, 8
[^51]: Francesco Ragusa, Rosario Leonardi, Michele Mazzamuto, Claudia Bonanno, Rosario Scavo, Antonino Furnari, and Giovanni Maria Farinella. Enigma-51: Towards a fine-grained understanding of human-object interactions in industrial scenarios. IEEE Winter Conference on Application of Computer Vision (WACV), 2024. 3
[^52]: James M Rehg and Takeo Kanade. Visual tracking of high DoF articulated structures: an application to human hand tracking. In ECCV, 1994. 3
[^53]: Lawrence G Roberts. Machine perception of three-dimensional solids. PhD thesis, Massachusetts Institute of Technology, 1963. 3
## [^54]: Javier Romero, Dimitrios Tzionas, and Michael J Black. Embodied hands: Modeling and capturing hands and bodies together. TOG, 2017. 4
---
[^55]: Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. IJCV, 2015. 3
[^56]: Daniel Scharstein and Richard Szeliski. A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. IJCV, 2002. 3
[^57]: Johannes L Schonberger and Jan-Michael Frahm. Structure-from-motion revisited. In CVPR, 2016. 5
[^58]: Dandan Shan, Jiaqi Geng, Michelle Shu, and David F Fouhey. Understanding human hands in contact at internet scale. In CVPR, 2020. 3
[^59]: Srinath Sridhar, Franziska Mueller, Michael Zollhöfer, Dan Casas, Antti Oulasvirta, and Christian Theobalt. Real-time joint tracking of a hand manipulating an object from RGB-D input. In ECCV, 2016. 3
[^60]: Jiaming Sun, Zihao Wang, Siyu Zhang, Xingyi He, Hongcheng Zhao, Guofeng Zhang, and Xiaowei Zhou. OnePose: One-shot object pose estimation without cad models. In CVPR, 2022. 3,5
[^61]: Xiao Sun, Yichen Wei, Shuang Liang, Xiaoou Tang, and Jian Sun. Cascaded hand pose regression. In CVPR, 2015. 3
[^62]: Martin Sundermeyer, Tomas Hodan, Yann Labbé, Gu Wang, Eric Brachmann, Bertram Drost, Carsten Rother, and Jiri Matas. BOP challenge 2022 on detection, segmentation and pose estimation of specific rigid objects. CVPRW, 2023. 3
[^63]: Danhang Tang, Hyung Jin Chang, Alykhan Tejani, and Tae-Kyun Kim. Latent regression forest: Structured estimation of 3D articulated hand posture. In CVPR, 2014. 3
[^64]: Jonathan Tompson, Murphy Stein, Yann Lecun, and Ken Perlin. Real-time continuous pose recovery of human hands using convolutional networks. ToG, 2014. 3
[^65]: Dimitrios Tzionas, Luca Ballan, Abhilash Srikantha, Pablo Aponte, Marc Pollefeys, and Juergen Gall. Capturing hands in action using discriminative salient points and physics simulation. IJCV, 2016. 3
[^66]: Dorin Ungureanu, Federica Bogo, Silvano Galliani, Pooja Sama, Xin Duan, Casey Meekhof, Jan Stühmer, Thomas J. Cashman, Bugra Tekin, Johannes L. Schönberger, et al. Hololens 2 research mode as a tool for computer vision research. arXiv, 2020.2
[^67]: Jikai Wang, Qifan Zhang, Yu-Wei Chao, Bowen Wen, Xiaohu Guo, and Yu Xiang. Ho-cap: A capture system and dataset for 3d reconstruction and pose tracking of hand-object interaction, 2024. 3, 4
[^68]: Pengyuan Wang, HyunJun Jung, Yitong Li, Siyuan Shen, Rahul Parthasarathy Srikanth, Lorenzo Garattoni, Sven Meier, Nassir Navab, and Benjamin Busam. Phocal: A multi-modal dataset for category-level object pose estimation with photometrically challenging objects. In CVPR, 2022. 3
[^69]: Yu Xiang, Tanner Schmidt, Venkatraman Narayanan, and Dieter Fox. PoseCNN: A convolutional neural network for 6D object pose estimation in cluttered scenes. RSS, 2018. 3
[^70]: Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything v2. arXiv:2406.09414, 2024. 7, 8
[^71]: Shanxin Yuan, Qi Ye, Guillermo Garcia-Hernando, and Tae-Kyun Kim. The 2017 HANDS in the million challenge on 3D hand pose estimation. arXiv preprint arXiv:1707.02237, 2017. 3
[^72]: Jiawei Zhang, Jianbo Jiao, Mingliang Chen, Liangqiong Qu, Xiaobin Xu, and Qingxiong Yang. 3D hand pose tracking and estimation using stereo matching. arXiv preprint arXiv:1610.07214, 2016. 3
[^73]: Lingzhi Zhang, Shenghao Zhou, Simon Stent, and Jianbo Shi. Fine-grained egocentric hand-object segmentation: Dataset, model, and applications. In ECCV, 2022. 3, 7
[^74]: Christian Zimmermann, Duygu Ceylan, Jimei Yang, Bryan Russell, Max Argus, and Thomas Brox. FreiHAND: A dataset for markerless capture of hand pose and shape from single RGB images. In ICCV, 2019. 3