# EgoPressure：一个用于自我中心视觉中的手部压力与姿态估计的数据集
赵一鸣 $^{1*}$ 权泰仁 $^{1*}$ 保罗·斯特雷利 $^{1*}$ 马克·波勒费斯 $^{1,2}$ 克里斯蒂安·霍尔茨 $^{1}$
$^{1}$苏黎世联邦理工学院 $^{2}$微软空间人工智能实验室，苏黎世
Figure 1. The EgoPressure dataset. We introduce a novel egocentric pressure dataset with hand poses. We label hand poses using our proposed optimization method across all static camera views (Cameras 1–7). The annotated hand mesh aligns well with the egocentric camera's view, indicating the high fidelity of our annotations. We project the pressure intensity and annotated hand mesh (Fig. i) to all camera views (Fig. a to h), and further provide the pressure applied over the hand as a UV texture map (Fig. j and k).
## 摘要
触觉接触与压力对于理解人类如何与物体交互及操控物体至关重要，这些认知能够显著推动混合现实和机器人领域应用的发展。然而，从以自我为中心的摄像头视角来估算这类交互极具挑战性，主要原因是缺乏既能提供接触表面上精确手部姿态、又能标注详细压力信息的综合性数据集。在本文中，我们提出了EgoPressure——一种全新的以自我为中心的数据集，用于捕捉精细的触觉接触与压力交互数据。EgoPressure为每个接触点提供了高分辨率的压力强度标注，并通过我们提出的基于多视图和序列优化的方法，利用8个摄像头组成的采集设备获取精确的手部姿态网格。该数据集包含了21名参与者长达5小时的互动记录，这些记录由1个头戴式摄像头和7个固定式Kinect摄像头同步采集，摄像头以30 Hz的频率拍摄RGB图像并生成深度图。为了支持未来的研究与基准测试，我们给出了几种基于RGB图像估算外部表面上所受压力的基线模型，无论是否包含手部姿态信息均适用。
我们还探索了手部网格与所施加压力的联合估算方法。实验结果表明，压力数据与手部姿态信息在理解手-物交互方面具有互补作用。项目页面：。
### 1. 引言
在手部-物体交互过程中感知触觉接触和压力，对于多种任务而言都至关重要——
物体交互对于增强现实领域的各类任务而言至关重要。
增强现实（AR）[^36] [^88]、虚拟现实（VR）[^25] [^91]，
以及机器人操控领域[^11] [^13] [^60]。具体而言，估算——
从自我中心的角度来理解这些物理属性
这是支持现实世界任务的核心前提 [^62] [^82]。在
AR/VR环境中，触觉接触和压力信息能够
提供更精准的控制与反馈 [^10]。例如
当用户在桌面上弹奏虚拟钢琴时，
琴音会根据按键时所受的压力发生变化，
从而带来当前AR/VR系统所欠缺的更精细的反馈体验
压力感知对于机器人准确复现人类操作而言也至关重要 [^62]。
机器人要能精准复制人类的动作，
就必须确定抓取物体所需的精确压力
而这仍是一项重大挑战 [^11] [^13] [^48]。
以往的研究采用手套 [^58] [^59] 和机器人
---
借助触觉传感器 [^48] [^98] 来采集物体操作过程中的压力数据。不过，这类仪器会阻碍触觉反馈，从而干扰自然的触感体验。相比之下，基于视觉的估算方法无需在手上安装任何仪器，而且摄像头早已集成到智能眼镜和混合现实头戴设备等设备中，这些设备被广泛用于从自我中心视角研究人类行为 [^26] [^27]。尽管有这些优势，但最先进（SOTA）模型的进展仍受限于缺乏能提供接触和压力数据的数据集。一个显著的例外是PressureVision数据集 [^24]，该数据集包含了四台固定摄像头拍摄的手部与压敏表面交互时的RGB视频，以及对应的投影压力图像。
在本文中，我们在前人工作的基础上进行了自然且重要的拓展 [^24] [^25]，推出了全新的EgoPressure数据集，该数据集从自我中心视角捕捉手部与表面的交互情况，还包含精确的手部姿态标注以及投影到手部网格上的压力图。我们的采集平台结合了SenseMorph触摸板、头戴式摄像头以及七台同步运行的Azure Kinect摄像头，可以30 Hz的频率录制RGB-D数据（见图1）。该数据集包含21名参与者录制的5小时视频，每位参与者需完成64组交互序列，平均长度为420帧——因此它是首个带有姿态和网格标注的裸手自我中心压力数据集。
我们还提供了基准模型，用以展示该数据集的潜力并为后续研究建立基准。首先，我们将PressureVisionNet [^24]设为该自我中心数据集上的基准模型，并将其与加入手部姿态作为额外输入的改进模型进行比较。利用HaMeR [^67]估算器从RGB图像中得到的手部姿态模型，其体积IoU误差比PressureVisionNet低5%以上；若使用真实的手部姿态，误差更是能降低7%以上。此外，我们还推出了首个能够利用自我中心的RGB摄像头同时估算手部姿态、手部网格以及网格和表面上压力的模型，从而实现三维空间中接触点和压力的定位。
我们将主要贡献总结如下：
1. EgoPressure是首个高质量的自我中心触觉接触和压力数据集，包含三维手部姿态信息。这有利于开发能够适配头戴式和穿戴式等移动摄像头的模型。论文被接收后，我们将公开分享该数据集及标注内容。
2. 我们提出了一种优化方法，可利用MANO [^73]从多视角采集系统中标注手部姿态。基于网格的手部姿态标注考虑了顶点位移因素，通过将SenseI Morph触摸板上的压力反向投影到手部网格上，可支持精准的手部操作分析。
3. 我们设立了两项全新基准：（1）利用自我中心的RGB图像估算接触压力，分别考虑有无手部姿态信息的情况；（2）联合重建三维手部姿态和压力，包括定位用户手部网格上的压力分布。
因此，EgoPressure为未来模型应对自我中心视角下的独特挑战、精准定位用户手部的压力分布提供了新的机遇。
### 2. 相关工作
我们的研究与手部-物体姿态估算、接触估算以及压力传感领域相关。
基于视觉的手-物体姿态估计。手部追踪一直是计算机视觉领域长期面临的挑战，其在机器人技术[^12] [^50]、人机交互[^31] [^32] [^70]以及医学领域[^3] [^35] [^57]均有广泛应用。过去十年间，得益于深度学习技术的进步[^65] [^67]以及相关数据集的收集[^64] [^93] [^97]，该领域取得了显著进展。虽然用于手势识别和直接输入的中心手追踪技术已经发展到可集成到AR和VR头戴设备等现代商用设备中的程度[^31] [^32]，但理解手部与外部物体的相互作用仍是一个活跃的研究领域[^20] [^21] [^27] [^47]。为帮助机器理解此类手-物体相互作用而收集的数据集，需要借助额外的工具对用户手部进行标记[^21]，或使用带有手部标记的动作捕捉系统[^20] [^84]，又或是多视角摄像机阵列[^7] [^30] [^47] [^92] [^94]，以便在物体造成的高度遮挡情况下获取用户手部的精确真实姿态。
手-物体接触点估计。除了物体相对手部姿态外，以往的研究还致力于估算用户手部与外部物体之间的接触点[^20] [^84]。研究表明，当作为输入代理时，现实世界中的物理物体能够改善输入控制并提供触觉反馈[^10]。在互动研究方面，研究人员利用外部追踪系统[^10] [^71]以及声学传感器[^77]和惯性测量单元等可穿戴传感器来估算接触点[^22] [^28] [^62] [^78] [^80] [^85]。此外，人们还开发了基于视觉的技术，利用基准标记[^49]、用于产生阴影的主动照明[^52] [^88]、振动检测[^81]或深度感知[6, 19, 29, 74, 76, 89–91]来实现接触点估算。更近期的研究则利用无需在用户手部或表面上添加额外设备的被动摄像头来估算触碰情况，从而能够将其部署在商用混合现实头戴设备上[^72] [^82]。更详细的接触图是基于追踪到的手部与物体网格的交集推断得出的[^20] [^23] [^47] [^84] [^94]，这要求达到亚毫米级的精度——由于软组织动态特性的影响，对于复杂手势而言这是一项极具挑战性的任务。为解决这一问题，
---
Table 1. Comparison between EgoPressure and selected hand-contact datasets. The overwhelming majority of prior datasets infer contacts based on hand and object pose. ContactLabelDB and PressureVisionDB also include ground-truth touch pressure but are limited to static cameras and do not provide accurate hand poses and meshes. Please see appendix for the full table.
| Dataset | frames | participants | hand pose | hand mesh | markerless | real | egocentric | multiview | RGB | depth | contact | pressure surface | hand |
| EgoPressure (ours) | 4.3M | 21 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | Pressure sensor | ✓ | ✓ |
| ContactLabelDB [^25] | 2.9M | 51 | ✗ | ✗ | ✓ | ✓ | ✗ | ✓ | ✓ | ✗ | Pressure sensor | ✓ | ✗ |
| PressureVisionDB [^24] | 3.0M | 36 | ✗ | ✗ | ✓ | ✓ | ✗ | ✓ | ✓ | ✗ | Pressure sensor | ✓ | ✗ |
| ContactPose [^4] | 3.0M | 50 | ✓ | ✓ | ✓ | ✓ | ✗ | ✓ | ✓ | ✓ | Thermal imprint | ✗ | ✗ |
| GRAB [^84] | 1.6M | 10 | ✓ | ✓ | ✗ | ✓ | ✗ | ✗ | ✗ | ✗ | Inferred from Pose | ✗ | ✗ |
| ARCTIC [^20] | 2.1M | 10 | ✓ | ✓ | ✗ | ✓ | ✓ | ✓ | ✓ | ✓ | Inferred from Pose | ✗ | ✗ |
| $H_{{20}}$ [^47] | 571k | 4 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | Inferred from Pose | ✗ | ✗ |
| OakInk [^92] | 230k | 12 | ✓ | ✓ | ✓ | ✓ | ✗ | ✓ | ✓ | ✓ | Inferred from Pose | ✗ | ✗ |
| OakInk-2 [^94] | 4.01M | 9 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | Inferred from Pose | ✗ | ✗ |
| DexYCB [^7] | 582k | 10 | ✓ | ✓ | ✓ | ✓ | ✗ | ✓ | ✓ | ✓ | Inferred from Pose | ✗ | ✗ |
| HO-3D [^30] | 103k | 10 | ✓ | ✓ | ✓ | ✓ | ✗ | ✓ | ✓ | ✓ | Inferred from Pose | ✗ | ✗ |
| TACO [^56] | 5.2M | 14 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | Inferred from Pose | ✗ | ✗ |
Brahmbhatt等人[^4]利用热成像技术来获取精确的接触图。此外，以往的研究还利用模拟方法来获取关于接触组织的更细致标签[^14] [^33] [^96]。
手部压力估计：超越简单的接触检测，以往的研究已经能够估算手部交互过程中施加的压力力，这对机器人抓取任务至关重要[^11] [^60]，同时也为输入控制提供了额外的维度[^69]。为了从单目图像中估算压力，研究人员利用了视觉线索，比如按压事件中的指甲变化[^8] [^61]或表面形变[^39] [^63]。物体轨迹和交互力的变化[^18] [^51] [^68]也能提供相关信息，但对于桌子和墙壁等静止物体而言效果有限。用于训练的精确压力标签通常需要给用户的手戴上手套[^5] [^53] [^83]，或在表面安装力传感器[^24] [^68] [^75]，理想情况下这些设备还应具备柔韧性或能贴合各种形状[^2] [^45] [^58]。然而，这种做法会改变手部和表面的外观及触觉特征，进而影响交互效果，并限制其在裸手和非受控表面上的普适性。Grady等人[^24] [^25]利用SenseMorph [^41]压力传感器收集了两套带有真实压力图的数据集，用以训练神经网络，从而从单张RGB图像中估算出表面的接触区域。不过，他们的方法依赖外部静态摄像头以及对应指尖的良好可见性。
借助EgoPressure，我们旨在填补这一空白，提供包含自中心视角的数据集，利用头戴式摄像头从这一视角更好地理解人类交互行为。我们的数据集还能在无需标记的情况下，从多个摄像头视角捕捉精确的手部姿态和网格模型。据我们所知，这是首个包含裸手与表面交互的自中心及多视角RGB-D图像的数据集，其中还同步记录了压力数据、手部姿态和网格模型（见表1）。
### 3. 无标记标注方法
为了在无标记的情况下捕捉手-表面交互过程中的精确手部姿态和网格模型，我们开发了一种基于多摄像头的手部姿态标注方法，该方法运用了MANO手部模型[^73]、可微渲染技术以及多目标优化算法。图2展示了我们方法的概览，该方法依赖于C个静态摄像头和一个压敏触摸板。关于我们标注方法的详细评估结果，请参阅补充材料。
#### 3.1. 手部姿态自动初始化
我们利用HaMeR [^67]来为每个静态摄像头估算初始的MANO手部姿态$\theta_{init}$和平移量$t_{init}$。由于HaMeR的预测是基于单张RGB图像，因此存在尺度-平移歧义，我们通过从7个静态摄像头视角对根部关节进行三角测量来解决这一问题。随后，根据单个摄像头视角的输出初始化手部的朝向和姿态。HaMeR还会提供一个边界框，我们将其与2D投影的手部根部一起作为输入，送入Segment-Anything (SAM) [^46]模型中，从而得到每个静态摄像头图像中手部的标注分割掩码$M_{gt}$。
#### 3.2. 标注细化
根据初始手部姿态，我们通过以下优化，利用来自 C 个静态相机的输入，获得细化的手部姿态标注。我们使用 MANO [^34] [^73] 模型进行网格表示，其中包含 25 个 PCA 分量，并采用 DIB-R [^9] 可微分渲染器。这些标注包括手部姿态 $\theta$、手部平移 t、位于世界坐标中的顶点位移 $D_{\mathrm{vert}}$，以及以纹理图 $\mathcal{T}_P$ 形式表示的手部网格上的压力。所有静态相机均预先完成标定，使我们能够使用外参 $[R_{\mathrm{cam}}^i | t_{\mathrm{cam}}^i]$ 将手部网格投影到每个静态相机 i 的坐标系中。
$\beta$校准：对于MANO形状参数$\beta$，我们会为每位参与者的每只手分别设置独立的校准序列——
---
Figure 2. Method overview. The input for our annotation method consists of RGB-D images captured by 7 static Azure Kinect cameras and the pressure frame from a SenseI Morph touchpad. We leverage Segment-Anything [^46] and HaMeR [^67] to obtain initial hand poses and masks. We refine the initial hand pose and shape estimates through differentiable rasterization [^9] optimization across all static camera views. Using an additional virtual orthogonal camera placed below the touchpad, we reproject the captured pressure frame onto the hand mesh by optimizing the pressure as a texture feature of the corresponding UV map, while ensuring contact between the touchpad and all contact vertices.
在该序列中，参与者会缓慢转动手部，使手部能从所有摄像头中可见，且手指呈展开状态。针对这些序列，我们会在之前的优化过程中加入$\beta$正则化项来进一步优化MANO形状参数$l_{2}$。之后，给定参与者的所有其他序列都会复用这些形状参数，且在后续优化过程中$\beta$保持固定不变。
遵循HARP算法[^44]的设计思路，我们的标注算法分为两个阶段：（1）姿态优化；（2）形状细化，分别对应渲染目标$\mathcal{L}_{\mathcal{R}}$和几何目标$\mathcal{L}_{\mathcal{G}}$。
首先进入第一阶段——姿态优化，该阶段的重点是标注手部姿态$\boldsymbol{\theta}$和平移量$\mathbf{t}$。由此可直接从MANO模型$\boldsymbol{\Theta}$中得出手部网格[^34]，其表达式即为$\boldsymbol{\Theta} = \text{MANO}(\boldsymbol{\theta}, \boldsymbol{\beta}) + \mathbf{t}$。需要注意的是，手部的某些部位（如手指）可能无法从所有拍摄角度看到——例如弯曲手势中手掌遮挡的手指。为解决这一问题，我们引入了网格相交损失$\mathcal{L}_{\text{insec}}$ [^43] [^86]。此时的目标函数定义如下：
$$ \mathcal{L}_{\mathrm{p o s e}}(\Theta)=\mathcal{L}_{\mathcal{R}}(\Theta)+\mathcal{L}_{\mathrm{i n s e c}}(\Theta) $$
渲染目标$\mathcal{L}_{\mathcal{R}}$和网格相交损失$\mathcal{L}_{\mathrm{insec}}$的详细内容将在补充材料中说明。在形状细化阶段，手部的姿态$\pmb{\theta}$和平移量t保持不变。优化过程会产生顶点位移$D_{\mathrm{vert}}$：每个顶点会沿其法向量$\overrightarrow{n}$方向偏移一定距离，该法向量取自姿态优化阶段的最后一个迭代结果，目的是最小化渲染损失$\mathcal{L}_{\mathcal{R}}(\pmb{\Theta}^{*})$。由此得到的精细化手部网格$\Theta^{*}$可表示为$\pmb{\Theta}^{*} = \pmb{\Theta} + \overrightarrow{n} \cdot D_{\mathrm{vert}}$。为确保网格合理性，几何目标$\mathcal{L}_{\mathcal{G}}$也会被纳入优化范畴。此外，我们还引入了虚拟渲染$\tilde{\mathcal{R}}$来优化作为UV贴图的压力分布$\mathcal{T}_{\mathrm{P}}$，并最小化手部网格$\Theta^{*}$与触摸板表面接触区域之间的距离。该阶段的目标函数$\mathcal{L}_{\mathrm{shape}}$如下：
$$ \mathcal{L}_{\mathrm{s h a p e}}(\boldsymbol{\Theta}^{*})=\mathcal{L}_{\mathcal{R}}(\boldsymbol{\Theta}^{*})+\mathcal{L}_{\mathcal{G}}(\boldsymbol{\Theta}^{*})+\mathcal{L}_{\mathcal{V}_{\mathcal{R}}}(\boldsymbol{\Theta}^{*}) $$
虚拟渲染$\mathcal{R}$及其目标$\mathcal{L}_{\mathcal{R}}$将在下一节中解释，而几何目标$\mathcal{L}_{\mathcal{G}}$中的其他项则会在补充材料中详细说明。
##### 3.2.1. 用于接触和压力检测的虚拟渲染
如图2所示，我们还将采集到的压力数据作为手部网格纹理特征纳入优化过程，以用于所提出的虚拟渲染方法。为此，我们在触摸板下方放置一台虚拟正交相机$\tilde{\mathcal{R}}$，该相机在世界坐标系中的朝向为向上。渲染尺寸与触摸板的分辨率相匹配，且相机的平面与触摸板的感应表面重合。我们的目标是让手部网格上经过优化的压力UV贴图$\tilde{\mathcal{R}}_{P}(\Theta^{*}, \mathcal{T}_{P})$所映射出的渲染压力$\mathcal{T}_{P}$，能够与输入压力$P_{gt}$保持一致。
此外，我们利用简单的压力阈值从$P_{gt}$中推断出接触区域。以该接触区域作为掩码，我们确保被掩码后的渲染z轴深度$\tilde{\mathcal{R}}_D(\Theta^*)[z]$与相机到触摸板的距离$Z_{v2p}$相一致，从而保证物理上的接触。
---
虚拟渲染的目标函数$\mathcal{L}_{\tilde{\mathcal{G}}}(\boldsymbol{\Theta}^{*})$如下：
$$ \begin{aligned}\mathcal{L}_{\tilde{\mathcal{R}}}(\boldsymbol{\Theta}^{*})=&\quad\mathrm{M S E}(\tilde{\mathcal{R}}_{P}(\boldsymbol{\Theta}^{*},\mathcal{T}_{P}),\boldsymbol{P}_{\mathrm{gt}})\\&+\left|\mathbb{I}(\boldsymbol{P}_{\mathrm{gt}}>0)\odot(\tilde{\mathcal{R}}_{D}(\boldsymbol{\Theta}^{*})[z]-\boldsymbol{Z}_{v2p})\right|_{1}.\end{aligned} $$
### 4. EgoPressure数据集
EgoPressure包含430万个RGB-D帧（静态相机分辨率为2560×1440，头戴式相机分辨率为1920×1080），用于记录左右手与触压敏感平面表面的交互情况。该数据集涵盖了21名参与者执行的31种不同手势，包括触摸、拖动、捏合和按压等，每种手势均针对每只手进行记录。数据集总共包含5.0小时的手部手势视频，由七台校准过的静态相机和一台头戴式相机同步采集的RGB-D帧组成，同时还有以30帧/秒速率采集的来自触压敏感表面的真实压力图。数据采集装置采用了四种不同的表面纹理，其中还包括一面绿墙以便于合成背景增强。此外，基于我们提出标注方法（详见第3节），我们还提供了交互过程中手部的高精度姿态和网格数据，以及头戴式相机的跟踪姿态。通过EgoPressure，我们希望提供一个用于研究以触摸这一基本模态为基础的人与刚性表面交互过程中头部佩戴设备下的手部姿态和压力估计的重要数据集，从而推动机器对人类与周围环境交互的理解。
#### 4.1. 数据采集设置
为了从头部佩戴视角下获取手部姿态和压力的真实标签，我们搭建了一个数据采集装置，该装置集成了用于触摸和压力感测的触压敏感触摸板（Sensel Morph [^41]），以及七台静态相机和一台头戴式RGB-D相机（Azure Kinect [^1]）来采集RGB和深度图像（见图3）。这块尺寸为$240 \times 169.5$毫米的Sensel Morph触摸板被安装在云台上。针对不同参与者，我们在Sensel Morph触摸板上分别覆盖了四种印刷在纸上的纹理覆盖层：白色、绿色、深木色和浅木色。七台静态Azure Kinect相机被固定在铝制框架上，头戴式相机则安装在头盔上。该框架还承载着电脑显示器，周围设有绿幕。
所有相机和触摸板均连接到两台工作站（Intel Core i7-9700K，Nvidia GeForce RTX 3070）上，它们的时间戳通过搭载Raspberry Pi CM4的设备利用PTP协议进行同步，该设备还能以30帧/秒的速率同时触发所有Azure Kinect相机。
(a) Static camera-rig visualization
(c) Recording
Figure 3. 7 static + 1 egocentric camera rig
Figure 4. Camera pose tracking with IR makers
头戴式相机跟踪：为了获取头戴式相机的精确姿态，我们在Sense1 Morph触控板上以非对称布局放置了九个主动红外标记（见图4）。这些由Raspberry Pi CM4控制的标记，可通过简单的阈值处理在Azure Kinect的红外图像中被识别出来（即饱和红外相机的值范围）。这些标记会同时被点亮，从而能够通过透视N点算法计算出相机的姿态，进而准确评估相机与触控板之间的时间同步性。
#### 4.2. 参与者
我们从机构中招募了21名参与者（6名女性，15名男性，年龄23–32岁，平均年龄26岁），以确保样本具有广泛的代表性，能够覆盖手部特征的解剖学差异。参与者的身高范围为160–194厘米（平均值174厘米，标准差9），体重范围为51–95公斤（平均值69公斤，标准差14），中指长度范围为7.3–9.2厘米（平均值7.9厘米，标准差0.5）（MANO $\beta$值的分布情况见图5）。关于给参与者的说明、知情同意书以及IRB批准的相关细节，请参阅补充材料。
#### 4.3. 数据采集流程
参与者坐在设备前的可调节凳子上，头戴装有相机的头盔，相机朝向Sensel Morph触控板；每只手上还戴着黑色手套，直至手腕处。
---
Figure 5. (a) t-SNE [^87] visualization of hand pose frames $\theta$ over our dataset, with color coding for different gestures. All gestures are listed in Table 9 of the supplementary material. (b) Ratio of touch frames with contact for each vertex. (c) Maximum pressure over hand vertices across dataset. (d) Mean length of performed gestures. (e) Distribution of $\beta$ values across participants.
Figure 6. Thumbnail of different poses in egocentric views
在开始采集数据前，实验员会向参与者解释任务内容及研究目的，随后参与者需签署知情同意书并提供人口统计学信息。首先，参与者要在相机范围内缓慢转动双手，手指张开，完成校准动作。校准完成后，参与者需在Sensel Morph触控板上做出31种不同的手势，包括不同力度的触摸、按压和拖动手势（手势的具体描述见补充材料）。若手势仅涉及单次触摸动作（如按压食指），则每种手势重复5次；若涉及一系列连续触摸动作（如书写字母），则重复3次。每次做手势前，参与者会观看演示对应手势的视频，同时计算机屏幕上会显示书面说明。
(a) Annotation examples for a right hand
(b) Annotation examples for a left hand
Figure 7. Sample data from EgoPressure
位于他们前方。实验者在整个研究过程中对参与者予以指导，每位参与者参与研究的时间约为1小时。参与者在完成每个手势后可以休息，作为参与研究的感谢，他们会收到一条巧克力棒。总共我们记录了6216种不同的手势，即21名参与者$\times$双手$\times$（1次校准 + 27个$\times$5 + 4个$\times$3）手势。
#### 4.4. 数据统计
每个动作序列的平均时长为14秒，捕捉左右手动作的帧数几乎相等。图5展示了不同手势下的平均序列时长。所有帧中约有45.1%捕捉到手部与压感触控板接触的情况。
---
图5b展示了特定顶点接触表面时的帧数占比，图5c则显示了每个顶点测得的最大压力值。参照Grady等人的研究[^24]，我们将0.5 kPa设为最小有效压力阈值，以此排除触控板产生的弥散读数。
### 5. 基准评估
以往的研究仅利用RGB图像来估算施加压力图[^24] [^25]。借助EgoPressure，我们探索了将精确的手部姿态作为额外输入所带来的优势——这种输入能自然提供关于交互过程的更丰富上下文。我们提出了新的基准测试，用于利用RGB图像和3D手部姿态来估算手部压力。此外，我们还提出了一种新颖的网络架构，能够从单张RGB图像中同时估算作用在外部表面以及手部的压力，从而更深入地了解交互过程中涉及的手部区域。
#### 5.1. 图像投影式压力基线
我们在自己的数据集上评估基于RGB的基线模型PressureVision-Net [^24]。由于PressureVision数据集[^24]仅包含静态相机视角，我们将基线实验分为以自我为中心的视角和以物体为中心的视角。具体而言，我们使用数据集中的第2、3、4、5号相机视角，因为这些视角与PressureVision中的相机在触摸板上的朝向和距离相近。
我们验证了将手部姿态作为额外输入能够提升压力估算精度的假设。为此，我们对PressureVisionNet [^24]进行了简单的扩展。我们为原本专为RGB输入设计的编码器-解码器分割架构新增了一个通道，用于输入2.5D手部关键点信息。具体做法是将21个3D手部关节投影到图像平面上，并加入它们相对于以自我为中心的相机坐标系的深度（z坐标），且数值已换算为毫米单位。
在评估时，我们既使用了标注数据中的真实手部关节信息，也使用了HaMeR [^67]预测出的手部关节信息。HaMeR估算出的手部姿态可作为公平的基线，反映当前最先进的基于RGB的手部姿态估算器的性能；而真实关节信息则构成了性能上限，展示了采用更精确的手部姿态所能实现的潜在提升。
表2总结了相关结果。我们发现，添加2.5D手部关节层后，无论是以自我为中心的视角还是以物体为中心的视角，模型性能均有所提升。值得注意的是，手部姿态还能增强模型对未见过的相机视角的泛化能力。我们分别在第2、3、4、5号相机视角上训练模型，并在第1、6、7号视角上进行评估（如表2第三行所示）。图8展示的定性结果进一步证实了纳入手部姿态信息的益处。
Figure 8. Qualitative results. We present the egocentric experiment results in Subfigure (a). In Subfigure (b), both baseline models are trained using camera views 2, 3, 4, and 5. We display the results for one seen view and one unseen view. Additionally, we overlay the 2D keypoints predicted by HaMeR [^67] and our annotated ground truth on the input image. For better visualization, the contour of the touch sensing area is also highlighted as a reference.
Table 2. Pressure inference on the full dataset (21 participants) using different input modalities. Our high-fidelity hand pose annotations improve contact IoU [%], volumetric IoU [%], MAE [Pa], and temporal accuracy [%] compared to using no hand poses or HaMeR [^67] hand poses as additional input for novel exocentric and egocentric views.
| Model | Train | Eval | Modality | Cont. IoU \uparrow | Vol. IoU \uparrow | MAE \downarrow | Temp. Acc \uparrow |
| PressureVisionNet [^24] | Ego | Ego | RGB | 55.73 | 38.64 | 53.60 | 91.68 |
| [^24] w. [^67] pose | Ego | Ego | RGB & pred pose | 56.25 | 40.52 | 55.23 | 91.67 |
| [^24] w. GT pose | Ego | Ego | RGB & GT pose | 58.80 | 41.39 | 53.79 | 92.17 |
| PressureVisionNet [^24] | Exo (2,3,4,5) | Exo (2,3,4,5) | RGB | 62.11 | 44.73 | 43.15 | 93.61 |
| [^24] w. [^67] pose | Exo (2,3,4,5) | Exo (2,3,4,5) | RGB & pred pose | 62.95 | 45.01 | 42.53 | 93.83 |
| [^24] w. GT pose | Exo (2,3,4,5) | Exo (2,3,4,5) | RGB & GT pose | 64.39 | 47.58 | 41.72 | 94.18 |
| PressureVisionNet [^24] | Exo (2,3,4,5) | Exo (1,6,7) | RGB | 36.82 | 25.05 | 62.22 | 83.40 |
| [^24] w. [^67] pose | Exo (2,3,4,5) | Exo (1,6,7) | RGB & pred pose | 38.46 | 28.10 | 51.50 | 86.34 |
| [^24] w. GT pose | Exo (2,3,4,5) | Exo (1,6,7) | RGB & GT pose | 43.04 | 31.39 | 49.45 | 89.78 |
#### 5.2. 首个手部投影式压力基线
最初的PressureVision框架[^24]及其后续版本PressureVision++ [^25]均能在图像平面上预测2D手部压力。但这种方式会导致在3D空间中难以确定这种压力在手与物体之间的具体表现形式。
为解决这一问题，我们提出了新的基线模型PressureFormer，该模型能将压力估算为3D手部网格的UV图，从而既能将压力投影到手部表面形成3D压力，也能将其投影到图像空间形成2D压力。
如图9所示，我们的模型基于HaMeR [^67]构建。它处理相机坐标系下的手部顶点$V_{hand}$以及来自HaMeR的视觉Transformer（ViT）[^17]的图像特征令牌。基于Transformer的解码器会将$V_{hand}$作为多个输入令牌，同时与ViT输出的图像特征令牌进行交叉注意力计算。每个输出令牌代表对应位置的D维特征。
---
Figure 9. PressureFormer uses HaMeR's hand vertices and image feature tokens to estimate the pressure distribution over the UV map. We employ a differentiable renderer [^9] to project the pressure back onto the image plane by texture-mapping it onto the predicted hand mesh.
我们对应地选取网格顶点，然后利用MANO模型的UV坐标将其映射到UV特征图上[^73]。由于投影后UV特征图较为稀疏，我们应用两层卷积层进行神经插值，并将维度缩减到力类别数C，从而预测量化的UV压力图$U_{pred}$。
首先，我们计算粗糙的UV压力损失$\mathcal{L}_c$，该损失存在于$U_{pred}$与真实UV压力图$U_{gt}$之间，而真实UV压力图是由数据集中的标量UV压力$\mathcal{T}_P$转换而来的。随后，我们基于顶点$P_{pred}$构成的$M_{hand}$网格以及预测的$V_{hand}$ UV压力图，将压力$U_{pred}$重新渲染到原始图像平面上。借助可微渲染器[^9]，我们对面顶点的z法线和z轴进行反转，从而识别出距离相机视图最远且不可见的网格面，将其标记为潜在的接触位置。借此我们可以计算相对于真实压力$\mathcal{L}_p$的压力损失$P_{gt}$。对于$\mathcal{L}_p$和$\mathcal{L}_c$，我们均采用交叉熵损失，最终得到PressureFormer的如下损失函数：
$$ \mathcal{L}_{\mathcal{P F}}=w_{1}\mathcal{L}_{c}+w_{2}\mathcal{L}_{p} $$
我们使用所有相机视角下的以手为中心的图像裁剪图来训练PressureFormer模型及基线模型。训练期间，我们采用了包括平移、缩放和旋转在内的数据增强技术。结果汇总于表3中，可视化效果见图10。补充材料中还包含了更多分析内容。
Table 3. Our method achieves the highest performance in terms of contact IoU and performs comparably to other approaches on additional evaluation metrics. Notably, it offers significant advantages over the image-projected pressure baselines by directly predicting pressure on the UV map, enabling the reconstruction of 3D pressure through projection onto the estimated hand surface.
| Model | Contact IoU $\uparrow$ | Vol. IoU $\uparrow$ | MAE [kPa] $\downarrow$ | Temp. Acc. [%] $\uparrow$ |
| PressureVisionNet [^24] | 40.71 | 32.11 | 44 | 90 |
| [^24] (w. HaMeR [^67] pose) | 42.52 | 35.40 | 49 | 92 |
| PressureFormer | 43.04 | 31.57 | 71 | 89 |
Figure 10. Qualitative Results PressureFormer on our dataset. We compare our PressureFormer with both PressureVision [^24] and our extended baseline model with HaMeR-estimated [^67] 2.5D joint positions. Additionally, we provide visualizations of the hand mesh estimated by HaMeR, alongside the 3D pressure distribution on the hand surface derived from our predicted UV-pressure in the last two columns. Note that we transform the left-hand UV maps into the right-hand format.
### 6. 结论
在本文中，我们介绍了EgoPressure——一种新型的自我中心式手部压力数据集，它搭配了多视角手部姿态估计方法。EgoPressure包含带有网格的精确手部姿态、多视角RGB和深度图像、自我中心视角图像以及高质量的压力数据。我们建立了新的基准测试，并证明了在压力估计中使用手部姿态数据的有效性。在未来的工作中，我们计划通过加入物体来丰富数据集，从而实现更复杂几何形状上的压力估计。总之，我们认为EgoPressure代表着朝着让机器更深入理解自我中心视角下的手-物交互迈出的重要一步。
---
## EgoPressure：用于自我中心视觉中的手部压力与姿态估计的数据集
补充材料
### 7. 关于基准测试评估的详情
在本节中，我们将进一步介绍第5节中基准测试评估实验的相关细节。
7.1. 图像投影压力基线模型的详情
7.1.1. 包含额外2.5D关键点的基线模型
Figure 11. Overview of the image-projected pressure baseline with additional hand pose input. The baseline receives an RGB image and a 2.5D keypoint depth map as inputs to an encoder-decoder segmentation network for pressure estimation.
此前用于预测手部压力的方法[^24]仅以RGB图像作为输入。相比之下，我们的新基准测试旨在纳入手部姿态这一额外模态。为了确保基线模型与我们提出的方法之间能够进行公平比较，我们对现有方法进行了扩展，加入了额外的手部姿态输入。除了PressureVision的三个RGB通道外，我们还将2.5D深度图作为额外输入通道添加到分割网络中。
编码器-解码器分割网络架构。与PressureVision类似，我们采用在ImageNet上预训练的Squeeze-and-Excitation Network（SEResNeXt50）[^37] [^38]作为编码器，它可接收RGB图像和3D手部姿态输入；同时采用特征金字塔网络[^40] [^54]作为解码器，用于生成压力图。
训练过程。在所有实验中，我们使用15名参与者的数据用于训练和验证，而另外6名参与者的数据则被留作测试集。训练时，我们采用Adam优化器，批量大小为8。训练过程首先以0.001的学习率进行10万次迭代，随后以0.0001的学习率再进行50万次迭代。
#### 7.1.2. 评估指标
评估时，我们采用了PressureVision中提出的四项指标[^24]：接触交并比（IoU）、体积交并比、平均绝对误差（MAE）以及时间准确性。
接触交并比通过计算估计的压力图与真实二值化压力图之间的IoU来衡量接触表面预测的准确性。体积交并比则进一步纳入了预测压力幅值的准确性，其计算方法为：每个像素处估计压力图与真实压力图中较小压力值之和，与较大压力值之和的比值。MAE则以千帕斯卡（kPa）/像素为单位，量化压力预测误差。时间准确性通过验证估计值与真实值之间逐帧的接触一致性，来评估接触状态随时间的变化一致性。
##### 7.2. 额外的定性结果
基线模型的更多定性结果如图27所示。标注相关的更多定性示例如图30、31、32、33、34和35所示。
我们还展示了第三人称视角相机实验的定性结果（详见主论文中的表2）。图28和图29对比了我们使用RGB图像和2.5D手部关键点输入的模型，与仅使用RGB输入的PressureVisionNet[^24]的性能。图28展示了这两个模型在相机2、3、4、5拍摄的图像上的定性表现，且这两个模型均是在针对这些视角单独划分的训练集上训练的。在图28中，我们还在训练集未包含的相机1、6、7拍摄的新视角下对相同模型进行了评估。
在图28和图29的第二列中，重投影后的触摸感应区域以白色轮廓显示，用于验证相机姿态。我们还提供了每个样本的MAE和接触交并比数值。值得注意的是，加入额外的手部姿态信息能够提升模型估算压力和接触的能力，尤其是对于手部被遮挡的部分（参见图28中的示例04，以及图29中的示例09、11、13）。
##### 7.3. 对PressureFormer的额外评估
PressureFormer改进了第5.1节中提到的基线模型，它能在重建的手部网格的UV图上直接估算压力。这种方法将基于估算的3D手部姿态的压力表示拓展到了3D空间。虽然基于手部网格的压力表示仍可被投影到图像平面上。
---
为了与先前的方法进行基准测试[^24] [^25]，它提供了关于施加压力的具体手部区域的额外信息。这种能力在涉及复杂手-物体交互的场景中尤为有用，例如手指部分被遮挡或与非平面表面交互时，此时基于图像投影的压力图可能会存在局限性并引入额外的歧义。这些触觉手部动态对于实现人形机器人中的精准抓取和物体操控也具有重要意义。
###### 7.3.1. 估计的UV压力图的准确性
在第5.2节中，我们将PressureFormer与PressureVisionNet [^24]及其经过2.5D手部关键点增强的基线模型进行比较，这两种模型都能直接估计相机图像投影的压力图。我们依据PressureVision中设定的评估指标（见表4）来进行这些比较。
我们进一步扩展了这一评估，考虑了PressureFormer直接估计的、投影到手部网格上的压力，即UV压力图（见图9）。作为对比，我们将PressureVisionNet及其经过手部姿态增强的基线模型生成的基于图像的压力图，投影到利用同一图像通过HaMeR [^67]估计出的对应手部网格上。这一过程包括识别出离相机最远的手部网格面（即被遮挡的顶点），并将2D压力图栅格化到UV图上（见图12）。此外，我们还评估了未受显式UV损失监督训练的PressureFormer变体。
因此，我们提出了一种全新的基准测试任务，用于评估手部表面压力的准确性以及同时估计压力和手部网格的性能。
评估指标。为了评估手部表面压力估计的准确性，我们在UV压力图上计算两个指标：接触IoU和体积IoU。
训练。模型使用所有相机视角的图像进行训练和评估。我们选取15名参与者用于训练和验证，其中6名参与者被留作测试集。预处理阶段，图像会围绕手部进行裁剪并调整大小以匹配网络的输入维度。评估时，我们确保裁剪过程中手部始终位于画面中央。所有方法均应用了包括平移、缩放和旋转在内的数据增强技术。训练采用Adam优化器，批量大小为8，前10万次迭代的学习率为0.001，随后的50万次迭代学习率则降至0.0001。PressureFormer的损失函数（见公式4）使用了权重参数$w_1 = 0.2$和$w_2 = 0.05$。
结果。第5.2节及基于UV图的评估结果汇总于表4中。在UV压力图上，PressureFormer在接触IoU和体积IoU方面的表现均优于所有基于图像投影的压力基线模型。
Figure 12. Pipeline for projecting the image-based pressure map (from PressureVision) onto the UV map: Starting with the predicted hand mesh and 2D pressure map, the normals and z-axis are inverted to identify occluded (invisible) faces of the mesh. The pressure is then mapped onto the UV space using rasterization.
Input GT Pressure GT UV GT on HandMesh [^67]Pred. Pres. Pred. UV On Hand PressureFormer PressureFormer w/o $\mathcal{L}_{C}$
Figure 13. Qualitative examples demonstrating the impact of coarse UV loss supervision $\mathcal{L}_{c}$. The coarse UV loss supervision $\mathcal{L}_{c}$ prevents the prediction of pressure in areas of the UV map that are not rendered on the image plane (see Section 5.2). These regions typically correspond to faces oriented toward the camera, where pressure and contact are not physically possible.
它在基于图像投影的压力图上也取得了最高的接触IoU值。直接预测相机图像上压力的、经过手部姿态增强的基线模型，在基于图像的压力图上实现了最佳的体积IoU值。这些结果凸显了将手部姿态信息纳入压力估计的重要性，同时也强调了进一步研究手部姿态与压力联合估计以构建更连贯的交互模型的潜力。
此外，结果还证明了粗糙的UV压力损失在提升UV图上压力预测准确性方面的作用（见图13）。
图14对三种基线方法所估算的UV压力图进行了定性比较。
###### 7.3.2. PressureFormer的泛化能力
采用UV压力图能够提升手部和压力预测在更复杂物体上的泛化能力。与仅关注手部表面交互的图像平面压力估算方式不同，UV压力映射能够通过直接预先……来突出以手为中心的压力分布。
---
Table 4. Performance comparison of our PressureFormer model against image-projected pressure baselines, evaluated using temporal accuracy [%], image-based pressure metrics (Image Contact IoU, Image Vol. IoU, Image MAE [kPa]), and UV map-based pressure metrics (UV Pressure IoU, UV Pressure Vol. IoU). PressureFormer demonstrates superior performance in UV pressure IoU and UV Pressure Vol. IoU, while also achieving higher scores in image-based Contact IoU. By directly predicting pressure on the UV map, PressureFormer offers advantages, enabling accurate 3D pressure reconstruction by projecting the results onto the estimated hand surface.
| Model | Im. Contact IoU $\uparrow$ | Im. Vol. IoU $\uparrow$ | Im. MAE $\downarrow$ | UV Press. Contact IoU $\uparrow$ | UV Press. Vol. IoU $\uparrow$ | Temp. Acc. $\uparrow$ |
| PressureVisionNet [^24] | 40.71 | 32.11 | 44 | 21.53 | 16.41 | 90 |
| [^24] (w. HaMeR [^67]) pose) | 42.52 | 35.40 | 49 | 24.10 | 17.36 | 92 |
| PressureFormer (Ours) | 43.04 | 31.57 | 71 | 33.12 | 24.54 | 89 |
| PressureFormer w/o $\mathcal{L}_{c}$ | 41.27 | 29.57 | 74 | 26.24 | 18.61 | 88 |
Figure 14. Qualitative comparison of UV Pressure. We compare our PressureFormer model against the original PressureVision [^24] and its extended version with additional hand keypoint inputs. For both PressureVision-based approaches, the UV pressure is obtained by baking the image-based pressure predictions onto the UV map of the hand mesh, using the hand mesh estimates provided by HaMeR [^67].
预测手部顶点上的压力。
我们的模型PressureFormer以预训练的HaMeR [^67]作为骨干网络，从视觉Transformer的标记中提取手部顶点和图像特征。这使得我们的方法能够有效处理各种手部姿态，同时整合编码在视觉Transformer标记中的以手为中心的图像纹理信息。图15所示的定性结果证明了PressureFormer对未见过的物体和环境的泛化能力。
#### 8. 标注方法的细节与评估
##### 8.1. 优化目标
在本节中，我们将描述与主论文中所述目标相结合以实现完整实施所需的优化目标。
###### 8.1.1. 渲染目标
由于手部网格$\Theta$是所有相机视角下唯一被渲染的物体，我们利用Segment-Anything (SAM) $M_{\mathrm{gt}}^{i}$生成的伪真实掩码[^46]，从输入的RGB图像$I_{\mathrm{gt}}^{i}=I_{\mathrm{in}}^{i}\otimes M_{\mathrm{gt}}^{i}$和深度图$D_{\mathrm{gt}}^{i}=D_{\mathrm{in}}^{i}\otimes M_{\mathrm{gt}}^{i}$中提取相关区域、外观特征$I_{\mathrm{in}}^{i}$以及深度信息$D_{\mathrm{in}}^{i}$。为了优化渲染后的外观$\mathcal{R}_{F}^{i}(\boldsymbol{\Theta},\mathcal{T})$，在一个包含连续多帧的输入批次中，所有相机视角会共享同一纹理，以此确保网格$\Theta$在不同相机和连续帧间保持一致。所有C个相机的渲染损失$\mathcal{L}_{\mathcal{R}}$由公式5表示。
深度体积IoU $\mathcal{L}_D(\Theta)$ [^24]定义在公式5的第三项中。我们将其应用于真实深度图和渲染深度图。表5展示了这两种损失：深度……
---
Figure 15. Qualitative evaluation of PressureFormer on diverse, real-world examples featuring various objects and scenes. Despite being trained exclusively on EgoPressure, the model recognizes pressure regions during corresponding contact events, demonstrating its potential for generalization.
体积IoU损失$\mathcal{L}_D(\Theta)$和掩码IoU损失$\mathcal{L}_M(\Theta)$是针对初始输入中的网格，即$\theta_{ini}$和$t_{ini}$，以及两个连续的标注阶段——姿态优化和形状精细化而言的。
###### 8.1.2. 几何目标
几何目标$(\mathcal{L}_{\mathcal{G}})$由几项组成：
$$ \mathcal{L}_{\mathcal{G}}=\mathcal{L}_{\mathrm{i n s e c}}+\mathcal{L}_{\mathrm{a r a p}}+\mathcal{L}_{\vec{\mathbf{n}}}+\mathcal{L}_{\mathrm{l a p}}+\mathcal{L}_{\mathrm{o f f s e t}} $$
其中，$\mathcal{L}_{\mathrm{insec}}$表示网格相交损失，该损失利用BVH树来识别网格内的自相交情况。
Table 5. Losses by Stages. We validate the quality of hand poses using two metrics, Depth Volumetric IoU Loss $\mathcal{L}_D$ (Eq. 5) and Mask IoU Loss $\mathcal{L}_M$ (Eq. 5), computed on $386,231 \times 7$ (static cameras) = 2,703,617 annotated frames. Of these, 2,192,633 (81%) show the hand in contact with the touchpad. We report the results before (initial) and after each consecutive optimization step: POSE OPTIMIZATION and SHAPE REFINEMENT.
| Category | $\mathcal{L}_{D}\downarrow$ | $\mathcal{L}_{M}\downarrow$ |
| Initial | Pose. | Pose. + Shape. | Initial | Pose. | Pose. + Shape. |
| Overall | 0.4443 | 0.1759 | 0.1317 | 0.3887 | 0.1165 | 0.0558 |
| With Contact | 0.4444 | 0.1752 | 0.1309 | 0.3891 | 0.1167 | 0.0562 |
| Without Contact | 0.4441 | 0.1790 | 0.1351 | 0.3871 | 0.1158 | 0.0545 |
根据这些检测结果，会相应地施加惩罚[^43] [^86]。
项$\mathcal{L}_{\text{arap}}$为尽可能保持刚性的损失，如[^79]中所介绍，该损失旨在提升3D网格的刚性，同时将长度变化分散到多条边上。边长的变化是相对于姿态优化最后一个阶段的网格来确定的。
$$ \mathcal{L}_{\mathrm{a r a p}}=\frac{1}{|E|}\sum_{v^{*}\in\Theta^{*}}\sum_{e^{*}\in E(v^{*},u^{*})}|\|e^{*}\|-\|e^{p}\|, $$
此处，$E(v^{*}, u^{*})$是边集E中连接顶点$v^{*}$和$u^{*}$的边，而边$e^{p}$则由网格中对应的顶点$v^{p}$和$u^{p}$构成，且不存在顶点位移$D_{vert}$。
网格顶点 $V_{\Theta^*}$ 通过拉普拉斯网格正则化 $\mathcal{L}_{\mathrm{lap}}$ [^16] 进行平滑处理，而法向一致性正则化 $\mathcal{L}_{\vec{\mathbf{n}}}$ 则用于平滑位移后的网格上的法向。最后，顶点偏移项 $\mathcal{L}_{\mathrm{offset}}$ 由 $\|D_{\mathrm{vert}}\|^2$ 计算得出。
###### 8.1.3. 深度剔除
在某些序列中，从某些相机视角看，手部可能会被Sense Morph触摸板部分遮挡，这可能会阻碍整体渲染掩码优化过程的收敛。为解决这一问题，我们对触摸板及其底座进行了建模。我们预先生成深度图 $D_o$ 来表示这些场景障碍物。随后，我们利用渲染出的深度图 $\mathcal{R}_D$ 进行简单的深度剔除，生成剔除掩码 $M_{dc} = \mathbb{I}(D_o > \mathcal{R}_D)$。借此我们可以在渲染深度图 $\mathcal{R}_D$、外观图 $\mathcal{R}_F$ 以及掩码 $\mathcal{R}_M$ 上创建裁剪区域，这些区域共同代表了位于场景障碍物前方的手部部件。初步测试后我们发现，这种深度剔除方式能促使手部网格与触摸板的交界处达到更低的掩码IoU损失 $\mathcal{L}_M$。因此，我们将触摸板的碰撞盒加入网格相交损失 $\mathcal{L}_{insec}$ 中，以惩罚这种相交情况。图16展示了相关示例。
---
$$ \mathcal{L}_{\mathcal{R}}(\boldsymbol{\Theta})=\sum_{i=0}^{C}[\lambda_{M}(\underbrace{1-\mathrm{I o U}(\mathcal{R}_{M}^{i}(\boldsymbol{\Theta}),M_{\mathrm{g t}}^{i})}_{\mathrm{M a s k I o U L o s s}\mathcal{L}_{M}(\boldsymbol{\Theta})})+\lambda_{A}\underbrace{\mathrm{M S E}(\mathcal{R}_{F}^{i}(\boldsymbol{\Theta},\mathcal{T}),\boldsymbol{I}_{\mathrm{g t}}^{i})}_{\mathrm{A p p e a r a n c e L o s s}\mathcal{L}_{A}(\boldsymbol{\Theta})}]+\lambda_{D}\underbrace{(1-\frac{|\operatorname*{m i n}(\mathcal{R}_{D}^{i}(\boldsymbol{\Theta}),D_{\mathrm{g t}}^{i})|_{1}}{|\operatorname*{m a x}(\mathcal{R}_{D}^{i}(\boldsymbol{\Theta}),D_{\mathrm{g t}}^{i})|_{1}})}_{D e n t h V o l u m e t r i c I o U L o s s}\mathcal{L}_{D}(\boldsymbol{\Theta}) $$
Figure 16. Depth Culling. (a) In the view of Camera 7, the thumb is behind the touchpad. (b) We compare the rendered depth of hand $\mathcal{R}_D$ and pre-rendered depth map of scene obstacles $D_o$, and (c) cutout the part which has a larger depth value than $D_o$. The thumb rendered in blue color is cutout due to the depth culling. (d) The collision box is rendered in 3D.
###### 8.1.4. 时间连续性
为确保标注的手部姿态在时间戳之间具有时间连续性，我们的优化过程会批量处理由7帧RGB-D图像和1帧压力数据组成的连续采集序列，批量大小为B。我们对从MANO网格回归得到的手部关节位置J的近似二阶导数施加正则化约束。时间连续性正则化表达式如下：
$$ \mathcal{L}_{\mathrm{t e m p}}=\frac{1}{B-2}\sum_{i=1}^{B-2}\|\mathbf{J}_{i+2}-2\mathbf{J}_{i+1}+\mathbf{J}_{i}\|_{2}. $$
##### 8.2. 标注保真度评估
###### 8.2.1. 手动标注与检查
为验证我们标注方法所得手部姿态的质量，我们手动标注了300组随机选取的7个静态视角和1个自中心视角的图像（共 $300\times8=2400$ 帧）。我们标注了所有相机视角下可见的指甲尖端点，共得到7176个2D点。随后对这些2D指甲尖端点进行三角测量以获得3D点。在将重投影误差阈值设为2像素以排除不一致的手动标注后，我们最终得到1114个至少在两个相机视角中可见的3D点。表6中报告了我们标注方法所得手尖位置相对于基于手动标注的3D手尖位置的距离误差。我们还附上了该方法的消融研究结果。手动标注与我们方法的效果对比图展示在图17中。
###### 8.2.2. 与基于学习的模型对比
与最先进的3D手部姿态估计器HaMeR相比，我们基于优化的方法具有显著优势，能够为我们的数据集生成高质量的标注。如图19所示，尽管从俯视图看手部姿态Table 6. 我们的标注方法与基于手动标注的三维指尖位置三角测量结果的定量评估。我们对不同的损失项进行了消融研究，包括外观损失$\mathcal{L}_{A}$（公式5）、深度体素IoU损失$\mathcal{L}_{D}$（公式5）、网格相交损失$\mathcal{L}_{insec}$（第8.1.2节）、尽可能保持刚性的损失$\mathcal{L}_{arap}$（第8.1.2节）、拉普拉斯平滑度$\mathcal{L}_{lap}$（第8.1.2节）、法向一致性正则化$\mathcal{L}_{\bar{u}}$（第8.1.2节）以及顶点偏移正则化$\mathcal{L}_{offset}$（第8.1.2节）。结果表明，每个损失项都对优化性能有所贡献。
| Losses | Ours | w/o $\mathcal{L}_{A}$ | w/o $\mathcal{L}_{D}$ | w/o $\mathcal{L}_{insert}$ | w/o $\mathcal{L}_{wrap}$ | w/o $\mathcal{L}_{lup}$ | w/o $\mathcal{L}_{fi}$ | w/o $\mathcal{L}_{offset}$ |
| $\mathcal{L}_{D} \downarrow$ | 0.1251 | 0.1404 | 0.1797 | 0.1368 | 0.1347 | 0.1396 | 0.1349 | 0.1477 |
| $\mathcal{L}_{M} \downarrow$ | 0.0488 | 0.0662 | 0.0724 | 0.0619 | 0.0597 | 0.0654 | 0.0653 | 0.0728 |
| 3D tips error [mm] $\downarrow$ | 5.68 | 7.66 | 8.45 | 7.99 | 8.61 | 8.49 | 8.39 | 8.28 |
手动标注 + 三角测量
Figure 17. Manual Verification Examples. We demonstrate our annotation is accurate compared to the manual annotations. (above) We re-project the triangulated nail tips. We only triangulated them when they are visible in at least 2 views. (bottom) We re-project our 3D annotations which also show invisible nail tips as well.
来自HaMeR [^67] 的结果从俯视图看似乎合理，但侧视图会暴露出不准确性和尺度模糊问题。相比之下，我们的标注方法在所有相机视角下都能产生稳健且一致的结果。在表2中，我们证明：使用我们高质量3D手部姿态的基线模型，在手部压力估计方面的表现优于使用HaMeR的[^67] 预测结果。
为了进一步评估标注质量，我们在表7中提供了静态视角下预测的指甲尖三角测量结果与手动标注的对比验证结果。此外，图20展示了定性比较。
---
结合了来自HaMeR [^67] 的额外姿态以及我们的真实标注的压力估计结果。这些结果从定量和定性两方面都凸显了我们优化方法所生成的高保真手部姿态标注的重要性，同时也强调了未来研究中推进手部姿态与压力图估计的必要性。
最后，我们在表8和图18中报告了在我们数据集上微调后的HaMeR方法的结果。尽管微调提升了性能，但仍有进一步改进的空间。这些结果为解决以自我为中心的视角下手部与物体交互时的3D手部姿态估计问题奠定了坚实基础。
Table 7. Hand pose verification. Triangulation is performed on the nail tips using HaMeR [^67] predictions across all static cameras, compared against manual annotations.
| 3D tips error [mm] | Std. |
| Ours | 5.68 | 4.9 |
| HaMeR [^67] | 12.37 | 6.3 |
Table 8. Fine-tuning results of HaMeR [^67] on EgoPressure demonstrate improved hand pose accuracy, underscoring the value of our dataset for 3D hand pose estimation.
| MPJPE [mm] | Reconstruction Error [mm] |
| Finetuned HaMeR [^67] | 10.75 | 6.10 |
| HaMeR [^67] | 18.58 | 8.11 |
#### 9. 关于数据集的详细说明
##### 9.1. 手势描述详情
表9列出了数据收集期间参与者执行的所有手势，包括使用的手部以及每种手势的重复次数。相关视觉示例可参见附带的视频。
##### 9.2. 数据集对比
表10对我们的提议数据集进行了全面对比。在现有专注于接触或手-物体姿态估计的公共数据集中，EgoPressure是首个将自我中心视角下的手部-物体交互视频数据、真实的接触与压力信息，以及高保真手部姿态和网格相结合的数据集。
##### 9.3. 主动式红外标记物详情
我们使用主动式红外标记物，其工作原理与被动式标记物类似，这类标记物会自身发射红外光，因此体积更小且更精确——在过滤后的红外图像中通常表现为微小的光点。这减少了镜头畸变对跟踪精度的影响。此外，这些标记物是可编程的，能够对其激活和停用进行精准控制，这一点
Input Our Annotation Fine-tuned Pretrained HaMeR [^67]
Figure 18. Hand pose prediction and ground truth pose visualization for each camera. We fine-tune HaMeR [^67] on our dataset, demonstrating improved detail in hand pose estimation, particularly in scenarios where the hand interacts with a surface.
这对我们系统内的同步至关重要。我们利用大光束角的红外LED（见图23）作为主动式红外标记物。
采用非对称布局的标记物，从标记物排列上方的上半球任意视角都能被唯一识别。这种独特的配置能够借助滤波后的红外图像实现稳健且精确的实时跟踪，其中标记物表现为半径为数个像素的光点。该过程在算法1中给出的伪代码里有详细说明。图24进一步展示了这种布局在辅助精确标记物识别与姿态估计方面的有效性，图中描绘了标记物的空间排列情况。此外，该流程也可推广到其他非对称布局中。
透视n点（PnP）算法用于根据红外帧中识别出的标记物来计算以自我为中心的相机的姿态。实验中，由识别良好的标记物计算得出的姿态的重投影误差平均值低于0.4像素。为确保识别的清晰性与可靠性，我们设定了1像素的阈值来过滤掉可能包含以下内容的帧：
---
Figure 19. Comparison of the estimated hand mesh from HaMeR [^67] and our annotation method in both egocentric and exocentric views. While the projected hand mesh from HaMeR appears visually plausible from an egocentric perspective, observable differences in hand articulation and mesh deformations become apparent from the exocentric viewpoint of the static cameras.
Figure 20. Qualitative results of the image-projected baselines on egocentric views, incorporating additional hand pose inputs using our annotations and predictions from HaMeR [^73]. We also reproject the area of the touchpad (indicated by white lines) to verify the egocentric camera pose.
---
Figure 21. Qualitative comparison of reprojected nail tips from our annotation method (center) and triangulation of HaMeR [^67] predictions (right). The left column displays the reprojection of triangulated manually annotated visible tips.
Figure 22. Marker visibility in infrared frame of head-mounted egocentric camera
Table 9. List of gestures performed by a participant during the data collection.
| Gesture | Left Hand | Right Hand | Number of Repetitions |
| i. | calibration routine | ✓ | ✓ | - |
| ii. | draw word | ✓ | ✓ | 3 |
| iii. | grasp edge curled thumb-down | ✓ | ✓ | 5 |
| iv. | grasp edge curled thumb-up | ✓ | ✓ | 5 |
| v. | grasp edge uncurled thumb-down | ✓ | ✓ | 5 |
| vi. | index press high force | ✓ | ✓ | 5 |
| vii. | index press low force | ✓ | ✓ | 5 |
| viii. | index press no-contact | ✓ | ✓ | 5 |
| ix. | index press pull | ✓ | ✓ | 5 |
| x. | index press push | ✓ | ✓ | 5 |
| xi. | index press rotate left | ✓ | ✓ | 5 |
| xii. | index press rotate right | ✓ | ✓ | 5 |
| xiii. | pinch thumb-down high force | ✓ | ✓ | 5 |
| xiv. | pinch thumb-down low force | ✓ | ✓ | 5 |
| xv. | pinch thumb-down no-contact | ✓ | ✓ | 5 |
| xvii. | pinch zoom | ✓ | ✓ | 5 |
| xviii. | press cupped onebyone high force | ✓ | ✓ | 3 |
| xvii. | press cupped onebyone low force | ✓ | ✓ | 3 |
| xix. | press fingers high force | ✓ | ✓ | 5 |
| xx. | press fingers low force | ✓ | ✓ | 5 |
| xxii. | press fingers no-contact | ✓ | ✓ | 5 |
| xxiii. | press flat onebyone high force | ✓ | ✓ | 3 |
| xxiii. | press flat onebyone low force | ✓ | ✓ | 3 |
| xxiv. | press palm high force | ✓ | ✓ | 5 |
| xxv. | press palm low force | ✓ | ✓ | 5 |
| xxvi. | press palm no-contact | ✓ | ✓ | 5 |
| xxvii. | press palm-and-fingers high force | ✓ | ✓ | 5 |
| xxviii. | press palm-and-fingers low force | ✓ | ✓ | 5 |
| xxix. | press palm-and-fingers no-contact | ✓ | ✓ | 5 |
| xxx. | pull towards | ✓ | ✓ | 5 |
| xxxi. | push away | ✓ | ✓ | 5 |
| xxxii. | touch iPad | ✓ | ✓ | 3 |
录制过程中标记物识别的歧义情况。此外，对于丢失跟踪的帧，会采用球面线性插值（Slerp）来估算相机姿态，从而保持跟踪数据的连续性与准确性。
Figure 23. Relative Radiant Intensity vs. Angular Displacement The marker enable a good visible radiant intensity of beam angle till 150 degree, which ensures good visibility in egocentric infrared camera
##### 9.4. 数据集采集中设备同步的相关细节
Sense Morph设备无缓冲机制，在120帧/秒的速率下能保持稳定的8毫秒延迟，而Azure Kinect摄像头则以30帧/秒的速度运行，可捕捉高分辨率的RGB图像和深度图。这得益于其出色的录制性能。
---
Table 10. Comparison between EgoPressure and extended list of hand-contact datasets.
| Dataset | frames | participants | hand pose | hand mesh | markerless | real | egocentric | multiview | RGB | depth | contact | pressure surface | hand |
| EgoPressure (ours) | 4.3M | 21 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | Pressure sensor | ✓ | ✓ |
| ContactLabelDB [^25] | 2.9M | 51 | ✗ | ✗ | ✓ | ✓ | ✗ | ✓ | ✓ | ✗ | Pressure sensor | ✓ | ✗ |
| PressureVisionDB [^24] | 3.0M | 36 | ✗ | ✗ | ✓ | ✓ | ✗ | ✓ | ✓ | ✗ | Pressure sensor | ✓ | ✗ |
| ContactPose [^4] | 3.0M | 50 | ✓ | ✓ | ✓ | ✓ | ✗ | ✓ | ✓ | ✓ | Thermal imprint | ✗ | ✗ |
| GRAB [^84] | 1.6M | 10 | ✓ | ✓ | ✗ | ✓ | ✗ | ✗ | ✗ | ✗ | Inferred from Pose | ✗ | ✗ |
| ARCTIC [^20] | 2.1M | 10 | ✓ | ✓ | ✗ | ✓ | ✓ | ✓ | ✓ | ✓ | Inferred from Pose | ✗ | ✗ |
| $H_{2}O$ [^47] | 571k | 4 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | Inferred from Pose | ✗ | ✗ |
| OakInk [^92] | 230k | 12 | ✓ | ✓ | ✓ | ✓ | ✗ | ✓ | ✓ | ✓ | Inferred from Pose | ✗ | ✗ |
| OakInk-2 [^94] | 4.01M | 9 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | Inferred from Pose | ✗ | ✗ |
| DexYCB [^7] | 582k | 10 | ✓ | ✓ | ✓ | ✓ | ✗ | ✓ | ✓ | ✓ | Inferred from Pose | ✗ | ✗ |
| $HO-3D$ [^30] | 103k | 10 | ✓ | ✓ | ✓ | ✓ | ✗ | ✓ | ✓ | ✓ | Inferred from Pose | ✗ | ✗ |
| TACO [^56] | 5.2M | 14 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | Inferred from Pose | ✗ | ✗ |
| Affordpose [^42] | 26.7k | - | ✓ | ✓ | ✓ | ✗ | ✗ | ✓ | ✗ | ✗ | Inferred from Pose | ✗ | ✗ |
| AssemblyHands [^66] | 3.03M | 34 | ✓ | ✗ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | Inferred from Pose | ✗ | ✗ |
| ContactArt [^96] | 332k | - | ✓ | ✓ | ✓ | ✗ | ✗ | ✓ | ✓ | ✓ | Simulated Pose | ✗ | ✗ |
| $HOI4D$ [^55] | 2.4M | 9 | ✓ | ✓ | ✓ | ✓ | ✓ | ✗ | ✓ | ✓ | Inferred from Pose | ✗ | ✗ |
| YCBAfford [^14] | 133k | - | ✓ | ✓ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | Simulated Pose | ✗ | ✗ |
| ObMan [^33] | 154k | - | ✓ | ✓ | ✓ | ✗ | ✗ | ✗ | ✓ | ✓ | Simulated Pose | ✗ | ✗ |
| FPHAB [^21] | 100k | 6 | ✓ | ✗ | ✗ | ✓ | ✓ | ✗ | ✓ | ✓ | ✗ | ✗ | ✗ |
| $HA-ViD$ [^95] | 1.5M | 30 | ✗ | ✗ | ✓ | ✓ | ✗ | ✓ | ✓ | ✓ | ✗ | ✗ | ✗ |
| Ego4d [^26] | 3670 hours | 923 | ✗ | ✗ | ✓ | ✓ | ✓ | ✗ | ✓ | ✗ | ✗ | ✗ | ✗ |
| EPIC-KITCHEN-100 [^15] | 20M | 37 | ✗ | ✗ | ✓ | ✓ | ✓ | ✗ | ✓ | ✗ | ✗ | ✗ | ✗ |
| Ego-Exo4D [^27] | 1422 hours | 740 | ✓ | ✗ | ✓ | ✓ | ✓ | ✓ | ✓ | ✗ | ✗ | ✗ | ✗ |
Figure 24. Layout of the Active Markers. The indices of the markers are aligned with the pseudocode provided in Algorithm 1. Starting from the asymmetrical anchor marker M, all markers can be identified by computing their relative distances and considering their spatial relationships.
由于Azure Kinect的性能特点，帧最初会被存储在设备的缓存中，因此无法依靠帧到达主机时的操作系统时间戳来与SenseI Morph的压力数据进行同步。
所有摄像头都可以通过树莓派CM4发出的30赫兹触发信号进行外部同步，以确保帧捕获同步。不过，由于设备自身的问题，录制开始时会出现1–3帧的帧丢失情况。由于设备时钟戳的绝对值没有实际意义，因此无法确定第一帧接收前到底丢失了多少帧。若依靠设备时钟戳差异来进行同步，可能会导致摄像头之间出现1–3帧的对齐偏差。
为解决这一问题，我们利用了主动式红外标记物的可编程特性，以及两台主机与树莓派CM4之间精度高达1毫秒的精确操作系统时间戳同步功能，这些功能由精密时间协议（PTP）提供支持。
```pseudo
// Algorithm 1: Identify Marker
1: procedure IDENTIFYMARKERS(filtered IR image)
2: Extract marker coordinates $(u, v)$ from the filtered IR image
3: Compute all pairwise distances among markers
4: Identify the pair with the smallest distance, initially labeled as 0 and M
5: Compute the vector from M to 0
6: Count the number of markers on each side of the vector line M - 0
7: if more markers lie on the right of the vector then
8: Confirm start point as M, endpoint as 0
9: else
10: Swap, set start point as 0 and endpoint as M
11: end if
12: Identify 2 and 4 as markers aligned with M - 0, on the same side relative to M
13: Check distances from M to 2 and 4 to determine which is closer
14: Identify L as the marker closest to the line extending through $(0, M, 2, 4)$ and on the same side as 0
15: Compute the centroid of all markers
16: Draw a line from 0 through the centroid
17: Identify 3 as the marker isolated on its side of the centroid line
18: Identify 5 as the closest marker to the line (0 - centroid) not already labeled
19: Determine 1 and R by their proximity to line $(2 - 5)$, with 1 being closer
20: end procedure
```
树莓派CM4配备了基础电子元件（见图22），它会在下一个曝光周期开始时发出信号，提供一个可靠的同步点，用以补偿最初丢失的帧。标记物激活时的精确操作系统时间戳会被清晰记录下来（见图26）。
通过从标记物首次出现的帧开始，根据设备时钟戳的偏移量来计算所有帧的实际操作系统时间戳，从而实现精确的同步。这种方法能有效对齐RGBD图像与压力数据，优化多模态传感器系统的数据整合。此外，这种利用外部主动光学标识符的同步机制还具备其他优势。
---
高效且经济，因此可推广到其他多传感器系统，例如依赖不同操作系统时间戳源、配备外部头戴式摄像头的动作捕捉系统。
Figure 25. Basic Electrical Elements Implementation, We use a D-type flip-flop and N-channel MOSFET to ensure the IR marker will be activated by the next beginning of exposure after receiving signal from PIN 23. And PIN 14 will monitor the activation to obtain its timestamp
Figure 26. Synchronization Diagram We set head-mounted egocentric camera to align with 30 Hz triggering signal emitted by the Raspberry Pi CM4, this signal will also go to PIN 18 as clock frequency of D-type Flip-flop Fig. 25). Then exposure $t_{exp}$ of all cameras is same. The other static cameras 1 to 7 will have a delay $\Delta t$ to trigger signal to avoid interference of infrared light. The marker will be activated at $t_0$ (around 300 milliseconds after start recording), which we know its global OS timestamp, then it will be visible to all camera at next exposure cycle. As verification, we deactivate marker by the very end of recording at the timestamp $t_1$, then the marker will be invisible for all cameras in the next frame capture. The good synchronization will have equal frame number between $t_0$ and $t_1$ for all cameras.
#### 10. 局限性
尽管EgoPressure为从自我中心视角理解压力提供了基础性研究，但仍有若干挑战尚未解决。这些挑战可分为三大类。
首先，在与普通物体交互时测量压力存在困难。目前我们的数据采集仅局限于对平面上的压力进行感知。虽然我们乐观地认为未来研究会拓展到更多种类的物体，但在任意表面上感知压力仍面临巨大挑战，因为这需要对用户的手部进行大量仪器安装，不仅会妨碍自然交互，还会在采集的数据中引入可见的伪影。为物体安装压力传感装置仍是一个持续的研究领域，近期进展主要集中在基础接触检测方面 [^4]。不过，我们预计随着这些挑战的解决，我们的标注方法会自然扩展到更复杂的物体和交互场景。PressureVision++ [^25] 利用弱标签来推断更复杂物体上的压力，但它仅考虑指尖交互，且由于获取精确压力的难度，其对压力回归的评估仍局限于平面。我们在图15中展示了PressureFormer在更多种类物体上的定性评估结果。
其次，当前数据集仅在室内环境中采集。我们的数据采集装置专为获取高保真度的手-表面交互标注而优化。为了增加背景环境的多样性以提升在真实场景中的泛化能力，我们在数据采集设备的背景以及压力垫上添加了绿色覆盖层。这便于进行背景替换，且已被证明能有效提升商用野外手部追踪性能 [^32] [^97]。
最后，当前设置仅考虑了单手交互。将双手使用的场景纳入研究将是本工作的自然延伸。
未来研究若能进一步解决这些挑战，就能改善真实场景中的压力估计效果，并拓宽其应用场景。
#### 11. 伦理考量
人类活动数据的记录与使用涉及重要的伦理问题。EgoPressure项目已获得苏黎世联邦理工学院伦理委员会的批准，提案编号为EK 2023-N-228。该批准涵盖数据采集及数据集的公开发布。所有参与者均书面明确同意录制他们的会话、创建数据集并予以发布（详见附带的知情同意书）。所有人口统计信息（如性别、年龄、体重和身高）以及传感器和视频数据均经过匿名化处理，为每位参与者分配一个数字代码。个人数据（性别、年龄、体重和身高）与传感器及视频数据分开存储，仅限参与研究的主要研究人员访问。我们并未采集或存储参与者的面部图像。
---
| | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 |
| MAE $\downarrow$ | Press.Vis. [^24] | 129.6 | 8.2 | 14.3 | 25.3 | 6.7 | 61.9 | 9.3 | 8.3 | 23.9 | 39.1 | 37.1 |
| [^24] w. GT poses | 116.4 | 5.3 | 11.8 | 17.3 | 4.8 | 62.2 | 5.1 | 6.2 | 13.5 | 30.6 | 35.6 |
| Contact IoU $\uparrow$ | Press.Vis. [^24] | 46.9 | 68.4 | 57.4 | 39.0 | 84.0 | 63.8 | 0.0 | 76.9 | 35.1 | 64.1 | 56.8 |
| [^24] w. GT poses | 55.2 | 79.9 | 63.9 | 67.8 | 86.8 | 65.6 | 58.7 | 82.6 | 73.2 | 72.1 | 65.4 |
| | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 |
| MAE $\downarrow$ | Press.Vis. [^24] | 6.2 | 50.1 | 39.4 | 54.7 | 11.7 | 10.7 | 32.0 | 37.0 | 67.4 | 21.5 | 8.29 |
| [^24] w. GT Keypoints | 4.7 | 51.7 | 35.6 | 30.4 | 10.0 | 10.3 | 24.9 | 32.6 | 35.5 | 14.7 | 4.89 |
| Contact IoU $\uparrow$ | Press.Vis. [^24] | 86.2 | 38.4 | 54.2 | 24.2 | 79.9 | 76.6 | 17.6 | 1.2 | 54.1 | 0.0 | 0.0 |
| [^24] w. GT Keypoints | 86.3 | 48.4 | 61.7 | 43.8 | 83.1 | 79.6 | 30.3 | 35.2 | 76.3 | 42.2 | 43.6 |
Figure 27. Qualitative comparison of pressure maps inferred using PressureVisionNet [^24] and our trained model with additional hand poses as input on representative cases across various gestures. The bottom table presents MAE [Pa] and Contact IoU [%] for pressure maps inferred using PressureVisionNet [^1] and our trained model on selected samples shown in the Figure.
---
Figure 28. Comparison of pressure maps estimated by PressureVisionNet [^24] and our adapted model, using separate training and validation sets, both consisting of images from camera views 2, 3, 4, and 5.
---
Figure 29. Comparison of pressure maps estimated by PressureVisionNet [^24] and our adapted model, evaluated using input images from cameras 1, 6, and 7. The models are the same as in Figure 28, which are trained on images from camera views 2, 3, 4, and 5.
---
(a) Camera 1
(b) Camera 2
(e) Camera 5
(c) Camera 3
(d) Camera 4
(f) Camera 6
(g) Camera 7
(h) Egocentric Camera
(i) Mesh
(j) Pressure texture
(k) Pressure on hand
Figure 30. Example of Annotation 1 Right hand with gesture: grasp edge with uncurled thumb down
(a) Camera 1
(b) Camera 2
(c) Camera 3
(d) Camera 4
(e) Camera 5
(f) Camera 6
(g) Camera 7
(h) Egocentric Camera
(i) Mesh
(j) Pressure texture
(k) Pressure on hand
Figure 31. Example of Annotation 2 Left hand with gesture: index press with high force
---
(a) Camera 1
(b) Camera 2
(e) Camera 5
(c) Camera 3
(d) Camera 4
(f) Camera 6
(g) Camera 7
(h) Egocentric Camera
(i) Mesh
(j) Pressure texture
(k) Pressure on hand
Figure 32. Example of Annotation 3 Left hand with gesture: pinch thumb down on the edge with high force
(a) Camera 1
(b) Camera 2
(e) Camera 5
(c) Camera 3
(d) Camera 4
(f) Camera 6
(g) Camera 7
(h) Egocentric Camera
(i) Mesh
(j) Pressure texture
(k) Pressure on hand
Figure 33. Example of Annotation 4 Right hand with gesture: grasp edge with curled thumb up
---
(a) Camera 1
(b) Camera 2
(c) Camera 3
(e) Camera 5
(d) Camera 4
(f) Camera 6
(g) Camera 7
(h) Egocentric Camera
(i) Mesh
(j) Pressure texture
(k) Pressure on hand
Figure 34. Example of Annotation 5 Right hand with gesture: pinch finger zoom in and out
(a) Camera 1
(b) Camera 2
(e) Camera 5
(c) Camera 3
(d) Camera 4
(f) Camera 6
(g) Camera 7
(h) Egocentric Camera
(i) Mesh
(i) Pressure texture
(k) Pressure on hand
Figure 35. Example of Annotation 6 Left hand with gesture: pull all finger towards participant
---
### 参考文献
[^1]: Azure Kinect. Azure kinect dk hardware specifications, 2019. 5
[^2]: Raunaq Bhirangi, Tess Hellebrekers, Carmel Majidi, and Abhinav Gupta. Reskin: versatile, replaceable, lasting tactile skins. In 5th Annual Conference on Robot Learning, 2021. 3
[^3]: Giulia Boato, Nicola Conci, Mattia Daldoss, Francesco GB De Natale, and Nicola Piotto. Hand tracking and trajectory analysis for physical rehabilitation. In 2009 IEEE International Workshop on Multimedia Signal Processing, pages 1–6. IEEE, 2009. 2
[^4]: Samarth Brahmbhatt, Chengcheng Tang, Christopher D Twigg, Charles C Kemp, and James Hays. Contactpose: A dataset of grasps with object contact and hand pose. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XIII 16, pages 361–378. Springer, 2020. 3, 9, 10
[^5]: Gereon H Büscher, Risto Kōiva, Carsten Schürmann, Robert Haschke, and Helge J Ritter. Flexible and stretchable fabric-based tactile sensor. Robotics and Autonomous Systems, 63:244–252, 2015. 3
[^6]: Jongeun Cha, Seung-man Kim, Ian Oakley, Jeha Ryu, and Kwan H. Lee. Haptic interaction with depth video media. In Advances in Multimedia Information Processing - PCM 2005, pages 420–430, Berlin, Heidelberg, 2005. Springer Berlin Heidelberg. 2
[^7]: Yu-Wei Chao, Wei Yang, Yu Xiang, Pavlo Molchanov, Ankur Handa, Jonathan Tremblay, Yashraj S Narang, Karl Van Wyk, Umar Iqbal, Stan Birchfield, et al. Dexycb: A benchmark for capturing hand grasping of objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9044–9053, 2021. 2, 3, 9
[^8]: Nutan Chen, Göran Westling, Benoni B Edin, and Patrick van der Smagt. Estimating fingertip forces, torques, and local curvatures from fingernail images. Robotica, 38(7):1242–1262, 2020. 3
[^9]: Wenzheng Chen, Jun Gao, Huan Ling, Edward Smith, Jaakko Lehtinen, Alec Jacobson, and Sanja Fidler. Learning to predict 3d objects with an interpolation-based differentiable renderer. In Advances In Neural Information Processing Systems, 2019. 3, 4, 8
[^10]: Yi Fei Cheng, Tiffany Luong, Andreas Rene Fender, Paul Streli, and Christian Holz. Comfortable user interfaces: Surfaces reduce input error, time, and exertion for tabletop and mid-air user interfaces. In 2022 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). IEEE, 2022. 1, 2
[^11]: Sammy Christen, Muhammed Kocabas, Emre Aksan, Jemin Hwangbo, Jie Song, and Otmar Hilliges. D-grasp: Physically plausible dynamic grasp synthesis for hand-object interactions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20577–20586, 2022. 1, 3
[^12]: Sammy Christen, Lan Feng, Wei Yang, Yu-Wei Chao, Otmar Hilliges, and Jie Song. Synh2r: Synthesizing hand-object motions for learning human-to-robot handovers. arXiv preprint arXiv:2311.05599, 2023. 2
[^13]: Jeremy A Collins, Cody Houff, Patrick Grady, and Charles C Kemp. Visual contact pressure estimation for grippers in the wild. In 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 10947–10954. IEEE, 2023. 1
[^14]: Enric Corona, Albert Pumarola, Guillem Alenya, Francesc Moreno-Noguer, and Grégory Rogez. Ganhand: Predicting human grasp affordances in multi-object scenes. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5031–5041, 2020. 3, 9
[^15]: Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Evangelos Kazakos, Jian Ma, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. Rescaling egocentric vision: Collection, pipeline and challenges for epic-kitchens-100. International Journal of Computer Vision, pages 1–23, 2022. 9
[^16]: Mathieu Desbrun, Mark Meyer, Peter Schröder, and Alan H Barr. Implicit fairing of irregular meshes using diffusion and curvature flow. In Proceedings of the 26th annual conference on Computer graphics and interactive techniques, pages 317–324, 1999. 4
[^17]: Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth $16 \times 16$ words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.7
[^18]: Kiana Ehsani, Shubham Tulsiani, Saurabh Gupta, Ali Farhadi, and Abhinav Gupta. Use the force, luke! learning to predict physical forces by simulating effects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 224–233, 2020. 3
[^19]: Neil Xu Fan and Robert Xiao. Reducing the latency of touch tracking on ad-hoc surfaces. Proc. ACM Hum.-Comput. Interact., 6(ISS), 2022. 2
[^20]: Zicong Fan, Omid Taheri, Dimitrios Tzionas, Muhammed Kocabas, Manuel Kaufmann, Michael J Black, and Otmar Hilliges. Arctic: A dataset for dexterous bimanual hand-object manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12943–12954, 2023. 2, 3, 9
[^21]: Guillermo Garcia-Hernando, Shanxin Yuan, Seungryul Baek, and Tae-Kyun Kim. First-person hand action benchmark with rgb-d videos and 3d hand pose annotations. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 409–419, 2018. 2, 9
[^22]: Jun Gong, Aakar Gupta, and Hrvoje Benko. Acustico: Surface tap detection and localization using wrist-based acoustic tdoa sensing. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology, page 406–419, New York, NY, USA, 2020. Association for Computing Machinery. 2
## [^23]: Patrick Grady, Chengcheng Tang, Christopher D Twigg, Minh Vo, Samarth Brahmbhatt, and Charles C Kemp. Contactopt: Optimizing contact to improve grasps. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1471–1481, 2021. 2
---
[^24]: Patrick Grady, Chengcheng Tang, Samarth Brahmbhatt, Christopher D Twigg, Chengde Wan, James Hays, and Charles C Kemp. Pressurevision: Estimating hand pressure from a single rgb image. In European Conference on Computer Vision, pages 328–345. Springer, 2022. 2, 3, 7, 8, 1, 9, 11, 12, 13
[^25]: Patrick Grady, Jeremy A Collins, Chengcheng Tang, Christopher D Twigg, Kunal Aneja, James Hays, and Charles C Kemp. Pressurevision++: Estimating fingertip pressure from diverse rgb images. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 8698–8708, 2024. 1, 2, 3, 7, 9, 10
[^26]: Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18995–19012, 2022. 2, 9
[^27]: Kristen Grauman, Andrew Westbury, Lorenzo Torresani, Kris Kitani, Jitendra Malik, Triantafyllos Afouras, Kumar Ashutosh, Vijay Baiyya, Siddhant Bansal, Bikram Boote, et al. Ego-exo4d: Understanding skilled human activity from first-and third-person perspectives. arXiv preprint arXiv:2311.18259, 2023. 2, 9
[^28]: Yizheng Gu, Chun Yu, Zhipeng Li, Weiqi Li, Shuchang Xu, Xiaoying Wei, and Yuanchun Shi. Accurate and low-latency sensing of touch contact on any surface with finger-worn imu sensor. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology, page 1059–1070, New York, NY, USA, 2019. Association for Computing Machinery. 2
[^29]: Sean Gustafson, Christian Holz, and Patrick Baudisch. Imaginary phone: Learning imaginary interfaces by transferring spatial memory from a familiar device. In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology, page 283–292, New York, NY, USA, 2011. Association for Computing Machinery. 2
[^30]: Shreyas Hampali, Mahdi Rad, Markus Oberweger, and Vincent Lepetit. Honnotate: A method for 3d annotation of hand and object poses. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3196–3206, 2020. 2, 3, 9
[^31]: Shangchen Han, Beibei Liu, Randi Cabezas, Christopher D Twigg, Peizhao Zhang, Jeff Petkau, Tsz-Ho Yu, Chun-Jung Tai, Muzaffer Akbay, Zheng Wang, et al. Megatrack: monochrome egocentric articulated hand-tracking for virtual reality. ACM Transactions on Graphics (ToG), 39(4):87–1, 2020. 2
[^32]: Shangchen Han, Po-chen Wu, Yubo Zhang, Beibei Liu, Linguang Zhang, Zheng Wang, Weiguang Si, Peizhao Zhang, Yujun Cai, Tomas Hodan, et al. Umetrack: Unified multiview end-to-end hand tracking for vr. In SIGGRAPH Asia 2022 Conference Papers, pages 1–9, 2022. 2, 10
[^33]: Yana Hasson, Gul Varol, Dimitrios Tzionas, Igor Kalevatykh, Michael J Black, Ivan Laptev, and Cordelia Schmid. Learning joint reconstruction of hands and manipulated objects. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11807–11816, 2019. 3,9
[^34]: Yana Hasson, Gül Varol, Dimitris Tzionas, Igor Kalevatykh, Michael J. Black, Ivan Laptev, and Cordelia Schmid. Learning joint reconstruction of hands and manipulated objects. In CVPR, 2019. 3,4
[^35]: Jonas Hein, Matthias Seibold, Federica Bogo, Mazda Farshad, Marc Pollefeys, Philipp Fürnstahl, and Nassir Navab. Towards markerless surgical tool and hand pose estimation. International journal of computer assisted radiology and surgery, 16:799–808, 2021. 2
[^36]: Steven Henderson and Steven Feiner. Opportunistic tangible user interfaces for augmented reality. IEEE Transactions on Visualization and Computer Graphics, 16(1):4–16, 2010. 1
[^37]: Jie Hu, Li Shen, and Gang Sun. Squeeze-and-excitation networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7132–7141, 2018. 1
[^38]: Jie Hu, Li Shen, Samuel Albanie, Gang Sun, and Enhua Wu. Squeeze-and-excitation networks, 2019. 1
[^39]: Wonjun Hwang and Soo-Chul Lim. Inferring interaction force from visual information without using physical force sensors. Sensors, 17(11):2455, 2017. 3
[^40]: Pavel Iakubovskii. Segmentation models pytorch. . pytorch, 2019. 1
[^41]: Sensel Inc. Sensel morph., 2024. 3, 5
[^42]: Juntao Jian, Xiuping Liu, Manyi Li, Ruizhen Hu, and Jian Liu. Affordpose: A large-scale dataset of hand-object interactions with affordance-driven hand pose. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14713–14724, 2023. 9
[^43]: Tero Karras. Maximizing parallelism in the construction of bvhs, octrees, and k-d trees. In Proceedings of the Fourth ACM SIGGRAPH/Eurographics Conference on High-Performance Graphics, pages 33–37. Eurographics Association, 2012. 4
[^44]: Korrawe Karunratanakul, Sergey Prokudin, Otmar Hilliges, and Siyu Tang. HARP: Personalized Hand Reconstruction from a Monocular RGB Video. 2023. 4
[^45]: Hong-Ki Kim, Seunggun Lee, and Kwang-Seok Yun. Capacitive tactile sensor array for touch screen application. Sensors and Actuators A: Physical, 165(1):2–7, 2011. 3
[^46]: Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything. arXiv:2304.02643, 2023. 3, 4
[^47]: Taein Kwon, Bugra Tekin, Jan Stühmer, Federica Bogo, and Marc Pollefeys. H2o: Two hands manipulating objects for first person interaction recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10138–10148, 2021. 2, 3, 9
[^48]: Mike Lambeta, Tingfan Wu, Ali Sengul, Victoria Rose Most, Nolan Black, Kevin Sawyer, Romeo Mercado, Haozhi Qi, Alexander Sohn, Byron Taylor, et al. Digitizing touch with an artificial multimodal fingertip. arXiv preprint arXiv:2411.02479, 2024. 1, 2
## [^49]: Minkyung Lee, Woontack Woo, et al. Arkb: 3d vision-based augmented reality keyboard. In ICAT, 2003. 2
---
[^50]: Shuang Li, Jiaxi Jiang, Philipp Ruppel, Hongzhuo Liang, Xiaojian Ma, Norman Hendrich, Fuchun Sun, and Jianwei Zhang. A mobile robot hand-arm teleoperation system by vision and imu. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 10900–10906. IEEE, 2020. 2
[^51]: Zongmian Li, Jiri Sedlar, Justin Carpentier, Ivan Laptev, Nicolas Mansard, and Josef Sivic. Estimating 3d motion and forces of person-object interactions from monocular video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8640–8649, 2019. 3
[^52]: Chen Liang, Xutong Wang, Zisu Li, Chi Hsia, Mingming Fan, Chun Yu, and Yuanchun Shi. Shadowtouch: Enabling free-form touch-based hand-to-surface interaction with wrist-mounted illuminant by shadow projection. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, pages 1–14, 2023. 2
[^53]: PPS UK Limited. Tactileglove - hand pressure and force measurement., 2023. 3
[^54]: Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2117–2125, 2017. 1
[^55]: Yunze Liu, Yun Liu, Che Jiang, Kangbo Lyu, Weikang Wan, Hao Shen, Boqiang Liang, Zhoujie Fu, He Wang, and Li Yi. Hoi4d: A 4d egocentric dataset for category-level human-object interaction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21013–21022, 2022. 9
[^56]: Yun Liu, Haolin Yang, Xu Si, Ling Liu, Zipeng Li, Yuxiang Zhang, Yebin Liu, and Li Yi. Taco: Benchmarking generalizable bimanual tool-action-object understanding. arXiv preprint arXiv:2401.08399, 2024. 3, 9
[^57]: Gabriel Lugo, Mario Ibarra-Manzano, Fang Ba, and Irene Cheng. Virtual reality and hand tracking system as a medical tool to evaluate patients with parkinson's. In Proceedings of the 11th EAI International Conference on Pervasive Computing Technologies for Healthcare, pages 405–408, 2017.
[^58]: Yiyue Luo, Yunzhu Li, Pratyusha Sharma, Wan Shou, Kui Wu, Michael Foshey, Beichen Li, Tomás Palacios, Antonio Torralba, and Wojciech Matusik. Learning human–environment interactions using conformal tactile textiles. Nature Electronics, 4(3):193–201, 2021. 1, 3
[^59]: Yiyue Luo, Chao Liu, Young Joong Lee, Joseph DelPreto, Kui Wu, Michael Foshey, Daniela Rus, Tomás Palacios, Yunzhu Li, Antonio Torralba, et al. Adaptive tactile interaction transfer via digitally embroidered smart gloves. Nature communications, 15(1):868, 2024.
[^60]: Priyanka Mandikal and Kristen Grauman. Learning dexterous grasping with object-centric visual affordances. In 2021 IEEE international conference on robotics and automation (ICRA), pages 6169–6176. IEEE, 2021. 1, 3
[^61]: Stephen A Mascaro and H Harry Asada. Measurement of finger posture and three-axis fingertip touch force using fingernail sensors. IEEE Transactions on Robotics and Automation, 20(1):26–35, 2004. 3
[^62]: Manuel Meier, Paul Streli, Andreas Fender, and Christian Holz. Rapid touch interaction in virtual reality using wearable sensing. In 2021 IEEE Virtual Reality and 3D User Interfaces (VR), pages 519–528. IEEE, 2021. 1, 2
[^63]: Vimal Mollyn and Chris Harrison. Egotouch: On-body touch input using ar/vr headset cameras. In Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology, pages 1–11, 2024. 3
[^64]: Gyeongsik Moon, Shoou-I Yu, He Wen, Takaaki Shiratori, and Kyoung Mu Lee. Interhand2.6m: A dataset and baseline for 3d interacting hand pose estimation from a single rgb image. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XX 16, pages 548–564. Springer, 2020. 2
[^65]: Franziska Mueller, Dushyant Mehta, Oleksandr Sotnychenko, Srinath Sridhar, Dan Casas, and Christian Theobalt. Real-time hand tracking under occlusion from an egocentric rgb-d sensor. In Proceedings of the IEEE International Conference on Computer Vision, pages 1154–1163, 2017. 2
[^66]: Takehiko Ohkawa, Kun He, Fadime Sener, Tomas Hodan, Luan Tran, and Cem Keskin. Assemblyhands: Towards egocentric activity understanding via 3d hand pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12999–13008, 2023. 9
[^67]: Georgios Pavlakos, Dandan Shan, Ilija Radosavovic, Angjoo Kanazawa, David Fouhey, and Jitendra Malik. Reconstructing hands in 3D with transformers. In CVPR, 2024. 2, 3, 4, 7, 8, 5, 6
[^68]: Tu-Hoa Pham, Nikolaos Kyriazis, Antonis A Argyros, and Abderrahmane Kheddar. Hand-object contact force estimation from markerless visual tracking. IEEE transactions on pattern analysis and machine intelligence, 40(12):2883–2896, 2017. 3
[^69]: Philip Quinn, Wenxin Feng, and Shumin Zhai. Deep touch: Sensing press gestures from touch image sequences. Artificial Intelligence for Human Computer Interaction: A Modern Approach, pages 169–192, 2021. 3
[^70]: James M Rehg and Takeo Kanade. Digiteyes: Vision-based hand tracking for human-computer interaction. In Proceedings of 1994 IEEE workshop on motion of non-rigid and articulated objects, pages 16–22. IEEE, 1994. 2
[^71]: Mark Richardson, Matt Durasoff, and Robert Wang. Decoding surface touch typing from hand-tracking. In Proceedings of the 33rd annual ACM symposium on user interface software and technology, pages 686–696, 2020. 2
[^72]: Mark Richardson, Fadi Botros, Yangyang Shi, Pinhao Guo, Bradford J Snow, Linguang Zhang, Jingming Dong, Keith Vertanen, Shugao Ma, and Robert Wang. Stegotype: Surface typing from egocentric cameras. In Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology, pages 1–14, 2024. 2
[^73]: Javier Romero, Dimitrios Tzionas, and Michael J. Black. Embodied hands: Modeling and capturing hands and bodies together. ACM Transactions on Graphics, (Proc. SIGGRAPH Asia), 2017. 2, 3, 8
## [^74]: Elliot N. Saba, Eric C. Larson, and Shwetak N. Patel. Dante vision: In-air and touch gesture sensing for natural surface
---
结合深度相机和热成像相机的交互。载于2012年IEEE新兴信号处理应用国际会议论文集，第167–170页，2012年。2
[^75]: Pressure Mapping Sensors. Tekscan., 2024. 3
[^76]: Vivian Shen, James Spann, and Chris Harrison. Farout touch: Extending the range of ad hoc touch sensing with depth cameras. In Proceedings of the 2021 ACM Symposium on Spatial User Interaction, New York, NY, USA, 2021. Association for Computing Machinery. 2
[^77]: Yilei Shi, Haimo Zhang, Jiashuo Cao, and Suranga Nanayakkara. Versatouch: A versatile plug-and-play system that enables touch interactions on everyday passive surfaces. In Proceedings of the Augmented Humans International Conference, New York, NY, USA, 2020. Association for Computing Machinery. 2
[^78]: Yilei Shi, Haimo Zhang, Kaixing Zhao, Jiashuo Cao, Meng-meng Sun, and Suranga Nanayakkara. Ready, steady, touch! sensing physical contact with a finger-mounted imu. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., 4(2), 2020. 2
[^79]: Olga Sorkine and Marc Alexa. As-rigid-as-possible surface modeling. In Proceedings of EUROGRAPHICS/ACM SIGGRAPH Symposium on Geometry Processing, pages 109–116, 2007. 4
[^80]: Paul Streli, Jiaxi Jiang, Andreas Fender, Manuel Meier, Hugo Romat, and Christian Holz. Taptype: Ten-finger text entry on everyday surfaces via bayesian inference. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, New York, NY, USA, 2022. Association for Computing Machinery. 2
[^81]: Paul Streli, Jiaxi Jiang, Juliete Rossie, and Christian Holz. Structured light speckle: Joint ego-centric depth estimation and low-latency contact detection via remote vibrometry. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, pages 1–12, 2023. 2
[^82]: Paul Streli, Mark Richardson, Fadi Botros, Shugao Ma, Robert Wang, and Christian Holz. Touchinsight: Uncertainty-aware rapid touch and text input for mixed reality from egocentric vision. In Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology, pages 1–16, 2024. 1, 2
[^83]: Subramanian Sundaram, Petr Kellnhofer, Yunzhu Li, Jun-Yan Zhu, Antonio Torralba, and Wojciech Matusik. Learning the signatures of the human grasp using a scalable tactile glove. Nature, 569(7758):698–702, 2019. 3
[^84]: Omid Taheri, Nima Ghorbani, Michael J Black, and Dimitrios Tzionas. Grab: A dataset of whole-body human grasping of objects. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part IV 16, pages 581–600. Springer, 2020. 2, 3, 9
[^85]: Ryo Takahashi, Masaaki Fukumoto, Changyo Han, Takuya Sasatani, Yoshiaki Narusue, and Yoshihiro Kawahara. Telemetring: A batteryless and wireless ring-shaped keyboard using passive inductive telemetry. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology, page 1161–1168, New York, NY, USA, 2020. Association for Computing Machinery. 2
[^86]: Dimitrios Tzionas, Luca Ballan, Abhilash Srikantha, Pablo Aponte, Marc Pollefeys, and Juergen Gall. Capturing hands in action using discriminative salient points and physics simulation. International Journal of Computer Vision (IJCV), 118(2):172–193, 2016. 4
[^87]: Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008. 6
[^88]: Andrew D Wilson. Playanywhere: a compact interactive tabletop projection-vision system. In Proceedings of the 18th annual ACM symposium on User interface software and technology, pages 83–92, 2005. 1, 2
[^89]: Andrew D. Wilson. Using a depth camera as a touch sensor. In ACM International Conference on Interactive Tabletops and Surfaces, page 69–72, New York, NY, USA, 2010. Association for Computing Machinery. 2
[^90]: Robert Xiao, Scott Hudson, and Chris Harrison. Direct: Making touch tracking on ordinary surfaces practical with hybrid depth-infrared sensing. In Proceedings of the 2016 ACM International Conference on Interactive Surfaces and Spaces, pages 85–94, 2016.
[^91]: Robert Xiao, Julia Schwarz, Nick Throm, Andrew D. Wilson, and Hrvoje Benko. Mrtouch: Adding touch input to head-mounted mixed reality. IEEE Transactions on Visualization and Computer Graphics, 24(4):1653–1660, 2018. 1,2
[^92]: Lixin Yang, Kailin Li, Xinyu Zhan, Fei Wu, Anran Xu, Liu Liu, and Cewu Lu. Oakink: A large-scale knowledge repository for understanding hand-object interaction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20953–20962, 2022. 2, 3, 9
[^93]: Shanxin Yuan, Qi Ye, Bjorn Stenger, Siddhant Jain, and Tae-Kyun Kim. Bighand2.2m benchmark: Hand pose dataset and state of the art analysis. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4866–4874, 2017. 2
[^94]: Xinyu Zhan, Lixin Yang, Yifei Zhao, Kangrui Mao, Hanlin Xu, Zenan Lin, Kailin Li, and Cewu Lu. Oakink2: A dataset of bimanual hands-object manipulation in complex task completion. arXiv preprint arXiv:2403.19417, 2024. 2, 3, 9
[^95]: Hao Zheng, Regina Lee, and Yuqian Lu. Ha-vid: A human assembly video dataset for comprehensive assembly knowledge understanding. Advances in Neural Information Processing Systems, 36, 2024. 9
[^96]: Zehao Zhu, Jiashun Wang, Yuzhe Qin, Deqing Sun, Varun Jampani, and Xiaolong Wang. Contactart: Learning 3d interaction priors for category-level articulated object and hand poses estimation. arXiv preprint arXiv:2305.01618, 2023. 3, 9
[^97]: Christian Zimmermann, Duygu Ceylan, Jimei Yang, Bryan Russell, Max Argus, and Thomas Brox. Freihand: A dataset for markerless capture of hand pose and shape from single rgb images. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 813–822, 2019. 2, 10
## [^98]: Lara Zlokapa, Yiyue Luo, Jie Xu, Michael Foshey, Kui Wu, Pulkit Agrawal, and Wojciech Matusik. An integrated design pipeline for tactile sensing robotic manipulators. In 2022 In-
---
国际机器人学与自动化会议（ICRA），第3136–3142页。IEEE，2022年。2