# HO-3D_v3：提升HO-3D数据集中手-物体标注的准确性

Shreyas Hampali $^{(1)}$, Sayan Deb Sarkar $^{(1)}$, Vincent Lepetit $^{(2,1)}$

$^{(1)}$奥地利格拉茨技术大学计算机图形与视觉研究所，格拉茨

(2) 法国巴黎东大学，巴黎高科桥路学院，巴黎

{<firstname>.<lastname>}@icg.tugraz.at

项目页面：<https://www.tugraz.at/index.php?id=40231>

## 1 HO-3D的‘v3’版本有哪些新内容？

HO-3D是一个数据集，提供了各种手-物体交互场景的图像序列，并标注了手和物体的3D姿态，它最初以HO-3D_v2版本的形式在[^1]中被介绍。这些标注是通过原始论文中提出的优化方法自动获得的。

HO-3D_v3为手和物体姿态提供了更精确的标注，从而能更好地估计手与物体之间的接触区域。新的标注采用了[^1]中介绍的优化技术的改进版本，具体细节见下文第2节。表1展示了HO-3D_v2和HO-3D_v3两个版本数据集的统计信息。

基于50帧人工标注的数据，表2显示了HO-3D_v3标注准确性的提升情况。图1展示了整个数据集中手表面与物体之间的接触区域，以及平均穿透距离。与HO-3D_v2相比，HO-3D_v3的标注显然产生了更多的接触点。

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'>Train set #images</td><td style='text-align: center; word-wrap: break-word;'>Test Set #images</td><td style='text-align: center; word-wrap: break-word;'>#objects</td><td style='text-align: center; word-wrap: break-word;'>#subjects</td><td style='text-align: center; word-wrap: break-word;'>Release date</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>HO-3D_v2</td><td style='text-align: center; word-wrap: break-word;'>66,034</td><td style='text-align: center; word-wrap: break-word;'>11,524</td><td style='text-align: center; word-wrap: break-word;'>10</td><td style='text-align: center; word-wrap: break-word;'>10</td><td style='text-align: center; word-wrap: break-word;'>9 Jan 2020</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>HO-3D_v3</td><td style='text-align: center; word-wrap: break-word;'>83,325</td><td style='text-align: center; word-wrap: break-word;'>20,137</td><td style='text-align: center; word-wrap: break-word;'>10</td><td style='text-align: center; word-wrap: break-word;'>10</td><td style='text-align: center; word-wrap: break-word;'>1 Jul 2021</td></tr></table>

<div style="text-align: center;">Table 1: Statistics of the HO-3D_v2 and HO-3D_v3 datasets.</div>

## 2 算法改进

为了提高姿态估计的准确性，我们对[^1]中提出的方法做了以下三项改动。

---

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'>mean (mm)</td><td style='text-align: center; word-wrap: break-word;'>std (mm)</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>HO-3D_v2</td><td style='text-align: center; word-wrap: break-word;'>12.31</td><td style='text-align: center; word-wrap: break-word;'>3.8</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>HO-3D_v3</td><td style='text-align: center; word-wrap: break-word;'>8.12</td><td style='text-align: center; word-wrap: break-word;'>2.2</td></tr></table>

<div style="text-align: center;">Table 2: Accuracy of our estimations with respect to manual annotations of fingertips using the point cloud built from all camera depth maps. We manually annotated 50 frames with all 5 fingertip locations. The frames were specifically chosen from parts of the sequence where HO-3D_v2 provides poor estimates.</div>

![Image](images/a39dce4d39a5d38f76e9b5f68f2d748b9480927f3d72242d47ee43d7e6bfb6b4.jpg)

<div style="text-align: center;">Figure 1: Contact maps for the HO-3D_v2 and HO-3D_v3 versions of the datasets for dynamic and static grasps. ‘Red’ regions denote complete contact and ‘Blue’ regions denote no contact. The mean penetration between hand and object are provided in the inset. The ‘v3’ dataset shows more contact regions than the ‘v2’ dataset with significant improvement in the case of dynamic grasps.</div>

### 2.1 基于交叉熵的轮廓差异项

我们将[^1]中公式(4)里的轮廓误差L2项替换为交叉熵损失。此外，我们不再像[^1]中那样使用基于合成数据训练的网络来获取手和物体的分割结果，而是采用MSeg [^2]的分割预测结果。

MSeg可预测包含194个类别的分割结果。我们将这些类别重新映射为3个类别：物体、人和背景。图2展示了使用MSeg网络对示例图像预测的这3个类别的置信度图。

---

物体、人物和背景。图2展示了使用MSeg网络在示例上对这3个类别预测的置信度图。

用$S_v(c)$表示相机v的分割置信度图，其中$c \in \{object, person, background\}$，而渲染后的分割结果为$S_v^R(c)$（即若$S_v^R(c)$中的像素属于类别c，则该像素值为1，否则为0），轮廓项定义如下：

 $$ E_{\mathrm{m a s k}}=\sum_{c}S_{v}(c)\cdot S_{v}^{R}(c)\;. $$

![Image](images/583c9bc9c1f91772401e5878e0f9baa28102ac9446f1fe01fde9389c8e15c875.jpg)

![Image](images/8bda5c948767efd4eed923c145092014fd1bf72554a1c23ca419ae8478ad3a92.jpg)

![Image](images/af2a5b52f9da62efbc3271a12a40f79ba2a9e6d001b50e3c5a3cea91d6d9e8c3.jpg)

![Image](images/39e8d01c0a36c23b8942447a3e890eb9540ddb9b476092c086212d15f157f55f.jpg)

<div style="text-align: center;">Figure 2: Confidence maps for the hand, object, and background classes obtained by merging the predictions made by MSeg [^2] for 194 categories.</div>

### 2.2 解剖学导向的关节轴

我们直接针对关节角度进行优化，而非其PCA分量。我们发现这种策略有助于：1）实现使用PCA分量时无法达到的抓取姿势；2）避免在基于分割图和深度图进行优化时陷入局部极小值。MANO [^3]模型中的关节轴并未沿手指的解剖方向排列，这使得定义关节角度限制变得困难。图3a展示了MANO模型中的关节轴，图3c则显示了由于关节角度限制不准确而导致的不合理的姿势。图3b展示了沿解剖方向手动对齐的关节轴，以及其在HO-3D_v3数据集中的对应姿势。

### 2.3 改进的排斥项

我们将 $E_{phy}$ 的式 (9) 中的物理合理性项 [^1] 替换为更简单的基于球体的排斥项。更具体地说，我们通过用球体填充手和物体的内部体积，获得了它们的“基于球体的表示”，如图 4 所示。我们将 $i^{th}$ 手部球体与 $j^{th}$ 物体球体之间的排斥项定义如下；这两个球体的半径分别记为 $r_{i}^{h}$、$r_{j}^{o}$，中心分别记为 $c_{i}^{h}$、$c_{j}^{o}$：

 $$ E_{\mathrm{p h y}}=\sum_{i}\sum_{j}\max(0,r_{i}^{h}+r_{j}^{o}-||c_{i}^{h}-c_{j}^{o}||_{2}-t), $$

其中t表示手与物体之间允许的穿透量。我们使用$t = 2mm$。

---

![Image](images/878b8bc818de7fb4a66cdde6d4edc763be0af3d78e3d7b16d771ac1942cd0ca8.jpg)

![Image](images/d45002279a4e6bfe4618d5518b65d58949c2f034fe8c690d80bcf2972b8f716f.jpg)

<div style="text-align: center;">(a) Joint axes not aligned along anatomical direction in MANO [^3]</div>

<div style="text-align: center;">(b) Joint axes manually aligned with joint direction in our method</div>

![Image](images/e083322188d55db7cb8d4ca1883f2435fff731dda1e707e74f6a35c0d5062d47.jpg)

<div style="text-align: center;">(c) Twisted thumb when using nonanatomically aligned joint axes</div>

![Image](images/68ee1ac759c03097f352020bc8809603e99f436cd3da6f2a0a5671370b32ea1c.jpg)

<div style="text-align: center;">(d) More natural pose when using anatomically aligned joint axes</div>

<div style="text-align: center;">Figure 3: We align the joint axes along anatomical direction of the fingers resulting in more natural poses in the HO-3D_v3 dataset.</div>

## 3 评估

我们在此报告HO-3D_v3相较于HO-3D_v2在手部姿势精度上的提升情况。为了衡量我们方法的精度，我们手动标注了50帧图像中点云中的指尖位置。该点云是通过将多摄像头设置下所有摄像头的点云合并得到的，如图5所示。此外，这50帧是特意从HO-3D_v2序列中手部姿势精度较差的部分选取的。如表2所示，HO-3D_v3的精度比HO-3D_v2提高了4毫米。

图2展示了HO-3D_v3和HO-3D_v2数据集在动态和静态抓取序列下的接触图，以及手与物体间的平均穿透量。如果手表面某点到物体表面的距离小于4毫米，我们就认为该点与物体相接触。我们允许这一容差，以弥补深度误差和相机校准误差。HO-3D_v3中的姿势表现出更多的手-物体接触，同时穿透程度相似。数据集中各序列的接触图分别显示在图6、图7和图8中。

---

![Image](images/50f60ac4173178e780e2f638d3dedd5d42af8e3fb5a6e310c3efd558d1d47598.jpg)

<div style="text-align: center;">(a) Bleach Cleanser object with its inner volume populated with spheres</div>

![Image](images/9a83daec147e61acd6ffefc5780bd553713d2df876e2c76868469e41d923232d.jpg)

<div style="text-align: center;">(b) Hand model and its sphere representation</div>

<div style="text-align: center;">Figure 4: Sphere-based object and hand models.</div>

![Image](images/d1b55fd2847252e7a463e666d3d41e9b1bab404e8ea2aee2edf82714ff4b7643.jpg)

<div style="text-align: center;">Figure 5: Point cloud from all cameras along with manual annotation of the finger tips.</div>

## 参考文献

[^1]: Shreyas Hampali, Mahdi Rad, Markus Oberweger, and Vincent Lepetit. Honnotate: A method for 3d annotation of hand and object poses. In Computer Vision and Pattern Recognition (CVPR), 2020. 1, 2, 3

[^2]: John Lambert, Zhuang Liu, Ozan Sener, James Hays, and Vladlen Koltun. MSeg: A composite dataset for multi-domain semantic segmentation. In Computer Vision and Pattern Recognition (CVPR), 2020. 2, 3

## [^3]: Javier Romero, Dimitrios Tzionas, and Michael J. Black. Embodied hands: Modeling and capturing hands and bodies together. ACM Transactions on Graphics, (Proc. SIGGRAPH Asia), 36(6):245:1–245:17, November 2017. 3,4

---

![Image](images/222e9c1e4fb6d05781ff8e5e7cebd85bbe6165bb8c51bc280bd76ec7f2650363.jpg)

<div style="text-align: center;">Figure 6: Comparison of contact maps between the 'v2' and 'v3' versions of the HO-3D dataset for all the dynamic grasp sequences. The mean penetration for each sequence is provided in the inset. We see an overall improvement in grasp quality.</div>

---

![Image](images/6c0fb51383d98a8cb13145fd7def17b650b2d38cbfc3c58a9a5bf457f513cb7e.jpg)

<div style="text-align: center;">Figure 7: Comparison of contact maps between the 'v2' and 'v3' versions of the HO-3D dataset for the static grasp sequences. The mean penetration for each sequence is provided in the inset. The grasp quality between the two versions are similar.</div>

---

![Image](images/22a679a360f49f9238ebe985924f65bec145fc6e7c91c0c42448062506be8cb7.jpg)

<div style="text-align: center;">Figure 8: Comparison of contact maps between the 'v2' and 'v3' versions of the HO-3D dataset for the static grasp sequences. The mean penetration for each sequence is provided in the inset. The grasp quality between the two versions are similar.</div>