# HO-3D_v3: Improving the Accuracy of Hand-Object Annotations of the HO-3D Dataset

Shreyas Hampali $^{(1)}$, Sayan Deb Sarkar $^{(1)}$, Vincent Lepetit $^{(2,1)}$

 $^{(1)}$Institute for Computer Graphics and Vision, Graz University of Technology, Graz, Austria

(2) Université Paris-Est, École des Ponts ParisTech, Paris, France

{<firstname>.<lastname>}@icg.tugraz.at

Project page: <https://www.tugraz.at/index.php?id=40231>

## 1 What's new in the 'v3' version of HO-3D?

HO-3D is a dataset providing image sequences of various hand-object interaction scenarios annotated with the 3D pose of the hand and the object and was introduced in [^1] as version HO-3D_v2. The annotations were obtained automatically using an optimization method introduced in the original paper.

HO-3D_v3 provides more accurate annotations for both the hand and object poses thus resulting in better estimates of contact regions between the hand and the object. The new annotations were obtained using an improved version of the optimization technique of [^1] as detailed in Section 2 below. Table 1 shows the statistics of the HO-3D_v2 and HO-3D_v3 versions of the dataset.

The improvement in accuracy of HO-3D_v3 annotations is shown in Table 2 based on 50 manually annotated frames. Figure 1 shows the contact regions between the hand and the objects over the hand surface averaged over the entire dataset together with the mean penetration distance. HO-3D_v3 annotations clearly results in more contacts compared to HO-3D_v2.

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'>Train set #images</td><td style='text-align: center; word-wrap: break-word;'>Test Set #images</td><td style='text-align: center; word-wrap: break-word;'>#objects</td><td style='text-align: center; word-wrap: break-word;'>#subjects</td><td style='text-align: center; word-wrap: break-word;'>Release date</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>HO-3D_v2</td><td style='text-align: center; word-wrap: break-word;'>66,034</td><td style='text-align: center; word-wrap: break-word;'>11,524</td><td style='text-align: center; word-wrap: break-word;'>10</td><td style='text-align: center; word-wrap: break-word;'>10</td><td style='text-align: center; word-wrap: break-word;'>9 Jan 2020</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>HO-3D_v3</td><td style='text-align: center; word-wrap: break-word;'>83,325</td><td style='text-align: center; word-wrap: break-word;'>20,137</td><td style='text-align: center; word-wrap: break-word;'>10</td><td style='text-align: center; word-wrap: break-word;'>10</td><td style='text-align: center; word-wrap: break-word;'>1 Jul 2021</td></tr></table>

<div style="text-align: center;">Table 1: Statistics of the HO-3D_v2 and HO-3D_v3 datasets.</div>

## 2 Algorithmic improvements

We made the following three changes to our method proposed in [^1] in order to improve the accuracy of the pose estimates.

---

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'>mean (mm)</td><td style='text-align: center; word-wrap: break-word;'>std (mm)</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>HO-3D_v2</td><td style='text-align: center; word-wrap: break-word;'>12.31</td><td style='text-align: center; word-wrap: break-word;'>3.8</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>HO-3D_v3</td><td style='text-align: center; word-wrap: break-word;'>8.12</td><td style='text-align: center; word-wrap: break-word;'>2.2</td></tr></table>

<div style="text-align: center;">Table 2: Accuracy of our estimations with respect to manual annotations of fingertips using the point cloud built from all camera depth maps. We manually annotated 50 frames with all 5 fingertip locations. The frames were specifically chosen from parts of the sequence where HO-3D_v2 provides poor estimates.</div>

![Image](images/a39dce4d39a5d38f76e9b5f68f2d748b9480927f3d72242d47ee43d7e6bfb6b4.jpg)

<div style="text-align: center;">Figure 1: Contact maps for the HO-3D_v2 and HO-3D_v3 versions of the datasets for dynamic and static grasps. ‘Red’ regions denote complete contact and ‘Blue’ regions denote no contact. The mean penetration between hand and object are provided in the inset. The ‘v3’ dataset shows more contact regions than the ‘v2’ dataset with significant improvement in the case of dynamic grasps.</div>

### 2.1 Cross-entropy based Silhouette Discrepancy Term

We replace the L2 term for the Silhouette error in Eq. (4) of [^1] with the cross-entropy loss. Moreover, instead of obtaining the segmentations for the hand and object from a network trained on synthetic data as in [^1], we use the segmentation prediction by MSeg [^2].

MSeg predicts a segmentation with 194 categories. We remap these categories to 3 classes: ob-

---

ject, person and background. Figure 2 shows the confidence maps predicted for these 3 categories using the MSeg network on an example.

Denoting the segmentation confidence maps for camera v by $S_v(c)$, where $c \in \{object, person, background\}$ and the rendered segmentation by $S_v^R(c)$ (i.e. a pixel in $S_v^R(c)$ has value 1 if lying on class c and 0 otherwise), the silhouette term is defined as

 $$ E_{\mathrm{m a s k}}=\sum_{c}S_{v}(c)\cdot S_{v}^{R}(c)\;. $$

![Image](images/583c9bc9c1f91772401e5878e0f9baa28102ac9446f1fe01fde9389c8e15c875.jpg)

![Image](images/8bda5c948767efd4eed923c145092014fd1bf72554a1c23ca419ae8478ad3a92.jpg)

![Image](images/af2a5b52f9da62efbc3271a12a40f79ba2a9e6d001b50e3c5a3cea91d6d9e8c3.jpg)

![Image](images/39e8d01c0a36c23b8942447a3e890eb9540ddb9b476092c086212d15f157f55f.jpg)

<div style="text-align: center;">Figure 2: Confidence maps for the hand, object, and background classes obtained by merging the predictions made by MSeg [^2] for 194 categories.</div>

### 2.2 Anatomically Oriented Joint Axes

We optimize directly on the joint angles instead of its PCA components. We have found this strategy helpful to 1) reach grasp poses that were not reachable when using PCA components and 2) avoid getting stuck in local minima especially when optimizing on segmentation and depth maps. The joint axes in the MANO [^3] model are not aligned along the anatomical directions of fingers which makes the task of defining the joint angle limits difficult. Figure 3a shows the joint axes in the MANO model and Figure 3c shows the resulting implausible pose due to inaccurate joint angle limits. Figure 3b shows the manually aligned joint axes along the anatomical direction and the resulting pose in the HO-3D_v3 dataset.

### 2.3 Improved Repulsion Term

We replace the physical plausibility term $E_{phy}$ in Eq. (9) of [^1] with a simpler sphere-based repulsion term. More specifically, we obtained a “sphere-based representation” for the hand and the object by populating their inner volumes with spheres as shown in Figure 4. We define the repulsion term between the $i^{th}$ hand sphere and the $j^{th}$ object sphere with radii denoted by $r_{i}^{h}$, $r_{j}^{o}$ and centers denoted by $c_{i}^{h}$, $c_{j}^{o}$, respectively as:

 $$ E_{\mathrm{p h y}}=\sum_{i}\sum_{j}\max(0,r_{i}^{h}+r_{j}^{o}-||c_{i}^{h}-c_{j}^{o}||_{2}-t), $$

where t is the allowed penetration between the hand and the object. We use $t = 2mm$.

---

![Image](images/878b8bc818de7fb4a66cdde6d4edc763be0af3d78e3d7b16d771ac1942cd0ca8.jpg)

![Image](images/d45002279a4e6bfe4618d5518b65d58949c2f034fe8c690d80bcf2972b8f716f.jpg)

<div style="text-align: center;">(a) Joint axes not aligned along anatomical direction in MANO [^3]</div>

<div style="text-align: center;">(b) Joint axes manually aligned with joint direction in our method</div>

![Image](images/e083322188d55db7cb8d4ca1883f2435fff731dda1e707e74f6a35c0d5062d47.jpg)

<div style="text-align: center;">(c) Twisted thumb when using nonanatomically aligned joint axes</div>

![Image](images/68ee1ac759c03097f352020bc8809603e99f436cd3da6f2a0a5671370b32ea1c.jpg)

<div style="text-align: center;">(d) More natural pose when using anatomically aligned joint axes</div>

<div style="text-align: center;">Figure 3: We align the joint axes along anatomical direction of the fingers resulting in more natural poses in the HO-3D_v3 dataset.</div>

## 3 Evaluations

We report here the improvement in hand pose accuracy of HO-3D_v3 with respect to HO-3D_v2. We manually annotated the point cloud of 50 frames with finger tip locations to measure the accuracy of our method. The point cloud is obtained by combining the point cloud from all cameras in the multi-camera setup as shown in Figure 5. Further, the 50 frames were specifically chosen from parts of the HO-3D_v2 sequences where the hand pose accuracy is poor. As shown in Table 2, HO-3D_v3 shows 4mm improvement in accuracy compared to HO-3D_v2.

Figure 2 shows the contact maps for dynamic and static grasp sequences for HO-3D_v3 and HO-3D_v2 datasets along with mean penetration between hand and object. We consider a point on the surface of the hand to be in contact with the object if the distance to the object surface is less than 4mm. We allow this tolerance to account for depth errors and error in camera calibration. The poses in HO-3D_v3 exhibit more hand-object contact while having similar inter-penetration. The contact maps for each of the sequences in the dataset are shown in Figures 6, 7 and 8.

---

![Image](images/50f60ac4173178e780e2f638d3dedd5d42af8e3fb5a6e310c3efd558d1d47598.jpg)

<div style="text-align: center;">(a) Bleach Cleanser object with its inner volume populated with spheres</div>

![Image](images/9a83daec147e61acd6ffefc5780bd553713d2df876e2c76868469e41d923232d.jpg)

<div style="text-align: center;">(b) Hand model and its sphere representation</div>

<div style="text-align: center;">Figure 4: Sphere-based object and hand models.</div>

![Image](images/d1b55fd2847252e7a463e666d3d41e9b1bab404e8ea2aee2edf82714ff4b7643.jpg)

<div style="text-align: center;">Figure 5: Point cloud from all cameras along with manual annotation of the finger tips.</div>

## References

[^1]: Shreyas Hampali, Mahdi Rad, Markus Oberweger, and Vincent Lepetit. Honnotate: A method for 3d annotation of hand and object poses. In Computer Vision and Pattern Recognition (CVPR), 2020. 1, 2, 3

[^2]: John Lambert, Zhuang Liu, Ozan Sener, James Hays, and Vladlen Koltun. MSeg: A composite dataset for multi-domain semantic segmentation. In Computer Vision and Pattern Recognition (CVPR), 2020. 2, 3

[^3]: Javier Romero, Dimitrios Tzionas, and Michael J. Black. Embodied hands: Modeling and capturing hands and bodies together. ACM Transactions on Graphics, (Proc. SIGGRAPH Asia), 36(6):245:1–245:17, November 2017. 3,4

---

![Image](images/222e9c1e4fb6d05781ff8e5e7cebd85bbe6165bb8c51bc280bd76ec7f2650363.jpg)

<div style="text-align: center;">Figure 6: Comparison of contact maps between the 'v2' and 'v3' versions of the HO-3D dataset for all the dynamic grasp sequences. The mean penetration for each sequence is provided in the inset. We see an overall improvement in grasp quality.</div>

---

![Image](images/6c0fb51383d98a8cb13145fd7def17b650b2d38cbfc3c58a9a5bf457f513cb7e.jpg)

<div style="text-align: center;">Figure 7: Comparison of contact maps between the 'v2' and 'v3' versions of the HO-3D dataset for the static grasp sequences. The mean penetration for each sequence is provided in the inset. The grasp quality between the two versions are similar.</div>

---

![Image](images/22a679a360f49f9238ebe985924f65bec145fc6e7c91c0c42448062506be8cb7.jpg)

<div style="text-align: center;">Figure 8: Comparison of contact maps between the 'v2' and 'v3' versions of the HO-3D dataset for the static grasp sequences. The mean penetration for each sequence is provided in the inset. The grasp quality between the two versions are similar.</div>