# UniHOPE: A Unified Approach for Hand-Only and Hand-Object Pose Estimation Yinqiao Wang $^{*}$ Hao Xu $^{*}$ Pheng-Ann Heng Chi-Wing Fu Department of Computer Science and Engineering Institute of Medical Intelligence and XR The Chinese University of Hong Kong {ygwang, xuhao, pheng, cwfu}@cse.cuhk.edu.hk ## Abstract Estimating the 3D pose of hand and potential hand-held object from monocular images is a longstanding challenge. Yet, existing methods are specialized, focusing on either bare-hand or hand interacting with object. No method can flexibly handle both scenarios and their performance degrees when applied to the other scenario. In this paper, we propose UniHOPE, a unified approach for general 3D hand-object pose estimation, flexibly adapting both scenarios. Technically, we design a grasp-aware feature fusion module to integrate hand-object features with an object switcher to dynamically control the hand-object pose estimation according to grasping status. Further, to uplift the robustness of hand pose estimation regardless of object presence, we generate realistic de-occluded image pairs to train the model to learn object-induced hand occlusions, and formulate multi-level feature enhancement techniques for learning occlusion-invariant features. Extensive experiments on three commonly-used benchmarks demonstrate UniHOPE's SOTA performance in addressing hand-only and hand-object scenarios. Code will be released on . ### 1. Introduction Estimating the 3D pose of hand and potential hand-held objects from monocular images is a long-standing task with applications in VR/AR, human-computer interactions, etc. However, existing methods are divided. As Fig. 1 illustrates, hand pose estimation (HPE) methods [^5] [^9] [^25] [^36] [^39] [^49] [^64] predict the 3D hand pose without considering the hand-held object. Conversely, hand-object pose estimation (HOPE) methods [^18] [^19] [^31] [^32] [^47] assume the presence of a hand-held object and perform object pose estimation with an extra object branch. Yet, they always make predictions even there is no object. Neither approach offers the flexibility to consider both hand-only and hand-object scenarios. Tab. 1 provides a detailed analysis of the performance of state-of-the-art (SOTA) HPE methods [^39] [^49] [^64] and ![Image](images/aa6270296342b63e00252298640b9c87af171ac167dc1a9c75e285292962a06d.jpg)

Figure 1. Existing approaches (top) for 3D hand pose estimation are either Hand Pose Estimation (HPE), which predicts hand pose only, or Hand-Object Pose Estimation (HOPE), which assumes hand-held object. Our novel UniHOPE approach (bottom) offers flexibility and robustness to handle both scenes in a unified manner.

HOPE methods [^18] [^31]. We observe an obvious performance degradation when these methods are applied across different scenes (see “Hand-Only↔Hand-Object Scene”) due to their task-specific designs. Though training on all scenes helps, it negatively impacts their original task performance (see “All→Hand-Only/Hand-Object Scene”), revealing their limited generalization capabilities. This observation motivates the need for a unified approach that can adapt effectively to both hand-only and hand-object scenes. In this work, we present UniHOPE, the first method to unify HPE and HOPE by addressing (i) basic criteria: adaptively switch between two scenes; and (ii) advanced criteria: robustly estimate hand pose regardless of object presence. First, to meet the basic criteria, we propose that the hand pose must always be predicted regardless of whether the hand is grasping an object or not, while the object pose should be estimated only if the object is present. Though a straightforward solution is to manually select existing SOTA HPE and HOPE methods according to the input scene, this approach is suboptimal, as switching between models leads to incoherent results and prevents joint optimization for one ---

HPE	Hand-Only Scene		Hand-Only $\rightarrow$ Hand-Object Scene		All $\rightarrow$ Hand-Only Scene		All $\rightarrow$ Hand-Object Scene
HPE	J-PE $\downarrow$	V-PE $\downarrow$	J-PE $\downarrow$	V-PE $\downarrow$	J-PE $\downarrow$	V-PE $\downarrow$	J-PE $\downarrow$	V-PE $\downarrow$
HandOccNet [^39]	12.98	12.52	19.60 (-6.62)	18.95 (-6.43)	13.16 (-0.18)	12.70 (-0.18)	14.58	14.10
H2ONet [^49]	13.34	13.13	21.98 (-8.64)	21.42 (-8.30)	14.14 (-0.80)	14.00 (-0.87)	15.20	15.03
SimpleHand [^64]	14.05	13.51	18.37 (-4.32)	17.54 (-4.03)	14.63 (-0.58)	13.96 (-0.45)	14.88	14.21

HOPE	Hand-Object Scene		Hand-Object $\rightarrow$ Hand-Only Scene		All $\rightarrow$ Hand-Object Scene		All $\rightarrow$ Hand-Only Scene
HOPE	J-PE $\downarrow$	V-PE $\downarrow$	J-PE $\downarrow$	V-PE $\downarrow$	J-PE $\downarrow$	V-PE $\downarrow$	J-PE $\downarrow$	V-PE $\downarrow$
Keypoint Trans. [^18]	17.99	17.57	25.10 (-7.11)	24.40 (-6.83)	18.79 (-1.00)	18.35 (-0.78)	19.75	19.26
HFL-Net [^31]	14.61	14.13	19.39 (-4.78)	18.61 (-4.48)	14.77 (-0.16)	14.29 (-0.16)	13.61	13.10

1. Existing HRE metnoas trained on nano-only scene exnion obvious performance degradation when testing on nano-object scene (1st vs. 2nd columns). Though training on all scenes (3rd & 4th columns) helps to improve metrics on the hand-object scene (2nd vs. 4th columns), their original performance is adversely affected (1st vs. 3rd columns). The HOPE methods also exhibit a similar pattern (see bottom part). These results demonstrate the inabilities of the existing methods to flexibly handle hand-only and hand-object scenes altogether.

model to work on both scenes. Importantly, we design an end-to-end method that dynamically controls object pose estimation through an internal object switcher by estimating the confidence of the grasping status, thereby promoting compatibility with various scenarios. However, solely adopting existing network architectures along with our object switcher encounters another issue caused by the commonly-used hand-object information interaction structure [^31] [^32] [^45]. When no object is present, extracting object features is unnecessary, so irrelevant object-to-hand feature transitions compromise hand pose estimation accuracy. To overcome this issue, we formulate a grasp-aware feature fusion module to utilize grasping confidence to select effective object features in hand-object feature fusion. Second, the advanced criteria emphasize coherent and robust hand pose estimation, regardless of whether the hand is grasping an object or not. As hand-held objects frequently cause severe occlusions, it is essential to learn occlusion-invariant features to accurately recover hand poses. Ideally, the feature of a non-occluded hand serves as the optimal representation for an occluded hand with the same pose, as it simplifies the prediction difficulty. To facilitate this, we propose training the model by transferring knowledge from the corresponding non-occluded hands to the occluded ones. Due to the scarcity of such paired data, we innovatively leverage diffusion-based generative models to create realistic de-occluded hand images from the originally-occluded ones. Subsequently, we adopt multi-level feature enhancement techniques to help the network simulate occlusion-invariant features by utilizing information from the de-occluded hand images in a self-distillation framework. Our main contributions are summarized as follows: - We demonstrate the necessity for a unified solution to hand-object pose estimation and propose UniHOPE, a novel approach to handle general hand-object scenarios. - We design an internal object switcher that provides flexibility across different scenes and formulate a grasp-aware feature fusion module to adaptively utilize the effective object information based on grasping status. - We propose an occlusion-invariant feature learning strategy for robustness, first using a generative de-occluder to prepare paired de-occluded hand images and then applying feature enhancement at multiple levels. • Extensive experiments on three widely-used datasets in our unified setting show the SOTA performance of UniHOPE. ### 2. Related Work Monocular 3D Hand Pose Estimation (HPE). Most methods formulate HPE by regressing MANO coefficients [2-4, 10, 51, 57, 60-63, 65]. Other common representations include voxels [^26] [^36] [^37] [^52], implicit functions [^25], and meshes [^8] [^9] [^29] [^49] [^64]. While they achieve superior performance in predicting hand poses, they do not account for object pose estimation during hand-object interactions, which is critical for practical applications. Monocular 3D Hand-Object Pose Estimation (HOPE). To jointly estimate the hand and object poses simultaneously, recent works can be categorized into two main streams: (i) Template-free methods reconstruct objects without knowing their 3D models. Hasson et al. [^19] recover the object mesh from a deformed icosphere. Tse et al. [^45] propose to optimize hand and object meshes iteratively. More recent works leverage implicit [^11] [^12] [^24] [^54] [^56] or neural fields [^13] to represent the hand and object; yet, these methods often struggle to accurately model unseen objects due to limited data prior. (ii) Template-based methods assume the 3D object model is known, focusing on regressing its pose. Liu et al. [^32] proposes a semi-supervised learning approach. Lin et al. [^31] design a dual-branch backbone to leverage mutual hand-object information. Though these methods produce promising results, they focus on the hand-object scene, lacking the flexibility to handle the hand-only scenario. Hand-Object Image Synthesis. Since we generate paired de-occluded hand images for unified scenarios, we also review methods for hand-object image synthesis. Rendering-based methods [^15] [^19] [^53] use common tools [^1] [^14] [^35] to render images as augmented data for HPE and HOPE tasks. With the advance of generative models [16, 21, 42–44, 48], recent methods generate more realistic data for various pur- --- ![Image](images/7d3bfac0019e3c781e3911cbf33f11d29943b239aea7edd7533c6cb4d701fb55.jpg)

Figure 2. Our UniHOPE framework. (i) We first de-occlude hand images occluded by objects to form pairs, conditioned on the depth map and hand-object mask, with adaptive selection of control strength to produce high-quality samples; (ii) to accommodate both hand-only and hand-object scenes, our object switcher dynamically controls the object output by predicting grasping status, which guides the feature fusion module to eliminate irrelevant object features; and (iii) to robustly estimate hand pose, our multi-level feature enhancement techniques utilize paired data to learn occlusion-invariant hand features.

poses, including image generation [^22] [^38] [^59], object-prior guidance [^55], hand re-malformation [^33], and training data augmentation [^40] [^50]. In this work, we propose to generate paired de-occluded hand images to facilitate learning occlusion-invariant hand features, thereby improving robustness to handle both hand-only and hand-object scenes. ### 3. Method #### 3.1. Overview Distinct from previous studies, we address a more general scenario, i.e., the model always predicts hand pose, regardless of whether the hand is grabbing an object or not. If an object is present, our model also estimates its pose. Fig. 2 shows our UniHOPE pipeline. First, we propose to dynamically estimate hand-object pose in an end-to-end manner (see the red section in Fig. 2), where our object switcher flexibly controls the output and our grasp-aware feature fusion module integrates grasp-relevant object information (Sec. 3.2). Next, to improve robustness against occlusions, we first design our generative de-occluder to prepare high-quality paired data by adaptively adjusting control strengths (see blue section), which is then used to learn occlusion-invariant features through our multi-level feature enhancement techniques (see yellow section, Sec. 3.3). Finally, we detail the loss functions used in our approach in Sec. 3.4. #### 3.2. Dynamic Hand-Object Pose Estimation To accommodate both HPE and HOPE, it is essential to consistently predict hand poses while estimating object poses only when an object is present. Directly combining existing HPE and HOPE methods is insufficient due to the incoherence introduced by model switching and the lack of joint optimization. In this section, we present our end-to-end dynamic hand-object pose estimation approach. To flexibly control the object output, we introduce an object switcher that predicts grasping status, trained with automatically-generated labels. Further, we propose a grasp-aware feature fusion module guided by the grasping status to prevent object-to-hand irrelevant feature transitions when no object is present. Grasping Label Preparation. To support model training, we automatically prepare grasping status labels, as existing datasets [^6] [^17] do not provide this information. Following [^50], we compute the isotropic Relative Rotation Error (RRE) and Relative Translation Error (RTE) between the object poses in the initial and current frames: $$ \mathrm{R R E}=\operatorname{a r c c o s}(\frac{\operatorname{t r a c e}(\xi_{R}^{t}{}^{\top}\xi_{R}^{0}-1)}{2}),\mathrm{R T E}=||\xi_{T}^{t}-\xi_{T}^{0}||_{2}, $$ where $\xi_R \in \mathbb{R}^{3 \times 3}$ and $\xi_T \in \mathbb{R}^3$ denote the object rotation matrix and translation vector, respectively. The superscripts 0 and t indicate the frame index. The object is labeled as grasped if the computed errors exceed a defined threshold. Object Switcher. With the prepared labels, we employ a multi-layer perceptron (MLP) $g(\cdot)$ to predict the grasping status from the object feature $\mathbf{F}_p^o$, which is extracted by the Feature Pyramid Network (FPN) [^30] from the input image. This process is supervised by the binary cross-entropy loss: $$ \mathcal{L}^{s}=-\sum_{j=0}^{1}\mathbb{1}(\hat{G}=j)\cdot\log\frac{\exp(g(\mathbf{F}_{p}^{o})_{j})}{\sum_{k=0}^{1}\exp(g(\mathbf{F}_{p}^{o})_{k})}, $$ where $\hat{G}$ is the ground-truth grasping label and $1(\cdot)$ is the indicator function. During testing, the object pose estimation branch is deactivated if predicted as non-grasping, providing more accurate responses for hand-object interactions. --- Grasp-aware Feature Fusion. Previous studies have shown that feature interaction between hand and object can effectively enhance performance in hand-object scenes [^31] [^32] [^45]; however, such interaction can disrupt hand feature learning in the hand-only scene due to the absence of objects (see Tab. 1). To mitigate interference from irrelevant object features, we design the grasp-aware feature fusion. During training, the object feature $\mathbf{F}_r^o$ and the hand feature $\mathbf{F}_p^h$ produced by RoIAlign [^20] are concatenated to form the feature $\mathbf{F}^H$ only when the object is predicted as grasped. Next, $\mathbf{F}^H$ is processed through a multi-head attention block [^46], resulting in the fused hand-object feature $\mathbf{F}^{OH}$: $$ \begin{aligned}{\mathbf{F}^{H}=\operatorname{C o n c a t}(\mathbf{F}_{r}^{h},s\mathbf{F}_{r}^{o}+(1-s)\mathbf{F}_{r}^{h})\quad\mathrm{a n d}}\\ {\mathbf{F}^{O H}=\operatorname{S o f t m a x}(\frac{\mathbf{F}^{H}\mathbf{F}^{H^{T}}}{\sqrt{d^{H}}})\mathbf{F}^{H},}\\ \end{aligned} $$ where $s=\mathrm{Argmax}(g(\mathbf{F}_{p}^{o}))$ indicates the predicted grasping status. Concat( $\cdot$) and Softmax( $\cdot$) represent the concatenation and soft-max operations along the channel dimension, respectively. $d^{H}$ is the channel dimension of $\mathbf{F}^{H}$. This approach enables the network to flexibly toggle object outputs while maintaining robust feature representations for the hand across various scenes. Then, similar to [^31], $\mathbf{F}^{OH}$ is fed into an hourglass-structured hand encoder to produce MANO-related features and regress 2D hand joint coordinates. #### 3.3. Occlusion-invariant Feature Learning Hands are frequently occluded when interacting with objects. To achieve robust estimation, the extracted hand feature for the same hand pose should be occlusion-invariant and irrespective of object presence. Given the fact that estimating the bare hand pose is easier than that of the one occluded by a held object, our key insight is to enable the network to simulate the non-occluded hand features from the occluded ones by transferring cross-domain knowledge. Thus, we propose generating plausible de-occluded hand images as pairs for training samples affected by object-caused occlusions, employing an adaptive adjustment strategy for control strength to maximize generation quality. Further, we design multi-level feature enhancement techniques that leverage this paired data to promote comprehensive hand feature learning. Generative De-occluder. For occluded hand images, our goal is to de-occlude by realistically removing the grasped object while preserving the hand pose to create paired data. Inspired by [^33] [^34], we utilize ControlNet [^58], pre-trained on synthetic hand images, to repaint the hand-object region M guided by a rendered hand depth map D. Specifically, following the latent diffusion model [^42], the original image X is first projected into latent space as $\mathbf{x}_0$ using a variational auto-encoder [^27]. We then follow the standard forward diffusion process as outlined in [^21]. In each reverse step $t \in \{T, T-1, ..., 1\}$, to preserve the known background region $(1 - \mathbf{M}) \odot \mathbf{X}$, we can alternate the corresponding feature using $(1 - \mathbf{m}) \odot \mathbf{x}_t$ as long as maintaining the correct properties of its distribution, as the transition from $\mathbf{x}_t$ to $\mathbf{x}_{t-1}$ depends solely on $\mathbf{x}_t$, i.e., $$ \mathbf{x}_{t-1}^{b g}\sim\mathcal{N}\big(\sqrt{\bar{\alpha}_{t}}\mathbf{x}_{0},(1-\bar{\alpha}_{t}\mathbf{I})\big), $$ where $\mathbf{m}$ is downsampled from $\mathbf{M}$ for calculations in latent space, and $\odot$ is the element-wise product. $\mathcal{N}(\cdot)$ denotes the Gaussian distribution. $\bar{\alpha}_t$ denotes the total noise variance at step t, as defined in [^44]. For the unknown hand-object region $\mathbf{M}\odot\mathbf{X}$, we perform the reverse diffusion process using the DDIM sampler [^44], i.e., $$ \mathbf{x}_{t-1}^{{h o}}=\operatorname{D D I M}(\epsilon_{\theta}(\mathbf{x}_{t},\mathbf{x}_{{m a s k}},\mathbf{D})), $$ where $\mathbf{x}_{mask}$ is the latent feature masked by $\mathbf{m}$. $\epsilon_{\theta}(\cdot)$ denotes the denoising model. Thus, the final expression of $\mathbf{x}_{t-1}$ during one reverse step is: $$ \mathbf{x}_{t-1}=\mathbf{m}\odot\mathbf{x}_{t-1}^{{h o}}+(1-\mathbf{m})\odot\mathbf{x}_{t-1}^{{b g}}, $$ which means $\mathbf{x}_{t-1}^{bg}$ is sampled using the known background pixels, while $\mathbf{x}_{t-1}^{ho}$ is sampled from the bare-hand data distribution. They are combined into the new $\mathbf{x}_{t-1}$ using the hand-object mask, ensuring both consistency and realism. After the iterative reverse process, the final denoised vector $\mathbf{x}_0$ is sent to the decoder [^27] to recover images from latent features. We show examples of different occlusion conditions along with their de-occluded counterparts in Fig. 3. Adaptive Control Strength Adjustment. To balance consistency with the condition and realism of the generated hand images, the user often needs to manually adjust the control strength parameter in the generative model. We visualize some examples generated using different control strengths in Fig. 4. In certain cases, such as the top row, too-small control strengths make the generated hand not align well with the depth condition (see (a-b)), while overly-large strengths result in unrealistic appearances (see (d)). Conversely, in other cases like the bottom row, a large control strength is needed to ensure correct hand anatomy (see (h)). Therefore, a fixed control strength cannot be universally applied, while manually setting the value for each case is impractical. To this end, we propose adaptively and automatically adjusting the control strength to enhance generation quality. Specifically, we first define candidate control strengths $\{s_1, s_2, ..., s_n \mid 0 < s_i \leq 1, \forall i\}$ and generate a de-occluded image for each. Then, we employ a pre-trained hand reconstruction model from [^9] to estimate the 3D hand poses from the generated images and evaluate the J-PE against the ground truth. The generated sample with the lowest J-PE is incorporated into the training process with the same ground-truth labels as the original sample, e.g., (c) and (h) in Fig. 4 are selected. This approach maximizes the generation quality by selecting a proper control strength that best balances realism and consistency for each case. --- ![Image](images/221720d2c2c6af7d3e2c3a1d2e0180c9d48622c9e15952b85ced8ec4affbde35.jpg) ![Image](images/5c47962df7408f16caffe161e2616e8601d149a0ff0e4f225b2c11ed046867d1.jpg)

Figure 3. De-occluded examples in various occlusion conditions.

Multi-level Feature Enhancement. Given the pair of the original image and the corresponding generated image (X, X'), our goal is to enhance the hand feature representation of X by leveraging information from X'. To achieve this, we introduce pair-wise feature constraints within a single network in a self-distillation manner. To holistically enhance the capability of recovering occluded information for hand, as shown in Fig. 2, we enhance hand features throughout the hand branch at multiple levels: (i) The initial-level feature $\mathbf{F}_i^h$, extracted from the initial layer of the FPN, captures the low-level information of the hand; (ii) the RoI-level feature $\mathbf{F}_r^h$, output from the FPN after the RoI-align operation, is utilized for adaptive fusion with the object feature; and (iii) the MANO-level feature $\mathbf{F}_m^h$, extracted before the MANO decoder, serves as the most pertinent feature for regressing the MANO coefficients. Since the MANO-level feature is at a relatively late stage, the occluded and de-occluded counterparts may not always reside in a similar feature space. Inspired by [^7], we adopt a multi-head attention block $h(\cdot)$ as the adaptation layer to improve knowledge transfer. Overall, the feature enhancement constraints are formulated as the L1 loss between the features of X and X': $$ \begin{array}{c c}{\mathcal{L}_{{i n i t}}^{{e n h}}=||\mathbf{F}_{i}^{h}-\mathbf{F}_{i}^{h^{\prime}}||_{1},}&{\mathcal{L}_{{R o I}}^{{e n h}}=||\mathbf{F}_{r}^{h}-\mathbf{F}_{r}^{h^{\prime}}||_{1},}\\ \end{array} $$ $$ \begin{array}{r l}{\mathrm{a n d}}&{\mathcal{L}_{M A N O}^{e n h}=||h(\mathbf{F}_{m}^{h})-h(\mathbf{F}_{m}^{h^{\prime}})||_{1},}\end{array} $$ where features with primes belong to the generated image. Occlusion-aware Case Filtering. During the knowledge transition from the generated samples to the original ones, we observe that the feature learning process may not benefit if the original hand is already non-occluded. In this case, the knowledge gap between de-occluded and occluded hand features disappears, which causes the feature constraints to focus on mitigating the sim-to-real domain gap, making the original features close to the simulated ones, thus yielding suboptimal performance. To address this issue, we filter out non-occluded samples for paired feature enhancement. Specifically, we compute the Intersection over Union (IoU) of the provided amodal hand mask (considering object-caused occlusions) and the rendered full hand mask as the ground-truth occlusion proportion. During training, feature constraints are exclusively applied to pairs whose original occlusion proportion $\hat{O}$ exceeds a pre-defined threshold $\tau$.

Figure 4. Visualization of our adaptive control strength adjustment.

Furthermore, non-grasping samples are excluded as they do not contribute additional information. Thus, the feature enhancement is conducted only on grasping and occluded samples, i.e., $$ \mathcal{L}_{*}^{e n h}=\mathbb{1}((\hat{O}\geq\tau)\land(\hat{G}=1))\cdot\mathcal{L}_{*}^{e n h}, $$ where $\tau = 0.1$ in our experiments. $\mathcal{L}_{*}^{enh}$ represents the feature enhancement constraints at multiple levels. #### 3.4. Loss Functions Our training loss includes regular hand and object losses similar to [^31], along with our object switcher loss $\mathcal{L}^s$, and feature enhancement constraints $\mathcal{L}_{init}^{enh}$, $\mathcal{L}_{RoI}^{enh}$, and $\mathcal{L}_{MANO}^{enh}$. First, the hand loss is computed as $\mathcal{L}^h = \mathcal{L}^J + \mathcal{L}^V + \mathcal{L}_{MANO}^{MANO}$, where $$ \begin{align*}\mathcal{L}^{J}&=||\mathbf{J}^{2D}-\hat{\mathbf{J}}^{2D}||_{2}+||\mathbf{J}^{3D}-\hat{\mathbf{J}}^{3D}||_{2},\\\mathcal{L}^{V}=&||\mathbf{V}-\hat{\mathbf{V}}||_{2},\text{and}\mathcal{L}^{MANO}=||(\theta;\boldsymbol{\beta})-(\hat{\theta};\hat{\boldsymbol{\beta}})||_{2}.\end{align*} $$ Here $\mathbf{J}^{2D}$, $\mathbf{J}^{3D}$, and V represent the 2D joint, 3D joint, and 3D vertex coordinates, respectively. $(\theta;\beta)$ represent the MANO coefficients. The hat superscript denotes the ground-truth label. The object loss $\mathcal{L}^{o}$ supervises the predictions of the 2D location (projected from 3D object keypoints) and their corresponding confidences from image grid proposals [^41], i.e., $$ \mathcal{L}^{o}=\sum_{g}\sum_{k=1}^{N^{o}}(\vert\vert p_{g,k}-\hat{p}_{g,k}\vert\vert_{1}+\vert\vert c_{g,k}-\hat{c}_{g,k}\vert\vert_{1}), $$ where $N^{o}$ is the number of keypoints in the 3D bounding box of object mesh. $p_{g,k}$ and $c_{g,k}$ are the pixel location and confidence value at the grid g and control point k, respectively. The hat superscript denotes the ground truth. We compute the object loss only for those grasping images, as they contain the complete object for pose estimation. Overall, the training loss is as follows. $$ \begin{aligned}\mathcal{L}^{total}=&\underbrace{\mathcal{L}^{h}+\mathcal{L}^{o}+\alpha\mathcal{L}^{s}}_{sample-wise}\\&+\underbrace{\gamma_{init}\mathcal{L}_{init}^{enh}+\gamma_{RoI}\mathcal{L}_{RoI}^{enh}+\gamma_{MANO}\mathcal{L}_{MANO}^{enh}}_{pair-wise}\end{aligned} $$ where $\alpha$ and $\gamma_{*}$ are weights to balance the loss terms. ---

	Methods	All Scenes				Hand-Only Scene				Hand-Object Scene
	Methods	J-PE $\downarrow$	PA-J-PE $\downarrow$	V-PE $\downarrow$	PA-V-PE $\downarrow$	J-PE $\downarrow$	PA-J-PE $\downarrow$	V-PE $\downarrow$	PA-V-PE $\downarrow$	J-PE $\downarrow$	PA-J-PE $\downarrow$	V-PE $\downarrow$	PA-V-PE $\downarrow$
HPE	HandOccNet [^39]	14.02	6.17	13.55	5.95	13.16	5.31	12.70	5.11	14.58	6.73	14.10	6.49
	MobRecon [^9]	15.02	6.71	13.93	5.91	14.43	5.88	13.45	5.15	15.40	7.25	14.24	6.39
	H2ONet [^49]	14.78	5.72	14.63	6.19	14.14	4.74	14.00	5.35	15.20	6.35	15.03	6.74
	SimpleHand [^64]	14.78	6.30	14.11	6.03	14.63	5.62	13.96	5.38	14.88	6.74	14.21	6.45
HOPE	Liu et al. [^32]	15.33	6.17	14.79	5.98	15.18	5.48	14.60	5.31	15.43	6.61	14.91	6.40
	Keypoint Trans. [^18]	19.16	7.70	18.71	7.96	19.75	7.59	19.26	7.98	18.79	7.77	18.35	7.94
	HFL-Net [^31]	14.32	6.08	13.83	5.86	13.61	5.20	13.10	5.01	14.77	6.64	14.29	6.41
Unified	H2ONet $^{\dagger}$ + HFL-Net $^{\dagger}$	14.12	5.83	13.75	5.83	13.29	4.70	13.08	5.06	14.66	6.55	14.18	6.33
	H2ONet $^{\dagger}$ + HFL-Net $^{\ddagger}$	14.54	5.90	14.19	6.00	14.14	4.76	14.00	5.35	14.79	6.63	14.31	6.41
	HandOccNet $^{\dagger}$ + HFL-Net $^{\dagger}$	14.34	6.06	13.86	5.85	13.85	5.28	13.35	5.08	14.66	6.56	14.18	6.34
	HandOccNet $^{\ddagger}$ + HFL-Net $^{\ddagger}$	14.52	6.15	14.02	5.93	14.09	5.37	13.58	5.17	14.79	6.65	14.31	6.42
	UniHOPE (ours)	13.03	5.59	12.59	5.40	12.59	4.83	12.12	4.66	13.31	6.08	12.89	5.87

Table 2. Hand-pose estimation results on DexYCB. $^{\dagger}$: pre-trained in the original setting. $^{\ddagger}$: re-trained in the unified setting. The best and second-best are marked in bold and $\underline{\text{underlined}}$. Our UniHOPE attains leading performance for almost all metrics in all scenarios.

Methods	gelatin_box	bleach_cleanser	wood_block	average $\uparrow$
Liu et al. [^32]	26.31	25.07	68.56	38.89
Keypoint Trans. [^18]	0.00	1.31	32.61	10.47
HFL-Net [^31]	25.88	32.08	70.16	41.59
H2ONet $^{\dagger}$ + HFL-Net $^{\dagger}$	29.26	30.71	64.84	40.69
H2ONet $^{\dagger}$ + HFL-Net $^{\dagger}$	26.12	29.33	69.40	40.51
HandOccNet $^{\dagger}$ + HFL-Net $^{\dagger}$	29.30	30.57	64.94	40.69
HandOccNet $^{\dagger}$ + HFL-Net $^{\dagger}$	26.22	29.51	69.40	40.61
UniHOPE (ours)	26.23	32.32	74.29	43.06

Table 3. Unseen object-pose estimation results on DexYCB.

	Methods	J-PE $\downarrow$	J-AUC $\uparrow$	V-PE $\downarrow$	V-AUC $\uparrow$	F@5 $\uparrow$	F@15 $\uparrow$
HPE	HandOccNet [^39]	28.94	47.96	28.10	49.23	23.34	67.86
	MobRecon [^9]	29.61	48.10	28.65	49.59	23.38	67.72
	H2ONet [^49]	30.46	47.09	29.55	48.31	22.10	66.04
	SimpleHand [^64]	29.01	47.58	28.09	49.00	21.93	65.90
HOPE	Liu et al. [^32]	29.54	47.55	28.66	48.83	21.82	67.34
	Keypoint Trans. [^18]	41.04	34.47	39.64	36.02	17.13	56.07
	HFL-Net [^31]	28.45	50.34	27.55	51.57	24.31	69.88
Unified	H2ONet $^{\dagger}$ + HFL-Net $^{\dagger}$	31.27	47.70	30.31	48.86	22.91	67.20
	H2ONet $^{\ddagger}$ + HFL-Net $^{\ddagger}$	28.49	50.45	27.59	51.67	24.33	69.86
	HandOccNet $^{\dagger}$ + HFL-Net $^{\dagger}$	30.96	47.85	30.01	49.02	23.04	67.47
	HandOccNet $^{\ddagger}$ + HFL-Net $^{\ddagger}$	28.33	50.45	27.44	51.67	24.39	69.99
	UniHOPE (ours)	26.23	52.26	25.41	53.52	24.64	70.77

Table 4. Hand-pose estimation results (Root-relative) on HO3D.

### 4. Experiments #### 4.1. Experimental Settings Datasets. In our unified setting, we organize the original dataset into hand-only and hand-object scenes based on the object grasping status. We conduct experiments on the following commonly-used datasets: (i) DexYCB [^6]: we use the more challenging “S3” split (train/test: 376,374/76,360 samples) with unseen grasped objects in the test set (train/test: 15/3 objects). We report performance on the entire dataset (all scenes) as well as separately for hand-only and hand-object scenes; (ii) HO3D [^17] (version 2, train/test: 66,034/11,524 samples): results are submitted to the online server as the ground-truth 3D hand annotations are not publicly accessible, hence results for each scene are unavailable. Further, to evaluate the generalization ability, we perform cross-dataset validation on the test set of (iii) FreiHAND [^65] (train/test: 130,240/3,960 samples), which mainly consists of bare-hand images and lacks object annotations for scene division. Consequently, results for each scene are also unavailable. More details and results on other data splits of DexYCB are provided in the Supp. Evaluation Metrics. We evaluate hand pose estimation using commonly-used metrics, as in [^9] [^39] [^49]: (i) J/V-PE denotes the mean per joint/vertex position error (also known as MPJPE/MPVPE) in mm measured by Euclidean distance between estimated and ground-truth 3D hand joint/vertex coordinates; (ii) J/V-AUC calculates the area under the curve (in percentage) of the percentage of correct keypoints (PCK) across different error thresholds for joint/vertex; and (iii) F@5/F@15 is the harmonic mean of recall and precision (in percentage) between estimated and ground-truth 3D hand vertices under 5mm/15mm thresholds. We report these metrics both before (i.e., root-relative) and after Procrustes Alignment (PA), which aligns the estimation with ground truths by global rotation, translation, and scale adjustment. For object pose evaluation, we measure the average 3D distance (ADD) of the grasped object. Specifically, we report ADD-0.5D, the percentage of objects whose ADD is within 50% of the object diameter as in [^23] considering the challenge of unseen-object pose estimation in DexYCB. Implementation Details. We train UniHOPE on eight NVidia RTX 2080Ti GPUs using a batch size of 64 and the Adam optimizer [^28] with an initial learning rate of $1e^{-4}$ (decay by 0.7 every 10 epochs). Input images are resized to $128 \times 128$ and augmented with random scaling, rotating, translating, and color jittering. To stabilize training, we first train the network with both original and generated images for 30 epochs, then incorporate the feature enhancement constraints for another 40 epochs. Please refer to our Supp. for more details. #### 4.2. Comparison with SOTA Methods We compare our UniHOPE with previous SOTA HPE and HOPE methods [^9] [^18] [^32] [^39] [^64] in our unified setting (trained using their officially-released code). To support --- ![Image](images/9785431dd402a96b95543806cc93230cce3a56492cfcbbdd67d89ed886adf80b.jpg)

Figure 5. Qualitative comparison between our method and SOTA HPE/HOPE methods on hand-only/hand-object scenarios across different datasets. The first and second rows in each example denote the original view and another view, respectively, for better comparison.

![Image](images/5f174845487ed2549e25356b7d170620d2f741e6b046914035db73c1e741f30d.jpg)

Figure 6. The joint AUC comparison under different thresholds. UniHOPE achieves better performance than others, consistently.

the unified task, one straightforward solution is to use a classifier to determine if the hand is grasping any object, then employ an existing HPE or HOPE method accordingly. We denote the combination as A + B, where A is the SOTA HPE method H2ONet [^49] or HandOccNet [^39], and B is the SOTA HOPE method, HFL-Net [^31]. Evaluation on DexYCB. We present quantitative comparisons of hand pose estimation on DexYCB in Tab. 2. Our UniHOPE achieves the best performance overall, showing its effectiveness in general hand-object interactions across various scenarios. The root-relative 3D joint PCK/AUC comparison under different thresholds is shown in Fig. 6, further confirming the comprehensive performance of our approach. For object pose estimation accuracy, we conduct comparisons using per-instance and average ADD-0.5D, as shown in Tab. 3. Compared with SOTA HOPE methods, our method achieves the highest average score on the test set with unseen objects, highlighting its superiority in estimating object pose. Qualitative comparisons with SOTA methods [^31] [^39] [^49] are illustrated in Fig. 5. In the hand-only scenario (see Fig. 5(a)), where the wood block has not been grasped yet, SOTA HOPE method [^31] inevitably produces an extra object pose due to its inflexibility; in contrast, our method does not, thanks to our object switcher's >95% grasping status prediction accuracy. Moreover, our method yields more plausible hand poses. In the hand-object scenario (see Fig. 5(b)), our method produces high-quality hand-object poses, whereas previous methods fail when the hand experiences moderate occlusion, indicating our robustness against such challenges. Evaluation on HO3D. We conduct the same experiment on HO3D [^17]. Tab. 4 shows the root-relative quantitative comparison. Our method achieves top performance across all metrics, demonstrating its effectiveness and robustness. The joint PCK/AUC curve in Fig. 6 also confirms the consistent best results of our approach. In addition, qualitative comparisons in Fig. 5 (d) clearly illustrate the superiority of our method in estimating 3D hand and object pose under partial object-caused occlusion. More quantitative results are available in the Supp. Evaluation on FreiHAND. To assess generalization ability, we perform cross-dataset validation by transferring models trained on DexYCB to the FreiHAND test set. As reported in Tab. 5, our method outperforms all SOTA HPE/HOPE methods, particularly by a substantial margin in root-relative metrics. The qualitative comparison in Fig. 5(c) also shows that UniHOPE produces more accurate hand poses in challenging cases, indicating improved generalization to unseen bare-hand scenes. Evaluation under Different Levels of Occlusion. To showcase the robustness of our method against object-caused occlusion, we partition the DexYCB test set into different occlusion levels based on the ground-truth hand-object occlusion proportion (as detailed in Sec. 3.3) and provide quantitative comparisons in Tab. 6. Our UniHOPE exhibits the best hand pose estimation performance across all occlusion levels, underscoring the efficacy of our feature enhancement techniques. Note that test samples where the hand being absent from the image region are excluded. ---

	Methods	Root-relative						Procrustes Alignment
	Methods	J-PE $\downarrow$	J-AUC $\uparrow$	V-PE $\downarrow$	V-AUC $\uparrow$	F@5 $\uparrow$	F@15 $\uparrow$	J-PE $\downarrow$	J-AUC $\uparrow$	V-PE $\downarrow$	V-AUC $\uparrow$	F@5 $\uparrow$	F@15 $\uparrow$
HPE	HandOccNet [^39]	58.03	29.07	56.06	29.40	14.89	47.75	14.52	71.34	14.09	72.07	40.02	88.08
	MobRecon [^9]	71.18	22.21	68.50	22.34	10.41	36.67	17.74	65.16	17.32	65.90	32.95	81.35
	H2ONet [^49]	79.14	19.88	76.29	19.38	9.69	36.71	14.56	71.21	14.25	71.69	38.73	87.69
	SimpleHand [^64]	60.82	26.09	58.85	26.34	12.84	43.53	15.79	68.90	15.37	69.61	37.03	85.40
HOPE	Liu et al. [^32]	59.40	28.84	57.39	29.04	14.92	47.76	14.00	72.30	13.59	73.02	41.53	89.11
	Keypoint Trans. [^18]	96.27	13.60	93.39	12.37	6.99	27.82	16.97	66.66	17.03	66.38	34.31	83.44
	HFL-Net [^31]	58.02	29.94	56.08	30.20	15.47	48.83	14.29	71.75	13.85	72.52	40.52	88.63
Unified	H2ONet $^{\dagger}$ + HFL-Net $^{\ddagger}$	68.88	24.27	66.49	24.21	12.33	42.38	14.47	71.36	14.10	71.99	39.48	88.14
	H2ONet $^{\dagger}$ + HFL-Net $^{\ddagger}$	68.25	24.49	65.90	24.40	12.30	42.51	14.42	71.47	14.06	72.09	39.54	88.18
	HandOccNet $^{\dagger}$ + HFL-Net $^{\ddagger}$	81.79	21.33	78.91	20.56	10.89	36.68	15.40	69.65	14.85	70.58	38.30	86.94
	HandOccNet $^{\ddagger}$ + HFL-Net $^{\ddagger}$	80.74	20.30	78.00	19.44	10.52	36.38	16.15	68.29	15.58	69.26	36.92	85.93
	UniHOPE (ours)	50.97	34.31	49.21	34.94	17.83	53.46	13.53	73.24	13.14	73.92	43.23	89.55

Table 5. Cross-dataset validation of hand-pose estimation on FreiHAND.

Methods	Occlusion (25%-50%)				Occlusion (50%-75%)				Occlusion (75%-100%)
Methods	J-PE $\downarrow$	PA-J-PE $\downarrow$	V-PE $\downarrow$	PA-V-PE $\downarrow$	J-PE $\downarrow$	PA-J-PE $\downarrow$	V-PE $\downarrow$	PA-V-PE $\downarrow$	J-PE $\downarrow$	PA-J-PE $\downarrow$	V-PE $\downarrow$	PA-V-PE $\downarrow$
HandOccNet [^39]	16.40	7.08	15.85	6.83	18.22	7.60	17.67	7.33	28.15	8.71	27.20	8.40
MobRecon [^9]	16.67	7.61	15.60	6.77	20.04	8.17	18.80	7.48	31.46	9.64	29.97	9.23
H2ONet [^49]	17.07	6.76	16.78	7.09	19.41	7.32	19.07	7.58	31.07	8.82	30.11	8.94
SimpleHand [^64]	16.43	7.05	15.78	6.75	19.33	7.55	18.38	7.39	38.52	10.57	36.85	10.40
Liu et al. [^32]	16.98	6.92	16.43	6.71	19.72	7.11	19.14	6.91	33.80	8.99	32.64	8.71
Keypoint Trans. [^18]	20.95	8.15	20.41	8.31	24.45	8.61	23.88	8.76	38.29	11.21	37.39	11.75
HFL-Net [^31]	16.33	7.00	15.81	6.77	18.66	7.33	18.11	7.11	28.95	8.80	27.94	8.53
H2ONet $^{\dagger}$ + HFL-Net $^{\dagger}$	16.07	6.84	15.57	6.67	19.39	7.40	18.82	7.22	30.32	8.43	29.27	8.28
H2ONet $^{\dagger}$ + HFL-Net $^{\dagger}$	16.43	6.93	15.94	6.79	18.76	7.29	18.25	7.14	31.02	8.55	29.91	8.45
HandOccNet $^{\dagger}$ + HFL-Net $^{\dagger}$	16.04	6.89	15.53	6.65	19.22	7.43	18.64	7.20	29.57	8.58	28.53	8.31
HandOccNet $^{\dagger}$ + HFL-Net $^{\dagger}$	16.41	7.00	15.88	6.77	18.64	7.33	18.09	7.10	29.22	8.76	28.20	8.47
UniHOPE (ours)	14.59	6.39	14.13	6.17	16.27	6.51	15.78	6.29	26.42	7.64	25.51	7.40

Table 6. Comparison with SOTA methods across different object-caused occlusion levels on DexYCB.

	Models	Root-relative		Procrustes Align.
	Models	J-PE $\downarrow$	V-PE $\downarrow$	J-PE $\downarrow$	V-PE $\downarrow$
(a)	Baseline	14.09	13.61	5.95	5.75
(b)	w/ Grasp-aware Feature Fusion	13.84	13.37	5.79	5.58
(c)	w/ Generative De-occluder	13.38	12.92	5.71	5.52
(d)	+ Image Feature Enhancement	13.23	12.79	5.64	5.44
(e)	+ RoI Feature Enhancement	13.18	12.73	5.64	5.45
(f)	+ MANO Feature Enhancement	13.12	12.67	5.63	5.43
(g)	+ Occlusion-aware Case Filtering	13.03	12.59	5.59	5.40

Table 7. Ablation study on major designs in UniHOPE.

#### 4.3. Ablation Studies We perform ablation studies on DexYCB to evaluate the effectiveness of our designs, as shown in Tab. 7. HFL-Net [^31] with our object switcher serves as the simplest baseline. Grasp-aware Feature Fusion. We first analyze the impact of the grasp-aware feature fusion module. Comparison of Rows (a-b) shows performance boosts across all metrics, indicating that integrating irrelevant object features affects hand pose estimation and our design alleviates this issue. Generative De-occluder. Next, we assess the effects of de-occluded hand images with in-distribution hand poses, as they provide extra information. Comparing Rows (b-c), the notable improvement upon incorporating paired data indicates the effectiveness of synthetic samples. More details on our adaptive control strength adjustment are in the Supp. Occlusion-invariant Feature Learning. Further, we show the individual effects of feature enhancement at various levels. Comparing Row (c) with (d-f), we note a progressive improvement in root-relative metrics at each level, showing the efficacy of knowledge transferring from de-occluded hands. Finally, Row (g) reveals that the effects are maximally realized through our occlusion-aware case filtering. ### 5. Conclusion We introduce UniHOPE, the first unified approach for hand-only and hand-object pose estimation, motivated by the inability of existing methods to handle both scenes. Our technical innovations are twofold: first, to enable flexibility in switching between different scenes, we incorporate an object switcher to control object-pose estimation and design a grasping-aware feature fusion module to selectively capture effective object features; second, to promote robustness against object-caused occlusion, we propose multi-level feature enhancement to learn occlusion-invariant hand features from generated realistic de-occluded hand images. Experimental results on three common benchmarks manifest the SOTA performance of UniHOPE. --- Acknowledgments This work is supported in part by the Research Grants Council of the Hong Kong Special Administrative Region, China, under Project T45-401/22-N and by the Hong Kong Innovation and Technology Fund, under Project MHP/086/21. ## References [^1]: Autodesk, INC. Maya. , 2018. 2 [^2]: Seungryul Baek, Kwang In Kim, and Tae-Kyun Kim. Pushing the envelope for RGB-based dense 3D hand pose estimation via neural rendering. In CVPR, pages 1067–1076, 2019. 2 [^3]: Seungryul Baek, Kwang In Kim, and Tae-Kyun Kim. Weakly-supervised domain adaptation via GAN and mesh model for estimating 3D hand poses interacting objects. In CVPR, pages 6121–6131, 2020. [^4]: Adnane Boukhayma, Rodrigo de Bem, and Philip H.S. Torr. 3D hand shape and pose from images in the wild. In CVPR, pages 10843–10852, 2019. 2 [^5]: Zhe Cao, Ilija Radosavovic, Angjoo Kanazawa, and Jitendra Malik. Reconstructing hand-object interactions in the wild. In ICCV, pages 12417–12426, 2021. 1 [^6]: Yu-Wei Chao, Wei Yang, Yu Xiang, Pavlo Molchanov, Ankur Handa, Jonathan Tremblay, Yashraj S. Narang, Karl Van Wyk, Umar Iqbal, Stan Birchfield, et al. DexYCB: A benchmark for capturing hand grasping of objects. In CVPR, pages 9044–9053, 2021. 3, 6 [^7]: Guobin Chen, Wongun Choi, Xiang Yu, Tony Han, and Manmohan Chandraker. Learning efficient object detection models with knowledge distillation. NeurIPS, 30, 2017. 5 [^8]: Xingyu Chen, Yufeng Liu, Chongyang Ma, Jianlong Chang, Huayan Wang, Tian Chen, Xiaoyan Guo, Pengfei Wan, and Wen Zheng. Camera-space hand mesh recovery via semantic aggregation and adaptive 2D-1D registration. In CVPR, pages 13274–13283, 2021. 2 [^9]: Xingyu Chen, Yufeng Liu, Yajiao Dong, Xiong Zhang, Chongyang Ma, Yanmin Xiong, Yuan Zhang, and Xiaoyan Guo. MobRecon: Mobile-friendly hand mesh reconstruction from monocular image. In CVPR, pages 20544–20554, 2022. 1, 2, 4, 6, 8 [^10]: Yujin Chen, Zhigang Tu, Di Kang, Linchao Bao, Ying Zhang, Xuefei Zhe, Ruizhi Chen, and Junsong Yuan. Model-based 3D hand reconstruction via self-supervised learning. In CVPR, pages 10451–10460, 2021. 2 [^11]: Zerui Chen, Yana Hasson, Cordelia Schmid, and Ivan Laptev. AlignSDF: Pose-aligned signed distance fields for hand-object reconstruction. In ECCV, pages 231–248. Springer, 2022. 2 [^12]: Zerui Chen, Shizhe Chen, Cordelia Schmid, and Ivan Laptev. gSDF: Geometry-driven signed distance functions for 3D hand-object reconstruction. In CVPR, pages 12890–12900, 2023. 2 [^13]: Hongsuk Choi, Nikhil Chavan-Dafle, Jiacheng Yuan, Volkan Isler, and Hyunsoo Park. HandNeRF: Learning to reconstruct hand-object interaction scene from a single RGB image. In ICRA, pages 13940–13946. IEEE, 2024. 2 [^14]: Blender Online Community. Blender. , 2019. 2 [^15]: Enric Corona, Albert Pumarola, Guillem Alenya, Francesc Moreno-Noguer, and Grégory Rogez. GanHand: Predicting human grasp affordances in multi-object scenes. In CVPR, pages 5031–5041, 2020. 2 [^16]: Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11):139–144, 2020. 2 [^17]: Shreyas Hampali, Mahdi Rad, Markus Oberweger, and Vincent Lepetit. HOnnotate: A method for 3D annotation of hand and object poses. In CVPR, pages 3196–3206, 2020. 3, 6, 7 [^18]: Shreyas Hampali, Sayan Deb Sarkar, Mahdi Rad, and Vincent Lepetit. Keypoint transformer: Solving joint identification in challenging hands and object interactions for accurate 3D pose estimation. In CVPR, pages 11090–11100, 2022. 1, 2, 6, 8 [^19]: Yana Hasson, Güil Varol, Dimitrios Tzionas, Igor Kalevatykh, Michael J. Black, Ivan Laptev, and Cordelia Schmid. Learning joint reconstruction of hands and manipulated objects. In CVPR, pages 11807–11816, 2019. 1,2 [^20]: Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask R-CNN. In ICCV, pages 2961–2969, 2017. 4 [^21]: Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurIPS, pages 6840–6851, 2020. 2, 4 [^22]: Hezhen Hu, Weilun Wang, Wengang Zhou, and Houqiang Li. Hand-object interaction image generation. NeurIPS, 35:23805–23817, 2022. 3 [^23]: Yinlin Hu, Pascal Fua, and Mathieu Salzmann. Perspective flow aggregation for data-limited 6D object pose estimation. In ECCV, pages 89–106. Springer, 2022. 6 [^24]: Di Huang, Xiaopeng Ji, Xingyi He, Jiaming Sun, Tong He, Qing Shuai, Wanli Ouyang, and Xiaowei Zhou. Reconstructing hand-held objects from monocular video. In SIGGRAPH Asia 2022 Conference Papers, pages 1–9, 2022. 2 [^25]: Lin Huang, Chung-Ching Lin, Kevin Lin, Lin Liang, Lijuan Wang, Junsong Yuan, and Zicheng Liu. Neural voting field for camera-space 3D hand pose estimation. In CVPR, pages 8969–8978, 2023. 1,2 [^26]: Umar Iqbal, Pavlo Molchanov, Thomas Breuel Juergen Gall, and Jan Kautz. Hand pose estimation via latent 2.5D heatmap regression. In ECCV, pages 118–134, 2018. 2 [^27]: Diederik P. Kingma. Auto-encoding variational Bayes. arXiv preprint arXiv:1312.6114, 2013. 4 [^28]: Diederik P. Kingma and Lei Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015. 6 [^29]: Dominik Kulon, Riza Alp Guler, Iasonas Kokkinos, Michael M. Bronstein, and Stefanos Zafeiriou. Weakly-supervised mesh-convolutional hand reconstruction in the wild. In CVPR, pages 4990–5000, 2020. 2 [^30]: Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In CVPR, pages 2117–2125, 2017. 3 --- [^31]: Zhifeng Lin, Changxing Ding, Huan Yao, Zengsheng Kuang, and Shaoli Huang. Harmonious feature learning for interactive hand-object pose estimation. In CVPR, pages 12989–12998, 2023. 1, 2, 4, 5, 6, 7, 8 [^32]: Shaowei Liu, Hanwen Jiang, Jiarui Xu, Sifei Liu, and Xiaolong Wang. Semi-supervised 3D hand-object poses estimation with interactions in time. In CVPR, pages 14687–14697, 2021. 1, 2, 4, 6, 8 [^33]: Wenquan Lu, Yufei Xu, Jing Zhang, Chaoyue Wang, and Dacheng Tao. HandRefiner: Refining malformed hands in generated images by diffusion-based conditional inpainting. In ACM MM, 2024. 3, 4 [^34]: Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In CVPR, pages 11461–11471, 2022. 4 [^35]: Matthew Matl. Pyrender. , 2019. 2 [^36]: Gyeongsik Moon and Kyoung Mu Lee. I2L-MeshNet: Image-to-pixel prediction network for accurate 3D human pose and mesh estimation from a single RGB image. In ECCV, pages 752–768, 2020. 1, 2 [^37]: Gyeongsik Moon, Shoou-I Yu, He Wen, Takaaki Shiratori, and Kyoung Mu Lee. InterHand2.6M: A dataset and baseline for 3D interacting hand pose estimation from a single RGB image. In ECCV, pages 548–564, 2020. 2 [^38]: Supreeth Narasimhaswamy, Uttaran Bhattacharya, Xiang Chen, Ishita Dasgupta, Saayan Mitra, and Minh Hoai. HanDiffuser: Text-to-image generation with realistic hand appearances. In CVPR, pages 2468–2479, 2024. 3 [^39]: JoonKyu Park, Yeonguk Oh, Gyeongsik Moon, Hongsuk Choi, and Kyoung Mu Lee. HandOccNet: Occlusion-robust 3D hand mesh estimation network. In CVPR, pages 1496–1505, 2022. 1, 2, 6, 7, 8 [^40]: Junho Park, Kyeongbo Kong, and Suk-Ju Kang. AttentionHand: Text-driven controllable hand image generation for 3D hand reconstruction in the wild. arXiv preprint arXiv:2407.18034, 2024. 3 [^41]: Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In CVPR, 2016. 5 [^42]: Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, pages 10684–10695, 2022. 2, 4 [^43]: Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In ICML, pages 2256–2265, 2015. [^44]: Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020. 2, 4 [^45]: Tze Ho Elden Tse, Kwang In Kim, Ales Leonardis, and Hyung Jin Chang. Collaborative learning for hand and object reconstruction with attention-guided graph convolution. In CVPR, pages 1664–1674, 2022. 2, 4 [^46]: Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. NeurIPS, pages 6000–6010, 2017. 4 [^47]: Rong Wang, Wei Mao, and Hongdong Li. Interacting hand-object pose estimation via dense mutual attention. In WACV, pages 5735–5745, 2023. 1 [^48]: Yinhui Wang, Jiwen Yu, and Jian Zhang. Zero-shot image restoration using denoising diffusion null-space model. ICLR, 2023. 2 [^49]: Hao Xu, Tianyu Wang, Xiao Tang, and Chi-Wing Fu. H2ONet: Hand-occlusion-and-orientation-aware network for real-time 3D hand mesh reconstruction. In CVPR, pages 17048–17058, 2023. 1, 2, 6, 7, 8 [^50]: Hao Xu, Haipeng Li, Yinqiao Wang, Shuaicheng Liu, and Chi-Wing Fu. HandBooster: Boosting 3D hand-mesh reconstruction by conditional synthesis and sampling of hand-object interactions. In CVPR, pages 10159–10169, 2024. 3 [^51]: Lixin Yang, Jiasen Li, Wenqiang Xu, Yiqun Diao, and Cewu Lu. BiHand: Recovering hand mesh with multi-stage bisected hourglass networks. In BMVC, 2020. 2 [^52]: Linlin Yang, Shicheng Chen, and Angela Yao. SemiHand: Semi-supervised hand pose estimation with consistency. In ICCV, pages 11364–11373, 2021. 2 [^53]: Lixin Yang, Kailin Li, Xinyu Zhan, Jun Lv, Wenqiang Xu, Jiefeng Li, and Cewu Lu. ArtiBoost: Boosting articulated 3D hand-object pose estimation via online exploration and synthesis. In CVPR, pages 2750–2760, 2022. 2 [^54]: Yufei Ye, Abhinav Gupta, and Shubham Tulsiani. What’s in your hands? 3D reconstruction of generic objects in hands. In CVPR, pages 3895–3905, 2022. 2 [^55]: Yufei Ye, Xueting Li, Abhinav Gupta, Shalini De Mello, Stan Birchfield, Jiaming Song, Shubham Tulsiani, and Sifei Liu. Affordance diffusion: Synthesizing hand-object interactions. In CVPR, pages 22479–22489, 2023. 3 [^56]: Yufei Ye, Abhinav Gupta, Kris Kitani, and Shubham Tulsiani. G-HOP: Generative hand-object prior for interaction reconstruction and grasp synthesis. In CVPR, pages 1911–1920, 2024. 2 [^57]: Baowen Zhang, Yangang Wang, Xiaoming Deng, Yinda Zhang, Ping Tan, Cuixia Ma, and Hongan Wang. Interacting two-hand 3D pose and shape reconstruction from single color image. In ICCV, pages 11354–11363, 2021. 2 [^58]: Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In ICCV, pages 3836–3847, 2023. 4 [^59]: Mengqi Zhang, Yang Fu, Zheng Ding, Sifei Liu, Zhuowen Tu, and Xiaolong Wang. HOIDiffusion: Generating realistic 3D hand-object interaction data. In CVPR, pages 8521–8531, 2024. 3 [^60]: Xiong Zhang, Qiang Li, Hong Mo, Wenbo Zhang, and Wen Zheng. End-to-end hand mesh recovery from a monocular RGB image. In ICCV, pages 2354–2364, 2019. 2 [^61]: Xiong Zhang, Hongsheng Huang, Jianchao Tan, Hongmin Xu, Cheng Yang, Guozhu Peng, Lei Wang, and Ji Liu. Hand image understanding via deep multi-task learning. In ICCV, pages 11281–11292, 2021. --- [^62]: Zimeng Zhao, Xi Zhao, and Yangang Wang. TravelNet: Self-supervised physically plausible hand motion learning from monocular color images. In ICCV, pages 11666–11676, 2021. [^63]: Yuxiao Zhou, Marc Habermann, Weipeng Xu, Ikhsanul Habibie, Christian Theobalt, and Feng Xu. Monocular real-time hand shape and motion capture using multi-modal data. In CVPR, pages 5346–5355, 2020. 2 [^64]: Zhishan Zhou, Shihao Zhou, Zhi Lv, Minqiang Zou, Yao Tang, and Jiajun Liang. A simple baseline for efficient hand mesh reconstruction. In CVPR, pages 1367–1376, 2024. 1, 2, 6, 8 [^65]: Christian Zimmermann, Duygu Ceylan, Jimei Yang, Bryan Russell, Max Argus, and Thomas Brox. FreiHAND: A dataset for markerless capture of hand pose and shape from single RGB images. In ICCV, pages 813–822, 2019. 2, 6 --- # UniHOPE: A Unified Approach for Hand-Only and Hand-Object Pose Estimation Supplementary Material In this supplementary material, we provide more qualitative and quantitative results to show the capabilities and robustness of UniHOPE (Sec. A). In Sec. B, we present the implementation details and in Sec. C, we discuss the limitations and future work. ### A. More Experimental Results #### A.1. Qualitative Results First of all, we present Figs. A to D, which show that UniHOPE is able to handle both hand-only scenario (left columns) and hand-object scenario (right columns). Comparison with SOTA Methods. Next, we provide more qualitative comparisons on the DexYCB (Fig. E), HO3D (Fig. F), and FreiHAND datasets (Fig. G). More De-occluded Examples. Furthermore, we present more de-occluded samples in Fig. H. #### A.2. Quantitative Results Additional Results of Tab. 1. The additional metrics of Tab. 1 in the main paper are provided in Tab. A. Both the metrics before & after PA show an overall performance degeneration of existing HPE/HOPE models when transferring to apply to the other scenario or testing in the original scenario even after re-training on both scenes. Comparison on Other Splits of DexYCB. We provide the quantitative results of hand pose estimation on the default “S0” split (same distribution for the training and test set) and “S1” split with unseen subjects (train/test: 7/2 subjects) of DexYCB in Tab. B and Tab. C, respectively. Our method achieves the best overall performance, especially in root-relative metrics. Comparison on HO3D. The remaining hand metrics on HO3D are reported in Tab. D. Though HFL-Net [^9] and the combination of H2ONet + HFL-Net achieve better PA results, our method outperforms them by a large margin in the metrics after scale-translation only alignment [^4], which takes both the global rotation and hand shape into consideration. We emphasize the importance of global rotation, since it better reflects the visualization quality, as indicated by the qualitative comparison results shown in Fig. F. #### A.3. Detailed Analysis on Performance In this work, we explore a new setting to address HPE and HOPE at once. Applying prior SOTA of HPE/HOPE is suboptimal, even re-trained on all scenarios, as they lack specific designs. For hand-only scenes, HOPE methods are affected by irrelevant object features, even no object is grasped, yet HPE methods may fail for unseen hand poses. For hand-object scenes, HOPE methods lack effective designs to handle severe occlusions, while HPE methods do not utilize object information to enhance performance. Our approach works better in each scene type. As Fig. I shows: (a) when the hand reaches out to grasp an object, our grasp-aware feature fusion reduces the adverse impact of non-grasped object; (b) for unseen hand poses from FreiHAND, our generated de-occluded images introduce richer hand poses to boost performance; (c) our multi-level feature enhancement improves robustness under severe object occlusions; and (d) when grasping objects, our method surpasses HPE methods by leveraging object information. These observations are consistent with the quantitative performance in Tab. 2, 5, 6 in the main paper. #### A.4. Additional Ablation Studies To be consistent with the main paper, we conduct all the ablation studies presented below on DexYCB. Additional Results of Tab. 7. Since the RHD [^22] and Static Gestures Dataset [^1] are utilized to fine-tune the ControlNet [^12], we also conduct an ablation study of pre-training on these synthetic datasets before training on DexYCB, using a network structure identical to our baseline model with the grasp-aware feature fusion module (Row (b) of Tab. 7 in the main paper). As shown in Tab. F, directly incorporating synthetic datasets into training leads to a minor improvement, indicating the limitation caused by the domain gap between the synthetic and real-world images. Conversely, our occlusion-invariant feature learning strategy substantially enhances the model performance through the foundational data prior provided by ControlNet [^20] and the multi-level feature enhancement. Ablation on Adaptive Control Strength Adjustment. Control strength (ranging from 0 to 1) is imposed on the connections between the ControlNet and Stable Diffusion, controlling the extent to which the output is consistent with the control signal. We propose to adaptively adjust its value with MobRecon [^3] pre-trained on DexYCB to avoid tedious manual tuning. The default control strength employed in [^12] is 0.55. In our work, we empirically select the candidate control strengths from $\{0.25, 0.4, 0.55, 0.7, 0.85, 1.0\}$, with a similar number of candidates as in [^12]. ---

Hand-Only Scene			Hand-Object Scene
![Image](images/5ee69c069870f5118c9bddc8baa5d5c098865efb92ecd9809cbf88cc4903698a.jpg)	![Image](images/c6c7874f0636a3664fe57f6ac8baddf4bfa26a7e7f9da544ae94f622dc5ad08d.jpg)		![Image](images/bd10b677f5b80284626ef4242f7c57ba8d08af314aaf690ca09f934a1ba3b35c.jpg)	![Image](images/2569eb35ab750b2a8bf58ae55c36f61374d263a5d64883a497513f434d922f01.jpg)	![Image](images/39daab4eba8a8dfadd00e87eb3bff39e741b4ebfb0e298953598f814b355af35.jpg)
![Image](images/a2c763ccc853d64c623903b92816e64bdbd115d96a8712e6927ca4ddf7932e73.jpg)	![Image](images/5efe5c361c3138d94c83aa54bc3731b02dbea3745875a7a22360cdd7c1f5690f.jpg)	![Image](images/e05f9d0a979d1a263d6b0040861c3ec63a64cef7cc31bf879aaab39325a95097.jpg)	![Image](images/faf27b8f46cdaba05d0f0bd2055adaac112307eb2139bf221b979e515d1a90f4.jpg)	![Image](images/3824d72bfd2cd1ca3eaee97ae7826b8c6076a8fca20ae6df4e3a449150630eeb.jpg)	![Image](images/b411a4e744e0b12e305c2723b75a0cdaeea8706bc215a20aab466f1447e0dbb5.jpg)
![Image](images/b9c92fe89792d0a3cc6eaa9d8450e7d3022125463f84f3e8e8547bd7bf67696e.jpg)	![Image](images/93e27e54e5ffeaea5d36cb7886219362201e71d806ebb71e18e76975d96df3fb.jpg)	![Image](images/9a27d1cba12524602a3b96f0e5c7515d87bc5af9542e6b9b49db674bfe4b2a8c.jpg)	![Image](images/054dedddfa97ff7c8681c5473d2cdd9acb9cf9adbcb11cd353e99528aa67599d.jpg)	![Image](images/0e061168cc6c10ac64ba5fc2cb3aa90eb4618b20620b8f5609c127f715d29f62.jpg)	![Image](images/f5ab0adb7e0c6ca2d2ba357b72b0d6b0e58aebf4e95c6f9008dde18a7dd07e4a.jpg)
![Image](images/1ba52f01c63426d2ac2e709b2a3adf1b60fc6d1d8ad72a8af997ad13a452f412.jpg)	![Image](images/19eed946acfa46f9b433ab4d9f4f178a94f22caef2e76844ebaced2f45194052.jpg)	![Image](images/0890fb4e68a913982a3a34610ab390d35190d363e5ae2cbf91f6da9c6c353b3f.jpg)	![Image](images/220ff67f9181feb81d8243453a9b48cc6f361f5ce3a5b86a295ae300e5c54e7a.jpg)	![Image](images/4fe518958e356a243be1710e844b518465f0d1cdbb380007e2139d59873b0a20.jpg)	![Image](images/6d57d67141338804e407736fbd232be17e42fab2c8460a67ebf45fd67869a72b.jpg)
![Image](images/17a40413144e9b95b1f696acc77773a3fd9de696e8445c759d6fc632519af12f.jpg)	![Image](images/33736e412113a5cb2c7f7a5798d761774b99c839e4c78d1231a09497b76abca5.jpg)	![Image](images/4f06e4b868f4d56a49e32507bcc96ef18e973335c7ecbd9f796b99e04450b5a3.jpg)	![Image](images/e86979569888fb9a6370e58e7277d7a54ab1463b95e73695667efd80bc7661ff.jpg)	![Image](images/af1b8c49e87bcb671fd43268c44e78975edce536f15c9e324c679a0c899b6e0a.jpg)	![Image](images/1f313fb2bd8504a16bc19a6157a4e1e732361d5bf0ae8e5daf7ebf1ef1fb8c2f.jpg)
Input	Output (view 1)	Output (view 2)	Input	Output (view 1)	Output (view 2)

Figure A. UniHOPE is able to handle both hand-only (left column) and hand-object scenarios (right column). Here, we show more qualitative results on DexYCB. For each example, the estimation results are rendered from the original (view 1) and another view (view 2) for clear visualization.

---

Hand-Only Scene

![Image](images/c958f0ac7e81bee4c0d81ae0c22c85c2d6ac3ebdb09f2529922752973cf324ef.jpg)	![Image](images/8789dc49623a41aeb80d65c854cd6f2cb42f8e73bada3d9be508c5ce201f6e7d.jpg)	![Image](images/e2c95a726c1099d6434966dc4578a32413c8f9efb46a908f4eb263c8e716c5af.jpg)	![Image](images/e897af396696e8bd69f6bfcff1c5cb76096eba1e9685f6e0407e3f537660bf50.jpg)	![Image](images/3690edd6bdc80762aa7f3630f9f625013609f296a271fb813c57b11c1f5d7ab9.jpg)
![Image](images/8da7856e454a5412d51cc8c55e806746be636c7cdf2b7fb1f614c6bf823bef97.jpg)	![Image](images/8f63c7131caecb015b132f1e6f7787cf5c14aea5ba96156f311918f7430ec00a.jpg)	![Image](images/08b52398ae4d0ec686ec6945e591444a5d16105642046e44c62f0e91c671df1b.jpg)	![Image](images/239cabb47b12925b0b765d5e4d481e3f264986ad2160afda52ed69f445de4eb6.jpg)	![Image](images/f153d78bc3091e33ded26fd52595dc66ed137ed7e268a78bd1c39db438425d3e.jpg)
![Image](images/029bbf92fa0baf9ba5f0e574db4d2780686cce233675b48b2e76cde53b6e67cd.jpg)	[F: [F8]	![Image](images/dd21cb0adfffeb44f096f1faf5f3de0e7cfa2f4b8621f675a25537afd47ebbbd.jpg)	![Image](images/e48807c76750f6cd1c803ba2733e114f369cdbf349ee5a4023b75f0b0155ac0e.jpg)	![Image](images/a00e6720d7ed21d19066c85d73f91155aba151b7826c7dc6dbfd3fc09adcd615.jpg)
![Image](images/3d86b2989bce683284d6fc3076263e55246588dadc58834df669c16e320850f5.jpg)	![Image](images/15790d24e7d2218ad802f221522c448d018e01b0b6572329f07da8135ebabbca.jpg)	![Image](images/5105f140aead30d959ba5c53b351d24a6649dcfe8622b65a9b9887ad16e2f056.jpg)	![Image](images/fcc56a26b91fa0c43df24d3c99091209f1728ee453a7147aa67a1f2643396df3.jpg)	![Image](images/946c317866de5eefadc6723f7ee6ab722ee9f8260d74a311f33629c78d41db22.jpg)
![Image](images/a1ef879eb55490ed1812458de251e65a5626e62c66e98a361af2ccf8dd4d55eb.jpg)	![Image](images/f4c19744b634a1a72ade8142df7c99fd00034fe4f491f17a6f50c2aeec4df657.jpg)	![Image](images/4bc0ca30c8c5f7896fa94ab463c0f0833cb9472af7f349b027efe7178250b3b2.jpg)	![Image](images/d4664d87a74714536d50417457c2b532e7cd70c5c17383ef2bec1927a7282e08.jpg)	![Image](images/3261264f3393e3592112e57104341de06c40dcf1750af32684753f0d4c57c2f3.jpg)

Hand-Object Scene

Input Output (view 1) Output (view 2) Input Output (view 1) Output (view 2)

Figure B. More qualitative results of UniHOPE on DexYCB.

---

Hand-Only Scene	Hand-Object Scene
![Image](images/1376909366b264f17ffab8fe0008b79514e07ae1fe3de139e41ea7d58387ae4b.jpg) ![Image](images/b63b195bea4d3f3794ac3e79afec130ecd41b121bcd8872b3aa47a82ac670a93.jpg)	![Image](images/edc21350dbb7751617269f1404da1c3b878d00c870ef16c4d7dde2246033e5d3.jpg) ![Image](images/79adad6c50dd892d53d56ae84ca43bca3e190053d4dccbbb552a7614dede8af6.jpg) ![Image](images/95e4303f688b546502fdf3d748e79061e7ac788b78bb740b70a6c39b8178d713.jpg)
![Image](images/be9fada8f3f217eb3902cdf6ef6982362ef32a0b49a5ee9e5fef05fb2a398dd2.jpg) ![Image](images/f8ed3795d0587bff26b11d7c67bc50b23b2b00bdece91846c66f6d91922043b3.jpg)	![Image](images/18845b5b2d1452ea985b8347c58011c072c7c5d4418f3ab7c498673b6b34d149.jpg) ![Image](images/ef88618fcda73b0d85c4197a4b43ae81130929036914eba4be786e967197ea49.jpg) ![Image](images/d0b352d46551ba7d082d1ff05157f3eaaf44cbd821fc7aaf813a6d748448555d.jpg)
![Image](images/3001f508a9c9b92a3f79c3e6f11b81c8b50e3953ff2089a1ed1847ad850982c4.jpg) ![Image](images/3af1bd19ffd4746af5aca38f03d51f4e6e1af9ebf4809d9bbaa776cbc4671034.jpg) ![Image](images/40eaae80e25fae727ca11f4f20823af2e28b7c352aac6f245de97128f2526992.jpg)	![Image](images/6915aef7fc95f0555899a6d8ecb5818edeaa22127795cbbcdf74a274be3fd791.jpg) ![Image](images/5f7f4fcc85f6499a28bac2c5ea4817ab30240c980eb4f52449845b4953ddad4d.jpg) ![Image](images/fca920c1e0237b5c94973172f8ed9c74592a0f09b7a0974d7dba611c58221ae0.jpg)
![Image](images/30899a98629026a4ef6c731332c925be08efd51128e41ecd3bba66a2a987e59f.jpg)	![Image](images/ae68942926afdac751c1bdb81fd1994ff9bba41c9303cd350486989de76bb6ba.jpg) ![Image](images/965fed0401833a8d28759f4f2628862fc0326f44864942436bfd9011587928a8.jpg) ![Image](images/4ee4d5756101b86dc04e037c6de2c9128116e08f09f8caa6f225a10511d8801d.jpg)
![Image](images/78d38a81e351f25995515f58d2dd8b9021b1d5adf502182ea47da044cec84f55.jpg) ![Image](images/70971528431402c68e8a8e3d65df1c64b9d457decd76e691f5b494431e974771.jpg) ![Image](images/8d08f0845b0e83c732015bdb3fe47d0839aa8bb53b2c2b7f4e5910f578b0b3f0.jpg)	![Image](images/08e29797b55fa7d791a6cc36c27dae595cba16eacd4d32984ddf15593617289b.jpg) ![Image](images/6742aa9e8c436a05ca4ed3b256a51f78bd52ceef0e59647369abd28eebe51bb0.jpg) ![Image](images/8c65f303ad5cc395ac1d7f28bd927574188167b3b3a147b4ca0cd450ff480023.jpg)
Input Output (view 1)	Output (view 2)

Figure C. More qualitative results of UniHOPE across hand-only (left column) and hand-object scenarios (right column) on HO3D.

--- ## Hand-Only Scene

![Image](images/80bcf885e4cb023d287da36cc29e53eb5707b5e2da61dc55a3470947f266ded7.jpg)	![Image](images/19bce30635f74b1296f1736a98b24b0e915c5dd3ef9e6660c610e27ee3db1031.jpg)	![Image](images/d7bcb4dde55b072c6616085c1c70071e231dbdc674207f06ec246f496e65e799.jpg)	![Image](images/a8605fb09e96d5eb3908d80e7dca6a6c8f7006f76024f2b94c1922a0988cf505.jpg)
![Image](images/3a2e9bd3d43f77775adc9df39733f42d29116b9a535776d2bc34f4c075d17b5b.jpg)		![Image](images/de0122711fd59a8211fe2a0de5f08eaa90938d59a63fa0ad4e6dc717fad4efab.jpg)	![Image](images/96994bf92a2b8798ebb1442ec4f5a1649d43af7dc032299adb5dfee4a269efb3.jpg) ![Image](images/cbfff91939f9f52ad91965b48794940cf218e824046f262ea5d59e5bfdcb233d.jpg)
![Image](images/1a5beb6f5e952cb230058ab877ea7cb2a25d24e903f9466671862c93510f6bd5.jpg)	![Image](images/5853d679460478c2123a62a688ac1a554916355fc09a9d071ed7a7abc77aca30.jpg)	![Image](images/fefef0a3a60d73c9afe403a547c6f2f6d95ef5575e8944b40817b5b34047ae26.jpg)	[F3]
![Image](images/6a541e4e50f1d76ea58735f48b7242d19d7718b7a1cd3908a49f5bd0973d3869.jpg)	![Image](images/000f35b804c337c85745301087d1849912e780c1bed6544e8d0c49747ec67779.jpg)	![Image](images/7a53ef5e94e7c0b8e5b39eeff155ff4924a53c7aa5eb20e30293b643997ee19e.jpg)	![Image](images/8e04774e592a9931b3d4d58fa892b6afc87516b7f9d9e80f2dad5d0390c76eee.jpg)
![Image](images/bfd590ed5c1270685e12474de4a247c11fb18c845e38ef7eb316bf84c9abee38.jpg)	![Image](images/47b3dd28ced004e94fc99a9ed52ed8959d5bc27ce1b91a12ec837365a60a1598.jpg)	![Image](images/97fe3fa7f41ca4729fbf53ef72cb2e5dc945ac2d6f3a380da9e015743a58a7f8.jpg)	![Image](images/c8a0675478e748d49689be33aa3b36f14425bbb024c42f747c2cad66888abc4d.jpg)

## Hand-Object Scene Input Output (view 1) Output (view 2) Input Output (view 1) Output (view 2)

Figure D. More qualitative results of UniHOPE on HO3D.

---

HPE	Hand-Only Scene				Hand-Only → Hand-Object Scene					All → Hand-Only Scene					All → Hand-Object Scene
HPE	J-PE $\downarrow$	PA-J-PE $\downarrow$	V-PE $\downarrow$	PA-V-PE $\downarrow$	J-PE $\downarrow$	PA-J-PE $\downarrow$	V-PE $\downarrow$	PA-V-PE $\downarrow$	J-PE $\downarrow$	PA-J-PE $\downarrow$	V-PE $\downarrow$	PA-V-PE $\downarrow$	J-PE $\downarrow$	PA-V-PE $\downarrow$	V-PE $\downarrow$	PA-V-PE $\downarrow$	J-PE $\downarrow$	PA-J-PE $\downarrow$	V-PE $\downarrow$	PA-V-PE $\downarrow$
[^14]	12.98	5.21	12.52	5.02	19.60	(-6.62)	7.71 (-2.50)	18.95 (-6.43)	7.42 (-2.40)	13.16 (-0.18)	5.31 (-0.10)	12.70 (-0.18)	5.11 (-0.09)	14.58	6.73	14.10	6.49
[^18]	13.34	4.69	13.13	5.05	21.98	(-8.64)	7.13 (-2.44)	21.42 (-8.30)	7.27 (-2.22)	14.14 (-0.80)	4.74 (-0.05)	14.00 (-0.87)	5.35 (-0.30)	15.20	6.35	15.03	6.74
[^21]	14.05	5.55	13.51	5.31	18.37	(-4.32)	7.42 (-1.87)	17.54 (-4.03)	6.91 (-1.60)	14.63 (-0.58)	5.62 (-0.07)	13.96 (-0.45)	5.38 (-0.07)	14.88	6.74	14.21	6.45
HOPE	Hand-Object Scene				Hand-Object → Hand-Only Scene					All → Hand-Object Scene					All → Hand-Only Scene
HOPE	J-PE $\downarrow$	PA-J-PE $\downarrow$	V-PE $\downarrow$	PA-V-PE $\downarrow$	J-PE $\downarrow$	PA-J-PE $\downarrow$	V-PE $\downarrow$	PA-V-PE $\downarrow$	J-PE $\downarrow$	PA-J-PE $\downarrow$	V-PE $\downarrow$	PA-V-PE $\downarrow$	J-PE $\downarrow$	PA-J-PE $\downarrow$	V-PE $\downarrow$	PA-V-PE $\downarrow$	V-PE $\downarrow$	PA-V-PE $\downarrow$	V-PE $\downarrow$	PA-V-PE $\downarrow$
[^5]	17.99	7.68	17.57	7.88	25.10	(-7.11)	7.62 (+0.06)	24.40 (-6.83)	7.88 (-0.00)	18.79 (-1.00)	7.77 (-0.09)	18.35 (-0.78)	7.94 (-0.06)	19.75	7.59	19.26	7.98
[^9]	14.61	6.56	14.13	6.33	19.39 (-4.78)	5.96 (+0.60)	18.61 (-4.48)	5.75 (-0.58)	14.77 (-0.16)	6.64 (-0.08)	14.29 (-0.16)	6.41 (-0.08)	6.41 (-0.08)	13.61	5.20	13.10	5.01

Table A. Full metrics of Tab.1 in the main paper.

	Methods	All Scenes				Hand-Only Scene				Hand-Object Scene
	Methods	J-PE $\downarrow$	PA-J-PE $\downarrow$	V-PE $\downarrow$	PA-V-PE $\downarrow$	J-PE $\downarrow$	PA-J-PE $\downarrow$	V-PE $\downarrow$	PA-V-PE $\downarrow$	J-PE $\downarrow$	PA-J-PE $\downarrow$	V-PE $\downarrow$	PA-V-PE $\downarrow$
HPE	HandOccNet [^14]	13.04	5.85	12.61	5.65	13.42	5.39	12.95	5.20	12.79	6.15	12.39	5.95
	MobRecon [^3]	14.34	6.50	13.40	5.74	14.57	5.91	13.74	5.29	14.18	6.88	13.19	6.03
	H2ONet [^18]	13.89	5.38	13.56	5.52	14.10	4.84	13.75	5.02	13.76	5.73	13.43	5.84
	SimpleHand [^21]	13.66	6.02	13.14	5.78	14.48	5.67	13.95	5.46	13.13	6.24	12.62	5.99
HOPE	Liu et al. [^11]	14.06	5.75	13.57	5.58	14.87	5.47	14.33	5.30	13.53	5.93	13.08	5.75
	Keypoint Trans. [^5]	16.61	6.84	16.21	7.05	18.50	7.03	18.00	7.32	15.39	6.71	15.05	6.88
	HFL-Net [^9]	13.02	5.58	12.58	5.39	13.41	5.19	12.92	5.00	12.77	5.84	12.35	5.64
Unified	H2ONet $^{\dagger}$ + HFL-Net $^{\dagger}$	13.08	5.47	12.71	5.43	13.81	4.85	13.50	5.06	12.61	5.87	12.20	5.68
	H2ONet $^{\dagger}$ + HFL-Net $^{\dagger}$	13.30	5.45	12.91	5.40	14.09	4.85	13.74	5.02	12.79	5.83	12.37	5.64
	HandOccNet $^{\dagger}$ + HFL-Net $^{\dagger}$	13.32	5.73	12.87	5.54	14.40	5.50	13.89	5.30	12.63	5.89	12.22	5.69
	HandOccNet $^{\dagger}$ + HFL-Net $^{\dagger}$	13.43	5.71	12.97	5.51	14.41	5.49	13.90	5.30	12.80	5.85	12.38	5.65
	UniHOPE (ours)	12.59	5.54	12.17	5.36	12.84	5.02	12.38	4.85	12.42	5.88	12.03	5.69

Table B. Quantitative comparison on DexYCB "S0" split.

	Methods	All Scenes				Hand-Only scene				Hand-Object scene
	Methods	J-PE $\downarrow$	PA-J-PE $\downarrow$	V-PE $\downarrow$	PA-V-PE $\downarrow$	J-PE $\downarrow$	PA-J-PE $\downarrow$	V-PE $\downarrow$	PA-V-PE $\downarrow$	J-PE $\downarrow$	PA-J-PE $\downarrow$	V-PE $\downarrow$	PA-V-PE $\downarrow$
HPE	HandOccNet [^14]	18.33	6.95	17.70	6.71	19.70	6.01	18.95	5.81	17.57	7.47	17.02	7.21
	MobRecon [^3]	18.62	7.18	17.73	6.61	19.36	6.27	18.42	5.75	18.21	7.68	17.36	7.09
	H2ONet [^18]	18.40	6.40	17.90	6.57	18.92	5.44	18.36	5.70	18.11	6.93	17.64	7.05
	SimpleHand [^21]	17.38	6.82	16.81	6.73	18.86	6.02	18.14	5.92	16.57	7.26	16.08	7.17
HOPE	Liu et al. [^11]	17.82	6.46	17.19	6.25	19.12	5.89	18.36	5.69	17.10	6.77	16.54	6.55
	Keypoint Trans. [^5]	21.61	8.15	21.18	8.36	22.84	7.32	22.24	7.59	20.93	8.61	20.60	8.79
	HFL-Net [^9]	17.77	6.58	17.16	6.36	18.42	5.72	17.72	5.52	17.41	7.06	16.86	6.82
Unified	H2ONet $^{\dagger}$ + HFL-Net $^{\dagger}$	17.49	6.36	16.94	6.25	19.24	5.50	18.60	5.59	16.54	6.83	16.02	6.61
	H2ONet $^{\dagger}$ + HFL-Net $^{\dagger}$	17.96	6.48	17.41	6.42	18.92	5.45	18.35	5.69	17.44	7.05	16.89	6.82
	HandOccNet $^{\dagger}$ + HFL-Net $^{\dagger}$	17.84	6.53	17.22	6.31	20.18	5.95	19.39	5.75	16.55	6.85	16.03	6.62
	HandOccNet $^{\dagger}$ + HFL-Net $^{\dagger}$	18.63	6.73	17.99	6.50	20.72	6.11	19.91	5.91	17.48	7.06	16.93	6.83
	UniHOPE (ours)	16.84	6.42	16.25	6.20	17.80	5.50	17.11	5.30	16.31	6.93	15.79	6.70

Table C. Quantitative comparison on DexYCB "S1" split.

To assess the effectiveness of our adaptive control strength adjustment, we compare our model (Row (c) of Tab. 7 in the main paper) with the ones trained with generated samples under fixed control strengths without incorporating the feature enhancement constraints. As shown in Tab. E, our adaptive strategy achieves the best performance in hand pose estimation compared to several control strengths. The samples generated under all candidate control strengths are provided in Fig. J, showing the need to adaptively select control strength for different cases. Effects of Hyperparameters. The default value of hyperparameter $\alpha$ is empirically set to 10 in Eq. (11) of the main paper. This is to ensure a prediction accuracy over 95%. For the hyperparameters controlling the feature enhancement at three different levels, we evaluate their effects on the hand pose estimation performance in Tab. G. Since the MANO-level feature is a late-stage feature employed to directly regress the final hand pose, an adaption layer is deployed to improve the knowledge transfer. We set a larger value for $\gamma_{MANO}$ to aim to strongly enforce this feature adaptation process. In our experiments, the values for $\gamma_{init}$, $\gamma_{RoI}$, and $\gamma_{MANO}$ are set to 0.1, 0.1, and 0.5, respectively. #### A.5. Computational Cost and Efficiency The training time of our model is 3 days for DexYCB (376k samples) and 12 hours for HO3D (66k samples), respectively, on eight NVidia RTX 2080Ti GPUs. Tab. H reports the inference speed (FPS, tested on a sin- ---

	Methods	Procrustes Alignment						Scale-Translation Aligned
	Methods	J-PE $\downarrow$	J-AUC $\uparrow$	V-PE $\downarrow$	V-AUC $\uparrow$	F@5 $\uparrow$	F@15 $\uparrow$	J-PE $\downarrow$	J-AUC $\uparrow$
HPE	HandOccNet [^14]	10.26	7.95	10.21	79.61	50.61	94.47	28.18	49.28
	MobRecon [^3]	10.47	79.14	10.76	78.54	47.57	93.59	29.36	49.36
	H2ONet [^18]	9.52	80.97	9.60	80.81	52.62	95.09	29.67	48.53
	SimpleHand [^21]	11.28	77.66	11.58	77.05	45.78	91.74	28.41	49.32
HOPE	Liu et al. [^11]	9.46	81.12	9.39	81.25	54.93	95.64	28.44	49.79
	Keypoint Trans. [^5]	12.00	76.24	12.18	75.83	44.71	91.60	40.00	36.36
	HFL-Net [^9]	9.01	82.02	8.92	82.18	57.01	96.19	27.97	51.33
Unified	H2ONet $^{\dagger}$ + HFL-Net $^{\dagger}$	9.49	81.04	9.43	81.16	54.54	95.54	30.60	48.93
	H2ONet $^{\dagger}$ + HFL-Net $^{\ddagger}$	8.97	82.10	8.88	82.26	57.08	96.22	28.00	51.44
	HandOccNet $^{\dagger}$ + HFL-Net $^{\dagger}$	9.56	80.89	9.50	81.02	54.23	95.47	30.29	49.09
	HandOccNet $^{\ddagger}$ + HFL-Net $^{\ddagger}$	9.05	81.94	8.96	82.10	56.79	96.14	27.83	51.45
	UniHOPE (ours)	9.60	80.82	9.45	81.12	52.57	95.68	25.53	53.70

Table D. Quantitative comparison (Procrustes Alignment & Scale-Translation Aligned) on HO3D.

Control Strength Selection	Root-relative		Procrustes Align.
Control Strength Selection	J-PE $\downarrow$	V-PE $\downarrow$	J-PE $\downarrow$	V-PE $\downarrow$
s = 0.4	13.76	13.30	5.85	5.65
s = 0.55	13.51	13.06	5.78	5.57
s = 0.7	13.43	12.98	5.75	5.55
Adaptive Adjustment (ours)	13.38	12.92	5.71	5.52

Table E. Quantitative results of our adaptive control strength adjustment vs. fixed control strengths.

Models	Root-relative		Procrustes Align.
	J-PE $\downarrow$	V-PE $\downarrow$	J-PE $\downarrow$	V-PE $\downarrow$
Baseline	13.84	13.37	5.79	5.58
w/ Grasp-aware Feature Fusion
w/ RHD [^22] & Static Gestures [^1]	13.79	13.32	5.73	5.53
Ours	13.03	12.59	5.59	5.40

Table F. Comparison with directly training with synthetic datasets used by [^12].

$\gamma_{init}/\gamma_{RoI}/\gamma_{MANO}$	Root-relative		Procrustes Align.
$\gamma_{init}/\gamma_{RoI}/\gamma_{MANO}$	J-PE $\downarrow$	V-PE $\downarrow$	J-PE $\downarrow$	V-PE $\downarrow$
0.001/0.001/0.005	13.17	12.72	5.61	5.41
0.01/0.01/0.05	13.13	12.69	5.62	5.42
0.1/0.1/0.5 (ours)	13.03	12.59	5.59	5.40
1.0/1.0/5.0	13.15	12.70	5.70	5.50
10.0/10.0/50.0	14.13	13.65	6.08	5.87

Table G. Effects of various hyperparameters of the multi-level feature constraints.

gle NVidia RTX 2080Ti GPU), FLOPs, and number of parameters of various models. Thanks to the lightweight object switcher in UniHOPE, UniHOPE has similar inference efficiency and model complexity as HFL-Net [^9]. Compared to other SOTA models, UniHOPE has a moderate model size and running speeds, enabling real-time applications. ### B. Implementation Details Scene Division. Following [^19], the thresholds for RRE and RTE in grasping label preparation are $5^{\circ}$ and 10mm, respectively. An image is categorized into the hand-only scenes, if determined as non-grasping, otherwise hand-object scenes. The numbers of samples in the two scenes are shown in Tab. I. Note that although FreiHAND [^23] contains a small number of images interacting with objects in both training and test sets, it cannot be divided due to the lack of object annotations. Generative De-occluder. We adopt the officially-released pre-trained weights from [^12], which fine-tunes ControlNet with synthetic hand images [^1] [^22]. The hand-object mask is obtained by applying dilation on the render mask of the 3D hand and object to ensure the hand-object region is covered for repainting. Then, we crop the original input image in the training set centered on the hand-object region and resize it to $512 \times 512$. The hand-object image and the hand-object mask are fed into the inpainting Stable Diffusion model, conditioned by the hand depth map. Besides, we adopt the positive prompt “a hand grasping gesture, indoor, in the lab” for image generation from the two laboratory benchmarks [^2] [^4], and the negative prompt is similar to the one in [^12]. During inference, the number of reverse steps for DDIM is set to 50 by default. Network Structure. (i) Backbone: Following [^9], we adopt ResNet50 [^6] as the backbone to extract features from the input image, in which a dual stream structure is adopted to relieve the competition between hand features and object features. (ii) Hand Encoder: The hand encoder takes $F^{OH}$ as input, first using an hourglass network [^13] to regress a feature map and the heatmap of 2D hand joints. Then, they are fused via a convolution layer and an element-wise addition, followed by four residual blocks to yield a 1024-dimensional vector. (iii) MANO Decoder: It consists of two fully connected layers to predict the hand pose and ---

Methods	HandOccNet [^14]	MobRecon [^3]	H2ONet [^18]	SimpleHand [^21]	Liu et al. [^11]	Keypoint Trans. [^5]	HFL-Net [^9]	H2ONet + HFL-Net	HandOccNet + HFL-Net	Ours
FPS	48	78	62	41	51	33	43	36	30	44
FLOPs	15.48G	0.46G	0.74G	9.96G	39.44G	12.66G	10.01G	0.77G / 10.04G	15.51G / 10.04G	10.04G
# Param.	37.22M	8.23M	25.88M	48.89M	34.48M	52.79M	46.08M	72.26M	83.60M	46.38M

Table H. Efficiency comparison with previous methods. Note that FLOPs for the “A+B” methods depend on the predicted grasping status, therefore reported as “FLOPs of (classifier + A) / FLOPs of (classifier + B)”.

Datasets (splits)	Training Set			Test Set
Datasets (splits)	All Scenes	Hand-Only Scene	Hand-Object Scene	All Scenes	Hand-Only Scene	Hand-Object Scene
DexYCB "S0"	401,507	153,210	248,297	78,768	30,848	47,920
DexYCB "S1"	351,943	138,775	213,168	104,128	36,912	67,216
DexYCB "S3"	376,374	145,051	231,323	76,360	29,912	46,448
HO3D	66,034	5,595	60,439	11,524	2,971	8,553
FreiHAND	130,240	N/A	N/A	3,960	N/A	N/A

Table I. Number of samples in hand-only/hand-object scenes for different datasets (splits).

shape parameters of the MANO model from the feature produced by the hand encoder. (iv) Object Decoder: Following [^9], the feature after RoIAlign from the hand branch is fused with the one from the object branch through a cross-attention layer, to enhance the object feature learning. The fused feature is then forwarded through six convolutional layers to predict the 2D projections of the 3D object corner keypoints and corresponding confidence. In testing, the object pose is computed by the Perspective-n-Point (PnP) algorithm [^8] using the correspondence between the predicted 2D and the original 3D keypoints on the object mesh. Training Details. Following [^9], we perform data augmentation on the training samples, including random scaling ( $\pm20\%$), rotating ( $\pm180^{\circ}$), translating ( $\pm10\%$), and color jittering ( $\pm50\%$). Our training process consists of two stages. In the first stage, the de-occluded images are incorporated into training without the feature enhancement loss for 30 epochs to first adapt the model to the domain of the generated data. In the second stage, the network is additionally supervised by the enhancement constraints between the image pairs for another 40 epochs under the same setting. ### C. Limitations and Future Work Limitations. Though we are able to predict the grasping status of unseen objects, the performance of their pose estimation tends to degrade when the object shape/appearance varies largely, due to the limited object categories in the training data. Besides, despite being provided in most existing public benchmarks, the object annotations are lacking in certain datasets, limiting the applicability of our approach as they are required for scene division and inpainting masks. Future Work. To improve the model's generalizability towards unseen objects, a promising direction is to utilize the knowledge prior from the various vision foundation models [^7] [^10] [^15], which demonstrated remarkable performance in zero-shot scenarios. Another approach that we are considering $f^{n}$ the model's generalizability is to 'atic data by leveraging ## References [^1]: Synthesis AI. Static gestures dataset, data retrieved from synthesis ai. , 2023. 1, 7 [^2]: Yu-Wei Chao, Wei Yang, Yu Xiang, Pavlo Molchanov, Ankur Handa, Jonathan Tremblay, Yashraj S. Narang, Karl Van Wyk, Umar Iqbal, Stan Birchfield, et al. DexYCB: A benchmark for capturing hand grasping of objects. In CVPR, pages 9044–9053, 2021. 7 [^3]: Xingyu Chen, Yufeng Liu, Yajiao Dong, Xiong Zhang, Chongyang Ma, Yanmin Xiong, Yuan Zhang, and Xiaoyan Guo. MobRecon: Mobile-friendly hand mesh reconstruction from monocular image. In CVPR, pages 20544–20554, 2022. 1, 6, 7, 8 [^4]: Shreyas Hampali, Mahdi Rad, Markus Oberweger, and Vincent Lepetit. HOnnotate: A method for 3D annotation of hand and object poses. In CVPR, pages 3196–3206, 2020. 1, 7 [^5]: Shreyas Hampali, Sayan Deb Sarkar, Mahdi Rad, and Vincent Lepetit. Keypoint transformer: Solving joint identification in challenging hands and object interactions for accurate 3D pose estimation. In CVPR, pages 11090–11100, 2022. 6, 7, 8 [^6]: Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pages 770–778, 2016. 7 [^7]: Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, et al. Segment anything. In ICCV, pages 4015–4026, 2023. 8 [^8]: Vincent Lepetit, Francesc Moreno-Noguer, and Pascal Fua. --- EPnP: An accurate O(n) solution to the PnP problem. IJCV, 81:155–166, 2009. 8 [^9]: Zhifeng Lin, Changxing Ding, Huan Yao, Zengsheng Kuang, and Shaoli Huang. Harmonious feature learning for interactive hand-object pose estimation. In CVPR, pages 12989–12998, 2023. 1, 6, 7, 8 [^10]: Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. NeurIPS, 36, 2024. 8 [^11]: Shaowei Liu, Hanwen Jiang, Jiarui Xu, Sifei Liu, and Xiaolong Wang. Semi-supervised 3D hand-object poses estimation with interactions in time. In CVPR, pages 14687–14697, 2021. 6, 7, 8 [^12]: Wenquan Lu, Yufei Xu, Jing Zhang, Chaoyue Wang, and Dacheng Tao. HandRefiner: Refining malformed hands in generated images by diffusion-based conditional inpainting. In ACM MM, 2024. 1,7 [^13]: Alejandro Newell, Kaiyu Yang, and Jia Deng. Stacked hour-glass networks for human pose estimation. In ECCV, pages 483–499. Springer International Publishing, 2016. 7 [^14]: JoonKyu Park, Yeonguk Oh, Gyeongsik Moon, Hongsuk Choi, and Kyoung Mu Lee. HandOccNet: Occlusion-robust 3D hand mesh estimation network. In CVPR, pages 1496–1505, 2022. 6, 7, 8 [^15]: Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, pages 8748–8763. PMLR, 2021. 8 [^16]: Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. NeurIPS, pages 6000–6010, 2017. 8 [^17]: Bowen Wen, Wei Yang, Jan Kautz, and Stan Birchfield. FoundationPose: Unified 6D pose estimation and tracking of novel objects. In CVPR, pages 17868–17879, 2024. 8 [^18]: Hao Xu, Tianyu Wang, Xiao Tang, and Chi-Wing Fu. H2ONet: Hand-occlusion-and-orientation-aware network for real-time 3D hand mesh reconstruction. In CVPR, pages 17048–17058, 2023. 6, 7, 8 [^19]: Hao Xu, Haipeng Li, Yinqiao Wang, Shuaicheng Liu, and Chi-Wing Fu. HandBooster: Boosting 3D hand-mesh reconstruction by conditional synthesis and sampling of hand-object interactions. In CVPR, pages 10159–10169, 2024. 7, 8 [^20]: Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In ICCV, pages 3836–3847, 2023. 1 [^21]: Zhishan Zhou, Shihao Zhou, Zhi Lv, Minqiang Zou, Yao Tang, and Jiajun Liang. A simple baseline for efficient hand mesh reconstruction. In CVPR, pages 1367–1376, 2024. 6, 7, 8 [^22]: Christian Zimmermann and Thomas Brox. Learning to estimate 3D hand pose from single RGB images. In ICCV, pages 4903–4911, 2017. 1, 7 [^23]: Christian Zimmermann, Duygu Ceylan, Jimei Yang, Bryan Russell, Max Argus, and Thomas Brox. FreiHAND: A dataset for markerless capture of hand pose and shape from single RGB images. In ICCV, pages 813–822, 2019. 7 --- ![Image](images/43df0b41475d2e65167fb2877dd59fd85a3c156abda2df311cde7a1485d649c1.jpg) ![Image](images/4d1a61850fb617847843da1c483c1fb6bf43edca8b50f7fe35f5d7a7126a6fd7.jpg) ![Image](images/141f9fcd17d135fa9bab9094c90c4c63bfdb1379ca0f2115e218872a768d798f.jpg) ![Image](images/32baa3c020f2ccf29bfd1274da356d83f34cb86e08294bbd6cb08cf698bc1e68.jpg) ![Image](images/d3af964aa7134040b87835e2c10db870c0930e4a292c658ebb8c3ebece92da12.jpg) ![Image](images/94ef4050451fcc2b1a9396dfe2951c690392b46fe75259e20834bd3b41d4e2e6.jpg) ![Image](images/36b4ef139ee5db8f14c38acb72c8b50aa967ac37702b63ef62fc5592d6ce9c8d.jpg) ![Image](images/5585fd74e5b8e712a82acdb413d4f7ad45f8fd7a140c24cc813df0cddbb7bb6c.jpg) ![Image](images/98f1da7a8d93f02dc26a475ea60e374e40f120f05d787a0849b033bbd536f196.jpg) ![Image](images/49bfc7ccd20b348eef6478cbaa7b84cc85a49b38d2bceb137014ef53d96fe548.jpg) ![Image](images/2a77b5d0a59a02e9438c9f7fc9d0d94d5d09017e8d4515779da652e88fee2be3.jpg) ![Image](images/9d7d1af1da0d893396161e6520cf25280c2a1c4f0e10eb357d59bd6c399f32a3.jpg) ![Image](images/4a8fa45e343378385a5cde054a7e58e714bca90f8ab19c232ad508560726beeb.jpg) ![Image](images/d697896153d244b2f36e0c39ab5def0a7ba831a0bf528f00a2edf102e0b66a18.jpg) ![Image](images/bc54494dece210e5c1fd8c208549fc1c84c1af15c03f30307a4e818c5c01eed1.jpg) Input GT Ours HFL-Net H2ONet HandOccNet ![Image](images/eb456787d96e448b59a3c55896188a9ed0a94825ac3fd831ab9982702dab8753.jpg) Input ![Image](images/788255227c8439d0e8db8a31b0d92f88901f7271924870e002c353f0b1c45de0.jpg) ![Image](images/8b7e675b01b7732949678238137c0e7720e474bdd135a9f616f700bf9791fee8.jpg) ![Image](images/1e4179176a86d378d4929e0be2b3377795ec30ecb6dca621caa4e1f788cc2935.jpg) Ours HFL-Net H2ONet HandOccNet

Figure E. Qualitative comparison between UniHOPE and SOTA HPE/HOPE methods across hand-only/hand-object scenarios in DexYCB ("S3" split), in which all the grasping objects are unseen during training.

---

Hand-Only Scene

![Image](images/efc25a42d5323cad8160bde9fce6b006bdfdf8b63e352d1a227ee6b5c856cb27.jpg)

Hand-Object Scene

![Image](images/f612e1915055192af57e5c233b1b4190610475241e889635a479e3db0a46f702.jpg) ![Image](images/3732c1536c57e8d41ec271795906122649331d07b279119f1fdaab09f55b7441.jpg) ![Image](images/c24124a64b2c3036bd6d0117986abf436d57d7e371bc79f2c53b5356c7cd6057.jpg) ![Image](images/ab9b023394d2f9afee7570e6662a10628032eaa809a77a30c57b1c97a44831b5.jpg) ![Image](images/a924338316685430577b800e65dd8a91134ec59f4ebd17e0a45d64c2e4b4f37e.jpg) ![Image](images/660b0ad33b5a8dd74487a32d6ab3ecb3dc9824d8339fea3c7ab5a1d84da605fb.jpg) ![Image](images/ca1eee7c2c7059b12a7dfdf6436a19bb3a8a9dbf0429d14985f153fccf8cfdfa.jpg) ![Image](images/1816f5c95795d866f6effded4c986fe801bd92f9440498760d838f34d45d8e57.jpg) ![Image](images/b15a1be36b72bb34972155f1130ea28853b39de49dfc6907d6722e2e5ab5a857.jpg) ![Image](images/1fa4e9564bd13469c69172a21871e211802cf2f282b30f4e1dfe34a3e8242166.jpg) ![Image](images/9f8d6b2c8ccb337d56c2901555fd5cc6649c1b5f9fb17b2114a03db5bf9fade8.jpg) ![Image](images/3800a6e2e81b58b679a7a17f0df86d58d406079cf163ec8185f462b9f36ba170.jpg) ![Image](images/f7a7fd6a37361db68c18e936915cdc7f6de47b43f8f24879aa49185c07d643ce.jpg) ![Image](images/2056ca3cc177e7eed820cdb9a8bc64fcdd8d6f76318521f66ba13cee5542836c.jpg) ![Image](images/478574415574650c392cd1859046873de8fd41f59e2f29ea569e8f71936b2b41.jpg) ![Image](images/4aa95a6765561c6b9642c821570625da5d6b80831d668da4bb2ff149d4d1e926.jpg) ![Image](images/1fdba2140f926ce555df7198221deacce973b2579c7f52c587ee332823ec7303.jpg) ![Image](images/133c76f33cbdc134332c872efcae256385046015251471d55d48a08d6d08c44c.jpg) ![Image](images/79bdb2b5fadc24281fe749584afb581158b8790c1ad6980e63a4831b4b5130de.jpg) ![Image](images/5c43071fdf6b49cc74a1adf386b13313edeec6ce7f842a8fcd53d68a3a3ea178.jpg) ![Image](images/fb43f182431f393292df6dbf8d252060b4970b725ddef9cb6f292061a87fa3c4.jpg) ![Image](images/589026aa80f69f62a0e609b53819bbc19b4d3b56878e1e950c68182d73fc2e7b.jpg) ![Image](images/80a28b1ec0f9b3027189ee37898c1ca9d89d7fceea31ce0b419e59fd1988f91a.jpg) ![Image](images/09b45ed9b50c347cac9e2aa1deda6153a747aa2806b68dbc63334b7c11957f6e.jpg) ![Image](images/597f1e053217f976043c467c804b40e777a9d7d48858c9262ba9ee5c92f2fede.jpg) ![Image](images/077829022fed7688e788e8243fd68b99c11077ae28d0ebd5c4479a3a5ddee3b2.jpg) ![Image](images/74899579d4f4d8b2b24121b3b53fdda1865fbd0e0704408faedacef4f3bbaf48.jpg) ![Image](images/d32731d0450a7d4db199c3267ebe241b3e2a74cf0a0a761106c53f39bb3b4de4.jpg) ![Image](images/be393cc88b2bc15a76842559522048fa8651020425e6e2a125a4ebeff4ccfc2d.jpg) ![Image](images/491a02669c21f20085d9b636e6ac7516912830ca2a5e93ff48bff3c2025c6f62.jpg) ![Image](images/681b59de79f54a45102d573c80ac3943b8f8f5080c2846a2e137f9cb50732b31.jpg) ![Image](images/5a468f3f34f7032490e80789ab53c92fbb7054add37322965a0ffa72e1ab0a6a.jpg) ![Image](images/f78e879164bd1a5ab26d640d38b724edfe5c3aef5d58047aef4b7c07d164068d.jpg) ![Image](images/00c80ccd225a37a7bb88b2e186e65038df71b2d18ac60dc1c814be6cd376e0aa.jpg) ![Image](images/dbfb5799249f1cc1747f6188a7b626279840d0729a153342e86b0b4d5a761bc7.jpg) ![Image](images/f0ab5a81505724370792b35ad30194ecb587dc22d85fc3287c5e4df7109c0774.jpg) ![Image](images/7d7e0477a40c39e06fde294e6c1efce0126eba26fbc862b7cf543a7ceadd485f.jpg) ![Image](images/db6d235291088c35b08aa99797bced4b7edf15b9f9665e75b2d994c8d2547e75.jpg) ![Image](images/3486650b379a08e67876c8200cdb78656c03f100fb16f93c94b868376dfc6878.jpg) ![Image](images/e4434b8c3760ff84f862150a3986a3c5c4e838c6c5e306d84fb9e2256f151401.jpg) ![Image](images/0934f334014bcca5c9b637f94deb778a29d291deb767411d7a3a00b1e3f6e17e.jpg) ![Image](images/27bd72240568cd1f0ec1a19c4d10c72814b741a71f91fa5d690690fab1c2c914.jpg) ![Image](images/6d8dfd47eff12379a9f231b9ec8359bcff09f3e0addf375c26688eb14d973d8c.jpg) ![Image](images/1d7c98c3b4ba402d09902b0fbaed52d09a1f66e1603cde7dcc6485ff9532ef05.jpg) ![Image](images/1eacffdf1f5fedb861ef3ff2943fc9d1ba056a98b3f8df4979357104e53c3ca5.jpg) ![Image](images/1b358a30cd458192daf6927c969d72439ba7f7228c04a920854a3285614a1374.jpg) Ours ![Image](images/89a8ab65a26d633442e91b9f0e2b6f7ae4963298067192ac4c2f4695a2f884cf.jpg) HFL-Net H2ONet ![Image](images/608aaa536c5ee2b30fea253565362c3efef94b2dfcd8c9cde4b50205ce3d5c50.jpg) ![Image](images/cb525206b6d946301ec7e9227fdd812e2566ed3c529fdda0ac01a6884ddf3016.jpg) Input ![Image](images/d7d0a47a59a7a577ac701a482759f66687cf1fef7f5a7dc988cdc40eae5af704.jpg) ![Image](images/d31e4459ae25c7cbf3b1ab2f2e8a8f426009e997520a710f292ea39ccc5e44bb.jpg) ![Image](images/dd33eb12c73c092c4198010ad41a7bfa715e0eb503f46583ae1bd3b31dff0c9f.jpg) HFL-Net ![Image](images/11140bc010b3bc562c6cfd91ae1fc72094be906324bd00811258d0fc6f4ba0f2.jpg) H2ONet

Input HandOccNet Ours et HandOccNet Figure F. Qualitative comparison between UniHOPE and SOTA HPE/HOPE methods across hand-only/hand-object scenarios in HO3D. The ground truths are not publicly available.

--- ![Image](images/bbc6bb71795a91e1a23ebf0cb2ec3d6097dad50db3c5ddf43c76acdc5af67266.jpg) ![Image](images/3c64f0bbc4a9e8b79a62cfcbe0d1fcd79432cf0906c5f71e1676e33d82d1faec.jpg) ![Image](images/afa6982a24ea3f09bcdad5fc52db37c0a95eb0e229cfeca63b6944d4985d7be8.jpg) ![Image](images/f663ef63aa5284f6259ddcac5c289fdb1d8b0a109e50cc65dbcaed7b197b98b7.jpg) ![Image](images/1d80a3c190972c89373775061c15847671d0ba53004cc681b3aad637353b2828.jpg) GT ![Image](images/0e14dbfbc2077a6e97288f1110625ff1abd62b657974a961d4c9709e5334ef20.jpg) Ours Input ![Image](images/eff4047fd2b4e1e3e7b2b065f8c99258589a0c8d1310c7b4a0e23655a6f85ddf.jpg) HFL-Net H2ONet HandOccNet ![Image](images/883b2a041f59a2c2af7b628920a48b4a17366774905a2e9ce5504bc93f0fbb66.jpg) ![Image](images/a02905d493b0f9a8c1dfe1f61e35e82d41e1821410c6a9ee0cba77e1002340eb.jpg) Input GT Ours HFL-Net

Figure G. Qualitative comparison between our method and SOTA HPE/HOPE methods on FreiHAND.

--- ![Image](images/7dc5771b71701b749165c4229f4770e6949c6f3193145b4b321ae3a8dfcf3354.jpg) ![Image](images/caab675da512523d6441209b0a5d6d6aa0a04949e2f5535d1ca4d8b6bd6e780a.jpg) ![Image](images/8f1e0444b4fe15753406de0f567fc20bf41591adfca9038339952ac26638a368.jpg) ![Image](images/e476885d96192f32eed17f69d519dc92f6128e156d08855bab8e999430bbdecd.jpg) ![Image](images/a5dae49c776b7113aedab6fb6f2ffec72b632c7be9857834304f25b2fff80a71.jpg) ![Image](images/d42e34fc785948630c4fb04f51600ed1f76db03b28541825553dae5cb837947b.jpg) ![Image](images/c4871aed1c8d066bcc9e5e22695f4a3758ed9f8b845b10365c8503566ed6d040.jpg) ![Image](images/b3a605b7963bc072bea4c0797301b67e8e1a386153b0cd72774935f8dcf14594.jpg) ![Image](images/5f58a4df8b38254873a9bf98294943387ce48deda6f678f291f6decb2eab7394.jpg) Input ![Image](images/2bd0b4c7683168aa2d7a37e3c2fe8d973a03ccf5d33beab41e1e929710076ff7.jpg) ![Image](images/c9df596e09f315aecb8a76badd84c71db71acaf4f4ee928fc103b9a2c513ad88.jpg) Conditions ![Image](images/e1608aa0eb5cf0ae3c1350067d8c05708421ab8114afd264a4cf205a8709fe9d.jpg) Result ![Image](images/c8ca75aef8e535948c4cf8bb6cca7742cd1842c4ecf40639c87a2be9a3552f0b.jpg) ![Image](images/3018f65e357e39cfbcff9f70c8ef81d79450248e6b960146ccab022ca4cae368.jpg) Input Conditions Result

Figure H. More examples of de-occluded hand images. Note that masks are overlaid on the original image for better visualization, the actual condition for our generative de-occluder is a binary mask.

w/o

![Image](images/a45d88e1fd3c4751581de6d3ee39c5655fe173e289258b4f8232cd92fb0fc79d.jpg) (a) Grasp-aware feature fusion ![Image](images/cb439919bbcc757d77296bcdfa00b1270e7ee0f86aa9e12d9638f9d0d1e96a47.jpg) (b) Generative de-occluder ![Image](images/1e6a5651fdb199be5e978584609b2ca6039bfd460d95079d6886cd448e67d9d3.jpg) (c) Multi-level feature enhancement ![Image](images/e2984ef35bb40a07576844071004d5c9b033aab0c2e59f938ecbe1cce20c3ffa.jpg) (d) Leveraging object information

Figure I. Effects of different designs in our pipeline.

--- ![Image](images/93f058d046ec6fe5019d3a3be86f219d7214bf0b306608b77cb5a45bcdc65aac.jpg) ![Image](images/20ddbe4dbabff3b04afe0fa8233b4a878de8d58c0038e076003b1b3f19d98348.jpg) ![Image](images/59ba402318892be23686ad4b95e5b0f2e099e636c01bf306f91cc877fe28eddb.jpg) ![Image](images/c7c0c64042cbbf83b02d78cb9172ae72cacfd60f99a537210f170aa04de894d9.jpg) ![Image](images/03c3e36be9f53d602b98e0013e1487d2b30a425935ca4096a3018c1a52cc078b.jpg) ![Image](images/12307016b605de7e7aadbc3a4b075b56af6de63878d1c3de867ec7669d6da65d.jpg) ![Image](images/1cea9e1f7db12f12685863b0d32038634040b878fc6d99cba5b880a29f0234b5.jpg) ![Image](images/3b1a48b714e2b686b9f4fd7782857a3f5b5c3310aaa0746dad7d0fa958c4d062.jpg) ![Image](images/f92af925c25caffc3951f8633c24081d040688fdb6ba3f8a4b91876b0f9a4bb6.jpg) ![Image](images/7bd2dad50a1a0b899adceccb52ba3e95c7be020d7b144f7501933ccb96ad13b2.jpg) ![Image](images/ca6d52025b7a998dcec98da83db3d8dc65b00d1c95ed7ffe95f3c3d73a97bb79.jpg) ![Image](images/52d45d36e0778c9205c80ea62432f96b14f0db5ee6f33e4628939e9fa85d4589.jpg) ![Image](images/9871012cfcb10e0c114194cfd9730ef0b8da122c12743716603f5937be89238d.jpg) ![Image](images/afd484298699971bcf3a64925340504e7a60118689ffd6d5d1b0db3ef77bb50e.jpg) ![Image](images/3aca15bc801b94c4e4a8d727cb9c8448031ee362f7996d004f8416b2ec8882df.jpg) ![Image](images/52489d7183cfb39f406ac4fb98dcab1f2b4a4a7897e7bc52dba2e6d7652036d5.jpg) ![Image](images/10c0935838fd9bc6f498a3c643369193e07d908e8860d2a9267dda82da6031b2.jpg) ![Image](images/6abcafd6bad41574b3cfb7b6ed95c75300f84aacdedefc7b81f40ad63bd6b342.jpg) ![Image](images/8c4e83c02edab9cc362ab7a7de0be206a4768ea0acd593fcea08602aa0696cd6.jpg) ![Image](images/77cace295b2b7cccec644dc706988a195ef4ada4151608086f12ae555775453c.jpg) ![Image](images/7d9825bd113a7e96b8002d42dd3d0fe8cc1c3253996203accaa755bd63e06bd3.jpg) ![Image](images/6f797b0be6533cccb84ae082ee3805400daee5e958624af39d19a87c4784fbc6.jpg) ![Image](images/003c68dc4843cb64c6ef69a6ed8152ee758e6fb5a832d0716709adee3dfa87b5.jpg) ![Image](images/2f7c30d3b54d2e5c2ab5685c460c992e5fde4e52eb469938952db14156275832.jpg) ![Image](images/128e94c42617b4afe8cf7d89e420795cd28aa4e9c9c6b51e649edf57cda11bf5.jpg) ![Image](images/67a1417e397c484b19ea2b2b413308af936466b80481bf0b91f5b1b5227eabc2.jpg) ![Image](images/fbf959ffc7a9d28fed56d32be9c8bf31e9d192289488bb4b7aea2cb33b2016b1.jpg) ![Image](images/0c2d5debcc7a718e70ee1e1f5805321b2909eb0a795e2cb34d99dc589b519c01.jpg) ![Image](images/604111dea1a1f5452613ddb64b73ea690844b6189f1b00a4282eceb390545c3b.jpg) ![Image](images/b308c82443b4ae6b1a33fbf0aa33f94dbf048a12e6190351b488b4431e9c4a66.jpg) ![Image](images/835f702a2b5c4b004db07e4e2b185f8b7242d6d39660ac66fdb3889846cb1a59.jpg) ![Image](images/d28c0d1a7348c29fcf04c61834251ee1d238ca8da1db6c43c7ad530340d7f60e.jpg) ![Image](images/98c6c567394371f7f76b1d45e635bf058ec265340970e0c97fb6cf8c7714a99b.jpg) Masked Input ![Image](images/88e0181db8cabbedb803f2b3ee5803c0feedb24a23fc75a458a827bbaa3eca36.jpg) ![Image](images/d0467952b5877cc985ae71363019dd630199f3390c7343c44ae6d4d51c8d084e.jpg) Depth Map s = 0.25 ![Image](images/dc60a4b94fb401e8f3dbe6c53792c469d5b516503a87a6adc480952cdd0a7ac5.jpg) s = 0.40 ![Image](images/5531c2d645afbd90afd52db076d9695cb673faef240667ff678e4bab6e863dfc.jpg) s = 0.55 ![Image](images/14a87102af788ba4421cbf5d3bba07f2de3788b5e04d357e2a500a085ef269ff.jpg) s = 0.70 ![Image](images/01e0cbb093e1bd670c7366dfbdc7b2bf5e09f39e590b2f9b122223bd76b0a25d.jpg) s = 0.85 ![Image](images/0bf8cdc7c63e2e814c48206c068d61f43a9ec224ea6e70182126245ef67a9979.jpg) s = 1.00

Figure J. The generated images with varying control strengths. Our adaptive strategy (metrics marked in bold) effectively balances fidelity and consistency.