# Affordance-Guided Diffusion Prior for 3D Hand Reconstruction

Naru Suzuki $^{1*}$, Takehiko Ohkawa $^{1*}$, Tatsuro Banno $^{1}$, Jihyun Lee $^{2}$, Ryosuke Furuta $^{1}$, and Yoichi Sato $^{1}$

 $^{1}$The University of Tokyo $^{2}$KAIST

Abstract. How can we infer a 3D hand pose when large portions of the hand are heavily occluded by itself or by objects? Humans often resolve such ambiguities by leveraging contextual knowledge—such as affordances, where an object's shape and function suggest how the object is typically grasped. Inspired by this observation, we propose a generative prior for 3D hand pose modeling guided by affordance-aware textual descriptions of hand-object interactions (HOI). Our method employs a diffusion-based generative model that learns the distribution of plausible hand poses conditioned on contextual signals, such as affordance descriptions and image features. The affordance descriptions are designed to represent the semantic intent and geometric structure of HOI, using the reasoning from a vision-language model (VLM) and grasp classification. We leverage the diffusion prior to refine the 3D pose predictions in hand reconstruction into more accurate and functionally coherent estimation. Our experiments demonstrate that our affordance-guided refinement significantly improves 3D hand pose estimation performance on 3D hand affordance datasets, HOGraspNet and HO3D, over state-of-the-art methods such as foundation models for hand reconstruction and the latest diffusion priors for 3D hands.

## 1 Introduction

Hand-object interactions (HOI) are ubiquitous in daily life, where human actions are inseparable from the surrounding context, including objects, environments, and the person's intent. Among them, reconstructing 3D hands (e.g., estimating pose and shape parameters of the MANO model [^47]) from a single RGB image has become a crucial task, enabling applications in virtual/augmented reality and robotic dexterous manipulation. However, this task remains highly challenging due to severe occlusions and inherent 3D ambiguities commonly present in real-life interactions. Despite these difficulties, humans can still infer plausible hand poses by exploiting contextual knowledge beyond the hand itself. In particular, the shape and function of interacting objects can provide strong cues for reasoning about feasible hand configurations during interaction.

Most previous works on 3D hand reconstruction [^10] [^39] [^41] [^42] have paid little attention to leveraging such contextual knowledge. Only a few studies have explored contextual signals in the following forms: (a) temporal contact states (e.g., pre-grasp, in-grasp, post-grasp) [^65], (b) short interaction captions (e.g., “a hand holding a spoon”) [^61], (c)

---

![Image](images/f0c948dd3f46773af1a6f3c828ca890c1700a5801f83460d0c3825466f94af97.jpg)

<div style="text-align: center;">Fig. 1: Can affordance-aware semantics benefit 3D hand reconstruction? By leveraging contextual cues about how an object is grasped, our affordance-guided diffusion prior refines coarse 3D hand predictions into functionally coherent and physically grounded poses, even under strong occlusion. Vertex error is color-coded on the hand mesh.</div>

action labels (e.g., “screw”, “open”) [^30] [^37], (d) intention labels (e.g., intend to “use” or “receive” an object) [^52] [^60], etc. These semantic labels have been used as conditional input in 3D modeling [^56] [^60] [^61], for additional semantic classification [^44] [^59] [^65], or to create specific evaluation categories [^4] [^30] [^37]. However, these existing labels remain coarse and fail to capture the rich details of HOI needed to resolve ambiguities.

To enrich contextual signals and better reflect the subtle nuances of HOI scenes, we emphasize the critical role of affordances [^1] [^6] [^14] as semantic guidance, which directly encodes how object properties constrain and guide possible hand interactions. Existing affordance-related studies are typically categorized into recognizing (i) motor actions or (ii) grasp types, defined by discrete labels [^63]. Motor actions, often expressed by verb labels (e.g., “pick up”), represent the high-level semantic intent of manipulation. In contrast, grasp types, defined through rigorous taxonomies (e.g., “lateral grasp”, “precision grasp”) [^4] [^12], provide the geometric structure of fine-grained hand configurations.

Based on this perspective, we integrate these two distinct aspects of affordances, i.e., both semantic intent and geometric structure of HOI, into a unified contextual representation for hand reconstruction. However, the way to combine these heterogeneous elements of affordances in a unified form is not obvious. Unlike following existing categories (e.g., “pick up” or “lateral grasp”), we tackle this issue by representing affordances through detailed textual descriptions that encode both semantic and geometric cues. This descriptive form offers greater flexibility to capture variations in object attributes, hand grasps, and interaction intent beyond predefined categories. In particular, we incorporate grasp types and object attributes (category, shape, and size), along with the underlying intent, such as interaction (action) and intention captions. To generate such a descriptive text reflecting multiple affordance aspects, we propose a step-by-step generation process by leveraging visual commonsense reasoning of a large vision-language model (VLM) [^2] with grasp taxonomy knowledge from the hand affordance dataset [^4], resulting in complete and interpretable descriptions of affordances.

With the affordance descriptions, we propose generative prior modeling that can integrate textual descriptions into 3D pose modeling (Fig. 1). We train a denoising diffusion model [^16] on hand pose data, while newly adopting affordance descriptions, encoded with DistilBERT [^48], and ViT-based image features [^9] as conditional input. We further utilize this prior to refine 3D pose estimates, inspired by [^38]. Specifically,

---

starting from an initial estimate by WiLoR [^42] or Hamba [^8], we iteratively denoise the pose so that it remains consistent with the affordance-aware diffusion prior and visible 2D evidence. To improve the refinement, we further detect occluded joints with ray casting from joints and hand segmentation to guide only uncertain joints.

Our experiments are conducted on 3D hand affordance datasets, HOGraspNet [^4] and HO3D [^15] where the grasp type labels are assigned by the classification model trained on HOGraspNet. Our affordance-guided diffusion prior consistently outperforms recent hand reconstruction models, including foundational transformers such as WiLoR [^42] and HaMeR [^41], the Mamba-based Hamba [^8], and hand diffusion priors based on InterHandGen [^24], with or without contextual reasoning. Furthermore, our qualitative study finds our method producing grasp poses that not only appear more natural but also align closely with the affordance descriptions.

Our main contributions are summarized as follows:

- We are the first to introduce affordance-aware textual descriptions into 3D hand reconstruction using VLMs and grasp classification.

- We design a diffusion-based generative prior that learns the hand pose distribution given the contextual signals, enabling semantically consistent generation.

We propose an occlusion-aware refinement approach that leverages the affordance-aware diffusion prior for 3D hand reconstruction, yielding substantial improvements in pose accuracy specifically when estimated joints are uncertain.

## 2 Related Work

3D hand reconstruction: This task involves estimating hand pose and shape to reconstruct hand mesh, typically from a single RGB image. The earlier efforts [^10] [^23] [^36] [^39] [^64] encode image features with CNN-based backbones and regress pose and shape parameters of the MANO model [^47]. Particularly, HandOccNet [^39] proposes robust feature extraction for occluded regions. Recent foundation models [^41] [^42] leverage a vision transformer [^9] with large-scale pretraining to enhance generalization ability across different domains. While these recent methods typically provide strong alignment to 2D observation, ambiguities still remain in the 3D space, e.g., depth ambiguity and uncertainty for occluded joints.

To address this, integrating with contextual cues is crucial as humans can infer feasible 3D poses given the context. Yet, this direction remains relatively underexplored. For instance, recent methods [^44] [^56] [^59] [^65] [^67] attempt to integrate semantic or physical cues (e.g., grasp type and state, body proportion, and action labels), but focus on coarse-level reasoning rather than fine-grained details of HOI scenes.

In contrast, our work is motivated by leveraging detailed textual descriptions regarding affordances (see Fig. 2) and we are the first to introduce such affordance descriptions into 3D hand reconstruction.

Affordance recognition and reasoning: The concept of affordances, introduced by Gibson [^13], describes the functional relationship between perception and potential action (e.g., a handle affords grasping or a button affords pressing). In computer vision, affordance recognition and reasoning have been studied through diverse formulations, including functional region segmentation [^11] [^18] [^25] [^34], grasp and contact prediction [^4] [^6] [^14], affordance-based HOI understanding [^7] [^22] and generation [^3] [^43] [^62] [^66],

---

and adaptation to robot manipulation [^1] [^26] [^58]. These approaches often operate at the object or scene level, capturing where and how an agent may interact. Yet, they typically rely on coarse labels [^63] such as verb categories (e.g., “pick up”, “place”) or predefined grasp taxonomies [^12]. Recent 3D hand affordance datasets [^4] [^18] further encourage affordance understanding with precise hand poses, object geometry, and vertex-wise contact.

In this work, we propose to represent affordances as rich, detailed, and multi-level semantic descriptions that unify motor intent, grasp configuration, and object attributes (see Fig. 2). Furthermore, unlike the existing formulations, we demonstrate that such affordance descriptions resolve the fine-grained geometric ambiguities central to 3D hand reconstruction.

Diffusion models: Denoising diffusion models [^16] have emerged as a powerful generative framework, owing to their strong capability to capture complex data distribution and their flexibility during inference. Diffusion models are trained to gradually transform Gaussian noise into realistic samples through an iterative denoising process.

Recent works adopt diffusion-based modeling in 3D hand and body modeling, outperforming the conventional VAEs-based modeling [^31] [^33]. The diffusion models can leverage various conditional inputs for 3D modeling [^5] [^24] [^29] [^38] [^50] [^54] [^61]. Human motion generation commonly leverages text prompts to guide motion patterns [^32] [^53] [^54], including hand-object motion generation [^5] [^27]. InterHandGen [^24] synthesizes two-hand interactions conditioned on a single hand, while G-HOP [^61] generates 3D hand-object interactions from object geometry. The trained diffusion priors can be further adapted to 3D reconstruction tasks with score-distillation sampling [^24] [^61] and refinement with keypoint guidance [^38] [^50].

Our work instead proposes a pose diffusion model that flexibly takes multi-modal input for detailed text and visual features, unlike existing frameworks, e.g., text-to-motion [^5] [^27], 3D two-hand [^24], and 3D hand-object generation [^61]. It further enhances the diffusion-based refinement with guidance from occlusion detection unlike [^38].

## 3 Method

Our work is motivated to leverage contextual signals from textual descriptions of affordances and visual evidence for 3D hand reconstruction. To incorporate these cues, we propose a generative prior model based on a diffusion process that learns the 3D hand pose distribution given the contextual signals. Our approach begins with an affordance description generation, which extracts key elements of affordances from input images using a vision-language model and grasp classification, as described in Sec. 3.1. We then construct a diffusion prior conditioned on the affordance descriptions and image features, as detailed in Sec. 3.2. Subsequently, the learned prior is adapted to correct occluded or uncertain joints with the refinement scheme in Sec. 3.3, yielding more accurate and functionally coherent hand poses.

### 3.1 Affordance description generation

Affordances represent how an object can be interacted with hands by capturing its actionable properties; thus we utilize them as contextual representation and guidance for

---

![Image](images/7bf8d82d3ca5dfdaa65647931816a4e281ca6e1dae3fe29fce2807fd331c1944.jpg)

<div style="text-align: center;">Fig. 2: Affordance description generation. (a) We show a naive baseline of caption generation (Simple VLM) from off-the-shelf VLMs (e.g., Qwen2.5-VL [^2]) given hand-object bounding boxes, lacking fine-grained affordance cues. (b) Inspired by chain-of-thought reasoning, our proposed generation first obtains parsed captions (items 1-5) by prompting the VLM to extract intended affordance elements. It further infers a grasp taxonomy (item 6) with a tailored classifier due to VLM's limited capability to predict hand grasps in a zero-shot manner. These generated affordance elements are summarized using LLM [^19], resulting in a single detailed description.</div>

3D hand reconstruction. Specifically, beyond standard categorical labels (e.g., grasp or action labels) of affordance recognition [^4] [^22] [^44] [^63], we enrich its form with textual descriptions that reflect both semantic and precise geometric properties, such as object category, shape, size, interaction (action), intention, and grasp type.

Our preliminary study finds that direct (one-step) captioning, which prompts VLMs to generate a description from an image, often produces a simplified sentence lacking fine-grained HOI properties. An example of this simply generated caption (“Simple VLM”) is found in Fig. 2(a). Additionally, while being able to reasonably describe HOI scenes, recent VLMs still struggle to capture precise grasp taxonomy as this annotation is not common in general QA tasks or VLMs’ pretraining data. To provide finer affordance details, we generate the affordance descriptions step-by-step in Fig. 2(b). Inspired by the chain-of-thought reasoning [^57] of LLMs, we break down the complex description task into small executable and interpretable steps. Our pipeline consists of three steps: (i) extracting key elements with zero-shot VLM reasoning, (ii) classifying the grasp with separate supervised training, and (iii) integrating all elements into a final description with LLM. This procedure ensures the completeness and interpretability of the generated affordance descriptions with improved caption quality. We detail the individual steps below.

Step 1 – Extract parsed captions with VLM: We leverage a vision-language model (Qwen2.5-VL [^2]) to extract structured captions describing different aspects of affordances from an image. Specifically, given the image together with the bounding boxes of the hand and the object, we design a prompt that queries the model to describe five key elements that characterize HOI properties. These include object category (e.g., "disk", "sphere"), object shape (e.g., "elongated", "flat"), object size (e.g., "small", "large"), interaction (action) type (e.g., "holding", "reaching"), and intention caption (e.g., "to lift", "to manipulate"). These elements are chosen because they capture the physical properties of the object and the semantic intent of the action. Further details are found in the supplement.

---

![Image](images/7a41e6ef4753bb09156eefa3e56803d107459678935fac3929c1d0a1147e3c58.jpg)

<div style="text-align: center;">Fig. 3: Denoising framework of affordance-guided diffusion prior (AffHandGen). The input diffused hand pose is denoised using a transformer decoder network conditioned on the affordance description and RGB image, each encoded by DistilBERT [^48] and the ViT backbone [^41]. These modality interactions across pose, text, and image are learned through cross-attention.</div>

Step 2 – Grasp classification: Accessing grasp information is essential in determining hand configurations. Yet, we observe that classifying grasp taxonomies using existing VLMs is highly challenging, as rigorous grasp labels [^12] may not be widely represented in their pretraining data. Therefore, we train a dedicated grasp classification model independently on a large-scale hand affordance dataset, HOGraspNet [^4]. Specifically, we take the RGB image as input and extract visual features encoded by the ViT and MANO pose parameters using HaMeR [^41]. The visual and pose features are concatenated and passed through a self-attention layer, followed by an MLP layer for final classification. Our model achieves 87.6% accuracy in the evaluation dataset, and further implementation details and results are provided in the supplement.

Step 3 – Summarize with LLM: Given the short captions for each element related to affordances, we transform these items into a single detailed description. We use a large language model (LLM), Mistral-7B [^19], with a summarization prompt that generates a final affordance description. This LLM-based summarization allows us to combine nuanced variations in HOI properties, producing a coherent sentence that flexibly represents semantic and geometric cues beyond what predefined categories can express.

Overall, this divide-and-conquer approach comprised of the three steps ensures that the resulting descriptions are (i) complete and specific by eliciting all key elements, (ii) consistent with the predicted grasp type via supervised classification, and (iii) coherent and interpretable as the conditional input (used for the subsequent diffusion training).

### 3.2 Affordance-guided diffusion prior

Using the affordance descriptions, we present a diffusion prior model conditioned on text and image features, dubbed AffHandGen. This model is trained to generate and refine 3D hand poses aligned with the contextual signals. We choose the diffusion modeling to learn data distribution across different modalities, such as image, text, and pose, inspired by its successful results in 3D human synthesis [^24] [^38] [^40] [^54]. It also benefits from being directly applicable to various downstream tasks without additional training [^50], making it flexible and broadly reusable.

Overview: With the goal of refining 3D hand pose in the downstream task, we construct a diffusion model in the hand pose space, where the shape estimation remains fixed. For

---

the input, we use the pose parameters of MANO [^47] as the target variable of denoising, i.e., $\mathbf{x}_0 = \boldsymbol{\theta} \in \mathbb{R}^{15 \times 3}$. We also use a diffusion timestep t, an affordance description D (generated in Sec. 3.1), and the corresponding RGB image I as the conditional input. We encode the affordance description using a text encoder, yielding textual features $\mathbf{c}_D$, and encode the RGB image with an image encoder, yielding image features $\mathbf{c}_I$. The resulting diffusion model parameterized by $\phi$ is formulated as $\hat{\mathbf{x}}_0 = f_\phi(\mathbf{x}_t, t, \mathbf{c})$, where $\mathbf{x}_t$ is the diffused pose at the timestep t and $\mathbf{c} = \{\mathbf{c}_D, \mathbf{c}_I\}$ denotes the text and image condition.

The overall architecture of $f_{\phi}(\cdot)$ is illustrated in Fig. 3. Our AffHandGen is a transformer network trained to denoise noisy hand pose input $\mathbf{x}_t$ at the timestep t. The hand pose and timestep are first encoded through linear layers and concatenated, followed by processing with transformer decoder layers and a final pose decoder with a linear layer. Cross-attention to modality tokens: For conditional data encoding, InterHandGen [^24] treats the condition $\mathbf{c}$ as a global feature vector and learns the relationships across the target pose, conditions, and timestep using self-attention. However, representing each conditional modality with a single vector loses spatial and semantic granularity. For instance, when two hand images involve the same object but different grasp types, the global visual or textual embeddings (e.g., from CNN or CLIP [^45]) tend to be similar, making it difficult for the diffusion model to distinguish subtle interaction differences.

To capture such fine-grained contextual distinctions, we represent each modality as a set of tokens and employ a cross-attention mechanism [^55] that learns localized correspondences between the denoising pose features and conditional modality tokens. This design enables the model to selectively attend to informative modality tokens, including those implicitly encoding grasp semantics, thereby achieving context-sensitive conditioning. Specifically, we use modality-specific tokenizers: DistilBERT [^48] for the text embedding and the ViT network trained in [^41] for the image embedding.

Diffusion process: We follow the DDPM formulation [^16], where the diffusion process consists of forward and reverse paths spanned with diffusion steps $t \in [0, T]$. The forward process gradually adds standard Gaussian noise to the input data $\mathbf{x}_0$, which is formulated as

 $$ \mathbf{x}_{t}:=\sqrt{\bar{\alpha}_{t}}\mathbf{x}_{0}+\sqrt{1-\bar{\alpha}_{t}}\mathbf{\epsilon}, $$

where $\epsilon \sim \mathcal{N}(0, I)$ is a noise variable and $\alpha_{1:T} \in (0, 1]^T$ is a sequence that controls the degree of noise at each timestep t. In contrast, the reverse process learns to iteratively transform noise $\mathbf{x}_T \sim \mathcal{N}(0, I)$ toward the target data distribution $\mathbf{x}_0 \sim q(x_0)$, defined as

 $$ p_{\phi}(\mathbf{x}_{0:T}):=p(\mathbf{x}_{T})\prod_{t=1}^{T}p_{\phi}(\mathbf{x}_{t-1}\mid\mathbf{x}_{t}), $$

where $p_\phi$ is a model distribution defined by the network $f_\phi(\cdot)$. We train the denoising network to approximate the original data $\mathbf{x}_0$ following [^24] [^38] [^54].

The training loss is computed between the original data $\mathbf{x}_0$ and the generated data $\hat{\mathbf{x}}_0$. We take the L2 loss between $\mathbf{x}_0$ and $\hat{\mathbf{x}}_0$ for the pose space, denoted as $L_\theta$. After passing to the MANO layer, we compute the L2 loss on the mesh vertices and 3D joint space, denoted as $L_v$ and $L_j$. The overall loss is formulated as

 $$ L_{t o t a l}=\lambda_{\theta}L_{\theta}+\lambda_{v}L_{v}+\lambda_{j}L_{j}. $$

---

```pseudo
// Algorithm 1: Diffusion-based pose refinement.

```

Diffuse 3D pose and visibility labels v of $\tilde{\mathbf{x}}_{0}$

1: $\tilde{\mathbf{x}}_{n} \leftarrow \sqrt{\tilde{\alpha}_{n} \tilde{\mathbf{x}}_{0}} + \sqrt{1 - \tilde{\alpha}_{n}} \epsilon$

2: $\mathbf{v} = V(\tilde{\mathbf{x}}_{0})$ // 1: visible, 0: occluded

3: for $t = n$ to 1 do

4: $\tilde{\mathbf{x}}_{0} \leftarrow f(\tilde{\mathbf{x}}_{t}, t, \mathbf{c})$

5: $2D$ keypoint fitting only for visible joints

6: $\tilde{\mathbf{x}}_{0} \leftarrow \tilde{\mathbf{x}}_{0} - \lambda_{2d} \nabla \tilde{\mathbf{x}}_{0} \mathcal{L}_{2}(M(\tilde{\mathbf{x}}_{0}), M(\tilde{\mathbf{x}}_{0})) \odot \mathbf{v}$

7: $\tilde{\mathbf{x}}_{t-1} \leftarrow \sqrt{\tilde{\alpha}_{t-1}} \tilde{\mathbf{x}}_{0} + \sqrt{1 - \tilde{\alpha}_{t-1}} \epsilon_{t}$

8: end for

9: return $\tilde{\mathbf{x}}_{0}$

![Image](images/ef36571aaa31dba75572f9279c237ecca60861b54ed502305a5df396f1e9c277.jpg)

<div style="text-align: center;">Fig. 4: Diffusion-based pose refinement algorithm. Our refinement takes the initial 3D pose estimates  $\tilde{x}_0$ from recent hand reconstruction methods (e.g., [^42]), conditional text and image features c, and detected visibility  $V(\cdot)$ in Sec. 3.3, and then corrects occluded joints (red and yellow in the right figure) while visible joints (green) remain unchanged. The visibility labels are obtained from two detection criteria: self-occlusion with ray casting on the MANO mesh and object occlusion with SAM2 [^46] mask for hands. This refinement is resumed at the middle of diffusion steps n, reducing the total inference cost, and optimized with the 2D keypoint loss based on camera projection function  $M(\cdot)$ and its loss weight  $\lambda_{2d}$.</div>

 $\lambda_{\theta}, \lambda_{v}$, and $\lambda_{j}$ are the weights for each loss term.

To improve the robustness of the model, we randomly dropout the conditional input $c_D$ and $c_T$, inspired by [^17] [^24] [^33] [^50]. Specifically, we randomly mask the affordance description and the RGB image with a probability of 0.1. This emulates conditional and unconditional generation within a single model. This strategy brings several benefits: (i) preventing the model from overfitting to a specific conditioning modality, leading to a strong prior learning over 3D hand poses, and (ii) improving flexibility at inference time, allowing seamless switching between conditional and unconditional sampling without retraining.

### 3.3 Occlusion-aware prior adaptation to 3D hand reconstruction

Once the diffusion prior is trained, we adapt the prior to guide and improve 3D hand reconstruction, specifically aiming to refine the 3D pose parameters. While foundation models for hand reconstruction (e.g., HaMeR [^41]) effectively align hand meshes with the visible 2D evidence in RGB images, they still struggle to recover accurate 3D poses under severe occlusion, where visual cues are limited. SCGen [^38] addresses this by applying a 3D human diffusion prior to refine 3D pose estimates with 2D keypoint constraints. Given an initial 3D pose $\tilde{x}_0$ (the target of refinement) and reliable 2D keypoints $y_0$ detected by human foundation models like Sapiens [^20], it first sets a diffused pose $\tilde{x}_n$ at the intermediate step $n (0 < n < T)$ and iteratively denoises the pose until $t = 0$, while minimizing the 2D projection loss between $\tilde{x}_0$ and $y_0$.

Inspired by the diffusion-based refinement for human pose [^38], we introduce an affordance-guided diffusion refinement tailored to 3D hand reconstruction (Fig. 4). Our approach newly incorporates contextual cues from textual descriptions and image features, enabling the model to reconstruct hand poses that are physically plausible and semantically consistent with the intended interaction. Unlike body poses, hands exhibit high articulation and frequent object contacts, leading to more complex occlusion patterns.

---

terms. To address this, we detect the visibility of each joint and apply selective 2D guidance only to visible joints, allowing the diffusion prior to correct uncertain (occluded) joints through contextual conditioning.

Our refinement framework requires the initial 3D pose $\bar{\mathbf{x}}_0$, its camera projection $(i.e.,\ 2D\ keypoints)$ $M(\bar{\mathbf{x}}_0)$, joint-wise visibility labels $\mathbf{v} \in \{0,1\}^J$ ( J is the number of joints), and contextual conditions $\mathbf{c} = \{\mathbf{c}_D, \mathbf{c}_I\}$ from text and image features. The visibility labels $\mathbf{v}$ are obtained through the detection scheme below and are used to mask the 2D keypoint loss. Since reliable 2D hand keypoints are more difficult to obtain than those in the body domain [^38], we instead use the projected 2D keypoints of the initial 3D pose, $M(\bar{\mathbf{x}}_0)$, along with the visibility labels as supervision signals. This allows the model to align the pose to confident 2D evidence and denoise uncertain occluded joints based on the contextual diffusion prior. We further apply the 2D keypoint fitting to the denoised pose $\bar{\mathbf{x}}_0$ as post-processing. This ensures that the resulting pose after the diffusion-based refinement is well-aligned to the visible 2D keypoints.

Occlusion detection: Estimating joint visibility is useful for downstream HOI tasks, yet challenging. We employ two criteria to determine the occlusion of hand joints regarding self-occlusion and object occlusion, resulting in the binary visibility labels v. For self-occlusion, we determine occluded joints by ray casting given the hand mesh derived from $\tilde{x}_0$. A ray is cast from each hand joint toward the camera, and its intersections with the hand surface are computed as [^49]. If the number of intersections is greater than or equal to two, the joint is occluded by the part of hand mesh, regarded as self-occlusion. For instance, a ray from the pinky fingertip of Fig. 4 penetrates into the index finger's mesh, judged as the self-occluded joint. For object occlusion, we identify object-occluded joints by comparing the projected 2D joints from the initial pose predictions, $M(\tilde{x}_0)$, and the hand mask obtained from SAM2 [^46]. If the 2D joint position is outside the mask, the joint is regarded as object occlusion.

## 4 Experiments

We present quantitative results on pose refinement, comparing against foundational models for 3D hand reconstruction and diffusion prior baselines with or without using contextual signals. We then present ablations and qualitative analyses of pose refinement and controllable generations. Additional results are found in the supplement.

Experimental settings: We use the HOGraspNet [^4] dataset for complex and diverse hand-object interactions, which is the only dataset offering RGB images, 3D assets, and grasp taxonomy labels needed for our task setup. We follow 28 of the 33 grasp types defined by [^12] to train a grasp classification network. We also use the HO3D [^15] dataset including RGB images and 3D hand-object annotations, as an additional validation and the HInt dataset [^41] for in-the-wild evaluation. As these two datasets lack grasp annotations, we apply the grasp classifier trained on [^4] to assign pseudo grasp labels. We subsample all datasets and use sparse frames as training and evaluation, because consecutive frames sharing similar visuals often contain the same grasp labels and almost identical VLM descriptions. To ensure diverse contextual signals without heavy semantic overlaps, we resample every 10 frames, resulting in 119K training and 3K evaluation images in [^4] and 65K training and 1K evaluation images in [^15]. HOGrasp

---

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td colspan="11">HOGraspNet</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Method</td><td style='text-align: center; word-wrap: break-word;'>Avg. $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>Low</td><td style='text-align: center; word-wrap: break-word;'>Med.</td><td style='text-align: center; word-wrap: break-word;'>High</td><td style='text-align: center; word-wrap: break-word;'>PA-MPVPE $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>PCK@5 $\uparrow$</td><td style='text-align: center; word-wrap: break-word;'>PCK@10 $\uparrow$</td><td style='text-align: center; word-wrap: break-word;'>PCK@15 $\uparrow$</td><td style='text-align: center; word-wrap: break-word;'>F@5 $\uparrow$</td><td style='text-align: center; word-wrap: break-word;'>F@15 $\uparrow$</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>HandOccNet [^39]</td><td style='text-align: center; word-wrap: break-word;'>15.09</td><td style='text-align: center; word-wrap: break-word;'>14.34</td><td style='text-align: center; word-wrap: break-word;'>15.55</td><td style='text-align: center; word-wrap: break-word;'>17.65</td><td style='text-align: center; word-wrap: break-word;'>12.93</td><td style='text-align: center; word-wrap: break-word;'>12.0</td><td style='text-align: center; word-wrap: break-word;'>42.8</td><td style='text-align: center; word-wrap: break-word;'>67.4</td><td style='text-align: center; word-wrap: break-word;'>0.482</td><td style='text-align: center; word-wrap: break-word;'>0.935</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>MeshGraphormer [^28]</td><td style='text-align: center; word-wrap: break-word;'>10.02</td><td style='text-align: center; word-wrap: break-word;'>9.39</td><td style='text-align: center; word-wrap: break-word;'>10.27</td><td style='text-align: center; word-wrap: break-word;'>13.55</td><td style='text-align: center; word-wrap: break-word;'>8.99</td><td style='text-align: center; word-wrap: break-word;'>24.7</td><td style='text-align: center; word-wrap: break-word;'>68.7</td><td style='text-align: center; word-wrap: break-word;'>87.5</td><td style='text-align: center; word-wrap: break-word;'>0.625</td><td style='text-align: center; word-wrap: break-word;'>0.977</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>HaMeR [^41]</td><td style='text-align: center; word-wrap: break-word;'>8.38</td><td style='text-align: center; word-wrap: break-word;'>7.37</td><td style='text-align: center; word-wrap: break-word;'>8.92</td><td style='text-align: center; word-wrap: break-word;'>12.68</td><td style='text-align: center; word-wrap: break-word;'>7.66</td><td style='text-align: center; word-wrap: break-word;'>38.6</td><td style='text-align: center; word-wrap: break-word;'>78.6</td><td style='text-align: center; word-wrap: break-word;'>91.7</td><td style='text-align: center; word-wrap: break-word;'>0.696</td><td style='text-align: center; word-wrap: break-word;'>0.988</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Hamba [^8]</td><td style='text-align: center; word-wrap: break-word;'>8.46</td><td style='text-align: center; word-wrap: break-word;'>7.44</td><td style='text-align: center; word-wrap: break-word;'>9.00</td><td style='text-align: center; word-wrap: break-word;'>12.72</td><td style='text-align: center; word-wrap: break-word;'>7.57</td><td style='text-align: center; word-wrap: break-word;'>38.3</td><td style='text-align: center; word-wrap: break-word;'>78.9</td><td style='text-align: center; word-wrap: break-word;'>91.8</td><td style='text-align: center; word-wrap: break-word;'>0.698</td><td style='text-align: center; word-wrap: break-word;'>0.987</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>+ Ours</td><td style='text-align: center; word-wrap: break-word;'>7.85</td><td style='text-align: center; word-wrap: break-word;'>7.49</td><td style='text-align: center; word-wrap: break-word;'>10.21</td><td style='text-align: center; word-wrap: break-word;'>9.46</td><td style='text-align: center; word-wrap: break-word;'>7.11</td><td style='text-align: center; word-wrap: break-word;'>41.2</td><td style='text-align: center; word-wrap: break-word;'>81.6</td><td style='text-align: center; word-wrap: break-word;'>93.2</td><td style='text-align: center; word-wrap: break-word;'>0.718</td><td style='text-align: center; word-wrap: break-word;'>0.991</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>WiLoR [^42]</td><td style='text-align: center; word-wrap: break-word;'>8.22</td><td style='text-align: center; word-wrap: break-word;'>7.42</td><td style='text-align: center; word-wrap: break-word;'>8.56</td><td style='text-align: center; word-wrap: break-word;'>12.41</td><td style='text-align: center; word-wrap: break-word;'>7.57</td><td style='text-align: center; word-wrap: break-word;'>39.7</td><td style='text-align: center; word-wrap: break-word;'>79.4</td><td style='text-align: center; word-wrap: break-word;'>92.2</td><td style='text-align: center; word-wrap: break-word;'>0.702</td><td style='text-align: center; word-wrap: break-word;'>0.989</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>+ InterHandGen [^24]</td><td style='text-align: center; word-wrap: break-word;'>8.99</td><td style='text-align: center; word-wrap: break-word;'>7.64</td><td style='text-align: center; word-wrap: break-word;'>9.72</td><td style='text-align: center; word-wrap: break-word;'>14.61</td><td style='text-align: center; word-wrap: break-word;'>7.91</td><td style='text-align: center; word-wrap: break-word;'>34.8</td><td style='text-align: center; word-wrap: break-word;'>75.4</td><td style='text-align: center; word-wrap: break-word;'>90.1</td><td style='text-align: center; word-wrap: break-word;'>0.681</td><td style='text-align: center; word-wrap: break-word;'>0.986</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>+ w/ Full Cond. ( $D+I$)</td><td style='text-align: center; word-wrap: break-word;'>8.06</td><td style='text-align: center; word-wrap: break-word;'>7.50</td><td style='text-align: center; word-wrap: break-word;'>8.29</td><td style='text-align: center; word-wrap: break-word;'>11.12</td><td style='text-align: center; word-wrap: break-word;'>7.29</td><td style='text-align: center; word-wrap: break-word;'>40.3</td><td style='text-align: center; word-wrap: break-word;'>79.6</td><td style='text-align: center; word-wrap: break-word;'>92.3</td><td style='text-align: center; word-wrap: break-word;'>0.713</td><td style='text-align: center; word-wrap: break-word;'>0.990</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>+ Ours</td><td style='text-align: center; word-wrap: break-word;'>7.53</td><td style='text-align: center; word-wrap: break-word;'>7.29</td><td style='text-align: center; word-wrap: break-word;'>7.57</td><td style='text-align: center; word-wrap: break-word;'>9.37</td><td style='text-align: center; word-wrap: break-word;'>6.90</td><td style='text-align: center; word-wrap: break-word;'>44.1</td><td style='text-align: center; word-wrap: break-word;'>82.5</td><td style='text-align: center; word-wrap: break-word;'>93.6</td><td style='text-align: center; word-wrap: break-word;'>0.731</td><td style='text-align: center; word-wrap: break-word;'>0.992</td></tr><tr><td colspan="11">HO3D</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Method</td><td style='text-align: center; word-wrap: break-word;'>Avg. $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>Low</td><td style='text-align: center; word-wrap: break-word;'>Med.</td><td style='text-align: center; word-wrap: break-word;'>High</td><td style='text-align: center; word-wrap: break-word;'>PA-MPVPE $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>PCK@5 $\uparrow$</td><td style='text-align: center; word-wrap: break-word;'>PCK@10 $\uparrow$</td><td style='text-align: center; word-wrap: break-word;'>PCK@15 $\uparrow$</td><td style='text-align: center; word-wrap: break-word;'>F@5 $\uparrow$</td><td style='text-align: center; word-wrap: break-word;'>F@15 $\uparrow$</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>HandOccNet [^39]</td><td style='text-align: center; word-wrap: break-word;'>8.83</td><td style='text-align: center; word-wrap: break-word;'>8.46</td><td style='text-align: center; word-wrap: break-word;'>9.44</td><td style='text-align: center; word-wrap: break-word;'>9.86</td><td style='text-align: center; word-wrap: break-word;'>8.80</td><td style='text-align: center; word-wrap: break-word;'>31.4</td><td style='text-align: center; word-wrap: break-word;'>69.3</td><td style='text-align: center; word-wrap: break-word;'>86.7</td><td style='text-align: center; word-wrap: break-word;'>0.572</td><td style='text-align: center; word-wrap: break-word;'>0.965</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>MeshGraphormer [^28]</td><td style='text-align: center; word-wrap: break-word;'>12.55</td><td style='text-align: center; word-wrap: break-word;'>11.71</td><td style='text-align: center; word-wrap: break-word;'>13.15</td><td style='text-align: center; word-wrap: break-word;'>18.79</td><td style='text-align: center; word-wrap: break-word;'>12.83</td><td style='text-align: center; word-wrap: break-word;'>16.6</td><td style='text-align: center; word-wrap: break-word;'>52.0</td><td style='text-align: center; word-wrap: break-word;'>73.3</td><td style='text-align: center; word-wrap: break-word;'>0.436</td><td style='text-align: center; word-wrap: break-word;'>0.908</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>HaMeR [^41]</td><td style='text-align: center; word-wrap: break-word;'>8.15</td><td style='text-align: center; word-wrap: break-word;'>7.34</td><td style='text-align: center; word-wrap: break-word;'>9.59</td><td style='text-align: center; word-wrap: break-word;'>9.54</td><td style='text-align: center; word-wrap: break-word;'>8.33</td><td style='text-align: center; word-wrap: break-word;'>37.7</td><td style='text-align: center; word-wrap: break-word;'>74.5</td><td style='text-align: center; word-wrap: break-word;'>89.0</td><td style='text-align: center; word-wrap: break-word;'>0.632</td><td style='text-align: center; word-wrap: break-word;'>0.973</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Hamba [^8]</td><td style='text-align: center; word-wrap: break-word;'>7.42</td><td style='text-align: center; word-wrap: break-word;'>7.10</td><td style='text-align: center; word-wrap: break-word;'>7.88</td><td style='text-align: center; word-wrap: break-word;'>8.71</td><td style='text-align: center; word-wrap: break-word;'>7.66</td><td style='text-align: center; word-wrap: break-word;'>40.3</td><td style='text-align: center; word-wrap: break-word;'>77.9</td><td style='text-align: center; word-wrap: break-word;'>91.3</td><td style='text-align: center; word-wrap: break-word;'>0.655</td><td style='text-align: center; word-wrap: break-word;'>0.982</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>+ Ours</td><td style='text-align: center; word-wrap: break-word;'>7.29</td><td style='text-align: center; word-wrap: break-word;'>7.25</td><td style='text-align: center; word-wrap: break-word;'>7.14</td><td style='text-align: center; word-wrap: break-word;'>8.39</td><td style='text-align: center; word-wrap: break-word;'>7.37</td><td style='text-align: center; word-wrap: break-word;'>41.1</td><td style='text-align: center; word-wrap: break-word;'>78.9</td><td style='text-align: center; word-wrap: break-word;'>91.9</td><td style='text-align: center; word-wrap: break-word;'>0.657</td><td style='text-align: center; word-wrap: break-word;'>0.983</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>WiLoR [^42]</td><td style='text-align: center; word-wrap: break-word;'>7.43</td><td style='text-align: center; word-wrap: break-word;'>7.19</td><td style='text-align: center; word-wrap: break-word;'>7.85</td><td style='text-align: center; word-wrap: break-word;'>7.87</td><td style='text-align: center; word-wrap: break-word;'>7.63</td><td style='text-align: center; word-wrap: break-word;'>39.0</td><td style='text-align: center; word-wrap: break-word;'>77.9</td><td style='text-align: center; word-wrap: break-word;'>91.9</td><td style='text-align: center; word-wrap: break-word;'>0.650</td><td style='text-align: center; word-wrap: break-word;'>0.983</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>+ InterHandGen [^24]</td><td style='text-align: center; word-wrap: break-word;'>7.55</td><td style='text-align: center; word-wrap: break-word;'>7.30</td><td style='text-align: center; word-wrap: break-word;'>7.99</td><td style='text-align: center; word-wrap: break-word;'>7.99</td><td style='text-align: center; word-wrap: break-word;'>7.64</td><td style='text-align: center; word-wrap: break-word;'>38.1</td><td style='text-align: center; word-wrap: break-word;'>77.6</td><td style='text-align: center; word-wrap: break-word;'>91.4</td><td style='text-align: center; word-wrap: break-word;'>0.638</td><td style='text-align: center; word-wrap: break-word;'>0.980</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>+ w/ Full Cond. ( $D+I$)</td><td style='text-align: center; word-wrap: break-word;'>7.65</td><td style='text-align: center; word-wrap: break-word;'>7.35</td><td style='text-align: center; word-wrap: break-word;'>8.15</td><td style='text-align: center; word-wrap: break-word;'>8.51</td><td style='text-align: center; word-wrap: break-word;'>7.72</td><td style='text-align: center; word-wrap: break-word;'>37.6</td><td style='text-align: center; word-wrap: break-word;'>77.1</td><td style='text-align: center; word-wrap: break-word;'>91.0</td><td style='text-align: center; word-wrap: break-word;'>0.633</td><td style='text-align: center; word-wrap: break-word;'>0.979</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>+ Ours</td><td style='text-align: center; word-wrap: break-word;'>7.37</td><td style='text-align: center; word-wrap: break-word;'>7.11</td><td style='text-align: center; word-wrap: break-word;'>7.83</td><td style='text-align: center; word-wrap: break-word;'>7.80</td><td style='text-align: center; word-wrap: break-word;'>7.46</td><td style='text-align: center; word-wrap: break-word;'>39.1</td><td style='text-align: center; word-wrap: break-word;'>78.7</td><td style='text-align: center; word-wrap: break-word;'>92.0</td><td style='text-align: center; word-wrap: break-word;'>0.645</td><td style='text-align: center; word-wrap: break-word;'>0.983</td></tr></table>

<div style="text-align: center;">Table 1: Results of 3D hand pose estimation on HOGraspNet and HO3D. We report PA-MPJPE in millimeters for varying occlusion levels: Low (0–5 occluded joints), Medium (6–10), and High (11–21), together with PA-MPVPE in mm, PCK @5/10/15, and F@5/15 mm. The diffusion baselines (InterHandGen, w/ Full Cond.) are applied on top of WiLoR, while our refinement is applied on top of both WiLoR and Hamba. Bold and underline denote the best and second-best.</div>

Net's evaluation includes unseen subjects (S1 split [^4]) and HO3D comprises unseen views, subjects, and objects.

For baselines of 3D hand reconstruction, we test a graph-based transformer, Mesh-Graphormer [^28], an occlusion-robust network, HandOccNet [^39], two foundational transformers, HaMeR [^41] and WiLoR [^42], and a Mamba-based model, Hamba [^8]. For the diffusion prior, we train a 3D hand diffusion model, InterHandGen [^24], by adapting its architecture from two-hand input to a single-hand input for this task, which lacks any contextual conditions. For a fair comparison with our method, we further implement InterHandGen w/ Full Cond. modified to take the text and image conditions as global feature vectors. These diffusion baselines are trained from scratch and applied to refine 3D poses from WiLoR with the refinement method in Sec. 3.3.

We evaluate reconstruction performance using both 3D and 2D metrics. For 3D joints and mesh vertices, we report Procrustes-aligned mean errors (PA-MPJPE and PA-MPVPE). We also report mesh F-scores under 5 mm and 15 mm thresholds (F@5 and F@15), which evaluate surface reconstruction accuracy within the corresponding distance thresholds. For 2D joint accuracy, we report PCK@5, PCK@10, and PCK@15, which measure the percentage of keypoints whose 2D errors fall within 5, 10, and 15 pixels, respectively. To measure the model's robustness under increasing ambiguities, we further provide evaluation subsets according to different occlusion levels (Low, Medium, High), grouped by the number of visible joints detected in Sec. 3.3.

Implementation details: For generation, our attention layers follow a latent size of 512 and four heads. The loss weights are set to $\lambda_{\theta} = 1$, $\lambda_{v} = 10000$, and $\lambda_{j} =$

---

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'>(a) Text type</td><td style='text-align: center; word-wrap: break-word;'>RGB?</td><td style='text-align: center; word-wrap: break-word;'>Avg.</td><td style='text-align: center; word-wrap: break-word;'>Low</td><td style='text-align: center; word-wrap: break-word;'>Med.</td><td style='text-align: center; word-wrap: break-word;'>High</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>(1) N/A</td><td style='text-align: center; word-wrap: break-word;'>-</td><td style='text-align: center; word-wrap: break-word;'>9.28</td><td style='text-align: center; word-wrap: break-word;'>8.16</td><td style='text-align: center; word-wrap: break-word;'>9.82</td><td style='text-align: center; word-wrap: break-word;'>14.6</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>(2) N/A</td><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>7.90</td><td style='text-align: center; word-wrap: break-word;'>7.69</td><td style='text-align: center; word-wrap: break-word;'>7.89</td><td style='text-align: center; word-wrap: break-word;'>9.91</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>(3) Simple VLM</td><td style='text-align: center; word-wrap: break-word;'>-</td><td style='text-align: center; word-wrap: break-word;'>8.67</td><td style='text-align: center; word-wrap: break-word;'>8.00</td><td style='text-align: center; word-wrap: break-word;'>9.03</td><td style='text-align: center; word-wrap: break-word;'>11.4</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>(4) Simple VLM</td><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>7.76</td><td style='text-align: center; word-wrap: break-word;'>7.64</td><td style='text-align: center; word-wrap: break-word;'>7.70</td><td style='text-align: center; word-wrap: break-word;'>9.50</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>(5) Aff.</td><td style='text-align: center; word-wrap: break-word;'>-</td><td style='text-align: center; word-wrap: break-word;'>8.43</td><td style='text-align: center; word-wrap: break-word;'>7.62</td><td style='text-align: center; word-wrap: break-word;'>8.90</td><td style='text-align: center; word-wrap: break-word;'>11.5</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>(6) Aff.</td><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>7.53</td><td style='text-align: center; word-wrap: break-word;'>7.29</td><td style='text-align: center; word-wrap: break-word;'>7.57</td><td style='text-align: center; word-wrap: break-word;'>9.37</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>(7) Aff. w/ GT grasp</td><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>7.51</td><td style='text-align: center; word-wrap: break-word;'>7.28</td><td style='text-align: center; word-wrap: break-word;'>7.54</td><td style='text-align: center; word-wrap: break-word;'>9.35</td></tr></table>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'>(b) Text input</td><td style='text-align: center; word-wrap: break-word;'>Avg. $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>FPS $\uparrow$</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>(1) Qwen + Mistral</td><td style='text-align: center; word-wrap: break-word;'>7.53</td><td style='text-align: center; word-wrap: break-word;'>0.31</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>(2) Qwen keywords</td><td style='text-align: center; word-wrap: break-word;'>7.59</td><td style='text-align: center; word-wrap: break-word;'>0.39</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>(3) Grasp labels</td><td style='text-align: center; word-wrap: break-word;'>7.69</td><td style='text-align: center; word-wrap: break-word;'>1242</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>(4) Qwen (one-shot)</td><td style='text-align: center; word-wrap: break-word;'>7.68</td><td style='text-align: center; word-wrap: break-word;'>1.19</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>(5) Aff. classifier $^{*}$</td><td style='text-align: center; word-wrap: break-word;'>7.57</td><td style='text-align: center; word-wrap: break-word;'>96</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Seg. mask</td><td style='text-align: center; word-wrap: break-word;'>Avg. $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>FPS $\uparrow$</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>(6) SAM2 [^46]</td><td style='text-align: center; word-wrap: break-word;'>7.53</td><td style='text-align: center; word-wrap: break-word;'>8.1</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>(7) InterFormer [^51]</td><td style='text-align: center; word-wrap: break-word;'>7.54</td><td style='text-align: center; word-wrap: break-word;'>16.3</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>(8) None</td><td style='text-align: center; word-wrap: break-word;'>7.55</td><td style='text-align: center; word-wrap: break-word;'>—</td></tr></table>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'>(c) 2D guid.</td><td style='text-align: center; word-wrap: break-word;'>Avg. \downarrow</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>$\lambda_{{2d}}=0.005$</td><td style='text-align: center; word-wrap: break-word;'>7.53</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>$\lambda_{{2d}}=0$</td><td style='text-align: center; word-wrap: break-word;'>7.65</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>V+L enc.</td><td style='text-align: center; word-wrap: break-word;'>Avg. \downarrow</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>HaMeR+DistilBERT</td><td style='text-align: center; word-wrap: break-word;'>7.53</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>CLIP</td><td style='text-align: center; word-wrap: break-word;'>11.36</td></tr></table>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'>(d) Aff. field</td><td style='text-align: center; word-wrap: break-word;'>Avg. $\downarrow$</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Full</td><td style='text-align: center; word-wrap: break-word;'>7.53</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>w/o obj. category</td><td style='text-align: center; word-wrap: break-word;'>7.53</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>w/o interaction</td><td style='text-align: center; word-wrap: break-word;'>7.54</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>w/o intention</td><td style='text-align: center; word-wrap: break-word;'>7.54</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>w/o obj. shape</td><td style='text-align: center; word-wrap: break-word;'>7.56</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>w/o obj. size</td><td style='text-align: center; word-wrap: break-word;'>7.59</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>w/o grasp tax.</td><td style='text-align: center; word-wrap: break-word;'>7.71</td></tr></table>

<div style="text-align: center;">Table 2: Ablation study on HOGraspNet (PA-MPJPE). (a) Conditional modality compares text inputs and RGB image features. (b) Text input and visibility masks, (c) 2D keypoint guidance and V+L encoder, and (d) affordance fields;  $*$ denotes a lightweight classifier trained on GT-based affordance elements. Shaded rows mark our setting. Bold and  $\underline{\text{underline}}$ denote the best and second-best.</div>

10000. We set the number of tokens for text and image embeddings to 64 and 192, respectively. We train diffusion models for 80 epochs with a batch size of 64 using an Adam optimizer [^21] with a learning rate of $1e^{-3}$. We set the maximum diffusion steps to $T = 1000$ and use a cosine scheduler [^35] for the forward process. For refinement, we implement faster sampling techniques. We adaptively set the start refinement step $n \in [n_{min}, n_{max}]$ to the value chosen linearly according to the number of occluded joints. Larger occlusion counts result in larger values of n that require more refinement steps, conversely setting fewer steps in less occluded samples. We also sparsely sample every m step in the reverse process. On HOGraspNet, we set $\lambda_{2d} = 0.005$, $n = [100, 1000]$, $m = 20$. HO3D requires fewer refinement steps and increased 2D fitting as the initial predictions are more reliable: we set $\lambda_{2d} = 0.5$, $n = [10, 20]$, $m = 3$. Additional analysis of hyperparameter sensitivity is found in the supplement.

### 4.1 Results

3D hand pose estimation: Tab. 1 reports the 3D pose evaluation on HOGraspNet [^4] and HO3D [^15]. Recent foundation models outperform the CNN-based HandOccNet [^39], yet the high-occlusion regime remains challenging (~12 mm error on the “High” subset). The diffusion baseline InterHandGen [^24] barely corrects poses without contextual signals, and even “w/ Full Cond.” (image and text as global vectors) only partially improves the “Med.”/“High” splits over WiLoR [^42]. In contrast, our prior consistently improves WiLoR across all occlusion levels, reaching an average error of 7.53 mm, with a substantial 3.04 mm (24%) reduction on the “High” occlusion subset. This gain comes from fine-grained cross-attention to image and text tokens and occlusion-aware refinement, rather than attending conditions as coarse global vectors like “IHGen w/ Full Cond.” The same trend holds on HO3D: although WiLoR and Hamba are already highly competitive, our prior refines both without degradation on average, achieving 7.37 mm with WiLoR and the best 7.29 mm with Hamba. This confirms that the gains are not specific to a single base estimator, but generalize across different estimators.

Ablation studies: All ablations are conducted on HOGraspNet and reported in Tab. 2. We ablate the main components of our pipeline, including conditional modalities, affordance text generation, visibility masking, 2D keypoint guidance, encoder design, and individual affordance fields.

---

![Image](images/b30a53d0e43b91c0b279ed85238de9f2ceb4a6040465b79d70dc2ec6347864c9.jpg)

<div style="text-align: center;">Fig. 5: Qualitative results of diffusion-based refinement on the HOGraspNet dataset [^4]. We show refinement results with our diffusion prior and a comparison diffusion baseline, InterHandGen (IHGen) [^24] with Full Cond. The vertex error compared to the ground-truth mesh is color-coded, which is clipped to  $[0, 20]$ mm. While the initial hand pose from WiLoR [^42] struggles to estimate joints in occluded regions, our method reasonably refines the estimates to plausibly grasp the object even under occlusion. The values under the samples denote PA-MPJPE.</div>

(a) Conditional modality: We evaluate the impact of image and text conditioning, comparing the guidance effect from affordance descriptions with simple VLM captions. The Simple VLM text is generated by prompting Qwen2.5-VL [^2] to describe the given HOI scene (e.g., “The right hand holds a screwdriver”). Compared with results of (1, 3, 5), the simple caption method does not offer significant improvement over the baseline without text conditioning (N/A), while the affordance descriptions (Aff.) help reduce the ambiguities. Conditioning image features with ViT encoding (2, 4, 6) further proves to be effective, serving to align the predicted pose with visual evidence. We observe that our configuration (6) performs the best 7.53 mm by utilizing the affordance descriptions and the image features jointly. We also assess the effect of predicted grasp types (see “Step 2” of Sec. 3.1) against using ground-truth grasp labels at test time. We find the results (6, 7) almost on par, indicating that the predicted grasps in (6) include few failures in classification and do not negatively affect the downstream task.

(b) Text input & segmentation mask: For the text input, our two-stage Qwen+Mistral description (1), where the VLM extracts affordance keywords that Mistral summarizes into a coherent sentence, outperforms three simpler alternatives: raw VLM keywords without summarization (2), grasp taxonomy labels alone (3), and a single-shot VLM prompt (4). This confirms that the structured summarization provides a stronger condition.

---

![Image](images/3fc69edae897673eecf54322e8851fdea95527a274be70004ec72dddf8417954.jpg)

<div style="text-align: center;">Fig. 6: Qualitative results of diffusion-based refinement on the HO3D dataset [^15]. We compare the refinement results of our diffusion prior against a diffusion baseline, InterHandGen (IHGen) [^24] with Full Cond., both applied to the initial hand poses predicted by WiLoR [^42]. Predicted hand joints are colored by finger, with ground truth displayed in black. Some joint errors are highlighted using colored ellipses. The values under the samples denote PA-MPJPE.</div>

tioning signal than flat labels or unsummarized text. A lightweight affordance classifier trained with GT-based affordance elements is then used to generate the affordance descriptions (5), recovering most of the accuracy (7.57 mm) while running at 96 fps. For the visibility mask, replacing SAM2 [^46] (6) with the faster InterFormer [^51] (7) doubles throughput at a negligible accuracy cost, and removing the mask entirely (8) only mildly degrades performance.

(c) 2D guidance & V+L encoder: The 2D keypoint guidance ( $\lambda_{2d}=0.005$) grounds the refinement to visual evidence during inference, and removing this guidance by setting $\lambda_{2d}=0$ degrades the accuracy. We further ablate the V+L encoder, which denotes the vision and language encoding modules used to condition our diffusion prior. Our modality-specific encoder, implemented with HaMeR for visual features and DistilBERT for textual affordance descriptions, is also essential to performance. In contrast, replacing these task-specific encoders with a CLIP encoder substantially degrades the error to 11.36 mm. This result suggests that CLIP's global embeddings lack the hand-specific spatial granularity required to distinguish subtle hand-object interactions.

(d) Affordance fields: Ablating each field of the affordance description shows that the grasp taxonomy contributes the most to the refinement. The object category, shape, size, interaction, and intention also provide complementary gains, highlighting the benefit of combining multiple elements for accurate refinement.

---

![Image](images/559a7729e784216dae531357637ad962cd43c3ac2c4102601a71f6ae09c4c15d.jpg)

<div style="text-align: center;">Fig. 7: In-the-wild sample evaluation on the HInt dataset [^41]. We present PCK (%) to the 2D keypoint GT (black skeleton). The highlighted circle and color in 2D kpt indicate the 2D joint error and its finger type (red circle means thumb error).</div>

<div style="text-align: center;">Fig. 8: Qualitative results of our diffusion prior. We show samples generated from partially different descriptions. We find that the diffusion-based generation responds to the change of object size, while remaining consistent with the given description.</div>

Qualitative results: Fig. 5 presents qualitative comparisons on HOGraspNet between the WiLoR predictions, the refined poses by InterHandGen (IHGen) with Full Cond., and our diffusion prior. The initial poses exhibit large errors (red vertices) in the occluded parts (e.g., the ring and pinky fingers are often hidden in the examples). Our method corrects these errors and improves over the IHGen (w/ Full Cond.) baseline.

Similarly, Fig. 6 visualizes qualitative comparisons on the HO3D dataset. As highlighted by the ellipses, the initial WiLoR predictions suffer from high errors at severely occluded joints. While the diffusion baseline, IHGen (w/ Full Cond.), attempts to correct them, it struggles to effectively reflect the textual affordance descriptions into the refined poses. In contrast, our diffusion prior successfully leverages these text cues, effectively correcting the hidden articulations to form plausible grasps.

Fig. 7 shows in-the-wild evaluation on the HInt dataset [^41], including proxy 2D keypoint evaluation (PCK). We find stable captioning with generalized VLM capability and successful diffusion-based refinement over WiLoR with improved PCK metrics. This suggests the applicability of our method in such unconstrained cooking environments where the affordance signals are abundant.

Controllability of diffusion-based generation: Fig. 8 presents qualitative examples of our diffusion model given different textual descriptions. We validate how pose generation responds to different affordance elements with the rest of the description fixed. Specifically, we alter only the object size (small vs. large), demonstrating that our model can produce hand poses that adapt to the size change, such as wider grasps for large objects and tighter grasps for small ones. This suggests that our diffusion prior can faithfully incorporate actionable semantics embedded in the affordance descriptions.

Object-property-wise evaluation: We provide further granular evaluations on [^4] with different object types and sizes. As shown in Fig. 9, while WiLoR suffers large errors on medium cylinders, large disks, and articulated shapes, our method consistently achieves robust refinement across these challenging categories.

---

![Image](images/cfb4e273c863fedab6296f656f5429d109d874c7df749999e4c0bf6a06c65e9a.jpg)

<div style="text-align: center;">Fig. 9: Object property-wise performance evaluation on HOGraspNet. The cell value "WiLoR/Ours (gain%)" shows the WiLoR baseline, our method, and the relative gain in PA-MPJPE. The x-axis represents cylinder, sphere, disk, cuboid, and articulated shapes.</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'>Method</td><td style='text-align: center; word-wrap: break-word;'>Unit</td><td style='text-align: center; word-wrap: break-word;'>GFLOPs $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>Throughput $\uparrow$ (frame/sec)</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>WiLoR</td><td style='text-align: center; word-wrap: break-word;'>per sample</td><td style='text-align: center; word-wrap: break-word;'>138.06</td><td style='text-align: center; word-wrap: break-word;'>21.0</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>AffHandGen</td><td style='text-align: center; word-wrap: break-word;'>per diff. step</td><td style='text-align: center; word-wrap: break-word;'>0.99</td><td style='text-align: center; word-wrap: break-word;'>389.1</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>AffHandGen</td><td style='text-align: center; word-wrap: break-word;'>per sample (≈8 steps)</td><td style='text-align: center; word-wrap: break-word;'>7.94</td><td style='text-align: center; word-wrap: break-word;'>48.6</td></tr></table>

<div style="text-align: center;">Table 3: Computational cost evaluation on HOGraspNet. We measure the inference burden of foundational transformers and our diffusion model. As it sparsely samples diffusion steps in inference, our method only requires on average approximately 8 diffusion steps per sample, resulting in significantly lower GFLOPs and higher throughput compared to WiLoR.</div>

Inference cost: Owing to our adaptive start steps in refinement (n in Fig. 4) and sparse time sampling, our method does not worsen the inference burden in GFLOPs and Throughput compared to the foundation model WiLoR in Tab. 3.

## 5 Conclusion

We present a novel diffusion prior that leverages affordance descriptions generated with VLM's reasoning and grasp classification. Our model learns the distribution of plausible and functionally coherent 3D hand poses, and then is adapted to refine 3D pose estimates, enabling robust reasoning about occluded joints and more accurate reconstructions in challenging hand-object interactions. Our experiments demonstrate that the guidance by affordance descriptions substantially improves the refinement quality over both foundational transformer methods and diffusion priors, with strong gains under severe occlusions. Our method also provides controllable and interpretable refinements, aligning pose generation with affordance descriptions.

We note some limitations that open interesting avenues for future work. Our method currently focuses on static single-hand reconstruction and does not yet explicitly account for 3D object geometry, relying instead on textual descriptions as a proxy to include the object information. Extending the approach to explicitly model 3D object interactions is promising for future work. Furthermore, extending our affordance-conditioned prior from static images to continuous temporal domains, such as egocentric video analysis and human motion synthesis, is a highly promising and exciting direction for modeling dynamic hand-object interactions.

## Acknowledgements

This work was supported by JST K Program Grant Number JPMJKP25V1 and JST ASPIRE Grant Number JPMJAP2303.

---

## References

[^1]: Bahl, S., Mendonca, R., Chen, L., Jain, U., Pathak, D.: Affordances from human videos as a versatile representation for robotics. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 13778–13790 (2023)

[^2]: Bai, S., Chen, K., Liu, X., Wang, J., Ge, W., Song, S., Dang, K., Wang, P., Wang, S., Tang, J., et al.: Qwen2. 5-xl technical report. arXiv preprint arXiv:2502.13923 (2025)

[^3]: Cha, J., Kim, J., Yoon, J.S., Baek, S.: Text2hoi: Text-guided 3d motion generation for hand-object interaction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 1577–1585 (2024)

[^4]: Cho, W., Lee, J., Yi, M., Kim, M., Woo, T., Kim, D., Ha, T., Lee, H., Ryu, J.H., Woo, W., et al.: Dense hand-object (ho) graspnet with full grasping taxonomy and dynamics. In: Proceedings of the European Conference on Computer Vision (ECCV). pp. 284–303 (2024)

5. Christen, S., Hampali, S., Sener, F., Remelli, E., Hodan, T., Sauer, E., Ma, S., Tekin, B.: Diff $^{2}$o: Diffusion-based synthesis of hand-object interactions from textual descriptions. In: Proceedings of the ACM SIGGRAPH Asia Conference. pp. 1–11 (2024)

[^6]: Corona, E., Pumarola, A., Alenya, G., Moreno-Noguer, F., Rogez, G.: Ganhand: Predicting human grasp affordances in multi-object scenes. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 5031–5041 (2020)

[^7]: Do, T.T., Nguyen, A., Reid, I.: Affordancenet: An end-to-end deep learning approach for object affordance detection. In: Proceedings of the IEEE International Conference on Robotics and Automation (ICRA). pp. 5882–5889 (2018)

[^8]: Dong, H., Chharia, A., Gou, W., Carrasco, F.V., De la Torre, F.: Hamba: Single-view 3d hand reconstruction with graph-guided bi-scanning mamba. In: Proceedings of the Advances in Neural Information Processing Systems (NeurIPS). vol. 37, pp. 2127–2160 (2024)

9. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., Housby, N.: An image is worth $16 \times 16$ words: Transformers for image recognition at scale. In: Proceedings of the International Conference on Learning Representations (ICLR). (2021)

[^10]: Fan, Z., Ohkawa, T., Yang, L., Lin, N., Zhou, Z., Zhou, S., Liang, J., Gao, Z., Zhang, X., Zhang, X., et al.: Benchmarks and challenges in pose estimation for egocentric hand interactions with objects. In: Proceedings of the European Conference on Computer Vision (ECCV). pp. 428–448 (2024)

[^11]: Fang, K., Wu, T.L., Yang, D., Savarese, S., Lim, J.J.: Demo2vec: Reasoning object affordances from online videos. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 2139–2147 (2018)

[^12]: Feix, T., Romero, J., Schmiedmayer, H.B., Dollar, A.M., Kragic, D.: The grasp taxonomy of human grasp types. IEEE Transactions on human-machine systems 46(1), 66–77 (2015)

[^13]: Gibson, J.J.: The theory of affordances: (1979). In: The people, place, and space reader, pp. 56–60. Routledge (2014)

[^14]: Goyal, M., Modi, S., Goyal, R., Gupta, S.: Human hands as probes for interactive object understanding. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 3293–3303 (2022)

[^15]: Hampali, S., Rad, M., Oberweger, M., Lepetit, V.: Honnotate: A method for 3d annotation of hand and object poses. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 3196–3206 (2020)

[^16]: Ho, J., Jain, A., Abbeel, P.: Denoising Diffusion Probabilistic Models. In: Proceedings of the Advances in Neural Information Processing Systems (NeurIPS). vol. 33, pp. 6840–6851 (2020)

---

[^17]: Ho, J., Salimans, T.: Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598 (2022)

[^18]: Jian, J., Liu, X., Li, M., Hu, R., Liu, J.: Affordpose: A large-scale dataset of hand-object interactions with affordance-driven hand pose. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). pp. 14713–14724 (2023)

[^19]: Jiang, A.Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D.S., de Las Casas, D., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., Lavaud, L.R., Lachaux, M., Stock, P., Scao, T.L., Lavril, T., Wang, T., Lacroix, T., Sayed, W.E.: Mistral 7b. arXiv preprint arXiv:2310.06825 (2024)

[^20]: Khirodkar, R., Bagautdinov, T.M., Martinez, J., Zhaoen, S., James, A., Selednik, P., Anderson, S., Saito, S.: Sapiens: Foundation for human vision models. In: Proceedings of the European Conference on Computer Vision (ECCV). vol. 15062, pp. 206–228 (2024)

[^21]: Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. In: Proceedings of the International Conference on Learning Representations (ICLR). (2015)

[^22]: Koppula, H.S., Gupta, R., Saxena, A.: Learning human activities and object affordances from rgb-d videos. The International journal of robotics research 32(8), 951–970 (2013)

[^23]: Kulon, D., Guler, R.A., Kokkinos, I., Bronstein, M.M., Zafeiriou, S.: Weakly-supervised mesh-convolutional hand reconstruction in the wild. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 4990–5000 (2020)

[^24]: Lee, J., Saito, S., Nam, G., Sung, M., Kim, T.K.: Interhandgen: Two-hand interaction generation via cascaded reverse diffusion. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 527–537 (2024)

[^25]: Li, G., Sun, D., Sevilla-Lara, L., Jampani, V.: One-shot open affordance learning with foundation models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 3086–3096 (2024)

[^26]: Li, G., Tsagkas, N., Song, J., Mon-Williams, R., Vijayakumar, S., Shao, K., Sevilla-Lara, L.: Learning precise affordances from egocentric videos for robotic manipulation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). pp. 10581–10591 (2025)

[^27]: Li, M., Christen, S., Wan, C., Cai, Y., Liao, R., Sigal, L., Ma, S.: Latenthoi: On the generalizable hand object motion generation with latent hand diffusion. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 17416–17425 (2025)

onal

[^29]: Liu, A.L., Chao, Y.W., Chen, Y.T.: Task-oriented human grasp synthesis via context- and task-aware diffusers. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). pp. 10375–10385 (2025)

[^30]: Liu, Y., Liu, Y., Jiang, C., Lyu, K., Wan, W., Shen, H., Liang, B., Fu, Z., Wang, H., Yi, L.: Hoi4d: A 4d egocentric dataset for category-level human-object interaction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 21013–21022 (2022)

[^31]: Lu, J., Lin, J., Dou, H., Zeng, A., Deng, Y., Liu, X., Cai, Z., Yang, L., Zhang, Y., Wang, H., et al.: Dposer-x: Diffusion model as robust 3d whole-body human pose prior. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). pp. 9988–9997 (2025)

[^32]: Lv, X., Xu, L., Yan, Y., Jin, X., Xu, C., Wu, S., Liu, Y., Li, L., Bi, M., Zeng, W., et al.: Himo: A new benchmark for full-body human interacting with multiple objects. In: Proceedings of the European Conference on Computer Vision (ECCV). pp. 300–318 (2024)

---

[^33]: Müller, L., Ye, V., Pavlakos, G., Black, M., Kanazawa, A.: Generative proxemics: A prior for 3d social interaction from images. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 9687–9697 (2024)

[^34]: Nagarajan, T., Feichtenhofer, C., Grauman, K.: Grounded human-object interaction hotspots from video. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). pp. 8688–8697 (2019)

[^35]: Nichol, A.Q., Dhariwal, P.: Improved denoising diffusion probabilistic models. In: Proceedings of the International Conference on Machine Learning (ICML). vol. 139, pp. 8162–8171 (2021)

[^36]: Ohkawa, T., Furuta, R., Sato, Y.: Efficient annotation and learning for 3d hand pose estimation: A survey. International Journal of Computer Vision (IJCV) 131, 3193–3206 (2023)

[^37]: Ohkawa, T., He, K., Sener, F., Hodan, T., Tran, L., Keskin, C.: Assemblyhands: Towards egocentric activity understanding via 3d hand pose estimation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 12999–13008 (2023)

[^38]: Ohkawa, T., Lee, J., Saito, S., Saragih, J., Prado, F., Xu, Y., Yu, S.I., Furuta, R., Sato, Y., Shiratori, T.: Generative modeling of shape-dependent self-contact human poses. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). pp. 5426–5436 (2025)

[^39]: Park, J., Oh, Y., Moon, G., Choi, H., Lee, K.M.: Handoccnet: Occlusion-robust 3d hand mesh estimation network. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 1496–1505 (2022)

[^40]: Patel, C., Nakamura, H., Kyuragi, Y., Kozuka, K., Niebles, J.C., Adeli, E.: Unigomotion: A unified model for egocentric motion reconstruction, forecasting, and generation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). pp. 10318–10329 (2025)

[^41]: Pavlakos, G., Shan, D., Radosavovic, I., Kanazawa, A., Fouhey, D., Malik, J.: Reconstructing hands in 3d with transformers. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 9826–9836 (2024)

[^42]: Potamias, R.A., Zhang, J., Deng, J., Zafeiriou, S.: Wilor: End-to-end 3d hand localization and reconstruction in-the-wild. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 12242–12254 (2025)

[^43]: Prakash, A., Lundell, B., Andreychuk, D., Forsyth, D., Gupta, S., Sawhney, H.: How do I do that? synthesizing 3d hand motion and contacts for everyday interactions. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 7026–7036 (2025)

[^44]: Prakash, A., Tu, R., Chang, M., Gupta, S.: 3d hand pose estimation in everyday egocentric images. In: Proceedings of the European Conference on Computer Vision (ECCV). pp. 183–202 (2024)

[^45]: Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: Proceedings of the International Conference on Machine Learning (ICML). pp. 8748–8763 (2021)

[^46]: Ravi, N., Gabeur, V., Hu, Y.T., Hu, R., Ryali, C., Ma, T., Khedr, H., Rädle, R., Rolland, C., Gustafson, L., et al.: Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00714 (2024)

[^47]: Romero, J., Tzionas, D., Black, M.J.: Embodied hands: modeling and capturing hands and bodies together. ACM Transactions on Graphics (ToG) 36(6), 245:1–245:17 (2017)

[^48]: Sanh, V., Debut, L., Chaumond, J., Wolf, T.: Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108 (2019)

---

[^49]: Smith, B., Wu, C., Wen, H., Peluse, P., Sheikh, Y., Hodgins, J.K., Shiratori, T.: Constraining dense hand surface tracking with elasticity. ACM Transactions on Graphics (ToG) 39(6), 219:1–219:14 (2020)

[^50]: Stathopoulos, A., Han, L., Metaxas, D.: Score-guided diffusion for 3d human recovery. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 906–915 (2024)

[^51]: Su, Y., Wang, Y., Yao, L., Cui, Y., Chau, L.P.: Interaction-aware representation modeling with co-occurrence consistency for egocentric hand-object parsing. In: Proceedings of the International Conference on Learning Representations (ICLR). (2026)

[^52]: Taheri, O., Ghorbani, N., Black, M.J., Tzionas, D.: GRAB: A dataset of whole-body human grasping of objects. In: Proceedings of the European Conference on Computer Vision (ECCV). pp. 581–600 (2020)

[^53]: Tevet, G., Gordon, B., Hertz, A., Bermano, A.H., Cohen-Or, D.: Motionclip: Exposing human motion generation to clip space. In: Proceedings of the European Conference on Computer Vision (ECCV). pp. 358–374 (2022)

[^54]: Tevet, G., Raab, S., Gordon, B., Shafir, Y., Cohen-Or, D., Bermano, A.H.: Human motion diffusion model. In: Proceedings of the International Conference on Learning Representations (ICLR) (2023)

[^55]: Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I.: Attention is all you need. In: Proceedings of the Advances in Neural Information Processing Systems (NeurIPS). vol. 30 (2017)

[^56]: Wang, Y., Sun, Y., Patel, P., Daniilidis, K., Black, M.J., Kocabas, M.: Prompthmr: Promptable human mesh recovery. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 1148–1159 (2025)

[^57]: Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q.V., Zhou, D., et al.: Chain-of-thought prompting elicits reasoning in large language models. In: Proceedings of the Advances in Neural Information Processing Systems (NeurIPS). vol. 35, pp. 24824–24837 (2022)

[^58]: Wei, Y.L., Jiang, J.J., Xing, C., Tan, X.T., Wu, X.M., Li, H., Cutkosky, M., Zheng, W.S.: Grasp as you say: Language-guided dexterous grasp generation. In: Proceedings of the Advances in Neural Information Processing Systems (NeurIPS). vol. 37, pp. 46881–46907 (2024)

[^59]: Wen, Y., Pan, H., Yang, L., Pan, J., Komura, T., Wang, W.: Hierarchical temporal transformer for 3d hand pose estimation and action recognition from egocentric RGB videos. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 21243–21253 (2023)

[^60]: Yang, L., Li, K., Zhan, X., Wu, F., Xu, A., Liu, L., Lu, C.: Oakink: A large-scale knowledge repository for understanding hand-object interaction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 20953–20962 (2022)

[^61]: Ye, Y., Gupta, A., Kitani, K., Tulsiani, S.: G-HOP: generative hand-object prior for interaction reconstruction and grasp synthesis. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 1911–1920 (2024)

[^62]: Ye, Y., Li, X., Gupta, A., De Mello, S., Birchfield, S., Song, J., Tulsiani, S., Liu, S.: Affordance diffusion: Synthesizing hand-object interactions. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 22479–22489 (2023)

[^63]: Yu, Z., Huang, Y., Furuta, R., Yagi, T., Goutsu, Y., Sato, Y.: Fine-grained affordance annotation for egocentric hand-object interaction videos. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). pp. 2155–2163 (2023)

[^64]: Zhang, X., Li, Q., Mo, H., Zhang, W., Zheng, W.: End-to-end hand mesh recovery from a monocular rgb image. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). pp. 2354–2364 (2019)

---

[^65]: Zhang, Y., Cui, Z., Kephart, J.O., Ji, Q.: Diffusion-based 3d hand motion recovery with intuitive physics. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). pp. 7306–7317 (2025)

[^66]: Zhou, B., Zhan, Y., Zhang, Z., Lu, Z.: MEgoHand: Multimodal egocentric hand-object interaction motion generation. In: Proceedings of the Advances in Neural Information Processing Systems (NeurIPS). vol. 38, pp. 49464–49490 (2025)

[^67]: Zhou, Y., Ohkawa, T., Zhou, G., Goto, K., Hirose, T., Sekikawa, Y., Inoue, N.: Df-mamba: Deformable state space modeling for 3d hand pose estimation in interactions. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). pp. 5352–5363 (2026)

---

# Affordance-Guided Diffusion Prior for 3D Hand Reconstruction — Supplementary Material —

## A Grasp Classification

As mentioned in Sec. 3.1, we adopt a grasp taxonomy classifier to classify grasp types from a single image. The overall architecture for grasp taxonomy classification is illustrated in Fig. 10. The input image is first encoded by a ViT to obtain visual tokens. In parallel, a MANO regressor predicts the hand pose parameters from the encoded visual tokens. Since grasp taxonomy depends not only on the hand pose but also on the visual context surrounding the hand-object interaction, we combine the visual tokens and the MANO pose parameters as a joint input to a self-attention module. We use the ViT encoder and the MANO regressor pretrained on [^41]. The output of the self-attention module is then fed into an MLP layer to produce the final grasp taxonomy classification. For training, we use the HOGraspNet [^4] dataset, adopting the same data size and train/validation split as in the AffHandGen training setup.

The overall classification accuracy is 87.6% on the evaluation split. We show the confusion matrix for grasp taxonomy classification in Fig. 13, where the original 33 grasp types are grouped into 6 categories based on grasp power and thumb posture, following the grouping introduced in [^63] and Fig. 14. While predicted taxonomies are mostly accurate, we find some failure examples of our classification model in Fig. 11. These failure cases correspond to grasp types that the model frequently confuses in the confusion matrix. Although the predictions are incorrect, the misclassified grasp types are visually and functionally similar. As shown in Tab. 2 rows (6) and (7), the performance gain when replacing predicted grasps with the ground-truth is marginal, indicating that these failures in grasp classification have limited practical impact.

![Image](images/47f758f3ac1480241c2e7630853534f63ebd99e278c74147151e4ebc45618dc6.jpg)

<div style="text-align: center;">Fig. 10: Architecture of grasp classification.</div>

![Image](images/f8e84109a0cff1bb46525f6bd89d471a17c47c21bcd455eef6121e2f012f62a0.jpg)

<div style="text-align: center;">Fig. 11: Failure cases of grasp classification.</div>

---

![Image](images/9db5273b6c2b753e64329228044d3f0ebc2bfccf93b954fba1edb3098d2111e7.jpg)

"A right hand, with an adducted thumb, is holding a medium-sized, cylindrical banana."

<div style="text-align: center;">Fig. 12: Examples of generated simple captions and affordance-aware descriptions.</div>

![Image](images/8fcea6e8aa3216c68c47cf4b154d96336785b4f3a11d5ae4171cf7a3ecf29431.jpg)

<div style="text-align: center;">Fig. 13: Confusion matrix of grasp classification.</div>

![Image](images/851d5b967435590837ace5ef37015f50da2c87dd2eaf7a60fa505cfb10fec4a8.jpg)

<div style="text-align: center;">Fig. 14: The 33 grasp labels [^12] are grouped into 6 categories. The figure is adapted from [^63].</div>

## B Description Generation

We design two types of descriptions in Sec. 3.1: a simple caption and an affordance-aware description. These descriptions are generated using a VLM (Qwen2.5-VL [^2]). Examples of both types of generated descriptions are provided in Fig. 12. Compared to the simple captions, our affordance-aware descriptions offer more detailed explanations of hand-object interaction, including grasp taxonomy, object size and shape, and other interaction-relevant attributes.

For simple caption generation, we feed the image together with the hand-object bounding boxes into the VLM and directly prompt it using the template shown in Fig. 15. In contrast, the affordance-aware description integrates additional structured cues. Specifically, we first feed the image and the hand-object bounding boxes into the VLM with the prompt shown in Fig. 16. In parallel, we perform grasp classifica-

---

tion on the input image and insert the predicted grasp taxonomy into a parsed caption template. The combined parsed caption is then provided to an LLM (Mistral-7B [^19]), which summarizes it into an affordance-aware description. We prompt the LLM with “Summarize the given parsed captions in a single sentence starting with ‘A right/left hand...’”, and use the resulting summary as our caption.

![Image](images/14ca22991e1b0b1187c2f4590a5f1908ef4e7d53c933f6c767813c48478bc7e5.jpg)

<div style="text-align: center;">Fig. 15: Prompt used to generate a simple caption.</div>

![Image](images/3b4d6a518a542c896ae0bded27ed76c494b5cfa89854d53d18e90f425a7146d4.jpg)

<div style="text-align: center;">Fig. 16: Prompt used to generate an affordance-aware description.</div>

---

![Image](images/a609f558502165e7f8d91efda43978c1fbdda9c030e0ad83a1c846bbd420b523.jpg)

<div style="text-align: center;">Fig. 17: Sensitivity analysis of the 2D reprojection loss weight  $\lambda_{2d}$ on the HOGraspNet dataset.</div>

## C Additional Results

Hyperparameter sensitivity: In the refinement stage (Sec. 3.3), we incorporate a 2D reprojection loss weighted by $\lambda_{2d}$ to encourage consistency between the optimized 3D hand pose and 2D keypoint estimates from an off-the-shelf 3D hand reconstruction method. We analyze the model's sensitivity to $\lambda_{2d}$ on the HOGraspNet dataset, keeping the maximum diffusion steps $T=1000$, start step $n \in [100, 1000]$, sampling interval $m=20$, and 100 post-processing steps, with image conditioning enabled. As shown in Fig. 17, sweeping $\lambda_{2d}$ across four orders of magnitude $(10^{-4}-10^{-1})$ reveals a broad plateau of near-optimal performance in the range of $10^{-4}$ to $10^{-2}$. We achieve the best PA-MPIPE of 7.51 mm at $\lambda_{2d}=0.005$ (compared to 8.22 mm before refinement). This broad stability region demonstrates that our refinement pipeline is highly robust and obviates the need for exhaustive hyperparameter tuning in practice.

In-the-wild results: To further assess generalization beyond the training datasets, we evaluate our method on HInt [^41], which includes challenging in-the-wild images from the New Days, VISOR, and Ego4D splits. As shown in Tab. 4, our prior improves WiLoR on occluded joints across all splits and HaMeR on most splits, indicating its effectiveness under unconstrained real-world ambiguities. We further present qualitative results in Fig. 19, where our method produces plausible hand poses that remain consistent with the interaction context across diverse real-world scenarios.

Failure cases: We finally inspect cases where our prior offers limited benefit (Fig. 18). For interactions far from the training distribution, such as unusual grasps or rare object affordances in in-the-wild HInt images, affordance descriptions can become less reliable, leading to weaker refinement over the initial WiLoR estimate. Broadening the affordance vocabulary and grounding it in explicit 3D object geometry may help close this gap, as discussed in the limitations below.

Text-conditional generation: In Fig. 20, we provide the qualitative results of our diffusion-based pose sampling from the text input. Our diffusion prior generates hand

---

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td rowspan="2" colspan="2">Method</td><td colspan="3">New Days</td><td colspan="3">VISOR</td><td colspan="3">Ego4D</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>@0.05 $\uparrow$</td><td style='text-align: center; word-wrap: break-word;'>@0.1 $\uparrow$</td><td style='text-align: center; word-wrap: break-word;'>@0.15 $\uparrow$</td><td style='text-align: center; word-wrap: break-word;'>@0.05 $\uparrow$</td><td style='text-align: center; word-wrap: break-word;'>@0.1 $\uparrow$</td><td style='text-align: center; word-wrap: break-word;'>@0.15 $\uparrow$</td><td style='text-align: center; word-wrap: break-word;'>@0.05 $\uparrow$</td><td style='text-align: center; word-wrap: break-word;'>@0.1 $\uparrow$</td><td style='text-align: center; word-wrap: break-word;'>@0.15 $\uparrow$</td></tr><tr><td rowspan="7">All</td><td style='text-align: center; word-wrap: break-word;'>HandOccNet [^39]</td><td style='text-align: center; word-wrap: break-word;'>7.1</td><td style='text-align: center; word-wrap: break-word;'>22.0</td><td style='text-align: center; word-wrap: break-word;'>37.8</td><td style='text-align: center; word-wrap: break-word;'>5.5</td><td style='text-align: center; word-wrap: break-word;'>18.6</td><td style='text-align: center; word-wrap: break-word;'>33.2</td><td style='text-align: center; word-wrap: break-word;'>2.9</td><td style='text-align: center; word-wrap: break-word;'>10.2</td><td style='text-align: center; word-wrap: break-word;'>19.6</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>MeshGraphormer [^28]</td><td style='text-align: center; word-wrap: break-word;'>15.0</td><td style='text-align: center; word-wrap: break-word;'>34.7</td><td style='text-align: center; word-wrap: break-word;'>49.1</td><td style='text-align: center; word-wrap: break-word;'>15.9</td><td style='text-align: center; word-wrap: break-word;'>39.0</td><td style='text-align: center; word-wrap: break-word;'>55.2</td><td style='text-align: center; word-wrap: break-word;'>7.2</td><td style='text-align: center; word-wrap: break-word;'>19.8</td><td style='text-align: center; word-wrap: break-word;'>31.8</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Hamba [^8]</td><td style='text-align: center; word-wrap: break-word;'>25.1</td><td style='text-align: center; word-wrap: break-word;'>54.3</td><td style='text-align: center; word-wrap: break-word;'>70.7</td><td style='text-align: center; word-wrap: break-word;'>19.7</td><td style='text-align: center; word-wrap: break-word;'>49.8</td><td style='text-align: center; word-wrap: break-word;'>69.1</td><td style='text-align: center; word-wrap: break-word;'>12.0</td><td style='text-align: center; word-wrap: break-word;'>33.0</td><td style='text-align: center; word-wrap: break-word;'>49.0</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>HaMeR [^41]</td><td style='text-align: center; word-wrap: break-word;'>27.7</td><td style='text-align: center; word-wrap: break-word;'>54.3</td><td style='text-align: center; word-wrap: break-word;'>68.9</td><td style='text-align: center; word-wrap: break-word;'>25.7</td><td style='text-align: center; word-wrap: break-word;'>55.2</td><td style='text-align: center; word-wrap: break-word;'>71.3</td><td style='text-align: center; word-wrap: break-word;'>18.3</td><td style='text-align: center; word-wrap: break-word;'>41.6</td><td style='text-align: center; word-wrap: break-word;'>56.4</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>+ Ours</td><td style='text-align: center; word-wrap: break-word;'>24.6</td><td style='text-align: center; word-wrap: break-word;'>52.6</td><td style='text-align: center; word-wrap: break-word;'>68.5</td><td style='text-align: center; word-wrap: break-word;'>24.3</td><td style='text-align: center; word-wrap: break-word;'>55.8</td><td style='text-align: center; word-wrap: break-word;'>72.8</td><td style='text-align: center; word-wrap: break-word;'>17.5</td><td style='text-align: center; word-wrap: break-word;'>42.3</td><td style='text-align: center; word-wrap: break-word;'>58.0</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>WiLoR [^42]</td><td style='text-align: center; word-wrap: break-word;'>24.9</td><td style='text-align: center; word-wrap: break-word;'>54.9</td><td style='text-align: center; word-wrap: break-word;'>71.6</td><td style='text-align: center; word-wrap: break-word;'>21.9</td><td style='text-align: center; word-wrap: break-word;'>53.5</td><td style='text-align: center; word-wrap: break-word;'>72.6</td><td style='text-align: center; word-wrap: break-word;'>12.5</td><td style='text-align: center; word-wrap: break-word;'>34.2</td><td style='text-align: center; word-wrap: break-word;'>50.3</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>+ Ours</td><td style='text-align: center; word-wrap: break-word;'>23.6</td><td style='text-align: center; word-wrap: break-word;'>54.5</td><td style='text-align: center; word-wrap: break-word;'>72.8</td><td style='text-align: center; word-wrap: break-word;'>21.7</td><td style='text-align: center; word-wrap: break-word;'>54.9</td><td style='text-align: center; word-wrap: break-word;'>74.8</td><td style='text-align: center; word-wrap: break-word;'>12.5</td><td style='text-align: center; word-wrap: break-word;'>35.6</td><td style='text-align: center; word-wrap: break-word;'>52.5</td></tr><tr><td rowspan="7">Visible</td><td style='text-align: center; word-wrap: break-word;'>HandOccNet [^39]</td><td style='text-align: center; word-wrap: break-word;'>8.4</td><td style='text-align: center; word-wrap: break-word;'>25.6</td><td style='text-align: center; word-wrap: break-word;'>43.0</td><td style='text-align: center; word-wrap: break-word;'>6.2</td><td style='text-align: center; word-wrap: break-word;'>19.8</td><td style='text-align: center; word-wrap: break-word;'>34.7</td><td style='text-align: center; word-wrap: break-word;'>3.1</td><td style='text-align: center; word-wrap: break-word;'>10.7</td><td style='text-align: center; word-wrap: break-word;'>20.5</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>MeshGraphormer [^28]</td><td style='text-align: center; word-wrap: break-word;'>18.0</td><td style='text-align: center; word-wrap: break-word;'>40.3</td><td style='text-align: center; word-wrap: break-word;'>55.0</td><td style='text-align: center; word-wrap: break-word;'>21.9</td><td style='text-align: center; word-wrap: break-word;'>49.5</td><td style='text-align: center; word-wrap: break-word;'>64.6</td><td style='text-align: center; word-wrap: break-word;'>9.4</td><td style='text-align: center; word-wrap: break-word;'>24.1</td><td style='text-align: center; word-wrap: break-word;'>36.1</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Hamba [^8]</td><td style='text-align: center; word-wrap: break-word;'>30.7</td><td style='text-align: center; word-wrap: break-word;'>63.5</td><td style='text-align: center; word-wrap: break-word;'>78.8</td><td style='text-align: center; word-wrap: break-word;'>25.7</td><td style='text-align: center; word-wrap: break-word;'>60.9</td><td style='text-align: center; word-wrap: break-word;'>79.0</td><td style='text-align: center; word-wrap: break-word;'>14.7</td><td style='text-align: center; word-wrap: break-word;'>39.5</td><td style='text-align: center; word-wrap: break-word;'>56.3</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>HaMeR [^41]</td><td style='text-align: center; word-wrap: break-word;'>33.2</td><td style='text-align: center; word-wrap: break-word;'>62.3</td><td style='text-align: center; word-wrap: break-word;'>76.1</td><td style='text-align: center; word-wrap: break-word;'>33.0</td><td style='text-align: center; word-wrap: break-word;'>65.5</td><td style='text-align: center; word-wrap: break-word;'>79.1</td><td style='text-align: center; word-wrap: break-word;'>23.6</td><td style='text-align: center; word-wrap: break-word;'>49.3</td><td style='text-align: center; word-wrap: break-word;'>62.6</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>+ Ours</td><td style='text-align: center; word-wrap: break-word;'>29.3</td><td style='text-align: center; word-wrap: break-word;'>60.5</td><td style='text-align: center; word-wrap: break-word;'>75.8</td><td style='text-align: center; word-wrap: break-word;'>29.9</td><td style='text-align: center; word-wrap: break-word;'>64.6</td><td style='text-align: center; word-wrap: break-word;'>79.8</td><td style='text-align: center; word-wrap: break-word;'>21.2</td><td style='text-align: center; word-wrap: break-word;'>49.4</td><td style='text-align: center; word-wrap: break-word;'>63.7</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>WiLoR [^42]</td><td style='text-align: center; word-wrap: break-word;'>30.0</td><td style='text-align: center; word-wrap: break-word;'>63.6</td><td style='text-align: center; word-wrap: break-word;'>79.9</td><td style='text-align: center; word-wrap: break-word;'>27.4</td><td style='text-align: center; word-wrap: break-word;'>63.9</td><td style='text-align: center; word-wrap: break-word;'>81.8</td><td style='text-align: center; word-wrap: break-word;'>15.4</td><td style='text-align: center; word-wrap: break-word;'>40.7</td><td style='text-align: center; word-wrap: break-word;'>57.4</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>+ Ours</td><td style='text-align: center; word-wrap: break-word;'>28.0</td><td style='text-align: center; word-wrap: break-word;'>62.6</td><td style='text-align: center; word-wrap: break-word;'>80.5</td><td style='text-align: center; word-wrap: break-word;'>26.0</td><td style='text-align: center; word-wrap: break-word;'>63.9</td><td style='text-align: center; word-wrap: break-word;'>83.0</td><td style='text-align: center; word-wrap: break-word;'>14.8</td><td style='text-align: center; word-wrap: break-word;'>41.2</td><td style='text-align: center; word-wrap: break-word;'>58.4</td></tr><tr><td rowspan="7">Occluded</td><td style='text-align: center; word-wrap: break-word;'>HandOccNet [^39]</td><td style='text-align: center; word-wrap: break-word;'>4.6</td><td style='text-align: center; word-wrap: break-word;'>16.1</td><td style='text-align: center; word-wrap: break-word;'>30.1</td><td style='text-align: center; word-wrap: break-word;'>4.7</td><td style='text-align: center; word-wrap: break-word;'>17.0</td><td style='text-align: center; word-wrap: break-word;'>31.1</td><td style='text-align: center; word-wrap: break-word;'>2.6</td><td style='text-align: center; word-wrap: break-word;'>9.3</td><td style='text-align: center; word-wrap: break-word;'>18.0</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>MeshGraphormer [^28]</td><td style='text-align: center; word-wrap: break-word;'>8.4</td><td style='text-align: center; word-wrap: break-word;'>24.1</td><td style='text-align: center; word-wrap: break-word;'>39.1</td><td style='text-align: center; word-wrap: break-word;'>8.3</td><td style='text-align: center; word-wrap: break-word;'>26.7</td><td style='text-align: center; word-wrap: break-word;'>45.1</td><td style='text-align: center; word-wrap: break-word;'>4.5</td><td style='text-align: center; word-wrap: break-word;'>14.6</td><td style='text-align: center; word-wrap: break-word;'>26.7</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Hamba [^8]</td><td style='text-align: center; word-wrap: break-word;'>14.6</td><td style='text-align: center; word-wrap: break-word;'>41.0</td><td style='text-align: center; word-wrap: break-word;'>60.6</td><td style='text-align: center; word-wrap: break-word;'>12.6</td><td style='text-align: center; word-wrap: break-word;'>36.9</td><td style='text-align: center; word-wrap: break-word;'>58.6</td><td style='text-align: center; word-wrap: break-word;'>8.6</td><td style='text-align: center; word-wrap: break-word;'>25.3</td><td style='text-align: center; word-wrap: break-word;'>41.1</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>HaMeR [^41]</td><td style='text-align: center; word-wrap: break-word;'>17.2</td><td style='text-align: center; word-wrap: break-word;'>42.1</td><td style='text-align: center; word-wrap: break-word;'>59.6</td><td style='text-align: center; word-wrap: break-word;'>16.0</td><td style='text-align: center; word-wrap: break-word;'>43.0</td><td style='text-align: center; word-wrap: break-word;'>62.7</td><td style='text-align: center; word-wrap: break-word;'>11.6</td><td style='text-align: center; word-wrap: break-word;'>32.5</td><td style='text-align: center; word-wrap: break-word;'>49.4</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>+ Ours</td><td style='text-align: center; word-wrap: break-word;'>15.9</td><td style='text-align: center; word-wrap: break-word;'>40.4</td><td style='text-align: center; word-wrap: break-word;'>59.2</td><td style='text-align: center; word-wrap: break-word;'>16.6</td><td style='text-align: center; word-wrap: break-word;'>45.4</td><td style='text-align: center; word-wrap: break-word;'>65.1</td><td style='text-align: center; word-wrap: break-word;'>12.6</td><td style='text-align: center; word-wrap: break-word;'>33.7</td><td style='text-align: center; word-wrap: break-word;'>51.5</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>WiLoR [^42]</td><td style='text-align: center; word-wrap: break-word;'>15.4</td><td style='text-align: center; word-wrap: break-word;'>42.1</td><td style='text-align: center; word-wrap: break-word;'>61.2</td><td style='text-align: center; word-wrap: break-word;'>15.4</td><td style='text-align: center; word-wrap: break-word;'>41.3</td><td style='text-align: center; word-wrap: break-word;'>62.9</td><td style='text-align: center; word-wrap: break-word;'>9.0</td><td style='text-align: center; word-wrap: break-word;'>26.6</td><td style='text-align: center; word-wrap: break-word;'>42.3</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>+ Ours</td><td style='text-align: center; word-wrap: break-word;'>15.1</td><td style='text-align: center; word-wrap: break-word;'>42.1</td><td style='text-align: center; word-wrap: break-word;'>63.0</td><td style='text-align: center; word-wrap: break-word;'>16.3</td><td style='text-align: center; word-wrap: break-word;'>44.3</td><td style='text-align: center; word-wrap: break-word;'>66.1</td><td style='text-align: center; word-wrap: break-word;'>9.5</td><td style='text-align: center; word-wrap: break-word;'>28.6</td><td style='text-align: center; word-wrap: break-word;'>45.5</td></tr></table>

<div style="text-align: center;">Table 4: Quantitative evaluation on the HInt benchmark [^41] (PCK@0.05/0.1/0.15). On occluded joints, our affordance-guided prior improves WiLoR across all three splits (New Days, VISOR, Ego4D) and HaMeR on most of them.</div>

![Image](images/526b14d17d50f571f9661ea7b315598226a5ee42ccfd1c8681948b05d2429dcc.jpg)

<div style="text-align: center;">Fig. 18: Failure cases on HInt. When the interaction deviates from the training distribution, the affordance prior provides weaker refinement over the initial WiLoR estimate. poses semantically aligned with affordance-aware descriptions, with high sample diversity.</div>

---

![Image](images/34d1dd975a2feef46ae6a0b491e23b5c5e09b746f8023894279a76b167d266c2.jpg)

<div style="text-align: center;">Fig. 19: Additional qualitative results on in-the-wild images from the HInt dataset [^41].</div>

---

![Image](images/fd2e1e6e18f0d7d3bd0f74d16948b01fed4dc994711f9db2ec99df60ef9de113.jpg)

<div style="text-align: center;">Fig. 20: Sampled by our affordance-description conditioned prior.</div>