Figure 1. We propose ForeHOI, the first feed-forward method that can directly reconstruct 3D object geometry from monocular hand-object interaction videos. Compared with previous methods that rely on complex pre-processing, our end-to-end pipeline achieves superior reconstruction performance under severe hand-object occlusion scenarios within one minute of inference time.
## Abstract
The ubiquity of monocular videos capturing daily hand-object interactions presents a valuable resource for embodied intelligence. While 3D hand reconstruction from in-the-wild videos has seen significant progress, reconstructing the involved objects remains challenging due to severe occlusions and the complex, coupled motion of the camera, hands, and object. In this paper, we introduce ForeHOI, a novel feed-forward model that directly reconstructs 3D object geometry from monocular hand-object interaction videos within one minute of inference time, eliminating the need for any pre-processing steps. Our key insight is that, the joint prediction of 2D mask inpainting and 3D shape completion in a feed-forward framework can effectively address the problem of severe occlusion in monocular hand-held object videos, thereby achieving results that outperform the performance of optimization-based methods. The information exchanges between the 2D and 3D shape completion boosts the overall reconstruction quality, enabling the framework to effectively handle severe hand-object occlusion. Furthermore, to support the training of our model, we contribute the first large-scale, high-
fidelity synthetic dataset of hand-object interactions with comprehensive annotations. Extensive experiments demonstrate that ForeHOI achieves state-of-the-art performance in object reconstruction, significantly outperforming previous methods with around a 100x speedup. Code and data are available at: .
### 1. Introduction
Every day, we unconsciously use our hands to interact with countless objects. With 3D hand-object interaction (HOI) data becoming increasingly important in embodied intelligence [^6] [^11] [^17] [^78], the comprehensive digitalization of 3D objects and hands is essential. 3D hand reconstruction for daily monocular videos has been widely studied in the past few years [^7] [^48] [^75]. However, research on object target reconstruction from such video data has been neglected due to two key challenges: 1): Such daily-life video usually does not provide full observation of the object due to hand-induced occlusions and object self-occlusions; 2): The camera, object and hands are all dynamic in one monocular video, making it hard to estimate the relative motion between the object and camera. In this context, we define
---
our task as the reconstruction of 3D object from daily hand-object interaction videos.
Since there exist unobserved areas for objects, this problem naturally needs strong object priors. As foundation models [^37] [^61] [^69] become ever more powerful, recent contributions have tried to tackle this problem using priors from foundation models. EasyHOI [^42] simplifies the problem to a single image input task and utilizes pretrained 2D inpainting models together with 3D generation models for complete object reconstruction. However, they are unable to utilize potential video sequence information, which is important for accurate 3D reconstruction in visible regions. MagicHOI [^64] takes short video clips as input and integrates a large pretrained novel view synthesis model [^37] into the object radiance field optimization pipeline for 3D object completion. However, its complex pipeline, including 2D inpainting and novel-view based 3D reconstruction, introduces view-inconsistent errors in each stage, leading to cumulative inaccuracies. Also, optimizing radiance fields usually takes hours, making it too slow for many applications. An alternative way for reconstructing hand-held objects involves first leveraging VGGT [^61] for initial reconstruction, followed by a 3D completion method to achieve full object reconstruction. However, under severe hand-object occlusion scenarios, VGGT fails to produce satisfactory initial results, which consequently leads to poor performance in the final completion results. Given the challenge of feed-forward reconstruction of severely occluded handheld objects in monocular video, it is crucial to incorporate both 2D and 3D shape completion of the occluded regions.
In this paper, we take a further step toward removing preprocessing steps by introducing ForeHOI, a feed-forward 3D reconstruction model that takes a video clip as input and predicts object geometry within one minute, making the reconstruction process robust and easy to use for daily HOI videos. Our key insight is that, the joint prediction of 2D mask inpainting and 3D shape completion in a feed-forward framework can effectively address the problem of severe occlusion in monocular hand-held object videos. We demonstrate that with sufficient data, hand-held object reconstruction can be directly learned by a single neural network, achieving results comparable with optimization-based methods but at a significantly higher inference speed. In detail, inspired by ReconViaGen [^5], we combine the reconstruction prior with a hand prior [^48] that provides the model with a deterministic understanding of the hand's shape and position as an additional condition. We then introduce a bidirectional cross-attention mechanism to the diffusion model to facilitate information exchange between the 3D features and the input image features. Through this bidirectional process, the inpainted 2D object mask and the complete 3D object geometry are jointly estimated. The mutual refinement between them boosts the overall quality, enabling the framework to effectively handle severe hand-object occlusion. Furthermore, as no large-scale HOI dataset is available to train our model, we build a synthetic dataset based on GraspXL [^77]: a robust RL-based grasp synthesis method. We first add photorealistic textures [^50] to the MANO [^53] hands and then render multi-view videos of the synthesized grasping process using the Objaverse [^14] dataset. This pipeline ultimately yields a large-scale synthetic dataset of 400K samples with comprehensive annotations, including hand masks, object masks, hand poses, object poses, and depth maps.
Extensive experiments conducted on the widely-used HO3D [^18] and recent proposed challenging HOT3D [^3] datasets demonstrate that ForeHOI achieves state-of-the-art performance in object geometry reconstruction, with significantly faster speed than optimization-based methods. Our contributions are summarized as follows:
- We propose ForeHOI, the first feed-forward method that effectively predicts object geometry from daily monocular HOI videos within one minute of inference time.
- We jointly generate 2D object mask inpainting and 3D object completion within our framework to address the reconstruction challenge provided by severe hand-object occlusion in daily monocular HOI videos.
- We contribute the first large-scale synthetic hand-object video dataset with high-fidelity rendering images.
### 2. Related Work
Object Reconstruction from generative priors. Preceding studies [^23] [^33] [^41] [^56] [^65] [^68] represented by Zero-1-to-3 [^37] focus mainly on leveraging 2D generative priors [^52] [^54] into 3D object reconstruction pipeline. These methods suffer from multi-view inconsistency problems and usually have lower quality compared with native 3D generation models [^30] [^35] [^57] [^60] [^69] on single-image input tasks. The very recent attempts CUPID [^24] and ReconViaGen [^5] leverage native 3D generation models for 3d reconstruction tasks by introducing pose-grounded generation and reconstruction priors, achieving superior performance on the single-image reconstruction task. Our work focuses on video input reconstruction under HOI scenarios, with a different approach, but leads to similar conclusions.
3D Hand Reconstruction. Over the past few decades, significant progress has been made in 3D hand reconstruction technology. With the introduction of the parametric hand model MANO [^53], recent studies [^2] [^19] [^31] [^36] [^38] [^46] [^79] have integrated it as a differentiable module into neural networks, enabling end-to-end reconstruction of dense 3D hand meshes directly from images or videos. State-of-the-art methods [^7] [^48] [^75] driven by large-scale data not only achieve robust 3D hand reconstruction performance but also demonstrate strong generalization capabilities to
---
in-the-wild images and videos. We use end-to-end hand reconstruction as guidance to improve object reconstruction.
Template-based Hand-held Object Reconstruction. Hand-held object reconstruction is more challenging than hands due to the object shape's natural diversity and hand-induced occlusion. Some existing methods simplify this problem by taking object templates as input [^4] [^13] [^15] [^39] [^59] [^70] [^71] and focusing mainly on estimating 6-DoF object pose. A recent contribution, Dynhor [^29] tackles the problem by generating the object template using a text-prompted 3D generation model with textual description coming from ChatGPT. Such a process creates large uncertainty due to the limitation of textual prompts. In contrast, our work predicts the object pose and shape simultaneously.
Template-free Hand-held Object Reconstruction. Recent advancements have introduced more generalizable methods [^16] [^21] [^25] [^27] [^49] [^51] [^63] [^66] [^67] [^73] [^74] with the development of general object reconstruction techniques like implicit fields [^44] [^62] and Diffusion models [^47]. However, these methods often impose strong constraints, including depth input [^66], accurate object pose input [^16] [^63], rigid hand-object interactions [^25] [^49], multi-view observations [^51], category-level optimization [^21] [^67], and complete object observations [^16]. Thus, they are difficult to work on daily hand-object interaction videos. On the contrary, our method takes only RGB video clips as input and ensures 3D reconstruction quality with a native 3D Diffusion model. Another branch of research [1, 8-10, 42, 72, 76] simplifies the problem to a single-image input task. Although they minimize input requirements, they are unable to consider potential video data, leading to an inability to exploit the temporal and multi-view information inherent in video sequences.
Hand-object interaction datasets. The lack of hand-object interaction data is a major bottleneck for the generalization capability of generalizable object reconstruction methods. Although several datasets have been proposed [^17] [^18] [^20] [^22] [^26] [^34] [^40] in previous research, the largest real-world dataset [^40] currently available contains only 417 objects, while the largest synthetic dataset [^22] comprises 2772 objects. Such a limited scale is far from sufficient for modeling complex object shapes. A recent contribution, GraspXL [^77] introduced a robust RL-based method for grasping synthesis and released their synthesizing results on Objaverse [^14]. We thus built the first large-scale hand-object interaction dataset based on it with more than 400k video sequences.
### 3. Methods
Figure 2 gives an overview of our method, ForeHOI, which reconstructs the object geometry from an RGB video sequence with limited viewpoints in a feed-forward manner. We begin by describing our image and hand feature encoding schemes in Sec. 3.1. Then Sec. 3.2 details a bidirectional cross-attention mechanism inside a diffusion transformer (DiT) that establishes correspondences between object voxels and multi-view masks. The training and inference process are detailed in Sec. 3.3. With the recovered mesh, per-frame object poses are obtained via rendering and comparison (Sec. 3.4). Finally, Sec. 3.5 introduces our new large-scale dataset.
#### 3.1. Image and hand prior encoding
We use DINOv2 [^45] as the image feature extractor to encode each frame of the input video. Apart from image features, we also encode hand features using a state-of-the-art hand pose estimation model [^48] for each image as input. Specifically, the image patch features of the hand ViT backbone $F_{hand} \in \mathbb{R}^{n \times 1024}$ are aggregated with the DINOv2 image patch features in a patch-to-patch manner for each input image into $F_{input} \in \mathbb{R}^{n \times 2048}$, finally we project it back to $\mathbb{R}^{n \times 1024}$ with a two-layer MLP.
By using the feature aggregator module to aggregate the 2D and 3D hand features, the model will have a deterministic understanding of the hand's shape and position. Note that since the features are pixel-aligned with each image, they contain implicit information in a local camera space, eliminating the multi-frame scale inconsistency problem. By using such an encoding strategy, we minimize the input requirements: no separate hand or object mask is needed during inference.
#### 3.2. Bidirectional cross attention
Our ultimate goal is to recover the object's complete shape. However, our early experiments show that directly predicting an accurate and complete geometry is highly challenging. We therefore decompose the problem into two complementary stages: 2D and 3D shape completion. Because the voxel-generation stage lacks sufficient appearance cues, we train the backbone with two objectives: per-frame pixel-aligned multi-view complete masks and the object's complete voxel representation. These two objectives are tightly coupled and mutually reinforcing. We elaborate on the benefits in both directions, beginning with object-voxel estimation: each input 2D image contains an incomplete observation of the object due to hand-induced occlusions. Under such circumstances, the 3D shape generation backbone must hallucinate the occluded regions without direct 2D evidence. Supervising the model with complete 2D object masks provides additional guidance on the full object contour in each view, thereby enhancing its ability to infer a coherent and complete 3D shape.
Given that our two tasks are highly interdependent and mutually reinforcing, effective information exchange between the two branches is essential. To this end, we adopt a
---
Figure 2. Pipeline overview of the proposed ForeHOL. Given a monocular video of hand-object interaction, we adopt a diffusion-based framework that jointly performs 2D object mask inpainting and 3D object completion to address the reconstruction challenge posed by severe hand-object occlusion. Moreover, the accurate object shape reconstruction achieved by our method leads to precise 3D object pose estimation through post-processing.
dual-branch architecture equipped with bidirectional cross-
attention. Starting from a noisy latent $\in \mathbb{R}^{64 \times 64 \times 64}$, the
geometry branch progressively denoises it into a coarse object
structure conditioned on the input features. Following
ReconViaGen [^5], we apply cross-attention between multiv-
view aggregated image features and the 3D latent representa-
tion within each DiT block using a weighted fusion
strategy. Differently, we replace the original input fe-
tures with features from the object-mask branch—layer by
layer from shallow to deep—so that the mask branch di-
rectly guides and supports the shape reconstruction process.
Just as mask-branch features are crucial for the geometry
branch, features from the geometry branch are equally im-
portant for completing object masks. Inspired by the semi-
nal multimodal alignment method [^43], we treat the 3D fe-
tures as contextual signals and feed them back into the mask
branch via additional cross-attention. The whole process
can be formulated as:
$$ \begin{align*}y_{j+1}&=\sum_{k=1}^{N}CrossAttn(Q(y_{j}),K(x_{i}^{n}),V(x_{i}^{n}))\cdot w_{n},\\&x_{i+1}=CrossAttn(Q(x_{i}),K(y_{j}),V(y_{j})),\\&j\in\{m\}_{m=1}^{M},i\in\{p\}_{p=1}^{P},k\in\{n\}_{n=1}^{N}.\end{align*} $$
where M is the number of DiT blocks in the geometry branch, P is the number of DiT blocks in the mask branch, N is the number of frames, $y_j$ denotes the 3D feature output of the j-th geometry branch block, while $x_i$ denotes the batch of image features after the i-th block of the mask branch. $w_n$ is the fusion weight associated with the n-th frame. Notably, because multi-view image features are fused through weighted cross-attention rather than concatenation, the model can accommodate relatively long input sequences with only a modest increase in memory consumption—unlike approaches that rely on concatenated multi-image features [^58] [^61].
#### 3.3. Training and inference
We train the network end-to-end by minimizing the conditional flow matching (CFM) objective for both 3D latent and 2D masks simultaneously with the same random t at each step, with the following loss:
$$ \begin{align*}\mathcal{L}_{CFM}(\boldsymbol{\theta})&=\mathbb{E}_{t,x_{0},\epsilon}||v_{\boldsymbol{\theta}}(x,t)-(\epsilon-x_{0})||_{2}^{2},\\\mathcal{L}&=\mathcal{L}_{CFM}^{2D}+\beta\mathcal{L}_{CFM}^{3D}.\end{align*} $$
where t is the continuous time step uniformly sampled from $\mathcal{U}(0,1)$. $x_0$ represents the ground-truth data sample, and $\epsilon$ is the noise sampled from a standard Gaussian prior distribution $\mathcal{N}(0,I)$. x denotes the interpolated intermediate state at time t. Accordingly, $v_\theta(x,t)$ is the velocity field predicted by the network, which is supervised by the ground-truth vector field ( $\epsilon - x_0$) pointing from the data to the noise. Furthermore, $\mathcal{L}_{CFM}^{2D}$ and $\mathcal{L}_{CFM}^{3D}$ denote the conditional flow matching losses for the 2D mask and 3D shape, $\beta$ is used to balance the loss of 2D mask and 3D shape.
To transform the voxel representation into a high-fidelity object surface, we train a masked multi-view structured-latent (SLat) flow following the strategy of ReconViaGen [^5]. Unlike the original formulation, we replace the
---
complete object images with occluded object images as inputs, and we directly adopt DINOv2 features instead of VGGT features, as the latter are unreliable when significant occlusions in object are present. The loss function is the same as the second stage of TRELLIS [^69].
At inference time, the multi-view object mask and object shape are simultaneously denoised through the CFM's denoising process, given 3D noisy latent volume or 2D noisy latent image as $x_0 \sim p_0$, the object's 3d shape and masks are denoised by the learned vector field $v(x, t) = \nabla_t x$ with Eular sampling methods:
$$ x(1)=x(0)+\int_{0}^{1}v_{\theta}(x(t),t)d t. $$
After object voxel generation, we elaborate its fined surface through masked multi-view structured-latent (SLat) flow finetuned on our data and decode it to 3D mesh following TRELLIS [^69].
#### 3.4. Object pose estimation
Since a large number of objects in real-world scenarios exhibit a high degree of symmetry, a general approach that relies solely on geometry for object pose estimation is not robust: symmetric objects introduce rotational ambiguity. Therefore, we adopt a pose estimation method based on textures derived from generated 3D models. The texture generation stage of the model is also trained on our dataset; further details on texture quality can be found in the appendix.
We employ a render-and-compare strategy to align all input views to the object coordinate system. Concretely, we begin by rendering 30 reference images from camera viewpoints uniformly distributed on a sphere. These rendered images are concatenated with the input frames and passed through VGGT [^61] to obtain coarse camera poses. Because the poses of the rendered views are known, we can derive the transformation that maps the predicted poses of VGGT [^61] into the object space, thereby yielding an initial pose estimation for each input image.
We then refine these initial poses through iterative correspondence-based alignment. Specifically, we render images using the coarse poses and apply Mast3R [^32] to establish 2D correspondences between each input frame and its rendered counterpart. We use Mast3R [^32] because it is trained with abundant data augmentation and is robust to our generated textures, which is not perfect. The coarse camera poses are then refined using a PnP solver with RANSAC. After several rounds of this refinement procedure, the initial predictions are substantially improved, resulting in highly accurate camera pose estimates.
#### 3.5. Dataset building
To build the large-scale high-fidelity synthetic dataset, we first incorporate a parametric hand texture model [^50] into the MANO [^53] hands, and randomize hand skin tone during rendering by adjusting its parameters. Then we carefully selected high-quality GraspXL synthesized grasping sequences on all the objects from Objaverse [^14] datasets by taking the object's texture quality and the object's degree of movement into consideration. Finally, we render the grasping processes as multi-view video sequences using Blender [^12]. During rendering, we automatically generate the camera moving trajectory for each grasping sequence that ensures the object and hand remain within the camera frustum at all times, while allowing for a limited range of object movement inside the rendered images during manipulation. Furthermore, Light intensity and direction are also randomized during rendering.
### 4. Experiments
We conduct experiments to effectively evaluate our model against state-of-the-art methods on the task of partially visible object reconstruction from daily hand-object-interaction videos. The objective is to recover both 3D object geometry accurately and the object pose from short monocular video sequences that provide only limited observation. Sec. 4.1 outlines the implementation details and training configurations of our framework. Sec. 4.2 introduces the datasets used for training and evaluation, including both existing benchmarks and our newly constructed dataset. Sec. 4.3 summarizes the evaluation metrics employed to assess 3D reconstruction quality, 2D prediction accuracy, and camera pose estimation. Sec. 4.4 presents quantitative and qualitative comparisons with recent state-of-the-art methods across multiple settings. Finally, Sec. 4.5 provides detailed ablation experiments to analyze the contribution of each component in our pipeline and to validate the design choices.
#### 4.1. Implementation Details
We adopt the same architecture as TRELLIS [^69]. We fine-tune the diffusion transformer in both stages of TRELIS using LoRA. In our configuration, the LoRA rank is set to 64, the scaling parameter (alpha) is set to 128, and the dropout for all LoRA modules is set to 0. The LoRA adapters are inserted exclusively into the qkv projection layers and the output projection layers of each attention block. The number of input views is randomly set as 2-6 while training, all the input images are resized to $518 \times 518$ images, resulting a $37 \times 37$ patches of image features, while the hand feature extractor takes $224 \times 224$ images as input, which is more compression, we then directly use a CNN with an MLP to fuse the two features as patch-level aggregated features. We randomly mix the images from different viewpoints at different timestamps as data augmentation. During training, we employ the Adam optimizer with a learning rate of $5e^{-4}$, the training takes 4 days on 8 NVIDIA L20 40G GPUs with a total batch size of 32.
---
Figure 3. Qualitative visual comparison results on the HO3D [^18] and HOT3D [^3] datasets. SfM Fails represent the failure camera estimation from the structure-from-motion (SfM) method COLMAP since the sparse-view in HOT3D video clips.
#### 4.2. Datasets
Training data We train our model only on our own large-scale synthetic dataset without any finetuning on real datasets, which highlights the strong data prior in our model and the importance of our dataset. Our data consists of over 400K video clips capturing hand-object interaction scenes in a multi-view manner. We randomly mix multi-view images with different timestamps on the same grasping sequence and randomly rotate or flip the input images as data enhancement while training.
HO3D We test our model on the HO3D [^18], a commonly used dataset for hand-object interaction reconstruction tasks. Specifically, we use the same sequences of HO3D-v3 as HOLD [^16], resulting in a total of 14 sequences. Since we are focusing on reconstructing objects from data with limited observations, following previous art MagicHOI [^64], we divide long sequences into 30-frame clips. For single-view reconstruction methods, HoRT [^10] and EasyHOI, we test them on each image and average the final metrics for all views of a sequence.
HOT3D Since our motivation is to digitize 3D objects from daily hand-object interaction videos, among which, egocentric data is the most challenging. HOT3D [^3] is a recent dataset captured with VR headsets. Different from other common datasets [^17] [^18] [^20] [^22] [^26] [^34] [^40] that intentionally rotate the object in hand for reconstruction, it is the first dataset that truly captures humans' daily interaction with
---
objects: we often interact with many objects at a short time and most of the interaction for one object happens only in 10-20 video frames, with highly dynamic and fast hand occlusions. To use the dataset for evaluation, we carefully pre-process the video, including cropping the video into short videos that describe the hand interaction with one object, undistorting the video frames, and sometimes cropping the video frames so that the object won't be too small in the video frame. Finally, we select 6 short clips at a length of 5-15 frames as evaluation data.
#### 4.3. Evaluation metrics
Since we focus mainly on the object's reconstruction, following [^10] [^28] [^64], we use the Chamfer distance (CD) in centimeters to assess object reconstruction quality, however, it is highly sensitive to outliers, so we also compute the F-score in percentage at 5mm (F5) and 10mm (F10) for a comprehensive evaluation of the object's shape. Additionally, we evaluate pose accuracy using standard visual odometry metrics [^55], including Absolute Trajectory Error (ATE) and Relative Pose Error (RPE). In which ATE measures the absolute difference between the estimated and ground truth camera translation, which is the object's translation in camera space, in our case, while RPE represents the relative rotation ( $RPE_{r}$) and translation ( $RPE_{t}$) error.
#### 4.4. Comparison with state of the art
Reconstruction results We compare our object shape reconstruction quality with state-of-the-art (SOTA) handheld object reconstruction methods in Tab. 2, including HOLD [^16], EasyHOI [^42], HORT [^10], and MagicHOI [^64]. Among them, HORT is trained on the HO3D dataset, so we only compare it on the HOT3D dataset. As shown in Fig. 3, due to a lack of data, HORT only works well on similar objects from their limited training data and is hard to generalize to unseen objects from HOT3D, it even completely fails for relatively complex geometry or unusual camera viewpoints. HOLD is the only non-prior method, without an object shape prior, it's hard for it to reconstruct the complete object shape. Although EasyHOI utilizes rich priors from 2D inpainting models and 3D generation models, due to the accumulative error brought by its complex pipeline, the final 3D shape actually deviates from the original image a lot, and it also fails in line 5 of Fig. 3 for complex geometry. By utilizing 2D novel-view-synthesis priors, MagicHOI can not only complete the occlusion area, but also align well with the image in observed areas, however, due to the natural inconsistency of 2D multi-view generation model, the generated back of the object may have strange shapes, see line 2 in Fig. 3, also, such as an inconsistency makes the optimized surface quite rough and bumpy. Noting that both HOLD and MagicHOI need the object pose and sparse points from Structure-from-Motion (SfM) as input, thus, the whole pipeline fails whenever SfM fails, which is quite common in daily HOI scenes.
| Methods | RPE(cm) | RPE(°) | ATE(m) |
| HOLD | 5.87 | 6.24 | 0.27 |
| Dynhor | 4.25 | 5.25 | 0.24 |
| Ours | 1.42 | 2.64 | 0.13 |
Table 1. Object pose estimation results compared with the HOLD and Dynhor method.
Object pose results We compare our object pose with Dynhor [^28] and HOLD [^16] in Tab. 1 on the HO3D dataset. Since HOLD actually takes Structure-from-Motion (SfM) poses as an initial pose and refines it, the quality of the initial pose matters for the final object pose. In our setting, the input is a video clip with a somewhat sparse viewpoint for the object, such a pattern is quite challenging for SfM methods, thus resulting in relatively poor pose estimation results. On the other hand, our method utilizes the reconstructed model for pose estimation, which is a template-based object estimation task, with the help of Mast3R's robust matching, the result turns out to be quite good. Dynhor claims to utilize a texture-to-3d generated template as input, we follow their paper to use Genie for shape generation, due to lack of text prompt of GPT's image capturing process, the resulting textured mesh has a high degree of randomness, thus influences the pose results, though they claimed that the following refining integrated in object reconstruction eliminates the errors in initial pose guessing process, this part is not open-sourced yet, so we use their pose at first step for comparison, which is not as good as the claim in their paper.
#### 4.5. Ablation studies
We ablate our two key designs: hand feature input and the 2D object mask completion branch of the pipeline, and the importance of our data in Tab. 3.
The importance of our dataset The problem of in-hand object reconstruction under dynamic occlusion can somewhat be viewed as a multi-view object reconstruction problem under occlusion: the camera movement with the object frozen is equal to the object movement under the object-camera coordinate. According to this assumption, we first trained our model on the commonly used object datasets [^69], which are easy to obtain in our early experiments. We add random masks to the multi-view images and select restricted views for each training step to imitate the hand-held environment. However, as shown in Tab. 3, the performance on the real-world dataset is very poor. Based on our observations, this is primarily attributable to two issues: Firstly, the semantic understanding and completion of 3D generative models rely on the object-centric coordinate system. When objects are rendered on a table, their orientations are generally aligned, whereas objects rotated by
---
| Methods | HO3D | HOT3D | Avg Time(min) |
| CD(cm) $\downarrow$ | F@5(%) $\uparrow$ | F@10(%) $\uparrow$ | CD(cm) $\downarrow$ | F@5(%) $\uparrow$ | F@10(%) $\uparrow$ |
| EasyHOI | 1.83 | 46.35 | 69.24 | 1.21 | 18.46 | 34.25 | 175 |
| HOLD | 1.36 | 66.42 | 82.43 | N/A | N/A | N/A | 330 |
| HORT | N/A | N/A | N/A | 2.32 | 16.52 | 20.43 | 0.8 |
| MagicHOI | 0.86 | 64.53 | 91.87 | N/A | N/A | N/A | 58 |
| Ours | 0.79 | 68.95 | 93.72 | 1.03 | 60.5 | 89.1 | 1.1 |
Table 2. Quantitative comparison with baseline methods on the real-world HO3D and HOI dataset. Our method outperforms all the baseline methods on geometry accuracy and inference time.
Figure 4. Qualitative comparisons for different variants of ForeHOI for ablative study on HO3D [^18] dataset. Zoom in for better visualization in detail.
hands represent a completely unfamiliar scenario for models trained on normal data. Secondly, the use of random occlusions devoid of semantic information results in the failure to learn hand-object interactions. This insight has inspired two other key components in our framework design: the incorporation of hand inputs and mask completion, and also underscores the importance of our data.
The influence of hand features One of the key challenges in object reconstruction of HOI scenes is analyzing the hand-object relationship. The hand feature gives the model a direct understanding of where the occlusion comes from. Fig. 4 shows that without the hand feature, the models sometimes fail to complete the object under hand contact areas with obvious fingerprint indentation.
The influence of 2D mask completion branch As previously mentioned, the masks caused by hand occlusion are relatively regular and contain information about hand-object interactions. After obtaining per-frame hand features, completing the full object mask becomes a relatively straightforward task. However, this simple task enhances the model's understanding of hand-object interactions, thereby significantly improving its performance in scenarios with substantial hand occlusion, as can be observed in Tab. 3 and Fig. 4.
| Strategy | CD(cm) $\downarrow$ | F@5(%) $\uparrow$ | F@10(%) $\uparrow$ |
| MV data | 1.23 | 25.54 | 68.54 |
| our data | 0.91 | 53.64 | 80.63 |
| + hand feats | 0.88 | 54.53 | 85.43 |
| + mask comp. | 0.79 | 68.95 | 93.72 |
Table 3. MV data means ReconViaGen [^5] backbone trained on their dataset with random image masking. our data means ReconViaGen [^5] backbone trained on our hand-object interaction data. Both +hand feats and +mask comp. represent versions trained on our data, with different architecture design.
### 5. Conclusion
In this paper, we present ForeHOI, a novel feed-forward model for fast and accurate 3D object reconstruction from monocular HOI videos. Our work demonstrates that high-fidelity reconstruction of hand-held objects under severe occlusion can be learned end-to-end within a single network. This is achieved by jointly predicting 2D mask inpainting and 3D shape completion, allowing the two tasks to benefit from each other to generate more accurate final geometry. Our method eliminates the dependency on cumbersome preprocessing steps and maintains fast inference speed. Qualitative and quantitative results show that ForeHOI outperforms state-of-the-art methods on in-the-wild settings.
---
### 6. Acknowledgments
The work was supported in part by Guangdong S&T Programme with Grant No. 2024B0101030002, the Basic Research Project No. HZQB-KCZYZ-2021067 of Hetao Shenzhen-HK S&T Cooperation Zone, by Guangdong Provincial Outstanding Youth Fund with No. 2023B1515020055, the Shenzhen Outstanding Talents Training Fund 202002, the NSFC with Grant No. 62293482, the Guangdong Research Projects No. 2017ZT07X152 and No. 2019CX01X104, the Guangdong Provincial Key Laboratory of Future Networks of Intelligence (Grant No. 2022B1212010001), and the Shenzhen Key Laboratory of Big Data and Artificial Intelligence (Grant No. SYSPG20241211173853027), the Guangdong Province Radio Science Data Center.
## References
[^1]: Ayce Idil Aytekin, Helge Rhodin, Rishabh Dabral, and Christian Theobalt. Follow my hold: Hand-object interaction reconstruction through geometric guidance, 2025. 3
[^2]: Seungryul Baek, Kwang In Kim, and Tae-Kyun Kim. Pushing the envelope for rgb-based dense 3d hand pose estimation via neural rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019. 2
[^3]: Prithviraj Banerjee, Sindi Shkodrani, Pierre Moulon, Shreyas Hampali, Shangchen Han, Fan Zhang, Linguang Zhang, Jade Fountain, Edward Miller, Selen Basol, Richard Newcombe, Robert Wang, Jakob Julian Engel, and Tomas Hodan. HOT3D: Hand and object tracking in 3D from egocentric multi-view videos. CVPR, 2025. 2, 6
[^4]: Zhe Cao, Ilija Radosavovic, Angjoo Kanazawa, and Jitendra Malik. Reconstructing hand-object interactions in the wild. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 12417–12426, 2021. 3
[^5]: Jiahao Chang, Chongjie Ye, Yushuang Wu, Yuantao Chen, Yidan Zhang, Zhongjin Luo, Chenghong Li, Yihao Zhi, and Xiaoguang Han. Reconviagen: Towards accurate multi-view 3d object reconstruction via generation, 2025. 2, 4, 8
[^6]: Hongyi Chen, Yunchao Yao, Yufei Ye, Zhixuan Xu, Homanga Bharadhwaj, Jiashun Wang, Shubham Tulsiani, Zackory Erickson, and Jeffrey Ichnowski. Web2grasp: Learning functional grasps from web images of hand-object interactions. arXiv preprint arXiv:2505.05517, 2025. 1
[^7]: Xingyu Chen, Zhuheng Song, Xiaoke Jiang, Yaoqing Hu, Junzhi Yu, and Lei Zhang. Handos: 3d hand reconstruction in one stage. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025. 1,2
[^8]: Zerui Chen, Yana Hasson, Cordelia Schmid, and Ivan Laptev. AlignSDF: Pose-Aligned signed distance fields for hand-object reconstruction. In ECCV, 2022. 3
[^9]: Zerui Chen, Shizhe Chen, Cordelia Schmid, and Ivan Laptev. gsdf: Geometry-driven signed distance functions for 3d hand-object reconstruction. In Proceedings of the IEEE/CVF
Conference on Computer Vision and Pattern Recognition (CVPR), pages 12890–12900, 2023.
[^10]: Zerui Chen, Rolandos Alexandros Potamias, Shizhe Chen, and Cordelia Schmid. HORT: Monocular hand-held objects reconstruction with transformers. arXiv preprint arXiv:2503.21313, 2025. 3, 6, 7
[^11]: Sammy Christen, Muhammed Kocabas, Emre Aksan, Jemin Hwangbo, Jie Song, and Otmar Hilliges. D-grasp: Physically plausible dynamic grasp synthesis for hand-object interactions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 1
[^12]: Blender Online Community. Blender - a 3D modelling and rendering package. Blender Foundation, Stichting Blender Foundation, Amsterdam, 2018. 5
[^13]: Enric Corona, Albert Pumarola, Guillem Alenya, Francesc Moreno-Noguer, and Gregory Rogez. Ganhand: Predicting human grasp affordances in multi-object scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020. 3
[^14]: Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects. arXiv preprint arXiv:2212.08051, 2022. 2, 3, 5
[^15]: Zicong Fan, Takehiko Ohkawa, Linlin Yang, Nie Lin, Zhishan Zhou, Shihao Zhou, Jiajun Liang, Zhong Gao, Xuanyang Zhang, Xue Zhang, Fei Li, Liu Zheng, Feng Lu, Karim Abou Zeid, Bastian Leibe, Jeongwan On, Seungryul Baek, Aditya Prakash, Saurabh Gupta, Kun He, Yoichi Sato, Otmar Hilliges, Hyung Jin Chang, and Angela Yao. Benchmarks and challenges in pose estimation for egocentric hand interactions with objects. In European Conference on Computer Vision (ECCV), 2024. 3
[^16]: Zicong Fan, Maria Parelli, Maria Eleni Kadoglou, Muhammed Kocabas, Xu Chen, Michael J Black, and Otmar Hilliges. HOLD: Category-agnostic 3d reconstruction of interacting hands and objects from video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 494–504, 2024. 3, 6, 7
[^17]: Rao Fu, Dingxi Zhang, Alex Jiang, Wanjia Fu, Austin Fund, Daniel Ritchie, and Srinath Sridhar. Gigahands: A massive annotated dataset of bimanual hand activities. 2025. 1, 3, 6
[^18]: Shreyas Hampali, Mahdi Rad, Markus Oberweger, and Vincent Lepetit. Honnotate: A method for 3d annotation of hand and object poses. In CVPR, 2020. 2, 3, 6, 8
[^19]: Shreyas Hampali, Sayan Deb Sarkar, Mahdi Rad, and Vincent Lepetit. Keypoint transformer: Solving joint identification in challenging hands and object interactions for accurate 3d pose estimation. In IEEE Computer Vision and Pattern Recognition Conference, 2022. 2
[^20]: Shreyas Hampali, Sayan Deb Sarkar, Mahdi Rad, and Vincent Lepetit. Keypoint transformer: Solving joint identification in challenging hands and object interactions for accurate 3d pose estimation. In CVPR, 2022. 3, 6
[^21]: Yana Hasson, Gul Varol, Dimitrios Tzionas, Igor Klevatykh, Michael J. Black, Ivan Laptev, and Cordelia Schmid. Learning joint reconstruction of hands and manipulated objects.
---
jects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019. 3
[^22]: Yana Hasson, Gül Varol, Dimitrios Tzionas, Igor Kalevatykh, Michael J. Black, Ivan Laptev, and Cordelia Schmid. Learning joint reconstruction of hands and manipulated objects. In CVPR, 2019. 3, 6
[^23]: Lukas Höllein, Aljaž Božič, Norman Müller, David Novotny, Hung-Yu Tseng, Christian Richardt, Michael Zollhöfer, and Matthias Nießner. Viewdiff: 3d-consistent image generation with text-to-image models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5043–5052, 2024. 2
[^24]: Binbin Huang, Haobin Duan, Yiqun Zhao, Zibo Zhao, Yi Ma, and Shenghua Gao. Cupid: Pose-grounded generative 3d reconstruction from a single image, 2025. 2
[^25]: Di Huang, Xiaopeng Ji, Xingyi He, Jiaming Sun, Tong He, Qing Shuai, Wanli Ouyang, and Xiaowei Zhou. Reconstructing hand-held objects from monocular video. In SIGGRAPH Asia Conference Proceedings, 2022. 3
[^26]: Juntao Jian, Xiuping Liu, Manyi Li, Ruizhen Hu, and Jian Liu. Affordpose: A large-scale dataset of hand-object interactions with affordance-driven hand pose. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 14713–14724, 2023. 3, 6
[^27]: Shijian Jiang, Qi Ye, Rengan Xie, Yuchi Huo, Xiang Li, Yang Zhou, and Jiming Chen. In-hand 3d object reconstruction from a monocular rgb video. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 2525–2533, 2024. 3
[^28]: Shijian Jiang, Qi Ye, Rengan Xie, Yuchi Huo, and Jiming Chen. Hand-held object reconstruction from RGB video with dynamic interaction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12220–12230, 2025. 7
[^29]: Shijian Jiang, Qi Ye, Rengan Xie, Yuchi Huo, and Jiming Chen. Hand-held object reconstruction from RGB video with dynamic interaction. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 12220–12230, 2025. 3
[^30]: Weitai Kang, Haifeng Huang, Yuzhang Shang, Mubarak Shah, and Yan Yan. Robin3d: Improving 3d large language model via robust instruction tuning. arXiv preprint arXiv:2410.00255, 2024. 2
[^31]: Dominik Kulon, Riza Alp Guler, Iasonas Kokkinos, Michael M. Bronstein, and Stefanos Zafeiriou. Weakly-supervised mesh-convolutional hand reconstruction in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020. 2
[^32]: Vincent Leroy, Yohann Cabon, and Jerome Revaud. Grounding image matching in 3d with mast3r, 2024. 5
[^33]: Jiahao Li, Hao Tan, Kai Zhang, Zexiang Xu, Fujun Luan, Yinghao Xu, Yicong Hong, Kalyan Sunkavalli, Greg Shakhnarovich, and Sai Bi. Instant3d: Fast text-to-3d with sparse-view generation and large reconstruction model. In International Conference on Representation Learning, pages 21896–21920, 2024. 2
[^34]: Qixiu Li, Yu Deng, Yaobo Liang, Lin Luo, Lei Zhou, Chengtang Yao, Lingqi Zeng, Zhiyuan Feng, Huizhi Liang,
Sicheng Xu, Yizhong Zhang, Xi Chen, Hao Chen, Lily Sun, Dong Chen, Jiaolong Yang, and Baining Guo. Scalable vision-language-action model pretraining for robotic manipulation with real-life human activity videos. arXiv preprint arXiv:2510.21571, 2025. 3, 6
[^35]: Yangguang Li, Zi-Xin Zou, Zexiang Liu, Dehu Wang, Yuan Liang, Zhipeng Yu, Xingchao Liu, Yuan-Chen Guo, Ding Liang, Wanli Ouyang, et al. Triposg: High-fidelity 3d shape synthesis using large-scale rectified flow models. arXiv preprint arXiv:2502.06608, 2025. 2
[^36]: Kevin Lin, Lijuan Wang, and Zicheng Liu. End-to-end human pose and mesh reconstruction with transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1954–1963, 2021. 2
[^37]: Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero-1-to-3: Zero-shot one image to 3d object, 2023. 2
[^38]: Shaowei Liu, Hanwen Jiang, Jiarui Xu, Sifei Liu, and Xiaolong Wang. Semi-supervised 3d hand-object poses estimation with interactions in time. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14687–14697, 2021. 2
[^39]: Shaowei Liu, Hanwen Jiang, Jiarui Xu, Sifei Liu, and Xiaolong Wang. Semi-supervised 3d hand-object poses estimation with interactions in time. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2021. 3
[^40]: Yunze Liu, Yun Liu, Che Jiang, Kangbo Lyu, Weikang Wan, Hao Shen, Boqiang Liang, Zhoujie Fu, He Wang, and Li Yi. Hoi4d: A 4d egocentric dataset for category-level human-object interaction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 21013–21022, 2022. 3, 6
[^41]: Yuan Liu, Cheng Lin, Zijiao Zeng, Xiaoxiao Long, Lingjie Liu, Taku Komura, and Wenping Wang. Syncdreamer: Generating multiview-consistent images from a single-view image. arXiv preprint arXiv:2309.03453, 2023. 2
[^42]: Yumeng Liu, Xiaoxiao Long, Zemin Yang, Yuan Liu, Marc Habermann, Christian Theobalt, Yuexin Ma, and Wenping Wang. Easyhoi: Unleashing the power of large models for reconstructing hand-object interactions in the wild. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 7037–7047, 2025. 2, 3, 7
[^43]: Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. Advances in neural information processing systems, 32, 2019. 4
[^44]: Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020. 3
[^45]: Maxime Oquab, Timothée Darcet, Theo Moutakanni, Huy V. Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Russell Howes, Po-Yao Huang, Hu Xu, Vasu Sharma, Shang-Wen Li, Wojciech Galuba, Mike Rabbat, Mido Assran, Nicolas Ballas, Gabriel Synnaeve, Ishan Misra, Herve Jegou,
---
Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision, 2023. 3
[^46]: Georgios Pavlakos, Dandan Shan, Ilija Radosavovic, Angjoo Kanazawa, David Fouhey, and Jitendra Malik. Reconstructing hands in 3d with transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9826–9836, 2024. 2
[^47]: Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv, 2022. 3
[^48]: Rolandos Alexandros Potamias, Jinglei Zhang, Jiankang Deng, and Stefanos Zafeiriou. Wilor: End-to-end 3d hand localization and reconstruction in-the-wild, 2024. 1, 2, 3
[^49]: Aditya Prakash, Matthew Chang, Matthew Jin, Ruisen Tu, and Saurabh Gupta. 3d reconstruction of objects in hands without real world 3d supervision. In European Conference on Computer Vision (ECCV), 2024. 3
[^50]: Neng Qian, Jiayi Wang, Franziska Mueller, Florian Bernard, Vladislav Golyanik, and Christian Theobalt. HTML: A Parametric Hand Texture Model for 3D Hand Reconstruction and Personalization. In Proceedings of the European Conference on Computer Vision (ECCV). Springer, 2020. 2, 5
[^51]: Wentian Qu, Zhaopeng Cui, Yinda Zhang, Chenyu Meng, Cuixia Ma, Xiaoming Deng, and Hongan Wang. Novel-view synthesis and pose estimation for hand-object interaction from sparse views. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 15100–15111, 2023. 3
[^52]: Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models, 2021. 2
[^53]: Javier Romero, Dimitrios Tzionas, and Michael J. Black. Embodied hands: modeling and capturing hands and bodies together. ACM Trans. Graph., 36(6), 2017. 2, 5
[^54]: Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language understanding. In Advances in Neural Information Processing Systems, pages 36479–36494. Curran Associates, Inc., 2022. 2
[^55]: Davide Scaramuzza and Friedrich Fraundorfer. Visual odometry [tutorial]. IEEE Robotics & Automation Magazine, 18(4):80–92, 2011. 7
[^56]: Ruoxi Shi, Hansheng Chen, Zhuoyang Zhang, Minghua Liu, Chao Xu, Xinyue Wei, Linghao Chen, Chong Zeng, and Hao Su. Zero123++: a single image to consistent multi-view diffusion base model. CoRR, abs/2310.15110, 2023. 2
[^57]: Tencent Hunyuan3D Team. Hunyuan3d 2.1: From images to high-fidelity 3d assets with production-ready pbr material, 2025. 2
[^58]: Tencent Hunyuan3D Team. Hunyuan3d 2.0: Scaling diffusion models for high resolution textured 3d assets generation, 2025. 4
[^59]: Bugra Tekin, Federica Bogo, and Marc Pollefeys. H+o: Unified egocentric recognition of 3d hand-object poses and interactions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019. 3
[^60]: Dmitry Tochilkin, David Pankratz, Zexiang Liu, Zixuan Huang, , Adam Letts, Yangguang Li, Ding Liang, Christian Laforte, Varun Jampani, and Yan-Pei Cao. Triposr: Fast 3d object reconstruction from a single image. arXiv preprint arXiv:2403.02151, 2024. 2
[^61]: Jianyuan Wang, Minghao Chen, Nikita Karaev, Andrea Vedaldi, Christian Rupprecht, and David Novotny. Vggt: Visual geometry grounded transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025. 2, 4, 5
[^62]: Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. arXiv preprint arXiv:2106.10689, 2021. 3
[^63]: Shibo Wang, Haonan He, Maria Parelli, Christoph Gebhardt, Zicong Fan, and Jie Song. Magichoi: Leveraging 3d priors for accurate hand-object reconstruction from short monocular video clips. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 5957–5968, 2025. 3
[^64]: Shibo Wang, Haonan He, Maria Parelli, Christoph Gebhardt, Zicong Fan, and Jie Song. Magichoi: Leveraging 3d priors for accurate hand-object reconstruction from short monocular video clips. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2025. 2, 6, 7
[^65]: Daniel Watson, William Chan, Ricardo Martin-Brualla, Jonathan Ho, Andrea Tagliasacchi, and Mohammad Norouzi. Novel view synthesis with diffusion models, 2022. 2
[^66]: Bowen Wen, Jonathan Tremblay, Valts Blukis, Stephen Tyree, Thomas Müller, Alex Evans, Dieter Fox, Jan Kautz, and Stan Birchfield. BundleSDF: Neural 6-DoF tracking and 3D reconstruction of unknown objects. In CVPR, 2023. 3
[^67]: Jane Wu, Georgios Pavlakos, Georgia Gkioxari, and Jitendra Malik. Reconstructing hand-held objects in 3d. arXiv preprint arXiv:2404.06507, 2024. 3
[^68]: Rundi Wu, Ben Mildenhall, Philipp Henzler, Keunhong Park, Ruiqi Gao, Daniel Watson, Pratul P. Srinivasan, Dor Verbin, Jonathan T. Barron, Ben Poole, and Aleksander Ho?y?ski. Reconfusion: 3d reconstruction with diffusion priors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 21551–21561, 2024. 2
[^69]: Jianfeng Xiang, Zelong Lv, Sicheng Xu, Yu Deng, Ruicheng Wang, Bowen Zhang, Dong Chen, Xin Tong, and Jiaolong Yang. Structured 3d latents for scalable and versatile 3d generation. arXiv preprint arXiv:2412.01506, 2024. 2, 5, 7
[^70]: Lixin Yang, Xinyu Zhan, Kailin Li, Wenqiang Xu, Jiefeng Li, and Cewu Lu. Cpf: Learning a contact potential field to model the hand-object interaction. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 11097–11106, 2021. 3
[^71]: Lixin Yang, Kailin Li, Xinyu Zhan, Jun Lv, Wenqiang Xu, Jiefeng Li, and Cewu Lu. Artiboost: Boosting articulated 3d
---
hand-object pose estimation via online exploration and synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2750–2760, 2022. 3
[^72]: Yufei Ye, Abhinav Gupta, and Shubham Tulsiani. What's in your hands? 3d reconstruction of generic objects in hands. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3895–3905, 2022. 3
[^73]: Yufei Ye, Poorvi Hebbar, Abhinav Gupta, and Shubham Tulsiani. Diffusion-guided reconstruction of everyday hand-object interaction clips. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 19717–19728, 2023. 3
[^74]: Yufei Ye, Abhinav Gupta, Kris Kitani, and Shubham Tulsiani. G-hop: Generative hand-object prior for interaction reconstruction and grasp synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1911–1920, 2024. 3
[^75]: Yufei Ye, Yao Feng, Omid Taheri, Haiwen Feng, Shubham Tulsiani, and Michael J Black. Predicting 4d hand trajectory from monocular videos. arXiv preprint arXiv:2501.08329, 2025. 1, 2
[^76]: Chenyangguang Zhang, Yan Di, Ruida Zhang, Guangyao Zhai, Fabian Manhardt, Federico Tombari, and Xiangyang Ji. Ddf-ho: Hand-held object reconstruction via conditional directed distance field. In Advances in Neural Information Processing Systems, pages 56871–56884. Curran Associates, Inc., 2023. 3
[^77]: Hui Zhang, Sammy Christen, Zicong Fan, Otmar Hilliges, and Jie Song. GraspXL: Generating grasping motions for diverse objects at scale. In European Conference on Computer Vision (ECCV), 2024. 2, 3
[^78]: Hui Zhang, Sammy Christen, Zicong Fan, Luocheng Zheng, Jemin Hwangbo, Jie Song, and Otmar Hilliges. ArtiGrasp: Physically plausible synthesis of bi-manual dexterous grasping and articulation. In International Conference on 3D Vision (3DV), 2024. 1
[^79]: Jason Y. Zhang, Sam Pepose, Hanbyul Joo, Deva Ramanan, Jitendra Malik, and Angjoo Kanazawa. Perceiving 3d human-object spatial arrangements from a single image in the wild. In European Conference on Computer Vision (ECCV), 2020. 2