# WHOLE: World-Grounded Hand-Object Lifted from Egocentric Videos Yufei Ye $^{1}$ Jiaman Li $^{2}$ Ryan Rong $^{1}$ C. Karen Liu $^{1,2}$ $^{1}$Stanford University $^{2}$Amazon FAR (Frontier AI & Robotics) ![Image](images/ec4009d4ac4f1b7ff11f90950b6a7f66375490c659be383fbc4057a42c07bf9c.jpg)
Figure 1. Given a metric-SLAMed egocentric video of a person interacting with the scene and the corresponding object templates, WHOLE reconstructs the motions of both hands and the objects of interest. The reconstruction is shown from the egocentric camera view, an allocentric view, and through the full 4D hand-object trajectories. In the example above, the person moves a box from the table on the left to the shelf on the right, takes a can from the shelf and places it on the center table, and finally picks up an orange box. For visual clarity, multiple objects are overlaid into a single scene, and the hands are displayed only when interacting with the box.
## Abstract Egocentric manipulation videos are highly challenging due to severe occlusions during interactions and frequent object entries and exits from the camera view as the person moves. Current methods typically focus on recovering either hand or object pose in isolation, but both struggle during interactions and fail to handle out-of-sight cases. Moreover, their independent predictions often lead to inconsistent hand-object relations. We introduce WHOLE, a method that holistically reconstructs hand and object motion in world space from egocentric videos given object templates. Our key insight is to learn a generative prior over hand-object motion to jointly reason about their interactions. At test time, the pretrained prior is guided to generate trajectories that conform to the video observations. This joint generative reconstruction substantially outperforms approaches that process hands and objects separately followed by post-processing. WHOLE achieves state-of-the-art performance. on hand motion estimation, 6D object pose estimation, and their relative interaction reconstruction. ### 1. Introduction Humans effortlessly connect what they see (egocentric) to a persistent 3D world (allocentric) - a core cognitive ability that underlies spatial reasoning and purposeful interaction. With the increasing availability of wearable cameras, egocentric videos have become a powerful medium for capturing such first-person experiences. These recordings depict everyday activities from the wearer's viewpoint like walking through a room, reaching for a can on the shelf, and pouring a milk bottle, as illustrated in Fig. 1. Among the many elements in these scenes, the hands and the objects they manipulate form the most direct interface through which humans act upon the world, making their 3D reconstruction a crucial step toward understanding egocentric --- experiences. Achieving this capacity of spatial reasoning about human interactions enables downstream applications such as robot learning from human demonstrations [^21] [^24] and immersive AR/VR environments. Our goal is to endow computers with a comparable ability: to reconstruct the motion of the active objects and both hands within a consistent world coordinate frame from metric-SLAMed egocentric videos that depict hand manipulation. However, this task is particularly challenging. Because the camera is mounted on a moving wearer, the resulting video often exhibits large egomotion even when the object itself does not move much. Objects may leave and re-enter the field of view, and frequent occlusions between hands and objects further complicate perception and reconstruction. Prior work has explored several closely related directions, yet often in isolation. Some methods focus exclusively on reconstructing humans [^45] [^69] [^70] or general objects [^12] [^58] in a consistent world coordinate frame, while others tackle the problem of estimating camera motion [^31] [^37] [^51] to align the egocentric viewpoint with the world space. However, simply combining these separate efforts is insufficient for egocentric interaction reconstruction from videos. Another line of research addresses hand-object interaction (HOI) reconstruction [^11] [^22] [^63], typically over a few-second clips to recover detailed object geometry and contact patterns. Yet, these approaches remain confined to local reference frames, without reasoning about motion and interaction within a persistent, global 3D world. Our key insight is that hand and object motions are inherently interdependent, and should be modeled jointly to capture coherent hand-object interactions. Building on this idea, we introduce WHOLE, which formulates reconstruction as a guided generation process based on a generative motion prior learned from hand-object interactions. Specifically, we train a diffusion-based motion prior to model the mutual dynamics between hands and objects, as well as the contact relationship between them. At test time, we guide this pretrained model using visual observations including segmentation masks and contact cues, to produce global 3D trajectories consistent with the input egocentric video. To obtain reliable contact information, we further enhance a vision-language model (VLM) with spatially grounded visual prompts to enable robust contact localization even in cluttered scenes. Together, it allows WHOLE to generate coherent plausible reconstructions of long hand-object interaction sequences in global 3D space. We train and evaluate WHOLE on the HOT3D [^2] dataset. We show that our learned motion prior generates diverse and plausible samples while producing reconstructions faithful to the input video. Across hand, object, and interaction evaluation settings, WHOLE consistently performs well compared to existing baselines that naively combine state-of-the-art methods from the respective hand and object estimation domains. We also find that VLM-annotated contact cues perform comparably to ground-truth labels, indicating that the designed visual prompt effectively improves the VLM's spatial grounding. Code and model will be public upon acceptance. ### 2. Related Work Egocentric Video Understanding. Egocentric perception has recently gained traction due to the availability of large-scale datasets [^6] [^15] [^16] and wearable cameras such as GoPro and Project Aria [^4] [^34] [^35]. While a line of work focuses on high-level semantic understanding such as action recognition and language grounding [^15] [^16] [^21] [^60], recent research has expanded to spatial perception tasks, including 2D detection, segmentation, and tracking [^7] [^76]. A few efforts have explored 3D understanding from egocentric inputs, such as camera localization, global human motion reconstruction, and global hand motion reconstruction [^28] [^40] [^54] [^67] [^73]. Our work follows this line of 3D egocentric understanding but differs in that we explicitly model interaction, i.e. jointly reconstruct both hands, object, and their contact. This allows for reconstruction of globally coherent hand-object interactions in the world coordinate frame. Video-Based Hand Pose Estimation. Estimating articulated hand motion from videos has long been a fundamental problem. Existing methods rely on multi-view setups or additional sensing modalities such as depth or IMU data [^10] [^13] [^26] [^52] [^53] to capture accurate 3D hand trajectories. Recent monocular approaches estimate hand poses directly from RGB videos by leveraging large-scale datasets [^23] [^36] [^38] [^78] and transformer-based architectures [^32] [^48] [^75] to regress MANO [^43] parameters. While these models achieve high accuracy for local hand poses, their predictions are expressed in canonical or camera-centric coordinates and thus cannot recover global trajectories in the world frame. To estimate global hand motion, recent works [^65] [^70] [^73] first predict local hand poses using a pretrained hand prior [^38], then infer world-space trajectories via test-time optimization or a learned neural model. However, these methods focus on hand motion in isolation. In contrast, we reconstruct object motion jointly with the hand in the world coordinate system, enabling accurate and contact-aware understanding of hand-object interactions. Hand-Object Interaction Reconstruction. Beyond pose estimation, a parallel line of research reconstructs or synthesizes 3D hand-object interactions. Template-based methods [^17] [^18] estimate object pose for a given template mesh, while template-free approaches [^11] [^22] [^56] [^62] [^64] directly --- ![Image](images/cc2c8c763ec3c317c0ee716b2971c1720208ffbdd7296c73c4ff7b5ba9b98f2b.jpg)
Figure 2. Reconstruction Using the Generative Motion Prior. Given a metric-SLAMed egocentric videos, and the object template O, we alternate the diffusion generation step and the guidance step to predict hand motion $\pmb{H}$, object 6D trajectory T, and binary contact $\pmb{C}$ as the final output $\pmb{x}_{0}$. The diffusion model $D_{\psi}$ is conditioned on object geometry and approximated hand $\pmb{H}$ from off-the-shelf hand estimator to diffuse the noisy parameters $\pmb{x}_{n}$. The guidance step refines the denoised output by optimizing task-specific objectives g to be consistent with the video observations $\hat{y}$ like 2D masks and contact. The contact labels $\hat{C}$ is automatically labeled by prompting a VLM.
recover object meshes with rich shape details. These methods reconstruct both hand and object motion during interaction but primarily emphasize detailed geometry, typically operating in hand- or object-centered coordinates and over short temporal windows. Our approach instead focuses on global 4D motion reconstruction, i.e. predicting how both the hand and the manipulated object move coherently through space and time, rather than recovering fine-grained surface geometry. 4D Reconstruction in the World Coordinate Frame. A related line of work aims to recover motion trajectories expressed in the world coordinate frame. Global human motion reconstruction methods [^14] [^45] [^46] [^61] [^72] estimate 3D body trajectories in world space by decoupling human and camera motion or leveraging learned motion priors. In the broader 4D tracking domain, approaches such as STaR-Track [^12] and SpatialTracker [^58] perform 3D-aware point tracking to capture scene-level dynamics and camera motion, without explicitly modeling human or object structure. FoundationPose [^55] instead tackles 6D object pose estimation given known object geometry, achieving robust per-frame alignment. Our approach also assumes an input object template but further estimates object poses and hand motion jointly from videos, capturing their coupled trajectories over time. ### 3. Method Given a metric-SLAMed egocentric video of object manipulation and a 3D object template, as shown in Figure 2, WHOLE holistically reconstructs both hand articulation and object 6D trajectories in world space. We first learn a generative motion prior of hand-object interactions in a gravity-aware local frame. At test time, we guide this pretrained prior by visual observations, object masks and VLM-derived binary contact cues, to generate globally consistent motions faithful to the input video. #### 3.1. Generative Hand-Object Motion Prior The generative prior is a diffusion model conditioned on a roughly estimated hand trajectory $\tilde{H}^{t=1:T}$ and an object template O, i.e. $c \equiv (\tilde{H}, O)$, generating refined hand motions $\tilde{H}^{t=1:T}$, object trajectories as SE(3) transforms $T^{t=1:T}$, and contact labels $C^{t=1:T}$, two binary indicators denoting contact with the left and right hands. It operates on a fix-length time window ( $T = 120$) and models how the object moves and interacts with the hands in a local gravity-aware coordinate frame, i.e. $p(\tilde{H}, T, C \mid O, \tilde{H})$. The approximated hand poses $\tilde{H}$ are obtained from an off-the-shelf hand estimator [^73] at test time, while during training, we also synthesize noisy hand tracks to avoid overfitting to a particular estimator. Interaction Motion Representation. We represent each hand using MANO parameters [^43], including global orientation $\Gamma$, translation $\Lambda$, articulation and shape $\Theta, \beta$, and joint positions and velocities J, J, i.e. $H \equiv (\Gamma, \Lambda, \Theta, J, J)$. We concatenate the left and right hand features and use the same representation for the approximated hand trajectory $\bar{H}$. We represent object poses in SE(3) using the 9D formulation [^77], and encode the (conditioning) object geometry with a BPS descriptor [^42] in its canonical frame, $O \equiv BPS(O)$. To better capture the spatial relationship between the hands and the object and to encourage the diffusion model to reason about fine-grained contact, we also compute a per-time "Ambient Sensor" feature [^47] [^74] between them. For each hand joint, we measure its displacement to the near- --- est point on the object surface transformed by the diffused pose $T_{i}[O]$, which is equivalent to a BPS of the posed object using hand joints as the vector basis i.e. $BPS_{J}(T_{i}[O])$. Gravity-Aware Local Coordinate Frame. Hands and objects are expressed in the gravity-aligned camera coordinate frame at the beginning of each sequence [^44] [^66]. Anchoring every sequence to the physical up direction and a consistent facing orientation allows our model to focus on relative hand-object motion rather than arbitrary global rotations. Specifically, we rotate the camera coordinates so that the z-axis aligns with the gravity vector. These canonicalized segments can later be transformed back to world coordinates and seamlessly stitched into long, continuous sequences during guided reconstruction. Approximating Inaccurate Hand Estimation. Our diffusion model refines approximated hand estimates $\tilde{H}$ at test time by reasoning about hand-object interactions. To enhance robustness and avoid overfitting to a specific off-the-shelf estimator, we synthesize imperfect conditioning hand tracks during training by perturbing ground-truth MANO parameters. We inject both trajectory-level noise $\varsigma^g$ and per-frame noise $\varsigma^t$ into the MANO parameters, then apply forward kinematics to produce perturbed joint positions that mimic real tracking noise, $\tilde{H}_{\varsigma^g,\varsigma^t}$. We further simulate occlusion and truncation by randomly dropping frames, and train jointly on both synthesized and real estimated hand conditions. Training Objective. The diffusion model learns to iteratively denoise from Gaussian noise z to generate clean interaction trajectories $x_{n=0}$. We adopt the conditional DDPM loss [^19], training the denoiser $D_{\phi}$ to recover clean trajectories from the corrupted ones $x_{n}$: $$ \begin{array}{r}{\mathcal{L}_{\mathrm{D D P M}}=\mathbb{E}_{t,\epsilon}\Big[w_{n}\left\|\tilde{\mathbf{x}}_{0}-D_{\phi}(\mathbf{x}_{n}\mid n,\bar{\boldsymbol{H}},\boldsymbol{O})\right\|_{2}^{2}\Big],}\end{array} $$ where $x_{n}$ is the diffusion parameters corrupted by Gaussian noise at step n and $\tilde{x}_{0}$ is the ground truth. $w_{n}$ is the variance schedule weight. The denoiser is implemented as a transformer decoder. In addition to the DDPM loss, we introduce auxiliary objectives to enhance realism. (1) Interaction Loss ( $L_{inter}$) encourages realistic hand-object contact by penalizing distances and ensuring near-rigid transport of contact points [^29]. (2) Consistency Loss ( $L_{const}$) promotes agreement between predicted hand features and their MANO forward kinematics. (3) Temporal Smoothness ( $L_{smooth}$) further penalizes large accelerations. We first warm up the model for 10k steps using only $L_{DDPM}$ before adding auxiliary terms. Please refer to appendix for detailed implementation. #### 3.2. Reconstruction as Guided Generation Sampling from the learned generative prior yields diverse and realistic hand-object motions. We leverage this motion ![Image](images/9ff545b47547cce1dbfa9df0e2437a5b7cbd709df62073e068b6770d63f91312.jpg)
Figure 3. Visual Prompt: We show two examples of the visual prompts provided to the VLM for contact detection.
prior for reconstruction from monocular videos by guiding the generation through classifier-guidance [^8]. Compared with another common guiding paradigm, score distillation sampling (SDS) [^41], classifier-guidance is faster, requiring only a single forward generation pass instead of thousands of optimization steps, and is less prone to model collapse. We find that two observations from the input video are crucial for guiding reconstruction: 2D masks that segments object and hand, and contact information indicating whether each hand is in contact with the object at the current time step. We employ a vision-language model (VLM) [^1] to obtain contact information. Test-Time Guidance Classifier guidance steers a diffusion model by modifying its score, the gradient of the log-probability $\nabla_{\boldsymbol{x}_n} \log p(\boldsymbol{x}_n)$, to incorporate task-specific objectives $g(\hat{\boldsymbol{y}}, \boldsymbol{x})$, $\hat{\boldsymbol{y}}$ are observations from the video. During sampling, the diffusion model predicts a score via the network $D_{\psi}$, which is then adjusted using the gradient of the task-specific objective g: $$ \begin{array}{r}{\tilde{\nabla}_{\pmb x_{n}}\log p(\pmb x_{n}\mathrm{~}|~{\hat{\pmb y}})=\nabla_{\pmb x_{n}}\log p(\pmb x_{n})-w\nabla_{\pmb x_{n}}g({\hat{\pmb y}},\pmb x_{n}),}\end{array} $$ This allows the generation process to remain plausible with respect to the learned prior while being steered toward samples faithful to the additional observations in the input videos. For the reconstruction task, we use three categories of objectives: (1) Reprojection term $g_{reproj}$, which aligns the generated sample with 2D observations, including contact binaries, 2D hand joints, and object masks. We apply a one-way Chamfer loss between the reprojected object and the segmented mask to handle occlusion and truncation; (2) Interaction term $g_{inter}$, which enforces realistic hand–object dynamics by minimizing distances, encouraging rigid transport under contact, and penalizing motion when contacts are absent in consecutive frames, following a similar formulation as $\mathcal{L}_{inter}$ during model training; (3) Temporal smoothness term $g_{temp}$, which regularizes temporal consistency and ensures smooth trajectories. VLM Contact Assignment. To detect hand-object contact, we use a Vision-Language Model (VLM) [^49] enhanced by spatial prompting and in-context learning. We --- ![Image](images/6f2eeef33482326ea08bb1626da6e68d75f5b6470f73ebfa71dc71b4375ba5ed.jpg)
Figure 4. HOI Generation Samples: We show two samples of interaction generated from our diffusion model with the same conditions. Objects are colored in red when contact is predicted and blue otherwise. We show 6 key frames among blended 150-frame generation.
segment and index hands and candidate objects, overlaying masks on the image to improve localization in cluttered scenes (Fig. 3). The VLM then identifies contacts via a JSON output, constrained by validation rules—such as a “one-out-of-k” contact limit—to distinguish true touch from proximity. As we find a tendency for false positives in VLM, we provide five annotated examples for calibration. These refinements increased the contact detection F1 score from 57% to 81%. For efficiency, we subsample videos to 3 fps and compute the reprojection term only at these frames. Blending Long Videos. Our diffusion model generates motion clips within a fixed temporal window of 120 frames. To reconstruct sequences longer than this window ( $L > 120$), we divide the sequence into overlapping sliding windows. During each diffusion step, we denoise all windows in parallel and blend overlapping regions [^3] and per-frame shape parameters into shared ones. This ensures smooth temporal transitions and consistency across windows. The blended full-length sequence is then refined under the guidance of the cost term g, after which each window's posterior $x_n$ is updated and the diffusion process continues. ### 4. Experiment We first train our diffusion model on HOT3D-CLIP [^2] and visualize its generated motions in Section 4. We then evaluate reconstructed motions both quantitatively and qualitatively on held-out sequences, comparing WHOLE with state-of-the-art baselines for hands, objects, and combined baselines formed by pairing the best-performing hand and object methods. In addition, we create specific subsets of the test set, including frames where objects are in contact with the hands, truncated, or out of view. We report reconstructed hand motion, object motion, and interaction motions both qualitatively and quantitatively (Section 4.2). Lastly, we analyze the effect of important components of our model (Section 4.3). Training Data and Setups. HOT3D is an egocentric dataset captured via Aria Glasses [^9], featuring real-world hand-object interactions. We utilize HOT3D-CLIP, a curated subset with verified annotations for hand-object poses, templates, and camera trajectories from shipped gravity-aware metric SLAM system. Each sequence consists of 150 frames (3 seconds). We train our diffusion model $D_{\psi}$ on 2,443 sequences. For evaluation, we hold out 50 dynamic object trajectories (displacement > 5 cm) from unseen sequences. While training contact labels are defined by proximity (< 5 mm), reconstruction labels are provided by prompting GPT-5 [^1]. #### 4.1. Visualizing Hand-Object Motion Generation We visualize conditional blended generation from our diffusion model in Figure 4. The conditioning hand motion $\bar{H}$ is taken from the test split, and we display six key frames from the full 150-frame sequence. Meshes are shown in red when hand-object contact is predicted. The two rows illustrate different generated samples conditioned on the same approximated hand motion. The generated hand-object motions are realistic and physically coherent. Within each frame, the spatial relationships between hands and objects are well captured, and the object dynamics appear plausible-objects stay relatively still when no contact is predicted and move naturally with ---
WA-MPJPE $\downarrow$W-MPJPE $\downarrow$ACC-NORM $\downarrow$PA-MPJPE $\downarrow$
HaMeR [^38]16.9328.3532.3112.76
HaWoR [^73]3.7611.264.158.99
FP+HaWoR-simple3.349.160.958.99
FP+HaWoR-contact5.8512.191.268.99
WHOLE (Ours)3.2610.410.586.67
Table 1. Hand Motion. We evaluate our method with state-of-the-art models in hand motion reconstruction. WA/W-MPJPE are in cm, PA-MPJPE is in mm.
the hands when contact occurs. The generated hand motion are similar as they just refine the approximated hand motion. Yet, the predicted contact timings vary, revealing different grasp and release moments. As a result, the reconstructed object trajectories also show diversity. For instance, in the second column, the mixer is picked up later in the first row. The hand-object relative poses are likewise diverse, such as the bottle being grasped in different ways in the last column. Please refer to the videos in the Sup. Mat. for the full generated motion dynamics. #### 4.2. Guided Reconstruction Metrics. Hands are evaluated using standard metrics from whole-body/hand pose estimation [^45] [^69] [^71] [^73]. W-MPJPE aligns all hand joints globally, WA-MPJPE aligns using the first frame, and PA-MPJPE performs per-frame Procrustes alignment. ACC-NORM measures joint acceleration error, reflecting temporal smoothness. Object poses are evaluated using common metrics from 6D pose estimation: AUC of ADD and ADD-S [^20] [^55] [^57]. ADD measures the average distance between corresponding model points transformed by the predicted and ground-truth poses, while ADD-S extends this metric to handle symmetric objects. To evaluate hand-object interaction, i.e., their relative spatial alignment, we report the AUC of ADD and ADD-S for object poses after globally aligning them using the predicted hand trajectories. Baselines. We compare WHOLE with state-of-the-art methods from their respective domains: HaWoR [^73] for world-grounded hand motion estimation and Foundation Pose (FP) [^55] for 6D object pose estimation. Since FP requires RGB-D input, we predict depth maps from RGB videos using Metric3D [^68] for metric depth estimation. (We also experiment with more recent depth estimation methods [^39] [^59] but observe surprisingly degraded performance.) To further evaluate joint reconstruction, we introduce combined baselines that integrate the best components from both domains, followed by test-time optimization: FP+HaWoR-simple/contact. These baselines use the same optimization objectives $g_{reproj}, g_{inter}, g_{temp}$ as our test-time guidance (Sec. 4.2). FP+HaWoR-simple optimizes hand and object trajectories with respect to observed evidence but excludes the interaction term $g_{inter}$, while FP+HaWoR-contact includes it to enforce hand-object consistency. ##### 4.2.1. Comparing Hand Motion HaMeR is an image-based hand prediction system thus its 4D trajectories in world coordinate are not aligned well. We find that HaWoR performs reasonably well in terms of hand motion quality. Note that the difference from the results reported in their paper comes from our evaluation on longer clips (150 frames instead of 100). Joint optimization without considering object trajectories with object motion in FP+HaWoR-simple yields smoother trajectories (lower ACC-NORM) and better global alignment. Adding the interaction loss (FP+HaWoR-contact) improves object performance (Table 2, 3) but reduces hand accuracy, revealing a trade-off between precise hand poses for stronger hand-object consistency. In contrast, WHOLE achieves the best overall performance (second best on one metric), excelling in global alignment, temporal smoothness, and local pose accuracy. Importantly, our method shows a clear improvement in local hand pose quality over HaWoR, highlighting the advantage of jointly reconstructing hands and objects within a unified generative framework. ##### 4.2.2. Object Motion FoundationPose (FP) performs moderately well when objects are in contact but struggles under truncation or out-of-view conditions. Note that for out-of-view frames, we interpolate 6D poses between the nearest visible frames in world space. Optimizing baseline only with reprojection and temporal smoothness (FP+HaWoR-simple) helps propagate poses to out-of-view frames but reduces accuracy on the contact subsets. It is likely because severe hand-object occlusion-reprojection loss on occluded masks can be misleading and ultimately degrades overall performance. Adding the interaction term (FP+HaWoR-contact) improves contact performance by leveraging hand cues but still falls short of the off-the-shelf FoundationPose. In contrast, WHOLE achieves consistently strong results across all subsets and metrics. It handles truncation and occlusion more robustly, producing smoother and more coherent object trajectories. By jointly reasoning about hand and object motion through a learned generative prior, it infers plausible movement even when parts of the object are missing or out of view. This demonstrates that our unified approach is more reliable than simply combining off-the-shelf methods with post-optimization. ##### 4.2.3. Interaction Quality While we report hand and object motion separately, we also evaluate their relative motion by measuring object error after alignment with the predicted hand trajectory. The combined baseline (FP+HaWoR-contact) shows clear improvements. ---
SplitAllContactTruncatedOut-of-view
ADD $\uparrow$ADD-S $\uparrow$ACC $\downarrow$ADD $\uparrow$ADD-S $\uparrow$ADD $\uparrow$ADD-S $\uparrow$ADD $\uparrow$ADD-S $\uparrow$ADD-S $\uparrow$
FoundationPose(FP) [^55]36.151.90.57946.666.618.633.84.819.6
FP+HaWoR-simple30.445.00.89638.657.519.428.97.813.1
FP+HaWoR-contact31.546.90.57242.362.422.735.88.917.7
Ours51.169.90.1155.877.342.560.228.843.8
Table 2. Object Motion. We evaluate our method with state-of-the-art baselines in object motion reconstruction. We report AUC of ADD, ADD-S on the whole test split (All) and 3 subsets (Contact, Truncated, and Out-of-View).
SplitAllContactTruncatedOut-of-view
ADD $\uparrow$ADD-S $\uparrow$ACC $\downarrow$ADD $\uparrow$ADD-S $\uparrow$ADD $\uparrow$ADD-S $\uparrow$ADD $\uparrow$ADD-S $\uparrow$
FP+HaWoR-simple29.044.00.89636.855.718.628.88.116.0
FP+HaWoR-contact36.853.80.57249.171.127.343.09.821.3
Table 3. Interactions Error. We evaluate our method with state-of-the-art baselines in object motion reconstruction. We report AUC of ADD, ADD-S on the whole test split (All) and 3 challenging subsets after aligning object motion with predicted hand motion.
ment over its variant without interaction optimization, highlighting the importance of modeling hand-object coupling. Finally, WHOLE surpasses both baselines by a large margin, demonstrating its strong capability to capture coherent and physically consistent hand-object interactions. We present visual comparisons in Figure 5, which reflect trends consistent with the quantitative results. In the top row of each example, we visualize the reconstructed hands and objects from the camera view, while the bottom row shows hand-object motion from an allocentric perspective across four keyframes. As shown, baseline methods often produce floating objects (highlighted in red) and unrealistic hand-object relations, particularly in the FP+HaWoR-simple variant. In contrast, WHOLE yields temporally smooth and spatially coherent reconstructions, capturing more natural and physically plausible hand-object interactions. #### 4.3. Ablation Study How good is VLM-annotated contact label? In Table 4, we replace the VLM-annotated contact labels at 3 fps with ground-truth contact annotations at 30 fps. While there remains some room for improvement, the VLM-based results is close to the ceiling performance across hand motion, object motion, and their relative spatial alignment. Is intertwined generation and optimization necessary? WHOLE alternates diffusion steps with task-specific guidance throughout the generation process. To examine the importance of this coupling, we compare it with a “Gen+Opt” variant that first performs unguided generation, then applies post-hoc optimization using the same objective terms. Results in Table 4 Line 2 show that incorporating guidance during diffusion is essential. It keeps the model's samples within the data manifold while progressively refining motion under task constraints.
HandObjectInteraction
W- $\downarrow$WA- $\downarrow$ACC $\downarrow$ADD $\uparrow$ADD-S $\uparrow$ACC $\downarrow$ADD $\uparrow$ADD-S $\uparrow$ACC $\downarrow$
GT contact10.790.556.6153.772.20.0756.174.50.07
Gen+Opt15.920.769.3444.361.00.1345.062.10.13
w/o $\mathcal{L}_{inter}$10.470.616.7028.542.90.0930.645.50.09
WHOLE10.410.586.6751.169.90.1153.572.30.11
Table 4. Ablation Study. We compare our full model on hand, object, and relative motion quality against variants that use ground-truth contact, not alternate diffusion and guidance step, or exclude the interaction objective.
Train DataTest DataTest Data
HOT3DH2O
ADD $\uparrow$ADD-S $\uparrow$ACC $\downarrow$ADD $\uparrow$ADD-S $\uparrow$ACC $\downarrow$
HOT3DWHOLE (Ours)51.169.90.1144.764.90.23
H2OH2OTR [^5]3.27.725.462.177.10.48
Table 5. Zero-Shot Generalization. Gray lines denote zero-shot test dataset while unshaded line denotes in-domain test split.
How much does the interaction loss help? In Table 4 Line 3, we show that the interaction objectives that capture the spatial relationships and relative dynamics conditioned on predicted contact is also important for producing faithful object motion and realistic hand-object interactions. How well does the model generalize? In Table 5, we evaluate WHOLE zero-shot on the unseen H2O dataset [^27]. While WHOLE experiences a moderate performance drop, RGB-conditioned baselines like H2OTR [^5] collapse catastrophically out-of-distribution. We attribute this robustness to our motion-space prior, which exhibits a smaller domain gap than the appearance-based representations used in prior work. While encouraging, scaling to more diverse training data and broader cross-dataset evaluation remain important directions for achieving truly generalizable reconstruction. #### 4.4. Application: Hand-Guided HOI Planner Our framework is flexible beyond reconstruction: given a coarse hand trajectory $\bar{H}$, picking and placing times (con- --- ![Image](images/8ebdc3c9a5c36daa417a2d2617d3706dbd87ad9b87371edd9b5c84e4b418bce7.jpg)
Figure 5. HOI Visualization. We show hand-object reconstructions from GT (green), FP+HaWor-Simple (purple), FP+HaWor-Contact (pink), and WHOLE (blue). Red circle highlights floating objects. We encourage readers to see videos in Sup. Mat.
tact label C), along with the object template, we can directly synthesize diverse hand-object interaction motions without any video input. This could potentially enable a robot planner to enumerate candidate object trajectories for a specific coarse hand motion at a specific picking and placing timing. Specifically, the coarse hand trajectory serves as the diffusion conditioning, while contact labels are injected at each guidance step via an L2 loss on binary contact predictions alongside $g_{\text{inter}}$ and $g_{\text{temp}}$. Please refer to our project page for video results. ### 5. Discussion Limitations and Future Work. Our current framework reconstructs each hand-object pair independently. A natural next step is extending it to scene-level, multi-object reconstruction through joint diffusion and scene-level objectives. The method also assumes a known object template, which could be relaxed by incorporating LLM-based retrieval $^{[56]}$ or template-free generation $^{[11,64]}$. Finally, since our generative prior is trained on a single dataset, scaling it with recent large-scale hand-object datasets $^{[25,33]}$ would improve generalization and robustness. Conclusion. We introduced WHOLE, a unified framework for reconstructing hand articulation and object trajectories from metric-SLAMed egocentric videos. By combining a learned generative motion prior with visual and contact guidance, it achieves coherent and plausible reconstructions under challenging conditions. We believe this framework offers a promising step toward scalable, scene-level hand-object understanding and generative modeling of human interactions in everyday environments. --- ## References [^1]: Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. 4, 5 [^2]: Prithviraj Banerjee, Sindi Shkodrani, Pierre Moulon, Shreyas Hampali, Shangchen Han, Fan Zhang, Linguang Zhang, Jade Fountain, Edward Miller, Selen Basol, et al. Hot3d: Hand and object tracking in 3d from egocentric multi-view videos. In CVPR, 2025. 2, 5 [^3]: Omer Bar-Tal, Lior Yariv, Yaron Lipman, and Tali Dekel. Multidiffusion: Fusing diffusion paths for controlled image generation.(2023). ICLR, 2023. 5 [^4]: Yu-Wei Chao, Wei Yang, Yu Xiang, Pavlo Molchanov, Ankur Handa, Jonathan Tremblay, Yashraj S. Narang, Karl Van Wyk, Umar Iqbal, Stan Birchfield, Jan Kautz, and Dieter Fox. DexYCB: A benchmark for capturing hand grasping of objects. In CVPR, 2021. 2 [^5]: Hoseong Cho, Chanwoo Kim, Jihyeon Kim, Seongyeong Lee, Elkhan Ismayilzada, and Seungryul Baek. Transformer-based unified recognition of two hands manipulating objects. In CVPR, 2023. 7 [^6]: Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. Scaling egocentric vision: The epic-kitchens dataset. In ECCV, 2018. 2 [^7]: Ahmad Darkhalil, Rhodri Guerrier, Adam W Harley, and Dima Damen. Egopoints: Advancing point tracking for egocentric videos. In WACV, 2025. 2 [^8]: Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. NeurIPS, 2021. 4 [^9]: Jakob Engel, Kiran Somasundaram, Michael Goesele, Albert Sun, Alexander Gamino, Andrew Turner, Arjang Talatof, Arnie Yuan, Bilal Souti, Brighid Meredith, et al. Project aria: A new tool for egocentric multi-modal ai research. arXiv preprint arXiv:2308.13561, 2023. 5 [^10]: Zicong Fan, Omid Taheri, Dimitrios Tzionas, Muhammed Kocabas, Manuel Kaufmann, Michael J Black, and Otmar Hilliges. Arctic: A dataset for dexterous bimanual hand-object manipulation. In CVPR, 2023. 2 [^11]: Zicong Fan, Maria Parelli, Maria Eleni Kadoglou, Xu Chen, Muhammed Kocabas, Michael J Black, and Otmar Hilliges. Hold: Category-agnostic 3d reconstruction of interacting hands and objects from video. In CVPR, 2024. 2, 8, 12 [^12]: Haiwen Feng, Junyi Zhang, Qianqian Wang, Yufei Ye, Pengcheng Yu, Michael J Black, Trevor Darrell, and Angjoo Kanazawa. St4rtrack: Simultaneous 4d reconstruction and tracking in the world. ICCV, 2025. 2, 3 [^13]: Guillermo Garcia-Hernando, Shanxin Yuan, Seungryul Baek, and Tae-Kyun Kim. First-person hand action benchmark with RGB-D videos and 3D hand pose annotations. In CVPR, 2018. 2 [^14]: Erik Gartner, Mykhaylo Andriluka, Hongyi Xu, and Cristian Sminchisescu. Pace: Physics-based animation and control of expressive 3D characters. In CVPR, 2022. 3 [^15]: Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, Miguel Martin, Tushar Nagarajan, Ilija Radosavovic, Santhosh Kumar Ramakrishnan, Fiona Ryan, Jayant Sharma, Michael Wray, Mengmeng Xu, Eric Zhongcong Xu, Chen Zhao, Siddhant Bansal, Dhruv Batra, Vincent Cartillier, Sean Crane, Tien Do, Morrie Doulaty, Akshay Erapalli, Christoph Feichtenhofer, Adriano Fragomeni, Qichen Fu, Christian Fuegen, Abrham Gebreselasie, Cristina Gonzalez, James Hillis, Xuhua Huang, Yifei Huang, Wenqi Jia, Weslie Khoo, Jachym Kolar, Satwik Kottur, Anurag Kumar, Federico Landini, Chao Li, Yanghao Li, Zhenqiang Li, Karttikeya Mangalam, Raghava Modhugu, Jonathan Munro, Tullie Murrell, Takumi Nishiyasu, Will Price, Paola Ruiz Puentes, Merey Ramazanova, Leda Sari, Kiran Somasundaram, Audrey Southerland, Yusuke Sugano, Ruijie Tao, Minh Vo, Yuchen Wang, Xindi Wu, Takuma Yagi, Yunyi Zhu, Pablo Arbelaez, David Crandall, Dima Damen, Giovanni Maria Farinella, Bernard Ghanem, Vamsi Krishna Ithapu, C. V. Jawahar, Hanbyul Joo, Kris Kitani, Haizhou Li, Richard Newcombe, Aude Oliva, Hyun Soo Park, James M. Rehg, Yoichi Sato, Jianbo Shi, Mike Zheng Shou, Antonio Toralba, Lorenzo Torresani, Mingfei Yan, and Jitendra Malik. Ego4d: Around the World in 3,000 Hours of Egocentric Video. In CVPR. 2022. 2 [^16]: Kristen Grauman, Andrew Westbury, Lorenzo Torresani, Kris Kitani, Jitendra Malik, Triantafyllos Afouras, Kumar Ashutosh, Vijay Baiyya, Siddhant Bansal, Bikram Boote, et al. Ego-exo4d: Understanding skilled human activity from first-and third-person perspectives. In CVPR, 2024. 2 [^17]: Shreyas Hampali, Mahdi Rad, Markus Oberweger, and Vincent Lepetit. Honnotate: A method for 3D annotation of hand and object poses. In CVPR, 2020. 2 [^18]: Yana Hasson, Gul Varol, Dimitrios Tzionas, Igor Kalevatykh, Michael J Black, Ivan Laptev, and Cordelia Schmid. Learning joint reconstruction of hands and manipulated objects. In CVPR, 2019. 2 [^19]: Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurIPS, 2020. 4 [^20]: Tomáš Hodaň, Martin Sundermeyer, Bertram Drost, Yann Labbé, Eric Brachmann, Frank Michel, Carsten Rother, and Jiří Matas. Bop challenge 2020 on 6d object localization. In ECCV, 2020. 6, 12 [^21]: Ryan Hoque, Peide Huang, David J Yoon, Mouli Sivapurapu, and Jian Zhang. Egodex: Learning dexterous manipulation from large-scale egocentric video. arXiv preprint arXiv:2505.11709, 2025. 2 [^22]: Di Huang, Xiaopeng Ji, Xingyi He, Jiaming Sun, Tong He, Qing Shuai, Wanli Ouyang, and Xiaowei Zhou. Reconstructing hand-held objects from monocular video. In SIGGRAPH Asia, 2022. 2 [^23]: Sheng Jin, Lumin Xu, Jin Xu, Can Wang, Wentao Liu, Chen Qian, Wanli Ouyang, and Ping Luo. Whole-body human pose estimation in the wild. In ECCV, 2020. 2 --- [^24]: Simar Kareer, Dhruv Patel, Ryan Punamiya, Pranay Mathur, Shuo Cheng, Chen Wang, Judy Hoffman, and Danfei Xu. Egomimic: Scaling imitation learning via egocentric video. In ICRA, 2025. 2 [^25]: Jeonghwan Kim, Jisoo Kim, Jeongheon Na, and Hanbyul Joo. Parahome: Parameterizing everyday home activities towards 3d generative modeling of human-object interactions. In CVPR, 2025. 8 [^26]: Taein Kwon, Bugra Tekin, Jan Stühmer, Federica Bogo, and Marc Pollefeys. H2o: Two hands manipulating objects for first person interaction recognition. In ICCV, 2021. 2 [^27]: Taein Kwon, Bugra Tekin, Jan Stühmer, Federica Bogo, and Marc Pollefeys. H2o: Two hands manipulating objects for first person interaction recognition. In ICCV, 2021. 7 [^28]: Jiaman Li, Karen Liu, and Jiajun Wu. Ego-body pose estimation via ego-head pose estimation. In CVPR, 2023. 2 [^29]: Jiaman Li, Jiajun Wu, and C Karen Liu. Object motion guided human motion synthesis. TOG, 2023. 4, 12 [^30]: Jiaman Li, Alexander Clegg, Roozbeh Mottaghi, Jiajun Wu, Xavier Puig, and C Karen Liu. Controllable human-object interaction synthesis. In ECCV, 2024. 12 [^31]: Zhengqi Li, Richard Tucker, Forrester Cole, Qianqian Wang, Linyi Jin, Vickie Ye, Angjoo Kanazawa, Aleksander Holynski, and Noah Snavely. Megasam: Accurate, fast and robust structure and motion from casual dynamic videos. In CVPR, 2025. 2 [^32]: Kevin Lin, Lijuan Wang, and Zicheng Liu. Mesh graphormer. In ICCV, 2021. 2 [^33]: Jiaxin Lu, Chun-Hao Paul Huang, Uttaran Bhattacharya, Qixing Huang, and Yi Zhou. Humoto: A 4d dataset of mocap human object interactions. ICCV, 2025. 8 [^34]: Zhaoyang Lv, Nicholas Charron, Pierre Moulon, Alexander Gamino, Cheng Peng, Chris Sweeney, Edward Miller, Huixuan Tang, Jeff Meissner, Jing Dong, Kiran Somasundaram, Luis Pesqueira, Mark Schwesinger, Omkar Parkhi, Qiao Gu, Renzo De Nardi, Shangyi Cheng, Steve Saarinen, Vijay Baiyya, Yuyang Zou, Richard Newcombe, Jakob Julian Engel, Xiaqing Pan, and Carl Ren. Aria everyday activities dataset, 2024. 2 [^35]: Lingni Ma, Yuting Ye, Fangzhou Hong, Vladimir Guzov, Yifeng Jiang, Rowan Postyeni, Luis Pesqueira, Alexander Gamino, Vijay Baiyya, Hyo Jin Kim, Kevin Bailey, David Soriano Fosas, C. Karen Liu, Ziwei Liu, Jakob Engel, Renzo De Nardi, and Richard Newcombe. Nymeria: A massive collection of multimodal egocentric daily motion in the wild. In ECCV, 2024. 2 [^36]: Gyeongsik Moon, Shoou-I Yu, He Wen, Takaaki Shiratori, and Kyoung Mu Lee. Interhand2.6m: A dataset and baseline for 3D interacting hand pose estimation from a single rgb image. In ECCV, 2020. 2 [^37]: Suvam Patra, Kartikeya Gupta, Faran Ahmad, Chetan Arora, and Subhashis Banerjee. Ego-slam: A robust monocular slam for egocentric videos. In WACV, 2019. 2 [^38]: Georgios Pavlakos, Dandan Shan, Ilija Radosavovic, Angjoo Kanazawa, David Fouhey, and Jitendra Malik. Reconstructing hands in 3D with transformers. In CVPR, 2024. 2, 6 [^39]: Luigi Piccinelli, Christos Sakaridis, Yung-Hsu Yang, Mattia Segu, Siyuan Li, Wim Abbeloos, and Luc Van Gool. Unidepthv2: Universal monocular metric depth estimation made simpler. TPAMI, 2025. 6 [^40]: Chiara Plizzari, Shubham Goel, Toby Perrett, Jacob Chalk, Angjoo Kanazawa, and Dima Damen. Spatial cognition from egocentric video: Out of sight, not out of mind. In 3DV, 2025. 2 [^41]: Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. ICLR, 2022. 4 [^42]: Sergey Prokudin, Christoph Lassner, and Javier Romero. Efficient learning on point clouds with basis point sets. In ICCV, 2019. 3 [^43]: Javier Romero, Dimitrios Tzionas, and Michael J. Black. Embodied hands: Modeling and capturing hands and bodies together. In TOG, 2017. 2, 3 [^44]: Zehong Shen, Huaijin Pi, Yan Xia, Zhi Cen, Sida Peng, Zechen Hu, Hujun Bao, Ruizhen Hu, and Xiaowei Zhou. World-grounded human motion recovery via gravity-view coordinates. In SIGGRAPH Asia, 2024. 4 [^45]: Soyong Shin, Juyong Kim, Eni Halilaj, and Michael J. Black. WHAM: Reconstructing world-grounded humans with accurate 3D motion. In CVPR, 2024. 2, 3, 6, 12 [^46]: Yu Sun, Wu Liu, Qian Bao, Yili Fu, Tao Mei, and Michael J Black. Trace: Monocular global human trajectory estimation in the wild. In CVPR, 2023. 3 [^47]: Omid Taheri, Yi Zhou, Dimitrios Tzionas, Yang Zhou, Duygu Ceylan, Soren Pirk, and Michael J. Black. GRIP: Generating interaction poses using latent consistency and spatial cues. In 3DV, 2024. 3 [^48]: Xiao Tang, Tianyu Wang, and Chi-Wing Fu. Towards accurate alignment in real-time 3D hand-mesh reconstruction. In ICCV, 2021. 2 [^49]: Gemini Robotics Team, Saminda Abeyruwan, Joshua Ainslie, Jean-Baptiste Alayrac, Montserrat Gonzalez Arenas, Travis Armstrong, Ashwin Balakrishna, Robert Baruch, Maria Bauza, Michiel Blokzijl, et al. Gemini robotics: Bringing ai into the physical world. arXiv preprint arXiv:2503.20020, 2025. 4 [^50]: Guy Tevet, Sigal Raab, Brian Gordon, Yonatan Shafir, Daniel Cohen-Or, and Amit H Bermano. Human motion diffusion model. ICLR, 2023. 12 [^51]: Vadim Tschernezki, Ahmad Darkhalil, Zhifan Zhu, David Fouhey, Iro Laina, Diane Larlus, Dima Damen, and Andrea Vedaldi. Epic Fields: Marrying 3D geometry and video understanding. NeurIPS, 2024. 2 [^52]: Ruocheng Wang, Pei Xu, Haochen Shi, Elizabeth Schumann, and C Karen Liu. F\"urelise: Capturing and physically synthesizing hand motions of piano performance. SIGGRAPH Asia, 2024. 2 [^53]: Robert Y Wang and Jovan Popović. Real-time hand-tracking with a color glove. TOG, 2009. 2 [^54]: Weizhuo Wang, C Karen Liu, and Monroe Kennedy III. Egonov: Egocentric scene-aware human trajectory prediction. arXiv preprint arXiv:2403.19026, 2024. 2 --- [^55]: Bowen Wen, Wei Yang, Jan Kautz, and Stan Birchfield. Foundationpose: Unified 6d pose estimation and tracking of novel objects. In CVPR, 2024. 3, 6, 7, 12 [^56]: Jane Wu, Georgios Pavlakos, Georgia Gkioxari, and Jitendra Malik. Reconstructing hand-held objects in 3d from images and videos. arXiv preprint arXiv:2404.06507, 2024. 2, 8 [^57]: Yu Xiang, Tanner Schmidt, Venkatraman Narayanan, and Dieter Fox. Posecnn: A convolutional neural network for 6d object pose estimation in cluttered scenes. RSS, 2017. 6 [^58]: Yuxi Xiao, Jianyuan Wang, Nan Xue, Nikita Karaev, Yuri Makarov, Bingyi Kang, Xing Zhu, Hujun Bao, Yujun Shen, and Xiaowei Zhou. Spatialtrackerv2: 3d point tracking made easy. In ICCV, 2025. 2, 3 [^59]: Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything v2. NeurIPS, 2024. 6 [^60]: Ruihan Yang, Qinxi Yu, Yecheng Wu, Rui Yan, Borui Li, An-Chieh Cheng, Xueyan Zou, Yunhao Fang, Hongxu Yin, Sifei Liu, Song Han, Yao Lu, and Xiaolong Wang. Egovla: Learning vision-language-action models from egocentric human videos, 2025. 2 [^61]: Vickie Ye, Georgios Pavlakos, Jitendra Malik, and Angjoo Kanazawa. Decoupling human and camera motion from videos in the wild. In CVPR, 2023. 3 [^62]: Yufei Ye, Abhinav Gupta, and Shubham Tulsiani. What's in your hands? 3d reconstruction of generic objects in hands. In CVPR, 2022. 2 [^63]: Yufei Ye, Abhinav Gupta, Kris Kitani, and Shubham Tulsiani. G-HOP: Generative hand-object prior for interaction reconstruction and grasp synthesis. In CVPR, 2024. 2, 12 [^64]: Yufei Ye, Abhinav Gupta, Kris Kitani, and Shubham Tulsiani. G-hop: Generative hand-object prior for interaction reconstruction and grasp synthesis. In CVPR, 2024. 2, 8 [^65]: Yufei Ye, Yao Feng, Omid Taheri, Haiwen Feng, Shubham Tulsiani, and Michael J Black. Predicting 4d hand trajectory from monocular videos. 3DV, 2026. 2 [^66]: Brent Yi, Vickie Ye, Maya Zheng, Yunqi Li, Lea Müller, Georgios Pavlakos, Yi Ma, Jitendra Malik, and Angjoo Kanazawa. Estimating body and hand motion in an ego-sensed world. In CVPR, 2025. 4 [^67]: Brent Yi, Vickie Ye, Maya Zheng, Yunqi Li, Lea Müller, Georgios Pavlakos, Yi Ma, Jitendra Malik, and Angjoo Kanazawa. Estimating body and hand motion in an ego-sensed world. In CVPR, 2025. 2 [^68]: Wei Yin, Chi Zhang, Hao Chen, Zhipeng Cai, Gang Yu, Kaixuan Wang, Xiaozhi Chen, and Chunhua Shen. Metric3d: Towards zero-shot metric 3d prediction from a single image. In ICCV, 2023. 6 [^69]: Paper Authors Your. Slahmr: Simultaneous localization and human mesh recovery. In CVPR, 2023. 2, 6, 12 [^70]: Zhengdi Yu, Stefanos Zafeiriou, and Tolga Birdal. Dynamr: Recovering 4d interacting hand motion from a dynamic camera. In CVPR, 2025. 2 [^71]: Zhengdi Yu, Stefanos Zafeiriou, and Tolga Birdal. Dynamr: Recovering 4d interacting hand motion from a dynamic camera. In CVPR, 2025. 6 [^72]: Ye Yuan, Umar Iqbal, Pavlo Molchanov, Kris Kitani, and Jan Kautz. Glamr: Global occlusion-aware human mesh recovery with dynamic cameras. In CVPR, 2022. 3 [^73]: Jinglei Zhang, Jiankang Deng, Chao Ma, and Rolandos Alexandros Potamias. Hawor: World-space hand motion reconstruction from egocentric videos. CVPR, 2025. 2, 3, 6, 12 [^74]: Wanyue Zhang, Rishabh Dabral, Vladislav Golyanik, Vasileios Choutas, Eduardo Alvarado, Thabo Beeler, Marc Habermann, and Christian Theobalt. Bimart: A unified approach for the synthesis of 3d bimanual interaction with articulated objects. In CVPR, 2025. 3 [^75]: Xiong Zhang, Qiang Li, Hong Mo, Wenbo Zhang, and Wen Zheng. End-to-end hand mesh recovery from a monocular rgb image. In ICCV, 2019. 2 [^76]: Yunhan Zhao, Haoyu Ma, Shu Kong, and Charless Fowlkes. Instance tracking in 3d scenes from egocentric videos. In CVPR, 2024. 2 [^77]: Yi Zhou, Connelly Barnes, Jingwan Lu, Jimei Yang, and Hao Li. On the continuity of rotation representations in neural networks. In CVPR, 2019. 3 [^78]: Christian Zimmermann, Duygu Ceylan, Jimei Yang, Bryan Russell, Max Argus, and Thomas Brox. Freihand: A dataset for markerless capture of hand pose and shape from single rgb images. In PICCV, 2019. 2 --- # WHOLE: World-Grounded Hand-Object Lifted from Egocentric Videos Supplementary Material In supplementary material, we provide further details on implementing network, full VLM prompt, and evaluation metrics (Sec. A). We also visualize more comparisons and results in supplementary videos. ### A. Implementation Details Network Architecture. We use a 4-layer transformer decoder with 4 attention heads for hand-to-object diffusion. The network is non-autoregressive, processing sequences jointly following [^50], with 12.35M parameters. The diffusion variable x is a 73-dimensional vector comprising: a 9D object state, a 2D bimanual contact indicator, and bimanual hand representations (2 × 31D)—global orientation (3), translation (3), pose PCA (15), and shape parameters (10). All inputs are projected to a consistent latent dimension of 512. The network is trained for 1,000,000 iterations using AdamW at a learning rate of $2 \times 10^{-4}$. To reduce overfitting, we augment the object template by sampling a random canonical pose applying a random rotation and a small translation jitter—for each training window. Training Loss. In addition to the DDPM loss, we introduce auxiliary objectives to enhance realism. (1) Interaction Loss ( $\mathcal{L}_{\text{inter}}$) encourages realistic contact between the predicted hand-object motions and contact labels. It penalizes hand-object distances when contact is predicted and enforces near-rigid transport of contact points across consecutive contact frames [^29] [^30]. Specifically, for each hand joint, we find its nearest object point $p^i$, rotate it by the object's relative motion, and penalize deviation from its counterpart $p^{i+1}$, i.e. $\|\boldsymbol{R}^{i+1}(\boldsymbol{R}^i)^T p^i - p^{i+1}\|$, where $\boldsymbol{R}$ is the object rotation. (2) Consistency Loss ( $\mathcal{L}_{\text{const}}$) promotes agreement among hand features before and after MANO forward kinematics, $\|\boldsymbol{J}_\psi - \text{MANO}(\boldsymbol{\Gamma}_\psi, \boldsymbol{\Lambda}_\psi, \boldsymbol{\Theta}_\psi)\|_2$. (3) Temporal Smoothness ( $\mathcal{L}_{\text{smooth}}$) further penalizes large accelerations. Running Time. On a single NVIDIA RTX 6000 Blackwell GPU, our model processes a 150-frame clip in an average of 59.34 seconds. The inference time is dominated by the guidance step (59.06s), with the diffusion step requiring only 0.28s. This represents an orders-of-magnitude speedup over prior works such as [^11] (30 hours) and [^63] (1 hour). The peak memory footprint is 14GB. VLM queries take 18.6s on average per image with GPT-5. VLM Prompt. We prompt a VLM to label contact info, with additional in-context-learning examples. Full prompts are illustrated in Table in this appendix. Evaluation Metrics. All metrics are computed on 150-frame clips, which correspond to the original sequence length in HOT3D-CLIP, in contrast to prior work [^73], which typically evaluates on shorter 60–100 frame segments taken from the middle of the videos. To compute W/WA-MPJPE, we align the predicted trajectory to the ground truth using an affine transformation (scale, rotation, translation) estimated from selected keypoints. WA-MPJPE uses all joints from all frames, while W-MPJPE uses only the joints from the first two frames. Although trajectory error could be computed without alignment given ground-truth cameras, we follow the standard alignment protocol used in prior work [^45] [^69] [^73]. For ADD/ADD-S of objects, we align predictions using the ground-truth camera poses. For HOI ADD/ADD-S, we first globally align the hand trajectory (as in WA-MPJPE) and then evaluate object error in this aligned space. The usual AUC [^20] [^55] threshold of 0.1 is overly strict for egocentric HOI due to severe occlusion, truncation, and out-of-view frames, leading to saturated low scores. We therefore use a more permissive threshold of 0.3 to obtain a more informative evaluation. --- You are a precise visual classifier for hand-object contact detection in cluttered scenes. CRITICAL CONSTRAINTS: 1. Each hand (left/right) can be in contact with AT MOST ONE object at a time. 2. "In contact" means direct physical touch: grasping, holding, pressing, or any visible contact. 3. If a hand is not clearly touching any object, you must mark all objects as 0 for that hand. ## User Prompt Template Analyze this image for hand{object contact (actual touching, not just reaching). VISUAL GUIDANCE: The image has been annotated with colored masks: - GREEN dot = Left hand - RED dot = Right hand - Other COLORED masks = Candidate objects (each object has a unique color) CANDIDATE OBJECTS (in order): 1. obj1 2. obj2 3. obj3 ... STRICT DEFINITION OF CONTACT: For this task, contact means clear physical touching in this frame only. Contact (label = 1) requires BOTH: 1. Mask intersection: - The hand mask and the object mask share some pixels or directly overlap at the boundary (no visible gap). 2. Touching region: - The overlap is at a plausible touching area (finger tips, fingers, palm, side of hand) on the visible surface of the object. NO Contact (label = 0) in all of these cases: - The hand is reaching toward, hovering above, or very close to an object with a visible gap between masks. - The hand is aligned in depth (e.g., above or behind the object) but the masks do not intersect. - The hand is in a pose that suggests future contact, but there is no current touching in this single frame. - There is only a tiny, ambiguous intersection (1{2 pixels) that could be noise or occlusion. In such uncertain cases, choose 0 (no contact). IMPORTANT: - **Reaching or hovering is NOT contact.** - **If you are unsure whether contact is happening, choose 0 (no contact).** CONSTRAINTS (VALIDATION CHECK): - Each hand can touch AT MOST ONE object. - Sum of left across all objects must be <= 1. - Sum of right across all objects must be <= 1. - If a hand is not clearly touching any object, it should have 0 for all objects. OUTPUT FORMAT: Return only a JSON object in this exact format (no extra text): { "obj1": {"left": 0, "right": 1}, "obj2": {"left": 0, "right": 0}, "obj3": {"left": 1, "right": 0} } --- } Where: - 1 = the specified hand is clearly touching that object in this frame.