# ACE-Ego-Hand: Repurposing Video Diffusion Models for Occlusion-Robust Egocentric 3D Hand Motion Recovery

Yufei Liu $^{1,4}$, Xixi Wang $^{2}$, Hao Li $^{3,4}$, Ganlong Zhao $^{3,4}$, Kaitong Cai $^{4}$, Chengkai Jin $^{2,4}$, Chunxiao Liu $^{4}$, Jianbo Liu $^{4}$, Siyuan Huang $^{4\dagger}$, Xingang Pan $^{2}$, Hongsheng Li $^{3,4\ddagger}$

 $^{1}$Shanghai Jiao Tong University $^{2}$Nanyang Technological University

 $^{3}$The Chinese University of Hong Kong $^{4}$ACE Robotics

† Project leader. ☑ Corresponding author.

![Image](images/a1f83758bdfed7588d5905b85d05fa4dc7e5326517c3474fbd9463457c319cf9.jpg)

![Image](images/0e5d6781d1956f81cdc94a2fb9b9274d3b429a8344ab905d0dd59b1cb84a88d0.jpg)

<div style="text-align: center;">Figure 1. Occlusion-robust egocentric 3D hand motion recovery. Left: ACE-Ego-Hand applied to a raw RGB video clip. ACE-Ego-Hand maintains hand identity and trajectory continuity even when a hand temporarily leaves the field of view (frames shaded yellow). Right: ACE-Ego-Hand leads on six of the seven metrics shown (F1, MPJPE-p, PA-p, EPE $_{2D}$-p, GO-p, CT-p, and Jitter, each averaged over ARCTIC, HOT3D, and HOI4D, with farther from the center being better) and matches the strongest baseline on detection F1.</div>

## Abstract

Egocentric video offers scalable manipulation data for embodied AI, yet recovering metric 3D hand trajectories remains challenging due to severe object occlusion and frequent out-of-sight gaps. Existing single-frame and windowed temporal regressors fail when a hand briefly leaves the frame, while recent video diffusion models (VDMs) rely on heavy, stochastic multi-step sampling as pixel-space renderers. We instead repurpose a VDM into a deterministic geometry encoder. A single forward pass over the clean latent exposes scene content beyond current observations, including occluded and out-of-sight hands. We introduce ACE-Ego-Hand, an offline clip-level framework that extracts features via a Deterministic Clean-Latent Encoder and decodes them with a Bidirectional Spatiotemporal Decoder. ACE-Ego-Hand recovers continuous bimanual trajectories with metric placement and no external detector, while a Ray-Based Camera Solver supports a second configuration that needs no test-time camera intrinsics. Across five egocentric benchmarks, ACE-Ego-Hand sets a

new state of the art, cutting MPJPE-p by 30% on occlusion-heavy ARCTIC and 40% on HOT3D. These gains reach 46%–61% once out-of-sight hands are included in the evaluation, offering a scalable path from everyday human video to robot manipulation data. The code is publicly available at github.com/ggxxii/ACE-Ego-Hand.

### 1. Introduction

Egocentric video offers a scalable source of robot manipulation data [^8] [^12] [^16] [^38], but exploiting this source requires recovering metric 3D hand motion from raw footage. Two constraints of egocentric capture make this recovery particularly challenging: frequent hand-object-interaction (HOI) occlusions, and swift head motion that pushes hands entirely out of sight (OOS) and leaves no visual evidence in the field of view (Figure 1). Existing 3D hand motion reconstruction methods rely heavily on per-frame detectors [5, 21, 23–25] or SLAM-anchored tracking [^36] [^39] [^40] [^42], the latter inherently suffering from error accumulation. Once a hand becomes unobserved or exits the camera

---

frustum, windowed temporal models [^4] [^18] [^28] lose historical context and struggle to resume coherent tracking upon re-emergence. To tackle this, we argue that metric 3D hand trajectory recovery from egocentric video is fundamentally an offline clip-level spatiotemporal reasoning task.

In order to effectively reconstruct plausible hand dynamics during OOS intervals, rich physical and temporal priors are essential. Pretrained video diffusion models (VDMs) naturally capture such priors of physical dynamics and human-object interaction. Recent generative frameworks [^37] exploit video diffusion priors for hand pose estimation under severe HOI occlusions. However, they rely on an indirect pixel-rendering pipeline—executing multi-step sampling to synthesize pixel-space outputs from which poses are then regressed. This roundabout route incurs sampling latency and may propagate generative artifacts into the pose estimates. Furthermore, directly regressing 3D hand coordinates in the camera frame conflates hand articulation with rapid camera ego-motion and imposes strict calibration dependencies.

To address these limitations, we present ACE-Ego-Hand, an offline clip-level 3D hand motion reconstruction framework. Our key insight is twofold: the VDM serves reconstruction best as an encoder rather than a renderer, and its generic features need end-to-end 3D supervision to become geometry-aware. ACE-Ego-Hand therefore repurposes a pretrained VDM [^33] as a Deterministic Clean-Latent Encoder. A single noise-free forward pass over the full video clip yields a dense spatiotemporal feature grid, adapted end-to-end via LoRA to capture structured physical representations without generative sampling latency. To recover continuous metric trajectories from these features, we propose a Bidirectional Spatiotemporal Decoder paired with a Ray-Based Camera Solver. Across the entire video clip, we aggregate long-range temporal context via unconstrained bidirectional attention. By anchoring joint-specific queries to spatial feature maps and enforcing a global shape prior, the model reliably interpolates plausible hand poses during severe occlusions and prolonged out-of-sight intervals. Concurrently, to decouple hand placement from camera intrinsics and rapid ego-motion, we estimate a 3D viewing-ray field directly from the diffusion latents. By optimizing translation against predicted 2D anchors during visible frames and relying on temporal bearing cues during visual gaps, ACE-Ego-Hand can recover metric hand translation without a supplied calibration matrix, which supports intrinsics-free (K-free) inference at test time.

Our primary contributions are summarized as follows:

• We introduce ACE-Ego-Hand, an offline clip-level framework that repurposes a video diffusion model into a Deterministic Clean-Latent Encoder, recovering metric bimanual trajectories without any external detector.

• We present a Bidirectional Spatiotemporal Decoder that synergizes spatially grounded queries, clip-level shape priors, and bidirectional reasoning to reconstruct out-of-sight hands. A Ray-Based Camera Solver further decouples hand motion from camera dynamics to enable test-time intrinsics-free inference.

• Extensive evaluations on five challenging egocentric benchmarks demonstrate that ACE-Ego-Hand achieves superior accuracy and robustness, outperforming state-of-the-art baselines on 45 of 48 primary metric comparisons and remaining stable under severe occlusions and hand absences.

### 2. Related Work

Single-frame hand reconstruction. Most single-frame methods follow a “detect–crop–regress” pipeline, localizing the hand and then regressing MANO parameters from the cropped image [^5] [^21] [^23] [^24]. For egocentric settings, WildHands [^25] conditions on camera intrinsics, and EgoForce [^20] adds a forearm-guided camera-space solver with causal translation filtering at inference. Despite the improved camera modeling, these methods remain crop-dependent. Scale and depth are tied to the detection box, and they produce no reconstruction when the hand is severely occluded or outside the field of view.

Video-based hand motion recovery. Video-based methods address missing observations through temporal fusion or motion-prior completion. Early works apply bidirectional modeling to per-frame features [^13] or aggregate multiple frames to predict the center frame [^4] [^28], and OmniHands [^18] fuses nine consecutive two-hand crops to recover hands fully occluded within the image. A second line completes invisible hands with motion priors. GENMO [^17] performs bidirectional motion modeling conditioned on video features, and EgoH4 [^7] causally predicts out-of-sight hands from estimated body poses. However, their visual front ends still rely on frame-wise detection and cropping, and they typically complete motion over normalized pose sequences rather than the original video [^31].

Generative video models as representations. Recent work increasingly repurposes generative video models for downstream perception, either by reading their activations [^34] or by prompting the generation process itself [^29]. The two routes differ in cost. Reading activations needs a single forward pass, while prompting the generation process pays for a full sampling trajectory. Self-supervised video encoders [^1] [^32] offer a discriminative alternative, trained to predict masked content rather than to synthesize it. ViDiHand [^37] applies the generation-prompting route to hand reconstruction. The method fine-tunes Wan through VACE [^10] to synthesize geometric overlays and regresses MANO from intermediate denoising features, using 12–25 sampling steps and a decoder trained on cached frozen fea

---

tures. The backbone therefore receives no gradient from the reconstruction objective, and those features stay generic. We instead read the clean latent in a single forward pass inside the training loop.

### 3. ACE-Ego-Hand

ACE-Ego-Hand processes an egocentric video clip $V = \{I_t\}_{t=1}^T$ to predict frame-wise 3D bimanual MANO parameters in a single deterministic pass: global orientation $\hat{R}_t$, articulation $\hat{\theta}_t$, camera-frame translation $\hat{\tau}_t$, and a per-clip shape $\hat{\beta}$. The MANO layer $\mathcal{M}$ maps the pose and shape parameters to hand meshes and 21 joints, and the network additionally predicts per-frame existence and visibility flags. Instead of iteratively sampling from generative models, we extract motion representations directly through three unified modules (Figure 2): a Deterministic Clean-Latent Encoder that reads features from a LoRA-adapted pretrained video diffusion model, a Bidirectional Spatiotemporal Decoder for sequence-wide trajectory estimation, and a Ray-Based Camera Solver. We consider two configurations: standard ACE-Ego-Hand and the intrinsic-free ACE-Ego-Hand.

#### 3.1. From Generator to Encoder

Feedforward encoding. The generator $\Phi$ is a Diffusion Transformer (DiT) pretrained via rectified flow on noisy latents $x_{\sigma} = (1-\sigma)z + \sigma\epsilon$, where z is the clean video latent, $\epsilon$ is Gaussian noise, and $\sigma \in [0,1]$ is the noise level. This sequence-level pretraining is expected to equip $\Phi$ with spatiotemporal priors such as object permanence, 3D structural consistency, and occlusion reasoning. Sec. 4.3 probes this premise by swapping feature sources. We therefore use $\Phi$ strictly as an encoder. A frozen VAE encoder $\mathcal{E}$ compresses the clip into the clean latent $z = \mathcal{E}(V)$, and a single deterministic forward pass runs at zero noise ( $\sigma = 0$):

 $$ F=\Phi_{0:L^{\star}}(z;\sigma=0), $$

where $\Phi_{0:L^*}$ truncates execution of block $L^* = 15$ of the zero-indexed 30-block stack, running the first 16 blocks. Bypassing the remaining blocks and generation head cuts per-pass computation by approximately half. A tap-depth sweep in Appendix D shows that block 15 retains nearly all of the available accuracy. The resulting feature sequence $F = \{F_\ell\}_{l=1}^{\ell}$ contains $T' = 21$ latent frames for an input of $T = 81$ frames. This setup yields $4\times$ temporal and $16\times$ spatial compression. Each latent frame $F_\ell$ forms a spatial grid of $D = 3072$-channel feature cells. For a $672 \times 480$ input, this corresponds to a $42 \times 30$ grid of $16 \times 16$ pixel patches. The latent features F then feed both the Bidirectional Spatiotemporal Decoder (Sec. 3.2) and the Ray-Based Camera Solver (Sec. 3.3).

End-to-end adaptation. We keep the encoder trainable within the optimization loop, utilizing LoRA [^9] on attention and feed-forward projections alongside a trainable patch embedding layer, so that backpropagated 3D supervision reshapes F into geometry-aware features end to end. We compare alternative latent representations in Sec. 4.3. A matched comparison in Appendix D further shows that reading the clean latent at $\sigma = 0$ outperforms reading noised latents.

#### 3.2. Bidirectional Spatiotemporal Decoder

Spatially grounded queries. We propose a lightweight Bidirectional Spatiotemporal Decoder with spatially grounded queries to extract representations from the tapped feature grid, as illustrated in Figure 3. The decoder processes 48 queries per latent frame: 2 hand tokens assigned to fixed left and right slots, 42 joint tokens, and 4 register tokens that serve as learned scratch space without updating F. Structuring joint queries as spatially grounded tokens binds features directly to physical keypoints, facilitating precise 3D hand pose estimation.

Each latent frame is tokenized into patch tokens by integrating two additive positional encodings:

 $$ \begin{array}{r}{X_{\ell}=\mathrm{L N}\big(W_{F}F_{\ell}\big)+P^{\mathrm{s p}}+g\big(\Gamma(\boldsymbol{\hat{r}})\big),}\end{array} $$

where LN denotes Layer Normalization and $W_{F}$ projects the D = 3072 backbone channels to the 384-dimensional decoder space. The spatial PE $P^{sp}$ provides explicit coordinates for attention, and the ray PE injects viewing geometry: $\Gamma$ Fourier-encodes the direction of each cell's predicted viewing ray $\hat{r}$, produced by the Ray Head (Sec. 3.3), and a zero-initialized MLP g maps that encoding to the decoder dimension (implementation details in Appendix B).

Spatial readout heads. The Joint Head estimates 2D joint locations directly from this spatial grid. For each joint j, the cross-attention weights from its corresponding token form a spatial heatmap $A_{j}$. A soft-argmax operation then aggregates these attention probabilities to derive 2D joint anchors $\hat{p}_{j}$.

 $$ \hat{p}_{j}=\sum_{u}A_{j}(u)u,\qquad\sum_{u}A_{j}(u)=1, $$

where u iterates over normalized grid-cell center coordinates in $[0, 1]^2$. This differentiable formulation anchors each keypoint with sub-cell precision, and an MLP then predicts wrist-relative 3D joint positions in meters. In parallel, the Pose Head regresses the global orientation $\hat{R}$ and articulation $\hat{\theta}$ as Gram–Schmidt-orthogonalized 6D rotations, and the Camera Head predicts a log-depth $\hat{\zeta}$ with $\hat{t}_z = \exp(\hat{\zeta})$ guaranteeing strictly positive metric depth. Existence and visibility confidence scores complete the frame-level readout, avoiding the need for Hungarian matching [^3] [^14].

---

![Image](images/164c0706ab1b2044eafb4c91d2c478d68e7f643e5f25d633ae6effc0e2720973.jpg)

<div style="text-align: center;">Figure 2. Overview of ACE-Ego-Hand. Raw video is encoded into latents by the Wan VAE and processed by the Wan DiT ( $\sigma = 0$). Block-15 features (grayed blocks skipped) branch to the Ray Head and Bidirectional Spatiotemporal Decoder. Only the patch embedding, LoRA, Ray Head, and Decoder are trained (parameter accounting in Appendix B). Camera intrinsics supervise the Ray Head and the K-free camera fit during training. At test time they are discarded ( K-free) or used only as the bearing source in the translation solve (standard).</div>

![Image](images/57318a3166a820147019cd7c69c79327c70c601b091075a7d1a752f80a3841f4.jpg)

<div style="text-align: center;">Figure 3. Architecture of the Bidirectional Spatiotemporal Decoder. The predicted ray field  $\hat{r}$ enters as a ray PE added to the spatial PE. Four layers alternate Spatial Cross-Attention with Temporal Self-Attention. The readout heads emit MANO parameters and feed the mixed-PnP translation solve. Our two configurations differ architecturally only in the bearing source, derived from the predicted ray field  $\hat{r}$ (K-free, shown) or computed from camera intrinsics (standard).</div>

Clip-level shape prior. In egocentric videos, an individual hand maintains a constant physical shape and scale throughout a continuous recording. Estimating mesh parameters independently per frame, however, risks size flickering and shape drift under camera motion and local occlusions. To enforce this physical invariant, the Shape Head predicts hand shape parameters $\hat{\beta}$ once per hand per video clip by temporally pooling hand tokens across all T frames, yielding a single mesh scale for the entire sequence.

Unconstrained bidirectional reasoning. We process the spatially grounded queries with four alternating attention layers. Frame $l'$s queries first extract per-frame visual details through Spatial Cross-Attention over $X_{\ell}$, and queries from all frames then exchange motion information via Temporal Self-Attention. Rotary relative positional encodings [^30] on the temporal axis remove absolute sequence constraints, so a single forward pass generalizes to long recordings (Appendix D). Temporal attention runs at the downsampled latent frame rate $T'$, with outputs linearly interpolated back to the T video frames, reducing whole-clip attention to roughly $(T'/T)^2 \approx 1/15$ of the full-resolution cost. We apply no causal mask: each frame conditions on both past and future context across the entire clip, so the decoder reconstructs occluded or out-of-sight hands rather than extrapolating unidirectionally.

#### 3.3. Ray-Based Camera Solver

Intrinsics-free ray field prediction. Camera geometry is needed at two points: the ray positional encoding in Eq. (2) and the metric projection of the predicted hand. Readout heads that memorize the pixel-to-metric mapping of one camera fail to generalize across intrinsic, so we instead predict a continuous viewing-ray field, following ray-based camera representations [^11] [^43]. The Ray Head, a zero-initialized $1 \times 1$ convolution on F, predicts a per-cell direction normalized to a unit ray $\hat{r} = (\hat{r}_x, \hat{r}_y, \hat{r}_z)$ in the camera frame. Temporal pooling then averages the per-frame predictions into a single field, since intrinsic are constant within a clip.

During training, ground-truth camera calibration supervises this ray field using a cosine distance loss:

 $$ \mathcal{L}_{\mathrm{r a y}}=\frac{1}{|\Omega|}\sum_{u\in\Omega}\big(1-\langle\hat{r}(u),r_{K}(u)\rangle\big), $$

where $\langle\cdot,\cdot\rangle$ denotes the inner product, $\Omega$ is the latent token grid, and $r_{K}(u)$ is the unit ray unprojected from the cell center under the calibrated camera model. One head thus accommodates both pinhole and fisheye camera models without intrinsic as input at test time.

Mixed-PnP translation. To determine metric placement, we employ a mixed Perspective-n-Point (PnP) scheme, similar in spirit to Wang et al. [^37] but formulated for the

---

calibration-free setting. Rather than predicting full 3D translations, the Camera Head regresses only the optical depth $\hat{t}_z = \exp(\hat{\zeta})$, and we solve the in-plane translation $(t_x, t_y)$ directly against the predicted 2D joint anchors. Per frame and hand (indices suppressed), the MANO forward pass yields $J^{\mathrm{can}} = \mathcal{M}(\hat{r}, \theta, \hat{\beta})$, the 21 joints posed and camera-oriented but not yet placed, so the depth of joint j along the optical axis is $z_j = J_{z,j}^{\mathrm{can}} + \hat{t}_z$. The bearing vector $b_j = (b_j^x, b_j^y)$, the dimensionless pair $(x/z, y/z)$ of the ray toward joint j, is evaluated analytically from the predicted ray field $\hat{r}$. A closed-form per-axis regression fits an effective pinhole camera $(\hat{f}, \hat{c})$ to $\hat{r}$ in normalized pixel units, with no calibration input, and $b_j = (\hat{p}_j - \hat{c})/\hat{f}$. The fitted camera extrapolates to out-of-frame anchors and denoises the per-token field. Standard ACE-Ego-Hand instead computes the bearings from the provided intrinsics as $((u_j - c_x)/f_x, (v_j - c_y)/f_y)$ with $(u_j, v_j) = \hat{p}_j$ in pixels. Each configuration is trained end to end with its own bearing source, the K-free rows in Table 1 report a separately trained model, not a test-time solver switch.

Perspective projection is linear in the in-plane shift, so $t_{x}$ admits a closed-form weighted least-squares solution:

 $$ b_{j}^{x}=\frac{J_{x,j}^{\mathrm{c a n}}+t_{x}}{z_{j}}\quad\Rightarrow\quad\hat{t}_{x}=\frac{\sum_{j}m_{j}z_{j}^{-1}\left(b_{j}^{x}-J_{x,j}^{\mathrm{c a n}}/z_{j}\right)}{\sum_{j}m_{j}z_{j}^{-2}}, $$

where $m_j \in \{0,1\}$ selects joints that lie in front of the camera with anchors inside the frame. We solve for $\hat{t}_y$ symmetrically, yielding $\hat{\tau} = (\hat{t}_x, \hat{t}_y, \hat{t}_z)$. With too few valid joints or an excessive re-projection residual, the hand falls back to its inverse-projected wrist ray at depth $\hat{t}_z$ (thresholds in Appendix B).

#### 3.4. Training Recipe

We optimize all trainable components, namely the patch embedding, the LoRA modules, the Ray Head, and the Bidirectional Spatiotemporal Decoder, jointly under a unified objective:

 $$ \mathcal{L}=\mathcal{L}_{\mathrm{r o t}}+\mathcal{L}_{\mathrm{j o i n t}}+\mathcal{L}_{\mathrm{i m g}}+\mathcal{L}_{\mathrm{c a m}}+\mathcal{L}_{\mathrm{p r e s}}+\mathcal{L}_{\mathrm{t m p}}+\mathcal{L}_{\mathrm{r a y}}. $$

$\mathcal{L}_{\mathrm{rot}}$ supervises orientation, articulation, and shape. $\mathcal{L}_{\mathrm{joint}}$ constrains root-relative, camera-frame, and wrist 3D positions. $\mathcal{L}_{\mathrm{img}}$ penalizes 2D anchors and re-projected MANO keypoints under the training camera. $\mathcal{L}_{\mathrm{cam}}$ supervises camera-frame translation, with gradients flowing through the PnP solver. $\mathcal{L}_{\mathrm{pres}}$ trains existence and visibility scores, and $\mathcal{L}_{\mathrm{tmp}}$ penalizes 3D joint accelerations for temporal smoothness. The K-free configuration adds $\mathcal{L}_{\mathrm{fit}}$, the bearing error of its learned camera (Sec. 3.3) against the calibrated one, whose gradient reaches the ray field only through the four fitted parameters. Loss weights are in Appendix B.

We highlight three key supervision strategies. First, we fully supervise out-of-sight hands rather than masking them out, forcing the bidirectional attention to reconstruct invisible hands from temporal context. Second, a source lacking 3D MANO annotations, RHD in our mixture, supervises only 2D anchors, 3D joints, and presence heads, a routing that adds appearance diversity while shielding the MANO parameter heads. Third, camera intrinsics never enter the encoder or decoder: they serve as training targets for the $\mathcal{L}_{\mathrm{img}}$ re-projections, $\mathcal{L}_{\mathrm{ray}}$, and $\mathcal{L}_{\mathrm{fit}}$, and in the standard configuration, they additionally supply the bearings of the translation solve.

### 4. Experiments

#### 4.1. Experimental Setup

Training data and benchmarks. Both ACE-Ego-Hand configurations are trained separately for 20k steps, the standard one on 16 A100 GPUs and the K-free one on 8, using a weighted mixture of MANO-labeled egocentric video (ARCTIC, HOT3D, H2O, and OakInk2), rendered egocentric video from Re:InterHand [^22], and two image-only hand datasets (FreiHAND [^45] and RHD [^44]). HOI4D serves as the held-out test set.

We evaluate ACE-Ego-Hand across five complementary benchmarks: ARCTIC [^6] for severe bimanual occlusion, HOT3D [^2] for complex egocentric viewpoints, HOI4D [^19] for category-level 4D interaction generalization, H2O [^15] for bimanual hand-object pose estimation, and OakInk2 [^41] for long-horizon multi-object manipulation. The main text reports ARCTIC, HOT3D, and held-out HOI4D. H2O and OakInk2 appear in Appendix C, and the splits and preprocessing in Appendix A.

Baselines. We compare ACE-Ego-Hand against ten representative methods under a unified setting. Single-frame baselines are InterWild [^21], HaMeR [^23], Hamba [^5], WiLoR [^24], WildHands [^25], and EgoForce [^20]. Video-based baselines include OmniHands [^18], Dyn-HaMR [^40], HaWoR [^42], and ViDiHand [^37].

Evaluation protocol. All methods are evaluated across sequence-disjoint test splits using identical video segments. Baseline rows in Table 1 quote the published ViDiHand evaluation under the identical protocol, and EgoForce plus our rows come from the same evaluator. Note that EgoForce $^{\dagger}$ post-filters its camera-space translation with a causal Kalman filter, and that HaWoR is scored on its camera-space front end with the SLAM and infilling stages disabled (Appendix A). HOI4D is held out from our training and evaluated zero-shot, while the H2O and OakInk2 comparisons in Appendix C place our in-domain model against baselines that never saw those datasets.

---

<div style="text-align: center;">Table 1. Main comparison on ARCTIC, HOT3D, and held-out HOI4D. Best result per column in bold. ↑/↓ give the direction of improvement. MPJPE $^{+005}$ averages over all hand-frames, in view and out of sight alike, so it needs out-of-sight ground truth, which HOI4D lacks (-).  $\dagger$ zero-shot evaluation,  $\ddagger$ causal Kalman filtering.</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td rowspan="2" colspan="2">Method</td><td colspan="3">Detection</td><td colspan="3">3D Pose</td><td colspan="3">Orient. &amp; Position</td><td style='text-align: center; word-wrap: break-word;'>Temporal</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>FAcc $\uparrow$</td><td style='text-align: center; word-wrap: break-word;'>Recall $\uparrow$</td><td style='text-align: center; word-wrap: break-word;'>F1 $\uparrow$</td><td style='text-align: center; word-wrap: break-word;'>MPJPE-p $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>PA-p $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>MPJPE+OOS $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>EPE2D-p $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>GO-p $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>CT-p $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>Jitter $\downarrow$</td></tr><tr><td rowspan="12">ARCTIC</td><td style='text-align: center; word-wrap: break-word;'>InterWild</td><td style='text-align: center; word-wrap: break-word;'>0.878</td><td style='text-align: center; word-wrap: break-word;'>0.943</td><td style='text-align: center; word-wrap: break-word;'>0.959</td><td style='text-align: center; word-wrap: break-word;'>30.817</td><td style='text-align: center; word-wrap: break-word;'>15.952</td><td style='text-align: center; word-wrap: break-word;'>39.435</td><td style='text-align: center; word-wrap: break-word;'>53.888</td><td style='text-align: center; word-wrap: break-word;'>25.386</td><td style='text-align: center; word-wrap: break-word;'>0.097</td><td style='text-align: center; word-wrap: break-word;'>46.577</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>HaMeR</td><td style='text-align: center; word-wrap: break-word;'>0.875</td><td style='text-align: center; word-wrap: break-word;'>0.943</td><td style='text-align: center; word-wrap: break-word;'>0.957</td><td style='text-align: center; word-wrap: break-word;'>29.197</td><td style='text-align: center; word-wrap: break-word;'>14.596</td><td style='text-align: center; word-wrap: break-word;'>38.183</td><td style='text-align: center; word-wrap: break-word;'>65.289</td><td style='text-align: center; word-wrap: break-word;'>24.907</td><td style='text-align: center; word-wrap: break-word;'>0.095</td><td style='text-align: center; word-wrap: break-word;'>18.279</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Hamba</td><td style='text-align: center; word-wrap: break-word;'>0.833</td><td style='text-align: center; word-wrap: break-word;'>0.912</td><td style='text-align: center; word-wrap: break-word;'>0.941</td><td style='text-align: center; word-wrap: break-word;'>31.233</td><td style='text-align: center; word-wrap: break-word;'>17.168</td><td style='text-align: center; word-wrap: break-word;'>40.039</td><td style='text-align: center; word-wrap: break-word;'>87.047</td><td style='text-align: center; word-wrap: break-word;'>27.822</td><td style='text-align: center; word-wrap: break-word;'>0.110</td><td style='text-align: center; word-wrap: break-word;'>15.357</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>WildHands</td><td style='text-align: center; word-wrap: break-word;'>0.879</td><td style='text-align: center; word-wrap: break-word;'>0.946</td><td style='text-align: center; word-wrap: break-word;'>0.960</td><td style='text-align: center; word-wrap: break-word;'>25.704</td><td style='text-align: center; word-wrap: break-word;'>13.941</td><td style='text-align: center; word-wrap: break-word;'>33.915</td><td style='text-align: center; word-wrap: break-word;'>50.517</td><td style='text-align: center; word-wrap: break-word;'>22.320</td><td style='text-align: center; word-wrap: break-word;'>0.058</td><td style='text-align: center; word-wrap: break-word;'>12.972</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>WiLoR</td><td style='text-align: center; word-wrap: break-word;'>0.919</td><td style='text-align: center; word-wrap: break-word;'>0.951</td><td style='text-align: center; word-wrap: break-word;'>0.974</td><td style='text-align: center; word-wrap: break-word;'>22.012</td><td style='text-align: center; word-wrap: break-word;'>11.873</td><td style='text-align: center; word-wrap: break-word;'>31.502</td><td style='text-align: center; word-wrap: break-word;'>71.527</td><td style='text-align: center; word-wrap: break-word;'>17.358</td><td style='text-align: center; word-wrap: break-word;'>0.075</td><td style='text-align: center; word-wrap: break-word;'>24.091</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>EgoForce $\ddagger$</td><td style='text-align: center; word-wrap: break-word;'>0.882</td><td style='text-align: center; word-wrap: break-word;'>0.930</td><td style='text-align: center; word-wrap: break-word;'>0.963</td><td style='text-align: center; word-wrap: break-word;'>22.388</td><td style='text-align: center; word-wrap: break-word;'>14.322</td><td style='text-align: center; word-wrap: break-word;'>32.193</td><td style='text-align: center; word-wrap: break-word;'>61.761</td><td style='text-align: center; word-wrap: break-word;'>21.073</td><td style='text-align: center; word-wrap: break-word;'>0.069</td><td style='text-align: center; word-wrap: break-word;'>24.158</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>OmniHands</td><td style='text-align: center; word-wrap: break-word;'>0.866</td><td style='text-align: center; word-wrap: break-word;'>0.949</td><td style='text-align: center; word-wrap: break-word;'>0.954</td><td style='text-align: center; word-wrap: break-word;'>29.674</td><td style='text-align: center; word-wrap: break-word;'>14.203</td><td style='text-align: center; word-wrap: break-word;'>38.191</td><td style='text-align: center; word-wrap: break-word;'>51.505</td><td style='text-align: center; word-wrap: break-word;'>24.580</td><td style='text-align: center; word-wrap: break-word;'>0.087</td><td style='text-align: center; word-wrap: break-word;'>45.312</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Dyn-HaMR</td><td style='text-align: center; word-wrap: break-word;'>0.842</td><td style='text-align: center; word-wrap: break-word;'>0.918</td><td style='text-align: center; word-wrap: break-word;'>0.951</td><td style='text-align: center; word-wrap: break-word;'>27.904</td><td style='text-align: center; word-wrap: break-word;'>17.017</td><td style='text-align: center; word-wrap: break-word;'>37.445</td><td style='text-align: center; word-wrap: break-word;'>85.723</td><td style='text-align: center; word-wrap: break-word;'>25.951</td><td style='text-align: center; word-wrap: break-word;'>0.121</td><td style='text-align: center; word-wrap: break-word;'>12.840</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>HaWoR</td><td style='text-align: center; word-wrap: break-word;'>0.700</td><td style='text-align: center; word-wrap: break-word;'>0.817</td><td style='text-align: center; word-wrap: break-word;'>0.895</td><td style='text-align: center; word-wrap: break-word;'>45.357</td><td style='text-align: center; word-wrap: break-word;'>26.375</td><td style='text-align: center; word-wrap: break-word;'>53.443</td><td style='text-align: center; word-wrap: break-word;'>158.062</td><td style='text-align: center; word-wrap: break-word;'>43.325</td><td style='text-align: center; word-wrap: break-word;'>0.149</td><td style='text-align: center; word-wrap: break-word;'>19.789</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>ViDiHand</td><td style='text-align: center; word-wrap: break-word;'>0.997</td><td style='text-align: center; word-wrap: break-word;'>0.999</td><td style='text-align: center; word-wrap: break-word;'>0.999</td><td style='text-align: center; word-wrap: break-word;'>21.668</td><td style='text-align: center; word-wrap: break-word;'>9.821</td><td style='text-align: center; word-wrap: break-word;'>31.045</td><td style='text-align: center; word-wrap: break-word;'>12.407</td><td style='text-align: center; word-wrap: break-word;'>14.642</td><td style='text-align: center; word-wrap: break-word;'>0.047</td><td style='text-align: center; word-wrap: break-word;'>3.183</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>ACE-Ego-Hand</td><td style='text-align: center; word-wrap: break-word;'>1.000</td><td style='text-align: center; word-wrap: break-word;'>1.000</td><td style='text-align: center; word-wrap: break-word;'>1.000</td><td style='text-align: center; word-wrap: break-word;'>15.256</td><td style='text-align: center; word-wrap: break-word;'>7.474</td><td style='text-align: center; word-wrap: break-word;'>16.783</td><td style='text-align: center; word-wrap: break-word;'>9.180</td><td style='text-align: center; word-wrap: break-word;'>11.807</td><td style='text-align: center; word-wrap: break-word;'>0.021</td><td style='text-align: center; word-wrap: break-word;'>2.700</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>ACE-Ego-Hand (K-free)</td><td style='text-align: center; word-wrap: break-word;'>0.999</td><td style='text-align: center; word-wrap: break-word;'>1.000</td><td style='text-align: center; word-wrap: break-word;'>1.000</td><td style='text-align: center; word-wrap: break-word;'>16.615</td><td style='text-align: center; word-wrap: break-word;'>8.756</td><td style='text-align: center; word-wrap: break-word;'>18.061</td><td style='text-align: center; word-wrap: break-word;'>11.014</td><td style='text-align: center; word-wrap: break-word;'>12.208</td><td style='text-align: center; word-wrap: break-word;'>0.028</td><td style='text-align: center; word-wrap: break-word;'>2.770</td></tr><tr><td rowspan="12">HOT3D</td><td style='text-align: center; word-wrap: break-word;'>InterWild</td><td style='text-align: center; word-wrap: break-word;'>0.669</td><td style='text-align: center; word-wrap: break-word;'>0.881</td><td style='text-align: center; word-wrap: break-word;'>0.868</td><td style='text-align: center; word-wrap: break-word;'>77.168</td><td style='text-align: center; word-wrap: break-word;'>24.811</td><td style='text-align: center; word-wrap: break-word;'>89.218</td><td style='text-align: center; word-wrap: break-word;'>71.482</td><td style='text-align: center; word-wrap: break-word;'>58.501</td><td style='text-align: center; word-wrap: break-word;'>0.213</td><td style='text-align: center; word-wrap: break-word;'>101.164</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>HaMeR</td><td style='text-align: center; word-wrap: break-word;'>0.692</td><td style='text-align: center; word-wrap: break-word;'>0.904</td><td style='text-align: center; word-wrap: break-word;'>0.883</td><td style='text-align: center; word-wrap: break-word;'>68.314</td><td style='text-align: center; word-wrap: break-word;'>21.455</td><td style='text-align: center; word-wrap: break-word;'>80.264</td><td style='text-align: center; word-wrap: break-word;'>59.077</td><td style='text-align: center; word-wrap: break-word;'>49.636</td><td style='text-align: center; word-wrap: break-word;'>0.102</td><td style='text-align: center; word-wrap: break-word;'>23.632</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Hamba</td><td style='text-align: center; word-wrap: break-word;'>0.632</td><td style='text-align: center; word-wrap: break-word;'>0.828</td><td style='text-align: center; word-wrap: break-word;'>0.853</td><td style='text-align: center; word-wrap: break-word;'>71.732</td><td style='text-align: center; word-wrap: break-word;'>29.620</td><td style='text-align: center; word-wrap: break-word;'>83.438</td><td style='text-align: center; word-wrap: break-word;'>107.625</td><td style='text-align: center; word-wrap: break-word;'>56.525</td><td style='text-align: center; word-wrap: break-word;'>0.128</td><td style='text-align: center; word-wrap: break-word;'>18.507</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>WildHands</td><td style='text-align: center; word-wrap: break-word;'>0.655</td><td style='text-align: center; word-wrap: break-word;'>0.863</td><td style='text-align: center; word-wrap: break-word;'>0.844</td><td style='text-align: center; word-wrap: break-word;'>52.791</td><td style='text-align: center; word-wrap: break-word;'>28.946</td><td style='text-align: center; word-wrap: break-word;'>60.491</td><td style='text-align: center; word-wrap: break-word;'>111.438</td><td style='text-align: center; word-wrap: break-word;'>53.933</td><td style='text-align: center; word-wrap: break-word;'>0.157</td><td style='text-align: center; word-wrap: break-word;'>22.885</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>WiLoR</td><td style='text-align: center; word-wrap: break-word;'>0.827</td><td style='text-align: center; word-wrap: break-word;'>0.897</td><td style='text-align: center; word-wrap: break-word;'>0.937</td><td style='text-align: center; word-wrap: break-word;'>30.966</td><td style='text-align: center; word-wrap: break-word;'>19.980</td><td style='text-align: center; word-wrap: break-word;'>52.014</td><td style='text-align: center; word-wrap: break-word;'>72.978</td><td style='text-align: center; word-wrap: break-word;'>25.746</td><td style='text-align: center; word-wrap: break-word;'>0.098</td><td style='text-align: center; word-wrap: break-word;'>17.976</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>EgoForce $\ddagger$</td><td style='text-align: center; word-wrap: break-word;'>0.769</td><td style='text-align: center; word-wrap: break-word;'>0.856</td><td style='text-align: center; word-wrap: break-word;'>0.916</td><td style='text-align: center; word-wrap: break-word;'>43.960</td><td style='text-align: center; word-wrap: break-word;'>25.709</td><td style='text-align: center; word-wrap: break-word;'>63.085</td><td style='text-align: center; word-wrap: break-word;'>83.521</td><td style='text-align: center; word-wrap: break-word;'>37.144</td><td style='text-align: center; word-wrap: break-word;'>0.130</td><td style='text-align: center; word-wrap: break-word;'>38.342</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>OmniHands</td><td style='text-align: center; word-wrap: break-word;'>0.649</td><td style='text-align: center; word-wrap: break-word;'>0.895</td><td style='text-align: center; word-wrap: break-word;'>0.868</td><td style='text-align: center; word-wrap: break-word;'>63.281</td><td style='text-align: center; word-wrap: break-word;'>22.682</td><td style='text-align: center; word-wrap: break-word;'>73.503</td><td style='text-align: center; word-wrap: break-word;'>68.437</td><td style='text-align: center; word-wrap: break-word;'>49.120</td><td style='text-align: center; word-wrap: break-word;'>0.133</td><td style='text-align: center; word-wrap: break-word;'>69.510</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Dyn-HaMR</td><td style='text-align: center; word-wrap: break-word;'>0.614</td><td style='text-align: center; word-wrap: break-word;'>0.811</td><td style='text-align: center; word-wrap: break-word;'>0.802</td><td style='text-align: center; word-wrap: break-word;'>74.214</td><td style='text-align: center; word-wrap: break-word;'>38.201</td><td style='text-align: center; word-wrap: break-word;'>80.952</td><td style='text-align: center; word-wrap: break-word;'>171.617</td><td style='text-align: center; word-wrap: break-word;'>43.851</td><td style='text-align: center; word-wrap: break-word;'>0.571</td><td style='text-align: center; word-wrap: break-word;'>44.942</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>HaWoR</td><td style='text-align: center; word-wrap: break-word;'>0.348</td><td style='text-align: center; word-wrap: break-word;'>0.499</td><td style='text-align: center; word-wrap: break-word;'>0.654</td><td style='text-align: center; word-wrap: break-word;'>71.396</td><td style='text-align: center; word-wrap: break-word;'>66.031</td><td style='text-align: center; word-wrap: break-word;'>84.733</td><td style='text-align: center; word-wrap: break-word;'>327.294</td><td style='text-align: center; word-wrap: break-word;'>79.350</td><td style='text-align: center; word-wrap: break-word;'>0.262</td><td style='text-align: center; word-wrap: break-word;'>23.872</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>ViDiHand</td><td style='text-align: center; word-wrap: break-word;'>0.948</td><td style='text-align: center; word-wrap: break-word;'>0.974</td><td style='text-align: center; word-wrap: break-word;'>0.983</td><td style='text-align: center; word-wrap: break-word;'>21.514</td><td style='text-align: center; word-wrap: break-word;'>11.383</td><td style='text-align: center; word-wrap: break-word;'>44.440</td><td style='text-align: center; word-wrap: break-word;'>14.953</td><td style='text-align: center; word-wrap: break-word;'>15.829</td><td style='text-align: center; word-wrap: break-word;'>0.040</td><td style='text-align: center; word-wrap: break-word;'>3.741</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>ACE-Ego-Hand</td><td style='text-align: center; word-wrap: break-word;'>0.986</td><td style='text-align: center; word-wrap: break-word;'>0.998</td><td style='text-align: center; word-wrap: break-word;'>0.996</td><td style='text-align: center; word-wrap: break-word;'>12.888</td><td style='text-align: center; word-wrap: break-word;'>6.436</td><td style='text-align: center; word-wrap: break-word;'>17.273</td><td style='text-align: center; word-wrap: break-word;'>6.418</td><td style='text-align: center; word-wrap: break-word;'>7.924</td><td style='text-align: center; word-wrap: break-word;'>0.025</td><td style='text-align: center; word-wrap: break-word;'>3.159</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>ACE-Ego-Hand (K-free)</td><td style='text-align: center; word-wrap: break-word;'>0.984</td><td style='text-align: center; word-wrap: break-word;'>0.999</td><td style='text-align: center; word-wrap: break-word;'>0.995</td><td style='text-align: center; word-wrap: break-word;'>13.535</td><td style='text-align: center; word-wrap: break-word;'>6.768</td><td style='text-align: center; word-wrap: break-word;'>18.703</td><td style='text-align: center; word-wrap: break-word;'>6.067</td><td style='text-align: center; word-wrap: break-word;'>8.362</td><td style='text-align: center; word-wrap: break-word;'>0.026</td><td style='text-align: center; word-wrap: break-word;'>3.426</td></tr><tr><td rowspan="12">HOI4D $\dagger$</td><td style='text-align: center; word-wrap: break-word;'>InterWild</td><td style='text-align: center; word-wrap: break-word;'>0.731</td><td style='text-align: center; word-wrap: break-word;'>0.922</td><td style='text-align: center; word-wrap: break-word;'>0.864</td><td style='text-align: center; word-wrap: break-word;'>53.072</td><td style='text-align: center; word-wrap: break-word;'>22.909</td><td style='text-align: center; word-wrap: break-word;'>-</td><td style='text-align: center; word-wrap: break-word;'>80.549</td><td style='text-align: center; word-wrap: break-word;'>41.743</td><td style='text-align: center; word-wrap: break-word;'>0.228</td><td style='text-align: center; word-wrap: break-word;'>98.866</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>HaMeR</td><td style='text-align: center; word-wrap: break-word;'>0.731</td><td style='text-align: center; word-wrap: break-word;'>0.923</td><td style='text-align: center; word-wrap: break-word;'>0.864</td><td style='text-align: center; word-wrap: break-word;'>44.481</td><td style='text-align: center; word-wrap: break-word;'>21.580</td><td style='text-align: center; word-wrap: break-word;'>-</td><td style='text-align: center; word-wrap: break-word;'>79.494</td><td style='text-align: center; word-wrap: break-word;'>33.557</td><td style='text-align: center; word-wrap: break-word;'>0.187</td><td style='text-align: center; word-wrap: break-word;'>20.068</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Hamba</td><td style='text-align: center; word-wrap: break-word;'>0.710</td><td style='text-align: center; word-wrap: break-word;'>0.885</td><td style='text-align: center; word-wrap: break-word;'>0.849</td><td style='text-align: center; word-wrap: break-word;'>47.161</td><td style='text-align: center; word-wrap: break-word;'>25.924</td><td style='text-align: center; word-wrap: break-word;'>-</td><td style='text-align: center; word-wrap: break-word;'>115.793</td><td style='text-align: center; word-wrap: break-word;'>37.390</td><td style='text-align: center; word-wrap: break-word;'>0.204</td><td style='text-align: center; word-wrap: break-word;'>21.556</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>WildHands</td><td style='text-align: center; word-wrap: break-word;'>0.730</td><td style='text-align: center; word-wrap: break-word;'>0.924</td><td style='text-align: center; word-wrap: break-word;'>0.864</td><td style='text-align: center; word-wrap: break-word;'>45.623</td><td style='text-align: center; word-wrap: break-word;'>23.601</td><td style='text-align: center; word-wrap: break-word;'>-</td><td style='text-align: center; word-wrap: break-word;'>82.246</td><td style='text-align: center; word-wrap: break-word;'>45.654</td><td style='text-align: center; word-wrap: break-word;'>0.159</td><td style='text-align: center; word-wrap: break-word;'>18.615</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>WiLoR</td><td style='text-align: center; word-wrap: break-word;'>0.962</td><td style='text-align: center; word-wrap: break-word;'>0.966</td><td style='text-align: center; word-wrap: break-word;'>0.972</td><td style='text-align: center; word-wrap: break-word;'>33.710</td><td style='text-align: center; word-wrap: break-word;'>14.903</td><td style='text-align: center; word-wrap: break-word;'>-</td><td style='text-align: center; word-wrap: break-word;'>41.579</td><td style='text-align: center; word-wrap: break-word;'>25.527</td><td style='text-align: center; word-wrap: break-word;'>0.115</td><td style='text-align: center; word-wrap: break-word;'>17.449</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>EgoForce $\ddagger$</td><td style='text-align: center; word-wrap: break-word;'>0.917</td><td style='text-align: center; word-wrap: break-word;'>0.941</td><td style='text-align: center; word-wrap: break-word;'>0.949</td><td style='text-align: center; word-wrap: break-word;'>44.746</td><td style='text-align: center; word-wrap: break-word;'>19.160</td><td style='text-align: center; word-wrap: break-word;'>-</td><td style='text-align: center; word-wrap: break-word;'>75.821</td><td style='text-align: center; word-wrap: break-word;'>38.861</td><td style='text-align: center; word-wrap: break-word;'>0.126</td><td style='text-align: center; word-wrap: break-word;'>83.452</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>OmniHands</td><td style='text-align: center; word-wrap: break-word;'>0.655</td><td style='text-align: center; word-wrap: break-word;'>0.937</td><td style='text-align: center; word-wrap: break-word;'>0.834</td><td style='text-align: center; word-wrap: break-word;'>44.255</td><td style='text-align: center; word-wrap: break-word;'>18.689</td><td style='text-align: center; word-wrap: break-word;'>-</td><td style='text-align: center; word-wrap: break-word;'>70.662</td><td style='text-align: center; word-wrap: break-word;'>34.392</td><td style='text-align: center; word-wrap: break-word;'>0.108</td><td style='text-align: center; word-wrap: break-word;'>24.212</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Dyn-HaMR</td><td style='text-align: center; word-wrap: break-word;'>0.750</td><td style='text-align: center; word-wrap: break-word;'>0.863</td><td style='text-align: center; word-wrap: break-word;'>0.845</td><td style='text-align: center; word-wrap: break-word;'>45.097</td><td style='text-align: center; word-wrap: break-word;'>29.259</td><td style='text-align: center; word-wrap: break-word;'>-</td><td style='text-align: center; word-wrap: break-word;'>144.643</td><td style='text-align: center; word-wrap: break-word;'>40.176</td><td style='text-align: center; word-wrap: break-word;'>0.258</td><td style='text-align: center; word-wrap: break-word;'>17.947</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>HaWoR</td><td style='text-align: center; word-wrap: break-word;'>0.869</td><td style='text-align: center; word-wrap: break-word;'>0.864</td><td style='text-align: center; word-wrap: break-word;'>0.919</td><td style='text-align: center; word-wrap: break-word;'>47.329</td><td style='text-align: center; word-wrap: break-word;'>28.851</td><td style='text-align: center; word-wrap: break-word;'>-</td><td style='text-align: center; word-wrap: break-word;'>135.748</td><td style='text-align: center; word-wrap: break-word;'>43.091</td><td style='text-align: center; word-wrap: break-word;'>0.139</td><td style='text-align: center; word-wrap: break-word;'>28.376</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>ViDiHand</td><td style='text-align: center; word-wrap: break-word;'>0.984</td><td style='text-align: center; word-wrap: break-word;'>0.991</td><td style='text-align: center; word-wrap: break-word;'>0.990</td><td style='text-align: center; word-wrap: break-word;'>30.090</td><td style='text-align: center; word-wrap: break-word;'>13.960</td><td style='text-align: center; word-wrap: break-word;'>-</td><td style='text-align: center; word-wrap: break-word;'>24.460</td><td style='text-align: center; word-wrap: break-word;'>23.420</td><td style='text-align: center; word-wrap: break-word;'>0.117</td><td style='text-align: center; word-wrap: break-word;'>4.010</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>ACE-Ego-Hand</td><td style='text-align: center; word-wrap: break-word;'>0.958</td><td style='text-align: center; word-wrap: break-word;'>0.996</td><td style='text-align: center; word-wrap: break-word;'>0.974</td><td style='text-align: center; word-wrap: break-word;'>23.031</td><td style='text-align: center; word-wrap: break-word;'>11.664</td><td style='text-align: center; word-wrap: break-word;'>-</td><td style='text-align: center; word-wrap: break-word;'>14.987</td><td style='text-align: center; word-wrap: break-word;'>18.426</td><td style='text-align: center; word-wrap: break-word;'>0.057</td><td style='text-align: center; word-wrap: break-word;'>2.393</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>ACE-Ego-Hand (K-free)</td><td style='text-align: center; word-wrap: break-word;'>0.975</td><td style='text-align: center; word-wrap: break-word;'>0.999</td><td style='text-align: center; word-wrap: break-word;'>0.985</td><td style='text-align: center; word-wrap: break-word;'>22.154</td><td style='text-align: center; word-wrap: break-word;'>11.213</td><td style='text-align: center; word-wrap: break-word;'>-</td><td style='text-align: center; word-wrap: break-word;'>18.004</td><td style='text-align: center; word-wrap: break-word;'>17.242</td><td style='text-align: center; word-wrap: break-word;'>0.055</td><td style='text-align: center; word-wrap: break-word;'>2.317</td></tr></table>

Metrics. Following Wang et al. [^37], a hand is defined as on-screen if any of its 21 ground-truth 3D joints projects within the image boundary at a depth above  $z_{\text{min}} = 1 \, \text{cm}$. Frame Accuracy (FAcc) measures the fraction of frames containing neither missed nor spurious hands, while Recall and F1 are calculated per hand. A hand counts as detected when its existence score exceeds 0.5 and it matches a same-side on-screen ground-truth hand by projected 2D overlap, with the on-screen gate applied to both sides. Because matching is scored under the predicted translation, detection reflects placement as well as presence. MPJPE-p (mm) is the mean joint position error after wrist alignment, and PA-p (mm) the mean joint error following Procrustes alignment.  $EPE_{2D-p}$ (px) is the mean 2D end-point error over in-frame joints, GO-p ( $^{\circ}$) the geodesic global rotation error, and CT-p (m) the camera-frame translation error  $\|\hat{\tau} - \tau\|_2$. Jitter  $\frac{\text{mm}}{\text{frame}^2}$ is the mean second temporal difference of the 3D joint predictions.

The -p suffix denotes a penalization protocol for false negatives: undetected on-screen hands are assigned a canonical MANO mesh placed at the camera origin, preventing methods from artificially boosting accuracy by dropping challenging detections. Additionally, MPJPE $^{OOS}$ (mm) evaluates all hands across the sequence including out-of-sight targets, weighted by frame counts, while MPJPE $^{OOS}$ (mm) restricts the same wrist-aligned error to the hand-frames that fail the on-screen gate. The two average over different populations and are not interchangeable. For our K-free variant, EPE $_{2D}$-p is evaluated by reprojecting predicted 3D joints using held-out ground-truth intrinsic, directly diagnosing the accuracy of implicit camera calibration, see formal definitions in Appendix A.

---

#### 4.2. Main Results

In-view accuracy. ACE-Ego-Hand achieves state-of-the-art overall tracking performance across ARCTIC, HOT3D, and HOI4D, leading on 27 of 29 evaluation metrics. Detection of the standard configuration is near saturation (FAcc, Recall, and F1 above 0.95, and 1.00 on ARCTIC), while seven baselines fall below 0.70 FAcc on wide-angle HOT3D sequences. Compared with ViDiHand, ACE-Ego-Hand reduces MPJPE-p by 30% (21.67 to 15.26 mm), 40% (21.51 to 12.89 mm), and 23% (30.09 to 23.03 mm) on ARCTIC, HOT3D, and zero-shot HOI4D, respectively, and leads on articulation, 2D localization, orientation, and translation throughout. The advantage persists on occlusion-heavy ARCTIC: ACE-Ego-Hand reduces PA-p by 24% over ViDiHand and nearly halves the MPJPE-p of OmniHands, a method designed for in-image occlusion recovery. ACE-Ego-Hand also yields the lowest Jitter (2.39–3.16 mm/frame $^2$), reflecting clip-level decoding and the explicit temporal-smoothness objective.

Intrinsics-free operation. The separately trained K-free configuration drops test-time calibration entirely while keeping detection at parity with the standard configuration on all five benchmarks, with 2D anchors on par with the calibrated bearings on wide-angle HOT3D (F1 0.995 vs. 0.996, EPE $_{2D}$-p 6.07 vs. 6.42px). Its residual cost is a small wrist-aligned pose gap on the narrower-field benchmarks (+1.36 mm MPJPE-p on ARCTIC, +1.09 on H2O), while zero-shot HOI4D improves (22.15 vs. 23.03 mm; the higher EPE $_{2D}$-p reflects recovered hard detections entering the matched set).

Out-of-sight recovery. Once out-of-sight hands enter the evaluation, ACE-Ego-Hand achieves 16.8 and 17.3 mm MPJPE+OOS on ARCTIC and HOT3D, reducing the 31.05 and 44.44 mm of ViDiHand by 45.9% and 61.1%, respectively. Restricted to the out-of-sight stratum alone, ACE-Ego-Hand attains 35.2 and 38.6 mm MPJPE+OOS against 149.1 and 151.7 mm for ViDiHand and 135.4 and 126.4 mm for WildHands, the strongest baseline on those frames. The K-free variant follows at 18.1 and 18.7 mm MPJPE+OOS, within 1.5 mm of the standard configuration. The encoder ablation traces this robustness to the generative VDM features (Table 2). Note that both metrics are wrist-aligned, reflecting articulation and global orientation through the gap. Absolute out-of-sight placement instead relies on depth and bearing regressed from temporal context through the solver fallback, and remains less constrained (Figure 4).

Efficiency. One deterministic pass makes ACE-Ego-Hand fast as well as accurate: 63.1 fps on one A100 (63.3 K-free) versus 1.91 fps for ViDiHand in its accuracy configuration, a $33\times$ gap (protocol in Appendix C).

<div style="text-align: center;">Table 2. Feature-source ablation on ARCTIC. MPJPE-p, PA-p and MPJPE $^{OOS}$ in mm, Jitter in mm/frame $^2$. MPJPE $^{OOS}$ is restricted to the hand-frames that fail the on-screen gate.</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'>Feature source</td><td style='text-align: center; word-wrap: break-word;'>MPJPE-p $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>PA-p $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>Jitter $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>MPJPE $^{OOS}$ $\downarrow$</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>ACE-Ego-Hand</td><td style='text-align: center; word-wrap: break-word;'>16.95</td><td style='text-align: center; word-wrap: break-word;'>8.16</td><td style='text-align: center; word-wrap: break-word;'>2.69</td><td style='text-align: center; word-wrap: break-word;'>41.5</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>w/o LoRA</td><td style='text-align: center; word-wrap: break-word;'>23.10</td><td style='text-align: center; word-wrap: break-word;'>10.83</td><td style='text-align: center; word-wrap: break-word;'>3.35</td><td style='text-align: center; word-wrap: break-word;'>50.1</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>V-JEPA 2 [^1]</td><td style='text-align: center; word-wrap: break-word;'>17.81</td><td style='text-align: center; word-wrap: break-word;'>8.74</td><td style='text-align: center; word-wrap: break-word;'>4.56</td><td style='text-align: center; word-wrap: break-word;'>66.1</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>VideoMAE [^32]</td><td style='text-align: center; word-wrap: break-word;'>22.53</td><td style='text-align: center; word-wrap: break-word;'>10.42</td><td style='text-align: center; word-wrap: break-word;'>5.60</td><td style='text-align: center; word-wrap: break-word;'>77.6</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>VAE latent</td><td style='text-align: center; word-wrap: break-word;'>32.50</td><td style='text-align: center; word-wrap: break-word;'>12.89</td><td style='text-align: center; word-wrap: break-word;'>3.35</td><td style='text-align: center; word-wrap: break-word;'>79.3</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>raw RGB</td><td style='text-align: center; word-wrap: break-word;'>46.50</td><td style='text-align: center; word-wrap: break-word;'>16.93</td><td style='text-align: center; word-wrap: break-word;'>4.37</td><td style='text-align: center; word-wrap: break-word;'>98.3</td></tr></table>

<div style="text-align: center;">Table 3. Camera and decoder ablations on ARCTIC. MPJPE-p and PA-p in mm, EPE_{2D}-p in px, CT-p in m, Jitter in mm/frame^{2}.</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'>Variant</td><td style='text-align: center; word-wrap: break-word;'>MPJPE-p $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>PA-p $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>EPE $_{2D}$-p $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>CT-p $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>Jitter $\downarrow$</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>ACE-Ego-Hand</td><td style='text-align: center; word-wrap: break-word;'>15.256</td><td style='text-align: center; word-wrap: break-word;'>7.474</td><td style='text-align: center; word-wrap: break-word;'>9.180</td><td style='text-align: center; word-wrap: break-word;'>0.021</td><td style='text-align: center; word-wrap: break-word;'>2.700</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Removing test-time intrinsics</td><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'></td></tr><tr><td style='text-align: center; word-wrap: break-word;'>ACE-Ego-Hand (K-free)</td><td style='text-align: center; word-wrap: break-word;'>15.264</td><td style='text-align: center; word-wrap: break-word;'>7.770</td><td style='text-align: center; word-wrap: break-word;'>13.168</td><td style='text-align: center; word-wrap: break-word;'>0.020</td><td style='text-align: center; word-wrap: break-word;'>2.704</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>w/o mixed-PnP (inverse proj.)</td><td style='text-align: center; word-wrap: break-word;'>16.134</td><td style='text-align: center; word-wrap: break-word;'>8.002</td><td style='text-align: center; word-wrap: break-word;'>14.425</td><td style='text-align: center; word-wrap: break-word;'>0.030</td><td style='text-align: center; word-wrap: break-word;'>2.874</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>w/o PnP solve (direct reg.)</td><td style='text-align: center; word-wrap: break-word;'>15.546</td><td style='text-align: center; word-wrap: break-word;'>7.955</td><td style='text-align: center; word-wrap: break-word;'>14.399</td><td style='text-align: center; word-wrap: break-word;'>0.023</td><td style='text-align: center; word-wrap: break-word;'>2.658</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Decoder design (ablated from K-free)</td><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'></td></tr><tr><td style='text-align: center; word-wrap: break-word;'>w/o spatial PE</td><td style='text-align: center; word-wrap: break-word;'>15.348</td><td style='text-align: center; word-wrap: break-word;'>8.101</td><td style='text-align: center; word-wrap: break-word;'>13.141</td><td style='text-align: center; word-wrap: break-word;'>0.023</td><td style='text-align: center; word-wrap: break-word;'>2.686</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>w/o joint queries (pooled)</td><td style='text-align: center; word-wrap: break-word;'>17.144</td><td style='text-align: center; word-wrap: break-word;'>8.916</td><td style='text-align: center; word-wrap: break-word;'>14.849</td><td style='text-align: center; word-wrap: break-word;'>0.029</td><td style='text-align: center; word-wrap: break-word;'>2.727</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>shape $\beta$ from registers</td><td style='text-align: center; word-wrap: break-word;'>15.622</td><td style='text-align: center; word-wrap: break-word;'>7.760</td><td style='text-align: center; word-wrap: break-word;'>14.078</td><td style='text-align: center; word-wrap: break-word;'>0.020</td><td style='text-align: center; word-wrap: break-word;'>2.714</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>w/o rotary PE (absolute PE)</td><td style='text-align: center; word-wrap: break-word;'>17.787</td><td style='text-align: center; word-wrap: break-word;'>8.671</td><td style='text-align: center; word-wrap: break-word;'>13.915</td><td style='text-align: center; word-wrap: break-word;'>0.033</td><td style='text-align: center; word-wrap: break-word;'>2.948</td></tr></table>

#### 4.3. Ablation Studies

Video encoder. To assess the necessity of VDM representations in ACE-Ego-Hand, we replace VDM features with VAE latents and raw RGB, remove LoRA to test task-specific adaptation, and substitute Wan 2.2 with frozen V-JEPA 2 and VideoMAE encoders to evaluate robustness across feature sources. All variants, including the full model, are trained on ARCTIC only for 10k steps, a reduced recipe whose MPJPE-p and PA-p sit above the 20k full-mix numbers of Table 1, so the entries are comparable only within this table.

Table 2 demonstrates that ACE-Ego-Hand consistently outperforms all alternative feature sources. Compared to raw RGB (46.50 mm) and VAE latents (32.50 mm), the full model lowers MPJPE-p to 16.95 mm and achieves the best MPJPE $^{OOS}$ (41.5 mm), confirming that high-level spatiotemporal features are essential for temporally coherent recovery. Disabling LoRA adaptation causes MPJPE-p and MPJPE $^{OOS}$ to rise to 23.10 mm and 50.1 mm, verifying the value of task-specific adaptation. Most importantly, while V-JEPA 2 performs reasonably on visible inputs, its out-of-sight error jumps to 66.1 mm, indicating that generative VDM priors provide superior predictive capabilities when visual cues are missing. Both substitutes run frozen while the reference row adapts Wan 2.2 with LoRA, but the matched frozen-to-frozen contrast points the same way, with V-JEPA 2 at 66.1 against 50.1 mm for Wan 2.2 without LoRA.

---

![Image](images/427fe91f5fe48ac10b9b47f58bd320c50e6dc363940f77fd7c5a9ade85d3e1ce.jpg)

<div style="text-align: center;">Figure 4. Qualitative comparison on ARCTIC (top) and HOT3D (bottom). For each dataset: the predicted mesh on the input frame, and the recovered hands in a shared world frame with wrist trails. Every method predicts in the camera frame, and the world view places those predictions with the ground-truth camera poses, so the view shows placement, not recovered camera motion. Under bimanual occlusion on ARCTIC several baselines recover only one hand. On wide-angle HOT3D they scatter the hands across the floor plane.</div>

Decoder and camera solve. We ablate the Ray-Based Camera Solver and the Bidirectional Spatiotemporal Decoder on ARCTIC (Table 3), training all variants under one shared recipe (HOI4D held out). All rows of this table omit $\mathcal{L}_{fit}$ and read bearings directly from the ray field, so they are comparable within the table but not with Table 1. Camera-level ablations evaluate the K-free variant, point-wise inverse projection, and direct translation regression. Decoder-level ablations remove spatial PE, replace joint queries with hand-pooled queries, predict clip-level $\hat{\beta}$ via register tokens, or swap rotary relative PE for absolute PE. Two variants, hand-pooled queries and absolute PE, run at half the data-parallel width of the rest at the same step count and the same clips per GPU, so part of their gap may reflect the smaller effective batch rather than the design change alone.

Within this table, the K-free row matches the standard configuration (15.264 vs. 15.256 mm MPJPE-p and 0.020 vs. 0.021 m CT-p), so learned ray fields replace explicit test-time intrinsics under the camera geometry of ARCTIC. Rotary relative PE and joint queries are pivotal: replacing relative PE with absolute PE degrades MPJPE-p by 2.523 mm and worsens Jitter from 2.704 to 2.948 mm/frame$^{2}$, while pooling joint queries adds 1.880 mm to MPJPE-p and 1.681 px to EPE$_{2D}$-p. For solver components, replacing mixed-PnP with inverse projection degrades CT-p (0.020 to 0.030 m), and direct translation regression increases EPE$_{2D}$-p by 1.231 px. Direct regression does return the lowest Jitter of the table (2.658 mm/frame$^{2}$), a trade-off we do not adopt, since it loosens image-space placement.

#### 4.4. Qualitative Comparison

As illustrated in Figure 4, single-frame and world-space baselines produce fragmented, frequently interrupted predictions, while the generative baseline produces smoother yet depth-biased ones in these examples. In contrast, ACE-Ego-Hand recovers continuous, temporally stable trajectories tightly aligned with ground truth under rapid motion, severe occlusion, and transient out-of-sight intervals. Retargeting the recovered trajectories onto a dexterous Inspire Hand yields natural, coordinated motions that closely mirror the source videos, shown as kinematic visualizations in Appendix E, which also extends this comparison to H2O, OakInk2, and two in-the-wild egocentric sources.

### 5. Conclusion and Discussion

We presented ACE-Ego-Hand, an offline clip-level framework that recovers metric bimanual hand trajectories from egocentric video. Rather than sampling a video diffusion model as a renderer, we read it once as a Deterministic Clean-Latent Encoder and let 3D supervision reshape its features into geometry-aware ones. A Bidirectional Spatiotemporal Decoder then reconstructs whole clips at once, and a Ray-Based Camera Solver places them metrically. ACE-Ego-Hand sets a new state of the art across five benchmarks while running in a single forward pass, $33\times$ faster than the strongest prior method. The best discriminative encoder approaches this accuracy while hands are visible but degrades far more once they leave view, suggesting that generative pretraining may contribute a temporal-geometric

---

prior that discriminative pretraining does not. We hope ACE-Ego-Hand serves as a practical bridge from large-scale human video to manipulation data for robot learning.

Several limitations remain. The predicted ray field generalizes best within camera families seen during training, and broader data coverage is the natural remedy. HOI4D and H2O rely partially on pseudo-ground-truth annotations, so higher-precision data is a promising path to further gains. Moreover, ACE-Ego-Hand operates offline over complete clips, which suits scalable data curation rather than closed-loop control. Distilling the encoder into a streaming variant is a natural step toward online deployment.

## References

[^1]: Mido Assran, Adrien Bardes, David Fan, Quentin Garrido, Russell Howes, Mojtaba Komeili, et al. V-JEPA 2: Self-supervised video models enable understanding, prediction and planning, 2025. Meta AI.

[^2]: Prithviraj Banerjee, Sindi Shkodrani, Pierre Moulon, Shreyas Hampali, Shangchen Han, Fan Zhang, Linguang Zhang, Jade Fountain, Edward Miller, Selen Basol, Richard Newcombe, Robert Wang, Jakob Julian Engel, and Tomas Hodan. HOT3D: Hand and object tracking in 3D from egocentric multi-view videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025. arXiv:2411.19167.

[^3]: Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In Proceedings of the European Conference on Computer Vision (ECCV), 2020.

[^4]: Hongsuk Choi, Gyeongsik Moon, Ju Yong Chang, and Kyoung Mu Lee. Beyond static features for temporally consistent 3D human pose and shape from a video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.

[^5]: Haoye Dong, Aviral Chharia, Wenbo Gou, Francisco Vicente Carrasco, and Fernando De la Torre. Hamba: Single-view 3D hand reconstruction with graph-guided bi-scanning mamba. In Advances in Neural Information Processing Systems (NeurIPS), 2024.

[^6]: Zicong Fan, Omid Taheri, Dimitrios Tzionas, Muhammed Kocabas, Manuel Kaufmann, Michael J. Black, and Otmar Hilliges. ARCTIC: A dataset for dexterous bimanual hand-object manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.

[^7]: Masashi Hatano, Zhifan Zhu, Hideo Saito, and Dima Damen. The invisible EgoHand: 3D hand forecasting through EgoBody pose estimation, 2025.

[^8]: Ryan Hoque, Peide Huang, David J. Yoon, Mouli Sivapurapu, and Jian Zhang. EgoDex: Learning dexterous manipulation from large-scale egocentric video. In International Conference on Learning Representations (ICLR), 2026. arXiv:2505.11709. Apple.

[^9]: Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.

LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR), 2022.

[^10]: Zeyinzi Jiang, Zhen Han, Chaojie Mao, Jingfeng Zhang, Yulin Pan, and Yu Liu. VACE: All-in-one video creation and editing. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2025. arXiv:2503.07598.

[^11]: Linyi Jin, Jianming Zhang, Yannick Hold-Geoffroy, Oliver Wang, Kevin Matzen, Matthew Sticha, and David F. Fouhey. Perspective fields for single image camera calibration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.

[^12]: Simar Kareer, Dhruv Patel, Ryan Punamiya, Pranay Mathur, Shuo Cheng, Chen Wang, Judy Hoffman, and Danfei Xu. EgoMimic: Scaling imitation learning via egocentric video. In IEEE International Conference on Robotics and Automation (ICRA), 2025. arXiv:2410.24221.

[^13]: Muhammed Kocabas, Nikos Athanasiou, and Michael J. Black. VIBE: Video inference for human body pose and shape estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.

[^14]: Harold W. Kuhn. The Hungarian method for the assignment problem. Naval Research Logistics Quarterly, 2, 1955.

[^15]: Taein Kwon, Bugra Tekin, Jan Stühmer, Federica Bogo, and Marc Pollefeys. H2O: Two hands manipulating objects for first person interaction recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021.

[^16]: Hao Li, Ganlong Zhao, Yufei Liu, Haotian Hou, Guoquan Ye, Tongyan Fang, Chunxiao Liu, Siyuan Huang, Jianbo Liu, Xiaogang Wang, and Hongsheng Li. ACE-Ego-0: Unifying egocentric human and robotic data for VLA pretraining, 2026.

[^17]: Jiefeng Li, Jinkun Cao, Haotian Zhang, Davis Rempe, Jan Kautz, Umar Iqbal, and Ye Yuan. GENMO: A generalist model for human motion. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2025. arXiv:2505.01425.

[^18]: Dixuan Lin, Yuxiang Zhang, Mengcheng Li, Wei Jing, Qian Han, Qianying Wang, Yehua Liu, and Hongwen Zhang. OmniHands: Towards robust 4D hand mesh recovery via a versatile transformer, 2024.

[^19]: Yunze Liu, Yun Liu, Che Jiang, Kangbo Lyu, Weikang Wan, Hao Shen, Boqiang Liang, Zhoujie Fu, He Wang, and Li Yi. HOI4D: A 4D egocentric dataset for category-level human-object interaction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.

[^20]: Christen Millerdurai, Shaoxiang Wang, Yaxu Xie, Vladislav Golyanik, Didier Stricker, and Alain Pagani. EgoForce: Forearm-guided camera-space 3D hand pose from a monocular egocentric camera. In ACM SIGGRAPH 2026 Conference Papers, 2026. arXiv:2605.12498.

[^21]: Gyeongsik Moon. Bringing inputs to shared domains for 3D interacting hands recovery in the wild. In Proceedings of

---

the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023. arXiv:2303.13652.

[^22]: Gyeongsik Moon, Shunsuke Saito, Weipeng Xu, et al. A dataset of relighted 3D interacting hands. In Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks, 2023.

[^23]: Georgios Pavlakos, Dandan Shan, Ilija Radosavovic, Angjoo Kanazawa, David Fouhey, and Jitendra Malik. Reconstructing hands in 3D with transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.

[^24]: Rolandos Alexandros Potamias, Jinglei Zhang, Jiankang Deng, and Stefanos Zafeiriou. WiLoR: End-to-end 3D hand localization and reconstruction in-the-wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025. arXiv:2409.12259.

[^25]: Aditya Prakash, Ruisen Tu, Matthew Chang, and Saurabh Gupta. 3D hand pose estimation in everyday egocentric images. In Proceedings of the European Conference on Computer Vision (ECCV), 2024.

[^26]: Javier Romero, Dimitrios Tzionas, and Michael J. Black. Embodied hands: Modeling and capturing hands and bodies together. ACM Transactions on Graphics, 36(6), 2017. Proc. SIGGRAPH Asia 2017; the MANO hand model.

[^27]: Ropedia. Xperience-10M: A large-scale egocentric multi-modal dataset with structured 3d/4d annotations. <https://huggingface.co/datasets/ropedia-ai/xperience-10m>, 2026. Dataset.

[^28]: Xiaolong Shen, Zongxin Yang, Xiaohan Wang, Jianxin Ma, Chang Zhou, and Yi Yang. Global-to-local modeling for video-based 3D human pose and shape estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.

[^29]: Ayush Shrivastava, Sanyam Mehta, Daniel Geng, and Andrew Owens. Point prompting: Counterfactual tracking with video diffusion models. In International Conference on Learning Representations (ICLR), 2026. arXiv:2510.11715.

[^30]: Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. RoFormer: Enhanced transformer with rotary position embedding. Neurocomputing, 568, 2024.

[^31]: Guy Tevet, Sigal Raab, Brian Gordon, Yoni Shafir, Daniel Cohen-Or, and Amit H. Bermano. Human motion diffusion model. In International Conference on Learning Representations (ICLR), 2023.

[^32]: Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. VideoMAE: Masked autoencoders are data-efficient learners for self-supervised video pre-training. In Advances in Neural Information Processing Systems (NeurIPS), 2022.

[^33]: Wan Team. Wan: Open and advanced large-scale video generative models, 2025. Alibaba; covers the Wan2.1/Wan2.2 model family.

[^34]: Letian Wang, Chuhan Zhang, Rishabh Kabra, Jasper Uijlings, Steven Waslander, Andrew Zisserman, Joao Carreira, Kaiming He, Misha Andriluka, Eduard Gabriel Bazavan, Andrei Zanfir, and Cristian Sminchisescu. Video generation models are general-purpose vision learners, 2026. GenCeption; Google DeepMind.

[^35]: Xin Wang, Taein Kwon, Mahdi Rad, Bowen Pan, Ishani Chakraborty, Sean Andrist, Dan Bohus, Ashley Feniello, Bugra Tekin, Felipe Vieira Frujeri, Neel Joshi, and Marc Pollefeys. HoloAssist: An egocentric human interaction dataset for interactive AI assistants in the real world. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023.

[^36]: Yufu Wang, Ziyun Wang, Lingjie Liu, and Kostas Daniilidis. TRAM: Global trajectory and motion of 3D humans from in-the-wild videos. In Proceedings of the European Conference on Computer Vision (ECCV), 2024. arXiv:2403.17346.

[^37]: Yuxi Wang, Chengkai Jin, Yufei Liu, Wenqi Ouyang, Tianyi Wei, Zhiwei Zeng, Siyuan Huang, Zhiqi Shen, and Xingang Pan. The surprising effectiveness of video diffusion models for hand motion reconstruction, 2026.

[^38]: Ruihan Yang, Qinxi Yu, Yecheng Wu, Rui Yan, Borui Li, An-Chieh Cheng, Xueyan Zou, Yunhao Fang, Xuxin Cheng, Ri-Zhao Qiu, Hongxu Yin, Sifei Liu, Song Han, Yao Lu, and Xiaolong Wang. EgoVLA: Learning vision-language-action models from egocentric human videos, 2025.

[^39]: Vickie Ye, Georgios Pavlakos, Jitendra Malik, and Angjoo Kanazawa. Decoupling human and camera motion from videos in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.

[^40]: Zhengdi Yu, Stefanos Zafeiriou, and Tolga Birdal. Dyn-HaMR: Recovering 4D interacting hand motion from a dynamic camera. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025. arXiv:2412.12861.

[^41]: Xinyu Zhan, Lixin Yang, Yifei Zhao, Kangrui Mao, Hanlin Xu, Zenan Lin, Kailin Li, and Cewu Lu. OakInk2: A dataset of bimanual hands-object manipulation in complex task completion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.

[^42]: Jinglei Zhang, Jiankang Deng, Chao Ma, and Rolandos Alexandros Potamias. HaWoR: World-space hand motion reconstruction from egocentric videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025. arXiv:2501.02973.

[^43]: Jason Y. Zhang, Amy Lin, Moneish Kumar, Tzu-Hsuan Yang, Deva Ramanan, and Shubham Tulsiani. Cameras as rays: Pose estimation via ray diffusion. In International Conference on Learning Representations (ICLR), 2024.

[^44]: Christian Zimmermann and Thomas Brox. Learning to estimate 3D hand pose from single RGB images. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2017.

[^45]: Christian Zimmermann, Duygu Ceylan, Jimei Yang, Bryan Russell, Max Argus, and Thomas Brox. FreiHAND: A dataset for markerless capture of hand pose and shape from single RGB images. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019.

---

# ACE-Ego-Hand: Repurposing Video Diffusion Models for Occlusion-Robust Egocentric 3D Hand Motion Recovery

Supplementary Material

The appendix is organized as follows. Appendix A states the evaluation protocol, the dataset splits, and how the baselines were reproduced, so that every number in the main text can be traced to a definition. Appendix B gives the architecture, optimization, and loss details needed to reproduce the model. Appendix C reports the two additional in-domain datasets, the out-of-sight stratification behind the two OOS metrics of the main text, the matched-detection check, and inference throughput. Appendix D analyzes the image-periphery failure that motivates the K-free camera fit and the cost of long decoding passes, and measures two design choices, the tap depth and the latent noise level. Appendix E gives extended qualitative comparisons and the retargeting application. Appendix F closes with the limitations, the failure signature each one produces, and our data-use statement.

### A. Experimental Setup and Protocol

#### A.1. Metric Definitions and Penalty Protocol

This section states the definitions behind every result table of the main text. Let a hand be a pair $(J, \tau)$ of 21 canonical joints and a camera-frame translation, and write $J^{\mathrm{cam}} = J + \tau$. Note that $\tau$ places the canonical hand and is therefore the MANO root translation, not the position of the wrist joint, which is $J_0 + \tau$. CT below is an error in $\tau$.

Table S1 lists every metric symbol used in this paper, the population it averages over, and whether the coverage penalty applies. The rest of this section defines each one. One further metric appears in Appendix C and is named in place: MPJPE restricted to matched detections, a third population again, used there to separate lost coverage from lost pose.

On-screen gate. A hand is on-screen if at least one of its 21 joints projects inside the image rectangle at positive depth,

 $$ \exists j:\;\pi(J_{j}^{\mathrm{c a m}})\in[0,W)\times[0,H)\;\land\;J_{z,j}^{\mathrm{c a m}}>z_{\mathrm{m i n}}, $$

with $\pi$ the calibrated projection and $z_{\min}=1$ cm. Ground-truth hands that fail this test are out of sight (OOS) and are excluded from the main metrics. The same test is applied to predictions, so a prediction that places its hand OOS is not charged as a false positive.

Detection. A predicted hand is active when its existence score exceeds 0.5. Active predictions are matched to on-screen ground-truth hands of the same side by the intersection-over-union of their projected mesh bounding boxes, with ground-truth boxes dilated by 10% before the overlap is computed. Each prediction takes its highest overlap candidate, and where two same-side predictions select the same ground-truth hand, the higher overlap wins. A prediction whose best overlap is zero, or whose best overlap falls on the opposite side, is counted as a false positive, so the rule is a strictly-positive overlap test on the matching side rather than a fixed IoU threshold. Unmatched ground-truth hands count as false negatives (FN), and unmatched predictions as false positives (FP). With per-side counts, and writing P and R for precision and recall,

 $$ \mathrm{P}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}},\qquad\mathrm{R}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}},\qquad\mathrm{F}1=\frac{2\mathrm{PR}}{\mathrm{P}+\mathrm{R}}, $$

and Frame Accuracy is the fraction of frames that contain no error on either side,

 $$ \mathrm{FAcc}=\frac{1}{N}\sum_{f=1}^{N}\mathbf{1}[\mathrm{FP}_{f}+\mathrm{FN}_{f}=0]. $$

Because matching uses the projected mesh, detection depends on predicted translation as well as on the existence score.

Pose metrics. For a matched pair, with $\bar{J} = J - J_{0}$ denoting wrist-relative joints and $\Lambda$ the Procrustes alignment,

 $$ \begin{array}{r l}&{\mathrm{M P J P E}=\frac{1}{21}\sum_{j}\lVert\bar{\hat{J}}_{j}-\bar{J}_{j}\rVert_{2},}\end{array} $$

 $$ \begin{array}{r l}&{\mathrm{P A}=\frac{1}{21}\sum_{j}\lVert\Lambda(\hat{J})_{j}-J_{j}\rVert_{2},}\end{array} $$

 $$ \begin{array}{r l}&{\mathrm{E P E_{2D}}=\frac{1}{|\mathcal{V}|}\sum_{j\in\mathcal{V}}\|\hat{p}_{j}-p_{j}\|_{2},}\end{array} $$

 $$  CT=||\hat{\tau}-\tau||_{2}, $$

where $\hat{p}_j$ is the model's own 2D anchor, $\mathcal{V}$ holds the joints whose ground truth falls inside the frame, and $\mathrm{GO} = \angle(\hat{R}, \hat{R})$ is the geodesic angle between predicted and ground-truth global orientations. One case needs stating separately. For the K-free configuration no intrinsic is applied to fix a pixel scale, so its $\hat{p}_j$ is obtained by re-projecting the predicted camera-frame joints under the clip's ground-truth intrinsic. Those intrinsic are used for scoring only and never reach the model, so that $\mathrm{EPE}_{2D-p}$ becomes a direct read-out of how well the predicted ray field stands in for calibration. Jitter is the mean second temporal difference of the assembled joints $\tilde{\boldsymbol{J}} = \hat{\tilde{\boldsymbol{J}}} + \hat{\tau}$, the wrist-relative joint set plus the predicted translation,

---

 $\frac{1}{T-2}\sum_t\|\tilde{J}_{t+1}-2\tilde{J}_t+\tilde{J}_{t-1}\|$, in  $\text{mm}/\text{frame}^2$. The difference runs over the frame index and is not normalized by frame rate. Jitter is evaluated only on frames for which a method produced a matched detection. For each segment and each ground-truth hand the matched predictions are partitioned into maximal runs of consecutive frames, and the second difference is averaged inside every run of at least three frames. A frame with no detection ends a run rather than being filled by interpolation or by a placeholder pose, and no run crosses a segment boundary. A missed detection therefore removes a frame from the computation instead of charging it, so the convention is conservative for a method that detects nearly every hand.

Reading Jitter. The Bidirectional Spatiotemporal Decoder emits one token set per latent frame, and the Wan VAE compresses time by 4×, so the 81 frames scored in Table 1 are carried by 21 latent frames. Hidden features and the direct joint coordinates are expanded to frame rate by linear interpolation along time, and the per-frame Pose Head and Camera Head then run on the expanded features, while the Shape Head reads a single $\hat{\beta}$ per hand per clip. The scored joints are therefore decoded per frame from a feature trajectory that is piecewise linear in time. Jitter, in mm/frame $^2$, measures the smoothness of that output. Fidelity to the ground truth is judged by the accuracy columns reported beside Jitter.

Coverage penalty (the -p suffix). Reporting errors only on matched hands rewards a method for skipping the hardest frames. Every metric therefore averages over true positives and false negatives, charging each FN the error of a canonical MANO [^26] hand: identity global orientation, zero articulation, mean shape, and $\tau = 0$,

 $$  metric-p=\frac{\sum_{i\in TP}e_{i}+\sum_{i\in FN}e_{i}^{can}}{|TP|+|FN|}. $$

The placeholder is computed per missed ground-truth hand, so the cost of a miss is deterministic and identical for every method. EPE$_{2D}$ is the one exception. Each joint of a missed hand is charged the image diagonal $\sqrt{W^2 + H^2}$, an upper bound on the distance between two points inside the frame. That diagonal is 826 px on ARCTIC and 679 px on HOT3D.

Coverage of out-of-sight hands: two distinct metrics. The on-screen gate restricts every metric above to ground-truth hands inside the frame, so none of them scores a hand while it is out of sight. Two further metrics close that gap. They are easy to confuse, so we state what separates them before defining either.

Both come from a second scoring pass that is gated by the ground truth alone. That pass walks every ground-truth hand-frame of the sequence and scores it against whatever the model emits in that hand slot, with no existence gate,

<div style="text-align: center;">Table S1. Metric symbols. "Averaged over" is the population each metric runs on. "Pen." marks the metrics that charge a placeholder for every missed on-screen detection. The upper block is detection-gated: restricted to on-screen ground truth, matched to predictions, with the -p metrics penalized for each miss. The lower block is ground-truth-gated: no matching, no placeholder, and the two differ only in which hand-frames they average over. The two families are not interchangeable.</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'>Symbol</td><td style='text-align: center; word-wrap: break-word;'>Averaged over</td><td style='text-align: center; word-wrap: break-word;'>Pen.</td><td style='text-align: center; word-wrap: break-word;'>Unit</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>FAcc</td><td style='text-align: center; word-wrap: break-word;'>frames</td><td style='text-align: center; word-wrap: break-word;'>-</td><td style='text-align: center; word-wrap: break-word;'>-</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Recall, F1</td><td style='text-align: center; word-wrap: break-word;'>on-screen hands</td><td style='text-align: center; word-wrap: break-word;'>-</td><td style='text-align: center; word-wrap: break-word;'>-</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>MPJPE-p</td><td style='text-align: center; word-wrap: break-word;'>on-screen hands</td><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>mm</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>PA-p</td><td style='text-align: center; word-wrap: break-word;'>on-screen hands</td><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>mm</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>EPE $_{2D}$-p</td><td style='text-align: center; word-wrap: break-word;'>on-screen joints</td><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>px</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>GO-p</td><td style='text-align: center; word-wrap: break-word;'>on-screen hands</td><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>°</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>CT-p</td><td style='text-align: center; word-wrap: break-word;'>on-screen hands</td><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>m</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Jitter</td><td style='text-align: center; word-wrap: break-word;'>matched runs</td><td style='text-align: center; word-wrap: break-word;'>-</td><td style='text-align: center; word-wrap: break-word;'>mm/frame $^{2}$</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>MPJPE $^{OOS}$</td><td style='text-align: center; word-wrap: break-word;'>out-of-sight hand-frames</td><td style='text-align: center; word-wrap: break-word;'>-</td><td style='text-align: center; word-wrap: break-word;'>mm</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>MPJPE+ $^{OOS}$</td><td style='text-align: center; word-wrap: break-word;'>all hand-frames</td><td style='text-align: center; word-wrap: break-word;'>-</td><td style='text-align: center; word-wrap: break-word;'>mm</td></tr></table>

no detection matching, and no false-negative placeholder. Neither metric therefore carries the -p coverage penalty, and neither is a function of MPJPE-p: MPJPE-p averages over matched detections plus a placeholder for each miss, whereas the two out-of-sight metrics average over ground-truth hand-frames regardless of whether the method detected anything, so the two scoring passes cannot be recombined arithmetically. Both are wrist-aligned, so they certify articulation and orientation through an out-of-sight interval rather than absolute placement.

Write $\mathcal{H}$ for the ground-truth hand-frames of a sequence, $\mathcal{H}^{\mathrm{IV}} \subseteq \mathcal{H}$ for those that pass the on-screen gate, the in-view stratum, and $\mathcal{H}^{\mathrm{OOS}} = \mathcal{H} \setminus \mathcal{H}^{\mathrm{IV}}$ for those that fail it, and let

 $$ \varepsilon_{i}=\frac{1}{21}\sum_{j}\|\bar{\hat{J}}_{j}^{(i)}-\bar{J}_{j}^{(i)}\|_{2} $$

be the wrist-aligned error of hand-frame i, written $\varepsilon$ to keep it distinct from the generic per-hand error e of the penalty above. The two metrics are the same error averaged over different populations,

 $$ \begin{align*}\mathrm{MPJPE}^{\mathrm{OOS}}&=\frac{1}{\left|\mathcal{H}^{\mathrm{OOS}}\right|}\sum_{i\in\mathcal{H}^{\mathrm{OOS}}}\varepsilon_{i},\\\mathrm{MPJPE}^{+}\mathrm{OOS}&=\frac{1}{\left|\mathcal{H}\right|}\sum_{i\in\mathcal{H}}\varepsilon_{i}.\end{align*} $$

The second is therefore the frame-count-weighted mixture of the two strata. Writing $\bar{\varepsilon}_{\mathrm{IV}}$ and $\bar{\varepsilon}_{\mathrm{OOS}}$ for the mean of $\varepsilon_{i}$ over each stratum, and $n_{\mathrm{IV}}$ and $n_{\mathrm{OOS}}$ for the two hand-frame counts,

 $$ MPJPE^{+OOS}=\frac{n_{IV}\bar{\varepsilon}_{IV}+n_{OOS}\bar{\varepsilon}_{OOS}}{n_{IV}+n_{OOS}}. $$

---

MPJPE $^{OOS}$ scores the out-of-sight stratum on its own. MPJPE $^{+OOS}$ pools both strata, so it is dominated by the larger one: out-of-sight hand-frames are 6.9% of ARCTIC, 17.6% of HOT3D, and 14.5% of OakInk2, so the blended figure is dominated by the in-view stratum on all three. A reduction on MPJPE $^{+OOS}$ is therefore not a reduction on out-of-sight hands, and the two must not be read as substitutes for one another. Table 1 (the main comparison) reports the blended form, Table 2 (the feature-source ablation) reports the stratum, and Table S4 of this appendix reports both, together with the in-view stratum, so that Eq. (S11) can be checked row by row.

#### A.2. Datasets, Splits, and Preprocessing

The five evaluated video datasets (ARCTIC [^6], HOT3D [^2], H2O [^15], OakInk2 [^41], and HOI4D [^19]) are processed at 30 fps without temporal subsampling. On the four of these that enter training, training operates on full-length recordings, drawing a random 21-latent-frame (81 RGB frame) window from each recording at every step, while evaluation decodes the fixed 81-frame test segments shared with all baselines. Three further datasets enter the training mixture only. They do not all follow this convention and are described at the end of this subsection. Input resolutions are $672 \times 480$ for ARCTIC and $480 \times 480$ for HOT3D, and H2O and OakInk2 are resized to a width of 832 with intrinsics rescaled accordingly, so that every dataset yields an even latent grid. Test splits are subject-disjoint on ARCTIC (test subject s05) and H2O (test subject 4), recording-level on HOT3D (126/72 recordings), and sequence-level on OakInk2 (evaluated on the 202-segment subset shared with the baselines), while HOI4D is excluded from training entirely and evaluated zero-shot. Each dataset ships its own hand annotation format, and we convert all of them into one shared MANO format so that a single loader and a single evaluator serve every dataset. As the main text notes, HOI4D and H2O rely partially on pseudo-ground-truth MANO annotation derived from a per-frame estimator, so on those two benchmarks the comparison measures agreement with the labels rather than absolute accuracy. The two differ in how much that matters. HOI4D is held out of training entirely, so its labels are noisy but the noise is the same for every method. H2O is in our training mixture, so our model additionally trains on the label distribution that later scores it, while every baseline meets that distribution for the first time at evaluation. ARCTIC, HOT3D, and OakInk2 carry native MANO ground truth, and the headline reductions are quoted on ARCTIC and HOT3D.

Training mixture. Both configurations train on the same seven-source mixture, with a dataset drawn per batch from the weights of Table S2, which sum to 100. Four sources are egocentric video with MANO, namely ARCTIC, HOT3D,

<div style="text-align: center;">Table S2. Training and evaluation data. Clip and frame counts are read from the dataset manifests. For the video sources one clip is one full-length recording, not an 81-frame window. FreiHAND and RHD are static image sources, so their clip column counts images. “Eval” is the number of 81-frame test segments shared with every baseline. For HOT3D and OakInk2 these segments cover only a subset of the available test video. “Wt.” is the per-batch sampling weight in percent and is identical for both configurations. HOI4D is held out of training.</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'></td><td colspan="2">Train</td><td colspan="2">Test</td><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'></td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Source</td><td style='text-align: center; word-wrap: break-word;'>clips</td><td style='text-align: center; word-wrap: break-word;'>frames</td><td style='text-align: center; word-wrap: break-word;'>clips</td><td style='text-align: center; word-wrap: break-word;'>frames</td><td style='text-align: center; word-wrap: break-word;'>Eval</td><td style='text-align: center; word-wrap: break-word;'>Wt.</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>ARCTIC</td><td style='text-align: center; word-wrap: break-word;'>267</td><td style='text-align: center; word-wrap: break-word;'>184,373</td><td style='text-align: center; word-wrap: break-word;'>34</td><td style='text-align: center; word-wrap: break-word;'>24,863</td><td style='text-align: center; word-wrap: break-word;'>291</td><td style='text-align: center; word-wrap: break-word;'>14</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>HOT3D</td><td style='text-align: center; word-wrap: break-word;'>126</td><td style='text-align: center; word-wrap: break-word;'>444,649</td><td style='text-align: center; word-wrap: break-word;'>72</td><td style='text-align: center; word-wrap: break-word;'>256,870</td><td style='text-align: center; word-wrap: break-word;'>437</td><td style='text-align: center; word-wrap: break-word;'>18</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>$H_{2}O$</td><td style='text-align: center; word-wrap: break-word;'>114</td><td style='text-align: center; word-wrap: break-word;'>68,929</td><td style='text-align: center; word-wrap: break-word;'>46</td><td style='text-align: center; word-wrap: break-word;'>30,724</td><td style='text-align: center; word-wrap: break-word;'>355</td><td style='text-align: center; word-wrap: break-word;'>10</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>OakInk2</td><td style='text-align: center; word-wrap: break-word;'>550</td><td style='text-align: center; word-wrap: break-word;'>935,260</td><td style='text-align: center; word-wrap: break-word;'>77</td><td style='text-align: center; word-wrap: break-word;'>69,371</td><td style='text-align: center; word-wrap: break-word;'>202</td><td style='text-align: center; word-wrap: break-word;'>14</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Re:InterHand</td><td style='text-align: center; word-wrap: break-word;'>43</td><td style='text-align: center; word-wrap: break-word;'>21,315</td><td style='text-align: center; word-wrap: break-word;'>-</td><td style='text-align: center; word-wrap: break-word;'>-</td><td style='text-align: center; word-wrap: break-word;'>-</td><td style='text-align: center; word-wrap: break-word;'>16</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>FreiHAND</td><td style='text-align: center; word-wrap: break-word;'>32,560</td><td style='text-align: center; word-wrap: break-word;'>162,800</td><td style='text-align: center; word-wrap: break-word;'>-</td><td style='text-align: center; word-wrap: break-word;'>-</td><td style='text-align: center; word-wrap: break-word;'>-</td><td style='text-align: center; word-wrap: break-word;'>14</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>RHD</td><td style='text-align: center; word-wrap: break-word;'>41,251</td><td style='text-align: center; word-wrap: break-word;'>206,255</td><td style='text-align: center; word-wrap: break-word;'>-</td><td style='text-align: center; word-wrap: break-word;'>-</td><td style='text-align: center; word-wrap: break-word;'>-</td><td style='text-align: center; word-wrap: break-word;'>14</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>HOI4D (held out)</td><td style='text-align: center; word-wrap: break-word;'>-</td><td style='text-align: center; word-wrap: break-word;'>-</td><td style='text-align: center; word-wrap: break-word;'>166</td><td style='text-align: center; word-wrap: break-word;'>49,800</td><td style='text-align: center; word-wrap: break-word;'>498</td><td style='text-align: center; word-wrap: break-word;'>0</td></tr></table>

H2O, and OakInk2. The remaining three are used for training only and are never evaluated on. They hold 44% of the sampling weight and add appearance and pose diversity from outside the egocentric video domain. The three differ in kind. Re:InterHand [^22] contributes relit studio captures rendered under egocentric fisheye cameras at 10 fps, so it enters as video and receives the full 21-latent-frame window like the video datasets above. FreiHAND [^45] and RHD [^44] are two static image sources. Each image is replicated into a static five-frame clip. Those batches contain almost no temporal variation, so they supply appearance and hand-pose diversity rather than motion. FreiHAND is right-hand only and carries a full MANO fit, as does Re:InterHand. RHD provides 21 3D joints and no MANO fit. Those samples supervise the 2D anchor, 3D joint, existence, and visibility heads while the rotation and shape terms are held at zero, following the loss routing described in the main text. Every batch is drawn from a single dataset, so clips of different length and different latent grid are never stacked together.

#### A.3. Baseline Reproduction

Every number we compute is produced by a single evaluation protocol applied to per-segment prediction files, and the values quoted from prior work follow that same protocol, so no metric definition changes between the rows of a table. That protocol is the one introduced by ViDiHand [^37], whose metric definitions, on-screen gate, and test segments we adopt unchanged.

Nine of the twelve rows in each block of Table 1 report methods that ViDiHand also evaluates, and for those the table quotes the published values. EgoForce is not among them, so its row comes from our own run under the same

---

protocol, as do the two ACE-Ego-Hand rows and every row of this appendix. Quoting across papers is only sound if the two implementations of the protocol agree, so we did not take that on trust. We re-ran the eight methods with public releases through our own evaluator, and for ViDiHand, which has no public release, we scored predictions that its authors produced at our request on the same test segments, comparing each outcome with the published number. The agreement is close enough for us to treat the two implementations as one protocol, and where a cell differs, the table keeps the published value rather than ours, so that the nine quoted rows remain consistent with the source they come from. The paragraphs below describe how those reproduction runs were set up. Except for ViDiHand, baseline predictions come from the official public release of each method, run without retraining and without edits to its model code, then converted to the shared MANO format before scoring.

HaMeR [^23] and Hamba [^5] use the released hamer. ckpt and hamba.ckpt behind the Detectron2 ViTDet-H detector and the ViTPose+-Huge whole-body keypoint model that produce their hand boxes. WildHands [^25] uses wildhands.ckpt and OmniHands [^18] its nine-frame video model Demo.Video.pth with config_video. yaml, both behind the same detector front end, that is, ViTDet-H plus ViTPose+-Huge. InterWild [^21] uses the egocentric release snapshot_6_ego.pth behind that front end as well, with its internal BoxNet bypassed at run time so that the same external hand boxes are injected, because that BoxNet mislocalizes hands on egocentric frames. WiLoR [^24] uses wilor_final.ckpt with its own YOLO detector, HaWoR [^42] uses hawor.ckpt, and EgoForce [^20] uses its released model_weights. pth. Dyn-HaMR [^40] runs the default optimization schedule of that repository on top of the HaMeR front end. Because every baseline is scored without retraining on our splits, the training data behind the baseline rows is whatever each model was trained on, and we do not control for that.

Three adaptations are worth stating. Two favor the baseline. WildHands and HaWoR, which take camera geometry as an explicit input, receive the ground-truth intrinsic of the clip, rescaled together with the image where the method requires its own input resolution. Dyn-HaMR receives the ground-truth extrinsics instead of running DROID-SLAM, which removes a failure source unrelated to hand pose. The third works against the baseline, and we state this as a cost to HaWoR. HaWoR's SLAM and infilling stages are skipped and its camera-space motion estimate is scored directly, because the protocol is camera-space and DROID-SLAM is unstable on the 81-frame (2.7 s) test segments. The two stages are coupled in the released code, since the infiller runs in world space on the SLAM trajectory, so dropping SLAM drops infilling with it. That removes the module meant to carry a hand through frames where it is not detected, so the HaWoR rows report the output of its camera-space hand-motion estimator rather than of its full published pipeline. The K-free configuration of ACE-Ego-Hand receives no camera information as input. Ground-truth intrinsic enters its evaluation only where the metric definitions above say so, namely to re-project its 3D joints when $EPE_{2D}$-p is scored.

### B. Implementation Details

#### B.1. Architecture

The spatial PE $P^{sp}$ is a learned $16 \times 16$ grid bilinearly resized to the token resolution, and the ray PE Fourier-encodes each ray's azimuth and elevation with sines and cosines at eight doubling frequencies, after which a zero-initialized MLP maps the resulting features to the decoder width, giving a smooth start to training. In the mixed-PnP solve, a joint votes ( $m_j = 1$) if it lies at least 5 cm in front of the camera and its anchor lies inside the frame by a margin of at least 2% of the image size. When fewer than six joints vote, or when the refit RMS anchor residual exceeds the larger of 15 px and a quarter of the hand's 2D bounding-box diagonal, the wrist is placed on its own inverse-projected ray at depth $\hat{t}_z$. The K-free camera fit is a closed-form, differentiable per-axis linear regression over the token grid, guarded by a variance floor of $10^{-4}$ and a bracket on the fitted focal length; a clip that fails either guard, including the fisheye Re:InterHand training arm, falls back to reading the ray field at the anchors, and both decode paths, the per-joint anchor bearings and the wrist fallback, use the fitted camera. The backbone is the Wan2.2-Fun-5B-Control release [^33] distributed with VideoX-Fun, loaded as its low-noise DiT submodel with 30 blocks of width 3072, feedforward width 14336, 148 input latent channels, and 48 output channels, and paired with the Wan 2.2 VAE. We keep the released input and output channel counts unchanged. LoRA adapters of rank 64 with $\alpha = 64$ and no dropout are injected into all ten linear layers of every block, namely the query, key, value, and output projections of both self-attention and cross-attention plus the two feed-forward layers. Adapters are matched by module name rather than by depth, so one set is instantiated in each of the 30 blocks, giving 5.37M adapter parameters per block and 161.219M in total. Three counts are worth keeping apart. Of the 5.002B pretrained weights the released DiT holds, only the 1.822M patch embedding is ever updated, and the 30 transformer blocks stay frozen throughout. With the adapters attached, the instantiated backbone holds 5.16B parameters. The optimizer is then handed 183.99M parameters, split by module into 161.219M in the LoRA adapters, 20.347M in the decoder and its readout heads, 1.822M in the patch embed-

---

ding, 0.596M in the DiT's own diffusion output head, and 9,219 in the Ray Head. These fall into the three learning-rate groups of the next subsection: the LoRA group, the patch-embedding group, and one group holding the decoder, its readout heads, the Ray Head, and the diffusion output head. Since the forward pass stops at the tap, the parameters a gradient can actually reach are the 85.983M of LoRA inside the executed blocks, the 1.822M patch embedding, the 12.287M of decoder parameters that this configuration routes through, and the Ray Head, for 100.10M in all. The adapters in the bypassed blocks and the diffusion output head sit on the skipped path and therefore never leave their initialization. We confirmed this on the released checkpoint: after 20k steps every LoRA B matrix past the tap is still exactly zero, which makes those adapters exact identity maps, and every tensor of the diffusion output head is bit-identical to the pretrained release. The decoder parameters outside the 12.287M belong to readout variants that this configuration does not select, and we keep them instantiated so that a single checkpoint schema covers every ablation.

#### B.2. Optimization

Only the 183.99M parameters listed above are registered with the optimizer. The decoder, the Ray Head, and the LoRA adapters are trained from scratch, with the Ray Head and the LoRA B matrices starting at zero so that the network begins the run as the unmodified backbone, while the patch embedding is fine-tuned from its released weights. We run 20k AdamW steps (weight decay $10^{-2}$, gradient clip 1.0, cosine decay, 200 warmup steps) at three learning rates: $2\times10^{-4}$ for the decoder and heads, $1\times10^{-4}$ for the LoRA adapters, $2\times10^{-5}$ for the patch embedding. Each GPU holds four clips, where a clip is 81 frames for the video datasets and five frames for the two image datasets. The standard configuration runs on 16 A100 GPUs, for an effective batch of 64 clips, and the K-free configuration on 8, for an effective batch of 32. The ablations of Table 3 follow the K-free configuration, except that the hand-pooled-query and absolute-PE variants run on half as many GPUs at the same step count and the same number of clips per GPU, the halved-effective-batch caveat noted alongside Table 3.

#### B.3. Loss Weights

ACE-Ego-Hand's trainable components, namely the LoRA adapters, the patch embedding, the Ray Head, and the Bidirectional Spatiotemporal Decoder, are optimized jointly under Eq. (6), $\mathcal{L} = \mathcal{L}_{\text{rot}} + \mathcal{L}_{\text{joint}} + \mathcal{L}_{\text{img}} + \mathcal{L}_{\text{cam}} + \mathcal{L}_{\text{pres}} + \mathcal{L}_{\text{tmp}} + \mathcal{L}_{\text{ray}}$. The per-term weights are as follows.

$\mathcal{L}_{\mathrm{rot}}$ supervises orientation and articulation with the geodesic distance $\operatorname{arccos}\left(\frac{1}{2}(\operatorname{tr}(\hat{R}^{\top}R)-1)\right)$ to the ground-truth rotation R, plus a rotation-matrix MSE (weight 1 each), and shape with an $\ell_1$ loss (0.1). $\mathcal{L}_{\mathrm{joint}}$ places $\ell_1$ losses on root-relative (weight 10, the dominant term), camera-frame (5), and wrist (2) 3D joints. $\mathcal{L}_{\mathrm{img}}$ supervises, under the training camera, the grounded soft-argmax 2D anchors and the re-projected MANO joints (weight 1, and 0.5 for the wrist). $\mathcal{L}_{\mathrm{cam}}$ is an $\ell_1$ loss on the assembled translation (weight 1), with the gradient flowing through the mixed-PnP solve. $\mathcal{L}_{\mathrm{pres}}$ applies binary cross-entropy to existence and visibility (0.5/0.25). $\mathcal{L}_{\mathrm{tmp}}$ penalizes the acceleration of the predicted 3D joints $\hat{J}_t, \|\hat{J}_{t+1}-2\hat{J}_t + \hat{J}_{t-1}\|_1$ (0.5). $\mathcal{L}_{\mathrm{ray}}$ carries weight 1, and the K-free configuration's $\mathcal{L}_{\mathrm{fit}}$ carries weight 5 with a linear warmup over the first 500 steps.

### C. Additional Quantitative Results

#### C.1. Additional In-Domain Datasets: $H_{2}O$ and OakInk2

The main comparison (Table 1) reports ARCTIC, HOT3D, and held-out HOI4D. Table S3 gives the full per-dataset results on two further in-domain datasets, H2O and OakInk2, scored with the same shared evaluator. ACE-Ego-Hand again leads every accuracy metric but one, cutting the best baseline MPJPE-p by 66% on OakInk2 (26.520→8.988) and by 43% on H2O (16.219→9.223). The K-free configuration matches the standard configuration's detection on both datasets, with identical detection columns on OakInk2, and stays within 1.1 mm of its MPJPE-p (10.31 vs. 9.22 on H2O and 9.81 vs. 8.99 on OakInk2). The only cell baseline wins is H2O CT-p, where the weak-perspective solve of WiLoR is marginally tighter (0.023 vs. 0.031). Two caveats apply. First, each ACE-Ego-Hand entry comes from a single training run. Second, both datasets are in-domain for us and out-of-domain for every baseline, so these two blocks read as an in-domain upper bound rather than as the like-for-like comparison ARCTIC and HOT3D provide. On H2O, whose labels are partially pseudo-ground-truth, our model has additionally trained on the distribution it is scored against, whereas OakInk2 carries native MANO ground truth.

#### C.2. Out-of-Sight Stratification

Table S4 decomposes the wrist-aligned error into its interview and out-of-sight strata for every method on the three datasets that carry out-of-sight ground truth. The table exists so that the two out-of-sight numbers of the main text can be told apart and audited separately, since the blended metric alone cannot distinguish a method that reconstructs out-of-sight hands from one that is merely accurate while the hands are in view.

The decomposition makes the difference concrete. On ARCTIC, ViDiHand and ACE-Ego-Hand differ by under 7 mm in view (22.311 against 15.423) but by 113.9 mm out of sight (149.113 against 35.165). Because only 6.9% of ARCTIC hand-frames are out of sight, the blended col

---

<div style="text-align: center;">Table S3. Results on H2O and OakInk2. Same protocol as Table 1. Best per column in bold. MPJPE $^{OOS}$ needs out-of-sight ground truth, which H2O lacks (-). OakInk2 uses the 202-segment subset shared with the baselines.  $^{\ddagger}$EgoForce post-filters translation with a causal Kalman filter.</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'></td><td colspan="3">Detection</td><td colspan="3">3D Pose</td><td colspan="3">Orient. &amp; Position</td><td style='text-align: center; word-wrap: break-word;'>Temporal</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Method</td><td style='text-align: center; word-wrap: break-word;'>FAcc $\uparrow$</td><td style='text-align: center; word-wrap: break-word;'>Recall $\uparrow$</td><td style='text-align: center; word-wrap: break-word;'>F1 $\uparrow$</td><td style='text-align: center; word-wrap: break-word;'>MPJPE-p $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>PA-p $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>MPJPE+OOS $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>EPE $_{2D}$-p $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>GO-p $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>CT-p $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>Jitter $\downarrow$</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>InterWild</td><td style='text-align: center; word-wrap: break-word;'>0.981</td><td style='text-align: center; word-wrap: break-word;'>0.990</td><td style='text-align: center; word-wrap: break-word;'>0.994</td><td style='text-align: center; word-wrap: break-word;'>21.526</td><td style='text-align: center; word-wrap: break-word;'>8.979</td><td style='text-align: center; word-wrap: break-word;'>-</td><td style='text-align: center; word-wrap: break-word;'>19.929</td><td style='text-align: center; word-wrap: break-word;'>17.695</td><td style='text-align: center; word-wrap: break-word;'>0.037</td><td style='text-align: center; word-wrap: break-word;'>16.153</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>HaMeR</td><td style='text-align: center; word-wrap: break-word;'>0.980</td><td style='text-align: center; word-wrap: break-word;'>0.990</td><td style='text-align: center; word-wrap: break-word;'>0.992</td><td style='text-align: center; word-wrap: break-word;'>20.079</td><td style='text-align: center; word-wrap: break-word;'>7.221</td><td style='text-align: center; word-wrap: break-word;'>-</td><td style='text-align: center; word-wrap: break-word;'>21.786</td><td style='text-align: center; word-wrap: break-word;'>17.971</td><td style='text-align: center; word-wrap: break-word;'>0.034</td><td style='text-align: center; word-wrap: break-word;'>9.307</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Hamba</td><td style='text-align: center; word-wrap: break-word;'>0.960</td><td style='text-align: center; word-wrap: break-word;'>0.979</td><td style='text-align: center; word-wrap: break-word;'>0.987</td><td style='text-align: center; word-wrap: break-word;'>21.412</td><td style='text-align: center; word-wrap: break-word;'>8.390</td><td style='text-align: center; word-wrap: break-word;'>-</td><td style='text-align: center; word-wrap: break-word;'>32.978</td><td style='text-align: center; word-wrap: break-word;'>19.510</td><td style='text-align: center; word-wrap: break-word;'>0.038</td><td style='text-align: center; word-wrap: break-word;'>8.285</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>WildHands</td><td style='text-align: center; word-wrap: break-word;'>0.989</td><td style='text-align: center; word-wrap: break-word;'>0.995</td><td style='text-align: center; word-wrap: break-word;'>0.996</td><td style='text-align: center; word-wrap: break-word;'>32.587</td><td style='text-align: center; word-wrap: break-word;'>10.508</td><td style='text-align: center; word-wrap: break-word;'>-</td><td style='text-align: center; word-wrap: break-word;'>26.297</td><td style='text-align: center; word-wrap: break-word;'>22.033</td><td style='text-align: center; word-wrap: break-word;'>0.082</td><td style='text-align: center; word-wrap: break-word;'>10.894</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>WiLoR</td><td style='text-align: center; word-wrap: break-word;'>0.994</td><td style='text-align: center; word-wrap: break-word;'>0.998</td><td style='text-align: center; word-wrap: break-word;'>0.998</td><td style='text-align: center; word-wrap: break-word;'>16.390</td><td style='text-align: center; word-wrap: break-word;'>5.633</td><td style='text-align: center; word-wrap: break-word;'>-</td><td style='text-align: center; word-wrap: break-word;'>14.497</td><td style='text-align: center; word-wrap: break-word;'>14.495</td><td style='text-align: center; word-wrap: break-word;'>0.023</td><td style='text-align: center; word-wrap: break-word;'>6.117</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>EgoForce $\ddagger$</td><td style='text-align: center; word-wrap: break-word;'>0.990</td><td style='text-align: center; word-wrap: break-word;'>0.996</td><td style='text-align: center; word-wrap: break-word;'>0.997</td><td style='text-align: center; word-wrap: break-word;'>16.219</td><td style='text-align: center; word-wrap: break-word;'>6.020</td><td style='text-align: center; word-wrap: break-word;'>-</td><td style='text-align: center; word-wrap: break-word;'>15.849</td><td style='text-align: center; word-wrap: break-word;'>10.416</td><td style='text-align: center; word-wrap: break-word;'>0.024</td><td style='text-align: center; word-wrap: break-word;'>8.046</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>OmniHands</td><td style='text-align: center; word-wrap: break-word;'>0.974</td><td style='text-align: center; word-wrap: break-word;'>0.986</td><td style='text-align: center; word-wrap: break-word;'>0.990</td><td style='text-align: center; word-wrap: break-word;'>19.835</td><td style='text-align: center; word-wrap: break-word;'>7.417</td><td style='text-align: center; word-wrap: break-word;'>-</td><td style='text-align: center; word-wrap: break-word;'>25.665</td><td style='text-align: center; word-wrap: break-word;'>18.032</td><td style='text-align: center; word-wrap: break-word;'>0.038</td><td style='text-align: center; word-wrap: break-word;'>6.321</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Dyn-HaMR</td><td style='text-align: center; word-wrap: break-word;'>0.988</td><td style='text-align: center; word-wrap: break-word;'>0.999</td><td style='text-align: center; word-wrap: break-word;'>0.997</td><td style='text-align: center; word-wrap: break-word;'>17.213</td><td style='text-align: center; word-wrap: break-word;'>6.390</td><td style='text-align: center; word-wrap: break-word;'>-</td><td style='text-align: center; word-wrap: break-word;'>10.790</td><td style='text-align: center; word-wrap: break-word;'>11.057</td><td style='text-align: center; word-wrap: break-word;'>0.030</td><td style='text-align: center; word-wrap: break-word;'>5.349</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>HaWoR</td><td style='text-align: center; word-wrap: break-word;'>0.946</td><td style='text-align: center; word-wrap: break-word;'>0.972</td><td style='text-align: center; word-wrap: break-word;'>0.985</td><td style='text-align: center; word-wrap: break-word;'>23.504</td><td style='text-align: center; word-wrap: break-word;'>9.570</td><td style='text-align: center; word-wrap: break-word;'>-</td><td style='text-align: center; word-wrap: break-word;'>39.735</td><td style='text-align: center; word-wrap: break-word;'>17.080</td><td style='text-align: center; word-wrap: break-word;'>0.045</td><td style='text-align: center; word-wrap: break-word;'>14.162</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>ViDiHand</td><td style='text-align: center; word-wrap: break-word;'>0.997</td><td style='text-align: center; word-wrap: break-word;'>0.999</td><td style='text-align: center; word-wrap: break-word;'>0.999</td><td style='text-align: center; word-wrap: break-word;'>17.320</td><td style='text-align: center; word-wrap: break-word;'>7.987</td><td style='text-align: center; word-wrap: break-word;'>-</td><td style='text-align: center; word-wrap: break-word;'>16.544</td><td style='text-align: center; word-wrap: break-word;'>12.159</td><td style='text-align: center; word-wrap: break-word;'>0.040</td><td style='text-align: center; word-wrap: break-word;'>1.990</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>ACE-Ego-Hand</td><td style='text-align: center; word-wrap: break-word;'>0.999</td><td style='text-align: center; word-wrap: break-word;'>1.000</td><td style='text-align: center; word-wrap: break-word;'>1.000</td><td style='text-align: center; word-wrap: break-word;'>9.223</td><td style='text-align: center; word-wrap: break-word;'>4.894</td><td style='text-align: center; word-wrap: break-word;'>-</td><td style='text-align: center; word-wrap: break-word;'>6.088</td><td style='text-align: center; word-wrap: break-word;'>5.689</td><td style='text-align: center; word-wrap: break-word;'>0.031</td><td style='text-align: center; word-wrap: break-word;'>0.707</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>ACE-Ego-Hand (K-free)</td><td style='text-align: center; word-wrap: break-word;'>0.999</td><td style='text-align: center; word-wrap: break-word;'>1.000</td><td style='text-align: center; word-wrap: break-word;'>1.000</td><td style='text-align: center; word-wrap: break-word;'>10.310</td><td style='text-align: center; word-wrap: break-word;'>5.522</td><td style='text-align: center; word-wrap: break-word;'>-</td><td style='text-align: center; word-wrap: break-word;'>9.464</td><td style='text-align: center; word-wrap: break-word;'>6.064</td><td style='text-align: center; word-wrap: break-word;'>0.024</td><td style='text-align: center; word-wrap: break-word;'>0.741</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>InterWild</td><td style='text-align: center; word-wrap: break-word;'>0.547</td><td style='text-align: center; word-wrap: break-word;'>0.722</td><td style='text-align: center; word-wrap: break-word;'>0.819</td><td style='text-align: center; word-wrap: break-word;'>51.013</td><td style='text-align: center; word-wrap: break-word;'>37.038</td><td style='text-align: center; word-wrap: break-word;'>65.619</td><td style='text-align: center; word-wrap: break-word;'>244.590</td><td style='text-align: center; word-wrap: break-word;'>54.710</td><td style='text-align: center; word-wrap: break-word;'>0.179</td><td style='text-align: center; word-wrap: break-word;'>43.499</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>HaMeR</td><td style='text-align: center; word-wrap: break-word;'>0.680</td><td style='text-align: center; word-wrap: break-word;'>0.799</td><td style='text-align: center; word-wrap: break-word;'>0.866</td><td style='text-align: center; word-wrap: break-word;'>43.804</td><td style='text-align: center; word-wrap: break-word;'>28.726</td><td style='text-align: center; word-wrap: break-word;'>59.471</td><td style='text-align: center; word-wrap: break-word;'>171.887</td><td style='text-align: center; word-wrap: break-word;'>46.340</td><td style='text-align: center; word-wrap: break-word;'>0.148</td><td style='text-align: center; word-wrap: break-word;'>19.510</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Hamba</td><td style='text-align: center; word-wrap: break-word;'>0.634</td><td style='text-align: center; word-wrap: break-word;'>0.755</td><td style='text-align: center; word-wrap: break-word;'>0.840</td><td style='text-align: center; word-wrap: break-word;'>46.517</td><td style='text-align: center; word-wrap: break-word;'>32.488</td><td style='text-align: center; word-wrap: break-word;'>61.863</td><td style='text-align: center; word-wrap: break-word;'>213.576</td><td style='text-align: center; word-wrap: break-word;'>51.295</td><td style='text-align: center; word-wrap: break-word;'>0.159</td><td style='text-align: center; word-wrap: break-word;'>13.811</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>WildHands</td><td style='text-align: center; word-wrap: break-word;'>0.731</td><td style='text-align: center; word-wrap: break-word;'>0.837</td><td style='text-align: center; word-wrap: break-word;'>0.890</td><td style='text-align: center; word-wrap: break-word;'>44.087</td><td style='text-align: center; word-wrap: break-word;'>27.115</td><td style='text-align: center; word-wrap: break-word;'>58.366</td><td style='text-align: center; word-wrap: break-word;'>151.305</td><td style='text-align: center; word-wrap: break-word;'>43.806</td><td style='text-align: center; word-wrap: break-word;'>0.146</td><td style='text-align: center; word-wrap: break-word;'>15.156</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>WiLoR</td><td style='text-align: center; word-wrap: break-word;'>0.921</td><td style='text-align: center; word-wrap: break-word;'>0.955</td><td style='text-align: center; word-wrap: break-word;'>0.973</td><td style='text-align: center; word-wrap: break-word;'>26.520</td><td style='text-align: center; word-wrap: break-word;'>12.481</td><td style='text-align: center; word-wrap: break-word;'>45.215</td><td style='text-align: center; word-wrap: break-word;'>39.928</td><td style='text-align: center; word-wrap: break-word;'>25.084</td><td style='text-align: center; word-wrap: break-word;'>0.085</td><td style='text-align: center; word-wrap: break-word;'>9.522</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>EgoForce $\ddagger$</td><td style='text-align: center; word-wrap: break-word;'>0.846</td><td style='text-align: center; word-wrap: break-word;'>0.912</td><td style='text-align: center; word-wrap: break-word;'>0.949</td><td style='text-align: center; word-wrap: break-word;'>36.725</td><td style='text-align: center; word-wrap: break-word;'>18.198</td><td style='text-align: center; word-wrap: break-word;'>55.095</td><td style='text-align: center; word-wrap: break-word;'>81.356</td><td style='text-align: center; word-wrap: break-word;'>35.671</td><td style='text-align: center; word-wrap: break-word;'>0.101</td><td style='text-align: center; word-wrap: break-word;'>30.476</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>OmniHands</td><td style='text-align: center; word-wrap: break-word;'>0.530</td><td style='text-align: center; word-wrap: break-word;'>0.688</td><td style='text-align: center; word-wrap: break-word;'>0.787</td><td style='text-align: center; word-wrap: break-word;'>54.712</td><td style='text-align: center; word-wrap: break-word;'>40.373</td><td style='text-align: center; word-wrap: break-word;'>68.024</td><td style='text-align: center; word-wrap: break-word;'>283.245</td><td style='text-align: center; word-wrap: break-word;'>60.157</td><td style='text-align: center; word-wrap: break-word;'>0.174</td><td style='text-align: center; word-wrap: break-word;'>22.562</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Dyn-HaMR</td><td style='text-align: center; word-wrap: break-word;'>0.882</td><td style='text-align: center; word-wrap: break-word;'>0.941</td><td style='text-align: center; word-wrap: break-word;'>0.947</td><td style='text-align: center; word-wrap: break-word;'>28.781</td><td style='text-align: center; word-wrap: break-word;'>14.945</td><td style='text-align: center; word-wrap: break-word;'>47.434</td><td style='text-align: center; word-wrap: break-word;'>46.902</td><td style='text-align: center; word-wrap: break-word;'>25.711</td><td style='text-align: center; word-wrap: break-word;'>0.102</td><td style='text-align: center; word-wrap: break-word;'>7.311</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>HaWoR</td><td style='text-align: center; word-wrap: break-word;'>0.818</td><td style='text-align: center; word-wrap: break-word;'>0.891</td><td style='text-align: center; word-wrap: break-word;'>0.935</td><td style='text-align: center; word-wrap: break-word;'>35.428</td><td style='text-align: center; word-wrap: break-word;'>20.670</td><td style='text-align: center; word-wrap: break-word;'>52.775</td><td style='text-align: center; word-wrap: break-word;'>96.442</td><td style='text-align: center; word-wrap: break-word;'>32.043</td><td style='text-align: center; word-wrap: break-word;'>0.120</td><td style='text-align: center; word-wrap: break-word;'>13.780</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>ViDiHand</td><td style='text-align: center; word-wrap: break-word;'>0.813</td><td style='text-align: center; word-wrap: break-word;'>0.886</td><td style='text-align: center; word-wrap: break-word;'>0.937</td><td style='text-align: center; word-wrap: break-word;'>38.996</td><td style='text-align: center; word-wrap: break-word;'>22.827</td><td style='text-align: center; word-wrap: break-word;'>56.047</td><td style='text-align: center; word-wrap: break-word;'>101.568</td><td style='text-align: center; word-wrap: break-word;'>38.119</td><td style='text-align: center; word-wrap: break-word;'>0.103</td><td style='text-align: center; word-wrap: break-word;'>3.728</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>ACE-Ego-Hand</td><td style='text-align: center; word-wrap: break-word;'>0.977</td><td style='text-align: center; word-wrap: break-word;'>0.985</td><td style='text-align: center; word-wrap: break-word;'>0.993</td><td style='text-align: center; word-wrap: break-word;'>8.988</td><td style='text-align: center; word-wrap: break-word;'>5.887</td><td style='text-align: center; word-wrap: break-word;'>8.272</td><td style='text-align: center; word-wrap: break-word;'>11.129</td><td style='text-align: center; word-wrap: break-word;'>7.222</td><td style='text-align: center; word-wrap: break-word;'>0.018</td><td style='text-align: center; word-wrap: break-word;'>1.089</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>ACE-Ego-Hand (K-free)</td><td style='text-align: center; word-wrap: break-word;'>0.977</td><td style='text-align: center; word-wrap: break-word;'>0.985</td><td style='text-align: center; word-wrap: break-word;'>0.993</td><td style='text-align: center; word-wrap: break-word;'>9.813</td><td style='text-align: center; word-wrap: break-word;'>6.362</td><td style='text-align: center; word-wrap: break-word;'>9.308</td><td style='text-align: center; word-wrap: break-word;'>12.046</td><td style='text-align: center; word-wrap: break-word;'>7.366</td><td style='text-align: center; word-wrap: break-word;'>0.018</td><td style='text-align: center; word-wrap: break-word;'>1.150</td></tr></table>

umn compresses those two very different gaps into 31.045 against 16.783. In HOT3D, 17.6% of hand-frames are out of sight, so more of the out-of-sight gap survives into the blend. The blended reduction is therefore larger on HOT3D (61.1%) than on ARCTIC (45.9%), while the stratum reduction runs the other way (74.5% against 76.4%).

Two further points are visible only in the stratum column. First, the baselines are not merely worse out of sight. They collapse into one narrow band far above their in-view errors: all ten sit between 126 and 164 mm on all three datasets, with no ordering that resembles their in-view ranking. This is what one would expect of methods that, in the configuration scored here, have no mechanism for producing a hand pose when no hand appears in the frame. Second, and as a consequence, the strongest baseline out of sight is not the strongest baseline in view: WildHands leads the out-of-sight stratum on ARCTIC (135.437) and HOT3D (126.358) despite trailing WiLoR and ViDiHand in view, and ACE-Ego-Hand still reduces WildHands' error by 74.0% and 69.4%. We read the out-of-sight margin as evidence of temporal reconstruction rather than of better perception, and we caution that it is a wrist-aligned margin: Appendix F states what that margin does not establish.

#### C.3. Matched-Detection Errors

Restricting the evaluation to matched detections separates the placement cost that arises at the image periphery without the camera fit from articulation accuracy. This defines a third population, narrower than the in-view stratum of Table S4 because it drops the hand-frames a method failed to detect. The matched-detection numbers are therefore near, but not equal to, the in-view column of that table. On HOT3D the variant without the fit ties the standard configuration once false negatives are excluded (12.75 mm against 12.68 mm, wrist-aligned), whereas its coverage-penalized gap in the same order is 18.04 against 12.89 mm. On OakInk2 the two remain close, 7.96 mm against 7.47 mm. That penalized gap therefore reflects lost coverage at the image periphery, not a loss of hand pose, and the camera fit recovers exactly that coverage (Table 1).

---

<div style="text-align: center;">Table S4. Out-of-sight stratification of the wrist-aligned error. The same wrist-aligned, ground-truth-gated error over three populations: the hand-frames that pass the on-screen gate, the hand-frames that fail it (MPJPE^{OOS}), and all of them together (MPJPE^{OOS}). Every row satisfies Eq. (S11) at the counts in the header. None carries the -p penalty, so the in-view column is close to but not the same as the MPJPE-p of Tables 1 and S3. HOI4D and H2O carry no out-of-sight ground truth. Lower is better.  ${}^{3}$EgoForce post-filters translation with a causal Kalman filter.</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td rowspan="2">Method</td><td colspan="3">ARCTIC (43,893 IV / 3,247 OOS)</td><td colspan="3">HOT3D (57,985 / 12,394)</td><td colspan="3">OakInk2 (27,988 / 4,736)</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>in view</td><td style='text-align: center; word-wrap: break-word;'>out of sight</td><td style='text-align: center; word-wrap: break-word;'>MPJPE $^{{+OOS}}$</td><td style='text-align: center; word-wrap: break-word;'>in view</td><td style='text-align: center; word-wrap: break-word;'>out of sight</td><td style='text-align: center; word-wrap: break-word;'>MPJPE $^{{+OOS}}$</td><td style='text-align: center; word-wrap: break-word;'>in view</td><td style='text-align: center; word-wrap: break-word;'>out of sight</td><td style='text-align: center; word-wrap: break-word;'>MPJPE $^{{+OOS}}$</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>InterWild</td><td style='text-align: center; word-wrap: break-word;'>32.000</td><td style='text-align: center; word-wrap: break-word;'>139.937</td><td style='text-align: center; word-wrap: break-word;'>39.435</td><td style='text-align: center; word-wrap: break-word;'>77.866</td><td style='text-align: center; word-wrap: break-word;'>142.326</td><td style='text-align: center; word-wrap: break-word;'>89.218</td><td style='text-align: center; word-wrap: break-word;'>49.364</td><td style='text-align: center; word-wrap: break-word;'>161.681</td><td style='text-align: center; word-wrap: break-word;'>65.619</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>HaMeR</td><td style='text-align: center; word-wrap: break-word;'>30.576</td><td style='text-align: center; word-wrap: break-word;'>141.005</td><td style='text-align: center; word-wrap: break-word;'>38.183</td><td style='text-align: center; word-wrap: break-word;'>67.956</td><td style='text-align: center; word-wrap: break-word;'>137.847</td><td style='text-align: center; word-wrap: break-word;'>80.264</td><td style='text-align: center; word-wrap: break-word;'>42.361</td><td style='text-align: center; word-wrap: break-word;'>160.588</td><td style='text-align: center; word-wrap: break-word;'>59.471</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Hamba</td><td style='text-align: center; word-wrap: break-word;'>32.513</td><td style='text-align: center; word-wrap: break-word;'>141.774</td><td style='text-align: center; word-wrap: break-word;'>40.039</td><td style='text-align: center; word-wrap: break-word;'>71.277</td><td style='text-align: center; word-wrap: break-word;'>140.333</td><td style='text-align: center; word-wrap: break-word;'>83.438</td><td style='text-align: center; word-wrap: break-word;'>45.119</td><td style='text-align: center; word-wrap: break-word;'>160.812</td><td style='text-align: center; word-wrap: break-word;'>61.863</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>WildHands</td><td style='text-align: center; word-wrap: break-word;'>26.404</td><td style='text-align: center; word-wrap: break-word;'>135.437</td><td style='text-align: center; word-wrap: break-word;'>33.915</td><td style='text-align: center; word-wrap: break-word;'>46.412</td><td style='text-align: center; word-wrap: break-word;'>126.358</td><td style='text-align: center; word-wrap: break-word;'>60.491</td><td style='text-align: center; word-wrap: break-word;'>41.184</td><td style='text-align: center; word-wrap: break-word;'>159.901</td><td style='text-align: center; word-wrap: break-word;'>58.366</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>WiLoR</td><td style='text-align: center; word-wrap: break-word;'>22.775</td><td style='text-align: center; word-wrap: break-word;'>149.477</td><td style='text-align: center; word-wrap: break-word;'>31.502</td><td style='text-align: center; word-wrap: break-word;'>30.921</td><td style='text-align: center; word-wrap: break-word;'>150.697</td><td style='text-align: center; word-wrap: break-word;'>52.014</td><td style='text-align: center; word-wrap: break-word;'>25.428</td><td style='text-align: center; word-wrap: break-word;'>162.143</td><td style='text-align: center; word-wrap: break-word;'>45.215</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>EgoForce $^{\ddagger}$</td><td style='text-align: center; word-wrap: break-word;'>23.533</td><td style='text-align: center; word-wrap: break-word;'>149.264</td><td style='text-align: center; word-wrap: break-word;'>32.193</td><td style='text-align: center; word-wrap: break-word;'>44.225</td><td style='text-align: center; word-wrap: break-word;'>151.322</td><td style='text-align: center; word-wrap: break-word;'>63.085</td><td style='text-align: center; word-wrap: break-word;'>36.690</td><td style='text-align: center; word-wrap: break-word;'>163.858</td><td style='text-align: center; word-wrap: break-word;'>55.095</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>OmniHands</td><td style='text-align: center; word-wrap: break-word;'>30.706</td><td style='text-align: center; word-wrap: break-word;'>139.371</td><td style='text-align: center; word-wrap: break-word;'>38.191</td><td style='text-align: center; word-wrap: break-word;'>61.152</td><td style='text-align: center; word-wrap: break-word;'>131.289</td><td style='text-align: center; word-wrap: break-word;'>73.503</td><td style='text-align: center; word-wrap: break-word;'>52.382</td><td style='text-align: center; word-wrap: break-word;'>160.461</td><td style='text-align: center; word-wrap: break-word;'>68.024</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Dyn-HaMR</td><td style='text-align: center; word-wrap: break-word;'>29.084</td><td style='text-align: center; word-wrap: break-word;'>150.464</td><td style='text-align: center; word-wrap: break-word;'>37.445</td><td style='text-align: center; word-wrap: break-word;'>67.786</td><td style='text-align: center; word-wrap: break-word;'>142.547</td><td style='text-align: center; word-wrap: break-word;'>80.952</td><td style='text-align: center; word-wrap: break-word;'>28.127</td><td style='text-align: center; word-wrap: break-word;'>161.532</td><td style='text-align: center; word-wrap: break-word;'>47.434</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>HaWoR</td><td style='text-align: center; word-wrap: break-word;'>46.406</td><td style='text-align: center; word-wrap: break-word;'>148.565</td><td style='text-align: center; word-wrap: break-word;'>53.443</td><td style='text-align: center; word-wrap: break-word;'>70.767</td><td style='text-align: center; word-wrap: break-word;'>150.074</td><td style='text-align: center; word-wrap: break-word;'>84.733</td><td style='text-align: center; word-wrap: break-word;'>34.343</td><td style='text-align: center; word-wrap: break-word;'>161.700</td><td style='text-align: center; word-wrap: break-word;'>52.775</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>ViDiHand</td><td style='text-align: center; word-wrap: break-word;'>22.311</td><td style='text-align: center; word-wrap: break-word;'>149.113</td><td style='text-align: center; word-wrap: break-word;'>31.045</td><td style='text-align: center; word-wrap: break-word;'>21.513</td><td style='text-align: center; word-wrap: break-word;'>151.703</td><td style='text-align: center; word-wrap: break-word;'>44.440</td><td style='text-align: center; word-wrap: break-word;'>38.109</td><td style='text-align: center; word-wrap: break-word;'>162.053</td><td style='text-align: center; word-wrap: break-word;'>56.047</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>ACE-Ego-Hand</td><td style='text-align: center; word-wrap: break-word;'>15.423</td><td style='text-align: center; word-wrap: break-word;'>35.165</td><td style='text-align: center; word-wrap: break-word;'>16.783</td><td style='text-align: center; word-wrap: break-word;'>12.703</td><td style='text-align: center; word-wrap: break-word;'>38.649</td><td style='text-align: center; word-wrap: break-word;'>17.273</td><td style='text-align: center; word-wrap: break-word;'>7.449</td><td style='text-align: center; word-wrap: break-word;'>13.131</td><td style='text-align: center; word-wrap: break-word;'>8.272</td></tr></table>

![Image](images/03989c5cbdf74f0f3982924fa02887e14fa75ea55e7825a52ff11373e68092ab.jpg)

<div style="text-align: center;">Figure S1. Accuracy against speed. We plot MPJPE-p averaged over ARCTIC, HOT3D, and HOI4D against end-to-end clip throughput on one A100, excluding model loading, video decoding, and rendering. Both ACE-Ego-Hand configurations run at the same speed.</div>

#### C.4. Inference Throughput

Figure S1 places every method on the accuracy-speed plane. This section states the protocol behind that figure. All methods are timed on the same 81-frame ARCTIC clip on one A100, from decoded frames in host memory to MANO parameters back on the host, excluding model loading, warmup, rendering, and file I/O. We report the median of at least three passes after a full-clip warmup, and we count the hands actually produced per frame so that missed detections cannot masquerade as speed. Several points depart from that protocol. HaMeR and WildHands warm up on five frames instead of a full clip. The timing for HaMeR is the mean of two passes rather than the median of at least three, and its frame decode sits inside the timed region rather than outside it. That decode costs a few tenths of a second out of 49.0 s, so its throughput is understated by well under one percent. Hamba is timed without the compiled selective-scan kernel, which is absent from our environment, so the model falls back to an uncompiled scan. That 1.5 fps is therefore a lower bound on what the method can reach. HaWoR is timed in the SLAM-free and infilling-free configuration that produced its accuracy row (Appendix A), so its figure does not describe its full published pipeline, which would be slower. The timing for EgoForce is inherited from a separate benchmarking run rather than measured with our own harness, and was taken in fp16 on a matched egocentric 81-frame clip rather than on our protocol clip. Both departures favor EgoForce. For ViDiHand we use its published accuracy row. Each baseline is timed in the configuration that produced its accuracy row in Table 1. ACE-Ego-Hand runs at 63.1 fps (63.3 fps in the K-free configuration) in a single deterministic pass whose runtime is 72% VAE encode, against 1.91 fps for ViDiHand, a 33× gap. The five crop-based baselines that share a ViTDet-H+ViTPose+ front end cluster at 1.5–1.7 fps, so their cost is set by the detector, not by the reconstruction network. The fastest crop-based method, WiLoR, reaches 15.5 fps.

### D. Analysis of Failure Modes and Design Choices

#### D.1. Errors Toward the Image Border

Without the camera fit of Sec. 3.3, K-free bearings would be read directly from the predicted ray field, which is defined only over the token grid and becomes unreliable near the image border. This section measures that failure di-

---

<div style="text-align: center;">Table S5. Error against distance from the image center. In-view hand-frames binned by normalized radius  r, with the share falling in each bin. The columns report the unpenalized in-view stratum, not the matched-detection numbers of Appendix C, so share-weighting a column reproduces the in-view column of Table S4. The two configurations agree to within a millimeter at the center and diverge in translation toward the border, which isolates the cost of sampling bearings from a predicted ray field.</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td rowspan="2">Radius  r</td><td rowspan="2">share</td><td colspan="2">$\overline{\text{ACE}}$-Ego-Hand</td><td colspan="2">w/o camera fit</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>MPJPE</td><td style='text-align: center; word-wrap: break-word;'>CT</td><td style='text-align: center; word-wrap: break-word;'>MPJPE</td><td style='text-align: center; word-wrap: break-word;'>CT</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>ARCTIC</td><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'></td></tr><tr><td style='text-align: center; word-wrap: break-word;'>0.00–0.25</td><td style='text-align: center; word-wrap: break-word;'>26.1%</td><td style='text-align: center; word-wrap: break-word;'>15.81</td><td style='text-align: center; word-wrap: break-word;'>0.019</td><td style='text-align: center; word-wrap: break-word;'>15.49</td><td style='text-align: center; word-wrap: break-word;'>0.019</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>0.25–0.50</td><td style='text-align: center; word-wrap: break-word;'>57.7%</td><td style='text-align: center; word-wrap: break-word;'>14.96</td><td style='text-align: center; word-wrap: break-word;'>0.021</td><td style='text-align: center; word-wrap: break-word;'>14.97</td><td style='text-align: center; word-wrap: break-word;'>0.020</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>0.50–0.75</td><td style='text-align: center; word-wrap: break-word;'>14.7%</td><td style='text-align: center; word-wrap: break-word;'>16.15</td><td style='text-align: center; word-wrap: break-word;'>0.028</td><td style='text-align: center; word-wrap: break-word;'>16.19</td><td style='text-align: center; word-wrap: break-word;'>0.039</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>&gt; 0.75</td><td style='text-align: center; word-wrap: break-word;'>1.5%</td><td style='text-align: center; word-wrap: break-word;'>19.17</td><td style='text-align: center; word-wrap: break-word;'>0.048</td><td style='text-align: center; word-wrap: break-word;'>19.00</td><td style='text-align: center; word-wrap: break-word;'>0.165</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>HOT3D</td><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'></td></tr><tr><td style='text-align: center; word-wrap: break-word;'>0.00–0.25</td><td style='text-align: center; word-wrap: break-word;'>7.8%</td><td style='text-align: center; word-wrap: break-word;'>12.62</td><td style='text-align: center; word-wrap: break-word;'>0.022</td><td style='text-align: center; word-wrap: break-word;'>12.97</td><td style='text-align: center; word-wrap: break-word;'>0.023</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>0.25–0.50</td><td style='text-align: center; word-wrap: break-word;'>59.3%</td><td style='text-align: center; word-wrap: break-word;'>12.28</td><td style='text-align: center; word-wrap: break-word;'>0.024</td><td style='text-align: center; word-wrap: break-word;'>12.57</td><td style='text-align: center; word-wrap: break-word;'>0.025</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>0.50–0.75</td><td style='text-align: center; word-wrap: break-word;'>28.5%</td><td style='text-align: center; word-wrap: break-word;'>12.68</td><td style='text-align: center; word-wrap: break-word;'>0.025</td><td style='text-align: center; word-wrap: break-word;'>13.12</td><td style='text-align: center; word-wrap: break-word;'>0.058</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>&gt; 0.75</td><td style='text-align: center; word-wrap: break-word;'>4.4%</td><td style='text-align: center; word-wrap: break-word;'>18.80</td><td style='text-align: center; word-wrap: break-word;'>0.035</td><td style='text-align: center; word-wrap: break-word;'>18.19</td><td style='text-align: center; word-wrap: break-word;'>0.190</td></tr></table>

rectly. For every in-view hand-frame we compute the normalized distance of its projected center from the principal point,  $r = \|(u - c_x, v - c_y)\| / (\text{diag}/2)$, and bin the errors by r (Table S5). The standard configuration and the variant without the fit are evaluated on identical dumps.

At the image center the two configurations agree to within a millimeter, with translation errors of 0.019 and 0.019 m on ARCTIC and 0.022 and 0.023 m on HOT3D, the standard configuration first. The gap then opens outward, negligibly in the second bin, where the variant without the fit is in fact marginally the tighter of the two on ARCTIC, and sharply in the outer two bins. In the outermost bin its translation error is 0.165 against 0.048 m on ARCTIC, a factor of 3.4, and 0.190 against 0.035 m on HOT3D, a factor of 5.4. Wrist-aligned pose is nearly identical between the two configurations in every bin, so the border effect is specific to metric placement rather than to hand pose. This is what the ray-field explanation predicts, because a cell near the frame edge has few neighbors to constrain its direction and the solve inherits that uncertainty as an in-plane shift. HOT3D shows the stronger effect, consistent with its wider field of view placing more hands in the outer bins.

The camera fit removes this failure mode: on an in-view/out-of-sight stratification of the translation error, it reduces the out-of-sight error from 0.55 to 0.10 m on ARCTIC, from 0.52 to 0.11 m on HOT3D, and from 0.45 to 0.06 m on OakInk2, and HOT3D false negatives fall from 2,491 to 74.

#### D.2. Decoded Sequence Length

Long-horizon decoding is a property of the ACE-Ego-Hand architecture rather than a separate mode. This section measures what that costs. All results in the main text are scored on the 81-frame segments shared with every baseline. Those 81 frames occupy 21 latent frames, and a segment whose first frame does not fall on a latent boundary is covered by one further latent frame, so the decoder is run over 21 or 22 latent frames depending on where the segment starts. Nothing in the model requires that length: the temporal attention uses rotary relative position encoding, so no absolute clip length is baked into the weights and the same checkpoint decodes any length in one pass. Figure S2 sweeps the length decoded in one pass on the identical 291 ARCTIC segments. Only the decoded window changes. The scored segments are fixed 81-frame slices of the 34 ARCTIC test recordings, which run from 594 to 1,087 frames. Each window is anchored at the segment it is scored on and clamped to end no later than the recording. The added context is therefore neighboring frames of the same recording, mostly the frames that follow the segment.

Accuracy is flat well past the training window: at 125 frames, more than 50% longer than that window, MPJPEp is marginally better than at the segment-length setting. Beyond that accuracy degrades gently, reaching 17.99 mm when each recording is decoded in a single pass, 7 to 13× the training window. Jitter, by contrast, falls from 2.700 to 2.585 mm/frame². The temporal resolution does not change with the pass length, because the VAE always carries four frames per latent. A longer window therefore adds context for the temporal attention to average over without adding any degree of freedom between latent samples. The net effect is 4% additional smoothing at a cost of 18% in MPJPE-p. Even so, ACE-Ego-Hand stays 17% below the windowed 21.668 mm of ViDiHand at that setting, so whole-recording decoding remains usable when the application needs a single pass. The main text reports the segment-length setting because it is the like-for-like comparison with the baselines.

#### D.3. Tap Depth

The tap sits at block 15 of the 30-block DiT, a choice fixed before the model reported in this paper was trained. This ablation retrains four models that differ only in tap depth and scores all four, to measure what tap depth costs and buys. The four runs share one reduced training configuration, 20k steps at effective batch 16 over a mixture of ARCTIC and HOT3D, rather than the seven training sets and the larger batch of the main recipe. They also decode translation by inverse projection rather than by the mixed-PnP solver of the main model. All four are scored at step 20k on the same 291-segment ARCTIC test split, so absolute errors are comparable within the sweep but not with Table 1. Table S6 reports the sweep.

Accuracy has saturated by the middle of the stack. Tapping at block 10, which runs 11 of the 30 blocks, costs

---

![Image](images/334709e5a23cbe39611d873faa622b34419337ed14a7d7fc3abec853829313c0.jpg)

<div style="text-align: center;">Figure S2. Effect of decoded sequence length. We plot ACE-Ego-Hand accuracy (MPJPE-p, left axis) and smoothness (Jitter, right axis) against the length decoded in one pass, MPJPE-p in mm and Jitter in mm/frame $^{2}$. The same 291 ARCTIC segments are scored at every length. The dotted vertical line marks the 81-frame training window, the dashed line marks 81-frame-window MPJPE-p, and the star marks every recording decoded end to end in one pass. The star's abscissa is nominal, at 645 frames: the recordings themselves run 594 to 1,087 frames, so the star understates that horizon.</div>

<div style="text-align: center;">Table S6. Tap depth. Four runs differing only in which DiT block is read, scored at step 20k on the same 291 ARCTIC segments under the reduced sweep recipe. Block indices are zero-based and a tap returns the output of its block, so a tap at block i runs the first  $i+1$ of the 30 blocks. Each row label gives that executed count. Lower is better throughout.</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'>Tap (blocks run)</td><td style='text-align: center; word-wrap: break-word;'>MPJPE-p</td><td style='text-align: center; word-wrap: break-word;'>PA-p</td><td style='text-align: center; word-wrap: break-word;'>$EPE_{2D}$-p</td><td style='text-align: center; word-wrap: break-word;'>GO-p</td><td style='text-align: center; word-wrap: break-word;'>CT-p</td><td style='text-align: center; word-wrap: break-word;'>Jitter</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Block 10 (11/30)</td><td style='text-align: center; word-wrap: break-word;'>17.77</td><td style='text-align: center; word-wrap: break-word;'>8.70</td><td style='text-align: center; word-wrap: break-word;'>11.63</td><td style='text-align: center; word-wrap: break-word;'>12.58</td><td style='text-align: center; word-wrap: break-word;'>0.030</td><td style='text-align: center; word-wrap: break-word;'>2.74</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Block 15 (16/30)</td><td style='text-align: center; word-wrap: break-word;'>16.10</td><td style='text-align: center; word-wrap: break-word;'>7.76</td><td style='text-align: center; word-wrap: break-word;'>9.99</td><td style='text-align: center; word-wrap: break-word;'>11.98</td><td style='text-align: center; word-wrap: break-word;'>0.028</td><td style='text-align: center; word-wrap: break-word;'>2.66</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Block 20 (21/30)</td><td style='text-align: center; word-wrap: break-word;'>15.93</td><td style='text-align: center; word-wrap: break-word;'>7.68</td><td style='text-align: center; word-wrap: break-word;'>9.87</td><td style='text-align: center; word-wrap: break-word;'>12.15</td><td style='text-align: center; word-wrap: break-word;'>0.021</td><td style='text-align: center; word-wrap: break-word;'>2.59</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Block 24 (25/30)</td><td style='text-align: center; word-wrap: break-word;'>15.95</td><td style='text-align: center; word-wrap: break-word;'>7.70</td><td style='text-align: center; word-wrap: break-word;'>9.93</td><td style='text-align: center; word-wrap: break-word;'>12.02</td><td style='text-align: center; word-wrap: break-word;'>0.023</td><td style='text-align: center; word-wrap: break-word;'>2.60</td></tr></table>

1.67 mm of MPJPE-p and is the only setting that is clearly worse, degrading every column at once. Descending past block 15 buys at most 0.17 mm while running 21 and 25 of the 30 blocks instead of 16, and the two deeper taps do not order consistently: block 20 is marginally better than block 24 on five of the six metrics but loses on global orientation. We therefore read the residual difference as run-to-run noise rather than as a trend. Block 15 therefore captures nearly all of the available accuracy at roughly half the forward cost, confirming the choice the main model was trained with.

#### D.4. Clean versus Noised Latents

ACE-Ego-Hand reads the backbone at $\sigma = 0$, where the input latent is the clean VAE code and no noise is added. The main text argues for this on determinism grounds. The measurement that supports that argument is reported here. We

<div style="text-align: center;">Table S7. Clean versus noised latent readout. Two runs that differ in the noise level  $\sigma$ at which the backbone is read, scored by the shared evaluator on the same 291 ARCTIC segments at the same 20k-step checkpoint. Lower is better throughout.</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'>Latent readout</td><td style='text-align: center; word-wrap: break-word;'>MPJPE-p</td><td style='text-align: center; word-wrap: break-word;'>PA-p</td><td style='text-align: center; word-wrap: break-word;'>$EPE_{2D}$-p</td><td style='text-align: center; word-wrap: break-word;'>GO-p</td><td style='text-align: center; word-wrap: break-word;'>CT-p</td><td style='text-align: center; word-wrap: break-word;'>Jitter</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>clean, $\sigma=0$</td><td style='text-align: center; word-wrap: break-word;'>15.26</td><td style='text-align: center; word-wrap: break-word;'>7.47</td><td style='text-align: center; word-wrap: break-word;'>9.18</td><td style='text-align: center; word-wrap: break-word;'>11.81</td><td style='text-align: center; word-wrap: break-word;'>0.021</td><td style='text-align: center; word-wrap: break-word;'>2.70</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>noised, $\sigma=0.5$</td><td style='text-align: center; word-wrap: break-word;'>17.66</td><td style='text-align: center; word-wrap: break-word;'>7.99</td><td style='text-align: center; word-wrap: break-word;'>9.78</td><td style='text-align: center; word-wrap: break-word;'>12.62</td><td style='text-align: center; word-wrap: break-word;'>0.034</td><td style='text-align: center; word-wrap: break-word;'>2.78</td></tr></table>

retrain with $\sigma = 0.5$ in place of $\sigma = 0$, so the encoder reads a half-noised latent at the timestep bound to that noise level. The two runs share the training data, the schedule, and the evaluator, are scored on the same 291 ARCTIC segments, and are read at the same 20k-step checkpoint (Table S7).

Reading the clean latent is better on every metric. MPJPE-p improves by 2.40 mm (17.66 to 15.26) and translation error by 39% (0.0343 to 0.0208 m), while the two runs sit within 0.08 mm/frame $^2$ of each other on Jitter. The gain is concentrated in metric placement rather than articulation, since PA-p moves by only 0.52 mm. Injecting noise degrades geometry that the deterministic read preserves, and the sample diversity it introduces goes unused because no generative pass is run. The comparison covers two noise levels and one seed per level, so it shows that $\sigma=0$ is the better of the two settings, not that error varies monotonically with $\sigma$. The gap is not an artifact of the checkpoint we read. Across the 31 validation checkpoints logged between step 5k and step 20k, the noised run is the worse of the two on 29, by a mean of 1.2 mm of wrist-aligned validation error.

### E. Qualitative Results and Application

#### E.1. Extended Qualitative Comparison

Figures S3 and S4 extend the comparison of Figure 4. Figure S3 covers the two main benchmarks, ARCTIC and HOT3D. Figure S4 covers H2O and OakInk2 together with two in-the-wild egocentric datasets, HoloAssist [^35] and Xperience-10M [^27]. Neither carries ground-truth hand annotation here, and neither enters any training mixture or evaluation table in this paper. We include them to place the comparison on footage that none of the methods was tuned on. Each block pairs a camera-space overlay row with the corresponding 3D reconstruction, and the columns hold the ground truth, where it exists, followed by ACE-EgoHand and six baselines. Every method predicts in the camera frame. For the datasets that provide camera poses, the 3D rows place those camera-frame predictions in a shared world frame, following the same convention as Figure 4. The clips are chosen to contain OOS intervals, because that is where the methods separate most visibly. Single-frame baselines drop the hand entirely once it leaves the frame and re-acquire it in a different pose, while world-space baselines keep producing a hand but drift away from the ground-truth.

---

trajectory. ACE-Ego-Hand carries the hand through the gap and re-aligns it with the visible evidence on re-entry.

#### E.2. Qualitative Results Without Ground Truth

Figure S5 shows ACE-Ego-Hand at eight moments drawn from two egocentric recordings for which no ground-truth annotation exists, so nothing in the pipeline can be tuned to them. Each row holds the source frame, the camera-space overlay of the prediction, and two renders of the same prediction, one from a fixed third-person viewpoint and one from the head-camera pose. ACE-Ego-Hand predicts hands in the camera frame and does not estimate camera motion, and these clips carry no camera annotation, so the two rendered views show the recovered hands themselves rather than a globally referenced trajectory. Without ground truth to overlay, what remains verifiable is whether the two hands stay metrically separated, keep a consistent scale as the head turns, and continue along a plausible path while they are outside the frame. The figure shows these properties, which the benchmark tables measure numerically, on footage that carries no annotation.

#### E.3. Retargeting to a Dexterous Hand

We show the step before policy learning, qualitatively: replaying the recovered bimanual trajectories on a dexterous Inspire Hand, shown in a neutral gray color scheme for legibility, with each panel rendered under the intrinsics of the source clip so robot and video share one projection. Finger flexion is read off the predicted 3D joints rather than the MANO articulation parameters. Figure S6 places the source frame, the recovered hand mesh, and the retargeted robot hand side by side for one clip from each of the four in-domain datasets, so the two conversion steps can be inspected separately. Figure S7 then follows a single recording over two minutes, sampled at four instants, to show that the conversion still holds at the end of a horizon far longer than the 81-frame training window.

A demonstration requires the three properties ACE-Ego-Hand provides: metric placement in the camera frame, cliplevel shape, and OOS coherence where per-frame methods drop frames. At 63.1 fps, the trajectory recovery that feeds this conversion is not the collection bottleneck. Whether such trajectories improve downstream policy learning remains for future work.

### F. Limitations, Failure Cases, and Data Use

#### F.1. Limitations and Failure Cases

Five limitations follow from the analyses above. Where possible, we state the concrete failure signature in the units a user would see.

The K-free configuration trades a small pose margin for calibration freedom. Its camera fit keeps detection and metric placement at the level of the calibrated solve (Table 1), and the residual cost sits in wrist-aligned pose: MPJPE-p runs 0.6–1.4 mm above the standard configuration on the in-domain benchmarks, consistent with the $\mathcal{L}_{fit}$ term competing for shared capacity; annealing its weight is a natural next step. The fit assumes a pinhole camera, so fisheye clips fall back to reading the ray field directly, and a radial bearing model remains future work. The failure signature is a slightly looser articulation, not a lost detection.

Long single-pass decoding trades accuracy for smoothness. Decoding a whole recording in one pass costs 18% in MPJPE-p relative to the 81-frame setting (15.26 to 17.99 mm), so a single long pass is an operating point rather than a free improvement. The failure signature is a trajectory that is smoother but less tightly fitted to the ground truth.

Ground truth and splits are uneven across the five benchmarks. Only ARCTIC and H2O are subject-disjoint. HOT3D is split at the recording level and OakInk2 at the sequence level, so subject overlap between train and test cannot be excluded there. As noted in Appendix A, HOI4D and H2O rely partially on pseudo-ground-truth annotation, which flatters any method that resembles the per-frame estimator behind those labels. On H2O, our own model was trained on that label distribution. The H2O and OakInk2 blocks of Table S3 also compare an in-domain ACE-Ego-Hand against baselines trained elsewhere, unlike the like-for-like ARCTIC and HOT3D comparison of Table 1.

The out-of-sight gain is wrist-aligned and does not certify placement. Both MPJPE $^{00S}$ and MPJPE $^{+00S}$ align the wrist before measuring, so the margins of Table S4 certify that the articulation and the global orientation of an out-of-sight hand stay close to the ground truth, and nothing more. Where the hand actually sits while it is out of sight depends on a regressed depth and on a bearing that no image evidence constrains, and on the solver fallback of Appendix B once too few joint anchors survive inside the frame. We therefore read the out-of-sight result as temporal coherence through the gap rather than as metric localization during the gap, and the failure signature is a plausibly articulated hand on a trajectory that has drifted in depth. No table in this paper measures absolute out-of-sight placement, because none of the five benchmarks scores it.

The method is offline by construction. What makes ACE-Ego-Hand offline is its dependence on clip-level context, not the direction of temporal attention in the decoder. The backbone attends over the whole clip in one pass, so latency is bounded below by the length of the clip rather than by the 63.1 fps throughput. ACE-Ego-Hand is a tool for curating manipulation data from recorded video, not a

---

![Image](images/32be4b3f2ca88cafd8fea747a61f16d48354198cf7a2d0b6d887b73570ba3aae.jpg)

<div style="text-align: center;">Figure S3. Extended qualitative comparison on ARCTIC and HOT3D. Columns: ground truth, ACE-Ego-Hand, and six baselines (ViDiHand, EgoForce, Dyn-HaMR, HaWoR, WiLoR, HaMeR). Each dataset block shows a camera-space overlay row above the corresponding 3D reconstruction, which uses the ground-truth camera poses these two datasets provide.</div>

---

![Image](images/185e8bff1d7d84745d6cc83e1e79cce30bae38094963737f94fe957629d070b3.jpg)

<div style="text-align: center;">Figure S4. Extended qualitative comparison on H2O, OakInk2, and in-the-wild HoloAssist and Xperience-10M. Same columns and same layout as Figure S3. H2O and OakInk2 supply ground-truth camera poses for the 3D row, while the two in-the-wild sources carry no annotation and are shown for qualitative inspection only.</div>

---

![Image](images/a1aaced902862d301875308b8bc0bdb14886649a3c3a76bd0ffd9a4b6016aa12.jpg)

<div style="text-align: center;">Figure S5. ACE-Ego-Hand qualitative results on in-the-wild videos without ground truth. Columns: source RGB, predicted hands overlaid on the source video, and the same prediction rendered from a fixed third-person viewpoint and from the head-camera pose.</div>

---

![Image](images/ecb55ced0c02f663f1d12c516aaab8c9082b44abe0ed09873b69e2dc74cf9772.jpg)

<div style="text-align: center;">Figure S6. From egocentric video to a dexterous hand, step by step. One clip from each of ARCTIC, HOT3D, H2O, and OakInk2. Columns: source RGB, the hand mesh recovered by ACE-Ego-Hand, and the trajectory retargeted to an Inspire Hand. Finger flexion is mapped from the predicted 3D joints rather than from the MANO articulation parameters.</div>

---

![Image](images/0ea1eef29aede69cc64cb069ca3de1c05375918efd8e3b0a2a5ac87a074b5737.jpg)

<div style="text-align: center;">Figure S7. Retargeting over a two-minute recording. A single recording sampled at four instants, with the input frame above and the retargeted dexterous hand below. The horizon is roughly 45 times the 81-frame training window, and both hands are still tracked and correctly identified at the last of the four instants.</div>

closed-loop controller. Three directions follow from that boundary. Distilling the encoder into a causal variant would trade clip-level context for streaming operation and would also quantify how much of the accuracy rests on future frames. Bounding the pass length while keeping identity across passes would remove the dependence of latency on clip length. Carrying hand identity and shape over recordings of several minutes would extend the horizon beyond what one pass now holds.

#### F.2. Ethics and Data Use

All five benchmarks and the three auxiliary training sources are public research datasets, as are the two in-the-wild datasets in Figure S4. We obtained each through its official channel and use it under the terms of that release. For Xperience-10M those terms are controlled access for non-commercial research. We collected no new human-subject data and we redistribute no dataset. Egocentric recordings can contain identifiable people and private environments, so we use them only to fit hand geometry. The model takes video, together with camera calibration in the standard configuration, and outputs MANO pose, shape, and camera-frame translation. The model neither receives nor produces identity, demographic, or biometric-identification attributes, and is not a person recognizer. Frames shown in figures are taken from those public releases and are reproduced under the same terms, with one exception. The frames in Figure S5 come from a licensed egocentric dataset outside those benchmarks, used for qualitative illustration only. That dataset enters no training mixture and no evaluation table. The intended downstream use is imitation learning from recordings whose subjects consented to being recorded. Running the model on third-party video without that consent is a misuse we do not endorse.