# ESTHER: Egocentric Stereo Hand Estimation and Reconstruction in the Wild

Hongyu Ma, Hairong Qu, Shiqi Zhao, Yongsong Yang, Peng Yin City University of Hong Kong

## Abstract

Human dexterity is guided by two eyes watching two hands: binocular vision supplies the metric 3D structure needed for precise manipulation. Egocentric stereo is therefore the natural perceptual interface for robots, AR, and VR. Yet egocentric stereo hand reconstruction still lacks an end-to-end model and an in-the-wild benchmark. We propose ESTHER, a model whose stereo geometry, temporal reasoning, and output representation are designed for wearable egocentric stereo. It is trained on pseudo labels from a calibrated labeling pipeline and in turn assembles our benchmark ESTHER3D, an egocentric stereo hand dataset pairing a large in-the-wild training set of model labels with a motion capture test set of true metric ground truth. Experiments show state-of-the-art accuracy, superior external generalization across stereo rigs and modalities with minimal fine-tuning, and robust metric reconstruction under missing views, monocular input, dropped frames, and lighting and motion blur in real egocentric capture. Project page.

### 1. Introduction

An embodied agent that sees the world through head-mounted stereo perceives its own hands much as people do: two eyes watch two hands, and binocular disparity carries the metric 3D structure that precise manipulation depends on. This makes egocentric stereo the natural signal for hand perception in robotics, AR, and VR: it provides the metric constraints that monocular RGB lacks. We study this setting as egocentric stereo 3D hand reconstruction: given short-baseline head-mounted stereo videos, the system should recover MANO hand pose and shape, and global wrist translation in metric 3D. Yet despite its practical importance, egocentric stereo hand reconstruction still lacks an end-to-end model and an in-the-wild benchmark. We identify two primary obstacles: the lack of scalable training data with reliable metric hand labels, and the mismatch between existing hand pose architectures and wearable stereo geometry.

Accurate 3D hand annotation for in-the-wild egocentric capture is hard: the label itself must be recovered from partial, moving, and frequently occluded observations. Ex-

![Image](images/5f3999d879ebc040d09d8763187cac3e4ea0e390e165a6be3f30235d18004a92.jpg)

stereo 2D projections + metric 3D mesh over time

<div style="text-align: center;">Figure 1. ESTHER at a glance. Letters: egocentric stereo frames from the six ESTHER3D industries. Bottom: ESTHER outputs stereo 2D projections and metric 3D meshes over time.</div>

listing egocentric hand datasets, including H2O [^17], AssemblyHands [^27], OakInk [^43], ARCTIC [^6], HOI4D [^21], HOT3D [^1], obtain labels through additional sensing or offline reconstruction; HOT3D is closest in viewpoint, but its stereo is monochrome and its ground truth comes from professional motion capture rather than scalable visual labeling of raw RGB stereo. How to scale metric hand labels from raw in-the-wild head-mounted stereo remains open. We therefore build a calibrated labeling pipeline combining 2D evidence, stereo geometry, and hand priors, validate it against motion capture ground truth, and train a scalable reconstruction model on the pseudo labels the pipeline generates; the resulting benchmark, ESTHER3D, pairs a mocap ground truth test set with these in-the-wild training sequences.

The model bottleneck can be traced through the evolution of hand pose architectures. Monocular mesh methods [^29] [^30] regress plausible hands, but from a single crop they can recover depth and scale only through learned priors rather than geometric measurement, so metric accuracy is not guaranteed. Multiview methods [^15] [^19] [^46] restore scale through calibrated triangulation, but are single-frame and assume at least two cameras see each point. Temporal models add video context in camera coordinates [^35] [^41] or lift monocular video into world space with SLAM [^47] [^48], recovering the camera only up to the scale ambiguity of

---

monocular SLAM and offline optimization. Egocentric stereo sharpens all three weaknesses at once. Triangulation anchors are not guaranteed: blur and occlusion routinely erase the hand from one of the two views, so a model dependent on triangulation loses its metric anchor exactly on the frames that matter. Detection dropout is the norm: the interacting hand is repeatedly missed for entire frames, so architectures that process each frame independently cannot run stably without temporal inpainting. Hand and head motion are coupled: observed image motion mixes articulation with head egomotion and crop reparameterization, which a model without explicit disentanglement absorbs into finger pose. The setting therefore requires a model designed for these failure modes.

We address this setting with ESTHER, built from two encoders and a decoder. A stereo image encoder fuses calibrated stereo evidence into joint tokens; a masked temporal motion encoder reasons over these tokens across time, with camera and crop motion supplied as conditioning tokens; and a query-gated fusion decoder reads both encoders through structured pose, shape, and wrist tokens and outputs MANO pose, shape shared over time, and a direct metric wrist translation.

In summary, our contributions are as follows:

- We propose ESTHER, an end-to-end egocentric stereo hand reconstruction model built from two encoders and a query-gated fusion decoder that fuses calibrated stereo geometry and temporal context to output metric MANO pose, shape, and an absolute wrist, with state-of-the-art accuracy and robustness.

- We introduce ESTHER3D, an in-the-wild egocentric stereo hand benchmark whose test set carries motion capture ground truth and whose training set consists of a large collection of stereo videos labeled by ESTHER, bootstrapped from a calibrated pipeline validated on that ground truth.

- We evaluate under a controlled protocol (shared backbone, data, and losses) on the ESTHER3D test set, on HOT3D, and with missing views and dropped frames, and uncover a binding effect between size and depth: short-baseline stereo guidance lets direct wrist regression internalize a prior tying apparent hand scale to metric depth, giving surprising accuracy when using the monocular fallback and cheap adaptation to different baselines, camera parameters, and even the grayscale modality.

### 2. Related Works

#### 2.1. Egocentric and Embodied Datasets

Egocentric datasets are central to embodied perception: EPIC-KITCHENS [^4] and Ego4D [^8] cover extended daily activities, Ego-Exo4D [^9] pairs egocentric and external views, and Nymeria [^23] adds multimodal wearable motion sensing.

Datasets of interactions between hands and objects such as FPHA [^7], H2O [^17], HOI4D [^21], OakInk [^43], DexYCB [^2], ARCTIC [^6], AssemblyHands [^27], and HOT3D [^1] provide important supervision for hands and interaction. HOT3D is closest, but its wearable stereo is monochrome and its metric labels come from motion capture. ESTHER3D instead targets in-the-wild calibrated short-baseline RGB stereo with metric MANO labels and train/test protocols.

#### 2.2. 3D Hand Pose and Mesh Estimation

Parametric hand reconstruction builds on MANO [^32], supported by datasets such as FreiHAND [^49], HO-3D [^10], InterHand2.6M [^26], DexYCB [^2], ObMan [^13], and ARCTIC [^6]. Monocular models from I2L-MeshNet [^25], Mesh Graphormer [^20], MobRecon [^3], and HandOccNet [^28] to HaMeR [^29] and WiLoR [^30] recover articulation well but must regress scale and metric depth from learned priors alone.

On the temporal side, HTT [^41], HandFormer [^35], and UmeTrack [^11] track hands in head-mounted settings, and HaWoR [^48] and Dyn-HaMR [^47] recover motion in world coordinates from monocular video via monocular SLAM and offline optimization, with ambiguous scale. ESTHER instead obtains metric scale feedforward from calibrated stereo and disentangles hand from head motion with conditioning tokens and masked temporal training.

#### 2.3. Multiview, Stereo, and Depth-Based Reconstruction

Multiview pose estimation has explored learnable triangulation, epipolar attention, volumetric fusion, and cross-view transformers [^14] [^15] [^19] [^37] [^39]; POEM-v2 [^46] studies multiview hand reconstruction with point embeddings. Closer to us, egocentric stereo has been used for absolute hand pose [^34] and stereo keypoints [^18], and event cameras for stereo hand pose [^38] and MANO mesh reconstruction [^12]. These estimate sparse keypoints, use non-RGB sensing, or are single-frame; we target temporally consistent metric MANO reconstruction from wearable RGB stereo, where neither visibility in both views nor detection in each frame can be assumed.

### 3. Method

ESTHER is built together with its supervision pipeline (Figure 2). The calibrated teacher labels an initial 10 hours of in-the-wild stereo using 2D evidence and a DPoser-X prior [^22]. ESTHER predictions then initialize teacher optimization on more samples; filtered labels feed back into training (Section 3.8). The final ESTHER supplies forward predictions for deployment and testing; the separate mocap collection provides metric evaluation.

---

![Image](images/cbe9b9cbe43c5b6374ada4a480037228aa3ac878ae826e43864c84211fc1b30a.jpg)

<div style="text-align: center;">Figure 2. Supervision pipeline and data engine. (a) The teacher turns stereo 2D keypoints and WiLoR predictions from each view into metric pseudo labels via optimization in two stages and a reprojection/human filter. (b) These train ESTHER, which labels in-the-wild stereo to build ESTHER3D.</div>

#### 3.1. Calibrated Teacher Labeling Pipeline

The teacher starts from 2D evidence, which is far easier to obtain reliably than metric 3D hands, and converts it into metric MANO pseudo labels via stereo calibration, hand priors, and optimization. We design a simple head-mounted RGB stereo rig (camera intrinsics and extrinsics in Appendix F); the formulation is not tied to it, and our experiments show the model adapts to different capture setups with only minimal fine-tuning.

An ensemble of 2D detectors (YOLO11-pose [^16] and ViTPose [^42]) provides keypoints on both views, triangulated through the calibrated rig and filtered by depth range and plausibility of bone lengths. Optimization is initialized from the triangulated wrist and monocular MANO estimates from each view from WiLoR [^30], and proceeds in two stages under the DPoser-X pose prior. Shape lock: on the 64 most confident frames, shape, pose, and wrist are jointly optimized against the filtered stereo observations, and the shapes estimated for individual frames are averaged into one fixed $\beta$ per hand. Pose and motion optimization: with the shape locked, pose and wrist are reoptimized per frame, then refined by 16-frame sliding-window motion optimization with velocity limits and an motion prior pretrained on AMASS [^5] [^24] [^31]. The detection filtering thresholds, initialization, full fitting objective, and stage details are given in Appendix A.

#### 3.2. Ground Truth Validation

Before scaling up, we verify that the automatic labels can serve as supervision on the full 10-hour motion capture test set (Appendix H). The teacher runs on it without access to the labels and reaches 15.9 mm MPJPE, 9.3 mm PA-MPJPE, and 19.1 mm wrist error zero-shot (Figure 4). This approaches the roughly 10 mm uncertainty in solving glove measurements of the ground truth itself and is on par with the quality of annotations obtained by fitting of widely used hand benchmarks [^10] [^49], meeting the accuracy required of pseudo labels; the full measurement chain is in Appendix I. The validation also exposes failure modes (2D misdetections under heavy occlusion), handled by confidence thresholds and manual quality control.

#### 3.3. In-the-Wild Data and Filtering

The validated teacher is applied to a subset of the raw in-the-wild collection, 10 hours drawn evenly from the six categories (collection statistics in the ESTHER3D section). Labels pass a filter with three stages: a coarse geometric prescreen on bone length and depth range between hand and camera, an automatic 2D check that reprojects each fitted hand into both views and discards hand frames disagreeing with the detections, and human review for the residual failures these cannot catch (wrong hand identity, implausible articulation). Qualified data trains ESTHER; its predictions initialize teacher refinement and the same filtering on more data (ESTHER3D Benchmark section).

#### 3.4. Model Overview

ESTHER starts from the standard pipeline of hand detection and cropping [^3] [^29] [^30]: a detector produces left/right crops with their virtual crop camera calibration, and the network consumes only these. Training uses right hands: left hands are mirrored with views swapped and predictions unmirrored,

---

![Image](images/6297375130304149b04603106a03ea32823965e35b5cadb99780ed623aa169d8.jpg)

<div style="text-align: center;">(a) ESTHER</div>

<div style="text-align: center;">(b) Stereo Transformer</div>

<div style="text-align: center;">(c) Temporal Transformer</div>

<div style="text-align: center;">Figure 3. ESTHER. (a) The stereo image encoder and masked temporal motion encoder feed a query-gated fusion decoder that outputs MANO with a metric wrist. (b, c) Stereo and temporal transformer blocks.</div>

![Image](images/8db77e1803639658c0fb6ee8ae507a639a1baff7f42a2ce1126369fa4da19081.jpg)

<div style="text-align: center;">Figure 4. Teacher pseudo labels vs. motion capture ground truth as MANO meshes projected into both stereo views (two ESTHER3D test scenes; the GT mesh is MANO fitted to the mocap joints).</div>

following WiLoR [^30], halving the pose space to learn.

The model consists of two encoders and a decoder (Figure 3):

 $$ \begin{aligned}\boldsymbol{F}&=E_{img}(\boldsymbol{I}^{l},\boldsymbol{I}^{r},\boldsymbol{\ell}^{l},\boldsymbol{\ell}^{r}),\\\boldsymbol{Z},\mathbf{z}^{root},\hat{\boldsymbol{o}}&=E_{mot}(\boldsymbol{F},\mathbf{f},\mathbf{r}),\\\hat{\boldsymbol{\Theta}}&=D_{fuse}(\boldsymbol{F},\boldsymbol{Z},\mathbf{z}^{root}),\end{aligned} $$

where, per hand, $\hat{\Theta} = (\hat{\theta}, \beta, \hat{\tau})$ collects the MANO pose $(16 \times 3\ \text{axis-angle})$, the 10-dimensional MANO shape, and the 3-dimensional global wrist translation. $I^l$, $I^r$ are cropped stereo images and $\ell^l$, $\ell^r$ the Plucker rays of their patches, computed from the crop camera calibration. At each frame, the stereo image encoder $E_{\text{img}}$ fuses the two views into 21 joint tokens F. The temporal motion encoder $E_{\text{mot}}$, conditioned on a camera motion flow token f and a learned root token $\mathbf{r}$, outputs a motion feature Z, a root feature $\mathbf{z}^{root}$, and hand visibility probabilities $\hat{o}$ that gates, at deployment, whether the decoded hand is emitted at all. The fusion decoder reads all sources with structured queries. During training the image encoder additionally emits 2D keypoints, visibility, and triangulated anchors as guidance, and the motion encoder is trained on randomly masked sequences; heads and losses are detailed in Appendix B.

#### 3.5. Stereo Image Encoder

The stereo image encoder pairs a frozen DINOv3-L image encoder [^36] with a stereo transformer (Figure 3b). It encodes the 16-frame clip together but injects no temporal information, fusing the two views purely through complementary image evidence; all temporal modeling is left to the motion encoder. Patch tokens of each view are augmented with embeddings of their calibrated Plucker rays, situating independently detected crops in the common metric geometry, and pass through alternating intra-view and cross-view attention, so spatial context within each view and cross-view stereo matching refine each other layer by layer; 21 joint queries per view then read from these fused tokens and are aggregated into the shared feature F. During training one entire view is randomly blanked and replaced by a learned view mask token, so the fusion explicitly survives a missing camera. Lightweight heads (heatmaps, 2D keypoints, visibility, depth bins, a differentiable triangulation) guide the encoder toward explicit stereo geometry. The DINOv3-L backbone stays frozen throughout; only the stereo transformer and its heads are trained, so the encoder inherits strong pretrained features while keeping the trainable parameter count small (51M of 355M total).

#### 3.6. Masked Temporal Motion Encoder

The motion encoder provides the temporal reasoning that egocentric capture demands. It feeds three inputs directly to a temporal transformer (Figure 3c): the fusion image feature F (with joint and time embeddings), the camera motion flow token f, and a learned root token $\mathbf{r}$ that aggregates global hand motion into the root feature $\mathbf{z}^{root}$.

Detection dropout and occlusion. Tokens are randomly masked per joint, per frame, and over contiguous spans

---

(anchor frames kept), each masked slot filled by a learned motion mask token (distinct from the stereo view mask token), and auxiliary heads reconstruct joints, pose, and root velocity there: the encoder learns to inpaint exactly the corruption of in-the-wild capture. A head predicts framewise hand visibility, using blank frames as negatives.

Coupling of hand and head motion. We compute dense classical optical flow online from consecutive RGB hand crops, including both hand and background pixels. The previous crop is warped into the current crop coordinates using their calibrated intrinsics to compensate changes in crop position and scale. A fast estimator that requires no training, computes flow on this compensated pair. Median pooling on an $8 \times 8$ grid gives 128 flow values; normalization by the 224-pixel crop side and concatenation with 64 validity flags form the 192-dimensional flow token (Appendix D). This unmasked input carries both camera and hand motion, providing context when image tokens are masked.

#### 3.7. Query-Gated Fusion Decoder

The fusion decoder lets pose, shape, and wrist read different evidence. As in Figure 3a, $\mathbf{Q}_p$ contains 16 pose queries along the kinematic tree, while $\mathbf{Q}_s$ and $\mathbf{Q}_w$ are the shape and wrist queries. Each layer applies self-attention among these 18 queries, yielding state $\mathbf{h}_i$ for query i, then reads $S_{\text{img}} = F$, $S_{\text{mot}} = Z$, and $S_{\text{root}} = \mathbf{z}^{root}$, $k \in \{\text{img}, \text{mot}, \text{root}\}$, with a softmax gate $\mathbf{g}_i$ for each token:

 $$ \mathbf{q}_{i}^{+}=\mathrm{F F N}\left(\mathbf{h}_{i}+\sum_{k}g_{i}^{k}\mathrm{C r o s s A t t n}(\mathbf{h}_{i},S_{k})\right), $$

where $\mathbf{q}_{i}^{+}$ is the updated token. The gate is predicted from the token state, pooled source summaries, and evidence statistics such as visibility and triangulation spread (Appendix E): the pose tokens can emphasize the image and motion features while the wrist token leans on the root feature when image evidence is ambiguous.

Pose is decoded from $\mathbf{Q}_p$ along the kinematic tree, conditioning each child rotation on its parent. After pooling over time, $\mathbf{Q}_s$ yields one shape per sequence. The final wrist query $\mathbf{Q}_{w}^{\text{out}}$ directly predicts metric wrist translation:

 $$ \begin{array}{r}{\hat{\mathbf{t}}=\mathrm{M L P}(\mathbf{Q}_{w}^{\mathrm{o u t}}).}\end{array} $$

MLP denotes a multilayer perceptron. We deliberately do not anchor the wrist to the triangulated point: an anchored wrist improves MPJPE with both views present in the ablation but requires a reliable stereo anchor. Triangulation instead contributes through its guidance loss and the gating statistics, and the decoded wrist survives monocular fallback. A MANO forward kinematics layer maps $\hat{\Theta}$ to joints and vertices.

#### 3.8. Training and Supervision

Iterative training flywheel. To limit labeling cost, the teacher optimizes an initial 10 hours of in-the-wild video

![Image](images/7ce8053460468396d75f3bef15aa8f8b5caa6d5835244e7f44c6832e9a0e9071.jpg)

<div style="text-align: center;">Figure 5. ESTHER3D across the six industries. Left: in-the-wild training hours (194 h). Right: motion capture test tasks (50 tasks,  $\approx$10 h).</div>

for approximately 2000 steps; filtered outputs train the first ESTHER model. On additional samples, ESTHER forward inference initializes MANO parameters, which the teacher refines with 200 steps of the same optimization. The same filtering (Appendix A) selects teacher outputs to retrain ESTHER. We repeat this cycle. Deployment and testing use forward inference of the final ESTHER model.

We pretrain the stereo image encoder with geometric guidance, then jointly train the full model with forward kinematics losses and intermediate encoder supervision. The image encoder retains 2D and triangulation objectives; the motion encoder reconstructs canonical joints, pose, and root velocity at masked positions and predicts hand visibility, learning temporal structure and inpainting (Appendix B). Losses exclude invalid targets. Motion token masking and image augmentation (noise, erasing, and blanking views and frames, with blank frames as visibility negatives) simulate corruption in the wild. The overall objective is

 $$ \mathcal{L}=\mathcal{L}_{\mathrm{F K}}+\mathcal{L}_{\mathrm{i m g}}+0.5\mathcal{L}_{\mathrm{m o t}}+0.5\mathcal{L}_{\mathrm{o n}}, $$

where $\mathcal{L}_{\mathrm{FK}}$ gathers joint, mesh, pose, shape, and wrist terms, $\mathcal{L}_{\mathrm{img}}$ the image guidance, $\mathcal{L}_{\mathrm{mot}}$ the masked reconstruction, and $\mathcal{L}_{\mathrm{on}}$ the on-screen classification (all terms in Appendix B); no oracle signal is provided at test time. The full model trains in about two days on four RTX 4090 GPUs.

### 4. ESTHER3D Benchmark

ESTHER3D consists of two parts: a large in-the-wild training set carrying metric pseudo labels from ESTHER, and a motion capture test set with true metric ground truth. Every frame carries MANO pose and shape for each hand, absolute metric wrist, calibrated camera parameters, 2D keypoint projections in both views, and hand bounding boxes. The training set comes from a distributed program that collects raw in-the-wild stereo video with our rig across six industries, yielding about 194 hours from 121 collectors.

---

<div style="text-align: center;">Table 1. Main comparison. Errors in mm, jitter in m/s². ETHER3D test = real mocap GT, zero-shot (z.s.) for all methods; HOT3D reported zero-shot and after fine-tuning (f.t.) on 10% of HOT3D. Best per block in bold;  $\downarrow$ means lower is better.</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td colspan="2">Method</td><td style='text-align: center; word-wrap: break-word;'>MPJPE $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>PA-MPJPE $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>Wrist $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>Jitter $\downarrow$</td></tr><tr><td rowspan="5">ESTHER3D</td><td style='text-align: center; word-wrap: break-word;'>Hiera-FTL [^33] [^50]</td><td style='text-align: center; word-wrap: break-word;'>43.4</td><td style='text-align: center; word-wrap: break-word;'>10.0</td><td style='text-align: center; word-wrap: break-word;'>277</td><td style='text-align: center; word-wrap: break-word;'>79.5</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>POEM-v2 [^46]</td><td style='text-align: center; word-wrap: break-word;'>19.4</td><td style='text-align: center; word-wrap: break-word;'>8.9</td><td style='text-align: center; word-wrap: break-word;'>22.5</td><td style='text-align: center; word-wrap: break-word;'>8.5</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>MVGFormer [^19]</td><td style='text-align: center; word-wrap: break-word;'>19.2</td><td style='text-align: center; word-wrap: break-word;'>8.9</td><td style='text-align: center; word-wrap: break-word;'>21.0</td><td style='text-align: center; word-wrap: break-word;'>8.7</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Epipolar Trans. [^14]</td><td style='text-align: center; word-wrap: break-word;'>19.0</td><td style='text-align: center; word-wrap: break-word;'>9.0</td><td style='text-align: center; word-wrap: break-word;'>20.9</td><td style='text-align: center; word-wrap: break-word;'>13.3</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>ESTHER (ours)</td><td style='text-align: center; word-wrap: break-word;'>17.7</td><td style='text-align: center; word-wrap: break-word;'>8.6</td><td style='text-align: center; word-wrap: break-word;'>21.3</td><td style='text-align: center; word-wrap: break-word;'>3.3</td></tr><tr><td rowspan="5">HOT3D z.s.</td><td style='text-align: center; word-wrap: break-word;'>Hiera-FTL [^33] [^50]</td><td style='text-align: center; word-wrap: break-word;'>103.2</td><td style='text-align: center; word-wrap: break-word;'>18.9</td><td style='text-align: center; word-wrap: break-word;'>324.3</td><td style='text-align: center; word-wrap: break-word;'>99.6</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>MVGFormer [^19]</td><td style='text-align: center; word-wrap: break-word;'>52.2</td><td style='text-align: center; word-wrap: break-word;'>18.7</td><td style='text-align: center; word-wrap: break-word;'>71.9</td><td style='text-align: center; word-wrap: break-word;'>31.5</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>POEM-v2 [^46]</td><td style='text-align: center; word-wrap: break-word;'>49.7</td><td style='text-align: center; word-wrap: break-word;'>16.7</td><td style='text-align: center; word-wrap: break-word;'>66.4</td><td style='text-align: center; word-wrap: break-word;'>33.4</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Epipolar Trans. [^14]</td><td style='text-align: center; word-wrap: break-word;'>48.3</td><td style='text-align: center; word-wrap: break-word;'>19.5</td><td style='text-align: center; word-wrap: break-word;'>83.8</td><td style='text-align: center; word-wrap: break-word;'>51.1</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>ESTHER (ours)</td><td style='text-align: center; word-wrap: break-word;'>30.5</td><td style='text-align: center; word-wrap: break-word;'>17.3</td><td style='text-align: center; word-wrap: break-word;'>54.2</td><td style='text-align: center; word-wrap: break-word;'>20.9</td></tr><tr><td rowspan="5">HOT3D f.t.</td><td style='text-align: center; word-wrap: break-word;'>Hiera-FTL [^33] [^50]</td><td style='text-align: center; word-wrap: break-word;'>45.3</td><td style='text-align: center; word-wrap: break-word;'>13.6</td><td style='text-align: center; word-wrap: break-word;'>46.0</td><td style='text-align: center; word-wrap: break-word;'>52.8</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>POEM-v2 [^46]</td><td style='text-align: center; word-wrap: break-word;'>35.5</td><td style='text-align: center; word-wrap: break-word;'>9.5</td><td style='text-align: center; word-wrap: break-word;'>25.9</td><td style='text-align: center; word-wrap: break-word;'>23.7</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Epipolar Trans. [^14]</td><td style='text-align: center; word-wrap: break-word;'>35.1</td><td style='text-align: center; word-wrap: break-word;'>9.6</td><td style='text-align: center; word-wrap: break-word;'>45.4</td><td style='text-align: center; word-wrap: break-word;'>55.9</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>MVGFormer [^19]</td><td style='text-align: center; word-wrap: break-word;'>34.9</td><td style='text-align: center; word-wrap: break-word;'>9.2</td><td style='text-align: center; word-wrap: break-word;'>24.7</td><td style='text-align: center; word-wrap: break-word;'>23.2</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>ESTHER (ours)</td><td style='text-align: center; word-wrap: break-word;'>29.5</td><td style='text-align: center; word-wrap: break-word;'>8.0</td><td style='text-align: center; word-wrap: break-word;'>20.8</td><td style='text-align: center; word-wrap: break-word;'>17.1</td></tr></table>

in 308 real venues, 44 scene categories, and roughly 800 manipulation tasks (Figure 5; breakdown in Appendix G). The collection covers common conditions for wearable systems: hands leaving the field of view, object interactions, self-occlusion, rapid head motion, and asymmetric visibility (Figure 1). The test set is captured in a motion capture studio mirroring the six industries (about 10 hours, 6 subjects, 50 scripted tasks spanning 14 hand action types; Appendix H).

#### 4.1. Labeling with ESTHER and Iterative Enhancement

ESTHER labels the raw collection at scale, in the spirit of depth data engines [^44] [^45] but grounded by stereo geometry and motion capture validation: each prediction is refined by 200 steps of teacher optimization under the DPoser-X prior and the 2D evidence (2D keypoints in both views), then passes the same reprojection and human review filter as the teacher. Surviving labels enter the benchmark and feed back into training through the flywheel in Section 3.8.

Label quality. The in-the-wild labels have a median reprojection residual of 5.3 px, corresponding to a median in-plane discrepancy of 2.8 mm. With the 6.2 cm stereo baseline, the propagation analysis in Appendix N indicates about 21 mm depth sensitivity under independent view errors. These are agreement and sensitivity measures, not bounds on true 3D label error. Metric accuracy is assessed against motion capture: the teacher reaches 15.9 mm MPJPE and 19.1 mm wrist error.

### 5. Experiments

The main text reports the controlled architecture comparison, the robustness studies, and the component ablation; remaining details are in Appendices A–O.

<div style="text-align: center;">Table 2. Robustness to missing views. One stereo view is dropped for methods trained with stereo (ESTHER falls back via its view mask token); monocular methods take their usual single view. Metrics as in Table 1.</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td colspan="2">Method</td><td style='text-align: center; word-wrap: break-word;'>MPIPE $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>PA-MPIPE $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>Wrist $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>Jitter $\downarrow$</td></tr><tr><td rowspan="8">ESTHER3D</td><td style='text-align: center; word-wrap: break-word;'>Hiera-FTL [^33] [^50]</td><td style='text-align: center; word-wrap: break-word;'>65.2</td><td style='text-align: center; word-wrap: break-word;'>9.9</td><td style='text-align: center; word-wrap: break-word;'>217</td><td style='text-align: center; word-wrap: break-word;'>55.2</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>HaWoR [^48] (mono, temporal)</td><td style='text-align: center; word-wrap: break-word;'>43.8</td><td style='text-align: center; word-wrap: break-word;'>11.2</td><td style='text-align: center; word-wrap: break-word;'>97.3</td><td style='text-align: center; word-wrap: break-word;'>4.8</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>HaMeR [^29] (mono)</td><td style='text-align: center; word-wrap: break-word;'>36.3</td><td style='text-align: center; word-wrap: break-word;'>11.8</td><td style='text-align: center; word-wrap: break-word;'>100</td><td style='text-align: center; word-wrap: break-word;'>10.5</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>WiLoR [^30] (mono)</td><td style='text-align: center; word-wrap: break-word;'>30.2</td><td style='text-align: center; word-wrap: break-word;'>11.1</td><td style='text-align: center; word-wrap: break-word;'>101.6</td><td style='text-align: center; word-wrap: break-word;'>9.5</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>POEM-v2 [^46]</td><td style='text-align: center; word-wrap: break-word;'>29.5</td><td style='text-align: center; word-wrap: break-word;'>9.7</td><td style='text-align: center; word-wrap: break-word;'>96.8</td><td style='text-align: center; word-wrap: break-word;'>8.8</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>MVGFormer [^19]</td><td style='text-align: center; word-wrap: break-word;'>28.9</td><td style='text-align: center; word-wrap: break-word;'>9.8</td><td style='text-align: center; word-wrap: break-word;'>97.1</td><td style='text-align: center; word-wrap: break-word;'>9.3</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Epipolar Trans. [^14]</td><td style='text-align: center; word-wrap: break-word;'>27.0</td><td style='text-align: center; word-wrap: break-word;'>9.5</td><td style='text-align: center; word-wrap: break-word;'>171.7</td><td style='text-align: center; word-wrap: break-word;'>10.5</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>ESTHER (ours)</td><td style='text-align: center; word-wrap: break-word;'>18.0</td><td style='text-align: center; word-wrap: break-word;'>9.0</td><td style='text-align: center; word-wrap: break-word;'>32.5</td><td style='text-align: center; word-wrap: break-word;'>4.3</td></tr><tr><td rowspan="8">HOT3D f.t.</td><td style='text-align: center; word-wrap: break-word;'>HaWoR [^48] (mono, temporal)</td><td style='text-align: center; word-wrap: break-word;'>71.8</td><td style='text-align: center; word-wrap: break-word;'>29.3</td><td style='text-align: center; word-wrap: break-word;'>214</td><td style='text-align: center; word-wrap: break-word;'>26.9</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>HaMeR [^29] (mono)</td><td style='text-align: center; word-wrap: break-word;'>69.5</td><td style='text-align: center; word-wrap: break-word;'>17.4</td><td style='text-align: center; word-wrap: break-word;'>126.8</td><td style='text-align: center; word-wrap: break-word;'>96.4</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Hiera-FTL [^33] [^50]</td><td style='text-align: center; word-wrap: break-word;'>52.4</td><td style='text-align: center; word-wrap: break-word;'>18.7</td><td style='text-align: center; word-wrap: break-word;'>115.6</td><td style='text-align: center; word-wrap: break-word;'>53.3</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Epipolar Trans. [^14]</td><td style='text-align: center; word-wrap: break-word;'>42.1</td><td style='text-align: center; word-wrap: break-word;'>15.8</td><td style='text-align: center; word-wrap: break-word;'>93.2</td><td style='text-align: center; word-wrap: break-word;'>45.8</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>MVGFormer [^19]</td><td style='text-align: center; word-wrap: break-word;'>39.9</td><td style='text-align: center; word-wrap: break-word;'>14.6</td><td style='text-align: center; word-wrap: break-word;'>90.0</td><td style='text-align: center; word-wrap: break-word;'>23.3</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>POEM-v2 [^46]</td><td style='text-align: center; word-wrap: break-word;'>37.2</td><td style='text-align: center; word-wrap: break-word;'>14.9</td><td style='text-align: center; word-wrap: break-word;'>130.2</td><td style='text-align: center; word-wrap: break-word;'>22.7</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>WiLoR [^30] (mono)</td><td style='text-align: center; word-wrap: break-word;'>31.0</td><td style='text-align: center; word-wrap: break-word;'>15.9</td><td style='text-align: center; word-wrap: break-word;'>111.5</td><td style='text-align: center; word-wrap: break-word;'>91.8</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>ESTHER (ours)</td><td style='text-align: center; word-wrap: break-word;'>30.2</td><td style='text-align: center; word-wrap: break-word;'>12.1</td><td style='text-align: center; word-wrap: break-word;'>33.9</td><td style='text-align: center; word-wrap: break-word;'>18.4</td></tr></table>

#### 5.1. Protocol and Metrics

All trainable methods share the frozen backbone, data, losses, and budget, so the comparison isolates architecture. Evaluation covers the ESTHER3D test set (zero-shot for every method) and HOT3D [^1], zero-shot and after fine-tuning on 10% of its data. MPJPE is mean joint position error relative to the wrist; PA-MPJPE adds Procrustes alignment. MPJPE-absolute denotes error without wrist alignment. We report MPJPE, PA-MPJPE, absolute wrist error (mm), and temporal jitter (m/s²; Appendix L). Ablations use MPJPE on the test set.

#### 5.2. Architecture Comparison

Baselines. We compare three groups: (1) multiview architectures adapted to stereo for individual frames: Epipolar Transformers [^14], MVGFormer [^19], POEM-v2 [^46]; (2) the HOT3D challenge winner, using Hiera [^33] with FTL camera fusion [^50]; and (3) pretrained monocular models: WiLoR [^30], HaMeR [^29], and the temporal HaWoR [^48].

Monocular wrist. The monocular methods predict a weak-perspective camera on the normalized crop, $x_{crop} = s(X + t_x)$, which involves no camera intrinsics. Because the MANO joints X are in meters, the scale s carries units of normalized crop per meter: metric scale already enters through MANO the moment s is predicted, and the hand lives directly in MANO metric coordinates. Writing the same point with the true pinhole, $u = f(X + t_x)/t_z + c$ and $x_{crop} = 2(u - c_{box})/b$ for a crop box of side b pixels, and matching the coefficient of X gives $s = 2f/(t_z b)$, i.e., $t_z = 2f/(s b)$, which is dimensionally meters. Recovering the wrist depth is thus a pure unit conversion rather than an intrinsic assumption, so their wrists are evaluated in the same metric camera frame as ESTHER's.

ESTHER3D test set. Against real ground truth (zero-shot for every method; Figure 6), ESTHER leads where stereo and temporal fusion matter: best pose (17.7 mm MPJPE) and three to four times lower jitter (3.3 versus 8.5-

---

![Image](images/750301fac5d11aee11fd2af6d6cb46bb8156256ec1674e2610760ec430c30345.jpg)

<div style="text-align: center;">Figure 6. Qualitative results on ESTHER3D and HOT3D: MANO meshes projected into both stereo views (pink right, blue left hand). Per block: top row full stereo, bottom row monocular.</div>

13.3 m/s $^{2}$). Independent frame processing cannot model this temporal behavior. ESTHER's wrist error on clean frames matches the multiview methods (21.3 versus 20.9–22.5 mm; absolute wrist prediction trades some accuracy on clean frames for robustness to missing views in Table 2), while Hiera-FTL's absolute head without an anchor fails outright (277 mm wrist).

External generalization. On HOT3D, ESTHER's zero-shot MPJPE (30.5 mm) beats even the fine-tuned baselines. After fine-tuning, ESTHER achieves the lowest MPJPE, PA-MPJPE, wrist error, and jitter among compared methods.

ESTHER is also stable across the HOT3D pinhole focal sweep (Appendix M).

#### 5.3. Robustness

One stereo view missing. We blank one view at inference, simulating a hand leaving or becoming occluded in one camera.

With a camera gone, the stereo baselines collapse in wrist: their triangulated anchor vanishes with the view. ESTHER degrades gracefully rather than collapsing: relative pose is nearly unchanged (17.7 → 18.0 mm MPJPE, PA-MPJPE 8.6 → 9.0 mm), while absolute wrist error increases with the loss of stereo depth cues, rising from 21.3 to 32.5 mm on ESTHER3D. This is below the baselines' wrist errors of 96.8–217 mm; training with blanked views and the view mask token let the model fall back to a single view without the wrist collapse that undoes the triangulation-anchored baselines. Even on a single grayscale view ESTHER surpasses the 1.3B-parameter video diffusion model ViDiHand [^40] at 355M total parameters (51M trainable atop a frozen DINOv3-L backbone). This robustness comes from the stereo training regime and the view mask token together: stereo supervision instills a metric depth prior, and under it the token learns to complete depth from one view, as the brain does from a single eye, using scene context (objects, arm) to stay stable and generalizable when a view is lost (Sec. 5.4). On HOT3D,

<div style="text-align: center;">Table 3. Robustness to dropped frames as N of 16 frames are blanked and masked at inference, on the ESTHER3D test set and HOT3D. Each cell: MPJPE / PA-MPJPE / wrist (mm) / jitter (m/s $^2$).</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'>N dropped / 16</td><td style='text-align: center; word-wrap: break-word;'>ESTHER3D</td><td style='text-align: center; word-wrap: break-word;'>HOT3D</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>0</td><td style='text-align: center; word-wrap: break-word;'>17.7 / 8.6 / 21.3 / 3.3</td><td style='text-align: center; word-wrap: break-word;'>29.5 / 8.0 / 20.8 / 17.1</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>2</td><td style='text-align: center; word-wrap: break-word;'>17.8 / 8.6 / 21.3 / 3.9</td><td style='text-align: center; word-wrap: break-word;'>29.7 / 8.2 / 20.9 / 18.2</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>4</td><td style='text-align: center; word-wrap: break-word;'>17.8 / 8.6 / 21.4 / 4.7</td><td style='text-align: center; word-wrap: break-word;'>29.8 / 8.1 / 20.9 / 19.5</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>6</td><td style='text-align: center; word-wrap: break-word;'>17.8 / 8.6 / 21.5 / 7.7</td><td style='text-align: center; word-wrap: break-word;'>29.8 / 8.1 / 21.2 / 23.8</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>8</td><td style='text-align: center; word-wrap: break-word;'>17.8 / 8.6 / 21.5 / 5.4</td><td style='text-align: center; word-wrap: break-word;'>30.0 / 8.2 / 21.0 / 20.6</td></tr></table>

inference from one view gives 30.2 mm MPJPE, 12.1 mm PA-MPJPE, 33.9 mm wrist error, and 18.4 m/s $^{2}$ jitter, the lowest values among the compared methods.

Dropped frames. We drop N random frames from each 16-frame clip (images blanked and tokens masked), simulating the detection dropout that is routine in egocentric capture, where fast motion and occlusion break detection in individual frames. Dropping up to half the frames (8 of 16) leaves MPJPE, PA-MPJPE, and wrist essentially unchanged on both datasets (Table 3). From $N=0$ to $N=8$, ESTHER3D MPJPE shifts by 0.1 mm and wrist error by 0.2 mm. Jitter increases overall but not monotonically with N, reflecting the extra frames interpolated between surviving observations. This graceful degradation comes from the objective for masked frames, which teaches the motion encoder to inpaint missing frames from temporal context. Appendix J reports bootstrap confidence intervals for these gaps and Appendix K a study of failure modes under lighting, motion blur, sensor noise, and skin tone shifts.

#### 5.4. Why the Monocular Fallback Stays Reliable

Table 2 shows that dropping a stereo view costs ESTHER only a graceful wrist increase, not the collapse the baselines based on anchors suffer. We probe why, comparing three configurations on the held-out motion capture set: stereo (both views), view mask (our model with one view dropped, its learned placeholder token standing in for the missing

---

![Image](images/0c4e2ead09c664bc6df9dc9195cafa939b6da9ea9208a45f6238a7d5de254e73.jpg)

<div style="text-align: center;">Figure 7. Cross-attention into the kept view (two held-out scenes, four transformer blocks L1–L4). Pure monocular and stereo attend almost identically to the hand; the view mask placeholder instead spreads onto surrounding objects and the arm, using scene context as a depth reference.</div>

view), and pure monocular, a model trained from scratch with the second view always removed (a separate checkpoint, not our model run monocularly).

Attention (Figure 7). We visualize, across the four stereo transformer blocks, the cross-attention that flows into the kept view. The pure monocular model and the stereo model attend almost identically: both concentrate on the hand itself. The view mask model is visibly different: its placeholder token queries beyond the hand, onto surrounding objects, the workspace, and the arm. Trained under stereo guidance, the token has learned to use scene context as a depth reference, analogous to the monocular depth prior used by the human visual system with one eye. This supports feedforward inference without disparity.

Depth scatter across frames (Figure 8). We fit $\hat{z} = a z_{gt} + b$ to the predicted wrist depth on held-out data and decompose the error into a systematic (linearly correctable) and a random (irreducible) part. Stereo tracks true depth tightly (correlation 0.96, random scatter 8 mm). The view mask model stays close (correlation 0.92, random scatter 11 mm): it still knows how depth varies frame to frame, and its residual error is mostly a systematic scale compression that a single linear term removes. The pure monocular model, despite being trained end-to-end for the monocular setting, collapses on held-out data: correlation drops to 0.69 and random scatter triples to 32 mm, i.e. it regresses toward a prior mean plus noise. The view mask token thus keeps depth estimates generalizable with low variance where a dedicated monocular model overfits its training cues, which is exactly why ESTHER degrades gracefully under a missing view.

#### 5.5. Ablation

Table 4 identifies stereo fusion as the core contributor, supported by complementary motion cues and auxiliary guidance. Geometric wrist anchoring also benefits articulated

![Image](images/ae91a4dce6ad10705d605da25131a110395d0b9e939dd86aab7b8fbdc01f0434.jpg)

<div style="text-align: center;">Figure 8. Held-out wrist depth prediction vs. ground truth for (a) stereo, (b) the view mask model with one view dropped, and (c) a separately trained pure monocular model. (a) and (b) track true depth (correlation 0.96/0.92, random scatter 8/11 mm); (c) collapses (correlation 0.69, random scatter 32 mm).</div>

<div style="text-align: center;">Table 4. Component ablation on the ESTHER3D test set (MPJPE, mm).  $\Delta$ is the change from the full model; negative values indicate improvement.</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'>Variant</td><td style='text-align: center; word-wrap: break-word;'>MPJPE $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>$\Delta$ vs full</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Full ESTHER</td><td style='text-align: center; word-wrap: break-word;'>17.7</td><td style='text-align: center; word-wrap: break-word;'>-</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>w/o Plucker ray embedding</td><td style='text-align: center; word-wrap: break-word;'>27.7</td><td style='text-align: center; word-wrap: break-word;'>+10.0</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>w/o cross-view attention</td><td style='text-align: center; word-wrap: break-word;'>26.4</td><td style='text-align: center; word-wrap: break-word;'>+8.7</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>w/o all guidance, masking, augmentation</td><td style='text-align: center; word-wrap: break-word;'>20.9</td><td style='text-align: center; word-wrap: break-word;'>+3.2</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>w/o flow token</td><td style='text-align: center; word-wrap: break-word;'>20.5</td><td style='text-align: center; word-wrap: break-word;'>+2.8</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>w/o 2D / heatmap guidance</td><td style='text-align: center; word-wrap: break-word;'>19.2</td><td style='text-align: center; word-wrap: break-word;'>+1.5</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>w/o image augmentation</td><td style='text-align: center; word-wrap: break-word;'>18.5</td><td style='text-align: center; word-wrap: break-word;'>+0.8</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>uniform gate (concatenation)</td><td style='text-align: center; word-wrap: break-word;'>18.0</td><td style='text-align: center; word-wrap: break-word;'>+0.3</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>w/o triangulation guidance</td><td style='text-align: center; word-wrap: break-word;'>17.8</td><td style='text-align: center; word-wrap: break-word;'>+0.1</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>triangulation-anchored wrist</td><td style='text-align: center; word-wrap: break-word;'>15.5</td><td style='text-align: center; word-wrap: break-word;'>-2.2</td></tr></table>

pose: since MPJPE is wrist-relative, the improvement reflects joint configuration and cannot arise solely from more accurate absolute wrist localization.

### 6. Conclusion

We presented ESTHER, an end-to-end model for metric 3D hand reconstruction from in-the-wild egocentric stereo, with ESTHER3D, a benchmark labeled by the model and validated by motion capture. ESTHER reaches state-of-the-art accuracy with superior temporal smoothness and generalization, and a binding effect between size and depth keeps its metric scale under a lost view, dropped frames, or monocular input. Streaming without detection and joint modeling of hands, objects, and bodies remain future work.

## References

[^1]: Prithviraj Banerjee et al. HOT3D: Hand and Object Tracking in 3D from Egocentric Multi-View Videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025. 1, 2, 6

[^2]: Yu-Wei Chao, Wei Yang, Yu Xiang, Pavlo Molchanov, Ankur Handa, Jonathan Tremblay, Yashraj S. Narang, Karl Van Wyk,

---

Umar Iqbal, Stan Birchfield, Jan Kautz, and Dieter Fox. DexYCB: A Benchmark for Capturing Hand Grasping of Objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021. 2

[^3]: Xingyu Chen, Yufeng Liu, Chi Ma, Jianlong Chang, Huamin Wang, Yi Chen, Xiaoguang Guo, Pengfei Wan, and Wen Zheng. MobRecon: Mobile-Friendly Hand Mesh Reconstruction from Monocular Image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022. 2, 3

[^4]: Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. Scaling Egocentric Vision: The EPICKITCHENS Dataset. In European Conference on Computer Vision, 2018. 2

[^5]: Enes Duran, Muhammed Kocabas, Vasileios Choutas, Zicong Fan, and Michael J. Black. HMP: Hand Motion Priors for Pose and Shape Estimation from Video. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2024. 3

[^6]: Zicong Fan, Omid Taheri, Dimitrios Tzionas, Muhammed Kocabas, Manuel Kaufmann, Michael J. Black, and Otmar Hilliges. ARCTIC: A Dataset for Dexterous Bimanual HandObject Manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023. 1,2

[^7]: Guillermo Garcia-Hernando, Shanxin Yuan, Seungryul Baek, and Tae-Kyun Kim. First-Person Hand Action Benchmark with RGB-D Videos and 3D Hand Pose Annotations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018. 2

[^8]: Kristen Grauman et al. Ego4D: Around the World in 3,000 Hours of Egocentric Video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022. 2

[^9]: Kristen Grauman et al. Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 2

[^10]: Shreyas Hampali, Mahdi Rad, Markus Oberweger, and Vincent Lepetit. HOnnotate: A Method for 3D Annotation of Hand and Object Poses. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020. 2, 3

[^11]: Shangchen Han et al. UmeTrack: Unified Multi-View End-to-End Hand Tracking for VR. In SIGGRAPH Asia 2022 Conference Papers, 2022. 2

[^12]: Ryosei Hara, Wataru Ikeda, Masashi Hatano, and Mariko Isogawa. EventEgoHands: Event-based Egocentric 3D Hand Mesh Reconstruction. In IEEE International Conference on Image Processing (ICIP), 2025. 2

[^13]: Yana Hasson, Gul Varol, Dimitrios Tzionas, Igor Kalevatykh, Michael J. Black, Ivan Laptev, and Cordelia Schmid. Learning Joint Reconstruction of Hands and Manipulated Objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019. 2

[^14]: Yihui He, Rui Yan, Katerina Fragkiadaki, and Shoou-I Yu. Epipolar Transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020. 2, 6

[^15]: Karim Iskakov, Egor Burkov, Victor Lempitsky, and Yury Malkov. Learnable Triangulation of Human Pose. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019. 1, 2

[^16]: Glenn Jocher and Jing Qiu. Ultralytics YOLO11. <https://github.com/ultralytics/ultralytics>, 2024. 3

[^17]: Taein Kwon, Bugra Tekin, Jan Stuhmer, Federica Bogo, and Marc Pollefeys. H2O: Two Hands Manipulating Objects for First Person Interaction Recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021. 1, 2

[^18]: Yuncheng Li, Zehao Xue, Yingying Wang, Liuhao Ge, Zhou Ren, and Jonathan Rodriguez. End-to-End 3D Hand Pose Estimation from Stereo Cameras. In British Machine Vision Conference (BMVC), 2019. 2

[^19]: Ziwei Liao, Jialiang Zhu, Chunyu Wang, Han Hu, and Steven L. Waslander. Multiple View Geometry Transformers for 3D Human Pose Estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024. 1, 2, 6

[^20]: Kevin Lin, Lijuan Wang, and Zicheng Liu. Mesh Graphormer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021. 2

[^21]: Yunze Liu et al. HOI4D: A 4D Egocentric Dataset for Category-Level Human-Object Interaction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022. 1,2

[^22]: Junzhe Lu, Jing Lin, Hongkun Dou, Ailing Zeng, Yue Deng, Xian Liu, Zhongang Cai, Lei Yang, Yulun Zhang, Haoqian Wang, and Ziwei Liu. DPoser-X: Diffusion Model as Robust 3D Whole-body Human Pose Prior. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2025. 2

[^23]: Lingni Ma et al. Nymeria: A Massive Collection of Multimodal Egocentric Daily Motion in the Wild. In European Conference on Computer Vision (ECCV), 2024. 2

[^24]: Naureen Mahmood, Nima Ghorbani, Nikolaus F. Troje, Gerard Pons-Moll, and Michael J. Black. AMASS: Archive of Motion Capture as Surface Shapes. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019. 3

[^25]: Gyeongsik Moon and Kyoung Mu Lee. I2L-MeshNet: Image-to-Lixel Prediction Network for Accurate 3D Human Pose and Mesh Estimation from a Single RGB Image. In European Conference on Computer Vision, 2020. 2

[^26]: Gyeongsik Moon, Shoou-I Yu, He Wen, Takaaki Shiratori, and Kyoung Mu Lee. InterHand2.6M: A Dataset and Baseline for 3D Interacting Hand Pose Estimation from a Single RGB Image. In European Conference on Computer Vision, 2020. 2

[^27]: Takehiko Ohkawa, Kun He, Fadime Sener, Tomas Hodan, Minh Tran, Cem Keskin, Jamie Shotton, Dima Damen, Greg Mori, and Yoichi Sato. AssemblyHands: Towards Egocentric

---

Activity Understanding via 3D Hand Pose Estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023. 1, 2

[^28]: JoonKyu Park, Youngjoong Oh, Gyeongsik Moon, Hong-suk Choi, and Kyoung Mu Lee. HandOccNet: Occlusion-Robust 3D Hand Mesh Estimation Network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022. 2

[^29]: Georgios Pavlakos, Dandan Shan, Ilija Radosavovic, Angjoo Kanazawa, David Fouhey, and Jitendra Malik. Reconstructing Hands in 3D with Transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024. 1, 2, 3, 6

[^30]: Rolandos Alexandros Potamias, Jinglei Zhang, Jiankang Deng, and Stefanos Zafeiriou. WiLoR: End-to-End 3D Hand Localization and Reconstruction in-the-wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025. 1, 2, 3, 4, 6

[^31]: Davis Rempe, Tolga Birdal, Aaron Hertzmann, Jimei Yang, Srinath Sridhar, and Leonidas J. Guibas. HuMoR: 3D Human Motion Model for Robust Pose Estimation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021. 3

[^32]: Javier Romero, Dimitrios Tzionas, and Michael J. Black. Embodied Hands: Modeling and Capturing Hands and Bodies Together. ACM Transactions on Graphics, 36(6), 2017. 2

[^33]: Chaitanya Ryali, Yuan-Ting Hu, Daniel Bolya, Chen Wei, Haoqi Fan, Po-Yao Huang, Vaibhav Aggarwal, Arkabandhu Chowdhury, Omid Poursaeed, Judy Hoffman, Jitendra Malik, Yanghao Li, and Christoph Feichtenhofer. Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistles. In International Conference on Machine Learning, 2023. 6

[^34]: Kyeongeun Seo, Hyeonjoong Cho, Daewoong Choi, and Taewook Heo. Stereo Feature Learning Based on Attention and Geometry for Absolute Hand Pose Estimation in Egocentric Stereo Views. IEEE Access, 9:116083–116093, 2021. 2

[^35]: Md Salman Shamil, Dibyadip Chatterjee, Fadime Sener, Shugao Ma, and Angela Yao. On the Utility of 3D Hand Poses for Action Recognition. In European Conference on Computer Vision, 2024. 1, 2

[^36]: Oriane Siméoni, Huy V. Vo, Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, Cijo Jose, Vasil Khalidov, Marc Szafraniec, Seungeun Yi, Michaël Ramamonjisoa, Francisco Massa, Daniel Haziza, Luca Wehrstedt, Jianyuan Wang, Timothée Darcet, Théo Moutakanni, Leonel Sentana, Claire Roberts, Andrea Vedaldi, Jamie Tolan, John Brandt, Camille Couprie, Julien Mairal, Hervé Jégou, Patrick Labatut, and Piotr Bojanowski. DINOv3. arXiv preprint arXiv:2508.10104, 2025. 4

[^37]: Hanyue Tu, Chunyu Wang, and Wenjun Zeng. VoxelPose: Towards Multi-Camera 3D Human Pose Estimation in Wild Environment. In European Conference on Computer Vision, 2020. 2

[^38]: Luming Wang, Hao Shi, Jiajun Zhai, Kailun Yang, and Kaiwei Wang. EgoEV-HandPose: Egocentric 3D Hand Pose Estimation and Gesture Recognition with Stereo Event Cameras. arXiv preprint arXiv:2605.12297, 2026. 2

[^39]: Tao Wang, Jianfeng Zhang, Yujun Cai, Shuicheng Yan, and Jiashi Feng. Direct Multi-view Multi-person 3D Pose Estimation. In Advances in Neural Information Processing Systems, 2021. 2

[^40]: Yuxi Wang, Chengkai Jin, Yufei Liu, Wenqi Ouyang, Tianyi Wei, Zhiwei Zeng, Siyuan Huang, Zhiqi Shen, and Xingang Pan. The Surprising Effectiveness of Video Diffusion Models for Hand Motion Reconstruction. arXiv preprint arXiv:2606.30308, 2026. 7

[^41]: Yilin Wen, Hao Pan, Lei Yang, Jia Pan, Taku Komura, and Wenping Wang. Hierarchical Temporal Transformer for 3D Hand Pose Estimation and Action Recognition from Egocentric RGB Videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023. 1, 2

[^42]: Yufei Xu, Jing Zhang, Qiming Zhang, and Dacheng Tao. ViT-Pose: Simple Vision Transformer Baselines for Human Pose Estimation. In Advances in Neural Information Processing Systems (NeurIPS), 2022. 3

[^43]: Lixin Yang, Xinyu Zhan, Kailin Li, Wenqiang Xu, Jiefeng Li, and Cewu Lu. OakInk: A Large-Scale Knowledge Repository for Understanding Hand-Object Interaction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022. 1, 2

[^44]: Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024. 6

[^45]: Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiao-gang Xu, Jiashi Feng, and Hengshuang Zhao. Depth Anything V2. In Advances in Neural Information Processing Systems, 2024. 6

[^46]: Lixin Yang, Licheng Zhong, Pengxiang Zhu, Xinyu Zhan, Junxiao Kong, Jian Xu, and Cewu Lu. Multi-view hand reconstruction with a point-embedded transformer. IEEE Transactions on Pattern Analysis and Machine Intelligence, 47(11):10680–10695, 2025. 1, 2, 6

[^47]: Zhengdi Yu, Stefanos Zafeiriou, and Tolga Birdal. Dyn-HaMR: Recovering 4D Interacting Hand Motion from a Dynamic Camera. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025. 1,2

[^48]: Jinglei Zhang, Jiankang Deng, Chao Ma, and Rolandos Alexandros Potamias. HaWoR: World-Space Hand Motion Reconstruction from Egocentric Videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025. 1, 2, 6

[^49]: Christian Zimmermann, Duygu Ceylan, Jimei Yang, Bryan Russell, Max Argus, and Thomas Brox. FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape from Single RGB Images. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019. 2, 3

[^50]: Minqiang Zou, Zhi Lv, Riqiang Jin, Tian Zhan, Mochen Yu, Yao Tang, and Jiajun Liang. 1st Place Solution of Multiview Egocentric Hand Tracking Challenge ECCV2024. arXiv preprint arXiv:2409.19362, 2024. 6

---

## Appendix

### A. Teacher Pipeline Details

The teacher first optimizes an initial 10 hours of in-the-wild video for approximately 2000 steps; filtered outputs train the first ESTHER. For later rounds, ESTHER forward inference supplies MANO initial estimates on additional samples. The teacher refines them with the same objectives for 200 steps, followed by the same geometric, reprojection, and human filtering described below. Accepted teacher outputs expand the training set. Deployment and testing use forward inference of the final ESTHER.

The 2D evidence is produced by an ensemble of detectors (YOLO11-pose and ViTPose) run independently on both rectified pinhole views. Each detector $d \in \{1,2\}$ returns, for every one of the 21 hand joints j, a 2D location $\mathbf{u}_j^d$ and confidence $c_j^d$, and each hand box a detector score $s^d$. Their outputs are fused using confidence, following WiLoR. With the valid set $V_j = \{d : c_j^d \geq \tau\}$ (threshold $\tau=0.3$), the fused keypoint is

 $$ \hat{\mathbf{u}}_{j}=\left\{\begin{aligned}{}&{{}\frac{c_{j}^{1}\mathbf{u}_{j}^{1}+c_{j}^{2}\mathbf{u}_{j}^{2}}{c_{j}^{1}+c_{j}^{2}},}&{|V_{j}|=2\operatorname{a n d}\|\mathbf{u}_{j}^{1}-\mathbf{u}_{j}^{2}\|\leq\epsilon w_{b},}\\ {}&{{}\mathbf{u}_{j}^{d^{\star}},}&{\mathrm{o t h e r w i s e},}\\ \end{aligned}\right. $$

where $d^* = \arg \max_{d \in V_j} c_j^d$ (near ties broken by the box score $s^d$), $w_b$ is the hand box side, and joints with $V_j = \emptyset$ are dropped. The agreement radius is $\epsilon = 0.1$ (two detections are averaged only if they fall within 10% of the hand box side of each other); it is set to about one MANO joint spacing at the working hand scale and the fusion is insensitive to it in the 0.08–0.15 range. In words: two detectors that agree are averaged with confidence weights, a lone confident detector is trusted, and disagreements fall back to the estimate with higher confidence, retaining each detector's reliable keypoints while suppressing the isolated false positives that either produces alone. The stereo filtering stage then removes geometrically implausible detections in two steps. First, joints whose triangulated depth falls outside the 0.1–1.0 m distance range between hand and camera typical of head-mounted capture are rejected. Second, a structural constraint on the relative distances between wrist and finger joints removes isolated matches whose reconstructed bone lengths deviate by more than 40% from the canonical MANO template. Surviving keypoints and coarse 3D anchors form the observation set $\mathcal{V}$ of the fitting objective

 $$ \begin{aligned}\mathcal{L}_{fit}=&\lambda_{2d}\mathcal{L}_{2d}+\lambda_{p}\mathcal{R}_{pose}(\boldsymbol{\theta})\\&+\lambda_{b}\|\boldsymbol{\beta}-\boldsymbol{\beta}_{0}\|_{2}^{2}+\lambda_{t}\|\mathbf{t}-\mathbf{t}_{0}\|_{2}^{2},\end{aligned} $$

where  $\mathcal{L}_{2d}$ is a robust reprojection loss between projected MANO joints and the filtered stereo keypoints over both views. Triangulation serves as both the initialization and a soft constraint: the triangulated wrist depth  $\hat{z}_{\text{wrist}}$ seeds the global translation  $\mathbf{t}_0$, and the mild  $\lambda_t\|\mathbf{t} - \mathbf{t}_0\|_2^2$ term then anchors the solution loosely to it while  $\mathcal{L}_{2d}$ drives the fit. The weight  $\lambda_t$ is small, so this is a soft prior the reprojection can override, not a hard depth constraint. For the initial 10-hour seed set, WiLoR initialization fuses MANO estimates from the left and right views in the calibrated stereo frame. In subsequent labeling rounds, ESTHER predictions replace this initialization; the optimization objectives and filtering remain the same.

The optimization runs in two stages. Stage 1 (shape lock) selects the 64 frames with the highest stereo detection confidence, optimizes $(\beta, \theta, \mathbf{t})$ jointly on them under $\mathcal{L}_{\mathrm{fit}}$, and averages the framewise shapes into one $\bar{\beta}$ per hand. Stage 2 reoptimizes pose and wrist per frame with $\bar{\beta}$ fixed, then applies motion optimization over 16-frame sliding windows with the objective

 $$ \begin{aligned}\mathcal{L}_{\mathrm{motion}}=&\sum_{t}\Big[\lambda_{2d}\mathcal{L}_{2d,t}+\lambda_{z}\rho\big(t_{z}^{(t)}-\hat{z}_{\mathrm{wrist}}^{(t)}\big)\\&+\lambda_{\omega}\rho\big(\angle\big(\phi_{t+1},\phi_{t}\big)-\omega_{\max}\big)\\&+\lambda_{v}\rho\big(\|\mathbf{t}_{t+1}-\mathbf{t}_{t}\|-v_{\max}\big)\Big]\\&+\lambda_{m}\mathcal{R}_{\mathrm{motion}}\big(\boldsymbol{\theta}_{t:t+15}\big),\end{aligned} $$

where $\phi_t$ is the global (wrist) orientation, i.e. the root axis-angle component of the pose $\boldsymbol{\theta}_t$ itself (its last three entries in the MANO parameterization); the angular penalty thus acts on a subvector of $\boldsymbol{\theta}_t$, and $\angle(\cdot, \cdot)$ is the geodesic angle between consecutive orientations. As in Stage 1, the triangulated wrist depth $\hat{z}_{\mathrm{wrist}}^{(t)}$ enters only as the same soft anchor on depth $t_z^{(t)}$ at each frame (small weight $\lambda_z$, robust $\rho$); the reprojection term can override it, so it is a prior rather than a hard depth constraint. Here, as in $\mathcal{L}_{2d}$, $\rho$ is the Huber function (quadratic within a small threshold $\delta$, linear beyond), which downweights single-frame outliers. It acts on the raw residual for the depth anchor and on the excess over the limit for the two velocity terms, so the velocity penalty is one-sided and active only above $\omega_{\max}$ and $v_{\max}$ ( $\omega_{\max} \approx 300^\circ/s$ and $v_{\max} \approx 1.5$ m/s at 30 fps); the orientation term carries the larger weight ( $\lambda_\omega > \lambda_v$), because over a short 16-frame window the global translation t is partly confounded by camera motion and is therefore constrained more loosely. $\mathcal{R}_{\text{motion}}$ is the energy of a 16-frame motion prior, from an encoder pretrained on AMASS, over the pose trajectory $\boldsymbol{\theta}_{t:t+15}$; it scores the orientation and articulation dynamics of the hand and is independent of the global translation. The smoothing weights are derived from the prior's gradient feedback and from differences between frames, so windows with implausible dynamics receive stronger regularization while genuinely fast motion is preserved.

The pipeline uses stereo only where it is geometrically reliable and lets the hand model and priors regularize unobserved structure: under object occlusion or self-occlusion

---

the desired annotation is a structured articulated state with semantic joints, MANO shape, and global wrist translation, not the depth of whatever surface happens to be visible. This suits egocentric capture, where hand size, camera distance, and motion centered at the wrist occupy a constrained range.

The filtering above (confidence, depth range, plausibility of bone lengths) is the coarse first stage: it prescreens the input 2D evidence before fitting. Fitted sequence labels pass two further stages before entering ESTHER3D. Stage two is a fine 2D check: it reprojects each fitted hand into both views and computes the residual r, the mean pixel distance to the fused keypoints over all joints of both views with confidence above $\tau$; hand frames with $r > 15$ px are discarded. Stage three is human review for the residual failures the automatic stages cannot catch (wrong hand identity, implausible articulation). This coarse (bone length/depth) $\rightarrow$ fine (2D reprojection) $\rightarrow$ human screen is the filter described in the main paper.

### B. Loss Definitions and Weights

Training proceeds in the two stages of the main paper: the stereo image encoder is first pretrained under its geometric guidance, then the full model is trained jointly.

Stage 1: image encoder pretraining. The image encoder is trained alone under the geometric guidance group $\mathcal{L}_{\mathrm{img}} = \lambda_{2d}\mathcal{L}_{2d} + \lambda_{hm}\mathcal{L}_{hm} + \lambda_{tri}\mathcal{L}_{tri}$ with $\lambda_{2d}=0.5$, $\lambda_{hm}=1.0$, $\lambda_{tri}=1.0$. The 2D loss is a Huber loss on coordinates normalized by image size, masked by joint visibility in each view; a blanked view contributes no 2D or heatmap supervision. The heatmap loss is a cross entropy against Gaussian soft targets on the prediction grid. The triangulation loss is an L1 loss between $\mathbf{X}^{tri}$ and metric 3D joints, applied only to joints visible in both views. Visibility uses binary cross entropy on valid image projections; depth uses cross entropy over logarithmically spaced bins.

Stage 2: joint training of the full model. The image encoder (keeping its guidance from Stage 1) is joined by the motion encoder and fusion decoder and trained end-to-end. The decoder output is mapped through the MANO forward kinematics layer, and $\mathcal{L}_{\mathrm{FK}}$ combines L1 losses on joints in the reference frame and canonical (wrist-relative) joints, an L1 mesh vertex loss, a geodesic rotation loss and 6D consistency term on pose, an L1 shape loss, an L1 wrist translation loss, and articulated priors (PCA pose prior, prior on joint limits, and shape regularization), with weights $\lambda_J=1.0$, $\lambda_{Jcan}=1.0$, $\lambda_V=0.5$, $\lambda_\theta=1.0$, $\lambda_{6d}=0.2$, $\lambda_\beta=0.01$, $\lambda_t=2.0$, and prior weights 0.1/0.05/0.02. In the same stage the motion encoder is supervised at masked positions, reconstructing canonical 3D joints (L1, per masked joint), MANO pose (L1, per masked frame), and root velocity (L1 on frame differences), summed as $\mathcal{L}_{\mathrm{mot}}$ with weight $\lambda_{mot}=0.5$; the on-screen term $\mathcal{L}_{\mathrm{on}}$ is a binary cross entropy against framewise labels indicating hand visibility in at least one retained view, with weight $\lambda_{on}=0.5$.

### C. Masking and Augmentation Recipe

Token masking rates: individual joints 0.20, entire frames 0.10, contiguous temporal span 0.10 (span length 3), entire views 0.05 (the first three via the motion mask token, the last via the view mask token); anchor frames (every T/4) are kept fully observed. Masked 2D observations always mask their associated ray information. Image augmentation: Gaussian noise in each view with standard deviation up to 0.08; random erasing with probability 0.5 (two rectangles covering 10–40% of each side); blanking entire views with probability 0.12; blanking entire frames with probability 0.08, which also provides on-screen negatives.

### D. Camera Motion Flow Token

The flow token is the dense classical optical flow of the whole crop between two consecutive frames, encoded directly as a conditioning input. It uses a fast classical estimator over all crop pixels, including the hand. The estimator runs online for every frame and requires neither training nor ground truth.

The crops follow the hand: each frame's crop has its own intrinsic $\mathbf{K}_t$ (its centre and scale track the detected hand every frame), so even a static camera produces apparent motion purely from the moving crop window. Using the crop intrinsic available at inference, we compensate this before measuring flow: the previous crop is warped by the homography $\mathbf{H}_t = \mathbf{K}_{t+1} \mathbf{K}_t^{-1}$ into the current crop's coordinates, so displacement from crop motion cancels and the remaining flow reflects real scene motion under the camera. Dense flow on this compensated pair is downsampled by median pooling over an $8 \times 8$ grid, normalized by the 224-pixel crop side, with a flag indicating image validity for each cell:

 $$ \mathbf{f}_{t}=[\operatorname{v e c}(\mathbf{F}_{t}/224),\operatorname{v e c}(\mathbf{M}_{t})]\in\mathbf{R}^{192}, $$

where $\mathbf{F}_t \in \mathbf{R}^{8\times8\times2}$ is the flow grid (128 values) and $\mathbf{M}_t \in \{0,1\}^{8\times8}$ (64 values) marks cells within the frame, for $128+64=192$ dimensions; median pooling rejects flow outliers.

The token carries both camera egomotion and hand motion without manually separating them. The token is fed to the motion encoder directly and, being a conditioning input, is excluded from random token masking. The image and motion tokens, by contrast, are masked during training; unable to lean on the masked appearance, the encoder is forced to read the global, scene wide component of the flow as camera motion and attribute the residual, hand localized component to articulation. The decoupling of hand and camera motion is thus learned from this always present flow cue, rather than absorbing head or crop motion into finger pose.

---

### E. Implementation Details

The model dimension is 512 with 8 attention heads: 4 stereo transformer blocks, 4 temporal transformer blocks, and 2 fusion decoder layers over 16-frame clips. The full model totals 354.5M parameters, of which the frozen DINOv3-L backbone accounts for 303.1M; the trainable remainder is 51.3M (image encoder blocks and heads 21.9M, motion encoder 12.8M, fusion decoder 16.6M). The decoder queries $Q_p$, $Q_s$, and $Q_w$ represent pose, shape, and wrist, respectively. Each query gate is predicted by an MLP from the query state, the pooled mean of each source, and four evidence statistics: the mean and minimum predicted visibility, the magnitude of the triangulated wrist, and the triangulation spread. The DINOv3-L backbone is frozen. The image encoder is pretrained, then the full model is trained jointly with AdamW (learning rate $2 \times 10^{-4}$, cosine schedule, 500 warmup steps, gradient clipping 1.0, bfloat16 autocast, losses in fp32). Left hands are handled by mirroring the stereo rig, and predictions are unmirrored for evaluation and projection. At deployment the detector is used only to localize, crop, and match hands across stereo views; the final ESTHER predicts MANO directly, without teacher optimization.

### F. Camera Setups

Our head-mounted RGB stereo rig is two synchronized color cameras on a rigid headband with a 6.2 cm baseline, calibrated once per device. It is a pinhole pair with focal length 624 px and principal point (924, 606) at $1920 \times 1200$ per eye. The network consumes these images directly. The HOT3D devices instead carry strongly distorted wide-FOV fisheye cameras: both Project Aria and Quest 3 use the Fisheye624 model (FisheyeRadTanThinPrism: six radial, two tangential, and four thin-prism coefficients), on monochrome streams ( $640 \times 480$ for Aria, $1280 \times 1024$ for Quest 3). Table 1 lists camera resolution, focal length, and baseline for each setup; full intrinsic, distortion, and extrinsic are released with the benchmark. Rectifying these fisheye views to pinhole is a design choice, so we ablate several undistortion strategies (Appendix M, Table 4); the HOT3D numbers in the main paper use the strategy with the best average accuracy, and ESTHER is in any case nearly invariant to the choice.

Why rectify HOT3D to pinhole. The HOT3D fisheye streams are undistorted to a pinhole crop before they enter the model, for two reasons. First, the 2D hand detectors (YOLO11-pose, ViTPose) are trained on perspective imagery and localize markedly worse on curved fisheye frames; rectification restores geometry of straight lines and thus reliable detection and cropping. Second, a pinhole crop makes the Plucker ray tokens trivial to compute, since each pixel projects back to a single straight ray through $K^{-1}$, and matches our RGB rig's parameterization, so the same network consumes the HOT3D fisheye rigs unchanged.

Undistortion and its costs. For the Fisheye624

<div style="text-align: center;">Table 1. Camera setups (camera resolution and focal length; stereo baseline). Fisheye rigs are rectified to pinhole before use.</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'>Setup</td><td style='text-align: center; word-wrap: break-word;'>Resolution</td><td style='text-align: center; word-wrap: break-word;'>Focal (px)</td><td style='text-align: center; word-wrap: break-word;'>Baseline (mm)</td><td style='text-align: center; word-wrap: break-word;'>Modality</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Collection rig (ours)</td><td style='text-align: center; word-wrap: break-word;'>$1920 \times 1200$</td><td style='text-align: center; word-wrap: break-word;'>623</td><td style='text-align: center; word-wrap: break-word;'>62</td><td style='text-align: center; word-wrap: break-word;'>RGB</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Test rig (mocap)</td><td style='text-align: center; word-wrap: break-word;'>$1920 \times 1200$</td><td style='text-align: center; word-wrap: break-word;'>623</td><td style='text-align: center; word-wrap: break-word;'>62</td><td style='text-align: center; word-wrap: break-word;'>RGB</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>HOT3D Project Aria</td><td style='text-align: center; word-wrap: break-word;'>640 $\times$ 480</td><td style='text-align: center; word-wrap: break-word;'>241</td><td style='text-align: center; word-wrap: break-word;'>138</td><td style='text-align: center; word-wrap: break-word;'>monochrome</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>HOT3D Quest 3</td><td style='text-align: center; word-wrap: break-word;'>$1280 \times 1024$</td><td style='text-align: center; word-wrap: break-word;'>504</td><td style='text-align: center; word-wrap: break-word;'>64</td><td style='text-align: center; word-wrap: break-word;'>monochrome</td></tr></table>

HOT3D cameras we unproject each pixel with the released model and reproject onto a pinhole camera, baking the fisheye→undistort→crop→resize chain into one crop intrinsic matrix K. Rectifying a wide-FOV fisheye to a pinhole plane is not free: the periphery is magnified, so edge pixels are upsampled and lose effective resolution; the full field of view cannot map to a finite pinhole image, so only a central FOV is kept and hands near the rim can be cropped out; and the distortion model leaves a growing residual reprojection error toward large incidence angles. Together with the grayscale modality and different focal length, this is the domain gap that the small HOT3D fine-tuning (main paper) closes.

### G. ESTHER3D Training Set: In-the-Wild Source Collection

The training set of ESTHER3D is the in-the-wild source collection, which comprises about 194 hours of egocentric stereo video spanning six industry categories, 44 scene categories, and 308 distinct real venues, with about 800 distinct manipulation tasks performed by 121 collectors (main paper). Below we list the scene categories and representative manipulation tasks per industry, ordered by prevalence; Figure 1 shows examples of these in-the-wild scenes.

Retail (44.0 h; 11 scene categories, 44 venues). Convenience stores dominate, followed by department stores, supermarkets, clothing stores, stores selling daily goods, food and beverage retail, leather fashion, hardware and tool shops, and toy stores. Representative tasks: stocking and displaying merchandise, arranging bagged and boxed snacks, hanging cable and accessory products, arranging bracelets and ornaments, tidying shelves, folding and arranging garments, and cleaning the store.

Household (40.0 h; 8 scene categories, 71 venues). Rural homes with multiple stories, penthouse apartments, and apartments of diverse layouts from one-bedroom to duplex. Representative tasks: cleaning floors, making beds, organizing clothes and wardrobes, tidying kitchens, bedrooms, and living rooms, arranging desktop items and sofa cushions, washing dishes, wiping surfaces, and cooking.

Dining (32.0h; 8 scene categories, 57 venues). Fast food restaurants in urban, village, and roadside settings, tea houses, cafes and drink shops, dine-in restaurants and barbecue venues, and food stalls. Representative tasks: arranging tables and chairs, cleaning counters and table tops, sweeping and mopping, laying out tableware, clearing dishes, washing

---

![Image](images/1d12fb604808c0b17841d42e7ee998d0500021230f6f8fd3ee4f14cb281694f6.jpg)

<div style="text-align: center;">Figure 1. ESTHER on in-the-wild ESTHER3D scenes across six industries. Each block shows five timesteps of one scene (columns, spaced 0.2 s apart); per timestep ESTHER's MANO meshes are projected into both stereo eyes side by side (L for left and R for right, right hands in pink, left in blue) with the same metric meshes shown in 3D below.</div>

kitchen containers, and sorting vegetables.

Life services (30.0h; 6 scene categories, 44 venues). Auto repair shops, shops providing daily services, beauty salons, and electronics, phone, and motorcycle repair shops. Representative tasks: removing and installing car headlight parts, chassis repair, phone repair, applying screen protectors, cleaning devices and counters, organizing tools, washing hair, and washing towels.

Factory (26.0 h; 5 scene categories, 42 venues). Printing and book printing workshops, metalwork shops for tools and doors, and wooden furniture workshops. Representative tasks: operating the main workstation, cutting panels, grinding and polishing, repairing machinery and press machines, positioning and punching, carrying materials, and packaging profiles.

Office (22.0 h; 6 scene categories, 50 venues). Corporate office towers and government service centers. Representative tasks: tidying office desks, arranging bookshelf items, tidying reception coffee tables and tea sets, cleaning office areas, and clearing floor litter.

### H. ESTHER3D Test Set: Motion Capture Studio

The test set of ESTHER3D is captured in a motion capture studio by the same head-mounted stereo rig (calibrated with Kalibr, 6.2 cm baseline) synchronized with a Vicon system, giving true metric ground truth (MANO recovered from gloves/BVH hand skeletons plus raw marker trajectories). It

---

totals about 10 hours from 6 subjects over 50 scripted manipulation tasks, distributed across six instrumented scene zones that mirror the six training industries: retail shelving (12 tasks), a life services corridor for walking/inspection (10), housekeeping (8), factory workbench (7), office reception (7), and dining (6). Each zone uses textured backgrounds so the onboard visual odometry stays stable, and every method is evaluated zero-shot.

To cover manipulation broadly, the protocol is organized around a taxonomy of 14 elementary hand actions, and the 50 tasks are designed so that every action is exercised. Each action maps to a concrete manipulation: wipe (surface and equipment cleaning), grasp (picking goods and tools), place (aligning and setting down), twist (turning screws and caps), fold/spread (folding garments, making beds), gather (clearing and tidying), shelve (restocking, book returns), press (buttons and switches), carry (two-handed lifting), flip (turning objects and pages), point (guiding and reception gestures), bimanual (asymmetric bimanual assembly and packaging), pinch (picking small parts and screws), and handover (passing an object to another person). Table 1 of the main paper reports the accuracy of each method on this set. Figure 2 visualizes the motion capture ground truth used for evaluation: MANO hand skeletons recovered from gloves and projected into the head-mounted camera across six test sequences and seven timesteps each.

### I. Teacher Pipeline Validation

We validate the teacher pipeline on the 10-hour motion capture test set of Appendix H, where it runs without access to the ground truth.

Mapping ground truth into camera coordinates. Predictions are compared to ground truth directly in the camera frame, with no individual clip alignment. The camera's pose in the mocap world has six degrees of freedom and comes from a rigid body with four markers on the rig combined with the calibrated extrinsic transform from the device to the camera; ground truth hands are mapped into the camera via a fixed similarity solved from wrist markers (residual 10.4 mm, approximately the offset between joint and marker). Each link of the chain is verified independently: forward kinematics bone lengths, joint correspondence via a shuffle test, and time synchronization solved twice by independent procedures: direct marker/detection alignment and a pose sweep, agreeing to 10 ms. Evaluation covers both hands throughout the 10 hours of mocap test video.

The teacher's accuracy on this set (and the comparison of methods) is reported in the main paper; here we only give the measurement chain above and additional ground truth examples in Figure 2.

<div style="text-align: center;">Table 2. Bootstrap statistics on the full 10-hour ESTHER3D test set (2000 resamples): mean ± std where available. Errors in mm, jitter in m/s². ESTHER's MPJPE is the updated point estimate only; its jitter 95% interval does not overlap any baseline's.</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'>Method</td><td style='text-align: center; word-wrap: break-word;'>MPJPE $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>PA-MPJPE $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>Wrist $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>Jitter $\downarrow$</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Hiera-FTL</td><td style='text-align: center; word-wrap: break-word;'>43.4 $\pm$ 1.2</td><td style='text-align: center; word-wrap: break-word;'>10.0 $\pm$ 0.2</td><td style='text-align: center; word-wrap: break-word;'>277 $\pm$ 5</td><td style='text-align: center; word-wrap: break-word;'>79.5 $\pm$ 3.6</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Epipolar Trans.</td><td style='text-align: center; word-wrap: break-word;'>19.0 $\pm$ 0.5</td><td style='text-align: center; word-wrap: break-word;'>9.0 $\pm$ 0.1</td><td style='text-align: center; word-wrap: break-word;'>20.9 $\pm$ 0.5</td><td style='text-align: center; word-wrap: break-word;'>13.3 $\pm$ 0.4</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>MVGFormer</td><td style='text-align: center; word-wrap: break-word;'>19.2 $\pm$ 0.6</td><td style='text-align: center; word-wrap: break-word;'>8.9 $\pm$ 0.1</td><td style='text-align: center; word-wrap: break-word;'>21.0 $\pm$ 0.4</td><td style='text-align: center; word-wrap: break-word;'>8.7 $\pm$ 0.3</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>POEM-v2</td><td style='text-align: center; word-wrap: break-word;'>19.4 $\pm$ 0.6</td><td style='text-align: center; word-wrap: break-word;'>8.9 $\pm$ 0.1</td><td style='text-align: center; word-wrap: break-word;'>22.5 $\pm$ 0.4</td><td style='text-align: center; word-wrap: break-word;'>8.5 $\pm$ 0.3</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>ESTHER (ours)</td><td style='text-align: center; word-wrap: break-word;'>17.7</td><td style='text-align: center; word-wrap: break-word;'>8.6 $\pm$ 0.1</td><td style='text-align: center; word-wrap: break-word;'>21.3 $\pm$ 0.5</td><td style='text-align: center; word-wrap: break-word;'>3.3 $\pm$ 0.1</td></tr></table>

### J. Statistical Rigor: Bootstrap Confidence Intervals

The architectures in Table 1 of the main paper share frozen image evidence, training data, forward kinematics losses, and budget. To assess sampling variability, we bootstrap clip scores from the full 10-hour ESTHER3D motion capture test set using 2000 resamples (Table 2). ESTHER's updated MPJPE point estimate is 17.7 mm; its corresponding bootstrap uncertainty is not reported here. For jitter, ESTHER's 95% interval [3.10, 3.52] lies below the best baseline's [7.98, 9.17]. These intervals quantify sampling variability for a fixed checkpoint, not variation across training seeds. The wrist error (21.3 mm) is on par with the best multiview methods. The perturbation sweeps in Tables 2 and 3 of the main paper use this same model.

### K. Robustness to Failure Modes

Beyond the missing views and dropped frames of the main paper, we stress ESTHER with seven photometric and temporal corruptions applied to the ESTHER3D test frames, spanning the main egocentric failure modes: overexposure, underexposure, and nonlinear gamma (lighting), motion blur, sensor noise, a hue shift (skin tone / color), and desaturation (Table 3, Figure 3). The gloved appearance gap is the test condition itself (Appendix H): ESTHER is trained on bare in-the-wild hands, yet every number here is measured on the gloved motion capture subject, so the shift from bare hands to gloves is already priced in.

Across all seven corruptions the metrics barely move: MPJPE stays within 0.5 mm of the clean 17.7 (worst case 18.2 under sensor noise), PA-MPJPE within 0.2 mm, wrist within about 2 mm of the clean 21.3, and jitter within approximately 0.4 m/s $^{2}$. Frozen DINOv3 evidence and heavy image augmentation during training make the encoder largely invariant to appearance, and the temporal motion encoder absorbs blur as it absorbs dropped frames. Skin tone (hue) and illumination shifts in particular leave MPJPE unchanged at 17.7 mm, matching clean input.

### L. On-Screen Gating and Jitter Protocol

Pose metrics. MPJPE measures wrist-relative joint error; PA-MPJPE additionally applies Procrustes alignment. MPJPE-absolute denotes joint error without wrist alignment.

---

![Image](images/c37014c6f3ef3da1c2ef4fba9cc2a52ea679e0c93bdf63983541c14df70b52e2.jpg)

<div style="text-align: center;">Figure 2. Motion capture ground truth of the ESTHER3D test set. Six sequences (Seq 1–6), shown in two rows for the left eye (L, cam0) and right eye (R, cam1), across seven timesteps (columns, labeled by time). The ground truth hands (MANO fitted to the motion capture hand joints) are projected into each view; they are the true metric supervision against which all methods are evaluated zero-shot.</div>

---

![Image](images/5e22adcb418976d756c139408f554f0dbded34b9457fdaa003d051ff9e7ff6d7.jpg)

<div style="text-align: center;">Figure 3. ESTHER under input corruptions on an ESTHER3D motion capture clip: MANO meshes projected on the degraded left view (top) and shown in 3D (bottom), for the clean input (a) and seven corruptions (b–h) spanning lighting, blur, noise, and color shifts. The mesh stays stable throughout.</div>

<div style="text-align: center;">Table 3. ESTHER under input corruption on the ESTHER3D test set. Errors in mm, jitter in m/s $^2$; MPJPE under corrupted input is within 0.5 mm of the clean condition.</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'>Corruption</td><td style='text-align: center; word-wrap: break-word;'>MPJPE</td><td style='text-align: center; word-wrap: break-word;'>PA-MPJPE</td><td style='text-align: center; word-wrap: break-word;'>Wrist</td><td style='text-align: center; word-wrap: break-word;'>Jitter</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>clean</td><td style='text-align: center; word-wrap: break-word;'>17.7</td><td style='text-align: center; word-wrap: break-word;'>8.60</td><td style='text-align: center; word-wrap: break-word;'>21.3</td><td style='text-align: center; word-wrap: break-word;'>3.31</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>overexposure $\times 1.6$</td><td style='text-align: center; word-wrap: break-word;'>17.6</td><td style='text-align: center; word-wrap: break-word;'>8.66</td><td style='text-align: center; word-wrap: break-word;'>20.1</td><td style='text-align: center; word-wrap: break-word;'>3.71</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>low light $\times 0.5$</td><td style='text-align: center; word-wrap: break-word;'>17.7</td><td style='text-align: center; word-wrap: break-word;'>8.57</td><td style='text-align: center; word-wrap: break-word;'>21.5</td><td style='text-align: center; word-wrap: break-word;'>3.34</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>gamma 2.2</td><td style='text-align: center; word-wrap: break-word;'>17.7</td><td style='text-align: center; word-wrap: break-word;'>8.49</td><td style='text-align: center; word-wrap: break-word;'>21.9</td><td style='text-align: center; word-wrap: break-word;'>3.33</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>motion blur $k=15$</td><td style='text-align: center; word-wrap: break-word;'>18.0</td><td style='text-align: center; word-wrap: break-word;'>8.66</td><td style='text-align: center; word-wrap: break-word;'>19.5</td><td style='text-align: center; word-wrap: break-word;'>3.24</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>sensor noise $\sigma=25$</td><td style='text-align: center; word-wrap: break-word;'>18.2</td><td style='text-align: center; word-wrap: break-word;'>8.68</td><td style='text-align: center; word-wrap: break-word;'>21.6</td><td style='text-align: center; word-wrap: break-word;'>3.73</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>hue shift $+25$, skin tone</td><td style='text-align: center; word-wrap: break-word;'>17.7</td><td style='text-align: center; word-wrap: break-word;'>8.60</td><td style='text-align: center; word-wrap: break-word;'>21.9</td><td style='text-align: center; word-wrap: break-word;'>3.37</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>desaturate $\times 0.4$</td><td style='text-align: center; word-wrap: break-word;'>17.8</td><td style='text-align: center; word-wrap: break-word;'>8.64</td><td style='text-align: center; word-wrap: break-word;'>22.1</td><td style='text-align: center; word-wrap: break-word;'>3.39</td></tr></table>

The ablations in the main paper use MPJPE on the ES-THER3D test set, with full ESTHER at 17.7 mm.

Jitter definition. Jitter is the mean second temporal difference of the joint positions,  $\|\mathbf{S}_{t-1}-2\mathbf{S}_t+\mathbf{S}_{t+1}\|$ averaged over the 21 joints and scaled by  $fps^2$ (fps = 30), taken only over triplets whose three frames are all on-screen. No acceleration is formed across an off-screen frame or an on/off boundary, so a hand that is gated off contributes nothing to jitter.

On-screen gating and off-screen frames. A hand is on-screen (ground truth) when at least half of its 21 joints project inside the image in at least one eye; ESTHER predicts hand visibility probabilities that we threshold at 0.5. Pose, wrist, and jitter are scored only on frames where the hand is visible. Frames without a visible hand have no valid target, so gating determines which frames enter each metric.

Gate accuracy and its effect on jitter. On the full 10-hour motion capture test set the gate is exact: precision, recall, and $F_1$ are all 1.00 over all evaluated hand frames (no false positives or negatives), so jitter over all frames and with ground truth or predicted gating coincides at $3.31 \, m/s^2$; the reported value therefore does not depend on the gating protocol here. To probe gate errors we run a stress test with the hand outside the view that blanks the middle four frames of every clip. The gate flags them off-screen (recall 1.00); jitter gated by predictions holds at $3.35 \, m/s^2$, whereas ungated jitter rises to 4.61 because the on/off/on transition injects a spurious acceleration spike. A missed off-screen frame (a false negative) would thus inflate jitter through such boundary spikes; the gate is what keeps the metric clean.

### M. HOT3D Robustness to Undistortion Strategy

Because the HOT3D cameras are wide-FOV Fisheye624 (Appendix F), the rectified pinhole crop fed to the network is a design choice, and a natural question is how much the results depend on it. On a held-out HOT3D split we rerun every method under four undistortion strategies: axis-aligned wide-pinhole undistortion at a virtual focal of 120 px (offbb120, our default), the same pipeline at 90 px (narrow) and 150 px (wide), and a ray-aligned adaptive-focal hand crop with minimal black border (hc). This is not a direct accuracy comparison: ESTHER is trained on the offbb120 rectification and the multiview baselines use their released weights, which would make an absolute ranking unfair. What Table 4 measures instead is each method's stability: how much its own numbers move when only the rectification changes, all else fixed.

ESTHER is essentially invariant to the rectification choice. Its MPJPE stays within 32.3–32.5 mm and its jitter within 16.3–16.8 m/s $^2$ across the three offbb focals, including the 90 and 150 px settings it never saw in training. In contrast, the multiview baselines vary substantially: POEM's wrist error moves 656 → 450 → 928 mm across the focal sweep alone, and the methods based on anchors only recover on the hc strategy that crops the hand directly. This follows from direct absolute wrist prediction: ESTHER's decoded wrist does not depend on a triangulated anchor whose geometry shifts with the rectification, so the reconstruction is stable under the undistortion pipeline rather than tuned to one.

### N. Label Quality Analysis

The motion capture test set measures the labeling procedure against true metric ground truth, but only in the studio. Here we report 2D agreement and geometric sensitivity to first order of the in-the-wild labels.

---

<div style="text-align: center;">Table 4. HOT3D robustness to undistortion strategy on a held-out split. Each cell: MPJPE / wrist (mm) / jitter (m/s²). The comparison measures stability across strategies: ESTHER barely moves (including on the 90/150 px focals it was not trained on) while the multiview baselines swing widely.</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'>Method</td><td style='text-align: center; word-wrap: break-word;'>offbb120</td><td style='text-align: center; word-wrap: break-word;'>offbb90</td><td style='text-align: center; word-wrap: break-word;'>offbb150</td><td style='text-align: center; word-wrap: break-word;'>hc</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>ESTHER (ours)</td><td style='text-align: center; word-wrap: break-word;'>32.4 / 20.8 / 16.3</td><td style='text-align: center; word-wrap: break-word;'>32.5 / 22.8 / 16.6</td><td style='text-align: center; word-wrap: break-word;'>32.3 / 23.6 / 16.8</td><td style='text-align: center; word-wrap: break-word;'>28.9 / 48.0 / 15.6</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>POEM-v2</td><td style='text-align: center; word-wrap: break-word;'>101.3 / 656 / 635</td><td style='text-align: center; word-wrap: break-word;'>100.9 / 450 / 395</td><td style='text-align: center; word-wrap: break-word;'>103.6 / 928 / 1044</td><td style='text-align: center; word-wrap: break-word;'>46.3 / 65.7 / 48</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>MVGFormer</td><td style='text-align: center; word-wrap: break-word;'>95.9 / 429 / 449</td><td style='text-align: center; word-wrap: break-word;'>95.9 / 291 / 265</td><td style='text-align: center; word-wrap: break-word;'>97.6 / 633 / 767</td><td style='text-align: center; word-wrap: break-word;'>43.1 / 69.9 / 52</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Epipolar Trans.</td><td style='text-align: center; word-wrap: break-word;'>69.4 / 414 / 318</td><td style='text-align: center; word-wrap: break-word;'>70.6 / 303 / 191</td><td style='text-align: center; word-wrap: break-word;'>69.9 / 571 / 541</td><td style='text-align: center; word-wrap: break-word;'>43.2 / 66.9 / 66</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Hiera-FTL</td><td style='text-align: center; word-wrap: break-word;'>80.9 / 401 / 105</td><td style='text-align: center; word-wrap: break-word;'>78.7 / 390 / 100</td><td style='text-align: center; word-wrap: break-word;'>82.8 / 435 / 118</td><td style='text-align: center; word-wrap: break-word;'>78.1 / 377 / 91</td></tr></table>

Residual. For every labeled hand frame we project the 21 label joints into both rectified pinhole views through the calibrated rig and compare them with the fused 2D keypoints of Appendix A (YOLO11-pose and ViTPose, detected on the same views); the residual r is the mean pixel distance over all joints of both views whose confidence exceeds $\tau=0.3$. This is exactly the quantity of the fine 2D check (Appendix A), which rejects hand frames with $r > 15$ px. The statistics cover the in-the-wild training set of ESTHER3D (Appendix G).

From 2D residual to 3D error. A residual of r pixels at hand depth z corresponds to a displacement parallel to the image plane of $rz/f$, the component of the 3D error that 2D observes directly. The component along the viewing ray is only constrained through the disparity between the two views: a disparity error $\delta d$ shifts the triangulated depth by $\delta z = z^2\delta d/(fb)$. With independent errors in the two views $\delta d \approx \sqrt{2}r$; if both views err in opposite directions, $\delta d \leq 2r$. Table 5 reports distributions of these quantities over hand frames.

<div style="text-align: center;">Table 5. In-the-wild label quality from 2D (hand frames passing the 15 px check). Depth z is the label wrist depth.</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'>Quantity</td><td style='text-align: center; word-wrap: break-word;'>Median</td><td style='text-align: center; word-wrap: break-word;'>Mean</td><td style='text-align: center; word-wrap: break-word;'>p90</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Reprojection residual  r (px)</td><td style='text-align: center; word-wrap: break-word;'>5.3</td><td style='text-align: center; word-wrap: break-word;'>5.8</td><td style='text-align: center; word-wrap: break-word;'>8.6</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Hand depth  z (m)</td><td style='text-align: center; word-wrap: break-word;'>0.32</td><td style='text-align: center; word-wrap: break-word;'>0.33</td><td style='text-align: center; word-wrap: break-word;'>0.46</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>In-plane discrepancy $r_z/f$ (mm)</td><td style='text-align: center; word-wrap: break-word;'>2.8</td><td style='text-align: center; word-wrap: break-word;'>3.0</td><td style='text-align: center; word-wrap: break-word;'>4.6</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Depth per pixel of disparity $z^2/(fb)$ (mm)</td><td style='text-align: center; word-wrap: break-word;'>2.8</td><td style='text-align: center; word-wrap: break-word;'>3.2</td><td style='text-align: center; word-wrap: break-word;'>5.6</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Depth sensitivity, independent views (mm)</td><td style='text-align: center; word-wrap: break-word;'>21.3</td><td style='text-align: center; word-wrap: break-word;'>25.1</td><td style='text-align: center; word-wrap: break-word;'>44.2</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Depth sensitivity, opposing view errors (mm)</td><td style='text-align: center; word-wrap: break-word;'>30.1</td><td style='text-align: center; word-wrap: break-word;'>35.5</td><td style='text-align: center; word-wrap: break-word;'>62.5</td></tr></table>

94.6% of the hand frames pass the 15 px check and 94.6% of those lie below 10 px.

Interpretation. The approximately 3 mm in-plane discrepancy and 21 mm depth sensitivity summarize reprojection agreement and error propagation under the stated assumptions. They do not bound true 3D label error, since the detectors used for fitting can share errors with the labels. The 21 mm quantity assumes independent view errors; opposing view errors give the separate 30.1 mm median in Table 5. Metric accuracy is measured against motion capture: teacher MPJPE is 15.9 mm and wrist error is 19.1 mm, as reported in the main paper.

### O. Ethics Statement

Participants in both collections gave informed consent before recording and were compensated. Consent covered model training, evaluation, and release as an anonymized benchmark for noncommercial research. The recordings contain hands and manipulated objects/workspaces, with no personally identifying facial imagery.