# MINT: A Unified Model for World-Space Camera and Hand Motion Estimation from Scalable Egocentric Pipeline Supervision

Zijie Zhu $^{1,3,4}$, Weiren Cai $^{3}$, Yizhou Wang $^{1,3}$, Zhenjie Yang $^{4}$, Yide Liu $^{3,5}$, Jiahao Chen $^{3,*}$, and Guanqi He $^{2,3,*}$

![Image](images/00f1789fbd041e20afb5ad3be7a19e0734d92cfe5b8fc73c0e5c3e7842190294.jpg)

![Image](images/50a1238be82af018ec97b29c526d128f1fbb43cd225f6d4b87ae5dc01b5e6e64.jpg)

<div style="text-align: center;">Fig. 1. MINT converts ordinary egocentric video into world-space motion supervision, and MINT recovers that motion with a single feed-forward model. Top: EgoPIPELINE labels 1,021 hours of public egocentric video drawn from Ego4D [^8], EPIC-KITCHENS [^4] and EgoDex [^10] (center) with camera trajectories and bimanual MANO [^21] states; each pair shows an input frame and the recovered hands reprojected onto it. Bottom: trained on that supervision, without any data from the evaluation domains, MINT transfers zero-shot to unseen domains.</div>

Abstract—Recovering camera and hand motion in world coordinates from egocentric video is a key capability for activity understanding, robot learning, and augmented reality. Existing systems typically decompose this problem into separate stages for camera motion, depth estimation, hand reconstruction, and trajectory refinement, resulting in substantial computational overhead and preventing the joint modeling of camera and hand motion. We introduce MINT (Minting IN-the-Wild Trajectories), a foundation model for world-space hand motion reconstruction from ego-centric RGB video. From a single shared spatiotemporal video representation, MINT jointly predicts the camera trajectory, field of view (FoV), camera-frame hand states, and per-frame hand observability, and then produces world-space hand motion via explicit coordinate transformations. Training such a model at scale is challenging, since paired world-space camera and hand annotations are scarce. We therefore develop an open-source labeling EgoPipeline that converts large collections of public egocentric videos into structured camera-and-hand trajectory supervision. MINT is first pretrained on these large-scale pseudo-labels and then fine-tuned on a small set of high-quality camera-and-hand annotations. Across public benchmarks MINT approaches state-of-the-art accuracy without seeing either benchmark in training, reaching 0.945 frame accuracy, 13.646 mm PA-MPJPE-p and 55.058 px EPE-p for camera-frame bimanual reconstruction on HOT3D, 4.690 mm RPE-T and 0.284° RPE-R for camera trajectory, and a 3.67× end-to-end speedup over the labeling pipeline that supervises it. We release the model, training and inference code, labeling pipeline, and a curated 1,021-hour egocentric trajectory dataset.

Index Terms—egocentric video; world-space hand reconstruction; camera trajectory; pipeline amortization; cross-domain generalization

### I. INTRODUCTION

Egocentric video records rich human motion and hand-object interactions, and recovering their world-space motion provides an important source of motion supervision for embodied AI training [^14], including robot imitation learning [^11]. Yet accurate world-space supervision remains scarce. Capturing such supervision typically requires specialized sensing, precise calibration, synchronized devices, or costly motion reconstruction, making it difficult to obtain large-scale, high-quality world-space supervision from publicly available egocentric video.

Existing reconstruction methods decompose egocentric motion into separate modules for hand detection, motion reconstruction, and camera trajectory estimation. Such pipelines can recover individual sequences, but typically require complex optimization procedures and substantial engineering effort. Hand estimation depends on a detector and can fail under severe occlusion, while errors in cascaded reconstruction propagate across stages [^28], [^34], and representations are difficult to share across tasks. Meanwhile, large-scale 3D reconstruction models provide a basis for jointly modeling camera and hand motion. Their pretraining captures long-range temporal correspondence, camera mo

---

tion, and scene geometry within a shared spatio-temporal representation. We therefore ask whether such a unified geometric representation can support both camera and hand motion reconstruction.

We present MINT, a unified model for recovering world-space camera and bimanual hand motion from egocentric RGB video. MINT adapts a pretrained geometric reconstruction model to egocentric data and predicts camera extrinsics, horizontal and vertical fields of view, left- and right-hand MANO $^{[21]}$ states, and per-frame hand observability from a shared representation. It requires no external hand detector, motion infiller, or test-time optimization, and no dense depth or point-map intermediate. For each 32-frame window, all outputs are obtained in a single encoder forward pass, while videos of arbitrary length are covered by chaining overlapping windows (Sec. IV-B), enabling unified motion reconstruction.

Real-world egocentric videos contain diverse scenes and hand motion, but lack accurate world-space hand supervision. We therefore build EgoPIPELINE to produce motion supervision at scale from ordinary video. We process 1,729 hours from Ego4D [^8], EPIC-KITCHENS [^4], and EgoDex [^10] into 1,021 hours of world-space camera and hand motion supervision (Fig. 1). Because the camera trajectories in this supervision inherit the scale ambiguity of monocular reconstruction [^36], we introduce a second training stage that adapts the camera head alone using a human egocentric dataset captured with high-precision devices. The pseudo-labels provide broad coverage, while this dataset provides precise absolute scale.

Neither HOT3D [^1] nor ARCTIC [^7] is included in the training data; both are therefore evaluated in the zero-shot setting. On HOT3D, MINT approaches state-of-the-art accuracy on all nine metrics we report, remaining close to a model trained on that benchmark. For camera trajectory estimation, MINT achieves the lowest relative translation error on ARCTIC.

## Our contributions are

- Unified egocentric reconstruction. MINT jointly recovers camera state, field of view, bimanual hand state, and hand observability from a unified spatio-temporal geometric representation. Each 32-frame window requires a single encoder forward pass, with no detector, motion infiller, or test-time optimization.

- Scalable egocentric data production. EgoPIPELINE converts large-scale ordinary egocentric video into structured world-space motion supervision and combines broad, automatically reconstructed data with a small amount of precise geometric supervision.

- Large-scale structured egocentric data. We provide a broad corpus of reconstructed egocentric motion containing camera trajectories, bimanual hand state, world-space motion, hand observability, and action descriptions.

### II. Related Work

### A. Egocentric Video Understanding

Large-scale egocentric video datasets, including Ego4D [^8], EPIC-KITCHENS [^4], Ego-Exo4D [^9], and EgoDex [^10], provide important resources for learning human actions and visual-language representations from diverse real-world activities. ViTRA [^14] further demonstrates the value of large-scale real-life human videos for pretraining, covering diverse tasks, objects, and environments and improving zero-shot action prediction. Extracting 3D hand-motion supervision from such videos typically requires multiple stages, including camera calibration, depth estimation, visual SLAM, hand reconstruction, and trajectory processing, resulting in a complex data production pipeline with substantial engineering effort for optimization and deployment.

### B. Video-Based Hand Pose Estimation

MANO [^21] provides a generic low-dimensional parameterization from images to hand pose and shape. Monocular hand reconstruction methods mainly recover hand pose and shape by estimating MANO [^21] parameters from RGB input. HaMeR [^18] employs a pretrained ViT with large-scale training data to improve generalization to real-world images, while WiLoR [^19] combines hand localization and 3D reconstruction for single-frame hand-state estimation [^25]. Video-based methods further introduce temporal modeling. Dyn-HaMR [^37] incorporates a two-hand interaction motion prior [^6] with test-time optimization, while HaWoR [^39] combines camera-frame hand estimation, egocentric SLAM trajectory estimation, and motion infilling based on monocular metric depth [^36]. ViDiHand [^33] exploits the spatiotemporal prior of a pretrained video diffusion model to jointly estimate hand presence and temporally stable 3D hand pose without an explicit detector or test-time optimization, while the reconstructed hand states remain in the camera frame.

The 3D supervision used by these methods is primarily obtained from dedicated sensing systems and staged laboratory captures, including the multi-depth-camera setup of DexYCB [^2], the head-mounted system of HOT3D [^1], and the multi-view marker-based motion capture system of ARCTIC [^7]. These systems provide accurate 3D annotations but rely on specialized acquisition setups, with limited coverage of hand motions, object interactions, and real-world scenes. Egocentric body-and-hand estimation faces the same supervision bottleneck [^35].

### C. Feed-Forward 3D Reconstruction

Camera and scene reconstruction has increasingly moved from per-video optimization toward learned feed-forward estimation [^12], [^27]. DROID-SLAM [^24] combines deep networks with differentiable optimization to jointly estimate camera poses and dense geometry, while MegaSaM [^13] incorporates learned geometric representations into SLAM for dynamic and casually captured videos.

Recent feed-forward methods directly predict camera and scene geometry from images or videos [^29], [^40]. VGGT [^32]

---

![Image](images/8981b65a7145cb10a9bbaff9c5d07726d670b50211aaa9441751dfb02882967a.jpg)

<div style="text-align: center;">Fig. 2. Overview of EgoPIPELINE. Egocentric videos are first pre-filtered by hand detection. The retained clips are then processed by GeoCalib for camera intrinsics, MoGe-2 for monocular depth, MegaSaM for camera pose, and HaWoR for bimanual hand reconstruction. A final post-processing stage stabilizes the recovered trajectories and composes the camera-frame hand states with the camera trajectory to produce world-space bimanual motion supervision.</div>

uses cross-view attention to estimate camera parameters and dense geometry from multiple frames, while $\pi^{3}$ [^31] employs permutation-equivariant modeling to predict affine-invariant camera poses and scale-invariant local point maps. LingBot-Map [^3] extends feed-forward reconstruction to long streaming videos by aggregating historical observations into a spatiotemporal geometric representation, enabling continuous estimation of camera trajectories and scene geometry over long sequences. These works establish a feed-forward paradigm for geometric reconstruction from long video sequences. Our work extends this paradigm to egocentric hand reconstruction by jointly modeling camera and bimanual motion and recovering temporally consistent 3D hand motion in a unified world coordinate system.

### III. EgoPipeline Structured Supervision

Existing egocentric hand reconstruction and hand–object interaction datasets are predominantly collected in controlled environments with predefined tasks, limiting the diversity of bimanual interactions and real-world conditions they capture [^1], [^2], [^7]. In contrast, publicly available egocentric videos cover diverse activities, objects, environments, and interactions. To exploit this diversity for large-scale 3D motion learning, we collect 1,729 hours of egocentric video spanning open-ended daily activities, fine-grained bimanual manipulation on tabletops, and structured kitchen interactions. These videos contain visual and motion variations that are difficult to reproduce in controlled capture, including complex backgrounds, dynamic object interactions, severe hand occlusions, variations in hand scale, and illumination conditions.

To extract 3D motion supervision from these videos, we develop a first-person video processing pipeline inspired by the data construction paradigm of ViTRA [^14], illustrated in Fig. 2. We first compress and encode the videos for improved I/O throughput and use high-confidence YOLO detections to filter clips with no detected hands for more than 1 s or more than two detected hands. For the remaining clips, we estimate camera intrinsics with GeoCalib [^26], undistort the frames, and inject the monocular metric depth prior [^36] from MoGe-2 [^30] into MegaSaM [^13] or DROID-SLAM [^24] to recover metric-scale camera trajectories. In parallel, HaWoR [^39] reconstruct both hands and estimates per-frame motion states in the camera coordinate system, with each hand represented by a 6D wrist pose and MANO [^21] parameters. We then combine the camera-space hand states with the recovered camera trajectory to obtain world-space bimanual motion.

Because HaWoR relies on YOLO-based hand detections, reconstruction failures and intermittent detections can produce outliers, abrupt trajectory changes, and short missing intervals. We therefore apply unified post-processing with outlier removal, missing-interval interpolation, and UKF temporal filtering to stabilize the recovered trajectories and provide more reliable motion supervision. Finally, we construct 1,021 hours of effective data with world-coordinate 3D bimanual motion supervision.

Despite enabling large-scale supervision generation, the pipeline relies on multiple heterogeneous models for hand detection, camera calibration, depth estimation, SLAM, and hand reconstruction. This design incurs substantial computational and engineering overhead, while errors from different

---

![Image](images/747afba1f759abc5cb7fc468db3de72cde5eb8d2181b7466bb7b3bd47e2bd79a.jpg)

<div style="text-align: center;">Fig. 3. Architecture of MINT. Each window of 32 egocentric frames is encoded at  $378 \times 518$ by a DINOv2 ViT-L/14 patch embedding, and the resulting frame tokens are processed by the spatiotemporal geometric aggregator of GCT, which alternates frame attention and global attention so that every frame of the window is encoded with bidirectional temporal context. The unified visual-geometric features carry camera, register, scale, and patch tokens, and feed four prediction heads for camera extrinsics, field of view, bimanual MANO parameters, and hand observability. Composing the predicted camera trajectory with the camera-frame hand states yields world-space bimanual reconstruction.</div>

stages can propagate through the reconstruction process. We therefore propose MINT, a unified feed-forward egocentric 3D reconstruction framework that jointly models camera and bimanual motion directly from video. MINT learns their coupled geometry and predicts camera and hand motion in a common world coordinate system.

### IV. Method

### A. Overview

Monocular hand reconstruction aims to recover 3D hand pose and motion from egocentric videos. Existing methods typically rely on a pipeline of hand detection, camera tracking, hand fitting, and post-processing, which requires substantial engineering effort, accumulates errors across stages, and limits overall processing throughput. We therefore leverage a pretrained Geometric Context Transformer (GCT), the geometric encoder of LingBot-Map [^3] trained for long-range 3D reconstruction from video, to learn a unified visual-geometric representation for hand reconstruction. GCT encodes a window of frames in a single bidirectional spatiotemporal pass, and four task heads regress camera extrinsics, field of view (FoV), MANO parameters for both hands, and hand observability, as shown in Fig. 3. MINT recovers camera and bimanual hand motion in the world coordinate frame from egocentric video with a single feed-forward model, without a hand detector, motion infiller, or test-time optimization. A window is processed in one pass, and videos longer than the window are covered by chaining windows as described in Sec. IV-B.

We adopt a two-stage training strategy. In the first stage, the visual-geometric representation and task heads are jointly trained on diverse egocentric datasets to establish coupled camera and hand reconstruction. In the second stage, while preserving the learned geometric representation and hand reconstruction capability, we adapt only the camera extrinsics head on a small in-house corpus whose camera trajectories are captured with high-precision devices. This curriculum transfers long-range 3D reconstruction priors to egocentric hand motion recovery while specializing camera estimation for accurate world-space annotation.

### B. Visual Geometry Grounded Backbone

Egocentric video interleaves articulated hand motion, hand–object contact, camera egomotion, and the surrounding 3D layout in a single stream, and none of them is fully recoverable from an isolated frame. To obtain temporally consistent geometric features, MINT adopts the pretrained geometric context transformer (GCT) of LingBot-Map [^3], a visual geometry grounded transformer [^32], as its shared spatiotemporal geometric encoder, in contrast to pairwise reconstruction backbones [^12], [^27]. Reconstruction training over a broad distribution of scenes has established in that model robust geometric priors for scene structure, viewpoint change, and trajectory consistency, and its aggregation of features along a video sequence also matches the temporal input of egocentric recordings more closely than models built for unordered image sets.

a) Geometric feature extraction: We retain the original visual encoder and geometric aggregator so that these pretrained priors are inherited. Each frame is resized to $378 \times 518$ and encoded by a ViT-L/14 encoder, initialized from DINOv2 [^17] and pretrained jointly with the aggregator, into 999 tokens of dimension 1024. One camera token, four register tokens, and one scale token are prepended to every frame, so that frame-level global quantities are carried in dedicated slots. The sequence then alternates between frame attention and global attention,

 $$ \begin{array}{r}{\mathbf{X}_{1:T}^{l}=G_{\mathrm{g l o b a l}}^{l}\left(G_{\mathrm{f r a m e}}^{l}(\mathbf{X}_{1:T}^{l-1})\right),}\end{array} $$

---

where a frame module models the spatial structure inside a single image and a global module propagates geometric context along time, so that egomotion is inferred from cross-frame evidence rather than from single-frame visual cues. Features taken from several depths are aggregated into shared prediction features, which the camera and bimanual heads of Sec. IV-C read from the same forward pass.

b) Long-sequence inference: The global attention of GCT operates on a clip of fixed length and is trained at T = 32 frames, so extending attention across a complete recording would let computation and memory grow with sequence length. We therefore segment a video of arbitrary length into overlapping sliding windows. Each window is encoded independently by the same shared network, the camera trajectories of adjacent windows are aligned and chained in SE(3), and the hand parameters and observability scores of the frames two windows share are fused. This design bounds the computation of a single pass to one fixed window while keeping the total cost approximately linear in the length of the video, which supports continuous geometric estimation over long sequences.

### C. Prediction Heads

Through large-scale geometric reconstruction pretraining, GCT acquires strong priors for scene geometry, viewpoint change, and temporal motion, allowing its features to encode scene structure, viewpoint changes, and cross-frame motion. However, these generic geometric features do not directly specify the structured state required for hand reconstruction. Unlike rigid scene geometry, egocentric hand reconstruction must jointly recover wrist pose, articulated joint motion, and hand shape while handling occlusion and partial observability during hand–object interaction. We therefore introduce dedicated hand and camera prediction heads on top of these geometric features, mapping them to structured bimanual MANO [^21] states and camera states. The hand state is recovered in the camera frame and coupled with camera motion through a differentiable camera-to-world transformation, while a hand observability head suppresses unreliable supervision from partially observed hands.

For each hand $s \in \{L, R\}$, we parameterize its state as

 $$ \begin{array}{r}{\hat{h}_{t,s}=[\hat{\mathbf{\Lambda}}_{t,s},\hat{\mathbf{\Gamma}}_{t,s},\hat{\mathbf{\Theta}}_{t,s},\hat{\mathbf{\beta}}_{t,s}]\in\mathbb{R}^{109},}\end{array} $$

where $\Lambda$, $\Gamma$, $\Theta$, and $\beta$ denote wrist translation, wrist rotation, fifteen joint rotations, and hand shape, respectively. All rotations use the continuous six-dimensional representation, resulting in 218 hand parameters for both hands at each frame.

The camera state is parameterized as

 $$ \hat{c}_{t}=[\hat{t}_{t},\hat{q}_{t},\hat{f}_{t}]\in\mathbb{R}^{9}, $$

comprising the camera translation $t_{t}$, a unit quaternion $q_{t}$ with rotation matrix $R_{t}$ under the world-to-camera convention, and the vertical and horizontal fields of view $f_{t}$.

a) Structured Hand Motion Prediction: Egocentric hand motion follows a structured kinematic model, and frame-wise features alone are insufficient to reliably recover the full MANO state, particularly under rapid motion and frequent occlusion. We therefore directly decode the structured states of both hands from the spatio-temporal geometric features.

Each hand carries four learnable queries corresponding to wrist translation, wrist rotation, joint rotations, and hand shape. Let $X_{t}$ denote the final-layer spatio-temporal tokens of frame t. Each query interacts with these tokens through

 $$ \begin{array}{r}{Q_{t,k}^{s}=\mathrm{C r o s s A t t n}(q_{k}^{s},X_{t},X_{t}),\quad k\in\{\Lambda,\Gamma,\Theta,\beta\}.}\end{array} $$

The four resulting features are jointly mapped to the MANO state.

Frame-wise decoding alone does not guarantee temporal consistency, since rapid motion and occlusion can cause discontinuities across neighboring frames. We therefore condition the hand queries on the previous prediction and perform two refinement iterations. The first iteration predicts absolute rotations, while subsequent iterations predict a bounded axis-angle increment in the current local frame:

 $$ \tilde{\omega}=\frac{\omega}{\|\omega\|}\omega_{\max}\tanh\left(\frac{\|\omega\|}{\omega_{\max}}\right),\qquad\omega_{\max}=30^{\circ}. $$

The update is then applied as $R^{(k+1)} = R^{(k)} \exp([\tilde{\omega}^{(k)}]_\times)$, and mapped back to the continuous six-dimensional representation. The residual layers are zero-initialized, making the refinement module an identity mapping at the beginning of training.

b) Camera State and Hand-Camera Geometric Coupling: The hand state is recovered in the camera frame, whereas the final hand motion must be expressed in a common world frame. The camera state therefore determines both camera motion and the mapping of the hand trajectory into world space.

We adopt the LingBot-Map [^3] camera head, which predicts camera pose through iterative refinement. The previous prediction is embedded and injected into a causal temporal trunk through adaptive layer normalization, which predicts the current update. We retain four refinement iterations and move the field of view out of the feedback path, reducing the iterative state from nine to seven dimensions.

The field of view is predicted by an independent branch:

 $$ \begin{array}{r}{\hat{\boldsymbol f}_{t}=\mathrm{S o f t p l u s}\big(H_{\mathrm{F o V}}(\boldsymbol X_{t}^{\mathrm{c a m}})\big),}\end{array} $$

where the camera token is first projected to 512 dimensions and processed by two temporal blocks. The predicted extrinsics and fields of view are then concatenated into the camera state in Eq. (3).

Let $p_{t,s}^{c}$ and $Q_{t,s}^{c}$ denote the wrist translation and rotation matrix of hand s in the camera frame. They are mapped to the world frame as

 $$ \boldsymbol{p}_{t,s}^{w}=\boldsymbol{R}_{t}^{\top}(\boldsymbol{p}_{t,s}^{c}-\boldsymbol{t}_{t}),\quad\boldsymbol{Q}_{t,s}^{w}=\boldsymbol{R}_{t}^{\top}\boldsymbol{Q}_{t,s}^{c}. $$

Joint rotations and hand shape remain invariant under this transformation. Since the transformation is differentiable, the world-frame loss can jointly constrain the camera and hand predictions, explicitly coupling their geometry.

---

<div style="text-align: center;">Table 1. Main comparison on the ARCTIC and HOT3D datasets. Best result per column in bold. ↑/↓ indicate the direction of improvement. MPJPE-p and PA-MPJPE-p are in mm, EPE-p in px, GO-p in degrees, CT-p in m and Jitter in mm/frame $^{2}$. MINT is evaluated in a zero-shot setting.</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td rowspan="2"></td><td rowspan="2">Method</td><td colspan="3">Detection</td><td colspan="2">3D Pose</td><td colspan="3">Orient. &amp; Position</td><td style='text-align: center; word-wrap: break-word;'>Temporal</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>FAcc $\uparrow$</td><td style='text-align: center; word-wrap: break-word;'>Recall $\uparrow$</td><td style='text-align: center; word-wrap: break-word;'>F1 $\uparrow$</td><td style='text-align: center; word-wrap: break-word;'>MPJPE-p $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>PA-MPJPE-p $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>EPE-p $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>GO-p $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>CT-p $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>Jitter $\downarrow$</td></tr><tr><td rowspan="8">ARCTIC</td><td style='text-align: center; word-wrap: break-word;'>InterWild [^16]</td><td style='text-align: center; word-wrap: break-word;'>0.878</td><td style='text-align: center; word-wrap: break-word;'>0.943</td><td style='text-align: center; word-wrap: break-word;'>0.959</td><td style='text-align: center; word-wrap: break-word;'>30.817</td><td style='text-align: center; word-wrap: break-word;'>15.952</td><td style='text-align: center; word-wrap: break-word;'>53.888</td><td style='text-align: center; word-wrap: break-word;'>25.386</td><td style='text-align: center; word-wrap: break-word;'>0.097</td><td style='text-align: center; word-wrap: break-word;'>46.577</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>HaMeR [^18]</td><td style='text-align: center; word-wrap: break-word;'>0.875</td><td style='text-align: center; word-wrap: break-word;'>0.943</td><td style='text-align: center; word-wrap: break-word;'>0.957</td><td style='text-align: center; word-wrap: break-word;'>29.197</td><td style='text-align: center; word-wrap: break-word;'>14.596</td><td style='text-align: center; word-wrap: break-word;'>65.289</td><td style='text-align: center; word-wrap: break-word;'>24.907</td><td style='text-align: center; word-wrap: break-word;'>0.095</td><td style='text-align: center; word-wrap: break-word;'>18.279</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Hamba [^5]</td><td style='text-align: center; word-wrap: break-word;'>0.833</td><td style='text-align: center; word-wrap: break-word;'>0.912</td><td style='text-align: center; word-wrap: break-word;'>0.941</td><td style='text-align: center; word-wrap: break-word;'>31.233</td><td style='text-align: center; word-wrap: break-word;'>17.168</td><td style='text-align: center; word-wrap: break-word;'>87.047</td><td style='text-align: center; word-wrap: break-word;'>27.822</td><td style='text-align: center; word-wrap: break-word;'>0.110</td><td style='text-align: center; word-wrap: break-word;'>15.357</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>WildHands [^20]</td><td style='text-align: center; word-wrap: break-word;'>0.879</td><td style='text-align: center; word-wrap: break-word;'>0.946</td><td style='text-align: center; word-wrap: break-word;'>0.960</td><td style='text-align: center; word-wrap: break-word;'>25.704</td><td style='text-align: center; word-wrap: break-word;'>13.941</td><td style='text-align: center; word-wrap: break-word;'>50.517</td><td style='text-align: center; word-wrap: break-word;'>22.320</td><td style='text-align: center; word-wrap: break-word;'>0.058</td><td style='text-align: center; word-wrap: break-word;'>12.972</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>OmniHands [^15]</td><td style='text-align: center; word-wrap: break-word;'>0.866</td><td style='text-align: center; word-wrap: break-word;'>0.949</td><td style='text-align: center; word-wrap: break-word;'>0.954</td><td style='text-align: center; word-wrap: break-word;'>29.674</td><td style='text-align: center; word-wrap: break-word;'>14.203</td><td style='text-align: center; word-wrap: break-word;'>51.505</td><td style='text-align: center; word-wrap: break-word;'>24.580</td><td style='text-align: center; word-wrap: break-word;'>0.087</td><td style='text-align: center; word-wrap: break-word;'>45.312</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>WiLoR [^19]</td><td style='text-align: center; word-wrap: break-word;'>0.919</td><td style='text-align: center; word-wrap: break-word;'>0.951</td><td style='text-align: center; word-wrap: break-word;'>0.974</td><td style='text-align: center; word-wrap: break-word;'>22.012</td><td style='text-align: center; word-wrap: break-word;'>11.873</td><td style='text-align: center; word-wrap: break-word;'>71.527</td><td style='text-align: center; word-wrap: break-word;'>17.358</td><td style='text-align: center; word-wrap: break-word;'>0.075</td><td style='text-align: center; word-wrap: break-word;'>24.091</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Dyn-HaMR [^37]</td><td style='text-align: center; word-wrap: break-word;'>0.842</td><td style='text-align: center; word-wrap: break-word;'>0.918</td><td style='text-align: center; word-wrap: break-word;'>0.951</td><td style='text-align: center; word-wrap: break-word;'>27.904</td><td style='text-align: center; word-wrap: break-word;'>17.017</td><td style='text-align: center; word-wrap: break-word;'>85.723</td><td style='text-align: center; word-wrap: break-word;'>25.951</td><td style='text-align: center; word-wrap: break-word;'>0.121</td><td style='text-align: center; word-wrap: break-word;'>12.840</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>HaWoR [^39]</td><td style='text-align: center; word-wrap: break-word;'>0.700</td><td style='text-align: center; word-wrap: break-word;'>0.817</td><td style='text-align: center; word-wrap: break-word;'>0.895</td><td style='text-align: center; word-wrap: break-word;'>45.357</td><td style='text-align: center; word-wrap: break-word;'>26.375</td><td style='text-align: center; word-wrap: break-word;'>158.062</td><td style='text-align: center; word-wrap: break-word;'>43.325</td><td style='text-align: center; word-wrap: break-word;'>0.149</td><td style='text-align: center; word-wrap: break-word;'>19.789</td></tr><tr><td rowspan="10">HOT3D</td><td style='text-align: center; word-wrap: break-word;'>MINT</td><td style='text-align: center; word-wrap: break-word;'>0.918</td><td style='text-align: center; word-wrap: break-word;'>0.957</td><td style='text-align: center; word-wrap: break-word;'>0.978</td><td style='text-align: center; word-wrap: break-word;'>26.527</td><td style='text-align: center; word-wrap: break-word;'>15.961</td><td style='text-align: center; word-wrap: break-word;'>55.038</td><td style='text-align: center; word-wrap: break-word;'>19.332</td><td style='text-align: center; word-wrap: break-word;'>0.069</td><td style='text-align: center; word-wrap: break-word;'>12.959</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>MINT + UKF</td><td style='text-align: center; word-wrap: break-word;'>0.918</td><td style='text-align: center; word-wrap: break-word;'>0.957</td><td style='text-align: center; word-wrap: break-word;'>0.978</td><td style='text-align: center; word-wrap: break-word;'>26.568</td><td style='text-align: center; word-wrap: break-word;'>15.955</td><td style='text-align: center; word-wrap: break-word;'>54.997</td><td style='text-align: center; word-wrap: break-word;'>19.353</td><td style='text-align: center; word-wrap: break-word;'>0.069</td><td style='text-align: center; word-wrap: break-word;'>2.621</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>InterWild [^16]</td><td style='text-align: center; word-wrap: break-word;'>0.669</td><td style='text-align: center; word-wrap: break-word;'>0.881</td><td style='text-align: center; word-wrap: break-word;'>0.868</td><td style='text-align: center; word-wrap: break-word;'>77.168</td><td style='text-align: center; word-wrap: break-word;'>24.811</td><td style='text-align: center; word-wrap: break-word;'>71.482</td><td style='text-align: center; word-wrap: break-word;'>58.501</td><td style='text-align: center; word-wrap: break-word;'>0.213</td><td style='text-align: center; word-wrap: break-word;'>101.164</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>HaMeR [^18]</td><td style='text-align: center; word-wrap: break-word;'>0.692</td><td style='text-align: center; word-wrap: break-word;'>0.904</td><td style='text-align: center; word-wrap: break-word;'>0.883</td><td style='text-align: center; word-wrap: break-word;'>68.314</td><td style='text-align: center; word-wrap: break-word;'>21.455</td><td style='text-align: center; word-wrap: break-word;'>59.077</td><td style='text-align: center; word-wrap: break-word;'>49.636</td><td style='text-align: center; word-wrap: break-word;'>0.102</td><td style='text-align: center; word-wrap: break-word;'>23.632</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Hamba [^5]</td><td style='text-align: center; word-wrap: break-word;'>0.632</td><td style='text-align: center; word-wrap: break-word;'>0.828</td><td style='text-align: center; word-wrap: break-word;'>0.853</td><td style='text-align: center; word-wrap: break-word;'>71.732</td><td style='text-align: center; word-wrap: break-word;'>29.620</td><td style='text-align: center; word-wrap: break-word;'>107.625</td><td style='text-align: center; word-wrap: break-word;'>56.525</td><td style='text-align: center; word-wrap: break-word;'>0.128</td><td style='text-align: center; word-wrap: break-word;'>18.507</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>WildHands [^20]</td><td style='text-align: center; word-wrap: break-word;'>0.655</td><td style='text-align: center; word-wrap: break-word;'>0.863</td><td style='text-align: center; word-wrap: break-word;'>0.844</td><td style='text-align: center; word-wrap: break-word;'>52.791</td><td style='text-align: center; word-wrap: break-word;'>28.946</td><td style='text-align: center; word-wrap: break-word;'>111.438</td><td style='text-align: center; word-wrap: break-word;'>53.933</td><td style='text-align: center; word-wrap: break-word;'>0.157</td><td style='text-align: center; word-wrap: break-word;'>22.885</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>OmniHands [^15]</td><td style='text-align: center; word-wrap: break-word;'>0.649</td><td style='text-align: center; word-wrap: break-word;'>0.895</td><td style='text-align: center; word-wrap: break-word;'>0.868</td><td style='text-align: center; word-wrap: break-word;'>63.281</td><td style='text-align: center; word-wrap: break-word;'>22.682</td><td style='text-align: center; word-wrap: break-word;'>68.437</td><td style='text-align: center; word-wrap: break-word;'>49.120</td><td style='text-align: center; word-wrap: break-word;'>0.133</td><td style='text-align: center; word-wrap: break-word;'>69.510</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>WiLoR [^19]</td><td style='text-align: center; word-wrap: break-word;'>0.827</td><td style='text-align: center; word-wrap: break-word;'>0.897</td><td style='text-align: center; word-wrap: break-word;'>0.937</td><td style='text-align: center; word-wrap: break-word;'>30.966</td><td style='text-align: center; word-wrap: break-word;'>19.980</td><td style='text-align: center; word-wrap: break-word;'>72.978</td><td style='text-align: center; word-wrap: break-word;'>25.746</td><td style='text-align: center; word-wrap: break-word;'>0.098</td><td style='text-align: center; word-wrap: break-word;'>17.976</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Dyn-HaMR [^37]</td><td style='text-align: center; word-wrap: break-word;'>0.614</td><td style='text-align: center; word-wrap: break-word;'>0.811</td><td style='text-align: center; word-wrap: break-word;'>0.802</td><td style='text-align: center; word-wrap: break-word;'>74.214</td><td style='text-align: center; word-wrap: break-word;'>38.201</td><td style='text-align: center; word-wrap: break-word;'>171.617</td><td style='text-align: center; word-wrap: break-word;'>43.851</td><td style='text-align: center; word-wrap: break-word;'>0.571</td><td style='text-align: center; word-wrap: break-word;'>44.942</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>HaWoR [^39]</td><td style='text-align: center; word-wrap: break-word;'>0.348</td><td style='text-align: center; word-wrap: break-word;'>0.499</td><td style='text-align: center; word-wrap: break-word;'>0.654</td><td style='text-align: center; word-wrap: break-word;'>71.396</td><td style='text-align: center; word-wrap: break-word;'>66.031</td><td style='text-align: center; word-wrap: break-word;'>327.294</td><td style='text-align: center; word-wrap: break-word;'>79.350</td><td style='text-align: center; word-wrap: break-word;'>0.262</td><td style='text-align: center; word-wrap: break-word;'>23.872</td></tr><tr><td rowspan="2">MINT</td><td style='text-align: center; word-wrap: break-word;'>MINT + UKF</td><td style='text-align: center; word-wrap: break-word;'>0.945</td><td style='text-align: center; word-wrap: break-word;'>0.983</td><td style='text-align: center; word-wrap: break-word;'>0.953</td><td style='text-align: center; word-wrap: break-word;'>29.926</td><td style='text-align: center; word-wrap: break-word;'>13.656</td><td style='text-align: center; word-wrap: break-word;'>55.117</td><td style='text-align: center; word-wrap: break-word;'>21.099</td><td style='text-align: center; word-wrap: break-word;'>0.196</td><td style='text-align: center; word-wrap: break-word;'>12.057</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>MINT + UKF</td><td style='text-align: center; word-wrap: break-word;'>0.945</td><td style='text-align: center; word-wrap: break-word;'>0.983</td><td style='text-align: center; word-wrap: break-word;'>0.953</td><td style='text-align: center; word-wrap: break-word;'>29.918</td><td style='text-align: center; word-wrap: break-word;'>13.646</td><td style='text-align: center; word-wrap: break-word;'>55.058</td><td style='text-align: center; word-wrap: break-word;'>21.091</td><td style='text-align: center; word-wrap: break-word;'>0.196</td><td style='text-align: center; word-wrap: break-word;'>2.373</td></tr></table>

c) Hand Observability: Hands in egocentric videos are frequently affected by object interaction, self-occlusion, and image-boundary truncation, making some frames unreliable for geometric supervision. We therefore introduce a lightweight hand observability head to determine whether each hand in the current frame can provide reliable supervision.

For each hand, a learnable query cross-attends to the current-frame patch tokens. The tokens are projected to 256 dimensions, and the head predicts the left- and right-hand observability logits $\hat{z}_{t,L}$ and $\hat{z}_{t,R}$. The corresponding probabilities $\hat{a}_{t,s}$ determine whether that hand contributes to the hand reconstruction loss and world-frame loss.

### D. Training Objectives

The egocentric reconstruction pipeline provides high-quality hand motion from large-scale videos, including wrist trajectories, joint motion, and hand–object interactions, while covering a broad range of tasks, scenes, and action variations. These data provide large-scale supervision for learning hand motion and geometric relationships in egocentric videos.

However, the corresponding camera trajectories typically rely on monocular depth and SLAM. Although monocular SLAM captures relative frame-to-frame motion well, its scale ambiguity can introduce systematic errors in the absolute trajectory scale. We therefore use two training stages: the first learns stable hand–camera geometry from large-scale and diverse data, while the second uses camera motion captured with high-precision devices to correct the remaining scale and absolute pose errors.

a) Stage 1: Learning Hand–Camera Coupled Geometry: The first stage does not directly fit an absolutely accurate camera trajectory. Instead, it exploits rich hand-motion supervision from large-scale egocentric data to learn a stable hand–camera geometric relationship. To improve generalization across tasks, actions, and scenes, we select clips with high scene and action diversity from Ego4D [^8], EgoDex [^10], and EPIC-KITCHENS [^4], and initialize the geometric encoder with the pretrained GCT from LingBot-Map [^3].

We jointly optimize four objectives: the camera motion loss $\mathcal{L}_{\mathrm{cam}}$, bimanual MANO loss $\mathcal{L}_{\mathrm{mano}}$, hand observability loss $\mathcal{L}_{\mathrm{obs}}$, and world-frame consistency loss $\mathcal{L}_{\mathrm{world}}$:

 $$ \mathcal{L}_{\mathrm{s t a g e1}}=\lambda_{\mathrm{c a m}}\mathcal{L}_{\mathrm{c a m}}+\lambda_{\mathrm{m a n o}}\mathcal{L}_{\mathrm{m a n o}}+\lambda_{\mathrm{o b s}}\mathcal{L}_{\mathrm{o b s}}+\lambda_{\mathrm{w o r l d}}\mathcal{L}_{\mathrm{w o r l d}}. $$

Through the differentiable transformation in Eq. (7), the world-frame loss jointly constrains the camera and hand predictions, allowing reliable hand-motion supervision to inform their geometric relationship.

The field of view and camera trajectory are predicted by separate heads, preserving the pretrained field-of-view capability while preventing camera-trajectory adaptation from interfering with it.

b) Stage 2: Camera Trajectory Correction: The first stage establishes stable hand–camera geometry, but monocular SLAM can still leave errors in absolute scale and pose. We therefore perform a second stage using camera trajectories captured with high-precision devices.

We freeze the learned geometric encoder, hand, observability, and field-of-view modules, and optimize only the camera trajectory prediction. The objective consists of a translation term $\mathcal{L}_{t}$, a rotation term $\mathcal{L}_{R}$, and a temporal consistency term $\mathcal{L}_{temp}$:

 $$ \mathcal{L}_{\mathrm{s t a g e2}}=\lambda_{t}\mathcal{L}_{t}+\lambda_{R}\mathcal{L}_{R}+\lambda_{\mathrm{t e m p}}\mathcal{L}_{\mathrm{t e m p}}. $$

Since hand reconstruction and hand–camera geometry are established in Stage 1, a small amount of high-precision camera trajectory data is sufficient to correct the absolute scale and pose while preserving the learned hand motion, hand–camera geometry, and field-of-view estimation.

---

### V. EXPERIMENTS

### A. Setup

Datasets. Stage 1 uses the 1,021 hours of EgoPIPELINE output from Sec. III, which contains no manually annotated hands and no ground-truth camera poses. Stage 2 trains the camera head alone on a small in-house corpus recorded with a single rig, using the camera trajectories that rig provides; no hand supervision is used in this stage. Neither stage sees HOT3D [^1] or ARCTIC [^7], on which all results below are therefore zero-shot.

Metrics. Standard hand pose metrics typically evaluate only successfully matched hands and therefore do not account for missed detections. We therefore use a coverage-aware evaluation in which missed hands are penalized rather than excluded from pose evaluation [^33], and we group the metrics by what they measure. Detection is measured by frame accuracy (FAcc), recall and F1. Camera-frame hand reconstruction is measured by MPJPE-p and PA-MPJPE-p for articulated pose in 3D, by EPE-p for the reprojected 2D keypoint error, by GO-p for wrist orientation and by CT-p for hand placement, all computed under the coverage-aware protocol. For camera trajectories we report RPE-T and RPE-R for relative motion accuracy, and the arc-length ratio for scale deviation. Trajectories are evaluated over the full sequence and aligned with $SE(3)$ without scale fitting, so the reported errors retain the metric scale the model recovers on its own. RPE is computed per sequence as an RMSE and summarized by a mean and a median. Temporal quality is assessed by RPE for the camera and by Jitter for the hands. RPE is particularly relevant to MINT because the model is trained with inter-frame motion increments, which makes it a direct measure of local motion consistency.

Implementation. MINT is trained on 32-frame clips at $378 \times 518$ resolution using AdamW with BF16 mixed precision. Stage 1 uses learning rates of $5 \times 10^{-5}$, $1 \times 10^{-4}$ and $5 \times 10^{-4}$ for the pretrained backbone, the camera and field-of-view heads and the hand and observability heads respectively, with an effective batch size of 1. In stage 2 the aggregator, the field-of-view head and the hand and observability heads are frozen, and only the camera head is optimized at a learning rate of $5 \times 10^{-5}$, with an effective batch size of 4. Both stages use a weight decay of 0.05 and a 2,000-step warmup.

### B. Camera-Frame Hands

Detection and pose. Predicting both hands on every frame with an explicit observability score, rather than delegating to an external detector, is what makes MINT reliable at finding hands under egocentric occlusion (Table 1). On HOT3D it lifts frame accuracy from 0.827 for the strongest baseline to 0.945 and recall from 0.904 to 0.983, so the share of frames carrying a missed hand falls from roughly one in six to one in twenty. On ARCTIC, recall and F1 are the highest of any method.

In pose accuracy MINT scores well on both zero-shot benchmarks. On ARCTIC it stays behind WiLoR [^19] on

<div style="text-align: center;">Table 2. World-frame camera trajectory results on the HOT3D [^1] and ARCTIC [^7] validation sets. All sequences are evaluated over their full trajectories using  $SE(3)$-only alignment without scale fitting, while the trajectory arc-length ratio measures scale consistency. MINT w/o stage 2 never sees metric ground truth.</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td rowspan="2">Method</td><td colspan="2">RPE-T $\downarrow$ (mm)</td><td colspan="2">RPE-R $\downarrow$ (deg)</td><td style='text-align: center; word-wrap: break-word;'>Arc len.</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>mean</td><td style='text-align: center; word-wrap: break-word;'>med.</td><td style='text-align: center; word-wrap: break-word;'>mean</td><td style='text-align: center; word-wrap: break-word;'>med.</td><td style='text-align: center; word-wrap: break-word;'>ratio $\rightarrow$ 1</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>DROID-SLAM [^24]</td><td style='text-align: center; word-wrap: break-word;'>5.362</td><td style='text-align: center; word-wrap: break-word;'>3.524</td><td style='text-align: center; word-wrap: break-word;'>0.227</td><td style='text-align: center; word-wrap: break-word;'>0.146</td><td style='text-align: center; word-wrap: break-word;'>0.778</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>InfiniteVGGT [^38]</td><td style='text-align: center; word-wrap: break-word;'>13.524</td><td style='text-align: center; word-wrap: break-word;'>9.673</td><td style='text-align: center; word-wrap: break-word;'>1.492</td><td style='text-align: center; word-wrap: break-word;'>0.392</td><td style='text-align: center; word-wrap: break-word;'>0.556</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>LingBot-Map [^3]</td><td style='text-align: center; word-wrap: break-word;'>7.566</td><td style='text-align: center; word-wrap: break-word;'>6.185</td><td style='text-align: center; word-wrap: break-word;'>0.684</td><td style='text-align: center; word-wrap: break-word;'>0.253</td><td style='text-align: center; word-wrap: break-word;'>0.712</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>MegaSaM [^13]</td><td style='text-align: center; word-wrap: break-word;'>3.187</td><td style='text-align: center; word-wrap: break-word;'>2.134</td><td style='text-align: center; word-wrap: break-word;'>0.082</td><td style='text-align: center; word-wrap: break-word;'>0.063</td><td style='text-align: center; word-wrap: break-word;'>0.716</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>MINT w/o stage 2</td><td style='text-align: center; word-wrap: break-word;'>8.752</td><td style='text-align: center; word-wrap: break-word;'>8.371</td><td style='text-align: center; word-wrap: break-word;'>0.234</td><td style='text-align: center; word-wrap: break-word;'>0.229</td><td style='text-align: center; word-wrap: break-word;'>0.466</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>MINT</td><td style='text-align: center; word-wrap: break-word;'>4.694</td><td style='text-align: center; word-wrap: break-word;'>4.783</td><td style='text-align: center; word-wrap: break-word;'>0.284</td><td style='text-align: center; word-wrap: break-word;'>0.259</td><td style='text-align: center; word-wrap: break-word;'>1.094</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>DROID-SLAM [^24]</td><td style='text-align: center; word-wrap: break-word;'>33.845</td><td style='text-align: center; word-wrap: break-word;'>14.256</td><td style='text-align: center; word-wrap: break-word;'>1.006</td><td style='text-align: center; word-wrap: break-word;'>0.423</td><td style='text-align: center; word-wrap: break-word;'>0.964</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>InfiniteVGGT [^38]</td><td style='text-align: center; word-wrap: break-word;'>16.215</td><td style='text-align: center; word-wrap: break-word;'>12.637</td><td style='text-align: center; word-wrap: break-word;'>1.265</td><td style='text-align: center; word-wrap: break-word;'>0.655</td><td style='text-align: center; word-wrap: break-word;'>0.284</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>LingBot-Map [^3]</td><td style='text-align: center; word-wrap: break-word;'>9.171</td><td style='text-align: center; word-wrap: break-word;'>8.472</td><td style='text-align: center; word-wrap: break-word;'>0.980</td><td style='text-align: center; word-wrap: break-word;'>0.717</td><td style='text-align: center; word-wrap: break-word;'>0.591</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>MegaSaM [^13]</td><td style='text-align: center; word-wrap: break-word;'>8.738</td><td style='text-align: center; word-wrap: break-word;'>5.581</td><td style='text-align: center; word-wrap: break-word;'>0.779</td><td style='text-align: center; word-wrap: break-word;'>0.725</td><td style='text-align: center; word-wrap: break-word;'>1.956</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>MINT w/o stage 2</td><td style='text-align: center; word-wrap: break-word;'>3.536</td><td style='text-align: center; word-wrap: break-word;'>3.624</td><td style='text-align: center; word-wrap: break-word;'>0.251</td><td style='text-align: center; word-wrap: break-word;'>0.243</td><td style='text-align: center; word-wrap: break-word;'>0.755</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>MINT</td><td style='text-align: center; word-wrap: break-word;'>3.392</td><td style='text-align: center; word-wrap: break-word;'>3.372</td><td style='text-align: center; word-wrap: break-word;'>0.256</td><td style='text-align: center; word-wrap: break-word;'>0.251</td><td style='text-align: center; word-wrap: break-word;'>1.412</td></tr></table>

the millimetre metrics while EPE-p comes within a few pixels of the best result, and on HOT3D it is the best method on every pose measure, reaching 29.918 mm MPJPE-p, 13.646 mm PA-MPJPE-p, 55.058 px EPE-p and 21.091° GO-p. MINT therefore maintains consistent performance across the two benchmarks, demonstrating strong cross-benchmark generalization.

The low EPE-p on HOT3D indicates accurate image-space localization, while CT-p reaches 0.069 m on ARCTIC and 0.196 m on HOT3D. These results indicate that the residual translation error primarily arises from metric depth estimation along the viewing ray, rather than from image-space localization.

Temporal smoothness. Egocentric hand trajectories are normally stabilized by a learned motion prior [^6], [^34] or by test-time optimization, both of which add inference cost, and MINT needs neither to reach 12.959 Jitter on ARCTIC, on par with the smoothest baseline, and 12.057 on HOT3D, the best of any method. The bidirectional spatiotemporal encoder resolves each frame against its neighbors instead of smoothing an independently estimated sequence after the fact. The UKF of Sec. III, applied at inference, drives Jitter to 2.621 and 2.373, close to a fivefold reduction on both benchmarks. Smoothness is therefore already a property of the learned representation, and the filter sharpens it without disturbing the recovered geometry.

### C. Camera Trajectory

On camera trajectory (Table 2) MINT more than halves the relative translation error on ARCTIC, from 8.730 mm for the strongest baseline to 3.390 mm, and reaches 4.690 mm zero-shot on HOT3D, second only to MegaSaM. Absolute error is where the dedicated systems stay ahead, since a feed-forward pass over 32-frame windows has neither loop closure nor global optimization and therefore accumulates drift over a full sequence.

The second stage acts on scale rather than on shape. Without it the model reaches an arc-length ratio of 0.466 on

---

<div style="text-align: center;">Table 3. Backbone initialization on ARCTIC [^7], for camera-frame hands and for the camera trajectory. Units follow Table 1; RPE-T is in mm, RPE-R in degrees and the arc-length ratio is best at 1. Best per column in bold.</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'></td><td colspan="6">Camera-frame hands</td><td colspan="6">Camera trajectory</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Init.</td><td style='text-align: center; word-wrap: break-word;'>MPJPE-p $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>PA-MPJPE-p $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>EPE-p $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>GO-p $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>CT-p $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>Jitter $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>RPE-T mean $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>RPE-T med. $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>RPE-R mean $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>RPE-R med. $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>Arc len.</td><td style='text-align: center; word-wrap: break-word;'></td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Random</td><td style='text-align: center; word-wrap: break-word;'>48.045</td><td style='text-align: center; word-wrap: break-word;'>26.869</td><td style='text-align: center; word-wrap: break-word;'>82.356</td><td style='text-align: center; word-wrap: break-word;'>24.562</td><td style='text-align: center; word-wrap: break-word;'>0.198</td><td style='text-align: center; word-wrap: break-word;'>18.758</td><td style='text-align: center; word-wrap: break-word;'>6.130</td><td style='text-align: center; word-wrap: break-word;'>6.280</td><td style='text-align: center; word-wrap: break-word;'>0.846</td><td style='text-align: center; word-wrap: break-word;'>0.850</td><td style='text-align: center; word-wrap: break-word;'>1.201</td><td style='text-align: center; word-wrap: break-word;'></td></tr><tr><td style='text-align: center; word-wrap: break-word;'>LingBot-Map [^3]</td><td style='text-align: center; word-wrap: break-word;'>26.527</td><td style='text-align: center; word-wrap: break-word;'>15.961</td><td style='text-align: center; word-wrap: break-word;'>55.038</td><td style='text-align: center; word-wrap: break-word;'>19.332</td><td style='text-align: center; word-wrap: break-word;'>0.069</td><td style='text-align: center; word-wrap: break-word;'>12.959</td><td style='text-align: center; word-wrap: break-word;'>3.390</td><td style='text-align: center; word-wrap: break-word;'>3.370</td><td style='text-align: center; word-wrap: break-word;'>0.256</td><td style='text-align: center; word-wrap: break-word;'>0.251</td><td style='text-align: center; word-wrap: break-word;'>1.412</td><td style='text-align: center; word-wrap: break-word;'></td></tr></table>

<div style="text-align: center;">Table 4. Data mixture for camera-frame hand reconstruction on ARCTIC [^7]; only the source composition changes. EPIC-KITCHENS [^4] and the single-rig in-house corpus appear only within the mixture.</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'>Sources</td><td style='text-align: center; word-wrap: break-word;'>MPJPE-p $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>PA-MPJPE-p $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>EPE-p $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>GO-p $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>CT-p $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>Jitter $\downarrow$</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Ego4D only</td><td style='text-align: center; word-wrap: break-word;'>49.594</td><td style='text-align: center; word-wrap: break-word;'>26.230</td><td style='text-align: center; word-wrap: break-word;'>64.449</td><td style='text-align: center; word-wrap: break-word;'>23.134</td><td style='text-align: center; word-wrap: break-word;'>0.142</td><td style='text-align: center; word-wrap: break-word;'>9.642</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>EgoDex only</td><td style='text-align: center; word-wrap: break-word;'>48.222</td><td style='text-align: center; word-wrap: break-word;'>25.675</td><td style='text-align: center; word-wrap: break-word;'>56.772</td><td style='text-align: center; word-wrap: break-word;'>24.102</td><td style='text-align: center; word-wrap: break-word;'>0.097</td><td style='text-align: center; word-wrap: break-word;'>15.783</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Full mixture</td><td style='text-align: center; word-wrap: break-word;'>26.527</td><td style='text-align: center; word-wrap: break-word;'>15.961</td><td style='text-align: center; word-wrap: break-word;'>55.038</td><td style='text-align: center; word-wrap: break-word;'>19.332</td><td style='text-align: center; word-wrap: break-word;'>0.069</td><td style='text-align: center; word-wrap: break-word;'>12.959</td></tr></table>

HOT3D, so the recovered path is less than half the length of the true one, the signature of pseudo-labels whose scale comes from monocular depth. Stage 2 moves that ratio to 1.094 and improves RPE-T from 8.750 to 4.690 mm. On ARCTIC the same correction overshoots, taking the ratio from 0.755 to 1.412, so the recovered path now exceeds the true length even though RPE-T still improves from 3.530 to 3.390 mm. Stage 2 therefore removes the systematic underscaling that dominates on HOT3D, at the price of some over-scaling on a benchmark whose motion is already closer to metric.

### D. Ablations

Data mixture. Table 4 varies only the source composition and scores camera-frame hand reconstruction on ARCTIC. Between the two single sources, EgoDex [^10] recovers the hand more accurately, reaching 48.222 mm MPJPE-p, 56.772 px EPE-p and 0.097 m CT-p, and Ego4D [^8] leads only on wrist orientation and smoothness. The mixture is best in five of the six columns and brings MPJPE-p to 26.527 mm and CT-p to 0.069 m. Since zero-shot transfer is the target, we train on the mixture, whose viewpoints, motions and intrinsic span what the benchmarks present rather than the capture conditions of any one source.

Initialization. Table 3 replaces the LingBot-Map [^3] backbone with random initialization and leaves everything else fixed. On camera-frame hands the pretrained backbone is ahead in every column, most visibly on EPE-p, which falls from 82.356 to 55.038 px, and on CT-p, which more than halves. The camera trajectory moves the same way, with relative translation error dropping from 6.130 to 3.390 mm and relative rotation error from $0.846^\circ$ to $0.256^\circ$. Geometric pretraining therefore supplies the correspondence and scene structure that the hand and camera heads both build on.

We evaluate inference efficiency at $512 \times 384$ and 30 fps under identical conditions. MINT encodes each video window in a single bidirectional spatiotemporal pass. On a single GPU, its per-frame latency is 72.4 ms, compared with 105.1 ms for HaWoR [^39] and 1260.0 ms for VITRA [^14].

### E. Inference Throughput

<div style="text-align: center;">Table 5. Inference efficiency comparison under identical conditions. VITRA [^14] is reproduced for reference.</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'>Method</td><td style='text-align: center; word-wrap: break-word;'>Time (ms/fr.) $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>fps $\uparrow$</td><td style='text-align: center; word-wrap: break-word;'>Speedup $\uparrow$</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>One GPU</td><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'></td></tr><tr><td style='text-align: center; word-wrap: break-word;'>VITRA [^14]</td><td style='text-align: center; word-wrap: break-word;'>1260.0</td><td style='text-align: center; word-wrap: break-word;'>0.8</td><td style='text-align: center; word-wrap: break-word;'>-</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>HaWoR [^39]</td><td style='text-align: center; word-wrap: break-word;'>105.1</td><td style='text-align: center; word-wrap: break-word;'>9.5</td><td style='text-align: center; word-wrap: break-word;'>12.0 $\times$</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>MINT</td><td style='text-align: center; word-wrap: break-word;'>72.4</td><td style='text-align: center; word-wrap: break-word;'>13.8</td><td style='text-align: center; word-wrap: break-word;'>17.4 $\times$</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Four GPUs</td><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'></td></tr><tr><td style='text-align: center; word-wrap: break-word;'>VITRA [^14]</td><td style='text-align: center; word-wrap: break-word;'>283.3</td><td style='text-align: center; word-wrap: break-word;'>3.5</td><td style='text-align: center; word-wrap: break-word;'>-</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>EGOPIPELINE</td><td style='text-align: center; word-wrap: break-word;'>83.4</td><td style='text-align: center; word-wrap: break-word;'>12.0</td><td style='text-align: center; word-wrap: break-word;'>3.4 $\times$</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>MINT</td><td style='text-align: center; word-wrap: break-word;'>22.7</td><td style='text-align: center; word-wrap: break-word;'>44.1</td><td style='text-align: center; word-wrap: break-word;'>12.5 $\times$</td></tr></table>

Since the windows are mutually independent, extended egocentric recordings admit parallel processing across GPUs without inter-window coordination, which reduces the mean per-frame latency to 22.7 ms. EgoPIPELINE, although it parallelizes an originally serial cascade, retains a fixed interstage dependency whose slowest stage bounds its attainable throughput, and requires 83.4 ms per frame. MINT therefore achieves a $3.67\times$ end-to-end speedup over EgoPIPELINE, and $12.5\times$ over our reproduction of VITRA, which requires 283.3 ms.

### VI. CONCLUSION

We presented MINT, a unified feed-forward model that recovers world-space camera and bimanual hand motion from egocentric video. No hand label in its training data is human-annotated. MINT is trained on 1,021 hours of motion supervision that EgoPIPELINE extracts automatically from publicly available egocentric video, and a short second stage uses device-captured camera trajectories from a small in-house corpus to correct absolute scale. The resulting model transfers to unseen domains and is competitive zero-shot on both camera-frame hand reconstruction and camera trajectory estimation. These results indicate that automatically reconstructed motion supervision from public video is sufficient to learn generalizable egocentric motion reconstruction.

a) Limitations and future work: MINT remains affected by the scale drift inherited from monocular reconstruction in the supervision pipeline, and the 32-frame training window leaves long-horizon drift open. Future work will explore more accurate and diverse metric supervision to reduce scale errors and improve long-range trajectory reconstruction.

## REFERENCES

[^1]: Prithviraj Banerjee, Sindi Shkodrani, Pierre Moulon, et al. Hot3d: Hand and object tracking in 3d from egocentric multi-view videos. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7061–7071, 2025.

[^2]: Yu-Wei Chao, Wei Yang, Yu Xiang, et al. Dexycb: A benchmark for capturing hand grasping of objects. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9040–9049, 2021.

---

[^3]: Lin-Zhuo Chen, Jian Gao, Yihang Chen, et al. Geometric context transformer for streaming 3d reconstruction. arXiv preprint arXiv:2604.14141, 2026.

[^4]: Dima Damen, Hazel Doughty, Giovanni Maria Farinella, et al. Scaling egocentric vision: the dataset. In European conference on computer vision, pages 753–771, 2018.

[^5]: Haoye Dong, Aviral Chharia, Wenbo Gou, Francisco Vicente Carrasco, and Fernando De la Torre. Hamba: Single-view 3d hand reconstruction with graph-guided bi-scanning mamba. arXiv preprint arXiv:2407.09646, 2024.

[^6]: Enes Duran, Muhammed Kocabas, Vasileios Choutas, Zicong Fan, and Michael J Black. Hmp: Hand motion priors for pose and shape estimation from video. In 2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 6341–6351, 2024.

[^7]: Zicong Fan, Omid Taheri, Dimitrios Tzionas, Muhammed Kocabas, Manuel Kaufmann, Michael J Black, and Otmar Hilliges. ARCTIC: A dataset for dexterous bimanual hand-object manipulation. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12943–12954, 2023.

[^8]: Kristen Grauman, Andrew Westbury, Eugene Byrne, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 18995–19012, 2022.

[^9]: Kristen Grauman, Andrew Westbury, Lorenzo Torresani, et al. Ego-exo4d: Understanding skilled human activity from first-and third-person perspectives. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 19383–19400, 2024.

[^10]: Ryan Hoque, Peide Huang, David Yoon, et al. Egodex: Learning dexterous manipulation from large-scale egocentric video. In International Conference on Learning Representations, pages 4218–4237, 2026.

[^11]: Simar Kareer, Dhruv Patel, Ryan Punamiya, Pranay Mathur, Shuo Cheng, Chen Wang, Judy Hoffman, and Danfei Xu. Egomimic: Scaling imitation learning via egocentric video. In 2025 IEEE International Conference on Robotics and Automation (ICRA), pages 13226–13233, 2025.

[^12]: Vincent Leroy, Yohann Cabon, and Jérôme Revaud. Grounding image matching in 3d with mast3r. In European conference on computer vision, pages 71–91, 2024.

[^13]: Zhengqi Li, Richard Tucker, Forrester Cole, Qianqian Wang, Linyi Jin, Vickie Ye, Angjoo Kanazawa, Aleksander Holynski, and Noah Snavely. Megasam: Accurate, fast, and robust structure and motion from casual dynamic videos. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10486–10496, 2025.

[^14]: Qixiu Li, Yu Deng, Yaobo Liang, et al. Scalable vision-language-action model pretraining for robotic manipulation with real-life human activity videos. arXiv preprint arXiv:2510.21571, 2025.

[^15]: Dixuan Lin, Yuxiang Zhang, Mengcheng Li, Wei Jing, Qi Yan, Qianying Wang, Yebin Liu, and Hongwen Zhang. Omnihands: Towards robust 4d hand mesh recovery via a versatile transformer. arXiv preprint arXiv:2405.20330, 2024.

[^16]: Gyeongsik Moon. Bringing inputs to shared domains for 3D interacting hands recovery in the wild. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 17028–17037, 2023.

[^17]: Maxime Oquab, Timothée Darcet, Théo Moutakanni, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023.

[^18]: Georgios Pavlakos, Dandan Shan, Ilija Radosavovic, Angjoo Kanazawa, David Fouhey, and Jitendra Malik. Reconstructing hands in 3d with transformers. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9826–9836, 2024.

[^19]: Rolandos Alexandros Potamias, Jinglei Zhang, Jiankang Deng, and Stefanos Zafeiriou. Wilor: End-to-end 3d hand localization and reconstruction in-the-wild. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12242–12254, 2025.

[^20]: Aditya Prakash, Ruisen Tu, Matthew Chang, and Saurabh Gupta. 3d hand pose estimation in everyday egocentric images. In European Conference on Computer Vision, pages 183–202, 2024.

[^21]: Javier Romero, Dimitrios Tzionas, and Michael J Black. Embodied hands: Modeling and capturing hands and bodies together. arXiv preprint arXiv:2201.02610, 2022.

[^22]: Zehong Shen, Huaijin Pi, Yan Xia, Zhi Cen, Sida Peng, Zechen Hu, Hujun Bao, Ruizhen Hu, and Xiaowei Zhou. World-grounded human

motion recovery via gravity-view coordinates. In SIGGRAPH Asia 2024 Conference Papers, pages 1–11, 2024.

[^23]: Soyong Shin, Juyong Kim, Eni Halilaj, and Michael J Black. Wham: Reconstructing world-grounded humans with accurate 3d motion. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2070–2080, 2024.

[^24]: Zachary Teed and Jia Deng. Droid-slam: Deep visual slam for monocular, stereo, and rgb-d cameras. Advances in neural information processing systems, 34:16558–16569, 2021.

[^25]: Eugene Valassakis and Guillermo Garcia-Hernando. Handdgp: Camera-space hand mesh prediction with differentiable global positioning. In European Conference on Computer Vision, pages 479–496, 2024.

[^26]: Alexander Veicht, Paul-Edouard Sarlin, Philipp Lindenberger, and Marc Pollefeys. Geocalib: Learning single-image calibration with geometric optimization. In European Conference on Computer Vision, pages 1–20, 2024.

[^27]: Shuzhe Wang, Vincent Leroy, Yohann Cabon, Boris Chidlovskii, and Jerome Revaud. Dust3r: Geometric 3d vision made easy. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 20697–20709, 2024.

[^28]: Yufu Wang, Ziyun Wang, Lingjie Liu, and Kostas Daniilidis. Tram: Global trajectory and motion of 3d humans from in-the-wild videos. In European Conference on Computer Vision, pages 467–487, 2024.

[^29]: Qianqian Wang, Yifei Zhang, Aleksander Holynski, Alexei A Efros, and Angjoo Kanazawa. Continuous 3d perception model with persistent state. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10510–10522, 2025.

[^30]: Ruicheng Wang, Sicheng Xu, Yue Dong, Yu Deng, Jianfeng Xiang, Zelong Lv, Guangzhong Sun, Xin Tong, and Jiaolong Yang. Moge2: Accurate monocular geometry with metric scale and sharp details. Advances in Neural Information Processing Systems, 38:35928–35959, 2026.

[^31]: Yifan Wang, Jianjun Zhou, Haoyi Zhu, Wenzheng Chang, Yang Zhou, Zizun Li, Junyi Chen, Jiangmiao Pang, Chunhua Shen, and Tong He. $\pi^{3}$: Permutation-Equivariant Visual Geometry Learning. In International Conference on Learning Representations, pages 10481–10497, 2026.

[^32]: Jianyuan Wang, Minghao Chen, Nikita Karaev, Andrea Vedaldi, Christian Rupprecht, and David Novotny. Vggt: Visual geometry grounded transformer. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5294–5306, 2025.

[^33]: Yuxi Wang, Chengkai Jin, Yufei Liu, Wenqi Ouyang, Tianyi Wei, Zhiwei Zeng, Siyuan Huang, Zhiqi Shen, and Xingang Pan. The Surprising Effectiveness of Video Diffusion Models for Hand Motion Reconstruction. arXiv preprint arXiv:2606.30308, 2026.

[^34]: Vickie Ye, Georgios Pavlakos, Jitendra Malik, and Angjoo Kanazawa. Decoupling human and camera motion from videos in the wild. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 21222–21232, 2023.

[^35]: Brent Yi, Vickie Ye, Maya Zheng, Yunqi Li, Lea Müller, Georgios Pavlakos, Yi Ma, Jitendra Malik, and Angjoo Kanazawa. Estimating body and hand motion in an ego-sensed world. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7072–7084, 2025.

[^36]: Wei Yin, Chi Zhang, Hao Chen, Zhipeng Cai, Gang Yu, Kaixuan Wang, Xiaozhi Chen, and Chunhua Shen. Metric3d: Towards zero-shot metric 3d prediction from a single image. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 9009–9019, 2023.

[^37]: Zhengdi Yu, Stefanos Zafeiriou, and Tolga Birdal. Dyn-hamr: Recovering 4d interacting hand motion from a dynamic camera. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 27716–27726, 2025.

[^38]: Shuai Yuan, Yantai Yang, Xiaotian Yang, Xupeng Zhang, Zhonghao Zhao, Lingming Zhang, and Zhipeng Zhang. Infinitevggt: Visual geometry grounded transformer for endless streams. arXiv preprint arXiv:2601.02281, 2026.

[^39]: Jinglei Zhang, Jiankang Deng, Chao Ma, and Rolandos Alexandros Potamias. Hawor: World-space hand motion reconstruction from egocentric videos. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) pages 1805–1815 2025

[^40]: Junyi Zhang, Charles Herrmann, Junhwa Hur, et al. Monst3r: A simple approach for estimating geometry in the presence of motion. In

---

International Conference on Learning Representations, pages 82863–82886, 2025.