# EGOWILD2DEX: Learning Dexterous Robotic Manipulation from In-the-Wild Human Experience

Kunyang Lin $^{\dagger}$, Xutao Wen, Jingxi Lin, Lanyong Lin, Jiaming Liu, Tianshuo Yang, Xianchi Chen, Yue Han, Yiduo Li, Zhanpeng Zhang, Ping Luo $^{*}$

The University of Hong Kong Kinetix AI

Project lead *Corresponding author

<https://mmlab.hk/egowild2dex>

![Image](images/6c67fc35e740f7c1dc46035101ff88e16ab6121b0a778ad74d5837e440e5ff99.jpg)

![Image](images/e58e42a02752b0893086739be11198485432710ab4e291ccf3a128f72f3172b0.jpg)

![Image](images/519835d8b181991b70ad54602a7343dd71b6a021272d72d67b353124298df288.jpg)

Cut box open, take the seen/unseen objects out.

![Image](images/b5ec9f85750013b82003c093ee5b034af1afe722875d73e6e0061dd703fbacde.jpg)

<div style="text-align: center;">(a) Human Experience</div>

![Image](images/2ea5bddce05249820ea1838b0384bbb758d35c46483e43944a0c32a8288977ce.jpg)

(b) EgoWild2Dex

![Image](images/c21e2fb9554c3545c888c9324f9a63f75b6b8aa7d13d4a775de3460c3667cd50.jpg)

Grip glue gun, glue shelf center, place figure on spot.

![Image](images/a3f1c950c2e549074ebffb3bf098f7a71c9f85f2e4d64a73127372e256541c94.jpg)

<div style="text-align: center;">(c) Dexterous Manipulation</div>

<div style="text-align: center;">Fig. 1. We introduce EGOWILD2DEX, a framework for learning dexterous robot manipulation from in-the-wild human experience. (a) EgoWild is an in-the-wild egocentric human manipulation dataset captured with head-mounted cameras during everyday activities. On average, each frame of EgoWild contains 34.78 segmented foreground instances (7.71× EgoDex [^1] and 1.53× EgoVerse [^2]), and cumulative head rotation is 15.93°/s (3.80× and 3.45×, respectively). (b) EGOWILD2DEX is a training pipeline that transfers in-the-wild human experience to dexterous robots with limited robot data in a shared robot-native action space. At its core is a differentiable geometric transformer that aligns task-specific ego observations with robot views for co-training. (c) We demonstrate our method on three long-horizon bimanual dexterous tasks. Each task presents challenges involving tool use, arm-hand coordination, and contact-rich interaction, and consists of multiple subtasks in sequence.</div>

Abstract—Egocentric human data provide a principled source of supervision for learning dexterous robot manipulation. Unlike prior approaches that often collect such data in constrained or specially constructed environments, we collect in-the-wild egocentric demonstrations in real-world settings, including homes, factories, and pharmacies, etc., where people perform their ordinary tasks while wearing head-mounted cameras. This collection protocol captures diverse workflows and hand-object interactions across long-tailed object and skill distributions, but also yields visually challenging observations due to scene clutter and head-motion-induced viewpoint changes (a mean cumulative rotation of $15.93^{\circ}/s$). To address these issues, we introduce EGOWILD2DEX, which transfers in-the-wild ego-human experience to dual-arm robots with dexterous hands by jointly aligning unstable egocentric views and human motions with robot observations and actions, respectively. This work offers three benefits. First, we introduce GEOFORMER, a differentiable geometric transformer that warps noisy human observations toward robot observations. Second, we design a human-robot training scheme to bridge the embodiment gap, enabling high task success with limited robot supervision. Third, we release EgoWild, a 538.9-hour in-the-wild egocentric human dataset comprising 179,049 episodes, 125,961 unique task descriptions, and 1,282 object categories. On real robots, EGOWILD2DEX achieves an average success rate of 96.7% across three long-horizon bimanual dexterous manipulation tasks and an average object-level zero-shot success rate of 33.3%. The data, models, and code will be released.

### I. INTRODUCTION

Large-scale ego-human demonstrations offer a scalable alternative to costly robot teleoperation trajectories. Beyond scale, they capture environmental dynamics, contact-induced object affordances, and dexterous arm–hand kinematics [^3] [^4] [^5]. Recent works have demonstrated that ego-human data can provide beneficial priors for robot manipulation [^6] [^7] [^8] [^9] [^10]. However, existing approaches often rely on massive amounts of non-public data, costly paired human–robot data, or specially constructed scenes [^1], [^8], [^9], [^11] [^12] [^13] [^14] [^15]. Moreover, most target gripper-based manipulation [^6], [^10], [^13] [^14] [^15] [^16] [^17] and do not consider multi-finger coordination for long-horizon dexterous tasks.

In this work, we seek to learn from naturally occurring human manipulation in everyday life, where actions are shaped by the available objects, surrounding clutter, and task demands. Such in-the-wild experience exposes the model to diverse objects and hand–object interactions in cluttered scenes with dynamic viewpoints, reducing its reliance on shortcuts specific to staged setups and encouraging it to learn manipulation patterns that transfer across environments.

However, transferring this experience to robots presents two challenges. First, egocentric observations change with

---

both manipulation and head motion. Consequently, the same manipulation can appear different under moving human cameras and fixed robot cameras. Second, broad human experience provides diverse manipulation behaviors but does not directly teach a robot how to execute a specific task under its own embodiment and physical constraints. The challenge is to turn this experience into reliable robot behavior.

To tackle these challenges, we introduce EGOWILD2DEX, a framework for learning dexterous robot manipulation from in-the-wild human experience. We propose Geometric Transformer (GEOFORMER) for lightweight projective alignment of task-specific ego observations with robot views, eliminating the need for precise camera calibration and the high computational and memory costs of existing 3D projection-and-inpainting pipelines [^18], [^19]. It learns a robot-to-ego homography and applies its inverse to bring ego observations closer to the robot view, providing visually aligned human demonstrations for co-training. To turn broad human experience into reliable robot execution, we design a progressive human–robot training scheme. Human-to-Robot learning first learns a broad prior from ego-human demonstrations. Human–Robot co-training then grounds this prior using task-specific human demonstrations and limited robot data. Finally, robot-domain refinement incorporates execution and recovery to reduce the remaining contact and dynamics gap. All stages share the policy architecture, robot-native action space, and flow-matching objective, while using different data and schedulers.

To support this framework, we construct EgoWild, a large-scale dataset of in-the-wild human manipulation, captured with head-mounted cameras during everyday tasks. As shown in Fig. 1, EgoWild contains more foreground instances per frame (34.78) and greater cumulative head rotation per second (15.93°) than EgoDex [^1] and EgoVerse [^2], featuring denser clutter and more dynamic viewpoints.

We evaluate our full system on three long-horizon tasks, each comprising multiple subtasks that involve tool use, bimanual coordination, and object manipulation. Empirical results show that, using less than one hour of robot demonstrations per task, EGOWILD2DEX achieves an average success rate of 96.7% across three long-horizon bimanual dexterous tasks. EGOWILD2DEX also achieves an average object-level zero-shot success rate of 33.3% on unseen objects. These results suggest that combining EgoWild with GEOFORMER and progressive human-robot training can effectively transfer in-the-wild human experience to robot manipulation.

Collectively, our contributions are threefold: (1) We introduce GEoFORMER, a differentiable geometric transformer for lightweight ego-to-robot view alignment without explicit 3D reconstruction or inpainting, delivering a speedup of up to $21.9\times$. (2) We develop a progressive human–robot training scheme that transfers broad human manipulation priors to dexterous robots, achieving a 96.7% average success rate across three long-horizon dexterous tasks with less than one hour of robot data per task. (3) We release EgoWild, a 538.9-hour in-the-wild egocentric dataset of unscripted bimanual activities across real-world environments, objects, and workflows. We demonstrate the potential of EgoWild in dexterous manipulation and believe it will benefit research in other domains.

### II. RELATED WORK

### A. Vision-Language-Action Models for Dexterous Hands

Vision–language–action (VLA) models predict robot actions from visual observations and language instructions. Existing VLAs acquire broad manipulation capabilities from Internet-scale knowledge and heterogeneous robot data [^20] [^21] [^22] [^23] [^24] [^25], but are predominantly developed around arm–gripper action spaces that do not specifically capture fine-grained multi-finger coordination. Unlike grippers, multi-DoF hands require coordinated control of multiple motors, and the resulting commands can exhibit jitter and abrupt changes under limited supervision. To study this, hierarchical dexterous systems connect semantic planners to grasping or reusable low-level skills [^26] [^27] [^28]; although these systems are modular, this separation can weaken closed-loop adaptation when contact outcomes deviate from the planned skill. End-to-end dexterous VLAs directly model high-DoF arm–hand actions [^8], [^29] [^30] [^31], but demand substantial robot-domain supervision and remain vulnerable to compounding errors over long horizons, motivating additional on-policy or reinforcement-learning post-training [^32], [^33]. In contrast, EGOWILD2DEX transfers in-the-wild human experience to dexterous VLAs through geometric view alignment and progressive human–robot training, reducing reliance on robot demonstrations.

### B. Ego-Human Data for Dexterous Robot Learning

Early work primarily uses human data either to pretrain visual representations or to support high-level planning, leaving downstream action learning to robot supervision [^15], [^34] [^35] [^36] [^37] [^38] [^39]. More recent methods exploit egocentric sensing and hand tracking as dense action supervision through explicit human–robot alignment and co-training [^6], [^40], [^41], pretraining followed by inverse kinematics and retargeting [^7], or large-scale cross-embodiment pretraining [^8], [^9], [^42]. In parallel, new datasets expand task, object, and environment coverage [^1], [^2], [^43], [^44], while some methods seek to synthesize robot-domain observations [^18], [^45], [^46]. Rather than scaling data indiscriminately, EGOWILD2DEX pairs a natural, manipulation-rich ego dataset with an explicit action-and view-alignment recipe. This data-efficient design achieves strong long-horizon bimanual dexterity with under one hour of robot data per task.

### III. METHODOLOGY

To transfer in-the-wild human experience to dexterous robots, we propose EGOWILD2DEX, a framework combining geometric view alignment with progressive human-robot training (Fig. 2). Specifically, GEOFORMER aligns task-specific ego observations with the robot view, providing visually aligned human demonstrations for co-training. Using a shared action space throughout, the training scheme first learns broad manipulation priors, then adapts them through task-specific human-robot co-training, and finally refines robot execution

---

![Image](images/b4e6618a13d9132a3ecafb31173bef58f970ce9e61a1041d3ba65da7d4b90a10.jpg)

<div style="text-align: center;">Fig. 2. Overview of EGOWILD2DEX. (a-1) Human-to-Robot learning builds broad manipulation priors by updating the vision–language model (VLM) on 538.9 hours of EgoWild data. (a-2) Human–Robot co-training first trains GEoFORMER on unpaired task-specific ego and robot images, then freezes it to align ego observations while updating the VLM and action expert. The task-specific ego, glove, and robot data used in this stage have a combined duration equivalent to just 1.6% of the data used in Stage 1. (a-3) Robot-Domain refinement improves execution and recovery by updating the VLM and action expert on robot data totaling just 0.1% of Stage 1's data duration. (b-1) Inverse kinematics (IK) and hand retargeting map all demonstrations into one robot-native action space. (b-2) GEoFORMER aligns ego observations to the robot view, while glove and robot data share the same camera configuration.</div>

with expert demonstrations and recovery trajectories. Algorithm 1 summarizes the training and deployment procedure.

### A. Geometric Transformer

A key challenge of using egocentric human data is that the same action appears different under head-mounted human cameras and stationary robot cameras. Traditional methods use 3D projection followed by inpainting to align the viewpoints [^18], [^19]. These methods require precise camera calibration and are computationally expensive. To overcome this, as shown in Fig. 3, we propose GEOFORMER, aligning task-specific ego observations to the robot view using a lightweight projective warp. GEOFORMER learns from unpaired images $I_{\text{ego}}, I_{\text{rob}} \in \mathbb{R}^{3 \times H_{\text{img}} \times W_{\text{img}}}$, where $H_{\text{img}}$ and $W_{\text{img}}$ denote image height and width. It randomly pairs each ego image with a robot-view image sampled from the corresponding task's robot-image pool during training. The model represents their geometric difference using an eight-dimensional homography parameter vector $\mathbf{p} = [p_1, \ldots, p_8]^\top$, which is then mapped to a homography $\mathcal{T}$ through

 $$ \begin{aligned}&P_{\mathbf{p}}:=\begin{bmatrix}p_{3}&p_{2}&p_{1}\\p_{6}&-p_{3}-p_{7}&p_{5}\\p_{4}&p_{8}&p_{7}\end{bmatrix},\quad\\ &\mathcal{T}_{\mathbf{p}}:=\exp(P_{\mathbf{p}})=\lim_{K\to\infty}\sum_{k=0}^{K}\frac{P_{\mathbf{p}}^{k}}{k!},\\ \end{aligned} $$

where $P_{p}$ is a traceless matrix of eight projective parameters, and its exponential $T_{p}$ defines the homography.

Starting from a perturbed estimate $p_{0}$, GEOFORMER progressively refines the warp. At the n-th step, it warps the robot image using the current estimate,

 $$ \begin{array}{r}{I_{\mathrm{r o b},n-1}^{\mathrm{w a r p}}=\mathrm{W a r p}(I_{\mathrm{r o b}};\mathcal{T}_{{\bf p}_{n-1}}),}\end{array} $$

where $\text{Warp}(I;\mathcal{T})$ denotes warping image I by the homography $\mathcal{T}$, and $I_{\text{rob},n-1}^{\text{warp}}$ is the robot image warped toward the ego view using the parameter estimate $\mathbf{p}_{n-1}$ from the previous step. The next residual is predicted as

 $$ \begin{array}{r}{\Delta\mathbf{p}_{n}=\mathcal{G}_{n}(I_{\mathrm{e g o}},I_{\mathrm{r o b},n-1}^{\mathrm{w a r p}}),}\end{array} $$

![Image](images/a3092bccf16397780e1edc3551648289d67b6453b9f2162bc127a9845195e020.jpg)

<div style="text-align: center;">Fig. 3. Geometric Transformer (GEOFORMER). (a) Alignment path: GEOFORMER predicts a robot-to-ego warp  $\mathcal{T}$ and applies  $\mathcal{T}^{-1}$ to align ego observations with the robot view. (b) Warp module: predictors  $\{\mathcal{G}_n\}$ refine homography parameters to make warped robot images appear natural when composited onto ego images, under adversarial supervision from the critic  $\mathcal{C}$. (c) VLA training: the aligned ego images are used alongside robot observations to train the VLA policy through human–robot co-training.</div>

where $\mathcal{G}_n$ is the n-th predictor that takes the concatenation of the ego image and the currently warped robot image as input and outputs an eight-dimensional residual in homography parameter space. Each predictor uses a convolutional neural network (CNN) to extract visual features, followed by a multilayer perceptron (MLP) to regress the residual [^47], [^48]. We then add this residual to the previous estimate, yielding $\mathbf{p}_n = \mathbf{p}_{n-1} + \Delta \mathbf{p}_n$.

The final robot-to-ego warp is $\mathcal{T} = \exp(P_{p_N})$, where N is the total number of cascaded refinement steps. During training, as illustrated in Fig. 3, we apply $\mathcal{T}$ to both the robot image and an all-ones mask of the same dimensions to obtain

 $$ I_{\mathrm{r o b}}^{\mathrm{w a r p}}=\mathrm{W a r p}(I_{\mathrm{r o b}};\mathcal{T}),\quad M_{\mathcal{T}}=\mathrm{W a r p}(\mathbf{1};\mathcal{T}), $$

where 1 is an all-ones image with the same dimensions as $I_{rob}$, and out-of-bounds samples are zero-padded. The resulting mask selects valid warped content to form the composite

 $$ \begin{array}{r}{I_{\mathrm{r o b+e g o}}=M_{\mathcal{T}}\odot I_{\mathrm{r o b}}^{\mathrm{w a r p}}+\left(1-M_{\mathcal{T}}\right)\odot I_{\mathrm{e g o}},}\end{array} $$

where $\odot$ denotes element-wise multiplication and $M_{\mathcal{T}}$ selects valid warped robot content, retaining the ego background elsewhere. The geometric predictor is trained with adversarial loss to warp robot-view content and paste it into the ego image

---

as a natural-looking composite. It learns to fool a Wasserstein critic [^49] that is trained simultaneously to distinguish real ego images from predictor-generated composites. Geometric regularization further keeps the pasted region at a plausible scale and maintains sufficient overlap with the ego canvas, preventing tiny patches or excessive out-of-frame warps.

After training GEOFORMER, the critic is discarded and the predictor is frozen during subsequent human-robot policy co-training. The learned warp maps robot views to ego views; policy inputs therefore use its inverse:

 $$ I_{\mathrm{e g o}}^{\mathrm{a l i g n}}=\mathrm{W a r p}(M_{\mathcal{T}}\odot I_{\mathrm{e g o}};\mathcal{T}^{-1}), $$

where $I_{\text{ego}}^{\text{align}}$ is the ego observation aligned toward the robot view by the inverse homography $\mathcal{T}^{-1}$. Despite fixed-resolution inputs, GEOFORMER supports inverse warping at arbitrary resolutions with coordinate scaling. The aligned frame is then processed by the same VLA vision encoder as a robot image. Compared with the Project+Inpaint pipeline using MoGe-based 3D reprojection [^50] with LaMa in-painting [^51], GEOFORMER achieves a $21.9\times$ speedup and improves both robot-view image similarity and downstream robot task performance (Sec. IV-E).

### B. Progressive Human-Robot Training

As shown in Fig. 2, we use a VLA architecture comprising a vision-language model backbone that processes images and task instructions, and a flow-matching action expert that predicts robot-native action chunks. We leverage a consistent action space to learn across human and robot demonstrations.

a) Shared robot-native action space: We calibrate operators' arm poses to approximately match the robot's reference configuration, establishing a common spatial reference for motion retargeting. Subsequent wrist and finger movements are converted into robot-native commands relative to this reference, reducing mismatches caused by arbitrary initial poses and providing consistent action supervision across training stages. The shared action vector contains the arm joint targets and hand motor commands for both sides,

 $$ A_{t}:=\left[\begin{matrix}{\mathbf{a}_{L,t}^{\operatorname{a r m}}}\\ {\mathbf{a}_{L,t}^{\operatorname{h a n d}}}\\ {\mathbf{a}_{R,t}^{\operatorname{a r m}}}\\ {\mathbf{a}_{R,t}^{\operatorname{h a n d}}}\\ \end{matrix}\right]\in\mathbb{R}^{d_{a}},\qquad d_{a}=2\left(d_{\operatorname{a r m}}+d_{\operatorname{h a n d}}\right), $$

where $\mathbf{a}_{j,t}^{\text{arm}}$ and $\mathbf{a}_{j,t}^{\text{hand}}$ are the commanded arm joint angles and hand motor values for side $j \in \{L, R\}$, and $d_{\text{arm}}$ and $d_{\text{hand}}$ denote the dimensions of the arm-joint and hand-motor command vectors for each side. Calibrated wrist and finger motions from every source are converted through the same retargeting and IK pipeline used for robot teleoperation. Across all stages, the robot-native action chunks serve as targets for the same flow-matching loss [^22], [^23], keeping the training objective and output representation unchanged.

Shared action targets make human and robot demonstrations compatible for optimization, but their contributions to learning remain complementary: broad human experience provides reusable manipulation priors, task-specific demonstrations connect these priors to the target task, and

## Algorithm 1 Training and deployment pipeline

Require: In-the-wild ego data $\mathcal{D}_{\mathrm{HB}}$, task-specific ego data $\mathcal{D}_{\mathrm{HT}}$, glove demonstrations $\mathcal{D}_{\mathrm{G}}$, robot data $\mathcal{D}_{\mathrm{R}}$, and instruction $\ell$.

Ensure: A deployed bimanual dexterous VLA policy $\pi$.

1: Build shared robot-native supervision:
2: Initialize processed subsets $\mathcal{D}_{s}$ for $s \in \{\mathrm{HB}, \mathrm{HT}, \mathrm{G}, \mathrm{R}\}$.
3: for each source s and trajectory $\tau \in \mathcal{D}_{s}$ do
4: Read wrist poses, finger signals, images, and instructions.
5: Apply shared calibration, dual-arm IK, and hand retargeting to obtain commands $A_{t} \in \mathbb{R}^{d_{a}}$.
6: Preserve source observations and store $(o_{t}, \ell, \mathbf{q}_{t}, A_{t})$ in $\widetilde{\mathcal{D}}_{s}$.
7: end for
8: Progressively train the VLA policy:
9: Perform Human-to-Robot learning on $\widetilde{\mathcal{D}}_{\mathrm{HB}}$.
10: Begin Human-Robot co-training by training GEOFORMER on unpaired images from $\mathcal{D}_{\mathrm{HT}}$ and $\mathcal{D}_{\mathrm{R}}$.
11: Freeze GEOFORMER and use it to align ego observations in $\mathcal{D}_{\mathrm{HT}}$.
12: Jointly train the VLM and action expert on $\widetilde{\mathcal{D}}_{\mathrm{HT}} \cup \widetilde{\mathcal{D}}_{\mathrm{G}} \cup \widetilde{\mathcal{D}}_{\mathrm{R}}$.
13: Refine in the robot domain using expert execution and recovery trajectories from $\widetilde{\mathcal{D}}_{\mathrm{R}}$.
14: Deploy with online correction:
15: while executing a task on the real robot do
16: Predict and execute action chunks from $\pi(\delta_{t}, \ell, \mathbf{q}_{t})$.
17: if human correction is required then
18: Apply Anchored Delta-Cmd and add corrections to $\widetilde{\mathcal{D}}_{\mathrm{R}}$, then continue refinement.
19: end if
20: end while
21: return Deployed policy $\pi$.

robot execution supplies contact and recovery experience. Motivated by this observation, we structure training around the progression, moving from broad prior learning through human-robot co-training to robot-domain refinement while retaining the same policy architecture.

b) Human-to-Robot learning: The first stage learns broad bimanual semantics and motion patterns from EgoWild demonstrations. Ego-human observations are preserved in their original form, allowing the policy to learn from the visual and behavioral diversity of the in-the-wild corpus.

c) Human–Robot co-training: The second stage first trains GEOFORMER on unpaired task-specific ego and robot images using the view-alignment objective in Sec. III-A. Then, we freeze GEOFORMER and use it to align task-specific ego observations while jointly training the VLM and action expert on the aligned ego data, robot-view glove demonstrations, and a small amount of robot demonstration data. This stage aims to adapt the broad policy to target tasks.

d) Robot-domain refinement: The final stage uses expert robot demonstrations and recovery trajectories to reduce the remaining embodiment and contact-dynamics gaps. Recovery trajectories are collected using Dataset Aggregation (DAgger) [^52], which augments the training set with expert recoveries from policy-visited states.

Every stage optimizes the same conditional mapping $(\tilde{o}_t^s, \ell, \mathbf{q}_t) \mapsto \mathbf{A}_t$, while the policy parameters continue to be updated. Here, $\tilde{o}_t^s$ is the visual observation at time t from data source s, after GEoFORMER alignment when applicable; $\mathbf{q}_t \in \mathbb{R}^{d_a}$ is the current measured joint state; $\ell$ is the language task instruction; and $\mathbf{A}_t := [A_t, A_{t+1}, \ldots, A_{t+H-1}] \in \mathbb{R}^{d_a \times H}$ is

---

an action chunk of horizon H. Each $A_{t}$ contains the arm joint targets and hand motor commands defined in Eq. (6).

### C. Data Sources

a) Ego-Human Demonstrations (EgoWild): We create EgoWild by collecting bimanual human behavior with VR devices at physically distinct real-world sites, including homes, supermarkets, parcel stations, factories, pharmacies, and offices, rather than recreating a small set of spaces. Objects, clutter, lighting, and layouts remain in their native state, exposing natural environmental diversity and scene clutter. Collection is open-ended and unscripted: operators plan and complete real tasks according to the local context without predefined action sequences. Raw long recordings preserve the behavioral chain, including observation, decision making, task switching, trial and error, locomotion, and inspection. For Stage 1 training, we retain quality-filtered manipulation clips with limited translational movement, keeping the supervision focused on manipulation.

EgoWild combines high-resolution egocentric video, tracked hand motion, and temporally structured language annotations. Videos are captured at $2048 \times 1536$, compared with $1920 \times 1080$ for EgoDex [^1] and $640 \times 360$ for EgoVerse [^2]. Collectors wear a calibrated tracker on each hand: in our 3D reaching test, median absolute fingertip-position error decreases from 4.21 cm without the tracker to 0.66 cm with it. Language annotations are organized at three timestamped levels: atomic-action segments, segment descriptions, and complete skills composed of multiple segments. In contrast, EgoDex primarily provides episode-level descriptions, while EgoVerse provides episode subtasks without corresponding temporal boundaries. The final processed corpus used for Stage 1 training contains 179,049 episodes and 538.9 hours of egocentric activity (58.2M frames at 30 Hz). As shown in Fig. 1, we use SAM 3.1 [^53] to segment foreground instances to quantify visual complexity. EgoWild contains an average of 34.78 foreground instances per frame, compared with 4.51 in EgoDex and 22.73 in EgoVerse. It also exhibits substantially greater mean cumulative head rotation per second ( $15.93^{\circ}$ versus $4.19^{\circ}$ and $4.62^{\circ}$). These metrics reflect denser scene clutter and more dynamic viewpoints in EgoWild. Additionally, we collect a task-specific corpus following the same data-collection pipeline in recreated downstream layouts for adaptation. GEOFORMER then transforms its ego observations toward the robot viewpoint; see Sec. III-A. For motion alignment, we express all poses in the initial headset coordinate frame and calibrate the VR pose corresponding to the robot-arm zero configuration. We then compute hand motions relative to this reference and obtain robot-relative trajectories through a fixed coordinate transform.

b) Glove Demonstrations: Glove demonstrations provide robot-view observations and retargeted arm-hand action labels. As depicted in Fig. 2, the operator performs downstream tasks under the same three-camera setup used at deployment, preserving viewpoint, workspace scale, and occlusion patterns of the robot. VR controllers track arm end-effector poses, while data gloves capture finger articulation; relative-pose mapping, dual-arm IK, and hand retargeting convert these signals into robot-native commands. Compared with headset-based hand tracking, controllers and gloves provide more accurate wrist-pose and finger-articulation measurements, respectively. Both devices are the same as those used for robot teleoperation; combined with the shared action mapping, this is expected to bring the resulting action supervision closer to the distribution of commands received by the robot. Thus, glove demonstrations supply deployment-matched observations and higher-fidelity action supervision before costly physical execution.

c) Real-Robot Demonstrations: Real-robot teleoperation applies the same mapping online and sends the resulting commands to the physical robot. Built on XRoboToolkit [^54], the system supports bare-hand tracking, auxiliary trackers, and VR controllers; we use VR controllers as they provide the most stable pose signal in our setup. Teleoperation supplies contact-grounded demonstrations and subsequently supports human intervention; light low-pass filtering suppresses sensing noise in the hand commands.

### D. Reliable Deployment and Recovery Collection

Contact-rich dexterous manipulation requires robust execution. To this end, we develop a reliable system for teleoperation, policy deployment, and human correction: we adopt curated calibration and filtering to produce smooth action targets; for low-level control, we apply compliant torque control to preserve contact intent while absorbing small pose errors; and overlap blending prevents discontinuities between predicted action chunks. For recovery collection, we propose Anchored Delta-Cmd, a human-takeover mechanism that anchors robot commands at their takeover values and updates them using subsequent relative operator motion, avoiding command jumps. These engineering components make data collection and closed-loop deployment robust and serve as the foundational infrastructure for our whole learning system.

### IV. EXPERIMENTS

We evaluate EGOWILD2DEX on real-world bimanual dexterous manipulation through four questions: RQ1: How does EGOWILD2DEX compare with direct robot-only adaptation under limited robot data? RQ2: How does each component contribute within the progressive training pipeline? RQ3: How does GEOFORMER affect visual transfer quality and efficiency? RQ4: How does the amount of human data affect downstream task performance, and how well does the learned policy generalize and transfer across objects and embodiments?

### A. Experimental Setup

a) Robot platform: We conduct experiments mainly on two six-joint Piper arms equipped with two six-channel Revo2 hands. Observations come from a fixed top-down camera and wrist cameras that move with the arms. The robot uses the arm-hand commands defined in Eq. (6).

---

<div style="text-align: center;">TABLE I</div>

<div style="text-align: center;">DIRECT ADAPTATION OF PUBLIC PRETRAINED VLAs. WE TRAIN THE PUBLIC PRETRAINED VLAs ON LIMITED ROBOT EXPERT DATA AND COMPARE THEIR PERFORMANCE WITH THAT OF OUR FULL PIPELINE.</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'>Method</td><td style='text-align: center; word-wrap: break-word;'>Open-Box</td><td style='text-align: center; word-wrap: break-word;'>Glue-Fig.</td><td style='text-align: center; word-wrap: break-word;'>Ice-Water</td><td style='text-align: center; word-wrap: break-word;'>Avg.</td></tr><tr><td colspan="5">Direct fine-tuning of pretrained VLAs</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>$\pi_0$ [^22]</td><td style='text-align: center; word-wrap: break-word;'>2.0/0.0</td><td style='text-align: center; word-wrap: break-word;'>3.7/0.0</td><td style='text-align: center; word-wrap: break-word;'>2.4/10.0</td><td style='text-align: center; word-wrap: break-word;'>2.7/3.3</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>$\pi_{0.5}$ [^23]</td><td style='text-align: center; word-wrap: break-word;'>4.0/0.0</td><td style='text-align: center; word-wrap: break-word;'>4.1/10.0</td><td style='text-align: center; word-wrap: break-word;'>4.6/10.0</td><td style='text-align: center; word-wrap: break-word;'>4.2/6.7</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>GR00T-N1.7 [^55]</td><td style='text-align: center; word-wrap: break-word;'>2.5/0.0</td><td style='text-align: center; word-wrap: break-word;'>1.4/0.0</td><td style='text-align: center; word-wrap: break-word;'>2.4/0.0</td><td style='text-align: center; word-wrap: break-word;'>2.1/0.0</td></tr><tr><td colspan="5">Human-robot alignment and co-training (full pipeline)</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>EGOWILD2DEX (Ours)</td><td style='text-align: center; word-wrap: break-word;'>9.5/90.0</td><td style='text-align: center; word-wrap: break-word;'>10.0/100.0</td><td style='text-align: center; word-wrap: break-word;'>10.0/100.0</td><td style='text-align: center; word-wrap: break-word;'>9.8/96.7</td></tr></table>

<div style="text-align: center;">TABLE II</div>

<div style="text-align: center;">ABLATION OF THE PROGRESSIVE TRAINING PIPELINE. WE ABLATE THE TWO PRECEDING HUMAN-DATA STAGES. THE SUPERSCRIPT  $\frac{1}{2}$ DENOTES ROBOT-DOMAIN REFINEMENT AUGMENTED WITH DAGGER.</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td colspan="3">Training Recipe</td><td colspan="2">Performance</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Human-to-Robot learning</td><td style='text-align: center; word-wrap: break-word;'>Human-Robot co-training</td><td style='text-align: center; word-wrap: break-word;'>Robot-domain refinement</td><td style='text-align: center; word-wrap: break-word;'>Score</td><td style='text-align: center; word-wrap: break-word;'>Succ.</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>✗</td><td style='text-align: center; word-wrap: break-word;'>✗</td><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>1.4</td><td style='text-align: center; word-wrap: break-word;'>0.0%</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>✗</td><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>5.1</td><td style='text-align: center; word-wrap: break-word;'>20.0%</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>✗</td><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>7.5</td><td style='text-align: center; word-wrap: break-word;'>40.0%</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>8.8</td><td style='text-align: center; word-wrap: break-word;'>70.0%</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>✓ $^{\ddagger}$</td><td style='text-align: center; word-wrap: break-word;'>9.5</td><td style='text-align: center; word-wrap: break-word;'>90.0%</td></tr></table>

b) Evaluation tasks: Our three long-horizon bimanual tasks comprise sequential subtasks requiring dexterous arm-hand coordination and contact-rich interaction. Scores measure subtask completion; numbers in parentheses below indicate the points awarded for completing each subtask.

Task I: Open-Box. The robot is required to grasp a utility knife (2), cut the tape (2), open the box (4), and remove the bouquet (2).

Task II: Glue-Figure. The robot grasps a hot glue gun (2), applies glue (3: dispensing 1, accurate placement 2), picks up the figure (2), and places it on the stand (3: placement 1, alignment 2).

Task III: Ice-Water. The robot grasps the scoop (2), transfers ice into a cup (3: scooping 1, pouring 2), positions the cup on the dispenser (2), and adds water (3: lever contact 1, aligned dispensing 2).

c) Metrics: We report the mean 10-point completion score and success rate over 10 independent trials per method and task; success requires completing the full instruction.

### B. Motivating Study: Limits of Robot-Only Adaptation

We compare direct robot-only adaptation of $\pi_0$ [^22], $\pi_{0.5}$ [^23], and GR00T-N1.7 (pretrained for dexterous hand manipulation on over 20,000 hours of human egocentric video [^55]) against the complete EGOWILD2DEX pipeline. Each model is fine-tuned on the same 100 expert robot demonstrations per task.

Table I evaluates direct adaptation of public pretrained VLAs to long-horizon bimanual dexterous manipulation using only 100 expert robot demonstrations per task. The full EGOWILD2DEX pipeline serves as an end-to-end reference.

Direct robot-only adaptation remains difficult despite initialization from large public pretrained models. The best adapted checkpoint, $\pi_{0.5}$, reaches an average completion score of 4.2 but completes only 6.7% of trials end-to-end,

![Image](images/eceb4a3866e37a728797a66db7d2aa55d35200ad4585b7dc098862a5c57e16c4.jpg)

<div style="text-align: center;">(a) Stage-1 evaluation loss</div>

![Image](images/b0a28f35b8a0e8f197354a873d3f0fcb1e686084a032ece47724ea318e3ed893.jpg)

<div style="text-align: center;">(b) Human-Robot co-training data scaling</div>

<div style="text-align: center;">Fig. 4. Human-to-Robot learning and human-data scaling. (a) Evaluation loss across action representations and ego-human corpora. (b) Independently varying task-specific ego data and glove demonstrations while keeping the remaining Human-Robot co-training sources fixed.</div>

Ego human (IK Pose) Real robot Overlap (IoU)

![Image](images/e2059eec03c6b3745a9f7e8ea664d373b15f3e99176b6f3c372b860ea4ef61fb.jpg)

<div style="text-align: center;">(a) Pooled PCA support</div>

![Image](images/b58ca4a60cef94cc35aa0d71d4d0e47404d08c0e5aa7bc92cac9b3a6bce2241a.jpg)

<div style="text-align: center;">(b) Group-wise magnitudes</div>

<div style="text-align: center;">Fig. 5. Distributional alignment between retargeted ego-human and real-robot actions. (a) Occupied support in the first two principal components of pooled normalized actions. (b) KDEs of normalized action magnitude for the left/right arms and hands.</div>

showing the limitations of direct adaptation with limited robot supervision and motivating a more effective adaptation strategy. Increasing the number of fine-tuning steps further lowers the training loss but brings little performance gain, reaching only 5.3/0.0% on Task I. For reference, the complete EGOWILD2DEX pipeline reaches a 96.7% average success rate. EGOWILD2DEX incorporates Human-to-Robot learning from in-the-wild ego-human data, task-aligned ego and glove demonstrations, and robot-domain refinement. The contribution of these components is examined next through ablations. To enable rapid iteration while controlling real-robot evaluation time and cost, subsequent ablations focus on the comparatively challenging Open-Box task (Task I).

### C. Ablation of Progressive Training

In this section, we examine the contributions of human-to-robot learning, human-robot co-training, and recovery-augmented robot-domain refinement. Human-robot co-training uses 1,000 task-specific ego demonstrations, 1,000 glove demonstrations, and 100 expert robot demonstrations per task. For robot-domain refinement, we first train on the same 100 expert robot demonstrations used in Human-Robot co-training. We then deploy the policy to collect 10 short DAgger recovery segments and continue fine-tuning on the combined robot data.

Ablation results in Table II highlight the criticality of each component. Conducting robot-domain refinement alone, without either human-data learning stage, achieves only

---

![Image](images/e9548c828bd7ffc9386b26c5a3acdec5ddb26f37c36e5ea8792a2e92eeb2c11b.jpg)

<div style="text-align: center;">(a) Raw ego</div>

![Image](images/9a1884af748f84b7f2221961f83c6e3231aa2c2dddc8d209a9acf435dfd890fe.jpg)

![Image](images/32000b68b7f265431422c6514699ffa9d9910fe1089b73769798d85f3311d794.jpg)

<div style="text-align: center;">(b) Project+Inpaint</div>

![Image](images/08ca9868d936f32085864beccb06eecb0d18ad62eef9bf0c0a1a08ced005d305.jpg)

<div style="text-align: center;">(c) GEOFORMER</div>

<div style="text-align: center;">(d) Robot view</div>

<div style="text-align: center;">Fig. 6. Qualitative comparison of visual alignment. (a) the raw ego observation, (b) MoGe reprojection followed by LaMa inpainting, (c) the GEOFORMER-aligned observation, and (d) the robot-view reference.</div>

1.4/0.0%. Adding only Human-to-Robot learning increases the performance to 5.1/20.0%, while adding only Human-Robot co-training reaches 7.5/40.0%, indicating that task-aligned human supervision provides stronger direct grounding. Combining both stages raises performance to 8.8/70.0%. This confirms our hypothesis that the broad prior and task-specific grounding are complementary. Anchored Delta-Cmd DAgger further improves performance to 9.5/90.0% through policy-failure recoveries that address the execution-distribution gap.

### D. Analysis of Human-to-Robot Learning

We investigate how action representation and the choice of ego-human dataset affect Stage 1 learning. We also compare retargeted human and real-robot command distributions to assess their statistical similarity in the shared action space.

We compare delta end-effector poses with absolute robot-native arm and hand commands on EgoWild. In Fig. 4(a), evaluation loss decreases and converges with absolute joint-angle targets but increases with delta end-effector pose targets. These results validate our choice of absolute robot-native commands for Human-to-Robot learning.

To assess this action space, we compare retargeted human commands with commands from real-robot demonstrations in Fig. 5. We measure overlap in a shared PCA projection and similarity between normalized action-magnitude distributions for each arm and hand. From 10,000 randomly sampled training frames in each of the ego-human and robot domains, we obtain a PCA-support IoU of 0.73, while the left-arm, left-hand, right-arm, and right-hand density overlaps are 0.77, 0.81, 0.70, and 0.87, respectively (mean 0.79). These results suggest our retargeting pipeline aligns human actions with the robot-native action space.

We next compare EgoWild with EgoDex [^1], processing both datasets with the same IK and hand-retargeting pipeline and using the same absolute robot-native action representation. EgoDex contains 314,834 episodes, $1.76\times$ as many as EgoWild (179,049 episodes). Nevertheless, the model trained on EgoWild achieves a lower evaluation loss in Fig. 4(a). We attribute this advantage to EgoWild's variation in environments, objects, viewpoints, and execution styles, which discourages setup-specific shortcuts and promotes manipulation patterns that transfer across environments.

<div style="text-align: center;">TABLE III</div>

### E. Efficiency and Quality of GEOFORMER

To evaluate the efficiency and quality of GEOFORMER, we compare GEOFORMER-aligned input with raw ego input and Project+Inpaint-aligned input [^50], [^51].

<div style="text-align: center;">EFFICIENCY, IMAGE SIMILARITY, AND DOWNSTREAM PERFORMANCE OF VISUAL ALIGNMENT. RAW USES UNaligned EGO IMAGES; PROJECT+INPAINT APPLIES MOGE REPROJECTION FOLLOWED BY LAMA INPAINTING; GEOFORMER IS OUR METHOD.</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td rowspan="2">Method</td><td colspan="3">Alignment Efficiency</td><td colspan="2">Image Similarity</td><td colspan="2">Task I: Open-Box</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Mean (ms) $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>P95 (ms) $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>GPU-h / 1M $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>PSNR $\uparrow$</td><td style='text-align: center; word-wrap: break-word;'>LPIPS $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>Score $\uparrow$</td><td style='text-align: center; word-wrap: break-word;'>Succ. $\uparrow$</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Raw</td><td style='text-align: center; word-wrap: break-word;'>-</td><td style='text-align: center; word-wrap: break-word;'>-</td><td style='text-align: center; word-wrap: break-word;'>-</td><td style='text-align: center; word-wrap: break-word;'>11.7187</td><td style='text-align: center; word-wrap: break-word;'>0.7293</td><td style='text-align: center; word-wrap: break-word;'>7.7</td><td style='text-align: center; word-wrap: break-word;'>60.0%</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Project+Inpaint</td><td style='text-align: center; word-wrap: break-word;'>162.58</td><td style='text-align: center; word-wrap: break-word;'>194.16</td><td style='text-align: center; word-wrap: break-word;'>45.16</td><td style='text-align: center; word-wrap: break-word;'>12.8213</td><td style='text-align: center; word-wrap: break-word;'>0.7146</td><td style='text-align: center; word-wrap: break-word;'>8.0</td><td style='text-align: center; word-wrap: break-word;'>80.0%</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>GEOFORMER</td><td style='text-align: center; word-wrap: break-word;'>7.43</td><td style='text-align: center; word-wrap: break-word;'>12.97</td><td style='text-align: center; word-wrap: break-word;'>2.06</td><td style='text-align: center; word-wrap: break-word;'>13.8749</td><td style='text-align: center; word-wrap: break-word;'>0.6327</td><td style='text-align: center; word-wrap: break-word;'>9.5</td><td style='text-align: center; word-wrap: break-word;'>90.0%</td></tr></table>

<div style="text-align: center;">TABLE IV</div>

<div style="text-align: center;">GENERALIZATION TO DIFFERENT PACKAGE CONTENTS. ALL VARIANTS ORIGINATE FROM THE BOUQUET PACKAGE-OPENING POLICY. UNSEEN OBJECTS ARE EVALUATED WITHOUT OBJECT-SPECIFIC ADAPTATION.</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td rowspan="2"></td><td style='text-align: center; word-wrap: break-word;'>Base</td><td colspan="4">Seen</td><td colspan="4">Unseen</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Bouquet</td><td style='text-align: center; word-wrap: break-word;'>Bread</td><td style='text-align: center; word-wrap: break-word;'>Pouch</td><td style='text-align: center; word-wrap: break-word;'>Plush</td><td style='text-align: center; word-wrap: break-word;'>Avg.</td><td style='text-align: center; word-wrap: break-word;'>Cable</td><td style='text-align: center; word-wrap: break-word;'>Tissues</td><td style='text-align: center; word-wrap: break-word;'>Coke</td><td style='text-align: center; word-wrap: break-word;'>Avg.</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Score</td><td style='text-align: center; word-wrap: break-word;'>9.5</td><td style='text-align: center; word-wrap: break-word;'>7.0</td><td style='text-align: center; word-wrap: break-word;'>8.8</td><td style='text-align: center; word-wrap: break-word;'>8.6</td><td style='text-align: center; word-wrap: break-word;'>8.1</td><td style='text-align: center; word-wrap: break-word;'>6.9</td><td style='text-align: center; word-wrap: break-word;'>7.4</td><td style='text-align: center; word-wrap: break-word;'>6.8</td><td style='text-align: center; word-wrap: break-word;'>7.0</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Succ.</td><td style='text-align: center; word-wrap: break-word;'>90.0%</td><td style='text-align: center; word-wrap: break-word;'>60.0%</td><td style='text-align: center; word-wrap: break-word;'>80.0%</td><td style='text-align: center; word-wrap: break-word;'>70.0%</td><td style='text-align: center; word-wrap: break-word;'>70.0%</td><td style='text-align: center; word-wrap: break-word;'>30.0%</td><td style='text-align: center; word-wrap: break-word;'>30.0%</td><td style='text-align: center; word-wrap: break-word;'>40.0%</td><td style='text-align: center; word-wrap: break-word;'>33.3%</td></tr></table>

a) Processing efficiency: Table III reports end-to-end alignment latency. The Project+Inpaint baseline applies MoGe reprojection followed by LaMa inpainting on every frame. For both methods, we exclude image decoding and model loading from timing and synchronize CUDA after each frame. Mean latency measures average processing time, P95 latency captures tail latency, and GPU-h/1M converts mean latency into the cost of processing one million frames. Raw ego has no alignment module and is therefore excluded from this comparison. GEOFORMER reduces mean latency from 162.58 ms to 7.43 ms and P95 latency from 194.16 ms to 12.97 ms. It also reduces the cost from 45.16 to 2.06 GPU-hours per million frames, saving 43.10 GPU-hours (95.4%).

b) Image similarity: For the image-similarity results in Table III, we compare action-matched ego and robot frames, summarize each episode by its median score, and then average equally across episodes. PSNR measures pixel-level fidelity; higher is better. LPIPS [^56] measures perceptual feature discrepancy; lower is better. Compared with Project+Inpaint, GEOFORMER improves PSNR by 1.0536 dB and reduces LPIPS by 11.46%. The qualitative comparison in Fig. 6 shows the same trend: Project+Inpaint introduces reconstruction artifacts, whereas GEOFORMER better preserves task-relevant objects and interactions. This stronger alignment also improves the VLA score from 8.0 to 9.5 and the success rate from 80.0% to 90.0%, outperforming Project+Inpaint.

### F. Effect of Human Data Volume

To investigate how human data volume affects downstream task performance, we vary the amounts of task-specific ego data and glove data. Fig. 4(b) shows that both data sources provide their largest measured gains at relatively small data volumes. Performance improves sharply when the first subsets are introduced and then exhibits diminishing returns as additional demonstrations are added. The ego-data curve shows a larger initial change, whereas the glove-data curve saturates earlier. We speculate that task-specific ego data primarily expand task and interaction coverage, while

---

a smaller amount of glove demonstrations is sufficient to anchor the deployment viewpoint and workspace geometry. The shared saturation suggests that further gains rely more on DAgger-style correction of policy-induced out-of-distribution states than additional human data.

### G. Generalization and Transfer

Objects. To inspect the object generalization of EGOWILD2DEX, we vary package contents in Task I, adapting the bouquet-based policy to three seen objects with fewer than 100 additional demonstrations per object. In Table IV, EGOWILD2DEX achieves an average score of 8.1 and a 70.0% success rate on the three seen objects. On three unseen objects, the policy also achieves a success rate of 30.0%–40.0% (33.3% on average). We attribute this promising performance to the broad in-the-wild ego-human prior obtained in Human-to-Robot learning.

Embodiments. To evaluate cross-embodiment transfer, we fine-tune on Tianji Marvin Pro arms with two more arm DoFs in total than Piper, different low-level control, and the same Revo2 hands. This variant achieves a mean score of 9.2 and a 60% success rate in Task I, suggesting that our method can transfer well across embodiments.

### V. CONCLUSION

We present EGOWILD2DEX for transferring in-the-wild human experience to dual-arm dexterous robots through GEOFORMER view alignment and progressive human–robot training in a shared robot-native action space. With 538.9 hours of human experience and less than one hour of robot data per task, our method achieves 90.0%–100.0% success across three long-horizon tasks and 30.0%–40.0% zero-shot success on unseen objects. We hope this work will advance research on scalable VLA models that learn from in-the-wild human experience.

### VI. LIMITATIONS AND FUTURE WORK

## REFERENCES

Learning solely from human demonstrations remains challenging for long-horizon sequential tasks and multi-finger coordination, so the framework still requires real-robot fine-tuning. In addition, GEoFORMER cannot reconstruct the missing content when head rotations move objects out of the camera's field of view. Finally, we have not yet identified a commercially available higher-DoF hand (e.g., 22 DoF) that jointly meets our requirements for operational stability, durability, thermal management, and affordability, and therefore leave evaluation on such hands to future work.

## ACKNOWLEDGMENTS

We thank the data collection team, especially Jian Yang and Yi Zhang. We are grateful to Peihuan Yang and Zhenkun Lin for their assistance with mechanical design and 3D-printed fabrication. We also thank Huige Tong, Haocheng Zhang, and Yuxing Zhang for their project management support. We further thank the maintenance and after-sales service teams at BrainCo, AgileX Robotics, VIRDYN, and Tianji.

[^1]: R. Hoque, P. Huang, D. J. Yoon, M. Sivapurapu, and J. Zhang, “EgoDex: Learning dexterous manipulation from large-scale egocentric video,” arXiv, 2025.

[^2]: R. Punamiya et al., “EgoVerse: An egocentric human dataset for robot learning from around the world,” arXiv, 2026.

[^3]: J. Ma et al., “Robot learning from human videos: A survey,” arXiv, 2026.

[^4]: T. He et al., “OmniH2O: Universal and dexterous human-to-humanoid whole-body teleoperation and learning,” arXiv, 2024.

[^5]: Spirit AI Team, “Spirit-v1.5: Clean data is the enemy of great robot foundation models,” January 2026, Accessed: Aug. 6, 2026. [Online]. Available: <https://www.spirit-ai.com/en/blog/spirit-v1-5>

[^6]: S. Kareer et al., “EgoMimic: Scaling imitation learning via egocentric video,” arXiv, 2024.

[^7]: R. Yang et al., “EgoVLA: Learning vision-language-action models from egocentric human videos,” arXiv, 2025.

[^8]: H. Luo et al., “Being-H0: vision-language-action pretraining from large-scale human videos,” arXiv, 2025.

[^9]: R. Zheng et al., “EgoScale: Scaling dexterous manipulation with diverse egocentric human data,” arXiv, 2026.

[^10]: G. Chen et al., “HOST: Robots acquire manipulation skills in seconds from a single human video,” arXiv, 2026.

[^11]: Dyna Robotics, “Dyna-2: A 1-million-hour scaling law for world-action models,” August 2026. [Online]. Available: <https://www.dyna.co/dyna-2>

[^12]: BeingBeyond Team, “Being-H0.8: A latent tactile world-action model at scale,” 2026. [Online]. Available: <https://research.beingbeyond.com/being-h08>

[^13]: S. Xie et al., “Human2Robot: Learning robot actions from paired human-robot videos,” in AAAI, 2026, pp. 11078–11086.

[^14]: V. Jain et al., “Vid2Robot: End-to-end video-conditioned policy learning with cross-attention transformers,” arXiv, 2024.

[^15]: C. Wang et al., “MimicPlay: Long-horizon imitation learning by watching human play,” in CoRL, 2023, pp. 201–221.

[^16]: Z. Wang et al., “HumanEgo: Zero-shot robot learning from minutes of human egocentric videos,” arXiv, 2026.

[^17]: V. Liu et al., “EgoZero: Robot learning from smart glasses,” arXiv, 2025.

[^18]: M. Shi et al., “Unlocking in-the-wild loco-manipulation with robot-free egocentric demonstration,” in RSS, 2026.

[^19]: Z. Zhao, Y. Zhao, G. Zhang, C. Liu, M. Zheng, and W. Lian, “HALOMI: Learning humanoid loco-manipulation with active perception from human demonstrations,” arXiv, 2026.

[^20]: B. Zitkovich et al., "RT-2: Vision-language-action models transfer web knowledge to robotic control," in CoRL, 2023, pp. 2165–2183.

[^21]: M. J. Kim et al., “OpenVLA: An open-source vision-language-action model,” arXiv, 2024.

[^22]: K. Black et al., “ $\pi_{0}$: A vision-language-action flow model for general robot control,” arXiv, 2024.

[^23]: K. Black et al., “ $\pi_{0.5}$: a vision-language-action model with open-world generalization,” in CoRL, 2025, pp. 17–40.

2. Cheang et al., “Gr-3 technical report,” arXiv, 2025.

J. Bjorck et al., “GR00T N1: An open foundation model for generalist humanoid robots,” arXiv, 2025.

[^26]: Y. Zhong et al., “DexGraspVLA: A vision-language-action framework towards general dexterous grasping,” in AAAI, 2026, pp. 18836–18844.

[^27]: H. Yuan et al., “Being-0: A humanoid robotic agent with vision-language models and modular skills,” arXiv, 2025.

[^28]: X. Mao et al., “DexSkills: Skill segmentation using haptic data for learning autonomous long-horizon robotic manipulation tasks,” in IROS, 2024, pp. 5104–5111.

[^29]: R. Wen et al., “Gr-dexter technical report,” arXiv, 2025.

[^30]: Z. Zhang et al., “Dexora: Open-source VLA for High-DoF bimanual dexterity,” in ICRA, 2026.

[^31]: Y.-L. Wei et al., “Cyclemanip: Enabling cyclic task manipulation via effective historical perception and understanding,” in CVPR, 2026.

[^32]: Physical Intelligence et al., “ $\pi_{0.6}^*$: A VLA that learns from experience,” arXiv, 2025.

[^33]: Y. Li et al., “GR-RL: Going dexterous and precise for long-horizon robotic manipulation,” arXiv, 2025.

[^34]: S. Nair, A. Rajeswaran, V. Kumar, C. Finn, and A. Gupta, "R3M: A universal visual representation for robot manipulation," in CoRL, 2023, pp. 892–909.

[^35]: T. Xiao, I. Radosavovic, T. Darrell, and J. Malik, “Masked visual pre-training for motor control,” arXiv, 2022.

---

[^36]: K. Lin, P. Chen, D. Huang, T. H. Li, M. Tan, and C. Gan, “Learning vision-and-language navigation from YouTube videos,” in ICCV, 2023, pp. 8317–8326.

[^37]: R. Zheng et al., “FLARE: Robot learning with implicit world modeling,” arXiv, 2025.

[^38]: M. Xu, Z. Xu, C. Chi, M. Veloso, and S. Song, “XSkill: Cross embodiment skill discovery,” in CoRL, 2023, pp. 3536–3555.

[^39]: M. Xu et al., “Flow as the cross-domain manipulation interface,” arXiv, 2024.

[^40]: R.-Z. Qiu et al., “Humanoid policy $\sim$ human policy,” arXiv, 2025.

[^41]: T. Tao, M. K. Srirama, J. J. Liu, K. Shaw, and D. Pathak, “DexWild: Dexterous human interactions for in-the-wild robot policies,” arXiv, 2025.

[^42]: S. Kareer et al., “Emergence of human to robot transfer in vision-language-action models,” arXiv, 2025.

[^43]: Y. Li et al., “EgoLive: A large-scale egocentric dataset from real-world human tasks,” arXiv, 2026.

[^44]: Z. Li et al., “Open-AoE: An open egocentric manipulation dataset and toolchain for embodied learning,” arXiv, 2026.

[^45]: H. Li et al., “ACE-Ego-0: Unifying egocentric human and robotic data for VLA pretraining,” arXiv, 2026.

[^46]: Y. Wang et al., “Ego2Robot: Scalable robot data synthesis from egocentric human data,” arXiv, 2026.

[^47]: M. Jaderberg, K. Simonyan, A. Zisserman, and K. Kavukcuoglu, “Spatial transformer networks,” in NeurIPS, 2015, pp. 2017–2025.

[^48]: C.-H. Lin, E. Yumer, O. Wang, E. Shechtman, and S. Lucey, "STGAN: Spatial transformer generative adversarial networks for image compositing," in CVPR, 2018, pp. 9455–9464.

[^49]: I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, and A. C. Courville, "Improved training of Wasserstein GANs," in NeurIPS, vol. 30, 2017, pp. 5767–5777.

[^50]: R. Wang et al., “MoGe: Unlocking accurate monocular geometry estimation for open-domain images with optimal training supervision,” in CVPR, 2025, pp. 5261–5271.

[^51]: R. Suvorov et al., “Resolution-robust large mask inpainting with fourier convolutions,” in WACV, 2022, pp. 2149–2159.

[^52]: M. Kelly, C. Sidrane, K. Driggs-Campbell, and M. J. Kochenderfer, “HG-DAgger: Interactive imitation learning with human experts,” in ICRA, 2019, pp. 8077–8083.

[^53]: N. Carion et al., “SAM 3: Segment anything with concepts,” arXiv, 2025.

[^54]: Z. Zhao, L. Yu, K. Jing, and N. Yang, "XRoboToolkit: A cross-platform framework for robot teleoperation," in SII. IEEE, 2026, pp. 15–20.

[^55]: E. Llontop and K. Vadrevu, “NVIDIA Isaac GR00T N1.7: Open reasoning VLA model for humanoid robots,” 2026, Accessed: Sep. 14, 2026. [Online]. Available: <https://huggingface.co/blog/nvidia/gr00t-n1-7>

[^56]: R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang, “The unreasonable effectiveness of deep features as a perceptual metric,” in CVPR, 2018, pp. 586–595.

[^57]: D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in ICLR, 2015.

[^58]: AgileX Robotics, “Piper robot arm SDK user manual,” Accessed: Sep. 13, 2026. [Online]. Available: <https://github.com/agilexrobotics/piper_sdk>

[^59]: C. Yu et al., “ $\chi_{0}$: Resource-aware robust manipulation via taming distributional inconsistencies,” arXiv, 2026.

[^60]: J. Yang et al., “Self-improving robot policy with compositional world model,” in RSS, 2026, paper 12.

[^61]: H. Sakoe and S. Chiba, “Dynamic programming algorithm optimization for spoken word recognition,” IEEE Transactions on Acoustics, Speech, and Signal Processing, vol. 26, no. 1, pp. 43–49, 1978.

[^62]: H. W. Kuhn, “The Hungarian method for the assignment problem,” Naval Research Logistics Quarterly, vol. 2, no. 1–2, pp. 83–97, 1955.

[^63]: Qwen Team, "Qwen3.8-Flash," 2026, Accessed: Sep. 13, 2026. [Online]. Available: <https://www.qwencloud.com/models/qwen3.8-flash>

## APPENDIX

The appendix is organized as follows.

- In Appendix Section A, we provide details of the hardware and data-collection setups, action alignment, GEOFORMER training, policy training, and deployment.

- In Appendix Section B, we describe the experimental protocols, baseline implementations, data composition,

![Image](images/76e21160e8d86bff3b6d5e0d110e888f198b58e9ba757181c24f1190ffc6155c.jpg)

<div style="text-align: center;">(a) Embodiment</div>

![Image](images/f7b3a8ce7972b0fa0ecb97df1e1936e21061d334b63e18cdcd0d9d2b631e1cfb.jpg)

<div style="text-align: center;">(b) Teleoperation</div>

<div style="text-align: center;">Fig. 7. Embodiment and teleoperation setup. (a) The bimanual robot platform with Piper arms; the upper-left inset uses Tianji Marvin Pro arms with the same Revo2 hands and camera layout. (b) The PICO 4 Ultra Enterprise headset and controllers for arm control, and mHandPro gloves for hand control.</div>

additional co-training ablations, action-distribution metrics, and ego-robot pairing procedure.

- In Appendix Section C, we report data-quality evaluations and detail the annotations, dataset curation, and semantic diversity of EgoWild.

- In Appendix Section D, we list the contributions of each author.

### A. Implementation Details

## 1 ) Hardware and Data Collection Setups

a) Robot platform: As shown in Fig. 7(a), we use two AgileX Piper arms for our bimanual robot platform. Each 6-DoF arm carries a tendon-driven 6-DoF BrainCo Revo2 hand and a wrist-mounted camera. We mount a top-down camera between the arms to provide a global view. All three cameras are Intel RealSense D435 cameras. The embodiment-transfer variant shown in the upper-left inset uses Tianji Marvin Pro arms while retaining the same hands and camera layout. We control the robot at 30 Hz.

b) Ego-human data collection: For broad and task-specific ego-human demonstrations, collectors interact with objects using their bare hands. PICO's built-in hand tracking records wrist poses and finger motion, while an auxiliary hand tracker improves tracking accuracy during collection. This lightweight setup allows collectors to record their daily activities naturally.

c) Glove and real-robot data collection: Fig. 7(b) shows the hardware used to collect glove demonstrations and real-robot demonstrations. A PICO 4 Ultra Enterprise headset and its handheld controllers record arm end-effector poses, while mHandPro gloves measure finger articulation. These measurements control the robot arms and dexterous hands during teleoperation. Before policy training, we process all modalities with the same calibration, IK, hand-retargeting, and command-safety pipeline described in the following subsection.

---

2) Unified Action Alignment: The wrist feature $\mathbf{b}_{j,t}^{\mathrm{wrist}}$ specifies the measured 6-DoF operator pose, while $\mathbf{b}_{j,t}^{\mathrm{finger}}$ contains the tracked or glove-based finger channels. Let $X_{j,t}^{\mathrm{src}} = \mathrm{Pose}(\mathbf{b}_{j,t}^{\mathrm{wrist}}) \in \mathrm{SE}(3)$. Relative motion from the calibrated configuration at $t_0$ defines the target robot end-effector pose,

 $$ \begin{array}{r}{X_{j,t}^{\star}=\mathrm{R e l P o s e}\big(X_{j,t}^{\mathrm{s r c}};X_{j,t_{0}}^{\mathrm{s r c}},X_{j,t_{0}}^{\mathrm{e e}}\big),}\end{array} $$

where $X_{j,t_0}^{src}$ and $X_{j,t_0}^{ee}$ are the operator and robot end-effector poses at calibration, respectively, and RelPose maps subsequent operator motion to the target robot pose $X_{j,t}^{\star}$ for side j. The arm and hand components are then generated in parallel,

 $$ \begin{aligned}\mathbf{a}_{j,t}^{\mathrm{arm}}&=\mathcal{H}_{\mathrm{arm}}\big(\mathbf{b}_{j,t}^{\mathrm{wrist}}\big):=\mathrm{IK}\big(X_{j,t}^{\star}\big),\\\mathbf{a}_{j,t}^{\mathrm{hand}}&=\mathcal{H}_{\mathrm{hand}}\Big(\mathbf{b}_{j,t}^{\mathrm{finger}}\Big),\end{aligned} $$

where $\mathcal{H}_{arm}$ combines calibrated relative-pose construction with constrained inverse kinematics, and $\mathcal{H}_{hand}$ maps calibrated finger features to normalized hand motor commands. The same calibration conventions, joint constraints, and channel ordering are used for every demonstration source.

## 3 ) GEOFORMER Training

a) Alignment process: GEoFORMER learns from unpaired ego- and robot-view images $I_{\text{ego}}, I_{\text{rob}} \in \mathbb{R}^{3 \times H_{\text{img}} \times W_{\text{img}}}$, where $H_{\text{img}} \times W_{\text{img}}$ denotes the input resolution. During both GEoFORMER training and subsequent ego-frame conversion for policy co-training, each ego image is paired with a robot-view image randomly sampled from the corresponding task's robot-image pool. The model represents their geometric difference by an eight-dimensional projective vector $\mathbf{p} \in \mathbb{R}^8$ and refines it through N predictors with independent parameters. A parameter vector $\mathbf{p} = [p_1, \ldots, p_8]^\top$ is first mapped to the traceless generator

 $$ P_{\mathbf{p}}=\left[\begin{matrix}{p_{3}}&{p_{2}}&{p_{1}}\\ {p_{6}}&{-p_{3}-p_{7}}&{p_{5}}\\ {p_{4}}&{p_{8}}&{p_{7}}\\ \end{matrix}\right], $$

where $P_{\mathbf{p}} \in \mathbb{R}^{3 \times 3}$ has zero trace and encodes the eight projective parameters. We approximate its matrix exponential using a finite series with $K_{\text{terms}}$ retained terms:

 $$ \mathcal{T}_{\mathbf{p}}=\exp(P_{\mathbf{p}})\approx\sum_{k=0}^{K_{\mathrm{t e r m s}}-1}\frac{P_{\mathbf{p}}^{k}}{k!}, $$

where $K_{terms}$ is the number of retained series terms and $\mathcal{T}_{p}$ is the resulting homography. The identity term makes small values of p produce transformations close to the identity, which suits progressive residual refinement.

During training, the initial estimate is $\mathbf{p}_0 = \sigma_{\text{pert}} \epsilon$, where $\epsilon \sim \mathcal{N}(\mathbf{0}_8, \text{Id}_8)$. At the n-th refinement step, we warp the robot view with the current estimate,

 $$ \begin{array}{r}{I_{\mathrm{r o b},n-1}^{\mathrm{w a r p}}=\mathrm{W a r p}(I_{\mathrm{r o b}};\mathcal{T}_{{\bf p}_{n-1}}),}\end{array} $$

where Warp denotes differentiable image warping. The next residual is predicted from the ego image and the currently warped robot image,

 $$ \begin{array}{r}{\Delta\mathbf{p}_{n}=\mathcal{G}_{n}(I_{\mathrm{e g o}},I_{\mathrm{r o b},n-1}^{\mathrm{w a r p}}),}\end{array} $$

where $\mathcal{G}_n$ is the n-th predictor. The estimate is refined as $\mathbf{p}_n = \mathbf{p}_{n-1} + \Delta \mathbf{p}_n$, and the final robot-to-ego transformation is $\mathcal{T} = \exp(P_{\mathbf{p}_N})$.

After a fixed reference mapping from normalized to image coordinates, each transformed homogeneous point $\tilde{\mathbf{h}} = [\tilde{h}_{u_2}, \tilde{h}_{v}, \tilde{h}_w]^\top$ is normalized by $u' = \tilde{h}_u / \tilde{h}_w$ and $v' = \tilde{h}_v / \tilde{h}_w$. We then use differentiable bilinear sampling and mark coordinates outside the image support as invalid.

For unpaired training, the final transformation is applied to both the robot-view image and an all-ones support mask:

 $$ I_{\mathrm{r o b}}^{\mathrm{w a r p}}=\mathrm{W a r p}(I_{\mathrm{r o b}};\mathcal{T}),\qquad M_{\mathcal{T}}=\mathrm{c l i p}(\mathrm{W a r p}(\mathbf{1};\mathcal{T}),0,1) $$

where $I_{\mathrm{rob}}^{\mathrm{warp}}$ is the warped robot image and $M_{\mathcal{T}} \in [0,1]^{H_{\mathrm{img}} \times W_{\mathrm{img}}}$ identifies its valid region. We form $I_{\mathrm{rob}+\mathrm{ego}} = M_{\mathcal{T}} \odot I_{\mathrm{rob}}^{\mathrm{warp}} + (1 - M_{\mathcal{T}}) \odot I_{\mathrm{ego}}$, where $\odot$ denotes element-wise multiplication. The critic evaluates whether the inserted robot content is geometrically compatible with the ego background, requiring neither paired images nor explicit 3D reconstruction.

After view-alignment training at the start of Stage 2, we discard the critic, freeze the predictor, and remove the random perturbation before training the VLM and action expert. Since training estimates the robot-to-ego transformation $\mathcal{T}$, its inverse produces the desired ego-to-robot-aligned observation:

 $$ I_{\mathrm{e g o}}^{\mathrm{a l i g n}}=\mathrm{W a r p}\big(M_{\mathcal{T}}\odot I_{\mathrm{e g o}};\mathcal{T}^{-1}\big), $$

where $I_{\text{ego}}^{\text{align}}$ is the ego observation aligned toward the robot view using the inverse homography $\mathcal{T}^{-1}$. This aligned image is passed directly to the robot policy without scene reconstruction or content synthesis.

b) Loss design: The geometric predictor is trained with the combined objective

 $$ \mathcal{L}_{G}=\mathcal{L}_{\mathrm{a d v}}+\lambda_{\mathrm{d i s p}}\mathcal{L}_{\mathrm{d i s p}}+\lambda_{\mathrm{c o v}}\mathcal{L}_{\mathrm{c o v}}, $$

where $\mathcal{L}_{\mathrm{adv}}$ aligns the transformed robot content with ego-view geometry, $\mathcal{L}_{\mathrm{disp}}$ limits the size of each refinement update, and $\mathcal{L}_{\mathrm{cov}}$ prevents the valid warped region from becoming too small or covering nearly the entire image. The weights $\lambda_{\mathrm{disp}}$ and $\lambda_{\mathrm{cov}}$ balance the regularizers against the adversarial term.

Let $\mathcal{C}$ denote the critic, which maps an image to a scalar score. The adversarial term $\mathcal{L}_{\mathrm{adv}} = -\mathbb{E}\left[\mathcal{C}(I_{\mathrm{rob}+\mathrm{ego}})\right]$ encourages the inserted robot region to be geometrically compatible with the ego-view background. The displacement term penalizes large residuals across all refinement steps, $\mathcal{L}_{\mathrm{disp}} = \sum_{n=1}^{N} \mathbb{E}\left[\|\Delta\mathbf{p}_n\|_2^2\right]$, which stabilizes the progressive refinement. The coverage term controls how much of the warped robot view remains valid. Let c denote the mean of the valid-support mask $M_\mathcal{T}$. We use $\mathcal{L}_{\mathrm{cov}} = [c_{\min} - c]_+ + [c - c_{\max}]_+$, where $[x]_+ = \max(x, 0)$. This rules out transformations that retain too little useful content or occupy almost the full frame. The critic is trained as a Wasserstein critic [^49]. For critic updates, we sample a recent composite $I_{\mathrm{rob}+\mathrm{ego}}^{\mathrm{hist}}$ from a history buffer and minimize

 $$ \begin{array}{r}{\mathcal{L}_{\mathcal{C}}=\mathbb{E}\big[\mathcal{C}\big(I_{\mathrm{r o b+e g o}}^{\mathrm{h i s t}}\big)\big]-\mathbb{E}[\mathcal{C}(I_{\mathrm{e g o}})]+\lambda_{\mathrm{g r a d}}\mathcal{L}_{\mathrm{g r a d}}.}\end{array} $$

---

<div style="text-align: center;">TABLE V</div>

<div style="text-align: center;">GEOFORMER ARCHITECTURE AND TRAINING HYPERPARAMETERS.</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'>Hyperparameter</td><td style='text-align: center; word-wrap: break-word;'>Value</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Input resolution</td><td style='text-align: center; word-wrap: break-word;'>$240 \times 320$</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Transformation model</td><td style='text-align: center; word-wrap: break-word;'>Homography</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Transformation dimension</td><td style='text-align: center; word-wrap: break-word;'>8</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Refinement steps N</td><td style='text-align: center; word-wrap: break-word;'>5</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Retained exponential-series terms $K_{terms}$</td><td style='text-align: center; word-wrap: break-word;'>20</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Training iterations</td><td style='text-align: center; word-wrap: break-word;'>5000</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Batch size</td><td style='text-align: center; word-wrap: break-word;'>8</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Predictor learning rate</td><td style='text-align: center; word-wrap: break-word;'>$1 \times 10^{-6}$</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Critic learning rate</td><td style='text-align: center; word-wrap: break-word;'>$1 \times 10^{-6}$</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Adam [^57] ( $\beta_1, \beta_2$)</td><td style='text-align: center; word-wrap: break-word;'>(0.5, 0.999)</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Predictor initialization std.</td><td style='text-align: center; word-wrap: break-word;'>0.02</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Critic initialization std.</td><td style='text-align: center; word-wrap: break-word;'>0.01</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Initial perturbation $\sigma_{pert}$</td><td style='text-align: center; word-wrap: break-word;'>0.1</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>$\lambda_{disp}$</td><td style='text-align: center; word-wrap: break-word;'>0.5</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>$\lambda_{grad}$</td><td style='text-align: center; word-wrap: break-word;'>10.0</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>$\lambda_{cov}$</td><td style='text-align: center; word-wrap: break-word;'>1.0</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Valid-coverage interval</td><td style='text-align: center; word-wrap: break-word;'>[0.4, 0.8]</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Predictor updates / iteration</td><td style='text-align: center; word-wrap: break-word;'>1</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Maximum critic updates / iteration</td><td style='text-align: center; word-wrap: break-word;'>2</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Critic margin threshold $\tau_C$</td><td style='text-align: center; word-wrap: break-word;'>0.3</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Predictor gradient clipping</td><td style='text-align: center; word-wrap: break-word;'>5.0</td></tr></table>

The first two terms separate generated and real ego images. Specifically, we interpolate $\widehat{I} = \alpha I_{\text{ego}} + (1 - \alpha)I_{\text{rob + ego}}^{\text{hist}}$, with $\alpha \sim \mathcal{U}(0,1)$, and apply

 $$ \mathcal{L}_{\mathrm{g r a d}}=\mathbb{E}_{\widehat{I}}\left[\left(\left\|\nabla_{\widehat{I}}\mathcal{C}(\widehat{I})\right\|_{2}-1\right)^{2}\right]. $$

Finally, a score-margin gate prevents the critic from becoming much stronger than the predictor. With $s_{\text{real}} = \mathbb{E}[\mathcal{C}(I_{\text{ego}})]$ and $s_{\text{fake}} = \mathbb{E}[\mathcal{C}(I_{\text{rob+ego}}^{\text{hist}})]$, we suspend critic updates when $s_{\text{real}} - s_{\text{fake}} \geq \tau_C$ while still updating the geometric predictor.

This promotes unit gradient norm along the interpolation path and stabilizes Wasserstein training.

c) Optimization: Each independent predictor uses five stride-2 convolutional layers with widths (32, 64, 128, 256, 512). At every scale, the average-pooled six-channel input is concatenated with the convolutional features. The resulting representation passes through a 256-dimensional fully connected layer and a linear eight-dimensional output head. ReLU is used throughout except at the output. The fully convolutional critic uses five $4 \times 4$ stride-2 layers with the same channel widths, followed by a linear $3 \times 3$ convolution that produces a spatial score map; its hidden layers use Leaky ReLU with slope 0.2. Neither network uses normalization. We initialize weights from zero-mean Gaussian distributions and biases to zero. The remaining architecture and training settings are summarized in Table V.

4) Policy Training Hyperparameters: Tables VI–VIII summarize the policy training settings for Stage 1 (Human-to-Robot learning), Stage 2 (Human–Robot co-training), and Stage 3 (robot-domain refinement). All stages use NVIDIA A100 GPUs, with 32 GPUs for Stage 1 and 8 GPUs each for Stages 2 and 3. All stages use a cosine learning-rate schedule with warmup and an action-chunk horizon of $H = 50$; batch

<div style="text-align: center;">TABLE VI</div>

<div style="text-align: center;">STAGE 1 (HUMAN-TO-ROBOT LEARNING) HYPERPARAMETERS.</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'>Hyperparameter</td><td style='text-align: center; word-wrap: break-word;'>Value</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Learning-rate schedule</td><td style='text-align: center; word-wrap: break-word;'>Cosine decay</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Warmup steps</td><td style='text-align: center; word-wrap: break-word;'>5,000</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Peak learning rate</td><td style='text-align: center; word-wrap: break-word;'>$7.0 \times 10^{-5}$</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Decay steps</td><td style='text-align: center; word-wrap: break-word;'>100,000</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Final learning rate</td><td style='text-align: center; word-wrap: break-word;'>$7.0 \times 10^{-6}$</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Training steps</td><td style='text-align: center; word-wrap: break-word;'>100,000</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Global batch size</td><td style='text-align: center; word-wrap: break-word;'>512</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Data-loader workers</td><td style='text-align: center; word-wrap: break-word;'>8</td></tr></table>

<div style="text-align: center;">TABLE VII</div>

<div style="text-align: center;">STAGE 2 (HUMAN–ROBOT CO-TRAINING) HYPERPARAMETERS. THE DATA RATIO IS MAINTAINED WITHIN EACH BATCH.</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'>Hyperparameter</td><td style='text-align: center; word-wrap: break-word;'>Value</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Learning-rate schedule</td><td style='text-align: center; word-wrap: break-word;'>Cosine decay</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Warmup steps</td><td style='text-align: center; word-wrap: break-word;'>2,800</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Peak learning rate</td><td style='text-align: center; word-wrap: break-word;'>$3.5 \times 10^{-5}$</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Decay steps</td><td style='text-align: center; word-wrap: break-word;'>50,000</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Final learning rate</td><td style='text-align: center; word-wrap: break-word;'>$3.5 \times 10^{-6}$</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Training steps</td><td style='text-align: center; word-wrap: break-word;'>50,000</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Global batch size</td><td style='text-align: center; word-wrap: break-word;'>128</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Data-loader workers</td><td style='text-align: center; word-wrap: break-word;'>8</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Batch ratio (ego : glove : robot)</td><td style='text-align: center; word-wrap: break-word;'>$\approx 1:1:1$</td></tr></table>

<div style="text-align: center;">TARI F VIII</div>

<div style="text-align: center;">STAGE 3 (ROBOT-DOMAIN REFINEMENT) HYPERPARAMETERS.</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'>Hyperparameter</td><td style='text-align: center; word-wrap: break-word;'>Value</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Learning-rate schedule</td><td style='text-align: center; word-wrap: break-word;'>Cosine decay</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Warmup steps</td><td style='text-align: center; word-wrap: break-word;'>2,800</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Peak learning rate</td><td style='text-align: center; word-wrap: break-word;'>$2.5 \times 10^{-5}$</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Decay steps</td><td style='text-align: center; word-wrap: break-word;'>50,000</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Final learning rate</td><td style='text-align: center; word-wrap: break-word;'>$2.5 \times 10^{-6}$</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Training steps</td><td style='text-align: center; word-wrap: break-word;'>50,000</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Global batch size</td><td style='text-align: center; word-wrap: break-word;'>128</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Data-loader workers</td><td style='text-align: center; word-wrap: break-word;'>8</td></tr></table>

sizes are global. Stage 2 (Human–Robot co-training) begins after GEOFORMER has been trained and frozen. Each Stage 2 batch maintains an approximately 1:1:1 ratio of task-specific ego, glove, and robot demonstrations.

5) Stable Teleoperation, Deployment, and Online Correction: Reliable robot commands are required during operator-collected demonstrations and recoveries, as well as when the learned policy controls the robot at inference time. We therefore stabilize the execution path at three levels: operator input, policy command execution, and human takeover.

a) Stable consumer-grade teleoperation: We use wired communication, packet validation, gap handling, controller-pivot calibration, translation-rotation filtering, rate limits, and temporally initialized IK to suppress tracking jitter and command discontinuities. Tracking loss, excessive IK residuals, or collision detection cause the system to hold the last safe command or stop. The specific calibration and filtering operations are described below.

b) Mixed Integrated Torque Control: During contact, a commanded target slightly inside an object can generate large interaction forces, making rigid position tracking unstable. Moreover, when the measured robot state stalls against the

---

environment, supervising the policy with the next proprioceptive state may suppress the operator's continued contact intent. We therefore supervise the policy on commanded actions and execute the resulting joint-position commands using Mixed Integrated Torque Control provided by the Piper arm SDK [^58]:

 $$ \boldsymbol{\tau}_{t}=\mathbf{K}_{p}\left(\hat{\mathbf{q}}_{t}-\mathbf{q}_{t}\right)-\mathbf{K}_{d}\dot{\mathbf{q}}_{t}, $$

where $\tau_{t}$ is the joint-torque vector, $\hat{q}_{t}$ is the commanded joint position, $\mathbf{q}_{t}$ and $\dot{\mathbf{q}}_{t}$ are the measured joint position and velocity, and $\mathbf{K}_{p}$ and $\mathbf{K}_{d}$ are diagonal proportional and derivative gain matrices. Eq. (18) acts as a joint-space spring–damper: the proportional term drives the robot toward the commanded configuration, while the derivative term damps joint motion. This preserves the commanded motion intent while allowing contact-induced spatial errors to be absorbed compliantly.

c) Streaming action-chunk execution: Like prior flow-matching VLA deployments [^59], [^60], at inference time t, the policy predicts an action chunk $\mathbf{A}_t$ of H future actions. If a new chunk $\mathbf{A}_{t'}$ is predicted before the active chunk $\mathbf{A}_t$ is exhausted ( $t < t' < t + H$), the two chunks overlap for $N_{\mathrm{ov}} = H + t - t'$ steps. Over this window, we linearly blend the action $A_{t' + r}^{old}$ from the active chunk and $A_{t' + r}^{new}$ from the new chunk to obtain the executed action $A_{t' + r}^{exec}$:

 $$ A_{t^{\prime}+r}^{\mathrm{e x e c}}=(1-\eta_{r})A_{t^{\prime}+r}^{\mathrm{o l d}}+\eta_{r}A_{t^{\prime}+r}^{\mathrm{n e w}},\quad\eta_{r}=\frac{r+1}{N_{\mathrm{o v}}+1}, $$

where $r \in \{0, \ldots, N_{\mathrm{ov}} - 1\}$ indexes the overlap window and $\eta_r$ is the linear blending weight. After the overlap window, the newly predicted chunk becomes the active chunk. This interpolation avoids abrupt command replacement between successive policy predictions.

d) Anchored Delta-Cmd DAgger: When the policy enters an out-of-distribution state, the operator can intervene to provide a recovery [^52]. An initial pose mismatch between the operator and robot may create a command jump when switching directly to absolute retargeting. We briefly pre-align the operator preview and anchor subsequent motion at the robot command at takeover time $t_{take}$. For the hand, only the post-takeover change in the operator's finger signal is added to the frozen robot command:

 $$ \begin{align*}\mathbf{m}_{j,t}^{\mathrm{exec}}=\mathrm{clip}\Big(\mathbf{m}_{j,t_{\mathrm{take}}}^{\mathrm{rob}}+\big[\mathcal{H}_{\mathrm{hand}}(\mathbf{b}_{j,t}^{\mathrm{finger}})-\mathcal{H}_{\mathrm{hand}}(\mathbf{b}_{j,t_{\mathrm{take}}}^{\mathrm{finger}})\big],\\ \mathbf{m}^{\mathrm{min}},\mathbf{m}^{\mathrm{max}}\Big),\end{align*} $$

where $\mathbf{m}_{j,t}^{exec}$ is the executed hand command, $\mathbf{m}_{j,t_{take}}^{rob}$ is the robot command frozen at takeover, and $\mathbf{m}_{\min}^{min}$, $\mathbf{m}_{max}^{max}$ are the motor limits. The difference between the two $\mathcal{H}_{\text{hand}}$ outputs captures only the operator's post-takeover change. At $t_{take}$, the correction is zero, making takeover continuous; afterward, only relative operator motion changes the command. The arm follows the same anchored relative-motion principle, and each recovery is recorded in the shared robot-native action space.

e) Controller-pivot calibration: To account for the offset from the controller tracking origin to the wrist rotation center, we estimate a fixed pivot from $N_{cal}$ calibration poses:

 $$ \begin{align*}(\boldsymbol{\rho}^{\star},\bar{\mathbf{x}}^{w})&=\arg\min_{\boldsymbol{\rho},\mathbf{x}}\sum_{\kappa=1}^{N_{\mathrm{cal}}}\|\mathbf{x}^{c}_{\kappa}+\mathbf{Q}^{c}_{\kappa}\boldsymbol{\rho}-\mathbf{x}\|^{2},\\\mathbf{x}^{w}_{t}&=\mathbf{x}^{c}_{t}+\mathbf{Q}^{c}_{t}\boldsymbol{\rho}^{\star},\end{align*} $$

where $\mathbf{x}_{\kappa}^c$ is the measured controller position, $\mathbf{Q}_\kappa^c \in \mathrm{SO}(3)$ is its orientation matrix, $\boldsymbol{\rho}^\star$ is the pivot in controller coordinates, and $\bar{\mathbf{x}}^w$ is the estimated stationary wrist center. This correction prevents pure wrist rotation from producing false translation.

f) Pose filtering and command safety: The corrected translation and measured orientation are filtered and rate-limited before retargeting and inverse kinematics:

 $$ \begin{align*}\mathbf{x}^{f}_{t}&=\alpha_{x}\mathbf{x}^{w}_{t}+(1-\alpha_{x})\mathbf{x}^{f}_{t-1},\\\mathbf{Q}^{f}_{t}&=Slerp(\mathbf{Q}^{f}_{t-1},\mathbf{Q}^{c}_{t},\alpha_{Q}),\\\|\Delta\mathbf{x}^{f}_{t}\|&\leq v_{\max}\Delta t,\qquad\angle(\Delta\mathbf{Q}^{f}_{t})\leq\omega_{\max}\Delta t,\end{align*} $$

where $\mathbf{x}_t^f$ and $\mathbf{Q}_t^f$ are the filtered position and orientation, $\Delta \mathbf{x}_t^f = \mathbf{x}_t^f - \mathbf{x}_{t-1}^f$ is the translation increment, and $\Delta \mathbf{Q}_t^f = (\mathbf{Q}_{t-1}^f)^\top \mathbf{Q}_t^f$ is the relative rotation. The coefficients $\alpha_x$ and $\alpha_Q$ control smoothing, and Slerp denotes spherical linear interpolation. The limits $v_{\max}$ and $\omega_{\max}$ bound translation and rotation rates over the control interval $\Delta t$. Dual-arm IK is initialized from the previous command and uses joint-rate constraints. Stale packets are rejected, short gaps are bridged by holding or interpolating commands, and prolonged tracking loss, excessive IK residuals, or collision detection freezes the last safe command or stops execution.

g) Pre-alignment and anchored takeover: Before takeover, the operator preview is compared with the frozen robot command at $t_{take}$. Control switches only after both arm-pose and hand-command discrepancies fall below fixed safety tolerances. Corrections follow the anchored Delta-Cmd rule above and are logged in the robot-native action space.

### B. Experimental Details

## 1 ) Baseline Models

a) $\pi_{0}$: This generalist VLA is initialized from an Internet-pretrained 3B vision–language model and pretrained on Open X-Embodiment together with cross-embodiment dexterous manipulation data collected across eight robot platforms [^22].

b) $\pi_{0.5}$: This successor to $\pi_{0}$ is designed for open-world generalization through heterogeneous co-training on multi-robot and multi-environment action data, multimodal Web tasks, high-level subtask labels, and verbal-instruction demonstrations [^23].

c) GR00T-N1.7: This open 3B cross-embodiment VLA combines a Cosmos-Reason2-2B backbone with a flow-matching diffusion-transformer action head. It is pretrained on diverse bimanual, semi-humanoid, and humanoid robot demonstrations together with 20,000 hours of EgoScale human video using a shared relative end-effector action representation [^55].

---

<div style="text-align: center;">TABLE IX</div>

<div style="text-align: center;">TASK-aligned Data Composition. Entries report total duration in hours; column headers give demonstration counts.</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'>Task</td><td style='text-align: center; word-wrap: break-word;'>Ego (1,000, h)</td><td style='text-align: center; word-wrap: break-word;'>Glove (1,000, h)</td><td style='text-align: center; word-wrap: break-word;'>Robot (110, h)</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Open-Box</td><td style='text-align: center; word-wrap: break-word;'>5.81</td><td style='text-align: center; word-wrap: break-word;'>5.11</td><td style='text-align: center; word-wrap: break-word;'>0.64</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Glue-Figure</td><td style='text-align: center; word-wrap: break-word;'>4.16</td><td style='text-align: center; word-wrap: break-word;'>4.37</td><td style='text-align: center; word-wrap: break-word;'>0.48</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Ice-Water</td><td style='text-align: center; word-wrap: break-word;'>3.09</td><td style='text-align: center; word-wrap: break-word;'>3.07</td><td style='text-align: center; word-wrap: break-word;'>0.51</td></tr></table>

<div style="text-align: center;">TABLE X</div>

<div style="text-align: center;">DATA-SOURCE ABLATIONS DURING HUMAN-ROBOT CO-TRAINING. DOWNSTREAM RESULTS ARE REPORTED ON TASK I: OPEN-BOX.</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'>Ego $^{TS}$</td><td style='text-align: center; word-wrap: break-word;'>Glove</td><td style='text-align: center; word-wrap: break-word;'>Robot</td><td style='text-align: center; word-wrap: break-word;'>Score</td><td style='text-align: center; word-wrap: break-word;'>Success Rate</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>✗</td><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>6.6</td><td style='text-align: center; word-wrap: break-word;'>30.0%</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>✗</td><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>6.5</td><td style='text-align: center; word-wrap: break-word;'>30.0%</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>✗</td><td style='text-align: center; word-wrap: break-word;'>5.8</td><td style='text-align: center; word-wrap: break-word;'>20.0%</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>7.5</td><td style='text-align: center; word-wrap: break-word;'>40.0%</td></tr></table>

![Image](images/1fab675bcb9076f01dc1391c8a8f671786132db31d2796ef8f0a699ffd24587c.jpg)

![Image](images/ec5dfb3bb8cf67c12c09e5bbfce52b3240a99a0a3ef9da7f72010fa06de682a4.jpg)

<div style="text-align: center;">Fig. 8. Human–Robot co-training optimization with a unified action representation. Each curve corresponds to the full Ego $^{TS}$+Glove+Robot mixture or one of the three data-source ablations, trained from the same base VLA without Human-to-Robot learning and with the same optimizer, batch size, and schedule. Left: training loss. Right: gradient norm.</div>

2) Data Composition and Physical Interaction: Table IX summarizes task-aligned data for each task. Every task uses 1,000 task-specific ego demonstrations, 1,000 glove demonstrations, and 110 real-robot trajectories: 100 expert demonstrations and 10 DAgger correction trajectories.

3) Human–Robot Co-Training Ablations: We further ablate the three data sources used during Human–Robot co-training without Human-to-Robot learning. Each variant starts from the same base VLA, follows the same optimization schedule, and retains the same robot-domain refinement; only one Stage 2 source is removed at a time.

Removing any Stage 2 source degrades downstream performance, with the removal of robot data producing the largest drop to 5.8/20.0%. Removing task-specific ego data or glove demonstrations yields comparable success of 30.0%, but lower scores of 6.6 and 6.5, respectively. Fig. 8 shows that directly co-training on task-specific ego and robot data without glove demonstrations leads to unstable training, whereas adding glove demonstrations stabilizes optimization. This suggests that glove demonstrations, which combine human manipulation with robot-view observations, help bridge the gap between ego-human and robot data during co-training.

4) Action-Distribution Alignment Metrics: We use the robot-native action $A_{t}$ defined in Eq. (6) and divide its coordinates into four groups: left arm, left hand, right arm, and right hand. Before comparing ego-human and robot actions, we standardize the complete action vector using statistics pooled from both domains:

 $$ \widehat{A}_{t}=\left(A_{t}-\mu\right)\oslash\boldsymbol{\sigma}, $$

where $\mu$ and $\sigma$ contain the pooled mean and standard deviation of each action coordinate, and $\varnothing$ denotes elementwise division.

For Fig. 5(a), we fit PCA jointly to the standardized ego-human and robot actions and project both domains onto the same first two principal components. We divide this shared two-dimensional plane into a $40 \times 40$ grid. Let $B_{ego}$ and $B_{robot}$ be the sets of grid cells occupied by samples from the two domains. Their support overlap is

 $$ \mathrm{I o U}_{\mathrm{P C A}}=\frac{\left|\mathcal{B}_{\mathrm{e g o}}\cap\mathcal{B}_{\mathrm{r o b o t}}\right|}{\left|\mathcal{B}_{\mathrm{e g o}}\cup\mathcal{B}_{\mathrm{r o b o t}}\right|}, $$

where  $|\cdot|$ denotes the number of cells in a set.

For Fig. 5(b), we take the Euclidean magnitude of each standardized arm or hand subvector: the square root of the sum of its squared coordinates. This produces one nonnegative value measuring how far that group's command is from the pooled mean. We estimate this magnitude distribution separately for each group. For any one of the four groups, let $p_{\text{ego}}(x)$ and $p_{\text{robot}}(x)$ be the normalized kernel density estimates for ego-human and robot actions at point x on a shared evaluation grid. Their density overlap is

 $$ OVL=\sum_{x}\min(p_{ego}(x),p_{robot}(x)), $$

where the sum runs over the grid and each density sums to one. At every grid point, the smaller density is the probability mass shared by both domains; summing this lower envelope gives OVL = 0 for disjoint distributions and OVL = 1 for identical distributions. We apply the same calculation separately to the left arm, left hand, right arm, and right hand. All reported alignment values use 10,000 randomly sampled training frames from each domain.

5) Action-Based Ego-Robot Episode and Frame Pairing: For the image-similarity evaluation in Table III, we pair ego and robot observations by how their actions evolve. From each pair of consecutive actions $A_{t-1}$ and $A_t$, we first compute the change in every command dimension. We then summarize the changes within each arm and hand subvector by their root mean square: square the changes, average them within the subvector, and take the square root. Each channel is smoothed with a nine-frame moving average and divided by its temporal 90th percentile within the episode, so a normalized value of 1 represents that channel's 90th-percentile motion level. We use a fixed cap of 3, corresponding to three times the 90th-percentile level, so isolated extreme changes do not dominate the subsequent matching cost. The four normalized channels form the motion-profile vector $\mathbf{v}_t \in \mathbb{R}^4$.

a) Episode-level pairing: We randomly sample equal numbers of ego and robot episodes. For a candidate pair, Dynamic Time Warping (DTW) [^61] aligns their sequences of four-channel motion-profile vectors, using the Euclidean

---

distance between two vectors as the local cost. DTW can therefore align similar arm-hand motion patterns performed at different speeds. The mean of these local distances along the optimal DTW path is denoted by $\bar{c}_{DTW}$. Let $F_{ego}$ and $F_{robot}$ denote the episode lengths in frames. The pair cost is

 $$ C_{\mathrm{p a i r}}=\bar{c}_{\mathrm{D T W}}+\lambda_{\mathrm{d u r}}\left|\log\frac{F_{\mathrm{e g o}}}{F_{\mathrm{r o b o t}}}\right|. $$

The first term measures motion-pattern mismatch, while $\lambda_{dur}$ controls the penalty for episodes with different durations. We use Hungarian assignment [^62] on these costs to obtain the minimum-cost one-to-one pairing.

b) Frame-level pairing: Within each paired episode, we sample every 15 frames, reducing the 30-Hz sequences to approximately 2 Hz, and run DTW again. Let i and j denote sampled ego and robot frame indices, and let $u_i^{\text{ego}}$ and $u_j^{\text{robot}}$ denote their normalized progress from zero at the start of an episode to one at the end. Their local matching cost is

 $$ c(i,j)=\|\mathbf{v}_{i}^{\mathrm{e g o}}-\mathbf{v}_{j}^{\mathrm{r o b o t}}\|_{2}+0.2|u_{i}^{\mathrm{e g o}}-u_{j}^{\mathrm{r o b o t}}|, $$

where  $\mathbf{v}_i^{\text{ego}}$ and  $\mathbf{v}_j^{\text{robot}}$ are the four-channel motion profiles defined above, and the second term penalizes differences in episode progress. We allow a candidate match only when  $|u_i^{\text{ego}} - u_j^{\text{robot}}| \leq 0.15$. DTW backtracking then gives a monotone, possibly many-to-one frame correspondence. Raw ego, Project+Inpaint, and GEOFORMER use exactly the same correspondences and are therefore evaluated on identical frame pairs.

For each method and metric, we report the unweighted mean of episode-pair median frame scores.

### C. Details of EgoWild

## 1 ) Data Quality Assessment

a) Action-annotation accuracy: We manually audit Qwen3.8-Flash [^63] annotations on 50 videos (2,926 s; 906 human-labeled actions). Of 989 predicted segments, 97.7% have exact action names and 98.6% have acceptable names (exact or semantically similar).

b) Temporal-segmentation quality: We inspect five frames within $\pm0.6$ s of each candidate cut. A cut is an over-segmentation error only if the hand and object remain in continuous motion, without completion, release, or pause. The confirmed over-segmentation rate is 1.6%; the combined segmentation-error rate is 1.7%.

c) Hand-tracking accuracy: Using the OpenXR middle-fingertip joint on a PICO 4 Ultra, we test unconstrained three-dimensional reaches: a 37 cm lift followed by a 70 cm planar motion. Median absolute fingertip-position error is 4.21 cm without an auxiliary hand tracker versus 0.66 cm with it. Collectors therefore wear the tracker during recording.

## 2 ) Dataset Curation and Diversity

a) Episode annotations: Each episode has one structured JSON annotation with 10 fields specifying its identifier, start and end times, atomic action, bilingual descriptions, interacting hand, target object class, and noise and cleaning flags for later checks.

![Image](images/967979b8cc4f95b6b0f9f16926b14a798164441648c164b047226eb089b1a78d.jpg)

<div style="text-align: center;">(a) Objects</div>

<div style="text-align: center;">(b) Skills</div>

<div style="text-align: center;">Fig. 9. Semantic word clouds for EgoWild. (a) Objects and (b) skills; font size indicates frequency.</div>

b) Auxiliary metadata and quality filtering: A recording-level filter uses synchronized body, hand, and camera metadata to find non-overlapping clips of at least 5 s with either hand visible in at least 90% of frames. We remove locomotion intervals lasting at least 1.0 s with at least 1.0 m of horizontal displacement (using head motion as fallback), their 1.0 s margins, and quality-control intervals with image blur, malformed hand skeletons, or similar issues. The valid ratio is the remaining duration divided by the full recording duration. Recordings with a valid ratio below 0.5 are labeled low quality and discarded; all others are labeled normal and retained in full for the next step.

For the second, clip-level filter, a VLM divides each retained recording into timestamped semantic action segments with descriptions, hand roles, and target objects. We retain segments that are bimanual and non-noise, exclude Idle, Inspect, and manually rejected intervals, and require each hand to be active in at least 90% of frames and both hands simultaneously active in at least 85%.

Finally, video, pose, and actions are nearest-neighbor aligned on a shared 30-Hz timeline. A frame is invalid if its pose or video timestamp error exceeds 20 ms or 25 ms, respectively. We split each VLM-labeled segment at invalid frames and keep continuous valid parts of at least 2 s as final episodes. Each episode keeps the original VLM description as its language instruction.

c) Qualitative coverage: To complement the category-level distributions, we visualize semantic coverage using the two word clouds in Fig. 9. The object cloud in panel (a) spans apparel, household items, and packaging. The skill cloud in panel (b) contains common actions such as pick, fold, place, hold, unfold, and rinse, indicating coverage beyond pick-and-place.

d) Category-level distribution: Figs. 10 and 11 show within-category frequencies for 1,282 object classes and 273 skill lemmas. Common objects and skills have many annotations, while less frequent entries form a long tail within most categories.

### D. Contributions

The authors contributed to this work as follows.

Project Lead: Kunyang Lin.

Core Contributors: Kunyang Lin, Xutao Wen, Jingxi Lin, Lanyong Lin, and Jiaming Liu.

Learning Algorithm: Kunyang Lin.

GeoFormer: Xutao Wen and Kunyang Lin.

EgoWild Dataset: Jiaming Liu, Kunyang Lin, and Yiduo Li.

---

Teleoperation and Data Collection System: Jingxi Lin, Lanyong Lin, and Kunyang Lin.

Training and Deployment: Kunyang Lin, Xutao Wen, Jingxi Lin, and Lanyong Lin.

Baseline Construction: Xutao Wen, Lanyong Lin, Kunyang Lin, and Xianchi Chen.

Writing and Illustration: Kunyang Lin, Xutao Wen, Tianshuo Yang, and Yue Han.

Discussion and Feedback: Ping Luo, Zhanpeng Zhang, and Tianshuo Yang.

Resources and Academic Supervision: Ping Luo.

---

![Image](images/9c2f4d81b5fc94fbac7a913eceb2963892aa9010cb71d262f4c5844fed7c4c85.jpg)

![Image](images/bee14fb5edbba1ee3bcd6f23b0a2e23345320a0fa6643ef1721f6a2985fc3c37.jpg)

![Image](images/c8aed38b794f6b496780de3196599ec3e5c4717a55bb59fcf38c0e3e4937a04d.jpg)

![Image](images/625f74be46998986909eedd0d71704865128d969915130d7f9c52e9c8158fce6.jpg)

![Image](images/3ab3f242f3deffc160caa5f34c207fe0294c25759726aad9cd92dfd2aaba0d8e.jpg)

![Image](images/0fbef65cf60f09b65a9b87a0f2a8b5d2d009d820592ac8416f2d6d5e97e9e367.jpg)

![Image](images/b3867907a69fbc18e8ad0db705d1df7659810c6fd378a9f390e65821a13a99b3.jpg)

![Image](images/f1516d939581423d9ddfcc1f41f02054474b4690aaa329931030da92c91cc4ae.jpg)

<div style="text-align: center;">Body, Living Things, Personal Care, Medical & Pet Supplies</div>

![Image](images/306204a7bfc5a6406c606205b2765af5b4e437aa0aed8d20633a6dcd2beddd27.jpg)

![Image](images/f1a31aaeb8aae67a302520249a57bf3f0658dca1471e55b0b47822dff1f806d3.jpg)

![Image](images/49485a1e2cefc1a67ebe57c0c8a76b90b369b875bb8b8809c04c619ca1b2ef75.jpg)

<div style="text-align: center;">Fig. 10. Object frequency distributions within broad semantic categories. The 1,282 fine-grained object classes are grouped into broad categories, sorted by frequency within each category, and displayed on a logarithmic scale.</div>

---

<div style="text-align: center;">Folding, Shaping & Textile Manipulation</div>

![Image](images/05fbcad098de9a77f1fb7f7c1b2ae85cdaa941ac374e73a00cbe06c479547b94.jpg)

<div style="text-align: center;">Opening, Closing, Connecting, Fastening, Wearing & Wrapping</div>

![Image](images/14dc0aa7c7b4b122022e95199ce67e48b5fbd3c2308c49ab497bbff47b959e94.jpg)

<div style="text-align: center;">Picking Up, Holding & Carrying</div>

![Image](images/fa8949159893204d259dcd5b3cc441fde6b27c974d044bfa40276c7d61cbf66b.jpg)

<div style="text-align: center;">Washing, Cleaning & Surface Care</div>

![Image](images/25d533602cbd98b8b00b5c43d5909046fbf65417d7a3162d2ba2cdf03816bcec.jpg)

<div style="text-align: center;">Moving, Placing & Body Movement</div>

![Image](images/6fbe68f45661223c0fc422044cf6056acd35dd875fd2b261c31e716e29002065.jpg)

<div style="text-align: center;">Cutting, Peeling, Separating & Removing</div>

![Image](images/0426e050b911de892b236585b3d50a086afc5e58ef42edc7cc64eb712ca4cad6.jpg)

<div style="text-align: center;">Arranging, Organizing, Sorting & Gathering</div>

![Image](images/2a0871a7e00611ebcd64ffb9cf40b7c8be65dcbe99367a30816dfebf8daa2636.jpg)

<div style="text-align: center;">Cooking. Mixing & Food Preparation</div>

![Image](images/4f6b679a499eebc8b2e486d9a3a928c47116ac6724beb2b4f45892f5daec54d5.jpg)

<div style="text-align: center;">Operating Devices & Tools, and Adjusting Controls</div>

![Image](images/341af1e9bd2f122187a67915052b0ac68deb8aab5f36db41f35461bbc14fddcd.jpg)

<div style="text-align: center;">Searching, Observing, Inspecting & Selecting</div>

![Image](images/42db6e5730c22f157ccc8dcd540dd326428f1c26e162c0eff04eaa980037f1f8.jpg)

<div style="text-align: center;">Task Execution. Preparation & Completion</div>

![Image](images/042b0b128316b1ddae309c2581900a307f49f8d85250c31fd30b54163d357372.jpg)

<div style="text-align: center;">Fig. 11. Skill frequency distributions within broad semantic categories. The 273 action-verb lemmas are grouped into broad skill categories, sorted by frequency within each category, and displayed on a logarithmic scale.</div>