# METIS: Multi-Source Egocentric Training for Integrated Dexterous Vision-Language-Action Model Yankai Fu $^{1,2,*}$, Ning Chen $^{1,2,*}$, Junkai Zhao $^{2,*†}$, Shaozhe Shan $^{1}$, Guocai Yao $^{2}$, Pengwei Wang $^{2}$, Zhongyuan Wang $^{2}$, Shanghang Zhang $^{1,2✉}$ $^{1}$State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University; $^{2}$Beijing Academy of Artificial Intelligence $^{*}$Equal contribution, $\dagger$Project leader, Corresponding author Project Webpage:
(a) EgoAtlas Dataset for Pretraining
![Image](images/467b9589d889c22c4e5b44a23c69e42502a85e2a8bd2e26e7cd4675a51a21ff9.jpg) ![Image](images/87ef933c6307880719d72ffa8284cada1c4c06532a14e8ec7f2af20f786fa5ed.jpg)
(b) METIS VLA Model
Figure 1: METIS is trained on a multi-source egocentric manipulation dataset EgoAtlas. It leverages motion-aware dynamics to extract manipulation-relevant dexterous features, and integrates reasoning and acting within a unified framework. METIS achieves strong performance across diverse dexterous manipulation tasks and exhibits remarkable generalization capability.
## Abstract Building a generalist robot that can perceive, reason, and act across diverse tasks remains an open challenge, especially for dexterous manipulation. A major bottleneck lies in the scarcity of large-scale, action-annotated data for dexterous skills, as teleoperation is difficult and costly. Human data, with its vast scale and diverse manipulation behaviors, provides rich priors for learning robotic actions. While prior works have explored leveraging human demonstrations, they are often constrained by limited scenarios and a large visual gap between human and robots. To eliminate these limitations, we propose METIS, a vision-language-action (VLA) model for dexterous manipulation pretrained on multi-source egocentric datasets. We first construct EgoAtlas, which integrates large-scale human and robotic data from multiple sources, all unified under a consistent action space. We further extract motion-aware dynamics, a compact and discretized motion representation, which provides efficient and expressive supervision for VLA training. Built upon them, METIS integrates reasoning and acting into a unified framework, enabling effective deployment to downstream dexterous manipulation tasks. Our method demonstrates exceptional dexterous manipulation capabilities, achieving highest average success rate in eight real-world tasks. Experimental results also highlight the superior generalization and robustness to out-of-distribution scenarios. These findings emphasize METIS as a promising step toward a generalist model for dexterous manipulation. --- ## 1 Introduction Recent advances in vision-language-action (VLA) models have achieved remarkable progress toward general-purpose embodied intelligence [^45] [^19] [^15] [^68]. However, such models heavily rely on scarce and expensive real-world robot data, often collected via human teleoperation. Despite extensive efforts to build large-scale robotic datasets [^34] [^17] [^5] [^54], the effective scale of robot interaction data remains orders of magnitude lower than those used for LLM training [^68], severely limiting the scalability and generalization of current VLAs. This issue is even more pronounced in dexterous manipulation, where acquiring high-quality demonstrations is extremely costly and complex. As a result, most existing VLA research focuses on simple gripper-based tasks [^14] [^26] [^6] [^30], leaving dexterous manipulation largely unexplored. In contrast to the scarcity of robotic datasets, human data are vastly abundant and rich in semantic information, providing valuable behavioral priors. Previous works have explored learning task-relevant representations from human videos, such as affordance [^1] [^21], latent action [^60] [^6], and keypoint flow [^65] [^62]. Recent progress such as EgoVLA [^59], Being-H0 [^32], and H-RDT [^2] leverage large-scale human datasets to pretrain VLA, providing a new paradigm for scaling robotic learning. However, human videos are often confined to specific household or workspace scenes (e.g., tabletop or kitchen), exhibiting uneven scene coverage and strong contextual biases. Moreover, there exists a substantial gap between human data and robot data, in both visual appearance and action space. Another line of research explores co-training strategies that jointly utilize self-collected human and robotic data. For example, HAT [^39] learns a human policy and demonstrates that incorporating human data can significantly enhance the generalization and robustness of robotic policies, while MotionTrans [^63] aligns human motions to robot-specific embodiments to achieve zero-shot skill transfer. These approaches collect manipulation-related human data but do not fully exploit the vast amount of human data available on the internet. In this work, we investigate these problems by introducing METIS, a vision-language-action (VLA) model for dexterous manipulation pretrained on multi-source egocentric datasets. We first construct EgoAtlas, a multi-source egocentric dataset that covers large-scale internet human data, robot data, and enhanced human data collected through a wearable system. EgoAtlas spans four major categories and eight sources, all aligned under a unified action space. We further propose motion-aware dynamics, a compact and discretized representation designed for dexterous manipulation. It captures both visual and motion dynamics, providing efficient and expressive supervision for training VLA. Built upon them, METIS is pretrained on EgoAtlas, unifying reasoning and acting within a single framework. This design enables efficient fine-tuning and deployment on downstream dexterous tasks. To comprehensively evaluate METIS, we conduct extensive real-world experiments on dexterous tasks. The comparative results show that our model achieves superior performance and efficiency, attaining the highest average success rate among all baseline VLAs. Besides, METIS also exhibits strong generalization to out-of-distribution scenarios, including unseen background, unseen object, unseen lighting condition, cluttered environment, and can be transferred to higher-DoF embodiments. Finally, ablation studies verify the contributions of both the multi-source egocentric dataset and the motion-aware dynamics to the overall system, highlighting the potential of learning robotic motion priors from human data. In summary, our contributions are as follows: • We construct a multi-source egocentric manipulation dataset EgoAtlas, which integrates diverse human and robotic data sources under a unified state-action space. - We present METIS, a VLA model for dexterous manipulation pretrained on EgoAtlas, which extracts motion-aware dynamics from dexterous hand motion and integrates reasoning and acting within a unified framework. - We demonstrate the effectiveness and generalization of our method through a range of real-world experiments. ## 2 Related Work ### 2.1 Dexterous Manipulation Dexterous manipulation is a key challenge in robotics, focusing on achieving human-like fine-grained manipulation skills. Traditional approaches are based on optimization and control algorithms. --- [^22]: [^51] [^27] [^56], typically assuming access to known dynamics and object models, and focusing on trajectory planning under physical constraints. While achieving strong performance in specific scenarios, these methods often struggle to generalize across diverse real-world settings. Recent learning-based approaches, including reinforcement learning (RL) and imitation learning (IL), have made remarkable progress across various tasks such as grasping [^55] [^48], in-hand manipulation [^37] [^36], and tool use [^41] [^61]. RL methods learn precise and dexterous skills through large-scale training in simulation, but often suffer from a significant sim-to-real gap. In contrast, IL methods leverage expert demonstrations to achieve robust performance in real-world dexterous manipulation tasks [^28] [^10] [^18] [^9], but typically rely on expensive human teleoperation data. In this work, we address this data bottleneck by pretraining our model on large-scale multi-source egocentric data that combines human and robotic demonstration to learn motion priors efficiently. ### 2.2 Learning Dexterity from Human Data Human data offer valuable priors for dexterous manipulation, featuring abundant samples, fine-grained hand motion, and rich semantic cues. There has been works on learning task-relevant representations from human videos, such as affordance [^1] [^21], latent action [^60] [^6], and keypoint flow [^65] [^62]. Ego-Only [^50] utilizes the MAE framework [^11] to extract actionable information from egocentric videos, while LAPA [^60] and UniVLA [^6] apply VQ-VAE [^47] to extract latent action from large-scale unlabeled human data. However, learning directly from human videos introduces redundant, manipulation-irrelevant content and underemphasizes the critical role of hand motion. To mitigate this gap, HAT [^39] and MotionTrans [^63] adopt a co-training strategy on both human and robotic data with explicit motion information, improving the robustness of the policy and enabling zero-shot skill transfer. In this work, we learn action priors from human data through joint modeling of visual and motion dynamics to support VLA learning. ### 2.3 Vision-Language-Action Model Vision-Language-Action (VLA) models have achieved unprecedented progress in recent years [^45] [^19] [^15] [^40], driven by advances in Vision-Language Models (VLMs) [^42] [^44] [^57] and the availability of large-scale robotic datasets [^34] [^17] [^5] [^54] [^13]. By processing multimodal inputs—such as visual observations and language instructions—these models enable robots to autonomously perform a wide range of tasks. Representative VLA models, such as RT-2 [^68] and OpenVLA [^19] generate action sequences autoregressively. In contrast, $\pi_0$ [^4] and DexVLA [^53] employ a diffusion-based policy, fitting continuous action distributions via iterative denoising. However, these approaches primarily focus on gripper-based manipulation, overlooking the rich motion and interaction dynamics inherent in dexterous tasks. Recently, several studies have extended VLA frameworks to dexterous manipulation. For example, GR00T N1 [^3] learns latent representations from hybrid data to train a humanoid manipulation policy, while EgoVLA [^59] and Being-H0 [^32] leverage large-scale human demonstrations for VLA pretraining, enabling the acquisition of motion priors. Despite their impressive results, the scene homogeneity of human videos leads to bias and a large visual gap from robotic observations. To overcome this issue, we leverage multi-source egocentric data, introduce an enhanced human dataset, and develop an integrated VLA model for unified reasoning and acting. ## 3 EgoAtlas Dataset ### 3.1 Wearable System for Enhanced Human Data Traditional hand motion capture systems rely on multi-camera or VR setups, which are limited by viewpoint dependency, occlusion, and constrained capture space. To overcome these issues, we develop a wearable glove-tracker system for portable, high-fidelity motion capture. Wearable Hand Motion Collection System. We use Manus Quantum Metagloves to record precise 3D hand keypoints (25 per hand). A VIVE Tracker mounted on each glove provides the 6-DoF wrist pose for global localization. To capture egocentric visual observations, a head-mounted camera is used to record the first-person view during manipulation, providing visual information closely aligned with the operator's perspective, while another VIVE Tracker is attached to the headset to perform extrinsic calibration. This calibration aligns the --- motion-capture and ego-camera coordinate frames, enabling the entire system to operate in the wild and ensuring scene diversity. We also provide a top-down third-person camera view which is optional and is not used for calibration, training, or inference in this work; it records RGB videos only for future research purposes. The entire system operates at 20 Hz, balancing motion fidelity with reliable multi-sensor synchronization. Subtask-Level Annotation. We performed subtask-level annotation to enrich the semantic structure of the dataset. Each trajectory is paired with a task-level language instruction, along with a fine-grained segmentation into multiple subtasks. This annotation design ![Image](images/6efa89d821c9a8015690f54a5340007001f67682a6d4b5cf0fdb569d948717b0.jpg)
Figure 2: Wearable Hand Motion Collection System.
enables the VLA to focus on hand motion and supports hierarchical reasoning over long-horizon manipulation tasks. More details can be found in Appendix A.5. ### 3.2 Data Statistics and Processing Built on Section 3.1, we introduce EgoAtlas, a large-scale multi-source egocentric dataset for VLA training. EgoAtlas integrates data from 8 heterogeneous sources spanning human and robot domains, including 10K self-collected high-fidelity trajectories using wearable system. The final dataset contains 343K trajectories and 89.72M image-action pairs, as summarized in Table 8. We apply weighted sampling to balance source distributions. More details are provided in Appendix A.7. Different embodiments operate in distinct state and action space. To bridge these heterogeneous sources, we construct a unified state–action representation that abstracts away embodiment-specific differences while preserving manipulation-relevant motion. The state–action space is composed of two components: wrist pose and fingertip pose. The wrist pose is represented as an 18-dimensional vector consisting of a 3D position and a 6D continuous rotation per hand in the egocentric camera frame. The hand configuration is represented by a 3D-dimensional fingertip pose, defined by the 3D positions of individual fingertips in the wrist coordinate frame. We apply robot-specific FK/IK (Forward Kinematics and Inverse Kinematics) and human motion retargeting algorithms to project heterogeneous motion representations into a shared state–action space, while aligning all data within a consistent coordinate system. More details can be found in Appendix A.3, A.4 and A.6. ## 4 Method ### 4.1 Problem Formulation Our goal is to learn a VLA model for dexterous manipulation from multi-source egocentric data. Formally, given a collection of human and robot trajectories $D = D_{robot} \cup D_{human}$ from the EgoAtlas dataset, each trajectory can be represented as a sequence of observation–action pairs $\tau = \{(o_t, a_t)\}_{t=1}^T$, where observation $o_t = \{I_t, S_t\}$ consists of the egocentric image input and the proprioceptive state. The training objective is to learn a policy $\pi_{\phi}(a_t | o_t, l)$ that predicts actions conditioned on egocentric observations $o_t$ and language instruction l. To bridge the embodiment gap, we construct a unified state-action space which includes an 18D wrist pose $P_{t}^{w}$ (3D position and 6D rotation vector per hand) and a 30D finger pose $P_{t}^{f}$ capturing the 3D position of individual fingertips. Here we describe the design of the three components in detail. - Image Observation $I_{t}$: We use egocentric image from a first-person camera, which provides a viewpoint that focuses on fine-grained interaction details. • Wrist Pose $P_{t}^{w}$: We represent the wrist pose as the 3D position and the 6D rotation vector following Zhou et al. [^67]. They are relative to the camera coordinate frame. - Finger Pose $P_{t}^{f}$: We represent the finger pose using 3D fingertip positions, defined in the wrist coordinate frame. --- ![Image](images/78f81495cfb3fcac854e113416183014f3b4c76f017536c20955ce26c4ae8286.jpg)
Figure 3: Overview Framework (a) We construct an expressive yet compact representation to capture the dynamics in dexterous manipulation. (b) METIS is pretrained on multi-source EgoAtlas dataset, where human and robot actions are align under a unified action space. (c) METIS integrates reasoning and acting within a framework, enabling effective deployment to downstream tasks.
### 4.2 Motion-Aware Dynamics Construction Recent VLA models typically employ generic tokenizers to discretize continuous actions, enabling autoregressive sequence modeling for policy learning. However, as the number of action chunks and system degrees of freedom increase, discrete action representations inevitably lead to longer token sequences, which in turn increases the computational cost and latency of autoregressive generation. Additionally, this tokenization approach often struggles to capture fine-grained motion details, which is particularly critical in dexterous manipulation, where subtle finger movements and contact interactions are essential. To address these challenges, we propose a compact and discretized representation that effectively constructs motion-aware dynamics for dexterous manipulation. This representation provides an efficient and expressive supervision signal for VLA pretraining. The proposed dynamics model consists of two key components, detailed as follows. Visual Dynamics Discretization. A key challenge in egocentric dexterous manipulation is modeling how hand motion induces structured visual change. Instead of discretizing raw visual observations, we explicitly model motion-conditioned visual dynamics that capture the causal effect of manipulation on future visual states. Specifically, we employ an Inverse Dynamics Model based encoder $\mathcal{I}(D_{vis}|I_t,I_{t+k},P_{t,t+1,...,t+k})$ and a Forward Dynamics Model based decoder $\mathcal{F}(I_{t+k}|I_t,D_{vis})$. The encoder captures motion-relevant visual dynamics by integrating visual observations with continuous motion information, while the decoder is trained to predict future observation given the constructed visual dynamics. Our encoder includes both spatial and temporal transformer while the decoder only contains spatial transformer. The visual dynamics $D_{vis}$ are quantized with the VQ-VAE objective [^47]. It maps continuous latent features into a finite set of discrete codebook embeddings, thereby enabling efficient and stable auto-regressive training for VLA. Following Bu et al. [^6], we do not reconstruct image pixels, as raw pixels contain substantial redundancy and details irrelevant to the manipulation task. Instead, we use pretrained DINOv2 [^33] to extract high-level semantic representations, that better reflect task-relevant visual dynamics. Motion Dynamics Quantization. Learning motion priors is essential for VLA models. To this end, we introduce a separate set of discretized codebook embeddings $D_{mot}$ to focus on motion capturing. We discretize hand motion trajectories into compact dynamics tokens that effectively capture proprioceptive information and reflect the fine-grained features of dexterous manipulation. We propose PoseNet as the encoder for 3D hand motion, combining multi-scale temporal convolutions --- with trajectory self-attention to capture spatio-temporal dynamics. The continuous motion features are then quantized using RQ-VAE [^25], which not only effectively prevents codebook collapse but also captures hierarchical motion patterns, from coarse to fine levels. A temporal convolutional network (TCN) [^24] is used as the decoder to reconstruct motion trajectories during training, as illustrated in Figure 3(a). More details about motion-aware dynamics can be found in Appendix B. ### 4.3 METIS Model As shown in Figure 3(c), our proposed METIS model is built upon a VLM, where the parameters are initialized from the Prismatic-7B [^16]. It incorporates a hybrid vision encoder that integrates SigLIP [^64] and DINOv2 [^33], capturing both global semantics and fine-grained spatial details. The resulting visual features $f^{SigLIP} \in \mathbb{R}^{N_v \times 1152}$ and $f^{DINO} \in \mathbb{R}^{N_v \times 1024}$, where $N_v$ represents the visual token dimension, are concatenated along the channel dimension. A projection layer is applied to align visual embeddings with the language modality. The 7B LLaMA-2 [^46] large language model is adopted as the LLM backbone, which employs a decoder-only Transformer architecture with 32 sequential blocks for auto-regressive language modeling. Previous work such as OpenVLA [^19] and RT-2 [^68] map the infrequently used words in LLaMA-2 to action bins uniformly distributed within $[-1, 1]$. While effective for low-dimensional control, they struggle to scale to high-dimensional, continuous actions, often resulting in inefficient and unstable behavior. METIS first extends the LLaMA tokenizer vocabulary with $|C_1| + |C_2|$ special tokens, where $|C_1|$ and $|C_2|$ correspond to the codebook sizes of the visual dynamics tokens and motion dynamics tokens, respectively. Using the dynamics model introduced in the previous section, we discretize each egocentric manipulation sequence into motion-aware action tokens, denoted as $D_{vis}$ and $D_{mot}$. Each dynamics feature is assigned to its nearest codebook entry, and the resulting discrete index is mapped to a unique special token. This design fully leverages the original VLM architecture, enabling autoregressive supervision for training. By preserving the language priors of the language model while injecting motion information, METIS learns to model the fine-grained dynamics essential for dexterous manipulation. The auto-regressive objective of METIS $\pi$ is to minimize the sum of next-dynamics negative log-probabilities: $$ \mathcal{L}_{a r}=\mathbb{E}_{o_{t},l,a_{d,