# Being-H0：基于大规模人类视频的视觉-语言-动作预训练

罗浩 $^{1,3,*}$ 冯一程 $^{1,3,*}$ 张万鹏 $^{1,3,*}$ 郑思鹏 $^{3,*}$ 王烨 $^{2,3}$ 袁浩奇 $^{1,3}$ 刘佳政 $^{1}$ 徐超逸 $^{3}$ 金琴 $^{2}$ 陆宗庆 $^{1,3,\dagger}$

$^{1}$北京大学 $^{2}$中国人民大学 $^{3}$BeingBeyond

<https://beingbeyond.github.io/Being-HO>

![Image](images/f0f45fcdca21f5159b75cbc22db75e6a6295aa39a86c0c53c1e49285851b711f.jpg)

<div style="text-align: center;">Figure 1: Being-HO acquires dexterous manipulation skills by learning from large-scale human videos in the UniHand dataset via physical instruction tuning. By explicitly modeling hand motion patterns, the resulting foundation model seamlessly transfers from human hand demonstrations to robotic manipulation.</div>

---

## 摘要

我们提出了\textbf{Being-HO}，一种在大规模人类视频上训练的灵巧视觉-语言-动作模型（VLA）。现有的VLA在处理需要高灵巧性的复杂操控任务时表现不佳，且在新场景和任务中的泛化能力较差，这主要是因为它们依赖存在显著仿真到现实差距的合成数据，或是缺乏规模与多样性的遥操作演示数据。为解决这一数据瓶颈，我们提出利用人类手部作为“基础操控器”，充分利用网络数据中蕴含的丰富灵巧性与可扩展性。我们的方法以物理指令微调为核心，这是一种全新的训练范式，结合了基于人类视频的大规模VLA预训练、用于三维推理的物理空间对齐，以及针对机器人任务的训练后适配。此外，我们提出了一种部件级运动分词方法，能够实现毫米级的重建精度，从而建模精确的手部轨迹以用于动作学习。为支持所提出的范式，我们进一步开发了一套完善的数据筛选流程，将动作捕捉、虚拟现实以及仅含RGB的视频等异构数据源整合为一个包含数百万条基于运动的指令实例的大规模数据集。我们通过实验验证了\textbf{Being-HO}在手部运动生成和指令跟随方面的卓越性能，且该模型在规模和数据量增加时仍能保持良好表现。重要的是，我们观察到当应用物理指令微调后，\textbf{Being-HO}在真实世界机器人操控任务中确实取得了预期的成效。

日期：2025年7月21日

## 1 引言

ChatGPT及其后续产品的出现彻底革新了人工智能研究领域，赋予了大型多模态模型（LMMs）强大的推理能力，使其在智能体系统、数学及编程等领域表现出色。然而，机器人领域仍未迎来属于它的“ChatGPT时刻”——即具身智能领域的突破性飞跃[^1] [^2]。近期的研究尝试通过将LMMs改造为视觉-语言-动作模型（VLAs）来填补这一缺口[^3] [^4] [^5] [^6] [^7]，利用它们的多模态感知能力和类人推理能力来完成机器人任务。

不过，这些方法仍受限于对远程操控演示数据的依赖，从而限制了其泛化能力和稳定性，尤其是在全新环境中。核心瓶颈在于：远程操控数据集[8–10]的规模比互联网规模的LMM训练数据小几个数量级，这使得具身智能[^1] [^2]深陷持久的“数据沼泽”之中。基于这类实验室采集的数据训练的模型[^5] [^11]，在应对各种物体和环境时的操控能力往往不尽如人意。对于灵巧手而言，这种数据稀缺问题尤为严重——因其操作复杂且硬件成本高昂，导致大多数VLA只能采用缺乏真正灵巧性的简单夹爪[12–15]。由于自由度有限且无法精细控制手指，这些末端执行器根本无法实现复杂交互所需的精准协调或细腻力控。尽管一些研究[16–20]试图利用模拟器来获取低成本的合成数据，但其多样性不足以及尚未解决的仿真到现实差距问题，依然阻碍着灵巧手在现实世界中的应用。

人类视频为VLA训练提供了另一种极具潜力的替代方案，它能提供丰富的真实世界数据，且与现实世界的差异极小。虽然以往的研究已采用多种隐式学习方法（如对比学习[^21]、掩码自编码器[^22]、潜动作表示[^23]）来提升机器人技能，但这些方法背后的学习机制及其迁移效果仍不清晰。值得注意的是，这些隐式方法并未能像LLM/LMM那样带来显著的性能提升；后者通过视觉指令微调[^24]等技术取得了令人瞩目的成果。我们认为，这种差异源于数据结构的根本性不同：在LLM和LMM中，预训练数据与下游微调数据具有同构性——文本推理与语言任务无缝衔接，视觉-文本理解也能自然迁移到多模态任务中。

---

相比之下，在VLA中这种对齐关系却变得异质化：文本/二维视觉输入与具备本体感知要求的三维动作空间之间存在显著鸿沟。近期的一些研究尝试采用以人为中心的显式表征[^25]并取得了不错的结果，但其训练规模有限，这与利用网络规模数据这一初衷相悖——正是大规模预训练让LLM/LMM取得了成功。

在本文中，我们旨在探讨以下问题：

我们能否像GPT-3那样，利用大规模人类视频预训练一个灵巧型VLA，使其能显式模仿人类动作，并通过后续训练适配到机器人手上？

我们的动机很直接：人类双手是灵巧操作 [^26] 的金标准，在无数自然场景采集的任务中展现出惊人的通用性。从手部运动中学习可以弥合预训练与下游任务之间的异质性，但将其扩展到海量人类视频会带来新的重大挑战。（1）数据异质性：人类视频数据横跨不同的相机系统、坐标系和录制条件，使模型学习复杂化。解决这一问题需要统一异构数据源，同时嵌入必要的 3D 空间推理能力。（2）手部量化：与简单的机器人动作不同，手部运动涉及精细的手指协调，需要谨慎的 tokenization 以保留细粒度控制。因此，连续手部运动必须被离散化为与语言兼容的表示，同时不牺牲毫米级精度。（3）跨模态推理：模型必须学习视觉观测、语言指令与精确手指运动之间的复杂依赖关系，这比传统 LMMs 更具挑战性。（4）机器人控制迁移：由于形态学差异，人类手部运动无法直接迁移到机器人，需要谨慎的技能迁移，以确保学到的策略能够有效适应。应对这些挑战需要一个全面的框架，系统地整合数据、运动表示、推理和迁移学习。

为实现这一目标，我们推出了\textbf{Being-HO}——一种先进且样本效率高的灵巧视觉-语言-动作模型，它基于大规模人类视频进行训练，如图1所示。为了训练该模型，我们提出了“物理指令微调”这一新范式，它通过三个关键组件将视觉指令微调[^27]扩展到物理领域：(1) 基于人类视频的视觉-语言-动作模型预训练，(2) 物理空间对齐，(3) 训练后的自适应调整。与传统视觉指令微调不同，我们的方法通过物理空间对齐来统一来自不同摄像系统和录制条件的异构数据，同时嵌入三维空间推理能力。此外，我们显式地将手部运动建模为先验知识，用以指导机器人的训练后调整，这与当前诸如GR00T [^21]等隐式学习方法[^23] [^23]有所不同。Being-HO采用统一的自回归架构，在视觉、语言和运动模块间共享注意力机制，从而实现无缝的跨模态推理。为了实现精确的运动分词，我们引入了基于分组残差量化（GRQ）[^28] [^29]的高效部件级运动分词方法，其重建精度可达毫米级。为了支持大规模学习，我们整理了\textbf{UniHand}——一个包含超过1.5亿个样本的综合数据集，涵盖了各种操控任务中的动作捕捉数据、虚拟现实录制内容以及仅含RGB信息的视频。据我们所知，这是首次尝试基于大规模人类视频中的显式运动建模来训练灵巧的视觉-语言-动作模型。本研究的主要贡献可总结如下：

- 物理指令微调：这是一种新范式，它将人类双手确立为机器人手部操控的基础，从而弥合人类视频与具身动作之间的差距。

- 部件级运动分词：这是一种量化方法，它能在保持连续手部运动毫米级精度的同时，使其与自回归语言模型的离散架构相兼容。

- UniHand：这是一个大规模数据集，包含超过一亿五千万个涵盖各种操作场景的指令跟随样本，该数据集是通过我们可扩展的数据管道收集的，该管道整合了动作捕捉、虚拟现实以及仅含RGB的视频。

- Being-H0：结合上述创新成果，我们推出了首个基于大规模人体运动视频数据训练的灵巧型视觉-语言-动作模型。我们的模型能够在视觉、语言和精细手部动作之间实现稳健的跨模态推理，针对下游机器人操作任务还采用了定制的适配策略。

---

## 2 相关工作

多模态大模型。Transformer架构[^30]彻底革新了语言模型[31–33]，具备强大的自回归文本理解与生成能力。这一成功延伸到了多模态大模型（LMMs）[34–38]领域，这类模型将LLM推理能力[39–41]与针对特定模态的编码器[^42] [^43]相结合，以实现统一的多模态理解。Flamingo[^44]等开创性工作通过视觉与文本输入之间的交叉注意力机制，展现了出色的少样本视觉问答性能；而LLaVA系列[^24] [^27] [^45]则引入了视觉指令微调技术，借助精心策划的数据集来提升模型的指令跟随能力。这些指令跟随数据集通常先利用视觉模型为图像标注，再由LLM生成问答对[^46] [^47]，或者直接借助专有的LMM进行标注[48–51]。虽然性能顶尖的LMMs[52–54]仍为闭源模型，但近期的研究通过发布模型权重[^55] [^56]、训练细节[^57]甚至数据制作方案[^58]，展现出日益开放的趋势。

人体运动生成。近期在人体运动建模领域取得的进展，得益于日益丰富的专业数据集和参数化模型。早期的运动数据集如KIT-ML[^59]和AMASS[^60]已发展为带文本标注的基准数据集（如HumanML3D[^61]、BABEL[^62]）以及大规模富有表现力的运动集合（如MotionX[^63]、EgoBody[^64]、Nymeria[^65]），最终形成了像MotionLib[^66]这样的百万序列级数据集。人体运动领域依赖SMPL[^67]及其扩展模型SMPL-X[^68]来为可微分的身体表示提供参数化标准。总体而言，运动生成遵循两种主流范式：一是能生成高保真运动的基于扩散的模型[69–75]；二是擅长长期建模和文本推理的自回归方法。对于自回归模型而言，它们通过使用先进的分词器将运动视为离散词元序列，这些分词器包括VQ-VAE[^76] [^77]、残差量化（RQ）[^28] [^78]、分层量化（H2VQ）[^79] [^80]，以及最近流行的无需查找表的量化方法[^66] [^81] [^82]。这些模型通常表现出优异的性能，尤其是在通过微调LLM来完成运动相关任务以解读人类意图方面[66, 83–89]。然而，一个关键挑战在于它们生成的运动缺乏物理真实性，会出现脚部滑动等伪影[^69]。鉴于此，近期的研究探索了利用强化学习（RL）和物理反馈来生成适用于人形机器人控制的物理上合理的运动[90–93]。

手部运动生成。与人类身体不同，手部运动研究主要聚焦于手与物体的交互[94–96]以及精细动作精度。尽管现有基准测试[97–99]能够捕捉这些交互，但其对动作捕捉系统或多摄像头设备的依赖限制了数据多样性，仅局限于桌面场景，从而阻碍了模型的泛化能力。相比之下，头戴式摄像头拍摄的第一人称视频[^94] [^100]虽然能提供丰富的环境多样性，但通常缺乏精确的手部标注。随着[^95] [^101]在单目重建技术上的突破，3D手部建模[102–104]的进步如今已能利用这些数据源提取伪标签。然而，由于这类方法基于较弱的透视假设，其生成的运动仅限于摄像机视锥范围内，因此无法与视角偏移的数据集兼容[^100] [^105]。Dyn-HaMR[^106]通过整合SLAM技术进行摄像头追踪和遮挡鲁棒性优化，解决了这一问题。除了单独建模手部运动外，近期的研究[^96]还致力于手与物体交互，预测交互热点、未来轨迹以及可操作性。

手-物交互（HOI）最初源于2D视觉识别[^107] [^108]和检测任务[109–111]，但逐渐转向用于手部运动生成的3D学习。早期方法采用多阶段流程[^112] [^113]，根据动作标签生成运动[^114] [^115]；而后续方法则借鉴人体运动生成的思路，采用扩散模型[^116]或自回归架构，其中一些方法还利用大语言模型作为统一骨干，以确保长期时间一致性[^117]。尽管取得了这些进展，大多数方法直到MEgoHand[^118]出现前都忽略了视觉输入。在本研究中，我们为Being-HO配备了基于大规模人类视频数据预训练的生成能力，从而为下游操控任务提供手部运动先验。需注意，本版本中并未纳入交互物体建模（如6D姿态），这将是未来的探索方向。

从人类视频中学习VLA。大语言模型的进步使得视觉-语言-动作模型（VLA）能够将多模态输入映射为物理动作。早期的RT-1/2[^3] [^4]等方法将7自由度动作量化以实现自回归预测。最近，OpenVLA[^5]及其后续工作如$\pi_0$ [^6] [^119]和GR00T-N1.5[^23]通过在Open X-Embodiment[^8]和Droid[^9]等数据集上进行大规模预训练，显著扩展了模型能力。FAST[^120]提出利用离散余弦变换来实现快速且可扩展的训练。Octo[^11]和RDT-1B[^121]则没有采用离散动作令牌，而是采用扩散头来实现灵活的动作预测。

---

![Image](images/d14a21b226d71bd07ab689cec9dc43ebebb803fe6a61013f74bb904b98394dd9.jpg)

<div style="text-align: center;">Figure 2: Overview of Being-HO. The text tokenizer and visual encoder are shared by both pretraining and post-training. For pretraining and hand motion/translation tasks, Being-HO generates outputs in an autoregressive manner. For post-training and downstream manipulation tasks, Being-HO incorporates a set of learnable queries as the action chunk for prediction.</div>

尽管取得了进展，当前的数据集仍局限于小规模的实验室采集——对于灵巧手操控而言尤其具有挑战性[^18]，因为其成本高于简单夹爪，导致目前尚无大规模的真实世界灵巧手数据可用。虽然仿真提供了一种可扩展的替代方案[^19] [^20] [^122] [^123]，但仿真到现实之间的差距依然显著，尤其在现实世界部署时会造成困难。

为解决数据瓶颈问题，人类视频提供了一种极具前景的解决方案。以往的研究通过提取可迁移的表征来应对这一挑战，例如视觉特征[^21]、3D感知先验[^124]以及交互知识（如可供性、接触点和抓取动作）[125–127]。然而，由于人类与机器人之间的结构差异，这些方法未能明确建立从人类动作到机器人动作的映射关系。部分研究通过在视觉层面运用数据编辑技术——如图像修复和渲染[^128]、视觉掩码处理[^129]——来实现不同实体间的观测对齐。另一些研究则着眼于动作空间的对齐，利用统一的人为中心的状态-动作空间[25, 129–132]，或借助强化学习/物理仿真技术对轨迹进行优化[133–136]，从而生成物理上合理且更平滑的动作。尽管取得了这些进展，现有方法仍局限于简化的夹持器，因为它们忽略了精细的手指动作对齐问题。在本文中，我们将人类手部视为下游操作任务的通用标准，使机器人能够从互联网视频中学习各种技能。据我们所知，我们是首个通过大规模人类视频中的显式动作建模来预训练可扩展且具备泛化能力的视觉-语言-动作模型的研究团队。

## 3 Being-H0概述

### 3.1 研究动机

尽管当前的视觉-语言-动作模型已取得显著进展，但在新场景和复杂物体交互中，其性能会大幅下降，甚至在指令跟随方面也不及LLaVA这类多模态大语言模型[^24]。这一局限源于模型对合成数据的依赖，而合成数据本身存在仿真到现实的差距[^137] [^138]；或者是因为远程操控的实验室演示数据缺乏多样性[^5]，尤其是在精细操作任务中[^5]。人类活动视频虽是一种有前景的替代方案，但也带来了新的挑战（详见第3.2节）。为应对这些挑战，我们分析了视觉指令微调的成功因素，并提出了物理指令微调这一全新范式，用于训练我们的精细操作视觉-语言-动作模型——Being-HO。如图2所示，该模型包含三个关键组成部分。

★ 预训练。现有的多模态大语言模型虽在多模态推理方面表现优异，但当被应用于操作任务的视觉-语言-动作模型时，其性能却大打折扣。如前所述，我们认为这是由于预训练数据与下游任务数据之间存在根本性的不匹配所致。为弥合这一差距，我们利用人类与机器人操作器之间的解剖学相似性，引入了手部动作生成预训练机制。在该预训练过程中，人类手部被视为理想的操作器，而机器人手部则作为其灵巧性的简化版本。我们利用多模态数据集训练基础视觉-语言-动作模型，使其能够根据视觉和语言信息预测手部动作。

---

$\mathcal{D}=\{(\mathbf{v}_{i},\mathbf{t}_{i},\mathbf{m}_{i})\}$，其中$\mathbf{v}$表示视觉输入，$\mathbf{t}$代表语言指令，而$\mathbf{m}=\{\theta,\mathbf{r}_{rot},\tau,\beta\}$则表示以MANO参数化的动作[^139]（包括关节角度$\theta$、手腕旋转$\mathbf{r}_{rot}$、平移运动$\tau$以及形状$\beta$）。每个样本都被视为一对指令跟随数据$\{\mathcal{X}_{Q},\mathcal{X}_{A}\}$，并通过以下方式优化：

 $$ \theta^{*}=\arg\min_{\theta}\sum_{i=1}^{N}\mathcal{L}(\Theta)=-\sum_{j=1}^{L}\log P_{\Theta}(y_{j}\mid\mathcal{X}_{Q},\hat{y}_{1:j-1}), $$

其中$\Theta$表示基础模型，$X_{A}=\{y_{i}\}$包含来自文本和动作模态的目标标记。这种统一的表述方式能够支持灵活的任务指定，包括从视觉到动作的生成、动作描述，以及针对各种手-物体交互场景的多模态条件化处理。

★ 物理空间对齐。我们的预训练过程旨在弥合视觉与动作之间的差距，以构建基础视觉-语言-动作模型，但这一过程面临着超出常规视觉指令微调范畴的独特对齐挑战。主要难点来自三个方面：（1）多源视觉输入在相机内参上存在差异，且是在动态世界坐标下采集的；（2）模型的骨干网络是基于二维视觉-文本预训练初始化而成的，缺乏关键的三维空间先验；（3）人类能够直观理解的力量、摩擦力等物理属性，在视频数据中根本不存在。与通过具身经验自然发展出三维感知能力的生物视觉系统不同，我们通过物理空间对齐来显式整合这些异构数据源——将各类观测结果统一到一致的坐标系中，从而赋予模型三维推理与物理理解能力。

★ 训练后适配。在完成基于物理空间对齐的预训练后，我们会对基础视觉-语言-动作模型进行下游操控任务的适配。在本研究中，我们采用了简单的基于多层感知机的投影策略，未来也计划探索更复杂的方法。

### 3.2 挑战与解决方案

在第1节中，我们提出了两个核心问题：（1）大规模人类视频能否用于灵巧视觉-语言-动作模型的预训练，使其能像GPT-3通过大规模预训练学习语言那样，明确理解并模仿人类动作？（2）这样的预训练模型能否通过训练后适配有效迁移到机器人操控任务中？为解答这些问题，我们必须克服若干关键挑战。下文我们将剖析这些难题并提出对应的解决方案：

预训练数据整理。与自然语言处理和计算机视觉领域相比，当前的视觉-语言-动作模型面临严重的数据稀缺问题。虽然存在Open X-Embodiment [^8]和AgiBot [^10]等数据集，但它们的规模仍比现有的多模态基准数据集小几个数量级，且主要聚焦于末端执行器控制，受硬件成本限制，忽略了精细的手指协调研究。人类视频或许能缓解这一问题，但目前尚未得到充分利用：大多数方法主要侧重于隐式对齐（例如GR00T N1.5的潜在动作优化 [^23]），且未证实其有效性。近期，一些研究开始探索基于实验室采集数据集 [^99] 的文本到动作生成任务，这类数据标签精确，但规模较小（不足3万条），因此缺乏多样性和泛化能力。相比之下，野外数据集（如Ego4D [^94]）虽能提供足够的数据量，却存在相机参数不一致和运动粒度不统一的问题。我们的解决方案是通过MANO参数标准化 [^139] 和弱透视对齐，系统性地整合异构数据源，最终打造出一个涵盖150多种任务、时长超过1000小时的统一数据集。

精细的手部量化。我们将手部动作视为一种“外语”，由此引出一个关键问题：“离散的动作词元能否保持足够的精度以支持动作预测？”尽管以往的研究认为量化会破坏姿态连续性并降低精度，但我们设计的基于向量量化的分词器 [^76] 通过精心优化，实现了毫米级的重建精度。我们使用一维卷积编码器对连续的MANO序列 $\mathcal{M} \in \mathbb{R}^{T \times D}$ 进行离散化处理，生成的特征图 $z \in \mathbb{R}^{[T/\alpha] \times d}$ 如下：

 $$ \mathcal{M}\xrightarrow{\mathrm{E n c o d e r}}z\xrightarrow{\mathrm{V Q}}\{m_{1},\ldots,m_{n}\}\xrightarrow{\mathrm{D e c o d e r}}\hat{\mathcal{M}}, $$

其中T表示帧数，$\alpha$ 为时间下采样率。动作词元 $m_i \in \{<motion\_{i}d\_0>, ..., <motion\_{i}d\_K>\}$ 由 <MOT> and </MOT> 分隔，从而形成连贯的动作块。

---

确保与我们统一的LMM模型中的文本实现无缝集成。

统一的多模态推理。为了建模视觉观测、语言指令和手部动作之间的复杂关系，我们将所有模态处理成一个统一的词元序列$\mathbf{S} = \{s_i\}$，其中每个词元$s_i$可以代表文本、视觉或动作信息。视觉词元会替换$\langle \text{IMG\_CONTEXT} \rangle$占位符，而动作词元则被组织成序列中的连贯块。通过共享注意力机制实现跨模态交互，其中查询$\mathbf{Q}_{v,t,m}$、键$\mathbf{K}_{v,t,m}$和值$\mathbf{V}_{v,t,m}$是根据拼接后的状态$\mathbf{H}_{v,t,m} = [\mathbf{H}_v; \mathbf{H}_t; \mathbf{H}_m]$计算得出的。这使得模型能够学习丰富的多模态依赖关系：将视觉场景映射为操作策略，把语言指令对应到精确的指部动作，以及让时序动作模式与任务目标保持一致。

自适应机器人控制迁移。尽管预训练好的基础VLA能够在保留广泛能力的同时生成连续动作，但由于运动学不匹配、自由度差异以及物理约束的存在，直接将人类手部动作迁移到下游操控器上仍颇具挑战性。为了验证从人类视频中进行大规模学习的有效性，我们采用了一种基于简单MLP的投影方法，该方法使用一组固定的可学习查询作为下游操控器的动作块。在未来的工作中，我们计划探索更高效且自适应的控制迁移策略，进一步缩小人类动作与机器人动作之间的差距。

下一节将详细阐述我们用于训练Being-HO的技术解决方案，包括物理指令微调流程以及我们构建的包含百万级样本的手部动作数据集UniHand。

## 4 物理指令微调

### 4.1 ★ 预训练

基于大规模人类视频，我们预训练了一个面向从夹爪到灵巧手等各类下游操控器的基础视觉-语言-动作模型（VLA）。人类手部具有极高的自由度，其能力涵盖了现有设备的范畴；因此，Being-HO以人为中心的训练能够通过运动学映射，广泛适用于各类操控任务。在本节中，我们首先简要介绍模型架构（4.1.1节），随后说明如何对人类的手部动作进行精确的词元化（4.1.2节）。接着，我们会详述多模态整合的过程（4.1.3节），并描述其训练流程（4.1.4节）和解码模式（4.1.5节）。

#### 4.1.1 模型架构

我们的模型Being-HO基于预训练的LMM构建，具体采用了InternVL3架构[^140]。该骨干网络由作为视觉编码器的预训练InternViT-300M和作为投影器的两层MLP组成。在每个时间步，模型会处理图像-文本对输入，以预测手部动作序列。参照[^141]的方法，我们采用了动态高分辨率策略，在保持宽高比的前提下将输入图像切分为图像块，从而最大程度减少失真，保留细粒度的视觉细节。受[^66]的启发，我们将手部动作视为一种“外语”，以实现其与LMM的无缝集成。在预训练阶段，手部动作分词器会将连续的动作特征量化为离散嵌入。为了将动作词元整合到LMM骨干网络中，我们为其词汇表扩展了来自动作码本的K个离散编码。我们还引入了两个特殊词元<MOT> and </MOT>，用于标记动作块的边界。

#### 4.1.2 手部动作词元化

运动分词器旨在将原始运动序列中的T帧手部特征$\mathcal{M} = \{m_1, m_2, ..., m_T\}$编码为维度为d的$[T/\alpha]$词嵌入，其中$\alpha$表示时间下采样率。

运动特征。我们采用3D模型MANO [^139]来表示手部姿态，其参数化为$m = \{\theta, \mathbf{r}_{rot}, \tau, \beta\}$。然而，高效且有效地表示手部运动至关重要。因此，选择合适的特征空间十分关键。在本文中，我们探讨了五种替代表示方式：

---

![Image](images/90d79b3a8f21b7dcebb0229b4196f3f49e6f3964dd64f41d03548419d194c3c7.jpg)

<div style="text-align: center;">Figure 3: Physical Instruction Tuning. Our training paradigm bridges human video datasets and robotic manipulation through a novel unified physical instruction tuning. Left: Part-level motion tokenization converts continuous hand motions into discrete tokens. Physical space alignment unifies heterogeneous data sources — from human hand demonstrations in the videos (dataset) to real-robot data — through coordinate system alignment and MANO parameterization, creating consistent representations for both pretraining and post-training supervision. Mid: During pretraining, we extend vision-text parameters  $\Theta_{v,t}$ to include motion parameters  $\Theta_m$, enabling multi-head attention across vision, text, and motion tokens within a unified sequence. We use blue to denote visual and text attention, red for motion attention, and yellow for cross-modal attention. Right: The extension phase shows how attention mechanisms adapt to pretrained cross-modal dependencies ( $Attn_{v,t|m}$), followed by post-training where action parameters  $\Theta_a$ are incorporated to produce the final VLA with parameters  $\Theta_{a,v,t|m}$ for downstream robotic tasks. The green part represents action attention.</div>

- MANO-D51：每一帧中的手部运动被编码为$m \in \mathbb{R}^{51}$，包含$\theta \in \mathbb{R}^{15 \times 3}$、$\mathbf{r}_{rot} \in \mathbb{R}^{3}$和$\tau \in \mathbb{R}^{3}$；其中$\theta$和$\mathbf{r}_{rot}$以轴角形式表示。

- MANO-D99：每一帧中的手部运动被编码为$m \in \mathbb{R}^{99}$。与MANO-D51不同，该特征采用6D旋转$\theta \in \mathbb{R}^{15 \times 6}$和$\mathbf{r}_{rot} \in \mathbb{R}^6$而非轴角形式。

- MANO-D109：它在MANO-D99的基础上额外加入了形状参数$\beta \in \mathbb{R}^{10}$。

- MANO-D114：它在MANO-D51的基础上增加了关节位置$j \in \mathbb{R}^{21 \times 3}$。需注意，关节位置仅在重建训练阶段作为辅助特征使用，而在评估和推理阶段，我们仅利用这51维参数。

• MANO-D162：与MANO-D114类似，它在MANO-D99的基础上增加了关节位置$j \in \mathbb{R}^{21 \times 3}$。

在我们的实验中观察到，6D旋转特征在手指关节旋转的重建质量上表现更佳，而轴角特征在手腕姿态的表示上更优。我们将这一现象归因于不同手部部位的结构特性。手腕通常表现出更大但更简单的旋转，因此受益于轴角公式的紧凑性和计算效率[^142] [^143]。相比之下，手指旋转涉及更精细的细节，这些细节能更好地被6D旋转表示的连续性和数值稳定性所捕捉。

尽管由于手腕姿态误差的主导影响，使用轴角特征时整体重建误差更低，但我们最终还是为手部运动分词器选择了6D旋转特征，因为它在Being-HO训练中表现更佳。一种可能的解释是，LMMs相对更容易学习手腕姿态模式，而建模精细的手指运动则更具挑战性。因此，在本文中，我们选择MANO-D162作为手部运动的特征。我们将在未来的工作中探索将手腕的轴角特征与手指的6D旋转特征相结合的方案。

---

分组残差量化。运动分词器的精度对生成的手部运动质量以及所学运动先验在下游操控任务中的可迁移性有着至关重要的影响。为确保最佳性能，我们为手部运动精心设计了专用的分词器。我们的架构基于分组残差量化变分自编码器（GRQ-VAE）[^29]，如图4所示。给定一个运动序列$\mathcal{M} \in \mathbb{R}^{T \times D}$，编码器会将其转换为特征图$z \in \mathbb{R}^{[T/\alpha] \times d}$。随后，分词器通过多阶段残差量化过程对特征图z进行离散化。首先，沿通道轴将特征维度d划分为n个组，每组独立使用残差向量进行量化。

![Image](images/6d73c0ee9591a40ad6c6985bf9ef8b07f70daba738490d8b9b4aa3caa63e23b6.jpg)

<div style="text-align: center;">Figure 4: Architecture of part-level hand motion tokenization based on GRQ.</div>

量化器（RQ）[^28]。对于每组$z^{(g)} \in \mathbb{R}^{[T/\alpha] \times d/n}$，会应用来自码本$\mathcal{C}^{(g)}$的一组L个编码，其中L代表RQ的层数。因此，组g中每个特征向量$z_i^{(g)}$的量化过程如下：

 $$ r_{0}=z_{i}^{(g)},\quad q_{l}=\underset{c\in\mathcal{C}^{(g)}}{\arg\min}\|r_{l-1}-c\|_{2},\quad r_{l}=r_{l-1}-q_{l}, $$

其中$i \in \lceil T/\alpha \rceil$、$q_l$和$r_l$分别表示第l层选定的编码和残差。随后，最终的量化表示计算如下：

 $$ \hat{z}_{i}^{(g)}=\sum_{l=1}^{L}q_{l}. $$

通过将特征通道分组并对每组进行多阶段残差量化，GRQ能够生成更具表现力的运动输入表示。我们发现，腕部参数$\mathbf{r}_{rot}$和$\tau$的重建误差主要源于它们在三维空间中分布范围较广，尽管它们对运动令牌化的精度至关重要。为解决这一问题，我们在整体量化目标中引入了针对腕部的重建损失项，其定义如下：

 $$ \mathcal{L}_{\mathrm{w r i s t}}=||\mathbf{w}-\hat{\mathbf{w}}||_{2}^{2},\quad\mathbf{w}=[\mathbf{r}_{r o t},\tau], $$

其中$\hat{w}$表示重建后的腕部参数。最终目标函数结合了重建损失和承诺损失[^28]，以及腕部相关项$(\lambda_{\{1,2\}})$（表示平衡权重）：

 $$ \mathcal{L}=\mathcal{L}_{\mathrm{r e c o n}}+\lambda_{1}\mathcal{L}_{\mathrm{c o m m i t}}+\lambda_{2}\mathcal{L}_{\mathrm{w r i s t}}, $$

我们采用指数移动平均（EMA）更新策略来更新码本。

部件级运动令牌化器。鉴于腕部参数重建的复杂度高于手指运动，我们借鉴[^66]的思路，为腕部和手指参数设计了独立的令牌化器，以便每个令牌化器能更好地建模部件级特征。具体而言，手部运动特征$m = \{\theta, \mathbf{r}_{rot}, \tau, \beta\}$被分解为用于全局姿态和精准定位的腕部运动$\{\mathbf{r}_{rot}, \tau\}$，以及用于精细操作的手指运动$\{\theta, \beta\}$。这种部件级令牌化不仅改善了特征建模效果，还提供了明确的令牌语义，使LMM骨干网络能更好地捕捉结构化的手部动态。使用部件级令牌化器时，会省略腕部损失$\mathcal{L}_{\text{wrist}}$。

---

#### 4.1.3 多模态集成

与传统LLM类似，它采用下一个令牌预测机制来生成输出。Being-HO将RGB视觉、文本和手部运动三种模态统一处理，将它们令牌化为离散令牌。文本的处理方式遵循LLM的流程，下面我们将详细介绍另外两种模态的处理过程。

视觉令牌。视觉输入需要经过特殊处理，以应对分辨率可变的图像和复杂的动态内容。对于输入图像，我们首先采用动态分块技术，根据图像内容的复杂程度生成N个图像块。参照InternVL [^57]的做法，分块策略中包含缩略图生成步骤以保留全局上下文：始终保留像素上采样比为0.5的降采样版本$I_{\mathrm{thumb}}$以及细节丰富的图像块。视觉处理流程首先利用视觉编码器从这些图像块中提取特征，再通过MLP层将其投影到统一的嵌入空间。视觉令牌用边界标记$\langle \mathrm{IMG} \rangle$和$\langle / \mathrm{IMG} \rangle$来界定，而$\langle \mathrm{IMG\_CONTEXT} \rangle$则作为占位符令牌，在处理过程中会被实际的视觉嵌入所替换。

运动令牌。在将运动数据整合到令牌流之前，会先对其进行量化处理。假设运动特征序列表示为$\mathcal{M}$，运动分词器会将其量化为一系列离散令牌$\{m_i\}$。运动序列通过边界令牌$\langle \text{MOT} \rangle$和$\langle \text{MOT} \rangle$来构建结构，每秒可生成包含128个令牌的运动块。这种结构化表示方式既能确保运动信息在令牌流中被清晰区分，又能与Transformer架构保持兼容。

多模态融合。模型通过统一的令牌空间处理所有模态数据，利用共享的嵌入层和注意力机制。在融合过程中，视觉令牌会替换<IMG_CONTEXT> placeholders while motion tokens are inserted as structured blocks within the text sequence. This generates a combined token sequence $\mathbf{S} = \{s_i\}$ where each element $s_i$ may represent text, visual, or motion content. The attention mechanisms operate across modalities simultaneously. For the concatenated multimodal hidden states $\mathbf{H}_{v,t,m} = [\mathbf{H}_v; \mathbf{H}_t; \mathbf{H}_m]$ (representing vision, text, and motion embeddings), we compute query, key and value through shared projections:

 $$ \mathbf{Q}_{v,t,m}=\mathbf{W}_{Q}\mathbf{H}_{v,t,m},\quad\mathbf{K}_{v,t,m}=\mathbf{W}_{K}\mathbf{H}_{v,t,m},\quad\mathbf{V}_{v,t,m}=\mathbf{W}_{V}\mathbf{H}_{v,t,m} $$

where $\mathbf{W}_{\{Q,K,V\}}$ denotes the weight matrices. This architecture enables direct cross-modal attention, allowing the model to capture rich interdependencies between modalities, such as correlating visual observations with specific hand movements, or grounding language instructions in motion sequences.

Figure 3 illustrates this multimodal integration process, where our pretraining extends the original vision-text parameters $\Theta_{v,t}$ to incorporate motion parameters $\Theta_m$, enabling unified processing of all three modalities through shared attention mechanisms. The model learns to generate coherent motion sequences by predicting discrete motion tokens within the broader context of visual observations and language instructions.

#### 4.1.4 Training Details

Hand Motion Tokenizer. We sample hand motion sequences at 15 frames per second (FPS) and tokenize them using fixed one-second windows. Noting that camera shift may occur within the temporal window, and Being-HO does not predict camera motion during inference. Considering this, we transform each sequence into the coordinate system of its first frame. To enable coherent generation of longer sequences, where each one-second segment within a multi-second output should be represented relative to the first frame of the entire sequence, we implement a specialized training strategy. For each one-second sample, we randomly select a reference frame from a larger temporal window (10 seconds in this paper) and transform the motion sequence relative to this frame. This approach allows the learned motion tokens to effectively represent movements relative to varying world coordinate systems while preserving long-term consistency.

Foundation VLA. The model is trained using standard next-token prediction. To optimize the integrated motion codes, we introduce a dual-level masking strategy to constrain the optimization at both the vocabulary and token-wise levels for more effective training.

---

- Vocabulary-Level Logit Masking. Since motion codes  $\mathcal{V}_{\text{motion}}$ occupy only a small portion of the foundation VLA’s vocabulary  $\mathcal{V}$, we selectively mask non-motion logits on motion labels with a probability of  $\mathcal{P}$. This would help to focus gradient updates on motion-related embedding space and avoid being diluted by trivial non-motion generations. Given predicted logits  $\mathbf{z} \in \mathbb{R}^{|\mathcal{V}|}$, we apply masking as:

 $$ \tilde{\mathbf{z}}_{i}=\begin{cases}{\mathbf{z}_{i}}&{i\in\mathcal{V}_{\mathbf{m o t i o n}}}\\ {-\infty}&{\mathit{o t h e r w i s e}.}\\ \end{cases}\quad\mathrm{(w i t h~p r o b a b i l i t y~}\mathcal{P}\mathcal{I}). $$

- Token-level Loss Masking. The token-wise cross-entropy losses are computed using the masked logits $\tilde{z}$. To address the natural variation in hand motion complexity (from static poses to unpredictable jitters), we filter extreme loss values, focusing learning on moderately challenging tokens. For per-token losses $L = \{\ell_1, \ldots, \ell_N\}$, we define the filtered loss set as:

 $$ \tilde{L}=\left\{\ell_{i}\in L\mid Q_{\mathrm{l o w}}\leq\ell_{i}\leq Q_{\mathrm{h i g h}}\right\}. $$

where $Q_{low}$, $Q_{high}$ denote the lower and upper percentiles of preset percentages.

The final motion loss is computed as the mean over the masked losses:

 $$ \mathcal{L}_{\mathrm{m o t i o n}}=\frac{1}{|\tilde{L}|}\sum_{\ell_{i}\in\tilde{L}}\ell_{i}. $$

#### 4.1.5 Decoding Modes

Our pretrained foundation VLA is able to generate both textual and motion outputs through unified next-token prediction, with valid motion blocks decoded into MANO parameters via the motion tokenizer. In this work, we provide three decoding modes to balance generation flexibility and the structural consistency required to recover valid hand motion: (1) Free-format Mode. This mode allows fully flexible autoregressive sampling without any explicit structural constraints. However, it risks producing invalid motion blocks that fail to decode. (2) Block-formatted Mode. This mode enforces structural consistency to the expected motion block by restricting sampling to the motion token vocabulary only between <MOT> and </MOT>分隔符。为了基于真实值运动块数量进行评估，我们避免过早生成<EOS> token until generating the target number of motion blocks. (3) Soft-formatted Mode. To evaluate local motion generation quality while accounting for the one-to-many nature between motion instruction and sequences, we employ a soft constraint strategy. After generating each motion block, we blend the predicted and ground-truth MANO parameters via their mean, then re-quantize this hybrid through the motion tokenizer. This enforces the generation anchored in a plausible neighborhood of the ground-truth, providing a more reliable estimate of the model's ability to produce high-quality motion in the vicinity of real trajectories.

### 4.2 ★ Physical Space Alignment

To build a sufficiently large-scale dataset of dexterous human videos, we collect samples from diverse datasets and public sources. However, this approach introduces variability in camera systems, posing challenges for effective pretraining. Furthermore, existing LMMs exhibit limited 3D perception capabilities — a well-documented limitation in prior research [^144] [^145] [^146] [^147]. To alleviate this, we introduce physical space alignment, a unified toolkit that maps videos recorded by different cameras into a consistent physical space, meanwhile incorporating 3D spatial reasoning and physical attributes (if available) to enhance geometric and perceptual consistency across the datasets. As illustrated in Figure 3, this alignment process serves as the critical bridge between human demonstrations from videos and robotic manipulations, enabling consistent MANO parameterization across both domains for effective learning. We introduce two strategies below, with more to be explored in future work.

---

#### 4.2.1 Weak-Perspective Projection Alignment

Inherent differences in camera systems across data sources introduce inconsistent projection of 3D space. Although humans can intuitively perceive depth and estimate distances between hands and objects for grasping, models trained on such multi-sourced datasets frequently struggle with accurately mapping the image projections to the actual 3D context, leading to errors in 3D spatial reasoning. To alleviate this, we establish a unified weak-perspective camera space, which ensures consistent alignment from 2D visual content to a shared 3D reference frame. This approach maintains uniform pixel scales for objects at similar depths, mitigating the inconsistencies caused by differing camera intrinsics. Specifically, given a source image with camera intrinsic $K = \{f_x, f_y, c_x, c_y\}$ and target camera intrinsic $K' = \{f'_x, f'_y, c'_x, c'_y\}$, we compute scale factors and translation offsets as:

 $$ s_{x}=\frac{f_{x}^{\prime}}{f_{x}},\quad s_{y}=\frac{f_{y}^{\prime}}{f_{y}},\quad\Delta x=c_{x}^{\prime}-s_{x}\cdot c_{x},\quad\Delta y=c_{y}^{\prime}-s_{y}\cdot c_{y}. $$

Each pixel $(u, v)$ in the source image is then transformed to $(u', v')$ via:

 $$ u^{\prime}=s_{x}\cdot u+\Delta x,\quad v^{\prime}=s_{y}\cdot v+\Delta y, $$

The source image is further cropped or padded to the same target resolution (W, H). For videos with severe lens distortion (e.g., fisheye cameras), we first normalize the field of view (FoV) to $90^{\circ}$ to minimize projection artifacts before alignment.

#### 4.2.2 View-Invariant Motion Distribution Balancing

Developing robust instruction-following capabilities requires careful pre-processing of instructional tuning data to ensure a balanced data distribution, particularly for physical instruction tuning. If a single camera configuration dominates the dataset, it may introduce bias into the 3D perception system, ultimately limiting the model's generalization ability in unseen camera settings. To mitigate this issue, we propose a novel distribution-balancing strategy to augment video-motion pairs from small-scale data sources, preventing them from being overshadowed by larger ones. During balancing, we vary hand pose distribution without introducing camera viewpoint and position changes. Importantly, our approach preserves weak-perspective consistency across motions from different sources, ensuring coherent 3D understanding. Unlike conventional image-only augmentations (e.g., random cropping or flipping) that may disrupt weak-perspective alignment between hand motion and visual observations, our strategy employs two complementary components:

Depth Scaling. For a hand pose in camera coordinate $m_c = \{\beta, \theta, R_c, \tau_c\}$, where $\tau_c = (\tau_c^x, \tau_c^y, \tau_c^z)$ denotes the wrist's 3D position and $R_c \in \mathbb{R}^{3 \times 3}$ is the rotation matrix form of $\mathbf{r}_{rot}$, we perturb human hand's depth by randomly sampling a scaling factor $\lambda_s$:

 $$ \tau_{c}^{z^{\prime}}=\lambda_{s}\cdot\tau_{c}^{z} $$

To maintain weak-perspective consistency, the paired image is rescaled by $1/\lambda_{s}$, yielding an intermediate resolution $(\mathcal{W}/\lambda_{s}, \mathcal{H}/\lambda_{s})$. We constrain $\lambda_{s}$ to plausible ranges to avoid unrealistic perspective distortions caused by the non-negligible physical size of the hand.

In-Plane Rotation. To diversify hand positions in the image plane without introducing geometric inconsistencies, we apply a rotation around the camera's optical axis (Z-axis) with a uniformly sampled angle $\varphi$. This updates both wrist position and global rotation:

 $$ \tau_{c}^{\prime}=R_{z}(\varphi)\cdot\tau_{c},\quad R_{c}{^{\prime}}=R_{z}(\varphi)\cdot R_{c}. $$

---

Here, $R_{z}(\varphi)$ is the rotation transform matrix. The corresponding image is rotated synchronously by $\varphi$ to maintain consistency. All transformed frames are then adjusted to the target resolution $(\mathcal{W}, \mathcal{H})$, ensuring weak-perspective projection integrity while increasing hand diversity.

#### 4.2.3 Further Discussion

Beyond the two strategies proposed above, we believe that integrating richer physical cues can further improve the model's understanding of spatial and physical environments. For instance, incorporating visual depth information, tactile feedback, or other multi-sensory signals may provide more grounded and embodied representations of human activities. These modalities offer complementary perspectives on physical interactions and 3D structure, which are often ambiguous or underspecified in 2D visual inputs alone.

Such multi-sensory integration could address fundamental limitations inherent in vision-only approaches. Depth information from RGB-D sensors could resolve spatial ambiguities that arise from weak-perspective projection, while tactile feedback could capture crucial contact dynamics, grip forces, and material properties that are invisible in visual observations but essential for successful manipulation. Audio signals from object interactions could further disambiguate manipulation strategies that appear visually similar but involve different physical processes, such as distinguishing between gentle placement and firm pressing actions.

These enhanced alignment strategies could create more robust representations that better capture the rich physical understanding humans naturally possess during manipulation tasks. As we scale our approach to larger and more diverse datasets, incorporating such multi-modal physical cues will become increasingly important for bridging the gap between human demonstration data and reliable robotic deployment across varied real-world scenarios.

### 4.3 ★ Post-Training for Dexterous Manipulation

Following pretraining and physical space alignment, our foundation VLA possesses comprehensive vision-language-motion understanding but requires adaptation to specific robotic manipulation tasks. As illustrated in Figure 3, this post-training stage extends our model parameters from $\Theta_{v,t,m}$ to $\Theta_{a,v,t,m}$, incorporating action parameters $\Theta_a$ that enable direct robotic control while leveraging the rich multimodal representations learned during pretraining. In this paper, we validate our approach using dexterous hands as the downstream manipulator. Even though our paradigm remains adaptable to various robotic hand configurations, including grippers. For instance, wrist motion can serve as a prior to guide end-effector movement, which has previously been explored [^129]. Therefore, we do not elaborate on further.

The kinematic differences between human and robotic hands prevent direct transfer of the foundation VLA and its motion tokens. Since our major focus is dexterous VLA learning from large-scale human videos, we bridge this gap with a non-autoregressive MLP-based projection method as shown in Figure 2, leaving further alternatives (e.g., discrete action tokens or diffusion policies) for future work. Specifically, we leverage the VLA backbone as a pretrained encoder, where a lightweight MLP $f_p$ projects the dexterous hand's proprioceptive states into its embedding space. The proprioceptive embedding is combined with visual-text tokens to form a unified context (ctx), enabling joint reasoning about sensory inputs, language, and current physical configuration. For action generation, we employ a set of learnable query tokens $\{\mathbf{q}_1, \ldots, \mathbf{q}_{N_a}\}$ that attend to these contexts within the pretrained encoder, with a regression policy head MLP $f_r$ transforming the pretrained encoder's outputs into executable dexterous poses. The objective of post-training is to replicate expert demonstrations through imitation learning. Given an expert action sequence $\mathbf{a}^* = \{\mathbf{a}_i^*\}$, we optimize the model by minimizing the L1 loss between predicted actions $\mathbf{a}$ and the expert data:

 $$ \mathbf{a}_{i}=f_{r}\Big(\Theta\big(\mathbf{q}_{i},\mathtt{c t x}\oplus f_{p}(\mathbf{p}_{t})\big)\Big),\quad\mathcal{L}(\Theta_{\mathbf{a}})=\frac{1}{N_{a}}\sum_{i=1}^{N_{a}}\|\mathbf{a}_{i}-\mathbf{a}_{i}^{*}\|_{1}, $$

where $N_{a}$ represents the action chunk size and $\Theta_{a}$ denotes all trainable parameters during post-training, including the foundation VLA $\Theta$, the action queries $q_{i}$, the proprioceptive projector $f_{p}$, and the policy head $f_{r}$. The term ctx represents the visual and textual context, and $\oplus$ denotes the concatenation operation. The

---

![Image](images/c03c9d20c47ecacb3fe2e7d14baf7648f0079f21b86ae4c708c9ebc848dcb692.jpg)

<div style="text-align: center;">Figure 5: The overview of our UniHand-2.5M. Left: The scenes and tasks from different data source types. Mid: The distribution of different data sources, data types, and durations. Right: Samples from different data types.</div>

term $\mathbf{a}_i^\ast \in \mathbb{R}^{D_a}$ denotes the actions from expert demonstrations, and $D_a$ is the dimension of the action space. This approach effectively evolves the pretrained foundation VLA to produce robot-executable control while maintaining cross-modal reasoning, with supportive tasks like: motion generation from visual-textual inputs, text-based captioning of observed motions, and robot control adaptation via domain-specific fine-tuning.

## 5 UniHand: Scaling up Hand Motion Instructional Data

Our pretrained foundation VLA requires massive human videos to learn diverse dexterous skills. In this section, we introduce the data curation pipeline in detail.

### 5.1 Data Sources and Statistics

We curate our dataset from three primary sources, each with distinct advantages and trade-offs: (1) Motion capture datasets [98, 148–150], which incorporate high-precision 3D annotations from multi-view motion capture systems in controlled environments (e.g., studio, lab), though their diversity is often limited. For example, OAKINK2 [^99] offers multi-view, object-centric recordings of real-world bimanual manipulation, including complex tasks. (2) VR-recorded datasets, which leverage VR devices (e.g., Apple Vision Pro) with calibrated cameras and SLAM-based tracking to capture natural hand-object interactions in less constrained settings while maintaining reliable 3D ground truth. A notable example is EgoDex [^151], which includes up to 194 household manipulation tasks such as tying shoelaces and folding laundry. (3) Pseudo-annotated datasets [^152], which we utilize off-the-shelf hand motion predictors [^152] to generate pseudo 3D labels from in-the-wild videos. While noisier, these datasets excel in scalability and diversity, as seen in prior large-scale applications [^153]. For instance, Taste-Rob [^154] includes around 100K egocentric videos with aligned language instructions recorded from a fixed viewpoint. We exclude samples with frequent hand occlusion, out-of-frame (e.g., Ego4D [^100]), or dynamic camera viewpoints despite cleaner visuals (e.g., EPIC-Kitchen [^105]), leaving the process of these datasets in our future work.

The recipe of our dataset UniHand is shown in Table 1. Our dataset is aggregated from 11 sources with detailed hand motion annotations along with corresponding RGB observations. In total, UniHand contains more than 440K task trajectories, covering over 130 million frames and over 1,100 hours of video, thus offering high diversity and strong coverage of real-world scenarios. Based on this rich data foundation, our instructional data labeling pipeline (detailed in the following sections) produces over 165 million motion-instruction pairs, offering large-scale supervision for dexterous VLA learning. Moreover, since our data curation pipeline supports both VR-recorded datasets and pseudo-annotated datasets, while the former is relatively easy to collect at scale, and the latter allows us to leverage the vast pool of in-the-wild human videos, more diverse data is on the way to be incorporated in the future. Due to computational cost constraints, we sample 2.5 million instruction data points from UniHand for pretraining. This subset is selected based on a balanced

---

<div style="text-align: center;">sampling strategy to ensure diversity in task types and sources, which we refer to as UniHand-2.5M, as shown in Figure 5.</div>

<div style="text-align: center;">Table 1: Statistics of UniHand. Our dataset aggregates hand motion data from 11 benchmarks across three types: motion capture, VR-recording, and pseudo-annotation. To the best of our knowledge, this represents the largest egocentric hand motion dataset to date. By regarding human hands as the golden standard for dexterous hands or grippers, we anticipate that UniHand will establish foundational references for VLA learning. #Inst refers to the number of our generated instructional samples.</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'>Dataset</td><td style='text-align: center; word-wrap: break-word;'>#Inst</td><td style='text-align: center; word-wrap: break-word;'>#Seq</td><td style='text-align: center; word-wrap: break-word;'>#Avg len</td><td style='text-align: center; word-wrap: break-word;'>#Frames</td><td style='text-align: center; word-wrap: break-word;'>#Hours</td><td style='text-align: center; word-wrap: break-word;'>Hand joint</td><td style='text-align: center; word-wrap: break-word;'>Hand Pose</td><td style='text-align: center; word-wrap: break-word;'>Ann Granularity</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>ARCTIC [^148]</td><td style='text-align: center; word-wrap: break-word;'>17.9M</td><td style='text-align: center; word-wrap: break-word;'>0.3K</td><td style='text-align: center; word-wrap: break-word;'>725.3</td><td style='text-align: center; word-wrap: break-word;'>245K</td><td style='text-align: center; word-wrap: break-word;'>2.3</td><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>Action</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>FPHA [^149]</td><td style='text-align: center; word-wrap: break-word;'>798K</td><td style='text-align: center; word-wrap: break-word;'>1.2K</td><td style='text-align: center; word-wrap: break-word;'>89.8</td><td style='text-align: center; word-wrap: break-word;'>105K</td><td style='text-align: center; word-wrap: break-word;'>1.0</td><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>✗</td><td style='text-align: center; word-wrap: break-word;'>Action</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>HoloAssist [^155]</td><td style='text-align: center; word-wrap: break-word;'>8.0M</td><td style='text-align: center; word-wrap: break-word;'>2.2K</td><td style='text-align: center; word-wrap: break-word;'>8081.7</td><td style='text-align: center; word-wrap: break-word;'>17.1M</td><td style='text-align: center; word-wrap: break-word;'>166</td><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>✗</td><td style='text-align: center; word-wrap: break-word;'>Segment</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>H2O [^150]</td><td style='text-align: center; word-wrap: break-word;'>3.7M</td><td style='text-align: center; word-wrap: break-word;'>0.9K</td><td style='text-align: center; word-wrap: break-word;'>121.5</td><td style='text-align: center; word-wrap: break-word;'>115K</td><td style='text-align: center; word-wrap: break-word;'>1.1</td><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>Action</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>HOI4D [^98]</td><td style='text-align: center; word-wrap: break-word;'>21.2M</td><td style='text-align: center; word-wrap: break-word;'>3.0K</td><td style='text-align: center; word-wrap: break-word;'>273.0</td><td style='text-align: center; word-wrap: break-word;'>825K</td><td style='text-align: center; word-wrap: break-word;'>7.6</td><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>Action</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>HOT3D [^97]</td><td style='text-align: center; word-wrap: break-word;'>8.7M</td><td style='text-align: center; word-wrap: break-word;'>2.8K</td><td style='text-align: center; word-wrap: break-word;'>150.0</td><td style='text-align: center; word-wrap: break-word;'>420K</td><td style='text-align: center; word-wrap: break-word;'>3.9</td><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>N/A</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>OAKINK2 [^99]</td><td style='text-align: center; word-wrap: break-word;'>18.5M</td><td style='text-align: center; word-wrap: break-word;'>2.8K</td><td style='text-align: center; word-wrap: break-word;'>244.4</td><td style='text-align: center; word-wrap: break-word;'>695K</td><td style='text-align: center; word-wrap: break-word;'>6.5</td><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>Action</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>TACO [^156]</td><td style='text-align: center; word-wrap: break-word;'>11.5M</td><td style='text-align: center; word-wrap: break-word;'>2.2K</td><td style='text-align: center; word-wrap: break-word;'>154.0</td><td style='text-align: center; word-wrap: break-word;'>340K</td><td style='text-align: center; word-wrap: break-word;'>3.2</td><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>Action</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>DexYCB [^157]</td><td style='text-align: center; word-wrap: break-word;'>3.6M</td><td style='text-align: center; word-wrap: break-word;'>5.6K</td><td style='text-align: center; word-wrap: break-word;'>72.8</td><td style='text-align: center; word-wrap: break-word;'>410K</td><td style='text-align: center; word-wrap: break-word;'>3.8</td><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>N/A</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Taste-Rob [^154]</td><td style='text-align: center; word-wrap: break-word;'>1.9M</td><td style='text-align: center; word-wrap: break-word;'>85K</td><td style='text-align: center; word-wrap: break-word;'>164.1</td><td style='text-align: center; word-wrap: break-word;'>14M</td><td style='text-align: center; word-wrap: break-word;'>130</td><td style='text-align: center; word-wrap: break-word;'>✗</td><td style='text-align: center; word-wrap: break-word;'>✗</td><td style='text-align: center; word-wrap: break-word;'>Trajectory</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>EgoDex [^151]</td><td style='text-align: center; word-wrap: break-word;'>70.6M</td><td style='text-align: center; word-wrap: break-word;'>338K</td><td style='text-align: center; word-wrap: break-word;'>264.8</td><td style='text-align: center; word-wrap: break-word;'>89.6M</td><td style='text-align: center; word-wrap: break-word;'>829.4</td><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>✗</td><td style='text-align: center; word-wrap: break-word;'>Trajectory</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Total</td><td style='text-align: center; word-wrap: break-word;'>166.5M</td><td style='text-align: center; word-wrap: break-word;'>444.1K</td><td style='text-align: center; word-wrap: break-word;'>-</td><td style='text-align: center; word-wrap: break-word;'>130M</td><td style='text-align: center; word-wrap: break-word;'>1155</td><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>Fine-Grained</td></tr></table>

### 5.2 Data Curation Pipeline

#### 5.2.1 Hand Pose Standardization

Our model learns an explicit mapping from 2D visual observations to 3D spatial coordinates by treating hand motions as 3D signals, ensuring both geometric precision and visual-semantic consistency. To address the heterogeneity in motion labels across datasets, we combine different data sources through hand pose standardization. For datasets with motion capture or SLAM-tracked labels, we directly extract their annotations in the form of MANO parameters [^139]. When only 3D hand joint positions are available, we derive the corresponding MANO parameters via gradient-based optimization. In cases where datasets lack 3D hand pose or joint annotations entirely, we leverage HaMer [^102] for frame-wise pose estimation to maintain consistent motion semantics. To enhance the reliability of HaMer's output, we detect and correct left-right hand mismatches by identifying pose discontinuities, followed by applying temporal interpolation to fill minor gaps. Additionally, the fitting process incorporates joint angle constraints and temporal smoothness regularization to ensure physically plausible and coherent hand motions.

#### 5.2.2 Task Description Labeling

To establish strong semantic grounding between vision, language, and motion, we introduce a structured hierarchical labeling framework that enriches motion semantics beyond the sparse or imprecise textual labels in existing datasets. This framework provides detailed and consistent textual descriptions, enabling our VLA to effectively align visual inputs, natural language instructions, and quantized hand motion representations. For structured coverage, we segment each video into non-overlapping chunks with a maximum length of 10 seconds, ensuring each captures a distinct phase of the task. We then samples frames at 2FPS and leverage Gemini-2.5-Flash-Lite [^53] to generate annotations at two temporal levels: At the chunk level, we produce imperative instructions and concise summaries that describe overarching hand activities and object interactions. At the more granular per-second level, we further divide each chunk into overlapping 1-second windows, annotating them with precise instructions and captions that detail contact states, object attributes, hand parts, and motion trajectories relative to the camera perspective. To ensure clarity and completeness, we separately annotate global two-handed actions and individual hand actions, capturing both bilateral and

---

unilateral descriptions. This multi-scale labeling strategy guarantees comprehensive and consistent coverage, bridging high-level task objectives with fine-grained hand-object interactions in a unified framework.

#### 5.2.3 Instructional Data Generation

Building on our systematic annotations, we construct instruction-following training data to explicitly establish rich vision-language-motion alignment for the foundation VLA. To achieve this, the designed instructional tasks focus on multiple grounding aspects for hand motion understanding, including spatial-temporal alignment of hand trajectories with visual context, precise object attributes and contact states, clear action intentions, and consistency between high-level instructions and fine-grained motion steps. Guided by these principles, we develop training data for three complementary task types: (1) Instructional motion generation, where the model learns to generate step-by-step motion sequences conditioned on scene images and task instructions; (2) Motion translation, where the model is required to convert the motion sequence and visual cues into language descriptions of hand-object interactions; and (3) Contextual motion prediction, where the model is asked to anticipate subsequent motion sequences based on prior motion history, the current scene observation, and optional instructions or task goals.

For implementation, we design approximately 20 base templates per task type and employ Gemini-2.5-Pro to generate diverse instructional variants. Each template explicitly incorporates target duration specifications, enabling the model to handle varying temporal granularities and sequence lengths. Through rule-based instantiation, we populate these templates with grounded instructions, motion tokens, and explicit length constraints.

To ensure a balanced visual perspective distribution in the training set, we apply the view-invariant motion distribution balancing method described in Section 4.2.2 to augment the data. Based on the balanced dataset, our pipeline produces over 165 million high-quality instructional pairs spanning multiple temporal scales, handedness configurations, and manipulation scenarios, with systematic quality checks for semantic coherence. To furthermore balance the distribution of data sources and task types in the training data, we sample a subset with 2.5 million instances from the full dataset that achieves a more even coverage of both task categories and data origins. The distribution of different tasks and data sources within UniHand-2.5M is shown in Figure 5, while the proportion of samples generated from the view-balanced data is illustrated in Figure 6. This unified design provides robust supervision for the model to learn consistent mappings between vision,

![Image](images/b32ab1661c19a2c35da6ffbc6a9eef99badb54baf8055536ba10f4df18347732.jpg)

<div style="text-align: center;">Figure 6: Comparison of the proportion of instruction samples derived from original data vs. view-balanced data in UniHand-2.5M.</div>

language, and structured motion. of bilateral and unilateral hand-object interactions. Together, this structured multi-scale annotation framework ensures comprehensive and consistent coverage of both high-level task objectives and fine-grained hand-object interactions, providing rich motion data for downstream modeling and analysis.

## 6 Experiments

In this section, we first introduce our experimental setups in Section 6.1, including details of the implementation and evaluation. To validate Being-HO, we demonstrate our principal comparison on hand motion generation and translation tasks in Section 6.2 and on long-range motion generation in Section 6.3, and carry out ablation studies in Section 6.4 and robot experiments in Section 6.6.

---

### 6.1 Experimental Setup

#### 6.1.1 Implementation Details

We encode motion sequences using a temporal downsampling ratio $\alpha = 4$, padding them with zeros to ensure that the length is a multiple of $\alpha$ to prevent information loss. Our part-level motion tokenizer employs an 8-layer Grouped Residual Quantization (GRQ) architecture with a default group size $n = 2$ by default, converting each one-second motion sequence into $2 \times n \times L \times \lceil T/\alpha \rceil = 128$ tokens. We use codebook sizes $K_w = K_f = 4096$ per part, shared across all RQ layers, with a code dimension of $d = 512$. The tokenizer is optimized with a batch size of 2048 and a learning rate of $2 \times 10^{-4}$. The loss function is weighted as $\lambda_1 = 0.02$, $\lambda_2 = 1.0$. For multimodal sequence modeling, we consider three scales of InternVL3 (1B, 8B, and 14B) as the backbone, trained on UniHand-2.5M. Each training instance includes a scene image scaled as $448 \times 448$ and a corresponding hand motion sequence aligned with the image under camera coordinates. Hand poses are represented using MANO-D162, sampled at 15 FPS and discretized into 128 motion tokens per hand per second via our tokenizer. During pertaining, the vocabulary-level logit masking is conducted with probability $\mathcal{P} = 50\%$, and the token-level loss is filtered within [15%, 95%] percentile. We use AdamW with a learning rate of $1 \times 10^{-5}$, a batch size of 128, and training on 32 NVIDIA A800-80G GPUs, jointly fine-tuning both the ViT adapter and LLM backbone.

#### 6.1.2 Evaluation on Hand Motion Generation

We conduct experiments on this benchmark to validate the effectiveness of our pretrained foundation VLA. Below, we describe the datasets, benchmarking tasks, and evaluation metrics used in our study.

Dataset Setups. For evaluation, we reserve 5% of videos along with their paired hand motion and textual annotations from sources in UniHand. Notably, the distributions of wrist translation vary across sources, with EgoDex contributing the majority of samples. To balance the overall distribution, we augment the other sources to expand their coverage in the translation space. As a result, EgoDex remains the central mode of the distribution, while the other sources form a broader yet sparser long-tail region. To reflect this structure, we evaluate the model on two distinct splits: the held-out EgoDex samples (referred to as the “head split”) and the combined samples from TACO, HOI4D, H2O, and OakInk2 (referred to as the “tail split”). This enables a systematic assessment of the model’s ability to capture dominant motion patterns while generalizing to less frequent motion contexts in our multi-source data.

Task Definition. To comprehensively evaluate our foundation VLA in grounding 2D visual cues to 3D hand motion, maintaining temporal motion coherence, and aligning vision-language-motion semantics, we design three complementary tasks.

- Visual-Grounded Hand Motion Generation. The model takes as input a static scene image, a textual instruction, and a specified duration, then directly generates a semantically aligned hand motion sequence in absolute 3D coordinates that matches the image. To accomplish this, the model must accurately establish an absolute spatial mapping from 2D to 3D while converting the instruction semantics into temporally coherent motion dynamics.

- Contextualized Hand Motion Generation. The model receives a scene image, a short segment of ground-truth hand motion as context, and a follow-up textual instruction to predict the subsequent hand motion sequence. By incorporating the motion context, this task eliminates ambiguity in the initial 2D-to-3D mapping, allowing the evaluation to focus on whether the generated motion remains semantically aligned with the instruction while preserving coherence with the provided context.

- Hand Motion Translation. The model is prompted with a scene image and a hand motion sequence to generate a textual description of the motion. This inverse task evaluates the model's ability to interpret hand motion semantics and accurately translate 3D motion back into linguistically grounded descriptions.

These tasks together constitute a holistic benchmark for assessing vision-language-motion grounding from both generation and understanding perspectives.

Evaluation Protocol. To comprehensively evaluate the quality of generated motions, we analyze three

---

key aspects: spatial accuracy, temporal coherence, and semantic consistency. For hand generation tasks, we employ five primary metrics to jointly assess spatial accuracy and semantic alignment of the generated hand motions:

- MPJPE (Mean Per Joint Position Error) measures overall spatial accuracy by computing the mean Euclidean distance between each generated joint and its ground-truth position in absolute 3D space.

- MWTE (Mean Wrist Translation Error) evaluates global trajectory fidelity through the mean Euclidean distance between predicted and ground-truth wrist positions across the sequence.

- PA-MPJPE (Procrustes Aligned MPJPE) isolates relative pose accuracy by aligning predicted joints to the ground truth via rigid transformation (including scaling, rotation, and translation).

- M2T R@3 (Motion-to-Text Retrieval Top-3 Accuracy) assesses semantic alignment by embedding generated motion into a shared representation space and retrieving the top-3 matching textual descriptions using a dataset-specific text-motion retrieval model (TMR [^158]).

• FID (Fréchet Inception Distance) quantifies the distribution similarity by comparing the generated and real motion embeddings in the dataset-specific latent space of the TMR model, measuring how well synthesized motions match the true data distribution.

For the hand motion translation task, we evaluate using a single retrieval metric: T2M R@3 (Text-to-Motion Retrieval Top-3 Accuracy), which reports how effectively the generated textual description retrieves its corresponding motion sequence from a database. This metric directly verifies whether the model's text output accurately captures the semantic content of the motion. We complement this with the valid generation rate for free-form generation, which reflects how consistently the model produces motion sequences adhering to the required structural format. Together, these metrics offer a holistic assessment of the model's capabilities to generate physically plausible, temporally coherent hand motions that remain faithful to the given instructions.

#### 6.1.3 Evaluation on Dexterous Manipulation

As intended, our Being-HO is able to execute dexterous manipulation tasks after physical instruction tuning. To assess the transferability of hand motion knowledge acquired from UniHand to downstream tasks, we conduct real-robot experiments for quantitative evaluation.

Robot System. Our experiments are conducted using a hardware setup including a 7-DoF Franka Research 3 arm, a 6-DoF Inspire hand, and a RealSense L515 camera for RGB streaming. To collect demonstrations for imitation learning, we introduce an improved teleoperation system that integrates the Gello exoskeleton [^159] for arm control with a RealSense D435i camera for hand pose estimation and retargeting [^160] [^161], as shown in Figure 7. The evaluation covers a diverse set of manipulation tasks, including grasp-and-place skills and interactions with both articulated and deformable objects. For each task, we collect 50–100 teleoperation trajectories to post-train our foundation VLA. At each step, the policy takes as input an egocentric RGB image and robot proprioception to generate an action chunk consisting of end-effector poses and dexterous hand joint positions.

![Image](images/89ee44ab9a2ea75d9d4fa2d26230e59dd875762bccd2157dd66512f12e65d03f.jpg)

<div style="text-align: center;">Figure 7: The hardware system.</div>

Evaluation Tasks. We design a suite of real-world manipulation tasks that assess fundamental skills while also challenging its generalization and precision in complex scenarios.

- Grasping and Placing (Pick-Place-Toy). The task includes three sub-scenarios. The Seen scenario assesses basic capabilities by requiring the robot to pick up a toy seen during post-training. To test object generalization, the Unseen scenario introduces a novel toy with visual properties (e.g., color). Finally,

---

the Clutter scenario evaluates advanced scene understanding by challenging the robot to identify and retrieve the target toy from among multiple distractors in a cluttered environment.

• Articulated Object Manipulation (Close-Toolbox, Close-Lid). The robot must perform delicate closure actions on a toolbox and a cup lid. These tasks rigorously assess the model's capability for accurate end-effector positioning and orientation, as well as stable interaction with object mechanics.

- Deformable Object Manipulation (Unfold-Clothes). The robot is required to unfold a piece of cloth, testing the model's ability to perform multi-finger fine-grained manipulation and dynamically alter the state of non-rigid objects.

- Precise Motion Control (Pour-Cup). In this task, the robot is asked to pour liquid from one cup to another, requiring the generation of smooth, stable motion trajectories to maintain temporal action coherence throughout the actions.

Evaluation Protocol. We evaluate dexterous manipulation performance using success rate as our primary metric, where task success follows strict binary criteria (e.g., complete lid closure, accurate toy placement). Each task undergoes 20 randomized trials with varied initial object positions to ensure statistical robustness. Our Being-HO is benchmarked against two baselines: GR00T N1.5 [^23] and InternVL3 [^57]. We select GR00T because it is the only large-scale VLA model pre-trained on egocentric human videos with a focus on dexterous manipulation. This stands in contrast to most other models, such as OpenVLA [^5], which are designed for grippers. Meanwhile, InternVL3 shares Being-HO's architecture and scale, but it lacks hand motion pretraining and physical alignment. All models receive identical post-training on the same teleoperation datasets to assess the benefits of pre-training on egocentric hand datasets. To better understand model capabilities and failure modes, we conduct qualitative analysis to perform model behavior analysis across three dimensions: motion precision (e.g., Close-Lid), semantic understanding of instructions (e.g., Pick the white duck), and robustness in complex tasks (e.g., Unfold-Clothes).

### 6.2 Comparisons on Hand Motion Generation and Translation

To begin with, we set the prediction horizon as one second for all experiments in this section. Our evaluation first examines the model's capability to produce motion sequences that adhere to the required structural format (<MOT>.....</MOT>)。如表2所示，基于自由格式模式（第4.1.5节）开展的视觉引导运动生成任务实验表明，不同模型规模在有效生成率上存在显著差异。虽然Being-HO-1B在保留运动块结构方面仅取得一般效果，但性能更强的Being-HO-8B和Being-HO-14B的有效率分别达到了99.8%和100%，这证明模型规模的扩大能显著改善对结构化运动格式的学习能力。

<div style="text-align: center;">Table 2: Comparison results on two tasks: visual-grounded motion generation and hand motion translation, where we use valid rate (%) and T2M R@3 (%) as the metrics.</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td rowspan="2">Model</td><td rowspan="2">Valid Rate $\uparrow$</td><td colspan="2">T2M R@3 $\uparrow$</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>head</td><td style='text-align: center; word-wrap: break-word;'>tail</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>ground truth</td><td style='text-align: center; word-wrap: break-word;'>-</td><td style='text-align: center; word-wrap: break-word;'>33.5</td><td style='text-align: center; word-wrap: break-word;'>42.7</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Being-H0-1B</td><td style='text-align: center; word-wrap: break-word;'>64.8</td><td style='text-align: center; word-wrap: break-word;'>12.5</td><td style='text-align: center; word-wrap: break-word;'>14.3</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Being-H0-8B</td><td style='text-align: center; word-wrap: break-word;'>99.8</td><td style='text-align: center; word-wrap: break-word;'>18.4</td><td style='text-align: center; word-wrap: break-word;'>19.7</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Being-H0-14B</td><td style='text-align: center; word-wrap: break-word;'>100.0</td><td style='text-align: center; word-wrap: break-word;'>19.0</td><td style='text-align: center; word-wrap: break-word;'>22.1</td></tr></table>

表2还通过手部运动翻译任务评估了模型的运动理解能力。我们采用T2M R@3指标，分别在头部和尾部数据集上测量生成的文本描述能够准确匹配相应运动序列的程度，以此验证模型文本输出的语义保真度。结果显示，规模更大的模型始终能获得更高的T2M R@3分数，证实了它们在运动与语言模态之间更强的双向对齐能力。

我们在表3中报告了视觉引导和语境化手部运动生成任务的主要定量结果。为了提供一个具有竞争力且可推广的基准，我们选用GR00T N1.5 [^23]作为基准模型。该模型最初是在多模态机器人及人类数据集上训练的，我们参照MEgoHand [^118]中的类似流程对其进行了适配。具体而言，我们将其动作表示重新定义为双手运动序列；当需要单手运动序列时，另一只手会用零填充。在语境化手部运动生成中，初始手部姿态会被用作动作头的输入状态，

---

<div style="text-align: center;">Table 3: Comparison results of visual-grounded and contextualized hand motion generation tasks on both head and tail splits, whose samples come from {EgoDex} and {TACO, HOI4D, H2O, OakInk2}, respectively. MPJPE, MWTE, and PA-MPJPE are reported in centimeters (cm), while the M2T R@3 is reported in percentage (%), which stays consistent in the rest of the article.</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td rowspan="2">Model</td><td colspan="2">MPJPE $\downarrow$</td><td colspan="2">MWTE $\downarrow$</td><td colspan="2">PA-MPJPE $\downarrow$</td><td colspan="2">M2T R@3 $\uparrow$</td><td colspan="2">FID $\downarrow$</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>head</td><td style='text-align: center; word-wrap: break-word;'>tail</td><td style='text-align: center; word-wrap: break-word;'>head</td><td style='text-align: center; word-wrap: break-word;'>tail</td><td style='text-align: center; word-wrap: break-word;'>head</td><td style='text-align: center; word-wrap: break-word;'>tail</td><td style='text-align: center; word-wrap: break-word;'>head</td><td style='text-align: center; word-wrap: break-word;'>tail</td><td style='text-align: center; word-wrap: break-word;'>head</td><td style='text-align: center; word-wrap: break-word;'>tail</td></tr><tr><td colspan="11"># Visual-Grounded Hand Motion Generation</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>GR00T N1.5 [^23]</td><td style='text-align: center; word-wrap: break-word;'>9.82</td><td style='text-align: center; word-wrap: break-word;'>15.35</td><td style='text-align: center; word-wrap: break-word;'>8.51</td><td style='text-align: center; word-wrap: break-word;'>11.20</td><td style='text-align: center; word-wrap: break-word;'>1.33</td><td style='text-align: center; word-wrap: break-word;'>1.41</td><td style='text-align: center; word-wrap: break-word;'>13.1</td><td style='text-align: center; word-wrap: break-word;'>14.8</td><td style='text-align: center; word-wrap: break-word;'>11.7</td><td style='text-align: center; word-wrap: break-word;'>14.4</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Being-H0-1B</td><td style='text-align: center; word-wrap: break-word;'>9.71</td><td style='text-align: center; word-wrap: break-word;'>17.21</td><td style='text-align: center; word-wrap: break-word;'>8.25</td><td style='text-align: center; word-wrap: break-word;'>12.04</td><td style='text-align: center; word-wrap: break-word;'>1.50</td><td style='text-align: center; word-wrap: break-word;'>1.55</td><td style='text-align: center; word-wrap: break-word;'>12.1</td><td style='text-align: center; word-wrap: break-word;'>15.3</td><td style='text-align: center; word-wrap: break-word;'>12.2</td><td style='text-align: center; word-wrap: break-word;'>13.1</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Being-H0-8B</td><td style='text-align: center; word-wrap: break-word;'>7.20</td><td style='text-align: center; word-wrap: break-word;'>9.02</td><td style='text-align: center; word-wrap: break-word;'>5.69</td><td style='text-align: center; word-wrap: break-word;'>8.11</td><td style='text-align: center; word-wrap: break-word;'>1.09</td><td style='text-align: center; word-wrap: break-word;'>1.32</td><td style='text-align: center; word-wrap: break-word;'>15.9</td><td style='text-align: center; word-wrap: break-word;'>18.7</td><td style='text-align: center; word-wrap: break-word;'>11.5</td><td style='text-align: center; word-wrap: break-word;'>13.4</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Being-H0-14B</td><td style='text-align: center; word-wrap: break-word;'>6.87</td><td style='text-align: center; word-wrap: break-word;'>8.11</td><td style='text-align: center; word-wrap: break-word;'>5.19</td><td style='text-align: center; word-wrap: break-word;'>7.41</td><td style='text-align: center; word-wrap: break-word;'>1.03</td><td style='text-align: center; word-wrap: break-word;'>1.20</td><td style='text-align: center; word-wrap: break-word;'>17.2</td><td style='text-align: center; word-wrap: break-word;'>20.5</td><td style='text-align: center; word-wrap: break-word;'>10.3</td><td style='text-align: center; word-wrap: break-word;'>11.8</td></tr><tr><td colspan="11"># Contextualized Hand Motion Generation</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>GR00T N1.5 [^23]</td><td style='text-align: center; word-wrap: break-word;'>7.14</td><td style='text-align: center; word-wrap: break-word;'>8.55</td><td style='text-align: center; word-wrap: break-word;'>6.65</td><td style='text-align: center; word-wrap: break-word;'>7.93</td><td style='text-align: center; word-wrap: break-word;'>1.11</td><td style='text-align: center; word-wrap: break-word;'>1.25</td><td style='text-align: center; word-wrap: break-word;'>16.1</td><td style='text-align: center; word-wrap: break-word;'>20.5</td><td style='text-align: center; word-wrap: break-word;'>11.2</td><td style='text-align: center; word-wrap: break-word;'>13.3</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Being-H0-1B</td><td style='text-align: center; word-wrap: break-word;'>8.73</td><td style='text-align: center; word-wrap: break-word;'>11.34</td><td style='text-align: center; word-wrap: break-word;'>7.88</td><td style='text-align: center; word-wrap: break-word;'>10.67</td><td style='text-align: center; word-wrap: break-word;'>1.17</td><td style='text-align: center; word-wrap: break-word;'>1.38</td><td style='text-align: center; word-wrap: break-word;'>15.4</td><td style='text-align: center; word-wrap: break-word;'>20.6</td><td style='text-align: center; word-wrap: break-word;'>14.7</td><td style='text-align: center; word-wrap: break-word;'>15.6</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Being-H0-8B</td><td style='text-align: center; word-wrap: break-word;'>6.67</td><td style='text-align: center; word-wrap: break-word;'>7.98</td><td style='text-align: center; word-wrap: break-word;'>5.03</td><td style='text-align: center; word-wrap: break-word;'>6.93</td><td style='text-align: center; word-wrap: break-word;'>0.90</td><td style='text-align: center; word-wrap: break-word;'>1.03</td><td style='text-align: center; word-wrap: break-word;'>19.7</td><td style='text-align: center; word-wrap: break-word;'>21.4</td><td style='text-align: center; word-wrap: break-word;'>10.1</td><td style='text-align: center; word-wrap: break-word;'>11.7</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Being-H0-14B</td><td style='text-align: center; word-wrap: break-word;'>6.21</td><td style='text-align: center; word-wrap: break-word;'>7.33</td><td style='text-align: center; word-wrap: break-word;'>4.89</td><td style='text-align: center; word-wrap: break-word;'>6.52</td><td style='text-align: center; word-wrap: break-word;'>0.92</td><td style='text-align: center; word-wrap: break-word;'>1.04</td><td style='text-align: center; word-wrap: break-word;'>20.1</td><td style='text-align: center; word-wrap: break-word;'>23.5</td><td style='text-align: center; word-wrap: break-word;'>9.8</td><td style='text-align: center; word-wrap: break-word;'>10.1</td></tr></table>

而在视觉引导的手部运动生成中则采用固定状态。为了确保不同模型规模和基准模型之间的评估一致性，我们采用了块格式化模式（第4.1.5节），强制要求生成的序列必须解码为正确的格式。我们的分析得出了三个关键结论。首先，规模更大的模型始终表现出更优的性能，其MPJPE、MWTE和PA-MPJPE分数更低，反映出更强的空间定位能力和更合理的姿态生成效果。其次，它们在M2T R@3和FID指标上的表现明显更好，说明生成的运动与输入指令之间的语义一致性更强。值得注意的是，这种性能优势在尾部数据集上尤为显著，表明模型规模的扩大能大幅提升其对各类运动分布的泛化能力。

#### 6.3 长距离运动生成对比

我们通过系统性的对比来评估不同Being-HO变体的长期运动生成能力，如表4所示。较长的运动序列本身容易因累积的预测误差而导致轨迹偏移或姿态质量下降。为缓解这一问题，我们在推理阶段采用了软约束模式（第4.1.5节），将生成的运动限制在与真实分布相符的合理范围内。为了进行全面分析，我们将结果分为短期（2–5秒）和长期（6–10秒）两类，以便精确考察随着序列长度增加而产生的质量退化情况。我们的评估采用MPJPE、MWTE和PA-MPJPE作为衡量空间精度和轨迹稳定性的核心指标。结果表明，随着序列长度的增加，生成质量自然会下降，这从长期范围内较高的MPJPE和MWTE数值中可见一斑，其根源在于误差的累积。不过，在软约束条件下，规模较大的模型仍能保持更稳定的空间精度，因为它们能借助部分真实上下文来锚定轨迹。

#### 6.4 消融研究

我们开展了系统的消融研究，以探究架构决策和训练协议对模型生成空间合理且语义连贯的手部运动能力的影响。这些研究聚焦于两项关键任务：（1）用于运动分词器的手部重建，旨在评估模型准确复现输入运动序列的能力；（2）面向基础视觉-语言-动作模型的视觉引导运动生成。

### 6.4.1 手部运动分词的最佳实践

局部级别与其他方式。我们在表5中的对比评估显示了局部级别分词相较于对整个手部进行统一量化的方法的优势。我们以两种GRQ变体作为基准进行测试：

---

<div style="text-align: center;">Table 4: Comparison results of hand motion generation tasks upon long-range sequences. We adopt the soft-formatted mode (Section 4.1.5) and report short-term (2–5s) and long-term (6–10s) results.</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td rowspan="3">Model</td><td colspan="6">Short-Term (2-5s)</td><td colspan="6">Long-Term (6-10s)</td></tr><tr><td colspan="2">MPJPE $\downarrow$</td><td colspan="2">MWTE $\downarrow$</td><td colspan="2">PA-MPJPE $\downarrow$</td><td colspan="2">MPJPE $\downarrow$</td><td colspan="2">MWTE $\downarrow$</td><td colspan="2">PA-MPJPE $\downarrow$</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>head</td><td style='text-align: center; word-wrap: break-word;'>tail</td><td style='text-align: center; word-wrap: break-word;'>head</td><td style='text-align: center; word-wrap: break-word;'>tail</td><td style='text-align: center; word-wrap: break-word;'>head</td><td style='text-align: center; word-wrap: break-word;'>tail</td><td style='text-align: center; word-wrap: break-word;'>head</td><td style='text-align: center; word-wrap: break-word;'>tail</td><td style='text-align: center; word-wrap: break-word;'>head</td><td style='text-align: center; word-wrap: break-word;'>tail</td><td style='text-align: center; word-wrap: break-word;'>head</td><td style='text-align: center; word-wrap: break-word;'>tail</td></tr><tr><td colspan="13"># Visual-Grounded Hand Motion Generation</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Being-H0-1B</td><td style='text-align: center; word-wrap: break-word;'>8.97</td><td style='text-align: center; word-wrap: break-word;'>9.96</td><td style='text-align: center; word-wrap: break-word;'>7.01</td><td style='text-align: center; word-wrap: break-word;'>8.75</td><td style='text-align: center; word-wrap: break-word;'>1.43</td><td style='text-align: center; word-wrap: break-word;'>1.67</td><td style='text-align: center; word-wrap: break-word;'>9.12</td><td style='text-align: center; word-wrap: break-word;'>11.24</td><td style='text-align: center; word-wrap: break-word;'>7.13</td><td style='text-align: center; word-wrap: break-word;'>9.91</td><td style='text-align: center; word-wrap: break-word;'>1.60</td><td style='text-align: center; word-wrap: break-word;'>1.81</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Being-H0-8B</td><td style='text-align: center; word-wrap: break-word;'>7.55</td><td style='text-align: center; word-wrap: break-word;'>8.45</td><td style='text-align: center; word-wrap: break-word;'>5.78</td><td style='text-align: center; word-wrap: break-word;'>7.51</td><td style='text-align: center; word-wrap: break-word;'>1.10</td><td style='text-align: center; word-wrap: break-word;'>1.30</td><td style='text-align: center; word-wrap: break-word;'>8.21</td><td style='text-align: center; word-wrap: break-word;'>9.98</td><td style='text-align: center; word-wrap: break-word;'>6.12</td><td style='text-align: center; word-wrap: break-word;'>8.34</td><td style='text-align: center; word-wrap: break-word;'>1.22</td><td style='text-align: center; word-wrap: break-word;'>1.36</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Being-H0-14B</td><td style='text-align: center; word-wrap: break-word;'>7.43</td><td style='text-align: center; word-wrap: break-word;'>8.39</td><td style='text-align: center; word-wrap: break-word;'>5.65</td><td style='text-align: center; word-wrap: break-word;'>7.39</td><td style='text-align: center; word-wrap: break-word;'>1.11</td><td style='text-align: center; word-wrap: break-word;'>1.28</td><td style='text-align: center; word-wrap: break-word;'>7.98</td><td style='text-align: center; word-wrap: break-word;'>9.72</td><td style='text-align: center; word-wrap: break-word;'>5.88</td><td style='text-align: center; word-wrap: break-word;'>8.01</td><td style='text-align: center; word-wrap: break-word;'>1.18</td><td style='text-align: center; word-wrap: break-word;'>1.32</td></tr><tr><td colspan="13"># Contextualized Hand Motion Generation</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Being-H0-1B</td><td style='text-align: center; word-wrap: break-word;'>8.44</td><td style='text-align: center; word-wrap: break-word;'>9.52</td><td style='text-align: center; word-wrap: break-word;'>6.71</td><td style='text-align: center; word-wrap: break-word;'>7.99</td><td style='text-align: center; word-wrap: break-word;'>1.20</td><td style='text-align: center; word-wrap: break-word;'>1.45</td><td style='text-align: center; word-wrap: break-word;'>9.01</td><td style='text-align: center; word-wrap: break-word;'>10.98</td><td style='text-align: center; word-wrap: break-word;'>6.98</td><td style='text-align: center; word-wrap: break-word;'>8.75</td><td style='text-align: center; word-wrap: break-word;'>1.35</td><td style='text-align: center; word-wrap: break-word;'>1.50</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Being-H0-8B</td><td style='text-align: center; word-wrap: break-word;'>7.67</td><td style='text-align: center; word-wrap: break-word;'>8.20</td><td style='text-align: center; word-wrap: break-word;'>5.81</td><td style='text-align: center; word-wrap: break-word;'>7.13</td><td style='text-align: center; word-wrap: break-word;'>1.01</td><td style='text-align: center; word-wrap: break-word;'>1.22</td><td style='text-align: center; word-wrap: break-word;'>8.23</td><td style='text-align: center; word-wrap: break-word;'>9.67</td><td style='text-align: center; word-wrap: break-word;'>6.23</td><td style='text-align: center; word-wrap: break-word;'>7.83</td><td style='text-align: center; word-wrap: break-word;'>1.14</td><td style='text-align: center; word-wrap: break-word;'>1.27</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Being-H0-14B</td><td style='text-align: center; word-wrap: break-word;'>7.39</td><td style='text-align: center; word-wrap: break-word;'>8.51</td><td style='text-align: center; word-wrap: break-word;'>5.77</td><td style='text-align: center; word-wrap: break-word;'>7.21</td><td style='text-align: center; word-wrap: break-word;'>1.05</td><td style='text-align: center; word-wrap: break-word;'>1.25</td><td style='text-align: center; word-wrap: break-word;'>8.01</td><td style='text-align: center; word-wrap: break-word;'>9.45</td><td style='text-align: center; word-wrap: break-word;'>6.02</td><td style='text-align: center; word-wrap: break-word;'>7.67</td><td style='text-align: center; word-wrap: break-word;'>1.18</td><td style='text-align: center; word-wrap: break-word;'>1.30</td></tr></table>

一种是4组配置（n = 4），另一种是16层残差量化架构（L = 16），两者均采用相同的码本大小（K = K_w + K_f = 8192）以确保实验严谨性；所有变体均遵循每秒固定词元数和一致的码本维度要求。所有评估均在UniHand的预留测试集上进行，该测试集仅包含训练数据中未出现过的运动序列。局部级别分词器表现出更优的性能，证明了对手腕和手指分别进行分词以建模手部运动的有效性。

MANO-D162与其他方式。基于我们的分词框架（第4.1.2节），我们通过GRQ重构来分析不同的运动特征。轴角表示法在整体重构精度上表现更优（MANO-D51 vs. MANO-D99；MANO-D114 vs. MANO-D162），而6D旋转表示法则能获得更好的PA-MPJPE分数，显示出其在建模手指动作方面的潜在优势。如表6所示，基于6D旋转的MANO-D162特征对Being-HO训练最为有效。我们进一步发现，加入辅助关节位置（j）能够提升性能，而建模手部形状参数（$\beta$）反而会导致结果变差。因此，我们保留了每个序列初始帧中的恒定形状参数，使分词器能够专注于运动动态。

<div style="text-align: center;">Table 5: Performance of different motion tokenization practices on the hand reconstruction task, including motion features and part-level tokenizing. Results are reported in centimeters (cm).</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td rowspan="2">Feature</td><td colspan="2">Part-Level</td><td colspan="2">4-Groups</td><td colspan="2">16-Layers</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>MPJPE $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>PA-MPJPE $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>MPJPE $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>PA-MPJPE $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>MPJPE $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>PA-MPJPE $\downarrow$</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>MANO-D51</td><td style='text-align: center; word-wrap: break-word;'>0.556</td><td style='text-align: center; word-wrap: break-word;'>0.209</td><td style='text-align: center; word-wrap: break-word;'>1.165</td><td style='text-align: center; word-wrap: break-word;'>0.184</td><td style='text-align: center; word-wrap: break-word;'>1.466</td><td style='text-align: center; word-wrap: break-word;'>0.243</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>MANO-D99</td><td style='text-align: center; word-wrap: break-word;'>0.584</td><td style='text-align: center; word-wrap: break-word;'>0.149</td><td style='text-align: center; word-wrap: break-word;'>1.093</td><td style='text-align: center; word-wrap: break-word;'>0.148</td><td style='text-align: center; word-wrap: break-word;'>1.510</td><td style='text-align: center; word-wrap: break-word;'>0.170</td></tr><tr><td colspan="7"># with shape parameters $\beta$</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>MANO-D109</td><td style='text-align: center; word-wrap: break-word;'>0.592</td><td style='text-align: center; word-wrap: break-word;'>0.160</td><td style='text-align: center; word-wrap: break-word;'>1.107</td><td style='text-align: center; word-wrap: break-word;'>0.140</td><td style='text-align: center; word-wrap: break-word;'>1.602</td><td style='text-align: center; word-wrap: break-word;'>0.201</td></tr><tr><td colspan="7"># with auxiliary joint positions  j</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>MANO-D114</td><td style='text-align: center; word-wrap: break-word;'>0.523</td><td style='text-align: center; word-wrap: break-word;'>0.167</td><td style='text-align: center; word-wrap: break-word;'>0.810</td><td style='text-align: center; word-wrap: break-word;'>0.202</td><td style='text-align: center; word-wrap: break-word;'>0.996</td><td style='text-align: center; word-wrap: break-word;'>0.253</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>MANO-D162</td><td style='text-align: center; word-wrap: break-word;'>0.573</td><td style='text-align: center; word-wrap: break-word;'>0.129</td><td style='text-align: center; word-wrap: break-word;'>0.704</td><td style='text-align: center; word-wrap: break-word;'>0.138</td><td style='text-align: center; word-wrap: break-word;'>1.054</td><td style='text-align: center; word-wrap: break-word;'>0.226</td></tr></table>

对手部运动生成的影响。我们采用MANO-D162 + Part-Level配置作为训练Being-HO的默认运动分词器。为了验证其有效性，我们还使用三种GRQ重建误差较低的替代配置来训练模型，分别是MANO-D114 + 4-Groups、MANO-D162 + 4-Groups和MANO-D114 + Part-Level。如表6所示，尽管MANO-D162在重建时的MPJPE略高（见表5），但它在手部运动生成任务中的表现始终优于其他配置。我们将此归因于6D旋转表示和部件级分解，这两者很可能有助于更好的时序建模以及细粒度手部运动的自回归生成。

---

<div style="text-align: center;">Table 6: Ablation on different motion tokenizer variants and data recipes for visual-grounded hand motion generation. Here “Trans” and “Context” denote the task types of motion translation, while “Balance” represents view-invariant motion distribution balancing.</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td rowspan="2">Variants</td><td colspan="2">MPJPE $\downarrow$</td><td colspan="2">MWTE $\downarrow$</td><td colspan="2">PA-MPJPE $\downarrow$</td><td colspan="2">M2T R@3 $\uparrow$</td><td colspan="2">FID $\downarrow$</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>head</td><td style='text-align: center; word-wrap: break-word;'>tail</td><td style='text-align: center; word-wrap: break-word;'>head</td><td style='text-align: center; word-wrap: break-word;'>tail</td><td style='text-align: center; word-wrap: break-word;'>head</td><td style='text-align: center; word-wrap: break-word;'>tail</td><td style='text-align: center; word-wrap: break-word;'>head</td><td style='text-align: center; word-wrap: break-word;'>tail</td><td style='text-align: center; word-wrap: break-word;'>head</td><td style='text-align: center; word-wrap: break-word;'>tail</td></tr><tr><td style='text-align: center; word-wrap: break-word;'># Base</td><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'></td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Being-H0-8B</td><td style='text-align: center; word-wrap: break-word;'>7.20</td><td style='text-align: center; word-wrap: break-word;'>9.02</td><td style='text-align: center; word-wrap: break-word;'>5.69</td><td style='text-align: center; word-wrap: break-word;'>8.11</td><td style='text-align: center; word-wrap: break-word;'>1.09</td><td style='text-align: center; word-wrap: break-word;'>1.32</td><td style='text-align: center; word-wrap: break-word;'>15.9</td><td style='text-align: center; word-wrap: break-word;'>18.7</td><td style='text-align: center; word-wrap: break-word;'>11.5</td><td style='text-align: center; word-wrap: break-word;'>13.4</td></tr><tr><td style='text-align: center; word-wrap: break-word;'># Tokenizer Variants</td><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'></td></tr><tr><td style='text-align: center; word-wrap: break-word;'>MANO-D114 + 4-Groups</td><td style='text-align: center; word-wrap: break-word;'>8.31</td><td style='text-align: center; word-wrap: break-word;'>10.35</td><td style='text-align: center; word-wrap: break-word;'>6.52</td><td style='text-align: center; word-wrap: break-word;'>9.11</td><td style='text-align: center; word-wrap: break-word;'>1.14</td><td style='text-align: center; word-wrap: break-word;'>1.35</td><td style='text-align: center; word-wrap: break-word;'>13.1</td><td style='text-align: center; word-wrap: break-word;'>14.7</td><td style='text-align: center; word-wrap: break-word;'>13.7</td><td style='text-align: center; word-wrap: break-word;'>15.6</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>MANO-D162 + 4-Groups</td><td style='text-align: center; word-wrap: break-word;'>7.98</td><td style='text-align: center; word-wrap: break-word;'>9.71</td><td style='text-align: center; word-wrap: break-word;'>5.58</td><td style='text-align: center; word-wrap: break-word;'>8.98</td><td style='text-align: center; word-wrap: break-word;'>1.09</td><td style='text-align: center; word-wrap: break-word;'>1.38</td><td style='text-align: center; word-wrap: break-word;'>15.4</td><td style='text-align: center; word-wrap: break-word;'>17.1</td><td style='text-align: center; word-wrap: break-word;'>11.9</td><td style='text-align: center; word-wrap: break-word;'>14.3</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>MANO-D114 + Part-Level</td><td style='text-align: center; word-wrap: break-word;'>7.74</td><td style='text-align: center; word-wrap: break-word;'>9.92</td><td style='text-align: center; word-wrap: break-word;'>6.11</td><td style='text-align: center; word-wrap: break-word;'>8.83</td><td style='text-align: center; word-wrap: break-word;'>1.16</td><td style='text-align: center; word-wrap: break-word;'>1.41</td><td style='text-align: center; word-wrap: break-word;'>12.3</td><td style='text-align: center; word-wrap: break-word;'>16.1</td><td style='text-align: center; word-wrap: break-word;'>13.1</td><td style='text-align: center; word-wrap: break-word;'>12.3</td></tr><tr><td style='text-align: center; word-wrap: break-word;'># Data Recipe</td><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'></td></tr><tr><td style='text-align: center; word-wrap: break-word;'>w/o Translate</td><td style='text-align: center; word-wrap: break-word;'>7.22</td><td style='text-align: center; word-wrap: break-word;'>9.11</td><td style='text-align: center; word-wrap: break-word;'>5.51</td><td style='text-align: center; word-wrap: break-word;'>8.12</td><td style='text-align: center; word-wrap: break-word;'>1.27</td><td style='text-align: center; word-wrap: break-word;'>1.46</td><td style='text-align: center; word-wrap: break-word;'>13.2</td><td style='text-align: center; word-wrap: break-word;'>11.6</td><td style='text-align: center; word-wrap: break-word;'>13.1</td><td style='text-align: center; word-wrap: break-word;'>15.7</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>w/o Context</td><td style='text-align: center; word-wrap: break-word;'>8.01</td><td style='text-align: center; word-wrap: break-word;'>10.97</td><td style='text-align: center; word-wrap: break-word;'>6.34</td><td style='text-align: center; word-wrap: break-word;'>9.03</td><td style='text-align: center; word-wrap: break-word;'>1.11</td><td style='text-align: center; word-wrap: break-word;'>1.52</td><td style='text-align: center; word-wrap: break-word;'>13.7</td><td style='text-align: center; word-wrap: break-word;'>15.2</td><td style='text-align: center; word-wrap: break-word;'>11.1</td><td style='text-align: center; word-wrap: break-word;'>14.9</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>w/o Balance</td><td style='text-align: center; word-wrap: break-word;'>8.54</td><td style='text-align: center; word-wrap: break-word;'>12.13</td><td style='text-align: center; word-wrap: break-word;'>7.74</td><td style='text-align: center; word-wrap: break-word;'>10.04</td><td style='text-align: center; word-wrap: break-word;'>1.24</td><td style='text-align: center; word-wrap: break-word;'>1.57</td><td style='text-align: center; word-wrap: break-word;'>11.3</td><td style='text-align: center; word-wrap: break-word;'>10.3</td><td style='text-align: center; word-wrap: break-word;'>15.8</td><td style='text-align: center; word-wrap: break-word;'>16.3</td></tr></table>

#### 6.4.2 数据配置的影响

为了优化Being-HO的性能，我们构建的UniHand不仅包含标准的手部运动生成数据，还涵盖了多种指令类型以及视角不变的平衡分布（见第5.2节）。为了了解UniHand的不同组成部分对Being-HO性能的影响，我们系统分析了指令数据类型、分布平衡和数据规模所产生的效果。

视角不变的运动分布平衡如何提升泛化能力？如第4.1.5节所述，我们的数据平衡策略旨在使不同数据源中的运动视角覆盖趋于一致。该策略通过增强手部姿态来实现，同时保留一致的弱透视投影特性。我们从两个角度评估其有效性。一方面，我们考察其对运动分词器学习的影响。具体而言，我们在UniHand的预留测试集上比较了采用该策略与未采用该策略时的GRQ重建性能，该测试集包含训练期间未出现且经过姿态增强的运动序列。如图8所示，采用平衡策略后，所有数据集上的GRQ重建误差均显著降低，即便对于没有显式姿态增强的数据源（如EgoDex和Taste-Rob）也是如此。这表明分词器从更均匀分布的运动视角覆盖中受益，进而能够

![Image](images/70550451e4e80e8c2cd1383e91ad7a7a9d749e57608335a1ca0a489ca00b13a5.jpg)

<div style="text-align: center;">Figure 8: Ablation of view-invariant motion distribution balancing ("Balance") on motion reconstruction.</div>

更精确地编码全局腕部运动和细粒度的手指关节动作。另一方面，我们也评估了该策略对基础VLA学习的影响。我们将$\mathbf{Being-HO}$与在未进行数据平衡的数据集上训练的变体进行比较。如表6所示，去除该策略会导致尾部细分上的性能大幅下降。缺乏平衡机制时，模型容易过度拟合于占主导地位摄像头配置，从而限制其对代表性不足的运动视角组合的泛化能力。这些结果凸显了视角不变平衡在提升分词器表征鲁棒性，以及增强运动模型在多样化视角下生成准确且语义合理的运动方面的双重作用。

辅助监督任务是否有助于视觉引导的手部运动生成？除了基础的指令性运动生成数据外，我们的数据集还纳入了两种额外的监督数据：手部运动翻译和上下文运动预测。它们分别对应评估

---

![Image](images/fabe9e513ae76624cebe3ead0261c9c293b2817748ed94c11dba7b5ee31d01d1.jpg)

<div style="text-align: center;">Figure 9: The performance of Being-HO-8B alongside the increasing training data scale for visual-grounded hand motion generation. All metrics are normalized to make the last checkpoint (2.5M training samples) represent 100%, with higher values indicating better performance. The metrics include MPJPE, MWTE, PA-MPJPE, M2T R@3, and FID and are averaged on the core split and the tail split.</div>

手部运动翻译与情境化手部运动生成任务。除了实现这些功能外，我们还进行了消融实验，以评估这些数据类型对视觉引导的手部运动生成这一核心任务性能的影响。如表6所示，移除翻译监督后，全局手腕指标（MPJPE、MWTE）仅发生细微变化，但PA-MPJPE、M2T R@3和FID指标却明显恶化。这表明翻译数据有助于捕捉与语义目标一致的精细手部动作。相比之下，移除情境运动预测会导致所有指标全面下降。这凸显了其在生成时间连贯且具备情境感知的运动中的核心作用。这些发现表明，辅助监督不仅能实现特定任务所需的功能，还能通过更丰富的语义和时间基准来提升核心运动生成质量。

模型性能是否会因训练规模扩大而受益？我们进一步研究了性能随训练样本数量的变化情况。如图9所示，当训练样本量达到250万时，模型性能会随着训练规模的扩大而稳步提升，这体现了扩展多样化运动-语言数据的价值。值得注意的是，尽管在最大规模下PA-MPJPE略有下降，但M2T R@3等语义对齐指标却持续上升。一种可能的解释是，更大的数据量增加了任务-物体组合及运动语义的多样性，促使模型优先考虑语义合理性而非手指姿态细节的精确复现。这种权衡反映了随着训练数据日益丰富，模型对功能性和情境正确性的重视程度不断提高。

#### 6.5 手部运动生成样例

为了直观展示Being-HO生成物理上合理的手部运动的能力，我们在图10中展示了具有代表性的样例。每一行对应不同的任务指令，涵盖了多种场景和摄像头视角。每个模块显示了在相应摄像头坐标系下的首帧视频上渲染出的手部运动序列。渲染出的手部用黄色和蓝色标记，分别表示左手和右手。

为了在统一的弱透视投影下标准化可视化效果，我们对帧应用了变换策略，因此图像周围会出现黑色边框。有效的图像区域（不包括黑色填充部分）反映了手-物体交互深度的差异，因为距离更近的交互会显得更大。我们的模型能够成功处理单手和双手运动，涵盖各种操作任务，展现了其在不同视角和物理环境下的泛化能力。

---

![Image](images/ea09d8dabb0ccab627b6786c45e5bd868704cc372738dea5be405a5d84f58c4d.jpg)

<div style="text-align: center;">Figure 10: Hand Motion Generation Samples from Being-HO-8B across various tasks, scenes, and viewpoints. A simplified task instruction is given for each block. The three frames illustrate the generated hand motion over time, rendered in the first-frame camera coordinate system and overlaid on the RGB image. Black padding around each image is introduced to enforce consistent weak-perspective projection.</div>

#### 6.6 真实世界实验

为了验证人类视频预训练在现实世界中的实际效果，我们在表7所示的灵巧手操控任务上开展了实验。此处，“InternVL3”指的是使用相同的VLA架构和参数，但未经过我们的物理指令微调。Being-HO在所有任务中始终能取得最高的成功率，每项任务均进行了20次试验，且物体的初始位置是随机设定的。在Pick-Place-Toy任务中，虽然经过微调的GR00T N1.5 [^23]在领域内已见过的物体上表现相当，但面对未见过的物体和杂乱场景时，其泛化能力会显著下降。值得注意的是，GR00T采用隐式学习方法来预测潜在动作，而我们的Being-HO通过显式的运动标记化，即便在训练后数据量更少的情况下也展现出更强的泛化能力。例如，图11显示Being-HO不仅能成功抓取已见过的黄色鸭子，还能泛化到未见过的绿色鸭子。更值得一提的是，在有多重干扰物的杂乱环境中，Being-HO能准确遵循“抓取白色鸭子”这一指令，正确识别并取出目标白色鸭子。这凸显了其在视觉、语言与动作理解方面的强大整合能力。

<div style="text-align: center;">Table 7: Performance comparison: Success rates (%) of Being-HO versus baseline models on real-world dexterous manipulation tasks.</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td rowspan="2">Task</td><td colspan="3">Pick-Place-Toy</td><td rowspan="2">Close-Toolbox</td><td rowspan="2">Close-Lid</td><td rowspan="2">Pour-Cup</td><td rowspan="2">Unfold-Clothes</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Seen.</td><td style='text-align: center; word-wrap: break-word;'>Unseen.</td><td style='text-align: center; word-wrap: break-word;'>Clutter.</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>GROOT N1.5 [^23]</td><td style='text-align: center; word-wrap: break-word;'>0.75</td><td style='text-align: center; word-wrap: break-word;'>0.40</td><td style='text-align: center; word-wrap: break-word;'>0.50</td><td style='text-align: center; word-wrap: break-word;'>0.80</td><td style='text-align: center; word-wrap: break-word;'>0.50</td><td style='text-align: center; word-wrap: break-word;'>0.90</td><td style='text-align: center; word-wrap: break-word;'>0.60</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>InternVL3 [^57]</td><td style='text-align: center; word-wrap: break-word;'>0.55</td><td style='text-align: center; word-wrap: break-word;'>0.55</td><td style='text-align: center; word-wrap: break-word;'>0.50</td><td style='text-align: center; word-wrap: break-word;'>0.50</td><td style='text-align: center; word-wrap: break-word;'>0.25</td><td style='text-align: center; word-wrap: break-word;'>0.55</td><td style='text-align: center; word-wrap: break-word;'>0.45</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Being-H0</td><td style='text-align: center; word-wrap: break-word;'>0.75</td><td style='text-align: center; word-wrap: break-word;'>0.65</td><td style='text-align: center; word-wrap: break-word;'>0.60</td><td style='text-align: center; word-wrap: break-word;'>0.85</td><td style='text-align: center; word-wrap: break-word;'>0.60</td><td style='text-align: center; word-wrap: break-word;'>1.00</td><td style='text-align: center; word-wrap: break-word;'>0.75</td></tr></table>

Being-HO的优势在需要精细操控的任务中尤为明显。基线模型InternVL3由于没有经过物理指令微调，因而缺乏对手部运动的先验认知，其表现明显较弱。图12中的定性对比清晰地揭示了InternVL3的失败模式：

- 关闭工具箱：其运动轨迹不够精确，常常无法接触到工具箱盖的边缘，导致无法施加足够的力将其关闭。

- 关闭盖子：它会出现位置偏差，经常将盖子错放在杯子边缘旁而非正确就位。

---

![Image](images/675645d1c59527447d7cdf2dd2440fc77658de9cb4c0f01d3606ed321921284d.jpg)

<div style="text-align: center;">Figure 11: Being-HO performing Pick-Place-Toy on seen objects, unseen objects, and in cluttered scenes.</div>

- 倒水：它的抓握不稳定，有时无法牢固地握住杯子，从而影响后续倒水动作的稳定性。

- 展开衣物：它对手部操作高度的判断有误，导致手指在错误的高度闭合，无法触及衣物边缘，最终展开动作失败。

与之形成鲜明对比的是，Being-HO在这些任务中表现出稳健且精准的行为。它能准确地定位并关闭盖子，捏住衣物边缘以展开它，并能稳定地握住杯子以实现顺畅的倒水。这些成功结果凸显了Being-HO将物理指令微调所获得的精准手部运动生成知识有效迁移到下游机器人任务中的能力，使其能在复杂且精细的灵巧操控任务中表现出色。

数据效率。为了评估Being-HO的数据效率，我们考察了在不同数量的演示数据训练下其下游任务的表现。我们将预训练模型（Being-HO）与未预训练的基线模型（InternVL3）进行对比，在多个任务中分别使用25%、50%和100%的可用演示数据进行评估。如图13所示，结果表明无论数据规模如何，Being-HO始终比基线模型保持一致且显著的性能优势。这一结果有力地证明了我们的物理指令微调所带来的巨大益处。预训练期间获得的通用手部运动先验为模型提供了强大的起点，使其能够更快适应下游的灵巧操控任务。

Being-HO仅使用少量数据就能达到甚至超越基线模型的性能。例如，在Pick-Place-Toy任务中，仅用25%的数据训练的Being-HO，其表现与用100%数据训练的基线模型相当；而在Close-Toolbox和Unfold-Clothes任务中，仅用25%数据训练的Being-HO，其表现与用50%数据训练的基线模型相当。在更具挑战性的Close-Lid任务中，仅用25%数据训练的基线模型完全无法完成任务（成功率为0%），而Being-HO却能达到15%的成功率，这凸显出从零开始学习此类任务的难度。这种出色的数据效率对现实世界的机器人应用而言至关重要：它能大幅减少大规模、高质量遥操作演示的需求，进而降低数据采集的成本和时间，加速灵巧机器人在复杂现实场景中的部署。

---

![Image](images/bdd4ceb5a6b75d8da1bf1c91db7dd77b945c61d4840ed59e96fa65197a1e0336.jpg)

<div style="text-align: center;">Figure 12: Qualitative comparisons of Being-HO and the baseline (InternVL3).</div>

![Image](images/3d926ef88a87a5f639833a35ee8081faf86b96020c69a2ca284fdb10e3830aab.jpg)

<div style="text-align: center;">Figure 13: Comparison of data efficiency between Being-HO and the baseline without human hand pre-training (InternVL3). For each task, the horizontal axis shows the percentage of teleoperation data used for finetuning, and the vertical axis reports the task success rate of the learned policy.</div>

### 7 结论与未来展望

我们提出了Being-HO——一种先进、可扩展且样本效率高的灵巧VLA，它通过我们创新的物理指令调优范式，利用大规模人类视频进行训练，该范式包含预训练、物理空间对齐和后训练阶段。我们以人手作为基础操纵器，将人类视频中的灵巧性迁移到机器人学习中，从而解决了将LMM适配为VLA时出现的预训练数据与下游数据不匹配问题。我们统一构建的百万级数据集UniHand系统性地整合了多种数据源（动作捕捉数据、VR录制视频以及纯RGB视频），以克服真实机器人数据稀缺的问题。

我们的方法解决了从人类视频中学习灵巧操控的四个关键挑战。在预训练数据整理方面，我们通过MANO参数标准化和投影对齐，系统性地整合异构数据源。在手部量化方面，我们的分组残差量化技术能够实现毫米级的重建精度，同时还能与语言模型无缝集成，有效将动作视为另一种语言。在跨模态推理方面，我们将所有模态统一到自回归序列中，构建复杂的跨模态依赖关系，将视觉场景与操控策略联系起来，并将语言指令精准映射到手指动作上。在机器人控制迁移方面，尽管人手与机器人操纵器之间存在运动学差异，但我们的物理指令调优技术仍能充分利用预训练的多模态表征。

---

人手与机器人操纵器之间的运动学差异。

本研究为大规模从人类视频中学习机器人操控奠定了基础，未来有若干极具前景的发展方向。首先，进一步发展物理空间对齐技术，将有助于实现从人类演示到机器人控制的迁移（例如引入深度和触觉反馈，以提升操控能力和物理合理性）。其次，将$\text{Being-HO}$扩展到涉及工具使用、多物体交互以及长时序推理的复杂场景中，是一个令人兴奋的前沿领域。最后，将其与仿真环境和强化学习相结合，有望实现更稳健的策略学习，并更安全地在现实世界中部署。我们希望能为这一日益重要的领域的研究铺平道路，推动开发出能够在各类现实环境中运行的更强大的机器人系统。

## 作者署名

### 模型训练

• 罗浩

冯一程

• 张万鹏

• 郑思鹏

### 动作分词

冯一程

• 郑思鹏

#### 架构与基础设施

• 张万鹏

• 罗浩

• 郑思鹏

#### 数据源与对齐

- 罗浩

- 王烨

- 张万鹏

- 冯一程

- 刘佳政

• 郑思鹏

#### 实际场景实验

实际场景实验

• 王烨

• 袁浩奇

机器人系统

• 袁浩奇

• 徐超逸

项目负责人

• 陆宗清

---

## 参考文献

[^1]: Suhas Kadalagere Sampath, Ning Wang, Hao Wu, and Chenguang Yang. Review on human-like robot manipulation using dexterous hands. $\underline{\text{Cogn. Comput. Syst.}}$, 5(1):14–29, 2023.

[^2]: Yayu Huang, Dongxuan Fan, Haonan Duan, Dashun Yan, Wen Qi, Jia Sun, Qian Liu, and Peng Wang. Human-like dexterous manipulation for anthropomorphic five-fingered hands: A review. $\underline{\text{Biomimetic Intelligence and Robotics}}$, page 100212, 2025.

[^3]: Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06817, 2022.

[^4]: Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.15818, 2023.

[^5]: Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailev, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source vision-language-action model. $\underline{\text{arXiv preprint arXiv:2406.09246}}$, 2024.

[^6]: Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al. $\pi0$: A vision-language-action flow model for general robot control corr, abs/2410.24164, 2024. doi: 10.48550. arXiv preprint ARXIV.2410.24164, 2024.

[^7]: Yueen Ma, Zixing Song, Yuzheng Zhuang, Jianye Hao, and Irwin King. A survey on vision-language-action models for embodied ai. arXiv preprint arXiv:2405.14093, 2024.

[^8]: Abby O'Neill, Abdul Rehman, Abhiram Maddukuri, Abhishek Gupta, Abhishek Padalkar, Abraham Lee, Acorn Pooley, Agrim Gupta, Ajay Mandlekar, Ajinkya Jain, et al. Open x-embodiment: Robotic learning datasets and rt-x models: Open x-embodiment collaboration 0. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 6892–6903. IEEE, 2024.

[^9]: Alexander Khazatsky, Karl Pertsch, Suraj Nair, Ashwin Balakrishna, Sudeep Dasari, Siddharth Karamcheti, Soroush Nasiriany, Mohan Kumar Srirama, Lawrence Yunliang Chen, Kirsty Ellis, et al. Droid: A large-scale in-the-wild robot manipulation dataset. arXiv preprint arXiv:2403.12945, 2024.

[^10]: Qingwen Bu, Jisong Cai, Li Chen, Xiuqi Cui, Yan Ding, Siyuan Feng, Shenyuan Gao, Xindong He, Xu Huang, Shu Jiang, et al. Agibot world colosseo: A large-scale manipulation platform for scalable and intelligent embodied systems. arXiv preprint arXiv:2503.06669, 2025.

[^11]: Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Tobias Kreiman, Charles Xu, et al. Octo: An open-source generalist robot policy. $\underline{\text{arXiv preprint}}$ arXiv:2405.12213, 2024.

[^12]: Elizabeth Cutler, Yuning Xing, Tony Cui, Brendan Zhou, Koen van Rijnsoever, Ben Hart, David Valencia, Lee Violet C Ong, Trevor Gee, Minas Liarokapis, et al. Benchmarking reinforcement learning methods for dexterous robotic manipulation with a three-fingered gripper. $\underline{\text{arXiv preprint arXiv:2408.14747}}$, 2024.

[^13]: Shan An, Ziyu Meng, Chao Tang, Yuning Zhou, Tengyu Liu, Fangqiang Ding, Shufang Zhang, Yao Mu, Ran Song, Wei Zhang, et al. Dexterous manipulation through imitation learning: A survey. $\underline{\text{arXiv preprint arXiv:2504.03515}}$, 2025.

[^14]: Pranav Guruprasad, Harshvardhan Sikka, Jaewoo Song, Yangyue Wang, and Paul Pu Liang. Benchmarking vision, language, & action models on robotic learning tasks. arXiv preprint arXiv:2411.05821, 2024.

[^15]: Vincent de Bakker, Joey Hejna, Tyler Ga Wei Lum, Onur Celik, Aleksandar Taranovic, Denis Blessing, Gerhard Neumann, Jeannette Bohg, and Dorsa Sadigh. Scaffolding dexterous manipulation with vision-language models. arXiv preprint arXiv:2506.19212, 2025.

### [^16]: Yinzhen Xu, Weikang Wan, Jialiang Zhang, Haoran Liu, Zikang Shan, Hao Shen, Ruicheng Wang, Haoran Geng, Yijia Weng, Jiayi Chen, et al. Unidexgrasp: Universal robotic dexterous grasping via learning diverse proposal generation and goal-conditioned policy. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4737–4746, 2023

---

[^17]: Weikang Wan, Haoran Geng, Yun Liu, Zikang Shan, Yaodong Yang, Li Yi, and He Wang. Unidexgrasp++: Improving dexterous grasping policy learning via geometry-aware curriculum and iterative generalist-specialist learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3891–3902, 2023.

[^18]: Yifan Zhong, Xuchuan Huang, Ruochong Li, Ceyao Zhang, Yitao Liang, Yaodong Yang, and Yuanpei Chen. Dexgraspvla: A vision-language-action framework towards general dexterous grasping. $\underline{\text{arXiv preprint}}$ arXiv:2502.20900, 2025.

[^19]: Jiawei He, Danshi Li, Xinqiang Yu, Zekun Qi, Wenyao Zhang, Jiayi Chen, Zhaoxiang Zhang, Zhizheng Zhang, Li Yi, and He Wang. Dexvlg: Dexterous vision-language-grasp model at scale. $\underline{\text{arXiv preprint arXiv:2507.02747}}$, 2025.

[^20]: Ziye Huang, Haoqi Yuan, Yuhui Fu, and Zongqing Lu. Efficient residual learning with mixture-of-experts for universal dexterous grasping. In The Thirteenth International Conference on Learning Representations, 2025.

[^21]: Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3m: A universal visual representation for robot manipulation. arXiv preprint arXiv:2203.12601, 2022.

[^22]: Ilija Radosavovic, Tete Xiao, Stephen James, Pieter Abbeel, Jitendra Malik, and Trevor Darrell. Real-world robot learning with masked visual pre-training. In $\underline{\text{Conference on Robot Learning}}$, pages 416–426. PMLR, 2023.

[^23]: Johan Bjorck, Fernando Castañeda, Nikita Cherniadev, Xingye Da, Runyu Ding, Linxi Fan, Yu Fang, Dieter Fox, Fengyuan Hu, Spencer Huang, et al. Gr00t n1: An open foundation model for generalist humanoid robots. arXiv preprint arXiv:2503.14734, 2025.

[^24]: Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In $\underline{\text{Proceedings of the IEEE/CVF conference on computer vision and pattern recognition}}$, pages 26296-26306, 2024.

[^25]: Ri-Zhao Qiu, Shiqi Yang, Xuxin Cheng, Chaitanya Chawla, Jialong Li, Tairan He, Ge Yan, David J Yoon, Ryan Hoque, Lars Paulsen, et al. Humanoid policy~ human policy. arXiv preprint arXiv:2503.13441, 2025.

[^26]: Uikyum Kim, Dawoon Jung, Heeyoen Jeong, Jongwoo Park, Hyun-Mok Jung, Joono Cheong, Hyouk Ryeol Choi, Hyunmin Do, and Chanhun Park. Integrated linkage-driven dexterous anthropomorphic robotic hand. $\underline{\text{Nature communications}}$, 12(1):7177, 2021.

[^27]: Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. $\underline{\text{Advances in neural information processing systems}}$, 36:34892-34916, 2023.

[^28]: Doyup Lee, Chiheon Kim, Saehoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive image generation using residual quantization. In $\underline{\text{Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}}$, pages 11523-11532, 2022.

[^29]: Dongchao Yang, Songxiang Liu, Rongjie Huang, Jinchuan Tian, Chao Weng, and Yuexian Zou. Hifi-codec: Group-residual vector quantization for high fidelity audio codec. arXiv preprint arXiv:2305.02765, 2023.

[^30]: Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.

[^31]: Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. In $\underline{\text{arxiv}}$. San Francisco, CA, USA, 2018.

[^32]: Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.

[^33]: Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020.

[^34]: Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023.

#### [^35]: Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. $\underline{\text{arXiv preprint arXiv:2409.12191}}$, 2024

---

[^36]: Sipeng Zheng, Bohan Zhou, Yicheng Feng, Ye Wang, and Zongqing Lu. Unicode: Learning a unified codebook for multimodal large language models. arXiv preprint arXiv:2403.09072, 2024.

[^37]: Wanpeng Zhang, Zilong Xie, Yicheng Feng, Yijiang Li, Xingrun Xing, Sipeng Zheng, and Zongqing Lu. From pixels to tokens: Byte-pair encoding on quantized visual modalities. In $\underline{\text{The Thirteenth International Conference on Learning Representations}}$, 2025.

[^38]: Wanpeng Zhang, Yicheng Feng, Hao Luo, Yijiang Li, Zihao Yue, Sipeng Zheng, and Zongqing Lu. Unified multimodal understanding via byte-pair visual encoding. In $\underline{\text{Proceedings of the IEEE/CVF International Conference on Computer Vision}}$, 2025.

[^39]: Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023.

[^40]: Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.

[^41]: Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. $\underline{\text{arXiv preprint arXiv:2309.16609}}$, 2023.

[^42]: Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In $\underline{\text{International conference on machine learning}}$, pages 8748-8763. PmLR, 2021.

[^43]: Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pretraining. In $\underline{\text{Proceedings of the IEEE/CVF international conference on computer vision}}$, pages 11975-11986, 2023.

[^44]: Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716–23736, 2022.

[^45]: Yadong Lu, Chunyuan Li, Haotian Liu, Jianwei Yang, Jianfeng Gao, and Yelong Shen. An empirical study of scaling instruct-tuned large multimodal models. $\underline{\text{arXiv preprint arXiv:2309.09958}}$, 2023.

[^46]: Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang. Ferret: Refer and ground anything anywhere at any granularity. $\underline{\text{arXiv preprint arXiv:2310.07704}}$, 2023.

[^47]: Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Fanyi Pu, Joshua Adrian Cahyono, Jingkang Yang, Chunyuan Li, and Ziwei Liu. Otter: A multi-modal model with in-context instruction tuning. $\underline{\text{IEEE Transactions on Pattern Analysis and Machine Intelligence}}$, 2025.

[^48]: Guiming Hardy Chen, Shunian Chen, Ruifei Zhang, Junying Chen, Xiangbo Wu, Zhiyi Zhang, Zhihong Chen, Jianquan Li, Xiang Wan, and Benyou Wang. Allava: Harnessing gpt4v-synthesized data for lite vision-language models. arXiv preprint arXiv:2402.11684, 2024.

[^49]: Sipeng Zheng, Jiazheng Liu, Yicheng Feng, and Zongqing Lu. Steve-eye: Equipping llm-based embodied agents with visual perception in open worlds. $\underline{\text{arXiv preprint arXiv:2310.13255}}$, 2023.

[^50]: Yicheng Feng, Yijiang Li, Wanpeng Zhang, Sipeng Zheng, and Zongqing Lu. Videoorion: Tokenizing object dynamics in videos. $\underline{\text{arXiv preprint arXiv:2411.16156}}$, 2024.

[^51]: Jiazheng Liu, Sipeng Zheng, Börje F Karlsson, and Zongqing Lu. Taking notes brings focus? towards multi-turn multimodal dialogue learning. arXiv preprint arXiv:2503.07002, 2025.

[^52]: Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023.

#### [^53]: Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261, 2025

---

[^54]: Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyam Anadkat, et al. Gpt-4 technical report. $\underline{\text{arXiv preprint arXiv:2303.08774}}$, 2023.

[^55]: Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2.5-vl technical report. arXiv preprint arXiv:2502.13923, 2025.

[^56]: Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models. $\underline{\text{arXiv preprint arXiv:2405.09818}}$, 2024.

[^57]: Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, et al. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504.10479, 2025.

[^58]: Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, et al. Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models. $\underline{\text{arXiv e-prints}}$, pages arXiv-2409, 2024.

[^59]: Matthias Plappert, Christian Mandery, and Tamim Asfour. The kit motion-language dataset. $\underline{\text{Big data, 4(4):236-252, 2016.}}$

[^60]: Naureen Mahmood, Nima Ghorbani, Nikolaus F Troje, Gerard Pons-Moll, and Michael J Black. Amass: Archive of motion capture as surface shapes. In $\underline{\text{Proceedings of the IEEE/CVF international conference on computer vision}}$, pages 5442-5451, 2019.

[^61]: Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang, Wei Ji, Xingyu Li, and Li Cheng. Generating diverse and natural 3d human motions from text. In $\underline{\text{Proceedings of the IEEE/CVF conference on computer vision and pattern recognition}}$, pages 5152-5161, 2022.

[^62]: Abhinanda R Punnakkal, Arjun Chandrasekaran, Nikos Athanasiou, Alejandra Quiros-Ramirez, and Michael J Black. Babel: Bodies, action and behavior with english labels. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 722–731, 2021.

[^63]: Jing Lin, Ailing Zeng, Shunlin Lu, Yuanhao Cai, Ruimao Zhang, Haoqian Wang, and Lei Zhang. Motion-x: A large-scale 3d expressive whole-body human motion dataset. $\underline{\text{Advances in Neural Information Processing Systems}}$, 36:25268–25280, 2023.

[^64]: Siwei Zhang, Qianli Ma, Yan Zhang, Zhiyin Qian, Taein Kwon, Marc Pollefeys, Federica Bogo, and Siyu Tang. Egobody: Human body shape and motion of interacting people from head-mounted devices. In $\underline{\text{European conference on computer vision}}$, pages 180-200. Springer, 2022.

[^65]: Lingni Ma, Yuting Ye, Fangzhou Hong, Vladimir Guzov, Yifeng Jiang, Rowan Postyeni, Luis Pesqueira, Alexander Gamino, Vijay Baiyya, Hyo Jin Kim, et al. Nymeria: A massive collection of multimodal egocentric daily motion in the wild. In $\underline{\text{European Conference on Computer Vision}}$, pages 445–465. Springer, 2024.

[^66]: Ye Wang, Sipeng Zheng, Bin Cao, Qianshan Wei, Weishuai Zeng, Qin Jin, and Zongqing Lu. Scaling large motion models with million-level human motions. In $\underline{\text{Forty-second International Conference on Machine Learning}}$, 2024.

[^67]: Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black. Smpl: A skinned multi-person linear model. In $\underline{\text{Seminal Graphics Papers: Pushing the Boundaries, Volume 2}}$, pages 851-866, 2023.

[^68]: Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed AA Osman, Dimitrios Tzionas, and Michael J Black. Expressive body capture: 3d hands, face, and body from a single image. In $\underline{\text{Proceedings of the IEEE/CVF conference on computer vision and pattern recognition}}$, pages 10975–10985, 2019.

[^69]: Guy Tevet, Sigal Raab, Brian Gordon, Yonatan Shafir, Daniel Cohen-Or, and Amit H Bermano. Human motion diffusion model. arXiv preprint arXiv:2209.14916, 2022.

[^70]: Mingyuan Zhang, Zhongang Cai, Liang Pan, Fangzhou Hong, Xinying Guo, Lei Yang, and Ziwei Liu. Motiondiffuse: Text-driven human motion generation with diffusion model. $\underline{\text{IEEE transactions on pattern analysis and machine intelligence}}$, 46(6):4115-4128, 2024.

#### [^71]: Xin Chen, Biao Jiang, Wen Liu, Zilong Huang, Bin Fu, Tao Chen, and Gang Yu. Executing your commands via motion diffusion in latent space. In $\underline{\text{Proceedings of the IEEE/CVF conference on computer vision and pattern recognition}}$, pages 18000-18010, 2023

---

[^72]: Ye Yuan, Jiaming Song, Umar Iqbal, Arash Vahdat, and Jan Kautz. Physdiff: Physics-guided human motion diffusion model. In $\underline{\text{Proceedings of the IEEE/CVF international conference on computer vision}}$, pages 16010–16021, 2023.

[^73]: Mingyuan Zhang, Xinying Guo, Liang Pan, Zhongang Cai, Fangzhou Hong, Huirong Li, Lei Yang, and Ziwei Liu. Remodiffuse: Retrieval-augmented motion diffusion model. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 364–373, 2023.

[^74]: Yunhong Lou, Linchao Zhu, Yaxiong Wang, Xiaohan Wang, and Yi Yang. Diversemotion: Towards diverse human motion generation via discrete diffusion. $\underline{\text{arXiv preprint arXiv:2309.01372}}$, 2023.

[^75]: Mingyuan Zhang, Daisheng Jin, Chenyang Gu, Fangzhou Hong, Zhongang Cai, Jingfang Huang, Chongzhi Zhang, Xinying Guo, Lei Yang, Ying He, et al. Large motion model for unified multi-modal motion generation. In $\underline{\text{European Conference on Computer Vision}}$, pages 397-421. Springer, 2024.

[^76]: Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. $\underline{\text{Advances in neural information processing systems}}$, 30, 2017.

[^77]: Jianrong Zhang, Yangsong Zhang, Xiaodong Cun, Yong Zhang, Hongwei Zhao, Hongtao Lu, Xi Shen, and Ying Shan. Generating human motion from textual descriptions with discrete representations. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14730–14740, 2023.

[^78]: Chuan Guo, Yuxuan Mu, Muhammad Gohar Javed, Sen Wang, and Li Cheng. Momask: Generative masked modeling of 3d human motions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1900–1910, 2024.

[^79]: Tackgeun You, Saehoon Kim, Chiheon Kim, Doyup Lee, and Bohyung Han. Locally hierarchical auto-regressive modeling for image generation. Advances in Neural Information Processing Systems, 35:16360–16372, 2022.

[^80]: Shunlin Lu, Ling-Hao Chen, Ailing Zeng, Jing Lin, Ruimao Zhang, Lei Zhang, and Heung-Yeung Shum. Humantomato: Text-aligned whole-body motion generation. arXiv preprint arXiv:2310.12978, 2023.

[^81]: Fabian Mentzer, David Minnen, Eirikur Agustsson, and Michael Tschannen. Finite scalar quantization: Vq-vae made simple. $\underline{\text{arXiv preprint arXiv:2309.15505}}$, 2023.

[^82]: Lijun Yu, José Lezama, Nitesh B Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Vighnesh Birodkar, Agrim Gupta, Xiuye Gu, et al. Language model beats diffusion-tokenizer is key to visual generation. arXiv preprint arXiv:2310.05737, 2023.

[^83]: Biao Jiang, Xin Chen, Wen Liu, Jingyi Yu, Gang Yu, and Tao Chen. Motiongpt: Human motion as a foreign language. Advances in Neural Information Processing Systems, 36:20067–20079, 2023.

[^84]: Yuan Wang, Di Huang, Yaqi Zhang, Wanli Ouyang, Jile Jiao, Xuetao Feng, Yan Zhou, Pengfei Wan, Shixiang Tang, and Dan Xu. Motiongpt-2: A general-purpose motion-language model for motion generation and understanding. arXiv preprint arXiv:2410.21747, 2024.

[^85]: Ling-Hao Chen, Shunlin Lu, Ailing Zeng, Hao Zhang, Benyou Wang, Ruimao Zhang, and Lei Zhang. Motionllm: Understanding human behaviors from human motions and videos. arXiv preprint arXiv:2405.20340, 2024.

[^86]: Zixiang Zhou, Yu Wan, and Baoyuan Wang. Avatargpt: All-in-one framework for motion understanding planning generation and beyond. In $\underline{\text{Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}}$, pages 1357-1366, 2024.

[^87]: Biao Jiang, Xin Chen, Chi Zhang, Fukun Yin, Zhuoyuan Li, Gang Yu, and Jiayuan Fan. Motionchain: Conversational motion controllers via multimodal prompts. In $\underline{\text{European Conference on Computer Vision}}$, pages 54–74. Springer, 2024.

[^88]: Zhe Li, Weihao Yuan, Yisheng He, Lingteng Qiu, Shenhao Zhu, Xiaodong Gu, Weichao Shen, Yuan Dong, Zilong Dong, and Laurence T Yang. Lamp: Language-motion pretraining for motion generation, retrieval, and captioning. arXiv preprint arXiv:2410.07093, 2024.

### [^89]: Lei Li, Sen Jia, Jianhao Wang, Zhongyu Jiang, Feng Zhou, Ju Dai, Tianfang Zhang, Zongkai Wu, and Jenq-Neng Hwang. Human motion instruction tuning. In $\underline{\text{Proceedings of the Computer Vision and Pattern Recognition Conference}}$, pages 17582-17591, 2025

---

[^90]: Junpeng Yue, Zepeng Wang, Yuxuan Wang, Weishuai Zeng, Jiangxing Wang, Xinrun Xu, Yu Zhang, Sipeng Zheng, Ziluo Ding, and Zongqing Lu. Rl from physical feedback: Aligning large motion models with humanoid control. arXiv preprint arXiv:2506.12769, 2025.

[^91]: Zhengyi Luo, Jinkun Cao, Kris Kitani, Weipeng Xu, et al. Perpetual humanoid control for real-time simulated avatars. In $\underline{\text{Proceedings of the IEEE/CVF International Conference on Computer Vision}}$, pages 10895-10904, 2023.

[^92]: Mazeyu Ji, Xuanbin Peng, Fangchen Liu, Jialong Li, Ge Yang, Xuxin Cheng, and Xiaolong Wang. Exbody2: Advanced expressive humanoid whole-body control. arXiv preprint arXiv:2412.13196, 2024.

[^93]: Gaoge Han, Mingjiang Liang, Jinglei Tang, Yongkang Cheng, Wei Liu, and Shaoli Huang. Reindiffuse: Crafting physically plausible motions with reinforced diffusion model. In $\underline{\text{2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)}}$, pages 2218–2227. IEEE, 2025.

[^94]: Kristen Grauman, Andrew Westbury, Lorenzo Torresani, Kris Kitani, Jitendra Malik, Triantafyllos Afouras, Kumar Ashutosh, Vijay Baiyya, Siddhant Bansal, Bikram Boote, et al. Ego-exo4d: Understanding skilled human activity from first-and third-person perspectives. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19383–19400, 2024.

[^95]: Hanwen Jiang, Shaowei Liu, Jiashun Wang, and Xiaolong Wang. Hand-object contact consistency reasoning for human grasps generation. In $\underline{\text{Proceedings of the IEEE/CVF international conference on computer vision}}$, pages 11107-11116, 2021.

[^96]: Shaowei Liu, Subarna Tripathi, Somdeb Majumdar, and Xiaolong Wang. Joint hand motion and interaction hotspots prediction from egocentric videos. In $\underline{\text{Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}}$, pages 3282-3292, 2022.

[^97]: Prithviraj Banerjee, Sindi Shkodrani, Pierre Moulon, Shreyas Hampali, Shangchen Han, Fan Zhang, Linguang Zhang, Jade Fountain, Edward Miller, Selen Basol, et al. Hot3d: Hand and object tracking in 3d from egocentric multi-view videos. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 7061–7071, 2025.

[^98]: Yunze Liu, Yun Liu, Che Jiang, Kangbo Lyu, Weikang Wan, Hao Shen, Boqiang Liang, Zhoujie Fu, He Wang, and Li Yi. Hoi4d: A 4d egocentric dataset for category-level human-object interaction. In $\underline{\text{Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}}$, pages 21013-21022, 2022.

[^99]: Xinyu Zhan, Lixin Yang, Yifei Zhao, Kangrui Mao, Hanlin Xu, Zenan Lin, Kailin Li, and Cewu Lu. Oakink2: A dataset of bimanual hands-object manipulation in complex task completion. In $\underline{\text{Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}}$, pages 445–456, 2024.

[^100]: Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 18995–19012, 2022.

[^101]: Yana Hasson, Gul Varol, Dimitrios Tzionas, Igor Kalevatykh, Michael J Black, Ivan Laptev, and Cordelia Schmid. Learning joint reconstruction of hands and manipulated objects. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11807–11816, 2019.

[^102]: Georgios Pavlakos, Dandan Shan, Ilija Radosavovic, Angjoo Kanazawa, David Fouhey, and Jitendra Malik. Reconstructing hands in 3d with transformers. In $\underline{\text{Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}}$, pages 9826–9836, 2024.

[^103]: Haoye Dong, Aviral Chharia, Wenbo Gou, Francisco Vicente Carrasco, and Fernando D De la Torre. Hamba: Single-view 3d hand reconstruction with graph-guided bi-scanning mamba. $\underline{\text{Advances in Neural Information Processing Systems}}$, 37:2127–2160, 2024.

[^104]: Zicong Fan, Maria Parelli, Maria Eleni Kadoglou, Xu Chen, Muhammed Kocabas, Michael J Black, and Otmar Hilliges. Hold: Category-agnostic 3d reconstruction of interacting hands and objects from video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 494–504, 2024.

### [^105]: Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. Scaling egocentric vision: The epic-kitchens dataset. In Proceedings of the European conference on computer vision (ECCV), pages 720–736, 2018

---

[^106]: Zhengdi Yu, Stefanos Zafeiriou, and Tolga Birdal. Dyn-hamr: Recovering 4d interacting hand motion from a dynamic camera. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 27716–27726, 2025.

[^107]: Georgia Gkioxari, Ross Girshick, and Jitendra Malik. Contextual action recognition with r* cnn. In Proceedings of the IEEE international conference on computer vision, pages 1080–1088, 2015.

[^108]: Arun Mallya and Svetlana Lazebnik. Learning models for actions and person-object interactions with transfer to question answering. In $\underline{\text{European Conference on Computer Vision}}$, pages 414-428. Springer, 2016.

[^109]: Sipeng Zheng, Boshen Xu, and Qin Jin. Open-category human-object interaction pre-training via language modeling framework. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19392–19402, 2023.

[^110]: Georgia Gkioxari, Ross Girshick, Piotr Dollár, and Kaiming He. Detecting and recognizing human-object interactions. In $\underline{\text{Proceedings of the IEEE conference on computer vision and pattern recognition}}$, pages 8359–8367, 2018.

[^111]: Siyuan Qi, Wenguan Wang, Baoxiong Jia, Jianbing Shen, and Song-Chun Zhu. Learning human-object interactions by graph parsing neural networks. In Proceedings of the European conference on computer vision (ECCV), pages 401-417, 2018.

[^112]: Sammy Christen, Shreyas Hampali, Fadime Sener, Edoardo Remelli, Tomas Hodan, Eric Sauser, Shugao Ma, and Bugra Tekin. Diff2o: Diffusion-based synthesis of hand-object interactions from textual descriptions. In SIGGRAPH Asia 2024 Conference Papers, pages 1–11, 2024.

[^113]: Junuk Cha, Jihyeon Kim, Jae Shin Yoon, and Seungryul Baek. Text2hoi: Text-guided 3d motion generation for hand-object interaction. In $\underline{\text{Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}}$, pages 1577-1585, 2024.

[^114]: Anindita Ghosh, Rishabh Dabral, Vladislav Golyanik, Christian Theobalt, and Philipp Slusallek. Imos: Intent-driven full-body motion synthesis for human-object interactions. In $\underline{\text{Computer Graphics Forum}}$, volume 42, pages 1–12. Wiley Online Library, 2023.

[^115]: Samarth Brahmbhatt, Cush Ham, Charles C Kemp, and James Hays. Contactdb: Analyzing and predicting grasp contact via thermal imaging. In $\underline{\text{Proceedings of the IEEE/CVF conference on computer vision and pattern recognition}}$, pages 8709–8719, 2019.

[^116]: Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. $\underline{\text{Advances in neural information processing systems}}$, 33:6840–6851, 2020.

[^117]: Mingzhen Huang, Fu-Jen Chu, Bugra Tekin, Kevin J Liang, Haoyu Ma, Weiyao Wang, Xingyu Chen, Pierre Gleize, Hongfei Xue, Siwei Lyu, et al. Hoigpt: Learning long-sequence hand-object interaction with language models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 7136–7146, 2025.

[^118]: Bohan Zhou, Yi Zhan, Zhongbin Zhang, and Zongqing Lu. Megohand: Multimodal egocentric hand-object interaction motion generation. $\underline{\text{arXiv preprint arXiv:2505.16602}}$, 2025.

[^119]: Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, et al. $\pi_{0.5}$: a vision-language-action model with open-world generalization. arXiv preprint arXiv:2504.16054, 2025.

[^120]: Karl Pertsch, Kyle Stachowicz, Brian Ichter, Danny Driess, Suraj Nair, Quan Vuong, Oier Mees, Chelsea Finn, and Sergey Levine. Fast: Efficient action tokenization for vision-language-action models. $\underline{\text{arXiv preprint}}$ arXiv:2501.09747, 2025.

[^121]: Songming Liu, Lingxuan Wu, Bangguo Li, Hengkai Tan, Huayu Chen, Zhengyi Wang, Ke Xu, Hang Su, and Jun Zhu. Rdt-1b: a diffusion foundation model for bimanual manipulation. arXiv preprint arXiv:2410.07864, 2024.

[^122]: Jianglong Ye, Keyi Wang, Chengjing Yuan, Ruihan Yang, Yiquan Li, Jiyue Zhu, Yuzhe Qin, Xueyan Zou, and Xiaolong Wang. Dex1b: Learning with 1b demonstrations for dexterous manipulation. $\underline{\text{arXiv preprint}}$ arXiv:2506.17198, 2025.

### [^123]: Shengliang Deng, Mi Yan, Songlin Wei, Haixin Ma, Yuxin Yang, Jiayi Chen, Zhiqi Zhang, Taoyu Yang, Xuheng Zhang, Heming Cui, et al. Graspvla: a grasping foundation model pre-trained on billion-scale synthetic action data. arXiv preprint arXiv:2505.03233, 2025

---

[^124]: Boshen Xu, Yuting Mei, Xinbi Liu, Sipeng Zheng, and Qin Jin. Egodtm: Towards 3d-aware egocentric video-language pretraining. $\underline{\text{arXiv preprint arXiv:2503.15470}}$, 2025.

[^125]: Alexey Gavryushin, Xi Wang, Robert JS Malate, Chenyu Yang, Xiangyi Jia, Shubh Goel, Davide Liconti, René Zurbrügg, Robert K Katzschmann, and Marc Pollefeys. Maple: Encoding dexterous robotic manipulation priors learned from egocentric videos. arXiv preprint arXiv:2504.06084, 2025.

[^126]: Hanzhi Chen, Boyang Sun, Anran Zhang, Marc Pollefeys, and Stefan Leutenegger. Vidbot: Learning generalizable 3d actions from in-the-wild 2d human videos for zero-shot robotic manipulation. In $\underline{\text{Proceedings of the Computer Vision and Pattern Recognition Conference}}$, pages 27661-27672, 2025.

[^127]: Teli Ma, Jia Zheng, Zifan Wang, Ziyao Gao, Jiaming Zhou, and Junwei Liang. Glover++: Unleashing the potential of affordance learning from human behaviors for robotic manipulation. $\underline{\text{arXiv preprint arXiv:2505.11865}}$. 2025.

[^128]: Marion Lepert, Jiaying Fang, and Jeannette Bohg. Phantom: Training robots without robots using only human videos. $\underline{\text{arXiv preprint arXiv:2503.00779}}$, 2025.

[^129]: Simar Kareer, Dhruv Patel, Ryan Punamiya, Pranay Mathur, Shuo Cheng, Chen Wang, Judy Hoffman, and Danfei Xu. Egomimic: Scaling imitation learning via egocentric video. arXiv preprint arXiv:2410.24221, 2024.

[^130]: Yaru Niu, Yunzhe Zhang, Mingyang Yu, Changyi Lin, Chenhao Li, Yikai Wang, Yuxiang Yang, Wenhao Yu, Tingnan Zhang, Bingqing Chen, et al. Human2locoman: Learning versatile quadrupedal manipulation with human pretraining. arXiv preprint arXiv:2506.16475, 2025.

[^131]: Himanshu Gaurav Singh, Antonio Loquercio, Carmelo Sferrazza, Jane Wu, Haozhi Qi, Pieter Abbeel, and Jitendra Malik. Hand-object interaction pretraining from videos. arXiv preprint arXiv:2409.08273, 2024.

[^132]: Haoqi Yuan, Bohan Zhou, Yuhui Fu, and Zongqing Lu. Cross-embodiment dexterous grasping with reinforcement learning. In $\underline{\text{The Thirteenth International Conference on Learning Representations}}$, 2025.

[^133]: Kailin Li, Puhao Li, Tengyu Liu, Yuyang Li, and Siyuan Huang. Maniptrans: Efficient dexterous bimanual manipulation transfer via residual learning. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 6991–7003, 2025.

[^134]: Zerui Chen, Shizhe Chen, Etienne Arlaud, Ivan Laptev, and Cordelia Schmid. Vividex: Learning vision-based dexterous manipulation from human videos. arXiv preprint arXiv:2404.15709, 2024.

[^135]: Bohan Zhou, Haoqi Yuan, Yuhui Fu, and Zongqing Lu. Learning diverse bimanual dexterous manipulation skills from human demonstrations. arXiv preprint arXiv:2410.02477, 2024.

[^136]: Huayi Zhou, Ruixiang Wang, Yunxin Tai, Yueci Deng, Guiliang Liu, and Kui Jia. You only teach once: Learn one-shot bimanual robotic manipulation from video demonstrations. arXiv preprint arXiv:2501.14208, 2025.

[^137]: Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. arXiv preprint arXiv:1709.10087, 2017.

[^138]: Ruicheng Wang, Jialiang Zhang, Jiayi Chen, Yinzhen Xu, Puhao Li, Tengyu Liu, and He Wang. Dexgraspnet: A large-scale robotic dexterous grasp dataset for general objects based on simulation. $\underline{\text{arXiv preprint}}$ $\underline{\text{arXiv:2210.02697}}$, 2022.

[^139]: Javier Romero, Dimitrios Tzionas, and Michael J. Black. Embodied hands: Modeling and capturing hands and bodies together. $\underline{\text{ACM Transactions on Graphics, (Proc. SIGGRAPH Asia)}}$, 36(6), November 2017.

[^140]: Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271, 2024.

[^141]: Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. Science China Information Sciences, 67(12):220101, 2024.

#### [^142]: Yannick Bukschat and Marcus Vetter. Efficientpose: An efficient, accurate and scalable end-to-end 6d multi object pose estimation approach. arXiv preprint arXiv:2011.04307, 2020

---

[^143]: Siddharth Mahendran, Haider Ali, and René Vidal. 3d pose regression using convolutional neural networks. In Proceedings of the IEEE international conference on computer vision workshops, pages 2174–2182, 2017.

[^144]: Wentao Yuan, Jiafei Duan, Valts Blukis, Wilbert Pumacay, Ranjay Krishna, Adithyavairavan Murali, Arsalan Mousavian, and Dieter Fox. Robopoint: A vision-language model for spatial affordance prediction for robotics. arXiv preprint arXiv:2406.10721, 2024.

[^145]: Wufei Ma, Haoyu Chen, Guofeng Zhang, Yu-Cheng Chou, Celso M de Melo, and Alan Yuille. 3dsbench: A comprehensive 3d spatial reasoning benchmark. $\underline{\text{arXiv preprint arXiv:2412.07825}}$, 2024.

[^146]: Yifu Yuan, Haiqin Cui, Yibin Chen, Zibin Dong, Fei Ni, Longxin Kou, Jinyi Liu, Pengyi Li, Yan Zheng, and Jianye Hao. From seeing to doing: Bridging reasoning and decision for robotic manipulation. $\underline{\text{arXiv preprint}}$ arXiv:2505.08548, 2025.

[^147]: Jihan Yang, Shusheng Yang, Anjali W Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. Thinking in space: How multimodal large language models see, remember, and recall spaces. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 10632–10643, 2025.

[^148]: Zicong Fan, Omid Taheri, Dimitrios Tzionas, Muhammed Kocabas, Manuel Kaufmann, Michael J Black, and Otmar Hilliges. Arctic: A dataset for dexterous bimanual hand-object manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12943–12954, 2023.

[^149]: Guillermo Garcia-Hernando, Shanxin Yuan, Seungryul Baek, and Tae-Kyun Kim. First-person hand action benchmark with rgb-d videos and 3d hand pose annotations. In $\underline{\text{Proceedings of the IEEE conference on computer vision and pattern recognition}}$, pages 409-419, 2018.

[^150]: Taein Kwon, Bugra Tekin, Jan Stühmer, Federica Bogo, and Marc Pollefeys. H2o: Two hands manipulating objects for first person interaction recognition. In $\underline{\text{Proceedings of the IEEE/CVF international conference on computer vision}}$, pages 10138-10148, 2021.

[^151]: Ryan Hoque, Peide Huang, David J Yoon, Mouli Sivapurapu, and Jian Zhang. Egodex: Learning dexterous manipulation from large-scale egocentric video. $\underline{\text{arXiv preprint arXiv:2505.11709}}$, 2025.

[^152]: Toby Perrett, Ahmad Darkhalil, Saptarshi Sinha, Omar Emara, Sam Pollard, Kranti Kumar Parida, Kaiting Liu, Prajwal Gatti, Siddhant Bansal, Kevin Flanagan, et al. Hd-epic: A highly-detailed egocentric video dataset. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 23901–23913, 2025.

[^153]: Zhongang Cai, Wanqi Yin, Ailing Zeng, Chen Wei, Qingping Sun, Wang Yanjun, Hui En Pang, Haiyi Mei, Mingyuan Zhang, Lei Zhang, et al. Simpler-x: Scaling up expressive human pose and shape estimation. $\underline{\text{Advances in Neural Information Processing Systems}}$, 36:11454-11468, 2023.

[^154]: Hongxiang Zhao, Xingchen Liu, Mutian Xu, Yiming Hao, Weikai Chen, and Xiaoguang Han. Taste-rob: Advancing video generation of task-oriented hand-object interaction for generalizable robotic manipulation. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 27683–27693, 2025.

[^155]: Xin Wang, Taein Kwon, Mahdi Rad, Bowen Pan, Ishani Chakraborty, Sean Andrist, Dan Bohus, Ashley Feniello, Bugra Tekin, Felipe Vieira Frujeri, et al. Holoassist: an egocentric human interaction dataset for interactive ai assistants in the real world. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 20270–20281, 2023.

[^156]: Yun Liu, Haolin Yang, Xu Si, Ling Liu, Zipeng Li, Yuxiang Zhang, Yebin Liu, and Li Yi. Taco: Benchmarking generalizable bimanual tool-action-object understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21740–21751, 2024.

[^157]: Yu-Wei Chao, Wei Yang, Yu Xiang, Pavlo Molchanov, Ankur Handa, Jonathan Tremblay, Yashraj S Narang, Karl Van Wyk, Umar Iqbal, Stan Birchfield, et al. Dexycb: A benchmark for capturing hand grasping of objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9044–9053, 2021.

[^158]: Mathis Petrovich, Michael J. Black, and Gül Varol. TMR: Text-to-motion retrieval using contrastive 3D human motion synthesis. In $\underline{\text{International Conference on Computer Vision (ICCV)}}$, 2023.

#### [^159]: Philipp Wu, Yide Shentui, Zhongke Yi, Xingyu Lin, and Pieter Abbeel. Gello: A general, low-cost, and intuitive teleoperation framework for robot manipulators. In $\underline{\text{2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)}}$, pages 12156-12163. IEEE, 2024

---

[^160]: Yuzhe Qin, Wei Yang, Binghao Huang, Karl Van Wyk, Hao Su, Xiaolong Wang, Yu-Wei Chao, and Dieter Fox. Anyteleop: A general vision-based dexterous robot arm-hand teleoperation system. $\underline{\text{arXiv preprint}}$ arXiv:2307.04577, 2023.

[^161] 袁浩奇、白宇、付玉辉、周博涵、冯一程、徐新润、詹毅、Börje F Karlsson和陆宗清。Being-0：一种具备视觉语言模型与模块化技能的人形机器人智能体。$\underline{\text{arXiv preprint}}$ arXiv:2503.12533，2025年。