Figure 1. EgoMAN project. We introduce 1) the EgoMAN dataset (top), a large-scale egocentric dataset for interaction stage–aware 3D hand trajectory prediction with 219K 6-DoF trajectories and 3M structured QA pairs for semantic, spatial, and motion reasoning. During inference, 2) the EgoMAN model (bottom) takes an image, past hand motion, and an intent query as input, performs stage-aware reasoning to infer intent-specific waypoints, and then generates 6-DoF hand trajectories of distinct motions for different intent queries.
Figure 2. Overview of the EgoMAN model. The EgoMAN model is a modular reasoning-to-motion framework that predicts future 6DoF hand trajectories from an egocentric RGB frame, past wrist trajectories, and a language intent. The Reasoning Module (a), built on QwenVL-7B, extracts semantic and spatial features and outputs trajectory tokens with waypoints and intent semantic cues. The Motion Expert (b), using Flow Matching, predicts future trajectories based on waypoints, past motion, intent semantics and visual input. The trajectory tokens of (a) form the Trajectory-Token Interface which replaces semantic and waypoint condition inputs of (b) to bridge from Reasoning to Motion Expert.
#### 4.3. 运动专家模块
运动专家模块以推理模块输出的轨迹标记为基础，通过建模精细的手部动力学来预测高频的 6DoF 手腕轨迹。该模块是一个基于 Flow Matching（FM）[^33] 的编码器-解码器式Transformer，其输入包括过去的手腕运动数据、意图语义、低层次视觉特征以及阶段相关的航点信息。FM 能够学习条件速度场，从而生成平滑且具备概率特性的轨迹，而三个航点标记则为其提供结构上的引导。
如图 2(b) 所示，编码器将所有输入组织成一个统一的序列。与运动相关的标记位于统一的时间轴上：过去的 H 个手腕运动点占据步骤 $1-H$，航点标记被放置在偏移 H 的预测时间戳处，而未来的查询则覆盖 $H+1-H+T$。这些时间标记会根据其时间戳获得相应的位置 ID。与此同时，意图语义和 DINOv3 [^52] 视觉特征则作为非时间性的上下文标记加入。随后，解码器通过关注这一完整的编码上下文来生成 T 个未来的 6DoF 轨迹点。
我们遵循标准的 FM 流程：将含噪样本 $x_{0}$ 与真实值 $x_{1}$ 进行插值，监督目标则为 $\hat{v} = x_{1} - x_{0}$。损失函数是对 3D 位置和 6D 旋转计算均方误差：
$$ \mathcal{L}_{\mathrm{F M}}=\|\hat{v}-(x_{1}-x_{0})\|_{2}^{2}. $$
在测试阶段，我们先采样一个初始随机轨迹 $x_{0}$，再将其速度场积分 N 步：
$$ x_{k+1}=x_{k}+\Delta t\cdot\hat{v}(x_{k},t_{k}),\quad\Delta t=\frac{1}{N}, $$
从而得到未来的手腕轨迹。
运动预训练。我们发现，由于学习目标不匹配，推理模块和运动专家模块联合训练时稳定性较差。为解决这一问题，我们在 EgoMAN 微调子集上单独对 FM 模型进行预训练（见第 3 节），以真实航点和动作短语语义作为条件。这种预训练能为联合训练提供强大的低层次运动先验，从而稳定与推理模块的合作。
#### 4.4. 推理到运动
一旦推理模块和运动专家模块都完成预训练，我们就将它们联合训练，以实现高层推理与低层次运动生成之间的衔接。这一阶段的关键挑战在于分布不匹配问题：推理模块是在真实值基础上预训练以预测标记的，而
---
Method
ADE (m) $\downarrow$
FDE (m) $\downarrow$
DTW (m) $\downarrow$
Rot (°) $\downarrow$
Dataset
K=1
K=5
K=10
K=1
K=5
K=10
K=1
K=5
K=10
K=1
K=5
K=10
USST $^{{*}}$ [^6]
0.233
0.233
0.233
0.394
0.394
0.394
0.220
0.220
0.220
46.98
46.98
46.98
EgoMAN
MMTwin $^{{*}}$ [^41]
0.213
0.208
0.206
0.261
0.257
0.256
0.211
0.206
0.204
49.53
49.12
48.98
HandsOnVLM $^{{*}}$ [^5]
0.176
0.172
0.171
0.232
0.228
0.228
0.166
0.162
0.161
35.49
35.29
35.22
FM-base
0.188
0.166
0.160
0.265
0.236
0.229
0.171
0.150
0.144
37.92
37.27
37.00
Unseen
EgoMAN-ACT
0.162
0.146
0.141
0.225
0.210
0.204
0.148
0.132
0.127
36.24
35.28
35.03
EgoMAN (Ours)
0.151
0.130
0.124
0.206
0.186
0.179
0.137
0.117
0.111
33.88
33.00
32.75
USST $^{{*}}$ [^6]
0.245
0.245
0.245
0.409
0.409
0.409
0.226
0.226
0.226
55.80
55.80
55.80
HOT3D
MMTwin $^{{*}}$ [^41]
0.214
0.210
0.209
0.263
0.260
0.259
0.212
0.208
0.207
44.75
44.46
44.37
HandsOnVLM $^{{*}}$ [^5]
0.197
0.194
0.194
0.266
0.262
0.262
0.191
0.188
0.186
38.29
38.18
38.13
FM-base
0.176
0.165
0.161
0.252
0.241
0.237
0.161
0.151
0.147
40.23
39.64
39.47
OOD
EgoMAN-ACT
0.172
0.158
0.153
0.247
0.232
0.228
0.159
0.145
0.141
39.60
38.68
38.42
EgoMAN (Ours)
0.166
0.147
0.141
0.246
0.224
0.217
0.155
0.137
0.130
36.11
35.40
35.09
Table 1. Comparison of 6DoF hand trajectory prediction on EgoMAN-Unseen and HOT3D-OOD. Lower is better. Best values are bold, second-best are underlined. Our EgoMAN model outperforms the strongest external baseline (HandsOnVLM) by 27.5% ADE on both the held-out EgoMAN-Unseen test split and the out-of-distribution HOT3D-OOD dataset.
Table 2. Waypoint prediction results. Lower is better for Contact and Traj; higher is better for FPS (averaged over 50 samples on an NVIDIA PG509-210, 80GB). EgoMAN-WP achieves the best accuracy, improving Contact by 33.8% and Traj by 52.8% on EgoMAN-Unseen, and runs orders of magnitude faster at 3.45 FPS.
Pretrain Reason FM
WP
ADE $\downarrow$
FDE $\downarrow$
DTW $\downarrow$
Rot $\downarrow$
✗
✗
0.273
0.308
0.260
51.79
✓
✗
6DoF
0.215
0.255
43.03
✗
✓
✗
0.162
0.225
36.24
✓
✓
✗
0.161
0.224
35.90
✓
✓
Emb
0.150
0.210
34.02
✓
✓
6DoF
0.151
0.206
33.88
Table 3. Ablation on EgoMAN-Unseen (K=1). Lower is better. Reason and FM pretraining with 6DoF waypoints yield the highest accuracy.
Figure 3. Qualitative comparisons on EgoMAN-Bench. We visualize best-of- $K=10$ predictions for waypoints and full trajectories. Left: and waypoint predictions compared with VRB $^{*}$ and VidBot. Right: 3D hand trajectory forecasts and 2D projections compared with prior baselines. Our EgoMAN model produces the smoothest and closest results to ground truth.
Figure 4. Qualitative results of diverse activities. EgoMAN generates accurate 6DoF hand trajectories for diverse activities, $\underline{\text{aligning motion with}}$ the intent description and scene spatial.
Figure 5. Multiple intents. With the same image and past motion, EgoMAN model produces distinct 6DoF trajectories for different intent queries, showing controllable intent-to-motion generation.
#### 5.5. 定性分析
我们在图3中展示了EgoMAN-Bench（EgoMAN-Unseen和HOT3D-OOD）上的最佳$K=10$预测结果。图3左侧对比了基于可供性方法的基线模型（VRB*、VidBot）与我们EgoMAN模型的航点。我们预测的接触点与终点与GT手腕位置高度吻合，而基于可供性的方法由于检测误差往往无法触及目标表面，或会偏离预定目标区域而朝向手部移动。图3右侧则将完整的6DoF轨迹与轨迹基线模型（USST、HandsOnVLM、MMTwin、FM-base）进行比较。EgoMAN生成的动作更平滑、更精准，既能到达目标位置，又能以正确的手腕方向完成操作；而基线模型在杂乱场景中或对陌生物体及意图描述的处理上，常常出现触及不足、过度延伸或轨迹偏移的问题。
图4进一步展示了关闭笔记本电脑、捡起袜子、打开柜子以及撕开调料包等多样化活动。在这些场景中，EgoMAN模型始终能生成符合动词短语和场景空间上下文的轨迹，证明这种“推理到运动”的流水线能够很好地泛化到各类现实世界的手物交互任务中。
最后，图5展示了在相同视觉和运动语境下基于意图的条件化运动生成结果。面对不同的意图查询，EgoMAN模型会先推理出相应动作，预测出不同的航点，进而生成对应的……
---
……截然不同但均有效的6DoF轨迹（例如打开烤箱、取叉子、戳食物、开启炉灶）。这种可控的意图到运动映射能够实现多样化的手部轨迹生成，从而支持机器人学习与数据增强。
### 6. 结论
我们推出了EgoMAN数据集，这是一个用于交互阶段感知的6自由度手部轨迹预测的大规模自中心基准数据集，其中包含能够捕捉语义、空间和运动推理的结构化问答对。我们还提出了EgoMAN模型，这是一种模块化的“推理到运动”框架，它通过轨迹标记接口和渐进式训练，将高层意图与基于物理约束的6自由度轨迹相匹配。实验表明，与仅关注运动的基线模型以及VLM基线模型相比，EgoMAN取得了显著优势：Flow Matching能够生成更平滑、更稳定的轨迹；基于VLM的推理提升了语义一致性以及对新场景和意图的泛化能力；而轨迹标记接口则实现了高效推理，将基于意图的阶段感知推理与精确的低级运动生成紧密衔接。总体而言，EgoMAN为上下文动作预测迈出了实用的一步，可应用于机器人操控、语言条件下的运动合成以及基于意图的辅助系统等领域。
## 致谢
我们感谢Lingni Ma在Nymeria数据集相关方面的宝贵讨论与支持。我们也感谢Hanzhi Chen协助我们将可供性模型适配到我们的航点评估协议中。我们衷心感谢Project Aria、EgoExo4D和HOT3D团队发布这些基础数据集，正是它们为本研究提供了可能。
## 参考文献
[^1]: Project Aria Machine Perception Services. 3
[^2]: Shikhar Bahl, Russell Mendonca, Lili Chen, Unnat Jain, and Deepak Pathak. Affordances from human videos as a versatile representation for robotics. In CVPR, 2023. 2, 3, 7
[^3]: Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2.5-vl technical report. arXiv preprint arXiv:2502.13923, 2025. 2, 3, 4, 12
[^4]: Prithviraj Banerjee, Sindi Shkodrani, Pierre Moulon, Shreyas Hampali, Fan Zhang, Jade Fountain, Edward Miller, Selen Basol, Richard Newcombe, Robert Wang, et al. Introducing hot3d: An egocentric dataset for 3d hand and object tracking. arXiv preprint arXiv:2406.09598, 2024. 2, 3, 12
[^5]: Chen Bao, Jiarui Xu, Xiaolong Wang, Abhinav Gupta, and Homanga Bharadhwaj. Handsonvlm: Vision-language models for hand-object interaction prediction. Transactions on Machine Learning Research, 2025. 2, 6, 14
[^6]: Wentao Bao, Lele Chen, Libing Zeng, Zhong Li, Yi Xu, Junsong Yuan, and Yu Kong. Uncertainty-aware state space transformer for egocentric 3d hand trajectory forecasting. In International Conference on Computer Vision (ICCV), 2023. 2, 6, 14
[^7]: Homanga Bharadhwaj, Abhinav Gupta, Vikash Kumar, and Shubham Tulsiani. Towards generalizable zero-shot manipulation via translating human interaction plans. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 6904–6911. IEEE, 2024. 2
[^8]: Homanga Bharadhwaj, Roozbeh Mottaghi, Abhinav Gupta, and Shubham Tulsiani. Track2act: Predicting point tracks from internet videos enables generalizable robot manipulation. In ECCV, 2024. 3
[^9]: Homanga Bharadhwaj, Debidatta Dwibedi, Abhinav Gupta, Shubham Tulsiani, Carl Doersch, Ted Xiao, Dhruv Shah, Fei Xia, Dorsa Sadigh, and Sean Kirmani. Gen2act: Human video generation in novel scenarios enables generalizable robot manipulation. Conference on Robot Learning, 2025. 3
[^10]: Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al. $\pi_{0}$: A vision-language-action flow model for general robot control. arXiv preprint arXiv:2410.24164, 2024. 2, 3
[^11]: Hanzhi Chen, Boyang Sun, Anran Zhang, Marc Pollefeys, and Stefan Leutenegger. VidBot: Learning generalizable 3d actions from in-the-wild 2d human videos for zero-shot robotic manipulation. In CVPR, 2025. 2, 7
[^12]: Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24185–24198, 2024. 2
[^13]: Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, and Pete Florence. Palm-e: an embodied multimodal language model. In Proceedings of the 40th International Conference on Machine Learning. JMLR.org, 2023. 2, 3
[^14]: Jakob Engel, Kiran Somasundaram, Michael Goesele, Albert Sun, Alexander Gamino, Andrew Turner, Arjang Talatof, Arnie Yuan, Bilal Souti, Brighid Meredith, et al. Project aria: A new tool for egocentric multi-modal ai research. arXiv preprint arXiv:2308.13561, 2023. 3
[^15]: Aaron Grattafiori et al. The llama 3 herd of models, 2024.
[^16]: Gemma Team et al. Gemma 3 technical report, 2025. 3
### [^17]: Kristen Grauman, Andrew Westbury, Lorenzo Torresani, Kris Kitani, Jitendra Malik, Triantafyllos Afouras, Kumar Ashutosh, Vijay Baiyya, Siddhant Bansal, Bikram Boote, et al. Ego-exo4d: Understanding skilled human activity from first-and third-person perspectives. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19383–19400, 2024. 2, 3, 12
---
[^18]: Chuan Guo, Xinxin Zuo, Sen Wang, and Li Cheng. Tm2t: Stochastic and tokenized modeling for the reciprocal generation of 3d human motions and texts. In ECCV, 2022. 13
[^19]: Masashi Hatano, Ryo Hachiuma, and Hideo Saito. Emag: Ego-motion aware and generalizable 2d hand forecasting from egocentric videos. In European Conference on Computer Vision Workshops (ECCVW), 2024. 2
[^20]: Masashi Hatano, Zhifan Zhu, Hideo Saito, and Dima Damen. The invisible egohand: 3d hand forecasting through egobody pose estimation. arXiv preprint arXiv:2504.08654, 2025. 2
[^21]: Ryan Hoque, Peide Huang, David J Yoon, Mouli Sivapurapu, and Jian Zhang. Egodex: Learning dexterous manipulation from large-scale egocentric video. arXiv preprint arXiv:2505.11709, 2025. 2, 3
[^22]: Mu Hu, Wei Yin, Chi Zhang, Zhipeng Cai, Xiaoxiao Long, Hao Chen, Kaixuan Wang, Gang Yu, Chunhua Shen, and Shaojie Shen. Metric3d v2: A versatile monocular geometric foundation model for zero-shot metric depth and surface normal estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024. 7
[^23]: Simar Kareer, Dhruv Patel, Ryan Punamiya, Pranay Mathur, Shuo Cheng, Chen Wang, Judy Hoffman, and Danfei Xu. Egomimic: Scaling imitation learning via egocentric video. In 2025 IEEE International Conference on Robotics and Automation (ICRA), pages 13226–13233. IEEE, 2025. 2, 3
[^24]: Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, Quan Vuong, Thomas Kollar, Benjamin Burchfiel, Russ Tedrake, Dorsa Sadigh, Sergey Levine, Percy Liang, and Chelsea Finn. Openvla: An open-source vision-language-action model. arXiv preprint arXiv:2406.09246, 2024. 2, 3
[^25]: Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. CoRR, abs/1312.6114, 2013. 7
[^26]: Taein Kwon, Bugra Tekin, Jan Stühmer, Federica Bogo, and Marc Pollefeys. H2o: Two hands manipulating objects for first person interaction recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 10138–10148, 2021. 2
[^27]: Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9579–9589, 2024. 3
[^28]: Jason Lee, Jiafei Duan, Haoquan Fang, Yuquan Deng, Shuo Liu, Boyang Li, Bohan Fang, Jieyu Zhang, Yi Ru Wang, Sangho Lee, et al. Molmoact: Action reasoning models that can reason in space. arXiv preprint arXiv:2508.07917, 2025. 2, 3
[^29]: Qixiu Li, Yu Deng, Yaobo Liang, Lin Luo, Lei Zhou, Chengtang Yao, Lingqi Zeng, Zhiyuan Feng, Huizhi Liang, Sicheng Xu, et al. Scalable vision-language-action model pretraining for robotic manipulation with real-life human activity videos. arXiv preprint arXiv:2510.21571, 2025. 3
[^30]: Yiming Li, Ziang Cao, Andrew Liang, Benjamin Liang, Luoyao Chen, Hang Zhao, and Chen Feng. Egocentric prediction of action target in 3d. In Proceedings of the IEEE/CVF Con-
计算机视觉与模式识别会议（CVPR），2022年。2
[^31]: Yi Li, Yuquan Deng, Jesse Zhang, Joel Jang, Marius Memmel, Raymond Yu, Caelan Reed Garrett, Fabio Ramos, Dieter Fox, Anqi Li, et al. Hamster: Hierarchical action models for open-world robot manipulation. arXiv preprint arXiv:2502.05485, 2025. 2, 3, 7
[^32]: Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Annual Meeting of the Association for Computational Linguistics, 2004. 14
[^33]: Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. In The Eleventh International Conference on Learning Representations, 2023. 2, 5, 12
[^34]: Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024. 2, 3
[^35]: Miao Liu, Siyu Tang, Yin Li, and James M Rehg. Forecasting human-object interaction: joint prediction of motor attention and actions in first person video. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part I 16, pages 704–721. Springer, 2020. 2
[^36]: Shaowei Liu, Subarna Tripathi, Somdeb Majumdar, and Xiaolong Wang. Joint hand motion and interaction hotspots prediction from egocentric videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 2
[^37]: Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2017. 12
[^38]: Hao Luo, Yicheng Feng, Wanpeng Zhang, Sipeng Zheng, Ye Wang, Haoqi Yuan, Jiazheng Liu, Chaoyi Xu, Qin Jin, and Zongqing Lu. Being-h0: vision-language-action pretraining from large-scale human videos. arXiv preprint arXiv:2507.15597, 2025. 2, 3
[^39]: Junyi Ma, Xieyuanli Chen, Wentao Bao, Jingyi Xu, and Hesheng Wang. Madiff: Motion-aware mamba diffusion models for hand trajectory prediction on egocentric videos, 2024. 2
[^40]: Junyi Ma, Jingyi Xu, Xieyuanli Chen, and Hesheng Wang. Diff-ip2d: Diffusion-based hand-object interaction prediction on egocentric videos. arXiv preprint arXiv:2405.04370, 2024.2
[^41]: Junyi Ma, Wentao Bao, Jingyi Xu, Guanzhong Sun, Xieyuanli Chen, and Hesheng Wang. Novel diffusion models for multi-modal 3d hand trajectory prediction, 2025. 2, 6, 14
[^42]: Lingni Ma, Yuting Ye, Fangzhou Hong, Vladimir Guzov, Yifeng Jiang, Rowan Postyeni, Luis Pesqueira, Alexander Gamino, Vijay Baiyya, Hyo Jin Kim, Kevin Bailey, David Soriano Fosas, C. Karen Liu, Ziwei Liu, Jakob Engel, Renzo De Nardi, and Richard Newcombe. Nymeria: A massive collection of multimodal egocentric daily motion in the wild. In the 18th European Conference on Computer Vision (ECCV), 2024. 2, 3, 12
### [^43]: Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint arXiv:2306.05424, 2023. 3
---
[^44]: Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018. 4, 12
[^45]: OpenAI and et al. Gpt-4 technical report, 2024. 3
[^46]: Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Annual Meeting of the Association for Computational Linguistics, 2002. 14
[^47]: Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron Courville. Film: visual reasoning with a general conditioning layer. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence. AAAI Press, 2018. 13
[^48]: Karl Pertsch, Kyle Stachowicz, Brian Ichter, Danny Driess, Suraj Nair, Quan Vuong, Oier Mees, Chelsea Finn, and Sergey Levine. Fast: Efficient action tokenization for vision-language-action models. arXiv preprint arXiv:2501.09747, 2025. 2
[^49]: Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, et al. $\pi_{0.5}$: A vision–language–action model with open-world generalization. arXiv preprint arXiv:2504.16054, 2025. 2, 3
[^50]: Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PmLR, 2021. 4, 13
[^51]: Hanoona Rasheed, Muhammad Maaz, Sahal Shaji, Abdelrahman Shaker, Salman Khan, Hisham Cholakkal, Rao M. Anwer, Eric Xing, Ming-Hsuan Yang, and Fahad S. Khan. Glamm: Pixel grounding large multimodal model. The IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024. 3
[^52]: Oriane Siméoni, Huy V. Vo, Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, Cijo Jose, Vasil Khalidov, Marc Szafraniec, Seungeun Yi, Michaël Ramamonjisao, Francisco Massa, Daniel Haziza, Luca Wehrstedt, Jianyuan Wang, Timothée Darcet, Théo Moutakanni, Leonel Sentana, Claire Roberts, Andrea Vedaldi, Jamie Tolan, John Brandt, Camille Couprie, Julien Mairal, Hervé Jégou, Patrick Labatut, and Piotr Bojanowski. DINOv3, 2025. 5
[^53]: Xin Wang, Taein Kwon, Mahdi Rad, Bowen Pan, Ishani Chakraborty, Sean Andrist, Dan Bohus, Ashley Feniello, Bugra Tekin, Felipe Vieira Frujeri, Neel Joshi, and Marc Pollefeys. Holoassist: an egocentric human interaction dataset for interactive ai assistants in the real world. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 20270–20281, 2023. 2
[^54]: Junjie Wen, Yichen Zhu, Jinming Li, Zhibin Tang, Chaomin Shen, and Feifei Feng. Dexvla: Vision-language model with plug-in diffusion expert for general robot control. arXiv preprint arXiv:2502.05855, 2025. 2, 3
[^55]: Ruihan Yang, Qinxi Yu, Yecheng Wu, Rui Yan, Borui Li, An-Chieh Cheng, Xueyan Zou, Yunhao Fang, Xuxin Cheng,
邱日昭等人。Egovla：从自中心人类视频中学习视觉-语言-动作模型。arXiv预印本arXiv:2507.12440，2025年。3
[^56]: Chengbo Yuan, Chuan Wen, Tong Zhang, and Yang Gao. General flow as foundation affordance for scalable robot learning. arXiv preprint arXiv:2401.11439, 2024. 3
[^57]: Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with BERT. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. 14
[^58]: Wenqi Zhang, Mengna Wang, Gangao Liu, Huixin Xu, Ywei Jiang, Yongliang Shen, Guiyang Hou, Zhe Zheng, Hang Zhang, Xin Li, Weiming Lu, Peng Li, and Yueting Zhuang. Embodied-reasoner: Synergizing visual search, reasoning, and action for embodied interactive tasks. arXiv preprint arXiv:2503.21696, 2025. 2, 3
[^59]: Yi Zhou, Connelly Barnes, Jingwan Lu, Jimei Yang, and Hao Li. On the continuity of rotation representations in neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5745–5753, 2019. 3, 4
### [^60]: Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Wohlhart, Stefan Welker, Ayzaan Wahid, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. In Conference on Robot Learning, pages 2165–2183. PMLR, 2023. 2, 3
---
### A. 附录概要
在本附录中，我们提供了：
1. 我们系统的视频演示，包括具有代表性的交互案例（见章节B）。
2. EgoMAN模型流程及渐进式训练策略的实现细节（见章节C）。
3. 主论文中所呈现的消融实验结果的详细分析（见章节D）。
4. 预测的手部轨迹与动作动词之间的语义一致性评估结果，以及与各基线模型的对比（见章节E）。
5. 不同参数规模的推理模块之间的对比，包括EgoMAN-QA准确率及轨迹预测性能（见章节F）。
6. 数据标注中使用的代表性提示示例（见章节G）。
7. 局限性与未来工作方向（见章节H）。
### B. 视频
我们的视频直观展示了EgoMAN的核心贡献，涵盖了数据集、模型架构以及定性演示内容。视频首先介绍了EgoMAN数据集，随后重点讲解了完整的EgoMAN流程，该流程包含推理模块、运动专家以及端到端的6自由度轨迹生成环节（通过轨迹标记接口实现从推理到运动的衔接）。
在数据集概览部分，我们展示了与EgoMAN中频率最高的十二个动词相对应的各种手-物体交互示例（例如抓取、打开、放置、倾倒、搅拌）。所展示的视频片段来自多个数据源，如EgoExo4D [^17]、Nymeria [^42]和HOT3D-Aria [^4]，它们体现了这些交互发生在具有自然噪声、杂乱环境和复杂视角的真实日常场景中。这些示例展现了数据集的规模、多样性和难度，从而凸显出构建鲁棒的、基于意图条件的3D手部轨迹模型的必要性。
定性结果部分重点展示了EgoMAN生成受意图引导的轨迹的能力。我们首先展示了各种场景下的数十个代表性案例：比如厨房场景中的搅拌牛奶和关闭炉灶、家居场景中的捡起袜子和开门、工作场景中的合上笔记本电脑和拔插电线，以及HOT3D场景中的操控碗或操控沙拉酱瓶。对于每个案例，视频会显示：(1)原始输入图像，(2)意图文本，(3)来自推理模块的中间航点预测结果，以及(4)最终的6自由度轨迹输出。这些预测结果既会在静态输入图像上可视化，也会叠加在未来的自拍视频帧上，以便更清晰地呈现空间精度和运动质量。
在所有演示中，EgoMAN都能准确预测目标物体周围的接触点和终点航点，生成的3D轨迹遵循符合预期语义的真实操作路径。虽然某些开放式任务（如开门、捡袜子）在最终姿态或时间上可能存在细微差异，或者非操作手相对于单一真实实例会有轻微偏差，但操作手的预测轨迹仍能在语义上与预期目标保持一致。这些结果凸显了EgoMAN在各类场景和交互类型中生成可靠的、受意图驱动的6自由度手部轨迹的能力。
我们还展示了目标导向的轨迹生成结果：同一输入图像搭配不同的意图描述时，EgoMAN模型仍能预测出与预期目标相符的不同轨迹，即便是在未见过的环境中也是如此。
请访问我们的项目网站，查看更多不同交互场景下的轨迹预测结果。
### C. 实现细节
推理模块。该推理模块采用bf16精度优化，并使用AdamW优化器以及余弦学习率衰减策略，其中引用了[^37]。视觉编码器和多模态投影层被冻结。我们设置的基础学习率为$1 \times 10^{-5}$，预热比例为0.02，权重衰减为0.05，最大梯度范数限制为1.0，在$8 \times$块NVIDIA A100 80GB GPU上以256的批量大小进行训练。训练共进行2个周期，涉及约100万个EgoMAN QA样本。图像采用动态缩放处理，其中max_pixels=50176，min_pixels=784。如果输入问题中提供了过往运动信息，我们会取最近5个手部的轨迹点，这些点以10 fps的频率进行分词处理；否则则使用初始化为零的运动历史数据。我们采用4层MLP从运动数据中提取特征，随后将这些特征输入Qwen2.5-VL模型，即[^3]。专门设计的航点解码器是轻量级的ReLU MLP，隐藏维度为768，用于预测时间戳、3D位置和6DoF旋转参数。动作语义解码器则是一个单层MLP（维度为768）。当批次中有效样本数量低于$\kappa=10$时，我们采用余弦相似度损失函数；否则应用InfoNCE损失，详见[^44]。损失权重分别设置为$\lambda_{wp}=0.3$和$\lambda_{act}=0.1$，内部权重则为$\lambda_t=1.0$、$\lambda_{3D}=2.0$、$\lambda_{2D}=0.5$、$\lambda_r=0.5$以及$\lambda_{geo}=0.15$。对于rot6D损失，我们应用Huber损失，相关参数为$\beta=0.2$；位置项的相关参数则为$\beta_{3D}=0.07$和$\beta_{2D}=0.02$。测地旋转损失仅应用于可见的航点。时间调制功能通过高斯时间窗口实现，相关参数为$\sigma_{time}=3.0$。
运动专家模块。我们基于约1.7万条轨迹数据，对基于流匹配（FM）[^33]的运动解码器进行预训练。输入数据包括DINOv3图像特征、真实动作描述、航点标记以及过往手腕运动信息。运动序列以10 fps的频率采样，未来预测范围最多为50步（即5秒）。该FM架构的隐藏维度为768，包含6个编码器层和6个解码器层Transformer，共有8个注意力头。此外还采用了正弦时间嵌入方式。
---
Pretrain Reason
FM
WP
ADE $\downarrow$
FDE $\downarrow$
DTW $\downarrow$
Rot $\downarrow$
✗
✗
✗
0.273
0.308
0.260
51.79
✓
✗
6DoF
0.215
0.255
0.198
43.03
✗
✓
✗
0.162
0.225
0.148
36.24
✓
✓
✗
0.161
0.224
0.147
35.90
✓
✓
Emb
0.150
0.210
0.138
34.02
✓
✓
6DoF
0.151
0.206
0.137
33.88
Table 4. Ablation on EgoMAN-Unseen (K=1). Lower is better. Reason and FM pretraining with 6DoF waypoints yield the highest accuracy.
Figure 6. Data efficiency results. ADE/FDE (m), best-of-10. The static baseline repeats the last observed hand location. Without pretraining, errors of EgoMAN-ACT rise sharply under limited data, while EgoMAN maintains strong performance even at 20% data, highlighting the benefit of waypoint-based Reasoning Module and pretraining.
通过航点实现轨迹-标记接口（WP）。WP为视觉语言模块与6自由度运动专家之间提供了结构化的接口。以预测的WP为条件进行约束，可提升模型的准确性和稳定性（ADE从0.161降至0.151，DTW从0.147降至0.137，旋转误差从35.90°降至33.88°）。若没有WP，模型性能会降至接近EgoMAN-ACT的水平，这表明学习到的WP强化了推理预训练的作用，其效果超出了隐式的语义嵌入。若用解码器的最终隐藏状态（Emb）替代6自由度航点，带来的差异微乎其微（FDE仅差约0.4厘米；其他指标几乎无变化）。
运动预训练（FM）。在保留推理预训练的前提下移除FM，会导致性能明显下滑（ADE从0.150升至0.215）。没有FM时，模型必须通过隐式语义接口同时学习语义和运动特征，进而导致长时序预测结果更嘈杂，旋转误差也更大。若同时移除FM和推理预训练，性能会进一步恶化（ADE达到0.273，旋转误差为51.79°）。尽管推理预训练和航点学习能缓解这一问题，但先对物理上合理的运动进行FM预训练，再与推理模块联合微调，能在所有指标上取得最显著的提升。
#### E. 运动与文本对齐
我们通过训练运动编码器，将手部轨迹映射到预先计算好的动词文本嵌入空间，并利用类CLIP的对比损失函数来评估轨迹与动作动词之间的语义对齐程度 [^18] [^50]。我们报告Recall@3的结果。
---
Method
R@3 $\uparrow$
FID $\downarrow$
USST $^{*}$ [^6]
15.0
0.22
MMTwin $^{*}$ [^41]
22.9
0.86
HandsonVLM $^{*}$ [^5]
27.9
0.10
FM-Base
39.7
0.05
EgoMAN
43.9
0.04
Table 5. Motion-to-Verb Text Retrieval. Train one encoder; evaluate verb text-motion relevance over 239 verb candidates.
Table 6. Effect of model scale on spatial reasoning, semantic alignment, and text QA on EgoMAN Unseen benchmark. We evaluate (i) waypoint spatial reasoning via 3D location, time, and rotation errors, (ii) semantic embedding alignment using R@3 (computed over 2,844 GT action-embedding candidates) and mean Pearson correlation, and (iii) semantic text QA using BERTScore, BLEU, and ROUGE. Best values are bolded; second-best are \underline{underlined}. Spatial reasoning performance saturates early, and models larger than 2B/3B provide consistently stronger performance. Semantic alignment benefits from larger models, with Qwen2.5-VL outperforming Qwen3-VL, while text QA remains relatively stable across scales, with Qwen3-VL slightly outperforming Qwen2.5-VL.
Model
Params
EgoMAN Unseen
HOT3D OOD
ADE $\downarrow$
FDE $\downarrow$
DTW $\downarrow$
Rot $^{\circ}$ $\downarrow$
ADE $\downarrow$
FDE $\downarrow$
DTW $\downarrow$
Rot $^{\circ}$ $\downarrow$
Qwen2.5-VL
3B
0.128
0.184
0.115
33.16
0.146
0.221
0.135
35.82
Qwen2.5-VL
7B
0.124
0.179
0.111
32.75
0.141
0.217
0.130
35.09
Qwen3-VL
2B
0.130
0.186
0.118
33.48
0.142
0.216
0.132
35.62
Qwen3-VL
4B
0.123
0.178
0.111
32.63
0.139
0.212
0.128
34.65
Qwen3-VL
8B
0.122
0.177
0.110
32.31
0.140
0.214
0.129
34.62
Table 7. Effect of Reasoning Module scale on trajectory prediction. Best results are bolded and second-best are $\underline{\text{underlined}}$. Larger reasoning models produce consistently more accurate 6-DoF trajectories on both EgoMAN Unseen and HOT3D OOD, with Qwen3-VL scaling smoothly and the 4B model offering an excellent speed-accuracy trade-off.