# Nymeria：一个庞大的野外自我中心日常运动多模态数据集 马玲妮$^{1}\textcircled{k}$$\textcircled{p}$, Yuting Ye$^{1}$$\textcircled{p}$、洪方舟$^{2}$$\textcircled{p}$, Vladimir Guzov$^{3}$$\textcircled{p}$、蒋一峰$^{4}$$\textcircled{p}$, Rowan Postyeni$^{1}$$\textcircled{p}$、路易斯·佩斯奎拉$^{1}$$\textcircled{p}$, Alexander Gamino$^{1}$$\textcircled{p}$、维杰·拜亚$^{1}$$\textcircled{p}$, Hyo Jin Kim$^{1}$$\textcircled{p}$、凯文·贝利$^{1}$$\textcircled{p}$, David S. Fosas$^{1}$$\textcircled{p}$、刘凯伦$^{4}$$\textcircled{p}$, Ziwei Liu$^{2}$$\textcircled{p}$、雅各布·恩格尔$^{1}$$\textcircled{p}$, Renzo De Nardi$^{1}$$\textcircled{p}$以及理查德·纽科姆$^{1}$$\textcircled{p}$ $ ^{1} $ Meta Reality Labs Research, Redmond, USA $ ^{2} $ Nanyang Technological University, Singapore, Singapore $ ^{3} $ University of Tübingen and Max Planck Institute for Informatics, Tübingen, Germany $ ^{4} $ Stanford University, Stanford, USA Abstract. We introduce Nymeria - a large-scale, diverse, richly annotated human motion dataset collected in the wild with multiple multimodal egocentric devices. The dataset comes with a) full-body ground-truth motion; b) multiple multimodal egocentric data from Project Aria devices with videos, eye tracking, IMUs and etc.; and c) an third-person perspective by an additional “observer”. All devices are precisely synchronized and localized in one metric 3D world. We derive hierarchical protocol to add in-context language descriptions of human motion, from fine-grain motion narrations, to simplified atomic actions and high-level activity summarization. To the best of our knowledge, Nymeria dataset is the world's largest human motion in the wild; first of its kind to provide synchronized and localized multi-device multimodal egocentric data; and the world's largest motion-language dataset. It provides 300 hours of daily activities from 264 participants across 50 locations, total travelling distance over 399 Km. The language descriptions contain 310.5K sentences in 8.64M words from a vocabulary size of 6545. To demonstrate the potential of the dataset we evaluate several SOTA algorithms for egocentric body tracking, motion synthesis, and action recognition. Keywords: human motion · egocentric · multimodal · dataset F. Hong, V. Guzov and Y. Jiang—Work done during internships at Meta Reality Labs Research. --- ![Image](images/646f9037a212ffd306fb8a32e0723d2205d1a5668da39b98c774961bd929e8a2.jpg)
Fig. 1. A glimpse of Nymeria dataset. The figure shows example indoor and outdoor activities captured on a campus, where the point clouds and trajectories are the SLAM output by tracking all egocentric devices i.e. the glasses and wristbands. Each sub-figure is a motion clip from a different participant, where the top left gives the latest egocentric view, the right is the 3D localized full-body motion synchronized with the headset and the bottom left provides an auxiliary third-person view.
## 1 Introduction The advent of AI is leading to a surge of smart glasses [1-3,5,6,9,11,30] and other wearables. These devices not only provide seamless access to LLM-based AI assistants, but also are multimodal data-capture vehicles that provide immediate and long-term personalized context, allowing AI assistants to evolve into the next generation of human-centric contextualized AI, and unlock a new era of contextualized computing combined with AR/VR technology. In this paradigm, the wearer's own body motion and action provides important context. The problem is challenging to solve given insufficient self-observations from wearable devices. In reality, a critical limiting factor to advance research is data. Currently, people either use real data with limited scale, diversity and modality [35,39,46,56,68,78,105,112] or simulations that lack realism and completeness [12,13,45,47,59,98]. There are three key technical challenges to create large motion datasets. Obtaining long-term ground-truth motion in the wild. There are two main motion capture (mocap) approaches. Vision-based solutions such as the ones relying on optical markers [56,68,78] or cameras [35,55,62,100,112] are adversely affected by line-of-sight visibility, and require a complex multi-camera-setup to cover limited range of motions inside a constrained volume. Inertial-based solutions [7,10] suffer from dead-reckoning and thus are inferior in global positioning accuracy [69,88]. Multi-device alignment. Combining multiple capture devices or ground-truth systems requires accurate temporal and spatial alignment, which can be challenging with off-the-shelf hardware that cannot be modified or lack support. --- for universal synchronization protocols. The existing datasets work around this problem using visual cues [35,55,112] or audio [39]. Such approaches offer limited accuracy and reliability. In order to counter clock drift for long recordings, they can be intrusive and interrupt the natural activity. Consequently, existing datasets mostly record short motions (cf. Table 2). Data processing and annotations. These are critical for a dataset to develop its full potential. In addition to the body motion, device localization and scene representation, we believe natural language descriptions are crucial for future research directions. Existing work provides simple descriptions or action labels without scene context [26,36,82], and is of significantly smaller scale compared to the text corpus for training LLMs [16,77,99]. To fill the gap and accelerate the research, we introduce Nymeria - the world largest human motion dataset with 300 hours in-the-wild daily activities from 264 participants performing 20 scenarios from 50 indoor and outdoor locations. With average 15-min per recording, the data captures natural activities with spontaneous unscripted actions and authentic interactions. Nymeria is first-of-its-kind dataset recorded with multiple multimodal egocentric devices. Participants worn XSens mocap suit [7], Project Aria glasses [30] and Aria-alike wristbands to record egocentric motion, RGB, grayscale, eye tracking (ET) videos, inertial measurement units (IMUs), magnetometer, barometer and etc. Devices are synchronized with a non-intrusive hardware solution with sub-millisecond accuracy and localized into a single metric 3D leveraging Project Aria Machine Perception Service (MPS) [8]. We also developed novel algorithms to retarget XSens skeleton motion into a full parametric human model and correct the global drift with optimization. To connect human motion with natural languages, we developed a coarse-to-fine narration schema to describe in-context human motion at different granularity. With 310.5K sentences and 8.64M words from 6545 vocabulary size, Nymeria stands out as the world's largest motion-language dataset (Table 1). ## 2 Related Works Motion Datasets – Scale, Multimodal, in-the-Wild and Perspectives. Datasets are crucial ingredients in developing algorithms, particularly machine learning approaches. AMASS [68] is a pioneering effort in large motion dataset, which unifies multiple marker-based datasets into SMPL [64]. While AMASS provides diverse human motion, it lacks scene context. Recent works [46,56,78] extend the solution to include objects. Monocular [18,33,50,62,67,80,87,89,93,107]
Table 1. Highlight statistics of Nymeria dataset. We capture 1200 sequences of 300-hour daily activity from 264 people performing 20 scenarios at 50 locations with 399 Km traveling distance, 260M body poses, 201.2M images, 11.7B IMUs, 10.8M gazes. The motion narrations contain 310.5K sentences in 8.64M words from 6545 vocabulary.
SeqQtyPtsSceLocPoseImgIMUGazeTrajSentWordVoc
1200300h2642050260M201M11.7B10.8M399 Km310.5K8.64M6545
--- and multi-view cameras [35,48,49,55,112] are common mocap alternatives. For monocular camera, Motion-X [62] stands out as a comprehensive large collection of whole-body motion with facial expressions and hand gestures. For multi-view setting, EgoExo4D [35] stands out as a large dataset of skilled activities. Vision-based algorithms require good line-of-sight. Consequently, they are better suited to record motion with clear body observations bounded by a volume. To capture data in the wild, mocap suit is a popular candidate [22,39,52,53,56,58,69,100,105]. To address dead-reckoning for inertial-based tracking, previous works fuse IMUs with vision [69,100,109], optimize motion with 3D scenes [39,58] and limit the range of motion locally [105]. Simulation as a more scalable solution, offers valuable supplement to real data. Existing works leverage gaming engine for character animation [19], render marker-based mocap with virtual characters and scenes [12,15,46], simulate motion with VR [13] or by generative algorithms [59] etc. Simulations often struggle to present noise characteristics of real data, result in domain gaps. While many solutions are developed for third-person views, egocentric motion datasets remain sparse, leaving a gap to the recent advance of egocentric perception [6,30,34]. Existing works focus on hands with object-interaction [14,24,25,57,90], lack ground-truth [34] or parametric body motion [35], or is limited in scale, diversity, and modality [12,39,55,58,78,102,103,112]. Nymeria is designed to fill the gaps with significant delta to existing datasets (cf. Table 2).
Table 2. Human motion datasets by releasing date. Columns 2 to 5 show activity in hour (q/h), pose frames in millions (p/M), mean sequence duration in minute ( $ \mu $m) and number of participants (pp). We then compare language narrations w.r.t. number of descriptions (tt/K) and vocabulary size (voc). The remaining columns mark following features: parametric model for motion representation (pm), egocentric head-mounted device (hd), third-person perspectives (3p), wristbands (wd), global positioning (gp), outdoor scene (od), gaze (gz), 3D scene representations (sr), multi-people scenarios (mp) and human-human interactions (hh). Note EgoExo4D [35] reports 1422 h by summing per camera recording time, where the total activity is 180 h, with 88.8 h annotated with MSCOCO keypoints. The underline numbers are reported for the full dataset.
datasetq/hp/M\mu /mpptt/Kvocpmhd3pwdgpodgzsrmphh
AMASS [68]420.90.22346✓
HPS [39]4.50.58.27✓✓✓✓✓✓✓
EgoBody [112]20.4136✓✓✓✓✓✓✓
HML3D [36]28.62.90.12-45.05371✓
EgoHuman [55]3.50.40.57✓✓✓✓✓✓
MotionX [62]14415.60.11-81.1-✓✓✓
DivaTrack [105]16.53.60.1322
EgoExo4D [35]88.89.62.67404324405✓✓✓✓✓
ParaHome [56]7.33564.430✓✓✓✓
LaHuman [22]3-0.51-12.3-✓✓✓✓✓✓✓
Nymeria (ours)30026015264310.56545✓✓✓✓✓✓✓✓✓✓
--- Motion with Natural Language. Adding language descriptions to motion leads to unique perspective in motion understanding, especially given the powerful capability of large language models (LLMs). Early datasets [17,32,38,40,42,92] offer sparse annotations as action categories or semantic labels [83]. KIT [82] is the first attempt in using complete sentences to describe locomotion. HumanML3D [36] enriches HumanAct12 [38] and AMASS [68] with multiple descriptions per motion clip. Motion-X [62] leverages narration algorithm to obtain large-scale fine-grained descriptions at sequence and frame levels. Compared to available text corpus, motion-language data is rather sparse with brief text on brief motion [22]. Ego4D [34] and EgoExo4D [35] offer large-scale descriptions of atomic actions, however, ground-truth motion is missing or without parametric representations. In our work, we provide in-context coarse-to-fine narration by annotators, with the amount of data a magnitude larger than prior works. Since Closely related to tracking is motion synthesis. Egocentric Body Tracking, Synthesis and Action Recognition. Tracking one's own body motion with wearable devices is challenging. Early approaches leverage dense body-worn IMUs [41,69,70]. Recent methods reduce IMUs to improve practicality [47,106,110,111], assume IMU-enabled AR/VR devices [44,45], combine IMU with cameras [39,102,103,109,117], mobile phones [72] and wristbands [58]. While sparse sensors are more practical, our work centers on constructing an offline dataset to serve "ground truth". Consequently, we use dense IMUs to ensure high accuracy and correct global drift via optimization. Since full-body motion is ill-posed given insufficient observation from egocentric headset, motion synthesis is often used to produce plausible motion. To condition the generation, research explores sparse motion measures [20,104,105], headset motion [60] and eye gaze [116] and text [37,81,95]. The success of diffusion models lead to active development in text-driven motion synthesis [21,23,51,91,96,115]. Similarly, LLMs inspire novel motion understanding algorithms [31,43,65,114] that tightly couple motion with natural language. Our work constructs a dataset with rich hierarchical narrations to inspire further breakthroughs in the field. ## 3 Building Nameria Dataset ### 3.1 Data Collection Setup Hardware. Each participant wears a mocap suit, a pair of glasses, two wristbands, and a synchronization device (cf. Fig. 2. XSens MVN Link [7] is adopted for mocap, which is a tight-fit body suit wired with 17 inertial trackers and a magnetometer. MVN Link supports on-device recording, making it ideal to collect in-the-wild data. We use Project Aria glasses [30] as a lightweight headset to record multimodal data. The sensor suite includes 1 RGB camera, 2 grayscale peripheral cameras, 2 ET cameras, 2 IMUs, 1 barometer, 1 magnetometer, 7 microphones, 1 thermometer, GNSS, WiFi and BT. We repackaged the electronics and sensors of Project Aria into a new wristband device called miniAria, in order to closely resemble current AR/VR headsets and provide data to better --- ![Image](images/9e3fa392ba584c97d059a05cde3927dcc6d20117f7f94484254fac096e238878.jpg)
(a)
![Image](images/b0a54c61876a852513629b7331039c75c488ff54be2f6a57c6f5d3d9dfec3a6a.jpg)
(b)
![Image](images/c864bfbdaa03b8dd6ef02d2c6f538fddf6d5eaaed84d95548b377b3512fa0f61.jpg)
(c)
![Image](images/bc73326c65a31bd4e45979538fc6e775e56ac2f6afdd8eeafbe17b741a1a36fc.jpg)
(d)
Fig. 2. Capture setup. (a) A full-dressed participant. (b) The set of hardwares including Project Aria glasses, two miniAria wristbands and synchronization device. (c) The sensor suite of Project Aria and (d) the miniAria wristband.
constrain body tracking algorithms. The wristband matches Project Aria's sensing ability, with exclusion of microphones, barometer, magnetometer and ET cameras. The supplementary provides detailed sensor configuration and recording profiles. Synchronization. Project Aria can record an externally provided time signal to aid synchronization. We further enable MVN Link to accept the same signal. A synchronization device is developed to supply the timestamps for all devices, which can optionally receive time from a wireless server located in radio range ( $ \sim $100 m). This facilitates synchronizing multiple devices with sub-millisecond accuracy. The alignment between XSens and Aria is within 1 motion frame i.e. 4.2 ms. To capture multiple people simultaneously, we replicate the described setup per participant and leverage a common time server. Recording Protocols. Data collection is managed by 2–3 onsite operators. In addition to participants, a trained observer wearing Project Aria is present to record participants from third-person perspective. All people interact naturally as per activity requires, contributing to rich dynamics as opposed to staged motion. To complete each recording, participants first perform a brief mocap calibration, then gaze calibration, and finally 15–20 min activity. We collect 4–8 recordings per person, where a bulk of data is captured at family houses. Scenarios. We define 20 scenarios (cf. examples in Fig.1 and Fig.3). For indoor activities, scenarios include cooking, working, entertaining, searching objects etc. For outdoor activities, scenarios include hiking, biking, dining, sports etc. To encourage natural interactions and authentic motions, participants are instructed with high-level guidelines e.g. "grab food in the cafeteria and eat on the patio". Operators also prompt in-context actions to increase dynamics. Privacy Considerations. We follow Project Aria research guideline for responsible innovation. Prior to data collection, consents were obtained from participants and home owners for recording and data usage. The SOTA de-identification algorithm EgoBlur [85] is used to blur faces and license plates for all videos. --- ![Image](images/b5d55414339fbff05cedd2a6ebf46f7c9333a18ab8493355168e9c1e3ded31a6.jpg)
Fig. 3. Diverse scenarios by diverse people. We show different participants performing different indoor/outdoor activities at different locations. In each subfigure, we show an egocentric view on the top left, a third-person view on the bottom left, and motion rendering on the right.
### 3.2 Data Processing To process data, we first use XSens software to obtain the skeleton motion, and Project Aria MPS [8] for device localization, scene representation and gaze estimation. Then motion is retarget to a parametric human model [4] and registered into the coordinates of Aria devices via optimization. Full-Body Mocap and Retargetting. We record motion at 240 Hz, following the recommended procedures: 1) carefully measuring body dimensions of participants; 2) performing calibration prior to every recording; and 3) processing XSens with the highest quality with single- or multi-floor specification. XSens represents the body motion as the global transformation and 3D local joint angles of a template skeleton. The skeleton consists of 23 segments, where each segment matches the measured body dimensions of the subject. In addition, a set of $ K = 79 $ anatomical landmarks are defined on the skeleton model [74]. Their global positions, $ \{\mathbf{p}_i\}_{K} $, can be computed by evaluating the forward kinematics at each frame. We utilize these landmarks to retarget the body motion onto an anatomically-inspired human model for improved realism and visual validation. Our human model is parameterized by $ \{\boldsymbol{\theta}, \phi\} $, where the pose parameters --- ![Image](images/cfec69ad296b8a49edec9120cacc7da10a50362d101c06c9e2a67f4c976df59b.jpg)
Fig. 4. Global aligned trajectories and point clouds by locations. We show examples of split-level residential house with gardens, where each contain $ \approx5 $h of recording. The left shows the top-down views of accumulated trajectories where red, green and blue indicate the head, the left and right wrist. On the right we sample closed-up views where the clusters emerge from human 3D motion distribution.
$ \theta $ define the global transformation and local joint angles, and the shape parameters $ \phi $ represent a global body scale and individual bone length. Given a motion of N frames, we solve the following inverse kinematics optimization problem: $$$ \arg\min_{\phi,\pmb{\theta}_{0}\cdots T-1,\mathbf{v}^{0\cdots K-1}}\sum_{t=0}^{N-1}\sum_{i=0}^{K-1}\left\|\boldsymbol{T}^{i}(\phi,\pmb{\theta}_{t})\mathbf{v}^{i}-\mathbf{p}_{t}^{i}\right\|^{2}, $$ where $ \mathbf{v}^{i} $ is the local offset of the $ i $th landmark defined on our model, and $ T^{i} $ is the global transformation of its parent joint. We initialize $ \mathbf{v}^{i} $ by manually placing them on the model. The supplementary provides more details about our human model and motion retargeting. 6DoF Localization and Mapping with Global Alignment. Data recorded at the same location are globally aligned into a single metric 3D world via Project Aria MPS [8], which employs state-of-the-art visual inertial odometry (VIO), SLAM and mapping algorithms [29,73,75]. In a nutshell, first SLAM is run for each individual recording independently. Subsequently, the resulting maps are loop-closed and jointly optimized via visual-inertial bundle adjustment. The output are highly accurate 1KHz trajectories (cf. Fig. 4 and supplementary) – allowing for example to visualize head- and wrist-motion-clusters respectively. --- ![Image](images/ef1fa890008e66cbd29edc8eda077fe6df301962d2aae98960df5d5efa45654d.jpg)
Fig. 5. End-to-end quality assessment. We uniformly sample a 20 min recording over 1.5 Km moving distance and project skeleton in observer's camera. The rendering and image aligns well, due to precise tracking and synchronization.
Given the device trajectories, we align the body poses into this same reference coordinates by correlating the dead-reckoned trajectory from XSens. Since the latter accumulates significant drift, there exists no static global transformation to align them. Instead, we assume a constant transformation $ T_{HD} $ between the user's head segment from XSens H and the Aria device D (Aria is firmly held in place with straps and participants are asked to avoid adjusting the glasses during the recording). We then cut the trajectory into a large number of short 4.2ms segments, and solve the following optimization problem $$ \arg\min_{T_{HD}}\sum_{t}\left\|\log\left(\left(T_{OH}^{t}{}^{-1}T_{OH}^{t+1}\right)\cdot\left(T_{HD}T_{WD}^{t}{}^{-1}T_{WD}^{t+1}T_{HD}{}^{-1}\right)^{-1}\right)\right\|^{2}, $$ 其中O是XSens的漂移里程计坐标系，W则是MPS输出的世界坐标系。这属于一个存在闭式解的HandEye标定问题[^94]。需要注意的是，该公式通过对比局部速度来对齐大量局部运动片段。实际操作中，Aria在录制过程中并非完全刚性，这也是导致误差的主要原因。通过滚动窗口优化可提升精度。图5从定性角度对我们的多设备定位、XSens运动配准以及时间同步效果进行了端到端评估。 ## 3.3 上下文运动语言描述 为了建立身体运动、自然语言与动作识别之间的联系，我们让人类标注员观看数据集的回放视频，从而为上下文中的人体运动撰写文本描述。标注员会将视频分割成若干片段，通过回答预设问题来完成描述撰写。为了让标注员对运动有全面的理解，回放视频会同步展示自我中心视角视频、第三人称视角视频以及渲染出的3D人体运动画面。 我们定义了三项标注任务，以实现从粗到细的运动描述，并合理扩展人力投入。最精细的层级是运动叙述，用于描述详细的身体姿态，例如运动方向、速度、互动情况以及注意力焦点。接下来是原子动作标注。与运动叙述相比，我们鼓励标注员尽可能使用动词，例如用“跳舞”代替“双臂摆动，同时身体向右旋转，双腿微微分开”。 --- ![Image](images/8b4a94156bc4139dacafa73c86924a126c79d2c37e1d3817299e2a304f9c1353.jpg)
Fig. 6. Hierarchical narration example. (left) motion narration of 5s clip – “C straightens the body to receive a badminton racket from his peer. C extends the right arms towards the bat, with the left arm slightly bent. C takes a step forward with the left foot. C focuses on the racket.” (middle) Atomic action of 5s clip – “C lifts the right leg to kick a soccer ball in front of a building.” (right) Activity summarization of 30s clip – “C bikes on the road with a peer”.
![Image](images/e6227a8b306b7d8c60ad862b0e590972e155c9fabb669737db256363ed031df4.jpg) 117.2K、2.72M、3739） ![Image](images/416c95dfb2e0c8e78800d50c06b9aafa88ff23008cc42e01c764588c70300a6e.jpg) 原子动作（207小时、170.6K、5.47M、5129） ![Image](images/e93f6f5f8fd23cc84bfdc9ed915f3cb2490ec52b713bdf5ba790bae68c5c8c35.jpg) 摘要生成（196小时、22.6K、0.45M、3168） ![Image](images/8944da7abfa456eaa298349779db22e0680d60c81a1b1be10ea8467c8e99f007.jpg) 综合任务（230小时、310.5K、8.64M、6545）
Fig. 7. Distribution of language descriptions. The word cloud visualizes annotations of each task in separation and with all data combined. The tuple (N, X, Y, Z) means N hours of data is described by X sentences, Y words, and Z vocabulary size.
前两项任务针对时长小于5秒的片段开展。最后一项任务——活动摘要生成，是为时长超过30秒的活动生成一句话总结。图6展示了各项标注任务的示例。该图体现了为标注员提供三种同步视图的优势：自我中心视角能捕捉近距离的手部与物体互动情况，而第三人称视角和运动渲染则有助于标注员全面理解动作内容。 ## 3.4 统计数据 数据统计。我们收集了来自264名参与者的300项日常活动，共计1200个序列，平均持续时间为15分钟。所有参与者累计产生的轨迹长度分别为：头戴设备记录399公里，两只手环记录1053公里。图8展示了参与者在自我报告的种族、年龄、身高和体重方面的人口统计数据。统计数据按性别划分，其中48.5%的参与者自认为女性，51.4%为男性。该数据集涵盖了47栋房屋，其中31栋为多层建筑；总共有201个房间和45个花园。我们还采集了来自开放式校园的三个地点数据，包括1间带有户外露台的食堂、1栋办公楼以及与之相连的公共停车场。 --- ![Image](images/00928c51185bc776a58cff7fd1dd55361748d173969e805df5f70cf64a1cfcc4.jpg) ![Image](images/8eac0b2c98f1314cc4d1a28fb23b04461bb134b5623d2ee784609e5f0232b543.jpg) ![Image](images/435eb4a055e117de1b6c353065570ebfe4c162268392fdd842b27f260471f3ca.jpg) ![Image](images/040b254f1e16dd709c155ab4f2a47549bcf5f47fff898c146624e7cab6d84ffe.jpg)
Fig. 8. Demographics by female and male. Meaning of symbols: Caucasian (Cau), Hispanic (His), African American (AA), East/South/Southeast Asian (EA/SA/SEA).
骑行/徒步路径。大约15%的录音包含户外活动内容。在所有场景中，出现频率最高的是烹饪、寻找物品以及即兴动作。详细的分类数据请参见补充材料。 标注统计。在语言描述方面，我们标注了38.6小时的运动叙述、207小时的原子动作描述以及196小时的活动总结。叙述和原子动作的平均视频片段时长为5秒，总结部分则为30秒。总体而言，该数据集提供了31.05万句句子，共864万字，涉及6545个不同的词汇。需要注意的是，平均每句单词数达到27.8个，这一数值明显长于现有的运动-语言叙述数据。图7可视化展示了语言分布情况。 ## 4 基准任务与基线 ### 4.1 研究机遇 凭借海量的情境化人体运动数据，Nymeria数据为前所未有的研究机遇提供了有力支撑。下文我们将重点介绍几个研究领域。这绝非详尽无遗的列表，旨在抛砖引玉，激发人们探索其全部潜力的新方向与新思路。 运动任务。Nymeria旨在辅助人体运动理解，重点涵盖但不仅限于以自我为中心的感知领域。这些数据支持多种研究主题，例如全身追踪、运动合成、运动预测、路径规划、动作识别、人类行为分析等。借助多模态、多设备的数据集，我们鼓励研究者探索具有新颖设定下的独特算法，例如基于视线条件的运动预测、利用头戴设备和手环进行动作识别、根据语言生成交互行为等。 多模态空间推理与视频理解。除了人体运动数据外，Nymeria还提供了带有精确定位与同步信息的丰富第一视角视频。因此，Nymeria对于相关算法而言是宝贵的资源。 ---
Table 3. AvatarPoser [^45] train/test with real vs. synthetic poses from Nymeria. The first row reports AvatarPoser train/test on AMASS [^68] for reference.
MPJPE (cm)Hand PE (cm)MPJVE (cm/s)
MeanLowerUpper
AMASS4.208.061.882.3428.23
Real7.9716.743.136.2516.71
Synthetic7.3115.972.513.4716.63
Table 4. Motion synthesis with EgoEgo [^60] and BoDiffusion [^20] on Nymeria. BoDiffusion(A) reports results train/test on AMASS [^68].
MPJPE (cm)FID
MeanLowerUpper
BoDiffusion(A)3.637.071.53-
BoDiffusion7.9815.275.282.32
EgoEgo13.2219.0310.005.14
这类算法需要以相机位姿作为先验信息，例如场景重建 [^54] [^71]。在这方面，丰富的动态数据带来了全新的现实世界挑战。Nymeria还有助于视频理解，因为其中包含大量上下文叙述。图像检索与重新定位也能从Nymeria中受益，因为我们针对每个地点采集了多段视频，并将其对齐到全局坐标系中。 仿真。仿真是一种获取海量数据的有效技术，不过合成数据通常难以直接应用于现实世界 [^15] [^46]。从设计初衷来看，Nymeria天然适用于驱动上下文角色动画。由于人体运动受环境制约，我们认为Nymeria也有助于3D场景仿真 [^108]。我们的数据同样能为传感器仿真提供帮助，尤其是结合运动先验的惯性测量单元、磁力计和气压计。 ### 4.2 运动任务基线 我们选取了三个案例研究来展示Nymeria上的运动算法。其目的是验证数据并为后续研究提供通用基准。由于篇幅限制，模型训练的详细信息详见补充材料。 基于稀疏输入的运动跟踪与合成。在该案例研究中，我们针对现代AR/VR场景中的单点/三点人体跟踪问题展开研究：在VR场景中，用户佩戴头戴式设备，也可选择佩戴控制器；在AR场景中，用户佩戴眼镜，也可选择佩戴腕带。任务目标是仅利用头戴式设备（单点）[^60] [^66] 或额外的腕部设备（三点）[^20] [^28] [^44] [^45] [^105] 来还原佩戴者的全身运动。由于下半身观测信息不足，该问题可视为跟踪与合成的混合问题。现有研究通常从运动数据集中模拟传感器输入，但这些模拟数据缺乏真实数据的噪声特征。Nymeria是首个包含多模态真实数据的大型数据集，可用于支持模型训练与评估。具体而言，我们提供了原始的IMU和设备姿态数据，而非以往数据集中发布的经过滤波的[^88]加速度和速度数据[^39] [^100] [^105]。在首个实验中，我们采用了三点回归方法AvatarPoser [^45]，在Nymeria数据集上训练两个变体模型：一个模型使用来自SLAM的“真实”设备姿态进行训练，另一个模型则使用与原始论文中相同的身体运动数据生成的“合成”输入进行训练。 --- 工作。评估采用与 AvatarPoser 相同的指标，包括平均关节位置误差 (MPJPE)、手部位置误差和平均关节速度误差 (MPJVE)。结果如表 3 所示。正如预期，由于额外的误差来源，“真实”数据的性能更差。然而差距很小，这表明了设备跟踪的质量。与在 AMASS 上训练的原版 AvatarPoser [^68] 相比，Nymeria 产生了合理但更高的 MPJPE，主要来自较差的下肢跟踪。我们假设我们的数据包含更困难的运动，例如在不平坦地形上徒步、上下楼梯、进行体育运动等。第二个实验评估了两个用于动作合成的扩散模型，即 3-point Bodiffusion [^20] 和 1-point EgoEgo [^60]。除 MPJPE 外，我们还报告了 Frechet Inception Distance (FID)，用于衡量生成动作与真实动作之间的分布距离，遵循 [^38] 中的相同流程（参见表 4）。从我们的数据训练得到的结果与在 AMASS 子集上训练的原始工作相当，其中 3-point diffusion 如预期那样产生了更好的性能。 人类运动流形的表示。学习嵌入空间以建模人类运动流形具有诸多优势，例如降维、学习运动先验、通过投影实现运动去噪、通过采样和插值实现运动合成等。以往的研究主要依赖AMASS数据集[^68]来学习该表示[^63] [^80] [^86] [^97]，这类数据主要聚焦于孤立的运动或专业运动场景。而我们的数据集涵盖了普通人与现实世界互动时的丰富日常活动，因此能更好地反映与日常人类活动相关的运动流形。为了进一步……
Table 5. Ablation of motion VQ-VAE trained on Nymeria (metric unit: mm). We compare product quantization (PQ), codebook (CB) size and latent dimensions (Dim). The first row show results of AMASS as reported in [^43]. PA stands for Procrustes-aligned and ACC for joint position acceleration.
PQCBDimMPJPEPA-MPJPEACC
$^{{*}}$512-55.8040.107.50
1204851251.6037.551.09
2204851239.6329.770.71
2409651239.2029.660.82
2163846434.4926.830.67
按照这个方向，我们针对Nymeria运动数据训练了向量量化变分自编码器（VQ-VAE）[^27] [^76]，这一工作延续了以往的研究[^43]。VQ-VAE可作为“运动分词器”，用于通过自回归方式生成运动[^65]，也能通过将输入投影到流形上来对运动去噪[^86]。我们进行了消融实验，以研究乘积量化、码本大小和潜变量维度的影响。在评估时，我们采用了[^43]中相同的评价指标，同时纳入在AMASS数据集上训练的VQ-VAE作为对比。如表5所示，Nymeria运动数据能够被很好地分词，其性能与用AMASS数据训练的VQ-VAE相近。通过利用乘积量化、增大码本大小并减小潜变量维度，重建质量得到了进一步提升。因此，所得到的运动分词器可以像语言分词器一样与大型语言模型配合使用，以促进对运动的理解[^43] [^114]，具体细节将在下一个案例研究中阐述。 --- 运动与语言。虽然参数化人体运动对机器算法而言很有用，但自然语言描述才是与人类交互的更好方式。Nymeria数据集的一个宝贵特性就是其高质量的层次化叙述。与现有数据[^36] [^62]相比，我们的叙述篇幅更长。
Table 6. Evaluation of motion-to-text. Models are trained with small subset of Nymeria.
BertBleu@1Bleu@4CIDErRougeL
TM2T11.0840.118.9920.8530.70
MotionGPT14.0942.2210.3137.2732.33
自然语句中包含了物体和环境的上下文描述。这类数据可用于训练用于文本驱动的动作生成以及动作转文本描述的模型。更重要的是，这些上下文描述不仅与人类动作相匹配，还与视频、点云以及其他感官数据和标注信息相关联。将二维和三维环境信息与语言相结合，为在物理世界中开展语言与动作研究提供了令人兴奋的契机。借助VQ-VAE实验，我们训练了MotionGPT [^43]和TM2T [^37]，用于执行动作转文本任务；我们使用了来自Nymeria数据集的30小时动作叙述子集（参见表6），以便与以往的研究结果直接对比，同时采用BERT [^113]、BLEU [^79]、CIDEr [^101]和ROUGE-L [^61]作为评估指标。由于TM2T缺乏基于T5 [^84]架构的强大语言先验，其表现不如MotionGPT。与最初使用HumanML3D [^36]和KIT [^82]训练的模型相比，我们的结果较差，这也在意料之中。尽管数据时长相近，但我们的叙述内容更为复杂且多样化。若使用完整的叙述数据，我们预计性能会有显著提升。 ### 5 结论与讨论 我们提出Nymeria数据集，旨在加速以自我为中心的运动理解研究。该数据集是全球最大的野外人体运动集合，包含300小时的日常活动数据、来自264名参与者的260百万个身体姿态数据，覆盖50个地点。我们提供了精确的六自由度跟踪数据、三维场景点云和视线数据，所有模态均同步对齐到一个统一的度量三维世界中。总体而言，该数据集记录了参与者行走的399公里路程，共产生2.012亿张以自我为中心的图像、117亿个惯性测量单元采样点和1080万个视线点。Nymeria数据集还是全球最大的运动-语言数据集，包含31.05万句叙述，总词数达864万词，其中精细运动叙述时长38.6小时，原子动作描述时长207小时，活动总结时长196小时。 局限性。动作捕捉服和腕带会导致视频中的人物外观不自然，还会限制某些运动范围。众所周知，XSens数据的质量会受到运动校准和身体测量因素的影响。我们的数据集仅涵盖了部分日常活动，未包含常见的公共场景。 社会影响。理解以自我为中心的全身运动对于开发情境感知型AI至关重要，但这涉及大量个人数据。我们尽最大努力通过征得同意、去标识化、最小化数据保留以及采用宽松的研究许可来尊重用户隐私。 --- 贡献声明。Lingni Ma主导了该项目，开发了用于构建数据集的流程，协调了数据收集与处理工作，并牵头开展了基线评估。Yuting Ye开发了人体运动重定向解决方案，为数据收集和评估提供了指导。Fangzhou Hong、Vladimir Guzov和Yifeng Jiang负责验证数据集并实现基线模型。Rowan Postyeni协助日常运营，处理了XSens运动数据，并对XSens运动数据和旁白进行了质量评估。Luis Pesquería担任数据收集和旁白制作的项目经理。Alexander Gamino负责多设备追踪工作。Vijay Baiyya负责Project Aria MPS的规模化处理工作。Hyo Jin Kim协助进行旁白标注。Kevin Bailey和David Soriano Fosas牵头研发miniAria腕带硬件。C. Karen Liu和Ziwei Liu担任数据收集、标注及基线评估方面的技术顾问。Jakob Engel、Renzo De Nardi和Richard Newcombe担任高级技术与科学顾问。 致谢。我们衷心感谢以下同事的宝贵讨论和技术支持：Genesis Mendoza、Jacob Alibadi、Ivan Soeria-Atmadja、Elena Shchetinina和Atishi Bali参与了数据收集工作；Yusuf Mansour协助在Project Aria上开展视线估计研究；Ahmed Elabbasy、Guru Somasundaram、Omkar Pakhi和Nikhil Raina协助开发EgoBlur匿名化视频解决方案，并探索了边界框标注方法；Evgeniy Oleinik、Maien Hamed和Mark Schwesinger协助将Nymeria数据集接入Project Aria数据集浏览器并完成数据发布；Melissa Hebra协助协调旁白标注工作；Edward Miller担任研究项目经理；Pierre Moulon为开源代码库提供了宝贵指导；Tassos Mourikis、Maurizio Monge、David Caruso、Duncan Frost和Harry Lanaras为SLAM技术提供了支持；Daniel DeTone、Dan Barnes、Raul Mur Artal、Thomas Whelan和Austin Kukay就语义边界框标注问题进行了深入讨论；Julian Nubert率先将该数据集用于内部测试；Pedro Cancel Rivera、Gustavo Solaira、Yang Lou和Yuyang Zou从Project Aria项目中给予了支持；Svetoslav Kolev提供了频繁的反馈；Arjang Talatto协助了MPS相关工作；Gerard Pons-Moll担任高级顾问；Carl Ren和Mingfei Yan担任高级经理。 ### 参考文献 1. Apple Vision Pro. [^2]: HTC VIVE. vive.com 3. Magic Leap 2. 4. Meta momentum库. 5. Meta Quest. 6. Microsoft HoloLens. 7. Movella XSens MVN Link动作捕捉系统. 8. Project Aria机器感知服务. 9. Ray-Ban Meta智能眼镜. 10. Rokoko. --- 11. Vuzix智能眼镜。 12. Akada, H.等人：UnrealEgo：一个用于鲁棒性第一人称3D人体动作捕捉的新数据集。载于：Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T.（编）《ECCV 2022》。LNCS，第13666卷，第1–17页。Springer出版社，Cham（2022年）。 [^13]: Araujo, J.P., et al.: CIRCLE: capture in rich contextual environments. In: CVPR (2023) [^14]: Banerjee, P., et al.: Introducing HOT3D: an egocentric dataset for 3D hand and object tracking (2024) [^15]: Black, M.J., Patel, P., Tesch, J., Yang, J.: BEDLAM: a synthetic dataset of bodies exhibiting detailed lifelike animated motion. In: Proceedings IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2023) [^16]: Brown, T., et al.: Language models are few-shot learners. In: Advances in Neural Information Processing Systems, vol. 33, pp. 1877–1901 (2020) 17. Cai, Z.等人：HuMMan：用于多功能感知与建模的多模态4D人体数据集。载于：Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T.（编）《ECCV 2022》。LNCS，第13667卷，第557–577页。Springer出版社，Cham（2022年）。 [^18]: Cai, Z., et al.: SMPLer-X: scaling up expressive human pose and shape estimation (2023) [^19]: Cai, Z., et al.: Playing for 3D human recovery. arXiv preprint arXiv:2110.07588 (2021) [^20]: Castillo, A., et al.: BoDiffusion: diffusing sparse observations for full-body human motion synthesis. In: ICCV (2023) [^21]: Chen, X., et al.: Executing your commands via motion diffusion in latent space. In: CVPR (2023) [^22]: Cong, P., et al.: LaserHuman: language-guided scene-aware human motion generation in free environment (2024) [^23]: Dabral, R., Mughal, M.H., Golyanik, V., Theobalt, C.: MoFusion: a framework for denoising-diffusion-based motion synthesis. In: Computer Vision and Pattern Recognition (CVPR) (2023) [^24]: Damen, D., et al.: Scaling egocentric vision: the epic-kitchens dataset. In: ECCV (2018) [^25]: Damen, D., et al.: The epic-kitchens dataset: collection, challenges and baselines. IEEE Trans. Pattern Anal. Mach. Intell. (TPAMI) 43(11), 4125–4141 (2021) 26. Delmas, G., Weinzaepfel, P., Lucas, T., Moreno-Noguer, F., Rogez, G.：PoseScript：从自然语言生成3D人体姿态。载于：Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T.（编）《ECCV 2022》。LNCS，第13666卷，第346–362页。Springer出版社，Cham（2022年）。 [^27]: Dhariwal, P., Jun, H., Payne, C., Kim, J.W., Radford, A., Sutskever, I.: Jukebox: a generative model for music. arXiv preprint arXiv:2005.00341 (2020) [^28]: Du, Y., Kips, R., Pumarola, A., Starke, S., Thabet, A., Sanakoyeu, A.: Avatars grow legs: generating smooth human motion from sparse tracking inputs with diffusion model. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 481–490 (2023) [^29]: Engel, J., Koltun, V., Cremers, D.: Direct sparse odometry (2016) [^30]: Engel, J., et al.: Project aria: a new tool for egocentric multi-modal AI research (2023) ## [^31]: Feng, Y., Lin, J., Dwivedi, S.K., Sun, Y., Patel, P., Black, M.J.: ChatPose: chatting about 3D human pose. In: CVPR (2024) --- [^32]: Ghorbani, S., et al.: MoVi: a large multi-purpose human motion and video dataset. PLoS ONE 16(6), e0253157 (2021) [^33]: Goel, S., Pavlakos, G., Rajasegaran, J., Kanazawa, A., Malik, J.: Humans in 4D: reconstructing and tracking humans with transformers. In: International Conference on Computer Vision (ICCV) (2023) [^34]: Grauman, K., et al.: Ego4D: around the world in 3,000 hours of egocentric video. In: CVPR, pp. 18995–19012 (2022) [^35]: Grauman, K., et al.: Ego-Exo4D: understanding skilled human activity from first- and third-person perspectives. In: CVPR (2024) [^36]: Guo, C., et al.: Generating diverse and natural 3D human motions from text. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5152–5161 (2022) 37. Guo, C., Zuo, X., Wang, S., Cheng, L.：TM2T：用于3D人体动作与文本相互生成的随机化与分词建模方法。载于：Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T.（编）《ECCV 2022》。LNCS，第13695卷，第580–597页。Springer出版社，Cham（2022年）。 [^38]: Guo, C., et al.: Action2Motion: conditioned generation of 3D human motions. In: Proceedings of the 28th ACM International Conference on Multimedia, pp. 2021–2029 (2020) [^39]: Guzov, V., Mir, A., Sattler, T., Pons-Moll, G.: Human poseitioning system (HPS): 3D human pose estimation and self-localization in large scenes from body-mounted sensors. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2021) [^40]: Harvey, F.G., Yurick, M., Nowrouzezahrai, D., Pal, C.: Robust motion in betweening. ACM Trans. Graph. (TOG) 39(4), 60–1 (2020) [^41]: Huang, Y., Kaufmann, M., Aksan, E., Black, M.J., Hilliges, O., Pons-Moll, G.: Deep inertial poser: learning to reconstruct human pose from sparse inertial measurements in real time. ACM Trans. Graph. (TOG) 37(6), 1–15 (2018) [^42]: Ionescu, C., Papava, D., Olaru, V., Sminchisescu, C.: Human3.6M: large scale datasets and predictive methods for 3d human sensing in natural environments. IEEE Trans. Pattern Anal. Mach. Intell. 36(7), 1325–1339 (2013) [^43]: Jiang, B., Chen, X., Liu, W., Yu, J., Yu, G., Chen, T.: MotionGPT: human motion as a foreign language. In: Advances in Neural Information Processing Systems (2024) [^44]: Jiang, J., Streli, P., Meier, M., Fender, A., Holz, C.: EgoPoser: robust real-time ego-body pose estimation in large scenes. arXiv preprint arXiv:2308.06493 (2023) 45. Jiang, J.等人：AvatarPoser：基于稀疏运动传感的全身关节姿态跟踪技术。载于：Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T.（编）《ECCV 2022》。LNCS，第13665卷，第443–460页。Springer出版社，Cham（2022年）。 [^46]: Jiang, N., et al.: Scaling up dynamic human-scene interaction modeling. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2024) [^47]: Jiang, Y., Ye, Y., Gopinath, D., Won, J., Winkler, A.W., Liu, C.K.: Transformer inertial poser: real-time human motion reconstruction from sparse IMUs with simultaneous terrain generation. In: SIGGRAPH Asia 2022 Conference Papers, pp. 1–9 (2022) ### [^48]: Joo, H., et al.: Panoptic studio: a massively multiview system for social motion capture. In: The IEEE International Conference on Computer Vision (ICCV) (2015) --- [^49]: Joo, H., et al.: Panoptic studio: a massively multiview system for social interaction capture. IEEE Trans. Pattern Anal. Mach. Intell. (2017) [^50]: Kanazawa, A., Zhang, J.Y., Felsen, P., Malik, J.: Learning 3D human dynamics from video. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5614–5623 (2019) [^51]: Karunratanakul, K., Preechakul, K., Suwajanakorn, S., Tang, S.: Guided motion diffusion for controllable human motion synthesis. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2151–2162 (2023) [^52]: Kaufmann, M., et al.: EMDB: the electromagnetic database of global 3D human pose and shape in the wild. In: International Conference on Computer Vision (ICCV) (2023) [^53]: Kaufmann, M., et al.: EM-pose: 3D human pose estimation from sparse electromagnetic trackers. In: The IEEE International Conference on Computer Vision (ICCV) (2021) 54. Kerbl, B., Kopanas, G., Leimkühler, T., Drettakis, G.：用于实时辐射场渲染的3D高斯泼溅技术。《ACM图形学汇刊》第42卷第4期（2023年）。 [^55]: Khirodkar, R., Bansal, A., Ma, L., Newcombe, R., Vo, M., Kitani, K.: EgoHumans: an egocentric 3D multi-human benchmark. In: ICCV (2023) [^56]: Kim, J., Kim, J., Na, J., Joo, H.: ParaHome: parameterizing everyday home activities towards 3D generative modeling of human-object interactions (2024) [^57]: Kwon, T., Tekin, B., Stühmer, J., Bogo, F., Pollefeys, M.: H2o: two hands manipulating objects for first person interaction recognition. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 10138–10148 (2021) [^58]: Lee, J., Joo, H.: Mocap everyone everywhere: lightweight motion capture with smartwatches and a head-mounted camera. arXiv preprint arXiv:2401.00847 (2024) [^59]: Li, G., Zhao, et al.: EgoGen: an egocentric synthetic data generator (2024) [^60]: Li, J., Liu, K., Wu, J.: Ego-body pose estimation via ego-head pose estimation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17142–17151 (2023) [^61]: Lin, C.Y.: ROUGE: a package for automatic evaluation of summaries. In: Text Summarization Branches Out, pp. 74–81 (2004) [^62]: Lin, J., et al.: Motion-X: a large-scale 3D expressive whole-body human motion dataset. In: Advances in Neural Information Processing Systems (2023) [^63]: Ling, H.Y., Zinno, F., Cheng, G., van de Panne, M.: Character controllers using motion VAEs. ACM Trans. Graph. 39(4), 1–40 (2020) [^64]: Loper, M., Mahmood, N., Romero, J., Pons-Moll, G., Black, M.J.: SMPL: a skinned multi-person linear model. ACM Trans. Graphics (Proc. SIGGRAPH Asia) 34(6), 248:1–248:16 (2015) 65. Lucas, T., Baradel, F., Weinzaepfel, P., Rogez, G.：PoseGPT：基于量化的3D人体动作生成与预测方法。载于：Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T.（编）《ECCV 2022》。LNCS，第13666卷，第417–435页。Springer出版社，Cham（2022年）。 [^66]: Luo, Z., Hachiuma, R., Yuan, Y., Kitani, K.: Dynamics-regulated kinematic policy for egocentric pose estimation. In: Neural Information Processing Systems (2021) ### [^67]: Luvizon, D., Habermann, M., Golyanik, V., Kortylewski, A., Theobalt, C.: Scene-aware 3D multi-human motion capture from a single camera. Comput. Graph. Forum 42(2), 371–383 (2023) --- [^68]: Mahmood, N., Ghorbani, N., Troje, N.F., Pons-Moll, G., Black, M.J.: AMASS: archive of motion capture as surface shapes. In: International Conference on Computer Vision, pp. 5442–5451 (2019) [^69]: von Marcard, T., Henschel, R., Black, M.J., Rosenhahn, B., Pons-Moll, G.: Recovering accurate 3D human pose in the wild using IMUs and a moving camera. In: Proceedings of the European Conference on Computer Vision (ECCV) (2018) [^70]: Marcard, T., Rosenhahn, B., Black, M., Pons-Moll, G.: Sparse inertial poser: automatic 3D human pose estimation from sparse IMUs. Comput. Graph. Forum 36(2) (2017). Proceedings of the 38th Annual Conference of the European Association for Computer Graphics (Eurographics) (2017) 71. Mildenhall, B., Srinivasan, P.P., Tancik, M., Barron, J.T., Ramamoorthi, R., Ng, R.：NeRF：将场景表示为用于视图合成的神经辐射场。载于：Vedaldi, A., Bischof, H., Brox, T., Frahm, J.M.（编）《ECCV 2020》。LNCS，第12346卷，第405–421页。Springer出版社，Cham（2020年）。 [^72]: Mollyn, V., Arakawa, R., Goel, M., Harrison, C., Ahuja, K.: IMUPoser: full-body pose estimation using IMUs in phones, watches, and earbuds. In: Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, CHI 2023. Association for Computing Machinery, New York (2023). [^73]: Mourikis, A.I., Roumeliotis, S.I.: A multi-state constraint Kalman filter for vision-aided inertial navigation. In: Proceedings 2007 IEEE International Conference on Robotics and Automation, pp. 3565–3572. IEEE (2007) 74. Movella：MVN用户手册。 [^75]: Mur-Artal, Raúl, M.J.M.M., Tardós, J.D.: ORB-SLAM: a versatile and accurate monocular SLAM system. IEEE Trans. Robot. 31(5), 1147–1163 (2015) [^76]: van den Oord, A., Vinyals, O., Kavukcuoglu, K.: Neural discrete representation learning. In: Proceedings of the 31st International Conference on Neural Information Processing Systems (2017) [^77]: OpenAI.: Achiam, J., et al.: GPT-4 technical report (2023) [^78]: Pan, X., et al.: Aria digital twin: a new benchmark dataset for egocentric 3D machine perception (2023) [^79]: Papineni, K., Roukos, S., Ward, T., Zhu, W.J.: BLEU: a method for automatic evaluation of machine translation. In: Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311–318 (2002) [^80]: Pavlakos, G., et al.: Expressive body capture: 3D hands, face, and body from a single image. In: Proceedings IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10975–10985 (2019) 81. Petrovich, M., Black, M.J., Varol, G.：TEMOS：从文本描述生成多样化人体动作的技术。载于：Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T.（编）《ECCV 2022》。LNCS，第13682卷，第480–497页。Springer出版社，Cham（2022年）。 [^82]: Plappert, M., Mandery, C., Asfour, T.: The KIT motion-language dataset. Big Data 4(4), 236–252 (2016) [^83]: Punnakkal, A.R., Chandrasekaran, A., Athanasiou, N., Quiros-Ramirez, A., Black, M.J.: BABEL: bodies, action and behavior with English labels. In: Proceedings IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 722–731 (2021) ## [^84]: Raffel, C., et al.: Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res. 21(1), 5485–5551 (2020) --- [^85]: Raina, N., et al.: EgoBlur: responsible innovation in aria. ArXiv abs/2308.13093 (2023) [^86]: Rempe, D., Birdal, T., Hertzmann, A., Yang, J., Sridhar, S., Guibas, L.J.: Humor: 3D human motion model for robust pose estimation. In: International Conference on Computer Vision (ICCV) (2021) [^87]: Alp Gueler, R., Natalia Neverova, I.K.: Densepose: dense human pose estimation in the wild. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2018) [^88]: Roetenberg, D., Luinge, H., Slycke, P.: Xsens MVN: full 6DOF human motion tracking using miniature inertial sensors. Xsens Motion Technol. BV Technical report 3 (2009) [^89]: Rong, Y., Shiratori, T., Joo, H.: FrankMocap: a monocular 3D whole-body pose estimation system via regression and integration. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1749–1759 (2021) [^90]: Sener, F., et al.: Assembly 101: a large-scale multi-view video dataset for understanding procedural activities. In: CVPR (2022) [^91]: Shafir, Y., Tevet, G., Kapon, R., Bermano, A.H.: Human motion diffusion as a generative prior. In: ICLR (2023) [^92]: Shahroudy, A., Liu, J., Ng, T.T., Wang, G.: NTU RGB+D: a large scale dataset for 3D human activity analysis. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1010–1019 (2016) [^93]: Shimada, S., Golyanik, V., Xu, W., Theobalt, C.: PhysCap: physically plausible monocular 3d motion capture in real time. ACM Trans. Graph. (ToG) 39(6), 1–16 (2020) [^94]: Sorkine-Hornung, O., Rabinovich, M.: Least-squares rigid motion using SVD. Computing 1(1), 1–5 (2017) 95. Tevet, G., Gordon, B., Hertz, A., Bermano, A.H., Cohen-Or, D.：MotionCLIP：将人体动作生成引入CLIP空间的技术。载于：Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T.（编）《ECCV 2022》，第XXII部分。LNCS，第13682卷，第358–374页。Springer出版社，Cham（2022年）。 [^96]: Tevet, G., Raab, S., Gordon, B., Shafir, Y., Cohen-or, D., Bermano, A.H.: Human motion diffusion model. In: ICLR (2023) 97. Tiwari, G., Antic, D., Lenssen, J.E., Sarafianos, N., Tung, T., Pons-Moll, G.：Pose-NDF：利用神经距离场建模人体姿态流形。载于：Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T.（编）《ECCV 2022》。LNCS，第13665卷，第572–589页。Springer出版社，Cham（2022年）。 [^98]: Tome, D., et al.: SelfPose: 3D egocentric pose estimation from a headset mounted camera. IEEE Trans. Pattern Anal. Mach. Intell. 45(6), 6794–6806 (2020) [^99]: Touvron, H., et al.: LLaMA: open and efficient foundation language models (2023) [^100]: Trumble, M., Gilbert, A., Malleson, C., Hilton, A., Collomosse, J.: Total capture: 3D human pose estimation fusing video and inertial sensors. In: Proceedings of 28th British Machine Vision Conference, pp. 1–13 (2017) [^101]: Vedantam, R., Lawrence Zitnick, C., Parikh, D.: CiDER: consensus-based image description evaluation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4566–4575 (2015) ## [^102]: Wang, J., Liu, L., Xu, W., Sarkar, K., Luvizon, D., Theobalt, C.: Estimating egocentric 3D human pose in the wild with external weak supervision. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 13157–13166 (2022) --- [^103]: Wang, J., Luvizon, D., Xu, W., Liu, L., Sarkar, K., Theobalt, C.: Scene-aware egocentric 3D human pose estimation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13031–13040 (2023) [^104]: Wouwe, T., Lee, S., Falisse, A., Delp, S., Liu, C.: Diffusion inertial poser: human motion reconstruction from arbitrary sparse IMU configurations. In: CVPR (2024) [^105]: Yang, D., Kang, J., Ma, L., Greer, J., Ye, Y., Lee, S.H.: DivaTrack: diverse bodies and motions from acceleration-enhanced three-point trackers. In: EuroGraphics (2024) [^106]: Yang, D., Kim, D., Lee, S.H.: LoBSTr: real-time lower-body pose prediction from sparse upper-body tracking signals. In: Computer Graphics Forum, vol. 40, pp. 265–275. Wiley Online Library (2021) [^107]: Ye, V., Pavlakos, G., Malik, J., Kanazawa, A.: Decoupling human and camera motion from videos in the wild. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2023) [^108]: Yi, H., Huang, C.H.P., Tripathi, S., Hering, L., Thies, J., Black, M.J.: MIME: human-aware 3D scene generation. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12965–12976, June 2023 [^109]: Yi, X., et al.: EgoLocate: real-time motion capture, localization, and mapping with sparse body-mounted sensors. ACM Trans. Graph. (TOG) 42(4), 1–17 (2023) [^110]: Yi, X., et al.: Physical inertial poser (PIP): physics-aware real-time human motion tracking from sparse inertial sensors. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13167–13178 (2022) [^111]: Yi, X., Zhou, Y., Xu, F.: Transpose: real-time 3D human translation and pose estimation with six inertial sensors. ACM Trans. Graph. (TOG) 40(4), 1–13 (2021) 112. Zhang, S.等人：EgoBody：从头戴式设备获取交互人群的人体形状与动作数据。载于：Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T.（编）《ECCV 2022》。LNCS，第13666卷，第180–200页。Springer出版社，Cham（2022年）。 [^113]: Zhang, T., Kishore, V., Wu, F., Weinberger, K.Q., Artzi, Y.: BERTScore: evaluating text generation with bert. arXiv preprint arXiv:1904.09675 (2019) [^114]: Zhang, Y., et al.: MotionGPT: finetuned LLMs are general-purpose motion generators. arXiv preprint arXiv:2306.10900 (2023) [^115]: Zhang, Z., Liu, R., Aberman, K., Hanocka, R.: TEDi: temporally-entangled diffusion for long-term motion synthesis (2023) 116. 郑Y.等人：GIMO：基于视线信息的人体运动预测。载于：Avidan S.、Brostow G.、Cissé M.、Farinella G.M.、Hassner T.（编）《ECCV 2022》。LNCS第13673卷，第676–694页。施普林格出版社， Cham（2022年）。 117. 郑Z.等人：HybridFusion：利用单个深度传感器和稀疏IMU实现实时动作捕捉。载于：Ferrari V.、Hebert M.、Sminchisescu C.、Weiss Y.（编）《ECCV 2018》。LNCS第11213卷，第389–406页。施普林格出版社，Cham（2018年）。