# EgoWholeView: Camera-Only Head-Mounted Capture of Body and Bimanual Hand Motion for Embodied Demonstrations
Yaoqing Hu $^{1}$, Xguo He $^{2}$, Daohan Cui $^{3}$, Yukun Shi $^{4}$, Kai Chen $^{2,4,\dagger}$
$^{1}$Zhongguancun Academy $^{2}$DeepCybo
$^{3}$Beijing University of Chemical Technology
$^{4}$Zhongguancun Institute of Artificial Intelligence
## Abstract
Head-mounted body and bimanual motion capture supports self-avatars, teleoperation, and embodied demonstrations. However, a single egocentric camera is vulnerable to a limited field of view, scale ambiguity, and hand–object occlusion. We present EgoWholeView, a wearable capture system of six synchronized wide-angle RGB cameras with a role-specialized pipeline. The outward views localize the rig in a world frame, while an overlapping downward pair independently predicts SMPL-X body and MANO hand parameters. Both predictors are adapted to egocentric fisheye data. To combine the two, reliability cues measured from the predictions themselves weight the fusion of body and hand joint rotations, while shape is aggregated across views and frames. The pair's baseline fixes metric scale, and triangulation recovers root translation. Relative to monocular prediction, fusion reduces PA-MPJPE by 3.7% for the body and 11.2% for the hands, and temporal refinement lowers angular-acceleration magnitude by 32.2% and 31.0%, respectively. On our six-camera recording, the pipeline produces body and hand states in a shared world frame using benchmark-fitted gates without returning, and temporal refinement lowers hand acceleration magnitude by 41.08%. Using only RGB cameras and no third-person view, EgoWholeView offers a low-cost, portable route to egocentric interaction data.
## 1 Introduction
Body and bimanual hand motion are core inputs to virtual reality and 3D user interfaces (Bowman et al., 2004). Self-avatars and embodied telepresence (Tome et al., 2019), hand-driven interaction in VR (Han et al., 2022), and robot demonstration capture (Khazatsky et al., 2024) all benefit from temporally coherent motion in a stable metric frame together with explicit reliability information (Kolotouros et al., 2021; Bramlage et al., 2023). Existing capture paradigms present a deployment–observability trade-off. External multi-camera systems are accurate but constrain the capture volume (Ionescu et al., 2014; Iskakov et al., 2019). Monocular egocentric RGB enables mobile capture but remains geometrically underconstrained: a single view covers only part of the near body, leaves a metric scale unobservable, and is frequently interrupted by hand–object occlusion (Prakash et al., 2024; Mucha et al., 2026). Moving hands can also degrade SLAM, and out-of-view motion requires explicit infilling (Zhang et al., 2025a). Recovering both hands together with the arms and torso is harder than isolated hand regression, and from a single head-mounted camera the body is often outside the reliable field of view (FOV).
Most commercial egocentric headsets already carry several cameras, but their sensing layouts are not designed for complete body capture, and such platforms remain costly or research-restricted. For example, HOT3D uses Meta Quest 3 for hand-object tracking alone (Banerjee et al., 2025). Recovering body and hand motion in a world frame requires two measurements that a single forward-facing view cannot supply: the metric pose of the rig itself in the world frame, and the metric geometry of the body and hands relative to that rig. The two measurements rely on different observations: the first on static scene features, the second on moving body parts. Camera identity, coordinate frame, timestamp, and the body-or-scene distinction therefore belong with every intermediate prediction. Overlapping views of the same joint are rarely equally reliable, so both contribute rather than one being selected.
In this work, we meet these requirements with a portable head-mounted RGB rig alone, through the role-specialized architecture of Figure 1. The rig carries six synchronized wide-angle cameras: forward and lateral outward views jointly recover its pose in a world frame, and an overlapping downward pair observes the near body and both hands. Released regressors target distant, weakly distorted
---
Figure 1: Overview of EgoWholeView. Six synchronized cameras on a wearable rig provide observations for continuous outward-view multi-camera trajectory estimation and downward-view body and bimanual pose estimation. Paired downward-view predictions undergo reliability-aware fusion in the canonical space, followed by temporal refinement of local rotations. The system exports world-frame body and bimanual motion.
views, which do not match the close-range, strongly distorted images that a head-mounted fisheye rig produces. We therefore adapt both predictors using egocentric recordings and synthetic fisheye images.
To combine the two downward views without assuming that either is reliable, the branches fuse the paired local rotations, with per-joint reliability weights deciding how much each view contributes. Fusion takes place in the canonical space of SMPLX (Pavlakos et al., 2019) and MANO (Romero et al., 2017), which is independent of any camera. Global orientation and monocular root translation are expressed in each camera's own frame and are handled separately: orientation is fused on $SO(3)$ once both views are brought into the frame of one downward camera, whereas metric root translation is recovered from calibrated stereo semantic-joint correspondences rather than from the two monocular root translation predictions. Articulation is evaluated under similarity-aligned protocols, and metric placement is demonstrated on the rig.
Contributions. Our main contributions are summarized as follows:
- Role-Specialized Six-Camera Wearable Rig: a synchronized, calibrated head-mounted rig whose outward views recover its pose while an overlapping downward pair observes the near body and both hands. Body and left/right hand assignments are retained throughout.
- Egocentric Adaptation and Canonical-Space Fusion: the SMPL-X and MANO predictors are adapted to egocentric fisheye views; their paired outputs are fused at the parameter level in the canonical space under per-joint reliability gates, and the fused rotations receive a disagreement-adaptive $SO(3)$ refinement to improve temporal coherence.
• Benchmark Evaluation and Real-Rig Demonstration: the method lowers PA-MPJPE by 3.7% on UnrealEgo-RW (Akada et al., 2022) and 11.2% on DexYCB (Chao et al., 2021) relative to monocular prediction, applies the DexYCB-fitted hand gate to HO3Dv3 (Hampali et al., 2020) without refitting, and reduces angular-acceleration magnitude by 32.2% for the body and 31.0% for the hands. On our own six-camera recording, gates frozen from the benchmarks are used without returning, and the pipeline runs end to end.
## 2 Related Work
### 2.1 Wearable Egocentric Capture for Embodied Applications
Embodied applications constrain both the content and the representation of captured motion. A self-avatar driven by the user's own movement can induce ownership of and agency over a virtual body (Slater et al., 2010; Kilteni et al., 2012). This effect is not fixed by appearance: personalization raises ownership and presence (Waltemate et al., 2018), tracking fidelity changes embodiment (Eubanks et al., 2020), and motion congruence shapes self-recognition (Kim et al., 2024).
Camera-based hand tracking is an established VR input (Han et al., 2020, 2022) but degrades during physical-object interaction (Xu et al., 2026; Cho et al., 2026), and how it fails matters as much
---
as its mean error: jitter and dropouts propagate into avatar motion (Maiorca et al., 2024). Capture inherits this difficulty because a recording system cannot require the wearer to keep both hands visible. Demonstration capture additionally requires a stable world frame (Zhang et al., 2018). Together, these applications call for temporally coherent, metric, parametric body-and-hand motion, with explicit reliability information (Kolotouros et al., 2021; Bramlage et al., 2023).
Existing wearables meet subsets of these needs. Project Aria is the closest platform, exposing calibrated outward fisheye streams with metric device tracking (Engel et al., 2023) and anchoring work on hand–object benchmarking (Banerjee et al., 2025) and dataset-scale full-body motion (Ma et al., 2024). These platforms observe the scene, so the wearer's body reaches the model indirectly. EgoAllo accepts that constraint and recovers allocentric body-and-hand motion from head-mounted sensing (Yi et al., 2025); EgoBody targets social interaction (Zhang et al., 2022). Downward-facing rigs take the opposite position, maximizing direct wearer visibility (Xu et al., 2019; Tome et al., 2019; Zhao et al., 2021; Akada et al., 2022). Either way the world frame comes from camera-only SLAM, now including generalized-camera tracking (Campos et al., 2021; Urban and Hinz, 2016; de Mayo et al., 2025).
Outward world sensing and downward body capture remain separate lines of work. To our knowledge, no wearable dedicates outward cameras to rig trajectory and an overlapping downward pair to directly observed body and hands, which is the system-level gap our six-camera design targets. Egocentric corpora already train embodied models and data engines (Grauman et al., 2022; Wang et al., 2023b; Nair et al., 2022; Brohan et al., 2023; Zhang et al., 2025b), but generally lack synchronized, metric, parametric body-and-hand states for motion supervision and retargeting.
### 2.2 Parametric Body and Hand Reconstruction
MANO and SMPL-X provide structured parametric representations for hands and expressive bodies, respectively: subject-specific shape and local joint rotations defined relative to kinematic trees (Romero et al., 2017; Pavlakos et al., 2019). Shape can be personalized to a wearer, and local rotations transferred to an articulated avatar more directly than loose keypoints. This factorization is important for a moving rig: the same articulation may be predicted from two cameras even though its global orientation and translation differ between camera frames. Transformer regressors produce strong monocular hand hypotheses from a single crop (Pavlakos et al., 2024; Potamias et al., 2025; Lin et al., 2026). Egocentric variants adapt these models to the close range, distortion, truncation, blur, and hand–object occlusion produced by head-mounted cameras (Prakash et al., 2024; Mucha et al., 2026). Egocentric whole-body regressors extend the same adaptation to the body (Na et al., 2026), where the domain gap is wider: downward views strongly foreshorten and crop the torso and rarely observe the legs.
On a moving wearer these regressors face two further difficulties. World-space reconstruction must separate camera and wearer motion because each prediction is expressed in a frame that is itself moving (Zhang et al., 2025a). Independently regressed frames also introduce jitter. Existing temporal methods learn body sequence priors or add video context for hand reconstruction (Kocabas et al., 2020; Lin et al., 2024), but couple temporal consistency to a sequence predictor and therefore change the deployed model. Keeping a released regressor frozen leaves only its per-frame output rotations to refine.
### 2.3 Reliability-Aware Multi-View Fusion
One camera may hold the wrist while losing the fingertips, and the other may fail on different joints. ProHMR and HuManiFlow model distributions over plausible bodies, POCO predicts per-sample uncertainty, and such estimates may still require post-hoc calibration (Kolotouros et al., 2021; Sengupta et al., 2023; Dwivedi et al., 2024; Bramlage et al., 2023). PARE and HandOccNet attend to occluded parts (Kocabas et al., 2021; Park et al., 2022); learnable triangulation scores camera-joint pairs before geometric aggregation (Iskakov et al., 2019). A single score for an entire view is therefore too coarse. The closest baselines fuse elsewhere: S2DHand converts two 21-joint predictions into pseudo-labels that update a single-view estimator (Liu et al., 2024), whereas Rule Meets Learning combines learned confidence with a fixed rule over per-view poses (Ren et al., 2025). Neither assigns view weights directly to MANO local rotations, the quantity that must be combined per joint.
Other multi-view methods fuse features or intermediate geometry. UmeTrack sits closest to deploy
---
ment, processing multiple views and frames jointly to predict one world-space hand pose (Han et al., 2022). Others embed cross-view evidence before regression: in image space, in a shared 3D point representation, or over a graph of joints and quality cues (Tse et al., 2023; Yang et al., 2023; Ren et al., 2022). HaMuCo co-trains a single-view estimator through cross-view interaction (Zheng et al., 2023), and annotation pipelines fit one MANO state to synchronized image and depth evidence (Hampali et al., 2020).
All of them need a trainable multi-view network, cross-view features, or iterative optimization. We instead study parameter-level fusion over released monocular regressors that are adapted with single-view supervision only and then frozen for gate fitting, fusion, and evaluation, so their weights never see a multi-view objective. Our fusion stage neither updates the monocular backbones nor performs cross-view feature fusion. It operates on the predicted parameters, reading intermediate features within a single view only, to derive one reliability cue. What the two views share is their predicted SMPL-X or MANO parameters, so we fuse local rotations in the camera-independent canonical space defined in Section 4.3. Triangulation is used only for metric root placement, not articulation. Recent rigs and datasets with similar sensor layouts still lack an established paired-MANO protocol (Rim et al., 2025; Liu et al., 2022; Ropedia, 2026).
## 3 Six-Camera Wearable Capture System
Each camera is assigned a specific sensing role, and every prediction retains its coordinate frame. We fuse synchronized predictions only when their parameterizations are compatible. Table 1 summarizes the frame, source cameras, and processing of each system output.
### 3.1 Camera Layout and Hardware Implementation
The rigid prototype carries three camera pairs, shown in Figure 2: a forward pair (Cam1/2), a lateral pair (Cam5/6), and an overlapping downward pair (Cam3/4). All six are commodity USB cameras with wide-angle optics spanning 170–220° FOV. All four outward cameras supply the static-scene observations for trajectory estimation, with the lateral pair extending the coverage of the forward one. The downward pair observes the near body and both hands, supplying the paired hypotheses for reliability-aware fusion.
### 3.2 System Overview and Capture Objective
EgoWholeView integrates the role-specialized views into a portable capture pipeline, as shown in Figure 3. The rig trajectory establishes a common world reference, after which egocentric networks produce per-view body and hand hypotheses. Paired SMPL-X and same-side MANO predictions are fused in the canonical space using branch-specific reliability gates.
### 3.3 Synchronization, Rig Calibration and Coordinate Frames
Each hardware trigger t produces one timestamped frame per camera. Frames from the same trigger form a six-view observation, and their timestamps provide a synchronization consistency check. We calibrate the rigid assembly before capture using a cylindrical target adapted to the converging multi-camera arrangement. Its curved marker layout follows self-identifying marker designs (Wang et al., 2024; Hu et al., 2024). This procedure estimates two quantities per view: a fisheye projection model $\pi_c$ (Scaramuzza et al., 2006), which maps a 3D point in camera c coordinates to a pixel, and the extrinsic transform of that camera relative to the rig frame, which stays fixed because the assembly is rigid.
Cam1 defines the rig coordinate frame, and the downward pair uses Cam3 as its own reference for root placement. The fixed extrinsic transform $T_c^{rig}$ maps points from camera c to the rig frame, while $T_{rig,t}^w$ describes the time-varying rig pose in the world frame. A point $\boldsymbol{x}^c$ expressed in camera c coordinates is therefore mapped to the world frame by
$$ \begin{array}{r}{T_{c,t}^{w}=T_{\mathrm{r i g},t}^{w}T_{c}^{\mathrm{r i g}},\;x^{w}=T_{c,t}^{w}x^{c}.}\end{array} $$
This transformation provides a common geometric reference for camera-relative joints and vertices.
### 3.4 Data Flow and System Outputs
With the instance set $\mathbf{I} = \{B, L, R\}$, the exported state at time t is:
$$ \begin{align*}S_{t}&=\{\boldsymbol{T}^{w}_{rig,t},\boldsymbol{X}^{B}_{t},\{\boldsymbol{X}^{h}_{t}\}_{h\in\{L,R\}},\mathbf{P}_{t}\},\\\mathbf{P}_{t}&=\{(\boldsymbol{R}^{i,o}_{root,t},\boldsymbol{t}^{i}_{root,t})\}_{i\in\mathbf{I}}.\end{align*} $$
Here $T_{rig,t}^{w}$ is the rig pose in the world frame, $X_{t}^{B}$ contains temporally refined root-relative SMPL-X local rotations together with sequence-level shape, and $X_{t}^{h}$ contains temporally refined MANO local
---
Table 1: System outputs, representations, coordinate frames, and processing.
| Output | Representation / frame | Cam. | Step |
| $Rig\ pose\ T_{rig,t}^{w}$ | $S\ E(3)$, rig to world | 1/2, 5/6 | multi-view SLAM |
| $Body\ X_{t}^{B}$ | SMPL-X, root relative | 3/4 | gated fusion + temporal |
| $Hand\ X_{t}^{h}$ | MANO, root relative | 3/4 | gated fusion + temporal |
| $Placement\ P_{t}$ | root $S\ E(3)$, Cam3 frame | 3/4 | $SO(3)$ orientation fusion + stereo triangulation |
Figure 2: Six-camera wearable prototype and task-specialized layout. (a) Physical prototype. (b) Side view: the forward pair (Cam1/2) and the overlapping downward pair (Cam3/4). (c) Front view: the lateral pair (Cam5/6) and the downward-pair baseline b. (d) Downward-view evidence coverage; full-body prediction extends beyond the directly observed regions.
rotations together with sequence-level shape for hand side h. The placement term $\mathbf{P}_{t}$ contains the Cam3-frame root orientation and metric root translation of the body, left hand, and right hand, where o marks fused quantities. If a hand is detected in only one downward view, its single-view MANO prediction is retained, but dual-view fusion and metric root placement are unavailable for that hand. Metric root placement is also unavailable when no valid triangulated joint remains. Per-view prediction, parameter fusion, temporal refinement, and root placement are described in Sections 4.2 to 4.5.
## 4 Method
### 4.1 Rig Trajectory and World-Frame Recovery
Using the synchronized images and calibrated camera transforms, we estimate the 6-DoF rig trajectory $T_{rig,t}^{w}$ from static-scene tracks after excluding the moving wearer. For a static landmark $P_{i}^{w}$ observed at pixel $u_{tci}$ in camera c, we define
$$ \begin{align*}e_{tci}&=\left\|\pi_{c}\Big(\boldsymbol{T}^{c}_{rig}(\boldsymbol{T}^{w}_{rig,t})^{-1}\boldsymbol{P}^{w}_{i}\Big)-\boldsymbol{u}_{tci}\right\|_{2},\\\mathcal{L}_{traj}&=\sum_{(t,c,i)\in O_{static}}\rho\big(e^{2}_{tci}\big).\end{align*} $$
where $T_{\text{rig}}^{\text{c}} = (T_{\text{c}}^{\text{rig}})^{-1}$ is the rig-to-camera transform, $\pi_c$ is the camera projection model, $O_{\text{static}}$ is the set of valid static observations, and $\rho$ is a robust penalty. Local bundle adjustment optimizes rig poses and landmarks jointly (Urban and Hinz, 2016), with loop closure supplied by a standard pose-graph backend (Campos et al., 2021). Static tracks are aggregated across all four outward views; dynamic-region rejection removes the wearer where the arms and hands enter those images (Zhang et al., 2025a), since mapping the moving body would corrupt the trajectory. The first frame defines the world frame, so $T_{\text{rig},t}^{\text{w}}$ gives the rig pose relative to the recording origin.
### 4.2 Egocentric Body and Hand Prediction Models
For each synchronized downward view $c \in \{3,4\}$, a body box $\boldsymbol{b}_{t}^{B,c}$ and a box $\boldsymbol{b}_{t}^{h,c}$ for each visible hand $h \in \{L,R\}$ define the two crops. Let $\mathcal{C}$ denote crop-and-resize preprocessing, with $\widetilde{I}_{t}^{B,c} = \mathcal{C}(I_{t}^{c}, \boldsymbol{b}_{t}^{B,c})$ and $\widetilde{I}_{t}^{h,c} = \mathcal{C}(I_{t}^{c}, \boldsymbol{b}_{t}^{h,c})$. The two branches process these crops independently before cross-view fusion, producing
$$ \widehat{\Theta}_{t}^{B,c}=f_{B}(\widetilde{I}_{t}^{B,c}),\qquad\widehat{\Theta}_{t}^{h,c}=f_{H}(\widetilde{I}_{t}^{h,c}). $$
We instantiate the two branches with existing parametric regressors. Specifically, we use EgoSMPLX (Na et al., 2026) for SMPL-X body prediction and WiLoR (Potamias et al., 2025), a ViT-based monocular hand regressor, for MANO prediction. Each branch predicts a parameter bundle $\widehat{\Theta}_t^{m,c} = \{\widehat{\theta}_{local,t}^{m,c}, \widehat{\beta}_t^{m,c}, \widehat{\omega}_{root,t}^{m,c}, \widehat{\omega}_{cam,t}^{m,c}\}$, where $m \in \{B, H\}$ denotes the body or hand branch. Here, $\text{Exp}(\omega)$ maps an axis-angle vector $\omega$ to a rotation matrix, while $\text{Log}^{\vee}(R)$ maps a rotation matrix back to its axis-angle vector. The SMPL-X
---
Figure 3: Processing overview. The outward branch recovers the rig trajectory; the downward branch predicts body and hand parameters per view. Paired predictions are fused in the canonical space: local rotations are weighted per joint, shape is aggregated across views and frames. The fused rotations are temporally refined and the roots placed metrically by triangulation.
Figure 4: Per-view prediction and egocentric-domain adaptation. Body and hand crops from each downward view are processed by separate SMPL-X and MANO regressors, with branch-specific weights shared between Cam3 and Cam4. The loss boxes apply during adaptation only. At inference, paired Cam3/Cam4 hypotheses enter the independently parameterized body or hand fusion module in Section 4.3.
or MANO layer first decodes root-relative joints and vertices, $(\widehat{J}_{t}^{m,c}, \widehat{V}_{t}^{m,c}) = M_{m}(\widehat{\theta}_{local,t}^{m,c}, \widehat{\beta}_{t}^{m,c})$. For every decoded point n, camera-frame placement is $\widehat{J}_{t,n}^{m,c} = \exp(\widehat{\omega}_{root,t}^{m,c})\widehat{J}_{t,n}^{m,c} + \widehat{t}_{cam,t}^{m,c}$, and analogously for vertices. The body branch predicts 21 local SMPL-X rotations and the hand branch 15 local MANO rotations, each alongside one root rotation. Egocentric model adaptation. Both predictors retain their original architectures and are fine-tuned end to end on egocentric fisheye data. Both share the same three supervision families, namely validity-masked parameter losses on rotations and shape, an $\ell_{1}$ loss on decoded root-relative joints, and a reprojection loss on those joints. Neither supervises mesh vertices,
$$ \mathcal{L}_{m}=\lambda_{\mathrm{p a r a m}}^{m}\mathcal{L}_{\mathrm{p a r a m}}^{m}+\lambda_{\mathrm{3D}}^{m}\mathcal{L}_{\mathrm{3D}}^{m}+\lambda_{\mathrm{2D}}^{m}\mathcal{L}_{\mathrm{2D}}^{m}+\mathcal{L}_{\mathrm{p r i o r}}^{m}. $$
They differ in two places. The body branch reprojects through the fisheye model of the sample's own camera and its crop transform, and adds an image-coordinate loss on the PositionNet head, whereas the hand branch follows WiLoR and reprojects in crop-normalized coordinates under a fixed virtual focal length, so fisheye distortion never enters its projection. The priors also differ: the hand branch uses WiLoR's adversarial pose–shape term, while the body branch adds a shape-norm prior, a DPoser-X diffusion prior whose weight decays from 0.1 to 0.01 over adaptation, and an explicit Smooth- $\ell_1$ loss on camera-frame root translation. Neither branch retunes the relative weights it inherits, and missing annotations are switched off by validity masks or zero weights. Figure 4 shows the inference branches and adaptation-only supervision.
### 4.3 Reliability-Aware Parametric Fusion
Figure 5 summarizes this stage. The two downward views provide paired hypotheses for one body instance and two hand instances. For $i \in \mathbf{I}$, the mapping $m(i) \in \{B, H\}$ selects its SMPL-X or MANO branch, with $K_H = 15$ local MANO rotations and $K_B = 21$ local SMPL-X body rotations in matrix form. A per-view hypothesis is
$$ \begin{array}{r}{\widehat{\Theta}_{t}^{i,c}=\{\widehat{\boldsymbol{R}}_{t,1:K_{m(i)}}^{i,c},\widehat{\boldsymbol{\beta}}_{t}^{i,c},\widehat{\boldsymbol{\omega}}_{\mathrm{r o o t},t}^{i,c},\widehat{\boldsymbol{t}}_{\mathrm{c a m},t}^{i,c}\},\quad c\in\{3,4\}.}\end{array} $$
The first two terms describe local joint rotations and shape parameters. Both are defined relative to the kinematic tree rather than to any camera, and we refer to the space they span as the canonical space. The last two terms place the prediction relative to its source camera and are substantially less reliable under close-range truncation. This subsection therefore fuses local rotations and shape only. Root orientation and monocular root translation are handled in Section 4.5.
---
Figure 5: Reliability-aware parametric fusion. (a) Gate structure, shared by the body and hand branches but fitted with independent parameters; every normalized cue and aggregate score follows the higher-is-more-reliable convention. (b)–(c) Per-joint view preference and the resulting weights on a MANO hand.
Independent body and hand reliability gates.
The body and hand branches use the same reliability-gate formulation but fit their parameters independently. Below, $c \in \{3,4\}$ indexes the downward camera, $j \in \{1,\ldots,K_m\}$ a local joint, and $\mathbf{K} = \{V, U, B, T, F\}$ the five cue types. For each indexed prediction, the frozen branch supplies
$$ \begin{array}{r}{\boldsymbol{x}_{t,j}^{i,c}=\left[V_{t,j}^{i,c},U_{t,j}^{i,c},B_{t,j}^{i,c},T_{t,j}^{i,c},F_{t,j}^{i,c}\right]^{\top},}\end{array} $$
where V is joint and kinematic-chain visibility, U test-time perturbation dispersion, B crop-boundary support, T temporal inconsistency, and F local feature dispersion. The symbol $x_{t,j,k}^{i,c}$ denotes the component associated with cue $k\in\mathbf{K}$. The V and B cues are computed from projected vertices affected by joint j, whereas F is sampled at its projected descendant joints. The per-view root pose is used only for projection. The resulting scores set the view weights for paired local-rotation fusion.
The five raw cues differ in scale and reliability direction. We therefore map each cue to a normalized reliability value:
$$ r_{t,j,k}^{i,c}=\sigma\Big(d_{k}^{m}(x_{t,j,k}^{i,c}-\mu_{k}^{m})/\eta_{k}^{m}\Big), $$
where $\sigma(z) = (1 + e^{-z})^{-1}$ is the logistic sigmoid; $d_k^m \in \{-1, +1\}$ orients each cue so that larger values always indicate higher reliability; and $\mu_k^m$ and $\eta_k^m > 0$ are its robust center and scale in branch m. Hence, larger $r_{t,j,k}^{i,c} \in (0,1)$ indicates greater joint-view reliability.
Let $\alpha^{m} = (\alpha_{V}^{m}, \alpha_{U}^{m}, \alpha_{B}^{m}, \alpha_{T}^{m}, \alpha_{F}^{m})$ denote the fitted weights of the five cues for branch m. The joint-level reliability score and the two view weights are
$$ \begin{aligned}&s_{t,j}^{i,c}=\sum_{k\in\mathbf{K}}\alpha_{k}^{m}r_{t,j,k}^{i,c},\quad\\ &\alpha_{k}^{m}\geq0,\quad\sum_{k\in\mathbf{K}}\alpha_{k}^{m}=1,\quad\\ &w_{t,j}^{i,3}=\sigma\Big((s_{t,j}^{i,3}-s_{t,j}^{i,4})/\tau_{\mathrm{gate}}^{m}\Big),\quad\\ &w_{t,j}^{i,4}=1-w_{t,j}^{i,3}.\\ \end{aligned} $$
Here $s_{t,j}^{i,c} \in (0,1)$ aggregates the five reliability values, while $\tau_{\mathrm{gate}}^{m} > 0$ controls the sharpness of the view-weight assignment. With $\boldsymbol{\mu}^{m} = (\boldsymbol{\mu}_{k}^{m})_{k \in \mathbf{K}}$ and $\boldsymbol{\eta}^{m} = (\boldsymbol{\eta}_{k}^{m})_{k \in \mathbf{K}}$, the branch gate is $g_{m} = \{\boldsymbol{\alpha}^{m}, \tau_{\mathrm{gate}}^{m}, \boldsymbol{\mu}^{m}, \boldsymbol{\eta}^{m}\}$. The body gate $g_{B}$ and hand gate $g_{H}$ are fitted independently.
Local-rotation and shape fusion. For $R_A, R_B \in SO(3)$, define
$$ \begin{aligned}\operatorname{GeoMean}_{SO(3)}(&\left(\boldsymbol{R}_{A},\boldsymbol{R}_{B};\boldsymbol{w}\right)\\=\boldsymbol{R}_{A}&\operatorname{Exp}\Big(w\operatorname{Log}^{\vee}\left(\boldsymbol{R}_{A}^{\top}\boldsymbol{R}_{B}\right)\Big).\end{aligned} $$
The fused local-pose and shape estimate is
$$ \begin{aligned}\boldsymbol{R}_{t,j}^{i,o}&=GeoMean_{SO(3)}(\widehat{\boldsymbol{R}}_{t,j}^{i,3},\widehat{\boldsymbol{R}}_{t,j}^{i,4};w_{t,j}^{i,4}),\\\boldsymbol{\beta}_{t}^{i}&=median_{u\in N_{t}}\frac{1}{2}(\widehat{\boldsymbol{\beta}}_{u}^{i,3}+\widehat{\boldsymbol{\beta}}_{u}^{i,4}),\ $ \bar{\boldsymbol{J}}_{t}^{i,o},\bar{\boldsymbol{V}}_{t}^{i,o})&=M_{m(i)}(\boldsymbol{R}_{t,1:K_{m(i)}}^{i,o},\boldsymbol{\beta}_{t}^{i}),\end{aligned} $$
where the median is coordinate-wise and $N_t$ contains the frames of the same synchronized sequence. The superscript o marks fused rotations and the quantities decoded from them, as opposed to the single-view predictions indexed by c; Section 4.4 treats them as its observations. For each instance $i \in \mathbf{I}$, let $\theta_{t,j}^{i,o} = \text{Log}^{\vee}(\mathbf{R}_{t,j}^{i,o})$. The fused body and hand states are
$$ \begin{align*}X^{B,o}_{t}&=\{\boldsymbol{\theta}^{B,o}_{t,1:K_{B}},\boldsymbol{\beta}^{B}_{t},\bar{\boldsymbol{J}}^{B,o}_{t},\bar{\boldsymbol{V}}^{B,o}_{t}\},\\X^{h,o}_{t}&=\{\boldsymbol{\theta}^{h,o}_{t,1:K_{H}},\boldsymbol{\beta}^{h}_{t},\bar{\boldsymbol{J}}^{h,o}_{t},\bar{\boldsymbol{V}}^{h,o}_{t}\}.\end{align*} $$
The two hand sides are processed independently.
---
### 4.4 Adaptive S O(3) Temporal Refinement
Frame-wise dual-view fusion can retain high-frequency rotation jitter. We temporally refine the $K_{m(i)}$ local rotations of each instance $i \in \mathbf{I} = \{B, L, R\}$. Sequence-level shape remains fixed, while root orientation and translation are not temporally filtered. For two joint-rotation matrices $\mathbf{R}_1, \mathbf{R}_2 \in SO(3)$, their angular distance is $d(\mathbf{R}_1, \mathbf{R}_2) = \cos^{-1}(((\mathbf{r}_1)^\top \mathbf{R}_2) - 1)/2)$. For joint j, cross-view disagreement, frame-to-frame motion, and fused reliability are $\Delta_{t,j}^i = d(\mathbf{R}_{t,j}^{i,3}, \widehat{\mathbf{R}}_{t,j}^{i,4})$, $\nu_{t,j}^i = d(\mathbf{R}_{t-1,j}^{i,0}, \mathbf{R}_{t,j}^{i,0})$, and $\vec{s}_{t,j}^i = \sum_{c \in \{3,4\}} w_{t,j}^{i,c} s_{t,j}^{i,c}$ respectively. These quantities set the two positive terms that a Kalman-style gain takes as process and observation variance:
$$ \begin{align*}\sigma^{2,i}_{\mathrm{proc},t,j}&=\sigma^{2}_{\mathrm{proc},0}+\lambda_{\nu}\left(\nu^{i}_{t,j}/\sigma_{\nu}\right)^{2},\\\sigma^{2,i}_{\mathrm{obs},t,j}&=\sigma^{2}_{\mathrm{obs},0}+\lambda_{\mathrm{rel}}(1-\bar{s}^{i}_{t,j})^{2}+\lambda_{\Delta}\left(\Delta^{i}_{t,j}/\sigma_{\Delta}\right)^{2}.\end{align*} $$
The subscript-0 terms are positive baselines, $\sigma_{\nu}, \sigma_{\Delta} > 0$ are fixed angular scales shared by all adaptive candidates and never fitted, and the $\lambda$ terms are nonnegative contribution weights. These quantities proxy variance; they are not recursively propagated covariance estimates. Inspired by Kalman-style gains (Kalman, 1960) and speed-adaptive smoothing (Casiez et al., 2012), we compute a joint-wise current-observation weight and apply it through geodesic interpolation on $S O(3)$ (Park and Ravani, 1997):
$$ \begin{align*}\kappa^{i}_{t,j}&=\sigma^{2,i}_{proc,t,j}/(\sigma^{2,i}_{proc,t,j}+\sigma^{2,i}_{obs,t,j}),\\\boldsymbol{R}^{i,\mathrm{temp}}_{t,j}&=\boldsymbol{R}^{i,\mathrm{temp}}_{t-1,j}\exp\left(\kappa^{i}_{t,j}\mathrm{Log}^{\vee}\left((\boldsymbol{R}^{i,\mathrm{temp}}_{t-1,j})^{\top}\boldsymbol{R}^{i,o}_{t,j}\right)\right).\end{align*} $$
Fast motion raises $\sigma_{proc}^{2}$ and thus $\kappa$, favoring the current observation and reducing lag. Low fused reliability or large cross-view disagreement raises $\sigma_{obs}^{2}$ and lowers $\kappa$, favoring the previous refined state. In implementation, we bound the gain between $\kappa_{min} = 0.2$ and $\kappa_{max} = 0.98$ to avoid complete freezing or direct observation pass-through. The refined rotations replace the unfiltered local rotations before SMPL-X or MANO decoding; placement then uses the decoded articulation without temporally filtering root orientation or translation. We reset the state at sequence boundaries or non-consecutive frames and initialize the first valid frame as $\boldsymbol{R}_{t,j}^{i,temp} = \boldsymbol{R}_{t,j}^{i,o}$. The recurrence is causal given its fused observations, but the full evaluated pipeline is offline because cue T uses frame $t+1$ and shape is aggregated over the sequence.
### 4.5 Global Orientation and Metric Root Placement
The two downward predictions express root orientation and monocular root translation in their respective camera frames. Because monocular root translation is unreliable under close-range truncation, we fuse orientation and recover metric root translation geometrically. Taking Cam3 as the reference, let $(R_4^3, t_4^3)$ denote the calibrated Cam4-to-Cam3 transform. The Cam3 orientation needs no transformation, whereas the Cam4 orientation is mapped by $R_4^3$. The two are then averaged with equal weight,
$$ \begin{array}{r l}&{\widetilde{R}_{\mathrm{r o o t},t}^{i,3}=\mathrm{E x p}(\widehat{\omega}_{\mathrm{r o o t},t}^{i,3}),}\\ &{\widetilde{R}_{\mathrm{r o o t},t}^{i,4}=\boldsymbol{R}_{4}^{3}\mathrm{E x p}(\widehat{\omega}_{\mathrm{r o o t},t}^{i,4}),}\\ &{\boldsymbol{R}_{\mathrm{r o o t},t}^{i,o}=\mathrm{G e o M e a n}_{S O(3)}\left(\widetilde{R}_{\mathrm{r o o t},t}^{i,3},\widetilde{R}_{\mathrm{r o o t},t}^{i,4};\frac{1}{2}\right).}\end{array} $$
Translation is recovered from the two downward views rather than from monocular depth. Each semantic joint provides a pair of matching pixels in Cam3 and Cam4. Body pixels come from the PositionNet head, whereas hand pixels are decoded MANO joints mapped from crop coordinates back to the source images. We use all 21 MANO joints for each hand and the neck, shoulders, elbows, wrists, and hips for the body, omitting the weakly observed lower limbs. Using the calibrated fisheye camera models (Scaramuzza et al., 2006) and the known position and orientation between Cam3 and Cam4, we triangulate each pixel pair to obtain the 3D position $X_{t,n}^{i}$ of joint n in Cam3 coordinates. The known distance between the cameras fixes the absolute scale.
The decoded model provides the corresponding root-relative joint $\vec{j}_{t,n}$. We discard triangulated joints with non-positive depth in either camera and let $N_{tri}$ be the number of remaining joints. Once root orientation is fixed, joint n estimates root translation as $X_n - R_{root}\vec{j}_n$. Omitting frame and instance indices for clarity, root translation is their arithmetic mean,
$$ \boldsymbol{t}_{\mathrm{r o o t}}=\frac{1}{N_{\mathrm{t r i}}}\sum_{n=1}^{N_{\mathrm{t r i}}}\left(\boldsymbol{X}_{n}-\boldsymbol{R}_{\mathrm{r o o t}}\ddot{\boldsymbol{j}}_{n}\right). $$
If $N_{tri} = 0$, no root translation is produced. This stage otherwise updates root translation while leaving articulation, orientation, and shape fixed. Because the body regressor's own hand estimate is unreliable at this range, the fused MANO pose replaces the SMPL-X hand pose, with each hand orientation converted to the wrist-local frame. Each hand is then placed at its triangulated root, not at the wrist implied by the arm chain. The resulting poses are exported as $\mathbf{P}_t = \{(\mathbf{R}_{\mathrm{root},t}^{i,o}, \mathbf{t}_{\mathrm{root},t}^{i})\}_{i \in \mathbf{I}}$
---
and transformed to the world frame by $T_{i,t}^{w} = T_{\mathrm{rig},t}^{w} T_{3}^{\mathrm{rig}} T_{i,t}^{3}$.
## 5 Experiments and Discussion
We evaluate body and hand fusion, adaptive $SO(3)$ temporal refinement, and the wearable's continuous trajectory and motion-capture outputs. The body and hand predictors are adapted independently, and their branches use separate gate parameters.
### 5.1 Experimental Setup
The body and hand gates of Section 4.3, fitted independently, are called VUBTF after their five cues V, U, B, T, and F. Equal-weight fusion fixes $w_{t,j}^{i,c} = 1/2$ in Equation (10); VUBTF soft fusion uses the fitted gate weights. Both predictors, both gates, all cue-normalization statistics, and all temporal settings are frozen before testing.
Body branch. The body predictor is fine-tuned from the released EgoSMPLX checkpoint, which uses the SMPLer-X-Huge backbone, on EgoPW (Wang et al., 2022), SceneEgo (Wang et al., 2023a), EgoWholeBody (Na et al., 2026), UnrealEgo-RW (Akada et al., 2022), and synthetic EgoFishBody data. The combined adaptation set contains 551,924 valid samples per epoch, and fine-tuning runs for 12 epochs. All subsequent body experiments use the final checkpoint.
We fit the body gate's five global cue weights and one temperature for eight epochs on 2,048 UnrealEgo-RW stereo pairs sampled from 12 subjects using Adam (learning rate 0.05, batch size 128). After each epoch, the gate is evaluated on 512 pairs from S16, which are excluded from gate training, and the epoch with the lowest validation PA-MPJPE is retained, here the last of the eight. The selected gate is then frozen for UnrealEgo-RW test evaluation.
Hand branch. The hand predictor is fine-tuned from the released WiLoR checkpoint on 391,003 cleaned HOI4D (Liu et al., 2022) samples and 155,000 synthetic EgoFishHands samples. Finetuning is performed using AdamW for 60 epochs (learning rate $10^{-5}$, weight decay $10^{-4}$, batch size 480), with the released loss weights $\lambda_{3D} = 0.05$, $\lambda_{2D} = 0.01$, $\lambda_{orient} = \lambda_{pose} = 0.001$, and $\lambda_{shape} = \lambda_{adv} = 0.0005$.
The hand gate fits the same six parameters: five global cue weights and one temperature. It is optimized for eight epochs on DexYCB S0 validation using Adam (learning rate 0.05, batch size 2,048).
Table 2: Learned normalized cue weights, raw-cue reliability directions, and gate temperatures for the body and hand gates.
| Cue | Reliable when | Body | Hand |
| Visibility V | Higher | 0.3228 | 0.3716 |
| Perturbation dispersion U | Lower | 0.0912 | 0.3243 |
| Crop-boundary support B | Higher | 0.1109 | 0.0029 |
| Temporal inconsistency T | Lower | 0.4551 | 0.2950 |
| Feature dispersion F | Lower | 0.0201 | 0.0062 |
| Gate temperature $\tau_{gate}$ | — | 0.1245 | 0.2285 |
This fitting split contains 2,841 synchronized times-tamps with three to eight valid views, yielding 21,497 unique images and 71,510 unordered pair appearances. The hand branch uses no held-out split for epoch selection, so the final-epoch gate and cue-normalization statistics are frozen for DexYCB testing and HO3Dv3 transfer.
Inputs. On the benchmarks the body branch takes the full image and the hand branch uses the dataset-provided boxes, so no detector is involved in training or benchmark evaluation.
Evaluation metric and gate parameters. For predicted and reference point matrices $\widehat{X}, X^* \in \mathbb{R}^{N \times 3}$, with one 3D point per row, define $r_n(s, \boldsymbol{Q}, \boldsymbol{b}) = \| s \widehat{X}_n \boldsymbol{Q}^\top + \boldsymbol{b} - X_n^* \|_2$. The standard similarity Procrustes alignment and the reported mean error are
$$ \begin{align*}(s^{*},\boldsymbol{Q}^{*},\boldsymbol{b}^{*})&=\underset{b\in\mathbb{R}^{3}}{\arg\min}\sum_{n=1}^{N}r_{n}(s,\boldsymbol{Q},\boldsymbol{b})^{2},\\E_{\mathrm{PA}}(\widehat{\boldsymbol{X}},\boldsymbol{X}^{*})&=\frac{1}{N}\sum_{n=1}^{N}r_{n}(s^{*},\boldsymbol{Q}^{*},\boldsymbol{b}^{*}).\end{align*} $$
PA-MPJPE applies $E_{\mathrm{PA}}$ to the joint matrices with $N=K$, and PA-MPVPE to the vertex matrices with $N=M$. For MANO, $(K,M)=(21,778)$, and for SMPL-X on UnrealEgo-RW, $K=15$. All distances are in millimeters.
Table 2 lists the fitted coefficients. Although the two gates share one structure, they learn different cue combinations: the body gate emphasizes temporal consistency and visibility, whereas the hand gate emphasizes visibility, perturbation stability, and temporal consistency. The body's larger crop-boundary weight is consistent with torso and upper-arm truncation at the image border, whereas the annotated hand boxes are generally centered on the hand.
### 5.2 Body Local-Pose Fusion on UnrealEgo-RW
Data and protocol. The body evaluation uses the released UnrealEgo-RW test split: 93 synchronized
---
Figure 6: Qualitative body local-pose fusion on UnrealEgo-RW. From left to right: stereo inputs with projected monocular SMPL-X predictions, left- and right-view predictions, equal-weight fusion, and VUBTF soft fusion. Predictions are independently Procrustes-aligned to GT15 (gray dashed), and values report PA-MPJPE in mm ( $\downarrow$). Yellow joints and white wireframes show the projected predictions; dotted ankle–foot links indicate the heel–toe correspondences used for GT15 evaluation. The examples have monocular errors close to the test-set mean.
stereo sequences from S1, S9, and S11, totaling 24,012 frame pairs (48,024 images), all held out from gate fitting, temporal-parameter fitting, and epoch selection.
Body joint accuracy uses GT15: the neck and bilateral shoulders, elbows, wrists, hips, knees, ankles, and ToeBase joints. Because SMPL-X lacks ToeBase joints, each annotation is approximately matched to the corresponding heel. Gate fitting excludes these two approximate matches and uses the remaining 13 joints. All reported test PA-MPJPE values use GT15. UnrealEgo-RW provides only sparse joint annotations, so we report joint error alone. The monocular baseline averages independently aligned per-view errors. Leg accuracy drops outside the downward views.
Quantitative results. In Table 3, equal-weight fusion improves PA-MPJPE over the monocular average by 2.5754 mm and VUBTF soft fusion by 2.9496 mm (3.69%), a further 0.3742 mm over equal-weight fusion. Figure 6 shows qualitative examples.
### 5.3 Reliability-Aware Dual-View MANO Fusion
Main DexYCB results. DexYCB S0 test contains 160 capture sequences holding 9,846 complete synchronized eight-view timestamps, or 78,768 images. All 28 camera combinations yield 275,688 pair appearances. Roles A and B are determined by camera serial number and do not denote fixed physical left and right positions.
The official DexYCB MANO annotations are decoded into the reference meshes used for PA-MPVPE. As Table 4 shows, equal-weight fusion reduces PA-MPJPE and PA-MPVPE relative to the monocular average by 0.4643 and 0.4450 mm, respectively. With the learned reliability gate, the two errors decrease further to 4.2872 and 4.1741 mm, corresponding to reductions of 0.0770 and 0.0681 mm over equal-weight fusion. Across all 28 camera pairs, mean PA-MPJPE decreases by 0.0196–0.1645 mm. VUBTF soft fusion outperforms equal-weight fusion on 56.0% of pair appearances and 139 of the 160 test sequences, with a sequence-block-bootstrap 95% interval of [0.0648, 0.0895] mm. Figure 7 visualizes the predictions and per-joint weights.
---
Table 3: SMPL-X fusion on the UnrealEgo-RW test split. PA-MPJPE uses GT15 and is given in mm.
| Method | Mean | Median | P90 | P95 | Gain vs. mono. avg. |
| Left-view prediction | 79.6188 | 74.1331 | 102.4118 | 115.0452 | — |
| Right-view prediction | 80.1623 | 74.5614 | 101.6197 | 116.6663 | — |
| Monocular average | 79.8905 | 75.1588 | 101.2589 | 112.1287 | 0.0000 |
| Equal-weight fusion | 77.3151 | 72.9845 | 96.8666 | 106.8710 | +2.5754 |
| VUBTF soft fusion | 76.9409 | 72.6824 | 96.0671 | 106.3893 | +2.9496 |
Table 4: Dual-view MANO fusion on DexYCB S0 test and frozen transfer to HO3Dv3 (mm). PA-MPJPE uses 21 joints and PA-MPVPE all 778 MANO vertices.
| Method | Mean | Median | P90 | P95 | Gain vs. mono. avg. |
| PA-MPJPE (21 joints) | | | | | |
| Mono A | 4.9092 | 4.3793 | 7.7093 | 9.1141 | — |
| Mono B | 4.7478 | 4.2562 | 7.4034 | 8.8242 | — |
| Monocular average | 4.8285 | 4.4214 | 7.2355 | 8.4284 | 0.0000 |
| Equal-weight fusion | 4.3642 | 3.9486 | 6.7328 | 7.9130 | +0.4643 |
| VUBTF soft fusion | 4.2872 | 3.8732 | 6.6027 | 7.7682 | +0.5413 |
| PA-MPVPE (778 vertices) | | | | | |
| Mono A | 4.7557 | 4.2541 | 7.3973 | 8.7742 | — |
| Mono B | 4.6186 | 4.1472 | 7.1306 | 8.4739 | — |
| Monocular average | 4.6872 | 4.3023 | 6.9567 | 8.1240 | 0.0000 |
| Equal-weight fusion | 4.2422 | 3.8499 | 6.4741 | 7.5945 | +0.4450 |
| VUBTF soft fusion | 4.1741 | 3.7837 | 6.3589 | 7.4551 | +0.5131 |
| $HO3Dv3$ PA-MPJPE (21 joints, frozen transfer) | | | | | |
| Mono A | 7.0625 | 6.8700 | 9.8260 | 10.8385 | — |
| Mono B | 7.1124 | 6.7238 | 10.0545 | 11.1588 | — |
| Monocular average | 7.0874 | 6.9924 | 9.4161 | 10.1860 | 0.0000 |
| Equal-weight fusion | 6.4953 | 6.3483 | 8.7991 | 9.6039 | +0.5921 |
| VUBTF soft fusion | 6.4775 | 6.3218 | 8.8179 | 9.6208 | +0.6099 |
| $HO3Dv3$ PA-MPVPE (778 vertices, frozen transfer) | | | | | |
| Mono A | 6.9523 | 6.7141 | 9.6297 | 10.4536 | — |
| Mono B | 7.0181 | 6.6324 | 9.8534 | 10.8022 | — |
| Monocular average | 6.9852 | 6.8803 | 9.1858 | 9.9498 | 0.0000 |
| Equal-weight fusion | 6.4274 | 6.3011 | 8.6324 | 9.3206 | +0.5578 |
| VUBTF soft fusion | 6.4141 | 6.2617 | 8.6532 | 9.3921 | +0.5711 |
(c)
| Prox. | Mid. | Dist. |
| Index | 0.66 | 0.69 | 0.62 |
| Middle | 0.57 | 0.45 | 0.33 |
| Pinky | 0.74 | 0.63 | 0.56 |
| Ring | 0.36 | 0.34 | 0.43 |
| Thumb | 0.56 | 0.53 | 0.50 |
(d)
Figure 7: Qualitative dual-view MANO fusion on DexYCB. (a)–(b) Synchronized views with monocular predictions and enlarged hand crops. (c) Ground-truth joints (GT), equal-weight fusion, and VUBTF soft fusion. (d) Per-joint view weights for proximal (Prox.), middle (Mid.), and distal (Dist.) finger joints.
Frozen transfer to HO3Dv3. With the adapted WiLoR checkpoint and DexYCB-fitted gate frozen, we evaluate 4,854 synchronized HO3Dv3 frames spanning 19,242 of the 20,137 images in the official evaluation split, which yield 33,456 unordered pair appearances. The remaining 895 images have no synchronized multi-view counterpart. As on DexYCB, all metrics are computed locally from the released MANO annotations, which we decode into the 778-vertex reference meshes used for PA-MPVPE.
As Table 4 shows, equal-weight fusion improves the HO3Dv3 monocular average by 0.5921 mm (8.35%) in PA-MPJPE and 0.5578 mm (7.99%) in PA-MPVPE. VUBTF soft fusion achieves the lowest mean and median on both metrics, improving it by 0.6099 mm (8.61%) and 0.5711 mm (8.18%).
---
### 5.4 Adaptive S O(3) Temporal Refinement
The temporal module refines only local rotations, preserving sequence-level shape. With $\boldsymbol{D}_{t,j} = \boldsymbol{R}_{t-1,j}^{\top} \boldsymbol{R}_{t,j}$ and $A_{t,j} = \boldsymbol{D}_{t,j}^{\top} \boldsymbol{D}_{t+1,j}$, the angular-velocity and angular-acceleration errors are $d(\hat{\boldsymbol{D}}, \boldsymbol{D}^{*})$ and $d(\hat{\boldsymbol{A}}, \boldsymbol{A}^{*})$, while acceleration magnitude is $d(\boldsymbol{I}, \hat{\boldsymbol{A}})$ and needs no reference. All three average over local rotations and over transitions within a sequence, in ${}^{\circ}/\text{frame}$ and ${}^{\circ}/\text{frame}^2$. UnrealEgo-RW has no ground-truth SMPL-X local rotations, so the body branch reports magnitude only.
Hand branch. We select the temporal-refinement settings on DexYCB S0 validation and keep them fixed during testing. On 71,510 validation pair appearances, we compare six fixed gains ( $\kappa = 0.50$, 0.65, 0.75, 0.85, 0.90, 0.95) with three adaptive variants that compute $\kappa$ for each frame and MANO joint. The variants differ in their baseline process and observation variances and in the weight on cross-view disagreement. The unrefined baseline obtains 4.0144 mm PA-MPJPE. The best fixed gain ( $\kappa = 0.75$) reduces it to 4.0013 mm, while the best adaptive variant reaches 3.9922 mm. The selected adaptive settings are
$$\begin{aligned}\left(\sigma_{proc,0}^{2},\lambda_{\nu},\right.&\left.\sigma_{obs,0}^{2},\lambda_{rel},\lambda_{\Delta}\right)\\=&\left(1.0,0.5,0.1,0.25,0.5\right)\end{aligned}$$
and $(\sigma_{v}, \sigma_{\Delta}) = (0.12, 0.25)$ rad. Both are frozen before testing.
On 275,688 test pair appearances (Table 5), adaptive refinement lowers the PA-MPJPE of the VUBTF observation, the fused estimate before refinement, by 0.0251 mm, with a sequence-block-bootstrap 95% interval of [0.0212,0.0290] mm. It improves all 28 camera-pair means and cuts acceleration magnitude by 31.04%.
Body branch. With the body predictor and gate frozen, we instead fit four effective scalars of Equation (12), namely $a_{\nu} = \lambda_{\nu}/\sigma_{\nu}^{2}$, $\sigma_{\mathrm{obs},0}^{2}$, $\lambda_{\mathrm{rel}}$, and $a_{\Delta} = \lambda_{\Delta}/\sigma_{\Delta}^{2}$, by minimizing reliable-13 PA-MPJPE over 105,126 consecutive training pairs from the gate-training subjects. Optimization uses Adam for 10 epochs with a learning rate of 0.02, and epoch 6 is selected by the same criterion on 4,662 S16 validation pairs. The fitted values are $a_{\nu} = 31.4855$, $\sigma_{\mathrm{obs},0}^{2} = 0.03987$, $\lambda_{\mathrm{rel}} = 0.83239$, and $a_{\Delta} = 3.56423$, with $\sigma_{\mathrm{proc},0}^{2}$ fixed to one.
On the S1, S9, and S11 test subjects, the fitted setup lowers GT15 PA-MPJPE from 76.9409 mm to 76.5428 mm, improving each subject. The resulting 0.3981 mm gain has a sequence-block-bootstrap 95% interval of [0.3012, 0.5045] mm, and angular-acceleration magnitude falls from 1.8535 to $1.2569^\circ/\text{frame}^2$ (32.19%).
### 5.5 Wearable-System Demonstration
According to the policy of the authors' institution, ethics review was not required for this study. This experiment tests whether the calibrated wearable produces uninterrupted trajectory and articulation outputs on a real recording.
#### 5.5.1 Outward-Looking Camera Tracking
Following Section 4.1, the rig trajectory places the downward-pair estimates in a common world frame. The four outward streams—the $170^{\circ}$ forward pair (Cam1/2) and the $220^{\circ}$ lateral pair (Cam5/6)—are tracked as a single calibrated generalized camera, with the extrinsics of Section 3.3 held fixed so that metric scale follows from the inter-camera baselines. Each view uses its calibrated Scaramuzza fisheye model (Scaramuzza et al., 2006). ORB descriptors are aggregated across the four views for loop-closure detection.
Demonstration sequence. From a 35.7-s synchronized six-camera recording at $1920 \times 1080$ and 59.94 fps, the four outward streams track without loss and export 2,121 timestamp-aligned 6-DoF rig poses. Calibration reached 0.093/0.106 px reprojection error. Figure 8 shows the recovered trajectory and its sparse map, with the raw views in Figure 1. During this recording, the wearer performs near-body interactions within a limited workspace. The estimated trajectory has a cumulative length of 4.635 m, stays within 0.279 m of the start, and ends 0.220 m away. Inter-frame translation and rotation stay below 0.022 m and $1.49^\circ$; over the sequence, orientation deviates from its initial value by at most $41.1^\circ$ as a geodesic angle on $SO(3)$.
#### 5.5.2 Real-World Body and Hand Motion Capture
The same synchronized six-camera recording is processed end to end. Metric root placement uses the downward pair's 152.02 mm baseline. Crops come from off-the-shelf detectors: the hand detector released with WiLoR (Potamias et al., 2025) gives the hand boxes and their left/right labels, and a Faster R-CNN (Ren et al., 2015) gives the body box. Neither is fine-tuned, both boxes are enlarged
---
Table 5: Temporal-refinement results on DexYCB S0 test. PA-MPJPE is in mm; angular metrics use ${}^{\circ}$/frame for velocity and ${}^{\circ}$/frame $^{2}$ for acceleration.
| Method | PA- MPJPE | Angular- velocity error | Angular- acceleration error | Acceleration magnitude |
| VUBTF observation | 4.2872 | 6.1053 | 8.1901 | 2.2269 |
| Fixed gain, $\kappa = 0.75$ | 4.2687 | 5.9673 | 7.8929 | 1.4968 |
| Adaptive gain | 4.2621 | 5.9601 | 7.8861 | 1.5356 |
Figure 8: World-frame rig trajectory and sparse map estimated from the four outward views, colored by time over the 35.7-s sequence. Sampled axes show rig orientation.
by $1.2 \times$ before cropping, and Cam3 and Cam4 are detected independently.
Outputs. Calibrated rig transforms express the head-mounted rig, body, and both hands in one world frame (Figure 1). Figure 9 projects the body and hand outputs back into the downward pair in four cases: (1) an unoccluded frame, (2) hand–object interaction, (3) hand–object occlusion, and (4) one hand outside the field of view. Cases (1) and (2) are the regime the paired-view fusion is designed for. In cases (3) and (4) one hand is missing from a single view, which is what the availability marking of Section 3.4 exists to handle: a hand missing from either view cannot be triangulated, so its metric placement is marked unavailable.
Detection is near-complete: Cam3 finds the left hand in 2,120 of the 2,121 frames and the right hand in 2,074, Cam4 in 2,069 and 2,048, and the body in every frame of both views. Triangulation therefore succeeds for the left hand in 2,069 frames and the right hand in 2,026, giving metric placement for both hands in 2,011 frames,
Figure 9: Downward-pair output: no occlusion, hand–object interaction, hand–object occlusion, and one hand out of view. SMPL-X body (white) and MANO hands (blue left, red right) are reprojected into Cam3 and Cam4. The right hand in case (3) and the left hand in case (4) are seen in one view only, so they are single-view predictions.
for one hand in 73, and for neither in 37. For the MANO branch, the gate and temporal configuration are frozen from DexYCB validation, and no ground truth from this recording is used. Over valid consecutive paired-hand segments, per-joint soft fusion reaches an acceleration magnitude of $3.5866^{\circ}/\text{frame}^2$, which the adaptive $SO(3)$ refinement lowers to $2.1132^{\circ}/\text{frame}^2$, a reduction of 41.08%. The gate stays active on this rig: Cam3 weights average 0.5116 but span 0.098 to 0.896 across joints, with a mean adaptive gain of 0.7079. This demonstration establishes synchronized output availability and end-to-end deployment.
### 5.6 Limitations and Future Work
The benchmark protocols report similarity-aligned error, which solves for scale, rotation, and transla-
---
tion and therefore cannot evaluate absolute placement at all. They also lack our downward-pair geometry, and the wearable recording carries no external reference, so the articulation gains are measured under the standard protocols while the trajectory and placement stages are demonstrated rather than quantified. Most of the benchmark gain comes from using both views; the learned gates add a further increment over equal-weight fusion. On HO3Dv3 the frozen gate improves the mean and median but not P90 or P95. The MANO hands are placed at their independently triangulated roots rather than being kinematically reconciled with the SMPL-X arm chains, so forearm length is not enforced and the body and hand shape parameters may disagree. Root orientation and translation are not temporally filtered, and the pipeline is offline because temporal inconsistency uses a following frame and shape uses the full sequence.
Four directions follow from the present design. Reconciling the triangulated MANO root with the arm chain would make the exported skeleton kinematically consistent and let forearm length constrain the hand. Replacing the two-sided temporal cue and the sequence-level shape estimate with causal counterparts would make the pipeline online. A controlled capture with an external tracking reference would turn metric placement from a demonstrated capability into a measured one. Reducing weight toward eyeglasses scale, with temple-mounted outward cameras and overlapping oblique-downward views, would preserve the same role split.
## 6 Conclusion
We presented EgoWholeView, a six-camera wearable and an offline pipeline that recovers body and bimanual hand motion by reliability-aware fusion and $SO(3)$ temporal refinement. Across UnrealEgo-RW, DexYCB, and HO3Dv3, dual-view fusion improves similarity-aligned local geometry, and temporal refinement improves all 28 DexYCB camera-pair means. The same recurrence applies to the body branch. On the rig itself, gates frozen from the benchmarks are used without returning: metric placement is available for both hands in 2,011 of 2,121 frames, and refinement lowers the hand acceleration magnitude by 41.08%. Together, these results support role-specialized sensing and canonical-space fusion.
## References
Hiroyasu Akada, Jian Wang, Soshi Shimada, Masaki Takahashi, Christian Theobalt, and Vladislav Golyanik. 2022. UnrealEgo: A new dataset for robust egocentric 3D human motion capture. In Proceedings of the European Conference on Computer Vision.
Prithviraj Banerjee, Sindi Shkodrani, Pierre Moulon, Shreyas Hampali, Shangchen Han, Fan Zhang, Linguang Zhang, Jade Fountain, Edward Miller, Selen Basol, Richard Newcombe, Robert Wang, Jakob Julian Engel, and Tomas Hodan. 2025. HOT3D: Hand and object tracking in 3D from egocentric multi-view videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7061–7071.
Doug A. Bowman, Ernst Kruijff, Joseph J. LaViola, and Ivan Poupyrev. 2004. 3D User Interfaces: Theory and Practice. Addison-Wesley.
Lennart Bramlage, Michelle Karg, and Cristobal Curio. 2023. Plausible uncertainties for human pose regression. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15133–15142.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. 2023. RT-2: Vision-language-action models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.15818.
Carlos Campos, Richard Elvira, Juan J. Gómez Rodríguez, José M. M. Montiel, and Juan D. Tardós. 2021. ORB-SLAM3: An accurate open-source library for visual, visual-inertial and multi-map SLAM. IEEE Transactions on Robotics, 37(6):1874–1890.
Géry Casiez, Nicolas Roussel, and Daniel Vogel. 2012. 1 euro filter: A simple speed-based low-pass filter for noisy input in interactive systems. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pages 2527–2530.
Yu-Wei Chao, Wei Yang, Yu Xiang, Pavlo Molchanov, Ankur Handa, Jonathan Tremblay, Yashraj S. Narang, Karl Van Wyk, Umar Iqbal, Stan Birchfield, Jan Kautz, and Dieter Fox. 2021. DexYCB: A benchmark for capturing hand grasping of objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.
Woojin Cho, Taewook Ha, Taejun Son, and Woontack Woo. 2026. A unified hand and gesture tracking via offloading framework for object-mediated interaction in wearable AR. In Proceedings of the IEEE Conference on Virtual Reality and 3D User Interfaces.
Mateo de Mayo, Daniel Cremers, and Taihú Pire. 2025. The Monado SLAM dataset for egocentric visual-inertial tracking. arXiv preprint arXiv:2508.00088.
---
Sai Kumar Dwivedi, Cordelia Schmid, Hongwei Yi, Michael J. Black, and Dimitrios Tzionas. 2024. POCO: 3d pose and shape estimation with confidence. In International Conference on 3D Vision, pages 85–95.
Jakob Engel, Kiran Somasundaram, Michael Goesele, et al. 2023. Project Aria: A new tool for egocentric multi-modal AI research. arXiv preprint arXiv:2308.13561.
Jessie Colette Eubanks, Alec G. Moore, Paul A. Fishwick, and Ryan P. McMahan. 2020. The effects of body tracking fidelity on embodiment of an inverse-kinematic avatar for male participants. In Proceedings of the IEEE International Symposium on Mixed and Augmented Reality, pages 54–63.
Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. 2022. Ego4D: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18973–18990.
Shreyas Hampali, Mahdi Rad, Markus Oberweger, and Vincent Lepetit. 2020. HOnnotate: A method for 3D annotation of hand and object poses. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.
Shangchen Han, Beibei Liu, Randi Cabezas, Christopher D. Twigg, Peizhao Zhang, Jeff Petkau, Tsz-Ho Yu, Chun-Jung Tai, Muzaffer Akbay, Zheng Wang, Asaf Nitzan, Gang Dong, Yuting Ye, Lingling Tao, Chengde Wan, and Robert Wang. 2020. MEgATrack: Monochrome egocentric articulated hand-tracking for virtual reality. ACM Transactions on Graphics, 39(4).
Shangchen Han, Po-Chen Wu, Yubo Zhang, Beibei Liu, Linguang Zhang, Zheng Wang, Weiguang Si, Peizhao Zhang, Yujun Cai, Tomas Hodan, Randi Cabezas, Luan Tran, Muzaffer Akbay, Tsz-Ho Yu, Cem Keskin, and Robert Wang. 2022. UmeTrack: Unified multi-view end-to-end hand tracking for VR. ACM Transactions on Graphics, 41(6).
Yaoqing Hu, Mingzhu Zhu, Shaoan Wang, Dongyue Li, Fusong Yuan, and Junzhi Yu. 2024. A novel lightweight navigation system for oral and maxillofacial surgery using an external curved self-identifying checkerboard. IEEE Transactions on Automation Science and Engineering, 21(2):1434–1444.
Catalin Ionescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu. 2014. Human3.6M: Large scale datasets and predictive methods for 3D human sensing in natural environments. IEEE Transactions on Pattern Analysis and Machine Intelligence, 36(7):1325–1339.
Karim Iskakov, Egor Burkov, Victor Lempitsky, and Yury Malkov. 2019. Learnable triangulation of human pose. In Proceedings of the IEEE/CVF
International Conference on Computer Vision, pages 7718–7727.
Rudolf E. Kalman. 1960. A new approach to linear filtering and prediction problems. Journal of Basic Engineering, 82(1):35–45.
Alexander Khazatsky, Karl Pertsch, Suraj Nair, Ashwin Balakrishna, Sudeep Dasari, Siddharth Karamcheti, Soroush Nasiriany, Mohan Kumar Srirama, Lawrence Yunliang Chen, Kirsty Ellis, et al. 2024. DROID: A large-scale in-the-wild robot manipulation dataset. In Proceedings of Robotics: Science and Systems.
Konstantina Kilteni, Raphaela Groten, and Mel Slater. 2012. The sense of embodiment in virtual reality. Presence: Teleoperators and Virtual Environments, 21(4):373–387.
Bowon Kim, Eugene Hwang, and Jeongmi Lee. 2024. Self-avatar recognition in virtual reality: the self advantage phenomenon and the relative importance of motion and visual congruence. In Proceedings of the IEEE International Symposium on Mixed and Augmented Reality, pages 643–651.
Muhammed Kocabas, Nikos Athanasiou, and Michael J. Black. 2020. VIBE: Video inference for human body pose and shape estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.
Muhammed Kocabas, Chun-Hao P. Huang, Otmar Hilliges, and Michael J. Black. 2021. PARE: Part attention regressor for 3d human body estimation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11127–11137.
Nikos Kolotouros, Georgios Pavlakos, Dinesh Jayaraman, and Kostas Daniilidis. 2021. Probabilistic modeling for human mesh recovery. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11605–11614.
Dixuan Lin, Yuxiang Zhang, Mengcheng Li, Yebin Liu, Wei Jing, Qi Yan, Qianying Wang, and Hongwen Zhang. 2024. Omnihands: Towards robust 4d hand mesh recovery via a versatile transformer. arXiv preprint arXiv:2405.20330.
Weiquan Lin, Yaoqing Hu, Liangchen Dai, Xu Tang, and Xingyu Chen. 2026. GeoHand: Unlocking prior geometry knowledge for monocular 3d hand reconstruction. arXiv preprint arXiv:2605.17354.
Ruicong Liu, Takehiko Ohkawa, Mingfang Zhang, and Yoichi Sato. 2024. Single-to-dual-view adaptation for egocentric 3D hand pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 677–686.
Yunze Liu, Yun Liu, Che Jiang, Kangbo Lyu, Weikang Wan, Hao Shen, Boqiang Liang, Zhoujie Fu, He Wang, and Li Yi. 2022. HOI4D: A 4d egocentric dataset for category-level human-object interaction.
---
In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.
Lingni Ma, Yuting Ye, Fangzhou Hong, Vladimir Guzov, Yifeng Jiang, Rowan Postyeni, Luis Pesqueira, Alexander Gamino, Vijay Baiyya, Hyo Jin Kim, Kevin Bailey, David Soriano Fosas, C. Karen Liu, Ziwei Liu, Jakob Engel, Renzo De Nardi, and Richard Newcombe. 2024. Nymeria: A massive collection of multimodal egocentric daily motion in the wild. In Proceedings of the European Conference on Computer Vision, pages 445–465.
Antoine Maiorca, Seyed Abolfazl Ghasemzadeh, Thierry Ravet, François Cresson, Thierry Dutoit, and Christophe De Vleeschouwer. 2024. Self-avatar animation in virtual reality: Impact of motion signals artifacts on the full-body pose reconstruction. arXiv preprint arXiv:2404.18628.
Wiktor Mucha, Michael Wray, and Martin Kampel. 2026. Towards egocentric 3d hand pose estimation in unseen domains. arXiv preprint arXiv:2601.06537.
Soyeon Na, Seung Young Noh, and Ju Yong Chang. 2026. Egocentric whole-body human mesh recovery with prior-guided learning. arXiv preprint arXiv:2605.08606.
Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. 2022. R3M: A universal visual representation for robot manipulation. In Proceedings of the Conference on Robot Learning.
Frank C. Park and Bahram Ravani. 1997. Smooth invariant interpolation of rotations. ACM Transactions on Graphics, 16(3):277–295.
JoonKyu Park, Yeonguk Oh, Gyeongsik Moon, Hong-suk Choi, and Kyoung Mu Lee. 2022. HandOccNet: Occlusion-robust 3d hand mesh estimation network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1496–1505.
Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed A. A. Osman, Dimitrios Tzionas, and Michael J. Black. 2019. Expressive body capture: 3D hands, face, and body from a single image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.
Georgios Pavlakos, Dandan Shan, Ilija Radosavovic, Angjoo Kanazawa, David Fouhey, and Jitendra Malik. 2024. Reconstructing hands in 3d with transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.
Rolandos Alexandros Potamias, Jinglei Zhang, Jiankang Deng, and Stefanos Zafeiriou. 2025. Wilor: End-to-end 3d hand localization and reconstruction in-the-wild. arXiv preprint arXiv:2409.12259.
Aditya Prakash, Ruisen Tu, Matthew Chang, and Saurabh Gupta. 2024. 3d hand pose estimation in everyday egocentric images. In European Conference on Computer Vision.
Pengfei Ren, Haifeng Sun, Jiachang Hao, Jingyu Wang, Qi Qi, and Jianxin Liao. 2022. Mining multi-view information: A strong self-supervised framework for depth-based 3d hand pose and mesh estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.
Pengfei Ren, Jingyu Wang, Haifeng Sun, Qi Qi, Jing Wang, and Jianxin Liao. 2025. Rule meets learning: Confidence-aware multi-view fusion for self-supervised 3D hand pose estimation. In Proceedings of the 33rd ACM International Conference on Multimedia, pages 1646–1655.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015. Faster R-CNN: Towards real-time object detection with region proposal networks. In Advances in Neural Information Processing Systems, pages 91–99.
Patrick Rim, Kun He, Kevin Harris, Braden Copple, Shangchen Han, Sizhe An, Ivan Shugurov, Tomas Hodan, He Wen, and Xu Xie. 2025. Ego-exo 3d hand tracking in the wild with a mobile multi-camera rig. arXiv preprint arXiv:2510.02601.
Javier Romero, Dimitrios Tzionas, and Michael J. Black. 2017. Embodied hands: Modeling and capturing hands and bodies together. ACM Transactions on Graphics, 36(6).
Ropedia. 2026. Xperience-10M: A large-scale egocentric multimodal dataset with structured 3D/4D annotations. Hugging Face dataset. Accessed July 12, 2026.
Davide Scaramuzza, Agostino Martinelli, and Roland Siegwart. 2006. A flexible technique for accurate omnidirectional camera calibration and structure from motion. In Proceedings of the IEEE International Conference on Computer Vision Systems, page 45.
Akash Sengupta, Ignas Budvytis, and Roberto Cipolla. 2023. HuManiFlow: Ancestor-conditioned normalising flows on SO(3) manifolds for human pose and shape distribution estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4779–4789.
Mel Slater, Bernhard Spanlang, Maria V. Sanchez-Vives, and Olaf Blanke. 2010. First person experience of body transfer in virtual reality. PLoS ONE, 5(5):e10564.
Denis Tome, Patrick Peluse, Lourdes Agapito, and Hernan Badino. 2019. xR-EgoPose: Egocentric 3D human pose from an HMD camera. In Proceedings of the IEEE/CVF International Conference on Computer Vision.
---
Tze Ho Elden Tse, Franziska Mueller, Zhengyang Shen, Danhang Tang, Thabo Beeler, Mingsong Dou, Yinda Zhang, Sasa Petrovic, Hyung Jin Chang, Jonathan Taylor, and Bardia Doosti. 2023. Spectral graphormer: Spectral graph-based transformer for egocentric two-hand reconstruction using multi-view color images. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14666–14677.
Steffen Urban and Stefan Hinz. 2016. MultiCol-SLAM: A modular real-time multi-camera SLAM system. arXiv preprint arXiv:1610.07336.
Thomas Waltemate, Dominik Gall, Daniel Roth, Mario Botsch, and Marc Erich Latoschik. 2018. The impact of avatar personalization and immersion on virtual body ownership, presence, and emotional response. IEEE Transactions on Visualization and Computer Graphics, 24(4):1643–1652.
Jian Wang, Lingjie Liu, Weipeng Xu, Kripasindhu Sarkar, Diogo Luvizon, and Christian Theobalt. 2022. Estimating egocentric 3d human pose in the wild with external weak supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13157–13166.
Jian Wang, Diogo Luvizon, Weipeng Xu, Lingjie Liu, Kripasindhu Sarkar, and Christian Theobalt. 2023a. Scene-aware egocentric 3d human pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.
Shaoan Wang, Mingzhu Zhu, Yaoqing Hu, Dongyue Li, Fusong Yuan, and Junzhi Yu. 2024. CylinderTag: An accurate and flexible marker for cylinder-shape objects pose estimation based on projective invariants. IEEE Transactions on Visualization and Computer Graphics, 30(12):7486–7499.
Xin Wang, Taein Kwon, Mahdi Rad, Bowen Pan, Ishani Chakraborty, Sean Andrist, Dan Bohus, Ashley Feniello, Felipe Vieira Frujeri, Neel Joshi, and Marc Pollefeys. 2023b. HoloAssist: An egocentric human interaction dataset for interactive AI assistants in the real world. In Proceedings of the IEEE/CVF International Conference on Computer Vision.
Vasco Xu, Brian Chen, Eric J. Gonzalez, Andrea Colaço, Henry Hoffmann, Mar Gonzalez-Franco, and Karan Ahuja. 2026. SurfaceXR: Fusing smartwatch IMUs and egocentric hand pose for seamless surface interactions. IEEE Transactions on Visualization and Computer Graphics. IEEE VR journal track; arXiv:2603.19529.
Weipeng Xu, Avishek Chatterjee, Michael Zollhoefer, Helge Rhodin, Pascal Fua, Hans-Peter Seidel, and Christian Theobalt. 2019. Mo2Cap2: Real-time mobile 3D motion capture with a cap-mounted fish-eye camera. IEEE Transactions on Visualization and Computer Graphics, 25(5):2093–2101.
Lixin Yang, Jian Xu, Licheng Zhong, Xinyu Zhan, Zhicheng Wang, Kejian Wu, and Cewu Lu. 2023.
POEM: Reconstructing hand in a point embedded multi-view stereo. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21108–21117.
Brent Yi, Vickie Ye, Maya Zheng, Yunqi Li, Lea Müller, Georgios Pavlakos, Yi Ma, Jitendra Malik, and Angjoo Kanazawa. 2025. Estimating body and hand motion in an ego-sensed world. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7072–7084.
Jinglei Zhang, Jiankang Deng, Chao Ma, and Rolandos Alexandros Potamias. 2025a. Hawor: World-space hand motion reconstruction from egocentric videos. arXiv preprint arXiv:2501.02973.
Siwei Zhang, Qianli Ma, Yan Zhang, Zhiyin Qian, Taein Kwon, Marc Pollefeys, Federica Bogo, and Siyu Tang. 2022. EgoBody: Human body shape and motion of interacting people from head-mounted devices. In Proceedings of the European Conference on Computer Vision.
Tianhao Zhang, Zoe McCarthy, Owen Jow, Dennis Lee, Xi Chen, Ken Goldberg, and Pieter Abbeel. 2018. Deep imitation learning for complex manipulation tasks from virtual reality teleoperation. In Proceedings of the IEEE International Conference on Robotics and Automation.
Yuhong Zhang, Zihan Gao, Shengpeng Li, Ling-Hao Chen, Kaisheng Liu, Runqing Cheng, Xiao Lin, Junjia Liu, Zhuoheng Li, Jingyi Feng, Ziyan He, Jintian Lin, Zheyan Huang, Zhifang Liu, and Haoqian Wang. 2025b. Robowheel: A data engine from real-world human demonstrations for cross-embodiment robotic learning. arXiv preprint arXiv:2512.02729.
Dongxu Zhao, Zhen Wei, Jisan Mahmud, and Jan-Michael Frahm. 2021. EgoGlass: Egocentric-view human pose estimation from an eyeglass frame. In Proceedings of the International Conference on 3D Vision, pages 32–41.
Xiaozheng Zheng, Chao Wen, Zhou Xue, Pengfei Ren, and Jingyu Wang. 2023. HaMuCo: Hand pose estimation via multiview collaborative self-supervised learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 20763–20773.