# YOCO: You Only Calibrate Once! Fast Mocap Calibration for Dexterous Teleoperation

Yu Zhang $^{1,2,3}$, Yunqi Li $^{1,3}$, Yushi Du $^{1}$, Yi Ma $^{1,2,3}$, Yanchao Yang $^{1,2,3}$

 $^{1}$The University of Hong Kong, $^{2}$Shenzhen Loop Area Institute, $^{3}$TranscEngram

![Image](images/e0cc02c0fdccd8157d060f02b1db119bfc24e8208f80602c1817597c9d8fc174.jpg)

<div style="text-align: center;">Figure 1: Overview of YOCO. Low-cost mocap streams suffer from user-, device-, and session-specific bias. Given a few paired calibration poses, YOCO predicts a calibration update that corrects future mocap poses and improves downstream dexterous teleoperation.</div>

Abstract: Dexterous teleoperation requires reliable human-hand state estimations. However, common low-cost motion-capture gloves and markerless trackers often exhibit biases that vary across users, glove fit, and recording sessions, degrading retargeting and demonstration quality. We present YOCO, a fast few-shot, fine-tuning-free calibration framework that corrects biased hand-pose streams from a small set of paired raw and target poses. Instead of optimizing a separate model for every operator or session, YOCO conditions a calibration HyperNet on the paired examples and predicts LoRA-style updates for a frozen MANO hand-estimation module, turning per-user calibration into a lightweight feed-forward adaptation step while preserving the geometric prior of MANO and the efficiency of a compact estimator. We train YOCO with synthetic drift augmentations on InterHand2.6M and evaluate on augmented InterHand sequences, offline real glove data, and dexterous teleoperation tasks. Across these settings, YOCO improves calibration efficiency, hand-state estimation quality and teleoperation performance compared with uncalibrated input and standard calibration baselines.

Keywords: Hand Pose Estimation and Calibration, Teleoperation

## 1 Introduction

Teleoperation is a key tool for collecting dexterous hand demonstrations, but its effectiveness depends on the fidelity of the operator's hand-state estimates. High-end motion-capture systems can provide accurate kinematics, yet they are expensive and often difficult to deploy at scale. Lower-cost gloves and markerless trackers are more accessible, but their outputs are affected by user hand morphology gap, device fit, sensor drift, and session-dependent bias. Such errors are particularly

---

detrimental in dexterous manipulation, where even small fingertip or joint offsets can alter contact patterns, pinch geometry, and retargeted robot motion.

Prior hand-mocap calibration typically operates within a particular sensing pipeline, learning glove-signal-to-pose mappings, correcting IMU alignment and drift, or personalizing sensor-to-pose models [^1] [^2] [^3] [^4]. We instead formulate few-shot calibration at the derived 3D hand-keypoint interface, after a tracker has produced its pose estimate. Given K paired raw and target poses from a user and session, YOCO conditions a HyperNet on those examples and generates LoRA updates for a frozen MANO-based estimator in one pass, without per-session gradient optimization. This couples a sensor-independent pose interface with amortized, session-conditioned weight generation: unlike a LoRA initialization shared across sessions, the generated updates depend on the current calibration set (Appendix B). The formulation requires no access to device-specific raw signals; our real-device evaluation remains limited to one tensile glove.

Our contributions are threefold. First, we formulate calibration of derived 3D hand keypoints from a few pose pairs, separating the correction from device-specific sensor-to-pose calibration. Second, we generate session-conditioned LoRA weights for a frozen MANO estimator in one pass; learned-prior and set-conditioned residual controls test whether the gains stem from pair-conditioned weight generation beyond a shared prior or set conditioning alone. Third, we evaluate accuracy and adaptation cost on augmented InterHand2.6M and paired real-glove/AVP data, then test the resulting hand poses in physical teleoperation.

## 2 Related Work

Mocap Devices: High-fidelity dexterous teleoperation depends on reliable capture of the operator hand, yet existing mocap interfaces exhibit a strong accuracy-accessibility trade-off. Glove- and exoskeleton-based systems provide stable kinematics but are costly, cumbersome, and sensitive to user-specific geometry and donning conditions [^5] [^1] [^6] [^2]. The resulting joint biases or fingertip drift propagate directly into the collected demonstrations. Markerless and RGB-based pipelines, including FreiHAND, MediaPipe Hands, and HaMeR, have improved accessibility [^7] [^8] [^9] [^10] [^11] [^12] [^13], but still struggle with occlusion, hand-object interaction, cross-user morphology, and temporal stability. YOCO is complementary: rather than introducing a new tracker, it calibrates existing glove-based streams so that inexpensive devices become more consistent for downstream use.

Prior Hand-Mocap Calibration: Wang and Neff [^1] learn a mapping from glove measurements to hand motion. Gosala et al. [^4] use optical reference data to personalize a wearable sensor-to-pose model through side-tuning. FSGlove [^2] jointly optimizes IMU alignment, hand shape, and kinematics, while the Transformer IMU Calibrator [^3] estimates time-varying IMU drift and offsets. These approaches address calibration within particular sensing pipelines. YOCO instead takes their downstream 3D-keypoint representation as its input and learns to generate a session-specific correction of a frozen MANO estimator from a few paired poses. This interface makes the formulation independent of raw sensor modality; generalization beyond the evaluated tensile glove remains to be tested.

Dexterous Hand Retargeting and Teleoperation: Recent teleoperation systems have made dexterous data collection cheaper, more immersive, and more hardware-agnostic. Vision-based systems such as DexPilot and AnyTeleop reduce dependence on instrumented gloves [^14] [^15] [^16], while VR-based platforms such as Open-TeleVision and OPEN TEACH improve operator immersion [^17] [^18]. In parallel, learning-based retargeting methods close the embodiment gap beyond purely kinematic alignment [^19] [^20] [^21] [^22], and new master-side or robot-side designs improve transferability and contact richness [^23] [^24] [^25]. These efforts largely assume that the upstream hand observation is already accurate; YOCO targets that bottleneck and is compatible with the downstream stacks above.

HyperNetworks and Parameter-Efficient Adaptation: HyperNetworks generate the weights of one network with another [^26], and task-conditioned variants support efficient cross-task adaptation

---

[^27] [^28] [^29] [^30] [^31]. In parallel, parameter-efficient methods such as Compacter and LoRA specialize a frozen backbone with compact low-rank updates [^32] [^33], and hypernetworks have been shown to predict such updates from limited conditioning input [^34]. YOCO brings these ideas to dexterous teleoperation: $h_{\psi}$ predicts LoRA-style updates $\Delta W$ for the hand estimator from a handful of paired calibration poses, yielding fast and robust adaptation.

## 3 Method

### 3.1 Problem Formulation

We consider the problem of calibrating noisy mocap hand poses using a small set of paired calibration examples. A hand observation is represented as 21 3D joint keypoints $x \in \mathbb{R}^{21 \times 3}$. Let

 $$ \mathcal{S}=\{(x_{i}^{n},x_{i}^{c})\}_{i=1}^{K} $$

denote the calibration set, where $x_{i}^{n}$ is a noisy mocap hand-pose keypoint observation sampled from the noisy pose distribution $\mathcal{D}_{\mathrm{noisy}}$, and $x_{i}^{c}$ is its corresponding target 3D keypoint pose. The calibration set S is assumed to capture the approximately stationary bias of the current setting, such as a particular subject, glove fit, or recording session.

We formulate calibration as the composition of two networks with distinct roles.

Target network. A MANO estimator

 $$ g_{\phi}:\mathbb{R}^{21\times3}\longrightarrow\mathbb{R}^{45},\qquad\hat{\theta}=g_{\phi}(x), $$

maps 21 3D hand keypoints to MANO pose parameters $\hat{\theta}$. Its parameters $\phi$ are trained once on clean data and then held fixed; per-user and per-session calibration acts on $g_{\phi}$ only through additive LoRA updates rather than by re-optimizing $\phi$.

Hypernetwork. A calibration HyperNet

 $$ h_{\psi}:\;\bigl(\mathbb{R}^{21\times3}\times\mathbb{R}^{21\times3}\bigr)^{K}\longrightarrow\mathcal{W},\qquad\Delta W=h_{\psi}(\mathcal{S}), $$

takes the calibration set as input and outputs a set of low-rank LoRA matrices $\Delta W$, where $\mathcal{W}$ denotes the space of LoRA adapters attached to $g_{\phi}$. The parameters $\psi$ are shared across all users and devices and constitute the only learned parameters of the calibration system; $\phi$ is fixed.

Calibrated estimator. Injecting $\Delta W$ into the target network yields a session-specific calibrated estimator

 $$ g_{\phi+\Delta W}:\mathbb{R}^{21\times3}\longrightarrow\mathbb{R}^{45}, $$

which, composed with MANO forward kinematics, produces the calibrated 3D joints

 $$ \begin{array}{r}{\hat{x}_{q}^{c}\;=\;\Pi_{\mathrm{M A N O}}\big(g_{\phi+h_{\psi}({\mathcal{S}})}\big(x_{q}^{n}\big),\;\beta\big)\;=\:f_{\psi}\big(x_{q}^{n};{\mathcal{S}}\big),}\end{array} $$

for any new query $x_q^n \sim \mathcal{D}_{\text{noisy}}$. The shape parameter $\beta$ is handled consistently across methods in each evaluation setting, as described in Section 5.

Learning objective. Since $\phi$ is frozen, only the hypernetwork parameters $\psi$ are trained, by minimizing the expected joint-position error of the resulting calibrated estimator:

 $$ \min_{\psi}\mathbb{E}_{\mathcal{S},\left(x_{q}^{n},x_{q}^{c}\right)}\big[\mathcal{L}\big(f_{\psi}\big(x_{q}^{n};\mathcal{S}\big),x_{q}^{c}\big)\big], $$

where $\mathcal{L}$ is an $L_2$ distance between hand joint positions. This objective encourages $h_\psi$ to infer the underlying mocap bias from the paired calibration set and produce LoRA updates that correct unseen noisy poses drawn from the same distribution.

### 3.2 Model Overview

Figure 2 illustrates how the target network $g_{\phi}$ and the hypernetwork $h_{\psi}$ defined in Section 3.1 are combined at deployment. Calibration consists of a one-time feed-forward conditioning pass

---

![Image](images/9ca43ac56e716a7637990cef72098f9ca1cb68bc1a1150097ad4d997fd10d20c.jpg)

<div style="text-align: center;">Figure 2: YOCO model architecture. Given a calibration set  $\mathcal{S} = \{(x_i^n, x_i^c)\}_{i=1}^K$ of  K paired noisy and target poses (left), the calibration HyperNet  $h_{\psi}$ (middle) encodes each pair, aggregates them through a transformer-based calibration mapper with one CLS token per LoRA matrix, and decodes the tokens into LoRA updates  $\Delta W$. These updates are injected once per session into the frozen MANO estimator  $g_{\phi}$ (right) to form  $g_{\phi+\Delta W}$, which maps each raw mocap query  $x_q^n$ at frame rate to calibrated MANO pose and 3D joints.</div>

followed by a frame-rate query path, both fully gradient-free: (i) the HyperNet ingests the calibration set and outputs LoRA matrices, $\Delta W = h_{\psi}(S)$; (ii) these matrices are injected into the frozen MANO estimator to form the calibrated estimator $g_{\phi+\Delta W}$; (iii) each subsequent raw mocap pose $x_q^n$ is mapped through $g_{\phi+\Delta W}$ to obtain $\hat{\theta}_q$, from which MANO reconstructs the calibrated joints $\hat{x}_q^c$. Step (i) is executed only once per session, while step (iii) runs at the mocap frame rate. The remainder of this section specifies the architectures of $g_{\phi}$ (Section 3.3) and $h_{\psi}$ (Section 3.4).

### 3.3 Target Network: MANO Estimator Architecture

The MANO estimator $g_{\phi}$ consists of a hand-keypoint encoder and a MANO parameter decoder. The hand-keypoint encoder uses finger-wise CNN keypoint encoders followed by a transformer fusion layer. The MANO parameter decoder uses finger-wise MLPs and a unified MANO $\theta$ decoder. The finger-wise design prevents each finger estimate from being overly affected by unrelated fingers, while the fusion layer captures global hand-pose correlation. The decoder is deliberately lightweight so that it can run in real time and accept LoRA updates from the HyperNet.

### 3.4 Hypernetwork: Calibration HyperNet Architecture

The calibration HyperNet $h_{\psi}$ realizes the mapping $\mathcal{S} \mapsto \Delta W$: its input is the K paired noisy/target poses in the calibration set, and its output is the LoRA low-rank matrices $\Delta W$ injected into the frozen MANO estimator $g_{\phi}$. To produce useful adapters from only a handful of pairs, $h_{\psi}$ must infer the session-specific drift pattern from a small amount of runtime calibration data.

The HyperNet consists of three submodules: (1) hand-keypoint encoder, (2) calibration mapping, and (3) LoRA parameter decoder. The hand-keypoint encoder reuses the encoder from the MANO estimator $g_{\phi}$ to extract pose features from each raw/calibrated pair. The calibration mapping is a transformer block with RoPE positional embedding [^35] that models the mapping relationship across the paired examples. It includes one learnable CLS token for each LoRA matrix to be generated; after calibration-map aggregation, each CLS token conditions the generation of its corresponding LoRA matrix. The LoRA parameter decoder is a DiT-style decoder [^36] that maps these LoRA-specific CLS tokens to the low-rank matrices in $\Delta W$. We use the same LoRA adapter footprint for YOCO and the LoRA-Finetune baseline; details are provided in Appendix A.2.

---

## 4 Model Training

Training proceeds in two stages that mirror the role separation introduced in Section 3.1: first the target network $g_{\phi}$ is trained on clean data, then it is frozen and the hypernetwork $h_{\psi}$ is trained to generate LoRA updates that correct simulated mocap bias.

### 4.1 Target Network Pretraining

We train the parameters $\phi$ of the MANO estimator $g_\phi$ on clean InterHand2.6M data, using the mean squared error between the predicted MANO pose parameter $\hat{\theta} = g_\phi(x)$ and the ground-truth MANO pose parameter. This pretraining establishes a hand-pose prior in MANO space that the HyperNet can later specialize through LoRA updates. After this stage, $\phi$ is frozen and never updated again.

### 4.2 Hypernetwork Training

With $\phi$ fixed, we train the hypernetwork parameters $\psi$ so that $h_{\psi}$ produces LoRA updates $\Delta W$ that compensate for realistic mocap bias. To expose $h_{\psi}$ to a diverse family of bias patterns, we apply synthetic augmentations to InterHand2.6M sequences that simulate the drift behavior of glove-based mocap devices. The augmentations are designed to be strong and varied so that the augmented distribution covers, or at least overlaps with, realistic distribution shifts observed in real mocap data; the full strategy is provided in Appendix A.1.

Each training iteration uses one augmented sequence as a session. From this sequence we sample K paired calibration examples $\mathcal{S} = \{(x_i^n, x_i^c)\}_{i=1}^K$ for the HyperNet, and a disjoint set of query frames $\{(x_q^n, x_q^c)\}$ for supervision. The HyperNet generates session-specific LoRA matrices $\Delta W = h_\psi(\mathcal{S})$, which is injected into the frozen target network to form $g_{\phi+\Delta W}$. Each query pose is then passed through this calibrated estimator and MANO forward kinematics to obtain $\hat{x}_q^c = f_\psi(x_q^n; \mathcal{S})$, and $\psi$ is updated by minimizing the mean squared error between $\hat{x}_q^c$ and the ground-truth calibrated joints $x_q^c$. Gradients flow only through $\psi$, and LoRA injection is differentiable in $\Delta W$.

## 5 Experiment

Our experiments are designed to evaluate YOCO as a deployable calibration system rather than only as a pose-estimation model. We organize the evaluation around five research questions that together cover calibration accuracy, the source of YOCO's advantage, sample efficiency in K, transfer to real low-cost glove data, and downstream impact on dexterous teleoperation: (1) Does YOCO reduce hand-pose error on biased mocap streams relative to uncalibrated input? (2) How does YOCO compare with optimization- and fine-tuning-based calibration baselines in accuracy and adaptation cost, and in particular, does the feed-forward HyperNet outperform gradient-based LoRA at matched adapter capacity? (3) How does performance scale with the number of paired calibration poses K? (4) Does YOCO transfer from synthetic drift to real low-cost glove data across users and glove fits? (5) Does improved hand-state calibration translate into measurable gains on downstream dexterous teleoperation tasks?

### 5.1 Experiment Setup

Training datasets. We pretrain the target network $g_{\phi}$ on the training subset of InterHand2.6M, which provides a large pool of clean MANO-annotated hand poses captured by a high-precision multi-view mocap system. We then train the hypernetwork $h_{\psi}$ on a synthetic dataset constructed by applying the structured drift augmentations described in Appendix A.1 to InterHand2.6M, while keeping $\phi$ frozen. Each augmented sequence simulates one user/device/session with an internally consistent bias pattern, providing the diverse session-level distribution that the HyperNet must learn to summarize from only K paired examples.

---

Test datasets. We evaluate YOCO on three settings of increasing realism. On the synthetic side, we apply the same augmentation family to the InterHand2.6M validation set, instantiating 20 augmented sets with distinct drift patterns; all reported synthetic numbers are aggregated over these 20 sets to assess robustness across drift types rather than performance on a single bias. For real low-cost-glove evaluation, we collect a paired dataset using an Apple Vision Pro (AVP) as a high-end reference tracker and a low-cost tensile glove worn simultaneously, treating the AVP 3D keypoints as the calibration target. Split and sequence statistics are provided in Appendix C.3. For downstream evaluation, we deploy YOCO in a live teleoperation system on four physical manipulation tasks.

Evaluation setup. Unless otherwise specified, all methods use $K = 4$ paired calibration poses. For offline evaluations on InterHand2.6M and the real glove dataset, $\mathcal{S}$ is sampled from the first 200 frames of each validation sequence with a fixed random seed; the remaining frames form the evaluation split. The real-glove targets are synchronized AVP reference poses. For deployment, we record a fixed library of four 21-joint target poses with AVP once offline: index pinch, fully open hand, claw/half-close, and four straight fingers with the thumb near the palm (Appendix C.3). The targets use the mocap stream's joint order and coordinate convention. At the start of each teleoperation trial, the operator performs these prompts wearing only the low-cost glove; each raw observation is paired with its stored library target. No AVP is needed at runtime. All methods share the same shape-handling protocol and do not calibrate MANO shape $\beta$; details are in Appendices C.1 and C.2.

#### 5.1.1 Teleoperation Protocol

We evaluate four physical tasks with a right robot hand: removing a fixed bottle's cap by twisting; lifting an earbud with a thumb–middle-finger pinch and holding it for 3 s; lifting a small glue bottle with the thumb, index, and middle fingers and holding it for 3 s; and unlocking a tissue-box lid by stabilizing it with four fingers and pulling with the thumb. These outcomes define task success. Two operators participate: one performs bottle-cap and earbud tasks, and the other performs glue-bottle and tissue-box tasks. The same operator tests all three calibration conditions for each assigned task. Operators can directly see the robot and thus can compensate for control error, but are blinded to the calibration method's identity. For each method–task pair, they receive 5 min of familiarization and then perform 10 evaluation trials; method order is randomized within each task. All conditions use the same trial budget and the same geometric-retargeting method [^22]. A separate retargeting model is trained for each calibration condition using the same configuration. We report success counts and completion time, with the failure penalty for overall time specified in Table 3.

Evaluation metrics. We report a comprehensive set of pose, geometry, and deployment metrics. MPJPE and PA-MPJPE measure 3D joint accuracy before and after Procrustes alignment: MPJPE preserves residual offset, scale, and orientation bias, while PA-MPJPE isolates articulated pose accuracy. AUCJ is the normalized area under the joint PCK curve, computed over 101 uniformly spaced thresholds from 0 to 0.05 m. Tip-PA is the PA-aligned mean L2 error over the five MANO fingertips and is reported separately because fingertip geometry dominates dexterous-manipulation outcomes. Pinch error is the L2 distance between two fingertips on frames where the ground-truth pinch distance is below 10 mm; this isolates the regime that contact-rich tasks depend on. Calibration time is the test-time conditioning or adaptation cost incurred before applying the calibrated estimator: for YOCO, this is one forward pass of $h_{\psi}$; for fine-tuning baselines, it is gradient-based adaptation on the paired calibration set. For downstream evaluation, we report task success count over 10 trials and average completion time.

### 5.2 Experiment Results

RQ1: YOCO substantially reduces hand-pose error on biased mocap streams. On the augmented InterHand2.6M evaluation set, YOCO reduces PA-MPJPE from 16.55 mm (no calibration) to 10.45 mm, a 37% relative reduction, with corresponding gains across MPJPE, Tip-PA, AUCJ, and pinch error (Table 1). The improvement is largest on fingertip-sensitive metrics (Tip-PA drops from 37.15 to 23.29 mm), which matches the design intuition that calibration matters most where

---

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'>Method</td><td style='text-align: center; word-wrap: break-word;'>MPJPE $\downarrow$ (mm)</td><td style='text-align: center; word-wrap: break-word;'>PA-MPJPE $\downarrow$ (mm)</td><td style='text-align: center; word-wrap: break-word;'>Tip-PA $\downarrow$ (mm)</td><td style='text-align: center; word-wrap: break-word;'>AUCJ $\uparrow$</td><td style='text-align: center; word-wrap: break-word;'>Pinch $\downarrow$ (mm)</td><td style='text-align: center; word-wrap: break-word;'>Calib. s $\downarrow$ (s)</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>No calibration</td><td style='text-align: center; word-wrap: break-word;'>21.52</td><td style='text-align: center; word-wrap: break-word;'>16.55</td><td style='text-align: center; word-wrap: break-word;'>37.15</td><td style='text-align: center; word-wrap: break-word;'>0.611</td><td style='text-align: center; word-wrap: break-word;'>48.49</td><td style='text-align: center; word-wrap: break-word;'>0.00</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Ridge</td><td style='text-align: center; word-wrap: break-word;'>17.62</td><td style='text-align: center; word-wrap: break-word;'>13.00</td><td style='text-align: center; word-wrap: break-word;'>28.28</td><td style='text-align: center; word-wrap: break-word;'>0.676</td><td style='text-align: center; word-wrap: break-word;'>45.35</td><td style='text-align: center; word-wrap: break-word;'>1.6</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Last-layer-Finetune</td><td style='text-align: center; word-wrap: break-word;'>16.62</td><td style='text-align: center; word-wrap: break-word;'>12.01</td><td style='text-align: center; word-wrap: break-word;'>25.92</td><td style='text-align: center; word-wrap: break-word;'>0.692</td><td style='text-align: center; word-wrap: break-word;'>39.18</td><td style='text-align: center; word-wrap: break-word;'>45.6</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Full-Finetune</td><td style='text-align: center; word-wrap: break-word;'>18.11</td><td style='text-align: center; word-wrap: break-word;'>12.77</td><td style='text-align: center; word-wrap: break-word;'>29.21</td><td style='text-align: center; word-wrap: break-word;'>0.668</td><td style='text-align: center; word-wrap: break-word;'>37.11</td><td style='text-align: center; word-wrap: break-word;'>95.4</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>LoRA-Finetune</td><td style='text-align: center; word-wrap: break-word;'>16.38</td><td style='text-align: center; word-wrap: break-word;'>11.67</td><td style='text-align: center; word-wrap: break-word;'>25.33</td><td style='text-align: center; word-wrap: break-word;'>0.694</td><td style='text-align: center; word-wrap: break-word;'>37.28</td><td style='text-align: center; word-wrap: break-word;'>89.4</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>YOCO (ours)</td><td style='text-align: center; word-wrap: break-word;'>15.64</td><td style='text-align: center; word-wrap: break-word;'>10.45</td><td style='text-align: center; word-wrap: break-word;'>23.29</td><td style='text-align: center; word-wrap: break-word;'>0.702</td><td style='text-align: center; word-wrap: break-word;'>34.90</td><td style='text-align: center; word-wrap: break-word;'>11.2</td></tr></table>

<div style="text-align: center;">Table 1: Calibration results on the augmented InterHand2.6M evaluation setting. Calib. s reports the total adaptation time in seconds for whole validation sets on one H100.</div>

small biases reshape contact geometry. These results confirm that the HyperNet successfully infers session-level drift from only K = 4 paired examples and that the predicted LoRA updates $\Delta W$ correct unseen queries from the same session.

RQ2: YOCO outperforms optimization- and fine-tuning-based baselines, and the advantage persists at matched LoRA capacity. We compare against five baselines spanning the natural spectrum of calibration strategies: (1) No Calibration, which uses raw mocap output; (2) Ridge, an optimization-based linear mapping fitted in MANO $\theta$ space from the paired examples; (3) Lastlayer-Finetune, which gradient-tunes only the final MLP of the MANO $\theta$ decoder on the paired examples; (4) Full-Finetune, which gradient-tunes the entire $g_{\phi}$; and (5) LoRA-Finetune, which injects the same LoRA adapter set as YOCO into $g_{\phi}$ and tunes only those adapters by gradient descent on the paired examples. LoRA-Finetune matches the adapter footprint, but its cold start does not control for YOCO's learned cross-session prior. Appendix B adds a meta-learned LoRA initialization, a set-conditioned residual predictor, and a published side-tuning baseline to examine that distinction.

As shown in Table 1, YOCO achieves the best score on every precision metric while being substantially faster than every gradient-based baseline: 11.2 s of aggregate H100 conditioning for YOCO versus 45.6–95.4 s for the fine-tuning baselines. The accuracy advantage over LoRA-Finetune at matched adapter capacity (PA-MPJPE 10.45 vs. 11.67 mm, Tip-PA 23.29 vs. 25.33 mm) shows that the gains are not attributable to the LoRA adapters alone: feed-forward weight generation by $h_{\psi}$ outperforms per-session gradient descent over the same parameter space. Ridge regression is the only baseline competitive on adaptation time, but its accuracy degrades severely. The deployment-latency gap is even larger on CPU, where YOCO requires 13.16 s versus 34.47–223.25 s for gradient fine-tuning baselines (Appendix C.4); this is the regime that matters for in-the-field teleoperation rigs without a dedicated GPU, which signifies the applicability of YOCO.

---

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'>Method</td><td style='text-align: center; word-wrap: break-word;'>MPJPE $\downarrow$ (mm)</td><td style='text-align: center; word-wrap: break-word;'>PA-MPJPE $\downarrow$ (mm)</td><td style='text-align: center; word-wrap: break-word;'>Tip-PA $\downarrow$ (mm)</td><td style='text-align: center; word-wrap: break-word;'>AUCJ $\uparrow$</td><td style='text-align: center; word-wrap: break-word;'>Pinch $\downarrow$ (mm)</td><td style='text-align: center; word-wrap: break-word;'>Calib $\downarrow$ (s)</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>No calibration</td><td style='text-align: center; word-wrap: break-word;'>20.49</td><td style='text-align: center; word-wrap: break-word;'>14.63</td><td style='text-align: center; word-wrap: break-word;'>36.22</td><td style='text-align: center; word-wrap: break-word;'>0.610</td><td style='text-align: center; word-wrap: break-word;'>51.39</td><td style='text-align: center; word-wrap: break-word;'>0.00</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Ridge</td><td style='text-align: center; word-wrap: break-word;'>24.71</td><td style='text-align: center; word-wrap: break-word;'>19.21</td><td style='text-align: center; word-wrap: break-word;'>45.50</td><td style='text-align: center; word-wrap: break-word;'>0.573</td><td style='text-align: center; word-wrap: break-word;'>57.54</td><td style='text-align: center; word-wrap: break-word;'>1.49</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Last-layer-Finetune</td><td style='text-align: center; word-wrap: break-word;'>18.86</td><td style='text-align: center; word-wrap: break-word;'>14.67</td><td style='text-align: center; word-wrap: break-word;'>33.88</td><td style='text-align: center; word-wrap: break-word;'>0.643</td><td style='text-align: center; word-wrap: break-word;'>43.63</td><td style='text-align: center; word-wrap: break-word;'>5.04</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Full-Finetune</td><td style='text-align: center; word-wrap: break-word;'>20.00</td><td style='text-align: center; word-wrap: break-word;'>15.73</td><td style='text-align: center; word-wrap: break-word;'>36.67</td><td style='text-align: center; word-wrap: break-word;'>0.630</td><td style='text-align: center; word-wrap: break-word;'>49.89</td><td style='text-align: center; word-wrap: break-word;'>13.04</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>LoRA-Finetune</td><td style='text-align: center; word-wrap: break-word;'>18.72</td><td style='text-align: center; word-wrap: break-word;'>13.94</td><td style='text-align: center; word-wrap: break-word;'>33.23</td><td style='text-align: center; word-wrap: break-word;'>0.648</td><td style='text-align: center; word-wrap: break-word;'>54.01</td><td style='text-align: center; word-wrap: break-word;'>14.57</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>YOCO (ours)</td><td style='text-align: center; word-wrap: break-word;'>17.75</td><td style='text-align: center; word-wrap: break-word;'>13.14</td><td style='text-align: center; word-wrap: break-word;'>31.73</td><td style='text-align: center; word-wrap: break-word;'>0.657</td><td style='text-align: center; word-wrap: break-word;'>42.79</td><td style='text-align: center; word-wrap: break-word;'>6.73</td></tr></table>

<div style="text-align: center;">Table 2: Real-world mocap calibration results. Metrics are averaged over 3 seeds. Calib. s reports the total conditioning or adaptation time in seconds for whole datasets on one H100. Corresponding user-level std is provided in Appendix C.5.</div>

RQ3: YOCO provides strong one-pass few-shot calibration. Figure 3 sweeps K from 1 to 16. Among methods that calibrate without further test-time optimization, YOCO achieves the lowest PA-MPJPE at $K \in \{1, 2, 4\}$: collecting four paired poses takes a few seconds, whereas collecting sixteen is closer to a minute. At K = 1, YOCO reaches 10.94 mm, below every gradient-based baseline at K = 4. At K = 8, gradient-based baselines close the gap, and at K = 16, Full-Finetune slightly surpasses the one-pass YOCO. The new YOCO-FT curve shows the effect of adding per-session fine-tuning after YOCO initialization; it improves accuracy further but gives up gradient-free adaptation. This pattern is consistent with an optimization or capacity limit at larger K, but does not isolate which factor dominates. Appendix C.7.

RQ4: YOCO transfers to real low-cost glove data and generalizes across users. We evaluate on the paired AVP/tensile-glove dataset described in Appendix C.3, which exposes YOCO to the actual user- and fit-dependent distortions of a low-cost glove rather than synthetic drift. As shown in Table 2, YOCO attains the best score on every primary metric (MPJPE, PA-MPJPE, Tip-PA, AUCJ, pinch error) while requiring 6.73 s of conditioning versus 14.57 s for LoRA-Finetune and 13.04 s for Full-Finetune. The per-user standard deviations in Appendix C.5 indicate that this advantage is consistent across the four users in the dataset rather than driven by any single operator. Notably, Ridge regression degrades below the nocalibration baseline on real data, which suggests that the bias structure of a real glove is

![Image](images/831f4fdd845736d3b09efee45dcc62e6caad4818ef6006d0faeec41344064b2a.jpg)

<div style="text-align: center;">Figure 3: PA-MPJPE versus calibration-pair count, including YOCO followed by test-time fine-tuning (YOCO-FT). The dotted line marks no calibration.</div>

The numerical multi-method sweep is reported in

![Image](images/de60b2c9428ff23c530731d80c9f8d5b4b284eb2a6b80e99bfbe3dd14368b0e8.jpg)

<div style="text-align: center;">Figure 4: Goal states for the downstream dexterous manipulation tasks: twisting and opening a bottle cap, pinch-grasping earbuds, three-finger glue grasping, and tissue-box opening.</div>

non-linear enough that linear correction in MANO $\theta$ space is actively harmful; this is the regime where a learned, structured calibrator earns its keep.

RQ5: The YOCO teleoperation pipeline improves task success. The tasks, operator assignment, visual feedback, familiarization, retargeting, and trial order are described in Section 5.1.1.

---

<div style="text-align: center;">As shown in Table 3, YOCO achieves an overall success rate of 90%, compared with 65% for LoRA-Finetune and 35% for the uncalibrated baseline, and the lowest overall completion time across the four tasks. The largest per-task gains appear on contact-sensitive tasks: Earbud Pinch Grasping rises from 1/10 (LoRA-Finetune) to 8/10 (YOCO), and Tissue Box Opening from 8/10 to 10/10. The earbud task uses a middle-finger pinch, whereas the calibration poses include an index-finger pinch but no middle-finger pinch, suggesting transfer to an unobserved fingertip configuration. Because retargeting models are trained separately for each condition, the task results characterize the complete teleoperation pipelines rather than isolating calibration as the sole cause.</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td rowspan="3">Method</td><td colspan="2">Bottle Cap</td><td colspan="2">Earbud Pinch</td><td rowspan="2" colspan="2">Glue Three-Finger Grasping</td><td colspan="2">Tissue Box</td><td rowspan="2" colspan="2">Overall</td></tr><tr><td colspan="2">Opening</td><td colspan="2">Grasping</td><td colspan="2">Opening</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Success $\uparrow$ Time (s) $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>Success $\uparrow$ Time (s) $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>Success $\uparrow$ Time (s) $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>Success $\uparrow$ Time (s) $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>Success $\uparrow$ Time (s) $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>Success $\uparrow$ Time (s) $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>Success $\uparrow$ Time (s) $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>Success $\uparrow$ Time (s) $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>Success Rate $\uparrow$</td><td style='text-align: center; word-wrap: break-word;'>Time (s) $\downarrow$</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>No calibration</td><td style='text-align: center; word-wrap: break-word;'>9/10</td><td style='text-align: center; word-wrap: break-word;'>35.6</td><td style='text-align: center; word-wrap: break-word;'>0/10</td><td style='text-align: center; word-wrap: break-word;'>-</td><td style='text-align: center; word-wrap: break-word;'>5/10</td><td style='text-align: center; word-wrap: break-word;'>7.0</td><td style='text-align: center; word-wrap: break-word;'>0/10</td><td style='text-align: center; word-wrap: break-word;'>-</td><td style='text-align: center; word-wrap: break-word;'>35%</td><td style='text-align: center; word-wrap: break-word;'>24.4</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>LoRA-Finetune</td><td style='text-align: center; word-wrap: break-word;'>9/10</td><td style='text-align: center; word-wrap: break-word;'>34.1</td><td style='text-align: center; word-wrap: break-word;'>1/10</td><td style='text-align: center; word-wrap: break-word;'>6.0</td><td style='text-align: center; word-wrap: break-word;'>8/10</td><td style='text-align: center; word-wrap: break-word;'>6.6</td><td style='text-align: center; word-wrap: break-word;'>8/10</td><td style='text-align: center; word-wrap: break-word;'>13.3</td><td style='text-align: center; word-wrap: break-word;'>65%</td><td style='text-align: center; word-wrap: break-word;'>17.9</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>YOCO (ours)</td><td style='text-align: center; word-wrap: break-word;'>10/10</td><td style='text-align: center; word-wrap: break-word;'>27.3</td><td style='text-align: center; word-wrap: break-word;'>8/10</td><td style='text-align: center; word-wrap: break-word;'>4.2</td><td style='text-align: center; word-wrap: break-word;'>8/10</td><td style='text-align: center; word-wrap: break-word;'>8.9</td><td style='text-align: center; word-wrap: break-word;'>10/10</td><td style='text-align: center; word-wrap: break-word;'>17.4</td><td style='text-align: center; word-wrap: break-word;'>90%</td><td style='text-align: center; word-wrap: break-word;'>14.9</td></tr></table>

<div style="text-align: center;">Table 3: Downstream dexterous teleoperation results. We evaluate 4 tasks and count each task success over 10 trials and average completion time over successful trials. Failed trials are capped at one minute and excluded from task-specific average completion time, while each failed trial is penalized by assigning it the longest observed trial time in the overall average time. More stats are provided in Appendix C.8.</div>

## 6 Conclusion

YOCO reframes mocap calibration as feed-forward weight generation: a hypernetwork $h_{\psi}$ reads a handful of paired poses and emits LoRA updates $\Delta W$ that specialize a frozen MANO estimator $g_{\phi}$, avoiding per-session optimization of the estimator. The downstream results show promising teleoperation gains for the evaluated calibration-and-retargeting pipelines.

Limitations YOCO focuses on pose calibration and leaves MANO shape estimation to a separate stage, which integrates cleanly with existing shape-fitting pipelines. The session-stationary bias assumption is consistent with sensor manufacturing tolerances of most commercial gloves, but high-frequency drift on less stable hardware would require re-calibration or a temporal extension of the HyperNet. The augmentation pipeline lets us scale training across diverse drift patterns at low cost, though covering high-frequency drift remains an open direction tied to sensor characterization.

## Acknowledgments

This work is supported by the Early Career Scheme of the Research Grants Council (RGC) grant #27207224, the HKU-100 Award, a donation from the Musketeers Foundation, and in part by the JC STEM Lab of Autonomous Intelligent Systems funded by The Hong Kong Jockey Club Charities Trust and computing resource provided by HKU-Shanghai Intelligent Computing Research Center. We thank TranscEngram for supporting data collection and providing hardware used in this project. This project is also supported and funded by Shenzhen Loop Area Institute under project FPF10120250005.

---

## References

[^1]: Y. Wang and M. Neff. Data-driven glove calibration for hand motion capture. In Proceedings of the 12th ACM SIGGRAPH/Eurographics Symposium on Computer Animation, pages 15–24, 2013. doi:10.1145/2485895.2485901.

[^2]: Y. Li, J. Zhang, W. Xu, T. Tang, and C. Lu. FSGlove: An inertial-based hand tracking system with shape-aware calibration. arXiv preprint arXiv:2509.21242, 2025. URL <https://arxiv.org/abs/2509.21242>.

[^3]: C. Zuo, J. Huang, X. Jiang, Y. Yao, X. Shi, R. Cao, X. Yi, F. Xu, S. Guo, and Y. Qin. Transformer IMU calibrator: Dynamic on-body IMU calibration for inertial motion capture. ACM Transactions on Graphics, 2025. URL <https://arxiv.org/abs/2506.10580>.

[^4]: N. Gosala, F. Wang, Z. Cui, H. Liang, O. Glauser, S. Wu, and O. Sorkine-Hornung. Self-calibrated multi-sensor wearable for hand tracking and modeling. IEEE Transactions on Visualization and Computer Graphics, 29(3):1769–1784, 2023. doi:10.1109/TVCG.2021.3131230.

[^5]: L. Dipietro, A. M. Sabatini, and P. Dario. A survey of glove-based systems and their applications. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 38(4):461–482, 2008. doi:10.1109/TSMCC.2008.923862.

[^6]: H. Zhang, S. Dalla Gasperina, S. N. Yousaf, T. Tsuboi, T. Narita, and A. D. Deshpande. Human-exoskeleton kinematic calibration to improve hand tracking for dexterous teleoperation. arXiv preprint arXiv:2507.23592, 2025. URL <https://arxiv.org/abs/2507.23592>.

[^7]: J. Romero, D. Tzionas, and M. J. Black. Embodied hands: Modeling and capturing hands and bodies together. ACM Transactions on Graphics, 36(6):245:1–245:17, 2017. doi:10.1145/3130800.3130883.

[^8]: C. Zimmermann and T. Brox. Learning to estimate 3d hand pose from single RGB images. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 4903–4911, 2017.

[^9]: C. Zimmermann, D. Ceylan, J. Yang, B. Russell, M. Argus, and T. Brox. FreiHAND: A dataset for markerless capture of hand pose and shape from single RGB images. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 813–822, 2019. doi:10.1109/ICCV.2019.00090.

[^10]: S. Hampali, M. Rad, M. Oberweger, and V. Lepetit. HOnnotate: A method for 3d annotation of hand and object poses. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3196–3206, 2020.

[^11]: F. Zhang, V. Bazarevsky, A. Vakunov, A. Tkachenka, G. Sung, C.-L. Chang, and M. Grundmann. MediaPipe hands: On-device real-time hand tracking. arXiv preprint arXiv:2006.10214, 2020. URL <https://arxiv.org/abs/2006.10214>.

[^12]: G. Moon, S.-I. Yu, H. Wen, T. Shiratori, and K. M. Lee. InterHand2.6M: A dataset and baseline for 3d interacting hand pose estimation from a single RGB image. In Computer Vision – ECCV 2020, pages 548–564. Springer, 2020. doi:10.1007/978-3-030-58565-5_33. URL <https://arxiv.org/abs/2008.09309>.

[^13]: G. Pavlakos, D. Shan, I. Radosavovic, A. Kanazawa, D. Fouhey, and J. Malik. Reconstructing hands in 3d with transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9826–9836, June 2024.

---

[^14]: A. Handa, K. Van Wyk, W. Yang, J. Liang, Y.-W. Chao, Q. Wan, S. Birchfield, N. Ratliff, and D. Fox. DexPilot: Vision-based teleoperation of dexterous robotic hand-arm system. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), pages 9164–9170, 2020. doi:10.1109/ICRA40945.2020.9197124. URL <https://arxiv.org/abs/1910.03135>.

[^15]: Y. Qin, W. Yang, B. Huang, K. Van Wyk, H. Su, X. Wang, Y.-W. Chao, and D. Fox. AnyTeleop: A general vision-based dexterous robot arm-hand teleoperation system. In Proceedings of Robotics: Science and Systems, Daegu, Republic of Korea, 2023. doi:10.15607/RSS.2023.XIX.015. URL <https://roboticsproceedings.org/rss19/p015.html>.

[^16]: T. Lin, Y. Zhang, Q. Li, H. Qi, B. Yi, S. Levine, and J. Malik. Learning visuotactile skills with two multifingered hands, 2024. URL <https://arxiv.org/abs/2404.16823>.

[^17]: X. Cheng, J. Li, S. Yang, G. Yang, and X. Wang. Open-television: Teleoperation with immersive active visual feedback. In Proceedings of The 8th Conference on Robot Learning, volume 270 of Proceedings of Machine Learning Research, pages 2729–2749. PMLR, 2025. URL <https://proceedings.mlr.press/v270/cheng25b.html>.

[^18]: A. Iyer, Z. Peng, Y. Dai, I. Guzey, S. Haldar, S. Chintala, and L. Pinto. OPEN TEACH: A versatile teleoperation system for robotic manipulation. arXiv preprint arXiv:2403.07870, 2024. URL <https://arxiv.org/abs/2403.07870>.

[^19]: A. Sivakumar, K. Shaw, and D. Pathak. Robotic telekinesis: Learning a robotic hand imitator by watching humans on YouTube. In Proceedings of Robotics: Science and Systems, New York City, NY, USA, 2022. doi:10.15607/RSS.2022.XVIII.023. URL <https://roboticsproceedings.org/rss18/p023.html>.

[^20]: S. Park, S. Lee, M. Choi, J. Lee, J. Kim, J. Kim, and H. Joo. Learning to transfer human hand skills for robot manipulations. arXiv preprint arXiv:2501.04169, 2025. URL <https://arxiv.org/abs/2501.04169>.

[^21]: K. Li, P. Li, T. Liu, Y. Li, and S. Huang. ManipTrans: Efficient dexterous bimanual manipulation transfer via residual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6991–7003, 2025. doi:10.1109/CVPR52734.2025.00656. URL <https://openaccess.thecvf.com/content/CVPR2025/html/Li_ManipTrans_Efficient_Dexterous_Bimanual_Manipulation_Transfer_via_Residual_Learning_CVPR_2025_paper.html>.

[^22]: Z.-H. Yin, C. Wang, L. Pineda, K. Bodduluri, T. Wu, P. Abbeel, and M. Mukadam. Geometric retargeting: A principled, ultrafast neural hand retargeting algorithm, 2025. URL <https://arxiv.org/abs/2503.07541>.

[^23]: Z. Wan, Z. Bi, Z. Zhou, H. Ren, Y. Zeng, Y. Li, L. Qi, X. Yang, M.-H. Yang, and H. Cheng. RAPID hand: A robust, affordable, perception-integrated, dexterous manipulation platform for generalist robot autonomy. arXiv preprint arXiv:2506.07490, 2025. URL <https://arxiv.org/abs/2506.07490>.

[^24]: H.-S. Fang, B. Romero, Y. Xie, A. Hu, B.-R. Huang, J. Alvarez, M. Kim, G. Margolis, K. Anbarasu, M. Tomizuka, E. Adelson, and P. Agrawal. DEXOP: A device for robotic transfer of dexterous human manipulation. arXiv preprint arXiv:2509.04441, 2025. URL <https://arxiv.org/abs/2509.04441>.

[^25]: K. Junge and J. Hughes. ADAPT-Teleop: Robotic hand with human matched embodiment enables dexterous teleoperated manipulation. npj Robotics, 3:31, 2025. doi:10.1038/s44182-025-00034-3.

[^26]: D. Ha, A. M. Dai, and Q. V. Le. HyperNetworks. In International Conference on Learning Representations (ICLR), 2017. URL <https://dblp.org/rec/conf/iclr/HaDL17>.

---

[^27]: J. von Oswald, C. Henning, B. F. Grewe, and J. Sacramento. Continual learning with hypernetworks. In International Conference on Learning Representations (ICLR), 2020. URL <https://openreview.net/forum?id=SJgwNerKvB>.

[^28]: R. Karimi Mahabadi, S. Ruder, M. Dehghani, and J. Henderson. Parameter-efficient multi-task fine-tuning for transformers via shared hypernetworks. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 565–576, 2021. URL <https://aclanthology.org/2021.acl-long.47/>.

[^29]: P. Zhou, W. Yao, Q. Luo, X. Zhou, and Y. Yang. Hyper-goalnet: Goal-conditioned manipulation policy learning with hypernetworks. arXiv preprint arXiv:2512.00085, 2025.

[^30]: H. Ren, L. Sun, X. Wang, P. Zhou, Z. Wu, S. Dong, D. Zou, Y. Zheng, and Y. Yang. HyPoGen: Optimization-biased hypernetworks for generalizable policy generation. In International Conference on Learning Representations (ICLR), 2025. URL <https://openreview.net/forum?id=CJWMXqAnAy>.

[^31]: L. Sun, J. Wu, F. Chen, R. Liu, and Y. Yang. HyperTASR: Hypernetwork-driven task-aware scene representations for robust manipulation, 2025. URL <https://arxiv.org/abs/2508.18802>.

[^32]: R. Karimi Mahabadi, J. Henderson, and S. Ruder. Compacter: Efficient low-rank hypercomplex adapter layers. In Advances in Neural Information Processing Systems (NeurIPS), volume 34, pages 1022–1035, 2021. URL <https://proceedings.neurips.cc/paper/2021/hash/081be9fdff07f3bc808f935906ef70c0-Abstract.html>.

[^33]: E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR), 2022. URL <https://openreview.net/forum?id=nZeVKeeFYf9>.

[^34]: N. Ruiz, Y. Li, V. Jampani, W. Wei, T. Hou, Y. Pritch, N. Wadhwa, M. Rubinstein, and K. Aberman. HyperDreamBooth: Hypernetworks for fast personalization of text-to-image models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6527–6536, 2024. URL <https://openaccess.thecvf.com/content/CVPR2024/html/Ruiz_HyperDreamBooth_HyperNetworks_for_Fast_Personalization_of_Text-to-Image_Models_CVPR_2024_paper.html>.

[^35]: J. Su, Y. Lu, S. Pan, A. Murtadha, B. Wen, and Y. Liu. Roformer: Enhanced transformer with rotary position embedding, 2023. URL <https://arxiv.org/abs/2104.09864>.

[^36]: W. Peebles and S. Xie. Scalable diffusion models with transformers, 2023. URL <https://arxiv.org/abs/2212.09748>.

[^37]: MANUS. MANUS Metagloves Pro, 2026. URL <https://www.manus-meta.com/products/metagloves-pro>.

[^38]: Rokoko. Rokoko Smartgloves, 2026. URL <https://www.rokoko.com/products/smartgloves>.

[^39]: StretchSense. StretchSense XR Gloves, 2026. URL <https://stretchsense.com/>.

[^40]: CyberGlove Systems LLC. CyberGlove II, 2026. URL <https://www.cyberglovesystems.com/cyberglove-ii>.

[^41]: CyberOrigin. CyberOrigin Data Glove and Motion Capture, 2025. URL <https://cyberorigin.ai/trending/post?id=5&title=posts%2F942b8c447cbb6586.md>.

---

## A Method Details

### A.1 Augmentation Strategy for HyperNet training

To train HyperLoRA under realistic calibration errors in glove-based mocap devices, we generate synthetic biases in the MANO pose space rather than directly perturbing 3D keypoints. Let $\theta_{d,t} \in R^{45}$ denote the MANO hand pose of frame t in sequence d, reshaped as $\theta_{d,t}^{f,j} \in R^{3}$ for finger f and joint level j. For each sequence, we sample a bias style $\phi_d$ once and keep it fixed across all frames. This reflects the observation that glove or hand-tracking miscalibration is usually session-consistent, rather than independent frame-wise noise.

The sampled bias combines several physically plausible error modes: per-finger zero offsets, per-joint residual offsets, global/finger/joint gain errors, adjacent-finger crosstalk, and proximal-to-distal coupling within each finger. Concretely, the biased pose is generated as

 $$ \tilde{\theta}_{d,t}^{f,j}=s_{d}^{f,j}\theta_{d,t}^{f,j}+a_{f,j}\sum_{g\in\mathcal{N}(f)}C_{d}^{f,g}s_{d}^{g,j}\theta_{d,t}^{g,j}+\sum_{k<j}U_{d}^{j,k}s_{d}^{f,k}\theta_{d,t}^{f,k}+o_{d}^{f,j}, $$

where $s_{d}^{f,j}$ is a clipped multiplicative gain, $C_{d}$ models leakage between adjacent fingers, $U_{d}$ models tendon-like proximal-to-distal coupling, $o_{d}^{f,j}$ is the additive offset, and $a_{f,j}$ attenuates perturbations from proximal to distal joints. These parameters are sampled from zero-mean Gaussian families with Bernoulli activation probabilities, so each sequence receives a different but internally consistent bias type.

We then apply MANO forward kinematics to both the calibrated and noisy poses:

 $$ x_{d,t}^{c}=\Pi_{\mathrm{M A N O}}(\theta_{d,t}),\qquad x_{d,t}^{n}=\Pi_{\mathrm{M A N O}}(\tilde{\theta}_{d,t}), $$

and store the resulting pair $(x_{d,t}^{n}, x_{d,t}^{c})$ offline. During training, the calibration model receives the noisy joints $x_{d,t}^{n}$ as input and predicts the calibrated MANO pose. YOCO is conditioned by a small set of calibration examples

 $$ \mathcal{P}_{d}=\left\{\left(x_{d,\tau_{i}}^{n},x_{d,\tau_{i}}^{c}\right)\right\}_{i=1}^{K}, $$

which are encoded by the calibration mapping to generate sequence-specific LoRA parameters. This augmentation exposes the model to a diverse family of structured, kinematically valid calibration errors while preserving the temporal consistency expected from real glove or hand-tracking biases.

### A.2 LoRA Adaptation Details

All layers in the MANO estimator are equipped with LoRA adapters. Unless otherwise specified, both YOCO and the LoRA-Finetune baseline use LoRA rank r = 8 and scaling factor $\alpha = 8$. YOCO generates the low-rank matrices for this full adapter set from the HyperNet, while the LoRA-Finetune baseline optimizes the same adapter set by gradient descent on the paired calibration examples. This keeps the adaptation capacity fixed across the feed-forward and optimization-based LoRA comparisons.

---

## B Learned-Prior Controls and Additional Baselines

Cold-start LoRA-Finetune uses the same adapter footprint as YOCO, but lacks its learned prior across calibration sessions. To separate this effect from conditioning on the specific K pairs, Meta-LoRA learns a session-independent LoRA initialization using the same target estimator and adapter footprint; Meta-LoRA+FT adapts that initialization by gradient descent on the pairs. Set-Cond. Residual instead predicts MANO-pose residuals from the paired set without test-time gradients. YOCO+LoRA-FT fine-tunes the adapters generated by YOCO. We also adapt the published side-tuning calibration approach of Gosala et al. [^4]; its original sensor inputs and supervision differ, so this is a related-method comparison rather than a matched-device calibration protocol.

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td rowspan="2">Method</td><td colspan="2">Real glove (AVP reference)</td><td colspan="2">Augmented InterHand2.6M</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>PA-MPJPE $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>AUCJ $\uparrow$</td><td style='text-align: center; word-wrap: break-word;'>PA-MPJPE $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>AUCJ $\uparrow$</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>LoRA-Finetune</td><td style='text-align: center; word-wrap: break-word;'>13.94</td><td style='text-align: center; word-wrap: break-word;'>0.648</td><td style='text-align: center; word-wrap: break-word;'>11.67</td><td style='text-align: center; word-wrap: break-word;'>0.694</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Set-Cond. Residual</td><td style='text-align: center; word-wrap: break-word;'>13.74</td><td style='text-align: center; word-wrap: break-word;'>0.645</td><td style='text-align: center; word-wrap: break-word;'>12.66</td><td style='text-align: center; word-wrap: break-word;'>0.664</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Side-Tuning [^4]</td><td style='text-align: center; word-wrap: break-word;'>13.48</td><td style='text-align: center; word-wrap: break-word;'>0.641</td><td style='text-align: center; word-wrap: break-word;'>15.35</td><td style='text-align: center; word-wrap: break-word;'>0.618</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Meta-LoRA</td><td style='text-align: center; word-wrap: break-word;'>13.51</td><td style='text-align: center; word-wrap: break-word;'>0.633</td><td style='text-align: center; word-wrap: break-word;'>13.94</td><td style='text-align: center; word-wrap: break-word;'>0.639</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Meta-LoRA+FT</td><td style='text-align: center; word-wrap: break-word;'>14.17</td><td style='text-align: center; word-wrap: break-word;'>0.664</td><td style='text-align: center; word-wrap: break-word;'>11.41</td><td style='text-align: center; word-wrap: break-word;'>0.702</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>YOCO</td><td style='text-align: center; word-wrap: break-word;'>13.14</td><td style='text-align: center; word-wrap: break-word;'>0.657</td><td style='text-align: center; word-wrap: break-word;'>10.45</td><td style='text-align: center; word-wrap: break-word;'>0.702</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>YOCO+LoRA-FT</td><td style='text-align: center; word-wrap: break-word;'>13.28</td><td style='text-align: center; word-wrap: break-word;'>0.665</td><td style='text-align: center; word-wrap: break-word;'>9.99</td><td style='text-align: center; word-wrap: break-word;'>0.707</td></tr></table>

<div style="text-align: center;">Table 4: Learned-prior controls and additional calibration baselines at K = 4. PA-MPJPE is in millimeters; AUCJ is the normalized joint-PCK area.</div>

Table 4 shows that, compared with the session-independent Meta-LoRA initialization, YOCO reduces PA-MPJPE from 13.51 to 13.14 mm on the real glove data and from 13.94 to 10.45 mm on augmented InterHand2.6M. The set-conditioned residual control and Side-Tuning provide additional checks on the adaptation mechanism and prior work. Further LoRA fine-tuning improves some scores, at the cost of test-time optimization.

## C Experiment Details

### C.1 MANO Beta Handling

For the evaluation on InterHand2.6M dataset, we use the ground truth shape parameter for all methods. For the evaluation on real-world dataset, we use optimization-based shape estimation method to estimate the shape parameter for all methods. For the evaluation on downstream dexterous manipulation tasks, we use a MANO shape parameter $\beta$ pre-computed when collecting the AVP mocap poses for the preset-pose library, and hold this same $\beta$ fixed for all methods.

### C.2 Target Pose Selection

For all offline evaluation, we report the average performance over 3 random seeds for the selection of paired calibration poses. In addition, the calibration poses are selected only from the first 200 frames of the validation sequence, and evaluation is performed on the remaining frames to avoid data leakage and evaluate the generalization ability of YOCO and the baselines to uncalibrated poses.

### C.3 Real-World Data Protocol

Our real-world glove/mocap dataset contains four users. For each frame, the noisy observation $x^n$ is the low-cost glove-derived hand keypoint pose, while the calibrated target $x^c$ is the synchronized 3D hand keypoint pose from Apple Vision Pro (AVP). AVP is used here as a high-end reference source for offline evaluation data, not as a device required during deployment-time calibration. Each user contributes one sequence for evaluation. Each sequence is approximately 1500 frames long at 30 FPS.

---

Index pinch. Thumb tip and index fingertip touch; remaining fingers stay naturally open.

![Image](images/554e37c98e273681cb814e0ac49bd02f101adda548aaa49fabd39ec22ff41147.jpg)

Fully open. Palm open with all five fingers extended.

Claw / half close.

Fingers are partially flexed in a mid-closure hand shape.

Four fingers straight,

thumb near palm.

Index, middle, ring, and little fingers are straight while the thumb is adducted close to the palm.

<div style="text-align: center;">Figure 5: Preset pose library used to obtain deployable calibration targets. The four prompts are index pinch, fully open, claw / half close, and four fingers straight with the thumb near the palm.</div>

### C.4 CPU Deployment Calibration Time

The Calibration time in Table 1 and 2 are computed using H100 GPU. However, in practical teleoperation deployments, calibration may need to run on a CPU-only host. Table 5 reports the average CPU adaptation time required before inference. We use the HyperNet forwarding time for YOCO and the optimization/solving time for gradient-based or regression baselines.

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'>Method</td><td style='text-align: center; word-wrap: break-word;'>Timing Source</td><td style='text-align: center; word-wrap: break-word;'>CPU calibration time (s)</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>YOCO (ours)</td><td style='text-align: center; word-wrap: break-word;'>HyperNet Forward</td><td style='text-align: center; word-wrap: break-word;'>13.16</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Ridge</td><td style='text-align: center; word-wrap: break-word;'>Closed Form Solver</td><td style='text-align: center; word-wrap: break-word;'>0.37</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Last-layer-Finetune</td><td style='text-align: center; word-wrap: break-word;'>Gradient Descent</td><td style='text-align: center; word-wrap: break-word;'>34.47</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>LoRA-Finetune</td><td style='text-align: center; word-wrap: break-word;'>Gradient Descent</td><td style='text-align: center; word-wrap: break-word;'>59.58</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Full-Finetune</td><td style='text-align: center; word-wrap: break-word;'>Gradient Descent</td><td style='text-align: center; word-wrap: break-word;'>223.25</td></tr></table>

<div style="text-align: center;">Table 5: Average CPU calibration time required before real deployment.</div>

### C.5 Real-World Per-User Variability

Table 6 reports the per-user standard deviation of the real-world mocap calibration results. This table corresponds to Table 2 to show how much each method varies across users.

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'>Method</td><td style='text-align: center; word-wrap: break-word;'>MPJPE $\downarrow$ (mm)</td><td style='text-align: center; word-wrap: break-word;'>PA-MPJPE $\downarrow$ (mm)</td><td style='text-align: center; word-wrap: break-word;'>Tip-PA $\downarrow$ (mm)</td><td style='text-align: center; word-wrap: break-word;'>AUCJ $\uparrow$</td><td style='text-align: center; word-wrap: break-word;'>Pinch $\downarrow$ (mm)</td><td style='text-align: center; word-wrap: break-word;'>Calib. s $\downarrow$ (s)</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>No calibration</td><td style='text-align: center; word-wrap: break-word;'>1.14</td><td style='text-align: center; word-wrap: break-word;'>0.76</td><td style='text-align: center; word-wrap: break-word;'>3.18</td><td style='text-align: center; word-wrap: break-word;'>0.021</td><td style='text-align: center; word-wrap: break-word;'>7.01</td><td style='text-align: center; word-wrap: break-word;'>0.00</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Ridge</td><td style='text-align: center; word-wrap: break-word;'>4.32</td><td style='text-align: center; word-wrap: break-word;'>2.95</td><td style='text-align: center; word-wrap: break-word;'>9.12</td><td style='text-align: center; word-wrap: break-word;'>0.053</td><td style='text-align: center; word-wrap: break-word;'>12.53</td><td style='text-align: center; word-wrap: break-word;'>0.68</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Last-layer-Finetune</td><td style='text-align: center; word-wrap: break-word;'>0.87</td><td style='text-align: center; word-wrap: break-word;'>1.01</td><td style='text-align: center; word-wrap: break-word;'>2.47</td><td style='text-align: center; word-wrap: break-word;'>0.017</td><td style='text-align: center; word-wrap: break-word;'>6.23</td><td style='text-align: center; word-wrap: break-word;'>0.55</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Full-Finetune</td><td style='text-align: center; word-wrap: break-word;'>1.80</td><td style='text-align: center; word-wrap: break-word;'>1.52</td><td style='text-align: center; word-wrap: break-word;'>4.79</td><td style='text-align: center; word-wrap: break-word;'>0.030</td><td style='text-align: center; word-wrap: break-word;'>7.65</td><td style='text-align: center; word-wrap: break-word;'>0.75</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>LoRA-Finetune</td><td style='text-align: center; word-wrap: break-word;'>1.22</td><td style='text-align: center; word-wrap: break-word;'>1.06</td><td style='text-align: center; word-wrap: break-word;'>3.73</td><td style='text-align: center; word-wrap: break-word;'>0.023</td><td style='text-align: center; word-wrap: break-word;'>9.50</td><td style='text-align: center; word-wrap: break-word;'>0.80</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>YOCO (ours)</td><td style='text-align: center; word-wrap: break-word;'>1.34</td><td style='text-align: center; word-wrap: break-word;'>1.07</td><td style='text-align: center; word-wrap: break-word;'>3.84</td><td style='text-align: center; word-wrap: break-word;'>0.029</td><td style='text-align: center; word-wrap: break-word;'>6.44</td><td style='text-align: center; word-wrap: break-word;'>3.30</td></tr></table>

<div style="text-align: center;">Table 6: Per-user standard deviation of the real-world mocap calibration results.</div>

### C.6 Mocap Device Selection

Several glove-based hand motion capture systems are currently available, including professional EMF-tracking gloves such as MANUS Metagloves Pro, IMU/EMF-based products such as Rokoko

---

Smartgloves, stretch-sensor gloves such as StretchSense gloves, and bend-sensor data gloves such as CyberGlove II [^37] [^38] [^39] [^40]. In our real-world experiment, we use OpenCyberGlove, a low-cost tensile-sensor-based glove [^41]. This choice is motivated by both practicality and evaluation purpose: compared with high-end commercial mocap gloves, OpenCyberGlove provides an affordable platform for real data collection, while its tensile/strain sensor readings are naturally affected by user-specific hand geometry, glove fit, hand size, and hand shape. Therefore, it provides a representative and challenging test case for our calibration setting, where the goal is precisely to compensate for person- and fit-dependent distortions and recover accurate hand motion from limited paired calibration data.

### C.7 Additional K-Sweep Results

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'>Method</td><td style='text-align: center; word-wrap: break-word;'>K</td><td style='text-align: center; word-wrap: break-word;'>MPJPE $\downarrow$ (mm)</td><td style='text-align: center; word-wrap: break-word;'>PA-MPJPE $\downarrow$ (mm)</td><td style='text-align: center; word-wrap: break-word;'>Tip-PA $\downarrow$ (mm)</td><td style='text-align: center; word-wrap: break-word;'>AUCJ $\uparrow$</td><td style='text-align: center; word-wrap: break-word;'>Pinch L2 $\downarrow$ (mm)</td></tr><tr><td rowspan="5">YOCO (ours)</td><td style='text-align: center; word-wrap: break-word;'>1</td><td style='text-align: center; word-wrap: break-word;'>16.07</td><td style='text-align: center; word-wrap: break-word;'>10.94</td><td style='text-align: center; word-wrap: break-word;'>24.40</td><td style='text-align: center; word-wrap: break-word;'>0.694</td><td style='text-align: center; word-wrap: break-word;'>38.04</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>2</td><td style='text-align: center; word-wrap: break-word;'>16.10</td><td style='text-align: center; word-wrap: break-word;'>10.86</td><td style='text-align: center; word-wrap: break-word;'>24.30</td><td style='text-align: center; word-wrap: break-word;'>0.694</td><td style='text-align: center; word-wrap: break-word;'>36.66</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>4</td><td style='text-align: center; word-wrap: break-word;'>15.64</td><td style='text-align: center; word-wrap: break-word;'>10.45</td><td style='text-align: center; word-wrap: break-word;'>23.29</td><td style='text-align: center; word-wrap: break-word;'>0.702</td><td style='text-align: center; word-wrap: break-word;'>34.90</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>8</td><td style='text-align: center; word-wrap: break-word;'>15.21</td><td style='text-align: center; word-wrap: break-word;'>10.19</td><td style='text-align: center; word-wrap: break-word;'>22.26</td><td style='text-align: center; word-wrap: break-word;'>0.709</td><td style='text-align: center; word-wrap: break-word;'>33.87</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>16</td><td style='text-align: center; word-wrap: break-word;'>15.10</td><td style='text-align: center; word-wrap: break-word;'>10.14</td><td style='text-align: center; word-wrap: break-word;'>21.94</td><td style='text-align: center; word-wrap: break-word;'>0.712</td><td style='text-align: center; word-wrap: break-word;'>33.70</td></tr><tr><td rowspan="5">LoRA FT</td><td style='text-align: center; word-wrap: break-word;'>1</td><td style='text-align: center; word-wrap: break-word;'>18.96</td><td style='text-align: center; word-wrap: break-word;'>14.07</td><td style='text-align: center; word-wrap: break-word;'>31.33</td><td style='text-align: center; word-wrap: break-word;'>0.651</td><td style='text-align: center; word-wrap: break-word;'>43.44</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>2</td><td style='text-align: center; word-wrap: break-word;'>17.80</td><td style='text-align: center; word-wrap: break-word;'>13.00</td><td style='text-align: center; word-wrap: break-word;'>28.54</td><td style='text-align: center; word-wrap: break-word;'>0.671</td><td style='text-align: center; word-wrap: break-word;'>42.01</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>4</td><td style='text-align: center; word-wrap: break-word;'>16.38</td><td style='text-align: center; word-wrap: break-word;'>11.67</td><td style='text-align: center; word-wrap: break-word;'>25.33</td><td style='text-align: center; word-wrap: break-word;'>0.694</td><td style='text-align: center; word-wrap: break-word;'>37.28</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>8</td><td style='text-align: center; word-wrap: break-word;'>15.44</td><td style='text-align: center; word-wrap: break-word;'>10.65</td><td style='text-align: center; word-wrap: break-word;'>22.92</td><td style='text-align: center; word-wrap: break-word;'>0.709</td><td style='text-align: center; word-wrap: break-word;'>33.42</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>16</td><td style='text-align: center; word-wrap: break-word;'>15.12</td><td style='text-align: center; word-wrap: break-word;'>10.29</td><td style='text-align: center; word-wrap: break-word;'>22.06</td><td style='text-align: center; word-wrap: break-word;'>0.714</td><td style='text-align: center; word-wrap: break-word;'>32.89</td></tr><tr><td rowspan="5">Last-layer FT</td><td style='text-align: center; word-wrap: break-word;'>1</td><td style='text-align: center; word-wrap: break-word;'>19.10</td><td style='text-align: center; word-wrap: break-word;'>14.22</td><td style='text-align: center; word-wrap: break-word;'>31.64</td><td style='text-align: center; word-wrap: break-word;'>0.649</td><td style='text-align: center; word-wrap: break-word;'>43.54</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>2</td><td style='text-align: center; word-wrap: break-word;'>18.04</td><td style='text-align: center; word-wrap: break-word;'>13.34</td><td style='text-align: center; word-wrap: break-word;'>29.20</td><td style='text-align: center; word-wrap: break-word;'>0.669</td><td style='text-align: center; word-wrap: break-word;'>42.57</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>4</td><td style='text-align: center; word-wrap: break-word;'>16.62</td><td style='text-align: center; word-wrap: break-word;'>12.01</td><td style='text-align: center; word-wrap: break-word;'>25.92</td><td style='text-align: center; word-wrap: break-word;'>0.692</td><td style='text-align: center; word-wrap: break-word;'>39.18</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>8</td><td style='text-align: center; word-wrap: break-word;'>15.54</td><td style='text-align: center; word-wrap: break-word;'>10.84</td><td style='text-align: center; word-wrap: break-word;'>23.16</td><td style='text-align: center; word-wrap: break-word;'>0.709</td><td style='text-align: center; word-wrap: break-word;'>35.72</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>16</td><td style='text-align: center; word-wrap: break-word;'>15.18</td><td style='text-align: center; word-wrap: break-word;'>10.50</td><td style='text-align: center; word-wrap: break-word;'>22.26</td><td style='text-align: center; word-wrap: break-word;'>0.715</td><td style='text-align: center; word-wrap: break-word;'>35.10</td></tr><tr><td rowspan="5">Ridge</td><td style='text-align: center; word-wrap: break-word;'>1</td><td style='text-align: center; word-wrap: break-word;'>20.18</td><td style='text-align: center; word-wrap: break-word;'>15.24</td><td style='text-align: center; word-wrap: break-word;'>33.87</td><td style='text-align: center; word-wrap: break-word;'>0.633</td><td style='text-align: center; word-wrap: break-word;'>47.44</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>2</td><td style='text-align: center; word-wrap: break-word;'>19.10</td><td style='text-align: center; word-wrap: break-word;'>14.35</td><td style='text-align: center; word-wrap: break-word;'>31.66</td><td style='text-align: center; word-wrap: break-word;'>0.654</td><td style='text-align: center; word-wrap: break-word;'>48.25</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>4</td><td style='text-align: center; word-wrap: break-word;'>17.62</td><td style='text-align: center; word-wrap: break-word;'>13.00</td><td style='text-align: center; word-wrap: break-word;'>28.28</td><td style='text-align: center; word-wrap: break-word;'>0.676</td><td style='text-align: center; word-wrap: break-word;'>45.35</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>8</td><td style='text-align: center; word-wrap: break-word;'>16.76</td><td style='text-align: center; word-wrap: break-word;'>12.13</td><td style='text-align: center; word-wrap: break-word;'>26.44</td><td style='text-align: center; word-wrap: break-word;'>0.688</td><td style='text-align: center; word-wrap: break-word;'>46.35</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>16</td><td style='text-align: center; word-wrap: break-word;'>16.07</td><td style='text-align: center; word-wrap: break-word;'>11.47</td><td style='text-align: center; word-wrap: break-word;'>24.87</td><td style='text-align: center; word-wrap: break-word;'>0.698</td><td style='text-align: center; word-wrap: break-word;'>45.03</td></tr><tr><td rowspan="5">Full FT</td><td style='text-align: center; word-wrap: break-word;'>1</td><td style='text-align: center; word-wrap: break-word;'>25.62</td><td style='text-align: center; word-wrap: break-word;'>20.20</td><td style='text-align: center; word-wrap: break-word;'>48.08</td><td style='text-align: center; word-wrap: break-word;'>0.570</td><td style='text-align: center; word-wrap: break-word;'>48.71</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>2</td><td style='text-align: center; word-wrap: break-word;'>22.79</td><td style='text-align: center; word-wrap: break-word;'>17.26</td><td style='text-align: center; word-wrap: break-word;'>40.76</td><td style='text-align: center; word-wrap: break-word;'>0.607</td><td style='text-align: center; word-wrap: break-word;'>45.11</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>4</td><td style='text-align: center; word-wrap: break-word;'>18.11</td><td style='text-align: center; word-wrap: break-word;'>12.77</td><td style='text-align: center; word-wrap: break-word;'>29.21</td><td style='text-align: center; word-wrap: break-word;'>0.668</td><td style='text-align: center; word-wrap: break-word;'>37.11</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>8</td><td style='text-align: center; word-wrap: break-word;'>15.39</td><td style='text-align: center; word-wrap: break-word;'>10.17</td><td style='text-align: center; word-wrap: break-word;'>22.39</td><td style='text-align: center; word-wrap: break-word;'>0.706</td><td style='text-align: center; word-wrap: break-word;'>32.60</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>16</td><td style='text-align: center; word-wrap: break-word;'>14.55</td><td style='text-align: center; word-wrap: break-word;'>9.47</td><td style='text-align: center; word-wrap: break-word;'>20.39</td><td style='text-align: center; word-wrap: break-word;'>0.722</td><td style='text-align: center; word-wrap: break-word;'>30.88</td></tr></table>

<div style="text-align: center;">Table 7: Detailed K-sweep results on the augmented InterHand2.6M evaluation setting. All rows use the overall split aggregated over 3 seeds and 20 augmented units.</div>

---

### C.8 Additional Teleoperation Statistics

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'>Method</td><td style='text-align: center; word-wrap: break-word;'>Bottle Cap Opening Avg. Time (s) $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>Earbud Pinch Grasping Avg. Time (s) $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>3-Finger Glue Grasping Avg. Time (s) $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>Tissue Box Opening Avg. Time (s) $\downarrow$</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>No calibration</td><td style='text-align: center; word-wrap: break-word;'>37.9</td><td style='text-align: center; word-wrap: break-word;'>7.0</td><td style='text-align: center; word-wrap: break-word;'>10.5</td><td style='text-align: center; word-wrap: break-word;'>42.0</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>LoRA-Finetune</td><td style='text-align: center; word-wrap: break-word;'>37.5</td><td style='text-align: center; word-wrap: break-word;'>6.9</td><td style='text-align: center; word-wrap: break-word;'>8.1</td><td style='text-align: center; word-wrap: break-word;'>19.0</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>YOCO (ours)</td><td style='text-align: center; word-wrap: break-word;'>27.3</td><td style='text-align: center; word-wrap: break-word;'>4.8</td><td style='text-align: center; word-wrap: break-word;'>9.9</td><td style='text-align: center; word-wrap: break-word;'>17.4</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Longest trial</td><td style='text-align: center; word-wrap: break-word;'>59.0</td><td style='text-align: center; word-wrap: break-word;'>7.0</td><td style='text-align: center; word-wrap: break-word;'>14.0</td><td style='text-align: center; word-wrap: break-word;'>42.0</td></tr></table>

Table 8: Detailed downstream teleoperation completion-time statistics. Values are average completion time in seconds over 10 trials; lower is better. The longest-trial row reports the longest observed trial time for each task.

### C.9 Device and Compute Platform

All training runs and timing measurements are performed on a single-GPU server. Table 9 summarizes the hardware and operating system used for model training and timing experiments.

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'>Component</td><td style='text-align: center; word-wrap: break-word;'>Value</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>CPU</td><td style='text-align: center; word-wrap: break-word;'>2 $\times$ Intel Xeon Platinum 8468V</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>RAM</td><td style='text-align: center; word-wrap: break-word;'>2.0 TiB</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>GPU</td><td style='text-align: center; word-wrap: break-word;'>1 $\times$ NVIDIA H100 80 GB HBM3</td></tr></table>

<div style="text-align: center;">Table 9: Device and compute platform used for training and timing experiments.</div>

### C.10 Training Hyperparameters

Table 10 reports the main optimization, HyperNet, LoRA, and paired-pose preference-extractor hyperparameters used in our training runs.

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'>Setting</td><td style='text-align: center; word-wrap: break-word;'>Value</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Seed</td><td style='text-align: center; word-wrap: break-word;'>42</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Optimizer</td><td style='text-align: center; word-wrap: break-word;'>Adam</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Total steps</td><td style='text-align: center; word-wrap: break-word;'>10,000</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Batch size</td><td style='text-align: center; word-wrap: break-word;'>1024</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Peak learning Rate</td><td style='text-align: center; word-wrap: break-word;'>3e-4</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Final Learning Rate</td><td style='text-align: center; word-wrap: break-word;'>1e-6</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Warmup Steps</td><td style='text-align: center; word-wrap: break-word;'>500</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Gradient clipping</td><td style='text-align: center; word-wrap: break-word;'>1.0</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>LoRA rank / alpha</td><td style='text-align: center; word-wrap: break-word;'>Rank 8; alpha 8.0</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>LoRA dropout</td><td style='text-align: center; word-wrap: break-word;'>0.1</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Calibration Mapper Transformer Depth</td><td style='text-align: center; word-wrap: break-word;'>6</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Calibration Mapper Transformer Number of Heads</td><td style='text-align: center; word-wrap: break-word;'>8</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Decoder Flow Steps</td><td style='text-align: center; word-wrap: break-word;'>8</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Decoder DIT Number of Layers</td><td style='text-align: center; word-wrap: break-word;'>4</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Decoder DIT Number of Heads</td><td style='text-align: center; word-wrap: break-word;'>4</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Hidden Dimension</td><td style='text-align: center; word-wrap: break-word;'>256</td></tr></table>

<div style="text-align: center;">Table 10: Training hyperparameters used by YOCO.</div>