# TransHands：将人体姿态编码器重新用于手部姿态编码

Milo Piccioli$^{1}$、Gianluca Amprimo$^{1}$$\textcircled{D}$, Claudia Ferraris$^{2}$$\textcircled{D}$和 Gabriella Olmo^{1}$$\textcircled{D}$

 $ ^{1} $ Politecnico di Torino, Torino, Italy <s359009@studenti.polito.it>, {gianluca.amprimo, gabriella.olmo}@polito.it

 $ ^{2} $ Consiglio Nazionale delle Ricerche, Torino, Italy <claudia.ferraris@cnr.it>

Abstract. Lifting 3D hand poses from 2D monocular representations remains challenging due to the limited availability of large-scale, diverse 3D-annotated hand datasets, in contrast to the abundance of human body motion data. We address this limitation by transferring motion representations learned from large body pose corpora to the hand domain. We introduce TransHands, a backbone-agnostic transfer learning framework that enables pre-trained human motion encoders to be effectively adapted for 3D hand pose estimation from 2D pose inputs. Rather than training hand-specific biomechanical models from scratch, TransHands combines a two-stage training and fine-tuning strategy with a lightweight hand-specific input adaptation module that aligns hand kinematics with the representation space learned for full-body motion. We evaluate TransHands across four state-of-the-art motion modeling architectures, including transformer-based, graph-based, and frequency-domain models. Results demonstrate that motion priors learned from body pose data transfer consistently across architectures, yielding consistent accuracy gains, strong cross-domain generalization, particularly in challenging egocentric settings, and applicability for downstream tasks in real-world contexts.

## 1 Introduction

Estimating 3D hand pose from 2D joint coordinates (2D-to-3D lifting) is a fundamental challenge in computer vision. While significant progress has been made in full-body pose estimation, benefiting from large-scale motion capture datasets such as Human3.6M [11] and AMASS [20], hand pose estimation remains constrained by the limited availability of diverse, large-scale annotated hand datasets. This data scarcity poses a critical bottleneck: hands exhibit complex biomechanical constraints and high-frequency articulations that are difficult to capture and annotate at scale. Existing hand-specific datasets [8,43] are orders of magnitude smaller than their body pose counterparts, limiting the capacity of data-driven models to learn robust kinematic priors. Meanwhile, state-of-the-art (SOTA) human motion encoders [38,40,42] have demonstrated remarkable

---

1. Body Motion Pre-training (Source Domain)

![Image](images/e15c0d909bda0a85872dc0e2dd9f1456dc04d0bfc21a78a8a7583066fd1dca90.jpg)

<div style="text-align: center;">Fig. 1: Framework Overview. TransHands bridges the domain gap between body and hand motion through a backbone-agnostic transfer learning strategy. We leverage (I) Body Motion Pre-training to capture spatio-temporal priors from heterogeneous body datasets. These priors are repurposed for hand pose estimation via (II) TransHands Framework, using a learnable Topological Adapter in input to align monocular hand sequences with the pre-trained body manifold.</div>

proficiency in modeling spatio-temporal dynamics from abundant body motion corpora.

We pose a natural question: Can motion priors learned from large-scale human body datasets be transferred to model hand biomechanical knowledge? Despite the topological differences, both domains share fundamental biomechanical principles such as temporal smoothness, joint angle constraints, and hierarchical structure. Recent work has demonstrated the effectiveness of pre-training, transfer learning and cross-domain adaptation in pose estimation [13,41,42], suggesting that learned motion representations can generalize across modalities; however, a systematic study on the transfer of body motion priors to the hand domain is still lacking.

To this end, we introduce TransHands, a backbone-agnostic framework that enables pre-trained human motion encoders to be effectively repurposed for 3D hand pose estimation from 2D inputs. Our key insight is that the high-level temporal dynamics and biomechanical knowledge encoded in body pose models can be preserved and adapted to hand kinematics through learnable topological alignment rather than architectural redesign. TransHands achieves this by encapsulating frozen body encoders within lightweight adaptation modules that bridge the domain gap while preserving the original kinematic reasoning capabilities. As illustrated in Figure 1, our approach comprises three main components: (1) a Topological Adapter in input that aligns hand joint hierarchies with the latent space of body-centric encoders, (2) a pre-trained motion backbone on large body datasets that extracts spatio-temporal features, and (3) a task-specific decoder that reconstructs 3D hand coordinates.

We validate TransHands across four diverse motion encoding architectures spanning transformer-based [11,38], frequency-domain [40], and graph-based [37] models. Our experiments demonstrate that: (1) motion priors transfer consis-

---

tently across architectural families, yielding consistent accuracy gains, (2) the learned representations exhibit strong cross-domain generalization, particularly in challenging egocentric settings, and (3) the framework requires minimal modification to existing backbones, enabling rapid integration of future motion modeling advances.

In summary, our primary contributions are threefold:

– To the best of our knowledge, we present the first systematic study on transferring body motion priors to hand pose estimation across multiple SOTA architectures.

– We propose a modular framework that decouples topological adaptation from temporal modeling, enabling backbone-agnostic transfer.

We demonstrate that biomechanical knowledge learned from body datasets provides strong inductive biases for modeling hand kinematics, opening new directions for data-efficient hand pose estimation and downstream applications in gesture recognition.

## 2 Related Works

Early learning-based approaches formulated 2D-to-3D pose uplifting as a direct regression, showing that even simple feed-forward models can exploit geometric consistency and temporal smoothness when trained with paired 2D–3D data [7,21]. The emergence of large-scale human motion datasets enabled a shift toward data-driven learning of kinematic priors. Datasets such as AMASS [20] facilitated the pretraining of spatio-temporal motion encoders that implicitly capture biomechanical properties, including joint coordination and long-range temporal dependencies. Building on this paradigm, recent work has demonstrated the effectiveness of powerful sequence models for 2D-to-3D lifting, including spatio-temporal transformers [38,42], frequency-domain formulations [40], and graph-based or hybrid architectures leveraging skeletal structure as an inductive bias [23,37]. These approaches encode biomechanical knowledge implicitly through learned motion dynamics rather than explicit constraints. In parallel, biomechanical plausibility has been addressed through model-based lifting approaches that fit parametric human models to monocular observations. Statistical body models such as SMPL [16] and GHUM [36] provide strong pose and shape priors and are widely used for monocular 3D reconstruction and tracking, including real-time systems such as BlazePose GHUM Holistic [10]. For hands, analogous constraints are commonly imposed through the MANO statistical model [29], to ensure anatomical validity of estimated coordinates [8,43]. While effective, these model-based methods rely on carefully engineered optimization pipelines and explicit skeletal modeling, limiting flexibility and cross-domain adaptability. Despite the success of data-driven motion encoders for full-body lifting, hand pose estimation remains comparatively data-limited. Transfer learning offers a natural alternative: pre-training on large body datasets followed by downstream adaptation has shown consistent benefits in tasks related to pose estimation [31,42]. To date, these approaches have largely assumed a

---

![Image](images/be07ec6d7c6a6a994ddb3a351369760ca4e7af7123f7c6f29081d0ec7e35a2fe.jpg)

<div style="text-align: center;">Fig. 2: Overview of the TransHands Framework. To bridge the domain gap between body and hand topology, we employ a learnable Topological Adapter ( $ \mathcal{A}_{in} $) in input that aligns hand joints with the latent space of a pre-trained Motion Encoder Backbone. The architecture adopts a Partial Unfreezing strategy: initially, the entire backbone is frozen ( $ \star $) to leverage pre-trained human motion priors (Stage 1); subsequently, the final block is unfrozen ( $ \blacklozenge $) to specialize the features for hand-specific kinematics (Stage 2). The Output Decoder then maps these refined features to the final 3D hand coordinates.</div>

shared skeletal topology, leaving open whether motion encoders pre-trained for body pose lifting can be systematically repurposed for hand pose estimation despite substantial topological mismatch. Our work addresses this gap by reframing hand pose estimation as a biomechanical lifting problem and adapting 2D-to-3D body motion encoders to hand kinematics via lightweight, modular alignment modules.

## 3 Method

### 3.1 Problem Formulation

We formulate 3D hand pose estimation as a sequence-to-sequence lifting task. Specifically, we focus on the geometric lifting process, assuming the availability of 2D skeletal coordinates either from off-the-shelf detectors or ground-truth projections. This allows us to isolate the motion transfer capabilities from the noise inherent in direct 2D-to-3D estimation from RGB sources. Given a sequence of input coordinates $ X \in \mathbb{R}^{T \times J_H \times C_{in}} $, where $ T $ denotes the sequence length, $ J_H = 21 $ represents the hand joints according to the MANO [29] topology, and $ C_{in} \in \{2, 3\} $ is the backbone-dependent input dimensionality, our objective is to regress the trajectory of 3D coordinates $ Y \in \mathbb{R}^{T \times J_H \times 3} $.

A fundamental challenge in leveraging body motion priors for this task is the topological mismatch: SOTA body encoders are typically optimized on skeletal full pose structures, rendering them incompatible with hand geometry. To bridge this domain gap without retraining the kinematic engine from scratch, we propose TransHands, the modular framework illustrated in Figure 2. It encapsulates a frozen body encoder $ \mathcal{E}_{\text{body}} $ within learnable adaptation layers. The mapping function is formalized as:

 $$。$\hat{Y}=\mathcal{D}_{\mathrm{out}}\left(\mathcal{P}\left(\mathcal{E}_{\mathrm{body}}\left(\mathcal{A}_{\mathrm{in}}(X)\right)\right)\right)$ $$

---

Here, $ \mathcal{A}_{\text{in}} $ projects the hand topology into the body's latent kinematic space, $ \mathcal{P} $ aligns the extracted features via projection, and $ \mathcal{D}_{\text{out}} $ decodes the processed representations back to the target $ \mathbb{R}^3 $ hand manifold. This formulation effectively decouples the learning of temporal dynamics, which is handled by the pre-trained $ \mathcal{E}_{\text{body}} $, from the spatial adaptation required for the new topology.

### 3.2 Topological Adapter

The primary function of the Topological Adapter ( $ A_{in} $) is to align hand kinematics with the representation space learned for full-body motion, resolving the dimensional and topological discrepancies between hand joints and body-centric pre-trained encoders. While SOTA body pose estimators are typically optimized on skeletal structures with $ J_{B} = 17 $ joints, as in Human3.6M [11] topology or $ J_{B} = 18 $, as in ST-GCN [37], standard hand models such as MANO define a distinct topology consisting of $ J_{H} = 21 $ joints. Direct application of the frozen body encoder $ \mathcal{E}_{\text{body}} $ is therefore infeasible due to this mismatch ( $ J_{H} \neq J_{B} $). Moreover, the two graphs share no common joints except for the wrist, which, however, plays significantly different roles in the two kinematic graphs (i.e., the root node in the hand graph vs. a distal node in the body pose graph).

To address this, we formulate the topological transformation as a continuous dynamic process using a Neural Ordinary Differential Equation (Neural ODE) [5]. Given the flattened input coordinates $ x \in \mathbb{R}^{J_H \cdot C_{in}} $, an initial encoder maps them into a latent state $ z(0) \in \mathbb{R}^{d_{emb}} $, where $ d_{emb} $ is the latent dimension. The continuous transformation of the hidden state is then defined by the ODE:

 $$ $\frac{dz(t)}{dt}=f_{\theta}(z(t),t)$ $$

where $ f_\theta $ is parameterized by a continuous dynamics neural network equipped with time-embeddings. We solve the initial value problem using a fixed-step fourth-order Runge-Kutta (RK4) integrator over the integration interval $ t \in [0,1] $ to obtain the terminal state $ z(1) $. Finally, a decoder maps $ z(1) $ into the compatibility vector $ z_{body} \in \mathbb{R}^{J_B \cdot C_{enc}} $ required by the backbone. This continuous flow formulation allows for a highly non-linear topological alignment without suffering from the rigidity of discrete linear projections.

Critically, $ A_{in} $ acts as a dynamic interface that abstracts the underlying backbone requirements from the raw input data:

– Channel Alignment: The adapter adjusts the channel dimensionality $ C_{enc} $ to match the specific requirement of the selected backbone. For instance, it synthesizes a confidence channel (C = 3) for MotionBERT [42], while preserving raw coordinates (C = 2) for architectures like MixSTE [38] and PoseFormerV2 [40].

- Latent Manifold Alignment: By optimizing the ODE trajectory, the adapter smoothly projects the hand topology onto the body's latent kinematic manifold, effectively reusing pre-trained priors without structural modifications to the frozen $ \mathcal{E}_{\text{body}} $.

---

### 3.3 Motion Encoder Backbones

The core component of our framework is the kinematic encoder $ \mathcal{E}_{\text{body}} $, which processes the adapted sequence $ Z_{\text{body}} = \mathcal{A}_{in}(X) $ to extract spatio-temporal features. To enforce the transfer of high-level motion priors (e.g., velocity continuity, biomechanical constraints), we adopt a Partial Unfreezing strategy. Instead of fine-tuning, we strategically enable gradients only for the deepest layers of the backbone, together with its embedding and output layers. This design choice ensures that the model reuses learned body dynamics while allowing the final representation stages to adapt to hand-specific micro-articulations. We validate the versatility of our approach by integrating four distinct SOTA architectures serving as Motion Encoder Backbone.

Spatio-Temporal Transformers. We integrate MotionBERT [42], utilizing its Dual-Stream Spatio-Temporal Transformer (DSTformer) to capture long-range dependencies. Additionally, we integrate MixSTE [38], a Seq2Seq model that decouples spatial and temporal attention, allowing us to evaluate the benefits of fine-grained frame-to-frame modeling for hand micro-articulations.

Frequency-Domain Transformers. To address long-sequence efficiency, we incorporate PoseFormerV2 [40], which operates in the frequency domain via Discrete Cosine Transform (DCT). This architecture is particularly relevant for analyzing the spectral properties of rapid hand gestures.

Graph-Based Architectures. To benchmark against purely convolutional approaches, we integrate the standalone implementation of ST-GCN [37], demonstrating the framework's adaptability to non-transformer backbones. For this backbone, the transfer is only partial: the temporal convolutions are initialized from the Kinetics checkpoint, whereas the graph branch (spatial convolutions, edge importance, and adaptive adjacency) is trained from scratch.

### 3.4 Latent Space Alignment

Since different backbones produce feature maps with varying distributions and sequential dynamics (covariate shift), we introduce a projection module $ \mathcal{P} $ to align the encoder output $ f_{enc} $ with the decoding task. Instead of a standard linear bottleneck or attention-based transformer, we utilize a multi-scale retention mechanism [32]. Formally:

 $$ $f_{proj}=W_{p}\cdot \text{LayerNorm}(f_{enc})$ $$

Subsequently, the sequence is processed by a stack of retention layers with residual connections:

 $$ $\tilde{f}=f_{proj}+\text{Retention}(\text{LayerNorm}(f_{proj})),\quad f_{ret}=\tilde{f}+\text{FFN}(\tilde{f})$ $$

where the retention operator applies learned exponential temporal decay to the position weights. The projection head standardizes heterogeneous encoder outputs to a common 256-dimensional latent before the decoding stage, preventing feature distortion during fine-tuning [12].

---

### 3.5 Task-Specific Output Decoder

The reconstruction of the target 3D pose is performed by the module $ \mathcal{D}_{out} $, a task-specific regressor. We design this decoder as a deep Multilayer Perceptron (MLP) utilizing Layer Normalization [1] for stable training across sequences. The architecture follows:

 $$ $h_{1}=\text{ReLU}(\text{LN}(W_{1}f_{ret}+b_{1})),\quad h_{2}=\text{ReLU}(\text{LN}(W_{2}h_{1}+b_{2})),$ $$

 $$ $\hat{Y}=\text{Reshape}_{J_{H}\times3}(W_{3}h_{2}+b_{3})$

其中 $W_1 \in \mathbb{R}^{512 \times 256}$、$W_2 \in \mathbb{R}^{512 \times 512}$ 和 $W_3 \in \mathbb{R}^{63 \times 512}$（对应 $J_H = 21$ 个关节）。引入层归一化并结合顺序隐藏层扩展，对于避免解码狭窄瓶颈时出现训练不稳定性至关重要。该模块的深度（三个线性变换，映射 $256 \to 512 \to 512 \to 63$）对于建模非线性逆运动学而言必不可少，只有这样才能从压缩的潜空间表示中恢复出复杂的手指关节动作。

## 4 实验

## 4.1 实验设置

数据集与数据说明。为了验证TransHands的有效性，我们采用Re:InterHand [^24] 作为主要基准数据集，这是一套大规模的重光照3D交互手部数据集，包含基于MANO的高保真标注，采集于受控的多视角摄影棚环境。为了评估其在不同场景下的鲁棒性，我们使用了AssemblyHands [^26]（ Egocentric视角）和GigaHands [^9]（双手活动场景）。这些数据集提供了高保真的2D真值以及对应的3D标注，使我们能够在不考虑2D检测误差的情况下，单独评估TransHands作为2D到3D的提升模块的性能。这能够分离出身体运动编码器的语义运动先验是否可迁移到准确的3D手部姿态提升中，且不受检测噪声的影响。虽然这种设置属于干净的受控环境，但TransHands本身具有即插即用的特性，可以集成任何2D追踪器以用于实际部署（见第5.1节）。

迁移协议。为了评估我们的迁移策略，我们采用了三阶段协议，逐步将骨干网络从完全冻结状态放松为部分解冻状态，在保留先验知识的同时实现领域适配。

- 第一阶段：几何对齐（S1）。我们用预训练的人体姿态权重初始化骨干网络 $\mathcal{E}_{body}$，并保持其冻结状态。仅在Re:InterHand数据集上优化拓扑适配器、投影头和输出解码器。此外，在此阶段会激活自监督的Lifted Masked Motion Modelling（L-MMM）目标函数（详见补充材料），通过从部分掩码的2D输入中重建完整的3D姿态，来提升模型对遮挡情况的鲁棒性。

– 第二阶段A：域内微调（S2-A）。从S1的检查点出发，我们有选择地解冻最后的Transformer块，并在

---

Re:InterHand数据集上使用有监督的加权MPJPE损失（$\mathcal{L}_{sup}$）进行优化，使模型能够适应手部细微动态，同时保留S1阶段获得的鲁棒表征。

- 第2阶段B：多数据集泛化（S2-B）。为了解决跨域泛化问题，我们针对目标域——AssemblyHands和GigaHands，对基于Re:InterHand对齐的S1模型进行专门训练，训练目标与S2-A阶段相同。我们采用平衡采样策略 [^27] [^42]，以确保在适配阶段所有域都能被均匀涉及。

实现细节。所有模型均基于PyTorch实现，并使用AdamW优化器 [^18]。第1阶段的学习率设为 $10^{-4}$。对于第2阶段（包括A和B），我们采用差异化的学习率：未冻结的骨干层学习率为 $5 \times 10^{-6}$，以保留预训练特征；适配器层的学习率则为 $5 \times 10^{-5}$。余弦热身调度器 [^17] [^35] 用于在每个阶段开始时稳定梯度。为了便于复现实验，我们公开了源代码 $^{3}$，其中包含所有骨干网络和各阶段的训练配置。

消融研究。第4.3节展示了关于迁移策略及其在低数据量场景下的表现的关键消融实验结果，同时也涉及了关键架构组件的影响。关于渐进式训练策略以及在第1阶段使用L-MMM目标的进一步分析，详见补充材料。

## 4.2 主要结果：骨干网络对比

表1报告了在Re:InterHand数据集上的适配效率与精细化效果，对比了冻结适配（S1）与部分解冻适配（S2-A）的情况。为评估框架的可扩展性，表2给出了跨域泛化性能的结果；此处我们在两个不同场景下对模型进行评估：冻结的以身体为中心的先验模型（S1）的零样本能力，以及多数据集训练（S2-B）在同一验证集上的表现。我们采用平均关节位置误差（MPJPE）[^21] 来衡量绝对位置精度，该指标计算的是根节点对齐后预测关节与真实关节之间的平均欧几里得距离。由于二维姿态提升存在尺度歧义，提升后的预测结果会归一化到参考骨骼长度，而RGB基线模型则可直接预测实际尺度。为了独立于全局因素评估重建结构的质量，我们还报告了经过普洛克鲁斯特斯对齐的MPJPE（PA-MPJPE），即在计算误差前先对预测结果进行刚性对齐（平移和旋转）。此外，补充材料中还提供了这些域上重建质量的定性可视化结果。

基于Transformer的模型占据主导地位。在最终精细化阶段，基于Transformer的架构明显优于图卷积网络。MixSTE在我们的基准测试中表现最佳，其误差低至7.16毫米。这一结果相比其本身已经很出色的冻结基线模型有显著改进（+21.34%）。MotionBERT紧随其后，其双流时空注意力机制的鲁棒性得到验证，误差降低幅度达到+21.45%，最终误差为7.47毫米。

---

<div style="text-align: center;">Table 1: Main Results In-Domain. We report parameter efficiency and metric progression on the Re:InterHand [^24] dataset for the four backbones ([^37] [^38] [^40] [^42]). Total denotes the backbone size, while Tr-S1 and Tr-S2 represent the trainable parameters in Stage 1 and 2 respectively. S1 refers to frozen adaptation (alignment), and S2-A to partial unfreezing (refinement). Gain quantifies the relative MPJPE improvement from S1 to S2-A.</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td rowspan="2">BACKBONE</td><td colspan="3">PARAMS (M)</td><td colspan="2">MPJPE $\downarrow$</td><td colspan="2">PA-MPJPE $\downarrow$</td><td rowspan="2">GAIN (%)</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>TOTAL</td><td style='text-align: center; word-wrap: break-word;'>TR-S1</td><td style='text-align: center; word-wrap: break-word;'>TR-S2</td><td style='text-align: center; word-wrap: break-word;'>S1</td><td style='text-align: center; word-wrap: break-word;'>S2-A</td><td style='text-align: center; word-wrap: break-word;'>S1</td><td style='text-align: center; word-wrap: break-word;'>S2-A</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>MOTIONBERT</td><td style='text-align: center; word-wrap: break-word;'>65.70</td><td style='text-align: center; word-wrap: break-word;'>2.24</td><td style='text-align: center; word-wrap: break-word;'>15.00</td><td style='text-align: center; word-wrap: break-word;'>9.51</td><td style='text-align: center; word-wrap: break-word;'>7.47</td><td style='text-align: center; word-wrap: break-word;'>6.81</td><td style='text-align: center; word-wrap: break-word;'>5.62</td><td style='text-align: center; word-wrap: break-word;'>+21.45</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>MIXSTE</td><td style='text-align: center; word-wrap: break-word;'>36.02</td><td style='text-align: center; word-wrap: break-word;'>2.24</td><td style='text-align: center; word-wrap: break-word;'>10.79</td><td style='text-align: center; word-wrap: break-word;'>9.09</td><td style='text-align: center; word-wrap: break-word;'>7.16</td><td style='text-align: center; word-wrap: break-word;'>6.69</td><td style='text-align: center; word-wrap: break-word;'>5.41</td><td style='text-align: center; word-wrap: break-word;'>+21.34</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>POSEFORMERV2</td><td style='text-align: center; word-wrap: break-word;'>16.62</td><td style='text-align: center; word-wrap: break-word;'>2.24</td><td style='text-align: center; word-wrap: break-word;'>5.89</td><td style='text-align: center; word-wrap: break-word;'>12.62</td><td style='text-align: center; word-wrap: break-word;'>8.01</td><td style='text-align: center; word-wrap: break-word;'>7.88</td><td style='text-align: center; word-wrap: break-word;'>5.88</td><td style='text-align: center; word-wrap: break-word;'>+36.53</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>ST-GCN</td><td style='text-align: center; word-wrap: break-word;'>6.61</td><td style='text-align: center; word-wrap: break-word;'>4.10</td><td style='text-align: center; word-wrap: break-word;'>5.00</td><td style='text-align: center; word-wrap: break-word;'>12.95</td><td style='text-align: center; word-wrap: break-word;'>11.56</td><td style='text-align: center; word-wrap: break-word;'>8.32</td><td style='text-align: center; word-wrap: break-word;'>7.66</td><td style='text-align: center; word-wrap: break-word;'>+10.73</td></tr></table>

<div style="text-align: center;">Table 2: Multi-Dataset Generalization. We compare the zero-shot performance of the frozen backbone (S1) against the multi-dataset refined model (S2-B) on AssemblyHands [^26] and GigaHands [^9].</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td rowspan="3">BACKBONE</td><td colspan="4">ASSEMBLYHANDS</td><td colspan="4">GIGAHANDS</td></tr><tr><td colspan="2">MPJPE $\downarrow$</td><td colspan="2">PA-MPJPE $\downarrow$</td><td colspan="2">MPJPE $\downarrow$</td><td colspan="2">PA-MPJPE $\downarrow$</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>S1</td><td style='text-align: center; word-wrap: break-word;'>S2-B</td><td style='text-align: center; word-wrap: break-word;'>S1</td><td style='text-align: center; word-wrap: break-word;'>S2-B</td><td style='text-align: center; word-wrap: break-word;'>S1</td><td style='text-align: center; word-wrap: break-word;'>S2-B</td><td style='text-align: center; word-wrap: break-word;'>S1</td><td style='text-align: center; word-wrap: break-word;'>S2-B</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>MOTIONBERT</td><td style='text-align: center; word-wrap: break-word;'>42.87</td><td style='text-align: center; word-wrap: break-word;'>11.73</td><td style='text-align: center; word-wrap: break-word;'>22.06</td><td style='text-align: center; word-wrap: break-word;'>6.96</td><td style='text-align: center; word-wrap: break-word;'>139.00</td><td style='text-align: center; word-wrap: break-word;'>2.21</td><td style='text-align: center; word-wrap: break-word;'>29.80</td><td style='text-align: center; word-wrap: break-word;'>1.89</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>MIXSTE</td><td style='text-align: center; word-wrap: break-word;'>38.58</td><td style='text-align: center; word-wrap: break-word;'>10.92</td><td style='text-align: center; word-wrap: break-word;'>21.71</td><td style='text-align: center; word-wrap: break-word;'>6.47</td><td style='text-align: center; word-wrap: break-word;'>121.20</td><td style='text-align: center; word-wrap: break-word;'>2.36</td><td style='text-align: center; word-wrap: break-word;'>28.27</td><td style='text-align: center; word-wrap: break-word;'>2.03</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>POSEFORMERV2</td><td style='text-align: center; word-wrap: break-word;'>39.70</td><td style='text-align: center; word-wrap: break-word;'>13.05</td><td style='text-align: center; word-wrap: break-word;'>21.93</td><td style='text-align: center; word-wrap: break-word;'>7.34</td><td style='text-align: center; word-wrap: break-word;'>115.90</td><td style='text-align: center; word-wrap: break-word;'>2.82</td><td style='text-align: center; word-wrap: break-word;'>27.68</td><td style='text-align: center; word-wrap: break-word;'>2.37</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>ST-GCN</td><td style='text-align: center; word-wrap: break-word;'>54.80</td><td style='text-align: center; word-wrap: break-word;'>20.66</td><td style='text-align: center; word-wrap: break-word;'>24.15</td><td style='text-align: center; word-wrap: break-word;'>11.34</td><td style='text-align: center; word-wrap: break-word;'>137.02</td><td style='text-align: center; word-wrap: break-word;'>6.12</td><td style='text-align: center; word-wrap: break-word;'>26.99</td><td style='text-align: center; word-wrap: break-word;'>4.38</td></tr></table>

最强的精细化提升幅度。从表1中可以发现一个关键现象：最高的适配增益并不一定属于参数量最大的模型。PoseFormerV2的参数量显著较少，且在S1阶段的初始误差（12.62毫米）也高于基准测试中的其他大型Transformer模型，但它在基准测试中却实现了最高的相对改进幅度（+36.53%）。这表明频域Transformer具有潜在的可塑性，一旦在S2-A阶段被激活，就能快速收敛到目标域的几何特征，成功将误差降至8.01毫米。

架构限制。ST-GCN是基准测试中体积最小且唯一的非Transformer模型，它似乎达到了容量上限。尽管在第二阶段其可使用的参数规模可与轻量级Transformer相媲美（500万参数），但它却产生了最高的S2-A误差（11.56毫米）以及最低的整体增益（+10.73%）。这表明：虽然局部微调对自注意力机制极为有效，但GCN仅靠局部解冻难以有效应对显著的领域偏移问题。

跨领域泛化能力。表2揭示了模型内在鲁棒性与适应性之间的显著权衡。在AssemblyHands数据集上的零样本设置（S1）中，纯Transformer模型表现出更优的鲁棒性：MixSTE取得了最佳的初始对齐效果（38.58毫米），这说明其表征机制能够有效编码相对几何关系，因此对未见领域中常见的全局坐标偏移不敏感。

---

该表征机制能有效编码相对几何关系，因此对未见领域中典型的全局坐标偏移不敏感。相反，在GigaHands数据集上，所有架构的初始MPJPE值都较高（> 115毫米）。不过它们的PA-MPJPE值仍具有竞争力（$\approx$ 24–30毫米）。这表明高误差源于全局刚性对齐偏差（即身体与手部空间之间的尺度/旋转差异），而非结构扭曲。多数据集适配阶段（S2-B）解决了这种对齐偏差问题，高容量的Transformer利用其潜在的塑性来恢复目标分布。MixSTE在AssemblyHands数据集上的误差降至10.92毫米，这证实了模型容量是解决几何歧义问题的关键驱动力。在GigaHands数据集上，同样的适配过程将所有骨干网络的初始误差从115毫米以上降至几毫米级别（例如MotionBERT仅为2.21毫米）；这种误差的降低反映了该领域本身的特性，而并非仅由模型容量决定——因为GigaHands包含更多静态、关节活动较少且标注时间上平滑的序列，一旦消除全局对齐偏差，它就更容易被建模。

### 4.3 消融实验

迁移学习的价值。我们假设本框架的高性能得益于迁移学习机制。具体而言，是指利用从全身姿态估计中学到的鲁棒运动先验进行适配，而非仅仅依赖目标手部数据集进行训练。为了验证这一点，我们测试了若从零开始训练，我们的流程是否能取得相近的结果。

实验设置。我们在四种运动编码器架构上采用完全相同的两阶段训练协议，对比两种初始化策略：

1. 随机初始化（基线）：编码器采用标准的随机权重初始化。这迫使模型只能从Re:InterHand数据集中学习三维结构。

2. 身体先验初始化（我们的方法）：编码器继承预训练好的身体权重，从而迁移已学到的时空特征与结构先验。

为了进一步评估所迁移的结构运动先验的有效性，我们接着在低数据量场景下对TransHands进行了测试。我们采用PoseFormerV2作为骨干网络，使用目标数据集中逐步减少的比例（从100%降至10%）来训练该框架，并与从零开始的基线模型进行对比。

结果。表3展示了各个阶段之间的性能差异。在S1阶段（冻结状态）下，随机基线在所有骨干网络上均表现不佳，因为冻结的随机编码器会产生噪声，而投影头无法将其映射为有效的姿态。相比之下，我们的方法从初始语义理解出发，使所有骨干网络的MPJPE值立即降至12.95至9.09毫米之间。当编码器在S2阶段（部分解冻）被允许学习时，随机基线的误差显著减小。然而，它仍达到了性能上限。与基线相比，我们的方法在所有骨干网络上均取得了更优的最终结果。PoseFormerV2从迁移学习中获益最多，性能提升了惊人的35.7%。

---

<div style="text-align: center;">Table 3: Transfer Learning Impact. Comparison between random initialization (Scratch) and our TransHands framework (Ours) on Re:InterHand. The Gain column highlights the MPJPE reduction achieved at S2 through motion prior transfer.</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td rowspan="3">BACKBONE</td><td colspan="4">MPJPE (MM) $\downarrow$</td><td colspan="4">PA-MPJPE (MM) $\downarrow$</td><td rowspan="3">S2 GAIN (%)</td></tr><tr><td colspan="2">S1 (FRz)</td><td colspan="2">S2 (PART)</td><td colspan="2">S1 (FRz)</td><td colspan="2">S2 (PART)</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>SCRATCH</td><td style='text-align: center; word-wrap: break-word;'>OURS</td><td style='text-align: center; word-wrap: break-word;'>SCRATCH</td><td style='text-align: center; word-wrap: break-word;'>OURS</td><td style='text-align: center; word-wrap: break-word;'>SCRATCH</td><td style='text-align: center; word-wrap: break-word;'>OURS</td><td style='text-align: center; word-wrap: break-word;'>SCRATCH</td><td style='text-align: center; word-wrap: break-word;'>OURS</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>POSEFORMERV2</td><td style='text-align: center; word-wrap: break-word;'>19.67</td><td style='text-align: center; word-wrap: break-word;'>12.62</td><td style='text-align: center; word-wrap: break-word;'>12.46</td><td style='text-align: center; word-wrap: break-word;'>8.01</td><td style='text-align: center; word-wrap: break-word;'>10.97</td><td style='text-align: center; word-wrap: break-word;'>7.88</td><td style='text-align: center; word-wrap: break-word;'>8.58</td><td style='text-align: center; word-wrap: break-word;'>5.88</td><td style='text-align: center; word-wrap: break-word;'>35.7%</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>MOTIONBERT</td><td style='text-align: center; word-wrap: break-word;'>10.76</td><td style='text-align: center; word-wrap: break-word;'>9.51</td><td style='text-align: center; word-wrap: break-word;'>8.40</td><td style='text-align: center; word-wrap: break-word;'>7.47</td><td style='text-align: center; word-wrap: break-word;'>8.00</td><td style='text-align: center; word-wrap: break-word;'>6.81</td><td style='text-align: center; word-wrap: break-word;'>6.47</td><td style='text-align: center; word-wrap: break-word;'>5.62</td><td style='text-align: center; word-wrap: break-word;'>11.1%</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>ST-GCN</td><td style='text-align: center; word-wrap: break-word;'>14.71</td><td style='text-align: center; word-wrap: break-word;'>12.95</td><td style='text-align: center; word-wrap: break-word;'>13.36</td><td style='text-align: center; word-wrap: break-word;'>11.56</td><td style='text-align: center; word-wrap: break-word;'>9.31</td><td style='text-align: center; word-wrap: break-word;'>8.32</td><td style='text-align: center; word-wrap: break-word;'>8.81</td><td style='text-align: center; word-wrap: break-word;'>7.66</td><td style='text-align: center; word-wrap: break-word;'>13.5%</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>MixSTE</td><td style='text-align: center; word-wrap: break-word;'>12.71</td><td style='text-align: center; word-wrap: break-word;'>9.09</td><td style='text-align: center; word-wrap: break-word;'>8.88</td><td style='text-align: center; word-wrap: break-word;'>7.16</td><td style='text-align: center; word-wrap: break-word;'>9.37</td><td style='text-align: center; word-wrap: break-word;'>6.69</td><td style='text-align: center; word-wrap: break-word;'>7.00</td><td style='text-align: center; word-wrap: break-word;'>5.41</td><td style='text-align: center; word-wrap: break-word;'>19.5%</td></tr></table>

<div style="text-align: center;">Table 4: Architectural Components Ablation (PoseFormerV2). Impact of replacing standard MLP and Linear layers with our proposed Neural ODE adapter and RetNet projection on Re:InterHand</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td rowspan="2">PROTOCOL</td><td rowspan="2" colspan="2">ODE ADAP. RETNET PROJ.</td><td colspan="2">MPJPE $\downarrow$</td><td colspan="2">PA-MPJPE $\downarrow$</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>S1</td><td style='text-align: center; word-wrap: break-word;'>S2-A</td><td style='text-align: center; word-wrap: break-word;'>S1</td><td style='text-align: center; word-wrap: break-word;'>S2-A</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>BASELINE (MLP + LIN.)</td><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'>18.45</td><td style='text-align: center; word-wrap: break-word;'>11.69</td><td style='text-align: center; word-wrap: break-word;'>11.04</td><td style='text-align: center; word-wrap: break-word;'>8.28</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>+ RETNET PROJ.</td><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>16.55</td><td style='text-align: center; word-wrap: break-word;'>11.65</td><td style='text-align: center; word-wrap: break-word;'>9.51</td><td style='text-align: center; word-wrap: break-word;'>8.45</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>+ ODE ADAPTER</td><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'>13.72</td><td style='text-align: center; word-wrap: break-word;'>9.47</td><td style='text-align: center; word-wrap: break-word;'>8.81</td><td style='text-align: center; word-wrap: break-word;'>6.97</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>OURS (FULL)</td><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>12.62</td><td style='text-align: center; word-wrap: break-word;'>8.01</td><td style='text-align: center; word-wrap: break-word;'>7.88</td><td style='text-align: center; word-wrap: break-word;'>5.88</td></tr></table>

在S2阶段也是如此。使用这种骨干网络进行的数据效率分析进一步表明：TransHands的性能可与在完整数据集上训练的从头训练基线相媲美（误差为12.46毫米），而它所使用的标注数据还不到25%。换句话说，对于PoseFormerV2而言，TransHands将标注需求减少了75%以上，这说明迁移得到的先验知识能够有效弥补大规模3D手部数据集稀缺的问题。

架构组件。我们引入了Neural ODE [^5]适配器以及RetNet [^32]投影头，用以填补手部序列与身体序列之间的拓扑和维度差异。在此，我们将其与标准基线进行对比，以明确它们各自的作用。

实验设置。利用PoseFormerV2及我们的两阶段训练方法，我们评估了四种配置：

1. 基线（MLP + Lin.）：用于输入拓扑映射的标准MLP适配器，以及用于输出投影的线性层。

2. + RetNet Proj.：MLP适配器搭配RetNet投影层，以保留时间上下文信息。

3. + ODE Adap.：连续型Neural ODE适配器搭配基线线性投影层。

4. 我们的方法（完整版）：结合了ODE适配器和RetNet投影层的完整TransHands框架。

结果。表4明确了各模块的贡献。基线模型（MLP适配器+线性投影）在S2-A阶段的误差为11.69毫米。仅添加Ret-

---

Net投影层后，误差仅略有变化（11.65毫米），这表明如果没有匹配的输入表示，更强的投影头也收效甚微。相比之下，Neural ODE适配器带来的误差降低幅度最大（9.47毫米），因为其连续深度建模方式能更好地拟合手部关节与身体关节之间的高度非线性映射关系。这两个模块具有互补性：二者结合后，误差可降至最低水平（8.01毫米），比单独使用ODE适配器时又降低了1.46毫米，这说明一旦ODE将输入对齐到身体流形上，RetNet投影层就能发挥其作用。

### 5 现实场景下的鲁棒性与泛化能力

在实际部署中，手部姿态估计模型必须在严重的域偏移情况下保持可靠性，能够处理来自无约束环境的含噪声的2D检测器输入，并支持在隐私保护限制下的手势识别等下游任务——这些限制可能不允许保留原始RGB数据。本节将从所有这些维度对TransHands与当前最先进的方法进行比较。首先，我们评估其在显著域偏移下的跨场景泛化能力，测试该框架在没有任何目标域监督的情况下对真实2D检测器噪声的抵御能力（第5.1节）。其次，我们通过跨外置视角和自中心视觉场景的手势识别下游任务，验证我们的紧凑型、仅骨骼表示是否编码了语义丰富且可迁移的运动先验（第5.2节）。

### 5.1 RGB到3D手部姿态估计

我们评估模型在域偏移下的鲁棒性：从Re:InterHand域到未见过的AssemblyHands域，使用现成的2D关键点检测器作为TransHands升维模块的输入。

零样本泛化。我们直接在AssemblyHands验证集上评估表现最优的S2-A模型（MixSTE和MotionBERT骨干网络），无需微调；对于每只手，我们汇总四个自中心相机视角的预测结果，取首个能观测到该手的那一视角的预测值。与表2中的域内适配情况不同，此处仅在Re:InterHand上训练的S2-A模型接收的是通过MediaPipe [^19]检测得到的含噪声关键点，而非真实的2D标注数据。表5显示，采用MotionBERT骨干网络时，我们的方法MPJPE达到92.97毫米，PA-MPJPE为23.82毫米；采用MixSTE骨干网络时则分别为93.44毫米和24.28毫米。

尽管我们的模型仅依赖现成检测器输出的稀疏2D关键点，但其性能仍优于RGB基线模型ArcticNet-SF [^8]所报告的MPJPE数值（110.76毫米）。在基于根部对齐的指标下，TransHands的92.97毫米与专门面向自中心场景的手部姿态估计方法V-HPOT [^25]的92.09毫米相当。这里只是对比不同技术路线，并非宣称孰优孰劣：V-HPOT是端到端的RGB模型，能够恢复度量级深度并在测试时自适应，而TransHands则是零样本地将含噪声的现成2D关键点升维，无需任何目标域适配即可达到相近的精度。此外，这一结果还提升了

---

<div style="text-align: center;">Table 5: Zero-Shot Robustness Analysis on AssemblyHands. Evaluation on the unseen AssemblyHands domain. Lift denotes methods that estimate 2D keypoints with an off-the-shelf detector and then uplift them to 3D (our pipeline and MediaPipe 3D, which share the same 2D input); RGB denotes end-to-end methods that estimate 3D directly from the image.</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'>METHOD</td><td style='text-align: center; word-wrap: break-word;'>TYPE</td><td style='text-align: center; word-wrap: break-word;'>MPJPE $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>PA-MPJPE $\downarrow$</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>ARCTICNET-SF [^8]</td><td style='text-align: center; word-wrap: break-word;'>RGB</td><td style='text-align: center; word-wrap: break-word;'>$110.76^{\dagger}$</td><td style='text-align: center; word-wrap: break-word;'>-</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>V-HPOT [^25]</td><td style='text-align: center; word-wrap: break-word;'>RGB</td><td style='text-align: center; word-wrap: break-word;'>$92.09^{\ddagger}$</td><td style='text-align: center; word-wrap: break-word;'>-</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>SMPLER-X [^3]</td><td style='text-align: center; word-wrap: break-word;'>RGB</td><td style='text-align: center; word-wrap: break-word;'>98.20</td><td style='text-align: center; word-wrap: break-word;'>19.40</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>MEDIAPIPE 3D [^19]</td><td style='text-align: center; word-wrap: break-word;'>LIFT</td><td style='text-align: center; word-wrap: break-word;'>130.77</td><td style='text-align: center; word-wrap: break-word;'>29.38</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>OURS (MOTIONBERT)</td><td style='text-align: center; word-wrap: break-word;'>LIFT</td><td style='text-align: center; word-wrap: break-word;'>92.97</td><td style='text-align: center; word-wrap: break-word;'>23.82</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>OURS (MIXSTE)</td><td style='text-align: center; word-wrap: break-word;'>LIFT</td><td style='text-align: center; word-wrap: break-word;'>93.44</td><td style='text-align: center; word-wrap: break-word;'>24.28</td></tr></table>

<div style="text-align: center;">As reported in [^28];  $\dagger$ As reported in [^25].</div>

<div style="text-align: center;">Table 6: Downstream Gesture Recognition. Evaluation of the frozen S2-B representations on Jester and EgoGesture datasets.</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td rowspan="2">BACKBONE</td><td colspan="2">JESTER</td><td colspan="2">EGOGESTURE</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Top-1 (%) $\uparrow$</td><td style='text-align: center; word-wrap: break-word;'>Top-5 (%) $\uparrow$</td><td style='text-align: center; word-wrap: break-word;'>Top-1 (%) $\uparrow$</td><td style='text-align: center; word-wrap: break-word;'>Top-5 (%) $\uparrow$</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>MOTIONBERT</td><td style='text-align: center; word-wrap: break-word;'>88.44</td><td style='text-align: center; word-wrap: break-word;'>95.80</td><td style='text-align: center; word-wrap: break-word;'>88.29</td><td style='text-align: center; word-wrap: break-word;'>97.36</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>MixSTE</td><td style='text-align: center; word-wrap: break-word;'>88.72</td><td style='text-align: center; word-wrap: break-word;'>95.76</td><td style='text-align: center; word-wrap: break-word;'>88.08</td><td style='text-align: center; word-wrap: break-word;'>97.53</td></tr></table>

在相同2D关键点输入下，MediaPipe 3D的性能——后者应用GHUM模型 [^36] 来实现2D到3D的升维。MediaPipe 3D的MPJPE达到130.77毫米；而我们的流程将其降低了约38毫米，这表明我们对3D手部几何的理解优于MediaPipe采用的升维策略。我们还与SMPLer-X [^3]进行了对比，它是目前最先进的全身网格模型，这一对比也凸显了在全身可见性受限的自中心场景中开展专门手部姿态任务的必要性。尽管存在这一局限，SMPLer-X仍取得了更低的PA-MPJPE（19.40毫米）：其参数化的SMPL-X先验强制手部形状符合解剖学规律，且天生不受单目升维中的深度符号歧义影响。不过，我们的流程MPJPE仅为92.97毫米，远低于SMPLer-X的98.20毫米，且参数量仅为其十分之一左右（包括MediaPipe 2D手部检测器在内共67.7百万个参数，而端到端的SMPLer-X流程参数量达703.58百万个）。这种参数量对比需结合具体场景理解：因为SMPLer-X针对的是范围更广的全身网格恢复任务。

### 5.2 下游应用价值：手势识别

为了评估我们框架所学到的时空表征的泛化能力和语义丰富度，我们在下游任务中对TransHands进行评估——

---

流动作识别任务。具体来说，我们从S2-B骨干网络中提取潜在嵌入，并利用它们对手势进行分类。

实验设置。我们冻结TransHands的预训练编码器和投影层，仅训练输入适配器，以严格评估预计算特征的质量。随后，这些潜在表示会被送入时序卷积网络（TCN）[^2]分类头进行处理。该网络采用结合Focal Loss [^15]和标签平滑[^33]的手势分类损失函数进行训练，以应对类别不平衡问题。优化过程通过AdamW优化器完成，并采用预热余弦学习率调度策略。

数据集与领域。我们在两个不同领域上评估这些表征：Jester [^22]是一个大规模人类手势数据集，由固定视角的第三方网络摄像头采集（静态领域）；EgoGesture [^39]则是一个具有挑战性的数据集，包含由可穿戴摄像头采集的第一人称视角手势，存在严重的透视畸变和头部运动问题（动态第一人称领域）。

结果。表6总结了从MotionBERT和MixSTE中提取的冻结表征的Top-1和Top-5准确率。两种模型均展现出跨域迁移能力。在Jester数据集上达到88%的准确率，在EgoGesture数据集上也达到约88%的准确率，且编码器的提升部分处于冻结状态，这一结果证实了我们学习到的运动先验能够编码高级语义动态信息。

与现有研究的对比。为了阐明这些结果的意义，基于全RGB模型的Jester数据集模型（如TSM [^14]）通过利用上下文线索可将准确率提升至95%以上，但计算成本较高。与Schlüner等人[^30]提出的仅基于姿态的方法（在10类子集上的准确率为81.2%）相比，TransHands在完整的27类词汇集上的准确率超过了88%。在EgoGesture数据集上，我们的模型（约88%）的表现优于VGG16+LSTM [^6]（74.7%）和C3D [^34]（86.4%）等RGB基线模型。虽然复杂的RGB-深度融合模型[^4]能达到92.2%的准确率，但TransHands提供了一种高效且能保护隐私的替代方案。

### 5.3 局限性与未来工作

我们的研究结果证实了在多数据集和多骨干网络框架中运用身体运动迁移的优势。未来将进一步开展与特定手部2D到3D提升方法的对比研究。此外，我们的评估主要依赖真实的2D关键点数据，实际场景中的准确率仍受限于2D检测器性能（见第5.1节）。通过TransHands实现2D估计与3D提升的进一步整合，将留待后续分析。

## 6 结论

我们提出了一种框架，将身体运动编码器重新用于从单目2D输入中提升3D手部姿态，将拓扑适配与时间建模分离，从而能够以最小的架构改动实现跨拓扑迁移。在多种编码器中，源自身体的先验为数据高效且鲁棒的手部姿态提升提供了强大的归纳偏差，这凸显了跨关节领域复用运动表征的潜力，以及其在现实场景下游应用中的重要性。

---

### 参考文献

1. Ba, J.L., Kiros, J.R., Hinton, G.E.: 层归一化（2016），<https://arxiv.org/abs/1607.06450>

2. Bai, S., Kolter, J.Z., Koltun, V.: 通用卷积神经网络与循环神经网络在序列建模中的实证评估（2018），<https://arxiv.org/abs/1803.01271>

[^3]: Cai, Z., Yin, W., Zeng, A., Wei, C., Sun, Q., Wang, Y., Pang, H.E., Mei, H., Zhang, M., Zhang, L., Loy, C.C., Yang, L., Liu, Z.: Simpler-x: scaling up expressive human pose and shape estimation. In: Proceedings of the 37th International Conference on Neural Information Processing Systems. NIPS '23, Curran Associates Inc., Red Hook, NY, USA (2023)

4. 曹C.、张Y.、吴Y.、陆H.、程J.：基于循环三维卷积神经网络与时空变换器模块的以自我为中心的手势识别。载于：2017年IEEE国际计算机视觉会议（ICCV）。第3783–3791页（2017年）。<https://doi.org/10.1109/ICCV.2017.406>

[^5]: Chen, R.T.Q., Rubanova, Y., Bettencourt, J., Duvenaud, D.: Neural ordinary differential equations. In: Advances in Neural Information Processing Systems (NeurIPS). pp. 6572–6583 (2018)

6. Donahue, J.、Hendricks, L.A.、Guadarrama, S.、Rohrbach, M.、Venugopalan, S.、Darrell, T.、Saenko, K.：用于视觉识别与描述的长时循环卷积神经网络。载于：IEEE计算机视觉与模式识别会议（CVPR）论文集。第2625–2634页（2015年）。<https://doi.org/10.1109/CVPR.2015.7298878>

7. Drover, D.、M. V、陈C.H.、Agrawal, A.、Tyagi, A.、Huynh, C.P.：仅通过二维投影就能学习三维姿态吗？载于：计算机视觉——ECCV 2018研讨会论文集：2018年9月8-14日于德国慕尼黑举行，第四部分。第78–94页。施普林格-维拉格出版社，柏林、海德堡（2018年）。<https://doi.org/10.1007/978-3-030-11018-5_7>，<https://doi.org/10.1007/978-3-030-11018-5_7>

8. 范Z.、Taheri, O.、Tzionas, D.、Kocabas, M.、Kaufmann, M.、Black, M.J.、Hilliges, O.：Arctic：一个关于灵巧双手与物体交互的数据集。载于：IEEE/CVF计算机视觉与模式识别会议（CVPR）论文集。第12943–12954页（2023年）。<https://doi.org/10.1109/CVPR52729.2023.01244>

[^9]: Fu, R., Zhang, D., Jiang, A., Fu, W., Funk, A., Ritchie, D., Sridhar, S.: Gigahands: A massive annotated dataset of bimanual hand activities. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2025)

[^10]: Grishchenko, I., Bazarevsky, V., Zanfir, A., Bazavan, E.G., Zanfir, M., Yee, R., Raveendran, K., Zhdanovich, M., Grundmann, M., Sminchisescu, C.: Blazepose ghum holistic: Real-time 3d human landmarks and pose estimation. arXiv preprint arXiv:2206.11678 (2022)

11. Ionescu, C.、Papava, D.、Olaru, V.、Sminchisescu, C.：Human3.6m：大规模数据集及自然环境中三维人体感知的预测方法。《IEEE模式分析与机器智能交易》第36卷第7期，第1325–1339页（2014年）。<https://doi.org/10.1109/TPAMI.2013.248>

### [^12]: Kumar, A., Raghunathan, A., Jones, R., Ma, T., Liang, P.: Fine-tuning can distort pretrained features and underperform out-of-distribution. In: International Conference on Learning Representations (ICLR) (2022)

---

13. 李Z.、张Y.、林J.、秦H.、顾J.、袁X.、孔L.、杨X.：Binaryhpe：基于二值化的三维人体姿态与形状估计（2025年），<https://arxiv.org/abs/2311.14323>

14. 林J.、甘C.、韩S.：Tsm：用于高效视频理解的时间移位模块。载于：IEEE/CVF国际计算机视觉会议（ICCV）论文集。第7082–7092页（2019年）。<https://doi.org/10.1109/ICCV.2019.00718>

15. 林T.Y.、Goyal, P.、Girshick, R.、何K.、Dollár, P.：用于密集物体检测的Focal损失。载于：IEEE国际计算机视觉会议（ICCV）论文集。第2999–3007页（2017年）。<https://doi.org/10.1109/ICCV.2017.324>

16. Loper, M.、Mahmood, N.、Romero, J.、Pons-Moll, G.、Black, M.J.：一种带皮肤的多人线性模型。《ACM图形学交易》第34卷第6期，第1–16页（2015年）。<https://doi.org/10.1145/2816795.2818013>

[^17]: Loshchilov, I., Hutter, F.: Sgdr: Stochastic gradient descent with warm restarts. In: International Conference on Learning Representations (ICLR) (2017)

[^18]: Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. In: International Conference on Learning Representations (ICLR) (2019)

19. Lugaresi, C.、唐J.、Nash, H.、McClanahan, C.、Uboweja, E.、Hays, M.、张F.、Chang, C.L.、Yong, M.、李J.、Chang, W.T.、华W.、Georg, M.、Grundmann, M.：MediaPipe：一种感知与处理现实的框架。载于：2019年IEEE计算机视觉与模式识别会议（CVPR）上的第三届AR/VR计算机视觉研讨会论文集（2019年），<https://mixedreality.cs.cornell.edu/s/NewTitle_May1_MediaPipe_CVPR_CV4ARVR_Workshop_2019.pdf>

20. Mahmood, N.、Ghorbani, N.、Troje, N.F.、Pons-Moll, G.、Black, M.J.：AMASS：作为表面形状的动作捕捉档案库。载于：IEEE/CVF国际计算机视觉会议（ICCV）论文集（2019年）。<https://doi.org/10.1109/ICCV.2019.00554>

21. Martinez, J.、Hossain, R.、Romero, J.、Little, J.J.：一种简单却有效的三维人体姿态估计基线。载于：IEEE国际计算机视觉会议（ICCV）论文集。第2659–2668页（2017年）。<https://doi.org/10.1109/ICCV.2017.288>

22. Materzynska, J.、Berger, G.、Bax, I.、Memisevic, R.：Jester数据集：一个大规模的人体手势视频数据集。载于：2019年IEEE/CVF国际计算机视觉研讨会（ICCVW）论文集。第2874–2882页（2019年）。<https://doi.org/10.1109/ICCVW.2019.00349>

23. Mehraban, S., Adeli, V., Taati, B.：Motionagformer：利用Transformer-GCNFormer网络提升3D人体姿态估计精度。载于：2024年IEEE/CVF计算机视觉应用冬季会议（WACV）。第6905–6915页（2024年）。<https://doi.org/10.1109/WACV57701.2024.00677>

[^24]: Moon, G., Saito, S., Xu, W., Joshi, R., Buffalini, J., Bellan, H., Rosen, N., Richardson, J., Mize, M., de Bree, P., Simon, T., Peng, B., Garg, S., McPhail, K., Shiratori, T.: A dataset of relighted 3d interacting hands. In: Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track (2023)

25. Mucha, W., Wray, M., Kampel, M.：面向未见域的以自我为中心的3D手部姿态估计研究。载于：2026年IEEE/CVF计算机视觉应用冬季会议（WACV）。第5776–5786页（2026年）。<https://doi.org/10.1109/WACV61042.2026.00560>

### [^26]: Ohkawa, T., He, K., Sener, F., Hodan, T., Tran, L., Keskin, C.: Assemblyhands: Towards egocentric activity understanding via 3d hand pose estimation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

---

载于：计算机视觉与模式识别会议（CVPR）。第12999–13008页（2023年）。<https://doi.org/10.1109/CVPR52729.2023.01249>

[^27]: Pavlakos, G., Shan, D., Radosavovic, I., Kanazawa, A., Fouhey, D., Malik, J.: Reconstructing hands in 3d with transformers. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 9826–9836 (2024)

[^28]: Prakash, A., Tu, R., Chang, M., Gupta, S.: 3d hand pose estimation in everyday egocentric images. In: European Conference on Computer Vision (ECCV) (2024)

29. Romero, J., Tzionas, D., Black, M.J.：具身化手部模型：对人体与手部进行联合建模与捕捉。ACM图形学交易刊第36卷第6期，第1–17页（2017年11月）。<https://doi.org/10.1145/3130800.3130883>，<http://dx.doi.org/10.1145/3130800.3130883>

30. Schlüserer, N., Bücker, M.：基于小样本学习模型的动态手势识别快速学习方法（2022年），<https://arxiv.org/abs/2212.08363>

31. Shan, W., Liu, Z., Zhang, X., Wang, S., Ma, S., Gao, W.：P-STMO：用于3D人体姿态估计的预训练时空多对一模型。载于：ECCV 2022计算机视觉会议。第461–478页。《计算机科学讲义》，施普林格出版社（2022年）。<https://doi.org/10.1007/978-3-031-20065-6_27>

32. Sun, Y., Dong, L., Huang, S., Ma, S., Xia, Y., Xue, J., Wang, J., Wei, F.：Retentive Network：大型语言模型中Transformer的继任者（2023年），<https://arxiv.org/abs/2307.08621>

33. Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.：重新思考计算机视觉领域的Inception架构。载于：IEEE计算机视觉与模式识别会议（CVPR）论文集。第2818–2826页（2016年）。<https://doi.org/10.1109/CVPR.2016.308>

34. Tran, D., Bourdev, L., Fergus, R., Torresani, L., Paluri, M.：利用3D卷积网络学习时空特征。载于：IEEE国际计算机视觉会议（ICCV）论文集。第4489–4497页（2015年）。<https://doi.org/10.1109/ICCV.2015.510>

[^35]: Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I.: Attention is all you need. In: Advances in Neural Information Processing Systems (NeurIPS). pp. 5998–6008 (2017)

36. Xu, H., Bazavan, E.G., Zanfir, A., Freeman, W.T., Sukthankar, R., Sminchisescu, C.：Ghum与GhumL：生成式3D人体形状与关节姿态模型。载于：2020年IEEE/CVF计算机视觉与模式识别会议（CVPR）。第6183–6192页（2020年）。<https://doi.org/10.1109/CVPR42600.2020.00622>

37. Yan, S., Xiong, Y., Lin, D.：基于骨骼的动作识别时空图卷积网络。载于：AAAI人工智能会议论文集。第32卷（2018年）。<https://doi.org/10.1609/aaai.v32i1.12328>

38. Zhang, J., Tu, Z., Yang, J., Chen, Y., Yuan, J.：MixSte：用于视频中3D人体姿态估计的Seq2Seq混合时空编码器。载于：IEEE/CVF计算机视觉与模式识别会议（CVPR）论文集。第13222–13232页（2022年）。<https://doi.org/10.1109/CVPR52688.2022.01288>

39. Zhang, Y., Cao, C., Cheng, J., Lu, H.：Egogesture：全新的以自我为中心的手势识别数据集与基准测试。IEEE多媒体汇刊第20卷第5期，第1038–1050页（2018年）。<https://doi.org/10.1109/TMM.2018.2808769>

## [^40]: Zhao, Q., Zheng, C., Liu, M., Wang, P., Chen, C.: Poseformerv2: Exploring frequency domain for efficient and robust 3d human pose estimation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

---

载于：计算机视觉与模式识别会议（CVPR）。第8877–8886页（2023年）。<https://doi.org/10.1109/CVPR52729.2023.00857>

41. Zhao, Z., Yang, L., Sun, P., Hui, P., Yao, A.：分析3D手部姿态估计中合成数据到真实数据的域差异。载于：2025年IEEE/CVF计算机视觉与模式识别会议（CVPR）。第12255–12265页（2025年）。<https://doi.org/10.1109/CVPR52734.2025.01144>

42. Zhu, W., Ma, X., Liu, Z., Liu, L., Wu, W., Wang, Y.：MotionBERT：学习人体运动表征的统一视角。载于：IEEE/CVF国际计算机视觉会议（ICCV）论文集。第15039–15053页（2023年）。<https://doi.org/10.1109/ICCV51070.2023.01385>

43. Zimmermann, C., Ceylan, D., Yang, J., Russell, B., Argus, M., Brox, T.：Freihand：一个用于从单张RGB图像中无标记捕捉手部姿态和形状的数据集。载于：IEEE/CVF国际计算机视觉会议（ICCV）论文集，第813–822页（2019年）。<https://doi.org/10.1109/ICCV.2019.00090>