# Estimating Accurate Hand Pose in Camera Space with Vision Transformer

Kaiwen Ren $^{*}$

Yiran Jiang $^{*}$

<mailto:renkaiwen23s@ict.ac.cn>

<mailto:jiangyiran25s@ict.ac.cn>

Institute of Computing Technology,

Chinese Academy of Sciences

University of Chinese Academy of Sciences

Beijing, China

Yongjing Ye

Institute of Computing Technology,

Chinese Academy of Sciences

Beijing, China

<mailto:yeyongjing@ict.ac.cn>

Shihong Xia $^{†}$

Institute of Computing Technology,

Chinese Academy of Sciences

University of Chinese Academy of Sciences

Beijing, China

<mailto:xsh@ict.ac.cn>

![Image](images/7e970841e4537ec0376f2975c053494104b3218dc35082812a1199589cdcf785.jpg)

Local hand pose estimation

![Image](images/7331bbf85bacd8a024b63f7b748cb4f710c16acdef147fbc37d47833644457e1.jpg)

![Image](images/b50be19b302b024345cb24048f227f4ec043dab08cab8ad1de458095f67d197d.jpg)

![Image](images/61316540474d51a7721a0667ce9a2f03d2ef7d609bed5a663287012a277c90ea.jpg)

![Image](images/201fbc36db04e2ff95bfffccc92bfda467cac89568d2b81b2933d639e6bcd0d1.jpg)

Camera-space hand pose estimation

(a) Task illustration

![Image](images/01023c39cffa0916a0eec96992653c7af394aa8d4ce002927b5b5273244870fd.jpg)

Different local pose, same distance to camera

![Image](images/2996f059c7b3a65b587858190e2d513387118996c5d4666611aaf74e455d3842.jpg)

(ii) Different local pose in camera space

![Image](images/7fa9d4cecb8f4283e14b6c61d18669ba96798712242aa0050ef8770c52d3bbef.jpg)

(iv) Highly-similar local projections

(b) Coupling of wrist position and local pose in 2D perspective projection results

<div style="text-align: center;">Figure 1: Iask illustration and challenges. (a) compares local versus camera-space pose estimation, where local estimation predicts joint-relative articulation while camera-space estimation recovers under camera's coordinate. (b) demonstrates the perspective coupling effect. The perspective projection can distorts the projection of same local pose as (i) and (iii). It can also compensate for difference in local pose and yield the same projection results as in (ii) and (iv).</div>

## Abstract

Monocular RGB-based hand pose estimation has emerged as a critical research frontier in computer vision. The local hand pose estimation methods predict hand poses relative to the wrist, while global hand pose estimation also requires estimating the wrist's position in the camera coordinate system. However, this camera-space estimation confronts two fundamental challenges: (1) depth ambiguity in monocular settings, and (2) the coupling effect of hand local poses and global wrist positions in the perspective projections. In particular, this coupling reflects that the projections are jointly determined by local hand poses, wrist positions, and camera intrinsics. To overcome these challenges, our framework proposes two key innovations: Transformation-Isomorphism Supervision for hand-depth information extraction and Perspective Information Embedding for resolving above coupling effect of local pose and wrist position, both integrated within the mainstream encoder-decoder architecture. Besides, we propose a novel framerate-aware multi-dataset training strategy for sequential pose refinement. Our fully integrated approach achieves at most 37.1% superiority in CS-MJE over SOTA on HO3D. Project page: <https://github.com/Mine268/CS-ViT>.

## CCS Concepts

• Human-centered computing → Interaction techniques;

Computing methodologies → Activity recognition and understanding; Motion capture.

## Keywords

---

## ACM Reference Format

Kaiwen Ren, Yiran Jiang, Yongjing Ye, and Shihong Xia. 2026. Estimating Accurate Hand Pose in Camera Space with Vision Transformer. In Proceedings of the 34th ACM International Conference on Multimedia (MM '26), November 10–14, 2026, Rio de Janeiro, Brazil. ACM, New York, NY, USA, 20 pages. <https://doi.org/10.1145/3767308.3836423>

## 1 Introduction

Accurate monocular RGB-based hand pose estimation plays a pivotal role for VR/AR systems and Human-Computer Interaction (HCI) applications, by facilitating seamless interaction with virtual objects without additional sensors. As shown in Fig 1(a), local hand pose estimation methods aim to regress wrist-relative hand poses, whereas camera-space hand pose estimation requires localizing the wrist joint in camera coordinates simultaneously. Camera-space hand pose estimation could provide more comprehensive hand information, therefore becoming the primary focus in our work.

Camera-space hand pose estimation faces two major challenges. First, inherent depth ambiguity in monocular views arises when objects of varying sizes at different distances yield identical projections, which is a common limitation in hand pose estimation. Local hand pose estimation methods try to estimate accurate hand size by introducing prior information [^27] [^29]. However, predicting camera-space hand pose not only requires accurate estimation of hand size, but also depth inference based on that size estimation [^10] [^12]. This imposes stricter requirements on the model's ability to perceive depth.

The second challenge is the irreducible coupling result of wrist positions and local hand poses under perspective projection, we claimed it as “perspective coupling effect”. As shown in Fig. 1(b), orthographic projection precludes perspective effects, thereby rendering the influences of local pose and root joint position mutually independent in the projected result. Consequently, the root joint’s x, y coordinates and local orientation can be directly inferred from the image, as illustrated in (i) and (ii). In contrast, perspective projection entangles root position and local pose, both contributing to the 2D hand orientation in the projection. This coupling makes it impossible to determine the relative contribution of each factor from the image alone, thereby complicating pose estimation. An extreme example is shown in (iv): three distinct combinations of local pose and root joint position produce identical 2D hand projections.

Perspective effects are therefore indispensable in camera-space pose estimation. Accurate recovery necessitates explicit modeling of camera intrinsics and perspective geometry. Most existing approaches[5, 21–23, 33] either overlook this requirement entirely or rely on virtual intrinsic assumptions; even the latter typically restrict such assumptions to auxiliary 2D projection loss computation, thereby introducing systematic errors into camera-space estimation. For full theoretical analysis please refer to Appendix.

To address the aforementioned challenges, the model must (1) sensitively incorporate hand priors to extract depth information from hand images and (2) precisely perceive perspective information of the hand projection. For the first requirement, inspired by [^26], we propose Transformation-Isomorphism Supervision (TIS), which aligns the scale and depth information through “rotation” & “scaling” in the latent space to enhance the perception of them.

![Image](images/2e0c24beeb8cca63260e5366497db5f2361f5981bb59af6968b04e7308478c41.jpg)

<div style="text-align: center;">Figure 2: The illustration of our intuitive baseline. The cropping image (#1) is treated as if it is taken by pointing camera directly to the hand (#2). Then the predicted dummy pose is rotated to match the actual position of the cropping, giving prediction of actual pose.</div>

for the backbone. For the second requirement, the Perspective Information Embedding (PIE) mechanism is inserted, enabling the ability of perspective information awareness by decorating the image latent patches with perspective vector. The significance of PIE is demonstrated through our carefully designed baseline (Fig. 2), which builds upon the observations in Fig. 1(b,iv) by first predicting a canonical dummy pose and then applying view-specific transformations to match the input image perspective. In addition, we propose framerate-aware multi-dataset approach to train our model on multiple datasets with various capture configurations, improving training data adaptability and pose estimation accuracy. In conclusion, our contributions are summarized as follows:

(2) Transformation-Isomorphism Supervision (TIS): A training component that improves the ability to perceive depth information and alleviates the ambiguity of depth.

(1) The concept of perspective coupling effect is formalized, with qualitative visualization and theoretical analysis of its impact on hand pose estimation.

(3) Perspective Information Embedding (PIE): A mechanism that helps the model to perceive the perspective information and to model the coupling of perspective on hand pose in the latent space.

(4) Framerate-Aware Multi-Dataset Training (MDT): This mechanism supports joint training across datasets with varying framerate, improving the performance of our pose estimation network.

By integrating the TIS and PIE mechanisms, we achieve accurate camera-space hand pose estimation, demonstrating 7.8% improvement over state-of-the-art methods [^12] on the HO3D dataset. This performance gain further expands to around 37.1% when incorporating the framerate-aware multi-dataset training mechanism.

## 2 Related Works

### 2.1 Local hand pose estimation

In recent years, research has been devoted to hand pose estimation, especially predicting local hand poses from monocular RGB-based image or sequence [4, 8, 14, 17–19, 21, 22, 25, 29, 32, 34, 35, 39].

Among various architectures, transformer-based models outperform others for its high accuracy and robustness $^{[22,24]}$ against

---

other architectures like CNN [^7] [^17] [^40] or graph-CNN [^6]. It has been proven that scaling laws [^11] still roughly hold in this field.

Specialized designs have also been proposed to resolve the inherent challenges of hand pose estimation: occlusion, self-occlusion and motion blur. H2ONet [^33] proposed learning occlusion information for hand-object interaction via dual-branch encoder. Deformer [^5] proposed a dynamic fusion transformer for temporal pose modeling to address motion blur. By incorporating vision transformer with large scale of data, HaMeR[^22] achieves impressive accuracy and robustness in local hand pose estimation. Based on that, WiLoR [^23] advances multi-hand localization and estimation with the novel image-aligned multi-scale sampling and a dataset including 2M+ in-the-wild images.

### 2.2 Hand pose estimation in camera space

Though lots of works with various architectures emerged to tackle the challenges in this field, few works provide solutions for global localization of hand in the camera space, leaving a substantial space for our work.

Camera-space and global hand pose estimation has drawn the attention of cutting-edge research. This distinct task focuses on estimating the pose of hand, including the position relative to camera. Recent studies [^36] [^37] integrate local hand pose estimation methods[^22] [^24] with SLAM-based camera tracking, enabling continuous hand pose tracking in world coordinates. We categorize them as global hand pose estimation.

For camera coordinate-based estimation, recent approaches address this challenge through novel paradigms. Huang et al. [^10] propose a Neural Voting Field (NVF) that projects 2D features into 3D voxels to predict joint-wise voting preferences, thereby locating hand joints in camera coordinates. This projection mechanism circumvents coupling challenges via voxel representation and reprojection. However, their approach predicts the joint's position instead of rotation due to the voxel restriction, limiting the applicability. Valassakis et al. [^12] introduce HandDGP, which established differentiable PnP solutions for derivable 2D-3D lifting. Their PnP algorithm bridges the "local to camera-space" estimation in learnable approach. However, they neglect the perspective coupling effect, resulting in accuracy degradation compared to our method, as shown in our experiment at Tab. 1.

### 2.3 Representation Learning for hand pose estimation

Some studies employ representation learning to enhance pose estimation methods. The goal of representation learning is to learn representations beneficial for downstream tasks, thereby improving both the performance of training and inference. As categorized by [^28], representation learning comprises two paradigms: MIM-based methods reconstruct images to learn representations [^9], while contrastive-based methods leverage positive-negative sample pairs for loss-driven representation learning [^2].

Most representation learning frameworks for hand pose estimation employ contrastive-based methods. For example, SiMHand [^13] constructs positive sample pairs by identifying images with similar poses via 2D pose detectors to enhance robustness against texture/lighting factors. We argue this is essentially knowledge distillation from 2D pose estimators. PeCLR [^29] adapts SimCLR [^3] framework by introducing geometric transformations to feature maps. This method directly applies a geometric inverse transformation in the image space to 2D feature maps, thereby directly constraining strong geometric priors in the latent space. We express concern about the practice of directly injecting intuitive constraints into the latent space. Similarly, [^26] also argues that such an approach can be further improved. Inspired by this, we propose Transformation-Isomorphic Supervision (TIS) to learn pose-enriched representations through parametric mappings isomorphic to geometry transformations across image and 3D domains.

## 3 Method

### 3.1 Overview of Our Pipeline

The spatial and temporal training phases are illustrated in Fig. 3. During the spatial training phase, an off-the-shelf hand detector is adopted to obtain hand bounding boxes from input images $I \in \mathbb{I}$. Subsequently, we utilize a Swin [^15] backbone $E : \mathbb{I} \to \mathbb{R}^{N \times d}$ to encode the cropped hand regions into latent patches $\{P_i \in \mathbb{R}^{d}\}_{i=1}^N$.

The Perspective Information Embedding mechanism leverages both the hand bounding box information and camera intrinsics to refine the latent patches into $\{P'_i\}_{i=1}^N$, which now incorporate pose information in the camera space. $\{P'_i\}_{i=1}^N$ are then processed by the Spatial Module (see Fig. 4) to extract the tokens for pose feature $Q_P, Q_T, Q_S \in \mathbb{R}^d$. Finally, a Pose Decoder $D: \mathbb{R}^{3 \times d} \to \mathbb{R}^{(16 \times 3) + 10 + 3}$ decodes these features into the global hand pose $(\theta, \beta, t)$ for joint rotation, shape, and wrist translation supervision respectively. In addition to supervising $\theta, \beta$ and t, we introduce a novel Transformation Isomorphism Supervision (TIS) mechanism. Specifically, we first apply learnable “scale-rotation” transformation $s_{latent}$ to $\{P'_i\}_{i=1}^N$ and decode it into pose $(\theta', \beta', t')$ parameters using the same Pose Decoder. Then $(\theta', \beta', t')$ is inversely transformed in 3D space and is also supervised against GT.

During the temporal training phase, as illustrated in Fig. 3(ii), we freeze the backbone and Spatial Module. We exclusively train the newly added Temporal Module, which is inserted after the Spatial Module. The Temporal Module takes as input the per-frame pose features $\{Q_P^i\}_{i=1}^{t}$, $\{Q_T^i\}_{i=1}^{t}$, and $\{Q_S^i\}_{i=1}^{t}$, and performs temporal information fusion to produce the refined pose features of the last frame, namely $Q_P^t$, $Q_T^t$, and $Q_S^t$. These fused features are then decoded into pose parameters using the Pose Decoder from the previous spatial training phase.

### 3.2 Perspective Information Embedding

The Fig. 1 demonstrates the coupling effect of wrist position and local hand pose: the projection of hand is the combination of hand pose and perspective information. Thus, we need a component to model the mechanism of this coupling process. Let us informally denote the 3D hand pose as X, the perspective information as K, and the 2D hand projection as Y. The projection process can be expressed as $Y = \Pi(X, K)$, where $\Pi$ is the perspective projection operator. For precise 3D pose recovery X, the 2D hand projection Y is inherently ambiguous without incorporating K. The recovery process can be expressed as $X = \Pi^{-1}(Y, K)$, with the symbol $\Pi^{-1}$ being the recovery process of hand pose estimation conditioned on K. The K is self-explained but the information contained in Y

---

![Image](images/d667a16ecd5ec8415a2756efee0e2e9b6088fbcf9214616cf11bd94373c2c2d3.jpg)

<div style="text-align: center;">Figure 3: Method illustration. (i) In single-frame training, an off-the-shelf detector localizes the hand, then Perspective Information Embedding (PIE) enhances latent feature with camera intrinsic and detection boxes. The spatial module extracts pose features, which are mapped to pose parameter by the Pose Decoder. Transformation Isomorphism Supervision (TIS) transforms patches and compares decoded poses with GT, imposing depth information into latent space. (ii) Temporal Module is added after the spatial module to exploit the temporal cues. We design the zero-init linear layer for training efficiency. The TIS mechanism is omitted in temporal training stage.</div>

![Image](images/d5a002477546ec9e06155c4ebfa69e6af10ed930d938e8b9913f22c23b5fde13.jpg)

<div style="text-align: center;">Figure 4: The spatial module extracts pose information from latent patches with query token  $Q_P^+, Q_T^+, Q_S^+$. The latent patches  $P'_1, \cdots, P'_N$ are discarded at the final layer.</div>

is more than hand image cropping: it also contains the location of cropping on the image, which can be described by the location of hand detection boxes.

The PIE mechanism aims at infusing this condition into the latent features $\{p_i\}_{i=1}^N$ of image cropping. Given the square cropping box of the hand, we uniformly sample $P \times Q$ points within the cropping as $\{p_{ij} \in \mathbb{R}^2\}_{ij}^{P \times Q}$, where $p_{ij}$ is the pixel coordinate of each sample point. For each sample point, we normalize its pixel coordinate $p_{ij}$ back to $u_{ij}$ using intrinsics:

 $$ u_{i j}=K^{-1}\big[p_{i j},1\big]^{\top}\in\mathbb{R}^{3}, $$

where  $K \in \mathbb{R}^{3 \times 3}$ is the intrinsics matrix. The direction vector from the ij-th sample on the normalized projection plane to the origin is denoted as  $\tilde{u}_{ij} = -u_{ij}/\|u_{ij}\|_2$. We drop the z-coordinates and concatenate all the  $\tilde{u}_{ij}$s to get the perspective map

 $$ F_{\mathrm{p e r s p}}=\mathrm{c a t}((\tilde{u}_{11})_{x y},\cdots,(\tilde{u}_{P Q})_{x y})\in\mathbb{R}^{P Q\times2}. $$

As shown in Fig. 3(i), the Perspective Information Embedder encodes the perspective map $F_{persp}$ into the perspective embedding vector $v_{persp} = \text{PIE}(F_{persp}) \in \mathbb{R}^d$, where d is the dimension of the backbone image patches. The vector $v_{persp}$ is then added to each latent patch $\{P_i\}_{i=1}^N$ to infuse perspective information into local image features. We expect this process to mimic the coupling of wrist position and local pose in the latent space, providing clues for camera-space hand pose in xy plane. For detailed architecture of PIE, please refer to the Appendix.

### 3.3 Transformation-Isomorphism Supervision

Transformation Isomorphism Supervision (TIS) stems from a fundamental geometric insight about perspective projection systems. This insight asserts a one-to-one correspondence between scalerotation transformations in the 2D space and those in the 3D pose space. Specifically, if a transformation is applied to a 2D projection result, a corresponding transformation can be applied to its 3D position such that the new 2D result remains the same as the projection of new 3D position, as shown in Fig. 5. A formal specification of these scale-rotation transformations is detailed in the Appendix.

---

Estimating Accurate Hand Pose in Camera Space with Vision Transformer

![Image](images/d75e33b08837b17777d0748157627caf039d38c394f7c3fabc4d5479c860d84d.jpg)

<div style="text-align: center;">(a) origin pose and projection</div>

![Image](images/eb243566516cb02317a9599ca300527c77de8390dcfa6a32c65c8c32ebd3b1bf.jpg)

<div style="text-align: center;">(b) transformed pose and projection</div>

<div style="text-align: center;">Figure 5: An example of transformation isomorphism. The projection relation holds after applying isomorphic transformations in pose space and image space respectively.</div>

This phenomenon is attributed to the intrinsic nature of hand geometry and the principles of perspective projection. The interplay between these two factors establishes a consistent relationship of depth variations across 2D and 3D spaces.

Let the pose estimation network f predict hand pose $\theta, \beta, t$ from image I, formulated as $(\theta, \beta, t) = f(I)$. As shown in Fig. 5, when applying scale-rotation transformation $sr_{2d}(\cdot)$ to the image, the resulting pose will change accordingly. This process can be formulated as $sr_{\mathrm{pose}}(\theta, \beta, t) = f(sr_{2d}(I))$, where the subscript denotes the 3D pose space and the 2D space, respectively. We note that $sr(\cdot)$ penetrates through the network: $f \circ sr_{2d} = sr_{\mathrm{pose}} \circ f$. In other words, the scale-rotation transformations in 2D and 3D pose space are in one-to-one correspondence. [^26] denotes such relation as transformation isomorphism. This relation potentially captures the depth information, alleviating the challenge of z-coordinate localization.

We naturally conjecture whether this isomorphic relationship between image space and pose space can be extended into the latent space. In our pipeline, the estimation process is formulated as $(\theta,\beta,t)=D(E(I))$. By introducing transformation $sr_{latent}(\cdot)$ to model the “scale & rotation” transformation in the latent space, now the transformation isomorphism relation becomes $^{1}$

 $$ s r_{\mathrm{p o s e}}(\theta,\beta,t)=D(s r_{\mathrm{l a t e n t}}(E(I)))=D(E(s r_{2\mathrm{d}}(I))). $$

By enforcing this constraint, TIS encourages the latent space to preserve structurally analogous relationships with both image and pose spaces, facilitating depth information embedding.

The mechanism of TIS is shown in Fig. 6. The parameterized network $sr_{latent} : \mathbb{R}^{Nd+2} \to \mathbb{R}^{Nd}$ is adopted to “scale” and “rotate” the latent patches $\{P'_i\}_{i=1}^N$ by random $s \sim \mathcal{U}$ and $\alpha \sim \mathcal{U}$. The shared decoder D interprets the latent patches into pose $\theta'$, wrist translation $t'$ and shape $\beta'$. By transformation isomorphism relation in eq. (3), we expect $sr_{pose}(\theta', \beta', t'; -\alpha, 1/s) = (\theta, \beta, t)$. We construct the objective of TIS as the regularization term with respect to E, D and $sr_{latent}$ as:

 $$ \begin{array}{r}{\mathcal{L}_{\mathrm{T I S}}=\mathbb{E}_{I,s r}\left[\left\|\mathbf{g t}-\big(s r_{\mathrm{p o s e}}\big)^{-1}\big(D(s r_{\mathrm{l a t e n t}}(E(I)))\big)\right\|\right],}\end{array} $$

where $(sr_{\mathrm{pose}})^{-1}(\cdot)$ denotes the inverse of $sr_{\mathrm{pose}}(\cdot)$, with factor 1/s and $-\alpha$. Our ablation studies exhibit the positive effect of TIS.

Unlike [^26], the transformation isomorphism relations we employ for supervision are established between the latent space and the pose space, whereas [^26] utilizes the relations between the image space and the latent space for supervision.

![Image](images/e773315a4a7c0e3c9316a9899ef10d540710b8bcb5885a598c67091c24f514de.jpg)

<div style="text-align: center;">Figure 6: Illustration of Transformation Isomorphism Supervision. A parameterized network  $sr_{latent}$ takes random rotation and scaling parameters  $\alpha$, s as input to transform the latent patches, and for the pose decoded from the transformed latent patches, it applies inverse 3D rotation and scaling transformations with  $\alpha$, s, then compares the results with the truth.</div>

### 3.4 Framerate-Aware Multi-Dataset Training

Large scale annotated data and temporal information fusion is vital for accurate global hand pose estimation. Incorporating multiple public datasets can solve the first problem, but also introduces the second problem: the capture configuration varies across datasets, especially the framerate. We propose our framerate-aware multi-dataset training (shortened as MDT) approach supporting different framerate in temporal training scenario.

As shown in Fig. 3(ii), given a sequence of images $\{I_i\}^t$ with corresponding timestamps $\{\tau_i\}^t$, the spatial module extracts feature sequences for local pose $\{Q_P^i\}^t$, root position $\{Q_T^i\}^t$, and shape $\{Q_S^i\}^t$. We then apply our Time-indexed RoPE (Ti-RoPE, based on RoPE[^30]) to these sequences. The last token of each sequence $Q_P^t$, $Q_T^t$, $Q_S^t$ is treated as a query to aggregate information from its respective sequence:

X can be P, T and S,

 $$ \begin{align*}\{\hat{Q}^{i}_{\mathsf{X}}\}^{t}&=\mathrm{Time-Indexed-RoPE}\left(\{Q^{i}_{\mathsf{X}}\}^{t},\{\tau_{i}\}^{t}\right),\\\tilde{Q}^{t}_{\mathsf{X}}&=\hat{Q}^{t}_{\mathsf{X}}+\mathrm{TM}_{\mathsf{X}}\left(\mathsf{q}=\hat{Q}^{t}_{\mathsf{X}},\mathsf{v}\&\mathsf{k}=\{\hat{Q}^{i}_{\mathsf{X}}\}^{t}\right),\end{align*} $$

where $\tilde{Q}_{X}^{t}$ are fused token for pose, wrist translation and shape across time and $TM_{P}$, $TM_{T}$, $TM_{S}$ are temporal module branches for local pose, wrist translation and shape feature tokens. Each of them is composed of layers of cross attention module.

Our TI-RoPE is a variant of Rotary Position Embedding [^31] designed to handle irregularly sampled time-series sequences by incorporating explicit temporal intervals into position encoding. In the original RoPE mechanism, the rotation angle $\theta$ of the embedding is determined solely by the token's index i. In our proposed TI-RoPE, we replace this with the corresponding timestamp $\tau_i$ of each token's position, thereby making the encoding sensitive to framerate variations.

For better training efficiency and accuracy, we zero initialize the last linear layer of the Temporal Module, following ControlNet [^38]. This forces the output of Temporal Module to be zero at the start

---

![Image](images/07e1469018e039b01afd1c936a9eb390c517987983ea33ddceaf88df5200a965.jpg)

<div style="text-align: center;">Figure 7: We present qualitative results on the HO3D [^7] datasets. Ground truth meshes are shown in green, and our predictions are in red. All predictions are rendered in camera space with entirely estimated 3D hand pose (predicted local pose parameter  $\theta$,  $\beta$ and wrist position t relative to camera). No ground-truth wrist position or wrist alignment is introduced.</div>

of the training, aligning the behavior to the pipeline without the Temporal Module.

### 3.5 Loss Functions

The loss function for spatial training is defined as followed:

 $$ \mathcal{L}=\mathcal{L}_{\mathrm{j o i n t}}+\mathcal{L}_{\mathrm{v e r t}}+\mathcal{L}_{\mathrm{b e t a}}+\mathcal{L}_{\mathrm{T I S}}, $$

where $\mathcal{L}_{\text{joint}}$ and $\mathcal{L}_{\text{vert}}$ are the L2 loss between the predictions and ground truth for 3D joints and mesh vertices, respectively. $\mathcal{L}_{\text{beta}}$ is the L1 loss between the predicted shape parameters and the ground truth. The loss function for temporal training is defined as:

 $$ \mathcal{L}=\sum_{t=1}^{T}\mathcal{L}_{\mathrm{j o i n t}}^{t}+\mathcal{L}_{\mathrm{v e r t}}^{t}+\mathcal{L}_{\mathrm{b e t a}}^{t}, $$

## 4 Experiments

where t stands for time index.

### 4.1 Implementation Details

We use Swin-Transformer [^15] with “base” scale as our vision backbone. The input image is fixed to $256 \times 256$, producing images patches in $\{P_1, \cdots, P_{64}\} \in \mathbb{R}^{64 \times 768}$. For each of $L = 6$ spatial modules, the standard self-attention module is implemented, with head number and hidden dimension equal to the configuration of backbone. The detailed architecture about spatial module can be found in Fig. 4. The perspective information embedding module is a 3-layer MLP. The $s_{latent}$ is composed of 2 standard self-attention layers. For temporal module, three individual branches with 2 cross-attention layers are inserted for pose, shape and translation.

During the spatial training phase, the temporal module are omitted and we update the backbone, spatial module, perspective embedding module and isomorphism transformation $s_{latent}$. We denoted the model trained under this phase as “Spatial (X)”, where ‘X’ refers the used dataset. In the temporal training phase, the temporal module is inserted and updated solely. The model under this phase is denoted “Temporal (X)”. The “MDT” setup refers the model trained under Temporal phase using all listed datasets. We use AdamW [^16] with learning rate $\sqrt{bs/44} \times 10^{-4}$ and clipping gradients below 5.0 for training, bs for batch size. Readers can refer to the Appendix for more training details.

### 4.2 Datasets and Metrics

DexYCB. [^1] is a large-scale RGB-D dataset capturing diverse natural hand-object interactions over 582K frames from multi-view recordings, annotated with accurate 3D hand poses and fingertip keypoints. The timestamps within the sequences need to be consecutive for temporal training, thus we use S1 protocol for our training/test splitting.

HO3D. [^7] is an RGB-D dataset focused on realistic hand-object interaction, providing over 103K frames of human manipulation captured under heavy mutual occlusion, each annotated with precise 3D hand poses and corresponding object poses.

InterHand2.6M. [^20] is a massive-scale RGB dataset for 3D hand pose estimation, distinguished by its extensive coverage of two-hand interactions. It comprises over 2.6 million annotated frames captured from a multi-view camera setup. The dataset provides accurate 3D hand joint annotations for both single-hand and interacting-hand scenarios.

CS-MJE, RS-MJE, RTE, PA-MJE. Following the metrics mentioned in Huang et al. [^10], we report the mean joint error in camera space (CS-MJE), root-aligned joint error (RS-MJE, RS for Root-Space) and wrist translation error in camera space (RTE) to evaluate the performance of our model. Other than that, we report the mean joint error after Procrustes analysis alignment as PA-MJE.

### 4.3 Qualitative Result

Our qualitative results are presented in Fig. 7. We visualize comparisons between our predicted poses (red) and the ground truth (green) from multiple perspectives. Accurate hand pose estimation in camera space, including root joint positioning, is validated by the significant mesh overlap observed in our results. It is noteworthy that Fig. 7 depicts raw network predictions rendered with no

---

![Image](images/3f501f9e6e5f36e10d6e1b10ba03e7f9e31f809f6e4c16bfea2156fd6e21510a.jpg)

![Image](images/f8ed941133d1853659d7de8791e83586d0318d3d26012fbccc73adaa61b89389.jpg)

<div style="text-align: center;">Figure 8: Wrist joint distribution of DexYCB and HO3D. The poses in the DexYCB dataset significantly surpass those in HO3D in terms of both quantity and range, resulting in greater difficulty for global pose estimation.</div>

<div style="text-align: center;">Table 1: Our metrics reported on HO3D. Our model outperforms the SOTA work in the same field [^12] by 7.8%, 11.7% and 37.1% when training on single frame, multiple frames and multiple datasets respectively. When compared with SOTA work of local hand pose estimation [^22], our model still exhibits overall superiority.</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'>Method</td><td style='text-align: center; word-wrap: break-word;'>CS-MJE $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>RS-MJE $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>PA-MJE $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>RTE $\downarrow$</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>NVF (relative) [^10]</td><td style='text-align: center; word-wrap: break-word;'>-</td><td style='text-align: center; word-wrap: break-word;'>23.2</td><td style='text-align: center; word-wrap: break-word;'>21.8</td><td style='text-align: center; word-wrap: break-word;'>-</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>HandDGP[^12]</td><td style='text-align: center; word-wrap: break-word;'>50.3</td><td style='text-align: center; word-wrap: break-word;'>-</td><td style='text-align: center; word-wrap: break-word;'>-</td><td style='text-align: center; word-wrap: break-word;'>-</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>HaMeR (ViT-H) [^22] + PnP</td><td style='text-align: center; word-wrap: break-word;'>33.8</td><td style='text-align: center; word-wrap: break-word;'>19.1</td><td style='text-align: center; word-wrap: break-word;'>7.2</td><td style='text-align: center; word-wrap: break-word;'>36.0</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>HaMeR (ViT-B) [^22] + PnP</td><td style='text-align: center; word-wrap: break-word;'>55.6</td><td style='text-align: center; word-wrap: break-word;'>35.8</td><td style='text-align: center; word-wrap: break-word;'>11.1</td><td style='text-align: center; word-wrap: break-word;'>48.5</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Spatial (HO3D)</td><td style='text-align: center; word-wrap: break-word;'>46.4</td><td style='text-align: center; word-wrap: break-word;'>26.3</td><td style='text-align: center; word-wrap: break-word;'>10.6</td><td style='text-align: center; word-wrap: break-word;'>44.3</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Temporal (HO3D)</td><td style='text-align: center; word-wrap: break-word;'>44.4</td><td style='text-align: center; word-wrap: break-word;'>25.9</td><td style='text-align: center; word-wrap: break-word;'>10.6</td><td style='text-align: center; word-wrap: break-word;'>41.9</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Spatial (HO3D+DexYCB)</td><td style='text-align: center; word-wrap: break-word;'>43.0</td><td style='text-align: center; word-wrap: break-word;'>23.4</td><td style='text-align: center; word-wrap: break-word;'>9.4</td><td style='text-align: center; word-wrap: break-word;'>39.7</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Temporal (HO3D+DexYCB)</td><td style='text-align: center; word-wrap: break-word;'>36.8</td><td style='text-align: center; word-wrap: break-word;'>23.8</td><td style='text-align: center; word-wrap: break-word;'>9.3</td><td style='text-align: center; word-wrap: break-word;'>33.8</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>MDT</td><td style='text-align: center; word-wrap: break-word;'>31.7</td><td style='text-align: center; word-wrap: break-word;'>17.9</td><td style='text-align: center; word-wrap: break-word;'>8.1</td><td style='text-align: center; word-wrap: break-word;'>31.4</td></tr></table>

ground-truth root or root joint alignment, in contrast to the visualization of local pose works [^5] [^22] [^23] [^33]. We provide more qualitative results and analyze the influence of root alignment in the supplementary.

### 4.4 Quantitative Result

Quantitative result on HO3D. Tab. 1 reports performance on HO3D [^7]. Note that NVF was trained with local pose supervision only, whereas camera-space estimation requires additional root joint localization (high-DoF). This methodological discrepancy renders direct comparison unfair; NVF results are thus included solely for reference completeness.

In contrast, HandDGP[^12] targets the same camera-space task. The proposed method consistently outperforms HandDGP across all configurations by significant margins (e.g., 7.8% reduction in CS-MJE), demonstrating superior camera-space estimation capability.

To further validate effectiveness against current SOTA of local pose estimation, we compare with HaMeR $^{[22]}$, the work incorporating a large amount of data and huge parameters (ViT-H). Since HaMeR predicts poses in local coordinates, we apply Perspectiven-Point (PnP) post-processing to lift predictions to camera space for equitable evaluation. Even though our model and data are not

<div style="text-align: center;">Table 2: Our metrics reported on DexYCB. The evaluation metrics on DexYCB are consistently lower than those on HO3D, primarily due to its broader pose distribution and frequent motion blur in captured images.</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'>Method</td><td style='text-align: center; word-wrap: break-word;'>CS-MJE $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>RS-MJE $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>PA-MJE $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>RTE $\downarrow$</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Spatial (DexYCB)</td><td style='text-align: center; word-wrap: break-word;'>50.1</td><td style='text-align: center; word-wrap: break-word;'>19.4</td><td style='text-align: center; word-wrap: break-word;'>6.9</td><td style='text-align: center; word-wrap: break-word;'>49.5</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Temporal (DexYCB)</td><td style='text-align: center; word-wrap: break-word;'>45.5</td><td style='text-align: center; word-wrap: break-word;'>18.0</td><td style='text-align: center; word-wrap: break-word;'>6.9</td><td style='text-align: center; word-wrap: break-word;'>44.2</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>MDT</td><td style='text-align: center; word-wrap: break-word;'>43.7</td><td style='text-align: center; word-wrap: break-word;'>15.5</td><td style='text-align: center; word-wrap: break-word;'>6.4</td><td style='text-align: center; word-wrap: break-word;'>42.7</td></tr></table>

comparable in scale to HaMeR, we still conducted experiments to demonstrate our method's hand geometry modeling capability in camera-space pose estimation. HaMeR is compared with the MDT setup in the following analysis.

As shown in Tab. 1, the proposed method achieves superior CS-MJE, RS-MJE, and RTE, with only marginally higher PA-MJE. This pattern stems from fundamental differences in handling perspective distortion: the proposed method explicitly encodes intrinsics and perspective effects, enabling disentanglement of global position and local pose that are coupled under perspective projection. This yields accurate camera-space localization (lower CS-MJE and RTE) and refined local pose estimation (lower RS-MJE).

The relative disadvantage in PA-MJE arises from a limitation of the metric itself rather than estimation quality. PA-MJE applies Procrustes alignment between predictions and ground truth, which compensates for local pose errors caused by perspective coupling. Consequently, HaMeR benefits from this alignment-based error cancellation, whereas our method—already accounting for perspective effects via explicit intrinsics—gains no such artificial advantage.

To isolate architectural contributions from model capacity effects, we trained a ViT-B variant of HaMeR (matching our backbone scale). With this configuration, all metrics degraded substantially, implying that HaMeR's effectiveness relies more on accurate data distribution fitting than on proper modeling of hand geometry.

Furthermore, results at varying training data scales demonstrate consistent performance gains with increased data volume, indicating the scalability of the proposed approach.

Quantitative result on DexYCB. To the best of our knowledge, we are the first to report camera-space hand pose estimation performance on DexYCB $^{[1]}$. Compared to the HO3D dataset, the DexYCB dataset features a wider range of hand motions, creating a broader spatial distribution (Fig. 8) with harder edge cases suffering greater perspective distortion. Our metrics on DexYCB are listed in Tab. 2. Due to wilder pose distribution (shown in Fig. 8), the pose estimation task on this dataset is harder. However, no significant accuracy drops are observed for this dataset.

### 4.5 Comparison with Our Observation Baseline

We proposed a baseline estimating the camera-space hand pose from local hand pose inspired by Fig. 1 (b,ii). As shown in the picture, different hands with the same distance to the camera and palm orientation towards the camera, produces almost the same projection. Local hand pose estimation networks, trained on cropped images, implicitly assume a canonical, hand-centric viewpoint. The

---

<div style="text-align: center;">Table 3: Comparison with our proposed baseline on HO3D. The “observ.” denotes the our proposed observation and “Spatial (HO3D) w/o TIS” denotes our method without TIS. The table shows the importance of PIE.</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'>Method</td><td style='text-align: center; word-wrap: break-word;'>CS-MJE $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>RS-MJE $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>PA-MJE $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>RTE $\downarrow$</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>observ.</td><td style='text-align: center; word-wrap: break-word;'>49.1</td><td style='text-align: center; word-wrap: break-word;'>25.1</td><td style='text-align: center; word-wrap: break-word;'>10.5</td><td style='text-align: center; word-wrap: break-word;'>46.4</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Spatial (HO3D) w/o TIS</td><td style='text-align: center; word-wrap: break-word;'>46.6</td><td style='text-align: center; word-wrap: break-word;'>25.6</td><td style='text-align: center; word-wrap: break-word;'>10.4</td><td style='text-align: center; word-wrap: break-word;'>45.1</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Spatial (HO3D)</td><td style='text-align: center; word-wrap: break-word;'>46.4</td><td style='text-align: center; word-wrap: break-word;'>26.3</td><td style='text-align: center; word-wrap: break-word;'>10.6</td><td style='text-align: center; word-wrap: break-word;'>44.3</td></tr></table>

<div style="text-align: center;">Table 4: Errors in x, y and z coordinate for all joints and root joints (with subscript r). Evaluated on HO3D.</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'>Method</td><td style='text-align: center; word-wrap: break-word;'>x</td><td style='text-align: center; word-wrap: break-word;'>$x_{r}$</td><td style='text-align: center; word-wrap: break-word;'>y</td><td style='text-align: center; word-wrap: break-word;'>$y_{r}$</td><td style='text-align: center; word-wrap: break-word;'>z</td><td style='text-align: center; word-wrap: break-word;'>$z_{r}$</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>observ.</td><td style='text-align: center; word-wrap: break-word;'>14.87</td><td style='text-align: center; word-wrap: break-word;'>15.37</td><td style='text-align: center; word-wrap: break-word;'>13.57</td><td style='text-align: center; word-wrap: break-word;'>13.26</td><td style='text-align: center; word-wrap: break-word;'>40.28</td><td style='text-align: center; word-wrap: break-word;'>37.78</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>PIE</td><td style='text-align: center; word-wrap: break-word;'>16.94</td><td style='text-align: center; word-wrap: break-word;'>15.07</td><td style='text-align: center; word-wrap: break-word;'>13.05</td><td style='text-align: center; word-wrap: break-word;'>11.48</td><td style='text-align: center; word-wrap: break-word;'>39.97</td><td style='text-align: center; word-wrap: break-word;'>37.01</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>PIE+TIS</td><td style='text-align: center; word-wrap: break-word;'>15.03</td><td style='text-align: center; word-wrap: break-word;'>14.81</td><td style='text-align: center; word-wrap: break-word;'>13.91</td><td style='text-align: center; word-wrap: break-word;'>13.65</td><td style='text-align: center; word-wrap: break-word;'>36.83</td><td style='text-align: center; word-wrap: break-word;'>35.58</td></tr></table>

<div style="text-align: center;">Table 5: Ablation result of TIS mechanism. Adding TIS in the first training stage reduces CS-MJE and RTE.</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'>Method</td><td style='text-align: center; word-wrap: break-word;'>Dataset</td><td style='text-align: center; word-wrap: break-word;'>CS-MJE $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>PA-MJE $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>RTE $\downarrow$</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>w/o TIS</td><td style='text-align: center; word-wrap: break-word;'>DexYCB</td><td style='text-align: center; word-wrap: break-word;'>51.8</td><td style='text-align: center; word-wrap: break-word;'>6.9</td><td style='text-align: center; word-wrap: break-word;'>51.5</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>w/ TIS</td><td style='text-align: center; word-wrap: break-word;'>DexYCB</td><td style='text-align: center; word-wrap: break-word;'>50.1</td><td style='text-align: center; word-wrap: break-word;'>6.9</td><td style='text-align: center; word-wrap: break-word;'>49.5</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>w/o TIS</td><td style='text-align: center; word-wrap: break-word;'>HO3D</td><td style='text-align: center; word-wrap: break-word;'>50.1</td><td style='text-align: center; word-wrap: break-word;'>10.7</td><td style='text-align: center; word-wrap: break-word;'>45.1</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>w/ TIS</td><td style='text-align: center; word-wrap: break-word;'>HO3D</td><td style='text-align: center; word-wrap: break-word;'>46.4</td><td style='text-align: center; word-wrap: break-word;'>10.6</td><td style='text-align: center; word-wrap: break-word;'>44.3</td></tr></table>

predicted 3D pose is therefore expressed in a canonical coordinate system. As illustrated in Fig. 2, we investigate whether applying the corresponding rotation to map this local pose back into the camera space is sufficient to yield an accurate hand pose.

We train the model based on this observation in HO3D and report the metrics, denoted as “observ”. Our proposed method is denoted as “PIE+TIS”. The result can be found on the Tab. 3 and 4. The proposed baseline deteriorates in camera-space estimation accuracy, proving the importance of perspective information embedding mechanism. The comparison between “observ.” and “PIE” in Tab. 4 shows that PIE improves the pose estimation in xy plane. The comparison between “PIE” and “TIS+PIE” proves that TIS is essential for accurate localization on z axis. Despite the negative impact of TIS on xy-plane accuracy, improved depth estimation (z-axis) compensates for this deficiency, resulting in improvements on the CS-MJE metric, which this task focuses more on (Tab. 3).

### 4.6 Ablation Studies

Ablation study about TIS. TIS enhances the model by improving the depth sensitivity of backbone. We conducted ablation study on TIS mechanism. As shown in Tab. 5, after removing TIS mechanism from the spatial training, the position error of global hand pose estimation increases by 3.2% and 7.4% on DexYCB and HO3D. It proves that by introducing latent transformation that corresponding to the ones in the 2D and 3D space, we can boost the backbone for

<div style="text-align: center;">Table 6: Ablation result of PIE mechanism. We evaluate the metrics on HO3D.</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'>Method</td><td style='text-align: center; word-wrap: break-word;'>CS-MJE $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>RS-MJE $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>PA-MJE $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>RTE $\downarrow$</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>addquery</td><td style='text-align: center; word-wrap: break-word;'>48.7</td><td style='text-align: center; word-wrap: break-word;'>27.2</td><td style='text-align: center; word-wrap: break-word;'>10.7</td><td style='text-align: center; word-wrap: break-word;'>47.1</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>sparse</td><td style='text-align: center; word-wrap: break-word;'>46.5</td><td style='text-align: center; word-wrap: break-word;'>26.8</td><td style='text-align: center; word-wrap: break-word;'>10.7</td><td style='text-align: center; word-wrap: break-word;'>44.4</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>addpatch</td><td style='text-align: center; word-wrap: break-word;'>46.4</td><td style='text-align: center; word-wrap: break-word;'>26.3</td><td style='text-align: center; word-wrap: break-word;'>10.6</td><td style='text-align: center; word-wrap: break-word;'>44.3</td></tr></table>

<div style="text-align: center;">Table 7: Ablation study of zero initialization mechanism in framerate-aware multi-dataset training. The comparison exhibits positive effect of our proposed method.</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'>Method</td><td style='text-align: center; word-wrap: break-word;'>CS-MJE $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>RS-MJE $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>PA-MJE $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>RTE $\downarrow$</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>random</td><td style='text-align: center; word-wrap: break-word;'>47.9</td><td style='text-align: center; word-wrap: break-word;'>26.4</td><td style='text-align: center; word-wrap: break-word;'>10.7</td><td style='text-align: center; word-wrap: break-word;'>43.8</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>zero</td><td style='text-align: center; word-wrap: break-word;'>44.8</td><td style='text-align: center; word-wrap: break-word;'>25.8</td><td style='text-align: center; word-wrap: break-word;'>10.6</td><td style='text-align: center; word-wrap: break-word;'>42.4</td></tr></table>

the extraction of depth information. Although the TIS mechanism improves the global positioning ability, it does not negatively affect the local hand pose estimation, keeping the PA-MJE performance almost unchanged.

Ablation study about PIE mechanism. We ablate the implementation of PIE mechanism. In our proposed PIE mechanism, we combine the perspective embedding vector $v_{persp}$ with the sequence of image patches $\{P_i\}_{i}^N$ through summation, denoted as “addpatch”. For ablation, we added $v_{persp}$ to pose query token $Q_p^+$, $Q_r^+$, $Q_s^+$, denoted as “addquery”. We also ablate the implementation of PIE input construction. Originally, we utilize dense perspective direction map $F_{persp}$ to represent the perspective information. We explore using the normalized coordinates of four corner points for the embedding of the perspective information, denoted as “sparse”.

The metrics of these ablations on HO3D are shown in Tab. 6. The table shows that “addpatch” exhibits best performance, demonstrating that integrating dense perspective information with image features improves camera-space localization accuracy.

Ablation study about zero initialization. The zero-initialization of last linear layer in temporal module makes entire pipeline behave as if there is no temporal module at the beginning of temporal training, as the output of temporal module is zeroed-out and hence has no influence on pose, shape and translation token from the spatial module. This improves the performance compared to the random initialization, as shown in Tab. 7. It proves that providing a fluent transition between stages is crucial for both training and inference.

## 5 Conclusion

We proposed a novel approach for accurate hand pose estimation in camera space. This method can enhance the quality of existing interaction devices by providing pose with respective to camera. The proposed TIS, PIE and MDT mechanism have proved their effectiveness through ablation studies and outperformed the SOTA both in local and camera-space pose estimation task. Further works on effectively utilizing distortion information and temporal clues are under investigation.

---

## Acknowledgments

This work was supported by The Innovation Funding of Institute of Computing Technology, Chinese Academy of Sciences (Grant No. E561010) and the National Key Research and Development Program of China (Grant No. 2022YFB3303202).

## References

[^1]: Yu-Wei Chao, Wei Yang, Yu Xiang, Pavlo Molchanov, Ankur Handa, Jonathan Tremblay, Yashraj S. Narang, Karl Van Wyk, Umar Iqbal, Stan Birchfield, Jan Kautz, and Dieter Fox. 2021. DexYCB: A Benchmark for Capturing Hand Grasping of Objects. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[^2]: Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A Simple Framework for Contrastive Learning of Visual Representations. In Proceedings of the 37th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 119), Hal Daumé III and Aarti Singh (Eds.). PMLR, 1597–1607. <https://proceedings.mlr.press/v119/chen20j.html>

[^3]: Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A Simple Framework for Contrastive Learning of Visual Representations. arXiv preprint arXiv:2002.05709 (2020).

[^4]: MMPose Contributors. 2020. OpenMMLab Pose Estimation Toolbox and Benchmark. <https://github.com/open-mmlab/mmpose>.

[^5]: Qichen Fu, Xingyu Liu, Ran Xu, Juan Carlos Niebles, and Kris M. Kitani. 2023. Deformer: Dynamic Fusion Transformer for Robust Hand Pose Estimation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). 23600–23611.

[^6]: Liuhao Ge, Zhou Ren, Yuncheng Li, Zehao Xue, Yingying Wang, Jianfei Cai, and Junsong Yuan. 2019. 3D Hand Shape and Pose Estimation from a Single RGB Image. In CVPR.

[^7]: Shreyas Hampali, Mahdi Rad, Markus Oberweger, and Vincent Lepetit. 2020. HOnnotate: A Method for 3D Annotation of Hand and Object Poses. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[^8]: Shreyas Hampali, Sayan Deb Sarkar, Mahdi Rad, and Vincent Lepetit. 2022. Keypoint Transformer: Solving Joint Identification in Challenging Hands and Object Interactions for Accurate 3D Pose Estimation. In IEEE Computer Vision and Pattern Recognition Conference.

[^9]: Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. 2021. Masked Autoencoders Are Scalable Vision Learners. arXiv:2111.06377 (2021).

[^10]: Lin Huang, Chung-Ching Lin, Kevin Lin, Lin Liang, Lijuan Wang, Junsong Yuan, and Zicheng Liu. 2023. Neural Voting Field for Camera-Space 3D Hand Pose Estimation. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, Vancouver, BC, Canada, 8969–8978. doi:10.1109/CVPR52729.2023.00866

[^11]: Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling Laws for Neural Language Models. arXiv:2001.08361 [cs.LG] <https://arxiv.org/abs/2001.08361>

[^12]: Ales Leonardis, Elisa Ricci, Stefan Roth, Olga Russakovsky, Torsten Sattler, and Gül Varol (Eds.). 2025. Computer Vision – ECCV 2024: 18th European Conference, Milan, Italy, September 29–October 4, 2024, Proceedings, Part XXXVIII. Lecture Notes in Computer Science, Vol. 15096. Springer Nature Switzerland, Cham. doi:10.1007/978-3-031-72920-1

[^13]: Nie Lin, Takehiko Ohkawa, Yifei Huang, Mingfang Zhang, Minjie Cai, Ming Li, Ryosuke Furuta, and Yoichi Sato. 2025. SiMHand: Mining Similar Hands for Large-Scale 3D Hand Pose Pre-training. In The Thirteenth International Conference on Learning Representations. <https://openreview.net/forum?id=96jZFqM5E0>

[^14]: Nie Lin, Takehiko Ohkawa, Mingfang Zhang, Yifei Huang, Ryosuke Furuta, and Yoichi Sato. 2024. Pre-Training for 3D Hand Pose Estimation with Contrastive Learning on Large-Scale Hand Images in the Wild. arXiv:2409.09714 [cs]. <http://arxiv.org/abs/2409.09714>

[^15]: Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, Furu Wei, and Baining Guo. 2022. Swin Transformer V2: Scaling Up Capacity and Resolution. In International Conference on Computer Vision and Pattern Recognition (CVPR).

[^16]: Ilya Loshchilov and Frank Hutter. 2019. Decoupled Weight Decay Regularization. arXiv:1711.05101 [cs.LG] <https://arxiv.org/abs/1711.05101>

[^17]: Camillo Lugaresi, Jiuqiang Tang, Hadon Nash, Chris McClanahan, Esha Uboweja, Michael Hays, Fan Zhang, Chuo-Ling Chang, Ming Guang Yong, Juhyun Lee, Wan-Teh Chang, Wei Hua, Manfred Georg, and Matthias Grundmann. 2019. MediaPipe: A Framework for Building Perception Pipelines. ArXiv abs/1906.08172 (2019). <https://api.semanticscholar.org/CorpusID:195069430>

[^18]: Hao Meng, Sheng Jin, Wentao Liu, Chen Qian, Mengxiang Lin, Wanli Ouyang, and Ping Luo. 2022. 3D Interacting Hand Pose Estimation by Hand De-occlusion

and Removal. In Computer Vision – ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part VI (Tel Aviv, Israel). Springer-Verlag, Berlin, Heidelberg, 380–397. doi:10.1007/978-3-031-20068-7_22

[^19]: Gyeongsik Moon. 2023. Bringing Inputs to Shared Domains for 3D Interacting Hands Recovery in the Wild. arXiv:2303.13652 [cs]. <http://arxiv.org/abs/2303.13652>

[^20]: Gyeongsik Moon, Shoou-I Yu, He Wen, Takaaki Shiratori, and Kyoung Mu Lee. 2020. InterHand2.6M: A Dataset and Baseline for 3D Interacting Hand Pose Estimation from a Single RGB Image. In European Conference on Computer Vision (ECCV).

[^21]: Yeonguk Oh, JoonKyu Park, Jaeha Kim, Gyeongsik Moon, and Kyoung Mu Lee. 2023. Recovering 3D Hand Mesh Sequence from a Single Blurry Image: A New Dataset and Temporal Unfolding. arXiv:2303.15417 [cs.CV] <https://arxiv.org/abs/2303.15417>

[^22]: Georgios Pavlakos, Dandan Shan, Ilija Radosavovic, Angjoo Kanazawa, David Fouhey, and Jitendra Malik. 2024. Reconstructing Hands in 3D with Transformers. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 9826–9836. doi:10.1109/CVPR52733.2024.00938

[^23]: Rolandos Alexandros Potamias, Jinglei Zhang, Jiankang Deng, and Stefanos Zafeiriou. 2024. WiLoR: End-to-end 3D Hand Localization and Reconstruction in-the-wild. arXiv:2409.12259 [cs.CV]

[^24]: Rolandos Alexandros Potamias, Jinglei Zhang, Jiankang Deng, and Stefanos Zafeiriou. 2025. WiLoR: End-to-end 3D Hand Localization and Reconstruction in the-wild. In Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR). 12242–12254.

[^25]: Aditya Prakash, Ruisen Tu, Matthew Chang, and Saurabh Gupta. 2024. 3D Hand Pose Estimation in Everyday Egocentric Images. In European Conference on Computer Vision (ECCV).

[^26]: Kaiwen Ren, Lei Hu, Zhiheng Zhang, Yongjing Ye, and Shihong Xia. 2025. Learning Transformation-Isomorphic Latent Space for Accurate Hand Pose Estimation. In Pacific Graphics Conference Papers, Posters, and Demos, Marc Christie, Ping-Hsuan Han, Shih-Syun Lin, Nico Pietroni, Teseo Schneider, Hsin-Ruey Tsai, Yu-Shuen Wang, and Eugene Zhang (Eds.). The Eurographics Association. doi:10.2312/pg.20251270

[^27]: Javier Romero, Dimitrios Tzionas, and Michael J. Black. 2017. Embodied Hands: Modeling and Capturing Hands and Bodies Together. ACM Transactions on Graphics, (Proc. SIGGRAPH Asia) 36, 6 (Nov. 2017).

[^28]: Devansh Singh, Aboli Marathe, Sidharth Roy, Rahee Walambe, and Ketan Kotecha. 2024. Explainable rotation-invariant self-supervised representation learning. MethodsX 13 (Dec. 2024), 102959. doi:10.1016/j.mex.2024.102959

[^29]: Adrian Spurr, Aneesh Dahiya, Xi Wang, Xucong Zhang, and Otmar Hilliges. 2022. PeCLR: Self-Supervised 3D Hand Pose Estimation from monocular RGB via Equivariant Contrastive Learning. arXiv:2106.05953 [cs]. <http://arxiv.org/abs/2106.05953>

[^30]: Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. 2024. RoFormer: Enhanced transformer with Rotary Position Embedding. Neurocomput. 568, C (Feb. 2024), 12 pages. doi:10.1016/j.neucom.2023.127063

[^31]: Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. 2021. RoFormer: Enhanced Transformer with Rotary Position Embedding. arXiv:2104.09864 [cs.CL]

[^32]: Zhigang Tu, Zhisheng Huang, Yujin Chen, Di Kang, Linchao Bao, Bisheng Yang, and Junsong Yuan. 2023. Consistent 3D Hand Reconstruction in Video via Self-Supervised Learning. IEEE Trans. Pattern Anal. Mach. Intell. 45, 8 (Aug. 2023), 9469–9485. doi:10.1109/TPAMI.2023.3247907

[^33]: Hao Xu, Tianyu Wang, Xiao Tang, and Chi-Wing Fu. 2023. H2ONet: Hand-Occlusion-and-Orientation-Aware Network for Real-Time 3D Hand Mesh Reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 17048–17058.

[^34]: Yufei Xu, Jing Zhang, Qiming Zhang, and Dacheng Tao. 2022. ViTPose: Simple Vision Transformer Baselines for Human Pose Estimation. In Advances in Neural Information Processing Systems.

[^35]: Yufei Xu, Jing Zhang, Qiming Zhang, and Dacheng Tao. 2022. ViTPose+: Vision Transformer Foundation Model for Generic Body Pose Estimation. arXiv preprint arXiv:2212.04246 (2022).

[^36]: Zhengdi Yu, Stefanos Zafeiriou, and Tolga Birdal. 2024. Dyn-HaMR: Recovering 4D Interacting Hand Motion from a Dynamic Camera. In arXiv preprint arXiv:2412.12861.

[^37]: Jinglei Zhang, Jiankang Deng, Chao Ma, and Rolandos Alexandros Potamias. 2025. HaWoR: World-Space Hand Motion Reconstruction from Egocentric Videos. arXiv preprint arXiv:2501.02973 (2025).

[^38]: Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. 2023. Adding Conditional Control to Text-to-Image Diffusion Models.

[^39]: Zhuoran Zhao, Linlin Yang, Pengzhan Sun, Pan Hui, and Angela Yao. 2025. Analyzing the Synthetic-to-Real Domain Gap in 3D Hand Pose Estimation. In Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR). 12255–12265.

[^40]: Zhishan Zhou, Shihao. Zhou, Zhi Lv, Minqiang Zou, Yao Tang, and Jiajun Liang. 2024. A Simple Baseline for Efficient Hand Mesh Reconstruction.

---

This is the Appendix/Supplementary pages for paper “Estimating Accurate Hand Pose in Camera Space with Vision Transformer”.

## A Perspective Information Embedding

We detailed the Perspective Information Embedding (PIE) module in Sec. 3.2 of the main paper. Here we provide more implementation details. As we stated above, the PIE module aims to model the effect of perspective projection process and camera intrinsics.

### A.1 Perspective Information Encoding from Camera Rays

Given the ROI of hands $\mathcal{B} = (x_1, y_1, x_2, y_2)$ in the image and camera intrinsics $K \in \mathbb{R}^{3 \times 3}$, we sample dense set of points within the bouding box to represent the perspective information. Specifically, we uniformly sample $N = P \times Q$ points within the bounding box $\mathcal{B}$, denoted as $\{p_{ij} \in \mathbb{R}^2\}_{ij}^N$, as shown in Fig. 9.

![Image](images/df5db7e252b098effc0d2def6df66c9fab596e78f4c4f11eb6f8a8265e3b1837.jpg)

<div style="text-align: center;">ROI box Sample point</div>

<div style="text-align: center;">Figure 9: Sampling of Perspective Information Embedding (PIE) module.</div>

The sampled points are then projected to the normalized camera space using the camera intrinsic K. This process normalized the pixel coordinates from image space into camera space, which is formulated as:

 $$ u_{i j}=K^{-1}\cdot[p_{i j};1],u_{i j}\in\mathbb{R}^{3}. $$

 $u_{ij}$ is the normalized camera coordinate of point $p_{ij}$ which can be seen as the intersection of the ray from camera center through pixel $p_{ij}$ with the normalized image plane at Z = 1. Homogeneous coordinate is used here for clarity, which is different from the main paper.

To simulate the process of perspective projection, we cast rays from $u_{ij}$ to the camera center and denoted each ray with the normalized direction vector:

 $$ \hat{u}_{i j}=\frac{-u_{i j}}{\|u_{i j}\|_{2}}. $$

As we have constrained the norm of $\hat{u}_{ij}$ to 1, we dropped the last element of coordinate for simplicity. Each simplified direction vector $(\hat{u}_{ij})_{xy}$ is concatenated into $F_{persp}$ as the perspective information:

 $$ F_{\mathrm{p e r s p}}=\mathrm{c a t}((\hat{u}_{11})_{x y},\cdots,(\hat{u}_{P Q})_{x y})\in\mathbb{R}^{N\times2}. $$

The $F_{\text{persp}}$ is then flatten into $\mathbb{R}^{2N}$ and passed through the Perspective Embedder (which is a MLP) to obtain the perspective embedding vector $v_{\text{persp}} = \text{PIE}(F_{\text{persp}}) \in \mathbb{R}^d$. We assert that this vector contains the perspective information of hand-related region in the input image, which can be helpful for alleviating the coupling of local and global pose under perspective projection.

### A.2 Focal-Length Normalization via Camera Intrinsic

PIE explicitly incorporates camera calibration by mapping an image point $p = [u, v, 1]^{\top}$ to its normalized camera coordinate:

 $$ q=K^{-1}p=\left[\left(u-c_{x}\right)/f_{x}\quad\left(v-c_{y}\right)/f_{y}\quad1\right]^{\top}. $$

Consider the same viewing ray observed by cameras with intrinsic matrices K and $K'$. Its image coordinates satisfy $p = Kq$ and $p' = K'q$, respectively, and therefore

 $$ K^{-1}p=K^{\prime-1}p^{\prime}=q. $$

For example, if the focal length and the corresponding image displacement from the principal point are both scaled by $\lambda$, then $f_{x}^{\prime}=\lambda f_{x}$ and $u^{\prime}-c_{x}^{\prime}=\lambda(u-c_{x})$, giving

 $$ \frac{u^{\prime}-c_{x}^{\prime}}{f_{x}^{\prime}}=\frac{u-c_{x}}{f_{x}}. $$

The same relation holds for the vertical coordinate. Thus, when image projection and the supplied camera intrinsics vary consistently, PIE represents the sampled pixels as the same normalized camera rays and explicitly accounts for focal-length differences. Therefore, given correctly calibrated intrinsics, PIE accounts for focal-length variations by encoding image points as normalized camera rays.

## B Specification of Scale-Rotation Transformation

The scale-rotation transformation is isomorphic in image and 3D space, which we will detail their specification in each space here.

In the image space, the scale-rotation transformation is defined as (i) first scale the image coordinates with a scale factor s around the center, and then (ii) rotate the scaled coordinates with a rotation matrix $R_2(\alpha) \in SO(2)$ around the same center. Formally, given a pixel coordinate $p \in \mathbb{R}^2$ in the image, the transformed pixel coordinate $p'$ is computed as:

 $$ p^{\prime}=R_{2}(\alpha)\cdot(p-c)\cdot s+c, $$

where $c \in \mathbb{R}^2$ is the center of transformation.

In the 3D space, the scale-rotation transformation is defined as (i) shift the z-axis by multiplying $1/s$, and then (ii) rotate the scaled coordinates with a rotation matrix $R_z(\alpha) \in SO(3)$ around the z-axis. Formally, given a 3D coordinate $P \in \mathbb{R}^3$, the transformed 3D coordinate $P'$ is computed as:

 $$ P^{\prime}=R_{z}(\alpha)\cdot P_{s},\quad P_{s}=\begin{bmatrix}P_{x}\\ P_{y}\\ P_{z}/s\end{bmatrix}. $$

For any 3D point P, its 2D projection under intrinsics $(f_x, f_y, c_x, c_y)$ is denoted as pixel coordinate p. After applying the above scaleration transformation in both image and 3D space, the transformed 3D point $P'$ will be projected to the transformed pixel coordinate $p'$. This property shows the isomorphism of the scale-rotation transformation in image and 3D space. Following the ordinary image-space rotation used here, we consider the case $f_x = f_y = f$ and let the transformation center coincide with the camera principal point, i.e., $c = [c_x, c_y]^\top$. Here is a brief proof:

---

Given a 3D point P and its projection p:

 $$ p_{x}=f_{x}\frac{P_{x}}{P_{z}}+c_{x},\quad p_{y}=f_{y}\frac{P_{y}}{P_{z}}+c_{y}. $$

After applying the scale-rotation transformation in 3D space, we have:

 $$ P^{\prime}=R_{z}(\alpha)\cdot\begin{bmatrix}P_{x}\\ P_{y}\\ P_{z}/s\end{bmatrix}=R_{z}(\alpha)\cdot P_{s}. $$

The projection of $P'$ is computed as:

 $$ p_{x}^{\prime}=f_{x}\frac{\left(R_{z}(\alpha)\cdot P_{s}\right)_{x}}{\left(R_{z}(\alpha)\cdot P_{s}\right)_{z}}+c_{x},\quad p_{y}^{\prime}=f_{y}\frac{\left(R_{z}(\alpha)\cdot P_{s}\right)_{y}}{\left(R_{z}(\alpha)\cdot P_{s}\right)_{z}}+c_{y}. $$

On the other hand, applying the scale-rotation transformation in image space, we have:

 $$ p^{\prime}=R_{2}(\alpha)\cdot(p-c)\cdot s+c. $$

Writing $R_{2}(\alpha) = [R_{ij}]$ and substituting p into the above equation, we have:

 $$ \begin{aligned}p_{x}^{\prime}=&\boldsymbol{R}_{11}\left(f_{x}\frac{P_{x}}{P_{z}}+c_{x}-c_{x}\right)\boldsymbol{s}\\&+\boldsymbol{R}_{12}\left(f_{y}\frac{P_{y}}{P_{z}}+c_{y}-c_{y}\right)\boldsymbol{s}+c_{x},\end{aligned} $$

 $$ \begin{aligned}\boldsymbol{p}_{y}^{\prime}=&\boldsymbol{R}_{21}\left(f_{x}\frac{P_{x}}{P_{z}}+c_{x}-c_{x}\right)\boldsymbol{s}\\&+R_{22}\left(f_{y}\frac{P_{y}}{P_{z}}+c_{y}-c_{y}\right)\boldsymbol{s}+c_{y}.\end{aligned} $$

With out loss of generality, we list the derivation for $p'_x$ here. Using $f_x = f_y = f$, the x coordinate can be simplified as:

 $$ \begin{aligned}p_{x}^{\prime}&=R_{11}(f\frac{P_{x}}{P_{z}})s+R_{12}(f\frac{P_{y}}{P_{z}})s+c_{x}\\&=f\frac{R_{11}P_{x}s+R_{12}P_{y}s}{P_{z}}+c_{x}\\&=f\frac{(R_{z}(\alpha)\cdot P_{s})_{x}}{(R_{z}(\alpha)\cdot P_{s})_{z}}+c_{x}\\&=f\frac{P_{x}^{\prime}}{P_{z}^{\prime}}+c_{x}.\\ \end{aligned} $$

The derivation for $p'_{y}$ is similar. Thus we prove that the scalerotation transformation is isomorphic in image and 3D space.

Specifically, since the input to $sr_{pose}$ consists of the joint rotations $\theta$, shape parameters $\beta$, and root joint translation t, we define the 3D space scale-rotation operation as applying a rotation to the root joint orientation and translation, and a z-axis scaling of the root joint translation.

Step-by-Step Example. Consider $P = [20, 10, 500]^\top$ mm, $f_x = f_y = 600$ px, $c = [320, 240]^\top$ px, $s = 1.2$, and $\alpha = 30^\circ$. As illustrated in Fig. 10, the camera-space path first scales the depth from 500 mm to 500/1.2 = 416.7 mm and then rotates (20, 10) to approximately (12.3, 18.7) mm; projecting the resulting point gives $p' = [337.7, 266.9]^\top$ px. In the image-space path, the original projection $p = [344, 252]^\top$ px is centered as $p - c = [24, 12]^\top$ px, scaled to $[28.8, 14.4]^\top$ px, rotated to $[17.7, 26.9]^\top$ px, and translated back by c, producing the same $p'$.

TIS uses this correspondence to regularize the learned latent transformation during training.

![Image](images/00caaa7e531b802c2f60ca8a6daec9766af9ae06d55a9f684da41c2e17b0aff2.jpg)

<div style="text-align: center;">Figure 10: Scale-rotation correspondence in camera and image spaces.</div>

![Image](images/6c2d0a99e155418f51ba6cf92c430244b4511fd4fcdcd04b1f0b1f55683a4a48.jpg)

<div style="text-align: center;">Figure 11: Images from FreiHAND dataset.</div>

## C About FreiHAND Dataset Results

FreiHAND is a large-scale, single-image 3D hand pose and shape estimation benchmark. It contains over 130,000 training frames, each providing accurate ground truth 3D hand joint annotations (21 keypoints) and MANO shape parameters. The dataset was captured using a multi-view green-screen setup, enabling easy background augmentation and robust training. It has become a standard benchmark for evaluating 3D hand reconstruction from a single RGB image.

As shown in Fig. 11, we note that the images in the FreiHAND dataset have already been normalized. The normalization method involves re-projecting the image via a perspective transformation, which aligns the camera's optical axis with the center of the hand bounding box, following resizing the image to $224 \times 224$.

After this transformation, the resulting image is analogous to one captured by a telephoto lens, exhibiting diminished perspective distortion and more closely approximating an orthographic projection. Moreover, in telephoto scenarios, estimating camera intrinsic parameters is more challenging. This is because variations in object distance produce only subtle changes in projection size. This effect, compounded by factors such as camera shake and defocus during capture, typically exacerbates errors in depth perception. For a machine learning model, this means it is highly prone to overfitting to the dataset's specific characteristics, rendering it unable to accurately estimate the 3D hand pose in camera space.

We trained our model on FreiHAND and plotted the CS-MJE, RTE, RS-MJE and PA-MJE of test set for each epoch, the results are shown in Fig. 12. We observed that the metrics reflecting local hand pose estimation (RS-MJE and PA-MJE) decreased and converged to

---

<div style="text-align: center;">Table 8: PnP post-processing results on the HO3D. GT replaces HaMeR-predicted 2D correspondences with ground-truth 2D joints. All errors are in millimeters.</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'>Method</td><td style='text-align: center; word-wrap: break-word;'>CS-MJE $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>RS-MJE $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>PA-MJE $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>RTE $\downarrow$</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>HaMeR-H + Root-PnP</td><td style='text-align: center; word-wrap: break-word;'>44.15</td><td style='text-align: center; word-wrap: break-word;'>14.50</td><td style='text-align: center; word-wrap: break-word;'>7.17</td><td style='text-align: center; word-wrap: break-word;'>44.19</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>HaMeR-H + Full-PnP</td><td style='text-align: center; word-wrap: break-word;'>33.83</td><td style='text-align: center; word-wrap: break-word;'>19.11</td><td style='text-align: center; word-wrap: break-word;'>7.17</td><td style='text-align: center; word-wrap: break-word;'>36.00</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>HaMeR-H + Full-PnP + GT</td><td style='text-align: center; word-wrap: break-word;'>27.55</td><td style='text-align: center; word-wrap: break-word;'>11.33</td><td style='text-align: center; word-wrap: break-word;'>7.17</td><td style='text-align: center; word-wrap: break-word;'>26.68</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>HaMeR-B + Root-PnP</td><td style='text-align: center; word-wrap: break-word;'>63.52</td><td style='text-align: center; word-wrap: break-word;'>35.37</td><td style='text-align: center; word-wrap: break-word;'>11.06</td><td style='text-align: center; word-wrap: break-word;'>56.77</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>HaMeR-B + Full-PnP</td><td style='text-align: center; word-wrap: break-word;'>55.60</td><td style='text-align: center; word-wrap: break-word;'>35.84</td><td style='text-align: center; word-wrap: break-word;'>11.06</td><td style='text-align: center; word-wrap: break-word;'>48.51</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>HaMeR-B + Full-PnP + GT</td><td style='text-align: center; word-wrap: break-word;'>53.37</td><td style='text-align: center; word-wrap: break-word;'>16.74</td><td style='text-align: center; word-wrap: break-word;'>11.06</td><td style='text-align: center; word-wrap: break-word;'>51.14</td></tr></table>

reasonable levels during training. However, the camera-space hand pose estimation metrics (CS-MJE and RTE) continuously oscillated at a high error level. By comparison, we can determine that the main source of this error is root joint localization, as the RTE metric remained persistently high. Conversely, Fig. 13 shows that the CS-MJE on the training set does decrease, converging to below 30 mm. This divergence from the test set performance confirms that the model has overfit on the global pose estimation task, which validates our theory.

![Image](images/9ce2ca31dc2fab42d2e9801904b68bb574201492b4cc5f512fae48d5505525f9.jpg)

<div style="text-align: center;">Figure 12: Per-epoch metrics on test set of FreiHAND.</div>

![Image](images/e7f874f0293d1d6763bc329523e9b7495e04ef6911733ef5dcfd37fd264c8ace.jpg)

<div style="text-align: center;">Figure 13: Per-step CS-MJE during the training.</div>

## D Additional Quantitative Evaluations

### D.1 PnP Error Analysis

We analyze the PnP post-processing used to lift HaMeR predictions into camera space on the HO3D evaluation set. Root-PnP recovers the camera-space root position by estimating only a global translation $(t_{\kappa}, t_{y}, t_{z})$, while retaining HaMeR's predicted global orientation and MANO-derived hand scale. The translation is obtained by minimizing the 2D reprojection error over all 21 joint correspondences. In contrast, Full-PnP jointly estimates global rotation and translation. Both variants use HaMeR-predicted 2D correspondences, HO3D ground-truth camera intrinsics, and GT-derived hand bounding boxes.

As shown in Tab. 8, HaMeR-H Root-PnP obtains 44.15 mm CS-MJE and 44.19 mm RTE, while its wrist-aligned RS-MJE is 14.50 mm. The difference shows that its camera-space error is dominated by global placement. Full-PnP reduces CS-MJE to 33.83 mm but increases RS-MJE to 19.11 mm, indicating that fitting rotation to predicted 2D joints can improve global placement while degrading root-relative orientation. HaMeR-B remains less accurate under both variants. Replacing predicted 2D correspondences with GT 2D joints substantially reduces RS-MJE for both backbones, confirming that 2D keypoint errors are an important, but not exclusive, source of the remaining error.

![Image](images/96dbfd533e8aeef74477a64dd3532ad6aa4c6357bbb2e36d9e4a0983daec11f8.jpg)

<div style="text-align: center;">Figure 14: Per-joint camera-space PnP errors. W denotes the wrist; T, I, M, R, and P denote the five fingers; levels 1–4 run from the palm-side joint to the fingertip.</div>

![Image](images/a53bdc68b913257c5895e71322e1d24a358684d767ec0aa40265b0d463558bc9.jpg)

<div style="text-align: center;">Figure 15: Per-joint root-relative PnP errors.</div>

---

The per-joint results in Fig. 15 show a consistent spatial pattern. Under Root-PnP, the mean root-relative error across the five fingers increases from 9.36 mm at level 1 to 21.36 mm at the fingertips for HaMeR-H, and from 25.94 to 48.02 mm for HaMeR-B. Translation-only fitting therefore cannot remove errors that accumulate along the kinematic chains.

### D.2 Robustness Analysis

Bounding-box perturbations. We separately evaluate center and scale errors on the HO3D evaluation set. For center perturbations, we fix the bounding-box scale to 1 and shift its center relative to the box size. Random selects a uniformly distributed direction for each sample, radial follows the direction from the camera principal point to the original box center, and tangential is perpendicular to the radial direction. Compared with the clean CS-MJE of 46.36 mm, offsets of 2.5% and 5% cause only modest degradation, while a 20% offset yields 48.95, 50.92, and 54.19 mm for radial, random, and tangential shifts, respectively. For scale perturbations, we keep the center fixed and vary the scale from 0.8 to 1.2. CS-MJE is lowest at scale 1.0 and reaches 47.55 and 47.47 mm at scales 0.8 and 1.2, showing a modest and approximately symmetric degradation as the scale departs from its original value.

![Image](images/c87f5c610923c50bf902459618354f7ced9e76e1d63bf765b2ce75ae90fda6b7.jpg)

<div style="text-align: center;">Figure 16: Bounding-box center robustness on HO3D.</div>

![Image](images/159cd3a829f5f17ba07b81575323e8e4dd9254ba4470493747113c3a36bb97ec.jpg)

<div style="text-align: center;">Figure 17: Bounding-box scale robustness on HO3D.</div>

Calibration-error sensitivity. We keep the image crop fixed and perturb only the camera intrinsics supplied to PIE. Focal-length errors within $\pm5\%$ change CS-MJE by less than 1%, while focal scales of 0.8 and 1.2 increase CS-MJE to 48.08 and 49.07 mm, respectively. A 5% principal-point shift increases CS-MJE to between 48.71 and 50.88 mm, depending on direction. Calibration errors therefore increase camera-space joint and wrist-translation errors, with larger perturbations causing greater degradation.

![Image](images/888729482944d9ed773c18b1a0462f35f79daa8b0f68274a0d1e6547d079cf8b.jpg)

<div style="text-align: center;">Figure 18: Focal-length robustness on HO3D.</div>

![Image](images/c2bf0c907c305b2d3ad537d62493c4adcd9e8c1daa36c7aad36660b79e0dad73.jpg)

<div style="text-align: center;">Figure 19: Principal-point robustness on HO3D.</div>

### D.3 Additional Ablation on DexYCB

We additionally evaluate the main components on DexYCB. The results are summarized in Tab. 9.

Compared with the observation baseline, PIE reduces RS-MJE from 18.9 to 18.2 mm and PA-MJE from 7.2 to 7.0 mm, improving local-pose accuracy. Adding TIS reduces CS-MJE from 51.8 to 50.1 mm and RTE from 51.5 to 49.5 mm, while RS-MJE increases from 18.2 to 19.4 mm. The DexYCB results therefore support a camera-space benefit from TIS while revealing a trade-off in root-relative pose accuracy.

---

<div style="text-align: center;">Table 9: DexYCB ablation results (mm).</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'>Method</td><td style='text-align: center; word-wrap: break-word;'>CS-MJE $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>RS-MJE $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>PA-MJE $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>RTE $\downarrow$</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>observ.</td><td style='text-align: center; word-wrap: break-word;'>51.3</td><td style='text-align: center; word-wrap: break-word;'>18.9</td><td style='text-align: center; word-wrap: break-word;'>7.2</td><td style='text-align: center; word-wrap: break-word;'>50.8</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>PIE</td><td style='text-align: center; word-wrap: break-word;'>51.8</td><td style='text-align: center; word-wrap: break-word;'>18.2</td><td style='text-align: center; word-wrap: break-word;'>7.0</td><td style='text-align: center; word-wrap: break-word;'>51.5</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>PIE + TIS</td><td style='text-align: center; word-wrap: break-word;'>50.1</td><td style='text-align: center; word-wrap: break-word;'>19.4</td><td style='text-align: center; word-wrap: break-word;'>6.9</td><td style='text-align: center; word-wrap: break-word;'>49.5</td></tr></table>

## E Training details

<div style="text-align: center;">Table 10: Details of spatial training phase for DexYCB and HO3D.</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'>Property</td><td style='text-align: center; word-wrap: break-word;'>Value</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>epoch</td><td style='text-align: center; word-wrap: break-word;'>30</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>batch size</td><td style='text-align: center; word-wrap: break-word;'>32</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>learning rate</td><td style='text-align: center; word-wrap: break-word;'>$10^{-4}$</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>optimizer</td><td style='text-align: center; word-wrap: break-word;'>AdamW</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>learning rate scheduler</td><td style='text-align: center; word-wrap: break-word;'>constant</td></tr></table>

<div style="text-align: center;">Table 11: Details of temporal training phase.</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'>Dataset</td><td style='text-align: center; word-wrap: break-word;'>Property</td><td style='text-align: center; word-wrap: break-word;'>Value</td></tr><tr><td rowspan="6">HO3D</td><td style='text-align: center; word-wrap: break-word;'>epoch</td><td style='text-align: center; word-wrap: break-word;'>30</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>batch size</td><td style='text-align: center; word-wrap: break-word;'>36</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>learning rate</td><td style='text-align: center; word-wrap: break-word;'>$10^{-4}$</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>frame length</td><td style='text-align: center; word-wrap: break-word;'>7</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>optimizer</td><td style='text-align: center; word-wrap: break-word;'>AdamW</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>learning rate scheduler</td><td style='text-align: center; word-wrap: break-word;'>constant</td></tr><tr><td rowspan="6">DexYCB</td><td style='text-align: center; word-wrap: break-word;'>epoch</td><td style='text-align: center; word-wrap: break-word;'>10</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>batch size</td><td style='text-align: center; word-wrap: break-word;'>32</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>learning rate</td><td style='text-align: center; word-wrap: break-word;'>$10^{-4}$</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>frame length</td><td style='text-align: center; word-wrap: break-word;'>7</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>optimizer</td><td style='text-align: center; word-wrap: break-word;'>AdamW</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>learning rate scheduler</td><td style='text-align: center; word-wrap: break-word;'>constant</td></tr><tr><td rowspan="6">MDT</td><td style='text-align: center; word-wrap: break-word;'>epoch</td><td style='text-align: center; word-wrap: break-word;'>1</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>batch size</td><td style='text-align: center; word-wrap: break-word;'>45</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>learning rate</td><td style='text-align: center; word-wrap: break-word;'>$10^{-4}$</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>frame length</td><td style='text-align: center; word-wrap: break-word;'>7</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>optimizer</td><td style='text-align: center; word-wrap: break-word;'>AdamW</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>learning rate scheduler</td><td style='text-align: center; word-wrap: break-word;'>constant</td></tr></table>

## F Qualitative Results

### F.1 More cases

Please refer to Fig. 22.

### F.2 Root alignment cases

Fig. 20 and Fig. 21 show the difference between two visualization routines on DexYCB and HO3D. In the "Predicted root" column, we render the predicted pose (red) and true pose (green) in the camera-space without any modifications to the predicted wrist. Whereas in local pose cases, the wrist position is manually modified to the ground-truth wrist position, as rendered in "GT-Aligned root" column.

Directly utilizing the estimated root joint position for rendering imposes stricter requirements on pose estimation accuracy, as the model must accurately localize the hand root joint within a vast free space to produce sufficiently accurate rendering results. In contrast, the root-joint alignment operation completely eliminates the difficulties arising from such high degrees of freedom, allowing the model to focus solely on local accuracy. Nevertheless, even when facing such challenges, our model can achieve higher pre-alignment accuracy than post-alignment accuracy in certain scenarios. This is because when root joint estimation is sufficiently accurate, the perspective coupling effect becomes pronounced. Our model pursues overall pose estimation accuracy in camera space (CS-MJE), whereas root joint alignment inadvertently increases error accumulation at distal joints. This characteristic enables our model to achieve more accurate estimation precision for joints that are more frequently involved in interactions, such as fingertips. Quantitative results on DexYCB for every joint are shown in Tab. 12 and Tab. 13, corresponding to row 1 and 2 in Fig. 20 respectively. Quantitative results on HO3D for every joint are shown in Tab. 14 and Tab. 15, corresponding to row 1 and 2 in Fig. 21 respectively.

### F.3 Failure cases

Failure cases are shown in Fig. 23.

## G Jacobian Analysis of Perspective Projection and Perspective Coupling Effect

### G.1 Pinhole Camera Model

The pinhole camera model describes the projection of a 3D spatial point $P_{\text{cam}} = [X, Y, Z]^\top$ to a 2D image point $p = [u, v]^\top$. This model is characterized by the camera intrinsic matrix $K \in \mathbb{R}^{3 \times 3}$:

 $$ K=\begin{bmatrix}f_{x}&0&c_{x}\\ 0&f_{y}&c_{y}\\ 0&0&1\end{bmatrix}, $$

where $f_x$, $f_y$ are the focal lengths in the x, y directions (in pixels), and $c_x$, $c_y$ are the principal point coordinates (image center). The projection process is:

 $$ p=\begin{bmatrix}u\\ v\end{bmatrix}=\pi(P_{cam})=\begin{bmatrix}f_{x}\frac{X}{Z}+c_{x}\\ f_{y}\frac{Y}{Z}+c_{y}\end{bmatrix}. $$

The projection equation eq. (23) reveals the fundamental property of perspective projection: depth Z appears in the denominator, causing the foreshortening effect—objects closer to the camera appear larger in the image.

### G.2 Complete Jacobian Matrix of Pixel Coordinates

To fully characterize the perspective coupling effect, we compute the Jacobian matrix of pixel coordinates $p_i = [u_i, v_i]^\top$ with respect to the six-dimensional vector $[J_{\mathrm{rel},i}^\top, t^\top]^\top$. This matrix comprehensively describes the combined effect of local pose changes $J_{\mathrm{rel},i}$ and root joint position changes t on the projection.

---

![Image](images/242de8cb2780823168164516e03177dc9e87b153a91407197eec80d693408ff7.jpg)

<div style="text-align: center;">Figure 20: DexYCB dataset. Qualitative and quantitative results for alignment choices. The “Predicted” use the wrist position predicted by our model and “GT-Aligned” use the ground-truth wrist position extracted from the annotations.</div>

<div style="text-align: center;">Figure 21: HO3D dataset. Qualitative and quantitative results for alignment choices. The “Predicted” use the wrist position predicted by our model and “GT-Aligned” use the ground-truth wrist position extracted from the annotations.</div>

Consider the i-th joint point in camera space. The joint coordinates are determined by both local pose $J_{rel,i}$ and global translation t:

 $$ J_{\mathrm{c a m},i}=J_{\mathrm{r e l},i}+t, $$

with its projection being $p_i = \pi(J_{\mathrm{cam},i}) = [u_i, v_i]^\top$, where:

 $$ u_{i}=f_{x}\frac{X_{i}}{Z_{i}}+c_{x},\quad v_{i}=f_{y}\frac{Y_{i}}{Z_{i}}+c_{y}, $$

 $$ \mathrm{a n d}X_{i}=J_{\mathrm{r e l},i,x}+t_{x},Y_{i}=J_{\mathrm{r e l},i,y}+t_{y},Z_{i}=J_{\mathrm{r e l},i,z}+t_{z}. $$

Computing partial derivatives of pixel coordinates with respect to $[J_{\mathrm{rel},i}^{\top}, t^{\top}]^{\top}$ yields the complete $2 \times 6$ Jacobian matrix:

 $$ \frac{\partial p_{i}}{\partial[J_{\mathrm{r e l},i}^{\top},t^{\top}]^{\top}}=\left[\begin{matrix}{\frac{f_{x}}{Z_{i}}}&{0}&{-\frac{f_{x}X_{i}}{Z_{i}^{2}}}&{\frac{f_{x}}{Z_{i}}}&{0}&{-\frac{f_{x}X_{i}}{Z_{i}^{2}}}\\ {0}&{\frac{f_{y}}{Z_{i}}}&{-\frac{f_{y}Y_{i}}{Z_{i}^{2}}}&{0}&{\frac{f_{y}}{Z_{i}}}&{-\frac{f_{y}Y_{i}}{Z_{i}^{2}}}\\ \end{matrix}\right]. $$

The Jacobian matrix has an important property: the corresponding components with respect to $J_{rel,i}$ and t have identical partial derivatives, i.e., $\partial u_i / \partial J_{rel,i,x} = \partial u_i / \partial t_x$, $\partial u_i / \partial J_{rel,i,z} = \partial u_i / \partial t_z$, etc. This symmetry arises from the additive structure of camera coordinates $J_{cam,i} = J_{rel,i} + t$-local pose changes and root joint translation changes have identical contributions to camera coordinates, and thus have identical effects on projected coordinates $p_i$.

### G.3 Depth Ambiguity Analysis via Jacobian Matrix

Based on the complete Jacobian matrix, this section analyzes the specific manifestation of depth ambiguity in camera-space hand pose estimation from the perspective of matrix symmetry. Depth ambiguity is an inherent difficulty in monocular vision: the absolute depth of an object cannot be uniquely determined from a single view.

---

Consider a spatial point $P_{\mathrm{cam}} = [X, Y, Z]^{\top}$ and its scaled version along the line of sight $P_{\mathrm{cam}}^{\prime} = \lambda P_{\mathrm{cam}} = [\lambda X, \lambda Y, \lambda Z]^{\top}$, where $\lambda > 0$. According to the projection equation eq. (23):

 $$ p=\pi(P_{\mathrm{c a m}})=\begin{bmatrix}f_{x}\frac{X}{Z}+c_{x}\\ f_{y}\frac{Y}{Z}+c_{y}\end{bmatrix}=\pi(P_{\mathrm{c a m}}^{\prime})=p^{\prime}. $$

Therefore, $P_{cam}$ and $P'_{cam}$ produce exactly the same projection p in the image, making it impossible to distinguish whether the object is at depth Z or $\lambda Z$ from a single image.

From the perspective of the Jacobian matrix, depth ambiguity is manifested in equation eq. (26) as the symmetry of the additive structure. The complete $2 \times 6$ Jacobian matrix shows complete symmetry between local pose $J_{rel,i}$ and global translation t in terms of projection change rates. This symmetry stems from the additivity of camera coordinates $J_{cam,i} = J_{rel,i} + t$: the contributions of $J_{rel,i}$ and t to $J_{cam,i}$ are completely equivalent, and thus their effects on projection are also equivalent. This means that mathematically, it is impossible to distinguish between “local scale changes” and “root joint translation changes” from a monocular image—both produce exactly the same effects on projection change rates.

This symmetry directly leads to the ill-posedness of depth estimation: the same projection change $\Delta p_i$ may be caused by local pose changes $\Delta J_{rel,i}$, or by root joint translation changes $\Delta t$, or even by any combination of the two. The non-zero off-diagonal elements $\partial u_i / \partial t_z$ and $\partial v_i / \partial t_z$ of the Jacobian matrix further indicate that depth estimation errors "leak" into the u, v coordinate estimates in the image plane, and vice versa.

### G.4 Perspective Information Loss and Coupling Effect Analysis

Based on the analysis of the complete Jacobian matrix, this section explores the perspective information loss caused by cropping operations and its interaction with the perspective coupling effect. In practical applications, hand pose estimation typically first obtains a hand bounding box through a detector, then crops and scales the region within the bounding box as input to the network. This preprocessing step simplifies the input but introduces the problem of perspective information loss.

In the original image, the position and size of the hand contain rich perspective geometric information—the farther the hand is from the camera, the smaller the image size, and the projection process is closer to the orthogonal projection. From an information-theoretic perspective, cropping reduces the amount of information for depth estimation. In the original image, the hand position $(u_{0}, v_{0})$ and size $s_{0}$ provide strong cues about depth $t_{z}$: according to perspective projection, size $s_{0} \propto 1/t_{z}$. After cropping, these cues are partially removed, increasing the uncertainty of depth estimation.

The perspective information loss caused by cropping exacerbates the perspective coupling effect. After losing the original position information, the network can only infer global position from the content of the cropped image, and must rely on indirect cues in the image content, which are often ambiguous and unreliable.

From the perspective of the Jacobian matrix, perspective information loss caused by cropping is closely related to the following properties of the matrix:

(1) Mixed influence of local pose and global position: Matrix elements depend on both local pose $J_{\text{rel},i}$ and global translation t. For example, $\partial u_i / \partial t_z = -f_x X_i / Z_i^2$ contains both local pose component $J_{\text{rel},i,x}$ and global translation component $t_x$. This means that projection changes are simultaneously affected by local pose and root joint position, and the effects of the two on the image plane are inseparable.

(2) Strong nonlinear coupling in the depth direction: Depth changes $t_{z}$ not only affect projection through the denominator effect of $Z_{i}$ (near objects appear larger), but also produce cross-coupling through product terms with local coordinates $J_{\mathrm{rel},i,x}$, $J_{\mathrm{rel},i,y}$. When hand pose ( $J_{\mathrm{rel},i}$) changes, the same depth change $t_{z}$ produces different projection offsets.

(3) Limitations of translation direction independence: Horizontal translation $t_{x}$ only affects the u coordinate, and vertical translation $t_{y}$ only affects the v coordinate (diagonal elements of the matrix), but this independence is local because $t_{x}, t_{y}$ indirectly affect depth derivative terms by changing $X_{i}, Y_{i}$. The cropping operation loses position information in the original image, making it difficult for the network to exploit this limited independence to infer translation components.

The non-zero off-diagonal elements $\partial u_i/\partial t_z$ and $\partial v_i/\partial t_z$ of the Jacobian matrix indicate that depth estimation errors “leak” into the u, v coordinate estimates in the image plane, and vice versa. This coupling makes direct estimation of camera-space pose from monocular images an ill-posed problem: the same projection change $\Delta p_i$ may be caused by many different combinations of $(J_{\text{rel},i}, t)$.

---

<div style="text-align: center;">Table 12: Per-joint error change before and after root alignment on case of row #1 of Fig. 20. Negative  $\Delta$ means the error decreases after alignment.</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'>Joint</td><td style='text-align: center; word-wrap: break-word;'>Before</td><td style='text-align: center; word-wrap: break-word;'>After</td><td style='text-align: center; word-wrap: break-word;'>$\Delta$ (After - Before)</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Wrist</td><td style='text-align: center; word-wrap: break-word;'>24.0174</td><td style='text-align: center; word-wrap: break-word;'>0.0000</td><td style='text-align: center; word-wrap: break-word;'>-24.0174</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Thumb_1</td><td style='text-align: center; word-wrap: break-word;'>15.9811</td><td style='text-align: center; word-wrap: break-word;'>8.7894</td><td style='text-align: center; word-wrap: break-word;'>-7.1917</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Thumb_2</td><td style='text-align: center; word-wrap: break-word;'>13.5497</td><td style='text-align: center; word-wrap: break-word;'>12.0409</td><td style='text-align: center; word-wrap: break-word;'>-1.5088</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Thumb_3</td><td style='text-align: center; word-wrap: break-word;'>12.5836</td><td style='text-align: center; word-wrap: break-word;'>13.5603</td><td style='text-align: center; word-wrap: break-word;'>0.9767</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Thumb_4</td><td style='text-align: center; word-wrap: break-word;'>15.4669</td><td style='text-align: center; word-wrap: break-word;'>14.2594</td><td style='text-align: center; word-wrap: break-word;'>-1.2074</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Index_1</td><td style='text-align: center; word-wrap: break-word;'>9.3494</td><td style='text-align: center; word-wrap: break-word;'>20.2602</td><td style='text-align: center; word-wrap: break-word;'>10.9108</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Index_2</td><td style='text-align: center; word-wrap: break-word;'>8.5350</td><td style='text-align: center; word-wrap: break-word;'>20.6546</td><td style='text-align: center; word-wrap: break-word;'>12.1196</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Index_3</td><td style='text-align: center; word-wrap: break-word;'>7.8767</td><td style='text-align: center; word-wrap: break-word;'>19.4604</td><td style='text-align: center; word-wrap: break-word;'>11.5837</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Index_4</td><td style='text-align: center; word-wrap: break-word;'>8.9758</td><td style='text-align: center; word-wrap: break-word;'>19.0349</td><td style='text-align: center; word-wrap: break-word;'>10.0591</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Middle_1</td><td style='text-align: center; word-wrap: break-word;'>9.5127</td><td style='text-align: center; word-wrap: break-word;'>18.7216</td><td style='text-align: center; word-wrap: break-word;'>9.2088</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Middle_2</td><td style='text-align: center; word-wrap: break-word;'>5.8279</td><td style='text-align: center; word-wrap: break-word;'>22.4163</td><td style='text-align: center; word-wrap: break-word;'>16.5884</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Middle_3</td><td style='text-align: center; word-wrap: break-word;'>2.7855</td><td style='text-align: center; word-wrap: break-word;'>25.2093</td><td style='text-align: center; word-wrap: break-word;'>22.4238</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Middle_4</td><td style='text-align: center; word-wrap: break-word;'>6.7389</td><td style='text-align: center; word-wrap: break-word;'>29.1953</td><td style='text-align: center; word-wrap: break-word;'>22.4564</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Ring_1</td><td style='text-align: center; word-wrap: break-word;'>11.7361</td><td style='text-align: center; word-wrap: break-word;'>13.2491</td><td style='text-align: center; word-wrap: break-word;'>1.5130</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Ring_2</td><td style='text-align: center; word-wrap: break-word;'>7.3535</td><td style='text-align: center; word-wrap: break-word;'>17.4583</td><td style='text-align: center; word-wrap: break-word;'>10.1048</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Ring_3</td><td style='text-align: center; word-wrap: break-word;'>4.3867</td><td style='text-align: center; word-wrap: break-word;'>21.7554</td><td style='text-align: center; word-wrap: break-word;'>17.3686</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Ring_4</td><td style='text-align: center; word-wrap: break-word;'>9.7153</td><td style='text-align: center; word-wrap: break-word;'>27.1253</td><td style='text-align: center; word-wrap: break-word;'>17.4100</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Pinky_1</td><td style='text-align: center; word-wrap: break-word;'>15.2682</td><td style='text-align: center; word-wrap: break-word;'>9.6775</td><td style='text-align: center; word-wrap: break-word;'>-5.5907</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Pinky_2</td><td style='text-align: center; word-wrap: break-word;'>13.6472</td><td style='text-align: center; word-wrap: break-word;'>13.1983</td><td style='text-align: center; word-wrap: break-word;'>-0.4490</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Pinky_3</td><td style='text-align: center; word-wrap: break-word;'>14.1184</td><td style='text-align: center; word-wrap: break-word;'>19.5819</td><td style='text-align: center; word-wrap: break-word;'>5.4635</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Pinky_4</td><td style='text-align: center; word-wrap: break-word;'>23.4038</td><td style='text-align: center; word-wrap: break-word;'>30.9909</td><td style='text-align: center; word-wrap: break-word;'>7.5871</td></tr></table>

<div style="text-align: center;">Table 13: Per-joint error change before and after root alignment on case of row #2 of Fig. 20. Negative  $\Delta$ means the error decreases after alignment.</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'>Joint</td><td style='text-align: center; word-wrap: break-word;'>Before</td><td style='text-align: center; word-wrap: break-word;'>After</td><td style='text-align: center; word-wrap: break-word;'>$\Delta$ (After - Before)</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Wrist</td><td style='text-align: center; word-wrap: break-word;'>20.9428</td><td style='text-align: center; word-wrap: break-word;'>0.0000</td><td style='text-align: center; word-wrap: break-word;'>-20.9428</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Thumb_1</td><td style='text-align: center; word-wrap: break-word;'>18.9970</td><td style='text-align: center; word-wrap: break-word;'>1.9609</td><td style='text-align: center; word-wrap: break-word;'>-17.0360</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Thumb_2</td><td style='text-align: center; word-wrap: break-word;'>15.5668</td><td style='text-align: center; word-wrap: break-word;'>5.4748</td><td style='text-align: center; word-wrap: break-word;'>-10.0920</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Thumb_3</td><td style='text-align: center; word-wrap: break-word;'>10.6571</td><td style='text-align: center; word-wrap: break-word;'>10.4247</td><td style='text-align: center; word-wrap: break-word;'>-0.2324</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Thumb_4</td><td style='text-align: center; word-wrap: break-word;'>7.8074</td><td style='text-align: center; word-wrap: break-word;'>15.1095</td><td style='text-align: center; word-wrap: break-word;'>7.3021</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Index_1</td><td style='text-align: center; word-wrap: break-word;'>10.1454</td><td style='text-align: center; word-wrap: break-word;'>11.4248</td><td style='text-align: center; word-wrap: break-word;'>1.2794</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Index_2</td><td style='text-align: center; word-wrap: break-word;'>8.6302</td><td style='text-align: center; word-wrap: break-word;'>20.2237</td><td style='text-align: center; word-wrap: break-word;'>11.5934</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Index_3</td><td style='text-align: center; word-wrap: break-word;'>10.9266</td><td style='text-align: center; word-wrap: break-word;'>22.7617</td><td style='text-align: center; word-wrap: break-word;'>11.8351</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Index_4</td><td style='text-align: center; word-wrap: break-word;'>10.5297</td><td style='text-align: center; word-wrap: break-word;'>21.4762</td><td style='text-align: center; word-wrap: break-word;'>10.9464</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Middle_1</td><td style='text-align: center; word-wrap: break-word;'>10.5208</td><td style='text-align: center; word-wrap: break-word;'>12.7205</td><td style='text-align: center; word-wrap: break-word;'>2.1997</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Middle_2</td><td style='text-align: center; word-wrap: break-word;'>11.5187</td><td style='text-align: center; word-wrap: break-word;'>15.0806</td><td style='text-align: center; word-wrap: break-word;'>3.5619</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Middle_3</td><td style='text-align: center; word-wrap: break-word;'>11.4452</td><td style='text-align: center; word-wrap: break-word;'>14.9854</td><td style='text-align: center; word-wrap: break-word;'>3.5403</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Middle_4</td><td style='text-align: center; word-wrap: break-word;'>10.7450</td><td style='text-align: center; word-wrap: break-word;'>13.7460</td><td style='text-align: center; word-wrap: break-word;'>3.0009</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Ring_1</td><td style='text-align: center; word-wrap: break-word;'>12.6135</td><td style='text-align: center; word-wrap: break-word;'>11.9443</td><td style='text-align: center; word-wrap: break-word;'>-0.6692</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Ring_2</td><td style='text-align: center; word-wrap: break-word;'>11.9614</td><td style='text-align: center; word-wrap: break-word;'>11.3974</td><td style='text-align: center; word-wrap: break-word;'>-0.5640</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Ring_3</td><td style='text-align: center; word-wrap: break-word;'>9.8892</td><td style='text-align: center; word-wrap: break-word;'>11.2968</td><td style='text-align: center; word-wrap: break-word;'>1.4076</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Ring_4</td><td style='text-align: center; word-wrap: break-word;'>11.6317</td><td style='text-align: center; word-wrap: break-word;'>16.0329</td><td style='text-align: center; word-wrap: break-word;'>4.4013</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Pinky_1</td><td style='text-align: center; word-wrap: break-word;'>14.5713</td><td style='text-align: center; word-wrap: break-word;'>11.5837</td><td style='text-align: center; word-wrap: break-word;'>-2.9876</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Pinky_2</td><td style='text-align: center; word-wrap: break-word;'>12.3349</td><td style='text-align: center; word-wrap: break-word;'>12.6961</td><td style='text-align: center; word-wrap: break-word;'>0.3613</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Pinky_3</td><td style='text-align: center; word-wrap: break-word;'>7.6028</td><td style='text-align: center; word-wrap: break-word;'>14.3917</td><td style='text-align: center; word-wrap: break-word;'>6.7889</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Pinky_4</td><td style='text-align: center; word-wrap: break-word;'>3.4800</td><td style='text-align: center; word-wrap: break-word;'>18.7052</td><td style='text-align: center; word-wrap: break-word;'>15.2252</td></tr></table>

---

<div style="text-align: center;">Table 14: Per-joint error change before and after root alignment on case of row #1 of Fig. 21. Negative  $\Delta$ means the error decreases after alignment.</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'>Joint</td><td style='text-align: center; word-wrap: break-word;'>Before</td><td style='text-align: center; word-wrap: break-word;'>After</td><td style='text-align: center; word-wrap: break-word;'>$\Delta$ (After - Before)</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Wrist</td><td style='text-align: center; word-wrap: break-word;'>19.4289</td><td style='text-align: center; word-wrap: break-word;'>0.0000</td><td style='text-align: center; word-wrap: break-word;'>-19.4289</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Thumb_1</td><td style='text-align: center; word-wrap: break-word;'>22.7313</td><td style='text-align: center; word-wrap: break-word;'>4.1163</td><td style='text-align: center; word-wrap: break-word;'>-18.6150</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Thumb_2</td><td style='text-align: center; word-wrap: break-word;'>17.3230</td><td style='text-align: center; word-wrap: break-word;'>4.0984</td><td style='text-align: center; word-wrap: break-word;'>-13.2246</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Thumb_3</td><td style='text-align: center; word-wrap: break-word;'>12.3879</td><td style='text-align: center; word-wrap: break-word;'>7.0410</td><td style='text-align: center; word-wrap: break-word;'>-5.3469</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Thumb_4</td><td style='text-align: center; word-wrap: break-word;'>12.5547</td><td style='text-align: center; word-wrap: break-word;'>16.0392</td><td style='text-align: center; word-wrap: break-word;'>3.4845</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Index_1</td><td style='text-align: center; word-wrap: break-word;'>12.2225</td><td style='text-align: center; word-wrap: break-word;'>9.3668</td><td style='text-align: center; word-wrap: break-word;'>-2.8557</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Index_2</td><td style='text-align: center; word-wrap: break-word;'>5.3559</td><td style='text-align: center; word-wrap: break-word;'>21.1668</td><td style='text-align: center; word-wrap: break-word;'>15.8109</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Index_3</td><td style='text-align: center; word-wrap: break-word;'>17.7139</td><td style='text-align: center; word-wrap: break-word;'>32.0996</td><td style='text-align: center; word-wrap: break-word;'>14.3857</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Index_4</td><td style='text-align: center; word-wrap: break-word;'>43.3812</td><td style='text-align: center; word-wrap: break-word;'>57.7669</td><td style='text-align: center; word-wrap: break-word;'>14.3857</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Middle_1</td><td style='text-align: center; word-wrap: break-word;'>8.0081</td><td style='text-align: center; word-wrap: break-word;'>12.1308</td><td style='text-align: center; word-wrap: break-word;'>4.1227</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Middle_2</td><td style='text-align: center; word-wrap: break-word;'>8.1967</td><td style='text-align: center; word-wrap: break-word;'>15.0336</td><td style='text-align: center; word-wrap: break-word;'>6.8368</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Middle_3</td><td style='text-align: center; word-wrap: break-word;'>11.5712</td><td style='text-align: center; word-wrap: break-word;'>19.4476</td><td style='text-align: center; word-wrap: break-word;'>7.8763</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Middle_4</td><td style='text-align: center; word-wrap: break-word;'>15.8597</td><td style='text-align: center; word-wrap: break-word;'>23.5558</td><td style='text-align: center; word-wrap: break-word;'>7.6961</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Ring_1</td><td style='text-align: center; word-wrap: break-word;'>7.3039</td><td style='text-align: center; word-wrap: break-word;'>12.1250</td><td style='text-align: center; word-wrap: break-word;'>4.8211</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Ring_2</td><td style='text-align: center; word-wrap: break-word;'>6.5721</td><td style='text-align: center; word-wrap: break-word;'>17.1794</td><td style='text-align: center; word-wrap: break-word;'>10.6073</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Ring_3</td><td style='text-align: center; word-wrap: break-word;'>10.6023</td><td style='text-align: center; word-wrap: break-word;'>21.6348</td><td style='text-align: center; word-wrap: break-word;'>11.0325</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Ring_4</td><td style='text-align: center; word-wrap: break-word;'>11.6765</td><td style='text-align: center; word-wrap: break-word;'>22.7090</td><td style='text-align: center; word-wrap: break-word;'>11.0325</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Pinky_1</td><td style='text-align: center; word-wrap: break-word;'>9.2459</td><td style='text-align: center; word-wrap: break-word;'>15.0597</td><td style='text-align: center; word-wrap: break-word;'>5.8139</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Pinky_2</td><td style='text-align: center; word-wrap: break-word;'>17.6427</td><td style='text-align: center; word-wrap: break-word;'>23.6320</td><td style='text-align: center; word-wrap: break-word;'>5.9894</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Pinky_3</td><td style='text-align: center; word-wrap: break-word;'>23.3074</td><td style='text-align: center; word-wrap: break-word;'>29.2968</td><td style='text-align: center; word-wrap: break-word;'>5.9894</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Pinky_4</td><td style='text-align: center; word-wrap: break-word;'>29.5787</td><td style='text-align: center; word-wrap: break-word;'>35.5681</td><td style='text-align: center; word-wrap: break-word;'>5.9894</td></tr></table>

<div style="text-align: center;">Table 15: Per-joint error change before and after root alignment on case of row #2 of Fig. 21. Negative  $\Delta$ means the error decreases after alignment.</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'>Joint</td><td style='text-align: center; word-wrap: break-word;'>Before</td><td style='text-align: center; word-wrap: break-word;'>After</td><td style='text-align: center; word-wrap: break-word;'>$\Delta$ (After - Before)</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Wrist</td><td style='text-align: center; word-wrap: break-word;'>8.4425</td><td style='text-align: center; word-wrap: break-word;'>0.0000</td><td style='text-align: center; word-wrap: break-word;'>-8.4425</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Thumb_1</td><td style='text-align: center; word-wrap: break-word;'>10.2311</td><td style='text-align: center; word-wrap: break-word;'>3.5969</td><td style='text-align: center; word-wrap: break-word;'>-6.6342</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Thumb_2</td><td style='text-align: center; word-wrap: break-word;'>4.4493</td><td style='text-align: center; word-wrap: break-word;'>8.8392</td><td style='text-align: center; word-wrap: break-word;'>4.3899</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Thumb_3</td><td style='text-align: center; word-wrap: break-word;'>21.4771</td><td style='text-align: center; word-wrap: break-word;'>29.1681</td><td style='text-align: center; word-wrap: break-word;'>7.6910</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Thumb_4</td><td style='text-align: center; word-wrap: break-word;'>46.5958</td><td style='text-align: center; word-wrap: break-word;'>54.2868</td><td style='text-align: center; word-wrap: break-word;'>7.6910</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Index_1</td><td style='text-align: center; word-wrap: break-word;'>21.6013</td><td style='text-align: center; word-wrap: break-word;'>13.1588</td><td style='text-align: center; word-wrap: break-word;'>-8.4425</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Index_2</td><td style='text-align: center; word-wrap: break-word;'>36.6657</td><td style='text-align: center; word-wrap: break-word;'>28.2232</td><td style='text-align: center; word-wrap: break-word;'>-8.4425</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Index_3</td><td style='text-align: center; word-wrap: break-word;'>38.3033</td><td style='text-align: center; word-wrap: break-word;'>29.8608</td><td style='text-align: center; word-wrap: break-word;'>-8.4425</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Index_4</td><td style='text-align: center; word-wrap: break-word;'>36.8067</td><td style='text-align: center; word-wrap: break-word;'>28.3642</td><td style='text-align: center; word-wrap: break-word;'>-8.4425</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Middle_1</td><td style='text-align: center; word-wrap: break-word;'>13.1708</td><td style='text-align: center; word-wrap: break-word;'>4.7395</td><td style='text-align: center; word-wrap: break-word;'>-8.4313</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Middle_2</td><td style='text-align: center; word-wrap: break-word;'>8.4589</td><td style='text-align: center; word-wrap: break-word;'>6.5191</td><td style='text-align: center; word-wrap: break-word;'>-1.9398</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Middle_3</td><td style='text-align: center; word-wrap: break-word;'>5.3740</td><td style='text-align: center; word-wrap: break-word;'>12.8966</td><td style='text-align: center; word-wrap: break-word;'>7.5227</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Middle_4</td><td style='text-align: center; word-wrap: break-word;'>22.6355</td><td style='text-align: center; word-wrap: break-word;'>30.3265</td><td style='text-align: center; word-wrap: break-word;'>7.6910</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Ring_1</td><td style='text-align: center; word-wrap: break-word;'>3.9240</td><td style='text-align: center; word-wrap: break-word;'>5.6899</td><td style='text-align: center; word-wrap: break-word;'>1.7659</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Ring_2</td><td style='text-align: center; word-wrap: break-word;'>1.6087</td><td style='text-align: center; word-wrap: break-word;'>8.9511</td><td style='text-align: center; word-wrap: break-word;'>7.3424</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Ring_3</td><td style='text-align: center; word-wrap: break-word;'>7.0678</td><td style='text-align: center; word-wrap: break-word;'>14.7588</td><td style='text-align: center; word-wrap: break-word;'>7.6910</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Ring_4</td><td style='text-align: center; word-wrap: break-word;'>18.9014</td><td style='text-align: center; word-wrap: break-word;'>26.5924</td><td style='text-align: center; word-wrap: break-word;'>7.6910</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Pinky_1</td><td style='text-align: center; word-wrap: break-word;'>3.9506</td><td style='text-align: center; word-wrap: break-word;'>9.6210</td><td style='text-align: center; word-wrap: break-word;'>5.6703</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Pinky_2</td><td style='text-align: center; word-wrap: break-word;'>5.8738</td><td style='text-align: center; word-wrap: break-word;'>9.6193</td><td style='text-align: center; word-wrap: break-word;'>3.7455</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Pinky_3</td><td style='text-align: center; word-wrap: break-word;'>10.7508</td><td style='text-align: center; word-wrap: break-word;'>12.5462</td><td style='text-align: center; word-wrap: break-word;'>1.7953</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Pinky_4</td><td style='text-align: center; word-wrap: break-word;'>12.6621</td><td style='text-align: center; word-wrap: break-word;'>19.2628</td><td style='text-align: center; word-wrap: break-word;'>6.6007</td></tr></table>

---

![Image](images/f204b15020a2edd562044e0ca69c4eff2de4720b71c26202abf81fe1d31206d4.jpg)

<div style="text-align: center;">Figure 22: Qualitative results on DexYCB and InterHand2.6M.</div>

---

![Image](images/8efb2615e6595a60ffd13c7783c98381849dbf7d51bbecd4cafc835c889fd986.jpg)

<div style="text-align: center;">Figure 23: Failure cases of DexYCB and InterHand2.6M.</div>