# ScaleHP: Scale-Mediated Optimization of Coupled Errors for Metric-Space Hand Pose Estimation Ruitao Jing $^{1,3,4,*}\dagger$, Xingyu Chen $^{2*}$, Hongyang Li $^{4,5}$, Qing Jiang $^{4,5}$, Yukai Shi $^{1,4}$, Lei Zhang $^{3,4,5}$ $^{1}$Tsinghua University $^{2}$Zhongguancun Academy $^{3}$Visincept $^{4}$International Digital Economy Academy (IDEA Research) $^{5}$South China University of Technology ## Abstract In this paper, we present ScaleHP, a unified framework that explicitly represents per-instance metric scale to resolve the coupled errors in calibrated camera-space hand pose estimation. Under the common root-relative-to-global paradigm, camera-space accuracy depends jointly on relative geometry, root localization, and their scale-dependent composition. ScaleHP treats scale as the shared interface among these terms rather than optimizing them in isolation. Its metric-aware 2D–3D Transformer decoder introduces a dedicated scale token that exchanges information with sparse, semantically indexed 2D/3D joint queries, allowing global metric reasoning and local joint geometry to mutually refine one another. A parameter-free calibrated module then recovers root translation from the predicted unit-scale joints, metric scale, and known camera intrinsics. ScaleHP demonstrates state-of-the-art CS-MPJPE on FreiHand (35.8 mm) and, under benchmark-specific training protocols, state-of-the-art PA-MPJPE on FreiHand, Dex YCB, and HO3Dv3 (5.0/4.6/5.9 mm), showing that improved global metric localization is attained together with improved local articulation. Ablations further show that scale-pose interaction improves both camera-space accuracy and relative depth geometry. ## 1 Introduction Hand pose estimation (HPE) is a basic perception primitive for interaction, manipulation, and teleoperation. In VR/AR, the hand must be placed consistently with the surrounding scene; in embodied AI and robotic teleoperation, a controller needs the hand's physical location and dimensions rather than only its articulation (Karim et al. 2025; Nostadt et al. 2020; Toet et al. 2020). These applications motivate metric camera-space hand pose, where the output must preserve both the articulated configuration and the hand's absolute position and size in the calibrated camera coordinate system. To obtain camera-space pose, a common paradigm first predicts root-relative hand geometry and then places it globally. Writing $\mathbf{Q}^{\mathrm{cam}} = s(\mathbf{J}^{\mathrm{rel}} + \mathbf{t}^{\mathrm{norm}})$ exposes three coupled error sources: (a) root-relative geometry in $\mathbf{J}^{\mathrm{rel}}$, (b) root localization in $\mathbf{t}^{\mathrm{norm}}$, and (c) the cascade produced when the metric scale s multiplies the composed term. Existing methods address these terms with different compromises. CMR (Chen et al. 2021a) follows the two-step recipe most explicitly: it predicts root-relative geometry and registers the root to image evidence, leaving positioning decoupled from the representation that produced the geometry. HandOS (Chen et al. 2025) brings 2D/3D queries and a translation variable into a one-stage decoder, but does not organize translation as an independently supervised metric-distance factor; reprojection alone cannot resolve the absolute metric gauge. HandDGP (Valassakis and Garcia-Hernando 2024) reconnects the stages with differentiable global positioning, yet its analysis reports a trade-off between global positioning and relative-geometry accuracy. NVF (Huang et al. 2023) bypasses the factorization with dense camera-frustum voting, using a substantially different sampling-based formulation. Overall, most prior systems primarily reduce error source (a), namely root-relative geometry. Error source (b) is commonly delegated to a learned translation or an external depth cue, which remains entangled with focal length and camera distance. Error source (c), the term most capable of widening camera-space gaps, is either left implicit or improved only at the expense of one of the first two terms. A related tension appears in metric-depth prediction: ScaleDepth (Zhu et al. 2026) and MoGe-2 (Wang et al. 2025) deliberately separate global metric quantities from local geometry, and their designs make clear that the two predictions can compete. This raises the central question of our work: can the three error sources be optimized together, with global metric reasoning and local hand geometry improving rather than undermining one another? We argue that the missing interface is an explicit instance scale. Under calibrated perspective projection, the same 2D projections can arise from different hand sizes paired with different depths. Without an explicit scale variable, existing camera-space formulations—including MANO-based parametric pipelines and other joint or mesh regressors—often misestimate metric scale during global positioning, or incorrectly compensate for that error in the recovered root translation. The absence of a separately supervised scale --- ![Image](images/1ec634072ec0b8a9eeac2d8d4dcfa1e2d128a705e784d0f460dfd58240355e60.jpg)
Figure 1: ScaleHP places metric scale at the center of calibrated camera-space hand pose estimation. Unlike prior approaches that leave scale implicit, supply it externally, or decouple it from pose learning, ScaleHP predicts per-instance metric scale jointly with unit-scale root-relative geometry and uses analytically recovered normalized translation to reconstruct the hand in camera space.
representation then makes this coupling hard to diagnose. Metric-depth methods instead estimate dense scene geometry and global scale from image-wise representations. Hand pose does not necessarily require a dense scene representation: an individual hand pose can be described by sparse, semantically indexed, and geometrically constrained 2D layouts and root-relative 3D joints. These representations can provide scale evidence without a dense point map or an external monocular-depth branch. Scale is therefore a natural bridge between local articulation and global camera-space coordinates, while remaining a learned statistical prior under metric supervision rather than a claim that monocular projection uniquely identifies hand size. Based on this insight, we present ScaleHP, a unified framework for calibrated metric-space HPE (Fig. 2). Its metric-aware 2D–3D Transformer decoder introduces a hand-specific scale token that exchanges information with sparse 2D/3D joint queries. Joint queries provide localized scale evidence, while scale supervision reaches later joint-query layers instead of acting only as post-hoc calibration. The decoder predicts 2D joints, unit-scale root-relative 3D joints, and instance metric scale from frozen-detector features. Given these outputs and known intrinsics, a parameter-free least-squares module recovers normalized root translation and composes metric camera-space joints. ScaleHP thereby coordinates (a), (b), and their scale-dependent interaction (c) without dense scene-depth prediction or iterative registration. The contributions of this work are: - Scale-mediated camera-space formulation. We expose relative geometry, root localization, and their scale-dependent composition as the three coupled sources of camera-space error, and identify per-instance metric scale as the interface between local hand structure and global coordinates. - Query-coupled ScaleHP. We present a unified framework for metric-space HPE whose scale token interacts with sparse semantic 2D/3D joint queries, allowing global scale supervision and local joint geometry to mutually refine one another. By jointly optimizing the learned relative-geometry and scale factors while calibrated positioning resolves root localization under known intrinsic scs, ScaleHP coordinates all three error sources in one pipeline. - State-of-the-art metric and relative accuracy. ScaleHP obtains the best listed FreiHand CS-MPJPE of 35.8 mm. Under benchmark-specific training protocols, it also achieves state-of-the-art PA-MPJPE of 5.0, 4.6, and 5.9 mm on FreiHand, DexYCB, and HO3Dv3, respectively, showing that global localization and local articulation improve together. ## 2 Related Works Camera-space HPE. Camera-space HPE must recover absolute placement and physical scale in addition to articulation. Iqbal et al. (Iqbal et al. 2018) lift a normalized pose with a supplied or mean bone length. CMR and related registration-based systems recover global position from 2D evidence and root-relative geometry (Chen et al. 2021a, 2022; Park et al. 2022; Xu et al. 2023; Wang et al. 2023), but metric scale remains implicit in their geometry and translation. NVF (Huang et al. 2023) instead supervises dense voting in a camera frustum. HandDGP (Valassakis and Garcia-Hernando 2024) makes global positioning differentiable but reports a tension with root-relative accuracy, whereas HandOS (Chen et al. 2025) jointly decodes geometry and translation without exposing an independently supervised metric gauge. WiLoR improves hand localization, and other systems exploit forearm or interaction context (Potamias et al. 2025; Millerdurai et al. 2026; Hasson et al. 2020; Spurr et al. 2020). These methods advance global positioning, but they do not provide an explicit instance-scale representation through which metric supervision can coordinate with relative hand geometry. Consequently, an aligned-pose gain need not reduce absolute --- error: plausible projections admit different hand-size–depth pairs, and implicit metric variables can compensate for one another during training. Root-relative HPE. Parametric, heatmap, and model-free methods respectively regress MANO parameters, discretize 2.5D/3D coordinates, or directly regress joints and vertices (Romero, Tzionas, and Black 2017b; Baek, Kim, and Kim 2019; Hasson et al. 2019; Boukhayma, Bem, and Torr 2019; Iqbal et al. 2018; Moon and Lee 2020; Moon et al. 2020; Chen et al. 2022; Ge et al. 2019; Kulon et al. 2020). Transformers and detector-based systems introduce global context and structured queries (Lin, Wang, and Liu 2021b,a; Pavlakos et al. 2024; Dong et al. 2024; Shi et al. 2022; Yang et al. 2023; Sun et al. 2024), while specialized designs address efficiency, deformation, and occlusion (Yoshiyasu 2023; Zhou et al. 2024; Saleem et al. 2025; Lu, Gou, and Li 2024; Ren et al. 2025). These methods achieve strong aligned accuracy, but alignment removes translation and common scale; it therefore does not test whether local geometry remains accurate when composed into metric camera space. Metric scale learning. Dense metric prediction couples a single global gauge to every local depth or point-map element. ScaleDepth (Zhu et al. 2026) explicitly separates scene-scale queries from normalized relative-depth bin queries so that the dense relative branch can model structure without absorbing cross-scene depth-range variation. MoGe2 (Wang et al. 2025) reaches the same conclusion more directly: its entangled metric point-map baseline is suboptimal, while decoupling a global CLS-derived scale from an affine-invariant point map better preserves relative geometry. Other metric-depth methods use adaptive bins, canonical-camera rays, or scale-aware dense representations (Bhat, Alhashim, and Wonka 2021; Li et al. 2024; Yin et al. 2023; Guizilini et al. 2023; Song et al. 2025). The shared lesson is not merely that scale can be predicted, but that folding a global scale into a high-dimensional local representation can make metric and relative objectives compete: the global quantity must determine metric extent, yet entangling it with dense local elements can distort their relative geometry. Hand pose provides a more compact interface because its geometry admits a sparse, semantically indexed joint representation. ScaleHP exploits this property by keeping scale and unit-scale joint tokens distinct while allowing them to interact, so global metric reasoning and local geometric learning can be optimized jointly. ## 3 Method ### 3.1 Why Scale Matters Table 1 contrasts how camera-space methods represent and expose metric scale. We use scale feedback to pose for the direction in which metric/global supervision affects pose features. Prior choices leave scale supplied externally or distributed across high-dimensional geometry and translation. Let the ground-truth camera-space pose be $\mathbf{Q} = s(\mathbf{J} + \mathbf{t})$, where $\mathbf{J}$ is unit-scale root-relative geometry and $\mathbf{t}$ is normalized root translation. With $\hat{s} = s + \Delta s$, $\hat{\mathbf{J}} = \mathbf{J} + \Delta\mathbf{J}$, and $\hat{\mathbf{t}} = \mathbf{t} + \Delta\mathbf{t}$, the exact error is $$ \begin{aligned}\hat{\mathbf{Q}}-\mathbf{Q}=&\underbrace{s\Delta\mathbf{J}}_{(a)relative~geometry}+\underbrace{s\Delta\mathbf{t}}_{(b)root~localization}\\&+\underbrace{\Delta s(\mathbf{J}+\mathbf{t})+\Delta s(\Delta\mathbf{J}+\Delta\mathbf{t})}_{(c)scale-mediated~coupling}.\end{aligned} $$ Since $\|\mathbf{t}\| \gg \|\mathbf{J}\|$ in camera space, $\|\Delta s(\mathbf{J} + \mathbf{t})\| \approx |\Delta s|\|\mathbf{t}\|:$ even a small scale error can induce a large depth displacement. Moreover, the last term couples global scale to both local geometry and root positioning. This motivates exposing s as a separately supervised instance variable rather than leaving it implicit in a dense pose or translation representation. Explicit scale prediction. We factor each metric joint as $\mathbf{Q}^{cam} = s(\mathbf{J}^{rel} + \mathbf{t}^{norm})$, where $\mathbf{J}^{rel}$ is root-relative and unit-scale. Perspective correspondences recover $\mathbf{t}^{norm}$ under known intrinsic, but cannot determine s: scaling both hand and translation leaves the projection unchanged. Hence s is learned from metric supervision and remains a data-driven monocular prior. Let $\mathbf{J}^{M*}$ denote the J ground-truth joint annotations in metric camera coordinates and let r index the root. The network does not regress $\mathbf{J}^{M*}$ directly. Instead, it receives the two targets induced by these annotations: projected pixel joints $\mathbf{J}^{2D*}$ and unit-scale root-relative joints $\mathbf{J}^{3D*}$. Motivated by their stable long-axis anatomy (Varu et al. 2015; Rastogi, Nagesh, and Yoganarasimha 2008; Manning 2002), let $\{l_i\}_{i=0}^{N-1}$ be the metric bone lengths in $\mathbf{J}^{M*}$ along the index, middle, and ring fingers (IMR). We define $$ \begin{align*}s^{*}&=\frac{1}{N}\sum_{i=0}^{N-1}l_{i},\\\mathbf{J}^{2D*}&=\pi(\mathbf{J}^{M*},\mathbf{K}_{cam}),\\\mathbf{J}^{3D*}&=\frac{\mathbf{J}^{M*}-\mathbf{J}_{r}^{M*}}{s^{*}}.\end{align*} $$ Thus the architecture outputs $\mathbf{J}^{2D} \to \mathbf{J}^{2D*}$, $\mathbf{J}^{3D} \to \mathbf{J}^{3D*}$, and $s \to s^*$; the analytical transfer recomposes them into metric camera space. While this target focuses on IMR fingers, it does not rule out scale evidence from other fingers; the ablation study and supplementary material provide further details. A scale token aggregates joint layout, unit-scale geometry, and image appearance, allowing metric supervision to affect pose features before calibrated positioning. ### 3.2 Architecture Overview. Figure 2 summarizes ScaleHP, which comprises a frozen DETR-style detector, a jointly trained metric-aware 2D–3D decoder, and a parameter-free analytical module. A scale token is updated jointly with hand-related 2D semantic and 3D geometric queries and cross-attends to image features. The decoder predicts an instance metric gauge alongside unit-scale pose, after which the analytical module solves normalized translation under known intrinsic. The complete system performs one-pass inference without iterative registration. ---
MethodMetric-scale sourceScale feedback to posePose representation
Iqbal et al.supplied/mean bone lengthpost-hoc2.5D joint heatmaps
CMRimplicit in root-relative geometrypost-hocsemantic feature aggregation
NVFdirect/test referencedirect camera-spacedense frustum voting field
HandDGPimplicit in root-relative geometrydifferentiable (trade-off)image-to-geometry regression
HandOSimplicit in geometry and translationone-way pose-to-translation2D-joint and 3D-vertex queries
EgoForcehand-arm shape/contextjoint camera-spacehand-arm parameter features
ScaleHPpredicted instance scalarbidirectional scale-posesparse joint and scale tokens
Table 1: Metric-scale handling in representative camera-space HPE. “Scale feedback to pose” summarizes how metric/global objectives communicate with pose features. The final column lists the intermediate pose representation, in which NVF’s voting field is dense.
![Image](images/927c1528359b4f1506b50a41dcb9c8076bbffbdfffb0b3f04d74857ef9ed018e.jpg)
Figure 2: Overview of ScaleHP. A frozen detector supplies image and instance features. The jointly trained decoder predicts 2D joints, unit-scale root-relative 3D joints, and a per-instance metric scale through interacting joint and scale tokens. A parameter-free calibrated solver then recovers normalized translation and composes metric camera-space joints.
DETR-Like Detector. Frozen Grounding DINO (Liu et al. 2024) with prompt hand fuses visual/text features and iteratively updates content queries and 2D references through deformable decoding (Zhu et al. 2021): $$ \mathbf{F}_{e}=\mathcal{E}(\mathcal{B}_{v}(\mathbf{I}),\mathcal{B}_{t}(“hand”).) $$ $$ (\mathbf{F}_{{G D}},\mathbf{Q},\mathbf{R},\mathbf{B},\mathbf{C})=\mathcal{D}_{{d e t}}(\mathbf{F}_{e}). $$ Here $\mathbf{F}_{GD}$ contains multi-scale features and $(\mathbf{Q}, \mathbf{R}, \mathbf{B}, \mathbf{C})$ are instance queries, references, boxes, and confidences. After score filtering and NMS, the retained set is $\{(\tilde{\mathbf{Q}}_k, \tilde{\mathbf{R}}_k, \tilde{\mathbf{B}}_k)\}_{k=1}^K$. The hand decoder processes all K instances; the complete interface is in the supplementary material. Metric 2D-3D Decoder. Each retained instance is expanded into $J = 21$ semantically indexed 2D and 3D joint queries using learnable embeddings $\mathbf{E}_{J}$; a hand-level scale token $\mathbf{q}^{s}$ is initialized from the same instance context. We suppress the instance index k below because the same decoder is applied independently to every retained hand: $$ \mathbf{Q}_{0}^{2D}=\mathbf{Q}_{0}^{3D}=\tilde{\mathbf{Q}}+\mathbf{E}_{J},\qquad\mathbf{q}_{0}^{s}=\tilde{\mathbf{Q}}. $$ At layer l, the scale token and both joint streams exchange information through global self-attention: $$ (\mathbf{q}_{l}^{s\prime},\mathbf{Q}_{l}^{2D},\mathbf{Q}_{l}^{3D})=\mathrm{S A}_{l}(\mathbf{q}_{l-1}^{s},\mathbf{Q}_{l-1}^{2D},\mathbf{Q}_{l-1}^{3D}). $$ This interaction is reciprocal: the scale token aggregates all joint layouts and normalized structures, and every joint attends to scale-supervised context. Repeated layers therefore refine both directions before prediction. Each stream then samples multi-scale appearance around its reference points: $$ \begin{aligned}\mathbf{q}_{l}^{s}&=\operatorname{Def}\operatorname{Attn}_{l}^{s}(\mathbf{q}_{l}^{s\prime},\mathbf{F}_{GD},\mathbf{R}_{l-1}^{s}),\\\mathbf{Q}_{l}^{d}&=\operatorname{Def}\operatorname{Attn}_{l}^{d}(\mathbf{Q}_{l}^{d},\mathbf{F}_{GD},\mathbf{R}_{l-1}^{d}),\\\mathbf{R}_{l}^{d}&=\mathbf{R}_{l-1}^{d}+\operatorname{FFN}_{R}^{d}(\mathbf{Q}_{l}^{d}),\end{aligned} $$ for $d \in \{2D, 3D\}$; the scale reference uses $\mathbf{R}_l^s = \mathbf{R}_{l-1}^s + \mathrm{FFN}_R^s(\mathbf{q}_l^s)$. Deformable attention adds local appearance and occupancy evidence, while reference refinement follows the evolving estimates. Finally, $$ \begin{aligned}\mathbf{J}^{2D}=\mathrm{FFN}_{2D}(\mathbf{Q}_{L}^{2D}),\quad&\mathbf{J}^{3D}=\mathrm{FFN}_{3D}(\mathbf{Q}_{L}^{3D}).\\ s=\mathrm{FFN}_{s}(\mathbf{q}_{L}^{s}).\end{aligned} $$ predict pixel coordinates, unit-scale root-relative joints, and the metric scalar supervised by Eq. 2, respectively. Transfer to Real Metric. The training-free analytical solver receives $\mathbf{J}^{2D}$, $\mathbf{J}^{3D}$, s, and known intrinsic. Let p be the root joint and let $\mathbf{A}$ stack $[u_j - u_p, v_j - v_p]^\top$ for $j \neq p$. Perspective projection gives the corresponding right-hand side $$ \mathbf{b}=\left[\begin{matrix}{f_{x}x_{j}-(u_{j}-c_{x})z_{j},}\\ {f_{y}y_{j}-(v_{j}-c_{y})z_{j}}\\ \end{matrix}\right]_{j\neq p}^{\top}. $$ The normalized translation is recovered without scale as $$ t_{z}^{n o r m}=\mathbf{A}^{\dagger}\mathbf{b}, $$ $$ t_{x}^{n o r m}=\frac{u_{p}-c_{x}}{f_{x}}t_{z}^{n o r m},\qquad t_{y}^{n o r m}=\frac{v_{p}-c_{y}}{f_{y}}t_{z}^{n o r m}. $$ --- Then $\mathbf{Q}^{cam} = s(\mathbf{J}^{3D} + \mathbf{t}^{norm})$ composes the metric prediction without learned translation regression or iterative registration. The supplementary material gives the full projection derivation. ### 3.3 Loss Functions ScaleHP is jointly optimized with 2D, unit-scale 3D, scale, and projection supervision. For ground truth denoted by (*), the direct terms are $$ \begin{align*}\mathcal{L}^{2D}&=\|\mathbf{J}^{2D}-\mathbf{J}^{2D*}\|_{1},\\\mathcal{L}^{3D}&=\|\mathbf{J}^{3D}-\mathbf{J}^{3D*}\|_{1},\\\mathcal{L}^{s}&=(s-s^{*})^{2}.\end{align*} $$ with an additional OKS loss on 2D joints. Following Chen et al. (2025), the differentiable solve $\hat{t}^{norm} = \mathrm{LS}(\mathbf{J}^{2D}, \mathbf{J}^{3D}, \mathbf{K}_{cam})$ supplies weak reprojection supervision: $$ \begin{aligned}\hat{\mathbf{J}}^{proj}&=\pi(\mathbf{J}^{3D}+\hat{\mathbf{t}}^{norm},\mathbf{K}_{cam}),\\\mathcal{L}_{2D}^{proj}&=\left\|\hat{\mathbf{J}}^{proj}-\mathbf{J}^{2D*}\right\|_{1},\\\mathcal{L}_{OKS}^{proj}&=1-OKS(\hat{\mathbf{J}}^{proj},\mathbf{J}^{2D*}).\end{aligned} $$ ## 4 Experiments The solver receives no separate ground-truth camera-space coordinate loss; joint learning refers to scale/pose representations and reprojection, not full camera-space supervision through every component. Setup. We evaluate FreiHand (Zimmermann et al. 2019), DexYCB (Chao et al. 2021), and HO3Dv3 (Hampali et al. 2020); HInt, COCO-WholeBody, CompHand, H2O3D and OneHand10K appear only in the combined-data protocol. Frozen Grounding DINO 1.5 processes full images at a 1280-pixel long edge. We train 45 epochs with Adam, batch size 16, $10^{-4}$ initial learning rate, and cosine decay on eight A100 GPUs. CS-MPJPE is unaligned camera-space error, R-MPJPE root-aligns the prediction to the ground-truth root, and PA-MPJPE applies a similarity Procrustes alignment, following established camera-space and aligned-pose protocols (Huang et al. 2023; Valassakis and Garcia-Hernando 2024; Millerdurai et al. 2026). Training details are supplementary. ### 4.1 Main Results CS-MPJPE is primary because it applies neither alignment nor post-hoc scale correction. We separate the evaluation into an in-dataset FreiHand comparison and cross-dataset transfer. Table 2 reports results from the original FreiHand protocols; Table 3 instead applies the same FreiHand-trained checkpoint to DexYCB and HO3Dv3 because baseline methods do not provide compatible checkpoints trained on those target datasets. On FreiHand, ScaleHP reduces CS-MPJPE from the best listed 42.4 mm to 35.8 mm (15.6%) while tying the best PA-MPJPE of 5.0 mm. DexYCB and HO3Dv3 in Table 3 are transfer tests, not target-dataset retraining. CMR and Hand-DGP lack standalone scale outputs, so their rows receive an external test-set-optimal dataset scalar; this oracle is needed
MethodCS-MPJPE $\downarrow$PA-MPJPE $\downarrow$
CMR-PG48.86.9
HandDGP46.37.4
MobRecon50.25.7
NVF42.4-
METRO-6.7
MaskHand-5.5
Hamba-5.7
HandOS-5.0
ScaleHP35.85.0
Table 2: FreiHand comparison under the protocols reported by the respective methods (mm). CS-MPJPE measures unaligned camera-space accuracy; PA-MPJPE applies Procrustes alignment. NVF's reported 42.4 mm uses the benchmark reference-bone scale (47.2 mm without it). ScaleHP uses RGB and known intrinsic without test-time ground-truth scale.
MethodDexYCB $\downarrow$HO3Dv3 $\downarrow$
CMR + oracle global scale183.2152.3
HandDGP + oracle global scale222.1132.6
ScaleHP136.350.7
Table 3: Cross-dataset CS-MPJPE using FreiHand-trained checkpoints (mm). The oracle global scale is a test-set-optimal dataset-level rescaling introduced only to give CMR and HandDGP, which do not expose an instance scale, a favorable transfer calibration. ScaleHP uses neither target-dataset training nor test-time ground-truth scale.
to avoid penalizing them for a dataset-level gauge mismatch when transferring from FreiHand. Even with that favorable calibration, ScaleHP reduces CS-MPJPE over the strongest available baseline by 25.6% on DexYCB and 61.8% on HO3Dv3. The within-dataset analyses below separate the three error sources. Qualitative Results. Figure 3 compares four FreiHand samples with metric GT. The metric space and top views respectively reveal absolute placement and depth offsets. ScaleHP follows GT more closely than HandDGP and CMR, particularly along global depth; additional views are in the supplementary material. Figure 4 shows four in-the-wild groups under occlusion, object interaction, and viewpoint changes. Although these images lie outside the training benchmarks, ScaleHP produces consistent metric reconstructions, indicating strong generalization to unconstrained, out-of-dataset imagery. ### 4.2 The Alleviation of the Three Error Sources In this subsection, we demonstrate that ScaleHP jointly optimizes the three coupled error sources: relative geometry, root localization, and their scale-mediated coupling. All diagnostics use a consistent checkpoint protocol. For each dataset, CS-MPJPE, aligned MPJPE, axis-wise error, and scale error are computed from the same predictions of the same dataset-specific ScaleHP checkpoint. They are distinct from --- ![Image](images/422790e252e8151ec196f0d7bcab3c16813855522875454bd78d1c14521708b4.jpg)
Figure 3: Main FreiHand camera-space comparisons on four samples. Yellow: GT; blue: ScaleHP; green: HandDGP; red: CMR. The metric space and top views show that ScaleHP remains closest to GT in absolute translation.
the explicitly cross-dataset protocol in Table 3; no aligned and camera-space metrics are selected from different checkpoints within a dataset. Relative Geometry (a). Table 4 tests whether absolute metric reasoning compromises articulation after global factors are removed. ScaleHP ties the best FreiHand PA-MPJPE (5.0 mm), reduces DexYCB R-MPJPE and PA-MPJPE to 10.3 and 4.6 mm, and reduces HO3Dv3 PA-MPJPE to 5.9 mm. Thus the camera-space gains in Tables 2 and 3 do not arise by trading away root-relative geometry. Because Procrustes alignment removes translation, rotation, and common scale, these results measure articulated geometry rather than a favorable metric composition. The result is therefore complementary to CS-MPJPE: scale-aware learning improves the local representation that is later lifted into camera space, rather than only correcting its final global gauge.
MethodFreiHandDexYCBHO3Dv3
PA-MPJPE$\downarrow$ R-MPJPE$\downarrow$ PA-MPJPE$\downarrow$ PA-MPJPE$\downarrow$ PA-MPJPE
CMR-PG6.9----
I2L-MeshNet7.4----
HandDGP7.4----
MobRecon5.714.26.4--
METRO6.715.27.0--
HandOccNet-14.05.8--
H2ONet-14.05.7--
Deformer-13.65.2--
Zhou et al.-12.45.5--
TI-Net-16.84.9--
MaskHand5.511.75.07.0-
Hamba5.7--6.9-
HandOS5.0-5.26.8-
ScaleHP5.010.34.65.9-
Table 4: Aligned/root-relative accuracy (mm). PA-MPJPE applies Procrustes alignment; R-MPJPE root-aligns translation only.
SettingPA-MPJPE $\downarrow$X error $\downarrow$Y error $\downarrow$Z error $\downarrow$
No scale-token5.62.32.43.5
ScaleHP5.02.22.23.0
Table 5: Scale-token regularization on FreiHand (mm). The first setting omits scale-token interaction; its global metric composition uses a fixed dataset scalar. The X/Y/Z columns are root-relative axis errors from the same prediction dump as PA-MPJPE.
The controlled analysis in Table 5 identifies how the scale token improves that geometry. Removing the token degrades every relative component, with the largest change on the depth axis: Z-axis error increases by 0.5 mm, versus 0.1 and 0.2 mm on the X and Y axes. Because PA-MPJPE removes the global metric gauge, this result cannot be explained by a better final rescaling. It shows that scale supervision acts as a regularizer during joint 2D–3D lifting, reducing the relative-depth component of error source (a). This anisotropic improvement is consequential under perspective projection: relative-depth error is the component most readily confounded with normalized root depth during metric composition. Global Localization (b). Because $\|t\|\gg\|J\|$, root-depth error can remain visually hidden under accurate reprojection. Fig. 3 holds image evidence fixed and exposes this error in metric/top views: all four inputs permit plausible image-plane overlays, but the green HandDGP and red CMR predictions retain visible root-depth offsets. The blue ScaleHP roots and joint sets remain closest to yellow GT along the camera-depth direction. These views therefore diagnose error source (b), which cannot be inferred from reprojection alone; CS-MPJPE then measures its combined effect with (a) and (c). --- ![Image](images/f3e60f558a7efffea27716cb18743288d1625788806ce9d330befddaab68dbe6.jpg)
Figure 4: Qualitative Results of ScaleHP on In-the-Wild Images. Each panel shows input, box and pose, metric 3D, and top view from left to right. ScaleHP remains consistent under hand–object and hand–hand interaction, viewpoint changes, and occlusion; no metric ground truth is available.
Training schedulePA-MPJPE $\downarrow$CS-MPJPE $\downarrow$Scale $L_1$ error $(10^{-3} \text{ m}) \downarrow$
Delayed introduction5.239.52.06
Joint5.035.81.94
Table 6: Scale-token introduction on FreiHand (mm). Delayed introduction first pre-trains without the token, then fine-tunes the entire interactive decoder after inserting it; all token-to-token attention is enabled. Joint training includes the token from initialization.
Scale-Mediated Coupling (c). The preceding results establish that the scale token benefits local geometry and global placement together. Table 6 tests whether this benefit requires scale to participate while the pose representation is learned. Both schedules enable full attention and update all decoder parameters during fine-tuning; only the time at which scale enters differs. Joint training improves PA-MPJPE by 0.2 mm, CS-MPJPE by 3.7 mm, and scale error by 5.8%. The agreement across relative pose, camera-space pose, and scale supports early scale-pose co-adaptation as a mechanism for reducing error source (c). In particular, the camera-space gain is not obtained by sacrificing aligned pose, unlike the relative-global trade-off motivating Sec. 3.1. The qualitative results make ScaleHP's improved global localization directly visible; further quantitative comparisons are provided in the supplementary material. ### 4.3 Ablation Study Scale-token coupling. Table 7 isolates the effect of an instance scale from a favorable dataset-level calibration. Even after the no-token model receives an oracle global scale, its CS-MPJPE is higher by 8.5, 10.2, and 8.6 mm on FreiHand, DexYCB, and HO3Dv3, respectively. A fixed scalar can correct an average dataset gauge but cannot represent the scale variation that composes with each hand's relative geometry and normalized root translation. The result therefore directly
SettingFreiHand CS-MPJPEDexYCB CS-MPJPEHO3Dv3 CS-MPJPE
No token + oracle scale44.340.242.0
ScaleHP35.830.033.4
Table 7: Per-instance scale-token coupling (mm). Each dataset uses its own checkpoint; the no-token setting receives an oracle dataset-level global scale.
supports the need for per-instance scale in the scale-mediated term. The global scalar is selected separately on every target test set, making this an intentionally favorable calibration for the no-token baseline rather than a deployment setting; the remaining gap is therefore instance-level rather than dataset-level scale variation. IMR scale target. The predicted scale $L_1$ errors are 1.94, 0.56, and $1.02 \times 10^{-3}$ m on FreiHand, DexYCB, and HO3Dv3. IMR has a higher between-subject-to-within-subject F-ratio than the thumb/little-finger target (6.17 versus 2.95) and stronger rank consistency with full-hand size (0.982 versus 0.868), supporting stable scalar supervision. In other words, IMR retains between-subject size variation while suppressing the finger-specific variation that makes a single short bone a noisier proxy. Background intervention. White-background FreiHand evaluation changes CS-MPJPE from 35.85 to 37.22 mm (3.8%, within 5%). Together with scale-token interaction with $\mathbf{J}^{2D}$ and $\mathbf{J}^{3D}$, this supports hand-specific evidence rather than a full-image scene cue. The paired distribution and limitations are reported in the supplement; this intervention does not establish strict background invariance. ## 5 Conclusion We introduced ScaleHP, a unified framework for calibrated metric-space hand pose estimation. Its central contribution is an explicit per-instance metric scale that interfaces unit-scale local geometry with global positioning. A dedicated scale to --- ken interacts with 2D/3D joint queries, and a parameter-free calibrated solver recovers normalized translation. ScaleHP achieves state-of-the-art camera-space accuracy on FreiHand and PA-MPJPE on FreiHand, DexYCB, and HO3Dv3, showing that global localization and local articulation improve together. Its transfer results require neither target-dataset training nor oracle scale, while ablations support instance-conditioned, joint scale-pose learning. --- ## A Detailed Architecture ### A.1 DETR-Like Detector ScaleHP uses a frozen Transformer-based detector to localize hands from a text prompt. We adopt Grounding DINO (Liu et al. 2024) as the representative implementation (Grounding DINO 1.5 in our experiments), while the framework is compatible with DETR-like or open-vocabulary detectors that provide multimodal features and query-based localization. Given an RGB image I and the generic prompt "hand", the encoder $\mathcal{E}$ fuses the outputs of the visual and text backbones: $$ \mathbf{F}_{e}=\mathcal{E}(\operatorname{Backbone}_{v}(\mathbf{I}),\operatorname{Backbone}_{t}(“hand”).). $$ The Transformer decoder $\mathcal{D}$ iteratively refines content queries $\mathbf{Q}_i$ and their 2D reference points $\mathbf{R}_i$. Deformable attention (Zhu et al. 2021) lets each query sample the encoded multi-scale features around its reference point and thereby capture local hand context: $$ \begin{array}{r}{({\bf Q}_{i+1},{\bf R}_{i+1})=\mathcal{D}_{i}({\bf Q}_{i},{\bf R}_{i},{\bf F}_{e}).}\end{array} $$ This process produces multi-scale image features $\mathbf{F}_{GD}$; the refined Q encodes holistic hand semantics, while $\mathbf{R}$ carries spatial locations. The detection head $\mathcal{H}$ maps them to bounding boxes $\mathbf{B}$ and confidence scores $\mathbf{C}$: $$ \mathbf{(B,C)}=\mathcal{H}(\mathbf{Q},\mathbf{R}). $$ The resulting detector representation is $$ \left\{\mathbf{F}_{G D},\mathbf{Q},\mathbf{R},\mathbf{B},\mathbf{C}\right\}. $$ which is passed to the Metric 2D-3D Decoder. After score filtering and NMS, suppose K hands remain, represented by $\{(\mathbf{Q}_k, \mathbf{R}_k, \mathbf{B}_k)\}_{k=1}^K$. Each hand has its own detector query and reference point, and the same hand decoder processes all K retained instances in parallel. ScaleHP therefore handles multiple hands without changing the decoder formulation. The detector remains frozen, so the scale, pose, and projection losses optimize only the hand decoder. The full image is resized to a 1280-pixel long edge and no externally supplied hand crop is required. ### A.2 Metric 2D-3D Decoder Instance expansion. For each retained hand, learnable joint embeddings $E_J$ expand the instance query into two semantically indexed sets of $J = 21$ tokens. The scale token shares the same instance context but has its own stream: $$ \mathbf{Q}_{0}^{2D}=\mathbf{Q}_{0}^{3D}=\tilde{\mathbf{Q}}+\mathbf{E}_{J},\qquad\mathbf{q}_{0}^{s}=\tilde{\mathbf{Q}}. $$ The 2D tokens represent joint locations in the image, the 3D tokens represent the corresponding unit-scale root-relative joints, and the single scale token represents a hand-level metric gauge. Initial spatial references are inherited from the corresponding detector instance and subsequently refined by each stream. Reciprocal scale-joint interaction. Let $\mathbf{Q}_l^J = [\mathbf{Q}_l^{2D}; \mathbf{Q}_l^{3D}]$ collect the two joint streams. At interactive layer l, the scale and joint streams are concatenated without a directional attention mask: $$ \mathbf{X}_{l-1}=[\mathbf{q}_{l-1}^{s};\mathbf{Q}_{l-1}^{J}],\qquad[\bar{\mathbf{q}}_{l}^{s};\bar{\mathbf{Q}}_{l}^{J}]=\mathrm{S A}_{l}(\mathbf{X}_{l-1}). $$ Consequently, the scale-token output aggregates keys and values from named 2D and 3D joints, while each joint output contains an attention-weighted contribution from the scale token. The former direction lets scale estimation use joint layout and normalized skeletal structure; the latter makes scale-supervised context available to pose features. Since the resulting states are fed to the next layer, repeated interaction permits reciprocal refinement rather than a one-time concatenation at the prediction head. Image interaction and spatial refinement. Self-attention is followed by stream-specific deformable cross-attention: $$ \begin{align*}\mathbf{Q}_{l}^{d}&=Def\mathrm{Attn}_{l}^{d}(\bar{\mathbf{Q}}_{l}^{d},\mathbf{F}_{GD},\mathbf{R}_{l-1}^{d}),\\\mathbf{R}_{l}^{d}&=\mathbf{R}_{l-1}^{d}+\mathrm{FFN}_{R}^{d}(\mathbf{Q}_{l}^{d}),\qquad d\in\{s,2D,3D\}.\end{align*} $$ where $\mathbf{Q}_l^s \equiv \mathbf{q}_l^s$. Thus the scale token combines skeletal evidence with multi-scale hand appearance and image occupancy, while 2D/3D tokens sample local evidence near their evolving references. Three independent heads at the final layer output $$ \begin{aligned}\mathbf{J}^{2D}=\mathrm{FFN}_{2D}(\mathbf{Q}_{L}^{2D}),\quad&\mathbf{J}^{3D}=\mathrm{FFN}_{3D}(\mathbf{Q}_{L}^{3D}),\\s=\mathrm{FFN}_{s}(\mathbf{q}_{L}^{s}).\end{aligned} $$ They are supervised by pixel joints $\mathbf{J}^{2D*}$, unit-scale joints $\mathbf{J}^{3D*}$, and IMR scale $s^*$ derived from the same metric annotation $\mathbf{J}^{M*}$. The analytical transfer below consumes these three outputs and known intrinsics; no learned translation head or iterative registration is used. ## B Derivation of Analytical Metric Transfer Once the global scale s is predicted, normalized root translation can be recovered from perspective constraints. Let joint j have predicted pixel coordinates $(u_j, v_j)$ and unit-scale root-relative coordinates $(x_j, y_j, z_j)$. We first derive the x-axis relation: $$ \frac{u_{j}-c_{x}}{f_{x}}=\frac{x_{j}+t_{x}^{norm}}{z_{j}+t_{z}^{norm}}. $$ Multiplying both sides by the denominators gives $$ (u_{j}-c_{x})(z_{j}+t_{z}^{n o r m})=f_{x}(x_{j}+t_{x}^{n o r m}). $$ Let p be $+1$ $\infty$ot-relative $c\circ$ d s $$ (u_{p}-c_{x})t_{z}^{n o r m}=f_{x}t_{x}^{n o r m}. $$ Substituting Eq. 24 back into the expanded projection gives $$ (u_{j}-c_{x})(z_{j}+t_{z}^{norm})=f_{x}\left(x_{j}+\frac{u_{p}-c_{x}}{f_{x}}t_{z}^{norm}\right) $$ Collecting all terms that contain $t_{z}^{norm}$ on the left produces $$ (u_{j}-u_{p})t_{z}^{n o r m}-f_{x}x_{j}=-(u_{j}-c_{x})z_{j}. $$ The same steps along the y-axis give $$ (v_{j}-v_{p})t_{z}^{n o r m}-f_{y}y_{j}=-(v_{j}-c_{y})z_{j}. $$ --- Stacking both equations for every non-root joint forms the overdetermined linear system $$ \underbrace{\left[\begin{matrix}{\vdots}\\ {u_{j}-u_{p}}\\ {v_{j}-v_{p}}\\ {\vdots}\\ \end{matrix}\right]}_{\mathbf{A}}t_{z}^{{n o r m}}=\underbrace{\left[\begin{matrix}{\vdots}\\ {f_{x}x_{j}-(u_{j}-c_{x})z_{j}}\\ {f_{y}y_{j}-(v_{j}-c_{y})z_{j}}\\ {\vdots}\\ \end{matrix}\right]}_{\mathbf{b}}. $$ We solve $t_z^{norm} = A^\dagger b$ by least squares and recover the other two components from the root projection: $$ t_{x}^{n o r m}=\frac{u_{p}-c_{x}}{f_{x}}t_{z}^{n o r m},\qquad t_{y}^{n o r m}=\frac{v_{p}-c_{y}}{f_{y}}t_{z}^{n o r m}. $$ $$ \mathbf{Q}^{c a m}=s(\mathbf{J}^{r e l}+\mathbf{t}^{n o r m}),\qquad\mathbf{t}^{M}=s\mathbf{t}^{n o r m}. $$ Because a common scale cancels under projection, the system estimates normalized translation. The predicted scale converts both the root-relative pose and its translation to metric camera coordinates: ## C Datasets, Training, and Reproducibility Details Datasets. We evaluate the performance of our method on the following three primary benchmarks, which provide standardized protocols for 3D hand pose estimation: FreiHand (Zimmermann et al. 2019): A widely recognized benchmark for 3D hand pose and MANO (Romero, Tzionas, and Black 2017a) fitting, containing 130,240 training and 3,960 test images. We follow the evaluation protocols in (Chen et al. 2021a; Huang et al. 2023), utilizing the provided 3D joint annotations for both supervision and quantitative assessment. DexYCB (Chao et al. 2021): A large-scale benchmark for 3D hand pose estimation and hand-object interaction. It comprises 429,616 training and 78,768 testing samples, featuring synchronized RGB-D sequences of human hands interacting with 20 YCB objects. The dataset provides accurate 3D annotations for both hand joints and object poses, offering an ideal platform for evaluating models in complex, real-world grasping scenarios with significant occlusions. HO3Dv3 (Hampali et al. 2020): This refined version of the HO3D dataset offers enhanced annotation accuracy and a larger scale, comprising 83,325 training and 20,137 test images. The dataset captures real-world 3D hand-object interactions and presents a significant challenge due to the severe occlusions caused by the manipulated objects. Optimization. The frozen Grounding DINO 1.5 detector processes the full image after resizing its long edge to 1280 pixels. The hand Decoder is trained for 45 epochs with Adam, total batch size 16, initial learning rate $10^{-4}$, and eight NVIDIA A100-80GB GPUs. The recorded configuration uses random seed 21 and a cosine-annealing schedule from epoch 7 to 45 with minimum learning rate $5 \times 10^{-6}$. The Decoder has six interactive loss layers, 21 joints per hand, and four image-feature levels. The first two layers are 2D-only; scale, projection, and 3D supervision begin at layer 2.
MethodFreiHandDexYCBHO3Dv3
Ours1.940.561.02
Table 8: Per-instance scale prediction error across datasets $(L_{1}$ distance, $10^{-3}$ meters $.
Training objective. Let $\ell \in \{0, \ldots, 5\}$ index the six decoder outputs. The complete enabled objective and its configured weights are $$ \begin{aligned}\mathcal{L}=\sum_{\ell=0}^{5}\left(10\mathcal{L}_{1,\ell}^{2D}+4\mathcal{L}_{\mathrm{OKS},\ell}^{2D}\right)\\+\sum_{\ell=2}^{5}\left(20\mathcal{L}_{\mathrm{MSE},\ell}^{s}+10\mathcal{L}_{1,\ell}^{\mathrm{proj}}+4\mathcal{L}_{\mathrm{OKS},\ell}^{\mathrm{proj}}\right)\\+10\mathcal{L}_{1,\ell}^{3D}+5\mathcal{L}_{\ell}^{\mathrm{edge}}\right)\end{aligned} $$ Thus the shared $L_{1}$, OKS, relative-edge, and scale-MSE weights are 10.0, 4.0, 5.0, and 20.0, respectively. The projection terms reuse the same $L_{1}$/OKS weights. ## D Accuracy of Metric Scale Prediction To evaluate the learned per-instance metric gauge separately from pose and translation, we report its average $L_{1}$ error against the annotation-derived target on three datasets. Table 8 shows errors below $2 \times 10^{-3}$ meters. This result measures in-distribution prediction of the supervised scale target. ## E Camera-Space Evaluation Protocol Camera-space comparisons depend on the information supplied at test time. ScaleHP consumes RGB and known camera intrinsics, but no ground-truth hand scale. The 42.4 mm NVF result in the main paper uses FreiHand's reference-bone scale; NVF reports 47.2 mm without it. For CMR and HandDGP, “+GS” denotes a test-set-optimal dataset-level rescaling applied to root-relative geometry and translation. This oracle calibration is used only for the cross-dataset control. Main-paper checkpoint mapping. Main-paper Table 2 evaluates a FreiHand-trained checkpoint on FreiHand. Main-paper Table 3 applies that same checkpoint directly to DexYCB and HO3Dv3, without target-dataset fine-tuning. In main-paper Table 4, each dataset column uses a checkpoint trained on that dataset. Main-paper Tables 5 and 6 contain FreiHand-trained ablations, whereas each column of main-paper Table 7 uses its corresponding dataset-specific checkpoint. The supplementary root-localization, white-background, and FreiHand qualitative results use the FreiHand checkpoint; the scale-token interaction ablation uses the corresponding checkpoint trained on each of the three datasets. All reported values are single-checkpoint point estimates computed consistently from the same prediction dump for each setting. ## F Global Root Localization Error To verify that joint optimization effectively reduces error source (b), we compare predicted wrist-root positions with ---
MethodRoot L2 $\downarrow$$|\Delta X| \downarrow$$|\Delta Y| \downarrow$$|\Delta Z| \downarrow$
CMR50.145.344.9648.59
HandDGP46.195.334.3444.65
ScaleHP35.744.343.7334.36
Table 9: FreiHand root-localization errors (mm). The best CMR-PG result in the paper is 48.8 mm CS-MPJPE, but its ResNet50 weights were not released; the CMR row therefore uses the public CMR-SG checkpoint.
ground truth in the FreiHand camera coordinate system. Table 9 reports the mean Euclidean root distance together with mean absolute errors on the three camera axes. For CMR, the reported 48.8 mm result belongs to CMR-PG with an unreleased ResNet50 backbone; our reproducible evaluation therefore uses the public CMR-SG ResNet18 checkpoint, its native $224 \times 224$ input and 0.2 scale, and the authors' UV/confidence/silhouette registration. HandDGP uses its public FreiHand checkpoint, rectified $224 \times 224$ input, canonical focal length 1000, principal point 112, and native 0.2 hand scale; this setting reproduces the paper result at 46.35 mm versus the reported 46.3 mm. As shown in Table 9, ScaleHP reduces mean root error by 10.45 mm relative to HandDGP and by 14.40 mm relative to the reproduced CMR model. The corresponding depth-error reductions are 10.29 and 14.23 mm, whereas the lateral-axis changes are below 1.4 mm. Thus the gain is concentrated on camera-depth localization, demonstrating that the joint scale-pose optimization directly suppresses the dominant component of error source (b). ## G Scale-Token-Joint Interaction Table 10 isolates how the metric scale representation communicates with joint queries while keeping the detector, data, objective, and optimization schedule fixed. The oracle-global-scale control removes per-instance scale prediction and uses the test-set-optimal dataset-level scalar. In the independent-branch variant, the filtered instance query is mapped directly to s by an MLP with scale supervision, without scale-joint attention. The two one-way variants apply a self-attention mask: either the scale token can attend to joint queries but joints cannot attend back, or joints can attend to scale but scale cannot attend back. The proposed model leaves both directions open.
SettingFreiHand $\downarrow$DexYCB $\downarrow$HO3Dv3 $\downarrow$
Oracle global scale44.340.242.0
Independent branch42.937.840.5
Scale sees joints only40.435.338.0
Joints see scale only40.235.437.6
Bidirectional interaction35.830.033.4
Table 10: Scale-token interaction ablation measured by CS-MPJPE (mm). Lower is better. Each dataset uses its corresponding within-dataset checkpoint. Both one-way interactions improve over an independent scale branch, but neither matches reciprocal attention. As shown in Table 10, bidirectional exchange lowers CS-MPJPE by between 4.4 and 5.4 mm relative to the stronger one-way result on each dataset. This consistent gain shows that the two directions solve complementary problems: joint geometry improves scale estimation, while scale context improves joint representation. Reciprocal interaction, rather than merely adding a scale branch, is therefore responsible for the best camera-space accuracy. ## H Aligned-Pose Comparison Table 11 evaluates articulated geometry after removing global factors. ScaleHP ties the best FreiHand Procrustes-aligned error and achieves the lowest listed errors on DexYCB and HO3Dv3. These results show that the camera-space improvement is not obtained by sacrificing root-relative hand geometry. ---
MethodFreiHandDexYCBHO3Dv3
CS-MPJPE $\downarrow$PA-MPJPE $\downarrow$R-MPJPE $\downarrow$PA-MPJPE $\downarrow$PA-MPJPE $\downarrow$
ObMan (Hasson et al. 2019)85.213.3---
MANO CNN (Zimmermann et al. 2019)71.311.0---
CMR-PG (Chen et al. 2021a)48.86.9---
I2L-MeshNet (Moon and Lee 2020)60.37.4---
HandDGP (Valassakis and Garcia-Hernando 2024)46.37.4---
MobRecon (Chen et al. 2022)50.25.714.26.4-
METRO (Lin, Wang, and Liu 2021a)-6.715.27.0-
HandOccNet (Park et al. 2022)--14.05.8-
H2ONet (Xu et al. 2023)--14.05.7-
Deformer (Yoshiyasu 2023)--13.65.2-
Zhou et al. (Zhou et al. 2024)--12.45.5-
TI-Net (Ren et al. 2025)--16.84.9-
MaskHand (Saleem et al. 2025)-5.511.75.07.0
$S^{2}HAND$ (Chen et al. 2021b)-11.8--11.5
ArtiBoost (Li et al. 2021)--12.8-10.8
HandGCAT (Wang et al. 2023)----9.1
AMVUR (Jiang et al. 2023)-6.2--8.7
SPMHand (Lu, Gou, and Li 2024)----8.6
Hamba (Dong et al. 2024)-5.7--6.9
HandOS (Chen et al. 2025)-5.0-5.26.8
ScaleHP35.85.010.34.65.9
Table 11: State-of-the-art comparison on FreiHand, DexYCB, and HO3Dv3 (mm).
## I White-Background Intervention To test whether scale prediction relies on scene background, we replace every FreiHand evaluation background with white while preserving foreground hand pixels and their image coordinates, as illustrated in Fig. 5. We then rerun the complete detector-decoder pipeline on all 3,960 paired samples. ![Image](images/6b4d70463bfb90cc7049127a4725fc3d3d00bdaede94f55414cdecd31052cbfc.jpg)
Figure 5: White-background intervention examples. Foreground hand pixels and their image coordinates are retained, while all other pixels are set to white; images are shown without cropping.
Table 12 shows that removing all background content changes camera-space error by only 1.370 mm (3.8%), while the median paired change is 0.113 mm. This small shift persists even though the upstream detector boxes and 2D joints also change. Together with the retained scale correlation, the result shows that the scale token is driven primarily by foreground hand appearance and joint geometry rather than background scene cues.
StatisticValue
Original CS-MPJPE35.854 mm
White-background CS-MPJPE37.225 mm
Mean paired change+1.370 mm
Median paired change+0.113 mm
Scale Pearson / Spearman0.622 / 0.595
Mean detection-box IoU0.840
Mean 2D-keypoint displacement4.343 px
Table 12: Full-pipeline white-background intervention on FreiHand.
## J Additional FreiHand Qualitative Results Figures 6 and 7 show all 16 additional FreiHand groups. Each row shows the input, metric camera-space reconstruction, and front, side, and top views. --- ![Image](images/ef41d6538087a3d59319300d115cfd8aa7bb3c36e35290af89707d50f63c36c0.jpg)
Figure 6: Additional FreiHand results (1/2). Yellow: GT; blue: ScaleHP; green: HandDGP; red: CMR. Best viewed zoomed in.
--- ![Image](images/e760297db9009815632920abdca92eaf4f33c39f9f0fd4688de0ebdcb0498975.jpg)
Figure 7: Additional FreiHand results (2/2). Yellow: GT; blue: ScaleHP; green: HandDGP; red: CMR. Best viewed zoomed in.
--- ## K More ScaleHP Qualitative Results Figures 8 and 9 show ScaleHP outputs on eight open-domain images. The examples include occlusion, motion blur, varied lighting, and interaction patterns.
Ex.1
Ex.2
Ex.3
Ex.4
![Image](images/f110584826109302b07552b98b96401ff5e2f048874ea883c2c977fac9ca01b9.jpg)
Figure 8: ScaleHP results on four in-the-wild examples (1/2). Best viewed zoomed in.
---
Ex.5
Ex.6
Ex.7
![Image](images/4c56d638c623b22d32089c7048b2a0099c6288a3c3cd987850e2192e9d5aab4e.jpg)
Figure 9: ScaleHP results on four in-the-wild examples (2/2). Best viewed zoomed in.
--- ## L Limitations and Future Works Limitations. Because the training datasets predominantly contain adult hands, metric-space accuracy may degrade for infants, young children, and populations whose hand size distribution is underrepresented during training. ScaleHP also assumes known camera intrinsics, so focal-length or principal-point errors can propagate to the recovered translation. Future Works. We plan to incorporate training data spanning broader ages and demographic variation, evaluate robustness to camera-calibration perturbations, and extend the frame-wise method with temporal constraints for stable video tracking. ## References Baek, S.; Kim, K. I.; and Kim, T. 2019. Pushing the Envelope for RGB-Based Dense 3D Hand Pose Estimation via Neural Rendering. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 1067–1076. IEEE. Bhat, S. F.; Alhashim, I.; and Wonka, P. 2021. Adabins: Depth estimation using adaptive bins. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 4009–4018. Boukhayma, A.; Bem, R. d.; and Torr, P. H. 2019. 3d hand shape and pose from images in the wild. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 10843–10852. Chao, Y.-W.; Yang, W.; Xiang, Y.; Molchanov, P.; Handa, A.; Tremblay, J.; Narang, Y. S.; Van Wyk, K.; Iqbal, U.; Birchfield, S.; et al. 2021. Dexycb: A benchmark for capturing hand grasping of objects. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 9044–9053. Chen, X.; Liu, Y.; Dong, Y.; Zhang, X.; Ma, C.; Xiong, Y.; Zhang, Y.; and Guo, X. 2022. MobRecon: Mobile-Friendly Hand Mesh Reconstruction from Monocular Image. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 20512–20522. IEEE. Chen, X.; Liu, Y.; Ma, C.; Chang, J.; Wang, H.; Chen, T.; Guo, X.; Wan, P.; and Zheng, W. 2021a. Camera-space hand mesh recovery via semantic aggregation and adaptive 2d-1d registration. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 13274–13283. Chen, X.; Song, Z.; Jiang, X.; Hu, Y.; Yu, J.; and Zhang, L. 2025. HandOS: 3D Hand Reconstruction in One Stage. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 17304–17314. IEEE. Chen, Y.; Tu, Z.; Kang, D.; Bao, L.; Zhang, Y.; Zhe, X.; Chen, R.; and Yuan, J. 2021b. Model-based 3D Hand Reconstruction via Self-Supervised Learning. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 10446–10455. Dong, H.; Chharia, A.; Gou, W.; Carrasco, F. V.; and la Torre, F. D. 2024. Hamba: Single-view 3D Hand Reconstruction with Graph-guided Bi-Scanning Mamba. In Advances in Neural Information Processing Systems (NeurIPS). Ge, L.; Ren, Z.; Li, Y.; Xue, Z.; Wang, Y.; Cai, J.; and Yuan, J. 2019. 3d hand shape and pose estimation from a single rgb image. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 10833–10842. Guizilini, V.; Vasiljevic, I.; Chen, D.; Ambrus, R.; and Gaidon, A. 2023. Towards zero-shot scale-aware monocular depth estimation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 9233–9243. Hampali, S.; Rad, M.; Oberweger, M.; and Lepetit, V. 2020. HOnnotate: A Method for 3D Annotation of Hand and Object Poses. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 3193–3203. IEEE. Hasson, Y.; Tekin, B.; Bogo, F.; Laptev, I.; Pollefeys, M.; and Schmid, C. 2020. Leveraging photometric consistency over time for sparsely supervised hand-object reconstruction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 571–580. Hasson, Y.; Varol, G.; Tzionas, D.; Kalevatykh, I.; Black, M. J.; Laptev, I.; and Schmid, C. 2019. Learning joint reconstruction of hands and manipulated objects. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 11807–11816. Huang, L.; Lin, C.-C.; Lin, K.; Liang, L.; Wang, L.; Yuan, J.; and Liu, Z. 2023. Neural voting field for camera-space 3D hand pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 8969–8978. Iqbal, U.; Molchanov, P.; Gall, T. B. J.; and Kautz, J. 2018. Hand pose estimation via latent 2.5 d heatmap regression. In Proceedings of the European Conference on Computer Vision, 118–134. Jiang, Z.; Rahmani, H.; Black, S.; and Williams, B. M. 2023. A Probabilistic Attention Model with Occlusion-aware Texture Regression for 3D Hand Reconstruction from a Single RGB Image. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 758–768. Karim, M.; Khalid, S.; Lee, S.; Almutairi, S. S.; Namoun, A.; and Abohashrh, M. 2025. Next Generation Human Action Recognition: A Comprehensive Review of State-of-the-Art Signal Processing Techniques. IEEE Access, 13: 135609–135633. Kulon, D.; Guler, R. A.; Kokkinos, I.; Bronstein, M. M.; and Zafeiriou, S. 2020. Weakly-supervised mesh-convolutional hand reconstruction in the wild. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 4990–5000. Li, K.; Yang, L.; Zhan, X.; Lv, J.; Xu, W.; Li, J.; and Lu, C. 2021. ArtiBoost: Boosting Articulated 3D HandObject Pose Estimation via Online Exploration and Synthesis. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2740–2750. Li, Z.; Wang, X.; Liu, X.; and Jiang, J. 2024. Binsformer: Re-visiting adaptive bins for monocular depth estimation. IEEE Transactions on Image Processing, 33: 3964–3976. Lin, K.; Wang, L.; and Liu, Z. 2021a. End-to-end human pose and mesh reconstruction with transformers. In Proceedings --- of the IEEE/CVF conference on computer vision and pattern recognition, 1954–1963. Lin, K.; Wang, L.; and Liu, Z. 2021b. Mesh graphormer. In Proceedings of the IEEE/CVF international conference on computer vision, 12939–12948. Liu, S.; Zeng, Z.; Ren, T.; Li, F.; Zhang, H.; Yang, J.; Jiang, Q.; Li, C.; Yang, J.; Su, H.; Zhu, J.; and Zhang, L. 2024. Grounding DINO: Marrying DINO with Grounded Pre-training for Open-Set Object Detection. In Computer Vision – ECCV 2024, 38–55. Springer. Lu, H.; Gou, S.; and Li, R. 2024. SPMHand: Segmentation-Guided Progressive Multi-Path 3D Hand Pose and Shape Estimation. IEEE Transactions on Multimedia, 26: 6822–6833. Manning, J. T. 2002. Digit Ratio: A Pointer to Fertility, Behavior, and Health. New Brunswick, NJ: Rutgers University Press. ISBN 978-0-8135-3030-7. Millerdurai, C.; Wang, S.; Xie, Y.; Golyanik, V.; Stricker, D.; and Pagani, A. 2026. EgoForce: Forearm-Guided Camera-Space 3D Hand Pose from a Monocular Egocentric Camera. In ACM SIGGRAPH 2026 Conference Papers. Moon, G.; and Lee, K. M. 2020. I2l-meshnet: Image-to-pixel prediction network for accurate 3d human pose and mesh estimation from a single rgb image. In European Conference on Computer Vision, 752–768. Springer. Moon, G.; Yu, S.-I.; Wen, H.; Shiratori, T.; and Lee, K. M. 2020. Interhand2.6m: A dataset and baseline for 3d interacting hand pose estimation from a single rgb image. In European Conference on Computer Vision, 548–564. Springer. Nostadt, N.; Abbink, D. A.; Christ, O.; and Beckerle, P. 2020. Embodiment, Presence, and Their Intersections: Teleoperation and Beyond. ACM Trans. Hum. Robot Interact., 9(4):28:1–28:19. Park, J.; Oh, Y.; Moon, G.; Choi, H.; and Lee, K. M. 2022. Handoccnet: Occlusion-robust 3d hand mesh estimation network. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 1496–1505. Pavlakos, G.; Shan, D.; Radosavovic, I.; Kanazawa, A.; Fouhey, D.; and Malik, J. 2024. Reconstructing hands in 3d with transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 9826–9836. Potamias, R. A.; Zhang, J.; Deng, J.; and Zafeiriou, S. 2025. WiLoR: End-to-end 3D Hand Localization and Reconstruction in-the-wild. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 12242–12254. IEEE. Rastogi, P.; Nagesh, K. R.; and Yoganarasimha, K. 2008. Estimation of stature from hand dimensions of north and south Indians. Legal Medicine, 10(4): 185–189. Ren, K.; Hu, L.; Zhang, Z.; Ye, Y.; and Xia, S. 2025. Learning Transformation-Isomorphic Latent Space for Accurate Hand Pose Estimation. CoRR, abs/2502.12535. Romero, J.; Tzionas, D.; and Black, M. J. 2017a. Embodied hands. ACM Transactions on Graphics (TOG), 36: 1–17. Romero, J.; Tzionas, D.; and Black, M. J. 2017b. Embodied hands: modeling and capturing hands and bodies together. ACM Transactions on Graphics (TOG), 36(6): 1–17. Saleem, M. U.; Pinyoanuntapong, E.; Patel, M. J.; Xue, H.; Helmy, A.; Das, S.; and Wang, P. 2025. MaskHand: Generative Masked Modeling for Robust Hand Mesh Reconstruction in the Wild. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 8372–8383. Shi, D.; Wei, X.; Li, L.; Ren, Y.; and Tan, W. 2022. End-to-end multi-person pose estimation with transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 11069–11078. Song, C.; Zhang, H.; Yuan, H.; Lin, H.; Tao, J.; Zhang, H.; and Yu, T. 2025. MetricHMSR: Metric Human Mesh and Scene Recovery from Monocular Images. CoRR, abs/2506.09919. Spurr, A.; Iqbal, U.; Molchanov, P.; Hilliges, O.; and Kautz, J. 2020. Weakly Supervised 3D Hand Pose Estimation via Biomechanical Constraints. In European Conference on Computer Vision. Springer. Sun, Q.; Wang, Y.; Zeng, A.; Yin, W.; Wei, C.; Wang, W.; Mei, H.; Leung, C.-S.; Liu, Z.; Yang, L.; et al. 2024. Aios: All-in-one-stage expressive human pose and shape estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 1834–1843. Toet, A.; Kuling, I. A.; Krom, B. N.; and van Erp, J. B. F. 2020. Toward Enhanced Teleoperation Through Embodiment. Frontiers Robotics AI, 7: 14. Valassakis, E.; and Garcia-Hernando, G. 2024. Hand-dgp: Camera-space hand mesh prediction with differentiable global positioning. In European Conference on Computer Vision, 479–496. Springer. Varu, P. R.; Manvar, P. J.; Mangal, H. M.; Kyada, H. C.; Vadgama, D. K.; and Bhuva, S. D. 2015. Determination of stature from hand dimensions. The Journal of Medical Research, 1(3): 104–107. Wang, R.; Xu, S.; Dong, Y.; Deng, Y.; Xiang, J.; Lv, Z.; Sun, G.; Tong, X.; and Yang, J. 2025. MoGe-2: Accurate Monocular Geometry with Metric Scale and Sharp Details. CoRR, abs/2507.02546. Wang, S.; Wang, S.; Yang, D.; Li, M.; Qian, Z.; Su, L.; and Zhang, L. 2023. HandGCAT: Occlusion-Robust 3D Hand Mesh Reconstruction from Monocular Images. 2023 IEEE International Conference on Multimedia and Expo (ICME), 2495–2500. Xu, H.; Wang, T.; Tang, X.; and Fu, C.-W. 2023. H2ONet: Hand-Occlusion-and-Orientation-Aware Network for Real-Time 3D Hand Mesh Reconstruction. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 17048–17058. Yang, J.; Zeng, A.; Liu, S.; Li, F.; Zhang, R.; and Zhang, L. 2023. Explicit box detection unifies end-to-end multi-person pose estimation. CoRR, abs/2302.01593. Yin, W.; Zhang, C.; Chen, H.; Cai, Z.; Yu, G.; Wang, K.; Chen, X.; and Shen, C. 2023. Metric3d: Towards zero-shot metric 3d prediction from a single image. In Proceedings of --- the IEEE/CVF international conference on computer vision, 9043–9053. Yoshiyasu, Y. 2023. Deformable mesh transformer for 3d human mesh recovery. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 17006–17015. Zhou, Z.; Zhou, S.; Lv, Z.; Zou, M.; Tang, Y.; and Liang, J. 2024. A simple baseline for efficient hand mesh reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 1367–1376. Zhu, R.; Wang, C.; Song, Z.; Liu, L.; He, J.; Deng, J.; Zhang, T.; and Zhang, Y. 2026. ScaleDepth: Decomposing Metric Depth Estimation Into Semantic-Aware Scale Prediction and Adaptive Relative Depth Estimation. IEEE Transactions on Circuits and Systems for Video Technology, 36(7): 9547–9560. Zhu, X.; Su, W.; Lu, L.; Li, B.; Wang, X.; and Dai, J. 2021. Deformable DETR: Deformable Transformers for End-to-End Object Detection. In International Conference on Learning Representations (ICLR). OpenReview.net. Zimmermann, C.; Ceylan, D.; Yang, J.; Russell, B.; Argus, M.; and Brox, T. 2019. Freihand: A dataset for markerless capture of hand pose and shape from single rgb images. In Proceedings of the IEEE/CVF international conference on computer vision, 813–822.