# OmniHands: Robust Motion Capture of Interactive Hands via A Versatile Transformer

DIXUAN LIN, Beijing Normal University, China

YUXIANG ZHANG, Tsinghua University, China

MENGCHENG LI, Tsinghua University, China

WEI JING, Lenovo, China

QI YAN, Lenovo, China

QIANYING WANG, Lenovo, China

YEBIN LIU, Tsinghua University, China

HONGWEN ZHANG $^{*}$, Beijing Normal University, China

![Image](images/8a92166f7fad3f3477e9854e9a6dfffc3fc77bbd722192af25ecdea7bedc3bef.jpg)

<div style="text-align: center;">Fig. 1. The proposed method, OmniHands, can robustly recover interactive hand motions and their relative movement from monocular inputs.</div>

In this paper, we introduce OmniHands, a universal approach to recovering interactive hand motions and their relative movement from monocular or multi-view inputs. Our approach addresses three major limitations of previous methods: lacking a unified solution for handling single hand and interactive hands image inputs, lacking a versatile framework to support various 4D tasks such as video-based tasks and multi-view tasks, and unable to maintain robust performance in real-world data. To overcome these challenges, we develop a universal architecture with novel tokenization and contextual feature fusion strategies, capable of adapting to a variety of tasks. Specifically, we propose a Relation-aware Two-Hand Tokenization (RAT)

method to embed positional relation information into the hand tokens. In this way, our network can handle both single-hand and two-hand inputs and explicitly leverage relative hand positions, facilitating the reconstruction of intricate hand interactions in real-world scenarios. As such tokenization indicates the relative relationship of two hands, it also supports more effective feature fusion. To this end, we further develop a 4D Interaction Reasoning (FIR) module to fuse hand tokens in 4D with attention and decode them into 3D hand meshes and relative temporal movements. The efficacy of our approach is validated on several benchmark datasets. The results on in-the-wild videos and real-world scenarios demonstrate the superior performances of our approach for interactive hand reconstruction. The code and video results can be found in omnihands.github.io.

CCS Concepts: • Computing methodologies → Motion capture.

Additional Key Words and Phrases: Hand motion capture, two-hand interaction, multi-view, temporal

## ACM Reference Format

Dixuan Lin, Yuxiang Zhang, Mengcheng Li, Wei Jing, Qi Yan, Qianying Wang, Yebin Liu, and Hongwen Zhang. 2026. OmniHands: Robust Motion Capture of Interactive Hands via A Versatile Transformer. ACM Trans. Graph. 45, 4, Article 39 (April 2026), 17 pages. <https://doi.org/10.1145/nnnnnnn.nnnnnnn>

## 1 Introduction

Hand mesh recovery is a fundamental technology for numerous advanced applications like virtual/augmented reality, human-computer interaction, robotics, and embodied AI.

Despite the remarkable progress achieved by existing methods [Li et al. 2022; Lin et al. 2021; Yu et al. 2023] in this field, three major challenges remain unresolved. First, there is a lack of a universal method

---

that can handle various input statuses for both single-handed and two-handed input. When using existing single-hand reconstruction models to reconstruct two hands, while they can individually reconstruct each hand, they fail to address the issue of hand occlusion and cannot compute the relative distance between the hands. On the other hand, current two-hand reconstruction models, due to their use of single images containing both hands as input, are limited to reconstructing scenarios where the hands are in close proximity, thereby exhibiting poor generalizability. Second, current hand recovery methods address specific subproblems such as monocular hand reconstruction, video-based reconstruction, and multi-view reconstruction, however, a unified and general model to handle these tasks is still lacking. Third, although current hand reconstruction methods perform well on test data, their effectiveness is in-the-wild scenarios is limited due to the inadequate size of training datasets, resulting in poor practical value.

The success of recent approaches in transformer-based large-scale models such as GPT-4 [Achiam et al. 2023], ViTpose [Xu et al. 2022], HaMeR [Pavlakos et al. 2023] provided us with the insight to investigate the usage of simple and high-capacity models along with large-scale in-the-wild data. Therefore, we proposed OmniHands, a transformer-based universal solution to recover interactive hand meshes cross various scenarios. The transformer possesses versatility, enabling it to perform unified forward operations on different input forms such as single-frame input, temporal input, and multi-view input. Additionally, multi-layer transformers' ability to process large datasets allows us to train a generalized model that remains robust in real-world scenarios across various lighting conditions, backgrounds, and interaction poses.

In the proposed method OmniHands, we unified the following aspects to achieve robust 4D hand mesh recovery: i) Status of Hand Images. The input can be single hand images or interactive hands images. In previous inter-hand recovery methods, a single image containing both hands is typically used as input. However, to accommodate both single-hand and inter-hand recovery scenarios, we need to unify the input format that handles both types of input while preserving the relative positions of the hands. ii) View Points. Inputting sequences of images from multiple camera perspectives captured at the same moment to enhance details through information supplementation across different viewpoints. iii) Temporal Sequences. Inputting a monocular dynamic sequence, make more accurate predictions by evaluating hand movements under conditions of motion blur and object occlusion. Since the multi-view data and temporal data can both be seen as contextual sequences, a unified module can be applied to fuse the contextual information.

To address these issues, we proposed a Relation-aware Two-Hand Tokenization (RAT) method and a 4D Interaction Reasoning (FIR) module to unifiedly process features in spatial context or temporal context.

In Relation-aware Two-Hand Tokenization (RAT), we crop and tokenize the RGB information of each hand individually, while maintaining hand-hand relative position information in tokens. Such a tokenization method brings the following benefits threefold. i) RAT is not reliant on the co-occurrence of two hands and can thus handle both single-hand and two-hand inputs in a unified manner. ii) RAT takes the overlapped status of two hands into account and adjusts the tokens accordingly, which allows the network to learn a more concentrated distribution of the hand tokens regardless of the various scales and positions. iii) RAT explicitly encodes the relative positions of two hands into the tokens, facilitating the reconstruction of close and intricate interaction in real-world scenarios.

Upon obtaining the image tokens, we integrate a Context Interaction Reasoning (FIR) module to unifiedly fuse contextual relations within the context channel. These contextual relations could be temporal (over time) or spatial (from multiple camera views). Thanks to the proposed tokenization method RAT, FIR can fuse both the contextual information more effectively as the hand tokens indicate the relative relationship of two hands, thereby enhancing the robustness of hand mesh and contextual movement recovery.

In summary, the main contributions of this work can be listed as follows:

- We introduce a Relation-aware Two-Hand Tokenization (RAT) in our transformer. By embedding positional relation information in the tokenization process, RAT enables a generic and informative process of hand inputs across diverse scenarios, regardless of whether the two hands are interacting or separated.

- We propose a 4D Interaction Reasoning (FIR) module to fuse spatial or temporal hand features. The collaboration of RAT and FIR further enhances the stability of hand mesh and temporal movement recovery.

- Our final solution, OmniHands, achieves robust hand motion capture in real-world scenarios, serving as a versatile solution to handle hand inputs in various forms, whether the images contain single or two hands, in single frames or temporal sequences or multi-view sequences.

## 2 Related Work

### 2.1 Monocular Hand Motion Capture

Recent years have witnessed great progress in hand mesh recovery from monocular RGB input. Following the advances in monocular human mesh recovery [Kanazawa et al. 2018; Kolotouros et al. 2019; Kolotouros et al. 2019; Tian et al. 2023; Zhang et al. 2022; Zhang et al. 2021], some researchers [Baek et al. 2019; Boukhayma et al. 2019; Park et al. 2022; Pavlakos et al. 2023; Zhang et al. 2019] use neural networks to regress the driving parameters of hand templates such as MANO [Romero et al. 2017] to recover hand meshes, while others prefer to directly estimate the vertex coordinates of hand meshes through GCN [Chen et al. 2021, 2022; Choi et al. 2020; Ge et al. 2019; Kulon et al. 2020; Tang et al. 2021] or transformer [Lin et al. 2021; Lin et al. 2021].

Recently, there have also been some efforts to simultaneously recover both two hands of humans. However, two hands mesh recovery is much more challenging than one hand, due to larger pose distribution and more severe occlusion. Furthermore, compared to the large-scale single-hand datasets [Pavlakos et al. 2023] with rich capture scenarios and postures, existing two-hand datasets are much fewer, and most of them are captured in limited environments.

A straightforward method for two-hand mesh recovery is to locate and crop each hand separately from the input image, and then convert the task into single-hand scenarios by processing two cropped

---

hand images individually. This strategy is widely used in full-body pose estimation frameworks [Choutas et al. 2020; Feng et al. 2021; Joo et al. 2018; Moon et al. 2022; Rong et al. 2021; Zhang et al. 2023, 2021b]. However, this approach is still based on single-hand recovery network, which usually performs poorly under close interaction situations. Moreover, the relative translation between two hands is ambiguous when the two hands are processed separately.

### 2.2 Hand Motion Capture under Interaction

To better address two-hands pose estimation and mesh recovery in close interaction scenarios, some researchers [Kim et al. 2021; Li et al. 2022, 2024; Meng et al. 2022; Moon et al. 2020; Yu et al. 2023; Zhang et al. 2021a; Zuo et al. 2023] used one image containing both hands as input. Zhang et al. [Zhang et al. 2021a] utilized 2d joints projection attention and context-aware refinement module to improve the pose accuracy. Li et al. [Li et al. 2022], Cho et al. [Cho et al. 2023] and Jiang et al. [Jiang et al. 2023] utilize transformer module [Dosovitskiy et al. 2020] to learn implicit attention between two hands. Zuo et al. [Zuo et al. 2023] propose a data-based prior of two interacting hands. Yu et al. [Yu et al. 2023] constructed the attention relationship between the two hands based on 2D hand center maps. Although the above approaches have produced promising results, there are still some limitations. Because these methods use a single image containing both hands as input, when the two hands are far apart, the area proportion of hands in the input image will be too small, which can reduce the accuracy of mesh recovery. Moon et al. [Moon 2023] separates the two hands as input, but its prediction of the relative position between the hands relies on bounding box information rather than visual cues from the image. Moreover, it also requires input images that contain both hands since the separation is conducted during forward process. Furthermore, these above network designs inherently lack the ability to recover single-hand mesh and only can operate under two-hand scenarios.

Therefore, we propose a new network that is compatible with both single-hand recovery and two-hand recovery. We take one or two cropped hand images as input, enabling utilize large-scale single-hand datasets to boost two-hand recovery results. We also tokenize the 2D relative distance and overlap between two cropped hand images by a powerful transformer [Dosovitskiy et al. 2020; Vaswani et al. 2017] module, resulting in accurate relative root translation estimation and interacting hands mesh recovery.

### 2.3 Temporal Hand Motion Capture

Several methods have been proposed for hand mesh recovery from temporal sequence data [Choi et al. 2021; Fu et al. 2023; Hasson et al. 2020; Kocabas et al. 2020; Liu et al. 2021; Yang et al. 2020]. RGB2Hands [Wang et al. 2020] proposed a real-time hand tracking approach that utilizes both segmentation and depth prediction to enhance performance. Hasson et al. [Hasson et al. 2020] leverage the optical flow of nearby frames to constrain the motion of hands and objects by photometric supervision. SeqHAND [Yang et al. 2020] incorporates the convolution-LSTM layer to capture the sequential relationship between consecutive hand poses. Deformer [Fu et al. 2023] proposes a novel dynamic fusion module that explicitly deforms nearby frames with clear hand visibility for robust hand pose estimation from temporal input. VIBE [Kocabas et al. 2020] and TCMR [Choi et al. 2021] utilize the recurrent neural network to produce temporally consistent motion.

However, these methods rely solely on sequence data. Our approach introduces a 4D Interaction Reasoning (FIR) module, enabling our method to not only recover accurate two-hand interacting meshes from single-frame input but also produce smooth and stable 4D hand recovery from occluded and motion-blurred temporal input.

### 2.4 Multi-View Hand Motion Capture

Recovering hands from multi-view input is a similar task as multi-view human reconstruction. Many recent researches have been proposed for multi-view 3D human pose estimation [Dong et al. 2019; Pavlakos et al. 2017; Qiu et al. 2019; Tu et al. 2020; Zhang et al. 2021b]. Some of the methods [Dong et al. 2019; Qiu et al. 2019; Tu et al. 2020] focus on matching the projection of 3D key-points with their corresponding 2D key-points from multiple camera views, and some [Pavlakos et al. 2017; Zhang et al. 2021b] focus on fusing the 2D visual features to build a 3D structure.

Recent research on multi-view hand reconstruction has seen various innovative approaches. Some researchers [Corona et al. 2022; Guo et al. 2023] utilize implicit fields to recover the shape and appearance of hands. Yang et al.[Yang et al. 2023] employ hand vertices as a point cloud embedded in the intersection volume of frustums from multiple cameras to integrate multi-view information. Wang et al.[Wang et al. 2024] adapt multi-view hand data to enhance the performance of single-view hand reconstruction. Additionally, Gan et al.[Gan et al. 2024] propose a neural rendering scheme to refine both the hand mesh and textures. However, most multi-view methods require extrinsic and intrinsic calibration information for each camera to accurately determine the 3D positions of hands. In contrast, our approach is capable of utilizing only visual information to recover precise hand meshes.

## 3 Method

In this Section, we present the technical details of OmniHands. As illustrated in Figure 2, OmniHands is a transformer-based network, which can accept multiple forms of input: single-frame images, videos, and multi-view images to estimates the hand meshes, as well as their relative position. In the proposed transformer, the key components include Relation-aware Two-hand Tokenization (RAT) and 4D Interaction Reasoning (FIR). As will be discussed shortly, the tokenization and reasoning strategies make OmniHands a generic solution to handle hand images in various forms, regardless of the images with single or two hands, in a single frame or temporal sequences or multi-view sequences.

### 3.1 Problem Formulation

Given a single-frame or a temporal sequence or a multi-view sequence of images I, our goal is to reconstruct the hand meshes and their relative position. We leverage the widely used parametric hand model MANO [Romero et al. 2017]. MANO defines a function $\mathbf{M}\mathbf{n}(\theta, \beta)$ that takes two parameters pose $\theta \in \mathbb{R}^{48}$ and shape $\beta \in \mathbb{R}^{10}$ as input, returns 3D hand mesh $V \in \mathbb{R}^{778 \times 3}$ and 3D joint keypoints

---

![Image](images/24fdbf1eb27284aca6c84cd74ec645f62bdd684a34bd232c88fcbeb495a8a201.jpg)

<div style="text-align: center;">Fig. 2. Overview of our OmniHands framework. OmniHands is a transformer-based network, which takes various forms of inputs and estimates two-hand meshes with their relative positions. General modules are used to process different forms of input data. For simplicity of representation, since the hand tokens of both hands will undergo the same processing in FIR, the hand tokens  $G^{*}$,  $L^{*}$ in the diagram represent those of one hand. In the top left corner we present our training set, indicating the data volume and whether including Interaction, Time sequences, Multi-view, Real-World data.</div>

 $V \in \mathbb{R}^{21 \times 3}$. The outputs of our model are MANO parameters of two hands: $\theta_R$, $\theta_L$, $\beta_R$, $\beta_L$, by which we can regress meshes and joints of each hand, and a vector v to represent the joint root distance of two hands. We define v to be the location of the left root joint location when we subtract the joints of both hands from the right hand's 3D root joint location. Note that our method can robustly handle both single-hand and two-hand inputs. When there is only a single hand visible in the images, our method is simply degraded to the estimation of the mesh of one hand.

### 3.2 Relation-aware Two-Hand Tokenization

Existing approaches to hand mesh recovery typically take the cropped hand images as input, but neglect the hand relationship in the images. Besides, these approaches handle two-hand and single-hand scenarios in different manners, resulting in jittering artifacts when the two hands operate in interaction and separately. To handle these issues in our transformer-based architecture, we propose a Relation-aware Two-Hand Tokenization (RAT) method to take the relation of two hands into consideration and handle inputs in various scenarios. Following recent state-of-the-art practices in 3D human/hand mesh recovery [Goel et al. 2023; Pavlakos et al. 2023; Xu et al. 2022], we utilize ViT [Dosovitskiy et al. 2020] as the visual encoder to tokenize image patches. In contrast to these methods, we handle both the single-hand and two-hand inputs in a unified manner and consider embedding positional relation information of each hand into its tokens during the tokenization process.

To achieve this, we first locate each hand from inputs and center it into the corresponding left and right hand images. This strategy

---

allows our method to learn a more concentrated distribution of the hand tokens regardless of the various scales and relative positions. Moreover, by tokenizing the images of two hands individually, our method can maintain the stability of inputs across various interaction scenarios, eventually enhancing the robustness of our solution. To compensate for the lack of relation information in the separated tokens, we further consider enriching the hand tokens with relative positional embedding information so that they can be aware of the relative position with respect to each hand and the input images.

Specifically, for each input image, we follow ViT [Dosovitskiy et al. 2020] to segment it into $H \times W$ image patches and subsequently feed them into the ViT backbone, resulting in a vision token map $M \in \mathbb{R}^{H \times W \times C}$. To enrich each token map with relationship information, we further calculate relation maps $\{\mathcal{R}_{R2L}, \mathcal{R}_{L2R}\}$ and embed them into the final image tokens, so that they can contain both the patch-wise visual knowledge and patch-to-patch relation information. Basically, the relation maps are the composition of the relative distance map and the overlapping map. Without losing generalization, we describe the right-hand relation map $\mathcal{R}_{R2L}$ in detail as follows. The left-hand relation map $\mathcal{R}_{L2R}$ can be calculated similarly.

Relative Distance Map. The relative distance map is calculated based on the hand position in the original images.

Let $\mathcal{B} = [c_x, c_y, s_x, s_y]$ be the bounding box of one hand, $c_x, c_y$ the x-coordinate and y-coordinate of the center position, and $s_x, s_y$ the box scale lengths on x-axis and y-axis. We obtain a position map $C \in \mathbb{R}^{H \times W \times 2}$, representing positions of the center in each image patch. For the patch $p(i, k)$ at the i-th column and the k-th row, its value is

 $$ C(p)=[c_{x}+\frac{(2i-W)\cdot s_{x}}{2W},c_{y}+\frac{(2k-H)\cdot s_{y}}{2H}]. $$

In our method, for each right hand and left hand, we first obtain their position maps $C_R, C_L$ and then calculate a relative distance map $\mathcal{D} \in \mathbb{R}^{H \times W \times 2}$. The relative distance token at $p(i, k)$ of the right hand is formulated as

 $$ \mathcal{D}_{R2L}(p)=\psi(\tau\cdot\frac{C_{R}(p)-C_{L}(p)}{s}) $$

where $s = (s_x, s_y)$ is the scale of right-hand bounding box, $\psi$ is an activation function, $\tau$ is a hyper-parameter used to adjust the activation distance. In this way, the relative distance map is activated when two hands are close. In our experiments, we use Sigmoid as the activation function $\psi$.

Overlapping Map. To explicitly represent the overlapped relation of two hands, we further introduce the overlapping map. Denoting Inside $(p(i,k), \mathcal{B})$ as a function that indicates whether the center of patch p is inside the bounding box $\mathcal{B}$, we formulate the the overlapping map $O \in \mathbb{R}^{H \times W \times 1}$ as

 $$ O_{R2L}(p)=\begin{cases}{1,}&{{I n s i d e}(p,\mathcal{B}_{L})}\\ {-1,}&{{o t h e r w i s e}}\\ \end{cases} $$

Given the above relative distance map and overlapping map, the relation map is calculated by first concatenating them in the last dimension and using an MLP to match the dimension with the image feature dimension C:

 $$ \mathcal{R}_{R2L}={M L P}(\mathcal{D}_{R2L}\oplus\mathcal{D}_{L2R}\oplus\mathcal{O}_{R2L}), $$

where $\oplus$ denotes the concatenation operation. Finally, the relation map acts as the relative positional embedding and enhances the original ViT tokens $M_R$ by concatenating them as the relation-aware tokens $M_R \in \mathbb{R}^{HW \times C}$.

 $$ \mathcal{M}_{\mathcal{R}}=\mathit{F l a t t e n}(M_{R}+\mathcal{R}_{R2L}) $$

Cross-hand Feature Fusion. With the token maps, we can fuse the cross-hand features by a Bimanual Decoder. Specifically, for each single frame we concatenate the tokens of two hands to $M = M_R \oplus M_L$. Then M is further fed into attention modules to fuse inter-hand information. To enhance the adaptability of our model, we first employ Spatio Encoder to proceed with the token sequence M:

 $$ \mathcal{M}^{*}=F F N(S o f t m a x(\frac{Q(\mathcal{M})K(\mathcal{M})}{\sqrt{d i m(\mathcal{M})}})V(\mathcal{M})) $$

where Q, K, V respectively represent the left-multiplied query, key, and value matrices.

As the relation-aware tokens are embedded with both visual and relation features, the self-attention module can adjust its weight adaptively based on visual and relative positional associativity and mitigate the interdependence of bilateral hand information. For instance, when relation-aware tokens indicate that the two hands are far away from each other, the self-attention can mainly interact with the tokens of each hand individually. The outputs $M^*$ from the self-attention module are further fed into Spatio Decoder, where we take a constant vector $\alpha \in \mathbb{R}^C$ as query vector, resulting in a global hand token $G \in \mathbb{R}^C$ corresponding to each hand:

 $$ G=F F N(S o f t m a x(\frac{Q(\alpha)K(\mathcal{M}^{*})}{\sqrt{d i m(\mathcal{M}^{*}))}})V(\mathcal{M}^{*})) $$

The vector $\alpha$ is predefined and has different value for left and right hands. Then the hand tokens are taken for further feature fusion according to the specific task demands.

### 3.3 4D Interaction Reasoning

Regardless of whether the input consists of monocular temporal frames or multi-view non-temporal frames, we can consider the set of inputs as contextually relative, therefore employ similar methods to correlate the information contained within them. Consequently, we aim to utilize a unified module 4D Interaction Reasoning (FIR) module in order to fuse the contextual knowledge to infer the hand poses and the relative position between two hands. We will discuss the processing differences specific to temporal inputs and multi-view inputs.

Temporal Context Fusion. When dealing with the temporal sequential inputs, we use $\{G_t\}_{t=1}^T$ to represent a temporal sequence of one of the two hands, for simplicity, and $\{G_t\}_{t=1}^T$ will be further proceeded by Global Context Decoder.

By fusing the sequential information, our model can predict the hand poses of intermediate frames more robustly in handling disturbances caused by object occlusions and dynamic blurring. Note that our method is also compatible with single-frame inputs. In this case,

---

we stack the hand tokens repetitively to obtain the sequential token $\mathcal{G} \in \mathbb{R}^{T \times C}$. Since the temporal input has a sequential order, we need to add position embedding to $\mathcal{G}$ before inputting it into Collective Decoder to get global feature. We only output the results of the middle $\frac{t}{2}$-th frame because, when predicting the middle frame, both past and future information is available, leading to more accurate predictions:

 $$ G^{*}=F F N(S o f t m a x(\frac{Q(\alpha)K(\mathcal{G})}{\sqrt{d i m(\mathcal{G})}})V(\mathcal{G}) $$

After the feature fusion of both spatial and temporal dimensions, we can obtain the integrated features $G_R^*, G_L^* \in \mathbb{R}^C$ as right hand collective token and left hand collective token, respectively. Finally, these features are proceeded by a Global Regressor, an MLP-based regression network to estimate the MANO parameters $\theta_R, \theta_L, \beta_R, \beta_L$ of two hands and their relative position v, and camera parameters $\pi$ for 2D projection.

Multi-view Context Fusion. When inputting frames of multiple camera views, we get hand tokens $\mathcal{G} \in \mathbb{R}^{T \times C}$ in the same way as mentioned above, where T is the number of cameras instead of time length. Then concatenating them to obtain the sequential global feature $\mathcal{G}$. Different with Temporal Context Fusion, since the camera parameters are not required as input, there isn't a fixed order of input frames and can be combined in any arbitrary sequence, therefore we don't add position embedding to $\mathcal{G}$ in this case. In temporal context fusion, we utilize information from preceding and succeeding frames to predict an accurate result of the intermediate frame. However, in multi-view context fusion, since there is no sequential order, the distance between frames is equivalent for all, enabling the mutual supplementation of information from each frame. Consequently, we can output predictions for all frames, with the collective information contributing to a precise result:

 $$ \mathcal{L}^{*}=F F N(S o f t m a x(\frac{Q(\mathcal{G})K(\mathcal{G})}{\sqrt{d i m(\mathcal{G})}})V(\mathcal{G}) $$

We get individual tokens $\mathcal{L}^* \in \mathbb{R}^{T \times C}$ from Individual Encoder (Eq 9) and collective features $G^* \in \mathbb{R}^{\times C}$ from Collective Decoder (Eq 8) after fusion. Given that all frames are taken at the same time from different views, the gesture pose $\theta_{3:45}$ and shape $\beta$ should be the same, while the root orientation $\theta_{0:3}$, camera parameters $\pi$ and root offset v varying. So we feed global feature $G^*$ into Collective Regressor, to regress the consistent parameters $\theta_{3:45}$ and shape $\beta$, and $\mathcal{L}^*$ into MLP-based Regressor to regress the inconsistent parameters $\theta_{0:3}$, $\pi$, v for each frame.

Although we predict the root orientations and offset vectors for each frame individually, the relative orientation and distance between the right hand and left hand should remain invariant across all frames. This means that when we transform all right hands to the zero position and zero rotation, the positions and orientations of the left hands should remain consistent. Therefore, we apply a constraint (Eq 17) to maintain this invariance. During inference, we average the relative translations and relative orientations to ensure consistency across views.

### 3.4 Losses

As our method predicts two hands in a disentangled way, we introduce single-hand losses and inter-hand losses to improve the accuracy of single-hand reconstruction and two-hand interaction.

#### 3.4.1 maxMSE Loss

Following [Fu et al. 2023], we use the maxMSE Loss to adjust the weights of each joint to maximize the MSE loss. Given joints locations $\mathcal{P}^*$, $\mathcal{P}$, the maxMSE loss is formulated as

 $$ m a x M S E(\mathcal{P},\mathcal{P}^{*})=\frac{\sum_{i=1}^{N}||\mathcal{P}_{i}-\mathcal{P}^{*}{}_{i}||^{4}}{\sum_{i=1}^{N}||\mathcal{P}_{i}-\mathcal{P}^{*}{}_{i}||^{2}}. $$

With maxMSE loss, the joints or vertices with large errors, typically the vertices of fingertips, will be assigned larger loss weights.

#### 3.4.2 Single-Hand Losses

For single-hand alignment, we train our model with MANO parameter losses, 3D losses, and 2D losses. For a single hand image, our model predicts its pose $\theta$, shape $\beta$ and camera parameter $\pi$. With ground truth MANO parameters $\tilde{\theta}$ and shape $\tilde{\beta}$, we optimize the MANO parameters using MSE loss:

 $$ L_{m a n o}=||\tilde{\theta}-\theta||^{2}+||\tilde{\beta}-\beta||^{2} $$

With the pose and shape parameters, we can obtain 3D vertices V and 3D joint keypoints J. With accurate 3D ground truth $\tilde{V}$ and $\tilde{J}$, we can supervise the output 3D keypoints with 3D loss using maxMSE

 $$ L_{3D}=max MSE(\tilde{V},V)+max MSE(\tilde{J},J) $$

With ground truth 3D joint keypoints and camera parameters, we can get 2D keypoints $\tilde{j} = \tilde{\pi}(\tilde{J})$ on the original image. The predicted 2D keypoints are calculated with predicted camera parameter $\pi$. The 2D projection loss is formulated

 $$ L_{2D}=maxMSE(\pi(J),\tilde{j}) $$

#### 3.4.3 Inter-Hand Losses

We utilize Inter-Hand losses in order to optimize the finger touching details of two hands. To calculate the relative distance between right hand keypoints and left hand keypoints, we relocate all keypoints to set the palm joint of right hand on the original coordinate. We denote $\phi_j(i, k) \in \mathbb{R}^3$ as the distance vector from the $i-th$ right hand joint to the $k-th$ left hand joint, and $\phi_v(i, k) \in \mathbb{R}^3$ as the distance vector of vertices, defined in the same way.

To supervise the interacting gesture, for ground truth joints and predicted joints, we calculate the distance from each right-hand joint to each left-hand joint, respectively. We formulate the joint-relation loss as:

 $$ L_{j rel}=\sum_{i=1}^{N_{j}}\sum_{k=1}^{N_{j}}||\tilde{\phi}_{j}(i,k)-\phi_{j}(i,k)||^{2} $$

where $N_{j} = 21$ represents the number of joint keypoints. For a finer reconstruction in contact of hands, we supervise the close vertices distances as:

 $$ L_{c l o s e}=\sum_{i=1}^{N_{v}}\sum_{k=1}^{N_{v}}\begin{cases}{||\tilde{\phi}_{v}(i,k)-\phi_{v}(i,k)||^{2},}&{\mathrm{i f~}||\tilde{\phi}_{v}(i,k)||^{2}\leq\alpha}\\ {0,}&{\mathrm{i f~}||\tilde{\phi}_{v}(i,k)||^{2}>\alpha}\\ \end{cases} $$

---

where $N_{v}=778$ is the number of vertices, and $\alpha$ is a hyper-parameter, representing the distance threshold. We set $\alpha=0.005$ in experiments.

#### 3.4.4 Sequential Consistency Losses

When using video as input, we use a smooth loss function to maintain the smoothness of the predicted mesh between video frames and reduce jitter:

 $$ L_{consis}=\sum_{t=1}^{N_{t}-1}||\theta_{t+1}-\theta_{t}||^{2} $$

where $N_{t}$ is the length of video sequence. We choose to constrain joint velocity rather than acceleration in the smoothness loss because acceleration involves second-order differences, which empirically yield weaker gradients and often lead to less stable or effective optimization during training.

When taking multi-view images as input, the inter-hand relative distance and relative rotation angle should be identical in all input images. Since the extrinsic parameters of the cameras from each viewpoint are different, predictions of the rotation angles and relative distances for each are coupled with the camera's rotation and translation predictions. Therefore, in each frame, we perform the same rotation and translation operations on the predicted joints of both hand to eliminate the effect of camera parameters. This process moves the root joint of the right hand to the coordinate origin and sets the rotation angle to zero, then denote the transformed left hand joints as $J^{*}$. The consistency loss is calculated:

 $$ L_{c o n s i s}=\sum_{i=1}^{N_{t}}\sum_{k=1}^{N_{t}}||J_{i}^{*}-J_{k}^{*}||^{2} $$

## 4 Experiments

### 4.1 Implementation details

We implement our model based on PyTorch [Paszke et al. 2019]. We use a pretrained Vision Transformer following a huge design (ViT-h) with 32 layers as the backbone to encode image features. The resolution of input image is $256 \times 192$ and the size of the feature map is $H = 16$, $W = 12$, $C = 1280$. We use transformer encoders and decoders in our feature fusion modules. We train our model for 1,000,000 steps using AdamW optimizer with a linear learning rate decay schedule from an initial learning rate $1 \times 10^{-5}$, and the decay rate is 0.7 for every 200,000. The training set is mixed of Interhand2.6m [Moon et al. 2020], Re:Interhand [Moon et al. 2024] and Single Hand Datasets, we use a random sampling strategy with sampling rate 40%, 40%, 20%. When taking temporal sequence as input, the length of an image sequence is set to 9 with a gap of 5 frames to sample images in a consecutive video sequence.

### 4.2 Datasets

We use a training dataset with a mix up of interacting hand datasets and single-hand datasets. The interacting hand datasets are used to optimize the hand relations in cases that two hands are close. The single-hand datasets are used to optimize the accuracy of single-hand mesh prediction in cases where two hands are far away.

Interhand2.6M [Moon et al. 2020] is the first publicly available dataset of sequences of interacting hands with accurate annotations, which is widely used for training and testing in two-hand reconstruction methods. InterHand2.6M in 30 fps is for video-related research and the 5 fps version is a subset of the 30 fps version which removes the redundancy for single image research. In our method, we make use of 30 fps and 5 fps interacting two-hand (IH) data with human and machine (H+M) annotations and filter invalid data following [Li et al. 2022]. We ultimately utilize 1.6M images for training and 1.2M images for testing.

Re:InterHand [Moon et al. 2024] has images with realistic and diverse appearances along with accurate GT 3D interacting hands. It has 738K video-based images in 3rd-person viewpoints. We take the whole dataset for training.

DexYCB [Chao et al. 2021] contains sequences of single hands interacting with objects. It consists of 1000 sequences with 582K images. Although it's not a two-hand dataset, we use it as part of the training set and testing set to enhance our model's performance in scenarios with object occlusion and motion blur.

Static Single Hand Datasets. To enhance the performance in wild videos, we use single-hand datasets that provide 3D hand annotations for training. Following the previous work [Pavlakos et al. 2023], we use the datasets with a combination of FreiHAND [Zimmermann et al. 2019], HO3D [Hampali et al. 2020], MTC [Xiang et al. 2019], RHD [Zimmermann and Brox 2017], H2O3D [Hampali et al. 2020], DEX YCB [Chao et al. 2021], COCO WholeBody [Jin et al. 2020], Halpe [Fang et al. 2022] and MPII NZSL [Simon et al. 2017]. We simulate a pair of widely separated hands by randomly combining two single-hand images and assigning them two non-overlapping bounding boxes.

In-the-wild Datasets. We use in-the-wild datasets HIC [Tzionas et al. 2016], ARCTIC [Fan et al. 2023] and RenderIH [Li et al. 2023] to evaluate the performance on various scenarios. HIC provides diverse hand-hand interacting and object-hand interacting sequences and contains 3D GT meshes of both hands. ARCTIC is a multi-view sequential dataset of two-hand interacting with daily use objects, involving occlusions and detailed interactive actions. RenderIH provides realistically rendered hands with a variety of poses, textures, backgrounds, and illuminations, which can simulate various scenarios in reality. For all in-the-wild datasets, we only use them for testing to evaluate the performance on unseen scenarios.

### 4.3 Metrics

We adopt Mean Per Joint Position Error (MPJPE) and Mean Per Vertex Position Error (MPVPE) and Area Under Curve (AUC) scores to measure the accuracy of the pose and shape of the estimated meshes, and Mean Relative-Root Position Error (MRRPE) to measure the accuracy of relative positions of two hands. For mean position error and AUC we get results under both Root-relative analysis (Ra) and Procrustes analysis (Pa). Following the prior method [Li et al. 2022], we perform root joint alignment and skeleton scaling when evaluating MPJPE. In addition, we use acceleration error Accel_E to evaluate the temporal consistency, following VIBE[Kocabas et al. 2020]. To better evaluate the accuracy of two-hand interactions, we propose Mean Inter-Joint Error (MIJE) as an evaluation metric.

---

![Image](images/1921da814eeaeebc7d3a142f7d5d7ac235d0fd504517c66591b307341a6ea50e.jpg)

<div style="text-align: center;">Fig. 3. Qualitative Comparison. We compare OmniHands with state-of-the-art methods on in-the-wild datasets ARCTIC[Fan et al. 2023] and RenderIH[Li et al. 2023]. ARCTIC provides two-hand object interacting images, and RenderIH provides interactive hands under different lighting conditions. We have demonstrated the results from both the camera perspective and an arbitrary perspective. The results show that our method outperforms the state-of-the-art methods in complex real-world environments.</div>

### 4.4 Qualitative Results

For monocular inputs, our qualitative comparisons are shown in Figure 3. We demonstrate results of in-the-wild datasets ARCTIC [Fan et al. 2023] and RenderIH [Li et al. 2023], which are unseen in the training phase. We compare with IntagHand[Li et al. 2022], ACR [Yu et al. 2023] and InterWild[Moon 2023] in camera perspective and an arbitrary perspective to show more details. We show more results on realistic complex cross-hand interacting samples in Figure 4, comparing with InterWild. The results demonstrate the stability of OmniHands in real-world scenarios and in complicated interaction cases.

For multi-view inputs, the qualitative results are shown in Figure 5, where 4 camera views are selected as demonstration. In every perspective, the hands are obscured to varying degrees. It can be observed that by integrating multi-view information, the hand interaction information is mutually supplemented, allowing successful reconstruction of the hand mesh corresponding to the respective view, even when the hands are completely occluded in some perspectives. It is worth noting that due to the absence of camera parameter

---

![Image](images/864ce0ca5b8ca1cb8b58c64397b182d4f3229abcd83b5334b6cdb252c6abb79b.jpg)

<div style="text-align: center;">Fig. 4. Qualitative Results of in-the-wild hard cases. We show our model's results on complex cross-hand interactions in realistic scenarios to demonstrate its performance. For a detailed perception, views of multiple perspectives are demonstrated.</div>

inputs, our model needs to predict the camera angle solely based on visual information and determine the hand orientation in the presence of occlusion, which is a non-trivial achievement.

### 4.5 Comparison with state-of-the-art methods

To demonstrate the versatility and stability of OmniHands across various scenarios, we conducted comparative analyses with state-of-the-art methods in multiple domains, including temporal two-hand evaluation, frame-based two-hand evaluation, temporal single hand evaluation, frame-based single hand evaluation and in-the-wild dataset evaluation.

Comparison with Temporal Two Hand Methods. In Table 1 we compare the proposed OmniHands with previous temporal methods on Interhand2.6m 30fps [Moon et al. 2020]. To demonstrate the benefits of using large mixed training datasets for the model, as well as the impact of using temporal features on model accuracy, we trained four groups of models based on whether additional training data and temporal features were used. For the training datasets, we use Interhand2.6m [Moon et al. 2020], Re:Interhand [Moon et al. 2024], and Single Hand Datasets mixed up as our training set for a better model generalization. To ensure a fair comparison with other two-hand reconstruction methods, which are trained only on Interhand2.6m, we use the same training set to train our model and the results are also shown marked with †. As previously mentioned, OmniHands can take sequential frames or single frame as input, to figure out the impact of using temporal features, we report and compare the results of OmniHands with single frame and sequential frames as input, denoting single-frame version with §. The results show that our method far outperforms the existing two-handed temporal models, with 23.34% (7.37 mm vs. 9.09 mm) improvement in MPJPE and 24.05% (2.81 mm/s² vs. 3.70 mm/s²) improvement in Accel_E. Comparing the results of whether using temporal features, it can be observed that using temporal feature results in better stability between frames and improvement of key points accuracy. The Accel_E improves 18.86% (2.81 mm/s² vs. 3.34 mm/s²) and 17.55% (3.02 mm/s² vs. 3.55 mm/s²) respectively, and there is also a notable improvement on MPJPE and MPVPE. By changing the training set, we can find that using extra training data can obviously improve the accuracy and stability.

Comparison with Frame-based Two Hand Methods. To compare with state-of-the-art frame-based methods and demonstrate

---

the performance in single frame input scenarios, we conduct experiments at Interhand2.6m 5fps [Moon et al. 2020] by setting the sequence length to 1. As can be observed at Table 2, even though we drop the temporal knowledge, our method still keep high performance and outperforms previous methods that were specially designed for single frame input scenarios, achieving 7.49 mm, 7.72 mm and 24.58 mm in MPJPE, MPVPE and MRRPE. The improvement of our model becomes more significant when trained on the full dataset, achieving 6.77 mm, 6.96 mm and 22.01 mm in MPJPE, MPVPE and MRRPE. We also report the results of multi-view frame inputs, without any camera parameter, our model achieves 5.93 mm, 6.20 mm and 14.08 mm in MPJPE, MPVPE and MRRPE, which significantly surpasses the accuracy of monocular estimation.

<div style="text-align: center;">Table 1. Quantitative comparisons with frame-based and temporal methods on Interhand2.6m 30fps dataset. The sign † stands for using only Interhand2.6m 30fps as training set, and § stands for using frame-based input.</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'>Methods</td><td style='text-align: center; word-wrap: break-word;'>Accel $E\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>MPJPE $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>MPVPE $\downarrow$</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>HPS [Boukhayma et al. 2019]</td><td style='text-align: center; word-wrap: break-word;'>8.14</td><td style='text-align: center; word-wrap: break-word;'>19.40</td><td style='text-align: center; word-wrap: break-word;'>19.78</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>ITH-3D[Zhang et al. 2021a]</td><td style='text-align: center; word-wrap: break-word;'>6.01</td><td style='text-align: center; word-wrap: break-word;'>14.45</td><td style='text-align: center; word-wrap: break-word;'>14.74</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>SeqHand[Yang et al. 2020]</td><td style='text-align: center; word-wrap: break-word;'>6.69</td><td style='text-align: center; word-wrap: break-word;'>18.20</td><td style='text-align: center; word-wrap: break-word;'>19.46</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Zhao et. al.[Zhao et al. 2024]</td><td style='text-align: center; word-wrap: break-word;'>3.70</td><td style='text-align: center; word-wrap: break-word;'>13.45</td><td style='text-align: center; word-wrap: break-word;'>13.91</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Intaghand[Li et al. 2022]</td><td style='text-align: center; word-wrap: break-word;'>4.40</td><td style='text-align: center; word-wrap: break-word;'>9.26</td><td style='text-align: center; word-wrap: break-word;'>9.71</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>DIR[Ren et al. 2023]</td><td style='text-align: center; word-wrap: break-word;'>8.92</td><td style='text-align: center; word-wrap: break-word;'>9.09</td><td style='text-align: center; word-wrap: break-word;'>9.43</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Ours $\dagger$ §</td><td style='text-align: center; word-wrap: break-word;'>3.55</td><td style='text-align: center; word-wrap: break-word;'>7.99</td><td style='text-align: center; word-wrap: break-word;'>8.15</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Ours $\dagger$</td><td style='text-align: center; word-wrap: break-word;'>3.02</td><td style='text-align: center; word-wrap: break-word;'>7.59</td><td style='text-align: center; word-wrap: break-word;'>7.79</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Ours §</td><td style='text-align: center; word-wrap: break-word;'>3.34</td><td style='text-align: center; word-wrap: break-word;'>7.80</td><td style='text-align: center; word-wrap: break-word;'>7.95</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Ours</td><td style='text-align: center; word-wrap: break-word;'>2.81</td><td style='text-align: center; word-wrap: break-word;'>7.37</td><td style='text-align: center; word-wrap: break-word;'>7.58</td></tr></table>

<div style="text-align: center;">Table 2. Performance comparisons on the Interhand2.6m 5fps dataset. We compare with the most recent methods. 'Ours†' was trained on Interhand2.6m 5fps. 'Ours' was trained on mixed dataset. 'Ours◇' takes multiview images as input and was trained on Interhand2.6m 5fps.</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'>Methods</td><td style='text-align: center; word-wrap: break-word;'>MPJPE $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>MPVPE $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>MRRPE $\downarrow$</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>InterWild[Moon 2023]</td><td style='text-align: center; word-wrap: break-word;'>14.76</td><td style='text-align: center; word-wrap: break-word;'>13.24</td><td style='text-align: center; word-wrap: break-word;'>26.25</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>IntagHand[Li et al. 2022]</td><td style='text-align: center; word-wrap: break-word;'>8.79</td><td style='text-align: center; word-wrap: break-word;'>9.02</td><td style='text-align: center; word-wrap: break-word;'>27.74</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>MEMAHand[Wang et al. 2023]</td><td style='text-align: center; word-wrap: break-word;'>8.65</td><td style='text-align: center; word-wrap: break-word;'>8.89</td><td style='text-align: center; word-wrap: break-word;'>-</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>ACR[Yu et al. 2023]</td><td style='text-align: center; word-wrap: break-word;'>8.41</td><td style='text-align: center; word-wrap: break-word;'>8.53</td><td style='text-align: center; word-wrap: break-word;'>-</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Zuo et.al[Zuo et al. 2023]</td><td style='text-align: center; word-wrap: break-word;'>8.34</td><td style='text-align: center; word-wrap: break-word;'>8.51</td><td style='text-align: center; word-wrap: break-word;'>-</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>DIR[Ren et al. 2023]</td><td style='text-align: center; word-wrap: break-word;'>7.51</td><td style='text-align: center; word-wrap: break-word;'>7.72</td><td style='text-align: center; word-wrap: break-word;'>28.98</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Ours $^{\dagger}$</td><td style='text-align: center; word-wrap: break-word;'>7.49</td><td style='text-align: center; word-wrap: break-word;'>7.72</td><td style='text-align: center; word-wrap: break-word;'>24.58</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Ours</td><td style='text-align: center; word-wrap: break-word;'>6.77</td><td style='text-align: center; word-wrap: break-word;'>6.96</td><td style='text-align: center; word-wrap: break-word;'>22.01</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Ours $\diamond$</td><td style='text-align: center; word-wrap: break-word;'>5.93</td><td style='text-align: center; word-wrap: break-word;'>6.20</td><td style='text-align: center; word-wrap: break-word;'>14.08</td></tr></table>

Comparison on Single hand Dataset. Although our framework is primarily designed for interacting hands reconstruction, it also achieves performance comparable to state-of-the-art single-hand methods on the task of single-hand reconstruction. We test OmniHands on 2 widely used benchmarks: Freihand [Zimmermann et al. 2019] and HO3Dv2 [Hampali et al. 2020], with the results presented in Table 6, Table 7. For a fair comparison, we adopt the same training settings as Hamba [Dong et al. 2024]. Specifically, the model in Table 6 is trained only using the FreiHAND dataset, and the model in Table 7 is trained on the same datasets as HaMeR [Pavlakos et al. 2023].

In the task of interacting-hand reconstruction, we primarily use MRRPE to evaluate the accuracy of the relative root positions, specifically the wrist positions, between the two hands as an indicator of interaction accuracy. However, such a numerical metric does not effectively capture the front-back ordering of the fingers between the hands, which can lead to noticeable differences in visual quality. Therefore, we propose Mean Inter-Joint Error (MIJE) to evaluate the hand interaction performance. The MIJE is computed as follows:

 $$ \mathbf{M I J E}=\frac{1}{\left|J_{r}\right|\cdot\left|J_{l}\right|}\sum_{j_{r}\in J_{r}}\sum_{j_{l}\in J_{l}}\|j_{r}-j_{l}\|_{2} $$

We evaluated this metric on InterHand2.6M and compared it with state-of-the-art methods, the results are reported in Table 5. As shown, our method achieves notably better performance in handling complex two-hand interaction poses. This advantage is also reflected in other qualitative results.

Comparison with Temporal Single Hand Methods. To further exploit the versatility of our method, we conduct experiments on temporal single hand dataset Dex-YCB [Chao et al. 2021]. We manage our input by flipping an input image as the opposite hand, and using far away bounding boxes to indicate the hands are not interacting. We report the results of Root-Aligned and Procrustes-aligned metrics. It can be observed in Table 8 that our method outperforms all existing single hand temporal methods in all metrics. It is noteworthy that our lead in Root-Aligned metrics is greater than those in Procrustes-aligned metrics (+38.63% vs. +9.00% in MPJPE, +12.70% vs. +0.89% in JAUC), indicating that we can predict accurate shapes and rotation angles without the assistance of ground truth alignment.

Quantitative Comparison on In-the-wild Datasets. We report the results on three in-the-wild datasets: Arctic [Fan et al. 2023], HIC [Tzionas et al. 2016] and RenderIH [Li et al. 2023], to compare the performance at in-the-wild scenarios. Arctic contains images of two hand interacting with objects, including a large number of cases with hand-object occlusion and hand-hand occlusion. HIC is a small dataset of two hand interaction sequences. And RenderIH contains images of complicated two hand interactions under various illumination conditions. We only use samples in which both hands are visible. For fair comparison, temporal data or multi-view data are not used in these experiments, a single image is taken as input for each case.

In Table 4 we compare the results with IntagHand [Li et al. 2022], InterWild [Moon 2023] and DIR [Ren et al. 2023]. IntagHand [Li et al. 2022] and DIR [Ren et al. 2023] are two open-source inter-hand reconstruction models that have demonstrated excellent performance in training and testing on Interhand2.6m [Moon et al. 2020]. InterWild [Moon 2023] is a SOTA method specifically designed for two-hand recovery in the wild. To demonstrate the efficiency of using large training data, we report results of models trained under two training set. One of our reported models is trained solely on InterHand2.6M, using the same training set as other comparison methods to enable a fair quantitative comparison; this model is marked with $\dagger$. The other unmarked model is trained on a mixed

---

<div style="text-align: center;">Table 3. Multi-view single hand recovery results on Dex-YCB, comparing with other single hand multi-view methods. The model reported was trained on Dex-YCB.</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'>Method</td><td style='text-align: center; word-wrap: break-word;'>Ra-MPJPE $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>Pa-MPJPE $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>Ra-JAUC $\uparrow$</td><td style='text-align: center; word-wrap: break-word;'>Ra-MPVPE $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>Pa-MPVPE $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>Ra-VAUC $\uparrow$</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>MVP[Zhang et al. 2021a]</td><td style='text-align: center; word-wrap: break-word;'>6.23</td><td style='text-align: center; word-wrap: break-word;'>4.26</td><td style='text-align: center; word-wrap: break-word;'>0.69</td><td style='text-align: center; word-wrap: break-word;'>9.77</td><td style='text-align: center; word-wrap: break-word;'>8.14</td><td style='text-align: center; word-wrap: break-word;'>0.53</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>PE-Mesh-TR[Yang et al. 2023]</td><td style='text-align: center; word-wrap: break-word;'>7.49</td><td style='text-align: center; word-wrap: break-word;'>4.76</td><td style='text-align: center; word-wrap: break-word;'>0.64</td><td style='text-align: center; word-wrap: break-word;'>7.41</td><td style='text-align: center; word-wrap: break-word;'>4.70</td><td style='text-align: center; word-wrap: break-word;'>0.64</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>FTL-Mesh-TR[Yang et al. 2023]</td><td style='text-align: center; word-wrap: break-word;'>8.66</td><td style='text-align: center; word-wrap: break-word;'>5.51</td><td style='text-align: center; word-wrap: break-word;'>0.59</td><td style='text-align: center; word-wrap: break-word;'>8.75</td><td style='text-align: center; word-wrap: break-word;'>5.75</td><td style='text-align: center; word-wrap: break-word;'>0.59</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>POEM[Yang et al. 2023]</td><td style='text-align: center; word-wrap: break-word;'>6.06</td><td style='text-align: center; word-wrap: break-word;'>3.93</td><td style='text-align: center; word-wrap: break-word;'>0.68</td><td style='text-align: center; word-wrap: break-word;'>6.13</td><td style='text-align: center; word-wrap: break-word;'>4.00</td><td style='text-align: center; word-wrap: break-word;'>0.70</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Multi-view Fit.[Yang et al. 2023]</td><td style='text-align: center; word-wrap: break-word;'>7.33</td><td style='text-align: center; word-wrap: break-word;'>5.29</td><td style='text-align: center; word-wrap: break-word;'>0.65</td><td style='text-align: center; word-wrap: break-word;'>7.22</td><td style='text-align: center; word-wrap: break-word;'>5.19</td><td style='text-align: center; word-wrap: break-word;'>0.65</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Ours</td><td style='text-align: center; word-wrap: break-word;'>6.44</td><td style='text-align: center; word-wrap: break-word;'>3.18</td><td style='text-align: center; word-wrap: break-word;'>0.87</td><td style='text-align: center; word-wrap: break-word;'>6.42</td><td style='text-align: center; word-wrap: break-word;'>3.26</td><td style='text-align: center; word-wrap: break-word;'>0.87</td></tr></table>

<div style="text-align: center;">Table 4. Comparison of two-hand reconstruction methods on in-the-wild datasets. Arctic, HIC and RenderlH are taken as test set, which are unseen during training. Our method marked with  $\dagger$ is trained on only Interhand2.6m, and the unmarked one is trained on mixed training set.</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td rowspan="2">Dataset</td><td rowspan="2">Method</td><td colspan="5">Joints</td><td colspan="4">Vertices</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Ra-MPJPE $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>Pa-MPJPE $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>Ra-JAUC $\uparrow$</td><td style='text-align: center; word-wrap: break-word;'>Pa-JAUC $\uparrow$</td><td style='text-align: center; word-wrap: break-word;'>MRRPE $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>Ra-MPVPE $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>Pa-MPVPE $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>Ra-VAUC $\uparrow$</td><td style='text-align: center; word-wrap: break-word;'>Pa-VAUC $\uparrow$</td></tr><tr><td rowspan="5">Arctic[Fan et al. 2023]</td><td style='text-align: center; word-wrap: break-word;'>IntagHand[Li et al. 2022]</td><td style='text-align: center; word-wrap: break-word;'>24.88</td><td style='text-align: center; word-wrap: break-word;'>12.02</td><td style='text-align: center; word-wrap: break-word;'>0.57</td><td style='text-align: center; word-wrap: break-word;'>0.76</td><td style='text-align: center; word-wrap: break-word;'>203.04</td><td style='text-align: center; word-wrap: break-word;'>26.30</td><td style='text-align: center; word-wrap: break-word;'>12.52</td><td style='text-align: center; word-wrap: break-word;'>0.55</td><td style='text-align: center; word-wrap: break-word;'>0.75</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>InterWild[Moon 2023]</td><td style='text-align: center; word-wrap: break-word;'>22.74</td><td style='text-align: center; word-wrap: break-word;'>11.31</td><td style='text-align: center; word-wrap: break-word;'>0.59</td><td style='text-align: center; word-wrap: break-word;'>0.72</td><td style='text-align: center; word-wrap: break-word;'>143.19</td><td style='text-align: center; word-wrap: break-word;'>23.92</td><td style='text-align: center; word-wrap: break-word;'>11.74</td><td style='text-align: center; word-wrap: break-word;'>0.56</td><td style='text-align: center; word-wrap: break-word;'>0.71</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>DIR[Ren et al. 2023]</td><td style='text-align: center; word-wrap: break-word;'>26.63</td><td style='text-align: center; word-wrap: break-word;'>11.97</td><td style='text-align: center; word-wrap: break-word;'>0.55</td><td style='text-align: center; word-wrap: break-word;'>0.76</td><td style='text-align: center; word-wrap: break-word;'>253.49</td><td style='text-align: center; word-wrap: break-word;'>28.33</td><td style='text-align: center; word-wrap: break-word;'>12.39</td><td style='text-align: center; word-wrap: break-word;'>0.52</td><td style='text-align: center; word-wrap: break-word;'>0.75</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>ours †</td><td style='text-align: center; word-wrap: break-word;'>14.21</td><td style='text-align: center; word-wrap: break-word;'>9.22</td><td style='text-align: center; word-wrap: break-word;'>0.69</td><td style='text-align: center; word-wrap: break-word;'>0.78</td><td style='text-align: center; word-wrap: break-word;'>112.81</td><td style='text-align: center; word-wrap: break-word;'>15.12</td><td style='text-align: center; word-wrap: break-word;'>9.76</td><td style='text-align: center; word-wrap: break-word;'>0.63</td><td style='text-align: center; word-wrap: break-word;'>0.81</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>ours</td><td style='text-align: center; word-wrap: break-word;'>13.57</td><td style='text-align: center; word-wrap: break-word;'>7.98</td><td style='text-align: center; word-wrap: break-word;'>0.74</td><td style='text-align: center; word-wrap: break-word;'>0.84</td><td style='text-align: center; word-wrap: break-word;'>105.43</td><td style='text-align: center; word-wrap: break-word;'>14.51</td><td style='text-align: center; word-wrap: break-word;'>8.75</td><td style='text-align: center; word-wrap: break-word;'>0.72</td><td style='text-align: center; word-wrap: break-word;'>0.82</td></tr><tr><td rowspan="5">HIC[Tzionas et al. 2016]</td><td style='text-align: center; word-wrap: break-word;'>IntagHand[Li et al. 2022]</td><td style='text-align: center; word-wrap: break-word;'>12.01</td><td style='text-align: center; word-wrap: break-word;'>7.46</td><td style='text-align: center; word-wrap: break-word;'>0.76</td><td style='text-align: center; word-wrap: break-word;'>0.85</td><td style='text-align: center; word-wrap: break-word;'>36.46</td><td style='text-align: center; word-wrap: break-word;'>12.77</td><td style='text-align: center; word-wrap: break-word;'>7.82</td><td style='text-align: center; word-wrap: break-word;'>0.74</td><td style='text-align: center; word-wrap: break-word;'>0.84</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>InterWild[Moon 2023]</td><td style='text-align: center; word-wrap: break-word;'>11.23</td><td style='text-align: center; word-wrap: break-word;'>6.62</td><td style='text-align: center; word-wrap: break-word;'>0.79</td><td style='text-align: center; word-wrap: break-word;'>0.86</td><td style='text-align: center; word-wrap: break-word;'>30.35</td><td style='text-align: center; word-wrap: break-word;'>11.65</td><td style='text-align: center; word-wrap: break-word;'>6.99</td><td style='text-align: center; word-wrap: break-word;'>0.76</td><td style='text-align: center; word-wrap: break-word;'>0.86</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>DIR [Ren et al. 2023]</td><td style='text-align: center; word-wrap: break-word;'>11.90</td><td style='text-align: center; word-wrap: break-word;'>7.12</td><td style='text-align: center; word-wrap: break-word;'>0.76</td><td style='text-align: center; word-wrap: break-word;'>0.85</td><td style='text-align: center; word-wrap: break-word;'>32.83</td><td style='text-align: center; word-wrap: break-word;'>12.79</td><td style='text-align: center; word-wrap: break-word;'>7.55</td><td style='text-align: center; word-wrap: break-word;'>0.75</td><td style='text-align: center; word-wrap: break-word;'>0.84</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>ours †</td><td style='text-align: center; word-wrap: break-word;'>9.47</td><td style='text-align: center; word-wrap: break-word;'>5.55</td><td style='text-align: center; word-wrap: break-word;'>0.81</td><td style='text-align: center; word-wrap: break-word;'>0.88</td><td style='text-align: center; word-wrap: break-word;'>26.84</td><td style='text-align: center; word-wrap: break-word;'>10.35</td><td style='text-align: center; word-wrap: break-word;'>6.07</td><td style='text-align: center; word-wrap: break-word;'>0.79</td><td style='text-align: center; word-wrap: break-word;'>0.87</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>ours</td><td style='text-align: center; word-wrap: break-word;'>8.66</td><td style='text-align: center; word-wrap: break-word;'>4.81</td><td style='text-align: center; word-wrap: break-word;'>0.82</td><td style='text-align: center; word-wrap: break-word;'>0.90</td><td style='text-align: center; word-wrap: break-word;'>26.56</td><td style='text-align: center; word-wrap: break-word;'>9.31</td><td style='text-align: center; word-wrap: break-word;'>5.30</td><td style='text-align: center; word-wrap: break-word;'>0.81</td><td style='text-align: center; word-wrap: break-word;'>0.89</td></tr><tr><td rowspan="5">RenderIH[Li et al. 2023]</td><td style='text-align: center; word-wrap: break-word;'>IntagHand[Li et al. 2022]</td><td style='text-align: center; word-wrap: break-word;'>33.84</td><td style='text-align: center; word-wrap: break-word;'>15.19</td><td style='text-align: center; word-wrap: break-word;'>0.47</td><td style='text-align: center; word-wrap: break-word;'>0.70</td><td style='text-align: center; word-wrap: break-word;'>80.28</td><td style='text-align: center; word-wrap: break-word;'>35.17</td><td style='text-align: center; word-wrap: break-word;'>15.34</td><td style='text-align: center; word-wrap: break-word;'>0.45</td><td style='text-align: center; word-wrap: break-word;'>0.69</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>InterWild[Moon 2023]</td><td style='text-align: center; word-wrap: break-word;'>19.12</td><td style='text-align: center; word-wrap: break-word;'>13.16</td><td style='text-align: center; word-wrap: break-word;'>0.62</td><td style='text-align: center; word-wrap: break-word;'>0.72</td><td style='text-align: center; word-wrap: break-word;'>56.10</td><td style='text-align: center; word-wrap: break-word;'>19.74</td><td style='text-align: center; word-wrap: break-word;'>12.77</td><td style='text-align: center; word-wrap: break-word;'>0.61</td><td style='text-align: center; word-wrap: break-word;'>0.73</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>DIR[Ren et al. 2023]</td><td style='text-align: center; word-wrap: break-word;'>29.48</td><td style='text-align: center; word-wrap: break-word;'>12.93</td><td style='text-align: center; word-wrap: break-word;'>0.53</td><td style='text-align: center; word-wrap: break-word;'>0.74</td><td style='text-align: center; word-wrap: break-word;'>78.53</td><td style='text-align: center; word-wrap: break-word;'>30.93</td><td style='text-align: center; word-wrap: break-word;'>12.89</td><td style='text-align: center; word-wrap: break-word;'>0.51</td><td style='text-align: center; word-wrap: break-word;'>0.74</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>ours †</td><td style='text-align: center; word-wrap: break-word;'>17.22</td><td style='text-align: center; word-wrap: break-word;'>10.91</td><td style='text-align: center; word-wrap: break-word;'>0.66</td><td style='text-align: center; word-wrap: break-word;'>0.78</td><td style='text-align: center; word-wrap: break-word;'>45.94</td><td style='text-align: center; word-wrap: break-word;'>18.01</td><td style='text-align: center; word-wrap: break-word;'>11.11</td><td style='text-align: center; word-wrap: break-word;'>0.64</td><td style='text-align: center; word-wrap: break-word;'>0.77</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>ours</td><td style='text-align: center; word-wrap: break-word;'>13.71</td><td style='text-align: center; word-wrap: break-word;'>9.17</td><td style='text-align: center; word-wrap: break-word;'>0.72</td><td style='text-align: center; word-wrap: break-word;'>0.81</td><td style='text-align: center; word-wrap: break-word;'>42.26</td><td style='text-align: center; word-wrap: break-word;'>14.32</td><td style='text-align: center; word-wrap: break-word;'>9.48</td><td style='text-align: center; word-wrap: break-word;'>0.71</td><td style='text-align: center; word-wrap: break-word;'>0.81</td></tr></table>

<div style="text-align: center;">Table 5. The Mean Inter-Joint Error (MIJE) metric, measured in millimeters, of our method compared to the SOTA in two-hand reconstruction on the InterHand2.6M dataset. Lower values indicate better performance.</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'>Ours</td><td style='text-align: center; word-wrap: break-word;'>IntagHand</td><td style='text-align: center; word-wrap: break-word;'>InterWild</td><td style='text-align: center; word-wrap: break-word;'>DIR</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>26.56</td><td style='text-align: center; word-wrap: break-word;'>32.36</td><td style='text-align: center; word-wrap: break-word;'>58.51</td><td style='text-align: center; word-wrap: break-word;'>73.40</td></tr></table>

<div style="text-align: center;">Table 6. Quantitative comparisons with other single hand reconstruction methods on Freihand dataset.</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'>Method</td><td style='text-align: center; word-wrap: break-word;'>PA-MPJPE $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>PA-MPVPE $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>F@5mm $\uparrow$</td><td style='text-align: center; word-wrap: break-word;'>F@15mm $\uparrow$</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>PointHMR[Kim et al. 2023]</td><td style='text-align: center; word-wrap: break-word;'>6.1</td><td style='text-align: center; word-wrap: break-word;'>6.6</td><td style='text-align: center; word-wrap: break-word;'>0.720</td><td style='text-align: center; word-wrap: break-word;'>0.984</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>zhou et. al [Zhou et al. 2024]</td><td style='text-align: center; word-wrap: break-word;'>5.7</td><td style='text-align: center; word-wrap: break-word;'>6.0</td><td style='text-align: center; word-wrap: break-word;'>0.772</td><td style='text-align: center; word-wrap: break-word;'>0.986</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>HaMeR [Pavlakos et al. 2023]</td><td style='text-align: center; word-wrap: break-word;'>6.0</td><td style='text-align: center; word-wrap: break-word;'>5.7</td><td style='text-align: center; word-wrap: break-word;'>0.785</td><td style='text-align: center; word-wrap: break-word;'>0.990</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Hamba [Dong et al. 2024]</td><td style='text-align: center; word-wrap: break-word;'>5.8</td><td style='text-align: center; word-wrap: break-word;'>5.5</td><td style='text-align: center; word-wrap: break-word;'>0.798</td><td style='text-align: center; word-wrap: break-word;'>0.991</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Ours</td><td style='text-align: center; word-wrap: break-word;'>5.2</td><td style='text-align: center; word-wrap: break-word;'>5.7</td><td style='text-align: center; word-wrap: break-word;'>0.779</td><td style='text-align: center; word-wrap: break-word;'>0.993</td></tr></table>

<div style="text-align: center;">Table 7. Quantitative comparisons with other single hand reconstruction methods on HO3Dv2 dataset.</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'>Method</td><td style='text-align: center; word-wrap: break-word;'>PA-MPJPE $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>PA-MPVPE $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>F@5mm $\uparrow$</td><td style='text-align: center; word-wrap: break-word;'>F@15mm $\uparrow$</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>H2ONet [Xu et al. 2023]</td><td style='text-align: center; word-wrap: break-word;'>8.5</td><td style='text-align: center; word-wrap: break-word;'>8.6</td><td style='text-align: center; word-wrap: break-word;'>0.966</td><td style='text-align: center; word-wrap: break-word;'>0.570</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>HandBooster [Xu et al. 2024]</td><td style='text-align: center; word-wrap: break-word;'>8.2</td><td style='text-align: center; word-wrap: break-word;'>8.4</td><td style='text-align: center; word-wrap: break-word;'>0.972</td><td style='text-align: center; word-wrap: break-word;'>0.585</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>HaMeR [Pavlakos et al. 2023]</td><td style='text-align: center; word-wrap: break-word;'>7.6</td><td style='text-align: center; word-wrap: break-word;'>7.9</td><td style='text-align: center; word-wrap: break-word;'>0.980</td><td style='text-align: center; word-wrap: break-word;'>0.635</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Hamba [Dong et al. 2024]</td><td style='text-align: center; word-wrap: break-word;'>7.5</td><td style='text-align: center; word-wrap: break-word;'>7.7</td><td style='text-align: center; word-wrap: break-word;'>0.982</td><td style='text-align: center; word-wrap: break-word;'>0.648</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Ours</td><td style='text-align: center; word-wrap: break-word;'>6.9</td><td style='text-align: center; word-wrap: break-word;'>7.7</td><td style='text-align: center; word-wrap: break-word;'>0.985</td><td style='text-align: center; word-wrap: break-word;'>0.616</td></tr></table>

dataset comprising InterHand2.6M, Re:InterHand, and single hand datasets.

<div style="text-align: center;">Table 8. Quantitative comparisons with other single hand temporal reconstruction methods on the Dex-YCB dataset. The model reported was trained on Dex-YCB</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'>Methods</td><td style='text-align: center; word-wrap: break-word;'>Ra-MAJPE $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>Ra-JAUC $\uparrow$</td><td style='text-align: center; word-wrap: break-word;'>Pa-MPJPE $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>Pa-JAUC $\uparrow$</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>HaMeR [Pavlakos et al. 2023]</td><td style='text-align: center; word-wrap: break-word;'>19.45</td><td style='text-align: center; word-wrap: break-word;'>68.4</td><td style='text-align: center; word-wrap: break-word;'>6.09</td><td style='text-align: center; word-wrap: break-word;'>87.8</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>VIBE[Kocabas et al. 2020]</td><td style='text-align: center; word-wrap: break-word;'>16.95</td><td style='text-align: center; word-wrap: break-word;'>67.5</td><td style='text-align: center; word-wrap: break-word;'>6.43</td><td style='text-align: center; word-wrap: break-word;'>87.1</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>TCMR[Choi et al. 2021]</td><td style='text-align: center; word-wrap: break-word;'>16.03</td><td style='text-align: center; word-wrap: break-word;'>70.1</td><td style='text-align: center; word-wrap: break-word;'>6.28</td><td style='text-align: center; word-wrap: break-word;'>87.5</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Tu et al. [Tu et al. 2023]</td><td style='text-align: center; word-wrap: break-word;'>19.67</td><td style='text-align: center; word-wrap: break-word;'>62.5</td><td style='text-align: center; word-wrap: break-word;'>7.27</td><td style='text-align: center; word-wrap: break-word;'>85.5</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>MeshGraphormer[Lin et al. 2021]</td><td style='text-align: center; word-wrap: break-word;'>16.21</td><td style='text-align: center; word-wrap: break-word;'>69.1</td><td style='text-align: center; word-wrap: break-word;'>6.41</td><td style='text-align: center; word-wrap: break-word;'>87.2</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Deformer[Fu et al. 2023]</td><td style='text-align: center; word-wrap: break-word;'>13.64</td><td style='text-align: center; word-wrap: break-word;'>74.0</td><td style='text-align: center; word-wrap: break-word;'>5.22</td><td style='text-align: center; word-wrap: break-word;'>89.6</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Ours</td><td style='text-align: center; word-wrap: break-word;'>8.37</td><td style='text-align: center; word-wrap: break-word;'>83.4</td><td style='text-align: center; word-wrap: break-word;'>4.75</td><td style='text-align: center; word-wrap: break-word;'>90.4</td></tr></table>

In all three unseen datasets, our method achieves the best performance and leads in all metrics. It can also be found that by using large training set, our model can handle situations in more diverse scenarios, resulting in higher accuracy in all three testing sets. HIC [Tzionas et al. 2016] includes interactive gestures that are not present in Interhand2.6m. However, the scenes are relatively simple and the test samples are few, allowing other methods to also achieve good results. In HIC our method improves 16.33% (8.66 mm vs. 10.35 mm) on Ra-MPJPE and 29.47% (4.81 mm vs. 3.34 mm) on Pa-MPJPE. Arctic [Fan et al. 2023] includes a large number of images of two hands interacting with objects, which significantly differ from the data in Interhand2.6m. Additionally, there are many instances where parts of the hands are occluded, making the situation more complex. In this dataset, other methods performed poorly, whereas our method maintained robustness and achieved excellent results.

---

![Image](images/bc23716f577c4c6ec7775ba138ff0716fb9d403264193736f2b1f8e29639cd04.jpg)

<div style="text-align: center;">Fig. 5. The qualitative results of OmniHands with multi-view inputs. We present the results on Interhand2.6m [Moon et al. 2020] at row 1 to row 5 and results on Arctic [Fan et al. 2023] at row 6 to row 9. In each row, we present pairs of input and output images from 4 different camera views. The images on the left side of each pair serve as the multi-view sequence input.</div>

---

improving 45.46% (13.57 mm vs. 24.88 mm) on Ra-MPJPE and 33.61% (7.98 mm vs. 12.02 mm) on Pa-MPJPE. In RenderIH [Li et al. 2023], the diversity of lighting and backgrounds places high demands on the cross-scene robustness of the model. Our proposed OmniHands obtains 13.89% (13.71 mm vs. 15.92 mm) improvement on Ra-MPJPE and 13.00% (9.17 mm vs. 10.54 mm) on Pa-MPJPE.

Comparison on Single hand Multi-view Dataset. To demonstrate the performance of our model when taking multi-view images as input, in Table 3 we compare with other multi-view single hand methods on Dex-YCB [Chao et al. 2021]. For a fair comparison, our model is trained on Dex-YCB. We also compare with Multi-view pose estimation reported in [Yang et al. 2023]. It's worth mentioning that other methods require camera parameters as input, which makes it easier for them to align different views but loses flexibility. Our method, on the other hand, does not need camera parameters as input. It only requires multiple images from different viewpoints, eliminating the tedious camera calibration steps while achieving higher accuracy than other methods.

### 4.6 Ablation Study

We evaluate the importance of each module in OmniHands by ablation experiments in temporal input conditions, using temporal dataset Interhand2.6m 30fps as training set and testing set. Moreover, we also conduct evaluations on the impact of camera numbers when taking multi-view images as input, using Interhand2.6m 5fps multi-view dataset.

4.6.1 Module Ablation. ViT Baseline. We trained a baseline using ViT and a transformer decoder on the full training set. Position map embedding, cross-hand feature interaction and temporal/multi-view interaction are not employed on the baseline, so the Bi-manual Decoder, Collective Decoder, and Individual Encoder are removed. We show the results in Table 9. Although ViT has high performance in detecting single hand, the occlusion caused by hands interaction reduces its predictive ability. Besides, without cross-hand knowledge, it separately predicts the positions of two hands, causing huge errors in relative position.

Effectiveness of Cross-hand Information. We experimented in adding cross-hand information in two ways: (1) Input holistic image including two hands(+Cross H). (2) Input separate images, each image contains a single hand(+Cross S). In both way we add Bimanual Decoder module to integrate cross-hand information, the results show that the last way achieves better accuracy. The main reason is that by separating two hands as input, the visual encoder focuses on extracting single hand features, allowing the ViT to learn more hand details when learning on large scale datasets.

Effectiveness of Relation-aware Tokenization. By removing position token map embedding in RAT(+Cross S w/o RAT), the model lost global position information during inference, which means that the model could only infer two-hand relative positions from visual information. In InterHand2.6m [Moon et al. 2020], the two hands are relatively close and the 2D scales of each hand are similar. Therefore, even without a global position information, the approximate positions of both hands can be inferred from the input image. We test cases when two hands are relatively far away and cases when 2D scales of each hand are different. As shown in Table 7,

![Image](images/92ec942ff1def01b6e22f64891562a43c6c3ae0e95f65fdcf4c038214696a492.jpg)

<div style="text-align: center;">Fig. 6. Ablation study on occlusion case. 'Full Model' is our model with all modules. 'w/o Temporal Fusion' means that the temporal integrating module is removed.</div>

without position token map, the model performs poor in predicting two-hand relative position.

![Image](images/fb4a23c8d1a9f19db8d04cc1bda516e5e14523312087272f203cf0324539bd28.jpg)

<div style="text-align: center;">Fig. 7. Ablation study of RAT. Two views of predicted hand meshes are visualized to demonstrate details of relative position. 'Full Model' is our model with all modules. 'w/o RAT' means that the position embedding is removed.</div>

Effectiveness of Temporal Fusion. By adding Collective Decoder for temporal information interacting(+Cross S + Seq), our method can make sequential consistent prediction therefore infer reasonable poses encountering occlusions, and achieves improvement of 0.32 mm in MPJPE. In Figure 6 we visualize the predicted mesh sequences in the case of severe occlusion. It shows that the left hand is entirely occluded, the model without temporal fusion makes erroneous prediction. Our full model predicts a consistent sequence with assistant information from the past and the future.

4.6.2 Ablation on training data. In this section, we conduct ablation studies to evaluate the impact of scaling up training data. We benchmark our method on three datasets: InterHand2.6m [Moon]

---

et al. 2020], HIC [Tzionas et al. 2016], and Arctic [Fan et al. 2023], and compare against two SOTA baselines: IntagHand [Li et al. 2022] and InterWild [Moon 2023]. The results are presented in Table 11. For each method, we train the models on both the InterHand2.6M 5fps dataset and mixed datasets. Specifically, the mixed datasets for OmniHands include InterHand2.6M, Re:InterHand, and Static Single Hand Datasets. The static single hand datasets simulate a pair of widely separated hands by randomly combining two single-hand images. However, such input is not suitable for IntagHand or InterWild, as their input requires images that contain both hands. Therefore, for IntagHand and InterWild, the mixed datasets comprise InterHand2.6M and Re:InterHand. Concerning the mixed datasets, while OmniHands leverages additional single-hand datasets during training, all methods are trained on the same interacting-hand datasets. As the prediction of relative hand positions is learned from the interacting-hand data, the MRRPE metric serves as a valid indicator for comparing the interacting-hand reconstruction performance among the methods.

The experimental results demonstrate that OmniHands benefits from the increased data diversity, exhibiting improved performance when trained with the hybrid dataset. In contrast, IntagHand suffers performance degradation with additional training data. This is likely due to its architectural limitations, which prevent it from effectively generalizing under increased data scale and complexity, such as varying scenes and lighting conditions, ultimately undermining its reliance on strong priors. Overall, these results validate that OmniHands consistently outperforms SOTA methods, regardless of whether the model is trained on a single-domain dataset or a more diverse hybrid dataset.

4.6.3 Camera view Ablation. In multi-view reconstruction, the number of input camera views is important to the resulting accuracy. We tried to figure out the impact of camera views, and the experimental results are shown in Table 10. We divided the experimental results into two groups based on whether mixed training data was used. 'I' indicates the use of only Interhand2.6m[Moon et al. 2020] as the training data, while 'I+R+A' represents the use of a mixed training dataset comprising Interhand2.6m, Re:Interhand[Moon et al. 2024], and Arctic[Fan et al. 2023].

From the results presented in the table, it can be observed that simply switching from monocular input to 2-camera multi-view input significantly improves our prediction accuracy, and this improvement is even more pronounced when training with a mixed dataset. As the number of cameras increases from 2 to 16, the accuracy metrics improve accordingly regardless of the training set. This aligns with the intuitive understanding that more perspective leads to more accurate predictions. This demonstrates that our FIR module effectively utilizes 4D data to enhance prediction results and shows significant flexibility in handling 4D data of varying lengths. Additionally, by comparing the results of 'I' and 'I+R+A' in the table, it is evident that the accuracy with the mixed training dataset is significantly higher than with a single training dataset, irrespective of the number of cameras. This highlights our model's capability to handle large-scale mixed data efficiently.

<div style="text-align: center;">Table 9. Ablation studies of the RAT and FIR modules on InterHand2.6M.</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'>Methods</td><td style='text-align: center; word-wrap: break-word;'>MPJPE $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>MPVPE $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>MRRPE $\downarrow$</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Baseline</td><td style='text-align: center; word-wrap: break-word;'>10.94</td><td style='text-align: center; word-wrap: break-word;'>11.35</td><td style='text-align: center; word-wrap: break-word;'>137.53</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>+Cross H</td><td style='text-align: center; word-wrap: break-word;'>8.78</td><td style='text-align: center; word-wrap: break-word;'>9.01</td><td style='text-align: center; word-wrap: break-word;'>28.65</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>+Cross S</td><td style='text-align: center; word-wrap: break-word;'>7.69</td><td style='text-align: center; word-wrap: break-word;'>7.92</td><td style='text-align: center; word-wrap: break-word;'>24.79</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>+Cross S w/o RAT</td><td style='text-align: center; word-wrap: break-word;'>7.81</td><td style='text-align: center; word-wrap: break-word;'>8.00</td><td style='text-align: center; word-wrap: break-word;'>25.81</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>+Cross S + Seq (Ours)</td><td style='text-align: center; word-wrap: break-word;'>7.37</td><td style='text-align: center; word-wrap: break-word;'>7.58</td><td style='text-align: center; word-wrap: break-word;'>24.48</td></tr></table>

<div style="text-align: center;">Table 10. Results on Interhand2.6m using different number of camera views. "I" means only using Interhand2.6m training set, and "I+R+A" means using Interhand2.6m, Re:Interhand and Arctic as training set.</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'>Train</td><td style='text-align: center; word-wrap: break-word;'>cameras</td><td style='text-align: center; word-wrap: break-word;'>MPJPE $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>MPVPE $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>MRRPE $\downarrow$</td></tr><tr><td rowspan="5">I</td><td style='text-align: center; word-wrap: break-word;'>2</td><td style='text-align: center; word-wrap: break-word;'>7.02</td><td style='text-align: center; word-wrap: break-word;'>7.28</td><td style='text-align: center; word-wrap: break-word;'>20.59</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>4</td><td style='text-align: center; word-wrap: break-word;'>6.88</td><td style='text-align: center; word-wrap: break-word;'>7.13</td><td style='text-align: center; word-wrap: break-word;'>18.90</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>8</td><td style='text-align: center; word-wrap: break-word;'>6.79</td><td style='text-align: center; word-wrap: break-word;'>7.05</td><td style='text-align: center; word-wrap: break-word;'>17.91</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>12</td><td style='text-align: center; word-wrap: break-word;'>6.65</td><td style='text-align: center; word-wrap: break-word;'>6.93</td><td style='text-align: center; word-wrap: break-word;'>16.86</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>16</td><td style='text-align: center; word-wrap: break-word;'>6.49</td><td style='text-align: center; word-wrap: break-word;'>6.76</td><td style='text-align: center; word-wrap: break-word;'>17.08</td></tr><tr><td rowspan="5">I+R+A</td><td style='text-align: center; word-wrap: break-word;'>2</td><td style='text-align: center; word-wrap: break-word;'>6.51</td><td style='text-align: center; word-wrap: break-word;'>6.74</td><td style='text-align: center; word-wrap: break-word;'>17.72</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>4</td><td style='text-align: center; word-wrap: break-word;'>6.28</td><td style='text-align: center; word-wrap: break-word;'>6.55</td><td style='text-align: center; word-wrap: break-word;'>16.49</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>8</td><td style='text-align: center; word-wrap: break-word;'>6.21</td><td style='text-align: center; word-wrap: break-word;'>6.47</td><td style='text-align: center; word-wrap: break-word;'>16.26</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>12</td><td style='text-align: center; word-wrap: break-word;'>6.14</td><td style='text-align: center; word-wrap: break-word;'>6.42</td><td style='text-align: center; word-wrap: break-word;'>14.15</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>16</td><td style='text-align: center; word-wrap: break-word;'>5.93</td><td style='text-align: center; word-wrap: break-word;'>6.20</td><td style='text-align: center; word-wrap: break-word;'>14.08</td></tr></table>

<div style="text-align: center;">Table 11. Albation study on training dataset. Models denoted with '-i' is trained on Interhand2.6m. Models denoted with '-m' is trained on mixed datasets.</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'>Dataset</td><td style='text-align: center; word-wrap: break-word;'>Methods</td><td style='text-align: center; word-wrap: break-word;'>MPJPE</td><td style='text-align: center; word-wrap: break-word;'>MPVPE</td><td style='text-align: center; word-wrap: break-word;'>MRRPE</td></tr><tr><td rowspan="6">Inter2.6m 5fps</td><td style='text-align: center; word-wrap: break-word;'>IntagHand-i</td><td style='text-align: center; word-wrap: break-word;'>8.79</td><td style='text-align: center; word-wrap: break-word;'>9.02</td><td style='text-align: center; word-wrap: break-word;'>27.74</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>IntagHand-m</td><td style='text-align: center; word-wrap: break-word;'>10.55</td><td style='text-align: center; word-wrap: break-word;'>10.98</td><td style='text-align: center; word-wrap: break-word;'>30.04</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>InterWild-i</td><td style='text-align: center; word-wrap: break-word;'>14.76</td><td style='text-align: center; word-wrap: break-word;'>13.24</td><td style='text-align: center; word-wrap: break-word;'>26.25</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>InterWild-m</td><td style='text-align: center; word-wrap: break-word;'>14.45</td><td style='text-align: center; word-wrap: break-word;'>13.01</td><td style='text-align: center; word-wrap: break-word;'>25.68</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Ours-i</td><td style='text-align: center; word-wrap: break-word;'>7.49</td><td style='text-align: center; word-wrap: break-word;'>7.72</td><td style='text-align: center; word-wrap: break-word;'>24.58</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Ours-m</td><td style='text-align: center; word-wrap: break-word;'>6.77</td><td style='text-align: center; word-wrap: break-word;'>6.96</td><td style='text-align: center; word-wrap: break-word;'>22.01</td></tr><tr><td rowspan="6">HIC</td><td style='text-align: center; word-wrap: break-word;'>IntagHand-i</td><td style='text-align: center; word-wrap: break-word;'>12.06</td><td style='text-align: center; word-wrap: break-word;'>12.77</td><td style='text-align: center; word-wrap: break-word;'>36.46</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>IntagHand-m</td><td style='text-align: center; word-wrap: break-word;'>12.54</td><td style='text-align: center; word-wrap: break-word;'>13.32</td><td style='text-align: center; word-wrap: break-word;'>35.41</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>InterWild-i</td><td style='text-align: center; word-wrap: break-word;'>11.23</td><td style='text-align: center; word-wrap: break-word;'>11.65</td><td style='text-align: center; word-wrap: break-word;'>30.35</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>InterWild-m</td><td style='text-align: center; word-wrap: break-word;'>11.13</td><td style='text-align: center; word-wrap: break-word;'>11.54</td><td style='text-align: center; word-wrap: break-word;'>30.59</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Ours-i</td><td style='text-align: center; word-wrap: break-word;'>9.47</td><td style='text-align: center; word-wrap: break-word;'>10.35</td><td style='text-align: center; word-wrap: break-word;'>26.84</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Ours-m</td><td style='text-align: center; word-wrap: break-word;'>8.66</td><td style='text-align: center; word-wrap: break-word;'>9.31</td><td style='text-align: center; word-wrap: break-word;'>26.56</td></tr><tr><td rowspan="6">Arctic</td><td style='text-align: center; word-wrap: break-word;'>IntagHand-i</td><td style='text-align: center; word-wrap: break-word;'>24.88</td><td style='text-align: center; word-wrap: break-word;'>26.30</td><td style='text-align: center; word-wrap: break-word;'>203.14</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>IntagHand-m</td><td style='text-align: center; word-wrap: break-word;'>27.64</td><td style='text-align: center; word-wrap: break-word;'>32.78</td><td style='text-align: center; word-wrap: break-word;'>225.49</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>InterWild-i</td><td style='text-align: center; word-wrap: break-word;'>22.74</td><td style='text-align: center; word-wrap: break-word;'>23.92</td><td style='text-align: center; word-wrap: break-word;'>143.19</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>InterWild-m</td><td style='text-align: center; word-wrap: break-word;'>21.85</td><td style='text-align: center; word-wrap: break-word;'>22.98</td><td style='text-align: center; word-wrap: break-word;'>134.69</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Ours-i</td><td style='text-align: center; word-wrap: break-word;'>14.21</td><td style='text-align: center; word-wrap: break-word;'>15.12</td><td style='text-align: center; word-wrap: break-word;'>112.81</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Ours-m</td><td style='text-align: center; word-wrap: break-word;'>13.57</td><td style='text-align: center; word-wrap: break-word;'>14.51</td><td style='text-align: center; word-wrap: break-word;'>105.43</td></tr></table>

## 5 Limitations

Although OmniHands has demonstrated high accuracy in offline video-based reconstruction, there remain several challenges that

---

currently hinder its deployment for real-time interacting-hand reconstruction. Our method utilizes ViT-h as the backbone, resulting in higher computational demands compared to CNN-based methods, which is a common issue with transformer-based models, as seen in HaMeR [Pavlakos et al. 2023] and Hamba [Dong et al. 2024]. During inference, running on a single A800 GPU with Python code, our model achieves 16.05 FPS, indicating that it is not suitable for real-time rendering, which requires an efficiency of at least 25 to 30 FPS. Additionally, since our approach relies on external detection models to localize hands and regress bounding boxes, the overall inference speed is also limited by the performance of these detection models. Another challenge in real-time reconstruction arises from the fact that we predict the middle frame, which introduces a minimum delay of 4 frames in a live video stream, as the model requires access to 4 future frames as input. We will focus on accelerating our model by reimplementing it in C++ instead of Python and optimizing the computation process to enable real-time rendering.

## 6 Conclusions

In this paper, we presented OmniHands, a robust solution for recovering interactive hand motions and their relative movements from monocular inputs. OmniHands is a transformer-based method with novel tokenization and feature fusion strategies. To this end, we introduced the Relation-aware Two-Hand Tokenization (RAT) method, which embeds positional information into hand tokens. This allows our network to handle both single-hand and two-hand inputs, leveraging their relative positions to reconstruct detailed hand interactions in real-world scenarios. Moreover, this tokenization approach enhances feature fusion by indicating the relative positions of the hands. To further this goal, we developed a 4D Interaction Reasoning (FIR) module that fuses hand tokens in 4D with attention, decoding them into 3D hand meshes and their temporal movements. The effectiveness of OmniHands is demonstrated through extensive evaluations on benchmark datasets, with results from in-the-wild videos and real-world scenarios showcasing its superior performance in interactive hand reconstruction.

## References

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyam Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023).

Seungryul Baek, Kwang In Kim, and Tae-Kyun Kim. 2019. Pushing the Envelope for RGB-Based Dense 3D Hand Pose Estimation via Neural Rendering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition.

Adnane Boukhayma, Rodrigo de Bem, and Philip HS Torr. 2019. 3d hand shape and pose from images in the wild. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition.

Yu-Wei Chao, Wei Yang, Yu Xiang, Pavlo Molchanov, Ankur Handa, Jonathan Tremblay, Yashraj S. Narang, Karl Van Wyk, Umar Iqbal, Stan Birchfield, Jan Kautz, and Dieter Fox. 2021. DexYCB: A Benchmark for Capturing Hand Grasping of Objects. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition.

Xingyu Chen, Yufeng Liu, Chongyang Ma, Jianlong Chang, Huayan Wang, Tian Chen, Xiaoyan Guo, Pengfei Wan, and Wen Zheng. 2021. Camera-Space Hand Mesh Recovery via Semantic Aggregation and Adaptive 2D-1D Registration. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition.

Xingyu Chen, Yufeng Liu, Dong Yajiao, Xiong Zhang, Chongyang Ma, Yanmin Xiong, Yuan Zhang, and Xiaoyan Guo. 2022. MobRecon: Mobile-Friendly Hand Mesh Reconstruction from Monocular Image. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition.

Hoseong Cho, Chanwoo Kim, Jihyeon Kim, Seongyeong Lee, Elkhan Ismayilzada, and Seungryul Baek. 2023. Transformer-based unified recognition of two hands

manipulating objects. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 4769–4778.

Hongsuk Choi, Gyeongsik Moon, Ju Yong Chang, and Kyoung Mu Lee. 2021. Beyond static features for temporally consistent 3d human pose and shape from a video. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 1964–1973.

Hongsuk Choi, Gyeongsik Moon, and Kyoung Mu Lee. 2020. Pose2Mesh: Graph Convolutional Network for 3D Human Pose and Mesh Recovery from a 2D Human Pose. In European Conference on Computer Vision (ECCV).

Vasileios Choutas, Georgios Pavlakos, Timo Bolkart, Dimitrios Tzionas, and Michael J. Black. 2020. Monocular Expressive Body Regression through Body-Driven Attention. In European Conference on Computer Vision (ECCV). <https://expose.is.tue.mpg.de>

Enric Corona, Tomas Hodan, Minh Vo, Francesc Moreno-Noguer, Chris Sweeney, Richard Newcombe, and Lingni Ma. 2022. Lisa: Learning implicit shape and appearance of hands. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition.

Haoye Dong, Aviral Chharia, Wenbo Gou, Francisco Vicente Carrasco, and Fernando De la Torre. 2024. Hamba: Single-view 3d hand reconstruction with graph-guided bi-scanning mamba. Advances in Neural Information Processing Systems 37 (2024), 2127–2160.

Junting Dong, Wen Jiang, Qixing Huang, Hujun Bao, and Xiaowei Zhou. 2019. Fast and robust multi-person 3d pose estimation from multiple views. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. 2020. An image is worth $16 \times 16$ words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020).

Zicong Fan, Omid Taheri, Dimitrios Tzionas, Muhammed Kocabas, Manuel Kaufmann, Michael J Black, and Otmar Hilliges. 2023. ARCTIC: A dataset for dexterous bimanual hand-object manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 12943–12954.

Hao-Shu Fang, Jiefeng Li, Hongyang Tang, Chao Xu, Haoyi Zhu, Yuliang Xiu, Yong-Lu Li, and Cewu Lu. 2022. Alphapose: Whole-body regional multi-person pose estimation and tracking in real-time. IEEE Transactions on Pattern Analysis and Machine Intelligence (2022).

Yao Feng, Vasileios Choutas, Timo Bolkart, Dimitrios Tzionas, and Michael J. Black. 2021. Collaborative Regression of Expressive Bodies using Moderation. In International Conference on 3D Vision (3DV).

Qichen Fu, Xingyu Liu, Ran Xu, Juan Carlos Niebles, and Kris M Kitani. 2023. Deformer: Dynamic fusion transformer for robust hand pose estimation. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 23600–23611.

Qijun Gan, Wentong Li, Jinwei Ren, and Jianke Zhu. 2024. Fine-grained multi-view hand reconstruction using inverse rendering. In Proceedings of the AAAI Conference on Artificial Intelligence.

Liuhao Ge, Zhou Ren, Yuncheng Li, Zehao Xue, Yingying Wang, Jianfei Cai, and Junsong Yuan. 2019. 3D Hand Shape and Pose Estimation From a Single RGB Image. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition.

Shubham Goel, Georgios Pavlakos, Jathushan Rajasegaran, Angjoo Kanazawa, and Jitendra Malik. 2023. Humans in 4d: Reconstructing and tracking humans with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 14783–14794.

Zhiyang Guo, Wengang Zhou, Min Wang, Li Li, and Houqiang Li. 2023. HandNeRF: Neural radiance fields for animatable interacting hands. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition.

Shreyas Hampali, Mahdi Rad, Markus Oberweger, and Vincent Lepetit. 2020. Honnotate: A method for 3d annotation of hand and object poses. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 3196–3206.

Yana Hasson, Bugra Tekin, Federica Bogo, Ivan Laptev, Marc Pollefeys, and Cordelia Schmid. 2020. Leveraging Photometric Consistency over Time for Sparsely Supervised Hand-Object Reconstruction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition.

Changlong Jiang, Yang Xiao, Cunlin Wu, Mingyang Zhang, Jinghong Zheng, Zhiguo Cao, and Joey Tianyi Zhou. 2023. A2j-transformer: Anchor-to-joint transformer network for 3d interacting hand pose estimation from a single rgb image. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 8846–8855.

Sheng Jin, Lumin Xu, Jin Xu, Can Wang, Wentao Liu, Chen Qian, Wanli Ouyang, and Ping Luo. 2020. Whole-body human pose estimation in the wild. In ECCV. Springer, 196–214.

Hanbyul Joo, Tomas Simon, and Yaser Sheikh. 2018. Total Capture: A 3D Deformation Model for Tracking Faces, Hands, and Bodies. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition.

Angjoo Kanazawa, Michael J Black, David W Jacobs, and Jitendra Malik. 2018. End-to-end recovery of human shape and pose. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 7122–7131.

Dong Uk Kim, Kwang In Kim, and Seungryul Baek. 2021. End-to-end detection and pose estimation of two interacting hands. In Proceedings of the IEEE/CVF International

---

Conference on Computer Vision.

Jeonghwan Kim, Mi-Gyeong Gwon, Hyunwoo Park, Hyukmin Kwon, Gi-Mun Um, and Wonjun Kim. 2023. Sampling is matter: Point-guided 3d human mesh reconstruction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 12880–12889.

Muhammed Kocabas, Nikos Athanasiou, and Michael J Black. 2020. Vibe: Video inference for human body pose and shape estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 5253–5263.

Nikos Kolotouros, Georgios Pavlakos, Michael J Black, and Kostas Daniilidis. 2019. Learning to reconstruct 3D human pose and shape via model-fitting in the loop. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 2252–2261.

Nikos Kolotouros, Georgios Pavlakos, and Kostas Daniilidis. 2019. Convolutional Mesh Regression for Single-Image Human Shape Reconstruction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition.

Dominik Kulon, Riza Alp Güler, Iasonas Kokkinos, Michael Bronstein, and Stefanos Zafeiriou. 2020. Weakly-Supervised Mesh-Convolutional Hand Reconstruction in the Wild. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition.

Lijun Li, Linrui Tian, Xindi Zhang, Qi Wang, Bang Zhang, Liefeng Bo, Mengyuan Liu, and Chen Chen. 2023. Renderih: A large-scale synthetic dataset for 3d interacting hand pose estimation. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 20395–20405.

Mengcheng Li, Liang An, Hongwen Zhang, Lianpeng Wu, Feng Chen, Tao Yu, and Yebin Liu. 2022. Interacting attention graph for single image two-hand reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2761–2770.

Mengcheng Li, Hongwen Zhang, Yuxiang Zhang, Ruizhi Shao, Tao Yu, and Yebin Liu. 2024. HHMR: Holistic Hand Mesh Recovery by Enhancing the Multimodal Controllability of Graph Diffusion Models. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.

Kevin Lin, Lijuan Wang, and Zicheng Liu. 2021. End-to-End Human Pose and Mesh Reconstruction with Transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition.

Kevin Lin, Lijuan Wang, and Zicheng Liu. 2021. Mesh graphormer. In Proceedings of the IEEE/CVF international conference on computer vision. 12939–12948.

Shaowei Liu, Hanwen Jiang, Jiarui Xu, Sifei Liu, and Xiaolong Wang. 2021. Semi-Supervised 3D Hand-Object Poses Estimation with Interactions in Time. In Proceedings of the IEEE conference on computer vision and pattern recognition.

Hao Meng, Sheng Jin, Wentao Liu, Chen Qian, Mengxiang Lin, Wanli Ouyang, and Ping Luo. 2022. 3D Interacting Hand Pose Estimation by Hand De-occlusion and Removal. In ECCV. Springer, 380–397.

Gyeongsik Moon. 2023. Bringing inputs to shared domains for 3D interacting hands recovery in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 17028–17037.

Gyeongsik Moon, Hongsuk Choi, and Kyoung Mu Lee. 2022. Accurate 3D Hand Pose Estimation for Whole-Body 3D Human Mesh Estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition Workshop.

Gyeongsik Moon, Shunsuke Saito, Weipeng Xu, Rohan Joshi, Julia Buffalo, Harley Bellan, Nicholas Rosen, Jesse Richardson, Mallorie Mize, Philippe De Bree, et al. 2024. A dataset of relighted 3d interacting hands. Advances in Neural Information Processing Systems 36 (2024).

Gyeongsik Moon, Shoou-I Yu, He Wen, Takaaki Shiratori, and Kyoung Mu Lee. 2020. InterHand2.6M: A Dataset and Baseline for 3D Interacting Hand Pose Estimation from a Single RGB Image. In ECCV.

JoonKyu Park, Yeonguk Oh, Gyeongsik Moon, Hongsuk Choi, and Kyoung Mu Lee. 2022. HandOccNet: Occlusion-Robust 3D Hand Mesh Estimation Network. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high-performance deep learning library. In NeurIPS.

Georgios Pavlakos, Dandan Shan, Ilija Radosavovic, Angjoo Kanazawa, David Fouhey, and Jitendra Malik. 2023. Reconstructing Hands in 3D with Transformers. arXiv preprint arXiv:2312.05251 (2023).

Georgios Pavlakos, Xiaowei Zhou, Konstantinos G Derpanis, and Kostas Daniilidis. 2017. Harvesting multiple views for marker-less 3d human pose annotations. In Proceedings of the IEEE conference on computer vision and pattern recognition.

Haibo Qiu, Chunyu Wang, Jingdong Wang, Naiyan Wang, and Wenjun Zeng. 2019. Cross view fusion for 3d human pose estimation. In Proceedings of the IEEE/CVF International Conference on Computer Vision.

Pengfei Ren, Chao Wen, Xiaozheng Zheng, Zhou Xue, Haifeng Sun, Qi Qi, Jingyu Wang, and Jianxin Liao. 2023. Decoupled iterative refinement framework for interacting hands reconstruction from a single rgb image. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 8014–8025.

Javier Romero, Dimitrios Tzionas, and Michael J. Black. 2017. Embodied Hands: Modeling and Capturing Hands and Bodies Together. In SIGGRAPH Asia.

Yu Rong, Takaaki Shiratori, and Hanbyul Joo. 2021. FrankMocap: Fast Monocular 3D Hand and Body Motion Capture by Regression and Integration. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshop.

Tomas Simon, Hanbyul Joo, Iain Matthews, and Yaser Sheikh. 2017. Hand keypoint detection in single images using multiview bootstrapping. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition. 1145–1153.

Xiao Tang, Tianyu Wang, and Chi-Wing Fu. 2021. Towards Accurate Alignment in Real-time 3D Hand-Mesh Reconstruction. In Proceedings of the IEEE/CVF International Conference on Computer Vision.

Yating Tian, Hongwen Zhang, Yebin Liu, and Limin Wang. 2023. Recovering 3D human mesh from monocular images: A survey. IEEE transactions on pattern analysis and machine intelligence (2023).

Hanyue Tu, Chunyu Wang, and Wenjun Zeng. 2020. Voxelpose: Towards multi-camera 3d human pose estimation in wild environment. In ECCV.

Zhigang Tu, Zhisheng Huang, Yujin Chen, Di Kang, Linchao Bao, Bisheng Yang, and Junsong Yuan. 2023. Consistent 3d hand reconstruction in video via self-supervised learning. IEEE Transactions on Pattern Analysis and Machine Intelligence (2023).

Dimitrios Tzionas, Luca Ballan, Abhilash Srikantha, Pablo Aponte, Marc Pollefeys, and Juergen Gall. 2016. Capturing hands in action using discriminative salient points and physics simulation. International Journal of Computer Vision 118 (2016), 172–193.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems (Long Beach, California, USA) (NIPS'17). Curran Associates Inc., Red Hook, NY, USA, 6000-6010.

Congyi Wang, Feida Zhu, and Shilei Wen. 2023. MeMaHand: Exploiting mesh-mono interaction for single image two-hand reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 564–573.

Jiayi Wang, Franziska Mueller, Florian Bernard, Suzanne Sorli, Oleksandr Sotnychenko, Neng Qian, Miguel A Otaduy, Dan Casas, and Christian Theobalt. 2020. Rgb2hands: real-time tracking of 3d hand interactions from monocular rgb video. ACM Transactions on Graphics (ToG) 39 (2020), 1–16.

Yinqiao Wang, Hao Xu, Pheng Ann Heng, and Chi-Wing Fu. 2024. SiMA-Hand: Boosting 3D Hand-Mesh Reconstruction by Single-to-Multi-View Adaptation. In Proceedings of the AAAI Conference on Artificial Intelligence.

Donglai Xiang, Hanbyul Joo, and Yaser Sheikh. 2019. Monocular total capture: Posing face, body, and hands in the wild. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 10965–10974.

Hao Xu, Haipeng Li, Yinqiao Wang, Shuaicheng Liu, and Chi-Wing Fu. 2024. Handbooster: Boosting 3d hand-mesh reconstruction by conditional synthesis and sampling of hand-object interactions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 10159–10169.

Hao Xu, Tianyu Wang, Xiao Tang, and Chi-Wing Fu. 2023. H2onet: Hand-occlusion-and-orientation-aware network for real-time 3d hand mesh reconstruction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 17048–17058.

Yufei Xu, Jing Zhang, Qiming Zhang, and Dacheng Tao. 2022. Vitpose: Simple vision transformer baselines for human pose estimation. Advances in Neural Information Processing Systems 35 (2022), 38571–38584.

John Yang, Hyung Jin Chang, Seungeui Lee, and Nojun Kwak. 2020. Seqhand: Rgb-sequence-based 3d hand pose and shape estimation. In ECCV. Springer, 122–139.

Lixin Yang, Jian Xu, Licheng Zhong, Xinyu Zhan, Zhicheng Wang, Kejian Wu, and Cewu Lu. 2023. Poem: reconstructing hand in a point embedded multi-view stereo. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition.

Zhengdi Yu, Shaoli Huang, Chen Fang, Toby P Breckon, and Jue Wang. 2023. Acr: Attention collaboration-based regressor for arbitrary two-hand reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 12955–12964.

Baowen Zhang, Yangang Wang, Xiaoming Deng, Yinda Zhang, Ping Tan, Cuixia Ma, and Hongan Wang. 2021a. Interacting two-hand 3d pose and shape reconstruction from single color image. In Proceedings of the IEEE/CVF international conference on computer vision. 11354–11363.

Hongwen Zhang, Jie Cao, Guo Lu, Wanli Ouyang, and Zhenan Sun. 2022. Learning 3D Human Shape and Pose from Dense Body Parts. TPAMI 44, 5 (2022), 2610–2627.

Hongwen Zhang, Yating Tian, Yuxiang Zhang, Mengcheng Li, Liang An, Zhenan Sun, and Yebin Liu. 2023. PyMAF-X: Towards Well-aligned Full-body Model Regression from Monocular Images. TPAMI (2023).

Hongwen Zhang, Yating Tian, Xinchi Zhou, Wanli Ouyang, Yebin Liu, Limin Wang, and Zhenan Sun. 2021. PyMAF: 3D Human Pose and Shape Regression With Pyramidal Mesh Alignment Feedback Loop. In Proceedings of the IEEE/CVF International Conference on Computer Vision.

Jianfeng Zhang, Yujun Cai, Shuicheng Yan, Jiashi Feng, et al. 2021a. Direct multi-view multi-person 3d pose estimation. Advances in Neural Information Processing Systems (2021).

Xiong Zhang, Qiang Li, Hong Mo, Wenbo Zhang, and Wen Zheng. 2019. End-to-End Hand Mesh Recovery From a Monocular RGB Image. In Proceedings of the IEEE/CVF International Conference on Computer Vision.

---

Yuxiang Zhang, Zhe Li, Liang An, Mengcheng Li, Tao Yu, and Yebin Liu. 2021b. Lightweight Multi-person Total Capture Using Sparse Multi-view Cameras. In Proceedings of the IEEE/CVF International Conference on Computer Vision.

Zhe Zhang, Chunyu Wang, Weichao Qiu, Wenhu Qin, and Wenjun Zeng. 2021b. Adaptive: Adaptive multiview fusion for accurate human pose estimation in the wild. International Journal of Computer Vision 129 (2021), 703–718.

Weichao Zhao, Hezhen Hu, Wengang Zhou, Li Li, and Houqiang Li. 2024. Exploiting Spatial-Temporal Context for Interacting Hand Reconstruction on Monocular RGB Video. ACM Transactions on Multimedia Computing, Communications and Applications 20, 6 (2024), 1–18.

Zhishan Zhou, Shihao Zhou, Zhi Lv, Minqiang Zou, Yao Tang, and Jiajun Liang. 2024. A simple baseline for efficient hand mesh reconstruction. In Proceedings of the

IEEE/CVF conference on computer vision and pattern recognition. 1367–1376.

Christian Zimmermann and Thomas Brox. 2017. Learning to estimate 3d hand pose from single rgb images. In Proceedings of the IEEE international conference on computer vision. 4903–4911.

Christian Zimmermann, Duygu Ceylan, Jimei Yang, Bryan Russell, Max Argus, and Thomas Brox. 2019. Freihand: A dataset for markerless capture of hand pose and shape from single rgb images. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 813–822.

Binghui Zuo, Zimeng Zhao, Wenqian Sun, Wei Xie, Zhou Xue, and Yangang Wang. 2023. Reconstructing interacting hands with interaction prior from monocular images. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 9054–9064.