# DF-Mamba：用于交互场景中3D手部姿态估计的可变形状态空间建模

周一凡 $^{1*}$、大川武彦 $^{1,3*}$、周冠文晓 $^{1}$、后藤佳乃子 $^{1}$、广濑拓海 $^{1}$、关川佑介 $^{2}$、井上中正 $^{1}$

$^{1}$东京科学大学 $^{2}$电装IT实验室 $^{3}$东京大学

## 摘要

对日常手部交互的建模常常面临严重遮挡的问题，比如双手重叠的情况，这凸显了在3D手部姿态估计（HPE）中学习鲁棒特征的必要性。为了处理这类被遮挡的手部图像，有效学习局部图像特征（例如被遮挡关节的特征）与全局上下文（例如关节间、手与手之间或场景中的线索）之间的关系至关重要。然而，目前大多数3D HPE方法仍依赖ResNet进行特征提取，由于CNN的归纳偏差有限，其建模全局上下文的能力可能并不适合3D HPE任务。为解决这一局限，我们提出了一种基于最新状态空间建模技术（即Mamba）的高效且轻量的3D HPE视觉特征提取框架，命名为可变形Mamba（DF-Mamba）。DF-Mamba旨在通过Mamba的选择性状态建模以及我们提出的可变形状态扫描机制，捕捉超越传统卷积的全局上下文线索。具体而言，对于卷积后的局部特征，我们的可变形扫描会在图像范围内聚合这些特征，同时选择性地保留能够代表全局上下文的有用线索。这种方法显著提升了结构化3D HPE的精度，且其推理速度与ResNet-50相当。我们的实验在五个不同类型的数据集上进行了广泛评估，涵盖单手和双手场景、仅手部及手与物体交互情况，以及RGB和深度图像估计任务。DF-Mamba在所有数据集上的表现均优于VMamba和Spatial-Mamba等最新的图像骨干网络，取得了最先进的性能。

### 1. 引言

日常的人类活动常常涉及复杂的手部交互，例如双手交互[^46] [^50]以及物体抓取[^16] [^49] [^51]，这就需要能够有效且高效地处理这些情况的推理模型，以估算三维手部姿态。

![Image](images/65495dbb99cb5b2cd3a0c981571944971729d0d6b8653d7ba6ea8c8953e4ce68.jpg)

![Image](images/9d68cf58d3caea7b94c4895ef353e5128c3845cc531e3ad1a84a1ea8ec5a80e0.jpg)

![Image](images/b4e7c31b087b47275589ca5863812a061863adea8af39edbb1a58fb47380c540.jpg)

![Image](images/f0f599c5ab6069d72fc1e90f306710de59d7ba6fbf0a2b5df2139af5994a5558.jpg)

<div style="text-align: center;">(b) Deformable scan (ours)</div>

<div style="text-align: center;">Figure 1. Deformable scan for DF-Mamba. (a) Conventional sweep scan uses a fixed grid pattern in the state space equations. (b) Our deformable scan adaptively adjusts the scanning pattern with multiple anchors  $a_{k}$ by predicting offset vectors  $o_{k,t}$ dependent on input visual features.</div>

为了应对这类极具挑战性的场景，需要高效的推理模型来估算三维手部姿态。由于存在严重的遮挡现象，从视觉数据（包括单张RGB图像[^32]、深度图像[^60]、 Egocentric视角图像[^16] [^38]等）中估算三维手部姿态（HPE）十分困难。同时，开发能够快速推理的模型对于支持实时应用至关重要，尤其是在AR/VR设备中[^6] [^36]。鉴于上述挑战，人们对骨干网络架构所引入的归纳偏置及其与三维HPE任务的协同作用关注甚少。这就凸显出设计既能有效捕捉复杂交互、又能满足实时推理效率要求的骨干网络的必要性。

为了针对复杂的手部交互实现鲁棒的特征学习，必须学会局部特征与全局上下文之间的关系__

---

图像特征（例如针对被遮挡的关节）以及全局上下文（例如来自关节间、手之间、手与物体之间或场景的线索）。卷积神经网络（CNN）在三维人体姿态估计中被广泛用作骨干架构，其中ResNet-50 [^26] 尤为常用 [^31] [^32] [^37] [^38] [^46] [^49] [^55] [^73]。这类CNN骨干网络依赖具有局部感受野的卷积运算，能够在精度与推理速度之间取得良好平衡。然而，这种局部卷积在捕捉全局上下文方面的能力有限。另一类近期研究则利用视觉Transformer（ViT）[^12] 作为手部网格重建的特征提取器 [^11] [^52] [^80]，但其较高的计算复杂度在实际场景中往往成为瓶颈。这表明，为了在高效实现手部姿态更好特征学习方面取得突破，骨干架构仍有很大的改进空间。

作为一种新兴的基础架构，基于状态空间模型（SSM）的Mamba [^20] 受到了广泛关注，它最初是为自然语言处理领域设计的。Mamba模型擅长选择性地聚焦输入标记（即强调特定信号），从而能高效地从长标记序列中捕捉全局上下文。近期已有多项研究将其拓展为图像骨干网络。例如，VisionMamba [^81] 引入了Vim模块，该模块采用二维双向扫描来实现空间感知的序列建模；VMamba [^40] 则进一步提出了基于四种不同扫描路径的VSS模块。不过，如图1(a)所示，这些扫描机制采用的是固定网格，当应用于三维人体姿态估计时，它们捕捉复杂手部姿态变化的能力受到限制。

针对这一局限，我们提出了一种高效且实用的骨干网络——可变形Mamba（DF-Mamba），它采用可变形状态空间模型（DSSM）来强化三维人体姿态估计中的视觉特征提取。DF-Mamba的核心思想是通过灵活的状态空间动态建模局部特征与全局上下文，从而实现特征提取。具体而言，我们的DSSM模块会根据可变形路径聚合局部特征，并选择性地存储有用线索以表征全局上下文。如图1(b)所示，扫描路径是通过带有局部锚点的可变形点采样以及依赖于输入特征的可学习偏移量来调整的。

DF-Mamba的整体架构采用三模块设计，包含卷积模块、DSSM模块和门控卷积模块，其模型规模与ResNet-50相当。这种方法有效利用了各模块类型的互补优势：低层通过卷积模块提取局部特征，下采样后在高层利用DSSM模块自适应地增强特征，最后再通过门控卷积模块进一步优化视觉表征。

在我们的实验中，我们将DF-Mamba集成到由Jiang等人[^32]和Zhou等人[^79]提出的两种具有代表性的3D手部姿态估计框架中，用我们的方法替换其骨干网络。我们的评估在五个数据集上进行：InterHand2.6M [^46]、RHP [^84]、NYU [^60]、DexYCB [^4]以及AssemblyHands [^49]，这些数据集涵盖了多种场景，包括单手和双手姿态估计、仅手部及手-物体交互，以及RGB和深度模态。结果表明，DF-Mamba的性能优于最新的基于Mamba的骨干网络，例如VMamba [^40]和SpatialMamba [^68]，在保持推理速度的同时实现了最先进的性能。我们的贡献总结如下。

1) 我们提出了DSSM，这是一种用于建模动态系统的新方法，通过灵活的状态空间来表示全局上下文。DSSM将可变形扫描机制融入状态空间方程中。

2) 我们引入了DF-Mamba，一种用于3D手部姿态估计的新型基于Mamba的骨干网络。它采用了由卷积层、DSSM和门控卷积块组成的三元设计。

3) 我们在五个数据集上验证了DF-Mamba的有效性和高效性。结果显示，DF-Mamba在3D手部姿态估计任务中的表现优于最新的图像骨干网络。

### 2. 相关工作

#### 2.1. 3D手部姿态估计

该任务旨在从单张图像中预测手部关节的3D坐标[^13] [^48]。这一任务已在多种交互场景中得到研究，例如单手姿态估计[^5] [^19] [^84] [^85]、自接触情况[^50] [^56]以及手-物体交互[^15] [^16]，还有基于RGB[^19] [^32] [^78]和深度的姿态估计[^60]。大多数现有方法依赖深度神经网络从视觉输入中预测手部姿态，这就需要设计高效且有效的架构。

为了实现精确估计，编码器-解码器设计最为常见，通常利用ResNet [^26]等图像骨干网络作为编码器，再配合复杂的解码器来准确预测关节位置。例如，Zhou等人[^79]提出了一种简单的回归解码头，可从ResNet-50提取的特征中估算2D热图和深度信息。近年来，利用注意力机制的基于Transformer的解码方式在3D手部姿态估计领域也日益流行[^28] [^32] [^59] [^66] [^67] [^73]。比如，Jiang等人[^32]将锚点偏移-权重估计机制整合到Transformer解码器中[^3] [^83]。尽管取得了这些成果，ResNet [^26]仍是3D手部姿态估计中最常用的骨干网络之一[^22] [^32] [^37] [^39] [^47] [^49] [^55] [^73] [^79]。虽然近期有研究采用了预训练的视觉Transformer作为手部网格重建的骨干网络[^11] [^52] [^80]，但注意力模块的高计算复杂度限制了它们的实际应用。此外，当目标数据分布与预训练数据分布差异较大时，

---

即便面对不同的领域（如双手交互[^46]、深度图像[^60]或自中心视角[^1] [^49]），使用基于ViT的骨干网络进行训练仍然颇具挑战性。为解决这一问题，我们提出了一种基于Mamba建模的多功能3D人体姿态估计骨干网络，它可替代常用的ResNet-50，并能适用于各类场景（如[^46] [^47] [^60]）。

#### 2.2. 图像骨干网络

基于CNN的骨干网络。CNN利用具有局部感受野的简单卷积操作来高效提取视觉特征。ResNet[^26]仍是各类视觉任务中最广泛采用的骨干网络之一。为了进一步提升卷积操作的适应性与灵活性，人们引入了可变形卷积[^9] [^82]和深度卷积[^7] [^27]等先进技术。近期，ConvNeXt[^42]通过融入视觉Transformer的设计元素对ResNet架构进行了改进，在各类任务中均取得了具有竞争力的性能。

基于Transformer的骨干网络。自视觉Transformer（ViT）[^12]问世以来，人们提出了多种基于Transformer的骨干网络以进一步优化图像骨干网络[^8] [^41] [^54] [^61] [^65] [^70]。其中，Swin Transformer[^41]引入了带有移位窗口注意力机制的层次化架构，凭借其强大的多尺度金字塔特征提取能力，在各种视觉任务中展现出了通用性。

基于Mamba的骨干网络。Mamba[^20]是首个基于状态空间建模（SSM）且在自然语言处理任务上性能可与Transformer相媲美的成功架构。它特别引入了选择性SSM，通过根据输入计算状态空间方程中的矩阵B和C[^2] [^33]，从而自适应地记忆或遗忘输入标记。

VisionMamba[^81]是首个基于Mamba的图像骨干网络，它用基于Mamba的编码器替换了ViT的编码器，并将原有的1D SSM扩展为2D双向SSM。后续研究进一步拓展了扫描机制与架构[^25] [^29] [^40] [^53] [^68] [^71] [^74]。例如，VMamba[^40]通过引入交叉扫描策略增强了2D扫描能力，可沿四个方向遍历图像。Spatial-Mamba[^68]是最新推出的基于Mamba的骨干网络之一，它引入了结构感知的状态融合方程，将2D卷积应用于状态空间方程中矩阵C的输入。由于以往Mamba模型采用的固定网格2D扫描方式会限制其在3D人体姿态估计等结构化预测任务中的能力，因此需要更自适应的扫描方法。Hamba[^11]则基于ViT骨干网络引入了用于手部网格重建的基于Mamba的解码器。其他研究也将基于Mamba的建模应用于人体姿态估计[^30] [^34] [^43] [^77]。

相比之下，我们研究了可用于各类场景的3D人体姿态估计骨干网络中的Mamba建模方案，这些场景包括双手姿态[^46]和自中心视角[^49]。

### 3. 可变形状态空间建模

我们提出了可变形状态空间建模（DSSM），这是一种用于建模具有灵活状态空间的动态系统的新方法。DSSM的核心思想是将局部视觉线索与Mamba的可变形扫描机制相结合，以捕捉全局上下文。下文我们将先简要回顾状态空间方程，随后介绍我们的DSSM公式。

#### 3.1. preliminaries

状态空间建模（SSM）。设$x(t) \in \mathbb{R}$为一维连续输入，其中$t \in \mathbb{R}_+$为时间索引。SSM表示一个动态线性系统[^2] [^33]，它通过隐状态$y(t) \in \mathbb{R}$生成输出$h(t) \in \mathbb{R}^{N \times 1}$，具体形式如下：

 $$ \left\{\begin{aligned}h^{\prime}(t)&=A h(t)+B x(t)\\ y(t)&=C h(t)+D x(t)\end{aligned}\right., $$

其中$A \in \mathbb{R}^{N \times N}$为转移矩阵，$B \in \mathbb{R}^{N \times 1}$、$C \in \mathbb{R}^{1 \times N}$和$D \in \mathbb{R}^{1 \times 1}$为用于控制动态系统的参数。

离散化。为了将SSM与深度神经网络结合，需对式(1)中的动态系统进行离散化处理，时间尺度为$\Delta$ [^18] [^21]。具体而言，离散的状态空间方程如下：

 $$ \left\{\begin{array}{l}h(t)=\bar{A}h(t-1)+\bar{B}x(t)\\ y(t)=C h(t)+D x(t)\end{array}\right., $$

此处采用零阶保持规则，即

 $$ \bar{A}=\exp(\Delta A),\;\bar{B}=(\Delta A)^{-1}(\exp(\Delta A)-I)(\Delta B). $$

图2(a)展示了SSM的计算流程。如图所示，输出y是根据式(2)中的递归关系，对x反复应用$\tilde{A}$得到的。在实际应用中，该模型被实现为SSM层，用于计算$Y = X * \tilde{K}$，其中$X \in \mathbb{R}^{d \times s}$为输入特征序列，$Y \in \mathbb{R}^{d \times s}$为输出序列，$\tilde{K} = [C\tilde{B}, C\tilde{A}\tilde{B}, \cdots, C\tilde{A}^{s-1}\tilde{B}]$为全局卷积核，d为维度，s为序列长度。此时各矩阵的大小也会扩展至d维度，即$B \in \mathbb{R}^{N \times d}$、$C \in \mathbb{R}^{d \times N}$和$D \in \mathbb{R}^{d \times d}$。Mamba模型[^20]会根据输入X计算B、C和$\Delta$，这些均通过可学习的线性层实现。

扫描机制。要将SSM应用于图像数据，需将视觉特征序列化。这一过程称为扫描方案，它将式(2)中的一维输入$x(t)$替换为二维

---

![Image](images/0a463e159e77664717ccd60e7f35d7892cac3a89f985ba60f2314909b222854d.jpg)

![Image](images/4b7f72abe029b37fc5cce4867c87419ef78c612a41a2f6a4e4a88a4874a0a8fa.jpg)

<div style="text-align: center;">Figure 2. Computational flow of SSM and DSSM. (a) As shown in Sec. 3.1, conventional SSM utilizes four matrices  $\overline{A}$,  $\overline{B}$,  C,  D, to compute the output  y from an input  x through intermediate representation  h. (b) Our DSSM incorporates weights  $b_k$ and offsets  $o_k$ for deformable scan into SSM.</div>

视觉特征图$x(p_t)$，其中$p_t \in \mathbb{R}^2$表示二维位置。图1(a)所示的扫描机制是一种典型的方法。具体而言，当特征图的大小为$H \times W$时，扫描机制确定的二维位置为

 $$ p_{t}=\left(\begin{matrix}{\bmod((t-1),W)+1}\\ {\lfloor t-1/W\rfloor+1}\\ \end{matrix}\right), $$

其中$t = 1, 2, \cdots, HW$。这种方式会在固定的二维网格上进行扫描。

然而，在三维人体姿态估计中，这种固定网格模式会导致效率低下，难以捕捉复杂的手部变化。例如，背景像素无法为局部关节位置提供任何线索，而输入图像的中心区域则包含更高密度的关节。这些现象表明，我们需要一种灵活的解决方案，以便根据输入自适应地建模空间局部偏差和全局上下文。

#### 3.2. 可变形公式

为了克服SSM的局限性，我们为SSM引入了数据驱动的自适应能力。我们提出加入可变形扫描机制，通过在输入采样中引入可学习的偏移量来动态调整采样点。这使我们能够平衡其感受野，选择性地聚合相关的局部线索以适应复杂的交互过程。具体而言，我们针对一维输入定义的DSSM动态系统如下：

 $$ \left\{\begin{array}{l}{h(t)=\bar{A}h(t-1)+\bar{B}x(t+\delta t)}\\ {y(t)=C h(t)+D x(t)}\end{array}\right., $$

其中$\delta t \in \mathbb{R}$是从x预测出的微小偏移量。

DSSM提升了SSM的灵活性，因为偏移量允许输入数据被自适应地移位。然而，将DSSM应用于图像数据并非易事，因为偏移量需要两个自由度，这使得相较于一维数据而言，偏移量的探索范围更大。为解决这一问题，我们通过引入空间分配的K个锚点来有效处理如此庞大的探索区域，这些锚点会从输入图像中采样多个点。具体而言，我们对二维输入的DSSM定义如下：

 $$ \left\{\begin{array}{l}h(t)=\bar{A}h(t-1)+\bar{B}\displaystyle\sum_{k=1}^{K}b_{k}x(p_{t}+a_{k}+o_{k,t})\\ y(t)=C h(t)+D x(p_{t})\end{array},\right. $$

其中$a_k \in \mathbb{R}^2$是固定锚点，$o_{k,t} \in \mathbb{R}^2$是偏移向量，而$b_k \in \mathbb{R}$是权重系数。$o_{k,t}$和$b_k$均通过可学习的线性层从x中预测得出。其计算流程如图2(b)所示。

具体而言，受可变形卷积$a_k \in \{(i, j) : i, j \in \{-1, 0, +1\}\}$的启发，K个锚点被对称地定义为[^9]。这为局部相对偏移提供了完整的初始分布，对于D个空间维度而言需要$3^D$个锚点（在本例中即为$D = 2$和$K = 9$）。以这些起始点为基础，通过偏移机制，锚点的空间分布便具备了可学习性。这种方式兼顾了锚点的初始覆盖范围与数据驱动的灵活性，而这正是DSSM的核心优势——二维扫描能力。

理论支撑方面，我们的DSSM是标准Mamba（SSM）模块[^20]和Spatial-Mamba模块[^68]的直接推广形式。例如，当锚点数量K=1，且偏移量为$o_{k,t} = 0$、缩放因子为$b_k = 1$时，我们的DSSM便会退化为普通的Mamba模块。虽然Spatial-Mamba在SSM中采用了膨胀操作，但可变形卷积的设计本身就将膨胀操作作为一种特殊情况纳入其中[^9]。因此，DSSM可视为Spatial-Mamba模块的一般形式，展现了相较于这些Mamba变体更强大的灵活性。

在3D人体姿态估计中的应用方面，当将DSSM用于3D人体姿态估计时，输入x对应于从单张图像中提取的局部特征，借助K个自适应锚点的引导来探测其邻近线索。随后，这些不断演化的局部线索会被聚合为隐状态h，该隐状态能够代表给定图像的全局上下文，例如关节间、手之间的线索，或是手与物体、场景之间的线索。因此，基于DSSM的这种自适应局部-全局特征学习方法，有助于建模3D人体姿态估计中手部动作的动态特性。

### 4. DF-Mamba骨干网络

结合第3节提出的DSSM，我们最终构建了整体骨干架构DF-Mamba。DF-Mamba是一种三模混合模型，由三种类型的模块组成：(i)卷积模块、(ii)DSSM模块以及(iii)门控卷积模块。由于(i)和(iii)属于现有模块，因此4.1节首先介绍(ii)DSSM模块的设计；详见图3(d)。接下来的4.2节将进一步详细说明如何在统一架构中整合这三种模块。

---

![Image](images/62394e1de04b4283df8c6e3e970428e29cd9dd286e798f3050c68508e4b946bc.jpg)

<div style="text-align: center;">Figure 3. Block architectures. (a) Vanilla Mamba block using SSM [^20] (b) Gated convolution block that omits the SSM layer from the Mamba block (e.g., MambaOut [^74]). (c) VSS block used in VMamba [^40]. (d) Our DF-Mamba block that replaces the SSM layer with the DSSM layer in the Mamba block. (e) General architecture inspired by the transformer architecture [^63]. Each subblock highlighted with a gray background acts as a token mixer.</div>

#### 4.1. DSSM模块

我们将 DSSM 与深度神经网络相结合，方法是定义其可堆叠为层的模块。图 3 比较了以下架构：(a) Mamba 中的原生 SSM 块 [^20]，(b) 门控卷积块 [^10]（例如 MambaOut [^74]），(c) VMamba 中的 VSS 块 [^40]，以及 (d) 我们的 DSSM 块。它们均以视觉特征图 X 作为输入，并应用层归一化和跳跃连接。(e) 展示了受原始 transformer [^63] 启发的一般设计。token 混合子块以灰色框标出，它们在 (a)–(d) 中互不相同。

具体来说，图3(b)中的门控卷积块通过省略SSM层形成了最简单的结构。给定经过层归一化后的视觉特征图$X' = \text{Norm}(X)$，该模块首先计算$Z_1 = \sigma(\text{Conv}(\text{Linear}(X')))$和$Z_2 = \sigma(\text{Linear}(X))$，其中$\sigma$是SiLU激活函数，Conv表示深度可分离的一维卷积层，Linear则表示线性层。随后对门控输出$Z_1 \odot Z_2$应用额外的线性层，再通过$Y = \text{Linear}(Z_1 \odot Z_2) + X$实现跳跃连接。而图3(a)中原始Mamba模型提出的SSM块[^20]则是在$Z_1$的计算过程中插入SSM层，即$Z_1 = \text{SSM}(\sigma(\text{Conv}(\text{Linear}(X’))))$。

受此SSM块的启发，我们图3(d)中的DSSM块将DSSM层插入其中，即$Z_1 = \text{DSSM}(\sigma(\text{Conv}(\text{Linear}(X''))))$。这样就能通过可变形扫描机制有效增强视觉特征图。如图3(e)所示，我们还在DSSM块中加入了前馈网络（FFN）模块，对应图4中的“DSSM+FFN块”。这种在基于Transformer的架构中提升性能的常见做法[^12] [^41] [^63]，在VMamba中也有应用[^40]。

#### 4.2. 三混合架构

大多数现有的Transformer骨干网络[^12] [^41]以及Mamba骨干网络[^40] [^68] [^81]都是由同一类型的模块堆叠而成。不过这些骨干网络往往会带来较高的计算复杂度，尤其是在视觉特征图分辨率较高的下层，这使得它们在3D人体姿态估计任务中实用性较低。

为解决这一问题，我们采用了一种将DSSM块、卷积块和门控卷积块相结合的三混合架构。这种方法能充分利用不同模块的互补优势：在下层通过卷积块提取特征，而在上层则利用DSSM块自适应地增强视觉特征图。

概述。图4展示了我们的DF-Mamba骨干网络的整体结构，它包含用于细粒度特征提取的卷积主干以及六个特征增强阶段。具体而言，这六个阶段分别采用三种模块类型：卷积模块、DSSM+FFN模块和门控卷积模块。在第1、2、4阶段末尾还添加了下采样层。该架构的设计目的是使计算复杂度与ResNet-50相当。由于多种3D人体姿态估计方法都利用金字塔特征来精确估计关节位置[^32]，图中也展示了从四个输出特征图中提取金字塔特征的流程。

卷积主干。对于尺寸为$H \times W \times C_0$的输入图像，卷积主干会应用两个连续的卷积层，每层之后依次接2D批量归一化和ReLU激活函数。第一个卷积层输出32个通道，第二个卷积层输出$C_1 = 80$个通道；两者均采用核大小为$3 \times 3$、步长为2的设置。最终得到的视觉特征图形状为$H/4 \times W/4 \times C_1$。

卷积模块。每个卷积模块采用ResNet-18的基本模块设计，激活函数由ReLU替换为GELU。两个卷积层均使用核大小为$3 \times 3$、步长为1的设置。第1和第2阶段各包含三个卷积模块。

DSSM+FFN模块。DSSM+FFN模块的架构遵循图3(e)中的样式，其中DSSM块位于…

---

![Image](images/5e6a821eb3cce3a16581ae2cba8c5d2ddd0c7435da43a85ce81bc24b8650ab8e.jpg)

<div style="text-align: center;">Figure 4. DF-Mamba backbone architecture. By combining three types of blocks, DF-Mamba improves the accuracy of 3D HPE while maintaining computational complexity comparable to or even lower than that of ResNet-50.</div>

图3(d)被用作令牌混合器。我们在第3阶段和第5阶段分别构建了五个和两个DSSM+FFN模块。之所以在这些阶段采用DSSM，是因为对这类下采样后的高级特征应用可变形扫描能够减少空间信息的丢失，同时更高效地完成状态扫描。

门控卷积模块。如[^74]所述，不含SSM的门控卷积模块计算效率较高，且作为令牌混合器依然有效。因此，我们在第4阶段和第6阶段分别构建了五个和一个门控卷积模块，如图3(b)所示；为降低计算复杂度，图中省略了FFN部分。在DSSM+FFN模块之后应用门控卷积模块，能够在保持高效性的同时增强视觉特征。

下采样操作。下采样层被添加在第1、第2和第4阶段的末尾。每个下采样层包含一个核大小为$3 \times 3$、步长为2的二维卷积层。

金字塔特征。视觉特征图从第1、第2、第4和第6阶段提取而来。对每个特征图分别应用层归一化和线性层处理，随后将得到的特征拼接起来以表示金字塔特征。

### 5. 实验

#### 5.1. 实验设置

数据集与评价指标。我们使用DF-Mamba在五个不同的数据集上开展大量实验：包含双手姿态数据的InterHand2.6M [^46]、包含单手姿态数据的RHP [^84]、包含深度图像的NYU [^60]、涉及手物交互的DexYCB [^4]，以及包含第一人称视角数据的AssemblyHands [^49]。我们按照各数据集指定的评估协议报告关节位置误差，同时给出DexYCB和AssemblyHands的AUC分数。更多细节详见附录。

基准模型。我们将DF-Mamba与六种骨干网络进行对比：ResNet-50 [^26]、ConvNeXt-T [^42]、ViT-S [^12]、Swin-T [^41]、VMamba-T [^40]以及Spatial-Mamba-T [^68]。为了进行3D手部姿态估计，我们将每种骨干网络集成到两种代表性框架中：一种由Jiang等人[^32]针对InterHand2.6M、RHP和NYU提出，另一种由Zhou等人[^79]针对DexYCB和AssemblyHands设计。这些框架包含解码器，且专为ResNet-50设计；因此所选的其他骨干网络具有相近的模型规模。

实现细节。我们遵循[^4]、[^32]和[^49]中描述的原始训练设置。所有模型均从零开始在同一数据划分上进行训练。详细的超参数见附录。所有实验均在单张NVIDIA H100 GPU上以FP32精度开展。推理速度（FPS）是在InterHand2.6M数据集上以批量大小1进行测量的。

#### 5.2. 实验结果

表1对比了DF-Mamba与其他骨干网络在五个数据集上的表现。DF-Mamba始终展现出更优的性能，在所有数据集上，其在联合位置误差和AUC分数方面均优于其他骨干网络。尽管一些传统骨干网络（如Swin-T、VMamba-T）在InterHand2.6M和RHP数据集上能提升性能，但在更具挑战性的场景中——比如基于深度的估计任务（NYU）和以自我为中心的估计任务（AssemblyHands）——它们的有效性便会大打折扣。在这些场景下，卷积网络固有的归纳偏置往往有助于高效学习；因此，基于Transformer和Mamba的架构会面临更大挑战。DF-Mamba凭借其带有可变形扫描机制的三元混合架构成功克服了这一局限。

在计算复杂度方面，DF-Mamba的模型尺寸和帧率与ResNet相当。从表1可以看出，ViT-S的帧率较低，这说明将Transformer模块应用于高分辨率特征图会带来较高的计算开销。Swin-T和VMamba-T通过分层架构解决了这一局限，而DF-Mamba则进一步利用卷积模块和门控卷积模块来提升效率。

下面，我们将 DF-Mamba 与每个数据集上的最先进 3D HPE 方法进行比较，并讨论其性能。InterHand2.6M。表 2 展示了 InterHand2.6M 上的结果。DF-Mamba 在复杂的双手交互中取得了最佳性能，MPJPE 得分为 7.94 mm（单

---

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td rowspan="2">Backbones</td><td rowspan="2">Size $\downarrow$ FPS $\uparrow$</td><td colspan="3">InterHand2.6M (MPJPE)</td><td style='text-align: center; word-wrap: break-word;'>RHP</td><td style='text-align: center; word-wrap: break-word;'>NYU</td><td colspan="2">DexYCB</td><td colspan="2">AssemblyHands</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Single $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>Two $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>All $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>EPE $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>ME $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>MPJPE $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>AUC $\uparrow$</td><td style='text-align: center; word-wrap: break-word;'>MPJPE $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>AUC $\uparrow$</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>ResNet-50 [^26]</td><td style='text-align: center; word-wrap: break-word;'>42M</td><td style='text-align: center; word-wrap: break-word;'>109.2</td><td style='text-align: center; word-wrap: break-word;'>8.10</td><td style='text-align: center; word-wrap: break-word;'>10.96</td><td style='text-align: center; word-wrap: break-word;'>9.63</td><td style='text-align: center; word-wrap: break-word;'>17.75</td><td style='text-align: center; word-wrap: break-word;'>8.43</td><td style='text-align: center; word-wrap: break-word;'>19.36</td><td style='text-align: center; word-wrap: break-word;'>84.80</td><td style='text-align: center; word-wrap: break-word;'>19.35</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>ConvNeXt-T [^42]</td><td style='text-align: center; word-wrap: break-word;'>45M</td><td style='text-align: center; word-wrap: break-word;'>101.1</td><td style='text-align: center; word-wrap: break-word;'>8.26 (+0.16)</td><td style='text-align: center; word-wrap: break-word;'>11.04 (+0.08)</td><td style='text-align: center; word-wrap: break-word;'>9.75 (+0.12)</td><td style='text-align: center; word-wrap: break-word;'>17.96 (+0.21)</td><td style='text-align: center; word-wrap: break-word;'>8.85 (+0.42)</td><td style='text-align: center; word-wrap: break-word;'>21.83 (+2.47)</td><td style='text-align: center; word-wrap: break-word;'>82.21 (-2.59)</td><td style='text-align: center; word-wrap: break-word;'>20.72 (+1.37)</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>ViT-S [^12]</td><td style='text-align: center; word-wrap: break-word;'>42M</td><td style='text-align: center; word-wrap: break-word;'>75.4</td><td style='text-align: center; word-wrap: break-word;'>-</td><td style='text-align: center; word-wrap: break-word;'>-</td><td style='text-align: center; word-wrap: break-word;'>-</td><td style='text-align: center; word-wrap: break-word;'>-</td><td style='text-align: center; word-wrap: break-word;'>-</td><td style='text-align: center; word-wrap: break-word;'>24.63 (+4.73)</td><td style='text-align: center; word-wrap: break-word;'>78.76 (-6.04)</td><td style='text-align: center; word-wrap: break-word;'>23.11 (+3.76)</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Swin-T [^41]</td><td style='text-align: center; word-wrap: break-word;'>45M</td><td style='text-align: center; word-wrap: break-word;'>103.4</td><td style='text-align: center; word-wrap: break-word;'>8.15 (+0.05)</td><td style='text-align: center; word-wrap: break-word;'>10.84 (-0.12)</td><td style='text-align: center; word-wrap: break-word;'>9.59 (-0.04)</td><td style='text-align: center; word-wrap: break-word;'>17.65 (-0.10)</td><td style='text-align: center; word-wrap: break-word;'>8.48 (+0.05)</td><td style='text-align: center; word-wrap: break-word;'>23.52 (+4.16)</td><td style='text-align: center; word-wrap: break-word;'>80.59 (-4.21)</td><td style='text-align: center; word-wrap: break-word;'>19.88 (+0.53)</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>VMamba-T [^40]</td><td style='text-align: center; word-wrap: break-word;'>46M</td><td style='text-align: center; word-wrap: break-word;'>100.8</td><td style='text-align: center; word-wrap: break-word;'>8.06 (-0.04)</td><td style='text-align: center; word-wrap: break-word;'>10.97 (+0.01)</td><td style='text-align: center; word-wrap: break-word;'>9.61 (-0.02)</td><td style='text-align: center; word-wrap: break-word;'>17.22 (-0.53)</td><td style='text-align: center; word-wrap: break-word;'>8.62 (+0.19)</td><td style='text-align: center; word-wrap: break-word;'>19.84 (+0.48)</td><td style='text-align: center; word-wrap: break-word;'>84.45 (-0.35)</td><td style='text-align: center; word-wrap: break-word;'>19.64 (+0.29)</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>SMamba-T [^68]</td><td style='text-align: center; word-wrap: break-word;'>43M</td><td style='text-align: center; word-wrap: break-word;'>92.8</td><td style='text-align: center; word-wrap: break-word;'>8.44 (+0.34)</td><td style='text-align: center; word-wrap: break-word;'>10.93 (-0.03)</td><td style='text-align: center; word-wrap: break-word;'>9.77 (+0.14)</td><td style='text-align: center; word-wrap: break-word;'>17.96 (+0.21)</td><td style='text-align: center; word-wrap: break-word;'>8.78 (+0.35)</td><td style='text-align: center; word-wrap: break-word;'>22.73 (+3.37)</td><td style='text-align: center; word-wrap: break-word;'>80.37 (-4.43)</td><td style='text-align: center; word-wrap: break-word;'>21.44 (+2.09)</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>DF-Mamba</td><td style='text-align: center; word-wrap: break-word;'>42M</td><td style='text-align: center; word-wrap: break-word;'>112.2</td><td style='text-align: center; word-wrap: break-word;'>7.94 (-0.16)</td><td style='text-align: center; word-wrap: break-word;'>10.53 (-0.43)</td><td style='text-align: center; word-wrap: break-word;'>9.32 (-0.31)</td><td style='text-align: center; word-wrap: break-word;'>17.16 (-0.59)</td><td style='text-align: center; word-wrap: break-word;'>7.96 (-0.47)</td><td style='text-align: center; word-wrap: break-word;'>17.80 (-1.56)</td><td style='text-align: center; word-wrap: break-word;'>87.31 (+2.51)</td><td style='text-align: center; word-wrap: break-word;'>18.78 (-0.57)</td></tr></table>

<div style="text-align: center;">Table 1. Comparison of backbone architectures across five datasets. Following the evaluation protocols for each dataset, Mean Per Joint Position Error (MPJPE), End Point Error (EPE), Mean Error (ME), and Area Under the Curve (AUC) scores are reported. For InterHand2.6M, MPJPE is reported for single-hand (Single), two-hand (Two), and overall (All) test sets. Numbers in parentheses indicate performance differences compared to ResNet-50, with green denoting improvement and red denoting degradation. Best results are highlighted in bold.</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td rowspan="2">Methods</td><td colspan="3">MPJPE</td><td rowspan="2">FPS $\uparrow$</td><td rowspan="2">Size $\downarrow$</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Single $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>Two $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>All $\downarrow$</td></tr><tr><td colspan="6">Model-based</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Zhang et al. [^76]</td><td style='text-align: center; word-wrap: break-word;'>-</td><td style='text-align: center; word-wrap: break-word;'>13.48</td><td style='text-align: center; word-wrap: break-word;'>-</td><td style='text-align: center; word-wrap: break-word;'>71.6</td><td style='text-align: center; word-wrap: break-word;'>143M</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Meng et al. [^44]</td><td style='text-align: center; word-wrap: break-word;'>8.51</td><td style='text-align: center; word-wrap: break-word;'>13.12</td><td style='text-align: center; word-wrap: break-word;'>10.97</td><td style='text-align: center; word-wrap: break-word;'>65.0</td><td style='text-align: center; word-wrap: break-word;'>55M</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Li et al. [^35]</td><td style='text-align: center; word-wrap: break-word;'>-</td><td style='text-align: center; word-wrap: break-word;'>8.79</td><td style='text-align: center; word-wrap: break-word;'>-</td><td style='text-align: center; word-wrap: break-word;'>76.0</td><td style='text-align: center; word-wrap: break-word;'>39M</td></tr><tr><td colspan="6">Model-free</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Moon et al. [^46]</td><td style='text-align: center; word-wrap: break-word;'>12.16</td><td style='text-align: center; word-wrap: break-word;'>16.02</td><td style='text-align: center; word-wrap: break-word;'>14.22</td><td style='text-align: center; word-wrap: break-word;'>330.5</td><td style='text-align: center; word-wrap: break-word;'>47M</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Fan et al. [^14]</td><td style='text-align: center; word-wrap: break-word;'>11.32</td><td style='text-align: center; word-wrap: break-word;'>15.57</td><td style='text-align: center; word-wrap: break-word;'>-</td><td style='text-align: center; word-wrap: break-word;'>87.1</td><td style='text-align: center; word-wrap: break-word;'>41M</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Hampali et al. [^23]</td><td style='text-align: center; word-wrap: break-word;'>10.99</td><td style='text-align: center; word-wrap: break-word;'>14.34</td><td style='text-align: center; word-wrap: break-word;'>12.78</td><td style='text-align: center; word-wrap: break-word;'>82.7</td><td style='text-align: center; word-wrap: break-word;'>48M</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Jiang et al. [^32]</td><td style='text-align: center; word-wrap: break-word;'>8.10</td><td style='text-align: center; word-wrap: break-word;'>10.96</td><td style='text-align: center; word-wrap: break-word;'>9.63</td><td style='text-align: center; word-wrap: break-word;'>109.2</td><td style='text-align: center; word-wrap: break-word;'>42M</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>DF-Mamba (Ours)</td><td style='text-align: center; word-wrap: break-word;'>7.94</td><td style='text-align: center; word-wrap: break-word;'>10.53</td><td style='text-align: center; word-wrap: break-word;'>9.32</td><td style='text-align: center; word-wrap: break-word;'>112.2</td><td style='text-align: center; word-wrap: break-word;'>42M</td></tr></table>

<div style="text-align: center;">Table 2. Comparison with SOTA methods on InterHand2.6M.</div>

手）、10.53 mm（双手）和 9.32 mm（全部）。与 Moon 等人 [^46] 提出的 InterHand 架构相比，计算成本仍高出约三倍。虽然我们的工作侧重于骨干网络架构，但未来工作需要进一步优化解码器，以在性能和效率之间取得更好的平衡。

在RHP数据集上，表3给出了相关对比结果。该数据集的训练图像数量有限，且背景多样。DF-Mamba无需依赖数据集中提供的真实尺度与手性信息（即无监督设置），便取得了最佳性能。这些结果表明，DF-Mamba能够有效利用有限数据进行学习，并泛化到多样化的场景。在仅使用深度信息的设置下，从表1可知只有DF-Mamba提升了性能，其平均误差仅为7.96毫米。其他基准方法的表现欠佳，可能是因为它们主要针对RGB图像训练设计。这些结果说明我们的DSSM在从深度图像中学习时具有更高的灵活性。表4也显示，DF-Mamba的性能优于专门处理深度图像的强基准方法（如[^17]）$^{1}$，这表明DF-Mamba在基于深度的3D手部姿态估计任务中表现强劲。

在DexYCB数据集上，表5展示了相关结果。该数据集专注于固定摄像头拍摄的手部与物体交互场景。DF-Mamba表现出稳健的性能，其MPJPE值降至17.80毫米，取得了显著提升。它的性能也优于Spurr等人提出的方法[^58]，后者采用HRNet32骨干网络[^64]。

表6展示了DF-Mamba在AssemblyHands数据集上与最先进方法的对比结果。AssemblyHands是一个更具挑战性的场景，涉及从以自我为中心的视角拍摄的真实世界手部与物体交互数据。我们的DF-Mamba在对比方法中实现了最低的MPJPE值，为18.78毫米。这证明了我们在涉及物体交互的动态、以自我为中心的场景中的优势。

#### 5.3. 消融研究与分析

可变形扫描的必要性。表7展示了未使用可变形扫描时的结果，此时模型退化为普通的Mamba SSM。我们发现性能显著下降，这凸显了引入可变形扫描的重要性。

网络架构。为了验证所提出的三元混合架构设计，我们针对其他混合架构和单模态架构开展了实验。我们的架构在第1至6阶段可表示为“CCDGDG”，其中C、D、G分别代表Conv、DDSM+FNN和门控卷积模块。我们进一步构建了由两种模块类型组合而成的混合架构（即CCGGGG和CCDDDD），以及仅由单一模块类型构成的单模态架构。表7中的实验结果表明，我们的三元混合架构的性能显著优于这两种混合及单模态变体，这证明了整合互补模块结构的有效性。

超参数K。我们通过改变K的取值来评估其影响。K的设计细节详见附录。表7中的结果显示，$K = 3^2$取得了最佳性能。即便采用$K = 1$，该模型的表现依然出色。

---

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'>Methods</td><td style='text-align: center; word-wrap: break-word;'>GT free</td><td style='text-align: center; word-wrap: break-word;'>EPE $\downarrow$</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Yang et al. [^72]</td><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'>19.95</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Spurr et al. [^57]</td><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'>19.73</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Moon et al. [^46]</td><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>20.89</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Jiang et al. [^32]</td><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>17.75</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Ours</td><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>17.16</td></tr></table>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'>Methods</td><td style='text-align: center; word-wrap: break-word;'>MA</td><td style='text-align: center; word-wrap: break-word;'>ME $\downarrow$</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Moon et al. [^45]</td><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'>9.22</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Xiong et al. [^69]</td><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'>8.61</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Fang et al. [^17]</td><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'>8.29</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Jiang et al. [^32]</td><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>8.43</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Ours</td><td style='text-align: center; word-wrap: break-word;'>✓</td><td style='text-align: center; word-wrap: break-word;'>7.96</td></tr></table>

<div style="text-align: center;">Table 3. Comparison on RHP.</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'>Methods</td><td style='text-align: center; word-wrap: break-word;'>MPJPE $\downarrow$</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Xiong et al. [^69]</td><td style='text-align: center; word-wrap: break-word;'>25.57</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Spurr et al. [^58]</td><td style='text-align: center; word-wrap: break-word;'>22.26</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Tse et al. [^62]</td><td style='text-align: center; word-wrap: break-word;'>21.22</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Zhou et al. [^79]</td><td style='text-align: center; word-wrap: break-word;'>19.36</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Ours</td><td style='text-align: center; word-wrap: break-word;'>17.80</td></tr></table>

<div style="text-align: center;">Table 4. Comparison on NYU.</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'>Methods</td><td style='text-align: center; word-wrap: break-word;'>MPJPE $\downarrow$</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Han et al. [^24]</td><td style='text-align: center; word-wrap: break-word;'>32.91</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Ohkawa et al. [^49]</td><td style='text-align: center; word-wrap: break-word;'>21.92</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Zhou et al. [^79]</td><td style='text-align: center; word-wrap: break-word;'>19.17</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Ours</td><td style='text-align: center; word-wrap: break-word;'>18.78</td></tr></table>

<div style="text-align: center;">Table 5. Comparison on DexYCB.</div>

<div style="text-align: center;">Table 6. Comparison on Assembly Hands.</div>

![Image](images/4c915ae9c3184c4d56f9c667ab4a248a9c6e3dbda2e03eb3eb7c6b81ca4f6a30.jpg)

<div style="text-align: center;">Figure 5. Qualitative examples. Predicted hand joints are color-coded by finger, with ground truth shown in black. The 3D visualizations provide rotated views, where several joint errors are highlighted using colored ellipses. The top two rows show examples from InterHand2.6M, and the bottom two rows from AssemblyHands.</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td rowspan="2">Methods</td><td colspan="2">DexYCB</td><td colspan="2">AssemblyHands</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>MPJPE $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>AUC $\uparrow$</td><td style='text-align: center; word-wrap: break-word;'>MPJPE $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>AUC $\uparrow$</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>DF-Mamba (Tribrid)</td><td style='text-align: center; word-wrap: break-word;'>17.80</td><td style='text-align: center; word-wrap: break-word;'>87.31</td><td style='text-align: center; word-wrap: break-word;'>18.78</td><td style='text-align: center; word-wrap: break-word;'>86.12</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>w/o deformable scan</td><td style='text-align: center; word-wrap: break-word;'>18.90</td><td style='text-align: center; word-wrap: break-word;'>85.59</td><td style='text-align: center; word-wrap: break-word;'>19.38</td><td style='text-align: center; word-wrap: break-word;'>84.93</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>w/ CCCCC (Monomodal)</td><td style='text-align: center; word-wrap: break-word;'>19.36</td><td style='text-align: center; word-wrap: break-word;'>84.80</td><td style='text-align: center; word-wrap: break-word;'>19.35</td><td style='text-align: center; word-wrap: break-word;'>85.24</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>w/ GGGGGG (Monomodal)</td><td style='text-align: center; word-wrap: break-word;'>18.70</td><td style='text-align: center; word-wrap: break-word;'>85.98</td><td style='text-align: center; word-wrap: break-word;'>19.13</td><td style='text-align: center; word-wrap: break-word;'>85.67</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>w/ DDDDDD (Monomodal)</td><td style='text-align: center; word-wrap: break-word;'>21.61</td><td style='text-align: center; word-wrap: break-word;'>85.57</td><td style='text-align: center; word-wrap: break-word;'>19.82</td><td style='text-align: center; word-wrap: break-word;'>84.44</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>w/ CCGGGG (Hybrid)</td><td style='text-align: center; word-wrap: break-word;'>18.67</td><td style='text-align: center; word-wrap: break-word;'>86.12</td><td style='text-align: center; word-wrap: break-word;'>18.97</td><td style='text-align: center; word-wrap: break-word;'>85.77</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>w/ CCDDDD (Hybrid)</td><td style='text-align: center; word-wrap: break-word;'>19.01</td><td style='text-align: center; word-wrap: break-word;'>85.25</td><td style='text-align: center; word-wrap: break-word;'>19.08</td><td style='text-align: center; word-wrap: break-word;'>85.61</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>w/ CCDGDG (Tribrid)</td><td style='text-align: center; word-wrap: break-word;'>17.80</td><td style='text-align: center; word-wrap: break-word;'>87.31</td><td style='text-align: center; word-wrap: break-word;'>18.78</td><td style='text-align: center; word-wrap: break-word;'>86.12</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>$K = 1$</td><td style='text-align: center; word-wrap: break-word;'>18.62</td><td style='text-align: center; word-wrap: break-word;'>86.09</td><td style='text-align: center; word-wrap: break-word;'>19.11</td><td style='text-align: center; word-wrap: break-word;'>85.77</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>$K = 3^{2}$ (default)</td><td style='text-align: center; word-wrap: break-word;'>17.80</td><td style='text-align: center; word-wrap: break-word;'>87.31</td><td style='text-align: center; word-wrap: break-word;'>18.78</td><td style='text-align: center; word-wrap: break-word;'>86.12</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>$K = 5^{2}$</td><td style='text-align: center; word-wrap: break-word;'>18.11</td><td style='text-align: center; word-wrap: break-word;'>86.68</td><td style='text-align: center; word-wrap: break-word;'>18.83</td><td style='text-align: center; word-wrap: break-word;'>86.06</td></tr></table>

<div style="text-align: center;">Table 7. Ablation and hyperparameter studies.</div>

该模型的表现优于其他骨干网络，而$K = 5^{2}$则取得了第二好的结果。

定性示例。图5展示了来自InterHand2.6M和AssemblyHands的定性示例。DF-Mamba能实现更精确的关节估计，这一点从彩色椭圆标出的位置误差减小中便可看出。在自中心视角场景下，即便关节被遮挡或无法直接从该视角看到，我们的方法仍能准确估计其位置，这体现了其在现实场景中的更强鲁棒性。

### 6. 结论

我们提出了DF-Mamba，这是一种用于3D手部姿态估计的新型骨干网络架构，它在低层结合了高效的卷积特征提取，在高层采用了可变形状态空间表示。我们在涵盖单手、双手及手物交互场景，以及RGB和深度模态的五个数据集上进行了大量实验，结果表明DF-Mamba的性能始终优于现有的骨干网络架构，同时推理速度也更快。未来值得研究的方向包括：1）将DF-Mamba拓展到3D人体姿态估计等更广泛的任务中；2）利用海量外部数据对其进行预训练；3）将其应用于最新的手部网格重建解码器（例如[^11] [^75]）；4）训练能够无需针对特定数据微调即可适配不同场景的基础模型。

---

致谢。本研究得到了DENSO IT LAB识别、控制与学习算法合作研究讲席项目（东京理科大学）的支持。本研究还得到了日本学术振兴会科研费资助，项目编号为23H00490和25K03135。感谢Nie Lin和Ruicong Liu在项目实施过程中提供的帮助。

## 参考文献

[^1]: Tatsuro Banno, Takehiko Ohkawa, Ruicong Liu, Ryosuke Furuta, and Yoichi Sato. Assemblyhands-x: Modeling 3d hand-body coordination for understanding bimanual human activities. 2025. 3

[^2]: William L. Brogan. Modern Control Theory. Quantum Publishers, 1974. 3

[^3]: Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In Proc. European Conference on Computer Vision (ECCV), pages 213–229, 2020. 2

[^4]: Yu-Wei Chao, Wei Yang, Yu Xiang, Pavlo Molchanov, Ankur Handa, Jonathan Tremblay, Yashraj S. Narang, Karl Van Wyk, Umar Iqbal, Stan Birchfield, Jan Kautz, and Dieter Fox. DexYCB: A benchmark for capturing hand grasping of objects. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021. 2, 6

[^5]: Liangjian Chen, Shih-Yao Lin, Yusheng Xie, Hui Tang, Yufan Xue, Xiaohui Xie, Yen-Yu Lin, and Wei Fan. Generating realistic training images based on tonality-alignment generative adversarial networks for hand pose estimation. arXiv preprint arXiv:1811.09916, 2018. 2

[^6]: Xingyu Chen, Yufeng Liu, Yajiao Dong, Xiong Zhang, Chongyang Ma, Yanmin Xiong, Yuan Zhang, and Xiaoyan Guo. Mobrecon: Mobile-friendly hand mesh reconstruction from monocular image. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 20544–20554, 2022. 1

[^7]: François Chollet. Xception: Deep learning with depthwise separable convolutions. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1251–1258, 2017. 3

[^8]: Xiangxiang Chu, Zhi Tian, Yuqing Wang, Bo Zhang, Haibing Ren, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Twins: Revisiting the design of spatial attention in vision transformers. In Proc. Annual Conference on Neural Information Processing Systems (NeurIPS), 2021. 3

[^9]: Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei. Deformable convolutional networks. In Proc. IEEE/CVF International Conference on Computer Vision (ICCV), pages 764–773, 2017. 3, 4

[^10]: Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In Proc. International Conference on Machine Learning (ICML), pages 933–941, 2017. 5

[^11]: Haoye Dong, Aviral Chharia, Wenbo Gou, Francisco Vicente Carrasco, and Fernando D De la Torre. Hamba: Single-view 3d hand reconstruction with graph-guided bi-scanning

Mamba. 载于神经信息处理系统年会（NeurIPS）论文集，第37卷：2127–2160页，2024年。2, 3, 8

[^12]: Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In Proc. International Conference on Learning Representations (ICLR), 2021. 2, 3, 5, 6, 7

[^13]: Ali Erol, George Bebis, Mircea Nicolescu, Richard D Boyle, and Xander Twombly. Vision-based hand pose estimation: A review. Computer Vision and Image Understanding, 108(1-2):52–73, 2007. 2

[^14]: Zicong Fan, Adrian Spurr, Muhammed Kocabas, Siyu Tang, Michael J Black, and Otmar Hilliges. Learning to disambiguate strongly interacting hands via probabilistic per-pixel part segmentation. In 2021 International Conference on 3D Vision (3DV), pages 1–10. IEEE, 2021. 7

[^15]: Zicong Fan, Omid Taheri, Dimitrios Tzionas, Muhammed Kocabas, Manuel Kaufmann, Michael J. Black, and Otmar Hilliges. ARCTIC: A dataset for dexterous bimanual hand-object manipulation. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023. 2

[^16]: Zicong Fan, Takehiko Ohkawa, Linlin Yang, Nie Lin, Zhishan Zhou, Shihao Zhou, Jiajun Liang, Zhong Gao, Xuanyang Zhang, Xue Zhang, et al. Benchmarks and challenges in pose estimation for egocentric hand interactions with objects. In Proc. European Conference on Computer Vision (ECCV), pages 428–448. Springer, 2024. 1, 2

[^17]: Linpu Fang, Xingyan Liu, Li Liu, Hang Xu, and Wenxiong Kang. Jgr-p2o: Joint graph reasoning based pixel-to-offset prediction network for 3d hand pose estimation from a single depth image. In Proc. European Conference on Computer Vision (ECCV), pages 120–137, 2020. 7, 8

[^18]: Daniel Y. Fu, Tri Dao, Khaled K. Saab, Armin W. Thomas, Atri Rudra, and Christopher Ré. Hungry hungry hippos: Towards language modeling with state space models. In Proc. International Conference on Learning Representations (ICLR), 2023. 3

[^19]: Liuhao Ge, Zhou Ren, Yuncheng Li, Zehao Xue, Yingying Wang, Jianfei Cai, and Junsong Yuan. 3d hand shape and pose estimation from a single rgb image. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10833–10842, 2019. 2

[^20]: Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. In Proc. Conference on Language Modeling (COLM), 2024. 2, 3, 4, 5

[^21]: Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, and Christopher Ré. Hippo: Recurrent memory with optimal polynomial projections. In Proc. Annual Conference on Neural Information Processing Systems (NeurIPS), pages 1474–1487, 2020. 3

[^22]: Shaoxiang Guo, Qing Cai, Lin Qi, and Junyu Dong. Clip-hand3d: Exploiting 3d hand pose estimation via context-aware prompting. In Proc. ACM International Conference on Multimedia (ACMMM), 2023. 2

## [^23]: Shreyas Hampali, Sayan Deb Sarkar, Mahdi Rad, and Vincent Lepetit. Keypoint transformer: Solving joint identifi-

---

在复杂的手物交互场景中实现精确的3D姿态估计。载于IEEE/CVF计算机视觉与模式识别会议（CVPR）论文集，第11090–11100页，2022年。7

[^24]: Shangchen Han, Po-chen Wu, Yubo Zhang, Beibei Liu, Linguang Zhang, Zheng Wang, Weiguang Si, Peizhao Zhang, Yujun Cai, Tomas Hodan, et al. Umetrack: Unified multiview end-to-end hand tracking for vr. In SIGGRAPH Asia 2022 Conference Papers, pages 1–9, 2022. 8

[^25]: Ali Hatamizadeh and Jan Kautz. Mambavision: A hybrid mamba-transformer vision backbone. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025. 3

[^26]: Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016. 2, 3, 6, 7

[^27]: Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861, 2017. 3

[^28]: Lin Huang, Jianchao Tan, Jingjing Meng, Ji Liu, and Junsong Yuan. Hot-net: Non-autoregressive transformer for 3d hand-object pose estimation. In Proc. ACM International Conference on Multimedia (ACMMM), pages 3136–3145, 2020. 2

[^29]: Tao Huang, Xiaohuan Pei, Shan You, Fei Wang, Chen Qian, and Chang Xu. LocalMamba: Visual state space model with windowed selective scan. arXiv preprint arXiv:2403.09338, 2024. 3

[^30]: Yunlong Huang, Junshuo Liu, Ke Xian, and Robert Caiming Qiu. Posemamba: Monocular 3d human pose estimation with bidirectional global-local spatio-temporal state space model. In Proc. AAAI Conference on Artificial Intelligence, pages 3842–3850, 2025. 3

[^31]: Umar Iqbal, Pavlo Molchanov, Thomas Breuel Juergen Gall, and Jan Kautz. Hand pose estimation via latent 2.5 d heatmap regression. In Proc. European Conference on Computer Vision (ECCV), pages 118–134, 2018. 2

[^32]: Changlong Jiang, Yang Xiao, Cunlin Wu, Mingyang Zhang, Jinghong Zheng, Zhiguo Cao, and Joey Tianyi Zhou. A2j-transformer: Anchor-to-joint transformer network for 3d interacting hand pose estimation from a single rgb image. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8846–8856, 2023. 1, 2, 5, 6, 7, 8

[^33]: Rudolph Emil Kalman. A new approach to linear filtering and prediction problems. Transactions of the ASME-Journal of Basic Engineering, 82(1):35–45, 1960. 3

[^34]: Bo Lang and Mooi Choo Chuah. Event-guided fusion-mamba for context-aware 3d human pose estimation. In Proc. IEEE Conference on Winter Conference on Applications of Computer Vision (WACV), pages 950–960, 2025. 3

[^35]: Mengcheng Li, Liang An, Hongwen Zhang, Lianpeng Wu, Feng Chen, Tao Yu, and Yebin Liu. Interacting attention graph for single image two-hand reconstruction. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2761–2770, 2022. 7

[^36]: Guan Ming Lim, Prayook Jatesiktat, and Wei Tech Ang. Mobilehand: Real-time 3d hand shape and pose estimation from color image. In Proc. International Conference on Neural Information Processing (ICONIP), pages 450–459, 2020. 1

[^37]: Nie Lin, Takehiko Ohkawa, Mingfang Zhang, Yifei Huang, Minjie Cai, Ming Li, Ryosuke Furuta, and Yoichi Sato. SiM-Hand: Pre-training for 3d hand pose estimation with contrastive learning on large-scale hand images in the wild. In Proc. International Conference on Learning Representations (ICLR), 2025. 2

[^38]: Ruicong Liu, Takehiko Ohkawa, Mingfang Zhang, and Yoichi Sato. Single-to-dual-view adaptation for egocentric 3d hand pose estimation. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 677–686, 2024. 1, 2

[^39]: Ruicong Liu, Takehiko Ohkawa, Tze Ho Elden Tse, Mingfang Zhang, Angela Yao, and Yoichi Sato. Leveraging RGB images for pre-training of event-based hand pose estimation. 2025. 2

[^40]: Yue Liu, Yunjie Tian, Yuzhong Zhao, Hongtian Yu, Lingxi Xie, Yaowei Wang, Qixiang Ye, Jianbin Jiao, and Yunfan Liu. VMamba: Visual state space model. In Proc. Annual Conference on Neural Information Processing Systems (NeurIPS), 2024. 2, 3, 5, 6, 7

[^41]: Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proc. IEEE/CVF International Conference on Computer Vision (ICCV), pages 10012–10022, 2021. 3, 5, 6, 7

[^42]: Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 3, 6, 7

[^43]: Ye Lu, Jie Wang, Jianjun Gao, Rui Gong, Chen Cai, and Kim-Hui Yap. A structure-aware and motion-adaptive framework for 3d human pose estimation with mamba. In Proc. IEEE/CVF International Conference on Computer Vision (ICCV), 2025. 3

[^44]: Hao Meng, Sheng Jin, Wentao Liu, Chen Qian, Mengxiang Lin, Wanli Ouyang, and Ping Luo. 3d interacting hand pose estimation by hand de-occlusion and removal. arXiv preprint arXiv:2207.11061, 2022. 7

[^45]: Gyeongsik Moon, Ju Yong Chang, and Kyoung Mu Lee. V2v-posenet: Voxel-to-voxel prediction network for accurate 3d hand and human pose estimation from a single depth map. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5079–5088, 2018. 8

[^46]: Gyeongsik Moon, Shoou-I Yu, He Wen, Takaaki Shiratori, and Kyoung Mu Lee. Interhand 2.6M: A dataset and baseline for 3d interacting hand pose estimation from a single rgb image. In Proc. European Conference on Computer Vision (ECCV), pages 548–564, 2020. 1, 2, 3, 6, 7, 8

## [^47]: Takehiko Ohkawa, Yu-Jhe Li, Qichen Fu, Ryosuke Furuta, Kris M Kitani, and Yoichi Sato. Domain adaptive hand keypoint and pixel localization in the wild. In Proc. European Conference on Computer Vision (ECCV), pages 68–87. Springer, 2022. 2, 3

---

[^48]: Takehiko Ohkawa, Ryosuke Furuta, and Yoichi Sato. Efficient annotation and learning for 3d hand pose estimation: A survey. International Journal of Computer Vision, 131(12):3193–3206, 2023.

[^49]: Takehiko Ohkawa, Kun He, Fadime Sener, Tomáš Hodan, Luan Tran, and Cem Keskin. Assembly Hands: towards egocentric activity understanding via 3d hand pose estimation. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12999–13008, 2023. 1, 2, 3, 6, 8

[^50]: Takehiko Ohkawa, Jihyun Lee, Shunsuke Saito, Jason Saragih, Fabian Prado, Yichen Xu, Shoou-I Yu, Ryosuke Furuta, Yoichi Sato, and Takaaki Shiratori. Generative modeling of shape-dependent self-contact human poses. In Proc. IEEE/CVF International Conference on Computer Vision (ICCV), 2025. 1, 2

[^51]: JoonKyu Park, Yeonguk Oh, Gyeongsik Moon, Hongsuk Choi, and Kyoung Mu Lee. Handoccnet: Occlusion-robust 3d hand mesh estimation network. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 1

[^52]: Georgios Pavlakos, Dandan Shan, Ilija Radosavovic, Angjoo Kanazawa, David Fouhey, and Jitendra Malik. Reconstructing hands in 3d with transformers. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9826–9836, 2024. 2

[^53]: Xiaohuan Pei, Tao Huang, and Chang Xu. Efficientvmamba: Atrous selective scan for light weight visual mamba. arXiv preprint arXiv:2403.09977, 2024. 3

[^54]: Yaopeng Peng, Milan Sonka, and Danny Z. Chen. Group vision transformer. In Proc. ACM International Conference on Multimedia (ACMMM), 2024. 3

[^55]: Aditya Prakash, Ruisen Tu, Matthew Chang, and Saurabh Gupta. 3d hand pose estimation in everyday egocentric images. In Proc. European Conference on Computer Vision (ECCV), 2024. 2

[^56]: Pengfei Ren, Chao Wen, Xiaozheng Zheng, Zhou Xue, Haifeng Sun, Qi Qi, Jingyu Wang, and Jianxin Liao. Decoupled iterative refinement framework for interacting hands reconstruction from a single rgb image. In Proc. IEEE/CVF International Conference on Computer Vision (ICCV), pages 8014–8025, 2023. 2

[^57]: Adrian Spurr, Jie Song, Seonwook Park, and Otmar Hilliges. Cross-modal deep variational hand pose estimation. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 89–98, 2018. 8

[^58]: Adrian Spurr, Umar Iqbal, Pavlo Molchanov, Otmar Hilliges, and Jan Kautz. Weakly supervised 3d hand pose estimation via biomechanical constraints. In Proc. European Conference on Computer Vision (ECCV), pages 211–228, 2020. 7, 8

[^59]: Naru Suzuki, Takehiko Ohkawa, Tatsuro Banno, Jihyun Lee, Ryosuke Furuta, and Yoichi Sato. Affordance-guided diffusion prior for 3d hand reconstruction. 2025. 2

[^60]: Jonathan Tompson, Murphy Stein, Yann Lecun, and Ken Perlin. Real-time continuous pose recovery of human hands using convolutional networks. ACM Transactions on Graphics (ToG), 33(5):1–10, 2014. 1, 2, 3, 6

[^61]: Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In Proc. International Conference on Machine Learning (ICML), pages 10347–10357, 2021. 3

[^62]: Tze Ho Elden Tse, Kwang In Kim, Ales Leonardis, and Hyung Jin Chang. Collaborative learning for hand and object reconstruction with attention-guided graph convolution. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1664–1674, 2022. 8

[^63]: Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Proc. Annual Conference on Neural Information Processing Systems (NeurIPS), 2017. 5

[^64]: Jingdong Wang, Ke Sun, Tianheng Cheng, Borui Jiang, Chaorui Deng, Yang Zhao, Dong Liu, Yadong Mu, Mingkui Tan, Xinggang Wang, et al. Deep high-resolution representation learning for visual recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 43(10):3349–3364, 2020. 7

[^65]: Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In Proc. IEEE/CVF International Conference on Computer Vision (ICCV), pages 568–578, 2021. 3

[^66]: Yilin Wen, Hao Pan, Lei Yang, Jia Pan, Taku Komura, and Wenping Wang. Hierarchical temporal transformer for 3d hand pose estimation and action recognition from egocentric rgb videos. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023. 2

[^67]: Yilin Wen, Hao Pan, Takehiko Ohkawa, Lei Yang, Jia Pan, Yoichi Sato, Taku Komura, and Wenping Wang. Generative hierarchical temporal transformer for hand pose and action modeling. Proc. European Conference on Computer Vision Workshops (ECCVW), 2024. 2

[^68]: Chaodong Xiao, Minghan Li, Zhengqiang Zhang, Deyu Meng, and Lei Zhang. Spatial-mamba: Effective visual state space models via structure-aware state fusion. In Proc. International Conference on Learning Representations (ICLR), 2025. 2, 3, 4, 5, 6, 7

[^69]: Fu Xiong, Boshen Zhang, Yang Xiao, Zhiguo Cao, Taidong Yu, Joey Tianyi Zhou, and Junsong Yuan. A2j: Anchor-to-joint regression network for 3d articulated pose estimation from a single depth image. In Proc. IEEE/CVF International Conference on Computer Vision (ICCV), pages 793–802, 2019. 8

[^70]: Guanyu Xu, Jiawei Hao, Li Shen, Han Hu, Yong Luo, Hui Lin, and Jialie Shen. Lgvit: Dynamic early exiting for accelerating vision transformer. In Proc. ACM International Conference on Multimedia (ACMMM), pages 1–12, 2023. 3

## [^71]: Chenhongyi Yang, Zehui Chen, Miguel Espinosa, Linus Ericsson, Zhenyu Wang, Jiaming Liu, and Elliot J. Crowley. Plainmamba: Improving non-hierarchical mamba in visual recognition. In Proc. British Machine Vision Conference (BMVC), 2024. 3

---

[^72]: Linlin Yang and Angela Yao. Disentangling latent hands for image synthesis and pose estimation. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9877–9886, 2019. 8

[^73]: Huan Yao, Changxing Ding, Xuanda Xu, and Zhifeng Lin. Decoupling heterogeneous features for robust 3d interacting hand poses estimation. In Proc. ACM International Conference on Multimedia (ACMMM), 2024. 2

[^74]: Weihao Yu and Xinchao Wang. Mambaout: Do we really need mamba for vision? In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025. 3, 5, 6

[^75]: Zhengdi Yu, Shaoli Huang, Chen Fang, Toby P Breckon, and Jue Wang. Acr: Attention collaboration-based regressor for arbitrary two-hand reconstruction. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12955–12964, 2023. 8

[^76]: Baowen Zhang, Yanggang Wang, Xiaoming Deng, Yinda Zhang, Ping Tan, Cuixia Ma, and Hongan Wang. Interacting two-hand 3d pose and shape reconstruction from single color image. In Proc. IEEE/CVF International Conference on Computer Vision (ICCV), pages 11354–11363, 2021. 7

[^77]: Xuan Zhang, Qian Bao, Qixuan Cui, Wankou Yang, and Qing Liao. Pose magic: Efficient and temporally consistent human pose estimation with a hybrid mamba-gcn network. In Proc. AAAI Conference on Artificial Intelligence, pages 10248–10256, 2025. 3

[^78]: Yumeng Zhang, Li Chen, Yufeng Liu, Junhai Yong, and Wen Zheng. Adaptive wasserstein hourglass for weakly supervised hand pose estimation from monocular rgb. In Proc. ACM International Conference on Multimedia (ACMMM), pages 1944–1952, 2020. 2

[^79]: Yuxiao Zhou, Marc Habermann, Weipeng Xu, Ikhsanul Habibie, Christian Theobalt, and Feng Xu. Monocular real-time hand shape and motion capture using multi-modal data. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5346–5355, 2020. 2, 6, 8

[^80]: Zhishan Zhou, Shihao Zhou, Zhi Lv, Minqiang Zou, Yao Tang, and Jiajun Liang. A simple baseline for efficient hand mesh reconstruction. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1367–1376, 2024. 2

[^81]: Lianghui Zhu, Bencheng Liao, Qian Zhang, Xinlong Wang, Wenyu Liu, and Xinggang Wang. Vision Mamba: Efficient visual representation learning with bidirectional state space model. In Proc. International Conference on Machine Learning (ICML), 2024. 2, 3, 5

[^82]: Xizhou Zhu, Han Hu, Stephen Lin, and Jifeng Dai. Deformable convnets v2: More deformable, better results. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9308–9316, 2019. 3

[^83]: Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. In Proc. International Conference on Learning Representations (ICLR), 2021. 2

[^84]: Christian Zimmermann and Thomas Brox. Learning to estimate 3d hand pose from single rgb images. In Proc.

IEEE/CVF国际计算机视觉会议（ICCV），第4903–4911页，2017年。2, 6

[^85]: Christian Zimmermann, Duygu Ceylan, Jimei Yang, Bryan Russell, Max Argus, and Thomas Brox. Freihand: A dataset for markerless capture of hand pose and shape from single rgb images. In Proc. IEEE/CVF International Conference on Computer Vision (ICCV), pages 813–822, 2019. 2