# GoTrack：通用的6自由度物体位姿优化与跟踪

Van Nguyen Nguyen $^{1*}$ Christian Forster $^{2}$ Sindi Shkodrani $^{2}$ Vincent Lepetit $^{1}$ Bugra Tekin $^{2}$ Cem Keskin $^{2}$ Tomas Hodan $^{2}$

 $^{1}$LIGM，桥路学校 $^{2}$Meta Reality Labs

## 摘要

我们提出了GoTrack，这是一种高效且精确的基于CAD的6自由度物体位姿优化与跟踪方法，无需针对特定物体进行训练即可处理各类物体。现有的跟踪方法仅依赖“分析-合成”思路来实现模型与帧之间的配准，而GoTrack还集成了帧间配准功能，既能节省计算资源，又能稳定跟踪效果。这两种配准方式均通过光流估计来实现。其中，模型与帧之间的配准过程比现有方法简单得多，仅需使用标准的神经网络模块（在DINOv2基础上训练的Transformer），无需单独的评分网络就能生成可靠的位姿置信度分数。至于帧间配准——由于连续视频帧通常极为相似，属于相对简单的问题——我们采用了轻量级的现成光流模型。实验表明，GoTrack可与现有的粗略位姿估计方法无缝结合，形成最简化的流程，在标准的6自由度物体位姿估计与跟踪基准测试中取得最先进的仅RGB图像结果。我们的源代码和训练好的模型可在<https://github.com/facebookresearch/gotrack>免费获取。

### 1. 引言

基于视觉的6自由度物体位姿估计是机器人技术与AR/VR领域的重要任务，过去十年间该领域取得了显著进展 [^21] [^32] [^34] [^35] [^37] [^40] [^44] [^48] [^49] [^51] [^54] [^66] [^71] [^81]。然而，大多数方法针对单张图像输入设计，而现实应用往往需要6自由度物体跟踪功能，即要在连续帧中估算物体的位姿。

传统的6自由度物体跟踪方法依赖于关键点 [^52] [^56] [^57] [^61] [^69]、边缘 [^6] [^13] [^19] [^58] [^62]，或直接优化 [^4] [^7] [^41] [^59] [^67]。更近期出现的基于学习的跟踪方法 [^8] [^17] [^42] [^72] [^75]展现出了卓越的性能

![Image](images/9709a9087bfb6172b28e176a4fc50da3e5377c7b248575539a1f3aac7ad79e6c.jpg)

<div style="text-align: center;">Figure 1. Given a CAD model of the target object, our method can reliably track the 6DoF object pose in challenging conditions such as dynamic scenes with hand-object interactions. The method is applicable to diverse objects and image types, without requiring any domain-specific training. Each row of this figure shows contours of the tracked model in videos from the HOT3D dataset [^3] captured by Quest 3 [^43] and Project Aria [^15] headsets.</div>

这类方法虽然性能不错，但依赖于RGB-D输入图像[^17] [^75]，或者需要针对每个目标物体专门定制的海量训练数据集[^8] [^42] [^72]，这限制了它们的应用范围。与姿态跟踪密切相关的一个问题是姿态优化。在这两类问题中，输入均为图像、相机内参以及初始物体姿态，输出则为优化后的物体姿态。近期用于6自由度物体姿态优化的方法[^35] [^44] [^77]均采用“分析-合成”思路，通常先渲染初始姿态下的3D网格模型，再预测其相对于输入帧的相对姿态。姿态的预测要么通过直接回归实现[^35] [^77]，要么借助2D方式完成。

---

在输入图像与物体模型的RGB-D渲染图之间建立3D对应关系[^44]。在这两种情况下，相关方法都需要解决模型到帧的配准问题，由于存在合成数据与真实数据之间的域差异，这一任务并不简单，且需要计算量较大的神经网络来支持。

在同步定位与地图构建（SLAM）[^14]中，3D场景模型与相机之间的姿态是通过视频帧中的两类配准操作持续估算得出的：模型到帧配准和帧到帧配准。模型到帧配准的主要作用是通过将当前帧与模型重新对齐，来纠正随时间积累的漂移现象，从而确保全局一致性。另一方面，帧到帧配准则侧重于在相机移动过程中提供平滑、连续的 motion 估算，使SLAM方法能够高效计算出帧间的相对变换关系。尽管这两类配准方式的协同作用已被证明对SLAM十分有效，但在6自由度物体跟踪领域，研究却一直仅聚焦于模型到帧配准。

在本文中，我们提出了GoTrack，一种通用的6自由度物体跟踪与姿态优化方法。该方法采用了与近期基于模型的姿态优化方法类似的“分析-合成”思路来实现模型到帧配准，同时借鉴了SLAM的理念，引入了帧到帧配准以降低跟踪时的计算成本。GoTrack中的这两类配准均通过光流估计来实现。

对于模型到帧配准，我们在DINOv2的基础上训练一个Transformer解码器，用以预测：(1) 合成物体模板与输入图像之间的光流场；(2) 输入图像中可见的模板像素掩码。利用这些预测结果，我们将可见像素与模型坐标系中的3D点关联起来，从而构建2D-3D对应关系；这些3D点是通过模板的深度通道计算得出的。随后，6自由度姿态便可通过PnP-RANSAC算法[^16] [^36]来计算得出。

对于帧到帧配准，我们基于这样一个观察：由于连续视频帧之间的差异通常很小，因此该问题相对容易解决，所以我们采用一个小型的现成光流估计网络来求解[^65]。帧到帧光流用于将2D-3D对应关系传递到下一帧，只有当帧到帧跟踪失效时，才会触发计算量更大的模型到帧配准。除了能节省计算资源外，使用帧到帧配准还能获得更稳定的跟踪结果，尤其是在物体被遮挡的情况下。

此外，我们还展示了所提出的模型到帧配准方法如何与FoundPose中的基于词袋模型的模板检索机制无缝结合[^51]，从而构建出高效且精准的端到端姿态估计流程。实验表明，该方法在标准的6自由度物体姿态优化[^28]及跟踪[^68] [^79]基准测试中均取得了业界领先的效果。

综上所述，我们的主要贡献如下：

1. 我们提出了一种基于流的6自由度姿态优化器，它比现有方法更简单，能够修正较大的姿态偏差，无需额外的评分网络即可生成可靠的姿态置信度分数，并且在6自由度物体姿态估计与跟踪基准测试中均能达到仅使用RGB数据的业界领先水平。

2. 我们设计了一种无缝集成的6自由度姿态估计流程，该流程通过模板检索快速生成粗略姿态，再利用基于流的优化器对其进行优化。由于两个阶段都依赖DINOv2，因此该流程易于实现。

3. 我们构建了鲁棒的6自由度姿态跟踪流程，该流程在优化器基础上增加了帧间光流估计功能，从而提升了跟踪效率，减少了抖动现象，并增强了对遮挡情况的适应能力。

### 2. 相关工作

在本节中，我们将概述与未知物体姿态估计、优化及跟踪密切相关的现有方法。

物体姿态估计与精化。最近的 6DoF 物体姿态估计研究引入了多种基准 [^5] [^10] [^12] [^22] [^26] [^31] [^64] [^80]，它们推动了许多基于深度学习的方法 [^29] [^32] [^34] [^37] [^38] [^40] [^53] [^54] [^66] [^81]。如 BOP Challenge 2023 的报告 [^28] 所示，大多数最先进的 6DoF 姿态估计方法都依赖于三阶段流程：(1) 2D 检测/分割，(2) 粗略姿态估计，以及 (3) 姿态精化。已见物体任务中排名前 16 的方法和未见物体任务中排名前 13 的方法均依赖于该流程。然而，第三阶段精化带来的精度提升通常伴随巨大的开销，例如，BOP 2023 的获胜者 GenFlow [^44] 的粗略版本，对于平均约含 6 个物体的图像，每幅耗时 3.8 秒。相比之下，基于渲染模板、通过对 DINOv2 [^50] 冻结特征进行最近邻搜索来估计粗略姿态的方法，例如 ZS6D [^2] 和 FoundPose [^51]，在速度快得多的同时也展现出了令人鼓舞的结果。受这些结果的启发，我们在冻结的 DINOv2 [^50] 之上构建姿态精化器，这也使我们能够与最新的粗略姿态方法无缝集成。

优化阶段对于实现最先进的精度至关重要，它通常通过迭代的渲染与比较方法来实现，该方法采用直接回归技术（对于已见过的物体使用CosyPose [^34]；对于未见过的物体则使用MegaPose [^35]和FoundationPose [^77]），或者利用2D-3D对应关系（GenFlow [^44]）。这些方法的一个共同特点是都需要进行姿态排序/选择步骤，而这需要一个额外的网络，该网络以测试图像以及估计姿态下的物体渲染图作为输入，并输出姿态得分。相比之下，我们的方法无需额外的评分网络即可生成可靠的姿态得分。

---

物体姿态跟踪。传统的物体跟踪方法可分为基于关键点的 [^52] [^56] [^57] [^61] [^69]、基于边缘的 [^6] [^13] [^19] [^58] [^62]，以及直接优化的 [^4] [^7] [^41] [^59] [^67]。虽然关键点和直接优化方法不适用于无纹理的物体，但基于边缘的方法通常难以应对背景杂乱和纹理干扰的问题。为解决这些问题，人们提出了基于学习的方法 [^8] [^17] [^42] [^72] [^75]。不过，这些方法要么依赖RGB-D输入图像 [^17] [^75]，要么需要针对每个目标物体的大量训练数据 [^8] [^42] [^72]。为解决这些问题，Nguyen等人 [^46] 提出了一种用于跟踪未见物体6DoF姿态的方法。尽管他们的方法仅在简单场景或简单运动下展示了效果，但我们关注的是更具挑战性的条件，即物体可能被严重遮挡或被手部操控的情况。另一类研究属于类别级方法 [^39] [^70]，它们专注于训练类别中未见过的实例。最近，出现了几种实例级跟踪方法 [^8] [^18] [^37] [^75]、基于模型的方法 [^30] [^63] [^78]，以及针对未见物体的无模型跟踪方法 [^74] [^76]。然而，这些方法要么仅针对已见过的物体，要么使用深度图像，这限制了它们的适用性。

针对未见物体的物体姿态优化以实现跟踪。与我们方法最为相似的是在[^35]、[^37]、[^44]以及[^77]中提出的方法。一个典型的例子是DeepIM [^37]，它提出了一种基于深度学习的迭代式渲染与比较姿态优化方法。尽管该方法对未见物体具有泛化能力，但实验仅局限于合成图像。MegaPose [^35]进一步拓展了这种方法，通过在大规模数据集上训练，证明了其在真实图像中对未见物体的强大泛化能力。在2023年的BOP挑战赛[^28]中，GenFlow [^44]凭借基于RAFT [^65]的架构来估计从模板到真实输入图像的流，最终获奖。在本研究中，我们也提出了一种基于流的未见物体姿态优化方法。不过，我们的方法无需依赖昂贵的形状约束、可微分的PnP、专门的神经架构，也无需姿态评分阶段即可达到最先进的性能。此外，现有方法都仅专注于模型到帧的配准。当应用于跟踪时，它们仅通过姿态参数过渡到下一帧，而丢弃了关于已建立配准的像素级信息。本研究证明，通过帧间流传播2D-3D对应关系能够提升效率并减少抖动。

### 3. 方法

在本节中，我们首先提出一种针对未见物体的6自由度姿态优化方法（第3.1节），随后展示该方法如何支持高效流程，以从单张图像中估算精确的物体姿态（第3.2节）以及跟踪图像序列中的物体姿态（第3.3节）。

#### 3.1. 基于流的物体姿态优化

问题表述：给定一个物体的CAD模型、一张已知内参且显示该物体处于未知姿态的RGB图像，以及对该物体姿态的粗略估计，我们的目标是优化姿态，使得模型的2D投影与图像中物体的外观高度吻合。我们假设物体是刚性的，用刚性变换(R, t)来表示其姿态，其中R是3D旋转，t是从模型坐标系到相机坐标系的3D平移。

方法概述：我们采用了分析-综合法，这与近期几种基于模型的姿态优化方法类似[^35]、[^37]、[^44]、[^77]。具体而言，我们根据以下两项来预测优化后的姿态：(i) 显示给定粗略姿态下CAD模型的RGB-D模板；(ii) 输入RGB图像中物体区域的裁剪图（该区域由CAD模型和粗略姿态确定）。我们并未像[^35]、[^37]、[^77]中那样直接回归渲染图与裁剪图之间的相对6自由度姿态，而是训练网络来预测密集的2D-2D对应关系（表示为光流场），同时生成两张图像中都可见的像素掩码。随后，利用模板的深度信息将对应关系提升至2D-3D，再应用PnP-RANSAC [^16]、[^36]来估算姿态。该优化方法可以迭代多次以提高精度。

预测二维输出（流和可见性掩码）而非相对的6自由度位姿，能使网络更易于训练、更具可解释性且应用范围更广——我们展示了如何利用预测的对应关系来计算可靠且可控的位姿评分，以及在跟踪场景下如何通过帧间流自然传播这些关系。这种基于对应关系的方法也能支持二维跟踪（只需关闭三维推理部分）。

预测二维-三维对应关系。设$I_c$为输入RGB图像中围绕给定物体二维边界框的透视裁剪区域。该裁剪区域的获取方式如[^51]所述：将图像变换到光轴穿过二维边界框中心的虚拟针孔相机视角下。接着设$I_t$为使用针孔相机渲染得到的RGB-D模板，该模板展示的是初始位姿为$(\mathbf{R}_0, \mathbf{t}_0)$的物体；设$\mathbf{M}_t$为$I_t$中物体轮廓的二值掩码，设O为表示物体模型表面的三维点集合。$I_t$与$I_c$的分辨率相同。

给定裁剪区域$\mathbf{I}_c$和模板$\mathbf{I}_t$，我们的目标是建立二者像素之间的对应关系。如果像素$\mathbf{u}_t \in \mathbf{I}_t$和$\mathbf{u}_c \in \mathbf{I}_c$与同一个三维点$\mathbf{x} \in O$的二维投影对齐，即满足$\mathbf{u}_t = \pi_t(\mathbf{R}_0 \mathbf{x} + \mathbf{t}_0)$和$\mathbf{u}_c = \pi_c(\mathbf{R} \mathbf{x} + \mathbf{t})$，其中$\pi$: $\mathbb{R}^3 \mapsto \mathbb{R}^2$为二维投影算子，$(\mathbf{R}, \mathbf{t})$为真实物体位姿，则我们认为这两个像素是对应的。

为了建立这种对应关系，我们训练神经网络为每个像素$\mathbf{u}_t \in \mathbf{M}_t$预测两个输出：类似——

---

![Image](images/f7d8ed32ad138e9669809e774dc1fc68d02025df01c3854ddab5562686981324.jpg)

<div style="text-align: center;">Figure 2. Overview of the proposed efficient pipeline for 6DoF pose estimation. Each object is first onboard by rendering RGB-D templates of its 3D model, and by extracting DINOv2 [^50] feature maps from RGB channels of the templates. At inference, we first crop the input image around a 2D object detection (from CNOS [^47]) and retrieve a template that shows the object in a similar pose as in the crop by the BoW-based approach from [^51]. Then we use a transformer decoder [^73] with a DPT head [^55] to predict (1) 2D-2D correspondences between the template and the input, and (2) a mask of template pixels that are visible in the input. Next we lift the 2D-2D correspondences established at visible template pixels to 2D-3D using the depth channel of the template, and estimate the pose by PnP-RANSAC.</div>

对应像素$v_{t\rightarrow c}(\mathbf{u}_t)$可见的概率$\mathbf{u}_c \in \mathbf{I}_c$，以及二维流向量$\mathbf{f}_{t\rightarrow c}(\mathbf{u}_t)$，使得$\mathbf{u}_t + \mathbf{f}_{t\rightarrow c}(\mathbf{u}_t) = \mathbf{u}_c$。根据网络的预测结果，我们通过关联像素$D = \{(\mathbf{u}_c, \mathbf{u}_t, w)\}$和$\mathbf{u}_u$来收集一组加权二维-二维对应关系$\mathbf{u}_t$，前提是它们满足$\mathbf{f}_{t\rightarrow c}(\mathbf{u}_t)$所描述的关系，且$v_{t\rightarrow c}(\mathbf{u}_t)$的值高于阈值$\tau_v$。权重w由$v_{t\rightarrow c}(\mathbf{u}_t)$决定。最后，我们利用模板的深度信息和已知的相机内参，将每个关联的模板像素$\mathbf{u}_t$转换为三维点$\mathbf{x} \in O$，从而得到一组加权的二维-三维对应关系$C = \{(\mathbf{u}_c, \mathbf{x}, w)\}$。

位姿拟合。通过求解透视n点（PnP）问题，即可从已建立的对应关系C中估算出精修后的位姿$(\mathbf{R}, \hat{\mathbf{t}})$。如[^51]所述，我们采用EPnP算法[^36]结合RANSAC拟合方案[^16]来求解该问题，以保证鲁棒性。在该方案中，PnP会在随机选取的包含4个对应关系的最小集合上反复求解，最终输出的是内点对应关系数量最多的位姿假设$C' \subseteq C$，且该假设下的二维重投影误差[^36]低于阈值$\tau_r$。随后，再利用所有内点通过Levenberg-Marquardt优化算法[^45]对最终位姿进行精修。

位姿估计的质量。我们将估计位姿$(\hat{\mathbf{R}},\hat{\mathbf{t}})$的质量定义为：$q = s'/s$，其中$s'$为内点对应关系C的权重w之和，C为所有对应关系C的权重之和。如第4节所示，这种简单的质量评估方法在多假设精修场景中挑选最佳位姿估计时竟出奇地有效。需要注意的是，现有方法[^35]、[^44]、[^77]为此都需要专门设计位姿评分网络。

网络架构与训练。我们采用一个冻结的DINOv2 [^50] 骨干网络，用于从图像裁剪区域 $I_c$ 和模板图像 $I_t$ 中提取特征。随后，我们将这些特征输入到CroCov2 [^73] 提出的双分支Transformer解码器中（一个分支处理模板，另一个分支处理裁剪区域），最后通过DPT头 [^55] 对模板分支的输出进行处理，从而得到预测结果。解码器和头网络通过最小化所有像素上的以下损失来联合训练 $\mathbf{u}_t \in \mathbf{M}_t$。

 $$ \begin{aligned}\mathcal{L}(\mathbf{u}_{t})=&\mathrm{BCE}\big(v_{t\to c}(\mathbf{u}_{t}),\bar{v}_{t\to c}(\mathbf{u}_{t})\big)\\&+\bar{v}_{t\to c}(\mathbf{u}_{t})\left\|\mathbf{f}_{t\to c}(\mathbf{u}_{t})-\bar{\mathbf{f}}_{t\to c}(\mathbf{u}_{t})\right\|_{1},\end{aligned} $$

其中BCE是预测的可见性 $v_{t\rightarrow c}(\mathbf{u}_t)$ 与二值真值可见性 $\bar{v}_{t\rightarrow c}(\mathbf{u}_t)$ 之间的二元交叉熵损失。与RAFT [^65] 类似，光流预测是通过最小化预测流向量 $\mathbf{f}_{t\rightarrow c}(\mathbf{u}_t)$ 与真值流向量 $\bar{\mathbf{f}}_{t\rightarrow c}(\mathbf{u}_t)$ 之间的L1距离来进行监督的。需要注意的是，仅当对应的裁剪像素 $\mathbf{u}_t \in \mathbf{M}_t$ 可见时，即满足 $\mathbf{u}_c \in \mathbf{M}_c$ 条件时，才对模板像素 $\bar{v}_{t\rightarrow c}(\mathbf{u}_t) = 1$ 处的光流预测进行监督。真值可见性数值 $\mathbf{u}_c \in \mathbf{M}_c$ 由 $\mathbf{M}_c(\mathbf{u}_t + \mathbf{f}_{t\rightarrow c}(\mathbf{u}_t))$ 定义，其中 $\mathbf{M}_c$ 是在裁剪区域内定义的模态掩码（仅在训练时需要 $(\mathbf{M}_c)$）。

我们在MegaPose-GSO数据集 [^35] 上训练网络，该数据集包含100万个用BlenderProc [^9] 渲染的PBR图像，展示了1000个GSO物体 [^11]，这些物体标注了6DoF位姿和2D分割掩码。与 [^35] 中的做法一致，我们在训练期间应用强烈的图像增强策略，以避免模型过度拟合合成图像域，从而使其能够在推理阶段应用于真实图像。我们同样按照 [^35] 中的方法随机扰动真值位姿。

---

#### 3.2. 高效的物体位姿估计流程

与针对未见物体的最先进6DoF位姿估计方法 [^28] 类似，我们采用了三阶段流程：2D物体检测、粗略6DoF位姿估计、6DoF位姿精修。虽然第一阶段可由任意2D物体检测器实现（例如CNOS [^47]），但我们流程的关键差异在于后两个阶段：我们采用高效的模板检索方法进行粗略位姿估计，并采用基于光流的方法进行位姿精修，这两者均基于冻结的DINOv2 [^50]。这一精简的流程在保持较高精度的同时，速度也快于现有方法（见第4节）。图2展示了该流程。

物体导入。首先，在离线导入阶段，我们预先渲染一组RGB-D模板（我们的实验中为800个），这一做法与 [^51] 类似。给定一个贴有纹理的3D物体模型，我们渲染出该模型在不同方向上的模板，这些方向是均匀覆盖3D旋转群SO(3)的采样结果 [^1]；模型渲染采用标准光栅化技术 [^60]，背景为灰色且光照固定。我们还会从模板的RGB通道中预先提取DINOv2特征图。

基于模板检索的粗略位姿估计。给定2D物体检测结果（例如来自CNOS [^47] 的结果），我们采用FoundPose [^51] 中的快速模板检索方法来生成粗略的物体位姿。在第4节中，我们将展示：与检索到的模板相关的位姿足以让基于光流的精修器收敛到精确的位姿。

无缝集成。从[^51]采用的模板检索方法利用了基于词袋（BoW）的描述符，这些描述符是由冻结的DINOv2模型生成的补丁特征计算而来。由于我们的细化器使用了相同的骨干网络，因此每个图像裁剪块只需执行一次特征提取过程。提取出的DINOv2特征既可用于检索模板，也可用于细化相关的姿态。除了共享同一骨干网络外，这两个阶段实际上还共享完全相同的基于模板的物体表示。

#### 3.3. 扩展到物体姿态跟踪

通过细化进行跟踪的局限性。所提出的细化器可直接应用于在图像序列中跟踪6自由度物体姿态。正如[^35]和[^77]中所述，这可以通过将前一帧的姿态作为下一帧的初始姿态来实现。尽管这种方法很简单，但它往往会出现抖动现象，即快速波动或不一致性，通常会导致跟踪输出出现摇晃或不稳定的情况。我们认为这种局限性的产生是因为连续帧之间的预测仅通过姿态参数相连，而前一帧中像素级的模型到图像配准信息被丢弃了。结果，作为所有“分析-合成”细化过程中的核心挑战，配准过程可能会导致跟踪输出的准确性显著降低。

![Image](images/d3b6fcf9feb769d7fede8ec3336cdc9e22eaa460427d12f5fec5a7ff8e8455f6.jpg)

![Image](images/666191a08d215a73ea6542f195e69cd045289dd95345be211c89e5a1f4c43cb0.jpg)

<div style="text-align: center;">Figure 3. Frame-to-frame consistency for tracking. Besides the model-to-frame registration (between a synthetic template and the real input image), which is the main building block of existing 6DoF object tracking methods, we introduce frame-to-frame registration, which reduces jitter and increases efficiency. As the difference between consecutive video frames is typically minimal, estimating frame-to-frame correspondences is a simpler problem and can be solved by a light-weight optical flow network such as RAFT [^65]. The model-to-frame correspondences need to be established only in case of a low frame-to-frame tracking confidence.</div>

传统方法在每个新帧中几乎都要重新启动。我们通过引入帧间一致性来解决这一局限性，具体内容将在下文介绍。

帧间一致性。受SLAM文献[^14]的启发——在该文献中，3D场景模型与相机之间的姿态不仅通过模型到帧的配准，还通过帧到帧的配准从视频帧中持续估计出来——我们通过在管道中预测前后视频帧之间的光流，引入了直接的帧间连接。

具体来说，我们通过构建一个虚拟相机来观察当前姿态估计下的物体模型，从而获得当前帧j的透视裁剪图；接着使用RAFT [^65]来预测前后裁剪图之间的光流。然后，我们将该光流应用到前一帧i中确定的内点2D-3D对应点$C'_i$的2D坐标上，从而得到一组传播后的内点$C'_{i\rightarrow j}$；再通过EPnP算法根据$C'_{i\rightarrow j}$重新估计姿态。如果相对于重新估计的姿态而言，内点比例$\tau_i$高于阈值（通常为0.8），我们就将重新估计的姿态作为帧j的最终姿态估计值，并继续处理下一帧。反之，如果内点比例低于阈值，我们就预测模型到帧的光流（如前所述）以获得新的2D-3D对应点集$C_j$。在这种情况下，为了增强姿态的时间平滑性，我们会合并集合$C'_{i\rightarrow j}$。

---

![Image](images/c1cb73579ef4c093e863a41ce8c415628bbc03f88a7c6c6808f61f6fee21a3f2.jpg)

<div style="text-align: center;">Figure 4. Example results of the GoTrack refiner on LM-O [^5], YCB-V [^79], and T-LESS [^23]. The input image is shown in the first column, and the template retrieved using the BoW-based approach [^51] in the second. The third and fourth columns show the predictions of our network, which can then be used to remap pixels from the template to the input image as shown in the fifth column. The last column presents the final pose estimated by PnP-RANSAC from 2D-3D correspondences (the contour of the object model in the initial pose is shown in blue, and in the estimated pose in red). As shown in the third column, our method can reliably predict which part of the object is visible, despite never seeing the object during training.</div>

通过从第一组中选取所有对应关系，并从第二组中随机选择最多 $C_j$ 个对应关系来生成 $r|C_{i\to j}'|$，也就是说，从两组中选取的对应关系比例最多为 r。随后利用 EPnP 算法根据合并后的数据集估算姿态，并将合并数据集中的内点对应关系传递到下一帧。

参数r用于控制从上一帧和当前帧中选取的对应点的比例，它可以有效调节抖动与漂移风险（即偏离实际物体姿态的风险）之间的权衡。当r值较小时，跟踪过程会更平滑，但可能会出现漂移现象；而当r值较大时，跟踪过程可能会产生抖动。

### 4. 实验

本节将介绍6自由度物体姿态精修（第4.1节）和6自由度物体姿态跟踪（第4.2节）的实验设置，并将GoTrack与针对这些任务的现有最先进的仅RGB方法进行比较。具体而言，我们会在BOP数据集[^35]上，将GoTrack与精修方法MegaPose [^44]、GenFlow [^28]进行对比；会在RBOT数据集[^67]上，将GoTrack与跟踪方法LDT3D [^62]、SRT3D [^72]、DeepAC [^68]进行对比；还会在YCB-V数据集[^79]上，将GoTrack与跟踪方法PoseCNN [^8]、PoseRBPF [^79]进行对比。在这些跟踪方法中，只有LDT3D和SRT3D能够处理未见过的物体，而DeepAC、PoseCNN和PoseRBPF则是在测试物体上训练的。

实现细节。本次评估所采用的精修方法版本使用了800个模板（其中所描绘的物体朝向之间的夹角约为$25^{\circ}$），模板和裁剪分辨率均为$280 \times 280$像素，采用基于ViT-S架构的DINOv2 [^50]模型，CroCov2 [^73]中的Transformer解码器包含12个模块，使用默认的DPT头[^55]，可见度阈值$\tau_v$设为0.3，PnP-RANSAC迭代次数最多为400次，重投影阈值$\tau_r$设为4像素。在姿态精修实验中，我们采用了5次精修迭代，这与以往的研究[^35] [^44]一致。尽管我们的基于流的重修器可以仅依赖在初始化阶段预渲染的模板（在每次精修迭代中，我们可以使用与当前姿态估计的朝向最匹配的模板），但我们发现这种方法的效果不如在线渲染——在线渲染会在当前姿态估计下渲染3D模型，因此不会受到姿态量化误差的影响。以往的研究[^35] [^44]也采用了在线渲染技术。

---

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td rowspan="2">#</td><td rowspan="2">Method</td><td colspan="3">LM-O</td><td colspan="2">T-LESS</td><td colspan="2">TUD-L</td><td colspan="2">IC-BIN</td><td colspan="2">ITODD</td><td colspan="2">HB</td><td colspan="2">YCB-V</td><td colspan="2">Average $\uparrow$</td><td rowspan="2">Time $\downarrow$</td><td rowspan="2">Nets $\downarrow$</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>AR_{MSPD}</td><td style='text-align: center; word-wrap: break-word;'>AR</td><td style='text-align: center; word-wrap: break-word;'>AR_{MSPD}</td><td style='text-align: center; word-wrap: break-word;'>AR</td><td style='text-align: center; word-wrap: break-word;'>AR_{MSPD}</td><td style='text-align: center; word-wrap: break-word;'>AR</td><td style='text-align: center; word-wrap: break-word;'>AR_{MSPD}</td><td style='text-align: center; word-wrap: break-word;'>AR</td><td style='text-align: center; word-wrap: break-word;'>AR_{MSPD}</td><td style='text-align: center; word-wrap: break-word;'>AR</td><td style='text-align: center; word-wrap: break-word;'>AR_{MSPD}</td><td style='text-align: center; word-wrap: break-word;'>AR</td><td style='text-align: center; word-wrap: break-word;'>AR_{MSPD}</td><td style='text-align: center; word-wrap: break-word;'>AR</td><td style='text-align: center; word-wrap: break-word;'>AR_{MSPD}</td><td style='text-align: center; word-wrap: break-word;'>AR</td><td style='text-align: center; word-wrap: break-word;'>AR_{MSPD}</td></tr><tr><td colspan="21">Coarse pose estimation:</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>1 FoundPose [^51]</td><td style='text-align: center; word-wrap: break-word;'>60.7</td><td style='text-align: center; word-wrap: break-word;'>39.6</td><td style='text-align: center; word-wrap: break-word;'>48.9</td><td style='text-align: center; word-wrap: break-word;'>33.8</td><td style='text-align: center; word-wrap: break-word;'>67.0</td><td style='text-align: center; word-wrap: break-word;'>46.7</td><td style='text-align: center; word-wrap: break-word;'>33.9</td><td style='text-align: center; word-wrap: break-word;'>23.9</td><td style='text-align: center; word-wrap: break-word;'>37.0</td><td style='text-align: center; word-wrap: break-word;'>20.4</td><td style='text-align: center; word-wrap: break-word;'>65.0</td><td style='text-align: center; word-wrap: break-word;'>50.8</td><td style='text-align: center; word-wrap: break-word;'>59.6</td><td style='text-align: center; word-wrap: break-word;'>45.2</td><td style='text-align: center; word-wrap: break-word;'>52.2</td><td style='text-align: center; word-wrap: break-word;'>37.2</td><td style='text-align: center; word-wrap: break-word;'>1.7 s</td><td style='text-align: center; word-wrap: break-word;'>1</td><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'></td></tr><tr><td style='text-align: center; word-wrap: break-word;'>2 GigaPose [^48]</td><td style='text-align: center; word-wrap: break-word;'>51.0</td><td style='text-align: center; word-wrap: break-word;'>29.6</td><td style='text-align: center; word-wrap: break-word;'>47.9</td><td style='text-align: center; word-wrap: break-word;'>26.4</td><td style='text-align: center; word-wrap: break-word;'>51.7</td><td style='text-align: center; word-wrap: break-word;'>30.0</td><td style='text-align: center; word-wrap: break-word;'>34.8</td><td style='text-align: center; word-wrap: break-word;'>22.3</td><td style='text-align: center; word-wrap: break-word;'>31.4</td><td style='text-align: center; word-wrap: break-word;'>17.5</td><td style='text-align: center; word-wrap: break-word;'>52.5</td><td style='text-align: center; word-wrap: break-word;'>34.1</td><td style='text-align: center; word-wrap: break-word;'>52.5</td><td style='text-align: center; word-wrap: break-word;'>27.8</td><td style='text-align: center; word-wrap: break-word;'>45.4</td><td style='text-align: center; word-wrap: break-word;'>26.8</td><td style='text-align: center; word-wrap: break-word;'>0.4 s</td><td style='text-align: center; word-wrap: break-word;'>1</td><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'></td></tr><tr><td style='text-align: center; word-wrap: break-word;'>3 BoW retrieval $^{{*}}$</td><td style='text-align: center; word-wrap: break-word;'>25.9</td><td style='text-align: center; word-wrap: break-word;'>11.6</td><td style='text-align: center; word-wrap: break-word;'>28.8</td><td style='text-align: center; word-wrap: break-word;'>15.4</td><td style='text-align: center; word-wrap: break-word;'>25.3</td><td style='text-align: center; word-wrap: break-word;'>13.7</td><td style='text-align: center; word-wrap: break-word;'>17.4</td><td style='text-align: center; word-wrap: break-word;'>10.1</td><td style='text-align: center; word-wrap: break-word;'>31.6</td><td style='text-align: center; word-wrap: break-word;'>17.4</td><td style='text-align: center; word-wrap: break-word;'>55.1</td><td style='text-align: center; word-wrap: break-word;'>42.4</td><td style='text-align: center; word-wrap: break-word;'>15.2</td><td style='text-align: center; word-wrap: break-word;'>11.6</td><td style='text-align: center; word-wrap: break-word;'>28.5</td><td style='text-align: center; word-wrap: break-word;'>17.5</td><td style='text-align: center; word-wrap: break-word;'>0.3 s</td><td style='text-align: center; word-wrap: break-word;'>1</td><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'></td></tr><tr><td colspan="21">Refinement using a single hypothesis from FoundPose [^51]:</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>4 GoTrack</td><td style='text-align: center; word-wrap: break-word;'>70.4</td><td style='text-align: center; word-wrap: break-word;'>56.5</td><td style='text-align: center; word-wrap: break-word;'>57.3</td><td style='text-align: center; word-wrap: break-word;'>50.4</td><td style='text-align: center; word-wrap: break-word;'>81.5</td><td style='text-align: center; word-wrap: break-word;'>67.2</td><td style='text-align: center; word-wrap: break-word;'>50.7</td><td style='text-align: center; word-wrap: break-word;'>43.2</td><td style='text-align: center; word-wrap: break-word;'>52.8</td><td style='text-align: center; word-wrap: break-word;'>39.3</td><td style='text-align: center; word-wrap: break-word;'>76.3</td><td style='text-align: center; word-wrap: break-word;'>72.2</td><td style='text-align: center; word-wrap: break-word;'>75.8</td><td style='text-align: center; word-wrap: break-word;'>63.1</td><td style='text-align: center; word-wrap: break-word;'>66.4</td><td style='text-align: center; word-wrap: break-word;'>56.0</td><td style='text-align: center; word-wrap: break-word;'>2.5 s</td><td style='text-align: center; word-wrap: break-word;'>1</td><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'></td></tr><tr><td style='text-align: center; word-wrap: break-word;'>5 MegaPose [^35]</td><td style='text-align: center; word-wrap: break-word;'>67.7</td><td style='text-align: center; word-wrap: break-word;'>55.4</td><td style='text-align: center; word-wrap: break-word;'>55.6</td><td style='text-align: center; word-wrap: break-word;'>51.0</td><td style='text-align: center; word-wrap: break-word;'>78.4</td><td style='text-align: center; word-wrap: break-word;'>63.3</td><td style='text-align: center; word-wrap: break-word;'>47.4</td><td style='text-align: center; word-wrap: break-word;'>43.0</td><td style='text-align: center; word-wrap: break-word;'>45.2</td><td style='text-align: center; word-wrap: break-word;'>34.6</td><td style='text-align: center; word-wrap: break-word;'>73.6</td><td style='text-align: center; word-wrap: break-word;'>69.5</td><td style='text-align: center; word-wrap: break-word;'>76.0</td><td style='text-align: center; word-wrap: break-word;'>66.1</td><td style='text-align: center; word-wrap: break-word;'>63.4</td><td style='text-align: center; word-wrap: break-word;'>54.7</td><td style='text-align: center; word-wrap: break-word;'>4.4 s</td><td style='text-align: center; word-wrap: break-word;'>2</td><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'></td></tr><tr><td colspan="21">Refinement using a single hypothesis from GigaPose [^48]:</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>6 GoTrack</td><td style='text-align: center; word-wrap: break-word;'>72.6</td><td style='text-align: center; word-wrap: break-word;'>58.4</td><td style='text-align: center; word-wrap: break-word;'>61.9</td><td style='text-align: center; word-wrap: break-word;'>54.9</td><td style='text-align: center; word-wrap: break-word;'>73.9</td><td style='text-align: center; word-wrap: break-word;'>61.3</td><td style='text-align: center; word-wrap: break-word;'>52.5</td><td style='text-align: center; word-wrap: break-word;'>45.3</td><td style='text-align: center; word-wrap: break-word;'>50.0</td><td style='text-align: center; word-wrap: break-word;'>39.5</td><td style='text-align: center; word-wrap: break-word;'>77.0</td><td style='text-align: center; word-wrap: break-word;'>72.9</td><td style='text-align: center; word-wrap: break-word;'>73.8</td><td style='text-align: center; word-wrap: break-word;'>62.0</td><td style='text-align: center; word-wrap: break-word;'>66.0</td><td style='text-align: center; word-wrap: break-word;'>56.3</td><td style='text-align: center; word-wrap: break-word;'>1.3 s</td><td style='text-align: center; word-wrap: break-word;'>2</td><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'></td></tr><tr><td style='text-align: center; word-wrap: break-word;'>7 GenFlow [^44]</td><td style='text-align: center; word-wrap: break-word;'>71.1</td><td style='text-align: center; word-wrap: break-word;'>59.5</td><td style='text-align: center; word-wrap: break-word;'>60.1</td><td style='text-align: center; word-wrap: break-word;'>55.0</td><td style='text-align: center; word-wrap: break-word;'>72.4</td><td style='text-align: center; word-wrap: break-word;'>60.7</td><td style='text-align: center; word-wrap: break-word;'>52.1</td><td style='text-align: center; word-wrap: break-word;'>47.8</td><td style='text-align: center; word-wrap: break-word;'>49.5</td><td style='text-align: center; word-wrap: break-word;'>41.3</td><td style='text-align: center; word-wrap: break-word;'>74.7</td><td style='text-align: center; word-wrap: break-word;'>72.2</td><td style='text-align: center; word-wrap: break-word;'>73.2</td><td style='text-align: center; word-wrap: break-word;'>60.8</td><td style='text-align: center; word-wrap: break-word;'>64.7</td><td style='text-align: center; word-wrap: break-word;'>56.8</td><td style='text-align: center; word-wrap: break-word;'>2.2 s</td><td style='text-align: center; word-wrap: break-word;'>2</td><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'></td></tr><tr><td style='text-align: center; word-wrap: break-word;'>8 MegaPose [^35]</td><td style='text-align: center; word-wrap: break-word;'>68.0</td><td style='text-align: center; word-wrap: break-word;'>55.7</td><td style='text-align: center; word-wrap: break-word;'>59.2</td><td style='text-align: center; word-wrap: break-word;'>54.1</td><td style='text-align: center; word-wrap: break-word;'>71.9</td><td style='text-align: center; word-wrap: break-word;'>58.0</td><td style='text-align: center; word-wrap: break-word;'>49.4</td><td style='text-align: center; word-wrap: break-word;'>45.0</td><td style='text-align: center; word-wrap: break-word;'>45.4</td><td style='text-align: center; word-wrap: break-word;'>37.6</td><td style='text-align: center; word-wrap: break-word;'>73.1</td><td style='text-align: center; word-wrap: break-word;'>69.3</td><td style='text-align: center; word-wrap: break-word;'>73.4</td><td style='text-align: center; word-wrap: break-word;'>63.2</td><td style='text-align: center; word-wrap: break-word;'>64.2</td><td style='text-align: center; word-wrap: break-word;'>54.7</td><td style='text-align: center; word-wrap: break-word;'>2.3 s</td><td style='text-align: center; word-wrap: break-word;'>2</td><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'></td></tr><tr><td colspan="21">Refinement using BoW retrieval for coarse pose estimation (from row 3):</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>9 GoTrack</td><td style='text-align: center; word-wrap: break-word;'>70.4</td><td style='text-align: center; word-wrap: break-word;'>56.9</td><td style='text-align: center; word-wrap: break-word;'>56.9</td><td style='text-align: center; word-wrap: break-word;'>50.4</td><td style='text-align: center; word-wrap: break-word;'>79.2</td><td style='text-align: center; word-wrap: break-word;'>65.2</td><td style='text-align: center; word-wrap: break-word;'>49.7</td><td style='text-align: center; word-wrap: break-word;'>42.4</td><td style='text-align: center; word-wrap: break-word;'>52.6</td><td style='text-align: center; word-wrap: break-word;'>40.2</td><td style='text-align: center; word-wrap: break-word;'>72.5</td><td style='text-align: center; word-wrap: break-word;'>69.5</td><td style='text-align: center; word-wrap: break-word;'>75.2</td><td style='text-align: center; word-wrap: break-word;'>62.9</td><td style='text-align: center; word-wrap: break-word;'>65.2</td><td style='text-align: center; word-wrap: break-word;'>55.4</td><td style='text-align: center; word-wrap: break-word;'>0.8 s</td><td style='text-align: center; word-wrap: break-word;'>1</td><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'></td></tr><tr><td colspan="21">Pose refinement using 5 hypotheses from GigaPose [^48]:</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>12 GoTrack</td><td style='text-align: center; word-wrap: break-word;'>78.1</td><td style='text-align: center; word-wrap: break-word;'>62.7</td><td style='text-align: center; word-wrap: break-word;'>66.3</td><td style='text-align: center; word-wrap: break-word;'>59.5</td><td style='text-align: center; word-wrap: break-word;'>80.8</td><td style='text-align: center; word-wrap: break-word;'>66.8</td><td style='text-align: center; word-wrap: break-word;'>54.4</td><td style='text-align: center; word-wrap: break-word;'>46.6</td><td style='text-align: center; word-wrap: break-word;'>56.3</td><td style='text-align: center; word-wrap: break-word;'>43.0</td><td style='text-align: center; word-wrap: break-word;'>81.9</td><td style='text-align: center; word-wrap: break-word;'>77.1</td><td style='text-align: center; word-wrap: break-word;'>80.6</td><td style='text-align: center; word-wrap: break-word;'>68.4</td><td style='text-align: center; word-wrap: break-word;'>71.2</td><td style='text-align: center; word-wrap: break-word;'>60.6</td><td style='text-align: center; word-wrap: break-word;'>3.0 s</td><td style='text-align: center; word-wrap: break-word;'>2</td><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'></td></tr><tr><td style='text-align: center; word-wrap: break-word;'>13 GenFlow [^44]</td><td style='text-align: center; word-wrap: break-word;'>75.3</td><td style='text-align: center; word-wrap: break-word;'>63.1</td><td style='text-align: center; word-wrap: break-word;'>63.7</td><td style='text-align: center; word-wrap: break-word;'>58.2</td><td style='text-align: center; word-wrap: break-word;'>79.4</td><td style='text-align: center; word-wrap: break-word;'>66.4</td><td style='text-align: center; word-wrap: break-word;'>55.1</td><td style='text-align: center; word-wrap: break-word;'>49.8</td><td style='text-align: center; word-wrap: break-word;'>55.1</td><td style='text-align: center; word-wrap: break-word;'>45.3</td><td style='text-align: center; word-wrap: break-word;'>78.3</td><td style='text-align: center; word-wrap: break-word;'>75.6</td><td style='text-align: center; word-wrap: break-word;'>78.8</td><td style='text-align: center; word-wrap: break-word;'>65.2</td><td style='text-align: center; word-wrap: break-word;'>69.4</td><td style='text-align: center; word-wrap: break-word;'>60.5</td><td style='text-align: center; word-wrap: break-word;'>10.6 s</td><td style='text-align: center; word-wrap: break-word;'>3</td><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'></td></tr><tr><td style='text-align: center; word-wrap: break-word;'>14 MegaPose [^35]</td><td style='text-align: center; word-wrap: break-word;'>73.6</td><td style='text-align: center; word-wrap: break-word;'>59.8</td><td style='text-align: center; word-wrap: break-word;'>62.0</td><td style='text-align: center; word-wrap: break-word;'>56.5</td><td style='text-align: center; word-wrap: break-word;'>77.9</td><td style='text-align: center; word-wrap: break-word;'>63.1</td><td style='text-align: center; word-wrap: break-word;'>52.8</td><td style='text-align: center; word-wrap: break-word;'>47.3</td><td style='text-align: center; word-wrap: break-word;'>48.5</td><td style='text-align: center; word-wrap: break-word;'>39.7</td><td style='text-align: center; word-wrap: break-word;'>76.5</td><td style='text-align: center; word-wrap: break-word;'>72.2</td><td style='text-align: center; word-wrap: break-word;'>77.7</td><td style='text-align: center; word-wrap: break-word;'>66.1</td><td style='text-align: center; word-wrap: break-word;'>65.3</td><td style='text-align: center; word-wrap: break-word;'>57.8</td><td style='text-align: center; word-wrap: break-word;'>7.7 s</td><td style='text-align: center; word-wrap: break-word;'>3</td><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'></td></tr></table>

<div style="text-align: center;">Table 1. Pose refinement performance on the seven core BOP datasets [^28]. Reported are accuracy scores, the average time required to estimate poses of all objects in an image (in seconds), and the number of different neural networks required by the pipelines.</div>

在物体跟踪实验中，触发模板与帧配准的参数$\tau_i$被设为0.8，混合比率r被设为2。我们采用Adam优化器[^35]，在MegaPose-GSO数据集[^33]上训练基于流的重修器直至收敛，同时冻结DINOv2模型的权重。

#### 4.1. 6自由度物体姿态精修

评估数据集。我们在七个核心BOP数据集[^28]上评估我们的方法：LM-O [^5]、T-LESS [^22]、TUD-L [^26]、IC-BIN [^10]、ITODD [^12]、[^31]以及YCB-V [^80]。这些数据集共包含132种不同的物体和19048个标注的测试物体实例。

评估指标。我们采用BOP评估协议[^27]，该协议基于三个指标：可见表面差异（VSD）、最大对称感知表面距离（MSSD）以及最大对称感知投影距离（MSPD）。最终的平均召回率（AR）是通过对这三个指标在一系列误差阈值下的单个AR分数取平均值得到的。

基线对比。我们将我们的方法与MegaPose [^35]和GenFlow [^44]进行比较，这两种方法目前是BOP基准测试中的最先进方法。MegaPose和GenFlow的结果源自BOP排行榜 [^28]。

实验结果。当以FoundPose [^51]估算的粗略姿态为起点时，我们的优化器在大多数BOP数据集上的表现优于MegaPose优化器 [^35]，平均AR值分别提升了+3.0 $_{MSPD}$和+1.3（见表1的第4行和第5行）。当以GigaPose [^48]估算的粗略姿态为起点时，我们的优化器在性能上同样能超越MegaPose（见第6行和第8行）。与GenFlow [^44]相比，我们的优化器可使AR值提升+1.3 $_{MSPD}$，而旋转误差则减少0.5（见第6行和第7行）。

从模型架构的复杂度来看，所评估的方法中最简单的是第3.2节提出的流程：它先利用基于BoW的模板检索来估算粗略姿态，随后再应用我们的优化器。该流程仅依赖一个神经网络（“Nets”列），且两个阶段共享这一网络，因此运行速度更快，也更容易部署到移动设备上。由于各方法的评估所用GPU不同（我们使用的是V100，而MegaPose在RTX 2080上评估，GenFlow则在RTX 3090上评估），直接比较它们的运行时间存在困难。不过各方法的复杂度差异显而易见：GenFlow和MegaPose的流程在粗略姿态估算和优化阶段分别采用不同的网络。在多假设场景下——即每个物体实例需考虑多个粗略姿态，且最终需对优化后的姿态进行排序以确定输出结果——GenFlow和MegaPose流程还需要额外的评分网络；而我们的方法只需通过计算第3.1节定义的加权内点比，就能得到可靠的姿态置信度分数。

#### 4.2.6 自由度物体姿态跟踪

评估数据集。我们在两个常用数据集上评估我们的跟踪方法：YCB-V [^79]和RBOT [^68]。关于最新HOT3D [^3]数据集的额外跟踪结果，详见附录。

评估指标。我们采用了与[^8]、[^62]、[^67]、[^72]、[^79]中相同的标准评估指标：曲线下面积

---

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'></td><td colspan="2">PoseCNN [^79]</td><td colspan="2">PoseRBFP [^8]</td><td colspan="2">GoTrack</td><td colspan="2">GoTrack+f2f</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Initialization</td><td colspan="2">-</td><td colspan="2">PoseCNN</td><td colspan="2">FoundPose</td><td colspan="2">FoundPose</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Unseen objects</td><td colspan="2">✗</td><td colspan="2">✗</td><td colspan="2">✓</td><td colspan="2">✓</td></tr><tr><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'>ADD</td><td style='text-align: center; word-wrap: break-word;'>ADD-S</td><td style='text-align: center; word-wrap: break-word;'>ADD</td><td style='text-align: center; word-wrap: break-word;'>ADD-S</td><td style='text-align: center; word-wrap: break-word;'>ADD</td><td style='text-align: center; word-wrap: break-word;'>ADD-S</td><td style='text-align: center; word-wrap: break-word;'>ADD</td><td style='text-align: center; word-wrap: break-word;'>ADD-S</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>002_master_chef_can</td><td style='text-align: center; word-wrap: break-word;'>50.9</td><td style='text-align: center; word-wrap: break-word;'>84.0</td><td style='text-align: center; word-wrap: break-word;'>58.0</td><td style='text-align: center; word-wrap: break-word;'>77.1</td><td style='text-align: center; word-wrap: break-word;'>71.0</td><td style='text-align: center; word-wrap: break-word;'>86.3</td><td style='text-align: center; word-wrap: break-word;'>71.3</td><td style='text-align: center; word-wrap: break-word;'>86.3</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>003_cracker_box</td><td style='text-align: center; word-wrap: break-word;'>51.7</td><td style='text-align: center; word-wrap: break-word;'>76.9</td><td style='text-align: center; word-wrap: break-word;'>76.8</td><td style='text-align: center; word-wrap: break-word;'>87.0</td><td style='text-align: center; word-wrap: break-word;'>79.6</td><td style='text-align: center; word-wrap: break-word;'>89.2</td><td style='text-align: center; word-wrap: break-word;'>80.6</td><td style='text-align: center; word-wrap: break-word;'>89.8</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>004_sugar_box</td><td style='text-align: center; word-wrap: break-word;'>68.6</td><td style='text-align: center; word-wrap: break-word;'>84.3</td><td style='text-align: center; word-wrap: break-word;'>75.9</td><td style='text-align: center; word-wrap: break-word;'>87.6</td><td style='text-align: center; word-wrap: break-word;'>80.4</td><td style='text-align: center; word-wrap: break-word;'>89.9</td><td style='text-align: center; word-wrap: break-word;'>80.5</td><td style='text-align: center; word-wrap: break-word;'>90.0</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>005_tomato_soup_can</td><td style='text-align: center; word-wrap: break-word;'>66.0</td><td style='text-align: center; word-wrap: break-word;'>80.9</td><td style='text-align: center; word-wrap: break-word;'>74.9</td><td style='text-align: center; word-wrap: break-word;'>84.5</td><td style='text-align: center; word-wrap: break-word;'>74.0</td><td style='text-align: center; word-wrap: break-word;'>87.2</td><td style='text-align: center; word-wrap: break-word;'>71.2</td><td style='text-align: center; word-wrap: break-word;'>85.8</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>006_mustard_bottle</td><td style='text-align: center; word-wrap: break-word;'>79.9</td><td style='text-align: center; word-wrap: break-word;'>90.2</td><td style='text-align: center; word-wrap: break-word;'>82.5</td><td style='text-align: center; word-wrap: break-word;'>91.0</td><td style='text-align: center; word-wrap: break-word;'>83.7</td><td style='text-align: center; word-wrap: break-word;'>91.9</td><td style='text-align: center; word-wrap: break-word;'>84.4</td><td style='text-align: center; word-wrap: break-word;'>92.2</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>007_tuna_fish_can</td><td style='text-align: center; word-wrap: break-word;'>70.4</td><td style='text-align: center; word-wrap: break-word;'>87.9</td><td style='text-align: center; word-wrap: break-word;'>59.0</td><td style='text-align: center; word-wrap: break-word;'>79.0</td><td style='text-align: center; word-wrap: break-word;'>81.4</td><td style='text-align: center; word-wrap: break-word;'>91.7</td><td style='text-align: center; word-wrap: break-word;'>82.0</td><td style='text-align: center; word-wrap: break-word;'>91.9</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>008_pudding_box</td><td style='text-align: center; word-wrap: break-word;'>62.9</td><td style='text-align: center; word-wrap: break-word;'>79.0</td><td style='text-align: center; word-wrap: break-word;'>57.2</td><td style='text-align: center; word-wrap: break-word;'>72.1</td><td style='text-align: center; word-wrap: break-word;'>82.8</td><td style='text-align: center; word-wrap: break-word;'>90.4</td><td style='text-align: center; word-wrap: break-word;'>83.5</td><td style='text-align: center; word-wrap: break-word;'>90.7</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>009_gelatin_box</td><td style='text-align: center; word-wrap: break-word;'>75.2</td><td style='text-align: center; word-wrap: break-word;'>87.1</td><td style='text-align: center; word-wrap: break-word;'>88.8</td><td style='text-align: center; word-wrap: break-word;'>93.1</td><td style='text-align: center; word-wrap: break-word;'>83.4</td><td style='text-align: center; word-wrap: break-word;'>90.9</td><td style='text-align: center; word-wrap: break-word;'>84.4</td><td style='text-align: center; word-wrap: break-word;'>91.5</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>010_potted_meat_can</td><td style='text-align: center; word-wrap: break-word;'>59.6</td><td style='text-align: center; word-wrap: break-word;'>78.5</td><td style='text-align: center; word-wrap: break-word;'>49.3</td><td style='text-align: center; word-wrap: break-word;'>62.0</td><td style='text-align: center; word-wrap: break-word;'>80.5</td><td style='text-align: center; word-wrap: break-word;'>91.3</td><td style='text-align: center; word-wrap: break-word;'>83.5</td><td style='text-align: center; word-wrap: break-word;'>92.8</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>011_banana</td><td style='text-align: center; word-wrap: break-word;'>72.3</td><td style='text-align: center; word-wrap: break-word;'>85.9</td><td style='text-align: center; word-wrap: break-word;'>24.8</td><td style='text-align: center; word-wrap: break-word;'>61.5</td><td style='text-align: center; word-wrap: break-word;'>53.8</td><td style='text-align: center; word-wrap: break-word;'>71.0</td><td style='text-align: center; word-wrap: break-word;'>56.0</td><td style='text-align: center; word-wrap: break-word;'>68.7</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>019_pitcher_base</td><td style='text-align: center; word-wrap: break-word;'>52.5</td><td style='text-align: center; word-wrap: break-word;'>76.8</td><td style='text-align: center; word-wrap: break-word;'>75.3</td><td style='text-align: center; word-wrap: break-word;'>88.4</td><td style='text-align: center; word-wrap: break-word;'>79.7</td><td style='text-align: center; word-wrap: break-word;'>90.0</td><td style='text-align: center; word-wrap: break-word;'>79.9</td><td style='text-align: center; word-wrap: break-word;'>90.1</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>021_bleach_cleanser</td><td style='text-align: center; word-wrap: break-word;'>50.5</td><td style='text-align: center; word-wrap: break-word;'>71.9</td><td style='text-align: center; word-wrap: break-word;'>54.5</td><td style='text-align: center; word-wrap: break-word;'>69.3</td><td style='text-align: center; word-wrap: break-word;'>68.9</td><td style='text-align: center; word-wrap: break-word;'>83.1</td><td style='text-align: center; word-wrap: break-word;'>69.3</td><td style='text-align: center; word-wrap: break-word;'>83.3</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>024_bowl</td><td style='text-align: center; word-wrap: break-word;'>6.5</td><td style='text-align: center; word-wrap: break-word;'>69.7</td><td style='text-align: center; word-wrap: break-word;'>36.1</td><td style='text-align: center; word-wrap: break-word;'>86.0</td><td style='text-align: center; word-wrap: break-word;'>35.1</td><td style='text-align: center; word-wrap: break-word;'>66.2</td><td style='text-align: center; word-wrap: break-word;'>34.8</td><td style='text-align: center; word-wrap: break-word;'>62.0</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>025_mug</td><td style='text-align: center; word-wrap: break-word;'>57.7</td><td style='text-align: center; word-wrap: break-word;'>78.0</td><td style='text-align: center; word-wrap: break-word;'>70.9</td><td style='text-align: center; word-wrap: break-word;'>85.4</td><td style='text-align: center; word-wrap: break-word;'>70.1</td><td style='text-align: center; word-wrap: break-word;'>87.1</td><td style='text-align: center; word-wrap: break-word;'>50.8</td><td style='text-align: center; word-wrap: break-word;'>87.4</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>035_power_drill</td><td style='text-align: center; word-wrap: break-word;'>55.1</td><td style='text-align: center; word-wrap: break-word;'>72.8</td><td style='text-align: center; word-wrap: break-word;'>70.9</td><td style='text-align: center; word-wrap: break-word;'>85.0</td><td style='text-align: center; word-wrap: break-word;'>81.9</td><td style='text-align: center; word-wrap: break-word;'>91.2</td><td style='text-align: center; word-wrap: break-word;'>82.1</td><td style='text-align: center; word-wrap: break-word;'>91.3</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>036_wood_block</td><td style='text-align: center; word-wrap: break-word;'>31.8</td><td style='text-align: center; word-wrap: break-word;'>65.8</td><td style='text-align: center; word-wrap: break-word;'>2.8</td><td style='text-align: center; word-wrap: break-word;'>33.3</td><td style='text-align: center; word-wrap: break-word;'>0.7</td><td style='text-align: center; word-wrap: break-word;'>30.4</td><td style='text-align: center; word-wrap: break-word;'>5.3</td><td style='text-align: center; word-wrap: break-word;'>40.9</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>037_scissors</td><td style='text-align: center; word-wrap: break-word;'>35.8</td><td style='text-align: center; word-wrap: break-word;'>56.2</td><td style='text-align: center; word-wrap: break-word;'>21.7</td><td style='text-align: center; word-wrap: break-word;'>33.0</td><td style='text-align: center; word-wrap: break-word;'>58.3</td><td style='text-align: center; word-wrap: break-word;'>76.5</td><td style='text-align: center; word-wrap: break-word;'>59.7</td><td style='text-align: center; word-wrap: break-word;'>77.8</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>040_large_marker</td><td style='text-align: center; word-wrap: break-word;'>58.0</td><td style='text-align: center; word-wrap: break-word;'>71.4</td><td style='text-align: center; word-wrap: break-word;'>48.7</td><td style='text-align: center; word-wrap: break-word;'>59.3</td><td style='text-align: center; word-wrap: break-word;'>60.0</td><td style='text-align: center; word-wrap: break-word;'>67.3</td><td style='text-align: center; word-wrap: break-word;'>65.0</td><td style='text-align: center; word-wrap: break-word;'>73.9</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>051_large_clamp</td><td style='text-align: center; word-wrap: break-word;'>25.0</td><td style='text-align: center; word-wrap: break-word;'>49.9</td><td style='text-align: center; word-wrap: break-word;'>47.3</td><td style='text-align: center; word-wrap: break-word;'>76.9</td><td style='text-align: center; word-wrap: break-word;'>64.1</td><td style='text-align: center; word-wrap: break-word;'>83.1</td><td style='text-align: center; word-wrap: break-word;'>64.4</td><td style='text-align: center; word-wrap: break-word;'>83.2</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>052_extra_large_clamp</td><td style='text-align: center; word-wrap: break-word;'>15.8</td><td style='text-align: center; word-wrap: break-word;'>47.0</td><td style='text-align: center; word-wrap: break-word;'>26.5</td><td style='text-align: center; word-wrap: break-word;'>69.5</td><td style='text-align: center; word-wrap: break-word;'>83.5</td><td style='text-align: center; word-wrap: break-word;'>93.5</td><td style='text-align: center; word-wrap: break-word;'>82.8</td><td style='text-align: center; word-wrap: break-word;'>93.0</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>061_foam_brick</td><td style='text-align: center; word-wrap: break-word;'>40.4</td><td style='text-align: center; word-wrap: break-word;'>87.8</td><td style='text-align: center; word-wrap: break-word;'>78.2</td><td style='text-align: center; word-wrap: break-word;'>89.7</td><td style='text-align: center; word-wrap: break-word;'>83.0</td><td style='text-align: center; word-wrap: break-word;'>92.3</td><td style='text-align: center; word-wrap: break-word;'>83.3</td><td style='text-align: center; word-wrap: break-word;'>92.4</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Average</td><td style='text-align: center; word-wrap: break-word;'>53.7</td><td style='text-align: center; word-wrap: break-word;'>75.9</td><td style='text-align: center; word-wrap: break-word;'>59.9</td><td style='text-align: center; word-wrap: break-word;'>77.5</td><td style='text-align: center; word-wrap: break-word;'>69.3</td><td style='text-align: center; word-wrap: break-word;'>82.9</td><td style='text-align: center; word-wrap: break-word;'>69.3</td><td style='text-align: center; word-wrap: break-word;'>83.6</td></tr></table>

<div style="text-align: center;">Table 2. Tracking performance on the YCB-V dataset [^79]. The accuracy is measured by AUC w.r.t. the ADD and ADD-S pose error functions. Despite not being trained on the YCB-V objects, our method outperforms PoseCNN [^79] and PoseRBPF [^8] which were trained on these specific objects. Adding the frame-to-frame consistency (f2f) makes our method 6X more efficient while slightly improving the accuracy (see text for details).</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td rowspan="2">Method</td><td colspan="6">with reset</td><td style='text-align: center; word-wrap: break-word;'>w/o reset</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Regular</td><td style='text-align: center; word-wrap: break-word;'>Dynamic</td><td style='text-align: center; word-wrap: break-word;'>Noisy</td><td style='text-align: center; word-wrap: break-word;'>Occlusion</td><td style='text-align: center; word-wrap: break-word;'>Mean</td><td style='text-align: center; word-wrap: break-word;'>Resets $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>Mean</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>SRT3D [^62]</td><td style='text-align: center; word-wrap: break-word;'>94.2</td><td style='text-align: center; word-wrap: break-word;'>94.6</td><td style='text-align: center; word-wrap: break-word;'>81.7</td><td style='text-align: center; word-wrap: break-word;'>93.2</td><td style='text-align: center; word-wrap: break-word;'>90.9</td><td style='text-align: center; word-wrap: break-word;'>6575</td><td style='text-align: center; word-wrap: break-word;'>19.0</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>LDT3D [^67]</td><td style='text-align: center; word-wrap: break-word;'>95.2</td><td style='text-align: center; word-wrap: break-word;'>95.4</td><td style='text-align: center; word-wrap: break-word;'>83.2</td><td style='text-align: center; word-wrap: break-word;'>94.9</td><td style='text-align: center; word-wrap: break-word;'>92.1</td><td style='text-align: center; word-wrap: break-word;'>6228</td><td style='text-align: center; word-wrap: break-word;'>18.8</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>DeepAC [^72] $^{{*}}$</td><td style='text-align: center; word-wrap: break-word;'>95.6</td><td style='text-align: center; word-wrap: break-word;'>95.6</td><td style='text-align: center; word-wrap: break-word;'>88.0</td><td style='text-align: center; word-wrap: break-word;'>94.0</td><td style='text-align: center; word-wrap: break-word;'>93.3</td><td style='text-align: center; word-wrap: break-word;'>4826</td><td style='text-align: center; word-wrap: break-word;'>30.3</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>GoTrack</td><td style='text-align: center; word-wrap: break-word;'>97.3</td><td style='text-align: center; word-wrap: break-word;'>96.2</td><td style='text-align: center; word-wrap: break-word;'>94.2</td><td style='text-align: center; word-wrap: break-word;'>95.4</td><td style='text-align: center; word-wrap: break-word;'>95.9</td><td style='text-align: center; word-wrap: break-word;'>3021</td><td style='text-align: center; word-wrap: break-word;'>86.7</td></tr></table>

<div style="text-align: center;">Table 3. Tracking on the RBOT dataset [^68] with/without reset as in [^62] [^67] [^72]. The accuracy is measured by  $5\,cm-5^{\circ}$ score and the number of reset. Character  $*$ indicates that DeepAC is a supervised method trained on the test objects. GoTrack significantly outperforms existing methods across all settings and metrics.</div>

（AUC），用于衡量YCB-V数据集上的ADD和ADD(-S)姿态误差函数，以及RBOT数据集上的$5\,cm-5^{\circ}$和重置次数。当跟踪方法失效——即平移误差超过5厘米或旋转误差超过5度时——便会计入一次重置，此时该方法会用真实姿态重新初始化。关于这些指标的更多细节详见附录。

基线对比。由于在YCB-V数据集上尚未有针对未见物体的纯RGB跟踪方法接受评估，因此我们将我们的方法与针对已见物体的跟踪方法（即在目标物体上训练的方法）进行比较，包括PoseCNN [^79]和PoseRBPF [^8]。PoseCNN是一种基于单张图像进行6自由度物体姿态估计的方法，评估时逐帧进行。PoseRBPF则在第一帧以PoseCNN估算的姿态作为初始值。由于PoseCNN的估算结果并未公开，我们便用FoundPose估算的姿态来初始化我们的跟踪方法。在RBOT数据集上，我们则将我们的方法与基于轮廓的方法DeepAC [^72]、SRT3D [^62]，以及基于优化的方法LDT3D [^67]进行比较。

YCB-V数据集上的实验结果 [^79]。如表2所示，尽管我们的方法并未在YCB-V物体上训练，但其性能仍显著优于专门针对这些物体训练的PoseCNN [^79]和PoseRBPF [^8]。加入帧间光流后，我们的方法效率提升了约6倍，同时精度也略有提高。我们用于模板到帧配准的基于光流的细化模块的前向传播耗时40毫秒，而用于帧间配准的RAFT-Small模型的前向传播仅需6毫秒。在YCB-V数据集的序列中，模板到帧的光流计算平均仅每50帧触发一次。加入帧间光流的另一个效果是减少了抖动现象，这一点可在补充视频（附录D）中看到。

RBOT数据集上的实验结果 [^68]。如表3所示，GoTrack在所有RBOT序列类型上的表现均优于LDT3D、SRT3D和DeepAC，且需要重新初始化的次数减少了37%到54%。最明显的差异体现在噪声较大的序列上，此时轮廓检测的可靠性较低。此外，这些方法仅在轮廓特征明显的物体上才能良好工作；如果相同的轮廓形状对应多种不同姿态，且只能根据纹理来估计实际姿态，它们便难以胜任（例如，在RBOT数据集中带有纹理的圆柱形汽水瓶上，DeepAC的5厘米-5°精度仅为70.1%，而GoTrack则达到了97.2%）。另外，表3显示，在无需重新初始化的情况下进行跟踪时，我们的方法性能显著优于其他方法（AUC值分别为86.7和30.3）。

### 5. 结论

我们提出了一种针对未见物体的6自由度姿态细化方法，该方法通过预测模板到帧的光流以及可见模板像素的掩码来实现。我们展示了如何将这种细化方法与基于词袋模型的模板检索无缝结合，从而构建出高效且精准的物体姿态估计流程。此外，我们还提出了一种通过利用帧间光流传播2D-3D对应关系来实现未见物体6自由度姿态跟踪的扩展方案。我们的方法在标准的6自由度物体姿态细化和跟踪基准测试中取得了业界领先的结果。在未来的工作中，我们计划从SLAM文献中汲取更多灵感，探索将该方法扩展到无模型场景的可能性。

---

## 参考文献

[^1]: Marc Alexa. Super-fibonacci spirals: Fast, low-discrepancy sampling of so (3). In CVPR, 2022. 5

[^2]: Philipp Ausserlechner, David Haberger, Stefan Thalhammer, Jean-Baptiste Weibel, and Markus Vincze. ZS6D: Zero-Shot 6D Object Pose Estimation Using Vision Transformers. 2023. 2

[^3]: Prithviraj Banerjee, Sindi Shkodrani, Pierre Moulon, Shreyas Hampali, Fan Zhang, Jade Fountain, Edward Miller, Selen Basol, Richard Newcombe, Robert Wang, et al. Introducing hot3d: An egocentric dataset for 3d hand and object tracking. arXiv preprint arXiv:2406.09598, 2024. 1, 7, 12, 13

[^4]: Selim Benhimane and Ezio Malis. Real-time image-based tracking of planes using efficient second-order minimization. In IEEE/RSJ International Conference on Intelligent Robots and Systems, 2004. 1, 3

[^5]: Eric Brachmann, Alexander Krull, Frank Michel, Stefan Gumhold, Jamie Shotton, and Carsten Rother. Learning 6D Object Pose Estimation Using 3D Object Coordinates. In ECCV, 2014. 2, 6, 7, 14

[^6]: Andrew I. Comport, Eric Marchand, Muriel Pressigout, and Francois Chaumette. Real-time markerless tracking for augmented reality: The virtual visual servoing framework. IEEE Transactions on Visualization and Computer Graphics, 2006. 1, 3

[^7]: Alberto Crivellaro and Vincent Lepetit. Robust 3D tracking with descriptor fields. In CVPR, 2014. 1, 3

[^8]: Xinke Deng, Arsalan Mousavian, Yu Xiang, Fei Xia, Timothy Bretl, and Dieter Fox. PoseRBPF: A Rao–Blackwellized particle filter for 6-D object pose tracking. IEEE Transactions on Robotics, 2021. 1, 3, 6, 7, 8, 12

[^9]: Maximilian Denninger, Martin Sundermeyer, Dominik Winkelbauer, Dmitry Olefir, Tomáš Hodaň, Youssef Zidan, Mohamad Elbadrawy, Markus Knauer, Harinandan Katam, and Ahsan Lodhi. BlenderProc: Reducing the reality gap with photorealistic rendering. RSS Workshops, 2020. 4

[^10]: Andreas Doumanoglou, Rigas Kouskouridas, Sotiris Malassiotis, and Tae-Kyun Kim. Recovering 6D Object Pose and Predicting Next-Best-View in the Crowd. In CVPR, 2016. 2, 7, 18

[^11]: Laura Downs, Anthony Francis, Nate Koenig, Brandon Kinman, Ryan Hickman, Krista Reymann, Thomas B McHugh, and Vincent Vanhoucke. Google scanned objects: A high-quality dataset of 3D scanned household items. ICRA, 2022.

[^12]: Bertram Drost, Markus Ulrich, Paul Bergmann, Philipp Hartinger, and Carsten Steger. Introducing MVTec ITODD – A dataset for 3D object recognition in industry. In ICCVW, 2017. 2, 7, 20

[^13]: Tom Drummond and Roberto Cipolla. Real-time visual tracking of complex structures. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2002. 1, 3

[^14]: Hugh Durrant-Whyte and Tim Bailey. Simultaneous localization and mapping: part i. IEEE robotics & automation magazine, 2006. 2, 5

[^15]: Jakob Engel, Kiran Somasundaram, Michael Goesele, Albert Sun, Alexander Gamino, Andrew Turner, Arjang Talatof, Arnie Yuan, Bilal Souti, Brighid Meredith, Cheng Peng, Chris Sweeney, Cole Wilson, Dan Barnes, Daniel DeTone, David Caruso, Derek Valleroy, Dinesh Ginjupalli, Duncan Frost, Edward Miller, Elias Mueggler, Evgeniy Oleinik, Fan Zhang, Guruprasad Somasundaram, Gustavo Solaira, Harry Lanaras, Henry Howard-Jenkins, Huixuan Tang, Hyo Jin Kim, Jaime Rivera, Ji Luo, Jing Dong, Julian Straub, Kevin Bailey, Kevin Eckenhoff, Lingni Ma, Luis Pesqueira, Mark Schwesinger, Maurizio Monge, Nan Yang, Nick Charron, Nikhil Raina, Omkar Parkhi, Peter Borschowa, Pierre Moulon, Prince Gupta, Raul Mur-Artal, Robbie Pennington, Sachin Kulkarni, Sagar Miglani, Santosh Gondi, Saransh Solanki, Sean Diener, Shangyi Cheng, Simon Green, Steve Saarinen, Suvam Patra, Tassos Mourikis, Thomas Whelan, Tripti Singh, Vasileios Balntas, Vijay Baiyya, Wilson Dreewes, Xiaqing Pan, Yang Lou, Yipu Zhao, Yusuf Mansour, Yuyang Zou, Zhaoyang Lv, Zijian Wang, Mingfei Yan, Carl Ren, Renzo De Nardi, and Richard Newcombe. Project Aria: A new tool for egocentric multi-modal AI research, 2023.1

[^16]: M. A. Fischler and R. C. Bolles. Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM, 1981. 2, 3, 4

[^17]: Mathieu Garon and Jean-François Lalonde. Deep 6-DOF tracking. IEEE transactions on visualization and computer graphics, 2017. 1, 3

[^18]: Mathieu Garon, Denis Laurendeau, and Jean-François Lalonde. A framework for evaluating 6-DoF object trackers. In ECCV, 2018. 3

[^19]: Chris Harris and Carl Stennett. RAPID - a video rate object tracker. In BMVC, 1990. 1,3

[^20]: Stefan Hinterstoißer, Vincent Lepetit, Slobodan Ilic, Stefan Holzer, Gary R. Bradski, Kurt Konolige, and Nassir Navab. Model Based Training, Detection and Pose Estimation of Texture-Less 3D Objects in Heavily Cluttered Scenes. In ACCV, 2012. 12

[^21]: Tomáš Hodaň, Dániel Baráth, and Jiří Matas. EPOS: Estimating 6D pose of objects with symmetries. CVPR, 2020. 1

[^22]: Tomas Hodan, Pavel Haluza, Stepan Obdrzalek, Jiri Matas, Manolis Lourakis, and Xenophon Zabulis. T-LESS: An RGB-D Dataset for 6D Pose Estimation of Texture-Less Objects. 2017. 2, 7, 15

[^23]: Tomáš Hodaň, Pavel Haluza, Štěpán Obdržálek, Jiří Matas, Manolis Lourakis, and Xenophon Zabulis. T-LESS: An RGB-D dataset for 6D pose estimation of texture-less objects. WACV, 2017. 6

[^24]: Tomáš Hodaň, Jiří Matas, and Štěpán Obdržálek. On evaluation of 6D object pose estimation. ECCVW, 2016. 12

## [^25]: Tomáš Hodaň, Frank Michel, Eric Brachmann, Wadim Kehl, Anders Glent Buch, Dirk Kraft, Bertram Drost, Joel Vidal, Stephan Ihrke, Xenophon Zabulis, Caner Sahin, Fabian Manhardt, Federico Tombari, Tae-Kyun Kim, Jiří Matas, and Carsten Rother. BOP: Benchmark for 6D object pose estimation. ECCV, 2018. 19

---

[^26]: Tomas Hodan, Frank Michel, Eric Brachmann, Wadim Kehl, Anders Glentbuch, Dirk Kraft, Bertram Drost, Joel Vidal, Stephan Ihrke, Xenophon Zabulis, and Others. Bop: Benchmark for 6D Object Pose Estimation. In ECCV, 2018. 2, 7

[^27]: Tomas Hodan, Martin Sundermeyer, Bertram Drost, Yann Labbé, Eric Brachmann, Frank Michel, Carsten Rother, and Jiri Matas. BOP Challenge 2020 on 6D Object Localization. In ECCV, 2020. 7

[^28]: Tomáš Hodaň, Martin Sundermeyer, Yann Labbé, Van Nguyen Nguyen, Gu Wang, Eric Brachmann, Bertram Drost, Vincent Lepetit, Carsten Rother, and Jiří Matas. BOP challenge 2023 on detection, segmentation and pose estimation of seen and unseen rigid objects. Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2024. 2, 3, 5, 6, 7, 12

[^29]: Yinlin Hu, Pascal Fua, Wei Wang, and Mathieu Salzmann. Single-Stage 6D Object Pose Estimation. In CVPR, 2020. 2

[^30]: Jan Issac, Manuel Wüthrich, Cristina Garcia Cifuentes, Jeannette Bohg, Sebastian Trimpe, and Stefan Schaal. Depth-based object tracking using a robust gaussian filter. In ICRA, 2016. 3

[^31]: Roman Kaskman, Sergey Zakharov, Ivan Shugurov, and Slobodan Ilic. Homebreweddb: RGB-D Dataset for 6D Pose Estimation of 3D Objects. In ICCV Workshops, 2019. 2, 7, 16

[^32]: Wadim Kehl, Fabian Manhardt, Federico Tombari, Slobodan Ilic, and Nassir Navab. SSD-6D: Making RGB-Based 3D Detection and 6D Pose Estimation Great Again. In ICCV, 2017. 1,2

[^33]: Diederik P Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014. 7

[^34]: Yann Labbé, Justin Carpentier, Mathieu Aubry, and Josef Sivic. CosyPose: Consistent multi-view multi-object 6D pose estimation. ECCV, 2020. 1, 2

[^35]: Yann Labbé, Lucas Manuelli, Arsalan Mousavian, Stephen Tyree, Stan Birchfield, Jonathan Tremblay, Justin Carpentier, Mathieu Aubry, Dieter Fox, and Josef Sivic. MegaPose: 6D Pose Estimation of Novel Objects via Render & Compare. In CoRL, 2022. 1, 2, 3, 4, 5, 6, 7

[^36]: Vincent Lepetit, Francesc Moreno-Noguer, and Pascal Fua. EPnP: An accurate O(n) solution to the PnP problem. IJCV, 2009. 2, 3, 4

[^37]: Yi Li, Gu Wang, Xiangyang Ji, Yu Xiang, and Dieter Fox. DeepIM: Deep iterative matching for 6d pose estimation. ECCV, 2018. 1, 2, 3

[^38]: Zhigang Li, Gu Wang, and Xiangyang Ji. CDPN: Coordinates-Based Disentangled Pose Network for Real-Time RGB-Based 6DoF Object Pose Estimation. In ICCV, 2019. 2

[^39]: Yunzhi Lin, Jonathan Tremblay, Stephen Tyree, Patricio A Vela, and Stan Birchfield. Keypoint-based category-level object pose tracking from an RGB sequence with uncertainty estimation. In ICRA, 2022. 3

[^40]: Xingyu Liu, Ruida Zhang, Chenyangguang Zhang, Bowen Fu, Jiwen Tang, Xiquan Liang, Jingyi Tang, Xiaotian Cheng, Yukang Zhang, Gu Wang, and Xiangyang Ji.

GDRNPP. <https://github.com/shanice-1/gdrnpp_bop2022>，2022.1,2

[^41]: Bruce D. Lucas and Takeo Kanade. An iterative image registration technique with an application to stereo vision. In Proceedings of the 7th International Joint Conference on Artificial Intelligence, 1981. 1, 3

[^42]: Mateusz Majcher and Bogdan Kwolek. 3d model-based 6d object pose tracking on rgb images using particle filtering and heuristic optimization. In VISIGRAPP (5: VISAPP), 2020. 1, 3

[^43]: Meta. Quest 3. <https://www.meta.com/quest/quest-3/,2023.1>

[^44]: Sungphill Moon, Hyeontae Son, Dongcheol Hur, and Sangwook Kim. GenFlow: Generalizable Recurrent Flow for 6D Pose Refinement of Novel Objects. In arXiv preprint arXiv:2403.11510, 2024. 1, 2, 3, 4, 6, 7

[^45]: Jorge J Moré. The levenberg-marquardt algorithm: implementation and theory. In Numerical analysis: proceedings of the biennial Conference held at Dundee, year=2006, organization=Springer. 4

[^46]: Van Nguyen Nguyen, Yuming Du, Yang Xiao, Michael Ramamonjisao, and Vincent Lepetit. PIZZA: A Powerful Image-Only Zero-Shot Zero-CAD Approach to 6 DoF Tracking. 2022. 3

[^47]: Van Nguyen Nguyen, Thibault Groueix, Georgy Ponimatkin, Vincent Lepetit, and Tomas Hodan. CNOS: A Strong Baseline for CAD-based Novel Object Segmentation. In ICCVW, 2023. 4,5

[^48]: Van Nguyen Nguyen, Thibault Groueix, Mathieu Salzmann, and Vincent Lepetit. Gigapose: Fast and robust novel object pose estimation via one correspondence. In CVPR, 2024. 1, 7

[^49]: Van Nguyen Nguyen, Yinlin Hu, Yang Xiao, Mathieu Salzmann, and Vincent Lepetit. Templates for 3D Object Pose Estimation Revisited: Generalization to New Objects and Robustness to Occlusions. In CVPR, 2022. 1

[^50]: Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023. 2, 4, 5, 6

[^51]: Evin Pınar Örnek, Yann Labbé, Bugra Tekin, Lingni Ma, Cem Keskin, Christian Forster, and Tomas Hodan. Found-pose: Unseen object pose estimation with foundation fea-tures. In ECCV, 2024. 1, 2, 3, 4, 5, 6, 7, 12, 13

[^52]: Mustafa Özuysal, Vincent Lepetit, François Fleuret, and Pascal Fua. Feature harvesting for tracking-by-detection. In Aleš Leonardis, Horst Bischof, and Axel Pinz, editors, ECCV, 2006. 1, 3

[^53]: Kiru Park, Timothy Patten, and Markus Vincze. Pix2pose: Pixel-Wise Coordinate Regression of Objects for 6D Pose Estimation. In ICCV, 2019. 2

### [^54]: Mahdi Rad and Vincent Lepetit. BB8: A Scalable, Accurate, Robust to Partial Occlusion Method for Predicting the 3D Poses of Challenging Objects Without Using Depth. In ICCV, 2017. 1,2

---

[^55]: René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. In ICCV, 2021. 4, 6

[^56]: Edward Rosten and Tom Drummond. Fusing points and lines for high performance tracking. In ICCV, 2005. 1,3

[^57]: Ethan Rublee, Vincent Rabaud, Kurt Konolige, and Gary Bradski. ORB: an efficient alternative to SIFT or SURF. In ICCV, 2011. 1, 3

[^58]: Byung-Kuk Seo, Hanhoon Park, Jong-Il Park, Stefan Hinterstoisser, and Slobodan Ilic. Optimal local searching for fast and robust textureless 3D object tracking in highly cluttered backgrounds. IEEE Transactions on Visualization and Computer Graphics, 2014. 1, 3

[^59]: Byung-Kuk Seo and Harald Wuest. A direct method for robust model-based 3D object tracking from a monocular RGB image. In ECCVW, 2016. 1, 3

[^60]: Dave Shreiner et al. OpenGL programming guide: the official guide to learning OpenGL, versions 3.0 and 3.1. Pearson Education, 2009. 5

[^61]: Iryna Skrypnyk and David G. Lowe. Scene modelling, recognition and tracking with invariant image features. In ISMAR, 2004. 1, 3

[^62]: Manuel Stoiber, Martin Pfanne, Klaus H Strobl, Rudolph Triebel, and Alin Albu-Schäffer. Srt3d: A sparse region-based 3d object tracking approach for the real world. International Journal of Computer Vision, 2022. 1, 3, 6, 7, 8, 12

[^63]: Manuel Stoiber, Martin Sundermeyer, and Rudolph Triebel. Iterative corresponding geometry: Fusing region and depth for highly efficient 3D tracking of textureless objects. In CVPR, 2022. 3

[^64]: Martin Sundermeyer, Tomáš Hodaň, Yann Labbe, Gu Wang, Eric Brachmann, Bertram Drost, Carsten Rother, and Jiří Matas. Bop Challenge 2022 on Detection, Segmentation and Pose Estimation of Specific Rigid Objects. In CVPR, 2023. 2

[^65]: Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In ECCV, 2020. 2, 3, 4, 5

[^66]: Bugra Tekin, Sudipta N. Sinha, and Pascal Fua. Real-Time Seamless Single Shot 6D Object Pose Prediction. In CVPR, 2018. 1,2

[^67]: Xuhui Tian, Xinran Lin, Fan Zhong, and Xueying Qin. Large-displacement 3d object tracking with hybrid non-local optimization. In ECCV, 2022. 1, 3, 6, 7, 8, 12

[^68]: Henning Tjaden, Ulrich Schwanecke, Elmar Schömer, and Daniel Cremers. A region-based gauss-newton approach to real-time monocular multiple object tracking. TPAMI, 2018. 2,6,7,8

[^69]: Luca Vacchetti, Vincent Lepetit, and Pascal Fua. Stable real-time 3D tracking using online and offline information. PAMI, 2004. 1, 3

[^70]: Chen Wang, Roberto Martín-Martín, Danfei Xu, Jun Lv, Cewu Lu, Li Fei-Fei, Silvio Savarese, and Yuke Zhu. 6-PACK: Category-level 6D pose tracker with anchor-based keypoints. In ICRA, 2020. 3

[^71]: Gu Wang, Fabian Manhardt, Federico Tombari, and Xiangyang Ji. GDR-Net: Geometry-Guided Direct Regressions

单目6D物体姿态估计的分割网络。收录于CVPR，2021年。1

[^72]: Long Wang, Shen Yan, Jianan Zhen, Yu Liu, Maojun Zhang, Guofeng Zhang, and Xiaowei Zhou. Deep active contours for real-time 6-DoF object tracking. In ICCV, 2023. 1, 3, 6, 7, 8, 12

[^73]: Philippe Weinzaepfel, Thomas Lucas, Vincent Leroy, Yohann Cabon, Vaibhav Arora, Romain Brégier, Gabriela Csurka, Leonid Antsfeld, Boris Chidlovskii, and Jérôme Revaud. Croco v2: Improved cross-view completion pretraining for stereo matching and optical flow. In ICCV, 2023. 4,6

[^74]: Bowen Wen and Kostas Bekris. Bundletrack: 6D pose tracking for novel objects without instance or category-level 3D models. In IROS, 2021. 3

[^75]: Bowen Wen, Chaitanya Mitash, Baozhang Ren, and Kostas E Bekris. se(3)-TrackNet: Data-driven 6D pose tracking by calibrating image residuals in synthetic domains. In IROS, 2020. 1,3

[^76]: Bowen Wen, Jonathan Tremblay, Valts Blukis, Stephen Tyree, Thomas Müller, Alex Evans, Dieter Fox, Jan Kautz, and Stan Birchfield. BundleSDF: Neural 6-DoF tracking and 3D reconstruction of unknown objects. In CVPR, 2023. 3

[^77]: Bowen Wen, Wei Yang, Jan Kautz, and Stan Birchfield. Foundationpose: Unified 6d pose estimation and tracking of novel objects. arXiv preprint arXiv:2312.08344, 2023. 1, 2, 3, 4, 5

[^78]: Manuel Wüthrich, Peter Pastor, Mrinal Kalakrishnan, Jeannette Bohg, and Stefan Schaal. Probabilistic object tracking using a range camera. In IROS, 2013. 3

[^79]: Yu Xiang, Tanner Schmidt, Venkatraman Narayanan, and Dieter Fox. PoseCNN: A convolutional neural network for 6D object pose estimation in cluttered scenes. RSS, 2018. 2, 6, 7, 8, 12, 13, 17

[^80]: Yu Xiang, Tanner Schmidt, Venkatraman Narayanan, and Dieter Fox. PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes. 2018. 2,7

### [^81]: Sergey Zakharov, Ivan S. Shugurov, and Slobodan Ilic. DPOD: 6D Pose Object Detector and Refiner. In ICCV, 2019. 1,2

---

### 附录

在本附录中，我们提供了物体跟踪实验中使用的评估指标详情（章节A）。我们还给出了HOT3D [^3]数据集上的定量跟踪结果（章节B），以及关于姿态细化和跟踪的额外定性结果（章节C和D）。

### A. 跟踪评估指标

正如主论文第4.2节所讨论的，我们采用了与以往研究相同标准评估指标[^8] [^62] [^67] [^72] [^79]：即在YCB-V数据集上针对ADD和ADD(-S)位姿误差函数的曲线下面积（AUC），以及$5cm-5^{\circ}$，还有在RBOT数据集上的重置次数。ADD误差[^20]用于衡量模型顶点在估计位姿下的平均偏差。ADD-S误差则衡量到最近模型点的平均距离[^20] [^24]，这一指标非常适合用于评估对称物体的位姿估计结果。AUC值是通过对0.1厘米到10厘米范围内的多个阈值对应的ADD和ADD-S召回率取平均值来计算的。$5cm-5^{\circ}$分数指的是跟踪位姿的成功率，其判定标准为平移误差小于5厘米且旋转误差小于5度。在RBOT数据集上，追踪器以真实位姿初始化后，会持续运行直至记录序列结束或跟踪失败。若跟踪失败，算法会用真实位姿重新初始化，我们会对重新初始化的总次数进行统计。

### B. 定量跟踪结果

表4展示了Aria和Quest 3设备采集的HOT3D[^3]录制数据上的实验结果。由于HOT3D数据集测试集的真实标注并未公开，我们选择在HOT3D的训练集上对GoTrack进行评估。尽管我们的方法并未在灰度图像上训练，但它在HOT3D-Quest3的灰度图像上取得的性能，与在HOT3D-Aria的RGB图像上的性能相近。

### C. 定性优化结果

图6至图12展示了在七个核心BOP数据集上仅经过一次迭代的多物体位姿优化结果[^28]。每一行代表一个样本：第一列显示输入图像，第二列展示通过FoundPose[^51]的快速模板检索得到的粗略位姿。第三列和第四列分别显示我们网络的预测结果（可见模板像素的掩码以及光流场），这些结果可用于将模板中的像素重映射到输入图像中，如第五列所示。最后一列呈现初始位姿（蓝色）以及由网络预测生成的2D-3D对应关系经PnP-RANSAC估算出的最终位姿（红色）。从所有图中的第二列和第三列可以看出，我们的方法对遮挡具有鲁棒性，能够准确估算可见掩码，进而得到精确的位姿估计。此外，许多样本（如图7的前两行）也表明我们的方法能够修正较大的位姿偏差。

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td style='text-align: center; word-wrap: break-word;'></td><td colspan="2">HOT3D-Aria</td><td colspan="2">HOT3D-Quest3</td></tr><tr><td style='text-align: center; word-wrap: break-word;'></td><td style='text-align: center; word-wrap: break-word;'>ADD</td><td style='text-align: center; word-wrap: break-word;'>ADD-S</td><td style='text-align: center; word-wrap: break-word;'>ADD</td><td style='text-align: center; word-wrap: break-word;'>ADD-S</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>dumbbell_5lb</td><td style='text-align: center; word-wrap: break-word;'>88.1</td><td style='text-align: center; word-wrap: break-word;'>93.5</td><td style='text-align: center; word-wrap: break-word;'>82.3</td><td style='text-align: center; word-wrap: break-word;'>89.5</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>cellphone</td><td style='text-align: center; word-wrap: break-word;'>84.8</td><td style='text-align: center; word-wrap: break-word;'>90.0</td><td style='text-align: center; word-wrap: break-word;'>81.2</td><td style='text-align: center; word-wrap: break-word;'>85.9</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>mouse</td><td style='text-align: center; word-wrap: break-word;'>88.5</td><td style='text-align: center; word-wrap: break-word;'>93.9</td><td style='text-align: center; word-wrap: break-word;'>86.1</td><td style='text-align: center; word-wrap: break-word;'>91.6</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>keyboard</td><td style='text-align: center; word-wrap: break-word;'>89.2</td><td style='text-align: center; word-wrap: break-word;'>92.1</td><td style='text-align: center; word-wrap: break-word;'>89.3</td><td style='text-align: center; word-wrap: break-word;'>91.8</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>mug_white</td><td style='text-align: center; word-wrap: break-word;'>89.0</td><td style='text-align: center; word-wrap: break-word;'>94.6</td><td style='text-align: center; word-wrap: break-word;'>84.5</td><td style='text-align: center; word-wrap: break-word;'>90.9</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>mug_patterned</td><td style='text-align: center; word-wrap: break-word;'>86.0</td><td style='text-align: center; word-wrap: break-word;'>92.6</td><td style='text-align: center; word-wrap: break-word;'>88.1</td><td style='text-align: center; word-wrap: break-word;'>94.1</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>mug_patterned</td><td style='text-align: center; word-wrap: break-word;'>87.9</td><td style='text-align: center; word-wrap: break-word;'>93.4</td><td style='text-align: center; word-wrap: break-word;'>88.9</td><td style='text-align: center; word-wrap: break-word;'>94.0</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>bowl</td><td style='text-align: center; word-wrap: break-word;'>62.6</td><td style='text-align: center; word-wrap: break-word;'>90.5</td><td style='text-align: center; word-wrap: break-word;'>64.9</td><td style='text-align: center; word-wrap: break-word;'>88.5</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>plate_bamboo</td><td style='text-align: center; word-wrap: break-word;'>62.7</td><td style='text-align: center; word-wrap: break-word;'>80.0</td><td style='text-align: center; word-wrap: break-word;'>63.7</td><td style='text-align: center; word-wrap: break-word;'>77.8</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>flask</td><td style='text-align: center; word-wrap: break-word;'>85.4</td><td style='text-align: center; word-wrap: break-word;'>92.5</td><td style='text-align: center; word-wrap: break-word;'>85.0</td><td style='text-align: center; word-wrap: break-word;'>91.7</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>vase</td><td style='text-align: center; word-wrap: break-word;'>85.5</td><td style='text-align: center; word-wrap: break-word;'>92.4</td><td style='text-align: center; word-wrap: break-word;'>87.8</td><td style='text-align: center; word-wrap: break-word;'>93.4</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>spoon_wooden</td><td style='text-align: center; word-wrap: break-word;'>70.4</td><td style='text-align: center; word-wrap: break-word;'>77.3</td><td style='text-align: center; word-wrap: break-word;'>58.4</td><td style='text-align: center; word-wrap: break-word;'>65.5</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>spatula_red</td><td style='text-align: center; word-wrap: break-word;'>65.7</td><td style='text-align: center; word-wrap: break-word;'>72.3</td><td style='text-align: center; word-wrap: break-word;'>57.6</td><td style='text-align: center; word-wrap: break-word;'>66.1</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>potato_masher</td><td style='text-align: center; word-wrap: break-word;'>85.4</td><td style='text-align: center; word-wrap: break-word;'>91.4</td><td style='text-align: center; word-wrap: break-word;'>78.8</td><td style='text-align: center; word-wrap: break-word;'>88.0</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>holder_black</td><td style='text-align: center; word-wrap: break-word;'>84.4</td><td style='text-align: center; word-wrap: break-word;'>92.6</td><td style='text-align: center; word-wrap: break-word;'>77.4</td><td style='text-align: center; word-wrap: break-word;'>87.8</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>holder_gray</td><td style='text-align: center; word-wrap: break-word;'>88.7</td><td style='text-align: center; word-wrap: break-word;'>93.7</td><td style='text-align: center; word-wrap: break-word;'>85.9</td><td style='text-align: center; word-wrap: break-word;'>92.7</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>birdhouse_toy</td><td style='text-align: center; word-wrap: break-word;'>88.4</td><td style='text-align: center; word-wrap: break-word;'>93.8</td><td style='text-align: center; word-wrap: break-word;'>91.2</td><td style='text-align: center; word-wrap: break-word;'>94.4</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>dino_toy</td><td style='text-align: center; word-wrap: break-word;'>87.1</td><td style='text-align: center; word-wrap: break-word;'>92.9</td><td style='text-align: center; word-wrap: break-word;'>85.1</td><td style='text-align: center; word-wrap: break-word;'>92.2</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>dvd_remote</td><td style='text-align: center; word-wrap: break-word;'>74.4</td><td style='text-align: center; word-wrap: break-word;'>79.6</td><td style='text-align: center; word-wrap: break-word;'>78.6</td><td style='text-align: center; word-wrap: break-word;'>85.1</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>whiteboard_eraser</td><td style='text-align: center; word-wrap: break-word;'>85.6</td><td style='text-align: center; word-wrap: break-word;'>90.8</td><td style='text-align: center; word-wrap: break-word;'>84.4</td><td style='text-align: center; word-wrap: break-word;'>90.1</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>whiteboard_marker</td><td style='text-align: center; word-wrap: break-word;'>85.2</td><td style='text-align: center; word-wrap: break-word;'>91.5</td><td style='text-align: center; word-wrap: break-word;'>83.8</td><td style='text-align: center; word-wrap: break-word;'>89.1</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>carton_milk</td><td style='text-align: center; word-wrap: break-word;'>89.4</td><td style='text-align: center; word-wrap: break-word;'>93.4</td><td style='text-align: center; word-wrap: break-word;'>85.9</td><td style='text-align: center; word-wrap: break-word;'>92.8</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>carton_oj</td><td style='text-align: center; word-wrap: break-word;'>93.6</td><td style='text-align: center; word-wrap: break-word;'>95.9</td><td style='text-align: center; word-wrap: break-word;'>86.2</td><td style='text-align: center; word-wrap: break-word;'>93.1</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>bottle_mustard</td><td style='text-align: center; word-wrap: break-word;'>88.4</td><td style='text-align: center; word-wrap: break-word;'>94.1</td><td style='text-align: center; word-wrap: break-word;'>79.5</td><td style='text-align: center; word-wrap: break-word;'>89.4</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>bottle_ranch</td><td style='text-align: center; word-wrap: break-word;'>86.8</td><td style='text-align: center; word-wrap: break-word;'>92.8</td><td style='text-align: center; word-wrap: break-word;'>80.9</td><td style='text-align: center; word-wrap: break-word;'>90.3</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>bottle_bbq</td><td style='text-align: center; word-wrap: break-word;'>76.8</td><td style='text-align: center; word-wrap: break-word;'>88.5</td><td style='text-align: center; word-wrap: break-word;'>81.2</td><td style='text-align: center; word-wrap: break-word;'>90.9</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>can_soup</td><td style='text-align: center; word-wrap: break-word;'>89.6</td><td style='text-align: center; word-wrap: break-word;'>94.3</td><td style='text-align: center; word-wrap: break-word;'>84.5</td><td style='text-align: center; word-wrap: break-word;'>90.9</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>can_parmesan</td><td style='text-align: center; word-wrap: break-word;'>91.0</td><td style='text-align: center; word-wrap: break-word;'>94.6</td><td style='text-align: center; word-wrap: break-word;'>88.9</td><td style='text-align: center; word-wrap: break-word;'>93.8</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>can_tomato_sauce</td><td style='text-align: center; word-wrap: break-word;'>90.6</td><td style='text-align: center; word-wrap: break-word;'>94.8</td><td style='text-align: center; word-wrap: break-word;'>85.8</td><td style='text-align: center; word-wrap: break-word;'>92.3</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>food_waffles</td><td style='text-align: center; word-wrap: break-word;'>85.1</td><td style='text-align: center; word-wrap: break-word;'>92.2</td><td style='text-align: center; word-wrap: break-word;'>86.3</td><td style='text-align: center; word-wrap: break-word;'>91.6</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>food_vegetables</td><td style='text-align: center; word-wrap: break-word;'>87.5</td><td style='text-align: center; word-wrap: break-word;'>92.6</td><td style='text-align: center; word-wrap: break-word;'>85.9</td><td style='text-align: center; word-wrap: break-word;'>91.3</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>puzzle_toy</td><td style='text-align: center; word-wrap: break-word;'>94.2</td><td style='text-align: center; word-wrap: break-word;'>96.6</td><td style='text-align: center; word-wrap: break-word;'>91.8</td><td style='text-align: center; word-wrap: break-word;'>95.3</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>aria_small</td><td style='text-align: center; word-wrap: break-word;'>65.7</td><td style='text-align: center; word-wrap: break-word;'>81.1</td><td style='text-align: center; word-wrap: break-word;'>70.9</td><td style='text-align: center; word-wrap: break-word;'>84.9</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Average</td><td style='text-align: center; word-wrap: break-word;'>83.7</td><td style='text-align: center; word-wrap: break-word;'>90.7</td><td style='text-align: center; word-wrap: break-word;'>81.5</td><td style='text-align: center; word-wrap: break-word;'>89.0</td></tr></table>

<div style="text-align: center;">Table 4. Tracking performance on the HOT3D dataset [^3]. The accuracy is measured by AUC w.r.t. the ADD and ADD-S pose error functions for multiple thresholds ranging from 0.1 cm to 10 cm (as in Table 2 of the main paper).</div>

在所有图表中的各列中，我们的方法对遮挡情况具有鲁棒性，能够准确估算可见掩码，从而得出精确的姿态估计结果。此外，许多样本（例如图 7 的前两行）也表明我们的方法能够修正较大的姿态偏差。

### D. 定性跟踪结果

• YCB-V数据集上的跟踪结果[^79]：

<https://youtu.be/6B1ZOQkvxH4>

• HOT3D-Aria数据集上的跟踪结果[^3]：

<https://youtu.be/74NXe03ySdM>;

• HOT3D-Quest3数据集上的跟踪结果[^3]：

<https://youtu.be/bjQWS9vgPT0>.

每个视频分别展示了输入数据、模型到帧的 $^{1}$ 光流、帧间光流、姿态拟合统计数据以及

---

![Image](images/3644af58721e4e77e8bd9db30d5b9c27d84e8ef8f881ab9171780191e01c8f6d.jpg)

<div style="text-align: center;">Figure 5. Visualization of tracking. Tiles ① and ② show crops of the previous and the current frame, respectively. Tile ⑤ shows a template rendered with the previous pose estimate. Tile ⑥ shows the masked template-to-frame flow, which is visualized for all frames but actually used only when the inlier ratio is below a threshold  $\tau_i$. Tile ⑦ shows the frame-to-frame flow which is calculated and used in every frame. Tile ③ shows a map of re-projection errors and several lines of text with information about the pose fitting. The second line of text shows the pose quality  q. The third line shows the statistics of the re-projection error (50th, 75th, and 100th percentile). The fourth line shows the number of used correspondences in the current frame (A), the number of inlier correspondences in the current frame (B), and the inlier ratio in the current frame (B/A). The fifth line shows the number of inlier correspondences in the current frame (B), the number of inlier correspondences in the last keyframe (C), i.e., the frame where we ran the template-to-frame registration last time, and the inlier ratio (B/C) – the template-to-frame registration is triggered if this inlier ratio is below  $\tau_i$. The last line shows whether the template-to-frame registration is triggered. Tile ④ shows a pose estimated using the full tracking approach described in Sec. 3.3 (i.e., with frame-to-frame flow). Tile ⑧ shows a pose that we would obtain if we ran the template-to-frame registration at every frame without frame-to-frame flow. Tile ⑨ shows the pose estimate from ④ in the raw input frame.</div>

样本序列的估算姿态（详情见图 5）。这些姿态是利用 FoundPose 中的模板检索方法来初始化的 [^51]。

如第3.3节所述，当内点比例低于$\tau_i = 0.8$时（即上一次模型与帧配准得到的内点对应关系中，当前帧中仍属于内点的比例不足80%），便会触发模型与帧的配准操作。触发模型与帧配准的频率取决于场景的动态特性：如果物体姿态和视角变化缓慢，比如在YCB-V数据集[^79]中，触发频率就会比HOT3D数据集[^3]中的动态序列更低，尤其是当物体被手部操控时。通过调低$\tau_i$的值可以降低触发频率，但可能会引入与实际姿态的偏差。在每一帧中，我们最多使用1万组对应关系（如果数量更多，则随机选取一个子集）。

---

![Image](images/8161262dfb92c3947968f1436d3201aad423f05ab5832f9c6bc81240d5f26728.jpg)

<div style="text-align: center;">Figure 6. Example results of the GoTrack refiner on LM-O [^5].</div>

---

![Image](images/65ea6b1cbfd382817389b45e5b03daad3b2279e5cb5120e5a3244d3596624cb2.jpg)

<div style="text-align: center;">Figure 7. Example results of the GoTrack refiner on T-LESS [^22].</div>

---

![Image](images/5a604c5f4bb440f8ac956e537017582909ea7696e1e1cb8a06526bf4fac0b1ce.jpg)

<div style="text-align: center;">Figure 8. Example results of the GoTrack refiner on HB [^31].</div>

---

![Image](images/d261bf6b181bcaee560f5dbb031ce0a0a6e9737cc8f3a91ffe8441a98ba438a7.jpg)

<div style="text-align: center;">Figure 9. Example results of the GoTrack refiner on YCBV [^79].</div>

---

![Image](images/339e4598bb5c6a1db712adba668179ec01e8e61ba8529f08fcbbfce88e181ad9.jpg)

<div style="text-align: center;">Figure 10. Example results of the GoTrack refiner on IC-BIN [^10].</div>

---

![Image](images/840ab6b2ab3191a0963418ac46cc15920fb2084d68f3a8113592795ecb0350a3.jpg)

<div style="text-align: center;">Figure 11. Example results of the GoTrack refiner on TUDL [^25].</div>

---

![Image](images/80ecbe4ea1b1f397bb21434e277ac3ff4d2ad275164e57548448e06a6c35c400.jpg)

<div style="text-align: center;">Figure 12. Example results of the GoTrack refiner on ITODD [^12].</div>