# FoundationPose：新型物体的统一6D姿态估计与跟踪
文博恩、韦杨、Jan Kautz、Stan Birchfield
英伟达
## 摘要
我们提出了FoundationPose，这是一个用于6D物体姿态估计与跟踪的统一基础模型，同时支持基于模型和无需模型的两种场景。只要给定物体的CAD模型或少量参考图像，我们的方法就能在测试时直接应用于新型物体，无需进行微调。得益于这一统一框架，两种场景下使用的下游姿态估计模块完全相同；当没有CAD模型可用时，该框架会采用神经隐式表示来实现高效的新视角合成。通过大规模合成数据训练，结合大型语言模型（LLM）、新颖的基于Transformer的架构以及对比学习机制，我们实现了强大的泛化能力。在涉及复杂场景和物体的多个公开数据集上进行的广泛评估表明，我们的统一方法相比针对各任务专门设计的方法有显著优势。此外，尽管假设条件更少，它的表现甚至能与实例级方法相媲美。项目页面：
### 1. 引言
计算物体到相机的刚性6D变换，也就是物体姿态估计，对机器人操控[^29] [^63] [^64]和混合现实[^40]等多种应用而言至关重要。传统方法[^19] [^20] [^30] [^47] [^62]属于实例级方法，因为它们仅能处理训练时确定的特定物体实例。这类方法通常需要带纹理的CAD模型来生成训练数据，无法在测试时应用于未见过的新型物体。虽然类别级方法[^5] [^32] [^54] [^58] [^69]消除了这些假设（即实例级训练和CAD模型要求），但它们仅能处理训练时所针对的预定义类别内的物体。此外，获取类别级训练数据极其困难，部分原因是必须执行额外的姿态规范化与校验步骤[^58]。
为解决这些局限性，近期的研究重点转向任意新型物体的即时姿态估计问题[^18] [^31] [^38] [^50] [^53]。根据测试时可用的信息不同，可分为两种不同的场景：
Figure 1. Our unified framework enables both 6D pose estimation and tracking for novel objects, supporting the model-based and model-free setups. On each of these four tasks, it outperforms prior work specially designed for the task (• indicates RGB-only; × indicates RGBD, like ours). The metric for each task is explained in detail in the experimental results.
一种是基于模型的场景，即提供物体的带纹理3D CAD模型；另一种是无模型场景，即提供一组物体的参考图像。尽管针对这两种场景分别已经取得了不少进展，但由于现实世界中的不同应用场景会提供不同类型的信息，仍需要一种能够统一处理两种场景的单一方法。
与单帧物体姿态估计不同，姿态跟踪方法[^8] [^28] [^34] [^37] [^51] [^57] [^61] [^66]利用时间线索，能够在视频序列上实现更高效、平滑且准确的姿态估计。根据其对物体知识的假设不同，这些方法存在与上述姿态估计方法类似的问题。
在本文中，我们提出了一种名为FoundationPose的统一框架，该框架能够利用RGBD图像在基于模型和无模型两种设置下对新物体进行姿态估计与跟踪。如图1所示，我们的方法在以上四项任务上均优于现有的专门方法。我们强大的泛化能力得益于大规模合成数据训练，这种训练得到了大型语言模型（LLM）的支持，同时也依托于新颖的基于Transformer的架构和对比学习机制。
---
我们通过神经隐式表示桥接了基于模型和无模型设置之间的鸿沟，该表示仅需少量（$\sim$16）参考图像即可实现高效的新视图合成，其渲染速度显著快于以往的渲染-比较方法[^31] [^34] [^61]。我们的贡献可总结如下：
- 我们提出了一种用于新物体姿态估计与跟踪的统一框架，支持基于模型和无模型两种设置。一种以物体为中心的神经隐式表示能够实现高效的新视图合成，从而弥合这两种设置之间的差距。
- 我们提出了一种由LLM辅助的合成数据生成流程，通过多样化的纹理增强来扩充3D训练资产的种类。
本研究开发的代码和数据将会被发布。
- 我们新颖设计的基于Transformer的网络架构以及对比学习机制，使得模型仅依靠合成数据训练就能具备强大的泛化能力。
- 在多个公开数据集上，我们的方法相比针对各任务专门设计的方法均有大幅提升；即便在假设条件更少的情况下，其性能仍可与实例级方法相媲美。
### 2. 相关工作
基于CAD模型的物体姿态估计。实例级姿态估计方法[^19] [^20] [^30] [^47]假定物体已提供带纹理的CAD模型，且训练与测试时使用的是完全相同的实例。物体姿态通常通过直接回归[^35] [^67]来求解，或通过构建2D-3D对应关系后再进行PnP计算[^47] [^55]，又或通过构建3D-3D对应关系后进行最小二乘拟合[^19] [^20]。为了放宽对物体知识的假设限制，类别级方法[^5] [^32] [^54] [^58] [^69] [^71]可应用于同一类别的新物体实例，但它们无法推广到预定义类别之外的任意新物体。为解决这一局限，近期的研究[^31] [^50]旨在只要测试时提供CAD模型，就能对任意新物体进行即时姿态估计。
少样本无模型物体姿态估计。无模型方法无需依赖显式的纹理模型，只需提供若干张包含目标物体的参考图像即可[^18] [^21] [^48] [^53]。RLLG [^3]和NeRF-Pose [^33]提出了无需物体CAD模型的实例级训练方式。具体而言，[^33]通过构建神经辐射场，为物体坐标图和掩码提供半监督信号。与之不同，我们引入了基于SDF表示的神经物体场，以实现高效的RGB和深度渲染，从而弥合基于模型与无模型场景之间的差距。此外，本研究聚焦于可泛化的新物体姿态估计，而这一点并非[^3]和[^33]所能实现的。为了处理新物体，Gen6D [^38]设计了检测、检索与优化流程；不过为避免在分布外测试集上出现困难，它需要进行微调。OnePose [^53]及其扩展版OnePose++ [^18]利用运动恢复结构（SfM）进行物体建模，并预训练2D-3D匹配网络以通过对应关系求解姿态。FS6D [^21]采用了类似的方案，且专注于RGBD模态。然而，当应用于无纹理物体或严重遮挡场景下时，对依赖对应关系的做法会变得不可靠。
物体姿态跟踪。6D物体姿态跟踪旨在利用时间线索，实现视频序列上更高效、平滑且准确的姿态预测。通过神经渲染技术，我们的方法能够轻松扩展到姿态跟踪任务，且效率极高。与单帧姿态估计类似，现有的跟踪方法可根据对物体知识的假设分为不同类型：包括实例级方法[^8] [^11] [^34] [^61]、类别级方法[^37] [^57]、基于模型的新物体跟踪[^28] [^51] [^66]以及无模型新物体跟踪[^60] [^65]。无论是在基于模型还是无模型设置下，我们都在公开数据集上创下了新的基准记录，甚至超越了那些需要实例级训练的最先进方法[^8] [^34] [^61]。
### 3. 方法
图2展示了我们整个系统的构成及各组件间的关系，下文各小节将对这些组件进行详细阐述。
#### 3.1. 大规模语言辅助数据生成
为了实现强大的泛化能力，训练时需要涵盖多种多样的物体和场景。在现实世界中获取此类数据并标注精确的6D姿态真值，不仅耗时且成本高昂。而合成数据往往在3D资产的数量和多样性上存在不足。我们开发了一种全新的合成数据生成流程用于训练，该流程依托于近期涌现的资源与技术：大规模3D模型数据库[^6] [^10]、大型语言模型（LLM）以及扩散模型[^4] [^23] [^49]。与以往的研究相比，这种方法大幅提升了数据的数量和多样性[^21] [^25] [^31]。
3D 资产。我们的训练资产来源于近期的大规模 3D 数据库，包括 Objaverse [^6] 和 GSO [^10]。对于 Objaverse [^6]，我们选取了来自 Objaverse-LVIS 子集的物体，该子集包含超过 4 万个属于 1156 个 LVIS [^13] 类别的物体。这份列表涵盖了日常生活中最常见、质量可靠且形状和外观多样的物体。它还为每个物体提供了描述其类别的标签。
---
Figure 2. Overview of our framework. To reduce manual efforts for large scale training, we developed a novel synthetic data generation pipeline by leveraging recent emerging techniques and resources including 3D model database, large language models and diffusion models (Sec. 3.1). To bridge the gap between model-free and model-based setup, we leverage an object-centric neural field (Sec. 3.2) for novel view RGBD rendering for subsequent renderings and compared. For pose estimation, we first initialize global poses uniformly around the object, which are then refined by the refinement network (Sec. 3.3). Finally, we forward the refined poses to the pose selection module which predicts their scores. The pose with the best score is selected as output (Sec. 3.4).
Figure 3. Top: Random texture blending proposed in FS6D [^21]. Bottom: Our LLM-aided texture augmentation yields more realistic appearance. Leftmost is the original 3D assets. Text prompts are automatically generated by ChatGPT.
这些标签在后续的 LLM 辅助纹理增强步骤中，对自动生成语言提示非常有帮助。
LLM 辅助纹理增强。虽然大多数 Objaverse 物体的形状质量较高，但它们的纹理保真度差异很大。FS6D [^21] 提出通过随机粘贴来自 ImageNet [^7] 或 MS-COCO [^36] 的图像来增强物体纹理。然而，由于随机 UV 映射的原因，该方法会在生成的带纹理网格上产生接缝等瑕疵（见图 3 顶部）；而且将整幅场景图像应用到物体上会导致不真实的结果。相比之下，我们探索如何利用大型语言和扩散模型的最新进展来实现更逼真（且完全自动化）的纹理增强。具体来说，我们会向 TexFusion [^4] 提供文本提示、物体形状以及随机初始化的噪声纹理，以生成增强后的带纹理模型。当然，如果想要为大量不同风格、在不同提示引导下生成的物体进行纹理增强，手动提供此类提示显然不具备可扩展性。因此，我们提出了一种两级分层提示策略。如图 2 左上所示，我们首先向 ChatGPT 提问，让其描述某物体的可能外观；该提示采用了模板化设计，每次只需替换与物体对应的标签即可，这些标签由 Objaverse-LVIS 列表提供。ChatGPT 给出的回答随后便成为提供给扩散模型用于纹理合成的文本提示 pro-
---
该文本提示被送入扩散模型以完成纹理合成。由于这种方法能实现纹理增强的全自动化，因此能够大规模地生成多样化的数据。图 3 展示了更多示例，包括同一物体的不同风格化效果。
数据生成。我们的合成数据生成是在 NVIDIA Isaac Sim 中实现的，利用路径追踪技术来渲染高保真的照片级图像。$^{1}$ 我们还会进行重力和物理模拟，以生成符合物理规律的场景。在每个场景中，我们会随机选取包括原始物体和纹理增强版本在内的物体。物体的尺寸、材质、相机姿态和光照也会被随机设置；更多细节可在附录中找到。
#### 3.2. 神经物体建模
在无模型设置下，当缺乏 3D CAD 模型时，如何有效表示物体以便为下游模块渲染出高质量图像便成为一个关键挑战。神经隐式表示既适用于新视角合成，又能在 GPU 上并行处理，因此在为下游姿态估计模块渲染多个姿态假设时具备很高的计算效率，如图 2 所示。基于此，我们借鉴以往的研究成果 [^42] [^59] [^65] [^68]，引入了以物体为中心的神经场表示来进行物体建模。
字段表示。我们用两个函数来表示该物体，如图2所示，具体为[^68]。首先，几何函数$\Omega : x \mapsto s$以三维点$x \in \mathbb{R}^3$作为输入，输出一个有符号距离值$s \in \mathbb{R}$。其次，外观函数$\Phi : (f_{\Omega(x)}, n, d) \mapsto c$接收来自几何网络的中间特征向量$f_{\Omega(x)}$、点法线$n \in \mathbb{R}^3$以及观察方向$d \in \mathbb{R}^3$，最终输出颜色值$c \in \mathbb{R}^3$。在实际应用中，在将输入数据传递给网络之前，我们会先对x应用多分辨率哈希编码[^42]。n和d均通过一组固定的二阶球面谐波系数进行嵌入。隐式物体表面是通过取有符号距离场（SDF）的零水平集得到的：$S = \{x \in \mathbb{R}^3 \mid \Omega(x) = 0\}$。与NeRF[^41]相比，SDF表示法$\Omega$能够提供更高质量的深度渲染效果，同时无需手动选择密度阈值。
字段学习。在纹理学习方面，我们采用基于截断近表面区域的体积渲染方法[^65]：
$$ c(r)=\int_{z(r)-\lambda}^{z(r)+0.5\lambda}w(x_{i})\Phi(f_{\Omega(x_{i})},n(x_{i}),d(x_{i}))d t, $$
$$ w(x_{i})=\frac{1}{1+e^{-\alpha\Omega(x_{i})}}\frac{1}{1+e^{\alpha\Omega(x_{i})}}, $$
其中，$w(x_i)$是钟形概率密度函数[^59]，它取决于该点到隐式物体表面的有符号距离$\Omega(x_i)$，而$\alpha$则用于调节分布的平滑程度。该概率在表面交点处达到峰值。在公式(1)中，$z(r)$是来自深度图像的射线深度值，$\lambda$则是截断距离。为了提高训练效率，我们会忽略距离表面超过$\lambda$的空旷空间的贡献；同时，为了建模自遮挡现象，我们仅积分到$0.5\lambda$这一穿透距离为止[^59]。在训练过程中，我们会将该数值与参考RGB图像进行比较，以实现颜色监督：
$$ \mathcal{L}_{c}=\frac{1}{|\mathcal{R}|}\sum_{r\in\mathcal{R}}\left\|c(r)-\bar{c}(r)\right\|_{2}, $$
其中，$\bar{c}(r)$表示光线r穿过像素处的真实颜色。
在几何学习方面，我们采用混合SDF模型[^65]，通过将空间划分为两个区域来学习SDF，进而得到空旷空间损失和近表面损失。我们还会对近表面SDF应用欧拉正则化约束[^12]：
$$ \mathcal{L}_{e}=\frac{1}{|\mathcal{X}_{e}|}\sum_{x\in\mathcal{X}_{e}}|\Omega(x)-\lambda|, $$
$$ \mathcal{L}_{s}=\frac{1}{|\mathcal{X}_{s}|}\sum_{x\in\mathcal{X}_{s}}\left(\Omega(x)+d_{x}-d_{D}\right)^{2}, $$
$$ \mathcal{L}_{e i k}=\frac{1}{|\mathcal{X}_{s}|}\sum_{x\in\mathcal{X}_{s}}(\|\nabla\Omega(x)\|_{2}-1)^{2}, $$
其中，x表示划分后的空间中沿光线采样的三维点；$d_x$和$d_D$分别为从光线起点到采样点的距离以及观测到的深度点。由于模型无先验设置下使用的模板图像是预先离线采集的，因此我们并未采用不确定的自由空间损失[^65]。总的训练损失为
$$ \mathcal{L}=w_{c}\mathcal{L}_{c}+w_{e}\mathcal{L}_{e}+w_{s}\mathcal{L}_{s}+w_{e i k}\mathcal{L}_{e i k}. $$
这种学习过程是针对每个物体单独优化的，无需依赖先验知识，且能在几秒内高效完成。对于新物体而言，神经场只需训练一次即可。
渲染。训练完成后，神经场可作为传统图形管线的直接替代品，用于高效渲染物体，以便进行后续的渲染与比较迭代。除了原始NeRF中的颜色渲染[^41]外，我们基于RGBD的位姿估计与跟踪还需要深度渲染。为此，我们采用Marching Cubes算法[^39]从SDF的零水平集提取带纹理的网格，并结合颜色投影。这一过程只需对每个物体执行一次即可。在推理阶段，给定物体位姿后，我们便按照光栅化流程渲染RGBD图像。或者，也可以直接使用$\Omega$在线进行球体追踪[^14]来直接渲染深度图像；不过我们发现这种方式效率较低，尤其是在需要并行渲染大量位姿假设时。
---
#### 3.3. 位姿假设生成
位姿初始化。给定RGBD图像后，可使用Mask R-CNN[^17]或CNOS[^44]等现成方法来检测物体。我们利用检测到的2D边界框内深度值的中位数所对应的3D点来初始化平移参数。对于旋转参数的初始化，我们从以物体为中心、相机朝向中心的球面上均匀采样$N_s$个视点。这些相机位姿还会进一步加上$N_i$离散化的平面内旋转，最终得到$N_s \cdot N_i$个全局位姿初始值，作为位姿优化器的输入。
位姿优化。由于上一步得到的粗略位姿初始化结果往往存在较大噪声，因此需要优化模块来提升位姿质量。具体来说，我们构建了一个位姿优化网络，其输入为基于粗略位姿渲染的物体图像以及相机拍摄的输入图像裁剪块；网络输出的位姿更新量可用来改善位姿质量。与MegaPose[^31]不同——后者会围绕粗略位姿渲染多个视角以找到锚点，我们发现仅渲染对应于粗略位姿的单视图就足够了。对于输入图像，我们并非基于固定的2D检测结果进行裁剪，而是采用受位姿约束的裁剪策略，从而为平移参数的更新提供反馈。具体而言，我们将物体原点投影到图像空间以确定裁剪中心；接着将略微放大的物体直径（物体表面上任意两点间的最大距离）投影出来，以此确定能包含物体及其附近上下文的裁剪尺寸。这种裁剪方式依赖于粗略位姿，能够促使网络更新平移参数，使裁剪区域与观测结果更对齐。优化过程可以重复多次，每次将最新更新的位姿作为输入送入下一次推理，从而逐步提升位姿质量。
图2展示了细化网络的架构；详情见附录。我们首先通过共享的CNN编码器从两个RGBD输入分支中提取特征图。这些特征图被拼接起来，送入带有残差连接的CNN模块[^16]，随后通过划分成图像块[^9]并加入位置嵌入来进行标记化。最终，网络会预测平移更新量$\Delta t \in \mathbb{R}^3$和旋转更新量$\Delta \boldsymbol{R} \in \mathbb{S}\mathbb{O}(3)$，二者分别由Transformer编码器处理[^56]，再线性投影到输出维度。更具体地说，$\Delta t$表示物体在相机坐标系中的平移偏移，$\Delta \boldsymbol{R}$表示物体在相机坐标系中的方向更新量。实际应用中，旋转采用轴角表示法来参数化。我们也尝试了6D表示方法[^72]，取得了类似的结果。接着，输入的粗略姿态$[\boldsymbol{R} | t] \in \mathbb{SE}(3)$会被更新。
Figure 4. Pose ranking visualization. Our proposed hierarchical comparison leverages the global context among all pose hypotheses for a better overall trend prediction that aligns both shape and texture. The true best pose is annotated with red circle.
更新方式如下：
$$ t^{+}=t+\Delta t $$
$$ \boldsymbol{R}^{+}=\Delta\boldsymbol{R}\otimes\boldsymbol{R}, $$
其中$\otimes$表示对$\mathbb{SO}(3)$的更新。这种解耦的表示方式没有采用单一的齐次姿态更新，从而在应用平移更新时不再依赖于已更新的方向。这种方式将更新量与输入观测值统一在相机坐标系中，简化了学习过程。网络训练由$L_2$损失函数监督：
$$ \begin{array}{r}{\mathcal{L}_{\mathrm{r e f i n e}}=w_{1}\left\|\Delta\boldsymbol{t}-\Delta\bar{\boldsymbol{t}}\right\|_{2}+w_{2}\left\|\Delta\boldsymbol{R}-\Delta\bar{\boldsymbol{R}}\right\|_{2},}\end{array} $$
其中$\bar{t}$和$\bar{R}$为真实值；$w_1$和$w_2$是用于平衡两种损失的权重，根据经验设为1。
#### 3.4. 姿态选择
给定一组细化后的姿态假设，我们使用分层姿态排序网络来计算它们的得分。得分最高的姿态被选为最终估计结果。
分层比较。该网络采用两级比较策略。首先，针对每个姿态假设，将渲染图像与裁剪后的输入观测值进行比较，这种基于姿态的裁剪操作在第3.3节中已有介绍。这种比较（见图2左下角）由姿态排序编码器执行，它采用与细化网络相同的骨干架构来提取特征。提取的特征被拼接、标记化后送入多头自注意力模块，以便更好地利用全局图像上下文进行对比。姿态排序编码器通过平均池化输出描述渲染结果与观测值之间对齐质量的特征嵌入$\mathcal{F} \in \mathbb{R}^{512}$（见图2中间下方）。此时，我们本可以直接将$\mathcal{F}$投影为相似度标量，就像通常所做的那样[^2] [^31] [^43]。但这样会忽略其他姿态假设，迫使网络输出绝对得分，而这类得分难以学习。
为了利用所有姿态假设的全局上下文来做出更明智的决策，我们引入了针对全部K个姿态假设的第二级比较。对拼接后的特征执行多头自注意力操作
---
特征嵌入$\mathbf{F} = [\mathcal{F}_0, \ldots, \mathcal{F}_{K-1}]^\top \in \mathbb{R}^{K \times 512}$，它编码了所有姿态中的姿态对齐信息。将$\mathbf{F}$视为一个序列后，这种方法便能自然地处理长度各异的K个姿态[^56]。我们不对$\mathbf{F}$应用位置编码，以保持其对排列顺序的无关性。随后，经注意力机制提取的特征会被线性投影为分数$\mathbf{S} \in \mathbb{R}^K$，用于分配给各姿态假设。图4展示了一个典型示例，说明了这种分层比较策略的有效性。
对比验证。为了训练姿态排序网络，我们提出了一种基于姿态的条件三元组损失函数：
$$ \mathcal{L}(i^{+},i^{-})=\operatorname*{m a x}(\mathbf{S}(i^{-})-\mathbf{S}(i^{+})+\alpha,0), $$
其中$\alpha$表示对比边界；$i^-$和$i^+$分别代表负样本和正样本姿态，它们是通过利用真实值计算ADD度量[^67]来确定的。需要注意的是，与标准三元组损失[^26]不同，在我们的场景中，正样本和负样本并不共享锚点样本，因为输入是根据每个姿态假设裁剪的，以考虑平移因素。虽然我们可以针对列表中的每对样本计算该损失，但当两个姿态都远离真实值时，比较结果会变得模糊不清。因此，我们只保留那些正样本视角与真实值足够接近、从而能使比较有意义的姿态对：
$$ \mathbb{V}^{+}=\{i:D(\mathbf{R}_{i},\bar{\mathbf{R}}) | PREDATOR [^27] | LoFTR [^52] | FS6D-DPM [^21] | Ours |
| Ref. images | 16 | 16 | 16 | 16 |
| Finetune-free | ✓ | ✓ | ✗ | ✓ |
| Metrics | ADD-S | ADD | ADD-S | ADD | ADD-S | ADD | ADD-S | ADD |
| 002_master_chef_can | 73.0 | 17.4 | 87.2 | 50.6 | 92.6 | 36.8 | 96.9 | 91.3 |
| 003_cracker_box | 41.7 | 8.3 | 71.8 | 25.5 | 83.9 | 24.5 | 97.5 | 96.2 |
| 004_sugar_box | 53.7 | 15.3 | 63.9 | 13.4 | 95.1 | 43.9 | 97.5 | 87.2 |
| 005_tomato_soup_can | 81.2 | 44.4 | 77.1 | 52.9 | 93.0 | 54.2 | 97.6 | 93.3 |
| 006_mustard_bottle | 35.5 | 5.0 | 84.5 | 59.0 | 97.0 | 71.1 | 98.4 | 97.3 |
| 007_tuna_fish_can | 78.2 | 34.2 | 72.6 | 55.7 | 94.5 | 53.9 | 97.7 | 73.7 |
| 008_pudding_box | 73.5 | 24.2 | 86.5 | 68.1 | 94.9 | 79.6 | 98.5 | 97.0 |
| 009_gelatin_box | 81.4 | 37.5 | 71.6 | 45.2 | 98.3 | 32.1 | 98.5 | 97.3 |
| 010_potted_meat_can | 62.0 | 20.9 | 67.4 | 45.1 | 87.6 | 54.9 | 96.6 | 82.3 |
| 011_banana | 57.7 | 9.9 | 24.2 | 1.6 | 94.0 | 69.1 | 98.1 | 95.4 |
| 019_pitcher_base | 83.7 | 18.1 | 58.7 | 22.3 | 91.1 | 40.4 | 97.9 | 96.6 |
| 021_bleach_cleanser | 88.3 | 48.1 | 36.9 | 16.7 | 89.4 | 44.1 | 97.4 | 93.3 |
| 024_bowl | 73.2 | 17.4 | 32.7 | 1.4 | 74.7 | 0.9 | 94.9 | 89.7 |
| 025_mug | 84.8 | 29.5 | 47.3 | 23.6 | 86.5 | 39.2 | 96.2 | 75.8 |
| 035_power_drill | 60.6 | 12.3 | 18.8 | 1.3 | 73.0 | 19.8 | 98.0 | 96.3 |
| 036_wood_block | 70.5 | 10.0 | 49.9 | 1.4 | 94.7 | 27.9 | 97.4 | 94.7 |
| 037_scissors | 75.5 | 25.0 | 32.3 | 14.6 | 74.2 | 27.7 | 97.8 | 95.5 |
| 040_large_marker | 81.8 | 38.9 | 20.7 | 8.4 | 97.4 | 74.2 | 98.6 | 96.5 |
| 051_large_clamp | 83.0 | 34.4 | 24.1 | 11.2 | 82.7 | 34.7 | 96.9 | 92.7 |
| 052_extra_large_clamp | 72.9 | 24.1 | 15.0 | 1.8 | 65.7 | 10.1 | 97.6 | 94.1 |
| 061_foam_brick | 79.2 | 35.5 | 59.4 | 31.4 | 95.7 | 45.8 | 98.1 | 93.4 |
| MEAN | 71.0 | 24.3 | 52.5 | 26.2 | 88.4 | 42.1 | 97.4 | 91.5 |
Table 1. Model-free pose estimation results measured by AUC of ADD and ADD-S on YCB-Video dataset. “Finetuned” means the method was fine-tuned with group split of object instances on the testing dataset, as introduced by [^21].
从数据集的训练划分中选取，并带有物体姿态的 ground-truth 标注，遵循 [^21]。对于 model-based 设置，为新物体提供 CAD 模型。在除消融实验外的所有评估中，我们的方法始终使用相同的训练模型和配置进行推理，无需任何微调。
#### 4.2. 评价指标
为了与各设置的基线协议保持一致，我们考虑以下评价指标：
- ADD和ADD-S [^67]曲线下的面积（AUC）。
- ADD小于物体直径0.1倍时的召回率（ADD-0.1d），这一指标在[^18]和[^21]中有所使用。
- BOP挑战赛[^25]中引入的VSD、MSSD和MSPD指标的平均召回率（AR）。
#### 4.3. 姿态估计对比
无模型方法。表1展示了在YCB-Video数据集上，本方法与最先进的RGBD方法的对比结果[^21] [^27] [^52]。基线结果取自[^21]。按照[^21]的做法，为确保所有方法可比性，所有方法均采用经过扰动的真实边界框作为2D检测结果。表2展示了在LINEMOD数据集上的对比结果，基线结果取自[^18] [^21]。基于RGB的方法[^18] [^38] [^53]被给予了数量更多的参考图像，以弥补深度信息的不足。在RGBD方法中，FS6D[^21]需要在目标数据集上进行微调。而我们的方法无需在目标数据集上微调或采用ICP优化，便能在两个数据集上显著优于现有方法。
#### 图5展示了定性比较结果。由于FS6D的代码并未公开发布，我们无法获取其姿态预测结果以用于定性分析，详见[^21]。
---
| Method | Modality | Finetune-free | Ref. images | ape | benchwise | cam | can | cat | driller | Objects | Avg. |
| duck | eggbox | glue | holepuncher | iron | lamp | phone |
| Gen6D [^38] | RGB | ✗ | 200 | - | 77 | 66.1 | - | 60.7 | 67.4 | 40.5 | 95.7 | 87.2 | - | - | - | - | - |
| Gen6D $^{*}$ [^38] | RGB | ✓ | 200 | - | 62.1 | 45.6 | - | 40.9 | 48.8 | 16.2 | - | - | - | - | - | - | - |
| OnePose [^53] | RGB | ✓ | 200 | 11.8 | 92.6 | 88.1 | 77.2 | 47.9 | 74.5 | 34.2 | 71.3 | 37.5 | 54.9 | 89.2 | 87.6 | 60.6 | 63.6 |
| OnePose++ [^18] | RGB | ✓ | 200 | 31.2 | 97.3 | 88.0 | 89.8 | 70.4 | 92.5 | 42.3 | 99.7 | 48.0 | 69.7 | 97.4 | 97.8 | 76.0 | 76.9 |
| LatentFusion [^48] | RGBD | ✓ | 16 | 88.0 | 92.4 | 74.4 | 88.8 | 94.5 | 91.7 | 68.1 | 96.3 | 94.9 | 82.1 | 74.6 | 94.7 | 91.5 | 87.1 |
| FS6D [^21] | RGBD | ✗ | 16 | 74.0 | 86.0 | 88.5 | 86.0 | 98.5 | 81.0 | 68.5 | 100.0 | 99.5 | 97.0 | 92.5 | 85.0 | 99.0 | 88.9 |
| FS6D [^21] + ICP | RGBD | ✗ | 16 | 78.0 | 88.5 | 91.0 | 89.5 | 97.5 | 92.0 | 75.5 | 99.5 | 99.5 | 96.0 | 87.5 | 97.0 | 97.5 | 91.5 |
| Ours | RGBD | ✓ | 16 | 99.0 | 100.0 | 100.0 | 100.0 | 100.0 | 100.0 | 99.4 | 100.0 | 100.0 | 99.9 | 100.0 | 100.0 | 100.0 | 99.9 |
Table 2. Model-free pose estimation results measured by ADD-0.1d on LINEMOD dataset. Gen6D $^{*}$ [^38] represents the variation without fine-tuning.
Figure 5. Qualitative comparison of pose estimation on LINEMOD dataset under the model-free setup. Images are cropped and zoomed-in for better visualization.
severe self-occlusion and lack of texture on the glue largely challenge OnePose++ [^18] and LatentFusion [^48], while our method successfully estimates the pose.
| Method | Unseen objects | Dataset | Mean |
| LM-O | T-LESS | YCB-V |
| SurfEmb [^15] + ICP | ✗ | 75.8 | 82.8 | 80.6 | 79.7 |
| OSOP [^50] + ICP | ✓ | 48.2 | - | 57.2 | - |
| (PPF, Sift) + Zephyr [^45] | ✓ | 59.8 | - | 51.6 | - |
| MegaPose-RGBD [^31] | ✓ | 58.3 | 54.3 | 63.3 | 58.6 |
| OVE6D [^2] | ✓ | 49.6 | 52.3 | - | - |
| GCPose [^70] | ✓ | 65.2 | 67.9 | - | - |
| Ours | ✓ | 78.8 | 83.0 | 88.0 | 83.3 |
Table 3. Model-based pose estimation results measured by AR score on representative BOP datasets. All methods use the RGBD modality.
基于模型的方法。表3展示了在BOP的三个核心数据集上，RGBD方法之间的对比结果：Occluded-LINEMOD[^1]、YCB-Video[^67]和TLESS[^24]。所有方法均采用Mask R-CNN[^17]进行2D检测。我们的方法在处理新物体以及实例级方法[^15]方面，均大幅优于现有的基于模型的方法。
#### 4.4. 姿态跟踪对比
| | se(3)- TrackNet [^61] | RGF [^28] | Bundle-Track [^60] | Bundle-SDF [^65] | Withrich [^66] | Ours | Ours $^{{3}}$ |
| Properties | Novel object | ✗ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| Initial pose | GT | GT | GT | GT | GT | GT | Est. |
| cracker_box | ADD-S | 94.06 | 55.44 | 89.41 | 90.63 | 88.13 | 95.10 | 94.92 |
| ADD | 90.76 | 34.78 | 85.07 | 83.37 | 79.00 | 91.32 | 91.54 |
| bleach_cleanser | ADD-S | 94.44 | 45.03 | 94.72 | 94.28 | 68.96 | 95.96 | 96.36 |
| ADD | 89.58 | 29.40 | 89.34 | 87.46 | 61.47 | 91.45 | 92.63 |
| sugar_box | ADD-S | 94.80 | 16.87 | 90.22 | 93.81 | 92.75 | 96.67 | 96.61 |
| ADD | 92.43 | 15.82 | 85.56 | 88.62 | 86.78 | 94.14 | 93.96 |
| tomato_soup_can | ADD-S | 96.95 | 26.44 | 95.13 | 95.24 | 93.17 | 96.58 | 96.54 |
| ADD | 93.40 | 15.13 | 86.00 | 83.10 | 63.71 | 91.71 | 91.85 |
| mustard_bottle | ADD-S | 97.92 | 60.17 | 95.35 | 95.75 | 95.31 | 97.89 | 97.77 |
| ADD | 97.00 | 56.49 | 92.26 | 89.87 | 91.31 | 96.34 | 95.95 |
| All | ADD-S | 95.53 | 39.90 | 92.53 | 93.77 | 89.18 | 96.42 | 96.40 |
| ADD | 92.66 | 29.98 | 87.34 | 86.95 | 78.28 | 93.09 | 93.22 |
Table 4. Pose tracking results of RGBD methods measured by AUC of ADD and ADD-S on YCBInEOAT dataset. Ours $^{\dagger}$ represents our unified pipeline that uses the pose estimation module for pose initialization.
除非另有说明，为评估长期跟踪的鲁棒性，在跟踪丢失的情况下不会对所评估的方法进行重新初始化。定性结果详见补充材料。
为了全面评估突发平面外旋转、动态外部遮挡以及相机运动解耦等挑战，我们在包含动态机器人操作视频的YCBInEOAT[^61]数据集上评估了姿态跟踪方法。基于模型设置下的结果见表4。我们的方法取得了最佳性能，甚至超过了需要真实姿态初始化的实例级训练方法[^61]。此外，我们的统一框架还支持无需外部姿态初始化的端到端姿态估计与跟踪，这是唯一具备该能力的方法，表中将其标记为Ours$^{\dagger}$。
表5展示了在YCB-Video[^67]数据集上的姿态跟踪对比结果。在基线方法中，DeepIM[^34]、se(3)-TrackNet[^61]和PoseRBPF[^8]需要在相同的物体实例上训练，而Wüthrich等人[^66]、RGF[^28]、ICG[^51]以及我们的方法，只需提供CAD模型即可直接应用于新物体。
### 4.5. 分析
消融实验。表6展示了关键设计选择的消融实验结果。结果通过YCB-Video数据集上ADD和ADD-S指标的AUC值来评估。我们的方法（所提出的方法）是在无模型（16张参考图像）设置下的默认版本。W/o LLM texture augmentation表示移除用于合成训练的LLM辅助纹理增强模块。在W/o transformer中，我们用卷积层和线性层替换基于Transformer的架构，同时保持相近的参数数量。W/o hierarchical comparison仅对比渲染结果与经姿态条件三元组损失函数（公式11）训练的裁剪输入，而不进行两级层次化对比。测试时，它将每个姿态假设与输入观测数据独立比较，并输出得分最高的姿态。图4展示了定性结果示例。Ours-InfoNCE用InfoNCE损失函数替换了对比验证式成对损失函数（公式14），该函数如[^43]中所用。
参考图像数量的影响。我们研究了参考图像数量对结果的影响，该结果通过YCB-Video数据集上ADD和ADD-S指标的AUC值来衡量，如图6所示。总体而言，我们的方法对参考图像数量具有鲁棒性，
---
| Approach | DeepIM [^34] | se(3)-TrackNet [^61] | PoseRBPF [^8] + SDF | Wüthrich [^66] | RGF [^28] | ICG [^51] | Ours | Ours $^{\dagger}$ |
| Initial pose | GT | GT | PoseCNN | GT | GT | GT | GT | GT |
| Re-initialization | Yes (290) | No | Yes (2) | No | No | No | No | No |
| Novel object | ✗ | ✗ | ✗ | ✓ | ✓ | ✓ | ✓ | ✓ |
| Object setup | Model-based | Model-based | Model-based | Model-based | Model-based | Model-based | Model-based | Model-free |
| Metric | ADD | ADD-S | ADD | ADD-S | ADD | ADD-S | ADD | ADD-S | ADD | ADD-S | ADD | ADD-S | ADD | ADD-S | ADD | ADD-S |
| 002_master_chef_can | 89.0 | 93.8 | 93.9 | 96.3 | 89.3 | 96.7 | 55.6 | 90.7 | 46.2 | 90.2 | 66.4 | 89.7 | 93.6 | 97.0 | 91.2 | 96.9 |
| 003_cracker_box | 88.5 | 93.0 | 96.5 | 97.2 | 96.0 | 97.1 | 96.4 | 97.2 | 57.0 | 72.3 | 82.4 | 92.1 | 96.9 | 97.8 | 96.2 | 97.5 |
| 004_sugar_box | 94.3 | 96.3 | 97.6 | 98.1 | 94.0 | 96.4 | 97.1 | 97.9 | 50.4 | 72.7 | 96.1 | 98.4 | 96.9 | 98.2 | 94.5 | 97.4 |
| 005_tomato_soup_can | 89.1 | 93.2 | 95.0 | 97.2 | 87.2 | 95.2 | 64.7 | 89.5 | 72.4 | 91.6 | 73.2 | 97.3 | 96.3 | 98.1 | 94.3 | 97.9 |
| 006_mustard_bottle | 92.0 | 95.1 | 95.8 | 97.4 | 98.3 | 98.5 | 97.1 | 98.0 | 87.7 | 98.2 | 96.2 | 98.4 | 97.3 | 98.4 | 97.3 | 98.5 |
| 007_tuna_fish_can | 92.0 | 96.4 | 86.5 | 91.1 | 86.8 | 93.6 | 69.1 | 93.3 | 28.7 | 52.9 | 73.2 | 95.8 | 96.9 | 98.5 | 84.0 | 97.8 |
| 008_pudding_box | 80.1 | 88.3 | 97.9 | 98.4 | 60.9 | 87.1 | 96.8 | 97.9 | 12.7 | 18.0 | 73.8 | 88.9 | 97.8 | 98.5 | 96.9 | 98.5 |
| 009_gelatin_box | 92.0 | 94.4 | 97.8 | 98.4 | 98.2 | 98.6 | 97.5 | 98.4 | 49.1 | 70.7 | 97.2 | 98.8 | 97.7 | 98.5 | 97.6 | 98.5 |
| 010_potted_meat_can | 78.0 | 88.9 | 77.8 | 84.2 | 76.4 | 83.5 | 83.7 | 86.7 | 44.1 | 45.6 | 93.3 | 97.3 | 95.1 | 97.7 | 94.8 | 97.5 |
| 011_banana | 81.0 | 90.5 | 94.9 | 97.2 | 92.8 | 97.7 | 86.3 | 96.1 | 93.3 | 97.7 | 95.6 | 98.4 | 96.4 | 98.4 | 95.6 | 98.1 |
| 019_pitcher_base | 90.4 | 94.7 | 96.8 | 97.5 | 97.7 | 98.1 | 97.3 | 97.7 | 97.9 | 98.2 | 97.0 | 98.8 | 96.7 | 98.0 | 96.8 | 98.0 |
| 021_bleach_cleanser | 81.7 | 90.5 | 95.9 | 97.2 | 95.9 | 97.0 | 95.2 | 97.2 | 95.9 | 97.3 | 92.6 | 97.5 | 95.5 | 97.8 | 94.7 | 97.5 |
| 024_bowl | 38.8 | 90.6 | 80.9 | 94.5 | 34.0 | 93.0 | 30.4 | 97.2 | 24.2 | 82.4 | 74.4 | 98.4 | 95.2 | 97.6 | 90.5 | 95.3 |
| 025_mug | 83.2 | 92.0 | 91.5 | 96.9 | 86.9 | 96.7 | 83.2 | 93.3 | 60.0 | 71.2 | 95.6 | 98.5 | 95.6 | 97.9 | 91.5 | 96.1 |
| 035_power_drill | 85.4 | 92.3 | 96.4 | 97.4 | 97.8 | 98.2 | 97.1 | 97.8 | 97.9 | 98.3 | 96.7 | 98.5 | 96.9 | 98.2 | 96.3 | 97.9 |
| 036_wood_block | 44.3 | 75.4 | 95.2 | 96.7 | 37.8 | 93.6 | 95.5 | 96.9 | 45.7 | 62.5 | 93.5 | 97.2 | 93.2 | 97.0 | 92.9 | 97.0 |
| 037_scissors | 70.3 | 84.5 | 95.7 | 97s | 72.7 | 85.5 | 4.2 | 16.2 | 20.9 | 38.6 | 93.5 | 97.3 | 94.8 | 97.5 | 95.5 | 97.8 |
| 040_large_marker | 80.4 | 91.2 | 92.2 | 96.0 | 89.2 | 97.3 | 35.6 | 53.0 | 12.2 | 18.9 | 88.5 | 97.8 | 96.9 | 98.6 | 96.6 | 98.6 |
| 051_large_clamp | 73.9 | 84.1 | 94.7 | 96.9 | 90.1 | 95.5 | 61.2 | 72.3 | 62.8 | 80.1 | 91.8 | 96.9 | 93.6 | 97.3 | 92.5 | 96.7 |
| 052_extra_large_clamp | 49.3 | 90.3 | 91.7 | 95.8 | 84.4 | 94.1 | 93.7 | 96.6 | 67.5 | 69.7 | 85.9 | 94.3 | 94.4 | 97.5 | 93.4 | 97.3 |
| 061_foam_brick | 91.6 | 95.5 | 93.7 | 96.7 | 96.1 | 98.3 | 96.8 | 98.1 | 70.0 | 86.5 | 96.2 | 98.5 | 97.9 | 98.6 | 96.8 | 98.3 |
| All Frames | 82.3 | 91.9 | 93.0 | 95.7 | 87.5 | 95.2 | 78.0 | 90.2 | 59.2 | 74.3 | 86.4 | 96.5 | 96.0 | 97.9 | 93.7 | 97.5 |
Table 5. Pose tracking results of RGBD methods measured by AUC of ADD and ADD-S on YCB-Video dataset. Ours $^{\dagger}$ represents our method under the model-free setup with reference images.
| ADD | ADD-S |
| Ours (proposed) | 91.52 | 97.40 |
| W/o LLM texture augmentation | 90.83 | 97.38 |
| W/o transformer | 90.77 | 97.33 |
| W/o hierarchical comparison | 89.05 | 96.67 |
| Ours-InfoNCE | 89.39 | 97.29 |
Table 6. Ablation study of critical design choices.
尤其是在ADD-S指标上，且两种指标在图像数量达到12张时便趋于饱和。值得注意的是，即便仅提供4张参考图像，我们的方法仍比配备16张参考图像的FS6D [^21]表现出更优的性能（见表1）。
训练数据规模规律。理论上，可以生成无限量的合成数据用于训练。图7展示了训练数据量对结果的影响，该结果通过YCB-Video数据集上ADD和ADD-S指标的AUC值来衡量。当数据量达到约1百万时，性能提升便趋于饱和。
运行时间。我们在Intel i9-10980XE CPU和NVIDIA RTX 3090 GPU硬件上测量了运行时间。对一个物体而言，姿态估计大约需要1.3秒，
Figure 6. Effects of number of reference images.
Figure 7. Effects of training data size.
其中姿态初始化耗时4毫秒，优化过程耗时0.88秒，姿态选择耗时0.42秒。跟踪过程的运行速度要快得多，可达$\sim$32 Hz，因为仅需进行姿态优化，且不存在多个姿态假设。实际应用中，我们可以先进行一次性姿态估计以完成初始化，随后切换至跟踪模式以实现实时性能。
## 5. 结论
我们提出了一种用于新物体6D姿态估计与跟踪的统一基础模型，它同时支持基于模型和无需模型两种设置。在4种不同任务组合上的大量实验表明，该模型不仅功能多样，而且性能远超为各任务专门设计的最新方法，甚至能达到那些需要实例级训练的方法所取得的效果。在未来的工作中，探索单个刚体之外的状态估计将会是一个有趣的方向。
---
## 参考文献
[^1]: Eric Brachmann, Alexander Krull, Frank Michel, Stefan Gumhold, Jamie Shotton, and Carsten Rother. Learning 6D object pose estimation using 3d object coordinates. In 13th European Conference on Computer Vision (ECCV), pages 536–551, 2014. 6, 7
[^2]: Dingding Cai, Janne Heikkilä, and Esa Rahtu. OVE6D: Object viewpoint encoding for depth-based 6D object pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6803–6813, 2022. 5, 7
[^3]: Ming Cai and Ian Reid. Reconstruct locally, localize globally: A model free method for object pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3153–3163, 2020. 2
[^4]: Tianshi Cao, Karsten Kreis, Sanja Fidler, Nicholas Sharp, and Kangxue Yin. TexFusion: Synthesizing 3D textures with text-guided image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 4169–4181, 2023. 2, 3
[^5]: Dengsheng Chen, Jun Li, Zheng Wang, and Kai Xu. Learning canonical shape space for category-level 6D object pose and size estimation. In Proceedings of the IEEE International Conference on Computer Vision (CVPR), pages 11973–11982, 2020. 1, 2
[^6]: Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3D objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13142–13153, 2023. 2
[^7]: Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 248–255, 2009. 3
[^8]: Xinke Deng, Arsalan Mousavian, Yu Xiang, Fei Xia, Timothy Bretl, and Dieter Fox. PoseRBPF: A Rao-Blackwellized particle filter for 6D object pose tracking. In Robotics: Science and Systems (RSS), 2019. 1, 2, 7, 8
[^9]: Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth $16 \times 16$ words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR), 2021. 5
[^10]: Laura Downs, Anthony Francis, Nate Koenig, Brandon Kinman, Ryan Hickman, Krista Reymann, Thomas B McHugh, and Vincent Vanhoucke. Google scanned objects: A high-quality dataset of 3D scanned household items. In International Conference on Robotics and Automation (ICRA), pages 2553–2560, 2022. 2
[^11]: Mathieu Garon, Denis Laurendeau, and Jean-François Lalonde. A framework for evaluating 6-dof object trackers. In Proceedings of the European Conference on Computer Vision (ECCV), pages 582–597, 2018. 2
[^12]: Amos Gropp, Lior Yariv, Niv Haim, Matan Atzmon, and Yaron Lipman. Implicit geometric regularization for learning shapes. In International Conference on Machine Learning (ICML), pages 3789–3799, 2020. 4
[^13]: Agrim Gupta, Piotr Dollar, and Ross Girshick. LVIS: A dataset for large vocabulary instance segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5356–5364, 2019. 2
[^14]: John C Hart. Sphere tracing: A geometric method for the antialiased ray tracing of implicit surfaces. The Visual Computer, 12(10):527–545, 1996. 4
[^15]: Rasmus Laurvig Haugaard and Anders Glent Buch. Surfemb: Dense and continuous correspondence distributions for object pose estimation with learnt surface embeddings. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6749–6758, 2022. 7
[^16]: Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016. 5
[^17]: Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask R-CNN. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2961–2969, 2017. 5, 7
[^18]: Xingyi He, Jiaming Sun, Yuang Wang, Di Huang, Hujun Bao, and Xiaowei Zhou. OnePose++: Keypoint-free one-shot object pose estimation without CAD models. Advances in Neural Information Processing Systems (NeurIPS), 35:35103–35115, 2022. 1,2,6,7
[^19]: Yisheng He, Wei Sun, Haibin Huang, Jianran Liu, Haoqiang Fan, and Jian Sun. PVN3D: A deep point-wise 3D keypoints voting network for 6DoF pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11632–11641, 2020. 1,2
[^20]: Yisheng He, Haibin Huang, Haoqiang Fan, Qifeng Chen, and Jian Sun. FFB6D: A full flow bidirectional fusion network for 6D pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3003–3013, 2021. 1, 2
[^21]: Yisheng He, Yao Wang, Haoqiang Fan, Jian Sun, and Qifeng Chen. FS6D: Few-shot 6D pose estimation of novel objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6814–6824, 2022. 2, 3, 6, 7, 8
[^22]: Stefan Hinterstoisser, Stefan Holzer, Cedric Cagniart, Slobodan Ilic, Kurt Konolige, Nassir Navab, and Vincent Lepetit. Multimodal templates for real-time detection of texture-less objects in heavily cluttered scenes. In International Conference on Computer Vision (ICCV), pages 858–865, 2011. 6
[^23]: Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems (NeurIPS), 33:6840–6851, 2020. 2
## [^24]: Tomáš Hodan, Pavel Haluza, Štepán Obdržálek, Jiri Matas, Manolis Lourakis, and Xenophon Zabulis. T-LESS: An RGB-D dataset for 6D pose estimation of texture-less objects. In IEEE Winter Conference on Applications of Computer Vision (WACV), pages 880–888, 2017. 6, 7
---
[^25]: Tomas Hodan, Frank Michel, Eric Brachmann, Wadim Kehl, Anders GlentBuch, Dirk Kraft, Bertram Drost, Joel Vidal, Stephan Ihrke, Xenophon Zabulis, et al. BOP: Benchmark for 6D object pose estimation. In Proceedings of the European Conference on Computer Vision (ECCV), pages 19–34, 2018. 2, 6
[^26]: Elad Hoffer and Nir Ailon. Deep metric learning using triplet network. In Third International Workshop on Similarity-Based Pattern Recognition (SIMBAD), pages 84–92, 2015. 6
[^27]: Shengyu Huang, Zan Gojcic, Mikhail Usvyatsov, Andreas Wieser, and Konrad Schindler. PREDATOR: Registration of 3D point clouds with low overlap. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4267–4276, 2021. 6
[^28]: Jan Issac, Manuel Wüthrich, Cristina Garcia Cifuentes, Jeannette Bohg, Sebastian Trimpe, and Stefan Schaal. Depth-based object tracking using a robust gaussian filter. In IEEE International Conference on Robotics and Automation (ICRA), pages 608–615, 2016. 1, 2, 7, 8
[^29]: Daniel Kappler, Franziska Meier, Jan Issac, Jim Mainprice, Cristina Garcia Cifuentes, Manuel Wüthrich, Vincent Berenz, Stefan Schaal, Nathan Ratliff, and Jeannette Bohg. Real-time perception meets reactive motion generation. IEEE Robotics and Automation Letters, 3(3):1864–1871, 2018.
[^30]: Yann Labbé, Justin Carpentier, Mathieu Aubry, and Josef Sivic. CosyPose: Consistent multi-view multi-object 6D pose estimation. In European Conference on Computer Vision (ECCV), pages 574–591, 2020. 1,2
[^31]: Yann Labbé, Lucas Manuelli, Arsalan Mousavian, Stephen Tyree, Stan Birchfield, Jonathan Tremblay, Justin Carpentier, Mathieu Aubry, Dieter Fox, and Josef Sivic. MegaPose: 6D pose estimation of novel objects via render & compare. In 6th Annual Conference on Robot Learning (CoRL), 2022. 1, 2, 5, 7
[^32]: Taeyeop Lee, Jonathan Tremblay, Valts Blukis, Bowen Wen, Byeong-Uk Lee, Inkyu Shin, Stan Birchfield, In So Kweon, and Kuk-Jin Yoon. TTA-COPE: Test-time adaptation for category-level object pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 21285–21295, 2023. 1,2
[^33]: Fu Li, Shishir Reddy Vutukur, Hao Yu, Ivan Shugurov, Benjamin Busam, Shaowu Yang, and Slobodan Ilic. NeRF-Pose: A first-reconstruct-then-regress approach for weakly-supervised 6D object pose estimation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 2123–2133, 2023. 2
[^34]: Yi Li, Gu Wang, Xiangyang Ji, Yu Xiang, and Dieter Fox. DeepIM: Deep iterative matching for 6D pose estimation. In Proceedings of the European Conference on Computer Vision (ECCV), pages 683–698, 2018. 1, 2, 7, 8
[^35]: Zhigang Li, Gu Wang, and Xiangyang Ji. CDPN: Coordinates-based disentangled pose network for real-time RGB-based 6-DoF object pose estimation. In CVF International Conference on Computer Vision (ICCV), pages 7677–7686, 2019. 2
[^36]: Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Common objects in context. In 13th European Conference on Computer Vision (ECCV), pages 740–755, 2014. 3
[^37]: Yunzhi Lin, Jonathan Tremblay, Stephen Tyree, Patricio A Vela, and Stan Birchfield. Keypoint-based category-level object pose tracking from an RGB sequence with uncertainty estimation. In International Conference on Robotics and Automation (ICRA), 2022. 1, 2
[^38]: Yuan Liu, Yilin Wen, Sida Peng, Cheng Lin, Xiaoxiao Long, Taku Komura, and Wenping Wang. Gen6D: Generalizable model-free 6-DoF object pose estimation from RGB images. ECCV, 2022. 1, 2, 6, 7
[^39]: William E Lorensen and Harvey E Cline. Marching cubes: A high resolution 3d surface construction algorithm. In Seminal graphics: pioneering efforts that shaped the field, pages 347–353. 1998. 4
[^40]: Eric Marchand, Hideaki Uchiyama, and Fabien Spindler. Pose estimation for augmented reality: A hands-on survey. IEEE Transactions on Visualization and Computer Graphics (TVCG), 22(12):2633–2651, 2015. 1
[^41]: Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99–106, 2021.
[^42]: Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM Trans. Graph., 41(4):102:1–102:15, 2022. 4
[^43]: Van Nguyen Nguyen, Yinlin Hu, Yang Xiao, Mathieu Salzmann, and Vincent Lepetit. Templates for 3D object pose estimation revisited: Generalization to new objects and robustness to occlusions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6771–6780, 2022. 5, 6, 7
[^44]: Van Nguyen Nguyen, Thibault Groueix, Georgy Ponimatkin, Vincent Lepetit, and Tomas Hodan. Cnos: A strong baseline for cad-based novel object segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2134–2140, 2023. 5
[^45]: Brian Okorn, Qiao Gu, Martial Hebert, and David Held. Zephyr: Zero-shot pose hypothesis rating. In IEEE International Conference on Robotics and Automation (ICRA), pages 14141–14148, 2021. 7
[^46]: Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018. 6
[^47]: Kiru Park, Timothy Patten, and Markus Vincze. Pix2Pose: Pixel-wise coordinate regression of objects for 6D pose estimation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 7668–7677, 2019. 1, 2
## [^48]: Keunhong Park, Arsalan Mousavian, Yu Xiang, and Dieter Fox. LatentFusion: End-to-end differentiable reconstruction and rendering for unseen object pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and
---
模式识别国际会议（CVPR），第10710–10719页，2020年。2, 7
[^49]: Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10684–10695, 2022. 2
[^50]: Ivan Shugurov, Fu Li, Benjamin Busam, and Slobodan Ilic. OSOP: A multi-stage one shot object pose estimation framework. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6835–6844, 2022. 1, 2, 7
[^51]: Manuel Stoiber, Martin Sundermeyer, and Rudolph Triebel. Iterative corresponding geometry: Fusing region and depth for highly efficient 3D tracking of textureless objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6855–6865, 2022. 1, 2, 7, 8
[^52]: Jiaming Sun, Zehong Shen, Yuang Wang, Hujun Bao, and Xiaowei Zhou. LoFTR: Detector-free local feature matching with transformers. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8922–8931, 2021. 6
[^53]: Jiaming Sun, Zihao Wang, Siyu Zhang, Xingyi He, Hongcheng Zhao, Guofeng Zhang, and Xiaowei Zhou. OnePose: One-shot object pose estimation without CAD models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6825–6834, 2022. 1, 2, 6, 7
[^54]: Meng Tian, Marcelo H Ang, and Gim Hee Lee. Shape prior deformation for categorical 6D object pose and size estimation. In Proceedings of the European Conference on Computer Vision (ECCV), pages 530–546, 2020. 1, 2
[^55]: Jonathan Tremblay, Thang To, Balakumar Sundaralingam, Yu Xiang, Dieter Fox, and Stan Birchfield. Deep object pose estimation for semantic robotic grasping of household objects. In Conference on Robot Learning (CoRL), pages 306–316, 2018. 2
[^56]: Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems (NeurIPS), 30, 2017. 5, 6
[^57]: Chen Wang, Roberto Martín-Martín, Danfei Xu, Jun Lv, Cewu Lu, Li Fei-Fei, Silvio Savarese, and Yuke Zhu. 6-PACK: Category-level 6D pose tracker with anchor-based keypoints. In IEEE International Conference on Robotics and Automation (ICRA), pages 10059–10066, 2020. 1,2
[^58]: He Wang, Srinath Sridhar, Jingwei Huang, Julien Valentin, Shuran Song, and Leonidas J Guibas. Normalized object coordinate space for category-level 6D object pose and size estimation. In Proceedings of the IEEE International Conference on Computer Vision (CVPR), pages 2642–2651, 2019. 1,2
[^59]: Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. NeuS: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. In Advances in Neural Information Processing Systems (NeurIPS), 2021. 4
[^60]: Bowen Wen and Kostas Bekris. BundleTrack: 6D pose tracking for novel objects without instance or category-level 3D models. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 8067–8074, 2021. 2, 7
[^61]: Bowen Wen, Chaitanya Mitash, Baozhang Ren, and Kostas E Bekris. se(3)-TrackNet: Data-driven 6D pose tracking by calibrating image residuals in synthetic domains. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 10367–10373, 2020. 1, 2, 6, 7, 8
[^62]: Bowen Wen, Chaitanya Mitash, Sruthi Soorian, Andrew Kimmel, Avishai Sintov, and Kostas E Bekris. Robust, occlusion-aware pose estimation for objects grasped by adaptive hands. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 6210–6217. IEEE, 2020. 1
[^63]: Bowen Wen, Wenzhao Lian, Kostas Bekris, and Stefan Schaal. CatGrasp: Learning category-level task-relevant grasping in clutter from simulation. In International Conference on Robotics and Automation (ICRA), pages 6401–6408, 2022. 1
[^64]: Bowen Wen, Wenzhao Lian, Kostas Bekris, and Stefan Schaal. You only demonstrate once: Category-level manipulation from single visual demonstration. RSS, 2022. 1
[^65]: Bowen Wen, Jonathan Tremblay, Valts Blukis, Stephen Tyree, Thomas Müller, Alex Evans, Dieter Fox, Jan Kautz, and Stan Birchfield. BundleSDF: Neural 6-DoF tracking and 3D reconstruction of unknown objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 606–617, 2023. 2, 4, 7
[^66]: Manuel Wüthrich, Peter Pastor, Mrinal Kalakrishnan, Jeannette Bohg, and Stefan Schaal. Probabilistic object tracking using a range camera. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 3195–3202, 2013. 1, 2, 7, 8
[^67]: Yu Xiang, Tanner Schmidt, Venkatraman Narayanan, and Dieter Fox. PoseCNN: A convolutional neural network for 6D object pose estimation in cluttered scenes. In Robotics: Science and Systems (RSS), 2018. 2, 6, 7
[^68]: Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan Atzmon, Basri Ronen, and Yaron Lipman. Multiview neural surface reconstruction by disentangling geometry and appearance. Advances in Neural Information Processing Systems (NeurIPS), 33:2492–2502, 2020. 4
[^69]: Ruida Zhang, Yan Di, Fabian Manhardt, Federico Tombari, and Xiangyang Ji. SSP-Pose: Symmetry-aware shape prior deformation for direct category-level object pose estimation. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 7452–7459, 2022. 1, 2
[^70]: Heng Zhao, Shenxing Wei, Dahu Shi, Wenming Tan, Zheyang Li, Ye Ren, Xing Wei, Yi Yang, and Shiliang Pu. Learning symmetry-aware geometry correspondences for 6D object pose estimation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 14045–14054, 2023. 7
## [^71]: Linfang Zheng, Chen Wang, Yinghan Sun, Esha Dasgupta, Hua Chen, Aleš Leonardis, Wei Zhang, and Hyung Jin
---
Chang. HS-Pose：用于类别级物体姿态估计的混合范围特征提取方法。载于IEEE/CVF计算机视觉与模式识别会议（CVPR）论文集，第17163–17173页，2023年。2
[^72]: Yi Zhou, Connelly Barnes, Jingwan Lu, Jimei Yang, and Hao Li. On the continuity of rotation representations in neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5745–5753, 2019. 5