# FUNREC：从以自我为中心的互动视频中重建功能性3D场景

![Image](images/4496885ab079697aec9445dae6e1a5d2a7412e8a7c4bc4043cfda427e4a9b3f3.jpg)

<div style="text-align: center;">Figure 1. Real-world functional digital twins. FUN(REC) takes a single egocentric RGB-D interaction video (top) and reconstructs a functional 3D digital twin of the environment (middle). The system automatically identifies articulated scene components, estimates their kinematic parameters along with per-timestep poses, and jointly reconstructs the static scene and each movable part, including interiors (see left and right). The final output is a simulation-compatible 3D scene representation with fully interactable articulated elements.</div>

## 摘要

我们提出了FunREC，这是一种直接从以自我为中心的RGB-D互动视频中重建室内场景的功能性3D数字孪生的方法。与现有的依赖受控设置、多状态捕获或CAD先验的关节重建方法不同，FunREC直接利用自然场景中的人类互动序列来恢复可交互的3D场景。它能自动发现关节部件、估算其运动学参数、跟踪它们的3D运动，并在规范空间中重建静态和动态几何结构，最终生成可用于仿真的网格模型。在全新的真实和模拟基准测试中，FunREC的表现大幅优于以往的研究工作：零件分割任务的mIoU指标提升了多达50%，关节误差和姿态误差降低了5到10倍，重建精度也显著提高。我们还展示了其在URDF/USD导出用于仿真、手绘动作映射以及机器人-场景交互等方面的应用。我们的项目页面为：functionalscenes.github.io。

运动映射与机器人场景交互。我们的项目页面为：functionalscenes.github.io。

### 1. 引言

人类理解世界并非仅仅依靠观察，而是通过与之互动来实现。重建功能性3D环境——不仅要捕捉静态几何结构，还要还原物体的运动与关节活动方式——是计算机视觉、机器人技术和具身人工智能领域的核心目标。尽管近期3D重建技术以及大规模RGB-D数据集的发展[^3] [^6] [^7] [^24] [^57] [^58] [^61] [^74] [^77]推动了静态场景理解的进步[^43] [^45]，但这些数据集仅能反映每个环境的单一状态。它们无法捕捉人类互动下场景的变化（例如门的开合、抽屉的滑动、冰箱门的开启及内部显露），而这些对于需要在物理世界中感知、规划并行动的智能体而言至关重要。这种对“功能性”的认知缺失，仍是3D场景重建领域的一大局限。

近期已有研究着手解决这一问题，

---

但仍存在明显局限。MultiScan[^44]需要手动对齐同一房间在不同状态下的多次扫描结果（例如橱柜开/闭状态），才能标注出关节部件。SceneFun3D[^11]和Articulate3D[^18]则是在静态激光雷达扫描数据的基础上添加细粒度的功能与可操作属性标注。关于“数字孪生”的并行研究[^8] [^22] [^78]会寻找与静态重建结果相似的CAD模型，以此获得可模拟、可交互的对应物。虽然这些研究能够生成可交互的场景，但它们都依赖于耗时费力的多状态捕获、人工标注或代理重建过程，且与实际3D几何结构的关联度较低。在物体层面，多项研究[^16] [^20] [^26] [^29] [^37] [^39] [^42] [^69] [^72]能够从演示数据中恢复关节物体，但通常都假设存在受控的设置、固定的摄像头或已知的CAD模型。它们仅局限于以物体为中心或合成场景，没有一种方法能够提供端到端的解决方案，直接从真实的以自我为中心的互动视频中重建场景级、基于物理规律的数字孪生。

我们的动机在于：人类互动为功能性场景重建提供了最直接且丰富的监督信息。当人们移动并操纵周围环境时，以自我为中心的观察自然会揭示哪些部分会发生运动、围绕何种关节运动、哪些体积会被暴露出来，以及相关的可操作性。基于这一洞察，我们推出了FunREC系统——它能直接从单段以自我为中心的RGB-D交互视频中，重建出真实环境的连贯且可动的3D数字孪生模型。

FunREC能够自动检测可动部件、估算其运动参数、同步追踪场景与部件的运动，同时重建静态和动态几何结构，包括被遮挡的内部区域。最终生成的是物理一致性高且可交互的3D场景，其中可动组件可沿着推断出的运动轴持续被操控。FunREC的核心是一个无需训练、基于优化的流程，它将几何推理与基础模型提供的语义及运动先验相结合。该系统会将输入视频拆分成短片段，借助视频-语言模型识别交互行为，把运动轨迹聚类为可动组件，进而得出像素级精度的交互部件掩码。随后，系统会联合优化部件姿态和运动参数，并利用估算出的相机与部件姿态来融合TSDF重建结果，最终得到全局对齐的功能性数字孪生模型。

我们在HOI4D [^40]数据集上对FunREC进行了评估，该数据集记录了实验室中的单个物体交互场景；此外我们还引入了两个新的以自我为中心的交互数据集，它们模拟了更贴近现实世界的场景：RealFun4D包含来自四个国家60套公寓室内空间的351段真实交互视频；OmniFun4D则由在12个OmniGibson场景中渲染的127段照片级真实序列组成 [^13] [^56]。在这三个数据集上，FunREC在运动估计、分割及重建质量方面均优于现有最先进的基线方法。最后，我们还展示了该方法在URDF/USD导出用于仿真、可操作性映射以及机器人-场景交互等实际应用场景中的价值。

### 2. 相关工作

静态与交互式3D场景理解。近年来，各类3D场景理解任务取得了显著进展，例如分割[^31] [^52]或物体检测[^33] [^53]。这些进展在很大程度上得益于大规模3D数据集的问世[^3] [^6] [^7] [^77]。虽然这些数据集基于静态扫描数据，但部分数据集已开始纳入随时间变化的信息，比如RIO数据集中的家具重新布置[^61]，或是NSS数据集中的建筑工地结构演变[^59]。最近，该领域开始转向对3D场景中物体的关节运动与功能进行建模[^11] [^18] [^44]。MultiScan数据集[^44]通过两次扫描同一房间——一次物体闭合、一次物体打开——并手动匹配不同状态下的部件，从而对铰接物体部件（如柜门和抽屉）进行标注。SceneFun3D[^11]和Articulate3D[^18]则专注于理解物体的功能与可用性，它们基于高分辨率激光雷达扫描数据提供详细标注，能够捕捉旋钮、开关等细微的交互元素。然而，由于这些场景是静态的，无法直接观察到铰接物体的因果与运动学特性。另一相关研究方向旨在重构真实环境的数字孪生体[^8] [^22] [^78]，即通过检索与观测场景相似的合成模型来实现。这些方法旨在提供具备功能性的数字副本，但依赖的是基于相似度的替换，而非真实的物理交互。在本研究中，我们提出了一种互补范式，用于捕捉具备功能性、因果关系的3D场景副本。我们并非从静态几何结构中推断关节运动，而是直接根据观测到的交互行为来重构场景，并还原其功能与运动学特性。

3D铰接物体重建。对交互式场景的建模始于对其基本组成部分——物体本身的理解。已有大量研究聚焦于3D铰接物体[^38]。大多数方法[^26] [^37] [^42] [^69] [^71]通过采集物体在不同状态下的多视角RGB(-D)数据，利用这些信息推断关节参数并重建部件级几何结构，从而实现铰接特性的恢复。然而，这种多视角采集方式使得数据采集过程复杂且实用性受限。另一类研究则结合视觉与语言模型，通过生成可执行代码[^83]或从视觉输入中检索CAD模型[^34]来推断关节运动；还有一些研究[^16] [^20] [^23] [^25] [^39] [^60] [^73]仅基于单幅观测数据进行建模。不过，由于CAD模型检索不准确，或是所依赖的2D关节估计器是在小规模数据集上训练的，这些研究在现实场景中的泛化能力仍受到限制。最近，一类新的研究方法应运而生……

---

如今已经出现了一些能够从人类演示视频中同时捕捉几何信息与运动信息的方法[^29] [^72]。不过，这些方法依赖于用固定摄像机录制的演示数据，且需要预先扫描的3D物体模型，这限制了它们在无约束的真实场景中的应用。与我们研究同期，iTACO[^50]和VideoArtGS[^41]能够从动态视频中重建可活动物体的模型，但它们仅在以物体为中心的场景下运行，并未解决野外场景级别的重建问题。ArtiPoint[^70]能从场景层面的交互视频中估算关节轴，但无法还原物体的几何结构。

Tracking Interactions in Dynamic 3D Scenes. While several impactful datasets have been proposed for dynamic object interactions ranging from rigid [^1] [^32] [^40] [^62] to articulated objects [^14] [^30], they primarily operate in a controlled, table-top setting with objects artificially scattered in the scene. Consequently, their applicability to in-the-wild scenes, e.g., real apartments, remains limited. Recent progress in 6D object pose tracking [66–68] has enabled several works that track objects and scene changes from egocentric videos [^4] [^17] [^81] in real scenes. Yet, they either only track rigid changes, or assume a given 3D scene scan pre-labeled with part-wise motions. Several recent works have focused on monocular 4D reconstruction [^35] [^36] [^64] [^65] [^79] [^82]. Typically trained to reconstruct the dynamic point maps and camera trajectories in a feedforward fashion, such methods do not understand the scene and object semantics, let alone the articulation parameters of the individual objects. Furthermore, they often operate on short video sequences and struggle with depth inconsistencies and flickering artifacts in the presence of occlusions, which are routinely encountered during in-the-wild egocentric scene interactions. While most methods do not predict 3D motion tracks, the ones that do [^15] [^76] struggle in the presence of occlusions due to interactions. Meanwhile, 2D and 3D point trackers [^19] [^27] [^28] [^46] [^54] [^63] [^75] [^80] provide useful motion priors but can be noisy. How to leverage them to robustly track the evolution of scene states in real-world articulated 3D environments remains an open question. We propose a training-free approach that leverages the semantic and motion priors of foundational models to robustly discover functionalities and track scene states from casually captured egocentric interaction videos.

### 3. Proposed Method

问题表述。设 $\mathcal{V} = \{(I_i, D_i)\}_{i=1}^N$ 表示一个由 N 帧组成的 RGB-D 视频，其中 $I_i$ 和 $D_i$ 分别表示第 i 时刻的 RGB 图像和深度图，且相机内参已知。该场景由静态背景几何结构 $\mathcal{P}^s$ 以及 K 个铰接部件 $\{\mathcal{P}^{m_k}\}_{k=1}^K$ 组成。每个部件 $m_k$ 由一组铰接参数 $\phi_k$ 和一系列刚体变换 $\{T_i^{m_k}\}_{i=1}^N$ 来参数化，其中每个 $T_i^{m_k} \in \text{SE}(3)$ 负责将该部件从其基准坐标系映射到第 i 时刻的世界坐标系。我们定义了两种铰接类型：棱柱关节。对于抽屉等可滑动的物体部件，我们定义一个单位平移轴 $\mathbf{a} \in \mathbb{S}^2$ 以及第 i 帧处的标量位移 $\lambda_i \in \mathbb{R}$，其中 $\mathbb{S}^2$ 是单位球面。第 i 帧处的变换如下所示：

 $$ \begin{array}{r}{\mathcal{T}_{\mathrm{p r i s}}(\mathbf{a},\lambda_{i})=\left[\begin{matrix}{\boldsymbol{I}}&{\lambda_{i}\mathbf{a}}\\ {\mathbf{0}^{\top}}&{1}\\ \end{matrix}\right],}\end{array} $$

此处 I 表示 $3 \times 3$ 单位矩阵。

旋转关节。对于门等可旋转的物体部件，我们定义单位旋转轴 $\mathbf{a} \in \mathbb{S}^2$、位于该轴上且最靠近场景原点 $\mathbf{p} \in \mathbb{R}^3$ 的枢轴点 $(i.e., \mathbf{a}^\top \mathbf{p} = 0)$，以及第 i 帧处的旋转角度 $\theta_i \in \mathbb{S}^1$，其中 $\mathbb{S}^1$ 是单位圆。相应的变换为：

 $$ \mathcal{T}_{\mathrm{r e v}}(\mathbf{a},\mathbf{p},\theta_{i})=\left[\begin{matrix}{R(\mathbf{a},\theta_{i})}&{(I-R(\mathbf{a},\theta_{i}))\mathbf{p}}\\ {\mathbf{0}^{\top}}&{1}\\ \end{matrix}\right], $$

此处 $R(\mathbf{a}, \theta)$ 表示绕轴 a 旋转角度 $3 \times 3$ 的 $\theta$ 旋转矩阵。

因此，部件 $m_{k}$ 在第 i 时刻的姿态为：

 $$ T_{i}^{m_{k}}=\left\{\begin{aligned}&\mathcal{T}_{\mathrm{p r i s}}(\mathbf{a}_{k},\lambda_{i,k}),&&\text{if}m_{k}\text{is prismatic},\\ &\mathcal{T}_{\mathrm{r e v}}(\mathbf{a}_{k},\mathbf{p}_{k},\theta_{i,k}),&&\text{if}m_{k}\text{is revolute}.\end{aligned}\right. $$

#### 3.1. 片段构建

输入视频 $\mathcal{V}$ 被划分为时间上连续的片段 $\mathcal{V}_k$（见图 2），满足 $\mathcal{V} = \bigcup_{k=1}^{K} \mathcal{V}_k$。每个片段会通过视频语言模型（VLM）[^10] 自动归类为静态（无交互）或动态（与铰接部件有交互）。对于每个动态片段，VLM 还会预测其铰接类型 $\sigma_k \in \{\text{prismatic}, \text{revolute}\}$。每个片段会按照下述方式独立处理。

#### 3.2. 动态片段重建

相机姿态估计。我们利用 RoMA [^12] 从连续的 RGB-D 帧中提取密集的点对应关系。接着使用来自 VISOR [^9] 的每帧手部掩码和交互物体掩码对这些匹配结果进行过滤：位于手部掩码 $\{M_i^h\}$ 内的点会被丢弃，而位于交互物体掩码 $\{M_i^{obj}\}$ 内的点则会通过缩放其 RoMA 置信度分数来降低权重。每个保留下来的 2D 对应关系会被提升到 3D 空间，随后利用 SuperRANSAC [^2] 估算帧间的相对相机运动。片段级别的姿态图优化可确保相机姿态具有全局一致性 $\{T_i^c\}$。

稀疏 3D 轨迹。我们利用 TAPIP3D [^80] 直接在世界坐标系中获取稀疏的 3D 点轨迹。这会生成轨迹位置 $\tau \in \mathbb{R}^{T \times N \times 3}$ 以及每帧的可见性得分 $o \in \mathbb{R}^{T \times N}$，其中 T 表示被跟踪点的数量。为了捕捉交互过程中显露出来的表面，会在均匀的 2D 网格上定期初始化新的轨迹。在确定了稀疏 3D 轨迹之后，接下来我们将这些轨迹分配给发生交互的铰接部件。

---

![Image](images/e567bf65ee53308a9e48acc626e917a3a1e3be5fcd2f723e5cf5c0b14ed35df7.jpg)

<div style="text-align: center;">Figure 2. Method overview. Given an egocentric RGB-D interaction video, FunREC first divides it into static and dynamic fragments. For each dynamic fragment, it estimates camera poses, computes sparse 3D point trajectories, and clusters them into articulated components via articulation-aware motion modeling. The interacting part is then segmented to obtain dense masks and reconstructed together with the static scene. Part pose and articulation parameters are optimized jointly to yield consistent motion across frames. Finally, all reconstructed fragments are globally aligned to produce a coherent, functional 3D digital twin where articulated parts can be interactively manipulated.</div>

基于关节运动的运动聚类。设 $\tau_l = \{\tau_{l,i}\}_{i=1}^N$ 表示第 l 条轨迹的 3D 坐标。通过设定最大位移阈值 $\epsilon_s$，可剔除运动微乎其微的轨迹。对于剩余每条轨迹，我们会估算出与其一致的逐轨迹关节运动假设 $\hat{\phi}_l$，该假设需符合片段整体的关节运动类型 $\sigma_k$，具体表述如下：

 $$ \hat{\phi}_{l}=\left\{\begin{aligned}&=(\hat{\mathbf{a}}_{l},\{\hat{\lambda}_{l,i}\}),&&if\sigma_{k}=prismatic,\\ &(\hat{\mathbf{a}}_{l},\hat{\mathbf{p}}_{l},\{\hat{\theta}_{l,i}\}),&&if\sigma_{k}=revolute.\end{aligned}\right. $$

每条轨迹的运动可被拟合为 3D 直线（棱柱关节）或圆（旋转关节）。我们通过利用初始点 $\hat{\tau}_{l,i}$ 结合估算出的运动参数，来计算预测的轨迹位置 $\tau_{l,1}$，计算公式如下：

 $$ \begin{array}{r}{\hat{\tau}_{l,i}=\left\{\begin{aligned}&\mathcal{T}_{\mathrm{p r i s}}(\hat{\mathbf{a}}_{l},\hat{\lambda}_{l,i})\tau_{l,1},&&\text{i f p r i s m a t i c},\\ &\mathcal{T}_{\mathrm{r e v}}(\hat{\mathbf{a}}_{l},\hat{\mathbf{p}}_{l},\hat{\theta}_{l,i})\tau_{l,1},&&\text{i f r e v o l u t e}.\end{aligned}\right.}\end{array} $$

若一条轨迹的平均拟合误差满足特定条件，则该轨迹会被保留。

 $$ \frac{1}{|\tau_{l}|}\sum_{\tau_{l,i}\in\tau_{l}}\|\tau_{l,i}-\hat{\tau}_{l,i}\|_{2}<\epsilon_{f}. $$

通过此筛选条件的轨迹会借助 HDBSCAN 算法进行聚类 [^5]，聚类依据是其拟合后的关节参数（轴线、枢轴及运动模式）的相似性，最终形成的运动聚类可代表独立运动的部件。

为确定与操作部件对应的聚类，我们会将各聚类与 2D 交互证据进行对比。设 $C_{i}^{obj}$ 为第 i 帧处交互物体掩码 $\mathcal{M}_{i}^{obj}$ 所对应的置信度。对于每个聚类 $\gamma$，我们会计算其一致性得分，计算公式如下：

 $$ s_{\gamma}=\sum_{l\in\gamma}\sum_{i}o_{l,i}\cdot\mathcal{C}_{i}^{o b j}\cdot\mathbb{I}\Big[\pi(\tau_{l,i})\in\mathcal{M}_{i}^{o b j}\Big], $$

其中 $\pi(\cdot)$ 表示投影到图像平面的操作。得分最高的聚类 $s_{\gamma}$ 即被判定为交互部件，进而得到运动轨迹集合 $\tau^{m}$ 以及静态集合 $\tau^{s}$。

像素级对齐的部件分割。给定运动轨迹集合 $\tau^{m}$，我们的目标是在每一帧中生成该关节部件的密集、像素级对齐的分割掩码。由于 $\tau^{m}$ 较为稀疏且可能包含异常值，直接投影轨迹或调用分割模型均不可靠。为获得可靠的分割结果，我们将轨迹提供的几何证据与图像级的语义分组相结合。

我们首先在片段中均匀选取一组关键帧 $\{I_q\}_{q=1}^{Q}$。在每个关键帧 $I_q$ 上，我们运用 SAM 的自动掩码生成器来产生过分割结果 $G_q: \{1, \ldots, H\} \times \{1, \ldots, W\} \to \mathbb{N}$，其中 $G_q(p)$ 会将每个像素 p 归类到某个语义区域（掩码）假设中，H 和 W 分别表示图像的高度和宽度。静态与动态 3D 轨迹会借助对应的相机姿态 $T_q^c$ 被投影到关键帧上，分别记为 $\pi(\tau_q^m)$ 和 $\pi(\tau_q^s)$，此处 $\pi: \mathbb{R}^3 \to \mathbb{R}^2$ 即为投影函数。

对于 $G_{q}$ 中的每个语义区域 r，我们会统计其中包含的投影后动态轨迹与静态轨迹的数量：

 $$ n_{r}^{m}=\sum_{p\in r}\mathbb{I}\bigl[p\in\pi(\tau_{q}^{m})\bigr],\quad n_{r}^{s}=\sum_{p\in r}\mathbb{I}\bigl[p\in\pi(\tau_{q}^{s})\bigr], $$

---

其中 $\mathbb{I}$ 为一个指示函数。我们进一步计算运动比率，用以衡量当前区域是否包含运动部件，计算公式为 $\gamma_r = n_r^m / (n_r^m + n_r^s + \epsilon)$，此处 $\epsilon$ 用于避免除以 0 的情况。运动比率满足 $\gamma_r > \eta_m$ 的区域被归类为属于运动中的关节部件，其中 $\eta_m$ 为设定的阈值。由此可在每个选定的关键帧 q 上生成运动掩码 $\mathcal{M}_q^{sm}$，公式如下：

 $$ \mathcal{M}_{q}^{s m}(p)=\begin{cases}{1,}&{\mathrm{i f~}G_{q}(p)\in\{r\mid\gamma_{r}>\eta_{m}\},}\\ {0,}&{\mathrm{o t h e r w i s e}.}\\ \end{cases} $$

关键帧上的运动掩码 $\{\mathcal{M}_q^{sm}\}$ 可作为 SAM2 视频传播模块的输入提示 [^55]，进而生成时间上连贯的关节部件掩码序列 $\{\mathcal{M}_i^m\}_{i=1}^N$。这些掩码能够精准勾勒出整个片段中关节部件的轮廓，为后续步骤中的密集部件重建提供必要的像素级支持。

零部件姿态与关节参数估计。给定运动轨迹集合 $\tau^m = \{\tau_{l,i}^m\}$ 及其对应的可见性得分 $o^m = \{o_{l,i}^m\}$，我们的目标是恢复全局一致的零部件姿态 $\{T_i^m\}_{i=1}^N$ 以及描述该零部件在所有帧中运动的关节参数 $\phi^m$。每个姿态 $T_i^m \in \mathrm{SE}(3)$ 将零部件的基准坐标系映射到时间 i 时的世界坐标系。关节参数 $\phi^m$ 用于编码关节模型，对于平移运动而言定义为 $(\mathbf{a}, \{\lambda_i\})$，对于旋转运动则定义为 $(\mathbf{a}, \mathbf{p}, \{\theta_i\})$。

对于片段内的每对帧 $(i, j)$，我们会在 $\tau^m$ 中的可见点之间构建三维-三维对应关系，并利用 SupeRANSAC $T_{i \to j}^m \in \text{SE}(3)$ 估算零部件的相对变换，记为 [^2]。每个对应关系的权重由其可见性得分 $o_{l,i}^m \cdot o_{l,j}^m$ 的乘积决定。由此得到的相对变换集合会形成一个连接片段内所有零部件姿态 $\{T_i^m\}$ 的姿态图。

为了联合恢复零部件的绝对姿态和关节参数，我们最小化以下目标函数：

 $$ \begin{align*}\mathcal{L}(T^{m},L^{m},\phi^{m})&=\sum_{i}f(T^{m}_{i},T^{m}_{i+1},T^{m}_{i\rightarrow i+1})\\+\sum_{i,j}l^{m}_{ij}f(T^{m}_{i},T^{m}_{j},T^{m}_{i\rightarrow j})+\mu\sum_{i,j}(\sqrt{l^{m}_{ij}}-1)^{2},\end{align*} $$

其中 $l_{ij}^{m} \in [0,1]$ 是待优化的闭环置信度，$\mu$ 用于控制其正则化项。该项为

 $$ f(T_{i}^{m},T_{j}^{m},T_{i\to j}^{m})=e_{i j}^{\top}\Omega_{i j}e_{i j}, $$

其中 $e_{ij} = \log_{\mathrm{SE}(3)} \left( \left( T_{i}^{m} \right)^{-1} T_{j}^{m} T_{i \rightarrow j}^{m} \right)$ 用于衡量估算出的相对变换 $(T_{i}^{m})^{-1} T_{j}^{m}$ 与观测到的变换 $T_{i \rightarrow j}^{m}$ 之间的差异，该差异由信息矩阵 $\Omega_{ij}$ 加权。对数 $\log_{\mathrm{SE}(3)}(\cdot)$ 会将残差映射到 $\mathrm{SE}(3)$ 的切空间内。

关节参数 $\phi^m$ 是通过对观测到的轨迹 $\tau^m$ 进行最小二乘拟合来初始化的，从而得到关节轴和运动状态的初步估计。随后，$\{T_i^m\}$ 和 $\phi^m$ 会通过 Ceres Solver 进行非线性优化，以联合优化它们的值；优化过程中采用流形优化技术，确保关节参数始终位于问题定义中所规定的相应流形上（详见第 3 节）。

重建与交互式操作。利用逐像素分割掩码和估算出的姿态，我们使用两个独立的截断符号距离函数（TSDF）体素来重建几何模型：一个用于静态背景，另一个用于关节零部件。静态 TSDF 体素利用估算出的相机姿态 $\{T_i^c\}$ 在世界坐标系中积分，同时排除属于手部或运动零部件区域的动态区域，这些区域分别由二进制掩码 $\mathcal{M}_i^h$ 和 $\mathcal{M}_i^m$ 标识。关节零部件则在基准坐标系下重建。对于每一帧 i，深度图会先通过 $(T_i^m)^{-1}T_i^c$ 从相机坐标系变换到零部件坐标系，且仅融合属于运动零部件掩码 $\mathcal{M}_i^m$ 的像素。这样就能得到干净的基准三维模型 $\mathcal{P}^m$，该模型不受相机运动和遮挡的影响。两个 TSDF 体素融合后，便可提取出静态场景 $\mathcal{P}^s$ 和关节零部件 $\mathcal{P}^m$ 的网格。时间 i 时的完整场景可表示为：$\mathcal{P}_i = \mathcal{P}^s \cup T_i^m (\mathcal{P}^m)$，其中 $T_i^m(\mathcal{P}^m)$ 会将零部件的估算姿态应用到其基准几何形状上，从而将其置于全局坐标系中。为了支持交互式操作，我们会根据追踪到的运动轨迹估算关节参数的可行运动范围：$\lambda \in [\lambda_{\min}, \lambda_{\max}]$ 或 $\theta \in [\theta_{\min}, \theta_{\max}]$，从而能够在与关节参数 $\phi^m$ 一致的任何中间状态下渲染或物理模拟重建后的场景。

#### 3.3. 全局片段对齐

每个动态片段都会生成一个局部子图 $\mathcal{S}_k$，其中包含其重建后的静态几何结构以及可动部件。为了生成静态片段子图，我们仅执行相机位姿估计，因为场景本身是静止的。为了形成全局一致的场景，我们需要将所有子图 $\{\mathcal{S}_k\}_{k=1}^K$ 对齐到一个共享的坐标系中。对于每一对子图 $(\mathcal{S}_k, \mathcal{S}_k')$，我们会提取它们之间的几何对应关系，并利用 PREDATOR [^21] 来估算相对的刚体变换。只有当点云对齐的均方根误差（RMSE）低于预设阈值时，才会接受闭环检测的结果。这些相对变换构成了基于子图的场景级位姿图，我们对其进行优化以获得全局一致的子图位姿。最终，对齐后的子图会被融合成一个统一的 TSDF 体积，从中我们可以提取出完整的静态场景 $\mathcal{P}^s$ 以及所有可动部件的网格模型 $\{\mathcal{P}^m_k\}$。

### 4. 数据采集

HOI4D [^40] 是我们这项任务中唯一已有的数据集，但它仅包含低运动幅度、单一物体的交互数据，且不包含完整场景。因此，我们收集了两个新的第一人称视角数据集，这些数据集涵盖了真实场景和模拟场景中的多样化交互情况（见图 3）。

---

![Image](images/c02a380c316bf45744756b6bac8af697ae85ad3b48cba75f11a23fb26ffa6320.jpg)

<div style="text-align: center;">Figure 3. Proposed Datasets. We introduce two datasets for functional 3D scene reconstruction and evaluation: RealFun4D (left), capturing egocentric interactions in real scenes, and OmniFun4D (right), providing photorealistic simulated interactions in synthetic scenes.</div>

![Image](images/e00a2e6a399f0d1b2ffd790f88bb7aa2bd28b13eaa5bbd2c3907aa850f91fec9.jpg)

<div style="text-align: center;">Figure 4. Qualitative comparisons. We show qualitative comparisons between baselines and our method. For each method, we accumulate the reconstructed point clouds of both the articulated part and the static scene across all timesteps, and visualize them under two selected scene states. Green indicates the articulated part, and red lines denote the estimated articulation axes.</div>

真实场景：RealFun4D。我们的真实数据集包含了在四个国家的 60 套公寓中记录的 351 段自然场景下的人与场景交互序列。每段序列都是用头戴式 Azure Kinect DK 设备采集的（$1920 \times 1080$，15 FPS），可同步获取 RGB 图像和深度数据。我们对交互区间和文本描述进行了标注，还为每帧图像标注了用于动态区域过滤和静态场景重建的 2D 手部及部件掩码 [^40]，同时标记了关节位置以及每帧的 2D 部件轨迹，以便进行对齐和完整的 3D 部件重建。

模拟场景：OmniFun4D。我们还在 12 个源自 iGibson [^13] 的 OmniGibson [^56] 场景中记录了 127 段交互数据。由人类操作员在场景中导航并触发预设的交互动作；相机位姿和事件会被记录下来，随后离线回放以利用 NVIDIA RTX Path Tracing 技术 [^48] 渲染高质量的 RGB-D 图像和掩码。我们还对相机位姿添加了随机高斯扰动，以模拟自然的头部运动。更多细节可在补充材料中查看。

### 5. 实验

数据集。我们在三个数据集上评估我们的方法。前两个数据集 RealFun4D 和 OmniFun4D 如第 4 节所述是新采集的。为了与基线方法进行比较，我们利用这些数据集中标注的交互区间，构建了一个包含 60 段交互序列的多样化评估集。此外，我们还使用了 HOI4D 数据集 [^40]，该数据集包含短小的、单一物体的第一人称 RGB-D 交互视频。为了适配我们的研究场景，我们从中选取了涉及四类可动物体（“笔记本电脑”、“橱柜”、“保险箱”和“垃圾桶”）的 30 段交互数据，并对提供的 6D 部件位姿进行了后处理，从而获取真实的关节运动参数。

评估任务与指标。我们考虑四项任务：可动部件运动估计、6D 部件位姿估计、部件分割以及 3D 重建。对于可动部件运动估计，我们报告轴向方向和位置误差，以及每个时间步的关节状态误差，其中旋转关节和移动关节的结果会分别列出。此外，我们还包括了

---

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td rowspan="2">Methods</td><td colspan="4">OmniFun4D</td><td colspan="4">HOI4D</td><td colspan="4">RealFun4D</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>Axis ( $^{\circ}$)</td><td style='text-align: center; word-wrap: break-word;'>Pos (m)</td><td style='text-align: center; word-wrap: break-word;'>State ( $^{\circ}$/m)</td><td style='text-align: center; word-wrap: break-word;'>Fail (%)</td><td style='text-align: center; word-wrap: break-word;'>Axis ( $^{\circ}$)</td><td style='text-align: center; word-wrap: break-word;'>Pos (m)</td><td style='text-align: center; word-wrap: break-word;'>State ( $^{\circ}$/m)</td><td style='text-align: center; word-wrap: break-word;'>Fail (%)</td><td style='text-align: center; word-wrap: break-word;'>Axis ( $^{\circ}$)</td><td style='text-align: center; word-wrap: break-word;'>Pos (m)</td><td style='text-align: center; word-wrap: break-word;'>State ( $^{\circ}$/m)</td><td style='text-align: center; word-wrap: break-word;'>Fail (%)</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>MonST3R [^82] (ICP)</td><td style='text-align: center; word-wrap: break-word;'>52.7/61.5</td><td style='text-align: center; word-wrap: break-word;'>1.14</td><td style='text-align: center; word-wrap: break-word;'>66.2/0.27</td><td style='text-align: center; word-wrap: break-word;'>11.7</td><td style='text-align: center; word-wrap: break-word;'>67.3/66.1</td><td style='text-align: center; word-wrap: break-word;'>0.45</td><td style='text-align: center; word-wrap: break-word;'>60.8/0.06</td><td style='text-align: center; word-wrap: break-word;'>0.0</td><td style='text-align: center; word-wrap: break-word;'>54.6/59.1</td><td style='text-align: center; word-wrap: break-word;'>0.68</td><td style='text-align: center; word-wrap: break-word;'>65.9/0.21</td><td style='text-align: center; word-wrap: break-word;'>0.0</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>MonST3R (CoTr3)</td><td style='text-align: center; word-wrap: break-word;'>46.8/58.9</td><td style='text-align: center; word-wrap: break-word;'>1.20</td><td style='text-align: center; word-wrap: break-word;'>45.3/0.18</td><td style='text-align: center; word-wrap: break-word;'>11.7</td><td style='text-align: center; word-wrap: break-word;'>54.7/51.5</td><td style='text-align: center; word-wrap: break-word;'>0.44</td><td style='text-align: center; word-wrap: break-word;'>44.8/0.07</td><td style='text-align: center; word-wrap: break-word;'>0.0</td><td style='text-align: center; word-wrap: break-word;'>56.8/52.4</td><td style='text-align: center; word-wrap: break-word;'>0.70</td><td style='text-align: center; word-wrap: break-word;'>57.4/0.22</td><td style='text-align: center; word-wrap: break-word;'>0.0</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>MonST3R (GT depth+CoTr3)</td><td style='text-align: center; word-wrap: break-word;'>57.6/63.5</td><td style='text-align: center; word-wrap: break-word;'>1.10</td><td style='text-align: center; word-wrap: break-word;'>46.5/0.18</td><td style='text-align: center; word-wrap: break-word;'>11.7</td><td style='text-align: center; word-wrap: break-word;'>48.6/30.1</td><td style='text-align: center; word-wrap: break-word;'>0.41</td><td style='text-align: center; word-wrap: break-word;'>37.6/0.06</td><td style='text-align: center; word-wrap: break-word;'>0.0</td><td style='text-align: center; word-wrap: break-word;'>51.4/56.2</td><td style='text-align: center; word-wrap: break-word;'>0.55</td><td style='text-align: center; word-wrap: break-word;'>54.3/0.21</td><td style='text-align: center; word-wrap: break-word;'>0.0</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>SpatialTrackerV2 [^76]</td><td style='text-align: center; word-wrap: break-word;'>59.8/69.1</td><td style='text-align: center; word-wrap: break-word;'>1.22</td><td style='text-align: center; word-wrap: break-word;'>48.6/0.19</td><td style='text-align: center; word-wrap: break-word;'>30.0</td><td style='text-align: center; word-wrap: break-word;'>57.9/44.6</td><td style='text-align: center; word-wrap: break-word;'>0.46</td><td style='text-align: center; word-wrap: break-word;'>41.9/0.07</td><td style='text-align: center; word-wrap: break-word;'>0.0</td><td style='text-align: center; word-wrap: break-word;'>59.4/49.8</td><td style='text-align: center; word-wrap: break-word;'>0.49</td><td style='text-align: center; word-wrap: break-word;'>57.5/0.22</td><td style='text-align: center; word-wrap: break-word;'>20.0</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>SpatialTrackerV2 (GT depth)</td><td style='text-align: center; word-wrap: break-word;'>48.7/69.3</td><td style='text-align: center; word-wrap: break-word;'>1.10</td><td style='text-align: center; word-wrap: break-word;'>46.8/0.19</td><td style='text-align: center; word-wrap: break-word;'>30.0</td><td style='text-align: center; word-wrap: break-word;'>54.3/38.8</td><td style='text-align: center; word-wrap: break-word;'>0.43</td><td style='text-align: center; word-wrap: break-word;'>38.7/0.07</td><td style='text-align: center; word-wrap: break-word;'>0.0</td><td style='text-align: center; word-wrap: break-word;'>60.8/42.6</td><td style='text-align: center; word-wrap: break-word;'>0.58</td><td style='text-align: center; word-wrap: break-word;'>55.6/0.22</td><td style='text-align: center; word-wrap: break-word;'>20.0</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>BundleSDF [^67] (GT mask)</td><td style='text-align: center; word-wrap: break-word;'>38.2/55.9</td><td style='text-align: center; word-wrap: break-word;'>0.95</td><td style='text-align: center; word-wrap: break-word;'>23.4/0.20</td><td style='text-align: center; word-wrap: break-word;'>55.0</td><td style='text-align: center; word-wrap: break-word;'>26.3/24.5</td><td style='text-align: center; word-wrap: break-word;'>0.24</td><td style='text-align: center; word-wrap: break-word;'>9.1/0.07</td><td style='text-align: center; word-wrap: break-word;'>43.3</td><td style='text-align: center; word-wrap: break-word;'>32.0/52.1</td><td style='text-align: center; word-wrap: break-word;'>0.58</td><td style='text-align: center; word-wrap: break-word;'>15.6/0.22</td><td style='text-align: center; word-wrap: break-word;'>36.7</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>ArtGS [^42]</td><td style='text-align: center; word-wrap: break-word;'>64.3/23.4</td><td style='text-align: center; word-wrap: break-word;'>1.69</td><td style='text-align: center; word-wrap: break-word;'>/</td><td style='text-align: center; word-wrap: break-word;'>95.0</td><td style='text-align: center; word-wrap: break-word;'>60.5/53.2</td><td style='text-align: center; word-wrap: break-word;'>0.35</td><td style='text-align: center; word-wrap: break-word;'>/</td><td style='text-align: center; word-wrap: break-word;'>66.7</td><td style='text-align: center; word-wrap: break-word;'>70.1/18.2</td><td style='text-align: center; word-wrap: break-word;'>0.34</td><td style='text-align: center; word-wrap: break-word;'>/</td><td style='text-align: center; word-wrap: break-word;'>66.7</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>FunREC (Ours)</td><td style='text-align: center; word-wrap: break-word;'>5.3/5.4</td><td style='text-align: center; word-wrap: break-word;'>0.03</td><td style='text-align: center; word-wrap: break-word;'>5.0/0.02</td><td style='text-align: center; word-wrap: break-word;'>1.7</td><td style='text-align: center; word-wrap: break-word;'>12.4/1.3</td><td style='text-align: center; word-wrap: break-word;'>0.06</td><td style='text-align: center; word-wrap: break-word;'>9.1/0.02</td><td style='text-align: center; word-wrap: break-word;'>0.0</td><td style='text-align: center; word-wrap: break-word;'>5.6/5.5</td><td style='text-align: center; word-wrap: break-word;'>0.05</td><td style='text-align: center; word-wrap: break-word;'>8.4/0.03</td><td style='text-align: center; word-wrap: break-word;'>0.0</td></tr></table>

<div style="text-align: center;">Table 1. Articulated motion estimation. We report articulation axis direction error (°), axis position error (meters; applicable only to revolute joints), joint state error (° for revolute and meters for prismatic joints), and failure rate (%). Values are shown as “XX / YY” for revolute (XX) and prismatic (YY) joints. Best, second-best, and third-best results are highlighted. All metrics are lower-is-better.</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td rowspan="3">Methods</td><td colspan="3">OmniFun4D</td><td colspan="3">HOI4D</td><td colspan="3">RealFun4D</td></tr><tr><td colspan="2">6D part pose</td><td style='text-align: center; word-wrap: break-word;'>Reconstruction</td><td colspan="2">6D part pose</td><td style='text-align: center; word-wrap: break-word;'>Reconstruction</td><td colspan="2">6D part pose</td><td style='text-align: center; word-wrap: break-word;'>Reconstruction</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>ADD-S (\%) $\uparrow$</td><td style='text-align: center; word-wrap: break-word;'>ADD (\%) $\uparrow$</td><td style='text-align: center; word-wrap: break-word;'>CD (cm) $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>ADD-S (\%) $\uparrow$</td><td style='text-align: center; word-wrap: break-word;'>ADD (\%) $\uparrow$</td><td style='text-align: center; word-wrap: break-word;'>CD (cm) $\downarrow$</td><td style='text-align: center; word-wrap: break-word;'>ADD-S (\%) $\uparrow$</td><td style='text-align: center; word-wrap: break-word;'>ADD (\%) $\uparrow$</td><td style='text-align: center; word-wrap: break-word;'>CD (cm) $\downarrow$</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>MonST3R [^82] (ICP)</td><td style='text-align: center; word-wrap: break-word;'>14.36</td><td style='text-align: center; word-wrap: break-word;'>9.62</td><td style='text-align: center; word-wrap: break-word;'>29.8</td><td style='text-align: center; word-wrap: break-word;'>29.41</td><td style='text-align: center; word-wrap: break-word;'>19.12</td><td style='text-align: center; word-wrap: break-word;'>6.4</td><td style='text-align: center; word-wrap: break-word;'>13.21</td><td style='text-align: center; word-wrap: break-word;'>9.11</td><td style='text-align: center; word-wrap: break-word;'>13.4</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>MonST3R [^82] (CoTr3)</td><td style='text-align: center; word-wrap: break-word;'>33.58</td><td style='text-align: center; word-wrap: break-word;'>21.94</td><td style='text-align: center; word-wrap: break-word;'>23.6</td><td style='text-align: center; word-wrap: break-word;'>44.72</td><td style='text-align: center; word-wrap: break-word;'>25.98</td><td style='text-align: center; word-wrap: break-word;'>2.1</td><td style='text-align: center; word-wrap: break-word;'>18.64</td><td style='text-align: center; word-wrap: break-word;'>11.35</td><td style='text-align: center; word-wrap: break-word;'>17.6</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>MonST3R [^82] (GT depth+CoTr3)</td><td style='text-align: center; word-wrap: break-word;'>37.12</td><td style='text-align: center; word-wrap: break-word;'>30.48</td><td style='text-align: center; word-wrap: break-word;'>13.9</td><td style='text-align: center; word-wrap: break-word;'>54.83</td><td style='text-align: center; word-wrap: break-word;'>37.11</td><td style='text-align: center; word-wrap: break-word;'>1.3</td><td style='text-align: center; word-wrap: break-word;'>20.11</td><td style='text-align: center; word-wrap: break-word;'>14.62</td><td style='text-align: center; word-wrap: break-word;'>13.2</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>SpatialTrackerV2 [^76]</td><td style='text-align: center; word-wrap: break-word;'>26.94</td><td style='text-align: center; word-wrap: break-word;'>13.11</td><td style='text-align: center; word-wrap: break-word;'>34.7</td><td style='text-align: center; word-wrap: break-word;'>61.25</td><td style='text-align: center; word-wrap: break-word;'>41.02</td><td style='text-align: center; word-wrap: break-word;'>0.9</td><td style='text-align: center; word-wrap: break-word;'>19.77</td><td style='text-align: center; word-wrap: break-word;'>13.92</td><td style='text-align: center; word-wrap: break-word;'>18.5</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>SpatialTrackerV2 [^76] (GT depth)</td><td style='text-align: center; word-wrap: break-word;'>29.71</td><td style='text-align: center; word-wrap: break-word;'>15.43</td><td style='text-align: center; word-wrap: break-word;'>9.88</td><td style='text-align: center; word-wrap: break-word;'>60.98</td><td style='text-align: center; word-wrap: break-word;'>40.21</td><td style='text-align: center; word-wrap: break-word;'>0.8</td><td style='text-align: center; word-wrap: break-word;'>22.94</td><td style='text-align: center; word-wrap: break-word;'>15.22</td><td style='text-align: center; word-wrap: break-word;'>13.5</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>BundleSDF [^67] (GT mask)</td><td style='text-align: center; word-wrap: break-word;'>22.84</td><td style='text-align: center; word-wrap: break-word;'>12.37</td><td style='text-align: center; word-wrap: break-word;'>17.1</td><td style='text-align: center; word-wrap: break-word;'>53.12</td><td style='text-align: center; word-wrap: break-word;'>35.22</td><td style='text-align: center; word-wrap: break-word;'>1.4</td><td style='text-align: center; word-wrap: break-word;'>20.88</td><td style='text-align: center; word-wrap: break-word;'>12.91</td><td style='text-align: center; word-wrap: break-word;'>10.6</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>ArtGS [^42]</td><td style='text-align: center; word-wrap: break-word;'>-</td><td style='text-align: center; word-wrap: break-word;'>-</td><td style='text-align: center; word-wrap: break-word;'>20.3</td><td style='text-align: center; word-wrap: break-word;'>-</td><td style='text-align: center; word-wrap: break-word;'>-</td><td style='text-align: center; word-wrap: break-word;'>6.8</td><td style='text-align: center; word-wrap: break-word;'>-</td><td style='text-align: center; word-wrap: break-word;'>-</td><td style='text-align: center; word-wrap: break-word;'>20.6</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>FunREC (Ours)</td><td style='text-align: center; word-wrap: break-word;'>78.96</td><td style='text-align: center; word-wrap: break-word;'>71.28</td><td style='text-align: center; word-wrap: break-word;'>3.2</td><td style='text-align: center; word-wrap: break-word;'>79.43</td><td style='text-align: center; word-wrap: break-word;'>69.85</td><td style='text-align: center; word-wrap: break-word;'>0.7</td><td style='text-align: center; word-wrap: break-word;'>75.62</td><td style='text-align: center; word-wrap: break-word;'>68.11</td><td style='text-align: center; word-wrap: break-word;'>6.1</td></tr></table>

<div style="text-align: center;">Table 2. 6D part pose estimation and reconstruction. We report 6D part pose accuracy using ADD-S and ADD (higher is better) and surface reconstruction quality using Chamfer Distance (CD, lower is better) across the OmniFun4D, HOI4D, and RealFun4D datasets.</div>

<table border=1 style='margin: auto; word-wrap: break-word;'><tr><td rowspan="2">Methods</td><td style='text-align: center; word-wrap: break-word;'>OmniFun4D</td><td style='text-align: center; word-wrap: break-word;'>HOI4D</td><td style='text-align: center; word-wrap: break-word;'>RealFun4D</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>mIoU $\uparrow$</td><td style='text-align: center; word-wrap: break-word;'>mIoU $\uparrow$</td><td style='text-align: center; word-wrap: break-word;'>mIoU $\uparrow$</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>MonST3R [^82]</td><td style='text-align: center; word-wrap: break-word;'>23.6</td><td style='text-align: center; word-wrap: break-word;'>26.8</td><td style='text-align: center; word-wrap: break-word;'>23.7</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>SpatialTrackerV2 [^76] (SAM2)</td><td style='text-align: center; word-wrap: break-word;'>6.2</td><td style='text-align: center; word-wrap: break-word;'>5.8</td><td style='text-align: center; word-wrap: break-word;'>13.4</td></tr><tr><td style='text-align: center; word-wrap: break-word;'>FunREC (Ours)</td><td style='text-align: center; word-wrap: break-word;'>77.9</td><td style='text-align: center; word-wrap: break-word;'>76.4</td><td style='text-align: center; word-wrap: break-word;'>74.8</td></tr></table>

<div style="text-align: center;">Table 3. Moving part segmentation. Mean Intersection-over-Union (mIoU) for moving-part segmentation.</div>

故障率是指该方法未能成功处理或预测出错误关节类型的视频所占比例。对于6D零件姿态估计，我们采用ADD-S和ADD指标进行标准评估 [^66] [^67]；而3D表面重建质量则通过倒角距离（Chamfer Distance, CD）来衡量。对于运动部件的分割任务，我们采用平均交并比（mean Intersection-over-Union, mIoU）作为评价指标。

基准方法。我们选取了三类具有代表性的方法进行比较：(类型1) 4D重建流程（MonST3R [^82]、SpatialTrackerV2 [^76]）；(类型2) 6D物体姿态跟踪方法（BundleSDF [^67]）；(类型3) 铰接物体重建方法（ArtGS [^42]）。对于MonST3R，我们集成了ICP和CoTracker3 [^28]以实现运动部件的姿态跟踪；SpatialTrackerV2则借助SAM2来获取部件分割结果。对于这两类基准方法，我们将动态区域或轨迹提升至3D空间，并通过基于RANSAC的模型拟合来估算部件姿态，必要时还会利用真实深度数据以确保比较的公平性。BundleSDF则被赋予真实的相机姿态和分割掩码，用以恢复零件的规范几何形状及每帧的部件变换参数。ArtGS仅能在两种静态铰接状态下工作，无法跟踪视频中的运动，因此我们只能报告其静态重建质量以及利用真实相机姿态估算出的铰接参数。更多细节详见补充材料。

利用真实相机姿态来估算铰接参数。更多细节详见补充材料。铰接运动估计方面，相关结果见表1，可视化示例见图4。FunREC在所有数据集和运动类型上均取得了最小的误差。在OmniFun4D数据集上，它估算出的铰接轴方向误差不超过$5.3^{\circ}$，位置误差不超过$0.03 \, m$，其性能远超BundleSDF [^67]，误差降低幅度超过$30^{\circ}$，距离误差更是缩小了一个数量级。在HOI4D [^40]和RealFun4D数据集上同样呈现此趋势：FunREC将状态预测误差分别降至$9.1^{\circ}/0.02 \, m$和$8.4^{\circ}/0.03 \, m$，且故障率仅为$0\%$。相比之下，现有的动态重建流程 [^76] [^82] 以及铰接建模方法 [^42] 则存在较大偏差且频繁出现故障。这些结果凸显了FunREC直接从第一人称视角视频中恢复物理上一致的关节参数的可靠性。

运动部件分割方面，相关结果见表3。在所有数据集上，FunREC均取得了最高的分割精度：在OmniFun4D、HOI4D和RealFun4D上的mIoU分数分别为77.9、76.4和74.8。这一结果相比以往的方法有显著提升，包括MonST3R [^82]（23.6–26.8 mIoU）和SpatialTrackerV2 [^76]（5.8–13.4 mIoU），表明FunREC能够在合成数据、受控环境以及真实场景中准确划分运动部件。

6D零件姿态估计方面，表2汇总了通过ADD-S和ADD指标衡量的6D零件姿态精度。在所有数据集上，FunREC均表现最佳：在OmniFun4D上，ADD-S达到79.0%，ADD达到71.3%；

---

在HOI4D上，ADD-S为79.4%，ADD为69.9%；在RealFun4D上，ADD-S为75.6%，ADD为68.1%。这些数值相比BundleSDF [^67]提升了两倍多，也明显优于动态跟踪器 [^76] [^82]。上述改进表明，所提出的零件姿态与铰接参数联合优化机制能够生成稳定且时间上一致的运动轨迹。

可动结构重建。最终，FunREC成功实现了高保真度的可动结构3D重建，如表2中的倒角距离（CD）所示。在OmniFun4D数据集上，其CD值达到3.2厘米；在HOI4D数据集上为0.7厘米；在RealFun4D数据集上则为6.1厘米，性能显著优于所有基准方法。图4中的示例表明，FunREC不仅能重建静态几何结构，还能准确还原可动部件的运动范围，从而支持在3D环境中对物体间的交互进行持续模拟。总体而言，这些结果证实：我们这种无需训练、基于优化的方法能够适用于合成数据、受控环境以及真实世界的以眼为中心的视频记录，生成物理一致性高且具备交互性的数字孪生模型。

### 6. 应用场景

可用于模拟的导出。利用重建后的几何结构与可动参数信息，我们能够生成文件，从而实现真实世界场景的交互式数字复制品，以便进行物理模拟。具体而言，我们利用可动参数来定义静态结构与可动部件之间的旋转关节与平移关节，并将其导出为URDF或USD格式。通过调用视觉-语言模型（如GPT-5），可从RGB图像中推断出质量、惯性等物理属性。这一过程使得重建后的场景可直接加载到物理模拟器中，进而支持机器人-场景交互等各类下游任务。图5展示了在Isaac Sim中的示例[^47]：模拟的机械臂与通过FunREC从真实扫描数据重建的场景进行交互。

![Image](images/2bad33d9a076243f7f44a694f12ae6cd7341fcff25cc91a29f439ef537a706e0.jpg)

![Image](images/92679030210b1e9416292899e5750d94a2a8d7937da99b1b28712471109c9494.jpg)

<div style="text-align: center;">Figure 5. Isaac Sim deployment. A mobile manipulator interacts with a drawer reconstructed from a real-world scan.</div>

手部引导的功能性映射。如图6所示，我们的框架可自然扩展以纳入以手为中心的功能性信息。通过现成的估算器提取3D手部网格[^49] [^51]，并将其对齐到我们的功能性数字孪生空间中，我们便能确定手部在3D空间中的位置，并还原出其在物体上的接触区域。这实现了对手部运动与被交互物体部件运动的联合推理。

![Image](images/658f985b145f6e61f2ce85a894be85fdb9a8274cfc73dc90c499fcb5da2d40b9.jpg)

<div style="text-align: center;">Figure 6. Hand-scene interaction. Estimated 3D hand mesh (left) and inferred affordance map (right). Integrating the hand pose into our functional reconstruction enables finding contact regions and consistent reasoning over the associated scene-part motion.</div>

基于人类演示的机器人-场景交互。功能性场景模型可直接迁移到移动操纵器上，从而实现基于人类演示的机器人-场景交互。图7展示了一台配备机械臂的Boston Dynamics Spot机器人在现实世界中与可动物体交互的情景。凭借推断出的接触点、可动参数以及交互轨迹，机器人能够可靠且稳定地重现相同的交互过程。

![Image](images/112b5cd700ad17f540211eb4ad2c8c04ce44f22fbc527d68cfca184ee7d03712.jpg)

<div style="text-align: center;">Figure 7. Robot-scene interaction. Left: Human demonstration of opening a cabinet. Center: The articulation trajectory derived from the functional scene model. Right: The robot leverages the functional information to reliably reproduce the same interaction.</div>

### 7. 结论

我们提出了FunREC——一种无需训练的方法，仅需单段以眼为中心的RGB-D交互视频，即可重建真实环境中的功能性可动3D数字孪生模型。通过将几何推理与基础模型提供的语义及运动先验相结合，FunREC能够同时估算相机运动、部件可动性以及场景几何结构。我们新构建的两个以眼为中心的数据集——RealFun4D和OmniFun4D，为功能性场景理解的定量评估及后续研究提供了支撑。在真实与模拟环境下的实验表明，FunREC在可动性估算、姿态估算、分割及重建质量方面均显著优于所有基准方法，且生成的数字孪生模型可直接用于模拟、功能性推理以及机器人-场景交互。

致谢。本研究得到了瑞士国家科学基金会高级资助项目216260“超越静止世界：从现实世界捕获功能性3D数字孪生”以及瑞士国家科学基金会博士后流动性资助项目222227的支持。作者还感谢瑞士人工智能小型项目资助以及英伟达学术资助的支持。Alexandros Delitzas的研究也得到了马克斯·普朗克ETH学习系统中心（CLS）的支持。

---

## 参考文献

[^1]: Prithviraj Banerjee, Sindi Shkodrani, Pierre Moulon, Shreyas Hampali, Shangchen Han, Fan Zhang, Linguang Zhang, Jade Fountain, Edward Miller, Selen Basol, Richard Newcombe, Robert Wang, Jakob Julian Engel, and Tomas Hodan. HOT3D: Hand and Object Tracking in 3D from Ego-centric Multi-View Videos. In International Conference on Computer Vision and Pattern Recognition (CVPR), 2025. 3

[^2]: Daniel Barath. Superansac: One ransac to rule them all. arXiv preprint arXiv:2506.04803, 2025. 3, 5

[^3]: Gilad Baruch, Zhuoyuan Chen, Afshin Dehghan, Tal Dimry, Yuri Feigin, Peter Fu, Thomas Gebauer, Brandon Joffe, Daniel Kurz, Arik Schwartz, et al. ARKitScenes: A Diverse Real-world Dataset for 3D Indoor Scene Understanding Using Mobile RGB-D Data. In International Conference on Neural Information Processing Systems (NeurIPS), 2021. 1, 2

[^4]: Tjark Behrens, René Zurbrügg, Marc Pollefeys, Zuria Bauer, and Hermann Blum. Lost & Found: Tracking Changes from Egocentric Observations in 3D Dynamic Scene Graphs. IEEE Robotics and Automation Letters (RA-L), 2025. 3

[^5]: Ricardo J. G. B. Campello, Davoud Moulavi, and Joerg Sander. Density-Based Clustering Based on Hierarchical Density Estimates. In Advances in Knowledge Discovery and Data Mining, 2013. 4

[^6]: Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Nießner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3D: Learning from RGBD Data in Indoor Environments. In International Conference on 3d Vision (3dV), 2017. 1, 2

[^7]: Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. ScanNet: Richly-Annotated 3D Reconstructions of Indoor Scenes. In International Conference on Computer Vision and Pattern Recognition (CVPR), 2017. 1, 2

[^8]: Tianyuan Dai, Josiah Wong, Yunfan Jiang, Chen Wang, Cem Gokmen, Ruohan Zhang, Jiajun Wu, and Li Fei-Fei. Automated Creation of Digital Cousins for Robust Policy Learning. In Conference on Robot Learning (CoRL), 2024. 2

[^9]: Ahmad Darkhalil, Dandan Shan, Bin Zhu, Jian Ma, Amlan Kar, Richard Higgins, Sanja Fidler, David Fouhey, and Dima Damen. EPIC-KITCHENS VISOR Benchmark: Video Segmentations and Object Relations. In International Conference on Neural Information Processing Systems (NeurIPS), 2022. 3

[^10]: Google Deepmind. Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities. arXiv preprint arXiv:2507.06261, 2025. 3

[^11]: Alexandros Delitzas, Ayca Takmaz, Federico Tombari, Robert Sumner, Marc Pollefeys, and Francis Engelmann. SceneFun3D: Fine-Grained Functionality and Affordance Understanding in 3D Scenes. In International Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 2

[^12]: Johan Edstedt, Qiyu Sun, Georg Bökman, Mårten Wadenbäck, and Michael Felsberg. RoMa: Robust Dense

特征匹配。载于2024年国际计算机视觉与模式识别会议（CVPR）。3

[^13]: Chengshu Li et al. BEHAVIOR-1K: A Human-Centered, Embodied AI Benchmark with 1,000 Everyday Activities and Realistic Simulation. arXiv preprint arXiv:2403.09227, 2024. 2, 6

[^14]: Zicong Fan, Omid Taheri, Dimitrios Tzionas, Muhammed Kocabas, Manuel Kaufmann, Michael J. Black, and Otmar Hilliges. ARCTIC: A Dataset for Dexterous Bimanual HandObject Manipulation. In International Conference on Computer Vision and Pattern Recognition (CVPR), 2023. 3

[^15]: Haiwen Feng, Junyi Zhang, Qianqian Wang, Yufei Ye, Pengcheng Yu, Michael J. Black, Trevor Darrell, and Angjoo Kanazawa. St4RTrack: Simultaneous 4D Reconstruction and Tracking in the World. In International Conference on Computer Vision and Pattern Recognition (CVPR), 2025. 3

[^16]: Pradyumn Goyal, Dmitry Petrov, Sheldon Andrews, Yizhak Ben-Shabat, Hsueh-Ti Derek Liu, and Evangelos Kalogerakis. GEOPARD: Geometric Pretraining for Articulation Prediction in 3D Shapes. In International Conference on Computer Vision (ICCV), 2025. 2

[^17]: Vladimir Guzov, Julian Chibane, Riccardo Marin, Yannan He, Yunus Saracoglu, Torsten Sattler, and Gerard Pons-Moll. Interaction Replica: Tracking human–object interaction and scene changes from human motion. In International Conference on 3d Vision (3dV), 2024. 3

[^18]: Anna-Maria Halacheva, Yang Miao, Jan-Nico Zaech, Xi Wang, Luc Van Gool, and Danda Pani Paudel. Holistic Understanding of 3D Scenes as Universal Scene Description. In International Conference on Computer Vision (ICCV), 2025. 2

[^19]: Adam W. Harley, Yang You, Xinglong Sun, Yang Zheng, Nikhil Raghuraman, Yunqi Gu, Sheldon Liang, Wen-Hsuan Chu, Achal Dave, Pavel Tokmakov, Suya You, Rares Ambrus, Katerina Fragkiadaki, and Leonidas J. Guibas. All-Tracker: Efficient Dense Point Tracking at High Resolution. In International Conference on Computer Vision (ICCV), 2025. 3

[^20]: Nick Heppert, Muhammad Zubair Irshad, Sergey Zakharov, Katherine Liu, Rares Andrei Ambrus, Jeannette Bohg, Abhinav Valada, and Thomas Kollar. CARTO: Category and Joint Agnostic Reconstruction of ARTiculated Objects. In International Conference on Computer Vision and Pattern Recognition (CVPR), 2023. 2

[^21]: Shengyu Huang, Zan Gojcic, Mikhail Usvyatsov, and Konrad Schindler Andreas Wieser. PREDATOR: Registration of 3D Point Clouds with Low Overlap. In International Conference on Computer Vision and Pattern Recognition (CVPR), 2021. 5

[^22]: Zhening Huang, Xiaoyang Wu, Fangcheng Zhong, Hengshuang Zhao, Matthias Nießner, and Joan Lasenby. LiteReality: Graphics-Ready 3D Scene Reconstruction from RGB-D Scans. arXiv preprint arXiv:2507.02861, 2025. 2

## [^23]: Zhao Huang, Boyang Sun, Alexandros Delitzas, Jiaqi Chen, and Marc Pollefeys. React3d: Recovering articulations for interactive physical 3d scenes. IEEE Robotics and Automation Letters (RA-L), 2026. 2

---

[^24]: Allison Janoch, Sergey Karayev, Yangqing Jia, Jonathan T. Barron, Mario Fritz, Kate Saenko, and Trevor Darrell. A Category-level 3D Object Dataset: Putting the Kinect to Work. In International Conference on Computer Vision (ICCV) Workshops, 2011. 1

[^25]: Hanxiao Jiang, Yongsen Mao, Manolis Savva, and Angel X Chang. OPD: Single-view 3D Openable Part Detection. In European Conference on Computer Vision (ECCV), 2022. 2

[^26]: Zhenyu Jiang, Cheng-Chun Hsu, and Yuke Zhu. Ditto: Building Digital Twins of Articulated Objects from Interaction. In International Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 2

[^27]: Nikita Karaev, Ignacio Rocco, Benjamin Graham, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. CoTracker: It is Better to Track Together. In European Conference on Computer Vision (ECCV), 2024. 3

[^28]: Nikita Karaev, Iurii Makarov, Jianyuan Wang, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. CoTracker3: Simpler and Better Point Tracking by Pseudo-Labelling Real Videos. In International Conference on Computer Vision (ICCV), 2025. 3, 7

[^29]: Justin Kerr, Chung Min Kim, Mingxuan Wu, Brent Yi, Qianqian Wang, Ken Goldberg, and Angjoo Kanazawa. Robot See Robot Do: Imitating Articulated Object Manipulation with Monocular 4D Reconstruction. In Conference on Robot Learning (CoRL), 2024. 2, 3

[^30]: Jeonghwan Kim, Jisoo Kim, Jeonghyeon Na, and Hanbyul Joo. Parahome: Parameterizing everyday home activities towards 3d generative modeling of human-object interactions. In International Conference on Computer Vision and Pattern Recognition (CVPR), 2025. 3

[^31]: Maxim Kolodiazhnyi, Anna Vorontsova, Anton Konushin, and Danila Rukhovich. OneFormer3D: One transformer for Unified Point Cloud Segmentation. In International Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 2

[^32]: Taein Kwon, Bugra Tekin, Jan Stühmer, Federica Bogo, and Marc Pollefeys. H2O: Two Hands Manipulating Objects for First Person Interaction Recognition. In International Conference on Computer Vision (ICCV), 2021. 3

[^33]: Justin Lazarow, David Griffiths, Gefen Kohavi, Francisco Crespo, and Afshin Dehghan. Cubify Anything: Scaling Indoor 3D Object Detection. In International Conference on Computer Vision and Pattern Recognition (CVPR), 2025. 2

[^34]: Long Le, Jason Xie, William Liang, Hung-Ju Wang, Yue Yang, Yecheng Jason Ma, Kyle Vedder, Arjun Krishna, Dinesh Jayaraman, and Eric Eaton. Articulate-anything: Automatic modeling of articulated objects via a vision-language foundation model. In International Conference on Learning Representations (ICLR), 2025. 2

[^35]: Jiahui Lei, Yijia Weng, Adam Harley, Leonidas Guibas, and Kostas Daniilidis. MoSca: Dynamic Gaussian Fusion from Casual Videos via 4D Motion Scaffolds. In International Conference on Computer Vision and Pattern Recognition (CVPR), 2025. 3

[^36]: Zhengqi Li, Richard Tucker, Forrester Cole, Qianqian Wang, Linyi Jin, Vickie Ye, Angjoo Kanazawa, Aleksander Holyn

Ski和Noah Snavely。MegaSaM：从普通动态视频中快速、精准且稳健地恢复结构与运动。载于2025年国际计算机视觉与模式识别会议（CVPR）。3

[^37]: Jiayi Liu, Ali Mahdavi-Amiri, and Manolis Savva. PARIS: Part-level Reconstruction and Motion Analysis for Articulated Objects. In International Conference on Computer Vision (ICCV), 2023. 2

[^38]: Jiayi Liu, Manolis Savva, and Ali Mahdavi-Amiri. Survey on Modeling of Human-made Articulated Objects. arXiv preprint arXiv:2403.14937, 2025. 2

[^39]: Xueyi Liu, Ji Zhang, Ruizhen Hu, Haibin Huang, He Wang, and Li Yi. Self-Supervised Category-Level Articulated Object Pose Estimation with Part-Level SE(3) Equivariance. In International Conference on Learning Representations (ICLR), 2023. 2

[^40]: Yunze Liu, Yun Liu, Che Jiang, Kangbo Lyu, Weikang Wan, Hao Shen, Boqiang Liang, Zhoujie Fu, He Wang, and Li Yi. HOI4D: A 4D Egocentric Dataset for Category-Level Human-Object Interaction. In International Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 2, 3, 5, 6, 7

[^41]: Yu Liu, Baoxiong Jia, Ruijie Lu, Chuyue Gan, Huayu Chen, Junfeng Ni, Song-Chun Zhu, and Siyuan Huang. Videoartgs: Building digital twins of articulated objects from monocular video. arXiv preprint arXiv:2509.17647, 2025. 3

[^42]: Yu Liu, Baoxiong Jia, Ruijie Lu, Junfeng Ni, Song-Chun Zhu, and Siyuan Huang. Building Interactable Replicas of Complex Articulated Objects via Gaussian Splatting. In International Conference on Learning Representations (ICLR), 2025. 2, 7

[^43]: Xianzheng Ma, Yash Bhalgat, Brandon Smart, Shuai Chen, Xinghui Li, Jian Ding, Jindong Gu, Dave Zhenyu Chen, Songyou Peng, Jia-Wang Bian, Philip H Torr, Marc Pollefeys, Matthias Nießner, Ian D Reid, Angel X. Chang, Iro Laina, and Victor Adrian Prisacariu. When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Models, 2024. 1

[^44]: Yongsen Mao, Yiming Zhang, Hanxiao Jiang, Angel X Chang, and Manolis Savva. MultiScan: Scalable RGBD Scanning for 3D Environments With Articulated Objects. In International Conference on Neural Information Processing Systems (NeurIPS), 2022. 2

[^45]: Muzammal Naseer, Salman Khan, and Fatih Porikli. Indoor Scene Understanding in 2.5/3D for Autonomous Agents: A Survey. IEEE Access, 2018. 1

[^46]: Tuan Ngo, Peiye Zhuang, Evangelos Kalogerakis, Chuang Gan, Sergey Tulyakov, Hsin-Ying Lee, and Chaoyang Wang. DELTA: Dense Efficient Long-Range 3D Tracking for Any Video. In International Conference on Learning Representations (ICLR), 2025. 3

[^47]: NVIDIA. Isaac Sim. 8

[^48]: NVIDIA. NVIDIA® RTX Path Tracing, 2023. 6

## [^49]: Georgios Pavlakos, Dandan Shan, Ilija Radosavovic, Angjoo Kanazawa, David Fouhey, and Jitendra Malik. Reconstructing Hands in 3D With Transformers. In International Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 8

---

[^50]: Weikun Peng, Jun Lv, Cewu Lu, and Manolis Savva. iTACO: Interactable Digital Twins of Articulated Objects from Casually Captured RGBD Videos. In International Conference on 3d Vision (3dV), 2026. 3

[^51]: Rolandos Alexandros Potamias, Jinglei Zhang, Jiankang Deng, and Stefanos Zafeiriou. Wilor: End-to-end 3d hand localization and reconstruction in-the-wild. In International Conference on Computer Vision and Pattern Recognition (CVPR), 2025. 8

[^52]: Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation. In International Conference on Computer Vision and Pattern Recognition (CVPR), 2017. 2

[^53]: Charles R Qi, Or Litany, Kaiming He, and Leonidas J Guibas. Deep Hugh Voting for 3D Object Detection in Point Clouds. In International Conference on Computer Vision (ICCV), 2019. 2

[^54]: Frano Rajič, Haofei Xu, Marko Mihajlovic, Siyuan Li, Irem Demir, Emircan Gündoğdu, Lei Ke, Sergey Prokudin, Marc Pollefeys, and Siyu Tang. Multi-view 3d point tracking. In International Conference on Computer Vision (ICCV), 2025. 3

[^55]: Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao-Yuan Wu, Ross Girshick, Piotr Dollár, and Christoph Feichtenhofer. SAM 2: Segment Anything in Images and Videos. arXiv preprint arXiv:2510.11340, 2024. 5

[^56]: Bokui Shen, Fei Xia, Chengshu Li, Roberto Martín-Martín, Linxi Fan, Guanzhi Wang, Claudia Pérez-D'Arpino, Shyamal Buch, Sanjana Srivastava, Lyne P. Tchapmi, Micael E. Tchapmi, Kent Vainio, Josiah Wong, Li Fei-Fei, and Silvio Savarese. iGibson 1.0: A Simulation Environment for Interactive Tasks in Large Realistic Scenes. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2021. 2, 6

[^57]: Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor Segmentation and Support Inference from RGBD Images. In European Conference on Computer Vision (ECCV), 2012. 1

[^58]: Shuran Song, Samuel P. Lichtenberg, and Jianxiong Xiao. SUN RGB-D: A RGB-D Scene Understanding Benchmark Suite. In International Conference on Computer Vision and Pattern Recognition (CVPR), 2015. 1

[^59]: Tao Sun, Yan Hao, Shengyu Huang, Silvio Savarese, Konrad Schindler, Marc Pollefeys, and Iro Armeni. Nothing Stands Still: A Spatiotemporal Benchmark on 3D Point Cloud Registration Under Large Geometric and Temporal Change. IS-PRS Journal of Photogrammetry and Remote Sensing, 2025. 2

[^60]: Xiaohao Sun, Hanxiao Jiang, Manolis Savva, and Angel Chang. OPDMulti: Openable Part Detection for Multiple Objects. In International Conference on 3d Vision (3dV), 2024. 2

[^61]: Johanna Wald, Armen Avetisyan, Nassir Navab, Federico Tombari, and Matthias Niessner. RIO: 3D Object Instance

变化中的室内环境定位。载于2019年国际计算机视觉会议（ICCV）。1, 2

[^62]: Jikai Wang, Qifan Zhang, Yu-Wei Chao, Bowen Wen, Xiaohu Guo, and Yu Xiang. HO-Cap: A Capture System and Dataset for 3D Reconstruction and Pose Tracking of HandObject Interaction. In International Conference on Neural Information Processing Systems (NeurIPS), 2025. 3

[^63]: Qianqian Wang, Yen-Yu Chang, Ruojin Cai, Zhengqi Li, Bharath Hariharan, Aleksander Holynski, and Noah Snavely. Tracking Everything Everywhere All at Once. In International Conference on Computer Vision (ICCV), 2023. 3

[^64]: Qianqian Wang, Vickie Ye, Hang Gao, Weijia Zeng, Jake Austin, Zhengqi Li, and Angjoo Kanazawa. Shape of Motion: 4D Reconstruction from a Single Video. In International Conference on Computer Vision (ICCV), 2025. 3

[^65]: Qianqian Wang, Yifei Zhang, Aleksander Holynski, Alexei A. Efros, and Angjoo Kanazawa. Continuous 3D Perception Model with Persistent State. In International Conference on Computer Vision and Pattern Recognition (CVPR), 2025. 3

[^66]: B Wen and Kostas E Bekris. BundleTrack: 6D Pose Tracking for Novel Objects without Instance or Category-Level 3D Models. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2021. 3, 7

[^67]: Bowen Wen, Jonathan Tremblay, Valts Blukis, Stephen Tyree, Thomas Muller, Alex Evans, Dieter Fox, Jan Kautz, and Stan Birchfield. BundleSDF: Neural 6-DoF Tracking and 3D Reconstruction of Unknown Objects. In International Conference on Computer Vision and Pattern Recognition (CVPR), 2023. 7, 8

[^68]: Bowen Wen, Wei Yang, Jan Kautz, and Stan Birchfield. FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objects. In International Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 3

[^69]: Yijia Weng, Bowen Wen, Jonathan Tremblay, Valts Blukis, Dieter Fox, Leonidas Guibas, and Stan Birchfield. Neural Implicit Representation for Building Digital Twins of Unknown Articulated Objects. In International Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 2

[^70]: Abdelrhman Werby, Martin Buechner, Adrian Roefer, Chenguang Huang, Wolfram Burgard, and Abhinav Valada. Articulated object estimation in the wild. In Conference on Robot Learning (CoRL), 2025. 3

[^71]: Di Wu, Liu Liu, Zhou Linli, Anran Huang, Liangtu Song, Qiaojun Yu, Qi Wu, and Cewu Lu. Reartgs: Reconstructing and generating articulated objects via 3d gaussian splatting with geometric and motion constraints. In International Conference on Neural Information Processing Systems (NeurIPS), 2025. 2

[^72]: Mingxuan Wu, Huang Huang, Justin Kerr, Chung Min Kim, Anthony Zhang, Brent Yi, and Angjoo Kanazawa. Predict-Optimize-Distill: A Self-Improving Cycle for 4D Object Un-derstanding. In International Conference on Computer Vi-sion (ICCV), 2025. 2, 3

## [^73]: Hongchi Xia, Entong Su, Marius Memmel, Arhan Jain, Raymond Yu, Numfor Mbiziwo-Tiapo, Ali Farhadi, Abhishek

---

Gupta、Shenlong Wang和Wei-Chiu Ma。Drawer：具有环境真实感的数字重建与关节建模。载于2025年国际计算机视觉与模式识别会议（CVPR）。2

[^74]: Jianxiong Xiao, Andrew Owens, and Antonio Torralba. SUN3D: A Database of Big Spaces Reconstructed Using SfM and Object Labels. In International Conference on Computer Vision (ICCV), 2013. 1

[^75]: Yuxi Xiao, Qianqian Wang, Shangzhan Zhang, Nan Xue, Sida Peng, Yujun Shen, and Xiaowei Zhou. SpatialTracker: Tracking Any 2D Pixels in 3D Space. In International Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 3

[^76]: Yuxi Xiao, Jianyuan Wang, Nan Xue, Nikita Karaev, Yuri Makarov, Bingyi Kang, Xing Zhu, Hujun Bao, Yujun Shen, and Xiaowei Zhou. Spatialtrackerv2: Advancing 3d point tracking with explicit camera motion. In International Conference on Computer Vision (ICCV), 2025. 3, 7, 8

[^77]: Chandan Yeshwanth, Yueh-Cheng Liu, Matthias Nießner, and Angela Dai. ScanNet++: A High-fidelity Dataset of 3D Indoor Scenes. In International Conference on Computer Vision (ICCV), 2023. 1, 2

[^78]: Huangyue Yu, Baoxiong Jia, Yixin Chen, Yandan Yang, Puhao Li, Rongpeng Su, Jiaxin Li, Qing Li, Wei Liang, Zhu Song-Chun, Tengyu Liu, and Siyuan Huang. METASCENES: Towards Automated Replica Creation for Real-world 3D Scans. In International Conference on Computer Vision and Pattern Recognition (CVPR), 2025. 2

[^79]: Chengbo Yuan, Geng Chen, Li Yi, and Yang Gao. Self-Supervised Monocular 4D Scene Reconstruction for Ego-centric Videos. In International Conference on Computer Vision (ICCV), 2025. 3

[^80]: Bowei Zhang, Lei Ke, Adam W Harley, and Katerina Fragki-adaki. TAPIP3D: Tracking Any Point in Persistent 3D Geometry. In International Conference on Neural Information Processing Systems (NeurIPS), 2025. 3

[^81]: Daiwei Zhang, Gengyan Li, Jiajie Li, Mickaël Bressieux, Omar Hilliges, Marc Pollefeys, Luc Van Gool, and Xi Wang. EgoGaussian: Dynamic Scene Understanding from Egocentric Video with 3D Gaussian Splatting. In International Conference on 3D Vision (3D V) 2025. 3

[^82]: Junyi Zhang, Charles Herrmann, Junhwa Hur, Varun Jampani, Trevor Darrell, Forrester Cole, Deqing Sun, and Ming-Hsuan Yang. MonST3R: A Simple Approach for Estimating Geometry in the Presence of Motion. In International Conference on Learning Representations (ICLR), 2025. 3, 7, 8

[^83]: Mandi Zhao, Yijia Weng, Dominik Bauer, and Shuran Song. Real2Code: Reconstruct Articulated Objects via Code Generation. In International Conference on Learning Representations (ICLR), 2025. 2