{
  "paper_id": "c4af90a411e6a30da226d3132f43036e",
  "paper_title": "ECHO: Ego-Centric modeling of Human-Object interactions",
  "paper_authors": [
    "Ilya A. Petrov",
    "Vladimir Guzov",
    "Riccardo Marin",
    "Emre Aksan",
    "Xu Chen",
    "Daniel Cremers",
    "Thabo Beeler",
    "Gerard Pons-Moll"
  ],
  "publication_date": "",
  "publication_venue": "unknown",
  "abstract": "Modeling human-object interactions (HOI) from an egocentric perspective is a critical yet challenging task, particularly when relying on sparse signals from wearable devices like smart glasses and watches. We present ECHO, the first unified framework to jointly recover human pose, object motion, and contact dynamics solely from head and wrist tracking. To tackle the underconstrained nature of this problem, we introduce a novel tri-variate diffusion process with independent noise schedules that models the mutual dependencies between the human, object, and interaction modalities. This formulation allows ECHO to operate with flexible input configurations, making it robust to intermittent tracking and capable of leveraging partial observations. Crucially, it enables training on a combination of large-scale human motion datasets and smaller HOI collections, learning strong priors while capturing interaction nuances. Furthermore, we employ a smooth inpainting inference mechanism that enables the generation of temporally consistent interactions for arbitrarily long sequences. Extensive evaluations demonstrate that ECHO achieves state-of-the-art performance, significantly outperforming existing methods lacking such flexibility.",
  "keywords": [
    "人-物交互",
    "自我中心感知",
    "扩散模型",
    "人体运动重建",
    "接触动力学",
    "可穿戴传感器",
    "稀疏追踪",
    "SMPL-X",
    "平滑修复"
  ],
  "paper_institutions": [
    "University of Tübingen, Germany",
    "Tübingen AI Center, Germany",
    "Technical University of Munich, Germany",
    "Munich Center for Machine Learning, Germany",
    "Google, Switzerland",
    "Max Planck Institute for Informatics, Saarland Informatics Campus, Germany"
  ],
  "output_language": "zh",
  "provider": "doubao",
  "model": "glm-5.3",
  "prompt_template": "simple",
  "extracted_at": "2026-09-17T09:36:25.211999Z",
  "template_tag": "simple",
  "summary": "### 摘要\n从智能眼镜、智能手表等可穿戴设备的稀疏信号（头部与手腕三点追踪）中以自我为中心重建人-物交互（HOI）是一个关键但欠约束的难题：现有方法（如iReplica）依赖全身IMU、预扫描3D场景或手工启发式约束，且现有HOI数据集规模与多样性远小于AMASS等大规模人体运动数据集，难以同时学习强运动先验与交互细节。ECHO提出首个仅凭头手三点追踪即可联合恢复人体姿态、物体运动与接触动力学的统一框架，其核心机制是具有独立噪声调度（$\\mathcal{T}_{\\mathcal{H}},\\mathcal{T}_{\\mathcal{O}},\\mathcal{T}_{\\mathcal{I}}$）的三变量扩散过程，训练目标为最小化 $\\mathbb{E}_{p}\\mathbb{E}_{\\mathcal{T}}\\mathbb{E}_{q}\\|\\mathrm{ECHO}_{\\psi}(\\mathcal{H}^{\\mathcal{T}_{\\mathcal{H}}},\\mathcal{O}^{\\mathcal{T}_{\\mathcal{O}}},\\mathcal{I}^{\\mathcal{T}_{\\mathcal{I}}};\\mathcal{T}_{\\mathcal{H}},\\mathcal{T}_{\\mathcal{O}},\\mathcal{T}_{\\mathcal{I}};\\mathcal{C}_{\\mathcal{O}},\\mathcal{E})-(\\mathcal{H}^{0},\\mathcal{O}^{0},\\mathcal{I}^{0})\\|_{2}$，该设计支持灵活输入配置、部分观测利用与间歇性追踪鲁棒性。模型采用57.7M参数的Diffusion Transformer（DiT）与逐帧头部中心坐标系表示，在AMASS（提供大规模运动先验）、BEHAVE与OMOMO（提供HOI细节）三数据集并集上联合训练——AMASS样本以可学习token替代物体条件，消融证明此联合训练至关重要；推理时结合平滑修复（smooth inpainting）实现任意长度序列的时序一致在线生成，并以分类器引导确保接触物理合理性。评估采用MPJPE、MPJVE、FC、$E_{v2v}$、$E_c$、旋转差与接触精度等指标：ECHO在BEHAVE上达MPJPE 6.8cm、$E_{v2v}$ 33.5cm，在OMOMO上达MPJPE 6.0cm、$E_{v2v}$ 26.5cm，在AMASS上达MPJPE 7.4cm，全面超越BoDiffusion+O与EgoAllo+H+O基线，验证了三模态联合建模、独立噪声调度与平滑修复等创新的有效性。未来方向包括引入动态环境约束、细粒度手指追踪与RGB视觉条件化；实际应用中，缺乏手指追踪限制了对笔、剪刀等小物体的灵巧交互重建，且该技术存在被用于未经授权监视的隐私风险，需以负责任的方式持续发展。\n\n### 关键要点\n- ECHO是首个仅从可穿戴设备三点（头部+双手）追踪联合恢复全身人-物交互序列的端到端方法，同时预测人体运动、物体轨迹与接触序列。\n- 核心创新为三变量扩散过程：对人体运动$\\mathcal{H}$、物体运动$\\mathcal{O}$与接触序列$\\mathcal{I}$采用独立噪声调度，支持灵活的跨模态条件化、部分观测与间歇性追踪鲁棒性。\n- 表示采用逐帧头部中心坐标系：人体用SMPL-X身体姿态$\\theta_{\\mathcal{H}}\\in\\mathbb{R}^{21\\times6}$，物体用头部坐标系下SE(3)变换序列（旋转转为$\\mathbb{R}^6$），接触用sigmoid映射的连续距离向量（含人-物与人-地面接触）。\n- 训练数据为AMASS、BEHAVE、OMOMO三数据集并集：AMASS提供大规模运动先验（物体条件以可学习token替代），BEHAVE/OMOMO提供交互细节；NoAMASS消融显示去除AMASS后其测试MPJPE从7.4cm退化至43.1cm。\n- 实现细节：57.7M参数DiT架构，AdamW优化器训练300k步（约30小时），批大小256，学习率5e-4，单张NVIDIA RTX 5090 GPU；推理用100步DDPM。\n- 推理性能：60帧窗口（30帧重叠）处理耗时640ms（无引导）/980ms（有引导），对应约46/30 FPS吞吐，每帧摊销延迟约21–33ms，支持低延迟实时应用。\n- 主要结果：BEHAVE上MPJPE 6.8±0.1cm、$E_{v2v}$ 33.5±0.5cm、$E_c$ 20.1±0.3cm；OMOMO上MPJPE 6.0±0.1cm、$E_{v2v}$ 26.5±1.1cm；均优于BoDiffusion+O与EgoAllo+H+O基线。\n- 噪声鲁棒性：随机丢弃手部追踪数据时，ECHO在缺失75%前性能基本稳定（BEHAVE MPJPE从6.8升至7.6cm），仅90%缺失时明显退化（9.3cm）。\n- 稀疏输入利用：提供50%物体观测使$E_{v2v}$从33.5降至10.75cm，提供50%人体观测使MPJPE从6.82降至3.71cm；接触条件化带来最大整体改进。\n- 消融验证：去除推理引导、使用非平滑修复、去除交互模态（仅H,O）或去除AMASS训练均导致性能下降，证明各组件必要性。\n- 物理合理性：ECHO在BEHAVE上最小人-物距离3.5cm、接触率79.5%，接近真值（3.3cm、80.2%），优于产生漂浮或穿透伪影的基线。\n- 局限与约束：缺乏细粒度手指追踪限制小物体灵巧交互重建；未建模动态环境约束；未来需扩展RGB视觉条件化并收集RGB自我中心HOI数据集；技术存在隐私滥用风险需负责任发展。"
}