{
  "paper_id": "6a1d0b146c35f29af83b75d43b68158d",
  "paper_title": "xR-EgoPose: Egocentric 3D Human Pose from an HMD Camera",
  "paper_authors": [
    "Denis Tome",
    "Patrick Peluse",
    "Lourdes Agapito",
    "Hernan Badino"
  ],
  "publication_date": "2019",
  "publication_venue": "CVPR 2019",
  "abstract": "We present a new solution to egocentric 3D body pose estimation from monocular images captured from a downward looking fish-eye camera installed on the rim of a head mounted virtual reality device. This unusual viewpoint, just 2 cm. away from the user's face, leads to images with unique visual appearance, characterized by severe self-occlusions and strong perspective distortions that result in a drastic difference in resolution between lower and upper body. Our contribution is two-fold. Firstly, we propose a new encoder-decoder architecture with a novel dual branch decoder designed specifically to account for the varying uncertainty in the 2D joint locations. Our quantitative evaluation, both on synthetic and real-world datasets, shows that our strategy leads to substantial improvements in accuracy over state of the art egocentric pose estimation approaches. Our second contribution is a new large-scale photorealistic synthetic dataset – xR-EgoPose – offering 383K frames of high quality renderings of people with a diversity of skin tones, body shapes, clothing, in a variety of backgrounds and lighting conditions, performing a range of actions. Our experiments show that the high variability in our new synthetic training corpus leads to good generalization to real world footage and to state of the art results on real world datasets with ground truth. Moreover, an evaluation on the Human3.6M benchmark shows that the performance of our method is on par with top performing approaches on the more classic problem of 3D human pose from a third person viewpoint.",
  "keywords": [
    "egocentric 3D human pose estimation",
    "head-mounted display",
    "fish-eye camera",
    "dual branch auto-encoder",
    "photorealistic synthetic dataset",
    "2D-to-3D lifting",
    "virtual reality"
  ],
  "paper_institutions": [
    "University College London",
    "Facebook Reality Lab"
  ],
  "output_language": "zh",
  "provider": "doubao",
  "model": "glm-5.3",
  "prompt_template": "simple",
  "extracted_at": "2026-09-17T09:37:17.642230Z",
  "template_tag": "simple",
  "summary": "### 摘要\n\n本文解决的问题是从安装在头戴式VR设备边缘、向下拍摄且距用户面部仅约2厘米的鱼眼单目相机图像中估计自我中心的全身体3D人体姿态；该特殊视角造成严重自遮挡、强烈透视畸变以及上下半身分辨率的巨大差异，使面向外部相机的现有方法表现不佳。相关研究涵盖外部相机视角的单目3D姿态估计（直接回归与\"2D检测+3D提升\"两步法）、第一人称姿态估计（如头盔立体鱼眼相机的EgoCap、棒球帽单目鱼眼相机的Mo2Cap2）以及IMU等可穿戴传感器方案，但它们或侵入性强、或精度不足。为此作者提出两步架构：先用ResNet101从输入图像回归15个身体关节的2D热图，再通过新颖的双分支自编码器将热图提升为16个3D关节坐标（含2D图像中不可观测的头部）；其核心机制是解码器一支从潜变量回归3D姿态、另一支重建输入2D热图，迫使潜变量编码2D关节估计的不确定性，总损失为 L_AE = λp(||P−P̂||² + R(P,P̂)) + λhm||HM−H̃M||²，其中R由肢体余弦相似度误差与肢体长度误差构成，且该设计允许混合仅有2D标注与含3D标注的数据同时训练。为解决该场景训练数据稀缺问题，作者构建了大规模照片级真实感合成数据集xR-EgoPose（383K帧，23男23女角色，身高155–189cm，多样肤色、体型、服装、背景、光照与9类动作，Maya+真实动捕数据动画化并用V-Ray物理渲染，1024×1024分辨率，训练/测试/验证集为252K/115K/16K帧），并采集带3D真值的小规模真实数据集xR-EgoPose^R（约10K帧、3名演员、6种动作、定制动捕系统提供真值），以验证模型从合成到真实的泛化能力。实现上模型从头训练3个epoch，学习率1e-3，batch size为16，Xavier初始化，λ权重经网格搜索设为λhm=1e-3、λp=1e-1、λθ=−1e-2、λL=0.5。评估采用MPJPE指标：合成测试集上双分支架构平均误差58.2mm，显著优于单分支消融（130.4mm）与Martinez基线（122.1mm），总体提升52.3%，证明不确定性编码分支是精度提升的关键；Mo2Cap2真实测试集上室内48.16mm、室外60.19mm，分别超越对手21.6%和25.4%；xR-EgoPose^R上平均61.71mm，显示良好的真实泛化；Human3.6M外部相机基准上Protocol 1误差53.4mm、Protocol 2为45.24mm，位列非时序方法第二，表明架构可无修改地泛化到标准第三人称视角；实验还证明额外2D标注（COCO、MPII）可进一步提升3D精度。未来方向是增加相机以覆盖更大视野并实现多视角感知。该技术面向VR/AR/MR中的沉浸式社交交互与化身驱动等真实场景，但受限于极端遮挡以及手部移出视野时无法测量等约束。\n\n### 关键要点\n\n- 任务挑战：HMD下视鱼眼相机距面部仅约2cm，图像存在严重自遮挡、极端透视效应以及下半身像素密度急剧下降，使面向外部相机的最先进姿态估计方法在该视角下失效。\n- 方法架构：采用两步流水线，第一模块用ResNet101（末端以反卷积层替换平均池化与全连接层）从368×368×3的RGB图像回归47×47×15的2D关节热图，第二模块用双分支自编码器输出16×3的3D姿态（含2D中不可见的头部关节）。\n- 核心创新：双分支解码器一支回归3D姿态、另一支重建输入2D热图，迫使潜变量编码热图的概率密度（即2D估计的不确定性），从而对遮挡或高不确定性部位推断更准确的关节姿态；该分支仅训练时使用，测试时可移除以保证速度且不损失性能。\n- 损失设计：总损失 L_AE = λp(||P−P̂||² + R(P,P̂)) + λhm||HM−H̃M||²，其中R由肢体余弦相似度误差θ与肢体长度误差L组成；若样本仅有2D标注则置λp=0，使模型可同时混合3D与2D数据集训练。\n- 合成数据集：xR-EgoPose共383K帧（训练252K、测试115K、验证16K），23男23女角色、身高155–189cm、六类肤色、多种服装鞋类与9种动作（游戏、打手势、问候、上下伸展、行走等），使用Maya+真实动捕数据动画化并用V-Ray标准化物理渲染，图像1024×1024、30fps，附带RGB、深度、法线、身体分割与元数据。\n- 真实数据集：xR-EgoPose^R约10K帧，由安装在VR HMD上的真实鱼眼相机采集，3名演员、不同服装、6种动作，3D真值由定制动作捕捉系统提供；网络先在合成语料上训练再用两名演员数据微调，在未见过的第三名演员上测试。\n- 训练设置：整个训练集训练3个epoch，学习率1e-3，batch size 16，Xavier初始化；编码器卷积/反卷积核大小4、步长2，激活函数为0.2泄漏系数的leaky ReLU；λhm=1e-3、λp=1e-1、λθ=−1e-2、λL=0.5由网格搜索确定。\n- 合成集结果：双分支平均MPJPE为58.2mm，相比单分支消融的130.4mm和Martinez基线的122.1mm，总体提升52.3%（上半身36.4%、下半身60%），证明重建热图的解码分支是精度大幅提升的关键。\n- 真实集结果：在Mo2Cap2真实测试集上，室内总误差48.16mm（超越Mo2Cap2的61.40mm约21.6%）、室外60.19mm（超越80.64mm约25.4%）；在xR-EgoPose^R上平均误差61.71mm，验证了合成训练语料向真实图像的良好泛化。\n- 泛化与基准：在Human3.6M外部相机基准上，Protocol 1误差53.4mm、Protocol 2误差45.24mm，在非时序方法中排名第二，表明该架构无需修改即可适用于标准第三人称视角。\n- 混合2D/3D训练：在xR-EgoPose上，将2D标注从50%增至100%（3D保持50%）使误差从68.04mm降至63.98mm；在Human3.6M上加入COCO与MPII的2D标注使误差从67.9mm降至53.4mm。\n- 误差与局限：逐关节误差分析显示手部（出视野遮挡，最高90.43mm）与脚部（自遮挡与低分辨率，脚趾约72–76mm）误差最大；极端遮挡和手部出视野时系统失败，未来工作计划增加相机以覆盖更大视野并实现多视角感知。"
}