# EgoFish3D: Egocentric 3D Pose Estimation From a Fisheye Camera via Self-Supervised Learning
Yuxuan Liu $^{ID}$, Graduate Student Member, IEEE, Jianxin Yang, Graduate Student Member, IEEE, Xiao Gu $^{ID}$, Graduate Student Member, IEEE, Yijun Chen, Yao Guo $^{ID}$, Member, IEEE, and Guang-Zhong Yang $^{ID}$, Fellow, IEEE
Abstract—Egocentric vision has gained increasing popularity recently, opening new avenues for human-centric applications. However, the use of the egocentric fisheye cameras allows wide angle coverage but image distortion is introduced along with strong human body self-occlusion imposing significant challenges in data processing and model reconstruction. Unlike previous work only leveraging synthetic data for model training, this paper presents a new real-world EgoCentric Human Pose (ECHP) dataset. To tackle the difficulty of collecting 3D ground truth using motion capture systems, we simultaneously collect images from a head-mounted egocentric fisheye camera as well as from two third-person-view cameras, circumventing the environmental restrictions. By using self-supervised learning under multi-view constraints, we propose a simple yet effective framework, namely EgoFish3D, for egocentric 3D pose estimation from a single image in different real-world scenarios. The proposed EgoFish3D incorporates three main modules. 1) The third-person-view module takes two exocentric images as input and estimates the 3D pose represented in the third-person camera frame; 2) the egocentric module predicts the 3D pose in the egocentric camera frame; and 3) the interactive module estimates the rotation matrix between the third-person and the egocentric views. Experimental results on our ECHP dataset and existing benchmark datasets demonstrate the effectiveness of the proposed EgoFish3D, which can achieve superior performance to existing methods.
Index Terms—Egocentric vision, 3D human pose estimation, Self-supervised learning, Multi-view constraints.
Manuscript received 16 January 2022; revised 8 October 2022 and 29 November 2022; accepted 23 January 2023. Date of publication 6 February 2023; date of current version 12 December 2023. This work was supported in part by the National Natural and Science Foundation of China under Grant 62203296, in part by Shanghai Pujiang Program under Grant 22PJ1405500, in part by the Science and Technology Commission of Shanghai Municipality under Grant 20DZ2220400, and in part by the Shanghai Pilot Program for Basic Research - Shanghai Jiao Tong University under Grant 21TQ1400203. The Associate Editor coordinating the review of this manuscript and approving it for publication was Prof. Mohammed Daoudi. (Corresponding authors: Yao Guo; Guang Zhong Yang.)
### I. INTRODUCTION
E GOCENTRIC vision is an emerging field in computer vision, involving the analysis of data captured from a head-mounted or chest-mounted wearable camera [^1], [^2], [^3]. When the egocentric camera is directed downwards, especially by incorporating a fisheye lens, the human body and the surrounding environment can be captured with an enlarged field-of-view, offering expanded visual cues for processing [^3], [^4], [^5], [^6]. Compared to the third-person view, egocentric vision is advantageous for long-term human-centric perception in a free-living environment, offering new opportunities for understanding human behavior and social activities [^5], [^6], [^7], [^8], [^9].
One of the prerequisites of egocentric vision for downstream applications is accurate pose estimation from egocentric views. Although extensive progress in monocular 2D/3D human pose estimation has been achieved in recent years [^10], [^11], [^12], [^13], these conventional third-person-view pose estimation methods are prone to errors when directly used to predict 2D/3D poses from novel first-person viewpoints, as illustrated in Fig. 1(b), which shows typical results that can be achieved. Hitherto, there are few dedicated egocentric human pose estimation methods available due to several inherent challenges in egocentric vision as shown in Fig. 1(c). First, there exist significant human body self-occlusions from the first-person view, especially for the head and lower limbs, making the estimation of occluded joints difficult. Second, although an egocentric camera with a fisheye lens enlarges the field of view and captures more details of the human body, the recorded images are severely distorted, increasing the difficulty of accurate annotation in practice. Furthermore, there is a lack of real-world datasets with accurate ground truth data as data collection using a motion capture (MoCap) system is labor-intensive and limited to dedicated laboratory settings. To partially address these issues, recent effort within the vision community has been directed to building public datasets using synthetic human models [^14], [^15]. However, training with synthetic datasets can affect the generalization capability of the model when applied to real-world scenarios.
To circumvent the above problems, it is necessary to develop a dedicated method for egocentric 3D human pose estimation. Furthermore, it is advantageous to incorporate self-supervised learning. Some recent work has already leveraged the intrinsic constraints across multiple third-person views, such as multiview geometry and view consistency [^16], [^17], [^18], to enable 3D human pose estimation without ground truth. These
---
Fig. 1. Egocentric 3D pose estimation from a single fisheye camera. a) Our proposed EgoFish3D can achieve accurate 2D and 3D pose estimation from distorted images captured by a single fisheye camera. b) Existing third-person-view 2D/3D pose estimation methods [^10], [^11] fail on this challenging task from images captured by the fisheye camera. c) The strong self-occlusion for lower limbs, the severe distortion of the fisheye camera, and the lack of real-world datasets are the inherent challenges in egocentric vision.
self-supervised methods have demonstrated comparative pose estimation performance against those fully-supervised counterparts. In practice, however, there is still the challenge of directly transferring the multi-view self-supervised mechanism for egocentric 3D pose estimation. First, the intrinsic parameters of the third-person-view and egocentric cameras are different. Second, there is often limited overlap between the third-person-view and first-person-view images, and the transformation between two views is hard to acquire. More importantly, conventional 2D/3D pose estimation methods can hardly work on egocentric images, thus limiting the direct use of self-supervised learning with multi-view constraints.
To address the aforementioned challenges, we first constructed the EgoCentric Human Pose (ECHP) dataset using a head-mounted GoPro camera with a fisheye lens, and two RGB cameras were used to simultaneously capture images from a third-person view. The training and validation datasets of ECHP consist of 30 video sequences (~75 k frames) recorded in 8 different real-world indoor/outdoor scenes, in which 10 different daily actions performed by 9 subjects with 20 different body textures were recorded. The test dataset consisting of 7 video sequences (~17 k frames) about 10 actions performed by 4 subjects with new body textures was simultaneously captured by a multi-camera motion capture system with ground truth annotations. This dataset can not only contribute as a new benchmark for egocentric 3D pose estimation, but also help enhance the generalization capability of the proposed method to real-world scenarios. Central to this paper, we propose a novel self-supervised method, namely EgoFish3D, for egocentric 3D human pose estimation from a single head-mounted fisheye camera. An overview of our proposed method is shown in Fig. 2. The EgoFish3D consists of three modules: 1) the third-person-view module; 2) the egocentric module; and 3) the interactive module. The third-person-view module first generates the 3D pose from two third-person-view cameras, providing a relatively accurate 3D pose represented in a third-person-view coordinate system. The egocentric module then takes the distorted first-person-view image as input, performs 2D pose estimation, and predicts the 3D pose in the egocentric coordinate system as the final output. Within this module, the latent features, 2D heatmaps and human masks are incorporated in order to improve the accuracy. Another interactive module is introduced to estimate the 3D rotation of two 3D poses, adding supervision rules for training the other two modules. During the training phase, we train these three modules in a self-supervised manner. During the inference stage, only the egocentric module is used to predict the 3D pose from a single egocentric fisheye image. The proposed work represents the first attempt in achieving egocentric 3D pose estimation from a fisheye camera looking downwards in a self-supervised manner without 3D pose ground truth as prior. Experimental results on our ECHP and the public synthetic datasets [^14], [^15] demonstrate that our method can achieve good accuracy compared to existing supervised approaches.
In summary, the main contributions of this paper include:
- A self-supervised method is proposed to achieve egocentric 3D pose estimation from a single image without the need for 3D ground truth annotations.
- A real-world dataset ECHP is constructed, which contains the synchronized images from two third-person-view cameras and an egocentric fisheye camera.
An interactive module is introduced to learn the relationship between the third-person and egocentric views.
### II. RELATED WORK
### A. Human Pose Estimation From Egocentric Camera
With increasing popularity of egocentric vision [^7], human pose estimation from a single egocentric camera has received significant attention in recent years. Unlike the image/video captured by a camera from the third-person view, egocentric vision still faces inherent challenges originating from the visual data captured from a novel first-person viewpoint. In previous works, the egocentric cameras are typically placed on the chest [^4] or head [^14], [^15], [^19], [^20], [^21], [^22], looking outwards [^4], [^19], [^20] or looking downwards [^14], [^15], [^21].
To mimic the visual perception of a human, one line of research is based on the egocentric camera looking outwards, but suffering from difficulties for recovering the 3D human pose only with limited observed human body. Jiang et al. [^4] took advantage of the dynamic motion signatures of the surroundings to infer the invisible pose from a chest-mounted camera. However, the estimated poses are inaccurate and easily affected by the changing of environment. Another group of research [^19], [^20] modeled the egocentric 3D pose estimation and forecasted as a Markov decision processing when the pose estimation is limited to a single mode of action, such as running or walking.
Compared to cameras looking outwards, some recent works change the egocentric camera to a downward-looking setting, which can consistently capture the human body in the field of view. Moreover, they all integrate the camera with a fisheye
---
Fig. 2. Illustration of the training and inference phases of our proposed EgoFish3D. During the training phase, the third-person view module takes two images from the third-person view as input and generates a relatively accurate 3D pose represented in the coordinate system of the external cameras, the interactive module predicts the rotation difference between the third-person-view and the egocentric coordinate systems, and the egocentric module estimates both 2D and 3D poses from an egocentric distorted image. During the inference phase, only the egocentric module can directly predict 2D and 3D poses from an egocentric image captured by a fisheye camera.
lens to enlarge the perception area of human body, which can boost the performance of egocentric 3D human pose estimation. By using a head-mounted stereo fisheye camera, Rhodin et al. [^21] proposed a markerless egocentric full-body motion capture method, but the stereo camera is inconvenient in practical applications. Xu et al. [^15] presented a real-time egocentric 3D pose estimation method with a single cap-mounted fisheye camera, which takes both original and zoom-in images as input to deal with the strong occlusion of the lower limbs. However, this method does not directly regress the 3D pose in the inference phase but predicts the absolute depths of the joints instead. Most recently, Tome et al. [^14] introduced a large corpus of synthetic dataset from a head-mounted fisheye camera, and proposed a three-branch network that achieves the state-of-the-art 3D pose estimation performance on this synthetic dataset. To deal with the problem of the severe distortion, Zhang et al. [^22] proposed an automatic calibration module to estimate the fisheye camera parameters, thus mitigating the effect of image distortions for robust egocentric 3D pose estimation. It should be pointed out that the models proposed in [^14], [^15], [^22] are trained on synthetic data under strong supervision, leading to degraded generalization of the model in real-world applications. Motivated by this, in this paper we establish a real-world egocentric human action dataset. In addition, we take full advantage of the multi-view constraints between the third-person-view and egocentric cameras to achieve egocentric 3D human pose estimation in a self-supervised manner.
### B. 3D Human Pose Estimation Via Self-Supervised Learning Under Multi-View Constraints
Recently, self-supervised learning has attracted increasing attention in estimating 3D human pose, in which the multi-view information is utilized to mitigate the ambiguity of learning 3D human poses from synchronized 2D images captured from third-person-view cameras [^23], [^24], [^25], [^26], [^27]. However, the fusion of multiple views and the annotation of 3D poses in different camera views are challenging. Employing some typical fusion methods, such as multi-view consistency of the same pose [^16], [^23], [^26] and triangulation [^25], [^28], can reduce the cost of 3D human pose labelling and make the network learn in a self-supervised manner.
Rhodin et al. [^23] employed the multi-view consistency to constrain the system to predict the same pose in all views with the help of only few annotated data. CanonPose [^16] disentangled the observed 2D pose into a canonical 3D pose and a camera orientation. Specifically, it contains several sub-networks inferring the same 3D pose from different views, and the predictions from all views are aggregated to produce the final 3D pose inference. However, only applying the multi-view consistency constraint is not sufficient enough because the model may get trapped in a trivial solution with different inputs [^23], [^29]. Iqbal et al. [^29] introduced a novel objective function based on normalized 3D bone lengths that are computed from Human3.6 M dataset. Differently, Rhodin et al. [^30] took advantages of the temporal consistency prior to first learn a geometry-aware body representation from sequential unlabelled multi-view images, and then map the novel geometry representation to actual 3D poses. In this paper, we aim to address the multi-view self-supervised learning from the combination of third-person and egocentric views.
The other branch of study in fusing multi-view data employed conventional triangulation method, given the intrinsic and extrinsic parameters of the cameras [^25], [^28]. Iskakov et al. [^28] firstly proposed a baseline method that computes the 3D human pose from multi-view 2D poses algebraically. Kocabas et al. [^25] utilized the epipolar geometry theorem to generate 3D pose annotations from multiple 2D poses. Our method also incorporates the triangulated 3D pose as the prior for faster convergence and better performance.
### III. EGOCENTRIC HUMAN POSE (ECHP) DATASET
There is a pressing need for available real-world data to enable the development of egocentric pose estimation algorithms
---
due to the unique camera placement and the images captured from the first-person view. However, the existing training methods for egocentric pose estimation are almost based on synthetic datasets, and real-world data is only used for test. Due to the domain gap between the synthetic environment and the real-world environment, the generalization capability of the model in real applications is limited. More importantly, it is faced with difficulties for simultaneously capturing ground truth data using MoCap system for supervised learning. This is because the multi-camera MoCap system is limited in a certain environment and the annotation is labour-intensive.
In this paper, we first construct an EgoCentric Human Pose (ECHP) dataset to enable the egocentric 3D human pose estimation in a self-supervised learning manner. Unlike previous datasets that mainly consist of synthetic images [^14], [^15], our ECHP is composed of real-world images collected by an egocentric camera with a fisheye lens in different scenarios. Our dataset consists of data collected from 9 different subjects with 20 different body textures performing 10 daily actions. The third-person-view images are captured by two Intel Realsense D455 cameras and the egocentric images are captured by a head-mounted GoPro Hero9 equipped fisheye lens.
### A. Data Collection
To overcome the difficulty of acquiring ground truth data by a MoCap system, two RGB cameras are used to capture images from the third-person view and a head-mounted GoPro camera with a fisheye lens captures the images from the egocentric view. In this manner, the ECHP dataset consists of well-synchronized images captured from two different views, i.e., the third-person view and the egocentric view. In practice, the egocentric camera is fixed on the head through a helmet and extends forward about 13–18cm, and tilts downwards about 15–25 degrees to reduce the self-occlusion and ensure that the lower limbs can be seen as much as possible. We also use Aruco [^31] marker to obtain the 6D pose of the egocentric camera represented in the third-person-view frames. All three cameras are well calibrated with a 25 mm chessboard to determine the intrinsic parameters [^32].
In the training and validation sets of our ECHP dataset, there are 9 different subjects with 20 different body textures performing 10 daily actions (i.e., squatting, walking, dancing, stretching, waving, boxing, kicking, touching, clamping, knocking). To improve the diversity of the dataset, real-world data are captured in 8 different scenes, both indoor and outdoor. In total, there are 30 video sequences about 75 k frames in the training and validation parts of ECHP dataset. To fully evaluate the performance as well as the generalization capability of different egocentric pose estimation algorithms, in the test set we simultaneously use the VICON MoCap system with a full-body gait model to collect these 10 daily actions performed by 4 subjects with new body textures, i.e., 7 video sequences about 17 K frames with 3D ground truth of anatomical joint positions. It should be emphasized that the test data is captured in the same indoor scene due to the use of MoCap system, and more importantly 2 of 4 subjects are unseen in the training set.
In summary, the ECHP dataset can be divided into three different parts: about 65 k images with both third-person-view
TABLE I DETAILS OF OUR ECHP DATASET
| Seq. | Frames | Subj(Gen) | Split | Seq. | Frames | Subj(Gen) | Split |
| #1 | 2652 | 1(F) | train | #20 | 2727 | 1(F) | val |
| #2 | 2625 | 2(M) | train | #21 | 2656 | 3(M) | train |
| #3 | 2573 | 1(F) | val | #22 | 3524 | 3(M) | train |
| #4 | 2795 | 2(M) | train | #23 | 2653 | 3(M) | train |
| #5 | 2279 | 1(F) | train | #24 | 2086 | 4(M) | train |
| #6 | 2359 | 2(M) | train | #25 | 2596 | 2(M) | train |
| #7 | 2268 | 1(F) | train | #26 | 2125 | 5(M) | train |
| #8 | 2339 | 2(M) | train | #27 | 1831 | 6(M) | train |
| #9 | 2218 | 3(M) | train | #28 | 3128 | 7(M) | train |
| #10 | 3215 | 1(F) | train | #29 | 2508 | 8(M) | train |
| #11 | 2797 | 2(M) | train | #30 | 1844 | 9(F) | train |
| #12 | 2833 | 4(M) | val | #31 | 1942 | 4(M) | test |
| #13 | 2984 | 4(M) | train | #32 | 2220 | 4(M) | test |
| #14 | 2001 | 4(M) | train | #33 | 2900 | 10(M) | test |
| #15 | 2563 | 1(F) | train | #34 | 2434 | 10(M) | test |
| #16 | 2807 | 2(M) | train | #35 | 2978 | 2(M) | test |
| #17 | 2373 | 1(F) | train | #36 | 2440 | 2(M) | test |
| #18 | 2213 | 2(M) | val | #37 | 2364 | 11(M) | test |
| #19 | 2427 | 1(F) | val | | | | |
Seq - Sequence; Subj - Subject; Gen - Gender; Val - Validation.
images and egocentric images are for training, the remaining 10 k images are for validation, and the egocentric images along with 3D ground truth captured by the VICON Mocap system form the test dataset. Table I lists the exact frame number, subject number and gender, and the train/validation/test split of the egocentric images in each video sequence.
### B. Data Preparation
It is noteworthy that our main focus is on egocentric pose estimation, so we exploit OpenPose [^10] to offline extract 2D joints of the target human given the rgb images captured from third-person-view cameras. These two 2D poses are served as the input to the third-person-view module of EgoFish3D. Besides, a pretrained human instance segmentation model [^33] is used to offline extract the human mask as the input to our egocentric module. The extrinsic parameters between the two third-person-view cameras are pre-measured and fixed during the data collection. To fit in our proposed network, we resize the images from the third-person view to $640 \times 480$ and the images from the egocentric view to $384 \times 384$.
### IV. EgoFish3D for EgoCENTRIC 3D Pose Estimation
### A. Problem Statement
In this paper, our aim is to perform egocentric 3D pose estimation from a single head-mounted fisheye camera by leveraging self-supervision provided by two third-person-view cameras. Let us denote $\{C_1\}, \{C_2\}$ as the coordinate systems of two third-person-view cameras $c_1$, $c_2$ and $\{C_{ego}\}$ indicates the local coordinate system of the egocentric fisheye camera $c_{ego}$. For a camera c, the captured image at each frame can be defined as $I_c$, the corresponding 2D joints of the target human is
---
Fig. 3. The details of our proposed ECHP dataset. (a) Placement of two third-person-view cameras (rgb camera of Realsense D455) and a head-mounted egocentric camera (GoPro) with a fisheye lens looking downwards; (b) Selected examples of the different scenes, subjects, and body textures in our ECHP dataset. In total, we capture the data of 9 subjects in 8 different scenes with 20 different body textures; (c) 10 different daily actions are recorded in our dataset.
Fig. 4. Overview of our proposed EgoFish3D. The figure shows the network architecture, which contains three modules: third-person-view module, egocentric module and interactive module. The black arrows indicate direction of information flow. The colorized lines with arrows indicate different loss functions. The yellow dotted line indicates that two sub-networks share the same weights.
$J_c = \{j_{c,1}, j_{c,2}, \ldots, j_{c,N}\} \in \mathbb{R}^{N \times 2}$, where $j_{c,i} = [u_{c,i}, v_{c,i}]$ indicates the pixel coordinates of a joint i and N is the number of key body joints. Accordingly, the estimated 3D joints represented in the camera c is $P_c = \{p_{c,1}, p_{c,2}, \ldots, p_{c,N}\} \in \mathbb{R}^{N \times 3}$, where $p_{c,i} = [x_{c,i}, y_{c,i}, z_{c,i}]$. Note that we first exploit OpenPose [^10] to extract the 2D pose $J_{c_1}, J_{c_2}$ from two third-person-view images $\mathbf{I}_{c_1}, \mathbf{I}_{c_2}$, and the transformation matrix $\{c_2 R_{c_1}, c_2 T_{c_1}\}$ between two third-person-view cameras is given.
Given an image $I_{c_{ego}}$ captured by the egocentric fisheye camera, a deep neural network $f_\theta(\cdot)$ with parameters $\theta$ is designed to first predict the 2D joint positions $J_{c_{ego}}$, and then estimate the 3D human pose $P_{c_{ego}}$ represented in the egocentric camera coordinate system. In this paper, we propose EgoFish3D to perform this task via self-supervised learning.
### B. Overview of EgoFish3D
The architecture of our proposed EgoFish3D is shown in Fig. 4. Specifically, our network contains three modules, i.e., third-person-view module $f_{\theta}^{trd}$, egocentric module $f_{\theta}^{ego}$, and the interactive module $f_{\theta}^{itr}$.
The third-person-view module $f_{\theta}^{trd}$ achieves 3D pose estimation from two external cameras, which aims to offer a relatively accurate 3D pose as the supervision represented in the third-person-view camera coordinate system. The input of this module is from two 2D poses $J_{c_1}, J_{c_2}$ estimated from two third-person-view images and the output is the 3D poses $\hat{P}_{c_1}, \hat{P}_{c_2}$ under the third-person-view coordinate system. The network is composed of several MLPs, where each MLP is the combination of a linear layer, a batchnorm layer and an activation function.
For egocentric module $f_{\theta}^{ego}$, the input is a single egocentric image $\mathbf{I}_{e_{ego}}$. We first perform 2D pose estimation to obtain the heatmaps of the body joints, and then predict the 3D pose $\hat{P}_{e_{ego}}$ under the egocentric coordinate system. To tackle the inherent challenges of pose estimation in distorted egocentric images, a feature fusion mechanism is proposed by combining the high-level features $\hat{F}^{T}_{T_{e_{ego}}}$ of the input image, 2D heatmaps of body joints $\hat{H}^{M}_{e_{ego}}$ and human masks $\hat{M}_{e_{ego}}$ together. The module is first built upon the CNN backbone to achieve feature fusion. Then the fused features are fed into an encoder-regressor, a combination of several CNN layers and MLPs, to generate the final 3D pose estimation.
Noted that the 3D poses estimated by the above two modules are represented in different coordinate systems, hence, it is necessary to perform the rotation alignment between third-person-view and the egocentric coordinate systems
---
across frames. To this end, we introduce an interactive module in this paper, which takes the paired 2D poses $\{J_{c_1}, \hat{J}_{c_{ego}}\}$, $\{J_{c_2}, \hat{J}_{c_{ego}}\}$ from different coordinate systems as input. The network structure is similar to the third-person-view module and is composed of several MLPs, predicting the rotation matrices $c_1, \hat{R}_{c_{ego}}, c_2, \hat{R}_{c_{ego}}$, respectively.
### C. Third-Person-View Module
This module aims to predict 3D poses under the third-person-view camera coordinate system, by giving two third-person-view images and the transformation matrix between these two cameras. In order to generate accurate pose estimation, we combine the conventional triangulation via multi-view constraints and a learning-based depth estimation model.
1) 3D Pose Triangulation: Given the intrinsic parameters and transformation matrix of two cameras, a point in 3D space can be determined by triangulation with respect to its projections on two images. Among different triangulation algorithms, we use the depth estimation related one. After obtaining 2D poses $J_{c_1}, J_{c_2}$ via OpenPose [^10], the analytical solution of 3D positions $P_{c_1}^{tri}, P_{c_2,i}^{tri}$ can be calculated.
2) 3D Pose Estimation Via Depth Prediction: To avoid inaccuracies in 3D pose estimation by triangulation, we also introduce a network $f_{\theta}^{trd}$ to predict the depth values of 2D joints, i.e., $d_{c} = f_{\theta}^{trd}(J_{c})$. Thus, the 3D joints $\hat{P}_{c}$ represented in each camera coordinate system can be calculated by leveraging the intrinsic parameters of this camera.
3) Loss Functions: In this module, two constraints are involved to facilitate the training of the network. First, the constraint $\mathcal{L}_{pose}^{trd}$ between the depth-based prediction and the triangulation result is introduced.
$$ \mathcal{L}_{p o s e}^{t r d}=\sum_{c=c_{1},c_{2}}^{}\sum_{i=1}^{N}\left\|\hat{p}_{c,i}-p_{c,i}^{t r i}\right\|_{1} $$
Besides, two estimated 3D poses represented in $\{C_{1}\}$ and $\{C_{2}\}$ are expected to be the same after transformation.
$$ \mathcal{L}_{t r a n}^{t r d}=\sum_{i=1}^{N}\|\hat{p}_{c_{1},i}-({}^{c_{2}}R_{c_{1}}\hat{p}_{c_{2},i}+{}^{c_{2}}T_{c_{1}})\|_{1} $$
During the training, the total loss $\mathcal{L}^{trd}$ can be formulated as the weighted sum $\mathcal{L}^{trd} = \omega_1 \mathcal{L}_{pose}^{trd} + \omega_2 \mathcal{L}_{tran}^{trd}$.
### D. Egocentric Module
This module aims to predict both 2D and 3D poses of the target human under the egocentric camera coordinate system given a distorted fisheye image, where the 3D pose is the final output. To achieve better performance on 3D pose estimation, we propose a feature fusion method to combine the high-level features of the input image, the 2D heatmap of body joints and the mask of human body together. Moreover, the 2D heatmap, as well as the 2D pose branch is incorporated with the reprojection constraints by leveraging the 3D pose estimated from the third-person-view module.
1) 2D Pose and Heatmap Prediction by Reprojection: After obtaining the 3D joints from the third-person-view module, an intuitive way is to project the 3D data onto the distorted egocentric image plane, thus providing a supervised information for training the egocentric 2D pose detector. The intrinsic parameters of the fisheye camera are extracted by the calibration method [^32]. As is well known, it is extremely challenging to directly predict the transformation between two cameras with less overlap and different intrinsic parameters. For simplicity, in our ECHP dataset, we make use of the Aruco marker to determine the 6D pose $\{c_1 \tilde{R}_{c_{ego}}, c_1 \tilde{T}_{c_{ego}}\}$ of egocentric camera in the third-person-view coordinate system. However, the detection is affected by the illumination and distance from the marker to camera. Thus, we only use the images with stable detection of the Aruco marker to train the egocentric 2D pose detector. Given the $P_{c_1}^{tri}$ estimated from the third-person-view camera $c_1$, it can be reprojected on the egocentric image. Accordingly, we can get the egocentric 2D pose $J_{c_{ego}}^{rep}$ as well as the heatmap $H M_{c_{ego}}^{rep}$. Next, the egocentric 2D pose detector is trained to predict the 2D heatmaps $\hat{H} M_{c_{ego}}$ under the supervision of $H M_{c_{ego}}^{rep}$, where MSE loss $\mathcal{L}_{rep}^{ego}$ is used as constraints.
$$ \mathcal{L}_{r e p}^{e g o}=\sum_{i=1}^{N}\left\|\hat{H M}_{c_{e g o},i}-H M_{c_{e g o},i}^{r e p}\right\|_{2}^{2} $$
In practice, the reprojected 2D pose is relatively low-accuracy due to the triangulated 3D pose and extrinsic parameters. It is noteworthy that the 2D pose detector can predict more accurate heatmaps compared to the reprojected ones.
2) Information Fusion for Egocentric 3D Pose Estimation: The main objective of this paper is to train network $f_{\theta}^{ego}$ that can predict 3D pose $\hat{P}_{c_{ego}}$ given a single fisheye image. To achieve this, we propose a feature fusion mechanism in the egocentric module to boost the pose estimation performance, which consists of three main branches. The first branch uses the pretrained 2D pose detector to obtain the heatmaps $\hat{M}_{c_{ego}}$, the second branch extracts the latent features $\hat{F}^{\hat{T}}c_{ego}$ of the input image in the feature space, and the third branch uses a pretrained human instance segmentation network [^33] to extract the human masks $\hat{M}_{c_{ego}}$. The final fusion is built upon an attention-aware manner, i.e., $\hat{P}_{c_{ego}} = f_{\theta}^{ego}\left(\hat{H}\hat{M}_{c_{ego}} \odot \hat{F}^{\hat{T}}c_{ego} \odot \hat{M}_{c_{ego}}\right)$. Then the fused feature is fed into an encoder and regressor to predict the final 3D poses. During training, two self-supervised constraints are used. One is the reprojection loss $\mathcal{L}_{rep}^{ego}$ mentioned above to force the egocentric 2D detector to generate reasonable heatmaps. The other one is the transformation constraint $\mathcal{L}_{pose}^{ego}$ of 3D poses between the third-person view and the egocentric view, which is supervised by the rotation matrix $c_{1}$ $\hat{R}_{c_{ego}}$ predicted by the interactive module. Note that we choose $\{C_{1}\}$ as the reference frame. Besides, an additional loss $\mathcal{L}_{bone}^{ego}$ on bone length $\hat{b}$ is utilized to force the length of L left and right links to be the same.
$$ \mathcal{L}_{p o s e}^{e g o}=\sum_{i=1}^{N}\left\|\hat{p}_{c_{e g o},i}-{}^{c_{1}}\hat{R}_{c_{e g o}}\hat{p}_{c_{1},i}\right\|_{2} $$
---
$$ \mathcal{L}_{b o n e}^{e g o}=\sum_{i=1}^{L}\left\|\hat{b}_{c_{e g o},i}^{l e f t}-\hat{b}_{c_{e g o},i}^{r i g h t}\right\|_{2} $$
3) Loss Functions: During the training of the egocentric module, the total loss $\mathcal{L}^{ego}$ is the weighted sum $\mathcal{L}^{ego} = \alpha_1\mathcal{L}_{rep}^{ego} + \alpha_2\mathcal{L}_{pose}^{ego} + \alpha_3\mathcal{L}_{bone}^{ego}$.
### E. Interactive Module
As the detection of Aruco markers is greatly affected by the environment and easily fails in some circumstances, we propose the interactive module to automatically predict the rotation matrix between the third-person-view and egocentric coordinate systems, offering the transformation constraints for training the egocentric module. Given pair of 2D poses $\{J_c, \hat{J}_{e_go}\}$, where $c \in \{c_1, c_2\}$, the interactive module $f_{θ}^{itr}$ predicts the Euler angles $[\hat{\theta}, \hat{\phi}, \hat{\psi}]$ through simple MLPs. Accordingly, the rotation matrix $\hat{R}$ can be determined. By leveraging the extrinsic parameters of two third-person-view cameras, the transformation constraint $\mathcal{L}_{tran}^{itr}$ can be defined as follow.
$$ \mathcal{L}_{t r a n}^{i t r}=\left\|^{c_{1}}\hat{R}_{c_{e g o}}-^{c_{2}}R_{c_{1}}{}^{c_{2}}\hat{R}_{c_{e g o}}\right\|_{1} $$
To ensure a fast convergence of the interactive module, we use extrinsic parameters $^c\tilde{R}_{ego}$ predicted by Aruco markers to form constraint $\mathcal{L}_{mat}^{itr}$ at the beginning of the training.
$$ \mathcal{L}_{m a t}^{i t r}=\left\|^{c}\hat{R}_{c_{e g o}}-^{c}\tilde{R}_{c_{e g o}}\right\|_{1},c=c_{1},c_{2} $$
During training, the total loss $\mathcal{L}^{itr}$ can be formulated as a weighted sum $\mathcal{L}^{itr} = \beta_1 \mathcal{L}_{tran}^{itr} + \beta_2 \mathcal{L}_{mat}^{itr}$. Noted that $\mathcal{L}_{mat}^{itr}$ is only used in the first 5 epochs.
In summary, the loss function for training the whole network of our EgoFish3D is the sum of losses for the aforementioned modules, i.e., $\mathcal{L} = \mathcal{L}^{trd} + \mathcal{L}^{ego} + \mathcal{L}^{itr}$.
### V. EXPERIMENTAL SETTINGS
### A. Dataset
For the ECHP dataset as mentioned in Section III, we train the model based on 65 k frames from the sequences {seq1-seq2, seq4-seq11, seq13-seq17, seq21-seq30}, and validate our model on the rest of images from {seq3, seq12, seq18-seq20} to demonstrate qualitative results. To demonstrate the effectiveness of our proposed self-supervised method, we also capture seven test videos named {test1-test7} of 4 subjects (2 of them are novel subjects and all body textures are unseen in the training set) with 3D pose ground truth provided by the VICON system to provide quantitative results. Note that we do not capture the joint position of nose in VICON system, so we only report the results of 14 body joints.
To evaluate the effectiveness of our method, detailed comparison experiments on the other two existing synthetic datasets were conducted. 1) Mo $^{2}$Cap $^{2}$ [^15] is a large-scale synthetic dataset that simulates the images captured by a single cap-mounted fish-eye camera, which contains 530 k rendered images including about 3000 different actions and 700 different body textures.
Fig. 5. Examples from our proposed ECHP dataset and the public datasets, i.e., xR-EgoPose [^14] and Mo $^{2}$Cap $^{2}$ [^15]. Compared to xR-EgoPose and Mo2 $^{2}$Cap $^{2}$, both training and test data in the ECHP dataset are real-world images and suffer from less occlusions of lower limbs, improving the generalization capability of the learned models in practical applications.
Besides, there is a real-world test set consisting of ~5.5 k frames for quantitative evaluation. 2) xR-EgoPose [^14] is also a synthetic dataset captured by a head-mounted fisheye camera. It has 383 k frames with 23 male and 23 female characters performing 9 different actions. Both Mo²Cap² and xR-EgoPose datasets contain the ground truth annotations of 2D and 3D joint positions. Fig. 5 demonstrates several examples of the three datasets. Although our ECHP contains fewer images than Mo²Cap² and xR-EgoPose, we believe that the real-world egocentric images in our dataset can contribute to the community for the development of egocentric pose estimation algorithms.
### B. Implementation Details
Since our method is based on self-supervised learning method, the network may not converge with an end-to-end training strategy. To make our model converge faster and reduce overfitting, we adopt a multi-stage training strategy instead. First, we train the third-person-view module with 3D pose triangulation and depth estimation methods to estimate the 3D pose under the third-person-view coordinate system with $\omega_1 = 1.0$, $\omega_2 = 0.5$. Then the egocentric 2D pose detector is trained to estimate the heatmaps with $\alpha_1 = 1.0$. Note that the input egocentric image is of size $384 \times 384$ and the dimension of heatmap is $48 \times 48$. To improve the generalization of the egocentric 2D pose estimation model in different real-world scenarios, we train the model on a combination of ECHP ( $\sim40$ k with available 2D reprojection from triangulated 3D pose) and Mo $^2$Cap $^2$ ( $\sim40$ k synthetic data with annotated 2D pose). Next, we train the interactive module to estimate the rotation matrix between the third-person-view coordinate system and the egocentric one. In the first 5 epochs, we use extrinsic parameters estimated from Aruco as the supervision to speed up the convergence of the network with $\beta_1 = 1.0$, $\beta_2 = 1.0$. Finally, we train the egocentric module and finetune the whole network for better performance with $\alpha_2 = 1.0$, $\alpha_3 = 0.05$. The proposed method and comparison methods are implemented by PyTorch, and we apply Adam for optimization with a learning rate of 0.001.
### C. Evaluation Metrics
Three evaluation protocols are used in this paper. The first one refers to Mean Per Joint Position Error (MPJPE), which calculates the average distance between the ground truth and
---
the predicted 3D joints as in Eq. (8).
$$ E(P,\hat{P})=\frac{1}{K}\frac{1}{N}\sum_{k=1}^{K}\sum_{i=1}^{N}\left\|p_{i}^{k}-\hat{p}_{i}^{k}\right\|_{2} $$
The second one refers to PA-MPJPE, which indicates the MPJPE after applying alignment by Procrustes Analysis to remove the global translation, rotation and scale of the two 3D poses. The third one refers to BA-MPJPE, which resizes the bone length of the estimated 3D pose and the ground truth 3D pose to a standard skeleton to remove the influence of the body scale and then performs PA-MPJPE.
### D. Comparison Methods
Since we are the first to propose a self-supervised method for egocentric 3D pose estimation from a looking downwards fisheye camera by leveraging multi-view constraints from the third-person view, we compare our proposed EgoFish3D with four existing supervised methods [^14], [^15], [^22], [^34], that use egocentric images with ground truth annotations. First, the comparison experiments are conducted on our ECHP dataset. For a fair comparison, we keep the third-person-view module and the interactive module consistent to generate the same pseudo labels and use them to train different methods. For the network proposed by Martinez [^34], we implement the network after extracting the 2D pose from the heatmaps we predicted. Since the code of [^14], [^15] are not publicly available, we implement these two models by ourselves instead. As it is difficult to determine the rotations between body parts in real-world data, only the first and the third branches of the network proposed by Tome [^14] is implemented for the comparison. For the network proposed by Xu [^15], we implement the heatmap-zoom and joint depth estimation modules, and generate the 3D pose by the reprojection formula of the egocentric fisheye camera. Second, we compare our proposed EgoFish3D with other methods on the Mo²Cap² [^14] and xR-EgoPose [^15] datasets. To compare the performance of 3D pose estimation, we retrain our egocentric view module on xR-EgoPose and Mo²Cap² datasets to report both quantitative and qualitative results. Third, to illustrate the generalization ability of our EgoFish3D, we directly apply our model trained on our ECHP dataset to the real-world test data in Mo²Cap² without finetuning on the synthetic dataset and demonstrate the qualitative results. The main comparison methods on our dataset are listed as follows.
Martinez [^34], a baseline method with several MLPs for 3D pose estimation, where the input is 2D pose.
- Tome [^14], a state-of-the-art supervised method with encoder-decoder network for egocentric 3D pose estimation.
- Xu [^15], a two-branch network that takes both original and zoom-in images as input for supervised pose estimation.
- Zhang [^22], a two-branch network introducing an automatic calibration module to deal with the distortion of fish-eye lens.
EgoFish3D, the full network of our proposed method.
### E. Ablation Study
We also conduct ablation studies of the proposed EgoFish3D to demonstrate the effectiveness of different sub-networks and loss functions. We remove or change the following parts of our network one by one.
- EgoFish3D, the full network of our proposed method.
w/o $\tilde{M}$, an ablated model by removing the human instance segmentation $\hat{M}_{c_{ego}}$.
w/o $\hat{F}T$, an ablated model by removing the branch of feature extraction $\hat{F}T_{c_{eqo}}$.
w/o $\hat{H}M$, an ablation study that removes the heatmap prediction branch $\hat{H}M_{ceqo}$ in the egocentric view module.
w/o $f_{\theta}^{trd}$, an ablated study removing the third-person-view module $f_{\theta}^{trd}$. Only triangulated 3D pose is for supervision.
w/o $\mathcal{L}_{pose}^{trd}$, the loss constraining the depth-based and triangulated poses in the third-person-view module is removed.
w/o $\mathcal{L}_{mat}^{itr}$, an ablated model without $\mathcal{L}_{mat}^{itr}$ as in (7).
### VI. EXPERIMENTAL RESULTS
### A. Quantitative Results
Without further clarify, the bold and $\underline{\text{underline}}$ values in the table indicate the best and the second best results in each column, respectively. All the elements indicate the result in millimeters (mm). By leveraging a full-body gait model provided by VICON MoCap system, the ground truth annotations are the 3D joint positions from the anatomical level.
1) ECHP Dataset: The comparison of 3D pose estimation results with other state-of-the-art methods on our ECHP dataset are reported in the upper part in Table II. The second column lists the average MPJPE(PA-MPJPE) results of all test data, and the remaining shows the results for each action.
It can be seen that our proposed EgoFish3D achieves the best average 3D pose estimation results (MPJPE = 107.9) under the evaluation protocol MPJPE against other comparison methods [^14], [^15], [^34]. This is because that our egocentric feature fusion method can leverage more useful information implied in the input image. For instance, the human mask is beneficial for removing the heatmaps with low confidence or beyond the scope of human body. It can also be found our method achieves the comparable performance (PA-MPJPE = 73.1) with Xu [^15] under the evaluation protocol PA-MPJPE. It should be noted that the method proposed by Xu [^15] is sensitive to the distorted egocentric images captured from a fisheye lens. Compared to our EgoFish3D, it requires the intrinsic parameters of the fisheye camera to compute the 3D pose during the inference phase. Hence, the performance of this method will be significantly degraded without the known intrinsic parameters of the camera. For action-level comparison, our EgoFish3D can still achieve the best or second best pose estimation for nine actions, except for Boxing with upper limb outside the field of view, which proves that our method is more stable to different actions than the comparison methods.
We also report the estimation error of each joint in Table V. It can be seen that distal joints (e.g., hands) are with larger PJPE and they become smaller after Procrustes analysis. This is mainly
---
TABLE II
COMPARISON MPJPE (PA-MPJPE) RESULTS OF EGOCENTRIC 3D POSE ESTIMATION IN MILLIMETERS (mm) ON ECHP DATASET
| Approach | All | Squatting | Walking | Dancing | Stretching | Waving | Boxing | Kicking | Touching | Clamping | Knocking |
| Martinez [^34] | 118.3(80.0) | 135.5(75.9) | 114.8(73.9) | 122.9(84.3) | 133.2(85.0) | 107.9(76.1) | 116.7(82.2) | 102.6(71.8) | 117.9(79.0) | 115.9(93.3) | 116.4(84.6) |
| Tome [^14] | 112.4(73.9) | 136.3(78.4) | 113.9(72.6) | 119.5(79.8) | 123.9(78.2) | 99.0(63.9) | 110.3(71.9) | 100.2(73.1) | 110.7(67.7) | 104.3(81.1) | 103.4(72.0) |
| Xu $^{\dagger}$ [^15] | 110.9(71.2) | 112.4(70.7) | 114.3(76.8) | 108.5(70.1) | 114.7(69.5) | 105.9(65.5) | 109.5(68.1) | 106.7(72.0) | 102.2(65.8) | 128.0(85.9) | 106.6(66.0) |
| EgoFish3D | 107.9(73.1) | 123.8(71.2) | 106.8(68.9) | 110.4(80.1) | 121.4(81.3) | 95.6(66.4) | 111.2(75.4) | 94.6(69.3) | 110.5(70.0) | 101.6(80.8) | 102.7(71.4) |
| Ablated models | All | Squatting | Walking | Dancing | Stretching | Waving | Boxing | Kicking | Touching | Clamping | Knocking |
| A: EgoFish3D | 107.9(73.1) | 123.8(71.2) | 106.8(68.9) | 110.4(80.1) | 121.4(81.3) | 95.6(66.4) | 111.2(75.4) | 94.6(69.3) | 110.5(70.0) | 101.6(80.8) | 102.7(71.4) |
| B: w/o $\hat{M}$ | 114.1(82.5) | 133.7(80.3) | 110.4(71.3) | 113.0(82.6) | 127.8(92.0) | 108.3(85.3) | 116.9(88.0) | 101.7(75.5) | 109.0(83.3) | 106.0(87.0) | 112.3(87.7) |
| C: w/o $\hat{F}$T | 114.5(73.5) | 135.8(77.5) | 116.6(70.2) | 120.2(79.9) | 126.8(80.5) | 102.0(64.5) | 113.4(73.5) | 101.2(69.3) | 110.9(68.9) | 107.3(81.9) | 107.6(71.1) |
| D: w/o $\hat{H}$M | 123.9(82.3) | 150.0(91.2) | 119.9(81.6) | 131.2(91.9) | 128.7(85.6) | 120.5(73.6) | 115.7(77.3) | 120.2(86.3) | 117.8(74.8) | 128.4(87.2) | 111.8(74.0) |
| E: w/o $\hat{t}_{b}^{trd}$ | 114.4(78.3) | 127.0(80.0) | 112.0(76.6) | 111.0(78.2) | 126.7(79.3) | 102.6(69.1) | 114.0(75.3) | 104.0(78.7) | 117.0(72.8) | 117.4(94.4) | 113.0(79.1) |
| F: w/o $\mathcal{L}_{pose}^{trd}$ | 111.1(79.5) | 148.1(86.1) | 114.1(87.8) | 115.9(88.6) | 117.1(82.0) | 92.5(64.3) | 111.3(77.4) | 98.5(78.5) | 106.1(68.5) | 99.9(76.6) | 101.0(74.9) |
| G: w/o $\mathcal{L}_{mat}^{tr}$ | 521.1(78.7) | 481.2(75.4) | 510.7(76.7) | 478.2(83.7) | 572.7(83.0) | 532.9(70.5) | 563.0(76.4) | 466.7(80.7) | 519.7(76.3) | 525.4(88.5) | 558.1(77.2) |
$^{\dagger}$ Require additional information (i.e., the intrinsic parameters of the fisheye camera) during the inference phase.
TABLE III
COMPARISON MPJPE RESULTS OF EGOCENTRIC 3D POSE ESTIMATION IN MILLIMETERS (mm) ON XR-EGOPOSE DATASET
| Approach | All | Gaming | Gesticulating | Greeting | Lower Stretching | Patting | Reacting | Talking | Upper Stretching | Walking |
| Martinez [^34] | 122.1 | 109.6 | 105.4 | 119.3 | 125.8 | 93.0 | 119.7 | 111.1 | 124.5 | 130.5 |
| Tome(p3d) [^14] | 130.4 | 138.3 | 108.5 | 100.3 | 133.3 | 117.8 | 175.6 | 93.5 | 129.0 | 131.9 |
| Tome(p3d+hm) [^14] | 58.2 | 56.0 | 50.2 | 44.6 | 51.1 | 59.4 | 60.8 | 43.9 | 53.9 | 57.7 |
| Tome(p3d+hm+rot) [^14] | 54.7 | 60.4 | 54.6 | 44.7 | 56.5 | 57.7 | 52.7 | 56.4 | 53.6 | 55.4 |
| Zhang [^22] | 50.0 | 36.8 | 34.1 | 36.7 | 50.1 | 57.2 | 34.4 | 32.8 | 54.3 | 52.6 |
| EgoFish3D | 48.0 | 48.2 | 42.5 | 47.3 | 48.8 | 53.6 | 47.2 | 36.2 | 48.9 | 46.1 |
TABLE IV
COMPARISON BA-MPJPE RESULTS OF EGOCENTRIC 3D POSE ESTIMATION IN MILLIMETERS (mm) ON MO $^{2}$CAP $^{2}$ DATASET
| Approach/Indoor | All | Walking | Sitting | Crawling | Crouching | Boxing | Dancing | Stretching | Waving |
| 3DV'17 [^35] | 76.3 | 48.8 | 101.2 | 119.0 | 94.9 | 57.3 | 61.0 | 111.4 | 64.5 |
| VNext [^36] | 97.8 | 65.3 | 130.0 | 133.1 | 120.4 | 78.4 | 82.5 | 153.2 | 83.9 |
| Zhang [^22] | 62.13 | 41.2 | 76.6 | 73.0 | 89.7 | 53.0 | 58.9 | 92.2 | 71.6 |
| Xu [^15] | 61.4 | 38.4 | 70.9 | 94.3 | 81.9 | 48.6 | 55.2 | 99.3 | 60.9 |
| EgoFish3D | 60.9 | 42.1 | 65.0 | 82.7 | 79.0 | 55.5 | 59.1 | 94.5 | 66.8 |
| Approach/Outdoor | All | Walking | Sitting | Crawling | Crouching | Boxing | Dancing | Stretching | Waving |
| 3DV'17 [^35] | 92.5 | 68.7 | 114.9 | 113.2 | 118.5 | 95.3 | 73.0 | 144.5 | 72.4 |
| VNext [^36] | 113.4 | 84.4 | 167.9 | 138.4 | 154.5 | 108.4 | 85.0 | 160.6 | 96.2 |
| Xu [^15] | 80.6 | 63.1 | 85.5 | 96.6 | 92.9 | 96.0 | 68.4 | 123.6 | 61.4 |
| EgoFish3D | 79.2 | 59.4 | 115.9 | 89.7 | 79.2 | 101.6 | 66.9 | 107.2 | 71.4 |
because the precise estimation of rotation across third-person and egocentric views is quite challenging. We also evaluate the performance of our proposed interactive module. We report mean angle error to evaluate the estimation of the camera orientation, which is $6.02^{\circ}$ for our interactive module.
2) xR-EgoPose Dataset: For xR-EgoPose dataset, the comparison MPJPE results are presented in Table III. We retrain our model on their synthetic dataset and compare the performance with three existing methods [^14], [^22], [^34]. Our method achieves the best average performance (MPJPE = 48.0) compared to Tome [^14] (MPJPE = 54.7) and Zhang [^22] (MPJPE = 50.0). It can be found that our method implements the same supervision cues as Tome(p3d+hm), i.e., 2D heatmaps and 3D pose, to train the model but achieves the superior performance than the three-branch network Tome(p3d+hm+rot) containing the extra supervised information of the relative rotations between body joints, which can not be easily acquired in real-world data. For different actions, our model achieves the best performance on four actions and the second best performance on four actions as well among the nine actions.
3) $Mo^{2}Cap^{2}$ Dataset: The results of different methods on $Mo^{2}Cap^{2}$ dataset are reported in Table IV. We retrain our model on this synthetic dataset and test our model on the public real-world egocentric images. We follow the evaluation protocol as [^15] and report BA-MPJPE. For the data captured in the indoor environment, we compare our method with four different methods [^15], [^22], [^35], [^36] and our method achieves the comparable performance (BA-MPJPE = 60.9) with Xu [^15] and outperforms the other three methods. For different actions, our method achieves the best or second best performance on five
---
TABLE V ERROR PER JOINT IN MILLIMETER
| Joint | PJPE | PA-PJPE |
| Right Shoulder | 96.7 | 49.1 |
| Right Elbow | 105.5 | 68.1 |
| Right Hand | 167.5 | 134.4 |
| Right Hip | 53.7 | 63.4 |
| Right Knee | 103.6 | 66.0 |
| Right Ankle | 128.9 | 69.7 |
| Left Shoulder | 95.6 | 69.0 |
| Left Elbow | 150.5 | 62.1 |
| Left Hand | 232.3 | 136.5 |
| Left Hip | 32.0 | 64.0 |
| Left Knee | 101.3 | 68.9 |
| Left Ankle | 142.3 | 81.2 |
PJPE - Per Joint Position Error PA - Procrustes Analysis
TABLE VI
RESULTS OF OUR PROPOSED THIRD-PERSON-VIEW MODULE COMPARED WITH DIFFERENT METHODS
| Approach | PA-MPJPE(mm) |
| Triangulation | 69.19 |
| Triangulation+smooth | 67.29 |
| PARE [^11] | 66.92 |
| Ours(third-person-view module) | 63.26 |
Fig. 6. Human body joints (N = 15) for ECHP dataset.
of eight actions. For the data captured in the outdoor environment, we make the comparison with three methods [^15], [^35], [^36] and the performance of our method (BA-MPJPE = 79.2) is the best. It can also be found that our method achieves the best performance on five of eight actions.
4) Ablation Study: The lower part of Table II lists the results of ablation studies on ECHP dataset. We report the average MPJPE(PA-MPJPE) results of all test data in the second column, and the rest shows the results for each action. It can be found that, compared to the other six ablated models B-G, our full network (model A) achieves the best on both MPJPE and PA-MPJPE. For action-level comparison, our full network can achieve the best or second best pose estimation for nine actions, which significantly outperforms other models.
Does the feature fusion method work? To evaluate the effectiveness of the proposed three-branch feature fusion mechanism in the egocentric view module, we remove each branch one by one corresponding to the models B-D. The results show that all
(b)
Fig. 7. Visualization results of egocentric 2D pose estimation by our proposed EgoFish3D. (a) On ECHP dataset; (b) On Mo $^{2}$Cap $^{2}$ dataset. The red points are the predicted joint positions and the colorized lines indicate the skeleton.
(a)
(b)
Fig. 8. Visualization of 2D pose by the proposed (a) egocentric module and (b) reprojection of the transformed 3D pose. The egocentric module can generate more accurate 2D pose than the reprojected one, especially for the joints of the upper limb.
three branches contribute to improve the pose estimation performance. The heatmap $\hat{H}M_{c_{ego}}$ introduces the prior of 2D joints, the mask $\hat{M}_{c_{ego}}$ removes the heatmap with low confidence or out of human body, and the latent feature $\hat{F}T_{c_{ego}}$ preserves the original information from images and compensates for the incorrect 2D pose detection.
Can we directly use the triangulated 3D pose for supervision? We perform the ablation study by removing the third-person-view module and directly apply the triangulated 3D pose as the
---
Fig. 9. Visualization of the egocentric 3D pose estimation by our EgoFish3D. (a) On ECHP dataset. Left: visualization results on the test set (two bottom rows are new subjects and all body textures are unseen in the training set); Right: visualization results on validation set. (b) On xR-EgoPose dataset. We retrain our model on their dataset. (c) On Mo²Cap² dataset. We directly apply our model trained on ECHP dataset to their dataset. The red color is the predicted 3D pose by our EgoFish3D and the blue color represents the ground truth.
supervision. As shown in model E, the performance is clearly worse than that of the full network architecture. This originates from the inaccurate 2D poses and trivial solution of the triangulated 3D pose in some circumstances. More importantly, with a well-trained third-person-view module, we can ease the data collection procedure by using only one third-person-view camera. We also provide more quantitative results of our proposed third-person-view module to prove the effectiveness of the designed regressor instead of the triangulation method in Table VI. We captured more third-person-view images along with the VICON system to perform the third-person-view pose estimation and compare our proposed third-person-view module with three methods, i.e., triangulation method and triangulation with temporal smoothing as well as the state-of-the-art third-person pose estimation method PARE [^11]. During inference, we only feed the exocentric images captured from a single-view camera to the third-person-view module. It can be found that our third-person-view module achieves the best performance compared to others, which highlights the effectiveness of the third-person-view module and it can generate relatively accurate weak labels.
Does the coarse prior knowledge help train the network? The models F and G highlight the significance of the prior knowledge on the 3D pose by triangulation and the rotation matrix by Aruco markers. With the help of these priors, it can be seen that our full network (model A) can consistently improve the performance. Especially for $\mathcal{L}_{mat}^{itr}$ calculated by Aruco-based rotation, the network without this loss can hardly estimate the 3D pose under the egocentric coordinate system.
### B. Qualitative Results
Fig. 7 demonstrates the visualization results of egocentric 2D pose estimation on ECHP and Mo²Cap² datasets by our proposed self-supervised learning based method. For ECHP dataset as in Fig. 7(a), the proposed EgoFish3D can predict relatively accurate 2D pose even with the occlusion of lower limbs. The generalization ability of the 2D pose predicted by our model on Mo²Cap² dataset is exhibited in Fig. 7(b), where the 2D pose estimator is trained by mixing ECHP dataset with a small number of synthetic data (~40 k) in Mo²Cap² dataset, without further finetuning on this dataset. Besides, compared to the reprojected 2D pose as shown in Fig. 8, our egocentric module can predict
---
more accurate 2D pose than direct reprojections, which proves the effectiveness of the reprojection loss $\mathcal{L}_{rep}^{ego}$.
The egocentric 3D pose estimation results by our method on our ECHP, xR-EgoPose, and Mo²Cap² datasets are presented in Fig. 9(a)–(c), respectively. To explore the generalization ability of our proposed method, we directly apply our trained model to the Mo²Cap² dataset without finetuning. We retrain our EgoFish3D on xR-EgoPose [^14] to conduct comparison on xR-EgoPose dataset. Given a single egocentric image captured by a fisheye lens, it can be seen that the proposed EgoFish3D can predict 3D pose for different actions, even on unseen subjects & textures and some occluded body parts.
### VII. CONCLUSION AND FUTURE WORK
This paper proposes a self-supervised egocentric pose estimation method called EgoFish3D to estimate both the 2D pose and 3D pose under the egocentric view from a single RGB image. This is achieved by leveraging the potential information from both the third-person view and the egocentric view. Specifically, the EgoFish3D incorporates three main modules: the third-person-view module, the egocentric module and the interactive module to improve the performance of our self-supervised method. This paper also proposes a real-world EgoCentric Human Pose dataset called ECHP to capture images from three different cameras circumventing the use of MoCap system to acquire the ground truth. Our experimental results demonstrate that our EgoFish3D can predict relatively accurate 2D and 3D pose. In future work, we aim to make our approach generalize well to the different placement of the egocentric camera and incorporate the human pose estimation with more egocentric vision tasks.
## REFERENCES
[^1]: S. Ardeshir and A. Borji, “Egocentric meets top-view,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 41, no. 6, pp. 1353–1366, Jun. 2019.
[^2]: A. Bandini and J. Zariffa, “Analysis of the hands in egocentric vision: A survey,” IEEE Trans. Pattern Anal. Mach. Intell., early access, Apr. 13, 2020, doi: 10.1109/TPAMI.2020.2986648.
[^3]: Y. Huang, X. Yang, J. Gao, and C. Xu, “Holographic feature learning of egocentric-exocentric videos for multi-domain action recognition,” IEEE Trans. Multimedia, vol. 24, pp. 2273–2286, 2022.
[^4]: H. Jiang and K. Grauman, “Seeing invisible poses: Estimating 3D body pose from egocentric video,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2017, pp. 3501–3509.
[^5]: M. Lu et al., “Deep attention network for egocentric action recognition,” IEEE Trans. Image Process., vol. 28, no. 8, pp. 3703–3713, Aug. 2019.
[^6]: Y. Zhang, C. Cao, J. Cheng, and H. Lu, “Egogesture: A new dataset and benchmark for egocentric hand gesture recognition,” IEEE Trans. Multimedia, vol. 20, no. 5, pp. 1038–1050, May 2018.
[^7]: S. Alletto, G. Serra, S. Calderara, and R. Cucchiara, “Understanding social relationships in egocentric vision,” Pattern Recognit., vol. 48, no. 12, pp. 4082–4096, 2015.
[^8]: F. Ragusa, A. Furnari, S. Battiato, G. Signorello, and G. M. Farinella, “EGO-CH: Dataset and fundamental tasks for visitors behavioral understanding using egocentric vision,” Pattern Recognit. Lett., vol. 131, pp. 150–157, 2020.
[^9]: J. Yang, Y. Liu, X. Gu, G.-Z. Yang, and Y. Guo, “POSESDF: Simultaneous 3D human shape reconstruction and gait pose estimation using signed distance functions,” in Proc. IEEE Int. Conf. Robot. Automat., 2022, pp. 1297–1303.
[^10]: Z. Cao, T. Simon, S.-E. Wei, and Y. Sheikh, “Openpose: Realtime multi-person 2D pose estimation using part affinity fields,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 43, no. 1, pp. 172–186, Jan. 2021.
[^11]: M. Kocabas, C.-H. P. Huang, O. Hilliges, and M. J. Black, “PARE: Part attention regressor for 3D human body estimation,” in Proc. IEEE/CVF Int. Conf. Comput. Vis., 2021, pp. 11127–11137.
[^12]: Q. Zhang et al., “Single person dense pose estimation via geometric equivariance consistency,” IEEE Trans. Multimedia, vol. 25, pp. 572–583, 2023.
[^13]: W. Li et al., “Exploiting temporal contexts with strided transformer for 3D human pose estimation,” IEEE Trans. Multimedia, early access, Jan. 07, 2022, doi: 10.1109/TMM.2022.3141231.
[^14]: D. Tome et al., “SELFPOSE: 3D egocentric pose estimation from a headset mounted camera,” IEEE Trans. Pattern Anal. Mach. Intell., early access, Oct. 8, 2020, doi: 10.1109/TPAMI.2020.3029700.
[^15]: W. Xu et al., “Mo $^{2}$ cap $^{2}$: Real-time mobile 3D motion capture with a cap-mounted fisheye camera,” IEEE Trans. Vis. Comput. Graph., vol. 25, no. 5, pp. 2093–2101, May 2019.
[^16]: B. Wandt, M. Rudolph, P. Zell, H. Rhodin, and B. Rosenhahn, “Canonpose: Self-supervised monocular 3D human pose estimation in the wild,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2021, pp. 13294–13304.
[^17]: I. Chang, M.-G. Park, J. Kim, and J. H. Yoon, “Multi-view 3D human pose estimation with self-supervised learning,” in Proc. IEEE Int. Conf. Artif. Intell. Inf. Commun., 2021, pp. 255–257.
[^18]: A. Bouazizi, J. Wiederer, U. Kressel, and V. Belagiannis, “Self-supervised 3D human pose estimation with multiple-view geometry,” in Proc. IEEE Int. Conf. Autom. Face Gesture Recognition., 2021, pp. 1–8.
[^19]: Y. Yuan and K. Kitani, “Ego-pose estimation and forecasting as real-time PD control,” in Proc. IEEE/CVF Int. Conf. Comput. Vis., 2019, pp. 10082–10092.
[^20]: Z. Luo, R. Hachiuma, Y. Yuan, and K. Kitani, “Dynamics-regulated kinematic policy for egocentric pose estimation,” in Adv. Neural Inf. Process. Syst., pp. 25019–25032, 2021.
[^21]: H. Rhodin et al., “Egocap: Egocentric marker-less motion capture with two fisheye cameras,” ACM Trans. Graph., vol. 35, no. 6, pp. 1–11, 2016.
[^22]: Y. Zhang, S. You, and T. Gevers, “Automatic calibration of the fisheye camera for egocentric 3D human pose estimation from a single image,” in Proc. IEEE/CVF Winter Conf. Appl. Comput. Vis., 2021, pp. 1772–1781.
[^23]: H. Rhodin et al., “Learning monocular 3D human pose estimation from multi-view images,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2018, pp. 8437–8446.
[^24]: X. Chen, K.-Y. Lin, W. Liu, C. Qian, and L. Lin, "Weakly-supervised discovery of geometry-aware representation for 3D human pose estimation," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2019, pp. 10895–10904.
[^25]: M. Kocabas, S. Karagoz, and E. Akbas, “Self-supervised learning of 3D human pose using multi-view geometry,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2019, pp. 1077–1086.
[^26]: C.-H. Chen et al., “Unsupervised 3D pose estimation with geometric self-supervision,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2019, pp. 5714–5724.
[^27]: X. Gu et al., “Occlusion-invariant rotation-equivariant semi-supervised depth based cross-view gait pose estimation,” 2021, arXiv:2109.01397.
[^28]: K. Iskakov, E. Burkov, V. Lempitsky, and Y. Malkov, “Learnable triangulation of human pose,” in Proc. IEEE/CVF Int. Conf. Comput. Vis., 2019, pp. 7718–7727.
[^29]: U. Iqbal, P. Molchanov, and J. Kautz, “Weakly-supervised 3 d human pose learning via multi-view images in the wild,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2020, pp. 5243–5252.
[^30]: H. Rhodin, M. Salzmann, and P. Fua, “Unsupervised geometry-aware representation for 3D human pose estimation,” in Proc. IEEE Eur. Conf. Comput. Vis., 2018, pp. 750–767.
[^31]: S. Garrido-Jurado, R. Muñoz-Salinas, F. J. Madrid-Cuevas, and M. J. Marín-Jiménez, “Automatic generation and detection of highly reliable fiducial markers under occlusion,” Pattern Recognit., vol. 47, no. 6, pp. 2280–2292, 2014.
[^32]: P. I. Corke, “The machine vision toolbox: A MATLAB toolbox for vision and vision-based control,” IEEE Robot. Autom. Mag., vol. 12, no. 4, pp. 16–25, Dec. 2005.
[^33]: P. Li, Y. Xu, Y. Wei, and Y. Yang, “Self-correction for human parsing,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 44, no. 6, pp. 3260–3271, Jun. 2022.
[^34]: J. Martinez, R. Hossain, J. Romero, and J. J. Little, “A simple yet effective baseline for 3D human pose estimation,” in Proc. IEEE/CVF Int. Conf. Comput. Vis., 2017, pp. 2640–2649.
[^35]: D. Mehta et al., “Monocular 3D human pose estimation in the wild using improved CNN supervision,” in Proc. IEEE Int. Conf. 3D Vis., 2017, pp. 506–516.
[^36]: D. Mehta et al., “Vnect: Real-time 3D human pose estimation with a single RGB camera,” ACM Trans. Graph., vol. 36, no. 4, pp. 1–14, 2017.