# HumanoidPano: Hybrid Spherical Panoramic-LiDAR Cross-Modal Perception for Humanoid Robots Qiang Zhang$^{1,2,*}$, Zhang Zhang$^{1,4,*}$, Wei Cui$^{1,*}$, Jingkai Sun$^{1,2}$, Jiahang Cao$^{1,2}$, Yijie Guo$^{1}$, Gang Han$^{1}$, Wen Zhao$^{1}$, Jiaxue Wang$^{1,2,3}$, Chenghao Sun$^{1}$, Lingfeng Zhang$^{2}$, Hao Cheng$^{2}$, Yujie Chen$^{5}$, Lin Wang$^{6}$, Jian Tang$^{1,\dagger}$, Renjing Xu$^{2,\dagger}$ $^{1}$ Beijing Innovation Center of Humanoid Robotics $^{2}$ Hong Kong University of Science and Technology (Guangzhou) $^{3}$ Hong Kong University of Science and Technology $^{4}$ Beijing Institute of Technology $^{5}$ ETH Zurich $^{6}$ Nanyang Technological University $^{*}$ Contributed equally. † Corresponding authors. ![Image](images/f229958e587888ee38e099f48fecf376dab6a4222d543328fda563f1a227e081.jpg)
Figure 1. The humanoid robot autonomously navigates complex environments using HumanoidPano, which fuses panoramic vision and LiDAR to generate real-time BEV semantic maps. The figure depicts motion trajectory while visualizing real-time visual sensor data and perception results in the upper-left corner, alongside detailed illustrations of the HumanoidPano universal perception module.
## Abstract The perceptual system design for humanoid robots poses unique challenges due to inherent structural con- straints that cause severe self-occlusion and limited field-of-view (FOV). We present HumanoidPano, a novel hybrid cross-modal perception framework that synergistically integrates panoramic vision and LiDAR sensing to over- --- come these limitations. Unlike conventional robot perception systems that rely on monocular cameras or standard multi-sensor configurations, our method establishes geometrically-aware modality alignment through a spherical vision transformer, enabling seamless fusion of $360^{\circ}$ visual context with LiDAR's precise depth measurements. First, Spherical Geometry-aware Constraints (SGC) leverage panoramic camera ray properties to guide distortion-regularized sampling offsets for geometric alignment. Second, Spatial Deformable Attention (SDA) aggregates hierarchical 3D features via spherical offsets, enabling efficient $360^{\circ}$-to-BEV fusion with geometrically complete object representations. Third, Panoramic Augmentation (AUG) combines cross-view transformations and semantic alignment to enhance BEV-panoramic feature consistency during data augmentation. Extensive evaluations demonstrate state-of-the-art performance on the 360BEV-Matterport benchmark. Real-world deployment on humanoid platforms validates the system's capability to generate accurate BEV segmentation maps through panoramic-LiDAR co-perception, directly enabling downstream navigation tasks in complex environments. Our work establishes a new paradigm for embodied perception in humanoid robotics. The codes will be released soon. ### 1. Introduction Robotic environmental perception driven by computer vision has achieved revolutionary advancements[^3] [^9] [^15] [^17] [^26] [^27] [^36] [^44] [^51] [^53], with groundbreaking innovations profoundly reshaping technological landscapes in core applications such as autonomous driving and industrial automation. Current industry standard solutions predominantly rely on rigid, occlusion-free sensor configuration assumptions: Autonomous vehicles acquire unobstructed omnidirectional perception through rigid roof mounted multi-sensor arrays (LiDAR, camere, millimeter wave radar etc.)[^2] [^4] [^6] [^18] [^19] [^31] [^40], while industrial robotic arms achieve exceptional positioning accuracy via fixed spatial relationships in Eye-in-Hand vision systems[^10] [^21]. These domains share a fundamental design paradigm ensuring sensor field-of-view (FOV) invariance through mechanical engineering. For instance, autonomous vehicles maintain inter sensor geometric consistency through rigid chassis structures, and warehouse robots eliminate dynamic occlusions using fixed height sensor columns. Humanoid robots, widely regarded as the most promising candidate for general purpose robotics and embodying humanity's vision for the pinnacle of robotic embodiment, have recently garnered unprecedented attention $^{[8,11,14,16,34,52]}$. However, their environmental perception capabilities have progressed relatively slowly compared to rapid advancements in mechanical design and locomotion strategies $^{[46]}$. The autonomous operation of humanoid robots in unstructured environments remains fundamentally constrained by an inherent paradox between their biomimetic structure and perceptual requirements. While anthropomorphic limb configurations enhance mobility, they simultaneously create persistent self-occlusion zones elbow joints obstruct chest mounted cameras during manipulation tasks, while thigh movements completely blind waist mounted sensors when navigating obstacles. This structure induced perceptual degradation renders conventional stereo vision solutions, widely adopted in general robotics, significantly less effective on humanoid platforms. As previously analyzed, the development of visual perception systems for humanoid robots faces multifaceted challenges. While multiview sensing configurations have demonstrated efficacy in autonomous driving, their adaptation to humanoid platforms encounters dual constraints: severe data scarcity hinders data driven model adaptation, and morphological diversity prevents the establishment of universal sensor placement standards. Although panoramic vision overcomes field-of-view limitations, it introduces triple feature degradation under spherical projection—structural fragmentation of linear elements; cross-regional scale disparity; and depth-aware feature degradation. Existing decoders relying on 2D offset prediction mechanisms fail to model the coupled effects of distortion and scene depth. In multimodal fusion, automotive LiDAR-camera registration paradigms prove inadequate due to geometric incompatibility between spherical projection and point clouds, while dynamic joint motions induce real-time drift in cross modal geometric alignment matrices. Furthermore, the complex spatial mapping required for unified panoramic-BEV representations suffers from geometric paucity in training data, with conventional augmentation strategies unable to preserve cross view metric consistency—critical limitations that severely constrain 3D environmental comprehension. We present HumanoidPano, a novel perception framework specifically designed for humanoid robots, achieving comprehensive environmental awareness through synergistic fusion of panoramic vision and LiDAR sensing. The framework operates through three stage hierarchical processing: pixel level semantic feature extraction via panoramic imaging, geometry-aware depth profiling using LiDAR point clouds, and cross modal feature fusion through our proposed geometrically grounded alignment mechanism. Our methodology reimagines robotic perception through four pivotal innovations: • Spherical Geometry-aware Constraints. (SGC): Introduces geometric constraints derived from panoramic imaging principles, enabling distortion-aware cross-modal alignment through ray-based feature modulation. --- • Spatial Deformable Attention. (SDA): Achieves efficient $360^{\circ}$-to-BEV fusion via adaptive spatial deformable attention and sampling. • Panoramic Augmentation (AUG): Introduces a novel data augmentation approach that integrates panoramic image and BEV space transformations—including random flipping, mixing, and semantic map alignment—to address alignment challenges in joint view augmentation. This geometrically-coherent framework establishes new standards for humanoid perception, effectively bridging the gap between panoramic distortion resilience and 3D scene understanding while maintaining real time operational efficiency. We establishes new methodological standards for humanoid robotic perception. By transforming structural constraints into design assets, HumanoidPano demonstrates that humanoid robots can transcend biological vision limitations when perception algorithms are deeply adapted to their morphological characteristics. Our contributions bridge critical technical gaps in environmental comprehension for humanoid platforms while providing a scalable architectural blueprint for future embodied intelligence systems. The principal advancements are: - A Universal Multimodal Humanoid Robots Perception Framework: HumanoidPano, a general purpose perception system optimized for humanoid biomechanical constraints, integrating panoramic vision and LiDAR through structural-aware fusion. - State-of-the-art results on the 360BEV Matterport dataset, surpassing previous methods in the precision of understanding the panoramic BEV scene. • Real World Humanoid Robot Validation: Demonstrated practical perception capabilities on physical humanoid platforms, enabling advanced downstream tasks like semantic map based navigation in complex environments. (Figure 1) These achievements collectively redefine the paradigm of environment interaction for humanoid robotics, establishing foundational infrastructure for next generation embodied AI systems. ### 2. Related Work #### 2.1. Humanoid Robot Perception In recent years, humanoid robots have garnered significant attention due to their human-like design and remarkable versatility in both living and working environments. To enhance the adaptability of humanoid locomotion and manipulation, it is essential to equip these robots with advanced perceptive abilities. Structured elevation maps, providing sampled height points as observations for the humanoid. Similarly, [^13] directly utilize egocentric depth images as perceptive input and leverage sampled height points from a simulator to supervise a height map predictor, facilitating improved sim-to-real transfer. However, these perception modalities primarily serve locomotion policies and thus contain only geometric information, lacking semantic understanding. In the field of locomotion, existing methods primarily rely on elevation maps [^35] and depth images. Studies such as [^33] [^37] [^42] [^57] employ LiDAR or depth cameras to con- For manipulation, perception modalities are typically more diverse and offer greater accuracy. Approaches such as [^16] [^22] [^23] [^25] use RGB cameras and teleportation devices to collect human demonstration data, enabling humanoid robots to autonomously complete tasks via imitation learning. [^28] further advances humanoid manipulation by leveraging RGB-D video demonstrations for skill acquisition. Additionally, [^48] integrate a 3D LiDAR sensor with an enhanced diffusion policy imitation learning framework, allowing humanoid robots to perform tasks autonomously in diverse real-world scenarios. [^56] build a navigation simulation environment with egocentric view RGB camera. Despite these advancements, existing methods are constrained by their limited field of view and lack the capability to understand and interact with large-scale surroundings. To address this limitation, we propose HumanoidPano, a novel framework that, for the first time, fuses panoramic cameras and LiDAR to achieve comprehensive environmental awareness. #### 2.2. Panoramic Perception With the rapid advancement of robotics and artificial intelligence, human perception of real-world scenes is no longer restricted to narrow fields of view and low-dimensional sensing devices. Panoramic perception enables more comprehensive environmental understanding and measurement. HoHoNet [^39] introduces a framework for indoor 360-degree panorama understanding using a Latent Horizontal Feature, achieving dense depth estimation, layout reconstruction, and semantic segmentation. [^43] enhances monocular depth estimation for 360-degree images, enabling effective knowledge transfer across different camera projections and data modalities. Several works [^29] [^38] [^47] incorporate global attention mechanisms, significantly improving 360-degree depth estimation. Semantic segmentation is also crucial for navigation and scene understanding. Research efforts such as [^12] [^24] [^32] [^45] [^49] [^54] [^55] provide a more comprehensive interpretation of indoor scenes through panoramic semantic segmentation. Additionally, [^41] extend panoramic perception to generate BEV (Bird's Eye View) segmentation maps. However, despite these advancements, many of these methods are constrained by the lack of real-world validation, as they primarily rely on limited datasets [^1] [^6] [^58]. --- ![Image](images/70266e3b6ddadb2cc09bba693d425a0763521188569f6ae414296216a77ef309.jpg)
Figure 2. HumanoidPano Framework Overview. The system addresses panoramic image distortion using Spherical Geometric Constraints (SGC) to guide 3D adaptive sampling offsets. By encoding camera ray properties into offset prediction, SGC aligns panoramic features with depth measurements, reducing projection artifacts. Spatial Deformable Attention (SDA) adaptively aggregates geometrically consistent object representations from panoramic images.
### 3. Humanoid Pano Motivated by the critical need for embodied scene understanding in humanoid robotics, we formalize the perceptual challenge as Panoramic-LiDAR Cross-modal BEV Semantic Segmentation, a novel task requiring joint interpretation of $360^{\circ}$ visual context and 3D spatial awareness under dynamic morphological constraints. Subsequently our proposed method will be shown in detail. #### 3.1. Problem Formulation In this work, we aim to obtain accurate semantic BEV map from panoramic image for humanoid robots. Formally, it can be defined as: $$ M=F_{seg}(I,D) $$ Where $F_{seg}$ is the segmentation model, $I \in R^{3 \times h \times w}$ is the input panoramic image, D represent the depth image, $(h, w)$ represent the height and width of the image, $M \in R^{L \times X \times Y}$ is the output semantic BEV map, $(X, Y)$ represent the perception range of the BEV space, L is the number of categories predicted. #### 3.2. Overall Architecture The proposed HumanoidPano framework as shown in Figure 2 includes three key components: (1) The transformer-based panoramic image encoder and transformation module. (2) The Panoramic Distortion-aware Attention (PD Attention) module which accomplishes the view transformation from panoramic image to BEV space and adaptive spatial feature extraction with spherical geometry constraints. (3) The BEV decoder parses the projected feature map and predicts the semantic BEV map. ##### 3.2.1. Panoramic Distortion-aware Attention Due to the huge and distorted panoramic image observed by the humanoid robot, it is difficult for the original 360 attention to learn the real 3D spatial distribution and complete representation information of objects with varying scales and distortions in panoramic images. Therefore, PD Attention is proposed, which learns feature aberrations in panoramic images through Spherical Geometry-aware Constraints (SGC) and grasps complete object information in real 3D space through Spatial Deformable Attention (SDA). Spherical Geometry-aware Constraints. Given the BEV queries $Q \in R^{N \times C}$ and the 3D reference points $P \in R^{N \times 3}$ as input, where N is the length of BEV queries and 3D reference points. The 3D reference points P are obtained by projecting the depth map D into the pseudo-point cloud. The mask() operation is applied on Q and P to mask out irrelevant points. The 3D reference points P are transformed onto the sphere to generate spherical reference points $S \in R^{N \times 2}$, as shown in Eq. (2): $$ \Phi=t a n^{-1}\frac{y}{x},\Theta=t a n^{-1}(\frac{x}{z}\cdot\frac{1}{c o s(\Phi)}) $$ Where $(x, y, z)$ represent 3D reference coordinates in cartesian coordinate system, $(\Phi, \Theta)$ represent azimuthal angle and polar angle in spherical coordinate system. The spherical sampling offsets $\Delta S \in R^{N \times 2}$ are obtained by BEV queries with linear layer, which generates spherical locations $S' \in R^{N \times 2}$ with the original spherical reference points $S \in R^{N \times 2}$, as shown in Eq. (3, 4): $$ (\Delta\Phi,\Delta\Theta)=f_{\Delta s}(m a s k(Q)+f_{p o s}(m a s k(S))) $$ $$ \Phi^{\prime}=\Phi+\Delta\Phi,\Theta^{\prime}=\Theta+\Delta\Theta $$ Where $f_{\Delta s}$ represent the linear layer for spherical angular deviations prediction and $f_{pos}$ represent the linear layer for spherical position encoding. Afterwards, we generate the 2D index $I \in R^{N \times 2}$ of the panoramic image from spherical locations $S'$, as shown in Eq. (5): $$ I_{h}=\lceil\frac{H\Theta^{\prime}}{\pi}\rceil,I_{w}=\lceil(\frac{\Phi^{\prime}}{\pi}-\frac{1}{W})\cdot\frac{W}{2}\rceil $$ --- Where $(H, W)$ represent height and width of panoramic image feature map, $(I_{h}, I_{w})$ represent the 2D index values of spherical locations $S'$ on the panoramic image feature map. Spatial Deformable Attention. We regard these 2D indexes I as the reference points of the BEV queries Q and sample the features from the panoramic image feature map $F \in R^{C \times H \times W}$. Finally, we perform a weighted sum of the sampled features as the output of SDA. The process of SDA can be formulated as: $$ SDA(Q,I,F)=\sum_{i}^{N_{ref}}A_{i}\cdot F[I+f_{\Delta ref}(mask(Q))] $$ Where $f_{\Delta ref}$ represent the linear layer for 2D offsets prediction and $A_{i}$ represent the attention weight of each sampled point. ##### 3.2.2. Panoramic-LiDAR Cross-Modal Perception In real-world robotic scenarios, depth information is typically acquired using two primary sensor types: 3D structured light depth cameras and LiDAR. We adopt panoramic LiDAR cross modal perception due to limitations of conventional structured light depth cameras in real-world robotic applications. These sensors often fail to provide the large range detection required for humanoid navigation, whereas LiDAR offers extended sensing capabilities. Although LiDAR generates sparser depth maps compared to structured light, its depth information can be leveraged for sparse semantic queries, which is critical for scene understanding in occluded environments. The primary distinction between LiDAR depth and structured light depth lies in data density. However, this sparsity does not compromise the framework's applicability. Our architecture is designed to inherently accommodate this characteristic through geometric constraint mechanisms, enabling seamless integration of panoramic and LiDAR modalities. This approach ensures robust environmental perception while maintaining computational efficiency for real-time humanoid operations. The paradigm of panoramic LiDAR cross modal perception is validated using real-world physical data. Specifically, the Insta360 X4 panoramic camera and Livox Mid360 LiDAR are utilized for data collection, with precise alignment achieved via timestamp synchronization. The point cloud data is transformed into the panoramic camera's coordinate system using an extrinsic calibration matrix, facilitating the construction of BEV-based map mask and map heights. HumanoidPano employs the PD Attention mechanism to generate 3D spatial points and 2D index values, subsequently performing inference to produce the final BEV semantic segmentation map. This study not only substantiates the effectiveness and robustness of the method in real-world scenarios but also pioneers new directions for panoramic vision and LiDAR fusion perception, expanding its applicability to autonomous navigation and beyond. Building upon our discussion of humanoid robot perception systems above, we present our hardware design in Figure 4, which features a universal multimodal perception module. The projection visualization process of LiDAR point clouds and panoramic images is presented in Figure 3. #### 3.3. Data Augmentation The benefits of data augmentation have formed a consensus in 2D image processing. We designed a data augmentation method for joint panoramic image space and BEV space, which is still not effectively explored. Visualizations and more details of data augmentation are presented in the appendix materials. Panoramic Image Augmentation. For a panoramic image, to enhance its diversity, we randomly flip it and randomly mix it with another panoramic image in the training set. BEV Augmentation. Similar to panoramic image augmentation, BEV augmentation imposes the random flipping and mixing described above. Note that data augmentation in the BEV space is more challenging than the image space because the image space and BEV space need to be aligned for proper view transformation. Therefore, we need to apply corresponding rotation and mixing to the ground truth of the semantic map in order to align the panoramic image augmentation. ### 4. Experiments In this section, we comprehensively present the performance of our proposed approach across multiple datasets. We begin by introducing the prevalent panoramic camera perception datasets in Section 4.1, which serve as the basis for training and evaluating our method. Subsequently, in Section 4.3, we demonstrate the high pixel wise accuracy, recall, and mean Intersection-over-Union (mIoU) achieved by our method on these datasets, highlighting its effectiveness in semantic segmentation tasks. Additionally, we conduct an ablation study in Section 4.3 to analyze the contributions of different components of our method. Regarding the real world deployment on a humanoid robot, considering its unique requirements, we have designed a specialized layout and sensor arrangement, details of which are presented in Section 4.4. Given the limited sensing range and field of view (FOV) of structured-light depth cameras in real world scenarios, we opted to use a LiDAR for depth information acquisition. Finally, validation demos carried out on an actual humanoid robot platform convincingly demonstrate the overall effectiveness of our solution, highlighting its potential for practical applications in humanoid robotics. --- ![Image](images/34f6415caa05a5889932744fdb8bac435a9f652b9e26dd784633ef9df90ecca9.jpg)
Figure 3. Visualization of LiDAR Depth Extraction and Projection Process. First, LiDAR captures raw point clouds, which are then projected onto panoramic images using the camera's intrinsic and extrinsic parameters. This projected depth information is fed into the network, enabling the retrieval of pixel-wise semantics corresponding to the depth values.
![Image](images/782b1bed4f9ac8cd6f028afa7c328726300826a33561ce304e74394a7a79205b.jpg)
Figure 4. Hardware design of our universal sensor module, integrating an Insta360 X4 panoramic camera and Livox Mid-360 LiDAR with precise spatial alignment. The compact, lightweight assembly minimizes self-occlusion while maximizing $360^{\circ}$ coverage. Its modular design supports flexible deployment across humanoid platforms for real-time panoramic-LiDAR fusion in navigation and manipulation tasks.
![Image](images/32d43223bc5ec46fe68f33c6f3ef25b500d5e4cbae539ec8a79704199ec3ca66.jpg)
Figure 5. The data distribution of 360BEV-Matterport visualized by category $^{[41]}$. We directly adopted the well-established indoor classification scheme and visual schematics from 360BEV-Matterport for dataset presentation, these categories enable humanoid robots to effectively perform indoor navigation tasks.
#### 4.1. Dataset The scarcity of panoramic datasets, compounded by the challenges of spherical image annotation, poses significant limitations to model generalization in panoramic perception research. However, we address this gap by leveraging the 360BEV-Matterport dataset $^{[41]}$, a tailored benchmark uniquely aligned with humanoid robotic perception requirements. Derived from Matterport3D, this dataset provides panoramic RGB images ( $512\times1024$ resolution) paired with dense BEV semantic annotations for 20 navigation-critical categories (Figure 5). Synthetic depth maps generated via ray casting ensure geometric consistency with panoramic inputs, enabling rigorous validation of our crossmodal fusion framework. The dataset's multi-scenario coverage of cluttered indoor environments (living spaces, offices) closely mirrors real-world humanoid operation scenarios. While BEV annotations directly align with our goal of generating navigable semantic maps. The spherical projection of panoramic images matches humanoid sensor configurations, and the combination of $360^{\circ}$ visual context, synthetic depth, and navigation-centric labels makes this dataset ideal for evaluating our distortion-aware panoramic-LiDAR fusion approach. #### 4.2. Training Details We adopted the training settings of 360BEV[^41]. Specifically, we trained the HumanoidPano models using 4 A100 GPUs. The initial learning rate was set to $6 \times 10^{-5}$, and the step strategy was employed for scheduling over 50 epochs. We utilized the AdamW optimizer with an epsilon value of $1 \times 10^{-8}$, a weight decay of 0.01, and a batch size of 4 on each GPU. The panoramic images in 360FV-Matterport $^{[41]}$ dataset had a size of $512 \times 1024$. For the HumanoidPano training, the panoramic images from the 360BEV-Stanford and 360BEV-Matterport datasets, which also had a resolution of $512 \times 1024$, were used as inputs. Meanwhile, the output BEV maps were configured to be $500 \times 500$, corresponding to a perception range of $10m \times 10m$. Following the works of [^5] [^7], we employed pixel-wise accuracy (Acc), pixel recall (mRecall), precision (mPrecision), and mean Intersection-over-Union (mIoU) as the evaluation metrics. #### 4.3. Panorama Semantic Mapping We compare our approach with three backbones, e.g., MiT-B0, MiT-B2 from SegFormer [^47] and MSCA-B from SegNeXt [^20] for the panoramic semantic mapping task. Where the E represents the early projection methods, the L represents the late projection methods and the I represents the intermediate projection methods. Intermediate projection can preserve dense visual cues and long-range information from front-view panoramas, and deliver more valuable context for BEV semantic mapping, leading to this superiority. ---
Table 1. Panoramic semantic mapping on the val set of 360 BEV-Matterport dataset. We marked the data of HumanoidPano with light blue. Meanwhile, we marked the state-of-the-art (SOTA) results with dark blue and indicated the sub-optimal results with underlines (on each backbone). † benefits from data augmentation.
MethodProjBackboneAccmRecallmPrecisionmIoU
SegFormer [^47]EMiT-B268.1241.3345.2529.22
SegNeXt [^20]EMSCA-B68.5342.1346.1230.01
HoHoNet [^39]LResNet10162.8438.9944.2226.21
Trans4PASS [^49]LMiT-B255.9929.5940.9120.07
Trans4PASS+ [^50]LMiT-B257.8932.7540.9321.58
SegFormer [^47]LMiT-B262.9841.8445.3027.78
Trans4Map [^7]IMiT-B070.1944.3150.3931.92
360BEV [^41]IMiT-B074.7247.1455.2035.46
HumanoidPanoIMiT-B076.1950.4157.8638.50
HumanoidPano †IMiT-B076.7152.8757.7640.12
BEVFormer [^30]IMiT-B272.9943.6151.7032.51
Trans4Map [^7]IMiT-B273.2851.6053.0236.72
360BEV [^41]IMiT-B276.9854.3059.6341.42
HumanoidPanoIMiT-B277.7257.5460.1843.52
HumanoidPano †IMiT-B278.4858.3261.5244.54
360BEV [^41]IMSCA-B78.3758.0862.6245.04
HumanoidPanoIMSCA-B79.2160.4163.1746.47
HumanoidPano †IMSCA-B79.4760.9262.6646.35
Table 2. Panoramic semantic mapping on the test set of 360 BEV-Matterport dataset. The annotation is consistent with Table 1.
MethodProjBackboneAccmRecallmPrecisionmIoU
SegFormer [^47]EMiT-B269.7235.2840.4124.04
SegNeXt [^20]EMSCA-B69.9936.2541.9625.22
HoHoNet [^39]LResNet10162.8935.1839.5422.01
Trans4PASS [^49]LMiT-B253.5029.3533.5316.53
Trans4PASS+ [^50]LMiT-B257.2430.6434.4917.72
SegFormer [^47]LMiT-B262.9135.3539.6422.02
Trans4Map [^7]IMiT-B071.7838.2743.7726.52
360BEV [^41]IMiT-B075.9844.7548.5931.81
HumanoidPanoIMiT-B076.4847.0951.4233.56
HumanoidPano †IMiT-B077.5049.2652.2335.37
BEVFormer [^30]IMiT-B272.0436.6947.9027.46
Trans4Map [^7]IMiT-B272.9445.4547.0331.08
360BEV [^41]IMiT-B277.9550.5553.0636.00
HumanoidPanoIMiT-B278.0253.2253.9337.95
HumanoidPano †IMiT-B278.1653.6054.8138.49
360BEV [^41]IMSCA-B79.0753.5956.4039.10
HumanoidPanoIMSCA-B78.9655.6456.6140.27
HumanoidPano †IMSCA-B78.9255.6956.8640.41
as compared to the other paradigms. As visualized in Figure 6, our method outperforms baselines in distorted panoramic regions—especially on severely warped objects—a capability vital for panoramic perception in humanoid and general embodied AI systems. Results on 360BEV-Matterport. We conduct extensive experiments on the val and test set of 360BEV-Matterport and compare our HumanoidPano with other state-of-the-
Table 3. Results under noise with backbone MiT-B0 on the val set of 360 BEV-Matterport dataset. The annotation is consistent with Table 1.
MethodPitchRollAccmRecallmPrecisionmIoU
360BEV--74.7247.1455.2035.46
360BEV✓-69.8540.8749.1129.74
360BEV-✓69.7440.7549.1929.64
360BEV✓✓66.6537.8244.9726.67 (-24.78%)
HumanoidPano--76.1950.4157.8638.50
HumanoidPano✓-71.4644.0151.8532.84
HumanoidPano-✓71.5043.8151.9732.79
HumanoidPano✓✓68.4140.3247.8729.58 (-23.16%)
HumanoidPano †--76.7152.8757.7640.12
HumanoidPano †✓-72.1246.6751.5034.28
HumanoidPano †-✓72.3546.6452.2134.55
HumanoidPano †✓✓69.1943.2548.2731.13 (-22.40%)
Table 4. Ablation study on the test set of 360 BEV-Matterport dataset. The annotation is consistent with Table 1.
SDASGCAUGBackboneAccmRecallmPrecisionmIoU
---MiT-B075.9844.7548.5931.81
✓--MiT-B076.3846.7849.7233.05
✓✓-MiT-B076.4847.0951.4233.56
✓✓✓MiT-B077.5049.2652.2335.37
art methods like BEVFormer [^30], Trans4Map [^7] and 360BEV [^41] to validate the effectiveness of the proposed HumanoidPano. In 360BEV-Matterport val set, as can be seen in Table 1, the proposed HumanoidPano with MiT-B0 surpasses 360BEV by a significant margin of 1.99%, 5.73%, 2.56% and 4.66% in Acc, mRecall, mPrecision and mIoU, respectively. The proposed HumanoidPano with MiT-B2 surpasses 360BEV by a significant margin of 1.50%, 4.02%, 1.89% and 3.12% in Acc, mRecall, mPrecision and mIoU, respectively. The proposed HumanoidPano with MSCA-B surpasses 360BEV by 0.84%, 2.33%, 0.55% and 1.43% in Acc, mRecall, mPrecision and mIoU, respectively. In 360BEV-Matterport test set, as can be seen in Table 2, the proposed HumanoidPano with MiT-B0 surpasses 360BEV by a significant margin of 1.52%, 4.51%, 3.64% and 3.56% in Acc, mRecall, mPrecision and mIoU, respectively. The proposed HumanoidPano with MiT-B2 surpasses 360BEV by a significant margin of 0.21%, 3.05%, 1.75% and 2.49% in Acc, mRecall, mPrecision and mIoU, respectively. The proposed HumanoidPano with MSCA-B surpasses 360BEV by 2.10%, 0.46% and 1.31% in mRecall, mPrecision and mIoU, respectively. Results on noisy settings. In the real world, the panoramic image of humanoid robots are susceptible to change by disturbances on camera. Here, we evaluate the performance of the proposed method in such noisy settings. Specifically, we transform the panoramic image to the spherical ---
Table 5. Per-class results with backbone MiT-B2 on the 360BEV-Matterport dataset. The annotation is consistent with Table 1.
MethodDatamIoUvoidwallfloorchairdoortablepicturefurnitureobjectswindowsofabedsinkstairsceilingtoiletmirrorshowerbedtubcountershelfing
Trans4Map [^7]val36.7247.8728.5282.9634.4422.2739.5816.2822.7526.2925.0842.8162.2513.9541.5137.7945.8219.5648.0547.7138.2527.31
360BEV [^41]val41.4278.7332.1084.2340.6023.9244.8918.1722.3022.9825.1748.3064.9416.9544.0149.4854.5119.1250.7159.0143.9325.85
HumanoidPano †val43.5273.3433.7285.9441.7425.0047.0720.9826.2023.5927.6349.3566.5221.5045.2952.8057.1227.5349.8760.9348.9028.93
HumanoidPano †val44.5476.3633.2985.9542.9827.3447.1821.7423.6528.8527.5650.8165.0919.6048.1158.1554.4926.9051.5366.7945.5833.28
Trans4Map [^7]test31.0840.5132.5480.2133.2320.8537.2119.0118.4623.0523.5632.3552.0815.3429.0218.2741.9015.3925.5848.1930.3815.52
360BEV [^41]test36.0063.4735.5084.9539.0217.8542.1219.9622.0723.0926.6036.1858.4822.8037.0847.0938.7617.0321.3852.8335.2614.66
HumanoidPano †test37.9560.4337.0484.7739.8921.6444.2522.1622.1124.4326.8235.7660.4227.7736.2348.2754.6215.7327.2253.7135.4018.27
HumanoidPano †test38.4961.3837.7284.3939.4122.9143.6020.9421.9724.7527.1337.4461.0524.6436.9350.0556.0919.4129.6452.6636.4019.78
coordinate system and then apply a rotation along the pitch and roll directions. The corresponding rotation matrix is $R(\alpha, \beta, 0)$, where the $\alpha$ is pitch angle and the $\beta$ is roll angle. As shown in Table 3, the performance of the existing methods degrades significantly when the panoramic image is disturbed. In such noise settings, the proposed HumanoidPano has less performance degradation and remains ahead compared to the 360BEV [^41] due to its ability to understand the 3D world. And the proposed HumanoidPano shows more robust performance in the data augmentation setting. Visualization results are presented in the appendix materials. Ablation Study. Experiments in Table 4 show the improvement when our Spatial Deformable Attention (SDA), Spherical Geometry-aware Constraints (SGC) and data augmentation (AUG) are applied. The proposed SDA complements the spatial context information and enhances the network's ability to understand the 3D world, obtaining a relatively significant improvement. The proposed SGC builds the panoramic perception capability and learns distortion features, forcing the SDA to better adapt to the structure of distorted objects in panoramic images. The proposed AUG creates several new scenes by random rotation and mixing, which enhances the diversity of the data, improves the robustness of the model, and also obtains a relatively significant improvement. Per-class Results. Per-class IoU scores on 360BEV-Matterport are shown in Table 5. Compared to Trans4Map [^7] and 360BEV [^41], our HumanoidPano with the same MiT-B2 backbone can achieve respective 44.54% and 38.49% in mIoU on the val and test set. The void class is also included on the 360BEV-Matterport dataset. #### 4.4. Implementation on Humanoid Robot We deploy and validate the HumanoidPano system on Tiangong, a full-sized humanoid platform (1.65m height, 23 DoF), through comprehensive indoor environmental testing. Developed by the Beijing Humanoid Robot Innovation Center, Tiangong demonstrates advanced all-terrain mobility. Our decision to test on this highly capable all-terrain humanoid robot allows us to better demonstrate the performance of our perception system. The perception module integrates an Insta360 X4 panoramic camera and Livox Mid-360 LiDAR, as illustrated in Figure 4. This configuration achieves minimal structural interference through kinematic collision analysis, ensuring continuous $360^{\circ}$ sensing coverage during dynamic motions such as crawling and stair navigation. A hardware-accelerated processing pipeline synchronizes spherical visual data with LiDAR point clouds at 10Hz by using a Nvidia Jetson Orin AGX, while an online calibration protocol dynamically adjusts extrinsic parameters to compensate for sensor displacement caused by joint movements. Evaluations in cluttered domestic environments demonstrate the system's operational efficacy. HumanoidPano maintains robust BEV semantic mapping during complex full-body interactions, including doorway traversal and obstacle navigation. The framework successfully resolves system-level challenges in humanoid robots' omnidirectional semantic perception systems, addressing the longstanding issue of simultaneously acquiring depth information and semantic understanding. By leveraging efficient perception neural networks, it provides robust semantic guidance for downstream navigation modules as shown in Figure 1. This implementation marks the first realization of embodied panoramic-BEV perception on a full-scale humanoid platform, proving its capability to support advanced navigation task in human-centric environments without external infrastructure dependencies. Video demonstration results are presented in the appendix materials. ### 5. Conclusion This work pioneers large-scale semantic mapping for humanoid robots through the HumanoidPano framework, establishing the first perception system capable of real-time environmental comprehension across full-body motion scenarios. By addressing the fundamental conflict between biomimetic morphology and sensing requirements, we demonstrate that structural constraints can be transformed into perceptual advantages when algorithms are co-designed with robotic embodiment. The methodology sets a new precedent for morphology-aware perception, providing both theoretical foundations and implementation guidelines for next-generation human-centered robotic systems. In the --- ![Image](images/547c03cbe8ca54cfcf4073b2a368ce8c75affa2976bc6c55058d0e2bfb740853.jpg)
Figure 6. The comparative experiments on the 360BEV-Matterport dataset. We compared the inference results of 360BEV [^41] and our model, where HumanoidPano demonstrated superior performance in areas with distortions.
future, we will further optimize the perception systems and algorithms for multi-view and panoramic cameras, while collecting and annotating scene data relevant to humanoid robots for further research. ### 6. Acknowledgments The authors would like to express sincere gratitude to the engineers at Beijing Innovation Center of Humanoid Robot for their invaluable technical expertise and dedicated collaboration throughout this research. Their profound insights were instrumental in tackling the challenges posed by the complex coupled tasks of the humanoid robot platform. Special thanks are extended to Arashi Vision Inc. (Insta360) for their generous support and cutting-edge technological assistance, which significantly contributed to the experimental implementation and data acquisition processes. This work would not have been possible without their professional commitment and interdisciplinary synergy. ## References [^1]: Georgios Albanis, Nikolaos Zioulis, Petros Drakoulis, Vasileios Gkitsas, Vladimiros Sterzentsenko, Federico Alvarez, Dimitrios Zarpalas, and Petros Daras. Pano3d: A holistic benchmark and a solid baseline for 360deg depth estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3727–3737, 2021. 3 [^2]: Iro Armeni, Sasha Sax, Amir R Zamir, and Silvio Savarese. Joint 2d-3d-semantic data for indoor scene understanding. arXiv preprint arXiv:1702.01105, 2017. 2 [^3]: Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06817, 2022. 2 [^4]: Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. Nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11621–11631, 2020. 2 [^5]: Vincent Cartillier, Zhile Ren, Neha Jain, Stefan Lee, Irfan Essa, and Dhruv Batra. Semantic mapnet: Building allocentric semantic maps and representations from egocentric views. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 964–972, 2021. 6 [^6]: Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3d: Learning from rgb-d data in indoor environments. arXiv preprint arXiv:1709.06158, 2017. 2, 3 [^7]: Chang Chen, Jiaming Zhang, Kailun Yang, Kunyu Peng, and Rainer Stiefelhagen. Trans4map: Revisiting holistic bird's-eye-view mapping from egocentric images to allocentric semantics with vision transformers. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 4013–4022, 2023. 6, 7, 8 [^8]: Penghui Chen, Yushi Wang, Changsheng Luo, Wenhan Cai, and Mingguo Zhao. Hifar: Multi-stage curriculum learning for high-dynamics humanoid fall recovery. arXiv preprint arXiv:2502.20061, 2025. 2 [^9]: Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research, page 02783649241273668, 2023. 2 [^10]: Cheng Chi, Zhenjia Xu, Chuer Pan, Eric Cousineau, Benjamin Burchfiel, Siyuan Feng, Russ Tedrake, and Shuran Song. Universal manipulation interface: In-the-wild robot teaching without in-the-wild robots. arXiv preprint arXiv:2402.10329, 2024. 2 [^11]: Wenhao Cui, Shengtao Li, Huaxing Huang, Bangyu Qin, Tianchu Zhang, Liang Zheng, Ziyang Tang, Chenxu Hu, NING Yan, Jiahao Chen, et al. Adapting humanoid locomotion over challenging terrain via two-phase training. In 8th Annual Conference on Robot Learning. 2 [^12]: Yuan Dong, Chuan Fang, Liefeng Bo, Zilong Dong, and Ping Tan. Panocontext-former: Panoramic total scene understanding with a transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 28087–28097, 2024. 3 [^13]: Helei Duan, Bikram Pandit, Mohitvishnu S Gadde, Bart Van Marum, Jeremy Dao, Chanho Kim, and Alan Fern. Learning vision-based bipedal locomotion for challenging terrain. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 56–62. IEEE, 2024. 3 [^14]: Yiqun Duan, Qiang Zhang, Jinzhao Zhou, Jingkai Sun, Xiaowei Jiang, Jiahang Cao, Jiaxhu Wang, Yiqian Yang, Wen --- Zhao, Gang Han, et al. E2h: A two-stage non-invasive neural signal driven humanoid robotic whole-body control framework. arXiv preprint arXiv:2410.02141, 2024. 2 [^15]: Yiqun Duan, Zhuoli Zhuang, Jinzhao Zhou, Yu-Cheng Chang, Yu-Kai Wang, and Chin-Teng Lin. Enhancing end-to-end autonomous driving systems through synchronized human behavior data. In Proceedings of the 1st International Workshop on Brain-Computer Interfaces (BCI) for Multimedia Understanding, pages 1–8, 2024. 2 [^16]: Zipeng Fu, Qingqing Zhao, Qi Wu, Gordon Wetzstein, and Chelsea Finn. Humanplus: Humanoid shadowing and imitation from humans. arXiv preprint arXiv:2406.10454, 2024. 2, 3 [^17]: Zipeng Fu, Tony Z Zhao, and Chelsea Finn. Mobile aloha: Learning bimanual mobile manipulation with low-cost whole-body teleoperation. arXiv preprint arXiv:2401.02117, 2024. 2 [^18]: Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In 2012 IEEE conference on computer vision and pattern recognition, pages 3354–3361. IEEE, 2012. 2 [^19]: Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The kitti dataset. The international journal of robotics research, 32(11):1231–1237, 2013. 2 [^20]: Meng-Hao Guo, Cheng-Ze Lu, Qibin Hou, Zhengning Liu, Ming-Ming Cheng, and Shi-Min Hu. Segnext: Rethinking convolutional attention design for semantic segmentation. Advances in neural information processing systems, 35:1140–1156, 2022. 6, 7 [^21]: Huy Ha, Yihuai Gao, Zipeng Fu, Jie Tan, and Shuran Song. Umi on legs: Making manipulation policies mobile with manipulation-centric whole-body controllers. arXiv preprint arXiv:2407.10353, 2024. 2 [^22]: Tairan He, Zhengyi Luo, Xialin He, Wenli Xiao, Chong Zhang, Weinan Zhang, Kris Kitani, Changliu Liu, and Guanya Shi. Omnih2o: Universal and dexterous human-to-humanoid whole-body teleoperation and learning. arXiv preprint arXiv:2406.08858, 2024. 3 [^23]: Tairan He, Zhengyi Luo, Wenli Xiao, Chong Zhang, Kris Kitani, Changliu Liu, and Guanya Shi. Learning human-to-humanoid real-time whole-body teleoperation. In 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 8944–8951. IEEE, 2024. 3 [^24]: Zhigang Jiang, Zhongzheng Xiang, Jinhua Xu, and Ming Zhao. Lgt-net: Indoor panoramic room layout estimation with geometry-aware transformer network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1654–1663, 2022. 3 [^25]: Daehwa Kim, Mario Srouji, Chen Chen, and Jian Zhang. Armor: Egocentric perception for humanoid robot collision avoidance and motion planning. arXiv preprint arXiv:2412.00396, 2024. 3 [^26]: Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source vision-language-action model. arXiv preprint arXiv:2406.09246, 2024. 2 [^27]: Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. Journal of Machine Learning Research, 17(39):1–40, 2016. [^28]: Jinhan Li, Yifeng Zhu, Yuqi Xie, Zhenyu Jiang, Mingyo Seo, Georgios Pavlakos, and Yuke Zhu. Okami: Teaching humanoid robots manipulation skills through single video imitation. In 8th Annual Conference on Robot Learning, 2024. 3 [^29]: Yuyan Li, Yuliang Guo, Zhixin Yan, Xinyu Huang, Ye Duan, and Liu Ren. Omnifusion: 360 monocular depth estimation via geometry-aware fusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2801–2810, 2022. 3 [^50]: Ziqi Li, Wenna Wang, Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Qiao Yu, and Jifeng Dai. Bevformer: learning bird's-eye-view representation from lidar-camera via spatiotemporal transformers. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024. 7 [^31]: Yiyi Liao, Jun Xie, and Andreas Geiger. Kitti-360: A novel dataset and benchmarks for urban scene understanding in 2d and 3d. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(3):3292–3310, 2022. 2 [^32]: Zhixin Ling, Zhen Xing, Xiangdong Zhou, Manliang Cao, and Guichun Zhou. Panoswin: a pano-style swin transformer for panorama understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17755–17764, 2023. 3 [^33]: Junfeng Long, Junli Ren, Moji Shi, Zirui Wang, Tao Huang, Ping Luo, and Jiangmiao Pang. Learning humanoid locomotion with perceptive internal model. arXiv preprint arXiv:2411.14386, 2024. 3 [^34]: Chenhao Lu, Xuxin Cheng, Jialong Li, Shiqi Yang, Mazeyu Ji, Chengjing Yuan, Ge Yang, Sha Yi, and Xiaolong Wang. Mobile-television: Predictive motion priors for humanoid whole-body control. arXiv preprint arXiv:2412.07773, 2024. 2 [^35]: Takahiro Miki, Lorenz Wellhausen, Ruben Grandia, Fabian Jenelten, Timon Homberger, and Marco Hutter. Elevation mapping for locomotion and navigation using gpu. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 2273–2280. IEEE, 2022. 3 [^36]: Zekun Qi, Wenyao Zhang, Yufei Ding, Runpei Dong, Xin-qiang Yu, Jingwen Li, Lingyun Xu, Baoyu Li, Xialin He, Guofan Fan, Jiazhao Zhang, Jiawei He, Jiayuan Gu, Xin Jin, Kaisheng Ma, Zhizheng Zhang, He Wang, and Li Yi. Sofar: Language-grounded orientation bridges spatial reasoning and object manipulation. arXiv preprint arXiv:2502.13143, 2025. 2 [^37]: Junli Ren, Tao Huang, Huayi Wang, Zirui Wang, Qingwei Ben, Jiangmiao Pang, and Ping Luo. Vb-com: Learning vision-blind composite humanoid locomotion against deficient perception. arXiv preprint arXiv:2502.14814, 2025. 3 [^38]: Zhijie Shen, Chunyu Lin, Kang Liao, Lang Nie, Zishuo Zheng, and Yao Zhao. Panformer: panorama transformer for indoor 360 depth estimation. In European Conference on Computer Vision, pages 195–211. Springer, 2022. 3 --- [^39]: Cheng Sun, Min Sun, and Hwann-Tzong Chen. Hohonet: 360 indoor holistic understanding with latent horizontal features. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2573–2582, 2021. 3, 7 [^40]: Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalability in perception for autonomous driving: Waymo open dataset. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2446–2454, 2020. 2 [^41]: Zhifeng Teng, Jiaming Zhang, Kailun Yang, Kunyu Peng, Hao Shi, Simon Reib, Ke Cao, and Rainer Stiefelhagen. 360bev: Panoramic semantic mapping for indoor bird's-eye view. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 373–382, 2024. 3, 6, 7, 8, 9 [^42]: Huayi Wang, Zirui Wang, Junli Ren, Qingwei Ben, Tao Huang, Weinan Zhang, and Jiangmiao Pang. Beamdojo: Learning agile humanoid locomotion on sparse footholds. arXiv preprint arXiv:2502.10363, 2025. 3 [^43]: Ning-Hsu Albert Wang and Yu-Lun Liu. Depth anywhere: Enhancing 360 monocular depth estimation via perspective distillation and unlabeled data augmentation. Advances in Neural Information Processing Systems, 37: 127739–127764, 2025. 3 [^44]: Kun Wu, Chengkai Hou, Jiaming Liu, Zhengping Che, Xiaozhu Ju, Zhuqin Yang, Meng Li, Yinuo Zhao, Zhiyuan Xu, Guang Yang, et al. Robomind: Benchmark on multi-embodiment intelligence normative data for robot manipulation. arXiv preprint arXiv:2412.13877, 2024. 2 [^45]: Haozheng Yu, Lu He, Bing Jian, Weiwei Feng, and Shan Liu. Panelnet: Understanding 360 indoor environment via panel representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 878–887, 2023. 3 [^46]: Kai Yuan, Noor Sajid, Karl Friston, and Zhibin Li. Hierarchical generative modelling for autonomous robots. Nature Machine Intelligence, 5(12):1402–1414, 2023. 2 [^47]: Ilwi Yun, Chanyang Shin, Hyunku Lee, Hyuk-Jae Lee, and Chae Eun Rhee. Eformer: Equirectangular geometry-biased transformer for 360 depth estimation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6101–6112, 2023. 3, 6, 7 [^48]: Yanjie Ze, Zixuan Chen, Wenhao Wang, Tianyi Chen, Xialin He, Ying Yuan, Xue Bin Peng, and Jiajun Wu. Generalizable humanoid manipulation with improved 3d diffusion policies. arXiv preprint arXiv:2410.10803, 2024. 3 [^49]: Jiaming Zhang, Kailun Yang, Chaoxiang Ma, Simon Reib, Kunyu Peng, and Rainer Stiefelhagen. Bending reality: Distortion-aware transformers for adapting to panoramic semantic segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16917–16927, 2022. 3, 7 [^50]: Jiaming Zhang, Kailun Yang, Hao Shi, Simon Reiß, Kunyu Peng, Chaoxiang Ma, Haodong Fu, Philip HS Torr, Kaiwei Wang, and Rainer Stiefelhagen. Behind every domain there is a shift: Adapting distortion-aware vision transformers for panoramic semantic segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024. 7 [^51]: Lingfeng Zhang, Xiaoshuai Hao, Qinwen Xu, Qiang Zhang, Xinyao Zhang, Pengwei Wang, Jing Zhang, Zhongyuan Wang, Shanghang Zhang, and Renjing Xu. Mapnav: A novel memory representation via annotated semantic maps for vlm-based vision-and-language navigation. arXiv preprint arXiv:2502.13451, 2025. 2 [^52]: Qiang Zhang, Peter Cui, David Yan, Jingkai Sun, Yiqun Duan, Gang Han, Wen Zhao, Weining Zhang, Yijie Guo, Arthur Zhang, et al. Whole-body humanoid robot locomotion with human reference. In 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 11225–11231. IEEE, 2024. 2 [^53]: Tony Z Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-grained bimanual manipulation with low-cost hardware. arXiv preprint arXiv:2304.13705, 2023. [^54]: Xu Zheng, Pengyuan Zhou, Athanasios V Vasilakos, and Lin Wang. Semantics distortion and style matter: Towards source-free uda for panoramic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 27885–27895, 2024. 3 [^55]: Xu Zheng, Peng Yuan Zhou, Athanasios V Vasilakos, and Lin Wang. 360sfuda++: Towards source-free uda for panoramic segmentation by learning reliable category prototypes. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024. 3 [^56]: Shaoting Zhu, Linzhan Mou, Derun Li, Baijun Ye, Runhan Huang, and Hang Zhao. Vr-robo: A real-to-sim-to-real framework for visual robot navigation and locomotion. arXiv preprint arXiv:2502.01536, 2025. 3 [^57]: Ziwen Zhuang, Shenzhen Yao, and Hang Zhao. Humanoid parkour learning. arXiv preprint arXiv:2406.10759, 2024. 3 [^58]: Nikolaos Zioulis, Antonis Karakottas, Dimitrios Zarpalas, Federico Alvarez, and Petros Daras. Spherical view synthesis for self-supervised 360 depth estimation. In 2019 International Conference on 3D Vision (3DV), pages 690–699. IEEE, 2019. 3 --- ### A. Appendix For evaluating the robustness of our methods, we rotate the panoramic image as showing in Figure 7. The benefits of data augmentation have formed a consensus in 2D image processing. Augmenting data helps models generalize better by introducing variations in the training set, reducing overfitting, and improving robustness to unseen scenarios. The visualization of data augmentation for joint panoramic image space, as Figure 8, Bird's Eye View (BEV) space like Figure 9. The augmentation techniques incorporate spatial transformations and feature-mixing strategies to enhance generalization while maintaining geometric alignment. Visualizations of data augmentation are provided in the appendix materials. For a panoramic image, data augmentation techniques are applied to increase diversity in training samples. Given a panoramic image I, a horizontal flip operation is applied with probability p. Let W be the width of the image, and let $I(x, y)$ represent the pixel value at coordinate $(x, y)$. The flipped image $I'$ is defined as: $$ I^{\prime}(x,y)=I(W-x,y),\quad\forall x\in[0,W],\quad\forall y\in[0,H], $$ where H is the height of the image. The horizontal flip is applied as follows: $$ I^{\prime}=\begin{cases}I(W-x,y),&with probability p,\\I(x,y),,&otherwise.\end{cases} $$ Since panoramic images wrap around at the boundaries, care must be taken to ensure continuity when flipping. To introduce additional variations, a random mixing strategy is applied. Given two panoramic images $I_{1}$ and $I_{2}$ sampled from the training set, a mixing factor $\lambda$ is used to blend them as follows: $$ I^{\prime}=\lambda I_{1}+(1-\lambda)I_{2}. $$ This operation encourages the model to learn smooth transitions between different scenes and reduces sensitivity to specific environmental conditions. BEV augmentation follows a similar approach to panoramic image augmentation, including random flipping and image mixing. However, data augmentation in the BEV space is more complex due to the need for geometric consistency between the panoramic and BEV representations. To ensure proper alignment, additional constraints are introduced. The flipping operation in the BEV space must correspond to the transformation applied in the panoramic space. Given a BEV semantic map S, the flipping function is defined as: $$ S^{\prime}(x,y)=S(W-x,y). $$ This ensures that any augmentation in the image space is accurately reflected in the BEV space. A random rotation is applied to maintain spatial coherence in BEV. Given a rotation angle, the transformed semantic map is computed using a 2D rotation matrix: $$ \begin{bmatrix}x^{\prime}\\ y^{\prime}\end{bmatrix}\begin{bmatrix}\cos\theta&-\sin\theta\\ \sin\theta&\cos\theta\end{bmatrix}\begin{bmatrix}x-x_{c}\\ y-y_{c}\end{bmatrix}+\begin{bmatrix}x_{c}\\ y_{c}\end{bmatrix}, $$ where is the center of rotation. In addition to flipping and rotation, BEV semantic map mixing is performed similarly to panoramic images: $$ S^{\prime}=\lambda S_{1}+(1-\lambda)S_{2}. $$ This ensures the semantic diversity of BEV annotations while preserving spatial alignment. The combination of these augmentation strategies enhances the robustness and generalization of the model in downstream tasks. --- ![Image](images/f1638a234253aeeba80e8fbe97c02e3d168d057ebbd5389191f04e9629911c67.jpg)
Figure 7. Visualizations Panoramic Robustness Evaluation. Roll and Pitch mean the rotation of images in the following direction.
![Image](images/3e494601e0e8adccc8e923bc155905ffda344f73284025549ecac291c4bb7569.jpg)
Figure 8. Visualizations Panoramic Images Augmentation. The horizontal flip and mix augmentation.
![Image](images/683d6e2cbecd484a0a590578a435716629675b2e9ce3e551f8fccbdbc6c8c090.jpg)
Figure 9. Visualizations BEV Images Augmentation. The horizontal flip and mix augmentation.