RepF-Net: Distortion-Aware Re-projection Fusion Network for Object Detection in Panorama Image

Abstract

Panorama image has a large 360$^{\circ }$ field of view, providing rich contextual information for object detection, widely used in virtual reality, augmented reality, scene understanding, etc. However, existing methods for object detection on panorama image still have some problems. When 360$^{\circ }$ content is converted to the projection plane, the geometric distortion brought by the projection model makes the neural network can not extract features efficiently, the objects at the boundary of the projection image are also incomplete. To solve these problems, in this paper, we propose a novel two-stage detection network, RepF-Net, comprehensively utilizing multiple distortion-aware convolution modules to deal with geometric distortion while performing effective features extraction, and using the non-maximum fusion algorithm to fuse the content of the detected object in the post-processing stage. Our proposed unified distortion-aware convolution modules can be used to deal with distortions from geometric transforms and projection models, and be used to solve the geometric distortion caused by equirectangular projection and stereographic projection in our network. Our proposed non-maximum fusion algorithm fuses the content of detected objects to deal with incomplete object content separated by the projection boundary. Experimental results show that our RepF-Net outperforms previous state-of-the-art methods by 6$\%$ on mAP. Based on RepF-Net, we present an implementation of 3D object detection and scene layout reconstruction application.