Pairwise Negative Sample Mining for Human-Object Interaction Detection

Abstract

In recent years, Human-Object Interaction (HOI) detection has been riding the wave of the development of object detectors. Typically, one-stage methods exploit them by instantiating HOI and detecting them in a coarse end-to-end learning scheme. With the invention detection transformer (DETR), more studies followed and addressed HOI detection in this novel set-prediction manner and achieved decent performance. However, the scarcity of positive samples in the dataset, especially among object-dense images, hinders learning and leads to lower detection quality. To alleviate this issue, we propose a sample mining technique that utilizes non-interactive human-object pairs to generate negative samples containing HOI features, enriching the sample sets to help the learning of queries. We also introduce interactivity priors, namely filtering out insignificant background objects to inhibit their disturbance. Our technique can be seamlessly integrated into an end-to-end training scheme. Additionally, we propose an unorthodox two-stage transformer-based method that separates pairwise detection and interaction inference to be handled by two cascade decoders, further exploiting this technique. Experimental results on mainstream datasets demonstrate that our approach achieves new state-of-the-art performance, surpassing both one-stage and traditional two-stage methods. Our study reveals the potential to convert between the two method types by adjusting data utilization.