Domain Generalization for Multimodal Disaster Tweet Classification

Abstract

Disaster-related information on social media platforms facilitates situational awareness of disaster events. Benefiting from the development of multimodal deep learning, it has become easy to extract and fuse information from different modalities of social media data. However, existing multimodal models are developed on limited disaster datasets and have problems in generalization. The lack of generalization is mainly manifested by significant performance degradation on disaster events outside the dataset. One key reason for overfitting might be that the models combine class information too closely with specific disaster events during training. In multimodal social media applications, existing models are rarely optimized for the representation of event domains. In this paper, we treat different disaster events as separate domains and propose a new domain generalization method for multimodal representation learning. Our method takes multimodal features as the sum of domain-specific features and class-specific features. The proposed model obtains domain-invariant representations through feature separation and reconstruction. In addition, the multimodal information exchange is realized by the cross-modal interaction module, and highly generalized class representations are finally obtained. Experimental results show that the proposed method improves generalization performance on the CrisisMMD benchmark.