Abstract
Multimodal reasoning is a challenging task that requires understanding and integrating information from different modalities, such as text and image. Existing methods for multimodal reasoning often fail to capture the rich structural information among visual and textual semantics in different modalities, which is crucial for generating accurate answers. In this paper, we propose a novel method that leverages graph neural networks to model the structural information to enhance multimodal reasoning. Specifically, we first use a Multimodal and Multi-hop reader to attend to different chunks in the context based on the question, and then search for multi-hop candidate tokens within these chunks. Next, we construct a graph to represent the relations among the chunks. Then we apply a Sparse Matrix-Tree algorithm to learn a hierarchical informative structure. Then, we use a Hierarchy-aware Message Passing mechanism to perform multi-hop reasoning on the selected edges and update the node representations. Finally, we use a graph-selection decoder to generate the answer based on the structure-enriched chunk representation. We conduct experiments on the WebQA dataset, which is a large-scale multimodal question answering dataset [1]. The results show that our method outperforms the baseline methods in terms of reasoning and the overall answer accuracy. We also provide some qualitative analysis to illustrate how our method benefits from the structural information among different modalities.
This work is supported by National Key Research and Development Program of China (2020AAA0109700), National Natural Science Foundation of China (62076167).
Q. Zhang and H. Lv—These authors contributed equally to this work.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Chang, Y., Narang, M., Suzuki, H., Cao, G., Gao, J., Bisk, Y.: WebQA: multihop and multimodal QA. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16495–16504 (2022)
Oguz, B., et al.: UniK-QA: unified representations of structured and unstructured knowledge for open-domain question answering. In: Findings of the Association for Computational Linguistics: NAACL 2022, pp. 1535–1546, July 2022
Qi, P., Lee, H., Sido, T., Manning, C.D.: Answering open-domain questions of varying reasoning steps from text. In: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 3599–3614, November 2021
Smith, D.A., Smith, N.A.: Probabilistic models of nonprojective dependency trees. In: Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), pp. 132–140, June 2007
Koo, T., Globerson, A., Carreras Pérez, X., Collins, M.: Structured prediction models via the matrix-tree theorem. In: Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), pp. 141–150 (2007)
McDonald, R., Satta, G.: On the complexity of non-projective data-driven dependency parsing. In: Proceedings of the Tenth International Conference on Parsing Technologies, pp. 121–132, June 2007
Talmor, A., et al.: MultiModalQA: complex question answering over text, tables and images (2021). ar**v preprint ar**v:2104.06039
Dhingra, B., **, Q., Yang, Z., Cohen, W., Salakhutdinov, R.: Neural models for reasoning over multiple mentions using coreference. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pp. 42–48, June 2018
Clark, C., Gardner, M.: Simple and effective multi-paragraph reading comprehension. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 845–855, July 2018
Tu, M., Wang, G., Huang, J., Tang, Y., He, X., Zhou, B.: Multi-hop reading comprehension across multiple documents by reasoning over heterogeneous graphs. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 2704–2713, July 2019
Tang, Z., Shen, Y., Ma, X., Xu, W., Yu, J., Lu, W.: Multi-hop reading comprehension across documents with path-based graph convolutional network. In: Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence, pp. 3905–3911, January 2021
Kundu, S., Khot, T., Sabharwal, A., Clark, P.: Exploiting explicit paths for multi-hop reading comprehension. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 2737–2747, July 2019
Chen, J., Lin, S., Durrett, G.: Multi-hop question answering via reasoning chains. ar**v preprint ar**v:1910.02610 (2019)
Min, S., Zhong, V., Zettlemoyer, L., Hajishirzi, H.: Multi-hop reading comprehension through question decomposition and rescoring. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 6097–6109, July 2019
Wang, D., Liu, P., Zheng, Y., Qiu, X., Huang, X.J.: Heterogeneous graph neural networks for extractive document summarization. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 6209–6219, July 2020
**, H., Wang, T., Wan, X.: SemSUM: semantic dependency guided neural abstractive summarization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, no. 05, pp. 8026–8033, April 2020
Qiu, Y., Cohen, S.B.: Abstractive summarization guided by latent hierarchical document structure. In: Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 5303–5317, December 2022
Wang, W., Pan, S.: Deep inductive logic reasoning for multi-hop reading comprehension. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 4999–5009, May 2022
Zhou, L., Palangi, H., Zhang, L., Hu, H., Corso, J., Gao, J.: Unified vision-language pre-training for image captioning and VQA. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, no. 07, pp. 13041–13049, April 2020
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Zhang, Q. et al. (2024). Discovering Multimodal Hierarchical Structures with Graph Neural Networks for Multi-modal and Multi-hop Question Answering. In: Liu, Q., et al. Pattern Recognition and Computer Vision. PRCV 2023. Lecture Notes in Computer Science, vol 14425. Springer, Singapore. https://doi.org/10.1007/978-981-99-8429-9_31
Download citation
DOI: https://doi.org/10.1007/978-981-99-8429-9_31
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-99-8428-2
Online ISBN: 978-981-99-8429-9
eBook Packages: Computer ScienceComputer Science (R0)