Abstract
With the rapid development of artificial intelligence, knowledge graph, image processing, etc. have been widely used, and smart medical care, as a major application scenario of artificial intelligence, has received a lot of attention. Traditional diagnostic methods have problems such as low accuracy and low efficiency, and the research and application of knowledge graph and image classification in the field of dermatology are also in the initial stage, but text-based knowledge graph technology and image-based image classification technology have developed very maturely. Considering that various current image classification algorithms extract features, feature calculation, and model matching from images, they do not consider obtaining information such as features or relationships that are not in images from text data to participate in image classification tasks. In this paper, the optimized hierarchical perception model H-HAKE based on hierarchical perception model KGE-HAKE calculates selector parameters by improving the hierarchical perception model to add category dimension to the TransE coordinate system, divide more image features and entities with the same attribute into the same level, increase the number of links between image and map entities, and produce better data coverage effect. Aiming at the image classification task, this paper proposes a game tree model to optimize the classification results, including calculating the confidence degree based on the map, the aggregation degree of the classification results, the inference value of the entities in the domain, etc., and comprehensively designing the fusion mode of knowledge graph and image classification algorithm KG-based CNN in scenarios such as multi-map input and feature pre-extraction. The mode is effective enough to enable the image classification task to utilize multimodal data, and the effectiveness is verified by multi-scenario and data ablation experiments on the public data collection.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Ridell, P., Spett, H.: Training Set Size for Skin Cancer Classification Using Google’s Inception v3 (2017)
Alabduljabbar, R., Alshamlan, H.: Intelligent multiclass skin cancer detection using convolution neural networks. (010):000 (2021)
Meng, T., Lin, L., Shyu, M.L., Chen, S.C.: Histology image classification using supervised classification and multimodal fusion. In: 2010 IEEE International Symposium on Multimedia, Taichung, Taiwan, pp. 145–152 (2010).https://doi.org/10.1109/ISM.2010.29
Znaidia, A., Shabou, A., Popescu, A., et al.: Multimodal feature generation framework for semantic image classification. In: ACM International Conference on Multimedia Retrieval, pp. 1–8. ACM (2012)
Ji, S., Pan, S., Cambria, E., et al.: A survey on knowledge graphs: representation, acquisition, and applications. IEEE Trans. Neural Netw. Learn. Syst. (99) (2021)
Zhen, W., Zhang, J., Feng, J., et al.: Knowledge graph embedding by translating on hyperplanes. In: National Conference on Artificial Intelligence. AAAI Press (2014)
Feng, J.: Knowledge graph embedding by translating on hyperplanes. In: AAAI, vol. 28, no. 1 (2014)
Moon, C., Harenberg, S., Slankas, J., et al.: Learning contextual embeddings for knowledge graph completion. In: The 21st Pacific Asia Conference on Information Systems, vol. 10 (2017)
Ji, G., He, S., Xu, L., et al.: Knowledge graph embedding via dynamic map** matrix. In: Meeting of the Association for Computational Linguistics & the International Joint Conference on Natural Language Processing, pp. 687–696 (2015)
Dettmers, T., Minervini, P., Stenetorp, P., et al.: Convolutional 2D Knowledge Graph Embeddings (2017)
Nguyen, D.Q., Vu, T., Nguyen, T.D., et al.: A capsule network-based embedding model for knowledge graph completion and search personalization (2018)
Yao, L., Mao, C., Luo, Y.: KG-BERT: BERT for knowledge graph completion (2019)
Nordhausen, K.: An introduction to statistical learning—with applications in R by Gareth James, Daniela Witten, Trevor Hastie & Robert Tibshirani. Int. Stat. Rev. 82(1), 156–157 (2014)
Wang, H., Zhang, F., **e, X., et al.: DKN: deep knowledgeÂaware network for news recommendation. In: Proceedings of the 2018 World Wide Web Conference on World Wide Web, pp. 1835–1844 (2018)
Wang, Q., Mao, Z., Wang, B., et al.: Knowledge graph embedding: a survey of approaches and applications. IEEE Trans. Knowl. Data Eng. 29(12), 2724–2743 (2017)
Liu, Y., Li, H., GarciaÂDuran, A., et al.: MMKG: multiÂmodal knowledge graphs. In: European Se mantic Web Conference, pp. 459–474 (2019)
MoussellyÂSergieh, H., Botschen, T., Gurevych, I., et al.: A multimodal translationÂbased approach for knowledge graph representation learning. In: Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, New Orleans, Louisiana, pp. 225–234 (2018)
Cun, Y.L., Boser, B., Denker, J.S., et al.: Handwritten digit recognition with a back-propagation network. Adv. Neural. Inf. Process. Syst. 2(2), 396–404 (1990)
Krizhevsky, A., Sutskever, I., Hinton, G.: ImageNet classification with deep convolutional neural networks. Adv. Neural Inf. Process. Syst. 25(2) (2012)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Xu, Z., Yang, X., **, Y., Chen, S. (2024). Research on Feature Fusion Methods for Multimodal Medical Data. In: Zhang, M., Xu, B., Hu, F., Lin, J., Song, X., Lu, Z. (eds) Computer Applications. CCF NCCA 2023. Communications in Computer and Information Science, vol 1959. Springer, Singapore. https://doi.org/10.1007/978-981-99-8764-1_8
Download citation
DOI: https://doi.org/10.1007/978-981-99-8764-1_8
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-99-8763-4
Online ISBN: 978-981-99-8764-1
eBook Packages: Computer ScienceComputer Science (R0)