Abstract
Image captioning is a complex artificial intelligence task that involves many fundamental questions of data representation, learning, and natural language processing. In addition, most of the work in this domain addresses the English language because of the high availability of annotated training data compared to other languages. Therefore, we investigate methods for image captioning in German that transfer knowledge from English training data. We explore four different methods for generating image captions in German, two baseline methods and two more advanced ones based on transfer learning. The baseline methods are based on a state-of-the-art model which we train using a translated version of the English MS COCO dataset and the smaller German Multi30K dataset, respectively. Both advanced methods are pre-trained using the translated MS COCO dataset and fine-tuned for German on the Multi30K dataset. One of these methods uses an alternative attention mechanism from the literature that showed a good performance in English image captioning. We compare the performance of all methods for the Multi30K test set in German using common automatic evaluation metrics. We show that our advanced method with the alternative attention mechanism presents a new baseline for German BLEU, ROUGE, CIDEr, and SPICE scores, and achieves a relative improvement of 21.2% in BLEU-4 score compared to the current state-of-the-art in German image captioning.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Anderson, P., Fernando, B., Johnson, M., Gould, S.: SPICE: semantic propositional image caption evaluation. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9909, pp. 382–398. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46454-1_24
Banerjee, S., Lavie, A.: METEOR: an automatic metric for MT evaluation with improved correlation with human judgments. In: Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pp. 65–72 (2005)
Biswas, R.: Diverse image caption generation and automated human judgement through active learning. Master’s thesis, Saarland University (2019)
Biswas, R., Barz, M., Sonntag, D.: Towards explanatory interactive image captioning using top-down and bottom-up features, beam search and re-ranking. KI - Künstliche Intell. German J. Artif. Intell. - Organ Fachbereiches “Künstliche Intell.” Gesellschaft für Inf. e.V. (KI) 34(4), 571–584 (2020). https://doi.org/10.1007/s13218-020-00679-2
Biswas, R., Mogadala, A., Barz, M., Sonntag, D., Klakow, D.: Automatic judgement of neural network-generated image captions. In: Martín-Vide, C., Purver, M., Pollak, S. (eds.) SLSP 2019. LNCS (LNAI), vol. 11816, pp. 261–272. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-31372-2_22
Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: ImageNet: a large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255. IEEE (2009)
Elliott, D., Frank, S., Hasler, E.: Multilingual image description with neural sequence models. ar**v Computation and Language (2015)
Elliott, D., Frank, S., Barrault, L., Bougares, F., Specia, L.: Findings of the second shared task on multimodal machine translation and multilingual image description. In: Proceedings of the Second Conference on Machine Translation, pp. 215–233. Association for Computational Linguistics, Copenhagen, Denmark, September 2017. https://doi.org/10.18653/v1/W17-4718. https://www.aclweb.org/anthology/W17-4718
Elliott, D., Frank, S., Sima’an, K., Specia, L.: Multi30K: multilingual English-German image descriptions. In: Proceedings of the 5th Workshop on Vision and Language, pp. 70–74. Association for Computational Linguistics, Berlin, August 2016. https://doi.org/10.18653/v1/W16-3210. https://www.aclweb.org/anthology/W16-3210
Gu, J., Joty, S., Cai, J., Wang, G.: Unpaired image captioning by language pivoting. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018, Part I. LNCS, vol. 11205, pp. 519–535. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01246-5_31
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778 (2016). https://doi.org/10.1109/CVPR.2016.90
Hitschler, J., Schamoni, S., Riezler, S.: Multimodal pivots for image caption translation. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2399–2409. Association for Computational Linguistics, Berlin, August 2016. https://doi.org/10.18653/v1/P16-1227. https://www.aclweb.org/anthology/P16-1227
Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997). https://doi.org/10.1162/neco.1997.9.8.1735
Huang, P.Y., Liu, F., Shiang, S.R., Oh, J., Dyer, C.: Attention-based multimodal neural machine translation. In: Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers, pp. 639–645. Association for Computational Linguistics, Berlin, August 2016. https://doi.org/10.18653/v1/W16-2360. https://www.aclweb.org/anthology/W16-2360
Jaffe, A.: Generating image descriptions using multilingual data. In: Proceedings of the Second Conference on Machine Translation, pp. 458–464. Association for Computational Linguistics, Copenhagen, September 2017. https://doi.org/10.18653/v1/W17-4750. https://www.aclweb.org/anthology/W17-4750
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, 7–9 May 2015. Conference Track Proceedings (2015). http://arxiv.org/abs/1412.6980
Lan, W., Li, X., Dong, J.: Fluency-guided cross-lingual image captioning. In: Proceedings of the 25th ACM International Conference on Multimedia, MM 2017, pp. 1549–1557. Association for Computing Machinery, New York (2017). https://doi.org/10.1145/3123266.3123366
Lin, C.: ROUGE: a package for automatic evaluation of summaries. Text Summarization Branches Out (2004)
Lin, T.-Y., et al.: Microsoft COCO: common objects in context. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8693, pp. 740–755. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10602-1_48
Miyazaki, T., Shimizu, N.: Cross-lingual image caption generation. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1780–1790. Association for Computational Linguistics, Berlin, August 2016. https://doi.org/10.18653/v1/P16-1168. https://www.aclweb.org/anthology/P16-1168
Ott, M., et al.: fairseq: a fast, extensible toolkit for sequence modeling. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations), pp. 48–53. Association for Computational Linguistics, Minneapolis, June 2019. https://doi.org/10.18653/v1/N19-4009. https://www.aclweb.org/anthology/N19-4009
Papineni, K., Roukos, S., Ward, T., Zhu, W.: BLEU: a method for automatic evaluation of machine translation, pp. 311–318. Association for Computational Linguistics (2002)
Specia, L., Frank, S., Sima’an, K., Elliott, D.: A shared task on multimodal machine translation and crosslingual image description. In: Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers, pp. 543–553. Association for Computational Linguistics, Berlin, August 2016. https://doi.org/10.18653/v1/W16-2346. https://www.aclweb.org/anthology/W16-2346
Thapliyal, A.V., Soricut, R.: Cross-modal language generation using pivot stabilization for web-scale language coverage. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 160–170. Association for Computational Linguistics, Online, July 2020. https://doi.org/10.18653/v1/2020.acl-main.16. https://www.aclweb.org/anthology/2020.acl-main.16
Vedantam, R., Zitnick, C., Parikh, D.: CIDEr: consensus-based image description evaluation. In: Computer Vision and Pattern Recognition, pp. 4566–4575 (2015)
Wu, Y., Zhao, S., Chen, J., Zhang, Y., Yuan, X., Su, Z.: Improving captioning for low-resource languages by cycle consistency. In: 2019 IEEE International Conference on Multimedia and Expo (ICME), pp. 362–367 (2019). https://doi.org/10.1109/ICME.2019.00070
Xu, K., et al.: Show, attend and tell: neural image caption generation with visual attention. In: Bach, F., Blei, D. (eds.) Proceedings of the 32nd International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 37, pp. 2048–2057. PMLR, Lille, 07–09 July 2015. http://proceedings.mlr.press/v37/xuc15.html
Acknowledgments
This work was funded by the German Federal Ministry of Research (BMBF) under grant number 01IW20005 (XAINES).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Biswas, R., Barz, M., Hartmann, M., Sonntag, D. (2021). Improving German Image Captions Using Machine Translation and Transfer Learning. In: Espinosa-Anke, L., Martín-Vide, C., Spasić, I. (eds) Statistical Language and Speech Processing. SLSP 2021. Lecture Notes in Computer Science(), vol 13062. Springer, Cham. https://doi.org/10.1007/978-3-030-89579-2_1
Download citation
DOI: https://doi.org/10.1007/978-3-030-89579-2_1
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-89578-5
Online ISBN: 978-3-030-89579-2
eBook Packages: Computer ScienceComputer Science (R0)