Abstract
In this paper, scientific species names from images of handwritten species observations are automatically recognised and annotated with semantic concepts, so that they can be used for document retrieval and faceted search. Until now, automated semantic annotation of such named entities was only applied to printed or digital text. We employ a two-step approach. First, word images are classified, identifying elements of scientific species names; Genus, species, author, using (i) visual structural features, (ii) position, and (iii) context. Second, the identified species names are semantically annotated according to the NHC-Ontology, an ontology that describes species observations. Internationalised Resource Identifiers (IRIs) are assigned to the elements so that they can be linked and disambiguated at a later stage by individual researchers. For the identification of scientific species names, we achieve an average F1 score of 0.86. Moreover, we discuss how our method will function in a semi-automated annotation process, with a fruitful dialogue between system and user as the main objective.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
- 2.
- 3.
- 4.
- 5.
10.5281/zenodo.2545573.
- 6.
- 7.
- 8.
- 9.
- 10.
nc: is the prefix for the http://makingsense.liacs.nl/rdf/nhc/nc# namespace.
- 11.
- 12.
http://makingsense.liacs.nl/rdf4j-server/repositories/SN, can be queried through a query editor such as: https://yasgui.org/.
References
Adak, C., Chaudhuri, B.B., Blumenstein, M.: Named entity recognition from unstructured handwritten document images. In: 12th IAPR Workshop on Document Analysis Systems (DAS), pp. 375–380. IEEE (2016)
Buda, M., Maki, A., Mazurowski, M.A.: A systematic study of the class imbalance problem in convolutional neural networks. Neural Netw. 106, 249–259 (2018)
Bulacu, M., van Koert, R., Schomaker, L., van der Zant, T.: Layout analysis of handwritten historical documents for searching the archive of the cabinet of the Dutch queen. In: Proceedings of the Ninth International Conference on Document Analysis and Recognition, vol. 1, 2, pp. 357–361. IEEE (2007)
Canfield, M.R.: Field Notes on Science & Nature. Harvard University Press, Cambridge (2011)
Carbonell, M., Villegas, M., Fornés, A., Lladós, J.: Joint recognition of handwritten text and named entities with a neural end-to-end model. ar**v preprint ar**v:1803.06252 (2018)
Deng, J., Dong, W., Socher, R., Li, L., Li, K., Fei-Fei, L.: Imagenet: a large-scale hierarchical image database. In: Proceedings of the 2009 Computer Vision and Pattern Recognition, CVPR, pp. 248–255. IEEE (2009)
Drew, J.A., Moreau, C.S., Stiassny, M.L.: Digitization of museum collections holds the potential to enhance researcher diversity. Nature Ecol. Evol. 1(12), 1789–1790 (2017)
Drinkwater, R.E., Cubey, R.W., Haston, E.M.: The use of optical character recognition (OCR) in the digitisation of herbarium specimen labels. PhytoKeys 38, 15–30 (2014)
Gwinn, N.E., Rinaldo, C.: The biodiversity heritage library: sharing biodiversity literature with the world. IFLA J. 35(1), 25–34 (2009)
Heidorn, P.B., Wei, Q.: Automatic metadata extraction from museum specimen labels. In: Proceedings of the 2008 International Conference on Dublin Core and Metadata Applications, pp. 57–68 (2008)
Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)
Kennedy, J.B., Kukla, R., Paterson, T.: Scientific names are ambiguous as identifiers for biological taxa: their context and definition are required for accurate data integration. In: Ludäscher, B., Raschid, L. (eds.) DILS 2005. LNCS, vol. 3615, pp. 80–95. Springer, Heidelberg (2005). https://doi.org/10.1007/11530084_8
Koning, D., Sarkar, I.N., Moritz, T.: Taxongrab: extracting taxonomic names from text. Biodivers. Inf. 2, 79–82 (2005)
LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)
MacGregor, A. (ed.): Naturalists in the Field. Brill, Leiden (2018)
Chollet, F., et al.: Keras (2015). https://keras.io
McCallum, A., Li, W.: Early results for named entity recognition with conditional random fields, feature induction and web-enhanced lexicons. In: Proceedings of the Seventh Conference on Natural Language Learning, vol. 4, pp. 188–191. Association for Computational Linguistics (2003)
Miracle, M.E.G.: On whose authority? Temminck’s debates on zoological classification and nomenclature: 1820–1850. J. Hist. Biol. 44(3), 445–481 (2011)
Müller-Wille, S.: Names and numbers: “data” in classical natural history, 1758–1859. Osiris 32(1), 109–128 (2017)
Oquab, M., Bottou, L., Laptev, I., Sivic, J.: Learning and transferring mid-level image representations using convolutional neural networks. In: Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1717–1724 (2014)
Page, L.M., MacFadden, B.J., Fortes, J.A., Soltis, P.S., Riccardi, G.: Digitization of biodiversity collections reveals biggest data on biodiversity. BioScience 65(9), 841–842 (2015)
Sarkar, I.N.: Biodiversity informatics: organizing and linking information across the spectrum of life. Briefings Bioinform. 8(5), 347–357 (2007)
Schomaker, L.: Design considerations for a large-scale image-based text search engine in historical manuscript collections. It - Inf. Technol. 58(2), 80–88 (2016)
Schuster, M., Paliwal, K.K.: Bidirectional recurrent neural networks. IEEE Trans. Signal Process. 45(11), 2673–2681 (1997)
Shi, Z.: Datefinder: detecting date regions on handwritten document images based on positional expectancy. Master’s thesis, University of Groningen, Groningen, the Netherlands (2016)
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. CoRR, abs/1409.1556 (2014)
Stork, L., et al.: Semantic annotation of natural history collections. Web Semant. Sci. Serv. Agents World Wide Web (2018). https://doi.org/10.1016/j.websem.2018.06.002
Toledo, J.I., Sudholt, S., Fornés, A., Cucurull, J., Fink, G.A., Lladós, J.: Handwritten word image categorization with convolutional neural networks and spatial pyramid pooling. In: Robles-Kelly, A., Loog, M., Biggio, B., Escolano, F., Wilson, R. (eds.) S+SSPR 2016. LNCS, vol. 10029, pp. 543–552. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-49055-7_48
Van der Zant, T., Schomaker, L., Haak, K.: Handwritten-word spotting using biologically inspired features. IEEE Trans. Pattern Anal. Mach. Intell. 30(11), 1945–1957 (2008)
van Oosten, J.-P., Schomaker, L.: Separability versus prototypicality in handwritten word-image retrieval. Pattern Recogn. 47(3), 1031–1038 (2014)
Weber, A., Ameryan, M., Wolstencroft, K., Stork, L., Heerlien, M., Schomaker, L.: Towards a digital infrastructure for illustrated handwritten archives. In: Ioannides, M. (ed.) Digital Cultural Heritage. LNCS, vol. 10605, pp. 155–166. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-75826-8_13
Acknowledgements
This work is supported by the Netherlands Organisation for Scientific Research (NWO), grant 652.001.001, and Brill publishers.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Stork, L., Weber, A., van den Herik, J., Plaat, A., Verbeek, F., Wolstencroft, K. (2019). Automated Semantic Annotation of Species Names in Handwritten Texts. In: Azzopardi, L., Stein, B., Fuhr, N., Mayr, P., Hauff, C., Hiemstra, D. (eds) Advances in Information Retrieval. ECIR 2019. Lecture Notes in Computer Science(), vol 11437. Springer, Cham. https://doi.org/10.1007/978-3-030-15712-8_43
Download citation
DOI: https://doi.org/10.1007/978-3-030-15712-8_43
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-15711-1
Online ISBN: 978-3-030-15712-8
eBook Packages: Computer ScienceComputer Science (R0)