Abstract
Handwritten chemical equation recognition is an appealing task, but its development is hampered by the lack of publicly available datasets. To this end, we propose a multi-level synthesis strategy to synthesize the corresponding handwritten equations from LaTeX expressions and regard the chemical equation recognition as an image-to-markup task. In particular, our approach first decomposes the LaTeX expression into a symbol layout tree (SLT) and obtains different multi-level components in stages by traversing the SLT. Then, online isolated symbols are placed in appropriate locations consistent with handwritten habits through a baseline-based layout strategy. Furthermore, expression patterns are enhanced at the local, component, and global levels to increase the diversity of synthesized data. It is worth noting that our synthesis strategy is theoretically applicable to any LaTeX-based expression. We also collected a real dataset containing 1595 handwritten chemical equations, and the experimental results confirm that our proposed method can effectively improve the performance of handwritten chemical equation recognition systems. The dataset we generated will be released.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Alonso, E., Moysset, B., Messina, R.: Adversarial generation of handwritten text images conditioned on sequences. In: 2019 International Conference on Document Analysis and Recognition (ICDAR), pp. 481–486. IEEE (2019)
Anderson, R.H.: Syntax-directed recognition of hand-printed two-dimensional mathematics. In: Symposium on Interactive Systems for Experimental Applied Mathematics: Proceedings of the Association for Computing Machinery Inc., Symposium, pp. 436–459 (1967)
Awal, A.-M., Mouchère, H., Viard-Gaudin, C.: Towards handwritten mathematical expression recognition. In: 2009 10th International Conference on Document Analysis and Recognition, pp. 1046–1050. IEEE (2009)
Belaid, A., Haton, J.-P.: A syntactic approach for handwritten mathematical formula recognition. IEEE Trans. Pattern Anal. Mach. Intell. PAMI-6(1), 105–111 (1984)
Bian, X., Qin, B., **n, X., Li, J., Xuefeng, S., Wang, Y.: Handwritten mathematical expression recognition via attention aggregation based bi-directional mutual learning. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, pp. 113–121 (2022)
Chang, M., Han, S., Zhang, D.: A unified framework for recognizing handwritten chemical expressions. In: 2009 10th International Conference on Document Analysis and Recognition, pp. 1345–1349. IEEE (2009)
Deng, Y., Kanervisto, A., Ling, J., Rush, A.M.: Image-to-markup generation with coarse-to-fine attention. In: International Conference on Machine Learning, pp. 980–989. PMLR (2017)
Graves, A.: Generating sequences with recurrent neural networks. ar**v preprint ar**v:1308.0850 (2013)
Hagag, A., Omara, I., Alfarra, A.N.K., Mekawy, F.: Handwritten chemical formulas classification model using deep transfer convolutional neural networks. In: 2021 International Conference on Electronic Engineering (ICEEM), pp. 1–6. IEEE (2021)
Khuong, V.T.M., Huy, U.Q., Masaki, N., Phan, M.K.: Generating synthetic handwritten mathematical expressions from a latex sequence or a mathml script. In: 2019 International Conference on Document Analysis and Recognition (ICDAR), pp. 922–927. IEEE (2019)
Le, A.D., Indurkhya, B., Nakagawa, M.: Pattern generation strategies for improving recognition of handwritten mathematical expressions. Pattern Recogn. Lett. 128, 255–262 (2019)
Li, B., et al.: When counting meets HMER: counting-aware network for handwritten mathematical expression recognition. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds.) ECCV 2022. LNCS, vol. 13688, pp. 197–214. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-19815-1_12
Li, Z., **, L., Lai, S., Zhu, Y.: Improving attention-based handwritten mathematical expression recognition with scale augmentation and drop attention. In: 2020 17th International Conference on Frontiers in Handwriting Recognition (ICFHR), pp. 175–180. IEEE (2020)
Liu, C.-L., Yin, F., Wang, D.-H., Wang, Q.-F.: Casia online and offline Chinese handwriting databases. In: 2011 International Conference on Document Analysis and Recognition, pp. 37–41. IEEE (2011)
Liu, X., Zhang, T., Yu, X.: An end-to-end trainable system for offline handwritten chemical formulae recognition. In: 2019 International Conference on Document Analysis and Recognition (ICDAR), pp. 577–582. IEEE (2019)
MacLean, S., Labahn, G., Lank, E., Marzouk, M., Tausky, D.: Grammar-based techniques for creating ground-truthed sketch corpora. Int. J. Doc. Anal. Recogn. (IJDAR) 14, 65–74 (2011)
Miller, E.G., Viola, P.A.: Ambiguity and constraint in mathematical expression recognition. In: AAAI/IAAI, pp. 784–791 (1998)
Mouchere, H., Viard-Gaudin, C., Zanibbi, R., Garain, U., Kim, D.H., Kim, J.H.: ICDAR 2013 crohme: third international competition on recognition of online handwritten mathematical expressions. In: 2013 12th International Conference on Document Analysis and Recognition, pp. 1428–1432. IEEE (2013)
Parmar, N., et al.: Image transformer. In: International Conference on Machine Learning, pp. 4055–4064. PMLR (2018)
Truong, T.-N., Nguyen, C.T., Nakagawa, M.: Syntactic data generation for handwritten mathematical expression recognition. Pattern Recogn. Lett. 153, 83–91 (2022)
Wang, Y., Zhang, T., Yu, X.: A component-detection-based approach for interpreting off-line handwritten chemical cyclic compound structures. In: 2021 IEEE International Conference on Engineering, Technology and Education (TALE), pp. 785–791. IEEE (2021)
Zanibbi, R., Blostein, D.: Recognition and retrieval of mathematical expressions. Int. J. Doc. Anal. Recogn. (IJDAR) 15, 331–357 (2012)
Zhang, J., Du, J., Dai, L.: Multi-scale attention with dense encoder for handwritten mathematical expression recognition. In: 2018 24th International Conference on Pattern Recognition (ICPR), pp. 2245–2250. IEEE (2018)
Zhang, J., et al.: Watch, attend and parse: an end-to-end neural network based approach to handwritten mathematical expression recognition. Pattern Recogn. 71, 196–206 (2017)
Zhang, T., Mouchère, H., Viard-Gaudin, C.: A tree-BLSTM-based recognition system for online handwritten mathematical expressions. Neural Comput. Appl. 32, 4689–4708 (2020)
Zhang, Z., He, T., Zhang, H., Zhang, Z., **e, J., Li, M.: Bag of freebies for training object detection neural networks. ar**v preprint ar**v:1902.04103 (2019)
Zhao, W., Gao, L., Yan, Z., Peng, S., Du, L., Zhang, Z.: Handwritten mathematical expression recognition with bidirectionally trained transformer. In: Lladós, J., Lopresti, D., Uchida, S. (eds.) ICDAR 2021. LNCS, vol. 12822, pp. 570–584. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-86331-9_37
Zhou, S., Chen, Q., Wang, X.: Hit-OR3C: an opening recognition corpus for Chinese characters. In: Proceedings of the 9th IAPR International Workshop on document analysis systems, pp. 223–230 (2010)
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Shen, H., Li, J., Lin, J., Wu, W. (2023). A Multi-level Synthesis Strategy for Online Handwritten Chemical Equation Recognition. In: Fink, G.A., Jain, R., Kise, K., Zanibbi, R. (eds) Document Analysis and Recognition - ICDAR 2023. ICDAR 2023. Lecture Notes in Computer Science, vol 14187. Springer, Cham. https://doi.org/10.1007/978-3-031-41676-7_12
Download citation
DOI: https://doi.org/10.1007/978-3-031-41676-7_12
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-41675-0
Online ISBN: 978-3-031-41676-7
eBook Packages: Computer ScienceComputer Science (R0)
