Abstract
Visualizations, such as charts, are crucial for interpreting complex data. However, they are often provided as raster images, which are not compatible with assistive technologies for people with blindness and visual impairments, such as embossed papers or tactile displays. At the same time, creating accessible vector graphics requires a skilled sighted person and is time-intensive. In this work, we leverage advancements in the field of chart analysis to generate tactile charts in an end-to-end manner. Our three key contributions are as follows: (1) introducing the ChartFormer model trained to convert raster chart images into tactile-accessible SVGs, (2) training this model on the Chart2Tactile dataset, a synthetic chart dataset we created following accessibility standards, and (3) evaluating the effectiveness of our SVGs through a pilot user study with an refreshable two-dimensional tactile display. Our work is publicly available at https://github.com/nsothman/ChartFormer.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Dürnegger, B., Feilmayr, C., Wöß, W.: Guided generation and evaluation of accessible scalable vector graphics. In: Miesenberger, K., Klaus, J., Zagler, W., Karshmer, A. (eds.) ICCHP 2010. LNCS, vol. 6179, pp. 27–34. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-14097-6_5
Engel, C., Müller, E.F., Weber, G.: SVGPlott: an accessible tool to generate highly adaptable, accessible audio-tactile charts for and from blind and visually impaired people. In: PETRA 2019 (2019)
Han, Y., et al.: ChartLlama: a multimodal LLM for chart understanding and generation (2023)
Liu, H., Li, C., Li, Y., Lee, Y.J.: Improved baselines with visual instruction tuning (2023)
Masson, D., Malacria, S., Vogel, D., Lank, E., Casiez, G.: Chartdetective: easy and accurate interactive data extraction from complex vector charts. In: CHI 2023 (2023)
Meng, F., et al.: Chartassisstant: a universal chart multimodal language model via chart-to-table pre-training and multitask instruction tuning (2024)
Moured, O., Alzalabny, S., Schwarz, T., Rapp, B., Stiefelhagen, R.: Accessible document layout: An interface for 2D tactile displays. In: Proceedings of the 16th International Conference on PErvasive Technologies Related to Assistive Environments, pp. 265–271 (2023)
Moured, O., Baumgarten-Egemole, M., Roitberg, A., Muller, K., Schwarz, T., Stiefelhagen, R.: Chart4blind: an intelligent interface for chart accessibility conversion. ar**v preprint ar**v:2403.06693 (2024)
Moured, O., Zhang, J., Roitberg, A., Schwarz, T., Stiefelhagen, R.: Line graphics digitization: a step towards full automation. In: Fink, G.A., Jain, R., Kise, K., Zanibbi, R. (eds.) ICDAR 2023. LNCS, vol. 14191, pp. 438–453. Springer, Cham (2023). https://doi.org/10.1007/978-3-031-41734-4_27
of North America, B.A.: Guidelines and standards for tactile graphics (2010). https://www.brailleauthority.org/tg/web-manual/index.html
Paterson, M.: Seeing with the hands’: Blindness, touch and the enlightenment spatial imaginary. Br. J. Vis. Impairment, 52–59 (2006)
Tang, B.J., Boggust, A., Satyanarayan, A.: Vistext: a benchmark for semantically rich chart captioning. ar**v preprint ar**v:2307.05356 (2023)
**a, R., et al.: Chartx & chartvlm: a versatile benchmark and foundation model for complicated chart reasoning (2024)
Acknowledgments
The authors would like to thank the HoreKa computing cluster at KIT for the computing resources used to conduct this research.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Ethics declarations
Disclosure of Interests
This research was funded by the European Research Council (ERC) grant and the European Union’s Horizon 2020 research and innovation program under the Marie Sklodowska-Curie grant agreements no. 816006 and 861166 respectively.
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Moured, O., Alzalabny, S., Osman, A., Schwarz, T., Müller, K., Stiefelhagen, R. (2024). ChartFormer: A Large Vision Language Model for Converting Chart Images into Tactile Accessible SVGs. In: Miesenberger, K., Peňáz, P., Kobayashi, M. (eds) Computers Hel** People with Special Needs. ICCHP 2024. Lecture Notes in Computer Science, vol 14750. Springer, Cham. https://doi.org/10.1007/978-3-031-62846-7_36
Download citation
DOI: https://doi.org/10.1007/978-3-031-62846-7_36
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-62845-0
Online ISBN: 978-3-031-62846-7
eBook Packages: Computer ScienceComputer Science (R0)