Abstract
This paper presents an approach to creating interpretable word embeddings, in which each component of the vector corresponds to some interpretable semantic category. To obtain such categories, a lexico-semantic resource is used in the form of the RuWordNet semantic network, as well as a representative corpus of Russian-language texts to train vector representations. The resulting interpretable embeddings were evaluated on semantic similarity tasks.
This is a preview of subscription content, log in via an institution to check access.
REFERENCES
J. Devlin, M. W. Chang, K. Lee, and K. Toutanova, ‘‘BERT: Pre-training of deep bidirectional transformers for language,’’ in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (2019), Vol. 1, pp. 4171–4186.
T. Mikolov et al., ‘‘Distributed representations of words and phrases and their compositionality,’’ ar**v: 1310.4546 (2013).
M. Artetxe, G. Labaka, and E. Agirre, ‘‘Learning principled bilingual map**s of word embeddings while preserving monolingual invariance,’’ in Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (2016), pp. 2289–2294.
T. Mikolov, Q. V. Le, and I. Sutskever, ‘‘Exploiting similarities among languages for machine translation,’’ ar**v: 1309.4168 (2013).
J. Yamane et al., ‘‘Distributional hypernym generation by jointly learning clusters and projections,’’ in Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers (2016), pp. 1871–1879.
M. Radovanovic, A. Nanopoulos, and M. Ivanovic, ‘‘Hubs in space: Popular nearest neighbors in high-dimensional data,’’ J. Mach. Learn. Res. 11, 2487–2531 (2010).
S. Ruder, I. Vulić, and A. Søgaard, ‘‘A survey of cross-lingual word embedding models,’’ J. Artif. Intell. Res. 65, 569–631 (2019).
G. A. Miller, WordNet: An Electronic Lexical Database (MIT, Boston, 1998).
N. V. Loukachevitch et al., ‘‘Creating Russian wordnet by conversion,’’ in Computational Linguistics and Intellectual Technologies: Proceedings of the Annual Conference Dialogue (2016), pp. 405–415.
L. Flekova and I. Gurevych, ‘‘Supersense embeddings: A unified model for supersense interpretation, prediction, and utilization,’’ in Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, Vol. 1: Long Papers (2016), pp. 2029–2041.
R. Navigli and S. P. Ponzetto, ‘‘BabelNet: Building a very large multilingual semantic network,’’ in Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics (2010), pp. 216–225.
F. Scozzafava et al., ‘‘Automatic identification and disambiguation of concepts and named entities in the multilingual wikipedia,’’ in AI* IA 2015 Advances in Artificial Intelligence: Proceedings of the 14th International Conference of the Italian Association for Artificial Intelligence, Ferrara, Italy, September 23–25, 2015 (Springer Int., Switzerland, 2015), pp. 357–366.
E. Agirre et al., ‘‘A study on similarity and relatedness using distributional and wordnet-based approaches,’’ in Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, Anthology N09-1003 (2009).
I. Leviant and R. Reichart, ‘‘Separated by an un-common language: Towards judgment language informed vector space modeling,’’ ar**v: 1508.00106 (2015).
A. Panchenko et al., ‘‘Human and machine judgements for Russian semantic relatedness,’’ in Proceedings of the International Conference on Analysis of Images, Social Networks and Texts (Springer, Cham, 2016), pp. 221–235.
C. Aloui et al., ‘‘Slice: Supersense-based lightweight interpretable contextual embeddings,’’ in Proceedings of the 28th International Conference on Computational Linguistics COLING 2020 (2020).
H. Le et al., ‘‘Flaubert: Unsupervised language model pre-training for french,’’ ar**v: 1912.05372 (2019).
L. K. Şenel et al., ‘‘Learning interpretable word embeddings via bidirectional alignment of dimensions with semantic concepts,’’ Inform. Process. Manage. 59, 102925 (2022).
N. Loukachevitch, G. Lashevich, and B. Dobrov, ‘‘Comparing two thesaurus representations for Russian,’’ in Proceedings of the 9th Global WordNet Conference GWC 2018 (2018), pp. 35–44.
S. Brin and L. Page, ‘‘The anatomy of a large-scale hypertextual web search engine,’’ Comput. Networks ISDN Syst. 30, 107–117 (1998).
F. Hill, R. Reichart, and A. Korhonen, ‘‘Simlex-999: Evaluating semantic models with (genuine) similarity estimation,’’ Comput. Linguist. 41, 665–695 (2015).
J. Shen et al., ‘‘TaxoExpan: Self-supervised taxonomy expansion with position-enhanced graph neural network,’’ in Proceedings of The Web Conference 2020 (2020), pp. 486–497.
T. N. Kipf and M. Welling, ‘‘Semi-supervised classification with graph convolutional networks,’’ ar**v: 1609.02907 (2016).
P. Veličkovič et al., ‘‘Graph attention networks,’’ ar**v: 1710.10903 (2017).
I. Nikishina et al., ‘‘Taxonomy enrichment with text and graph vector representations,’’ Semantic Web 13, 441–475 (2022).
Funding
The research is carried out using the equipment of the shared research facilities of HPC computing resources at Lomonosov Moscow State University. The study was funded by a grant Russian Science Foundation (project no. 21-71-30003). The work of Mikhail Tikhomirov in conducting comparative experiments was supported by Non-commercial Foundation for Support of Science and Education ‘‘INTELLECT.’’
Author information
Authors and Affiliations
Corresponding authors
Additional information
(Submitted by E. E. Tyrtyshnikov)
Rights and permissions
About this article
Cite this article
Tikhomirov, M.M., Loukachevitch, N.V. Method for Generating Interpretable Embeddings Based on Superconcepts. Lobachevskii J Math 44, 3169–3177 (2023). https://doi.org/10.1134/S199508022308053X
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1134/S199508022308053X