Topical Extractive Summarization

Abstract

In this paper we propose a graph-based biased LexRank approach combined with topic modeling to create a topic-based extractive summarization model. Topical summarization task is used to obtain a customized summary for a particular reader. We achieve so by including information about topics of interest into an extractive summarization model. Topical information is derived via aspect embedding vectors from the ABAE model and used as a topic input for the biased LexRank model that runs on the heterogeneous graph of topics and sentences. Inclusion of the topical structure into a summary increases the targeting of the summary and makes the overall summarization model benefit from it. We conduct experiments on a novel dataset for extractive summarization quality evaluation constructed by ourselves from Wikipedia articles.

A Appendix

1.1 A.1 Experiments with BigARTM and WikiPersons

BigARTM [7, 20] is an open source project for topic modeling of large collections. It allows to train various topic models including multimodal, hierarchical, temporal additively regularized variants. Several experiments on Wikipedia corpus show that BigARTM performs faster and gives better perplexity comparing to other popular packages, such as Vowpal Wabbit and Gensim.

The topic model was trained with the following parameters:

• number of most frequently used words in the text corpora to be included into the dictionary: 2000;

• number of topics: 50;

• weight coefficient for SmoothSparsePhiRegularizer: $-2$;

• weight coefficient for DecorrelatorPhiRegularizer: 0.00001;

• number of collection passes during the training: 35.

We also conducted the series of experiments varying the parameter d (dam** factor), testing the overall model performance on CNN/Daily Mail dataset (Table 3):

Table 3. Topical extractive summarization quality with additively regularized topic model and various d (dam** factor) tested on CNN/Daily Mail dataset.

1.2 A.2 Experiments with Neural Topic Models and WikiPersons

Neural topic models (NTM) provide an optimizable coherence awareness from a text corpora. NTM uses neural variational inference, while basic models such as LDA relies on Gibbs sampling and Bayesian variational inference which appear to be mathematically cumbersome and should be re-derivated even after a small change of modeling assumptions. In our experiments we considered following neural topic models:

• NVDM-GSM [12]. The architecture of the model is a simple VAE [9], which takes the BOW (bag of words) of a document as an input. The topic vector is sampled from the distribution Q(z|x) and then normalized using softmax function (for more details please refer to Fig. 3, which is taken from the Github repo “Neural Topic Models” [24]).

• ETM [4]. The architecture is a straightforward VAE [9], with the topic-word distribution matrix decomposed as the product of the matrix with topic vectors and the matrix with word vectors (Fig. 4). This model improves the interpretability of topics by locating the topic vectors and the word vectors in the same vector space.

• WTM-GMM. An improved model of the WLDA (Weak supervised LDA). Architecturally it is a Wasserstein Auto-Encoder (WAE) [?], that takes Gaussian mixture distribution as a prior distribution and uses Gaussian softmax. The number of components in the Gaussian mixture is usually the same as the number of topics in the model. A detailed scheme of WTM-GMM is presented in Fig. 5 (the picture is taken from the Github repo “Neural Topic Models” [24]).

Fig. 3.

NVDM-GSM architecture

Fig. 4.

ETM architecture

Fig. 5.

WTM-GMM architecture

All model implementations were taken from the Neural Topic Models repo¹. For all the models we set number of topics to 300 and batch size to 512. NVDM-GSM and WTM-GMM were trained for 150 epochs, while ETM was trained for 160 epochs.

Table 4. Comparison between neural topic models (no heterogeneous graph) on WikiPersons dataset

We provide qualitative comparison between the aforementioned neural topic models in Table 4. We compare different NTM using the following criteria (these metrics are taken from the Coherence model from the library Gensim [17]):

• c_v measure is based on a sliding window, one-set segmentation of the top words and an indirect confirmation measure that uses normalized pointwise mutual information (NPMI) and the cosine similarity

• c_uci measure is based on a sliding window and the pointwise mutual information (PMI) of all word pairs of the given top words:

• c_npmi is an enhanced version of the c_uci coherence using the normalized pointwise mutual information (NPMI).

According to the Table 4 and also manual evaluation of topics by its top tokens, NVDM-GSM model performs better than all the other mentioned models.

Other experimentation included varying the parameter d. However, the average precision increased from 0.643 only to 0.65 with the growth of the dam** value.