SpringerLink
Log in

Pruned Graph Neural Network for Short Story Ordering

  • Conference paper
  • First Online:
AIxIA 2021 – Advances in Artificial Intelligence (AIxIA 2021)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 13196))

Abstract

Text coherence is a fundamental problem in natural language generation and understanding. Organizing sentences into an order that maximizes coherence is known as sentence ordering. This paper is proposing a new approach based on the graph neural network approach to encode a set of sentences and learn orderings of short stories. We propose a new method for constructing sentence-entity graphs of short stories to create the edges between sentences and reduce noise in our graph by replacing the pronouns with their referring entities. We improve the sentence ordering by introducing an aggregation method based on majority voting of state-of-the-art methods and our proposed one. Our approach employs a BERT-based model to learn semantic representations of the sentences. The results demonstrate that the proposed method significantly outperforms existing baselines on a corpus of short stories with a new state-of-the-art performance in terms of Perfect Match Ratio (PMR) and Kendall’s Tau (\(\tau \)) metrics. More precisely, our method increases PMR and \(\tau \) criteria by more than 5% and 4.3%, respectively. These outcomes highlight the benefit of forming the edges between sentences based on their cosine similarity. We also observe that replacing pronouns with their referring entities effectively encodes sentences in sentence-entity graphs.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
EUR 29.95
Price includes VAT (Germany)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
EUR 93.08
Price includes VAT (Germany)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
EUR 117.69
Price includes VAT (Germany)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free ship** worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    The entity should be common to at least two sentences.

  2. 2.

    The paragraph vector is nonetheless influenced by the permutations of input sentences.

  3. 3.

    We use the Stanford’s tool [25].

  4. 4.

    For example, either \(s_1 s_2\) or \(s_2 s_1\) occurs, and without a doubt, the co-occurrence of these is a vast and impossible contradiction.

  5. 5.

    Suppose the outputs of the three methods for arranging \(sentence_1\) (\(s_1\)) and \(sentence_2\) (\(s_2\)) are: Method 1: \(s_1\) \(s_2\), Method 2: \(s_1\) \(s_2\), and Method 3: \(s_2\) \(s_1\). Therefore, the order \(s_1\) \(s_2\) gets two points and the order \(s_2\) \(s_1\) gets one, so \(s_1\) \(s_2\) applies to the final output.

  6. 6.

    Either of the two pair orderings that have an “or” between them.

  7. 7.

    [10] did not train ATTOrderNet on the ROCStories dataset.

  8. 8.

    Entity nodes are not connected to all nodes.

  9. 9.

    We train SE-Graph on ROCStories since [47] did not.

  10. 10.

    To demonstrate the advantages of the PG’s BERT-based sentence encoder, this component is considered exactly like the sentence encoder of SE-Graph and ATTOrderNet.

  11. 11.

    Entity nodes can only have a link to sentence nodes.

References

  1. Agrawal, H., Chandrasekaran, A., Batra, D., Parikh, D., Bansal, M.: Sort story: sorting jumbled images and captions into stories. In: Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 925–931. Association for Computational Linguistics, Austin (2016). https://doi.org/10.18653/v1/D16-1091

  2. Anderson, P., et al.: Bottom-up and top-down attention for image captioning and visual question answering. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6077–6086 (2018)

    Google Scholar 

  3. Barzilay, R., Elhadad, N.: Inferring strategies for sentence ordering in multidocument news summarization. J. Artif. Intell. Res. 17, 35–55 (2002)

    Article  Google Scholar 

  4. Barzilay, R., Lapata, M.: Modeling local coherence: an entity-based approach. Comput. Linguist. 34(1), 1–34 (2008)

    Article  Google Scholar 

  5. Barzilay, R., Lee, L.: Catching the drift: probabilistic content models, with applications to generation and summarization. In: Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL 2004, pp. 113–120 (2004)

    Google Scholar 

  6. Beck, D., Haffari, G., Cohn, T.: Graph-to-sequence learning using gated graph neural networks. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (vol. 1: Long Papers), pp. 273–283. Association for Computational Linguistics, Melbourne (2018). https://doi.org/10.18653/v1/P18-1026

  7. Chandu, K., Nyberg, E., Black, A.W.: Storyboarding of recipes: grounded contextual generation. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 6040–6046 (2019)

    Google Scholar 

  8. Chen, X., Qiu, X., Huang, X.: Neural sentence ordering. ar**v preprint ar**v:1607.06952 (2016)

  9. Christensen, J., Soderland, S., Etzioni, O.: Towards coherent multi-document summarization. In: Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1163–1173. Association for Computational Linguistics, Atlanta (2013). https://www.aclweb.org/anthology/N13-1136

  10. Cui, B., Li, Y., Chen, M., Zhang, Z.: Deep attentive sentence ordering network. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 4340–4349 (2018)

    Google Scholar 

  11. Elsner, M., Austerweil, J., Charniak, E.: A unified local and global model for discourse coherence. In: Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics, Proceedings of the Main Conference, pp. 436–443 (2007)

    Google Scholar 

  12. Fan, A., Lewis, M., Dauphin, Y.: Strategies for structuring story generation. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 2650–2660. Association for Computational Linguistics, Florence (2019). https://doi.org/10.18653/v1/P19-1254

  13. Galanis, D., Lampouras, G., Androutsopoulos, I.: Extractive multi-document summarization with integer linear programming and support vector regression. In: Proceedings of COLING 2012, pp. 911–926 (2012)

    Google Scholar 

  14. Golestani, M., Razavi, S.Z., Borhanifard, Z., Tahmasebian, F., Faili, H.: Using BERT encoding and sentence-level language model for sentence ordering. In: Ekštein, K., Pártl, F., Konopík, M. (eds.) Text, Speech, and Dialogue, pp. 318–330. Springer International Publishing, Cham (2021)

    Chapter  Google Scholar 

  15. Gong, J., Chen, X., Qiu, X., Huang, X.: End-to-end neural sentence ordering using pointer network. ar**v preprint ar**v:1611.04953 (2016)

  16. Guinaudeau, C., Strube, M.: Graph-based local coherence modeling. In: Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (vol. 1: Long Papers), pp. 93–103 (2013)

    Google Scholar 

  17. Hamilton, W.L., Ying, R., Leskovec, J.: Inductive representation learning on large graphs. In: Proceedings of the 31st International Conference on Neural Information Processing Systems, pp. 1025–1035. NIPS 2017, Curran Associates Inc., Red Hook, NY (2017)

    Google Scholar 

  18. Hu, J., Cheng, Y., Gan, Z., Liu, J., Gao, J., Neubig, G.: What makes a good story? Designing composite rewards for visual storytelling. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, pp. 7969–7976 (2020)

    Google Scholar 

  19. Johnson, D.: Learning graphical state transitions. In: ICLR (2017)

    Google Scholar 

  20. Kipf, T.N., Welling, M.: Semi-supervised classification with graph convolutional networks. ar**v preprint ar**v:1609.02907 (2017)

  21. Konstas, I., Lapata, M.: Concept-to-text generation via discriminative reranking. In: Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (vol. 1: Long Papers), pp. 369–378 (2012)

    Google Scholar 

  22. Kumar, P., Brahma, D., Karnick, H., Rai, P.: Deep attentive ranking networks for learning to order sentences. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, pp. 8115–8122 (2020)

    Google Scholar 

  23. Lapata, M.: Probabilistic text structuring: experiments with sentence ordering. In: Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics, pp. 545–552 (2003)

    Google Scholar 

  24. Lapata, M.: Automatic evaluation of information ordering: Kendall’s tau. Comput. Linguist. 32(4), 471–484 (2006). https://doi.org/10.1162/coli.2006.32.4.471

    Article  MATH  Google Scholar 

  25. Lee, H., Peirsman, Y., Chang, A., Chambers, N., Surdeanu, M., Jurafsky, D.: Stanford’s multi-pass sieve coreference resolution system at the CoNLL-2011 shared task. In: Proceedings of the 15th Conference on Computational Natural Language Learning: Shared Task, pp. 28–34. Association for Computational Linguistics (2011)

    Google Scholar 

  26. Li, J., Hovy, E.: A model of coherence based on distributed sentence representation. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 2039–2048 (2014)

    Google Scholar 

  27. Li, J., Jurafsky, D.: Neural net models of open-domain discourse coherence. In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 198–209. Association for Computational Linguistics, Copenhagen (2017). https://doi.org/10.18653/v1/D17-1019

  28. Liu, X., Shen, Y., Duh, K., Gao, J.: Stochastic answer networks for machine reading comprehension. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (vol. 1: Long Papers), pp. 1694–1704. Association for Computational Linguistics, Melbourne (2018). https://doi.org/10.18653/v1/P18-1157

  29. Logeswaran, L., Lee, H., Radev, D.: Sentence ordering and coherence modeling using recurrent neural networks. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 32 (2018)

    Google Scholar 

  30. Louis, A., Nenkova, A.: A coherence model based on syntactic patterns. In: Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pp. 1157–1168 (2012)

    Google Scholar 

  31. Marcheggiani, D., Titov, I.: Encoding sentences with graph convolutional networks for semantic role labeling. In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 1506–1515. Association for Computational Linguistics, Copenhagen (2017). https://doi.org/10.18653/v1/D17-1159

  32. Mostafazadeh, N., et al.: A corpus and cloze evaluation for deeper understanding of commonsense stories. In: Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 839–849. Association for Computational Linguistics, San Diego (2016). https://doi.org/10.18653/v1/N16-1098

  33. Mostafazadeh, N., Roth, M., Louis, A., Chambers, N., Allen, J.: LSDSem 2017 shared task: the story cloze test. In: Proceedings of the 2nd Workshop on Linking Models of Lexical, Sentential and Discourse-level Semantics, pp. 46–51. Association for Computational Linguistics, Valencia (2017). https://doi.org/10.18653/v1/W17-0906

  34. Nallapati, R., Zhai, F., Zhou, B.: SummaRuNNer: a recurrent neural network based sequence model for extractive summarization of documents. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 31 (2017)

    Google Scholar 

  35. Pour, M.G., Razavi, S.Z., Faili, H.: A new sentence ordering method using BERT pretrained model. In: 2020 11th International Conference on Information and Knowledge Technology (IKT), pp. 132–138. IEEE (2020)

    Google Scholar 

  36. Prabhumoye, S., Salakhutdinov, R., Black, A.W.: Topological sort for sentence ordering. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 2783–2792. Association for Computational Linguistics, Online (2020). https://doi.org/10.18653/v1/2020.acl-main.248

  37. Rahutomo, F., Kitasuka, T., Aritsugi, M.: Semantic cosine similarity. In: The 7th International Student Conference on Advanced Science and Technology ICAST, vol. 4, p. 1 (2012)

    Google Scholar 

  38. Reiter, E., Dale, R.: Building applied natural language generation systems. Nat. Lang. Eng. 3(1), 57–87 (1997)

    Article  Google Scholar 

  39. Song, L., Zhang, Y., Wang, Z., Gildea, D.: N-ary relation extraction using graph-state LSTM. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2226–2235. Association for Computational Linguistics, Brussels (2018). https://doi.org/10.18653/v1/D18-1246

  40. Sutskever, I., Vinyals, O., Le, Q.V.: Sequence to sequence learning with neural networks. In: Proceedings of the 27th International Conference on Neural Information Processing Systems, vol. 2, pp. 3104–3112. NIPS 2014, MIT Press, Cambridge (2014)

    Google Scholar 

  41. Wang, B., Kuo, C.C.J.: SBERT-WK: a sentence embedding method by dissecting BERT-based word models. IEEE/ACM Trans. Audio Speech Lang. Process. 28, 2146–2157 (2020)

    Article  Google Scholar 

  42. Wang, T., Wan, X.: Hierarchical attention networks for sentence ordering. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 7184–7191 (2019)

    Google Scholar 

  43. Wang, Z., Lv, Q., Lan, X., Zhang, Y.: Cross-lingual knowledge graph alignment via graph convolutional networks. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 349–357. Association for Computational Linguistics, Brussels (2018). https://doi.org/10.18653/v1/D18-1032

  44. Yanase, T., et al.: Learning sentence ordering for opinion generation of debate. In: Proceedings of the 2nd Workshop on Argumentation Mining, pp. 94–103 (2015)

    Google Scholar 

  45. Yasunaga, M., Zhang, R., Meelu, K., Pareek, A., Srinivasan, K., Radev, D.: Graph-based neural multi-document summarization. In: Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pp. 452–462. Association for Computational Linguistics, Vancouver (2017). https://doi.org/10.18653/v1/K17-1045

  46. Yin, Y., et al.: Enhancing pointer network for sentence ordering with pairwise ordering predictions. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, pp. 9482–9489 (2020)

    Google Scholar 

  47. Yin, Y., Song, L., Su, J., Zeng, J., Zhou, C., Luo, J.: Graph-based neural sentence ordering. In: Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI 2019, pp. 5387–5393. International Joint Conferences on Artificial Intelligence Organization (2019). https://doi.org/10.24963/ijcai.2019/748

  48. Yu, A.W., et al.: QANet: combining local convolution with global self-attention for reading comprehension. ar**v preprint ar**v:1804.09541 (2018)

  49. Zeng, X., Li, J., Wang, L., Beauchamp, N., Shugars, S., Wong, K.F.: Microblog conversation recommendation via joint modeling of topics and discourse. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, vol. 1 (Long Papers), pp. 375–385 (2018)

    Google Scholar 

  50. Zhang, Y., Liu, Q., Song, L.: Sentence-state LSTM for text representation. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (vol. 1: Long Papers), pp. 317–327. Association for Computational Linguistics, Melbourne (2018). https://doi.org/10.18653/v1/P18-1030

  51. Zhu, Y., Song, R., Dou, Z., Nie, J.Y., Zhou, J.: ScriptWriter: narrative-guided script generation. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 8647–8657. Association for Computational Linguistics, Online (2020). https://doi.org/10.18653/v1/2020.acl-main.765

Download references

Author information

Authors and Affiliations

  1. School of Electrical and Computer Engineering, College of Engineering, University of Tehran, Tehran, Iran

    Melika Golestani, Zeinab Borhanifard, Farnaz Tahmasebian & Heshaam Faili

  2. Alumni of Emory University, Atlanta, GA, USA

    Farnaz Tahmasebian

Authors
  1. Melika Golestani
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zeinab Borhanifard
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Farnaz Tahmasebian
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Heshaam Faili
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Melika Golestani .

Editor information

Editors and Affiliations

  1. Department of Informatics, Systems and Communication, University of Milano-Bicocca, Milan, Italy

    Stefania Bandini

  2. Department of Informatics, Systems and Communication, University of Milano-Bicocca, Milan, Italy

    Francesca Gasparini

  3. Department of Informatics, Bioengineering, Robotics and Systems Engineering, University of Genoa, Genova, Italy

    Viviana Mascardi

  4. Department of Informatics, Systems and Communication, University of Milano-Bicocca, Milan, Italy

    Matteo Palmonari

  5. Department of Informatics, Systems and Communication, University of Milano-Bicocca, Milan, Italy

    Giuseppe Vizzari

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Golestani, M., Borhanifard, Z., Tahmasebian, F., Faili, H. (2022). Pruned Graph Neural Network for Short Story Ordering. In: Bandini, S., Gasparini, F., Mascardi, V., Palmonari, M., Vizzari, G. (eds) AIxIA 2021 – Advances in Artificial Intelligence. AIxIA 2021. Lecture Notes in Computer Science(), vol 13196. Springer, Cham. https://doi.org/10.1007/978-3-031-08421-8_15

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-08421-8_15

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-08420-1

  • Online ISBN: 978-3-031-08421-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation