SpringerLink
Log in

Improving rating predictions with time-varying attention and dual-optimizer

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Due to the rapid expansion of online platforms, review-based recommender systems have become the standard way to determine consumers’ preferences for various goods. The purpose of this study is to address three significant issues with review-based approaches. First, these techniques suffer from a class-imbalance problem, where rating levels with lower proportions are partially neglected. As a result, their rating performance for such rare levels is unsatisfactory. To address this drawback, this paper proposes a flexible dual-optimizer network that enhances performance by utilizing classification and regression losses in the first attempt to solve this problem. Second, the typical review-based approach of preprocessing review texts with word embeddings will result in inadequate contextual information extraction. Thus, the bidirectional encoder representations from the transformers (BERT) method is introduced as the preprocessing technique to fully extract semantic information. Third, existing techniques make recommendations without considering that the user preferences may change over time. We suggest a time-varying feature extraction scheme that includes a multiscale convolutional neural network and bidirectional long short-term memory to solve this problem. We then develop an interactive component that summarizes the contextual information associated with the user-item pairings. The efficiency of the proposed time-varying attention with dual-optimizer (TADO) model is demonstrated through extensive tests on 12 benchmark datasets derived from Amazon Product Reviews. Regarding the existing state-of-the-art techniques, the proposed model attains a substantial performance increase of 22.69%, 8.75%, and 17.35%, respectively, over the Aspect-aware Latent Factor Model (ALFM), Multi-Pointer Co-Attention Networks for recommendation (MPCN), and Aspect-based Neural Recommender (ANR). Moreover, an ablation study demonstrates the importance of jointly employing the components developed for TADO.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Algorithm 1
Fig. 4

Similar content being viewed by others

Notes

  1. http://jmcauley.ucsd.edu/data/amazon/

  2. https://nlp.stanford.edu/projects/glove/

References

  1. Koren Y (2008) “Factorization meets the neighborhood: a multifaceted collaborative filtering model.” In Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 426-434

  2. Aljunid MF, Dh M (2020) An Efficient Deep Learning Approach for Collaborative Filtering Recommender System. Proc Comput Sci 171:829–836

    Article  Google Scholar 

  3. Li C, Quan C, Peng L, Qi Y, Deng Y, Wu L (2019) “A Capsule Network for Recommendation and Explaining What You Like and Dislike.” In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 275-284

  4. Yin R, Li K, Zhang G, Lu J (2019) A deeper graph neural network for recommender systems. Knowl Based Syst 185:105020

  5. Seo S, Huang J, Yang H, Liu Y (2017) “Interpretable convolutional neural networks with dual local and global attention for review rating prediction.” In Proceedings of the eleventh ACM conference on recommender systems, pp. 297-305

  6. Zheng L, Noroozi V, Yu PS (2017) “Joint deep modeling of users and items using reviews for recommendation.” In Proceedings of the Tenth ACM International Conference on Web Search and Data Mining, pp. 425-434

  7. McAuley J, Leskovec J (2013) “Hidden factors and hidden topics: understanding rating dimensions with review text.” In Proceedings of the 7th ACM conference on Recommender systems, pp. 165-172

  8. Bauman K, Liu B, Tuzhilin A (2017) “Aspect based recommendations: Recommending items with the most valuable aspects based on user reviews.” In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 717-725

  9. Hu G (2019) “Personalized Neural Embeddings for Collaborative Filtering with Text.” In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 2082-2088

  10. Nikolenko SI, Tutubalina E, Malykh V, Shenbin I, Alekseev A (2019) “AspeRa: aspect-based rating prediction model.” In European Conference on Information Retrieval, pp. 163-171. Springer, Cham

  11. He X, Liao L, Zhang H, Nie L, Hu X, Chua TS (2017) “Neural collaborative filtering.” In Proceedings of the 26th international conference on world wide web, pp. 173-182

  12. Catherine R, Cohen W (2017) “Transnets: Learning to transform for recommendation.” In Proceedings of the eleventh ACM conference on recommender systems, pp. 288-296

  13. Tay, Yi, Anh Tuan Luu, and Siu Cheung Hui. “Multi-pointer co-attention networks for recommendation.” In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 2309-2318. 2018

  14. Chin JY, Zhao K, Joty S, Cong G (2018) “ANR: Aspect-based neural recommender.” In Proceedings of the 27th ACM International Conference on Information and Knowledge Management, pp. 147-156

  15. Guan X, Cheng Z, He X, Zhang Y, Zhu Z, Peng Q, Chua TS (2019) Attentive aspect modeling for review-aware recommendation. ACM Trans Inf Syst 37(3):1–27

    Article  Google Scholar 

  16. Devlin J, Chang MW, Lee K, Toutanova K (2018) “Bert: Pre-training of deep bidirectional transformers for language understanding.” ar**v preprint ar**v:1810.04805

  17. He R, Lee WS, Ng HT, Dahlmeier D (2017) “An unsupervised neural attention model for aspect extraction.” In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 388–397

  18. Seo S, Huang J, Yang H, Liu Y (2017) “Interpretable convolutional neural networks with dual local and global attention for review rating prediction.” In Proceedings of the eleventh ACM conference on recommender systems, pp. 297–305

  19. Nallapati R, Zhou B, dos Santos C, Gulçehre Ç, **ang B Abstractive Text Summarization using Sequence-to-sequence RNNs and Beyond. In Proceedings of The 20th Conference on Computational Natural Language Learning pp. 280-290

  20. Parikh A, Täckström O, Das D, Uszkoreit J (2016) “A Decomposable Attention Model for Natural Language Inference.” In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2249-2255

  21. Tay Y, Tuan LA, Hui SC (2018) “Compare, Compress and Propagate: Enhancing Neural Architectures with Alignment Factorization for Natural Language Inference.” In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 1565-1575

  22. Cheng Z, Ding Y, Zhu L, Kankanhalli M (2018) “Aspect-aware latent factor model: Rating prediction with ratings and reviews.” In Proceedings of the 2018 world wide web conference, pp. 639-648

  23. Chen C, Zhang M, Liu Y, Ma S (2018) “Neural attentional rating regression with review-level explanations.” In Proceedings of the 2018 World Wide Web Conference, pp. 1583-1592

  24. Cheng Z, Ding Y, He X, Zhu L, Song X, Kankanhalli MS (2018) “A\(\hat{3}\)NCF: An Adaptive Aspect Attention Model for Rating Prediction.” In IJCAI, pp. 3748-3754

  25. Eskandanian F, Mobasher B (2019) “Modeling the Dynamics of User Preferences for Sequence-Aware Recommendation Using Hidden Markov Models.” In The Thirty-Second International Flairs Conference

  26. Chen Q, Zhao H, Li W, Huang P, Ou W (2019) “Behavior sequence transformer for e-commerce recommendation in Alibaba.” In Proceedings of the 1st International Workshop on Deep Learning Practice for High-Dimensional Sparse Data, pp. 1-4

  27. Song K, Ji M, Park S, Moon IC (2019) Hierarchical Context Enabled Recurrent Neural Network for Recommendation. In Proceedings of the AAAI Conference on Artificial Intelligence 33:4983–4991

  28. Pennington J, Socher R, Manning CD (2014) “Glove: Global vectors for word representation.” In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 1532-1543

  29. Bauman K, Liu B, Tuzhilin A (2017) “Aspect based recommendations: Recommending items with the most valuable aspects based on user reviews.” In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 717-725

  30. Peters ME, Neumann M, Iyyer M, Gardner M, Clark C, Lee K, Zettlemoyer L (2018) “Deep contextualized word representations.” In Proceedings of NAACL-HLT, pp. 2227-2237

  31. Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser Ł, Polosukhin I (2017) “Attention is all you need.” In Advances in neural information processing systems, pp. 5998-6008

  32. Devlin J, Chang MW, Lee K, Toutanova K (2018) Bert: Pre-training of deep bidirectional transformers for language understanding. ar**v preprint ar**v:1810.04805

  33. Radford A, Narasimhan K, Salimans T, Sutskever I “Improving Language Understanding by Generative Pre-Training.”

  34. Seo S, Huang J, Yang H, Liu Y (2017) “Representation learning of users and items for review rating prediction using attention-based convolutional neural network.” In International Workshop on Machine Learning Methods for Recommender Systems

  35. Wilcoxon F (1945) Individual comparisons by ranking methods. Biometrics Bull 1(6):80–83

    Article  Google Scholar 

  36. Wu Y, Gao T, Wang S, **ong Z (2020) “TADO: Time-Varying Attention with Dual-Optimizer Model,” 2020 IEEE International Conference on Data Mining (ICDM), pp. 1340-1345. IEEE

Download references

Acknowledgements

This work is supported by the National Natural Science Foundation of China (No.61872254, No.62162057), the Sichuan Science and Technology Program (2021JDRC0004), and the Key Lab of Information Network Security of the Ministry of Public Security (C20606). Zhengji Li and Yuexin Wu contributed equally to this paper. We wish to express our appreciation to ** Yang (corresponding author) for his valuable advice when writing this essay. Moreover, we wish to convey our appreciation to Tianyu Gao, who helped write this manuscript. This paper extends our 2020 IEEE International Conference on Data Mining (ICDM) paper [36]. Codes are published at Code Ocean (https://doi.org/10.24433/CO.8288426.v1).

Author information

Authors and Affiliations

  1. School of Cyber Science and Engineering, Sichuan University, Chengdu, 610207, Sichuan, China

    Zhengji Li & ** Yang

  2. Department of Information Engineering and Computer Science, University of Trento, Trento, 38123, Trento, Italy

    Zhengji Li

  3. Department of Computer Science, The University of Memphis, Memphis, TN, 38152, USA

    Yuexin Wu

  4. School of Computer Science, China West Normal University, Nanchong, 637002, Sichuan, China

    Jiangchuan Chen

  5. School of Computing and Artificial Intelligence, Southwest Jiaotong University, Chengdu, 611756, Sichuan, China

    Tianrui Li

Authors
  1. Zhengji Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yuexin Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. ** Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jiangchuan Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Tianrui Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to ** Yang.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Zhengji Li and Yuexin Wu contributed equally to this work.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, Z., Wu, Y., Yang, J. et al. Improving rating predictions with time-varying attention and dual-optimizer. Appl Intell 53, 26098–26109 (2023). https://doi.org/10.1007/s10489-023-04943-4

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-023-04943-4

Keywords

Search

Navigation