SpringerLink
Log in

Quantum self-attention neural networks for text classification

  • Research Paper
  • Published:
Science China Information Sciences Aims and scope Submit manuscript

Abstract

An emerging direction of quantum computing is to establish meaningful quantum applications in various fields of artificial intelligence, including natural language processing (NLP). Although some efforts based on syntactic analysis have opened the door to research in quantum NLP (QNLP), limitations such as heavy syntactic preprocessing and syntax-dependent network architecture make them impracticable on larger and real-world data sets. In this paper, we propose a new simple network architecture, called the quantum self-attention neural network (QSANN), which can compensate for these limitations. Specifically, we introduce the self-attention mechanism into quantum neural networks and then utilize a Gaussian projected quantum self-attention serving as a sensible quantum version of self-attention. As a result, QSANN is effective and scalable on larger data sets and has the desirable property of being implementable on near-term quantum devices. In particular, our QSANN outperforms the best existing QNLP model based on syntactic analysis as well as a simple classical self-attention neural network in numerical experiments of text classification tasks on public data sets. We further show that our method exhibits robustness to low-level quantum noises and showcases resilience to quantum neural network architectures.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price includes VAT (Germany)

Instant access to the full article PDF.

Institutional subscriptions

References

  1. Preskill J. Quantum computing 40 years later. 2021. Ar**v:2106.10522

  2. Harrow A W, Montanaro A. Quantum computational supremacy. Nature, 2017, 549: 203–209

    Article  Google Scholar 

  3. Childs A M, van Dam W. Quantum algorithms for algebraic problems. Rev Mod Phys, 2010, 82: 1–52

    Article  MathSciNet  Google Scholar 

  4. Montanaro A. Quantum algorithms: an overview. npj Quantum Inf, 2016, 2: 15023

    Article  Google Scholar 

  5. Childs A M, Maslov D, Nam Y, et al. Toward the first quantum simulation with quantum speedup. Proc Natl Acad Sci USA, 2018, 115: 9456–9461

    Article  MathSciNet  Google Scholar 

  6. Biamonte J, Wittek P, Pancotti N, et al. Quantum machine learning. Nature, 2017, 549: 195–202

    Article  Google Scholar 

  7. Brandao F G S L, Svore K M. Quantum speed-ups for solving semidefinite programs. In: Proceedingsof the 58th Annual Symposium on Foundations of Computer Science (FOCS), 2017. 415–426

  8. Xu F, Ma X, Zhang Q, et al. Secure quantum key distribution with realistic devices. Rev Mod Phys, 2020, 92: 25002

    Article  MathSciNet  Google Scholar 

  9. McArdle S, Endo S, Aspuru-Guzik A, et al. Quantum computational chemistry. Rev Mod Phys, 2020, 92: 015003

    Article  MathSciNet  Google Scholar 

  10. Cao Y, Romero J, Olson J P, et al. Quantum chemistry in the age of quantum computing. Chem Rev, 2019, 119: 10856–10915

    Article  Google Scholar 

  11. Rebentrost P, Mohseni M, Lloyd S. Quantum support vector machine for big data classification. Phys Rev Lett, 2014, 113: 130503

    Article  Google Scholar 

  12. Huang H Y, Broughton M, Mohseni M, et al. Power of data in quantum machine learning. Nat Commun, 2021, 12: 2631

    Article  Google Scholar 

  13. Schuld M, Petruccione F. Machine Learning with Quantum Computers. Berlin: Springer, 2021

    Book  Google Scholar 

  14. Preskill J. Quantum computing in the NISQ era and beyond. 2018. Ar**v:1801.00862

  15. Arute F, Arya K, Babbush R, et al. Quantum supremacy using a programmable superconducting processor. Nature, 2019, 574: 505–510

    Article  Google Scholar 

  16. Zhong H S, Wang H, Deng Y H, et al. Quantum computational advantage using photons. Science, 2020, 370: 1460–1463

    Article  Google Scholar 

  17. Bharti K, Cervera-Lierta A, Kyaw T H, et al. Noisy intermediate-scale quantum (NISQ) algorithms. 2021. Ar**v:2101.08448

  18. Cerezo M, Arrasmith A, Babbush R, et al. Variational quantum algorithms. Nat Rev Phys, 2021, 3: 625–644

    Article  Google Scholar 

  19. Endo S, Cai Z, Benjamin S C, et al. Hybrid quantum-classical algorithms and quantum error mitigation. J Phys Soc Jpn, 2021, 90: 032001

    Article  Google Scholar 

  20. Peruzzo A, McClean J, Shadbolt P, et al. A variational eigenvalue solver on a photonic quantum processor. Nat Commun, 2014, 5: 4213

    Article  Google Scholar 

  21. Farhi E, Goldstone J, Gutmann S. A quantum approximate optimization algorithm. 2014. Ar**v:1411.4028

  22. Havlíček V, Córcoles A D, Temme K, et al. Supervised learning with quantum-enhanced feature spaces. Nature, 2019, 567: 209–212

    Article  Google Scholar 

  23. Schuld M, Bocharov A, Svore K M, et al. Circuit-centric quantum classifiers. Phys Rev A, 2020, 101: 032308

    Article  MathSciNet  Google Scholar 

  24. Mitarai K, Negoro M, Kitagawa M, et al. Quantum circuit learning. Phys Rev A, 2018, 98: 032309

    Article  Google Scholar 

  25. Yang C H H, Qi J, Chen S Y C, et al. When bert meets quantum temporal convolution learning for text classification in heterogeneous computing. In: Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022. 8602–8606

  26. Qi J, Tejedor J. Classical-to-quantum transfer learning for spoken command recognition based on quantum neural networks. In: Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022. 8627–8631

  27. Yang C H H, Li B, Zhang Y, et al. A quantum kernel learning approach to acoustic modeling for spoken command recognition. In: Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2023. 1–5

  28. Benedetti M, Lloyd E, Sack S, et al. Parameterized quantum circuits as machine learning models. Quantum Sci Technol, 2019, 4: 043001

    Article  Google Scholar 

  29. Farhi E, Neven H. Classification with quantum neural networks on near term processors. 2018. Ar**v:1802.06002

  30. Yu Z, Yao H S, Li M J, et al. Power and limitations of single-qubit native quantum neural networks. In: Proceedings of Advances in Neural Information Processing Systems, 2022. 27810–27823

  31. Caro M C, Huang H Y, Cerezo M, et al. Generalization in quantum machine learning from few training data. Nat Commun, 2022, 13: 4919

    Article  Google Scholar 

  32. Li G X, Ye R L, Zhao X Q, et al. Concentration of data encoding in parameterized quantum circuits. In: Proceedings of the 36th Conference on Neural Information Processing Systems (NeurIPS 2022), 2022

  33. Du Y, Tu Z, Yuan X, et al. Efficient measure for the expressivity of variational quantum algorithms. Phys Rev Lett, 2022, 128: 80506

    Article  MathSciNet  Google Scholar 

  34. Jerbi S, Fiderer L J, Nautrup H P, et al. Quantum machine learning beyond kernel methods. Nat Commun, 2023, 14: 517

    Article  Google Scholar 

  35. Yu Z, Zhao X, Zhao B, et al. Optimal quantum dataset for learning a unitary transformation. Phys Rev Appl, 2023, 19: 034017

    Article  Google Scholar 

  36. Wang K, Song Z, Zhao X, et al. Detecting and quantifying entanglement on near-term quantum devices. npj Quantum Inf, 2022, 8: 52

    Article  Google Scholar 

  37. Zhao X, Zhao B, Wang Z, et al. Practical distributed quantum information processing with LOCCNet. npj Quantum Inf, 2021, 7: 159

    Article  Google Scholar 

  38. Tian J K, Sun X Y, Du Y X, et al. Recent advances for quantum neural networks in generative learning. 2022. Ar**v:2206.03066

  39. Wang Y, Li G, Wang X. A hybrid quantum-classical Hamiltonian learning algorithm. Sci China Inf Sci, 2023, 66: 129502

    Article  Google Scholar 

  40. Abbas A, Sutter D, Zoufal C, et al. The power of quantum neural networks. Nat Comput Sci, 2021, 1: 403–409

    Article  Google Scholar 

  41. Sordoni A, Nie J T, Bengio Y. Modeling term dependencies with quantum language models for IR. In: Proceedings of the 36th International ACM SIGIR Conference on Research and Development in Information Retrieval, New York, 2013

  42. Zhang P, Niu J B, Su Z, et al. End-to-end quantum-like language models with application to question answering. In: Proceedings of the AAAI Conference on Artificial Intelligence, 2018

  43. Zhang Y, Song D, Zhang P, et al. A quantum-inspired sentiment representation model for twitter sentiment analysis. Appl Intell, 2019, 49: 3093–3108

    Article  Google Scholar 

  44. Basile I, Tamburini F. Towards quantum language models. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2017. 1840–1849

  45. Zeng W, Coecke B. Quantum algorithms for compositional natural language processing. 2016. Ar**v:1608.01406

  46. Meichanetzidis K, Gogioso S, de Felice G, et al. Quantum natural language processing on near-term quantum computers. 2020. Ar**v:2005.04147

  47. Wiebe N, Bocharov A, Smolensky P, et al. Quantum language processing. 2019. Ar**v:1902.05162

  48. Chen S Y C, Yoo S, Fang Y L L. Quantum long short-term memory. 2020. Ar**v:2009.01783

  49. Devlin J, Chang M W, Lee K, et al. Bert: pre-training of deep bidirectional transformers for language understanding. 2018. Ar**v:1810.04805

  50. Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need. In: Proceedings of the 31st International Conference on Neural Information Processing Systems, 2017. 6000–6010

  51. Li X P, Song J K, Gao L L, et al. Beyond RNNs: positional self-attention with co-attention for video question answering. In: Proceedings of the AAAI Conference on Artificial Intelligence, 2019. 8658–8665

  52. Guo Q P, Qiu X P, Liu P F, et al. Multi-scale self-attention for text classification. In: Proceedings of the AAAI Conference on Artificial Intelligence, 2020. 7847–7854

  53. Cha P, Ginsparg P, Wu F, et al. Attention-based quantum tomography. 2020. Ar**v:2006.12469

  54. Lorenz R, Pearson A, Meichanetzidis K, et al. QNLP in practice: running compositional models of meaning on a quantum computer. 2021. Ar**v:2102.12846

  55. Nielsen M A, Chuang I. Quantum Computation and Quantum Information. Cambridge: Cambridge University Press, 2011

    Google Scholar 

  56. Micchelli C A, Xu Y S, Zhang H Z. Universal kernels. J Mach Learn Res, 2006, 7: 2651–2667

    MathSciNet  Google Scholar 

  57. Di Sipio R, Huang J H, Chen S Y C, et al. The dawn of quantum natural language processing. In: Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022. 8612–8616

  58. Ziegel E R, Lehmann E L, Casella G. Theory of point estimation. Technometrics, 1999, 41: 274

    Google Scholar 

  59. Bottou L. Stochastic learning. In: Proceedings of Advanced Lectures on Machine Learning, 2004. 146–168

  60. Goodfellow I, Bengio Y, Courville A. Deep Learning. Cambridge: MIT Press, 2016

    Google Scholar 

  61. Liu Y, Arunachalam S, Temme K. A rigorous and robust quantum speed-up in supervised machine learning. Nat Phys, 2021, 17: 1013–1017

    Article  Google Scholar 

  62. Qi J, Yang C H H, Chen P Y, et al. Theoretical error performance analysis for variational quantum circuit based functional regression. npj Quantum Inf, 2023, 9: 4

    Article  Google Scholar 

  63. Kotzias D, Denil N, de Freitas N, et al. From group to individual labels using deep features. In: Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2015. 597–606

  64. Ma Y J, Yu D H, Wu T, et al. PaddlePaddle: an open-source deep learning platform from industrial practice. Front Data Domput, 2019, 1: 105–115

    Google Scholar 

  65. Dua D, Graff C. UCI machine learning repository. 2017. http://archive.ics.uci.edu/ml

  66. Kingma D P, Ba J L. Adam: a method for stochastic optimization. In: Proceedings of the 3rd International Conference on Learning Representations, 2015

Download references

Acknowledgements

This work was partially supported by Guangdong Provincial Quantum Science Strategic Initiative (Grant No. GDZX2303007). Guangxi LI acknowledges the support from Quantum Science Center of Guangdong-Hong Kong-Macao Greater Bay Area, Baidu-UTS AI Meets Quantum project, the China Scholarship Council (Grant No. 201806070139), and Australian Research Council Project (Grant No. DP180100691). **n WANG was partially supported by Start-up Fund (Grant No. G0101000151) from The Hong Kong University of Science and Technology (Guangzhou), Innovation Program for Quantum Science and Technology (Grant No. 2021ZD0302901), and Education Bureau of Guangzhou Municipality. We would like to thank Prof. Sanjiang LI and Prof. Yuan FENG for their helpful discussions. We also thank Zihe WANG and Chenghong ZHU for their help related to the experiments. Part of this work was done when all of the authors were at Baidu Research.

Author information

Authors and Affiliations

  1. Institute for Quantum Computing, Baidu Research, Bei**g, 100193, China

    Guangxi Li, Xuanqiang Zhao & **n Wang

  2. Centre for Quantum Software and Information, University of Technology Sydney, Sydney, NSW, 2007, Australia

    Guangxi Li

  3. QICI Quantum Information and Computation Initiative, Department of Computer Science, The University of Hong Kong, Hong Kong, 999077, China

    Xuanqiang Zhao

  4. Thrust of Artificial Intelligence, Information Hub, Hong Kong University of Science and Technology (Guangzhou), Guangzhou, 511453, China

    **n Wang

Authors
  1. Guangxi Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xuanqiang Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. **n Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to **n Wang.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, G., Zhao, X. & Wang, X. Quantum self-attention neural networks for text classification. Sci. China Inf. Sci. 67, 142501 (2024). https://doi.org/10.1007/s11432-023-3879-7

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11432-023-3879-7

Keywords

Search

Navigation