SpringerLink
Log in

MSTS-Net: malignancy evolution prediction of pulmonary nodules from longitudinal CT images via multi-task spatial-temporal self-attention network

  • Original Article
  • Published:
International Journal of Computer Assisted Radiology and Surgery Aims and scope Submit manuscript

Abstract

Purpose

Longitudinal CT images contain the law of lesion growth and evolution over time. Therefore, our purpose is to explore the growth and evolution law of pulmonary lesions in the time dimension to improve the performance of predicting the malignant evolution of pulmonary nodules.

Methods

In this paper, we propose a Multi-task Spatial-Temporal Self-attention network (MSTS-Net) to predict the malignancy growth trend of pulmonary nodules from different periods. More specifically, the model achieves lesion segmentation task and lesion prediction task by sharing the same encoder. Segmentation task boosts the performance of the prediction task. In addition, a Static Context Spatial Self-attention Module and a Dynamic Adaptive Temporal Self-Attention Module are introduced to capture both static spatial coherence patterns between consecutive slices of lesions in the same period and temporal dynamics across different time points.

Results

We repeatedly evaluated the proposed method on the National Lung Screening Trial dataset and the Shanxi Cancer Hospital dataset. The final experimental results show that our MSTS-Net has an area under the ROC curve score of 0.919.

Conclusion

In the computer-aided prediction of the malignant evolution of pulmonary nodules, combining the characteristics of the temporal dimension of pulmonary nodules with CT data can effectively improve the accuracy of prediction. The MSTS-Net we developed has high predictive value and broad prospects for clinical application.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Ferlay J, Colombet M, Soerjomataram I, Parkin DM, Piñeros M, Znaor A, Bray F (2021) Cancer statistics for the year 2020: an overview. Int J Cancer 149(4):778–789

    Article  CAS  Google Scholar 

  2. Chen K, Liu L, Nie B, Binchun L, Lidan F, He Z, Li W, Pi X, Liu H (2021) Recognizing lung cancer and stages using a self-developed electronic nose system. Comput Biol Med 131:104294

    Article  CAS  PubMed  Google Scholar 

  3. Cai Z, Liu Q (2021) Understanding the global cancer statistics 2018: implications for cancer control. Sci Chin Life Sci 64(6):1017–1020

    Article  Google Scholar 

  4. Tang Z, Zhenhua X, Zhu X, Zhang J (2021) New insights into molecules and pathways of cancer metabolism and therapeutic implications. Cancer Commun 41(1):16–36

    Article  Google Scholar 

  5. Han J, **ao N, Yang W, Luo S, Zhao J, Qiang Y, Chaudhary S, Zhao J (2022) MS-ResNet: disease-specific survival prediction using longitudinal CT images and clinical data. Int J Comput Assist Radiol Surg. pp 1–9

  6. Doppalapudi S, Qiu RG, Badr Y (2021) Lung cancer survival period prediction and understanding: deep learning approaches. Int J Med Inform 148:104371

    Article  PubMed  Google Scholar 

  7. Mastouri R, Khlifa N, Neji H, Hantous-Zannad S (2021) A bilinear convolutional neural network for lung nodules classification on ct images. Int J Comput Assist Radiol Surg 16(1):91–101

    Article  PubMed  Google Scholar 

  8. Toda R, Teramoto A, Tsujimoto M, Toyama H, Imaizumi K, Saito K, Fujita H (2021) Synthetic ct image generation of shape-controlled lung cancer using semi-conditional infogan and its applicability for type classification. Int J Comput Assist Radiol Surg 16(2):241–251

    Article  PubMed  Google Scholar 

  9. Mi H, Petitjean C, Vera P, Ruan S (2015) Joint tumor growth prediction and tumor segmentation on therapeutic follow-up pet images. Med Image Anal 23(1):84–91

    Article  PubMed  Google Scholar 

  10. Zhao Z, Zhao J, Song K, Hussain A, Qianqian D, Dong Y, Liu J, Yang X (2020) Joint dbn and fuzzy c-means unsupervised deep clustering for lung cancer patient stratification. Eng Appl Artif Intell 91:103571

    Article  Google Scholar 

  11. Ghani T, Oommen BJ (2020) Enhancing the prediction of lung cancer survival rates using 2d features from 3d scans. In: International Conference on Image Analysis and Recognition, pp 202–215. Springer

  12. Gong J, Liu J, Hao W, Nie S, Zheng B, Wang S, Peng W (2020) A deep residual learning network for predicting lung adenocarcinoma manifesting as ground-glass nodule on ct images. Eur Radiol 30(4):1847–1855

    Article  PubMed  Google Scholar 

  13. Causey Jason L, Zhang J, Ma S, Jiang Jason L, Qualls Jake A, Politte David G, Prior F, Zhang S, Huang X (2018) Highly accurate model for prediction of lung nodule malignancy with ct scans. Sci Rep 8(1):1–12

    Google Scholar 

  14. Liao F, Liang M, Li Z, **aolin H, Song S (2019) Evaluate the malignancy of pulmonary nodules using the 3-d deep leaky noisy-or network. IEEE Trans Neural Netw Learn Syst 30(11):3484–3495

    Article  PubMed  Google Scholar 

  15. Yao J, Shi Y, Lu L, **ao J, Zhang L (2020) Deepprognosis: preoperative prediction of pancreatic cancer survival and surgical margin via contrast-enhanced ct imaging. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp 272–282. Springer

  16. Elazab A, Wang C, Gardezi SJS, Bai H, Qingmao H, Wang T, Chang C, Lei B (2020) Gp-gan: brain tumor growth prediction using stacked 3d generative adversarial networks from longitudinal mr images. Neural Netw 132:321–332

    Article  PubMed  Google Scholar 

  17. **a Y, Yu Q, Shen W, Zhou Y, Fishman EK, Yuille AL (2020) Detecting pancreatic ductal adenocarcinoma in multi-phase ct scans via alignment ensemble. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp 285–295. Springer, 2020

  18. Liu X, Wang M, Rukhma Aftab R (2022) Study on the prediction method of long-term benign and malignant pulmonary lesions based on lstm. Front Bioeng Biotechnol. p 30

  19. Veasey Benjamin P, Justin B, Michael D, Albert S, Amini Amir A (2020) Lung nodule malignancy prediction from longitudinal ct scans with siamese convolutional attention networks. IEEE Open J Eng Med Biol 1:257–264

    Article  CAS  PubMed  Google Scholar 

  20. Mehdi Farhangi M, Petrick N, Sahiner B, Frigui H, Amini Amir A, Aria P (2020) Recurrent attention network for false positive reduction in the detection of pulmonary nodules in thoracic ct scans. Med Phys 47(5):2150–2160

    Article  PubMed  Google Scholar 

  21. Bietti A, Mairal J (2019) On the inductive bias of neural tangent kernels. Adv Neural Inf Process Syst. p 32

  22. Liu Z, Lin Y, Cao Y, Hu H, Wei Y, Zhang Z, Lin S, Guo B (2021) Swin transformer: hierarchical vision transformer using shifted windows. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp 10012–10022

  23. Ronneberger O, Fischer P, Brox T (2015) U-net: convolutional networks for biomedical image segmentation. In: International Conference on Medical image computing and computer-assisted intervention. pp 234–241. Springer

  24. Xue Y, Tao X, Zhang H, Rodney Long L, **aolei H (2018) Segan: adversarial network with multi-scale l1 loss for medical image segmentation. Neuroinformatics 16(3):383–392

  25. Winter A, Aberle Denise R, Hsu W (2019) External validation and recalibration of the brock model to predict probability of cancer in pulmonary nodules using nlst data. Thorax 74(6):551–563

    Article  PubMed  Google Scholar 

  26. Gao R, Huo Y, Bao S, Tang Y, Antic SL, Epstein ES, Balar AB, Deppen S, Paulson AB, Sandler KL, Massion PP (2019) Distanced lstm: time-distanced gates in long short-term memory models for lung cancer detection. In: International Workshop on Machine Learning in Medical Imaging. pp 310–318. Springer

  27. Ghazipour A, Veasey B, Seow A, Amini AA (2021) Joint learning for deformable registration and malignancy classification of lung nodules. In: 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI). pp 1807–1811. IEEE

  28. Perez G, Arbelaez P (2020) Automated lung cancer diagnosis using three-dimensional convolutional neural networks. Med Biol Eng Comput 58(8):1803–1815

    Article  PubMed  Google Scholar 

  29. Gao R, Tang Y, Xu K, Huo Y, Bao S, Antic SL, Epstein ES, Deppen S, Paulson AB, Sandler KL, Massion PP (2020) Time-distanced gates in long short-term memory networks. Med Image Anal 65:101785

    Article  PubMed  PubMed Central  Google Scholar 

  30. Ardila D, Kiraly AP, Bharadwaj S, Choi B, Reicher JJ, Peng L, Tse D, Etemadi M, Ye W, Corrado G, Naidich DP, Shravya S (2019) End-to-end lung cancer screening with three-dimensional deep learning on low-dose chest computed tomography. Nat Med 25(6):954–961

    Article  CAS  PubMed  Google Scholar 

Download references

Acknowledgements

This work is supported by the National Natural Science Foundation of China [grant numbers 61872261]; the National Natural Science Foundation of China [grant numbers U21A20469]; the Shanxi Provincial Basic Research Program [grant numbers 202103021224066].

Author information

Authors and Affiliations

  1. College of Information and Computer, Taiyuan University of Technology, Taiyuan, China

    ** Song, Jiaxin Hou, Ning **ao, Jun Zhao, Juanjuan Zhao & Yan Qiang

  2. College of Information, **zhong College of Information, **zhong, China

    Juanjuan Zhao & Qianqian Yang

Authors
  1. ** Song
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jiaxin Hou
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ning **ao
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jun Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Juanjuan Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Yan Qiang
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Qianqian Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Juanjuan Zhao.

Ethics declarations

Conflicts of interest

The authors declare that they have no conflict of interest.

Human and animal rights

This article does not contain any studies with human participants or animals performed by any of the authors.

Informed consent

Informed consent was obtained from all individual participants included in the study.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Song, P., Hou, J., **ao, N. et al. MSTS-Net: malignancy evolution prediction of pulmonary nodules from longitudinal CT images via multi-task spatial-temporal self-attention network. Int J CARS 18, 685–693 (2023). https://doi.org/10.1007/s11548-022-02744-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11548-022-02744-7

Keywords

Search

Navigation