Abstract
Background
Despite endovascular coiling as a valid modality in treatment of aneurysmal subarachnoid hemorrhage (aSAH), there is a risk of poor prognosis. However, the clinical utility of previously proposed early prediction tools remains limited. We aimed to develop a clinically generalizable machine learning (ML) models for accurately predicting unfavorable outcomes in aSAH patients after endovascular coiling.
Methods
Functional outcomes at 6 months after endovascular coiling were assessed via the modified Rankin Scale (mRS) and unfavorable outcomes were defined as mRS 3-6. Five ML algorithms (logistic regression, random forest, support vector machine, deep neural network, and extreme gradient boosting) were used for model development. The area under precision-recall curve (AUPRC) and receiver operating characteristic curve (AUROC) was used as main indices of model evaluation. SHapley Additive exPlanations (SHAP) method was applied to interpret the best-performing ML model.
Results
A total of 371 patients were eventually included into this study, and 85.4% of them had favorable outcomes. Among the five models, the DNN model had a better performance with AUPRC of 0.645 (AUROC of 0.905). Postoperative GCS score, size of aneurysm, and age were the top three powerful predictors. The further analysis of five random cases presented the good interpretability of the DNN model.
Conclusion
Interpretable clinical prediction models based on different ML algorithms have been successfully constructed and validated, which would serve as reliable tools in optimizing the treatment decision-making of aSAH. Our DNN model had better performance to predict the unfavorable outcomes at 6 months in aSAH patients compared with Yan’s nomogram model.
Similar content being viewed by others
Data availability
The datasets generated and analyzed during the current study are not publicly available due protection of patients’ personal information but are available from the corresponding author on reasonable request.
Abbreviations
- aSAH :
-
aneurysmal subarachnoid hemorrhage
- AUPRC :
-
area under precision-recall curve
- AUROC :
-
area under receiver operating characteristic curve
- DNN :
-
deep neural network
- FGB :
-
fasting blood glucose
- GCS :
-
Glasgow Coma Scale
- HbA1c :
-
glycated hemoglobin
- LASSO :
-
least absolute shrinkage and selection operator
- LR :
-
logistic regression
- ML :
-
machine learning
- NPV :
-
negative predictive value
- mRS :
-
modified Rankin Scale
- PPV :
-
positive predictive value
- RF :
-
random forest
- SBP :
-
systolic blood pressure
- SHAP :
-
SHapley Additive exPlanations
- SVM :
-
support vector machine
- XGBoost :
-
extreme gradient boost
References
Etminan N, Chang H-S, Hackenberg K et al (2019) Worldwide incidence of aneurysmal subarachnoid hemorrhage according to region, time period, blood pressure, and smoking prevalence in the population: a systematic review and meta-analysis. JAMA Neurol 76:588–597. https://doi.org/10.1001/jamaneurol.2019.0006
Gawlitza M, Soize S, Barbe C et al (2019) Aneurysm characteristics, study population, and endovascular techniques for the treatment of intracranial aneurysms in a large, prospective, multicenter cohort: results of the analysis of recanalization after endovascular treatment of intracranial aneurysm study. AJNR Am J Neuroradiol 40:517–523. https://doi.org/10.3174/ajnr.A5991
Pierot L, Wakhloo AK (2013) Endovascular treatment of intracranial aneurysms: current status. Stroke 44:2046–2054. https://doi.org/10.1161/STROKEAHA.113.000733
Campos JK, Lien BV, Wang AS, Lin L-M (2020) Advances in endovascular aneurysm management: coiling and adjunctive devices. Stroke Vasc Neurol 5:14–21. https://doi.org/10.1136/svn-2019-000303
Molyneux A, Kerr R, Stratton I et al (2002) International Subarachnoid Aneurysm Trial (ISAT) of neurosurgical clip** versus endovascular coiling in 2143 patients with ruptured intracranial aneurysms: a randomised trial. Lancet 360:1267–1274. https://doi.org/10.1016/s0140-6736(02)11314-6
Jaja B, Saposnik G, Lingsma HF et al (2018) Development and validation of outcome prediction models for aneurysmal subarachnoid haemorrhage: the SAHIT multinational cohort study. BMJ 360:j5745
Naval NS, Kowalski RG, Chang TR et al (2014) The SAH score: a comprehensive communication tool. J Stroke Cerebrovasc Dis 23:902–909. https://doi.org/10.1016/j.jstrokecerebrovasdis.2013.07.035
van Donkelaar CE, Bakker NA, Birks J et al (2019) Prediction of outcome after aneurysmal subarachnoid hemorrhage. Stroke 50:837–844. https://doi.org/10.1161/STROKEAHA.118.023902
Yan Y, Hu J, Fang X et al (2021) Predicting the poor recovery risk of aneurysmal subarachnoid hemorrhage: clinical evaluation and management based on a new predictive nomogram. Clin Neurol Neurosurg 200:106302. https://doi.org/10.1016/j.clineuro.2020.106302
Connolly ES, Rabinstein AA, Carhuapoma JR et al (2012) Guidelines for the management of aneurysmal subarachnoid hemorrhage: a guideline for healthcare professionals from the American Heart Association/American Stroke Association. Stroke 43:1711–1737. https://doi.org/10.1161/STR.0b013e3182587839
Stienen MN, Germans M, Burkhardt J-K et al (2018) Predictors of in-hospital death after aneurysmal subarachnoid hemorrhage: analysis of a nationwide database (Swiss SOS [Swiss Study on Aneurysmal Subarachnoid Hemorrhage]). Stroke 49:333–340. https://doi.org/10.1161/STROKEAHA.117.019328
van Niftrik CHB, van der Wouden F, Staartjes VE et al (2019) Machine learning algorithm identifies patients at high risk for early complications after intracranial tumor surgery: registry-based cohort study. Neurosurgery 85:E756–E764. https://doi.org/10.1093/neuros/nyz145
Senders JT, Arnaout O, Karhade AV et al (2018) Natural and artificial intelligence in neurosurgery: a systematic review. Neurosurgery 83:181–192. https://doi.org/10.1093/neuros/nyx384
Park A, Chute C, Rajpurkar P et al (2019) Deep learning-assisted diagnosis of cerebral aneurysms using the HeadXNet model. JAMA Netw Open 2:e195600. https://doi.org/10.1001/jamanetworkopen.2019.5600
Tanioka S, Ishida F, Nakano F et al (2019) Machine learning analysis of matricellular proteins and clinical variables for early prediction of delayed cerebral ischemia after aneurysmal subarachnoid hemorrhage. Mol Neurobiol 56:7128–7135. https://doi.org/10.1007/s12035-019-1601-7
Diringer MN, Bleck TP, Claude Hemphill J et al (2011) Critical care management of patients following aneurysmal subarachnoid hemorrhage: recommendations from the Neurocritical Care Society’s Multidisciplinary Consensus Conference. Neurocrit Care 15:211–240. https://doi.org/10.1007/s12028-011-9605-9
Horan TC, Andrus M, Dudeck MA (2008) CDC/NHSN surveillance definition of health care-associated infection and criteria for specific types of infections in the acute care setting. Am J Infect Control 36:309–332. https://doi.org/10.1016/j.ajic.2008.03.002
Teasdale G, Jennett B (1974) Assessment of coma and impaired consciousness. A practical scale. Lancet 2:81–84. https://doi.org/10.1016/s0140-6736(74)91639-0
Brennan PM, Murray GD, Teasdale GM (2020) A practical method for dealing with missing Glasgow Coma Scale verbal component scores. J Neurosurg:1–6. https://doi.org/10.3171/2020.6.JNS20992
Li X, Pan X, Jiang C et al (2020) Predicting 6-month unfavorable outcome of acute ischemic stroke using machine learning. Front Neurol 11:539509. https://doi.org/10.3389/fneur.2020.539509
Vasquez MM, Hu C, Roe DJ et al (2016) Least absolute shrinkage and selection operator type methods for the identification of serum biomarkers of overweight and obesity: simulation and application. BMC Med Res Methodol 16:154. https://doi.org/10.1186/s12874-016-0254-8
Shalabi LA, Shaaban Z, Kasasbeh B (2006) Data mining: a preprocessing engine. J Comput Sci 2:735–739. https://doi.org/10.3844/jcssp.2006.735.739
Okada S, Ohzeki M, Taguchi S (2019) Efficient partition of integer optimization problems with one-hot encoding. Sci Rep 9:13036. https://doi.org/10.1038/s41598-019-49539-6
Davis J, Goadrich M (2006) The relationship between Precision-Recall and ROC curves. In: Proceedings of the 23rd international conference on Machine learning - ICML ’06. ACM Press, Pittsburgh, Pennsylvania, pp 233–240
Saito T, Rehmsmeier M (2015) The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets. PLoS One 10:e0118432. https://doi.org/10.1371/journal.pone.0118432
DeLong ER, DeLong DM, Clarke-Pearson DL (1988) Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics 44:837–845
Lundberg SM, Lee S–I (2017) A unified approach to interpreting model predictions. In: Proceedings of the 31st International Conference on Neural Information Processing Systems. Long Beach, CA, USA
Challen R, Denny J, Pitt M et al (2019) Artificial intelligence, bias and clinical safety. BMJ Qual Saf 28:231–237. https://doi.org/10.1136/bmjqs-2018-008370
Ribeiro MT, Singh S, Guestrin C (2016) “Why should i trust you?”: explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, New York, USA, pp 1135–1144
Shrikumar A, Greenside P, Shcherbina A, Kundaje A (2016) Not just a black box: learning important features through propagating activation differences. https://doi.org/10.3171/2014.11.JNS141607
Wartenberg KE, Schmidt JM, Claassen J et al (2006) Impact of medical complications on outcome after subarachnoid hemorrhage. Crit Care Med 34:617–623; quiz 624. https://doi.org/10.1097/01.ccm.0000201903.46435.35
Jabbarli R, Reinhard M, Roelz R et al (2016) The predictors and clinical impact of intraventricular hemorrhage in patients with aneurysmal subarachnoid hemorrhage. Int J Stroke 11:68–76. https://doi.org/10.1177/1747493015607518
Zheng K, Zhong M, Zhao B et al (2019) Poor-grade aneurysmal subarachnoid hemorrhage: risk factors affecting clinical outcomes in intracranial aneurysm patients in a multi-center study. Front Neurol 10:123. https://doi.org/10.3389/fneur.2019.00123
Ryttlefors M, Enblad P, Kerr RSC, Molyneux AJ (2008) International subarachnoid aneurysm trial of neurosurgical clip** versus endovascular coiling: subgroup analysis of 278 elderly patients. Stroke 39:2720–2726. https://doi.org/10.1161/STROKEAHA.107.506030
Bjerkne Wenneberg S, Odenstedt Hergès H, Svedin P et al (2021) Association between inflammatory response and outcome after subarachnoid haemorrhage. Acta Neurol Scand 143:195–205. https://doi.org/10.1111/ane.13353
Vergouwen MDI, Ilodigwe D, Macdonald RL (2011) Cerebral infarction after subarachnoid hemorrhage contributes to poor outcome by vasospasm-dependent and -independent effects. Stroke 42:924–929. https://doi.org/10.1161/STROKEAHA.110.597914
Galea JP, Dulhanty L, Patel HC, UK and Ireland Subarachnoid Hemorrhage Database Collaborators (2017) Predictors of outcome in aneurysmal subarachnoid hemorrhage patients: observations from a multicenter data set. Stroke 48:2958–2963. https://doi.org/10.1161/STROKEAHA.117.017777
Abla AA, Wilson DA, Williamson RW et al (2014) The relationship between ruptured aneurysm location, subarachnoid hemorrhage clot thickness, and incidence of radiographic or symptomatic vasospasm in patients enrolled in a prospective randomized controlled trial. J Neurosurg 120:391–397. https://doi.org/10.3171/2013.10.JNS13419
Pierot L, Cognard C, Anxionnat R et al (2010) Ruptured intracranial aneurysms: factors affecting the rate and outcome of endovascular treatment complications in a series of 782 patients (CLARITY study). Radiology 256:916–923. https://doi.org/10.1148/radiol.10092209
Spetzler RF, McDougall CG, Zabramski JM et al (2015) The barrow ruptured aneurysm trial: 6-year results. J Neurosurg 123:609–617. https://doi.org/10.3171/2014.9.JNS141749
Fisher CM, Kistler JP, Davis JM (1980) Relation of cerebral vasospasm to subarachnoid hemorrhage visualized by computerized tomographic scanning. Neurosurgery 6:1–9. https://doi.org/10.1227/00006123-198001000-00001
Funding
This study was supported by the National Natural Science Foundation of China (82173899), Jiangsu Pharmaceutical Association (JY202207, Q202202, A2021024, H202108), Hunan Natural Science Foundation (2021JJ70021), and Hunan innovation guidance grant of clinical medical technology (2020SK50920).
Author information
Authors and Affiliations
Contributions
ZZ and DAR conceived and designed the study. YYQ and JYZ provided a review of the previous literature. XWX, LX, and LHG were responsible for data collection. JYZ, YYQ, and DZZ carried out the statistical analyses and interpreted the results. ZZ and DAR drafted the manuscript. ZZ, DAR, YYQ, JYZ, JFP, ZZH, and JJZ participated in the discussion of the study. LX, DZZ, and XWX polished this manuscript, including grammatical checks. All authors contributed to manuscript revision, read, and approved the submitted version.
Corresponding authors
Ethics declarations
Ethical approval and Informed consent
The studies involving human participants were reviewed and approved by the Human Research Ethics Committee of Hunan Provincial People’s Hospital (article number: [2015]-10). The Ethics Committee waived the requirement of written informed consent for participation.
Conflict of interest
The authors declare no competing interests.
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary information
ESM 1
(DOCX 182 kb)
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Zhou, ., Dai, A., Yan, Y. et al. Accurately predicting the risk of unfavorable outcomes after endovascular coil therapy in patients with aneurysmal subarachnoid hemorrhage: an interpretable machine learning model. Neurol Sci 45, 679–691 (2024). https://doi.org/10.1007/s10072-023-07003-4
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10072-023-07003-4