Abstract
To validate the application of machine learning (ML) to rock engineering practice, it is crucial that algorithm developers use appropriate methods to quantify how closely the ML reproduces the observed rock mass deformation. Input variable selection (IVS) is one approach that examines how ML uses the given data, or inputs, to forecast rock mass behavior. Three IVS methods were developed for two convolutional neural network (CNN) architectures that predict tunnel liner yield at the Cigar Lake Mine, which exhibits time-dependent squeezing deformation. One model architecture focused on accurately predicting the higher tunnel liner yield classes, while the second architecture prioritized prediction accuracy across all tunnel liner yield classes. The three IVS methods investigated herein were channel activation strength (CAS), input omission (IO), and partial correlation (PC). The IO and PC approaches proposed are novel approaches proposed for CNNs using a spatial and temporal geomechanical dataset. Performance of all models was compared using the corrected Akaike information criterion (AICc), where lower values indicate better performance. Each IVS method was used to produce a unique ranking for each model architecture and training/testing data split: CAS produced an activation ranking, IO produced an Omission Ranking, and PC produced a correlation ranking. The activation rankings showed that the geology input had the lowest activation strength in the CNN relative to the other inputs (ground freezing, primary installed support class, and radial tunnel displacement). Geology had the highest omission ranking, resulting from it having the most negative impact on performance as compared to the other inputs when it was omitted from the models entirely. The PC approach, using the Correlation Rankings, found that the highest model performances were reached when the most recent radial tunnel displacement was added into the pool of candidate inputs. The three IVS approaches and their respective rankings proved to be useful for analyzing the CNN inputs in terms of importance and confirming underlying assumption about the deformation mechanics at Cigar Lake Mine. Collectively, the IVS analyses indicated that all of the available digitized inputs for the Cigar Lake Mine CNNs are needed to produce good model performances. Each IVS method revealed different insights into this CNN development. Undertaking IVS for ML developed using geomechanical datasets allows for verification of the algorithms and thereby a better understanding of the nuance of the rock mass deformation. At Cigar Lake Mine, these findings may be used to assist in forecasting the schedule and budget for ground support rehabilitation.
Highlights
-
Convolutional neural networks presented to predict tunnel liner yield in squeezing ground at Cigar Lake Mine.
-
Three input variable selection (IVS) methods are proposed to examine machine learning (ML) input saliency.
-
Channel activation strength method is adapted for CNNs; novel input omission and partial correlation methods are developed and proposed.
-
IVS reveals input dominance, whether to remove redundant data, and temporal effects of available inputs.
-
The authors suggest that IVS become standard practice to validate ML for rock mechanics and rock engineering.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Akaike H (1969) Fitting autoreggressive models for prediction. Ann I Stat Math 21:243–247
Barla G (2002) Tunnelling under squeezing rock conditions. In: Kolymbas D (ed) Tunnelling mechanics—advances in geotechnical engineering and tunnelling, vol 5. Springer Science & Business Media, pp 169–268
Barla G, Borgna S (1999) Tunnelling in squeezing rock conditions. In: Proceedings of ROCKSITE-99, pp 97–108
Barla G, Bonini M, Semeraro M (2011) Analysis of the behaviour of a yield-control support system in squeezing rock. Tunn Undergr Space Technol 26(1):146–154. https://doi.org/10.1016/j.tust.2010.08.001
Bishop S, Mainville A, Yesnik L (2016) Cigar Lake Operation, National Instrument 43-101 Technical Report
Bizjak KF, Petkovšek B (2004) Displacement analysis of tunnel support in soft rock around a shallow highway tunnel at Golovec. Eng Geol 75(1):89–106. https://doi.org/10.1016/j.enggeo.2004.05.003
Burnham KP, Anderson DR (2002) Model selection and multi-model inference: a practical information-theoretic approach, 2nd edn. Springer, New York. https://doi.org/10.1007/b97636
Elmo D, Stead D (2021) The role of behavioural factors and cognitive biases in rock engineering. Rock Mech Rock Eng 54:2109–2128. https://doi.org/10.1007/s00603-021-02385-3
Elmo D, Stead D, Yang B, Tsai R, Fogel Y (2020) Can new technologies shake the empirical foundations of rock engineering ? In: Wesseloo J (ed) Proceedings of the second international conference on underground mining technology. Australian Centre for Geomechanics, pp 107–116. https://doi.org/10.36487/ACG_repo/2035_01
Fathipour-Azar H (2021) Data-driven estimation of joint roughness coefficient. J Rock Mech Geotech Eng 13(6):1428–1437. https://doi.org/10.1016/j.jrmge.2021.09.003
Golder Associates (2001) Numerical analysis to estimate stresses in crosscut linings at Cigar Lake Mine
He J, Valeo C, Chu A, Neumann N (2011) Prediction of event-based stormwater runoff quantity and quality by ANNs developed using PMI-based input selection. J Hydrol 400(1–2):10–23. https://doi.org/10.1016/j.jhydrol.2011.01.024
Hurvich C, Tsai CL (1989) Regression and time series model selection in small samples. Biometrika 76(2):297–307. https://doi.org/10.1093/biomet/76.2.297
Isleyen E, Duzgun S, McKell Carter R (2021) Interpretable deep learning for roof fall hazard detection in underground mines. J Rock Mech Geotech Eng 13(6):1246–1255. https://doi.org/10.1016/j.jrmge.2021.09.005
Khan UT, Valeo C (2017) Optimising fuzzy neural network architecture for dissolved oxygen prediction and risk analysis. Water (switzerland) 9(6):381. https://doi.org/10.3390/w9060381
Kudo M, Toyama J, Shimbo M (1999) Multidimensional curve classification using passing-through regions. Pattern Recognit Lett 20(11–13):1103–1111. https://doi.org/10.1016/S0167-8655(99)00077-X
Kumar M, Samui P, Naithani AK (2013) Determination of uniaxial compressive strength and modulus of elasticity of travertine using machine learning techniques. Int J Adv Soft Comput Appl 5(3):2074–8523
Lawal AI, Kwon S (2020) Application of artificial intelligence to rock mechanics: An overview. J Rock Mech Geotech 13(1):248–266. https://doi.org/10.1016/j.jrmge.2020.05.010
LeCun Y, Boser B, Denker JS, Henderson D, Howard RE, Hubbard W, Jackel LD (1989) Backpropagation applied to handwritten zip code recognition. Neural Comput 1(4):541–551. https://doi.org/10.1162/neco.1989.1.4.541
Leu SS, Chen CN, Chang SL (2001) Data mining for tunnel support stability: neural network approach. Automat Constr 10(4):429–441. https://doi.org/10.1016/S0926-5805(00)00078-9
Lipton ZC (2016) The mythos of model interpretability. In: 2016 ICML workshop on human interpretability in machine learning. http://arxiv.org/abs/1606.03490
Liu K, Liu B (2017) Optimization of smooth blasting parameters for mountain tunnel construction with specified control indices based on a GA and ISVR coupling algorithm. Tunn Underground Space Technol 70:363–374. https://doi.org/10.1016/j.tust.2017.09.007
MathWorks Inc (2019) MATLAB R2019b 9.7.0.1261785
May RJ, Dandy GC, Maier HR, Nixon JB (2008) Application of partial mutual information variable selection to ANN forecasting of water quality in water distribution systems. Environ Model Softw 23(10–11):1289–1299. https://doi.org/10.1016/j.envsoft.2008.03.008
May R, Dandy G, Maier H (2011) Review of input variable selection methods for artificial neural networks. In: Suzuki K (ed) Artificial neural networks—methodological advances and biomedical applications. Intech, pp 19–44. https://doi.org/10.5772/16004
Mcgaughey WJ (2019) Data-driven geotechnical hazard assessment: practice and pitfalls. In: Wesseloo J (ed) Proceedings of the first international conference on mining geomechanical risk. Australian Centre for Geomechanics, pp 219–232. https://doi.org/10.36487/ACG_rep/1905_11_McGaughey
Mcgaughey J (2020) Artificial intelligence and big data analytics in mining geomechanics. J South Afr Inst Min Metall. https://doi.org/10.17159/2411
Millar D, Clarici E (1994) Investigation of back-propagation artificial neural networks in modelling the stress-strain behaviour of sandstone rock. In: Proceedings of 1994 IEEE international conference on neural networks (ICNN’94), vol 5, pp 3326–3331. https://doi.org/10.1109/icnn.1994.374770
Mnih V, Hinton G (2012) Learning to label aerial images from noisy data. In: Proceedings of the 29th international conference on machine learning, ICML 2012, vol 1, pp 567–574
Molnar C (2022) Interpretable machine learning: a guide for making black boxes explainable, 2nd edn. http://christophm.github.io/interpretable-ml-book/
Morgenroth J (2021) Cigar Lake Mine convolutional neural network
Morgenroth J, Khan UT, Perras MA (2019) An overview of opportunities for machine learning methods in rock mechanics. Geosci J 9(12):504–524
Morgenroth J, Perras MA, Khan UT (2020) Convolutional neural networks for predicting tunnel support and liner performance: Cigar Lake Mine case study. In: Proceedings of the 54th US rock mechanics/geomechanics symposium
Morgenroth J, Perras MA, Khan UT (2021a) A convolutional neural network approach for predicting tunnel liner yield at Cigar Lake Mine. Rock Mech Rock Eng. https://doi.org/10.1007/s00603-021-02563-3
Morgenroth J, Perras MA, Khan UT (2021b) An input variable selection approach for a convolutional neural network that forecasts tunnel liner yield at the Cigar Lake Mine. In: Rocscience international conference 2021b—the evolution of geotech: 25 years of innovation
Paudel B, Jafarpour M, Brummer R (2012) Cigar Lake Mine MDS tunnel liner loading analysis at Cameco. ITASCA International Inc
Pu Y, Apel DB, Lingga B (2018) Rockburst prediction in kimberlite using decision tree with incomplete data. J Sustain Min 17(3):158–165. https://doi.org/10.1016/j.jsm.2018.07.004
Pu Y, Apel DB, Liu V, Mitri H (2019) Machine learning methods for rockburst prediction-state-of-the-art review. Int J Min Sci Technol 29(4):565–570. https://doi.org/10.1016/j.ijmst.2019.06.009
Qi C, Fourie A, Du X, Tang X (2018) Prediction of open stope hangingwall stability using random forests. Nat Hazards 92(2):1179–1197. https://doi.org/10.1007/s11069-018-3246-7
Reed SE, Lee H, Anguelov D, Szegedy C, Erhan D, Rabinovich A (2015) Training deep neural networks on noisy labels with bootstrap**. In: Bengio Y, LeCun Y (eds) 3rd international conference on learning representations, ICLR 2015—workshop track proceedings May 7–9, 2015. ICLR, pp 1–11
Ribeiro e Sousa L, Miranda T, Leal e Sousa R, Tinoco J (2017) The use of data mining techniques in rockburst risk assessment. Engineering 3(4):552–558. https://doi.org/10.1016/J.ENG.2017.04.002
Roworth M (2013) Understanding the effect of freezing on rock mass behaviour as applied to the Cigar Lake mining method (MASc Thesis). The University of British Columbia
Samanta A, Saha A, Satapathy SC, Fernandes SL, Zhang YD (2020) Automated detection of diabetic retinopathy using convolutional neural networks on a small dataset. Pattern Recognit Lett 135:293–298. https://doi.org/10.1016/j.patrec.2020.04.026
Setiono R, Liu H (1997) Neural-network feature selector. IEEE Trans Neural Netw 8(3):654–662. https://doi.org/10.1109/72.572104
Shlens J (2014) A tutorial on principal component analysis. http://arxiv.org/abs/1404.1100
Sklavounos P, Sakellariou M (1995) Intelligent classification of rock masses. Trans Info Commun Technol 8:387–393. https://doi.org/10.2495/AI950411
Snieder E, Shakir R, Khan UT (2019) A comprehensive comparison of four input variable selection methods for artificial neural network flow forecasting models. J Hydrol 583:124299. https://doi.org/10.1016/j.jhydrol.2019.124299
Song ZP, Jiang AN, Jiang ZB (2015) Back analysis of geomechanical parameters using hybrid algorithm based on difference evolution and extreme learning machine. Math Probl Eng 2015:1–11. https://doi.org/10.1155/2015/821534
Sun Y, Feng X, Yang L (2018) Predicting tunnel squeezing using multiclass support vector machines. Adv Civ Eng 2018:1–12. https://doi.org/10.1155/2018/4543984
WSP (formerly Parson Brinkerhoff Quade & Douglas Inc.) (1999) Cigar Lake mine global mine model study: final report
Zeiler MD, Fergus R (2014) Visualizing and understanding convolutional network. In: Fleet D, Pajdla T, Schiele B, Tuytelaars T (eds) Computer vision—ECCV 2014, vol 8689. Springer International Publishing Switzerland, pp 818–833
Zhao K, Bonini M, Debernardi D, Janutolo M, Barla G, Chen G (2015) Computational modelling of the mechanised excavation of deep tunnels in weak rock. Comput Geotech 66:158–171. https://doi.org/10.1016/j.compgeo.2015.01.020
Acknowledgements
The authors would like to extend special thanks to Cameco, and particularly Chris Twiggs, Imre Bartha, and Kirk Lamont for their constructive feedback and informative conversations. This work is funded in part by the Natural Sciences and Engineering Research Council of Canada through the Discovery Grant program and the joint Innovation York and National Research Council Canada’s Industry Research Assistance Program—Artificial Intelligence Industry Partnership Fund, in partnership with Yield Point Inc. This work is also funded by the NSERC Postgraduate Scholarships—Doctoral program.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Appendix
Appendix
See Tables 4, 5, 6 and Figs. 11, 12.
Results of partial correlation (PC) input variable selection (IVS) approach for Cigar Lake Mine targeted class 2/3 models. Each plot shows the boxplot of performance across an ensemble of 30 models of the convolutional neural network as each successive input is added, where the order is determined by the partial correlation of the candidate inputs and the target
Results of partial correlation (PC) input variable selection (IVS) approach for Cigar Lake Mine global balanced models. Each plot shows the boxplot of performance across an ensemble of 30 models of the convolutional neural network as each successive input is added, where the order is determined by the partial correlation of the candidate inputs and the target
Rights and permissions
About this article
Cite this article
Morgenroth, J., Perras, M.A. & Khan, U.T. On the Interpretability of Machine Learning Using Input Variable Selection: Forecasting Tunnel Liner Yield. Rock Mech Rock Eng 55, 6779–6804 (2022). https://doi.org/10.1007/s00603-022-02987-5
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00603-022-02987-5