Abstract
The random forest (RF) algorithm is known for its predictive performance but has been criticized for its lack of interpretability due to its complex ensemble nature. To address the issue of explainability our study questions the traditional approach of using most representative trees (MRTs) to simplify RF interpretation, highlighting the potential for misinterpretation due to non-informative early splits. To overcome these limitations, we propose a new method involving the construction of artificial representative trees (ARTs) through a greedy algorithm that iteratively builds a tree to minimize the distance to the RF ensemble, thereby preserving the predictive performance of the RF. We give a detailed description of the methodological framework for ART construction, including strategies for reducing computational complexity through variable preselection and quantile-based splitting. Results from extensive simulations demonstrate that ARTs provide a more accurate reflection of the RF's predictive performance and substantially reduce the false discovery rate, thus offering a more reliable interpretative model. The findings suggest that ARTs represent an advance in addressing the interpretation of RF models.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Data Availability
All methods developed in this manuscript are publicly available within the R package timbR on GitHub (https://github.com/imbs-hl/timbR). Additionally, precalculated results of the simulations can be accessed in a shiny app (https://bioweb2.imbs.uksh.de/rsconnect/ART_paper/) and code for reproducing the simulation study and regenerating figures is publicly available on GitHub (https://github.com/imbs-hl/ART_paper).
References
Breiman, L.: Random Forests. Mach. Learn. 45, 5–32 (2001). https://doi.org/10.1023/A:1010933404324
Strobl, C., Boulesteix, A.-L., Zeileis, A., Hothorn, T.: Bias in random forest variable importance measures: Illustrations, sources and a solution. BMC Bioinform. 8, 25 (2007). https://doi.org/10.1186/1471-2105-8-25
Nembrini, S., König, I.R., Wright, M.N.: The revival of the Gini importance? Bioinformatics 34, 3711–3718 (2018). https://doi.org/10.1093/bioinformatics/bty373
Banerjee, M., Ding, Y., Noone, A.-M.: Identifying representative trees from ensembles. Stat. Med. 31, 1601–1616 (2012). https://doi.org/10.1002/sim.4492
Laabs, B.-H., Westenberger, A., König, I.R.: Identification of representative trees in random forests based on a new tree-based distance measure. Adv. Data Anal. Classif. (2023). https://doi.org/10.1007/s11634-023-00537-7
Meinshausen, N.: Node harvest. Ann. Appl. Stat. 4 (2010). https://doi.org/10.1214/10-AOAS367
Deng, H.: Interpreting tree ensembles with inTrees. Int. J. Data Sci. Anal. 7, 277–287 (2019). https://doi.org/10.1007/s41060-018-0144-8
Bénard, C., Biau, G., da Veiga, S., Scornet, E.: Interpretable Random Forests via Rule Extraction (2020). https://doi.org/10.48550/ARXIV.2004.14841
Domingos, P.: Knowledge discovery via multiple models. Intell. Data Anal. 2, 187–202 (1998). https://doi.org/10.1016/S1088-467X(98)00023-7
Estruch, V., Ferri, C., Hernández-Orallo, J., Ramírez-Quintana, M. J.: Simple mimetic classifiers. In: Perner, P., Rosenfeld, A. (eds.) MLDM 2003. LNCS, vol. 2734, pp. 156–171. Springer, Heidelberg (2003). https://doi.org/10.1007/3-540-45065-3_14
Breiman, L. (ed.): Classification and Regression Trees. Chapman & Hall/CRC, Boca Raton (1998)
Breiman, L.: Bagging predictors. Mach. Learn. 24, 123–140 (1996). https://doi.org/10.1007/BF00058655
Seifert, S., Gundlach, S., Szymczak, S.: Surrogate minimal depth as an importance measure for variables in random forests. Bioinformatics 35, 3663–3671 (2019). https://doi.org/10.1093/bioinformatics/btz149
Voges, L.F., Jarren, L.C., Seifert, S.: Exploitation of surrogate variables in random forests for unbiased analysis of mutual impact and importance of features. Bioinformatics. 39, btad471 (2023). https://doi.org/10.1093/bioinformatics/btad471
Morris, T.P., White, I.R., Crowther, M.J.: Using simulation studies to evaluate statistical methods. Stat. Med. 38, 2074–2102 (2019). https://doi.org/10.1002/sim.8086
Wright, M.N., Ziegler, A.: ranger : a fast implementation of random forests for high dimensional data in C++ and R. J. Stat. Soft. 77 (2017). https://doi.org/10.18637/jss.v077.i01
Lang, M., Bischl, B., Surmann, D.: batchtools: tools for R to work on batch systems. JOSS. 2, 135 (2017). https://doi.org/10.21105/joss.00135
Janitza, S., Celik, E., Boulesteix, A.-L.: A computationally fast variable importance test for random forests for high-dimensional data. Adv. Data Anal. Classif. 12, 885–915 (2018). https://doi.org/10.1007/s11634-016-0276-4
Kursa, M.B., Rudnicki, W.R.: Feature selection with the Boruta package. J. Stat. Soft. 36 (2010). https://doi.org/10.18637/jss.v036.i11
Acknowledgments
This study was funded by the Medical Section of the University of Lübeck (J01–2024, BL).
Funding
All authors declare that they have no conflict of interest.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Appendix
Appendix
Comparison of artificial (ART, orange) and selected most representative trees (MRT, blue) with regard to their deviation in predictions from the original forest predictions for all five simulation scenarios in 100 repetitions. ARTs were generated using the five (10, 15, 20, 25, and 100) most important variables and 25% quantiles to limit the number of split points for continuous variables. Deviation was measured by the mean squared difference of the predictions made by the surrogate model and the original RF on an independent validation data set. (Color figure online)
Comparison of artificial (ART, orange) and selected most representative trees (MRT, blue) with regard to included variables in the surrogate model for all five simulation scenarios in 100 repetitions. ARTs were generated using the five (10, 15, 20, 25, and 100) most important variables and 25% quantiles to limit the number of split points for continuous variables. The fractions of covered effect variables were calculated by the number of effect variables that occur at least once in the surrogate model divided by the number of all simulated effect variables. (Color figure online)
Comparison of artificial (ART, orange) and selected most representative trees (MRT, blue) with regard to included variables in the surrogate model for all five simulation scenarios in 100 repetitions. ARTs were generated using the five (10, 15, 20, 25, and 100) most important variables and 25% quantiles to limit the number of split points for continuous variables. The fractions of covered noise variables were calculated by the number of noise variables that occur at least once in the surrogate model divided by the number of all simulated noise variables. (Color figure online)
Comparison of artificial (ART, orange) and selected most representative trees (MRT, blue) with regard to included variables in the surrogate model for all five simulation scenarios in 100 repetitions. ARTs were generated using the five (10, 15, 20, 25, and 100) most important variables and 25% quantiles to limit the number of split points for continuous variables. False discovery rate is represented by the number of splits in the surrogate model using one of the noise variables divided by the total number of splits in the surrogate model. (Color figure online)
Comparison of artificial (ART, orange) and selected most representative trees (MRT, blue) with regard to runtime for all five simulation scenarios in 100 repetitions. ARTs were generated using the five (10, 15, 20, 25, and 100) most important variables and 25% quantiles to limit the number of split points for continuous variables. (Color figure online)
Comparison of artificial (ART, orange) and selected most representative trees (MRT, blue) with regard to their deviation in predictions from the original forest predictions for all five simulation scenarios in 100 repetitions. ARTs were generated using the five most important variables and 50%, 25%, and 10% quantiles to limit the number of split points for continuous variables. Deviation was measured by the mean squared difference of the predictions made by the surrogate model and the original RF on an independent validation data set. (Color figure online)
Comparison of artificial (ART, orange) and selected most representative trees (MRT, blue) with regard to included variables in the surrogate model for all five simulation scenarios in 100 repetitions. ARTs were generated using the five most important variables and 50%, 25%, and 10% quantiles to limit the number of split points for continuous variables. The fractions of covered effect variables were calculated by the number of effect variables that occur at least once in the surrogate model divided by the number of all simulated effect variables. (Color figure online)
Comparison of artificial (ART, orange) and selected most representative trees (MRT, blue) with regard to included variables in the surrogate model for all five simulation scenarios in 100 repetitions. ARTs were generated using the five most important variables and 50%, 25%, and 10% quantiles to limit the number of split points for continuous variables. The fractions of covered noise variables were calculated by the number of noise variables that occur at least once in the surrogate model divided by the number of all simulated noise variables. (Color figure online)
Comparison of artificial (ART, orange) and selected most representative trees (MRT, blue) with regard to included variables in the surrogate model for all five simulation scenarios in 100 repetitions. ARTs were generated using the five most important variables and 50%, 25%, and 10% quantiles to limit the number of split points for continuous variables. False discovery rate is represented by the number of splits in the surrogate model using one of the noise variables divided by the total number of splits in the surrogate model. (Color figure online)
Comparison of artificial (ART, orange) and selected most representative trees (MRT, blue) with regard to runtime for all five simulation scenarios in 100 repetitions. ARTs were generated using the five most important variables and 50%, 25%, and 10% quantiles to limit the number of split points for continuous variables. (Color figure online)
Relationship of number of splits (simplicity) for artificial (ART, orange) and selected most representative trees (MRT, blue) with deviation of the surrogate model predictions from the original random forest predictions. (Color figure online)
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Laabs, BH., Kronziel, L.L., König, I.R., Szymczak, S. (2024). Construction of Artificial Most Representative Trees by Minimizing Tree-Based Distance Measures. In: Longo, L., Lapuschkin, S., Seifert, C. (eds) Explainable Artificial Intelligence. xAI 2024. Communications in Computer and Information Science, vol 2154. Springer, Cham. https://doi.org/10.1007/978-3-031-63797-1_15
Download citation
DOI: https://doi.org/10.1007/978-3-031-63797-1_15
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-63796-4
Online ISBN: 978-3-031-63797-1
eBook Packages: Computer ScienceComputer Science (R0)