Abstract
The use of Shap scores has become widespread in Explainable AI. However, their computation is in general intractable, in particular when done with a black-box classifier, such as neural network. Recent research has unveiled classes of open-box Boolean Circuit classifiers for which Shap can be computed efficiently. We show how to transform binary neural networks into those circuits for efficient Shap computation. We use logic-based knowledge compilation techniques. The performance gain is huge, as we show in the light of our experiments.
L. Bertossi—Member of the Millennium Institute for Foundational Research on Data (IMFD, Chile).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
California Housing Prices dataset: https://www.kaggle.com/datasets/camnugent/california-housing-prices.
- 2.
- 3.
We could also used binarized sigmoid and softmax functions.
- 4.
At this point is where using \(1,-1\) in the BNN instead of 1, 0 becomes useful.
- 5.
We say “a CNF” meaning “a formula in CNF”. Similarly in plural.
- 6.
- 7.
Binarization could be achieved in other ways, depending on the feature, for better interaction with the feature independence assumption.
- 8.
The experiments were run on Google Colab (with an NVIDIA Tesla T4 enabled). Algorithm 1 was programmed in Python. The complete code for Google Colab can be found at: https://github.com/Jorvan758/dDBCSFi2.
- 9.
As done in [4], but with only the entity sample.
References
Arenas, M., Barceló, P., Bertossi, L., Monet, M.: On the complexity of SHAP-score-based explanations: tractability via knowledge compilation and non-approximability results. J. Mach. Learn. Res. 24(63), 1–58 (2023). Extended version of [2]
Arenas, M., Barceló, P., Bertossi, L., Monet, M.: The tractability of SHAP-score-based explanations for classification over deterministic and decomposable Boolean circuits. In: Proceedings of the 35th AAAI Conference on Artificial Intelligence, pp. 6670–6678 (2021)
Audemard, G., Koriche, F., Marquis, P.: On tractable XAI queries based on compiled representations. In: Proceedings KR 2020, pp. 838–849 (2020)
Bertossi, L., Li, J., Schleich, M., Suciu, D., Vagena, Z.: Causality-based explanation of classification outcomes. In: Proceedings of the 4th International Workshop on “Data Management for End-to-End Machine Learning” (DEEM) at ACM SIGMOD/PODS, pp. 1–10 (2020). Posted as Corr ar**v Paper ar**v:2003.06868
Bertossi, L.: Declarative approaches to counterfactual explanations for classification. Theory Pract. Logic Program. 23(3), 559–593 (2023)
Bollig, B., Buttkus, M.: On the relative succinctness of sentential decision diagrams. Theory Comput. Syst. 63(6), 1250–1277 (2019)
Bova, S.: SDDs are exponentially more succinct than OBDDs. In: Proceedings of the 30th AAAI Conference on Artificial Intelligence, pp. 929–935 (2016)
Bryant, R.E.: Graph-based algorithms for Boolean function manipulation. IEEE Trans. Comput. C-35(8), 677–691 (1986)
Choi, A., Darwiche, A.: Dynamic minimization of sentential decision diagrams. In: Proceedings of the 27th AAAI Conference on Artificial Intelligence, pp. 187–194 (2013)
Choi, A., Darwiche, A.: SDD Advanced-User Manual Version 2.0. Automated Reasoning Group, UCLA (2018)
Darwiche, A., Hirth, A.: On the reasons behind decisions. In: Proceedings of the 24th European Conference on Artificial Intelligence, pp. 712–720 (2020)
Darwiche, A., Marquis, P.: A knowledge compilation map. J. Artif. Intell. Res. 17(1), 229–264 (2002)
Darwiche, A.: On the tractable counting of theory models and its application to truth maintenance and belief revision. J. Appl. Non-Classical Logics 11(1–2), 11–34 (2011)
Darwiche, A.: SDD: a new canonical representation of propositional knowledge bases. In: Proceedings of the 22th International Joint Conference on Artificial Intelligence (IJCAI 2011), pp. 819–826 (2011)
Flum, J., Grohe, M.: Parameterized Complexity Theory. Springer, Heidelberg (2006). https://doi.org/10.1007/3-540-29953-X
Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5), 1–42 (2018)
Hubara, I., Courbariaux, M., Soudry, D., El-Yaniv, R., Bengio, Y.: Binarized neural networks. In: Proceedings of the NIPS 2016, pp. 4107–4115 (2016)
Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Proceedings of the 31st International Conference on Neural Information Processing Systems, pp. 4768–4777 (2017). ar**v Paper ar**v:1705.07874
Lundberg, S., et al.: From local explanations to global understanding with explainable AI for trees. Nat. Mach. Intell. 2(1), 56–67 (2020). ar**v Paper ar**v:1905.04610
Manthey, N.: RISS tool collection (2017). https://github.com/nmanthey/riss-solver
Meert, W., Choi, A.: Python Wrapper Package to Interactively Use Sentential Decision Diagrams (SDD) (2018). https://github.com/wannesm/PySDD
Nakamura, K., Denzumi, S., Nishino, M.: Variable shift SDD: a more succinct sentential decision diagram. In: Proceedings of the 18th International Symposium on Experimental Algorithms (SEA 2020). Leibniz International Proceedings in Informatics, vol. 160, pp. 22:1–22:13 (2020)
Narodytska, N., Kasiviswanathan, S., Ryzhyk, L., Sagiv, M., Walsh, T.: Verifying properties of binarized deep neural networks. In: Proceedings of the 32nd AAAI Conference on Artificial Intelligence, pp. 6615–6624 (2018)
Nugent, C.: California Housing Prices (2018). https://www.kaggle.com/datasets/camnugent/california-housing-prices
Oztok, U., Darwiche, A.: On compiling CNF into decision-DNNF. In: O’Sullivan, B. (ed.) CP 2014. LNCS, vol. 8656, pp. 42–57. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10428-7_7
Oztok, U., Darwiche, A.: On compiling DNNFs without determinism (2017). ar**v:1709.07092
Qin, H., Gong, R., Liu, X., Bai, X., Song, J., Sebe, N.: Binary neural networks: a survey. Pattern Recogn. 105, 107281 (2020)
Ras, G., **e, N., van Gerven, M., Doran, D.: Explainable deep learning: a field guide for the uninitiated. J. Artif. Intell. Res. 73, 329–396 (2022)
Roth, A.: The Shapley Value: Essays in Honor of Lloyd S. Shapley. Cambridge University Press (1988)
Rudin, C.: Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nat. Mach. Intell. 1, 206–215 (2019). ar**v Paper ar**v:1811.10154
Shapley, L.S.: A value for n-person games. In: Contributions to the Theory of Games (AM-28), vol. 2, pp. 307–318 (1953)
Shih, A., Van den Broeck, G., Beame, P., Amarilli, A.: Smoothing structured decomposable circuits. In: Proceedings of the NeurIPS (2019)
Shi, W., Shih, A., Darwiche, A., Choi, A.: On tractable representations of binary neural networks. In: Proceedings of the 17th International Conference on Principles of Knowledge Representation and Reasoning, pp. 882–892 (2020)
Shih, A., Darwiche, A., Choi, A.: Verifying binarized neural networks by Angluin-Style learning. In: Janota, M., Lynce, I. (eds.) SAT 2019. LNCS, vol. 11628, pp. 354–370. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-24258-9_25
Simons, T., Lee, D.-J.: A review of binarized neural networks. Electronics 8(6), 661 (2019)
Van den Broeck, G., Darwiche, A.: On the role of canonicity in knowledge compilation. In: Proceedings of the 29th AAAI Conference on Artificial Intelligence, pp. 1641–1648 (2015)
Van den Broeck, G., Lykov, A., Schleich, M., Suciu, D.: On the tractability of SHAP explanations. In: Proceedings of the 35th AAAI Conference on Artificial Intelligence, pp. 6505–6513 (2021)
Acknowledgments
Special thanks to Arthur Choi, Andy Shih, Norbert Manthey, Maximilian Schleich and Adnan Darwiche, for their valuable help. Work was funded by ANID - Millennium Science Initiative Program - Code ICN17002; CENIA, FB210017 (Financiamiento Basal para Centros Científicos y Tecnológicos de Excelencia de ANID), Chile; SKEMA Business School, and NSERC-DG 2023-04650. L. Bertossi is a Professor Emeritus at Carleton University, Canada.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Bertossi, L., León, J.E. (2023). Efficient Computation of Shap Explanation Scores for Neural Network Classifiers via Knowledge Compilation. In: Gaggl, S., Martinez, M.V., Ortiz, M. (eds) Logics in Artificial Intelligence. JELIA 2023. Lecture Notes in Computer Science(), vol 14281. Springer, Cham. https://doi.org/10.1007/978-3-031-43619-2_4
Download citation
DOI: https://doi.org/10.1007/978-3-031-43619-2_4
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-43618-5
Online ISBN: 978-3-031-43619-2
eBook Packages: Computer ScienceComputer Science (R0)