SpringerLink
Log in

Exploring methods for the generation of visual counterfactuals in the latent space

  • Short Paper
  • Published:
Pattern Analysis and Applications Aims and scope Submit manuscript

Abstract

In the field of eXplainable Artificial Intelligence (XAI), the generation of counterfactuals is a promising method for human-interpretable explanations. A counterfactual explanation describes a causal situation in the form: “If X had not occurred, Y would not have occurred”. In this work, we study the generation of visual counterfactuals in the latent space for deep learning image classification models. We explore how to adapt the training environment to facilitate the generation of counterfactuals, combining ideas coming from different fields such as multitasking or generative learning, with the aim of develo** more interpretable models. We study well-known counterfactual methods and how to apply them in the latent space. Furthermore, we propose a new way of generating counterfactuals working in the latent space and compare it with the other studied approaches, achieving competitive results.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Algorithm 1
Fig. 2

Data Availability

The data that support the findings of this study are available from the corresponding author upon reasonable request.

Notes

  1. https://docs.seldon.io/projects/alibi/en/latest/methods/CF.html.

  2. https://docs.seldon.io/projects/alibi/en/latest/methods/CFProto.html.

References

  1. Abbaasi S, Ghiasi-Shirazi K, Harati A (2023) A multi-prototype capsule network for image recognition with high intra-class variations. Neural Process Lett 55(5):1–15

    Article  Google Scholar 

  2. Alvarez Melis D, Jaakkola T (2018) Towards robust interpretability with self-explaining neural networks. Adv Neural Inf Process Syst 31

  3. Arrieta AB, Díaz-Rodríguez N, Del Ser J, Bennetot A, Tabik S, Barbado A, García S, Gil-López S, Molina D, Benjamins R et al (2020) Explainable artificial intelligence (XAI): concepts, taxonomies, opportunities and challenges toward responsible AI. Inf Fusion 58:82–115

    Article  Google Scholar 

  4. Artelt A, Vaquet V, Velioglu R, Hinder F, Brinkrolf J, Schilling M, Hammer B (2021) Evaluating robustness of counterfactual explanations. In: 2021 IEEE symposium series on computational intelligence (SSCI), pp. 01–09

  5. Bennetot A, Laurent J-L, Chatila R, Díaz-Rodríguez N (2019) Towards explainable neural-symbolic visual reasoning. In: IJCAI neural-symbolic learning and reasoning workshop

  6. Freiesleben T (2021) The intriguing relation between counterfactual explanations and adversarial examples. Minds Mach 32(1):1–33

    Google Scholar 

  7. Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2020) Generative adversarial networks. Commun ACM 63(11):139–144. https://doi.org/10.1145/3422622

    Article  MathSciNet  Google Scholar 

  8. Khan Salman H, Hayat M, Barnes N (2018) Adversarial training of variational auto-encoders for high fidelity image generation. In: 2018 IEEE winter conference on applications of computer vision (WACV), pp 1312–1320, https://doi.org/10.1109/WACV.2018.00148

  9. Kingma Diederik P, Ba J (2015) Adam: a method for stochastic optimization. In: 3rd international conference on learning representations, pp 1–15

  10. Kingma Diederik P, Welling M (2014) Auto-encoding variational Bayes in 2nd international conference on learning representations. In: ICLR 2014-conference track proceedings

  11. Kingma Diederik P, Welling M (2019) An introduction to variational autoencoders. ar**v preprint ar**v:1906.02691

  12. Klaise J, Van Looveren A, Vacanti G, Coca A (2021) Alibi explain: algorithms for explaining machine learning models. J Mach Learn Res 22(181):1–7

    Google Scholar 

  13. LeCun Y, Cortes C, Burges CJ (2010) Mnist handwritten digit database. ATT Labs. Available http://yann.lecun.com/exdb/mnist

  14. Molnar C (2020) Interpretable machine learning. Lulu.com

  15. Nemirovsky D, Thiebaut N, Xu Y, Gupta A (2020) Countergan: generating realistic counterfactuals with residual generative adversarial nets. ar**v preprint ar**v:2009.05199

  16. Perarnau G, Van De Weijer J, Raducanu B, Álvarez JM (2016) Invertible conditional gans for image editing. ar**v preprint ar**v:1611.06355

  17. Pesteie M, Abolmaesumi P, Rohling RN (2019) Adaptive augmentation of medical data using independently conditional variational auto-encoders. IEEE Trans Med Imaging 38(12):2807–2820. https://doi.org/10.1109/TMI.2019.2914656

    Article  Google Scholar 

  18. Ribeiro MT, Singh S, Guestrin C (2016) Why should I trust you? Explaining the predictions of any classifier. In: ACM SIGKDD international conference on knowledge discovery and data mining, pp 1135–1144

  19. Singla S, Pollack B, Wallace S, Batmanghelich K (2021) Explaining the black-box smoothly-a counterfactual approach. ar**v preprint ar**v:2101.04230

  20. Van Looveren A, Klaise J (2021) Interpretable counterfactual explanations guided by prototypes. In: Joint European conference on machine learning and knowledge discovery in databases. Springer: New York, pp 650–665

  21. Wachter S, Mittelstadt B, Russell C (2017) Counterfactual explanations without opening the black box: automated decisions and the GDPR. Harv JL Tech 31:841

    Google Scholar 

  22. Wang P, Vasconcelos N (2020) Scout: self-aware discriminant counterfactual explanations. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR)

  23. **ao H, Rasul K, Vollgraf R (2017) Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. CoRR, http://arxiv.org/abs/1708.07747

  24. Zhou B, Khosla A, Lapedriza A, Oliva A, Torralba A (2016) Learning deep features for discriminative localization. In: IEEE conference on computer vision and pattern recognition, pp 2921–2929

Download references

Author information

Authors and Affiliations

  1. University of Granada, 18071, Granada, Spain

    David Morales, Manuel P. Cuéllar & Diego P. Morales

  2. HAT.tec, Lilienthalstraße 15, 85579, Neubiberg, Germany

    David Morales

Authors
  1. David Morales
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Manuel P. Cuéllar
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Diego P. Morales
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to David Morales.

Ethics declarations

Conflict of interest

The authors declare that they have no Conflict of interest.

Additional information

Publisher's Note

Publisher's Note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Partial financial support was received from HAT.tec GmbH. The funders had no role in the study design, data collection, analysis, and preparation of the manuscript.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Morales, D., Cuéllar, M.P. & Morales, D.P. Exploring methods for the generation of visual counterfactuals in the latent space. Pattern Anal Applic 27, 81 (2024). https://doi.org/10.1007/s10044-024-01299-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10044-024-01299-4

Keywords

Search

Navigation