SpringerLink
Log in

Implicit Embeddings via GAN Inversion for High Resolution Chest Radiographs

  • Conference paper
  • First Online:
Medical Applications with Disentanglements (MAD 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13823))

Included in the following conference series:

  • 418 Accesses

Abstract

Generative models allow for the creation of highly realistic artificial samples, opening up promising applications in medical imaging. In this work, we propose a multi-stage encoder-based approach to invert the generator of a generative adversarial network (GAN) for high resolution chest radiographs. This gives direct access to its implicitly formed latent space, makes generative models more accessible to researchers, and enables to apply generative techniques to actual patient’s images. We investigate various applications for this embedding, including image compression, disentanglement in the encoded dataset, guided image manipulation, and creation of stylized samples. We find that this type of GAN inversion is a promising research direction in the domain of chest radiograph modeling and opens up new ways to combine realistic X-ray sample synthesis with radiological image analysis.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
EUR 29.95
Price includes VAT (Germany)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
EUR 46.00
Price includes VAT (Germany)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
EUR 58.84
Price includes VAT (Germany)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free ship** worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Abdal, R., Qin, Y., Wonka, P.: Image2StyleGAN: how to embed images into the StyleGAN latent space? In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4432–4441 (2019)

    Google Scholar 

  2. Abdal, R., Qin, Y., Wonka, P.: Image2StyleGAN++: how to edit the embedded images? In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8296–8305 (2020)

    Google Scholar 

  3. Bau, D., et al.: Seeing what a GAN cannot generate. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4502–4511 (2019)

    Google Scholar 

  4. Bermano, A.H., et al.: State-of-the-art in the architecture, methods and applications of StyleGAN. In: Computer Graphics Forum, vol. 41, pp. 591–611. Wiley Online Library (2022)

    Google Scholar 

  5. Chai, L., Wulff, J., Isola, P.: Using latent space regression to analyze and leverage compositionality in GANs. In: 9th International Conference on Learning Representations, ICLR (2021)

    Google Scholar 

  6. Chen, X., Fan, H., Girshick, R., He, K.: Improved Baselines with Momentum Contrastive Learning. ar**v:2003.04297 (2020)

  7. Creswell, A., Bharath, A.A.: Inverting the generator of a generative adversarial network. IEEE Trans. Neural Netw. Learn. Syst. 30(7), 1967–1974 (2018)

    Article  Google Scholar 

  8. Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: ImageNet: a large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255. IEEE (2009)

    Google Scholar 

  9. Donahue, J., Krähenbühl, P., Darrell, T.: Adversarial feature learning. In: 5th International Conference on Learning Representations, ICLR (2017)

    Google Scholar 

  10. Ester, M., Kriegel, H.P., Sander, J., Xu, X.: A density-based algorithm for discovering clusters in large spatial databases with noise. In: Proceedings of the Second International Conference on Knowledge Discovery and Data Mining, KDD 1996, Portland, Oregon, pp. 226–231. AAAI Press (1996)

    Google Scholar 

  11. Fetty, L., et al.: Latent space manipulation for high-resolution medical image synthesis via the StyleGAN. Z. Med. Phys. 30(4), 305–314 (2020)

    Article  Google Scholar 

  12. Goetschalckx, L., Andonian, A., Oliva, A., Isola, P.: GANalyze: toward visual definitions of cognitive image properties. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5744–5753 (2019)

    Google Scholar 

  13. Goodfellow, I., et al.: Generative adversarial networks. In: Advances in Neural Information Processing Systems, vol. 27 (2014)

    Google Scholar 

  14. Han, L., Lyu, Y., Peng, C., Zhou, S.K.: GAN-based disentanglement learning for chest X-ray rib suppression. Med. Image Anal. 77 (2022)

    Google Scholar 

  15. Han, T., et al.: Breaking medical data sharing boundaries by using synthesized radiographs. Sci. Adv. 6(49), eabb7973 (2020)

    Google Scholar 

  16. Härkönen, E., Hertzmann, A., Lehtinen, J., Paris, S.: GANSpace: discovering interpretable GAN controls. Adv. Neural. Inf. Process. Syst. 33, 9841–9850 (2020)

    Google Scholar 

  17. He, K., Fan, H., Wu, Y., **e, S., Girshick, R.: Momentum contrast for unsupervised visual representation learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9729–9738 (2020)

    Google Scholar 

  18. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)

    Google Scholar 

  19. Irvin, J., et al.: CheXpert: a large chest radiograph dataset with uncertainty labels and expert comparison. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 590–597 (2019)

    Google Scholar 

  20. Karras, T., Aila, T., Laine, S., Lehtinen, J.: Progressive growing of GANs for improved quality, stability, and variation. In: 6th International Conference on Learning Representations, ICLR (2018)

    Google Scholar 

  21. Karras, T., et al.: Alias-free generative adversarial networks. In: Advances in Neural Information Processing Systems, vol. 34 (2021)

    Google Scholar 

  22. Karras, T., Laine, S., Aila, T.: A style-based generator architecture for generative adversarial networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4401–4410 (2019)

    Google Scholar 

  23. Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., Aila, T.: Analyzing and improving the image quality of StyleGAN. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8110–8119 (2020)

    Google Scholar 

  24. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: 3rd International Conference on Learning Representations, ICLR (2017)

    Google Scholar 

  25. Li, Z., Li, H., Han, H., Shi, G., Wang, J., Zhou, S.K.: Encoding CT anatomy knowledge for unpaired chest X-ray image decomposition. In: Shen, D., et al. (eds.) MICCAI 2019. LNCS, vol. 11769, pp. 275–283. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-32226-7_31

    Chapter  Google Scholar 

  26. Liu, Z., Mao, H., Wu, C.Y., Feichtenhofer, C., Darrell, T., **e, S.: A ConvNet for the 2020s. ar**v:2201.03545 (2022)

  27. McInnes, L., Healy, J., Saul, N., Großberger, L.: UMAP: uniform manifold approximation and projection. J. Open Source Softw. 3(29), 861 (2018)

    Article  Google Scholar 

  28. Menon, S., Damian, A., Hu, S., Ravi, N., Rudin, C.: PULSE: self-supervised photo upsampling via latent space exploration of generative models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2437–2445 (2020)

    Google Scholar 

  29. Nitzan, Y., Bermano, A., Li, Y., Cohen-Or, D.: Face identity disentanglement via latent space map**. ACM Trans. Graph. 39(6), 225:1–225:14 (2020)

    Google Scholar 

  30. Rajpurkar, P., et al.: CheXNet: Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning. ar**v:1711.05225 (2017)

  31. Ren, Z., Yu, S.X., Whitney, D.: Controllable medical image generation via GAN. J. Percept. Imaging 5, 000502-1–000502-15 (2022)

    Google Scholar 

  32. Richardson, E., et al.: Encoding in style: a StyleGAN encoder for image-to-image translation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2287–2296 (2021)

    Google Scholar 

  33. Schutte, K., Moindrot, O., Hérent, P., Schiratti, J.B., Jégou, S.: Using StyleGAN for Visual Interpretability of Deep Learning Models on Medical Images. ar**v:2101.07563 (2021)

  34. Segal, B., Rubin, D.M., Rubin, G., Pantanowitz, A.: Evaluating the clinical realism of synthetic chest X-rays generated using progressively growing GANs. SN Comput. Sci. 2(4), 1–17 (2021). https://doi.org/10.1007/s42979-021-00720-7

    Article  Google Scholar 

  35. Shen, Y., Gu, J., Tang, X., Zhou, B.: Interpreting the latent space of GANs for semantic face editing. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9243–9252 (2020)

    Google Scholar 

  36. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: 3rd International Conference on Learning Representations, ICLR (2015)

    Google Scholar 

  37. Sundaram, S., Hulkund, N.: GAN-based data augmentation for chest X-ray classification. ar**v:2107.02970 (2021)

  38. Tang, Y., Tang, Y., Zhu, Y., **ao, J., Summers, R.M.: A disentangled generative model for disease decomposition in chest X-rays via normal image synthesis. Med. Image Anal. 67 (2021)

    Google Scholar 

  39. Tov, O., Alaluf, Y., Nitzan, Y., Patashnik, O., Cohen-Or, D.: Designing an encoder for StyleGAN image manipulation. ACM Trans. Graph. (TOG) 40(4), 1–14 (2021)

    Article  Google Scholar 

  40. Wang, X., Peng, Y., Lu, L., Lu, Z., Bagheri, M., Summers, R.M.: Chestx-ray8: hospital-scale chest X-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2097–2106 (2017)

    Google Scholar 

  41. Wei, T., et al.: E2Style: improve the efficiency and effectiveness of StyleGAN inversion. IEEE Trans. Image Process. 31, 3267–3280 (2022)

    Article  Google Scholar 

  42. **a, W., Zhang, Y., Yang, Y., Xue, J.H., Zhou, B., Yang, M.H.: GAN Inversion: A Survey. ar**v:2101.05278 (2022)

  43. Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The unreasonable effectiveness of deep features as a perceptual metric. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 586–595 (2018)

    Google Scholar 

  44. Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2223–2232 (2017)

    Google Scholar 

Download references

Acknowledgments

This work has been funded by the German Federal Ministry of Education and Research and the Bavarian State Ministry for Science and the Arts. The authors of this work take full responsibility for its content. The authors gratefully acknowledge LMU Klinikum for providing computing resources on their Clinical Open Research Engine (CORE). We thank the anonymous reviewers for their constructive comments, which helped us to improve the manuscript.

Author information

Authors and Affiliations

  1. Department of Statistics, LMU Munich, Munich, Germany

    Tobias Weber, Bernd Bischl & David Rügamer

  2. Department of Radiology, University Hospital, LMU Munich, Munich, Germany

    Tobias Weber & Michael Ingrisch

Authors
  1. Tobias Weber
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Michael Ingrisch
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Bernd Bischl
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. David Rügamer
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Tobias Weber .

Editor information

Editors and Affiliations

  1. Essen University Hospital, Essen, Germany

    Jana Fragemann

  2. Graz University of Technology, Graz, Austria

    Jianning Li

  3. University of Edinburgh, Edinburgh, UK

    **ao Liu

  4. University of Edinburgh, Edinburgh, UK

    Sotirios A. Tsaftaris

  5. Graz University of Technology, Graz, Austria

    Jan Egger

  6. German Cancer Consortium, Essen, Germany

    Jens Kleesiek

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Weber, T., Ingrisch, M., Bischl, B., Rügamer, D. (2023). Implicit Embeddings via GAN Inversion for High Resolution Chest Radiographs. In: Fragemann, J., Li, J., Liu, X., Tsaftaris, S.A., Egger, J., Kleesiek, J. (eds) Medical Applications with Disentanglements. MAD 2022. Lecture Notes in Computer Science, vol 13823. Springer, Cham. https://doi.org/10.1007/978-3-031-25046-0_3

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-25046-0_3

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-25045-3

  • Online ISBN: 978-3-031-25046-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Societies and partnerships

Search

Navigation