SpringerLink
Log in

A Wasserstein GAN for Joint Learning of Inpainting and Spatial Optimisation

  • Conference paper
  • First Online:
Image and Video Technology (PSIVT 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13763))

Included in the following conference series:

Abstract

Image inpainting is a restoration method that reconstructs missing image parts. However, a carefully selected mask of known pixels that yield a high quality inpainting can also act as a sparse image representation. This challenging spatial optimisation problem is essential for practical applications such as compression. So far, it has been almost exclusively addressed by model-based approaches. First attempts with neural networks seem promising, but are tailored towards specific inpainting operators or require postprocessing. To address this issue, we propose the first generative adversarial network (GAN) for spatial inpainting data optimisation. In contrast to previous approaches, it allows joint training of an inpainting generator and a corresponding mask optimisation network. With a Wasserstein distance, we ensure that our inpainting results accurately reflect the statistics of natural images. This yields significant improvements in visual quality and speed over conventional stochastic models. It also outperforms current spatial optimisation networks.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
EUR 29.95
Price includes VAT (Germany)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
EUR 46.00
Price includes VAT (Germany)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
EUR 58.84
Price includes VAT (Germany)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free ship** worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Alt, T., Peter, P., Weickert, J.: Learning sparse masks for diffusion-based image inpainting. In: Pinho, A.J., Georgieva, P., Teixeira, L.F., Sánchez, J.A. (eds.) IbPRIA 2022. LNCS, vol. 13256, pp. 528–539. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-04881-4_42

    Chapter  Google Scholar 

  2. Arbelaez, P., Maire, M., Fowlkes, C., Malik, J.: Contour detection and hierarchical image segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 33(5), 898–916 (2011)

    Article  Google Scholar 

  3. Arjovsky, M., Chintala, S., Bottou, L.: Wasserstein generative adversarial networks. In: Precup, D., Teh, Y.W. (eds.) Proceedings of the 34th International Conference on Machine Learning. Proceedings of Machine Learning Research, Sydney, Australia, vol. 70, pp. 214–223, August 2017

    Google Scholar 

  4. Ballé, J., Laparra, V., Simoncelli, E.P.: End-to-end optimised image compression. In: Proceedings of the 5th International Conference on Learning Representations, Toulon, France, April 2017

    Google Scholar 

  5. Belhachmi, Z., Bucur, D., Burgeth, B., Weickert, J.: How to choose interpolation data in images. SIAM J. Appl. Math. 70(1), 333–352 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  6. Bertalmío, M., Caselles, V., Masnou, S., Sapiro, G.: Inpainting. In: Ikeuchi, K. (ed.) Computer Vision: A Reference Guide, pp. 401–416. Springer, New York (2014)

    Chapter  Google Scholar 

  7. Bonettini, S., Loris, I., Porta, F., Prato, M., Rebegoldi, S.: On the convergence of a linesearch based proximal-gradient method for nonconvex optimization. Inverse Probl. 33(5), 055005 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  8. Chan, T.F., Shen, J.: Non-texture inpainting by curvature-driven diffusions (CDD). J. Vis. Commun. Image Represent. 12(4), 436–449 (2001)

    Article  Google Scholar 

  9. Chen, Y., Ranftl, R., Pock, T.: A bi-level view of inpainting-based image compression. In: Kúkelová, Z., Heller, J. (eds.) Proceedings of the 19th Computer Vision Winter Workshop, Křtiny, Czech Republic, February 2014

    Google Scholar 

  10. Chizhov, V., Weickert, J.: Efficient data optimisation for harmonic inpainting with finite elements. In: Tsapatsoulis, N., Panayides, A., Theocharides, T., Lanitis, A., Pattichis, C., Vento, M. (eds.) CAIP 2021. LNCS, vol. 13053, pp. 432–441. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-89131-2_40

    Chapter  Google Scholar 

  11. Dai, Q., Chopp, H., Pouyet, E., Cossairt, O., Walton, M., Katsaggelos, A.K.: Adaptive image sampling using deep learning and its application on X-Ray fluorescence image reconstruction. IEEE Trans. Multimedia 22(10), 2564–2578 (2019)

    Article  Google Scholar 

  12. Esedoglu, S., Shen, J.: Digital inpainting based on the Mumford-Shah-Euler image model. Eur. J. Appl. Math. 13(4), 353–370 (2002)

    Article  MathSciNet  MATH  Google Scholar 

  13. Galić, I., Weickert, J., Welk, M., Bruhn, A., Belyaev, A., Seidel, H.P.: Image compression with anisotropic diffusion. J. Math. Imaging Vis. 31(2–3), 255–269 (2008). https://doi.org/10.1007/s10851-008-0087-0

    Article  MathSciNet  MATH  Google Scholar 

  14. Goodfellow, I., et al.: Generative adversarial nets. In: Advances in Neural Information Processing Systems, vol. 27, pp. 2672–2680 (2014)

    Google Scholar 

  15. Hoeltgen, L., Setzer, S., Weickert, J.: An optimal control approach to find sparse data for Laplace interpolation. In: Heyden, A., Kahl, F., Olsson, C., Oskarsson, M., Tai, X.-C. (eds.) EMMCVPR 2013. LNCS, vol. 8081, pp. 151–164. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-40395-8_12

    Chapter  Google Scholar 

  16. Iijima, T.: Basic theory on normalization of pattern (in case of typical one-dimensional pattern). Bull. Electrotechnical Lab. 26, 368–388 (1962). in Japanese

    Google Scholar 

  17. Karos, L., Bheed, P., Peter, P., Weickert, J.: Optimising data for exemplar-based inpainting. In: Blanc-Talon, J., Helbert, D., Philips, W., Popescu, D., Scheunders, P. (eds.) ACIVS 2018. LNCS, vol. 11182, pp. 547–558. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01449-0_46

    Chapter  Google Scholar 

  18. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: Proceedings of the 3rd International Conference on Learning Representations, San Diego, CA, May 2015

    Google Scholar 

  19. Liu, G., Reda, F.A., Shih, K.J., Wang, T.-C., Tao, A., Catanzaro, B.: Image inpainting for irregular holes using partial convolutions. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11215, pp. 89–105. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01252-6_6

    Chapter  Google Scholar 

  20. Liu, H., Jiang, B., **ao, Y., Yang, C.: Coherent semantic attention for image inpainting. In: Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision, Seoul, Korea, pp. 4170–4179, October 2017

    Google Scholar 

  21. Mainberger, M., et al.: Optimising spatial and tonal data for homogeneous diffusion inpainting. In: Bruckstein, A.M., ter Haar Romeny, B.M., Bronstein, A.M., Bronstein, M.M. (eds.) SSVM 2011. LNCS, vol. 6667, pp. 26–37. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-24785-9_3

    Chapter  Google Scholar 

  22. Masnou, S., Morel, J.M.: Level lines based disocclusion. In: Proceedings of the 1998 IEEE International Conference on Image Processing, Chicago, IL, vol. 3, pp. 259–263, October 1998

    Google Scholar 

  23. Pathak, D., Krähenbühl, P., Donahue, J., Darrell, T., Efros, A.A.: Context encoders: feature learning by inpainting. In: Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, pp. 2536–2544, June 2016

    Google Scholar 

  24. Peter, P., Hoffmann, S., Nedwed, F., Hoeltgen, L., Weickert, J.: Evaluating the true potential of diffusion-based inpainting in a compression context. Sig. Process. Image Commun. 46, 40–53 (2016)

    Article  Google Scholar 

  25. Peter, P., Weickert, J., Munk, A., Krivobokova, T., Li, H.: Justifying tensor-driven diffusion from structure-adaptive statistics of natural images. In: Tai, X.-C., Bae, E., Chan, T.F., Lysaker, M. (eds.) EMMCVPR 2015. LNCS, vol. 8932, pp. 263–277. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-14612-6_20

    Chapter  Google Scholar 

  26. Schmaltz, C., Peter, P., Mainberger, M., Ebel, F., Weickert, J., Bruhn, A.: Understanding, optimising, and extending data compression with anisotropic diffusion. Int. J. Comput. Vis. 108(3), 222–240 (2014). https://doi.org/10.1007/s11263-014-0702-z

    Article  MathSciNet  Google Scholar 

  27. Theis, L., Shi, W., Cunningham, A., Huszár, F.: Lossy image compression with compressive autoencoders. In: Proceedings of the 5th International Conference on Learning Representations, Toulon, France, April 2016

    Google Scholar 

  28. Ulyanov, D., Vedaldi, A., Lempitsky, V.: Deep image prior. In: Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, pp. 9446–9454, June 2018

    Google Scholar 

  29. Vaserstein, L.N.: Markov processes over denumerable products of spaces, describing large systems of automata. Problemy Peredachi Informatsii 5(3), 64–72 (1969)

    MathSciNet  MATH  Google Scholar 

  30. Vašata, D., Halama, T., Friedjungová, M.: Image inpainting using Wasserstein generative adversarial imputation network. In: Farkaš, I., Masulli, P., Otte, S., Wermter, S. (eds.) ICANN 2021. LNCS, vol. 12892, pp. 575–586. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-86340-1_46

    Chapter  Google Scholar 

  31. Wang, N., Zhang, Y., Zhang, L.: Dynamic selection network for image inpainting. IEEE Trans. Image Process. 30, 1784–1798 (2021)

    Article  Google Scholar 

  32. Wang, W., Zhang, J., Niu, L., Ling, H., Yang, X., Zhang, L.: Parallel multi-resolution fusion network for image inpainting. In: Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision, pp. 14559–14568, October 2021

    Google Scholar 

  33. Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13, 600–612 (2004)

    Article  Google Scholar 

  34. **e, J., Xu, L., Chen, E.: Image denoising and inpainting with deep neural networks. In: Proceedings of the 26th International Conference on Neural Information Processing Systems, Lake Tahoe, NV, pp. 350–358, December 2012

    Google Scholar 

  35. Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., Huang, T.S.: Free-form image inpainting with gated convolution. In: Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision, Seoul, Korea, pp. 4471–4480, October 2019

    Google Scholar 

  36. Yu, T., et al.: Region normalization for image inpainting. In: Proceedings of the AAAI Conference on Artificial Intelligence, New York, NY, vol. 34, pp. 12733–12740, February 2020

    Google Scholar 

  37. Zhou, M., et al.: Nonparametric Bayesian dictionary learning for analysis of noisy and incomplete images. IEEE Trans. Image Process. 21(1), 130–144 (2011)

    Article  MathSciNet  Google Scholar 

Download references

Acknowledgements

This work has received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement no. 741215, ERC Advanced Grant INCOVID). We thank Dai et al. [11] for providing their reference dataset.

Author information

Authors and Affiliations

  1. Mathematical Image Analysis Group, Faculty of Mathematics and Computer Science, Saarland University, Campus E1.7, 66041, Saarbrücken, Germany

    Pascal Peter

Authors
  1. Pascal Peter
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Pascal Peter .

Editor information

Editors and Affiliations

  1. **amen University Malaysia, Sepang, Malaysia

    Han Wang

  2. Singapore Institute of Manufacturing Technology, Singapore, Singapore

    Wei Lin

  3. Charles Sturt University, Bathurst, NSW, Australia

    Paul Manoranjan

  4. Minjiang University, Fuzhou, China

    Guobao **ao

  5. Yau Lee Holdings Ltd., Hong Kong, Hong Kong

    Kap Luk Chan

  6. Tsinghua University, Bei**g, China

    **aonan Wang

  7. Nanyang Technological University, Singapore, Singapore

    Guiju **

  8. Nanyang Technological University, Singapore, Singapore

    Haoge Jiang

Rights and permissions

Reprints and permissions

Copyright information

© 2023 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Peter, P. (2023). A Wasserstein GAN for Joint Learning of Inpainting and Spatial Optimisation. In: Wang, H., et al. Image and Video Technology. PSIVT 2022. Lecture Notes in Computer Science, vol 13763. Springer, Cham. https://doi.org/10.1007/978-3-031-26431-3_11

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-26431-3_11

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-26430-6

  • Online ISBN: 978-3-031-26431-3

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation