SpringerLink
Log in

Blind Image Inpainting with Sparse Directional Filter Dictionaries for Lightweight CNNs

  • Published:
Journal of Mathematical Imaging and Vision Aims and scope Submit manuscript

Abstract

Blind inpainting algorithms based on deep learning architectures have shown a remarkable performance in recent years, typically outperforming model-based methods both in terms of image quality and run time. However, neural network strategies typically lack a theoretical explanation, which contrasts with the well-understood theory underlying model-based methods. In this work, we leverage the advantages of both approaches by integrating theoretically founded concepts from transform domain methods and sparse approximations into a CNN-based approach for blind image inpainting. To this end, we present a novel strategy to learn convolutional kernels that applies a specifically designed filter dictionary whose elements are linearly combined with trainable weights. Numerical experiments demonstrate the competitiveness of this approach. Our results show not only an improved inpainting quality compared to conventional CNNs but also significantly faster network convergence within a lightweight network design. Our code is available at https://github.com/cv-stuttgart/SDPF_Blind-Inpainting.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Notes

  1. Retraining the network became necessary since the pre-trained VCNet model provided by the authors in [35] did not achieve comparable results even after tuning it on our data set for 100,000 additional iterations.

References

  1. Abadi, M., Barham, P., Chen, J., et al.: TensorFlow: a system for large-scale machine learning. OSDI 16, 265–283 (2016)

    Google Scholar 

  2. Aharon, M., Elad, M., Bruckstein, A.: K-SVD: an algorithm for designing overcomplete dictionaries for sparse representation. IEEE Trans. Signal Process. 54(11), 4311–4322 (2006)

    Article  MATH  Google Scholar 

  3. Arias, P., Facciolo, G., Caselles, V., et al.: A variational framework for exemplar-based image inpainting. Int. J. Comput. Vis. 93(3), 319–347 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  4. Cai, J.F., Dong, B., Osher, S., et al.: Image restoration: total variation, wavelet frames, and beyond. J. Am. Math. Soc. 25(4), 1033–1089 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  5. Cai, N., Su, Z., Lin, Z., et al.: Blind inpainting using the fully convolutional neural network. Vis. Comput. 33(2), 249–261 (2017)

    Article  Google Scholar 

  6. Candès, A., Donoho, D.: Curvelets a surprisingly effective nonadaptive representation for objects with edges. In: Schumaker, L. (ed.) Curves and Surfaces, pp. 105–120. Vanderbilt University Press, Nashville (2000)

    Google Scholar 

  7. Candès, E.J., Donoho, D.L.: New tight frames of curvelets and optimal representations of objects with piecewise C2 singularities. Commun. Pure Appl. Math. 57(2), 219–266 (2004)

    Article  MATH  Google Scholar 

  8. Chan, T.F., Shen, J.: Variational image inpainting. Commun. Pure Appl. Math. 58(5), 579–619 (2005)

    Article  MathSciNet  MATH  Google Scholar 

  9. Chan, T.F., Shen, J., Zhou, H.M.: Total variation wavelet inpainting. J. Math. Imaging Vis. 25(1), 107–125 (2006)

    Article  MathSciNet  MATH  Google Scholar 

  10. Chaudhury, S., Roy, H.: Can fully convolutional networks perform well for general image restoration problems? In: Proceedings of International Conference on Machine Vision Applications (MVA), pp. 254–257 (2017)

  11. Dong, B., Ji, H., Li, J., et al.: Wavelet frame based blind image inpainting. Appl. Comput. Harmon. Anal. 32(2), 268–279 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  12. Donoho, D.L., Vetterli, M., DeVore, R.A., et al.: Data compression and harmonic analysis. IEEE Trans. Inf. Theory 44(6), 2435–2476 (1998)

  13. Elad, M., Starck, J.L., Querre, P., et al.: Simultaneous cartoon and texture image inpainting using morphological component analysis (MCA). Appl. Comput. Harmon. Anal. 19(3), 340–358 (2005)

    Article  MathSciNet  MATH  Google Scholar 

  14. Esedoglu, S., Shen, J.: Digital inpainting based on the Mumford–Shah–Euler image model. Eur. J. Appl. Math. 13(4), 353–370 (2002)

    Article  MathSciNet  MATH  Google Scholar 

  15. Guo, K., Labate, D.: Optimally sparse multidimensional representation using shearlets. SIAM J. Math. Anal. 39, 298–318 (2007)

    Article  MathSciNet  MATH  Google Scholar 

  16. Guo, K., Labate, D., Ayllon, J.P.R.: Image inpainting using sparse multiscale representations: image recovery performance guarantees. Appl. Comput. Harmon. Anal. 49(2), 343–380 (2020)

    Article  MathSciNet  MATH  Google Scholar 

  17. Jacobsen, J.H., Van Gemert, J., Lou, Z., et al.: Structured receptive fields in CNNs. In: Proceedings of Conference on Computer Vision and Pattern Recognition, pp. 2610–2619 (2016)

  18. Karantzas, N., Atreas, N., Papadakis, M., et al.: On the design of multi-dimensional compactly supported Parseval framelets with directional characteristics. Linear Algebra Appl. 582, 1–36 (2019)

    Article  MathSciNet  MATH  Google Scholar 

  19. Köhler, R., Schuler, C., Schölkopf, B., et al.: Mask-specific inpainting with deep neural networks. In: Pattern Recognition, vol. 8753, pp. 523–534. Springer, Berlin (2014)

  20. King, E.J., Kutyniok, G., Zhuang, X.: Analysis of inpainting via clustered sparsity and microlocal analysis. J. Math. Imaging Vis. 48(2), 205–234 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  21. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: Proceedings of International Conference on Learning Representations (2015)

  22. Kutyniok, G., Lim, W.Q.: Compactly supported shearlets are optimally sparse. J. Approx. Theory 163, 1564–1589 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  23. Labate, D., Lim, W.Q., Kutyniok, G., et al.: Sparse multidimensional representation using shearlets. In: Wavelets XI, International Society for Optics and Photonics, p. 59140U (2005)

  24. Labate, D., Safari, K., Karantzas, N., et al.: Structured receptive field networks and applications to hyperspectral image classification. In: Wavelets and Sparsity XVIII, International Society for Optics and Photonics, pp. 218–226 (2019)

  25. Liu, G., Reda, F.A., Shih, K.J., et al.: Image inpainting for irregular holes using partial convolutions. In: Proceedings of European Conference on Computer Vision, pp. 85–100 (2018)

  26. Liu, Y., Pan, J., Su, Z.: Deep blind image inpainting. In: Proceedings of International Conference on Intelligent Science and Big Data Engineering, pp. 128–141 (2019)

  27. Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: Proceedings of Conference on Computer Vision and Pattern Recognition, pp. 3431–3440 (2015)

  28. Mairal, J., Elad, M., Sapiro, G.: Sparse representation for color image restoration. IEEE Trans. Image Process. 17(1), 53–69 (2008)

  29. Mairal, J., Sapiro, G., Elad, M.: Learning multiscale sparse representations for image and video restoration. Multiscale Model. Simul. 7(1), 214–241 (2008)

  30. Mallat, S.: A Wavelet Tour of Signal Processing. Elsevier, Amsterdam (1999)

    MATH  Google Scholar 

  31. Pathak, D., Krahenbuhl, P., Donahue, J., et al.: Context encoders: feature learning by inpainting. In: Proceedings of Conference on Computer Vision and Pattern Recognition, pp. 2536–2544 (2016)

  32. Shen, J., Chan, T.F.: Mathematical models for local nontexture inpaintings. SIAM J. Appl. Math. 62(3), 1019–1043 (2002)

    Article  MathSciNet  MATH  Google Scholar 

  33. Shen, L., Xu, Y., Zeng, X.: Wavelet inpainting with the \(\ell _0\) sparse regularization. Appl. Comput. Harmon. Anal. 41(1), 26–53 (2016)

    Article  MathSciNet  Google Scholar 

  34. Szegedy, C., Liu, W., Jia, Y., et al.: Going deeper with convolutions. In: Proceedings of Conference on Computer Vision and Pattern Recognition, pp. 1–9 (2015)

  35. Wang, Y., Chen, Y.C., Tao, X., et al.: VCnet: a robust approach to blind image inpainting. In: Proceedings of European Conference on Computer Vision, pp. 752–768 (2020)

  36. **e, J., Xu, L., Chen, E.: Image denoising and inpainting with deep neural networks. In: Proceedings of Advances in Neural Information Processing Systems, pp. 341–349 (2012)

  37. Yi, Z., Tang, Q., Azizi, S., et al.: Contextual residual aggregation for ultra high-resolution image inpainting. In: Proceedings of Conference on Computer Vision and Pattern Recognition, pp. 7508–7517 (2020)

  38. Yu, J., Lin, Z., Yang, J., et al.: Generative image inpainting with contextual attention. In: Proceedings of Computer Vision and Pattern Recognition, pp. 5505–5514 (2018)

  39. Yu, J., Lin, Z., Yang, J., et al.: Free-form image inpainting with gated convolution. In: Proceedings of International Conference on Computer Vision, pp. 4471–4480 (2019)

  40. Zhang, Y., Tiňo, P., Leonardis, A., et al.: A survey on neural network interpretability. IEEE Trans. Emerg. Top. Comput. Intell. 5(5), 726–742 (2021)

    Article  Google Scholar 

  41. Zhou, B., Lapedriza, A., Khosla, A., et al.: Places: a 10 million image database for scene recognition. IEEE Trans. Pattern Anal. Mach. Intell. 40(6), 1452–1464 (2018)

    Article  Google Scholar 

Download references

Acknowledgements

Authors thank Prof. Dr. D. Göddeke for enabling this collaboration and his constant support and advise, Prof. Dr. B. Haasdonk and Prof. Dr. C. Rohde for providing the handwritten notes, and K. Safari for his help with the implementation. DL acknowledges support by NSF-DMS 1720487, 1720452 and HPE DSI/IT at UH. JS acknowledges partial funding by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) - Project-ID 251654672 - TRR 161 (Project B04) and support by the International Max Planck Research School for Intelligent Systems (IMPRS-IS).

Author information

Authors and Affiliations

  1. Institute for Visualization and Interactive Systems (VIS), University of Stuttgart, Universitätsstraße 38, 70569, Stuttgart, Germany

    Jenny Schmalfuss, Erik Scheurer & Andrés Bruhn

  2. Department of Mathematics, University of Houston, 651 Phillip G Hoffman, Houston, 77204, TX, USA

    Heng Zhao, Nikolaos Karantzas & Demetrio Labate

Authors
  1. Jenny Schmalfuss
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Erik Scheurer
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Heng Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Nikolaos Karantzas
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Andrés Bruhn
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Demetrio Labate
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Demetrio Labate.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Schmalfuss, J., Scheurer, E., Zhao, H. et al. Blind Image Inpainting with Sparse Directional Filter Dictionaries for Lightweight CNNs. J Math Imaging Vis 65, 323–339 (2023). https://doi.org/10.1007/s10851-022-01119-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10851-022-01119-6

Keywords

Search

Navigation