SpringerLink
Log in

Influence of Color Spaces for Deep Learning Image Colorization

  • Reference work entry
  • First Online:
Handbook of Mathematical Models and Algorithms in Computer Vision and Imaging

  • 2088 Accesses

Abstract

Colorization is a process that converts a grayscale image into a colored one that looks as natural as possible. Over the years this task has received a lot of attention. Existing colorization methods rely on different color spaces: RGB, YUV, Lab, etc. In this chapter, we aim to study their influence on the results obtained by training a deep neural network, to answer the following question: “Is it crucial to correctly choose the right color space in deep learning-based colorization?” First, we briefly summarize the literature and, in particular, deep learning-based methods. We then compare the results obtained with the same deep neural network architecture with RGB, YUV, and Lab color spaces. Qualitative and quantitative analysis do not conclude similarly on which color space is better. We then show the importance of carefully designing the architecture and evaluation protocols depending on the types of images that are being processed and their specificities: strong/small contours, few/many objects, recent/archive images.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 899.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Hardcover Book
USD 949.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free ship** worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  • Afifi, M., Brubaker, M.A., Brown, M.S.: HistoGAN: controlling colors of gan-generated and real images via color histograms. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 7941–7950 (2021)

    Google Scholar 

  • Agustsson, E., Timofte, R.: Ntire 2017 challenge on single image super-resolution: dataset and study. In: Conference on Computer Vision and Pattern Recognition Workshops, pp. 126–135 (2017)

    Google Scholar 

  • Antic, J.: Deoldify (2019). https://github.com/jantic/DeOldify

    Google Scholar 

  • Anwar, S., Tahir, M., Li, C., Mian, A., Khan, F.S., Muzaffar, A.W.: Image colorization: a survey and dataset (2020). ar**v preprint ar**v:2008.10774

    Google Scholar 

  • Arbelot, B., Vergne, R., Hurtut, T., Thollot, J.: Automatic texture guided color transfer and colorization. In: Expressive, Elsevier, pp. 21–32 (2016)

    Google Scholar 

  • Arbelot, B., Vergne, R., Hurtut, T., Thollot, J.: Local texture-based color transfer and colorization. Comput. Graph. 62, 15–27 (2017)

    Article  Google Scholar 

  • Arjovsky, M., Chintala, S., Bottou, L.: Wasserstein generative adversarial networks. In: International Conference on Machine Learning, vol. 70, pp. 214–223 (2017)

    Google Scholar 

  • Bugeau, A., Ta, V.-T.: Patch-based image colorization. In: International Conference on Pattern Recognition, pp. 3058–3061 (2012)

    Google Scholar 

  • Bugeau, A., Ta, V.-T., Papadakis, N.: Variational exemplar-based image colorization. IEEE Trans. Image Process. 23(1), 298–307 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  • Cao, Y., Zhou, Z., Zhang, W., Yu, Y.: Unsupervised diverse colorization via generative adversarial networks. In: Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 151–166 (2017)

    Google Scholar 

  • Charpiat, G., Hofmann, M., Schölkopf, B.: Automatic image colorization via multimodal predictions. In: European Conference on Computer Vision, pp. 126–139 (2008)

    Google Scholar 

  • Cheng, Z., Yang, Q., Sheng, B.: Deep colorization. In: IEEE International Conference on Computer Vision, pp. 415–423 (2015)

    Google Scholar 

  • Chia, A.Y.-S., Zhuo, S., Gupta, R.K., Tai, Y.-W., Cho, S.-Y., Tan, P., Lin, S.: Semantic colorization with internet images. In: ACM SIGGRAPH ASIA (2011)

    Google Scholar 

  • Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., Fei-Fei, L.: Imagenet: a large-scale hierarchical image database. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255 (2009)

    Google Scholar 

  • Deshpande, A., Rock, J., Forsyth, D.: Learning large-scale automatic image colorization. In: IEEE International Conference on Computer Vision (2015)

    Google Scholar 

  • Deshpande, A., Lu, J., Yeh, M.-C., ** Chong, M., Forsyth, D.: Learning diverse image colorization. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 6837–6845 (2017)

    Google Scholar 

  • Di Blasi, G., Reforgiato, D.: Fast colorization of gray images. In: Eurographics Italian, Eurographics Association (2003)

    Google Scholar 

  • Ding, X., Xu, Y., Deng, L., Yang, X.: Colorization using quaternion algebra with automatic scribble generation. In: Advances in Multimedia Modeling, pp. 103–114 (2012)

    Google Scholar 

  • Ding, K., Ma, K., Wang, S., Simoncelli, E.P.: Comparison of full-reference image quality models for optimization of image processing systems. Int. J. Comput. Vis. 129(4), 1258–1281 (2021)

    Article  Google Scholar 

  • Dowson, D., Landau, B.: The Fréchet distance between multivariate normal distributions. J. Multivar. Anal. 12(3), 450–455 (1982)

    Article  MATH  Google Scholar 

  • Drew, M.S., Finlayson, G.D.: Improvement of colorization realism via the structure tensor. Int. J. Image Graph. 11(4), 589–609 (2011)

    Article  MathSciNet  Google Scholar 

  • Ebner, M.: Color Constancy, vol. 7. Wiley, Hoboken (2007)

    MATH  Google Scholar 

  • Efros, A., Leung, T.: Texture synthesis by non-parametric sampling. In: IEEE International Conference on Computer Vision, pp. 1033–1038 (1999)

    Google Scholar 

  • Fairchild, M.D. : Color Appearance Models. Wiley, Hoboken (2013)

    Book  Google Scholar 

  • Fang, F., Wang, T., Zeng, T., Zhang, G.: A superpixel-based variational model for image colorization. IEEE Trans. Vis. Comput. Graph. 26(10), 2931–2943 (2019)

    Article  Google Scholar 

  • Gatys, L.A., Ecker, A.S., Bethge, M.: Image style transfer using convolutional neural networks. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 2414–2423 (2016a)

    Google Scholar 

  • Gatys, L.A., Ecker, A.S., Bethge, M.: A neural algorithm of artistic style. J. Vis. 16(12), 326 (2016b)

    Article  Google Scholar 

  • Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. In: Advances in Neural Information Processing Systems (2014)

    Google Scholar 

  • Guadarrama, S., Dahl, R., Bieber, D., Norouzi, M., Shlens, J., Murphy, K.: Pixcolor: pixel recursive colorization. In: British Machine Vision Conference (2017)

    Google Scholar 

  • Gu, S., Timofte, R., Zhang, R.: Ntire 2019 challenge on image colorization: report. In: Conference on Computer Vision and Pattern Recognition Workshops (2019)

    Google Scholar 

  • Gupta, R.K., Chia, A.Y.-S., Rajan, D., Ng, E.S., Zhiyong, H.: Image colorization using similar images. In: ACM International Conference on Multimedia, pp. 369–378 (2012)

    Google Scholar 

  • He, K., Gkioxari, G., Dollár, P., Girshick, R.: Mask R-CNN. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2961–2969 (2017)

    Google Scholar 

  • He, M., Chen, D., Liao, J., Sander, P.V., Yuan, L.: Deep exemplar-based colorization. ACM Trans. Graph. 37(4), 1–16 (2018)

    Google Scholar 

  • Heu, J., Hyun, D.-Y., Kim, C.-S., Lee, S.-U.: Image and video colorization based on prioritized source propagation. In: IEEE International Conference on Image Processing, pp. 465–468 (2009)

    Google Scholar 

  • Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S.: GANs trained by a two time-scale update rule converge to a local Nash equilibrium. In: Advances in Neural Information Processing Systems, vol. 30 (2017)

    Google Scholar 

  • Ho, J., Kalchbrenner, N., Weissenborn, D., Salimans, T.: Axial attention in multidimensional transformers. ar**v preprint ar**v:1912.12180 (2019)

    Google Scholar 

  • Huang, Y.-C., Tung, Y.-S., Chen, J.-C., Wang, S.-W., Wu, J.-L.: An adaptive edge detection based colorization algorithm and its applications. In: ACM International Conference on Multimedia, pp. 351–354 (2005)

    Google Scholar 

  • Huang, G.B., Ramesh, M., Berg, T., Learned-Miller, E.: Labeled faces in the wild: a database for studying face recognition in unconstrained environments. Technical Report 07-49, University of Massachusetts, Amherst (2007)

    Google Scholar 

  • Iizuka, S., Simo-Serra, E., Ishikawa, H.: Let there be color!: joint end-to-end learning of global and local image priors for automatic image colorization with simultaneous classification. ACM Trans. Graph. 35(4), 1–11 (2016)

    Article  Google Scholar 

  • Irony, R., Cohen-Or, D., Lischinski, D.: Colorization by example. In: Eurographics Conference on Rendering Techniques (2005)

    Google Scholar 

  • Isola, P., Zhu, J.-Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1125–1134 (2017)

    Google Scholar 

  • Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: European Conference on Computer Vision, pp. 694–711 (2016)

    Google Scholar 

  • Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., Aila, T.: Analyzing and improving the image quality of stylegan. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 8110–8119 (2020)

    Google Scholar 

  • Kawulok, M., Kawulok, J., Smolka, B.: Discriminative textural features for image and video colorization. IEICE Trans. Inf. Syst. 95-D(7), 1722–1730 (2012)

    Article  Google Scholar 

  • Kong, G., Tian, H., Duan, X., Long, H.: Adversarial edge-aware image colorization with semantic segmentation. IEEE Access 9, 28194–28203 (2021)

    Article  Google Scholar 

  • Krizhevsky, A., Hinton, G., et al.: Learning multiple layers of features from tiny images. Technical report, University of Toronto (2009)

    Google Scholar 

  • Kumar, M., Weissenborn, D., Kalchbrenner, N.: Colorization transformer. ar**v preprint ar**v:2102.04432 (2021)

    Google Scholar 

  • Lagodzinski, P., Smolka, B.: Digital image colorization based on probabilistic distance transformation. In: 50th International Symposium ELMAR, vol. 2, pp. 495–498 (2008)

    Google Scholar 

  • Larsson, G., Maire, M., Shakhnarovich, G.: Learning representations for automatic colorization. In: European Conference on Computer Vision, pp. 577–593 (2016)

    Google Scholar 

  • Levin, A., Lischinski, D., Weiss, Y.: Colorization using optimization. ACM Trans. Graph. 23(3), 689–694 (2004)

    Article  Google Scholar 

  • Lézoray, O., Ta, V.-T., Elmoataz, A.: Nonlocal graph regularization for image colorization. In: International Conference on Pattern Recognition, pp. 1–4 (2008)

    Google Scholar 

  • Li, B., Lai, Y.-K., Rosin, P.L.: Example-based image colorization via automatic feature selection and fusion. Neurocomputing 266, 687–698 (2017a)

    Article  Google Scholar 

  • Li, B., Zhao, F., Su, Z., Liang, X., Lai, Y.-K., Rosin, P.L.: Example-based image colorization using locality consistent sparse representation. IEEE Trans. Image Process. 26(11), 5188–5202 (2017b)

    Article  MathSciNet  MATH  Google Scholar 

  • Li, B., Lai, Y.-K., John, M., Rosin, P.L.: Automatic example-based image colorization using location-aware cross-scale matching. IEEE Trans. Image Process. 28(9), 4606–4619 (2019)

    Article  MathSciNet  MATH  Google Scholar 

  • Li, B., Lai, Y.-K., Rosin, P.L.: A review of image colourisation. In: Handbook of Pattern Recognition and Computer Vision, p. 139. World Scientific, Singapore (2020)

    Google Scholar 

  • Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft COCO: common objects in context. In: European Conference on Computer Vision, pp. 740–755 (2014)

    Google Scholar 

  • Ling, Y., Au, O.C., Pang, J., Zeng, J., Yuan, Y., Zheng, A.: Image colorization via color propagation and rank minimization. In: IEEE International Conference on Image Processing, pp. 4228–4232 (2015)

    Google Scholar 

  • Liu, S., Zhang, X.: Automatic grayscale image colorization using histogram regression. Pattern Recogn. Lett. 33(13), 1673–1681 (2012)

    Article  Google Scholar 

  • Luan, Q., Wen, F., Cohen-Or, D., Liang, L., Xu, Y.-Q., Shum, H.-Y.: Natural image colorization. In: Eurographics Conference on Rendering Techniques, pp. 309–320 (2007)

    Google Scholar 

  • Luo, X., Zhang, X., Yoo, P., Martin-Brualla, R., Lawrence, J., Seitz, S.M.: Time-travel rephotography. ar**v preprint ar**v:2012.12261 (2020)

    Google Scholar 

  • Mouzon, T., Pierre, F., Berger, M.-O.: Joint CNN and variational model for fully-automatic image colorization. In: Scale Space and Variational Methods in Computer Vision, pp. 535–546 (2019)

    Google Scholar 

  • Nazeri, K., Ng, E., Ebrahimi, M.: Image colorization using generative adversarial networks. In: International Conference on Articulated Motion and Deformable Objects, pp. 85–94 (2018)

    Google Scholar 

  • Oord, A.V.D., Kalchbrenner, N., Vinyals, O., Espeholt, L., Graves, A., Kavukcuoglu, K.: Conditional image generation with PixelCNN decoders. In: Advances in Neural Information Processing Systems (2016)

    Google Scholar 

  • Pang, J., Au, O.C., Tang, K., Guo, Y.: Image colorization using sparse representation. In: IEEE International Conference on Acoustics, Speech, and Signal Processing, pp. 1578–1582 (2013)

    Google Scholar 

  • Pierre, F., Aujol, J.-F.: Recent approaches for image colorization. In: Handbook of Mathematical Models and Algorithms in Computer Vision and Imaging, springer (2020)

    Google Scholar 

  • Pierre, F., Aujol, J.-F., Bugeau, A., Ta, V.-T.: A unified model for image colorization. In: European Conference on Computer Vision Workshops, pp. 297–308 (2014)

    Google Scholar 

  • Pierre, F., Aujol, J.-F., Bugeau, A., Papadakis, N., Ta, V.-T.: Luminance-chrominance model for image colorization. SIAM J. Imaging Sci. 8(1), 536–563 (2015a)

    Article  MathSciNet  MATH  Google Scholar 

  • Pierre, F., Aujol, J.-F., Bugeau, A., Ta, V.-T.: Luminance-Hue Specification in the RGB Space. In: Scale Space and Variational Methods in Computer Vision, pp. 413–424. Springer, Cham (2015b)

    Google Scholar 

  • Pucci, R., Micheloni, C., Martinel, N.: Collaborative image and object level features for image colourisation. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 2160–2169 (2021)

    Google Scholar 

  • Radford, A., Metz, L., Chintala, S.: Unsupervised representation learning with deep convolutional generative adversarial networks. In: International Conference on Learning Representations (2016)

    Google Scholar 

  • Ren, X., Malik, J.: Learning a classification model for segmentation. In: IEEE International Conference on Computer Vision, pp. 10–17 (2003)

    Google Scholar 

  • Riba, E., Mishkin, D., Ponsa, D., Rublee, E., Bradski, G.: Kornia: an open source differentiable computer vision library for PyTorch. In: Winter Conference on Applications of Computer Vision, pp. 3674–3683 (2020)

    Google Scholar 

  • Royer, A., Kolesnikov, A., Lampert, C.H.: Probabilistic image colorization. In: British Machine Vision Conference (2017)

    Google Scholar 

  • Salimans, T., Karpathy, A., Chen, X., Kingma, D.P.: PixelCNN++: improving the PixelCNN with discretized logistic mixture likelihood and other modifications. ar**v preprint ar**v:1701.05517 (2017)

    Google Scholar 

  • Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: International Conference on Learning Representations (2015)

    Google Scholar 

  • Su, J.-W., Chu, H.-K., Huang, J.-B.: Instance-aware image colorization. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 7968–7977 (2020)

    Google Scholar 

  • Tai, Y.-W., Jia, J., Tang, C.-K.: Local color transfer via probabilistic segmentation by expectation-maximization. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 747–754 (2005)

    Google Scholar 

  • Vitoria, P., Raad, L., Ballester, C.: ChromaGAN: adversarial picture colorization with semantic class distribution. In: Winter Conference on Applications of Computer Vision, pp. 2445–2454 (2020)

    Google Scholar 

  • Wan, S., **a, Y., Qi, L., Yang, Y.-H., Atiquzzaman, M.: Automated colorization of a grayscale image with seed points propagation. IEEE Trans. Multimedia 22(7), 1756–1768 (2020a)

    Article  Google Scholar 

  • Wan, Z., Zhang, B., Chen, D., Zhang, P., Chen, D., Liao, J., Wen, F.: Bringing old photos back to life. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 2747–2757 (2020b)

    Google Scholar 

  • Wang, S., Zhang, Z.: Colorization by matrix completion. In: AAAI Conference on Artificial Intelligence (2012)

    Google Scholar 

  • Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)

    Article  Google Scholar 

  • Welsh, T., Ashikhmin, M., Mueller, K. Transferring color to greyscale images. ACM Trans. Graph. 21(3), 277–280 (2002)

    Article  Google Scholar 

  • **ao, J., Hays, J., Ehinger, K.A., Oliva, A., Torralba, A.: Sun database: large-scale scene recognition from abbey to zoo. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 3485–3492 (2010)

    Google Scholar 

  • Yao, Q., James, T.K.: Colorization by patch-based local low-rank matrix completion. In: AAAI Conference on Artificial Intelligence (2015)

    Google Scholar 

  • Yatziv, L., Sapiro, G.: Fast image and video colorization using chrominance blending. IEEE Trans. Image Process. 15(5), 1120–1129 (2006)

    Article  Google Scholar 

  • Yoo, S., Bahng, H., Chung, S., Lee, J., Chang, J., Choo, J.: Coloring with limited data: few-shot colorization via memory augmented networks. In: IEEE Conference on Computer Vision and Pattern Recognition (2019)

    Google Scholar 

  • Yu, F., Seff, A., Zhang, Y., Song, S., Funkhouser, T., **ao, J.: LSUN: construction of a large-scale image dataset using deep learning with humans in the loop. ar**v preprint ar**v:1506.03365 (2015)

    Google Scholar 

  • Zhang, R., Isola, P., Efros, A.A.: Colorful image colorization. In: European Conference on Computer Vision, pp. 649–666 (2016)

    Google Scholar 

  • Zhang, R., Zhu, J.-Y., Isola, P., Geng, X., Lin, A.S., Yu, T., Efros, A.A.: Real-time user-guided image colorization with learned deep priors. ACM Trans. Graph. 36, 1–11 (2017)

    Google Scholar 

  • Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The unreasonable effectiveness of deep features as a perceptual metric. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 586–595 (2018)

    Google Scholar 

  • Zhou, B., Lapedriza, A., Khosla, A., Oliva, A., Torralba, A.: Places: a 10 million image database for scene recognition. IEEE Trans. Pattern Anal. Mach. Intell. 40(6), 1452–1464 (2017)

    Article  Google Scholar 

Download references

Acknowledgements

This study has been carried out with financial support from the French Research Agency through the PostProdLEAP project (ANR-19-CE23-0027-01) and from the EU Horizon 2020 research and innovation program NoMADS (Marie Skodowska-Curie grant agreement No 777826). This chapter was written together with another chapter of the current handbook, Chap. 21, “Analysis of Different Losses for Deep Learning Image Colorization”.All authors have contributed to both chapters.

Author information

Authors and Affiliations

  1. LaBRI, CNRS, UMR5800, Univ. Bordeaux, F-33400 Talence, France

    Aurélie Bugeau

  2. Institut universitaire de France (IUF), Paris, France

    Aurélie Bugeau

  3. Univ. Bordeaux, CNRS, IMS UMR5251, Bordeaux INP, F-33400 Talence, France

    Rémi Giraud

  4. LIGM, CNRS, Univ Gustave Eiffel, F-77454 Marne-la-Vallée, France

    Lara Raad

Authors
  1. Aurélie Bugeau
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Rémi Giraud
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Lara Raad
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Aurélie Bugeau .

Editor information

Editors and Affiliations

  1. Department of Mathematical Sciences, The University of Liverpool, Liverpool, UK

    Ke Chen

  2. Department of Applied Mathematics and Theoretical Physics, University of Cambridge, Cambridge, UK

    Carola-Bibiane Schönlieb

  3. Hong Kong Center for Cerebrocardiovascular Health Engineering (COCHE), Shatin, Hong Kong

    Xue-Cheng Tai

  4. Department of Applied Mathematics and Statistics, Johns Hopkins University, Baltimore, MD, USA

    Laurent Younes

Rights and permissions

Reprints and permissions

Copyright information

© 2023 Springer Nature Switzerland AG

About this entry

Check for updates. Verify currency and authenticity via CrossMark

Cite this entry

Bugeau, A., Giraud, R., Raad, L. (2023). Influence of Color Spaces for Deep Learning Image Colorization. In: Chen, K., Schönlieb, CB., Tai, XC., Younes, L. (eds) Handbook of Mathematical Models and Algorithms in Computer Vision and Imaging. Springer, Cham. https://doi.org/10.1007/978-3-030-98661-2_125

Download citation

Publish with us

Policies and ethics

Search

Navigation