Abstract
In 2020 and beyond, there are more and more opportunities to communicate with others while wearing a face mask. Since masks hide the mouth and facial muscles, it becomes more challenging to convey facial expressions to others while wearing a face mask. In this study, we propose using generative adversarial networks (GAN) to complement facial regions hidden by masks on images and videos. We defined the custom loss function that focuses on the error of the feature point coordinates of the face and the pixels in the masked region. As a result, we were able to generate higher-quality images than existing methods. Even when the input was video-based, our approach generated high-quality videos with fewer jittering and pixel errors than existing methods.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Myheritage deep nostalgia. https://www.myheritage.jp/deep-nostalgia
Anwar, A., Raychowdhury, A.: Masked face recognition for secure authentication. ar**v abs/2008.11104 (2020)
Cai, J., Han, H., Shan, S., Chen, X.: FCSR-GAN: joint face completion and super-resolution via multi-task learning. IEEE Trans. Biom. Behav. Identity Sci. 2(2), 109–121 (2020). https://doi.org/10.1109/TBIOM.2019.2951063
Goodfellow, I., et al.: Generative adversarial nets. In: Advances in Neural Information Processing Systems, vol. 27, pp. 2672–2680. Curran Associates, Inc. (2014)
Iizuka, S., Simo-Serra, E., Ishikawa, H.: Globally and locally consistent image completion. ACM Trans. Graph. 36(4), 107:1–107:14 (2017). (Proc. of SIGGRAPH 2017)
Isola, P., Zhu, J., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition, pp. 5967–5976 (2017). https://doi.org/10.1109/CVPR.2017.632
Karras, T., Laine, S., Aila, T.: A style-based generator architecture for generative adversarial networks. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4396–4405 (2019). https://doi.org/10.1109/CVPR.2019.00453
Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., Aila, T.: Analyzing and improving the image quality of stylegan. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8107–8116 (2020). https://doi.org/10.1109/CVPR42600.2020.00813
King, D.E.: Dlib-ml: a machine learning toolkit. J. Mach. Learn. Res. 10(60), 1755–1758 (2009)
Lundqvist, D., Flykt, A., Öhman, A.: The Karolinska Directed Emotional Faces. Karolinska Institutet (1998)
Mirza, M., Osindero, S.: Conditional generative adversarial nets. ar**v abs/1411.1784 (2014)
Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28
Telea, A.: An image inpainting technique based on the fast marching method. J. Graph. Tools 9, 23–34 (2004). https://doi.org/10.1080/10867651.2004.10487596
Tsujimura, Y., Nishimura, S., Iijima, A., Kobayashi, R., Miyajima, N.: Comparing different levels of smiling with and without a surgical mask. J. Compr. Nurs. Res. 19(2), 3–9 (2020). https://doi.org/10.14943/95250
Ud Din, N., Javed, K., Bae, S., Yi, J.: A novel GAN-based network for unmasking of masked face. IEEE Access 8, 44276–44287 (2020). https://doi.org/10.1109/ACCESS.2020.2977386
Wang, M., Wen, X., Hu, S.: Faithful face image completion for HMD occlusion removal. In: 2019 IEEE International Symposium on Mixed and Augmented Reality Adjunct, pp. 251–256 (2019). https://doi.org/10.1109/ISMAR-Adjunct.2019.00-36
**e, J., Xu, L., Chen, E.: Image denoising and inpainting with deep neural networks. In: Advances in Neural Information Processing Systems, vol. 25, pp. 341–349 (2012)
Yoshihashi, H., Ienaga, N., Sugimoto, M.: Gan-based face mask removal using facial landmarks and pixel errors in masked region. In: the 17th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications, vol. 5, pp. 125–133 (2022). https://doi.org/10.5220/0010827500003124
Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., Huang, T.: Free-form image inpainting with gated convolution. In: 2019 IEEE/CVF International Conference on Computer Vision, pp. 4470–4479 (2019). https://doi.org/10.1109/ICCV.2019.00457
Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., Huang, T.S.: Generative image inpainting with contextual attention. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5505–5514 (2018). https://doi.org/10.1109/CVPR.2018.00577
Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The unreasonable effectiveness of deep features as a perceptual metric. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 586–595 (2018). https://doi.org/10.1109/CVPR.2018.00068
Zhang, Z., Song, Y., Qi, H.: Age progression/regression by conditional adversarial autoencoder. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition, pp. 4352–4360 (2017). https://doi.org/10.1109/CVPR.2017.463
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Yoshihashi, H., Ienaga, N., Sugimoto, M. (2023). A Quantitative and Qualitative Analysis on a GAN-Based Face Mask Removal on Masked Images and Videos. In: de Sousa, A.A., et al. Computer Vision, Imaging and Computer Graphics Theory and Applications. VISIGRAPP 2022. Communications in Computer and Information Science, vol 1815. Springer, Cham. https://doi.org/10.1007/978-3-031-45725-8_3
Download citation
DOI: https://doi.org/10.1007/978-3-031-45725-8_3
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-45724-1
Online ISBN: 978-3-031-45725-8
eBook Packages: Computer ScienceComputer Science (R0)