Abstract
Specular highlight detection and removal is a fundamental problem in computer vision and image processing. In this paper, we present an efficient end-to-end deep learning model for automatically detecting and removing specular highlights in a single image. In particular, an encoder—decoder network is utilized to detect specular highlights, and then a novel Unet-Transformer network performs highlight removal; we append transformer modules instead of feature maps in the Unet architecture. We also introduce a highlight detection module as a mask to guide the removal task. Thus, these two networks can be jointly trained in an effective manner. Thanks to the hierarchical and global properties of the transformer mechanism, our framework is able to establish relationships between continuous self-attention layers, making it possible to directly model the map** between the diffuse area and the specular highlight area, and reduce indeterminacy within areas containing strong specular highlight reflection. Experiments on public benchmark and real-world images demonstrate that our approach outperforms state-of-the-art methods for both highlight detection and removal tasks.
Article PDF
Similar content being viewed by others
Avoid common mistakes on your manuscript.
References
Arbeláez, P.; Maire, M.; Fowlkes, C.; Malik, J. Contour detection and hierarchical image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 33, No. 5, 898–916, 2011.
Tao, M. W.; Su, J. C.; Wang, T. C.; Malik, J.; Ramamoorthi, R. Depth estimation and specular removal for glossy surfaces using point and line consistency with light-field cameras. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 38, No. 6, 1155–1169, 2016.
Ramadan, H.; Lachqar, C.; Tairi, H. A survey of recent interactive image segmentation methods. Computational Visual Media Vol. 6, No. 4, 355–384, 2020.
Khanian, M.; Boroujerdi, A. S.; Breuß, M. Photometric stereo for strong specular highlights. Computational Visual Media Vol. 4, No. 1, 83–102, 2018.
Cui, Z. P.; Gu, J. W.; Shi, B. X.; Tan, P.; Kautz, J. Polarimetric multi-view stereo. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 369–378, 2017.
Xue, M. L.; Shivakumara, P.; Zhang, C.; **ao, Y.; Lu, T.; Pal, U.; Lopresti, D.; Yang, Z. Arbitrarily-oriented text detection in low light natural scene images. IEEE Transactions on Multimedia Vol. 23, 2706–2720, 2021.
Osadchy, M.; Jacobs, D. W.; Ramamoorthi, R. Using specularities for recognition. In: Proceedings of the 9th IEEE International Conference on Computer Vision, 1512–1519, 2003.
Li, R. Y.; Pan, J. J.; Si, Y. Q.; Yan, B.; Hu, Y.; Qin, H. Specular reflections removal for endoscopic image sequences with adaptive-RPCA decomposition. IEEE Transactions on Medical Imaging Vol. 39, No. 2, 328–340, 2020.
Artusi, A.; Banterle, F.; Chetverikov, D. A survey of specularity removal methods. Computer Graphics Forum Vol. 30, No. 8, 2208–2230, 2011.
Shafer, S. A. Using color to separate reflection components. Color Research & Application Vol. 10, No. 4, 210–218, 1985.
Shen, H. L.; Zhang, H. G.; Shao, S. J.; **n, J. H. Chromaticity-based separation of reflection components in a single image. Pattern Recognition Vol. 41, No. 8, 2461–2469, 2008.
Brainard, D. H.; Freeman, W. T. Bayesian color constancy Journal of the Optical Society of America A Vol. 14, No. 7, 1393–1411, 1997.
Finlayson, G. D.; Hordley, S. D.; HubeL, P. M. Color by correlation: A simple, unifying framework for color constancy. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 23, No. 11, 1209–1221, 2001.
Tan, R. T.; Nishino, K.; Ikeuchi, K. Color constancy through inverse-intensity chromaticity space. Journal of the Optical Society of America A Vol. 21, No. 3, 321–334, 2004.
Shi, J.; Dong, Y.; Su, H.; Yu, S. X. Learning non-Lambertian object intrinsics across ShapeNet categories. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 5844–5853, 2017.
Lin, J.; El Amine Seddik, M.; Tamaazousti, M.; Tamaazousti, Y.; Bartoli, A. Deep multi-class adversarial specularity removal. In: Image Analysis. Lecture Notes in Computer Science, Vol. 11482. Felsberg, M.; Forssén, P. E.; Sintorn, I. M.; Unger, J. Eds. Springer Cham, 3–15, 2019.
Muhammad, S.; Dailey, M. N.; Farooq, M.; Majeed, M. F.; Ekpanyapong, M. Spec-Net and Spec-CGAN: Deep learning models for specularity removal from faces. Image and Vision Computing Vol. 93, 103823, 2020.
Xu, Y. F.; Wei, H. P.; Lin, M. X.; Deng, Y. Y.; Sheng, K. K.; Zhang, M. D.; Tang, F.; Dong, W.; Huang, F.; Xu, C. Transformers in computational visual media: A survey. Computational Visual Media Vol. 8, No. 1, 33–62, 2022.
Liu, Z.; Lin, Y. T.; Cao, Y.; Hu, H.; Wei, Y. X.; Zhang, Z.; Lin, S.; Guo, B. Swin transformer: Hierarchical vision transformer using shifted windows. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, 9992–10002, 2021.
Maloney, L. T.; Wandell, B. A. Color constancy: A method for recovering surface spectral reflectance. Journal of the Optical Society of America A Vol. 3, No. 1, 29–33, 1986.
Park, J. B.; Kak, A. C. A truncated least squares approach to the detection of specular highlights in color images. In: Proceedings of the IEEE International Conference on Robotics and Automation, 1397–1403, 2003.
Meslouhi, O.; Kardouchi, M.; Allali, H.; Gadi, T.; Benkaddour, Y. Automatic detection and inpainting of specular reflections for colposcopic images. Central European Journal of Computer Science Vol. 1, No. 3, 341–354, 2011.
Zhang, W. M.; Zhao, X.; Morvan, J. M.; Chen, L. M. Improving shadow suppression for illumination robust face recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 41, No. 3, 611–624, 2019.
Hoyer, P. O. Non-negative matrix factorization with sparseness constraints. Journal of Machine Learning Research Vol. 5, 1457–1469, 2004.
Fu, G.; Zhang, Q.; Lin, Q. F.; Zhu, L.; **ao, C. X. Learning to detect specular highlights from real-world images. In: Proceedings of the 28th ACM International Conference on Multimedia, 1873–1881, 2020.
Tan, P.; Quan, L.; Lin, S. Separation of highlight reflections on textured surfaces. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 1855–1860, 2006.
Shen, H.-L.; Cai, Q.-Y. Simple and efficient method for specularity removal in an image. Applied Optics Vol. 48, No. 14, 2711, 2009.
Shen, H. L.; Zheng, Z. H. Real-time highlight removal using intensity ratio. Applied Optics Vol. 52, No. 19, 4483–4493, 2013.
Yang, J. W.; Liu, L. X.; Li, S. Z. Separating specular and diffuse reflection components in the HSI color space. In: Proceedings of the IEEE International Conference on Computer Vision Workshops, 891–898, 2013.
Yang, Q. X.; Tang, J. H.; Ahuja, N. Efficient and robust specular highlight removal. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 37, No. 6, 1304–1311, 2015.
Akashi, Y.; Okatani, T. Separation of reflection components by sparse non-negative matrix factorization. In: Computer Vision — ACCV 2014. Lecture Notes in Computer Science, Vol. 9007. Cremers, D.; Reid, I.; Saito, H.; Yang, M. H. Eds. Springer Cham, 611–625, 2015.
Guo, J.; Zhou, Z. J.; Wang, L. M. Single image highlight removal with a sparse and low-rank reflection model. In: Computer Vision — ECCV 2018. Lecture Notes in Computer Science, Vol. 11208. Ferrari, V.; Hebert, M.; Sminchisescu, C.; Weiss, Y. Eds. Springer Cham, 282–298, 2018.
Fu, G.; Zhang, Q.; Song, C. F.; Lin, Q. F.; **ao, C. X. Specular highlight removal for real-world images. Computer Graphics Forum Vol. 38, No. 7, 253–263, 2019.
Nayar, S. K.; Fang, X. S.; Boult, T. Separation of reflection components using color and polarization. International Journal of Computer Vision Vol. 21, No. 163–186, 1997.
Umeyama, S.; Godin, G. Separation of diffuse and specular components of surface reflection by use of polarization and statistical analysis of images. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 26, No. 5, 639–647, 2004.
Wang, F.; Ainouz, S.; Petitjean, C.; Bensrhair, A. Specularity removal: A global energy minimization approach based on polarization imaging. Computer Vision and Image Understanding Vol. 158, 31–39, 2017.
Wen, S.; Zheng, Y.; Lu, F. Polarization guided specular reflection separation. IEEE Transactions on Image Processing Vol. 30, 7280–7291, 2021.
Sapiro, G. Color and illuminant voting. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 21, No. 11, 1210–1215, 1999.
Imai, Y.; Kato, Y.; Kadoi, H.; Horiuchi, T.; Tominaga, S. Estimation of multiple illuminants based on specular highlight detection. In: Computational Color Imaging. Lecture Notes in Computer Science, Vol. 6626. Schettini, R.; Tominaga, S.; Trémeau, A. Eds. Springer Berlin Heidelberg, 85–98, 2011.
Forsyth, D. A. A novel algorithm for color constancy. International Journal of Computer Vision Vol. 5, No. 1, 5–35, 1990.
Hansen, T.; Olkkonen, M.; Walter, S.; Gegenfurtner, K. R. Memory modulates color appearance. Nature Neuroscience Vol. 9, No. 11, 1367–1368, 2006.
Joze, H. R. V.; Drew, M. S. Exemplar-based color constancy and multiple illumination. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 36, No. 5, 860–873, 2014.
Lin, P.; Quan, L.; Shum, H.-Y. Highlight removal by illumination-constrained inpainting. In: Proceedings of the 9th IEEE International Conference on Computer Vision, 164–169, 2003.
Tan, R. T.; Ikeuchi, K. Separating reflection components of textured surfaces using a single image. In: Digitally Archiving Cultural Objects. Springer Boston MA, 353–384, 2008.
Tan, T. T.; Nishino, K.; Ikeuchi, K. Illumination chromaticity estimation using inverse-intensity chromaticity space. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, I, 2003.
Funke, I.; Bodenstedt, S.; Riediger, C.; Weitz, J.; Speidel, S. Generative adversarial networks for specular highlight removal in endoscopic images. In: Proceedings of the SPIE 10576, Medical Imaging 2018: Image-Guided Procedures, Robotic Interventions, and Modeling, 1057604, 2018.
Wu, Z. Q.; Zhuang, C. Q.; Shi, J.; **ao, J.; Guo, J. W. Deep specular highlight removal for single real-world image. In: Proceedings of the SIGGRAPH Asia 2020 Posters, Article No. 34, 2020.
Wu, Z. Q.; Zhuang, C. Q.; Shi, J.; Guo, J. W.; **ao, J.; Zhang, X. P.; Yan, D.-M. Single-image specular highlight removal via real-world dataset construction. IEEE Transactions on Multimedia Vol. 24, 3782–3793, 2022.
Yi, R. J.; Tan, P.; Lin, S. Leveraging multi-view image sets for unsupervised intrinsic image decomposition and highlight separation. Proceedings of the AAAI Conference on Artificial Intelligence Vol. 34, No. 7, 12685–12692, 2020.
Fu, G.; Zhang, Q.; Zhu, L.; Li, P.; **ao, C. X. A multitask network for joint specular highlight detection and removal. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 7748–7757, 2021.
Ronneberger, O.; Fischer, P.; Brox, T. U-Net: Convolutional networks for biomedical image segmentation. In: Medical Image Computing and Computer-Assisted Intervention — MICCAI 2015. Lecture Notes in Computer Science, Vol. 9351. Navab, N.; Hornegger, J.; Wells, W.; Frangi, A. Eds. Springer Cham, 234–241, 2015.
Cao, H.; Wang, Y.; Chen, J.; Jiang, D.; Zhang, X.; Tian, Q.; Wang, M. Swin-Unet: Unet-like pure transformer for medical image segmentation. ar**v preprint ar**v:2105.05537, 2021.
Wei, K. X.; Yang, J. L.; Fu, Y.; Wipf, D.; Huang, H. Single image reflection removal exploiting misaligned training data and network enhancements. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 8170–8179, 2019.
Gatys, L. A.; Ecker, A. S.; Bethge, M. Image style transfer using convolutional neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2414–2423, 2016.
Simonyan, K.; Zisserman, A. Very deep convolutional networks for large-scale image recognition. ar**v preprint ar**v:1409.1556, 2014.
Murmann, L.; Gharbi, M.; Aittala, M.; Durand, F. A dataset of multi-illumination images in the wild. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, 4079–4088, 2019.
Guo, X. J.; Cao, X. C.; Ma, Y. Robust separation of reflection from multiple images. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2195–2202, 2014.
Kingma, D. P.; Ba, J. Adam: A method for stochastic optimization. In: Proceedings of the 3rd International Conference for Learning Representations, 2015.
Yamamoto, T.; Kitajima, T.; Kawauchi, R. Efficient improvement method for separation of reflection components based on an energy function. In: Proceedings of the IEEE International Conference on Image Processing, 4222–4226, 2017.
Hou, S.; Wang, C.; Quan, W.; Jiang, J.; Yan, D. M. Text-aware single image specular highlight removal. In: Pattern Recognition and Computer Vision. Lecture Notes in Computer Science, Vol. 13022. Springer Cham, 115–127, 2021.
Acknowledgements
This work was partially funded by the National Natural Science Foundation of China (U21A20515, 62172416, 62172415, U2003109), and Youth Innovation Promotion Association of the Chinese Academy of Sciences (2022131).
Author information
Authors and Affiliations
Corresponding authors
Ethics declarations
The authors have no competing interests to declare that are relevant to the content of this article.
Additional information
Zhongqi Wu received her master degree from the School of Artificial Intelligence of the University of the Chinese Academy of Sciences in 2019. She is currently working towards her Ph.D degree at the National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences. Her research interests include image processing and computer vision.
Jianwei Guo is an associate professor in the National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences (CASIA). He received his Ph.D. degree in computer science from CASIA in 2016, and bachelor degree from Shandong University in 2011. His research interests include computer vision, computer graphics, and image processing.
Chuanqing Zhuang is working toward a master degree in School of Artificial Intelligence, the University of the Chinese Academy of Sciences. He received his bachelor degree in engineering from Tsinghua University in 2019. His re-search interests include computer vision and image processing.
Jun **ao is a professor in the University of the Chinese Academy of Sciences. He obtained his Ph.D. degree in communication and information system from the Graduate University of the Chinese Academy of Sciences in 2008. His research interests include computer graphics, computer vision, image processing, and 3D reconstruction.
Dong-Ming Yan is a professor in the National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences. He received his Ph.D. degree in computer science from Hong Kong University in 2010, and his master and bachelor degrees in computer science and technology from Tsinghua University in 2005 and 2002, respectively. His research interests include image processing, geometric processing, and visualization.
**aopeng Zhang received his Ph.D. degree in computer science from the Institute of Software, Chinese Academic of Sciences in 1999, where he is a professor. He received a National Scientific and Technological Progress Prize (second class) in 2004 and a Chinese Award of Excellent Patents in 2012. His main research interests include image processing, computer graphics, and computer vision.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made.
The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.
To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
Other papers from this open access journal are available free of charge from http://www.springer.com/journal/41095. To submit a manuscript, please go to https://www.editorialmanager.com/cvmj.
About this article
Cite this article
Wu, Z., Guo, J., Zhuang, C. et al. Joint specular highlight detection and removal in single images via Unet-Transformer. Comp. Visual Media 9, 141–154 (2023). https://doi.org/10.1007/s41095-022-0273-9
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s41095-022-0273-9