SpringerLink
Log in

Face photo-sketch recognition based on multi-directional line features projection

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Face photo-sketch recognition plays an important role in law enforcement, particularly in narrowing down the search for potential suspects based on limited sketch information. However, the issues of large modality gap and having a relatively small number of sketch samples for training remained a challenging task. In this paper, we propose a novel feature descriptor network for automated face photo-sketch recognition that is suitable for modality discrepancy and small dataset learning. By stacking a multi-directional image difference operation over a pooling projection in a multilayer fashion, our proposal forms an interpretable learning system that does not show obvious overfitting on limited training data. Extensive evaluation using three public face photo-sketch databases shows competing rank-1 recognition accuracy of the proposed method comparing with state-of-the-art methods. In terms of average ranking on the three experimented databases, the proposed method has the top average rank of 2 among 17 algorithms with the runner-up LFDA algorithm having an average rank of 2.83.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (Germany)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

Data availability

The datasets analysed during the current study have been based on [3, 6, 40]. These datasets are available from the following public domain resources: http://mmlab.ie.cuhk.edu.hk/archive/facesketch.html; http://mmlab.ie.cuhk.edu.hk/archive/cufsf/; https://biometrics.cse.msu.edu/Publications/Databases/PRIP-VSGC-Release.zip.

References

  1. Chalabi NE, Attia A, Bouziane A, Hassaballah M, Akhtar Z (2022) Recent trends in face recognition using metaheuristic optimization. In: Handbook of nature-inspired optimization algorithms: the state of the art, Volume II, Solving Constrained Single Objective Real-Parameter Optimization Problems. Springer, pp 85–112

  2. Galea C, Farrugia RA (2017) Forensic face photo-sketch recognition using a deep learning-based architecture. IEEE Signal Process Lett 24(11):1586–1590

    Article  Google Scholar 

  3. Wang X, Tang X (2009) Face photo-sketch synthesis and recognition. IEEE Trans Pattern Anal Mach Intell 31(11):1955–1967

    Article  Google Scholar 

  4. Hassaballah M, Aly S (2015) Face recognition: challenges, achievements and future directions. IET Comput Vis 9(4):614–626

    Article  Google Scholar 

  5. Sarfraz MS, Stiefelhagen R (2017) Deep perceptual map** for cross-modal face recognition. Int J Comput Vis 122(3):426–438

    Article  Google Scholar 

  6. Zhang W, Wang X, Tang X (2011) Coupled information-theoretic encoding for face photo-sketch recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE, pp 513–520

  7. Klare B, Li Z, Jain AK (2010) Matching forensic sketches to mug shot photos. IEEE Trans Pattern Anal Mach Intell 33(3):639–646

    Article  Google Scholar 

  8. Klare BF, Jain AK (2012) Heterogeneous face recognition using kernel prototype similarities. IEEE Trans Pattern Anal Mach Intell 35(6):1410–1422

    Article  Google Scholar 

  9. Setumin S, Suandi SA (2019) Cascaded static and dynamic local feature extractions for face sketch to photo matching. IEEE Access 7:27135–27145

    Article  Google Scholar 

  10. Tang X, Wang X (2003) Face sketch synthesis and recognition. In: Proceedings ninth IEEE international conference on computer vision. IEEE, pp 687–694

  11. Liu Q, Tang X, ** H, Lu H, Ma S (2005) A nonlinear approach for face sketch synthesis and recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. vol. 1. IEEE, pp 1005–1010

  12. Zhang L, Lin L, Wu X, Ding S, Zhang L (2015) End-to-end photo-sketch generation via fully convolutional representation learning. In: Proceedings of the 5th ACM on international conference on multimedia retrieval, pp 627–634

  13. Wang N, Gao X, Li J (2018) Random sampling for fast face sketch synthesis. Pattern Recogn 76:215–227

    Article  Google Scholar 

  14. Zheng J, Song W, Wu Y, Xu R, Liu F (2019) Feature encoder guided generative adversarial network for face photo-sketch synthesis. IEEE Access 7:154971–154985

    Article  Google Scholar 

  15. Zhu M, Li J, Wang N, Gao X (2021) Learning deep patch representation for probabilistic graphical model-based face sketch synthesiss. Int J Comput Vis. 129(6):1820–1836

    Article  Google Scholar 

  16. Klare B, Jain AK (2010) Sketch-to-photo matching: a feature-based approach. In: Biometric technology for human identification VII. vol. 7667. International Society for Optics and Photonics, pp 766702

  17. Galoogahi HK, Sim T (2012) Face sketch recognition by local Radon binary pattern: LRBP. In: 2012 19th IEEE international conference on image processing. IEEE, pp 1837–1840

  18. Ojala T, Pietikainen M, Maenpaa T (2002) Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Trans Pattern Anal Mach Intell 24(7):971–987

    Article  MATH  Google Scholar 

  19. Dalal N, Triggs B (2005) Histograms of oriented gradients for human detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition. vol 1. IEEE, pp 886–893

  20. Wen Y, Zhang K, Li Z, Qiao Y (2019) A comprehensive study on center loss for deep face recognition. Int J Comput Vis 127(6):668–683

    Article  Google Scholar 

  21. Han C, Shan S, Kan M, Wu S, Chen X (2022) Personalized convolution for face recognition. In: International journal of computer vision, pp 1–19

  22. Saxena S, Verbeek J (2016) Heterogeneous face recognition with CNNs. In: European conference on computer vision. Springer, pp 483–491

  23. Hu G, Peng X, Yang Y, Hospedales TM, Verbeek J (2017) Frankenstein: learning deep face representations using small data. IEEE Trans Image Process 27(1):293–303

    Article  MathSciNet  MATH  Google Scholar 

  24. Mittal P, Vatsa M, Singh R (2015) Composite sketch recognition via deep network-a transfer learning approach. In: 2015 international conference on biometrics (ICB). IEEE, pp 251–256

  25. Fu C, Wu X, Hu Y, Huang H, He R (2021) Dvg-face: Dual variational generation for heterogeneous face recognition. In: IEEE transactions on pattern analysis and machine intelligence

  26. Wu X, Song L, He R, Tan T (2018) Coupled deep learning for heterogeneous face recognition. In: Proceedings of the AAAI conference on artificial intelligence, vol 32

  27. Wan W, Gao Y, Lee HJ (2019) Transfer deep feature learning for face sketch recognition. Neural Comput Appl 31(12):9175–9184

    Article  Google Scholar 

  28. Simard PY, Steinkraus D, Platt JC, et al (2003) Best practices for convolutional neural networks applied to visual document analysis. In: Icdar, vol 3

  29. Cao B, Wang N, Li J, Gao X (2018) Data augmentation-based joint learning for heterogeneous face recognition. IEEE Trans Neural Netw Learn Syst 30(6):1731–1743

    Article  MathSciNet  Google Scholar 

  30. Williford JR, May BB, Byrne J (2020) Explainable face recognition. In: European conference on computer vision. Springer, pp 248–263

  31. Fan KC, Hung TY (2014) A novel local pattern descriptor-local vector pattern in high-order derivative space for face recognition. IEEE Trans Image Process 23(7):2877–2891

    Article  MathSciNet  MATH  Google Scholar 

  32. Kim J, Oh K, Oh BS, Lin Z, Toh KA (2019) A line feature extraction method for finger-knuckle-print verification. Cogn Comput 11(1):50–70

    Article  Google Scholar 

  33. LeCun Y, Bottou L, Bengio Y, Haffner P (1998) Gradient-based learning applied to document recognition. Proc IEEE. 86(11):2278–2324

    Article  Google Scholar 

  34. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778

  35. Tolias G, Sicre R, Jégou H (2016) Particular object retrieval with integral max-pooling of CNN activations. In: International Conference on Learning Representations, pp 1–12

  36. Pinheiro PO, Collobert R (2015) From image-level to pixel-level labeling with convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1713–1721

  37. Edmonds J (1971) Matroids and the greedy algorithm. Math Program 1(1):127–136

    Article  MathSciNet  MATH  Google Scholar 

  38. Zhang Y, Tiňo P, Leonardis A, Tang K (2021) A survey on neural network interpretability. In: IEEE transactions on emerging topics in computational intelligence

  39. Caruana R, Lawrence S, Giles C (2000) Overfitting in neural nets: backpropagation, conjugate gradient, and early stop**. In; Advances in neural information processing systems, vol 13

  40. Han H, Klare BF, Bonnen K, Jain AK (2012) Matching composite sketches to face photos: a component-based approach. IEEE Trans Inf Forensics Secur 8(1):191–204

    Article  Google Scholar 

  41. Chan CH, Kittler J, Messer K (2007) Multi-scale local binary pattern histograms for face recognition. In: International conference on biometrics. Springer, pp 809–818

  42. Petpon A, Srisuk S (2009) Face recognition with local line binary pattern. In: 2009 Fifth international conference on image and graphics. IEEE, pp 533–539

  43. Ding C, Choi J, Tao D, Davis LS (2015) Multi-directional multi-level dual-cross patterns for robust face recognition. IEEE Trans Pattern Anal Mach Intell 38(3):518–531

    Article  Google Scholar 

  44. Parkhi OM, Vedaldi A, Zisserman A (2015) Deep face recognition. In: Proceedings of the british machine vision conference. BMVA Press, pp 41.1–41.12

  45. Wu X, He R, Sun Z, Tan T (2018) A light cnn for deep face representation with noisy labels. IEEE Trans Inf Forensics Secur 13(11):2884–2896

    Article  Google Scholar 

  46. Wang H, Wang Y, Zhou Z, Ji X, Gong D, Zhou J, et al (2018) Cosface: large margin cosine loss for deep face recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 5265–5274

  47. Deng J, Guo J, Xue N, Zafeiriou S (2019) Arcface: Additive angular margin loss for deep face recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4690–4699

  48. Zeiler MD, Fergus R (2014) Visualizing and understanding convolutional networks. In: European conference on computer vision. Springer, pp 818–833

  49. Brazdil PB, Soares C (2000) A comparison of ranking methods for classification algorithm selection. In: European conference on machine learning. Springer, pp 63–75

Download references

Acknowledgements

This research was supported in part by Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education, Science and Technology (NRF-2021R1A2C1093425), and in part by the NRF under the program of Basic Research Laboratory (NRF-2022R1A4A2000748).

Author information

Authors and Affiliations

  1. School of Electrical and Electronic Engineering, Yonsei University, Seoul, 03722, Korea

    Jooyoung Kim, Donghyun Kim & Kar-Ann Toh

  2. School of Electrical and Electronic Engineering, Nanyang Technological University, 639798, Singapore , Singapore

    Zhi** Lin

Authors
  1. Jooyoung Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zhi** Lin
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Donghyun Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Kar-Ann Toh
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Kar-Ann Toh.

Ethics declarations

Conflict of Interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kim, J., Lin, Z., Kim, D. et al. Face photo-sketch recognition based on multi-directional line features projection. Neural Comput & Applic 35, 20697–20715 (2023). https://doi.org/10.1007/s00521-023-08801-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-023-08801-9

Keywords

Search

Navigation