SpringerLink
Log in

Pupil center detection inspired by multi-task auxiliary learning characteristic

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Robust and accurate pupil center detection is a prerequisite for eye-tracking system. In natural environments, due to rapid illumination transformation, pupil occlusion, specular reflection, etc., existing pupil detection methods are not easy to detect the pupil center robustly and accurately. To solve this problem, inspired by the multi-task auxiliary learning characteristic of human beings, we propose a coarse-to-fine pupil center detection method. We explore the hidden relationship between multi-task auxiliary learning characteristic and the pupil detection task. Our method can be divided into two stages, i.e., coarse classification and fine regression. Firstly, in the coarse classification stage, we use multiple subtasks to assist the main task so as to improve the robustness of the model. Secondly, in the fine regression stage, we use the main task from coarse classification to assist the regression task to improve the accuracy of the model. The results of the coarse classification stage are refined to determine the final accurate pupil center. The proposed method achieves 97% detection rate on LPW with a distance error lower than 5 pixels. In addition, our method also shows the state-of-the-art results on the ExCuSe, ElSe, and PupilNet datasets. Finally, a large number of experiments are taken to verify the effectiveness and advancement of the pupil center detection method inspired by the multi-task auxiliary learning characteristic.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price includes VAT (Germany)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Alex K, Roberto C (2018) Multi-task learning using uncertainty to weigh losses for scene geometry and semantics 30–43 ar**v:1705.07115

  2. Chen LC, Papandreou G, Schroff F, Adam H (2017) Rethinking atrous convolution for semantic image segmentation, pp 30–43. ar**v:1706.05587

  3. Chen C, Wang J, Lin Y (2019) A visual interactive reading system based on eye tracking technology to improve digital reading performance. The Electronic Library 37:680–702

    Article  Google Scholar 

  4. Chinsatit W, Saitoh T (2017) Cnn-based pupil center detection for wearable gaze estimation system. Appl Comput Intell Soft Comput 2017:1–10

    Google Scholar 

  5. Didday RL, Arbib MA (1975) Eye movements and A two visual perception: visual system model. Int J Man Mach Stud 7:547–569

    Article  Google Scholar 

  6. Dowiasch S, Backasch B, Einhäuser W., Leube D, Kircher T, Bremmer F (2016) Eye movements of patients with schizophrenia in a natural environment. Eur Arch Psychiatry Clin Neurosci 266:43–54

    Article  Google Scholar 

  7. Edewaard DE, Tyrrell RA, Duchowski AT, Szubski EC, King SS (2020) Using eye tracking to assess the temporal dynamics by which drivers notice cyclists in daylight: Drivers becoming aware of cyclists. In: ETRA ’20: 2020 symposium on eye tracking research and applications, pp 36:1–36:5

  8. Eivazi S, Santini T, Keshavarzi A, Kubler T, Mazzei A (2019) Improving real-time cnn-based pupil detection through domain-specific data augmentation. In: Proceedings of the 11th ACM symposium on eye tracking research and amp applications, pp 40:1–40:6

  9. Fang A, Zhao X, Yang J, Cao S, Zhang Y (2020) Ae-net: Autonomous evolution image fusion method inspired by human cognitive mechanism, 30–43

  10. Fang A, Zhao X, Yang J, Zhang Y (2019) A cross-modal image fusion method guided by human visual characteristics. ar**v:1912.08577

  11. Fang A, Zhao X, Yang J, Zhang Y (2019) Non-linear and selective fusion of cross-modal images. ar**v:1912.10738

  12. Fang A, Zhao X, Zhang Y (2020) Cross-modal image fusion guided by subjective visual attention. Neurocomputing 49:26719–26730

    Google Scholar 

  13. Figueiredo GR, Ripka WL, Romaneli EFR, Ulbricht L (2019) Attentional bias for emotional faces in depressed and non-depressed individuals: an eye-tracking study. In: 41st annual international conference of the IEEE engineering in medicine and biology society, pp 5419–5422

  14. Fuhl W, Kübler TC, Sippel K, Rosenstiel W, Kasneci E (2015) Excuse: Robust pupil detection in real-world scenarios. In: Computer analysis of images and patterns 16th international conference, vol 9256, pp 39–51

  15. Fuhl W, Santini T, Kasneci G, Kasneci E (2016) Pupilnet: convolutional neural networks for robust pupil detection. Revista De Odontologia Da Unesp 19:806–821

    Google Scholar 

  16. Fuhl W, Santini T, Kübler T, Kasneci E (2016) Else: ellipse selection for robust pupil detection in real-world environments. In: Proceedings of the Ninth Biennial ACM Symposium on eye tracking research and applications, vol 14, pp 123–130

  17. Gomolka Z, Kordos D, Zeslawska E (2020) The application of flexible areas of interest to pilot mobile eye tracking. Sensors 20:986–986

    Article  Google Scholar 

  18. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), vol 2016, pp 770–778

  19. Javadi A-H, Hakimi Z, Barati M, Walsh V, Tcheang L (2015) Set: a pupil detection method using sinusoidal approximation. Frontiers in Neuroengineering 8:4–4

    Article  Google Scholar 

  20. Joo HJ, Jeong HY (2020) A study on eye-tracking-based interface for vr/ar education platform. Multim Tools Appl 79(23-24):16719–16730

    Article  Google Scholar 

  21. Kang J, Han X, Song J, Niu Z, Li X (2020) The identification of children with autism spectrum disorder by SVM approach on eeg and eye-tracking data. Comput Biol Med 120:103722–103722

    Article  Google Scholar 

  22. Kim HC, ** S, Jo S, Lee JH (2020) A naturalistic viewing paradigm using 360 degree panoramic video clips and real-time field-of-view changes with eye-gaze tracking. NeuroImage 216:116617–116617

    Article  Google Scholar 

  23. Kirkpatrick J, Pascanu R, Rabinowitz N, Veness J, Desjardins G, Rusu AA, Milan K, Quan J, Ramalho T, Grabska-Barwinska A, Hassabis D, Clopath C, Kumaran D, Hadsell R (2017) Overcoming catastrophic forgetting in neural networks. Proc Natl Acad Sci U S A 114:3521–3526

    Article  MathSciNet  Google Scholar 

  24. Krizhevsky A, Sutskever I, Hinton G (2017) Imagenet classification with deep convolutional neural networks, 60:84–90

  25. Lee GJ, Jang SW, Kim GY (2020) Pupil detection and gaze tracking using a deformable template. Multimed Tools Appl 79:12939–12958

    Article  Google Scholar 

  26. Li H, Lin Z, Shen X, Brandt J, Hua G (2015) A convolutional neural network cascade for face detection. In: IEEE conference on computer vision and pattern recognition, pp 5325–5334

  27. Li D, Winfield D, Parkhurst DJ (2005) Starburst: A hybrid algorithm for video-based eye tracking combining feature-based and model-based approaches. In: IEEE conference on computer vision and pattern recognition, pp 79–79

  28. Lim JZ, Mountstephens J, Teo J (2020) Emotion recognition using eye-tracking: taxonomy, review and currentchallenges. Sensors 20:2384–2384

    Article  Google Scholar 

  29. Long X, Tonguz OK, Kiderman A (2007) A high speed eye tracking system with robust pupil center estimation algorithm. In: 2007 29th annual international conference of the IEEE engineering in medicine and biology society, pp 3331–3334

  30. Miller EK, Cohen JD (2001) An integrative theory of prefrontal cortex function. Annu Rev Neurosci 24:167–202

    Article  Google Scholar 

  31. Park H, Lee S, Lee M, Chang MS, bWan Kwak H (2016) Using eye movement data to infer human behavioral intentions. Comput Hum Behav 63:796–804

    Article  Google Scholar 

  32. Riba P, Llads J, Forns A (2020) Hierarchical graphs for coarse-to-fine error tolerant matching. Pattern Recognit Lett 134:116–124

    Article  Google Scholar 

  33. Ruiz N, Chong E, M Rehg J (2018) Fine-grained head pose estimation without keypoints. In: 2018 IEEE conference on computer vision and pattern recognition, pp 2074–2083

  34. Ryan WJ, Duchowski AT, Birchfield ST (2008) Limbus/pupil switching for wearable eye tracking under variable lighting conditions. In: Proceedings of the eye tracking research and application symposium, pp 61–64

  35. Santini T, Fuhl W, Kasneci E (2018) Pure: Robust pupil detection for real-time pervasive eye tracking. Comput Vis Image Underst 170:40–50

    Article  Google Scholar 

  36. Simonyan K, Zisserman A (2015) Very deep convolutional networks for large-scale image recognition. In: 3Rd international conference on learning representations, ar**v:1409.1556

  37. Strnádelová B, Halamová J, Kanovský M (2019) Eye-tracking of facial emotions in relation to self-criticism and self-reassurance. Appl Artif Intell 33:839–862

    Article  Google Scholar 

  38. Szegedy C, Liu W, Jia Y, P. S. et al (2015) Going deeper with convolutions. In: 2015 IEEE conference on computer vision and pattern recognition, pp 1–9

  39. Świrski L, Bulling A, Dodgson N (2012) Robust real-time pupil tracking in highly off-axis images. In: Proceedings of the symposium on eye tracking research and applications, pp 173–176

  40. Valenti R, Gevers T (2012) Accurate eye center location through invariant isocentric patterns. IEEE Trans Pattern Anal Mach Intell 34:1785–1798

    Article  Google Scholar 

  41. Vera-Olmos F, Pardo E, Melero H, Malpica N (2019) Deepeye: Deep convolutional network for pupil detection in real environments. Integrated Computer-Aided Engineering 26:85–95

    Article  Google Scholar 

  42. Wang Y, Liang W, Shen J, Jia Y, Yu L (2019) A deep coarse-to-fine network for head pose estimation from synthetic data. Pattern Recognit 94:196–206

    Article  Google Scholar 

  43. Wang X, Zhao X, Ren J (2019) A new type of eye movement model based on recurrent neural networks for simulating the gaze behavior of human reading. Complexity 2019:12–12

    Google Scholar 

  44. Zhang J, Mei K, Zheng Y, Fan J (2019) Learning multi-layer coarse-to-fine representations for large-scale image classification. Pattern Recognit 91:175–189

    Article  Google Scholar 

  45. Zhang Y, Yang Q (2018) An overview of multi-task learning. Nat Sci Rev 5:30–43

    Article  Google Scholar 

  46. Zhou X, Gao X, Wang J, Yu H, Wang Z, Chi Z (2017) Eye tracking data guided feature selection for image classification. Pattern Recognit 63:56–70

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported by the National Natural Science Foundation of China under Grants Nos. 61871326, and the Shaanxi Natural Science Basic Research Program under Grants Nos. 2018JM6116, and Science and Technology on Electro-optic Control Laboratory Jointly funded with Aviation Science Fund under Grants Nos. 20185153031.

Author information

Authors and Affiliations

  1. School of Software, Northwestern Polytechnical University, **’an, 710072, China

    Zheng **ang

  2. School of computer Science, Northwestern Polytechnical University, **’an, 710072, China

    **nbo Zhao & Aiqing Fang

  3. Science and Technology on Electro-optic Control Laboratory, Luoyang, 471009, China

    **nbo Zhao

Authors
  1. Zheng **ang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. **nbo Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Aiqing Fang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to **nbo Zhao.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

**ang, Z., Zhao, X. & Fang, A. Pupil center detection inspired by multi-task auxiliary learning characteristic. Multimed Tools Appl 81, 40067–40088 (2022). https://doi.org/10.1007/s11042-022-12278-4

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-022-12278-4

Keywords

Search

Navigation