SpringerLink
Log in

FRSE-Net: low-illumination object detection network based on feature representation refinement and semantic-aware enhancement

  • Original article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

Deep learning-based object detection methods have achieved great performance improvement. However, the current mainstream object detectors focus on normal illumination images, while low-illumination object detection is often ignored. It is still a challenging task to detect objects in low-illumination scenes due to insufficient illumination and low visibility. To address this issue, we propose a low-illumination object detection network based on feature representation refinement and semantic-aware enhancement, called FRSE-Net. There are two key components in the proposed network, including a feature capture module (FCM) and a semantic aggregation module (SAM). First, the FCM is designed to enhance the feature representation of the feature map, thus making the object features more discriminative. This is beneficial to capture more effective feature information for subsequent prediction tasks. Furthermore, the SAM is introduced to enhance the semantic-aware ability of the model in low-light images, which makes the detection network focus on the objects of interest to learn rich semantic information. Finally, the experimental results on two low-light image datasets demonstrate the effectiveness and superiority of the proposed network when compared with other advanced low-illumination detection methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (Germany)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

References

  1. Wei, L., Cui, W., Hu, Z., Sun, H., Hou, S.: A single-shot multi-level feature reused neural network for object detection. Vis. Comput. 37(1), 133–142 (2021)

    Article  Google Scholar 

  2. Kim, S., Winovich, N., Chi, H.-G., Lin, G., Ramani, K.: Latent transformations neural network for object view synthesis. Vis. Comput. 36(8), 1663–1677 (2020)

    Article  Google Scholar 

  3. Everingham, M., Van Gool, L., Williams, C.K., Winn, J., Zisserman, A.: The pascal visual object classes (VOC) challenge. Int. J. Comput. Vis. 88(2), 303–338 (2010)

    Article  Google Scholar 

  4. Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: common objects in context. In: European Conference on Computer Vision, pp. 740–755. Springer (2014)

  5. Loh, Y.P., Chan, C.S.: Getting to know low-light images with the exclusively dark dataset. Comput. Vis. Image Underst. 178, 30–42 (2019)

    Article  Google Scholar 

  6. Zaidi, S.S.A., Ansari, M.S., Aslam, A., Kanwal, N., Asghar, M., Lee, B.: A survey of modern deep learning based object detection models. Digit. Signal Process. 126, 103514 (2022)

    Article  Google Scholar 

  7. Huang, Y., Jiang, Z., Lan, R., Zhang, S., Pi, K.: Infrared image super-resolution via transfer learning and PSRGAN. IEEE Signal Process. Lett. 28, 982–986 (2021)

    Article  Google Scholar 

  8. Kera, S.B., Tadepalli, A., Ranjani, J.J.: A paced multi-stage block-wise approach for object detection in thermal images. Vis. Comput. (2022). https://doi.org/10.1007/s00371-022-02445-x

    Article  Google Scholar 

  9. Huang, Y., Wang, Q., Omachi, S.: Rethinking degradation: radiograph super-resolution via aid-srgan. In: International Workshop on Machine Learning in Medical Imaging, pp. 43–52. Springer (2022)

  10. Huang, Y., Jiang, Z., Wang, Q., Jiang, Q., Pang, G.: Infrared image super-resolution via heterogeneous convolutional WGAN. In: Pacific Rim International Conference on Artificial Intelligence, pp. 461–472. Springer (2021)

  11. Bochkovskiy, A., Wang, C.-Y., Liao, H.-Y.M.: Yolov4: optimal speed and accuracy of object detection (2020). ar**v preprint ar**v:2004.10934

  12. Li, C., Guo, C., Han, L., Jiang, J., Cheng, M.-M., Gu, J., Loy, C.C.: Low-light image and video enhancement using deep learning: a survey. IEEE Trans. Pattern Anal. Mach. Intell. 44(12), 9396–9416 (2021)

    Article  Google Scholar 

  13. Arici, T., Dikbas, S., Altunbasak, Y.: A histogram modification framework and its application for image contrast enhancement. IEEE Trans. Image Process. 18(9), 1921–1935 (2009)

    Article  MathSciNet  Google Scholar 

  14. Kokufuta, K., Maruyama, T.: Real-time processing of local contrast enhancement on FPGA. In: 2009 International Conference on Field Programmable Logic and Applications, pp. 288–293 (2009)

  15. Guo, X., Li, Y., Ling, H.: Lime: low-light image enhancement via illumination map estimation. IEEE Trans. Image Process. 26(2), 982–993 (2017)

    Article  MathSciNet  Google Scholar 

  16. Zhang, Q., Nie, Y., Zheng, W.-S.: Dual illumination estimation for robust exposure correction. Comput. Graph. Forum 38(7), 243–252 (2019)

    Article  Google Scholar 

  17. Jiang, Y., Gong, X., Liu, D., Cheng, Y., Fang, C., Shen, X., Yang, J., Zhou, P., Wang, Z.: Enlightengan: deep light enhancement without paired supervision. IEEE Trans. Image Process. 30, 2340–2349 (2021)

    Article  Google Scholar 

  18. Li, C., Guo, C., Loy, C.C.: Learning to enhance low-light image via zero-reference deep curve estimation. IEEE Trans. Pattern Anal. Mach. Intell. 44(8), 4225–4238 (2022)

    Article  Google Scholar 

  19. Wu, W., Weng, J., Zhang, P., Wang, X., Yang, W., Jiang, J.: Uretinex-net: Retinex-based deep unfolding network for low-light image enhancement. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5901–5910 (2022)

  20. Ma, L., Ma, T., Liu, R., Fan, X., Luo, Z.: Toward fast, flexible, and robust low-light image enhancement. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5637–5646 (2022)

  21. Li, C., Guo, C., Han, L., Jiang, J., Cheng, M.-M., Gu, J., Loy, C.C.: Lighting the darkness in the deep learning era (2021). ar**v preprint ar**v: 2104.10729

  22. Meng, Z., Xu, R., Ho, C.M.: Gia-net: global information aware network for low-light imaging. In: European Conference on Computer Vision, pp. 327–342. Springer (2020)

  23. Huang, Y., Miyazaki, T., Liu, X., Omachi, S.: Infrared image super-resolution: systematic review, and future trends (2022). ar**v preprint ar**v:2212.12322

  24. Cui, Z., Qi, G.-J., Gu, L., You, S., Zhang, Z., Harada, T.: Multitask aet with orthogonal tangent regularity for dark object detection. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2553–2562 (2021)

  25. Liu, W., Ren, G., Yu, R., Guo, S., Zhu, J., Zhang, L.: Image-adaptive yolo for object detection in adverse weather conditions. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, pp. 1792–1800 (2022)

  26. Wang, W., Yang, W., Liu, J.: Hla-face: joint high-low adaptation for low light face detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16195–16204 (2021)

  27. Lin, T.-Y., Dollár, P., Girshick, R., He, K., Hariharan, B., Belongie, S.: Feature pyramid networks for object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2117–2125 (2017)

  28. Guo, Z., Shuai, H., Liu, G., Zhu, Y., Wang, W.: Multi-level feature fusion pyramid network for object detection. Vis. Comput. (2022). https://doi.org/10.1007/s00371-022-02589-w

    Article  Google Scholar 

  29. Ulyanov, D., Vedaldi, A., Lempitsky, V.: Instance normalization: the missing ingredient for fast stylization (2016). ar**v preprint ar**v:1607.08022

  30. Liu, S., Qi, L., Qin, H., Shi, J., Jia, J.: Path aggregation network for instance segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8759–8768 (2018)

  31. Shao, M., Zhang, W., Li, Y., Fan, B.: Branch aware assignment for object detection. Vis. Comput. (2022). https://doi.org/10.1007/s00371-022-02691-z

    Article  Google Scholar 

  32. Wu, X., Wu, Z., Guo, H., Ju, L., Wang, S.: Dannet: a one-stage domain adaptation network for unsupervised nighttime semantic segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15769–15778 (2021)

  33. Luo, Y., Cao, X., Zhang, J., Guo, J., Shen, H., Wang, T., Feng, Q.: Ce-fpn: enhancing channel information for object detection. Multimed. Tools Appl. 81, 30685–30704 (2022)

    Article  Google Scholar 

  34. Hu, J., Shen, L., Sun, G.: Squeeze-and-excitation networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7132–7141 (2018)

  35. Guo, M.-H., Xu, T.-X., Liu, J.-J., Liu, Z.-N., Jiang, P.-T., Mu, T.-J., Zhang, S.-H., Martin, R.R., Cheng, M.-M., Hu, S.-M.: Attention mechanisms in computer vision: a survey. Comput. Vis. Med. 8, 331–368 (2022)

    Article  Google Scholar 

  36. Zhu, J.-Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2223–2232 (2017)

  37. Loshchilov, I., Hutter, F.: Sgdr: stochastic gradient descent with warm restarts (2016). ar**v preprint ar**v:1608.03983

  38. Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., Fei-Fei, L.: Imagenet: a large-scale hierarchical image database. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 248– 255 (2009)

  39. Woo, S., Park, J., Lee, J.-Y., Kweon, I.S.: Cbam: convolutional block attention module. In: European Conference on Computer Vision, pp. 3– 19. Springer (2018)

  40. Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: visual explanations from deep networks via gradient-based localization. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 618–626 (2017)

Download references

Acknowledgements

This work is supported by Nature Science Foundation of China (62172118) and Nature Science Key Foundation of Guangxi (2021GXNSFDA196002); in part by the Guangxi Key Laboratory of Image and Graphic Intelligent Processing under Grants (GIIP2203, GIIP2204); in part by the Innovation Project of Guangxi Graduate Education under Grants (YCB2021070, YCBZ2018052, YCSW2022269); and in part by the Innovation Project of GUET Graduate Education under Grants (2021YCXS071).

Author information

Authors and Affiliations

  1. Guangxi Key Laboratory of Image and Graphic Intelligent Processing, Guilin University of Electronic Technology, Guilin, 541004, China

    Zetao Jiang & Daoquan Shi

  2. Nanchang Hangkong University, Nanchang, 330063, China

    Shaoqin Zhang

Authors
  1. Zetao Jiang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Daoquan Shi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Shaoqin Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Daoquan Shi.

Ethics declarations

Conflict of interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Data availability statement

The data generated and analyzed during the current study are available from the corresponding author on reasonable request.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Daoquan Shi and Zetao Jiang contributed equally to this work.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Jiang, Z., Shi, D. & Zhang, S. FRSE-Net: low-illumination object detection network based on feature representation refinement and semantic-aware enhancement. Vis Comput 40, 3233–3247 (2024). https://doi.org/10.1007/s00371-023-03024-4

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-023-03024-4

Keywords

Search

Navigation