SpringerLink
Log in

Hierarchical Curriculum Learning for No-Reference Image Quality Assessment

  • Published:
International Journal of Computer Vision Aims and scope Submit manuscript

Abstract

Despite remarkable success has been achieved by convolutional neural networks (CNNs) in no-reference image quality assessment (NR-IQA), there still exist many challenges in improving the performance of IQA for authentically distorted images. An important factor is that the insufficient annotated data limits the training of high-capacity CNNs to accommodate diverse distortions, complicated semantic structures and high-variance quality scores of these images. To address this problem, this paper proposes a hierarchical curriculum learning (HCL) framework for NR-IQA. The main idea of the proposed framework is to leverage the external data to learn the prior knowledge about IQA widely and progressively. Specifically, as a closely-related task with NR-IQA, image restoration is used as the first curriculum to learn the image quality related knowledge (i.e., semantic and distortion information) on massive distorted-reference image pairs. Then multiple lightweight subnetworks are designed to learn human scoring rules on multiple available synthetic IQA datasets independently, and a cross-dataset quality assessment correlation (CQAC) module is proposed to fully explore the similarities and diversities of different scoring rules. Finally, the whole model is fine-tuned on the target authentic IQA dataset to fuse the learned knowledge and adapt to the target data distribution. Experimental results show that our model achieves state-of-the-art performance on multiple standard authentic IQA datasets. Moreover, the generalization of our model is fully validated by the cross-dataset evaluation and the gMAD competition. In addition, extensive analyses prove that the proposed HCL framework is effective in improving the performance of our model.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (France)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Data Availability

The data that support the findings of this study are available from the corresponding author upon reasonable request.

References

  • A, T. D., B, K. G., C, L. N., et al. (2018). Referenceless quality metric of multiply-distorted images based on structural degradation. Neurocomputing, 290, 185–195.

    Article  Google Scholar 

  • Bosse, S., Maniry, D., Müller, K. R., et al. (2017). Deep neural networks for no-reference and full-reference image quality assessment. IEEE Transactions on Image Processing (TIP), 27(1), 206–219.

    Article  MathSciNet  MATH  Google Scholar 

  • Ciancio, A., da Silva, E. A., Said, A., et al. (2010). No-reference blur assessment of digital pictures based on multifeature classifiers. IEEE Transactions on Image Processing (TIP), 20(1), 64–75.

    Article  MathSciNet  MATH  Google Scholar 

  • Deng, J., Dong, W., Socher, R., et al. (2009). Imagenet: A large-scale hierarchical image database. In Conference on computer vision and pattern recognition (CVPR) (pp. 248–255).

  • Fang, Y., Ma, K., Wang, Z., et al. (2015). No-reference quality assessment of contrast-distorted images based on natural scene statistics. IEEE Signal Processing Letters, 22(7), 838–842. https://doi.org/10.1109/LSP.2014.2372333

    Article  Google Scholar 

  • Fang, Y., Zhu, H., Zeng, Y., et al. (2020). Perceptual quality assessment of smartphone photography. In Conference on computer vision and pattern recognition (CVPR).

  • Ghadiyaram, D., & Bovik, A. C. (2015). Massive online crowdsourced study of subjective and objective picture quality. IEEE Transactions on Image Processing (TIP), 25(1), 372–387.

    Article  MathSciNet  MATH  Google Scholar 

  • Ghadiyaram, D., & Bovik, A. C. (2017). Perceptual quality prediction on authentically distorted images using a bag of features approach. Journal of Vision, 17(1), 32–32.

    Article  Google Scholar 

  • Golestaneh, S. A., Dadsetan, S., & Kitani, K. M. (2022). No-reference image quality assessment via transformers, relative ranking, and self-consistency. In IEEE Winter conference on applications of computer vision (WACV) (pp. 3989–3999). https://doi.org/10.1109/WACV51458.2022.00404

  • He, K., Zhang, X., Ren, S., et al. (2016). Deep residual learning for image recognition. In Conference on computer vision and pattern recognition (CVPR) (pp. 770–778).

  • Hosu, V., Lin, H., Sziranyi, T., et al. (2020). Koniq-10k: An ecologically valid database for deep learning of blind image quality assessment. IEEE Transactions on Image Processing (TIP), 29, 4041–4056.

    Article  MATH  Google Scholar 

  • Kang, L., Ye, P., Li, Y., et al. (2014). Convolutional neural networks for no-reference image quality assessment. In Conference on computer vision and pattern recognition (CVPR) (pp. 1733–1740).

  • Kang, L., Ye, P., Li, Y., et al. (2015). Simultaneous estimation of image quality and distortion via multi-task convolutional neural networks. In International conference on image processing (ICIP), IEEE (pp. 2791–2795).

  • Ke, J., Wang, Q., Wang, Y., et al. (2021). Musiq: Multi-scale image quality transformer. In International conference on computer vision (ICCV) (pp. 5148–5157).

  • Kim, J., & Lee, S. (2016). Fully deep blind image quality predictor. IEEE Journal of Selected Topics in Signal Processing, 11(1), 206–220.

    Article  Google Scholar 

  • Kim, J., Nguyen, A. D., & Lee, S. (2018). Deep cnn-based blind image quality predictor. IEEE Transactions on Neural Networks and Learning Systems, 30(1), 11–24.

    Article  Google Scholar 

  • Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). Imagenet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems (NeurIPS), 25, 1097–1105.

    Google Scholar 

  • Larson, E. C., & Chandler, D. M. (2010). Most apparent distortion: Full-reference image quality assessment and the role of strategy. Journal of Electronic Imaging, 19(1), 011006.

    Article  Google Scholar 

  • Li, D., Jiang, T., Lin, W., et al. (2018). Which has better visual quality: The clear blue sky or a blurry animal? IEEE Transactions on Multimedia (TMM), 21(5), 1221–1234.

    Article  Google Scholar 

  • Li, D., Jiang, T., & Jiang, M. (2020). Norm-in-norm loss with faster convergence and better performance for image quality assessment. In ACM International conference on multimedia (ACM MM) (pp. 789–797). ACM.

  • Li, D., Jiang, T., & Jiang, M. (2021). Unified quality assessment of in-the-wild videos with mixed datasets training. International Journal of Computer Vision (IJCV), 129, 1238–1257.

    Article  Google Scholar 

  • Li, L., **a, W., Lin, W., et al. (2017). No-reference and robust image sharpness evaluation based on multiscale spatial and spectral features. IEEE Transactions on Multimedia (TMM), 19(5), 1030–1040. https://doi.org/10.1109/TMM.2016.2640762

    Article  Google Scholar 

  • Li, L., Song, T., Wu, J., et al. (2022). Blind image quality index for authentic distortions with local and global deep feature aggregation. IEEE Transactions on Circuits and Systems for Video Technology (TCSTV), 32(12), 8512–8523. https://doi.org/10.1109/TCSVT.2021.3112197

    Article  Google Scholar 

  • Li, Q., Lin, W., Xu, J., et al. (2016). Blind image quality assessment using statistical structural and luminance features. IEEE Transactions on Multimedia (TMM), 18(12), 2457–2469.

    Article  Google Scholar 

  • Li, X. (2002). Blind image quality assessment. In International conference on image processing (ICIP) (pp. I–I). https://doi.org/10.1109/ICIP.2002.1038057

  • Lin, H., Hosu, V., & Saupe, D. (2019). Kadid-10k: A large-scale artificially distorted iqa database. In Conference on quality of multimedia experience (QoMEX), IEEE (pp. 1–3).

  • Lin, K. Y., & Wang, G. (2018) Hallucinated-iqa: No-reference image quality assessment via adversarial learning. In Conference on computer vision and pattern recognition (CVPR) (pp. 732–741).

  • Liu, L., Dong, H., Huang, H., et al. (2014). No-reference image quality assessment in curvelet domain. Signal Processing: Image Communication, 29(4), 494–505.

    Google Scholar 

  • Liu, X., Van De Weijer, J., & Bagdanov, A. D. (2017). Rankiqa: Learning from rankings for no-reference image quality assessment. In International conference on computer vision (ICCV) (pp. 1040–1049).

  • Ma, J., Wu, J., Li, L., et al. (2021). Blind image quality assessment with active inference. IEEE Transactions on Image Processing (TIP), 30, 3650–3663.

    Article  Google Scholar 

  • Ma, K., Duanmu, Z., Wu, Q., et al. (2016). Waterloo exploration database: New challenges for image quality assessment models. IEEE Transactions on Image Processing (TIP), 26(2), 1004–1016.

    Article  MathSciNet  MATH  Google Scholar 

  • Ma, K., Liu, W., Liu, T., et al. (2017). dipiq: Blind image quality assessment by learning-to-rank discriminable image pairs. IEEE Transactions on Image Processing (TIP), 26(8), 3951–3964.

    Article  MathSciNet  MATH  Google Scholar 

  • Ma, K., Duanmu, Z., Zhou, W., et al. (2018). Group maximum differentiation competition: Model comparison with few samples. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), PP(99), 1–1.

    Google Scholar 

  • Mittal, A., Moorthy, A. K., & Bovik, A. C. (2012). No-reference image quality assessment in the spatial domain. IEEE Transactions on Image Processing (TIP), 21(12), 4695–4708.

    Article  MathSciNet  MATH  Google Scholar 

  • Moorthy, A. K., & Bovik, A. C. (2010). A two-step framework for constructing blind image quality indices. IEEE Signal Processing Letters, 17(5), 513–516. https://doi.org/10.1109/LSP.2010.2043888

    Article  Google Scholar 

  • Pan, X., Zhan, X., Dai, B., et al. (2022). Exploiting deep generative prior for versatile image restoration and manipulation. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 44(11), 7474–7489. https://doi.org/10.1109/TPAMI.2021.3115428

    Article  Google Scholar 

  • Pan, Z., Yuan, F., Lei, J., et al. (2022). Vcrnet: Visual compensation restoration network for no-reference image quality assessment. IEEE Transactions on Image Processing (TIP), 31, 1613–1627. https://doi.org/10.1109/TIP.2022.3144892

    Article  Google Scholar 

  • Ponomarenko, N., Ieremeiev, O., Lukin, V., et al. (2013). Color image database tid2013: Peculiarities and preliminary results. In European workshop on visual information processing (EUVIP) (pp. 106–111)

  • Ren, H., Chen, D., & Wang, Y. (2017). Ran4iqa: Restorative adversarial nets for no-reference image quality assessment. In AAAI conference on artificial intelligence (AAAI) (Vol. 32). https://doi.org/10.1609/aaai.v32i1.12258

  • Ren, H., Chen, D., & Wang, Y. (2018). Ran4iqa: Restorative adversarial nets for no-reference image quality assessment. In AAAI Conference on artificial intelligence (AAAI).

  • Saad, M. A., Bovik, A. C., & Charrier, C. (2012). Blind image quality assessment: A natural scene statistics approach in the dct domain. IEEE Transactions on Image Processing (TIP), 21(8), 3339–3352.

    Article  MathSciNet  MATH  Google Scholar 

  • Sheikh, H. R., Sabir, M. F., & Bovik, A. C. (2006). A statistical evaluation of recent full reference image quality assessment algorithms. IEEE Transactions on Image Processing (TIP), 15(11), 3440–3451.

    Article  Google Scholar 

  • Simonyan, K., & Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. ar**v preprint ar**v:1409.1556

  • Song, T., Li, L., Zhu, H., et al. (2021). Ie-iqa: Intelligibility enriched generalizable no-reference image quality assessment. Frontiers in Neuroscience, 15, 739138.

    Article  Google Scholar 

  • Song, T., Li, L., Chen, P., et al. (2022). Blind image quality assessment for authentic distortions by intermediary enhancement and iterative training. IEEE Transactions on Circuits and Systems for Video Technology (TCSVT), 32(11), 7592–7604. https://doi.org/10.1109/TCSVT.2022.3179744

    Article  Google Scholar 

  • Song, T., Li, L., Wu, J., et al. (2022). Knowledge-guided blind image quality assessment with few training samples. IEEE Transactions on Multimedia (TMM). https://doi.org/10.1109/TMM.2022.3233244

    Article  Google Scholar 

  • Su, S., Yan, Q., Zhu, Y., et al. (2020). Blindly assess image quality in the wild guided by a self-adaptive hyper network. In Conference on computer vision and pattern recognition (CVPR).

  • Tan, M., & Le, Q. (2019). Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning (ICML) (pp. 6105–6114).

  • Thomee, B., Shamma, D. A., Friedland, G., et al. (2016). Yfcc100m: The new data in multimedia research. Communications of the ACM, 59(2), 64–73.

    Article  Google Scholar 

  • Touvron, H., Cord, M., Douze, M., et al. (2021). Training data-efficient image transformers & distillation through attention. In International conference on machine learning (ICML) (pp. 10,347–10,357)

  • Vaswani, A., Shazeer, N., Parmar, N., et al. (2017). Attention is all you need. Advances in Neural Information Processing Systems, 30. https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf

  • Wang, J., Fan, H., Hou, X., et al. (2022). Mstriq: No reference image quality assessment based on swin transformer with multi-stage fusion. In Conference on computer vision and pattern recognition workshops (CVPRW) (pp. 1268–1277). https://doi.org/10.1109/CVPRW56347.2022.00133

  • Wang, Z., & Ma, K. (2021). Active fine-tuning from gmad examples improves blind image quality assessment. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI).

  • Wang, Z., Bovik, A. C., Sheikh, H. R., et al. (2004). Image quality assessment: From error visibility to structural similarity. IEEE Transactions on Image Processing (TIP), 13(4), 600–612.

    Article  Google Scholar 

  • Wu, Q., Wang, Z., & Li, H. (2015). A highly efficient method for blind image quality assessment. In International conference on image processing (ICIP) (pp. 339–343). IEEE.

  • Xu, J., Ye, P., Li, Q., et al. (2016). Blind image quality assessment based on high order statistics aggregation. IEEE Transactions on Image Processing (TIP), 25(9), 4444–4457.

    Article  MathSciNet  MATH  Google Scholar 

  • Xue, W., Mou, X., Zhang, L., et al. (2014). Blind image quality assessment using joint statistics of gradient magnitude and laplacian features. IEEE Transactions on Image Processing (TIP), 23(11), 4850–4862. https://doi.org/10.1109/TIP.2014.2355716

    Article  MathSciNet  MATH  Google Scholar 

  • Yan, B., Bare, B., & Tan, W. (2019). Naturalness-aware deep no-reference image quality assessment. IEEE Transactions on Multimedia (TMM), 21(10), 2603–2615. https://doi.org/10.1109/TMM.2019.2904879

    Article  Google Scholar 

  • Yan, Q., Gong, D., & Zhang, Y. (2019). Two-stream convolutional networks for blind image quality assessment. IEEE Transactions on Image Processing (TIP), 28(5), 2200–2211. https://doi.org/10.1109/TIP.2018.2883741

    Article  MathSciNet  Google Scholar 

  • Yan, Q., Gong, D., & Zhang, Y. (2019). Two-stream convolutional networks for blind image quality assessment. IEEE Transactions on Image Processing (TIP), 28(5), 2200–2211. https://doi.org/10.1109/TIP.2018.2883741

    Article  MathSciNet  Google Scholar 

  • Yang, S., Wu, T., Shi, S., et al (2022). Maniqa: Multi-dimension attention network for no-reference image quality assessment. In Conference on computer vision and pattern recognition workshops (CVPRW) (pp. 1190–1199). https://doi.org/10.1109/CVPRW56347.2022.00126

  • Ye, P., Kumar, J., Kang, L., et al. (2012). Unsupervised feature learning framework for no-reference image quality assessment. In Conference on computer vision and pattern recognition (CVPR) (pp. 1098–1105). https://doi.org/10.1109/CVPR.2012.6247789

  • Yin, G., Wang, W., Yuan, Z., et al. (2022). Content-variant reference image quality assessment via knowledge distillation. In AAAI Conference on artificial intelligence (AAAI).

  • You, J., & Korhonen, J. (2021). Transformer for image quality assessment. In International conference on image processing (ICIP) (pp. 1389–1393). https://doi.org/10.1109/ICIP42928.2021.9506075

  • Zeng, H., Zhang, L., & Bovik, A. C. (2017). A probabilistic quality representation approach to deep blind image quality prediction. ar**v preprint ar**v:1708.08190

  • Zhang, L., Zhang, L., & Bovik, A. C. (2015). A feature-enriched completely blind image quality evaluator. IEEE Transactions on Image Processing (TIP), 24(8), 2579–2591.

    Article  MathSciNet  MATH  Google Scholar 

  • Zhang, W., Ma, K., Yan, J., et al. (2018). Blind image quality assessment using a deep bilinear convolutional neural network. IEEE Transactions on Circuits and Systems for Video Technology (TCSVT), 30(1), 36–47.

    Article  Google Scholar 

  • Zhang, W., Ma, K., Zhai, G., et al. (2021). Uncertainty-aware blind image quality assessment in the laboratory and wild. IEEE Transactions on Image Processing (TIP), 30, 3474–3486.

  • Zhang, W., Li, D., Ma, C., et al. (2023). Continual learning for blind image quality assessment. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 45(3), 2864–2878. https://doi.org/10.1109/TPAMI.2022.3178874

    Article  Google Scholar 

  • Zhao, K., Yuan, K., Sun, M., et al. (2023). Quality-aware pre-trained models for blind image quality assessment. In Conference on computer vision and pattern recognition (CVPR).

  • Zhu, H., Li, L., Wu, J., et al. (2020). Metaiqa: Deep meta-learning for no-reference image quality assessment. In Conference on computer vision and pattern recognition (pp. 14,143–14,152).

  • Zhu, Y., Li, Y., Sun, W., et al. (2022). Blind image quality assessment via cross-view consistency. IEEE Transactions on Multimedia (TMM) 1–14. https://doi.org/10.1109/TMM.2022.3224319

Download references

Funding

This work is supported by the National Key Research and Development Program of China (Grant No. 2020AAA0106800), the Natural Science Foundation of China (Grant No. 62202470, 61972397, 62122086, U1936204, 62036011, 62192782, 61721004, U2033210), Bei**g Natural Science Foundation (Grant No. 4224093, JQ21017, L223003), the Major Projects of Guangdong Education Department for Foundation Research and Applied Research (Grant No. 2017KZDXM081, 2018KZDXM066), Guangdong Provincial University Innovation Team Project (Project No. 2020KCXTD045) and Youth Innovation Promotion Association, CAS.

Author information

Author notes

  1. Juan Wang and Zewen Chen contributed equally to this work.

Authors and Affiliations

  1. State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Bei**g, 100190, China

    Juan Wang, Zewen Chen, Chunfeng Yuan, Bing Li & Weiming Hu

  2. School of Artificial Intelligence, University of Chinese Academy of Sciences, Bei**g, 100190, China

    Zewen Chen & Weiming Hu

  3. OPPO Corp., LTD, Shanghai, 201615, China

    Wentao Ma

  4. School of Information Science and Technology, ShanghaiTech University, Shanghai, 201210, China

    Weiming Hu

  5. People AI, Inc., Bei**g, 100080, China

    Bing Li

Authors
  1. Juan Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zewen Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Chunfeng Yuan
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Bing Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Wentao Ma
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Weiming Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Juan Wang designed the framework, performed the research and wrote the paper. Zewen Chen conducted the experiments and analyzed the results. Chunfeng Yuan reorganized the structure of the paper and carried out the experimental analysis. Bing Li refined the idea of the paper and provided some technical guidance (e.g., network design and training approach). Wentao Ma gave useful technical comments (e.g., experience in image quality assessment). Weiming Hu reviewed the paper and examined the technique details (e.g., mathematical formula and algorithm description). All authors read and approved the final manuscript.

Corresponding author

Correspondence to Chunfeng Yuan.

Ethics declarations

Conflict of interest

The authors have no competing interests to declare that are relevant to the content of this article.

Additional information

Communicated by Stephen Lin.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file 1 (pdf 423 KB)

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, J., Chen, Z., Yuan, C. et al. Hierarchical Curriculum Learning for No-Reference Image Quality Assessment. Int J Comput Vis 131, 3074–3093 (2023). https://doi.org/10.1007/s11263-023-01851-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11263-023-01851-5

Keywords

Search

Navigation