SpringerLink
Log in

Impact of Autotuned Fully Connected Layers on Performance of Self-supervised Models for Image Classification

  • Research Article
  • Published:
Machine Intelligence Research Aims and scope Submit manuscript

Abstract

With the recent advancements of deep learning-based methods in image classification, the requirement of a huge amount of training data is inevitable to avoid overfitting problems. Moreover, supervised deep learning models require labelled datasets for training. Preparing such a huge amount of labelled data requires considerable human effort and time. In this scenario, self-supervised models are becoming popular because of their ability to learn even from unlabelled datasets. However, the efficient transfer of knowledge learned by self-supervised models into a target task, is an unsolved problem. This paper proposes a method for the efficient transfer of knowledge learned by a self-supervised model, into a target task. Hyperparameters such as the number of layers, the number of units in each layer, learning rate, and dropout are automatically tuned in these fully connected (FC) layers using a Bayesian optimization technique called the tree-structured parzen estimator (TPE) approach algorithm. To evaluate the performance of the proposed method, state-of-the-art self-supervised models such as SimClr and SWAV are used to extract the learned features. Experiments are carried out on the CIFAR-10, CIFAR-100, and Tiny ImageNet datasets. The proposed method outperforms the baseline approach with margins of 2.97%, 2.45%, and 0.91% for the CIFAR-100, Tiny ImageNet, and CIFAR-10 datasets, respectively.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price includes VAT (Germany)

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. R. Zhang, P. Isola, A. A. Efros. Colorful image colorization. In Proceedings of the 14th European Conference on Computer Vision, Springer, Amsterdam, The Netherlands, pp. 649–666, 2016. DOI: https://doi.org/10.1007/978-3-319-46487-9_40.

    Google Scholar 

  2. C. Doersch, A. Gupta, A. A. Efros. Unsupervised visual representation learning by context prediction. In Proceedings of IEEE International Conference on Computer Vision, IEEE, Santiago, Chile, pp. 1422–1430, 2015. DOI: https://doi.org/10.1109/ICCV.2015.167.

    Google Scholar 

  3. M. Noroozi, P. Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In Proceedings of the 14th European Conference on Computer Vision, Springer, Amsterdam, The Netherlands, pp. 69–84, 2016. DOI: https://doi.org/10.1007/978-3-319-46466-4_5.

    Google Scholar 

  4. D. Pathak, P. Krähenbühl, J. Donahue, T. Darrell, A. A. Efros. Context encoders: Feature learning by inpainting. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, IEEE, Las Vegas, USA, pp. 2536–2544, 2016. DOI: https://doi.org/10.1109/CVPR.2016.278.

    Google Scholar 

  5. S. Gidaris, P. Singh, N. Komodakis. Unsupervised representation learning by predicting image rotations. In Proceedings of the 6th International Conference on Learning Representations, Vancouver, Canada, 2018.

  6. I. Misra, C. L. Zitnick, M. Hebert. Shuffle and learn: Unsupervised learning using temporal order verification. In Proceedings of the 14th European Conference on Computer Vision, Springer, Amsterdam, The Netherlands, pp. 527–544, 2016. DOI: https://doi.org/10.1007/978-3-319-46448-0_32.

    Google Scholar 

  7. T. Chen, S. Kornblith, M. Norouzi, G. Hinton. A simple framework for contrastive learning of visual representations. In Proceedings of the 37th International Conference on Machine Learning, Article number 149, 2020.

  8. K. M. He, H. Q. Fan, Y. X. Wu, S. N. **e, R. Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, Seattle, USA, pp. 9726–9735, 2020. DOI: https://doi.org/10.1109/CVPR42600.2020.00975.

    Google Scholar 

  9. X. L. Chen, H. Q. Fan, R. Girshick, K. M. He. Improved baselines with momentum contrastive learning, [Online], Available: https://arxiv.org/abs/2003.04297, 2020.

  10. M. Caron, I. Misra, J. Mairal, P. Goyal, P. Bojanowski, A. Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In Proceedings of the 34th International Conference on Neural Information Processing Systems, Vancouver, Canada, Article No. 831, 2020.

  11. J. Bergstra, R. Bardenet, Y. Bengio, B. Kégl. Algorithms for hyper-parameter optimization. In Proceedings of the 24th International Conference on Neural Information Processing Systems, Granada, Spain, pp. 2546–2554, 2011.

  12. T. Elsken, J. H. Metzen, F. Hutter. Neural architecture search: A survey. The Journal of Machine Learning Research, vol. 20, no. 1, pp. 1997–2017, 2019.

    MathSciNet  Google Scholar 

  13. M. Wistuba, A. Rawat, T. Pedapati. A survey on neural architecture search, [Online], Available: https://arxiv.org/abs/1905.01392, 2019.

  14. S. Kaplan, R. Giryes. Self-supervised neural architecture search, [Online], Available:https://arxiv.org/abs/2007.01500, 2020.

  15. S. H. S. Basha, S. K. Vinakota, S. R. Dubey, V. Pulabaigari, S. Mukherjee. AutoFCL: Automatically tuning fully connected layers for handling small dataset. Neural Computing and Applications, vol. 33, no. 13, pp. 8055–8065, 2021. DOI: https://doi.org/10.1007/s00521-020-05549-4.

    Article  Google Scholar 

  16. S. H. S. Basha, S. K. Vinakota, V. Pulabaigari, S. Mukherjee, S. R. Dubey. AutoTune: Automatically tuning convolutional neural networks for improved transfer learning. Neural Networks, vol. 133, pp. 112–122, 2021. DOI: https://doi.org/10.1016/j.neunet.2020.10.009.

    Article  Google Scholar 

  17. L. Yang, A. Shami. On hyperparameter optimization of machine learning algorithms: Theory and practice. Neurocomputing, vol. 415, pp. 295–316, 2020. DOI: https://doi.org/10.1016/j.neucom.2020.07.061.

    Article  Google Scholar 

  18. D. Baymurzina, E. Golikov, M. Burtsev. A review of neural architecture search. Neurocomputing, vol. 474, pp. 82–93, 2022. DOI: https://doi.org/10.1016/j.neucom.2021.12.014.

    Article  Google Scholar 

  19. C. X. Liu, B. Zoph, M. Neumann, J. Shlens, W. Hua, L. J. Li, L. Fei-Fei, A. Yuille, J. Huang, K. Murphy. Progressive neural architecture search. In Proceedings of the 15th European Conference on Computer Vision, Springer, Munich, Germany, pp. 19–35, 2018. DOI: https://doi.org/10.1007/978-3-030-01246-5_2.

    Google Scholar 

  20. B. Zoph, V. Vasudevan, J. Shlens, Q. V. Le. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, Salt Lake City, USA, pp. 8697–8710, 2018. DOI: https://doi.org/10.1109/CVPR.2018.00907.

    Google Scholar 

  21. H. X. Liu, K. Simonyan, Y. M. Yang. DARTS: Differentiable architecture search. In Proceedings of the 7th International Conference on Learning Representations, New Orleans, USA, 2019.

  22. D. Polap, M. Wozniak, W. Holubowski, R. Damaševičius. A heuristic approach to the hyperparameters in training spiking neural networks using spike-timing-dependent plasticity. Neural Computing and Applications, vol. 34, no. 16, pp. 13187–13200, 2021. DOI: https://doi.org/10.1007/s00521-021-06824-8.

    Article  Google Scholar 

  23. M. Subramanian, K. Shanmugavadivel, P. S. Nandhini. On fine-tuning deep learning models using transfer learning and hyper-parameters optimization for disease identification in maize leaves. Neural Computing and Applications, vol. 34, no. 16, pp. 13951–13968, 2022. DOI: https://doi.org/10.1007/s00521-022-07246-w.

    Article  Google Scholar 

  24. B. Zoph, Q. V. Le. Neural architecture search with reinforcement learning. In Proceedings of the 5th International Conference on Learning Representation, Toulon, France, 2017.

  25. X. Y. Gong, S. Y. Chang, Y. F. Jiang, Z. Y. Wang. AutoGAN: Neural architecture search for generative adversarial networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, IEEE, Seoul, Republic of Korea, pp. 3223–3233, 2019. DOI: https://doi.org/10.1109/IC-CV.2019.00332.

    Google Scholar 

  26. J. Y. Li, Z. H. Zhan, J. Zhang. Evolutionary computation for expensive optimization: A survey. Machine Intelligence Research, vol. 19, no. 1, pp. 3–23, 2022. DOI: https://doi.org/10.1007/s11633-022-1317-4.

    Article  Google Scholar 

  27. X. Y. Dong, L. Liu, K. Musial, B. Gabrys. NATS-bench: Benchmarking NAS algorithms for architecture topology and size. IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 7, pp. 3634–3646, 2022. DOI: https://doi.org/10.1109/TPAMI.2021.3054824.

    Google Scholar 

  28. D. M. Han, Q. G. Liu, W. G. Fan. A new image classification method using CNN transfer learning and web data augmentation. Expert Systems with Applications, vol. 95, pp. 43–56, 2018. DOI: https://doi.org/10.1016/j.eswa.2017.11.028.

    Article  Google Scholar 

  29. H. Mendoza, A. Klein, M. Feurer, J. T. Springenberg, F. Hutter. Towards automatically-tuned neural networks. In Proceedings of the Workshop on Automatic Machine Learning, New York, USA, pp. 58–65, 2016.

  30. P. I. Frazier. A tutorial on Bayesian optimization, [Online], Available: https://arxiv.org/abs/1807.02811, 2018

  31. H. Cho, Y. Kim, E. Lee, D. Choi, Y. Lee, W. Rhee. Basic enhancement strategies when using Bayesian optimization for hyperparameter tuning of deep neural networks. IEEE Access, vol. 8, pp. 52588–52608, 2020. DOI: https://doi.org/10.1109/ACCESS.2020.2981072.

    Article  Google Scholar 

  32. A. H. Victoria, G. Maragatham. Automatic tuning of hyperparameters using Bayesian optimization. Evolving Systems, vol. 12, no. 1, pp. 217–223, 2021. DOI: https://doi.org/10.1007/s12530-020-09345-2.

    Article  Google Scholar 

  33. H. P. Nguyen, J. Liu, E. Zio. A long-term prediction approach based on long short-term memory neural networks with automatic parameter optimization by tree-structured parzen estimator and applied to time-series data of NPP steam generators. Applied Soft Computing, vol. 89, Article number 106116, 2020. DOI: https://doi.org/10.1016/j.asoc.2020.106116.

  34. X. L. Liang, Y. Liu, J. H. Luo, Y. Q. He, T. J. Chen, Q. Yang. Self-supervised cross-silo federated neural architecture search, [Online], Available: https://arxiv.org/abs/2101.11896, 2021.

  35. S. H. S. Basha, S. R. Dubey, V. Pulabaigari, S. Mukherjee. Impact of fully connected layers on performance of convolutional neural networks for image classification. Neurocomputing, vol. 378, pp. 112–119, 2020. DOI: https://doi.org/10.1016/j.neucom.2019.10.008.

    Article  Google Scholar 

  36. A. Krizhevsky, I. Sutskever, G. E. Hinton. ImageNet classification with deep convolutional neural networks. In Proceedings of the 25th International Conference on Neural Information Processing Systems, Lake Tahoe, USA, pp. 1097–1105, 2012.

  37. K. Simonyan, A. Zisserman. Very deep convolutional networks for large-scale image recognition, [Online], Available: https://arxiv.org/abs/1409.1556, 2014.

  38. E. A. Falcon. WA: Pytorch lightning. GitHub, [Online], Available: https://github.com/PyTorchLightning/pytorch-lightning, 3, 2019.

  39. W. Falcon, K. Cho. A framework for contrastive self-supervised learning and designing a new approach, [Online], Available: https://arxiv.org/abs/2009.00104, 2020.

  40. T. Akiba, S. Sano, T. Yanase, T. Ohta, M. Koyama. Optuna: A next-generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, ACM, Anchorage, USA, pp. 2623–2631, 2019. DOI: https://doi.org/10.1145/3292500.3330701.

    Chapter  Google Scholar 

  41. O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. A. Ma, Z. H. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, L. Fei-Fei. ImageNet large scale visual recognition challenge. International Journal of Computer Vision, vol. 115, no. 3, pp. 211–252, 2015. DOI: https://doi.org/10.1007/s11263-015-0816-y.

    Article  MathSciNet  Google Scholar 

  42. A. Krizhevsky. Learning Multiple Layers of Features from tiny Images, Master dissertation, Department of Computer Science, University of Toronto, Canada, 2009.

    Google Scholar 

  43. Y. Le, X. Yang. Tiny imagenet visual recognition challenge. CS 231N, vol. 7, no. 7, Article number 3, 2015.

  44. K. M. He, X. Y. Zhang, S. Q. Ren, J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, IEEE, Las Vegas, USA, pp. 770–778, 2016. DOI: https://doi.org/10.1109/CVPR.2016.90.

    Google Scholar 

  45. P. A. Knight. The sinkhorn–knopp algorithm: Convergence and applications. SIAM Journal on Matrix Analysis and Applications, vol. 30, no. 1, pp. 261–275, 2008. DOI: https://doi.org/10.1137/060659624.

    Article  MathSciNet  Google Scholar 

  46. P. Bachman, R. D. Hjelm, W. Buchwalter. Learning representations by maximizing mutual information across views. In Proceedings of the 33rd International Conference on Neural Information Processing Systems, Vancouver, Canada, Article number 1392, 2019.

  47. J. B. Grill, F. Strub, F. Altché, C. Tallec, P. H. Richemond, E. Buchatskaya, C. Doersch, B. A. Pires, Z. D. Guo, M. G. Azar, B. Piot, K. Kavukcuoglu, R. Munos, M. Valko. Bootstrap your own latent a new approach to self-supervised learning. In Proceedings of the 34th International Conference on Neural Information Processing Systems, Vancouver, Canada, Article No. 1786, 2020.

  48. X. L. Chen, K. M. He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, Nashville, USA, pp. 15745–15753, 2021. DOI: https://doi.org/10.1109/CV-PR46437.2021.01549.

    Google Scholar 

  49. Y. Zhong, H. Tang, J. Chen, J. Peng, Y. X. Wang. Is self-supervised contrastive learning more robust than supervised learning?. In Proceedings of the 1st Workshop on Pre-training: Perspectives, Pitfalls, and Paths Forward at ICML, 2022.

  50. S. C. Ren, H. Y. Wang, Z. Q. Gao, S. F. He, A. Yuille, Y. Y. Zhou, C. H. **e. A simple data mixing prior for improving self-supervised learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, New Orleans, USA, pp. 14575–14584, 2022. DOI: https://doi.org/10.1109/CVPR52688.2022.01419.

    Google Scholar 

  51. C. H. Tseng, S. J. Lee, J. N. Feng, S. Z. Mao, Y. P. Wu, J. Y. Shang, M. C. Tseng, X. J. Zeng. UPANets: Learning from the universal pixel attention networks, [Online], Available: https://arxiv.org/abs/2103.08640, 2021.

  52. Z. C. Liu, S. Y. Li, D. Wu, Z. H. Liu, Z. Y. Chen, L. R. Wu, S. Z. Li. AutoMix: Unveiling the power of mixup for stronger classifiers. In Proceedings of the 17th European Conference on Computer Vision, Springer, Tel Aviv, Israel, pp. 441–458, 2022. DOI: https://doi.org/10.1007/978-3-031-20053-3_26.

    Google Scholar 

  53. J. H. Kim, W. Choo, H. O. Song. Puzzle mix: Exploiting saliency and local statistics for optimal mixup. In Proceedings of the 37th International Conference on Machine Learning, Article number 489, 2020.

  54. M. Z. Alom, M. Hasan, C. Yakopcic, T. M. Taha, V. K. Asari. Improved inception-residual convolutional neural network for object recognition. Neural Computing and Applications, vol. 32, no. 1, pp. 279–293, 2020. DOI: https://doi.org/10.1007/s00521-018-3627-6.

    Article  Google Scholar 

  55. H. Lee, S. J. Hwang, J. Shin. Self-supervised label augmentation via input transformations. In Proceedings of the 37th International Conference on Machine Learning, Article number 530, 2020.

  56. D. X. Yao, L. Y. **ang, Z. F. Wang, J. Y. Xu, C. Li, X. B. Wang. Context-aware compilation of DNN training pipelines across edge and cloud. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, vol. 5, no. 4, Article number Article number 188, 2021. DOI: https://doi.org/10.1145/3494981.

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science and Engineering, Shiv Nadar Institute of Eminence, Delhi NCR, 201314, India

    Jaydeep Kishore & Snehasis Mukherjee

Authors
  1. Jaydeep Kishore
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Snehasis Mukherjee
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Snehasis Mukherjee.

Ethics declarations

The authors declared that they have no conflicts of interest to this work.

Additional information

Colored figures are available in the online version at https://springer.longhoe.net/journal/11633

Jaydeep Kishore received the B. Tech. degree in computer science and engineering (CSE) from Uttar Pradesh Technical University, India in 2005, received the M.Tech. degree in computer science and engineering from Uttarakhand Technical University, India in 2010. Currently, he is a Ph.D. degree candidate in Department of computer science and engineering, Shiv Nadar University, India. He has 15 years of academic experience in various institutions.

His research interests include computer vision and neural architecture search.

Snehasis Mukherjee received the Ph.D. degree in computer science from Indian Statistical Institute, India in 2012. Currently, he is an assistant professor at Shiv Nadar University, India since May 2020. He worked as a post doctoral fellow at National Institute Standards and Technology, USA. Then he spent 6 years as an assistant professor at IIIT Sri City, India till April 2020. He has authored several peer-reviewed research papers in reputed journals and conferences. He is an associate editor of the Springer Journal of SN Computer Science. He is an active reviewer of several reputed journals such as IEEE TNNLS, IEEE TCSVT, IEEE TIP, IEEE TETCI, IEEE THMS, IEEE TCyb, IEEE CIM, CVIU, Pattern Recognition, Neural Networks, Neurocomputing, and many more. He chaired sessions in several prestigious conferences such as ICARCV, ICVGIP and NCV-PRIPG.

His research interests include computer vision, machine learning, image processing and graphics.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kishore, J., Mukherjee, S. Impact of Autotuned Fully Connected Layers on Performance of Self-supervised Models for Image Classification. Mach. Intell. Res. (2024). https://doi.org/10.1007/s11633-023-1435-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11633-023-1435-7

Keywords

Search

Navigation