SpringerLink
Log in

Diversified deep hierarchical kernel ensemble regression

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Deep ensemble learning models that combine multiple independent deep learning models with multi-layer processing architectures have proven to be effective techniques for improving the accuracy and robustness of deep learning models. In this paper, we propose a diversified kernel ensemble regression method, which is developed from the well-known kernel ridge regression methods. Motivated by multi-view data modeling ideas, we treat each individual kernel as one view of original data in kernel representation space. Therefore, we develop a deep kernel ensemble ridge regression method in Neural Tangent Kernel (NTK) to address the problem faced by traditional kernel ridge regression methods in finding appropriate types of kernels and their parameters. Specifically, as multiple deep kernel regressors share common information from the multi-view kernel representations, our proposed method is built through a hierarchical modeling method, where deep kernel regressors share a common parameter and also have model-specific parameters in individual regressors, which is further helpful in improving our model’s performance. Furthermore, to achieve better diversified deep kernel representations in our proposed method, the Hilbert-Schmidt Independence Criterion (HSIC) is used to regularize our proposed model. In this way, we can find more diversified kernel representations among multiple kernel ensemble regressors to achieve better generalization performance. Experiments on several classification and regression datasets, such as MNIST, TinyImageNet-200, ORL, Credit and YearPredictionMSD demonstrate that our proposed method can achieve best regression performance than other state-of-the-art methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (Germany)

Instant access to the full article PDF.

Institutional subscriptions

Algorithm 1
Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Data Availability

The UCI datasets analysed during this study are available in the UCI repository, http://archive.ics.uci.edu/. And other datasets analysed during this study are available from the corresponding author on reasonable request.

References

  1. Borandağ E, Özçift A, Kaygusuz Y (2021) Development of majority vote ensemble feature selection algorithm augmentedwith rank allocation to enhance turkish text categorization. Turk J Electr Eng Comput Sci 29(2):514–530

  2. Yu J, Cai Z, He P, **e G, Ling Q (2022) Multi-model ensemble learning method for human expression recognition. ar**v preprint ar**v:2203.14466

  3. Ahn E, Kumar A, Feng D, Fulham M, Kim J (2019) Unsupervised feature learning with k-means and an ensemble of deep convolutional neural networks for medical image classification. ar**v preprint ar**v:1906.03359

  4. Kazemi S, Minaei Bidgoli B, Shamshirband S, Karimi SM, Ghorbani MA, Chau K-W, Kazem Pour R (2018) Novel genetic-based negative correlation learning for estimating soil temperature. Eng Appl Comput Fluid Mech 12(1):506–516

  5. Wu Y, Liu L, **e Z, Chow K-H, Wei W (2021) Boosting ensemble accuracy by revisiting ensemble diversity metrics. In: Proc IEEE Int Conf Comput Vis Pattern Recognit pp 16469–16477

  6. Bartlett P, Freund Y, Lee WS, Schapire RE (1998) Boosting the margin: A new explanation for the effectiveness of voting methods. Ann Stat 26(5):1651–1686

    Article  MathSciNet  Google Scholar 

  7. Breiman L (1996) Bagging predictors. Mach Learn 24(2):123–140

    Article  Google Scholar 

  8. Breiman L (2001) Random forests. Mach Learn 45(1):5–32

    Article  Google Scholar 

  9. Freund Y, Schapire RE (1996) Experiments with a new boosting algorithm. In: Icml, vol 96, pp 148–156. Citeseer

  10. Friedman JH (2001) Greedy function approximation: a gradient boosting machine. Annals of statistics, 1189–1232

  11. Wang R, Kwong S, Wang X, Jia Y (2021) Active k-labelsets ensemble for multi-label classification. Pattern Recogn 109:107583

    Article  Google Scholar 

  12. Wang B, Xue B, Zhang M (2020) Particle swarm optimisation for evolving deep neural networks for image classification by evolving and stacking transferable blocks. In: 2020 IEEE Congr Evol Comput (CEC) pp 1–8. IEEE

  13. Liu B, Gu L, Lu F (2019) Unsupervised ensemble strategy for retinal vessel segmentation. In: Int Conf Med Image Comput Comput Assist Interv, pp 111–119. Springer

  14. Ali F, El-Sappagh S, Islam SR, Kwak D, Ali A, Imran M, Kwak K-S (2020) A smart healthcare monitoring system for heart disease prediction based on ensemble deep learning and feature fusion. Inf Fusion 63:208–222

  15. Zhang W, Jiang J, Shao Y, Cui B (2020) Snapshot boosting: a fast ensemble framework for deep neural networks. Sci China Inf Sci 63(1):1–12

  16. Zhang S, Liu M, Yan J (2020) The diversified ensemble neural network. Adv Neural Inf Process Syst 33:16001–16011

    Google Scholar 

  17. Lakshminarayanan B, Pritzel A, Blundell C (2017) Simple and scalable predictive uncertainty estimation using deep ensembles. Adv Neural Inf Process Syst 30

  18. Bhadra S, Kaski S, Rousu J (2017) Multi-view kernel completion. Mach Learn 106(5):713–739

    Article  MathSciNet  Google Scholar 

  19. Khan GA, Hu J, Li T, Diallo B, Zhao Y (2022) Multi-view low rank sparse representation method for three-way clustering. Int J Mach Learn Cybern 13(1):233–253

    Article  Google Scholar 

  20. Jacot A, Gabriel F, Hongler C (2018) Neural tangent kernel: Convergence and generalization in neural networks. Adv Neural Inf Process Syst 31

  21. Gretton A, Bousquet O, Smola A, Schölkopf B (2005) Measuring statistical dependence with hilbert-schmidt norms. In: Int Conf Algorithmic Learning Theory, pp 63–77. Springer

  22. Mukkamala S, Sung AH, Abraham A (2003) Intrusion detection using ensemble of soft computing paradigms. In: Intell Syst Design Appl, pp 239–248. Springer, ???

  23. Chen T, Guestrin C (2016) Xgboost: A scalable tree boosting system. In: Proceed 22nd Acm Sigkdd Int Conf Knowl Discov Data Min, pp 785–794

  24. Ke G, Meng Q, Finley T, Wang T, Chen W, Ma W, Ye Q, Liu T-Y (2017) Lightgbm: A highly efficient gradient boosting decision tree. Adv Neural Inf Process Syst 30

  25. Pal M (2005) Random forest classifier for remote sensing classification. Int J Remote Sens 26(1):217–222

    Article  Google Scholar 

  26. Ogunleye A, Wang Q-G (2019) Xgboost model for chronic kidney disease diagnosis. IEEE/ACM Trans Comput Biol Bioinf 17(6):2131–2140

    Article  Google Scholar 

  27. Shi Z, Zhang L, Liu Y, Cao X, Ye Y, Cheng M-M, Zheng G (2018) Crowd counting with deep negative correlation learning. In: Proceed IEEE Conf Comput Vis Pattern Recognit, pp 5382–5390

  28. Xue J, Wang Z, Kong D, Wang Y, Liu X, Fan W, Yuan S, Niu S, Li D (2021) Deep ensemble neural-like p systems for segmentation of central serous chorioretinopathy lesion. Inf Fusion 65:84–94

  29. Lin Z, Chen M, Ma Y (2010) The augmented lagrange multiplier method for exact recovery of corrupted low-rank matrices. ar**v preprint ar**v:1009.5055

  30. Schölkopf B, Smola AJ, Bach F (2002) Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond. MIT press, ???

  31. Vorontsov MA, Sivokon VP (1998) Stochastic parallel-gradient-descent technique for high-resolution wave-front phase-distortion correction. JOSA A 15(10):2745–2758

    Article  Google Scholar 

  32. Ma C, Qiu X, Beutel D, Lane N (2023) Gradient-less federated gradient boosting tree with learnable learning rates. In: Proceed 3rd Workshop Mach Learn Syst, pp 56–63

  33. Lalev A, Alexandrova A (2023) Recurrent neural networks for forecasting social processes. In: 2023 Int Conf Big Data Knowl Control Syst Eng (BdKCSE), pp 1–5. IEEE

  34. Wan A, Dunlap L, Ho D, Yin J, Lee S, ** H, Petryk S, Bargal SA, Gonzalez JE (2020) Nbdt: neural-backed decision trees. ar**v preprint ar**v:2004.00221

  35. Luo ZT, Sang H, Mallick B (2022) Bamdt: Bayesian additive semi-multivariate decision trees for nonparametric regression. In: Int Conf Mach Learn, pp 14509–14526. PMLR

  36. Zhou Z-H, Feng J (2019) Deep forest. National Science Review 6(1):74–86

    Article  MathSciNet  Google Scholar 

  37. Fang C, Cheng L, Mao Y, Zhang D, Fang Y, Li G, Qi H, Jiao L (2023) Separating noisy samples from tail classes for long-tailed image classification with label noise. IEEE Trans Neural Netw Learn Syst

  38. Fonti V, Belitser E (2017) Feature selection using lasso. VU Amsterdam research paper in business analytics 30:1–25

    Google Scholar 

  39. Fang C, Wang Q, Cheng L, Gao Z, Pan C, Cao Z, Zheng Z, Zhang D (2023) Reliable mutual distillation for medical image segmentation under imperfect annotations. IEEE Trans Med Imaging

  40. Yao J, Han L, Guo G, Zheng Z, Cong R, Huang X, Ding J, Yang K, Zhang D, Han J (2024) Position-based anchor optimization for point supervised dense nuclei detection. Neural Netw 171:159–170

    Article  Google Scholar 

Download references

Author information

Author notes

  1. These authors contributed equally to this work.

Authors and Affiliations

  1. School of Computer Science and Communication Engineering, JiangSu University, Zhenjiang, 212013, China

    Zhifeng Liu, Zhengqin Xu, Stanley Ebhohimhen Abhadiomhen & **ang-Jun Shen

  2. Department of Computer Science, University of Nigeria, Nsukka, 410105, Nigeria

    Stanley Ebhohimhen Abhadiomhen

  3. Department of Ultrasound, Affiliated People’s Hospital of Jiangsu University, Zhenjiang, 212002, China

    **aoqin Qian

Authors
  1. Zhifeng Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zhengqin Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Stanley Ebhohimhen Abhadiomhen
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. **aoqin Qian
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. **ang-Jun Shen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to **ang-Jun Shen.

Ethics declarations

Conflicts of interest

This work was funded in part by the National Natural Science Foundation of China (62376108).

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liu, Z., Xu, Z., Ebhohimhen Abhadiomhen, S. et al. Diversified deep hierarchical kernel ensemble regression. Multimed Tools Appl (2024). https://doi.org/10.1007/s11042-024-19637-3

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11042-024-19637-3

Keywords

Search

Navigation