SpringerLink
Log in

An Adaptive, Energy-Efficient DRL-Based and MCMC-Based Caching Strategy for IoT Systems

  • Conference paper
  • First Online:
Algorithmic Aspects of Cloud Computing (ALGOCLOUD 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14053))

Included in the following conference series:

  • 151 Accesses

Abstract

The Internet of Things (IoT) has seen remarkable growth in recent years, but the data volatility and limited energy resources in these networks pose significant challenges. In addition, traditional quality of service metrics like throughput, latency, packet delay variation, and error rate remain important benchmarks. In this work, we explore the application of Markov Chain Monte Carlo (MCMC) methods to address these issues by designing efficient caching policies. Without the necessity for prior knowledge or context, MCMC methods provide a promising alternative to traditional caching schemes and existing machine learning models. We propose an MCMC-based caching strategy that can improve both cache hit rates and energy efficiency in IoT networks. Additionally, we introduce a hierarchical caching structure that allows parent nodes to process requests from several edge nodes and make autonomous caching decisions. Our experimental results indicate that the MCMC-based approach outperforms both traditional and other ML-based caching policies significantly. For environments where file popularity changes over time, we propose an MCMC-based adaptive caching solution. This solution detects shifts in popularity distribution using clustering and cluster similarity metrics, leading to an MCMC adaptation process. This adaptability further enhances the efficiency and effectiveness of our caching scheme, reducing training time and improving overall performance.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (Brazil)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 49.99
Price excludes VAT (Brazil)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 64.99
Price excludes VAT (Brazil)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free ship** worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Akhtar, N., Khan, M.A., Ullah, A., Javed, M.Y.: Congestion avoidance for smart devices by caching information in manets and iot. IEEE Access 7, 71459–71471 (2019)

    Article  Google Scholar 

  2. Al-Abiad, M.S., Hassan, M.Z., Hossain, M.J.: A joint reinforcement-learning enabled caching and cross-layer network code in f-ran with d2d communications. IEEE Trans. Commun. 70(7), 4400–4416 (2022)

    Article  Google Scholar 

  3. Alduayji, S., Belghith, A., Gazdar, A., Al-Ahmadi, S.: Pf-clustercache: popularity and freshness-aware collaborative cache clustering for named data networking of things. Appl. Sci. 12(13) (2022). https://doi.org/10.3390/app12136706. https://www.mdpi.com/2076-3417/12/13/6706

  4. Asmat, H., Ullah, F., Zareei, M., Khan, A., Mohamed, E.M.: Energy-efficient centrally controlled caching contents for information-centric internet of things. IEEE Access 8, 126358–126369 (2020). https://doi.org/10.1109/ACCESS.2020.3008193

    Article  Google Scholar 

  5. Bando, Y., et al.: Caching mechanisms towards single-level storage systems for internet of things. In: 2015 Symposium on VLSI Circuits (VLSI Circuits), pp. C132–C133 (2015). https://doi.org/10.1109/VLSIC.2015.7231352

  6. Bonte, P., Turck, F.D., Ongenae, F.: Bridging the gap between expressivity and efficiency in stream reasoning: a structural caching approach for IoT streams. Knowl. Inf. Syst. 64(7), 1781–1815 (2022)

    Article  Google Scholar 

  7. Chen, Y., Sun, Y., Yang, B., Taleb, T.: Joint caching and computing service placement for edge-enabled IoT based on deep reinforcement learning. IEEE Internet Things J. 9(19), 19501–19514 (2022)

    Article  Google Scholar 

  8. Coutinho, R.W.L., Boukerche, A.: Modeling and analysis of a shared edge caching system for connected cars and industrial IoT-based applications. IEEE Trans. Ind. Inf. 16(3), 2003–2012 (2020). https://doi.org/10.1109/TII.2019.2938529

    Article  Google Scholar 

  9. Elgendy, I.A., Zhang, W.Z., He, H., Gupta, B.B., Abd El-Latif, A.A.: Joint computation offloading and task caching for multi-user and multi-task mec systems: reinforcement learning-based algorithms. Wirel. Netw. 27(3), 2023–2038 (2021)

    Article  Google Scholar 

  10. Gupta, D., Rani, S., Ahmed, S.H., Verma, S., Ijaz, M.F., Shafi, J.: Edge caching based on collaborative filtering for heterogeneous ICN-IoT applications. Sensors 21(16), 5491 (2021)

    Article  Google Scholar 

  11. He, X., Wang, K., Huang, H., Miyazaki, T., Wang, Y., Guo, S.: Green resource allocation based on deep reinforcement learning in content-centric IoT. IEEE Trans. Emerg. Top. Comput. 8(3), 781–796 (2018)

    Article  Google Scholar 

  12. He, X., Wang, K., Xu, W.: QoE-driven content-centric caching with deep reinforcement learning in edge-enabled IoT. IEEE Comput. Intell. Mag. 14(4), 12–20 (2019)

    Article  Google Scholar 

  13. He, Y., Zhao, N., Yin, H.: Integrated networking, caching, and computing for connected vehicles: a deep reinforcement learning approach. IEEE Trans. Veh. Technol. 67(1), 44–55 (2017)

    Article  Google Scholar 

  14. Jazaeri, S.S., Asghari, P., Jabbehdari, S., Javadi, H.H.S.: Toward caching techniques in edge computing over SDN-IoT architecture: a review of challenges, solutions, and open issues. Multimedia Tools Appl. 1–67 (2023)

    Google Scholar 

  15. Jiang, W., Feng, G., Qin, S., Liu, Y.: Multi-agent reinforcement learning based cooperative content caching for mobile edge networks. IEEE Access 7, 61856–61867 (2019)

    Article  Google Scholar 

  16. Karras, A., Karras, C., Giotopoulos, K.C., Tsolis, D., Oikonomou, K., Sioutas, S.: Federated edge intelligence and edge caching mechanisms. Information 14(7), 414 (2023)

    Article  Google Scholar 

  17. Karras, C., Karras, A., Avlonitis, M., Sioutas, S.: An overview of MCMC methods: from theory to applications. In: Maglogiannis, I., Iliadis, L., Macintyre, J., Cortez, P. (eds.) IFIP International Conference on Artificial Intelligence Applications and Innovations, pp. 319–332. Springer, Heidelberg (2022). https://doi.org/10.1007/978-3-031-08341-9_26

  18. Khodaparas, S., Benslimane, A., Yousefi, S.: A software-defined caching scheme for the internet of things. Comput. Commun. 158, 178–188 (2020)

    Article  Google Scholar 

  19. Khodaparas, S., Yousefi, S., Benslimane, A.: A multi criteria cooperative caching scheme for internet of things. In: ICC 2019–2019 IEEE International Conference on Communications (ICC), pp. 1–6 (2019). https://doi.org/10.1109/ICC.2019.8761546

  20. Lai, L., Zheng, F.C., Wen, W., Luo, J., Li, G.: Dynamic content caching based on actor-critic reinforcement learning for IoT systems. In: 2022 IEEE 96th Vehicular Technology Conference (VTC2022-Fall), pp. 1–6. IEEE (2022)

    Google Scholar 

  21. Meddeb, M., Dhraief, A., Belghith, A., Monteil, T., Drira, K.: How to cache in ICN-based IoT environments? In: 2017 IEEE/ACS 14th International Conference on Computer Systems and Applications (AICCSA), pp. 1117–1124. IEEE (2017)

    Google Scholar 

  22. Nasehzadeh, A., Wang, P.: A deep reinforcement learning-based caching strategy for internet of things. In: 2020 IEEE/CIC International Conference on Communications in China (ICCC), pp. 969–974 (2020). https://doi.org/10.1109/ICCC49849.2020.9238811

  23. Nath, S., Wu, J.: Deep reinforcement learning for dynamic computation offloading and resource allocation in cache-assisted mobile edge computing systems. Intell. Conv. Netw. 1(2), 181–198 (2020)

    Article  Google Scholar 

  24. Niesen, U., Shah, D., Wornell, G.W.: Caching in wireless networks. IEEE Trans. Inf. Theory 58(10), 6524–6540 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  25. Nomikos, N., Zoupanos, S., Charalambous, T., Krikidis, I.: A survey on reinforcement learning-aided caching in heterogeneous mobile edge networks. IEEE Access 10, 4380–4413 (2022)

    Article  Google Scholar 

  26. Nour, B.: ICN communication optimization for Internet of Things. Ph.D. thesis, Bei**g Institute of Technology (2020)

    Google Scholar 

  27. Qiao, D., Guo, S., Liu, D., Long, S., Zhou, P., Li, Z.: Adaptive federated deep reinforcement learning for proactive content caching in edge computing. IEEE Trans. Parallel Distrib. Syst. 33(12), 4767–4782 (2022)

    Article  Google Scholar 

  28. Sadeghi, A., Wang, G., Giannakis, G.B.: Deep reinforcement learning for adaptive caching in hierarchical content delivery networks. IEEE Trans. Cogn. Commun. Netw. 5(4), 1024–1033 (2019)

    Article  Google Scholar 

  29. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., Klimov, O.: Proximal policy optimization algorithms. ar**v preprint ar**v:1707.06347 (2017)

  30. Sutar, A., Gaikwad, S., Bhadani, R., Dere, A., Domb, R., Malve, S.: Comparison and loss analysis of efficient optical routers. In: 2023 International Conference for Advancement in Technology (ICONAT), pp. 1–4 (2023). https://doi.org/10.1109/ICONAT57137.2023.10080535

  31. Tang, F., Chen, X., Rodrigues, T.K., Zhao, M., Kato, N.: Survey on digital twin edge networks (diten) toward 6G. IEEE Open J. Commun. Soc. 3, 1360–1381 (2022)

    Article  Google Scholar 

  32. Vlachou, E., Karras, C., Karras, A., Tsolis, D., Sioutas, S.: EVCA classifier: a MCMC-based classifier for analyzing high-dimensional big data. Information 14(8) (2023). https://doi.org/10.3390/info14080451. https://www.mdpi.com/2078-2489/14/8/451

  33. Wang, S., Zhang, X., Zhang, Y., Wang, L., Yang, J., Wang, W.: A survey on mobile edge networks: convergence of computing, caching and communications. IEEE Access 5, 6757–6779 (2017)

    Article  Google Scholar 

  34. Wang, X., Li, R., Wang, C., Li, X., Taleb, T., Leung, V.C.: Attention-weighted federated deep reinforcement learning for device-to-device assisted heterogeneous collaborative edge caching. IEEE J. Sel. Areas Commun. 39(1), 154–169 (2020)

    Article  Google Scholar 

  35. Wang, X., Wang, C., Li, X., Leung, V.C., Taleb, T.: Federated deep reinforcement learning for internet of things with decentralized cooperative edge caching. IEEE Internet Things J. 7(10), 9441–9455 (2020)

    Article  Google Scholar 

  36. Wei, Y., Yu, F.R., Song, M., Han, Z.: Joint optimization of caching, computing, and radio resources for fog-enabled IoT using natural actor-critic deep reinforcement learning. IEEE Internet Things J. 6(2), 2061–2073 (2018)

    Article  Google Scholar 

  37. Wu, B., Xu, K., Li, Q., Ren, S., Liu, Z., Zhang, Z.: Toward blockchain-powered trusted collaborative services for edge-centric networks. IEEE Netw. 34(2), 30–36 (2020)

    Article  Google Scholar 

  38. Wu, H., Nasehzadeh, A., Wang, P.: A deep reinforcement learning-based caching strategy for IoT networks with transient data. IEEE Trans. Veh. Technol. 71(12), 13310–13319 (2022). https://doi.org/10.1109/TVT.2022.3199677

    Article  Google Scholar 

  39. Wu, Q., Zhao, Y., Fan, Q., Fan, P., Wang, J., Zhang, C.: Mobility-aware cooperative caching in vehicular edge computing based on asynchronous federated and deep reinforcement learning. IEEE J. Sel. Topics Signal Process. 17(1), 66–81 (2022)

    Article  Google Scholar 

  40. Yao, J., Ansari, N.: Caching in dynamic IoT networks by deep reinforcement learning. IEEE Internet Things J. 8(5), 3268–3275 (2020). https://doi.org/10.1109/JIOT.2020.3004394

    Article  Google Scholar 

  41. Zahmatkesh, H., Al-Turjman, F.: Fog computing for sustainable smart cities in the IoT era: caching techniques and enabling technologies-an overview. Sustain. Urban Areas 59, 102139 (2020)

    Google Scholar 

  42. Zameel, A., Najmuldeen, M., Gormus, S.: Context-aware caching in wireless IoT networks. In: 2019 11th International Conference on Electrical and Electronics Engineering (ELECO), pp. 712–717 (2019). https://doi.org/10.23919/ELECO47770.2019.8990647

  43. Zhang, Y., et al.: Cooperative edge caching: a multi-agent deep learning based approach. IEEE Access 8, 133212–133224 (2020)

    Article  Google Scholar 

  44. Zhong, C., Gursoy, M.C., Velipasalar, S.: A deep reinforcement learning-based framework for content caching. In: 2018 52nd Annual Conference on Information Sciences and Systems (CISS), pp. 1–6. IEEE (2018)

    Google Scholar 

  45. Zhu, H., Cao, Y., Wang, W., Jiang, T., **, S.: Deep reinforcement learning for mobile edge caching: review, new features, and open issues. IEEE Netw. 32(6), 50–57 (2018)

    Article  Google Scholar 

Download references

Acknowledgments

The financial support of the European Union and Greece (Partnership Agreement for the Development Framework 2014–2020) under the Regional Operational Programme Ionian Islands 2014–2020 for the project “Laertis” is gratefully acknowledged.

Author information

Authors and Affiliations

  1. Computer Engineering and Informatics Department, University of Patras, 26504, Patras, Greece

    Aristeidis Karras, Christos Karras & Spyros Sioutas

  2. Department of Informatics, Ionian University, 49100, Corfu, Greece

    Ioannis Karydis & Markos Avlonitis

Authors
  1. Aristeidis Karras
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Christos Karras
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ioannis Karydis
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Markos Avlonitis
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Spyros Sioutas
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Aristeidis Karras .

Editor information

Editors and Affiliations

  1. Sapienza University of Rome, Rome, Italy

    Ioannis Chatzigiannakis

  2. Ionian University, Corfu, Greece

    Ioannis Karydis

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Karras, A., Karras, C., Karydis, I., Avlonitis, M., Sioutas, S. (2024). An Adaptive, Energy-Efficient DRL-Based and MCMC-Based Caching Strategy for IoT Systems. In: Chatzigiannakis, I., Karydis, I. (eds) Algorithmic Aspects of Cloud Computing. ALGOCLOUD 2023. Lecture Notes in Computer Science, vol 14053. Springer, Cham. https://doi.org/10.1007/978-3-031-49361-4_4

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-49361-4_4

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-49360-7

  • Online ISBN: 978-3-031-49361-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation