SpringerLink
Log in

Deep Reinforcement Learning and Markov Decision Problem for Task Offloading in Mobile Edge Computing

  • Research
  • Published:
Journal of Grid Computing Aims and scope Submit manuscript

Abstract

Mobile Edge Computing (MEC) offers cloud-like capabilities to mobile users, making it an up-and-coming method for advancing the Internet of Things (IoT). However, current approaches are limited by various factors such as network latency, bandwidth, energy consumption, task characteristics, and edge server overload. To address these limitations, this research propose a novel approach that integrates Deep Reinforcement Learning (DRL) with Deep Deterministic Policy Gradient (DDPG) and Markov Decision Problem for task offloading in MEC. Among DRL algorithms, the ITODDPG algorithm based on the DDPG algorithm and MDP is a popular choice for task offloading in MEC. Firstly, the ITODDPG algorithm formulates the task offloading problem in MEC as an MDP, which enables the agent to learn a policy that maximizes the expected cumulative reward. Secondly, ITODDPG employs a deep neural network to approximate the Q-function, which maps the state-action pairs to their expected cumulative rewards. Finally, the experimental results demonstrate that the ITODDPG algorithm outperforms the baseline algorithms regarding average compensation and convergence speed. In addition to its superior performance, our proposed approach can learn complex non-linear policies using DNN and an information-theoretic objective function to improve the performance of task offloading in MEC. Compared to traditional methods, our approach delivers improved performance, making it highly effective for develo** IoT environments. Experimental trials were carried out, and the results indicate that the suggested approach can enhance performance compared to the other three baseline methods. It is highly scalable, capable of handling large and complex environments, and suitable for deployment in real-world scenarios, ensuring its widespread applicability to a diverse range of task offloading and MEC applications.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Data Availability

The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.

References

  1. Li, B., Zhou, X., Ning, Z., Guan, X., Yiu, K.C.: Dynamic event-triggered security control for networked control systems with cyber-attacks: A model predictive control approach. Inf. Sci. 612, 384–398 (2022)

    Article  Google Scholar 

  2. Nath, S., Wu, J.: Deep reinforcement learning for dynamic computation offloading and resource allocation in cache-assisted mobile edge computing systems. Intell. Converged Netw. 1(2), 181–198 (2020)

    Article  Google Scholar 

  3. Liu, X., Jiang, S., Wu, Y.: A novel deep reinforcement learning approach for task offloading in MEC systems. Appl. Sci. 12(21), 11260 (2022)

    Article  Google Scholar 

  4. Liang, X., Huang, Z., Yang, S., Qiu, L.: Device-free motion & trajectory detection via RFID. ACM Trans. Embed. Comput. Syst. 17(4), 78 (2018)

    Article  Google Scholar 

  5. Zhang, B., Zhang, G., Sun, W., Yang, K.: Task offloading with power control for mobile edge computing using reinforcement learning-based markov decision process. Mob. Inf. Syst. 2020, 1–6 (2020)

    Google Scholar 

  6. Qian, L., Zheng, Y., Li, L., Ma, Y., Zhou, C.,... Zhang, D.: A new method of inland water ship trajectory prediction based on long short-term memory network optimized by genetic algorithm. Appl. Sci. 12(8), 4073 (2022)

  7. Zhang, X., Wang, Y., Yuan, X., Shen, Y., Lu, Z.,... Wang, Z.: Adaptive Dynamic Surface Control with Disturbance Observers for Battery/Supercapacitor-based Hybrid Energy Sources in Electric Vehicles. IEEE Trans. Transp. Electrif. (2022)

  8. Yao, Y., Shu, F., Li, Z., Cheng, X., Wu, L.: Secure transmission scheme based on joint radar and communication in mobile vehicular networks. IEEE Trans. Intell. Transpo. Syst. (2023)

  9. Guo, F., Zhou, W., Lu, Q., Zhang, C.: Path extension similarity link prediction method based on matrix algebra in directed networks. Comput. Commun.. Commun. 187, 83–92 (2022)

    Article  Google Scholar 

  10. Dai, X., **ao, Z., Jiang, H., Alazab, M., Lui, J. C. S., Min, G.,... Liu, J.: Task Offloading for Cloud-Assisted Fog Computing With Dynamic Service Caching in Enterprise Management Systems. IEEE Trans. Ind. Inform. 19(1), 662–672 (2023)

  11. Dai, X., **ao, Z., Jiang, H., Alazab, M., Lui, J. C. S., Dustdar, S.,... Liu, J.: Task co-offloading for D2D-assisted mobile edge computing in industrial internet of things. IEEE Trans. Ind. Inform. 19(1), 480–490 (2023)

  12. Jiang, H., Dai, X., **ao, Z., Iyengar, A. K, Joint Task Offloading and Resource Allocation for Energy-Constrained Mobile Edge Computing. IEEE Trans. Mobile Comput. (2022)

  13. Dai, X., **ao, Z., Jiang, H., Lui, J. C. S.: UAV-assisted task offloading in vehicular edge computing networks. IEEE Trans. Mobile Comput. (2023)

  14. Li, J., Deng, Y., Sun, W., Li, W., Li, R., Li, Q.,... Liu, Z.: Resource orchestration of cloud-edge–based smart grid fault detection. ACM Trans. Sen. Netw. 18(3) (2022)

  15. Li, Z., Zhou, X., Huang, S.: Managing skill certification in online outsourcing platforms: A perspective of buyer-determined reverse auctions. Int. J. Prod. Econ. 238, 108166 (2021)

    Article  Google Scholar 

  16. Gong, J., Rezaeipanah, A.: A fuzzy delay-bandwidth guaranteed routing algorithm for video conferencing services over SDN networks. Multimed. Tools Appl. (2023)

  17. Chen, G., Chen, P., Huang, W., Zhai, J.: Continuance intention mechanism of middle school student users on online learning platform based on qualitative comparative analysis method. Math. Prob. Eng. 2022(3215337), 12 (2022)

  18. Ni, Q., Guo, J., Wu, W., Wang, H., Wu, J.: Continuous Influence-Based Community Partition for Social Networks. IEEE Trans. Netw. Sci. Eng. 9(3), 1187–1197 (2022)

    Article  MathSciNet  Google Scholar 

  19. Zhou, G., Zhang, R., Huang, S.: Generalized buffering algorithm. IEEE Access 9, 27140–27157 (2021)

    Article  Google Scholar 

  20. Yuan, H., Yang, B.: System dynamics approach for evaluating the interconnection performance of cross-border transport infrastructure. J. Manage. Eng. 38(3) (2022)

  21. Chen, P., Liu, H., **n, R., Carval, T., Zhao, J., **a, Y.,... Zhao, Z.: Effectively detecting operational anomalies in large-scale IoT data infrastructures by using a GAN-based predictive model. Comput. J. 65(11), 2909–2925 (2022)

  22. Sharma, S., Hong, Y.: A hybrid multiple access scheme via deep learning-based detection. IEEE Syst. J. 15(1), 981–984 (2020)

    Article  Google Scholar 

  23. Sharma, S., Hong, Y.: UWB receiver via deep learning in MUI and ISI scenarios. IEEE Trans. Veh. Technol. 69(3), 3496–3499 (2020)

    Article  Google Scholar 

  24. Peng, Y., Zhao, Y., & Hu, J, On The role of community structure in evolution of opinion formation: a new bounded confidence opinion dynamics. Inf. Sci. 621, 672–690 (2023)

  25. Li, D., Ge, S.S., Lee, T.H.: Fixed-Time-Synchronized Consensus Control of Multiagent Systems. IEEE Trans Control Netw. Syst. 8(1), 89–98 (2021)

    Article  MathSciNet  Google Scholar 

  26. Ma, Q., Meng, Q., & Xu, S.: Distributed optimization for uncertain high-order nonlinear multiagent systems via dynamic gain approach. IEEE Trans. Syst. Man. Cybern. Syst. 53(7), 4351–4357 (2023)

  27. Zhang, H., Mi, Y., Fu, Y., Liu, X., Zhang, Y., Wang, J.,... Tan, J.: Security defense decision method based on potential differential game for complex networks. Comput. Secur. 129, 103187 (2023)

  28. Cheng, B., Wang, M., Zhao, S., Zhai, Z., Zhu, D.,... Chen, J.: Situation-aware dynamic service coordination in an IoT environment. IEEE/ACM Trans. Network. 25(4), 2082–2095 (2017)

  29. Lu, S., Liu, M., Yin, L., Yin, Z., Liu, X., Zheng, W.,... Kong, X.: The multi-modal fusion in visual question answering: a review of attention mechanisms. PeerJ Comput. Sci. 9, e1400 (2023)

  30. Li, B., Tan, Y., Wu, A., Duan, G.: A distributionally robust optimization based method for stochastic model predictive control. IEEE Trans. Autom. ControlAutom. Control 67(11), 5762–5776 (2021)

    Article  MathSciNet  Google Scholar 

  31. Cao, K., Wang, B., Ding, H., Lv, L., Dong, R., Cheng, T.,... Gong, F.: Improving physical layer security of uplink NOMA via energy harvesting jammers. IEEE Trans. Inform. Forensics Secur. 16, 786–799 (2021)

  32. Yang, S., Li, Q., Li, W., Li, X., Liu, A.: Dual-Level Representation Enhancement on Characteristic and Context for Image-Text Retrieval. IEEE Trans. Circuits Syst. Video Technol. 32(11), 8037–8050 (2022)

    Article  Google Scholar 

  33. ** via dynamic representation of gripper-object interaction. ACM Trans. Graph. 41(4) (2022)

  34. Zhao, K., Jia, Z., Jia, F., Shao, H.: Multi-scale integrated deep self-attention network for predicting remaining useful life of aero-engine. Eng. Appl. Artif. Intell.Artif. Intell. 120, 105860 (2023)

    Article  Google Scholar 

  35. Wang, B., Zhu, D., Han, L., Gao, H., Gao, Z.,... Zhang, Y.: Adaptive fault-tolerant control of a hybrid canard rotor/wing UAV under transition flight subject to actuator faults and model uncertainties. IEEE Trans. Aerosp. Electron. Syst. (2023)

Download references

Funding

This research received no specific grant from any funding agency.

Author information

Authors and Affiliations

  1. School of Electronics and Information, Jiangsu Vocational College of Business, Nantong, 226011, China

    **aohu Gao

  2. Jiangsu Provincial Research and Development Center for Internet of Things and Visual Intelligent Processing Engineering Technology, Jiangsu Vocational College of Business, Nantong, 226011, China

    **aohu Gao

  3. Institute of Visual Informatics, Universiti Kebangsaan Malaysia, 43600, Bangi, Selangor, Malaysia

    Mei Choo Ang

  4. Department of Computer Science, College of Computer and Information Sciences, Majmaah University, 11952, Al-Majmaah, Saudi Arabia

    Sara A. Althubiti

Authors
  1. **aohu Gao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mei Choo Ang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sara A. Althubiti
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

**aohu Gao: Conceptualization, Methodology, Formal analysis, Supervision, Writing—original draft, Writing—review & editing.

MEI CHOO ANG: Writing—original draft, Writing—review & editing.

Sara A Althubiti: Investigation, Data Curation, Validation, Resources, Writing—review & editing.

Corresponding author

Correspondence to **aohu Gao.

Ethics declarations

Ethics Approval and Consent to Participate

Not applicable.

Consent for Publication

Not applicable.

Competing Interests

The authors declare that they have no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Gao, X., Ang, M.C. & Althubiti, S.A. Deep Reinforcement Learning and Markov Decision Problem for Task Offloading in Mobile Edge Computing. J Grid Computing 21, 78 (2023). https://doi.org/10.1007/s10723-023-09708-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10723-023-09708-4

Keywords

Search

Navigation