SpringerLink
Log in

Distributed multi-agent deep reinforcement learning for trajectory planning in UAVs-assisted edge offloading

  • Regular Paper
  • Published:
CCF Transactions on Pervasive Computing and Interaction Aims and scope Submit manuscript

Abstract

To deal with the diverse computing tasks generated by Internet of Things devices (IoTDs), unmanned aerial vehicles (UAVs)-assisted edge offloading technology has emerged. However, there are many challenges in edge offloading, such as the dynamic change of channel state and the limited computing resources. Therefore, it is crucial to plan the trajectories of UAVs intelligently to improve the offloading efficiency. In this paper, we propose an individual-inference-based distributed deep deterministic policy gradient (IID-DDPG) algorithm for multi-UAV trajectory planning. The proposed IID-DDPG algorithm adopts a distributed training method, which only involves the interactions of neighbor agents. Consequently, the IID-DDPG algorithm has strong scalability and small communication burden. In addition, the causal inference method is first used to measure the importance of neighbor information. Secondly, an information aggregation network is designed to assist UAV agents to infer global knowledge, so as to enhance the stability of distributed training. Finally, the effectiveness and superiority of the proposed IID-DDPG algorithm are verified by extensive simulations.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price includes VAT (Canada)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Algorithm 1
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Data availability

No data was used for the research described in the article.

References

  • Akhlaqi, M.Y., Hanapi, Z.B.M.: Task offloading paradigm in mobile edge computing-current issues, adopted approaches, and future directions. J. Netw. Comput. Appl. 212, 103568 (2023)

    Article  Google Scholar 

  • Cai, T., Yang, Z., Chen, Y., Chen, W., Zheng, Z., Yu, Y., Dai, H.-N.: Cooperative data sensing and computation offloading in uav-assisted crowdsensing with multi-agent deep reinforcement learning. IEEE Trans. Netw. Sci. Eng. 9(5), 3197–3211 (2021)

    Article  Google Scholar 

  • Cheng, Z., Gao, Z., Liwang, M., Huang, L., Du, X., Guizani, M.: Intelligent task offloading and energy allocation in the uav-aided mobile edge-cloud continuum. IEEE Netw. 35(5), 42–49 (2021)

    Article  Google Scholar 

  • Gao, A., Wang, Q., Liang, W., Ding, Z.: Game combined multi-agent reinforcement learning approach for uav assisted offloading. IEEE Trans. Veh. Technol. 70(12), 12888–12901 (2021)

    Article  Google Scholar 

  • Han, H., Zhan, C., Lv, J., Xu, C.: Energy minimization for cellular connected aerial edge computing system with binary offloading. IEEE Internet Things J. (2023). https://doi.org/10.1109/JIOT.2023.3323289

    Article  Google Scholar 

  • Hoang, L.T., Nguyen, C.T., Pham, A.T.: Deep reinforcement learning-based online resource management for uav-assisted edge computing with dual connectivity. IEEE/ACM Trans. Networking. 31(6), 2761–2776 (2023)

    Article  Google Scholar 

  • Jaques, N., Lazaridou, A., Hughes, E., Gulcehre, C., Ortega, P., Strouse, D., Leibo, J.Z., De Freitas, N.: Social influence as intrinsic motivation for multi-agent deep reinforcement learning. In: International Conference on Machine Learning, 3040–3049 (2019). PMLR

  • Lai, C.-C., Tsai, A.-H., Wang, L.-C., et al.: Adaptive and fair deployment approach to balance offload traffic in multi-uav cellular networks. IEEE Trans. Vehicular Technol. 72(3), 3724–3738 (2022)

    Article  Google Scholar 

  • Li, X., Teng, M., Wu, J., Qin, X.: Dual-label aware service replacement for interaction quality improvement in heterogeneous mec system. CCF Trans. Pervasive Comput. Interact. 3, 129–146 (2021)

    Article  Google Scholar 

  • Li, B., Yang, R., Liu, L., Wang, J., Zhang, N., Dong, M.: Robust computation offloading and trajectory optimization for multi-uav-assisted mec: a multi-agent drl approach. IEEE Internet Things J. 11(3), 4775–4786 (2024)

    Article  Google Scholar 

  • Li, S., Zhang, S., Wang, Z., Zhou, Z., Wang, X., Mumtaz, S., Guizani, M., Frascolla, V.: Asynchronous fdrl-based low-latency computation offloading for integrated terrestrial and non-terrestrial power iot. IEEE Netw. (2023b). https://doi.org/10.1109/MNET.2023.3320894

    Article  Google Scholar 

  • Lin, N., Tang, H., Zhao, L., Wan, S., Hawbani, A., Guizani, M.: A pddqnlp algorithm for energy efficient computation offloading in uav-assisted mec. IEEE Trans. Wirel. Commun. 22(12), 8876–8890 (2023). https://doi.org/10.1109/TWC.2023.3266497

    Article  Google Scholar 

  • Miao, Y., Hwang, K., Wu, D., Hao, Y., Chen, M.: Drone swarm path planning for mobile edge computing in industrial internet of things. IEEE Trans. Ind. Inf. 19(5), 6836–6848 (2022)

    Article  Google Scholar 

  • Ning, Z., Hu, H., Wang, X., Guo, L., Guo, S., Wang, G., Gao, X.: Mobile edge computing and machine learning in the internet of unmanned aerial vehicles: a survey. ACM Comput. Surveys. 56(1), 1–31 (2023)

    Article  Google Scholar 

  • Peng, H., Shen, X.: Multi-agent reinforcement learning based resource management in mec-and uav-assisted vehicular networks. IEEE J. Select. Areas Commun. 39(1), 131–141 (2020)

    Article  Google Scholar 

  • Shah, Z., Javed, U., Naeem, M., Zeadally, S., Ejaz, W.: Mobile edge computing (mec)-enabled uav placement and computation efficiency maximization in disaster scenario. IEEE Trans. Veh. Technol. 72(10), 13406–13416 (2023)

    Article  Google Scholar 

  • Shen, M., Gu, A., Kang, J., Tang, X., Lin, X., Zhu, L., Niyato, D.: Blockchains for artificial intelligence of things: a comprehensive survey. IEEE Internet Things J. 10(16), 14483–14506 (2023)

    Article  Google Scholar 

  • Shi, H., Tian, Y., Li, H., Huang, J., Shi, L., Zhou, Y.: Task offloading and trajectory scheduling for uav-enabled mec networks: an madrl algorithm with prioritized experience replay. Ad Hoc Netw. 154, 103371 (2024)

    Article  Google Scholar 

  • Song, Z., Qin, X., Hao, Y., Hou, T., Wang, J., Sun, X.: A comprehensive survey on aerial mobile edge computing: challenges, state-of-the-art, and future directions. Comput. Commun. 191, 233–256 (2022)

    Article  Google Scholar 

  • Tang, Q., Chang, L., Yang, K., Wang, K., Wang, J., Sharma, P.K.: Task number maximization offloading strategy seamlessly adapted to uav scenario. Comput. Commun. 151, 19–30 (2020)

    Article  Google Scholar 

  • Wang, L., Wang, K., Pan, C., Xu, W., Aslam, N., Hanzo, L.: Multi-agent deep reinforcement learning-based trajectory planning for multi-uav assisted mobile edge computing. IEEE Trans. Cognit. Commun. Netw 7(1), 73–84 (2020)

    Article  Google Scholar 

  • Wang, H., Yu, Y., Jiang, Y.: Fully decentralized multiagent communication via causal inference. IEEE Trans. Neural Netw. Learn. Syst. 34(12), 10193–10202 (2022a)

    Article  Google Scholar 

  • Wang, C., Deng, D., Xu, L., Wang, W.: Resource scheduling based on deep reinforcement learning in uav assisted emergency communication networks. IEEE Trans. Commun. 70(6), 3834–3848 (2022b)

    Article  Google Scholar 

  • Wang, L., Zhao, X., Lu, Z., Wang, L., Zhang, S.: Enhancing privacy preservation and trustworthiness for decentralized federated learning. Inf. Sci. 628, 449–468 (2023). https://doi.org/10.1016/j.ins.2023.01.130

    Article  Google Scholar 

  • Wei, D., Ma, J., Luo, L., Wang, Y., He, L., Li, X.: Computation offloading over multi-uav mec network: a distributed deep reinforcement learning approach. Comput. Netw. 199, 108439 (2021)

    Article  Google Scholar 

  • Wu, S., Xu, W., Wang, F., Li, G., Pan, M.: Distributed federated deep reinforcement learning based trajectory optimization for air-ground cooperative emergency networks. IEEE Trans. Veh. Technol. 71(8), 9107–9112 (2022)

    Article  Google Scholar 

  • Xu, J., Li, D., Gu, W., Chen, Y.: Uav-assisted task offloading for iot in smart buildings and environment via deep reinforcement learning. Build. Environ. 222, 109218 (2022a)

    Article  Google Scholar 

  • Xu, Y., Zhang, T., Zou, Y., Liu, Y.: Reconfigurable intelligence surface aided uav-mec systems with noma. IEEE Commun. Lett. 26(9), 2121–2125 (2022b)

    Article  Google Scholar 

  • Ye, Y., Wei, W., Geng, D., He, X.: Dynamic coordination in uav swarm assisted mec via decentralized deep reinforcement learning. In: 2020 International Conference on Wireless Communications and Signal Processing (WCSP), 1064–1069 (2020). IEEE

  • Zhang, X., Debroy, S.: Resource management in mobile edge computing: a comprehensive survey. ACM Comput. Surveys. 55(13s), 1–37 (2023)

    Article  Google Scholar 

  • Zhang, T., Chen, C., Xu, Y., Loo, J., Xu, W.: Joint task scheduling and multi-uav deployment for aerial computing in emergency communication networks. Sci. China Inf. Sci. 66(9), 192303 (2023)

    Article  MathSciNet  Google Scholar 

  • Zhao, N., Cheng, Y., Pei, Y., Liang, Y.-C., Niyato, D.: Deep reinforcement learning for trajectory design and power allocation in uav networks. In: ICC 2020-2020 IEEE International Conference on Communications (ICC), 1–6 (2020). IEEE

  • Zhao, N., Ye, Z., Pei, Y., Liang, Y.-C., Niyato, D.: Multi-agent deep reinforcement learning for task offloading in uav-assisted mobile edge computing. IEEE Trans. Wirel. Commun. 21(9), 6949–6960 (2022)

    Article  Google Scholar 

  • Zhou, H., Wu, T., Chen, X., He, S., Guo, D., Wu, J.: Reverse auction-based computation offloading and resource allocation in mobile cloud-edge computing. IEEE Trans. Mobile Comput. 22(10), 6144–6159 (2023a). https://doi.org/10.1109/TMC.2022.3189050

    Article  Google Scholar 

  • Zhou, H., Li, M., Wang, N., Min, G., Wu, J.: Accelerating deep learning inference via model parallelism and partial computation offloading. IEEE Trans. Distrib. Syst. 34(2), 475–488 (2023b)

    Article  Google Scholar 

  • Zhou, H., Jiang, K., He, S., Min, G., Wu, J.: Distributed deep multi-agent reinforcement learning for cooperative edge caching in internet-of-vehicles. IEEE Trans. Wirel. Commun. 22(12), 9595–9609 (2023c). https://doi.org/10.1109/TWC.2023.3272348

    Article  Google Scholar 

Download references

Funding

This work was supported in part by the National Natural Science Foundation of China (No. 62373102), the Jiangsu Natural Science Foundation (No. BK20221455), and the Anhui Provincial Key Research and Development Project (No. 2022i01020013).

Author information

Authors and Affiliations

  1. School of Automation, Southeast University, **njiekou Street, Nan**g, 210096, Jiangsu, China

    Chenchen Fan & Qingling Wang

  2. College of Intelligence Science and Technology, National University of Defense Technology, Kaifu District, Changsha, 410073, Hunan, China

    **angke Wang

Authors
  1. Chenchen Fan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Qingling Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. **angke Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Qingling Wang.

Ethics declarations

Conflict of interest

The authors have no Conflict of interest.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Fan, C., Wang, Q. & Wang, X. Distributed multi-agent deep reinforcement learning for trajectory planning in UAVs-assisted edge offloading. CCF Trans. Pervasive Comp. Interact. (2024). https://doi.org/10.1007/s42486-024-00159-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s42486-024-00159-8

Keywords

Search

Navigation