SpringerLink
Log in

Digital twin-enabled adaptive scheduling strategy based on deep reinforcement learning

  • Article
  • Published:
Science China Technological Sciences Aims and scope Submit manuscript

Abstract

The modern complicated manufacturing industry and smart manufacturing tendency have imposed new requirements on the scheduling method, such as self-regulation and self-learning capabilities. While traditional scheduling methods cannot meet these needs due to their rigidity. Self-learning is an inherent ability of reinforcement learning (RL) algorithm inhered from its continuous learning and trial-and-error characteristics. Self-regulation of scheduling could be enabled by the emerging digital twin (DT) technology because of its virtual-real map** and mutual control characteristics. This paper proposed a DT-enabled adaptive scheduling based on the improved proximal policy optimization RL algorithm, which was called explicit exploration and asynchronous update proximal policy optimization algorithm (E2APPO). Firstly, the DT-enabled scheduling system framework was designed to enhance the interaction between the virtual and the physical job shops, strengthening the self-regulation of the scheduling model. Secondly, an innovative action selection strategy and an asynchronous update mechanism were proposed to improve the optimization algorithm to strengthen the self-learning ability of the scheduling model. Lastly, the proposed scheduling model was extensively tested in comparison with heuristic and meta-heuristic algorithms, such as well-known scheduling rules and genetic algorithms, as well as other existing scheduling methods based on reinforcement learning. The comparisons have proved both the effectiveness and advancement of the proposed DT-enabled adaptive scheduling strategy.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Błażewicz J, Domschke W, Pesch E. The job shop scheduling problem: Conventional and new solution techniques. Eur J Operational Res, 1996, 93: 1–33

    Article  MATH  Google Scholar 

  2. Fisher M L. Optimal solution of scheduling problems using Lagrange multipliers: Part I. Oper Res, 1973, 21: 1114–1127

    Article  MATH  Google Scholar 

  3. Lomnicki Z A. A “branch-and-bound” algorithm for the exact solution of the three-machine scheduling problem. J Operational Res Soc, 1965, 16: 89–100

    Article  Google Scholar 

  4. Zhou H, Cheung W, Leung L C. Minimizing weighted tardiness of job-shop scheduling using a hybrid genetic algorithm. Eur J Operational Res, 2009, 194: 637–649

    Article  MATH  Google Scholar 

  5. Salido M A, Escamilla J, Giret A, et al. A genetic algorithm for energy-efficiency in job-shop scheduling. Int J Adv Manuf Technol, 2016, 85: 1303–1314

    Article  Google Scholar 

  6. Lin Q, Li J, Du Z, et al. A novel multi-objective particle swarm optimization with multiple search strategies. Eur J Operational Res, 2015, 247: 732–744

    Article  MathSciNet  MATH  Google Scholar 

  7. Liu X, Ni Z, Qiu X. Application of ant colony optimization algorithm in integrated process planning and scheduling. Int J Adv Manuf Technol, 2016, 84: 393–404

    Article  Google Scholar 

  8. Karaboga D. Artificial bee colony algorithm. Scholarpedia, 2010, 5: 6915

    Article  Google Scholar 

  9. Pan Q K. An effective co-evolutionary artificial bee colony algorithm for steelmaking-continuous casting scheduling. Eur J Operational Res, 2016, 250: 702–714

    Article  MathSciNet  MATH  Google Scholar 

  10. Qu M, Zuo Y, **ang F, et al. An improved electromagnetism-like mechanism algorithm for energy-aware many-objective flexible job shop scheduling. Int J Adv Manuf Technol, 2022, 119: 4265–4275

    Article  Google Scholar 

  11. Li X Y, **e J, Ma Q J, et al. Improved gray wolf optimizer for distributed flexible job shop scheduling problem. Sci China Tech Sci, 2022, 65: 2105–2115

    Article  Google Scholar 

  12. Wang X, Wang R, Shu G Q, et al. Energy management strategy for hybrid electric vehicle integrated with waste heat recovery system based on deep reinforcement learning. Sci China Tech Sci, 2022, 65: 713–725

    Article  Google Scholar 

  13. Emary E, Zawbaa H M, Grosan C. Experienced gray wolf optimization through reinforcement learning and neural networks. IEEE Trans Neural Netw Learn Syst, 2018, 29: 681–694

    Article  MathSciNet  Google Scholar 

  14. Shahrabi J, Adibi M A, Mahootchi M. A reinforcement learning approach to parameter estimation in dynamic job shop scheduling. Comput Industrial Eng, 2017, 110: 75–82

    Article  Google Scholar 

  15. Zhao H, Zhang C. A decomposition-based many-objective artificial bee colony algorithm with reinforcement learning. Appl Soft Computing, 2020, 86: 105879

    Article  Google Scholar 

  16. Stricker N, Kuhnle A, Sturm R, et al. Reinforcement learning for adaptive order dispatching in the semiconductor industry. CIRP Ann, 2018, 67: 511–514

    Article  Google Scholar 

  17. Lin C C, Deng D J, Chih Y L, et al. Smart manufacturing scheduling with edge computing using multiclass deep Q network. IEEE Trans Ind Inf, 2019, 15: 4276–4284

    Article  Google Scholar 

  18. Shi D, Fan W, **ao Y, et al. Intelligent scheduling of discrete automated production line via deep reinforcement learning. Int J Prod Res, 2020, 58: 1–19

    Article  Google Scholar 

  19. Palombarini J A, Martínez E C. Automatic generation of rescheduling knowledge in socio-technical manufacturing systems using deep reinforcement learning. In: Proceedings of 2018 IEEE Biennial Congress of Argentina. San Miguel de Tucuman, 2018. 1–8

  20. **a K, Sacco C, Kirkpatrick M, et al. A digital twin to train deep reinforcement learning agent for smart manufacturing plants: Environment, interfaces and intelligence. J Manuf Syst, 2021, 58: 210–230

    Article  Google Scholar 

  21. Park I B, Huh J, Kim J, et al. A reinforcement learning approach to robust scheduling of semiconductor manufacturing facilities. IEEE Trans Autom Sci Eng, 2020, 17: 1420–1431

    Google Scholar 

  22. Zhou L, Zhang L, Horn B K P. Deep reinforcement learning-based dynamic scheduling in smart manufacturing. Procedia CIRP, 2020, 93: 383–388

    Article  Google Scholar 

  23. Tao F, Zhang M, Nee A Y C. Digital Twin Driven Smart Manufacturing. Elsevier Academic Press, 2019

  24. Grieves M. Digital twin: Manufacturing excellence through virtual factory replication. White Paper, 2014. 1–7

  25. Zhang J, Deng T, Jiang H, et al. Bi-level dynamic scheduling architecture based on service unit digital twin agents. J Manuf Syst, 2021, 60: 59–79

    Article  Google Scholar 

  26. Tao F, Zhang M, Liu Y, et al. Digital twin driven prognostics and health management for complex equipment. CIRP Ann, 2018, 67: 169–172

    Article  Google Scholar 

  27. Schluse M, Priggemeyer M, Atorf L, et al. Experimentable digital twins—Streamlining simulation-based systems engineering for Industry 4.0. IEEE Trans Ind Inf, 2018, 14: 1722–1731

    Article  Google Scholar 

  28. Li Y, Tao Z, Wang L, et al. Digital twin-based job shop anomaly detection and dynamic scheduling. Robotics Comput-Integrated Manuf, 2023, 79: 102443

    Article  Google Scholar 

  29. Yan Q, Wang H, Wu F. Digital twin-enabled dynamic scheduling with preventive maintenance using a double-layer Q-learning algorithm. Comput Operations Res, 2022, 144: 105823

    Article  MATH  Google Scholar 

  30. Zhang M, Tao F, Nee A Y C. Digital twin enhanced dynamic job-shop scheduling. J Manuf Syst, 2021, 58: 146–156

    Article  Google Scholar 

  31. Negri E, Ardakani H D, Cattaneo L, et al. A digital twin-based scheduling framework including equipment health index and genetic algorithms. In: Proceedings of 13th International-Federation-of-Automatic-Control (IFAC) Workshop on Intelligent Manufacturing Systems (IMS). Oshawa, 2019. 52: 43–48

  32. Pinedo M. Scheduling: Theory, Algorithms, and Systems. New York: NYU Stern School of Business, 2016. 207–214

    Book  MATH  Google Scholar 

  33. Wang W Q, Ye C M, Tan X J. Job shop dynamic scheduling based on Q-learning algorithm. Comput Syst Appl, 2020, 29: 218–226

    Google Scholar 

  34. Adams J, Balas E, Zawack D. The shifting bottleneck procedure for job shop scheduling. Manage Sci, 1988, 34: 391–401

    Article  MathSciNet  MATH  Google Scholar 

  35. Schulman J, Wolski F, Dhariwal P, et al. Proximal policy optimization algorithms. ar**v: 1707.06347, 2017

  36. Girshick R. Fast R-CNN. In: Proceedings of IEEE International Conference on Computer Vision. Santiago, 2015. 1440–1448

  37. Zoph B, Le Q V. Searching for activation functions. In: Proceedings of International Conference of Learning Representation. Vancouver, Canada, 2018. 1–13

  38. Niu G G, Sun S D, Lafon P, et al. A decomposition approach to jobshop scheduling problem with discretely controllable processing times. Sci China Tech Sci, 2011, 54: 1240–1248

    Article  MATH  Google Scholar 

  39. Wu Z, Pan S, Chen F, et al. A comprehensive survey on graph neural networks. IEEE Trans Neural Netw Learn Syst, 2020, 32: 4–24

    Article  MathSciNet  Google Scholar 

  40. Fisher H. Probabilistic learning combinations of local job-shop scheduling rules. Ind Sched, 1963, 225–251

  41. Lawrence S. Resouce constrained project scheduling: An experimental investigation of heuristic scheduling techniques (Supplement). Dissertation for the Doctoral Degree. Pennsylvania: Carnegie-Mellon University. 1984

    Google Scholar 

  42. Applegate D, Cook W. A computational study of the job-shop scheduling problem. ORSA J Computing, 1991, 3: 149–156

    Article  MATH  Google Scholar 

  43. Demirkol E, Mehta S, Uzsoy R. Benchmarks for shop scheduling problems. Eur J Operational Res, 1998, 109: 137–141

    Article  MATH  Google Scholar 

  44. Taillard E. Benchmarks for basic scheduling problems. Eur J Operational Res, 1993, 64: 278–285

    Article  MATH  Google Scholar 

  45. Zhang C, Song W, Cao Z, et al. Learning to dispatch for job shop scheduling via deep reinforcement learning. Adv Neural Inf Process Syst, 2020, 33: 1621–1632

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Mechanical Engineering, Guizhou University, Guiyang, 550025, China

    XueMei Gan

  2. School of Automation Science and Electrical Engineering, Beihang University, Bei**g, 100191, China

    Ying Zuo & Fei Tao

  3. State Key Laboratory of Public Big Data, Guizhou University, Guiyang, 550025, China

    AnSi Zhang & ShaoBo Li

Authors
  1. XueMei Gan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ying Zuo
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. AnSi Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. ShaoBo Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Fei Tao
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ying Zuo.

Additional information

This work was supported by the National Key R&D Program of China (Grant No. 2020YFB1713300), the Joint Open Fund of Wuhan Textile University (Grant No. KT202201005), and the Foundation of Key Laboratory of Advanced Manufacturing Technology, Ministry of Education, Guizhou University (Grant No. GZUAMT2021KF11). The authors thank for the computing support of the State Key Laboratory of Public Big Data, Guizhou University.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Gan, X., Zuo, Y., Zhang, A. et al. Digital twin-enabled adaptive scheduling strategy based on deep reinforcement learning. Sci. China Technol. Sci. 66, 1937–1951 (2023). https://doi.org/10.1007/s11431-022-2413-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11431-022-2413-5

Search

Navigation