Abstract
Battery energy storage systems are widely used in microgrids integrated with volatile energy resources for their ability in peak load shifting. Security constrained economic dispatch over the system’s lifecycle is a constrained multi-period stochastic optimization problem, which is intractable. We propose an improved actor-critic-based reinforcement learning combined with a protection layer security control method for this issue, where the distributional critic net is applied to estimate the expected total reward value of a period more accurately, and the policy net with a mask action layer is used to make secure and real-time decision. Additionally, we propose a protection layer to assist the policy net as a secondary control to prevent the unsafe state of microgrids due to the trial-and-error learning of reinforcement learning. Numerical test results show the proposed algorithm can perform better than the conventional economic dispatch and other reinforcement learning algorithms while guaranteeing safe operation.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Birge, J.R., Louveaux, F.: Introduction to Stochastic Programming. Springer Science & Business Media (2011). 10.1007/978-1-4614-0237-4
Carpinelli, G., Fazio, D., Rita, A., Khormali, S., Mottola, F.: Optimal sizing of battery storage systems for industrial applications when uncertainties exist. Energies 7, 130–149 (2014)
Christiano, P.F., Leike, J., Brown, T., Martic, M., Legg, S., Amodei, D.: Deep reinforcement learning from human preferences. Adv. Neural Inform. Process. Syst. 30, 4299–4307 (2017)
Divya, K., Østergaard, J.: Battery energy storage technology for power systems–an overview. Electr. Power Syst. Res. 79(4), 511–520 (2009)
Eseye, A.T., Lehtonen, M., Tukia, T., Uimonen, S., Millar, R.J.: Optimal energy trading for renewable energy integrated building microgrids containing electric vehicles and energy storage batteries. IEEE Access 7, 106092–106101 (2019)
Hill, C.A., Such, M.C., Chen, D., Gonzalez, J., Grady, W.M.: Battery energy storage for enabling integration of distributed solar power generation. IEEE Trans. Smart Grid 3(2), 850–857 (2012)
Huang, J., Wu, F., Precup, D., Cai, Y.: Learning safe policies with expert guidance. In: Advances in Neural Information Processing Systems, pp. 9105–9114 (2018)
Keirstead, J., Jennings, M., Sivakumar, A.: A review of urban energy system models: approaches, challenges and opportunities. Renew. Sustain. Energy Rev. 16(6), 3847–3866 (2012)
Lambert, T., Gilman, P., Lilienthal, P.: Micropower system modeling with homer. Integr. Altern. Sour. Energy 1(1), 379–385 (2006)
Liu, C., Wang, X., Wu, X., Guo, J.: Economic scheduling model of microgrid considering the lifetime of batteries. IET Gener. Trans. Distrib. 11(3), 759–767 (2017)
Luo, X., Wang, J., Dooner, M., Clarke, J.: Overview of current development in electrical energy storage technologies and the application potential in power system operation. Appl. Energ. 137, 511–536 (2015)
Ma, T., Yang, H., Lu, L.: A feasibility study of a stand-alone hybrid solar-wind-battery system for a remote island. Appl. Energ. 121, 149–158 (2014)
Manwell, J.F., McGowan, J.G.: Lead acid battery storage model for hybrid energy systems. Solar Energ. 50(5), 399–405 (1993)
Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., Riedmiller, M.: Playing atari with deep reinforcement learning. ar**v preprint ar**v:1312.5602 (2013)
Nair, N.K.C., Garimella, N.: Battery energy storage systems: assessment for small-scale renewable energy integration. Energ. Build. 42(11), 2124–2130 (2010)
Nechyba, M.: Maximum-likelihood estimation for mixture models: the em algorithm. EEL6935 Fall (2001)
Schulman, J., Moritz, P., Levine, S., Jordan, M., Abbeel, P.: High-dimensional continuous control using generalized advantage estimation. ar**v preprint ar**v:1506.02438 (2015)
Shang, Y., et al.: Stochastic dispatch of energy storage in microgrids: an augmented reinforcement learning approach. Appl. Energ. 261, 114423 (2020)
Silver, D., et al.: Mastering the game of go without human knowledge. Nature 550(7676), 354–359 (2017)
Vinyals, O., et al.: Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature 575(7782), 350–354 (2019)
Wu, J., Yan, J., Jia, H., Hatziargyriou, N., Djilali, N., Sun, H.: Integrated energy systems. Appl. Energ. 167, 155–157 (2016)
Zha, Z.Y., Wang, B., Tang, X.S.: Evaluate, explain, and explore the state more exactly: an improved actor-critic algorithm for complex environment. Neural Comput. Appl. 4, 1–12 (2021)
Zhang, C., Wu, J., Zhou, Y., Cheng, M., Long, C.: Peer-to-peer energy trading in a microgrid. Appl. Energ. 220, 1–12 (2018)
Zia, M.F., Elbouchikhi, E., Benbouzid, M.: Microgrids energy management systems: a critical review on methods, solutions, and prospects. Appl. Energ. 222, 1033–1055 (2018)
Acknowledgement
This research is surpported by state grid corporation of China headquarters science and technology project (grant number: 5100-202099522A-0-0-00)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Zha, Z., Wang, B., Fan, H., Liu, L. (2021). An Improved Reinforcement Learning for Security-Constrained Economic Dispatch of Battery Energy Storage in Microgrids. In: Zhang, H., Yang, Z., Zhang, Z., Wu, Z., Hao, T. (eds) Neural Computing for Advanced Applications. NCAA 2021. Communications in Computer and Information Science, vol 1449. Springer, Singapore. https://doi.org/10.1007/978-981-16-5188-5_22
Download citation
DOI: https://doi.org/10.1007/978-981-16-5188-5_22
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-16-5187-8
Online ISBN: 978-981-16-5188-5
eBook Packages: Computer ScienceComputer Science (R0)