Abstract
In this chapter, a novel data-driven method, which is called the deep deterministic policy gradient (DDPG), is applied for optimally controlling the multi-zone residential heating, ventilation, and air conditioning (HVAC) system. The DDPG method is a type of model-free deep reinforcement learning (deep RL) method that can generate HVAC control strategies without referring to any complex modeling formulation. The applied deep RL–based method can learn the optimal control strategy through continuous interaction with the simulated building environment. Simulation results of DDPG on real-world use cases and comparisons with the benchmark cases demonstrate the effectiveness and the generalization ability of DDPG in saving energy cost while maintaining occupant comfort, which proves its feasibility in solving real-world high-dimensional control problems with hidden information or vast solution spaces.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
L. Pérez-Lombard, J. Ortiz, C. Pout, A review on buildings energy consumption information. Energy Build. 40, 394–398 (2008)
A. Costa, M.M. Keane, J.I. Torrens, E. Corry, Building operation and energy performance: Monitoring, analysis and optimisation toolkit. Appl. Energy 101, 310–316 (2013)
X. Kou, F. Li, J. Dong, et al., A scalable and distributed algorithm for managing residential demand response programs using Alternating Direction Method of Multipliers (ADMM). IEEE Trans. Smart Grid 2020; in-press
K. Ma, G. Hu, C.J. Spanos, Energy management considering load operations and forecast errors with application to HVAC systems. IEEE Trans. Smart Grid 9, 605–614 (2016)
O. Erdinc, A. Taşcıkaraoğlu, N.G. Paterakis, Y. Eren, J.P. Catalão, End-user comfort oriented day-ahead planning for responsive residential HVAC demand aggregation considering weather forecasts. IEEE Trans. Smart Grid 8, 362–372 (2016)
X. Wu, J. He, Y. Xu, J. Lu, N. Lu, X. Wang, Hierarchical control of residential HVAC units for primary frequency regulation. IEEE Trans. Smart Grid 9, 3844–3856 (2017)
Y. Lin, P. Barooah, J.L. Mathieu, Ancillary services through demand scheduling and control of commercial buildings. IEEE Trans. Power Syst. 32, 186–197 (2016)
L. Yu, T. Jiang, Y. Zou, Online energy management for a sustainable smart home with an HVAC load and random occupancy. IEEE Trans. Smart Grid 10, 1646–1659 (2017)
H. Hao, C.D. Corbin, K. Kalsi, R.G. Pratt, Transactive control of commercial buildings for demand response. IEEE Trans. Power Syst. 32, 774–783 (2016)
D. Silver, A. Huang, C.J. Maddison, A. Guez, L. Sifre, G. Van Den Driessche, et al., Mastering the game of go with deep neural networks and tree search. Nature 529, 484–489 (2016)
F. Li, Y. Du, From AlphaGo to power system AI: What engineers can learn from solving the most complex board game. IEEE Power Energy Mag. 16, 76–84 (2018)
Y. Du, F. Li, J. Li, T. Zheng, Achieving 100x acceleration for N-1 contingency screening with uncertain scenarios using deep convolutional neural network. IEEE Trans. Power Syst. 34, 3303–3305 (2019)
Y. Du, F. Li, Multi-microgrid energy management based on deep neural network and model-free reinforcement learning. IEEE Trans. Smart Grid 11, 1066–1076 (2020)
T. Huang, Q. Guo, H. Sun, A distributed computing platform supporting power system security knowledge discovery based on online simulation. IEEE Trans. Smart Grid 8, 1513–1524 (2017)
Y. Wu, H. Tan, J. Peng, H. Zhang, H. He, Deep reinforcement learning of energy management with continuous control strategy and traffic information for a series-parallel plug-in hybrid electric bus. Appl. Energy 247, 454–466 (2019)
X. Han, H. He, J. Wu, J. Peng, Y. Li, Energy management based on reinforcement learning with double deep Q-learning for a hybrid electric tracked vehicle. Appl. Energy 254, 113708 (2019)
H. Hua, Y. Qin, C. Hao, J. Cao, Optimal energy management strategies for energy internet via deep reinforcement learning approach. Appl Energy 239, 598–609 (2019)
R. Rocchetta, L. Bellani, M. Compare, E. Zio, E. Patelli, A reinforcement learning framework for optimal operation and maintenance of power grids. Appl. Energy 241, 291–301 (2019)
P. Kou, D. Liang, C. Wang, Z. Wu, L. Gao, Safe deep reinforcement learning-based constrained optimal control scheme for active distribution networks. Appl. Energy 264, 114772 (2020)
T. Wei, S. Ren, Q. Zhu, Deep reinforcement learning for joint datacenter and HVAC load control in distributed mixed-use buildings. IEEE Trans. Sustain. Comput (2019); early access
B.J. Claessens, P. Vrancx, F. Ruelens, Convolutional neural networks for automatic state-time feature extraction in reinforcement learning applied to residential load control. IEEE Trans. Smart Grid 9, 3259–3269 (2016)
E. Mocanu, D.C. Mocanu, P.H. Nguyen, A. Liotta, M.E. Webber, M. Gibescu, J.G. Slootweg, On-line building energy optimization using deep reinforcement learning. IEEE Trans. Smart Grid 10, 3698–3708 (2018)
Y. Wang, K. Velswamy, B. Huang, A long-short term memory recurrent neural network based reinforcement learning controller for office heating ventilation and air conditioning systems. Processes 5, 46–63 (2017)
Z. Zhang, A. Chong, Y. Pan, C. Zhang, K.P. Lam, Whole building energy model for HVAC optimal control: A practical framework based on deep reinforcement learning. Energy Build. 199, 472–490 (2019)
K.U. Ahn, C.S. Park, Application of deep Q-networks for model-free optimal control balancing between different HVAC systems. Sci. Technol. Built. Environ. 26, 61–74 (2019)
G. Gao, J. Li, Y. Wen, DeepComfort: Energy-efficient thermal comfort control in buildings via reinforcement learning. IEEE Internet Things J. (2020), early access
L. Yu, Y. Sun, Z. Xu, C. Shen, D. Yue, T. Jiang, et al., Multi-agent deep reinforcement learning for HVAC control in commercial buildings. IEEE Trans. Smart Grid early access (2020)
Z. Zou, X. Yu, S. Ergan, Towards optimal control of air handling units using deep reinforcement learning and recurrent neural network. Build. Environ. 168, 1–15 (2020)
N. Lu, An evaluation of the HVAC load potential for providing load balancing service. IEEE Trans. Smart Grid 3, 1263–1270 (2012)
B. Cui, J. Joe, J. Munk, J. Sun, T. Kuruganti, Load Flexibility Analysis of Residential HVAC and Water Heating and Commercial Refrigeration (Oak Ridge National Lab, Oak Ridge, TN (United States), 2019 Sep 1)
V. Mnih, K. Kavukcuoglu, D. Silver, A.A. Rusu, J. Veness, M.G. Bellemare, A. Graves, M. Riedmiller, A.K. Fidjeland, G. Ostrovski, S. Petersen, Human-level control through deep reinforcement learning. Nature 518, 529–533 (2015)
T.P. Lillicrap, J.J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, D. Wierstra, Continuous control with deep reinforcement learning. ar**v preprint ar**v:1509.02971
B. Cui, J. Munk, R. Jackson, D. Fugate, M. Starke, Building thermal model development of typical house in US for virtual storage control of aggregated building loads based on limited available information, in 30th International Conference on Efficiency, Cost, Optimisation, Simulation and Environmental Impact of Energy Systems. San Diego, California, US (2017)
Clean Power Research. [online]: https://www.cleanpower.com/
TensorFlow. [online]: https://www.tensorflow.org/
PJM market. [online]: https://www.pjm.com/
Author information
Authors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this chapter
Cite this chapter
Li, F., Du, Y. (2024). Intelligent Multi-zone Residential HVAC Control Strategy Based on Deep Reinforcement Learning. In: Deep Learning for Power System Applications. Power Electronics and Power Systems. Springer, Cham. https://doi.org/10.1007/978-3-031-45357-1_4
Download citation
DOI: https://doi.org/10.1007/978-3-031-45357-1_4
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-45356-4
Online ISBN: 978-3-031-45357-1
eBook Packages: EnergyEnergy (R0)