Multi-agent reinforcement learning for fast-timescale demand response of residential loads

Abstract

To integrate high amounts of renewable energy resources, electrical power grids must be able to cope with high amplitude, fast timescale variations in power generation. Frequency regulation through demand response has the potential to coordinate temporally flexible loads, such as air conditioners, to counteract these variations. Existing approaches for discrete control with dynamic constraints struggle to provide satisfactory performance for fast timescale action selection with hundreds of agents. We propose a decentralized agent trained with multi-agent proximal policy optimization with localized communication. We explore two communication frameworks: hand-engineered, or learned through targeted multi-agent communication. The resulting policies perform well and robustly for frequency regulation, and scale seamlessly to arbitrary numbers of houses for constant processing times.

Appendices

Appendix A: Notation

Table 7 contains the different notations we use in this paper.

Table 7 Notation table

Appendix B: Carbon emissions of the research project

As a significant amount of electricity has been used to train and run the models for this work, we publish its estimated carbon footprint.

Experiments were conducted using a private infrastructure, which has a carbon efficiency of 0.049 kg CO$_2$eq/kWh. A cumulative of 10895 days, or 261480 h, of computation was mainly performed on CPU of type Intel Xeon Processor E5-2683 v4 (TDP of 120W). We assume on average a power usage of half the TDP for CPUs.

The total emissions are estimated to be 628 kg CO$_2$eq of which 0% were directly offset. This is equivalent to 2550 km driven by an average car, or 314 kg of burned coal.

These estimations were conducted using the MachineLearning Impact calculator (Lacoste et al., 2019).

Appendix C: Environment details

1.1 C.1: Detailed house thermal model

The air temperature in each house evolves separately, based on its thermal characteristics $\theta _h$, its current state, the outdoor conditions such as outdoor temperature and solar gain, and the status of the air conditioner in the house. The second-order model is based on Gridlab-D’s Residential module user’s guide (Betelle Memorial Institute, 2022).

Using Gridlab-D’s module, we model an 8$\times$12.5 m, one level rectangular house, with a ceiling height of 2.5 m, four 1.8m$^2$, 2-layer, aluminum windows, and two 2 m$^2$ wooden doors, leading to the values presented in Table 8.

Table 8 Default house thermal parameters $\theta _h$

To model the evolution of the house’s air temperature $T_{h,t}$ and its mass temperature $T_{m,t}$, we assume that this temperature is homogeneous and do not consider the heat propagation in the house. We define the following variables:

$$\begin{aligned} a =&C_m C_h / H_m \\ b =&C_m (U_h + H_m)/H_m + C_h \\ c =&U_h\\ d =&Q_{a,t} + Q_{s,t} + U_hT_{o,t}\\ dT_{h,t}/dt =&\left( H_mT_{m,t} - (U_h + H_m) T_{h,t}\right. \\&\left. + U_hT_{o,t} + Q_{h,t} + Q_{s,t}\right) /C_h. \end{aligned}$$

The following coefficient are then computed:

$$\begin{aligned} r_1 =&(-b + \sqrt{b^2 - 4ac})/2a\\ r_2 =&(-b - \sqrt{b^2 - 4ac})/2a\\ A_1 =\,&(r_2 T_{h,t} - dT_{h,t}/dt - r_2d/c)/(r2-r1)\\ A_2 =&T_{h,t} - d/c - A_1\\ A_3 =&(r_1 C_h + U_h + H_m)/H_m\\ A_4 =&(r_2 C_h + U_h + H_m)/H_m. \end{aligned}$$

These coefficients are finally applied to the following dynamic equations:

$$\begin{aligned} T_{h,t+1} =&A_1 e^{r_1 \delta t} + A_2 e^{r_2 \delta t} + d/c \\ T_{m, t+1} =\,&A_1A_3 e^{r_t \delta t} + A_2A_4 e^{r_2 \delta _t} + d/c. \end{aligned}$$

1.1.1 C.1.1: Solar gain

It is possible to add the solar gain to the simulator. It is computed based on the CIBSE Environmental Design Guide (CIBSE, 2015).

The house’s lighting characteristics $\theta _S$, which include the window area and the shading coefficient of 0.67 are needed to model the solar gain, $Q_{s,t}$.

Then, the following assumptions are made:

• The latitude is 30$^\circ$.

• The solar gain is negligible before 7:30 am and after 5:30 pm at such latitude.

• The windows are distributed evenly around the building, in the 4 orientations.

• All windows are vertical.

This allows us to compute the coefficients of a fourth-degree bivariate polynomial to model the solar gain of the house based on the time of the day and the day of the year.

1.2 C.2: Detailed air conditioner model

Once again based on the Gridlab-D Residential module user’s guide (Betelle Memorial Institute, 2022), we model the air conditioner’s power consumption $P_{a,t}$ when turned on, and the heat removed from the air $Q_{a,t}$, based on its characteristics $\theta _H$, such as cooling capacity $K_a$, coefficient of performance $COP_a$, and the latent cooling fraction $L_a$.

$COP_a$ and $L_a$ are considered constant and based on default values of the guide: $COP_a = 2.5$ and $L_a = 0.35$. We have:

$$\begin{aligned} Q_{a,t}&= - \dfrac{K_a}{1+L_a} \\ P_{a,t}&= \dfrac{K_a}{COP_a}. \end{aligned}$$

We set $K_a$ to 15 kW, or 50 000 BTU/hr, to be able to control the air temperature even with high outdoor temperatures. This is higher than most house ACs, but allows to have sufficient flexibility even at high outdoor temperatures (a 5kW AC would have to be always on to keep a 20$^{\circ }$C temperature when it is 38$^{\circ }$C outside). This choice does not significantly affect our results: with lower outdoor temperatures, the problem is equivalent with lower AC power.

1.3 C.3: Regulation signal

1.3.1 C.3.1: Interpolation for the base signal

As described in Sect. 3.1.4, we estimate $D_{a,t}$ by interpolation. A bang-bang controller is ran without lockout for 5 min, and we compute the average power that was consumed. This gives a proxy for the amount of power necessary in a given situation.

A database was created by estimating $D_{a,t}$ for a single house for more than 4 million combinations of the following parameters: the house thermal characteristics $\theta _h$, the differences between its air and mass temperatures $T_{a,t}$ and $T_{m,t}$ and the target temperature $T_T$, the outdoor temperature $T_{o,t}$, and the AC’s cooling capacity $K_a$. If the solar gain is added to the simulation, the hour of the day and the day of the year are also considered.

When the environment is simulated, every 5 min, $D_{a,t}$ is computed by summing the interpolated necessary consumption of every house of the cluster. The interpolation process is linear for most parameters except for the 4 elements of $\theta _h$ and for $K_a$, which are instead using nearest neighbours to reduce the complexity of the operation.

1.3.2 C.3.2: Perlin noise

1-D Perlin noise is used to compute $\delta _{\Pi ,t}$, the power generation high-frequency element. Designed for the field of computer image generation, this noise has several interesting properties for our use case.

Perlin noise is most of the time generated by the superposition of several sub-noises called octaves. It is possible to restrict the span of the values that they can take. Thus, it is possible to test the agents in an environment taking into account several frequencies of non-regular noise, but whose values are restricted within realistic limits. Moreover, the average value of the noise can be easily defined and does not deviate, which ensures that for a sufficiently long time horizon, the noise average is 0.

Each octave is characterized by 2 parameters: an amplitude and a frequency ratio. The frequency represents the distance between two random deviations. The amplitude represents the magnitude of the variation. Normally the frequency increases as the amplitude decreases. This way, high-amplitude noise is spread over a wider interval and lower amplitude noise is more frequent and compact. This is illustrated in Fig. 7.

Fig. 7

Illustration of how several octaves add up to form Perlin noise. The frequency of the octaves increases as their amplitude decreases

In our case, we use 5 octaves, with an amplitude ratio of 0.9 between each octave and a frequency proportional to the number of the octave.

Appendix D: Algorithm details

1.1 D.1: Model Predictive Control

Our MPC is based on a centralized model. At each time step, information about the state of the agents is used to find the future controls that minimize the reward function over the next H time steps. The optimal immediate action is then communicated to the agents. At each time step, the algorithm calculates the ideal control combination for the H-time step horizon.

The cost function for both the signal and the temperature to minimize being the RMSE, the problem is modeled as a quadratic mixed-integer program. The solver used to solve the MPC is the commercial solver Gurobi 9.5.1 (Gurobi Optimization, 2022) together with CVXPY 1.3 (Diamond & Boyd, 2016). Gurobi being a licensed solver, its exact internal behavior is unknown to us and it acts as a black box for our MPC. However, we know that it solves convex integer problems using the branch and bound algorithm. The speed of resolution depends mainly on the quality of the solver’s heuristics.

The computation time required for each step of the MPC increases drastically with the number of agents and/or H. To be able to test this approach with enough agents and a rolling horizon allowing to have reasonable performance, it was necessary to increase the time step at which the agents make decisions to 12 s (instead of 4 for other agents).

It was impossible to launch an experiment with the MPC agent for 48 h in a reasonable time. To compensate, we launched in parallel 200 agents having been started at random simulated times. In order to reach quickly the stability of the environment, the noise on the temperature was reduced to 0.05$^{\circ }$C. We then measured the average RMSE over the first 2 h of simulation for each agent.

Despite this, it was impossible to test the MPC with more than 10 agents while kee** the computation time reasonable enough to be used in real time. That is to say, in a time shorter than the duration between two-time steps.

At each time step, the MPC solves the following optimization problem:

$$\min _{a \in \left\{ {0,1} \right\}^{N \times H} } \sum_{t \in H} {\alpha _{{\text{sig}}} } \left( {\sum_{i{{\epsilon }}N} {P_{i,t} } - s_0 } \right)^2 + \alpha _{{\text{temp}}} \sum_{i{{\epsilon }}N} {\left( {T_{h,t,i} - T_{t,t,i} } \right)^2 } ,$$

such that it obeys the following physical constraints of the environment:

$$\begin{aligned} T_{h,t,i}, T_{m,t,i} = F_1(a_{i,t}, T_{h,t-1,i},T_{m,t-1,i}) \ {}&\forall \ t \in H, i \in N \\ P_{h,t,i} = a_{i,t} F_2(\theta _{a}^i) \&\forall \ t \in H, i \in N, \end{aligned}$$

and the lockout constraint:

$$\begin{aligned} l_{max}(a_{i,t} - \omega _{i,t-1}) - \sum _{k=0}^{l_{max}} (1-\omega _{i,t-k})\le 0 \ \forall \ t \in H, i \in N, \end{aligned}$$

where $F_1$ and $F_2$ are convex functions that can be deduced from the physical equations given in Sect. 3.

1.2 D.2: Learning-based methods

1.2.1 D.2.1: TarMAC and MA-PPO

The original implementation of TarMAC (Das et al., 2019) is built over the Asynchronous Advantage Actor-Critic (A3C) algorithm (Mnih et al., 2016). The environments on which it is trained have very short episodes, making it possible for the agents to train online over the whole memory as one mini-batch.

This is not possible with our environment where training episodes last around 16000 time steps. As a result, we built TarMAC over our existing MA-PPO implementation. The same loss functions were used to train the actor and the critic.

The critic is given all agents’ observation as an input.

The actor’s architecture is described in Fig. 8. Agent i’s observations are passed through a first multi-layer perceptron (MLP), outputting a hidden state x. x is then used to produce a key, a value, and a query by three MLPs. The key and value are sent to the other agents, while agent i receives the other agents’ keys and values. The other agents’ keys are multiplied using a dot product with agent i’s query, and passed through a softmax to produce the attention. Here, a mask is applied to impose the localized communication constraints and ensure agent i only listen to its neighbours. The attention is then used as weights for the values, which are summed together to produce the communication vector for agent i. For multi-round communication, the communication vector and x are concatenated and passed through another MLP to produce a new x, and the communication process is repeated for the number of communication hops. Once done, the final x and communication vector are once more concatenated and passed through the last MLP, the actor, to produce the action probabilities.

We take advantage of the centralized training approach to connect the agents’ communications in the computational graph during training. Once trained, the agents can be deployed in a decentralized way.

In order to maintain the privacy constraint of local communications, meaning that information about an agent is only shared with its immediate neighbours, two design choices have been made. First, the agents do not retain any memory of past messages received, relying solely on their present observations to generate messages. Second, the communication is limited to a single hop, ensuring that messages travel only to the neighbouring agents.

Fig. 8

Architecture of the TarMAC-PPO actor

1.2.2 D.2.2: Neural networks architecture and optimization

For MA-DQN as well as for MA-PPO-HE, every neural network has the same structure, except for the number of inputs and outputs. The networks are composed of 2 hidden layers of 100 neurons, activated with ReLU, and are trained with Adam (Kingma & Ba, 2017).

For TarMAC-PPO, the actor’s obs2hidden, hidden2key, hidden2val, hidden2query and actor MLPs (as shown in Fig. 8) all have one hidden layer of size 32. obs2hidden and actor are activated by ReLU whereas the three communication MLPs are activated by hyperbolic tangent. The hidden state x also has a size of 32.

The centralized critic is an MLP with two hidden layers of size 128 activated with ReLU. The input size is the number of agents multiplied by their observation size, and the output size is the number of agents.

For all networks, the inputs are normalized by constants approximating the mean and standard deviation of the features, to facilitate the training. The networks are optimized using Adam.

1.2.3 D.2.3: Hyperparameters

We carefully tuned the hyperparameters through grid searches. Table 9 shows the hyperparameters selected for the agents presented in the paper.

Table 9 Training hyperparameters

Appendix E: $N_\textrm{de}$ and per-agent RMSE

In this section, we discuss the relation between the per-agent signal RMSE of an aggregation of N homogeneous agents if N is multiplied by an integer $k \in \mathbb {N}$.

We consider the aggregation of size kN as the aggregation of k homogeneous groups $g_j$ of N agents which consumes a power $P^j_{g,t} = \sum _i^N P^i_{a,t}$. We have: $P_t = \sum _i^{kN} P^i_{a,t} = \sum _j^k P^j_{g,t}$.

We assume that each group tracks an equal portion of the signal $s^j_t = s_t/k$. We assume that the tracking error $P^j_{g,t} - s^j_t$ follows a 0-mean Gaussian of standard deviation $\sigma _g$. This Gaussian error is uncorrelated to the noise of other groups.

It follows from the properties of Gaussian random variables that the aggregation signal error $P_t - s_t$ follows a Gaussian distribution of mean $\mu _k = 0$ and standard deviation $\sigma _k = \sqrt{k}\sigma _g$ for all $k \ge 1$ with $k \in \mathbb {N}$.

Hence the signal’s RMSE of a group of kN agents, which is a measured estimation of $\sigma _k$, is approximately $\sqrt{k}$ times the RMSE of a group of N agents, which estimates $\sigma _g$. Finally, the per-agent RMSE is computed as the group’s RMSE divided by the number of agents. We therefore have that the per-agent RMSE of kN agents is approximately $\sqrt{k}/k = 1/\sqrt{k}$ times the RMSE of N agents.

This discussion provides an intuitive explanation for the diminution of the relative RMSE when the number of agents increases. However, it is based on the assumption that the error of each group is not biased, which is not necessarily true with our agents. This explains why the RMSEs are not 10 times lower passing from $N_\textrm{de} = 10$ to $N_\textrm{de} = 1000$.