1 Introduction

Energy transition implies a global effort to shift from traditional conventional or fossil fuel-based energy to RES or sustainable energy alternatives. The integration of a higher volume of RES, namely the installed power of the PV systems and wind power plants, is a key component of the energy transition, as they ensure a cleaner environment that would rely less on coal and gas and more on wind and solar energy [1, 2]. AI has a significant role in supporting the energy transition and predicting imbalance volume and electricity prices [3, 4]. Replacing coal and gas is expensive and faces many barriers (such as unemployment), but more efforts should be involved to reduce annual deaths from carbon dioxide emissions [5, 6]. AI will further enhance the smart grid concept that is one of the pillars of the security of the electricity supply [7]. Among many applications, a brief selection is described in the following paragraphs. AI supports a smoother integration of RES into the meshed power systems [8]. AI has the capacity to handle historical data, learn from the previous patterns, predict the output of various RES (solar, wind-based power plants) and assist grid operators in dispatching volatile generation, maintain system stability and mix RES with other sources [9]. AI also optimizes the operation of these systems by predicting energy generation, managing energy storage and minimizing downtime. Moreover, flexibility usage is optimized by AI means, adjusting demand and supply accordingly [10]. AI enhances the management and control of the power system. It analyses real-time data from smart meters and regulators, monitoring the grid performance, identifies anomalies or faults, and optimizes the power flows. AI algorithms enable dynamic grid balancing, where RES generation is matched with demand in real time, reducing reliance on fossil fuel-based backup generation [11]. AI has the potential to revolutionize the energy sector by optimizing energy generation, transmission, distribution and consumption—thus, the entire chain is revitalized by AI. It monitors and automatically controls grid operations to prevent outages and reduce downtime. AI facilitates predictive maintenance of energy infrastructure, such as power plants, transformers and transmission lines. By analysing real-time data provided by sensors and historical maintenance records, AI algorithms predict equipment failures, schedule maintenance activities and optimize asset management strategies [12].

The electricity price forecasting is facilitated by analysing the historical energy market data, trends, other exogenous factors, such as inflation, interest rate and other related prices (CO2 certificates, oil and gas prices), hydrology, market conditions and regulatory policies, to predict energy prices and traded volumes [13, 14]. This prediction of prices provides a valuable estimation to the energy market participants, such as utilities companies, traders and suppliers to make informed decisions regarding energy procurement, planning and investment [15]. Performant price predictions also leverage an effective integration of RES by offering insights into the economic viability of RES projects. Demand response (DR) programmes are further enhanced by AI that provides tools for analysing historical consumption data and extracting patterns, and analysing weather data and market price signals. It can provide personalized recommendations to consumers based on their pattern and preferences on how to reduce energy consumption and shift usage to off-peak hours [16]. AI-powered systems automate and optimize the control of electric devices and households’ appliances to minimize energy waste [17]. By predicting electricity prices, identifying adequate time-of-use (ToU) tariff rates [18] and assessing consumers’ behaviour, AI identifies opportunities for load shifting, rescheduling flexible appliances, optimal charging and discharging schedules of the energy storage systems, direct load control and DR implementation, maximizing the cost savings while supporting grid stability and energy efficiency [19]. It helps consumers, prosumers and storage facilities owners maximize the economic value of storage assets, support RES integration and improve grid resilience [20].

Moreover, the policy and investment decisions are improved by AI that provides insights into the long-term planning and investment decisions associated with the energy transition [21]. By identifying the most favourable regions or power systems nodes for storage and RES development, optimizing investment strategies and supporting the achievement of energy transition goals [22]. Furthermore, AI through digital twin and blockchain technologies sustains the transition to a cleaner, more sustainable energy future. It identifies energy-saving and market opportunities in public and private buildings, industries and individual consumers. By analysing data, AI algorithms detect patterns and anomalies, recommend energy-efficient measures and optimize system performance. This leads to reduced energy waste and lower carbon emissions [23]. Overall, the integration of AI technologies in the energy sector holds great potential for improving energy efficiency, reliability and sustainability [24]. Smarter and more resilient power systems that meet the growing global energy demand are enhanced by means of AI technologies [25]. A brief image of the AI applications helpful in energy transition is presented in Fig. 1.

Fig. 1
figure 1

AI applications in energy transition

Full size image

On a timeline, the balancing market (BM) is the last stage for creating offers from generating units to increase/decrease the output so as to balance the system and adjust the difference between notifications and real-time operation. On long and mid-term, both producers and suppliers sign contracts to sell/buy electricity at lower prices [26, 27]. the long-term contracts regulated market (LTCRM) is characterized by standard products that are rigid especially for RES generation that can be better predicted closer to the real-time operation [28, 29]. These products impose fixed intervals and quantities that can be combined so as to cover the entire supplying interval. However, significant differences exist between requirements in terms of consumption/generation and the contracted quantities. These differences (about 40% of the traded electricity) are settled on the day-ahead market (DAM) [30]. As the electricity is traded in 24 h, more accuracy is expected from the prediction models [31]. Therefore, both suppliers and producers are expected to create more precise bids/offer to trade the electricity on DAM [32, 33], usually at higher prices than prices on LTCRM. On this market, the market price is a uniform price; therefore, the most expensive unit of electricity gives the price for all transactions on DAM. If a market participant offers a price higher than the market price, he will fail to trade on DAM. The risk of trading failure on DAM is usually considered 25–30%. Another chance to trade and not to create an imbalance is the intraday market (IDM) that is similar to DAM [34]. There can be several IDM, three, four or six sessions between DAM and real-time operation [35]. For balancing purposes, generators are required to offer to increase the output, the remaining capacity that was not traded on DAM/IDM and to reduce the output, the entire capacity up to zero or minimum technical power. The risk of no trading on BM is higher (65–75%); therefore, there is a lower probability to trade on BM than on DAM. On BM, usually, the prices are higher than on DAM/IDM and market participants should avoid being in imbalance [36]. The power system imbalance is characterized by sign, volume (MWh—or the lack of energy) and prices for deficit/surplus. If the system is in deficit, additional energy has to be produced by generators and they are paid as they bid. Therefore, the market price is a pay-as-bid price mechanism [37]. Similarly, if the system is in surplus, the output of generating units has to be reduced. The difference that has to be adjusted is the imbalance volume and it influences the sign and the price of imbalance. However, several events and changes in the power system landscape influenced the wholesale electricity markets and balancing market. Events like COVID-19 and conflict in Ukraine and changes such as smart energy hubs and energy communities [38, 39], including higher RES penetration, have led to a highly unpredictable economic environment.

Several studies investigated the imbalance volume forecasting methods using ML and DL. A UK case study for short-term net imbalance volume forecasting through ML and DL was proposed [40]. In recent years, the dynamic nature of energy markets has made price forecasting increasingly vital. This trend is further amplified by the introduction of new roles and business models like aggregating and energy flexibility trading, especially as the European electricity markets continue to open up. The ability to predict energy price trends and flows becomes now crucial for business success. Additionally, the growing use of RES, which are inherently volatile and intermittent, presents further challenges by potentially causing significant imbalances that could destabilize the overall system. To mitigate power imbalances, forecasting the volume of these imbalances is beneficial for both system operators, who can reduce mitigation costs, and market participants, who can maximize profits and manage risks more effectively. This research proposed a DL algorithm to predict the net imbalance volume in the UK market, comparing it with a traditional gradient boosting trees regression and ARIMA models [40]. The primary contributing variables for these models were the historical values of net imbalance volume. The DL model provided a root mean squared error (RMSE) of 200 and mean absolute error (MAE) of 152 MWh. In contrast, the gradient boosting trees model shows an RMSE of 203 and MAE of 154 MWh, compared to an ARIMA model with 226 and 173 MWh, respectively. Moreover, methods for predicting load using LSTM and genetic algorithms providing a comparison of the ML algorithms are implemented in [41, 42], LSTM for univariate household energy forecasting in [43], ensemble learning approach for demand forecasting in [44] and LSTM for predicting wind generation in [45]. Additionally, a survey of LSTM and related models in wind energy predictive analytics was provided [46]. These methods are versatile and can be implemented for various purposes including transaction frauds detection [47], detecting phishing [48], customer satisfaction [49], stock price prediction [50], etc.

The energy transition is not just a technical shift, but a critical response to the escalating impacts of climate change, necessitating an urgent move from fossil fuels to RES. This transition, while essential, presents significant challenges, especially in integrating a higher volume of variable power sources such as solar and wind. These challenges underscore the need for innovative solutions to manage energy generation variability and ensure a stable supply. In this context, the economic and social impacts of moving away from conventional energy sources cannot be understated. Transitioning to RES involves navigating economic implications, like the cost of replacing coal and gas, and addressing potential unemployment in traditional energy sectors. Simultaneously, this transition has the potential to significantly reduce annual deaths caused by carbon dioxide emissions, highlighting its importance beyond environmental concerns. AI emerges as a pivotal tool in this landscape. Its role extends beyond forecasting and balancing to facilitating the integration of smart grids, optimizing RES use and enhancing the efficiency of energy distribution and consumption. This includes AI’s capability to support complex tasks such as electricity price forecasting, which directly influences decisions made by energy market participants and affects the economic viability of RES projects. AI’s impact is also profound in the realm of DR programmes. These programmes, powered by AI, lead to more efficient energy use and support the integration of RES. AI enables personalized energy consumption strategies, optimizing cost savings and grid stability. Furthermore, AI plays a critical role in informing long-term policy and investment decisions related to the energy transition. By identifying optimal locations for RES and storage and optimizing investment strategies, AI supports the overarching goals of the energy transition.

Emerging technologies like digital twins and blockchain further enhance the potential of AI in the energy sector. These technologies identify energy-saving opportunities and optimize system performance across various sectors, leading to reduced energy waste and lower carbon emissions. Understanding the dynamics of different electricity markets, such as the BM, DAM and IDM, is crucial. AI’s role in these markets includes aiding in navigating market complexities, minimizing imbalances and reducing risks associated with electricity trading. Lastly, the backdrop of recent global events and changes in the energy landscape, such as the COVID-19 pandemic and geopolitical conflicts, add layers of complexity to the energy market. These developments make the need for advanced AI-driven solutions even more pronounced. By exploring these facets, this paper underscores the multi-dimensional and evolving nature of the energy transition, highlighting the critical role of AI in navigating its challenges and opportunities. This approach paints a comprehensive picture of the research context, emphasizing the significance of the problem being addressed. To synthesize the findings, Table 1 highlights the key characteristics of previous studies, focusing on their differences and constraints.

Table 1 Comparison of previous studies
Full size table

Table 1 provides a condensed view of the wide range of studies, highlighting their focus areas, differences and constraints. Based on the context provided by the previous work, several limitations in the existing approaches to predicting electricity market trends and imbalance volumes can be identified. The following major limitations relevant to our proposal are identified. (a) Complexity in BM dynamics: the BM involves complex interactions and decisions. Previous works did not adequately capture this complexity, especially in terms of the decision-making process of producers in creating offers on the BM. (b) Inadequate handling of exogenous variables: prior methods did not efficiently incorporate exogenous variables, which can affect the accuracy of the predictions. (c) Reliance on single forecasting techniques: relying solely on either ML classifiers or traditional forecasting methods did not provide the comprehensive analysis required for accurate imbalance volume prediction. Our proposal aims to address these limitations by using a combination of classification ML algorithms and RNN, particularly with the memory capabilities of LSTM models. Our approach is designed to enhance the accuracy of imbalance volume predictions by incorporating both the classification of imbalance signs and the memorization of variations in the data.

Compared with the previous research, the main contribution of the current work consists of combining ML algorithms (such as: eXtreme gradient boosting (XGB), light gradient boosting model (LGBM), random forest (RF) and multi-layer perceptron (MLP)) to perform classification of the sign and RNN to predict the imbalance volume. Its novelty lies in its proposal of a specific method to predict electricity market imbalances using a combination of ML algorithms and RNN, specifically the LSTM model. This approach is innovative in several ways:

  1. 1)

    Combining classification and prediction models: The method involves predicting the imbalance sign (positive or negative) using various classifiers. This prediction is then integrated into a dataset for forecasting the volume of the imbalance. This dual-step approach (classification followed by prediction) is a novel way to handle the complexity of electricity markets.

  2. 2)

    Use of LSTM in electricity market prediction: The application of LSTM models, which are adept at memorizing and utilizing time-series data, to predict electricity market imbalances is a forward-thinking approach. LSTM's ability to remember long-term dependencies makes it particularly suitable for this task where past data significantly influences future outcomes.

  3. 3)

    Incorporation of exogenous variables from the previous day: The method involves shifting certain exogenous variables (market data: prices and quantities from DAM and BM; and operational data: total generation and its breakdown on sources—both renewable and non-renewable, total consumption, exchange power with neighbouring countries) to values from the previous day (d − 1). This indicates a unique approach to incorporating historical data, assuming that the very recent past (day before) has a significant influence on the current day’s market conditions.

  4. 4)

    Focus on BM dynamics: The emphasis on predicting imbalances in the BM and understanding how producers and the TSO interact in this market is particularly relevant for ensuring a stable and efficient energy supply.

  5. 5)

    Aiming at sustainable energy development: The proposed method aligns with broader goals of sustainable development, highlighting the importance of affordable electricity and the role of AI in advancing cleaner, more efficient energy systems.

This combination of advanced predictive modelling, specific focus on energy market dynamics and alignment with sustainable development goals makes the proposed method an innovative contribution to the field of energy market analysis and AI applications in environmental sustainability. The current paper consists of several sections. In Sect. 2, the input data analysis is performed considering the dataset created from scratch from different open data sources. In Sect. 3, a method is proposed to first classify the power system sign and then predict the imbalance volume. The results are presented in Sect. 4, discussions in Sect. 5 and conclusion in the final section.

2 Input Data Analysis

A data set that consists of the following variables is created: date—Date, hour—Hour, total imbalance volume—Total_Imb, electricity price on DAM—Price_DAM, traded volume of electricity on DAM—Q_DAM, total consumption in Romania—Consumption, total generation in Romania—Generation, generation based on coal—Coal_gen, generation based on oil and gas—Oil&Gas_gen, hydro-based generation—Hydro_gen, nuclear based generation—Nuclear_gen, wind generation—Wind_gen, PV generation—PV_gen, biomass generation—Biomass_gen, sold or exchange, either import or export—Exchange, imbalance price for surplus—ImbPrice_s, financial neutrality for surplus—FinNeutrality_s, imbance for deficit—ImbPrice_d, financial neutrality for deficit—FinNeutrality_d, forecast of the solar energy for DAM—GenSolarDAM, forecast of the solar energy for IDM—GenSolarIDM, forecast of the wind energy for DAM—GenWindDAM, forecast of the wind energy for IDM—GenWindIDM, imbalance sign—ImbSign. The data extraction interval started from 1st of February 2021 until 31st of August 2022, as the records from ENTSO-E were available from 1st of February 2021. From the three public data sources, 21 exogenous variables are extracted plus 1—the imbalance volume—as target.

The first two hourly variables: Price_DAM and Q_DAM were scraped from the OPCOM websiteFootnote 1 using two Python libraries: BeautifulSoup and Selenium [51], the next ten variables recorded at 10-min interval were downloaded from Transelectrica website,Footnote 2 then the last nine variables recorded at 15-min interval were downloaded from ENTSO-E website.Footnote 3 The three data sets recorded at different time intervals were merged using date and hour columns. The flowchart of the data pre-processing is presented in Fig. 2. Figure 3 shows the hourly average prices for deficit/surplus and imbalance during the analysed interval.

Fig. 2
figure 2

Flowchart of the input data pre-processing

Full size image
Fig. 3
figure 3

Hourly average prices for deficit/surplus and imbalance during the analysed interval

Full size image

Moreover, the moving average of the total imbalance is displayed in Fig. 3, showing an opposite variation of the prices and volume. The prices for both surplus and deficit were higher when the imbalance was negative, and the system was in deficit. Additionally, Fig. 3 shows two more interesting insights:

  • The prices were higher at peak hours, in the morning and in the evening.

  • On average, the deficit price was always higher than the surplus price; therefore, the market participants would prefer to be balanced or on the surplus side.

The average imbalance volume per day type and hour is depicted in Fig. 4, where 1–5 is Monday–Friday, 6 Saturday and 0 Sunday. The negative values are more present during the weekdays and less on the weekend days.

Fig. 4
figure 4

Average imbalance volume per day type and hour

Full size image

The variations of hourly average consumption, generation based on oil, gas and hydro, price on DAM and imbalance volume are shown in Fig. 5.

Fig. 5
figure 5

Average variations of hourly total consumption, generation based on oil, gas and hydro, price on DAM and imbalance volume

Full size image

Figure 5 displays the variation of the imbalance volume that reflect the link between peak hours when the consumption is usually higher and the deficit volumes that are noticeable at 8 and 9 in the morning and during evening (between 18 and 21).

Figure 6 shows the evolution of the imbalance volume (a) and prices (b) and (c) in the analysed interval. One can notice the shape of the prices with a narrow variation until November 2021, when the prices significantly increased for both deficit and surplus cases. The prices started to increase when the lockdowns were removed and after the war has started in the Black Sea region, increasing the pressure on the economy. On the other hand, the imbalance volume evolution shows more spikes when the prices were lower. Furthermore, one can notice more spikes in 2021 on the positive axis, while in 2022, more spikes were encountered on the negative axis.

Fig. 6
figure 6

Evolution of the imbalance volume (a) and prices (b) and (c) in the analysed interval

Full size image

3 Method

Both machine learning algorithms and recurrent neural networks are trained to accurately predict the imbalance volume. First, the imbalance sign (negative 0 or positive 1) is predicted using several classification algorithms, such as: eXtreme gradient boosting (XGB), light gradient boosting model (LGBM), random forest (RF) and multi-layer perceptron (MLP). eXtreme gradient boosting is a performant ML algorithm that belongs to the family of gradient boosting methods. It was proposed by Tianqi Chen in 2014 and proved its efficiency gaining a widespread utilization [52]. It is based on an ensemble learning technique in which multiple weak learners (such as decision trees) are used to obtain a predictive model by incorporating several optimization processes, and is faster and more accurate than its competitors [53, 54]. Light gradient boosting is also a popular ML algorithm, proposed more recently (in 2017) by Microsoft. Like XGB, LGBM belongs to the family of gradient boosting methods, but it brings exclusive feature bundling and native flexibility to the categorical features that make it particularly performant and scalable especially for large datasets, improving the training speed and memory usage [55, 56]. Random forest is also a well-known ensemble learning technique proposed by Leo Breiman in 2001 [57]. It combines the predictions of multiple individual decision trees that can be parallelized to create a better output. RF trains multiple decision trees on different data subsets using the bootstrapped sampling technique [58]. It combines the predictions to obtain a better robustness avoiding overfitting. Other advantages of RF are handling high-dimensional datasets, outlier robustness and effectiveness [59]. Multi-layer perceptron is a type of artificial neural network (ANN) and is one of the well-known model architectures in DL. It consists of multiple hidden layers of interconnected neurons. They have associated weights that are updated and during training through the process of forward and backpropagation. The backpropagation is an optimization algorithm to further adjust the loss function and minimize the prediction error [60]. To improve its performance, regularization, dropout, batch normalization, activation functions and advanced optimization algorithms can be used [61].

First, the three datasets are merged using date and hour columns. The classification result obtained with one of the above-mentioned classifiers is inserted into the merged dataset that includes the predicted sign after the classification process. Then, a multivariate stacked LSTM model is applied to predict the total imbalance volume. The flowchart of the proposed method is depicted in Fig. 7. LSTM is a recurrent neural network (RNN) proposed by Sepp Hochreiter and Jürgen Schmidhuber in 1997 [62]. RNNs provide feedback connections to process sequential data by maintaining hidden states that represent information from previous time steps (the so-called memory). It may encounter the vanishing gradient problem, having difficulties to capture long-term dependencies in sequential data. By LSTM, the vanishing gradient problem is avoided using a complex gating mechanism that allows the RNN to selectively remember or forget specific data in time [63, 64].

Fig. 7
figure 7

Flowchart of the proposed method

Full size image

The LSTM setup to predict the total imbalance volume is shown in Table 2. The first step consists of splitting a multivariate sequence into samples defining the procedure split_sequences with two parameters. The second step consists of defining the input sequence that has to be an array. The third step is to choose a number of time steps that is one of the split_sequences parameters. The fourth step is to call the procedure split_sequences. The fifth step consists of defining the sequential model, including the number of features or variables (as the model is multivariate). The proposed model is trained using the fit procedure and the prediction is obtained with predict procedure.

Table 2 Steps to perform LSTM for predicting total imbalance volume
Full size table

4 Results

The number of deficit and surplus cases is rather balanced; therefore, the dataset is feasible for processing without SMOTE, ADASYN or other methods to artificially balance the target. Therefore, accuracy is also a metric that can be taken into account. For classification, the AUC score is considered and the best classifier is in this case RF, closely followed by LGBM and XGB (AUC = 0.79 XGB, AUC = 0.82 RF, AUC = 0.7 MLP, AUC = 0.8 LGBM). The confusion matrix for each algorithm is presented in Fig. 8. It compares the accuracy/errors and has two diagonals: the first diagonal should contain the highest values, whereas the second diagonal should contain the lowest values (or errors) to obtain a performant prediction.

Fig. 8
figure 8

Confusion matrix for each classification algorithm (RF, XGB, LGBM, MLP)

Full size image

The two heatmaps with (a) and without (b) the imbalance sign (ImbSign) variable are presented in Fig. 9. The total imbalance is highly correlated with the imbalance sign (0.84), inversely moderately correlated with the price on DAM—PriceDAM ( – 0.43), inversely weakly correlated with the imbalance price for deficit ImbPrice_d ( – 0.27) and ImbPrice_s surplus ( – 0.23).

Fig. 9
figure 9

Heatmaps with (a) and without (b) imbalance sign variable

Full size image

The imbalance sign feature is inserted into the model to predict the total imbalance volume. The training interval starts on 1st of November 2021 and the prediction is daily performed to obtain the hourly forecast (as in Fig. 10).

Fig. 10
figure 10

Training interval for predicting the total imbalance volume

Full size image

We test the prediction model for two intervals:

  • The first 4 days in March 2022: 1st–4th of March 2022 to show the results in detail (as in Appendix A, Figure 13).

  • The first 4 days in April 2022: 1st–4th of April 2022 to show the results in detail (as in Appendix A, Figure 14). Table 3 provides the performance metrics at the daily level, entire months (March and April, 2022) and 8-month interval (January–August 2022).

Table 3 Prediction metrics with and without imbalance sign as input
Full size table

Usually, several performance metrics are calculated as they provide more insights regarding the results. The following metrics were calculated:

  • Mean absolute error (MAE) is a commonly used metric to assess the performance of the predictive model. It also measures the average deviation between the predicted and the actual values (as in Eq. (1)). However, MAE does not involve squaring the errors like RMSE, making MAE less sensitive to outliers and large errors. Particularly, MAE calculates the absolute differences between the predicted and the actual values, showing the absolute differences.

    $${\text{MAE}}=\frac{1}{T}\sum_{h=1}^{T}\left|{y}^{h}-\widehat{{y}_{i}^{h}}\right|$$
    (1)

  • Root mean squared error (RMSE) measures the average deviation between the predicted values and the actual values as in Eq. (2). RMSE is especially useful as it penalizes large errors more than small errors due to the squaring operation (root).

    $${\text{MAE}}=\frac{1}{T}\sum_{h=1}^{T}\left|{y}^{h}-\widehat{{y}_{i}^{h}}\right|$$
    (2)

  • Mean absolute percentage error (MAPE) calculates the percentage difference between the predicted and the actual values (as in Eq. (3)). The absolute percentage differences are important to avoid the issue of positive and negative errors that cancel each other out. MAPE is useful because it provides a relative measure of the forecast performance, which ensures a better comparison across various time series and datasets.

    $${\text{MAPE}}=\frac{1}{T}\sum_{h=1}^{T}\frac{\left|{y}^{h}-\widehat{{y}_{i}^{h}}\right|}{\left|{y}^{h}\right|}\times 100\%.$$
    (3)

  • Coefficient of determination (\({R}^{2}\)) evaluates how a regression model fits the data. It represents the proportion of the variance in the dependent variable that is predictable from the independent variables. Therefore, it assesses the goodness of fit of the regression model to a certain dataset as in Eq. (4). It can also be written using the ratio between \({{\text{SS}}}_{{\text{residual}}}\) and \({{\text{SS}}}_{{\text{total}}}\), where \({{\text{SS}}}_{{\text{residual}}}\) is the sum of squares of the residuals or the sum of squared differences between the actual and the predicted values; and \({{\text{SS}}}_{{\text{total}}}\) is the total sum of squares, which is the sum of squared differences between the actual and the mean of the dependent variable.

    $${R}^{2}=1-\frac{\sum_{h=1}^{T}{\left({y}^{h}-\widehat{{y}_{i}^{h}}\right)}^{2}}{\sum_{h=1}^{T}{\left({y}^{h}- \overline{{y }^{h}}\right)}^{2}}=1-\frac{{{\text{SS}}}_{{\text{residual}}}}{{{\text{SS}}}_{{\text{total}}}}.$$
    (4)

The graphical results (total imbalance volume actual—blue vs. forecast—orange) testing the prediction for the first 4 days in March are depicted in Fig. 11.

Fig. 11
figure 11

Forecast of the imbalance volume for 1st–4th of March with (right) and without (left) imbalance sign

Full size image

Additionally, the training interval spanned from 1st of November 2021 until 31st of March 2022 and we tested the results for the first 4 days of the next month (April, 2022) to better validate the prediction output.

The graphical results (total imbalance volume actual—blue vs. forecast—orange) testing the prediction for the first 4 days in April are depicted in Fig. 12.

Fig. 12
figure 12

Forecast of the imbalance volume for 1st–4th of April with (right) and without (left) imbalance sign

Full size image

From Fig. 12, one can notice that on the 1st, 2nd and 4th, there was visible improvement of the prediction, whereas on the 3rd of April, the improvement was moderate. In the first cases, the proposed prediction model was able to anticipate valley and spikes and the differences between prediction and actual values were rather small. For April, the training interval increased by 31 days that led to a similar performance. Therefore, increasing the training interval did not significantly improve the results. The numerical results in terms of prediction metrics are synthetically provided at the daily, monthly and 8-month interval (January–August 2022) level in Table 3.

5 Discussion and Comparison

This table presents a comparative analysis of performance metrics for two scenarios labelled “Without imbalance sign” and “With imbalance sign” across different dates in 2022. The metrics used are MAE, RMSE, MAPE and R2 (coefficient of determination). For each date, the performance of the proposed is evaluated using these metrics both in the presence (“With”) and absence (“Without”) of the imbalance sign. The data shows a clear trend of improvement in all metrics when the imbalance sign is present, as indicated by lower MAE, RMSE and MAPE values and higher R2 values in the “With” column. This suggests that the imbalance sign being considered has a positive impact on the performance of the process or model. In the following paragraphs, each interval (daily, monthly and 8-month interval) is discussed and compared:

5.1 Daily-Level Analysis (1st–4th of March and April 2022)

  • MAE: There is a consistent decrease in MAE in the “With” case, indicating improved accuracy. For example, on the 1st of March, MAE decreases from 1.352 to 0.762.

  • RMSE: The RMSE values are generally lower in the “With” scenario, suggesting better predictive accuracy. On the 4th of March, RMSE dropped from 0.901 to 0.547.

  • MAPE: The “With” case shows a reduction in MAPE values, indicating more accurate predictions relative to actual values. The reduction is notable on the 1st of March, from 0.109 to 0.039.

  • R2: Higher R2 values in the “With” case across all days indicate better model fit or explanatory power. On the 1st of March, R2 increased from 0.965 to 0.998.

5.2 Monthly-Level Analysis (March and April 2022)

  • MAE: For both March and April, the “With” case shows a significant reduction in MAE. In March, MAE decreases from 1.205 to 0.546.

  • RMSE: The “With” scenario has consistently lower RMSE values in both months, suggesting improved model performance.

  • MAPE: The “With” scenario demonstrates a decrease in MAPE for both months, indicating greater prediction accuracy.

  • R2: There is an increase in R2 in the “With” case for both months, pointing to a better fit of the model to the data.

5.3 Overall Interval Analysis (Jan.–Aug. 2022)

  • MAE: The overall MAE is reduced in the “With” case (from 1.037 to 0.725), indicating better accuracy over the longer period.

  • RMSE: The “With” scenario sees a decrease in RMSE (from 1.428 to 0.987), suggesting improved overall predictive performance.

  • MAPE: The reduction in MAPE in the “With” case (from 0.112 to 0.063) points to more accurate predictions over the longer term.

  • R2: The R2 value is higher in the “With” case (0.982 vs. 0.939), indicating a stronger model performance.

Therefore, the results indicate a clear trend of improved performance in the "With" scenario across all metrics and time frames. This improvement is especially pronounced in the R2 values, which suggest a better fit of the model or process being evaluated. The consistent reduction in MAE, RMSE and MAPE in the "With" case suggests that the intervention or factor applied in this scenario (the imbalance sign classification as input feature) contributes significantly to enhanced accuracy and predictive performance of the proposed method. This trend is consistent across both short-term (daily) and long-term (monthly and interval) analyses.

6 Conclusion

AI applications bring an invaluable assistance in the energy transition towards clean environment. They are widely applied to large-scale systems, being reliable in monitoring and controlling the power grid to optimize its operation states. Additionally, they have proved useful to prosumers, consumers, storage facilities and RES owners, having a major impact on their business models. Optimization algorithms, predictive models, digital twin technologies and blockchain have led to a paradigm shift. The conventional flows from large power plants to end consumers using transmission and distribution infrastructure are no longer the only way to ensure the security of supply. More distributed energy resources (DER) have emerged, bringing new challenges and opportunities, such as the local electricity market and revenue stacking. Mixing DER and AI, residential consumers and commercial enterprises, industry and agriculture benefit from cheaper electricity and more energy independence.

Knowing that the imbalance volume brings benefits to the producers and assist them in creating strategies in trading on the BM, in this paper, we build a dataset by merging three datasets from three websites (OPCOM, Transelectrica and ENSTO-E). The input data was analysed in the context of post-COVID-19 era and war in the Black Sea region. First, the imbalance sign was predicted using four classifiers. The predicted input variable is inserted into the dataset to predict the imbalance volume using a multivariate stacked LSTM model.

Several metrics were used to assess the results. The model was trained using data recorded from 1st of November, whereas the testing intervals took into account the first 4 days from March and April 2022. In March, MAE decreased on average from 1.20 to 0.54 (almost 55%) and R2 increased from 0.95 to 0.99 proving that the proposed solution outperforms the approach without considering the imbalance sign as an input feature. Additionally, in April, MAE decreased from 1.28 to 0.68, whereas the coefficient of determination increased from 0.92 to almost 0.97, showing that the prediction model is more accurate. Regarding the performance metrics between two scenarios, referred to as “With imbalance sign” and “Without imbalance sign”, observed from January to August 2022, it highlights that in the “With” case, there is a significant improvement in accuracy, as evidenced by the reduction in the MAE from 1.037 to 0.725. Similarly, the RMSE decreases from 1.428 to 0.987 in the "With" scenario, suggesting enhanced predictive performance. Additionally, the MAPE in the “With” case shows more accurate long-term predictions, drop** from 0.112 to 0.063. Finally, the R2 value, which indicates the strength of the model’s performance, is higher in the “With” case (0.982) compared to the “Without” scenario (0.939). Overall, these improvements across all metrics in the “With” scenario point to better accuracy, predictive performance, and method strength.