1 Introduction

The quantification of energy transport between ecosystems and the atmosphere provides an important basis for various areas of environmental science. Some examples are operational weather forecasting and climate modelling (e.g. Arneth et al. 2012; Cuxart et al. 2015; Green et al. 2017), or investigating the reaction of ecosystems to changing climate conditions (e.g. Cremonese et al. 2017; Fu et al. 2020; Qu et al. 2016; Reichstein et al. 2007; van Gorsel et al. 2016) to come up with sustainable management strategies for ecosystems and adapt agriculture (e.g. Bernacchi et al. 2005, 2006; Ceschia et al. 2010; Graham et al. 2016; O’Dell et al. 2020; Stork and Menzel 2016). For decades, the eddy covariance (EC) method has been used to quantify energy transport in the atmospheric boundary layer in terms of sensible and latent heat fluxes at the ecosystem scale, as it facilitates the direct measurement of turbulent energy fluxes without disturbing the investigated ecosystem (Aubinet et al. 2012; Baldocchi 2003, 2014; Foken 2017; Mauder et al. 2007a).

However, the systematic study of measurements at EC stations has shown that the surface-energy balance (SEB, the balance between incoming and outgoing energy at the surface in the form of radiation, heat fluxes and storage change) is usually not closed, but shows a gap of 10–30%, always in the same direction (Hendricks-Franssen et al. 2010; Soltani et al. 2018; Stoy et al. 2013). The SEB can therefore be expressed in simplified form as:

$$ R_{{{\text{net}}}} = H_{t} + \lambda E_{t} + G + I, $$
(1)

where \({R}_{net}\) is the net radiation, which is defined to be positive during daytime, \({H}_{t}\) and \({\lambda E}_{t}\) are the sensible and latent turbulent fluxes measured with the EC method, \(G\) is the ground heat flux and \(I\) is the remaining SEB gap (Mauder et al. 2020). The terms on the right-hand side of the equation are defined to be positive when they transport energy away from the surface.

The SEB gap has been the subject of research for a long time and many factors, such as systematic errors in the measurement of the vertical wind component and humidity (Frank et al. 2013; Goulden et al. 1996; Kochendorfer et al. 2012; Laubach et al. 1994; Mauder 2013; Nakai and Shimoyama 2012), which are used to calculate heat fluxes, and systematic errors in the measurement of other components of the SEB, such as the radiation and ground heat flux (Foken 2008; Kohsiek et al. 2007; Liebethal et al. 2005), and the scale mismatch between the footprint of the EC measurement and the radiation measurement (Schmid 1997) have been studied. Especially for measurement towers above high vegetation, the energy storage in the air volume and biomass below the measurement instruments also plays a non-negligible role and should be captured by profile measurements of temperature and humidity (Leuning et al. 2012; Lindroth et al. 2010; Moderow et al. 2009; Xu et al. 2019). The influences of systematic measurement errors were minimized or can be accounted for by appropriate measures in the data processing (Mauder et al. 2006; Mauder and Foken 2006; Twine et al. 2000) but a significant gap in the SEB remains (Mauder et al. 2006, 2007c, 2020).

One reason for the remaining SEB gap could be the fact that the EC method, given typical approaches and measurement limitations, only captures turbulent transport: the flux is calculated as the covariance of the vertical wind speed and a scalar of interest (e.g., temperature for the sensible heat flux \(H\) or humidity for the latent heat flux \(\lambda E\)) from high-frequency time series. The covariance is based on the fluctuations around an average over a certain averaging period of typically 30 min (Aubinet et al. 1999; Mauder et al. 2020). The averages of the wind speed and scalar of interest are not considered and any energy transport by structures that contribute to the mean vertical wind speed is neglected (Metzger and Holmes 2007). The temporal or spatial averaging period used to represent the Reynolds’ average defines which atmospheric structures ultimately contribute to “turbulent” and which contribute to “non-turbulent” transport.

After assuming for a long time that energy transport in the atmospheric boundary layer (ABL) is exclusively turbulent in nature, i.e., that all relevant transport is captured by applying 30-min averaging intervals, it has been found that large eddies, also known as secondary circulations (SCs) reach near the surface (Eder et al. 2015; Patton et al. 2016). SCs contribute to the mean wind speed at typical EC averaging periods, which is why the energy transport by SCs is not considered in EC measurements and instead contributes to the SEB gap (Foken 2008; Mauder et al. 2020).

Two main types of SCs can be distinguished. The first type are randomly develo** large eddies that are turbulent in nature but so large that they are not considered in the covariance over the averaging period and even form over homogeneous surfaces, so called turbulent organized structures (TOSs) (Inagaki et al. 2006; Kanda et al. 2004). The second type are systematically develo** secondary circulations caused by differential heating at the earth surface and therefore bound in space, so called thermally-induced mesoscale circulations (TMCs) (Bou-Zeid et al. 2020; Foken 2008; Kenny et al. 2017; Mauder et al. 2020). Increasing the averaging period might sample more of the energy transported due to the randomly organized and quasi-stationary TOSs with longer time scales as they move slowly with the mean wind within the tower footprint. However, TMCs are spatially bound to surface characteristics which is why they still contribute to the mean wind even with long averaging periods as they might not be sampled by fixed-point EC tower measurements. This may explain why in some cases, increasing the averaging period had no effect on the SEB closure in some cases (Charuchittipan et al. 2014) but improved the SEB closure, though not entirely resolved it, in other cases (Finnigan et al. 2003; Mauder et al. 2006). Furthermore, increasing the averaging period in EC measurements beyond multiple hours will eventually violate the assumption of stationarity (Mauder et al. 2006, 2020). Wavelet-based flux calculation approaches that incorporate longer wavelength turbulent transport have been shown to improve the SEB gap, also implying a role for SCs in energy transport (Metzger et al. 2013; Xu et al. 2020). The SEB gap is therefore to some extent a result of the different scales at which ecosystem-atmosphere interactions take place (Desai et al. 2022b).

The contribution of the transport by secondary circulations can be quantified by spatial measurements using an array of tower measurements or aircraft measurements (Feigenwinter et al. 2008; Mahrt 1998; Mauder et al. 2008; Paleri et al. 2022b; Steinfeld et al. 2007). However, those measurements are expensive and cannot be carried out at each EC station or over long periods. Another possibility is to model the energy transport by secondary circulations based on atmospheric and landscape parameters that can be gathered for single-tower EC measurements (e.g. Choi et al. 2009; De Roo et al. 2018; Eder et al. 2014; Huang et al. 2008; Mauder et al. 2021; Panin and Bernhofer 2008).

Systematic studies at multiple EC sites were carried out that found surface heterogeneity (Foken et al. 2010; Mauder et al. 2007b; Morrison et al. 2021; Panin et al. 1998; Panin and Bernhofer 2008), friction velocity \({u}_{*}\) (Barr et al. 2006; Hendricks-Franssen et al. 2010; Stoy et al. 2013; Wilson et al. 2002), and atmospheric stability (Barr et al. 2006; Hendricks-Franssen et al. 2010; Stoy et al. 2006, 2013) to be related to the SEB gap. The SEB gap was also studied with large-eddy simulations (LES), which also found surface heterogeneity (De Roo and Mauder 2018; Zhou et al. 2019), \({u}_{*}\) (Schalkwijk et al. 2016) and atmospheric stability (Huang et al. 2008; Zhou et al. 2018) were influencing the SEB gap.

That the SEB gap depends on atmospheric stability follows from the fact that convectively driven eddies predominate under unstable conditions. Their vertical extent can reach the ABL depth (Maronga and Raasch 2013; Sühring and Raasch 2013), and their horizontal extent can reach two to three times the boundary layer depth (Paleri et al. 2022a; Stull 1988). These large eddies are not captured in 30-min EC measurements and therefore are considered TOSs. Furthermore, large horizontal wind speeds that are associated with neutral to stable atmospheric conditions increase the horizontal mixing (Katul 2019; Schalkwijk et al. 2016), and TOSs are carried along faster with the wind, increasing the likelihood of them being captured within a typical 30-min measurement period. The thermal surface heterogeneity affects the SEB gap through its influence on the formation of SCs: different surface properties of the individual patches cause the surfaces and also the air above them to heat up to different extents, resulting in the formation of TMCs in addition to the TOSs that are already randomly formed. The magnitude of the TMCs depends on the prevalent size and temperature amplitude of the individual surface patches (Inagaki et al. 2006; Letzel and Raasch 2003; Sühring et al. 2018; Zhou et al. 2019).

Several studies proposed models of the SEB gap based on atmospheric stability and measurement height (e.g. De Roo et al. 2018; Huang et al. 2008). Recently, a model of the SEB gap including the effect of thermal surface heterogeneity was published (Wanner et al. 2022b), however, it was only developed to predict the imbalance in the sensible heat flux. These models all have in common that they model the entire SEB gap, i.e., the gap between the available energy at the surface and the measured turbulent heat fluxes.

However, the energy being stored in the air layer and biomass beneath the instruments were identified as another major contribution to the SEB gap in EC measurements over tall vegetation, as mentioned above. We hypothesize that the amount of energy supplied to this storage does not depend on the same factors as the magnitude of the energy transport by secondary circulations. It is possible to measure the energy storage change in the air by simply adding few sensors below the EC instrumentation (Lindroth et al. 2010; Moderow et al. 2009). This is already implemented by default in some large EC networks like ICOS (Heiskanen et al. 2022) and provides a more direct way of obtaining the storage change contribution to the SEB than predicting it by a model. Since storage change is being routinely measured at EC sites and its effect on the long-term mean SEB is negligible, we propose to measure the storage directly and model the energy transport by secondary circulations separately.

Considering the mentioned contributions to the SEB gap, we can rewrite Eq. 1 as:

$$ R_{net} = H_{t} + \lambda E_{t} + H_{nt} + \lambda E_{nt} + G + H_{\Delta St,a} + \lambda E_{\Delta St,a} + H_{\Delta St,b} + I, $$
(2)

where \({H}_{nt}\) and \({\lambda E}_{nt}\) are non-turbulent transport of sensible and latent heat which is missed by EC the measurement, \({H}_{\Delta St,a}\) and \({\lambda E}_{\Delta St,a}\) are storage changes of sensible heat in the air volume below the EC measurement, and \({H}_{\Delta St,b}\) is the storage change of sensible heat in the biomass. If a small volume surrounding the EC site is considered, the non-turbulent fluxes \({H}_{nt}\) and \({\lambda E}_{nt}\) contribute to horizontal and vertical advection. In an infinite observation area, they contribute to the dispersive fluxes (Raupach and Shaw 1982; Wilson and Shaw 1977), that are calculated as the spatial covariance of 30-min averaged values of \(w\) and \(\theta \) or \(q\). This yields:

$${H}_{d}(z)=\langle {\overline{w} }^{*}{\overline{\theta }}^{*}\rangle (z) {c}_{p}\rho, $$
(3)

for the sensible dispersive heat flux \({H}_{d}\), where \({c}_{p}\) is the specific heat capacity of air and \(\rho \) is the air density. The overbar refers to temporal averaging over 30 min and angular brackets denote spatial averaging over the observation area. The upper * refers to spatial fluctuations which are averaged over all grid boxes horizontally. The spatial covariance is therefore defined as:

$$\langle {\overline{w} }^{*}{\overline{\theta }}^{*}\rangle \left(z\right)=\frac{1}{{n}_{x}\cdot {n}_{y}}\sum_{x=0}^{{x}_{max}}\sum_{y=0}^{{y}_{max}}\left(\overline{w }\left(x,y,z\right)-\langle \overline{w }\rangle \left(z\right)\right)\left(\overline{\theta }\left(x,y,z\right)-\langle \overline{\theta }\rangle \left(z\right)\right),$$
(4)

where and \({n}_{x}\) and \({n}_{y}\) refer to the number of grid points in x- and y-direction. The latent dispersive heat flux \({\lambda E}_{d}\) is calculated similarly following:

$${\lambda E}_{d}(z)=\langle {\overline{w} }^{*}{\overline{q} }^{*}\rangle (z) {\lambda }_{v}\rho, $$
(5)

where \({\lambda }_{v}\) is the latent heat of vaporization and:

$$\langle {\overline{w} }^{*}{\overline{q} }^{*}\rangle \left(z\right)=\frac{1}{{n}_{x}\cdot {{n}_{y}}}\sum_{x=0}^{{x}_{max}}\sum_{y=0}^{{y}_{max}}\left(\overline{w }\left(x,y,z\right)-\langle \overline{w }\rangle \left(z\right)\right)\left(\overline{q }\left(x,y,z\right)-\langle \overline{q }\rangle \left(z\right)\right).$$
(6)

By considering \({H}_{nt}\), \({\lambda E}_{nt}\), \({H}_{\Delta St,a}\), \({\lambda E}_{\Delta St,a}\), and \({H}_{\Delta St,b}\) \(I\) becomes smaller. Therefore, to close the energy balance gap eventually, all known factors contributing to the energy balance must be quantified.

Since the long-term measurement of non-turbulent fluxes at all EC sites is not possible, the main objective of this study is to develop a model that can predict the transport of sensible and latent heat by SCs under unstable conditions at the site of an EC station based on as few additional measurements as possible. Further objectives are (1) to test how well the model predicts the partitioning into sensible and latent energy transport by secondary circulations in a more realistic LES, and (2) to show how the model can be applied to field measurements and test if considering the modelled dispersive fluxes results in an SEB closure in EC field measurements. The main objective is achieved by further develo** an existing model (Wanner et al. 2022b). Following De Roo et al. (2018) it uses the atmospheric stability measure \({u}_{*}/{w}_{*}\), where \({w}_{*}\) is the Deardorff velocity, and the measurement height \(z\) normalized with the atmospheric boundary-layer height \({z}_{i}\), as well as a thermal surface heterogeneity parameter \(\mathcal{H}\) (Margairaz et al. 2020b) to predict the entire energy balance gap. The new model developed in this study additionally considers the gradients of potential temperature \(\theta \) to predict the non-turbulent transport of \(H\) and mixing ratio \(q\) to predict the non-turbulent transport of \(\lambda E\). We hypothesize that the consideration of these gradients improves the prediction of energy transport by SCs since they define whether there is any energy to be transported by secondary circulations. The model is based on a large set of idealized LESs introduced in Sect. 2.1 and also contributes to a better understanding of the partitioning of the SEB gap into sensible and latent heat as well as non-turbulent fluxes and storage change, respectively.

The additional objectives to test the model are addressed by applying the model to more realistic LES (Sect. 2.3, Paleri et al. (2023)) and field measurements at EC stations (Sect. 2.4) from the CHEESEHEAD19 campaign (Butterworth et al. 2021). The application to the more realistic LES allows a separate analysis of the surface balances of sensible and latent heat, which is not possible with field measurements, where only the total available energy can be used as a reference. The test using the field measurements is intended to show how practicable the model application is and how well the findings from the LES can be transferred to the real world.

The development and application of the model are described in Sect. 2. The results are shown in Sect. 3 and discussed in Sect. 4.

2 Methods

2.1 Set-up of Idealized Large-Eddy Simulations

We used PALM V6, revision 4849 to run a set of idealized simulations. PALM is a parallelized LES model based on the non-hydrostatic incompressible Boussinesq equations (Maronga et al. 2020; Raasch and Schröter 2001). The sub-grid model was based on the kinetic energy scheme of Deardorff (1980) which was modified by Moeng and Wyngaard (1988) and Saiki et al. (2000). A fifth-order scheme (Wicker and Skamarock 2002) was used to discretize the advection terms, and for the time integration, a third-order Runge–Kutta scheme (Williamson 1980) was used.

In total, we ran 148 simulations with different combinations of geostrophic wind speeds (\({U}_{g}\), 0.5–9.0 m s−1), surface patch sizes of sensible (\({H}_{{\text{s}}}\)) and latent (\({\lambda E}_{{\text{s}}}\)) surface fluxes (homogeneous, 200 m, 400 m, 800 m), and Bowen ratios (\(\beta \)) of surface fluxes (0.1–1.3), following the setup used in Margairaz et al. (2020a) and Wanner et al. (2022b). The randomly assigned surface fluxes followed a Gaussian normal distribution, where the standard deviation was also varied. Table 6 in Appendix 1 gives an overview over the individual simulations. The variation in \({U}_{g}\), \({H}_{{\text{s}}}\), and \({\lambda E}_{{\text{s}}}\) led to differences in atmospheric stability and the variation in heterogeneity scales of surface fluxes caused the development different patterns of surface temperature (\({T}_{{\text{s}}}\)). Examples of the spatial distribution of surface fluxes and resulting surface temperatures can be found in Fig. 11 (Appendix 1).

Following Margairaz et al. (2020a) and Wanner et al. (2022b), the horizontal grid spacing was 24.5 m and the vertical grid spacing was 7.8 m. With (x,y,z) = 256 × 256 × 256 grid points, this resulted in a horizontal domain extent of 6272 × 6272 m2 and a vertical domain extent of roughly 2000 m. All simulations were carried out with cyclic horizontal boundary conditions. At the lower boundary, Dirichlet boundary conditions were used for the wind velocity components, while Monin–Obukhov similarity theory was used to determine the lower boundary condition for the momentum equations, and Neumann boundary conditions were used for potential temperature, humidity, pressure and turbulent kinetic energy. The roughness length was set to 0.1 m, following Margairaz et al. (2020a) and Wanner et al. (2022b).

The initialization profiles were set up following De Roo et al. (2018). The same horizontally homogeneous initial vertical profiles of potential temperature \(\theta \) and mixing ratio \(q\) were used in all 148 simulations. The surface temperature was horizontally homogeneous and set to 285 K. A vertical gradient of 3·10−3 K m−1 was applied between 40 and 800 m and a vertical gradient of 8·10−3 K m−1 was applied between 800 and 1000 m, ensuring that the top of the domain lay within an inversion layer and the atmospheric processes in the boundary layer were not affected by the upper border of the domain. The humidity was set to 1·10−3 kg kg−1 at the surface and only between 1000 and 1100 m, a vertical gradient of − 1·10−5 kg kg−1 m−1 was applied.

The initial profile of horizontal wind speed was horizontally and vertically constant. The wind speed varied among the simulations between 0.5–9.0 m s−1 (Table 6, Appendix 1) and generated a horizontal pressure gradient in x-direction. A small vertical pressure gradient to counteract excessive boundary-layer growth was introduced by a vertical subsidence velocity gradient (− 4·10−5 s−1 between 0 and 800 m and − 2·10−5 s−1 between 8 and 1000 m).

After a 4-h spin-up period, the output data was collected for a 30-min averaging period representing typical EC averaging periods (Rebmann et al. 2012). The profiles of temperature, humidity, and horizontal wind speed during the data collection period is shown as an example for two simulations in Fig. 12, Appendix 1.

2.2 Development of the Dispersive Flux Model

2.2.1 Data Processing

Following the approach of former models, we developed a model using horizontally domain-averaged values from an idealized set of LESs. Since the idealized LESs had cyclic horizontal boundary conditions, the horizontal and vertical advection fluxes (\({H}_{a}\), \({\lambda E}_{a}\)) became zero on the ecosystem scale and all energy transport by secondary circulations was combined in the dispersive fluxes. Therefore, the horizontally averaged total available energy at the surface, i.e., the heat fluxes originating from the surface (\(\langle {H}_{{\text{s}}}\rangle \), \(\langle {\lambda E}_{{\text{s}}}\rangle \)), was divided into horizontally averaged non-organized randomly distributed turbulent fluxes (\(\langle {H}_{t}\rangle \), \(\langle {\lambda E}_{t}\rangle \)), including the sub-grid-scale contributions (\(\langle {H}_{{\text{sgs}}}\rangle \), \(\langle {\lambda E}_{{\text{sgs}}}\rangle \)), dispersive fluxes (\({H}_{d}\), \({\lambda E}_{d}\)), and horizontally averaged storage change of energy in the underlying air mass (\(\langle {H}_{\Delta St}\rangle \), \(\langle {\lambda E}_{\Delta St}\rangle \)) as follows:

$$\langle {H}_{{\text{s}}}\rangle +\langle {\lambda E}_{{\text{s}}}\rangle =\langle {H}_{t}\rangle +\langle {\lambda E}_{t}\rangle +{H}_{d}+{\lambda E}_{d}+\langle {H}_{\Delta St}\rangle +\langle {\lambda E}_{\Delta St}\rangle +{H}_{a}+{\lambda E}_{a},$$
(7)

with \({H}_{a}={\lambda E}_{a}=0\) W m−1.

The 30-min horizontally averaged turbulent sensible heat flux \(\langle {H}_{t}\rangle \) was calculated for each grid level up to \(z/{z}_{i}= 0.1\) using the 30-min averaged vertical wind speed \(w\) and potential temperature \(\theta \), the 30-min averaged temporal covariance of \(w\) and \(\theta \), and the sub-grid-scale contribution \({H}_{{\text{sgs}}}\), following:

$$\langle {H}_{t}\rangle \left(z\right)= \langle \overline{w\theta }\left(z\right)-\overline{w }\left(z\right) \overline{\theta }\left(z\right)\rangle {c}_{p}\rho +\langle {H}_{{\text{sgs}}}\rangle .$$
(8)

The horizontal averaging was performed over the entire extent of the domain. The same procedure was used to calculate the horizontally averaged 30-min turbulent latent heat flux \(\langle {\lambda E}_{t}\rangle \), however, instead of \(\theta \), the mixing ratio \(q\) was used:

$$\langle {\lambda E}_{t}\rangle \left(z\right)= \langle \overline{wq }\left(z\right)-\overline{w }\left(z\right) \overline{q }\left(z\right)\rangle {\lambda }_{v}\rho +\langle {\lambda E}_{{\text{sgs}}}\rangle .$$
(9)

The dispersive sensible and latent heat fluxes \({H}_{d}\) and \({\lambda E}_{d}\) were calculated over the entire horizontal extent of the domain following Eqs. 3–6. Equations 3 and 5 show that \({H}_{d}\) and \({\lambda E}_{d}\) were by definition already horizontally averaged.

Since the LESs had cyclic boundary conditions and any other flux contributions could be ruled out, the change of energy stored in the underlying air mass (\({H}_{\Delta St}\), \({\lambda E}_{\Delta St}\)) was calculated as the residual of the other contributions to the total available surface flux:

$$\langle {H}_{\Delta St}\rangle =\langle {H}_{{\text{s}}}\rangle -\langle {H}_{t}\rangle -{H}_{d},$$
(10)

and:

$$\langle {\lambda E}_{\Delta St}\rangle =\langle {\lambda E}_{{\text{s}}}\rangle -\langle {\lambda E}_{t}\rangle -{\lambda E}_{d}.$$
(11)

To model \({H}_{d}\) and \({\lambda E}_{d}\), we proposed variables of atmospheric stability \(\langle {u}_{*}/{w}_{*}\rangle \), the thermal heterogeneity parameter \(\mathcal{H}\), the measurement height \({z}_{m}\) normalized with the atmospheric boundary layer height \(\langle \overline{{z }_{i}}\rangle \), and the vertical differences of potential temperature theta \(\langle \Delta \overline{\theta }\rangle \) (for \({H}_{d}\)) and mixing ratio \(\langle \Delta \overline{q }\rangle \) (for \({\lambda E}_{d}\)), respectively, as our driving factors.

We calculated \(\langle {u}_{*}/{w}_{*}\rangle \), for one grid level near the surface based on the 30-min averaged covariances following:

$${u}_{*}={\left({({\overline{{u }{\prime}{w}{\prime}}}_{{\text{res}}}+\langle {\overline{{u }{\prime}{w}{\prime}}}_{{\text{sgs}}}\rangle )}^{2}+{({\overline{{v }{\prime}{w}{\prime}}}_{{\text{res}}}+\langle {\overline{{v }{\prime}{w}{\prime}}}_{{\text{sgs}}}\rangle )}^{2}\right)}^{1/4},$$
(12)

where \(u\) and \(v\) are the horizontal wind speeds in x- and y-direction, and the index \({\text{res}}\) indicates the covariance resolved by the grid, and:

$${w}_{*}={\left(\frac{g}{\overline{\theta }}\langle {z}_{i}\rangle ({\overline{{w }{\prime}{\theta }{\prime}}}_{{\text{res}}}+\langle {\overline{{w }{\prime}{\theta }{\prime}}}_{{\text{sgs}}}\rangle \right)}^{1/3},$$
(13)

where \(g\) is the gravitational acceleration (9,81 m s−2) and \({z}_{i}\) was determined as the height at which the total sensible heat flux crossed the zero value prior to reaching the cap** inversion.

The thermal heterogeneity parameter was introduced by Margairaz et al. (2020b) as:

$$\mathcal{H}=\frac{g \overline{{L }_{h}}}{{\langle \overline{{U }_{g}}\rangle }^{2}}\frac{\Delta \overline{T} }{\langle \overline{{T }_{{\text{s}}}}\rangle },$$
(14)

where \(\langle \overline{{T }_{{\text{s}}}}\rangle \) is the averaged surface temperature, \(\Delta \overline{T }\) is the amplitude of the surface temperature heterogeneities, calculated following:

$$\Delta \overline{T }=\langle \left|\overline{{T }_{{\text{s}}}}-\langle \overline{{T }_{{\text{s}}}}\rangle \right|\rangle ,$$
(15)

and \(\langle \overline{{U }_{g}}\rangle \) is the prescribed geostrophic wind speed. The heterogeneity length scale \(\overline{{L }_{h}}\) was calculated following an approach of Panin and Bernhofer (2008). We calculated the spatial spectra of \(\overline{{T }_{{\text{s}}}}\) by performing a Fourier transformation along ten transects in the x- and y-direction of the domain, respectively. Before performing the Fourier transformation, a bell taper was applied to smooth the edges and thereby reduce leakage (Stull 1988). The length scale that contributed the most to the variability in \(\overline{{T }_{{\text{s}}}}\) along each transect was identified as the location of the maximum of the spectrum. By averaging over the predominant length scales derived from each transect along both directions, we derived the predominant length scale \(\overline{{L }_{h}}\) of the entire domain.

The vertical differences \(\langle \Delta \overline{\theta }\rangle \) and \(\langle \Delta \overline{q }\rangle \) were calculated following:

$$\langle \Delta \overline{\theta }\rangle = \langle \overline{\theta }\rangle \left(0 m\right)-\langle \overline{\theta }\rangle \left({0.5 z}_{i}\right),$$
(16)

and:

$$\langle \Delta \overline{q }\rangle = \langle \overline{q }\rangle \left(0 m\right)-\langle \overline{q }\rangle \left({0.5 z}_{i}\right).$$
(17)

2.2.2 Model Fitting

We used the Random Forest machine-learning method (Breiman 2001) to predict \({H}_{d}\) and \({\lambda E}_{d}\) from 148 idealized LESs that were set up to represent a wide range of boundary conditions with respect to thermal surface heterogeneity and atmospheric stability (Sect. 2.1). The models were trained separately for \({H}_{d}\) and \({\lambda E}_{d}\) using \(\langle {u}_{*}/{w}_{*}\rangle \), \(z/\langle {z}_{i}\rangle \), \(\mathcal{H}\), and \(\langle \Delta \overline{\theta }\rangle \) (only for \({H}_{d}\)) or \(\langle \Delta \overline{q }\rangle \) (only for \({\lambda E}_{d}\)) as predictor variables. The entire training data is attached as a supplementary file. The modelling was performed in Python 3.7 using the Random Forest implementation in scikit-learn (Pedregosa et al. 2015; Hirschi et al. 2017; Mauder et al. 2007a, 2020; Mauder et al. 2013a, b; Twine et al. 2000). However, the relation between Bowen ratios of storage change and surface, turbulent, and dispersive heat fluxes is different in the more realistic CHEESEHEAD19 LES which is discussed in the next section. It furthermore remains questionable whether this assumption still holds if locally measured heat fluxes differ considerably from the average heat fluxes of the surroundings, which is typically the case in heterogeneous areas.

Due to the rather coarse resolution in the idealized LESs that were used to train the model, one has to assume that the strength of the SCs is underestimated, especially near the lower boundary of the domain. It is therefore likely that the model underestimates the transport of energy by SCs. The magnitude of this systematic error, however, is not known and requires further investigation in follow-up studies.

4.2 Application to the More Realistic CHEESEHEAD19 LES

Figure 4 shows that the model could be applied to almost all 30-min intervals with unstable conditions and for a wide range of measurement heights. A few observations near the canopy top had to be discarded because of the low values of \(z/\langle \overline{{z }_{i}}\rangle \). However, this affected mostly measurement heights of less than 16.5 m, which corresponds to approximately 10 m above the canopy top. To avoid individual roughness elements affecting the measured turbulent fluxes, EC measurements should not be carried out within the roughness sublayer (Katul et al. 1999) and are often located more than 10 m above the canopy top at forest locations.

In comparison with the dispersive fluxes calculated from the CHEESEHEAD19 LES output, the model slightly overestimates \({\lambda E}_{d}\) and clearly overestimates \({H}_{d}\) (Fig. 5), resulting in a general overestimation of dispersive fluxes (Fig. 6) and a slightly higher Bowen ratio (Table 4). The fact that \({H}_{d,{\text{mod}}}\) and \({\lambda E}_{d,{\text{mod}}}\) are slightly larger than the actual fluxes could be due to non-cyclic boundary conditions in the CHEESEHEAD19 LES. This arrangement raises the possibility that a small share of energy is transported by horizontal advection and is therefore not captured in the dispersive fluxes. Another reason could be the different setup of the simulations. In the idealized simulations, the fluxes were prescribed at the surface, whereas in the realistic simulations, the surface fluxes were provided by a combination of LSM and PCM. Wanner et al. (2022a) showed that the use of LSM and PCM increases the dispersive heat fluxes directly above the canopy but decreases the dispersive fluxes further up in the ABL, compared to dispersive fluxes produced in simulations with prescribed surface fluxes, owing to feedbacks between atmosphere and surface.

In contrast to the results from the idealized LESs, \(\beta (\langle {F}_{d,{\text{LES}}}\rangle )\) does not equal \(\beta (\langle {F}_{t}\rangle )\) in the CHEESEHEAD19 LES but is considerably lower, indicating that SCs have a larger relative share in the transport of latent heat than in the transport of sensible heat. It follows that the Bowen ratio approach to fill the SEB gap is not so well suited under more realistic conditions. Our dispersive flux model is capable of partially capturing the difference between \(\beta (\langle {F}_{d,{\text{LES}}}\rangle )\) and \(\beta (\langle {F}_{t}\rangle )\), but due to the strong overestimation of the sensible dispersive heat flux by roughly 18%, the partitioning of dispersive fluxes in the realistic LES is not ideally captured by the model.

Another factor that could alter the partitioning is the influence of entrainment on the transport of energy by secondary circulations (Gao et al. 2017; Huang et al. 2008; Mauder et al. 2020) which is not considered in our model. A temperature or humidity gradient in the boundary layer is the basic prerequisite for any transport of sensible or latent energy through SC which is why we included \(\langle \overline{\Delta \theta }\rangle \) and \(\langle \overline{\Delta q }\rangle \) as predictors in our model. The properties of the air mixed into the boundary layer at the upper boundary should also affect the transport of sensible heat or water vapor. Since \(\langle \overline{\Delta \theta }\rangle \) and \(\langle \overline{\Delta q }\rangle \) only represent the lower half of the atmospheric boundary layer (see Eqs. 16–17), this influence is only partially considered in the model and could also contribute to the underestimation of the dispersive fluxes. Since the ABL is confined by an inversion at the top, the influence of entrainment on the \({H}_{d}\) may be quite small. However, the humidity above the boundary layer is highly variable, so the effect of entrainment could also be a reason for the high uncertainty in the prediction of \({\lambda E}_{d}\). The heat flux profiles (not shown) confirm that the flux regime in the CHEESEHEAD19 LES was driven rather by dry air entrainment at the ABL top than by the surface fluxes at some times. As already discussed in the previous section, including entrainment as a predictor variable might therefore improve the model performance. For instance, this could be considered by the ratio of entrainment flux and the surface flux for sensible and latent heat.

Furthermore, with non-cyclic boundary conditions it is possible that weather fronts pass through the domain which can carry warmer/cooler or moister/dryer air into the domain, affecting \(\overline{w }\), \(\overline{\theta }\), and \(\overline{q }\) used to calculate the dispersive fluxes. In the CHEESEHEAD19 LES, a front passed through the domain in the afternoon on the first day, causing the boundary layer to collapse at 14.5 h simulated time which corresponds to 1430 LT, which is shown by the increase of \(z/\langle \overline{{z }_{i}}\rangle \) (Fig. 4). Figure 13 (Appendix 2) shows that the surface fluxes started decreasing at 1300 LT, indicating that advection may already have played a minor role, affecting \({H}_{d,{\text{LES}}}\) and \({\lambda E}_{d,{\text{LES}}}\). The effect of weather fronts passing by is generally less relevant for EC measurements, as they would typically be discarded in the affected periods, due to the violation of the stationarity requirement (Mauder et al. 2006).

4.3 Application to the CHEESEHEAD19 Field Measurements

The application of the model to field measurements was very limited because the ranges of the predictor variables in the field measurements were significantly larger than covered by the idealized LESs used to train the model (Fig. 7). The ranges of \(\langle {u}_{*}/{w}_{*}\rangle \), \(\mathcal{H}\) and \(\langle \overline{\Delta q }\rangle \) encountered in the ERA5 reanalysis data is much larger than in the idealized LES. However, this seems to be caused by extreme cases and outliers since a considerable number of observations (45.06%, 99.77%, and 67.94%, respectively) were covered by the training data (Table 5). The range of \(z/\langle \overline{{z }_{i}}\rangle \) encountered in the field is only slightly larger than in the idealized LES. Nevertheless, only 37.90% were covered by the training data (Table 5). This is due to the low measurement heights in some of the field stations (12 m at NW2 and SE5 and 3 m at NW3 and SE5) whose data were entirely discarded. Such low measurement heights could not be covered by the idealized LES due to the coarse grid resolution. Most measurements had to be discarded due to insufficient coverage of \(\langle \overline{\Delta \theta }\rangle \), as only 27.69% of the observed values were covered by the idealized LES (Table 5).

To investigate whether the large ranges in predictor variables are realistic, we compared the predictor variables derived from the ERA5 reanalysis data to the predictor variables derived from the CHEESEHEAD19 LES for 22–23 August 2019 (Fig. 10). On average, they show a good agreement in \(\mathcal{H}\) and \(\langle \overline{{z }_{i}}\rangle \) where the values are distributed evenly around unity, although ERA5 does not capture the variability in \(\langle \overline{{z }_{i}}\rangle \) as well as the CHEESEHEAD19 LES. Compared to the CHEESEHEAD19 LES, ERA5 underestimates \(\langle {u}_{*}/{w}_{*}\rangle \) and overestimates \(\langle \overline{\Delta \theta }\rangle \) and\(\langle \overline{\Delta q }\rangle \). The comparison of hourly vertical profiles (Fig. 14, Appendix 3) shows that the overestimation of \(\langle \overline{\Delta \theta }\rangle \) results mainly from a stronger increase in theta near the surface, whereas the profiles agree very well within the mixed layer (Fig. 11).

Fig. 10
figure 10

Comparison of predictor variables for the dispersive flux model from the CHEESEHEAD19 LES (x-axis) and ERA5 reanalysis data (y-axis) for all 30-min intervals on 22–23 August used for the analysis. Note that the second panel shows \(\langle \overline{{z }_{i}}\rangle \) instead of \(z/\langle \overline{{z }_{i}}\rangle \) for better comparability

Full size image
Fig. 11
figure 11

Comparison of hourly averaged vertical profiles of potential temperature (\(\langle \overline{\theta }\rangle \)) and mixing ratio (\(\langle \overline{q }\rangle \)) between the CHEESEHEAD LES and ERA5 reanalysis data for 23 August, 2019

Full size image

The CHEESEHEAD19 LES was validated against field measurements and the domain-averaged vertical profiles showed good agreement with radiosonde measurements (Paleri et al. (2023)). Nevertheless, it must be noted, that the domain was mostly forested with an average canopy height of 22.08 m. The active surface is therefore not clearly defined and the lowest grid points receive less radiation. \(\langle \overline{\theta }\rangle \) may therefore be slightly underestimated near the surface.

Figure 8 shows that adding the modelled energy transport by secondary circulations considerably decreases the SEB gap from 23. 8% on average to 18.6% on average but the SEB is still not closed. As mentioned in Sect. 4.1, one reason for the remaining gap is that the model might underestimate the transport by SCs due to the coarse grid spacing. However, there are more reasons why considering the energy transport by SCs alone does not result in SEB closure.

Because the model predicts only the dispersive heat fluxes and not the entire SEB gap, the storage of energy in the underlying air volume and canopy must be measured at tall EC stations since it also contributes significantly to the SEB gap. The measurement of the energy storage in the air volume beneath an EC station can be realized by simple profile measurements of temperature and humidity. This is already standard in some EC stations (Heiskanen et al. 2022) and was also included in the CHEESEHEAD19 measurements. However, the biomass storage change is typically not measured at EC stations and adds uncertainty to the energy storage quantification. It was not included in the energy storage measurements at the CHEESEHEAD19 sites, though tree biomass heat storage was measured at several sites subsequent to the campaign, where it averaged 6.5% of the available energy (Butterworth et al. 2024). This agrees with other studies which have found the biomass storage change to be of the same magnitude as the storage change in the air (Lindroth et al. 2010) or even twice as high (Haverd et al. 2007) at forest locations. Figure 9 shows that the addition of modelled dispersive heat fluxes to measured heat fluxes and air storage change decreases the SEB gap from 22.82 ± 17.67% to 15.75 ± 19.40%. Including the biomass storage change would further decrease the SEB gap to roughly 9% on average.

As discussed in Sect. 4.1, a slight underestimation of the dispersive fluxes by the model is expected due to the coarse grid spacing in the idealized LESs providing the training data. However, there are some more minor factors that contribute to the SEB gap that were not considered in this analysis.

First, the model only considers the effect of atmospheric stability and thermal surface heterogeneity on the energy transport by secondary circulations, which were identified as the major factors. However, topography effects can also enhance the energy transport of energy by SCs (Finnigan et al. 2003) but are not included in the model. Even though the measurement area was chosen in part because of its weak topography, the area in which the CHEESEHEAD19 campaign took place is not entirely flat. This may have strengthened the SCs, especially near the numerous water bodies in the domain which would also explain why the dispersive fluxes in the field measurements are larger than in both the idealized and CHEESEHEAD19 LESs.

Second, the turbulent heat fluxes might be underestimated as CSAT3AW (Campbell Scientific) sonic anemometers were used. For this type of sonic anemometer, a correction to account for transducer shadowing was developed that increases the fluxes by up to 5% (Horst et al. 2015) which was not applied to the CHEESEHEAD19 field measurements. Also, minor contributions to the SEB gap are not considered, like energy consumption by photosynthesis, which is estimated to account for roughly 1% of \({R}_{net}\) (Finnigan 2008). Taking these factors into account, the remaining SEB gap would be only about 3%, which suggests that the underestimation of energy transport by SCs by our model is quite small.

4.4 Different Data Availability in Idealized LES, More Realistic LES, and Field Measurements

Fundamental differences in data availability in the idealized and more realistic LES and field measurements complicate the accurate application of the model. This includes the determination of \(\mathcal{H}\), the true dispersive fluxes and the vertical differences \(\langle \overline{\Delta \theta }\rangle \) and \(\langle \overline{\Delta q }\rangle \).

The calculation of \(\mathcal{H}\) involves \(\Delta \overline{T }\), \(\langle \overline{{T }_{{\text{s}}}}\rangle \), and \({L}_{h}\). All three parameters are calculated based on the two-dimensional surface temperature. In the idealized LES, the surface temperature is clearly defined as the temperature at the lower boundary of the domain. In the CHEESEHEAD19 simulations, where a plant canopy model has been used in the forested areas, the surface is not so clearly defined. The temperature at the lower boundary of the domain is not very meaningful because it is mostly shaded and the majority of radiative transfer happens at the canopy level. We therefore used the air temperature at the top of the vegetation instead of the actual surface temperature as \({T}_{{\text{s}}}\), but this is based on the assumption that the vegetation has the same temperature as the air and introduces additional uncertainty. In field measurements, \({T}_{{\text{s}}}\) is difficult to determine at sufficiently high spatial and especially temporal resolution. Therefore, we used the surface-temperature maps from Desai et al. (2021), which are a fusion data product using land-surface models, satellite and hyperspectral imagery to apply the model to the CHEESEHEAD19 field measurements.

The dispersive fluxes can be computed directly as the spatial covariance of \(w\) and \(\theta \) or \(w\) and \(q\) in the idealized LESs with cyclic boundary conditions and thus quasi-infinite horizontal extent. In the realistic LES, the dispersive fluxes can also be computed as spatial covariance in a defined area. This area must be sufficiently large to fully cover the horizontal extent of the secondary circulations, similarly to how the averaging period in classical EC measurements must theoretically be sufficiently long to cover all relevant time scales. Since the horizontal extent of secondary circulations can reach about 2–3 times \({z}_{i}\) (Paleri et al. 2022a; Stull 1988), we recommend using an area with an edge length of at least 4 times \({z}_{i}\) to calculate the dispersive fluxes. If the area is too small, SCs partially contribute to horizontal transport, i.e., advection, that is not captured by the spatial covariance of \(w\) and \(\theta \) or \(q\). If the area is very large or includes an area where land cover or topography changes significantly the resulting dispersive flux is not representative of the area in which the EC measurement takes place. For typical single-tower field measurements, the dispersive flux cannot be calculated directly, as the calculation of covariances would require high-resolution spatial measurements of \(w\), \(q\) and \(\theta \). Instead, the amount of energy transported by the SCs must be calculated indirectly as a residual from the available energy at the surface (\(R_{{{\text{net}}}} {-} G\)) minus the energy stored in the air volume between the surface and the measurement height (\({H}_{\Delta St}+{\lambda E}_{\Delta St}\)) and the turbulent transport of energy (\({H}_{t}+{\lambda E}_{t}\)). A drawback of this approach is the scale mismatch between the measurement of \({R}_{net}\) and \(G\), which is only representative for the location of the EC station, and \(H\) and \(\lambda E\), which reflect the energy flux of the entire footprint, which covers a much larger area depending on the measurement height and atmospheric stability (Kljun et al. 2015).

For the model training, the output from the idealized LESs was used where the vertical differences \(\langle \overline{\Delta \theta }\rangle \) and \(\langle \overline{\Delta q }\rangle \) were calculated as the difference between surface values and the 0.5 \(\langle \overline{{z }_{i}}\rangle \). Since surface values are not available in the ERA5 reanalysis data, we used the 2 m temperature and humidity measurements instead to predict the dispersive fluxes in the CHEESEHEAD19 measurements. Under unstable conditions, the temperature gradient near the surface is very steep. Thus, the 2 m temperature is expected to be lower than the air temperature directly above the surface, resulting in an underestimation of the vertical temperature difference between the surface and the middle of the boundary layer. Over vegetated surfaces, which typically serve as a source of moisture unless they experience water stress, using the 2 m humidity instead of a near-ground measurement also results in an underestimation of the vertical difference. However, as shown in Fig. 10, \(\langle \Delta \overline{\theta }\rangle \) derived from ERA5 reanalysis data is not underestimated but overestimated compared to the CHEESEHEAD19 LES.

In the more realistic CHEESEHEAD19 LES, a PCM is used in forested areas, which is why the active surface is not as clearly defined as in the idealized LES. This increases the uncertainty in the calculation of \({T}_{{\text{s}}}\), \(\langle \Delta \overline{\theta }\rangle \) and \(\langle \Delta \overline{q }\rangle \) and \(\langle {u}_{*}/{w}_{*}\rangle \), as well as the surface fluxes.

4.5 Feasibility of the Model Application to Eddy-Covariance Stations

Single point EC field measurements typically only represent a limited area, referred to as footprint. The size of this area depends on various factors such as measurement height, atmospheric stability, and horizontal wind speed (Kljun et al. 2015), but it is always only a subset of the area that influences the development of SCs. Since the horizontal extent of SCs can reach 2–3 times \({z}_{i}\), they can span multiple kilometres and the area that has an effect on their development is correspondingly large. Therefore, EC measurements are likely to differ from the average values representing the entire area and are not suited as predicting variables for a model based on domain-averaged values.

This is why we used the ERA5 reanalysis data to provide the majority of the predictor variables for the application of the model to the CHEESEHEAD19 EC measurements. One advantage of the ERA5 reanalysis data is that it is available in all locations where EC measurements are carried out and thus can be used to predict dispersive fluxes at any tower without installing additional measurement instruments. This also facilitates the prediction of dispersive fluxes for EC measurements carried out in the past for which no field measurements of predictor variables are available. Since the ERA5 reanalysis data is available for the past 80 decades it can be used to consistently improve the energy-balance closure in long existing time series. However, with a spatial resolution of 0.25°, one grid cell in the ERA5 data is representative of up to 28 × 28 km2, which might include landscapes that are not representative of the surroundings of an EC tower.

The only parameters not derived from the ERA5 reanalysis data were \(\Delta \overline{T }\), \(\langle \overline{{T }_{{\text{s}}}}\rangle \), and \({L}_{h}\) used to calculate the thermal heterogeneity parameter as their calculation requires spatially highly resolved surface-temperature maps. We used the land-surface temperature fusion maps provided by Desai et al. (2021) which are derived by combining the information gathered from different satellite observations with high temporal resolution (NLDAS-2 (** algorithm for operational GOES-R land surface temperature product. IEEE Trans Geosci Remote Sens 47:936–951. https://doi.org/10.1109/TGRS.2008.2006180 " href="/article/10.1007/s10546-024-00868-8#ref-CR122" id="ref-link-section-d27679394e19476">2009)) and high spatial resolution (ECOSTRESS (Hulley et al. 2022)). Desai et al. (2021) showed that these maps represent the spatial and temporal variation in surface temperature in the CHEESEHEAD19 domain. Their approach can be used to generate land-surface temperature fusion maps with a spatial resolution of 50 m and a temporal resolution of 1 h for any location where temporally frequent geostationary (such as GOES) and high-spatial resolution less frequent land surface-temperature data (such as ECOSTRESS) are available and therefore can as well be derived for many EC stations without carrying out additional measurements.

5 Conclusion

We have developed a model to predict 30-min dispersive heat fluxes, i.e., the energy transport by SCs. A Python script to apply the model, including the training dataset, is included in the supplements.

By applying it to the CHEESEHEAD19 LES, we have shown that it reproduces the partitioning into sensible and latent heat fluxes quite well. The application to CHEESEHEAD19 field measurements showed that the model currently covers only a limited range of atmospheric conditions and is only applicable to measurement heights where \({z}_{{\text{m}}}\)-\({z}_{{\text{d}}}\) > 10 m. Therefore, to predict dispersive fluxes under all typical conditions at eddy covariance stations, the training data set must be extended to cover a wider range of \(\langle \overline{\Delta \theta }\rangle \), \(\langle \overline{\Delta q }\rangle \), \(\mathcal{H}\), and \(z/\langle {z}_{i}\rangle \).

However, application to the measurements that were within the model limits showed that the modelled dispersive fluxes resulted in a significant reduction in the SEB gap, but a SEB gap of 18.6% still remained. A variety of possible reasons were discussed, including the low resolution of the LESs the training data was extracted from, the different availability of information on vertical differences of \(\langle \overline{\theta }\rangle \) and \(\langle \overline{q }\rangle \) in the LES and the ERA5 reanalysis data, and uncertainties associated with the use of modelled surface temperature and atmospheric variables as predictor variables. The dispersive flux model could therefore be further improved by increasing the resolution of the LESs that provide the training data, which will allow for an enhanced representation of SCs near the surface and the differences of \(\langle \overline{\theta }\rangle \) and \(\langle \overline{q }\rangle \) can be defined in accordance to data availability in field measurements.

We conclude that these factors likely cause a slight underestimation of the dispersive fluxes by the model. However, a large portion of the remaining gap could be filled by adding the ignored biomass storage change, which is estimated to account for 6.5% of the total available energy at 30-min intervals.

We have also shown that it is possible to apply this model of dispersive heat fluxes to field measurements without performing any additional field measurements. The application of the model is based on ERA5 reanalysis data and remote sensing products that are available for most EC stations around the world and can also be used to model the dispersive fluxes retroactively.