On convergence of covariance matrix of empirical Bayes hyper-parameter estimator

Abstract

Regularized system identification has become the research frontier of system identification in the past decade. One related core subject is to study the convergence properties of various hyper-parameter estimators as the sample size goes to infinity. In this paper, we consider one commonly used hyper-parameter estimator, the empirical Bayes (EB). Its convergence in distribution has been studied, and the explicit expression of the covariance matrix of its limiting distribution has been given. However, what we are truly interested in are factors contained in the covariance matrix of the EB hyper-parameter estimator, and then, the convergence of its covariance matrix to that of its limiting distribution is required. In general, the convergence in distribution of a sequence of random variables does not necessarily guarantee the convergence of its covariance matrix. Thus, the derivation of such convergence is a necessary complement to our theoretical analysis about factors that influence the convergence properties of the EB hyper-parameter estimator. In this paper, we consider the regularized finite impulse response (FIR) model estimation with deterministic inputs, and show that the covariance matrix of the EB hyper-parameter estimator converges to that of its limiting distribution. Moreover, we run numerical simulations to demonstrate the efficacy of our theoretical results.

Appendix A

This section includes the proof of Proposition 1.

1.1 Proof of Proposition 1

To prove Proposition 1, we first derive some preliminary results as shown in Lemmas A.1–A.2.

Lemma A.1

There exists a compact set $\widetilde{\varOmega }\subset \varOmega$ containing $\eta _{{{\,\textrm{b}\,}}}^{*}$, such that for $k,l=1,2,\ldots ,p$

$$\begin{aligned}&\Vert P\Vert _{F},\ \text {and}\ \Vert \hat{S}^{-1}\Vert _{F}<\Vert P^{-1}\Vert _{F}\ \text {are}\ \text {bounded}, \end{aligned}$$

(A1a)

$$\begin{aligned}&\left\| \frac{\partial P}{\partial \eta _{l}}\right\| _{F}\ \text {and}\ \left\| \frac{\partial ^2 P}{\partial \eta _{k}\partial \eta _{l}}\right\| _{F}\ \text {are}\ \text {bounded}, \end{aligned}$$

(A1b)

$$\begin{aligned}&\left\| \frac{\partial \hat{S}^{-1}}{\partial \eta _{k}}\right\| _{F}\ \text {and}\ \left\| \frac{\partial P^{-1}}{\partial \eta _{k}} \right\| _{F}\ \text {are}\ \text {bounded}, \end{aligned}$$

(A1c)

$$\begin{aligned}&\left\| \frac{\partial ^2 \hat{S}^{-1}}{\partial \eta _{k}\partial \eta _{l}}\right\| _{F}\ \text {and}\ \left\| \frac{\partial ^2 P^{-1}}{\partial \eta _{k}\partial \eta _{l}}\right\| _{F}\ \text {are}\ \text {bounded}, \end{aligned}$$

(A1d)

and there exists $\widetilde{M}_{1,{{\,\textrm{b}\,}}}>0$, irrespective of $\eta$, such that

$$\begin{aligned}&\sup _{\eta \in \widetilde{\varOmega }}\left\| \frac{\partial \hat{S}^{-1}}{\partial \eta _{l}}-\frac{\partial P^{-1}}{\partial \eta _{l}}\right\| _{F}\!\!\le \! \frac{1}{N}\widetilde{M}_{1,{{\,\textrm{b}\,}}}\widehat{\sigma ^2}\Vert N(\varPhi ^\textrm{T}\varPhi )^{-1}\Vert _{F}, \end{aligned}$$

(A2a)

$$\begin{aligned}&\sup _{\eta \in \widetilde{\varOmega }}\left\| \frac{\partial ^2 \hat{S}^{-1}}{\partial \eta _{k}\partial \eta _{l}}-\frac{\partial ^2 P^{-1}}{\partial \eta _{k}\partial \eta _{l}}\right\| _{F}\!\!\le \! \frac{1}{N}\widetilde{M}_{1,{{\,\textrm{b}\,}}}\widehat{\sigma ^2}\Vert N(\varPhi ^\textrm{T}\varPhi )^{-1}\Vert _{F}, \end{aligned}$$

(A2b)

where $\hat{S}$ is defined in (23).

Proof

If Assumption 4 holds, we have (A1a)–(A1b). Moreover, note that

$$\begin{aligned} \frac{\partial P^{-1}}{\partial \eta _{k}}&= -P^{-1}\frac{\partial P}{\partial \eta _{k}}P^{-1} \end{aligned}$$

(A3a)

$$\begin{aligned} \frac{\partial \hat{S}^{-1}}{\partial \eta _{k}}&=-\hat{S}^{-1}\frac{\partial P}{\partial \eta _{k}}\hat{S}^{-1}, \end{aligned}$$

(A3b)

$$\begin{aligned} \frac{\partial ^2 P^{-1}}{\partial \eta _{k}\partial \eta _{l}}&=P^{-1}\frac{\partial P}{\partial \eta _{l}}P^{-1}\frac{\partial P}{\partial \eta _{k}}P^{-1} -P^{-1}\frac{\partial ^2 P}{\partial \eta _{k}\partial \eta _{l}}P^{-1}\nonumber \\&\quad +P^{-1}\frac{\partial P}{\partial \eta _{k}}P^{-1}\frac{\partial P}{\partial \eta _{l}}P^{-1} \end{aligned}$$

(A3c)

$$\begin{aligned} \frac{\partial ^2 \hat{S}^{-1}}{\partial \eta _{k}\partial \eta _{l}}&=\hat{S}^{-1}\frac{\partial P}{\partial \eta _{l}}\hat{S}^{-1}\frac{\partial P}{\partial \eta _{k}}\hat{S}^{-1} -\hat{S}^{-1}\frac{\partial ^2 P}{\partial \eta _{k}\partial \eta _{l}}\hat{S}^{-1}\nonumber \\&\quad +\hat{S}^{-1}\frac{\partial P}{\partial \eta _{k}}\hat{S}^{-1}\frac{\partial P}{\partial \eta _{l}}\hat{S}^{-1}. \end{aligned}$$

(A3d)

It can be seen that $\frac{\partial \hat{S}^{-1}}{\partial \eta _{k}}$, $\frac{\partial P^{-1}}{\partial \eta _{k}}$, $\frac{\partial ^2 \hat{S}^{-1}}{\partial \eta _{k}\partial \eta _{l}}$ and $\frac{\partial ^2 P^{-1}}{\partial \eta _{k}\partial \eta _{l}}$ are all made of $P^{-1}$, $\hat{S}^{-1}$, $\frac{\partial P}{\partial \eta _{k}}$, $\frac{\partial ^2 P}{\partial \eta _{k}\partial \eta _{l}}$. Since $\forall \eta \in \widetilde{\varOmega }$, (A1a) and (A1b) both hold, it follows that (A1c) and (A1d) also hold.

Moreover, since

$$\begin{aligned}&\frac{\partial \hat{S}^{-1}}{\partial \eta _{k}}-\frac{\partial P^{-1}}{\partial \eta _{k}}\\&\quad =(P^{-1}-\hat{S}^{-1})\frac{\partial P}{\partial \eta _{k}}P^{-1} +\hat{S}^{-1}\frac{\partial P}{\partial \eta _{k}}(P^{-1}-\hat{S}^{-1}),\\&\quad =\widehat{\sigma ^2}P^{-1}(\varPhi ^\textrm{T}\varPhi )^{-1}\hat{S}^{-1}\frac{\partial P}{\partial \eta _{k}}P^{-1}\\&\qquad +\widehat{\sigma ^2}\hat{S}^{-1}\frac{\partial P}{\partial \eta _{k}}P^{-1}(\varPhi ^\textrm{T}\varPhi )^{-1}\hat{S}^{-1}, \end{aligned}$$

we have

$$\begin{aligned}&\sup _{\eta \in \widetilde{\varOmega }}\left\| \frac{\partial \hat{S}^{-1}}{\partial \eta _{l}}-\frac{\partial P^{-1}}{\partial \eta _{l}}\right\| _{F}\nonumber \\&\quad \le \frac{1}{N}\widehat{\sigma ^2}\Vert N(\varPhi ^\textrm{T}\varPhi )^{-1}\Vert _{F}\sup _{\eta \in \widetilde{\varOmega }}\left\| \frac{\partial P}{\partial \eta _{k}}\right\| _{F}\nonumber \\&\qquad \cdot \bigg (\sup _{\eta \in \widetilde{\varOmega }}\Vert P^{-1}\Vert _{F}^2\sup _{\eta \in \widetilde{\varOmega }}\Vert \hat{S}^{-1}\Vert _{F}\nonumber \\&\qquad +\sup _{\eta \in \widetilde{\varOmega }}\Vert \hat{S}^{-1}\Vert _{F}^2\sup _{\eta \in \widetilde{\varOmega }}\Vert P^{-1}\Vert _{F}\bigg ). \end{aligned}$$

(A4)

Using (A1a)–(A1b) and [32, Lemma B.1], we can see that (A4) leads to (A2a). Similarly, we can also derive (A2b) from [32, Lemma B.1], (A1c)–(A1d).

Lemma A.2

Under Assumptions 1–2, there exists $\widetilde{M}_{2,{{\,\textrm{b}\,}}}>0$, such that

$$\begin{aligned}&{\mathbb {E}}\bigg (\bigg \Vert \frac{\varPhi ^\textrm{T}V}{N}\bigg \Vert _{2}^8\bigg )\le \frac{1}{N^4}\widetilde{M}_{2,{{\,\textrm{b}\,}}}, \end{aligned}$$

(A5)

$$\begin{aligned}&{\mathbb {E}}\big [\big (\widehat{\sigma ^2}\big )^8\big ]\le \widetilde{M}_{2,{{\,\textrm{b}\,}}}, \end{aligned}$$

(A6)

$$\begin{aligned}&{\mathbb {E}}\left( \Vert \hat{\theta }^{{{\,\textrm{LS}\,}}}\Vert _{2}^8 \right) \le \widetilde{M}_{2,{{\,\textrm{b}\,}}}. \end{aligned}$$

(A7)

Proof

We shall derive (A5)–(A7), respectively.

For (A5), we have

$$\begin{aligned}&{\mathbb {E}}\bigg (\bigg \Vert \frac{\varPhi ^{\textrm{T}}V}{N}\bigg \Vert _{2}^8\bigg )\nonumber \\&\quad =\frac{1}{N^8}{\mathbb {E}}\left\{ \textstyle \sum \limits _{i=1}^{n} \left[ \textstyle \sum \limits _{t=1}^{N}u(t)v(t+i)\right] ^2\right\} ^4\nonumber \\&\quad \le \frac{1}{N^8} (2^3)^{n-1}\textstyle \sum \limits _{i=1}^{n}\left\{ {\mathbb {E}}\left[ \textstyle \sum \limits _{t=1}^{N}u(t)v(t+i)\right] ^8 \right\} \nonumber \\&\qquad (\text {using}~\text {[29, Theorem 2.2]})\nonumber \\&\quad \le \frac{1}{N^4}\widetilde{M}_{2,{{\,\textrm{b}\,}}}, \end{aligned}$$

(A8)

where the last step is derived due to that $\{v(t)\}_{t=1}^{N}$ is independent and has bounded moments of order $8+\delta$ for $\delta >0$, and the boundedness of u(t) as mentioned in Assumptions 1–2. Note that the boundedness of higher order moments can lead to the boundedness of lower order ones.

For (A6), we have

$$\begin{aligned}&{\mathbb {E}}(\widehat{\sigma ^2})^8\\&\quad =\frac{1}{(N-n)^8}{\mathbb {E}}(\Vert Y-\varPhi \hat{\theta }^{{{\,\textrm{LS}\,}}}\Vert _{2}^2)^8\\&\quad =\frac{1}{(N-n)^8}{\mathbb {E}}\left[ V^\textrm{T}V-V^\textrm{T}\varPhi (\varPhi ^\textrm{T}\varPhi )^{-1}\varPhi ^\textrm{T}V \right] ^8\\&\quad \le \frac{1}{(N-n)^8}2^7\left[ {\mathbb {E}}(V^\textrm{T}V)^8+{\mathbb {E}}(V^\textrm{T}\varPhi (\varPhi ^\textrm{T} \varPhi )^{-1}\varPhi ^\textrm{T}V)^8 \right] \\&\quad =\frac{1}{(N-n)^8}2^7\Big \{{\mathbb {E}}\Big [\textstyle \sum \limits _{t=1}^{N}v^2(t) \Big ]^8 \\&\qquad +{\mathbb {E}}\Big [\textstyle \sum \limits _{t_{1}=1}^{N}\textstyle \sum \limits _{t_{2}=1}^{N}v(t_{1}) v(t_{2})h_{t_{1},t_{2}} \Big ]^8 \Big \}\\&\quad \le \widetilde{M}_{2,{{\,\textrm{b}\,}}}, \end{aligned}$$

where $h_{t_{1},t_{2}}$ is the $(t_{1},t_{2})$th element of $\varPhi (\varPhi ^\textrm{T}\varPhi )^{-1}\varPhi ^\textrm{T}$ and is bounded, and the last step is due to that $\{v(t)\}_{t=1}^{N}$ is independent and has bounded moments of order $16+\delta$ for $\delta >0$ as shown in Assumption 2.

For (A7), we have

$$\begin{aligned}&{\mathbb {E}}\left( \Vert \hat{\theta }^{{{\,\textrm{LS}\,}}}\Vert _{2}^8 \right) \\&\quad ={\mathbb {E}}\left( \Vert \theta _{0}+(\varPhi ^\textrm{T}\varPhi )^{-1}\varPhi ^\textrm{T}V\Vert _{2}^8 \right) \\&\quad ={\mathbb {E}}\left( \Vert \theta _{0}\Vert _{2}^{2}\!+\!V^\textrm{T}\varPhi (\varPhi ^\textrm{T}\varPhi ) ^{-1}\varPhi ^\textrm{T}V\!+\!2\theta _{0}^\textrm{T}(\varPhi ^\textrm{T}\varPhi )^{-1}\varPhi ^\textrm{T}V\right) ^4\\&\quad \le 2^{3}\Big [ \Vert \theta _{0}\Vert _{2}^{8}+{\mathbb {E}}(V^\textrm{T}\varPhi (\varPhi ^\textrm{T}\varPhi )^{-1}\varPhi ^\textrm{T}V)^4\\&\qquad +{\mathbb {E}}(2\theta _{0}^\textrm{T}(\varPhi ^\textrm{T}\varPhi )^{-1}\varPhi ^\textrm{T}V)^4 \Big ]\\&\quad \le \widetilde{M}_{2,{{\,\textrm{b}\,}}}, \end{aligned}$$

where for the last second step, we apply [29, Theorem 2.2], and for the last step, we utilize Assumptions 1–2.

Now, using Lemmas A.1–A.2, we shall derive (24), (25) and (26), respectively.

Proof of (24)

We first rewrite the difference between $\overline{\mathscr {F}_{{{\,\textrm{EB}\,}}}}$ in (22) and $W_{{{\,\textrm{b}\,}}}$ in (16b) as

$$\begin{aligned}&\overline{\mathscr {F}_{{{\,\textrm{EB}\,}}}}-W_{{{\,\textrm{b}\,}}}\nonumber \\&\quad =(\hat{\theta }^{{{\,\textrm{LS}\,}}}-\theta _{0})^\textrm{T}\hat{S}^{-1}\hat{\theta }^{{{\,\textrm{LS}\,}}}+\theta _{0}^\textrm{T}(\hat{S}^{-1}-P^{-1})\hat{\theta }^{{{\,\textrm{LS}\,}}}\nonumber \\&\qquad +\theta _{0}^\textrm{T}P^{-1}(\hat{\theta }^{{{\,\textrm{LS}\,}}}-\theta _{0})+[\log \det (\hat{S})-\log \det (P)]\nonumber \\&\quad =\Big (\frac{\varPhi ^\textrm{T}V}{N} \Big )^\textrm{T}N(\varPhi ^\textrm{T}\varPhi )^{-1}\hat{S}^{-1}\hat{\theta }^{{{\,\textrm{LS}\,}}}\nonumber \\&\qquad -\widehat{\sigma ^2}\theta _{0}^\textrm{T}P^{-1}(\varPhi ^\textrm{T}\varPhi )^{-1}\hat{S}^{-1}\hat{\theta }^{{{\,\textrm{LS}\,}}}\nonumber \\&\qquad +\theta _{0}^\textrm{T}P^{-1}N(\varPhi ^\textrm{T}\varPhi )^{-1}\frac{\varPhi ^\textrm{T}V}{N}\!+\!\log \det (P^{-1/2}\hat{S}P^{-1/2}). \end{aligned}$$

(A9)

It follows that:

$$\begin{aligned}&\sup _{\eta \in \widetilde{\varOmega }}\vert \overline{\mathscr {F}_{{{\,\textrm{EB}\,}}}}-W_{{{\,\textrm{b}\,}}}\vert \nonumber \\&\quad \le \sup _{\eta \in \widetilde{\varOmega }}\bigg \{\left| \left( \frac{\varPhi ^\textrm{T}V}{N} \right) ^\textrm{T}N(\varPhi ^\textrm{T}\varPhi )^{-1}\hat{S}^{-1}\hat{\theta }^{{{\,\textrm{LS}\,}}}\right| \nonumber \\&\qquad +\left| \widehat{\sigma ^2}\theta _{0}^\textrm{T}P^{-1}(\varPhi ^\textrm{T}\varPhi )^{-1}\hat{S}^{-1}\hat{\theta }^{{{\,\textrm{LS}\,}}}\right| \nonumber \\&\qquad +\bigg \vert \theta _{0}^\textrm{T}P^{-1}N(\varPhi ^\textrm{T}\varPhi )^{-1}\frac{\varPhi ^\textrm{T}V}{N} \bigg \vert \nonumber \\&\qquad + \max \Big \{\left| {{\,\textrm{Tr}\,}}[\widehat{\sigma ^{2}}(\varPhi ^\textrm{T}\varPhi )^{-1}\hat{S}^{-1}]\right| ,\nonumber \\&\qquad \left| {{\,\textrm{Tr}\,}}[\widehat{\sigma ^2}(\varPhi ^\textrm{T}\varPhi )^{-1}P^{-1}]\right| \Big \}\bigg \}, \end{aligned}$$

(A10)

where we apply

$$\begin{aligned}&{{\,\textrm{Tr}\,}}(I_{n}-P^{1/2}\hat{S}^{-1}P^{1/2})\\&\quad \le \log \det (P^{-1/2}\hat{S}P^{-1/2})\!\le \!{{\,\textrm{Tr}\,}}(P^{-1/2}\hat{S}P^{-1/2}\!-\!I_{n}) \end{aligned}$$

using [32, Lemma B.16] with $A=P^{-1/2}\hat{S}P^{-1/2}$. Then, according to [32, Lemma B.1] and (A1a), there exists $\widetilde{M}_{3,{{\,\textrm{b}\,}}}>0$, such that (A10) can be further derived as

$$\begin{aligned}&\sup _{\eta \in \widetilde{\varOmega }}\vert \overline{\mathscr {F}_{{{\,\textrm{EB}\,}}}}-W_{{{\,\textrm{b}\,}}}\vert \nonumber \\&\quad \le \widetilde{M}_{3,{{\,\textrm{b}\,}}}\left\| \frac{\varPhi ^\textrm{T}V}{N}\right\| _{2}\Vert N(\varPhi ^\textrm{T}\varPhi )^{-1}\Vert _{F}\Vert \hat{\theta }^{{{\,\textrm{LS}\,}}}\Vert _{2}\nonumber \\&\qquad +\frac{1}{N}\widetilde{M}_{3,{{\,\textrm{b}\,}}}\Vert \theta _{0}\Vert _{2}\widehat{\sigma ^2}\Vert N(\varPhi ^\textrm{T}\varPhi )^{-1}\Vert _{F}\Vert \hat{\theta }^{{{\,\textrm{LS}\,}}}\Vert _{2}\nonumber \\&\qquad +\frac{1}{N}\widetilde{M}_{3,{{\,\textrm{b}\,}}}\widehat{\sigma ^2}\Vert N(\varPhi ^\textrm{T}\varPhi )^{-1}\Vert _{F}. \end{aligned}$$

(A11)

According to [32, Lemmas B.5-B.6], we have

$$\begin{aligned}&{\mathbb {E}}\left( \sup _{\eta \in \widetilde{\varOmega }}\vert \overline{\mathscr {F}_{{{\,\textrm{EB}\,}}}}-W_{{{\,\textrm{b}\,}}}\vert \right) ^4\nonumber \\&\quad \le 2^6\widetilde{M}_{3,{{\,\textrm{b}\,}}}^4\Vert N(\varPhi ^\textrm{T}\varPhi )^{-1}\Vert _{F}^4\bigg \{\Big [{\mathbb {E}}\Big (\Big \Vert \frac{\varPhi ^\textrm{T}V}{N}\Big \Vert _{2}^8\Big ){\mathbb {E}}\Big (\Vert \hat{\theta }^{{{\,\textrm{LS}\,}}}\Vert _{2}^{8}\Big )\Big ]^{1/2}\nonumber \\&\qquad +\frac{1}{N^4}\Vert \theta _{0}\Vert _{2}^{4}\Vert N(\varPhi ^\textrm{T}\varPhi )^{-1}\Vert _{F}^4\left[ {\mathbb {E}}\left( \widehat{\sigma ^2}\right) ^8{\mathbb {E}}\left( \Vert \hat{\theta }^{{{\,\textrm{LS}\,}}}\Vert _{2}^{8}\right) \right] ^{1/2}\nonumber \\&\qquad +\frac{1}{N^4}\Vert N(\varPhi ^\textrm{T}\varPhi )^{-1}\Vert _{F}^4{\mathbb {E}}\big [\big (\widehat{\sigma ^2}\big )^4 \big ]\bigg \}. \end{aligned}$$

(A12)

Thus, using Lemma A.2 and (A5), we can show that $\exists \ \widetilde{M}_{4,{{\,\textrm{b}\,}}}>0$, such that

$$\begin{aligned} {\mathbb {E}}\left( \sup _{\eta \in \widetilde{\varOmega }}\vert \overline{\mathscr {F}_{{{\,\textrm{EB}\,}}}}-W_{{{\,\textrm{b}\,}}}\vert \right) ^4\le \frac{1}{N^2}\widetilde{M}_{4,{{\,\textrm{b}\,}}}. \end{aligned}$$

(A13)

$\square$

Proof of (25)

According to the definition of Frobenius norm, it can be known that

$$\begin{aligned}&{\mathbb {E}}\left( \sup _{\eta \in \widetilde{\varOmega }} \left\| \frac{\partial ^2 \overline{\mathscr {F}_{{{\,\textrm{EB}\,}}}}}{\partial \eta \partial \eta ^\textrm{T}}-\frac{\partial ^2 W_{{{\,\textrm{b}\,}}}}{\partial \eta \partial \eta ^\textrm{T}}\right\| _{F}\right) ^4\nonumber \\&\quad \le {\mathbb {E}}\left( \sup _{\eta \in \widetilde{\varOmega }} \textstyle \sum \limits _{k=1}^{p}\textstyle \sum \limits _{l=1}^{p} \left| \dfrac{\partial ^2 \overline{\mathscr {F}_{{{\,\textrm{EB}\,}}}}}{\partial \eta _{k}\partial \eta _{l}}-\dfrac{\partial ^2 W_{{{\,\textrm{b}\,}}}}{\partial \eta _{k}\partial \eta _{l}}\right| ^2 \right) ^2\nonumber \\&\quad \le {\mathbb {E}}\left( \textstyle \sum \limits _{k=1}^{p}\textstyle \sum \limits _{l=1}^{p} \sup _{\eta \in \widetilde{\varOmega }} \left| \dfrac{\partial ^2 \overline{\mathscr {F}_{{{\,\textrm{EB}\,}}}}}{\partial \eta _{k}\partial \eta _{l}}-\dfrac{\partial ^2 W_{{{\,\textrm{b}\,}}}}{\partial \eta _{k}\partial \eta _{l}}\right| ^2 \right) ^2\nonumber \\&\quad \le 2^{p^2-1}\textstyle \sum \limits _{k=1}^{p}\textstyle \sum \limits _{l=1}^{p}{\mathbb {E}}\left( \sup _{\eta \in \widetilde{\varOmega }} \left| \dfrac{\partial ^2 \overline{\mathscr {F}_{{{\,\textrm{EB}\,}}}}}{\partial \eta _{k}\partial \eta _{l}}-\dfrac{\partial ^2 W_{{{\,\textrm{b}\,}}}}{\partial \eta _{k}\partial \eta _{l}}\right| \right) ^4, \end{aligned}$$

(A14)

where the last step is derived using [29, Theorem 2.2].

Recall [20, (A.6)–(A.7)] and matrix norm inequalities in [32, Lemma B.1]. It follows that:

$$\begin{aligned}&\sup _{\eta \in \widetilde{\varOmega }} \left| \frac{\partial ^2 \overline{\mathscr {F}_{{{\,\textrm{EB}\,}}}}}{\partial \eta _{k}\partial \eta _{l}}-\frac{\partial ^2 W_{{{\,\textrm{b}\,}}}}{\partial \eta _{k}\partial \eta _{l}}\right| \nonumber \\&\quad \le \sup _{\eta \in \widetilde{\varOmega }}\Bigg \{\left\| \frac{\varPhi ^\textrm{T}V}{N}\right\| _{2}\Vert N(\varPhi ^\textrm{T}\varPhi )^{-1}\Vert _{F} \left\| \frac{\partial ^2 \hat{S}^{-1}}{\partial \eta _{k}\partial \eta _{l}}\right\| _{F}\Vert \hat{\theta }^{{{\,\textrm{LS}\,}}}\Vert _{2}\nonumber \\&\qquad +\Vert \theta _{0}\Vert _{2}\left\| \frac{\partial ^2 \hat{S}^{-1}}{\partial \eta _{k}\partial \eta _{l}}-\frac{\partial ^2 P^{-1}}{\partial \eta _{k}\partial \eta _{l}}\right\| _{F}\Vert \hat{\theta }^{{{\,\textrm{LS}\,}}}\Vert _{2}\nonumber \\&\qquad +\Vert \theta _{0}\Vert _{2}\left\| \frac{\partial ^2 P^{-1}}{\partial \eta _{k}\partial \eta _{l}}\right\| _{F}\Vert N(\varPhi ^\textrm{T}\varPhi )^{-1}\Vert _{F}\left\| \frac{\varPhi ^\textrm{T}V}{N}\right\| _{2}\nonumber \\&\qquad +\sqrt{n}\left\| \frac{\partial \hat{S}^{-1}}{\partial \eta _{l}}-\frac{\partial P^{-1}}{\partial \eta _{l}}\right\| _{F} \left\| \frac{\partial P}{\partial \eta _{k}}\right\| _{F}\nonumber \\&\qquad +\frac{1}{N}\sqrt{n}\widehat{\sigma ^2}\Vert \hat{S}^{-1}\Vert _{F}\Vert N(\varPhi ^\textrm{T}\varPhi )^{-1}\Vert _{F}\nonumber \\&\qquad \cdot \Vert P^{-1}\Vert _{F}\left\| \frac{\partial ^2 P}{\partial \eta _{k}\partial \eta _{l}}\right\| _{F}\Bigg \}. \end{aligned}$$

(A15)

Then, according to (A1a), (A1b), (A1d), (A2a), and (A2b), there exists $\widetilde{M}_{5,{{\,\textrm{b}\,}}}>0$, such that we can rewrite (A15) as

$$\begin{aligned}&\sup _{\eta \in \widetilde{\varOmega }} \left| \frac{\partial ^2 \overline{\mathscr {F}_{{{\,\textrm{EB}\,}}}}}{\partial \eta _{k}\partial \eta _{l}}-\frac{\partial ^2 W_{{{\,\textrm{b}\,}}}}{\partial \eta _{k}\partial \eta _{l}}\right| \nonumber \\&\quad \le \widetilde{M}_{5,{{\,\textrm{b}\,}}}\left\| \frac{\varPhi ^\textrm{T}V}{N}\right\| _{2}\Vert N(\varPhi ^\textrm{T}\varPhi )^{-1}\Vert _{F}\Vert \hat{\theta }^{{{\,\textrm{LS}\,}}}\Vert _{2}\nonumber \\&\qquad +\widetilde{M}_{5,{{\,\textrm{b}\,}}}\left\| \frac{\varPhi ^\textrm{T}V}{N}\right\| _{2}\Vert N(\varPhi ^\textrm{T}\varPhi )^{-1}\Vert _{F}\nonumber \\&\qquad +\widetilde{M}_{5,{{\,\textrm{b}\,}}}\frac{1}{N}\widehat{\sigma ^2}\Vert N(\varPhi ^\textrm{T}\varPhi )^{-1}\Vert _{F}\Vert \hat{\theta }^{{{\,\textrm{LS}\,}}}\Vert _{2}\nonumber \\&\qquad +\widetilde{M}_{5,{{\,\textrm{b}\,}}}\frac{1}{N}\widehat{\sigma ^2}\Vert N(\varPhi ^\textrm{T}\varPhi )^{-1}\Vert _{F}. \end{aligned}$$

(A16)

Moreover, using (A16) and [29, Theorems 2.2, 3.1], we can further show that

$$\begin{aligned}&{\mathbb {E}}\left( \sup _{\eta \in \widetilde{\varOmega }} \left| \frac{\partial ^2 \overline{\mathscr {F}_{{{\,\textrm{EB}\,}}}}}{\partial \eta _{k}\partial \eta _{l}}-\frac{\partial ^2 W_{{{\,\textrm{b}\,}}}}{\partial \eta _{k}\partial \eta _{l}}\right| \right) ^4\nonumber \\&\quad \le 2^9 \widetilde{M}_{5,{{\,\textrm{b}\,}}}^4\Vert N(\varPhi ^\textrm{T}\varPhi )^{-1}\Vert _{F}^4\Bigg \{ \Big [{\mathbb {E}}\Big (\Big \Vert \frac{\varPhi ^\textrm{T}V}{N}\Big \Vert _{2}^8\Big ) {\mathbb {E}}\Big (\Vert \hat{\theta }^{{{\,\textrm{LS}\,}}}\Vert _{2}^8\Big ) \Big ]^{1/2} \nonumber \\&\qquad + {\mathbb {E}}\Big (\Big \Vert \frac{\varPhi ^\textrm{T}V}{N}\Big \Vert _{2}^4\Big ) +\frac{1}{N^4}\left[ {\mathbb {E}}\left( (\widehat{\sigma ^2})^8\right) {\mathbb {E}}(\Vert \hat{\theta }^{{{\,\textrm{LS}\,}}}\Vert _{2}^8) \right] ^{1/2}\nonumber \\&\qquad +\frac{1}{N^4}{\mathbb {E}}\big (\widehat{\sigma ^2}\big )^4\Bigg \}. \end{aligned}$$

(A17)

Together (A14) with Lemma A.2 and (A17), we have $\exists \ \widetilde{M}_{6,{{\,\textrm{b}\,}}}>0$, such that

$$\begin{aligned} {\mathbb {E}}\left( \sup _{\eta \in \widetilde{\varOmega }} \left\| \frac{\partial ^2 \overline{\mathscr {F}_{{{\,\textrm{EB}\,}}}}}{\partial \eta \partial \eta ^\textrm{T}}-\frac{\partial ^2 W_{{{\,\textrm{b}\,}}}}{\partial \eta \partial \eta ^\textrm{T}}\right\| _{F}\right) ^4 \le \frac{1}{N^2} \widetilde{M}_{6,{{\,\textrm{b}\,}}}. \end{aligned}$$

(A18)

$\square$

Proof of (26)

Let us consider the boundedness of ${\mathbb {E}}\left\| \left. {\partial \overline{\mathscr {F}_{{{\,\textrm{EB}\,}}}}}/{\partial \eta }\right| _{\eta ^{*}_{{{\,\textrm{b}\,}}}} \right\| _{2}^4$ first. Under Assumption 6, $\eta _{{{\,\textrm{b}\,}}}^{*}$ should satisfy the first-order optimality condition, i.e., ${\partial W_{{{\,\textrm{b}\,}}}}/{\partial \eta }\vert _{\eta ^{*}_{{{\,\textrm{b}\,}}}}=0$. It yields that

$$\begin{aligned}&{\mathbb {E}}\left\| \left. \frac{\partial \overline{\mathscr {F}_{{{\,\textrm{EB}\,}}}}}{\partial \eta }\right| _{\eta ^{*}_{{{\,\textrm{b}\,}}}} \right\| _{2}^4\nonumber \\&\quad ={\mathbb {E}}\left\| \left. \left( \frac{\partial \overline{\mathscr {F}_{{{\,\textrm{EB}\,}}}}}{\partial \eta } -\frac{\partial W_{{{\,\textrm{b}\,}}}}{\partial \eta } \right) \right| _{\eta ^{*}_{{{\,\textrm{b}\,}}}} \right\| _{2}^4\nonumber \\&\quad \le {\mathbb {E}}\left( \sup _{\eta \in \widetilde{\varOmega }}\left\| \frac{\partial \overline{\mathscr {F}_{{{\,\textrm{EB}\,}}}}}{\partial \eta } -\frac{\partial W_{{{\,\textrm{b}\,}}}}{\partial \eta } \right\| _{2}\right) ^4\nonumber \\&\quad \le {\mathbb {E}}\left[ \textstyle \sum \limits _{k=1}^{p}\sup _{\eta \in \widetilde{\varOmega }}\left( \frac{\partial \overline{\mathscr {F}_{{{\,\textrm{EB}\,}}}}}{\partial \eta _{k}} -\frac{\partial W_{{{\,\textrm{b}\,}}}}{\partial \eta _{k}} \right) ^2 \right] ^2\nonumber \\&\quad \le 2^{p-1}\textstyle \sum \limits _{k=1}^{p}{\mathbb {E}}\left( \sup _{\eta \in \widetilde{\varOmega }}\left| \frac{\partial \overline{\mathscr {F}_{{{\,\textrm{EB}\,}}}}}{\partial \eta _{k}} -\frac{\partial W_{{{\,\textrm{b}\,}}}}{\partial \eta _{k}} \right| \right) ^4, \end{aligned}$$

(A19)

where the last step is derived from [29, Theorem 2.2].

For any $k=1,\ldots ,p$, using [20, (A.11)–(A.15)], we know that

$$\begin{aligned}&\sup _{\eta \in \widetilde{\varOmega }}\left| \frac{\partial \overline{\mathscr {F}_{{{\,\textrm{EB}\,}}}}}{\partial \eta _{k}} -\frac{\partial W_{{{\,\textrm{b}\,}}}}{\partial \eta _{k}} \right| \\&\quad \le \sup _{\eta \in \widetilde{\varOmega }} \Bigg \{ \Vert \theta _{0}\Vert _{2}\left\| \frac{\partial \hat{S}^{-1}}{\partial \eta _{k}}-\frac{\partial P^{-1}}{\partial \eta _{k}}\right\| _{F}\Vert \hat{\theta }^{{{\,\textrm{LS}\,}}}\Vert _{2}\\&\qquad +\left\| \frac{\varPhi ^\textrm{T}V}{N}\right\| _{2}\Vert N(\varPhi ^\textrm{T}\varPhi )^{-1}\Vert _{F}\left\| \frac{\partial \hat{S}^{-1}}{\partial \eta _{k}}\right\| _{F}\Vert \hat{\theta }^{{{\,\textrm{LS}\,}}}\Vert _{2}\\&\qquad +\left\| \frac{\varPhi ^\textrm{T}V}{N}\right\| _{2}\Vert N(\varPhi ^\textrm{T}\varPhi )^{-1}\Vert _{F}\left\| \frac{\partial P^{-1}}{\partial \eta _{k}}\right\| _{F}\Vert \theta _{0}\Vert _{2}\\&\qquad +\!\!\frac{1}{N}\sqrt{n}\widehat{\sigma ^2}\Vert \hat{S}^{-1}\Vert _{F}\Vert N(\varPhi ^\textrm{T}\varPhi )^{-1}\Vert _{F}\Vert P^{-1}\Vert _{F}\left\| \frac{\partial P}{\partial \eta _{k}}\right\| _{F}\!\!\Bigg \}. \end{aligned}$$

Applying (A1a)–(A1c), (A2a), and [32, Lemma B.1], we can further derive that there exists $\widetilde{M}_{7,{{\,\textrm{b}\,}}}>0$, such that

$$\begin{aligned}&\sup _{\eta \in \widetilde{\varOmega }}\left| \frac{\partial \overline{\mathscr {F}_{{{\,\textrm{EB}\,}}}}}{\partial \eta _{k}} -\frac{\partial W_{{{\,\textrm{b}\,}}}}{\partial \eta _{k}} \right| \\&\quad \le \widetilde{M}_{7,{{\,\textrm{b}\,}}} \frac{1}{N}\widehat{\sigma ^2}\Vert N(\varPhi ^\textrm{T}\varPhi )^{-1}\Vert _{F}\Vert \hat{\theta }^{{{\,\textrm{LS}\,}}}\Vert _{2}\\&\qquad +\widetilde{M}_{7,{{\,\textrm{b}\,}}}\left\| \frac{\varPhi ^\textrm{T}V}{N}\right\| _{2}\Vert N(\varPhi ^\textrm{T}\varPhi )^{-1}\Vert _{F} \Vert \hat{\theta }^{{{\,\textrm{LS}\,}}}\Vert _{2}\\&\qquad +\widetilde{M}_{7,{{\,\textrm{b}\,}}}\left\| \frac{\varPhi ^\textrm{T}V}{N}\right\| _{2}\Vert N(\varPhi ^\textrm{T}\varPhi )^{-1}\Vert _{F}\\&\qquad +\widetilde{M}_{7,{{\,\textrm{b}\,}}}\frac{1}{N}\widehat{\sigma ^2}\Vert N(\varPhi ^\textrm{T}\varPhi )^{-1}\Vert _{F}. \end{aligned}$$

Then, using [29, Theorems 2.2, 3.1], we have

$$\begin{aligned}&{\mathbb {E}}\left( \sup _{\eta \in \widetilde{\varOmega }}\left| \frac{\partial \overline{\mathscr {F}_{{{\,\textrm{EB}\,}}}}}{\partial \eta _{k}} -\frac{\partial W_{{{\,\textrm{b}\,}}}}{\partial \eta _{k}} \right| \right) ^4\nonumber \\&\quad \le 2^9\widetilde{M}_{7,{{\,\textrm{b}\,}}}^4\left\{ \frac{1}{N^4}\Vert N(\varPhi ^\textrm{T}\varPhi )^{-1}\Vert _{F}^4{\mathbb {E}}\left( \widehat{\sigma ^2} \Vert \hat{\theta }^{{{\,\textrm{LS}\,}}}\Vert _{2}\right) ^4 \right. \nonumber \\&\qquad +\Vert N(\varPhi ^\textrm{T}\varPhi )^{-1}\Vert _{F}^4\Big [ {\mathbb {E}}\Big (\Big \Vert \frac{\varPhi ^\textrm{T}V}{N}\Big \Vert _{2}\Big )^8 {\mathbb {E}}\Big ( \Vert \hat{\theta }^{{{\,\textrm{LS}\,}}}\Vert _{2}\Big )^8 \Big ]^{1/2}\nonumber \\&\qquad +\Vert N(\varPhi ^\textrm{T}\varPhi )^{-1}\Vert _{F}^4 {\mathbb {E}}\Big (\Big \Vert \frac{\varPhi ^\textrm{T}V}{N}\Big \Vert _{2}\Big )^4\nonumber \\&\qquad +\left. \frac{1}{N^4}\Vert N(\varPhi ^\textrm{T}\varPhi )^{-1}\Vert _{F}^4{\mathbb {E}}\big ( \widehat{\sigma ^2}\big )^4\right\} . \end{aligned}$$

(A20)

Analogously, our next step is to apply Lemma A.2 and (A20) to (A19), and then, there exists $\widetilde{M}_{8,{{\,\textrm{b}\,}}}>0$, such that

$$\begin{aligned} {\mathbb {E}}\bigg \Vert \frac{\partial \overline{\mathscr {F}_{{{\,\textrm{EB}\,}}}}}{\partial \eta }\Big \vert _{\eta =\eta _{{{\,\textrm{b}\,}}}^{*}} \bigg \Vert _{2}^4 \le \frac{1}{N^2}\widetilde{M}_{8,{{\,\textrm{b}\,}}}. \end{aligned}$$

(A21)

$\square$

Define

$$\begin{aligned} \check{M}_{{{\,\textrm{b}\,}}}=\max \big \{\widetilde{M}_{4,{{\,\textrm{b}\,}}},\widetilde{M}_{6,{{\,\textrm{b}\,}}}, \widetilde{M}_{8,{{\,\textrm{b}\,}}}\big \}, \end{aligned}$$

(A22)

and then, our proof is complete.