Regularized Estimation of Covariance Structure Through Quadratic Loss Function

Abstract

Estimation of high-dimensional covariance structure is an interesting topic in statistics. Motivated by the work of Lin et al. [9], in this paper, the quadratic loss function is proposed to measure the discrepancy between a real covariance matrix and its candidate covariance matrix, where the latter has a regular structure. A commonly encountered candidate structures including MA(1), compound symmetry, AR(1), and banded Toeplitz matrix are considered. Regularization is made by selecting the optimal structure from a potential class of candidate covariance structures through minimizing the discrepancy, i.e., the quadratic loss function, between the given matrix and the candidate covariance class. Analytical or numerical solutions to the optimization problems are obtained and simulation studies are also conducted, showing that the proposed approach provides a reliable method to regularize covariance structures. It is applied to analyze real data problems for illustration of the use of the proposed method.

Appendix

We need the following lemma to judge the determinant sign of a matrix.

Lemma 4.1

If $\mathbf {A}$ and $\mathbf {B}$ are positive semidefinite matrices of the same order, then

$$ 0\le {\text {Tr}}(\mathbf {AB})\le {\text {Tr}}(\mathbf {A})\cdot {\text {Tr}}(\mathbf {B}), $$

and

$$ {\text {Tr}}(\mathbf {AB})\le ({\text {Tr}}(\mathbf {A}^2))^\frac{1}{2}\cdot ({\text {Tr}}(\mathbf {B}^2))^\frac{1}{2}. $$

A1. The proof of $\mathrm{det}(\nabla ^2 L(c,\sigma ))>0$ for MA(1) case.

We first note the first order partial derivative for $L(c,\sigma )$ is

$$ \begin{array}{rcl} \nabla L(c,\sigma ) &{} := &{} \left( \begin{array}{c} \frac{\partial L}{\partial c} \\ \frac{\partial L}{\partial \sigma } \end{array}\right) \\ &{} \phantom {:}= &{} \left( \begin{array}{c} 2\sigma ^4x+2c\sigma ^4y-2\sigma ^2z \\ 4\sigma ^3{\text {Tr}}({\pmb {\Sigma }}^{-2})+8c\sigma ^3x+4\sigma ^3c^2y-4\sigma {\text {Tr}}({\pmb {\Sigma }}^{-1})-4c\sigma z \end{array}\right) . \end{array} $$

Then the Hessian matrix is

$$ \begin{array}{l} \nabla ^2\,L(c,\sigma ) = \\ \ \left( \begin{array}{cc} 2\sigma ^4y &{} \quad 8\sigma ^3x+8c\sigma ^3y-4\sigma z \\ 8\sigma ^3x+8c\sigma ^3y-4\sigma z &{} \quad 12\sigma ^2 {\text {Tr}}({\pmb {\Sigma }}^{-2})+24c\sigma ^2x+12\sigma ^2c^2y-4{\text {Tr}}({\pmb {\Sigma }}^{-1})-4cz \end{array} \right) \\ \ = \\ \ \left( \begin{array}{cc} 2\sigma ^4y &{} \quad 4\sigma ^3x+4c\sigma ^3y \\ 4\sigma ^3x+4c\sigma ^3y &{} \quad 8\sigma ^2 {\text {Tr}}({\pmb {\Sigma }}^{-2})+16c\sigma ^2x+8\sigma ^2c^2y \end{array}\right) , \end{array} $$

where $x={\text {Tr}}({\pmb {\Sigma }}^{-2}{} \mathbf{T} _1)$, $y={\text {Tr}}({\pmb {\Sigma }}^{-2}{} \mathbf{T} _1^2)$, $z={\text {Tr}}({\pmb {\Sigma }}^{-1}{} \mathbf{T} _1)$. Thus

$$ \begin{array}{rcl} \mathrm{det}(\nabla ^2\,L(c,\sigma )) &{} = &{} 16\sigma ^6(y{\text {Tr}}({\pmb {\Sigma }}^{-2})-x^2) \\ &{} = &{} 16\sigma ^6({\text {Tr}}({\pmb {\Sigma }}^{-2}{} \mathbf{T} _1^2)\cdot {\text {Tr}}({\pmb {\Sigma }}^{-2})-({\text {Tr}}({\pmb {\Sigma }}^{-2}{} \mathbf{T} _1))^2). \end{array} $$

According to Lemma 4.1, we have $\mathrm{det}(\nabla ^2 L(c,\sigma ))>0$.

A2. The proof of $\mathrm{det}(\nabla ^2 L(c,\sigma ))>0$ for CS case.

Since

$$ \begin{array}{l} \nabla ^2\,L(c,\sigma ) = \\ \ \left( \begin{array}{cc} 2\sigma ^4v &{} \quad 8\sigma ^3w+8c\sigma ^3v-4\sigma u \\ 8\sigma ^3w+8c\sigma ^3v-4\sigma u &{} \quad 12\sigma ^2 {\text {Tr}}({\pmb {\Sigma }}^{-2})+24c\sigma ^2w+12\sigma ^2c^2v-4{\text {Tr}}({\pmb {\Sigma }}^{-1})-4cu \end{array}\right) \\ \ = \left( \begin{array}{cc} 2\sigma ^4v &{} \quad 4\sigma ^3w+4c\sigma ^3v \\ 4\sigma ^3w+4c\sigma ^3v &{} \quad 8\sigma ^2 {\text {Tr}}({\pmb {\Sigma }}^{-2})+16c\sigma ^2w+8\sigma ^2c^2v \end{array}\right) , \end{array} $$

where $u={\text {Tr}}({\pmb {\Sigma }}^{-1}(\mathbf{J} _m-\mathbf{I} _m))$, $v={\text {Tr}}({\pmb {\Sigma }}^{-2}(\mathbf{J} _m-\mathbf{I} _m)^2)$, $w={\text {Tr}}({\pmb {\Sigma }}^{-2}(\mathbf{J} _m-\mathbf{I} _m))$. Then, following Lemma 4.1, we have

$$ \begin{array}{rcl} \mathrm{det}(\nabla ^2\,L(c,\sigma )) &{} = &{} 16\sigma ^6(v {\text {Tr}}({\pmb {\Sigma }}^{-2})-w^2)\\ &{} = &{} 16\sigma ^6\left[ {\text {Tr}}({\pmb {\Sigma }}^{-2}(\mathbf{J} _m-\mathbf{I} _m)^2)\cdot {\text {Tr}}({\pmb {\Sigma }}^{-2})-({\text {Tr}}({\pmb {\Sigma }}^{-2}(\mathbf{J} _m-\mathbf{I} _m)))^2\right] >0. \end{array} $$