Variational inference for sparse spectrum Gaussian process regression

Abstract

We develop a fast variational approximation scheme for Gaussian process (GP) regression, where the spectrum of the covariance function is subjected to a sparse approximation. Our approach enables uncertainty in covariance function hyperparameters to be treated without using Monte Carlo methods and is robust to overfitting. Our article makes three contributions. First, we present a variational Bayes algorithm for fitting sparse spectrum GP regression models that uses nonconjugate variational message passing to derive fast and efficient updates. Second, we propose a novel adaptive neighbourhood technique for obtaining predictive inference that is effective in dealing with nonstationarity. Regression is performed locally at each point to be predicted and the neighbourhood is determined using a measure defined based on lengthscales estimated from an initial fit. Weighting dimensions according to lengthscales, this downweights variables of little relevance, leading to automatic variable selection and improved prediction. Third, we introduce a technique for accelerating convergence in nonconjugate variational message passing by adapting step sizes in the direction of the natural gradient of the lower bound. Our adaptive strategy can be easily implemented and empirical results indicate significant speedups.

Appendices

Appendix 1: Derivation of $E_q(Z)$ and $E_q(Z^TZ)$

Lemma 1

Suppose $\lambda \sim N(\mu ,{\varSigma })$ and $t_1$, $t_2$ are fixed vectors the same length as $\lambda$. Let $t_{12}^-=t_1-t_2$ and $t_{12}^+=t_1+t_2$, then

$$\begin{aligned}&E\left\{ \cos \left( t_1^T\lambda \right) \cos \left( t_2^T\lambda \right) \right\} =\tfrac{1}{2}\left[ \exp \left( -\tfrac{1}{2}{t_{12}^-}^T{\varSigma } t_{12}^-\right) \right. \\&\quad \left. \cdot \cos \left( {t_{12}^-}^T\mu \right) +\exp \left( -\tfrac{1}{2}{t_{12}^+}^T{\varSigma } t_{12}^+\right) \cos \left( {t_{12}^+}^T\mu \right) \right] \\&E\left\{ \sin \left( t_1^T\lambda \right) \sin \left( t_2^T \lambda \right) \right\} =\tfrac{1}{2}\left[ \exp \left( -\tfrac{1}{2}{t_{12}^-}^T{\varSigma } t_{12}^-\right) \right. \\&\quad \left. \cdot \cos \left( {t_{12}^-}^T\mu \right) -\exp \left( -\tfrac{1}{2}{t_{12}^+}^T{\varSigma } t_{12}^+\right) \cos \left( {t_{12}^+}^T\mu \right) \right] \\&E\left\{ \sin \left( t_1^T\lambda \right) \cos \left( t_2^T \lambda \right) \right\} =\tfrac{1}{2}\left[ \exp \left( -\tfrac{1}{2}{t_{12}^-}^T{\varSigma } t_{12}^-\right) \right. \\&\quad \left. \cdot \sin \left( {t_{12}^-}^T\mu \right) + \exp \left( -\tfrac{1}{2}{t_{12}^+}^T{\varSigma } t_{12}^+\right) \sin \left( {t_{12}^+}^T\mu \right) \right] \end{aligned}$$

By setting $t_2=0$ in the first and third expressions, we get

$$\begin{aligned}&E\left\{ \cos \left( t_1^T\lambda \right) \right\} = \exp \left( -\tfrac{1}{2}t_1^T{\varSigma } t_1\right) \cos \left( t_1^T\mu \right) \;\; \text {and} \\&E\left\{ \sin \left( t_1^T\lambda \right) \right\} =\exp \left( -\tfrac{1}{2}t_1^T{\varSigma } t_1\right) \sin \left( t_1^T\mu \right) . \end{aligned}$$

Proof

$E[\exp \{i\lambda ^T(t_1-t_2)\}]=\exp \{i \mu ^T (t_1-t_2)- \tfrac{1}{2}(t_1-t_2)^T {\varSigma } (t_1-t_2)\}$ implies

$$\begin{aligned} E\left[ \cos \left\{ \lambda ^T\left( t_1-t_2\right) \right\} \right]= & {} E\left\{ \cos \left( t_1^T\lambda \right) \cos \left( t_2^T\lambda \right) \right. \nonumber \\&\left. +\sin \left( t_1^T\lambda \right) \sin \left( t_2^T\lambda \right) \right\} \nonumber \\= & {} \exp \left\{ -\tfrac{1}{2}(t_1-t_2)^T {\varSigma } (t_1-t_2)\right\} \nonumber \\&\cdot \cos \left\{ \mu ^T (t_1-t_2)\right\} \end{aligned}$$

(17)

and

$$\begin{aligned} E\left[ \sin \left\{ \lambda ^T\left( t_1-t_2\right) \right\} \right]= & {} E\left\{ \sin \left( t_1^T\lambda \right) \cos \left( t_2^T\lambda \right) \right. \nonumber \\&\left. -\cos \left( t_1^T\lambda \right) \sin \left( t_2^T\lambda \right) \right\} \nonumber \\= & {} \exp \left\{ -\tfrac{1}{2}(t_1-t_2)^T {\varSigma } (t_1-t_2)\right\} \nonumber \\&\cdot \sin \left\{ \mu ^T (t_1-t_2)\right\} . \end{aligned}$$

(18)

Replacing $t_2$ by $-t_2$, we get

$$\begin{aligned} E\left[ \cos \left\{ \lambda ^T\left( t_1+t_2\right) \right\} \right]= & {} E\left\{ \cos \left( t_1^T\lambda \right) \cos \left( t_2^T\lambda \right) \right. \nonumber \\&\left. -\sin \left( t_1^T\lambda \right) \sin \left( t_2^T\lambda \right) \right\} \nonumber \\= & {} \exp \left\{ -\tfrac{1}{2}(t_1+t_2)^T {\varSigma } (t_1+t_2)\right\} \nonumber \\&\cdot \cos \left\{ \mu ^T (t_1+t_2)\right\} \end{aligned}$$

(19)

and

$$\begin{aligned} E[\sin \{\lambda ^T(t_1+t_2)\}]= & {} E\left\{ \sin \left( t_1^T\lambda \right) \cos \left( t_2^T\lambda \right) \right. \nonumber \\&\left. +\cos \left( t_1^T\lambda \right) \sin \left( t_2^T\lambda \right) \right\} \nonumber \\= & {} \exp \left\{ -\tfrac{1}{2}(t_1+t_2)^T {\varSigma } (t_1+t_2)\right\} \nonumber \\&\cdot \sin \left\{ \mu ^T (t_1+t_2)\right\} . \end{aligned}$$

(20)

(17) + (19) gives the first equation of the lemma, (17) – (19) gives the second and (18) + (20) gives the third. $\square$

Using Lemma 1, we have

$$\begin{aligned} E_q(Z)=[E_q(Z_1), \dots , E_q(Z_n)]^T, \end{aligned}$$

where

and $t_{ir}=s_r \odot x_i$ for $i=1,\dots ,n$, $r=1,\dots ,m$. We also have $E_q(Z^TZ)=\sum _{i=1}^n E_q(Z_iZ_i^T)$ where $E_q(Z_iZ_i^T)=\left[ {\begin{matrix} P_i &{} Q_i^T \\ Q_i &{} R_i \end{matrix}}\right]$, where $P_i$, $Q_i$, $R_i$ are all $m\times m$ matrices and

$$\begin{aligned} {P_i}_{rl}= & {} \frac{1}{2}\left\{ \exp \left( -\frac{1}{2}{t_{irl}^-}^T{\varSigma }_\lambda ^qt_{irl}^-\right) \cos \left( {t_{irl}^-}^T\right) \mu _\lambda ^q \right. \\&\left. + \exp \left( -\frac{1}{2}{t_{irl}^+}^T{\varSigma }_\lambda ^qt_{irl}^+\right) \cos \left( {t_{irl}^+}^T\right) \mu _\lambda ^q\right\} ,\\ {Q_i}_{rl}= & {} \frac{1}{2}\left\{ -\exp \left( -\frac{1}{2}{t_{irl}^-}^T{\varSigma }_\lambda ^qt_{irl}^-\right) \sin \left( {t_{irl}^-}^T\right) \mu _\lambda ^q \right. \\&\left. + \exp \left( -\frac{1}{2}{t_{irl}^+}^T{\varSigma }_\lambda ^qt_{irl}^+\right) \sin \left( {t_{irl}^+}^T\right) \mu _\lambda ^q\right\} ,\\ {R_i}_{rl}= & {} \frac{1}{2}\left\{ \exp \left( -\frac{1}{2}{t_{irl}^-}^T{\varSigma }_\lambda ^qt_{irl}^-\right) \cos \left( {t_{irl}^-}^T\right) \mu _\lambda ^q \right. \\&\left. - \exp \left( -\frac{1}{2}{t_{irl}^+}^T{\varSigma }_\lambda ^qt_{irl}^+\right) \cos \left( {t_{irl}^+}^T\right) \mu _\lambda ^q\right\} , \end{aligned}$$

$t_{irl}^-=t_{ir}-t_{il}$, $t_{irl}^+=t_{ir}+t_{il}$ for $r=1,\dots ,m$, $l=1,\dots ,m$.

Appendix 2: Derivation of lower bound

From (6), the lower bound is given by

$$\begin{aligned} \mathcal {L}=E_q\{\log p(y,\theta )\}-E_q\{\log q(\theta )\} \end{aligned}$$

where

$$\begin{aligned} E_q\{\log p(y,\theta )\}= & {} E_q\{\log p(y|\alpha ,\lambda ,\gamma )\} + E_q\{\log p(\alpha |\sigma )\} \\&+E_q\{\log p(\lambda )\}+E_q\{\log p(\sigma )\} \\&+E_q\{\log p(\gamma )\}, \end{aligned}$$

$$\begin{aligned} E_q\{\log q(\theta )\}= & {} E_q\{\log q(\alpha )\} +E_q\{\log q(\lambda )\} \\&+E_q\{\log q(\sigma )\}+E_q\{\log q(\gamma )\}. \end{aligned}$$

The terms in the lower bound can be evaluated as follows:

$$\begin{aligned}&E_q\{\log p(y|\alpha ,\beta ,\lambda ,\gamma )\}=-\frac{n}{2}\log (2\pi )-\frac{n}{2}E_q(\log \gamma ^2) \\&\quad -\frac{1}{2}\big [y^Ty-2y^TE_q(Z)\mu _\alpha ^q+\text {tr}\{(\mu _\alpha ^q{\mu _\alpha ^q}^T+{\varSigma }_\alpha ^q)E_q(Z^TZ)\}\big ] \\&\quad \cdot {\mathcal {H}(n,C_\gamma ^q,A_\gamma ^2)}/{\mathcal {H}(n-2,C_\gamma ^q,A_\gamma ^2)}\\&E_q\{\log p(\alpha |\sigma )\}=-m\log (2\pi ) -mE_q\{\log \sigma ^2\} \\&\quad +m\log m -\frac{m}{2}\frac{\mathcal {H}(2m,C_\sigma ^q,A_\sigma ^2)}{\mathcal {H}(2m-2,C_\sigma ^q,A_\sigma ^2)}\{{\mu _\alpha ^q}^T\mu _\alpha ^q+\text {tr}({\varSigma }_\alpha ^q)\}\\&E_q\{\log p(\lambda )\}=-\frac{d}{2}\log (2\pi ) -\quad \frac{1}{2}\log |{\varSigma }_\lambda ^0| \\&\quad -\frac{1}{2}(\mu _\lambda ^q-\mu _\lambda ^0)^T{{\varSigma }_\lambda ^0}^{-1}(\mu _\lambda ^q-\mu _\lambda ^0)-\frac{1}{2}\text {tr}({{\varSigma }_\lambda ^0}^{-1}{\varSigma }_\lambda ^q)\\&E_q\{\log p(\sigma )\}=\log (2A_\sigma )-\log \pi -E_q\{\log (A_\sigma ^2+\sigma ^2)\}\\&E_q\{\log p(\gamma )\}=\log (2A_\gamma )-\log \pi -E_q\{\log (A_\gamma ^2+\gamma ^2)\}\\&E_q\{\log q(\alpha )\}=-m\log (2\pi )-\frac{1}{2}\log |{\varSigma }_\alpha ^q|-m\\&E_q\{\log q(\lambda )\}=-\frac{d}{2}\log (2\pi ) -\frac{1}{2}\log |{\varSigma }_\lambda ^q|-\frac{d}{2}\\&E_q\{\log q(\sigma )\}=-C_\sigma ^q \frac{\mathcal {H}(2m,C_\sigma ^q,A_\sigma ^2)}{\mathcal {H}(2m-2,C_\sigma ^q,A_\sigma ^2)} - 2mE_q\{\log \sigma \}\\&\quad -\log \mathcal {H}(2m-2,C_\sigma ^q,A_\sigma ^2) -E_q\{\log (A_\sigma ^2+\sigma ^2)\}\\&E_q\{\log q(\gamma )\}=-C_\gamma ^q {\mathcal {H}(n,C_\gamma ^q,A_\gamma ^2)} /{\mathcal {H}(n-2,C_\gamma ^q,A_\gamma ^2)} \\&\quad -\log \mathcal {H}(n-2,C_\gamma ^q,A_\gamma ^2)-nE_q\{\log \gamma \}-E_q\{\log (A_\gamma ^2+\gamma ^2)\} \end{aligned}$$

Putting these terms together and making use of the updates in steps 5 and 6 of Algorithm 1 gives the lower bound in (12).

Appendix 3: Derivation of simplified updates in Algorithm 2

It can be shown (see Wand 2014; Tan and Nott 2013) that the natural parameter of $q(\lambda )=N(\mu _\lambda ^q,{\varSigma }_\lambda ^q)$ is

$$\begin{aligned} \eta _{\lambda } = \left[ {\begin{array}{c}{- \frac{1}{2}D_{d}^{T} {\text {vec}}\left( {\mathop \sum \nolimits _{\lambda }^{{q^{{ - 1}} }} } \right) } \\ {\sum \nolimits _{\lambda }^{{q^{{ - 1}} }} {\mu _{\lambda }^{q} } } \\ \end{array} } \right] , \end{aligned}$$

where $D_d$ is a unique $d^2 \times \tfrac{d}{2}(d+1)$ matrix that transforms $\text {vech}(A)$ into $\text {vec}(A)$ for any $d \times d$ symmetric square matrix A, that is, $D_d\text {vech}(A)=\text {vec}(A)$. We use $\text {vech}(A)$ to denote the $\tfrac{1}{2}d(d+1) \times 1$ vector obtained from $\text {vec}(A)$ by eliminating all supradiagonal elements of A. Magnus and Neudecker (1988) is a good reference for the matrix differential calculus involved in the derivation below. From (13) and (Tan and Nott 2013, pg. 7), we have

$$\begin{aligned} \left[ {\begin{array}{c} { - \frac{1}{2}D_{d}^{T} {\text {vec}}\left( {\mathop \sum \nolimits _{\lambda }^{{q(t)^{{ - 1}} }} } \right) {\text { }}} \\ {\sum \nolimits _{\lambda }^{{q(t)^{{ - 1}} }} {\mu _{\lambda }^{{q(t)}} } {\text { }}} \\ \end{array} } \right] = (1 - a_{t} )\cdot \left[ {\,\begin{array}{c} { - \frac{1}{2}D_{d}^{T} {\text {vec}}\left( {\sum _{\lambda }^{{q(t - 1)^{{ - 1}} }} } \right) } \\ {\sum \nolimits _{\lambda }^{{q(t - 1)^{{ - 1}} }} {\mu _{\lambda }^{{q(t - 1)}} } } \\ \\ \end{array} } \right] \nonumber \\ + a_t \left[ {\begin{array}{cc} {D_{d}^{T} } &{} \quad 0 \\ { - 2(\mu _{\lambda }^{{q(t - 1)^{T} }} \otimes I)D_{d}^{{ + T}} D_{d}^{T} } &{} \quad I \\ \end{array} } \right] \sum \limits _{{a \in N(\lambda )}} {\left[ {\begin{array}{c} {\frac{{\partial S_{a} }}{{\partial {\text {vec}}(\sum _{\lambda }^{q} )}}} \\ {\frac{{\partial S_{a} }}{{\partial \mu _{\lambda }^{q} }}} \\ \end{array} } \right] }, \nonumber \\ \end{aligned}$$

(21)

where $\dfrac{\partial S_a}{\partial \text {vec}({\varSigma }_\lambda ^q)}$ and $\dfrac{\partial S_a}{\partial \mu _\lambda ^q}$ are evaluated at

$$\begin{aligned} {\varSigma }_\lambda ^q={{{\varSigma }_\lambda ^q}^{(t)}}^{-1} \;\; \text {and} \;\; \mu _\lambda ^q={\mu _\lambda ^q}^{(t-1)}. \end{aligned}$$

Let

$$\begin{aligned} \sum _{a \in N(\lambda )} \frac{\partial S_a}{\partial \text {vec}({\varSigma }_\lambda ^q)} = -\frac{1}{2}\text {vec}(G). \end{aligned}$$

The first line of (21) simplifies to

$$\begin{aligned}&{{{\varSigma }_\lambda ^q}^{(t)}}^{-1} = (1-a_t) {{{\varSigma }_\lambda ^q}^{(t)}}^{-1} + a_t G \\&\Rightarrow {{\varSigma }_\lambda ^q}^{(t)} =\{(1-a_t) {{{\varSigma }_\lambda ^q}^{(t)}}^{-1} + a_t G\}^{-1}. \end{aligned}$$

The second line of (21) gives

$$\begin{aligned} {{{\varSigma }_\lambda ^q}^{(t)}}^{-1} {\mu _\lambda ^q}^{(t)}= & {} (1-a_t) {{{\varSigma }_\lambda ^q}^{(t-1)}}^{-1} {\mu _\lambda ^q}^{(t-1)} \\&+ a_t G {\mu _\lambda ^q}^{(t-1)} + a_t \sum _{a \in N(\lambda )}\frac{\partial S_a}{\partial \mu _\lambda ^q} \\= & {} {{{\varSigma }_\lambda ^q}^{(t)}}^{-1} {\mu _\lambda ^q}^{(t-1)} + a_t \sum _{a \in N(\lambda )}\frac{\partial S_a}{\partial \mu _\lambda ^q} \\ \Rightarrow {\mu _\lambda ^q}^{(t)}= & {} {\mu _\lambda ^q}^{(t-1)} + a_t {{\varSigma }_\lambda ^q}^{(t)} \sum _{a \in N(\lambda )}\frac{\partial S_a}{\partial \mu _\lambda ^q}. \end{aligned}$$