An augmented Lagrangian method with constraint generation for shape-constrained convex regression problems

Abstract

Shape-constrained convex regression problem deals with fitting a convex function to the observed data, where additional constraints are imposed, such as component-wise monotonicity and uniform Lipschitz continuity. This paper provides a unified framework for computing the least squares estimator of a multivariate shape-constrained convex regression function in ${\mathbb {R}}^d$. We prove that the least squares estimator is computable via solving an essentially constrained convex quadratic programming (QP) problem with $(d+1)n$ variables, $n(n-1)$ linear inequality constraints and n possibly non-polyhedral inequality constraints, where n is the number of data points. To efficiently solve the generally very large-scale convex QP, we design a proximal augmented Lagrangian method (proxALM) whose subproblems are solved by the semismooth Newton method. To further accelerate the computation when n is huge, we design a practical implementation of the constraint generation method such that each reduced problem is efficiently solved by our proposed proxALM. Comprehensive numerical experiments, including those in the pricing of basket options and estimation of production functions in economics, demonstrate that our proposed proxALM outperforms the state-of-the-art algorithms, and the proposed acceleration technique further shortens the computation time by a large margin.

Appendices

Appendices

Derivation of the proximal map** and generalized Jacobian associated with $\mathcal {D}=\{x\in {\mathbb {R}}^d\mid \Vert x\Vert _{1}\le L\}$

For $x\in {\mathbb {R}}^d$, let $P_x=\mathrm{Diag}(\mathrm{sign}(x))\in {\mathbb {R}}^{d\times d}$, then

$$\begin{aligned} \varPi _\mathcal {D}(x)&= \underset{y\in {\mathbb {R}}^d}{\arg \min }\ \Big \{\frac{1}{2}\Vert y-x\Vert ^2\mid \Vert y\Vert _1\le L\Big \}\\&= LP_x \Big ( \underset{y\in {\mathbb {R}}^d}{\arg \min }\ \Big \{\frac{1}{2}\Vert y-P_x x/L\Vert ^2\mid e_d^T y \le 1,y\ge 0\Big \}\Big )\\&=\left\{ \begin{aligned}&x&\text{ if }\ \Vert x\Vert _1\le L, \\&L P_x \varPi _{\varDelta _d}(P_x x/L)&\text{ otherwise, } \end{aligned}\right. \end{aligned}$$

where $\varDelta _d=\{x\in {\mathbb {R}}^d\mid e_d^T x=1,x\ge 0\}$. To derive the generalized Jacobian of $\varPi _\mathcal {D}(\cdot )$, we need the generalized Jacobian of $\varPi _{\varDelta _d}(\cdot )$. Following the idea in [17, 26], we can explicitly compute an element of the generalized Jacobian of $\varPi _{\varDelta _d}(\cdot )$ at $P_x x/L$. Let K be the set of index i such that $(\varPi _{\varDelta }(P_x x/L))_i=0$. Then

$$\begin{aligned} {\widetilde{H}}=I_d-\begin{bmatrix} I_K^T&e_d \end{bmatrix}\Big ( \begin{bmatrix} I_K \\ e_d^T \end{bmatrix} \begin{bmatrix} I_K^T&e_d \end{bmatrix} \Big )^{\dagger }\begin{bmatrix} I_K \\ e_d^T \end{bmatrix} \end{aligned}$$

is an element in $\partial \varPi _{\varDelta _d}(P_x x/L)$, where $I_K$ means the matrix consisting of the rows of the identity matrix $I_d$, indexed by K. After some algebraic computation, we can see

$$\begin{aligned} {\widetilde{H}}&=I_d-\begin{bmatrix} I_K^T&e_d \end{bmatrix} \begin{bmatrix} I_{|K|}+\frac{1}{n-|K|}e_{|K|} e_{|K|}^T &{} -\frac{1}{n-|K|}e_{|K|} \\ -\frac{1}{n-|K|}e_{|K|}^T &{} \frac{1}{n-|K|} \end{bmatrix} \begin{bmatrix} I_K \\ e_d^T \end{bmatrix}=\mathrm{Diag}(r)-\frac{1}{\mathrm{nnz}(r)}rr^T, \end{aligned}$$

where $r\in {\mathbb {R}}^d$ is defined as $r_i=1$ if $(\varPi _{\varDelta }(P_x x/L))_i\ne 0$ and $r_i=0$ otherwise. Therefore,

$$\begin{aligned} H\in \partial \varPi _\mathcal {D}(x),\quad \text{ where } H=\left\{ \begin{aligned}&I_d&\text{ if }\ \Vert x\Vert _1\le L, \\&P_x {\widetilde{H}} P_x&\text{ otherwise }. \end{aligned}\right. \end{aligned}$$

A symmetric Gauss–Seidel based alternating direction method of multipliers (sGS-ADMM) for (P)

In the literature, popular first-order methods based on the framework of the alternating direction method of multipliers have been applied to solve (P). In [30, Section A.2], the problem (P) is reformulated as

$$\begin{aligned} \min _{\theta \in {\mathbb {R}}^n,\xi \in {\mathbb {R}}^{dn},\eta \in {\mathbb {R}}^{n^2}}\ \Big \{\frac{1}{2}\Vert \theta -Y\Vert ^2+p(\xi )+\delta _{+}(\eta )\mid A\theta +B\xi -\eta =0\Big \}. \end{aligned}$$

The corresponding augmented Lagrangian function for a fixed $\sigma >0$ is defined by

$$\begin{aligned} \widetilde{\mathcal {L}}_{\sigma }(\theta ,\xi ,\eta ;u)=\frac{1}{2}\Vert \theta -Y\Vert ^2+p(\xi )+\delta _{+}(\eta )+\frac{\sigma }{2}\Vert A\theta +B\xi -\eta -\frac{u}{\sigma }\Vert ^2-\frac{1}{2\sigma }\Vert u\Vert ^2. \end{aligned}$$

Then the two-block ADMM is given as

$$\begin{aligned} \left\{ \begin{aligned}&\xi ^{k+1} = \arg \min \ \widetilde{\mathcal {L}}_{\sigma }(\theta ^{k},\xi ,\eta ^{k};u^{k})=\arg \min \ \Big \{ p(\xi )+\frac{\sigma }{2}\Vert A\theta ^k+B\xi -\eta ^k-\frac{u^k}{\sigma }\Vert ^2\Big \},\\&(\theta ^{k+1},\eta ^{k+1}) = \arg \min \ \widetilde{\mathcal {L}}_{\sigma }(\theta ,\xi ^{k+1},\eta ;u^{k}),\\&u^{k+1}=u^k-\tau \sigma (A\theta ^{k+1}+B\xi ^{k+1}-\eta ^{k+1}), \end{aligned}\right. \end{aligned}$$

where $\tau \in (0,(1+\sqrt{5}/2))$ is a given step length. As described in [30], the subproblem of updating $\xi$ is separable in the variables $\xi _i$’s for $i=1,\ldots ,n$, and the update of each $\xi _i$ can be solved by using an interior point method. The update of $\theta$ and $\eta$ is performed by using a block coordinate descent method, which may converge slowly. One can also apply the directly extended three-block ADMM algorithm as in [30, Section 2.1] to solve (P), and the steps are given by

$$\begin{aligned} \left\{ \begin{aligned}&\xi ^{k+1} = \arg \min \ \widetilde{\mathcal {L}}_{\sigma }(\theta ^{k},\xi ,\eta ^{k};u^{k}),\\&\theta ^{k+1} = \arg \min \ \widetilde{\mathcal {L}}_{\sigma }(\theta ,\xi ^{k+1},\eta ^k;u^{k}),\\&\eta ^{k+1} = \arg \min \ \widetilde{\mathcal {L}}_{\sigma }(\theta ^{k+1},\xi ^{k+1},\eta ;u^{k}),\\&u^{k+1}=u^k-\tau \sigma (A\theta ^{k+1}+B\xi ^{k+1}-\eta ^{k+1}). \end{aligned}\right. \end{aligned}$$

In the directly extended three-block ADMM, the subproblem of updating $\theta$ can be computed by solving a linear system, and that of updating $\eta$ can be solved by the projection onto ${\mathbb {R}}_{+}^{n^2}$. However, it is shown in [9] that the directly extended three-block ADMM may not be convergent. Thus it is desirable to employ an algorithm that is guaranteed to converge.

In this section, we aim to present an efficient and convergent multi-block ADMM for solving (P). The authors in [11] have proposed an inexact symmetric Gauss–Seidel based multi-block ADMM for solving high-dimensional convex composite conic optimization problems, and it was demonstrated to perform better than the possibly nonconvergent directly extended multi-block ADMM. To adapt the sGS-ADMM in [11] to solve (P), we first rewrite (P) as follows:

$$\begin{aligned} \min _{\theta \in {\mathbb {R}}^n,\xi ,y\in {\mathbb {R}}^{dn},\eta \in {\mathbb {R}}^{n^2}} \ \Big \{\frac{1}{2}\Vert \theta -Y\Vert ^2+p(y)+\delta _{+}(\eta )\Bigm |A\theta +B\xi -\eta =0,\ \xi -y=0\Big \}. \end{aligned}$$

(24)

Given a parameter $\sigma >0$, the augmented Lagrangian function associated with (24) is defined by

$$\begin{aligned} \widehat{\mathcal {L}}_{\sigma }(\theta ,\xi ,y,\eta ;u,v)&=\frac{1}{2}\Vert \theta -Y\Vert ^2+p(y)+\delta _{+}(\eta ) \nonumber \\&\qquad -\langle u,A\theta +B\xi -\eta \rangle -\langle v,\xi -y\rangle \nonumber \\&\qquad +\frac{\sigma }{2}\Vert A\theta +B\xi -\eta \Vert ^2+\frac{\sigma }{2}\Vert \xi -y\Vert ^2\nonumber \\&=\frac{1}{2}\Vert \theta -Y\Vert ^2+p(y)+\delta _{+}(\eta )+\frac{\sigma }{2}\Vert A\theta +B\xi -\eta \nonumber \\&\qquad -\frac{u}{\sigma }\Vert ^2+\frac{\sigma }{2}\Vert \xi -y-\frac{v}{\sigma }\Vert ^2\nonumber \\&\qquad -\frac{1}{2\sigma }\Vert u\Vert ^2-\frac{1}{2\sigma }\Vert v\Vert ^2. \end{aligned}$$

(25)

Then the sGS-ADMM algorithm for solving (P) is given as in Algorithm 4.

In Algorithm 4, all the subproblems can be solved explicitly. In Step 1, $\eta ^{k+1}$ and $y^{k+1}$ are separable and can be solved independently as

$$\begin{aligned} y^{k+1} = \mathrm{Prox}_{p/\sigma }(\xi ^k-v^k/\sigma ),\quad \eta ^{k+1} = \varPi _{+}(A\theta ^k+B\xi ^k-u^k/\sigma ), \end{aligned}$$

where $\varPi _{\pm }(\cdot )$ denotes the projection onto ${\mathbb {R}}^{n^2}_{\pm }$. In Step 2a and Step 2c, $\theta$ can be computed by solving the following linear system

$$\begin{aligned} (I_n+\sigma A^T A)\theta = Y-\sigma A^T (B\xi -\eta -u/\sigma ). \end{aligned}$$

By noting that $A^TA=2nI_n-2e_ne_n^T$, one can apply the Sherman-Morrison-Woodbury formula to compute

$$\begin{aligned} (I_n+\sigma A^T A)^{-1} = \frac{1}{1+2\sigma n}(I_n+2\sigma e_ne_n^T). \end{aligned}$$

Thus $\theta$ can be computed in O(n) operations. For Step 2b, $\xi ^{k+1}$ can be computed by solving the linear equation

$$\begin{aligned} (I_{dn}+B^T B)\xi = y^{k+1}+v^k/\sigma -B^T(A{\widehat{\theta }}^{k+1}-\eta ^{k+1}-u^k/\sigma ). \end{aligned}$$

As the coefficient matrix $I_{dn}+B^TB$ is a block diagonal matrix consisting of n blocks of $d\times d$ submatrices, each $\xi _i$ can be computed separately, and the inverse of each block only needs to be computed once.

The convergence result of Algorithm 4 is presented in the following theorem, which is taken directly from [11, Theorem 5.1].

Theorem 5

Suppose that the solution set to the KKT system (10) is nonempty. Let $\{(\theta ^k,\xi ^k,y^k,\eta ^k,u^k,v^k)\}$ be the sequence generated by Algorithm 4. Then $\{(\theta ^k,\xi ^k,y^k,\eta ^k)\}$ converges to an optimal solution of problem (24), and $\{(u^k,v^k)\}$ converges to an optimal solution of its dual (D).

More results on comparison of algorithms for solving (P)

Tables 7, 8, 9, 10, 11 and 12 show the comparison among proxALM, sGS-ADMM and MOSEK on instances with relatively large d and n. Note that here we set the stop** criterion to $R_{\mathrm{KKT}}\le 10^{-6}$ to show that our proposed proxALM is capable of solving the problem (P) to relatively high accuracy. As one can see that, when estimating the function $\psi (x)=\exp (p^T x)$ for moderate $(d,n)=(100,1000)$, proxALM is about 3 times faster than sGS-ADMM, and about 29 times faster than MOSEK. For the case when $d=100$, $n=4000$, which is a large problem with 404, 000 variables and about 16, 000, 000 inequality constraints, MOSEK runs out of memory, while proxALM could solve it within 7 minutes and sGS-ADMM takes 17 minutes. From the tables, we can see that sGS-ADMM performs much better than MOSEK in each instance, and proxALM performs even better than sGS-ADMM. In most of the cases, proxALM is at least 10 times faster than MOSEK.

Table 7 Convex regression for test function $\psi (x)=\exp (p^T x)$, where p is a given random vector with each coordinate drawn from the standard normal distribution

Table 8 Convex regression with monotone constraint (non-decreasing) for the test function $\psi (x)=(e_d^T x)_{+}$

Table 9 Convex regression with box constraint ($L=0_d$, $U=e_d$) for the test function $\psi (x)=\ln (1+\exp (e_d^Tx))$

Table 10 Convex regression with Lipschitz constraint ($p=1$, $q=\infty$, $L=1$) for the test function $\psi (x)=\sqrt{1+x^Tx}$

Table 11 Convex regression with Lipschitz constraint ($p=2$, $q=2$, $L=\lambda _{\mathrm{max}}(Q)$) for the test function $\psi (x)=\sqrt{x^T Qx}$

Table 12 Convex regression with Lipschitz constraint ($p=\infty$, $q=1$, $L=1$) for the test function $\psi (x)=\ln (1+e^{x_1}+\cdots +e^{x_d})$

Property of basket option of two European call options

The function V(x, y) is differentiable since it is the solution of the Black-Scholes PDE. By the definition of V, we can see that V is non-decreasing in x and y, which means that $\nabla V(x,y)\ge 0$. According to the distribution of $S_T^1$ and $S_T^2$, we have that

$$\begin{aligned} V(x,y) = e^{-r(T-t)} {\mathbb {E}}_z f(x,y,z), \end{aligned}$$

where

$$\begin{aligned} f(x,y,z)&=(xw_1 e^{(r-\sigma _1^2/2)(T-t)+\sqrt{T-t}z_1}+yw_2 e^{(r-\sigma _2^2/2)(T-t)+\sqrt{T-t}z_2}-K)_+,\\ \begin{pmatrix} z_1\\ z_2 \end{pmatrix}&\sim \mathcal {N}(0,\begin{pmatrix} \sigma _1^2 &{} \rho \sigma _1\sigma _2\\ \rho \sigma _1\sigma _2 &{}\sigma _2^2 \end{pmatrix}). \end{aligned}$$

For any $x_1,x_2,y\in {\mathbb {R}}$, we can see that

$$\begin{aligned} |V(x_1,y)-V(x_2,y)|&=e^{-r(T-t)} \Big | {\mathbb {E}}_z [f(x_1,y,z)-f(x_1,y,z)]\Big |\\&\le e^{-r(T-t)} {\mathbb {E}}_z |f(x_1,y,z)-f(x_1,y,z)|\\&\le e^{-r(T-t)} {\mathbb {E}}_z [w_1e^{(r-\sigma _1^2/2)(T-t)+\sqrt{T-t}z_1} |x_1-x_2|]\\&=w_1|x_1-x_2|e^{-\sigma _1^2/2(T-t)}{\mathbb {E}}_z [e^{\sqrt{T-t}z_1}]\\&=w_1|x_1-x_2|. \end{aligned}$$

Similarly, we can prove that for any $x,y_1,y_2\in {\mathbb {R}}$,

$$\begin{aligned} |V(x,y_1)-V(x,y_2)|\le w_2|y_1-y_2|. \end{aligned}$$

Therefore, we have that fact that $0\le \nabla V(x,y)\le w$ for any x, y.

A finite difference method for estimating the basket option of two European call options

It is well-known that the function $V(x,y)=U(0,x,y)$, where U satisfies the Black-Scholes PDE

$$\begin{aligned} \left\{ \begin{aligned}&\frac{\partial U}{\partial t}+rx\frac{\partial U}{\partial x}+ry\frac{\partial U}{\partial y} +\frac{1}{2}\sigma _1^2x^2\frac{\partial ^2 U}{\partial ^2 x^2}+\rho \sigma _1\sigma _2xy\frac{\partial ^2 U}{\partial xy}+\frac{1}{2}\sigma _2^2y^2\frac{\partial ^2 U}{\partial ^2 y^2}-rU=0,\\&U(T,x,y)=(w_1x+w_2y-K)^+. \end{aligned} \right. \end{aligned}$$

Let $\tau = T-t$, $u(\tau ,x,y)=U(t,x,y)$, then u satisfies

$$\begin{aligned} \left\{ \begin{aligned}&\frac{\partial u}{\partial \tau }-rx\frac{\partial u}{\partial x}-ry\frac{\partial u}{\partial y} -\frac{1}{2}\sigma _1^2x^2\frac{\partial ^2 u}{\partial ^2 x^2}-\rho \sigma _1\sigma _2xy \frac{\partial ^2 u}{\partial xy}-\frac{1}{2}\sigma _2^2y^2\frac{\partial ^2 u}{\partial ^2 y^2}+ru=0,\\&u(0,x,y)=(w_1x+w_2y-K)^+. \end{aligned} \right. \end{aligned}$$

The above convection-diffusion equation can be solved numerically on a bounded region $(0,x_{\max })\times (0,y_{\max })$ by the standard finite difference method with the artificial boundary conditions

$$\begin{aligned} \left\{ \begin{aligned}&u(\tau ,x,0)=c(w_1x,K,r,\tau ,\sigma _1),\\&u(\tau ,0,y)=c(w_2y,K,r,\tau ,\sigma _2),\\&\frac{\partial }{\partial x}u(\tau ,x_{\max },y)=w_1,\\&\frac{\partial }{\partial y}u(\tau ,x,y_{\max })=w_2,\\ \end{aligned} \right. \end{aligned}$$

where

$$\begin{aligned} c(x,K,r,\tau ,\sigma )=x\varPhi (d_1)-Ke^{-r\tau }\varPhi (d_2),\quad d_{1,2} = \frac{\log \frac{x}{K}+(r\pm \frac{1}{2}\sigma ^2)\tau }{\sigma \sqrt{\tau }}, \end{aligned}$$

and $\varPhi (\cdot )$ is the cumulative distribution function of the standard normal distribution.