A difference-of-convex approach for split feasibility with applications to matrix factorizations and outlier detection

Abstract

The split feasibility problem is to find an element in the intersection of a closed set C and the linear preimage of another closed set D, assuming the projections onto C and D are easy to compute. This class of problems arises naturally in many contemporary applications such as compressed sensing. While the sets C and D are typically assumed to be convex in the literature, in this paper, we allow both sets to be possibly nonconvex. We observe that, in this setting, the split feasibility problem can be formulated as an optimization problem with a difference-of-convex objective so that standard majorization-minimization type algorithms can be applied. Here we focus on the nonmonotone proximal gradient algorithm with majorization studied in Liu et al. (Math Program, 2019. https://doi.org/10.1007/s10107-018-1327-8, Appendix A). We show that, when this algorithm is applied to a split feasibility problem, the sequence generated clusters at a stationary point of the problem under mild assumptions. We also study local convergence property of the sequence under suitable assumptions on the closed sets involved. Finally, we perform numerical experiments to illustrate the efficiency of our approach on solving split feasibility problems that arise in completely positive matrix factorization, (uniformly) sparse matrix factorization, and outlier detection.

Proof of Theorem 2

Proof

The boundedness of $\{x^t\}$ follows from Theorem 1(i). We now prove convergence of the whole sequence. By assumption, $x^*$ is an accumulation point of $\{x^t\}$ so that the function

$$\begin{aligned} \kappa (u):= \frac{1}{2}d^2(u,D) \end{aligned}$$

is continuously differentiable at $Ax^*$ with locally Lipschitz gradient. Then we have from [30, Example 8.53], [25, Theorem 1.110(ii)] and the chain rule that

$$\begin{aligned} \nabla (\kappa \circ A)(x^*) = A^T(Ax^* - \mathrm{Proj}_D(Ax^*)). \end{aligned}$$

Using this and the fact that $x^*$ is a stationary point of the split feasibility problem (2) (see Theorem 1(ii)), we deduce further that

$$\begin{aligned} \begin{aligned} 0&\in A^T(Ax^* - \mathrm{Proj}_D(Ax^*)) + N_C(x^*) \\&= \nabla (\kappa \circ A)(x^*) + N_C(x^*) = \partial F(x^*), \end{aligned} \end{aligned}$$

where the last equality follows from [30, Exercise 8.8(c)]. In particular, it holds that $x^*\in \mathrm{dom}\partial F$.

Since F is a KL function and $x^*\in \mathrm{dom}\partial F$, there exist $\epsilon > 0$ and a continuous concave function $\psi$ as in Definition 1 so that

$$\begin{aligned} \psi '(F(x) - F(x^*))\cdot d(0,\partial F(x))\ge 1 \end{aligned}$$

(35)

whenever $\Vert x - x^*\Vert \le \epsilon$ and $F(x^*)< F(x) < F(x^*) + \epsilon$. Moreover, by shrinking $\epsilon$ if necessary, we may assume without loss of generality that $\nabla \kappa$ is globally Lipschitz in $\{Ax:\; x\in B(x^*,\epsilon )\}$ with Lipschitz modulus $\tau$.

Next, observe from (7) with $M = 0$ that $\{F(x^t)\}$ is nonincreasing. Since F is also nonnegative, we deduce that the limit $\lim \limits _{t\rightarrow \infty }F(x^t)$ exists. In addition, notice that F is continuous in its closed domain and $x^*$ is an accumulation point of $\{x^t\}$. Thus, we conclude that $\lim \limits _{t\rightarrow \infty }F(x^t) = F(x^*)$.

Now, if $F(x^{t_0}) = F(x^*)$ for some $t_0 \ge 0$, then we see from (7) with $M = 0$ and $\lim \limits _{t\rightarrow \infty }F(x^t) = F(x^*)$ that $x^{t+1}=x^t$ for all $t\ge t_0$, which implies that the sequence $\{x^t\}$ converges (finitely). Thus, from now on, we focus on the case that $F(x^t) > F(x^*)$ for all $t\ge 0$.

In this case, note from Lemma 2 that there exists $N_0 > 1$ so that $\Vert x^{t}-x^{t-1}\Vert \le \frac{\epsilon }{2}$ whenever $t\ge N_0$. Also, using Lemma 2, the definition of accumulation point and the fact that $\lim \limits _{t\rightarrow \infty }F(x^t) = F(x^*)$, there exists $N_1 \ge N_0$ so that

• (i) $\Vert x^{N_1}-x^*\Vert \le \frac{\epsilon }{2}$ and $F(x^*)< F(x^{N_1}) < F(x^*) + \epsilon$.

• (ii) $\Vert x^{N_1}-x^*\Vert + \Vert x^{N_1} - x^{N_1-1}\Vert + C_1\psi (F(x^{N_1})-F(x^*))\le \frac{\epsilon }{2}$,

where $C_1 := \frac{2(\tau \lambda _{\max }(A^TA) + \beta )}{c}$, c is as in (7), $\tau$ is the Lipschitz continuity modulus of $\nabla \kappa$ on $\{Ax:\; x\in B(x^*,\epsilon )\}$, $\beta = \sup _t {\bar{L}}_t$ with ${\bar{L}}_t$ defined in Step 2 of $\textsf {SpFeas}_{\mathrm{DC}_{\textsf {ls}}}$, and $\beta$ is finite according to Lemma 2.

We claim that if $t\ge N_1$ and $\Vert x^t - x^*\Vert \le \epsilon /2$, then

$$\begin{aligned} 2\Vert x^{t+1}-x^t\Vert \le \Vert x^t-x^{t-1}\Vert + C_1 \left[ \psi (F(x^t) - F(x^*)) - \psi (F(x^{t+1}) - F(x^*))\right] . \end{aligned}$$

(36)

To this end, note that since $x^t\in B(x^*,\epsilon /2)$ and $t\ge N_1\ge N_0$, we have $\Vert x^{t}-x^{t-1}\Vert \le \frac{\epsilon }{2}$ and hence $\Vert x^{t-1} - x^*\Vert \le \epsilon$. Thus, $\kappa$ is continuously differentiable at $Ax^{t-1}$ and $Ax^t$. Moreover, we see from [30, Example 8.53] and [25, Theorem 1.110(ii)] (see also (10)) that $\nabla (\kappa \circ A)(x^{t-1}) = A^T(Ax^{t-1} - \mathrm{Proj}_D(Ax^{t-1}))$. Using this and the definition of $x^t$, we deduce that

$$\begin{aligned} x^{t}\in \mathrm{Proj}_C\left( x^{t-1}-\frac{\nabla (\kappa \circ A)(x^{t-1})}{\bar{L}_{t-1}}\right) . \end{aligned}$$

Thus, according to (1),

$$\begin{aligned} v^t:={{\bar{L}}}_{t-1}(x^{t-1}-x^t)-\nabla (\kappa \circ A)(x^{t-1})\in N_C(x^t). \end{aligned}$$

Moreover, using the definition of $v^t$, we have

$$\begin{aligned} \begin{aligned} \Vert v^t+\nabla (\kappa \circ A)(x^t)\Vert&\le \Vert \nabla (\kappa \circ A)(x^t)-\nabla (\kappa \circ A)(x^{t-1})\Vert +{{\bar{L}}}_{t-1}\Vert x^{t}-x^{t-1}\Vert \\&\le (\tau \lambda _{\max }(A^TA) + \beta )\Vert x^{t}-x^{t-1}\Vert , \end{aligned} \end{aligned}$$

(37)

where the second inequality holds because $\beta = \sup _{t}{\bar{L}}_t$ and $\nabla \kappa$ is globally Lipschitz in $\{Ax:\;x\in B(x^*,\epsilon )\}$ with Lipschitz modulus $\tau$. Since $v^t+\nabla (\kappa \circ A)(x^t)\in N_C(x^t) + \nabla (\kappa \circ A)(x^t) = \partial F(x^t)$, we obtain from (37) that

$$\begin{aligned} d(0,\partial F(x^t))\le \left( \tau \lambda _{\max }(A^TA) +\beta \right) \Vert x^t-x^{t-1}\Vert . \end{aligned}$$

Making use of this, the concavity of $\psi$ and (7) with $M = 0$, we see further that

$$\begin{aligned} \begin{aligned}&\left( \tau \lambda _{\max }(A^TA) + \beta \right) \Vert x^t-x^{t-1}\Vert \cdot \left[ \psi (F(x^t) - F(x^*)) - \psi (F(x^{t+1}) - F(x^*))\right] \\&\quad \ge d(0,\partial F(x^t))\cdot \left[ \psi (F(x^t) - F(x^*)) - \psi (F(x^{t+1}) - F(x^*))\right] \\&\quad \ge d(0,\partial F(x^t))\cdot \psi '(F(x^t) - F(x^*))\cdot \left[ F(x^t) - F(x^{t+1})\right] \\&\quad \ge \frac{c}{2}\Vert x^{t+1}-x^t\Vert ^2, \end{aligned} \end{aligned}$$

where the last inequality follows from (7) with $M=0$, (35), and the facts that $\Vert x^t-x^*\Vert \le \epsilon /2$ and that $F(x^*)< F(x^t)\le F(x^{N_1}) < F(x^*)+\epsilon$ (since $t\ge N_1$). Dividing both sides of the above inequality by $\frac{c}{2}$, taking square root, using the relation $\sqrt{ab}\le \frac{a+b}{2}$ for any nonnegative numbers a and b and invoking the definition of $C_1$, we obtain further that

$$\begin{aligned} \begin{aligned} \Vert x^{t+1}-x^t\Vert&\le \sqrt{\Vert x^t-x^{t-1}\Vert \cdot C_1\left[ \psi (F(x^t) - F(x^*)) - \psi (F(x^{t+1}) - F(x^*))\right] }\\&\le \frac{1}{2}\left( \Vert x^t-x^{t-1}\Vert + C_1\left[ \psi (F(x^t) - F(x^*)) - \psi (F(x^{t+1}) - F(x^*))\right] \right) , \end{aligned} \end{aligned}$$

from which (36) follows immediately.

Next, we show by induction that $x^t\in B(x^*,\epsilon /2)$ whenever $t\ge N_1$. The case $t = N_1$ follows from construction. Suppose that $x^t\in B(x^*,\epsilon /2)$ whenever $t = N_1, \ldots , N_1+k-1$ for some $k\ge 1$. Then

$$\begin{aligned} \Vert x^{N_1+k} - x^*\Vert\le & {} \Vert x^{N_1}-x^*\Vert + \sum _{t=N_1}^{N_1+k-1}\Vert x^{t+1}-x^t\Vert \overset{\mathrm{(a)}}{\le }\Vert x^{N_1}-x^*\Vert \\&+ \sum _{t=N_1}^{N_1+k-1}\left( \Vert x^t-x^{t-1}\Vert - \Vert x^{t+1}-x^t\Vert \right. \\&\left. + C_1 \left[ \psi (F(x^t) - F(x^*)) - \psi (F(x^{t+1}) - F(x^*))\right] \right) \\\le & {} \Vert x^{N_1}-x^*\Vert + \Vert x^{N_1}-x^{N_1-1}\Vert + C_1\psi (F(x^{N_1}) - F(x^*)) \overset{\mathrm{(b)}}{\le }\frac{\epsilon }{2}, \end{aligned}$$

where (a) follows from the induction hypothesis and (36), and (b) follows from the definition of $N_1$. Thus, $x^t\in B(x^*,\epsilon /2)$ whenever $t\ge N_1$ by induction.

Since $x^t\in B(x^*,\epsilon /2)$ whenever $t\ge N_1$, we can sum both sides of (36) from $N_1$ to $\infty$ and obtain

$$\begin{aligned} \begin{aligned}&\sum _{t=N_1}^\infty \Vert x^{t+1}-x^t\Vert \\&\quad \le \sum _{t=N_1}^\infty \bigg (\Vert x^t-x^{t-1}\Vert - \Vert x^{t+1}-x^t\Vert + C_1 \left[ \psi (F(x^t) - F(x^*)) - \psi (F(x^{t+1}) - F(x^*))\right] \bigg )\\&\quad \le \Vert x^{N_1}-x^{N_1-1}\Vert +C_1\psi (F(x^{N_1}) - F(x^*)) < \infty . \end{aligned} \end{aligned}$$

Thus, the sequence $\{x^t\}$ is Cauchy and is hence convergent. $\square$