Search
Search Results
-
Nonsmooth Nonconvex Stochastic Heavy Ball
Motivated by the conspicuous use of momentum-based algorithms in deep learning, we study a nonsmooth nonconvex stochastic heavy ball method and show...
-
An accelerated lyapunov function for Polyak’s Heavy-ball on convex quadratics
In 1964, Polyak showed that the Heavy-ball method, the simplest momentum technique, accelerates convergence of strongly-convex problems in the...
-
Universal heavy-ball method for nonconvex optimization under Hölder continuous Hessians
We propose a new first-order method for minimizing nonconvex functions with Lipschitz continuous gradients and Hölder continuous Hessians. The...
-
Heavy-Ball-Based Optimal Thresholding Algorithms for Sparse Linear Inverse Problems
Linear inverse problems arise in diverse engineering fields especially in signal and image reconstruction. The development of computational methods...
-
On the Convergence Analysis of Aggregated Heavy-Ball Method
Momentum first-order optimization methods are the workhorses in various optimization tasks, e.g., in the training of deep neural networks. Recently,...
-
Convergence rates of the Heavy-Ball method under the Łojasiewicz property
In this paper, a joint study of the behavior of solutions of the Heavy Ball ODE and Heavy Ball type algorithms is given. Since the pioneering work of...
-
Learning Proper Orthogonal Decomposition of Complex Dynamics Using Heavy-ball Neural ODEs
Proper orthogonal decomposition (POD) allows reduced-order modeling of complex dynamical systems at a substantial level, while maintaining a high...
-
A new proximal heavy ball inexact line-search algorithm
We study a novel inertial proximal-gradient method for composite optimization. The proposed method alternates between a variable metric...
-
Convergence rates for the heavy-ball continuous dynamics for non-convex optimization, under Polyak–Łojasiewicz condition
We study convergence of the trajectories of the Heavy Ball dynamical system, with constant dam** coefficient, in the framework of convex and...
-
On the Dynamics of a Heavy Symmetric Ball that Rolls Without Sliding on a Uniformly Rotating Surface of Revolution
We study the class of nonholonomic mechanical systems formed by a heavy symmetric ball that rolls without sliding on a surface of revolution, which...
-
Stochastic Heavy-Ball Method for Constrained Stochastic Optimization Problems
In this paper, we consider a heavy-ball method for the constrained stochastic optimization problem by focusing to the situation that the constraint...
-
Liouvillian Solutions in the Problem of Rolling of a Heavy Homogeneous Ball on a Surface of Revolution
AbstractThe problem of a heavy homogeneous ball rolling without slip** on a surface of revolution is a classical problem of the nonholonomic system...
-
Non-monotone Behavior of the Heavy Ball Method
We focus on the solutions of second-order stable linear difference equations and demonstrate that their behavior can be non-monotone and exhibit peak...
-
Gradient Method
In this chapter we introduce the gradient method, which is one of the first methods proposed for the unconstrained minimization of differentiable...
-
Discriminative Bayesian filtering lends momentum to the stochastic Newton method for minimizing log-convex functions
To minimize the average of a set of log-convex functions, the stochastic Newton method iteratively updates its estimate using subsampled versions of...
-
The Inertial Krasnosel’skiı̆–Mann Iteration
The inertial type algorithms [172] originate from the heavy ball method of the so-called heavy ball with friction dynamical system:...
-
Another Approach to Build Lyapunov Functions for the First Order Methods in the Quadratic Case
AbstractLyapunov functions play a fundamental role in analyzing the stability and convergence properties of optimization methods. In this paper, we...
-
A randomized block Douglas–Rachford method for solving linear matrix equation
The Douglas-Rachford method (DR) is one of the most computationally efficient iterative methods for the large scale linear systems of equations....
-
Shuffling Momentum Gradient Algorithm for Convex Optimization
The Stochastic Gradient Method (SGD) and its stochastic variants have become methods of choice for solving finite-sum optimization problems arising...
-
Higher-Order Iterative Learning Control Algorithms for Linear Systems
AbstractIterative learning control (ILC) algorithms appeared in connection with the problems of increasing the accuracy of performing repetitive...
