A family of optimal weighted conjugate-gradient-type methods for strictly convex quadratic minimization (original) (raw)
Related papers
A hybrid gradient method for strictly convex quadratic programming
Numerical Linear Algebra with Applications, 2020
In this paper, we present a reliable hybrid algorithm for solving convex quadratic minimization problems. At the k-th iteration, two points are computed: first, an auxiliary pointẋ k is generated by performing a gradient step using an optimal steplength, and secondly, the next iterate x k+1 is obtained by means of weighted sum ofẋ k with the penultimate iterate x k−1. The coefficient of the linear combination is computed by minimizing the residual norm along the line determined by the previous points. In particular, we adopt an optimal, non-delayed steplength in the first step and then use a smoothing technique to impose a delay on the scheme. Under a modest assumption, we show that our algorithm is Q-linearly convergent to the unique solution of the problem. Finally, we report numerical experiments on strictly convex quadratic problems, showing that the proposed method is competitive in terms of CPU-time and iterations with the conjugate gradient method. KEYWORDS Gradient methods, convex quadratic optimization and linear system of equations.
Properties of the delayed weighted gradient method
Computational Optimization and Applications
The delayed weighted gradient method, recently introduced in [13], is a low-cost gradient-type method that exhibits a surprisingly and perhaps unexpected fast convergence behavior that competes favorably with the well-known conjugate gradient method for the minimization of convex quadratic functions. In this work, we establish several orthogonality properties that add understanding to the practical behavior of the method, including its finite termination. We show that if the n × n real Hessian matrix of the quadratic function has only p < n distinct eigenvalues, then the method terminates in p iterations. We also establish an optimality condition, concerning the gradient norm, that motivates the use of this novel scheme when low precision is required for the minimization of non-quadratic functions.
Computational Optimization and Applications, 2007
We consider the class of quadratically-constrained quadratic-programming methods in the framework extended from optimization to more general variational problems. Previously, in the optimization case, Anitescu (SIAM J. Optim. 12, 949-978, 2002) showed superlinear convergence of the primal sequence under the Mangasarian-Fromovitz constraint qualification and the quadratic growth condition. Quadratic convergence of the primal-dual sequence was established by Fukushima, Luo and Tseng (SIAM J. Optim. 13, 1098-1119, 2003) under the assumption of convexity, the Slater constraint qualification, and a strong second-order sufficient condition. We obtain a new local convergence result, which complements the above (it is neither stronger nor weaker): we prove primal-dual quadratic convergence under the linear independence constraint qualification, strict complementarity, and a secondorder sufficiency condition. Additionally, our results apply to variational problems beyond the optimization case. Finally, we provide a necessary and sufficient condition for superlinear convergence of the primal sequence under a Dennis-Moré type condition.
A q-Gradient Descent Algorithm with Quasi-Fejér Convergence for Unconstrained Optimization Problems
Fractal and Fractional, 2021
We present an algorithm for solving unconstrained optimization problems based on the q-gradient vector. The main idea used in the algorithm construction is the approximation of the classical gradient by a q-gradient vector. For a convex objective function, the quasi-Fejér convergence of the algorithm is proved. The proposed method does not require the boundedness assumption on any level set. Further, numerical experiments are reported to show the performance of the proposed method.
2005
A new active set based algorithm is proposed that uses the conjugate gradient method to explore the face of the feasible region defined by the current iterate and the reduced gradient projection with the fixed steplength to expand the active set. The precision of approximate solutions of the auxiliary unconstrained problems is controlled by the norm of violation of the Karush-Kuhn-Tucker conditions at active constraints and the scalar product of the reduced gradient and the reduced gradient projection. The modifications were exploited to find the rate of convergence in terms of the spectral condition number of the Hessian matrix, to preserve its finite termination property even for problems whose solution does not satisfy the strict complementarity condition, and to avoid any backtracking at the cost of evaluation of an upper bound for the spectral radius of the Hessian matrix. The performance of the algorithm is illustrated on solution of the inner obstacle problems. The result is an important ingredient in development of scalable algorithms for numerical solution of elliptic variational inequalities.
Computational Optimization and Applications, 2005
A new active set based algorithm is proposed that uses the conjugate gradient method to explore the face of the feasible region defined by the current iterate and the reduced gradient projection with the fixed steplength to expand the active set. The precision of approximate solutions of the auxiliary unconstrained problems is controlled by the norm of violation of the Karush-Kuhn-Tucker conditions at active constraints and the scalar product of the reduced gradient with the reduced gradient projection. The modifications were exploited to find the rate of convergence in terms of the spectral condition number of the Hessian matrix, to prove its finite termination property even for problems whose solution does not satisfy the strict complementarity condition, and to avoid any backtracking at the cost of evaluation of an upper bound for the spectral radius of the Hessian matrix. The performance of the algorithm is illustrated on solution of the inner obstacle problems. The result is an important ingredient in development of scalable algorithms for numerical solution of elliptic variational inequalities.
Computational Optimization and Applications
In this paper, we consider the nonconvex quadratically constrained quadratic programming (QCQP) with one quadratic constraint. By employing the conjugate gradient method, an efficient algorithm is proposed to solve QCQP that exploits the sparsity of the involved matrices and solves the problem via solving a sequence of positive definite system of linear equations after identifying suitable generalized eigenvalues. Some numerical experiments are given to show the effectiveness of the proposed method and to compare it with some recent algorithms in the literature.
Steepest descent with momentum for quadratic functions is a version of the conjugate gradient method
Neural Networks, 2004
It is pointed out that the so called momentum method, much used in the neural network literature as an acceleration of the backpropagation method, is a stationary version of the conjugate gradient method. Connections with the continuous optimization method known as heavy ball with friction are also made. In both cases, adaptive (dynamic) choices of the so called learning rate and momentum parameters are obtained using a control Liapunov function analysis of the system.
Sequential Quadratic Programming Algorithms for Optimization
1989
The problem considered in this dissertation is that of finding local minimizers for a function subject to general nonlinear inequality constraints, when first and perhaps second derivatives are available. The methods studied belong to the class of sequential quadratic programming (SQP) algorithms. In particular, the methods are based on the SQP algorithm embodiod in tho co'r' ".P;OL. which was developed at the Systems Optimization Laboratory, Stanford University. The goal of the dissertation is to develop SQP algorithms that allow some flexibility in their design. Specifically, we are interested in introducing modifications that enable the algorithms to solve large-scale problems efficiently. The following issues are considered in detail: a The use of approximate solutions for the QP subproblem. Instead of trying to obtain the search direction as a minimizer for the QP, the solution process is terminated after a limited number of iterations. Suitable termination criteria are defined that ensure convergence for an algorithm that uses a quasi-Newton approximation for the full Hessian. Theorems concerning the rate of convergence are also given. e The use of approximations for the reduced Hessian in tne construction of the QP subproblems. For many problems the reduced Hessian is considerably smaller than the full Hessian. Consequently, there are considerable practical benefits to be gained by orfly requiring an approximation to the reduced Hessian. Theorems are proved concerning the convergence and rate of convergence for an algorithm that uses a quasi-Newton approximation for the reduced Hessian when early termination of the QP subproblem is enforced. Aceeg ion For iN71S GEA&I D7:7 TA;
SIAM Journal on Optimization
We propose a gradient-based method for quadratic programming problems with a single linear constraint and bounds on the variables. Inspired by the GPCG algorithm for boundconstrained convex quadratic programming [J.J. Moré and G. Toraldo, SIAM J. Optim. 1, 1991], our approach alternates between two phases until convergence: an identification phase, which performs gradient projection iterations until either a candidate active set is identified or no reasonable progress is made, and an unconstrained minimization phase, which reduces the objective function in a suitable space defined by the identification phase, by applying either the conjugate gradient method or a recently proposed spectral gradient method. However, the algorithm differs from GPCG not only because it deals with a more general class of problems, but mainly for the way it stops the minimization phase. This is based on a comparison between a measure of optimality in the reduced space and a measure of bindingness of the variables that are on the bounds, defined by extending the concept of proportional iterate, which was proposed by some authors for box-constrained problems. If the objective function is bounded, the algorithm converges to a stationary point thanks to a suitable application of the gradient projection method in the identification phase. For strictly convex problems, the algorithm converges to the optimal solution in a finite number of steps even in case of degeneracy. Extensive numerical experiments show the effectiveness of the proposed approach.