Diagonal Preconditioning: Theory and Algorithms (original) (raw)

2020

Diagonal preconditioning has been a staple technique in optimization and machine learning. It often reduces the condition number of the design or Hessian matrix it is applied to, thereby speeding up convergence. However, rigorous analyses of how well various diagonal preconditioning procedures improve the condition number of the preconditioned matrix and how that translates into improvements in optimization are rare. In this paper, we first provide an analysis of a popular diagonal preconditioning technique based on column standard deviation and its effect on the condition number using random matrix theory. Then we identify a class of design matrices whose condition numbers can be reduced significantly by this procedure. We then study the problem of optimal diagonal preconditioning to improve the condition number of any full-rank matrix and provide a bisection algorithm and a potential reduction algorithm with O(log(1/ϵ)) iteration complexity, where each iteration consists of an SDP...

Additive Preconditioning for Matrix Computations

2008

Our weakly random additive preconditioners facilitate the solution of linear systems of equations and other fundamental matrix computations. Compared to the popular SVD-based multiplicative preconditioners, these preconditioners are generated more readily and for a much wider class of input matrices. Furthermore they better preserve matrix structure and sparseness and have a wider range of applications, in particular to linear systems with rectangular coefficient matrices. We study the generation of such preconditioners and their impact on conditioning of the input matrix. Our analysis and experiments show the power of our approach even where we use very weak randomization and choose sparse and/or structured preconditioners.

A Randomized Algorithm for Preconditioner Selection

ArXiv, 2019

The task of choosing a preconditioner boldsymbolM\boldsymbol{M}boldsymbolM to use when solving a linear system boldsymbolAx=boldsymbolb\boldsymbol{Ax}=\boldsymbol{b}boldsymbolAx=boldsymbolb with iterative methods is difficult. For instance, even if one has access to a collection boldsymbolM1,boldsymbolM2,ldots,boldsymbolMn\boldsymbol{M}_1,\boldsymbol{M}_2,\ldots,\boldsymbol{M}_nboldsymbolM1,boldsymbolM2,ldots,boldsymbolMn of candidate preconditioners, it is currently unclear how to practically choose the boldsymbolMi\boldsymbol{M}_iboldsymbolMi which minimizes the number of iterations of an iterative algorithm to achieve a suitable approximation to boldsymbolx\boldsymbol{x}boldsymbolx. This paper makes progress on this sub-problem by showing that the preconditioner stability ∣boldsymbolI−boldsymbolM−1boldsymbolA∣mathsfF\|\boldsymbol{I}-\boldsymbol{M}^{-1}\boldsymbol{A}\|_\mathsf{F}∣boldsymbolI−boldsymbolM−1boldsymbolA∣mathsfF, known to forecast preconditioner quality, can be computed in the time it takes to run a constant number of iterations of conjugate gradients through use of sketching methods. This is in spite of folklore which suggests the quantity is impractical to compute, and a proof we give that ensures the quantity could not possibly be approximate...

Two-level Nystr\"om--Schur preconditioner for sparse symmetric positive definite matrices

2021

Randomized methods are becoming increasingly popular in numerical linear algebra. However, few attempts have been made to use them in developing preconditioners. Our interest lies in solving large-scale sparse symmetric positive definite linear systems of equations where the system matrix is preordered to doubly bordered block diagonal form (for example, using a nested dissection ordering). We investigate the use of randomized methods to construct high quality preconditioners. In particular, we propose a new and efficient approach that employs Nyström’s method for computing low rank approximations to develop robust algebraic two-level preconditioners. Construction of the new preconditioners involves iteratively solving a smaller but denser symmetric positive definite Schur complement system with multiple right-hand sides. Numerical experiments on problems coming from a range of application areas demonstrate that this inner system can be solved cheaply using block conjugate gradients...

Preconditioners for singular black box matrices

2005

This paper develops preconditioners for singular black box matrix problems. We introduce networks of arbitrary radix switches for matrices of any square dimension, and we show random full Toeplitz matrices are adequate switches for these networks. We also show a random full Toeplitz matrix to satisfy all requirements of the Kaltofen-Saunders black box matrix rank algorithm without requiring a diagonal

Loading...

Loading Preview

Sorry, preview is currently unavailable. You can download the paper by clicking the button above.