Abstract
With the recent emergence of mixed precision hardware, there has been a renewed interest in its use for solving numerical linear algebra problems fast and accurately. The solution of total least squares problems, i.e., solving \(\varvec{\min }_{\varvec{E,r}} \Vert [\varvec{E, r}]\Vert _{\varvec{F}}\) subject to \(({\varvec{A}}+{\varvec{E}}){\varvec{x}}={\varvec{b}}+{\varvec{r}}\), arises in numerous applications. Solving this problem requires finding the smallest singular value and corresponding right singular vector of \([\varvec{A,b}]\), which is challenging when \(\varvec{A}\) is large and sparse. An efficient algorithm for this case due to Björck et al. (SIAM J. Matrix Anal. Appl. 22(2), 413–429 2000), called RQI-PCGTLS, is based on Rayleigh quotient iteration coupled with the preconditioned conjugate gradient method. We develop a mixed precision variant of this algorithm, RQI-PCGTLS-MP, in which up to three different precisions can be used. We assume that the lowest precision is used in the computation of the preconditioner and give theoretical constraints on how this precision must be chosen to ensure stability. In contrast to standard least squares, for total least squares, the resulting constraint depends not only on the matrix \(\varvec{A}\), but also on the right-hand side \(\varvec{b}\). We perform a number of numerical experiments on model total least squares problems used in the literature, which demonstrate that our algorithm can attain the same accuracy as RQI-PCGTLS albeit with a potential convergence delay due to the use of low precision. Performance modeling shows that the mixed precision approach can achieve up to a \(\varvec{4}\times \) speedup depending on the size of the matrix and the number of Rayleigh quotient iterations performed.
Similar content being viewed by others
Availability of data and materials
The authors confirm that all data generated or analyzed in the development of this work are adequately included or referenced in the article itself.
Code Availability
The authors confirm that all codes written in the development of this work are referenced in the article itself.
References
Abdelfattah, A., Anzt, H., Boman, E., Carson, E., Cojean, T., Dongarra, J., Fox, A., Gates, M., Higham, N., Li, X., Loe, J., Luszczek, P., Pranesh, S., Rajamanickam, S., Ribizel, T., Smith, B., Swirydowicz, K., Thomas, S., Tomov, S., Yang, U.: A survey of numerical linear algebra methods utilizing mixed-precision arithmetic. Int. J. High Perform. Comput. Appl. 109434202110033 (2021). https://doi.org/10.1177/10943420211003313
Higham, N.J., Mary, T.: Mixed precision algorithms in numerical linear algebra. Acta Numer. 31, 347–414 (2022)
Björck, Å.: Iterative refinement of linear least squares solutions I. Numer. Mat. 7(4), 257–278 (1967)
Demmel, J., Hida, Y., Riedy, E.J., Li, X.S.: Extra-precise iterative refinement for overdetermined least squares problems. ACM Trans. Math. Softw. (TOMS) 35(4), 1–32 (2009)
Carson, E., Higham, N.J., Pranesh, S.: Three-precision GMRES-based iterative refinement for least squares problems. SIAM J. Sci. Comput. 42(6), 4063–4083 (2020). https://doi.org/10.1137/20M1316822
Higham, N.J., Pranesh, S.: Exploiting lower precision arithmetic in solving symmetric positive definite linear systems and least squares problems. SIAM J. Sci. Comput. 43(1), 258–277 (2021)
Yamazaki, I., Tomov, S., Dongarra, J.: Mixed-precision Cholesky QR factorization and its case studies on multicore CPU with multiple GPUs. SIAM J. Sci. Comput. 37(3), 307–330 (2015)
Golub, G.H., Van Loan, C.F.: An analysis of the total least squares problem. SIAM J. Numer. Anal. 17(6), 883–893 (1980)
Björck, Å., Heggernes, P., Matstoms, P.: Methods for large scale total least squares problems. SIAM J. Matrix Anal. Appl. 22(2), 413–429 (2000)
Van Huffel, S.: Iterative algorithms for computing the singular subspace of a matrix associated with its smallest singular values. Linear Algebra Appl. 154, 675–709 (1991)
Hnětynková, I., Plesinger, M., Strakoš, Z.: The core problem within a linear approximation problem \(AX\approx B\) with multiple right-hand sides. SIAM J. Matrix Anal. Appl. 34(3), 917–931 (2013)
Connolly, M.P., Higham, N.J.: Probabilistic rounding error analysis of Householder QR factorization. Technical Report 2022.5 (February 2022). Revised August 2022. http://eprints.maths.manchester.ac.uk/2865/
Björck, Å.: Newton and Rayleigh quotient methods for total least squares problems. In: Recent advances in total least squares techniques and errors–in–variables modeling: proceedings of the second international workshop on total least squares and errors–in–variables modeling, pp. 149–160 (1997)
Szyld, D.B.: Criteria for combining inverse and Rayleigh quotient iteration. SIAM J. Numer. Anal. 25(6), 1369–1375 (1988)
Freitag, M.A., Spence, A.: Convergence theory for inexact inverse iteration applied to the generalised nonsymmetric eigenproblem. Electron. Trans. Numer. Anal. 28, 40–64 (2007)
Simoncini, V., Eldén, L.: Inexact Rayleigh quotient-type methods for eigenvalue computations. BIT Numer. Math. 42(1), 159–182 (2002)
Kelley, C.: Newton’s method in mixed precision. SIAM Rev. 64(1), 191–211 (2022)
Tisseur, F.: Newton’s method in floating point arithmetic and iterative refinement of generalized eigenvalue problems. SIAM J. Matrix Anal. Appl. 22(4), 1038–1057 (2001)
Björck, Å.: Numerical Methods for Least Squares Problems. SIAM, Philadelphia (1996)
Higham, N.J.: Accuracy and Stability of Numerical Algorithms. SIAM, Philadelphia (2002)
Wilkinson, J.H.: Modern error analysis. SIAM Rev. 13(4), 548–568 (1971)
Demmel, J.: On floating point errors in Cholesky. Technical Report UT-CS-89-87, Computer Science Department, University of Tennessee, Tennessee (1989)
Multiprecision Computing Toolbox for MATLAB, Advanpix LLC. Version 4.8.3.14460 (2021). http://www.advanpix.com/
Higham, N.J., Pranesh, S.: Simulating low precision floating-point arithmetic. SIAM J. Sci. Comput. 41(5), 585–602 (2019)
Diao, H.-A., Sun, Y.: Mixed and componentwise condition numbers for a linear function of the solution of the total least squares problem. Linear Algebra Appl. 544, 1–29 (2018). https://doi.org/10.1016/j.laa.2018.01.008
Trussell, H.: Convergence criteria for iterative restoration methods. IEEE Transactions on acoustics, speech, and signal processing 31(1), 129–136 (1983)
Van Huffel, S., Vandewalle, J.: The Total Least Squares Problem: Computational Aspects and Analysis. SIAM, Philadelphia (1991)
Acknowledgements
This work is co-funded by the Charles University grant no. SVV-2023-260711, GAUK project no. 202722, the Exascale Computing Project (17-SC-20-SC), a collaborative effort of the U.S. Department of Energy Office of Science and the National Nuclear Security Administration, and by the European Union (ERC, inEXASCALE, 101075632). Views and opinions expressed are those of the authors only and do not necessarily reflect those of the European Union or the European Research Council. Neither the European Union nor the granting authority can be held responsible for them.
Funding
Both authors are supported by the Charles University grant no. SVV-2023-260711, GAUK project no. 202722, the Exascale Computing Project (17-SC-20-SC), a collaborative effort of the U.S. Department of Energy Office of Science and the National Nuclear Security Administration, and by the European Union (ERC, inEXASCALE, 101075632). Views and opinions expressed are those of the authors only and do not necessarily reflect those of the European Union or the European Research Council. Neither the European Union nor the granting authority can be held responsible for them.
Author information
Authors and Affiliations
Contributions
Both authors contributed equally to the manuscript and read and approved the final manuscript.
Corresponding author
Ethics declarations
Ethics approval
Not applicable.
Consent for publication
Both authors consent to publication.
Conflict of interest
The authors declare no competing interests.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Both authors contributed equally to this work.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Oktay, E., Carson, E. Mixed precision Rayleigh quotient iteration for total least squares problems. Numer Algor 96, 777–798 (2024). https://doi.org/10.1007/s11075-023-01665-z
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11075-023-01665-z