Abstract
Classical Krylov subspace projection methods for the solution of linear problem \(Ax = b\) output an approximate solution \({\widetilde{x}}\simeq x\). Recently, it has been recognized that projection methods can be understood from a statistical perspective. These probabilistic projection methods return a distribution \(p({\widetilde{x}})\) in place of a point estimate \({\widetilde{x}}\). The resulting uncertainty, codified as a distribution, can, in theory, be meaningfully combined with other uncertainties, can be propagated through computational pipelines, and can be used in the framework of probabilistic decision theory. The problem we address is that the current probabilistic projection methods lead to the poorly calibrated posterior distribution. We improve the covariance matrix from previous works in a way that it does not contain such undesirable objects as \(A^{-1}\) or \(A^{-1}A^{-T}\), results in nontrivial uncertainty, and reproduces an arbitrary projection method as a mean of the posterior distribution. We also propose a variant that is numerically inexpensive in the case the uncertainty is calibrated a priori. Since it usually is not, we put forward a practical way to calibrate uncertainty that performs reasonably well, albeit at the expense of roughly doubling the numerical cost of the underlying projection method.
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11222-021-10031-9/MediaObjects/11222_2021_10031_Fig1_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11222-021-10031-9/MediaObjects/11222_2021_10031_Fig2_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11222-021-10031-9/MediaObjects/11222_2021_10031_Fig3_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11222-021-10031-9/MediaObjects/11222_2021_10031_Fig4_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11222-021-10031-9/MediaObjects/11222_2021_10031_Fig5_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11222-021-10031-9/MediaObjects/11222_2021_10031_Fig6_HTML.png)
Similar content being viewed by others
Notes
We do not consider “lucky breakdowns” (Saad 2003, Section 6.3.1)
Note that the choice \(\alpha = \beta = 0\) leads to the improper prior. In the present case the posterior distribution is always proper, so noninformative prior seems harmless. Moreover, s is a scale parameter so \(p(s)\propto s^{-1}\) is a reasonable choice (see (Gelman et al. 2013), Sect. 2.8).
This equation can be rearranged into an ordinary linear system \(Ax = b\), where A is a matrix with two indices, by the use of lexicographic order. We do not cover this here in details, consult https://github.com/VLSF/BayesKrylov for the implementation.
References
Bartels, S., Cockayne, J., Ipsen, I.C.F., Hennig, P.: Probabilistic linear solvers: a unifying view. Stat. Comput. 29(6), 1249–1263 (2019)
Bernardo, J.M., Smith, A.F.: Bayesian theory, vol. 405. Wiley, Newyork (2009)
Bezanson, J., Edelman, A., Karpinski, S., Shah, V.B.: Julia: a fresh approach to numerical computing. SIAM Rev. 59(1), 65–98 (2017)
Ciarlet, P. G. (2002). The finite element method for elliptic problems. SIAM
Cockayne, J., Oates, C.J., Ipsen, I.C.F., Girolami, M.: A Bayesian conjugate gradient method (with discussion). Bayesian Anal. 14(3), 937–1012 (2019)
Gelman, A., Carlin, J.B., Stern, H.S., Dunson, D.B., Vehtari, A., Rubin, D.B.: Bayesian data analysis. CRC Press, USA (2013)
Hackbusch, W. (2016). Iterative solution of large sparse systems of equations, volume 95 of Applied Mathematical Sciences. Springer, [Cham], second edition
Hennig, P.: Probabilistic interpretation of linear solvers. SIAM J. Optim. 25(1), 234–260 (2015)
Hennig, P., Kiefel, M.: Quasi-Newton methods: a new direction. J. Mach. Learn. Res. 14, 843–865 (2013)
Hestenes, M.R., Stiefel, E., et al.: Methods of conjugate gradients for solving linear systems, vol. 49. NBS Washington, DC (1952)
Kennedy, M.C., O’Hagan, A.: Predicting the output from a complex computer code when fast approximations are available. Biometrika 87(1), 1–13 (2000)
Krishnamoorthy, K. (2016). Handbook of statistical distributions with applications. CRC Press, Boca Raton, FL, second edition
Lattimore, T., Szepesvári, C.: Bandit algorithms. Cambridge University Press, Cambridge (2020)
Peherstorfer, B., Willcox, K., Gunzburger, M.: Survey of multifidelity methods in uncertainty propagation, inference, and optimization. Siam Review 60(3), 550–591 (2018)
Pletcher, R.H., Tannehill, J.C., Anderson, D.: Computational fluid mechanics and heat transfer. CRC Press, USA (2012)
Reid, T. W., Ipsen, I. C., Cockayne, J., and Oates, C. J. (2020). A probabilistic numerical extension of the conjugate gradient method. ar**v preprint ar**v:2008.03225
Saad, Y.: Iterative methods for sparse linear systems. Society for Industrial and Applied Mathematics (SIAM), Philadelphia, PA, second edition (2003)
Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., De Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2015)
Tong, C.H., Chan, T.F., Kuo, C.J.: Multilevel filtering preconditioners: Extensions to more general elliptic problems. SIAM J. Sci. Stat. Comput. 13(1), 227–242 (1992)
Trefethen, L. N. and Bau, III, D.: Numerical linear algebra. Society for Industrial and Applied Mathematics (SIAM), Philadelphia, PA (1997)
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Fanaskov, V. Uncertainty calibration for probabilistic projection methods. Stat Comput 31, 56 (2021). https://doi.org/10.1007/s11222-021-10031-9
Received:
Accepted:
Published:
DOI: https://doi.org/10.1007/s11222-021-10031-9