Abstract
Physics-informed deep learning has recently emerged as an effective tool for leveraging both observational data and available physical laws. Physics-informed neural networks (PINNs) and deep operator networks (DeepONets) are two such models. The former encodes the physical laws via the automatic differentiation, while the latter learns the hidden physics from data. Generally, the noisy and limited observational data as well as the over-parameterization in neural networks (NNs) result in uncertainty in predictions from deep learning models. In paper “MENG, X., YANG, L., MAO, Z., FERRANDIS, J. D., and KARNIADAKIS, G. E. Learning functional priors and posteriors from data and physics. Journal of Computational Physics, 457, 111073 (2022)”, a Bayesian framework based on the generative adversarial networks (GANs) has been proposed as a unified model to quantify uncertainties in predictions of PINNs as well as DeepONets. Specifically, the proposed approach in “MENG, X., YANG, L., MAO, Z., FERRANDIS, J. D., and KARNIADAKIS, G. E. Learning functional priors and posteriors from data and physics. Journal of Computational Physics, 457, 111073 (2022)” has two stages: (i) prior learning, and (ii) posterior estimation. At the first stage, the GANs are utilized to learn a functional prior either from a prescribed function distribution, e.g., the Gaussian process, or from historical data and available physics. At the second stage, the Hamiltonian Monte Carlo (HMC) method is utilized to estimate the posterior in the latent space of GANs. However, the vanilla HMC does not support the mini-batch training, which limits its applications in problems with big data. In the present work, we propose to use the normalizing flow (NF) models in the context of variational inference (VI), which naturally enables the mini-batch training, as the alternative to HMC for posterior estimation in the latent space of GANs. A series of numerical experiments, including a nonlinear differential equation problem and a 100-dimensional (100D) Darcy problem, are conducted to demonstrate that the NFs with full-/mini-batch training are able to achieve similar accuracy as the “gold rule” HMC. Moreover, the mini-batch training of NF makes it a promising tool for quantifying uncertainty in solving the high-dimensional partial differential equation (PDE) problems with big data.
Article PDF
Similar content being viewed by others
Avoid common mistakes on your manuscript.
References
MENG, X., YANG, L., MAO, Z., FERRANDIS, J. D., and KAENIADAKIS, G. E. Learning functional priors and posteriors from data and physics. Journal of Computational Physics, 457, 111073 (2022)
MENG, X. and KARNIADAKIS, G. E. A composite neural network that learns from multi fidelity data: application to function approximation and inverse PDE problems. Journal of Computational Physics, 401, 109020 (2020)
MENG, X., WANG, Z., FAN, D., TRIANTAFYLLOU, M. S., and KARNIADAKIS, G. E. A fast multi-fidelity method with uncertainty quantification for complex data correlations: application to vortex-induced vibrations of marine risers. Computer Methods in Applied Mechanics and Engineering, 386, 114212 (2021)
MENG, X., BABAEE, H., and KARNIADAKIS, G. E. Multi-fidelity bayesian neural networks: algorithms and applications. Journal of Computational Physics, 438, 110361 (2021)
RAISSI, M., PERDIKARIS, P., and KARNIADAKIS, G. E. Physics-informed neural networks: a deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378, 686–707 (2019)
SIRIGNANO, J. and SPILIOPOULOS, K. DGM: a deep learning algorithm for solving partial differential equations. Journal of Computational Physics, 375, 1339–1364 (2018)
HAN, J. and JENTZEN, A. W. E. Solving high-dimensional partial differential equations using deep learning. Proceedings of the National Academy of Sciences, 115(34), 8505–8510 (2018)
WEINAN, E. and YU, B. The deep Ritz method: a deep learning-based numerical algorithm for solving variational problems. Communications in Mathematics and Statistics, 6(1), 1–12 (2018)
LU, L., JIN, P., PANG, G., ZHANG, Z., and KARNIADAKIS, G. E. Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators. Nature Machine Intelligence, 3(3), 218–229 (2021)
LI, Z., KOVACHKI, N., AZIZZADENESHELI, K., LIU, B., BHATTACHARYA, K., STUART, A., and ANANDKUMAR, A. Fourier neural operator for parametric partial differential equations. ar**v Preprint, ar**v: 2010.08895 (2020) https://doi.org/10.48550/ar**v.2010.08895
ABDAR, M., POURPANAH, F., HUSSAIN, S., REZAZADEGAN, D., LIU, L., GHAVAMZADEH, M., FIEGUTH, P., CAO, X. C., KHOSRAVI, A., ACHARYA, U. R., MAKARENKOV, V., and NAHAVANDI, S. A review of uncertainty quantification in deep learning: techniques, applications and challenges. Information Fusion, 76, 243–297 (2021)
PICKERING, E., GUTH, S., KARNIADAKIS, G. E., and SAPSIS, T. P. Discovering and forecasting extreme events via active learning in neural operators. Nature Computational Science, 2(12), 823–833 (2022)
LINKA, K., SCHÄFER, A., MENG, X., ZOU, Z., KARNIADAKIS, G. E., and KUHL, E. Bayesian physics informed neural networks for real-world nonlinear dynamical systems. Computer Methods in Applied Mechanics and Engineering, 402, 115346 (2022)
NEAL, R. M. Bayesian Learning for Neural Networks, Springer Science and Business Media, Berlin (2012)
YANG, L., MENG, X., and KARNIADAKIS, G. E. B-PINNs: Bayesian physics-informed neural networks for forward and inverse PDE problems with noisy data. Journal of Computational Physics, 425, 109913 (2021)
LAKSHMINARAYANAN, B., PRITZEL, A., and BLUNDELL, C. Simple and scalable predictive uncertainty estimation using deep ensembles. 31st Annual Conference on Neural Information Processing Systems, Long Beach, State of California (2017)
PEARCE, T., LEIBFRIED, F., and BRINTRUP, A. Uncertainty in neural networks: approximately Bayesian ensembling. International Conference on Artificial Intelligence and Statistics, 234–244 (2020)
ZHANG, D., LU, L., GUO, L., and KARNIADAKIS, G. E. Quantifying total uncertainty in physics-informed neural networks for solving forward and inverse stochastic problems. Journal of Computational Physics, 397, 108850 (2019)
YAO, J., PAN, W., GHOSH, S., and DOSHI-VELEZ, F. Quality of uncertainty quantificatio for Bayesian neural network inference. ar**v Preprint, ar**v: 1906.09686 (2019) https://doi.org/10.48550/ar**v.1906.09686
PSAROS, A. F., MENG, X., ZOU, Z., GUO, L., and KARNIADAKIS, G. E. Uncertainty quantification in scientific machine learning: methods, metrics, and comparisons. Journal of Computational Physics, 477, 111902 (2023)
CHEN, T., FOX, E., and GUESTRIN, C. Stochastic gradient Hamiltonian Monte Carlo. International Conference on Machine Learning, 1683–1691 (2014)
BLUNDELL, C., CORNEBISE, J., KAVUKCUOGLU, K., and WIERSTRA, D. Weight uncertainty in neural networks. ar**v Preprint, ar**v: 1505.05424 (2015) https://doi.org/10.48550/ar**v.1505.05424
REZENDE, D. and MOHAMED, S. Variational inference with normalizing flows. International Conference on Machine Learning, 1530–1538 (2015)
ZOU, Z., MENG, X., PSAROS, A. F., and KARNIADAKIS, G. E. NeuralUQ: a comprehensive library for uncertainty quantification in neural differential equations and operators. ar**v Preprint, ar**v: 2208.11866 (2022) https://doi.org/10.48550/ar**v.2208.11866
DINH, L., SOHL-DICKSTEIN, J., and BENGIO, S. Density estimation using real NVP. ar**v Preprint, ar**v: 1605.08803 (2016) https://doi.org/10.48550/ar**v.1605.08803
PAPAMAKARIOS, G., PAVLAKOU, T., and MURRAY, I. Masked autoregressive flow for density estimation. 31 st Annual Conference on Neural Information Processing Systems, Long Beach, State of California (2017)
KINGMA, D. P., SALIMANS, T., JOZEFOWICZ, R., CHEN, X., SUTSKEVER, I., and WELLING, M. Improved variational inference with inverse autoregressive flow. 30th Conference on Neural Information Processing Systems, Barcelona, Spain (2016)
KINGMA, D. P. and DHARIWAL, P. Glow: generative flow with invertible 1 × 1 convolutions. 31st Annual Conference on Neural Information Processing Systems, Long Beach, State of California (2017)
LU, L., MENG, X., CAI, S., MAO, Z., GOSWAMI, S., ZHANG, Z., and KARNIADAKIS, G. E. A comprehensive and fair comparison of two neural operators (with practical extensions) based on fair data. Computer Methods in Applied Mechanics and Engineering, 393, 114778 (2022)
HOFFMAN, M. D. and GELMAN, A. The No-U-Turn sampler: adaptively setting path lengths in Hamiltonian Monte Carlo. Journal of Machine Learning Research, 15(1), 1593–1623 (2014)
LAO, J., SUTER, C., LANGMORE, I., CHIMISOV, C., SAXENA, A., SOUNTSOV, P., MOORE, D., SAUROUS, R. A., HOFFMAN, M. D., and DILLON, J. V. Tfp. mcmc: modern Markov chain Monte Carlo tools built for modern hardware. ar**v Preprint, ar**v: 2002.01184 https://doi.org/10.48550/ar**v.2002.01184
Acknowledgements
The author would like to acknowledge the support of the CCF-Baidu Open Fund and the **aomi Young Talents Program. The author also thanks Dr. Liu YANG for his helpful discussion.
Author information
Authors and Affiliations
Corresponding author
Additional information
Conflict of interest
The author declares no conflict of interest.
Citation: MENG, X. H. Variational inference in neural functional prior using normalizing flows: application to differential equation and operator learning problems. Applied Mathematics and Mechanics (English Edition), 44(7), 1111–1124 (2023) https://doi.org/10.1007/s10483-023-2997-7
Project supported by the National Natural Science Foundation of China (No. 12201229)
Rights and permissions
Open access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Meng, X. Variational inference in neural functional prior using normalizing flows: application to differential equation and operator learning problems. Appl. Math. Mech.-Engl. Ed. 44, 1111–1124 (2023). https://doi.org/10.1007/s10483-023-2997-7
Received:
Revised:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10483-023-2997-7
Key words
- uncertainty quantification (UQ)
- physics-informed neural network (PINN)
- deep operator network (DeepONet)
- generative adversarial network (GAN)
- normalizing flow (NF)
- differential equation