SpringerLink
Log in

A model validation framework based on parameter calibration under aleatory and epistemic uncertainty

  • Research Paper
  • Published:
Structural and Multidisciplinary Optimization Aims and scope Submit manuscript

Abstract

Model validation methods have been widely used in engineering design to evaluate the accuracy and reliability of simulation models with uncertain inputs. Most of the existing validation methods for aleatory and epistemic uncertainty are based on the Bayesian theorem, which needs a vast number of data to update the posterior distribution of the model parameter. However, when a single simulation is time-consuming, the required simulation cost for the validation of a simulation model may be unaffordable. To overcome this difficulty, a new model validation framework based on parameter calibration under aleatory and epistemic uncertainty is proposed. In the proposed method, a stochastic kriging model is constructed to predict the validity of the candidate simulation model under different uncertainty input parameters. Then, an optimization problem is defined to calibrate the epistemic uncertainty parameters to minimize the discrepancy between the simulation model and the experimental model. K–S test finally decides whether to accept or reject the calibrated simulation model. The performance of the proposed approach is illustrated through a cantilever beam example and a turbine blade validation problem. Results show that the proposed framework can identify the most appropriate parameters to calibrate the simulation model and provide a correct judgment about the validity of the candidate model, which is useful for the validation of simulation models in practical engineering design.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

References

  • An H, Chen S, Huang H (2018) Multi-objective optimization of a composite stiffened panel for hybrid design of stiffener layout and laminate stacking sequence. Struct Multidiscip Optim 57:1411–1426

    MathSciNet  Google Scholar 

  • Ankenman B, Nelson BL, Staum J (2010) Stochastic kriging for simulation metamodeling. Oper Res 58:371–382

    MathSciNet  MATH  Google Scholar 

  • Ao D, Hu Z, Mahadevan S (2017) Dynamics model validation using time-domain metrics. Journal of Verification, Validation and Uncertainty Quantification 2:011004

    Google Scholar 

  • Bauchau O, Craig J (2009) Euler-Bernoulli beam theory. In: Structural analysis. Springer, pp 173–221

  • Chen X, Kim K-K (2014) Stochastic kriging with biased sample estimates. ACM Transactions on Modeling and Computer Simulation (TOMACS) 24:8

    MathSciNet  MATH  Google Scholar 

  • Chen W, Baghdasaryan L, Buranathiti T, Cao J (2004) Model validation via uncertainty propagation and data transformations. AIAA J 42:1406–1415

    Google Scholar 

  • Chen X, Ankenman BE, Nelson BL (2013) Enhancing stochastic kriging metamodels with gradient estimators. Oper Res 61:512–528

    MathSciNet  MATH  Google Scholar 

  • L. Davis (1991) Handbook of genetic algorithms

    Google Scholar 

  • Deng W, Lu X, Deng Y (2018) Evidential model validation under epistemic uncertainty. Math Probl Eng 2018:1–11

    MathSciNet  Google Scholar 

  • Efron B, Tibshirani R (1997) Improvements on cross-validation: the 632+ bootstrap method. J Am Stat Assoc 92:548–560

    MathSciNet  MATH  Google Scholar 

  • Ferson S, Oberkampf WL, Ginzburg L (2008) Model validation and predictive capability for the thermal challenge problem. Comput Methods Appl Mech Eng 197:2408–2430

    MATH  Google Scholar 

  • Gorguluarslan RM, Choi S-K, Saldana CJ (2017) Uncertainty quantification and validation of 3D lattice scaffolds for computer-aided biomedical applications. J Mech Behav Biomed Mater 71:428–440

    Google Scholar 

  • Helton JC, Davis FJ (2003) Latin hypercube sampling and the propagation of uncertainty in analyses of complex systems. Reliability Engineering & System Safety 81:23–69

    Google Scholar 

  • Hu Z, Mahadevan S (2017) Uncertainty quantification and management in additive manufacturing: current status, needs, and opportunities. Int J Adv Manuf Technol 93:2855–2874

    Google Scholar 

  • Hu Z, Mahadevan S, Ao D (2018) Uncertainty aggregation and reduction in structure–material performance prediction. Comput Mech 61:237–257

    MathSciNet  MATH  Google Scholar 

  • Jiang C, Zheng J, Han X (2018) Probability-interval hybrid uncertainty analysis for structures with both aleatory and epistemic uncertainties: a review. Struct Multidiscip Optim 57:2485–2502

    Google Scholar 

  • Jung BC, Park J, Oh H, Kim J, Youn BD (2015) A framework of model validation and virtual product qualification with limited experimental data based on statistical inference. Struct Multidiscip Optim 51:573–583

  • Kennedy MC, O’Hagan A (2001) Bayesian calibration of computer models. Journal of the Royal Statistical Society: Series B (Statistical Methodology) 63:425–464

    MathSciNet  MATH  Google Scholar 

  • Lee G, Kim W, Oh H, Youn BD, Kim NH (2019) Review of statistical model calibration and validation—from the perspective of uncertainty structures. Struct Multidiscip Optim 60:1619–1644

    MathSciNet  Google Scholar 

  • Li C, Mahadevan S (2016) Role of calibration, validation, and relevance in multi-level uncertainty integration. Reliability Engineering & System Safety 148:32–43

    Google Scholar 

  • Li W, Chen W, Jiang Z, Lu Z, Liu Y (2014) New validation metrics for models with multiple correlated responses. Reliability Eng Syst Saf 127:1–11

    Google Scholar 

  • Ling Y, Mahadevan S (2013) Quantitative model validation techniques: new insights. Reliability Engineering & System Safety 111:217–231

    Google Scholar 

  • Liu Y, Shi Y, Zhou Q, **u R (2016) A sequential sampling strategy to improve the global fidelity of metamodels in multi-level system design. Struct Multidiscip Optim 53:1295–1313

    MathSciNet  Google Scholar 

  • Lü H, Shangguan W-B, Yu D (2018) A unified method and its application to brake instability analysis involving different types of epistemic uncertainties. Appl Math Model 56:158–171

    MathSciNet  MATH  Google Scholar 

  • Marsaglia G, Tsang WW, Wang J (2003) Evaluating Kolmogorov’s distribution. J Stat Softw 8:1–4

    Google Scholar 

  • Massey FJ Jr (1951) The Kolmogorov-Smirnov test for goodness of fit. J Am Stat Assoc 46:68–78

    MATH  Google Scholar 

  • McKeand AM, Gorguluarslan RM, Choi S-K (2018) A stochastic approach for performance prediction of aircraft engine components under manufacturing uncertainty. In: ASME 2018 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference. American Society of Mechanical Engineers Digital Collection, Quebec City, Quebec, Canada

    Google Scholar 

  • Moon M-Y, Choi K, Cho H, Gaul N, Lamb D, Gorsich D (2017) Reliability-based design optimization using confidence-based model validation for insufficient experimental data. J Mech Des 139:031404

    Google Scholar 

  • R.T. Muehleisen, J. Bergerson (2016) Bayesian calibration-what, Why And How

    Google Scholar 

  • Mullins J, Mahadevan S (2016) Bayesian uncertainty integration for model calibration, validation, and prediction. Journal of Verification, Validation and Uncertainty Quantification 1:011006

    Google Scholar 

  • Mullins J, Ling Y, Mahadevan S, Sun L, Strachan A (2016) Separation of aleatory and epistemic uncertainty in probabilistic model validation. Reliability Eng Syst Saf 147:49–59

    Google Scholar 

  • Oh H, Kim J, Son H, Youn BD, Jung BC (2016) A systematic approach for model refinement considering blind and recognized uncertainties in engineered product development. Struct Multidiscip Optim 54:1527–1541

    Google Scholar 

  • Peacock JA (1983) Two-dimensional goodness-of-fit testing in astronomy. Mon Not R Astron Soc 202:615–627

    Google Scholar 

  • Qian X, Li W, Yang M (2016) Two-stage nested optimization-based uncertainty propagation method for model calibration. International Journal of Modeling, Simulation, and Scientific Computing 7:1541003

    Google Scholar 

  • Qian J, Yi J, Cheng Y, Liu J, Zhou Q (2020) A sequential constraints updating approach for Kriging surrogate model-assisted engineering optimization design problem. Eng Comput 36:993–1009

  • Rebba R, Mahadevan S (2006) Model predictive capability assessment under uncertainty. AIAA J 44:2376–2384

    Google Scholar 

  • Rebba R, Mahadevan S (2008) Computational methods for model reliability assessment. Reliability Engineering & System Safety 93:1197–1207

    Google Scholar 

  • Ruan X, Zhou Q, Shu L, Hu J, Cao L (2018) Accurate prediction of the weld bead characteristic in laser keyhole welding based on the stochastic Kriging model. Metals 8:486

    Google Scholar 

  • Ruan X, Jiang P, Zhou Q, Hu J, Shu L (2020) Variable-fidelity probability of improvement method for efficient global optimization of expensive black-box problems. Struct Multidiscip Optim:1–32

  • Sankararaman S, Mahadevan S (2011a) Likelihood-based representation of epistemic uncertainty due to sparse point data and/or interval data. Reliability Engineering & System Safety 96:814–824

    Google Scholar 

  • Sankararaman S, Mahadevan S (2011b) Model validation under epistemic uncertainty. Reliability Engineering & System Safety 96:1232–1241

    Google Scholar 

  • Sankararaman S, Mahadevan S (2015) Integration of model verification, validation, and calibration for uncertainty quantification in engineering systems. Reliabil Eng Syst Saf 138:194–209

    MATH  Google Scholar 

  • Sargent RG (2010) Verification and validation of simulation models. In: Proceedings of the 2010 Winter Simulation Conference, IEEE, pp 166–183

    Google Scholar 

  • Shen Z, Chen X, He Q, Zang CP (2015) Study on area metric based upon multiple correlated system response quantities. In: SAE Technical Paper

    Google Scholar 

  • Staum J (2009) Better simulation metamodeling: the why, what, and how of stochastic kriging. In: Proceedings of the 2009 Winter Simulation Conference (WSC). IEEE, pp 119–133

  • Wang N, Yao W, Zhao Y, Chen X, Zhang X, Li L (2018a) A new interval area metric for model validation with limited experimental data. J Mech Des 140:061403

    Google Scholar 

  • Wang C, Matthies HG, Xu M, Li Y (2018b) Epistemic uncertainty-based model validation via interval propagation and parameter calibration. Comput Methods Appl Mech Eng 342:161–176

    MathSciNet  MATH  Google Scholar 

  • **ong Y, Chen W, Tsui K-L, Apley DW (2009) A better understanding of model updating strategies in validating engineering models. Comput Methods Appl Mech Eng 198:1327–1337

    MATH  Google Scholar 

  • Youn BD, ** Z, Wang P (2008) Eigenvector dimension reduction (EDR) method for sensitivity-free probability analysis. Struct Multidiscip Optim 37:13–28

    MathSciNet  MATH  Google Scholar 

  • Youn BD, Jung BC, ** Z, Kim SB, Lee W (2011) A hierarchical framework for statistical model calibration in engineering product development. Comput Methods Appl Mech Eng 200:1421–1431

    MATH  Google Scholar 

  • Zhang R, Wang Z (1996) Uniform design sampling and its fine properties. Chin J Appl Probab Stat 12:337–347

    MATH  Google Scholar 

  • Zou L, Zhang X (2018) Stochastic kriging for inadequate simulation models. ar**v preprint ar**v:1802.00677

Download references

Funding

This research has been supported by the National Natural Science Foundation of China (NSFC) under Grant No. 51805179, No. 51775203, and No. 51721092, the National Defense Innovation Program under Grant No. 18-163-00-TS-004-033-01, the research fund under Grant No.61400020401, the Research Funds of the Maritime Defense Technologies Innovation under Grant YT19201901, and the China Scholarship Council with a Scholarship (No. 201706160153).

Author information

Authors and Affiliations

  1. School of Aerospace Engineering, Huazhong University of Science & Technology, Wuhan, 430074, People’s Republic of China

    Jiexiang Hu & Qi Zhou

  2. School of Material Science and Engineering, Huazhong University of Science & Technology, Wuhan, 430074, People’s Republic of China

    Jiexiang Hu

  3. George W. Woodruff School of Mechanical Engineering, Georgia Institute of Technology, Atlanta, GA, 30332, USA

    Austin McKeand, Tingli **e & Seung-Kyum Choi

Authors
  1. Jiexiang Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Qi Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Austin McKeand
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Tingli **e
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Seung-Kyum Choi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Qi Zhou.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Replication of results

The main step for applying the validation framework has been presented in Section 3. To help readers understand better, the validation code in Section 4.1 has been attached to the supplementary material.

Additional information

Responsible Editor: Nam Ho Kim

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

ESM 1

(ZIP 10.3 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hu, J., Zhou, Q., McKeand, A. et al. A model validation framework based on parameter calibration under aleatory and epistemic uncertainty. Struct Multidisc Optim 63, 645–660 (2021). https://doi.org/10.1007/s00158-020-02715-z

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00158-020-02715-z

Keywords

Search

Navigation