SpringerLink
Log in

Evaluation and comparison of CMIP6 and CMIP5 model performance in simulating the runoff

  • Original Paper
  • Published:
Theoretical and Applied Climatology Aims and scope Submit manuscript

Abstract

This study evaluates and compares the performance of Coupled Model Intercomparison Project Phase 6 (CMIP6) and CMIP5 in simulating the runoff on global-scale and eight large-scale basins, over the period 1981–2005 using percent bias (PBIAS), correlation coefficient (CC), root-mean-square error (RMSE), Theil-Sen median trend, and the Taylor diagram. The CMIP models are ranked by comprehensive rating index (MR), which is determined by PBIAS, CC, and RMSE three metrics. Linear Optimal Runoff Aggregate (LORA), Global Runoff Reconstruction (GRUN), and ERA5-Land were selected as reference datasets. LORA was used as the main reference data to evaluate the historical runoff results of CMIP from 1981 to 2012 for three aspects: trend, PBIAS, and uncertainty. Results reveal that (i) CMIP6 models have obviously overvalued on the global and basins (except Amazon and Lena basin); this phenomenon was more prominent in arid and semi-arid areas (Murray-Darling and Nile basin). (ii) Compared with CMIP5 models, CMIP6 models have less uncertainty on the global scale, but it has not made outstanding progress on the basin scale. (iii) CMIP6 multi-model ensemble mean (CMIP6_MMEs) has better simulation effect than most individual models, which reduces the uncertainty among different models to some extent. (iv) There were differences in trends and PBIAS between the three reference datasets at both the global and basin scale. However, the interannual fluctuations of the three datasets were basically the same and have high correlation coefficient (except for ERA5 in the world and Nile basin), which shows that LORA dataset has high reliability. The global comprehensive rating metric (GR) of CMIP6_MMEs was better than CMIP5_MMEs in all metrics, but this result was not found in eight basins. This shows that CMIP6 models has better effect in simulating global runoff and related diagnostic indicators. Implying further improvements are needs for the runoff simulation capability at the basin scale.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Data availability

The datasets used in the present study are freely available: (i) The CMIP6 and CMIP5 datasets from https://esgf-node.llnl.gov/search/ (Eyring et al. 2016a, b). (ii) The LORA dataset is freely available for download on https://geonetwork.nci.org.au/geonetwork/srv/eng/catalog.search#/metadata/f9617_9854_8096_5291 (Hobeichi et al. 2019). (iii) The GRUN dataset is available from the ETHZ Research Collection at https://doi.org/10.3929/ethz-b-000324386 (Ghiggi et al. 2019). (iv) The ERA5-Land reanalysis datasets from https://www.ecmwf.int/en/era5-land (C3S 2019).

Code availability

Not applicable.

References

  • Abramowitz G et al (2019) ESD Reviews: model dependence in multi-model climate ensembles: weighting, sub-selection and out-of-sample testing, Earth System. Dynamics 10:91–105

    Google Scholar 

  • Adnan M, Nabi G, Saleem Poomee M, Ashraf A (2017) Snowmelt runoff prediction under changing climate in the Himalayan cryosphere: a case of Gilgit River Basin. Geosci Front 8:941–949

    Article  Google Scholar 

  • Alkama R, Marchand L, Ribes A, Decharme B (2013) Detection of global runoff changes: results from observations and CMIP5 experiments. Hydrol Earth Syst Sci 17:2967–2979

    Article  Google Scholar 

  • Beck HE et al (2017) MSWEP: 3-hourly 0.25° global gridded precipitation (1979–2015) by merging gauge, satellite, and reanalysis data. Hydrol Syst Sci 21:589–615

    Article  Google Scholar 

  • Beck HE et al (2018) Present and future Koppen-Geiger climate classification maps at 1-km resolution. Sci Data 5:180214

    Article  Google Scholar 

  • C3S ERA5-land Reanalysis. Copernicus Climate Change Service date of access December, 2019, https://cds.climate.copernicus.eu/cdsapp#!/home (2019).

  • Chen H, Xu C-Y, Guo S (2012) Comparison and evaluation of multiple GCMs, statistical downscaling and hydrological models in the study of climate change impacts on runoff. J Hydrol 434–435:36–45

    Article  Google Scholar 

  • Coppola E et al (2021) Climate hazard indices projections based on CORDEX-CORE, CMIP5 and CMIP6 ensemble, Clim Dyn.

  • Dankers R, Kundzewicz ZW (2020) Grappling with uncertainties in physical climate impact projections of water resources. Clim Change 163:1379–1397

    Article  Google Scholar 

  • Deb P, Kiem AS, Willgoose G (2019) A linked surface water-groundwater modelling approach to more realistically simulate rainfall-runoff non-stationarity in semi-arid regions. J Hydrol 575:273–291

    Article  Google Scholar 

  • Dobrovolski SG, Yushkov VP, Istomina MN (2019) Statistical modeling of the global river runoff using GCMs: comparison with the observational data and reanalysis results. Water Resour 46:S17–S24

    Article  Google Scholar 

  • Espinoza Villar JC et al (2009) Contrasting regional discharge evolutions in the Amazon basin (1974–2004). J Hydrol 375:297–311

    Article  Google Scholar 

  • Evans J, Abramowitz G, Hobeichi S (2020) Conserving Land-Atmosphere Synthesis Suite (CLASS). J Clim 33:1821–1844

    Article  Google Scholar 

  • Eyring V et al (2016) Overview of the Coupled Model Intercomparison Project Phase 6 (CMIP6) experimental design and organization. Geosci Model Dev 9:1937–1958

    Article  Google Scholar 

  • Eyring V et al (2016) Towards improved and more routine Earth system model evaluation in CMIP, Earth System. Dynamics 7:813–830

    Google Scholar 

  • Fiedler S et al (2020) Simulated tropical precipitation assessed across three major phases of the Coupled Model Intercomparison Project (CMIP). Mon Weather Rev 148:3653–3680

    Article  Google Scholar 

  • Gain AK, Apel H, Renaud FG, Giupponi C (2013) Thresholds of hydrologic flow regime of a river and investigation of climate change impact—the case of the Lower Brahmaputra river Basin. Clim Change 120:463–475

    Article  Google Scholar 

  • Gao HK et al (2021) Assessing glacier retreat and its impact on water resources in a headwater of Yangtze River based on CMIP6 projections, Sci Total Environ 765.

  • Gelfan A et al (2020) Does a successful comprehensive evaluation increase confidence in a hydrological model intended for climate impact assessment? Clim Change 163:1165–1185

    Article  Google Scholar 

  • Gettelman A et al (2019) High climate sensitivity in the Community Earth System Model Version 2 (CESM2). Geophys Res Lett 46:8329–8337

    Article  Google Scholar 

  • Ghiggi G, Humphrey V, Seneviratne SI, Gudmundsson L (2019) GRUN: an observation-based global gridded runoff dataset from 1902 to 2014. Earth Syst Sci Data 11:1655–1674

    Article  Google Scholar 

  • Giuntoli I, Vidal JP, Prudhomme C, Hannah DM (2015) Future hydrological extremes: the uncertainty from multiple global climate and global hydrological models, Earth System. Dynamics 6:267–285

    Google Scholar 

  • Gosling SN, Arnell NW (2013) A global assessment of the impact of climate change on water scarcity. Clim Change 134:371–385

    Article  Google Scholar 

  • Gudmundsson L, Seneviratne SI (2015) Towards observation-based gridded runoff estimates for Europe. Hydrol Earth Syst Sci 19:2859–2879

    Article  Google Scholar 

  • Hersbach H (2000) Decomposition of the continuous ranked probability score for ensemble prediction systems. Weather Forecast 15:559–570

    Article  Google Scholar 

  • Hobeichi S, Abramowitz G, Evans J, Beck HE (2019) Linear Optimal Runoff Aggregate (LORA): a global gridded synthesis runoff product. Hydrol Earth Syst Sci 23:851–870

    Article  Google Scholar 

  • Jahfer S, Vinayachandran PN, Nanjundiah RS (2017) Long-term impact of Amazon river runoff on northern hemispheric climate. Sci Rep 7:10989

    Article  Google Scholar 

  • Jiang Z, Li W, Xu J, Li L (2015) Extreme precipitation indices over China in CMIP5 models. Part i: Model Eval, J Clim 28:8603–8619

    Google Scholar 

  • Kendall MG (1975) Rank Correlation Methods, 4th ed.; Charless Grin: London, UK, ISBN 0195208374

  • Kim MK et al (2020) Performance evaluation of CMIP5 and CMIP6 models on heatwaves in Korea and associated teleconnection Patterns, J Geophys Res: Atmos 125.

  • Kling H, Stanzel P, Fuchs M, Nachtnebel H-P (2015) Performance of the COSERO precipitation–runoff model under non-stationary conditions in basins with different climates. Hydrol Sci J 60:1374–1393

    Article  Google Scholar 

  • Knutti R et al (2017) A climate model projection weighting scheme accounting for performance and interdependence, Geophys Res Lett.

  • Knutti R, Sedláček J (2012) Robustness and uncertainties in the new CMIP5 climate model projections, Nature. Clim Change 3:369–373

    Google Scholar 

  • Kooperman GJ et al (2018) Plant physiological responses to rising CO2 modify simulated daily runoff intensity with implications for global-scale flood risk assessment, Geophys Res Lett 45.

  • Krysanova V et al (2017) Intercomparison of regional-scale hydrological models and climate change impacts projected for 12 large river basins worldwide—a synthesis, Environ Res Lett 12.

  • Kumar S et al (2016) Terrestrial contribution to the heterogeneity in hydrological changes under global warming. Water Resour Res 52:3127–3142

    Article  Google Scholar 

  • Lehner F et al (2020) Partitioning climate projection uncertainty with multiple large ensembles and CMIP5/6, Earth System. Dynamics 11:491–508

    Google Scholar 

  • Levizzani V, Cattani E (2019) Satellite remote sensing of precipitation and the terrestrial water cycle in a changing climate, Remote Sens 11.

  • Lloyd EA (2012) The role of ‘complex’ empiricism in the debates about satellite data and climate models. Stud Hist Philos Sci 43:390–401

    Article  Google Scholar 

  • Lutz AF, Immerzeel WW, Shrestha AB, Bierkens MFP (2014) Consistent increase in High Asia’s runoff due to increasing glacier melt and precipitation, Nature. Clim Change 4:587–592

    Google Scholar 

  • Mann HB (1945) Nonparametric tests against trend. Econometrica 13:245–259

    Article  Google Scholar 

  • Massonnet F et al (2012) Constraining projections of summer Arctic sea ice. Cryosphere 6:1383–1394

    Article  Google Scholar 

  • Massoud EC, Espinoza V, Guan B, Waliser DE (2019) Global climate model ensemble approaches for future projections of atmospheric rivers, Earth’s. Future 7:1136–1151

    Google Scholar 

  • Mauritsen T et al (2019) Developments in the MPI-M Earth System Model version 1.2 (MPI-ESM1.2) and its response to increasing CO2. J Adv Model Earth Syst 11:998–1038

    Article  Google Scholar 

  • Milly PC, Dunne KA, Vecchia AV (2005) Global pattern of trends in streamflow and water availability in a changing climate. Nature 438:347–350

    Article  Google Scholar 

  • Mockler EM, Chun KP, Sapriza-Azuri G, Bruen M, Wheater HS (2016) Assessing the relative importance of parameter and forcing uncertainty and their interactions in conceptual hydrological model simulations. Adv Water Resour 97:299–313

    Article  Google Scholar 

  • Moriasi DN et al (2007) Model evaluation guidelines for systematic quantification of accuracy in watershed simulations. Trans ASABE 50:885–900

    Article  Google Scholar 

  • Padrón RS et al (2020) Observed changes in dry-season water availability attributed to human-induced climate change. Nat Geosci 13:477–481

    Article  Google Scholar 

  • Pokorny S et al (2021) Cumulative effects of uncertainty on simulated streamflow in a hydrologic modeling environment, Elementa-Sci Anthropocene 9.

  • Sarmadi F, Huang Y, Thompson G, Siems ST, Manton MJ (2019) Simulations of orographic precipitation in the Snowy Mountains of Southeastern Australia. Atmos Res 219:183–199

    Article  Google Scholar 

  • Schewe J et al (2014) Multimodel assessment of water scarcity under climate change. Proc Natl Acad Sci U S A 111:3245–3250

    Article  Google Scholar 

  • Seibert J, Beven KJ (2009) Gauging the ungauged basin: how many discharge measurements are needed? Hydrol Earth Syst Sci 13:883–892

    Article  Google Scholar 

  • Sharma, Ojha (2019) Changes of annual precipitation and probability distributions for different climate types of the world, Water 11.

  • Simpkins G (2017) Progress in climate modelling, Nature. Clim Change 7:684–685

    Google Scholar 

  • Taylor KE (2001) Summarizing multiple aspects of model performance in a single diagram. J Geophys Res: Atmos 106:7183–7192

    Article  Google Scholar 

  • Teklesadik AD et al (2017) Inter-model comparison of hydrological impacts of climate change on the Upper Blue Nile basin using ensemble of hydrological models and global climate models. Clim Change 141:517–532

    Article  Google Scholar 

  • Travis KR et al (2016) Why do models overestimate surface ozone in the Southeastern United States?, Atmos. Chem Phys 16:13561–13577

    Google Scholar 

  • Vaze J et al (2010) Climate non-stationarity — validity of calibrated rainfall–runoff models for use in climate change studies. J Hydrol 394:447–457

    Article  Google Scholar 

  • Waliser D et al (2020) Observations for Model Intercomparison Project (Obs4MIPs): status for CMIP6. Geosci Model Dev 13:2945–2958

    Article  Google Scholar 

  • Wang G, Dommenget D, Frauen C (2014) An evaluation of the CMIP3 and CMIP5 simulations in their skill of simulating the spatial structure of SST variability. Clim Dyn 44:95–114

    Article  Google Scholar 

  • Wang Z, Zhan C, Ning L, Guo H (2020) Evaluation of global terrestrial evapotranspiration in CMIP6 models. Theoret Appl Climatol 143:521–531

    Article  Google Scholar 

  • Watterson IG (2015) Improved simulation of regional climate by global models with higher resolution: skill scores correlated with grid length*. J Clim 28:5985–6000

    Article  Google Scholar 

  • Wen X et al (2018) Future changes in Yuan River ecohydrology: individual and cumulative impacts of climates change and cascade hydropower development on runoff and aquatic habitat quality. Sci Total Environ 633:1403–1417

    Article  Google Scholar 

  • Wen X et al (2019) Two-phase extreme learning machines integrated with the complete ensemble empirical mode decomposition with adaptive noise algorithm for multi-scale runoff prediction problems. J Hydrol 570:167–184

    Article  Google Scholar 

  • Xu Z, Hou Z, Han Y, Guo W (2016) A diagram for evaluating multiple aspects of model performance in simulating vector fields. Geosci Model Dev 9:4365–4380

    Article  Google Scholar 

  • Yin JB et al (2021) Does the hook structure constrain future flood intensification under anthropogenic climate warming?, Water Resour Res 57.

  • Zhang Y, Shao Q, Zhang S, Zhai X, She D (2016) Multi-metric calibration of hydrological model to capture overall flow regimes. J Hydrol 539:525–538

    Article  Google Scholar 

  • Zhou X et al (2012) Benchmarking global land surface models against the observed mean annual runoff from 150 large basins. J Hydrol 470–471:269–279

    Article  Google Scholar 

  • Zhou RR, Li Y, Lu D, Liu HX, Zhou HC (2016) An optimization based sampling approach for multiple metrics uncertainty analysis using generalized likelihood uncertainty estimation. J Hydrol 540:274–286

    Article  Google Scholar 

  • Zumwald M et al (2020) Understanding and assessing uncertainty of observational climate datasets for model evaluation using ensembles, WIREs Clim Change 11.

Download references

Acknowledgements

We acknowledge Climate Change Research Centre, University of New South Wales, for the data from LORA, ETH Zurich, for the data from GRUN, European Centre for Medium-Range Weather Forecasts (ECMWF) for the reanalysis dataset from ERA5-land. We also gratefully acknowledge the World Climate Research Programme’s Working Group on Coupled Modelling, which is responsible for CMIP, and we thank the climate modeling groups for producing and making available their model output, the Earth System Grid Federation (ESGF) for archiving the data and providing access, and the multiple funding agencies who support CMIP6 and ESGF. We sincerely appreciate the anonymous reviewers’ helpful comments and the editor’s efforts in improving this manuscript.

Funding

This research was funded by the National Key R&D Program of China (grant number 2017YFA0603702), the National Natural Science Foundation of China (grant number 41701023), and the Natural Science Foundation of China (grant number 41971232).

Author information

Authors and Affiliations

  1. Key Laboratory of Ecosystem Network Observation and Modeling, Institute of Geographic Sciences and Natural Resources Research, Chinese Academy of Sciences, Bei**g, 100101, China

    Hai Guo, Chesheng Zhan, Like Ning & Zhonghe Li

  2. University of Chinese Academy of Sciences, Bei**g, 100049, China

    Hai Guo & Zhonghe Li

  3. Key Laboratory of Water Cycle and Related Land Surface Processes, Institute of Geographic Sciences and Natural Resources Research, Chinese Academy of Sciences, Bei**g, 100101, China

    Shi Hu

Authors
  1. Hai Guo
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Chesheng Zhan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Like Ning
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Zhonghe Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Shi Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All authors contributed to the study conception and design. Hai Guo: data curation, formal analysis, visualization, software, writing—original draft preparation. Zhonghe Li: visualization. Like Ning: conceptualization, methodology, writing—reviewing and editing. Shi Hu: revision of the article. Chesheng Zhan: supervision. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Chesheng Zhan.

Ethics declarations

Ethics approval

This research did not involve human subjects. Meteorological datasets used in this study can all be obtained from publicly accessible archives.

Consent to participate

This research did not involve human subjects.

Consent for publication

This research did not involve personal information for which consent was to be sought.

Conflict of interest

The authors declare no competing interests.

Additional information

Publisher's note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendices

Appendix 1

Table 4 Model names, institution, and resolution for CMIP6 models used in the paper
Full size table
Table 5 CMIP5 models used in the paper, details are the same as Table 4
Full size table

Appendix 2 Model ranking in basin

Fig. 11
figure 11

The portrait diagram for the rankings of PBIAS, CC, and RMSE. Upper panel is the CMIP6 model, the middle panel is the CMIP5 model, and the bottom panel is reference dataset. The AR, LR, MEKR, MISR, M-DR, NR, RR, and YR are the comprehensive rating metrics of Amazon, Lena, Mekong, Mississippi, Murray-Darling, Nile, Rhine, and Yangtze basin, respectively. The basin comprehensive rating metrics (BR) is the comprehensive ranking of three indicators in eight basins

Full size image

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Guo, H., Zhan, C., Ning, L. et al. Evaluation and comparison of CMIP6 and CMIP5 model performance in simulating the runoff. Theor Appl Climatol 149, 1451–1470 (2022). https://doi.org/10.1007/s00704-022-04118-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00704-022-04118-0

Search

Navigation