Abstract
Efficient execution of large-scale and extremely demanding computational scenarios is a challenge for both the infrastructure providers and end-users, usually scientists, that need to develop highly scalable computational codes. Nevertheless, at this time, on the eve of exa-scale supercomputers, the particular role has to be given also to the intermediate software that can help in the preparation of applications so they can be efficiently executed on the emerging HPC systems. The efficiency and scalability of such software can be seen as priorities, however, these are not the only elements that should be addressed. Equally important is to offer software that is elastic, portable between platforms of different sizes, and easy to use. Trying to fulfill all the above needs we present QCG-PilotJob, a tool designed to enable flexible execution of numerous potentially dynamic and interdependent computing tasks in a single allocation on a computing cluster. QCG-PilotJob is built on many years of collaboration with computational scientists representing various domains and it responses to the practical requirements of real scientific use-cases. In this paper, we focus on the recent integration of QCG-PilotJob with the EasyVVUQ library and its successful use for Uncertainty Quantification workflows of several complex multiscale applications being developed within the VECMA project. However, we believe that with a well-thought-out design that allows for fully user-space execution and straightforward installation, QCG-PilotJob may be easily exploited in many other application scenarios, even by inexperienced users.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
- 2.
- 3.
- 4.
- 5.
- 6.
- 7.
- 8.
References
Babuji, Y., et al.: Parsl: pervasive parallel programming in python. In: Proceedings of the 28th International Symposium on High-Performance Parallel and Distributed Computing, pp. 25–36. HPDC 2019, New York, NY, USA (2019). https://doi.org/10.1145/3307681.3325400
Cabellos, L., Campos, I., del Castillo, E.F., Owsiak, M., Palak, B., Płóciennik, M.: Scientific workflow orchestration interoperating HTC and HPC resources. Comput. Phys. Commun. 182(4), 890–897 (2011). https://doi.org/10.1016/j.cpc.2010.12.020
Edeling, W., et al.: The impact of uncertainty on predictions of the CovidSim epidemiological code. Nat. Comput. Sci. 1(2), 128–135 (2021). https://doi.org/10.1038/s43588-021-00028-9
Groen, D., et al.: VECMAtk: a scalable verification, validation and uncertainty quantification toolkit for scientific simulations. Philos. Trans. R. Soc. A: Mathe. Phys. Eng. Sci. 379(2197), 20200221 (2021). https://doi.org/10.1098/rsta.2020.0221
Luk, O., Hoenen, O., Bottino, A., Scott, B., Coster, D.: Compat framework for multiscale simulations applied to fusion plasmas. Comput. Phys. Commun. (2019). https://doi.org/10.1016/j.cpc.2018.12.021
Merzky, A., Turilli, M., Titov, M., Al-Saadi, A., Jha, S.: Design and performance characterization of RADICAL-PILOT on leadership-class platforms (2021)
Piontek, T., et al.: Development of science gateways using QCG – lessons learned from the deployment on large scale distributed and HPC infrastructures. J. Grid Comput. 14, 559–573 (2016)
Richardson, R., Wright, D., Edeling, W., Jancauskas, V., Lakhlili, J., Coveney, P.: EasyVVUQ: a library for verification, validation and uncertainty quantification in high performance computing. J. Open Res. Softw. 8, 1–8 (2020)
Wozniak, J.M., Armstrong, T.G., Wilde, M., Katz, D.S., Lusk, E., Foster, I.T.: Swift/t: large-scale application composition via distributed-memory dataflow processing. In: 2013 13th IEEE/ACM International Symposium on Cluster, Cloud, and Grid Computing, pp. 95–102 (2013). https://doi.org/10.1109/CCGrid.2013.99
Wright, D.W., et al.: Building confidence in simulation: applications of EasyVVUQ. Adv. Theory Simul. 3(8), 1900246 (2020)
Acknowledgments
This work received funding from the VECMA project realised under grant agreement 800925 of the European Union’s Horizon 2020 research and innovation programme. We are thankful to the Poznan Supercomputing and Networking Center for providing its computational infrastructure. We are also grateful to the VECMA partners for the invaluable motivation.
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Bosak, B., Piontek, T., Karlshoefer, P., Raffin, E., Lakhlili, J., Kopta, P. (2021). Verification, Validation and Uncertainty Quantification of Large-Scale Applications with QCG-PilotJob. In: Paszynski, M., Kranzlmüller, D., Krzhizhanovskaya, V.V., Dongarra, J.J., Sloot, P.M. (eds) Computational Science – ICCS 2021. ICCS 2021. Lecture Notes in Computer Science(), vol 12746. Springer, Cham. https://doi.org/10.1007/978-3-030-77977-1_39
Download citation
DOI: https://doi.org/10.1007/978-3-030-77977-1_39
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-77976-4
Online ISBN: 978-3-030-77977-1
eBook Packages: Computer ScienceComputer Science (R0)