Abstract
The efficient use of High Performance Computing (HPC) resources is essential and requires continuous performance monitoring. The NHR HPC Center at TU Dresden has been using PIKA [5], a continuous job-level performance monitoring system, for more than five years. It is active by default and allows retrospective analysis and comparison with previous jobs. Its results are available to users for their jobs as well as to admins and HPC support for all jobs. It has proven to be very useful for reactive user support on various aspects of efficient use of HPC resources in general as well as on specific performance issues of individual users.
At the same time, the continuously collected data can be scanned proactively, as users may not yet be aware of performance issues. In this article, we report on our methods for scanning for job inefficiencies, and to inspire discussion about appropriate methods. It covers the most useful heuristic checks for a variety of aspects. We focus on meaningful performance criteria and commonly observed performance problems. All parameters and thresholds are derived from experience and tuned to detect the most severe cases in the average job mix. The heuristics range from simple cases that compare against appropriate thresholds to more sophisticated tests with some pre-processing.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
InfluxDB: Scalable datastore for metrics, events, and real-time analytics. https://github.com/influxdata/influxdb. Accessed February 2023
NumPy: The fundamental package for scientific computing with Python. https://numpy.org/. Accessed February 2023
SciPy: Fundamental algorithms for scientific computing in Python. https://scipy.org/. Accessed February 2023
Agrawal, K., Fahey, M.R., McLay, R., James, D.: User environment tracking and problem detection with XALT. In: 2014 First International Workshop on HPC User Support Tools, pp. 32–40 (2014). https://doi.org/10.1109/HUST.2014.6
Dietrich, R., Winkler, F., Knüpfer, A., Nagel, W.: PIKA: center-wide and job-aware cluster monitoring. In: Workshop on Monitoring and Analysis for High Performance Computing Systems Plus Applications. HPCMASPA (2020). https://doi.org/10.1109/CLUSTER49012.2020.00061
Eitzinger, J., Gruber, T., Afzal, A., Zeiser, T., Wellein, G.: ClusterCockpit—a web application for job-specific performance monitoring. In: 2019 IEEE International Conference on Cluster Computing (CLUSTER), pp. 1–7 (2019). https://doi.org/10.1109/CLUSTER.2019.8891017
Evans, T., et al.: Comprehensive resource use monitoring for HPC systems with TACC stats. In: 2014 First International Workshop on HPC User Support Tools, pp. 13–21 (2014). https://doi.org/10.1109/HUST.2014.7
Fischer, F.: Metrics for job similarity based on hardware performance data. Master’s thesis, Technische Universität München (2020)
**dal, A., Staab, P., Kulkarni, P., Cardoso, J., Gerndt, M., Podolskiy, V.: Memory leak detection algorithms in the cloud-based infrastructure. CoRR abs/2106.08938 (2021). https://arxiv.org/abs/2106.08938
Multani, M.: Statistical characterization of HPC monitoring data (2021)
Ozer, G., Netti, A., Tafani, D., Schulz, M.: Characterizing HPC performance variation with monitoring and unsupervised learning. In: Jagode, H., Anzt, H., Juckeland, G., Ltaief, H. (eds.) ISC High Performance 2020. LNCS, vol. 12321, pp. 280–292. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-59851-8_18
Röhl, T., Eitzinger, J., Hager, G., Wellein, G.: LIKWID monitoring stack: a flexible framework enabling job specific performance monitoring for the masses. In: 2017 IEEE International Conference on Cluster Computing (CLUSTER), pp. 781–784 (2017). https://doi.org/10.1109/CLUSTER.2017.115
Roigk, J.: Feasibility study for detecting different job stages using a system monitoring daemon. Master’s thesis (2022)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Winkler, F., Knüpfer, A. (2023). Automatic Detection of HPC Job Inefficiencies at TU Dresden’s HPC Center with PIKA. In: Bienz, A., Weiland, M., Baboulin, M., Kruse, C. (eds) High Performance Computing. ISC High Performance 2023. Lecture Notes in Computer Science, vol 13999. Springer, Cham. https://doi.org/10.1007/978-3-031-40843-4_22
Download citation
DOI: https://doi.org/10.1007/978-3-031-40843-4_22
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-40842-7
Online ISBN: 978-3-031-40843-4
eBook Packages: Computer ScienceComputer Science (R0)