Using Resource Use Data and System Logs for HPC System Error Propagation and Recovery Diagnosis

Thuan Chuah, Arshad Jhumka, Samantha Alt, JJ Villalobos, Josh Fryman, Bill Barth, Manish Parashar

Research output: Chapter in Book/Report/Conference proceedingPublished conference contribution

4 Citations (Scopus)


Analyzing failures is important for the reliability of HPC systems and failure diagnosis based only on system logs is incomplete. Resource use data - made available recently - is another potential source of data for failure analysis. Recent work that combines analysis of system logs with resource use data show promising results. In this paper, we describe a new workflow for combining system resource usage and failure logs for diagnosis. The workflow - called EXERMEST - identifies significant system counters and events then correlates them to failures and recovery. We apply EXERMEST on the Ranger HPC system cluster log-data and show that it improves diagnosis over previous research. EXERMEST: (i) show that more system counters and errors can be identified only by applying more feature extractors, (ii) identify CPU I/O bottlenecks and Lustre client eviction, (iii) identify network packet drops and Lustre I/O errors, (iv) identify virtual memory and harddisk I/O errors, (v) show that time-bins of different granularities are required for identifying the errors. EXERMEST is available on the public domain for supporting system administrators in failure diagnosis.
Original languageEnglish
Title of host publication2019 IEEE International Symposium on Parallel and Distributed Processing with Applications (ISPA)
PublisherIEEE Explore
Number of pages10
Publication statusPublished - 18 Dec 2019

Bibliographical note

Acknowledgements: The Ranger cluster log-data was provided by the Texas
Advanced Computing Center (TACC). We like to thank Karl
Solchenbach (Intel) for granting access to his engineers. We
thank the anonymous reviewers for their constructive feedback
which helped improve the paper significantly. This research is
supported by The Alan Turing Institute under the EPSRC grant
EP/N510129/1, The Alan Turing Institute-Intel partnership, The
University of Warwick Department of Computer Science scholarship, The National Science Foundation under OCI awards
#0622780, #1203604 and #1134872 to TACC at The University
of Texas at Austin.


Dive into the research topics of 'Using Resource Use Data and System Logs for HPC System Error Propagation and Recovery Diagnosis'. Together they form a unique fingerprint.

Cite this