Using Message Logs and Resource Use Data for Cluster Failure Diagnosis

Thuan Chuah, Arshad Jhumka, James Browne, Nentawe Gurumdimma, Sai Narasimhamurthy, Bill Barth

Research output: Chapter in Book/Report/Conference proceedingPublished conference contribution

7 Citations (Scopus)


Failure diagnosis for large compute clusters using
only message logs is known to be incomplete. Recent availability
of resource use data provides another potentially useful source of
data for failure detection and diagnosis. Early work combining
message logs and resource use data for failure diagnosis has
shown promising results. This paper describes the CRUMEL
framework which implements a new approach to combining
rationalized message logs and resource use data for failure diagnosis. CRUMEL identifies patterns of errors and resource use and
correlates these patterns by time with system failures. Application
of CRUMEL to data from the Ranger supercomputer has yielded
improved diagnoses over previous research. CRUMEL has: (i)
showed that more events correlated with system failures can
only be identified by applying different correlation algorithms,
(ii) confirmed six groups of errors, (iii) identified Lustre I/O
resource use counters which are correlated with occurrence of
Lustre faults which are potential flags for online detection of
failures, (iv) matched the dates of correlated error events and
correlated resource use with the dates of compute node hangups and (v) identified two more error groups associated with
compute node hang-ups. The pre-processed data will be put on
the public domain in September, 2016.
Original languageEnglish
Title of host publication2016 IEEE 23rd International Conference on High Performance Computing (HiPC)
PublisherIEEE Explore
Number of pages10
Publication statusPublished - Dec 2016

Bibliographical note

We would like to thank the Texas Advanced Computing Center for providing the Ranger system logs and case studies. This research was supported in part by the National Science Foundation under DCI award #0622780 to the Texas Advanced Computing Center at the University of Texas at Austin.


Dive into the research topics of 'Using Message Logs and Resource Use Data for Cluster Failure Diagnosis'. Together they form a unique fingerprint.

Cite this