Enabling Dependability-Driven Resource Use and Message Log-Analysis for Cluster System Diagnosis

Thuan Chuah, Arshad Jhumka, Samantha Alt, Theodoros Damoulas, Nentawe Gurumdimma, Marie-Christine Sawley, Bill Barth, Tommy Minyard, James Browne

Research output: Chapter in Book/Report/Conference proceedingPublished conference contribution

10 Citations (Scopus)


Recent work have used both failure logs and resource use data separately (and together) to detect system failure-inducing errors and to diagnose system failures. System failure occurs as a result of error propagation and the (unsuccessful) execution of error recovery mechanisms. Knowledge of error propagation patterns and unsuccessful error recovery is important for more accurate and detailed failure diagnosis, and knowledge of recovery protocols deployment is important for improving system reliability. This paper presents the CORRMEXT framework which carries failure diagnosis another significant step forward by analyzing and reporting error propagation patterns and degrees of success and failure of error recovery protocols. CORRMEXT uses both error messages and resource use data in its analyses. Application of CORRMEXT to data from the Ranger supercomputer have produced new insights. CORRMEXT has: (i) identified correlations between resource use counters that capture recovery attempts after an error, (ii) identified correlations between error events to capture error propagation patterns within the system, (iii) identified error propagation and recovery paths during system execution to explain system behaviour, (iv) showed that the earliest times of change in system behaviour can only be identified by analyzing both the correlated resource use counters and correlated errors. CORRMEXT will be installed on the HPC clusters at the Texas Advanced Computing Center in Autumn 2017.
Original languageEnglish
Title of host publication2017 IEEE 24th International Conference on High Performance Computing (HiPC)
PublisherIEEE Explore
Publication statusPublished - 21 Dec 2017

Bibliographical note

We would like to thank the Texas Advanced Computing Center (TACC) for providing the Ranger cluster log data and granting access to their systems administrators. We also thank Karl Solchenbach (Intel Corporation, Europe) for granting access to his research scientists. This research is supported by The Alan Turing Institute under the EPSRC grant EP/N510129/1, The Alan Turing Institute-Intel partnership and the National Science Foundation under OCI awards #0622780 and #1203604 to TACC at the University of Texas at Austin.


Dive into the research topics of 'Enabling Dependability-Driven Resource Use and Message Log-Analysis for Cluster System Diagnosis'. Together they form a unique fingerprint.

Cite this