Abstract
Failure analysis plays an important role in the reliability of data centers and high-performance computing (HPC) systems. Recent work have shown that both resource use data and failure logs can, separately and together, be used to detect system failure-inducing errors and diagnose system failures; the result of error propagation and (unsuccessful) execution of error recovery mechanisms. For more accurate and detailed failure diagnosis, knowledge of error propagation patterns and unsuccessful error recovery is important. To improve system reliability, knowledge of recovery protocols deployment is important. This paper describes and demonstrates application of a new diagnostics framework (CORRMEXT). CORRMEXT analyzes and reports error propagation patterns and degrees of success and failure of error recovery protocols. The steps in the framework are correlations of resource use metrics and error messages, and identification of the earliest times of change of system behaviour. The framework is illustrated with analyses of resource use data and message logs for three HPC systems operated by the Texas Advanced Computing Center (TACC). The illustrations are focused on groups of resource use counters and groups of errors; they reveal many interesting insights into patterns of: (i) network data and software errors, (ii) Lustre file-system and Linux operating system process errors, and (iii) memory and storage errors. We also confirm that: (i) correlations of resource use and errors can only be identified by applying different correlation algorithms, and (ii) the earliest times of change in system behaviour can only be identified by analyzing both the correlated resource use counters and correlated errors. We believe CORRMEXT is the first tool that have diagnosed error propagation paths and error recovery attempts on three different HPC systems. CORRMEXT will be put on the public domain to support systems administrators in diagnosing HPC system failures, on August 2018.
| Original language | English |
|---|---|
| Pages (from-to) | 95-112 |
| Number of pages | 18 |
| Journal | Journal of Parallel and Distributed Computing (JPDC) |
| Volume | 132 |
| Early online date | 7 Jun 2019 |
| DOIs | |
| Publication status | Published - Oct 2019 |
Bibliographical note
Acknowledgements: We would like to thank the Texas Advanced Computing Center (TACC) for providing the Stampede, Ranger & Lonestar4 cluster log-data, and to Tommy Minyard, William Lee Barth and Richard Todd Evans for granting access to their data and HPC systems. We would also like to thank Karl Solchenbach and Marie-Christine Sawley (Intel Corporation, Europe) for granting access to their research scientists, and Theo Damoulas (University of Warwick, UK) for his contribution to validating the CORRMEXT framework. We would also like to thank the anonymous reviewers for their constructive feedback which helped improve the paper significantly. This research is supported by The Alan Turing Institute under the EPSRC, UK grant EP/N510129/1, The Alan Turing Institute-Intel partnership and The National Science Foundation, USA under OCI awards #0622780, #1203604 and #1134872 to TACC at The University of Texas at Austin.This paper is dedicated to the memory of Professor Emeritus James Clayton Browne (January 16, 1935–January 19, 2018). Dr. Browne’s contributions to developing the CORRMEXT framework have been crucial but sadly he passed away while we were working on the paper. As such, we wish to keep his name as an author posthumously.
Data Availability Statement
No data availability statementKeywords
- Large HPC systems
- Correlation
- Variance extraction
- Error propagation and recovery
- Cluster log-data