Towards Increasing the Error Handling Time Window in Large-Scale Distributed Systems Using Console and Resource Usage Logs

Nentawe Gurumdimma, Arshad Jhumka, Maria Liakata, Thuan Chuah, James Browne

Research output: Chapter in Book/Report/Conference proceedingPublished conference contribution

3 Citations (Scopus)


Resource-intensive applications such as scientific applications require the architecture or system on which they execute to display a very high level of dependability to reduce the impact of faults. Typically, the state of the underlying system is captured through messages that are recorded in a log file, which has been proven useful to system administrators in understanding the root-causes of system failures (and for their subsequent debugging). However, the time window between when the first error message is detected in the log file and time of the ensuing failure may not be large enough to allow the administrators to save the state of the running application, which will result in lost execution time. We thus address this fundamental question: Is it possible to extend this time window? The answer is positive: We show that, by using (i) resource usage logs to track anomalous resource usage and (ii) error logs to identify root-causes of system failures, it is possible to increase the time window, on average, by 50 minutes. These files were those obtained for the Ranger Supercomputer from TACC. We achieve this by applying anomaly detection techniques on resource usage data and conducting a root-cause analysis on error log files.
Original languageEnglish
Title of host publication2015 IEEE Trustcom/BigDataSE/ISPA
PublisherIEEE Explore
ISBN (Electronic)978-1-4673-7952-6
Publication statusPublished - Aug 2015
Event2015 IEEE Trustcom/BigDataSE/ISPA - Helsinki, Finland , Helsinki, Finland
Duration: 20 Aug 201522 Aug 2015


Conference2015 IEEE Trustcom/BigDataSE/ISPA

Bibliographical note

We thank the Texas Advanced Computing Center (TACC) for providing the Ranger message logs. We also thank PTDF Nigeria and the University of Jos-Nigeria for partly funding this research.


Dive into the research topics of 'Towards Increasing the Error Handling Time Window in Large-Scale Distributed Systems Using Console and Resource Usage Logs'. Together they form a unique fingerprint.

Cite this