The ability to automatically detect faults or fault patterns to enhance system reliability is important for system administrators in reducing system failures. To achieve this objective, the message logs from cluster system are augmented with failure information, i.e., The raw log data is labelled. However, tagging or labelling of raw log data is very costly. In this paper, our objective is to detect failure patterns in the message logs using unlabelled data. To achieve our aim, we propose a methodology whereby a pre-processing step is first performed where redundant data is removed. A clustering algorithm is then executed on the resulting logs, and we further developed an unsupervised algorithm to detect failure patterns in the clustered log by harnessing the characteristics of these sequences. We evaluated our methodology on large production data, and results shows that, on average, an f-measure of 78% can be obtained without having data labels. The implication of our methodology is that a system administrator with little knowledge of the system can detect failure runs with reasonably high accuracy.
|Title of host publication||2015 IEEE International Parallel and Distributed Processing Symposium Workshop|
|Publication status||Published - May 2015|
|Event||2015 IEEE International Parallel and Distributed Processing Symposium Workshop - Hyderabad, India , Hyderabad, India|
Duration: 25 May 2015 → 29 May 2015
|Conference||2015 IEEE International Parallel and Distributed Processing Symposium Workshop|
|Period||25/05/15 → 29/05/15|