Parameter-Efficient Log Anomaly Detection based on Pre-training model and LORA

Logs record both the normal and abnormal system operating status at any time, which are crucial data during system operation. Log anomaly detection can help with system debugging and analyzing root causes, such as system fault, shutdown fault, null-pointer exception, illegal-argument exception, and class cast exception. Deep learning is widely applied to log anomaly detection to enhance detection accuracy. However, the deep learning model requires a lot of label logs, which consume large amounts of labor and time. To tackle this label requirement problem, the pre-training model is introduced, for instance, the Bidirectional Encoder Representations from Transformers (BERT). However, the pre-training model brings new problems. The parameters of BERT needed to be fine-tuned are huge, resulting in a high training overhead. Besides, the direct word sequence input representation of BERT ignores the semantic information among logs. Therefore, we propose a parameter-efficient log anomaly detection scheme (LogBP-LORA) based on BERT and Low-Rank Adaptation (LORA). LORA is an effective parameter-tuning strategy. LogBP-LORA increases bypass weight matrices and only updates the bypass parameters instead of all the original parameters to reduce the training overhead. Additionally, LogBP-LORA exploits log event sequence representation to obtain more semantic information with a shorter sequence length. Extensive experiments carry on three public log datasets, BGL, Thunderbird, and HDFS, demonstrate LogBP-LORA can obtain favorable performance with lower resource consumption. When fewer label data is available, LogBP-LORA achieves about 10%-99% higher F1-score compared with Neurallog, Deeplog, MADDC, and Loganomaly. The training parameters of LogBP-LoRA are only 0.06% of the original parameters of BERT.