Abstract
Fully supervised log anomaly detection methods suffer the heavy burden of annotating massive unlabeled log data. Recently, many semi-supervised methods have been proposed to reduce annotation costs with the help of parsed templates. However, these methods consider each keyword independently, which disregards the correlation between keywords and the contextual relationships among log sequences. In this paper, we propose a novel weakly supervised log anomaly detection framework, named LogLG, to explore the semantic connections among keywords from sequences. Specifically, we design an end-to-end iterative process, where the keywords of unlabeled logs are first extracted to construct a log-event graph. Then, we build a subgraph annotator to generate pseudo labels for unlabeled log sequences. To ameliorate the annotation quality, we adopt a self-supervised task to pre-train a subgraph annotator. After that, a detection model is trained with the generated pseudo labels. Conditioned on the classification results, we re-extract the keywords from the log sequences and update the log-event graph for the next iteration. Experiments on five benchmarks validate the effectiveness of LogLG for detecting anomalies on unlabeled log data and demonstrate that LogLG, as the state-of-the-art weakly supervised method, achieves significant performance improvements compared to existing methods.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
References
Chen, J., et al.: An empirical investigation of incident triage for online service systems. In: ICSE 2019 (2019)
Chen, Y., et al.: Identifying linked incidents in large-scale online service systems. In: ESEC/FSE 2020 (2020)
Devlin, J., Chang, M., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. In: NAACL-HLT 2019 (2019)
Du, M., Li, F., Zheng, G., Srikumar, V.: Deeplog: anomaly detection and diagnosis from system logs through deep learning. In: CCS 2017 (2017)
Guo, H., Lin, X., Yang, J., Liu, J., Zhuang, Y., Bai, J., Zheng, T., Zhang, B., Li, Z.: Translog: A unified transformer-based framework for log anomaly detection. CoRR (2022)
Guo, H., et al.: Lvp-m3: language-aware visual prompt for multilingual multimodal machine translation. In: EMNLP (2022)
He, P., Zhu, J., Zheng, Z., Lyu, M.R.: Drain: an online log parsing approach with fixed depth tree. In: ICWS 2017 (2017)
He, S., Zhu, J., He, P., Lyu, M.R.: Loghub: a large collection of system log datasets towards automated log analytics. CoRR abs/2008.06448 (2020)
Le, V., Zhang, H.: Log-based anomaly detection without log parsing. In: ASE 2021 (2021)
Lee, Y., Kim, J., Kang, P.: Lanobert: system log anomaly detection based on BERT masked language model. CoRR (2021)
Liang, Y., Zhang, Y., **ong, H., Sahoo, R.K.: Failure prediction in IBM bluegene/l event logs. In: ICDM 2007 (2007)
Liu, J., Guo, J., Xu, D.: Geometrymotion-transformer: an end-to-end framework for 3d action recognition. IEEE TMM (2022)
Liu, J., Yu, T., Peng, H., Sun, M., Li, P.: Cross-lingual cross-modal consolidation for effective multilingual video corpus moment retrieval. In: NAACL 2022
Meng, W., et al.: Loganomaly: unsupervised detection of sequential and quantitative anomalies in unstructured logs. In: IJCAI 2019 (2019)
Wittkopp, T., et al.: A2log: attentive augmented log anomaly detection. In: HICSS 2022 (2022)
Xu, K., Hu, W., Leskovec, J., Jegelka, S.: How powerful are graph neural networks? In: ICLR 2019 (2019)
Yang, J., et al.: UM4: unified multilingual multiple teacher-student model for zero-resource neural machine translation. In: IJCAI 2022, pp. 4454–4460 (2022)
Yang, L., et al.: Semi-supervised log-based anomaly detection via probabilistic label estimation. In: ICSE 2021 (2021)
Zhang, L., Ding, J., Xu, Y., Liu, Y., Zhou, S.: Weakly-supervised text classification based on keyword graph. In: EMNLP 2021 (2021)
Zhang, X., et al.: Robust log-based anomaly detection on unstable log data. In: ESEC/SIGSOFT FSE (2019)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Guo, H. et al. (2023). LogLG: Weakly Supervised Log Anomaly Detection via Log-Event Graph Construction. In: Wang, X., et al. Database Systems for Advanced Applications. DASFAA 2023. Lecture Notes in Computer Science, vol 13946. Springer, Cham. https://doi.org/10.1007/978-3-031-30678-5_36
Download citation
DOI: https://doi.org/10.1007/978-3-031-30678-5_36
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-30677-8
Online ISBN: 978-3-031-30678-5
eBook Packages: Computer ScienceComputer Science (R0)