Abstract
Log files are created by devices or systems in order to provide information about processes or actions that were performed. Detailed inspection of security logs can reveal potential security breaches and it can show us system weaknesses. In our work we propose a novel anomaly-based detection approach based on data mining techniques for log analysis. Our approach uses Apache Hadoop technique to enable processing of large data sets in a parallel way. Dynamic rule creation enables us to detect new types of breaches without further human intervention. Overall error rates of our method are below 10%.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
J. Dean and S. Ghemawat. MapReduce: Simplified Data Processing on Large Clusters. Commun. ACM, 51(1):107–113, January 2008.
A Frei and M. Rennhard. Histogram matrix: Log file visualization for anomaly detection. In Availability, Reliability and Security, 2008. ARES 08. Third International Conference on, pages 610–617, March 2008.
Q. Fu, J.-G. Lou, Y. Wang, and J. Li. Execution anomaly detection in distributed systems through unstructured log analysis. In Proceedings of the 2009 Ninth IEEE International Conference on Data Mining, ICDM ’09, pages 149–158, Washington, DC, USA, 2009. IEEE Computer Society.
L.K.J. Grace, V. Maheswari, and D. Nagamalai. Web log data analysis and mining. In Natarajan Meghanathan, BrajeshKumar Kaushik, and Dhinaharan Nagamalai, editors, Advanced Computing, volume 133 of Communications in Computer and Information Science, pages 459–469. Springer Berlin Heidelberg, 2011.
A Makanju, A.N. Zincir-Heywood, and E.E. Milios. Investigating event log analysis with minimum apriori information. In Integrated Network Management (IM 2013), 2013 IFIP/IEEE International Symposium on, pages 962–968, May 2013.
M.G. Schultz, E. Eskin, E. Zadok, and S.J. Stolfo. Data mining methods for detection of new malicious executables. In Security and Privacy, 2001. S P 2001. Proceedings. 2001 IEEE Symposium on, pages 38–49, 2001.
M. A. Siddiqui. Data mining methods for malware detection. ProQuest, 2011.
R. Winding, T. Wright, and M. Chapple. System Anomaly Detection: Mining Firewall Logs. In Securecomm and Workshops, 2006, pages 1–5, Aug 2006.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2015 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Breier, J., Branišová, J. (2015). Anomaly Detection from Log Files Using Data Mining Techniques. In: Kim, K. (eds) Information Science and Applications. Lecture Notes in Electrical Engineering, vol 339. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-662-46578-3_53
Download citation
DOI: https://doi.org/10.1007/978-3-662-46578-3_53
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-662-46577-6
Online ISBN: 978-3-662-46578-3
eBook Packages: EngineeringEngineering (R0)