Abstract
Network-based fuzz testing has become an effective mechanism to ensure the security and reliability of communication protocol systems. However, fuzz testing is still conducted in an ad-hoc manner with considerable manual effort, which is mainly due to the unavailability of protocol model. In this paper we present our on-going work of develo** an automated and measurable protocol fuzz testing approach that uses a formally synthesized approximate formal protocol specification to guide the testing process. We adopt the Finite State Machine protocol model and study two formal methods for protocol synthesis: an active black-box checking algorithm that has provable optimality and a passive trace minimization algorithm that is less accurate but much more efficient. We also present our preliminary results of using this method to implementations of the MSN instant messaging protocol: MSN clients Gaim (pidgin) and aMSN. Our testing reveals some serious reliability and security flaws by automatically crashing both of them.
Chapter PDF
Similar content being viewed by others
References
Angulin, D.: Learning regular sets from queries and counterexamples. Information and Computation 75, 87–106 (1987)
Cui, W., Kannan, J., Wang, H.: Discoverer: Automatic Protocol Reverse Engineering from Network Traces. In: The 16th USENIX Security Symposium (2007)
Dolev, D., Yao, A.: On the security of public-key protocols. IEEE Transaction on Information Theory 29, 198–208 (1983)
Godefroid, P., Klarlund, N., Sen, K.: DART: Directed Automated Random Testing. In: Proceedings of PLDI 2005 (ACM SIGPLAN 2005 Conference on Programming Language Design and Implementation), pp. 213–223 (2005)
Godefroid, P., Levin, M.Y., Molnar, D.: Automated Whitebox Fuzz Testing. Technical Report MS-TR-2007-58, Microsoft (May 2007)
Gören, S., Ferguson, F.J.: On state reduction of incompletely specified finite state machines. Computers and Electrical Engineering 33(1), 58–69 (2007)
Howard, M.: Inside the Windows Security Push. IEEE Security & Privacy, 57–61 (2003)
Lee, D., Yannakakis, M.: Principles and methods of testing finite state machines - A survey. In: Proceedings of the IEEE, 1090–1123 (1996)
Oehlert, P.: Violating Assumptions with Fuzzing. IEEE Security & Privacy, pp. 58-62 (2005)
Peled, D., Vardi, M.Y., Yannakakis, M.: Black-box checking. In: Proceedings of IFIP FORTE/PSTV (1999)
Shu, G., Lee, D.: Testing Security Properties of protocol implementations – a machine learning based approach. In: Proceedings of IEEE ICDCS (2007)
Wang, L., Ellis, C., Yin, W., Luong, D.D.: Hercules: An Environment for Large-Scale Enterprise Infrastructure Testing. In: Proceedings of the Workshop on Advances and Innovations in Systems Testing (2007)
Author information
Authors and Affiliations
Editor information
Rights and permissions
Copyright information
© 2008 IFIP International Federation for Information Processing
About this paper
Cite this paper
Shu, G., Hsu, Y., Lee, D. (2008). Detecting Communication Protocol Security Flaws by Formal Fuzz Testing and Machine Learning. In: Suzuki, K., Higashino, T., Yasumoto, K., El-Fakih, K. (eds) Formal Techniques for Networked and Distributed Systems – FORTE 2008. FORTE 2008. Lecture Notes in Computer Science, vol 5048. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-68855-6_19
Download citation
DOI: https://doi.org/10.1007/978-3-540-68855-6_19
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-68854-9
Online ISBN: 978-3-540-68855-6
eBook Packages: Computer ScienceComputer Science (R0)