Abstract
With the growth of internet encryption to protect users’ privacy, malware has evolved to employ encryption protocols such as TLS/SSL to obfuscate the contents of malicious communications. Unfortunately, decrypting network data before it reaches the signature-based Intrusion Detection System (IDS) to identify TLS-based malware is impractical since it adds infrastructure complexity and compromises user privacy. As a result, various studies have moved to investigate anomaly-based detection without decryption using different TLS features and techniques such as Machine Learning (ML). This paper aims to review TLS-based malware anomaly detection studies and analyze the employment of TLS features and machine learning in these works to understand the field’s current state better. Furthermore, this study highlights the strengths of the related research and offers several recommendations on its shortcomings and TLS features for future effective detection systems.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
“Defining Malware: FAQ | Microsoft Docs”. https://docs.microsoft.com/en-us/previous-versions/tn-archive/dd632948(v=technet.10)?redirectedfrom=MSDN (accessed Apr. 29, 2021).
- 2.
“Malware Statistics & Trends Report | AV-TEST”. https://www.av-test.org/en/statistics/malware/ (accessed Apr. 29, 2021).
- 3.
L. Nagy, “Nearly a quarter of malware now communicates using TLS – Sophos News”, Feb. 18, 2020. https://news.sophos.com/en-us/2020/02/18/nearly-a-quarter-of-malware-now-communicates-using-tls/ (accessed May 02, 2021).
- 4.
- 5.
- 6.
- 7.
- 8.
References
Durumeric, Z., et al.: The security impact of HTTPS interception. In: NDSS (2017). https://doi.org/10.14722/ndss.2017.23456
Kok, S.H., Abdullah, A., Jhanjhi, N.Z., Supramaniam, M.: A review of intrusion detection system using machine learning approach. Int. J. Eng. Res. Technol. 12(1), 8–15 (2019)
Singh, A.P., Singh, M.: A comparative review of malware analysis and detection in HTTPs traffic. Int. J. Comput. Digit. Syst. 10(1), 2210–3142 (2021). https://doi.org/10.12785/ijcds/100111
Understanding malware & other threats - Windows security | Microsoft Docs. https://docs.microsoft.com/en-us/windows/security/threat-protection/intelligence/understanding-malware. Accessed 20 Sept 2020
What Is the Difference: Viruses, Worms, Trojans, and Bots? https://tools.cisco.com/security/center/resources/virus_differences. Accessed 08 Oct 2020
Gadhiya, S., Bhavsar, K.H.: Techniques for malware analysis. Int. J. Adv. Res. Comput. Sci. Softw. Eng. 3, 2277–3128 (2013)
Jenseg, O.: A machine learning approach to detecting malware in TLS traffic using resilient network features. Master’s thesis, NTNU (2019)
Patrick, N.: The TLS Handshake: taking a closer look - Hashed Out by The SSL StoreTM (2019). https://www.thesslstore.com/blog/explaining-ssl-handshake/. Accessed 08 July 2020
Liu, J., Zeng, Y., Shi, J., Yang, Y., Wang, R., He, L.: Maldetect: a structure of encrypted malware traffic detection. Comput. Mater. Contin. 60(2), 721–739 (2019). https://doi.org/10.32604/cmc.2019.05610
Anderson, B., Paul, S., McGrew, D.: Deciphering malware’s use of TLS (without decryption). J. Comput. Virol. Hack. Tech. 14(3), 195–211 (2017). https://doi.org/10.1007/s11416-017-0306-6
Senecal, D., Kahn, A., Segal, O., et al.: Bot detection in an edge network using Transport Layer Security (TLS) fingerprint. Google Patents (2019)
Anderson, B., McGrew, D.: Leveraging point inferences on HTTP transactions for HTTPS malware detection. Google Patents (2019)
Roques, O., Maffeis, S., Cova, M.: Detecting malware in TLS traffic. Ph.D. diss., Imperial College London (2019)
Calderon, P., Hasegawa, H., Yamaguchi, Y., Shimada, H.: Malware detection based on HTTPS characteristic via machine learning. In: ICISSP 2018 – Proceedings of 4th International Conference on Information Systems Security Privacy, vol. 2018-Janua, no. Icissp, pp. 410–417 (2018). https://doi.org/10.5220/0006654604100417
Zheng, R., et al.: Two-layer detection framework with a high accuracy and efficiency for a malware family over the TLS protocol. PLoS ONE 15(5), e0232696 (2020). https://doi.org/10.1371/journal.pone.0232696
Dai, R., Gao, C., Lang, B., Yang, L., Liu, H., Chen, S.: SSL malicious traffic detection based on multi-view features. In: ACM International Conference on Proceeding Series, pp. 40–46 (2019). https://doi.org/10.1145/3371676.3371697
Bazuhair, W., Lee, W.: Detecting malign encrypted network traffic using perlin noise and convolutional neural network. In: 2020 10th Annual Computing and Communication Workshop and Conference, CCWC 2020, January 2020, pp. 200–206 (2020). https://doi.org/10.1109/CCWC47524.2020.9031116
Prasse, P., Gruben, G., Pevny, T., Sofka, M., Scheffer, T.: Malware detection by HTTPS traffic analysis. Math. Fak. Potsdam Univ. (2017). https://doi.org/10.1109/OCEANS.2001.968684
Puuska, S., Kokkonen, T., Alatalo, J., Heilimo, E.: Anomaly-based network intrusion detection using wavelets and adversarial autoencoders. In: Lanet, J.-L., Toma, C. (eds.) SECITC 2018. LNCS, vol. 11359, pp. 234–246. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-12942-2_18
Anderson, B., McGrew, D.: Identifying encrypted malware traffic with contextual flow data. In: AISec 2016 - Proceedings of the 2016 ACM Workshop on Artificial Intelligence and Security Co-located with CCS 2016, pp. 35–46 (2016). https://doi.org/10.1145/2996758.2996768
Anderson, B., McGrew, D.: Machine learning for encrypted malware traffic classification: Accounting for noisy labels and non-stationarity. In: Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, vol. Part F1296, pp. 1723–1732 (2017). https://doi.org/10.1145/3097983.3098163
Prasse, P., Machlica, L., Pevny, T., Havelka, J., Scheffer, T.: Malware detection by analysing network traffic with neural networks. In: Proceedings of the 2017 IEEE Symposium on Security and Privacy Workshops, SPW 2017, vol. 2017-Decem, pp. 205–210 (2017). https://doi.org/10.1109/SPW.2017.8
Franc, V., Sofka, M., Bartos, K.: Learning detector of malicious network traffic from weak labels. In: Bifet, A., et al. (eds.) ECML PKDD 2015. LNCS (LNAI and LNB), vol. 9286, pp. 85–99. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-23461-8_6
Bortolameotti, R.: C&C botnet detection over SSL. Master’s thesis, University of Twente (2014)
Strasák, F.: Detection of HTTPS malware traffic. Bachelor’s thesis, Czech Technical University in Prague, pp. 1–49 (2017)
Kato, H., Haruta, S., Sasase, I.: Android malware detection scheme based on level of SSL server certificate. In: 2019 IEEE Global Communications Conference, GLOBECOM 2019 - Proceedings, no. 2, pp. 379–389 (2019). https://doi.org/10.1109/GLOBECOM38437.2019.9013483
Fehrman, B., Woody, E., et al.: Connection information. Google Patents (2020)
Senecal, D., Kahn, A., Segal, O., et al.: Bot detection in an edge network using Transport Layer Security (TLS) fingerprint. Google Patents, vol. 1 (2019)
De Lucia, M. J., Cotton, C.: Detection of Encrypted Malicious Network Traffic using Machine Learning. In: Proceedings of the IEEE Military Communications Conference, MILCOM, vol. 2019-Novem, pp. 1–6 (2019). https://doi.org/10.1109/MILCOM47813.2019.9020856
Torroledo, I., Camacho, L. D., Bahnsen, A. C.: Hunting malicious TLS certificates with deep neural networks. In: Proceedings of the ACM Conference on Computer and Communications Security, pp. 64–73 (2018). https://doi.org/10.1145/3270101.3270105
Kohout, J., Komárek, T., Čech, P., Bodnár, J., Lokoč, J.: Learning communication patterns for malware discovery in HTTPs data. Expert Syst. Appl. 101, 129–142 (2018). https://doi.org/10.1016/j.eswa.2018.02.010
Maroušek, J.: Efficient kNN classification of malware from HTTPS data. Master’s thesis, Charles University, Faculty of Mathematics and Physics (2017)
What is an Extended Validation Certificate? :: What is an Extended Validation Certificate? :: GlobalSign GMO Internet, Inc. https://www.globalsign.com/en/ssl-information-center/what-is-an-extended-validation-certificate. Accessed 08 Oct 2020
Allix, K., Bissyandé, T. F., Klein, J., Le Traon, Y.: AndroZoo: collecting millions of android apps for the research community. In: Proceedings - 13th Working Conference on Mining Software Repositories, MSR 2016, pp. 468–471 (2016). https://doi.org/10.1145/2901739.2903508.
Brownlee, J.: Supervised and unsupervised machine learning algorithms (2016). https://machinelearningmastery.com/supervised-and-unsupervised-machine-learning-algorithms/. Accessed 03 June 2021
Commonly Used Machine Learning Algorithms | Data Science. https://www.analyticsvidhya.com/blog/2017/09/common-machine-learning-algorithms/. Accessed 28 Sept 2020
Kononenko, I., Kukar, M.: Machine Learning and Data Mining. Horwood Publishing (2007)
Yao, B., Li, F., Kumar, P.: K nearest neighbor queries and KNN-joins in large relational databases (almost) for free. In: Proceedings - International Conference on Data Engineering, pp. 4–15 (2010). https://doi.org/10.1109/ICDE.2010.5447837
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Keshkeh, K., Jantan, A., Alieyan, K., Gana, U.M. (2021). A Review on TLS Encryption Malware Detection: TLS Features, Machine Learning Usage, and Future Directions. In: Abdullah, N., Manickam, S., Anbar, M. (eds) Advances in Cyber Security. ACeS 2021. Communications in Computer and Information Science, vol 1487. Springer, Singapore. https://doi.org/10.1007/978-981-16-8059-5_13
Download citation
DOI: https://doi.org/10.1007/978-981-16-8059-5_13
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-16-8058-8
Online ISBN: 978-981-16-8059-5
eBook Packages: Computer ScienceComputer Science (R0)