Purdue University Graduate School

File(s) under embargo





until file(s) become available

Deep Learning-Based Anomaly  Detection in TLS Encrypted Traffic

posted on 2023-08-03, 15:27 authored by Kehinde AyanoKehinde Ayano

 The growing trend of encrypted network traffic is changing the cybersecurity threat scene. Most critical infrastructures and organizations enhance service delivery by embracing digital platforms and applications that use encryption to ensure that data and Information are moved across networks in an encrypted form to improve security. While this protects data confidentiality, hackers are also taking advantage of encrypted network traffic to hide malicious software known as malware that will easily bypass the conventional detection mechanisms on the system because the traffic is not transparent for the monitoring mechanism on the system to analyze. Cybercriminals leverage encryption using cryptographic protocols such as SSL/TLS to launch malicious attacks. This hidden threat exists because of the SSL encryption of benign traffic. Hence, there is a need for visibility in encrypted traffic. This research was conducted to detect malware in encrypted network traffic without decryption. The existing solution involves bulk decryption, analysis, and re-encryption. However, this method is prone to privacy issues, is not cost-efficient, and is time-consuming, creating huge overhead on the network. In addition, limited research exists on detecting malware in encrypted traffic without decryption. There is a need to strike a balance between security and privacy by building an intelligent framework that can detect malicious activity in encrypted network traffic without decrypting the traffic prior to inspection. With the payload still encrypted, the study focuses on extracting metadata from flow features to train the machine-learning model. It further deployed this set of features as input to an autoencoder, leveraging the construction error of the autoencoder for anomaly detection. 


Degree Type

  • Doctor of Philosophy


  • Computer and Information Technology

Campus location

  • West Lafayette

Advisor/Supervisor/Committee Chair

Kathryn Seigfried-Spellar, Co-Chair

Advisor/Supervisor/Committee co-chair

John Springer, Co-Chair

Additional Committee Member 2

Jin Kocsis

Additional Committee Member 3

Vetria Byrd