Improving detection and annotation of malware downloads and infections through deep packet inspection
Nelms, Terry Lee
MetadataShow full item record
Malware continues to be one of the primary tools employed by attackers. It is used in attacks ranging from click fraud to nation state espionage. Malware infects hosts over the network through drive-by downloads and social engineering. These infected hosts communicate with remote command and control (C&C) servers to perform tasks and exfiltrate data. Malware's reliance on the network provides an opportunity for the detection and annotation of malicious communication. This thesis presents four main contributions. First, we design and implement a novel incident investigation system, named WebWitness. It automatically traces back and labels the sequence of events (e.g., visited web pages) preceding malware downloads to highlight how users reach attack pages on the web; providing a better understanding of current attack trends and aiding in the development of more effective defenses. Second, we conduct the first systematic study of modern web based social engineering malware download attacks. From this study we develop a categorization system for classifying social engineering downloads and use it to measure attack properties. From these measurements we show that it is possible to detect the majority of social engineering downloads using features from the download path. Third, we design and implement ExecScent, a novel system for mining new malware C&C domains from live networks. ExecScent automatically learns C&C traffic models that can adapt to the deployment network's traffic. This adaptive approach allows us to greatly reduce the false positives while maintaining a high number of true positives. Lastly, we develop a new packet scheduling algorithm for deep packet inspection that maximizes throughput by optimizing for cache affinity. By scheduling for cache affinity, we are able to deploy our systems on multi-gigabit networks.