Improving host-based computer security using secure active monitoring and memory analysis

Abstract

Thirty years ago, research in designing operating systems to defeat malicious software was very popular. The primary technique was to design and implement a small security kernel that could provide security assurances to the rest of the system. However, as operating systems grew in size throughout the 1980's and 1990's, research into security kernels slowly waned. From a security perspective, the story was bleak. Providing security to one of these large operating systems typically required running software within that operating system. This weak security foundation made it relatively easy for attackers to subvert the entire system without detection.

The research presented in this thesis aims to reimagine how we design and deploy computer systems. We show that through careful use of virtualization technology, one can effectively isolate the security critical components in a system from malicious software. Furthermore, we can control this isolation to allow the security software a complete view to monitor the running system. This view includes all of the necessary information for implementing useful security applications including the system memory, storage, hardware events, and network traffic. In addition, we show how to perform both passive and active monitoring securely, using this new system architecture.

Security applications must be redesigned to work within this new monitoring architecture. The data acquired through our monitoring is typically very low-level and difficult to use directly. In this thesis, we describe work that helps bridge this semantic gap by locating data structures within the memory of a running virtual machine. We also describe work that shows a useful and novel security framework made possible through this new monitoring architecture. This framework correlates human interaction with the system to distinguish legitimate and malicious outgoing network traffic.