RADAR: compiler and architecture supported intrusion prevention, detection, analysis and recovery
MetadataShow full item record
In this dissertation, we propose RADAR - compileR and micro-Architecture supported intrusion prevention, Detection, Analysis and Recovery. RADAR is an infrastructure to help prevent, detect and even recover from attacks to critical software. Our approach emphasizes collaborations between compiler and micro-architecture to avoid the problems of purely software or hardware based approaches. With hardware support for cryptographic operations, our infrastructure can achieve strong process isolation to prevent attacks from other processes and to prevent certain types of hardware attacks. Moreover, we show that an unprotected system address bus leaks critical control flow information of the protected software but has never been carefully addressed previously. To enhance intrusion prevention capability of our infrastructure further, we present a scheme with both innovative hardware modification and extensive compiler support to eliminate most of the information leakage on system address bus. However, no security system is able to prevent all attacks. In general, we have to assume that certain attacks will get through our intrusion prevention mechanisms. To protect software from those attacks, we build a second line of defense consisted of intrusion detection and intrusion recovery mechanisms. Our intrusion detection mechanisms are based on anomaly detection. In this dissertation, we propose three anomaly detection schemes. We demonstrate the effectiveness of our anomaly detection schemes thus the great potential of what compiler and micro-architecture can do for software security. The ability to recover from an attack is very important for systems providing critical services. Thus, intrusion recoverability is an important goal of our infrastructure. We focus on recovery of memory state in this dissertation, since most attacks break into a system by memory tampering. We propose two schemes for intrusion analysis. The execution logging based scheme incurs little performance overhead but has higher demand for storage and memory bandwidth. The external input points tagging based scheme is much more space and memory bandwidth efficient, but leads to significant performance degradation. After intrusion analysis is done and tampered memory state is identified, tampered memory state can be easily recovered through memory updates logging or memory state checkpointing.