Novel System Compartmentalization and Reverse Engineering Methods

The need to secure software systems is more important than ever. However, while a lot of work exists to design and implement secure systems, a fundamental weakness remains. Instead of implementing software with least privilege policies, developers create monolithic systems that allow any instruction near universal memory access. This dissertation attempts to rectify this fundamental weakness to software design through three different contributions.

First, I address the monolithic software design problem by proposing and evaluating a novel compartmentalization enforcement mechanism called Hardware-Assisted Kernel Compartmentalization (HAKC). HAKC is capable of enforcing an arbitrary compartmentalization policy using features of the ARMv9 ISA, without the need of any extra virtualization or trusted software layer. I then introduce a method of determining an optimal compartmentalization policy based on user performance and security constraints called FlexC, which is tested using HAKC as the enforcement mechanism. The end result is a hardened, com-partmentalized kernel, customized to a user’s needs, which enforces a least privilege policy that minimizes overhead. Finally, as an avenue for further compartmentalization policy generation, I introduce a novel program analysis framework called IOVec Function Identifier (IOVFI), which foregoes the use of language processing and model learning, but instead uses program state changes as a unique function fingerprint. I show that IOVFI is a more stable and accurate function identifier than the state-of-the-art, even in the presence of differing compilation environments, purposeful obfuscations, and even architecture changes.