FORENSICS AND FORMALIZED PROTOCOL CUSTOMIZATION FOR ENHANCING NETWORKING SECURITY

Comprehensive networking security is a goal to achieve for enterprise networks. In forensics, the traffic analysis, causality dependence in intricate program network flows is needed in flow-based attribution techniques. The provenance, the connection between stealthy advanced persistent threats (APTs) and the execution of loadable modules is stripped because loading a module does not guarantee an execution. The reports of common vulnerabilities and exposures (CVE) demonstrate that lots of vulnerabilities have been introduced in protocol engineering process, especially for the emerging Internet-of-Things (IoT) applications. A code generation framework targeting secure protocol implementations can substantially enhance security.

A novel automaton-based technique, NetCrop, to infer fine-grained program behavior by analyzing network traffic is proposed in this thesis. Based on network flow causality, it constructs automata that describe both the network behavior and the end-host behavior of a whole program to attribute individual packets to their belonging programs and fingerprint the high-level program behavior. A novel provenance-oriented library tracing system, Lprov, which enforces library tracing on top of existing syscall logging based provenance tracking approaches is investigated. With the dynamic library call stack, the provenance of implicit library function execution is revealed and correlated to system events, facilitating the locating and defense of malicious libraries. The thesis presents ProFactory, in which a protocol is modeled, checked and securely generated, averting common vulnerabilities residing in protocol implementations.