On the effectiveness of virtualization-based security

Abstract: Protecting commodity operating systems and applications against malware and targeted attacks has proven to be difficult. In recent years, virtualization has received attention from security researchers who utilize it to harden existing systems and provide strong security guarantees. This has lead to interesting use cases such as cloud computing where possibly sensitive data is processed on remote, third party systems. The migration and processing of data in remote servers, poses new technical and legal questions, such as which security measures should be taken to protect this data or how can it be proven that execution of code wasn't tampered with. In this paper we focus on technological aspects. We discuss the various possibilities of security within the virtualization layer and we use as a case study \HelloRootkitty{}, a lightweight invariance-enforcing framework which allows an operating system to recover from kernel-level attacks. In addition to \HelloRootkitty{}, we also explore the use of special hardware chips as a way of further protecting and guaranteeing the integrity of a virtualized system.