Do we still need canaries in the coal mine? Measuring shadow stack effectiveness in countering stack smashing

Abstract: Stack canaries and shadow stacks are widely deployed mitigations to memory-safety vulnerabilities. While stack canaries are introduced by the compiler and rely on sentry values placed between variables and control data, shadow stack implementations protect return addresses explicitly and rely on hardware features available in modern processor designs for efficiency. In this paper we hypothesize that stack canaries and shadow stacks provide similar levels of protections against sequential stack-based overflows. Based on the Juliet test suite, we evaluate whether 64-bit x86 (x86-64) systems benefit from enabling stack canaries in addition to the x86-64 shadow stack enforcement. We observe divergence in overflow detection rates between the GCC and Clang compilers and across optimization levels, which we attribute to differences in stack layouts generated by the compilers. We also find that x86-64 shadow stack implementations are more effective and outperform stack canaries when combined with a stack-protector-like stack layout. We implement and evaluate an enhancement to the Clang x86-64 shadow stack instrumentation that improves the shadow stack detection accuracy based on this observation.