Resilient Self-Debugging Software Protection

Abstract: Debuggers are a popular reverse engineering and tampering tool. Self-debugging is an effective technique for applications to defend themselves against hostile debuggers. In penetration tests on state-of-the-art self-debugging, we observed several opportunities through which it could be attacked, however. We therefore improved upon the existing technique, making it more resilient by introducing reciprocal debugging and making the transfers of control between protected application and self-debugger more stealthy. This paper presents the improved self-debugging design, and details our research efforts into realizing reciprocal debugging. In our evaluation we show that the improved design is significantly harder for attackers to defeat.