Identification of Bugs and Vulnerabilities in TLS Implementation for Windows Operating System Using State Machine Learning

Abstract: TLS protocol is an essential part of secure Internet communication. In past, many attacks have been identified on the protocol. Most of these attacks are due to flaws in protocol implementation. The flaws are due to improper design and implementation of program logic by programmers. One of the widely used implementation of TLS is SChannel which is used in Windows operating system since its inception. We have used protocol state fuzzing to identify vulnerable and undesired state transitions in the state machine of the protocol for various versions of SChannel. The client as well as server components have been analyzed thoroughly using this technique and various flaws have been discovered in the implementation. Exploitation of these flaws under specific circumstances may lead to serious attacks which could disrupt secure communication. In this paper, we analyze state machine models of TLS protocol implementation of SChannel library and describe weaknesses and design flaws in these models, found using protocol state fuzzing.