Topological Phase Transitions in a Hybridized Three-Dimensional Topological Insulator

Abstract: As the thickness of a three-dimensional (3D) topological insulator (TI) becomes comparable to the penetration depth of the surface states, quantum tunneling between surfaces turns their gapless Dirac electronic structure into a gapped surface state. Analytical formulation suggests that the hybridization gap scales exponentially with decrease in number of layers while the system oscillates between topologically trivial and non-trivial insulators. This work explores the transport properties of a 3D TI in the inter-surface hybridization regime. By experimentally probing the hybridization gap as a function of BiSbTeSe2 thickness using three different methods, we map the crossover from the 3D to 2D state. In the 2D topological state, we observe a finite longitudinal conductance at ~2e2/h when the Fermi level is aligned within the surface gap, indicating a quantum spin Hall (QSH) state. Additionally, we study the response of trivial and non-trivial hybridization gapped states modulated by external out-of-plane magnetic and electric fields. Our revelations of surface gap-closing and/or reopening features are strongly indicative of topological phase transitions (TPTs) in the hybridization gap regime, realizing magnetic/electric field switching between band insulating and QSH states with immense potential for practical applications.