Phonon mode calculations and Raman spectroscopy of the bulk-insulating topological insulator BiSbTeSe$_2$

Abstract: The tetradymite compound BiSbTeSe$2$, is one of the most bulk-insulating three-dimensional topological insulators, which makes it important in the topological insulator research. It is a member of the solid-solution system Bi${2-x}$Sb$x$Te${3-y}$Se$_y$, for which the local crystal structure, such as the occupation probabilities of each atomic site, is not well understood. We have investigated the temperature and polarization dependent spontaneous Raman scattering in BiSbTeSe$_2$, revealing a much higher number of lattice vibrational modes than predicted by group theoretical considerations for the space group R$\bar{3}$m corresponding to an ideally random solid-solution situation. The density functional calculations of phonon frequencies show a very good agreement with experimental data for parent material Bi$_2$Te$_3$, where no disorder effects were found. In comparison to Bi$_2$Te$_3$ the stacking disorder in BiSbTeSe$_2$, causes a discrepancy between theory and experiment. Combined analysis of experimental Raman spectra and DFT calculated phonon spectra for different types of atomic orders showed coexistence of different sequences of layers in the material and that those with Se in the center and a local order of Se-Bi-Se-Sb-Te, are the most favored.