Order by disorder and phase transitions in a highly frustrated spin model on the triangular lattice

Abstract: Frustration has proved to give rise to an extremely rich phenomenology in both quantum and classical systems. The leading behavior of the system can often be described by an effective model, where only the lowest-energy degrees of freedom are considered. In this paper we study a system corresponding to the strong trimerization limit of the spin 1/2 kagome antiferromagnet in a magnetic field. It has been suggested that this system can be realized experimentally by a gas of spinless fermions in an optical kagome lattice at 2/3 filling. We investigate the low-energy behavior of both the spin 1/2 quantum version and the classical limit of this system by applying various techniques. We study in parallel both signs of the coupling constant J since the two cases display qualitative differences. One of the main peculiarities of the J>0 case is that, at the classical level, there is an exponentially large manifold of lowest-energy configurations. This renders the thermodynamics of the system quite exotic and interesting in this case. For both cases, J>0 and J<0, a finite-temperature phase transition with a breaking of the discrete dihedral symmetry group D_6 of the model is present. For J<0, we find a transition temperature T^<_c/|J| = 1.566 +/- 0.005, i.e., of order unity, as expected. We then analyze the nature of the transition in this case. While we find no evidence for a discontinuous transition, the interpretation as a continuous phase transition yields very unusual critical exponents violating the hyperscaling relation. By contrast, in the case J>0 the transition occurs at an extremely low temperature, T^>_c ~= 0.0125 J. Presumably this low transition temperature is connected with the fact that the low-temperature ordered state of the system is established by an order-by-disorder mechanism in this case.