Success probability of the Babai estimators for box-constrained integer linear models

Abstract: In many applications including communications, one may encounter a linear model where the parameter vector $\hbx$ is an integer vector in a box. To estimate $\hbx$, a typical method is to solve a box-constrained integer least squares (BILS) problem. However, due to its high complexity, the box-constrained Babai integer point $\x^\sBB$ is commonly used as a suboptimal solution. In this paper, we first derive formulas for the success probability $P^\sBB$ of $\x^\sBB$ and the success probability $P^\sOB$ of the ordinary Babai integer point $\x^\sOB$ when $\hbx$ is uniformly distributed over the constraint box. Some properties of $P^\sBB$ and $P^\sOB$ and the relationship between them are studied. Then, we investigate the effects of some column permutation strategies on $\P^\sBB$. In addition to V-BLAST and SQRD, we also consider the permutation strategy involved in the LLL lattice reduction, to be referred to as LLL-P. On the one hand, we show that when the noise is relatively small, LLL-P always increases $P^\sBB$ and argue why both V-BLAST and SQRD often increase $P^\sBB$; and on the other hand, we show that when the noise is relatively large, LLL-P always decreases $P^\sBB$ and argue why both V-BLAST and SQRD often decrease $P^\sBB$. We also derive a column permutation invariant bound on $P^\sBB$, which is an upper bound and a lower bound under these two opposite conditions, respectively. Numerical results demonstrate our findings. Finally, we consider a conjecture concerning $\x^\sOB$ proposed by Ma et al. We first construct an example to show that the conjecture does not hold in general, and then show that it does hold under some conditions.