Optimal Transmit Filters for ISI Channels under Channel Shortening Detection

Abstract: We consider channels affected by intersymbol interference with reduced-complexity, mutual information optimized, channel-shortening detection. For such settings, we optimize the transmit filter, taking into consideration the reduced receiver complexity constraint. As figure of merit, we consider the achievable information rate of the entire system and with functional analysis, we establish a general form of the optimal transmit filter, which can then be optimized by standard numerical methods. As a corollary to our main result, we obtain some insight of the behavior of the standard waterfilling algorithm for intersymbol interference channels. With only some minor changes, the general form we derive can be applied to multiple-input multiple-output channels with intersymbol interference. To illuminate the practical use of our results, we provide applications of our theoretical results by deriving the optimal shaping pulse of a linear modulation transmitted over a bandlimited additive white Gaussian noise channel which has possible applications in the faster-than-Nyquist/time packing technique.