Information Structures for Feedback Capacity of Channels with Memory and Transmission Cost: Stochastic Optimal Control & Variational Equalities-Part I

Abstract: The Finite Transmission Feedback Information (FTFI) capacity is characterized for any class of channel conditional distributions $\big{{\bf P}{B_i|B^{i-1}, A_i} :i=0, 1, \ldots, n\big}$ and $\big{ {\bf P}{B_i|B_{i-M}^{i-1}, A_i} :i=0, 1, \ldots, n\big}$, where $M$ is the memory of the channel, $B^n {\stackrel{\triangle}{=}} {B_j: j=\ldots, 0,1, \ldots, n}$ are the channel outputs and $A^n{\stackrel{\triangle}{=}} {A_j: j=\ldots, 0,1, \ldots, n}$ are the channel inputs. The characterizations of FTFI capacity, are obtained by first identifying the information structures of the optimal channel input conditional distributions ${\cal P}{[0,n]} {\stackrel{\triangle}{=}} \big{ {\bf P}{A_i|A^{i-1}, B^{i-1}}: i=0, \ldots, n\big}$, which maximize directed information. The main theorem states, for any channel with memory $M$, the optimal channel input conditional distributions occur in the subset satisfying conditional independence $\stackrel{\circ}{\cal P}{[0,n]}{\stackrel{\triangle}{=}} \big{ {\bf P}{A_i|A^{i-1}, B^{i-1}}= {\bf P}{A_i|B{i-M}^{i-1}}: i=1, \ldots, n\big}$, and the characterization of FTFI capacity is given by $C_{A^n \rightarrow B^n}^{FB, M} {\stackrel{\triangle}{=}} \sup_{ \stackrel{\circ}{\cal P}{[0,n]} } \sum{i=0}^n I(A_i; B_i|B_{i-M}^{i-1}) $. The methodology utilizes stochastic optimal control theory and a variational equality of directed information, to derive upper bounds on $I(A^n \rightarrow B^n)$, which are achievable over specific subsets of channel input conditional distributions ${\cal P}_{[0,n]}$, which are characterized by conditional independence. For any of the above classes of channel distributions and transmission cost functions, a direct analogy, in terms of conditional independence, of the characterizations of FTFI capacity and Shannon's capacity formulae of Memoryless Channels is identified.