Entropic Conjugation Framework

• Entropic conjugation is a duality operator that links joint entropies, unifying metrics such as total correlation, dual total correlation, and others.
• It decomposes high-order dependencies into symmetric (total interdependence) and skew-symmetric (redundancy vs. synergy) components using an averaged conditional mutual information basis.
• Applications in spin systems, neuroscience, and statistical mechanics illustrate its potential for uncovering new metric directions, especially in larger systems (n > 5).

The entropic conjugation framework is a foundational mathematical structure for analyzing high-order interdependencies in collections of random variables, unifying a broad family of information-theoretic metrics and elucidating their structural relations. The framework centers on a canonical duality—entropic conjugation—over functionals representing multivariate joint entropies, yielding a principled taxonomy of high-order informational quantities characterized by symmetry and skew-symmetry. The approach is applicable across diverse domains, including statistical mechanics, neuroscience, and complex systems, where distinguishing low- and high-order effects is central.

1. Formal Definition of Entropic Conjugation

Let $X_1, \dots, X_n$ denote random variables indexed by $I_n = \{1, \ldots, n\}$, with joint entropies $H(\bm X^{\bm a})$ for every subset $\bm a \subseteq I_n$. The entropic conjugation operator $^*$ acts on single-term entropies as

$$\bigl( H(\bm X^{\bm a}) \bigr)^* = H(\bm X^{-\bm a}) - H(\bm X)$$

where $-\bm a$ is the complement of $\bm a$ in $I_n$. Linearity extends $^*$ to any functional $\phi(\bm X)$ expressible as $$\sum_{\bm a \subseteq I_n} \lambda_{\bm a} H(\bm X^{\bm a})$$, so

$$\left( \phi(\bm X) \right)^* = \sum_{\bm a \subseteq I_n} \lambda_{\bm a} \left( H(\bm X^{\bm a}) \right)^*.$$

For conditional mutual informations,

$$\bigl( I(\bm X^{\bm a}; \bm X^{\bm b} \mid \bm X^{\bm c}) \bigr)^* = I(\bm X^{\bm a}; \bm X^{\bm b} \mid \bm X^{-(\bm a \cup \bm b \cup \bm c)}).$$

This establishes a concrete involutive transformation within the space of entropy-based metrics.

2. Basis and Duality in High-Order Measures

A central role is played by the averaged conditional mutual information basis

$$u_k(\bm X) = \frac{1}{\binom{n}{k+1}} \binom{k+1}{2} \sum_{i<j,\,|\bm a|=k-1} I(X_i;X_j \mid \bm X^{\bm a}), \quad k = 1, \ldots, n-1.$$

Entropic conjugation yields a duality: $\left( u_k(\bm X) \right)^* = u_{n-k}(\bm X).$ The proof leverages average entropy coefficients $$r_k(\bm X) = \frac{1}{\binom{n}{k}} \sum_{|\bm a| = k} H(\bm X^{\bm a})$$ and their transformation properties under conjugation, together with the identity $u_k = -\left( r_{k+1} - 2r_k + r_{k-1} \right)$. This establishes a rigorous mapping between metrics targeting low-order and high-order interdependencies.

3. Mathematical Properties: Linearity, Involution, Symmetry Structures

The entropic conjugation operator $^*$ is linear on any entropy combination. It is involutive: $(\cdot)^{**}$ recovers the identity, i.e.,

$$\left( H(\bm X^{\bm a})^* \right)^* = H(\bm X^{\bm a}).$$

Symmetry properties are defined for any functional $\phi(\bm X)$:

• Symmetric if $\phi^* = \phi$
• Skew-symmetric if $\phi^* = -\phi$ Expressed via the $u_k$ basis (Te Sun Han’s decomposition) as $\phi(\bm X) = \sum_{k=1}^{n-1} c_k u_k(\bm X)$, symmetry corresponds to $c_k = c_{n-k}$ and skew-symmetry to $c_k = -c_{n-k}$. These properties undergird the classification of all entropy-driven high-order metrics.

4. Unification of Multivariate Information-Theoretic Metrics

All label-symmetric, dependency-zero functionals are representable as $\sum_k c_k u_k$. Notable metrics include:

• Total correlation (TC): $$\mathrm{TC}(\bm X) = \sum_{k=1}^{n-1} (n-k) u_k(\bm X)$$, with $\mathrm{TC}^* = \mathrm{DTC}$
• Dual total correlation (DTC): $$\mathrm{DTC}(\bm X) = \sum_{k=1}^{n-1} k u_k(\bm X)$$, with $\mathrm{DTC}^* = \mathrm{TC}$
• S-information ($\Sigma$): $\Sigma(\bm X) = n \sum_{k=1}^{n-1} u_k(\bm X)$, with $\Sigma^* = \Sigma$ (symmetric)
• TSE complexity: $$\mathrm{TSE}(\bm X) = \sum_{k=1}^{n-1} \frac{k(n-k)}{2} u_k(\bm X)$$, with $\mathrm{TSE}^* = \mathrm{TSE}$ (symmetric)
• O-information ($\Omega$): $\Omega(\bm X) = \sum_{k=1}^{n-1} (n-2k) u_k(\bm X)$, with $\Omega^* = -\Omega$ (skew-symmetric)
• Interaction information (II): $$\mathrm{II}(\bm X) = \sum_{k=1}^{n-1} (-1)^{k+1} \binom{n-2}{k-1} u_k(\bm X)$$, with $\mathrm{II}^* = (-1)^n \mathrm{II}$

TC and DTC form a conjugate pair; $\Sigma$ and TSE are symmetric measures of total interdependence; $\Omega$ is the unique linear skew-symmetric measure; II alternates symmetry class with $n$. This unifying structure facilitates systematic selection and interpretation of high-order informational metrics.

5. Dimensional Analysis and Gaps in Metric Space

The vector space $\mathcal{I}_n$ of all labelling-symmetric, dependency-zero functionals has dimension $n-1$. The entropic conjugation operator partitions $\mathcal{I}_n$ into $\lfloor n/2 \rfloor$ symmetric and $\lfloor (n-1)/2 \rfloor$ skew-symmetric directions. For $n \leq 5$, the cited six metrics span all possible directions:

• $n=3$: $\Sigma$ (symmetric), $\Omega$ (skew-symmetric)
• $n=4$: symmetric $\{\Sigma, II\}$; skew $\{\Omega\}$
• $n=5$: symmetric $\{\Sigma, TSE\}$; skew $\{\Omega, II\}$

For $n > 5$, additional directions emerge, representing an unexplored class of high-order metrics. Computational efficiency imposes further constraints: among symmetric and skew-symmetric metrics computable in $O(n)$ entropies, only $\Sigma$ and $\Omega$ satisfy this criterion.

6. Practical Implications and Illustrative Applications

The entropic conjugation framework enables any high-order measure to be decomposed into a symmetric part (overall interdependence strength) and a skew-symmetric part (balance of low- vs. high-order effects). In practical terms, symmetric measures track total interdependence, while skew-symmetric measures distinguish synergistic versus redundant regimes.

Applied to spin systems ($n=8$), where $X_i \in \{-1,1\}$ are distributed under Boltzmann statistics with pairwise couplings $J_{ij} \sim \mathcal{N}(\mu, \sigma^2)$, analysis of $(u_1, \ldots, u_7)$ via principal component analysis yields:

• The first principal component (correlated with $\Sigma$) represents total interdependence strength.
• The second principal component (aligned with $\Omega$) reflects the relative influence of low- and high-order interactions, separating ferromagnetic ($\mu > 0$), weak ($\mu \approx 0$), and frustrated ($\mu < 0$) regimes.

This decomposition provides immediate guidance: $\Sigma$ is optimal for balanced total interdependence analysis; $\Omega$ is optimal for balanced redundancy-versus-synergy analysis; their combination delivers rich profiling for complex systems.

7. Impact, Limitations, and Outlook

Entropic conjugation establishes a rigorous duality for Shannon-based high-order metrics, systematically exchanges low and high-order effects, and classifies all symmetric and skew-symmetric combinations up to moderate $n$. Existing measures are positioned within this taxonomy, revealing unexplored metric directions for $n > 5$ and informing computational feasibility. A plausible implication is that new high-order metrics—beyond those explicitly characterized—remain to be discovered for large systems. The framework provides both theoretical and practical roadmaps for the analysis of multivariate dependencies in complex physical and biological systems (Rosas et al., 2024).