Model-free Analysis of Scattering and Imaging Data with Escort-Weighted Shannon Entropy and Divergence Matrices

Abstract: We demonstrate a model-free data analysis framework that leverages escort-weighted Shannon Entropy and several divergence matrices to detect phase transitions in scattering and imaging datasets. By establishing a connection between physical entropy and informational entropy, this approach provides a sensitive method for identifying phase transitions without an explicit physical model or order parameter. We further show that pairwise divergence matrices, including Kullback-Leibler divergence, Jeffrey Divergence, Jensen-Shannon Divergence and antisymmetric Kullback-Leibler divergence, provide more comprehensive measures of statistical changes than scalar entropy alone. Our approach successfully detects the onset of both long- and short-range order in neutron and X-ray scattering data, as well as a non-trivial phase transition in magnetic skyrmion lattices observed through Lorentz-transition electron microscopy. These results establish a framework for automated, model-free analysis of experimental data with broad applications in materials science and condensed matter physics.

• The paper introduces a novel model-free framework using escort-weighted Shannon entropy and divergence matrices to detect both pronounced and subtle phase transitions in experimental datasets.
• It leverages escort distributions with an adjustable parameter to enhance sensitivity and denoise features in scattering and imaging data effectively.
• The approach is validated on neutron, X-ray, and TEM datasets, demonstrating robustness and broad applicability across different experimental modalities.

Model-Free Informational Entropy Analysis of Scattering and Imaging Data

Theoretical Foundations

This work constructs a model-free framework for analyzing experimental scattering and imaging datasets by leveraging escort-weighted Shannon entropy and a suite of divergence matrices. The foundation rests on a formal connection between physical (thermodynamic) entropy and informational (Shannon) entropy, consistent with Jaynes’ and Landauer’s formulations. The authors implement the escort distribution, parametrized by an artificial temperature, to tune the sensitivity of information measures to features within data distributions.

For each dataset—normalized to a probability space—the framework computes Shannon entropy and then systematically explores the space of escort-weighted entropies. The escort transformation emphasizes different regions of the probability landscape: for exponent $n>1$, the method accentuates high-probability, sharply localized regions, whereas $n<1$ increases sensitivity to diffuse features. Crucially, these mappings remain monotonic with respect to the ordering of microstates, yielding a robust data-driven representation of order-disorder phenomena.

Relative entropy (Kullback-Leibler divergence, KLD) and its symmetric (Jeffrey, Jensen-Shannon) and antisymmetric (a-KLD) extensions are used to construct divergence matrices quantifying statistical changes between datasets as external parameters vary. These matrices reveal the onset, evolution, and boundaries of emergent phases by identifying block structures and abrupt transitions.

Application to Experimental Datasets

The framework is systematically validated on representative neutron diffraction data (Eu$_3$Sn$_2$S$_7$), X-ray scattering data (Cd$_2$Re$_2$O$_7$), and real-space Lorentz-TEM imaging data (Fe$_3$GeTe$_2$ skyrmions). In neutron diffraction, the analysis identifies both long- and short-range magnetic ordering, with divergence matrices displaying low-divergence blocks below the antiferromagnetic transition temperature and revealing features not directly visible in intensity data. Field-induced transitions appear as well-defined boundaries in divergence matrices, with escort-weighted entropy (e.g., $n=2$) significantly improving sensitivity over standard Shannon entropy.

In the case of X-ray scattering datasets, the technique identifies multiple phase transitions—including subtle ones—across vast, high-dimensional data acquired with modern pixelated detectors. The antisymmetric KLD is found to be particularly sensitive to fine structure, revealing up to four distinct phase transitions, fitting well with previously reported structural phase boundaries and corresponding to nontrivial changes in short-range order and diffuse scattering.

For real-space TEM imaging of skyrmion lattices, the method generalizes beyond reciprocal space. Escort-weighted entropy captures the transition from disordered to partially ordered skyrmion phases; divergence matrices exhibit block structure marking the evolution and crossover of magnetic textures, despite the absence of direct correspondence with traditional order parameters.

Numerical and Methodological Significance

The analysis framework is characterized by several strong features:

• Detection of Known and Subtle Phase Transitions: The entropy and divergence metrics sensitively reveal both pronounced and subtle transitions in magnetic and structural order across all tested probe modalities.
• Enhanced Sensitivity via Escort Distributions: Escort-weighted entropy with appropriate $n$ substantially denoises and sharpens transition contrast, outperforming scalar entropy measures in identifying weak-order transitions, especially in data with significant background or incommensurability.
• Generality and Robustness: Applicability is demonstrated on neutron, X-ray, and electron microscopy datasets, encompassing both reciprocal and real space and multiple physical observables.

Notably, divergence matrices not only locate known transitions but also provide sensitivity to experimental artifacts (e.g., background or instrument changes), highlighting their comprehensiveness but also pointing to the necessity for careful interpretation in the presence of uncontrolled nonphysical variables.

Practical and Theoretical Implications

This framework introduces a model-free, unsupervised, and automatable approach that integrates seamlessly into high-throughput experimental paradigms, complementing and enhancing conventional model-dependent analyses. With the proliferation of large-area detectors and dramatically increasing data rates, its ability to flag candidate phase transitions without a priori order parameters or explicit structure models is highly significant.

The methodology can be used as a rapid, on-the-fly indicator system within measurement pipelines or as input for subsequent machine learning workflows requiring high-quality, unsupervised feature extraction. The artificial temperature parameter provides a form of regularization and "contrast tuning," effectively acting as a diagnostic for measurement efficiency.

Theoretically, the mapping reinforces the paradigm that many physical transitions manifest as detectable statistical changes in data distributions—regardless of the underlying order parameter—and that information theoretic quantities suffice for robust detection. This perspective aligns with generalized entropic measures gaining importance in quantum information and complex systems.

Future Directions

Several open directions emerge from this work. Future development includes integration of this analysis into real-time measurement environments for adaptive experiment control, deeper theoretical exploration of the artificial temperature’s meaning, and targeted deployment in systems where order parameters remain theoretically or experimentally inaccessible (e.g., topologically ordered, glassy, or hidden-order materials).

Extensions to higher-order entropy measures and more complex divergence structures may yield additional insight into hierarchical and multifractal phenomena. Machine learning methods can be hybridized with this framework for further automation and discovery in massive experimental datasets.

Conclusion

The escort-weighted information entropy and divergence matrix framework demonstrated here constitutes a robust, model-free statistical tool for the analysis of scattering and imaging datasets. Its capacity to detect both conventional and subtle transitions without explicit physical models, and its compatibility with modern high-throughput experimental data, marks it as a valuable addition to contemporary condensed matter physics and materials science data analysis. The approach provides a flexible, generalizable pathway for the unsupervised identification of phase transitions and can be further developed for inline data assessment, exploration of hidden order, and integration with advanced data-driven discovery platforms.