Data Hardness Transferability

Updated 31 January 2026

The paper introduces data hardness transferability as the extent to which intrinsic data difficulty—quantified by metrics like conditional entropy and KL divergence—remains effective across various tasks and architectures.
Empirical benchmarks in supervised classification, MLIP, and cryptographic reductions reveal strong correlations between hardness measurements and transfer error or security degradation.
New algorithms and frameworks guide the selection of hard data subsets and hardness-preserving reductions, offering actionable insights to enhance learning stability and system security.

Transferability of data hardness refers to the extent to which the property of "hardness"—the intrinsic difficulty posed by data points, tasks, or distributions—persists or translates when transferred across models, tasks, problem domains, or learning architectures. This concept appears across supervised classification, cryptographic reductions, transfer learning, and machine-learned interatomic potentials (MLIPs), each context demanding rigorous definitions and metrics for both "hardness" and its transferability. Central to this discussion are new information-theoretic and empirical frameworks measuring data and task hardness, quantifying transferability of such hardness, and establishing both theoretical and practical limits.

1. Definitions and Information-Theoretic Frameworks

The foundational approach to data hardness and its transferability models the learning problem in terms of random variables over fixed input sequences. For supervised classification, consider two tasks defined by label sequences $Z = (z_1,\ldots,z_n)$ (source) and $Y = (y_1,\ldots,y_n)$ (target), both indexed over the same list of inputs $X = (x_1,\ldots,x_n)$ . The empirical joint distribution $\hat{P}(y,z)$ is estimated directly from the label assignments.

Conditional entropy $H(Y|Z) = -\sum_{y,z} \hat{P}(y,z) \log \frac{\hat{P}(y,z)}{\hat{P}(z)}$ plays a central role. It quantifies the expected uncertainty in the target labels given the source labels, providing a scalar measurement of transferability—lower $H(Y|Z)$ indicates greater alignment and greater potential for successful transfer between tasks.

For task hardness, with a trivial (constant) source task $C$ , $H(Z|C)$ approximates the intrinsic entropy of a task's labels. This enables a label-only, solution-agnostic estimate of task difficulty without requiring any trained classifiers or feature representations (Tran et al., 2019).

In cryptographic search problems, relative entropy (Kullback–Leibler divergence) is used to formalize "hardness" of generating or simulating solution–instance pairs. Hardness in KL, and its blockwise decompositions (pseudoentropy, inaccessible entropy), encode the resistance of a task or function to simulation or inversion, and underpin modular reductions relevant to the transferability of computational hardness in cryptographic constructions (Agrawal et al., 2019).

2. Measurement Methodologies and Algorithms

For classification tasks, the complete pipeline for estimating task hardness and transferability consists of two principal computational phases:

Two-pass Algorithm: (1) Iterate through the $n$ label pairs $(y_i, z_i)$ to count co-occurrences, populating a $|\mathcal{Y}| \times |\mathcal{Z}|$ matrix; (2) derive joint and marginal empirical probabilities, then accumulate the conditional entropy $H(Y|Z)$ . The complete runtime is $O(n + |\mathcal{Y}| \cdot |\mathcal{Z}|)$ —feasible even for datasets with $n > 10^5$ and $|\mathcal{Z}| \sim 10^3$ (Tran et al., 2019).
Task Pair Evaluation: To empirically validate transferability, models (e.g., ResNet-18) are trained on source tasks, features are frozen, and linear classifiers or SVMs are trained on target tasks. Empirical transfer error is then compared to the precomputed $H(Y|Z)$ .

In MLIP transfer scenarios, data hardness is characterized at the configuration level via: - Committee variance: For a configuration $X$ , $\sigma_F^2(X)$ quantifies prediction spread among a model ensemble—high variance denotes hard points. - Trajectory failure time $t_{\text{stab}}$ and thermodynamic error metrics (KL divergence between predicted and reference distributions)—these operationalize the dynamic and collective consequences of unlearned hardness in training data (Niblett et al., 2024).

In cryptography, hardness-preserving reductions utilize explicit algorithmic constructions—such as cuckoo hashing for domain extension—to guarantee that the hardness of a primitive is preserved with respect to a new domain, often via tight reduction arguments (Berman et al., 2021).

3. Empirical Patterns and Validation across Domains

Extensive empirical benchmarks confirm the strong correlation between theoretical measures of data or task hardness and empirical transferability or difficulty:

Supervised Classification: Across CelebA (41 tasks), AwA2 (135 tasks), and CUB-200 (512 tasks)—totaling 437 tasks and $\sim 10^5$ task pairs—Pearson correlations between conditional entropy $H(Y|Z)$ and transfer error exceed $r \approx +0.85$ to $+0.90$ (all $p \ll 10^{-8}$ ). For task hardness, $H(Z|C)$ correlates $r \approx +0.80$ –$0.85$ with final test error (Tran et al., 2019).
Transfer Learning via Hard Subsets: Metrics like LEEP and NCE, when computed only on the hardest $20$– $40\%$ of target samples (scored via class-agnostic or class-specific criteria), yield $+130\%$ average improvement in accuracy correlation (LEEP), $+29\%$ (NCE), and up to $+182\%$ in segmentation benchmarks. The improvement is most pronounced on the hardest data subsets and is robust to the source architecture used to compute hardness (Menta et al., 2023).
MLIP Data Reuse: Adding "hard" configurations (volume scans, single-molecule distortions) to the training set for DeePMD or similar NN architectures increases simulation stability time $t_{\rm stab}$ from sub-ps to $\sim 100$ ps, even before active learning. In contrast, active-learned frames from one MLIP architecture (e.g., GAP) offer minimal transfer benefit to structurally different models (e.g., DeePMD, MACE), as their sampled "holes" are often model-specific (Niblett et al., 2024).
Cryptographic Reductions: Hardness-transfer theorems (hardness in KL or via cuckoo-hashing domain extension) show that adversarial advantage or entropy gaps degrade only negligibly, even under blockwise and online decompositions (Agrawal et al., 2019, Berman et al., 2021).

4. Theoretical Bounds and Central Analytic Results

The analytic connection between transferability and hardness is formalized as a lower bound: for any cross-entropy trained source model transferred to a target, the expected transferred log-likelihood is at least the source likelihood minus the conditional entropy, $\ell_Y(\text{transferred}) \geq \ell_Z(\text{source}) - H(Y|Z)$ . This demonstrates that smaller $H(Y|Z)$ strictly bounds achievable target performance, and no model-free metric will improve upon this limit without additional side information (Tran et al., 2019).

In transfer learning, HASTE (Hard Subset TransfErability) guarantees that hard-subset modifed metrics (e.g., HASTE-LEEP) always lie between the optimal average log-likelihood achievable with retraining on the subset and the negative hard-subset conditional entropy, yielding a theoretically sound and tighter sandwich of the true fine-tuned accuracy than global metrics (Menta et al., 2023).

Cryptographic reductions are underpinned by modular proofs where KL-hardness implies both next-block pseudoentropy and next-block inaccessible entropy, with all parameters scaling only logarithmically in block size and polynomially in time overhead. These proofs formalize the transfer of computational hardness across problem structures, one-way functions, and complex primitives (Agrawal et al., 2019).

5. Transfer Mechanisms: Architecture and Data Specificity

Transferability of data hardness is highly sensitive to the match between the probing mechanism that generates hard configurations and the architecture under consideration.

Model-Agnostic Hardness: Configurations probing universal failure modes (e.g., volume scans for high-density overlap, isolated-molecule distortions) act as model-agnostic hardness probes. When added to the training set, these configurations enhance robustness and stability across disparate MLIP architectures, forming an “architecture-blind” data-hardness transfer (Niblett et al., 2024).
Model-Specific Hardness: Active learning based on committee variance or other error signals from a specific architecture (e.g., GAP) typically generates data addressing idiosyncratic “holes” in that architecture’s sampled feature space. Such configurations tend not to transfer efficiently to other architectures, since their error landscape is not shared (Niblett et al., 2024). A plausible implication is that data-hardened by one model’s active learning can have limited cross-architectural value unless its error modes are universal.
Hardness in Task Transfer: In supervised classification, the conditional entropy $H(Y|Z)$ is independent of model details, providing a universal predictor of transferability rooted solely in label statistics (Tran et al., 2019). However, achieved transfer accuracy can vary based on representational alignment between frozen source features and target label geometry.
Cryptographic Transformations: Hardness-preserving reductions (e.g., cuckoo-hashing domain extension) are rigorously constructed to guarantee transferability of security guarantees (hardness). Parameters can be tuned so that security degradation is negligible, and constructions are black-box by design (Berman et al., 2021).

6. Practical Recommendations and Guidelines

For supervised classification, label-only computation of $H(Y|Z)$ or $H(Z|C)$ is sufficient for robust screening of source–target pairs without any training. This allows efficient batch selection of promising transfer pipelines (Tran et al., 2019).
In transfer learning, incorporating only the hardest 20–40% of the target examples into transferability scoring yields both tighter correlation with actual accuracy and better discriminative power for model or dataset selection. Both class-specific (Mahalanobis) and class-agnostic (layerwise cosine) scores are effective; selection of hard-subset size in this range yields optimal results (Menta et al., 2023).
For MLIPs, augment initial training sets with a minimal “starter kit” of classical MD data, several volume scans, and isolated-molecule distortions, rather than relying on model-specific active-learned points alone. This expedites convergence, improves cross-architecture transferability, and mitigates high-energy catastrophic failures during subsequent active learning (Niblett et al., 2024).
Whenever transferring data between ML learning systems, empirically evaluate whether hard configurations stem from universal physical or statistical properties, or model-specific error regions, and prioritize the former for improved transferability.

In cryptographic theory, the formal transferability of hardness underpins the security of pseudorandom generators, statistically hiding commitment schemes, and universal one-way hash functions. From a single on-average hard search problem or one-way function, one can, via structured KL-divergence arguments, derive pseudoentropy and inaccessible entropy gaps sufficient for constructing high-assurance cryptographic primitives. The only substantial loss in quality occurs as a logarithmic penalty in block-based simulations necessary for practical reductions (Agrawal et al., 2019).

In foundation MLIP and fine-tuning contexts, the observation that training on “hard” data not only improves stability and accuracy within the trained domain but also enhances generalization to unseen but structurally similar systems suggests an underlying link between data hardness and model extrapolation ability. This suggests a research direction focused on universal hardness probes as a foundation for broad generalization and transfer in physical and chemical learning systems (Niblett et al., 2024).

Transferability of data hardness, in its various rigorous instantiations, provides a unifying lens for understanding not just cross-task or cross-model generalization, but also the limitations imposed by intrinsic data complexity, architecture-specific representations, and the theoretical possibilities and bounds inherent in information-theoretic and cryptographic perspectives.