Data Mixture Laws & Optimization

• Data mixture laws are mathematical frameworks that define how a system's performance responds to varied proportions of data sources, guiding model scaling and domain adaptation.
• The topic covers diverse optimization methods including convex programming, minimax duality, and scaling law frameworks to rigorously tune mixture loss functions.
• Recent advances integrate algorithmic design with empirical guidelines to achieve dynamic mixture optimization and robust performance across applications, from language models to physical systems.

A data mixture law specifies how the performance of a learning system responds to the proportions of various data sources in the training mixture. Optimization of these laws is central to modern statistical learning, unsupervised representation modeling, and domain adaptation, as well as in engineering control of physical mixtures. The mathematical theory spans convex and nonconvex programming, function space minimax duality, moment and scaling law frameworks, as well as algorithmic design for efficient mixture identification. This article surveys the principal theoretical and algorithmic foundations with reference to recent advances in data mixture law formulation and optimization.

1. Mathematical Formulations of Data Mixture Laws

The most general formulation treats a set of source distributions $\{p_i\}_{i=1}^k$, with a mixture vector $w \in \Delta^{k}$ (the probability simplex), and aims to optimize a loss functional $L(w)$. Typical scenarios:

• Parametric mixture matching: Given a family $\{p(\cdot|\theta): \theta \in \Theta\}$ and moments $\{y_\alpha\}$ of a target $\mu$, search for a discrete mixing measure $\varphi = \sum_{i=1}^r w_i\delta_{\theta_i}$ minimizing $W_2^2(\mu,\nu_\varphi)$ or $\|\mu - \nu_\varphi\|_{TV}$ (Đurašinović et al., 26 Sep 2025).
• Finite mixture MLE: Optimize $p\in\Delta_K$ (possibly with shape constraints) for $$f(p) = -\frac{1}{N}\sum_{j=1}^N \log\left(\sum_{k=1}^K p_k \psi_k(X_j)\right)$$, possibly subject to convex polyhedral constraints (Wang et al., 2021).
• Scaling law–based mixture loss: For foundation models, fit a law $L(N, D, w)$ relating model size $N$, data size $D$, and mixture $w$, for loss prediction, and select $w^*$ to minimize $L$ (Shukor et al., 12 Jul 2025, Kang et al., 2 Oct 2025, Ye et al., 2024).
• Bi-level objectives: Find $$\alpha^* = \arg\min_{\alpha\in\Delta^P} E_{(x,y)\sim d_t}[\ell(f_{\theta^*(\alpha)}(x),y)]$$, subject to $$\theta^*(\alpha) = \arg\min_\theta E_{(x,y)\sim \sum_p \alpha_p p_p}[\ell(f_\theta(x),y)]$$ (e.g. MixMin (Thudi et al., 14 Feb 2025)).
• DRO in function space: For $\{P_g\}_{g=1}^G$, minimize $\sup_{g} \mathbb{E}_{P_g}[\ell(f(X), Y)]$, or, equivalently, maximize over $\lambda \in \Delta_G$ the (concave) risk of the mixture (Thudi et al., 2024).
• Dynamic (online) laws: Adapt $w^t$ during training using estimated data-mixing laws of the type $$\ell_j^{t+1}(w^t) = c_j + b_j \sigma(-\sum_i A_{ji}w_i^t)$$, and perform mirror descent or exponentiated-gradient steps (Chen et al., 2024).

Mixture law functional forms range from exponential, power-law, and rational to general nonparametric regressors—often motivated by empirical scaling behavior in model performance (Ye et al., 2024, Shukor et al., 12 Jul 2025).

2. Convexity, Nonconvexity, and Duality

The convexity structure in mixture optimization is pivotal for tractability:

• Mixing over universal approximation function spaces: For cross-entropy or MSE losses, group DRO over $L^\infty$ reduces to a convex program over the simplex, with parameter-function minimax duality (Sion's theorem) holding even in nonparametric regimes (Thudi et al., 2024).
• Bilevel collapse: For rich enough hypothesis classes and no covariate shift, the bilevel objective in MixMin is provably convex in $\alpha$, reducing optimization to a single convex minimization (Thudi et al., 14 Feb 2025).
• Finite mixture MLE: The negative log-likelihood for discrete mixtures is convex in $p$ over the simplex and shape-constraint polyhedra, enabling efficient cubic regularized Newton methods with global convergence (Wang et al., 2021).
• Minimax in data-adaptive domain adaptation: Convex–nonconcave compositional minimax problems arise in multi-source adaptation, with the outer minimization in mixture weights being strongly convex and the inner maximization possibly nonconcave (Deng et al., 2023).

In non-Euclidean settings (e.g., moment space, measures), semidefinite programming (SDP) relaxations are employed: hierarchies of SDPs yield asymptotic convergence to the optimum with finite extraction of K-atomic minimizers if flatness is detected (Đurašinović et al., 26 Sep 2025).

3. Data Mixture Law Fitting and Scaling Laws

Empirical work demonstrates that mixture-performance laws admit concise, predictable forms:

• Exponential and linear mixing laws: Cross-domain loss responds exponentially or linearly to mixture weightings, as observed in DML and Aioli frameworks (Chen et al., 2024).
• Chinchilla-style power-law scaling: The loss for LLMs, vision models, and MoEs can be modeled as $$L = E + A/N^\alpha + B/D^\beta + 1/\left(\sum_i C_i h_i^{\gamma_i}\right)$$, enabling mixture optimization independent of or interacting with dataset/model scale (Shukor et al., 12 Jul 2025, Kang et al., 2 Oct 2025, Krajewski et al., 2024, Ludziejewski et al., 7 Feb 2025).
• Per-domain and per-domain-pair scaling: For each domain $k$, per-domain scaling laws $L_k(n) = \epsilon_k + \beta_k n^{-\alpha_k}$ yield instantaneous learning curves that guide dynamic mixture adjustment (as in ADO (Jiang et al., 2024)).
• Regression and nonparametric fitting: Methods such as RegMix use LightGBM or linear regression over mixture-loss pairs $(m_j, f(m_j))$ from small proxy models to predict validation loss on unseen mixtures (Liu et al., 2024). Nonparametric regressors are required when mixture-response is sufficiently idiosyncratic (Yen et al., 26 Mar 2025).
• Mixture-of-experts scaling: Granularity $G$ and expert count $E$ determine MoE scaling laws, allowing for memory–compute-aware hyperparameter selection (Krajewski et al., 2024, Ludziejewski et al., 7 Feb 2025).

Careful calibration and pilot experimentation at small scale is central to robust parameter identification for scaling-law-based approaches (Shukor et al., 12 Jul 2025, Yen et al., 26 Mar 2025).

4. Optimization Algorithms for Data Mixtures

Optimization techniques bifurcate along static–dynamic and offline–online lines:

• Offline static mixture optimization: Includes convex or bilevel minimization (MixMin (Thudi et al., 14 Feb 2025), function-fitting via regression or scaling law fitting (Ye et al., 2024, Shukor et al., 12 Jul 2025, Liu et al., 2024)). Mirror descent and exponentiated gradient are standard for the simplex (Thudi et al., 14 Feb 2025, Thudi et al., 2024).
• Proxy-based regression approaches: Use ensembles of proxies/small models on sampled mixtures to fit regressors or surrogates $\hat{L}(w)$, then optimize/interpolate over the simplex (Liu et al., 2024).
• Dynamic online adaptation: Algorithms such as ADO (Jiang et al., 2024), Aioli (Chen et al., 2024), and ODM maintain and continually update data group weights $w^t$, informed by per-domain scaling curves or linear/exponential local response models, optimizing to maximize instantaneous expected learning gain or validation improvement.
• Bayesian optimization: Data mixture selection as black-box optimization, integrating multi-fidelity (proxy and full-scale) evaluations, is tackled via Gaussian process surrogates with acquisition functions emphasizing cost-sensitive exploration (ADMIRE-BayesOpt (Chen et al., 15 Aug 2025), MFMS-GP (Yen et al., 26 Mar 2025)).
• Semidefinite programming and moment extraction: Population mixture problems with limited moment information are handled via hierarchies of SDPs with moment extraction for finite atomicity under the Curto–Fialkow condition (Đurašinović et al., 26 Sep 2025).

Bandit approaches and meta-learning schemes have also been developed for regimes with large source numbers, uncertain validation, or non-stationary data composition.

5. Empirical Laws and Practical Guidelines

Convergent empirical findings emerge across approaches and domains:

• Synthetic–natural mixture sweet-spot: For LLM pretraining, mixtures with $\sim$30% rephrased synthetic data frequently minimize validation loss, delivering 5–10$\times$ faster convergence to a fixed loss than pure natural data (Kang et al., 2 Oct 2025).
• Proxy scale-invariance: Data-mixture weights selected using small (proxy) models generalize robustly to larger-scale downstream training, barring radical covariate shift (Thudi et al., 14 Feb 2025, Liu et al., 2024, Chen et al., 15 Aug 2025).
• Per-domain scaling as online signal: ADO demonstrates that cheap, online-fitted per-domain scaling exponents are sufficient to drive dynamic weighting, without proxy runs or Hessian computation, matching or exceeding proxy-heavy workflows (Jiang et al., 2024).
• Mixture regularization for stability: Injecting modest fractions ($1\%$) of pretraining data into finetuning batches prevents catastrophic forgetting with negligible loss in adaptation quality (Bethune et al., 9 Feb 2025).
• MoE and granularity scaling: MoEs with non-default granularity parameters are always compute-optimal if the routing overhead is controlled, and for fixed memory, MoEs can attain a lower irreducible loss than any dense parameter-matched model (Krajewski et al., 2024, Ludziejewski et al., 7 Feb 2025).
• Surveyed trade-offs: Offline methods are reusable but risk proxy–target gaps; online methods adapt but add compute overhead; function-fitting offers a middle ground if the parametric assumption matches observed mixture laws (Liu et al., 27 May 2025).
• Challenges: Scaling laws may be brittle outside the calibration regime, and fully dynamic or curriculum-based schedules (non-static mixture) remain less theoretically understood (Yen et al., 26 Mar 2025).

6. Applications Beyond Language Modeling

These frameworks are applicable in statistical estimation, clustering, chemical mixture design, and robust supervised learning:

• Moment-based mixture identification: Semidefinite relaxations for moment-constrained mixture approximation provide quantifiable convergence and atomicity guarantees, with application in cluster analysis and parametric density estimation (Đurašinović et al., 26 Sep 2025).
• Shape-constrained MLE: Convex optimization with polyhedral shape constraints enables structured mixture estimation in settings requiring monotonicity, unimodality, or concavity (Wang et al., 2021).
• Physical-chemical mixture optimization: Differentiable physics–machine learning hybrids (e.g., DiffMix) leverage learnable mixture laws for closed-loop robotic materials optimization, guided by physical parameterizations of mixture thermodynamics and transport (Zhu et al., 2023).
• Distributional robustness: Explicit mixture optimization in function space (MixMax) is provably equivalent to group DRO for broad loss classes, providing efficient minimax procedures for fairness and robustness-critical applications (Thudi et al., 2024).

Synthetic–real mixture engineering, robust domain adaptation, and general model–mixture co-design across modalities and physical systems are all beneficiaries of this mathematical and algorithmic toolbox.

7. Open Problems and Future Directions

Several critical open challenges are highlighted:

• Dynamic and curriculum mixtures: Development of functional laws and algorithms for non-stationary mixtures, where proportions change adaptively or in response to feedback.
• High-dimensional domain decomposition: Automatic discovery and scalable optimization of fine-grained domain mixtures, especially as $k$ grows large (Liu et al., 27 May 2025).
• Robustness under covariate shift: Extending convexity and transfer principles to non-shared input distributions and non-convex loss functions (Thudi et al., 14 Feb 2025, Deng et al., 2023).
• Integration with end-task metrics: Bridging mixture law fidelity on validation loss to concrete downstream utility measures.
• Refined theoretical guarantees: Sharp regret and convergence bounds for dynamic, hierarchical, and online mixture learning methods.
• Scalable functional fitting and uncertainty quantification: Improved nonparametric and probabilistic extrapolation schemes for black-box response surfaces, incorporating model uncertainty and epistemic error (Yen et al., 26 Mar 2025, Chen et al., 15 Aug 2025).

These lines of inquiry are essential for principled and efficient design of data mixtures in increasingly complex, dynamic, and interdisciplinary learning systems.