Simulation-Based Composite Metrics

• Simulation-based composite metrics are performance measures constructed from simulated, analytic, and empirical data to estimate system-level properties.
• They enable calibration of inaccessible parameters and quantify trade-offs via Monte Carlo adjustments, moment-matching, and surrogate modeling.
• Applications span composite likelihood inference, computational materials design, and resource scheduling, ensuring reliable risk and performance assessments.

A simulation-based composite metric is any performance or validation metric for a complex, structured, or composite system that is explicitly constructed or adjusted using simulated data. This paradigm is central to fields ranging from composite likelihood inference and engineered material design to validation of composite cyber-physical systems. Simulation-based composite metrics provide a principled approach for estimating or calibrating otherwise inaccessible parameters, quantifying trade-offs among competing objectives, or certifying system reliability when only partial experimental or real-world data are available.

1. Definition and General Framework

Simulation-based composite metrics formalize system-level performance or evidence quantification by synthesizing heterogeneous sources of information—often from simulations, analytic models, and limited empirical observations. The metric may serve multiple roles:

• Estimator of effective properties (e.g., elasticity, conductivity) in multi-phase materials constructed from lower-scale simulations.
• Test statistic adjustment in inferential settings, utilizing Monte Carlo estimation to recover standard limiting distributions.
• Performance or risk bound in validation of complex, layered systems subject to discrepancies between simulation and deployment regimes.

Canonical examples include the simulation-adjusted composite likelihood ratio (Cattelan et al., 2014), effective moduli in mesoscale finite element simulations of composites (Collins et al., 2021), and modular risk bounds for composite cyber-physical systems using discrepancy propagation (Reeb et al., 2022).

2. Composite Metrics in Statistical Inference

In composite likelihood inference, the true likelihood is replaced by a sum (or product) of lower-dimensional marginal or conditional likelihoods, such as the pairwise (bivariate-marginal) likelihood. For a sample $y = (y_1, ..., y_n)$, parameter $\theta$, and composite log-likelihood $l_C(\theta; y)$, the unadjusted ratio test statistic,

$$W_C = 2\{ l_C(\widehat\theta_C; y) - l_C(\theta_0; y) \},$$

is asymptotically a weighted sum of chi-squared variables rather than $\chi^2_d$.

To obtain accurate inference, this composite metric must be adjusted. The required sensitivity and variability matrices,

$$H(\theta) = -E_\theta[\partial^2_\theta l_C(\theta;Y)],\quad J(\theta) = \mathrm{Var}_\theta[\partial_\theta l_C(\theta;Y)],$$

cannot always be obtained analytically. When Monte Carlo simulation from the model is feasible, $H, J$ are estimated by generating $M$ i.i.d. samples $Y^{(m)} \sim f(\cdot; \theta_0)$ and computing averages over the simulated composite scores and Hessians (Cattelan et al., 2014). This simulation-based construction enables stringent control over type I error, as numerically verified in spatial Gaussian random field and multivariate probit models.

Adjustment formulas include:

• Moment-matching: $W_C^{(1)} = W_C / w$ with $w = \mathrm{tr}(H^{-1}J)/d$.
• Satterthwaite: $W_C^{(2)} = W_C / k$; $k, \nu$ defined from eigenvalues of $H^{-1}J$.
• Invariant/PSS adjustment: $W_C^{\rm PSS}$ constructed to recover $\chi^2_d$ asymptotics.

Monte Carlo adjustment is strongly preferred when direct analytic calculation or massive independent replicated sampling is impractical (Cattelan et al., 2014).

3. Simulation-Based Composite Metrics in Materials Science

Simulation-based metrics are foundational in computational materials design, especially for multi-scale or heterogeneous composites. Effective material properties—such as homogenized moduli, thermal conductivity, specific heat, permeability, and tortuosity—are computed from finite-element simulation of parameterized unit cells, with mesoscale geometry and constituent properties sampled from design-variable ranges (Collins et al., 2021).

Numerical metrics are aggregated to form a composite figure-of-merit,

$$M = \alpha_1 \frac{k_{\rm eff}}{k_{\rm ref}} + \alpha_2 \frac{E_{\rm eff}}{E_{\rm ref}} - \alpha_3 \frac{\tau}{\tau_{\rm ref}} + \dots,$$

where $\alpha_i$ weight the importance of each property (e.g., thermal conductivity $k_{\rm eff}$, effective Young's modulus $E_{\rm eff}$, tortuosity $\tau$) and each term is normalized relative to a reference value. Polynomial Chaos Expansion (PCE) surrogates and global sensitivity indices (e.g., Sobol') accelerate exploration of the design space. Optimization is conducted over the surrogate-augmented metric, reflecting application-specific performance trade-offs (Collins et al., 2021).

4. Metrics from Simulation in Discrete and Scheduling Systems

In resource-constrained scheduling, simulation-based composite metrics evaluate the quality of activity-selection heuristics under stochastic tie-breaking and lexicographically ordered multi-criteria rules. The composite priority rule, formalized as a sequential "drawers" system—where each candidate activity is assigned the lowest-indexed drawer whose Boolean criterion is satisfied—yields a priority key $P(a) = (d(a), \tau(a))$. Here $d(a)$ is the index of the first criterion satisfied and $\tau(a)$ is a tie-breaking key, often randomized.

One may write an explicit (although unneeded for implementation) composite metric: $$M(a) = \sum_{i=1}^n W_i \delta_{i,d(a)} + \epsilon \tau(a),$$ with $W_1 \gg W_2 \gg \dots \gg W_n$ and $\epsilon$ infinitesimal (Alvarez-Campana et al., 2024). The effectiveness of the composite schedule metric (portfolio makespan) is measured over multiple simulation runs, with best-of-$R$ selection establishing policy performance.

5. Validation and Risk Quantification via Simulation-Based Composite Metrics

Certification and reliability assessment for complex composite systems increasingly rely on simulation-propagated composite metrics. The "discrepancy propagation" method constructs a validation bound for the probability of system failure by propagating local estimates of distributional discrepancy (e.g., maximum mean discrepancy, MMD) through the system's component graph. For a composite system $S$ with $C$ modules and simulator $M$, the method recursively computes at each inter-component edge a tight upper bound $B^{c\to c''}$ on the discrepancy between simulated and real intermediate distributions, using semidefinite programming relaxations on convex quadratically-constrained subproblems (Reeb et al., 2022).

The propagated final bound $B^y$ on output discrepancy enables a rigorously valid upper bound $F_{\max}$ on event risk $p_{\rm fail}$, computed as the optimal value of a convex program over all distributions $\beta$ consistent with $B^y$. This approach rigorously accounts for covariate shift and model bias at any module, yielding nontrivial, nontrivially conservative risk bounds across a variety of benchmarks.

6. Computational and Practical Aspects

Simulation-based estimation of composite metrics balances trade-offs between computational cost, parallelizability, and empirical accuracy:

• Statistical inference: MC-based estimation of sensitivity/variability matrices is strongly preferred when analytic calculation is infeasible. Parallelization over MC replications is highly effective, with empirical test-size stabilization observed for moderate $M$ (250–500) (Cattelan et al., 2014).
• Material design: Surrogate evaluation (e.g., PCE) makes rapid screening and gradient-free optimization feasible, bypassing high-fidelity simulation at every candidate point (Collins et al., 2021).
• Scheduling: Simulation replications scale linearly with computational resources, enabling practical evaluation of stochastic heuristic performance (Alvarez-Campana et al., 2024).
• Validation: Convex relaxations in risk propagation are empirically nearly tight (>99% in most subproblems), and total computational expenditure scales polylinearly with system size and validation set cardinality (Reeb et al., 2022).

7. Limitations and Extensions

Simulation-based composite metrics inherit limitations from their underlying models and sampling strategies:

• Inference: Validity may break down under model misspecification or insufficient MC replication. Adjustments require ability to simulate from the full model; some high-dimensional or computationally intensive settings may restrict feasibility (Cattelan et al., 2014).
• Materials science: Fidelity of property aggregation depends on representativeness of parameter sampling, surrogate model accuracy, and scale separation (Collins et al., 2021).
• Scheduling heuristics: Composite metric effectiveness is tied to adequacy of Boolean drawer criteria; no universal ordering is optimal over all instance classes (Alvarez-Campana et al., 2024).
• Validation: Worst-case propagation of discrepancy may induce conservative (loose) bounds in the presence of limited or unrepresentative validation data, though empirical studies demonstrate tightness in typical cases (Reeb et al., 2022).

Plausible extensions involve generalization to non-Gaussian or multi-modal settings, refinement of surrogate models, incorporation of prior information in discrepancy propagation, and use of hierarchical or multi-fidelity simulation schemes aligned with specific composite metric objectives.