Adversarial Compilation Configurations

• Adversarial compilation configurations are systematic selections of compiler options that intentionally amplify or reveal security-sensitive behaviors in binaries.
• They underpin methodologies such as robust hyperproperty preservation, robustly safe compilation, and contract-aware secure compilation to balance security and performance.
• These configurations are applied adversarially for evasion and defensively for fault localization and side-channel mitigation in safety-critical systems.

Adversarial compilation configurations are systematic choices of compiler settings and transformations—sometimes guided or selected by attackers or context-sensitive security requirements—that intentionally force, amplify, or expose security-relevant behaviors in output binaries, machine code, or deployed artifacts. These configurations occupy a central position in secure compilation theory, vulnerability analysis, anti-diffing and malware evasion, side-channel robustness, and fault localization. They refer both to configurations used by attackers (to break invariants, evade detection, or trigger weird machines) and to those employed defensively (to enforce contracts, isolate components, or probe for faults).

1. Formal Frameworks and Secure Compilation Criteria

Secure compilation research formalizes the notion of adversarial contexts and configuration-induced attacks via hyperproperty preservation, robustly-safe compilation, and contract-aware secure compilation.

• Robust Hyperproperty Preservation: Models a compiler's obligation to preserve classes of security properties (trace properties, hyperproperties, hypersafety, relational hyperproperties) against all possible adversarial contexts or configurations. Robust preservation is expressed as

$$\forall P. \forall H \in \mathcal{H}.~ (P \models_s^{rob} H) \implies (\llbracket P \rrbracket \models_t^{rob} H)$$

where source partial programs $P$ are compiled to $\llbracket P \rrbracket$, and $H$ denotes the property class. Distinct characterizations cover trace properties, $K$-hypersafety, and full relational property preservation (Garg et al., 2017, Abate et al., 2018).

• Robustly Safe Compilation (RSC): Focuses on preserving safety properties under any adversarial target context. Compilers must defend the component's integrity (e.g., memory regions, invariants) by suitable configuration, such as capability-based isolation or enclave partitioning. The formal criterion is

$$\forall C,~ M.~ (M \vdash C : rs) \implies (M \vdash \lceil C \rceil : rs)$$

where $M$ is a finite-state monitor over sensitive regions, and $\lceil \cdot \rceil$ denotes compilation. The property-free variant shows no new target traces violating $M$ exist (Patrignani et al., 2018).

• Contract-Aware Secure Compilation (CASCO): Parameterizes both the compiler pipeline and hardware characterization by a formal "security contract" $c = (CS, LC, \rightarrow_c)$, encoding microarchitectural observability. A contract-aware compiler $\llbracket c \rrbracket$ must guarantee that only contract-permitted leaks occur; security criterion:

$$\forall c \in C,~ \forall p,p'.~ \text{if } ATR(p) = ATR(p')~\text{then } CTR_c(\llbracket c \rrbracket(p)) = CTR_c(\llbracket c \rrbracket(p'))$$

(Guarnieri et al., 2020).

2. Compiler Configuration Spaces: Security and Attack Surfaces

Adversarial compilation configurations arise by selecting or tuning options that affect control-flow, memory layout, optimizations, architectural mappings, or code generation so as to maximize, minimize, or control behavioral differences:

• Optimization Flags (GCC/LLVM): Switching from default $-O3$ to customized sequences (e.g., $-funroll-loops$, $-ftree-vectorize$) can systematically mutate binaries, disrupt static and semantic code diffing, and lower detection by AV or clone-diffing engines (Ren et al., 2021). Genetic search and Normalized Compression Distance (NCD) metrics enable automatic maximization of binary code differences.
• DL Compiler Knobs: Neural model compilers (XLA, TVM, ONNX Runtime) expose options for operator fusion, loop reordering, floating-point precision, hardware backend, and reduction scheduling. These can be adversarially selected to introduce model backdoors at compile-time or to counter side-channel risks (via tensor schedule tuning) (Chen et al., 14 Sep 2025, Trawicki et al., 2023).
• Hardware/Software Security Contracts: Selection of contracts, e.g., $c^{spec}$ vs $c^{seq-spec}$ in CASCO, determines the insertion or omission of fences, masks, or constant-time primitives (Guarnieri et al., 2020).
• Compartmentalization Parameters: In secure compilation pipelines for unsafe languages, adversarial configurations may specify compartment maps, SFI masks, or tag-rule tables to isolate components or probe for faults (Fachini et al., 2017).

3. Dynamic Compromise and Mutual Distrust

Modern secure compilation models extend adversarial configuration analysis to account for dynamic compromise and mutually distrustful components:

• Dynamic Compromise Model: Each component becomes "compromised" precisely at the instant it executes undefined behavior (e.g., out-of-bounds write). The RSCC criterion requires that any target-level trace be replayable at the source level by sequentially "replacing" each compromised component with an arbitrary safe attacker stub at the point of compromise (Abate et al., 2018, Fachini et al., 2017). This is formalized via the dynamic-compromise relation $t' \prec_{C_k} m$.
• Mutual Distrust: Full-abstraction frameworks now require configuration and runtime enforcement so that any set of compromised components cannot penetrate the invariants of uncompromised peers, irrespective of how the set $K\subseteq C$ of compromised components is chosen (Juglaret et al., 2015). Enforcement via tag-based micro-policies and compartmentalized linking/loader pipelines prevents cross-component leakage, arbitrary jumps, and tag-forging attacks.

4. Adversarial Configuration in Binary Difference and Malware Evasion

Adversarial configurations play a key role in challenging the resilience of diffing, clone-detection, and malware analysis pipelines:

• BinTuner (Editor’s term): Systematic auto-tuning of compiler optimization flags using metaheuristics (GA + NCD) produces binaries with up to 18–60% higher difference scores than default settings, defeating the most prominent binary diffing and clone-detection tools (Ren et al., 2021). Tuned malware variants exhibit up to 50% reduction in AV detection rates (Coreutils, OpenSSL, Mirai, LightAidra, BASHLIFE).
• Evaluation Recommendations: The traditional evaluation assumption ("worst case is $O3$ vs $O0$") is empirically invalid. Robust adversarial compilation should include random and search-based non-default flags in resilience studies; ML-based diffing must incorporate tuned binaries in training data (Ren et al., 2021).

5. Defensive Compilation: Fault Localization, Side-Channel Mitigation, and Compliance

Adversarial configuration techniques are also utilized to enhance defense robustness and for compiler fault diagnosis:

• MultiConf: Construction of multiple adversarial pairs (FailConf, PassConf) differing only in minimal fine-grained compiler options enables more accurate spectrum-based file-level fault localization in complex compilers (e.g., GCC). Aggregation of independent rankings by weighted voting yields up to 45% Top-1 localization success, outperforming prior techniques (Li et al., 27 Dec 2025).
• Tensor Schedule Tuning in DL Compilers: Defensive configuration of scheduling (tiling, fusion, memory layout) reduces side-channel leakage (e.g., GPU L2 cache patterns, attack fidelity $f$ down by ~40% at $<8\%$ performance cost). Multi-objective cost modeling and auto-tuning via AutoTVM are recommended best practices (Trawicki et al., 2023).
• Contract Selection and Secure Compilation Pipelines: Security- and performance-sensitive applications can select contracts (e.g., $c^{seq}$, $c^{spec}$, $c^{seq-spec}$) to balance fence insertion and hardware mitigations, shifting security enforcement between software and microarchitecture (Guarnieri et al., 2020).

6. Weird Machines: Exploit Generation and Secure Mitigation

Adversarial compilation can be conceptualized as a generator of "weird machines"—computational behaviors emergent from semantic gaps between source abstraction and target implementation:

• Characterization: The set of target behaviors accessible to attacker contexts but impossible in any source-level context corresponds precisely to failures in property-preserving compilation. Formally, $$W^{Target}(V) = Behav_T(C\langle V \rangle) \setminus \bigcup_{C_S} Behav_S(C_S[V])$$ (Paykin et al., 2019).
• Exploit Classes: Buffer overflows (ROP), use-after-free, type confusion (synthetic method access), and side-channel timing attacks are all manifestations of adversarially configured compilation chains.
• Mitigations: Bounds-check insertion, pointer tagging (CHERI), control-flow integrity, and constant-time compilation are defensive configurations that restore security by ensuring robust preservation of trace, hyper, and relational properties.

7. Configuration-Driven Trade-Offs: Security, Performance, and Proof Simplicity

Selection of adversarial or defensive compilation configurations exposes trade-offs in overhead, abstraction preservation, and proof complexity:

Configuration Type	Security Strength	Typical Overhead	Proof Complexity
RSC (safety)	Coarse-grained safety (integrity, memory safety)	Minimal (capabilities, enclaves)	Simulation or invariant-based
Full abstraction	Observational equivalence, defense-in-depth	High (universal checks, code padding)	Complex back-translation
CASCO (contract-aware)	Configurable (2-hypersafety, Spectre mitigations)	Tunable: fence insertion, hardware support	Compositional via contract proofs
BinTuner/MultiConf	Adversarial binary/metamorphism resistance	Search/iterative compile cost	Empirical optimization

In sum, the study and deployment of adversarial compilation configurations span attacker-centric evasion and exploitation channels, rigorous formal security models, robust mitigation strategies, and defensive diagnostic techniques. The richness of the configuration space, coupled with evolving threat models and hardware semantics, demands that secure compilation approaches be both theory-grounded and empirically validated, with explicit articulation of the configuration’s provenance, intent, and impact.