BASE Scale: Benchmarking Autonomy in Experiments

• The BASE Scale is a six-level hierarchical taxonomy that quantifies and standardizes autonomy in scientific experiments under strict operational constraints.
• It delineates the transition from manual processes to full autonomy by defining benchmarks like the Inference Barrier and Decision Manifold for risk-sensitive automation.
• It enables facility leaders, beamline scientists, and experiment architects to align procurement, safety, and performance through precise latency and digital-twin specifications.

The Benchmarking Autonomy in Scientific Experiments (BASE) Scale is a six-level hierarchical taxonomy designed to rigorously quantify and standardize the degrees of autonomy in experimental workflows at large-scale scientific user facilities, such as synchrotrons, neutron sources, and free-electron lasers. The BASE Scale addresses the critical need for an operationally relevant, risk-sensitive vocabulary to distinguish between mere automation and true autonomy within environments constrained by scarce beamtime, stringent operational design domains (ODDs), and prohibitive latency for in situ agent retraining. It defines precise technical benchmarks for each level, introduces the concepts of the "Inference Barrier" and "Decision Manifold," and operationalizes the assignment of algorithmic liability in autonomous experimentation (Houx, 11 Jan 2026).

1. Motivation and Context

Large-scale user facilities historically automated mechanical tasks while relying on human decision-making for data interpretation, resulting in high-throughput but information-poor data collection. The operational environment is characterized by:

• Severe beamtime scarcity (typically 48–96 hours per user), precluding on-instrument agent retraining.
• Scientific reasoning dependent on high-level, reconstructed physical quantities (e.g., 3D densities, strain fields) rather than raw detector data.
• Latency bottlenecks in real-time semantic inversion, blocking closed-loop semantic feedback.
• Lack of a shared, facility-appropriate standard to demarcate automation from autonomy for the purposes of risk assessment, liability, procurement, and insurance.

The BASE Scale provides a taxonomy tailored for these constraints, in contrast to prior owner-operator models, such as the SAE J3016 standard, that do not align with zero-shot deployment and liability frameworks required in these facilities (Houx, 11 Jan 2026).

2. The Six BASE Levels: Taxonomy and Transition Points

The BASE Scale defines six discrete tiers, characterizing each by who actuates experimental components, who performs data interpretation, feedback loop structure, and necessary technical infrastructure. The two central transitions are the semantic perception inflection ("Inference Barrier" at Level 3) and the first transfer of algorithmic liability away from humans (Level 4).

BASE Level	Execution	Perception	Feedback/Capability
0: Manual	Human controls all	Human interprets data	Reactive OODA loop; no automation
1: Scripted	System follows scripts	Static, no data interpretation	Deterministic, high-throughput; not adaptive
2: Reflexive	System executes moves	Scalar feedback from sensors	Closed-loop optimization (e.g. beam alignment)
3: Heuristic	System + inference engine	Digital twin, semantic	Autonomous feature targeting; within ODD; passes Inference Barrier
4: Supervisory	System manages campaigns	Strategy agent + human veto	Strategic, validated deployment; liability shifts
5: Full	Fully generative system	Full agent-driven autonomy	Hypothesis generation, execution, model updates

Level 3 (Heuristic Autonomy) marks the crossing of the Inference Barrier, permitting semantic feedback and multi-dimensional state-space reasoning under hard latency constraints. Level 4 (Supervisory Autonomy) introduces batch-scale strategy and the formalization of liability denomination (Houx, 11 Jan 2026).

3. Formalism: Inference Barrier, Latency, and Decision Manifold

Crossing the Inference Barrier requires that the closed-loop latency for semantic feedback remains a controlled fraction of the dynamical process timescale:

$$\tau_{\mathrm{loop}} = \tau_{\mathrm{readout}} + \tau_{\mathrm{transport}} + (\tau_{\mathrm{reduce}} + \tau_{\mathrm{inference}}) + \tau_{\mathrm{actuation}} \leq \frac{t_{\mathrm{dynamics}}}{\kappa}, \quad \kappa \geq 10$$

• $\tau_{\mathrm{readout}}$: detector deadtime
• $\tau_{\mathrm{transport}}$: frame transmission ($D_{\mathrm{frame}}/B_{\mathrm{net}}$)
• $\tau_{\mathrm{reduce}} + \tau_{\mathrm{inference}}$: reconstruction + inference
• $\tau_{\mathrm{actuation}}$: mechanical settling

Exceeding this latency threshold requires reverting to lower autonomy. The decision manifold at Level 3 is expressed as a heuristic policy $\pi$ mapping observations $\mathbf{o}_t$ to actions $\mathbf{a}_t$ to maximize entropy-scaled measurement efficiency $E_{\mathrm{SME}}$ under resource constraints, with the admissible state constrained to the ODD:

$$\pi^* = \underset{\pi}{\arg\max} \mathbb{E}_{\mathbf{D} \sim P_{\mathrm{sim}}} \left[\frac{I(M; \mathbf{D}_\pi)}{\mathbf{w}^\intercal\mathbf{C}(\mathbf{a})}\right]$$

with required safety ($\mathbf{s}_t \in \mathcal{O}_{\mathrm{ODD}}$) and observability ($$\tau_{\mathrm{loop}} \le t_{\mathrm{dynamics}}/\kappa$$) constraints.

Spatial to Temporal Extension: Embedding the process velocity $v_{\mathrm{evol}}$ and spatial sampling $\delta x$ allows decision-making to extend from spatial targeting ("where") to temporal gating ("when"), enabling capture of transient phenomena (e.g., crack propagation at $\mu$m/s) with microsecond-level acquisition synchronization (Houx, 11 Jan 2026).

4. Liability, Validation, and Quantification

The BASE Scale provides a formal logic for assigning liability in autonomous experimentation based on the sim-to-real gap ($\Delta_{\mathrm{gap}}$), defined as the negative log-likelihood of the observed experimental outcome under the digital twin model:

$$\Delta_{\mathrm{gap}}(\mathbf{s}_t, \mathbf{a}_t, \mathbf{s}'_{\mathrm{obs}}) = -\ln P_{\mathrm{sim}}(\mathbf{s}'_{\mathrm{obs}} \mid \mathbf{s}_t, \mathbf{a}_t)$$

Liability assignment:

$$\mathcal{L} = \begin{cases} \text{Facility}, & \Delta_{\mathrm{gap}} > \epsilon_{\mathrm{valid}} \quad \text{(model failure)} \ \text{User}, & \Delta_{\mathrm{gap}} \leq \epsilon_{\mathrm{valid}} \quad \text{(policy failure)} \end{cases}$$

This formalism allows facility directors and funding bodies to specify procurement and risk metrics in terms of BASE Level and compliance with latency/ODD constraints, while enabling precise demarcation of responsibility for experimental outcomes.

5. Operationalization and Use Cases

• Facility leadership can define procurement requirements as specific BASE Levels (e.g., "must support Level 3 semantic autonomy with $\tau_{\mathrm{loop}} < 10$ ms and integrated Digital Twin"), commission validation tools for agent certification (e.g., Virtual Beamline), and prioritize infrastructure for semantic middleware and digital-twin fidelity.
• Beamline scientists and experiment architects select autonomy tiers appropriate for experiment type, define ODDs in proposals, and employ the BASE Matrix (a technical communication tool detailing necessary bandwidth, latency, and digital-twin specifications) for engineering integration.
• Benchmarking existing systems: The BASE taxonomy enables mapping of current workflows (e.g., ANDiE and Edge-to-Exascale at Level 3; adaptive XRD and GP-MAPs at Level 2) and the identification of necessary infrastructure improvements to achieve higher autonomy (Houx, 11 Jan 2026).

6. Broader Implications and Connections

The BASE Scale establishes a standardized and quantifiable metric for evaluating end-to-end scientific autonomy, analogous in function but distinct in constraints from SAE J3016 in automotive autonomy. Unlike frameworks that presume agent retraining, BASE enforces zero-shot deployment and operational safety criteria congruent with user-facility realities, explicitly formalizing the technical and legal inflection points for semantic feedback and liability. A plausible implication is that future automated benchmarking frameworks for large multimodal models—such as APEx, which features end-to-end, LLM-driven orchestrated experiment design—may serve as practical testbeds for BASE Scale assessment, providing a path toward the quantification of system autonomy based on the number of independent, fully closed-loop experimental cycles completed without human intervention (Conti et al., 2024).

In summary, the BASE Scale unifies the operational, technical, and risk domains for benchmarking autonomy in experimental science, introducing sharp definitions for semantic feedback, latency governance, and algorithmic liability, and offering a rigorous, facility-relevant framework for the evaluation, certification, and governance of autonomous experimental workflows (Houx, 11 Jan 2026).