Open Source AutoML Benchmark

• The paper outlines a modular evaluation pipeline with fixed data splits, isolated execution environments, and uniform resource allocation to ensure reproducibility.
• It employs statistical methods, including the Bradley–Terry model, for rigorous pairwise comparisons and performance quantification across diverse metrics.
• The framework emphasizes extensibility and reproducibility through version-controlled configuration management, continuous integration, and provenance-aware artifact storage.

Open source AutoML benchmarks provide rigorous, reproducible, and extensible frameworks for the comparative evaluation of automated machine learning (AutoML) systems and pipeline components. These benchmarks span supervised tabular learning, text and vision, time series forecasting, multiple-instance learning, and specialized modalities. This article details the design principles, architectures, methodologies, and analytical standards in state-of-the-art open source AutoML benchmarks, highlighting formal workflow definitions, statistical analysis techniques, extensibility and reproducibility frameworks, and meta-benchmarking advances.

1. Formal Benchmark Architecture and Workflow Design

Leading open-source AutoML benchmarks such as AMLB implement a modular evaluation pipeline with uniform resource allocation, fixed datasets, and strictly specified procedures to ensure comparability and reproducibility across frameworks (Gijsbers et al., 2022).

Each benchmark “task” is described by a YAML or JSON configuration declaring:

• The data source (OpenML dataset ID or local file)
• Task type: binary/multiclass classification or regression
• The target variable and optional column-type annotations

The pipeline comprises these canonical stages:

1. Data Loading and Partitioning: Automatic retrieval (OpenML/file), application of consistent, time-stable train/validation/test splits (often 10-fold cross-validation or fixed holdouts).
2. Framework Isolation and Job Spawning: Each framework is installed in an isolated conda or Docker environment. Jobs are launched per (framework, task, fold) tuple, with resource flags (CPU, RAM, GPU) enforced at the runner level.
3. Training and Inference: Framework APIs are called in standardized signature (fit(X_train, y_train), predict(X_test), optionally predict_proba(X_test)).
4. Metrics Collection: Wall-clock training_time, per-fold and per-instance inference_time, peak RAM, and all pipeline artifacts (e.g., feature importances, serialized models) are logged in a structured result directory.
5. Summary and Downstream Analysis: Results are compiled into SQLite or JSON summary tables for formal statistical treatment.

An explicit plugin interface allows new AutoML frameworks to be added by implementing two shell/Python methods (train, predict), with frameworks registered via YAML files supplying invocation templates and resource constraints.

CLI and direct Python APIs enable flexible execution:

from amlb.core.runner import Runner runner = Runner( frameworks=["frameworks/autosklearn.yaml", "frameworks/h2o.yaml"], tasks=["tasks/classification.yaml"], results="./results" ) runner.set_resource_limits(cores=4, memory_gb=16, gpu=False) runner.run() summary = runner.get_results()

2. Evaluation Metrics and Statistical Comparison

Benchmarks provide a comprehensive suite of performance metrics standardized across frameworks and data modalities:

• Classification: Accuracy, log-loss, AUC, F1-score.
• Regression: RMSE, MAE, $R^2$.
• Resource Usage: Training/inference times, peak memory.

For rigorous system-level comparison, the Bradley–Terry model is used for pairwise win/loss analysis and latent strength estimation (Gijsbers et al., 2022). Given all pairwise outcomes, the latent strength parameters $\{\pi_i\}$ for frameworks $i$ are fitted by maximum likelihood:

$$P(i\text{ beats }j) = \frac{\exp(\pi_i)}{\exp(\pi_i) + \exp(\pi_j)}$$

“Bradley–Terry trees” recursively partition the task space by meta-features (number of classes, sample size, feature count, etc.), building a hierarchical decision tree in which internal nodes each fit their own BT model. Statistical tests are applied at each split to ascertain whether framework rankings differ significantly within a subtree.

All metrics, timing, and errors are stored per (framework, task, fold), making it possible to create normalized rank or win/failure heatmaps as well as resource–performance trade-off plots for multi-criteria analysis.

3. Extensibility, Reproducibility, and Configuration Management

A central tenet is ease of extensibility and strict reproducibility (Gijsbers et al., 2022). This is achieved by:

• Declarative addition of new tasks or frameworks (YAML entries + plugin wrappers).
• Pinning all random seeds and dependency versions (enforced by conda/docker environments).
• Centralized configuration management: all critical parameters (task lists, framework scripts, seeds, budgets) reside in version-controlled YAML files.
• Continuous Integration: Nightly/commit-triggered workflows run small “smoke tests” across all combinations, automatically updating a public results database and re-rendering benchmark web dashboards.

Artifacts (models, logs, metrics) are persistently stored with explicit linkage to code and configuration versions, yielding strong provenance and alignment with best practices for scientific computing.

4. Analysis of Framework Trade-offs and Failure Modes

Comprehensive open source benchmarks expose not only performance but also systematic trade-offs and failure states:

• Resource allocation is uniform: Each system is restricted to identical CPU/GPU counts, RAM, and wall-clock limits, eliminating confounding from overprovisioning.
• Failures (job crashes, timeouts, OOM errors) are logged per (framework, task, fold), providing a granular failure map for subsequent analysis.
• The interface enables direct inspection of resource–performance curves (training/inference time vs. accuracy). This illuminates cases where model accuracy improvements are accompanied by unacceptable increases in cost or where certain systems fail to produce any valid result within the allocation.
• Subset analysis using decision-tree meta-models can reveal regimes (e.g., high-dimensional tasks, high-class cardinality) where some frameworks systematically outperform others, informing both method selection and future AutoML research.

5. Project Ecosystem and Community Tools

AMLB and similar benchmarks are Apache 2.0-licensed, with explicit contributions guides, CI integration, and active user communities (Gijsbers et al., 2022).

The project supports:

• CLI utilities for summarizing outcomes, plotting per-task or per-framework comparisons, and exporting results as HTML reports.
• Interactive Web UIs (e.g., https://amlbench.info) enabling drill-down by dataset meta-features, visualization of Bradley–Terry rankings, and inspection of per-task learning/inference curves.
• Community participation mechanisms: bug reports, pull requests for new tasks or frameworks, and live support channels (Slack/Gitter).

Interpretation tools extend to model- and framework-level diagnostic analysis, enabling researchers to directly visualize, audit, and compare performance and resource trends.

6. Significance and Future Directions

Open source AutoML benchmarks formalize best practices in system-level comparison, reduce methodological errors in the literature, and provide a continuously updated leaderboard of public results. The explicit separation of configuration, execution, and evaluation enables both rapid experimentation and rigorous scientific audit (Gijsbers et al., 2022).

The use of hierarchical statistical ranking (e.g., Bradley–Terry trees) and provenance-aware artifact management mark significant advances over ad hoc or purely leaderboard-style benchmarks. As AutoML frameworks diversify, such open-source, extensible benchmarks are essential infrastructure for both academic and applied machine learning research.