CASH: Algorithm Selection & Hyperparameter Tuning

• The Combined Algorithm Selection and Hyperparameter (CASH) problem is a key AutoML challenge that integrates algorithm choice and hyperparameter tuning to minimize empirical loss.
• Hierarchical search spaces, Bayesian optimization, and surrogate modeling are employed to efficiently navigate expansive and conditional hyperparameter domains.
• Advanced strategies like bandit formulations, ensemble methods, and distributed systems yield improved accuracy and reduced computational cost.

The Combined Algorithm Selection and Hyperparameter (CASH) Problem is a core challenge in Automated Machine Learning (AutoML), requiring efficient search over both candidate algorithms and their associated hyperparameter spaces to minimize task-specific loss. This problem arises due to the vast diversity of algorithm families, each with distinct, often high-dimensional and conditional hyperparameter domains, and the expensive computational cost of empirical performance evaluation.

1. Formal Definitions and Complexity

CASH is classically formulated: for a set of algorithms $\mathcal{A}$ and respective hyperparameter domains $\Lambda_a$, given dataset $\mathcal{D}$, find

$$(a^*,\lambda^*) = \arg\min_{a \in \mathcal{A}, \lambda\in\Lambda_a} \mathcal{L}(a, \lambda; \mathcal{D})$$

where $\mathcal{L}$ is an empirical loss measure such as cross-validation error. In concrete systems (e.g., Weka), $|\mathcal{A}|$ might exceed 50, with $\Lambda_a$ encompassing categorical, continuous, and conditional dimensions, sometimes yielding hundreds of variables (Wang et al., 2019).

Jointly solving CASH is NP-hard: the space $\bigsqcup_{a} \Lambda_a$ is highly structured, often hierarchical, and function evaluations (e.g., CV) are resource-intensive. Direct enumeration or grid search rapidly becomes infeasible.

2. Hierarchical and Conditional Search Spaces

Modern CASH solvers leverage hierarchical representations:

• Introduce a root “algorithm” variable, whose setting activates the relevant hyperparameter sub-tree for that algorithm;
• Conditional blocks encode meta-methods, base-classifier choices, ensembles, and possibly feature selection (Thornton et al., 2012, Lindauer et al., 2021).

In SMAC3, the ConfigSpace library instantiates this as a tree-structured space, permitting optimized traversal, surrogate modeling, and conditional activation of relevant hyperparameters only (Lindauer et al., 2021).

3. Bayesian Optimization and Surrogate Modeling

The dominant approach to CASH is Sequential Model-Based Optimization (SMBO), typically using Bayesian optimization:

• Random Forests (SMAC, Auto-WEKA, SMAC3), or Gaussian Processes (single or multi-task) fit observed configurations $(a, \lambda) \mapsto \mathcal{L}$;
• Acquisition functions (e.g., Expected Improvement, UCB) are optimized over the conditional/hierarchical space, often with multi-start and local search (Thornton et al., 2012, Lindauer et al., 2021, Ishikawa et al., 13 Feb 2025).

Recent advances embed each algorithm's hyperparameter space $\Lambda^m$ into a shared latent space $Z$ using multilayer perceptron mappings $\varphi^m:\Lambda^m\to Z$; a multi-task GP is then fitted over $Z \times \{A\}$, enabling cross-algorithm information sharing and accelerated convergence. Pre-training with adversarial regularization and meta-feature-based ranking of latent embeddings further boosts sample efficiency (Ishikawa et al., 13 Feb 2025).

4. Meta-Learning and Human Knowledge Mining

Meta-learning approaches compress prior empirical knowledge:

• Paper-mining: Auto-Model constructs a directed graph of extracted “experience” tuples from the literature, encoding (dataset, best-performing algorithm) with edge weights reflecting publication reliability (impact factor, citation count). Breadth-first search and in-degree analysis yield robust per-dataset algorithm recommendations.
• Automatic meta-feature selection: Genetic Algorithms and Deep Q-Networks prune dataset meta-features to those most predictive of best algorithm, accelerating the selector model and reducing overfitting (Wang et al., 2019, Mu et al., 2020).

Meta-models, frequently trained offline (Random Forest, MLP regressor), then provide instant algorithm recommendations per dataset. Upon selection, a specialized HPO—Bayesian Optimization, Genetic Algorithm, or other—is chosen adaptively based on evaluation cost (Wang et al., 2019).

5. Bandit and Decomposed Two-Level Techniques

Casting CASH as a multi-armed or max $K$-armed bandit yields resource allocation strategies:

• Each algorithm is an arm; each “pull” assigns a time-slice or an HPO iteration to that algorithm.
• Reward functions penalize empirical risk or reward top region performance (e.g., extreme-region UCB focuses on maximal observed outcomes instead of means) (Efimova et al., 2016, Hu et al., 2019, Balef et al., 8 May 2025).

MaxUCB, tailored for CASH, adapts its optimism index to light-tailed reward distributions, maximizing $\max_{t\le T}r_{I_t,t}$ with bounded regret $O(K \ln T / \sqrt{T})$ (Balef et al., 8 May 2025).

6. Ensemble and Diversity-Aware Extensions

Traditional CASH algorithms, in focusing on single best $(A,\lambda)$, often under-explore configuration diversity—limiting ensemble learning gains. Diversity-aware frameworks such as DivBO introduce:

• Diversity surrogates for pairwise configuration diversity (trained with LightGBM);
• Weighted acquisition functions combining validation performance and minimal similarity to a temporary candidate pool;
• Dynamic schedules to shift search weight from pure exploitation to diversity injection, yielding superior generalization in ensemble methods (Shen et al., 2023).

7. Distributed, Agent-Based and Pipeline-Generalized CASH

Distributed resource architectures (HAMLET) arrange ML resources as agents in a holonic tree. Hierarchical queries are matched across agents using parametric similarity, enabling fully automatic, scalable algorithm selection and hyperparameter tuning on distributed platforms, with linear space and time complexity and formal verification (Esmaeili et al., 2023).

CASH frameworks are being extended to encompass broader ML pipeline elements—fine-tuning, ensembling, heterogeneous workflow arms—with decision algorithms (posterior-sampling PFNs) designed for budget-aware horizon optimization and cost-sensitive pulls (Balef et al., 19 Aug 2025).

8. Empirical Results and Benchmarks

CASH methodologies are consistently evaluated on large-scale public datasets (Weka’s 21 datasets, OpenML tasks, BBOB optimization functions), showing that hierarchical, meta-learned, and bandit/Bayesian approaches outperform grid/random search, classical two-stage meta-learning, and uniform sampling baselines in almost all regimes.

For instance, Auto-Model achieves $0.82$ average accuracy on 21 datasets in $30$ sec versus $0.78$ for Auto-WEKA; with $5$ min, Auto-Model at $0.83$, Auto-WEKA at $0.80$. Bandit formulations reduce search cost by an order of magnitude with comparable or better test error (Wang et al., 2019, Efimova et al., 2016, Balef et al., 8 May 2025).

9. Limitations and Future Directions

Current limitations include manual curation of meta-knowledge, domain specificity (most frameworks validated only for classification), lack of ensemble or pipeline heterogeneity optimization, and computational cost of deep or adversarial latent-space pre-training. Promising future directions identified in the literature:

• NLP-driven automatic knowledge extraction from papers;
• Meta-learning for feature, algorithm, and HPO-strategy selection;
• Incorporation of multi-objective criteria (accuracy, runtime, fairness);
• Distributed and incremental learning from user interaction streams;
• Theoretical guarantees for adaptive cost-aware and dynamic acquisition schemes (Wang et al., 2019, Ishikawa et al., 13 Feb 2025, Balef et al., 19 Aug 2025).

Solution/Framework	Key Feature(s)	arXiv ID
Auto-WEKA	Hierarchical SMBO with RF surrogate over joint algorithm/HP space	(Thornton et al., 2012)
SMAC3	Modular BO, ConfigSpace, robust racing over conditional hierarchies	(Lindauer et al., 2021)
Auto-Model	Paper-mined knowledge graph; GA meta-feature selection; fast meta-model	(Wang et al., 2019)
Bandit-Based Methods	Decomposition, resource allocation, max-UCB, extreme-region UCB	(Efimova et al., 2016)/(Hu et al., 2019)/(Balef et al., 8 May 2025)
Latent-Space BO	Shared embedding, multi-task GP, adversarial pre-training	(Ishikawa et al., 13 Feb 2025)
DivBO	Diversity-aware surrogate and ensemble-selection in BO	(Shen et al., 2023)
HAMLET Agent System	Distributed agent-based, hierarchical query protocol	(Esmaeili et al., 2023)
Weighted Sampling	Hyperparameter-space-informed sampling distribution	(Sarigiannis et al., 2019)
Pipeline-Generalized CASH	Bandit allocation over heterogeneous pipelines; cost-aware PFNs	(Balef et al., 19 Aug 2025)
LB-MCTS	LLM+BO within MCTS, dynamic exploration-exploitation, selective memory	(Xu et al., 18 Jan 2026)

CASH remains an active and rapidly evolving research topic at the intersection of optimization, meta-learning, automated reasoning, and systems design.