Decision Tree-Based Surrogate Model

• Decision tree-based surrogate models are interpretable approximations that mimic complex black-box models using recursive, axis-aligned partitioning.
• They leverage greedy algorithms like CART and extend to model-based variants to provide clear, actionable decision rules for optimization and explainability.
• They balance fidelity and interpretability, enabling transparent evaluations in clinical screening, fairness auditing, and surrogate-based optimization tasks.

A decision tree-based surrogate model is an interpretable predictive or explanatory model that mimics the behavior of a complex, typically black-box machine learning model (e.g., ensemble methods, neural networks), or directly approximates an unknown objective for purposes such as optimization or explainability. These surrogates leverage the axis-aligned, recursive partitioning structure of decision trees to provide human-accessible decision logic, with direct control of interpretability via tree depth and node count, and offer transparent rules or piecewise approximations of the target function (Hasnat et al., 3 Nov 2025, Herbinger et al., 2023, Verdasco et al., 2024, Leenders et al., 16 Dec 2025, Naharro et al., 2024).

1. Formal Definition and Construction

Let $f: \mathcal{X} \rightarrow \mathcal{Y}$ be a black-box model whose input-output mapping is complex or opaque. The surrogate decision tree $h: \mathcal{X} \rightarrow \mathcal{Y}$ is constructed by training on a dataset $\{(x_i, f(x_i))\}_{i=1}^n$ sampled from the relevant feature space. The chosen function class $H_d$ consists of all decision trees of depth at most $d$.

The surrogate tree is selected to minimize an empirical fidelity loss,

$$L_n(f, h) = \frac{1}{n} \sum_{i=1}^n \mathbb{I}[f(x_i) \neq h(x_i)],$$

and thus,

$h^* = \arg\min_{h \in H_d} L_n(f, h).$

for classification; analogous formulations apply for regression (e.g., squared error loss) or custom objectives in model-based trees (Hasnat et al., 3 Nov 2025, Herbinger et al., 2023).

Surrogates may also be fitted directly to pairwise labels (preferential optimization), rules extracted from boosting ensembles, or additive models inside each leaf (model-based tree surrogates) (Leenders et al., 16 Dec 2025, Herbinger et al., 2023, Markovitch et al., 2021).

2. Training Algorithms and Splitting Criteria

The canonical approach is greedy, recursive partitioning as in CART. At each node, candidate splits are evaluated—often by impurity metrics (Gini index for classification, variance reduction for regression). For increased expressiveness or task-adaptivity:

• Pairwise surrogate trees for preference learning split nodes based on a "consistency score" that maximizes the separation of pairwise comparison outcomes, rather than label purity (Leenders et al., 16 Dec 2025).
• Model-based tree surrogates (MOB, SLIM, GUIDE, CTree) fit parametric (often linear additive) models per region; splits are chosen to maximize reduction in local loss or via statistical tests for parameter instability (Herbinger et al., 2023, Zhou et al., 2022).
• Surrogate-based rule extraction (e.g., SRules) may enforce split order based on global feature-importance, followed by pruning for statistical significance and interpretability (Verdasco et al., 2024).
• Special-purpose regularization during black-box training (e.g., L₁-orthogonal penalization) can facilitate higher-fidelity surrogates by aligning network structure with axis-parallel decision boundaries (Schaaf et al., 2019).

Pseudocode for canonical construction (classification setting) (Hasnat et al., 3 Nov 2025):

Input: f ← trained black-box oracle X ← input features d_max ← maximum tree depth y_tilde ← f(X) # pseudo-labels tree = DecisionTreeClassifier(criterion='gini', max_depth=d_max) tree.fit(X, y_tilde) return tree

3. Application Contexts and Model Variants

Surrogate decision trees are deployed across a broad range of contexts:

• Explainability: Translating high-performance models into actionable decision logic (e.g., CVD-risk flowcharts at depth 4 with ∼90% fidelity) (Hasnat et al., 3 Nov 2025), creating concise, human-auditable rule sets (SRules, Concept Trees, AdaBoost stumps) (Verdasco et al., 2024, Chatzimparmpas et al., 2023, Renard et al., 2019).
• Optimization and Active Learning: Serving as the core approximator in surrogate-based optimization (e.g., Bayesian, preferential, and derivative-free optimization), handling both regression and pairwise discovery (Leenders et al., 16 Dec 2025, Naharro et al., 2024, Neufang et al., 2024). Surrogates may be single trees, ensembles (RF, boosted), or decision-tree hybrids (TreePCE) (Said et al., 16 Sep 2025).
• Fairness and XAI Auditing: Illuminating black-box discrimination patterns; noting that sampling design, class imbalance, and split criteria can obscure or reveal sensitive features at arbitrary depths (Wilhelm et al., 2024).
• Policy Surrogates: High-dimensional control policies distilled to local tree-based surrogates, emphasizing local fidelity and clustering for interpretability (Mern et al., 2021).
• Shapley Value Computation and Model-based Distillation: Leveraging tree structure for efficient conditional expectation calculations and unified global/local feature attribution (Zhou et al., 2022, Herbinger et al., 2023).

Advanced forms include model-based trees (locally linear, spline, or additive expansions in leaves) (Herbinger et al., 2023, Said et al., 16 Sep 2025), decision-stump ensembles (Chatzimparmpas et al., 2023), and robust optimization surrogates encoded as MILPs (Goerigk et al., 2024).

4. Evaluation Metrics and Fidelity-Interpretability Trade-offs

Metrics for surrogate model evaluation include:

• Fidelity (Mimicry Accuracy): Agreement rate between surrogate and black-box predictions (e.g., $89.9\%$ at depth 4 in clinical CVD risk screening); loss as $L_n$ (Hasnat et al., 3 Nov 2025). For regression, $R^2$ with respect to the black-box output (Herbinger et al., 2023).
• Coverage: Fraction of data by which the extracted rule set or tree applies (Verdasco et al., 2024).
• Confidence/Precision: Accuracy of surrogate predictions within each rule (Verdasco et al., 2024).
• Complexity: Tree depth, number of leaves, or total rules; direct proxies for interpretability (Hasnat et al., 3 Nov 2025, Herbinger et al., 2023, Chatzimparmpas et al., 2023).
• Stability: Repeatability of tree structure under bootstrapped resampling (quantified by the Rand index or similar) (Herbinger et al., 2023).
• Locality: Ability to yield interpretability in a focused region of the input space (local surrogates) (Mern et al., 2021).
• Runtime and Scalability: Owing to low complexity, tree-based surrogates are tractable for interactive or high-dimensional settings (Leenders et al., 16 Dec 2025, Kim et al., 2022).

Performance always exhibits a fundamental fidelity–interpretability trade-off: increasing tree depth or number of leaves improves fidelity but reduces the ease of human comprehension (Hasnat et al., 3 Nov 2025, Herbinger et al., 2023, Chatzimparmpas et al., 2023, Verdasco et al., 2024).

5. Strengths, Limitations, and Safeguards

Advantages:

• Direct interpretability (flowchart-like decisions, if–then rules).
• Fast inference and low computational overhead for training small trees (Hasnat et al., 3 Nov 2025, Leenders et al., 16 Dec 2025).
• Flexible to regression, classification, preference learning, and optimization contexts (Naharro et al., 2024, Neufang et al., 2024).
• Amenable to quantification of fidelity and coverage, with systematic rule extraction and post-hoc evaluation (Verdasco et al., 2024).
• Uniquely suited for domains where transparent, actionable models are required (clinical, regulatory, public sector applications) (Hasnat et al., 3 Nov 2025, Wilhelm et al., 2024).

Limitations and Cautions:

• Surrogates mimic the black box; any bias, error, or discriminatory structure present in $f$ is replicated in $h$, often without exposure of causal pathways (Hasnat et al., 3 Nov 2025, Wilhelm et al., 2024).
• Tree structure can obscure discriminatory or sensitive features if data distributions are imbalanced or if malicious ordering of sample frequencies is allowed; sensitive attributes can be relegated to deep levels, escaping scrutiny (Wilhelm et al., 2024).
• Limited capacity for representing continuous, smooth, or high-order interaction effects without loss of interpretability (unless extended to model-based leaves) (Herbinger et al., 2023, Said et al., 16 Sep 2025).
• Hyperparameter selection (depth, min_samples, splitting criterion) must balance readability vs. accuracy; no one-size-fits-all setting (Hasnat et al., 3 Nov 2025, Verdasco et al., 2024).
• Ensembles of trees (RF, boosted trees) may offer higher predictive fidelity as surrogates but at the expense of lost transparency unless further distilled (Neufang et al., 2024, Kim et al., 2022).

Recommended Safeguards:

• Enforce balanced sampling, ensemble surrogates with multiple seeds/bootstraps, and explicit fairness tests to ensure critical features do not remain hidden (Wilhelm et al., 2024).
• Combine global surrogates with local explanations (e.g., SHAP, LIME) for completeness.
• Use automated tooling (e.g., DeforestVis, SRules, Concept Tree) for interactive, human-in-the-loop reconciliation of fidelity and interpretability (Chatzimparmpas et al., 2023, Verdasco et al., 2024, Renard et al., 2019).

6. Recent Advances and Practical Implementations

• Clinical screening: CART surrogates of LightGBM ensembles with $\leq 5$ nodes, $89.9\%$ fidelity, and specific feature split ordering enable clinical rationale without discarding nearly 10% fidelity relative to the black box (Hasnat et al., 3 Nov 2025).
• Optimization: Decision tree surrogates enable scalable, interpretable Bayesian Optimization workflows (tree-based preferential BO, robust optimization), outperforming Gaussian Processes on spiky or high-dimensional tasks and providing direct robustness guarantees under uncertainty sets (Leenders et al., 16 Dec 2025, Goerigk et al., 2024).
• Model distillation: Combining local additive models in leaf nodes—in SLIM, GUIDE, MOB, or CTree—yields surrogates with higher fidelity to black-box predictors, transparent region-wise explanations, and direct quantification of interaction/covariate effects (Herbinger et al., 2023, Zhou et al., 2022).
• Fairness/explainability auditing: Empirical and theoretical demonstrations that surrogate trees do not inherently "surface" hidden discrimination; meta-algorithmic checks and randomized surrogates are needed for trustworthy XAI (Wilhelm et al., 2024).
• User-centered rule extraction and visualization: Interactive surrogate tree/ensemble frameworks (DeforestVis, SRules, Concept Trees) offer targeted explanations, complexity–fidelity trade-offs, and integration with domain experts for actionable deployment (Chatzimparmpas et al., 2023, Verdasco et al., 2024, Renard et al., 2019).

7. Comparative Summary of Major Variants

Surrogate Variant	Interpretability	Fidelity Potential	Target Application	Key Innovations
Simple CART surrogate	High	Moderate	XAI, screening	Single depth/mastery, actionable rules (Hasnat et al., 3 Nov 2025)
Model-based tree surrogate	Moderate	High	Model distillation, local explanation	Additive models in leaves (Herbinger et al., 2023, Zhou et al., 2022)
Surrogate rule ensemble	High	Varies	Rule-based explanations	Pruned rule extraction (Verdasco et al., 2024, Markovitch et al., 2021)
Robust surrogate tree	High	Moderate	Optimization under uncertainty	MILP encoding, scenario generation (Goerigk et al., 2024)
Stump-based ensemble (AdaBoost)	Moderate	High	Visual analytics, explanation	Interactive complexity-fidelity analysis (Chatzimparmpas et al., 2023)
Tree-PCE surrogate	Low-moderate	High	Sensitivity analysis, discontinuities	Piecewise polynomial expansions (Said et al., 16 Sep 2025)

Each variant involves principled choices regarding split criterion, leaf model, post-processing, and metric for interpretability, tailored to application constraints and the nature of the black-box or latent function being approximated.

---

References:

• "Interpretable Heart Disease Prediction via a Weighted Ensemble Model: A Large-Scale Study with SHAP and Surrogate Decision Trees" (Hasnat et al., 3 Nov 2025)
• "Explainable Preference Learning: a Decision Tree-based Surrogate Model for Preferential Bayesian Optimization" (Leenders et al., 16 Dec 2025)
• "Comparative study of regression vs pairwise models for surrogate-based heuristic optimisation" (Naharro et al., 2024)
• "Hacking a surrogate model approach to XAI" (Wilhelm et al., 2024)
• "Towards Robust Interpretable Surrogates for Optimization" (Goerigk et al., 2024)
• "DeforestVis: Behavior Analysis of Machine Learning Models with Surrogate Decision Stumps" (Chatzimparmpas et al., 2023)
• "Shapley Computations Using Surrogate Model-Based Trees" (Zhou et al., 2022)
• "A tree-based Polynomial Chaos expansion for surrogate modeling and sensitivity analysis of complex numerical models" (Said et al., 16 Sep 2025)
• "On Uncertainty Estimation by Tree-based Surrogate Models in Sequential Model-based Optimization" (Kim et al., 2022)
• "An Interpretable Rule Creation Method for Black-Box Models based on Surrogate Trees -- SRules" (Verdasco et al., 2024)
• "Improved prediction rule ensembling through model-based data generation" (Markovitch et al., 2021)
• "Enhancing Decision Tree based Interpretation of Deep Neural Networks through L1-Orthogonal Regularization" (Schaaf et al., 2019)
• "Surrogate-Based Optimization Techniques for Process Systems Engineering" (Neufang et al., 2024)
• "Interpretable Local Tree Surrogate Policies" (Mern et al., 2021)
• "Concept Tree: High-Level Representation of Variables for More Interpretable Surrogate Decision Trees" (Renard et al., 2019)
• "Leveraging Model-based Trees as Interpretable Surrogate Models for Model Distillation" (Herbinger et al., 2023)