DUMBer Methodology Overview

• DUMBer methodology is a systematic, taxonomically grounded framework that evaluates adversarial robustness by varying dataset sources, model architectures, and class balances.
• It integrates multiple adversarial attack strategies and training regimes to rigorously assess performance in both image classification and deepfake detection tasks.
• Extensive experiments reveal that adaptive and ensemble adversarial training, along with proper architectural matching, are key for enhancing real-world model robustness.

The DUMBer methodology is a systematic, taxonomically grounded framework for evaluating the robustness of adversarially trained machine learning models under realistic transfer-based adversarial attacks. By operationalizing the axes of heterogeneity in attacker and defender setups—specifically dataset sources, model architectures, and class balance—DUMBer establishes a rigorous, reproducible protocol for adversarial stress-testing that mirrors the variability encountered in real-world deployment scenarios. Its extensibility has been demonstrated in both general image classification (Marchiori et al., 23 Jun 2025) and deepfake detection (Serrano et al., 9 Jan 2026).

1. Foundational Principles and Taxonomy

DUMBer builds on the DUMB taxonomy, which examines robustness across three core axes:

• Dataset Sources (D): Models are evaluated using two or more independently collected datasets (e.g., Bing and Google images, FaceForensics++ and Celeb-DF-V2).
• Model Architectures (U or M): Multiple standard network backbones are instantiated (e.g., AlexNet, ResNet18, VGG11 in vision; Xception, UCF, RECCE, SPSL, SRM in deepfake detection).
• Class Balance (B): Systematic variation of class priors in the training set (e.g., $50/50$, $40/60$, $30/70$, $20/80$), while keeping validation and test distributions balanced.

A DUMB "configuration" is specified by a tuple $(\mathcal{D}_i, \mathcal{M}_j, \pi_k)$, encapsulating one dataset, one architecture, and one class balance regime. DUMBer further introduces diversity in training strategies, particularly with adversarial training (AT), expanding the configuration space to $$(\mathcal{D}_i, \mathcal{M}_j, \pi_k, \text{train}_\ell)$$, where $\text{train}_\ell$ denotes the training regime (baseline or adversarial variant).

This taxonomy structures an attacker–defender matrix of $2 \times 3 \times 4 \times 10 = 240$ models per task in the canonical evaluation (Marchiori et al., 23 Jun 2025). For deepfake detection, the methodology is adapted to two datasets, five detectors, and fixed class balance, yielding a $2 \times 5 \times 12$ experimental grid (Serrano et al., 9 Jan 2026).

2. Formal Attack and Transferability Modeling

Central to DUMBer is the explicit modeling of evasion-style adversarial threat scenarios. Let $f_\theta : \mathbb{R}^d \to \{1, \dots, C\}$ be a trained classifier. The adversarial perturbation problem is posed as:

$$\delta^* = \arg\max_{\delta \in \Delta_p} \mathcal{L}(f_\theta(x + \delta), y), \quad \Delta_p = \{\delta: \|\delta\|_p \leq \varepsilon\},$$

where $\mathcal{L}$ is the loss function (typically cross-entropy) and $\varepsilon$ specifies the adversarial budget.

To measure transferability, adversarial examples are crafted on a surrogate (source) model $A$ and evaluated on a distinct (victim) model $B$ using the transfer success rate:

$$T_{A \to B} = \frac{1}{N}\sum_{i=1}^N \mathbf{1}\bigl\{f_B(x_i + \delta^*_i) \neq y_i\bigr\}$$

Eight real-world attacker–victim configurations are identified, covering all combinations of matches/mismatches along the DUMB axes (from pure white-box to full black-box). In deepfake detection, the space is restricted to four operational DUMB cases (C₁: white-box, C₃: cross-model, C₅: cross-dataset, C₇: cross-dataset/cross-model) (Serrano et al., 9 Jan 2026).

3. Experimental Pipeline and Evaluation Design

DUMBer's experimental protocol is characterized by:

• Large-Scale Model Generation: For each task, the full grid of dataset, architecture, balance, and training strategies is instantiated, resulting in large populations (e.g., 240 models per image classification task; 60 detectors in deepfake detection).
• Attack Suite: Attacks encompass both mathematical (e.g., FGSM, PGD, BIM, DeepFool, RFGSM, TIFGSM, Square) and non-mathematical (e.g., Gaussian noise, box blur, grayscale, occlusion, color inversion) perturbations (Marchiori et al., 23 Jun 2025); deepfake applications use FGSM, PGD, and spectrum-specific strategies such as FPBA (Serrano et al., 9 Jan 2026).
• Adversarial Training Integration: Multiple AT regimes are evaluated, including single-attack, dual-attack, triple-attack/ensemble, adversarial surrogate training, curriculum AT (increasing $\varepsilon$), and mixed augmentation (“NonMathMix-AT”) (Marchiori et al., 23 Jun 2025, Serrano et al., 9 Jan 2026).
• Parallelized Orchestration: All (task, source, attack) jobs are independent, allowing for embarrassingly-parallel execution in high-throughput compute environments.

Prototypical experimental workflow (from (Marchiori et al., 23 Jun 2025)):

Stage	Operation
Model Gen	For each combination of dataset, architecture, balance, and training: Train and store a model
Adv Gen	On each “baseline” model: Generate adversarial examples via each attack type and transfer to all target models
Perf Eval	On each model: Compute clean accuracy, adversarial accuracy, ASR, AMR, and summary statistics

The scope includes 120,960 mathematical and 8,640 non-mathematical attack evaluations per task (three tasks), yielding upwards of 130,000 transfer experiments for robust statistical analysis (Marchiori et al., 23 Jun 2025). Deepfake detection evaluation comprises 1,920 experiment points (Serrano et al., 9 Jan 2026).

4. Performance Metrics and Statistical Summaries

Quantitative evaluation in DUMBer employs the following primary metrics:

• Clean Accuracy ($\mathrm{Acc}_{clean}$): Fraction of correctly classified samples on unperturbed input.
• Adversarial/Rugged Accuracy ($\mathrm{Acc}_{adv}$): Fraction correctly classified under adversarial perturbation.
• Attack Success Rate (ASR): Proportion of adversarially perturbed samples misclassified, computed only over inputs initially classified correctly.
• Attack Mitigation Rate (AMR): Relative reduction in ASR due to adversarial training:

$$\mathrm{AMR} = \frac{\mathrm{ASR}_{base} - \mathrm{ASR}_{atk\_trained}}{\mathrm{ASR}_{base}}, \quad \mathrm{AMR} \in [-1, 1]$$

• Area Under the ROC Curve (AUC): Used in deepfake detection for nominal binary classification performance.

Performance aggregation includes mean and variance calculations over DUMB axes, bootstrapped 95% confidence intervals, and severity bucketization of ASR results (stratifying attacks by impact level to avoid bias from low-impact cases) (Marchiori et al., 23 Jun 2025).

5. Representative Findings and Insights

Analysis across extensive experimental regimes reveals the following:

• Superiority of Adaptive and Ensembled AT: Adaptive adversarial training (scheduled $\varepsilon$) yields the highest AMR (up to ~96%) against high-severity attacks; ensembles of diverse attacks further drive down ASR in white-box and gray-box conditions (Marchiori et al., 23 Jun 2025).
• Generalization Caveats: Curriculum and surrogate-based AT generalize better to attacker–defender mismatches in dataset and class balance, but gains shrink dramatically under cross-dataset shifts; several AT strategies can even degrade robustness relative to baseline in these scenarios (Serrano et al., 9 Jan 2026).
• Importance of Architectural Matching: Transferability is maximized when attacker and victim share the same baseline architecture; white-box evaluations may therefore dramatically overestimate real-world model robustness.
• Low-Regret Value of Non-Mathematical Training: Mixed augmentations with non-mathematical perturbations provide consistent, though modest (20–40%), robustness improvements even when gradient-based attacks are out-of-scope.
• Class Balance Effects: Moderate training class imbalance slightly degrades robustness, whereas extreme imbalance can substantially increase transferability; close monitoring of class priors is recommended for deployment.

Practitioner guidance includes recommending non-mathematical noise and curriculum AT for black-box safety, validating defense generality on DUMB populations, and thorough reporting of robust performance stratified by DUMB axes with confidence intervals (Marchiori et al., 23 Jun 2025, Serrano et al., 9 Jan 2026).

6. Implications and Case-Aware Recommendations

The DUMBer methodology demonstrates that adversarial training protocols must be tailored to anticipated threat models and evaluated comprehensively for real-world credibility:

• In-Distribution Robustness: Adversarial training, particularly with a diverse attack set and surrogates, meaningfully protects against attacks where test data and architecture are matched or similar to the training scenario—relevant in controlled white- and gray-box environments.
• Cross-Dataset Vulnerability: When facing out-of-distribution shifts (e.g., new datasets in the wild), even ensembles and surrogate-based AT can fail to confer significant robustness; negative AMR values have been observed, underscoring risk of overfitting defense strategies (Serrano et al., 9 Jan 2026).
• Case-Aware Defense Strategy: Mapping security protocols to specific DUMB cases is essential; evaluating robustness only in white-box or in-distribution settings leads to over-optimistic claims. Real-world deployment should favor defense diversification, cross-domain generalization, and transparency in reporting robust performance.

A plausible implication is that DUMBer’s taxonomic structuring provides a blueprint for future robust machine learning assessment, emphasizing heterogeneity and comprehensive adversarial testing as preconditions for credible model deployment.

7. Methodological Extensions and Applications

DUMBer’s core schema—systematically varying dataset, architecture, and balance, combined with a spectrum of adversarial attacks and training strategies—readily generalizes beyond standard image classification. In the deepfake detection domain, DUMBer’s operationalization highlights both the strengths and intrinsic limits of adversarial training under transferability constraints, reinforcing the necessity of domain-generalization and hybrid defenses in security-critical applications (Serrano et al., 9 Jan 2026).

The approach has catalyzed development of reproducible testbeds, rigorous reporting paradigms, and transferability-aware benchmarking in adversarial machine learning. Its continued adaptation to emerging modalities is likely to be central for realistic appraisal of model robustness in operational contexts.