RocketStack: Level-aware deep recursive ensemble learning framework with adaptive feature fusion and model pruning dynamics

Abstract: Ensemble learning remains a cornerstone of machine learning, with stacking used to integrate predictions from multiple base learners through a meta-model. However, deep stacking remains rare, as most designs prioritize horizontal diversity over recursive depth due to model complexity, feature redundancy, and computational burden. To address these challenges, RocketStack, a level-aware recursive ensemble framework, is introduced and explored up to ten stacking levels, extending beyond prior architectures. The framework incrementally prunes weaker learners at each level, enabling deeper stacking without excessive complexity. To mitigate early performance saturation, mild Gaussian noise is added to out-of-fold (OOF) scores before pruning, and compared against strict OOF pruning. Further both per-level and periodic feature compressions are explored using attention-based selection, Simple, Fast, Efficient (SFE) filter, and autoencoders. Across 33 datasets (23 binary, 10 multi-class), linear-trend tests confirmed rising accuracy with depth in most variants, and the top performing meta-model at each level increasingly outperformed the strongest standalone ensemble. In the binary subset, periodic SFE with mild OOF-score randomization reached 97.08% at level 10, 5.14% above the strict-pruning configuration and cut runtime by 10.5% relative to no compression. In the multi-class subset, periodic attention selection reached 98.60% at level 10, exceeding the strongest baseline by 6.11%, while reducing runtime by 56.1% and feature dimensionality by 74% compared to no compression. These findings highlight mild randomization as an effective regularizer and periodic compression as a stabilizer. Echoing the design of multistage rockets in aerospace (prune, compress, propel) RocketStack achieves deep recursive ensembling with tractable complexity.

• The paper introduces RocketStack, a recursive ensemble framework that leverages deep stacking with adaptive feature fusion and dynamic model pruning to address complexity.
• The methodology employs periodic feature compression using SFE filtering and attention mechanisms, achieving notable accuracy improvements up to 97-98% while reducing runtime.
• The study demonstrates a scalable and modular approach that enhances predictive performance and mitigates overfitting through controlled stochastic pruning and diverse model integration.

"RocketStack: Level-aware deep recursive ensemble learning framework with adaptive feature fusion and model pruning dynamics"

Introduction to RocketStack

The paper presents RocketStack, a novel recursive ensemble learning framework designed to address the complexity and redundancy challenges inherent in deep stacking architectures. Although ensemble learning, particularly through stacking, has demonstrated substantial benefits in mitigating overfitting and improving predictive accuracy, traditional approaches prioritize horizontal diversity at the expense of recursive depth. RocketStack proposes a structured method to enable deeper stacking, reaching up to ten levels, while incorporating adaptive feature fusion and model pruning dynamics to curb excess complexity.

Methodology

RocketStack's architecture is founded on the principle of recursive stacking, where predictions from base learners are fused with original features at each level. The framework introduces a systematic feature compression and model pruning mechanism across levels:

• Feature Fusion: Combines the predictions of base learners with original input features. At each level, the new meta-feature matrix comprises both the fused features and those carried over from previous levels.
• Model Pruning: Implements dynamic pruning based on out-of-fold (OOF) score thresholds, enhanced by stochastic perturbation via Gaussian noise. This prevents premature convergence and fosters accuracy and diversity among retained models.
• Periodic Compression: Uses feature selection strategies such as Simple, Fast, Efficient (SFE) filtering and attention-based mechanisms, selectively applied at key levels (3, 6, and 9). This periodic schedule balances feature accumulation and dimensionality reduction to enhance both accuracy and computational efficiency.

  Figure 1: Accuracy (\%) across ensembling depths for each individual binary classification dataset for each RocketStack variant.

Evaluation and Results

The framework was evaluated across 33 datasets, encompassing both binary and multi-class classification tasks. Results demonstrated consistent accuracy improvements as stacking depth increased. Notably, periodic feature compression exhibited superior performance relative to each-level approaches by preventing excessive feature growth without compromising predictive quality.

• Binary Classification: In periodic SFE configurations, mild randomization surpassed deterministic selections in both performance and runtime. Maximum accuracy reached 97.08\% at level 10, with substantial runtime reductions compared to non-compressed configurations.
• Multi-Class Classification: Attention-based periodic compression displayed significant gains, achieving 98.60\% accuracy at level 10. It demonstrated the greatest efficiency in reducing feature dimensionality (by 74\%) and runtime (by 56.1\%), indicating effective complexity stabilization.

  Figure 2: Accuracy (\%) across ensembling depths for each individual multi-class classification dataset for each RocketStack variant.

Discussion

RocketStack embodies a shift from shallow ensemble architectures to deep recursive designs, establishing a modular and scalable foundation for decision fusion systems in tabular domains.

• Scalability and Modularity: Its depth-aware framework supports structured optimization across recursive levels, offering a robust solution for evolving feature spaces.
• Regularization Through Stochastic Pruning: By introducing controlled noise into model selection, RocketStack enhances diversity and mitigates the risk of overfitting, akin to common regularization techniques in machine learning.

Conclusion

RocketStack leverages modular recursive stacking supported by adaptive feature fusion and pruning dynamics to achieve deeper ensemble architectures while maintaining computationally efficient growth. Its unique design, inspired by aerodynamics principles in aerospace, fosters scalable learning systems that reconcile predictive accuracy with resource feasibility in complex ensemble domains.

In closing, RocketStack offers a promising path for expanding the depth and effectiveness of ensemble learning frameworks while addressing traditional challenges of feature redundancy, runtime cost, and model overfitting within deep stacking paradigms. Future exploration could integrate RocketStack with more diverse and complex datasets to further enhance its applicability and optimize its architectural prowess in broader contexts.