TabPack: Efficient Hyperparameter Ensembles for Tabular Deep Learning

Abstract: In deep learning for tabular data, efficient ensembles of multilayer perceptrons (MLPs) have recently emerged as effective and practical architectures. Existing methods of this kind use the same hyperparameters for all underlying MLPs, which requires hyperparameter tuning for achieving the best performance. In this work, we introduce TabPack, an efficient MLP ensemble with strong out-of-the-box performance and reduced reliance on traditional tuning. In a single run, TabPack samples and trains many MLPs with different hyperparameters efficiently in parallel and selects ensemble members on the fly during training. Thus, TabPack only requires specifying ranges from which to sample MLP hyperparameter rather than exact hyperparameter values, which naturally demands less precision for good performance. In experiments on medium-to-large public datasets, TabPack with default settings performs on par with extensively tuned prior methods, thus substantially reducing effort and compute resources needed to achieve competitive results on tabular tasks. Notably, running the default TabPack configuration on a modern MacBook took less time than tuning some baselines on an industry-grade GPU.

• The paper introduces a tuning-free paradigm that leverages hyperparameter diversity through random sampling in packed ensembles of MLPs for tabular data.
• Its novel architecture packs diverse MLPs with individual optimizers in a single module, achieving significant speedups and resource efficiency on consumer hardware.
• Empirical results on large datasets illustrate that TabPack outperforms tuned baselines while enabling rapid deployment and practical scalability.

TabPack: Efficient Hyperparameter Ensembles for Tabular Deep Learning

Overview and Motivation

TabPack introduces an efficient paradigm for constructing and training ensembles of multilayer perceptrons (MLPs) with diverse hyperparameters for tabular machine learning tasks. In contrast to conventional tabular deep learning methods, which either employ costly hyperparameter tuning or use homogeneous ensembles requiring tuned parameters, TabPack bypasses the necessity for precise hyperparameter specification by leveraging random sampling within user-defined ranges. This ensemble-first methodology integrates diverse model and optimizer configurations in a single training pipeline, enabling rapid, tuning-free deployment with competitive performance and efficiency.

Technical Design

TabPack operationalizes the packed ensemble framework by stacking multiple base MLPs—each with randomly sampled architecture and optimizer hyperparameters—into a unified module. The entire ensemble is trained in parallel, leveraging the broadcast and vectorization capabilities of standard deep learning libraries for accelerated throughput.

Figure 1: TabPack architecture packing diverse base models and optimizers with randomly sampled hyperparameters, enabling efficient parallel training and online ensemble selection.

To address the challenge of heterogeneity among base MLPs (e.g., varying depths and widths), TabPack pads input representations and parameters to the maximum dimensions required and applies masking during the forward pass to preserve correctness.

Figure 2: Packing diverse MLP architectures into a single weight tensor, maintaining individual model behaviors through padding and masking.

Optimizer diversity is similarly achieved by maintaining individual optimizers (with different learning rates, weight decays, etc.) for each base model in the pack. The ensemble is continuously updated during training (online ensembling), using greedy selection to maximize collective validation performance.

TabPack supports both offline and online ensemble selection strategies. The online approach rebuilds the ensemble once per epoch, sampling intermediate base-model states, which empirically results in superior performance and efficiency compared to offline selection.

Computational Efficiency

The core advantage of TabPack lies in computational throughput. Empirical analysis demonstrates dramatic inference and training speedups compared to single-process or multi-process training of separate MLP instances.

Figure 3: Inference throughput scaling for packed vs. single/multi-process MLP training regimes, showing substantial efficiency gains for packed ensembles.

Packing enables practical scaling to tens or hundreds of base models on a single device—even consumer-grade hardware, such as MacBook M4 Pro—without loss in ensemble predictive quality.

Memory consumption during training is linear in the number of base models, but empirical results show that TabPack fits comfortably within typical GPU RAM constraints for up to 128 base models on large tabular datasets.

Empirical Evaluation

TabPack was benchmarked on a diverse suite of medium-to-large public datasets (varied domains, up to 13.6M instances, up to 900+ features), against strong baselines including XGBoost, TabM, ModernNCA, RealMLP, TabICLv2, and TabPFN-3. Key findings:

• Out-of-the-box performance: TabPack (even with default configurations) matches or exceeds extensively tuned baselines in rank and absolute metrics, especially on large regression tasks.
• Efficiency: Default TabPack runs on a MacBook completed faster than tuning classical methods on an NVIDIA A100 GPU, with task performance on par or superior.
• Time-budget robustness: TabPack rapidly realizes most of its performance potential under even tight time constraints, outperforming competitors across budgets.
• Ensemble member diversity: Final ensembles retain nontrivial hyperparameter diversity, covering substantial fractions of the sampling space for depth, dropout, learning rate, etc.

  Figure 4: Aggregate performance and runtime summary for TabPack relative to state-of-the-art baselines, highlighting efficiency and competitive accuracy with minimal tuning.

  

  Figure 5: Task performance across datasets, showing TabPack's stability and superiority in mean-rank compared to DL and GBDT models.

  

  Figure 6: Performance as a function of runtime budget, demonstrating TabPack's rapid convergence and practical deployment readiness.

Additional evaluation on million-plus instance datasets further substantiates TabPack's scalability and competitiveness in industrial contexts.

Architectural and Hyperparameter Analysis

Empirical ablation studies reveal that the primary gain from hyperparameter diversity is facilitating tuning-free operation rather than improving ensemble accuracy through increased heterogeneity. Ensembles constructed from models with identical tuned hyperparameters marginally outperform those with randomly sampled parameters but incur substantial computational overhead.

Figure 7: Impact of varying architectural hyperparameters (depth, number of base models, maximum patience) on ensemble performance; vertical lines denote defaults.

TabPack periodically samples feature embedding hyperparameters, using efficient periodic embeddings. The strategy efficiently balances model expressivity and computational cost.

Practical and Theoretical Implications

TabPack represents a shift from post hoc ensemble construction toward ensemble-first architectures, tightly integrating ensemble member diversity and parallelism at both algorithmic and systems levels. This enables:

• Automated and tuning-free tabular ML pipelines: Practitioners can deploy strong models without time-consuming hyperparameter optimization.
• Resource-efficient research workflows: Fast experiment cycles on consumer hardware unlock broad access to state-of-the-art tabular deep learning.
• Scalability without loss in predictive quality: TabPack scales linearly in memory and computation, retaining practical throughput and inference efficiency for real-world applications.

The theoretical implication is the decoupling of hyperparameter precision from ensemble performance, relying instead on diversity + online selection for robust generalization. This paradigm suggests broader applicability to other domains (e.g., vision, sequence modeling) where ensemble diversity can be realized via architectural or optimizer heterogeneity.

Figure 8: Inference throughput achieved on A100 GPU for TabPack ensembles, establishing practical feasibility for large-scale deployments.

Limitations and Future Directions

TabPack relies on validation sets for online ensemble selection and early stopping, which may limit generalization in cases of small validation sizes or distribution shift. Extension to alternative ensemble selection algorithms and base architectures is warranted. GPU memory utilization grows with ensemble size; offline/partitioned training or hierarchical ensembling may alleviate this for ultra-large settings.

Potential future directions include:

• Extending hyperparameter diversity to loss functions and data preprocessing.
• Combining architectural elements (e.g., normalization, residual connections) with dynamic sampling.
• Enhancing online ensemble algorithms to improve selection robustness and reduce validation reliance.
• Adapting TabPack's packed ensemble strategy to foundation models and other modalities.

Conclusion

TabPack delivers an efficient, tuning-free methodology for constructing robust deep ensembles in tabular ML, achieving comparable or superior accuracy to tuned baselines with significant reductions in compute and time. The ensemble-first approach sets a blueprint for future research in scalable deep learning architectures, emphasizing diversity, parallelism, and workflow automation.