Tree-Structured Modelling of Categorical Predictors in Regression

Abstract: Generalized linear and additive models are very efficient regression tools but the selection of relevant terms becomes difficult if higher order interactions are needed. In contrast, tree-based methods also known as recursive partitioning are explicitly designed to model a specific form of interaction but with their focus on interaction tend to neglect the main effects. The method proposed here focusses on the main effects of categorical predictors by using tree type methods to obtain clusters. In particular when the predictor has many categories one wants to know which of the categories have to be distinguished with respect to their effect on the response. The tree-structured approach allows to detect clusters of categories that share the same effect while letting other variables, in particular metric variables, have a linear or additive effect on the response. An algorithm for the fitting is proposed and various stopping criteria are evaluated. The preferred stopping criterion is based on $p$-values representing a conditional inference procedure. In addition, stability of clusters are investigated and the relevance of variables is investigated by bootstrap methods. Several applications show the usefulness of tree-structured clustering and a small simulation study demonstrates that the fitting procedure works well.

• The paper introduces a semiparametric framework that fuses tree-based clustering for categorical predictors with additive modeling for continuous covariates.
• It employs ordered category splits and p-value guided stopping to efficiently fuse similar groups and ensure model stability.
• Empirical results across diverse datasets demonstrate accurate cluster recovery and competitive predictive performance.

Tree-Structured Modelling of Categorical Predictors in Regression

Motivation and Limitations of Classical Approaches

Conventional regression modeling with categorical predictors faces severe challenges, notably when high-cardinality factors are present. Generalized linear models (GLMs) and generalized additive models (GAMs) require explicit parameterization for each category, causing increased model complexity and instability as the number of categorical levels grows. Regularization or penalization methods partially mitigate this but become computationally prohibitive with many categories. Tree-based models, especially CART, naturally handle interactions but tend to overfit interactions at the expense of main effects, and do not facilitate linear or smooth effects for continuous covariates within the same modeling framework. Identifying clusters of similar categories—fusing categories that exhibit statistically indistinguishable effects—remains nontrivial in high-dimensional settings.

Methodological Framework

The proposed method introduces a partially linear tree-structured regression framework aimed at resolving these issues. The response mean is linked to covariates via a predictor that decomposes into a tree component for categorical variables and an additive (potentially smooth) part for continuous variables. Specifically,

$\eta = tr(z) + \sum_{j=1}^q f_j(x_j)$

where $tr(z)$ encodes tree-based clustering on categorical predictors $z$, and $f_j$ are unspecified smooth or linear functions for continuous covariates.

Tree construction iteratively partitions the set of categories for either nominal or ordinal variables. For nominal variables, an optimal split is found by ordering categories according to empirical outcome means, then treating them as ordered for splitting, yielding computational tractability. For ordinal factors, only contiguous fusions are considered. Splitting is guided by statistical fit, with deviance as the objective.

Distinct from recursive partitioning, all components of the model—including the smooth and linear terms—are estimated jointly at each iteration, preserving inference on main effects while allowing flexible, data-driven clustering of categorical levels.

Algorithm and Model Selection

The fitting algorithm proceeds via forward selection: at each step, all possible unselected splits are evaluated, and the split yielding the largest improvement in deviance (or lowest deviance) is retained. The process naturally accommodates multiple categorical predictors, each with its own candidate splitting set. Termination is governed by stopping criteria critical for model parsimony and statistical validity.

Preferred stopping employs a $p$-value based conditional inference approach, controlling for multiple testing (e.g., Bonferroni correction). Alternatives such as cross-validation (CV), Akaike information criterion (AIC), and Bayesian information criterion (BIC) are also benchmarked, but simulation results indicate that the significance-based approach yields superior cluster recovery and parameter estimation accuracy.

Inference and Stability Considerations

Asymptotic standard errors for the tree component parameters lack a closed-form, so the method utilizes bootstrap resampling for uncertainty quantification. Stability of identified clusters is assessed through bootstrap-derived similarity matrices; for each pair of categories, the proportion of bootstrap replicates in which they are fused provides a cluster stability measure.

In empirical applications, clusters for high-cardinality nominal factors (e.g., urban districts in rental data or country in household data) display varying stability, often with high-fidelity recovery of true clusters as seen in simulation experiments.

Empirical Results and Performance

The methodology is validated via simulation and multiple real-world datasets:

• Munich Rent Standard: The approach clusters urban districts, construction years, and room numbers into groups with homogeneous effects and yields interpretable effects for other predictors. Floor space is modeled with a penalized spline, capturing nonlinear effects. Bootstrap-based intervals establish the significance and stability of clusters. Predictive deviance, evaluated by 5-fold CV, is competitive with GAMs and outperforms conventional trees and model-based partitioning.
• Household Car Ownership (German SOEP): The model identifies meaningful clusters among German federal states, highlighting urban city-states as distinct from other regions in car ownership patterns, correcting for socioeconomic covariates. The structured regression delivers compact, interpretable clusters and robust inference.
• Motivational States Questionnaire: In high-dimensional rating scale data, significant reduction is achieved as only a handful of ordinal predictors are retained, with clustering of levels reducing them further to effective binary distinctions for their association with the probability of being sad.

Simulation studies explore MSE, number of detected clusters, false positive/negative rates across diverse stopping rules, and reveal that the tree-structured approach, with a $p$-value threshold, achieves nearly optimal recovery of true data-generating clusters and maintains low error rates. The results also reflect that additive terms can be reliably estimated even in the presence of categorical clustering.

Comparison with Related Methods

Compared to penalized regression methods that fuse categories by direct penalization on pairwise differences, the tree-based approach is markedly more scalable and effective for categorical predictors with many levels. Unlike boosting strategies, which favor weak learners and build smooth fits, tree-structured clustering exploits strong splits to maximize interpretability and cluster detection.

Model-based partitioning, although flexible, emphasizes interaction effects and fits different models in terminal nodes, thus failing to directly address main effect clustering of categorical predictors. The method presented here is better suited for identifying operational clusters under the presence of additional continuous or smooth predictors.

Theoretical and Practical Implications

The proposed tree-structured framework significantly enhances interpretability and parsimony in regression models with categorical predictors, especially in high-cardinality or complex domains. It enables effect fusion for categorical levels, yielding models that respect the true response structure with minimal complexity. The approach also enhances stability and transparency of inference for main effects—often missed by pure tree or black-box models—while retaining predictive accuracy. From a theoretical standpoint, it fills a crucial gap between parametric and interaction-oriented machine learning models.

Practically, the approach is broadly applicable in domains where interpretability is crucial and categorical variables are prevalent—such as urban studies, socio-economic modeling, and psychometrics. The methodology suggests new standards for variable encoding and effect assessment, potentially impacting the way variable importance and regularization are conducted in large-scale applied regression.

Conclusion

Tree-structured modeling of categorical predictors addresses the core limitations of classic parametric and tree-based regression for high-cardinality and complex categorical covariates. The method achieves data-driven clustering of categories within a unified semiparametric model, ensuring parsimonious, interpretable, and statistically robust inference. Simulations and empirical results confirm strong recovery properties, stable clustering, and competitive predictive accuracy, establishing this method as a practical and theoretically sound enhancement for regression modeling with categorical predictors (1504.04700).