Consistency of Random Forest Type Algorithms under a Probabilistic Impurity Decrease Condition

Abstract: This paper derives a unifying theorem establishing consistency results for a broad class of tree-based algorithms. It improves current results in two aspects. First of all, it can be applied to algorithms that vary from traditional Random Forests due to additional randomness for choosing splits, extending split options, allowing partitions into more than two cells in a single iteration step, and combinations of those. In particular, we prove consistency for Extremely Randomized Trees, Interaction Forests and Oblique Regression Trees using our general theorem. Secondly, it can be used to demonstrate consistency for a larger function class compared to previous results on Random Forests if one allows for additional random splits. Our results are based on the extension of the recently introduced notion of sufficient impurity decrease to a probabilistic sufficient impurity decrease condition.