Decision trees with Classification and Regression Trees (CART) methodology form the building blocks of some of the most important and powerful algorithms in statistical learning (e.g., random forests, boosting). For regression models, at each node of the tree, the data is recursively divided into two child nodes according to a split point that minimizes a sum of squares error along a particular variable. In this talk, we will investigate the statistical properties of regression trees grown according to such a scheme. First, we study their asymptotic bias and show that it decays exponentially with the depth—theoretically confirming a common empirical observation that deeply grown trees exhibit low bias. Second, we show how trees with cost-complexity pruning can overcome the curse of dimensionality in certain settings where conventional nonparametric methods fail. The analysis highlights the critical role that data dependent splits play in determining such favorable performance.
S. S. Wilks Memorial Seminar in Statistics