Mdl = fitcensemble (X, Y) grows 100 boosted decision trees on the NxP predictor matrix X and the class labels Y, LogitBoost for two classes and AdaBoostM2 for more, and returns a ClassificationEnsemble. With 'Method' set to 'Bag', or with a boosting method that resamples, it returns a ClassificationBaggedEnsemble.

Y holds a class label per row of X, as a numeric or logical vector, a categorical, string or character array, or a cell array of character vectors. A row missing a predictor or a class is left out.

The boosting methods, y being +1 for the first class and -1 for the second, d the observation weights, which start at W times the total cost of misclassifying each observation’s class, and eta the learning rate:

'AdaBoostM1'

Two classes. Each tree is grown with the weights d; h is +1 where it predicts the first class and -1 elsewhere, e its weighted error, and its weight eta * log ((1 - e) / e) / 2. The weights are then multiplied by exp (-weight * y .* h).

'AdaBoostM2'

More than two classes. A weight is kept for each observation and each class other than its own, and each tree is grown with their sums; its pseudo-loss e over its class probabilities gives its weight as for AdaBoostM1, and the scores are the weighted sums of the class probabilities.

'RUSBoost'

Two or more classes, for classes of unequal size. Each tree is grown without weights on a sample of every class, round (r * m) rows, m being the size of the smallest class and r the class’s element of 'RatioToSmallest', drawn in proportion to d and with replacement only when the class holds fewer rows. Its pseudo-loss e over all the observations, the weight of each spread evenly over the classes other than its own, gives its weight as for AdaBoostM1, the weights are then multiplied by the mean over those classes of exp (-weight * (1 + h_true - h_k)), and the scores are as for AdaBoostM2. A perfect tree is kept as for AdaBoostM2.

'LPBoost'

Two or more classes. Each tree is grown with the weights d, and its margins and edge are as for TotalBoost. A linear program over the trees with the new one, solved by GLPK, gives the least over distributions of their largest edge; when the smallest edge is no more than 'MarginPrecision' above it, the new tree is not kept and the fit stops. Otherwise the program gives the learner weights that maximise the smallest margin, and as its dual the next weights d. The scores are as for TotalBoost. Where the program has several solutions, MATLAB and this package may choose different ones: the learner weights, and the weights d with the trees grown on them, then differ, and so may the number of trees kept. MATLAB R2024a keeps a tree only when that gap is also above 0.01, so a 'MarginPrecision' below 0.01 acts there as 0.01; here it is taken as given.

'TotalBoost'

Two or more classes. Each tree is grown with the weights d; its margin on an observation is the probability it gives the observation’s class less the largest it gives another, and its edge is those margins weighted by d. The fit stops, without that tree, once the least over distributions of the largest edge of the trees exceeds the smallest edge less 'MarginPrecision'. Otherwise d takes one quadratic step towards the least relative entropy to the starting weights, every edge held at most the smallest edge less 'MarginPrecision', and the learner weights are those that maximise the smallest margin, a linear program solved by GLPK. A tree scores each class with twice its probability less one. Where several learner weights maximise that margin equally, MATLAB and this package may choose different ones, and the scores then differ.

'GentleBoost'

Two classes. Each regression tree is fitted to y with the weights d, its prediction h added to the score times eta, and the weights multiplied by exp (-eta * y .* h).

'LogitBoost'

Two classes. With p the probability of the first class, each regression tree is fitted to (y01 - p) ./ (p .* (1 - p)) with the weights d .* p .* (1 - p); its prediction times eta / 2 is added to the score f, and p = 1 ./ (1 + exp (-f)).

The 'Subspace' method fits each learner, a nearest neighbour or discriminant classifier, on NPredToSample predictors drawn at random without replacement, or on every combination of that many with 'NumLearningCycles' set to 'AllPredictorCombinations', and scores each observation with the plain average of the learners’ class probabilities. UsePredForLearner records the predictors of each learner; every combination is taken in the order of nchoosek, which MATLAB R2024a reverses for some subset sizes, changing the order of the learners but not the scores. MATLAB takes observation weights here and passes them to the learners; the learners in this package take none, so weights that are not uniform are refused.

A boosting method resamples when 'Resample' is 'on' or 'FResample' or 'Replace' is given; RUSBoost and Subspace cannot. Each learner is then grown on ceil (FResample * N) rows, drawn with replacement in proportion to d, each draw weighing the same, or without replacement uniformly, each row keeping its weight in d. Its error is taken on those rows with those weights, and only they are reweighted, rescaled to the weight they carried over their draws. AdaBoostM2 then keeps one weight per observation, spread evenly over the classes other than its own, as RUSBoost does, and LogitBoost advances the score of the rows drawn only. The ensemble is a ClassificationBaggedEnsemble, which records the rows each learner drew and estimates the out-of-bag error. These rules reproduce MATLAB R2024a’s fits on its own draws, except LogitBoost with replacement, whose reweighting in MATLAB was not identified; here it follows the others.

A boosting method whose tree classifies the data without error, zero error for AdaBoostM1 or zero pseudo-loss for AdaBoostM2, keeps that tree with the weight an error of eps gives and stops; one whose error is greater than 0.5 is not kept and stops. MATLAB discards a perfect tree, so a fit whose first tree is perfect is empty there. GentleBoost applies the learning rate to the fit itself, as the other boosting methods do; MATLAB scales only the learner weights, growing the same trees whatever the rate.

Name-Value arguments:

'CrossVal' set to 'on', 'KFold', 'Holdout', 'Leaveout' or 'CVPartition', only one of them, fits the ensemble and cross-validates it as crossval does, returning a ClassificationPartitionedEnsemble.

The method 'RobustBoost', binning and hyperparameter optimization are not implemented, and an option asking for one of them is refused. 'CategoricalPredictors' cannot be used with 'Subspace', its nearest neighbour and discriminant learners taking no categorical predictors here; MATLAB passes them to its nearest neighbour learners and drops them, with a warning, from its discriminant ones.

See also: ClassificationEnsemble, ClassificationBaggedEnsemble, CompactClassificationEnsemble, templateTree, TreeBagger

Source Code: fitcensemble