RegressionTree
statistics: RegressionTree
Binary decision tree for regression
The RegressionTree class implements a CART binary decision tree.
Growth splits each node on the single predictor and cut point that lower
the squared error of the response the most, and stops when a node is too
small to be a parent, has no split leaving enough observations on both
sides, or already accounts for all but a tolerance of the error the root
carried. The grown tree is then optionally reduced, first by merging
the leaves that buy no accuracy and then by cost complexity pruning,
which orders the branch nodes by how little error their subtrees remove
and records that order so a subtree of any size can be recovered
afterwards with prune.
Create a RegressionTree object by using the fitrtree
function or the class constructor.
The fit is carried out by the compiled engine treetrain and
predictions by treepredict, which the classification tree shares.
An observation missing the predictor a node cuts on descends to neither
child. It is counted in that node and in every node above it, and
predict stops it there and gives it that node’s answer, so a row
is never sent down a branch on evidence it does not carry.
A categorical predictor is split into two sets of levels by ordering the levels by their mean response, and an observation whose level a node did not see stops there, as one missing the predictor does. The lower means go left; MATLAB does not always keep that side, and equally good splits may be chosen differently, so a node’s sides and numbering may differ.
What this class does not do yet. Surrogate splits are not
implemented, and an option asking for them is refused rather than
quietly ignored. The six Surrogate properties are therefore
always empty, as they are in MATLAB without surrogate splits.
See also: fitrtree, ClassificationTree, treetrain, treepredict
Source Code: RegressionTree
The RegressionTree class contains the following properties:
A numeric matrix holding the predictor data the model was fitted on. Each column is one predictor and each row one observation. This property is read-only.
A numeric column vector with one element per row of X, holding the observed response of each observation. This property is read-only.
A positive integer, the number of observations the model was fitted on. It counts the rows kept, so it is smaller than the number of rows given whenever a response was missing. This property is read-only.
A logical column vector with one element per row of the predictor data
as it was given, true for each row used for fitting. It is empty,
[], when every row was used, so a non-empty value means that
rows were dropped. Only a missing response drops a row; a row missing
some of its predictors is kept. This property is read-only.
A numeric column vector of the weights the fit used, one per retained observation. They are the weights given, scaled to sum to one. This property is read-only.
A cell array of character vectors with one name per column of X. This property is read-only.
A character vector naming the response. This property is read-only.
A row vector of column indices into X, naming the predictors treated as categorical, empty when none is. This property is read-only.
A cell array of character vectors. It differs from
PredictorNames only when a categorical predictor has been
expanded into one column per level, which this class does not do, so
the two are always equal. This property is read-only.
A cell array with one column vector of bin edges per predictor, empty unless the predictors were binned before fitting. Binning is not implemented, so this is always empty. This property is read-only.
A structure recording the options the tree was grown under:
SplitCriterion, MinParent, MinLeaf,
MaxSplits, NVarToSample, MergeLeaves,
Prune, PruneCriterion, QEToler,
NSurrogate, MaxCat, AlgCat,
PredictorSelection, Method and Type.
SplitCriterion and PruneCriterion are both
'mse', the only criterion a regression tree has, and
QEToler is the tolerance growth stops at.
MinParent is the value the fit used, which is
max (MinParentSize, 2 * MinLeafSize) and so may exceed the
'MinParentSize' asked for. This property is read-only.
Hyperparameter optimization is not implemented, so this is always empty. This property is read-only.
A positive integer, the number of nodes the tree holds, branch nodes and leaves together. Nodes are numbered as they are created, so a parent always carries a lower number than either of its children. This property is read-only.
A NumNodesx2 matrix naming the left and the right child of each node. A leaf carries a zero in both columns. This property is read-only.
A column vector naming the parent of each node. The root carries a zero. This property is read-only.
A logical column vector, true for each node that carries a split and false for each leaf. This property is read-only.
A cell array of character vectors with one entry per node, holding the name of the predictor the node splits on and an empty character vector at a leaf. This property is read-only.
A column vector holding, for each node, the column of X the node splits on, and zero at a leaf. This property is read-only.
A column vector holding, for each node, the value the split compares
the predictor against: an observation goes left when its value is less
than the cut point and right otherwise. A leaf and a categorical cut
carry NaN.
This property is read-only.
A cell array of character vectors holding 'continuous' at a
branch node that cuts a numeric predictor at a point,
'categorical' at one that splits a set of levels, and an empty
character vector at a leaf. This property is read-only.
A NumNodesx2 cell array holding, for a node that cuts a categorical predictor, the levels sent left and the levels sent right. Every other entry is empty. This property is read-only.
A Nx2 cell array with one row per categorical split, in node order, holding the levels sent left and the levels sent right. It is empty when no split is categorical. This property is read-only.
A column vector holding how many training observations reached each node. A row missing the predictor its node cuts on is counted at that node and at none below it, so a parent’s size is not in general the sum of its children’s. This property is read-only.
A column vector holding, for each node, the weighted mean of the
response over the observations that reached it. It is what
predict answers for a row that comes to rest there. This
property is read-only.
A column vector holding, for each node, the weighted mean squared error of the response about the node’s mean. This property is read-only.
A column vector holding, for each node, the total weight of the
observations that reached it, the weights being those in W.
The root carries one. This property is read-only.
A column vector holding, for each node, its mean squared error weighted by the probability of reaching it, which is the squared error the node contributes to the whole tree. This property is read-only.
A column vector holding, for each branch node, the level of the cost
complexity sequence at which it stops being a branch node, and zero at
a leaf. Pruning the tree to level L turns every node whose
level is between one and L into a leaf. It is empty when
neither 'Prune' nor 'MergeLeaves' was asked for, since
no sequence was then estimated. This property is read-only.
A column vector with one element per level of the pruning sequence,
the first of which is zero and stands for the unpruned tree. Level
L is the smallest subtree that is optimal for every complexity
parameter from PruneAlpha(L+1) up to the next one. This
property is read-only.
Surrogate splits are not implemented, so this is always empty. This property is read-only.
Surrogate splits are not implemented, so this is always empty. This property is read-only.
Surrogate splits are not implemented, so this is always empty. This property is read-only.
Surrogate splits are not implemented, so this is always empty. This property is read-only.
Surrogate splits are not implemented, so this is always empty. This property is read-only.
Surrogate splits are not implemented, so this is always empty. This property is read-only.
A character vector naming the function predict applies to the
response it predicts before returning it, or a function handle taking
and returning an array of the same size. The default is
'none'.
The RegressionTree class offers the following public methods:
RegressionTree: obj = RegressionTree (X, Y)
RegressionTree: obj = RegressionTree (…, name, value)
obj = RegressionTree (X, Y) grows a tree on
the NxP numeric matrix X of predictor data and the
Nx1 numeric response Y, and returns it as a
RegressionTree object.
obj = RegressionTree (…, name, value)
takes the options below.
| Name | Value |
|---|---|
'CategoricalPredictors' | The predictors whose values
are levels, as indices, as a logical vector with one element per
predictor, or as 'all'. |
'MaxNumCategories' | A nonnegative integer, recorded
in ModelParameters. The default is 10. A regression orders
the levels by their mean response, which finds the best split whatever
the number of levels. |
'MaxNumSplits' | A nonnegative integer, the largest number of branch nodes the tree may take. The default is one less than the number of observations, which is as many as a tree can have. |
'MergeLeaves' | 'on' (default) or
'off'. When on, a pair of leaves whose parent is no worse
than the two of them together is merged back into that parent. |
'MinLeafSize' | A positive integer, the fewest observations a leaf may hold. The default is 1. A split leaving fewer than this on either side is not taken. |
'MinParentSize' | A positive integer, the fewest
observations a node must hold to be split at all. The default is 10.
The value the fit uses is max (MinParentSize, 2 * MinLeafSize),
since a smaller node cannot give both children a legal leaf. |
'PredictorNames' | A cell array of character vectors naming the columns of X. |
'Prune' | 'on' (default) or 'off'.
When on, the cost complexity pruning sequence is estimated and
reported in PruneList and PruneAlpha. The tree returned
is the unpruned one either way; prune takes a subtree out of
the sequence. |
'PruneCriterion' | 'mse', the only criterion
a regression tree has. |
'QuadraticErrorTolerance' | A positive scalar. A node whose squared error has fallen to this fraction of the root’s is not split further. The default is 1e-6. |
'ResponseName' | A character vector naming the
response. The default is 'Y'. |
'ResponseTransform' | A character vector naming a
transform to apply to the predicted response, or a function handle.
The default is 'none'. |
'SplitCriterion' | 'mse', the only criterion
a regression tree has. |
'Weights' | A nonnegative numeric vector with one element per observation. The default is uniform. |
See also: fitrtree, ClassificationTree, treetrain, treepredict
RegressionTree: yFit = predict (obj, XC)
RegressionTree: [yFit, node] = predict (…)
yFit = predict (obj, XC) sends each row of
XC down the tree and returns the mean response of the node it
comes to rest at, after ResponseTransform. XC must have
as many columns as the predictor data the model was fitted on.
[yFit, node] = predict (…) also returns the
number of the node each row landed in.
A row missing the predictor a node cuts on is stopped at that node and takes its answer, rather than being sent down a branch on evidence the row does not carry.
See also: RegressionTree, fitrtree
RegressionTree: obj2 = prune (obj)
RegressionTree: obj2 = prune (obj, 'Level', L)
RegressionTree: obj2 = prune (obj, 'Alpha', A)
RegressionTree: obj2 = prune (obj, 'Nodes', N)
obj2 = prune (obj) returns the tree unchanged.
obj2 = prune (obj, turns
every branch node whose 'Level', L)PruneList level is between one and
L into a leaf and discards everything below it. Level zero is
the tree itself and the largest level is the root alone. A level
above the largest prunes to the root and warns.
obj2 = prune (obj, prunes
to the smallest subtree that is optimal for the cost complexity
parameter A, which is the largest level whose 'Alpha', A)PruneAlpha
does not exceed it.
obj2 = prune (obj, turns
the branch nodes named in N into leaves, along with everything
below them, and leaves the rest of the tree alone.
'Nodes', N)
Pruning renumbers the nodes, so the properties of the returned tree are those of a tree of that shape and not a subset of the original’s.
See also: RegressionTree, fitrtree, RegressionTree.PruneList, RegressionTree.PruneAlpha
RegressionTree: CMdl = compact (obj)
CMdl = compact (obj) returns a
CompactRegressionTree object carrying the tree and everything
predict needs, but not the observations the model was fitted
on. It answers new data identically and is far smaller to keep or to
ship.
See also: CompactRegressionTree, RegressionTree
RegressionTree: CVMdl = crossval (obj)
RegressionTree: CVMdl = crossval (…, name, value)
CVMdl = crossval (obj) partitions the training data
into ten folds, or into as many folds as there are observations when
there are fewer than ten, grows a tree on the training part of each
and returns them as a RegressionPartitionedModel.
CVMdl = crossval (…, name, value) takes
one of the following, and one only.
| Name | Value |
|---|---|
'KFold' | An integer greater than 1, the number of folds. |
'Holdout' | A value between 0 and 1, the fraction of the data held out for testing, which gives a single fold. |
'Leaveout' | 'on' or 'off', one fold
per observation. |
'CVPartition' | A cvpartition object. |
Every fold is grown with the growth parameters the parent was grown with and a slice of its observation weights.
See also: RegressionPartitionedModel, RegressionTree, cvpartition
RegressionTree: E = cvloss (obj)
RegressionTree: [E, SE, Nleaf, BestLevel] = cvloss (obj)
RegressionTree: […] = cvloss (…, name, value)
E = cvloss (obj) partitions the training data into
ten folds, grows a tree on the training part of each, and returns the
mean squared error of the held-out part.
[E, SE, Nleaf, BestLevel] = cvloss
(…) also returns SE, the standard error of E over
the folds, Nleaf, the number of leaves each subtree holds, and
BestLevel, the pruning level chosen by 'TreeSize'. Each
has one element per subtree asked for.
[…] = cvloss (…, name, value) takes the
options below.
| Name | Value |
|---|---|
'SubTrees' | A vector of pruning levels in ascending
order, or 'all' for every level of the sequence. The default
is 0, the unpruned tree. |
'TreeSize' | 'se' (default), the smallest
subtree whose loss is within one standard error of the smallest loss,
or 'min', the smallest subtree of least loss. |
'KFold' | An integer greater than 1, the number of folds. The default is 10. A value above the number of observations is reduced to it. |
A fold’s tree is pruned to the level its own sequence gives for the geometric mean of the parent’s two neighbouring complexity parameters, which is the classical way a fold is matched to a subtree of the whole tree. The last level takes every fold’s tree back to its root. The partition is drawn over the observations rather than over a response there is nothing to stratify, and the loss is weighed by the model’s own weights.
The standard error is not MATLAB’s. This is the standard error of the loss over the folds, which is what the name means. Its value is not MATLAB’s, whose formula is not recoverable from what it reports; E, Nleaf and BestLevel are measured and match.
See also: RegressionTree, RegressionTree.prune, RegressionTree.crossval, RegressionTree.loss
RegressionTree: imp = predictorImportance (obj)
imp = predictorImportance (obj) returns a row
vector with one element per predictor, holding the total drop in risk
the splits on that predictor bring about, divided by the number of
branch nodes. A predictor the tree never splits on scores zero.
The drop at a branch node is its NodeRisk less the risk of its
two children and less what it holds back, so a predictor that is
chosen often, high up, and on nodes it separates well, scores highest.
The numbers are comparable between predictors of one tree and not
between trees.
See also: RegressionTree, fitrtree, RegressionTree.NodeRisk
RegressionTree: r = nodeVariableRange (obj, node)
r = nodeVariableRange (obj, node) returns a
structure with one field per predictor the path from the root to
node cuts on, holding the two-element range of values that reach
the node, or for a categorical predictor the levels that reach it. A
predictor the path never cuts on is unconstrained and is left out, so
the root gives a structure with no fields.
See also: RegressionTree, fitrtree
RegressionTree: view (obj)
view (obj) prints one line per node: a branch node names
the predictor it cuts on, the cut point, and the node each side leads
to, and a leaf names the response it fits. A branch node’s line ends
with the response it would fit itself, which is the answer an
observation missing that predictor gets.
See also: RegressionTree, fitrtree
RegressionTree: L = loss (obj, X, Y)
RegressionTree: L = loss (…, name, value)
L = loss (obj, X, Y) returns the
weighted mean squared error of the response the model predicts for
X against the observed response Y. A row whose response
is missing is dropped, as it is when fitting.
L = loss (…, name, value) takes the
following options.
| Name | Value |
|---|---|
'LossFun' | 'mse', the default, or a function
handle taking the true response, the predicted response and the
weights, and returning a numeric scalar. |
'Weights' | A numeric vector of observation weights, one per row of X, normalized to sum to one before it is applied. |
See also: RegressionTree, fitrtree, RegressionTree.predict
RegressionTree: yFit = resubPredict (obj)
yFit = resubPredict (obj) is
predict (obj, obj.X).
See also: RegressionTree, RegressionTree.predict
RegressionTree: L = resubLoss (obj)
RegressionTree: L = resubLoss (…, name, value)
L = resubLoss (obj) is loss over the
training data, weighed as the fit weighed it, and takes the same
'LossFun' option. Giving 'Weights' weighs the
training data some other way instead.
See also: RegressionTree, RegressionTree.loss
RegressionTree: savemodel (obj, filename)
savemodel (obj, filename) saves each property of a
RegressionTree object into an Octave binary file, the name of which is
specified in filename, along with an extra variable, which
defines the type of regression object these variables constitute. Use
loadmodel in order to load a regression object into Octave’s
workspace.
See also: loadmodel, fitrtree, RegressionTree