TreeBagger - Ensemble of bagged decision trees - MATLAB (original) (raw)

Ensemble of bagged decision trees

Description

A TreeBagger object is an ensemble of bagged decision trees for either classification or regression. Individual decision trees tend to overfit.Bagging, which stands for bootstrap aggregation, is an ensemble method that reduces the effects of overfitting and improves generalization.

Creation

The TreeBagger function grows every tree in theTreeBagger ensemble model using bootstrap samples of the input data. Observations not included in a sample are considered "out-of-bag" for that tree. The function selects a random subset of predictors for each decision split by using the random forest algorithm [1].

Syntax

Description

Tip

By default, the TreeBagger function grows classification decision trees. To grow regression decision trees, specify the name-value argumentMethod as "regression".

[Mdl](#mw%5Fc81f7559-993c-4728-97b0-f6c3e6f2978b) = TreeBagger([NumTrees](#mw%5F4336452d-cf59-4727-a2bd-c40afe6621d5),[Tbl](#mw%5F56dbafd4-a0d4-4811-b8cd-cb09ff05eeb8),[ResponseVarName](#mw%5F1c7889a4-e3b4-4532-8788-33ca561ede37)) returns an ensemble object (Mdl) of NumTrees bagged classification trees, trained by the predictors in the table Tbl and the class labels in the variable Tbl.ResponseVarName.

example

[Mdl](#mw%5Fc81f7559-993c-4728-97b0-f6c3e6f2978b) = TreeBagger([NumTrees](#mw%5F4336452d-cf59-4727-a2bd-c40afe6621d5),[Tbl](#mw%5F56dbafd4-a0d4-4811-b8cd-cb09ff05eeb8),[formula](#mw%5F5cfa92c7-05bc-4611-8b9f-b86481b51f4a)) returns Mdl trained by the predictors in the tableTbl. The input formula is an explanatory model of the response and a subset of predictor variables in Tbl used to fitMdl. Specify formula using Wilkinson Notation.

[Mdl](#mw%5Fc81f7559-993c-4728-97b0-f6c3e6f2978b) = TreeBagger([NumTrees](#mw%5F4336452d-cf59-4727-a2bd-c40afe6621d5),[Tbl](#mw%5F56dbafd4-a0d4-4811-b8cd-cb09ff05eeb8),[Y](#mw%5Faa94994c-b0c1-4125-9223-951db78ca1f3)) returns Mdl trained by the predictor data in the tableTbl and the class labels in the array Y.

[Mdl](#mw%5Fc81f7559-993c-4728-97b0-f6c3e6f2978b) = TreeBagger([NumTrees](#mw%5F4336452d-cf59-4727-a2bd-c40afe6621d5),[X](#mw%5F89fa73ad-81f8-4fa1-b2b8-a5fbf78cdec6),[Y](#mw%5Faa94994c-b0c1-4125-9223-951db78ca1f3)) returns Mdl trained by the predictor data in the matrixX and the class labels in the array Y.

example

[Mdl](#mw%5Fc81f7559-993c-4728-97b0-f6c3e6f2978b) = TreeBagger(___,[Name=Value](#namevaluepairarguments)) returns Mdl with additional options specified by one or more name-value arguments, using any of the previous input argument combinations. For example, you can specify the algorithm used to find the best split on a categorical predictor by using the name-value argument PredictorSelection.

example

Input Arguments

expand all

Number of decision trees in the bagged ensemble, specified as a positive integer.

Data Types: single | double

Sample data used to train the model, specified as a table. Each row ofTbl corresponds to one observation, and each column corresponds to one predictor variable. Optionally, Tbl can contain one additional column for the response variable. Multicolumn variables and cell arrays other than cell arrays of character vectors are not allowed.

• If Tbl contains the response variable, and you want to use all remaining variables in Tbl as predictors, then specify the response variable by using ResponseVarName.
• If Tbl contains the response variable, and you want to use only a subset of the remaining variables in Tbl as predictors, then specify a formula by using formula.
• If Tbl does not contain the response variable, then specify a response variable by using Y. The length of the response variable and the number of rows in Tbl must be equal.

Response variable name, specified as the name of a variable inTbl.

You must specify ResponseVarName as a character vector or string scalar. For example, if the response variable Y is stored asTbl.Y, then specify it as "Y". Otherwise, the software treats all columns of Tbl, including Y, as predictors when training the model.

The response variable must be a categorical, character, or string array; a logical or numeric vector; or a cell array of character vectors. If Y is a character array, then each element of the response variable must correspond to one row of the array.

A good practice is to specify the order of the classes by using theClassNames name-value argument.

Data Types: char | string

Explanatory model of the response variable and a subset of the predictor variables, specified as a character vector or string scalar in the form "Y~x1+x2+x3". In this form, Y represents the response variable, andx1, x2, and x3 represent the predictor variables.

To specify a subset of variables in Tbl as predictors for training the model, use a formula. If you specify a formula, then the software does not use any variables in Tbl that do not appear informula.

The variable names in the formula must be both variable names in Tbl (Tbl.Properties.VariableNames) and valid MATLAB® identifiers. You can verify the variable names in Tbl by using the isvarname function. If the variable names are not valid, then you can convert them by using the matlab.lang.makeValidName function.

Data Types: char | string

Class labels or response variable to which the ensemble of bagged decision trees is trained, specified as a categorical, character, or string array; a logical or numeric vector; or a cell array of character vectors.

• If you specify Method as "classification", the following apply for the class labels Y:
  • Each element of Y defines the class membership of the corresponding row of X.
  • If Y is a character array, then each row must correspond to one class label.
  • The TreeBagger function converts the class labels to a cell array of character vectors.
• If you specify Method as "regression", the response variable Y is an _n_-by-1 numeric vector, where n is the number of observations. Each entry inY is the response for the corresponding row ofX.

The length of Y and the number of rows of X must be equal.

Data Types: categorical | char | string | logical | single | double | cell

Predictor data, specified as a numeric matrix.

Each row of X corresponds to one observation (also known as an instance or example), and each column corresponds to one variable (also known as a feature).

The length of Y and the number of rows of X must be equal.

Data Types: double

Name-Value Arguments

expand all

Specify optional pairs of arguments asName1=Value1,...,NameN=ValueN, where Name is the argument name and Value is the corresponding value. Name-value arguments must appear after other arguments, but the order of the pairs does not matter.

Example: TreeBagger(100,X,Y,Method="regression",Surrogate="on",OOBPredictorImportance="on") creates a bagged ensemble of 100 regression trees, and specifies to use surrogate splits and to store the out-of-bag information for predictor importance estimation.

Number of observations in each chunk of data, specified as a positive integer. This option applies only when you use TreeBagger on tall arrays. For more information, see Extended Capabilities.

Example: ChunkSize=10000

Data Types: single | double

Misclassification cost, specified as a square matrix or structure.

• If you specify the square matrix Cost and the true class of an observation is i, then Cost(i,j) is the cost of classifying a point into class j. That is, rows correspond to the true classes and columns correspond to the predicted classes. To specify the class order for the corresponding rows and columns of Cost, use theClassNames name-value argument.
• If you specify the structure S, then it must have two fields:
  • S.ClassNames, which contains the class names as a variable of the same data type as Y
  • S.ClassificationCosts, which contains the cost matrix with rows and columns ordered as in S.ClassNames

The default value is Cost(i,j)=1 if i~=j, andCost(i,j)=0 if i=j.

For more information on the effect of a highly skewed Cost, seeAlgorithms.

Example: Cost=[0,1;2,0]

Data Types: single | double | struct

Categorical predictors list, specified as one of the values in this table.

By default, if the predictor data is in a table (Tbl), TreeBagger assumes that a variable is categorical if it is a logical vector, categorical vector, character array, string array, or cell array of character vectors. If the predictor data is a matrix (X), TreeBagger assumes that all predictors are continuous. To identify any other predictors as categorical predictors, specify them by using the CategoricalPredictors name-value argument.

For the identified categorical predictors, TreeBagger creates dummy variables using two different schemes, depending on whether a categorical variable is unordered or ordered. For an unordered categorical variable,TreeBagger creates one dummy variable for each level of the categorical variable. For an ordered categorical variable, TreeBagger creates one less dummy variable than the number of categories. For details, see Automatic Creation of Dummy Variables.

Example: CategoricalPredictors="all"

Data Types: single | double | logical | char | string | cell

Type of decision tree, specified as "classification" or"regression". For regression trees, Y must be numeric.

Example: Method="regression"

Minimum number of leaf node observations, specified as a positive integer. Each leaf has at least MinLeafSize observations per tree leaf. By default,MinLeafSize is 1 for classification trees and5 for regression trees.

Example: MinLeafSize=4

Data Types: single | double

Number of predictor variables (randomly selected) for each decision split, specified as a positive integer or "all". By default,NumPredictorsToSample is the square root of the number of variables for classification trees, and one third of the number of variables for regression trees. If the default number is not an integer, the software rounds the number to the nearest integer in the direction of positive infinity. If you set NumPredictorsToSample to any value except "all", the software uses Breiman's random forest algorithm [1].

Example: NumPredictorsToSample=5

Data Types: single | double | char | string

Number of grown trees (training cycles) after which the software displays a message about the training progress in the command window, specified as a nonnegative integer. By default, the software displays no diagnostic messages.

Example: NumPrint=10

Data Types: single | double

Fraction of input data to sample with replacement from the input data for growing each new tree, specified as a positive scalar in the range (0,1].

Example: InBagFraction=0.5

Data Types: single | double

Indicator to store out-of-bag information in the ensemble, specified as"on" or "off". SpecifyOOBPrediction as "on" to store information on which observations are out-of-bag for each tree. TreeBagger can use this information to compute the predicted class probabilities for each tree in the ensemble.

Example: OOBPrediction="off"

Indicator to store out-of-bag estimates of feature importance in the ensemble, specified as "on" or "off". If you specifyOOBPredictorImportance as "on", theTreeBagger function sets OOBPrediction to"on". If you want to analyze predictor importance, specifyPredictorSelection as "curvature" or"interaction-curvature".

Example: OOBPredictorImportance="on"

Options for computing in parallel and setting random streams, specified as a structure. Create the Options structure using statset. This table lists the option fields and their values.

Note

You need Parallel Computing Toolbox™ to run computations in parallel.

Example: Options=statset(UseParallel=true,UseSubstreams=true,Streams=RandStream("mlfg6331_64"))

Data Types: struct

Predictor variable names, specified as a string array of unique names or cell array of unique character vectors. The functionality of PredictorNames depends on how you supply the training data.

• If you supply X and Y, then you can usePredictorNames to assign names to the predictor variables inX.
  • The order of the names in PredictorNames must correspond to the column order of X. That is,PredictorNames{1} is the name of X(:,1),PredictorNames{2} is the name of X(:,2), and so on. Also, size(X,2) and numel(PredictorNames) must be equal.
  • By default, PredictorNames is{'x1','x2',...}.
• If you supply Tbl, then you can usePredictorNames to choose which predictor variables to use in training. That is, TreeBagger uses only the predictor variables inPredictorNames and the response variable during training.
  • PredictorNames must be a subset ofTbl.Properties.VariableNames and cannot include the name of the response variable.
  • By default, PredictorNames contains the names of all predictor variables.
  • A good practice is to specify the predictors for training using eitherPredictorNames or formula, but not both.

Example: PredictorNames=["SepalLength","SepalWidth","PetalLength","PetalWidth"]

Data Types: string | cell

Indicator for sampling with replacement, specified as "on" or"off". Specify SampleWithReplacement as"on" to sample with replacement, or as "off" to sample without replacement. If you set SampleWithReplacement to"off", you must set the name-value argumentInBagFraction to a value less than 1.

Example: SampleWithReplacement="on"

Prior probability for each class for two-class learning, specified as a value in this table.

If you specify a cost matrix, the Prior property of theTreeBagger model stores the prior probabilities adjusted for the misclassification cost. For more details, see Algorithms.

This argument is valid only for two-class learning.

Example: Prior=struct(ClassNames=["setosa" "versicolor" "virginica"],ClassProbs=1:3)

Data Types: char | string | single | double | struct

Output Arguments

expand all

Ensemble of bagged decision trees, returned as a TreeBagger object.

Properties

expand all

Bagging Properties

This property is read-only.

Indicator to compute out-of-bag predictions for training observations, specified as a numeric or logical 1 (true) or 0 (false). If this property is true:

• The TreeBagger object has the propertiesOOBIndices and OOBInstanceWeight.
• You can use the object functions oobError,oobMargin, and oobMeanMargin.

This property is read-only.

Indicator to compute the out-of-bag variable importance, specified as a numeric or logical 1 (true) or 0 (false). If this property istrue:

• The TreeBagger object has the propertiesOOBPermutedPredictorDeltaError,OOBPermutedPredictorDeltaMeanMargin, andOOBPermutedPredictorCountRaiseMargin.
• The property ComputeOOBPrediction is alsotrue.

This property is read-only.

Fraction of observations that are randomly selected with replacement (in-bag observations) for each bootstrap replica, specified as a numeric scalar. The size of each replica is Nobs_×InBagFraction, where_Nobs is the number of observations in the training data.

Data Types: single | double

This property is read-only.

Out-of-bag indices, specified as a logical array. This property is a_Nobs_-by-NumTrees array, where_Nobs_ is the number of observations in the training data, andNumTrees is the number of trees in the ensemble. If the`OOBIndices`(_i_,_j_) element is true, the observation i is out-of-bag for the tree j (that is, the TreeBagger function did not select the observation i for the training data used to grow the tree_j_).

This property is read-only.

Number of out-of-bag trees for each observation, specified as a numeric vector. This property is a Nobs_-by-1 vector, where Nobs is the number of observations in the training data. The`OOBInstanceWeight`(_i_) element contains the number of trees used for computing the out-of-bag response for observation_i.

Data Types: single | double

This property is read-only.

Predictor variable (feature) importance for raising the margin, specified as a numeric vector. This property is a 1-by-Nvars vector, where_Nvars_ is the number of variables in the training data. For each variable, the measure is the difference between the number of raised margins and the number of lowered margins if the values of that variable are permuted across the out-of-bag observations. This measure is computed for every tree, then averaged over the entire ensemble and divided by the standard deviation over the entire ensemble.

This property is empty ([]) for regression trees.

Data Types: single | double

This property is read-only.

Predictor variable (feature) importance for prediction error, specified as a numeric vector. This property is a 1-by-Nvars vector, where_Nvars_ is the number of variables (columns) in the training data. For each variable, the measure is the increase in prediction error if the values of that variable are permuted across the out-of-bag observations. This measure is computed for every tree, then averaged over the entire ensemble and divided by the standard deviation over the entire ensemble.

Data Types: single | double

This property is read-only.

Predictor variable (feature) importance for the classification margin, specified as numeric vector. This property is a 1-by-Nvars vector, where_Nvars_ is the number of variables (columns) in the training data. For each variable, the measure is the decrease in the classification margin if the values of that variable are permuted across the out-of-bag observations. This measure is computed for every tree, then averaged over the entire ensemble and divided by the standard deviation over the entire ensemble.

This property is empty ([]) for regression trees.

Data Types: single | double

Tree Properties

This property is read-only.

Split criterion contributions for each predictor, specified as a numeric vector. This property is a 1-by-Nvars vector, where_Nvars_ is the number of changes in the split criterion. The software sums the changes in the split criterion over splits on each variable, then averages the sums across the entire ensemble of grown trees.

Data Types: single | double

This property is read-only.

Indicator to merge leaves, specified as a numeric or logical 1 (true) or 0 (false). This property is true if the software merges the decision tree leaves with the same parent, for splits that do not decrease the total risk. Otherwise, this property is false.

This property is read-only.

Minimum number of leaf node observations, specified as a positive integer. Each leaf has at least MinLeafSize observations. By default,MinLeafSize is 1 for classification trees and 5 for regression trees. For decision tree training, fitctree and fitrtree set the name-value argument MinParentSize to2*MinLeafSize.

Data Types: single | double

This property is read-only.

Number of decision trees in the bagged ensemble, specified as a positive integer.

Data Types: single | double

This property is read-only.

Indicator to estimate the optimal sequence of pruned subtrees, specified as a numeric or logical 1 (true) or 0 (false). ThePrune property is true if the decision trees are pruned, and false if they are not. Pruning decision trees is not recommended for ensembles.

This property is read-only.

Indicator to sample each decision tree with replacement, specified as a numeric or logical 1 (true) or 0 (false). This property istrue if the TreeBagger function samples each decision tree with replacement, and false otherwise.

This property is read-only.

Predictive measures of variable association, specified as a numeric matrix. This property is an Nvars_-by-Nvars matrix, where_Nvars is the number of predictor variables. The property contains the predictive measures of variable association, averaged across the entire ensemble of grown trees.

• If you grow the ensemble with the Surrogate name-value argument set to "on", this matrix, for each tree, is filled with the predictive measures of association averaged over the surrogate splits.
• If you grow the ensemble with the Surrogate name-value argument set to "off", theSurrogateAssociation property is an identity matrix. By default, Surrogate is set to"off".

Data Types: single | double

This property is read-only.

Name-value arguments specified for the TreeBagger function, specified as a cell array. The TreeBagger function uses these name-value arguments when it grows new trees for the bagged ensemble.

This property is read-only.

Decision trees in the bagged ensemble, specified as a NumTrees-by-1 cell array. Each tree is a CompactClassificationTree orCompactRegressionTree object.

Predictor Properties

This property is read-only.

Number of decision splits for each predictor, specified as a numeric vector. This property is a 1-by-Nvars vector, where_Nvars_ is the number of predictor variables. Each element ofNumPredictorSplit represents the number of splits on the predictor summed over all trees.

Data Types: single | double

This property is read-only.

Number of predictor variables to select at random for each decision split, specified as a positive integer. By default, this property is the square root of the total number of variables for classification trees, and one third of the total number of variables for regression trees.

Data Types: single | double

This property is read-only.

Outlier measure for each observation, specified as a numeric vector. This property is a_Nobs_-by-1 vector, where Nobs is the number of observations in the training data.

Data Types: single | double

This property is read-only.

Predictor names, specified as a cell array of character vectors. The order of the elements in PredictorNames corresponds to the order in which the predictor names appear in the training data X.

This property is read-only.

Predictors used to train the bagged ensemble, specified as a numeric array. This property is a Nobs_-by-Nvars array, where_Nobs is the number of observations (rows) and Nvars is the number of variables (columns) in the training data.

Data Types: single | double

Response Properties

Default prediction value returned by predict oroobPredict, specified as "","MostPopular", or a numeric scalar. This property controls the predicted value returned by the predict or oobPredict object function when no prediction is possible (for example, when oobPredict predicts a response for an observation that is in-bag for all trees in the ensemble).

• For classification trees, you can set DefaultYfit to either"" or "MostPopular". If you specify"MostPopular" (default for classification), the property value is the name of the most probable class in the training data. If you specify "", the in-bag observations are excluded from computation of the out-of-bag error and margin.
• For regression trees, you can set DefaultYfit to any numeric scalar. The default value for regression is the mean of the response for the training data. If you set DefaultYfit to NaN, the in-bag observations are excluded from computation of the out-of-bag error and margin.

Example: Mdl.DefaultYfit="MostPopular"

Data Types: single | double | char | string

This property is read-only.

Class labels or response data, specified as a cell array of character vectors or a numeric vector.

• If you set the Method name-value argument to "classification", this property represents class labels. Each row of Y represents the observed classification of the corresponding row of X.
• If you set the Method name-value argument to"regression", this property represents response data and is a numeric vector.

Data Types: single | double | cell

Training Properties

This property is read-only.

Type of ensemble, specified as "classification" for classification ensembles or "regression" for regression ensembles.

This property is read-only.

Proximity between training data observations, specified as a numeric array. This property is a Nobs_-by-Nobs array, where_Nobs is the number of observations in the training data. The array contains measures of the proximity between observations. For any two observations, their proximity is defined as the fraction of trees for which these observations land on the same leaf. The array is symmetric, with ones on the diagonal and off-diagonal elements ranging from 0 to 1.

Data Types: single | double

This property is read-only.

Observation weights, specified as a vector of nonnegative values. This property has the same number of rows as Y. Each entry in W specifies the relative importance of the corresponding observation in Y. TheTreeBagger function uses the observation weights to grow each decision tree in the ensemble.

Data Types: single | double

Classification Properties

This property is read-only.

Unique class names used in the training model, specified as a cell array of character vectors.

This property is empty ([]) for regression trees.

This property is read-only.

Misclassification cost, specified as a numeric square matrix. The elementCost(i,j) is the cost of classifying a point into classj if its true class is i. The rows correspond to the true class and the columns correspond to the predicted class. The order of the rows and columns of Cost corresponds to the order of the classes inClassNames.

This property is empty ([]) for regression trees.

Data Types: single | double

This property is read-only.

Prior probabilities, specified as a numeric vector. The order of the elements inPrior corresponds to the order of the elements inMdl.ClassNames.

If you specify a cost matrix by using the Cost name-value argument of the TreeBagger function, the Prior property of theTreeBagger model object stores the prior probabilities (specified by thePrior name-value argument) adjusted for the misclassification cost. For more details, see Algorithms.

This property is empty ([]) for regression trees.

Data Types: single | double

Object Functions

expand all

Examples

collapse all

Create an ensemble of bagged classification trees for Fisher's iris data set. Then, view the first grown tree, plot the out-of-bag classification error, and predict labels for out-of-bag observations.

Load the fisheriris data set. Create X as a numeric matrix that contains four measurements for 150 irises. Create Y as a cell array of character vectors that contains the corresponding iris species.

load fisheriris X = meas; Y = species;

Set the random number generator to default for reproducibility.

Train an ensemble of bagged classification trees using the entire data set. Specify 50 weak learners. Store the out-of-bag observations for each tree. By default, TreeBagger grows deep trees.

Mdl = TreeBagger(50,X,Y,... Method="classification",... OOBPrediction="on")

Mdl = TreeBagger Ensemble with 50 bagged decision trees: Training X: [150x4] Training Y: [150x1] Method: classification NumPredictors: 4 NumPredictorsToSample: 2 MinLeafSize: 1 InBagFraction: 1 SampleWithReplacement: 1 ComputeOOBPrediction: 1 ComputeOOBPredictorImportance: 0 Proximity: [] ClassNames: 'setosa' 'versicolor' 'virginica'

Properties, Methods

Mdl is a TreeBagger ensemble for classification trees.

The Mdl.Trees property is a 50-by-1 cell vector that contains the trained classification trees for the ensemble. Each tree is a CompactClassificationTree object. View the graphical display of the first trained classification tree.

view(Mdl.Trees{1},Mode="graph")

Plot the out-of-bag classification error over the number of grown classification trees.

plot(oobError(Mdl)) xlabel("Number of Grown Trees") ylabel("Out-of-Bag Classification Error")

The out-of-bag error decreases as the number of grown trees increases.

Predict labels for out-of-bag observations. Display the results for a random set of 10 observations.

oobLabels = oobPredict(Mdl); ind = randsample(length(oobLabels),10); table(Y(ind),oobLabels(ind),... VariableNames=["TrueLabel" "PredictedLabel"])

ans=10×2 table TrueLabel PredictedLabel ______________ ______________

{'setosa'    }    {'setosa'    }
{'virginica' }    {'virginica' }
{'setosa'    }    {'setosa'    }
{'virginica' }    {'virginica' }
{'setosa'    }    {'setosa'    }
{'virginica' }    {'virginica' }
{'setosa'    }    {'setosa'    }
{'versicolor'}    {'versicolor'}
{'versicolor'}    {'virginica' }
{'virginica' }    {'virginica' }

Create an ensemble of bagged regression trees for the carsmall data set. Then, predict conditional mean responses and conditional quartiles.

Load the carsmall data set. Create X as a numeric vector that contains the car engine displacement values. Create Y as a numeric vector that contains the corresponding miles per gallon.

load carsmall X = Displacement; Y = MPG;

Set the random number generator to default for reproducibility.

Train an ensemble of bagged regression trees using the entire data set. Specify 100 weak learners.

Mdl = TreeBagger(100,X,Y,... Method="regression")

Mdl = TreeBagger Ensemble with 100 bagged decision trees: Training X: [94x1] Training Y: [94x1] Method: regression NumPredictors: 1 NumPredictorsToSample: 1 MinLeafSize: 5 InBagFraction: 1 SampleWithReplacement: 1 ComputeOOBPrediction: 0 ComputeOOBPredictorImportance: 0 Proximity: []

Properties, Methods

Mdl is a TreeBagger ensemble for regression trees.

For 10 equally spaced engine displacements between the minimum and maximum in-sample displacement, predict conditional mean responses (YMean) and conditional quartiles (YQuartiles).

predX = linspace(min(X),max(X),10)'; YMean = predict(Mdl,predX); YQuartiles = quantilePredict(Mdl,predX,... Quantile=[0.25,0.5,0.75]);

Plot the observations, estimated mean responses, and estimated quartiles.

hold on plot(X,Y,"o"); plot(predX,YMean) plot(predX,YQuartiles) hold off ylabel("Fuel Economy") xlabel("Engine Displacement") legend("Data","Mean Response",... "First Quartile","Median",..., "Third Quartile")

Create two ensembles of bagged regression trees, one using the standard CART algorithm for splitting predictors, and the other using the curvature test for splitting predictors. Then, compare the predictor importance estimates for the two ensembles.

Load the carsmall data set and convert the variables Cylinders, Mfg, and Model_Year to categorical variables. Then, display the number of categories represented in the categorical variables.

load carsmall Cylinders = categorical(Cylinders); Mfg = categorical(cellstr(Mfg)); Model_Year = categorical(Model_Year);

numel(categories(Cylinders))

numel(categories(Model_Year))

Create a table that contains eight car metrics.

Tbl = table(Acceleration,Cylinders,Displacement,... Horsepower,Mfg,Model_Year,Weight,MPG);

Set the random number generator to default for reproducibility.

Train an ensemble of 200 bagged regression trees using the entire data set. Because the data has missing values, specify to use surrogate splits. Store the out-of-bag information for predictor importance estimation.

By default, TreeBagger uses the standard CART, an algorithm for splitting predictors. Because the variables Cylinders and Model_Year each contain only three categories, the standard CART prefers splitting a continuous predictor over these two variables.

MdlCART = TreeBagger(200,Tbl,"MPG",... Method="regression",Surrogate="on",... OOBPredictorImportance="on");

TreeBagger stores predictor importance estimates in the property OOBPermutedPredictorDeltaError.

impCART = MdlCART.OOBPermutedPredictorDeltaError;

Train a random forest of 200 regression trees using the entire data set. To grow unbiased trees, specify to use the curvature test for splitting predictors.

MdlUnbiased = TreeBagger(200,Tbl,"MPG",... Method="regression",Surrogate="on",... PredictorSelection="curvature",... OOBPredictorImportance="on");

impUnbiased = MdlUnbiased.OOBPermutedPredictorDeltaError;

Create bar graphs to compare the predictor importance estimates impCART and impUnbiased for the two ensembles.

tiledlayout(1,2,Padding="compact");

nexttile bar(impCART) title("Standard CART") ylabel("Predictor Importance Estimates") xlabel("Predictors") h = gca; h.XTickLabel = MdlCART.PredictorNames; h.XTickLabelRotation = 45; h.TickLabelInterpreter = "none";

nexttile bar(impUnbiased); title("Curvature Test") ylabel("Predictor Importance Estimates") xlabel("Predictors") h = gca; h.XTickLabel = MdlUnbiased.PredictorNames; h.XTickLabelRotation = 45; h.TickLabelInterpreter = "none";

For the CART model, the continuous predictor Weight is the second most important predictor. For the unbiased model, the predictor importance of Weight is smaller in value and ranking.

Train an ensemble of bagged classification trees for observations in a tall array, and find the misclassification probability of each tree in the model for weighted observations. This example uses the data set airlinesmall.csv, a large data set that contains a tabular file of airline flight data.

When you perform calculations on tall arrays, MATLAB® uses either a parallel pool (default if you have Parallel Computing Toolbox™) or the local MATLAB session. To run the example using the local MATLAB session when you have Parallel Computing Toolbox, change the global execution environment by using the mapreducer function.

Create a datastore that references the location of the folder containing the data set. Select a subset of the variables to work with, and treat "NA" values as missing data so that the datastore function replaces them with NaN values. Create the tall table tt to contain the data in the datastore.

ds = datastore("airlinesmall.csv"); ds.SelectedVariableNames = ["Month" "DayofMonth" "DayOfWeek",... "DepTime" "ArrDelay" "Distance" "DepDelay"]; ds.TreatAsMissing = "NA"; tt = tall(ds)

tt =

M×7 tall table

Month    DayofMonth    DayOfWeek    DepTime    ArrDelay    Distance    DepDelay
_____    __________    _________    _______    ________    ________    ________

 10          21            3          642          8         308          12   
 10          26            1         1021          8         296           1   
 10          23            5         2055         21         480          20   
 10          23            5         1332         13         296          12   
 10          22            4          629          4         373          -1   
 10          28            3         1446         59         308          63   
 10           8            4          928          3         447          -2   
 10          10            6          859         11         954          -1   
  :          :             :           :          :           :           :
  :          :             :           :          :           :           :

Determine the flights that are late by 10 minutes or more by defining a logical variable that is true for a late flight. This variable contains the class labels Y. A preview of this variable includes the first few rows.

Y =

M×1 tall logical array

1 0 1 1 0 1 0 0 : :

Create a tall array X for the predictor data.

X =

M×6 tall double matrix

      10          21           3         642           8         308
      10          26           1        1021           8         296
      10          23           5        2055          21         480
      10          23           5        1332          13         296
      10          22           4         629           4         373
      10          28           3        1446          59         308
      10           8           4         928           3         447
      10          10           6         859          11         954
      :           :            :          :           :           :
      :           :            :          :           :           :

Create a tall array W for the observation weights by arbitrarily assigning double weights to the observations in class 1.

Remove the rows in X, Y, and W that contain missing data.

R = rmmissing([X Y W]); X = R(:,1:end-2); Y = R(:,end-1); W = R(:,end);

Train an ensemble of 20 bagged classification trees using the entire data set. Specify a weight vector and uniform prior probabilities. For reproducibility, set the seeds of the random number generators using rng and tallrng. The results can vary depending on the number of workers and the execution environment for the tall arrays. For details, see Control Where Your Code Runs.

rng("default") tallrng("default") tMdl = TreeBagger(20,X,Y,... Weights=W,Prior="uniform")

Evaluating tall expression using the Local MATLAB Session:

• Pass 1 of 1: Completed in 0.44 sec Evaluation completed in 0.47 sec Evaluating tall expression using the Local MATLAB Session:
• Pass 1 of 1: Completed in 1.5 sec Evaluation completed in 1.6 sec Evaluating tall expression using the Local MATLAB Session:
• Pass 1 of 1: Completed in 3.8 sec Evaluation completed in 3.8 sec

tMdl = CompactTreeBagger Ensemble with 20 bagged decision trees: Method: classification NumPredictors: 6 ClassNames: '0' '1'

Properties, Methods

tMdl is a CompactTreeBagger ensemble with 20 bagged decision trees. For tall data, the TreeBagger function returns a CompactTreeBagger object.

Calculate the misclassification probability of each tree in the model. Attribute a weight contained in the vector W to each observation by using the Weights name-value argument.

terr = error(tMdl,X,Y,Weights=W)

Evaluating tall expression using the Local MATLAB Session:

• Pass 1 of 1: Completed in 4.7 sec Evaluation completed in 4.7 sec

terr = 20×1

0.1420
0.1214
0.1115
0.1078
0.1037
0.1027
0.1005
0.0997
0.0981
0.0983
  ⋮

Find the average misclassification probability for the ensemble of decision trees.

More About

expand all

In addition to its Name-Value Arguments, theTreeBagger function accepts the following name-value arguments offitctree and fitrtree.

*When you specify the ClassNames name-value argument as a logical vector, use 0 and 1 values. Do not use false and true values. For example, you can specify ClassNames as [1 0 1].

Tips

• For a TreeBagger model Mdl, theTrees property contains a cell vector of Mdl.NumTrees CompactClassificationTree or CompactRegressionTree objects. View the graphical display of the t grown tree by entering:
• For regression problems, TreeBagger supports mean and quantile regression (that is, quantile regression forest [5]).
  • To predict mean responses or estimate the mean squared error given data, pass aTreeBagger model object and the data to predict or error, respectively. To perform similar operations for out-of-bag observations, use oobPredict or oobError.
  • To estimate quantiles of the response distribution or the quantile error given data, pass a TreeBagger model object and the data to quantilePredict or quantileError, respectively. To perform similar operations for out-of-bag observations, use oobQuantilePredict or oobQuantileError.
• Standard CART tends to select split predictors containing many distinct values, such as continuous variables, over those containing few distinct values, such as categorical variables[4]. Consider specifying the curvature or interaction test if either of the following is true:
  • The data has predictors with relatively fewer distinct values than other predictors; for example, the predictor data set is heterogeneous.
  • Your goal is to analyze predictor importance. TreeBagger stores predictor importance estimates in the OOBPermutedPredictorDeltaError property.
    For more information on predictor selection, see the name-value argumentPredictorSelection for classification trees or the name-value argument PredictorSelection for regression trees.

Algorithms

• If you specify the Cost,Prior, and Weights name-value arguments, the output model object stores the specified values in the Cost,Prior, and W properties, respectively. TheCost property stores the user-specified cost matrix (C) without modification. The Prior andW properties store the prior probabilities and observation weights, respectively, after normalization. For model training, the software updates the prior probabilities and observation weights to incorporate the penalties described in the cost matrix. For details, see Misclassification Cost Matrix, Prior Probabilities, and Observation Weights.
• The TreeBagger function generates in-bag samples by oversampling classes with large misclassification costs and undersampling classes with small misclassification costs. Consequently, out-of-bag samples have fewer observations from classes with large misclassification costs and more observations from classes with small misclassification costs. If you train a classification ensemble using a small data set and a highly skewed cost matrix, then the number of out-of-bag observations per class might be very low. Therefore, the estimated out-of-bag error might have a large variance and be difficult to interpret. The same phenomenon can occur for classes with large prior probabilities.
• For details on how the TreeBagger function selects split predictors, and for information on node-splitting algorithms when the function grows decision trees, seeAlgorithms for classification trees and Algorithms for regression trees.

References

[2] Breiman, Leo, Jerome Friedman, Charles J. Stone, and R. A. Olshen. Classification and Regression Trees. Boca Raton, FL: CRC Press, 1984.

[3] Loh, Wei-Yin. "Regression Trees with Unbiased Variable Selection and Interaction Detection." Statistica Sinica 12, no. 2 (2002): 361–386. https://www.jstor.org/stable/24306967.

[4] Loh, Wei-Yin, and Yu-Shan Shih. "Split Selection for Classification Trees." Statistica Sinica 7, no. 4 (1997): 815–840. https://www.jstor.org/stable/24306157.

[6] Genuer, Robin, Jean-Michel Poggi, Christine Tuleau-Malot, and Nathalie Villa-Vialanei. "Random Forests for Big Data." Big Data Research 9 (2017): 28–46. https://doi.org/10.1016/j.bdr.2017.07.003.

Extended Capabilities

expand all

This function supports tall arrays with the following limitations.

• The TreeBagger function supports these syntaxes for tallX, Y, and Tbl:
  • B = TreeBagger(NumTrees,Tbl,Y)
  • B = TreeBagger(NumTrees,X,Y)
  • B = TreeBagger(___,Name=Value)
• For tall arrays, the TreeBagger function supports classification but not regression.
• The TreeBagger function supports these name-value arguments:
  • NumPredictorsToSample — The default value is the square root of the number of variables for classification.
  • MinLeafSize — The default value is 1 if the number of observations is less than 50,000. If the number of observations is 50,000 or greater, then the default value is max(1,min(5,floor(0.01*NobsChunk))), where NobsChunk is the number of observations in a chunk.
  • ChunkSize (only for tall arrays) — The default value is50000.
    In addition, the TreeBagger function supports these name-value arguments of fitctree:
  • AlgorithmForCategorical
  • CategoricalPredictors
  • Cost — The columns of the cost matrix C cannot contain Inf or NaN values.
  • MaxNumCategories
  • MaxNumSplits
  • MergeLeaves
  • PredictorNames
  • PredictorSelection
  • Prior
  • Prune
  • PruneCriterion
  • SplitCriterion
  • Surrogate
  • Weights
• For tall data, the TreeBagger function returns a CompactTreeBagger object that contains most of the same properties as a fullTreeBagger object. The main difference is that the compact object is more memory efficient. The compact object does not contain properties that include the data, or that include an array of the same size as the data.
• The number of trees contained in the returned CompactTreeBagger object can differ from the number of trees specified as input to the TreeBagger function. TreeBagger determines the number of trees to return based on factors that include the size of the input data set and the number of data chunks available to grow trees.
• Supported CompactTreeBagger object functions are:
  • combine
  • error
  • margin
  • meanMargin
  • predict
  • setDefaultYfit
    The error, margin,meanMargin, and predict object functions do not support the name-value arguments Trees,TreeWeights, or UseInstanceForTree. The meanMargin function also does not support theWeights name-value argument.
• The TreeBagger function creates a random forest by generating trees on disjoint chunks of the data. When more data is available than is required to create the random forest, the function subsamples the data. For a similar example, see Random Forests for Big Data [6].
  Depending on how the data is stored, some chunks of data might contain observations from only a few classes out of all the classes. In this case, the TreeBagger function might produce inferior results compared to the case where each chunk of data contains observations from most of the classes.
• During training of the TreeBagger algorithm, the speed, accuracy, and memory usage depend on a number of factors. These factors include the values forNumTrees and the name-value arguments ChunkSize,MinLeafSize, and MaxNumSplits.
  For an _n_-by-p tall array X,TreeBagger implements sampling during training. This sampling depends on these variables:
  • Number of trees NumTrees
  • Chunk size ChunkSize
  • Number of observations n
  • Number of chunks r (approximately equal to_n_/ChunkSize)
    Because the value of n is fixed for a givenX, your settings for NumTrees andChunkSize determine how TreeBagger samplesX.
  1. If r > NumTrees, thenTreeBagger samples ChunkSize * NumTrees observations from X, and trains one tree per chunk (with each chunk containingChunkSize number of observations). This scenario is the most common when you work with tall arrays.
  2. If r ≤ NumTrees, thenTreeBagger trains approximatelyNumTrees/_r_ trees in each chunk, using bootstrapping within the chunk.
  3. If n ≤ ChunkSize, thenTreeBagger uses bootstrapping to generate samples (each of size_n_) on which to train individual trees.
• When you specify a value for NumTrees, consider the following:
  • If you run your code on Apache® Spark™, and your data set is distributed with Hadoop® Distributed File System (HDFS™), start by specifying a value for NumTrees that is at least twice the number of partitions in HDFS for your data set. This setting prevents excessive data communication amongApache Spark executors and can improve performance of the TreeBagger algorithm.
  • TreeBagger copies fitted trees into the client memory in the resulting CompactTreeBagger model. Therefore, the amount of memory available to the client creates an upper bound on the value you can set forNumTrees. You can tune the values of MinLeafSize andMaxNumSplits for more efficient speed and memory usage at the expense of some predictive accuracy. After tuning, if the value of NumTrees is less than twice the number of partitions in HDFS for your data set, then consider repartitioning your data in HDFS to have larger partitions.
    After you specify a value for NumTrees, setChunkSize to ensure that TreeBagger uses most of the data to grow trees. Ideally, ChunkSize * NumTrees should approximate_n_, the number of rows in your data. Note that the memory available in the workers for training individual trees can also determine an upper bound forChunkSize.
    You can adjust the Apache Spark memory properties to avoid out-of-memory errors and support your workflow. See parallel.cluster.Hadoop (Parallel Computing Toolbox) for more information.

For more information, see Tall Arrays for Out-of-Memory Data.

To run in parallel, specify the Options name-value argument in the call to this function and set the UseParallel field of the options structure to true usingstatset:

Options=statset(UseParallel=true)

For more information about parallel computing, see Run MATLAB Functions with Automatic Parallel Support (Parallel Computing Toolbox).

Version History

Introduced in R2009a

expand all

Starting in R2022a, the Cost property stores the user-specified cost matrix. The software stores normalized prior probabilities (Prior) and observation weights (W) that do not reflect the penalties described in the cost matrix.

Note that model training has not changed and, therefore, the decision boundaries between classes have not changed.

For training, the fitting function updates the specified prior probabilities by incorporating the penalties described in the specified cost matrix, and then normalizes the prior probabilities and observation weights. This behavior has not changed. In previous releases, the software stored the default cost matrix in the Cost property and stored the prior probabilities and observation weights used for training in thePrior and W properties, respectively. Starting in R2022a, the software stores the user-specified cost matrix without modification, and stores normalized prior probabilities and observation weights that do not reflect the cost penalties. For more details, see Misclassification Cost Matrix, Prior Probabilities, and Observation Weights.

The oobError and oobMeanMargin functions use the observation weights stored in the W property. Therefore, if you specify a nondefault cost matrix when you train a classification model, the object functions return a different value compared to previous releases.

If you want the software to handle the cost matrix, prior probabilities, and observation weights in the same way as in previous releases, adjust the prior probabilities and observation weights for the nondefault cost matrix, as described in Adjust Prior Probabilities and Observation Weights for Misclassification Cost Matrix. Then, when you train a classification model, specify the adjusted prior probabilities and observation weights by using the Prior and Weights name-value arguments, respectively, and use the default cost matrix.

See Also

Objects

• CompactTreeBagger | ClassificationTree | RegressionTree

Functions

• predict | oobPredict | error | oobError | view | view | fitctree | fitrtree

Topics

• Bootstrap Aggregation (Bagging) of Regression Trees Using TreeBagger
• Bootstrap Aggregation (Bagging) of Classification Trees Using TreeBagger
• Framework for Ensemble Learning
• Working with Quantile Regression Models