Decision Tree Regressor, Explained: A Visual Guide with Code Examples | by Samy Baladram | Oct, 2024

REGRESSION ALGORITHM

Trimming branches neatly with cost-complexity pruning

Determination Timber aren’t restricted to categorizing information — they’re equally good at predicting numerical values! Classification bushes typically steal the highlight, however Determination Tree Regressors (or Regression Timber) are highly effective and versatile instruments on the earth of steady variable prediction.

Whereas we’ll talk about the mechanics of regression tree building (that are principally much like the classification tree), right here, we’ll additionally advance past the pre-pruning strategies like “minimal pattern leaf” and “max tree depth” launched within the classifier article. We’ll discover the commonest publish-pruning methodology which is price complexity pruning, that introduces a complexity parameter to the choice tree’s price operate.

A Determination Tree for regression is a mannequin that predicts numerical values utilizing a tree-like construction. It splits information based mostly on key options, ranging from a root query and branching out. Every node asks a couple of function, dividing information additional till reaching leaf nodes with ultimate predictions. To get a end result, you comply with the trail matching your information’s options from root to leaf.

To exhibit our ideas, we’ll work with our standard dataset. This dataset is used to foretell the variety of golfers visiting on a given day and consists of variables like climate outlook, temperature, humidity, and wind situations.

import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
# Create dataset
dataset_dict = {
'Outlook': ['sunny', 'sunny', 'overcast', 'rain', 'rain', 'rain', 'overcast', 'sunny', 'sunny', 'rain', 'sunny', 'overcast', 'overcast', 'rain', 'sunny', 'overcast', 'rain', 'sunny', 'sunny', 'rain', 'overcast', 'rain', 'sunny', 'overcast', 'sunny', 'overcast', 'rain', 'overcast'],
'Temp.': [85.0, 80.0, 83.0, 70.0, 68.0, 65.0, 64.0, 72.0, 69.0, 75.0, 75.0, 72.0, 81.0, 71.0, 81.0, 74.0, 76.0, 78.0, 82.0, 67.0, 85.0, 73.0, 88.0, 77.0, 79.0, 80.0, 66.0, 84.0],
'Humid.': [85.0, 90.0, 78.0, 96.0, 80.0, 70.0, 65.0, 95.0, 70.0, 80.0, 70.0, 90.0, 75.0, 80.0, 88.0, 92.0, 85.0, 75.0, 92.0, 90.0, 85.0, 88.0, 65.0, 70.0, 60.0, 95.0, 70.0, 78.0],
'Wind': [False, True, False, False, False, True, True, False, False, False, True, True, False, True, True, False, False, True, False, True, True, False, True, False, False, True, False, False],
'Num_Players': [52, 39, 43, 37, 28, 19, 43, 47, 56, 33, 49, 23, 42, 13, 33, 29, 25, 51, 41, 14, 34, 29, 49, 36, 57, 21, 23, 41]
}
df = pd.DataFrame(dataset_dict)
# One-hot encode 'Outlook' column
df = pd.get_dummies(df, columns=['Outlook'],prefix='',prefix_sep='')
# Convert 'Wind' column to binary
df['Wind'] = df['Wind'].astype(int)
# Break up information into options and goal, then into coaching and take a look at units
X, y = df.drop(columns='Num_Players'), df['Num_Players']
X_train, X_test, y_train, y_test = train_test_split(X, y, train_size=0.5, shuffle=False)

The Determination Tree for regression operates by recursively dividing the information based mostly on options that finest scale back prediction error. Right here’s the final course of:

1. Start with the complete dataset on the root node.
2. Select the function that minimizes a selected error metric (akin to imply squared error or variance) to separate the information.
3. Create youngster nodes based mostly on the break up, the place every youngster represents a subset of the information aligned with the corresponding function values.
4. Repeat steps 2–3 for every youngster node, persevering with to separate the information till a stopping situation is reached.
5. Assign a ultimate predicted worth to every leaf node, usually the common of the goal values in that node.

We’ll discover the regression half within the determination tree algorithm CART (Classification and Regression Timber). It builds binary bushes and usually follows these steps:

1.Start with all coaching samples within the root node.

2.For every function within the dataset:
a. Kind the function values in ascending order.
b. Take into account all midpoints between adjoining values as potential break up factors.

3. For every potential break up level:
a. Calculate the imply squared error (MSE) of the present node.
b. Compute the weighted common of errors for the ensuing break up.

4. After evaluating all options and break up factors, choose the one with lowest weighted common of MSE.

5. Create two youngster nodes based mostly on the chosen function and break up level:
– Left youngster: samples with function worth <= break up level
– Proper youngster: samples with function worth > break up level

6. Recursively repeat steps 2–5 for every youngster node. (Proceed till a stopping criterion is met.)

7. At every leaf node, assign the common goal worth of the samples in that node because the prediction.

from sklearn.tree import DecisionTreeRegressor, plot_tree
import matplotlib.pyplot as plt
# Prepare the mannequin
regr = DecisionTreeRegressor(random_state=42)
regr.match(X_train, y_train)
# Visualize the choice tree
plt.determine(figsize=(26,8))
plot_tree(regr, feature_names=X.columns, stuffed=True, rounded=True, impurity=False, fontsize=16, precision=2)
plt.tight_layout()
plt.present()

Right here’s how a regression tree makes predictions for brand new information:
1. Begin on the high (root) of the tree.
2. At every determination level (node):
– Take a look at the function and break up worth.
– If the information level’s function worth is smaller or equal, go left.
– If it’s bigger, go proper.
3. Hold transferring down the tree till you attain the tip (a leaf).
4. The prediction is the common worth saved in that leaf.

Analysis Step

After constructing the tree, the one factor we have to fear about is the tactic to make the tree smaller to forestall overfitting. Usually, the tactic of pruning could be categorized as:

Pre-pruning

Pre-pruning, also called early stopping, entails halting the expansion of a call tree in the course of the coaching course of based mostly on sure predefined standards. This strategy goals to forestall the tree from turning into too advanced and overfitting the coaching information. Widespread pre-pruning methods embody:

1. Most depth: Limiting how deep the tree can develop.
2. Minimal samples for break up: Requiring a minimal variety of samples to justify splitting a node.
3. Minimal samples per leaf: Making certain every leaf node has no less than a sure variety of samples.
4. Most variety of leaf nodes: Limiting the entire variety of leaf nodes within the tree.
5. Minimal impurity lower: Solely permitting splits that lower impurity by a specified quantity.

These strategies cease the tree’s development when the desired situations are met, successfully “pruning” the tree throughout its building part.
(We have discussed these methods before, which is exactly the same in regression case.)

Submit-pruning

Submit-pruning, however, permits the choice tree to develop to its full extent after which prunes it again to cut back complexity. This strategy first builds a whole tree after which removes or collapses branches that don’t considerably contribute to the mannequin’s efficiency. One frequent post-pruning approach is known as Value-Complexity Pruning.

Step 1: Calculate the Impurity for Every Node

For every interim node, calculate the impurity (MSE for regression case). We then sorted this worth from the bottom to highest.

# Visualize the choice tree
plt.determine(figsize=(26,8))
plot_tree(regr, feature_names=X.columns, stuffed=True, rounded=True, impurity=True, fontsize=16, precision=2)
plt.tight_layout()
plt.present()

Step 2: Create Subtrees by Trimming The Weakest Hyperlink

The objective is to regularly flip the interim nodes into leaves ranging from the node with the bottom MSE (= weakest hyperlink). We are able to create a path of pruning based mostly on that.

Step 3: Calculate Complete Leaf Impurities for Every Subtree

For every subtree T, whole leaf impurities (R(T)) could be calculated as:

R(T) = (1/N) Σ I(L) * n_L

the place:
· L ranges over all leaf nodes
· n_L is the variety of samples in leaf L
· N is the entire variety of samples within the tree
· I(L) is the impurity (MSE) of leaf L

Step 4: Compute the Value Operate

To regulate when to cease turning the interim nodes into leaves, we examine the associated fee complexity first for every subtree T utilizing the next formulation:

Value(T) = R(T) + α * |T|

the place:
· R(T) is the entire leaf impurities
· |T| is the variety of leaf nodes within the subtree
· α is the complexity parameter

Step 5: Choose the Alpha

The worth of alpha management which subtree we’ll find yourself with. The subtree with the bottom price would be the ultimate tree.

Whereas we will freely set the α, in scikit-learn, it’s also possible to get the smallest worth of α to acquire a specific subtree. That is known as efficient α.

# Compute the cost-complexity pruning path
tree = DecisionTreeRegressor(random_state=42)
effective_alphas = tree.cost_complexity_pruning_path(X_train, y_train).ccp_alphas
impurities = tree.cost_complexity_pruning_path(X_train, y_train).impurities
# Operate to depend leaf nodes
count_leaves = lambda tree: sum(tree.tree_.children_left[i] == tree.tree_.children_right[i] == -1 for i in vary(tree.tree_.node_count))
# Prepare bushes and depend leaves for every complexity parameter
leaf_counts = [count_leaves(DecisionTreeRegressor(random_state=0, ccp_alpha=alpha).fit(X_train_scaled, y_train)) for alpha in effective_alphas]
# Create DataFrame with evaluation outcomes
pruning_analysis = pd.DataFrame({
'total_leaf_impurities': impurities,
'leaf_count': leaf_counts,
'cost_function': [f"{imp:.3f} + {leaves}α" for imp, leaves in zip(impurities, leaf_counts)],
'effective_α': effective_alphas
})
print(pruning_analysis)

Ultimate Remarks

Pre-pruning strategies are typically quicker and extra memory-efficient, as they forestall the tree from rising too massive within the first place.

Submit-pruning can doubtlessly create extra optimum bushes, because it considers the complete tree construction earlier than making pruning selections. Nonetheless, it may be extra computationally costly.

Each approaches purpose to discover a steadiness between mannequin complexity and efficiency, with the objective of making a mannequin that generalizes properly to unseen information. The selection between pre-pruning and post-pruning (or a mixture of each) typically is dependent upon the particular dataset, the issue at hand, and naturally, computational assets obtainable.

In observe, it’s frequent to make use of a mixture of those strategies, like making use of some pre-pruning standards to forestall excessively massive bushes, after which utilizing post-pruning for fine-tuning the mannequin’s complexity.

import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.metrics import root_mean_squared_error
from sklearn.tree import DecisionTreeRegressor
from sklearn.preprocessing import StandardScaler
# Create dataset
dataset_dict = {
'Outlook': ['sunny', 'sunny', 'overcast', 'rain', 'rain', 'rain', 'overcast', 'sunny', 'sunny', 'rain', 'sunny', 'overcast', 'overcast', 'rain', 'sunny', 'overcast', 'rain', 'sunny', 'sunny', 'rain', 'overcast', 'rain', 'sunny', 'overcast', 'sunny', 'overcast', 'rain', 'overcast'],
'Temperature': [85.0, 80.0, 83.0, 70.0, 68.0, 65.0, 64.0, 72.0, 69.0, 75.0, 75.0, 72.0, 81.0, 71.0, 81.0, 74.0, 76.0, 78.0, 82.0, 67.0, 85.0, 73.0, 88.0, 77.0, 79.0, 80.0, 66.0, 84.0],
'Humidity': [85.0, 90.0, 78.0, 96.0, 80.0, 70.0, 65.0, 95.0, 70.0, 80.0, 70.0, 90.0, 75.0, 80.0, 88.0, 92.0, 85.0, 75.0, 92.0, 90.0, 85.0, 88.0, 65.0, 70.0, 60.0, 95.0, 70.0, 78.0],
'Wind': [False, True, False, False, False, True, True, False, False, False, True, True, False, True, True, False, False, True, False, True, True, False, True, False, False, True, False, False],
'Num_Players': [52,39,43,37,28,19,43,47,56,33,49,23,42,13,33,29,25,51,41,14,34,29,49,36,57,21,23,41]
}
df = pd.DataFrame(dataset_dict)
# One-hot encode 'Outlook' column
df = pd.get_dummies(df, columns=['Outlook'], prefix='', prefix_sep='', dtype=int)
# Convert 'Wind' column to binary
df['Wind'] = df['Wind'].astype(int)
# Break up information into options and goal, then into coaching and take a look at units
X, y = df.drop(columns='Num_Players'), df['Num_Players']
X_train, X_test, y_train, y_test = train_test_split(X, y, train_size=0.5, shuffle=False)
# Initialize Determination Tree Regressor
tree = DecisionTreeRegressor(random_state=42)
# Get the associated fee complexity path, impurities, and efficient alpha
path = tree.cost_complexity_pruning_path(X_train, y_train)
ccp_alphas, impurities = path.ccp_alphas, path.impurities
print(ccp_alphas)
print(impurities)
# Prepare the ultimate tree with the chosen alpha
final_tree = DecisionTreeRegressor(random_state=42, ccp_alpha=0.1)
final_tree.match(X_train_scaled, y_train)
# Make predictions
y_pred = final_tree.predict(X_test)
# Calculate and print RMSE
rmse = root_mean_squared_error(y_test, y_pred)
print(f"RMSE: {rmse:.4f}")