Robust Regression for Machine Learning in Python (original) (raw)

Last Updated : 23 Jul, 2025

Simple linear regression aims to find the best fit line that describes the linear relationship between some input variables(denoted by X) and the target variable(denoted by y). This has some limitations as in real-world problems, there is a high probability that the dataset may have outliers. This results in biased model fitting. To overcome this limitation of the biased fitted model, robust regression was introduced. In this article, we will learn about some state-of-the-art machine learning models which are robust to outliers.

One of the most used algorithms for Robust Regression is Random Sample Consensus (RANSAC). It is an iterative and non-deterministic method that is used to estimate the values of parameters used to build a machine-learning model from a set of observed data that contains outliers. When outliers are to be accorded, there is no influence on the estimated value. Hence, it also can be interpreted as a method for outlier detection. It produces a reasonable result only with a certain probability. This probability increases as more iterations are allowed.

Importing Libraries and Dataset

Here we will import a dataset and use it with some of the robust linear regression models. Python libraries make it easy for us to handle the data and perform typical and complex tasks with a single line of code.

import numpy as np import matplotlib.pyplot as plt import pandas as pd from sklearn import datasets from sklearn.preprocessing import StandardScaler from sklearn.model_selection import train_test_split from sklearn.linear_model import LinearRegression, RANSACRegressor from sklearn.metrics import r2_score, mean_squared_error

import warnings warnings.filterwarnings('ignore')

`

Sklearn datasets library contains some exemplary datasets for testing ideas and illustrations.

Python3 `

Load the Boston Housing dataset

for training

boston = datasets.load_boston()

Load the columns present in the dataset

df = pd.DataFrame(boston.data) df.columns = ['CRIM', 'ZN', 'INDUS', 'CHAS', 'NOX', 'RM', 'AGE', 'DIS', 'RAD', 'TAX', 'PTRATIO', 'B', 'LSTAT']

Set target as column "MEDV"

df['MEDV'] = boston.target

Select Avg. No of rooms per dwelling

as one feature

X = df['RM'].to_numpy().reshape(-1, 1) y = df['MEDV'].to_numpy().reshape(-1, 1)

`

RANSAC Regressor

In this model first data is separated into inliers and outliers then the model is trained on the inlier's data. Training model in this way helps the model to learn patterns instead of any noises.

Python3 `

Create a model

model = RANSACRegressor(base_estimator=LinearRegression(), min_samples=50, max_trials=100, loss='absolute_loss', random_state=42, residual_threshold=10)

Fit the model

model.fit(X, y)

`

Output:

RANSACRegressor(base_estimator=LinearRegression(), loss='absolute_loss',

min_samples=50, random_state=42, residual_threshold=10)

Now let's check the mean absolute error of the model.

Python3 `

y_pred = model.predict(X) print(metrics.mean_absolute_error(y, y_pred))

`

Output:

4.475672221331006

Theil Sen Regressor

This model is somehow similar to the random forest model in which we train multiple decision trees and average their results. This helps us to eliminate the problem of overfitting. This model also trains multiple regression models on the subsets of the training data and then the coefficients of those models are combined. This averaging step of the coefficient is exactly the step where the model becomes robust to the outliers.

Python3 `

from sklearn.linear_model import TheilSenRegressor

Create a model

model = TheilSenRegressor(random_state=42)

Fit the model

model.fit(X, y)

`

Output:

TheilSenRegressor(max_subpopulation=10000, random_state=42)

Now let's check the mean absolute error of the model.

Python3 `

y_pred = model.predict(X) print(metrics.mean_absolute_error(y, y_pred))

`

Output:

4.442032221450043

Huber Regressor

In this model, weights are optimized by giving higher preferences to the data points which are inliers. This again helps to learn the patterns specifically instead of the noise present in the data.

Python3 `

from sklearn.linear_model import HuberRegressor

Create a model

model = HuberRegressor()

Fit the model

model.fit(X, y)

`

Now let's check the mean absolute error of the model.

Python3 `

y_pred = model.predict(X) print(metrics.mean_absolute_error(y, y_pred))

`

Output:

4.437123637682936

If the dataset you are using contains a lot of outliers then do try to train above mentioned robust regression models and choose the best out of them by using the validation dataset.