Polynomial Regression and Regularization

This project is a beginner-friendly demonstration of overfitting and regularization with polynomial regression.

It will compare:

• Unregularized polynomial regression
• Ridge regression
• Lasso regression

The goal is educational clarity. Each module should stay small, readable, and easy to review.

Project Structure

Polynomial-Regression-Regularization/
├── README.md
├── requirements.txt
├── src/
│   ├── data_generator.py
│   ├── models.py
│   ├── experiments.py
│   ├── experiment/
│   │   ├── no_regularization.py
│   │   ├── l1_regression.py
│   │   ├── l2_regression.py
│   │   ├── auto_selection.py
│   │   ├── runner.py
│   │   └── common.py
│   └── plots.py
└── tests/

Planned Workflow

1. Generate a synthetic regression dataset.
2. Fit polynomial regression models with different degrees.
3. Compare unregularized, Ridge, and Lasso models.
4. Visualize how regularization changes model behavior.

Data Generation

The data generator supports both the reference essay dataset and custom polynomial datasets.

Use the default reference relationship:

from src.data_generator import generate_polynomial_data

x_values, y_values = generate_polynomial_data(
    n_samples=100,
    noise=0.1,
    random_state=42,
)

Use custom coefficients for any number of input variables:

from src.data_generator import generate_polynomial_data

coefficients = {
    (0, 0): 1.0,
    (1, 0): 2.0,
    (0, 2): -0.5,
}

x_values, y_values = generate_polynomial_data(
    n_samples=100,
    noise=0.1,
    random_state=42,
    coefficients=coefficients,
)

Generate a random polynomial dataset and keep the coefficients:

from src.data_generator import generate_random_polynomial_data

x_values, y_values, coefficients = generate_random_polynomial_data(
    n_samples=100,
    n_features=4,
    degree=2,
    random_state=42,
)

Use a compact coefficient input string:

from src.data_generator import generate_polynomial_data_from_input

input_text = "2;(0,0)=1;(1,0)=2;(0,2)=-0.5"

x_values, y_values, coefficients = generate_polynomial_data_from_input(
    input_text,
    n_samples=100,
    noise=0.1,
    random_state=42,
)

The input format is:

n;(e1,e2,...,en)=a;(e1,e2,...,en)=b;...

The first field n is a single positive integer. Here, (e1,e2,...,en) is the exponent tuple for one polynomial term. For example, (0,2) means the term using x2^2, and (1,0) means the term using x1.

Save generated data to CSV:

from src.data_generator import generate_polynomial_data, save_dataset_to_csv

x_values, y_values = generate_polynomial_data(
    n_samples=100,
    noise=0.1,
    random_state=42,
)

save_dataset_to_csv(x_values, y_values, "dataset.csv")

CSV Regression Experiments

Experiment helpers read generated datasets from CSV files. The expected CSV format is:

x1,x2,...,y
1.0,2.0,...,3.5

The y column is the target value, and all other columns are treated as input features.

Run unregularized, L1, and L2 polynomial regression from a CSV file:

from src.experiments import run_regression_comparison_from_csv

polynomials = run_regression_comparison_from_csv(
    csv_path="dataset.csv",
    degree=2,
    l1_alpha=0.01,
    l2_alpha=1.0,
)

print(polynomials["none"])
print(polynomials["l1"])
print(polynomials["l2"])

Each regression type is also available as a separate experiment file:

from src.experiment.no_regularization import fit_no_regularization_polynomial_from_csv
from src.experiment.l1_regression import fit_l1_polynomial_from_csv
from src.experiment.l2_regression import fit_l2_polynomial_from_csv

The output is a readable fitted polynomial, such as:

y = 1.02 + 1.98*x1 - 0.49*x2^2

The regression degree must be provided because polynomial regression first expands the input features up to a chosen maximum degree. L1 and L2 regularization shrink or remove coefficients after that feature expansion; they do not decide the maximum degree by themselves.

Automatically choose the regression type, degree, and regularization strength:

from src.experiments import find_best_regularized_polynomial_from_csv

results = find_best_regularized_polynomial_from_csv("dataset.csv")

print(results["best"]["regularization"])
print(results["best"]["degree"])
print(results["best"]["alpha"])
print(results["best"]["polynomial"])

This automatic search only requires the CSV file. It uses a deterministic train/validation split, tests unregularized, L1, and L2 models, searches degrees from 1 to 8, and searches regularization strengths from 1e-6 to 100 for L1/L2. The unregularized baseline uses alpha 0. The selected model is the one with the lowest validation MSE. If two models are effectively tied, the simpler lower-degree model is preferred.

Plotting

Basic matplotlib plots are available in src.plots.

Plot model predictions for a one-feature dataset:

from src.plots import plot_model_predictions

fig = plot_model_predictions(x_values, y_values, predictions)
fig.savefig("predictions.png")

Plot a simple model metric comparison:

from src.plots import plot_metric_comparison

fig = plot_metric_comparison({"none": 3.0, "l1": 1.0, "l2": 2.0})
fig.savefig("metrics.png")

Plot automatic-search validation MSE by degree:

from src.experiments import find_best_regularized_polynomial_from_csv
from src.plots import plot_best_validation_mse_by_degree

results = find_best_regularized_polynomial_from_csv("dataset.csv")
fig = plot_best_validation_mse_by_degree(results["candidates"])
fig.savefig("validation_mse_by_degree.png")

Setup

Install the project dependencies:

pip install -r requirements.txt

Run tests:

pytest

Current Status

This repository currently contains the initial project structure and a synthetic data generator based on the reference essay. It can also generate random polynomial datasets with custom feature counts and coefficients. CSV-based L1 and L2 polynomial regression helpers are also available alongside an unregularized baseline, including automatic degree and regularization-strength selection from a CSV file. Basic plotting helpers are available for model predictions, metric comparisons, and automatic-search validation curves.