🌀 Lorentzian Solver with Cauchy Loss — IRLS + L-BFGS

Author: Thibaut LOMBARD
License: MIT

This repository contains a robust solver based on a quasi-Lorentzian approximation of the Cauchy loss, implemented via an IRLS — Iteratively Reweighted Least Squares scheme.

The goal is to provide a robust alternative to classical methods (MSE, L-BFGS, Adam), capable of resisting outliers and avoiding degenerate minima linked to the Gaussian error hypothesis.

---

📚 Documentation PDF

Complete technical documentation is available in PDF format:

Language	Document	Description
🇫🇷 French	optimisation_quasi_lorentzienne_irls.pdf	10-page technical document with theory, algorithms, Python implementation, and fine-tuning applications
🇬🇧 English	quasi_lorentzian_irls_optimization.pdf	Full English translation covering IRLS optimization for LLM fine-tuning on limited hardware

Both documents include:

• Mathematical foundations (Cauchy loss, Lorentzian weights)
• Complete IRLS algorithm
• Python implementation (CauchyIRLSSolver class)
• Experimental results (linear and quadratic regression with outliers)
• Comparison with other anti-outlier methods (OWQ, SmoothQuant, AWQ, Huber)
• Fine-tuning integration guide for LLM (LoRA/QLoRA on Tesla T4)
• Industrial applications (Computer Vision, Robotics, Signal Processing)

---

🚀 Why a "Lorentzian" Solver?

Most optimizations use quadratic loss:

L_MSE = ∑ (y - fθ(x))²

It explodes in the presence of outliers.

The Cauchy loss, from a Lorentzian model, limits the influence of large errors:

L_Cauchy = ∑ log(1 + r² / σ²)

where r = y − fθ(x) and σ controls the scale of residuals.

---

🔁 Quasi-Lorentzian Approach via IRLS + L-BFGS

The Cauchy loss is not quadratic → difficult to minimize directly.
We iteratively approximate it by a weighted quadratic loss:

log(1 + r² / σ²) ≈ w(r) · r² + constant

with the Lorentzian weight:

w(r) = 1 / (σ² + r²)

→ Distant points (outliers) get low weight
→ Reliable points truly guide the descent

---

🧮 IRLS Loop

For data {(xi, yi)} and a model ŷ = fθ(x):

1. Initialize parameters θ
2. Choose or estimate σ
3. Repeat:

ri = yi - fθ(xi)          # residuals
wi = 1 / (σ² + ri²)       # Lorentzian weights
Minimize ∑ wi · (yi - fθ(xi))² with L-BFGS
σ ← median(|ri|)          # optional: automatic adaptation

→ Each iteration solves a locally quadratic problem
→ L-BFGS ensures stable and precise update
→ Outliers see their weight tend towards zero

---

🤖 Fine-Tuning LLM Application

This technique is particularly effective for fine-tuning Large Language Models on limited hardware:

Target Configuration

• GPU: NVIDIA Tesla T4 (or GTX 1050 4GB VRAM minimum)
• CPU: Intel i7-7700HQ
• RAM: 16GB DDR4
• Method: LoRA/QLoRA with 4-bit/16-bit quantization

Why Use IRLS Cauchy for LLM Fine-Tuning?

Problem	Standard MSE/CE	IRLS Cauchy
Outliers dominate loss	✅ Yes	❌ No (weighted down)
Gradient explosion	✅ Risk	❌ Stable
Noisy training data	❌ Poor handling	✅ Robust
4-bit quantization sensitivity	✅ High	❌ Reduced

Integration with PyTorch/Unsloth

import torch
from cauchy_irls import CauchyIRLSSolver

# Wrapper for SFTTrainer
def cauchy_loss_wrapper(logits, labels, sigma=0.5):
    """Cauchy loss wrapper for SFTTrainer"""
    residuals = logits - labels
    loss = torch.sum(torch.log(1 + (residuals / sigma) ** 2))
    return loss

# Complete IRLS training step
def irls_cauchy_training_step(model, batch, optimizer, 
                                sigma=1.0, max_inner_iter=10):
    """Training step with Cauchy IRLS"""
    for inner_iter in range(max_inner_iter):
        optimizer.zero_grad()
        # Forward pass
        outputs = model(**batch)
        residuals = outputs.logits - batch['labels']
        # Calculate Lorentzian weights
        weights = 1.0 / (sigma**2 + residuals**2 + 1e-8)
        # Weighted loss
        loss = torch.sum(weights * residuals**2)
        # Backward and optimization
        loss.backward()
        optimizer.step()
        # Adaptive sigma update
        with torch.no_grad():
            sigma = torch.median(torch.abs(residuals)) / 0.6745
    return loss

Google Colab Tesla T4

Ready-to-use notebook: Google Colab Tesla T4

---

🧠 Usage with Neural Networks (PyTorch)

With PyTorch, we cannot solve weighted least squares analytically.
We therefore apply an internal optimization loop — the IRLS idea remains identical.

import torch

def cauchy_irls_step(model, X, y, sigma=1.0):
    model.eval()
    with torch.no_grad():
        y_pred = model(X)
        r = y - y_pred
        w = 1.0 / (sigma**2 + r**2)

    model.train()
    optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)
    for _ in range(5):  # internal mini-optimization
        optimizer.zero_grad()
        y_pred = model(X)
        loss = (w * (y - y_pred)**2).mean()
        loss.backward()
        optimizer.step()

---

✅ Advantages

Criterion	Lorentzian
Sensitive to outliers	❌
Stable update	✔️ (L-BFGS)
Interpretable (weights)	✔️
Compatible with complex models	✔️
Automatic σ adaptation	✔️

---

📦 Installation

git clone https://github.com/Lombard-Web-Services/cauchy
python solver.py

---

🛠️ Quasi-Lorentzian Optimizer via IRLS

Transform any optimizer (L-BFGS, Adam, SGD…) into an outlier-robust solver — without rewriting a single line of code.

📘 Read the quick start guide
→ 2-minute startup, ready-to-use examples, immediate integration.

📚 Explore the complete documentation
→ Theory, IRLS scheme, Lorentzian weights, adaptive σ update, applications in vision, robotics and signal processing.

💻 Test the Python code directly
→ A single class (CauchyIRLSSolver), two methods (solve_linear, solve_nonlinear), NumPy/SciPy compatible, production-ready.

📄 Download PDF Documentation (FR)
→ Complete 10-page guide with fine-tuning focus.

📄 Download PDF Documentation (EN)
→ Full English version for international teams.