Smart ML Gateway

Secure Deployment Infrastructure for Machine Learning Models.

A high-performance MLOps tool written in Go, designed to act as an intelligent gateway in front of ML models. The goal is to enable advanced deployment strategies (Canary, Shadow, Blue-Green) without coupling to the client or model code.

Main Features

• High Performance: Built on the Gin framework and using Goroutines for concurrent processing with very low overhead.

• Canary Deployment: Weighted traffic distribution (e.g., 80% v1, 20% v2) for gradual validation with real users.

• Shadow Mode (Dark Launch): Real-time traffic mirroring. The request goes to V1 (production) and V2 (candidate) simultaneously.

• The user receives the response from V1 instantly.

• The V2 response is compared asynchronously (JSON Diff) for regression detection.

• Blue-Green Deployment: Instant major version switching via API, with no downtime.

• Dynamic Control Plane: Internal REST API for changing strategies at runtime (Hot-Reload configuration).

How to Run

Prerequisites

• Go 1.25+

• Two ML models running locally (or via Docker) for testing.

• Example: V1 on port 8001 and V2 on port 8002.

Running the Gateway

1. Clone the repository and download the dependencies:

go mod tidy

2. Start the server:

go run cmd/server/main.go

The server will start on port :8080.

API Reference

1. Data Plane (Inference)

Public route used by clients to obtain predictions.

• POST /predict

• Response Headers:

• X-Model-Used: Indicates which model responded (v1 or v2).

• X-Latency: Total processing time.

curl -X POST http://localhost:8080/predict \

-H "Content-Type: application/json" \

-d '{"sepal_length": 5, "sepal_width": 4, "petal_length": 1, "petal_width": 0}'

2. Control Plane (Administration)

Internal route to manage deployment strategies.

• PUT /admin/config

Example: Enable Canary (20% for V2)

{
"primary_url": "http://localhost:8001/predict",
"candidate_url": "http://localhost:8002/predict",
"active_strategy": "canary",
"canary_weight": 0.2
}

Example: Enable Shadow Mode

{
"primary_url": "http://localhost:8001/predict",
"candidate_url": "http://localhost:8002/predict",
"active_strategy": "shadow",

"canary_weight": 0.0
}

Roadmap & Next Steps

• Deployment Strategies (Canary, Shadow, Blue-Green)
• Basic Observability (Latency and Diff Logs)

---

Developed as an SRE/MLOps tool to ensure the sanity of models in production.