LLM Gateway AI

An gateway that handles retries of chat with LLMs.

Architecture

flowchart LR
    Client["Client"]
    LLM["LLM Client"]
    Issue["Issue"]
    Retryable["Retryable Errors (rate limit / server issues)"]
    NonRetryable["Non-Retryable Issues (user issues)"]

    Client -->|"POST /client"| LLM
    LLM -->|response| Client
    LLM --> Issue
    Issue -->|"non-retriable / user issue"| NonRetryable
    Issue -->|"rate limit / server issue"| Retryable
    Retryable -->|"loop: retry"| LLM

Loading

Upcoming

• fallback providers.
• circuit breakers.