Skip to content

Home

Jump to bottom
Rafael Gumieri edited this page Aug 27, 2026 · 6 revisions

Nenya AI Gateway

A lightweight, zero-dependency AI API Gateway written in Go. Nenya sits between your AI coding clients and upstream LLM providers, adding secret redaction, context management, agent routing, and MCP tool integration — all with transparent SSE streaming. Security-hardened: non-root execution, mlock for secrets, seccomp + no-new-privileges.

Compatible with any provider that implements the OpenAI or Anthropic Chat Completions API. For 24 providers we ship built-in adapters with specialized handling.

go-version License zero-deps

How Nenya handles the requests

flowchart TD
    CLIENT["Client<br/>Cursor / OpenCode / Aider / etc.<br/>POST /v1/chat/completions · /v1/messages<br/>Bearer token"]

    subgraph GW["Nenya Gateway"]
        direction TB
        AUTH["Auth + RBAC"]
        RESOLVE["Parse body · resolve agent + targets<br/>strategy: fallback · round-robin · sticky"]
        CACHE{"Response cache"}
        MCPINJ["MCP auto-search + tool injection"]
    end

    CHAIN["Interceptor chain (best-effort)<br/>redact → entropy → TF-IDF → bouncer"]
    TRIM["Token budget trim (hard limit)"]

    subgraph LOOP["Dispatch loop — per target"]
        direction TB
        GUARDS["Circuit breaker · rate limits · cost guard"]
        MODES["A standard forward<br/>B MCP multi-turn tool loop<br/>C context-limit retry"]
    end

    UPSTREAM["Upstream LLM providers<br/>23 built-in adapters"]

    subgraph SSE["SSE pipeline"]
        direction TB
        PROBE["Stream-head probe (pre-header)<br/>empty / early-error failover"]
        XFORM["Adapter transforms · format conversion"]
        WATCH["Stall watchdog · stream continuation"]
        ACCT["Usage accounting · cache capture · MCP auto-save"]
    end

    OUT["Client receives transparent SSE"]

    CLIENT --> AUTH --> RESOLVE --> CACHE
    CACHE -- "HIT → replay" --> OUT
    CACHE -- "miss" --> MCPINJ --> CHAIN --> TRIM --> GUARDS --> MODES --> UPSTREAM
    UPSTREAM --> PROBE --> XFORM --> WATCH --> ACCT --> OUT
    PROBE -. "failover → next target" .-> GUARDS

    classDef io fill:#e8ebf0,stroke:#57606a,color:#1f2328
    classDef gw fill:#ddf4ff,stroke:#0969da,color:#1f2328
    classDef pipe fill:#fff8c5,stroke:#9a6700,color:#1f2328
    classDef sse fill:#dafbe1,stroke:#1a7f37,color:#1f2328

    class CLIENT,OUT io
    class GW,LOOP gw
    class CHAIN,TRIM pipe
    class SSE sse
Loading

Flow notes:

  • /v1/* endpoints require client bearer auth; /healthz, /statsz, /metrics do not.
  • Pipeline failures degrade gracefully and forward the request instead of returning a 500.
  • MCP-enabled agents can run local/remote tools without exposing MCP complexity to the client.
  • Sticky strategy pins a session (agent + system prompt + first user message) to one provider/model, keeping provider-side prefix caches warm across turns.
  • Upstream streams are probed before headers commit: empty streams or early SSE errors fail over to the next target, and interrupted streams are auto-resumed via stream continuation.

Features

Routing & Agents

  • Config-driven provider registry — add providers via JSON, zero code changes
  • 24 built-in providers with specialized adapters for wire format differences
  • Dynamic model discovery — fetches live model catalogs from providers at startup and on reload
  • Model registry — reference models by string shorthand with automatic provider/context resolution
  • Multi-provider model resolution — when a model exists in multiple providers, all are added to the agent's fallback chain
  • Three-tier model resolution — config overrides > discovered models > static registry
  • Per-model wire format — models from multi-format gateways (OpenCode Zen) auto-convert between OpenAI, Anthropic, and Gemini wire formats
  • Agent fallback chains — round-robin or sequential with circuit breaker and automatic failover
  • Latency-aware routing — auto-reorder targets by historical median response time with ±5% jitter to prevent thundering herd
  • Multi-account provider selection — LRU-based account pool with per-account credential, error classification, and backoff
  • Billing-aware routing — per-account spend tracking, quota polling (API/headers), exhaustion filtering, free model scoring bonuses, and per-agent budget limits
  • Per-agent system prompts — inline or file-based

Security & Privacy

  • Tier-0 regex secret filter — always-on redaction of AWS keys, GitHub tokens, passwords, etc.
  • Pluggable Interceptor Chain — priority-ordered pipeline (regex redaction, entropy filtering, TF-IDF relevance scoring, engine summarization) with per-step metrics
  • 3-Tier content pipeline — pass-through, engine summarization, or TF-IDF relevance-scored truncation
  • Context window compaction — sliding window summarization with configurable engine
  • Stale tool call pruning — compact old assistant+tool response pairs to save tokens
  • Thought pruning — strip reasoning blocks from assistant message history
  • Input validation — strict body limits, JSON sanitization, header filtering
  • Graceful degradation — never blocks requests due to engine or pipeline failures
  • Role-Based Access Control (RBAC) — per-API key roles (admin, user, read-only) with agent and endpoint restrictions
  • Secure memory — mlock-protected token storage, read-only sealing, core dump prevention

Hardening (Deployment Security)

  • Secure memory (default): All tokens stored in mlock-protected RAM, sealed read-only after init, core dumps disabled
  • Non-root execution — runs as UID 65532 with dropped capabilities
  • Memory protection — LimitMEMLOCK=infinity and LimitCORE=0 in systemd
  • Read-only filesystem — immutable root + private /tmp
  • Seccomp + no-new-privileges — restricted syscalls, prevents privilege escalation
  • Zero-trust secrets — loaded via systemd credentials or container mounts, never to disk
  • Socket activation — seamless restarts with zero dropped connections

Reliability

  • Zero external dependencies — Go standard library only
  • Hot reload — systemctl reload nenya for zero-downtime config changes
  • Circuit breaker — per agent+provider+model with automatic failover, exponential backoff, and semantic error classification
  • Rate limiting — per upstream host (RPM/TPM) with per-provider overrides
  • Response cache — in-memory LRU with SHA-256 fingerprinting
  • Context-Limit Auto-Retry — automatic summarization via engine and retry on upstream context-length errors
  • Local Engine Lifecycle — preload and unload local Ollama models with LRU eviction
  • Graceful shutdown — 30s grace period for in-flight requests, MCP client cleanup

MCP Tool Integration

  • Tool discovery — connect to MCP servers for automatic tool injection
  • Multi-turn execution — intercept tool calls, execute against MCP servers, forward results
  • Auto-search — pre-fetch relevant context from MCP servers before forwarding
  • Auto-save — persist assistant responses to MCP memory servers

Quick Start

1. Install

curl -fsSL https://raw.githubusercontent.com/gumieri/nenya/main/install.sh | sudo sh

2. Run with Podman

Create minimal config and secrets:

mkdir -p config secrets
cat > config/config.json << 'EOF'
{
  "server": { "listen_addr": ":8080" },
  "agents": {
    "default": {
      "strategy": "fallback",
      "models": ["gemini-3-flash"]
    }
  }
}
EOF

cat > secrets/provider_keys.json << 'EOF'
{
  "provider_keys": {
    "gemini": "AIza..."
  }
}
EOF

cat > secrets/client.json << 'EOF'
{
  "client_token": "nk-$(openssl rand -hex 32)"
}
EOF

Run the container:

podman run -d \
  --name nenya \
  -p 8080:8080 \
  -v ./config:/etc/nenya:ro \
  -v ./secrets:/run/secrets/nenya:ro \
  -e NENYA_SECRETS_DIR=/run/secrets/nenya \
  --cap-drop=ALL \
  --cap-add=IPC_LOCK \
  --security-opt=no-new-privileges:true \
  --read-only \
  --tmpfs /tmp:rw,noexec,nosuid,size=64M \
  ghcr.io/gumieri/nenya:latest

Test it:

curl -H "Authorization: Bearer $(jq -r '.client_token' secrets/client.json)" \
  http://localhost:8080/healthz

Runtime Configuration

Variable Default Description
PORT 8080 Listening port (overrides server.listen_addr)
HOST — Optional bind address (e.g. 127.0.0.1). Only used when combined with PORT
NENYA_CONFIG_DIR /etc/nenya/ Configuration directory path
NENYA_CONFIG_FILE — Single config file path (takes precedence over NENYA_CONFIG_DIR)
NENYA_SECRETS_DIR — Secrets directory (overrides CREDENTIALS_DIRECTORY)

Navigation

Getting Started

  • Quick Start — Detailed installation and first run
  • Client Setup — Configure OpenCode, Cursor, and other clients
  • Deployment — Bare metal (systemd), container, and Kubernetes guides

Core Concepts

  • Configuration — Full config reference with examples
  • Providers — All 24 providers, capabilities, and special behaviors
  • Routing — Latency-aware routing and fallback chains
  • Architecture — Package overview, request lifecycle, circuit breaker
  • MCP Integration — Model Context Protocol server integration

Reference

Operations

  • Demo — Step-by-step testing of all pipeline tiers
  • Troubleshooting — Common issues and solutions
  • FAQ — Frequently asked questions
  • Security — Security policy and vulnerability reporting

Project

  • Roadmap — Planned features and improvements
  • Disclaimer — Legal disclaimer and usage terms

Nenya on GitHub | Apache 2.0 License

Nenya on GitHub | Report an Issue | Apache 2.0 License

Getting Started

Core Concepts

Reference

Operations

  • Demo — Test all pipeline tiers
  • Troubleshooting — Common issues and solutions
  • FAQ — Frequently asked questions
  • Security — Security policy and vulnerability reporting

Project

Clone this wiki locally