diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 6dfc4d6..cc9dc30 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -17,6 +17,7 @@ jobs: cache: pnpm - run: pnpm install --frozen-lockfile - run: pnpm verify + - run: pnpm build - run: pnpm audit --prod --audit-level=high e2e: diff --git a/CHANGELOG.md b/CHANGELOG.md index aab7c1a..97fdb83 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,16 @@ All notable changes to Sentinel are documented here. The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## [Unreleased] + +### Fixed + +- The gateway production build (`pnpm build`) no longer pulls dashboard sources into the Node build; CI now runs `pnpm build` on every PR so it can't silently break again. + +### Changed + +- Documentation-accuracy pass: corrected stack/roadmap references (Node load harness, not k6; single-node in-memory cache + SQLite storage; dashboard is shipped; providers are OpenAI-compatible with native Anthropic planned), reconciled `package.json` versions to `0.1.0`, and added a **Known limitations** section to the README. + ## [0.1.0] — 2026-06-28 First public release: a self-hostable, OpenAI-compatible **verifying LLM gateway** that @@ -44,6 +54,6 @@ dashboard. Adopting it is a one-line base-URL change. - Per-client rate limiting (`CLIENT_RPM`) keyed by hashed API key — a single noisy key gets 429s without affecting other clients. - Upstream fetches refuse redirects (SSRF hardening); authorization headers redacted from logs. - CI runs typecheck + lint + the full coverage gate plus a production dependency audit. Security posture tracked in `SECURITY_REVIEW_LOG.md`. -- Reproducible load harness (`pnpm load`) producing the headline benchmarks in `load/RESULTS.md`: 50% cache cost-reduction on a 50%-repeat workload, zero unhandled 429s, 100% PII guardrail catch-rate, and ~14 ms p99 added overhead. +- Reproducible load harness (`pnpm load`) producing the headline benchmarks in `load/RESULTS.md`: 50% cache cost-reduction on a 50%-repeat workload, zero unhandled 429s, 100% PII guardrail catch-rate, and ~14 ms p99 added overhead (measured in-process against **mock upstreams**; the catch-rate is the deterministic-guardrail rate — see `load/RESULTS.md` for full method and caveats). [0.1.0]: https://github.com/MANVENDRA-github/sentinel/releases/tag/v0.1.0 diff --git a/CLAUDE.md b/CLAUDE.md index ba3e7f2..11fa30f 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -13,15 +13,15 @@ The product source of truth is `PRP_SPEC.md`. The build order is `ROADMAP.md`. B - Package manager / monorepo: **pnpm** workspaces. - Gateway server: **Fastify** (streaming, hooks, `.inject()` testing). - Schemas & validation: **Zod** at every trust boundary (config, request/response, guardrails). -- Providers: OpenAI-compatible interface (`openai` SDK) with adapters for **OpenAI, Anthropic, Google Gemini (AI Studio), Ollama**. +- Providers: a unified, hand-rolled **OpenAI-compatible** HTTP adapter (`fetch`) — OpenAI, Groq, Gemini's OpenAI endpoint, Ollama, and other OpenAI-API providers. A native **Anthropic** (Messages API) adapter is on the near-term roadmap. - Local models (keyless): **Ollama** — judge `qwen2.5:7b`, embeddings `nomic-embed-text`. -- Cache & rate-limit state: **Redis** (`ioredis`); in-memory fallback for dev/tests. -- Storage (traces, eval results): **better-sqlite3** (dev) / **Postgres** (prod). +- Cache & rate-limit state: **in-memory** (single-node v1). A **Redis** (`ioredis`) backend is a planned swap behind the cache/throttle interfaces. +- Storage (traces, eval results): **better-sqlite3** / in-memory (single-node v1). A **Postgres** backend is a planned swap behind the `TraceStore` interface. - Telemetry: **OpenTelemetry** (`@opentelemetry/sdk-node`). Logging: **pino**. - Tests: **Vitest** (unit/integration), **Playwright** (E2E + Playwright MCP for the dashboard). - Quality: **ESLint** (flat config, `typescript-eslint` strict) + **Prettier**. -- Infra: **Docker** + docker-compose (gateway + redis + ollama). CI: **GitHub Actions**. -- Dashboard (later phase): **React + Vite + TS**. +- Infra: **Docker** + docker-compose for local dependencies (redis + ollama). CI: **GitHub Actions**. +- Dashboard: **React + Vite + TS**. **Commands** (the only canonical ones — use these, don't invent): - Install: `pnpm install` @@ -37,8 +37,8 @@ The product source of truth is `PRP_SPEC.md`. The build order is `ROADMAP.md`. B **Structure** (target monorepo — build it per `ROADMAP.md`, don't scaffold ahead of the current phase): - `packages/gateway/` — the Fastify proxy: providers, router, cache, verify (guardrails + judge), telemetry. The core. - `packages/sdk/` — thin client/config helpers (later phase). -- `packages/dashboard/` — React + Vite UI (later phase). -- Tests: `*.test.ts` next to source (unit); `packages/*/test/` (integration); `e2e/` (Playwright). +- `packages/dashboard/` — React + Vite UI (shipped). +- Tests: `*.test.ts` next to source (unit + integration via Fastify `.inject()`); `packages/dashboard/e2e/` (Playwright). - Root docs: `PRP_SPEC.md`, `ROADMAP.md`, `PLAN_TEMPLATE.md`, `TEST_CONTRACT.md`, `SECURITY_REVIEW_LOG.md`. ## Coding style (strict — these are not suggestions) diff --git a/README.md b/README.md index 14a8c3d..be7921f 100644 --- a/README.md +++ b/README.md @@ -6,7 +6,7 @@ A self-hostable **verifying LLM gateway** — a drop-in, OpenAI-compatible proxy that **routes** (cheapest capable model + fallback), **semantically caches**, and **verifies** (deterministic guardrails inline + a local Ollama judge) every LLM call, with full OpenTelemetry tracing. Unlike after-the-fact observability tools, it can flag or block a bad response _before it returns_. -> **v0.1.0 — feature-complete and hardened.** All seven build phases are done: drop-in proxy, OpenTelemetry tracing, semantic cache, routing / fallback / rate-limit survival, in-path verification (guardrails + a local judge), a React dashboard, and security hardening with a reproducible load harness. Product spec in [`PRP_SPEC.md`](./PRP_SPEC.md), phased build in [`ROADMAP.md`](./ROADMAP.md), contributor/agent guidance in [`CLAUDE.md`](./CLAUDE.md). +> **v0.1.0 — first release.** All build phases (0–7) are done: drop-in proxy, OpenTelemetry tracing, semantic cache, routing / fallback / rate-limit survival, in-path verification (guardrails + a local judge), a React dashboard, and security hardening with a reproducible load harness. It runs **single-node** (in-memory cache + SQLite) — see [Known limitations](#known-limitations). Product spec in [`PRP_SPEC.md`](./PRP_SPEC.md), phased build in [`ROADMAP.md`](./ROADMAP.md), contributor/agent guidance in [`CLAUDE.md`](./CLAUDE.md). ## What it does @@ -89,7 +89,7 @@ A provider may also set an `rpm` (requests-per-minute) ceiling; the optional `ro ### Bring your own Ollama -Sentinel never hardcodes a machine or a model. Set `OLLAMA_BASE_URL` in `.env` to point at _your_ Ollama (default `http://localhost:11434/v1`), list the models you've pulled in the `models` map, and you're set. The local **judge** that verifies outputs in a later phase works the same way — it reads `JUDGE_MODEL` from your environment, so every clone uses its own local model, with no API key and no quota. +Sentinel never hardcodes a machine or a model. Set `OLLAMA_BASE_URL` in `.env` to point at _your_ Ollama (default `http://localhost:11434/v1`), list the models you've pulled in the `models` map, and you're set. The local **judge** that verifies outputs works the same way — it reads `JUDGE_MODEL` from your environment, so every clone uses its own local model, with no API key and no quota. ## Observability @@ -155,6 +155,17 @@ Real numbers from the bundled load harness (`pnpm load` spins up a mock upstream Framing, stated plainly: overhead is Sentinel's _own_ per-request cost, not a model's latency; cost-reduction scales with your traffic's repeat rate; and the catch-rate is the deterministic guardrail rate — the async LLM judge needs a real model and is covered by the unit tests. Reproduce it all with `pnpm load`. +## Known limitations + +Sentinel v0.1.0 is a **single-node, self-hosted** gateway. Honest boundaries today: + +- **In-memory cache & rate-limit state.** The semantic cache and token buckets live in-process — not shared across instances, and cleared on restart. A Redis-backed swap is planned behind the existing interfaces. +- **SQLite (or in-memory) trace storage.** Ideal for single-node; a Postgres backend for multi-instance is planned. +- **The async judge needs a local Ollama** with `JUDGE_MODEL` pulled. Without it, judging degrades to `unscored` (never a false pass). The bundled benchmarks run against **mock upstreams**, so the headline catch-rate is the _deterministic guardrail_ rate; the LLM judge is covered by unit tests. +- **Inline guardrails apply to non-streaming responses.** Streamed responses are judged from their buffered output _after_ completion (inline blocking of a live stream is on the roadmap). +- **Providers are OpenAI-compatible.** OpenAI, Groq, Gemini's OpenAI endpoint, Ollama, and other OpenAI-API providers work today; a native **Anthropic** (Messages API) adapter is planned. +- **Cost reduction is measured by request volume** (cache hits × avoided upstream calls) on a repeat-heavy workload; per-request dollar accounting is planned. + ## Development ```bash diff --git a/ROADMAP.md b/ROADMAP.md index 84629a9..5d4b174 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -52,9 +52,9 @@ A modular, sequential build. **Work one phase at a time.** Each phase is sized t ## Phase 7 — Hardening, load test & launch **Goal:** prove the numbers; ship it. -**Context:** `e2e/`, `load/` (k6), `README.md`, docs. -**Deliverables:** k6 load test; measured cost-reduction %, p99 overhead, 429-elimination, quality catch-rate (fills `PRP_SPEC.md` §5); full security pass (`SECURITY_REVIEW_LOG.md`); README + quickstart; OSS release. -**Exit:** headline metrics captured and documented; security checklist complete; v1.0 tagged. +**Context:** `packages/dashboard/e2e/`, `load/` (Node harness, `load/run.ts`), `README.md`, docs. +**Deliverables:** Node load harness (`load/run.ts`); measured cost-reduction %, p99 overhead, 429-elimination, quality catch-rate (fills `PRP_SPEC.md` §5); full security pass (`SECURITY_REVIEW_LOG.md`); README + quickstart; OSS release. +**Exit:** headline metrics captured and documented; security checklist complete; v0.1.0 tagged. ## Phase 8 — (Advanced, optional) Fine-tuned judge **Goal:** a sharper, cheaper judge — and the model-internals milestone. diff --git a/TEST_CONTRACT.md b/TEST_CONTRACT.md index 914d56b..e393d2e 100644 --- a/TEST_CONTRACT.md +++ b/TEST_CONTRACT.md @@ -21,7 +21,7 @@ A PR whose diff adds or changes behavior without a corresponding test that would | Integration | **Vitest** + Fastify **`.inject()`** | The HTTP surface with providers / Redis / Ollama **mocked or faked**. Request → route → cache → verify → response. | | Contract | **Vitest** + recorded fixtures | Provider adapters against **recorded** real responses (record once, replay in CI). Catches provider-shape drift. | | E2E | **Playwright** (+ **Playwright MCP**) | Dashboard flows against a seeded gateway. Agent-driven exploratory passes via Playwright MCP. | -| Load | **k6** (Phase 7) | Throughput, p99 overhead, 429-elimination, cost-reduction numbers. | +| Load | **Node harness** (`load/run.ts`) | Throughput, p99 overhead, 429-elimination, cost-reduction numbers. | ## Determinism rules (CI must be hermetic) @@ -49,6 +49,6 @@ A PR whose diff adds or changes behavior without a corresponding test that would ## Conventions -- Test files: `*.test.ts` next to source (unit); `packages/*/test/**` (integration); `e2e/**` (Playwright). +- Test files: `*.test.ts` next to source (unit + integration); `packages/dashboard/e2e/**` (Playwright). - One behavior per test; name as `describe('feature') > it('does X when Y')`. - Arrange–Act–Assert; no logic in tests; shared fixtures in `test/fixtures/`. diff --git a/package.json b/package.json index ed4aef8..e8301e9 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "sentinel", - "version": "0.0.0", + "version": "0.1.0", "private": true, "type": "module", "description": "A self-hostable verifying LLM gateway — routes, semantically caches, and verifies every LLM call.", diff --git a/packages/dashboard/package.json b/packages/dashboard/package.json index e9330c2..4f5509a 100644 --- a/packages/dashboard/package.json +++ b/packages/dashboard/package.json @@ -1,6 +1,6 @@ { "name": "@sentinel/dashboard", - "version": "0.0.0", + "version": "0.1.0", "private": true, "type": "module", "description": "Sentinel dashboard — a read-only React UI over the gateway's admin trace API.", diff --git a/packages/gateway/package.json b/packages/gateway/package.json index 1c2759c..d4427af 100644 --- a/packages/gateway/package.json +++ b/packages/gateway/package.json @@ -1,9 +1,9 @@ { "name": "@sentinel/gateway", - "version": "0.0.0", + "version": "0.1.0", "private": true, "type": "module", - "description": "Sentinel gateway — the verifying LLM proxy core. (Scaffold; built out from ROADMAP Phase 1.)", + "description": "Sentinel gateway — the verifying LLM proxy core: providers, router, cache, verify, telemetry.", "dependencies": { "@opentelemetry/api": "^1.9.1", "@opentelemetry/core": "^2.8.0", diff --git a/tsconfig.build.json b/tsconfig.build.json index 505b54e..6eccd36 100644 --- a/tsconfig.build.json +++ b/tsconfig.build.json @@ -5,6 +5,6 @@ "rootDir": ".", "outDir": "dist" }, - "include": ["packages/*/src/**/*.ts"], + "include": ["packages/gateway/src/**/*.ts"], "exclude": ["**/*.test.ts"] }