Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ jobs:
cache: pnpm
- run: pnpm install --frozen-lockfile
- run: pnpm verify
- run: pnpm build
- run: pnpm audit --prod --audit-level=high

e2e:
Expand Down
12 changes: 11 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,16 @@ All notable changes to Sentinel are documented here. The format follows
[Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to
[Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [Unreleased]

### Fixed

- The gateway production build (`pnpm build`) no longer pulls dashboard sources into the Node build; CI now runs `pnpm build` on every PR so it can't silently break again.

### Changed

- Documentation-accuracy pass: corrected stack/roadmap references (Node load harness, not k6; single-node in-memory cache + SQLite storage; dashboard is shipped; providers are OpenAI-compatible with native Anthropic planned), reconciled `package.json` versions to `0.1.0`, and added a **Known limitations** section to the README.

## [0.1.0] — 2026-06-28

First public release: a self-hostable, OpenAI-compatible **verifying LLM gateway** that
Expand Down Expand Up @@ -44,6 +54,6 @@ dashboard. Adopting it is a one-line base-URL change.
- Per-client rate limiting (`CLIENT_RPM`) keyed by hashed API key — a single noisy key gets 429s without affecting other clients.
- Upstream fetches refuse redirects (SSRF hardening); authorization headers redacted from logs.
- CI runs typecheck + lint + the full coverage gate plus a production dependency audit. Security posture tracked in `SECURITY_REVIEW_LOG.md`.
- Reproducible load harness (`pnpm load`) producing the headline benchmarks in `load/RESULTS.md`: 50% cache cost-reduction on a 50%-repeat workload, zero unhandled 429s, 100% PII guardrail catch-rate, and ~14 ms p99 added overhead.
- Reproducible load harness (`pnpm load`) producing the headline benchmarks in `load/RESULTS.md`: 50% cache cost-reduction on a 50%-repeat workload, zero unhandled 429s, 100% PII guardrail catch-rate, and ~14 ms p99 added overhead (measured in-process against **mock upstreams**; the catch-rate is the deterministic-guardrail rate — see `load/RESULTS.md` for full method and caveats).

[0.1.0]: https://github.com/MANVENDRA-github/sentinel/releases/tag/v0.1.0
14 changes: 7 additions & 7 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,15 +13,15 @@ The product source of truth is `PRP_SPEC.md`. The build order is `ROADMAP.md`. B
- Package manager / monorepo: **pnpm** workspaces.
- Gateway server: **Fastify** (streaming, hooks, `.inject()` testing).
- Schemas & validation: **Zod** at every trust boundary (config, request/response, guardrails).
- Providers: OpenAI-compatible interface (`openai` SDK) with adapters for **OpenAI, Anthropic, Google Gemini (AI Studio), Ollama**.
- Providers: a unified, hand-rolled **OpenAI-compatible** HTTP adapter (`fetch`) — OpenAI, Groq, Gemini's OpenAI endpoint, Ollama, and other OpenAI-API providers. A native **Anthropic** (Messages API) adapter is on the near-term roadmap.
- Local models (keyless): **Ollama** — judge `qwen2.5:7b`, embeddings `nomic-embed-text`.
- Cache & rate-limit state: **Redis** (`ioredis`); in-memory fallback for dev/tests.
- Storage (traces, eval results): **better-sqlite3** (dev) / **Postgres** (prod).
- Cache & rate-limit state: **in-memory** (single-node v1). A **Redis** (`ioredis`) backend is a planned swap behind the cache/throttle interfaces.
- Storage (traces, eval results): **better-sqlite3** / in-memory (single-node v1). A **Postgres** backend is a planned swap behind the `TraceStore` interface.
- Telemetry: **OpenTelemetry** (`@opentelemetry/sdk-node`). Logging: **pino**.
- Tests: **Vitest** (unit/integration), **Playwright** (E2E + Playwright MCP for the dashboard).
- Quality: **ESLint** (flat config, `typescript-eslint` strict) + **Prettier**.
- Infra: **Docker** + docker-compose (gateway + redis + ollama). CI: **GitHub Actions**.
- Dashboard (later phase): **React + Vite + TS**.
- Infra: **Docker** + docker-compose for local dependencies (redis + ollama). CI: **GitHub Actions**.
- Dashboard: **React + Vite + TS**.

**Commands** (the only canonical ones — use these, don't invent):
- Install: `pnpm install`
Expand All @@ -37,8 +37,8 @@ The product source of truth is `PRP_SPEC.md`. The build order is `ROADMAP.md`. B
**Structure** (target monorepo — build it per `ROADMAP.md`, don't scaffold ahead of the current phase):
- `packages/gateway/` — the Fastify proxy: providers, router, cache, verify (guardrails + judge), telemetry. The core.
- `packages/sdk/` — thin client/config helpers (later phase).
- `packages/dashboard/` — React + Vite UI (later phase).
- Tests: `*.test.ts` next to source (unit); `packages/*/test/` (integration); `e2e/` (Playwright).
- `packages/dashboard/` — React + Vite UI (shipped).
- Tests: `*.test.ts` next to source (unit + integration via Fastify `.inject()`); `packages/dashboard/e2e/` (Playwright).
- Root docs: `PRP_SPEC.md`, `ROADMAP.md`, `PLAN_TEMPLATE.md`, `TEST_CONTRACT.md`, `SECURITY_REVIEW_LOG.md`.

## Coding style (strict — these are not suggestions)
Expand Down
15 changes: 13 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@

A self-hostable **verifying LLM gateway** — a drop-in, OpenAI-compatible proxy that **routes** (cheapest capable model + fallback), **semantically caches**, and **verifies** (deterministic guardrails inline + a local Ollama judge) every LLM call, with full OpenTelemetry tracing. Unlike after-the-fact observability tools, it can flag or block a bad response _before it returns_.

> **v0.1.0 — feature-complete and hardened.** All seven build phases are done: drop-in proxy, OpenTelemetry tracing, semantic cache, routing / fallback / rate-limit survival, in-path verification (guardrails + a local judge), a React dashboard, and security hardening with a reproducible load harness. Product spec in [`PRP_SPEC.md`](./PRP_SPEC.md), phased build in [`ROADMAP.md`](./ROADMAP.md), contributor/agent guidance in [`CLAUDE.md`](./CLAUDE.md).
> **v0.1.0 — first release.** All build phases (0–7) are done: drop-in proxy, OpenTelemetry tracing, semantic cache, routing / fallback / rate-limit survival, in-path verification (guardrails + a local judge), a React dashboard, and security hardening with a reproducible load harness. It runs **single-node** (in-memory cache + SQLite) — see [Known limitations](#known-limitations). Product spec in [`PRP_SPEC.md`](./PRP_SPEC.md), phased build in [`ROADMAP.md`](./ROADMAP.md), contributor/agent guidance in [`CLAUDE.md`](./CLAUDE.md).

## What it does

Expand Down Expand Up @@ -89,7 +89,7 @@ A provider may also set an `rpm` (requests-per-minute) ceiling; the optional `ro

### Bring your own Ollama

Sentinel never hardcodes a machine or a model. Set `OLLAMA_BASE_URL` in `.env` to point at _your_ Ollama (default `http://localhost:11434/v1`), list the models you've pulled in the `models` map, and you're set. The local **judge** that verifies outputs in a later phase works the same way — it reads `JUDGE_MODEL` from your environment, so every clone uses its own local model, with no API key and no quota.
Sentinel never hardcodes a machine or a model. Set `OLLAMA_BASE_URL` in `.env` to point at _your_ Ollama (default `http://localhost:11434/v1`), list the models you've pulled in the `models` map, and you're set. The local **judge** that verifies outputs works the same way — it reads `JUDGE_MODEL` from your environment, so every clone uses its own local model, with no API key and no quota.

## Observability

Expand Down Expand Up @@ -155,6 +155,17 @@ Real numbers from the bundled load harness (`pnpm load` spins up a mock upstream

Framing, stated plainly: overhead is Sentinel's _own_ per-request cost, not a model's latency; cost-reduction scales with your traffic's repeat rate; and the catch-rate is the deterministic guardrail rate — the async LLM judge needs a real model and is covered by the unit tests. Reproduce it all with `pnpm load`.

## Known limitations

Sentinel v0.1.0 is a **single-node, self-hosted** gateway. Honest boundaries today:

- **In-memory cache & rate-limit state.** The semantic cache and token buckets live in-process — not shared across instances, and cleared on restart. A Redis-backed swap is planned behind the existing interfaces.
- **SQLite (or in-memory) trace storage.** Ideal for single-node; a Postgres backend for multi-instance is planned.
- **The async judge needs a local Ollama** with `JUDGE_MODEL` pulled. Without it, judging degrades to `unscored` (never a false pass). The bundled benchmarks run against **mock upstreams**, so the headline catch-rate is the _deterministic guardrail_ rate; the LLM judge is covered by unit tests.
- **Inline guardrails apply to non-streaming responses.** Streamed responses are judged from their buffered output _after_ completion (inline blocking of a live stream is on the roadmap).
- **Providers are OpenAI-compatible.** OpenAI, Groq, Gemini's OpenAI endpoint, Ollama, and other OpenAI-API providers work today; a native **Anthropic** (Messages API) adapter is planned.
- **Cost reduction is measured by request volume** (cache hits × avoided upstream calls) on a repeat-heavy workload; per-request dollar accounting is planned.

## Development

```bash
Expand Down
6 changes: 3 additions & 3 deletions ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,9 +52,9 @@ A modular, sequential build. **Work one phase at a time.** Each phase is sized t

## Phase 7 — Hardening, load test & launch
**Goal:** prove the numbers; ship it.
**Context:** `e2e/`, `load/` (k6), `README.md`, docs.
**Deliverables:** k6 load test; measured cost-reduction %, p99 overhead, 429-elimination, quality catch-rate (fills `PRP_SPEC.md` §5); full security pass (`SECURITY_REVIEW_LOG.md`); README + quickstart; OSS release.
**Exit:** headline metrics captured and documented; security checklist complete; v1.0 tagged.
**Context:** `packages/dashboard/e2e/`, `load/` (Node harness, `load/run.ts`), `README.md`, docs.
**Deliverables:** Node load harness (`load/run.ts`); measured cost-reduction %, p99 overhead, 429-elimination, quality catch-rate (fills `PRP_SPEC.md` §5); full security pass (`SECURITY_REVIEW_LOG.md`); README + quickstart; OSS release.
**Exit:** headline metrics captured and documented; security checklist complete; v0.1.0 tagged.

## Phase 8 — (Advanced, optional) Fine-tuned judge
**Goal:** a sharper, cheaper judge — and the model-internals milestone.
Expand Down
4 changes: 2 additions & 2 deletions TEST_CONTRACT.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ A PR whose diff adds or changes behavior without a corresponding test that would
| Integration | **Vitest** + Fastify **`.inject()`** | The HTTP surface with providers / Redis / Ollama **mocked or faked**. Request → route → cache → verify → response. |
| Contract | **Vitest** + recorded fixtures | Provider adapters against **recorded** real responses (record once, replay in CI). Catches provider-shape drift. |
| E2E | **Playwright** (+ **Playwright MCP**) | Dashboard flows against a seeded gateway. Agent-driven exploratory passes via Playwright MCP. |
| Load | **k6** (Phase 7) | Throughput, p99 overhead, 429-elimination, cost-reduction numbers. |
| Load | **Node harness** (`load/run.ts`) | Throughput, p99 overhead, 429-elimination, cost-reduction numbers. |

## Determinism rules (CI must be hermetic)

Expand Down Expand Up @@ -49,6 +49,6 @@ A PR whose diff adds or changes behavior without a corresponding test that would

## Conventions

- Test files: `*.test.ts` next to source (unit); `packages/*/test/**` (integration); `e2e/**` (Playwright).
- Test files: `*.test.ts` next to source (unit + integration); `packages/dashboard/e2e/**` (Playwright).
- One behavior per test; name as `describe('feature') > it('does X when Y')`.
- Arrange–Act–Assert; no logic in tests; shared fixtures in `test/fixtures/`.
2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "sentinel",
"version": "0.0.0",
"version": "0.1.0",
"private": true,
"type": "module",
"description": "A self-hostable verifying LLM gateway — routes, semantically caches, and verifies every LLM call.",
Expand Down
2 changes: 1 addition & 1 deletion packages/dashboard/package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "@sentinel/dashboard",
"version": "0.0.0",
"version": "0.1.0",
"private": true,
"type": "module",
"description": "Sentinel dashboard — a read-only React UI over the gateway's admin trace API.",
Expand Down
4 changes: 2 additions & 2 deletions packages/gateway/package.json
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
{
"name": "@sentinel/gateway",
"version": "0.0.0",
"version": "0.1.0",
"private": true,
"type": "module",
"description": "Sentinel gateway — the verifying LLM proxy core. (Scaffold; built out from ROADMAP Phase 1.)",
"description": "Sentinel gateway — the verifying LLM proxy core: providers, router, cache, verify, telemetry.",
"dependencies": {
"@opentelemetry/api": "^1.9.1",
"@opentelemetry/core": "^2.8.0",
Expand Down
2 changes: 1 addition & 1 deletion tsconfig.build.json
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,6 @@
"rootDir": ".",
"outDir": "dist"
},
"include": ["packages/*/src/**/*.ts"],
"include": ["packages/gateway/src/**/*.ts"],
"exclude": ["**/*.test.ts"]
}
Loading