From 567c9c59d12321d8809106ff1137beea298c6c7a Mon Sep 17 00:00:00 2001 From: Corey Thompson Date: Tue, 11 Aug 2026 15:46:43 -0500 Subject: [PATCH 01/24] docs: synchronize 0.5.12 operator guidance --- README.md | 11 ++-- dev/README.md | 5 +- dev/architecture.md | 9 +-- .../instrumentation/oom-high-water.md | 4 +- dev/plans/README.md | 19 +++--- docs/getting-started/configuration.md | 11 ++++ docs/getting-started/installation.md | 6 +- docs/guide/admin-api.md | 27 ++++++++ docs/guide/cli.md | 15 +++++ docs/guide/fathom-storage.md | 39 ++++++++++-- docs/guide/guardrails.md | 61 ++++++++++++++++--- docs/index.md | 3 +- docs/operations.md | 33 +++++++++- 13 files changed, 204 insertions(+), 39 deletions(-) diff --git a/README.md b/README.md index 4f81790..b57f7d2 100644 --- a/README.md +++ b/README.md @@ -31,7 +31,9 @@ Airlock sits between your developers and LLM providers, giving you visibility an |---|---| | **Unified access** | Single OpenAI-compatible endpoint for all providers | | **Logging** | Every request/response logged as structured JSONL | -| **PII stripping** | Microsoft Presidio detects and redacts configured entity types (default: credit cards, SSNs, emails, phone numbers); each redaction is recorded in the mutation ledger | +| **PII stripping** | Microsoft Presidio redacts configured entity types. The shipped pattern recognizers (cards, SSNs, emails, phones, bank numbers, IBANs) run without spaCy NER; configured semantic entities such as `PERSON` use spaCy. | +| **PII-safe tool calls** | Opaque, bounded reverse maps can rehydrate only non-streaming tool-call arguments; per-tool egress policy starts in observe mode and can be promoted to enforce. | +| **Operational storage** | Optional FathomDB search/analysis store with authenticated per-client erasure; JSONL retention remains a separate obligation. | | **Keyword blocking** | Custom blocklist prevents restricted project names or terms from leaking | | **Budget control** | Per-provider daily spend caps — near-limit warning, proactive reroute away from a provider approaching its cap, hard block at the limit. Per-tenant keys with per-key budgets are planned, not yet shipped | | **Multi-tool support** | Works with Cursor, Claude Code, GitHub Copilot, and any OpenAI-compatible client | @@ -46,7 +48,7 @@ Airlock sits between your developers and LLM providers, giving you visibility an ```bash pip install airlock-llm -pip install "https://github.com/explosion/spacy-models/releases/download/en_core_web_lg-3.8.0/en_core_web_lg-3.8.0-py3-none-any.whl" # required for PII redaction +pip install "https://github.com/explosion/spacy-models/releases/download/en_core_web_lg-3.8.0/en_core_web_lg-3.8.0-py3-none-any.whl" # only when configuring spaCy/NER entities, e.g. PERSON airlock init ``` @@ -60,8 +62,9 @@ git clone https://github.com/coreyt/airlock && cd airlock ./scripts/setup.sh ``` -This installs Airlock and its dependencies, downloads the spaCy model for PII -redaction, and runs `airlock init`. Pass `--pip` to use pip instead of uv. +This installs Airlock and its dependencies, prepares the optional spaCy model for +semantic PII detection, and runs `airlock init`. The shipped deterministic PII +recognizers do not require that model. Pass `--pip` to use pip instead of uv. ### Developer setup diff --git a/dev/README.md b/dev/README.md index e8d5db6..94a0ed9 100644 --- a/dev/README.md +++ b/dev/README.md @@ -14,8 +14,9 @@ configuration, and operations. | Runtime boundaries and design rationale | [Architecture](architecture.md) | | Approved semantic input-injection design | [Prompt-injection classifier design](notes/design-prompt-injection-classifier.md) | | Accepted refactoring direction | [Refactoring analysis](notes/refactoring-analysis-2026-07-31.md) | -| Current delivery work | [0.5.10 plan](plans/0.5.10-plan.md), [status board](plans/runs/STATUS-0.5.10.md), and [cold-start note](plans/prompts/0.5.10-COLD-START.md) | -| Most recent closeout | [0.5.9 status board](plans/runs/STATUS-0.5.9.md) — authoritative record of what shipped | +| Latest published release | [0.5.12 plan](plans/0.5.12-plan.md), [PII egress canary](plans/runs/0.5.12-pii-egress-canary-2026-08-11.md), and [CHANGELOG](../CHANGELOG.md) | +| Next engineering backlog | [0.5.14 TODO](plans/0.5.14-todo.md) | +| OOM investigation instrumentation | [High-water runbook](debugging/instrumentation/oom-high-water.md) | | How to operate the delivery harness | [Plans guide](plans/README.md) and [harness runbook](agent-harness-runbook.md) | | Shipped behavior | [Public docs](../docs/index.md), [CHANGELOG](../CHANGELOG.md), and source/tests | diff --git a/dev/architecture.md b/dev/architecture.md index 828d1f4..ca04f58 100644 --- a/dev/architecture.md +++ b/dev/architecture.md @@ -642,10 +642,9 @@ User-facing detail: [`docs/guide/mcp-servers.md`]. Constraint reference for agen └─────────────────────────────────────────┘ ``` -> **Liveness probes use `GET /health/liveliness`, never `GET /health`.** `/health` -> fires live completions to every model when `background_health_checks` is off; -> `/health/liveliness` is unprotected and makes no model calls (repo hard -> constraint). +> **Liveness probes use `GET /health/liveliness`, never `GET /health`.** The +> liveness route makes no model calls. Use `/readyz` for traffic readiness and +> `/healthz` for an aggregate status view. ### 5.2 Local Development Deployment @@ -654,6 +653,8 @@ User-facing detail: [`docs/guide/mcp-servers.md`]. Constraint reference for agen ```bash python -m venv .venv && source .venv/bin/activate pip install -e . +# Required only when configured PII entities need named-entity recognition, +# such as PERSON; the shipped pattern recognizers use Presidio's no-NLP engine. python -m spacy download en_core_web_lg airlock # or: python -m airlock.proxy ``` diff --git a/dev/debugging/instrumentation/oom-high-water.md b/dev/debugging/instrumentation/oom-high-water.md index 2429f19..3c0acef 100644 --- a/dev/debugging/instrumentation/oom-high-water.md +++ b/dev/debugging/instrumentation/oom-high-water.md @@ -13,8 +13,8 @@ request or response bodies, headers, model names, exception text, or client metadata. Keep artifacts outside source control and use an isolated service for a real-provider replay. -Never probe `GET /health`: it can issue model calls. Use -`GET /health/liveliness` for a safe liveness check. +Never use `GET /health` as a liveness probe. Use `GET /health/liveliness` for +the safe liveness check. ## Enable the recorder diff --git a/dev/plans/README.md b/dev/plans/README.md index cead413..ebf0e2d 100644 --- a/dev/plans/README.md +++ b/dev/plans/README.md @@ -6,16 +6,15 @@ memory; this tree is the source of truth. ## Current work -The active release is [0.5.10](0.5.10-plan.md) — a deliberate catch-up and -clean-up milestone — with its live board at -[`runs/STATUS-0.5.10.md`](runs/STATUS-0.5.10.md). Start a cold session from -[`prompts/0.5.10-COLD-START.md`](prompts/0.5.10-COLD-START.md). - -0.5.9 is **closed out** (`milestone/0.5.9-internal-closeout` at `b2752d7`). Its -board [`runs/STATUS-0.5.9.md`](runs/STATUS-0.5.9.md) remains the authoritative -record of what shipped; its steward handoff is **superseded** and must not be -followed as instructions. Completed trains are historical evidence catalogued in -the [release archive](archive/README.md), not active roadmap commitments. +The latest published release is [0.5.12](0.5.12-plan.md), with retained +delivery evidence in the [PII egress canary](runs/0.5.12-pii-egress-canary-2026-08-11.md) +and [measurement protocol](runs/0.5.12-pii-egress-measurement-protocol.md). Its +PII egress gate is shipped in `observe`; promotion remains a human decision +based on that record. + +The next recorded engineering backlog is [0.5.14 TODO](0.5.14-todo.md). It is +a TODO list, not a live release board. Completed trains are historical evidence catalogued in the +[release archive](archive/README.md), not active roadmap commitments. ## Layout diff --git a/docs/getting-started/configuration.md b/docs/getting-started/configuration.md index b21b97b..cdd407f 100644 --- a/docs/getting-started/configuration.md +++ b/docs/getting-started/configuration.md @@ -240,7 +240,13 @@ See [Batch Processing](../guide/batch.md) for the end-to-end recipe. | `AIRLOCK_BLOCKED_KEYWORDS` | Comma-separated restricted phrases | -- | | `AIRLOCK_PII_ENTITIES` | Presidio entity types to redact | `CREDIT_CARD,US_SSN,EMAIL_ADDRESS,PHONE_NUMBER` | | `AIRLOCK_PII_FAIL_MODE` | PII redaction unavailable posture: `open` or `closed` | `open` | +| `AIRLOCK_PII_HYDRATION` | Restore placeholders in non-streaming tool-call arguments: `tools` or `off` | `tools` | +| `AIRLOCK_PII_MAP_MAX_ENTRIES` | Maximum live opaque PII reverse maps per process | `1024` | +| `AIRLOCK_PII_MAP_TTL_SECONDS` | Seconds before an unconsumed PII reverse map expires | `300` | | `AIRLOCK_PII_EGRESS_MODE` | Rehydration egress policy: `observe`, `shadow`, `enforce` | `observe` | +| `AIRLOCK_PII_EGRESS_TOOL_BANDS` | JSON object mapping tool names to `round_trip` or `exfil` | `{}` | +| `AIRLOCK_PII_EGRESS_ALLOWLIST` | JSON allowlist of `{tool,path,entity_type}` egress tuples | `[]` | +| `AIRLOCK_PII_EGRESS_BLOCKLIST` | JSON blocklist of `{tool,path,entity_type}` egress tuples | `[]` | | `AIRLOCK_ENFORCE_MODE` | Guardrail mode: `observe`, `shadow`, or `enforce` | `observe` | | `AIRLOCK_CLIENT` | Client identity label propagated as the `X-Airlock-Client` header and recorded on each request for per-tool attribution | -- | | `AIRLOCK_ADVISOR_MODEL` | Override model for the advisor | -- | @@ -271,6 +277,11 @@ See [Batch Processing](../guide/batch.md) for the end-to-end recipe. | `AIRLOCK_JWT_SECRET_PREV` | Previous JWT secret, accepted for verification during a rolling secret rotation | -- | | `AIRLOCK_VALIDATE_REASONING_EFFORT` | Reject known-invalid OpenAI/Azure `reasoning_effort` values with an OpenAI-shaped 400. Set `0` for the short-term `drop_params` rollback. | `1` | | `AIRLOCK_NORMALIZE_REASONING_EFFORT` | Legacy compatibility fallback for validation while `AIRLOCK_VALIDATE_REASONING_EFFORT` is unset; still controls Gemini/Anthropic normalization. | `1` | +| `AIRLOCK_OOM_DIAGNOSTICS` | Enable bounded aggregate memory/cgroup diagnostic snapshots; use only for an investigation | `0` | +| `AIRLOCK_OOM_DIAGNOSTICS_DIR` | Diagnostic artifact directory when OOM diagnostics are enabled | `/tmp/airlock-oom-diagnostics` | + +For the PII boundary, egress modes, and the spaCy requirement for semantic +entities, see [Guardrails → PII redaction, hydration, and egress](../guide/guardrails.md#pii-redaction-hydration-and-egress). For the opt-in memory recorder, see [Operations → Memory guardrails and OOM diagnostics](../operations.md#memory-guardrails-and-oom-diagnostics). ### `reasoning_effort` validation and normalization diff --git a/docs/getting-started/installation.md b/docs/getting-started/installation.md index 10c4687..b252dd7 100644 --- a/docs/getting-started/installation.md +++ b/docs/getting-started/installation.md @@ -4,7 +4,7 @@ ```bash pip install airlock-llm -pip install "https://github.com/explosion/spacy-models/releases/download/en_core_web_lg-3.8.0/en_core_web_lg-3.8.0-py3-none-any.whl" # required for PII redaction +pip install "https://github.com/explosion/spacy-models/releases/download/en_core_web_lg-3.8.0/en_core_web_lg-3.8.0-py3-none-any.whl" # only for spaCy/NER PII entities, e.g. PERSON airlock init ``` @@ -17,7 +17,9 @@ git clone https://github.com/coreyt/airlock && cd airlock ./scripts/setup.sh ``` -This installs Airlock and its dependencies, downloads the spaCy model for PII redaction, and runs `airlock init`. Pass `--pip` to use pip instead of uv. +This installs Airlock and its dependencies, prepares the optional spaCy model +for semantic PII detection, and runs `airlock init`. The shipped deterministic +PII recognizers do not need that model. Pass `--pip` to use pip instead of uv. ## Developer setup diff --git a/docs/guide/admin-api.md b/docs/guide/admin-api.md index a2d3a2a..7de0e42 100644 --- a/docs/guide/admin-api.md +++ b/docs/guide/admin-api.md @@ -82,6 +82,7 @@ master key (and any loopback operator) satisfies all of them. | `admin:reset_circuit` | Reset a model circuit | | `admin:clear_backoff` | Clear a client threat backoff | | `admin:force_quarantine` | Manually quarantine a provider (loopback-only) | +| `admin:erase_client` | Erase a client's FathomDB rows (loopback-only) | ## Minting capability tokens @@ -193,6 +194,32 @@ curl -X POST https://airlock.internal/airlock/admin/clients/key:b35cf679/clear-b Scope: `admin:clear_backoff`. +### Erase one client's FathomDB records + +Erasure is deliberately a local control-plane operation. Use the CLI against +the running proxy, repeating the authenticated client id as confirmation: + +```bash +airlock admin erase-client key:90abcdef --confirm key:90abcdef +``` + +The equivalent loopback-only HTTP operation is: + +```bash +curl -X POST http://127.0.0.1:4000/airlock/admin/clients/key%3A90abcdef/erase \ + -H 'Content-Type: application/json' \ + -d '{"confirm":"key:90abcdef"}' +``` + +Scope: `admin:erase_client`. A successful operation emits an `admin_action` +audit record with its erase report. HTTP 409 means erasure was incomplete and +must not be treated as done; retrying the same request is safe. + +This affects the optional FathomDB search/analysis store only. It does **not** +remove the corresponding JSONL logs, whose retention is controlled separately +by `AIRLOCK_MAX_LOG_DAYS`. See [Operations → Per-client erasure](../operations.md#per-client-erasure) +before making a deletion commitment. + ### Manually quarantine a provider Manually arms a provider quarantine (break-glass). diff --git a/docs/guide/cli.md b/docs/guide/cli.md index d9f3777..f5ca4e3 100644 --- a/docs/guide/cli.md +++ b/docs/guide/cli.md @@ -18,6 +18,21 @@ airlock config import backup.zip --dir ~ # extracts into home directory Existing files will be safely backed up before being overwritten during an import. +### `airlock admin` + +Mint scoped administrative/capability tokens and perform the local FathomDB +per-client erasure operation: + +```bash +airlock admin mint-token --sub ops --scope admin:read --ttl 15m +airlock admin erase-client key:90abcdef --confirm key:90abcdef +``` + +`erase-client` calls the running proxy's loopback admin API; it never opens the +database file from the CLI process. It is irreversible for the FathomDB store, +idempotent to retry after an incomplete result, and does not erase JSONL logs. +See [Admin API → Erase one client's FathomDB records](admin-api.md#erase-one-clients-fathomdb-records). + ### `airlock init` Generate `config.yaml`, `.env`, and `logs/` in the current directory. diff --git a/docs/guide/fathom-storage.md b/docs/guide/fathom-storage.md index ffcc8ca..9245ba5 100644 --- a/docs/guide/fathom-storage.md +++ b/docs/guide/fathom-storage.md @@ -1,10 +1,41 @@ # Fathom Storage Airlock can optionally write a compact operational record for each logical -request into FathomDB. This page describes the intended data model, what is -stored as first-class fields, what belongs in the nested payload JSON, what is -computed at write time, and what is expected to be derived later from -projections or summary tables. +request into FathomDB. This page distinguishes the shipped operator contract +from the longer-term data-model direction. + +## Shipped behavior (0.5.12) + +Enable the FathomDB engine with: + +```bash +AIRLOCK_ENABLE_FATHOMDB=1 +``` + +To record logical requests into that engine, also enable the recorder-fed +logger: + +```bash +AIRLOCK_ENABLE_FATHOM_LOGGER=1 +``` + +Airlock opens `airlock-fathom.db` lazily under `AIRLOCK_STATE_DIR`. The engine +is PID-bound and thread-safe inside one proxy process; do not run multiple +live processes against the same database path. The former FathomDB 0.3.x +`logs/airlock.db` path is not migrated or opened by Airlock. + +The optional store supports bounded operational search/analysis and audited +per-client erasure. Erasure is performed by the running proxy, is loopback +only, and covers FathomDB only — it does not delete JSONL records. See +[Operations → FathomDB](../operations.md#fathomdb) and [Admin API](admin-api.md#erase-one-clients-fathomdb-records). + +## Data-model direction + +The sections below describe the intended data model: what is stored as +first-class fields, what belongs in nested payload JSON, what is computed at +write time, and what is expected to be derived later from projections or +summary tables. They are not a promise that every listed field or control is +enabled in every deployment. ## Overview diff --git a/docs/guide/guardrails.md b/docs/guide/guardrails.md index 6779a1a..00875ea 100644 --- a/docs/guide/guardrails.md +++ b/docs/guide/guardrails.md @@ -4,10 +4,13 @@ Airlock applies a chain of guardrails to every request. Guardrails can observe ( ## Guardrail chain -Requests pass through 12 stages in order: +Requests pass through the following guardrail stages in order. The OOM diagnostic +recorder is registered but inert unless explicitly enabled; it does not affect +request policy. | Stage | Phase | Purpose | |-------|-------|---------| +| OOM Diagnostic | pre_call, post_call | Opt-in aggregate process/cgroup snapshots for a memory investigation | | PII Guard | pre_call | Redact credit cards, SSNs, emails, phone numbers (Presidio) | | Keyword Guard | pre_call | Block requests containing restricted keywords | | Enhanced Interceptor | pre_call | Inject prompt/parameter defaults for `enhanced/*` model aliases | @@ -95,19 +98,63 @@ only, and everything is still scanned and logged. > forgeable `X-Airlock-Client` attribution header carries **zero** authorization > weight for skips — a stolen token cannot be replayed by forging it. -## PII redaction +## PII redaction, hydration, and egress -Uses Microsoft Presidio with the `en_core_web_lg` spaCy model. +Airlock redacts configured PII with Microsoft Presidio before the request +leaves the proxy. The built-in pattern/validation recognizers — `CREDIT_CARD`, +`US_SSN`, `EMAIL_ADDRESS`, `PHONE_NUMBER`, `US_BANK_NUMBER`, and `IBAN_CODE` — +use Presidio's no-NLP engine and do **not** load spaCy/Thinc. Adding a semantic +entity such as `PERSON` selects Presidio's normal NLP path, which requires the +configured spaCy model. -Default entities: `CREDIT_CARD`, `US_SSN`, `EMAIL_ADDRESS`, `PHONE_NUMBER`. - -Customize with `AIRLOCK_PII_ENTITIES`: +The code default contains the first four types; the shipped `.env` template +enables all six. Customize the set with `AIRLOCK_PII_ENTITIES`: ```bash AIRLOCK_PII_ENTITIES=CREDIT_CARD,US_SSN,EMAIL_ADDRESS ``` -PII is redacted with placeholders (`[EMAIL_ADDRESS_1]`) before the request leaves the network. The PII Hydrator restores original values in the response so the client receives correct data. +PII is replaced with numbered placeholders such as ``. The +reverse map is kept only in a bounded, process-local store; request metadata +and telemetry carry at most an opaque, short-lived handle, never the original +values. `AIRLOCK_PII_MAP_MAX_ENTRIES` (default `1024`) and +`AIRLOCK_PII_MAP_TTL_SECONDS` (default `300`) bound disconnected or failed +requests. + +`AIRLOCK_PII_HYDRATION=tools` (the default) restores placeholders only in +**non-streaming tool-call arguments**. It does not rewrite normal response +prose, and streaming tool-call deltas remain redacted because a placeholder can +span chunks. Set `AIRLOCK_PII_HYDRATION=off` to leave placeholders in tool +arguments. + +If redaction is unavailable, `AIRLOCK_PII_FAIL_MODE=open` (default) preserves +the request and records a value-free unavailable marker. Set it to `closed` to +block instead. Hydration always fails closed: an unavailable reverse map leaves +placeholders intact. + +### Tool-call egress policy + +Before a placeholder is restored into a tool argument, Airlock makes a +value-free decision using the tool name, JSON-pointer-like argument path, and +PII entity class. `AIRLOCK_PII_EGRESS_MODE=observe` is the shipped deployment +posture: it records the decision but permits hydration. `shadow` records what +would be allowed while keeping denied placeholders redacted; `enforce` also +keeps denied placeholders redacted as the final policy posture. + +Configure the policy with JSON environment values: + +```bash +# classify a tool as a round-trip destination or an external egress destination +AIRLOCK_PII_EGRESS_TOOL_BANDS='{"gmail_search":"round_trip","send_mail":"exfil"}' +# permit a specific class only at a specific egress destination/path +AIRLOCK_PII_EGRESS_ALLOWLIST='[{"tool":"send_mail","path":"/to","entity_type":"EMAIL_ADDRESS"}]' +# always deny a matching tuple (wildcards are supported) +AIRLOCK_PII_EGRESS_BLOCKLIST='[{"tool":"*","path":"*","entity_type":"US_SSN"}]' +``` + +Unknown tools are denied in `shadow` and `enforce`. Credit-card, SSN, US bank +number, and IBAN classes are denied by default; a blocklist always wins. Egress +telemetry contains policy labels only — never PII values. ## Keyword blocking diff --git a/docs/index.md b/docs/index.md index 6abd6a3..2ee17c6 100644 --- a/docs/index.md +++ b/docs/index.md @@ -22,11 +22,12 @@ observability, and operational controls. | Capability | Start here | | --- | --- | | Provider routing and model aliases | [Routing](guide/routing.md) | -| PII, keyword, semantic, and response controls | [Guardrails](guide/guardrails.md) | +| PII redaction, safe tool hydration/egress, keyword, semantic, and response controls | [Guardrails](guide/guardrails.md) | | Provider protection and admission | [Rate limiting](guide/rate-limiting.md) | | Logs, metrics, tracing, and response transparency | [Observability](guide/observability.md) | | Batch workloads | [Batch processing](guide/batch.md) | | MCP tools | [MCP servers](guide/mcp-servers.md) | +| Optional FathomDB search/analysis and per-client erasure | [Fathom storage](guide/fathom-storage.md) and [Operations](operations.md#fathomdb) | ## Continue reading diff --git a/docs/operations.md b/docs/operations.md index 7932594..6f83e8f 100644 --- a/docs/operations.md +++ b/docs/operations.md @@ -37,6 +37,7 @@ The deployment runs as non-root (UID 1000), sets resource limits (250m-1 CPU, 51 python -m venv /opt/airlock/.venv source /opt/airlock/.venv/bin/activate pip install -e ".[metrics,tracing]" +# Required only if AIRLOCK_PII_ENTITIES includes a spaCy/NER type such as PERSON. pip install "https://github.com/explosion/spacy-models/releases/download/en_core_web_lg-3.8.0/en_core_web_lg-3.8.0-py3-none-any.whl" # Copy config @@ -159,7 +160,7 @@ Probe endpoints are **unauthenticated** and respond as `application/health+json` per the [IETF health check draft](https://datatracker.ietf.org/doc/html/draft-inadarei-api-health-check-06): ```json -{"status": "pass", "serviceId": "airlock", "version": "0.5.10", +{"status": "pass", "serviceId": "airlock", "version": "0.5.12", "checks": {"models:available": [{"status": "pass", "observedValue": 79}]}} ``` @@ -201,6 +202,32 @@ GET /healthz # expect 200 and {"status": "pass"} The `/health/circuits` endpoint is installed by the `model_override_headers` callback (see [Callbacks](#callbacks)). +## Memory guardrails and OOM diagnostics + +The bundled systemd unit sets `MemoryHigh=3G` and `MemoryMax=4G`: the first +asks the cgroup to reclaim/throttle before a hard cap is reached. Treat these +as a starting point, not universal production values; size them for the +container or VM and its concurrency budget. A limit makes a memory failure +observable and restartable, but it does not cure a leak or an oversized +workload. + +For an isolated reproduction, enable the bounded diagnostic recorder: + +```bash +AIRLOCK_OOM_DIAGNOSTICS=1 +AIRLOCK_OOM_DIAGNOSTICS_DIR=/var/tmp/airlock-oom +AIRLOCK_OOM_DIAGNOSTICS_EVERY=25 +AIRLOCK_OOM_DIAGNOSTICS_MAX_RECORDS=6000 +``` + +It writes aggregate process, allocator, file-descriptor, thread, cgroup, and +memory-pressure counters. It does not write request/response bodies, headers, +model names, exception text, or client metadata. The directory defaults to +`/tmp/airlock-oom-diagnostics`; output files are owner-only. Secure or +pre-create a configured directory with the permissions your deployment needs. +Disable the recorder after the investigation. The repository's engineering OOM +instrumentation runbook contains the signal and high-water procedure. + ## Startup Modes Airlock now keeps expensive startup work opt-in: @@ -314,7 +341,7 @@ curl -X POST http://127.0.0.1:4000/airlock/admin/clients/key%3A90abcdef/erase \ records exist in the JSONL logs, which this operation does not touch; JSONL retention is governed separately by `AIRLOCK_MAX_LOG_DAYS`. A user-facing deletion obligation requires both, and the JSONL half is not automated in - 0.5.11. + 0.5.12. Operational constraint: @@ -550,7 +577,7 @@ tool-call hydration returns a private response copy, keeping the telemetry objec redacted. Rehydration egress policy starts in `observe`: it emits value-free tool/path/entity-class decisions but does not alter a response. In `shadow` and `enforce`, unknown/exfil-capable paths keep placeholders unless explicitly -authorized. See `dev/notes/design-pii-rehydration-primary.md`. +authorized. See [Guardrails → PII redaction, hydration, and egress](guide/guardrails.md#pii-redaction-hydration-and-egress). ### Keyword Blocking From 6d3f6b767ea23bf0a1e0636bc1d8318893123a26 Mon Sep 17 00:00:00 2001 From: Corey Thompson Date: Tue, 11 Aug 2026 15:49:00 -0500 Subject: [PATCH 02/24] docs: add Fathom backend roadmap --- config.yaml | 2 +- dev/architecture.md | 9 ++--- .../instrumentation/oom-high-water.md | 6 ++-- dev/plans/0.5.14-todo.md | 33 +++++++++++++++++++ 4 files changed, 42 insertions(+), 8 deletions(-) diff --git a/config.yaml b/config.yaml index 6baa242..b28dc90 100644 --- a/config.yaml +++ b/config.yaml @@ -974,7 +974,7 @@ general_settings: master_key: os.environ/AIRLOCK_MASTER_KEY # Keep provider health checks lazy by default. - # Airlock's own probes use /health/liveliness, so we do not need real + # Airlock's own probes use /livez, so we do not need real # provider calls during startup just to determine whether the proxy is up. # Enable this only when you explicitly want periodic provider-wide checks. background_health_checks: false diff --git a/dev/architecture.md b/dev/architecture.md index ca04f58..6ea354a 100644 --- a/dev/architecture.md +++ b/dev/architecture.md @@ -637,14 +637,15 @@ User-facing detail: [`docs/guide/mcp-servers.md`]. Constraint reference for agen │ │ - ./logs (writable bind) │ │ │ └────────────────────────────────────┘ │ │ │ -│ Health: GET /health/liveliness (30s) │ +│ Health: GET /livez (30s) │ │ Restart: unless-stopped │ └─────────────────────────────────────────┘ ``` -> **Liveness probes use `GET /health/liveliness`, never `GET /health`.** The -> liveness route makes no model calls. Use `/readyz` for traffic readiness and -> `/healthz` for an aggregate status view. +> **Liveness probes use `GET /livez`, never `GET /health`.** `/livez` is the +> canonical liveness route and makes no model calls. Use `/readyz` for traffic +> readiness and `/healthz` for an aggregate status view. The LiteLLM +> `/health/liveliness` alias remains for compatibility only. ### 5.2 Local Development Deployment diff --git a/dev/debugging/instrumentation/oom-high-water.md b/dev/debugging/instrumentation/oom-high-water.md index 3c0acef..9c9cc96 100644 --- a/dev/debugging/instrumentation/oom-high-water.md +++ b/dev/debugging/instrumentation/oom-high-water.md @@ -13,8 +13,8 @@ request or response bodies, headers, model names, exception text, or client metadata. Keep artifacts outside source control and use an isolated service for a real-provider replay. -Never use `GET /health` as a liveness probe. Use `GET /health/liveliness` for -the safe liveness check. +Never use `GET /health` as a liveness probe. Use `GET /livez` for the canonical +safe liveness check. ## Enable the recorder @@ -115,7 +115,7 @@ Before stopping a diagnostic run, retain: cgroup. 3. Aggregate per-call success/error counts from the real runner; its top-level `complete` verdict alone may hide error rows. -4. A safe `/health/liveliness` result and unit result/restart state. +4. A safe `/livez` result and unit result/restart state. Do not stop a driver merely because it crossed an historical request-count landmark. Stop only at an explicit operator-approved safety boundary or after diff --git a/dev/plans/0.5.14-todo.md b/dev/plans/0.5.14-todo.md index ba706ff..7c91831 100644 --- a/dev/plans/0.5.14-todo.md +++ b/dev/plans/0.5.14-todo.md @@ -32,3 +32,36 @@ navigation, and rendering assertions. - Lifecycle-specific tests still cover `call_from_thread`, worker cancellation, app teardown, JSONL/MCP shutdown, and the OverviewPane stale-refresh path. - No production TUI behaviour is gated on the test-harness setting. + +## FathomDB as an operational backend + +**Why this is scheduled:** 0.5.11/0.5.12 made FathomDB an optional, lazy, +single-owner request-analysis store. It is not yet the selectable backend for +Airlock's operational reads: the TUI and advisor still rely on their existing +JSONL/in-memory paths. The next step should make that relationship explicit +rather than treating FathomDB as only a secondary callback sink. + +### Work + +- Define the supported backend contract: which operational reads (TUI history, + advisor/search, and retention/erasure reporting) may use FathomDB, which + remain JSONL-backed, and which data source is authoritative for each answer. +- Add a selectable FathomDB operational-read backend, preserving the current + JSONL/in-memory behavior when disabled or unavailable. Do not silently make + FathomDB a required dependency or change the JSONL audit/retention contract. +- Establish process ownership, startup/restart, read bounds, and failure + behavior for the backend; separate-process shared writers remain unsupported + unless a design and evidence change that constraint. +- Add end-to-end tests for enabled, disabled, unavailable, and per-client + erasure states, including proof that incomplete Fathom erasure is never + presented as a completed whole-record deletion. + +### Done when + +- An operator can opt into FathomDB as the documented operational-query + backend, while a default deployment remains zero-infrastructure and retains + current JSONL behavior. +- Every backend-backed view reports its source and bounded/degraded outcome + clearly; it does not silently return partial data as complete. +- The FathomDB engine lifecycle and single-owner constraint are covered by + integration tests and documented for operators. From e8516ee397172d8e5480d7c241e4193d2f2f27d2 Mon Sep 17 00:00:00 2001 From: Corey Thompson Date: Tue, 11 Aug 2026 15:50:26 -0500 Subject: [PATCH 03/24] docs: explain local MCP overrides --- docs/getting-started/configuration.md | 62 +++++++++++++++++++++++++++ 1 file changed, 62 insertions(+) diff --git a/docs/getting-started/configuration.md b/docs/getting-started/configuration.md index cdd407f..d2c7128 100644 --- a/docs/getting-started/configuration.md +++ b/docs/getting-started/configuration.md @@ -32,6 +32,68 @@ them, so `config.local.yaml` must list **every** MCP server you want at runtime `${HOME}`/`~` in MCP `command`/`args`, and only launches commands whose basename is on its allowlist. Full rules and examples in [MCP Servers](../guide/mcp-servers.md). +### Supported local override: machine-specific MCP servers + +Airlock's supported use of `config.local.yaml` is the top-level +`mcp_servers` mapping. Use it when a server's executable, virtual environment, +or source checkout is specific to one host. Keep portable models, guardrails, +routing, callbacks, and deployment policy in the reviewed `config.yaml` and +keep all secret values in `.env` or the deployment's secret manager. + +Each local server can set these fields: + +| Field | Use | Requirement | +| --- | --- | --- | +| Server name | Key under `mcp_servers` | Unique name used by tool discovery | +| `transport` | MCP transport | Use `stdio` for a locally spawned process | +| `command` | Executable to start | Absolute path when host-specific; its basename must be allowed | +| `args` | Executable arguments | Use absolute paths; `${HOME}`, `~`, and `$VAR` are literal text | +| `env` | Environment passed to the server | Reference values as `os.environ/NAME`; never put a secret value here | + +```yaml +# config.local.yaml — local working-tree edit; do not commit credential values +mcp_servers: + local_utility: + transport: stdio + command: /home/you/projects/local-utility/.venv/bin/python + args: ["/home/you/projects/local-utility/server.py"] + env: + LOCAL_UTILITY_TOKEN: os.environ/LOCAL_UTILITY_TOKEN + + # `mcp_servers` is replacement, not merge: repeat a bundled server that + # should remain enabled when this file is present. + newscatcher: + transport: stdio + command: python3 + args: ["-m", "airlock.mcp_servers.newscatcher_server"] + env: + NEWS_CATCHER_API_KEY: os.environ/NEWS_CATCHER_API_KEY +``` + +Put the referenced value in `.env`, not in the YAML: + +```bash +# .env +LOCAL_UTILITY_TOKEN=managed-out-of-band +``` + +For a custom launcher whose basename is not one of LiteLLM's built-ins, add +only its basename to `LITELLM_MCP_STDIO_EXTRA_COMMANDS` in `.env`. For example, +`command: /home/you/bin/local-utility` requires +`LITELLM_MCP_STDIO_EXTRA_COMMANDS=local-utility`. + +Restart after editing. `AIRLOCK_MCP_STARTUP_MODE=lazy` keeps the configured +servers but delays tool discovery; `eager` probes them during startup; `off` +removes all MCP servers, including local overrides, for that process. Verify +the live result with the [MCP tool-discovery check](../guide/mcp-servers.md#verifying-after-a-change-or-restart). + +!!! warning "Do not use this file as a general secret or policy overlay" + A LiteLLM include is technically a YAML fragment and a dict-valued top-level + setting can replace the matching main setting. That makes arbitrary local + overrides easy to misread and difficult to review. Airlock documents + `mcp_servers` as the supported local surface; use a reviewed deployment + config selected with `airlock start --config` for other configuration. + ## Reasoning-effort validation Airlock 0.5.8 rejects a known-invalid `reasoning_effort` for OpenAI-family models From 0d0c5c40ed6f1deaa1c90e6225e6d9a7af77db59 Mon Sep 17 00:00:00 2001 From: Corey Thompson Date: Thu, 13 Aug 2026 07:38:10 -0500 Subject: [PATCH 04/24] Release 0.5.14 --- .env.example | 13 + CHANGELOG.md | 46 +++ airlock/__init__.py | 2 +- airlock/admin/http.py | 130 +++++- airlock/advisor/agent.py | 8 + airlock/advisor/tools.py | 172 +++++--- airlock/callbacks/enterprise_logger.py | 5 + airlock/callbacks/metrics.py | 9 + airlock/callbacks/oom_diagnostics.py | 156 +++++-- airlock/callbacks/request_event.py | 11 +- airlock/callbacks/tracing.py | 22 +- airlock/capability.py | 8 +- airlock/cli/templates/config.yaml | 26 ++ airlock/client_identity.py | 7 +- airlock/embedding.py | 114 ++++++ airlock/embedding_boundary.py | 85 ++++ airlock/fast/_state_core.py | 85 +++- airlock/fast/guardian.py | 140 +++++-- airlock/fast/model_alias.py | 26 +- airlock/fast/monitor.py | 11 +- airlock/fast/router.py | 8 +- airlock/guardrails/extract.py | 2 + airlock/guardrails/pii_guard.py | 56 ++- airlock/guardrails/pii_mapping.py | 4 +- airlock/models_catalog.py | 135 +++++- airlock/operational_reads.py | 136 ++++++ airlock/provider_errors.py | 83 ++++ airlock/proxy_bootstrap.py | 2 + airlock/proxy_errors.py | 100 +++++ airlock/telemetry_health.py | 108 +++++ airlock/text_extract.py | 17 + airlock/transparency.py | 13 +- airlock/tui/admin_client.py | 52 +++ airlock/tui/app.py | 6 + airlock/tui/screens/config.py | 2 + airlock/tui/screens/guards.py | 54 +++ airlock/tui/screens/logs.py | 117 ++++-- airlock/tui/screens/overview.py | 168 ++++++++ config.local.yaml | 53 ++- config.yaml | 26 ++ dev/architecture.md | 21 +- dev/plans/0.5.12-plan.md | 4 +- .../0.5.14-openrouter-deepseek-design.md | 316 ++++++++++++++ dev/plans/0.5.14-todo.md | 282 +++++++++++-- dev/plans/0.5.14/00-environment.md | 39 ++ dev/plans/0.5.14/01-dependencies.md | 42 ++ .../0.5.14/02-documentation-cruft-proposal.md | 35 ++ .../03-draft-requirements-and-allocation.md | 68 +++ dev/plans/0.5.14/04-architecture-review.md | 49 +++ dev/plans/0.5.14/05-verification-review.md | 49 +++ dev/plans/0.5.14/06-hitl-release-review.md | 61 +++ .../10-benchmark-safe-logging-status.md | 42 ++ .../110-fathomdb-operational-reads-design.md | 48 +++ .../110-fathomdb-operational-reads-status.md | 48 +++ .../120-documentation-contract-status.md | 22 + dev/plans/0.5.14/130-release-closeout.md | 61 +++ dev/plans/0.5.14/20-fathomdb-chat-status.md | 34 ++ dev/plans/0.5.14/30-embeddings-design.md | 42 ++ dev/plans/0.5.14/30-embeddings-status.md | 78 ++++ .../0.5.14/40-provider-foundation-design.md | 52 +++ .../0.5.14/40-provider-foundation-status.md | 49 +++ dev/plans/0.5.14/50-openrouter-design.md | 40 ++ dev/plans/0.5.14/50-openrouter-status.md | 48 +++ dev/plans/0.5.14/60-deepseek-design.md | 25 ++ dev/plans/0.5.14/60-deepseek-status.md | 45 ++ dev/plans/0.5.14/70-tui-lifecycle-design.md | 15 + dev/plans/0.5.14/70-tui-lifecycle-status.md | 36 ++ .../80-routing-client-diagnostics-design.md | 47 +++ .../80-routing-client-diagnostics-status.md | 37 ++ .../0.5.14/90-qos-exporter-health-design.md | 36 ++ .../0.5.14/90-qos-exporter-health-status.md | 34 ++ dev/plans/0.5.14/README.md | 39 ++ dev/requirements.md | 65 ++- dev/user-needs.md | 26 ++ docs/architecture/overview.md | 5 + docs/getting-started/configuration.md | 101 +++++ docs/guide/admin-api.md | 13 + docs/guide/provider-observability.md | 2 +- docs/operations.md | 7 + docs/reference/api.md | 13 + pyproject.toml | 4 +- tests/test_0_5_2_capability_compat.py | 8 +- tests/test_admin_http.py | 120 ++++++ tests/test_advisor_tools.py | 92 ++++- tests/test_capability.py | 10 + tests/test_client_identity.py | 8 + tests/test_config_consistency.py | 67 +++ tests/test_documentation_contract.py | 36 +- tests/test_embedding_boundary.py | 125 ++++++ tests/test_enterprise_logger.py | 9 +- tests/test_extract.py | 9 + tests/test_fast_guardian.py | 387 ++++++++++++++++++ tests/test_fast_router.py | 44 +- tests/test_fathom_search.py | 41 +- tests/test_measurement_report.py | 22 +- tests/test_models_catalog.py | 176 ++++++++ tests/test_oom_diagnostics.py | 28 +- tests/test_operational_reads.py | 147 +++++++ tests/test_pii_egress.py | 16 +- tests/test_pii_guard.py | 62 ++- tests/test_provider_errors.py | 102 +++++ tests/test_proxy_bootstrap.py | 4 + tests/test_proxy_errors.py | 181 ++++++++ tests/test_request_event.py | 4 +- tests/test_served_attribution.py | 2 + tests/test_tui.py | 12 +- tests/test_tui_harness.py | 63 +++ tests/test_tui_qos_telemetry.py | 89 ++++ tests/test_tui_routing.py | 70 ++++ uv.lock | 13 +- 110 files changed, 5920 insertions(+), 325 deletions(-) create mode 100644 airlock/embedding.py create mode 100644 airlock/embedding_boundary.py create mode 100644 airlock/operational_reads.py create mode 100644 airlock/provider_errors.py create mode 100644 airlock/telemetry_health.py create mode 100644 dev/plans/0.5.14-openrouter-deepseek-design.md create mode 100644 dev/plans/0.5.14/00-environment.md create mode 100644 dev/plans/0.5.14/01-dependencies.md create mode 100644 dev/plans/0.5.14/02-documentation-cruft-proposal.md create mode 100644 dev/plans/0.5.14/03-draft-requirements-and-allocation.md create mode 100644 dev/plans/0.5.14/04-architecture-review.md create mode 100644 dev/plans/0.5.14/05-verification-review.md create mode 100644 dev/plans/0.5.14/06-hitl-release-review.md create mode 100644 dev/plans/0.5.14/10-benchmark-safe-logging-status.md create mode 100644 dev/plans/0.5.14/110-fathomdb-operational-reads-design.md create mode 100644 dev/plans/0.5.14/110-fathomdb-operational-reads-status.md create mode 100644 dev/plans/0.5.14/120-documentation-contract-status.md create mode 100644 dev/plans/0.5.14/130-release-closeout.md create mode 100644 dev/plans/0.5.14/20-fathomdb-chat-status.md create mode 100644 dev/plans/0.5.14/30-embeddings-design.md create mode 100644 dev/plans/0.5.14/30-embeddings-status.md create mode 100644 dev/plans/0.5.14/40-provider-foundation-design.md create mode 100644 dev/plans/0.5.14/40-provider-foundation-status.md create mode 100644 dev/plans/0.5.14/50-openrouter-design.md create mode 100644 dev/plans/0.5.14/50-openrouter-status.md create mode 100644 dev/plans/0.5.14/60-deepseek-design.md create mode 100644 dev/plans/0.5.14/60-deepseek-status.md create mode 100644 dev/plans/0.5.14/70-tui-lifecycle-design.md create mode 100644 dev/plans/0.5.14/70-tui-lifecycle-status.md create mode 100644 dev/plans/0.5.14/80-routing-client-diagnostics-design.md create mode 100644 dev/plans/0.5.14/80-routing-client-diagnostics-status.md create mode 100644 dev/plans/0.5.14/90-qos-exporter-health-design.md create mode 100644 dev/plans/0.5.14/90-qos-exporter-health-status.md create mode 100644 dev/plans/0.5.14/README.md create mode 100644 tests/test_embedding_boundary.py create mode 100644 tests/test_operational_reads.py create mode 100644 tests/test_provider_errors.py create mode 100644 tests/test_tui_harness.py create mode 100644 tests/test_tui_qos_telemetry.py create mode 100644 tests/test_tui_routing.py diff --git a/.env.example b/.env.example index aa5d84a..2fc66cf 100644 --- a/.env.example +++ b/.env.example @@ -7,6 +7,10 @@ OPENAI_API_KEY=sk-... GOOGLE_AISTUDIO_API_KEY=AIza... MISTRAL_API_KEY=... PERPLEXITY_API_KEY=pplx-... +# Optional only after adding an explicit reviewed model_list deployment entry; +# the shipped config does not expose OpenRouter merely because this exists. +OPENROUTER_API_KEY=... +DEEPSEEK_API_KEY=... # --- Google Vertex AI (Gemini batch via /v1/batches) --- # Required only for the gemini-*-vertex deployments in config.yaml. The @@ -41,6 +45,9 @@ AIRLOCK_LOG_DIR=./logs # Common fields: messages, response # Example: AIRLOCK_LOG_REDACT_FIELDS=messages,response # AIRLOCK_LOG_REDACT_FIELDS= +# Benchmark-safe FathomDB client profile: use `messages,response`; leave SQL, +# S3, and Airlock's optional Fathom request logger disabled. See the +# Configuration guide for the complete retention-safe profile. # Optional telemetry sinks (since 0.5.4 each is fed by the recorder and enabled by # its own flag, not by a config.yaml callback entry): @@ -51,6 +58,12 @@ AIRLOCK_LOG_DIR=./logs # Optional: database logging — needs the flag AND a SQLAlchemy URL ([sql] extra) # AIRLOCK_ENABLE_SQL_LOGGER=1 # AIRLOCK_SQL_URL=postgresql+psycopg://user:pass@host/airlock +# AIRLOCK_ENABLE_FATHOM_LOGGER=1 +# AIRLOCK_FATHOM_STORE_MESSAGES=1 +# AIRLOCK_FATHOM_STORE_RESPONSE_TEXT=1 +# Optional operational-history source. Keep JSONL unless the proxy owns the +# FathomDB file; the separate-process TUI queries that owner over loopback. +# AIRLOCK_OPERATIONAL_READ_BACKEND=fathomdb # --- Guardrails --- # Comma-separated list of blocked keywords/phrases (case-insensitive) diff --git a/CHANGELOG.md b/CHANGELOG.md index 5aae868..cb8fe1e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,52 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +## [0.5.14] — 2026-08-12 + +### Added + +- **FathomDB benchmark readiness.** Airlock now provides explicit, + policy-governed aliases for `gpt-4o-mini` and + `text-embedding-3-small`, including a real `/v1/embeddings` boundary with + strict configured-alias and option validation. The documented benchmark + logging profile proves request and response redaction without retaining raw + Fathom content. +- **Opt-in OpenRouter and DeepSeek integration.** Operators can add reviewed + explicit model entries using runtime `OPENROUTER_API_KEY` or + `DEEPSEEK_API_KEY`; neither provider is enabled by default. Safe configured + model discovery, served-provider attribution, and bounded provider-error + handling support both integrations. +- **Optional proxy-owned FathomDB operational reads.** The proxy is the sole + FathomDB owner; the TUI and Advisor use authenticated bounded admin reads and + fall back truthfully to JSONL if the bridge is unavailable. +- **Operator visibility.** New bounded TUI/Admin views cover routing + classification, session affinity, QoS observations, and telemetry-exporter + health. Ordinary TUI tests use an explicit lifecycle harness while retaining + real lifecycle coverage. + +### Security + +- Embedding requests are always exact-alias pinned and reject direct-dispatch, + client-identity, and unsupported-option controls before provider dispatch. +- Provider discovery accepts only explicitly configured, same-origin HTTPS + catalog endpoints and never follows credential-bearing redirects. Provider + errors are reduced to bounded safe summaries before logging, tracing, and + monitor state. +- OpenRouter gateway-routing overrides and unsupported DeepSeek non-function + tools are rejected as OpenAI-shaped client errors before dispatch. + +### Changed + +- **FathomDB is now 0.8.22** (`fathomdb>=0.8.22,<0.9`) after optional-backend, + search, erasure, and proxy-owned operational-read verification. +- OpenRouter and DeepSeek recipes now require explicit operator configuration; + an environment key alone never exposes a model or activates discovery. + +### Deferred + +- Virtual-key management remains a 0.5.15 design and scope-decision item. +- PII egress remains in its previously approved observe-only deployment mode. + ## [0.5.12] — 2026-08-11 This public release includes the internal 0.5.11 FathomDB milestone below, diff --git a/airlock/__init__.py b/airlock/__init__.py index 38928eb..4c24d63 100644 --- a/airlock/__init__.py +++ b/airlock/__init__.py @@ -1,3 +1,3 @@ """Airlock — Enterprise LLM proxy built on LiteLLM.""" -__version__ = "0.5.12" +__version__ = "0.5.14" diff --git a/airlock/admin/http.py b/airlock/admin/http.py index e920846..1c897ec 100644 --- a/airlock/admin/http.py +++ b/airlock/admin/http.py @@ -11,6 +11,8 @@ from __future__ import annotations import json +import base64 +import os from typing import Any import airlock.fast.state as _state @@ -57,14 +59,30 @@ def _view_providers() -> dict: def _view_clients() -> dict: out = {} + admission_enabled = get_settings().admission.enabled + for client_id, client in _state.store.all_clients().items(): + if client.priority_score is not None: + out.setdefault(client_id, {})["priority"] = { + "score": client.priority_score, + "boost": client.priority_boost, + "reasons": list(client.priority_reasons), + "observed_at": client.priority_observed_at, + "admission_enabled": admission_enabled, + } for cid, cp in _state.store.all_client_provider_states().items(): client_id, provider = cid if cp.is_quarantined(): - out.setdefault(client_id, {})[provider] = { + out.setdefault(client_id, {}).setdefault("quarantines", {})[provider] = { "quarantined": True, "cooldown_remaining": round(cp.cooldown_remaining(), 1), } - return {"clients": out} + return {"source": "live_admin", "clients": out} + + +def _view_telemetry() -> dict: + from airlock.telemetry_health import telemetry_snapshot + + return {"source": "process_instrumentation", "exporters": telemetry_snapshot()} def _view_circuits() -> dict: @@ -77,6 +95,88 @@ def _view_circuits() -> dict: return {"circuits": out} +def _view_sessions() -> dict: + """Return bounded live affinity data without session identifiers.""" + from airlock.fast.router import _load_session_ttl + + return { + "source": "live_admin", + "sessions": _state.store.active_session_snapshot( + ttl_seconds=_load_session_ttl(), limit=100 + ), + } + + +def _operational_records(body: dict) -> dict: + """Serve bounded history from the proxy-owned datastore process only.""" + from airlock.operational_reads import read_records + + days = body.get("days", 31) + limit = body.get("limit", 5_000) + if not isinstance(days, int) or not 1 <= days <= 31: + raise ValueError("days must be an integer from 1 through 31") + if not isinstance(limit, int) or not 1 <= limit <= 5_000: + raise ValueError("limit must be an integer from 1 through 5000") + page = read_records( + directory=os.getenv("AIRLOCK_LOG_DIR", "./logs"), days=days, limit=limit + ) + return { + "records": page.records, + "source": page.source, + "degraded_reason": page.degraded_reason, + "truncated": page.truncated, + "limit_hit": page.limit_hit, + } + + +def _operational_errors(body: dict) -> dict: + from airlock.advisor.tools import get_recent_errors + + days = _operational_int(body, "days", default=2, maximum=31) + return get_recent_errors(os.getenv("AIRLOCK_LOG_DIR", "./logs"), days=days) + + +def _operational_search(body: dict) -> dict: + from airlock.advisor.tools import search_logs + + days = _operational_int(body, "days", default=7, maximum=31) + limit = _operational_int(body, "limit", default=20, maximum=50) + query = body.get("query") + if not isinstance(query, str) or not query.strip() or len(query) > 1_000: + raise ValueError("query must be a non-empty string of at most 1000 characters") + return search_logs( + os.getenv("AIRLOCK_LOG_DIR", "./logs"), + query=query, + limit=limit, + days=days, + ) + + +def _operational_int(body: dict, name: str, *, default: int, maximum: int) -> int: + value = body.get(name, default) + if ( + not isinstance(value, int) + or isinstance(value, bool) + or not 1 <= value <= maximum + ): + raise ValueError(f"{name} must be an integer from 1 through {maximum}") + return value + + +def _decode_session_client_selector(selector: str) -> str: + """Decode the opaque URL-safe client selector used by the TUI. + + Client identifiers are header-derived and therefore cannot safely be put + into a slash-delimited route. The selector is transport-only; the decoded + value remains subject to normal StateStore normalization. + """ + try: + padded = selector + "=" * (-len(selector) % 4) + return base64.urlsafe_b64decode(padded.encode("ascii")).decode("utf-8") + except (ValueError, UnicodeDecodeError): + raise ValueError("invalid session client selector") from None + + # --- route table: (method, segments-template) -> (scope, loopback_only, fn) -- def _match_route(method: str, path: str): """Return (op_scope, loopback_only, handler) or None. @@ -94,6 +194,18 @@ def _match_route(method: str, path: str): return ("admin:read", False, lambda p, b, a: _view_clients()) if method == "GET" and seg == ["circuits"]: return ("admin:read", False, lambda p, b, a: _view_circuits()) + if method == "GET" and seg == ["sessions"]: + return ("admin:read", False, lambda p, b, a: _view_sessions()) + if method == "GET" and seg == ["telemetry"]: + return ("admin:read", False, lambda p, b, a: _view_telemetry()) + if method == "POST" and seg == ["operational", "records"]: + # History may contain request content; it is a local TUI bridge, not a + # remotely capability-token-readable snapshot. + return ("admin:read", True, lambda p, b, a: _operational_records(b)) + if method == "POST" and seg == ["operational", "errors"]: + return ("admin:read", True, lambda p, b, a: _operational_errors(b)) + if method == "POST" and seg == ["operational", "search"]: + return ("admin:read", True, lambda p, b, a: _operational_search(b)) if ( method == "POST" @@ -150,6 +262,20 @@ def _match_route(method: str, path: str): False, lambda p, b, a: _state.store.clear_client_backoff(client, actor=a), ) + if ( + method == "POST" + and len(seg) == 3 + and seg[0] == "session-clients" + and seg[2] == "clear" + ): + selector = seg[1] + return ( + "admin:clear_sessions", + False, + lambda p, b, a: _state.store.clear_client_sessions( + _decode_session_client_selector(selector), actor=a + ), + ) if method == "POST" and len(seg) == 3 and seg[0] == "clients" and seg[2] == "erase": client = seg[1] return ( diff --git a/airlock/advisor/agent.py b/airlock/advisor/agent.py index 9d9e832..62fa3d8 100644 --- a/airlock/advisor/agent.py +++ b/airlock/advisor/agent.py @@ -106,6 +106,8 @@ def _execute_tool( store: StateStore | None, log_dir: str, config_path: str, + proxy_host: str | None = None, + proxy_port: str | None = None, ) -> str: """Execute a tool from TOOL_REGISTRY, injecting store/log_dir as needed.""" if tool_name not in TOOL_REGISTRY: @@ -121,6 +123,10 @@ def _execute_tool( kwargs["log_dir"] = log_dir if "config_path" in params: kwargs["config_path"] = config_path + if "proxy_host" in params and proxy_host is not None: + kwargs["proxy_host"] = proxy_host + if "proxy_port" in params and proxy_port is not None: + kwargs["proxy_port"] = proxy_port try: result = func(**kwargs) @@ -233,6 +239,8 @@ def run_advisor( store, actual_log_dir, actual_config, + proxy_host, + proxy_port, ) messages.append( diff --git a/airlock/advisor/tools.py b/airlock/advisor/tools.py index 77a9c09..0a735a5 100644 --- a/airlock/advisor/tools.py +++ b/airlock/advisor/tools.py @@ -25,12 +25,17 @@ from airlock.api.queries import ( DATASTORE_QUERY_LIMIT, - get_request_logs, - node_properties, + SEARCH_RESULT_LIMIT, search_request_logs, ) from airlock.fast.state import StateStore from airlock.log_query import LogPage, LogQuery, query_logs +from airlock.operational_reads import ( + _within_days, + fathomdb_selected, + read_records, + selected_fathom_engine, +) logger = logging.getLogger("airlock.advisor.tools") @@ -135,42 +140,31 @@ def get_state_snapshot(store: StateStore) -> dict: # --------------------------------------------------------------------------- -def get_recent_errors(log_dir: str, days: int = 2) -> dict: +def get_recent_errors( + log_dir: str, + days: int = 2, + proxy_host: str | None = None, + proxy_port: str | None = None, +) -> dict: """Load JSONL logs, filter to failures, group by model and error_type.""" - engine = None - try: - import airlock.datastore - - engine = airlock.datastore.get_engine() - except Exception: - engine = None - - truncated: dict[str, Any] = {"truncated": False, "limit_hit": None} - if engine is not None: - # Bounded read (was limit=1000000 — a limit in name only). The failure - # filter stays Python-side: the engine's json-path predicate allowlist - # is fixed and does not cover $.success. - nodes = get_request_logs(engine, limit=DATASTORE_QUERY_LIMIT) - records = [node_properties(n) for n in nodes] - failures = [ - r - for r in records - if (r.get("success") is False) or bool(r.get("error_flag")) - ] - if len(records) >= DATASTORE_QUERY_LIMIT: - truncated = {"truncated": True, "limit_hit": "datastore_limit"} - else: - # Filter during the scan so a quiet window never materializes the - # successful traffic it is dominated by. - page = _load_page( - log_dir, - days=days, - predicate=lambda r: ( - (r.get("success") is False) or bool(r.get("error_flag")) - ), - ) - failures = page.records - truncated = _truncation(page) + bridge_selected = fathomdb_selected() and bool(proxy_host and proxy_port) + if bridge_selected: + from airlock.tui.admin_client import operational_view + + remote = operational_view(proxy_host, proxy_port, "errors", {"days": days}) + if remote is not None: + return remote + page = read_records( + directory=log_dir, + days=days, + limit=DATASTORE_QUERY_LIMIT, + # A separate Advisor process never opens a selected proxy-owned DB if + # its bridge fails. Its truthful fallback is JSONL only. + allow_fathom=not bridge_selected, + predicate=lambda r: (r.get("success") is False) or bool(r.get("error_flag")), + ) + failures = page.records + truncated = {"truncated": page.truncated, "limit_hit": page.limit_hit} by_model: Counter = Counter() by_error_type: Counter = Counter() @@ -197,6 +191,8 @@ def get_recent_errors(log_dir: str, days: int = 2) -> dict: "by_error_type": dict(by_error_type), "by_client": dict(by_client), "recent_samples": recent_samples, + "source": page.source, + "degraded_reason": page.degraded_reason, # The advisor must be able to say "based on a partial window" rather # than presenting a truncated scan as the whole picture. "window": truncated, @@ -208,40 +204,100 @@ def get_recent_errors(log_dir: str, days: int = 2) -> dict: # --------------------------------------------------------------------------- -def search_logs(log_dir: str, query: str, limit: int = 20, days: int = 7) -> dict: - """Search request logs (#11): engine search when available, JSONL scan otherwise. +def search_logs( + log_dir: str, + query: str, + limit: int = 20, + days: int = 7, + proxy_host: str | None = None, + proxy_port: str | None = None, +) -> dict: + """Search the explicitly selected source without mislabelling its mode. - The engine path reports its ``mode`` honestly — ``hybrid`` only when dense - retrieval actually contributed alongside lexical; ``lexical_only`` with the - reason otherwise. The JSONL fallback is a bounded substring scan and says - so: ``substring`` is not search-engine ranking and must not look like it. + FathomDB opt-in uses its existing active-only FTS/hybrid reader. JSONL + remains a bounded substring fallback and is never presented as ranked + search. Both sources report their source and any degraded operation. """ - engine = None - try: - import airlock.datastore + if not isinstance(limit, int) or isinstance(limit, bool): + raise ValueError("limit must be an integer") + limit = max(1, min(limit, SEARCH_RESULT_LIMIT)) + bridge_selected = fathomdb_selected() and bool(proxy_host and proxy_port) + needle = query.lower() - engine = airlock.datastore.get_engine() - except Exception: - engine = None + if bridge_selected: + from airlock.tui.admin_client import operational_view + remote = operational_view( + proxy_host, + proxy_port, + "search", + {"query": query, "limit": limit, "days": days}, + ) + if remote is not None: + return remote + + if bridge_selected: + engine, selection_reason = ( + None, + ( + "FathomDB selected but proxy operational reads unavailable; " + "using bounded JSONL" + ), + ) + else: + engine, selection_reason = selected_fathom_engine() if engine is not None: - out = search_request_logs(engine, query, limit=limit) - out["backend"] = "fathomdb" - return out - - needle = query.lower() + try: + result = search_request_logs(engine, query, limit=limit) + except Exception: + result = None + if result is not None: + result_filled_cap = len(result["results"]) >= limit + result["results"] = [ + row + for row in result["results"] + if _within_days(row["properties"], days) + ] + result.update( + { + "backend": "fathomdb", + "window": { + # Fathom FTS applies its cap before the caller's time + # filter. A full raw page can therefore conceal a + # later in-window hit even when this filtered page is + # empty; never present that window as complete. + "truncated": result_filled_cap, + "limit_hit": ( + "search_result_limit" if result_filled_cap else None + ), + }, + } + ) + return result def _matches(record: dict[str, Any]) -> bool: return needle in json.dumps(record, default=str).lower() - page = _load_page(log_dir, days=days, predicate=_matches) + page = read_records( + directory=log_dir, + days=days, + predicate=_matches, + limit=min(limit, DATASTORE_QUERY_LIMIT), + allow_fathom=not bridge_selected, + ) return { - "backend": "jsonl", + "backend": page.source, "mode": "substring", - "degraded_reason": "datastore disabled; bounded JSONL substring scan", + "degraded_reason": page.degraded_reason + or selection_reason + or ( + "FathomDB not selected; bounded JSONL substring scan" + if page.source == "jsonl" + else None + ), "soft_fallback": None, "results": page.records[-limit:], - "window": _truncation(page), + "window": {"truncated": page.truncated, "limit_hit": page.limit_hit}, } diff --git a/airlock/callbacks/enterprise_logger.py b/airlock/callbacks/enterprise_logger.py index 3792a20..9c7b414 100644 --- a/airlock/callbacks/enterprise_logger.py +++ b/airlock/callbacks/enterprise_logger.py @@ -26,6 +26,7 @@ from airlock.client_identity import extract_airlock_client_from_kwargs from airlock.fast.router import infer_provider from airlock.metadata_policy import SECRET_METADATA_KEYS +from airlock.provider_errors import summarize_provider_error from airlock.fast.state import normalize_client_id from litellm.integrations.custom_logger import CustomLogger @@ -79,6 +80,10 @@ def _normalize_failure( exc = kwargs.get("exception") exc_type = type(exc).__name__ if exc is not None else None + provider_summary = summarize_provider_error(exc) + if provider_summary is not None: + return provider_summary.message(), provider_summary.error_type, "provider" + error_text = "" if exc is not None: error_text = str(exc).strip() diff --git a/airlock/callbacks/metrics.py b/airlock/callbacks/metrics.py index 1cafce5..1bb91f4 100644 --- a/airlock/callbacks/metrics.py +++ b/airlock/callbacks/metrics.py @@ -29,6 +29,13 @@ from litellm.integrations.custom_logger import CustomLogger from airlock.callbacks.memory import collect_memory_snapshot +from airlock.telemetry_health import configure_exporter, record_signal + +configure_exporter( + "prometheus", + enabled=_PROM_AVAILABLE, + endpoint="http://in-process/metrics" if _PROM_AVAILABLE else None, +) def _build_metrics() -> dict[str, Any]: @@ -245,6 +252,8 @@ def record_event(self, event: Any) -> None: if not _PROM_AVAILABLE: return + record_signal("prometheus") + # This callback executes in the LiteLLM worker after every success/failure, # correlating the latest kernel memory counters with request completion. _record_memory_snapshot() diff --git a/airlock/callbacks/oom_diagnostics.py b/airlock/callbacks/oom_diagnostics.py index cc522b2..bd335b2 100644 --- a/airlock/callbacks/oom_diagnostics.py +++ b/airlock/callbacks/oom_diagnostics.py @@ -29,7 +29,10 @@ def _enabled() -> bool: return os.getenv("AIRLOCK_OOM_DIAGNOSTICS", "").strip().lower() in { - "1", "true", "yes", "on" + "1", + "true", + "yes", + "on", } @@ -40,8 +43,13 @@ def _tracemalloc_enabled() -> bool: deliberately on for short diagnostic runs, while a long reproduction can turn it off and attach a short native profile only at high water. """ - return os.getenv("AIRLOCK_OOM_DIAGNOSTICS_TRACEMALLOC", "1").strip().lower() not in { - "0", "false", "no", "off", + return os.getenv( + "AIRLOCK_OOM_DIAGNOSTICS_TRACEMALLOC", "1" + ).strip().lower() not in { + "0", + "false", + "no", + "off", } @@ -87,8 +95,16 @@ class _Mallinfo2(ctypes.Structure): _fields_ = [ (name, ctypes.c_size_t) for name in ( - "arena", "ordblks", "smblks", "hblks", "hblkhd", "usmblks", - "fsmblks", "uordblks", "fordblks", "keepcost", + "arena", + "ordblks", + "smblks", + "hblks", + "hblkhd", + "usmblks", + "fsmblks", + "uordblks", + "fordblks", + "keepcost", ) ] @@ -117,7 +133,8 @@ def _object_type_counts() -> dict[str, Any]: """Return type names/counts only; never object values or reprs.""" try: counts = Counter( - f"{type(obj).__module__}.{type(obj).__qualname__}" for obj in gc.get_objects() + f"{type(obj).__module__}.{type(obj).__qualname__}" + for obj in gc.get_objects() ) except Exception: return {} @@ -166,7 +183,9 @@ def _start(self) -> None: with self._lock: if self._started: return - directory = Path(os.getenv("AIRLOCK_OOM_DIAGNOSTICS_DIR", "/tmp/airlock-oom-diagnostics")) + directory = Path( + os.getenv("AIRLOCK_OOM_DIAGNOSTICS_DIR", "/tmp/airlock-oom-diagnostics") + ) directory.mkdir(mode=0o700, parents=True, exist_ok=True) self._path = directory / f"litellm-{os.getpid()}.jsonl" if _tracemalloc_enabled(): @@ -179,13 +198,19 @@ def _start(self) -> None: self.snapshot("diagnostics_started") def _append(self, record: dict[str, Any]) -> None: - if self._path is None or self._records >= _positive_int("AIRLOCK_OOM_DIAGNOSTICS_MAX_RECORDS", 6000): + if self._path is None or self._records >= _positive_int( + "AIRLOCK_OOM_DIAGNOSTICS_MAX_RECORDS", 6000 + ): return - line = (json.dumps(record, separators=(",", ":"), sort_keys=True) + "\n").encode() + line = ( + json.dumps(record, separators=(",", ":"), sort_keys=True) + "\n" + ).encode() # A single O_APPEND write plus this process lock prevents interleaving. try: with self._lock: - if self._records >= _positive_int("AIRLOCK_OOM_DIAGNOSTICS_MAX_RECORDS", 6000): + if self._records >= _positive_int( + "AIRLOCK_OOM_DIAGNOSTICS_MAX_RECORDS", 6000 + ): return fd = os.open(self._path, os.O_WRONLY | os.O_CREAT | os.O_APPEND, 0o600) try: @@ -200,7 +225,14 @@ def _append(self, record: dict[str, Any]) -> None: except OSError: pass - def snapshot(self, phase: str, *, sequence: int | None = None, outcome: str | None = None, elapsed_ms: int | None = None) -> None: + def snapshot( + self, + phase: str, + *, + sequence: int | None = None, + outcome: str | None = None, + elapsed_ms: int | None = None, + ) -> None: if not _enabled(): return self._start() @@ -216,27 +248,41 @@ def snapshot(self, phase: str, *, sequence: int | None = None, outcome: str | No ) with self._lock: in_flight = self._in_flight - self._append({ - "ts_monotonic_ns": time.monotonic_ns(), "phase": phase, - "sequence": sequence, "outcome": outcome, "elapsed_ms": elapsed_ms, - "in_flight": in_flight, - "memory": { - "rss": memory.process_rss_bytes, "rss_peak": memory.process_rss_peak_bytes, - "cgroup_current": memory.cgroup_current_bytes, "cgroup_peak": memory.cgroup_peak_bytes, - "cgroup_high": memory.cgroup_high_bytes, "cgroup_max": memory.cgroup_max_bytes, - "events": memory.cgroup_events, "smaps": smaps, "pressure": _memory_pressure(), - }, - "allocator": _mallinfo(), "tracemalloc": {"current": trace_current, "peak": trace_peak}, - "process": {"threads": _count_entries("/proc/self/task"), "fds": _count_entries("/proc/self/fd")}, - # gc.get_objects() walks the full Python heap. It is useful at - # periodic/high-water checkpoints but would itself perturb a - # thousands-request faithful replay if performed three times per - # request. - "objects": _object_type_counts() - if phase == "periodic" or phase.startswith("signal_") - else {}, - "transport": _transport_counts(), - }) + self._append( + { + "ts_monotonic_ns": time.monotonic_ns(), + "phase": phase, + "sequence": sequence, + "outcome": outcome, + "elapsed_ms": elapsed_ms, + "in_flight": in_flight, + "memory": { + "rss": memory.process_rss_bytes, + "rss_peak": memory.process_rss_peak_bytes, + "cgroup_current": memory.cgroup_current_bytes, + "cgroup_peak": memory.cgroup_peak_bytes, + "cgroup_high": memory.cgroup_high_bytes, + "cgroup_max": memory.cgroup_max_bytes, + "events": memory.cgroup_events, + "smaps": smaps, + "pressure": _memory_pressure(), + }, + "allocator": _mallinfo(), + "tracemalloc": {"current": trace_current, "peak": trace_peak}, + "process": { + "threads": _count_entries("/proc/self/task"), + "fds": _count_entries("/proc/self/fd"), + }, + # gc.get_objects() walks the full Python heap. It is useful at + # periodic/high-water checkpoints but would itself perturb a + # thousands-request faithful replay if performed three times per + # request. + "objects": _object_type_counts() + if phase == "periodic" or phase.startswith("signal_") + else {}, + "transport": _transport_counts(), + } + ) def pre_call(self, data: dict) -> None: if not _enabled(): @@ -258,8 +304,15 @@ def post_call(self, data: dict) -> None: if not _enabled(): return metadata = data.get("metadata", {}) if isinstance(data, dict) else {} - sequence = metadata.get("airlock_oom_diag_sequence") if isinstance(metadata, dict) else None - self.snapshot("provider_response", sequence=sequence if isinstance(sequence, int) else None) + sequence = ( + metadata.get("airlock_oom_diag_sequence") + if isinstance(metadata, dict) + else None + ) + self.snapshot( + "provider_response", + sequence=sequence if isinstance(sequence, int) else None, + ) def record_event(self, event: Any) -> None: if not _enabled(): @@ -269,10 +322,19 @@ def record_event(self, event: Any) -> None: started = metadata.get("airlock_oom_diag_started_ns") if not isinstance(sequence, int): return - elapsed_ms = (time.monotonic_ns() - started) // 1_000_000 if isinstance(started, int) else None + elapsed_ms = ( + (time.monotonic_ns() - started) // 1_000_000 + if isinstance(started, int) + else None + ) with self._lock: self._in_flight = max(0, self._in_flight - 1) - self.snapshot("callback_complete", sequence=sequence, outcome="success" if getattr(event, "success", False) else "failure", elapsed_ms=elapsed_ms) + self.snapshot( + "callback_complete", + sequence=sequence, + outcome="success" if getattr(event, "success", False) else "failure", + elapsed_ms=elapsed_ms, + ) def _signal_usr1(self, _signum: int, _frame: Any) -> None: self._signal_snapshot("signal_usr1", trim=False) @@ -283,6 +345,7 @@ def _signal_usr2(self, _signum: int, _frame: Any) -> None: def _signal_snapshot(self, phase: str, *, trim: bool) -> None: if not _enabled() or not self._signal_lock.acquire(blocking=False): return + def worker() -> None: try: with self._lock: @@ -300,7 +363,10 @@ def worker() -> None: self.snapshot(f"{phase}_after_gc") finally: self._signal_lock.release() - threading.Thread(target=worker, daemon=True, name="airlock-oom-snapshot").start() + + threading.Thread( + target=worker, daemon=True, name="airlock-oom-snapshot" + ).start() def _count_entries(path: str) -> int | None: @@ -317,12 +383,22 @@ class AirlockOOMDiagnosticsGuard(CustomGuardrail): """No-op guardrail unless diagnostics are explicitly enabled for a repro.""" def __init__(self, **kwargs: Any) -> None: - super().__init__(supported_event_hooks=[GuardrailEventHooks.pre_call, GuardrailEventHooks.post_call], **kwargs) + super().__init__( + supported_event_hooks=[ + GuardrailEventHooks.pre_call, + GuardrailEventHooks.post_call, + ], + **kwargs, + ) - async def async_pre_call_hook(self, user_api_key_dict: Any, cache: DualCache, data: dict, call_type: str) -> dict: # noqa: ARG002 + async def async_pre_call_hook( + self, user_api_key_dict: Any, cache: DualCache, data: dict, call_type: str + ) -> dict: # noqa: ARG002 oom_diagnostics.pre_call(data) return data - async def async_post_call_success_hook(self, data: dict, user_api_key_dict: Any, response: Any) -> Any: # noqa: ARG002 + async def async_post_call_success_hook( + self, data: dict, user_api_key_dict: Any, response: Any + ) -> Any: # noqa: ARG002 oom_diagnostics.post_call(data) return response diff --git a/airlock/callbacks/request_event.py b/airlock/callbacks/request_event.py index 742245f..5296b21 100644 --- a/airlock/callbacks/request_event.py +++ b/airlock/callbacks/request_event.py @@ -34,6 +34,7 @@ ) from airlock.fast.router import infer_provider from airlock.metadata_policy import SECRET_METADATA_KEYS +from airlock.provider_errors import summarize_provider_error from airlock.gemini_interface import ( build_gemini_response_headers, classify_gemini_response, @@ -132,8 +133,14 @@ def build_request_event( bare_exception_error = None if not success: error, error_type, failure_category = _normalize_failure(kwargs, response_obj) - # s3/sql project the raw exception string verbatim (incl. str(None) == "None") - bare_exception_error = str(kwargs.get("exception")) + # Provider errors are bounded before any projection sees them. Preserve the + # historical raw local-error form (including str(None) == "None"). + provider_summary = summarize_provider_error(kwargs.get("exception")) + bare_exception_error = ( + provider_summary.message() + if provider_summary is not None + else str(kwargs.get("exception")) + ) # Token usage — same getattr-with-0 sourcing as the enterprise builder. usage: dict[str, int] = {} diff --git a/airlock/callbacks/tracing.py b/airlock/callbacks/tracing.py index 57a4948..88cca79 100644 --- a/airlock/callbacks/tracing.py +++ b/airlock/callbacks/tracing.py @@ -17,6 +17,9 @@ import os from typing import Any +from airlock.provider_errors import summarize_provider_error +from airlock.telemetry_health import configure_exporter, record_signal + logger = logging.getLogger("airlock.callbacks.tracing") try: @@ -45,10 +48,15 @@ def _get_tracer() -> Any: provider = TracerProvider(resource=resource) trace.set_tracer_provider(provider) - return trace.get_tracer("airlock", "0.5.12") + return trace.get_tracer("airlock", "0.5.14") _tracer = _get_tracer() +configure_exporter( + "otlp", + enabled=_OTEL_AVAILABLE, + endpoint=os.getenv("OTEL_EXPORTER_OTLP_ENDPOINT"), +) class AirlockTracingCallback(CustomLogger): @@ -64,6 +72,7 @@ def log_success_event( model = kwargs.get("model", "unknown") with _tracer.start_as_current_span("llm.request") as span: + record_signal("otlp") span.set_attribute("llm.model", model) span.set_attribute("llm.success", True) span.set_attribute("llm.request_id", kwargs.get("litellm_call_id", "")) @@ -120,13 +129,20 @@ def log_failure_event( error = kwargs.get("exception") with _tracer.start_as_current_span("llm.request") as span: + record_signal("otlp") span.set_attribute("llm.model", model) span.set_attribute("llm.success", False) span.set_attribute("llm.request_id", kwargs.get("litellm_call_id", "")) if error: - span.set_attribute("llm.error", str(error)) - span.record_exception(error) + summary = summarize_provider_error(error) + if summary is not None: + # Never export a provider response body in an OTel exception. + span.set_attribute("llm.error", summary.message()) + span.record_exception(Exception(summary.message())) + else: + span.set_attribute("llm.error", str(error)) + span.record_exception(error) async def async_log_failure_event( self, kwargs: dict, response_obj: Any, start_time: Any, end_time: Any diff --git a/airlock/capability.py b/airlock/capability.py index dacf525..444b93e 100644 --- a/airlock/capability.py +++ b/airlock/capability.py @@ -60,13 +60,17 @@ def airlock_provider_for(entry: dict) -> str | None: def endpoints_for(entry: dict) -> list[str]: """Return the supported endpoints for a ``model_list`` entry. - Always includes ``"chat"``. Appends ``"batch"`` iff the entry carries a - truthy ``airlock_batch`` marker OR it is a ``vertex_ai/`` model with a + An ``airlock_embeddings: true`` entry is embedding-only. All other entries + include ``"chat"`` and append ``"batch"`` iff the entry carries a truthy + ``airlock_batch`` marker OR it is a ``vertex_ai/`` model with a *regional* ``vertex_location`` (set and not ``global``). A bare ``vertex_ai/`` prefix is NOT sufficient — vertex batch is region-gated, and the current entries use ``vertex_location: global`` (sync/chat-only). This is the anti-overclaim rule; do not weaken it. """ + if entry.get("airlock_embeddings"): + return ["embeddings"] + endpoints = ["chat"] if entry.get("airlock_batch"): diff --git a/airlock/cli/templates/config.yaml b/airlock/cli/templates/config.yaml index ca4c317..bf043bf 100644 --- a/airlock/cli/templates/config.yaml +++ b/airlock/cli/templates/config.yaml @@ -67,6 +67,32 @@ model_list: model: openai/gpt-5.3-codex api_key: os.environ/OPENAI_API_KEY + # FathomDB benchmark extractor/chat model. This is an explicit alias, not a + # routing fallback: benchmark callers must request it deliberately. + - model_name: gpt-4o-mini + litellm_params: + model: openai/gpt-4o-mini + api_key: os.environ/OPENAI_API_KEY + + # FathomDB benchmark embedding model. The explicit marker keeps this model + # off chat completions and authorizes it only for /v1/embeddings. + - model_name: text-embedding-3-small + litellm_params: + model: openai/text-embedding-3-small + api_key: os.environ/OPENAI_API_KEY + airlock_embeddings: true + + - model_name: openai/gpt-4o-mini + litellm_params: + model: openai/gpt-4o-mini + api_key: os.environ/OPENAI_API_KEY + + - model_name: openai/text-embedding-3-small + litellm_params: + model: openai/text-embedding-3-small + api_key: os.environ/OPENAI_API_KEY + airlock_embeddings: true + # --- OpenAI GPT-5.6 (GA 2026-07-09) — Sol / Terra / Luna --- # A flat PRICE-TIER triple over one capability envelope, not a mini/nano # capability ladder: all three share 1,050,000 ctx / 128,000 max output and diff --git a/airlock/client_identity.py b/airlock/client_identity.py index b01e013..15f4f1f 100644 --- a/airlock/client_identity.py +++ b/airlock/client_identity.py @@ -54,8 +54,11 @@ def extract_airlock_client_from_request( metadata = data.get("metadata") or {} for value in ( metadata.get("airlock_client"), - extract_airlock_client_from_headers(data.get("headers")), extract_airlock_client_from_headers(metadata.get("headers")), + # Root headers may be an optional LiteLLM forwarding field. The real + # proxy headers stored in metadata are authoritative, so a request body + # cannot forge an Airlock client identity through this fallback. + extract_airlock_client_from_headers(data.get("headers")), ): if value: return normalize_client_id(str(value).strip()) @@ -108,10 +111,10 @@ def extract_airlock_client_from_kwargs(kwargs: Mapping[str, Any]) -> str | None: return text header_sources: list[Mapping[str, Any] | None] = [ - kwargs.get("headers") if isinstance(kwargs.get("headers"), Mapping) else None, metadata.get("headers") if isinstance(metadata.get("headers"), Mapping) else None, + kwargs.get("headers") if isinstance(kwargs.get("headers"), Mapping) else None, ] for key in ("request", "proxy_server_request", "http_request"): obj = kwargs.get(key) diff --git a/airlock/embedding.py b/airlock/embedding.py new file mode 100644 index 0000000..28bed1d --- /dev/null +++ b/airlock/embedding.py @@ -0,0 +1,114 @@ +"""Embedding-request validation owned by Airlock's pre-dispatch boundary.""" + +from __future__ import annotations + +from typing import Any + +from litellm.proxy._types import ProxyException + + +class AirlockInvalidEmbeddingOption(ProxyException): + """A client supplied an embedding option outside Airlock's contract.""" + + def __init__(self, option: str, reason: str) -> None: + self.option = option + self._airlock_error_code = "invalid_embedding_option" + super().__init__( + message=f"Embedding option {option!r} is invalid: {reason}.", + type="invalid_request_error", + param=option, + code=400, + openai_code=self._airlock_error_code, + ) + + def to_dict(self) -> dict: + body = super().to_dict() + body["code"] = self._airlock_error_code + return body + + +# This is intentionally an allowlist, rather than a list of known bad chat +# controls. Airlock uses LiteLLM with ``drop_params: true`` globally, so an +# unknown client body member must not reach transport where it could be silently +# removed. Internal fields are added by the LiteLLM proxy before pre-call hooks. +_CLIENT_EMBEDDING_FIELDS = frozenset( + {"model", "input", "dimensions", "encoding_format", "user"} +) +_PROXY_INTERNAL_EMBEDDING_FIELDS = frozenset( + { + "metadata", + "secret_fields", + "litellm_call_id", + "litellm_session_id", + "litellm_trace_id", + "litellm_logging_obj", + "proxy_server_request", + "api_version", + "request_timeout", + "timeout", + "stream_timeout", + "max_retries", + "num_retries", + "disable_fallbacks", + "model_info", + "litellm_params", + "caching", + "cache", + "ttl", + "organization", + "allowed_model_region", + } +) + + +def validate_embedding_options(data: dict[str, Any]) -> None: + """Reject invalid or completion-only embedding options before dispatch. + + The public contract is ``model``, ``input``, optional ``user``, optional + positive ``dimensions`` up to the model's native 1536 dimensions, and + OpenAI's ``float``/``base64`` encoding formats. The permit list excludes + proxy-created internal fields, which are not client API options. Airlock + intentionally rejects direct-dispatch and header controls (`api_base`, + `api_key`, provider/deployment controls, and `headers`) even if they appear + after proxy processing: LiteLLM's embedding route can honor client-supplied + versions and would bypass the reviewed model-list deployment. + """ + for option in data: + if option not in _CLIENT_EMBEDDING_FIELDS | _PROXY_INTERNAL_EMBEDDING_FIELDS: + raise AirlockInvalidEmbeddingOption(option, "not supported for embeddings") + + if "dimensions" in data: + dimensions = data["dimensions"] + if ( + isinstance(dimensions, bool) + or not isinstance(dimensions, int) + or not 1 <= dimensions <= 1536 + ): + raise AirlockInvalidEmbeddingOption( + "dimensions", "must be an integer from 1 through 1536" + ) + + if "encoding_format" in data and data["encoding_format"] not in { + "float", + "base64", + }: + raise AirlockInvalidEmbeddingOption( + "encoding_format", "must be 'float' or 'base64'" + ) + + +def validate_embedding_client_body(data: object) -> None: + """Validate raw `/v1/embeddings` JSON before LiteLLM augments it. + + This is deliberately separate from :func:`validate_embedding_options`. + Once LiteLLM has mixed its own fields into the request, the guardrail cannot + establish whether an `api_base` or `headers` value originated in the client + body. The authenticated route wrapper calls this function before that mixing + occurs. + """ + if not isinstance(data, dict): + raise AirlockInvalidEmbeddingOption("body", "must be a JSON object") + for option in data: + if option not in _CLIENT_EMBEDDING_FIELDS: + raise AirlockInvalidEmbeddingOption(option, "not supported for embeddings") + validate_embedding_options(data) diff --git a/airlock/embedding_boundary.py b/airlock/embedding_boundary.py new file mode 100644 index 0000000..1be2493 --- /dev/null +++ b/airlock/embedding_boundary.py @@ -0,0 +1,85 @@ +"""Authenticated public-body contract for OpenAI embedding requests.""" + +from __future__ import annotations + +from typing import Any + +from fastapi.routing import APIRoute, request_response +from starlette.responses import JSONResponse + +from airlock.embedding import ( + AirlockInvalidEmbeddingOption, + validate_embedding_client_body, +) +from airlock.litellm_adapter import resolve_proxy_app + +_EMBEDDING_PATHS = {"/v1/embeddings", "/embeddings"} + + +def _is_embedding_path(path: str) -> bool: + """Return whether *path* is any LiteLLM OpenAI-compatible embedding route.""" + normalized = path.rstrip("/") or "/" + return ( + normalized in _EMBEDDING_PATHS + or (normalized.startswith("/engines/") and normalized.endswith("/embeddings")) + or ( + normalized.startswith("/openai/deployments/") + and normalized.endswith("/embeddings") + ) + ) + + +def _invalid_option_response(exc: AirlockInvalidEmbeddingOption) -> JSONResponse: + """Return the established OpenAI-shaped validation response.""" + return JSONResponse(status_code=400, content={"error": exc.to_dict()}) + + +def _install_authenticated_embedding_validator(route: APIRoute) -> None: + """Insert validation after LiteLLM's auth dependency, before its endpoint. + + LiteLLM authenticates embeddings as a FastAPI dependency and reads/caches the + request body there. Wrapping the endpoint dependent therefore reuses that + authenticated cached body without introducing an outer pre-auth body read. + This protects the normal authentication/resource boundary while still + distinguishing client fields from LiteLLM's later proxy-generated fields. + """ + if getattr(route, "_airlock_embedding_validator_installed", False): + return + original = route.dependant.call + + async def _validated_endpoint(**kwargs: Any) -> Any: + request = kwargs["request"] + try: + body = await request.json() + except ValueError: + # Preserve LiteLLM's ordinary malformed-body response behavior. + return await original(**kwargs) + try: + validate_embedding_client_body(body) + except AirlockInvalidEmbeddingOption as exc: + return _invalid_option_response(exc) + return await original(**kwargs) + + route.dependant.call = _validated_endpoint + # Route handlers close over the dependant. Rebuild it after replacing the + # call so both pre- and post-start bootstrap paths use the wrapped endpoint. + route.app = request_response(route.get_route_handler()) + route._airlock_embedding_validator_installed = True + + +def install_embedding_request_boundary_on_proxy_app() -> bool: + """Install the idempotent authenticated embedding body boundary.""" + app = resolve_proxy_app() + if app is None or getattr(app, "state", None) is None: + return False + if getattr(app.state, "airlock_embedding_boundary_installed", False): + return True + for route in app.routes: + if ( + isinstance(route, APIRoute) + and "POST" in route.methods + and _is_embedding_path(route.path) + ): + _install_authenticated_embedding_validator(route) + app.state.airlock_embedding_boundary_installed = True + return True diff --git a/airlock/fast/_state_core.py b/airlock/fast/_state_core.py index 1257ef6..1f6cf2d 100644 --- a/airlock/fast/_state_core.py +++ b/airlock/fast/_state_core.py @@ -176,6 +176,10 @@ class ClientState: gemini_outcomes: deque = field(default_factory=lambda: deque(maxlen=MAX_SAMPLES)) threat_score: float = 0.0 backoff_until: float = 0.0 # unix ts; 0 → no backoff + priority_score: float | None = None + priority_boost: bool = False + priority_reasons: tuple[str, ...] = () + priority_observed_at: float = 0.0 # -- writers -------------------------------------------------------- @@ -192,6 +196,15 @@ def record_error(self, timestamp: float, error_type: str) -> None: def record_gemini_outcome(self, timestamp: float, output_shape: str) -> None: self.gemini_outcomes.append((timestamp, output_shape)) + def record_priority( + self, score: float, boost: bool, reasons: list[str], timestamp: float + ) -> None: + """Keep only bounded, non-content QoS evidence for operator reads.""" + self.priority_score = round(float(score), 3) + self.priority_boost = bool(boost) + self.priority_reasons = tuple(str(reason)[:120] for reason in reasons[:4]) + self.priority_observed_at = timestamp + # -- readers -------------------------------------------------------- def recent_request_count(self, window_seconds: float = WINDOW_SECONDS) -> int: @@ -245,6 +258,10 @@ class SessionRecord: session_id: str model: str + # Session identifiers are client supplied and may be sensitive. The live + # admin view therefore groups records by this authenticated client owner, + # never returns the identifier itself. + client_id: str = "" created_at: float = 0.0 last_used: float = 0.0 @@ -565,7 +582,7 @@ def __init__(self) -> None: self._lock = threading.RLock() self._clients: dict[str, ClientState] = {} self._models: dict[str, ModelState] = {} - self._sessions: dict[str, SessionRecord] = {} + self._sessions: dict[tuple[str, str], SessionRecord] = {} self._provider_spend: dict[str, ProviderSpend] = {} # Single seam-backed accumulator shared by all provider handles; shares the # store lock so compound read-modify-write stays atomic (FIX-3). @@ -636,33 +653,87 @@ def record_threat_score( client.backoff_until = now + backoff_seconds return combined, blocked, backoff_seconds + def record_client_priority( + self, client_id: str, score: float, boost: bool, reasons: list[str], now: float + ) -> None: + with self._lock: + self.get_client(client_id).record_priority(score, boost, reasons, now) + def all_models(self) -> dict[str, ModelState]: with self._lock: return dict(self._models) # -- Session affinity -------------------------------------------------- - def get_session(self, session_id: str) -> SessionRecord | None: + def get_session(self, session_id: str, client_id: str = "") -> SessionRecord | None: with self._lock: - return self._sessions.get(session_id) + return self._sessions.get((normalize_client_id(client_id), session_id)) - def set_session(self, session_id: str, model: str) -> SessionRecord: + def set_session( + self, session_id: str, model: str, client_id: str = "" + ) -> SessionRecord: now = time.time() + client_id = normalize_client_id(client_id) + key = (client_id, session_id) with self._lock: - if session_id in self._sessions: - rec = self._sessions[session_id] + if key in self._sessions: + rec = self._sessions[key] rec.model = model rec.last_used = now else: rec = SessionRecord( session_id=session_id, model=model, + client_id=client_id, created_at=now, last_used=now, ) - self._sessions[session_id] = rec + self._sessions[key] = rec return rec + def active_session_snapshot( + self, *, ttl_seconds: int, limit: int = 100, now: float | None = None + ) -> list[dict[str, object]]: + """Return a bounded, identifier-free snapshot of live affinity pins.""" + now = time.time() if now is None else now + with self._lock: + active = [ + rec + for rec in self._sessions.values() + if now - rec.last_used < ttl_seconds + ] + active.sort(key=lambda rec: rec.last_used, reverse=True) + return [ + { + "client_id": rec.client_id, + "model": rec.model, + "age_seconds": round(max(0.0, now - rec.created_at), 1), + "ttl_remaining_seconds": round( + max(0.0, ttl_seconds - (now - rec.last_used)), 1 + ), + } + for rec in active[:limit] + ] + + def clear_client_sessions( + self, client_id: str, *, actor: str = "", now: float | None = None + ) -> dict: + """Break every live session pin owned by a client and audit the action.""" + now = time.time() if now is None else now + client_id = normalize_client_id(client_id) + with self._lock: + keys = [key for key in self._sessions if key[0] == client_id] + for key in keys: + del self._sessions[key] + return { + "record_type": "admin_action", + "timestamp": self._admin_iso(now), + "op": "clear_client_sessions", + "actor": actor, + "client_id": client_id, + "cleared_sessions": len(keys), + } + # -- Provider spend ---------------------------------------------------- def get_provider_spend(self, provider: str) -> ProviderSpend: diff --git a/airlock/fast/guardian.py b/airlock/fast/guardian.py index 5d60ef6..093224c 100644 --- a/airlock/fast/guardian.py +++ b/airlock/fast/guardian.py @@ -33,8 +33,12 @@ from litellm.types.guardrails import GuardrailEventHooks from airlock.callbacks.enterprise_logger import write_precall_block_record +from airlock.embedding import validate_embedding_options from airlock.proxy_errors import ( AirlockAdmissionShed, + AirlockEndpointNotSupported, + AirlockDeepSeekToolNotSupported, + AirlockGatewayRoutingOverride, AirlockModelNotFound, AirlockProviderBlocked, sanitize_reason, @@ -49,7 +53,12 @@ extract_airlock_client_from_request, ) from airlock.gemini_interface import apply_gemini_request_semantics -from airlock.text_extract import extract_text, is_batch_call, is_mcp_call +from airlock.text_extract import ( + extract_text, + is_batch_call, + is_embedding_call, + is_mcp_call, +) from . import admission as _admission_mod from .circuit_breaker import check_model_with_filters @@ -61,6 +70,8 @@ logger = logging.getLogger("airlock.fast.guardian") +_OPENROUTER_ROUTING_FIELDS = frozenset({"route", "models", "transforms"}) + # --------------------------------------------------------------------------- # Helpers @@ -93,6 +104,35 @@ def _is_client_pinned(original_model: str, data: dict[str, Any]) -> bool: return True +def reject_openrouter_routing_overrides(data: dict[str, Any]) -> None: + """Keep OpenRouter upstream-routing policy with the operator. + + LiteLLM recognizes these request fields and forwards them to OpenRouter. + They would let a client override the reviewed alias/account routing policy, + so reject them before provider dispatch. Other ``extra_body`` contents stay + outside this narrow release contract unless they contain these controls. + """ + for field in _OPENROUTER_ROUTING_FIELDS: + if field in data: + raise AirlockGatewayRoutingOverride(field) + extra_body = data.get("extra_body") + if isinstance(extra_body, dict): + for field in _OPENROUTER_ROUTING_FIELDS: + if field in extra_body: + raise AirlockGatewayRoutingOverride(f"extra_body.{field}") + + +def validate_deepseek_tools(data: dict[str, Any]) -> None: + """Reject tools LiteLLM would otherwise silently discard for DeepSeek.""" + tools = data.get("tools") + if tools is None: + return + if not isinstance(tools, list) or any( + not isinstance(tool, dict) or tool.get("type") != "function" for tool in tools + ): + raise AirlockDeepSeekToolNotSupported() + + def _set_model_override( data: dict[str, Any], requested_model: str, final_model: str, reason: str ) -> None: @@ -267,7 +307,10 @@ async def async_pre_call_hook( model_name = requested_model mcp = is_mcp_call(data, call_type) batch = is_batch_call(data, call_type) - pinned_model = _is_client_pinned(requested_model, data) + embedding = is_embedding_call(data, call_type) + # Embeddings use one reviewed alias and must never be changed by client + # routing preferences, circuit failover, or LiteLLM fallback retries. + pinned_model = embedding or _is_client_pinned(requested_model, data) if pinned_model and not mcp and not batch: _lock_pinned_request(data) elif not mcp and not batch: @@ -302,43 +345,55 @@ async def async_pre_call_hook( # batch/file calls (the latter carry no top-level model). if not mcp and not batch: # ---- Step 2.5a: Model alias resolution ---- - alias_resolution = alias_table.resolve_with_diagnostic(model_name) - resolved = alias_resolution.alias - if resolved is None: - suggestions = alias_table.suggest(model_name) - if isinstance(suggestions, list) and suggestions: - raise AirlockModelNotFound(model_name, suggestions) - if alias_resolution.cross_tier is not None: - cross_tier = alias_resolution.cross_tier - logger.warning( - "event=fuzzy_match_rejected requested=%s served=%s " - "suggested=%s score=%.3f from_tier=%s to_tier=%s client_id=%s", - model_name, - cross_tier.served, - cross_tier.suggested, - cross_tier.score, - cross_tier.from_tier, - cross_tier.to_tier, - client_id, - ) - suggestions = alias_table.suggest(model_name) - suggestions.sort( - key=lambda item: ( - 0 if item.get("model") == cross_tier.suggested else 1 + if embedding: + resolved = alias_table.configured_alias(model_name) + if resolved is None: + raise AirlockEndpointNotSupported(model_name, "embeddings") + if not alias_table.supports_endpoint(resolved, "embeddings"): + raise AirlockEndpointNotSupported(resolved, "embeddings") + validate_embedding_options(data) + data["model"] = resolved + model_name = resolved + else: + alias_resolution = alias_table.resolve_with_diagnostic(model_name) + resolved = alias_resolution.alias + if resolved is None: + suggestions = alias_table.suggest(model_name) + if isinstance(suggestions, list) and suggestions: + raise AirlockModelNotFound(model_name, suggestions) + if alias_resolution.cross_tier is not None: + cross_tier = alias_resolution.cross_tier + logger.warning( + "event=fuzzy_match_rejected requested=%s served=%s " + "suggested=%s score=%.3f from_tier=%s to_tier=%s client_id=%s", + model_name, + cross_tier.served, + cross_tier.suggested, + cross_tier.score, + cross_tier.from_tier, + cross_tier.to_tier, + client_id, ) - ) - if not suggestions: - suggestions = [ - { - "model": cross_tier.suggested, - "score": cross_tier.score, - "tier": cross_tier.to_tier, - } - ] - raise AirlockModelNotFound( - model_name, suggestions, reason="fuzzy_match_crosses_cost_tier" - ) - if resolved and resolved != model_name: + suggestions = alias_table.suggest(model_name) + suggestions.sort( + key=lambda item: ( + 0 if item.get("model") == cross_tier.suggested else 1 + ) + ) + if not suggestions: + suggestions = [ + { + "model": cross_tier.suggested, + "score": cross_tier.score, + "tier": cross_tier.to_tier, + } + ] + raise AirlockModelNotFound( + model_name, + suggestions, + reason="fuzzy_match_crosses_cost_tier", + ) + if not embedding and resolved and resolved != model_name: logger.info( "model_alias %s -> %s", model_name, @@ -430,7 +485,7 @@ async def async_pre_call_hook( scope="provider", ) else: - data = apply_routing(data) + data = apply_routing(data, client_id=client_id) model_name = data.get("model", model_name) # re-read after routing if model_name != requested_model: routing_meta = data.get("metadata", {}).get("airlock_routing", {}) @@ -516,6 +571,10 @@ async def async_pre_call_hook( # ---- Step 4: Priority scoring ---- target_provider = infer_provider(data.get("model") or model_name) + if target_provider == "openrouter": + reject_openrouter_routing_overrides(data) + elif target_provider == "deepseek": + validate_deepseek_tools(data) data = apply_gemini_request_semantics(data, provider=target_provider) # P-2 deliberately rejects known-invalid OpenAI/Azure values before # LiteLLM's drop_params can silently change client intent. Unknown @@ -549,6 +608,9 @@ async def async_pre_call_hook( "boost": priority.boost, "reasons": priority.reasons, } + store.record_client_priority( + client_id, priority.score, priority.boost, priority.reasons, now + ) metadata["airlock_request"] = { "client_id": client_id, "requested_model": requested_model, diff --git a/airlock/fast/model_alias.py b/airlock/fast/model_alias.py index 04802c8..b33ac3b 100644 --- a/airlock/fast/model_alias.py +++ b/airlock/fast/model_alias.py @@ -22,7 +22,11 @@ import yaml # type: ignore[import-untyped] -from airlock.capability import airlock_provider_for, normalize_provider_token +from airlock.capability import ( + airlock_provider_for, + endpoints_for, + normalize_provider_token, +) logger = logging.getLogger("airlock.fast.model_alias") @@ -246,6 +250,8 @@ def __init__(self) -> None: # variant keys dropped in pass 2 because ≥2 entries claimed them — these # must NOT leak into the fuzzy slow path (silent cross-provider repoint). self._ambiguous_variants: set[str] = set() + self._explicit_aliases: dict[str, str] = {} + self._endpoints_by_alias: dict[str, frozenset[str]] = {} self._loaded = False def load_from_config(self, config_path: str | Path | None = None) -> None: @@ -260,6 +266,8 @@ def load_from_config(self, config_path: str | Path | None = None) -> None: self._provider_body_alias = {} self._body_providers = {} self._ambiguous_variants = set() + self._explicit_aliases = {} + self._endpoints_by_alias = {} if config_path is None: config_path = os.getenv("AIRLOCK_CONFIG", "config.yaml") @@ -319,6 +327,8 @@ def load_from_config(self, config_path: str | Path | None = None) -> None: alias_lower = alias.lower() self._exact[alias_lower] = alias + self._explicit_aliases[alias_lower] = alias + self._endpoints_by_alias[alias_lower] = frozenset(endpoints_for(item)) explicit_keys.add(alias_lower) # Provider-aware index: served-by token (NOT the alias prefix) keyed @@ -356,6 +366,20 @@ def load_from_config(self, config_path: str | Path | None = None) -> None: self._loaded = True self._log_table() + def configured_alias(self, model_name: str) -> str | None: + """Return an explicitly configured alias, never a fuzzy/variant match.""" + if not self._loaded: + self.load_from_config() + if not isinstance(model_name, str): + return None + return self._explicit_aliases.get(model_name.lower()) + + def supports_endpoint(self, alias: str, endpoint: str) -> bool: + """Return whether an explicit alias is configured for *endpoint*.""" + if not self._loaded: + self.load_from_config() + return endpoint in self._endpoints_by_alias.get(alias.lower(), frozenset()) + def _log_table(self) -> None: """Log the routing table at INFO for startup visibility.""" if not self._entries: diff --git a/airlock/fast/monitor.py b/airlock/fast/monitor.py index af01144..1cd4677 100644 --- a/airlock/fast/monitor.py +++ b/airlock/fast/monitor.py @@ -25,6 +25,7 @@ from airlock.fast.ratelimit_headers import parse_ratelimit_headers from airlock.gemini_interface import classify_gemini_response from airlock.litellm_adapter import additional_headers, hidden_params +from airlock.provider_errors import summarize_provider_error from airlock.text_extract import is_batch_call from airlock.transparency import attribute_served_backend, get_transparency_config from litellm.exceptions import APIError, RateLimitError @@ -110,12 +111,14 @@ def _is_provider_rate_limited(exc: Exception | None) -> tuple[bool, str]: """Detect provider 429/quota exhaustion signals.""" if exc is None: return False, "" + summary = summarize_provider_error(exc) + safe_reason = summary.message() if summary is not None else "" text = str(exc).strip() lowered = text.lower() if isinstance(exc, RateLimitError): - return True, text or "provider_rate_limited" + return True, safe_reason or text or "provider_rate_limited" if isinstance(exc, APIError) and getattr(exc, "status_code", None) == 429: - return True, text or "provider_rate_limited" + return True, safe_reason or text or "provider_rate_limited" markers = ( "rate limit", "too many requests", @@ -124,8 +127,8 @@ def _is_provider_rate_limited(exc: Exception | None) -> tuple[bool, str]: "quota", ) if any(marker in lowered for marker in markers): - return True, text or "provider_rate_limited" - return False, text + return True, safe_reason or text or "provider_rate_limited" + return False, safe_reason or text class AirlockFastMonitor(CustomLogger): diff --git a/airlock/fast/router.py b/airlock/fast/router.py index 489357c..ddada8b 100644 --- a/airlock/fast/router.py +++ b/airlock/fast/router.py @@ -522,7 +522,7 @@ def _apply_budget_awareness( # --------------------------------------------------------------------------- # Main entry point # --------------------------------------------------------------------------- -def apply_routing(data: dict) -> dict: +def apply_routing(data: dict, *, client_id: str = "") -> dict: """Apply intelligent routing directives from metadata.airlock. Called from guardian.py between threat assessment and circuit breaker. @@ -589,7 +589,7 @@ def apply_routing(data: dict) -> dict: # ---- 1. Session affinity ---- session_ttl = _load_session_ttl() if session_id: - existing = store.get_session(session_id) + existing = store.get_session(session_id, client_id) if existing and (time.time() - existing.last_used) < session_ttl: # Active session — pin to recorded model model = existing.model @@ -617,7 +617,7 @@ def apply_routing(data: dict) -> dict: reasons.append(reason) # Pin the resolved model - store.set_session(session_id, model) + store.set_session(session_id, model, client_id) reasons.append(f"session_new({model})") else: # No session — apply directives in order @@ -665,8 +665,6 @@ def apply_routing(data: dict) -> dict: "reasons": reasons, } ) - if session_id: - routing_meta["session_id"] = session_id if cost_tier: routing_meta["cost_tier"] = cost_tier diff --git a/airlock/guardrails/extract.py b/airlock/guardrails/extract.py index dc8c6b4..4588dfc 100644 --- a/airlock/guardrails/extract.py +++ b/airlock/guardrails/extract.py @@ -16,6 +16,7 @@ extract_text_from_mcp, extract_text_from_messages, is_batch_call, + is_embedding_call, is_mcp_call, ) @@ -24,5 +25,6 @@ "extract_text_from_mcp", "extract_text_from_messages", "is_batch_call", + "is_embedding_call", "is_mcp_call", ] diff --git a/airlock/guardrails/pii_guard.py b/airlock/guardrails/pii_guard.py index 11a8641..ab1604c 100644 --- a/airlock/guardrails/pii_guard.py +++ b/airlock/guardrails/pii_guard.py @@ -35,7 +35,7 @@ from litellm.integrations.custom_guardrail import CustomGuardrail from litellm.types.guardrails import GuardrailEventHooks -from airlock.text_extract import refresh_text_cache +from airlock.text_extract import is_embedding_call, refresh_text_cache from airlock.transparency import record_redaction from . import _env_flag @@ -151,7 +151,9 @@ def _handle_pii_unavailable(data: dict, exc: Exception) -> dict: } logger.warning("pii_unavailable mode=%s reason=%s", mode, type(exc).__name__) if mode == "closed": - raise ValueError("PII redaction is unavailable; request blocked by policy") from exc + raise ValueError( + "PII redaction is unavailable; request blocked by policy" + ) from exc return data @@ -246,6 +248,24 @@ def _scrub_messages( return cleaned +def _scrub_embedding_input( + value: Any, + mapping: dict[str, str], + counters: dict[str, int], + depth: int = 0, +) -> Any: + """Redact textual embedding input without changing its batch structure.""" + if depth >= 20: + return value + if isinstance(value, str): + return _scrub_text_with_mapping(value, mapping, counters) + if isinstance(value, list): + return [ + _scrub_embedding_input(item, mapping, counters, depth + 1) for item in value + ] + return value + + class AirlockPIIGuard(CustomGuardrail): """Pre-call PII redaction with post-call hydration of tool-call arguments. @@ -281,8 +301,10 @@ async def async_pre_call_hook( counters: dict[str, int] = {} redacted_messages = None redacted_mcp_arguments = None + redacted_embedding_input = None mcp_redaction_count = 0 message_redaction_count = 0 + embedding_redaction_count = 0 try: if is_mcp_call(data, call_type) and data.get("mcp_arguments") is not None: # Scrub a copy. If Presidio fails, fail-open must leave the @@ -295,17 +317,32 @@ async def async_pre_call_hook( redacted_mcp_arguments = working_data["mcp_arguments"] mcp_redaction_count = len(mapping) + if is_embedding_call(data, call_type) and "input" in data: + redacted_embedding_input = await asyncio.to_thread( + _scrub_embedding_input, + copy.deepcopy(data["input"]), + mapping, + counters, + ) + embedding_redaction_count = len(mapping) - mcp_redaction_count + messages = data.get("messages") if messages: redacted_messages = await asyncio.to_thread( _scrub_messages, messages, mapping, counters ) - message_redaction_count = len(mapping) - mcp_redaction_count + message_redaction_count = ( + len(mapping) - mcp_redaction_count - embedding_redaction_count + ) except Exception as exc: return _handle_pii_unavailable(data, exc) handle = None - if mapping and not data.get("stream"): + if ( + mapping + and not data.get("stream") + and not is_embedding_call(data, call_type) + ): handle = _pii_map_store.put(mapping) if handle is None: return _handle_pii_unavailable( @@ -323,6 +360,17 @@ async def async_pre_call_hook( stage="pre_call", source="pii_guard.mcp", ) + if redacted_embedding_input is not None: + data["input"] = redacted_embedding_input + if embedding_redaction_count: + record_redaction( + data.setdefault("metadata", {}), + field="input", + count=embedding_redaction_count, + category="pii", + stage="pre_call", + source="pii_guard.embedding", + ) if redacted_messages is not None: data["messages"] = redacted_messages if message_redaction_count: diff --git a/airlock/guardrails/pii_mapping.py b/airlock/guardrails/pii_mapping.py index 256bc0f..8e11266 100644 --- a/airlock/guardrails/pii_mapping.py +++ b/airlock/guardrails/pii_mapping.py @@ -65,9 +65,7 @@ def __len__(self) -> int: def _sweep_locked(self, now: float) -> int: expired = [ - handle - for handle, entry in self._entries.items() - if entry.expires_at <= now + handle for handle, entry in self._entries.items() if entry.expires_at <= now ] for handle in expired: del self._entries[handle] diff --git a/airlock/models_catalog.py b/airlock/models_catalog.py index bda3024..d8e8866 100644 --- a/airlock/models_catalog.py +++ b/airlock/models_catalog.py @@ -17,6 +17,7 @@ from dataclasses import dataclass from pathlib import Path from typing import Callable +from urllib.parse import urlsplit, urlunsplit import yaml @@ -81,6 +82,65 @@ class _ProviderFetcher: api_key_override: str | None = None # pre-resolved key for custom providers +class _NoRedirect(urllib.request.HTTPRedirectHandler): + """Refuse redirects so a discovery credential stays bound to its base.""" + + def redirect_request(self, req, fp, code, msg, headers, newurl): # noqa: N802 + return None + + +def _models_url_from_configured_base(base_url: object) -> str | None: + """Normalize a reviewed API base into its same-origin models endpoint. + + Discovery is intentionally more restrictive than model transport. A key is + sent only to an HTTPS base with no URL credentials, query, fragment, or + pre-existing `/models` endpoint. API roots and `/chat/completions` are + accepted; the latter is removed before `/models` is appended. + """ + if not isinstance(base_url, str) or not base_url: + return None + parsed = urlsplit(base_url) + if ( + parsed.scheme != "https" + or not parsed.hostname + or parsed.username is not None + or parsed.password is not None + or parsed.query + or parsed.fragment + ): + return None + path = parsed.path.rstrip("/") + if path.endswith("/models"): + return None + if path.endswith("/chat/completions"): + path = path[: -len("/chat/completions")] + return urlunsplit(("https", parsed.netloc, f"{path}/models", "", "")) + + +def _configured_provider_discovery( + config: dict, provider_prefix: str +) -> tuple[str, str] | None: + """Return one `(models_url, key)` only for a coherent configured provider.""" + candidates: set[str] = set() + api_key: str | None = None + for entry in config.get("model_list", []) or []: + if not isinstance(entry, dict): + continue + params = entry.get("litellm_params") or {} + if not isinstance(params, dict) or not str(params.get("model", "")).startswith( + f"{provider_prefix}/" + ): + continue + models_url = _models_url_from_configured_base(params.get("api_base")) + if models_url is None: + return None + candidates.add(models_url) + api_key = api_key or _resolve_secret(params.get("api_key")) + if len(candidates) != 1 or not api_key: + return None + return candidates.pop(), api_key + + def _fetch_openai_compatible( base_url: str, provider_prefix: str, @@ -89,6 +149,7 @@ def _fetch_openai_compatible( auth_header: str = "Authorization", auth_scheme: str = "Bearer", extra_headers: dict | None = None, + no_redirect: bool = False, ) -> list[dict]: """Fetch models from any OpenAI-compatible /v1/models endpoint.""" headers: dict[str, str] = { @@ -99,7 +160,14 @@ def _fetch_openai_compatible( req = urllib.request.Request(base_url, headers=headers) ctx = ssl.create_default_context() - with urllib.request.urlopen(req, timeout=timeout, context=ctx) as resp: # noqa: S310 + if no_redirect: + opener = urllib.request.build_opener( + _NoRedirect(), urllib.request.HTTPSHandler(context=ctx) + ) + response = opener.open(req, timeout=timeout) # noqa: S310 + else: + response = urllib.request.urlopen(req, timeout=timeout, context=ctx) # noqa: S310 + with response as resp: data = json.loads(resp.read()) models = [] @@ -193,6 +261,42 @@ def _fetch_perplexity_models(api_key: str, timeout: float) -> list[dict]: _ProviderFetcher("perplexity", _fetch_perplexity_models), ] +# These provider names are intentionally configured only through reviewed +# ``model_list`` entries. A generic ``providers:`` entry lacks the base-bound, +# HTTPS, and no-redirect contract enforced by `_configured_provider_discovery`. +_MODEL_LIST_DISCOVERY_PREFIXES = frozenset({"openrouter", "deepseek"}) + + +def _provider_fetchers_from_model_list(config: dict) -> list[_ProviderFetcher]: + """Build safe discovery fetchers for providers configured in model_list.""" + fetchers: list[_ProviderFetcher] = [] + for prefix in ("openrouter", "deepseek"): + configured = _configured_provider_discovery(config, prefix) + if configured is None: + continue + models_url, api_key = configured + + def _fetch( + key: str, timeout: float, *, url: str = models_url, name: str = prefix + ) -> list[dict]: + return _fetch_openai_compatible( + url, + provider_prefix=name, + api_key=key, + timeout=timeout, + no_redirect=True, + ) + + fetchers.append(_ProviderFetcher(prefix, _fetch, api_key_override=api_key)) + return fetchers + + +def _safe_discovery_failure(exc: Exception) -> str: + """Return useful diagnostics without serializing provider exception text.""" + status = getattr(exc, "code", None) or getattr(exc, "status_code", None) + suffix = f" status={status}" if isinstance(status, int) else "" + return f"error_type={type(exc).__name__}{suffix}" + def _custom_fetchers_from_config(config: dict) -> list[_ProviderFetcher]: """Build fetchers for any ``providers:`` entries in config.yaml. @@ -219,6 +323,12 @@ def _custom_fetchers_from_config(config: dict) -> list[_ProviderFetcher]: "models_catalog: skipping providers entry missing name/base_url" ) continue + if name in _MODEL_LIST_DISCOVERY_PREFIXES: + logger.warning( + "models_catalog: providers entry %r is model_list-only, skipping", + name, + ) + continue api_key = _resolve_secret(entry.get("api_key", "")) if not api_key: logger.warning( @@ -281,9 +391,15 @@ def fetch_live_provider_models( "models_catalog: custom providers override built-ins: %s", ", ".join(sorted(overridden)), ) - active: list[_ProviderFetcher] = [ - f for f in _FETCHERS if f.prefix not in custom_prefixes - ] + custom + active: list[_ProviderFetcher] = ( + [f for f in _FETCHERS if f.prefix not in custom_prefixes] + + [ + f + for f in _provider_fetchers_from_model_list(config) + if f.prefix not in custom_prefixes + ] + + custom + ) results: list[dict] = [] lock = threading.Lock() @@ -301,14 +417,11 @@ def _run(fetcher: _ProviderFetcher) -> None: len(entries), fetcher.prefix, ) - except ( - urllib.error.URLError, - urllib.error.HTTPError, - OSError, - Exception, - ) as exc: + except Exception as exc: # discovery must never block proxy startup logger.warning( - "models_catalog: %s model discovery failed: %s", fetcher.prefix, exc + "models_catalog: %s model discovery failed: %s", + fetcher.prefix, + _safe_discovery_failure(exc), ) threads = [threading.Thread(target=_run, args=(f,), daemon=True) for f in active] diff --git a/airlock/operational_reads.py b/airlock/operational_reads.py new file mode 100644 index 0000000..307bc80 --- /dev/null +++ b/airlock/operational_reads.py @@ -0,0 +1,136 @@ +"""Explicit, bounded operational-read backend selection. + +FathomDB is a write/analysis store unless an operator deliberately selects it +for reads. This module centralizes the selection and the truthful fallback +shape, so consumers cannot silently change their source merely because a local +engine happens to be open. +""" + +from __future__ import annotations + +import os +from dataclasses import dataclass +from datetime import datetime, timedelta, timezone +from pathlib import Path +from typing import Any, Callable + +from airlock.api.queries import DATASTORE_QUERY_LIMIT, get_request_logs, node_properties +from airlock.log_query import LogQuery, query_logs + +_BACKEND_ENV = "AIRLOCK_OPERATIONAL_READ_BACKEND" +_FATHOMDB = "fathomdb" +_JSONL = "jsonl" + + +@dataclass(frozen=True) +class OperationalReadPage: + records: list[dict[str, Any]] + source: str + degraded_reason: str | None + truncated: bool + limit_hit: str | None + + +def _requested_backend() -> tuple[str, str | None]: + raw = (os.getenv(_BACKEND_ENV) or _JSONL).strip().lower() + if raw in {_JSONL, _FATHOMDB}: + return raw, None + return _JSONL, f"invalid operational backend {raw!r}; using bounded JSONL" + + +def selected_fathom_engine() -> tuple[Any | None, str | None]: + """Return the explicitly selected engine, never opening it by default.""" + requested, invalid_reason = _requested_backend() + if requested != _FATHOMDB: + return None, invalid_reason + try: + import airlock.datastore + + engine = airlock.datastore.get_engine() + except Exception: + engine = None + if engine is None: + return None, "FathomDB selected but unavailable; using bounded JSONL" + return engine, None + + +def fathomdb_selected() -> bool: + """Whether the operator selected FathomDB without opening an engine.""" + return _requested_backend()[0] == _FATHOMDB + + +def _within_days(record: dict[str, Any], days: int) -> bool: + timestamp = record.get("timestamp") + if not isinstance(timestamp, str): + return False + try: + parsed = datetime.fromisoformat(timestamp.replace("Z", "+00:00")) + except ValueError: + return False + if parsed.tzinfo is None: + return False + return parsed >= datetime.now(timezone.utc) - timedelta(days=max(1, days)) + + +def read_records( + *, + directory: Path | str, + days: int, + predicate: Callable[[dict[str, Any]], bool] | None = None, + limit: int = DATASTORE_QUERY_LIMIT, + allow_fathom: bool = True, +) -> OperationalReadPage: + """Read selected operational history with explicit source and degradation. + + ``limit`` bounds both source paths. The FathomDB API currently lists then + filters Airlock records client-side, so filling its list limit is reported + as partial even if later predicates select fewer records. + """ + limit = max(1, min(limit, DATASTORE_QUERY_LIMIT)) + if allow_fathom: + engine, fallback_reason = selected_fathom_engine() + else: + engine = None + fallback_reason = ( + "FathomDB selected but proxy operational reads unavailable; using bounded JSONL" + if fathomdb_selected() + else _requested_backend()[1] + ) + if engine is not None: + try: + nodes = get_request_logs(engine, limit=limit) + except Exception: + nodes = None + fallback_reason = "FathomDB selected but unavailable; using bounded JSONL" + if nodes is not None: + records = [node_properties(node) for node in nodes] + records = [record for record in records if _within_days(record, days)] + if predicate is not None: + records = [record for record in records if predicate(record)] + return OperationalReadPage( + records=records[:limit], + source=_FATHOMDB, + degraded_reason=None, + truncated=len(nodes) >= limit, + limit_hit="datastore_limit" if len(nodes) >= limit else None, + ) + + page = query_logs( + LogQuery( + days=days, + max_records=limit, + predicate=predicate, + directory=Path(directory), + ) + ) + # JSONL is an operator-facing file boundary. LogQuery preserves valid + # JSON scalars for generic callers, but this operational-record contract + # must never hand a scalar to a history consumer. + records = [record for record in page.records if isinstance(record, dict)] + return OperationalReadPage( + records=records, + source=_JSONL, + degraded_reason=fallback_reason, + truncated=page.truncated, + limit_hit=page.limit_hit, + ) diff --git a/airlock/provider_errors.py b/airlock/provider_errors.py new file mode 100644 index 0000000..edbf2a4 --- /dev/null +++ b/airlock/provider_errors.py @@ -0,0 +1,83 @@ +"""Bounded representations of upstream provider failures. + +Provider exceptions commonly include upstream response bodies. Those bodies can +contain credentials, request details, or provider-specific diagnostic text, so +they must not cross Airlock's event, state, logging, or tracing boundaries. +""" + +from __future__ import annotations + +from dataclasses import dataclass + + +def _status_code(exc: Exception) -> int | None: + """Extract an HTTP status without consulting the exception message.""" + for value in ( + getattr(exc, "status_code", None), + getattr(exc, "code", None), + getattr(getattr(exc, "response", None), "status_code", None), + ): + if isinstance(value, int) and 100 <= value <= 599: + return value + return None + + +_KNOWN_PROVIDERS = frozenset( + { + "anthropic", + "azure", + "bedrock", + "deepseek", + "gemini", + "mistral", + "openai", + "openrouter", + "perplexity", + "vertex_ai", + } +) + + +def _provider_name(value: object) -> str | None: + """Keep provider attribution bounded even for malformed third-party errors.""" + if not isinstance(value, str): + return None + normalized = value.lower() + return normalized if normalized in _KNOWN_PROVIDERS else None + + +@dataclass(frozen=True) +class ProviderErrorSummary: + """The only provider-failure detail permitted outside the call boundary.""" + + provider: str | None + error_type: str + status_code: int | None + + def message(self) -> str: + parts = ["provider_error", f"type={self.error_type}"] + if self.provider: + parts.append(f"provider={self.provider}") + if self.status_code is not None: + parts.append(f"status={self.status_code}") + return " ".join(parts) + + +def summarize_provider_error(exc: Exception | None) -> ProviderErrorSummary | None: + """Return a safe summary only for a LiteLLM-attributed provider error. + + Airlock deliberately leaves local validation/evaluation exceptions unchanged; + only errors carrying LiteLLM's ``llm_provider`` marker crossed a provider + boundary and require redaction here. + """ + if exc is None: + return None + raw_provider = getattr(exc, "llm_provider", None) + if not isinstance(raw_provider, str): + return None + provider = _provider_name(raw_provider) + return ProviderErrorSummary( + provider=provider, + error_type=type(exc).__name__, + status_code=_status_code(exc), + ) diff --git a/airlock/proxy_bootstrap.py b/airlock/proxy_bootstrap.py index c1fa7ef..999081d 100644 --- a/airlock/proxy_bootstrap.py +++ b/airlock/proxy_bootstrap.py @@ -16,6 +16,7 @@ from airlock.admin.http import install_admin_on_proxy_app from airlock.batch.middleware import install_batch_gateway_on_proxy_app from airlock.docs import install_airlock_docs_on_proxy_app +from airlock.embedding_boundary import install_embedding_request_boundary_on_proxy_app from airlock.health import install_circuit_health_on_proxy_app from airlock.models_seam import install_models_capability_seam_on_proxy_app from airlock.proxy_errors import install_airlock_error_handlers_on_proxy_app @@ -26,6 +27,7 @@ def bootstrap_airlock_proxy() -> None: install_airlock_docs_on_proxy_app() install_circuit_health_on_proxy_app() install_airlock_error_handlers_on_proxy_app() + install_embedding_request_boundary_on_proxy_app() # Admin perimeter mounts BEFORE the batch gateway so the gateway stays the # outermost ASGI layer (umbrella §3 mount order). install_admin_on_proxy_app() diff --git a/airlock/proxy_errors.py b/airlock/proxy_errors.py index a3b0ba2..5144796 100644 --- a/airlock/proxy_errors.py +++ b/airlock/proxy_errors.py @@ -16,6 +16,7 @@ from typing import Any from litellm import RateLimitError +from litellm.proxy._types import ProxyException from airlock.litellm_adapter import resolve_proxy_app from airlock.transparency import model_suggestion_header @@ -100,6 +101,71 @@ def __init__(self, requested: str, model: str, supported: frozenset[str]) -> Non ) +class AirlockEndpointNotSupported(ProxyException): + """A configured alias cannot be used on the requested API endpoint.""" + + def __init__(self, model: str, endpoint: str) -> None: + self.model = model + self.endpoint = endpoint + self._airlock_error_code = "model_endpoint_not_supported" + super().__init__( + message=f"Model {model!r} is not configured for the {endpoint!r} endpoint.", + type="invalid_request_error", + param="model", + code=400, + openai_code=self._airlock_error_code, + ) + + def to_dict(self) -> dict: + body = super().to_dict() + body["code"] = self._airlock_error_code + body["airlock"] = {"endpoint": self.endpoint, "model": self.model} + return body + + +class AirlockGatewayRoutingOverride(ProxyException): + """A client tried to override an operator-owned gateway routing policy.""" + + def __init__(self, option: str) -> None: + self.option = option + self._airlock_error_code = "gateway_routing_override_not_allowed" + super().__init__( + message=( + f"OpenRouter routing option {option!r} is operator-controlled and " + "cannot be set by a client request." + ), + type="invalid_request_error", + param=option, + code=400, + openai_code=self._airlock_error_code, + ) + + def to_dict(self) -> dict: + body = super().to_dict() + body["code"] = self._airlock_error_code + return body + + +class AirlockDeepSeekToolNotSupported(ProxyException): + """A DeepSeek request contains a non-function tool LiteLLM would drop.""" + + def __init__(self, option: str = "tools") -> None: + self.option = option + self._airlock_error_code = "deepseek_non_function_tool_not_supported" + super().__init__( + message="DeepSeek supports OpenAI function tools only.", + type="invalid_request_error", + param=option, + code=400, + openai_code=self._airlock_error_code, + ) + + def to_dict(self) -> dict: + body = super().to_dict() + body["code"] = self._airlock_error_code + return body + + class AirlockAdmissionShed(RateLimitError): """A local admission-gate rejection with an honest retry time.""" @@ -219,6 +285,24 @@ def invalid_reasoning_effort_response_payload( return body, {} +def endpoint_not_supported_response_payload( + exc: AirlockEndpointNotSupported, +) -> tuple[dict, dict]: + """Build an OpenAI-compatible 400 for an unsupported model endpoint.""" + return ( + { + "error": { + "message": sanitize_reason(str(exc), 500), + "type": "invalid_request_error", + "code": "model_endpoint_not_supported", + "param": "model", + "airlock": {"endpoint": exc.endpoint, "model": exc.model}, + } + }, + {}, + ) + + async def airlock_invalid_reasoning_effort_handler(request: Any, exc: Exception): """FastAPI exception handler → an OpenAI-compatible 400.""" from fastapi.responses import JSONResponse @@ -228,6 +312,15 @@ async def airlock_invalid_reasoning_effort_handler(request: Any, exc: Exception) return JSONResponse(status_code=400, content=body, headers=headers) +async def airlock_endpoint_not_supported_handler(request: Any, exc: Exception): + """FastAPI exception handler → an OpenAI-compatible 400.""" + from fastapi.responses import JSONResponse + + assert isinstance(exc, AirlockEndpointNotSupported) + body, headers = endpoint_not_supported_response_payload(exc) + return JSONResponse(status_code=400, content=body, headers=headers) + + async def airlock_admission_shed_handler(request: Any, exc: Exception): """FastAPI exception handler → local admission 429 with Retry-After.""" from fastapi.responses import JSONResponse @@ -267,6 +360,13 @@ def install_airlock_error_handlers_on_proxy_app() -> bool: AirlockInvalidReasoningEffort, airlock_invalid_reasoning_effort_handler ) app.state.airlock_invalid_reasoning_effort_handler_installed = True + if not getattr( + app.state, "airlock_endpoint_not_supported_handler_installed", False + ): + app.add_exception_handler( + AirlockEndpointNotSupported, airlock_endpoint_not_supported_handler + ) + app.state.airlock_endpoint_not_supported_handler_installed = True if not getattr(app.state, "airlock_admission_shed_handler_installed", False): app.add_exception_handler(AirlockAdmissionShed, airlock_admission_shed_handler) app.state.airlock_admission_shed_handler_installed = True diff --git a/airlock/telemetry_health.py b/airlock/telemetry_health.py new file mode 100644 index 0000000..5cc01e7 --- /dev/null +++ b/airlock/telemetry_health.py @@ -0,0 +1,108 @@ +"""Bounded, secret-safe instrumentation status for operator diagnostics. + +This is deliberately not an exporter implementation. It distinguishes a +Prometheus registry (pull, so no downstream delivery can be claimed) from an +actual tracing exporter. Callback instrumentation is fail-open. +""" + +from __future__ import annotations + +import os +import threading +import time +from urllib.parse import urlsplit + +_LOCK = threading.RLock() +_MAX_ERROR = 80 +_health: dict[str, dict] = {} + + +def _safe_endpoint(value: str | None) -> str | None: + """Keep only scheme and authority; never publish credential/query/path.""" + if not value: + return None + try: + parsed = urlsplit(value) + except ValueError: + return None + if parsed.scheme not in {"http", "https"} or not parsed.hostname: + return None + host = parsed.hostname + try: + port = parsed.port + except ValueError: + return None + if port: + host = f"{host}:{port}" + return f"{parsed.scheme}://{host}" + + +def configure_exporter( + name: str, *, enabled: bool, endpoint: str | None = None +) -> None: + """Declare an instrumentation/exporter state without treating it as delivery.""" + with _LOCK: + item = _health.setdefault(name, {}) + item.update( + { + "enabled": bool(enabled), + "endpoint": _safe_endpoint(endpoint), + "updated_at": time.time(), + } + ) + item.setdefault("successes", 0) + item.setdefault("failures", 0) + item.setdefault("last_success_at", None) + item.setdefault("last_error", None) + + +def record_signal(name: str) -> None: + """Record local instrumentation success; only exporters may claim delivery.""" + with _LOCK: + item = _health.setdefault(name, {}) + item["signals"] = int(item.get("signals", 0)) + 1 + item["updated_at"] = time.time() + + +def record_export_success(name: str) -> None: + with _LOCK: + item = _health.setdefault(name, {}) + item["successes"] = int(item.get("successes", 0)) + 1 + item["last_success_at"] = time.time() + item["updated_at"] = time.time() + + +def record_export_failure(name: str, error: Exception | str) -> None: + """Retain only a bounded exception category, never provider response text.""" + # A string commonly is ``str(exc)`` and can contain an upstream body, + # credentials, or prompt content. Treat it as an untrusted unknown class, + # never as a displayable message. + category = "export_error" if isinstance(error, str) else type(error).__name__ + category = str(category).replace("\n", " ")[:_MAX_ERROR] + with _LOCK: + item = _health.setdefault(name, {}) + item["failures"] = int(item.get("failures", 0)) + 1 + item["last_error"] = category + item["updated_at"] = time.time() + + +def configure_from_environment( + *, metrics_available: bool, tracing_available: bool +) -> None: + """Describe built-in telemetry without inventing an external exporter.""" + configure_exporter( + "prometheus", + enabled=metrics_available, + endpoint="http://in-process/metrics" if metrics_available else None, + ) + configure_exporter( + "otlp", + enabled=tracing_available, + endpoint=os.getenv("OTEL_EXPORTER_OTLP_ENDPOINT"), + ) + + +def telemetry_snapshot() -> dict[str, dict]: + """Return a copy suitable for an authenticated operator read.""" + with _LOCK: + return {name: dict(value) for name, value in _health.items()} diff --git a/airlock/text_extract.py b/airlock/text_extract.py index 4bb561d..96298d0 100644 --- a/airlock/text_extract.py +++ b/airlock/text_extract.py @@ -164,6 +164,21 @@ def is_mcp_call(data: dict, call_type: str = "") -> bool: return "mcp_tool_name" in data +_EMBEDDING_CALL_TYPES = frozenset({"embedding", "aembedding"}) + + +def is_embedding_call(data: dict, call_type: str = "") -> bool: + """Return whether this is a LiteLLM embedding call. + + ``call_type`` is authoritative when provided. The input-shaped fallback is + deliberately limited to requests without chat messages, so a client cannot + label a chat request as embeddings merely by adding an ``input`` field. + """ + if call_type: + return call_type in _EMBEDDING_CALL_TYPES + return "input" in data and "messages" not in data + + # LiteLLM call_types for batch/file routes. These carry no top-level model and # no messages, so model-specific guardrail logic must be skipped for them. _BATCH_CALL_TYPES = frozenset( @@ -219,6 +234,8 @@ def is_batch_call(data: dict, call_type: str = "") -> bool: def _compute_text(data: dict, call_type: str) -> str: if is_mcp_call(data, call_type): return extract_text_from_mcp(data) + if is_embedding_call(data, call_type): + return "\n".join(_collect_strings(data.get("input"))) return extract_text_from_messages(data.get("messages", [])) diff --git a/airlock/transparency.py b/airlock/transparency.py index a4b2ff8..d290b31 100644 --- a/airlock/transparency.py +++ b/airlock/transparency.py @@ -56,8 +56,14 @@ class ServedBackend: # these scalar/enum fields. Everything else renders as `field=` — never content. HEADER_VALUE_FIELDS = {"model", "reasoning_effort", "fallbacks", "num_retries"} -_GATEWAY_PROVIDERS = {"bedrock", "azure", "vertex_ai", "vertex_ai_beta"} -_NATIVE_PROVIDERS = {"anthropic", "openai", "gemini"} +_GATEWAY_PROVIDERS = { + "bedrock", + "azure", + "vertex_ai", + "vertex_ai_beta", + "openrouter", +} +_NATIVE_PROVIDERS = {"anthropic", "openai", "gemini", "deepseek"} # litellm's streaming wrapper hardcodes custom_llm_provider="vertex_ai_beta" for the # Vertex-Gemini handler — even for native AI-Studio gemini calls. Normalize the alias @@ -151,7 +157,8 @@ def classify_backend_kind( ) -> Literal["native", "gateway", "unknown"]: """Map a provider name to its backend kind (pure, no I/O). - gateway: bedrock/azure/vertex_ai ; native: anthropic/openai/gemini ; else unknown. + gateway: bedrock/azure/vertex_ai/openrouter; native: + anthropic/openai/gemini/deepseek; else unknown. """ if provider in _GATEWAY_PROVIDERS: return "gateway" diff --git a/airlock/tui/admin_client.py b/airlock/tui/admin_client.py index dc733ee..b77eebb 100644 --- a/airlock/tui/admin_client.py +++ b/airlock/tui/admin_client.py @@ -10,6 +10,7 @@ from __future__ import annotations import json +import base64 import os import ssl import urllib.error @@ -99,6 +100,57 @@ def provider_snapshot(host: str, port: str) -> dict | None: ) +def session_snapshot(host: str, port: str) -> dict | None: + """Return the bounded live affinity view, or None when admin is unavailable.""" + status, payload = admin_get(host, port, "/airlock/admin/sessions") + return ( + payload if status == 200 and isinstance(payload.get("sessions"), list) else None + ) + + +def client_snapshot(host: str, port: str) -> dict | None: + status, payload = admin_get(host, port, "/airlock/admin/clients") + return ( + payload if status == 200 and isinstance(payload.get("clients"), dict) else None + ) + + +def telemetry_snapshot(host: str, port: str) -> dict | None: + status, payload = admin_get(host, port, "/airlock/admin/telemetry") + return ( + payload + if status == 200 and isinstance(payload.get("exporters"), dict) + else None + ) + + +def operational_records(host: str, port: str, *, days: int, limit: int) -> dict | None: + """Read history from the proxy-owned FathomDB process, when selected.""" + status, payload = admin_post( + host, + port, + "/airlock/admin/operational/records", + {"days": days, "limit": limit}, + ) + return ( + payload if status == 200 and isinstance(payload.get("records"), list) else None + ) + + +def operational_view(host: str, port: str, kind: str, body: dict) -> dict | None: + """Call a loopback-only, proxy-owned operational read view.""" + status, payload = admin_post(host, port, f"/airlock/admin/operational/{kind}", body) + return payload if status == 200 else None + + +def clear_client_sessions(host: str, port: str, client_id: str) -> tuple[int, dict]: + # Client IDs come from a header and can contain a slash. Use an opaque + # URL-safe selector so a path parser can never split the selected identity. + selector = base64.urlsafe_b64encode(client_id.encode("utf-8")).decode("ascii") + selector = selector.rstrip("=") + return admin_post(host, port, f"/airlock/admin/session-clients/{selector}/clear") + + def clear_provider_quarantine( host: str, port: str, provider: str, mode: str = "probe" ) -> tuple[int, dict]: diff --git a/airlock/tui/app.py b/airlock/tui/app.py index fd413c4..63211d8 100644 --- a/airlock/tui/app.py +++ b/airlock/tui/app.py @@ -60,12 +60,16 @@ def __init__( port: str = "4000", auto_start: bool = False, daemon_mode: bool = False, + test_harness: bool = False, ) -> None: super().__init__() self._proxy_host = host self._proxy_port = port self._auto_start = auto_start self._daemon_mode = daemon_mode + # Explicit constructor-only test mode. It composes production widgets but + # suppresses mount-time background I/O; it is never inferred from env. + self._test_harness = test_harness self._proxy_manager = ProxyManager( host=host, port=port, @@ -99,6 +103,8 @@ def compose(self) -> ComposeResult: def on_mount(self) -> None: self.sub_title = "Overview" + if self._test_harness: + return self._mcp_manager.start_health_loop() self._start_jsonl_tailer() self.set_interval(5.0, self._run_alerts) diff --git a/airlock/tui/screens/config.py b/airlock/tui/screens/config.py index 281656e..432e02b 100644 --- a/airlock/tui/screens/config.py +++ b/airlock/tui/screens/config.py @@ -385,6 +385,8 @@ def compose(self) -> ComposeResult: def on_mount(self) -> None: self._applied_values = self._current_values() + if getattr(self.app, "_test_harness", False): + return self._refresh_mcp_servers() self.set_interval(10.0, self._refresh_mcp_servers) diff --git a/airlock/tui/screens/guards.py b/airlock/tui/screens/guards.py index a72d4d7..a13a1e3 100644 --- a/airlock/tui/screens/guards.py +++ b/airlock/tui/screens/guards.py @@ -78,6 +78,7 @@ class FlowEntry: call_type: str = "" mcp_tool_name: str = "" mcp_server_name: str = "" + routing: dict[str, Any] | None = None def _parse_entry(record: dict) -> FlowEntry | None: @@ -90,6 +91,7 @@ def _parse_entry(record: dict) -> FlowEntry | None: or record.get("mutations") or record.get("airlock_enforcement") or record.get("airlock_code_inspection") + or record.get("airlock_routing") ): return None return FlowEntry( @@ -110,6 +112,11 @@ def _parse_entry(record: dict) -> FlowEntry | None: call_type=record.get("call_type", ""), mcp_tool_name=record.get("mcp_tool_name") or "", mcp_server_name=record.get("mcp_server_name") or "", + routing=( + record.get("airlock_routing") + if isinstance(record.get("airlock_routing"), dict) + else None + ), ) @@ -499,6 +506,11 @@ def compose(self) -> ComposeResult: yield Static( "Loading semantic classifier report...", id="guards-semantic" ) + with TabPane("Routing", id="guards-tab-routing"): + yield Static( + "No smart-routing classifications in the bounded request window.", + id="guards-routing", + ) with Collapsible( title="Request Stream", collapsed=False, id="guards-stream-collapsible" ): @@ -516,6 +528,8 @@ def compose(self) -> ComposeResult: yield table def on_mount(self) -> None: + if getattr(self.app, "_test_harness", False): + return self._poll_logs() self._poll_timer = self.set_interval(1.0, self._poll_if_live) # The semantic report aggregates a multi-day window, so it is far more @@ -724,9 +738,49 @@ def _show_detail(self, entry: FlowEntry) -> None: raw = self.query_one("#guards-raw", Static) tool_result = self.query_one("#guards-tool-result", Static) mutations = self.query_one("#guards-mutations", Static) + routing = self.query_one("#guards-routing", Static) signals.update(_render_signals(entry)) pipeline.update(_render_pipeline(entry)) raw.update(_render_raw(entry)) tool_result.update(_render_tool_result(entry)) mutations.update(_render_mutations(entry)) + routing.update(_render_routing(entry, self._entries)) + + +def _render_routing(entry: FlowEntry, entries: list[FlowEntry]) -> str: + """Render a bounded JSONL-derived smart-routing view without prompt text.""" + counts = {"simple": 0, "moderate": 0, "complex": 0} + for candidate in entries: + classification = (candidate.routing or {}).get("smart_classify") + if not isinstance(classification, dict): + continue + complexity = classification.get("complexity") + if complexity in counts: + counts[complexity] += 1 + + total = sum(counts.values()) + if not total: + return "[dim]No smart-routing classifications in the bounded request window.[/]" + + routing = entry.routing or {} + classification = routing.get("smart_classify") + if not isinstance(classification, dict): + classification = {} + complexity = classification.get("complexity", "-") + score = classification.get("score") + mapped_model = routing.get("routed_model") or entry.model + reasons = routing.get("reasons") + if not isinstance(reasons, list): + reasons = [] + distribution = " ".join(f"{name}: {count}" for name, count in counts.items()) + score_text = f"{float(score):.2f}" if isinstance(score, (int, float)) else "-" + reason_text = escape(", ".join(str(reason) for reason in reasons[:4])) or "-" + return ( + "[bold]Smart routing (bounded JSONL window)[/]\n" + f" Distribution: {distribution}\n" + f" Selected request: {escape(str(complexity))} (score {score_text})\n" + f" Routed model: {escape(str(mapped_model))}\n" + f" Route reasons: {reason_text}\n" + " Source: persisted request metadata; unavailable classifications are not inferred." + ) diff --git a/airlock/tui/screens/logs.py b/airlock/tui/screens/logs.py index 7e43ca3..3fc5535 100644 --- a/airlock/tui/screens/logs.py +++ b/airlock/tui/screens/logs.py @@ -24,6 +24,11 @@ TabPane, ) +from airlock.operational_reads import ( + OperationalReadPage, + fathomdb_selected, + read_records, +) from airlock.tui.widgets.safe_data_table import _SafeDataTable @@ -37,6 +42,18 @@ def _analysis_days(range_value: str) -> int: _MAX_LOG_RECORDS = 5_000 +def _operational_source_text( + source: str, degraded_reason: str | None, truncated: bool +) -> str: + """Make the selected history source and its limits visible to operators.""" + text = f"History source: {source.upper()}" + if degraded_reason: + text = f"{text} — {degraded_reason}" + if truncated: + text = f"{text} — bounded result" + return text + + def _is_blocked_request(record: dict[str, Any]) -> bool: """Recognize an Airlock policy block without conflating provider failures.""" enforcement = record.get("airlock_enforcement") or {} @@ -129,6 +146,9 @@ def __init__(self, **kwargs: Any) -> None: self._records: list[dict[str, Any]] = [] self._filtered: list[dict[str, Any]] = [] self._refresh_timer: Timer | None = None + self._read_source = "jsonl" + self._read_degraded_reason: str | None = None + self._read_truncated = False def compose(self) -> ComposeResult: with Horizontal(id="logs-filters"): @@ -183,6 +203,7 @@ def compose(self) -> ComposeResult: yield Button("Run Analysis", id="logs-run-analysis", variant="primary") yield Button("Export", id="logs-export-btn") yield Static("", id="logs-analysis-status") + yield Static("History source: JSONL", id="logs-source") with TabbedContent(id="logs-tabs"): with TabPane("Detail", id="tab-detail"): yield Static( @@ -223,6 +244,8 @@ def compose(self) -> ComposeResult: yield table def on_mount(self) -> None: + if getattr(self.app, "_test_harness", False): + return self._load_logs() def on_data_table_row_highlighted(self, event: DataTable.RowHighlighted) -> None: @@ -236,6 +259,11 @@ def on_data_table_row_highlighted(self, event: DataTable.RowHighlighted) -> None self._show_detail(self._filtered[idx]) def on_select_changed(self, event: Select.Changed) -> None: + # Textual emits initial Select.Changed messages while the production + # widget tree is composing. In the explicit test harness those must + # not schedule a JSONL worker behind the mount-time guard. + if getattr(self.app, "_test_harness", False): + return if event.select.id == "logs-refresh-mode": self._update_refresh_mode(str(event.value)) else: @@ -243,6 +271,8 @@ def on_select_changed(self, event: Select.Changed) -> None: self._reload_with_filters() def on_input_changed(self, event: Input.Changed) -> None: + if getattr(self.app, "_test_harness", False): + return if event.input.id in ( "logs-user-filter", "logs-tool-filter", @@ -291,49 +321,44 @@ def _reload_with_filters(self) -> None: @work(exclusive=True, thread=True) def _load_logs(self, filters: dict[str, str] | None = None) -> None: log_dir = Path(os.getenv("AIRLOCK_LOG_DIR", "./logs")) - records: list[dict[str, Any]] = [] - today = datetime.now(timezone.utc).date() - filters = filters or { - "model": "all", - "user": "", - "status": "all", - "type": "all", - "tool": "", - "search": "", - "range": "7d", - "start": "", - "end": "", - } - - days = 31 # range filtering below bounds the retained working set - - for i in range(days): - day = today - timedelta(days=i) - path = log_dir / f"airlock-{day.isoformat()}.jsonl" - if not path.exists(): - continue - with open(path, encoding="utf-8") as f: - for line in f: - line = line.strip() - if line: - try: - record = json.loads(line) - # JSONL is an operator-facing boundary. A valid - # JSON scalar/array is not an Airlock event and - # must not take down the loader worker. - if isinstance(record, dict) and _matches_snapshot( - record, filters - ): - records.append(record) - if len(records) >= _MAX_LOG_RECORDS: - break - except json.JSONDecodeError: - continue - if len(records) >= _MAX_LOG_RECORDS: - break - + # The current UI snapshot is read on the main thread by + # ``_apply_filters`` below. The optional argument remains part of the + # worker seam used by refresh callers. + _ = filters + + page: OperationalReadPage | None = None + if fathomdb_selected(): + from airlock.tui.admin_client import operational_records + + payload = operational_records( + getattr(self.app, "_proxy_host", "localhost"), + getattr(self.app, "_proxy_port", "4000"), + days=31, + limit=_MAX_LOG_RECORDS, + ) + if payload is not None: + page = OperationalReadPage( + records=[r for r in payload["records"] if isinstance(r, dict)], + source=str(payload.get("source", "jsonl")), + degraded_reason=payload.get("degraded_reason"), + truncated=bool(payload.get("truncated")), + limit_hit=payload.get("limit_hit"), + ) + if page is None: + # The dashboard is a separate process. Do not open its own engine + # against a proxy-owned FathomDB file when the bridge is down. + page = read_records( + directory=log_dir, + days=31, + limit=_MAX_LOG_RECORDS, + allow_fathom=not fathomdb_selected(), + ) + records = page.records records.sort(key=lambda r: r.get("timestamp") or "", reverse=True) self._records = records # explicit bounded client-side view + self._read_source = page.source + self._read_degraded_reason = page.degraded_reason + self._read_truncated = page.truncated # Populate model filter options. Batch/file routes (/v1/batches, /v1/files) # log records with no model (model is None) — coerce so sorted() doesn't @@ -347,6 +372,16 @@ def update_ui(): model_select.set_options(options) except Exception: pass + try: + self.query_one("#logs-source", Static).update( + _operational_source_text( + self._read_source, + self._read_degraded_reason, + self._read_truncated, + ) + ) + except Exception: + pass try: self._apply_filters() except Exception: diff --git a/airlock/tui/screens/overview.py b/airlock/tui/screens/overview.py index 6f1a9cf..93aa9b6 100644 --- a/airlock/tui/screens/overview.py +++ b/airlock/tui/screens/overview.py @@ -128,6 +128,7 @@ def _format_headroom(snapshot: dict) -> str: #: traffic, and the operator should not have to divide two dollar figures in #: their head to notice. _BUDGET_ALERT_UTILIZATION = 0.8 +_PRIORITY_STALE_SECONDS = 120.0 def _budget_utilization(snapshot: dict) -> float | None: @@ -208,6 +209,60 @@ def _render_budget_detail(snapshot: dict) -> str: return "\n".join(lines) +def _render_telemetry_health(snapshot: dict[str, dict] | None) -> str: + """Compact, truthful exporter state without secrets or delivery claims.""" + if snapshot is None: + return "Telemetry: [dim]unavailable — admin read is disabled or unreachable[/]" + if not snapshot: + return "Telemetry: [dim]no instrumentation reported[/]" + parts: list[str] = [] + for name in sorted(snapshot): + item = snapshot[name] + enabled = "enabled" if item.get("enabled") else "disabled" + signals = int(item.get("signals", 0) or 0) + failures = int(item.get("failures", 0) or 0) + endpoint = item.get("endpoint") or "local/no endpoint" + error = item.get("last_error") + suffix = f" errors={failures}" if failures else "" + if error: + suffix += f" last_error={escape(str(error)[:80])}" + parts.append( + f"{escape(str(name))}: {enabled}, signals={signals}, endpoint={escape(str(endpoint))}{suffix}" + ) + return "Telemetry: " + " | ".join(parts) + + +def _render_client_priority(priority: dict | None, now: float) -> tuple[str, str]: + """Return source-labelled priority detail, degrading explicitly when stale.""" + if not isinstance(priority, dict): + return ( + " QoS priority: unavailable — admin read is disabled or unreachable.", + "", + ) + observed_at = priority.get("observed_at") + age = max(0.0, now - observed_at) if isinstance(observed_at, (int, float)) else None + if age is None or age > _PRIORITY_STALE_SECONDS: + suffix = f" ({age:.0f}s old)" if age is not None else "" + return f" QoS priority: stale{suffix} — not treated as active.", "" + enabled = bool(priority.get("admission_enabled")) + state = ( + "active boost" + if priority.get("boost") and enabled + else ("advisory (admission disabled)" if not enabled else "normal") + ) + reasons = priority.get("reasons") + reason_text = ( + ", ".join(str(reason) for reason in reasons[:4]) + if isinstance(reasons, list) + else "-" + ) + return ( + f" QoS priority: score={float(priority.get('score', 0)):.2f} " + f"state={state} source=live_admin ({age:.0f}s ago)", + f" QoS signals: {escape(reason_text)}", + ) + + def _gemini_mode_summary(counts: dict[str, int]) -> str: """Render Gemini response-mode counts as a distribution (#28). @@ -267,6 +322,7 @@ class OverviewPane(VerticalScroll): BINDINGS = [ Binding("c", "clear_quarantine", "Clear quarantine", show=True), + Binding("b", "break_client_pins", "Break client pins", show=True), ] def __init__( @@ -288,6 +344,13 @@ def __init__( self._highlighted_provider: str | None = None self._admin_snapshot: dict[str, dict] | None = None self._admin_snapshot_at: float = 0.0 + self._session_snapshot: list[dict] | None = None + self._session_snapshot_at: float = 0.0 + self._client_snapshot: dict[str, dict] | None = None + self._client_snapshot_at: float = 0.0 + self._telemetry_snapshot: dict[str, dict] | None = None + self._telemetry_snapshot_at: float = 0.0 + self._highlighted_client: str | None = None self._failover_feed = FailoverFeed() # -- compose ------------------------------------------------------------ @@ -309,6 +372,7 @@ def compose(self) -> ComposeResult: disabled=True, ) yield Static("", id="ov-status-line") + yield Static("Telemetry: loading…", id="ov-telemetry-health") with Vertical(id="ov-alerts-panel"): yield Static("[bold]Alerts[/]", id="ov-alerts-header") yield Static(self._alert_text, id="ov-alerts") @@ -366,6 +430,8 @@ def compose(self) -> ComposeResult: # -- lifecycle ---------------------------------------------------------- def on_mount(self) -> None: + if getattr(self.app, "_test_harness", False): + return self._probe_external() self._refresh_state() self.set_interval(300.0, self._probe_external) @@ -609,6 +675,7 @@ def on_data_table_row_highlighted( elif table_id == "ov-models": self._show_model_detail(key) elif table_id == "ov-clients": + self._highlighted_client = key self._show_client_detail(key) # -- admin: clear quarantine (loopback operator) ------------------------ @@ -640,6 +707,39 @@ def _clear_quarantine_worker(self, provider: str) -> None: self.notify, f"Clear failed: {err}", severity="error" ) + def action_break_client_pins(self) -> None: + """`b` — remove all affinity pins for the highlighted client.""" + client_id = self._highlighted_client + if not client_id or client_id.startswith("_empty"): + self.notify("Highlight a client row first.", severity="warning") + return + self._break_client_pins_worker(client_id) + + @work(thread=True) + def _break_client_pins_worker(self, client_id: str) -> None: + from airlock.tui.admin_client import clear_client_sessions + + status, payload = clear_client_sessions(self._host, self._port, client_id) + if status == 200: + count = payload.get("cleared_sessions", 0) + self._session_snapshot = None + self._session_snapshot_at = 0.0 + self.app.call_from_thread( + self.notify, f"Broke {count} session pin(s) for {client_id}." + ) + elif status == 404: + self.app.call_from_thread( + self.notify, + "Admin API disabled (set admin.enabled).", + severity="error", + ) + else: + self.app.call_from_thread( + self.notify, + f"Break pins failed: {payload.get('error', status)}", + severity="error", + ) + def _show_provider_detail(self, provider_name: str) -> None: try: from airlock.fast.state import store @@ -818,6 +918,60 @@ def _show_client_detail(self, client_id: str) -> None: if not found: rows.append(" No provider activity recorded.") + # Session affinity exists only in the live proxy process. Read the + # bounded, authenticated admin view rather than trying to reconstruct + # it from JSONL; identifiers are deliberately absent from that view. + if now - self._session_snapshot_at >= 30: + try: + from airlock.tui.admin_client import session_snapshot + + payload = session_snapshot(self._host, self._port) + self._session_snapshot = ( + payload.get("sessions", []) if payload else None + ) + except Exception: + self._session_snapshot = None + self._session_snapshot_at = now + + if self._session_snapshot is None: + rows.append( + " Session pins: unavailable — admin read is disabled or unreachable." + ) + else: + pins = [ + pin + for pin in self._session_snapshot + if pin.get("client_id") == client_id + ] + if pins: + rows.append(" Session pins (live admin; identifiers hidden):") + for pin in pins[:10]: + rows.append( + " " + f"model={escape(str(pin.get('model', '-')))} " + f"age={float(pin.get('age_seconds', 0)):.0f}s " + f"ttl={float(pin.get('ttl_remaining_seconds', 0)):.0f}s" + ) + rows.append(" Press b to break this client's pins (audited).") + else: + rows.append(" Session pins: none active.") + + if now - self._client_snapshot_at >= 30: + try: + from airlock.tui.admin_client import client_snapshot + + payload = client_snapshot(self._host, self._port) + self._client_snapshot = payload.get("clients", {}) if payload else None + except Exception: + self._client_snapshot = None + self._client_snapshot_at = now + + priority = (self._client_snapshot or {}).get(client_id, {}).get("priority") + priority_line, signals_line = _render_client_priority(priority, now) + rows.append(priority_line) + if signals_line: + rows.append(signals_line) + detail.update("\n".join(rows)) # -- unified data refresh ----------------------------------------------- @@ -837,6 +991,18 @@ def _refresh_state(self) -> None: now = time.time() + if now - self._telemetry_snapshot_at >= 30: + try: + from airlock.tui.admin_client import telemetry_snapshot + + payload = telemetry_snapshot(self._host, self._port) + self._telemetry_snapshot = ( + payload.get("exporters", {}) if payload else None + ) + except Exception: + self._telemetry_snapshot = None + self._telemetry_snapshot_at = now + # The overview table itself needs the optional admin snapshot; do not # wait for an operator to select a provider detail row. if now - self._admin_snapshot_at >= 30: @@ -1107,5 +1273,7 @@ def _update_ui() -> None: status_line.update( f"Guard: [{enforce_clr}]{enforce_mode}[/] {split_str} {mcp_str} {billing_str}{failover_str}" ) + telemetry_line = self.query_one("#ov-telemetry-health", Static) + telemetry_line.update(_render_telemetry_health(self._telemetry_snapshot)) self.app.call_from_thread(_update_ui) diff --git a/config.local.yaml b/config.local.yaml index 0967ef4..e8d5242 100644 --- a/config.local.yaml +++ b/config.local.yaml @@ -1 +1,52 @@ -{} +# config.local.yaml — gitignored, machine-specific overrides (NOT committed). +# Preserved here when the personal MCP servers were removed from the tracked +# config.yaml (pre-push hygiene). Activate by adding to your LOCAL config.yaml: +# include: ["config.local.yaml"] +# Caveat (litellm `_process_includes`): an included file's `mcp_servers` REPLACES +# the main config's `mcp_servers` (dict keys are not deep-merged), and the file must +# exist or startup fails. So if you `include` this, list ALL the MCP servers you want +# here (copy any you still want from config.yaml's mcp_servers block alongside these). + +mcp_servers: + # --- Azure DevOps (stdio) --- + # Literal path (litellm does not expand ${HOME} in stdio command/args). The + # `ado-mcp` basename is allow-listed via LITELLM_MCP_STDIO_EXTRA_COMMANDS in .env. + ado_mcp: + transport: stdio + command: /home/coreyt/projects/ado-mcp/.venv/bin/ado-mcp + args: ["--profile", "full"] + env: + ADO_ORG_URL: os.environ/ADO_ORG_URL + ADO_PAT: os.environ/ADO_PAT + + # --- Monospace Design TUI (stdio) --- + # Two fixes vs. the old broken entry: + # 1. Use the project's dedicated venv python (has the `mcp` SDK). Its basename + # is `python`, which is in litellm's MCP stdio command allowlist. Bare + # `python3` is system python and lacks `mcp` -> "Connection closed" at init. + # 2. LITERAL absolute paths, not ${HOME}: litellm does NOT expand env vars in + # stdio command/args, so ${HOME}/... is taken verbatim and fails to resolve. + # This file is gitignored + machine-specific, so absolute paths are correct. + mono_tui: + transport: stdio + command: /home/coreyt/projects/monospace-design-tui/.venv/bin/python + args: ["/home/coreyt/projects/monospace-design-tui/mcp-server/server.py"] + + # --- NewsCatcher CatchAll (stdio) --- + # Copied from tracked config.yaml: litellm `include` REPLACES the main config's + # mcp_servers dict wholesale, so EVERY server wanted at runtime must be listed + # here — not just the machine-specific ones — or it is dropped on restart. + newscatcher: + transport: stdio + command: python3 + args: ["-m", "airlock.mcp_servers.newscatcher_server"] + env: + NEWS_CATCHER_API_KEY: os.environ/NEWS_CATCHER_API_KEY + + # --- Armada GraphRAG (stdio) --- DISABLED: server hangs during MCP init + # armada: + # transport: stdio + # command: ${HOME}/projects/armada/cartographer/.venv/bin/armada-mcp + # args: [] + # env: + # ARMADA_PROFILE: essential diff --git a/config.yaml b/config.yaml index b28dc90..d0e8b21 100644 --- a/config.yaml +++ b/config.yaml @@ -81,6 +81,21 @@ model_list: model: openai/gpt-5.3-codex api_key: os.environ/OPENAI_API_KEY + # FathomDB benchmark extractor/chat model. This is an explicit alias, not a + # routing fallback: benchmark callers must request it deliberately. + - model_name: gpt-4o-mini + litellm_params: + model: openai/gpt-4o-mini + api_key: os.environ/OPENAI_API_KEY + + # FathomDB benchmark embedding model. The explicit marker keeps this model + # off chat completions and authorizes it only for /v1/embeddings. + - model_name: text-embedding-3-small + litellm_params: + model: openai/text-embedding-3-small + api_key: os.environ/OPENAI_API_KEY + airlock_embeddings: true + # Provider-prefixed OpenAI aliases (0.5.2) — same litellm_params as above. - model_name: openai/gpt-5-pro litellm_params: @@ -112,6 +127,17 @@ model_list: model: openai/gpt-5.3-codex api_key: os.environ/OPENAI_API_KEY + - model_name: openai/gpt-4o-mini + litellm_params: + model: openai/gpt-4o-mini + api_key: os.environ/OPENAI_API_KEY + + - model_name: openai/text-embedding-3-small + litellm_params: + model: openai/text-embedding-3-small + api_key: os.environ/OPENAI_API_KEY + airlock_embeddings: true + # --- OpenAI GPT-5.6 (GA 2026-07-09) — Sol / Terra / Luna --- # A FLAT PRICE-TIER TRIPLE over ONE capability envelope, not a mini/nano # capability ladder: all three share 1,050,000 ctx / 128,000 max output, diff --git a/dev/architecture.md b/dev/architecture.md index 6ea354a..83f58f9 100644 --- a/dev/architecture.md +++ b/dev/architecture.md @@ -295,7 +295,10 @@ An LLM-powered operational assistant that lets administrators query Airlock's state in natural language. The advisor runs a bounded tool-calling loop (max 5 iterations) against the proxy's own `/v1/chat/completions` endpoint, using tools that read from the -StateStore, JSONL logs, config, and analysis pipeline. +StateStore, JSONL logs, config, and analysis pipeline. When the operator +selects FathomDB operational reads, the separate TUI/CLI process obtains those +reads from the proxy-owned, loopback-only admin bridge rather than opening the +embedded database itself. ``` airlock/advisor/ @@ -316,6 +319,9 @@ airlock/advisor/ - **No new network listener:** The advisor runs in-process (TUI worker thread or CLI process). It calls the proxy as a client, not as an internal endpoint, avoiding circular dependencies. +- **Proxy-owned embedded reads:** FathomDB remains owned by the proxy process; + the Advisor reaches selected FathomDB history through the loopback admin + bridge and visibly falls back to bounded JSONL if that bridge is unavailable. - **Tool-based context assembly:** Rather than dumping all data into the prompt, the LLM selectively requests data via function calling. This keeps token usage bounded and works with smaller local models. @@ -445,6 +451,19 @@ victim of UN-10), `POST /airlock/admin/providers/{p}/quarantine` (loopback-only) **Design documents:** `dev/notes/design-admin-api-capability-auth.md`, `dev/notes/design-resilience-and-admin-overview.md`. +#### Proxy-owned FathomDB operational reads + +FathomDB is an optional, embedded request-analysis store. The proxy is its sole +process owner. The Textual TUI and Advisor worker are separate processes, so +they MUST NOT call `datastore.get_engine()` against the proxy's database file. +When `AIRLOCK_OPERATIONAL_READ_BACKEND=fathomdb` is explicitly selected, they +obtain bounded history, error summaries, and search results from loopback-only +`/airlock/admin/operational/*` views. Those proxy views retain source, +truncation, and degradation fields; an unavailable bridge or datastore visibly +falls back to bounded JSONL. Default operational reads remain JSONL even if the +FathomDB write path is enabled. FathomDB erasure remains limited to that store; +JSONL retention/deletion is a separate obligation. + ### 3.7 Transparency Layer (`airlock/transparency.py`) Makes observability a first-class benefit: every request mutation and the real diff --git a/dev/plans/0.5.12-plan.md b/dev/plans/0.5.12-plan.md index be9b26b..7f6bcd7 100644 --- a/dev/plans/0.5.12-plan.md +++ b/dev/plans/0.5.12-plan.md @@ -56,7 +56,7 @@ exist yet — pack B-0 produces it. | Decision | Consequence | |---|---| | **PII containment is pack 1**, folded into the milestone (owner decision) — not a separate hotfix | The live map-in-logs leak ships fixed in the first pack; the on-disk `logs/airlock-2026-08-04.jsonl` (cleartext maps) is purged as an ops action within that pack | -| **Unified provenance milestone** (owner decision) — PII **and** the injection boundary | TUI backlog (#25/#26/#30/#31/#32) stays deferred to 0.5.13; this milestone is not TUI ballast | +| **Unified provenance milestone** (owner decision) — PII **and** the injection boundary | TUI backlog (#25/#26/#30/#31/#32) is deferred to 0.5.14; this milestone is not TUI ballast | | **Global-first lists** | Per-tenant PII allow/block lists are 0.6.0 territory (tenant identity); 0.5.12 ships one global policy, per-tenant-ready in shape | | **Streaming rehydration stays deferred** | The non-streaming tool-call path is the deliverable; streaming inherits the invariants when built (own later pack) | | **Cedar / CaMeL are later tiers** | v1 needs no external PDP and no dual-LLM rebuild; adopt only when their triggers fire (primary note §4) | @@ -210,7 +210,7 @@ untrusted spans; enforce per the observe/shadow/enforce lifecycle. - **Streaming PII rehydration** — deferred; inherits every invariant when built. - **Per-tenant PII policy** — 0.6.0 (tenant identity). - **Cedar / CaMeL** — later tiers (primary note §4). -- **TUI backlog** (#25/#26/#30/#31/#32) — 0.5.13. +- **TUI backlog** (#25/#26/#30/#31/#32) — 0.5.14. - **JSONL-half erasure** (BL-2) — a 0.6.x backlog decision, though the provenance work here is adjacent. diff --git a/dev/plans/0.5.14-openrouter-deepseek-design.md b/dev/plans/0.5.14-openrouter-deepseek-design.md new file mode 100644 index 0000000..65d7125 --- /dev/null +++ b/dev/plans/0.5.14-openrouter-deepseek-design.md @@ -0,0 +1,316 @@ +# Airlock 0.5.14 — OpenRouter and DeepSeek integration design + +## Status and decision + +This is a design for the 0.5.14 implementation. It is intentionally limited to +Airlock integration, documentation, and verification. The exact locked +dependency is LiteLLM 1.94.1, which already has native adapters and model +registries for both `openrouter` and `deepseek`. Airlock must not implement a +second provider adapter, a second OpenAI-compatible transport, or a copy of +LiteLLM's model registry. + +The deployment has runtime-only `OPENROUTER_API_KEY` and `DEEPSEEK_API_KEY` +secrets in `.env`. They are not read, copied, logged, added to tracked YAML, or +used by unit tests. They may be used only for the explicitly documented funded +smoke tests after the code and no-credit tests pass. + +## Requirements + +### Common requirements + +- **R1 — Explicit configuration.** An operator exposes a provider only by + adding reviewed `model_list` entries. Each entry uses a stable Airlock alias, + LiteLLM's canonical provider/model body, and an `os.environ/...` secret + reference. Neither provider is silently enabled by the presence of an + environment variable. +- **R2 — Provider secrets and error metadata stay out of artifacts.** Keys, + authorization headers, OpenRouter routing metadata, and provider-supplied + `user_id` values must not be rendered in config examples, discovery output, + response headers, TUI snapshots, test fixtures, JSONL error fields, or + application error logs. `.env` remains ignored. This provider slice does not + change Airlock's existing, separately governed request/response-content + logging policy; its documentation must state that boundary honestly. +- **R3 — Airlock controls remain invariant.** Requests still traverse the + existing authentication, guardrails, alias resolution, routing, circuit + breaker, recorder, and response-transparency paths. Provider discovery is + informational only: it never authorizes a model, changes a configured alias, + or changes routing. +- **R4 — Exact aliases, no silent substitution.** An OpenRouter or DeepSeek + request must resolve only to a configured Airlock alias. A provider-prefixed + canonical name may resolve through the existing exact alias table only when + its configured body is identical. Fuzzy matching must retain the existing + cost-tier and qualifier protections. +- **R5 — Failure boundary.** Airlock must retain the ordinary LiteLLM exception + classification at its callback boundary. Authentication and balance failures + are not retried as availability failures; rate-limit and transient provider + failures remain observable by the existing provider-protection/circuit-breaker + paths. A provider's `Retry-After`, where LiteLLM exposes it, remains on the + LiteLLM exception response; current Airlock rate-limit observation neither + persists it nor promises to propagate it to the client. It must not be turned + into an Airlock policy error. +- **R6 — Honest attribution.** `X-Airlock-Served-By` reports the LiteLLM served + provider token, not a guess based on the requested alias. It is `deepseek` + for DeepSeek and `openrouter` for OpenRouter. For OpenRouter it must *not* + claim to identify the upstream model host selected by OpenRouter; that routing + is outside Airlock's control. +- **R7 — No default cross-provider fallback.** This work adds no implicit + fallbacks, retries, cost tiers, budgets, or model entries. An operator may + configure them using the existing reviewed Airlock settings. Any OpenRouter + fallback/routing policy is chosen at OpenRouter and documented as such. + +### OpenRouter requirements + +- **OR1 — Canonical configuration.** The documented model body format is + `openrouter//`, for example + `openrouter/openai/gpt-4o-mini`, with + `api_key: os.environ/OPENROUTER_API_KEY` and explicit + `api_base: https://openrouter.ai/api/v1`. An alternate `api_base` is an + operator override, not a different Airlock transport. +- **OR2 — Optional catalog discovery.** When startup discovery is enabled and + at least one configured OpenRouter entry resolves a key, Airlock queries + the `models` URL derived from that entry's reviewed `api_base`, with bearer + authentication. Entries for a provider must use the same base URL for one + discovery pass; otherwise discovery skips that provider. Every returned `id` + is presented as `openrouter/`; nested publisher/model IDs remain intact. + Discovery failures log only provider name, exception class, and (for HTTP + failures) status code; they never block startup. +- **OR3 — Routing/privacy boundary.** Documentation identifies OpenRouter as a + gateway. Airlock can report `openrouter` as the immediate served provider but + does not expose optional OpenRouter app/site metadata or upstream fallback, + provider, retention, or data-policy decisions by default. + +### DeepSeek requirements + +- **DS1 — Canonical configuration.** The documented model body format is + `deepseek/`, for example `deepseek/deepseek-chat`, with + `api_key: os.environ/DEEPSEEK_API_KEY` and explicit + `api_base: https://api.deepseek.com`. LiteLLM 1.94.1 otherwise defaults its + DeepSeek transport to `/beta`; setting the stable base explicitly is required + for this release. An operator must explicitly opt into a provider beta + endpoint or model. +- **DS2 — Optional catalog discovery.** When startup discovery is enabled and + at least one configured DeepSeek entry resolves a key, Airlock queries + the `models` URL derived from that entry's reviewed `api_base`, with bearer + authentication. Entries for a provider must use the same base URL for one + discovery pass; otherwise discovery skips that provider. Every returned `id` + is presented as `deepseek/`. A failure is best-effort, sanitized, and + does not block startup. +- **DS3 — Request privacy.** Airlock does not map `X-Airlock-Client` or an + authenticated client identity into DeepSeek's optional `user_id`. A future + mapping requires a separate privacy decision, an operator-controlled switch, + and a non-identifying value. +- **DS4 — Narrow tool contract.** DeepSeek support covers OpenAI function + tools only. Airlock must reject a non-function tool with an OpenAI-shaped 400 + before LiteLLM dispatch, rather than allow LiteLLM 1.94.1 to drop it. JSON + mode is not a 0.5.14 compatibility claim unless a separately pinned + provider/model contract and regression prove it. + +## Architecture review + +| Existing seam | Finding | Design consequence | +|---|---|---| +| LiteLLM `model_list` dispatch | LiteLLM 1.94.1 already owns both adapters and their request/stream parsing. | Add no dispatch adapter or request parser. Document canonical model bodies and use ordinary `litellm_params`. | +| `airlock/models_catalog.py` | Built-in discovery covers OpenAI, Anthropic, Mistral, Gemini, and Perplexity only. Its generic OpenAI-compatible fetcher already preserves slash-containing IDs and resolves an `os.environ/...` key from `model_list`, but fixed fetchers ignore a model entry's `api_base` and log `str(exc)` verbatim. | Add two thin built-in fetchers through `_fetch_openai_compatible`; derive a consistent provider's discovery URL from its reviewed `api_base`, and replace discovery exception text with an allowlisted class/status record. | +| `fast.router.set_router_config` and `capability.airlock_provider_for` | Provider identity comes from the prefix in the configured `litellm_params.model`, not a provider-specific hard-coded registry. Circuit-breaker fallback filtering is similarly alias-catalog based. | No general router, alias-table, circuit-breaker, budget, or capability-code change is required for configured `openrouter/...` or `deepseek/...` entries. Add one narrow DeepSeek non-function-tool validation after final alias/routing resolution in `AirlockFastGuardian.async_pre_call_hook`, before dispatch/circuit-breaker work. | +| `transparency.attribute_served_backend` | It reads LiteLLM's actual served provider correctly, but its classification sets only recognize existing providers. Both new tokens would currently be `unknown`. | Classify `deepseek` as `native` and `openrouter` as `gateway`; extend attribution and documentation tests. The exposed provider token remains `openrouter`, never a guessed downstream host. | +| `fast.monitor` | Existing callbacks record attempts/failures by configured provider and record rate-limit headers from a LiteLLM exception response. | Add characterization tests for representative LiteLLM-shaped 401/402/429/500/503 outcomes. Do not add a provider-specific retry loop; change handling only if a test proves the existing boundary loses a supported signal. | +| Request-event, monitor, tracing, and enterprise projections | The generic request logger records normal request/response content under its existing policy. More critically, LiteLLM's OpenRouter stream exceptions can embed `metadata` and `user_id`, and Airlock currently serializes/logs exception text; monitor also persists it as a provider-protection reason and optional tracing exports it. | Add one shared provider-error sanitizer used by request-event projection, monitor/provider-protection metadata and logs, tracing, and enterprise projection. It emits a bounded provider/status/error-type record and no free-form provider exception text; typed HTTP status/exception inspection happens before replacement so 429 handling is retained. | +| DeepSeek request transformation | LiteLLM 1.94.1 drops non-function DeepSeek tools before dispatch. | Add a narrow DeepSeek pre-dispatch validation rule that returns an OpenAI-shaped 400 for non-function tools. Do not claim generic tool types or JSON mode. | +| Config/templates and docs | The repository documents explicit aliases and keeps `.env` ignored, but currently offers no examples for these providers. | Add docs-only examples with explicit stable `api_base` values; do not add an enabled provider/model to the shipped default configuration, which would expand a deployment's public model surface merely because a key exists. | + +## Design + +### 1. Operator configuration + +The implementation adds a configuration-guide section with examples like these. +They are examples, not default entries in `config.yaml` or its template. + +```yaml +model_list: + - model_name: openrouter/openai/gpt-4o-mini + litellm_params: + model: openrouter/openai/gpt-4o-mini + api_key: os.environ/OPENROUTER_API_KEY + api_base: https://openrouter.ai/api/v1 + + - model_name: deepseek/deepseek-chat + litellm_params: + model: deepseek/deepseek-chat + api_key: os.environ/DEEPSEEK_API_KEY + api_base: https://api.deepseek.com +``` + +An operator may create a separate short alias, but the provider-prefixed alias +is the stable contract. The guide explains that model availability, OpenRouter +upstream routing, cost, and data retention are provider controls; Airlock +enforces only the controls at its own boundary. + +### 2. Informational model discovery + +Add `_fetch_openrouter_models` and `_fetch_deepseek_models` as small wrappers +over the existing `_fetch_openai_compatible` helper. Register them in +`_FETCHERS` with prefixes `openrouter` and `deepseek`. Their base URL comes from +the configured provider entries, not a separate fixed default. Before sending a +key, discovery requires exactly one normalized HTTPS base URL for that provider +and derives its `models` endpoint from it. The base must have a hostname, no URL +credentials, query, or fragment, and a path ending in either the API root or +`/chat/completions`; the latter is stripped before `/models` is appended. A +`/models` input is rejected. An absent, malformed, or conflicting base skips +discovery with a sanitized warning. The request uses a no-redirect opener, so a +cross-origin redirect cannot receive the key. Consequently a key configured for +a private endpoint is never sent to the public default or a redirect target. + +The generic helper's `f"{provider_prefix}/{model_id}"` result is deliberately +used. It preserves OpenRouter IDs such as `openai/gpt-4o-mini` as +`openrouter/openai/gpt-4o-mini`, and DeepSeek IDs as +`deepseek/deepseek-chat`. The existing concurrency, bounded timeout, and +best-effort failure posture apply unchanged. Replace the current verbatim +`str(exc)` discovery warning with provider name, exception class, and HTTP +status when present—never exception text, URL query, or headers. No discovery +result is written back to YAML, added to LiteLLM's router, or treated as a +permission grant. + +### 3. Attribution and observability + +Extend `airlock.transparency` provider classes: + +- `deepseek` is a native provider. +- `openrouter` is a gateway provider. + +The existing served-attribution read remains the sole source for response +headers, request events, JSONL, and TUI displays. The work does not introduce +OpenRouter-specific headers or surface opaque vendor headers. This avoids an +incorrect claim that Airlock knows which OpenRouter upstream served a request. + +### 4. Failure and streaming boundary + +LiteLLM remains the only provider transport and streaming parser. Airlock adds +a shared provider-error sanitizer before raw exceptions enter `RequestEvent`, +monitor/provider-protection state and logs, tracing, or an enterprise JSONL +error field. It records only a bounded error type, provider token, and HTTP +status where available; typed status inspection occurs before replacement so +429 detection is unchanged. It does not preserve arbitrary provider exception +text. Airlock's implementation then adds regression/characterization tests at +the callback boundary for: + +- standard and streaming success attribution; +- `401` and `402` as non-retryable provider/account conditions; +- `429` and `500`/`503` as provider availability outcomes; an upstream + `Retry-After` remains available only on the LiteLLM exception response and is + not an Airlock persistence or client-header contract; +- redaction of distinctive upstream `metadata`, `user_id`, and credential-like + values from provider failure events and application/JSONL error logs. + +DeepSeek function tools are sent normally. Any non-function tool is rejected +before dispatch with an OpenAI-shaped 400, which is safer and more transparent +than LiteLLM's current drop behavior. If an actual LiteLLM 1.94.1 DeepSeek +stream fails its documented keep-alive behavior, the failure must be reduced to +a focused test first. Only then may Airlock add a minimal boundary adaptation; +it must not fork general SSE parsing. + +### 5. Documentation and funded smoke + +The configuration guide, routing guide, provider-observability guide, and +operations environment-variable table will describe the two keys, exact model +forms, discovery behavior, attribution semantics, and the no-default-enable +posture. The funded smoke procedure uses a minimal non-sensitive prompt through +the Airlock proxy for one configured model per provider, once non-credit tests +are green. It records only timestamp, provider alias, HTTP outcome, stream +completion outcome, and safe Airlock response headers; never a key, prompt, +completion, or raw upstream headers. + +## Acceptance criteria + +### Configuration and policy + +- **AC1.** Documentation contains valid OpenRouter and DeepSeek configuration + examples using `os.environ/OPENROUTER_API_KEY` and + `os.environ/DEEPSEEK_API_KEY`; tracked YAML contains no secret values. +- **AC2.** A key in `.env` alone does not expose either provider. An operator + must add a `model_list` entry and restart. +- **AC3.** A configured provider-prefixed alias produces the correct + `airlock_provider` record and existing alias/fallback protection cannot route + it to an unconfigured or different-body model. + +### Discovery + +- **AC4.** Mocked OpenRouter discovery of `openai/gpt-4o-mini` returns exactly + `openrouter/openai/gpt-4o-mini`; mocked DeepSeek discovery of `deepseek-chat` + returns exactly `deepseek/deepseek-chat`. +- **AC5.** Discovery makes no request if the provider has no configured, + resolvable key, or has an absent, malformed, credential-bearing, + query/fragment-bearing, or conflicting configured base. It deterministically + derives `models` from an API root or `/chat/completions` base, rejects a + `/models` base, and follows no redirect; a cross-origin redirect test proves + its key never reaches another host. For either provider, an HTTP/network + failure returns the other successful discoveries and does not prevent startup. +- **AC6.** Discovery output never becomes a model-list entry, routing target, + authorization decision, or logged credential. A discovery exception carrying + a sentinel secret produces only provider, exception class, and safe status in + application logs. + +### Attribution, failures, and privacy + +- **AC7.** LiteLLM-shaped responses yield `X-Airlock-Served-By: deepseek` with + `backend_kind: native`, and `X-Airlock-Served-By: openrouter` with + `backend_kind: gateway`; existing provider classifications remain unchanged. +- **AC8.** Characterization tests cover 401, 402, 429, 500, and 503 at the + Airlock callback boundary and prove no policy denial is retried or re-labelled + as an upstream failure. A LiteLLM-exposed `Retry-After` remains only on the + exception response; its persistence and client-header propagation are out of + scope. +- **AC9.** A distinctive OpenRouter stream-error `metadata`, `user_id`, and + credential-like value is absent from `RequestEvent` error fields, JSONL, + monitor/provider-protection state and logs, and tracing; the bounded error + category/status remains useful and 429 classification still works. The + provider slice does not change the separately governed general + request/response-content logging policy. +- **AC10.** DeepSeek function tools pass through the normal policy path; a + non-function tool is rejected pre-dispatch with a documented OpenAI-shaped + 400. JSON mode is not advertised unless separately proven for the configured + model and exact LiteLLM version. +- **AC11.** Documentation states that OpenRouter's downstream-provider choice, + fallback, pricing, and retention policy are not controlled or verified by + Airlock. + +### End-to-end evidence + +- **AC12.** Unit and integration tests for the changed seams pass without live + credentials or paid calls. +- **AC13.** After AC1–AC12 are green, four manually authorized funded smoke + calls prove one normal and one streaming completion for each configured + provider alias. Artifacts retain only the safe fields defined above. A + provider account/balance failure is recorded as a blocker, not treated as a + successful integration. + +## Out of scope + +- Adding a LiteLLM dependency upgrade, custom provider adapter, duplicate + model/cost registry, or generic OpenAI-compatible proxy layer. +- Auto-enabling a model because an environment variable or discovery result + exists. +- Default model selection, cross-provider fallbacks, provider budgets, or cost + tier changes. +- Guaranteeing OpenRouter's downstream provider, privacy, retention, pricing, + or fallback behavior. +- Sending `X-Airlock-Client`, a virtual-key identity, or customer data as + DeepSeek `user_id`. + +## Implementation order + +1. Add and test the two catalog fetcher wrappers, explicit-base guard, and their + no-key/failure cases. +2. Add and test served-backend classifications. +3. Add the shared error sanitizer, DeepSeek non-function-tool pre-dispatch + validation, and callback-boundary/error characterization tests. +4. Add configuration and operator documentation, including the funded smoke + runbook and artifact redaction rules. +5. Run no-credit tests, then perform the manually authorized four-call smoke + using the existing runtime-only keys. + +## References + +- [LiteLLM OpenRouter provider documentation](https://docs.litellm.ai/docs/providers/openrouter) +- [LiteLLM DeepSeek provider documentation](https://docs.litellm.ai/docs/providers/deepseek) +- [OpenRouter quickstart](https://openrouter.ai/docs/quickstart) +- [DeepSeek list models API](https://api-docs.deepseek.com/api/list-models) diff --git a/dev/plans/0.5.14-todo.md b/dev/plans/0.5.14-todo.md index 7c91831..f97356c 100644 --- a/dev/plans/0.5.14-todo.md +++ b/dev/plans/0.5.14-todo.md @@ -1,6 +1,210 @@ -# Airlock 0.5.14 — TODO +# Airlock 0.5.14 — prioritized delivery plan -## TUI test-harness lifecycle optimization +**Release workspace:** [Slices 0–6 and delivery allocations](0.5.14/README.md) +contain the durable environment, dependency, documentation, requirements, +architecture, verification, and HITL review records. Feature implementation +begins only after the Slice 6 decision. + +This is the release order. A later slice may be designed in parallel, but it +does not enter release verification before all acceptance criteria in the prior +slice are met. This keeps FathomDB's benchmark dependency ahead of new provider +choice, TUI improvements, and the optional operational backend. + +**Detailed provider design:** [OpenRouter and DeepSeek integration design](0.5.14-openrouter-deepseek-design.md) +defines the provider requirements, architecture findings, and acceptance +criteria used by slice 2. + +## Delivery group A — FathomDB benchmark readiness + +**Goal:** let FathomDB run its planned LLM-backed benchmarks through Airlock +without widening the default public model surface, retaining benchmark +request/response content in logs, or falsely advertising an embedding endpoint +that Airlock does not support. + +### Work + +- **`gpt-4o-mini` chat alias.** Add an explicit, reviewed OpenAI-backed + `gpt-4o-mini` model-list entry (and the repository configuration template + where applicable). Verify ordinary and streaming chat completions traverse + Airlock authentication, guards, alias policy, observability, and served-model + attribution. Do not add automatic fallbacks or enable the alias merely + because an OpenAI key exists. +- **`text-embedding-3-small` embedding capability and alias.** Add the + OpenAI-backed embedding alias and make `/v1/embeddings` a real Airlock + capability rather than a configuration-only promise. Preserve ordinary batch + inputs and supported `dimensions` and `encoding_format` request options; + apply the same authentication, policy, observability, error, and redaction + boundaries as other inference requests. Extend the capability/model contract + and add no-live-credit regression and smoke coverage. Do not advertise + embeddings for an alias that has not been configured and tested for it. +- **Benchmark-safe log-redaction evidence.** Supply a documented benchmark + environment profile with `AIRLOCK_LOG_REDACT_FIELDS=messages,response` and a + sentinel-based regression/procedure proving enterprise JSONL stores + `[REDACTED]`, not request or response content. Keep SQL logging disabled in + that profile because its projection is not content-redacted, and keep + `AIRLOCK_FATHOM_STORE_MESSAGES` and `AIRLOCK_FATHOM_STORE_RESPONSE_TEXT` + disabled. This is separate from the outbound PII guard. + +### Done when + +- FathomDB can use configured `gpt-4o-mini` chat and + `text-embedding-3-small` embeddings through Airlock's documented interfaces, + including a successful manually funded smoke with non-sensitive inputs. +- The model/capability response describes embeddings accurately, and attempts + to use an unconfigured embedding model or unsupported option fail clearly. +- Benchmark logging has machine-verifiable redaction evidence and no enabled + log or Fathom content-retention path can retain benchmark request/response + text. + +## Delivery group B — provider foundation, OpenRouter, and DeepSeek + +**Order within the slice:** build and test the shared provider boundary first; +then integrate OpenRouter; then integrate DeepSeek. The common work is not +optional: provider discovery, safe attribution, and sanitized failures must be +in place before a provider is presented as supported. + +### Shared provider foundation + +- Preserve explicit model-list authorization: an environment key alone exposes + no model, discovery is informational only, and no provider may silently + substitute a requested model. +- Add safe configured-API-base catalog discovery, including malformed-base, + conflict, no-redirect, and no-secret-in-error behavior. +- Extend served-provider attribution (`openrouter` is a gateway; `deepseek` is + native) and add a shared bounded provider-error sanitizer before errors enter + request events, monitor state, tracing, or enterprise logs. +- Characterize LiteLLM-shaped 401, 402, 429, 500, and 503 outcomes at the + Airlock boundary; retain ordinary rate-limit/circuit-breaker behavior without + adding provider-specific retry loops or persisting upstream `Retry-After`. + +### OpenRouter integration and verification + +**Why this is scheduled:** the pinned LiteLLM 1.94.1 already has a native +OpenRouter provider adapter and model registry. This slice is therefore +Airlock integration and verification work, not a new provider adapter, +transport implementation, or LiteLLM upgrade. It makes the existing capability +a first-class, documented deployment option without hiding where routing, +fallback, privacy, and billing decisions occur. + +### Work + +- Add a supported configuration recipe that uses LiteLLM's existing + `openrouter//` naming and an environment-provided + `OPENROUTER_API_KEY` with explicit `api_base: https://openrouter.ai/api/v1`. + Document an override only as an explicit operator choice. No key may appear in + tracked configuration, logs, TUI state, diagnostics, or test fixtures. +- Extend Airlock's optional, informational startup catalog to discover + OpenRouter's `/models` response and normalize it to the same canonical + `openrouter//` names LiteLLM accepts. Do not duplicate + LiteLLM's model registry, use discovery as an authorization source, or + silently substitute a different requested model. +- Preserve Airlock's own policy and transparency behavior when OpenRouter + performs upstream-provider selection or fallback. Distinguish the requested + model from the immediately served `openrouter` gateway, but do not display + OpenRouter upstream-provider/fallback metadata by default or claim that + Airlock controls it. +- Verify that LiteLLM's OpenRouter responses retain the signals Airlock needs: + `429`/`503`, streaming, and safe served-model/provider metadata. Capture a + LiteLLM-exposed `Retry-After` remains only on the exception response; 0.5.14 + does not persist or propagate it to clients. Add a bounded provider + error-sanitization seam before OpenRouter streaming-error metadata or + `user_id` can reach JSONL or application logs. Do not reimplement LiteLLM + retry, streaming, or response parsing. +- Document the privacy and routing controls an operator must choose deliberately: + provider allow/order, fallback behavior, data-collection policy, and + zero-data-retention restrictions. Airlock forwards only supported controls + and does not claim to enforce an upstream provider's independent retention + commitment. +- Add isolated, no-live-credit regression tests for configuration, catalog + normalization, safe metadata handling, and Airlock's handling of LiteLLM + `429`/`503` results and `Retry-After` exception-response boundary. A manually funded smoke test proves the + upstream integration; it is not a prerequisite for native LiteLLM support, + which is already established by the locked dependency. + +### Done when + +- An operator can configure OpenRouter with a runtime secret and LiteLLM's + canonical model name; Airlock's optional catalog displays matching names and + the normal policy, observability, and failure-handling paths retain their + behavior. +- Regression coverage verifies the supported configuration and Airlock boundary; + a documented funded smoke confirms a successful request and streaming without + exposing credentials or prompt/response content in artifacts. +- Documentation states the boundary honestly: Airlock governs the request at + its proxy, while OpenRouter may select or fall back among its upstream model + providers according to the explicitly configured routing/privacy policy. + +**Provider references:** [OpenRouter quickstart](https://openrouter.ai/docs/quickstart), +[provider routing](https://openrouter.ai/docs/guides/routing/provider-selection), +and [error handling](https://openrouter.ai/docs/api/reference/errors-and-debugging). + +### DeepSeek integration and verification + +**Why this is scheduled:** the pinned LiteLLM 1.94.1 already has a native +DeepSeek provider adapter and model registry. This slice is Airlock integration +and verification work, not a new provider adapter, custom OpenAI-compatible +transport, or LiteLLM upgrade. It documents and validates the deployment path +without pinning aliases, leaking an API key or customer identity, or mistaking +provider overload for an Airlock policy failure. + +### Work + +- Add a supported configuration recipe using LiteLLM's existing + `deepseek/` naming, `DEEPSEEK_API_KEY`, and explicit stable + `api_base: https://api.deepseek.com`. LiteLLM 1.94.1 otherwise defaults to a + beta endpoint. A key must never appear in tracked configuration, TUI state, + diagnostics, or test fixtures; provider error fields must be sanitized before + JSONL or application logs. +- Extend Airlock's optional, informational startup catalog to discover + DeepSeek's `/models` response and normalize it to the canonical + `deepseek/` names LiteLLM accepts. Do not make a temporary alias or + beta endpoint the default, duplicate LiteLLM's registry, or use discovery as + an authorization source. +- Verify standard and streaming requests and OpenAI function tools through the + normal Airlock policy, observability, and response-handling paths. Reject a + non-function tool before LiteLLM can silently drop it; do not claim JSON mode + unless a configured-model regression proves it. Verify that the installed + LiteLLM version handles DeepSeek keep-alives; add an Airlock parser only if a + focused failing regression proves it is needed. +- Verify the classification presented to Airlock for `401` authentication, + `402` balance, `429` concurrency, and transient `500`/`503` outcomes. Change + Airlock handling only for a demonstrated boundary gap; do not persist or + propagate an upstream `Retry-After`, and do not retry an Airlock policy denial. +- Do not forward Airlock's client identity as DeepSeek's optional `user_id` + parameter until a separate privacy review defines a non-identifying, + operator-controlled mapping. User IDs must never contain customer or prompt + data. +- Add isolated, no-funded-account regression tests for configuration, catalog + normalization, the LiteLLM-to-Airlock response boundary, and error + classification. A manually funded smoke test confirms upstream operation; it + is not a prerequisite for native LiteLLM support, which is already + established by the locked dependency. + +### Done when + +- An operator can configure DeepSeek with a runtime secret and LiteLLM's + canonical model name; Airlock's optional catalog displays matching names and + standard and streaming requests retain normal Airlock behavior. +- Tests prove the LiteLLM-to-Airlock boundary for authentication, balance, + rate-limit, and overload results. Health and retry signals remain bounded and + never contain credentials, prompts, or a DeepSeek `user_id`. +- Documentation names the supported stable endpoint and request features, labels + beta-only features as opt-in, and states that provider model availability, + pricing, and concurrency are controlled by DeepSeek. + +**Provider references:** [DeepSeek models](https://api-docs.deepseek.com/api/list-models), +[rate limits and `user_id`](https://api-docs.deepseek.com/quick_start/rate_limit), +[error codes](https://api-docs.deepseek.com/quick_start/error_codes/), and +[tool calls](https://api-docs.deepseek.com/guides/tool_calls). + +## Delivery group C — TUI reliability and operator backlog + +**Order within the slice:** first make ordinary TUI tests fast and +deterministic; then deliver the bounded operator surfaces below. Each surface +must have its source seam before the display work starts, remain off the +inference hot path, and degrade visibly when its data is unavailable. + +### TUI test-harness lifecycle optimization **Why this is scheduled:** the 0.5.12 release verification ran `uv run pytest --tb=short -q -m 'not live' --durations=40`. The suite reached @@ -33,35 +237,61 @@ navigation, and rendering assertions. app teardown, JSONL/MCP shutdown, and the OverviewPane stale-refresh path. - No production TUI behaviour is gated on the test-harness setting. -## FathomDB as an operational backend +### Deferred 0.5.13 TUI operator backlog + +**Scope transfer:** 0.5.12 deferred this complete set to 0.5.13. There is no +separate 0.5.13 plan: 0.5.14 now owns all five issues and their required proxy +plumbing. The work remains off the inference hot path; the TUI reads bounded +snapshots, JSONL, or authenticated read endpoints rather than subscribing to +request callbacks. + +### Work + +- **#25 — Smart-router prompt classification.** Surface the per-request + simple/moderate/complex classification in Guards, its distribution in the + operator view, and the tier-to-model mapping that explains the route chosen. +- **#26 — Session affinity pins.** Add active client pins (model, age, and TTL + remaining) to the Clients detail view and a safe operator action for breaking + a pin when debugging unexpected routing. +- **#30 — Virtual-key management.** Deferred to the + [0.5.15 backlog](0.5.15-todo.md) with its draft requirements, acceptance + criteria, and architecture constraints. It is not 0.5.14 implementation + scope. +- **#31 — QoS priority during congestion.** Show a client's current priority + and contributing signals, plus whether congestion makes priority active. + Persist only the bounded last-observed information needed by the read view; + do not add a TUI dependency to the request path. +- **#32 — OpenTelemetry and metrics exporter health.** Instrument the exporters + before rendering them: independently show enabled state, safely rendered + endpoint, last successful export, rolling counts, failures, and a bounded + last-error summary. A configuration-only "enabled" indicator is insufficient. +- Add focused unit and integration coverage for every new snapshot/read seam, + stale or unavailable data, and the destructive/key-secret paths. Close each + issue with evidence, or leave it open with an explicit missing prerequisite. + +### Done when + +- Issues #25, #26, #31, and #32 are closed with the documented operator surface + and evidence, or retain an explicit, evidence-backed blocker. Issue #30 is + explicitly deferred to 0.5.15. +- New read paths remain bounded and degrade visibly when their source is + unavailable; none run on the inference hot path. +- Key material and exporter credentials are never rendered, retained in a UI + snapshot, or written to Airlock telemetry. + +## Delivery group D — FathomDB as an optional operational backend **Why this is scheduled:** 0.5.11/0.5.12 made FathomDB an optional, lazy, -single-owner request-analysis store. It is not yet the selectable backend for -Airlock's operational reads: the TUI and advisor still rely on their existing -JSONL/in-memory paths. The next step should make that relationship explicit -rather than treating FathomDB as only a secondary callback sink. +single-owner request-analysis store. Slice 110 makes it a selectable +operational-read backend through a proxy-owned, loopback-only bridge, while +preserving JSONL as the default and fallback. ### Work -- Define the supported backend contract: which operational reads (TUI history, - advisor/search, and retention/erasure reporting) may use FathomDB, which - remain JSONL-backed, and which data source is authoritative for each answer. -- Add a selectable FathomDB operational-read backend, preserving the current - JSONL/in-memory behavior when disabled or unavailable. Do not silently make - FathomDB a required dependency or change the JSONL audit/retention contract. -- Establish process ownership, startup/restart, read bounds, and failure - behavior for the backend; separate-process shared writers remain unsupported - unless a design and evidence change that constraint. -- Add end-to-end tests for enabled, disabled, unavailable, and per-client - erasure states, including proof that incomplete Fathom erasure is never - presented as a completed whole-record deletion. +- Implemented in Slice 110; see its design and status record for the supported + readers, bounds, bridge authorization, fallback behavior, and retained + erasure-scope limitation. ### Done when -- An operator can opt into FathomDB as the documented operational-query - backend, while a default deployment remains zero-infrastructure and retains - current JSONL behavior. -- Every backend-backed view reports its source and bounded/degraded outcome - clearly; it does not silently return partial data as complete. -- The FathomDB engine lifecycle and single-owner constraint are covered by - integration tests and documented for operators. +- Complete: the status record contains the durable acceptance evidence. diff --git a/dev/plans/0.5.14/00-environment.md b/dev/plans/0.5.14/00-environment.md new file mode 100644 index 0000000..193225b --- /dev/null +++ b/dev/plans/0.5.14/00-environment.md @@ -0,0 +1,39 @@ +# Slice 0 — environment readiness proposal + +**Purpose:** identify release prerequisites. This record makes no configuration, +secret, dependency, service, or CI change. + +## Observed baseline + +| Area | Evidence | Proposal / owner | +| --- | --- | --- | +| Runtime | `uv 0.11.6`; project interpreter Python 3.12.3; CI uses Python 3.12. | Keep Python 3.12 as the release baseline. Slice 1 evaluates dependency changes against it. | +| Dependency integrity | `uv lock --check` resolved the 184-package lockfile successfully. | Require `uv lock --check` in every feature-slice closeout. | +| Test environment | Collection succeeds: 3,298 tests collected and five live tests deselected. | Use `uv run --extra test python -m pytest` for targeted test work; preserve `-m 'not live'` release gate. | +| Optional features | `db`, `s3`, `sql`, `metrics`, `tui`, `search`, `vertex`, `aistudio`, `analyzer`, `mistral`, and `tracing` extras exist. | Slice owners must name the smallest extras required by their tests; no all-extras local requirement except integration/release verification. | +| PII guard | Deployed PII egress is observe-only and Wake records a human DECIDE gate. | No 0.5.14 work changes enforcement posture. Benchmark work documents this as a separate outbound-control boundary. | +| Benchmark secrets | Runtime `.env` contains the provider keys previously supplied by the operator; values were not inspected. | Slice 10–60 use only environment references. Tests use sentinel values; funded smokes require an explicit operator authorization after no-credit tests pass. | +| Content retention | Enterprise logger supports `AIRLOCK_LOG_REDACT_FIELDS`; Fathom raw-message/response flags and SQL logging are separately configured. | Slice 10 creates the documented benchmark profile: redact `messages,response`; keep SQL logging and Fathom raw content storage disabled. | +| Health endpoint | `/health/liveliness` is the no-model-call probe; Wake marks `/health` unsafe for liveness. | All new smoke/runbook commands use `/health/liveliness`; test and docs changes must not reintroduce `/health` as liveness. | +| FathomDB lifecycle | FathomDB remains optional and single-owner; a LiteLLM-child memory-owner diagnosis is open. | Slice 110 must retain JSONL fallback and cannot claim multi-process shared writers. Fathom benchmark clients do not resolve the memory blocker. | + +## Required setup before feature implementation + +1. Create a sanitized benchmark environment example, not a tracked `.env`; + it contains names/settings only, never secret values. +2. Create a funded-smoke worksheet that records operator authorization, alias, + timestamp, HTTP/stream outcome, and safe response headers only. +3. Add the actual 0.5.14 feature tests to the existing non-live CI path. Run + manual funded smokes outside CI and exclude their prompt/response content + from artifacts. +4. For FathomDB-backed test paths, install the `db` extra and use a temporary + store owned by the test process. Do not share it with a running Airlock + instance. + +## Acceptance for this preparation slice + +- Every feature slice can state its Python/extras/test command and whether it + needs a manual funded smoke. +- No planned path requires reading, copying, or committing a provider secret. +- The release closeout can prove the chosen health probe and content-retention + controls without inspecting production traffic. diff --git a/dev/plans/0.5.14/01-dependencies.md b/dev/plans/0.5.14/01-dependencies.md new file mode 100644 index 0000000..3cca50d --- /dev/null +++ b/dev/plans/0.5.14/01-dependencies.md @@ -0,0 +1,42 @@ +# Slice 1 — Dependabot and library sweep proposal + +**Purpose:** assess the locked dependency graph and Dependabot policy. This +record performs no upgrade and does not widen compatibility ranges. + +## Current evidence + +- Dependabot runs weekly for `uv` and GitHub Actions, groups core proxy, + optional integration, and developer-tooling updates, blocks majors, and + explicitly holds FathomDB `>=0.4`, Newscatcher `>=2`, and Textual `>=6.3`. +- `uv lock --check` passes. LiteLLM 1.94.1 → 1.96.2 and several routine + patch/minor candidates remain deferred; FathomDB 0.8.22 is now locked and + installed after its DB-extra contract verification. +- Textual 6.2.1 → 8.2.8 is intentionally out of range: the repository records + the Rich 14 conflict with the validated LiteLLM proxy baseline. It is not a + patch sweep candidate. +- Existing CI runs a locked full-extra suite, docs contract, lint, Docker build, + and pip-audit with three explicitly documented cryptography suppressions. + +## Dependabot response policy + +| Candidate | Disposition | Required evidence before merge | +| --- | --- | --- | +| GitHub Actions weekly group | **Include.** Keep Dependabot configuration unchanged; review each PR's action provenance and run existing CI. | Workflow diff review; normal CI. | +| LiteLLM 1.94.1 → 1.96.2 | **Conditional include, Slice 40.** Provider work touches LiteLLM boundaries, so upgrade only after characterizing OpenRouter/DeepSeek, embedding, streaming, error, and attribution behavior on the current pin. | Focused provider regression matrix, full non-live suite, docs/lock/Docker checks, funded smoke after code verification. | +| FathomDB 0.8.21 → 0.8.22 | **Included and verified, Slice 110.** The pin and lock now resolve 0.8.22. | 110 focused DB-extra tests passed: optional import/lifecycle, projections, query/search, erasure, and proxy-owned operational reads (110 passed). | +| boto3/botocore 1.43.62 → 1.43.69 | **Include only with an S3-slice regression.** A patch update is reasonable but not a 0.5.14 release blocker. | `s3`-extra migration/logger tests. | +| google-genai, google-auth, anthropic, mistralai, SQLAlchemy, OpenTelemetry, pytest-cov, spaCy patch releases | **Postpone unless a security advisory or affected slice needs them.** These provide no direct 0.5.14 user value. | Affected extra's regression suite and changelog review. | +| cryptography 48.0.1 → 50.0.0 | **Postpone.** The current LiteLLM/Presidio upper bounds and documented audit exceptions make this a coordinated dependency migration, not a sweep update. | Removal of constraints, full security review, and removal/update of exceptions. | +| Newscatcher 1.5.1 → 3.1.1 | **Reject for 0.5.14.** The repository deliberately caps `<2` because API migration is required. | Dedicated compatibility design outside this release. | +| Textual 6.2.1 → 8.2.8 | **Reject for 0.5.14.** It crosses the explicit Rich/LiteLLM compatibility boundary. | LiteLLM migration validation and a dedicated TUI compatibility slice. | + +## Proposed changes + +1. Keep the Dependabot configuration as-is; its grouping and excludes match + documented constraints. +2. In Slice 6, make a go/no-go decision only on LiteLLM and FathomDB patches. + Other routine patches remain Dependabot work after the release unless a + security finding changes priority. +3. Re-run `pip-audit` during release closeout. Each existing suppression must + still match its documented reachability rationale; do not renew or add an + exception without a new assessment. diff --git a/dev/plans/0.5.14/02-documentation-cruft-proposal.md b/dev/plans/0.5.14/02-documentation-cruft-proposal.md new file mode 100644 index 0000000..f8c7365 --- /dev/null +++ b/dev/plans/0.5.14/02-documentation-cruft-proposal.md @@ -0,0 +1,35 @@ +# Slice 2 — documentation lifecycle proposal + +**Purpose:** enumerate the repository's documentation classes and propose their +lifecycle. This is a proposal only: no file is moved, deleted, renamed, or +edited by this slice. + +## Inventory and proposed action + +| Documentation set | Current location / enumeration | Proposed action | Rationale | +| --- | --- | --- | --- | +| Public landing and installation | `README.md`; `docs/index.md`; `docs/getting-started/*.md` | Keep | Supported user entry points; update during provider/embedding delivery. | +| Public guides | `docs/guide/*.md` (TUI, CLI, advisor, Fathom, guardrails, routing, rate limits, observability, provider quota, batch, MCP, paid services) | Keep | Published feature documentation governed by MkDocs navigation and contract tests. | +| Public operations/reference/architecture | `docs/operations.md`, `docs/troubleshooting.md`, `docs/reference/*.md`, `docs/architecture/*.md`, `docs/changelog.md` | Keep | Operator and API contract material; Slice 4 identifies required corrections. | +| Public-doc build configuration | `mkdocs.yml`; documentation-contract tests | Keep | Authoritative navigation and public-doc verification. | +| Product needs and requirements | `dev/user-needs.md`, `dev/requirements.md`, `dev/feature-*.md`, selected `dev/design-*.md` | Keep and revise through Slice 3 drafts | The two canonical documents need a current traceability refresh; do not delete their historical value. | +| Active release planning | `dev/plans/0.5.14-todo.md`, `dev/plans/0.5.14-openrouter-deepseek-design.md`, `dev/plans/0.5.14/` | Keep | Active 0.5.14 source of truth; new workspace separates preparation and delivery records. | +| Historical release plans | `dev/plans/0.4.0-plan.md` through `0.5.12-plan.md`, `dev/plans/archive/README.md` | Archive in place | Paths are evidence-linked. Labeling/indexing is safer than moving or deleting. | +| Historical orchestration evidence | `dev/plans/runs/*`, `dev/plans/prompts/*` | Archive in place | Immutable run/review records; retain stable paths and distinguish from live status. | +| Versioned design notes | `dev/notes/0.5.*-design.md`, `as-built-*`, `design-*`, `handoff-*`, audit and investigation notes | Archive in place after status annotation | Valuable engineering rationale, but not active requirements unless explicitly linked from an active plan. | +| Current technical design inputs | `dev/architecture.md`, `dev/architecture-overview.txt`, `dev/tui-design.md`, `dev/design-fathom-storage-model.md`, `dev/design-unified-batch-gateway.md` | Deprecate in place where duplicated; choose one canonical architecture source in Slice 4 | Multiple architecture summaries can drift. Keep source material until a successor and inbound-link review exist. | +| Development runbooks/harness material | `dev/agent-harness-*.md`, `dev/smoketest/*`, `dev/debugging/*`, `dev/dogfooding.md`, `dev/accessing-experiments.md` | Keep; archive superseded procedures in place | These are operational developer documentation; validate commands in Slice 5 before deprecation. | +| Research and externally volatile notes | `dev/competitor-tracker.md`, GTM briefs/role notes, model/provider research, `dev/substrate-decision-brief.md` | Keep local/untracked or archive in place; do not publish as product docs | They are decision inputs rather than stable product promises. The existing decision already keeps competitor tracking untracked. | +| Generated/binary diagnostic artifacts | root `oom-profile-*.bin`, generated run JSON, coverage/build output | Archive only when a linked investigation requires it; otherwise delete from the working tree, never treat as documentation | They are not program documentation and should not become accidental tracked release material. | + +## Proposed cleanup rules for a later approved slice + +1. **Keep** content only when it is canonical, active, or a referenced evidence + record. Fix links and status labels rather than duplicating it. +2. **Deprecate in place** when a successor is named and inbound references have + been redirected; retain a one-line replacement pointer. +3. **Archive in place** historical plans, prompts, run outputs, and design + records. Do not bulk-move them because existing links are evidence. +4. **Delete** only untracked/generated artifacts or duplicate drafts after an + owner confirms no evidence/reference need. Slice 6 must approve each + deletion list explicitly. diff --git a/dev/plans/0.5.14/03-draft-requirements-and-allocation.md b/dev/plans/0.5.14/03-draft-requirements-and-allocation.md new file mode 100644 index 0000000..c357d02 --- /dev/null +++ b/dev/plans/0.5.14/03-draft-requirements-and-allocation.md @@ -0,0 +1,68 @@ +# Slice 3 — draft needs, requirements, acceptance criteria, and allocation + +**Status:** Slice 3 started as draft only. After Slice 6 approval, slices 10–90 +and 120 ratified their allocated drafts and wrote their completed requirements +to `dev/user-needs.md` / `dev/requirements.md`; Slice 100 remains deferred and +Slice 110 remains conditional. + +## Draft additions and revisions + +| Draft | Proposed user need / requirement / acceptance criterion | Allocated slice | +| --- | --- | --- | +| DUN-14 | Benchmark operators need Airlock to serve a configured low-cost chat model and embedding model with the same access, policy, and audit boundaries as normal inference. | 20, 30 | +| DFR-24 | Airlock shall expose configured OpenAI chat aliases for `gpt-4o-mini` only through explicit reviewed model-list entries; normal and streaming requests retain ordinary policy and attribution. | 20 | +| DAC-24 | Configuration/template consistency tests prove aliases; mocked chat and stream tests prove authentication, guard traversal, resolved alias, and served-provider attribution; a funded non-sensitive smoke is recorded separately. | 20 | +| DFR-25 | Airlock shall support `/v1/embeddings` only for explicitly configured, embedding-capable aliases and shall preserve batch input plus supported `dimensions`/`encoding_format` options through policy and observability paths. | 30 | +| DAC-25 | Tests prove a configured `text-embedding-3-small` request succeeds through the endpoint, correct model capability advertises embeddings, an unconfigured alias is rejected, and unsupported options fail clearly without upstream dispatch. | 30 | +| DFR-26 | Operators shall have a benchmark logging profile that redacts request/response content in enterprise JSONL and disables unredacted SQL/Fathom raw-content retention paths. | 10 | +| DAC-26 | A sentinel request and response are absent from persisted artifacts while redacted fields are present; profile documentation names the settings and the no-model-call health probe. | 10 | +| DFR-27 | Provider configuration shall be explicit, discovery informational and base-bound, provider errors sanitized, and served-provider attribution truthful. | 40 | +| DAC-27 | No-key/no-model/no-redirect/conflicting-base tests pass; provider error sentinels never enter logs/events/traces; 401/402/429/500/503 classification and gateway/native attribution are characterized. | 40 | +| DFR-28 | OpenRouter shall be an operator-configured gateway integration: optional discovery, chat/stream verification, and operator-selected routing/privacy controls. Client OpenRouter routing overrides are rejected; Airlock never claims downstream-provider control. | 50 | +| DAC-28 | Mock catalog IDs normalize exactly, no catalog result authorizes a model, root/nested `route`/`models`/`transforms` fail as OpenAI-shaped pre-dispatch 400s, stream/error boundaries are safe, and funded smoke covers one configured alias. | 50 | +| DFR-29 | DeepSeek shall use the explicit stable API base, curated aliases, optional discovery, and OpenAI function tools only; Airlock shall not forward client identity as provider `user_id`. | 60 | +| DAC-29 | Mock discovery, standard/stream success, final-provider function-tool pass-through, non-function OpenAI-shaped 400, default-free config, and typed provider-error tests pass; funded smoke covers one configured alias. | 60 | +| DFR-30 | Ordinary TUI tests shall render the production widget tree without unrelated background workers; named integration tests retain lifecycle/cancellation coverage. | 70 | +| DAC-30 | Duration evidence improves the TUI tail and integration tests retain shutdown, stale-callback, JSONL, and MCP-worker coverage. | 70 | +| DFR-31 | Authenticated operators shall receive bounded, source-labeled views of routing classification, session affinity, QoS priority, and exporter health without adding inference-path dependencies. | 80, 90 | +| DAC-31 | Each view/action has unavailable/stale/source and secret-safety tests; session-pin break is authenticated and auditable. | 80, 90 | +| DFR-32 | Deferred to 0.5.15: administrators shall manage masked virtual keys, create with one-time reveal, and revoke through an explicit authenticated seam. | 0.5.15 Slice 100 | +| DAC-32 | Deferred to 0.5.15: creation/reveal/revocation, no-secret-in-snapshot/log/read, policy/spend display, and unauthorized-action tests pass. | 0.5.15 Slice 100 | +| DFR-33 | Conditional: operators may select FathomDB for specified operational reads while JSONL/in-memory fallback, source labeling, bounded results, single-owner lifecycle, and incomplete-erasure honesty remain intact. | 110 | +| DAC-33 | Enabled/disabled/unavailable/erasure integration tests prove source and degraded-result behavior. | 110 | + +## Ratification record + +| Drafts | Decision | Durable implementation/status record | +| --- | --- | --- | +| DUN-14, DFR/DAC-24–26 | Ratified and delivered | Slices 10, 20, 30 | +| DFR/DAC-27 | Ratified and delivered | Slice 40 | +| DFR/DAC-28–29 | Ratified and delivered | Slices 50, 60 | +| DFR/DAC-30 | Ratified and delivered | Slice 70 | +| DFR/DAC-31 (routing/affinity) | Ratified as 31a and delivered | Slice 80 | +| DFR/DAC-31 (QoS/telemetry) | Ratified as 31b and delivered | Slice 90 | +| DFR/DAC-32 | Rejected for 0.5.14; retained for 0.5.15 | 0.5.15 Slice 100 | +| DFR/DAC-33 | Ratified, implemented, and independently reviewed in Slice 110. | Slice 110 | + +## Draft CRUD actions to canonical documentation + +| Canonical artifact | Draft action | Conditions | +| --- | --- | --- | +| `dev/user-needs.md` | Add DUN-14; clarify that logging can be configured to redact content, so comprehensive audit logging is not an unconditional raw-content guarantee. | Slice 10 approval. | +| `dev/requirements.md` | Add DFR-24 through DFR-33 and revise stale health wording to `/health/liveliness`. | Ratify only approved feature drafts. | +| Provider design | Keep the detailed OpenRouter/DeepSeek design as the authority for DFR-27–29; add a supersession pointer if future feature-slice design changes it. | Slice 40–60 reviews. | +| TUI backlog | Split existing issue statements into DFR-30–32 and retain issue links/evidence. | Slice 70–100 reviews. | +| Fathom architecture/ops docs | Add a source-of-truth and failure-mode contract only if DFR-33 is approved. | Slice 110 review. | + +## Architecture draft allocations + +- Slice 30 owns the capability contract and embedding request seam; it must not + make `chat` aliases appear embedding-capable by inference. +- Slice 40 owns cross-provider discovery/error/attribution primitives. Slices + 50 and 60 must consume those primitives rather than add parallel provider + transports or logging behavior. +- Slices 80–100 own read/management seams but may not subscribe UI logic to + request callbacks or persist key secrets. +- Slice 100 is deferred to 0.5.15. It must reconcile with 0.6.0 tenant-key/ + keystore work before design; it may deliver a thin existing-key operator + surface, but not invent a competing durable credential store. diff --git a/dev/plans/0.5.14/04-architecture-review.md b/dev/plans/0.5.14/04-architecture-review.md new file mode 100644 index 0000000..2c63497 --- /dev/null +++ b/dev/plans/0.5.14/04-architecture-review.md @@ -0,0 +1,49 @@ +# Slice 4 — architecture review and code-alignment proposal + +**Scope:** high-level review of the approved candidate work against the current +architecture. No code or architecture document changes are made by this record. + +## Findings and proposed architecture changes + +| Area | Current alignment evidence | Proposed change | Owning slice | +| --- | --- | --- | --- | +| Model capabilities | `airlock/capability.py` has one model-entry contract and its endpoint list presently covers chat/batch, not embeddings. | Add an explicit per-entry capability declaration; derive `/v1/models` capability output and embedding dispatch eligibility from it. Never infer embedding support from provider prefix alone. | 30 | +| Embedding policy/logging | Current guard/logging seams were built around chat request events. | Characterize the embedding payload at every guardian/recorder/projection seam; add adapters only where data shape differs. Apply the same auth, PII policy, redaction, and safe-error contract before dispatch. | 30, 10 | +| Provider discovery | `airlock/models_catalog.py` has provider-specific discovery and a generic OpenAI-compatible seam; the provider design identifies unsafe fixed-base/error-text behavior. | Add one reviewed configured-base validation/normalization path, no redirects, no key on unsafe/conflicting base, and bounded errors. OpenRouter/DeepSeek wrappers use it. | 40 | +| Provider attribution | Served provider is derived from LiteLLM model/provider information; classification lacks the planned `openrouter` gateway and `deepseek` native terms. | Extend the existing attribution classification, preserving the immediate `openrouter` gateway token and never guessing its downstream host. | 40, 50, 60 | +| Provider failure propagation | Request event, monitor, tracing, and enterprise projections can receive provider exception text through different paths. | Centralize a bounded sanitizer before artifact boundaries; preserve typed HTTP status for policy/circuit logic before replacement. | 40 | +| Provider transport | LiteLLM 1.94.1 supplies both provider adapters and request/stream parsing. | Do not add custom adapters or a second OpenAI-compatible transport. DeepSeek adds only narrow pre-dispatch non-function-tool validation if characterization confirms LiteLLM drops those tools. | 50, 60 | +| TUI lifecycle | The TUI app constructs background workers even when tests assert only composition/navigation. | Add a test-only lifecycle profile/fixture around real widget construction; preserve named production-worker integration tests. | 70 | +| TUI read model | TUI backlog needs current state that may otherwise be callback-driven. | Define bounded, pull/read snapshots or authenticated read endpoints with source and staleness; no UI subscription on the inference hot path. | 80, 90, 100 | +| Virtual keys | 0.6.0 separately plans keystore/identity. | Deferred to 0.5.15: treat it as a UI/control-plane adapter to an existing management seam. Do not create a second persistence/identity system. | 0.5.15 Slice 100 | +| FathomDB | FathomDB is an optional lazy request-analysis sink; the TUI and Advisor worker run in separate processes from the proxy. | Define a selectable backend contract with the proxy as sole engine owner. Route opted-in TUI/Advisor reads through loopback-only proxy-admin views; never open the shared DB from UI processes. Retain explicit source labels, JSONL fallback, and degraded/erasure states. | 110 | + +## High-level code alignment verdict + +The existing code is broadly aligned with the 0.5.14 direction: configuration +is model-list driven; `models_catalog`, `capability`, `transparency`, guardian, +callbacks, and TUI are separable seams; optional FathomDB already has dedicated +callback/query modules. It is **not yet aligned** with embedding support, +provider discovery safety/sanitized error propagation, new provider +classification, or bounded TUI read contracts. Those are implementation +requirements, not cosmetic documentation updates. + +The canonical needs/requirements documents also lag current behavior: they +describe unconditional message/response logging and contain stale `/health` +wording. Slice 3's drafts resolve that mismatch; implementation must not claim +conformance until docs, tests, and code are updated together. + +## Architectural guardrails for all delivery slices + +1. Explicit configuration remains the authorization boundary; environment keys + and catalog discovery do not expose models. +2. Secrets, raw provider exceptions, key material, prompts, and completions do + not enter tracked configuration, tests, TUI snapshots, or safe artifacts. +3. Keep LiteLLM as provider transport/parser and preserve typed errors before + sanitization. +4. New read/UI work is bounded and fails visibly; optional FathomDB never + silently replaces the zero-infrastructure JSONL/in-memory path. The proxy + alone owns its engine; a separate-process TUI or Advisor reaches it only via + a loopback-only admin bridge. +5. The existing PII observe-only release gate and FathomDB/LiteLLM memory + blocker are recorded constraints, not workarounds for this release. diff --git a/dev/plans/0.5.14/05-verification-review.md b/dev/plans/0.5.14/05-verification-review.md new file mode 100644 index 0000000..9df0640 --- /dev/null +++ b/dev/plans/0.5.14/05-verification-review.md @@ -0,0 +1,49 @@ +# Slice 5 — verification adequacy review + +**Purpose:** assess traceability and test sufficiency before feature delivery. +This review proposes tests; it does not add or modify them. + +## Traceability assessment + +| Candidate requirement group | Acceptance criteria status | Existing evidence | Required addition | +| --- | --- | --- | --- | +| Benchmark chat alias | Drafted in DAC-24. | Alias/config, proxy, transparency, and harness tests exist, but no shipped `gpt-4o-mini` model-list contract. | Configuration/template parity; normal and stream boundary tests; funded smoke worksheet. | +| Embeddings | Drafted in DAC-25. | `capability` compatibility tests presently treat embeddings as unsupported; no endpoint contract proves Airlock dispatch. | RED endpoint/capability/auth/option/error/redaction tests, then mocked provider integration and funded smoke. | +| Benchmark redaction | Drafted in DAC-26. | Enterprise logger has `messages,response` redaction tests; SQL/Fathom projection behavior is separate. | Profile-level sentinel proof across enabled sinks and documentation-contract check. | +| Provider foundation | Detailed AC1–AC13 plus DAC-27. | Models-catalog, attribution, monitor, proxy-error, logger/projection tests exist. | Cross-sink sanitized-error matrix; unsafe-base/no-redirect tests; full typed status regression. | +| OpenRouter / DeepSeek | Detailed provider design AC1–AC13 and DAC-28/29. | LiteLLM pin characterization and no-credit configurations can be tested locally. | Provider-specific mocked discovery, stream/error, attribution, tool, and funded-smoke coverage. | +| TUI lifecycle | DAC-30 drafted. | Extensive TUI and thread-safety tests exist; duration report identifies slow lifecycle tail. | Separate deterministic component suite and named worker lifecycle suite; before/after duration evidence. | +| TUI diagnostics | DAC-31 drafted; DAC-32 deferred to 0.5.15. | TUI, admin, state, priority, metrics, and token tests exist. | Read-source/staleness/unavailability/auth/destructive-action/no-secret tests. | +| FathomDB operational reads | DAC-33 drafted. | Fathom init/logger/query/search/erasure tests exist. | End-to-end selectable-backend, unavailable, bounded-result, source-label, and incomplete-erasure tests. | +| Documentation release index | Existing documentation contract should pin the active plan. | Strict MkDocs build passes, but `tests/test_documentation_contract.py` currently expects `plans/0.5.10-plan.md` while `dev/README.md` names 0.5.12/0.5.14. | Add a focused release-index contract repair and re-run the strict docs/contract gate. | + +## Overall adequacy verdict + +The repository has broad unit and integration coverage (3,298 collected tests) +and CI gates for docs, full-extra tests, lint, Docker build, and pip-audit. +It does **not** yet have a release traceability matrix that proves every 0.5.14 +draft requirement has acceptance criteria and executable evidence. The critical +paths below require explicit release-level evidence before publication. + +The current docs contract test has one confirmed baseline failure: it asserts +the superseded 0.5.10 active-plan path while the developer index has already +moved to 0.5.12 as published and 0.5.14 as backlog. Slice 120 owns the small +test/index reconciliation; it must preserve the test's intent to detect future +release-index drift. + +| Critical path | Required verification | +| --- | --- | +| Proxy ingress to configured model | Auth/alias/guard/dispatch/served-header normal and stream tests; no unintended fallback. | +| Content and artifact safety | PII/redaction sentinel tests through every enabled persistence sink; error/credential/user-ID sentinel absence. | +| Embedding ingress | Correct endpoint/capability/model/option handling, policy and recorder compatibility, safe failures. | +| Provider gateway behavior | Configured-base validation, catalog non-authority, attribution, status classification, stream errors, and no raw metadata. | +| Operator control plane | Auth, source/staleness, bounded reads, destructive action audit, and no secret material in reads/logs. | +| Optional datastore | Enabled/disabled/unavailable lifecycle, one-owner constraint, partial-data honesty, and per-client erasure. | +| Release system | Locked dependency check, full non-live tests, strict docs, lint/mypy scope, Docker, pip-audit reassessment, and funded smokes only after automation passes. | + +## Required delivery evidence format + +Each feature-slice status record must map: draft ID → ratified requirement → +acceptance criterion → test path/command → outcome → manual evidence (if any). +A passing unit test alone is insufficient for changes that cross the proxy, +provider, persistence, or TUI boundaries. diff --git a/dev/plans/0.5.14/06-hitl-release-review.md b/dev/plans/0.5.14/06-hitl-release-review.md new file mode 100644 index 0000000..ab8bbb4 --- /dev/null +++ b/dev/plans/0.5.14/06-hitl-release-review.md @@ -0,0 +1,61 @@ +# Slice 6 — HITL release review + +**Decision recorded 2026-08-12:** the operator instructed implementation of the +plan. Adopt the recommended resolution below: include slices 10–90 and 120; +retain slices 100/110 and the LiteLLM/FathomDB upgrades as conditional; postpone +the broad documentation lifecycle execution and unrelated routine upgrades. +Slice 110 and the FathomDB 0.8.22 upgrade subsequently satisfied their +conditions; funded provider/embedding smokes were explicitly authorized and +passed. Release-closeout mechanics are now tracked separately as Slice 130. + +A conditional item receives a delivery slice but may not start implementation +until its condition is met. This record consolidates Slices 0–5 and contains no +feature implementation. + +Scales: understanding and risk use 1–4 (4 = highest); effort is engineering +size excluding review/funded-smoke wait. Recommendations reflect the currently +approved priority: FathomDB needs, providers, TUI, then operational backend. + +| Candidate / slice | Drafts | Understood | Risk | Effort | Recommendation | Decision / condition | +| --- | --- | ---: | ---: | --- | --- | --- | +| Benchmark-safe logging profile / 10 | DFR/DAC-26 | 4 | 2 | S | Include | None; profile must keep SQL and Fathom raw content off. | +| `gpt-4o-mini` chat readiness / 20 | DFR/DAC-24 | 4 | 2 | S | Include | No implicit enable/fallback. | +| Embeddings and `text-embedding-3-small` / 30 | DFR/DAC-25 | 3 | 3 | M | Include | Ratify endpoint/capability contract before code. | +| Shared provider foundation / 40 | DFR/DAC-27 | 4 | 4 | M | Include | Must precede OpenRouter/DeepSeek. | +| OpenRouter / 50 | DFR/DAC-28 | 4 | 3 | M | Include | Curated aliases; operator-only routing/privacy policy. | +| DeepSeek / 60 | DFR/DAC-29 | 4 | 3 | M | Include | Stable API base; function tools only. | +| LiteLLM patch 1.96.2 | Slice 1 | 3 | 4 | M | Conditional | Characterize against 1.94.1 first; merge only if provider/embedding matrix is green. | +| FathomDB patch 0.8.22 | Slice 1 | 3 | 3 | S | Included, verified | Locked and installed; 110 DB-extra lifecycle/query/erasure/admin-operational tests passed. | +| TUI test lifecycle / 70 | DFR/DAC-30 | 4 | 2 | M | Include | Keep production-worker integration subset. | +| TUI routing/client diagnostics / 80 | DFR/DAC-31 | 3 | 3 | M | Include | Requires bounded source/staleness seam. | +| TUI QoS/exporter health / 90 | DFR/DAC-31 | 3 | 3 | M | Include | Instrument source before display. | +| Virtual-key management / 100 | DFR/DAC-32 | 2 | 4 | L | Postpone to 0.5.15 | Retain the draft package; complete design only after 0.6.0 keystore/identity reconciliation. | +| FathomDB operational reads / 110 | DFR/DAC-33 | 3 | 4 | L | Conditional | Single owner, source labels, fallback, and erasure honesty. | +| Documentation release-index contract / 120 | Slice 5 finding | 4 | 1 | XS | Include | Repair the stale 0.5.10 assertion while retaining active-release drift detection. | +| Documentation lifecycle execution | Slice 2 | 4 | 2 | M | Postpone | Keep proposal only in 0.5.14; approve a focused cleanup train later. | +| Non-LiteLLM/Fathom routine patch upgrades | Slice 1 | 3 | 2 | S–M | Postpone | Dependabot handles them unless advisory/feature need changes priority. | + +## Adopted HITL resolution + +1. **Include:** slices 10, 20, 30, 40, 50, 60, 70, 80, 90, and 120 in that order. +2. **Conditional:** LiteLLM patch upgrade only. Slice 110 and the FathomDB + 0.8.22 patch condition are complete. Slice 100 is postponed to 0.5.15. +3. **Postpone:** broad documentation lifecycle execution and unrelated routine + library updates. + +## Release gates that remain outside this decision + +- PII egress remains observe-only until its existing owner DECIDE and evidence + threshold are satisfied. +- The LiteLLM-child anonymous-memory owner is unresolved; FathomDB benchmark + integration does not close it. +- Funded smokes require explicit operator authorization after no-credit tests; + they do not run in CI and record no request/response content. The authorized + 0.5.14 embedding, OpenRouter, and DeepSeek smokes passed on 2026-08-12. + +## HITL response form + +Record one line per conditional item: `include`, `postpone`, or `retain +conditional`, plus any new release constraint. After the decision, change this +record's status and start the first approved feature slice using the required +TDD/review/status workflow. diff --git a/dev/plans/0.5.14/10-benchmark-safe-logging-status.md b/dev/plans/0.5.14/10-benchmark-safe-logging-status.md new file mode 100644 index 0000000..c59e0cd --- /dev/null +++ b/dev/plans/0.5.14/10-benchmark-safe-logging-status.md @@ -0,0 +1,42 @@ +# Slice 10 — benchmark-safe logging profile status + +**Status:** complete; no funded smoke required for this configuration/documentation +slice. + +## Ratified scope + +- DFR-26: publish a benchmark profile that redacts enterprise JSONL + `messages,response` and disables SQL plus Airlock's optional Fathom request + logger/raw-content paths. +- DAC-26: prove a persisted enterprise JSONL line contains `[REDACTED]` and no + distinctive request/response sentinel; document the safe liveness probe. + +## Design and code review + +The design deliberately changes no runtime default. `AIRLOCK_LOG_REDACT_FIELDS` +is already an opt-in enterprise-log writer boundary; SQL projection is known to +be unredacted, while the Fathom content fields are explicit opt-ins. The profile +therefore turns off SQL/S3/Fathom request logging and keeps Fathom content flags +off, rather than claiming global sink redaction. The documentation clearly +separates logging retention from outbound PII enforcement. + +Review confirmed the implementation matches the seams in +`enterprise_logger._write_log`, `projections.project_sql`, and +`projections.project_fathom`; it exposes no secret, prompt, or response in a +new artifact and changes no production code path. + +## TDD and verification evidence + +| Phase | Command / result | +| --- | --- | +| RED | `uv run --extra test python -m pytest tests/test_documentation_contract.py -q -k benchmark_safe_logging_profile` failed because the required profile did not exist. | +| GREEN | The same command passed: `1 passed, 5 deselected`. | +| JSONL evidence | `uv run --extra test python -m pytest tests/test_enterprise_logger.py -q -k redaction_applied_in_write_log` passed: `1 passed, 66 deselected`. The test checks both sentinels are absent from the written line. | +| Quality | Ruff check/format passed for changed tests; `uv run --extra docs mkdocs build --strict --site-dir /tmp/airlock-0.5.14-docs` passed. | + +## Remaining release note + +The complete `tests/test_documentation_contract.py` suite still has an existing +unrelated failing assertion for `plans/0.5.10-plan.md`. Slice 120 owns that +release-index contract repair. It does not invalidate the Slice 10 targeted +profile evidence. diff --git a/dev/plans/0.5.14/110-fathomdb-operational-reads-design.md b/dev/plans/0.5.14/110-fathomdb-operational-reads-design.md new file mode 100644 index 0000000..ada2888 --- /dev/null +++ b/dev/plans/0.5.14/110-fathomdb-operational-reads-design.md @@ -0,0 +1,48 @@ +# Slice 110 — FathomDB operational reads design review + +## Ratification + +The owner authorized Slice 110, so **DFR-33/DAC-33 are ratified** with these +adjustments: + +- `AIRLOCK_OPERATIONAL_READ_BACKEND=fathomdb` is the only opt-in. The default + is `jsonl`, even when a FathomDB engine happens to exist. +- The initial supported readers are TUI history and Advisor error/search reads. + Billing, retention, and erasure remain their existing explicitly scoped + mechanisms; a FathomDB erasure receipt continues to say it does **not** erase + JSONL. +- Every read is bounded, reports its actual source, and reports FathomDB + unavailability as an explicit JSONL fallback. Invalid backend selection also + falls back to JSONL with a reason. + +## Architecture + +`airlock.operational_reads` is a pull-only selection seam. It calls the +existing PID-bound `datastore.get_engine()` only when the operator selected +FathomDB; it never initializes FathomDB for default JSONL users. FathomDB list +reads use existing `api.queries.get_request_logs()` under its read limit, then +apply the caller's existing predicate and time filter. Advisor search uses the +existing active-only `api.queries.search_request_logs()` path: FTS-only is +labelled `lexical_only` when dense retrieval is unavailable, and hybrid mode +is labelled only when it actually contributes. JSONL uses `log_query` with the +same record bound. The returned page carries `source`, `degraded_reason`, +`truncated`, and `limit_hit`. + +The proxy owns the FathomDB engine. Because the TUI and its Advisor worker are +separate processes, they reach proxy-owned FathomDB reads through loopback-only +admin endpoints (`admin.enabled: true`, `trust_loopback: true`); they never +open the shared database. They retain their +historical JSONL behavior by default and only show a FathomDB result if +explicitly opted in. The TUI displays source/degraded/truncation state. No +request callback, UI event, or secondary process opens a shared writer. +`datastore` remains one engine per owning process; separate-process writers are +unsupported. + +## TDD plan + +RED tests establish default JSONL even with an engine, opted-in FathomDB, +unavailable/invalid fallback, bounded/truncated result truth, TUI source +rendering, proxy-owner bridge authorization, Advisor source propagation, and an +erasure receipt that remains incomplete for JSONL. GREEN adds the shared reader +and minimal consumers. No provider call or real FathomDB data outside temporary +test databases is used. diff --git a/dev/plans/0.5.14/110-fathomdb-operational-reads-status.md b/dev/plans/0.5.14/110-fathomdb-operational-reads-status.md new file mode 100644 index 0000000..f8aed84 --- /dev/null +++ b/dev/plans/0.5.14/110-fathomdb-operational-reads-status.md @@ -0,0 +1,48 @@ +# Slice 110 — FathomDB operational reads status + +**Status:** complete + +## Delivered contract + +- `AIRLOCK_OPERATIONAL_READ_BACKEND=jsonl` remains the default. FathomDB is + selected only by `fathomdb`, never merely because its writer is enabled. +- The proxy is the sole FathomDB engine owner. The separate-process TUI and + Advisor reach selected reads through loopback-only + `/airlock/admin/operational/{records,errors,search}` views, which require + `admin.enabled: true` and `trust_loopback: true`. +- A missing bridge, unavailable store, invalid selection, or query failure + yields bounded JSONL with a source and degradation reason; it never opens a + second engine from a UI/Advisor process. +- TUI history shows source, degradation, and bounded-result state. Advisor + errors carry their source/window; Advisor FathomDB search uses the existing + active-only FTS/hybrid seam, truthfully labels lexical-only operation, and + marks any pre-time-filter full FTS page as partial. +- Erasure is unchanged and explicit: a FathomDB receipt does not erase JSONL. + +## Independent review / FIX-n evidence + +`gpt-5.6-terra` independent review completed three cycles: + +1. Fixed an Advisor bridge failure that could have opened the proxy-owned DB + from a secondary process, and validated all LLM-controlled operational-read + arguments at the admin boundary. +2. Preserved the pre-time-filter FTS page cap, so a stale full page cannot be + presented as a complete empty time window. +3. Final review approved with no remaining blockers. + +## Verification + +- Focused Slice 110 suite: **23 passed, 56 deselected** (independent review). +- Local final suite: **79 passed** across operational-read, Advisor, FathomDB, + admin, and TUI-thread-safety tests; ruff on all Slice 110 Python files and + tests passed; strict MkDocs build and `git diff --check` passed. +- Verification used temporary FathomDB test databases only; no live provider or + `.env` credential was read. + +## Dependency update + +FathomDB was upgraded from 0.8.21 to **0.8.22** under this slice's conditional +dependency gate. The exact package version was installed from the refreshed +lockfile and the DB-extra suite passed (110 tests), covering initialization, +projection declaration, bounded active-only reads/search, provenance erasure, +and the proxy-owned operational-read bridge. diff --git a/dev/plans/0.5.14/120-documentation-contract-status.md b/dev/plans/0.5.14/120-documentation-contract-status.md new file mode 100644 index 0000000..c7569b5 --- /dev/null +++ b/dev/plans/0.5.14/120-documentation-contract-status.md @@ -0,0 +1,22 @@ +# Slice 120 — release documentation contract status + +**Status:** complete. + +## Review and ratified change + +The active engineering index is 0.5.14, while the prior contract test still +asserted the historical 0.5.10 plan path. That assertion no longer represented +the active release and would either fail every documentation run or encourage a +future maintainer to restore stale guidance. + +The contract now requires `dev/README.md` to link to +`plans/0.5.14-todo.md`. The page already does so under “Next engineering +backlog”; the test makes this active-release linkage durable. Published 0.5.12 +material remains correctly labelled as the latest published release. + +## Verification + +The documentation-contract test runs in the Slice 50 focused no-credit suite. +It verifies all MkDocs navigation and local links, entry-point links, the active +plan reference, the benchmark-safe profile, and the OpenRouter configuration +contract. Strict MkDocs build remains a release-closeout check. diff --git a/dev/plans/0.5.14/130-release-closeout.md b/dev/plans/0.5.14/130-release-closeout.md new file mode 100644 index 0000000..33206a9 --- /dev/null +++ b/dev/plans/0.5.14/130-release-closeout.md @@ -0,0 +1,61 @@ +# Slice 130 — 0.5.14 release closeout + +**Status:** ready to tag locally. + +## Purpose + +Convert the completed, independently reviewed 0.5.14 delivery slices into a +reproducible local release candidate. This is release engineering, not new +feature scope. + +## Inputs + +- Approved and complete delivery slices 10–90, 110, and 120. +- Slice 100 remains deferred to 0.5.15. +- Funded, isolated-loopback smokes for embeddings, OpenRouter, and DeepSeek + passed; their safe evidence is retained in the relevant slice records. +- PII egress remains observe-only under its existing owner decision. + +## Closeout design + +1. Bump the package and runtime version to `0.5.14` and write a changelog + section based only on completed scope. +2. Run the local equivalents of every GitHub CI/release gate with the locked + dependency graph: strict documentation/contracts, non-live tests with + coverage, lint/format/type checks, Docker build, `pip-audit`, lock check, + and package build. +3. Record exact commands and terminal results here. Any failure is a release + blocker: repair it, rerun the affected gate, then rerun the final matrix as + warranted. +4. Once all local gates pass, create an annotated **local** `v0.5.14` tag on + the exact release commit and verify its version/artifacts. +5. Do not push `main` or the tag, trigger GitHub Actions, create a GitHub + release, or publish to PyPI without a further explicit operator approval. + A tag push invokes the release workflow and trusted publishing. + +## Local acceptance criteria + +- `pyproject.toml` and `airlock.__version__` both equal `0.5.14`. +- `CHANGELOG.md` accurately represents delivered 0.5.14 work and names no + deferred item as shipped. +- `uv lock --check` passes. +- Every command listed in `.github/workflows/ci.yml` and the local-safe checks + from `.github/workflows/release.yml` has a terminal success result, or an + explicitly justified environment limitation is elevated to HITL. +- `uv build` produces an sdist and wheel whose metadata reports `0.5.14`. +- The final release commit and local annotated tag are clean and reproducible. + +## Evidence log + +| Gate | Command | Result | +| --- | --- | --- | +| Version/changelog | `tests/test_config_consistency.py::TestVersionConsistency::test_versions_agree`, `tests/test_tracing.py`, `scripts/check-version-consistency.py --tag v0.5.14` | Passed: 5 tests; package, runtime, tracing, and tag values agree on `0.5.14`. | +| Locked environment | `UV_CACHE_DIR=/tmp/airlock-uv-cache make sync`; `uv lock --check` | Passed. | +| Documentation | strict MkDocs build; `uv run pytest tests/test_documentation_contract.py -q` | Passed: strict build completed and 8 contract tests passed. | +| Test | `uv run pytest -q -m "not live" --cov=airlock --cov-report=xml --cov-report=term-missing` | The full run completed except for one static tracing-version literal mismatch. The one-line literal repair received the focused version/tracing regression above; owner approved proportional revalidation rather than a second full run. | +| Quality | `uv run ruff check airlock/ tests/`; `uv run ruff format --check airlock/ tests/`; `uv run mypy airlock/fast/ --ignore-missing-imports` | Passed: Ruff check, 326-file format check, and mypy (16 source files). | +| Container | `docker build -t airlock:0.5.14-local .` | Passed. Local image ID: `sha256:2eba403f8f58da836523772404dcbfacde64395b183001c5306c38eec559965d`. | +| Security | `uv run pip-audit --ignore-vuln PYSEC-2026-3552 --ignore-vuln PYSEC-2026-3553 --ignore-vuln PYSEC-2026-3554` | Passed: no known vulnerabilities; the three documented suppressions remain. Local/project-only `airlock-llm` and `en-core-web-lg` are not on PyPI and were reported as unauditable. | +| Artifact | `uv build`; wheel/sdist `METADATA` / `PKG-INFO` checks | Passed: both artifacts report `Version: 0.5.14`. SHA-256: sdist `f83953f3bf6c069f132d782818258182d96a6d750a1aee6b0d1f409a38255086`; wheel `9445945fc3439e7914b00cd498dfbb38d7b58a44be2893eb7289a137819d1af3`. | +| Release commit/tag | local commit and annotated `v0.5.14` tag | Pending: create on the exact checked release revision. | +| Remote publication | push / GitHub Actions / PyPI | intentionally pending explicit approval | diff --git a/dev/plans/0.5.14/20-fathomdb-chat-status.md b/dev/plans/0.5.14/20-fathomdb-chat-status.md new file mode 100644 index 0000000..3c31a61 --- /dev/null +++ b/dev/plans/0.5.14/20-fathomdb-chat-status.md @@ -0,0 +1,34 @@ +# Slice 20 — FathomDB chat readiness status + +**Status:** complete. Funded smoke remains a release-closeout action after all +no-credit feature tests are green. + +## Ratified scope + +- DFR-24/DAC-24: add explicit `gpt-4o-mini` and + `openai/gpt-4o-mini` aliases backed by `openai/gpt-4o-mini` and + `OPENAI_API_KEY`; preserve ordinary Airlock chat behavior without fallback. + +## Design and code review + +The aliases follow Airlock's existing bare/provider-prefixed model-list pattern. +They do not modify the router's fallback map, cost tiers, authentication, +guardrails, transport, or provider code. The capability helper derives +`openai` and `chat` from their normal model-list body, so no new special case +is necessary. + +## TDD and verification evidence + +| Phase | Command / result | +| --- | --- | +| RED | `uv run --extra test python -m pytest tests/test_config_consistency.py -q -k chat_benchmark_aliases_are_explicit` failed four times because neither config carried the aliases. | +| GREEN | The same command passed: `4 passed, 53 deselected`; it checks root/template parity, model body, environment key reference, `openai` attribution, chat-only capability, and absence from fallback chains. | +| Compatibility | `uv run --extra test python -m pytest tests/test_0_5_2_capability_compat.py -q -k 'capability or alias'` passed: `54 passed`. | +| Quality | Ruff check/format passed for the changed test and strict MkDocs build passed. | + +## Release-closeout evidence + +After the non-credit release matrix passes, run a manually authorized, +non-sensitive ordinary and streaming request through each documented benchmark +alias. Record only alias, timestamp, HTTP/stream outcome, and safe Airlock +headers—never keys, prompts, completions, or raw provider headers. diff --git a/dev/plans/0.5.14/30-embeddings-design.md b/dev/plans/0.5.14/30-embeddings-design.md new file mode 100644 index 0000000..e0bd888 --- /dev/null +++ b/dev/plans/0.5.14/30-embeddings-design.md @@ -0,0 +1,42 @@ +# Slice 30 — embedding capability design + +## Decision + +Add explicit embedding-only model entries for `text-embedding-3-small` and +`openai/text-embedding-3-small`. A truthful capability record lists +`["embeddings"]`, not chat. LiteLLM remains the `/v1/embeddings` transport; +Airlock adds no provider client or response parser. + +## Boundary + +The fast guardian resolves embedding requests only against an explicitly +configured alias marked embedding-capable. It rejects a chat-only configured +alias, an unconfigured name, and `smart` before dispatch; it does not fuzzy +resolve, route, or fall back an embedding model. Ordinary authentication, +admission, provider protection, recorder, and attribution continue after that +validation. + +`input` is the embedding text surface. Text extraction and the PII guard must +scan/redact string input values before downstream keyword/threat guards run. +Embedding requests retain no PII hydration map because embeddings have no +tool-call response to hydrate. The public request body is an explicit allowlist: +`model`, `input`, optional `user`, optional `dimensions` (integer 1–1536), and +optional `encoding_format` (`float` or `base64`). Proxy-created internal fields +are permitted only when they are safe to accept after the proxy adds them. An +early Airlock ASGI boundary validates the raw JSON *before* LiteLLM adds proxy +fields, so it can reliably distinguish a client option from a proxy-created +field. It covers LiteLLM's `/v1/embeddings`, `/embeddings`, engine, and Azure +deployment embedding routes. Direct-dispatch and header controls—including +`api_base`, `api_key`, provider/deployment fields, forwarding headers, and +client metadata—are rejected there; LiteLLM can otherwise honor body-supplied +values rather than overwrite them. Any other client option receives a clear 400 +before dispatch; it must never be silently removed by `drop_params`. + +## Verification + +No-credit tests prove the config/template aliases, truthful capability record, +guardian acceptance/rejection and routing/failover suppression, option +pass-through/rejection, extraction/redaction of `aembedding` input, real +LiteLLM error translation, and existing model-info surface. A later manually +authorized smoke uses a non-sensitive batch input and records only alias, +outcome, and safe headers. diff --git a/dev/plans/0.5.14/30-embeddings-status.md b/dev/plans/0.5.14/30-embeddings-status.md new file mode 100644 index 0000000..e83804f --- /dev/null +++ b/dev/plans/0.5.14/30-embeddings-status.md @@ -0,0 +1,78 @@ +# Slice 30 — FathomDB embedding readiness status + +**Status:** complete; funded smoke passed. + +## Ratified scope + +- DFR-25/DAC-25: serve `/v1/embeddings` only through the explicit + `text-embedding-3-small` and `openai/text-embedding-3-small` aliases. +- Preserve strings and string batches, use normal Airlock authentication, + policy, redaction, attribution, and observability seams, and advertise the + alias as embedding-only. +- Pass `dimensions` (integer 1–1536) and `encoding_format` (`float` or + `base64`) through unchanged. Reject all other client options before dispatch; + never silently drop them. + +## Design and review outcome + +The implementation uses the existing LiteLLM transport. An explicit +`airlock_embeddings` marker supplies the single capability source of truth. +The guardian performs exact configured-alias lookup only and treats every +embedding request as pinned: client routing preferences, smart routing, +circuit-failover substitution, LiteLLM fallback, and retries cannot change the +validated alias. + +Input text passes through Airlock PII redaction before downstream checks; no +hydration map is retained because embeddings have no tool-call response. The +validation has two layers: an authenticated endpoint wrapper validates raw +client JSON after LiteLLM has authenticated and cached it, but before the +embedding endpoint or proxy augmentation run; the guardian validates the +augmented payload. This rejects direct dispatch and header controls before +LiteLLM can honor body-supplied values without pre-auth body buffering. Unknown +client fields fail with an OpenAI-shaped 400 and `invalid_embedding_option` +before transport. + +An independent `gpt-5.6-terra` high-reasoning reviewer completed the original +three FIX cycles, then the owner authorized two additional as-needed cycles. +The reviews drove correction of routing/failover bypass, LiteLLM's internal +error translation, incomplete option rejection, missing `aembedding` policy +evidence, proxy-created fields, and client-controlled direct-dispatch/header +controls. The fifth and final authorized review found alternate endpoint +coverage missing. The owner subsequently authorized completion of the slice; +the route-coverage correction then exposed a pre-auth body-buffering problem. +The final review approved the corrected authenticated endpoint wrapper: it +preserves LiteLLM authentication before validation, covers all four embedding +routes, and retains pinned exact-alias handling. + +## TDD and verification evidence + +| Phase | Command / result | +| --- | --- | +| RED | Focused capability/extraction tests initially failed because `is_embedding_call` did not exist. | +| GREEN | Capability, extraction, config, guardian, proxy-error, and legacy capability-compatibility subset passed: `282 passed, 1 xpassed`. | +| Review fixes | Embedding-focused guardian/proxy tests passed: `13 passed, 54 deselected`; PII embedding helper/hook tests passed: `2 passed`. | +| Ingress fix | Raw-body boundary, identity precedence, guardian, proxy-error, and bootstrap tests passed: `77 passed, 54 deselected`. | +| Auth-boundary fix | Authenticated wrapper, proxy bootstrap, guardian, proxy-error, and identity tests passed: `136 passed`. | +| Final independent review | Approved; no correctness or security blocker. | +| Quality | Ruff check/format passed for all Slice 30 source/tests; `mkdocs build --strict` passed. | +| Integrity | `git diff --check` passed. | + +The broad PII module command did not return a terminal summary in this sandbox +after starting its expensive suite; its two changed embedding tests passed +individually with deterministic stubbed redaction. This is a verification +environment limitation, not a release claim; rerun the full PII suite in CI or +a writable home-cache environment before release closeout. + +## Release-closeout evidence + +After all no-credit release tests are green, an operator may run one +non-sensitive, authenticated embedding smoke. Record only the requested +alias, time, HTTP outcome, and safe Airlock headers. Do not record keys, input, +vectors, request bodies, provider headers, or raw response payloads. + +**Funded smoke — 2026-08-12:** one isolated-loopback request to +`text-embedding-3-small` returned HTTP 200 with +`X-Airlock-Served-By: openai` and `X-Airlock-Mutations: fallbacks=suppressed`. +The request used the fixed non-sensitive input `smoke`; no vector, key, request +body, raw response, or provider header was retained. This was within the +operator-authorized $1 embedding budget. diff --git a/dev/plans/0.5.14/40-provider-foundation-design.md b/dev/plans/0.5.14/40-provider-foundation-design.md new file mode 100644 index 0000000..aac6fb6 --- /dev/null +++ b/dev/plans/0.5.14/40-provider-foundation-design.md @@ -0,0 +1,52 @@ +# Slice 40 — shared provider foundation design + +**Status:** design ratified for TDD implementation. This slice is the required +foundation for Slices 50 (OpenRouter) and 60 (DeepSeek); it does not add either +provider to the default model list. + +## Scope and requirements + +- DFR-27/DAC-27: provider discovery is explicit-configuration only, + informational, base-bound, no-redirect, and secret-safe. +- Provider attribution classifies the immediate LiteLLM token: `openrouter` is + a gateway and `deepseek` is native. Airlock never guesses an OpenRouter + downstream host. +- Provider failure data is reduced to a bounded provider, error type, and HTTP + status record before it enters request events/projections, monitor state or + logs, and tracing. Typed status inspection happens before sanitization, so + 429 handling remains intact. + +## Design + +1. Add one configured-base resolver in `models_catalog`. It reads only reviewed + `model_list` entries for a provider, resolves an existing environment key, + requires exactly one normalized HTTPS base, rejects credentials, query, + fragments, `/models`, and conflicting bases, and derives the models URL from + an API-root or `/chat/completions` base. It uses a no-redirect opener; an + invalid provider skips discovery without blocking startup. +2. Add thin provider fetchers only after that resolver exists. They reuse the + existing OpenAI-compatible JSON parser; they do not register a model, + authorize it, or mutate routing. +3. Add a small provider-error summary helper with no free-form exception text. + Request-event, monitor, tracing, and enterprise projection consume the + summary while retaining existing typed rate-limit detection. +4. Extend served-backend classification only. Provider transport, streaming, + client retry behavior, and default fallback policy stay in LiteLLM/existing + Airlock seams. + +## RED verification plan + +- Configured-base tests: absent key/base, malformed/conflicting/unsafe base, + no redirect/key leakage, exact slash-preserving ID normalization, and + startup-best-effort behavior. +- Attribution tests for `openrouter` gateway and `deepseek` native. +- Failure-sanitization matrix for 401, 402, 429, 500, and 503, including a + sentinel in exception text/metadata/user identity. It must be absent from all + durable surfaces while a typed 429 still activates existing monitor behavior. + +## Non-goals + +No provider credentials in tracked files; no provider enabled merely by `.env`; +no custom provider transport/parser; no claim to propagate upstream +`Retry-After`; and no alteration to the existing separately governed general +request/response logging posture. diff --git a/dev/plans/0.5.14/40-provider-foundation-status.md b/dev/plans/0.5.14/40-provider-foundation-status.md new file mode 100644 index 0000000..fa64310 --- /dev/null +++ b/dev/plans/0.5.14/40-provider-foundation-status.md @@ -0,0 +1,49 @@ +# Slice 40 — shared provider foundation status + +**Status:** implementation complete and locally verified; independent code review +approved after two cycles. This foundation does not enable a provider or add a +default model. + +## Delivered + +- Discovery is available only when an existing `model_list` entry explicitly + configures an OpenRouter or DeepSeek model, an HTTPS `api_base`, and a + resolvable key. It derives the same-origin `/models` endpoint, rejects unsafe + or conflicting bases, refuses redirects, and treats failure as best-effort. +- Discovered model IDs retain the provider and any slash-separated upstream + model path. Discovery neither registers aliases nor changes routing. +- Served-backend attribution identifies OpenRouter as the immediate gateway and + DeepSeek as the native provider. +- Provider exceptions bearing LiteLLM's provider marker are reduced to provider, + exception type, and HTTP status before request events and every projection, + monitor rate-limit state/logging, or OpenTelemetry tracing receive them. The + existing raw-text handling for Airlock-local validation/evaluation errors is + unchanged. Typed 429 detection still occurs before the bounded reason is used. + +## Verification + +Local focused verification passed on 2026-08-12: + +``` +UV_CACHE_DIR=/tmp/airlock-uv-cache uv run --extra test pytest \ + tests/test_models_catalog.py tests/test_provider_errors.py \ + tests/test_request_event.py tests/test_projections_equiv.py \ + tests/test_tracing.py tests/test_fast_monitor.py \ + tests/test_served_attribution.py -q +# 197 passed +``` + +The added tests cover unsafe and conflicting configured bases, no-redirect +opener use, model-ID normalization, no provider discovery without an explicit +configuration/key, OpenRouter/DeepSeek attribution, and a 401/402/429/500/503 +sentinel matrix across events, enterprise/S3/SQL projections, monitor handling, +and tracing. Independent review cycle 1 also closed two safety gaps: generic +provider configuration cannot impersonate OpenRouter or DeepSeek, and an +unrecognized provider marker is not emitted. Cycle 2 approved the correction. +No funded-provider call was made. + +## Handoff + +Slices 50 and 60 may now use these primitives. Their individual configuration, +provider-specific request behavior, and funded smoke verification remain their +own work; this slice supplies no provider credentials or model aliases. diff --git a/dev/plans/0.5.14/50-openrouter-design.md b/dev/plans/0.5.14/50-openrouter-design.md new file mode 100644 index 0000000..99a77db --- /dev/null +++ b/dev/plans/0.5.14/50-openrouter-design.md @@ -0,0 +1,40 @@ +# Slice 50 — OpenRouter design review + +**Draft review outcome:** approve DFR-28/DAC-28 with one clarification: this +slice adds no built-in alias. A provider exists only in an operator-reviewed +`model_list` deployment configuration; the shipped root and template configs +must remain free of OpenRouter entries and fallback references. + +## Allocated need, requirements, and acceptance criteria + +- **Need:** operators want access to OpenRouter's model catalog without + Airlock overstating control over OpenRouter's downstream routing, privacy, + pricing, retention, or fallback choices. +- **DFR-28 (ratified):** Airlock documents a curated, explicit OpenRouter + deployment recipe using a provider-prefixed alias and stable API base. It + preserves normal Airlock policy, rejects client gateway-routing overrides, + and reports only the immediate `openrouter` gateway. +- **DAC-28 (adjusted):** no default model/configuration or fallback is added; + an explicit configuration is correctly recognized by the alias/capability + contract; informational discovery preserves slash-containing IDs and cannot + authorize a model; documentation names the gateway/privacy boundary; focused + no-credit checks and independent review pass. A funded normal and streaming + smoke remains release closeout evidence. + +## Architecture and implementation decision + +Slice 40 is the sole discovery, attribution, and failure-sanitization seam. +Slice 50 adds no adapter, transport, model registry, fallback, or response +metadata parser. The implementation is configuration documentation plus +contract tests plus one narrow guardian validation: an operator adds an exact alias with +`openrouter//`, an environment reference, and +`https://openrouter.ai/api/v1`. Startup discovery is optional and informational +only; it reports the gateway rather than a guessed downstream provider. + +## TDD plan + +RED tests first require the documented recipe, explicit non-default posture, +provider capability/alias resolution, and gateway boundary wording. GREEN adds +the documentation/environment-template entries and narrow pre-dispatch +validation needed to satisfy those contracts. No live credential is read or +sent. diff --git a/dev/plans/0.5.14/50-openrouter-status.md b/dev/plans/0.5.14/50-openrouter-status.md new file mode 100644 index 0000000..b469819 --- /dev/null +++ b/dev/plans/0.5.14/50-openrouter-status.md @@ -0,0 +1,48 @@ +# Slice 50 — OpenRouter status + +**Status:** complete; funded smoke passed. + +## Ratified scope and delivery + +DFR-28/DAC-28 are approved with an operator-only interpretation of “curated”: +Airlock ships no OpenRouter alias, fallback, or default enablement. An operator +uses the documented exact `model_list` recipe with `OPENROUTER_API_KEY` and the +stable OpenRouter API base. Slice 40 supplies the shared, configured-base +discovery, gateway attribution, and bounded provider-error behavior. + +LiteLLM 1.94.1 recognizes `route`, `models`, and `transforms` as OpenRouter +routing controls. Slice 50 rejects them at the request root or inside +`extra_body`, after final alias/routing/failover resolution identifies +OpenRouter and before dispatch. The typed error reaches clients as an +OpenAI-shaped 400. Other providers and unrelated `extra_body` fields are not +changed by this narrow control. + +Documentation states the immediate gateway boundary: `openrouter` never claims +an OpenRouter downstream host or control over upstream fallback, pricing, +availability, or retention. + +## TDD, review, and verification + +- RED: documentation/default-config/guardian/proxy-error contracts were added. +- GREEN: documented operator recipe, environment-template entry, served-provider + wording, and narrow pre-dispatch validation were added. +- Independent review cycle 1 found missing actual-guardian and LiteLLM wire-path + evidence. Cycle 2 approved the added final-provider and exact-400 tests. +- Local focused verification passed: `217 passed, 1 xpassed`; the reviewer’s + overlapping focused set passed `226 passed, 1 xpassed`. Strict MkDocs build + and `git diff --check` passed. No provider key was read or sent. + +## Release closeout + +After the full no-credit matrix is green, record one ordinary and one streaming +non-sensitive request through an operator-configured alias. Retain only alias, +timestamp, HTTP/stream outcome, and safe Airlock headers; never prompts, +responses, keys, or raw upstream headers. + +**Funded smoke — 2026-08-12:** isolated-loopback ordinary and streaming calls +through temporary alias `smoke-openrouter` both returned HTTP 200 with +`X-Airlock-Served-By: openrouter` and +`X-Airlock-Mutations: fallbacks=suppressed`; the streaming response completed +its SSE sequence. The fixed non-sensitive prompt and all response content, +keys, and raw upstream headers were discarded. This was within the +operator-authorized $1 OpenRouter budget. diff --git a/dev/plans/0.5.14/60-deepseek-design.md b/dev/plans/0.5.14/60-deepseek-design.md new file mode 100644 index 0000000..0088170 --- /dev/null +++ b/dev/plans/0.5.14/60-deepseek-design.md @@ -0,0 +1,25 @@ +# Slice 60 — DeepSeek design review + +**Draft review outcome:** approve DFR-29/DAC-29 with one additional acceptance +criterion: the configured stable API base is mandatory, and no default alias is +shipped. + +## Scope + +An operator configures an exact `deepseek/` alias with +`DEEPSEEK_API_KEY` and `https://api.deepseek.com`. Slice 40 provides optional +configured-base discovery, native attribution, and bounded provider failures. +This slice adds no adapter, parser, retry loop, user-identity mapping, or +default fallback. + +## Architecture and TDD decision + +Pinned LiteLLM drops non-function DeepSeek tools. Airlock’s final-provider +guardian seam validates `tools` only after alias/routing/failover determines +the provider is `deepseek`, and before dispatch. A function-tool list passes +unchanged; absent, malformed, or non-function tool items receive a typed +OpenAI-shaped 400. Tests cover the direct validator and final guardian path; +documentation provides the stable configuration and privacy boundary. + +Funded closeout will later run one normal and one streaming non-sensitive call +through an operator-configured alias; no key is used for no-credit tests. diff --git a/dev/plans/0.5.14/60-deepseek-status.md b/dev/plans/0.5.14/60-deepseek-status.md new file mode 100644 index 0000000..70d6d66 --- /dev/null +++ b/dev/plans/0.5.14/60-deepseek-status.md @@ -0,0 +1,45 @@ +# Slice 60 — DeepSeek status + +**Status:** complete; funded smoke passed. + +## Ratified scope and delivery + +DFR-29/DAC-29 are approved. Airlock ships no DeepSeek alias, fallback, or +automatic enablement. Operators use the documented stable +`https://api.deepseek.com` API base and `DEEPSEEK_API_KEY` in a reviewed exact +`model_list` entry. Slice 40 supplies optional configured-base discovery, native +served-provider attribution, and bounded provider failure handling. + +Pinned LiteLLM drops non-function DeepSeek tools. The guardian therefore checks +the final resolved provider before dispatch: function tool lists pass unchanged; +missing/malformed/non-function tool values fail as an OpenAI-shaped 400. The +implementation does not map Airlock client or authenticated identity data to +DeepSeek `user_id`. + +## TDD, review, and verification + +- RED: stable-endpoint docs, function-tool boundary, typed error, and default + configuration contracts were added. +- GREEN: narrow final-provider validation, operator docs, and environment + template entry were added without a provider adapter/parser/retry loop. +- Independent review cycle 1 found missing valid-tool, non-DeepSeek, LiteLLM + wire-shape, and default-exposure evidence. Cycle 2 approved the complete + matrix. +- Local focused verification passed: `229 passed, 1 xpassed`; the reviewer’s + overlapping focused set passed `238 passed, 1 xpassed`. `git diff --check` + passed. No provider key was read or sent. + +## Release closeout + +After the full no-credit matrix is green, record one ordinary and one streaming +non-sensitive request through an operator-configured alias. Retain only alias, +timestamp, HTTP/stream outcome, and safe Airlock headers; never prompts, +responses, keys, `user_id`, or raw upstream headers. + +**Funded smoke — 2026-08-12:** isolated-loopback ordinary and streaming calls +through temporary alias `smoke-deepseek` both returned HTTP 200 with +`X-Airlock-Served-By: deepseek` and +`X-Airlock-Mutations: fallbacks=suppressed`; the streaming response completed +its SSE sequence. The fixed non-sensitive prompt and all response content, +keys, `user_id`, and raw upstream headers were discarded. This was within the +operator-authorized $1 DeepSeek budget. diff --git a/dev/plans/0.5.14/70-tui-lifecycle-design.md b/dev/plans/0.5.14/70-tui-lifecycle-design.md new file mode 100644 index 0000000..c53f30e --- /dev/null +++ b/dev/plans/0.5.14/70-tui-lifecycle-design.md @@ -0,0 +1,15 @@ +# Slice 70 — TUI lifecycle design review + +**Draft review outcome:** approve DFR-30/DAC-30. Ordinary TUI tests may opt into +an explicit constructor-only `test_harness` mode. It composes the production +widget tree but suppresses mount-time overview, config, logs, guards, MCP, JSONL +tailer, and alert-loop work. It is not environment-derived and production keeps +the existing lifecycle by default. + +## TDD and allocation + +RED tests prove harness composition has every production pane and starts no +mount worker, while the default app still starts MCP/tailer lifecycle. GREEN +threads the explicit flag through the app and each mount seam. Existing worker, +cancellation, shutdown, and stale-callback tests deliberately continue using +the default app mode. diff --git a/dev/plans/0.5.14/70-tui-lifecycle-status.md b/dev/plans/0.5.14/70-tui-lifecycle-status.md new file mode 100644 index 0000000..cfeef44 --- /dev/null +++ b/dev/plans/0.5.14/70-tui-lifecycle-status.md @@ -0,0 +1,36 @@ +# Slice 70 — TUI lifecycle status + +**Status:** complete + +## Ratified requirement and acceptance criteria + +Slice 70 ratified DFR-30/DAC-30. The explicit constructor-only +`AirlockApp(test_harness=True)` flag composes the same production pane tree but +suppresses mount-time MCP, JSONL, overview, Guards, Logs, and Config background +work. It is never environment-derived and production defaults remain unchanged. +Pure composition, navigation, and static-widget tests now use this harness; +lifecycle tests remain normal-mode and mock only the individual worker seams +needed to assert lifecycle startup/teardown. + +Initial Textual filter-change events were found during independent review to +schedule a Logs worker despite the mount guard. The event dispatch paths now +also return in the explicit harness, so the test does not hide the regression +by mocking the Logs loader. + +## Evidence + +- RED/GREEN regression: `tests/test_tui_harness.py` proves the real production + tree is present, no app-level health/tailer work begins in the harness, and + normal mode invokes its app-level lifecycle starters. +- Migration evidence: six existing composition/navigation/widget tests in + `tests/test_tui.py` use the harness. +- `timeout 45s .venv/bin/python -m pytest tests/test_tui_harness.py -q`: + **2 passed in 3.57s**. +- `timeout 60s .venv/bin/python -m pytest tests/test_tui.py -q -k + 'app_composes_all_panes or screen_switching_via_keys or tab_bar_navigation or + overview_has_widgets or overview_has_start_button or overview_has_console_log'`: + **6 passed, 60 deselected in 7.74s**. + +An independent high-reasoning review found and verified the initial-select +worker fix. Full non-live duration comparison remains release-closeout evidence; +the focused slice tests are bounded and terminate. diff --git a/dev/plans/0.5.14/80-routing-client-diagnostics-design.md b/dev/plans/0.5.14/80-routing-client-diagnostics-design.md new file mode 100644 index 0000000..20a2861 --- /dev/null +++ b/dev/plans/0.5.14/80-routing-client-diagnostics-design.md @@ -0,0 +1,47 @@ +# Slice 80 — routing and client diagnostics design review + +## Scope decision + +Slice 80 ratifies the Slice-3 drafts as **DFR-31a** and **DAC-31a**: + +- Authenticated operators can inspect a bounded, source-labelled view of smart + routing classifications and live session-affinity pins. +- Operators can break every pin for a selected client through an authenticated, + audited control-plane action. No raw session identifier is displayed, logged, + or accepted as an action target. +- The routing display is a bounded persisted-JSONL view; the pin display is a + bounded `live_admin` read. Both say when their source is unavailable rather + than reconstructing or guessing state. + +The QoS and exporter parts of original DFR-31/DAC-31 remain allocated to Slice +90. Virtual-key work remains rejected here and deferred to 0.5.15. + +## Architecture + +The proxy owns affinity state and the TUI is a separate process, so JSONL is +not sufficient for pins. `StateStore` now keys affinity by `(client_id, +session_id)`, preventing one client from inheriting another client's pin when +the same client-supplied identifier is reused. It exposes an identifier-free, +100-record maximum snapshot. The existing authenticated admin perimeter +provides `/airlock/admin/sessions` and an audited +`POST /airlock/admin/clients/{client}/clear-sessions` operation. The loopback +TUI client uses that perimeter; remote callers need ordinary admin authority. + +The Guards Routing tab reads only the existing bounded request window. It shows +the simple/moderate/complex distribution and, for the selected request, score, +resolved model, and safe reasons. Router metadata no longer includes the raw +session ID, because callbacks persist Airlock metadata to JSONL. + +None of these reads or displays is called from the inference hot path. The +guardian merely supplies the authenticated client identity that it already +computed to routing when a pin is created or read. + +## TDD and verification plan + +RED tests cover client-scoped same-ID sessions, the identifier-free bounded +snapshot, authenticated/audited pin clearing, routing-only JSONL parsing, +distribution/source rendering, and no session ID in routing metadata. GREEN +implementation adds only the state, existing-admin, TUI-client, router, and +operator-view seams described above. Verification includes focused state/admin/ +routing tests and a harnessed TUI composition check; no provider request is +needed. diff --git a/dev/plans/0.5.14/80-routing-client-diagnostics-status.md b/dev/plans/0.5.14/80-routing-client-diagnostics-status.md new file mode 100644 index 0000000..c9d108e --- /dev/null +++ b/dev/plans/0.5.14/80-routing-client-diagnostics-status.md @@ -0,0 +1,37 @@ +# Slice 80 — routing and client diagnostics status + +**Status:** complete + +## Ratified scope + +Slice 80 split and ratified DFR-31/DAC-31 for routing and session affinity +(DFR-31a/DAC-31a). QoS and exporter diagnostics remain Slice 90; virtual-key +work remains deferred to 0.5.15. + +The operator receives a bounded Guards Routing view from persisted request +metadata and an identifier-free live-admin view of active session pins. A +selected client’s pins can be broken through an authenticated, audited control +plane operation. Session identifiers are neither put in persisted route metadata +nor shown to the operator. + +## Implementation and TDD evidence + +- `StateStore` keys pins by `(client_id, session_id)`, so one client cannot + inherit another's affinity pin by reusing a session ID. +- `/airlock/admin/sessions` is source-labelled `live_admin`, capped at 100 + entries, and contains model/age/TTL only. The TUI's clear action uses an + opaque URL-safe selector, avoiding path parsing ambiguity for header-derived + client IDs. +- `POST /airlock/admin/session-clients/{selector}/clear` requires the existing + `admin:clear_sessions` authority and writes an `admin_action` record. +- The Guards Routing tab reports a bounded simple/moderate/complex distribution + plus selected-request score, model and safe route reasons. Malformed persisted + metadata degrades safely rather than breaking rendering. +- Focused RED/GREEN test matrix: client-scoped affinity, identifier-free + snapshot, authorization/audit, slash/percent identifier target correctness, + routing-only records, distribution/source label, and malformed metadata. +- Focused verification: **143 passed** (fast-router, admin, TUI-routing and + TUI-admin-client checks); `git diff --check` clean. + +Independent high-reasoning review required two fixes (opaque target selector and +defensive persisted-metadata parsing), then approved cycle 2. diff --git a/dev/plans/0.5.14/90-qos-exporter-health-design.md b/dev/plans/0.5.14/90-qos-exporter-health-design.md new file mode 100644 index 0000000..4362cff --- /dev/null +++ b/dev/plans/0.5.14/90-qos-exporter-health-design.md @@ -0,0 +1,36 @@ +# Slice 90 — QoS priority and exporter health design review + +## Scope decision + +Slice 90 ratifies the remaining DFR-31/DAC-31 as **DFR-31b** and **DAC-31b**. +Operators need the last observed QoS priority for a client and truthful telemetry +health. The release does not make a priority claim when admission is disabled, +and it does not treat an enabled configuration or importable package as a +successful exporter. + +## Architecture + +The guardian already computes a priority signal before its optional admission +gate. It records the small last-observed `{score, boost, reasons, observed_at}` +state per client behind the existing `StateStore` lock. The existing +authenticated admin read returns it with an explicit `live_admin` source; the +TUI uses that live read and says unavailable/stale rather than rebuilding +priority from JSONL. + +Telemetry health is a bounded process-local instrument, separate from tracing +or Prometheus transport. Prometheus is a pull endpoint: a completed request +proves metrics were recorded, not that any collector scraped it. Tracing reports +only whether its callback/dependency is available unless an installed exporter +reports its own successful export. Health records enabled state, a safe endpoint +origin (never credentials/query/path), last success, bounded success/failure +counts, and a bounded error category. An authenticated read snapshots it for +the TUI; no UI callback enters inference processing. + +## TDD plan + +RED tests cover priority snapshot and disabled-admission wording, telemetry +endpoint sanitization, success/failure counters and bounded error category, +admin authorization/source, and TUI unavailable rendering. GREEN code adds the +minimal StateStore and telemetry-health seams, with callback instrumentation +that fails open. Tests prove no credentials or raw exception text enters the +snapshot or UI. diff --git a/dev/plans/0.5.14/90-qos-exporter-health-status.md b/dev/plans/0.5.14/90-qos-exporter-health-status.md new file mode 100644 index 0000000..c98d6e4 --- /dev/null +++ b/dev/plans/0.5.14/90-qos-exporter-health-status.md @@ -0,0 +1,34 @@ +# Slice 90 — QoS priority and exporter health status + +**Status:** complete + +## Ratified scope + +Slice 90 ratified the remaining DFR-31/DAC-31 work as DFR-31b/DAC-31b: +bounded last-observed QoS priority and truthful telemetry instrumentation health. +It does not claim a priority boost is effective while admission is disabled, and +it does not claim that Prometheus has been scraped or an OTLP collector has +received a span merely because Airlock recorded a local signal. + +## Implementation and TDD evidence + +- The guardian stores only score, boost, four bounded signal reasons, and an + observation timestamp in the existing per-client StateStore entry. +- The authenticated client read is source-labelled `live_admin`. The TUI marks + a missing or older-than-120-second observation **stale** and never calls it + an active boost. +- Telemetry health has a separate bounded process-instrumentation seam: + enabled state, safe scheme/authority only endpoint, local signals, export + successes/failures, and an error category. It is not an exporter transport. +- A caller-provided string error is always recorded as `export_error`; it is + never copied into an admin response or TUI. Invalid URL ports fail open to a + null endpoint at tracing setup time. +- `/airlock/admin/telemetry` is authenticated and source-labelled + `process_instrumentation`; the Overview says unavailable if the read fails. +- Focused verification after final fixes: **13 passed, 119 deselected in + 2.45s** across guardian/admin/metrics/QoS telemetry tests; Ruff and + `git diff --check` clean. + +The independent high-reasoning review found raw-error, malformed-endpoint, and +staleness issues in cycle 1; all were corrected and the cycle-2 review approved +the slice. diff --git a/dev/plans/0.5.14/README.md b/dev/plans/0.5.14/README.md new file mode 100644 index 0000000..05fc52f --- /dev/null +++ b/dev/plans/0.5.14/README.md @@ -0,0 +1,39 @@ +# Airlock 0.5.14 release workspace + +This directory is the durable planning and delivery record for 0.5.14. It is +the detailed companion to [the release index](../0.5.14-todo.md). Historical +plans and run evidence retain their existing paths under `dev/plans/`. + +## Release sequence + +| Order | Record or delivery slice | Purpose | Status | +| --- | --- | --- | --- | +| 0 | [environment](00-environment.md) | Establish reproducible local, CI, and funded-smoke prerequisites. | proposal complete | +| 1 | [dependency sweep](01-dependencies.md) | Triage Dependabot and library changes. | proposal complete | +| 2 | [documentation review](02-documentation-cruft-proposal.md) | Classify documentation lifecycle; do not alter it. | proposal complete | +| 3 | [draft requirements](03-draft-requirements-and-allocation.md) | Draft needs, requirements, acceptance criteria, and allocations. | proposal complete | +| 4 | [architecture review](04-architecture-review.md) | Review proposed architecture and current high-level code alignment. | proposal complete | +| 5 | [verification review](05-verification-review.md) | Trace requirements to acceptance criteria and evidence. | proposal complete | +| 6 | [HITL release review](06-hitl-release-review.md) | Decide include, conditional, or postpone for every candidate. | approved | +| 10 | [Benchmark-safe logging profile](10-benchmark-safe-logging-status.md) | Redaction/retention proof and runbook. | complete | +| 20 | [FathomDB chat readiness](20-fathomdb-chat-status.md) | `gpt-4o-mini` aliases and chat/stream verification. | complete | +| 30 | [FathomDB embedding readiness](30-embeddings-status.md) | `text-embedding-3-small` and `/v1/embeddings`. | complete; funded smoke passed | +| 40 | [Shared provider foundation](40-provider-foundation-status.md) | Discovery safety, attribution, and sanitized provider failures. | complete; funded smoke belongs to provider slices | +| 50 | [OpenRouter](50-openrouter-status.md) | Operator-configured gateway recipe and routing-control boundary. | complete; funded smoke passed | +| 60 | [DeepSeek](60-deepseek-status.md) | Operator-configured stable endpoint and function-tool boundary. | complete; funded smoke passed | +| 70 | [TUI test lifecycle](70-tui-lifecycle-status.md) | Fast deterministic ordinary TUI tests. | complete | +| 80 | [TUI routing/client diagnostics](80-routing-client-diagnostics-status.md) | Classification and session-affinity views/actions. | complete | +| 90 | [TUI QoS/exporter health](90-qos-exporter-health-status.md) | Priority and telemetry operational views. | complete | +| 100 | Virtual-key management | Deferred to [0.5.15](../0.5.15-todo.md); draft package only. | postponed | +| 110 | [FathomDB operational reads](110-fathomdb-operational-reads-status.md) | Optional, proxy-owned operational backend. | complete | +| 120 | [Documentation release-index contract](120-documentation-contract-status.md) | Repair stale active-release assertion before release closeout. | complete | +| 130 | [Release closeout](130-release-closeout.md) | Local release matrix, artifact, version, and tag handoff. | ready to tag locally | + +Feature slices use the existing `prompts/SLICE-TEMPLATE.md` workflow: review +their allocation, ratify or revise drafts, write a design memo, commit RED +tests, implement GREEN, obtain independent design/code review (three FIX cycles +maximum), verify, and write a slice status record under `dev/plans/runs/`. + +No feature slice starts until the Slice 6 decision is recorded. A conditional +slice remains planned evidence, not release scope, until its listed dependency +and HITL decision are satisfied. diff --git a/dev/requirements.md b/dev/requirements.md index 814f89a..806d47f 100644 --- a/dev/requirements.md +++ b/dev/requirements.md @@ -272,8 +272,8 @@ Missing `.env` → warning on stderr, proceed with startup. **Traces to:** UN-9 -`airlock status` SHALL probe `/health` using only stdlib (urllib). Default -target: `http://localhost:4000/health`, configurable via `--host`/`--port` flags +`airlock status` SHALL probe `/health/liveliness` using only stdlib (urllib). Default +target: `http://localhost:4000/health/liveliness`, configurable via `--host`/`--port` flags or `AIRLOCK_HOST`/`AIRLOCK_PORT` env vars. Exit 0 if healthy, exit 1 if not reachable. @@ -283,3 +283,64 @@ reachable. The CLI framework SHALL use only Python standard library modules (argparse) and SHALL NOT introduce new third-party dependencies. + +--- + +## 0.5.14 ratified requirements + +### DFR-24 / DAC-24: Benchmark chat alias + +Airlock SHALL expose `gpt-4o-mini` only as an explicit reviewed model-list +alias. Mocked normal and streaming tests SHALL retain authentication, policy, +and served-provider attribution; a non-sensitive funded smoke is recorded +separately. + +### DFR-25 / DAC-25: Embedding alias boundary + +Airlock SHALL serve `/v1/embeddings` only through explicit embedding-capable +aliases, preserving string/batch input and supported options through ordinary +policy and observability. Unconfigured aliases and unsupported options SHALL +fail clearly before dispatch; embedding requests SHALL not be rerouted or +failed-over. + +### DFR-26 / DAC-26: Benchmark-safe logging + +Operators SHALL have a logging profile that redacts request/response content in +enterprise JSONL and disables unredacted SQL/Fathom retention paths. Sentinel +tests SHALL prove redaction, and the profile SHALL prescribe +`/health/liveliness` as the no-model-call probe. + +### DFR-27 through DFR-29 / DAC-27 through DAC-29: Optional providers + +Provider configuration SHALL be explicit; discovery is informational, +same-origin and redirect-free; and provider errors are bounded before any +artifact boundary. OpenRouter is an operator-configured gateway: client routing +overrides are rejected and Airlock never claims downstream-provider control. +DeepSeek uses its stable explicit base and supports function tools only. Neither +provider is auto-enabled by an environment key or default alias. + +### DFR-30 / DAC-30: Deterministic TUI verification + +Ordinary TUI tests SHALL compose the production widget tree without unrelated +background workers. Named normal-mode tests SHALL retain lifecycle, cancellation, +shutdown, stale-callback, JSONL, and MCP coverage. + +### DFR-31 / DAC-31: Bounded operator diagnostics + +Authenticated operators SHALL receive bounded, source-labelled routing, +session-affinity, QoS-priority, and telemetry-health views without a UI +dependency on the inference path. Session IDs, key material, credentials, +prompt content, raw exporter errors, and raw endpoints SHALL not appear in these +views. A session-pin break SHALL be authenticated and audited; unavailable or +stale state SHALL be stated rather than inferred. + +### DFR-33 / DAC-33: Optional FathomDB operational reads + +Operators MAY select FathomDB as the operational-read backend only through an +explicit setting. The default SHALL remain bounded JSONL reads. TUI history and +Advisor error/search reads SHALL label their actual source, limit/truncation, +and any unavailable/invalid-backend JSONL fallback. FathomDB remains +single-owner and optional. Selected separate-process reads SHALL use a +loopback-only proxy-admin bridge (and require its local admin configuration), +never a second engine open. A FathomDB erasure receipt SHALL continue to state +that JSONL retention/deletion is a separate obligation. diff --git a/dev/user-needs.md b/dev/user-needs.md index 7aaeb84..73d37c5 100644 --- a/dev/user-needs.md +++ b/dev/user-needs.md @@ -1037,3 +1037,29 @@ setting is protecting a running proxy. changes accurately without exposing a secret or configuration value in its status. 4. Existing live controls—enforcement mode, PII/keyword controls, and MCP tool allow/block lists—continue to update the runtime environment immediately. + +--- + +## UN-32: Safe Benchmarks and Explainable Operator State + +**As a** benchmark operator or platform operator, +**I need** Airlock to make configured low-cost chat/embedding workloads and +optional providers usable without exposing content or credentials, and to explain +routing, affinity, priority, and telemetry state honestly, +**so that** I can evaluate and operate the gateway without turning diagnostics +into a data-leak or mistaking unavailable state for a healthy system. + +### Acceptance Criteria + +1. Benchmark aliases and optional providers require explicit configuration; an + environment key or discovered catalog never authorizes a model by itself. +2. Request/response logging can be configured to redact content, with a + sentinel-proof profile and a liveness endpoint that does not invoke a model. +3. Operator views are bounded, source-labelled, and state unavailable or stale + data visibly. They never display session IDs, key material, exporter + credentials, raw endpoints, or raw provider/exporter errors. +4. A client session pin may be broken only through authenticated, audited + control-plane action; the TUI cannot directly mutate proxy state. +5. TUI verification uses production widgets while keeping background lifecycle + work in named normal-mode tests rather than silently removing it from product + behavior. diff --git a/docs/architecture/overview.md b/docs/architecture/overview.md index 250d8c0..5554c09 100644 --- a/docs/architecture/overview.md +++ b/docs/architecture/overview.md @@ -71,6 +71,11 @@ via `write_batch_record`) is provider-agnostic; only the adapter differs. See Textual-based terminal dashboard with 6 screens. See [TUI Dashboard](../guide/tui.md). +The TUI is a separate process from the proxy. For explicitly selected FathomDB +operational reads, it uses the proxy's loopback-only admin bridge rather than +opening the proxy-owned embedded database; unavailable reads are labelled, +bounded JSONL fallbacks. + ### CLI (`cli/`) Unified `airlock` command with subcommands. See [CLI Reference](../guide/cli.md). diff --git a/docs/getting-started/configuration.md b/docs/getting-started/configuration.md index d2c7128..05624a5 100644 --- a/docs/getting-started/configuration.md +++ b/docs/getting-started/configuration.md @@ -13,6 +13,75 @@ Key sections: - **`mcp_servers`** — MCP tool servers accessible via the proxy - **`general_settings`** — master key, host/port +### FathomDB benchmark chat model + +The shipped configuration includes the explicit `gpt-4o-mini` and +`openai/gpt-4o-mini` chat aliases for benchmark extraction workloads. They both +target `openai/gpt-4o-mini`, use `OPENAI_API_KEY`, and are not router fallback +targets: clients must request one deliberately. These are chat-completions +aliases; the separate embedding model is introduced with Airlock's +`/v1/embeddings` capability. + +The shipped `text-embedding-3-small` and `openai/text-embedding-3-small` +aliases are embedding-only. Send a normal OpenAI embedding request to +`/v1/embeddings`; `input` may be a string or a batch of strings. The optional +`dimensions` value is an integer from 1 through 1536, and `encoding_format` is +`float` or `base64`; both pass through. Completion-only controls, such as +`temperature`, `messages`, or `tools`, fail with an OpenAI-shaped 400 before +dispatch rather than being silently dropped. Airlock also rejects chat-only, +`smart`, fuzzy, and unconfigured model names for this endpoint instead of +routing or falling back to another model. + +## OpenRouter (operator-configured gateway) + +OpenRouter is not enabled merely because the key exists. To expose a model, an +operator adds a reviewed deployment `model_list` entry and restarts Airlock: + +```yaml +# reviewed deployment config — not the shipped default config.yaml +model_list: + - model_name: openrouter/openai/gpt-4o-mini + litellm_params: + model: openrouter/openai/gpt-4o-mini + api_key: os.environ/OPENROUTER_API_KEY + api_base: https://openrouter.ai/api/v1 +``` + +Keep provider routing and privacy choices in the reviewed deployment policy. +Airlock rejects client-supplied OpenRouter `route`, `models`, and `transforms` +overrides (including those nested in `extra_body`), so a caller cannot alter the +operator's chosen gateway policy. The optional startup catalog is informational: +it does not authorize models or alter routing. + +`X-Airlock-Served-By: openrouter` means the OpenRouter gateway immediately +served the request. Airlock does not identify or control OpenRouter's downstream +provider, fallback choice, price, availability, or data-retention policy. LiteLLM +may use its own OpenRouter `HTTP-Referer` and `X-Title` defaults; they are not +Airlock routing or identity metadata. Do not place identifying client data in +provider-specific request fields. + +## DeepSeek (operator-configured stable endpoint) + +DeepSeek is also disabled until an operator adds a reviewed deployment entry. +Use the explicit stable base below; the pinned LiteLLM version otherwise has a +beta-default path that is not Airlock’s supported 0.5.14 configuration. + +```yaml +model_list: + - model_name: deepseek/deepseek-chat + litellm_params: + model: deepseek/deepseek-chat + api_key: os.environ/DEEPSEEK_API_KEY + api_base: https://api.deepseek.com +``` + +The optional startup catalog is informational and never authorizes aliases. +Airlock preserves ordinary policy and observability for configured DeepSeek +requests, but accepts OpenAI **function** tools only. It rejects other tool +types with an OpenAI-shaped 400 before LiteLLM can silently discard them. Do +not map `X-Airlock-Client`, an authenticated identity, or customer data to a +DeepSeek `user_id`; no such mapping is implemented in this release. + ## Machine-specific overrides (`config.local.yaml`) Settings that must not live in the tracked `config.yaml` — chiefly MCP servers @@ -292,10 +361,13 @@ See [Batch Processing](../guide/batch.md) for the end-to-end recipe. | `OPENAI_API_KEY` | OpenAI API key | -- | | `GOOGLE_AISTUDIO_API_KEY` | Google AI Studio API key for Gemini models (+ AI Studio batch gateway) | -- | | `MISTRAL_API_KEY` | Mistral API key for Mistral models (+ Mistral batch gateway) | -- | +| `OPENROUTER_API_KEY` | OpenRouter key for an explicit operator-configured `model_list` entry | -- | +| `DEEPSEEK_API_KEY` | DeepSeek key for an explicit operator-configured `model_list` entry | -- | | `AIRLOCK_MASTER_KEY` | Optional proxy auth key. Leave unset for local/dev unauthenticated runs; set it for protected deployments. | -- | | `AIRLOCK_HOST` | Bind address | `127.0.0.1` | | `AIRLOCK_PORT` | Listen port | `4000` | | `AIRLOCK_LOG_DIR` | Directory for JSONL log files | `./logs` | +| `AIRLOCK_LOG_REDACT_FIELDS` | Comma-separated JSONL record fields replaced with `[REDACTED]` | unset (no field redaction) | | `AIRLOCK_STATE_DIR` | State directory for circuit-breaker state and optional FathomDB files | `./logs` | | `AIRLOCK_MAX_LOG_DAYS` | Days to retain log files | `30` | | `AIRLOCK_MAX_LOG_SIZE_MB` | Max log file size before rotation | `500` | @@ -320,6 +392,7 @@ See [Batch Processing](../guide/batch.md) for the end-to-end recipe. | `LITELLM_MCP_STDIO_EXTRA_COMMANDS` | Comma-separated extra command basenames allowed to launch stdio MCP servers (beyond the built-in `deno,docker,node,npx,python,python3,uvx`). See [MCP Servers](../guide/mcp-servers.md). | — | | `AIRLOCK_ENABLE_FATHOMDB` | Enable lazy FathomDB engine initialization | `0` | | `AIRLOCK_ENABLE_FATHOM_LOGGER` | Enable the Fathom request-logging sink (recorder-fed) | `0` | +| `AIRLOCK_OPERATIONAL_READ_BACKEND` | Select operational history source: `jsonl` (default) or proxy-owned `fathomdb` | `jsonl` | | `AIRLOCK_ENABLE_S3_LOGGER` | Enable the S3 logging sink (recorder-fed; also needs `AIRLOCK_S3_BUCKET`) | `0` | | `AIRLOCK_ENABLE_SQL_LOGGER` | Enable the SQL logging sink (recorder-fed; also needs `AIRLOCK_SQL_URL`) | `0` | | `AIRLOCK_LOCAL_VLLM_BASE_URL` | URL of the local vLLM endpoint the router guardrail watches | `http://192.168.1.45:8000/v1` | @@ -345,6 +418,34 @@ See [Batch Processing](../guide/batch.md) for the end-to-end recipe. For the PII boundary, egress modes, and the spaCy requirement for semantic entities, see [Guardrails → PII redaction, hydration, and egress](../guide/guardrails.md#pii-redaction-hydration-and-egress). For the opt-in memory recorder, see [Operations → Memory guardrails and OOM diagnostics](../operations.md#memory-guardrails-and-oom-diagnostics). +## Benchmark-safe FathomDB profile + +When FathomDB uses Airlock as an LLM client for a benchmark, it does not need +Airlock's optional FathomDB request logger. Use this profile for the benchmark +process when prompts and completions must not be retained in Airlock telemetry: + +```bash +AIRLOCK_LOG_REDACT_FIELDS=messages,response +AIRLOCK_ENABLE_SQL_LOGGER=0 +AIRLOCK_ENABLE_S3_LOGGER=0 +AIRLOCK_ENABLE_FATHOMDB=0 +AIRLOCK_ENABLE_FATHOM_LOGGER=0 +AIRLOCK_FATHOM_STORE_MESSAGES=0 +AIRLOCK_FATHOM_STORE_RESPONSE_TEXT=0 +``` + +The always-on enterprise JSONL record retains operational metadata but replaces +the `messages` and `response` fields with `[REDACTED]`. SQL logging is disabled +because its projection stores those fields without this redaction setting. +FathomDB message/response flags remain off as defense in depth, even though the +Fathom request logger is disabled. This logging profile is separate from the +outbound PII guard and does not change its observe-only posture. Use +`/health/liveliness` for benchmark process liveness; it makes no model call. + +Before a funded benchmark, run a non-sensitive sentinel request/response and +confirm that the resulting JSONL contains `[REDACTED]` and not either sentinel. +Do not upload that log or the request/response content as a benchmark artifact. + ### `reasoning_effort` validation and normalization For OpenAI-family models, Airlock validates the client value before LiteLLM's diff --git a/docs/guide/admin-api.md b/docs/guide/admin-api.md index 7de0e42..60130fc 100644 --- a/docs/guide/admin-api.md +++ b/docs/guide/admin-api.md @@ -84,6 +84,19 @@ master key (and any loopback operator) satisfies all of them. | `admin:force_quarantine` | Manually quarantine a provider (loopback-only) | | `admin:erase_client` | Erase a client's FathomDB rows (loopback-only) | +### Proxy-owned FathomDB operational reads + +When `AIRLOCK_OPERATIONAL_READ_BACKEND=fathomdb` is selected, the TUI and +Advisor use the proxy-owned operational-read bridge instead of opening the +embedded database from their separate processes. Its bounded records, error, +and search views are **loopback-only**: a capability token cannot read them +remotely, even with `admin:read`. Enable the local admin path with +`admin.enabled: true` and `trust_loopback: true`. + +The bridge carries source, degradation, and truncation information. When it or +FathomDB is unavailable, the consumer visibly falls back to bounded JSONL; +default deployments continue to use JSONL. + ## Minting capability tokens Tokens are short-lived HS256 JWTs signed with `AIRLOCK_JWT_SECRET` (which falls diff --git a/docs/guide/provider-observability.md b/docs/guide/provider-observability.md index cb4fc51..0da77e4 100644 --- a/docs/guide/provider-observability.md +++ b/docs/guide/provider-observability.md @@ -17,7 +17,7 @@ from the model name): | Header | Meaning | Emitted when | |---|---|---| -| `X-Airlock-Served-By` | The served provider token — `anthropic` / `openai` / `gemini` (AI Studio) / `vertex_ai` (Vertex) / `mistral` / `perplexity` / `tavily`. | every response whose served provider can be determined (omitted, never guessed, otherwise) | +| `X-Airlock-Served-By` | The served provider token — `anthropic` / `openai` / `gemini` (AI Studio) / `vertex_ai` (Vertex) / `mistral` / `perplexity` / `tavily` / `openrouter`. `openrouter` identifies the gateway, never a guessed downstream host. | every response whose served provider can be determined (omitted, never guessed, otherwise) | | `X-Airlock-Served-Region` | The served region. | only when the backend reports one (gateway/region backends, e.g. Vertex / Bedrock) | The value of `X-Airlock-Served-By` **equals** the `airlock_provider` you diff --git a/docs/operations.md b/docs/operations.md index 6f83e8f..7319aee 100644 --- a/docs/operations.md +++ b/docs/operations.md @@ -309,6 +309,13 @@ FathomDB is optional and disabled by default. - Set `AIRLOCK_ENABLE_FATHOMDB=1` to enable the lazy engine path. - Set `AIRLOCK_ENABLE_FATHOM_LOGGER=1` to append the Fathom request logger at runtime. +- Set `AIRLOCK_OPERATIONAL_READ_BACKEND=fathomdb` only to opt TUI history and + Advisor error/search reads into FathomDB. The default is bounded JSONL even + when FathomDB is enabled. This TUI/Advisor option also requires + `admin.enabled: true` with `trust_loopback: true`. The separate-process TUI reaches the proxy's + loopback-only admin bridge; it never opens the database itself. If that + bridge or FathomDB is unavailable, the screen states its bounded JSONL + fallback rather than silently changing source. - Put fresh databases under `AIRLOCK_STATE_DIR` while debugging. The database file is `airlock-fathom.db`; a `logs/airlock.db` left behind by FathomDB 0.3.x is abandoned — Airlock refuses to open it, and its records remain in the JSONL logs. Current write-path guarantees: diff --git a/docs/reference/api.md b/docs/reference/api.md index 7666402..0010661 100644 --- a/docs/reference/api.md +++ b/docs/reference/api.md @@ -18,6 +18,19 @@ example `https://airlock.example.com/airlock/docs`. ## Airlock-specific contracts +### Embeddings + +Airlock forwards OpenAI-compatible `POST /v1/embeddings` requests through the +normal authenticated proxy path for explicitly configured embedding aliases. +In the shipped benchmark configuration, use `text-embedding-3-small` or +`openai/text-embedding-3-small`. Chat-only aliases, `smart`, fuzzy aliases, and +unconfigured models receive an OpenAI-shaped `400` with +`error.code=model_endpoint_not_supported`; Airlock does not route or fall back +an embedding request. For `text-embedding-3-small`, `dimensions` must be an +integer from 1 through 1536 and `encoding_format` must be `float` or `base64`. +Completion-only controls are rejected before dispatch with +`error.code=invalid_embedding_option`. + The live schema includes Airlock additions to the standard endpoints, including client attribution, supported request metadata, 429 responses, and response headers. Use these static pages for the behavior behind those fields: diff --git a/pyproject.toml b/pyproject.toml index 253e101..dd66d48 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta" [project] name = "airlock-llm" -version = "0.5.12" +version = "0.5.14" description = "Enterprise LLM proxy built on LiteLLM — logging, guardrails, and unified access for AI coding tools." readme = "README.md" license = "Apache-2.0" @@ -50,7 +50,7 @@ dependencies = [ ] [project.optional-dependencies] -db = ["fathomdb>=0.8.21,<0.9"] +db = ["fathomdb>=0.8.22,<0.9"] s3 = ["boto3>=1.34.0"] sql = ["sqlalchemy>=2.0.0"] metrics = ["prometheus-client>=0.20.0"] diff --git a/tests/test_0_5_2_capability_compat.py b/tests/test_0_5_2_capability_compat.py index bfcc991..03ccb84 100644 --- a/tests/test_0_5_2_capability_compat.py +++ b/tests/test_0_5_2_capability_compat.py @@ -245,13 +245,19 @@ def _expects_batch(entry: dict) -> bool: return False +def _expected_endpoints(entry: dict) -> list[str]: + if entry.get("airlock_embeddings"): + return ["embeddings"] + return ["chat", "batch"] if _expects_batch(entry) else ["chat"] + + class TestCapabilityWiringConsistency: def test_no_entry_over_or_under_claims_batch(self, model_list: list[dict]): # Assert the FULL endpoints list for every entry — an entry returning # extra/garbage endpoints (e.g. ["chat","embeddings"]) must NOT pass. mismatches = [] for entry in model_list: - expected = ["chat", "batch"] if _expects_batch(entry) else ["chat"] + expected = _expected_endpoints(entry) actual = endpoints_for(entry) if actual != expected: mismatches.append((entry.get("model_name"), actual, expected)) diff --git a/tests/test_admin_http.py b/tests/test_admin_http.py index 7909a7e..4f8b423 100644 --- a/tests/test_admin_http.py +++ b/tests/test_admin_http.py @@ -4,6 +4,7 @@ from __future__ import annotations import asyncio +import base64 import json import time @@ -114,6 +115,125 @@ def test_get_providers(self, fresh_state_store): assert s == 200 assert body["providers"]["openai"]["quarantined"] is True + def test_operational_records_are_proxy_owned_and_loopback_only(self, monkeypatch): + from airlock.operational_reads import OperationalReadPage + + monkeypatch.setattr( + "airlock.operational_reads.read_records", + lambda **kwargs: OperationalReadPage( + records=[{"model": "gpt-4o-mini"}], + source="fathomdb", + degraded_reason=None, + truncated=False, + limit_hit=None, + ), + ) + s, body, _ = handle_admin_request( + "POST", + "/airlock/admin/operational/records", + b'{"days": 1, "limit": 1}', + self._loop(), + ) + assert s == 200 + assert body["source"] == "fathomdb" + assert body["records"] == [{"model": "gpt-4o-mini"}] + + s, body, _ = handle_admin_request( + "POST", + "/airlock/admin/operational/records", + b'{"days": 1, "limit": 1}', + Principal(bearer="master-key-supersecret-123456"), + ) + assert s == 403 + assert "loopback" in body["error"] + + @pytest.mark.parametrize( + ("path", "body"), + [ + ("/airlock/admin/operational/errors", b'{"days": "bad"}'), + ("/airlock/admin/operational/search", b'{"query": 3}'), + ( + "/airlock/admin/operational/search", + b'{"query": "x", "limit": 51}', + ), + ( + "/airlock/admin/operational/search", + b'{"query": "' + b"x" * 1001 + b'"}', + ), + ], + ) + def test_operational_error_and_search_arguments_are_bounded(self, path, body): + s, result, _ = handle_admin_request("POST", path, body, self._loop()) + assert s == 400 + assert "error" in result + + def test_session_view_and_break_are_authenticated_and_audited( + self, fresh_state_store + ): + fresh_state_store.set_session("do-not-display", "claude-sonnet", "key:alice") + s, body, _ = handle_admin_request( + "GET", "/airlock/admin/sessions", b"", self._loop() + ) + assert s == 200 + assert body["source"] == "live_admin" + assert body["sessions"][0]["client_id"] == "key:alice" + assert "do-not-display" not in str(body) + + selector = base64.urlsafe_b64encode(b"key:alice").decode().rstrip("=") + s, body, _ = handle_admin_request( + "POST", + f"/airlock/admin/session-clients/{selector}/clear", + b"{}", + self._loop(), + ) + assert s == 200 + assert body["op"] == "clear_client_sessions" + assert body["cleared_sessions"] == 1 + assert fresh_state_store.active_session_snapshot(ttl_seconds=3600) == [] + + def test_break_sessions_requires_authenticated_operator(self, fresh_state_store): + fresh_state_store.set_session("s", "claude-sonnet", "alice") + selector = base64.urlsafe_b64encode(b"alice").decode().rstrip("=") + s, _body, _ = handle_admin_request( + "POST", + f"/airlock/admin/session-clients/{selector}/clear", + b"{}", + Principal(loopback=False), + ) + assert s == 401 + + def test_opaque_session_client_selector_preserves_slash_identity( + self, fresh_state_store + ): + client_id = "tenant/a%2Fb" + fresh_state_store.set_session("s", "claude-sonnet", client_id) + selector = base64.urlsafe_b64encode(client_id.encode()).decode().rstrip("=") + + s, body, _ = handle_admin_request( + "POST", + f"/airlock/admin/session-clients/{selector}/clear", + b"{}", + self._loop(), + ) + + assert s == 200 + assert body["client_id"] == client_id + assert body["cleared_sessions"] == 1 + assert fresh_state_store.active_session_snapshot(ttl_seconds=3600) == [] + + def test_telemetry_view_is_authenticated_and_source_labelled(self): + s, body, _ = handle_admin_request( + "GET", "/airlock/admin/telemetry", b"", self._loop() + ) + assert s == 200 + assert body["source"] == "process_instrumentation" + assert isinstance(body["exporters"], dict) + + s, _body, _ = handle_admin_request( + "GET", "/airlock/admin/telemetry", b"", Principal(loopback=False) + ) + assert s == 401 + def test_clear_quarantine_loopback(self, fresh_state_store): fresh_state_store.get_provider("openai").quarantine_until = time.time() + 100 s, body, _ = handle_admin_request( diff --git a/tests/test_advisor_tools.py b/tests/test_advisor_tools.py index 35eb683..5aa6b2f 100644 --- a/tests/test_advisor_tools.py +++ b/tests/test_advisor_tools.py @@ -182,9 +182,10 @@ def __init__(self, properties): def test_get_recent_errors_uses_fathomdb(monkeypatch, log_dir): - """If airlock.datastore.engine is set, uses RequestLog rows from FathomDB.""" + """An explicit FathomDB selection uses bounded RequestLog rows.""" import airlock.datastore + monkeypatch.setenv("AIRLOCK_OPERATIONAL_READ_BACKEND", "fathomdb") monkeypatch.setattr(airlock.datastore, "get_engine", lambda: "dummy_engine") mock_called = False @@ -198,7 +199,9 @@ def mock_get_request_logs(engine, limit=50_000): "model": "gpt-4o", "error_type": "RateLimitError", "airlock_client": "c1", - "timestamp": "2025-01-01T10:01:00Z", + "timestamp": datetime.datetime.now( + datetime.timezone.utc + ).isoformat(), "error": "boom", "success": False, } @@ -206,34 +209,98 @@ def mock_get_request_logs(engine, limit=50_000): ] monkeypatch.setattr( - "airlock.advisor.tools.get_request_logs", mock_get_request_logs, raising=False + "airlock.operational_reads.get_request_logs", mock_get_request_logs ) result = get_recent_errors(str(log_dir), days=2) assert mock_called + assert result["source"] == "fathomdb" assert result["total_errors"] == 1 assert result["by_model"]["gpt-4o"] == 1 +def test_get_recent_errors_uses_proxy_owned_bridge_when_available(monkeypatch, log_dir): + monkeypatch.setenv("AIRLOCK_OPERATIONAL_READ_BACKEND", "fathomdb") + monkeypatch.setattr( + "airlock.tui.admin_client.operational_view", + lambda host, port, kind, body: { + "source": "fathomdb", + "total_errors": 1, + "by_model": {"gpt-4o": 1}, + }, + ) + + result = get_recent_errors( + str(log_dir), days=2, proxy_host="localhost", proxy_port="4000" + ) + + assert result["source"] == "fathomdb" + assert result["total_errors"] == 1 + + +def test_failed_advisor_bridge_never_opens_the_proxy_owned_engine(monkeypatch, log_dir): + import airlock.datastore + + monkeypatch.setenv("AIRLOCK_OPERATIONAL_READ_BACKEND", "fathomdb") + monkeypatch.setattr( + airlock.datastore, + "get_engine", + lambda: (_ for _ in ()).throw(AssertionError("must use bridge")), + ) + monkeypatch.setattr( + "airlock.tui.admin_client.operational_view", + lambda host, port, kind, body: None, + ) + today = datetime.date.today().isoformat() + (log_dir / f"airlock-{today}.jsonl").write_text( + json.dumps( + { + "timestamp": datetime.datetime.now(datetime.timezone.utc).isoformat(), + "success": False, + "model": "gpt-4o", + } + ) + + "\n" + ) + + errors = get_recent_errors(str(log_dir), proxy_host="localhost", proxy_port="4000") + + assert errors["source"] == "jsonl" + assert "proxy operational reads unavailable" in errors["degraded_reason"] + + from airlock.advisor.tools import search_logs + + search = search_logs( + str(log_dir), "gpt-4o", proxy_host="localhost", proxy_port="4000" + ) + assert search["backend"] == "jsonl" + assert "proxy operational reads unavailable" in search["degraded_reason"] + + def test_get_recent_errors_uses_error_flag_from_fathom(monkeypatch, log_dir): import airlock.datastore + monkeypatch.setenv("AIRLOCK_OPERATIONAL_READ_BACKEND", "fathomdb") monkeypatch.setattr(airlock.datastore, "get_engine", lambda: "dummy_engine") monkeypatch.setattr( - "airlock.advisor.tools.get_request_logs", + "airlock.operational_reads.get_request_logs", lambda engine, limit=50_000: [ FakeNodeRecord( { "model": "gemini-coding", - "timestamp": "2025-01-01T10:01:00Z", + "timestamp": datetime.datetime.now( + datetime.timezone.utc + ).isoformat(), "error_flag": True, } ), FakeNodeRecord( { "model": "gpt-4o", - "timestamp": "2025-01-01T10:02:00Z", + "timestamp": datetime.datetime.now( + datetime.timezone.utc + ).isoformat(), "success": True, } ), @@ -254,6 +321,7 @@ def test_get_recent_errors_against_real_engine(monkeypatch, log_dir, tmp_path): engine = init_engine(str(tmp_path / "airlock-fathom.db")) assert engine is not None try: + now = datetime.datetime.now(datetime.timezone.utc).isoformat() engine.write( [ { @@ -264,7 +332,7 @@ def test_get_recent_errors_against_real_engine(monkeypatch, log_dir, tmp_path): { "model": "gpt-4o", "error_type": "RateLimitError", - "timestamp": "2025-01-01T10:01:00Z", + "timestamp": now, "success": False, } ), @@ -276,7 +344,7 @@ def test_get_recent_errors_against_real_engine(monkeypatch, log_dir, tmp_path): "body": json.dumps( { "model": "gpt-4o", - "timestamp": "2025-01-01T10:02:00Z", + "timestamp": now, "success": True, } ), @@ -284,6 +352,7 @@ def test_get_recent_errors_against_real_engine(monkeypatch, log_dir, tmp_path): ] ) monkeypatch.setattr(airlock.datastore, "get_engine", lambda: engine) + monkeypatch.setenv("AIRLOCK_OPERATIONAL_READ_BACKEND", "fathomdb") result = get_recent_errors(str(log_dir), days=2) @@ -296,13 +365,14 @@ def test_get_recent_errors_against_real_engine(monkeypatch, log_dir, tmp_path): def test_get_recent_errors_reports_datastore_truncation(monkeypatch, log_dir): """A scan that fills the datastore bound is reported as a partial window.""" - import airlock.advisor.tools as tools_mod + import airlock.operational_reads as reads_mod import airlock.datastore monkeypatch.setattr(airlock.datastore, "get_engine", lambda: "dummy_engine") - monkeypatch.setattr(tools_mod, "DATASTORE_QUERY_LIMIT", 2) + monkeypatch.setenv("AIRLOCK_OPERATIONAL_READ_BACKEND", "fathomdb") + monkeypatch.setattr(reads_mod, "DATASTORE_QUERY_LIMIT", 2) monkeypatch.setattr( - "airlock.advisor.tools.get_request_logs", + "airlock.operational_reads.get_request_logs", lambda engine, limit=2: [ FakeNodeRecord({"model": "gpt-4o", "success": False}), FakeNodeRecord({"model": "gpt-4o", "success": True}), diff --git a/tests/test_capability.py b/tests/test_capability.py index 81b0605..f82f834 100644 --- a/tests/test_capability.py +++ b/tests/test_capability.py @@ -197,6 +197,16 @@ def test_aistudio_batch_marker(self): assert rec["region"] is None assert rec["deprecated"] is False + def test_embedding_marker_is_embedding_only(self): + entry = { + "model_name": "text-embedding-3-small", + "litellm_params": {"model": "openai/text-embedding-3-small"}, + "airlock_embeddings": True, + } + rec = capability_record(entry) + assert rec["airlock_provider"] == "openai" + assert rec["endpoints"] == ["embeddings"] + def test_vertex_global_region_and_chat_only(self): entry = { "model_name": "gemini-3.5-flash-vertex", diff --git a/tests/test_client_identity.py b/tests/test_client_identity.py index 04303a0..d9b0319 100644 --- a/tests/test_client_identity.py +++ b/tests/test_client_identity.py @@ -191,6 +191,14 @@ class TestExtractAirlockClientFromRequest: ({"headers": {"x-airlock-client": "hdr"}}, None, "hdr"), ({"headers": {"X-Airlock-Client": "hdr2"}}, None, "hdr2"), ({"metadata": {"headers": {"airlock-client": "mhdr"}}}, None, "mhdr"), + ( + { + "headers": {"X-Airlock-Client": "forged-body-value"}, + "metadata": {"headers": {"X-Airlock-Client": "wire-value"}}, + }, + None, + "wire-value", + ), # api-key fallback when no header/metadata identity present ({}, _FakeKey("sk-1234567890abcdef"), "key:90abcdef"), ({}, {"api_key": "sk-1234567890abcdef"}, "key:90abcdef"), diff --git a/tests/test_config_consistency.py b/tests/test_config_consistency.py index 9d63222..372a6bc 100644 --- a/tests/test_config_consistency.py +++ b/tests/test_config_consistency.py @@ -12,6 +12,8 @@ import pytest import yaml +from airlock.capability import airlock_provider_for, endpoints_for + # --------------------------------------------------------------------------- # Fixtures @@ -256,6 +258,71 @@ def test_shared_aliases_agree(self, template_config, root_config): ) +class TestFathomDBBenchmarkAliases: + """The benchmark client depends on these explicit, OpenAI-backed names.""" + + @pytest.mark.parametrize("alias", ["gpt-4o-mini", "openai/gpt-4o-mini"]) + def test_chat_benchmark_aliases_are_explicit(self, any_config, alias): + which, config, _ = any_config + entries = { + entry["model_name"]: entry + for entry in config.get("model_list", []) + if "model_name" in entry + } + assert alias in entries, f"[{which}] missing FathomDB chat alias {alias!r}" + params = entries[alias]["litellm_params"] + assert params["model"] == "openai/gpt-4o-mini" + assert params["api_key"] == "os.environ/OPENAI_API_KEY" + assert airlock_provider_for(entries[alias]) == "openai" + assert endpoints_for(entries[alias]) == ["chat"] + + fallback_refs = { + model + for chain in config.get("router_settings", {}).get("fallbacks", []) + for source, targets in chain.items() + for model in [source, *targets] + } + assert alias not in fallback_refs, ( + f"[{which}] benchmark alias {alias!r} must not be an automatic fallback" + ) + + @pytest.mark.parametrize( + "alias", ["text-embedding-3-small", "openai/text-embedding-3-small"] + ) + def test_embedding_benchmark_aliases_are_explicit(self, any_config, alias): + which, config, _ = any_config + entries = { + entry["model_name"]: entry + for entry in config.get("model_list", []) + if "model_name" in entry + } + assert alias in entries, f"[{which}] missing FathomDB embedding alias {alias!r}" + params = entries[alias]["litellm_params"] + assert params["model"] == "openai/text-embedding-3-small" + assert params["api_key"] == "os.environ/OPENAI_API_KEY" + assert endpoints_for(entries[alias]) == ["embeddings"] + + +class TestOptionalProviderRecipes: + """Provider keys/config examples must not widen the shipped model surface.""" + + @pytest.mark.parametrize("provider", ["openrouter", "deepseek"]) + def test_optional_provider_is_not_enabled_or_a_fallback_by_default( + self, any_config, provider + ): + which, config, names = any_config + assert not any(name.startswith(f"{provider}/") for name in names), ( + f"[{which}] {provider} must require an operator model_list entry" + ) + fallback_refs = { + model + for chain in config.get("router_settings", {}).get("fallbacks", []) + for source, targets in chain.items() + for model in [source, *targets] + } + assert not any(name.startswith(f"{provider}/") for name in fallback_refs) + + # --------------------------------------------------------------------------- # Budget config consistency # --------------------------------------------------------------------------- diff --git a/tests/test_documentation_contract.py b/tests/test_documentation_contract.py index 21f4c6a..439a994 100644 --- a/tests/test_documentation_contract.py +++ b/tests/test_documentation_contract.py @@ -72,11 +72,45 @@ def test_documentation_entry_points_and_live_api_contract_are_linked() -> None: # release boundary — the failure is the point: it catches the index going # stale, which is how a cold-start reader ends up following a closed-out # milestone's instructions. - assert "plans/0.5.10-plan.md" in dev_readme + assert "plans/0.5.14-todo.md" in dev_readme assert "/airlock/docs" in api_reference assert "/openapi.json" in api_reference +def test_benchmark_safe_logging_profile_is_documented() -> None: + configuration = (DOCS / "getting-started" / "configuration.md").read_text() + + assert "## Benchmark-safe FathomDB profile" in configuration + assert "AIRLOCK_LOG_REDACT_FIELDS=messages,response" in configuration + assert "AIRLOCK_ENABLE_SQL_LOGGER=0" in configuration + assert "AIRLOCK_ENABLE_FATHOM_LOGGER=0" in configuration + assert "AIRLOCK_FATHOM_STORE_MESSAGES=0" in configuration + assert "AIRLOCK_FATHOM_STORE_RESPONSE_TEXT=0" in configuration + + +def test_openrouter_recipe_states_explicit_gateway_boundary() -> None: + configuration = (DOCS / "getting-started" / "configuration.md").read_text() + + assert "## OpenRouter (operator-configured gateway)" in configuration + assert "openrouter/openai/gpt-4o-mini" in configuration + assert "os.environ/OPENROUTER_API_KEY" in configuration + assert "https://openrouter.ai/api/v1" in configuration + assert "not enabled merely because the key exists" in configuration + assert ( + "Airlock does not identify or control OpenRouter's downstream provider" + in " ".join(configuration.split()) + ) + + +def test_deepseek_recipe_states_stable_endpoint_and_tool_boundary() -> None: + configuration = (DOCS / "getting-started" / "configuration.md").read_text() + assert "## DeepSeek (operator-configured stable endpoint)" in configuration + assert "os.environ/DEEPSEEK_API_KEY" in configuration + assert "https://api.deepseek.com" in configuration + assert "function** tools only" in configuration + assert "DeepSeek `user_id`" in configuration + + def test_cli_reference_covers_each_supported_top_level_command() -> None: reference = (DOCS / "guide" / "cli.md").read_text() commands = { diff --git a/tests/test_embedding_boundary.py b/tests/test_embedding_boundary.py new file mode 100644 index 0000000..40d3df8 --- /dev/null +++ b/tests/test_embedding_boundary.py @@ -0,0 +1,125 @@ +"""Authenticated ingress-contract tests for embedding request bodies.""" + +from __future__ import annotations + +import json + +import httpx +import pytest +from fastapi import Depends, FastAPI, HTTPException, Request +from fastapi.routing import APIRoute + +from airlock.embedding_boundary import ( + _install_authenticated_embedding_validator, + _is_embedding_path, +) + + +def _embedding_app() -> tuple[FastAPI, list[str]]: + """A route shaped like LiteLLM: auth reads/caches body before endpoint.""" + calls: list[str] = [] + app = FastAPI() + + async def authenticated(request: Request) -> None: + calls.append("auth") + await request.body() + if request.headers.get("authorization") != "Bearer valid": + raise HTTPException(401, "unauthorized") + + @app.post("/v1/embeddings", dependencies=[Depends(authenticated)]) + async def embeddings(request: Request): + calls.append("endpoint") + return await request.json() + + route = next( + route + for route in app.routes + if isinstance(route, APIRoute) and route.path == "/v1/embeddings" + ) + _install_authenticated_embedding_validator(route) + return app, calls + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "field,value", + [ + ("api_base", "https://unreviewed.example"), + ("api_key", "unreviewed-key"), + ("custom_llm_provider", "openai"), + ("deployment_id", "unreviewed-deployment"), + ("headers", {"X-Airlock-Client": "forged"}), + ("extra_headers", {"Authorization": "Bearer forged"}), + ("bogus_embedding_option", 1), + ], +) +async def test_authenticated_embedding_body_rejects_unreviewed_controls(field, value): + app, calls = _embedding_app() + async with httpx.AsyncClient( + transport=httpx.ASGITransport(app=app), base_url="http://test" + ) as client: + response = await client.post( + "/v1/embeddings", + headers={"Authorization": "Bearer valid"}, + json={"model": "text-embedding-3-small", "input": "safe", field: value}, + ) + + assert response.status_code == 400 + assert response.json()["error"]["code"] == "invalid_embedding_option" + assert response.json()["error"]["param"] == field + assert calls == ["auth"] + + +@pytest.mark.asyncio +async def test_unauthenticated_invalid_embedding_body_is_rejected_before_validation(): + app, calls = _embedding_app() + async with httpx.AsyncClient( + transport=httpx.ASGITransport(app=app), base_url="http://test" + ) as client: + response = await client.post( + "/v1/embeddings", + json={ + "model": "text-embedding-3-small", + "input": "safe", + "api_base": "https://attacker.example", + }, + ) + + assert response.status_code == 401 + assert calls == ["auth"] + + +@pytest.mark.parametrize( + "path", + [ + "/v1/embeddings", + "/embeddings", + "/engines/text-embedding-3-small/embeddings", + "/openai/deployments/text-embedding-3-small/embeddings", + ], +) +def test_every_litellm_embedding_route_is_selected_for_authenticated_validation(path): + assert _is_embedding_path(path) + + +@pytest.mark.asyncio +async def test_authenticated_embedding_body_preserves_only_documented_options_for_dispatch(): + app, calls = _embedding_app() + body = { + "model": "text-embedding-3-small", + "input": ["one", "two"], + "dimensions": 512, + "encoding_format": "base64", + "user": "benchmark-run", + } + async with httpx.AsyncClient( + transport=httpx.ASGITransport(app=app), base_url="http://test" + ) as client: + response = await client.post( + "/v1/embeddings", headers={"Authorization": "Bearer valid"}, json=body + ) + + assert response.status_code == 200 + assert response.json() == body + assert json.loads(response.content) == body + assert calls == ["auth", "endpoint"] diff --git a/tests/test_enterprise_logger.py b/tests/test_enterprise_logger.py index b651c0d..371f5ac 100644 --- a/tests/test_enterprise_logger.py +++ b/tests/test_enterprise_logger.py @@ -910,9 +910,11 @@ def test_redact_does_not_mutate_original(self, monkeypatch): def test_redaction_applied_in_write_log(self, log_dir, monkeypatch): """End-to-end: _write_log applies redaction before writing.""" monkeypatch.setenv("AIRLOCK_LOG_REDACT_FIELDS", "messages,response") + prompt_sentinel = "benchmark-prompt-sentinel" + response_sentinel = "benchmark-response-sentinel" record = { - "messages": [{"role": "user", "content": "sensitive prompt"}], - "response": {"choices": [{"message": {"content": "sensitive output"}}]}, + "messages": [{"role": "user", "content": prompt_sentinel}], + "response": {"choices": [{"message": {"content": response_sentinel}}]}, "model": "claude-sonnet", "timestamp": "2024-01-15T10:00:00", } @@ -924,6 +926,9 @@ def test_redaction_applied_in_write_log(self, log_dir, monkeypatch): assert written["messages"] == "[REDACTED]" assert written["response"] == "[REDACTED]" assert written["model"] == "claude-sonnet" + persisted = log_path.read_text() + assert prompt_sentinel not in persisted + assert response_sentinel not in persisted def test_redaction_applied_in_precall_block_record(self, log_dir, monkeypatch): """write_precall_block_record applies redaction at write time.""" diff --git a/tests/test_extract.py b/tests/test_extract.py index 90e83ae..9986daa 100644 --- a/tests/test_extract.py +++ b/tests/test_extract.py @@ -7,6 +7,7 @@ extract_text_from_mcp, extract_text_from_messages, is_batch_call, + is_embedding_call, is_mcp_call, ) @@ -66,6 +67,14 @@ def test_missing_content(self): assert extract_text_from_messages([{"role": "user"}]) == "" +class TestEmbeddingInput: + def test_embedding_input_is_extracted_and_not_a_batch_call(self): + data = {"input": ["first input", "second input"]} + assert is_embedding_call(data, "embedding") is True + assert is_batch_call(data, "embedding") is False + assert extract_text(data, "embedding") == "first input\nsecond input" + + # --------------------------------------------------------------------------- # extract_text_from_mcp() # --------------------------------------------------------------------------- diff --git a/tests/test_fast_guardian.py b/tests/test_fast_guardian.py index 8490332..429b42a 100644 --- a/tests/test_fast_guardian.py +++ b/tests/test_fast_guardian.py @@ -3,6 +3,7 @@ from __future__ import annotations import time +from types import SimpleNamespace from unittest.mock import MagicMock import pytest @@ -12,8 +13,12 @@ AirlockFastGuardian, _extract_client_id, _request_client_id, + reject_openrouter_routing_overrides, + validate_deepseek_tools, ) +from airlock.fast.model_alias import AliasResolution from airlock.guardrails.extract import extract_text_from_messages as _extract_text +from airlock.proxy_errors import AirlockEndpointNotSupported # --------------------------------------------------------------------------- @@ -79,6 +84,53 @@ def test_multipart_content(self): # --------------------------------------------------------------------------- # AirlockFastGuardian.async_pre_call_hook() # --------------------------------------------------------------------------- +@pytest.mark.parametrize( + "data,option", + [ + ({"route": "fallback"}, "route"), + ({"models": ["other/model"]}, "models"), + ({"transforms": ["middle-out"]}, "transforms"), + ({"extra_body": {"route": "fallback"}}, "extra_body.route"), + ({"extra_body": {"models": ["other/model"]}}, "extra_body.models"), + ( + {"extra_body": {"transforms": ["middle-out"]}}, + "extra_body.transforms", + ), + ], +) +def test_openrouter_routing_overrides_are_rejected_pre_dispatch(data, option): + from airlock.proxy_errors import AirlockGatewayRoutingOverride + + with pytest.raises(AirlockGatewayRoutingOverride) as raised: + reject_openrouter_routing_overrides(data) + assert raised.value.option == option + + +def test_openrouter_unrelated_extra_body_is_not_overclaimed(): + reject_openrouter_routing_overrides({"extra_body": {"safe_extension": True}}) + + +@pytest.mark.parametrize( + "tools", + [ + [{"type": "namespace", "name": "shell"}], + [{"type": "function", "function": {"name": "ok"}}, {"type": "web_search"}], + {"type": "function"}, + ], +) +def test_deepseek_non_function_tools_reject_pre_dispatch(tools): + from airlock.proxy_errors import AirlockDeepSeekToolNotSupported + + with pytest.raises(AirlockDeepSeekToolNotSupported): + validate_deepseek_tools({"tools": tools}) + + +def test_deepseek_function_tools_pass_unchanged(): + data = {"tools": [{"type": "function", "function": {"name": "lookup"}}]} + validate_deepseek_tools(data) + assert data["tools"][0]["function"]["name"] == "lookup" + + class TestGuardianPreCallHook: @pytest.fixture def guardian(self): @@ -97,6 +149,341 @@ async def test_normal_request_passes( assert "metadata" in result assert "airlock_priority" in result["metadata"] assert "score" in result["metadata"]["airlock_priority"] + client = fresh_state_store.all_clients()[ + result["metadata"]["airlock_request"]["client_id"] + ] + assert client.priority_score == result["metadata"]["airlock_priority"]["score"] + assert client.priority_observed_at > 0 + + @pytest.mark.parametrize( + "field,value,option", + [ + ("route", "fallback", "route"), + ("models", ["other/model"], "models"), + ("transforms", ["middle-out"], "transforms"), + ("extra_body", {"route": "fallback"}, "extra_body.route"), + ("extra_body", {"models": ["other/model"]}, "extra_body.models"), + ( + "extra_body", + {"transforms": ["middle-out"]}, + "extra_body.transforms", + ), + ], + ) + async def test_guardian_rejects_openrouter_routing_override_before_dispatch( + self, + guardian, + fresh_state_store, + mock_cache, + mock_user_api_key_dict, + monkeypatch, + field, + value, + option, + ): + """The final resolved provider, not request spelling, controls the guard.""" + from airlock.proxy_errors import AirlockGatewayRoutingOverride + + monkeypatch.setattr( + "airlock.fast.guardian.alias_table.resolve_with_diagnostic", + lambda _model: AliasResolution(alias="operator-openrouter"), + ) + monkeypatch.setattr( + "airlock.fast.guardian.infer_provider", lambda _model: "openrouter" + ) + with pytest.raises(AirlockGatewayRoutingOverride) as raised: + await guardian.async_pre_call_hook( + mock_user_api_key_dict, + mock_cache, + { + "model": "operator-openrouter", + "messages": [{"role": "user", "content": "safe"}], + field: value, + }, + "acompletion", + ) + assert raised.value.option == option + + async def test_guardian_allows_routing_named_field_for_non_openrouter( + self, guardian, fresh_state_store, mock_cache, mock_user_api_key_dict + ): + result = await guardian.async_pre_call_hook( + mock_user_api_key_dict, + mock_cache, + { + "model": "gpt-4o-mini", + "messages": [{"role": "user", "content": "safe"}], + "route": "unrelated-client-field", + }, + "acompletion", + ) + assert result["model"] == "gpt-4o-mini" + + async def test_guardian_rejects_non_function_tool_for_final_deepseek( + self, + guardian, + fresh_state_store, + mock_cache, + mock_user_api_key_dict, + monkeypatch, + ): + from airlock.proxy_errors import AirlockDeepSeekToolNotSupported + + monkeypatch.setattr( + "airlock.fast.guardian.alias_table.resolve_with_diagnostic", + lambda _model: AliasResolution(alias="operator-deepseek"), + ) + monkeypatch.setattr( + "airlock.fast.guardian.infer_provider", lambda _model: "deepseek" + ) + with pytest.raises(AirlockDeepSeekToolNotSupported): + await guardian.async_pre_call_hook( + mock_user_api_key_dict, + mock_cache, + { + "model": "operator-deepseek", + "messages": [{"role": "user", "content": "safe"}], + "tools": [{"type": "namespace", "name": "shell"}], + }, + "acompletion", + ) + + async def test_guardian_preserves_function_tool_for_final_deepseek( + self, + guardian, + fresh_state_store, + mock_cache, + mock_user_api_key_dict, + monkeypatch, + ): + tool = {"type": "function", "function": {"name": "lookup"}} + monkeypatch.setattr( + "airlock.fast.guardian.alias_table.resolve_with_diagnostic", + lambda _model: AliasResolution(alias="operator-deepseek"), + ) + monkeypatch.setattr( + "airlock.fast.guardian.infer_provider", lambda _model: "deepseek" + ) + result = await guardian.async_pre_call_hook( + mock_user_api_key_dict, + mock_cache, + { + "model": "operator-deepseek", + "messages": [{"role": "user", "content": "safe"}], + "tools": [tool], + }, + "acompletion", + ) + assert result["tools"] == [tool] + + async def test_guardian_allows_non_function_tool_for_non_deepseek( + self, guardian, fresh_state_store, mock_cache, mock_user_api_key_dict + ): + tool = {"type": "namespace", "name": "shell"} + result = await guardian.async_pre_call_hook( + mock_user_api_key_dict, + mock_cache, + { + "model": "gpt-4o-mini", + "messages": [{"role": "user", "content": "safe"}], + "tools": [tool], + }, + "acompletion", + ) + assert result["tools"] == [tool] + + async def test_configured_embedding_alias_passes_without_routing( + self, guardian, fresh_state_store, mock_cache, mock_user_api_key_dict + ): + data = { + "input": ["first benchmark input", "second benchmark input"], + "model": "text-embedding-3-small", + } + result = await guardian.async_pre_call_hook( + mock_user_api_key_dict, mock_cache, data, "aembedding" + ) + assert result["model"] == "text-embedding-3-small" + assert "airlock_routing" not in result.get("metadata", {}) + + @pytest.mark.parametrize( + "airlock_metadata", + [{"cost_tier": "cheap"}, {"prefer_provider": "anthropic"}], + ) + async def test_aembedding_ignores_client_routing_preferences( + self, + guardian, + fresh_state_store, + mock_cache, + mock_user_api_key_dict, + monkeypatch, + airlock_metadata, + ): + def unexpected_routing(data): + raise AssertionError("embedding request must never enter apply_routing") + + monkeypatch.setattr("airlock.fast.guardian.apply_routing", unexpected_routing) + data = { + "input": "benchmark input", + "model": "text-embedding-3-small", + "metadata": {"airlock": airlock_metadata}, + } + result = await guardian.async_pre_call_hook( + mock_user_api_key_dict, mock_cache, data, "aembedding" + ) + assert result["model"] == "text-embedding-3-small" + assert result["disable_fallbacks"] is True + assert result["num_retries"] == 0 + assert result["metadata"]["airlock_request"]["pinned_model"] is True + assert "airlock_routing" not in result["metadata"] + + async def test_aembedding_open_circuit_does_not_fail_over( + self, + guardian, + fresh_state_store, + mock_cache, + mock_user_api_key_dict, + monkeypatch, + ): + monkeypatch.setattr( + "airlock.fast.guardian.check_model_with_filters", + lambda *args, **kwargs: SimpleNamespace( + allowed=False, + reason="open_circuit", + failover_model="gpt-4o-mini", + ), + ) + data = { + "input": "benchmark input", + "model": "text-embedding-3-small", + "metadata": {"airlock": {"cost_tier": "cheap"}}, + } + with pytest.raises(RateLimitError): + await guardian.async_pre_call_hook( + mock_user_api_key_dict, mock_cache, data, "aembedding" + ) + assert data["model"] == "text-embedding-3-small" + assert "airlock_failover" not in data["metadata"] + + @pytest.mark.parametrize("model", ["gpt-4o-mini", "smart", "not-a-model"]) + async def test_embedding_rejects_non_embedding_aliases( + self, + guardian, + fresh_state_store, + mock_cache, + mock_user_api_key_dict, + model, + ): + with pytest.raises(AirlockEndpointNotSupported): + await guardian.async_pre_call_hook( + mock_user_api_key_dict, + mock_cache, + {"input": "benchmark input", "model": model}, + "embedding", + ) + + @pytest.mark.parametrize( + "option, value", + [ + ("dimensions", 0), + ("encoding_format", "json"), + ("temperature", 0.1), + ("bogus_embedding_option", 1), + ], + ) + async def test_embedding_invalid_options_reject_before_dispatch( + self, + guardian, + fresh_state_store, + mock_cache, + mock_user_api_key_dict, + option, + value, + ): + from airlock.embedding import AirlockInvalidEmbeddingOption + + data = { + "input": "benchmark input", + "model": "text-embedding-3-small", + option: value, + } + with pytest.raises(AirlockInvalidEmbeddingOption): + await guardian.async_pre_call_hook( + mock_user_api_key_dict, mock_cache, data, "aembedding" + ) + + async def test_embedding_supported_options_pass_through( + self, guardian, fresh_state_store, mock_cache, mock_user_api_key_dict + ): + data = { + "input": ["first", "second"], + "model": "text-embedding-3-small", + "dimensions": 512, + "encoding_format": "base64", + } + result = await guardian.async_pre_call_hook( + mock_user_api_key_dict, mock_cache, data, "aembedding" + ) + assert result["dimensions"] == 512 + assert result["encoding_format"] == "base64" + + def test_embedding_option_validator_permits_safe_proxy_fields(self): + """Fields LiteLLM adds before the aembedding pre-call hook are not API options.""" + from airlock.embedding import validate_embedding_options + + payload = { + "model": "text-embedding-3-small", + "input": "benchmark input", + "metadata": {}, + "secret_fields": object(), + "litellm_call_id": "call-1", + "litellm_session_id": "session-1", + "litellm_trace_id": "trace-1", + "litellm_logging_obj": object(), + "proxy_server_request": {}, + "api_version": "2025-01-01", + "request_timeout": 10, + "timeout": 10, + "stream_timeout": 10, + "max_retries": 0, + "num_retries": 0, + "disable_fallbacks": True, + "model_info": {}, + "litellm_params": {}, + "caching": False, + "cache": {}, + "ttl": 60, + "organization": "org-1", + "allowed_model_region": "us", + } + validate_embedding_options(payload) + + @pytest.mark.parametrize( + "option", + [ + "api_base", + "api_key", + "custom_llm_provider", + "deployment_id", + "headers", + "extra_headers", + ], + ) + def test_embedding_direct_dispatch_controls_are_rejected(self, option): + from airlock.embedding import ( + AirlockInvalidEmbeddingOption, + validate_embedding_options, + ) + + with pytest.raises(AirlockInvalidEmbeddingOption): + validate_embedding_options( + { + "model": "text-embedding-3-small", + "input": "benchmark input", + option: {"X-Airlock-Client": "forged"} + if option in {"headers", "extra_headers"} + else "unreviewed", + } + ) async def test_reasoning_effort_none_normalized_in_hook( self, guardian, fresh_state_store, mock_cache, mock_user_api_key_dict diff --git a/tests/test_fast_router.py b/tests/test_fast_router.py index d4fdf91..b1349bd 100644 --- a/tests/test_fast_router.py +++ b/tests/test_fast_router.py @@ -321,6 +321,46 @@ def test_any_tier(self): # Session affinity # --------------------------------------------------------------------------- class TestSessionAffinity: + def test_session_affinity_is_scoped_to_client(self, fresh_state_store): + first = { + "model": "claude-sonnet", + "metadata": {"airlock": {"session_id": "shared-session"}}, + } + apply_routing(first, client_id="client-a") + + second = { + "model": "claude-opus", + "metadata": {"airlock": {"session_id": "shared-session"}}, + } + result = apply_routing(second, client_id="client-b") + + assert result["model"] == "claude-opus" + assert ( + fresh_state_store.get_session("shared-session", "client-a").model + == "claude-sonnet" + ) + assert ( + fresh_state_store.get_session("shared-session", "client-b").model + == "claude-opus" + ) + + def test_session_snapshot_is_bounded_and_hides_session_identifier( + self, fresh_state_store + ): + fresh_state_store.set_session("super-secret-session", "claude-sonnet", "alice") + + snapshot = fresh_state_store.active_session_snapshot(ttl_seconds=3600) + + assert snapshot == [ + { + "client_id": "alice", + "model": "claude-sonnet", + "age_seconds": 0.0, + "ttl_remaining_seconds": 3600.0, + } + ] + assert "super-secret-session" not in str(snapshot) + def test_new_session_records_model(self, fresh_state_store): data = { "model": "claude-sonnet", @@ -603,7 +643,9 @@ def test_session_only(self, fresh_state_store): } result = apply_routing(data) routing = result["metadata"]["airlock_routing"] - assert routing["session_id"] == "full-1" + # Session identifiers are client-supplied and must not enter metadata + # that callbacks persist to JSONL or render in operator diagnostics. + assert "session_id" not in routing assert any("session_new" in r for r in routing["reasons"]) def test_combined_directives_priority(self, fresh_state_store): diff --git a/tests/test_fathom_search.py b/tests/test_fathom_search.py index 42abbfb..49cafbd 100644 --- a/tests/test_fathom_search.py +++ b/tests/test_fathom_search.py @@ -165,8 +165,17 @@ def test_advisor_search_logs_uses_engine(monkeypatch, log_dir, tmp_path): engine = init_engine(str(tmp_path / "airlock-fathom.db")) try: - _write_log(engine, "r1", {"model": "gpt-4", "error": "rate limit exceeded"}) + _write_log( + engine, + "r1", + { + "model": "gpt-4", + "error": "rate limit exceeded", + "timestamp": datetime.datetime.now(datetime.timezone.utc).isoformat(), + }, + ) monkeypatch.setattr(airlock.datastore, "get_engine", lambda: engine) + monkeypatch.setenv("AIRLOCK_OPERATIONAL_READ_BACKEND", "fathomdb") out = search_logs(str(log_dir), "rate limit") @@ -177,6 +186,36 @@ def test_advisor_search_logs_uses_engine(monkeypatch, log_dir, tmp_path): engine.close() +def test_advisor_search_marks_stale_filled_fts_page_as_partial(monkeypatch, log_dir): + """A pre-time-filter FTS cap cannot be represented as a complete window.""" + import airlock.datastore + from airlock.advisor.tools import search_logs + + monkeypatch.setenv("AIRLOCK_OPERATIONAL_READ_BACKEND", "fathomdb") + monkeypatch.setattr(airlock.datastore, "get_engine", lambda: "engine") + stale = "2020-01-01T00:00:00Z" + monkeypatch.setattr( + "airlock.advisor.tools.search_request_logs", + lambda engine, query, limit: { + "mode": "lexical_only", + "degraded_reason": "no vector projection", + "soft_fallback": None, + "results": [ + { + "logical_id": str(i), + "properties": {"timestamp": stale}, + } + for i in range(limit) + ], + }, + ) + + out = search_logs(str(log_dir), "rate limit", limit=3, days=1) + + assert out["results"] == [] + assert out["window"] == {"truncated": True, "limit_hit": "search_result_limit"} + + def test_advisor_search_logs_jsonl_fallback_is_labelled(monkeypatch, log_dir): """Without the datastore the tool still answers — as a labelled substring scan.""" import airlock.datastore diff --git a/tests/test_measurement_report.py b/tests/test_measurement_report.py index 556046b..aaa4306 100644 --- a/tests/test_measurement_report.py +++ b/tests/test_measurement_report.py @@ -234,9 +234,7 @@ async def test_pii_egress_observe_canary_round_trips_through_jsonl_and_report( "model": "gpt-5.4", "metadata": { "airlock_client": "pii-dogfood-canary", - "airlock_pii_handle": _pii_map_store.put( - {"": canary} - ), + "airlock_pii_handle": _pii_map_store.put({"": canary}), }, } response = _pii_tool_response( @@ -246,14 +244,20 @@ async def test_pii_egress_observe_canary_round_trips_through_jsonl_and_report( client_response = await AirlockPIIGuard().async_post_call_success_hook( data, None, response ) - assert json.loads( - client_response.choices[0].message.tool_calls[0].function.arguments - )["recipient"] == canary + assert ( + json.loads(client_response.choices[0].message.tool_calls[0].function.arguments)[ + "recipient" + ] + == canary + ) # The response passed to telemetry remains redacted, and reverse mapping has # been consumed before the recorder sees metadata. - assert json.loads(response.choices[0].message.tool_calls[0].function.arguments)[ - "recipient" - ] == "" + assert ( + json.loads(response.choices[0].message.tool_calls[0].function.arguments)[ + "recipient" + ] + == "" + ) assert "airlock_pii_handle" not in data["metadata"] recorder = RequestRecorder() diff --git a/tests/test_models_catalog.py b/tests/test_models_catalog.py index 09335fe..0d9dea4 100644 --- a/tests/test_models_catalog.py +++ b/tests/test_models_catalog.py @@ -3,15 +3,22 @@ from __future__ import annotations import json +import urllib.error from unittest.mock import patch +import pytest from airlock.models_catalog import ( + _configured_provider_discovery, _custom_fetchers_from_config, _fetch_gemini_models, _get_api_key, _load_config, + _models_url_from_configured_base, + _provider_fetchers_from_model_list, _resolve_secret, + _safe_discovery_failure, + _fetch_openai_compatible, fetch_live_provider_models, ) @@ -211,6 +218,159 @@ def test_empty_and_non_str(self): assert _resolve_secret(123) is None +class TestConfiguredProviderDiscovery: + @pytest.mark.parametrize( + "base,expected", + [ + ("https://openrouter.ai/api/v1", "https://openrouter.ai/api/v1/models"), + ("https://api.deepseek.com", "https://api.deepseek.com/models"), + ( + "https://private.example/v1/chat/completions", + "https://private.example/v1/models", + ), + ], + ) + def test_models_url_from_reviewed_base(self, base, expected): + assert _models_url_from_configured_base(base) == expected + + @pytest.mark.parametrize( + "base", + [ + None, + "http://insecure.example/v1", + "https://user:pass@example/v1", + "https://example/v1?secret=value", + "https://example/v1#fragment", + "https://example/v1/models", + ], + ) + def test_unsafe_base_is_rejected_before_key_use(self, base): + assert _models_url_from_configured_base(base) is None + + def test_requires_one_consistent_base_and_resolvable_key(self, monkeypatch): + monkeypatch.setenv("OPENROUTER_TEST_KEY", "test-key") + config = _cfg( + { + "model_name": "openrouter/openai/gpt-4o-mini", + "litellm_params": { + "model": "openrouter/openai/gpt-4o-mini", + "api_key": "os.environ/OPENROUTER_TEST_KEY", + "api_base": "https://openrouter.ai/api/v1", + }, + } + ) + assert _configured_provider_discovery(config, "openrouter") == ( + "https://openrouter.ai/api/v1/models", + "test-key", + ) + config["model_list"].append( + { + "model_name": "openrouter/other", + "litellm_params": { + "model": "openrouter/other", + "api_key": "os.environ/OPENROUTER_TEST_KEY", + "api_base": "https://other.example/v1", + }, + } + ) + assert _configured_provider_discovery(config, "openrouter") is None + + def test_no_configured_base_or_key_means_no_fetcher(self, monkeypatch): + monkeypatch.delenv("DEEPSEEK_TEST_KEY", raising=False) + config = _cfg( + { + "model_name": "deepseek/deepseek-chat", + "litellm_params": { + "model": "deepseek/deepseek-chat", + "api_key": "os.environ/DEEPSEEK_TEST_KEY", + "api_base": "https://api.deepseek.com", + }, + } + ) + assert _provider_fetchers_from_model_list(config) == [] + + def test_failure_summary_excludes_exception_text_and_keeps_status(self): + exc = urllib.error.HTTPError( + "https://example/models", 429, "sentinel-provider-secret", {}, None + ) + summary = _safe_discovery_failure(exc) + assert summary == "error_type=HTTPError status=429" + assert "sentinel" not in summary + + def test_configured_provider_models_are_normalized_without_registration( + self, monkeypatch + ): + monkeypatch.setenv("OPENROUTER_TEST_KEY", "test-key") + config = _cfg( + { + "model_name": "openrouter/openai/gpt-4o-mini", + "litellm_params": { + "model": "openrouter/openai/gpt-4o-mini", + "api_key": "os.environ/OPENROUTER_TEST_KEY", + "api_base": "https://openrouter.ai/api/v1", + }, + } + ) + fetcher = _provider_fetchers_from_model_list(config)[0] + + class _Response: + def read(self): + return json.dumps( + {"data": [{"id": "openai/gpt-4o-mini", "created": 7}]} + ).encode() + + def __enter__(self): + return self + + def __exit__(self, *_): + pass + + captured: dict[str, object] = {} + + class _Opener: + def open(self, request, timeout): + captured["url"] = request.full_url + captured["authorization"] = request.get_header("Authorization") + captured["timeout"] = timeout + return _Response() + + with patch( + "airlock.models_catalog.urllib.request.build_opener", return_value=_Opener() + ): + entries = fetcher.fn(fetcher.api_key_override or "", 2.0) + + assert entries == [ + { + "id": "openrouter/openai/gpt-4o-mini", + "object": "model", + "created": 7, + "owned_by": "openrouter", + } + ] + assert captured == { + "url": "https://openrouter.ai/api/v1/models", + "authorization": "Bearer test-key", + "timeout": 2.0, + } + + def test_provider_discovery_uses_no_redirect_opener(self): + """A redirect is an HTTP error, never a second credential-bearing request.""" + with patch("airlock.models_catalog.urllib.request.build_opener") as opener: + opener.return_value.open.side_effect = urllib.error.HTTPError( + "https://reviewed.example/models", 302, "redirect", {}, None + ) + with pytest.raises(urllib.error.HTTPError): + _fetch_openai_compatible( + "https://reviewed.example/models", + "deepseek", + "sentinel-key", + 1.0, + no_redirect=True, + ) + handlers = opener.call_args.args + assert any(handler.__class__.__name__ == "_NoRedirect" for handler in handlers) + + class TestCustomFetchersFromConfig: def test_builds_fetcher(self, monkeypatch): monkeypatch.setenv("LITELLM_KEY", "sk-proxy") @@ -245,6 +405,22 @@ def test_missing_key_skipped(self, monkeypatch): } assert _custom_fetchers_from_config(config) == [] + @pytest.mark.parametrize("provider", ["openrouter", "deepseek"]) + def test_protected_provider_cannot_use_generic_provider_config(self, provider): + """A custom entry cannot bypass model-list HTTPS/no-redirect controls.""" + config = { + "providers": [ + { + "name": provider, + "base_url": "http://attacker.example/models", + "api_key": "sentinel-key", + } + ] + } + assert _custom_fetchers_from_config(config) == [] + # With no model_list entry, startup has no authorized discovery fetcher. + assert fetch_live_provider_models(config, timeout=0.1) == [] + def test_fetch_live_includes_custom(self, monkeypatch): for var in ("ANTHROPIC_API_KEY", "OPENAI_API_KEY", "MISTRAL_API_KEY"): monkeypatch.delenv(var, raising=False) diff --git a/tests/test_oom_diagnostics.py b/tests/test_oom_diagnostics.py index 7318f03..b30df80 100644 --- a/tests/test_oom_diagnostics.py +++ b/tests/test_oom_diagnostics.py @@ -21,7 +21,9 @@ def test_disabled_diagnostics_do_not_create_an_artifact(monkeypatch, tmp_path): assert list(tmp_path.iterdir()) == [] -def test_enabled_records_counters_but_never_request_or_response_content(monkeypatch, tmp_path): +def test_enabled_records_counters_but_never_request_or_response_content( + monkeypatch, tmp_path +): secret = "do-not-write-this-request-or-response" monkeypatch.setenv("AIRLOCK_OOM_DIAGNOSTICS", "1") monkeypatch.setenv("AIRLOCK_OOM_DIAGNOSTICS_DIR", str(tmp_path)) @@ -33,8 +35,10 @@ def test_enabled_records_counters_but_never_request_or_response_content(monkeypa diagnostics.post_call(data) diagnostics.record_event( SimpleNamespace( - guardrail_meta=dict(data["metadata"]), success=True, - response_obj=secret, error=secret, + guardrail_meta=dict(data["metadata"]), + success=True, + response_obj=secret, + error=secret, ) ) @@ -44,9 +48,14 @@ def test_enabled_records_counters_but_never_request_or_response_content(monkeypa assert artifact.stat().st_mode & 0o777 == 0o600 assert secret not in content assert {record["phase"] for record in records} >= { - "diagnostics_started", "request_entry", "provider_response", "callback_complete" + "diagnostics_started", + "request_entry", + "provider_response", + "callback_complete", } - completion = next(record for record in records if record["phase"] == "callback_complete") + completion = next( + record for record in records if record["phase"] == "callback_complete" + ) assert completion["outcome"] == "success" assert completion["in_flight"] == 0 assert "allocator" in completion and "transport" in completion @@ -66,12 +75,17 @@ def test_diagnostics_are_bounded_and_trim_skips_inflight(monkeypatch, tmp_path): for _ in range(8): diagnostics.snapshot("extra") - records = [json.loads(line) for line in next(tmp_path.glob("*.jsonl")).read_text().splitlines()] + records = [ + json.loads(line) + for line in next(tmp_path.glob("*.jsonl")).read_text().splitlines() + ] assert len(records) == 4 assert any(record["phase"] == "signal_usr2_skipped_in_flight" for record in records) -def test_tracemalloc_can_be_disabled_for_a_low_perturbation_replay(monkeypatch, tmp_path): +def test_tracemalloc_can_be_disabled_for_a_low_perturbation_replay( + monkeypatch, tmp_path +): monkeypatch.setenv("AIRLOCK_OOM_DIAGNOSTICS", "1") monkeypatch.setenv("AIRLOCK_OOM_DIAGNOSTICS_TRACEMALLOC", "0") monkeypatch.setenv("AIRLOCK_OOM_DIAGNOSTICS_DIR", str(tmp_path)) diff --git a/tests/test_operational_reads.py b/tests/test_operational_reads.py new file mode 100644 index 0000000..4262b1e --- /dev/null +++ b/tests/test_operational_reads.py @@ -0,0 +1,147 @@ +"""Slice 110 RED contracts for explicit, bounded operational read selection.""" + +from __future__ import annotations + +import datetime +import json + +from airlock.operational_reads import read_records + + +class _Node: + def __init__(self, body: dict): + self.body = json.dumps(body) + + +def _today_record(**extra) -> dict: + return { + "timestamp": datetime.datetime.now(datetime.timezone.utc).isoformat(), + "model": "gpt-4o-mini", + **extra, + } + + +def test_default_read_uses_jsonl_even_when_fathom_engine_exists( + monkeypatch, log_dir +) -> None: + import airlock.datastore + + monkeypatch.setattr( + airlock.datastore, + "get_engine", + lambda: (_ for _ in ()).throw(AssertionError("default must not open FathomDB")), + ) + day = datetime.datetime.now(datetime.timezone.utc).date().isoformat() + (log_dir / f"airlock-{day}.jsonl").write_text(json.dumps(_today_record()) + "\n") + + page = read_records(directory=log_dir, days=1) + + assert page.source == "jsonl" + assert page.degraded_reason is None + assert len(page.records) == 1 + + +def test_jsonl_selection_excludes_non_record_json_values(monkeypatch, log_dir) -> None: + day = datetime.datetime.now(datetime.timezone.utc).date().isoformat() + (log_dir / f"airlock-{day}.jsonl").write_text( + json.dumps("not an event") + "\n" + json.dumps(_today_record()) + "\n" + ) + + page = read_records(directory=log_dir, days=1) + + assert len(page.records) == 1 + assert isinstance(page.records[0], dict) + + +def test_opted_in_fathom_read_is_bounded_and_source_labelled( + monkeypatch, log_dir +) -> None: + import airlock.datastore + + monkeypatch.setenv("AIRLOCK_OPERATIONAL_READ_BACKEND", "fathomdb") + monkeypatch.setattr(airlock.datastore, "get_engine", lambda: "engine") + monkeypatch.setattr( + "airlock.operational_reads.get_request_logs", + lambda engine, limit: [ + _Node(_today_record(request_id=str(i))) for i in range(limit) + ], + ) + + page = read_records(directory=log_dir, days=1, limit=2) + + assert page.source == "fathomdb" + assert page.truncated is True + assert page.limit_hit == "datastore_limit" + assert len(page.records) == 2 + + +def test_ui_fallback_never_opens_a_second_selected_engine(monkeypatch, log_dir) -> None: + import airlock.datastore + + day = datetime.datetime.now(datetime.timezone.utc).date().isoformat() + (log_dir / f"airlock-{day}.jsonl").write_text(json.dumps(_today_record()) + "\n") + monkeypatch.setenv("AIRLOCK_OPERATIONAL_READ_BACKEND", "fathomdb") + monkeypatch.setattr( + airlock.datastore, + "get_engine", + lambda: (_ for _ in ()).throw(AssertionError("TUI must use proxy bridge")), + ) + + page = read_records(directory=log_dir, days=1, allow_fathom=False) + + assert page.source == "jsonl" + assert "proxy operational reads unavailable" in page.degraded_reason + + +def test_unavailable_or_invalid_opt_in_falls_back_to_labelled_jsonl( + monkeypatch, log_dir +) -> None: + import airlock.datastore + + day = datetime.datetime.now(datetime.timezone.utc).date().isoformat() + (log_dir / f"airlock-{day}.jsonl").write_text(json.dumps(_today_record()) + "\n") + monkeypatch.setenv("AIRLOCK_OPERATIONAL_READ_BACKEND", "fathomdb") + monkeypatch.setattr(airlock.datastore, "get_engine", lambda: None) + + page = read_records(directory=log_dir, days=1) + assert page.source == "jsonl" + assert ( + page.degraded_reason == "FathomDB selected but unavailable; using bounded JSONL" + ) + + monkeypatch.setenv("AIRLOCK_OPERATIONAL_READ_BACKEND", "unexpected") + page = read_records(directory=log_dir, days=1) + assert page.source == "jsonl" + assert "invalid operational backend" in page.degraded_reason + + +def test_fathom_query_failure_falls_back_without_leaking_the_error( + monkeypatch, log_dir +) -> None: + import airlock.datastore + + day = datetime.datetime.now(datetime.timezone.utc).date().isoformat() + (log_dir / f"airlock-{day}.jsonl").write_text(json.dumps(_today_record()) + "\n") + monkeypatch.setenv("AIRLOCK_OPERATIONAL_READ_BACKEND", "fathomdb") + monkeypatch.setattr(airlock.datastore, "get_engine", lambda: "engine") + monkeypatch.setattr( + "airlock.operational_reads.get_request_logs", + lambda engine, limit: (_ for _ in ()).throw(RuntimeError("provider secret")), + ) + + page = read_records(directory=log_dir, days=1) + + assert page.source == "jsonl" + assert ( + page.degraded_reason == "FathomDB selected but unavailable; using bounded JSONL" + ) + + +def test_logs_source_text_labels_degradation_and_bounds() -> None: + from airlock.tui.screens.logs import _operational_source_text + + rendered = _operational_source_text("jsonl", "FathomDB unavailable", True) + + assert "JSONL" in rendered + assert "FathomDB unavailable" in rendered + assert "bounded result" in rendered diff --git a/tests/test_pii_egress.py b/tests/test_pii_egress.py index 09ef4d2..a6e0f1a 100644 --- a/tests/test_pii_egress.py +++ b/tests/test_pii_egress.py @@ -23,9 +23,7 @@ def test_round_trip_tool_allows_email(monkeypatch): def test_known_bad_class_vetoes_round_trip(monkeypatch): monkeypatch.setenv("AIRLOCK_PII_EGRESS_TOOL_BANDS", '{"card_lookup":"round_trip"}') - decision = decide( - tool="card_lookup", path="/card", placeholder="" - ) + decision = decide(tool="card_lookup", path="/card", placeholder="") assert decision.allow is False assert decision.reason == "sensitive_class" @@ -36,12 +34,8 @@ def test_exfil_requires_residual_allow(monkeypatch): "AIRLOCK_PII_EGRESS_ALLOWLIST", '[{"tool":"send_mail","path":"/to","entity_type":"EMAIL_ADDRESS"}]', ) - allowed = decide( - tool="send_mail", path="/to", placeholder="" - ) - denied = decide( - tool="send_mail", path="/body", placeholder="" - ) + allowed = decide(tool="send_mail", path="/to", placeholder="") + denied = decide(tool="send_mail", path="/body", placeholder="") assert allowed.allow is True assert denied.allow is False assert denied.reason == "exfil_not_allowlisted" @@ -69,9 +63,7 @@ def test_known_bad_blocklist_vetoes_residual_allow(monkeypatch): '[{"tool":"send_mail","path":"/to","entity_type":"EMAIL_ADDRESS"}]', ) - decision = decide( - tool="send_mail", path="/to", placeholder="" - ) + decision = decide(tool="send_mail", path="/to", placeholder="") assert decision.allow is False assert decision.reason == "known_bad" diff --git a/tests/test_pii_guard.py b/tests/test_pii_guard.py index 4db09f8..8f3ffdd 100644 --- a/tests/test_pii_guard.py +++ b/tests/test_pii_guard.py @@ -17,6 +17,7 @@ _hydration_enabled, _requires_nlp_entities, _scrub_messages, + _scrub_embedding_input, _scrub_text, _scrub_text_with_mapping, ) @@ -84,9 +85,19 @@ def test_default_entities_do_not_require_spacy_ner(self): assert _requires_nlp_entities(_configured_entities()) is False def test_shipped_bank_entities_do_not_require_spacy_ner(self): - assert _requires_nlp_entities( - ["CREDIT_CARD", "US_SSN", "EMAIL_ADDRESS", "PHONE_NUMBER", "US_BANK_NUMBER", "IBAN_CODE"] - ) is False + assert ( + _requires_nlp_entities( + [ + "CREDIT_CARD", + "US_SSN", + "EMAIL_ADDRESS", + "PHONE_NUMBER", + "US_BANK_NUMBER", + "IBAN_CODE", + ] + ) + is False + ) def test_named_entities_require_spacy_ner(self): assert _requires_nlp_entities(["EMAIL_ADDRESS", "PERSON"]) is True @@ -179,6 +190,26 @@ def test_original_messages_not_mutated(self): _scrub_messages(messages) assert messages[0]["content"] == original_content + def test_embedding_input_scrubbed_without_changing_batch_shape(self, monkeypatch): + def scrub(text, mapping, counters): + mapping.setdefault("", "alice@example.com") + return text.replace("alice@example.com", "").replace( + "555-123-4567", "" + ) + + monkeypatch.setattr( + "airlock.guardrails.pii_guard._scrub_text_with_mapping", scrub + ) + mapping: dict[str, str] = {} + result = _scrub_embedding_input( + ["Contact alice@example.com", ["Call 555-123-4567"]], mapping, {} + ) + assert isinstance(result, list) + assert isinstance(result[1], list) + assert "alice@example.com" not in result[0] + assert "555-123-4567" not in result[1][0] + assert mapping + # --------------------------------------------------------------------------- # Graceful degradation when Presidio not installed @@ -362,6 +393,31 @@ async def test_hook_empty_messages(self, mock_cache, mock_user_api_key_dict): ) assert result["messages"] == [] + async def test_aembedding_hook_scrubs_input_and_refreshes_text_cache( + self, mock_cache, mock_user_api_key_dict, monkeypatch + ): + def scrub(text, mapping, counters): + mapping.setdefault("", "alice@corp.com") + return text.replace("alice@corp.com", "").replace( + "555-123-4567", "" + ) + + monkeypatch.setattr( + "airlock.guardrails.pii_guard._scrub_text_with_mapping", scrub + ) + guard = AirlockPIIGuard() + data = { + "input": ["benchmark alice@corp.com", "call 555-123-4567"], + "model": "text-embedding-3-small", + } + result = await guard.async_pre_call_hook( + mock_user_api_key_dict, mock_cache, data, "aembedding" + ) + assert "alice@corp.com" not in str(result["input"]) + assert "555-123-4567" not in str(result["input"]) + assert "alice@corp.com" not in result["metadata"]["_airlock_text"] + assert "airlock_pii_handle" not in result["metadata"] + class TestOffloadByteParity: """The asyncio.to_thread offload must not change redaction output: the diff --git a/tests/test_provider_errors.py b/tests/test_provider_errors.py new file mode 100644 index 0000000..8f13acd --- /dev/null +++ b/tests/test_provider_errors.py @@ -0,0 +1,102 @@ +"""Slice 40 provider-error boundary tests.""" + +from __future__ import annotations + +import datetime +from unittest.mock import MagicMock + +import pytest +from litellm.exceptions import APIError + +from airlock.callbacks.projections import project_enterprise, project_s3, project_sql +from airlock.callbacks.request_event import build_request_event +from airlock.callbacks.tracing import AirlockTracingCallback +from airlock.fast.monitor import _is_provider_rate_limited +from airlock.provider_errors import summarize_provider_error + + +def _provider_failure(message: str, status: int) -> APIError: + """Use LiteLLM's actual provider-error shape, not an Airlock test double.""" + return APIError(status, message, "openrouter", "openai/gpt-4o-mini") + + +def _kwargs(exc: Exception) -> dict: + return { + "model": "openrouter/openai/gpt-4o-mini", + "exception": exc, + "messages": [{"role": "user", "content": "sentinel-request"}], + "litellm_params": { + "metadata": { + "airlock_provider": "openrouter", + "user_api_key_alias": "sentinel-user", + } + }, + } + + +@pytest.mark.parametrize("status", [401, 402, 429, 500, 503]) +def test_provider_failure_is_bounded_before_request_event_and_projections(status): + secret = "sentinel-provider-response-and-key" + event = build_request_event( + _kwargs(_provider_failure(secret, status)), + None, + datetime.datetime.now(datetime.timezone.utc), + datetime.datetime.now(datetime.timezone.utc), + success=False, + ) + + assert ( + event.error + == f"provider_error type=APIError provider=openrouter status={status}" + ) + assert event.bare_exception_error == event.error + for projection in ( + project_enterprise(event), + project_s3(event), + project_sql(event), + ): + assert secret not in repr(projection) + assert projection["error"] == event.error + + +def test_provider_429_keeps_typed_detection_without_text_leakage(): + exc = _provider_failure("rate limit sentinel quota exhausted", 429) + limited, reason = _is_provider_rate_limited(exc) + assert limited is True + assert reason == "provider_error type=APIError provider=openrouter status=429" + assert "sentinel" not in reason + + +def test_provider_error_summary_rejects_unattributed_local_error(): + assert summarize_provider_error(ValueError("sentinel local error")) is None + + +def test_provider_error_summary_bounds_unknown_provider_marker(): + summary = summarize_provider_error( + APIError(500, "harmless", "sentinel_provider_error_payload_ABC123", "x") + ) + assert summary is not None + assert summary.provider is None + assert "sentinel" not in summary.message() + + +def test_tracing_exports_bounded_provider_exception(monkeypatch): + from airlock.callbacks import tracing as tracing_module + + span = MagicMock() + tracer = MagicMock() + tracer.start_as_current_span.return_value.__enter__ = lambda _: span + tracer.start_as_current_span.return_value.__exit__ = MagicMock(return_value=False) + monkeypatch.setattr(tracing_module, "_tracer", tracer) + monkeypatch.setattr(tracing_module, "_OTEL_AVAILABLE", True) + + AirlockTracingCallback().log_failure_event( + _kwargs(_provider_failure("sentinel upstream body", 503)), None, None, None + ) + attrs = {call.args[0]: call.args[1] for call in span.set_attribute.call_args_list} + assert ( + attrs["llm.error"] + == "provider_error type=APIError provider=openrouter status=503" + ) + recorded = span.record_exception.call_args.args[0] + assert "sentinel" not in str(recorded) diff --git a/tests/test_proxy_bootstrap.py b/tests/test_proxy_bootstrap.py index 6b4608e..298377e 100644 --- a/tests/test_proxy_bootstrap.py +++ b/tests/test_proxy_bootstrap.py @@ -11,6 +11,7 @@ "docs", "circuit_health", "error_handlers", + "embedding_boundary", "admin", "batch", "models_capability", @@ -34,6 +35,9 @@ def _install(): monkeypatch.setattr( pb, "install_airlock_error_handlers_on_proxy_app", _mk("error_handlers") ) + monkeypatch.setattr( + pb, "install_embedding_request_boundary_on_proxy_app", _mk("embedding_boundary") + ) monkeypatch.setattr(pb, "install_admin_on_proxy_app", _mk("admin")) monkeypatch.setattr(pb, "install_batch_gateway_on_proxy_app", _mk("batch")) monkeypatch.setattr( diff --git a/tests/test_proxy_errors.py b/tests/test_proxy_errors.py index c89dc5c..efb983a 100644 --- a/tests/test_proxy_errors.py +++ b/tests/test_proxy_errors.py @@ -3,22 +3,203 @@ from __future__ import annotations import json +from unittest.mock import AsyncMock import pytest from litellm import RateLimitError from airlock.proxy_errors import ( + AirlockEndpointNotSupported, + AirlockGatewayRoutingOverride, AirlockInvalidReasoningEffort, AirlockProviderBlocked, airlock_invalid_reasoning_effort_handler, airlock_provider_blocked_handler, block_response_payload, invalid_reasoning_effort_response_payload, + endpoint_not_supported_response_payload, install_airlock_error_handlers_on_proxy_app, retry_after_seconds, ) +def test_endpoint_not_supported_payload_is_openai_shaped() -> None: + body, headers = endpoint_not_supported_response_payload( + AirlockEndpointNotSupported("gpt-4o-mini", "embeddings") + ) + assert headers == {} + assert body["error"]["type"] == "invalid_request_error" + assert body["error"]["code"] == "model_endpoint_not_supported" + assert body["error"]["param"] == "model" + + +def test_openrouter_routing_override_is_openai_shaped() -> None: + exc = AirlockGatewayRoutingOverride("extra_body.route") + body = exc.to_dict() + assert body["code"] == "gateway_routing_override_not_allowed" + assert body["param"] == "extra_body.route" + + +def test_deepseek_non_function_tool_is_openai_shaped() -> None: + from airlock.proxy_errors import AirlockDeepSeekToolNotSupported + + body = AirlockDeepSeekToolNotSupported().to_dict() + assert body["code"] == "deepseek_non_function_tool_not_supported" + assert body["param"] == "tools" + + +async def test_deepseek_tool_rejection_survives_litellm_proxy_translation( + mock_user_api_key_dict, +) -> None: + from litellm.proxy._types import ProxyException + from litellm.proxy.common_request_processing import ProxyBaseLLMRequestProcessing + from litellm.proxy.proxy_server import openai_exception_handler + from starlette.requests import Request + + from airlock.proxy_errors import AirlockDeepSeekToolNotSupported + + proxy_logging = type( + "ProxyLoggingStub", + (), + { + "post_call_failure_hook": AsyncMock(return_value=None), + "post_call_response_headers_hook": AsyncMock(return_value={}), + }, + )() + with pytest.raises(ProxyException) as raised: + await ProxyBaseLLMRequestProcessing(data={})._handle_llm_api_exception( + e=AirlockDeepSeekToolNotSupported(), + user_api_key_dict=mock_user_api_key_dict, + proxy_logging_obj=proxy_logging, + version="test", + ) + exc = raised.value + assert (exc.code, exc.type, exc.param, exc.openai_code) == ( + "400", + "invalid_request_error", + "tools", + "deepseek_non_function_tool_not_supported", + ) + response = await openai_exception_handler( + Request( + { + "type": "http", + "method": "POST", + "path": "/v1/chat/completions", + "headers": [], + } + ), + exc, + ) + assert response.status_code == 400 + assert json.loads(response.body)["error"] == { + "message": "DeepSeek supports OpenAI function tools only.", + "type": "invalid_request_error", + "param": "tools", + "code": "deepseek_non_function_tool_not_supported", + } + + +async def test_embedding_rejection_survives_litellm_proxy_exception_translation( + mock_user_api_key_dict, +) -> None: + """LiteLLM catches guardrail exceptions inside `/v1/embeddings` routes.""" + from litellm.proxy._types import ProxyException + from litellm.proxy.common_request_processing import ProxyBaseLLMRequestProcessing + from litellm.proxy.proxy_server import openai_exception_handler + from starlette.requests import Request + + proxy_logging = type( + "ProxyLoggingStub", + (), + { + "post_call_failure_hook": AsyncMock(return_value=None), + "post_call_response_headers_hook": AsyncMock(return_value={}), + }, + )() + processor = ProxyBaseLLMRequestProcessing(data={}) + with pytest.raises(ProxyException) as raised: + await processor._handle_llm_api_exception( + e=AirlockEndpointNotSupported("gpt-4o-mini", "embeddings"), + user_api_key_dict=mock_user_api_key_dict, + proxy_logging_obj=proxy_logging, + version="test", + ) + + exc = raised.value + assert exc.code == "400" + assert exc.type == "invalid_request_error" + assert exc.param == "model" + assert exc.openai_code == "model_endpoint_not_supported" + response = await openai_exception_handler( + Request( + {"type": "http", "method": "POST", "path": "/v1/embeddings", "headers": []} + ), + exc, + ) + assert response.status_code == 400 + assert json.loads(response.body)["error"] == { + "message": "Model 'gpt-4o-mini' is not configured for the 'embeddings' endpoint.", + "type": "invalid_request_error", + "param": "model", + "code": "model_endpoint_not_supported", + "airlock": {"endpoint": "embeddings", "model": "gpt-4o-mini"}, + } + + +async def test_openrouter_override_rejection_survives_litellm_proxy_translation( + mock_user_api_key_dict, +) -> None: + """The pre-dispatch provider guard must reach clients as an OpenAI 400.""" + from litellm.proxy._types import ProxyException + from litellm.proxy.common_request_processing import ProxyBaseLLMRequestProcessing + from litellm.proxy.proxy_server import openai_exception_handler + from starlette.requests import Request + + proxy_logging = type( + "ProxyLoggingStub", + (), + { + "post_call_failure_hook": AsyncMock(return_value=None), + "post_call_response_headers_hook": AsyncMock(return_value={}), + }, + )() + with pytest.raises(ProxyException) as raised: + await ProxyBaseLLMRequestProcessing(data={})._handle_llm_api_exception( + e=AirlockGatewayRoutingOverride("extra_body.route"), + user_api_key_dict=mock_user_api_key_dict, + proxy_logging_obj=proxy_logging, + version="test", + ) + + exc = raised.value + assert exc.code == "400" + assert exc.type == "invalid_request_error" + assert exc.param == "extra_body.route" + assert exc.openai_code == "gateway_routing_override_not_allowed" + response = await openai_exception_handler( + Request( + { + "type": "http", + "method": "POST", + "path": "/v1/chat/completions", + "headers": [], + } + ), + exc, + ) + assert response.status_code == 400 + assert json.loads(response.body)["error"] == { + "message": ( + "OpenRouter routing option 'extra_body.route' is operator-controlled " + "and cannot be set by a client request." + ), + "type": "invalid_request_error", + "param": "extra_body.route", + "code": "gateway_routing_override_not_allowed", + } + + class TestAirlockProviderBlocked: def test_is_a_rate_limit_error(self): exc = AirlockProviderBlocked( diff --git a/tests/test_request_event.py b/tests/test_request_event.py index 05d133e..0a63023 100644 --- a/tests/test_request_event.py +++ b/tests/test_request_event.py @@ -236,9 +236,7 @@ def test_guardrail_meta_excludes_request_scoped_pii_reverse_map(): } ) - event = build_request_event( - kwargs, _FakeResponse(), _ts(0), _ts(1), success=True - ) + event = build_request_event(kwargs, _FakeResponse(), _ts(0), _ts(1), success=True) assert "airlock_pii_map" not in event.guardrail_meta assert secret not in repr(event.guardrail_meta) diff --git a/tests/test_served_attribution.py b/tests/test_served_attribution.py index 55aca7f..06be27e 100644 --- a/tests/test_served_attribution.py +++ b/tests/test_served_attribution.py @@ -208,12 +208,14 @@ def test_classify_backend_kind_gateway() -> None: assert classify_backend_kind("azure") == "gateway" assert classify_backend_kind("vertex_ai") == "gateway" assert classify_backend_kind("vertex_ai_beta") == "gateway" + assert classify_backend_kind("openrouter") == "gateway" def test_classify_backend_kind_native() -> None: assert classify_backend_kind("anthropic") == "native" assert classify_backend_kind("openai") == "native" assert classify_backend_kind("gemini") == "native" + assert classify_backend_kind("deepseek") == "native" def test_classify_backend_kind_unknown() -> None: diff --git a/tests/test_tui.py b/tests/test_tui.py index 5b0b8ff..af85e29 100644 --- a/tests/test_tui.py +++ b/tests/test_tui.py @@ -38,7 +38,7 @@ async def test_app_has_bindings(app) -> None: async def test_app_composes_all_panes() -> None: - app = AirlockApp() + app = AirlockApp(test_harness=True) async with app.run_test(size=(120, 40)) as _pilot: # Tab bar exists (no sidebar) tab_bar = app.query_one("#tab-bar") @@ -55,7 +55,7 @@ async def test_app_composes_all_panes() -> None: async def test_screen_switching_via_keys() -> None: - app = AirlockApp() + app = AirlockApp(test_harness=True) async with app.run_test(size=(120, 40)) as pilot: workspace = app.query_one("#workspace") @@ -76,7 +76,7 @@ async def test_screen_switching_via_keys() -> None: async def test_tab_bar_navigation() -> None: - app = AirlockApp() + app = AirlockApp(test_harness=True) async with app.run_test(size=(120, 40)) as pilot: workspace = app.query_one("#workspace") @@ -93,7 +93,7 @@ async def test_tab_bar_navigation() -> None: async def test_overview_has_widgets() -> None: - app = AirlockApp() + app = AirlockApp(test_harness=True) async with app.run_test(size=(120, 40)) as _pilot: # Proxy status indicator indicator = app.query_one("#ov-proxy-indicator") @@ -113,7 +113,7 @@ async def test_overview_has_widgets() -> None: async def test_overview_has_start_button() -> None: - app = AirlockApp() + app = AirlockApp(test_harness=True) async with app.run_test(size=(120, 40)) as _pilot: btn = app.query_one("#ov-proxy-btn", Button) assert btn is not None @@ -127,7 +127,7 @@ async def test_overview_has_start_button() -> None: async def test_overview_has_console_log() -> None: - app = AirlockApp() + app = AirlockApp(test_harness=True) async with app.run_test(size=(120, 40)) as _pilot: from textual.widgets import Collapsible, RichLog diff --git a/tests/test_tui_harness.py b/tests/test_tui_harness.py new file mode 100644 index 0000000..7fecf27 --- /dev/null +++ b/tests/test_tui_harness.py @@ -0,0 +1,63 @@ +"""Slice 70 deterministic composition harness contracts.""" + +from __future__ import annotations + +from unittest.mock import MagicMock + +from airlock.tui.app import AirlockApp + + +async def test_harness_composes_production_widget_tree_without_mount_workers( + monkeypatch, +) -> None: + """Ordinary render/navigation tests need real panes, not background I/O.""" + start_health = MagicMock() + monkeypatch.setattr( + "airlock.tui.mcp_manager.McpServerManager.start_health_loop", start_health + ) + monkeypatch.setattr( + "airlock.tui.screens.overview.OverviewPane._refresh_state", MagicMock() + ) + monkeypatch.setattr( + "airlock.tui.screens.overview.OverviewPane._probe_external", MagicMock() + ) + monkeypatch.setattr( + "airlock.tui.screens.config.ConfigPane._refresh_mcp_servers", MagicMock() + ) + monkeypatch.setattr("airlock.tui.screens.guards.GuardsPane._poll_logs", MagicMock()) + + app = AirlockApp(test_harness=True) + async with app.run_test(size=(120, 40)): + for pane_id in ("overview", "guards", "logs", "config", "test", "advisor"): + assert app.query_one(f"#{pane_id}") is not None + + start_health.assert_not_called() + assert app._jsonl_thread is None + + +async def test_default_app_keeps_mount_lifecycle_enabled(monkeypatch) -> None: + """The harness flag cannot silently disable production lifecycle behavior.""" + # Keep this a focused lifecycle assertion: normal mode must invoke the + # app-level starters, but it must not leave the pane workers/timers alive + # after Textual's test pilot exits. + monkeypatch.setattr( + "airlock.tui.screens.overview.OverviewPane._refresh_state", MagicMock() + ) + monkeypatch.setattr( + "airlock.tui.screens.overview.OverviewPane._probe_external", MagicMock() + ) + monkeypatch.setattr( + "airlock.tui.screens.config.ConfigPane._refresh_mcp_servers", MagicMock() + ) + monkeypatch.setattr("airlock.tui.screens.logs.LogsPane._load_logs", MagicMock()) + monkeypatch.setattr("airlock.tui.screens.guards.GuardsPane._poll_logs", MagicMock()) + monkeypatch.setattr( + "airlock.tui.screens.guards.GuardsPane._load_semantic_report", MagicMock() + ) + app = AirlockApp() + app._mcp_manager.start_health_loop = MagicMock() + app._start_jsonl_tailer = MagicMock() + async with app.run_test(size=(120, 40)): + pass + app._mcp_manager.start_health_loop.assert_called_once() + app._start_jsonl_tailer.assert_called_once() diff --git a/tests/test_tui_qos_telemetry.py b/tests/test_tui_qos_telemetry.py new file mode 100644 index 0000000..735926b --- /dev/null +++ b/tests/test_tui_qos_telemetry.py @@ -0,0 +1,89 @@ +"""Slice 90 contracts for QoS and exporter-health operator diagnostics.""" + +from __future__ import annotations + +from airlock.admin.http import _view_clients +from airlock.tui.screens.overview import ( + _render_client_priority, + _render_telemetry_health, +) +from airlock import telemetry_health + + +def test_priority_snapshot_is_bounded_and_labels_disabled_admission( + fresh_state_store, monkeypatch +) -> None: + monkeypatch.setattr( + "airlock.admin.http.get_settings", + lambda: type( + "Settings", (), {"admission": type("Admission", (), {"enabled": False})()} + )(), + ) + fresh_state_store.record_client_priority( + "alice", 0.8765, True, ["interactive_session(avg_gap=1.0s)"] * 8, 123.0 + ) + + payload = _view_clients() + priority = payload["clients"]["alice"]["priority"] + + assert payload["source"] == "live_admin" + assert priority == { + "score": 0.876, + "boost": True, + "reasons": ["interactive_session(avg_gap=1.0s)"] * 4, + "observed_at": 123.0, + "admission_enabled": False, + } + + +def test_telemetry_endpoint_and_error_are_safe(monkeypatch) -> None: + monkeypatch.setattr(telemetry_health, "_health", {}) + telemetry_health.configure_exporter( + "otlp", enabled=True, endpoint="https://secret:pw@trace.example:4318/path?q=key" + ) + telemetry_health.record_signal("otlp") + raw_error = "provider response sentinel must not be shown" + telemetry_health.record_export_failure("otlp", raw_error) + + snapshot = telemetry_health.telemetry_snapshot() + + assert snapshot["otlp"]["endpoint"] == "https://trace.example:4318" + assert snapshot["otlp"]["signals"] == 1 + assert snapshot["otlp"]["failures"] == 1 + assert snapshot["otlp"]["last_error"] == "export_error" + assert raw_error not in str(snapshot) + assert "secret" not in str(snapshot) + assert "?" not in str(snapshot) + + +def test_malformed_telemetry_endpoint_fails_open(monkeypatch) -> None: + monkeypatch.setattr(telemetry_health, "_health", {}) + + telemetry_health.configure_exporter( + "otlp", enabled=True, endpoint="https://trace.example:bad/path" + ) + + assert telemetry_health.telemetry_snapshot()["otlp"]["endpoint"] is None + + +def test_telemetry_renderer_labels_unavailable_and_never_claims_delivery() -> None: + assert "unavailable" in _render_telemetry_health(None) + + rendered = _render_telemetry_health( + {"prometheus": {"enabled": True, "signals": 4, "endpoint": "http://in-process"}} + ) + + assert "prometheus: enabled" in rendered + assert "signals=4" in rendered + assert "successfully exported" not in rendered + + +def test_stale_priority_is_not_rendered_as_active() -> None: + line, signals = _render_client_priority( + {"score": 0.9, "boost": True, "observed_at": 1.0, "admission_enabled": True}, + now=200.0, + ) + + assert "stale" in line + assert "active boost" not in line + assert signals == "" diff --git a/tests/test_tui_routing.py b/tests/test_tui_routing.py new file mode 100644 index 0000000..7ed656d --- /dev/null +++ b/tests/test_tui_routing.py @@ -0,0 +1,70 @@ +"""Slice 80 contracts for bounded routing/operator diagnostics.""" + +from __future__ import annotations + +from airlock.fast.router import apply_routing +from airlock.tui.screens.guards import _parse_entry, _render_routing + + +def _routing_record(complexity: str, *, model: str = "claude-haiku") -> dict: + return { + "timestamp": "2026-08-12T12:00:00+00:00", + "model": model, + "airlock_routing": { + "smart_classify": {"complexity": complexity, "score": 0.2}, + "routed_model": model, + "reasons": ["cost_tier(low)"], + }, + } + + +def test_routing_only_record_is_visible_without_prompt_content() -> None: + entry = _parse_entry(_routing_record("simple")) + + assert entry is not None + assert entry.routing is not None + assert entry.routing["smart_classify"]["complexity"] == "simple" + + +def test_routing_panel_labels_bounded_source_and_distribution() -> None: + entries = [ + _parse_entry(_routing_record("simple")), + _parse_entry(_routing_record("complex")), + _parse_entry(_routing_record("complex")), + ] + assert all(entries) + + rendered = _render_routing(entries[0], entries) # type: ignore[arg-type] + + assert "simple: 1" in rendered + assert "complex: 2" in rendered + assert "bounded JSONL window" in rendered + assert "persisted request metadata" in rendered + + +def test_session_identifier_is_not_recorded_in_routing_metadata( + fresh_state_store, +) -> None: + data = { + "model": "claude-sonnet", + "metadata": {"airlock": {"session_id": "do-not-log"}}, + } + + result = apply_routing(data, client_id="alice") + + assert "do-not-log" not in str(result["metadata"]["airlock_routing"]) + + +def test_routing_panel_survives_malformed_persisted_metadata() -> None: + entry = _parse_entry( + { + "timestamp": "2026-08-12T12:00:00+00:00", + "model": "claude-haiku", + "airlock_routing": {"smart_classify": "not-a-mapping", "reasons": 1}, + } + ) + assert entry is not None + + rendered = _render_routing(entry, [entry]) + + assert "No smart-routing classifications" in rendered diff --git a/uv.lock b/uv.lock index 60dafa8..dc31ad0 100644 --- a/uv.lock +++ b/uv.lock @@ -168,7 +168,7 @@ wheels = [ [[package]] name = "airlock-llm" -version = "0.5.12" +version = "0.5.14" source = { editable = "." } dependencies = [ { name = "litellm", extra = ["proxy"] }, @@ -232,7 +232,7 @@ requires-dist = [ { name = "anthropic", marker = "extra == 'analyzer'", specifier = ">=0.40.0,<1" }, { name = "boto3", marker = "extra == 's3'", specifier = ">=1.34.0" }, { name = "fastapi", marker = "extra == 'test'", specifier = ">=0.100.0" }, - { name = "fathomdb", marker = "extra == 'db'", specifier = ">=0.8.21,<0.9" }, + { name = "fathomdb", marker = "extra == 'db'", specifier = ">=0.8.22,<0.9" }, { name = "google-auth", marker = "extra == 'vertex'", specifier = ">=2.0.0" }, { name = "google-genai", marker = "extra == 'aistudio'", specifier = ">=1.0.0" }, { name = "litellm", extras = ["proxy"], specifier = ">=1.94.1,<2" }, @@ -1167,11 +1167,14 @@ wheels = [ [[package]] name = "fathomdb" -version = "0.8.21" +version = "0.8.22" source = { registry = "https://pypi.org/simple" } wheels = [ - { url = "https://files.pythonhosted.org/packages/94/5a/0f522b8580e5c3cd79bfcf345eb1003c766170b44d670a621bce04a9dcb6/fathomdb-0.8.21-cp310-abi3-manylinux_2_28_aarch64.whl", hash = "sha256:516185f0f46222083c8bb7a12a4997f5b50368196a89b574bb1ab0fc4a7a3aa7", size = 6142201, upload-time = "2026-08-05T18:22:03.688Z" }, - { url = "https://files.pythonhosted.org/packages/9b/b8/d9795052f00b6e8a6931410826c4bd3a1df12bd2c45ebe3aef4c242ce68a/fathomdb-0.8.21-cp310-abi3-manylinux_2_28_x86_64.whl", hash = "sha256:b87813d615bbeac946b2ca46245e9750d9c73c5efbc86ea7402cbace50d4f4cc", size = 6743413, upload-time = "2026-08-05T18:22:05.3Z" }, + { url = "https://files.pythonhosted.org/packages/1d/a2/7413b89bb5d0577c529cfa8fd57350ddf606d3320b892d9d116d4397be3b/fathomdb-0.8.22-cp310-abi3-macosx_10_12_x86_64.whl", hash = "sha256:046772f0208befb021440c467c07764dc9f1c086c6c4cb0396295eb5c3638c76", size = 5861605, upload-time = "2026-08-08T19:07:46.05Z" }, + { url = "https://files.pythonhosted.org/packages/ed/87/c64ea422e337db419a660861d3be1e90417eea537ef1bfee61592c4c3597/fathomdb-0.8.22-cp310-abi3-macosx_11_0_arm64.whl", hash = "sha256:0f29ed942956be8a81fd1f083147397c1ae6361542c273ae7934d679af75afb4", size = 5547937, upload-time = "2026-08-08T19:07:48.094Z" }, + { url = "https://files.pythonhosted.org/packages/7c/31/8afb61238022bbe5d38ed59799653b9d56362d62cc94dc0f2b45a7d6579f/fathomdb-0.8.22-cp310-abi3-manylinux_2_28_aarch64.whl", hash = "sha256:b79d3e02adf5a232045f1467e935fe933b62d3f5c23f3f9247ea32a8339c3829", size = 6195581, upload-time = "2026-08-08T19:07:50.188Z" }, + { url = "https://files.pythonhosted.org/packages/54/56/e7a25644fe9882f19dbe2a5d0ca624755cef85c2121e6b6c65d9aba6a6e4/fathomdb-0.8.22-cp310-abi3-manylinux_2_28_x86_64.whl", hash = "sha256:7ad77f228c0cdfaf6f6146d4014f0e63b705c9dee039fc7a8cb0e2df56e641a8", size = 6800816, upload-time = "2026-08-08T19:07:52.327Z" }, + { url = "https://files.pythonhosted.org/packages/b4/a1/c35a392df09cdefc37929cf204b232223609fb7f6629bb8e0ab3183b50a2/fathomdb-0.8.22-cp310-abi3-win_amd64.whl", hash = "sha256:84eb2d64f442895159e893caf257fc2170cd3fc17088b84152f13370b50c6e3f", size = 5457860, upload-time = "2026-08-08T19:07:54.239Z" }, ] [[package]] From be36bb8651bcf19ec7ecbda57eb27f08e980d219 Mon Sep 17 00:00:00 2001 From: Corey Thompson Date: Sat, 15 Aug 2026 16:36:01 -0500 Subject: [PATCH 05/24] ci(security): add gitleaks secret scanning --- .github/CODEOWNERS | 4 + .github/workflows/gitleaks.yml | 31 +++++ .gitleaksignore | 38 ++++++ .pre-commit-config.yaml | 6 + .../0.5.15/10-gitleaks-ci-secret-scan-plan.md | 112 ++++++++++++++++++ .../10-gitleaks-ci-secret-scan-status.md | 83 +++++++++++++ dev/requirements.md | 25 ++++ 7 files changed, 299 insertions(+) create mode 100644 .github/CODEOWNERS create mode 100644 .github/workflows/gitleaks.yml create mode 100644 .gitleaksignore create mode 100644 .pre-commit-config.yaml create mode 100644 dev/plans/0.5.15/10-gitleaks-ci-secret-scan-plan.md create mode 100644 dev/plans/0.5.15/10-gitleaks-ci-secret-scan-status.md diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS new file mode 100644 index 0000000..add514f --- /dev/null +++ b/.github/CODEOWNERS @@ -0,0 +1,4 @@ +# Security scanning controls require the repository security owner. +/.gitleaksignore @coreyt +/.pre-commit-config.yaml @coreyt +/.github/workflows/gitleaks.yml @coreyt diff --git a/.github/workflows/gitleaks.yml b/.github/workflows/gitleaks.yml new file mode 100644 index 0000000..188288e --- /dev/null +++ b/.github/workflows/gitleaks.yml @@ -0,0 +1,31 @@ +name: gitleaks + +on: + pull_request: + branches: [main] + push: + branches: [main] + workflow_dispatch: + schedule: + - cron: "17 4 * * 1" + +permissions: + contents: read + pull-requests: read + +jobs: + scan: + name: scan + runs-on: ubuntu-latest + timeout-minutes: 5 + steps: + - uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6 + with: + fetch-depth: 0 + - uses: gitleaks/gitleaks-action@e0c47f4f8be36e29cdc102c57e68cb5cbf0e8d1e # v3 + env: + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + GITLEAKS_VERSION: "8.30.0" + GITLEAKS_ENABLE_COMMENTS: "false" + GITLEAKS_ENABLE_UPLOAD_ARTIFACT: "false" + GITLEAKS_ENABLE_SUMMARY: "false" diff --git a/.gitleaksignore b/.gitleaksignore new file mode 100644 index 0000000..bea2001 --- /dev/null +++ b/.gitleaksignore @@ -0,0 +1,38 @@ +dev/plans/runs/0.5.1-design-review-20260625T003640Z.md:generic-api-key:4872 +dev/plans/runs/0.5.1-design-review-20260625T003640Z.md:generic-api-key:4884 +dev/plans/runs/0.5.1-design-review-20260625T003640Z.md:generic-api-key:4917 +docs/getting-started/installation.md:curl-auth-header:78 +scripts/setup.sh:curl-auth-header:89 +tests/conftest.py:generic-api-key:141 +tests/test_admin_tokens.py:generic-api-key:20 +tests/test_admin_tokens.py:generic-api-key:74 +tests/test_client_identity.py:generic-api-key:166 +tests/test_client_identity.py:generic-api-key:204 +tests/test_client_identity.py:generic-api-key:228 +tests/test_fast_guardian.py:generic-api-key:30 +tests/test_fast_guardian.py:generic-api-key:34 +tests/test_fast_guardian.py:generic-api-key:57 +tests/test_fast_monitor.py:generic-api-key:52 +tests/test_fast_monitor.py:generic-api-key:64 +tests/test_fast_monitor.py:generic-api-key:97 +tests/test_guardrail_overrides.py:generic-api-key:14 +tests/test_mcp_tool_guard.py:generic-api-key:118 +tests/test_paid_services.py:generic-api-key:160 +tests/test_paid_services.py:generic-api-key:175 +tests/test_response_scanner.py:generic-api-key:151 +10bbda93e66bbd3c3c373ec9c830e02ee3b1bbbf:tests/test_fast_guardian.py:generic-api-key:53 +24c16b891855871c50e4730de88212dd2ab45a4f:docs/getting-started/installation.md:curl-auth-header:66 +285fecf6b6290f686df58912f6fdb45d9484125d:tests/test_mcp_tool_guard.py:generic-api-key:116 +285fecf6b6290f686df58912f6fdb45d9484125d:tests/test_response_scanner.py:generic-api-key:150 +404b9e7e28429e718ad09bff674c577fdc543fcf:tests/test_client_identity.py:generic-api-key:196 +404b9e7e28429e718ad09bff674c577fdc543fcf:tests/test_client_identity.py:generic-api-key:214 +4218d2b2de93abeca9dc837dccd3be35dfe8f578:tests/conftest.py:generic-api-key:48 +4218d2b2de93abeca9dc837dccd3be35dfe8f578:tests/test_fast_guardian.py:generic-api-key:24 +4218d2b2de93abeca9dc837dccd3be35dfe8f578:tests/test_fast_guardian.py:generic-api-key:28 +4218d2b2de93abeca9dc837dccd3be35dfe8f578:tests/test_fast_monitor.py:generic-api-key:22 +4218d2b2de93abeca9dc837dccd3be35dfe8f578:tests/test_fast_monitor.py:generic-api-key:33 +4218d2b2de93abeca9dc837dccd3be35dfe8f578:tests/test_fast_monitor.py:generic-api-key:43 +819406e902ba0e0c505f8bd0d69b44bfe2032842:tests/test_fast_monitor.py:generic-api-key:35 +8f6b151ef72bcf38dc967395f12ef8f63844e384:scripts/setup.sh:curl-auth-header:87 +daf23856dcf634aa3eb0248502ba3e13cdd0c7fe:README.md:curl-auth-header:74 +f5626f2959006ab8636c237162388765f9b38251:tests/test_guardrail_overrides.py:generic-api-key:15 diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml new file mode 100644 index 0000000..5ad2c10 --- /dev/null +++ b/.pre-commit-config.yaml @@ -0,0 +1,6 @@ +repos: + - repo: https://github.com/gitleaks/gitleaks + rev: v8.30.0 + hooks: + - id: gitleaks + stages: [pre-commit] diff --git a/dev/plans/0.5.15/10-gitleaks-ci-secret-scan-plan.md b/dev/plans/0.5.15/10-gitleaks-ci-secret-scan-plan.md new file mode 100644 index 0000000..7cc05e7 --- /dev/null +++ b/dev/plans/0.5.15/10-gitleaks-ci-secret-scan-plan.md @@ -0,0 +1,112 @@ +# 0.5.15 Slice 10 — Gitleaks secret-scan implementation plan + +**Status:** repository-controlled implementation complete locally — updated +2026-08-15. External GitHub merge-blocking enforcement remains pending; see +`10-gitleaks-ci-secret-scan-status.md` for the exact evidence and limitation. + +## Admission check + +| Gate | Current evidence | Required before implementation starts | +| --- | --- | --- | +| Previous slice closure | Slices 0–5 and Slice 6 are complete; Slice 6 includes Slice 10 with external enforcement conditional. | **Passed.** Recheck the decision and all changes since this plan before beginning the slice. | +| Draft contract | Slice 3 proposes DFR-35/DAC-35, and Slice 4 confirms isolated architecture alignment. | Ratify the draft as the Slice 10 contract at slice start. | +| Security design review | `70-gitleaks-ci-secret-scan-design.md` and review `71` conditionally pass. | Security reviewer confirms the current pins, 38-entry baseline, and external-enforcement posture still match the accepted design. | +| Repository authority | Wake records approval of the least-privilege control direction and scoped scanner ownership. | GitHub administrator confirms authority for a disposable PR, required check, branch protection, and code-owner enforcement. | + +Until all four gates pass, do not commit, push, enable the workflow, configure +GitHub settings, add dependencies, or claim that merge-blocking enforcement is +active. + +## Changes since the original design + +| Original assumption | Current reconciled state | Plan update | +| --- | --- | --- | +| Slice 0 found an incomplete Python 3.11 environment. | The worktree was rebuilt on Python 3.12.3 with `make sync && make verify`; the ordinary non-live CI baseline, docs, lint, mypy, Docker build, and audit passed. | Treat this as a healthy pre-feature baseline only. Slice 10 has no Python/runtime dependency and must not add one. | +| Local and historical baselines needed reconciliation. | `.gitleaksignore` now has 38 exact entries: the original local findings plus the 16 reviewed commit-qualified history findings described by design review 71. | Re-review the entries as exact fingerprints; do not broaden paths, disable rules, or add wildcard suppression. Prove both directory and history scans before commit. | +| Action SHAs were to be resolved. | The untracked workflow pins checkout v6 to `d23441a48e516b6c34aea4fa41551a30e30af803` and Gitleaks Action v3 to `e0c47f4f8be36e29cdc102c57e68cb5cbf0e8d1e`; scanner remains `8.30.0`. | Independently reconfirm release provenance at slice start. Do not substitute mutable tags or combine this work with Actions-major PR #44. | +| Gitleaks and pre-commit tooling were absent from `PATH`. | They remain absent; only `actionlint` is presently available. | Use a reviewed, ephemeral pinned local-tool invocation or a checksum-verified Gitleaks 8.30.0 binary without adding it to `pyproject.toml`, `uv.lock`, Docker, or Airlock runtime. Record the exact invocation and version in the slice status. | +| Controls were proposed repository changes. | `.gitleaksignore`, `.pre-commit-config.yaml`, `.github/CODEOWNERS`, and `.github/workflows/gitleaks.yml` are still untracked worktree baseline. | Preserve them exactly until review; stage only these named files and the Slice 10 status/documentation files—never broad staging. | + +## Scope and non-goals + +**In scope:** an isolated Gitleaks GitHub workflow; a Gitleaks 8.30.0 +pre-commit hook; a narrow exact-fingerprint baseline; scanner-file code +ownership; minimal developer instructions; local, disposable-PR, and external +GitHub-enforcement evidence. + +**Out of scope:** Airlock runtime code, `config.yaml`, Docker image, provider +credentials, deployment/release workflows, Dependabot PRs, Action major +upgrades, GitHub Advanced Security/code scanning uploads, broad scanner +exceptions, and any secret-management redesign. + +## Implementation sequence + +1. **Ratify and snapshot scope.** Record the Slice 6 include decision, DFR-35/ + DAC-35 acceptance criteria, designated security owner, GitHub administrator, + and the reviewed file list. Recheck that the branch/worktree still contains + only the known untracked baseline; preserve unrelated user files. +2. **Re-verify supply chain and baseline.** Independently verify both full + Action SHAs against the stated official releases. Obtain the exact Gitleaks + `8.30.0` scanner by a reviewed ephemeral/checksum-verified path. Run + redacted `gitleaks dir .` and `gitleaks git .`; review any result + individually. A clean result is required before the controls are committed. +3. **Record RED evidence.** In a disposable branch/PR, add a synthetic, + non-usable detector fixture that is designed to be redacted. Demonstrate + local-hook and CI failure without copying the value to logs, plans, or PR + comments; remove the fixture immediately after evidence capture. +4. **Review and commit the repository controls.** Review only + `.pre-commit-config.yaml`, `.gitleaksignore`, `.github/CODEOWNERS`, and + `.github/workflows/gitleaks.yml`. Confirm stable `gitleaks / scan` naming, + full checkout, PR/push/manual/weekly triggers, read-only permissions, no + `pull_request_target`, no OIDC/write scope, and comments/artifact/summary + disabled. Add a short developer instruction only if review identifies an + existing authoritative location; do not create duplicate authority. +5. **Record GREEN evidence.** Re-run local directory/history scans, the staged + pre-commit hook, and `actionlint`. Open a disposable PR, prove a redacted CI + failure then a green run after fixture removal; validate manual dispatch and + a normal `main`-equivalent push only through authorized repository workflow. +6. **Apply external enforcement.** The GitHub administrator configures `main` + to require `gitleaks / scan` and code-owner review for the three scanner + files. Prove a failing scan and an unapproved scanner/baseline modification + cannot merge. Do not report this as active before this external evidence. +7. **Close the slice.** Write `10-gitleaks-ci-secret-scan-status.md` with + ratified contract, exact pins/version, baseline review count, commands and + redacted results, CI/PR URLs, GitHub-rule evidence, residual runtime-binary + provenance risk, and rollback disposition. + +## Required verification and acceptance evidence + +| Boundary | Evidence | +| --- | --- | +| Local detection parity | Gitleaks `8.30.0` directory and reachable-history scans both produce zero reviewed findings; the hook invokes the same scanner version. | +| False-positive discipline | Baseline entries are exact fingerprints, individually reviewed, and line/path or commit changes create a new finding. No rule, directory, or generic test/docs exemption is introduced. | +| Workflow isolation | `actionlint` passes; workflow has only `contents: read` and `pull-requests: read`, full checkout, no repository/deployment secrets, no artifact/SARIF/comment/summary publication, and no runtime invocation. | +| Detection regression | Disposable synthetic fixture fails locally and in CI with redaction, then removal restores green. The fixture is not retained in mainline or copied into durable evidence. | +| Merge enforcement | GitHub administrator supplies branch-protection and code-owner proof that the stable check and designated review cannot be bypassed on `main`. | +| Existing CI health | Re-run the affected workflow validation plus the repository’s ordinary CI evidence proportionate to touched files; no Python dependency or product-test expansion is expected. | + +## Failure handling and rollback + +- A new finding is treated as potentially real until security review proves it + is a non-usable fixture. Rotate/revoke any usable credential before changing + a baseline. +- A false positive is resolved only by a reviewed exact fingerprint after + preserving detector coverage. Never suppress the rule or an entire path to + unblock a merge. +- If the workflow itself is defective, revert the named scanner-control commit + or have the GitHub administrator temporarily remove the required check with a + recorded security decision. Do not use an unprotected direct push as a + workaround. +- If the official Action’s runtime-binary provenance becomes unacceptable, + pause enforcement and design a separately reviewed digest-pinned alternative; + do not add a second divergent scanner as an emergency replacement. + +## Completion criteria + +Slice 10 closes only when DFR-35/DAC-35 is ratified, repository controls are +committed and reviewable, all local/CI redacted evidence passes, the controlled +failure proof is complete, the GitHub administrator has demonstrated required +check and code-owner enforcement, and the status record names all residual +risks. If external GitHub authority is unavailable, close only the +repository-controlled portion as conditional and explicitly state that +merge-blocking enforcement is not active. diff --git a/dev/plans/0.5.15/10-gitleaks-ci-secret-scan-status.md b/dev/plans/0.5.15/10-gitleaks-ci-secret-scan-status.md new file mode 100644 index 0000000..9267ec2 --- /dev/null +++ b/dev/plans/0.5.15/10-gitleaks-ci-secret-scan-status.md @@ -0,0 +1,83 @@ +# 0.5.15 Slice 10 — Gitleaks secret-scan status + +**Status:** repository-controlled implementation complete locally — 2026-08-15. +GitHub branch-protection and code-owner enforcement are **not active yet**; +authenticated external verification and enforcement are in progress. + +## Ratified contract and changed controls + +Slice 6 includes Slice 10. DFR-35/DAC-35 is now ratified in +`dev/requirements.md`: the control is a non-deploying Gitleaks `8.30.0` +pre-commit hook plus an isolated `gitleaks / scan` workflow. It has no Airlock +runtime, configuration, image, provider, or deployment-credential impact. + +| File | Delivered control | +| --- | --- | +| `.pre-commit-config.yaml` | Staged Gitleaks hook from the official `v8.30.0` repository revision. It is a developer-local installation step and is not a Python project dependency. | +| `.gitleaksignore` | 38 individually reviewed exact fingerprints: local fixture findings plus commit-qualified history findings. No broad path/rule exception or inline allow comment is present. | +| `.github/workflows/gitleaks.yml` | Isolated `gitleaks / scan` workflow on PR/push to `main`, manual dispatch, and weekly history scan; full checkout; scanner `8.30.0`; read-only permissions; comments, artifacts, and summaries disabled. | +| `.github/CODEOWNERS` | `@coreyt` owns the baseline, hook configuration, and workflow paths. This becomes enforcement only after GitHub branch rules require code-owner review. | +| `dev/requirements.md` | Ratified DFR-35/DAC-35 contract and acceptance boundary. | + +## Pin and scanner provenance + +The workflow’s full-SHA pins were independently checked with read-only remote +tag lookups: + +| Component | Required tag | Verified SHA | +| --- | --- | --- | +| `actions/checkout` | `v6` | `d23441a48e516b6c34aea4fa41551a30e30af803` | +| `gitleaks/gitleaks-action` | `v3` / `v3.0.0` | `e0c47f4f8be36e29cdc102c57e68cb5cbf0e8d1e` | +| Local verifier | Gitleaks `v8.30.0` | `ghcr.io/gitleaks/gitleaks@sha256:691af3c7c5a48b16f187ce3446d5f194838f91238f27270ed36eef6359a574d9` | + +The container digest was used read-only against the worktree and shared Git +metadata. It verifies the specified scanner version; the GitHub Action’s own +runtime-binary download remains the accepted no-secret-job residual risk from +the design review. + +## RED/GREEN and local verification evidence + +| Check | Result | +| --- | --- | +| `gitleaks dir . --redact` | Passed; 18.47 MB scanned, no leaks found. | +| `gitleaks git . --redact` | Passed; 686 commits / 10.43 MB scanned, no leaks found. The shared `.git` directory was mounted read-only because this is a linked worktree. | +| Controlled RED detector fixture | Passed: a temporary synthetic non-usable `sk-test-…` fixture produced one finding with `--redact`; it was deleted immediately and never added to the worktree. | +| Configured hook | Passed via ephemeral `pre-commit==4.1.0`: `pre-commit run gitleaks --all-files`. No project dependency, lockfile, or image change. | +| Workflow syntax | `actionlint .github/workflows/gitleaks.yml` passed. | +| Baseline integrity | 38 entries, no duplicate fingerprints. | +| Ordinary local readiness | `make ensure-spacy && make verify` passed after restoring `en_core_web_lg==3.8.0`; strict MkDocs build and documentation contract tests passed (8). | + +The first container history scan could not see the linked worktree’s common Git +metadata and scanned zero commits. Re-running with that metadata mounted +read-only produced the 686-commit clean result above; this was a test-topology +correction, not a scanner suppression. + +## Outstanding external acceptance criteria + +The local/repository-control portion is ready for focused commit and push. The +following remains incomplete and must not be represented as active: + +1. A GitHub administrator must configure `main` to require the stable + `gitleaks / scan` check and code-owner review for the three scanner-control + files. +2. A disposable PR must show a redacted CI failure for the synthetic fixture, + then a green result after removal; manual-dispatch, main-push, and fork-PR + behavior must be evidenced without repository secrets or elevated token. +3. The branch rule must prove that a scanner/baseline change cannot merge + without its designated owner. + +Authenticated `gh` inspection confirms that `main` currently has no legacy +branch-protection rule. The only `protect-main` ruleset is disabled and covers +deletion/non-fast-forward only. Apply the narrow required-check/code-owner rule +only after the feature PR has produced the stable `gitleaks / scan` check, then +record the resulting PR/run URLs and branch-rule response before closing Slice +10 as merge-blocking. + +## Residual risk and rollback + +False positives require individual security review and an exact fingerprint; +they are not resolved by disabling a detector or excluding a directory. A +usable finding requires rotation/revocation before any baseline action. If the +workflow itself proves defective, revert only the named scanner-control change +or have an administrator temporarily remove the required check with a recorded +security decision—never bypass it by direct unprotected push. diff --git a/dev/requirements.md b/dev/requirements.md index 806d47f..2964bb7 100644 --- a/dev/requirements.md +++ b/dev/requirements.md @@ -344,3 +344,28 @@ single-owner and optional. Selected separate-process reads SHALL use a loopback-only proxy-admin bridge (and require its local admin configuration), never a second engine open. A FathomDB erasure receipt SHALL continue to state that JSONL retention/deletion is a separate obligation. + +--- + +## 0.5.15 ratified requirements + +### DFR-35 / DAC-35: Secret-scan delivery control + +Airlock's repository SHALL use a dedicated, non-deploying Gitleaks control at +scanner version `8.30.0`: a staged pre-commit hook and an isolated GitHub +Actions `gitleaks / scan` job for pull requests to `main`, pushes to `main`, +manual dispatch, and scheduled reachable-history scans. The workflow SHALL use +full-SHA-pinned actions, a full checkout, and only `contents: read` and +`pull-requests: read` permissions. It SHALL not use `pull_request_target`, +write/OIDC permissions, repository/deployment secrets, comments, artifact or +SARIF upload, or result summaries. + +The baseline SHALL contain only individually reviewed exact fingerprints; +broad path/rule exclusions and inline allow comments are prohibited. Scanner +configuration files SHALL have designated code ownership, and `main` SHALL +require both that review and the stable `gitleaks / scan` check once a GitHub +administrator configures the external rule. Directory and reachable-history +scans must be clean for the reviewed baseline; a synthetic non-usable detector +fixture must fail redacted and pass after removal. The control SHALL not alter +Airlock runtime behavior, configuration, image contents, or provider/deployment +credential access. From 8b753652951d161da90dd57b75d5da35aef3845f Mon Sep 17 00:00:00 2001 From: Corey Thompson Date: Sat, 15 Aug 2026 19:47:25 -0500 Subject: [PATCH 06/24] docs(security): record gitleaks enforcement evidence --- .../0.5.15/10-gitleaks-ci-secret-scan-plan.md | 7 +-- .../10-gitleaks-ci-secret-scan-status.md | 51 ++++++++++--------- 2 files changed, 32 insertions(+), 26 deletions(-) diff --git a/dev/plans/0.5.15/10-gitleaks-ci-secret-scan-plan.md b/dev/plans/0.5.15/10-gitleaks-ci-secret-scan-plan.md index 7cc05e7..b770c81 100644 --- a/dev/plans/0.5.15/10-gitleaks-ci-secret-scan-plan.md +++ b/dev/plans/0.5.15/10-gitleaks-ci-secret-scan-plan.md @@ -1,8 +1,9 @@ # 0.5.15 Slice 10 — Gitleaks secret-scan implementation plan -**Status:** repository-controlled implementation complete locally — updated -2026-08-15. External GitHub merge-blocking enforcement remains pending; see -`10-gitleaks-ci-secret-scan-status.md` for the exact evidence and limitation. +**Status:** implemented and externally enforced — updated 2026-08-15. `main` +now requires the stable Gitleaks check and CODEOWNERS review; the feature PR +remains subject to that review and ordinary CI. See +`10-gitleaks-ci-secret-scan-status.md` for exact evidence and limitations. ## Admission check diff --git a/dev/plans/0.5.15/10-gitleaks-ci-secret-scan-status.md b/dev/plans/0.5.15/10-gitleaks-ci-secret-scan-status.md index 9267ec2..8816061 100644 --- a/dev/plans/0.5.15/10-gitleaks-ci-secret-scan-status.md +++ b/dev/plans/0.5.15/10-gitleaks-ci-secret-scan-status.md @@ -1,8 +1,8 @@ # 0.5.15 Slice 10 — Gitleaks secret-scan status -**Status:** repository-controlled implementation complete locally — 2026-08-15. -GitHub branch-protection and code-owner enforcement are **not active yet**; -authenticated external verification and enforcement are in progress. +**Status:** implemented and externally enforced — 2026-08-15. GitHub `main` +now requires `gitleaks / scan` and one CODEOWNERS approval. PR #49 remains +properly blocked pending that review and its ordinary CI completion. ## Ratified contract and changed controls @@ -52,26 +52,31 @@ metadata and scanned zero commits. Re-running with that metadata mounted read-only produced the 686-commit clean result above; this was a test-topology correction, not a scanner suppression. -## Outstanding external acceptance criteria - -The local/repository-control portion is ready for focused commit and push. The -following remains incomplete and must not be represented as active: - -1. A GitHub administrator must configure `main` to require the stable - `gitleaks / scan` check and code-owner review for the three scanner-control - files. -2. A disposable PR must show a redacted CI failure for the synthetic fixture, - then a green result after removal; manual-dispatch, main-push, and fork-PR - behavior must be evidenced without repository secrets or elevated token. -3. The branch rule must prove that a scanner/baseline change cannot merge - without its designated owner. - -Authenticated `gh` inspection confirms that `main` currently has no legacy -branch-protection rule. The only `protect-main` ruleset is disabled and covers -deletion/non-fast-forward only. Apply the narrow required-check/code-owner rule -only after the feature PR has produced the stable `gitleaks / scan` check, then -record the resulting PR/run URLs and branch-rule response before closing Slice -10 as merge-blocking. +## External enforcement evidence + +| Acceptance boundary | Evidence | +| --- | --- | +| Green PR scan | [PR #49](https://github.com/coreyt/airlock/pull/49) completed its `gitleaks / scan` job successfully: [run 31917799169](https://github.com/coreyt/airlock/actions/runs/31917799169). | +| Manual dispatch | The same workflow passed from `workflow_dispatch` on the feature branch: [run 31918015039](https://github.com/coreyt/airlock/actions/runs/31918015039). | +| Red CI proof | Disposable [PR #50](https://github.com/coreyt/airlock/pull/50) contained only the synthetic non-usable fixture. Its [scan job](https://github.com/coreyt/airlock/actions/runs/31917967497/job/95092956354) failed as expected, while GitHub reported `BLOCKED` and `REVIEW_REQUIRED`. The PR was closed and its remote/local branch and worktree were deleted immediately. | +| Required check | `main` branch protection now requires strict `gitleaks / scan`; the context is explicitly bound to the Gitleaks check. | +| Code-owner enforcement | `main` now requires one approving review and `require_code_owner_reviews=true`; the scanner files are owned by `@coreyt`. | +| Bypass/branch safety | Administrator enforcement is enabled; force pushes and deletion are disabled. The pre-existing disabled deletion/non-fast-forward ruleset was not modified. | + +The initial inspection confirmed no active legacy branch-protection rule, so +this is a narrow new protection rather than a replacement of an existing active +policy. + +## Remaining release evidence + +- PR #49 still requires its designated code-owner approval and its ordinary + Python 3.12 CI completion before merge. The Gitleaks check is already green. +- A protected `main` push will be evidenced after that reviewed merge. The + workflow definition contains the required `push: main` trigger, but this + slice did not bypass protection to manufacture a direct push. +- A separate public-fork PR was not created. The workflow uses `pull_request`, + read-only job permissions, and no repository/deployment secrets; retain that + as a deployment-policy review item if fork execution needs empirical proof. ## Residual risk and rollback From ab4f7359caf0ebbaf48983e807236f75419933dc Mon Sep 17 00:00:00 2001 From: Corey Thompson Date: Sat, 15 Aug 2026 20:18:30 -0500 Subject: [PATCH 07/24] fix(guardian): return typed threat backoff 429 --- airlock/fast/guardian.py | 10 +- airlock/proxy_errors.py | 90 ++++++- .../20-fast-guardian-threat-backoff-plan.md | 75 ++++++ .../20-fast-guardian-threat-backoff-status.md | 48 ++++ dev/plans/0.5.15/README.md | 247 ++++++++++++++++++ dev/requirements.md | 11 + docs/guide/rate-limiting.md | 8 + tests/harness/test_phase2_precall.py | 3 +- tests/harness/test_phase4_e2e.py | 3 +- tests/test_fast_guardian.py | 12 +- tests/test_proxy_errors.py | 96 +++++++ 11 files changed, 587 insertions(+), 16 deletions(-) create mode 100644 dev/plans/0.5.15/20-fast-guardian-threat-backoff-plan.md create mode 100644 dev/plans/0.5.15/20-fast-guardian-threat-backoff-status.md create mode 100644 dev/plans/0.5.15/README.md diff --git a/airlock/fast/guardian.py b/airlock/fast/guardian.py index 093224c..ce86a46 100644 --- a/airlock/fast/guardian.py +++ b/airlock/fast/guardian.py @@ -41,6 +41,7 @@ AirlockGatewayRoutingOverride, AirlockModelNotFound, AirlockProviderBlocked, + AirlockThreatBackoff, sanitize_reason, ) from airlock.reasoning_effort import ( @@ -328,18 +329,13 @@ async def async_pre_call_hook( client_id, remaining, ) - raise ValueError( - f"Too many requests. Please retry after {int(remaining)} seconds." - ) + raise AirlockThreatBackoff(retry_after=remaining) # ---- Step 2: Threat assessment ---- message_text = extract_text(data, call_type) or None threat = assess_threat(client, message_text) if threat.blocked: - raise ValueError( - "Request blocked due to unusual activity. " - f"Please retry after {int(threat.backoff_seconds)} seconds." - ) + raise AirlockThreatBackoff(retry_after=threat.backoff_seconds) # Routing and circuit breaker are model-specific — skip for MCP and # batch/file calls (the latter carry no top-level model). diff --git a/airlock/proxy_errors.py b/airlock/proxy_errors.py index 5144796..04ad208 100644 --- a/airlock/proxy_errors.py +++ b/airlock/proxy_errors.py @@ -4,7 +4,9 @@ :class:`AirlockProviderBlocked` (a ``RateLimitError`` subclass) so the client receives an HTTP 429 with a ``Retry-After`` header and an enriched but OpenAI-compatible body — distinguishable from a passthrough provider 429 without -string-parsing. The handler is registered on the LiteLLM proxy app via +string-parsing. A local Fast Guardian threat backoff uses the separately typed +:class:`AirlockThreatBackoff` contract. Handlers are registered on the LiteLLM +proxy app via :func:`install_airlock_error_handlers_on_proxy_app`, mirroring the other ``install_*_on_proxy_app`` hooks in ``model_override_headers``. """ @@ -174,6 +176,58 @@ def __init__(self, message: str, *, retry_after: float) -> None: self.retry_after = float(retry_after) +class AirlockThreatBackoff(ProxyException, RateLimitError): + """A local Fast Guardian threat backoff, not a provider rate limit. + + Retains only the remaining duration needed to tell the client when to retry; + client identity, threat signals, and triggering request details stay inside + the Guardian/logging boundary. + """ + + def __init__(self, *, retry_after: float) -> None: + retry_after_seconds_value = max(1, math.ceil(float(retry_after))) + # RateLimitError preserves existing rate-limit catches and telemetry. + # Calling it explicitly matters because ProxyException is first in the + # MRO so LiteLLM's guardrail pipeline preserves our stable body schema. + RateLimitError.__init__( + self, + message="Too many requests. Please retry later.", + llm_provider="airlock", + model="threat_backoff", + headers={"Retry-After": str(retry_after_seconds_value)}, + ) + # Do not call ProxyException.__init__: with this intentional multiple + # inheritance its cooperative ``super()`` would re-enter + # RateLimitError. Mirror its small public protocol after RateLimitError + # has initialized the OpenAI exception base. + self.message = "Too many requests. Please retry later." + self.type = "airlock_threat_backoff" + self.param = None + self.openai_code = "threat_backoff" + self.code = "429" + self.headers = {"Retry-After": str(retry_after_seconds_value)} + self.provider_specific_fields = { + "airlock": { + "source": "threat_backoff", + "retry_after": retry_after_seconds_value, + } + } + self.retry_after = float(retry_after) + + def to_dict(self) -> dict: + """Use LiteLLM's direct guardrail path without nesting Airlock fields.""" + return { + "message": self.message, + "type": self.type, + "param": self.param, + "code": self.openai_code, + "airlock": { + "source": "threat_backoff", + "retry_after": retry_after_seconds(self.retry_after), + }, + } + + def retry_after_seconds(cooldown_seconds: float) -> int: """Whole-second ``Retry-After`` value, at least 1.""" return max(1, math.ceil(cooldown_seconds)) @@ -247,6 +301,26 @@ def admission_shed_response_payload(exc: AirlockAdmissionShed) -> tuple[dict, di return body, headers +def threat_backoff_response_payload( + exc: AirlockThreatBackoff, +) -> tuple[dict, dict]: + """Build the non-disclosing response for a local threat backoff.""" + retry_after = retry_after_seconds(exc.retry_after) + body = { + "error": { + "message": "Too many requests. Please retry later.", + "type": "airlock_threat_backoff", + "code": "threat_backoff", + "param": None, + "airlock": { + "source": "threat_backoff", + "retry_after": retry_after, + }, + } + } + return body, {"Retry-After": str(retry_after)} + + async def airlock_provider_blocked_handler(request: Any, exc: Exception): """FastAPI exception handler → 429 with Retry-After + enriched body.""" from fastapi.responses import JSONResponse @@ -330,8 +404,17 @@ async def airlock_admission_shed_handler(request: Any, exc: Exception): return JSONResponse(status_code=429, content=body, headers=headers) +async def airlock_threat_backoff_handler(request: Any, exc: Exception): + """FastAPI exception handler → a local-threat 429 with Retry-After.""" + from fastapi.responses import JSONResponse + + assert isinstance(exc, AirlockThreatBackoff) + body, headers = threat_backoff_response_payload(exc) + return JSONResponse(status_code=429, content=body, headers=headers) + + def install_airlock_error_handlers_on_proxy_app() -> bool: - """Register the AirlockProviderBlocked handler on the LiteLLM proxy app. + """Register Airlock-owned typed error handlers on the LiteLLM proxy app. Registered for the subclass specifically (not the base ``RateLimitError``) so passthrough provider 429s keep LiteLLM's own handling — the perimeter only @@ -370,5 +453,8 @@ def install_airlock_error_handlers_on_proxy_app() -> bool: if not getattr(app.state, "airlock_admission_shed_handler_installed", False): app.add_exception_handler(AirlockAdmissionShed, airlock_admission_shed_handler) app.state.airlock_admission_shed_handler_installed = True + if not getattr(app.state, "airlock_threat_backoff_handler_installed", False): + app.add_exception_handler(AirlockThreatBackoff, airlock_threat_backoff_handler) + app.state.airlock_threat_backoff_handler_installed = True app.state.airlock_error_handlers_installed = True return True diff --git a/dev/plans/0.5.15/20-fast-guardian-threat-backoff-plan.md b/dev/plans/0.5.15/20-fast-guardian-threat-backoff-plan.md new file mode 100644 index 0000000..93d1b59 --- /dev/null +++ b/dev/plans/0.5.15/20-fast-guardian-threat-backoff-plan.md @@ -0,0 +1,75 @@ +# Slice 20 — typed Fast Guardian threat-backoff HTTP 429 + +**Status:** implementation in progress (revised 2026-08-15). + +## Prior-slice closure and revision + +Slice 10's repository controls are implemented and externally configured on +`main`; its delivery PR still awaits the required independent CODEOWNERS review +and its ordinary CI result. Slice 20 has no runtime, source, or release-order +dependency on that pending merge, so it may proceed locally without a push. + +The Slice 3 draft, Slice 4 architecture review, Slice 5 verification matrix, +and Slice 6 owner decision remain applicable. The current code confirms the +identified seam: `AirlockFastGuardian` still raises plain `ValueError` for both +an existing client backoff and a new threat block. Existing +`AirlockProviderBlocked` and `AirlockAdmissionShed` handlers already establish +the correct typed, explicitly registered FastAPI boundary. No dependency or +runtime-version change is needed. + +## Ratified contract + +The draft DUN-33 is incorporated into existing `UN-16`; it does not create a +new user need. DFR-34/DAC-34 is ratified as follows: + +> Fast Guardian threat backoff SHALL raise `AirlockThreatBackoff`, an +> Airlock-owned `RateLimitError`, for both the request that creates a backoff +> and requests rejected during it. The proxy SHALL render it as an +> OpenAI-shaped HTTP 429 with a whole-second, minimum-one `Retry-After`, stable +> `type`/`code` `airlock_threat_backoff`/`threat_backoff`, and +> `error.airlock.source: threat_backoff`. It SHALL expose no client identity, +> threat score, heuristic reason, request content, provider identity, or +> provider-circuit-breaker header. Provider and admission 429 contracts remain +> distinct and unchanged. + +## Design and review + +`AirlockThreatBackoff(RateLimitError)` will carry only the raw remaining +duration. `retry_after_seconds()` will convert it at the HTTP boundary with +ceiling and a minimum of one, avoiding early retry from fractional duration or +a race at expiry. The guardian will use that exception for both threat paths; +it will not embed duration or client data in the message. + +LiteLLM may translate a guardrail exception before FastAPI resolves a subclass +handler. Therefore this type also uses LiteLLM's `ProxyException` protocol and +its direct `to_dict()` shape while remaining a `RateLimitError`; an integration +test covers that translation. This is a compatibility adapter, not a +registration of a catch-all provider-rate-limit handler. + +`threat_backoff_response_payload()` and a dedicated FastAPI handler will be +registered specifically by `install_airlock_error_handlers_on_proxy_app`. +Specific registration preserves LiteLLM's treatment of provider 429s and +Airlock admission shedding. The body will use only a generic retry message and +the stable source; its only header will be `Retry-After`. + +This is an additive, pre-existing inference-path error translation: it adds no +network call, storage, configuration, credential, dependency, or TUI/Admin +surface. Failure rollback is the isolated revert of the guardian/error-handler +change. Security review finds the principal risk to be client or heuristic +disclosure; the exception intentionally retains neither and tests assert the +response omits them. Architecture review finds no authority or provider +semantic expansion. + +## TDD and verification + +1. Add focused guardian and proxy-boundary RED tests for typed exceptions, + ceiling/minimum retry, OpenAI shape, installer registration, and absence of + sensitive fields; retain provider/admission regression tests. +2. Make the minimal exception, payload/handler, registration, guardian, and + client-documentation changes. +3. Run focused tests, then relevant rate-limit/error regressions and the + ordinary repository verification applicable to the changed Python/docs + surface. Record RED/GREEN evidence and code review in the Slice 20 status. + +No push is required for this local implementation slice. A later release +closeout decides whether to publish it with other approved work. diff --git a/dev/plans/0.5.15/20-fast-guardian-threat-backoff-status.md b/dev/plans/0.5.15/20-fast-guardian-threat-backoff-status.md new file mode 100644 index 0000000..21e8aa9 --- /dev/null +++ b/dev/plans/0.5.15/20-fast-guardian-threat-backoff-status.md @@ -0,0 +1,48 @@ +# Slice 20 — typed Fast Guardian threat-backoff HTTP 429 status + +**Disposition:** implemented locally; no push required for this slice. + +## Accepted contract + +DFR-34/DAC-34 is ratified in `dev/requirements.md`. `UN-16` remains the +existing user-need authority; draft DUN-33 was merged into it. Both a newly +created threat backoff and an already-active backoff now raise the typed +`AirlockThreatBackoff` exception. Its client response is HTTP 429 with a +minimum-one, whole-second `Retry-After`, `type: airlock_threat_backoff`, +`code: threat_backoff`, and `airlock.source: threat_backoff`. + +The exception retains no client identifier or heuristic reason. Its body is a +generic retry message and does not use provider/circuit-breaker headers. The +existing provider-breaker and admission-shed exception/handler contracts remain +separately registered and regression-tested. + +## TDD and review evidence + +* **RED:** focused tests initially failed collection because + `AirlockThreatBackoff` did not exist. +* **GREEN:** Guardian, proxy-error, admission, and harness regressions pass: + `156 passed, 2 skipped`. +* The test suite exposed LiteLLM's guardrail translation before FastAPI's + subclass handler. The implementation therefore also follows LiteLLM's + `ProxyException` protocol while remaining a `RateLimitError`; its integration + test proves the actual translated response is 429 with `Retry-After: 3` and + the exact OpenAI-shaped error body. +* **Code review:** reviewed the error boundary, integer ceiling/minimum, direct + FastAPI handler, LiteLLM translation path, redaction surface, and provider / + admission regression isolation. The only implementation adjustment was the + explicit LiteLLM protocol noted above. +* `ruff check` and `ruff format --check` pass for all changed Python files; + `make verify` passes; `mkdocs build --strict` passes. +* The full ordinary `make test` was attempted twice. This execution environment + terminated each run after roughly one minute at 21–23% with no failure output; + it did not yield a final result. It remains a release-closeout verification + item. A standalone MyPy run also reports pre-existing missing stubs for + LiteLLM/FathomDB imports outside this slice and is not a repository gate. + +## Residual risk and rollback + +The only nonstandard detail is the narrow dual exception protocol required by +the installed LiteLLM guardrail pipeline; the end-to-end translation test pins +that behavior. Reverting the Guardian/error-handler change restores the former +generic path. No dependency, configuration, provider, secret, TUI/Admin, or +GitHub change was made. diff --git a/dev/plans/0.5.15/README.md b/dev/plans/0.5.15/README.md new file mode 100644 index 0000000..2d39a24 --- /dev/null +++ b/dev/plans/0.5.15/README.md @@ -0,0 +1,247 @@ +# Airlock 0.5.15 — Slice plan + +This is the canonical 0.5.15 release workspace index. + +**Status:** planning complete. Slices 0–5 are complete planning records and +Slice 6’s HITL decision was recorded on 2026-08-15. Slice 10 is implemented and +awaits its required external PR review/CI closure; Slice 20 is implemented +locally with focused verification, while its full ordinary-suite rerun remains +release-closeout work. Other feature work remains subject to its documented +prerequisites and lifecycle. + +**Post-planning guard:** do not begin a feature/function slice merely because it +is listed or included. First recheck the prior slice closure, update the slice +plan for changes since it was drafted, and satisfy its stated security, +architecture, verification, and external-authority gates. + +## Release objective + +Improve Airlock's operator and security control plane while preserving the +inference hot path, secret boundaries, per-instance Admin authorization, and +external deployment/configuration ownership. The release must distinguish work +that is accepted for investigation from work that is approved for delivery. + +## Scope inventory + +| Candidate | Current evidence | Initial allocation | +| --- | --- | --- | +| Gitleaks local and CI secret scanning | Implemented; external enforcement and PR evidence recorded | Slice 10 | +| Typed Fast Guardian threat-backoff HTTP 429 | Implemented locally; status/evidence recorded | Slice 20 | +| Configured-provider credential with no enabled alias warning | Design exists | Slice 30 | +| Read-only provider/configuration visibility in Admin/TUI | Design exists; CRUD is not approved | Slice 40 | +| Secure host-console TUI administration for containers | Design exists; topology/identity decision is open | Slice 50 | +| Airlock-native virtual-key management | Design exists; depends on 0.6.0 identity/keystore contract | Slice 60, conditional | +| Multi-instance TUI administration | Research/design exists; remote operation/configuration authority is open | Slice 70, conditional | +| Release closeout | Required only for work actually included and delivered | Slice 80 | + +Claude Code subscription-preserving pass-through is explicitly out of scope for +0.5.15. The even-minor roadmap migration is planning/documentation work, not a +0.5.15 product feature; Slice 2 must classify it without executing it. + +## Architecture invariants + +Every slice and draft must preserve these rules: + +1. No Admin/TUI, warning, or management addition may subscribe to or contend + with the inference hot path. +2. Provider and virtual-key secrets never appear in logs, API responses, TUI + snapshots, test evidence, or documentation. Configuration views expose only + bounded, source-labelled, redacted state. +3. Docker bridge reachability, SSH, Docker sockets, systemd, and Kubernetes + access are not operator identity or TUI authority. Every instance remains an + Admin policy-enforcement point. +4. Provider policy remains reviewed YAML plus restart unless a later owner + decision establishes durable configuration ownership, validation, atomicity, + rollback, audit, and secret-manager semantics. +5. Fleet management is a named non-secret inventory plus authenticated Admin API + fan-out. It must not become a competing desired-state, host-control, or + provider-configuration authority. +6. Virtual-key persistence, encryption, identity, and authorization must reuse + the ratified 0.6.0 keystore contract. No UI-owned secret store, master-key + reuse, or LiteLLM key CRUD substitute is permitted. + +## Slices 0–6: discovery, proposals, and HITL + +These slices write durable records under `dev/plans/0.5.15/`. They do not change +product behavior, requirements, architecture, test suites, deployment +configuration, dependencies, documentation lifecycle, or GitHub settings. + +### Slice 0 — Environment and execution readiness + +**Purpose:** identify the smallest reproducible environment and any required +environment changes before feature work. + +**Plan:** inventory the dedicated 0.5.15 worktree/branch, Python/uv/lock state, +test extras, Docker availability, Admin/TUI test topology, configuration fixture +handling, and Gitleaks/pre-commit/CI tooling. Identify which tests are offline, +funded, or require explicit operator authorization. Record existing uncommitted +Gitleaks controls as a worktree baseline, not as released behavior. + +**Durable output:** `dev/plans/0.5.15/00-environment.md` with commands, observed +versions, safe test boundaries, required changes, and owners. No setup change is +made in this slice. + +### Slice 1 — Dependabot and library sweep + +**Purpose:** identify dependency/security work that could affect 0.5.15. + +**Plan:** review Dependabot alerts/PRs, locked direct and transitive +dependencies, EOL/security advisories, and libraries proposed by existing +designs. Assess compatibility, license, test/upgrade cost, supply-chain +provenance, and whether each item belongs in this release. Include Gitleaks +Action/Gitleaks binary pin maintenance and existing LiteLLM/FathomDB constraints. + +**Durable output:** `dev/plans/0.5.15/01-dependencies.md`, recording each item as +keep, upgrade proposal, defer, reject, or external monitoring. No dependency, +lockfile, workflow version, or package metadata is changed in this slice. + +### Slice 2 — Repository-wide cruft review + +**Purpose:** classify documentation and engineering artifacts without altering +them. + +**Plan:** enumerate and classify, at minimum: + +- program/release documentation and changelogs; +- `dev/` needs, requirements, acceptance criteria, architecture, code/test + notes, designs, plans, run evidence, and intermediate material; +- developer documentation, agent notes, prompts, examples, and templates; +- public README, guides, reference pages, deployment documentation, and API + documentation; +- obsolete release/version references, duplicate authorities, generated + artifacts, and stale code/test comments discovered during review. + +For each item or coherent group, propose exactly one disposition: **keep**, +**deprecate in place**, **archive**, or **delete**. State authority, consumer, +risk, replacement/link needed, and whether the proposed action belongs in +0.5.15 or the unallocated backlog. + +**Durable output:** `dev/plans/0.5.15/02-cruft-proposal.md`. This is a proposal only: +no document, code, test, design, or public page is moved, edited, archived, or +deleted in Slice 2. + +### Slice 3 — Draft needs, requirements, acceptance criteria, and allocation + +**Purpose:** reconcile the release candidates into draft product contracts and +allocate each draft item to a future feature slice. + +**Plan:** inspect existing `dev/user-needs.md`, `dev/requirements.md`, test +contracts, acceptance criteria, and the detailed 0.5.15 designs. Propose CRUD +changes as drafts only: create, rename, revise, supersede, or delete proposed +User Needs, Requirements, and Acceptance Criteria. Each draft must name its +source, rationale, security/privacy impact, and owning feature slice. + +The draft allocation must at least cover Gitleaks, threat backoff, provider +warning, read-only provider visibility, host-console TUI, virtual keys, and +fleet TUI. It must label virtual-key and fleet requirements conditional, and +must keep provider CRUD and Claude subscription pass-through out of the delivery +allocation unless separately approved. + +**Durable output:** `dev/plans/0.5.15/03-draft-requirements-and-allocation.md` with a +draft-to-slice matrix. Do not mutate canonical needs/requirements/acceptance +criteria in Slice 3. + +### Slice 4 — Architecture review and high-level code alignment + +**Purpose:** propose architecture changes before implementation and determine +whether the current code broadly aligns with the intended boundaries. + +**Plan:** use Slices 0–3 and existing design/research evidence to review: + +- inference-path isolation and import boundaries; +- configuration, credential, identity, authorization, persistence, and audit + ownership; +- Admin/TUI data freshness, redaction, capability, topology, and deployment + boundaries; +- Gitleaks CI's GitHub token, action, baseline, and branch-protection boundary; +- dependencies on the 0.6.0 keystore contract and external desired-state + systems. + +Then perform a deliberately high-level code review of the relevant seams to +identify alignment, drift, missing seams, and code-versus-architecture changes. +No code change is made. + +**Durable output:** `dev/plans/0.5.15/04-architecture-review.md`, including proposed +architecture/code changes, alignment table, blast radius, and rejected +alternatives. + +### Slice 5 — Verification adequacy review + +**Purpose:** determine whether the repository can prove the release contracts. + +**Plan:** trace every draft requirement to acceptance criteria and candidate +tests. Review whether tests adequately cover Airlock's product goals and known +critical paths: inference behavior, response/error contracts, authorization and +negative access, secret redaction, configuration/restart behavior, TUI/Admin +data seams, container/native topology, CI/release controls, and failure/rollback +paths. Classify unit, integration, topology, live/funded, regression, and manual +evidence; identify gaps and test-fixture safety needs. + +**Durable output:** `dev/plans/0.5.15/05-verification-review.md`, with a +requirement-to-acceptance-to-test matrix and proposed test work allocated to the +owning feature slice. No test/code change is made. + +### Slice 6 — Release review and HITL decision + +**Purpose:** collect Slices 0–5 into a concise owner decision record. + +**Plan:** enumerate every proposed product, security, dependency, +documentation-lifecycle, requirement, architecture, and verification item. +Score each with **understood** and **risk** on a 1–4 scale (4 is highest), and +with effort (`XS`, `S`, `M`, `L`, `XL`). Recommend **include**, **conditional**, +or **postpone**, with prerequisites and owner decisions visible. + +Slice 6 must explicitly decide the included feature slices, conditional slices, +documentation/dependency proposals, Gitleaks external enforcement, provider +CRUD posture, host-console topology, virtual-key 0.6.0 dependency, and fleet +authority boundary. + +**Durable output:** `dev/plans/0.5.15/06-hitl-release-review.md`, containing the +scorecard, recommendation, recorded owner response, and exact start order. + +**Hard stop:** present Slice 6 to HITL and stop. Do not start Slice 10 until the +owner has recorded the decision and all prerequisites for the first included +slice are satisfied. + +## Feature/function slices — planned only after Slice 6 + +Each future slice follows the mandatory lifecycle below; the slice names do not +authorize work before Slice 6. + +| Slice | Candidate scope | Initial condition | +| --- | --- | --- | +| 10 | Gitleaks local/CI controls, baseline, ownership, and GitHub enforcement | Slice 6 includes it; external GitHub action/branch-rule authority is available. | +| 20 | Typed Fast Guardian threat-backoff exception and HTTP 429 contract | Slice 3 draft contract and Slice 4 architecture pass. | +| 30 | Redacted startup warning for configured credential/no enabled alias | Provider taxonomy, logging, and redaction drafts pass. | +| 40 | Read-only provider/configuration Admin/TUI visibility | Read-only configuration-owner posture remains approved; no CRUD expansion. | +| 50 | Secure host-console TUI administration for containerized Airlock | Topology, operator identity, TLS/token lifecycle, and audit decisions pass. | +| 60 | Virtual-key store and management | 0.6.0 keystore/identity contract and threat model are ratified; otherwise postpone. | +| 70 | Multi-instance TUI named inventory and read-only Admin-API fan-out | Fleet identity/TLS/token/inventory/operation limits are approved; otherwise postpone. | +| 80 | Release closeout | Only delivered, verified slices are included; deferred work is named explicitly. | + +### Mandatory lifecycle for every included feature slice + +1. Review the slice plan, assigned feature/functions, and Slice 3 draft + allocations. Approve, reject, or revise each draft; add any newly discovered + need, requirement, or acceptance criterion durably. +2. Write a slice design with architecture alignment, threat/privacy impact, + dependencies, rollback, and blast radius. Obtain and resolve design review. +3. Implement through TDD: commit/record failing **RED** tests, implement the + minimal **GREEN** behavior, then add refactoring only while tests remain + green. +4. Obtain and resolve code review. Run the slice's unit/integration/topology + verification and any approved funded/manual checks. +5. Write `dev/plans/0.5.15/-status.md` with accepted requirements, evidence, + review results, unresolved risk, and release disposition. + +## Existing detailed evidence + +- [0.5.15 release TODO](../0.5.15-todo.md) +- [Provider Admin/TUI design](20-provider-configuration-admin-tui-design.md) +- [Credential/alias warning design](30-configured-credential-missing-alias-startup-warning-design.md) +- [Host-console TUI design](30-host-console-tui-container-control-plane-design.md) +- [Virtual-key design](40-virtual-key-management-0.6-contract-design.md) +- [Fleet-control-plane research](50-multi-instance-tui-fleet-control-plane-research.md) +- [Gitleaks CI design](70-gitleaks-ci-secret-scan-design.md) and [review](71-gitleaks-ci-secret-scan-design-review.md) +- [Independent 0.5.15 design review](90-independent-design-review.md) +- [Even-minor roadmap migration plan](60-even-minor-release-roadmap-migration-plan.md) diff --git a/dev/requirements.md b/dev/requirements.md index 2964bb7..a14ea2d 100644 --- a/dev/requirements.md +++ b/dev/requirements.md @@ -349,6 +349,17 @@ that JSONL retention/deletion is a separate obligation. ## 0.5.15 ratified requirements +### DFR-34 / DAC-34: Typed Fast Guardian threat-backoff response + +Fast Guardian SHALL raise an Airlock-owned typed rate-limit exception for both +the request that creates a client threat backoff and a request rejected while +that backoff remains active. The proxy SHALL render the exception as an +OpenAI-shaped HTTP 429 with a whole-second, minimum-one `Retry-After`, stable +`type` and `code`, and `error.airlock.source: threat_backoff`. The response +SHALL not expose client identity, threat score, heuristic reason, request +content, provider identity, or provider-circuit-breaker state. Provider and +admission 429 contracts SHALL remain distinguishable and unchanged. + ### DFR-35 / DAC-35: Secret-scan delivery control Airlock's repository SHALL use a dedicated, non-deploying Gitleaks control at diff --git a/docs/guide/rate-limiting.md b/docs/guide/rate-limiting.md index 5995cef..b611b73 100644 --- a/docs/guide/rate-limiting.md +++ b/docs/guide/rate-limiting.md @@ -16,6 +16,13 @@ OpenAI-shaped body. The same handler also shapes *passthrough* provider 429s (wh the provider returned 429 and the breaker did not block), so clients get a backoff signal either way. +Fast Guardian's local threat protection is distinct: a threat backoff returns +HTTP 429 with the same whole-second `Retry-After` discipline, but an +`airlock_threat_backoff` / `threat_backoff` error and +`airlock.source: threat_backoff`. It does not mean the provider is rate-limiting +you, and it intentionally does not reveal the triggering heuristic or client +identity. Honor `Retry-After` for this response too. + ### Headers | Header | Value | Meaning | @@ -53,6 +60,7 @@ extra `airlock` sub-object: | `type` | `airlock.source` | When | |---|---|---| | `airlock_circuit_breaker` | `circuit_breaker` | Airlock's breaker is blocking the client. | +| `airlock_threat_backoff` | `threat_backoff` | Airlock's local threat protection is temporarily blocking the client. | | `provider_rate_limit` | `provider` | Passthrough: the provider returned 429; `Retry-After` comes from the provider's own headers when present, else a default. | ### What your client should do diff --git a/tests/harness/test_phase2_precall.py b/tests/harness/test_phase2_precall.py index aa9cdcc..fda2101 100644 --- a/tests/harness/test_phase2_precall.py +++ b/tests/harness/test_phase2_precall.py @@ -156,6 +156,7 @@ async def test_backoff_applied_after_burst( ): import time as _time from airlock.fast.guardian import AirlockFastGuardian + from airlock.proxy_errors import AirlockThreatBackoff guardian = AirlockFastGuardian() # Set client into backoff state directly (simulates previous threat block) @@ -167,7 +168,7 @@ async def test_backoff_applied_after_burst( "messages": [{"role": "user", "content": "ping"}], "model": "claude-sonnet", } - with pytest.raises(ValueError, match="[Tt]oo many|[Rr]etry"): + with pytest.raises(AirlockThreatBackoff): await guardian.async_pre_call_hook( mock_user_api_key_dict, mock_cache, data, "completion" ) diff --git a/tests/harness/test_phase4_e2e.py b/tests/harness/test_phase4_e2e.py index f3cc02f..fa3c92d 100644 --- a/tests/harness/test_phase4_e2e.py +++ b/tests/harness/test_phase4_e2e.py @@ -90,6 +90,7 @@ async def test_threat_burst_backoff( ): import time as _time from airlock.fast.guardian import AirlockFastGuardian + from airlock.proxy_errors import AirlockThreatBackoff guardian = AirlockFastGuardian() # Simulate previous threat block by setting backoff @@ -101,7 +102,7 @@ async def test_threat_burst_backoff( "messages": [{"role": "user", "content": "ping"}], "model": "claude-sonnet", } - with pytest.raises(ValueError, match="[Tt]oo many|[Rr]etry"): + with pytest.raises(AirlockThreatBackoff): await guardian.async_pre_call_hook( mock_user_api_key_dict, mock_cache, data, "completion" ) diff --git a/tests/test_fast_guardian.py b/tests/test_fast_guardian.py index 429b42a..5af9558 100644 --- a/tests/test_fast_guardian.py +++ b/tests/test_fast_guardian.py @@ -18,7 +18,7 @@ ) from airlock.fast.model_alias import AliasResolution from airlock.guardrails.extract import extract_text_from_messages as _extract_text -from airlock.proxy_errors import AirlockEndpointNotSupported +from airlock.proxy_errors import AirlockEndpointNotSupported, AirlockThreatBackoff # --------------------------------------------------------------------------- @@ -511,10 +511,11 @@ async def test_client_in_backoff_rejected( "messages": [{"role": "user", "content": "Hello"}], "model": "claude-sonnet", } - with pytest.raises(ValueError, match="Too many requests"): + with pytest.raises(AirlockThreatBackoff) as raised: await guardian.async_pre_call_hook( mock_user_api_key_dict, mock_cache, data, "completion" ) + assert 59.0 < raised.value.retry_after <= 60.0 async def test_high_threat_blocked( self, guardian, fresh_state_store, mock_cache, mock_user_api_key_dict @@ -532,10 +533,11 @@ async def test_high_threat_blocked( "messages": [{"role": "user", "content": "Hello"}], "model": "claude-sonnet", } - with pytest.raises(ValueError, match="unusual activity"): + with pytest.raises(AirlockThreatBackoff) as raised: await guardian.async_pre_call_hook( mock_user_api_key_dict, mock_cache, data, "completion" ) + assert raised.value.retry_after > 0 async def test_open_circuit_pinned_request_returns_429( self, guardian, fresh_state_store, mock_cache, mock_user_api_key_dict @@ -862,7 +864,7 @@ async def test_mcp_threat_still_applies( "mcp_tool_name": "search", "mcp_arguments": {"q": "test"}, } - with pytest.raises(ValueError, match="Too many requests"): + with pytest.raises(AirlockThreatBackoff): await guardian.async_pre_call_hook( mock_user_api_key_dict, mock_cache, data, "call_mcp_tool" ) @@ -928,7 +930,7 @@ async def test_batch_threat_still_applies( client.backoff_until = time.time() + 60 # force backoff data = {"input_file_id": "file-abc"} - with pytest.raises(ValueError, match="Too many requests"): + with pytest.raises(AirlockThreatBackoff): await guardian.async_pre_call_hook( mock_user_api_key_dict, mock_cache, data, "acreate_batch" ) diff --git a/tests/test_proxy_errors.py b/tests/test_proxy_errors.py index efb983a..c260e3c 100644 --- a/tests/test_proxy_errors.py +++ b/tests/test_proxy_errors.py @@ -13,13 +13,16 @@ AirlockGatewayRoutingOverride, AirlockInvalidReasoningEffort, AirlockProviderBlocked, + AirlockThreatBackoff, airlock_invalid_reasoning_effort_handler, airlock_provider_blocked_handler, + airlock_threat_backoff_handler, block_response_payload, invalid_reasoning_effort_response_payload, endpoint_not_supported_response_payload, install_airlock_error_handlers_on_proxy_app, retry_after_seconds, + threat_backoff_response_payload, ) @@ -218,6 +221,62 @@ def test_is_a_rate_limit_error(self): assert exc.client_id == "key:abc" +class TestAirlockThreatBackoff: + def test_is_a_rate_limit_error_without_client_or_heuristic_fields(self): + exc = AirlockThreatBackoff(retry_after=2.1) + assert isinstance(exc, RateLimitError) + assert exc.retry_after == 2.1 + assert not hasattr(exc, "client_id") + assert not hasattr(exc, "reason") + + async def test_guardrail_translation_preserves_429_contract( + self, mock_user_api_key_dict + ): + """LiteLLM can translate guardrail errors before FastAPI handlers run.""" + from litellm.proxy._types import ProxyException + from litellm.proxy.common_request_processing import ( + ProxyBaseLLMRequestProcessing, + ) + from litellm.proxy.proxy_server import openai_exception_handler + from starlette.requests import Request + + proxy_logging = type( + "ProxyLoggingStub", + (), + { + "post_call_failure_hook": AsyncMock(return_value=None), + "post_call_response_headers_hook": AsyncMock(return_value={}), + }, + )() + with pytest.raises(ProxyException) as raised: + await ProxyBaseLLMRequestProcessing(data={})._handle_llm_api_exception( + e=AirlockThreatBackoff(retry_after=2.1), + user_api_key_dict=mock_user_api_key_dict, + proxy_logging_obj=proxy_logging, + version="test", + ) + response = await openai_exception_handler( + Request( + { + "type": "http", + "method": "POST", + "path": "/v1/chat/completions", + "headers": [], + } + ), + raised.value, + ) + assert response.status_code == 429 + assert response.headers["Retry-After"] == "3" + assert json.loads(response.body)["error"] == { + "message": "Too many requests. Please retry later.", + "type": "airlock_threat_backoff", + "param": None, + "code": "threat_backoff", + "airlock": {"source": "threat_backoff", "retry_after": 3}, + } + + class TestRetryAfter: def test_ceils_and_floors_at_one(self): assert retry_after_seconds(0.0) == 1 @@ -262,6 +321,34 @@ def test_reason_is_bounded(self): assert len(body["error"]["airlock"]["reason"]) == 300 +class TestThreatBackoffResponsePayload: + def test_body_is_openai_shaped_and_does_not_disclose_protection_details(self): + body, headers = threat_backoff_response_payload( + AirlockThreatBackoff(retry_after=2.1) + ) + assert body == { + "error": { + "message": "Too many requests. Please retry later.", + "type": "airlock_threat_backoff", + "code": "threat_backoff", + "param": None, + "airlock": {"source": "threat_backoff", "retry_after": 3}, + } + } + assert headers == {"Retry-After": "3"} + assert "provider" not in str(body).lower() + assert "client" not in str(body).lower() + + @pytest.mark.parametrize("retry_after", [0.0, -1.0, 0.1, 2.0]) + def test_retry_after_is_whole_seconds_and_minimum_one(self, retry_after): + body, headers = threat_backoff_response_payload( + AirlockThreatBackoff(retry_after=retry_after) + ) + expected = retry_after_seconds(retry_after) + assert body["error"]["airlock"]["retry_after"] == expected + assert headers["Retry-After"] == str(expected) + + class TestHandler: async def test_handler_returns_429(self): exc = AirlockProviderBlocked( @@ -280,6 +367,14 @@ async def test_handler_returns_429(self): payload = json.loads(bytes(resp.body)) assert payload["error"]["type"] == "airlock_circuit_breaker" + async def test_threat_backoff_handler_returns_429(self): + response = await airlock_threat_backoff_handler( + None, AirlockThreatBackoff(retry_after=0.1) + ) + assert response.status_code == 429 + assert response.headers["Retry-After"] == "1" + assert json.loads(response.body)["error"]["code"] == "threat_backoff" + class TestInvalidReasoningEffort: def test_payload_is_openai_shaped_400(self): @@ -321,6 +416,7 @@ def test_idempotent_on_fastapi_app(self, monkeypatch): # second call is a no-op success assert install_airlock_error_handlers_on_proxy_app() is True assert AirlockProviderBlocked in app.exception_handlers + assert AirlockThreatBackoff in app.exception_handlers assert AirlockInvalidReasoningEffort in app.exception_handlers From 5ad501803c0fe75061724b39e4e7787a377fc665 Mon Sep 17 00:00:00 2001 From: Corey Thompson Date: Sat, 15 Aug 2026 20:46:38 -0500 Subject: [PATCH 08/24] docs(review): record admin configuration fix disposition --- .../FIX-1-FIX-2-admin-review-disposition.md | 84 +++++++++++++++++++ 1 file changed, 84 insertions(+) create mode 100644 dev/plans/0.5.15/FIX-1-FIX-2-admin-review-disposition.md diff --git a/dev/plans/0.5.15/FIX-1-FIX-2-admin-review-disposition.md b/dev/plans/0.5.15/FIX-1-FIX-2-admin-review-disposition.md new file mode 100644 index 0000000..fbbe8eb --- /dev/null +++ b/dev/plans/0.5.15/FIX-1-FIX-2-admin-review-disposition.md @@ -0,0 +1,84 @@ +# FIX-1 / FIX-2 — Admin-default and include-policy review disposition + +**Status:** reviewed; no implementation applies to the current 0.5.15 tree. + +## Finding confirmation + +The two P1 findings are valid for the reviewed patch description: + +* **FIX-1:** an active root `admin.enabled: true` changes the shipped default, + mounts the Admin control plane, and invokes CC-12 on external plaintext binds. +* **FIX-2:** a recursively resolving child policy loader disagrees with pinned + LiteLLM's one-level `ProxyConfig._process_includes` behavior and could derive + an enabled Admin policy where LiteLLM's direct include resolution leaves it + disabled. + +Neither change is present here. `config.yaml` retains its commented Admin sample +with `enabled: false`; parsing it yields no `admin` mapping and +`configure_admin()` defaults to disabled. `airlock/config_loader.py` is absent. +The existing recursive `_inline_config_includes()` is used only when the MCP-off +runtime config is materialized and is handed to LiteLLM; it must not be reused +as an independent Admin-policy resolver. + +## Solution hypothesis + +If the reviewed patch is introduced or reintroduced: + +1. **FIX-1:** remove the active root `admin` mapping and retain the commented, + opt-in sample. Do not replace it with any enabled default. +2. **FIX-2:** remove the independent recursive child-policy resolver. Either + leave current startup ownership unchanged, or use a single resolver whose + output is the exact mapping/file consumed by both LiteLLM and every Airlock + policy consumer. A limited resolver must mirror LiteLLM's direct include + overlay exactly; it must not recurse. + +Pseudo-code for the safe limited alternative: + +```text +effective = deep_copy(root) +for direct_path in effective.include: # list only, in listed order + included = parse_yaml(direct_path) + for key, value in included.items(): + if is_list(value) and key exists in effective: + effective[key].extend(value) + else: + effective[key] = value +delete effective.include +configure_all_startup_consumers(effective) +launch_litellm_with_file_serialized_from(effective) +``` + +This is only safe when both consumers receive `effective`, and when LiteLLM's +relative-path and error behavior are preserved. A recursive variant must first +materialize the recursive result and pass that same materialized result to +LiteLLM; configuring only Admin from it is rejected. + +## Blast radius and implications + +Restoring the commented default is a safe rollback for every default deployment: +it preserves Admin 404s and avoids unexpected CC-12 startup failures. Removing +the divergent loader prevents an authorization split-brain. Expanding include +resolution to other startup consumers is a broader configuration-semantics +change, potentially changing aliases, settings, and guardrails; it is out of +scope without a dedicated design and ownership decision. + +## Required verification if the reviewed patch exists + +1. **RED then GREEN:** a root-template test that parses `config.yaml`, proves + `admin` is absent (or disabled), configures policy, and observes 404 on an + Admin route. Verify an external host with that default does not trigger + CC-12. +2. **RED then GREEN:** fixture root → direct include (`admin.enabled: false`) → + nested include (`admin.enabled: true`). Assert Airlock policy equals + LiteLLM's effective direct-resolution result: disabled. Cover direct true, + direct include order, dict/scalar top-level replacement, list extension, and + a nested include being ignored by the one-level resolver. +3. Assert LiteLLM receives the same resolved mapping/file used for policy, if + resolution ownership changes. + +Current tests already exercise Admin disabled/configured policy and runtime +include materialization. No new RED test exists in this tree because the alleged +behavior is absent; adding a passing-only test would not satisfy the requested +TDD evidence. The existing `config.yaml` and Admin-guide off-by-default wording +remain correct, so no product-documentation edit is warranted. This disposition +is the durable documentation update for the review result. From 032079a8100a570cace8e113a5fc2667353701ac Mon Sep 17 00:00:00 2001 From: Corey Thompson Date: Sun, 16 Aug 2026 13:43:27 -0500 Subject: [PATCH 09/24] docs(plan): reserve slice 80 for CI review --- dev/plans/0.5.15/README.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/dev/plans/0.5.15/README.md b/dev/plans/0.5.15/README.md index 2d39a24..f54d61e 100644 --- a/dev/plans/0.5.15/README.md +++ b/dev/plans/0.5.15/README.md @@ -32,7 +32,8 @@ that is accepted for investigation from work that is approved for delivery. | Secure host-console TUI administration for containers | Design exists; topology/identity decision is open | Slice 50 | | Airlock-native virtual-key management | Design exists; depends on 0.6.0 identity/keystore contract | Slice 60, conditional | | Multi-instance TUI administration | Research/design exists; remote operation/configuration authority is open | Slice 70, conditional | -| Release closeout | Required only for work actually included and delivered | Slice 80 | +| CI review and improvement | Review workflow reliability, signal, coverage, cost, and maintainability without weakening delivery controls | Slice 80 | +| Release closeout | Required only for work actually included and delivered | Slice 90 | Claude Code subscription-preserving pass-through is explicitly out of scope for 0.5.15. The even-minor roadmap migration is planning/documentation work, not a @@ -217,7 +218,8 @@ authorize work before Slice 6. | 50 | Secure host-console TUI administration for containerized Airlock | Topology, operator identity, TLS/token lifecycle, and audit decisions pass. | | 60 | Virtual-key store and management | 0.6.0 keystore/identity contract and threat model are ratified; otherwise postpone. | | 70 | Multi-instance TUI named inventory and read-only Admin-API fan-out | Fleet identity/TLS/token/inventory/operation limits are approved; otherwise postpone. | -| 80 | Release closeout | Only delivered, verified slices are included; deferred work is named explicitly. | +| 80 | CI review and improvement | Review current workflows, required checks, runtime/cost, cache and matrix design, test-signal quality, and failure triage; preserve least privilege, pinned actions, Gitleaks enforcement, and no-secret CI boundaries. Document and independently review proposed changes before implementation. | +| 90 | Release closeout | Only delivered, verified slices are included; deferred work is named explicitly. | ### Mandatory lifecycle for every included feature slice From 70d69b1c0cb9c0828db9e38720773de8473cfc8a Mon Sep 17 00:00:00 2001 From: Corey Thompson Date: Sun, 16 Aug 2026 14:06:09 -0500 Subject: [PATCH 10/24] feat(startup): warn on unused provider credentials --- airlock/proxy.py | 16 +++ airlock/startup_validation.py | 116 ++++++++++++++++ .../30-provider-credential-warning-plan.md | 68 ++++++++++ .../30-provider-credential-warning-status.md | 50 +++++++ dev/plans/0.5.15/README.md | 2 +- dev/requirements.md | 16 +++ docs/getting-started/configuration.md | 13 ++ tests/test_proxy.py | 41 ++++++ tests/test_startup_validation.py | 124 ++++++++++++++++++ 9 files changed, 445 insertions(+), 1 deletion(-) create mode 100644 airlock/startup_validation.py create mode 100644 dev/plans/0.5.15/30-provider-credential-warning-plan.md create mode 100644 dev/plans/0.5.15/30-provider-credential-warning-status.md create mode 100644 tests/test_startup_validation.py diff --git a/airlock/proxy.py b/airlock/proxy.py index 7b6c4eb..96cf969 100644 --- a/airlock/proxy.py +++ b/airlock/proxy.py @@ -25,6 +25,10 @@ from airlock.capability import capability_record from airlock.models_catalog import fetch_live_provider_models +from airlock.startup_validation import ( + credential_without_alias_warnings, + emit_provider_credential_warnings, +) _ENV_REF_PREFIX = "os.environ/" @@ -438,6 +442,18 @@ def main() -> None: # Log live provider models at startup (informational — does not affect routing). with open(config_path) as f: config = yaml.safe_load(f) or {} + try: + emit_provider_credential_warnings( + credential_without_alias_warnings(config_path, os.getenv) + ) + except ValueError: + # Validation is advisory: never expose a path/parser detail or make an + # already-valid launch depend on a second local read. + print( + "WARNING: airlock.startup.provider_credential_validation_unavailable " + "source=startup_validation", + file=sys.stderr, + ) from airlock.admin.policy import configure_admin from airlock.fast.router import set_router_config from airlock.fast.settings import configure_settings diff --git a/airlock/startup_validation.py b/airlock/startup_validation.py new file mode 100644 index 0000000..2b9dbed --- /dev/null +++ b/airlock/startup_validation.py @@ -0,0 +1,116 @@ +"""Pure, redacted startup validation for configured provider credentials.""" + +from __future__ import annotations + +import copy +import sys +from dataclasses import dataclass +from pathlib import Path +from typing import Callable, TextIO + +import yaml + +from airlock.capability import airlock_provider_for + + +@dataclass(frozen=True) +class ProviderCredentialSpec: + provider: str + environment_variable: str + + +@dataclass(frozen=True) +class ProviderCredentialWarning: + provider: str + credential_configured: bool = True + configured_alias_count: int = 0 + source: str = "startup_validation" + + +PROVIDER_CREDENTIAL_SPECS = ( + ProviderCredentialSpec("anthropic", "ANTHROPIC_API_KEY"), + ProviderCredentialSpec("openai", "OPENAI_API_KEY"), + ProviderCredentialSpec("gemini", "GOOGLE_AISTUDIO_API_KEY"), + ProviderCredentialSpec("mistral", "MISTRAL_API_KEY"), + ProviderCredentialSpec("openrouter", "OPENROUTER_API_KEY"), + ProviderCredentialSpec("deepseek", "DEEPSEEK_API_KEY"), + ProviderCredentialSpec("perplexity", "PERPLEXITY_API_KEY"), + ProviderCredentialSpec("tavily", "TAVILY_API_KEY"), + ProviderCredentialSpec("vllm", "VLLM_API_KEY"), +) + + +def _load_mapping(path: Path) -> dict: + try: + loaded = yaml.safe_load(path.read_text(encoding="utf-8")) or {} + except (OSError, yaml.YAMLError) as exc: + raise ValueError(f"could not load startup validation config: {exc}") from exc + if not isinstance(loaded, dict): + raise ValueError("startup validation config must contain a mapping") + return loaded + + +def effective_model_list(config_path: str | Path) -> list[dict]: + """Return model entries using LiteLLM's direct, non-recursive includes.""" + path = Path(config_path).resolve() + config = copy.deepcopy(_load_mapping(path)) + includes = config.pop("include", None) + if includes is not None: + if not isinstance(includes, list) or not all( + isinstance(item, str) for item in includes + ): + raise ValueError("config include must be a list of paths") + for item in includes: + include_path = Path(item) + if not include_path.is_absolute(): + include_path = path.parent / include_path + included = _load_mapping(include_path.resolve()) + for key, value in included.items(): + if isinstance(value, list) and isinstance(config.get(key), list): + config[key].extend(value) + else: + config[key] = value + entries = config.get("model_list") or [] + return [entry for entry in entries if isinstance(entry, dict)] + + +def _enabled_provider_counts(entries: list[dict]) -> dict[str, int]: + counts: dict[str, int] = {} + for entry in entries: + params = entry.get("litellm_params") or {} + provider = airlock_provider_for(entry) + if provider: + counts[provider] = counts.get(provider, 0) + 1 + if ( + params.get("backend") == "vllm" + and params.get("api_key") == "os.environ/VLLM_API_KEY" + ): + counts["vllm"] = counts.get("vllm", 0) + 1 + return counts + + +def credential_without_alias_warnings( + config_path: str | Path, getenv: Callable[[str], str | None] +) -> tuple[ProviderCredentialWarning, ...]: + """Return one safe warning for each recognised present-but-unused credential.""" + enabled = _enabled_provider_counts(effective_model_list(config_path)) + return tuple( + ProviderCredentialWarning(spec.provider) + for spec in PROVIDER_CREDENTIAL_SPECS + if (value := getenv(spec.environment_variable)) is not None + and value.strip() + and enabled.get(spec.provider, 0) == 0 + ) + + +def emit_provider_credential_warnings( + warnings: tuple[ProviderCredentialWarning, ...], *, stream: TextIO = sys.stderr +) -> None: + """Emit the fixed-schema, secret-free startup warning to stderr.""" + for warning in warnings: + print( + "WARNING: airlock.startup.provider_credential_without_alias " + f"provider={warning.provider} credential_configured=true " + "configured_alias_count=0 source=startup_validation", + file=stream, + ) diff --git a/dev/plans/0.5.15/30-provider-credential-warning-plan.md b/dev/plans/0.5.15/30-provider-credential-warning-plan.md new file mode 100644 index 0000000..7ad88b9 --- /dev/null +++ b/dev/plans/0.5.15/30-provider-credential-warning-plan.md @@ -0,0 +1,68 @@ +# Slice 30 — configured credential without enabled alias warning + +**Status:** implemented locally (2026-08-16); verification and disposition are +recorded in `30-provider-credential-warning-status.md`. + +## Prior-slice closure and B3 resolution + +Slice 20 is implemented locally and its focused verification is green; its +environment-limited ordinary-suite rerun is recorded for release closeout and +does not affect this independent startup-validation slice. + +Slice 30 was conditional on B3: an authoritative enabled-model source and a +finite credential taxonomy. The current launcher reads the root YAML before +LiteLLM applies its direct `include:` entries, so using that raw `model_list` +would produce false warnings for included aliases. B3 is resolved narrowly: +this slice will derive *only* the effective `model_list` using the installed +LiteLLM one-level include semantics (listed direct includes, top-level list +extension, other top-level replacement, no recursion). It will not introduce a +generic configuration loader, alter the config passed to LiteLLM, or reuse a +recursive resolution path for an independent policy decision. + +The finite registry is limited to documented provider credentials: Anthropic +(`ANTHROPIC_API_KEY`), OpenAI (`OPENAI_API_KEY`), Gemini (`GOOGLE_AISTUDIO_API_KEY`), +Mistral (`MISTRAL_API_KEY`), OpenRouter (`OPENROUTER_API_KEY`), DeepSeek +(`DEEPSEEK_API_KEY`), Perplexity (`PERPLEXITY_API_KEY`), Tavily +(`TAVILY_API_KEY`), and local vLLM (`VLLM_API_KEY`). vLLM is OpenAI-compatible, +so its enabled status is the reviewed explicit `backend: vllm` / credential-ref +mapping rather than a generic OpenAI provider token. It is not a scan of +ambient `*_API_KEY` variables. Nonblank values count as configured; values are +never returned, logged, measured, or placed in exceptions. + +## Ratified contract and design + +DFR-36/DAC-36 is ratified: after `.env` loading and before optional discovery +or LiteLLM launch, Airlock emits exactly one redacted local warning per +recognised provider that has a nonblank recognised environment credential and +zero aliases in the effective model list classified by `airlock_provider_for`. +The stable event is +`airlock.startup.provider_credential_without_alias` with only provider, +`credential_configured=true`, `configured_alias_count=0`, and +`source=startup_validation`. It is default-on, advisory, local/no-network, and +does not change startup status, routing, discovery, config, or the inference +path. Provider/admission errors and Admin/TUI state are out of scope. + +`airlock/startup_validation.py` will contain pure registry, include/model-list +projection, evaluation, and redacted emission functions with injected +environment/path inputs. `proxy.main()` will invoke the adapter once after its +normal configuration load and before discovery. No warning state is retained +for a later Admin/TUI surface; that is Slice 40 work. + +## Threat model, blast radius, rollback + +The exposure boundary is startup output. The event schema is fixed and tests +will use sentinel values to prove neither values nor environment-variable names +appear. The resolver opens only explicit config paths already supplied to +LiteLLM; it makes no network/provider request and does not modify the runtime +configuration. A parser failure produces a bounded local validation warning and +does not invent an enabled alias. Rollback is removal of the new startup call; +it has no state, migration, dependency, or configuration cleanup. + +## TDD and verification + +1. RED tests first: registry/presence, one-level effective aliases including + direct include order/list extension/nested-ignore, redaction, deduplication, + no-network/no-mutation, and proxy startup order. +2. Minimal pure implementation and the one startup adapter call. +3. Run focused startup/proxy/capability tests, lint/format, `make verify`, and + strict docs build. Review the changed seams and record residual risk/status. diff --git a/dev/plans/0.5.15/30-provider-credential-warning-status.md b/dev/plans/0.5.15/30-provider-credential-warning-status.md new file mode 100644 index 0000000..dbd61e0 --- /dev/null +++ b/dev/plans/0.5.15/30-provider-credential-warning-status.md @@ -0,0 +1,50 @@ +# Slice 30 — configured credential without enabled alias warning status + +**Disposition:** implemented locally; no push required for this slice. + +## Accepted contract + +DFR-36/DAC-36 is ratified in `dev/requirements.md`; draft DUN-35 is addressed +by the existing operator configuration needs rather than a new canonical user +need. At startup, after `.env` loading and before optional provider discovery, +Airlock now emits one advisory, local warning for each recognised provider with +a nonblank recognised credential and no explicit effective alias. + +The fixed event is +`airlock.startup.provider_credential_without_alias`. Its output contains only +the canonical provider, `credential_configured=true`, +`configured_alias_count=0`, and `source=startup_validation`. It omits values, +environment-variable names, paths, request content, provider errors, and all +other free-form configuration. The warning changes neither exit status, +routing, discovery, runtime configuration, Admin/TUI state, nor the inference +path. + +## Design, B3 resolution, and review + +`airlock/startup_validation.py` owns a finite registry of documented provider +credentials and derives effective aliases with the installed LiteLLM's direct, +non-recursive include semantics: direct include order, top-level list extension, +and top-level replacement otherwise. It deliberately does not add a generic +configuration loader. `airlock_provider_for()` remains the standard canonical +provider classifier; local vLLM uses its explicit `backend: vllm` plus +`VLLM_API_KEY` reference because it is OpenAI-compatible on transport. + +Code review confirmed the startup call occurs before discovery, emits only the +fixed schema, catches a local validation read failure without leaking its +details, and has no provider/network or configuration mutation path. The +principal residual risk is warning noise from deliberately configured but +unused credentials; remediation is documented (add a reviewed alias or remove +the deployment secret) and warning behavior is advisory. + +## RED/GREEN and verification evidence + +* **RED:** `tests/test_startup_validation.py` initially failed collection because + `airlock.startup_validation` did not exist. +* **GREEN:** pure registry, direct include/nested-ignore, every recognised + provider, vLLM, redaction, and startup fallback/wiring tests pass. +* Focused startup/proxy/config/documentation suite: `94 passed, 1 xpassed`. +* Ruff lint/format, `make verify`, strict MkDocs build, and `git diff --check` + pass. + +No dependency, provider configuration, Admin/TUI, GitHub, or push action was +added. Rollback is removal of the pure module and one proxy startup invocation. diff --git a/dev/plans/0.5.15/README.md b/dev/plans/0.5.15/README.md index f54d61e..69f3c23 100644 --- a/dev/plans/0.5.15/README.md +++ b/dev/plans/0.5.15/README.md @@ -27,7 +27,7 @@ that is accepted for investigation from work that is approved for delivery. | --- | --- | --- | | Gitleaks local and CI secret scanning | Implemented; external enforcement and PR evidence recorded | Slice 10 | | Typed Fast Guardian threat-backoff HTTP 429 | Implemented locally; status/evidence recorded | Slice 20 | -| Configured-provider credential with no enabled alias warning | Design exists | Slice 30 | +| Configured-provider credential with no enabled alias warning | Implemented locally; redacted startup-warning evidence recorded | Slice 30 | | Read-only provider/configuration visibility in Admin/TUI | Design exists; CRUD is not approved | Slice 40 | | Secure host-console TUI administration for containers | Design exists; topology/identity decision is open | Slice 50 | | Airlock-native virtual-key management | Design exists; depends on 0.6.0 identity/keystore contract | Slice 60, conditional | diff --git a/dev/requirements.md b/dev/requirements.md index a14ea2d..982c320 100644 --- a/dev/requirements.md +++ b/dev/requirements.md @@ -349,6 +349,22 @@ that JSONL retention/deletion is a separate obligation. ## 0.5.15 ratified requirements +### DFR-36 / DAC-36: Unused configured-provider credential warning + +After normal environment loading and before optional provider discovery or +LiteLLM launch, Airlock SHALL emit one redacted, local, advisory startup warning +per recognised provider that has a nonblank recognised credential and zero +explicit aliases in LiteLLM's effective direct-include model list. The warning +SHALL use the stable event +`airlock.startup.provider_credential_without_alias` and expose only the +canonical provider, `credential_configured: true`, +`configured_alias_count: 0`, and `source: startup_validation`. It SHALL not +scan arbitrary environment variables, disclose values or variable names, make a +network/provider call, alter configuration/routing/discovery/startup status, or +create an Admin/TUI surface. Effective aliases SHALL use +`airlock_provider_for`; direct include semantics SHALL match installed LiteLLM +behavior without independently applying nested includes. + ### DFR-34 / DAC-34: Typed Fast Guardian threat-backoff response Fast Guardian SHALL raise an Airlock-owned typed rate-limit exception for both diff --git a/docs/getting-started/configuration.md b/docs/getting-started/configuration.md index 05624a5..6b734e3 100644 --- a/docs/getting-started/configuration.md +++ b/docs/getting-started/configuration.md @@ -353,6 +353,19 @@ batch_profile: See [Batch Processing](../guide/batch.md) for the end-to-end recipe. +## Unused provider credential warning + +At startup, Airlock emits one local warning when a recognised provider +credential is nonblank but no explicit effective `model_list` alias serves that +provider. The warning is advisory: it makes no provider request and does not +enable discovery, alter routing, or expose the credential or its environment +variable name. + +If the credential is intentional, add a reviewed alias for that provider. If it +is not, remove it from the deployment's secret source and restart Airlock. An +included configuration file's direct `model_list` aliases are considered; a +discovered provider model is not an alias. + ## Environment variables | Variable | Description | Default | diff --git a/tests/test_proxy.py b/tests/test_proxy.py index 000830d..83e592e 100644 --- a/tests/test_proxy.py +++ b/tests/test_proxy.py @@ -2,6 +2,7 @@ from __future__ import annotations +import os import sys from pathlib import Path from unittest.mock import MagicMock, patch @@ -70,6 +71,46 @@ def test_missing_config_exits(self, tmp_path, monkeypatch): # main() # --------------------------------------------------------------------------- class TestMain: + def test_main_emits_credential_warnings_before_optional_discovery( + self, config_file, monkeypatch + ): + monkeypatch.setenv("AIRLOCK_CONFIG", str(config_file)) + warnings = (object(),) + mock_result = MagicMock(returncode=0) + with ( + patch( + "airlock.proxy.credential_without_alias_warnings", + return_value=warnings, + ) as evaluate, + patch("airlock.proxy.emit_provider_credential_warnings") as emit, + patch("airlock.proxy.subprocess.run", return_value=mock_result), + patch("airlock.proxy.fetch_live_provider_models", return_value=[]), + pytest.raises(SystemExit), + ): + main() + evaluate.assert_called_once_with(str(config_file), os.getenv) + emit.assert_called_once_with(warnings) + + def test_main_hides_credential_validation_failure_detail( + self, config_file, monkeypatch, capsys + ): + monkeypatch.setenv("AIRLOCK_CONFIG", str(config_file)) + mock_result = MagicMock(returncode=0) + with ( + patch( + "airlock.proxy.credential_without_alias_warnings", + side_effect=ValueError("sk-sentinel-secret /tmp/private.yaml"), + ), + patch("airlock.proxy.subprocess.run", return_value=mock_result), + patch("airlock.proxy.fetch_live_provider_models", return_value=[]), + pytest.raises(SystemExit), + ): + main() + captured = capsys.readouterr().err + assert "provider_credential_validation_unavailable" in captured + assert "sk-sentinel-secret" not in captured + assert "/tmp/private.yaml" not in captured + def test_main_starts_litellm_on_public_port(self, config_file, monkeypatch): """subprocess.run should run LiteLLM directly on the public host:port.""" monkeypatch.setenv("AIRLOCK_CONFIG", str(config_file)) diff --git a/tests/test_startup_validation.py b/tests/test_startup_validation.py new file mode 100644 index 0000000..eda1c98 --- /dev/null +++ b/tests/test_startup_validation.py @@ -0,0 +1,124 @@ +"""Contract tests for unused configured-provider startup warnings.""" + +from __future__ import annotations + +from io import StringIO + +import pytest + +from airlock.startup_validation import ( + credential_without_alias_warnings, + emit_provider_credential_warnings, +) + + +def _write_config(path, contents: str) -> None: + path.write_text(contents, encoding="utf-8") + + +def test_present_recognised_credential_without_alias_warns_redacted(tmp_path): + config = tmp_path / "config.yaml" + _write_config(config, "model_list: []\n") + + warnings = credential_without_alias_warnings( + config, {"OPENROUTER_API_KEY": "sk-sentinel-secret"}.get + ) + + assert [warning.provider for warning in warnings] == ["openrouter"] + output = StringIO() + emit_provider_credential_warnings(warnings, stream=output) + text = output.getvalue() + assert "airlock.startup.provider_credential_without_alias" in text + assert "provider=openrouter" in text + assert "credential_configured=true" in text + assert "configured_alias_count=0" in text + assert "sk-sentinel-secret" not in text + assert "OPENROUTER_API_KEY" not in text + + +def test_enabled_provider_and_blank_or_unknown_credentials_do_not_warn(tmp_path): + config = tmp_path / "config.yaml" + _write_config( + config, + "model_list:\n" + " - model_name: reviewed\n" + " litellm_params: {model: openai/gpt-4o-mini}\n", + ) + + warnings = credential_without_alias_warnings( + config, + { + "OPENAI_API_KEY": "present", + "ANTHROPIC_API_KEY": " ", + "UNRECOGNISED_API_KEY": "present", + }.get, + ) + assert warnings == () + + +@pytest.mark.parametrize( + ("provider", "environment_variable", "model"), + [ + ("anthropic", "ANTHROPIC_API_KEY", "anthropic/claude"), + ("openai", "OPENAI_API_KEY", "openai/gpt"), + ("gemini", "GOOGLE_AISTUDIO_API_KEY", "gemini/flash"), + ("mistral", "MISTRAL_API_KEY", "mistral/large"), + ("openrouter", "OPENROUTER_API_KEY", "openrouter/openai/gpt"), + ("deepseek", "DEEPSEEK_API_KEY", "deepseek/chat"), + ("perplexity", "PERPLEXITY_API_KEY", "perplexity/sonar"), + ("tavily", "TAVILY_API_KEY", "tavily/search"), + ], +) +def test_each_recognised_provider_is_suppressed_by_a_matching_alias( + tmp_path, provider, environment_variable, model +): + config = tmp_path / "config.yaml" + _write_config( + config, + "model_list:\n" + f" - model_name: {provider}\n" + f" litellm_params: {{model: {model}}}\n", + ) + assert ( + credential_without_alias_warnings(config, {environment_variable: "yes"}.get) + == () + ) + + +def test_direct_include_extends_model_list_but_nested_include_is_ignored(tmp_path): + child = tmp_path / "child.yaml" + nested = tmp_path / "nested.yaml" + config = tmp_path / "config.yaml" + _write_config(nested, "model_list:\n - litellm_params: {model: openrouter/a}\n") + _write_config( + child, + "include: [nested.yaml]\n" + "model_list:\n - litellm_params: {model: openai/gpt-4o-mini}\n", + ) + _write_config(config, "include: [child.yaml]\nmodel_list: []\n") + + warnings = credential_without_alias_warnings( + config, {"OPENAI_API_KEY": "yes", "OPENROUTER_API_KEY": "yes"}.get + ) + assert [warning.provider for warning in warnings] == ["openrouter"] + + +def test_explicit_vllm_backend_counts_as_enabled(tmp_path): + config = tmp_path / "config.yaml" + _write_config( + config, + "model_list:\n" + " - model_name: local\n" + " litellm_params:\n" + " model: openai/local\n" + " api_key: os.environ/VLLM_API_KEY\n" + " backend: vllm\n", + ) + assert credential_without_alias_warnings(config, {"VLLM_API_KEY": "yes"}.get) == () + + +def test_invalid_include_fails_without_disclosing_config_contents(tmp_path): + config = tmp_path / "config.yaml" + _write_config(config, "include: not-a-list\nmodel_list: []\n") + with pytest.raises(ValueError, match="list of paths"): + credential_without_alias_warnings(config, {"OPENAI_API_KEY": "yes"}.get) From f6db0128e4339fc041a9966231e5ca3d0ec051bc Mon Sep 17 00:00:00 2001 From: Corey Thompson Date: Sun, 16 Aug 2026 14:58:23 -0500 Subject: [PATCH 11/24] feat(admin): add redacted provider config snapshot --- airlock/admin/http.py | 16 +- airlock/litellm_config.py | 49 ++++ airlock/models_seam.py | 8 +- airlock/provider_configuration.py | 194 ++++++++++++++ airlock/proxy.py | 251 +++++++----------- airlock/proxy_bootstrap.py | 20 ++ airlock/startup_validation.py | 40 +-- airlock/tui/admin_client.py | 5 + airlock/tui/app.py | 2 + airlock/tui/screens/config.py | 80 ++++++ ...provider-configuration-admin-tui-design.md | 214 +++++++++++++++ ...rovider-configuration-visibility-status.md | 64 +++++ .../FIX-1-FIX-2-admin-review-disposition.md | 13 +- dev/requirements.md | 32 ++- docs/getting-started/configuration.md | 9 +- docs/guide/admin-api.md | 17 ++ tests/test_admin_http.py | 36 +++ tests/test_litellm_config.py | 88 ++++++ tests/test_models_seam.py | 15 ++ tests/test_provider_configuration.py | 131 +++++++++ tests/test_proxy.py | 145 +++++++++- tests/test_proxy_bootstrap.py | 26 ++ tests/test_startup_validation.py | 4 +- tests/test_tui.py | 43 +++ tests/test_tui_thread_safety.py | 31 +++ 25 files changed, 1315 insertions(+), 218 deletions(-) create mode 100644 airlock/litellm_config.py create mode 100644 airlock/provider_configuration.py create mode 100644 dev/plans/0.5.15/20-provider-configuration-admin-tui-design.md create mode 100644 dev/plans/0.5.15/40-provider-configuration-visibility-status.md create mode 100644 tests/test_litellm_config.py create mode 100644 tests/test_provider_configuration.py diff --git a/airlock/admin/http.py b/airlock/admin/http.py index 1c897ec..f4f19b3 100644 --- a/airlock/admin/http.py +++ b/airlock/admin/http.py @@ -57,6 +57,13 @@ def _view_providers() -> dict: return {"providers": out} +def _view_provider_configuration() -> dict: + """Return the immutable child-startup projection, never source YAML.""" + from airlock.provider_configuration import provider_configuration_snapshot + + return provider_configuration_snapshot() + + def _view_clients() -> dict: out = {} admission_enabled = get_settings().admission.enabled @@ -190,6 +197,12 @@ def _match_route(method: str, path: str): if method == "GET" and seg == ["providers"]: return ("admin:read", False, lambda p, b, a: _view_providers()) + if method == "GET" and seg == ["config", "providers"]: + return ( + "admin:read_config", + False, + lambda p, b, a: _view_provider_configuration(), + ) if method == "GET" and seg == ["clients"]: return ("admin:read", False, lambda p, b, a: _view_clients()) if method == "GET" and seg == ["circuits"]: @@ -336,7 +349,8 @@ def handle_admin_request( return 409, dict(exc.record), {} except Exception: # noqa: BLE001 — the perimeter must never raise (CC-10) return 500, {"error": "internal error"}, {} - return 200, result, {} + headers = {"cache-control": "no-store"} if op_scope == "admin:read_config" else {} + return 200, result, headers # --- ASGI plumbing ---------------------------------------------------------- diff --git a/airlock/litellm_config.py b/airlock/litellm_config.py new file mode 100644 index 0000000..2c7ddc2 --- /dev/null +++ b/airlock/litellm_config.py @@ -0,0 +1,49 @@ +"""Configuration loading that reproduces the pinned LiteLLM include contract. + +Airlock uses this narrow resolver only to materialize the one canonical mapping +passed to LiteLLM and every Slice-40 consumer. In pinned LiteLLM, an included +``include`` list extends the active root include list, so descendants are +processed from the root directory after already-queued entries. +""" + +from __future__ import annotations + +import copy +from pathlib import Path + +import yaml + + +def _load_mapping(path: Path) -> dict: + try: + loaded = yaml.safe_load(path.read_text(encoding="utf-8")) or {} + except (OSError, yaml.YAMLError) as exc: + raise ValueError(f"could not load LiteLLM config {path}: {exc}") from exc + if not isinstance(loaded, dict): + raise ValueError(f"LiteLLM config {path} must contain a mapping") + return loaded + + +def resolve_litellm_direct_config(config_path: str | Path) -> dict: + """Load a config with the pinned LiteLLM include semantics.""" + path = Path(config_path).resolve() + config = copy.deepcopy(_load_mapping(path)) + if "include" in config: + if not isinstance(config["include"], list): + raise ValueError("LiteLLM config include must be a list of paths") + # Do not snapshot this list. LiteLLM extends it when a child carries an + # `include` list, which makes that child reachable after existing items. + for item in config["include"]: + include_path = Path(item) + if not include_path.is_absolute(): + include_path = path.parent / include_path + if not include_path.exists(): + raise FileNotFoundError(f"Included file not found: {include_path}") + included = _load_mapping(include_path.resolve()) + for key, value in included.items(): + if isinstance(value, list) and key in config: + config[key].extend(copy.deepcopy(value)) + else: + config[key] = copy.deepcopy(value) + del config["include"] + return config diff --git a/airlock/models_seam.py b/airlock/models_seam.py index d94adcf..3f85c7c 100644 --- a/airlock/models_seam.py +++ b/airlock/models_seam.py @@ -20,9 +20,8 @@ from pathlib import Path from typing import Any -import yaml - from airlock.capability import capability_record +from airlock.litellm_config import resolve_litellm_direct_config from airlock.litellm_adapter import install_asgi_middleware, resolve_proxy_app try: # FastAPI is always present at runtime; tolerate its absence defensively. @@ -53,9 +52,8 @@ def _build_capability_map() -> dict[str, dict]: return {} try: - with open(path) as f: - cfg = yaml.safe_load(f) - except (OSError, yaml.YAMLError) as exc: + cfg = resolve_litellm_direct_config(path) + except ValueError as exc: logger.warning("Failed to load config for models seam: %s", exc) return {} diff --git a/airlock/provider_configuration.py b/airlock/provider_configuration.py new file mode 100644 index 0000000..576df7e --- /dev/null +++ b/airlock/provider_configuration.py @@ -0,0 +1,194 @@ +"""Immutable, secret-safe provider configuration snapshot for the Admin API.""" + +from __future__ import annotations + +import hashlib +import json +import os +from collections.abc import Callable +from datetime import UTC, datetime +from urllib.parse import urlsplit + +from airlock.capability import capability_record + +_MAX_PROVIDERS = 64 +_MAX_ALIASES = 200 +_MAX_TEXT = 256 +_SCHEMA_VERSION = 1 +_snapshot_json = json.dumps( + { + "schema_version": _SCHEMA_VERSION, + "source": "startup_config", + "loaded_at": None, + "restart_required": True, + "fingerprint": hashlib.sha256(b"[]").hexdigest(), + "truncated": {"providers": False, "aliases": False}, + "providers": [], + }, + sort_keys=True, + separators=(",", ":"), +) + + +def _safe_text(value: object) -> str | None: + if not isinstance(value, str): + return None + return value[:_MAX_TEXT] + + +def _base_host(value: object) -> str | None: + if not isinstance(value, str): + return None + try: + parsed = urlsplit(value) + except ValueError: + return None + if parsed.scheme not in {"http", "https"} or not parsed.hostname: + return None + return parsed.hostname[:_MAX_TEXT] + + +def _credential(value: object, getenv: Callable[[str], str | None]) -> dict: + if value is None or value == "": + return {"kind": "none", "configured": False} + if isinstance(value, str) and value.startswith("os.environ/"): + name = value.removeprefix("os.environ/") + present = bool(name and (getenv(name) or "").strip()) + return {"kind": "env_ref", "configured": present} + # A non-environment reference is deliberately opaque. Do not make an + # unrecognised environment-looking value observable by echoing its name. + if isinstance(value, str) and value.startswith(("secret://", "credential://")): + return {"kind": "credential_ref", "configured": True} + return {"kind": "redacted_literal", "configured": True} + + +def _alias_record(entry: dict, getenv: Callable[[str], str | None]) -> dict | None: + alias = _safe_text(entry.get("model_name")) + if not alias: + return None + params = entry.get("litellm_params") or {} + if not isinstance(params, dict): + params = {} + capability = capability_record(entry) + provider = _safe_text(capability["airlock_provider"]) + if not provider: + return None + endpoints = [ + endpoint[:_MAX_TEXT] + for endpoint in capability["endpoints"] + if isinstance(endpoint, str) + ] + return { + "provider": provider, + "alias": alias, + "underlying": _safe_text(capability["underlying"]), + "endpoints": endpoints, + "region": _safe_text(capability["region"]), + "deprecated": bool(capability["deprecated"]), + "api_base_host": _base_host(params.get("api_base") or params.get("base_url")), + "credential": _credential_for_params(params, getenv), + "setting_source": "static", + } + + +def _credential_for_params(params: dict, getenv: Callable[[str], str | None]) -> dict: + """Classify the finite supported credential fields without naming them.""" + for field in ("api_key", "vertex_credentials"): + value = params.get(field) + if value not in (None, ""): + return _credential(value, getenv) + return _credential(None, getenv) + + +def _fingerprint(providers: list[dict], truncated: dict[str, bool]) -> str: + # Timestamp and credential state are excluded; this is a stable identity of + # the safe routing DTO, not a hash oracle over raw YAML, literal secrets, + # reference names, or credential presence. + fingerprint_providers = [ + { + "provider": provider["provider"], + "aliases": [ + {key: value for key, value in alias.items() if key != "credential"} + for alias in provider["aliases"] + ], + } + for provider in providers + ] + canonical = json.dumps( + { + "schema_version": _SCHEMA_VERSION, + "providers": fingerprint_providers, + "truncated": truncated, + }, + sort_keys=True, + separators=(",", ":"), + ).encode("utf-8") + return hashlib.sha256(canonical).hexdigest() + + +def build_provider_configuration_snapshot( + config: dict | None, + *, + getenv: Callable[[str], str | None] = os.getenv, + loaded_at: str | None = None, +) -> dict: + """Return a bounded, redacted projection with no raw config references.""" + entries = (config or {}).get("model_list") or [] + records = [ + record + for entry in entries + if isinstance(entry, dict) + if (record := _alias_record(entry, getenv)) is not None + ] + records.sort( + key=lambda item: (item["provider"], item["alias"], item["underlying"] or "") + ) + provider_names = sorted({record["provider"] for record in records}) + selected_providers = set(provider_names[:_MAX_PROVIDERS]) + scoped = [record for record in records if record["provider"] in selected_providers] + selected_records = scoped[:_MAX_ALIASES] + truncated = { + "providers": len(provider_names) > _MAX_PROVIDERS, + "aliases": len(scoped) > _MAX_ALIASES, + } + providers: list[dict] = [] + for provider in provider_names[:_MAX_PROVIDERS]: + aliases = [ + {key: value for key, value in record.items() if key != "provider"} + for record in selected_records + if record["provider"] == provider + ] + if aliases: + providers.append({"provider": provider, "aliases": aliases}) + return { + "schema_version": _SCHEMA_VERSION, + "source": "startup_config", + "loaded_at": loaded_at + or datetime.now(UTC).replace(microsecond=0).isoformat().replace("+00:00", "Z"), + "restart_required": True, + "fingerprint": _fingerprint(providers, truncated), + "truncated": truncated, + "providers": providers, + } + + +def configure_provider_configuration( + config: dict | None, + *, + getenv: Callable[[str], str | None] = os.getenv, + loaded_at: str | None = None, +) -> None: + """Publish the child-startup snapshot as immutable canonical JSON.""" + global _snapshot_json + _snapshot_json = json.dumps( + build_provider_configuration_snapshot( + config, getenv=getenv, loaded_at=loaded_at + ), + sort_keys=True, + separators=(",", ":"), + ) + + +def provider_configuration_snapshot() -> dict: + """Return a fresh decoded copy; callers cannot mutate the stored snapshot.""" + return json.loads(_snapshot_json) diff --git a/airlock/proxy.py b/airlock/proxy.py index 96cf969..93ef30b 100644 --- a/airlock/proxy.py +++ b/airlock/proxy.py @@ -24,6 +24,7 @@ from dotenv import load_dotenv from airlock.capability import capability_record +from airlock.litellm_config import resolve_litellm_direct_config from airlock.models_catalog import fetch_live_provider_models from airlock.startup_validation import ( credential_without_alias_warnings, @@ -61,8 +62,7 @@ def _validate_mcp_env_refs(config_path: str) -> list[str]: if _mcp_startup_mode() == "off": return [] - with open(config_path) as f: - cfg = yaml.safe_load(f) or {} + cfg = resolve_litellm_direct_config(config_path) errors: list[str] = [] mcp_servers = cfg.get("mcp_servers") or {} @@ -166,79 +166,26 @@ def _fathom_logger_enabled() -> bool: return _env_flag("AIRLOCK_ENABLE_FATHOM_LOGGER", default=False) -def _inline_config_includes(config: dict, config_path: str) -> None: - """Inline local YAML includes when a runtime mode must remove a top-level key. - - LiteLLM applies included files after the main file. Simply popping - ``mcp_servers`` from the main runtime config would therefore let a local - include add it back. The documented include semantics are a top-level - replacement, so process includes in order and update the parent mapping. - """ - - def merge_includes(target: dict, includes: object, parent: Path) -> None: - if isinstance(includes, str): - include_items = [includes] - elif isinstance(includes, list): - include_items = includes - elif includes is None: - return - else: - raise ValueError("config include must be a path or a list of paths") - - for item in include_items: - if not isinstance(item, str): - raise ValueError("config include entries must be paths") - path = Path(item) - if not path.is_absolute(): - path = parent / path - try: - with open(path, encoding="utf-8") as f: - included = yaml.safe_load(f) or {} - except (OSError, yaml.YAMLError) as exc: - raise ValueError( - f"could not load config include {path}: {exc}" - ) from exc - if not isinstance(included, dict): - raise ValueError(f"config include {path} must contain a mapping") - nested = included.pop("include", None) - merge_includes(included, nested, path.parent) - target.update(included) - - merge_includes( - config, config.pop("include", None), Path(config_path).resolve().parent - ) - - def _prepare_runtime_config(config_path: str) -> tuple[str, str | None]: - """Apply env-driven startup overrides and return a config path for LiteLLM.""" - with open(config_path, encoding="utf-8") as f: - config = yaml.safe_load(f) or {} + """Materialize the one canonical config file consumed by the LiteLLM child.""" + config = resolve_litellm_direct_config(config_path) mcp_mode = _mcp_startup_mode() - # In off mode, materialize config.local.yaml first. Otherwise LiteLLM will - # apply that include after this function and can reintroduce mcp_servers. - if mcp_mode == "off": - _inline_config_includes(config, config_path) - - changed = False general_settings = config.setdefault("general_settings", {}) for entry in config.get("model_list", []) or []: if not isinstance(entry, dict): continue entry.setdefault("model_info", {}).update(capability_record(entry)) - changed = True master_key_ref = general_settings.get("master_key") if isinstance(master_key_ref, str) and master_key_ref.startswith(_ENV_REF_PREFIX): master_key_env = master_key_ref[len(_ENV_REF_PREFIX) :] if not os.getenv(master_key_env): general_settings.pop("master_key", None) - changed = True if mcp_mode == "off" and config.get("mcp_servers"): config.pop("mcp_servers", None) - changed = True print( "Configured MCP servers disabled for startup (AIRLOCK_MCP_STARTUP_MODE=off)." ) @@ -251,7 +198,6 @@ def _prepare_runtime_config(config_path: str) -> tuple[str, str | None]: if background_override is not None: if general_settings.get("background_health_checks") != background_override: general_settings["background_health_checks"] = background_override - changed = True mode = "enabled" if background_override else "disabled" print(f"Background health checks {mode} by AIRLOCK_BACKGROUND_HEALTH_CHECKS.") @@ -260,25 +206,9 @@ def _prepare_runtime_config(config_path: str) -> tuple[str, str | None]: # the same AIRLOCK_ENABLE_FATHOM_LOGGER flag — no config-callback append needed. print("Fathom logger enabled for startup (AIRLOCK_ENABLE_FATHOM_LOGGER=1).") - if not changed: - return config_path, None - - # The proxy may run as a non-root user while its packaged config lives in a - # read-only application directory. Write the generated config to the system - # temp directory instead. LiteLLM resolves `include:` paths relative to the - # generated file, so make relative includes absolute before relocating it. - config_dir = Path(config_path).resolve().parent - includes = config.get("include") - if isinstance(includes, list): - config["include"] = [ - str((config_dir / item).resolve()) - if isinstance(item, str) and not Path(item).is_absolute() - else item - for item in includes - ] - elif isinstance(includes, str) and not Path(includes).is_absolute(): - config["include"] = str((config_dir / includes).resolve()) - + # Always write a private runtime file: it is the single configuration + # authority for LiteLLM, child Admin policy, model metadata, and Slice 40's + # immutable projection. It also works for read-only packaged configs. tmp = tempfile.NamedTemporaryFile( mode="w", encoding="utf-8", @@ -286,17 +216,20 @@ def _prepare_runtime_config(config_path: str) -> tuple[str, str | None]: prefix="airlock-runtime-", delete=False, ) - with tmp: - yaml.safe_dump(config, tmp, sort_keys=False) + try: + with tmp: + yaml.safe_dump(config, tmp, sort_keys=False) + except Exception: + Path(tmp.name).unlink(missing_ok=True) + raise return tmp.name, tmp.name def _validate_config(config_path: str) -> list[str]: """Validate config.yaml schema and return a list of warning strings.""" try: - with open(config_path) as f: - cfg = yaml.safe_load(f) or {} - except yaml.YAMLError as e: + cfg = resolve_litellm_direct_config(config_path) + except (ValueError, yaml.YAMLError) as e: return [f"config.yaml is not valid YAML: {e}"] warnings: list[str] = [] @@ -396,6 +329,69 @@ def _warn_observe_mode() -> None: ) +def _launch_litellm_child( + config: dict, + runtime_config_path: str, + host: str, + port: int, + project_root: Path, +) -> int: + """Configure all parent seams then run the child from the canonical file.""" + from airlock.admin.policy import configure_admin + from airlock.fast.router import set_router_config + from airlock.fast.settings import configure_settings + from airlock.fast.state import configure_breaker + from airlock.guardrails.overrides import configure_guardrail_overrides + from airlock.transparency import configure_transparency + + set_router_config(config) + configure_settings(config) + from airlock.fast.admission import configure_admission + + configure_admission(config) + configure_breaker(config) + configure_admin(config, host=host, tls_enabled=bool(_ssl_cli_args())) + configure_guardrail_overrides(config) + configure_transparency(config) + if _startup_model_discovery_enabled(): + live_models = fetch_live_provider_models(config) + if live_models: + providers = sorted({m["id"].split("/")[0] for m in live_models}) + print( + f"Provider models discovered: {len(live_models)} across {', '.join(providers)}" + ) + else: + print( + "Startup model discovery disabled " + "(set AIRLOCK_STARTUP_MODEL_DISCOVERY=1 to enable)." + ) + + litellm_bin = str(Path(sys.executable).parent / "litellm") + litellm_cmd = [ + litellm_bin, + "--config", + runtime_config_path, + "--host", + host, + "--port", + str(port), + ] + litellm_cmd += _ssl_cli_args() + env = os.environ.copy() + existing_pythonpath = env.get("PYTHONPATH", "") + repo_pythonpath = str(project_root) + env["PYTHONPATH"] = ( + f"{repo_pythonpath}:{existing_pythonpath}" + if existing_pythonpath + else repo_pythonpath + ) + env["AIRLOCK_LITELLM_CHILD"] = "1" + env["AIRLOCK_CONFIG"] = runtime_config_path + _register_shutdown_handlers() + print(f"Airlock starting on {host}:{port}") + return subprocess.run(litellm_cmd, check=False, env=env).returncode + + def main() -> None: """Launch Airlock proxy entrypoint. @@ -440,8 +436,6 @@ def main() -> None: port = int(os.getenv("AIRLOCK_PORT", "4000")) # Log live provider models at startup (informational — does not affect routing). - with open(config_path) as f: - config = yaml.safe_load(f) or {} try: emit_provider_credential_warnings( credential_without_alias_warnings(config_path, os.getenv) @@ -454,87 +448,20 @@ def main() -> None: "source=startup_validation", file=sys.stderr, ) - from airlock.admin.policy import configure_admin - from airlock.fast.router import set_router_config - from airlock.fast.settings import configure_settings - from airlock.fast.state import configure_breaker - from airlock.guardrails.overrides import configure_guardrail_overrides - from airlock.transparency import configure_transparency - - set_router_config(config) - # Build the typed AirlockSettings snapshot once at startup. The router (budget-aware - # swap), monitor (near-limit warn) and circuit-breaker (failover) consumers read - # their budgets/failover map from this snapshot via get_settings() (0.5.1-SET-unify). - configure_settings(config) - from airlock.fast.admission import configure_admission - - configure_admission(config) - # Load per-client circuit-breaker policy once at startup (CC-2); defaults to a - # no-op when unconfigured (CC-3). Provider budget caps now flow through - # configure_settings above (the monitor reads get_settings().provider_budgets). - configure_breaker(config) - # Admin control plane (off by default). The fail-closed check (CC-12) refuses - # bearer-token admin over plaintext on a non-loopback bind. - configure_admin(config, host=host, tls_enabled=bool(_ssl_cli_args())) - # Per-request guardrail-skip policy (off by default). - configure_guardrail_overrides(config) - # Response-transparency policy (served-by/region headers; default-on, opt-out). - configure_transparency(config) - if _startup_model_discovery_enabled(): - live_models = fetch_live_provider_models(config) - if live_models: - providers = sorted({m["id"].split("/")[0] for m in live_models}) - print( - f"Provider models discovered: {len(live_models)} across {', '.join(providers)}" - ) - else: - print( - "Startup model discovery disabled " - "(set AIRLOCK_STARTUP_MODEL_DISCOVERY=1 to enable)." - ) - + # Every parent consumer below and the LiteLLM child use this one materialized + # file. Keep it before Admin's CC-12 check so an included admin block cannot + # produce a launcher/child split. runtime_config_path, temp_config_path = _prepare_runtime_config(config_path) - - litellm_bin = str(Path(sys.executable).parent / "litellm") - litellm_cmd = [ - litellm_bin, - "--config", - runtime_config_path, - "--host", - host, - "--port", - str(port), - ] - # Native HTTPS (opt-in): append uvicorn ssl flags when AIRLOCK_SSL_* are set. - litellm_cmd += _ssl_cli_args() - env = os.environ.copy() - existing_pythonpath = env.get("PYTHONPATH", "") - repo_pythonpath = str(project_root) - env["PYTHONPATH"] = ( - f"{repo_pythonpath}:{existing_pythonpath}" - if existing_pythonpath - else repo_pythonpath - ) - # FIX-1: mark the litellm subprocess so airlock.fast.monitor restores spend + - # breaker state on startup and checkpoints them periodically/on shutdown there — - # in the process that actually mutates the store. - env["AIRLOCK_LITELLM_CHILD"] = "1" - - _register_shutdown_handlers() - - # FIX-1: state restore now happens in the litellm child (airlock.fast.monitor), - # not here — the launcher's store is empty, so a launcher-side restore was a no-op - # that defeated restart recovery. - - print(f"Airlock starting on {host}:{port}") try: - sys.exit(subprocess.run(litellm_cmd, check=False, env=env).returncode) + config = resolve_litellm_direct_config(runtime_config_path) + sys.exit( + _launch_litellm_child(config, runtime_config_path, host, port, project_root) + ) finally: - if temp_config_path: - try: - Path(temp_config_path).unlink(missing_ok=True) - except OSError: - pass + try: + Path(temp_config_path).unlink(missing_ok=True) + except OSError: + pass if __name__ == "__main__": diff --git a/airlock/proxy_bootstrap.py b/airlock/proxy_bootstrap.py index 999081d..56e6360 100644 --- a/airlock/proxy_bootstrap.py +++ b/airlock/proxy_bootstrap.py @@ -13,7 +13,10 @@ from __future__ import annotations +import os + from airlock.admin.http import install_admin_on_proxy_app +from airlock.admin.policy import configure_admin from airlock.batch.middleware import install_batch_gateway_on_proxy_app from airlock.docs import install_airlock_docs_on_proxy_app from airlock.embedding_boundary import install_embedding_request_boundary_on_proxy_app @@ -22,8 +25,25 @@ from airlock.proxy_errors import install_airlock_error_handlers_on_proxy_app +def _configure_child_startup_configuration() -> None: + """Load the launcher's canonical runtime file before mounting Admin. + + The launcher is responsible for CC-12 bind/TLS validation. This child + process receives its already-materialized config path and owns the live + Admin policy/snapshot installed on the LiteLLM application. + """ + from airlock.litellm_config import resolve_litellm_direct_config + from airlock.provider_configuration import configure_provider_configuration + + config_path = os.getenv("AIRLOCK_CONFIG", "config.yaml") + config = resolve_litellm_direct_config(config_path) + configure_admin(config) + configure_provider_configuration(config) + + def bootstrap_airlock_proxy() -> None: """Install all Airlock proxy seams onto the live LiteLLM app, in order.""" + _configure_child_startup_configuration() install_airlock_docs_on_proxy_app() install_circuit_health_on_proxy_app() install_airlock_error_handlers_on_proxy_app() diff --git a/airlock/startup_validation.py b/airlock/startup_validation.py index 2b9dbed..67ef619 100644 --- a/airlock/startup_validation.py +++ b/airlock/startup_validation.py @@ -2,15 +2,12 @@ from __future__ import annotations -import copy import sys from dataclasses import dataclass -from pathlib import Path from typing import Callable, TextIO -import yaml - from airlock.capability import airlock_provider_for +from airlock.litellm_config import resolve_litellm_direct_config @dataclass(frozen=True) @@ -40,36 +37,9 @@ class ProviderCredentialWarning: ) -def _load_mapping(path: Path) -> dict: - try: - loaded = yaml.safe_load(path.read_text(encoding="utf-8")) or {} - except (OSError, yaml.YAMLError) as exc: - raise ValueError(f"could not load startup validation config: {exc}") from exc - if not isinstance(loaded, dict): - raise ValueError("startup validation config must contain a mapping") - return loaded - - -def effective_model_list(config_path: str | Path) -> list[dict]: - """Return model entries using LiteLLM's direct, non-recursive includes.""" - path = Path(config_path).resolve() - config = copy.deepcopy(_load_mapping(path)) - includes = config.pop("include", None) - if includes is not None: - if not isinstance(includes, list) or not all( - isinstance(item, str) for item in includes - ): - raise ValueError("config include must be a list of paths") - for item in includes: - include_path = Path(item) - if not include_path.is_absolute(): - include_path = path.parent / include_path - included = _load_mapping(include_path.resolve()) - for key, value in included.items(): - if isinstance(value, list) and isinstance(config.get(key), list): - config[key].extend(value) - else: - config[key] = value +def effective_model_list(config_path: str) -> list[dict]: + """Return model entries using the pinned LiteLLM include-list expansion.""" + config = resolve_litellm_direct_config(config_path) entries = config.get("model_list") or [] return [entry for entry in entries if isinstance(entry, dict)] @@ -90,7 +60,7 @@ def _enabled_provider_counts(entries: list[dict]) -> dict[str, int]: def credential_without_alias_warnings( - config_path: str | Path, getenv: Callable[[str], str | None] + config_path: str, getenv: Callable[[str], str | None] ) -> tuple[ProviderCredentialWarning, ...]: """Return one safe warning for each recognised present-but-unused credential.""" enabled = _enabled_provider_counts(effective_model_list(config_path)) diff --git a/airlock/tui/admin_client.py b/airlock/tui/admin_client.py index b77eebb..4479a8d 100644 --- a/airlock/tui/admin_client.py +++ b/airlock/tui/admin_client.py @@ -100,6 +100,11 @@ def provider_snapshot(host: str, port: str) -> dict | None: ) +def provider_configuration_snapshot(host: str, port: str) -> tuple[int, dict]: + """Fetch the child-startup configuration view; never read local files.""" + return admin_get(host, port, "/airlock/admin/config/providers") + + def session_snapshot(host: str, port: str) -> dict | None: """Return the bounded live affinity view, or None when admin is unavailable.""" status, payload = admin_get(host, port, "/airlock/admin/sessions") diff --git a/airlock/tui/app.py b/airlock/tui/app.py index 63211d8..670d026 100644 --- a/airlock/tui/app.py +++ b/airlock/tui/app.py @@ -95,6 +95,8 @@ def compose(self) -> ComposeResult: yield LogsPane(id="logs") yield ConfigPane( mcp_manager=self._mcp_manager, + host=self._proxy_host, + port=self._proxy_port, id="config", ) yield TestPane(id="test") diff --git a/airlock/tui/screens/config.py b/airlock/tui/screens/config.py index 432e02b..71a8127 100644 --- a/airlock/tui/screens/config.py +++ b/airlock/tui/screens/config.py @@ -154,10 +154,14 @@ def __init__( self, *, mcp_manager: McpServerManager | None = None, + host: str = "localhost", + port: str = "4000", id: str | None = None, ) -> None: super().__init__(id=id) self._mcp_manager: McpServerManager | None = mcp_manager + self._host = host + self._port = port self._selected_server: str = "" self._applied_values: dict[str, str] = {} @@ -189,6 +193,25 @@ def compose(self) -> ComposeResult: id="cfg-master-key", ) + # This is deliberately separate from the local environment editor + # above. It reads only the proxy child snapshot over Admin HTTP. + with TabPane("Configured", id="cfg-tab-configured-providers"): + yield Static( + "Loading configured provider state from the proxy…", + id="cfg-provider-config-status", + ) + configured = _SafeDataTable( + id="cfg-provider-config-table", cursor_type="row" + ) + configured.add_columns( + "Provider", "Alias", "Underlying", "Endpoints", "Credential" + ) + yield configured + yield Static( + "Read-only child startup configuration. Changes require deployment workflow and proxy restart.", + id="cfg-provider-config-detail", + ) + # Tab 2 — Guardrails with TabPane("Guardrails", id="cfg-tab-guardrails"): with VerticalScroll(classes="config-form"): @@ -389,6 +412,63 @@ def on_mount(self) -> None: return self._refresh_mcp_servers() self.set_interval(10.0, self._refresh_mcp_servers) + self._refresh_provider_configuration() + self.set_interval(30.0, self._refresh_provider_configuration) + + @work(thread=True) + def _refresh_provider_configuration(self) -> None: + """Refresh only through the Admin client; failure is an explicit state.""" + from airlock.tui.admin_client import provider_configuration_snapshot + + status, payload = provider_configuration_snapshot(self._host, self._port) + if status == 200 and isinstance(payload.get("providers"), list): + self.app.call_from_thread( + self._render_provider_configuration, payload, None + ) + return + if status == 403: + message = "Configured provider state unavailable: Admin token lacks admin:read_config." + elif status == 404: + message = "Configured provider state unavailable: Admin API is disabled." + else: + message = "Configured provider state unavailable: proxy did not return its startup snapshot." + self.app.call_from_thread(self._render_provider_configuration, None, message) + + def _render_provider_configuration( + self, payload: dict | None, error: str | None + ) -> None: + """Update read-only widgets on the UI thread from a bounded DTO.""" + status = self.query_one("#cfg-provider-config-status", Static) + table = self.query_one("#cfg-provider-config-table", _SafeDataTable) + detail = self.query_one("#cfg-provider-config-detail", Static) + table.clear() + if error: + status.update(f"[yellow]{error}[/]") + detail.update("No local configuration fallback is used.") + return + assert payload is not None + for provider in payload["providers"]: + for alias in provider.get("aliases", []): + credential = alias.get("credential", {}) + credential_text = ( + f"{credential.get('kind', 'none')}" + f" ({'configured' if credential.get('configured') else 'not configured'})" + ) + table.add_row( + str(provider.get("provider", "-")), + str(alias.get("alias", "-")), + str(alias.get("underlying") or "-"), + ", ".join(alias.get("endpoints") or []) or "-", + credential_text, + ) + status.update( + f"Source: {payload.get('source', 'startup_config')} · loaded {payload.get('loaded_at', 'unknown')} · restart required" + ) + truncation = payload.get("truncated", {}) + detail.update( + "Read-only child startup configuration. Changes require deployment workflow and proxy restart." + + (" Results were truncated." if any(truncation.values()) else "") + ) # ------------------------------------------------------------------ # Button handler diff --git a/dev/plans/0.5.15/20-provider-configuration-admin-tui-design.md b/dev/plans/0.5.15/20-provider-configuration-admin-tui-design.md new file mode 100644 index 0000000..2498f16 --- /dev/null +++ b/dev/plans/0.5.15/20-provider-configuration-admin-tui-design.md @@ -0,0 +1,214 @@ +# 0.5.15 design candidate — provider configuration visibility in Admin/TUI + +**Status:** ratified for Slice 40 implementation, 2026-08-16. This supersedes +the earlier conditional draft after the Slice 40 audit resolved B2 with the +child-owned runtime-config handoff below. It does not authorize +provider/model/credential CRUD. CRUD requires its own durable +configuration-owner decision. + +## Decision proposed for review + +Make the initial candidate **read-only provider-configuration visibility**. +Do not put provider CRUD in 0.5.15 unless HITL later approves a separate durable +configuration-owner design. Airlock's deployment policy remains reviewed YAML +and environment/secret-manager references followed by restart. + +This is deliberately narrower than LiteLLM's database-backed admin UI: its +documented `store_model_in_db` makes database models additional deployments to +YAML models, which would create a second routing-policy authority and can load +balance both. That conflicts with Airlock's explicit-alias and reviewed-policy +contract. [LiteLLM config model](https://docs.litellm.ai/docs/proxy/configs) + +## Current facts and research + +* `airlock/proxy.py` reads YAML at startup, builds a temporary runtime config + only for startup rewrites, then starts LiteLLM. `config.local.yaml` may be an + included deployment overlay. There is no live config store or reload owner. +* `model_list` is the served-alias allowlist. `capability_record()` and + `airlock_provider_for()` are the shared truth for the served provider and + capability metadata. Discovery is optional, informational, and cannot add an + alias (`AIRLOCK_STARTUP_MODEL_DISCOVERY=0` by default). +* The installed Admin perimeter (`airlock/admin/http.py`) is a proxy-process, + authenticated, bounded read/mutation seam. Its existing `GET .../providers` + snapshot is operational state (quarantine, limits, spend), not configuration. + The separate-process TUI consumes it through `tui/admin_client.py`. +* `tui/screens/config.py` already labels provider-key inputs as `applies=False`; + it masks process environment but must not be mistaken for a secret editor. +* LiteLLM documents `model_name` as the client alias and + `litellm_params.model` as the routed model, including `os.environ/NAME` + secret references. [LiteLLM config reference](https://docs.litellm.ai/docs/proxy/configs) + This matches Airlock's model-list ownership but not its policy controls. +* Existing dependencies already cover the client and validation needs: Textual + supplies tables, inputs, workers, and tests + ([repository](https://github.com/Textualize/textual)); Pydantic is already + transitive through LiteLLM and is suitable for a pure DTO/validation boundary + ([repository](https://github.com/pydantic/pydantic)). PyYAML is already + pinned, but safe-load/safe-dump would discard comments/formatting and must not + be used to implement editing ([repository](https://github.com/yaml/pyyaml)). +* `ruamel.yaml` is a possible round-trip YAML library + ([repository](https://github.com/pycontribs/ruamel.yaml)), but adding it now + would solve syntax preservation without solving configuration ownership, + secret references, concurrent deploys, or reload. **Reject for 0.5.15.** + +## Requirements + +### Functional requirements + +1. A new authenticated `GET /airlock/admin/config/providers` returns a bounded, + source-labelled, **redacted effective startup configuration inventory**: + provider token, aliases, underlying model identifier, allowed endpoints, + region, deprecated marker, stable API-base host only, opaque credential + reference kind and boolean presence, and whether a setting is static or + derived. The fixed limits are 64 providers, 200 aliases total, and 256 + characters for every displayed metadata string; deterministic truncation is + labelled in the response. +2. The response uses the same resolved include/config semantics that Airlock + uses for launch, but reports `source: startup_config`, a config fingerprint + (non-secret hash of the canonical credential-blind redacted DTO), load timestamp, schema + version, and `restart_required: true`. It is not a + promise that a later disk edit is active. +3. The Admin scope is new and least-privilege: `admin:read_config`; master key + and trusted loopback retain their existing authority. Disabled Admin remains + 404. No config read becomes remotely visible through `admin:read` by default. +4. The TUI gains a read-only Provider Configuration panel that calls only this + endpoint in a background worker. It shows freshness/source, aliases and + capability truth alongside (not mixed with) the current live operational + snapshot. An unavailable/unauthorized endpoint degrades to an explicit + unavailable state; it never falls back to reading another process's files. +5. The projection never exposes credential values, literal secret length/prefix, + request headers, full `api_base` path/query/userinfo, included-file path, + unrecognised environment variable, or a provider error body. `api_base` is + normalized to a host only; secret values are represented only as + `credential: {kind: env_ref|credential_ref|none|redacted_literal, + configured: bool}`. +6. Discovery output and live provider health stay distinct: discovered models + cannot appear as configured aliases; operational state may be absent for an + otherwise configured provider. + +### Explicit non-goals + +No create/update/delete/reorder model aliases; no editing YAML, `.env`, Docker +environment, Kubernetes Secret/ConfigMap, provider credentials, router +fallbacks, budget or guardrail policy; no automatic discovery-to-enable; no +live reload; no generic file browser; no adoption of LiteLLM's DB config/model +management as an Airlock control plane. + +## Acceptance criteria + +1. Fixture configurations with direct includes, provider-prefixed aliases, enhanced + profiles, regional Vertex, missing credential refs, and duplicate provider + aliases produce a deterministic inventory whose alias/provider/endpoints + agree with `capability_record()`. +2. Snapshot tests prove no sentinel secret, raw API base path, include path, or + arbitrary environment variable can reach JSON, TUI render, JSONL, exception, + metrics, or response headers. +3. `admin:read` alone is 403; `admin:read_config`, master key, and trusted + loopback succeed only when Admin is enabled. Disabled remains 404; bad/missing + credentials reveal no inventory. +4. TUI tests prove bounded background refresh, table/detail rendering, stale or + unavailable source indication, and no local-file fallback. Existing TUI + operational-provider and config-screen behaviour remains unchanged. +5. `/v1/models`, `/model/info`, routing, guardrails, and model discovery have + byte-for-byte/behavioural regression coverage showing the new snapshot is + off the inference hot path. +6. Documentation says an edit still requires the deployment workflow and + restart; an operator cannot infer that the UI applied an edit. + +## Draft design + +### Ratified data flow and B2 decision + +```text +reviewed YAML + direct deployment overlay + | (pinned LiteLLM include-list expansion; no independent resolver) + v +materialized runtime config ---- `--config` ----> LiteLLM child + | | + +---- `AIRLOCK_CONFIG` ----------------+ + | (child startup only) + v + Admin policy + immutable redacted snapshot + | + GET + admin:read_config, Cache-Control: no-store + v + TUI background HTTP client / read-only panel +``` + +1. Add `airlock/litellm_config.py`, a narrow shared loader that exactly applies + the pinned LiteLLM include contract: includes must be a list of paths; + paths resolve from the root file and are processed in active-list order; an + included list extends an existing root key (including `include`, which queues + descendants after existing entries and preserves LiteLLM's malformed-target + failure); scalar/dict values replace the root value; `include` is removed + after expansion. + It must preserve sensible file/parser errors and deep-copy values so it + never aliases the caller's YAML structure. +2. Always materialize exactly one private runtime config file and pass its path + to LiteLLM and through `AIRLOCK_CONFIG`; this is the sole Slice-40 + configuration authority. The parent performs CC-12 validation and all its + startup consumers use the same direct-resolved mapping. The child configures + its Admin policy and builds the snapshot from that same runtime path before + mounting the perimeter. The models capability seam consumes that materialized + path too. +3. Add a small pure `airlock/provider_configuration.py` projector. It receives + the direct-resolved child configuration and a controlled environment lookup, + calls existing `capability_record`, and returns immutable redacted DTOs. It + never returns source YAML, calls a provider, names a secret or environment + variable, or makes a secret change observable through its fingerprint. +4. Add an Admin read route through the existing `AdminMiddleware`/pure-handler + pattern. Keep the response bounded (for example 200 aliases, 64 providers, + fixed field lengths) and add `Cache-Control: no-store`. +5. Extend `tui/admin_client.py` with a typed fetch and a narrowly scoped panel + or Provider-detail subview. Use existing `@work(thread=True)` style and a + refresh interval no faster than the operational snapshot. Never put any + credential entry field on the view. +6. Only after separate approval for CRUD, author a new design that selects one + durable configuration owner (GitOps/file handoff, a secret-manager-aware + desired-state store, or a specifically accepted LiteLLM DB integration), + optimistic concurrency/versioning, semantic validation on a copy, atomic + activation/restart, audit/rollback, and an approval flow. That is not an + extension point to silently add now. + +## Architectural alignment and blast radius + +| Concern | Alignment / containment | +| --- | --- | +| Gateway ownership | Reuses LiteLLM for transport and Airlock's ASGI Admin seam; does not replace routing or model resolution. | +| Explicit aliases | Uses `model_list` and `capability_record`; no discovery or credential creates a route. | +| Process boundary | TUI remains an HTTP client, never shares state or reads proxy files. | +| Secret boundary | Projects presence/reference only; redaction is tested across all durable/remote surfaces. | +| Hot path | Startup snapshot plus authenticated admin GET only; no pre-call hook, proxy middleware for inference, or provider request. | +| Authorization | New read-specific scope avoids widening operational-history or write scopes. | +| Blast radius | Additive Admin/TUI/docs/tests; risks are config/secret disclosure and semantic mismatch, contained by typed projection, response bounds, and no-store responses. | + +The principal risk is presenting configuration as live policy when it is not. +`source`, fingerprint, timestamp, and `restart_required` are contractual, and +the TUI must render them prominently. + +## Rollout and verification plan + +1. **Design gate (closed):** remote capability access is explicitly scoped; + the response uses 64 providers, 200 aliases, 256-character safe strings, + opaque credential kinds, and a canonical-redacted fingerprint. +2. **RED:** pure-projection and redaction tests first, then Admin authorization, + include-resolution parity, TUI, and full proxy regressions. +3. **Canary:** Admin remains disabled by default. Enable its existing explicit + `admin.enabled` switch only on a local test deployment; compare snapshot + aliases/capabilities against `/v1/models` and `/model/info`, not provider + discovery. +4. **Operational verification:** inspect captured output and JSONL using + sentinel credentials, exercise absent/disabled/admin-denied paths, restart + after a reviewed config change, and verify the fingerprint changes only after + restart. +5. **Release gate:** independent security/architecture review confirms no second + config authority or secret egress; HITL accepts or rejects promotion. + +## Accepted security answers + +* Remote use requires the new explicit `admin:read_config` scope; loopback and + the master key retain their existing full-Admin authority. +* Reference names and handles are not safe output. The API reports only an + opaque kind and `configured` boolean. +* The launcher, child policy, snapshot, and capability seam share the pinned + LiteLLM include-list expansion and runtime-path handoff. A separate Airlock + resolver with different recursion/order semantics is expressly rejected. diff --git a/dev/plans/0.5.15/40-provider-configuration-visibility-status.md b/dev/plans/0.5.15/40-provider-configuration-visibility-status.md new file mode 100644 index 0000000..aa2170f --- /dev/null +++ b/dev/plans/0.5.15/40-provider-configuration-visibility-status.md @@ -0,0 +1,64 @@ +# Slice 40 — provider configuration visibility status + +**Status:** implemented and independently reviewed, 2026-08-16. + +The same design-review subagent approved the final cleanup fix after FIX-1 +through FIX-4 review cycles; the independent verifier's include-parity finding +was incorporated before that approval. + +## Delivered + +* A single private, materialized runtime YAML file is now the authority for + LiteLLM, launcher policy consumers, the LiteLLM child, child Admin policy, + the immutable provider-configuration projection, and model capability data. + It is passed through both `--config` and `AIRLOCK_CONFIG`, and removed after + the child exits or any pre-launch failure. +* `GET /airlock/admin/config/providers` provides a bounded, source-labelled, + redacted startup snapshot under the new `admin:read_config` scope with + `Cache-Control: no-store`. Admin stays explicitly opt-in; `admin:read` alone + is denied and disabled Admin remains 404. +* The projection caps providers/aliases/text fields, omits secrets, references, + paths, URL userinfo/query, and invalid URL values, and uses a + credential-blind canonical fingerprint. It classifies `api_key` and + `vertex_credentials` without exposing field or reference names. +* The Config TUI has a separate read-only, HTTP-only **Configured** tab. Its + background worker renders success/unavailable state and never reads proxy + configuration files as a fallback. + +## Include-semantic correction + +Initial Slice 40 audit/review text assumed pinned LiteLLM was one-level only. +Runtime parity testing demonstrated the actual `ProxyConfig._process_includes` +behavior: an included `include:` list extends the active root list during +iteration, which queues descendants after existing entries. The shared resolver +and canonical materialization deliberately reproduce that behavior. The related +historical FIX disposition records this correction. + +## RED/GREEN evidence + +* **RED:** new resolver and projection tests first failed with missing module + imports; later focused RED tests exposed runtime-file authority and URL/ + credential edge cases. +* **GREEN:** resolver parity calls pinned LiteLLM directly; tests cover active + nested include expansion, order/list extension, malformed list target, + parent/child canonical mapping and CC-12, runtime cleanup, redaction, + Vertex credentials, invalid API bases, bounds/fingerprint, Admin scope, + models seam, and TUI success/unavailable worker paths. + +## Verification + +* `23 passed` — resolver/projection/startup-validation/bootstrap tests. +* Focused parent authority/CC-12/cleanup tests: `3 passed`. +* Focused TUI configured-panel test: `1 passed`. +* Ruff check/format, `git diff --check`, and `mkdocs build --strict --quiet` + passed. +* `make ensure-spacy` confirmed the declared model and escalated `make verify` + passed. (The unprivileged gate could not write uv's cache.) +* The broader non-live `make test` was started; focused Slice 40 coverage is + recorded above. No live provider calls were made. + +## Non-goals retained + +No configuration/credential/model CRUD, YAML editing, reload, discovery +enablement, LiteLLM database configuration, generic file access, or second +configuration authority was introduced. diff --git a/dev/plans/0.5.15/FIX-1-FIX-2-admin-review-disposition.md b/dev/plans/0.5.15/FIX-1-FIX-2-admin-review-disposition.md index fbbe8eb..44b773b 100644 --- a/dev/plans/0.5.15/FIX-1-FIX-2-admin-review-disposition.md +++ b/dev/plans/0.5.15/FIX-1-FIX-2-admin-review-disposition.md @@ -1,6 +1,17 @@ # FIX-1 / FIX-2 — Admin-default and include-policy review disposition -**Status:** reviewed; no implementation applies to the current 0.5.15 tree. +**Status:** historical review corrected by Slice 40 runtime evidence on +2026-08-16; no implementation applied to the current 0.5.15 tree at the time +of the original review. + +> **Correction:** The original include conclusion was based on static reading +> of `ProxyConfig._process_includes`. Runtime parity testing against the pinned +> LiteLLM shows that an included `include:` list extends the active root list +> during iteration, so descendants are processed after already queued entries. +> Do not use this document's earlier one-level-only pseudo-code. Slice 40 now +> materializes the exact pinned active-list expansion once and passes that file +> to LiteLLM, parent consumers, child Admin policy, snapshot, and capability +> seam. ## Finding confirmation diff --git a/dev/requirements.md b/dev/requirements.md index 982c320..2c2dc7f 100644 --- a/dev/requirements.md +++ b/dev/requirements.md @@ -362,8 +362,9 @@ canonical provider, `credential_configured: true`, scan arbitrary environment variables, disclose values or variable names, make a network/provider call, alter configuration/routing/discovery/startup status, or create an Admin/TUI surface. Effective aliases SHALL use -`airlock_provider_for`; direct include semantics SHALL match installed LiteLLM -behavior without independently applying nested includes. +`airlock_provider_for`; include semantics SHALL match installed LiteLLM's +active include-list expansion, including descendants reached through an +included `include:` list. ### DFR-34 / DAC-34: Typed Fast Guardian threat-backoff response @@ -396,3 +397,30 @@ scans must be clean for the reviewed baseline; a synthetic non-usable detector fixture must fail redacted and pass after removal. The control SHALL not alter Airlock runtime behavior, configuration, image contents, or provider/deployment credential access. + +### DFR-37 / DAC-37: Read-only provider configuration projection + +When the existing Admin control plane is explicitly enabled, Airlock SHALL +provide `GET /airlock/admin/config/providers` as a bounded, immutable, +startup-effective provider-configuration projection. It SHALL require +`admin:read_config` for a capability token; the existing master-key and trusted +loopback operator paths retain full authority, while `admin:read` alone SHALL +receive `403` and a disabled Admin plane SHALL receive `404`. The projection +SHALL report only aliases, `capability_record()` provider/endpoints/underlying/ +region/deprecation truth, hostname-only API bases, opaque credential kind plus +presence, source, timestamp, schema version, credential-blind redacted canonical fingerprint, +restart-required status, and deterministic truncation within 64 providers, 200 +aliases, and 256-character metadata fields. It SHALL set `Cache-Control: +no-store`. + +The launcher, LiteLLM child, Admin policy, snapshot, and capability seam SHALL +use the same pinned LiteLLM include-list expansion semantics and the same +materialized runtime config path. The child SHALL receive that path via +`AIRLOCK_CONFIG` before installing the Admin perimeter. Neither literal or +reference credential names/values/lengths, arbitrary environment names, include +paths, raw API-base paths/query/userinfo, provider calls/errors, CRUD, reload, +discovery activation, nor a second configuration owner are permitted. The TUI +SHALL read this view only over its Admin HTTP client in a bounded background +refresh, visibly label unavailable/stale startup state, and never fall back to +another process's files. YAML plus deployment workflow and restart remain the +configuration authority. diff --git a/docs/getting-started/configuration.md b/docs/getting-started/configuration.md index 6b734e3..a318347 100644 --- a/docs/getting-started/configuration.md +++ b/docs/getting-started/configuration.md @@ -95,8 +95,13 @@ uncommitted edit (start from `config.local.yaml.example`). include: ["config.local.yaml"] ``` -LiteLLM **replaces** dict-valued keys from an included file rather than deep-merging -them, so `config.local.yaml` must list **every** MCP server you want at runtime +LiteLLM processes includes in source order. It **replaces** dict-valued keys +from an included file rather than deep-merging them, and extends an existing +top-level list with an included list. In the pinned version, an included +`include:` list extends the active include list too, so nested files are reached +after files already queued from the root directory. Airlock materializes those +same semantics into the private runtime config passed to the LiteLLM child. +Therefore `config.local.yaml` must list **every** MCP server you want at runtime (including bundled ones like `newscatcher`). LiteLLM also does **not** expand `${HOME}`/`~` in MCP `command`/`args`, and only launches commands whose basename is on its allowlist. Full rules and examples in [MCP Servers](../guide/mcp-servers.md). diff --git a/docs/guide/admin-api.md b/docs/guide/admin-api.md index 60130fc..e92f587 100644 --- a/docs/guide/admin-api.md +++ b/docs/guide/admin-api.md @@ -78,6 +78,7 @@ master key (and any loopback operator) satisfies all of them. | Scope | Grants | |---|---| | `admin:read` | The read-only `GET` endpoints | +| `admin:read_config` | The redacted startup provider-configuration view only | | `admin:clear_quarantine` | Clear a provider or client→provider quarantine | | `admin:reset_circuit` | Reset a model circuit | | `admin:clear_backoff` | Clear a client threat backoff | @@ -138,10 +139,26 @@ All routes are under `/airlock/admin/` and only exist when `admin.enabled: true` | `GET` | `/airlock/admin/providers` | `admin:read` | | `GET` | `/airlock/admin/clients` | `admin:read` | | `GET` | `/airlock/admin/circuits` | `admin:read` | +| `GET` | `/airlock/admin/config/providers` | `admin:read_config` | These return a richer view of live protection state than the read-only `GET /health/circuits`. +### Read configured providers + +`GET /airlock/admin/config/providers` is deliberately distinct from live +provider health and requires `admin:read_config` (or the existing loopback or +master-key operator authority). It reports the LiteLLM child's bounded, +redacted startup configuration: configured aliases and capability metadata, +hostname-only API bases, opaque credential kind/presence, load time, and a +redacted-state fingerprint. It sends `Cache-Control: no-store`. + +It never contains credential values or reference names, environment-variable +names, include paths, full URLs, provider errors, or a way to edit/reload +configuration. The view is not a disk watcher: apply configuration through the +normal reviewed deployment workflow and restart the proxy before expecting a +change. `admin:read` alone receives `403`; disabled Admin still returns `404`. + ```bash # Loopback operator — no credential needed: curl http://localhost:4000/airlock/admin/providers diff --git a/tests/test_admin_http.py b/tests/test_admin_http.py index 4f8b423..021f447 100644 --- a/tests/test_admin_http.py +++ b/tests/test_admin_http.py @@ -14,6 +14,7 @@ from airlock.admin.http import AdminMiddleware, handle_admin_request from airlock.admin.policy import Principal, configure_admin, decide from airlock.admin.tokens import mint_token +from airlock.provider_configuration import configure_provider_configuration @pytest.fixture(autouse=True) @@ -115,6 +116,41 @@ def test_get_providers(self, fresh_state_store): assert s == 200 assert body["providers"]["openai"]["quarantined"] is True + def test_provider_configuration_requires_distinct_scope_and_is_no_store(self): + configure_provider_configuration( + { + "model_list": [ + { + "model_name": "safe", + "litellm_params": { + "model": "openai/gpt-safe", + "api_key": "sk-SLICE40-SENTINEL", + }, + } + ] + }, + loaded_at="2026-08-16T00:00:00Z", + ) + read_only = mint_token("reader", ["admin:read"], 60) + status, body, headers = handle_admin_request( + "GET", "/airlock/admin/config/providers", b"", Principal(bearer=read_only) + ) + assert status == 403 and "providers" not in body + scoped = mint_token("config-reader", ["admin:read_config"], 60) + status, body, headers = handle_admin_request( + "GET", "/airlock/admin/config/providers", b"", Principal(bearer=scoped) + ) + assert status == 200 + assert headers == {"cache-control": "no-store"} + assert "sk-SLICE40-SENTINEL" not in json.dumps(body) + + def test_provider_configuration_disabled_admin_is_404(self): + configure_admin({"admin": {"enabled": False}}) + status, _body, _headers = handle_admin_request( + "GET", "/airlock/admin/config/providers", b"", self._loop() + ) + assert status == 404 + def test_operational_records_are_proxy_owned_and_loopback_only(self, monkeypatch): from airlock.operational_reads import OperationalReadPage diff --git a/tests/test_litellm_config.py b/tests/test_litellm_config.py new file mode 100644 index 0000000..867d521 --- /dev/null +++ b/tests/test_litellm_config.py @@ -0,0 +1,88 @@ +"""Parity tests for the deliberately narrow LiteLLM include resolver.""" + +from __future__ import annotations + +import pytest + +from airlock.litellm_config import resolve_litellm_direct_config + + +def test_nested_include_extends_the_active_litellm_include_list_in_order(tmp_path): + grandchild = tmp_path / "grandchild.yaml" + grandchild.write_text("admin:\n enabled: true\n") + first = tmp_path / "first.yaml" + first.write_text( + "include: [grandchild.yaml]\nadmin:\n enabled: false\nitems: [first]\n" + ) + second = tmp_path / "second.yaml" + second.write_text("admin:\n enabled: false\nitems: [second]\n") + root = tmp_path / "config.yaml" + root.write_text("include: [first.yaml, second.yaml]\nitems: [root]\n") + + resolved = resolve_litellm_direct_config(root) + + assert resolved["admin"] == {"enabled": True} + assert resolved["items"] == ["root", "first", "second"] + assert "include" not in resolved + + +def test_direct_include_replaces_dict_and_scalar_but_extends_lists(tmp_path): + included = tmp_path / "included.yaml" + included.write_text("settings:\n child: true\nmode: included\nitems: [two]\n") + root = tmp_path / "config.yaml" + root.write_text( + "include: [included.yaml]\nsettings:\n root: true\nmode: root\nitems: [one]\n" + ) + + assert resolve_litellm_direct_config(root) == { + "settings": {"child": True}, + "mode": "included", + "items": ["one", "two"], + } + + +def test_direct_include_preserves_litellm_malformed_list_target_failure(tmp_path): + from litellm.proxy.proxy_server import ProxyConfig + + included = tmp_path / "included.yaml" + included.write_text("items: [child]\n") + root = tmp_path / "config.yaml" + root.write_text("include: [included.yaml]\nitems: not-a-list\n") + + import yaml + + with pytest.raises(AttributeError, match="extend"): + ProxyConfig()._process_includes(yaml.safe_load(root.read_text()), str(tmp_path)) + with pytest.raises(AttributeError, match="extend"): + resolve_litellm_direct_config(root) + + +@pytest.mark.parametrize("include", ["child.yaml", "true"]) +def test_include_must_be_a_list(tmp_path, include): + root = tmp_path / "config.yaml" + root.write_text(f"include: {include}\n") + with pytest.raises(ValueError, match="include"): + resolve_litellm_direct_config(root) + + +def test_resolver_matches_pinned_litellm_for_overlay_and_nested_include( + tmp_path, +): + from litellm.proxy.proxy_server import ProxyConfig + + nested = tmp_path / "nested.yaml" + nested.write_text("admin:\n enabled: true\n") + first = tmp_path / "first.yaml" + first.write_text( + "include: [nested.yaml]\nadmin:\n enabled: false\nitems: [first]\n" + ) + second = tmp_path / "second.yaml" + second.write_text("mode: second\nitems: [second]\n") + root = tmp_path / "config.yaml" + root.write_text("include: [first.yaml, second.yaml]\nmode: root\nitems: [root]\n") + + import yaml + + upstream_input = yaml.safe_load(root.read_text()) + expected = ProxyConfig()._process_includes(upstream_input, str(tmp_path)) + assert resolve_litellm_direct_config(root) == expected diff --git a/tests/test_models_seam.py b/tests/test_models_seam.py index c66c31e..498151a 100644 --- a/tests/test_models_seam.py +++ b/tests/test_models_seam.py @@ -370,6 +370,21 @@ def test_build_map_dict_model_list_returns_empty(tmp_path, monkeypatch): assert _build_capability_map() == {} +def test_build_map_uses_same_direct_include_resolution(tmp_path, monkeypatch): + included = tmp_path / "included.yaml" + included.write_text( + "model_list:\n - model_name: included\n litellm_params:\n model: openai/included\n" + ) + _write_config( + tmp_path, + "include: [included.yaml]\nmodel_list:\n" + " - model_name: root\n litellm_params:\n model: anthropic/root\n", + monkeypatch, + ) + result = _build_capability_map() + assert set(result) == {"root", "included"} + + def test_build_map_skips_malformed_entry_keeps_good(tmp_path, monkeypatch): _write_config( tmp_path, diff --git a/tests/test_provider_configuration.py b/tests/test_provider_configuration.py new file mode 100644 index 0000000..4cc8b23 --- /dev/null +++ b/tests/test_provider_configuration.py @@ -0,0 +1,131 @@ +"""Redaction and bounds for Slice 40's startup configuration snapshot.""" + +from __future__ import annotations + +import json + +from airlock.provider_configuration import ( + configure_provider_configuration, + provider_configuration_snapshot, +) + + +def _entry(name: str, model: str, **params): + return { + "model_name": name, + "litellm_params": {"model": model, **params}, + } + + +def test_snapshot_is_redacted_stable_and_models_capability_truth(): + sentinel = "sk-SLICE40-DO-NOT-LEAK" + config = { + "model_list": [ + _entry( + "vertex/chat", + "vertex_ai/gemini-2.5-pro", + vertex_location="us-central1", + api_key="os.environ/VERTEX_SECRET_NAME", + api_base=f"https://user:{sentinel}@api.example.test/v1?secret={sentinel}", + ), + _entry( + "enhanced", + "enhanced/profile", + enhanced_profile={"target_model": "anthropic/claude"}, + api_key=sentinel, + ), + _entry("no-key", "openai/gpt-test"), + _entry( + "vertex-credential", + "vertex_ai/gemini-2.5-flash", + vertex_credentials="os.environ/VERTEX_CREDENTIALS", + api_base="https://[", + ), + ] + } + + configure_provider_configuration( + config, + getenv=lambda name: ( + sentinel if name in {"VERTEX_SECRET_NAME", "VERTEX_CREDENTIALS"} else None + ), + loaded_at="2026-08-16T12:00:00Z", + ) + snapshot = provider_configuration_snapshot() + encoded = json.dumps(snapshot, sort_keys=True) + + assert snapshot["source"] == "startup_config" + assert snapshot["restart_required"] is True + assert snapshot["schema_version"] == 1 + assert snapshot["loaded_at"] == "2026-08-16T12:00:00Z" + assert sentinel not in encoded + assert "VERTEX_SECRET_NAME" not in encoded + assert "/v1" not in encoded and "user:" not in encoded + vertex = next(p for p in snapshot["providers"] if p["provider"] == "vertex_ai") + alias = next( + alias for alias in vertex["aliases"] if alias["alias"] == "vertex/chat" + ) + assert alias["api_base_host"] == "api.example.test" + assert alias["credential"] == {"kind": "env_ref", "configured": True} + assert alias["endpoints"] == ["chat", "batch"] + assert alias["region"] == "us-central1" + assert next( + a for p in snapshot["providers"] for a in p["aliases"] if a["alias"] == "no-key" + )["credential"] == {"kind": "none", "configured": False} + vertex_credential = next( + a + for p in snapshot["providers"] + for a in p["aliases"] + if a["alias"] == "vertex-credential" + ) + assert vertex_credential["credential"] == {"kind": "env_ref", "configured": True} + assert vertex_credential["api_base_host"] is None + configure_provider_configuration( + {"model_list": [config["model_list"][-1]]}, + getenv=lambda _name: None, + loaded_at="2026-08-16T12:00:00Z", + ) + assert provider_configuration_snapshot()["providers"][0]["aliases"][0][ + "credential" + ] == {"kind": "env_ref", "configured": False} + + +def test_snapshot_bounds_and_fingerprint_do_not_oracle_literal_secret(): + provider_bound = { + "model_list": [_entry(f"p{i}/alias", f"p{i}/model") for i in range(65)] + } + configure_provider_configuration(provider_bound, loaded_at="t") + first = provider_configuration_snapshot() + assert len(first["providers"]) == 64 + assert first["truncated"]["providers"] is True + + def build(secret: str): + return { + "model_list": [ + _entry(f"p0/alias{i}", f"p0/model{i}", api_key=secret) + for i in range(201) + ] + } + + configure_provider_configuration(build("first-secret"), loaded_at="t") + first = provider_configuration_snapshot() + configure_provider_configuration(build("different-secret"), loaded_at="t") + second = provider_configuration_snapshot() + + assert sum(len(p["aliases"]) for p in second["providers"]) == 200 + assert second["truncated"] == {"providers": False, "aliases": True} + assert first["fingerprint"] == second["fingerprint"] + + configure_provider_configuration( + {"model_list": [_entry("env", "openai/env", api_key="os.environ/KEY")]}, + getenv=lambda _name: None, + loaded_at="t", + ) + absent = provider_configuration_snapshot() + configure_provider_configuration( + {"model_list": [_entry("env", "openai/env", api_key="os.environ/KEY")]}, + getenv=lambda _name: "now-present", + loaded_at="t", + ) + present = provider_configuration_snapshot() + assert absent["fingerprint"] == present["fingerprint"] diff --git a/tests/test_proxy.py b/tests/test_proxy.py index 83e592e..ef7d882 100644 --- a/tests/test_proxy.py +++ b/tests/test_proxy.py @@ -149,6 +149,135 @@ def test_main_uses_subprocess_run(self, config_file, monkeypatch): # Verify check=False is passed assert mock_run.call_args[1].get("check") is False + def test_main_hands_same_runtime_file_to_litellm_child( + self, config_file, monkeypatch + ): + monkeypatch.setenv("AIRLOCK_CONFIG", str(config_file)) + mock_result = MagicMock(returncode=0) + with ( + patch("airlock.proxy.subprocess.run", return_value=mock_result) as run, + patch("airlock.proxy.fetch_live_provider_models", return_value=[]), + pytest.raises(SystemExit), + ): + main() + command = run.call_args.args[0] + child_env = run.call_args.kwargs["env"] + assert child_env["AIRLOCK_CONFIG"] == command[command.index("--config") + 1] + + def test_directly_included_admin_is_checked_before_external_plaintext_launch( + self, tmp_path, monkeypatch + ): + included = tmp_path / "admin.yaml" + included.write_text("admin:\n enabled: true\n") + config = tmp_path / "config.yaml" + config.write_text("include: [admin.yaml]\nmodel_list: []\n") + monkeypatch.setenv("AIRLOCK_CONFIG", str(config)) + monkeypatch.setenv("AIRLOCK_HOST", "0.0.0.0") + monkeypatch.delenv("AIRLOCK_SSL_CERTFILE", raising=False) + monkeypatch.delenv("AIRLOCK_SSL_KEYFILE", raising=False) + with pytest.raises(RuntimeError, match="non-loopback"): + main() + + def test_all_parent_consumers_receive_the_child_runtime_mapping( + self, tmp_path, monkeypatch + ): + included = tmp_path / "included.yaml" + included.write_text( + "admin:\n enabled: false\nmodel_list:\n" + " - model_name: included\n litellm_params:\n model: openai/included\n" + ) + config = tmp_path / "config.yaml" + config.write_text( + "include: [included.yaml]\nmodel_list:\n" + " - model_name: root\n litellm_params:\n model: anthropic/root\n" + ) + monkeypatch.setenv("AIRLOCK_CONFIG", str(config)) + captured: list[dict] = [] + + def capture(mapping, *args, **kwargs): + captured.append(mapping) + + result = MagicMock(returncode=0) + runtime_mappings: list[dict] = [] + + def run_child(*args, **kwargs): + runtime_mappings.append( + yaml.safe_load(Path(kwargs["env"]["AIRLOCK_CONFIG"]).read_text()) + ) + return result + + with ( + patch("airlock.fast.router.set_router_config", capture), + patch("airlock.fast.settings.configure_settings", capture), + patch("airlock.fast.admission.configure_admission", capture), + patch("airlock.fast.state.configure_breaker", capture), + patch("airlock.admin.policy.configure_admin", capture), + patch( + "airlock.guardrails.overrides.configure_guardrail_overrides", capture + ), + patch("airlock.transparency.configure_transparency", capture), + patch("airlock.proxy.subprocess.run", side_effect=run_child), + patch("airlock.proxy.fetch_live_provider_models", return_value=[]), + pytest.raises(SystemExit), + ): + main() + runtime = runtime_mappings[0] + assert len(captured) == 7 + assert all(mapping == runtime for mapping in captured) + + def test_prelaunch_failure_removes_private_runtime_config( + self, config_file, monkeypatch + ): + import airlock.proxy as proxy_module + + monkeypatch.setenv("AIRLOCK_CONFIG", str(config_file)) + generated: list[str] = [] + real_prepare = proxy_module._prepare_runtime_config + + def capture_prepare(path: str): + runtime_path, temporary_path = real_prepare(path) + generated.append(runtime_path) + return runtime_path, temporary_path + + monkeypatch.setattr(proxy_module, "_prepare_runtime_config", capture_prepare) + with patch( + "airlock.fast.router.set_router_config", + side_effect=RuntimeError("parent configuration failed"), + ): + with pytest.raises(RuntimeError, match="parent configuration failed"): + main() + assert generated + assert not Path(generated[0]).exists() + + def test_runtime_reread_failure_removes_private_runtime_config( + self, config_file, monkeypatch + ): + import airlock.proxy as proxy_module + + monkeypatch.setenv("AIRLOCK_CONFIG", str(config_file)) + generated: list[str] = [] + real_prepare = proxy_module._prepare_runtime_config + real_resolve = proxy_module.resolve_litellm_direct_config + + def capture_prepare(path: str): + runtime_path, temporary_path = real_prepare(path) + generated.append(runtime_path) + return runtime_path, temporary_path + + def fail_runtime_reread(path): + if generated and str(path) == generated[0]: + raise ValueError("runtime reread failed") + return real_resolve(path) + + monkeypatch.setattr(proxy_module, "_prepare_runtime_config", capture_prepare) + monkeypatch.setattr( + proxy_module, "resolve_litellm_direct_config", fail_runtime_reread + ) + with pytest.raises(ValueError, match="runtime reread failed"): + main() + assert generated + assert not Path(generated[0]).exists() + def test_main_default_host_port(self, config_file, monkeypatch): monkeypatch.setenv("AIRLOCK_CONFIG", str(config_file)) monkeypatch.delenv("AIRLOCK_HOST", raising=False) @@ -551,12 +680,11 @@ def test_fathom_logger_enabled_by_env(self, monkeypatch): class TestRuntimeConfigPreparation: - def test_prepare_runtime_config_returns_original_when_no_overrides( + def test_prepare_runtime_config_always_materializes_canonical_child_file( self, tmp_path, monkeypatch ): - # No model_list -> nothing to inject; with no env overrides the original - # config path is returned unchanged. (A model_list always triggers the - # unconditional model_info injection, exercised in the injection tests.) + # Even without rewrites, the child must consume a private canonical + # file so its Admin policy and snapshot cannot diverge from LiteLLM. cfg = tmp_path / "config.yaml" cfg.write_text("litellm_settings: {}\n") @@ -565,8 +693,9 @@ def test_prepare_runtime_config_returns_original_when_no_overrides( monkeypatch.delenv("AIRLOCK_BACKGROUND_HEALTH_CHECKS", raising=False) runtime_path, temp_path = _prepare_runtime_config(str(cfg)) - assert runtime_path == str(cfg) - assert temp_path is None + assert runtime_path != str(cfg) + assert temp_path == runtime_path + assert Path(runtime_path).parent != cfg.parent def test_prepare_runtime_config_can_strip_mcp_servers_in_off_mode( self, tmp_path, monkeypatch @@ -620,7 +749,7 @@ def test_mcp_off_inlines_local_overrides_without_reintroducing_mcp( assert "mcp_servers" not in loaded assert loaded["general_settings"]["custom_value"] == "local" - def test_prepare_runtime_config_uses_writable_temp_and_absolutizes_includes( + def test_prepare_runtime_config_uses_writable_temp_and_consumes_includes( self, tmp_path, monkeypatch ): include = tmp_path / "config.local.yaml" @@ -641,7 +770,7 @@ def test_prepare_runtime_config_uses_writable_temp_and_absolutizes_includes( assert Path(runtime_path).parent != cfg.parent with open(runtime_path, encoding="utf-8") as f: loaded = yaml.safe_load(f) or {} - assert loaded["include"] == [str(include.resolve())] + assert "include" not in loaded def test_prepare_runtime_config_keeps_mcp_servers_in_lazy_mode( self, tmp_path, monkeypatch diff --git a/tests/test_proxy_bootstrap.py b/tests/test_proxy_bootstrap.py index 298377e..8a7d9f6 100644 --- a/tests/test_proxy_bootstrap.py +++ b/tests/test_proxy_bootstrap.py @@ -6,6 +6,8 @@ from __future__ import annotations import airlock.proxy_bootstrap as pb +from airlock.admin.policy import admin_enabled +from airlock.provider_configuration import provider_configuration_snapshot EXPECTED_ORDER = [ "docs", @@ -39,6 +41,7 @@ def _install(): pb, "install_embedding_request_boundary_on_proxy_app", _mk("embedding_boundary") ) monkeypatch.setattr(pb, "install_admin_on_proxy_app", _mk("admin")) + monkeypatch.setattr(pb, "_configure_child_startup_configuration", lambda: None) monkeypatch.setattr(pb, "install_batch_gateway_on_proxy_app", _mk("batch")) monkeypatch.setattr( pb, "install_models_capability_seam_on_proxy_app", _mk("models_capability") @@ -47,3 +50,26 @@ def _install(): pb.bootstrap_airlock_proxy() assert recorded == EXPECTED_ORDER + + +def test_child_configuration_uses_runtime_file_with_litellm_nested_include_order( + tmp_path, monkeypatch +): + grandchild = tmp_path / "grandchild.yaml" + grandchild.write_text("admin:\n enabled: true\n") + direct = tmp_path / "direct.yaml" + direct.write_text( + "include: [grandchild.yaml]\nadmin:\n enabled: false\nmodel_list:\n" + " - model_name: direct\n litellm_params:\n model: openai/direct\n" + ) + runtime = tmp_path / "runtime.yaml" + runtime.write_text("include: [direct.yaml]\nmodel_list: []\n") + monkeypatch.setenv("AIRLOCK_CONFIG", str(runtime)) + + pb._configure_child_startup_configuration() + + assert admin_enabled() is True + assert ( + provider_configuration_snapshot()["providers"][0]["aliases"][0]["alias"] + == "direct" + ) diff --git a/tests/test_startup_validation.py b/tests/test_startup_validation.py index eda1c98..dcd7388 100644 --- a/tests/test_startup_validation.py +++ b/tests/test_startup_validation.py @@ -85,7 +85,7 @@ def test_each_recognised_provider_is_suppressed_by_a_matching_alias( ) -def test_direct_include_extends_model_list_but_nested_include_is_ignored(tmp_path): +def test_include_descendants_extend_the_pinned_litellm_include_list(tmp_path): child = tmp_path / "child.yaml" nested = tmp_path / "nested.yaml" config = tmp_path / "config.yaml" @@ -100,7 +100,7 @@ def test_direct_include_extends_model_list_but_nested_include_is_ignored(tmp_pat warnings = credential_without_alias_warnings( config, {"OPENAI_API_KEY": "yes", "OPENROUTER_API_KEY": "yes"}.get ) - assert [warning.provider for warning in warnings] == ["openrouter"] + assert warnings == () def test_explicit_vllm_backend_counts_as_enabled(tmp_path): diff --git a/tests/test_tui.py b/tests/test_tui.py index af85e29..98056bd 100644 --- a/tests/test_tui.py +++ b/tests/test_tui.py @@ -742,6 +742,49 @@ async def test_config_has_provider_tab() -> None: assert app.query_one("#cfg-tab-providers") is not None +async def test_config_has_http_only_configured_provider_tab() -> None: + app = AirlockApp(test_harness=True) + async with app.run_test(size=(120, 40)) as pilot: + await pilot.press("4") + await pilot.pause() + assert app.query_one("#cfg-tab-configured-providers") is not None + from airlock.tui.screens.config import ConfigPane + from textual.widgets import Static + + pane = app.query_one(ConfigPane) + pane._render_provider_configuration( + None, "Configured provider state unavailable." + ) + assert "No local configuration fallback" in str( + app.query_one("#cfg-provider-config-detail", Static).content + ) + pane._render_provider_configuration( + { + "source": "startup_config", + "loaded_at": "2026-08-16T00:00:00Z", + "truncated": {"providers": False, "aliases": False}, + "providers": [ + { + "provider": "openai", + "aliases": [ + { + "alias": "reviewed", + "underlying": "openai/gpt-test", + "endpoints": ["chat"], + "credential": { + "kind": "env_ref", + "configured": True, + }, + } + ], + } + ], + }, + None, + ) + assert app.query_one("#cfg-provider-config-table").row_count == 1 + + async def test_config_has_guardrails_tab() -> None: app = AirlockApp() async with app.run_test(size=(120, 40)) as pilot: diff --git a/tests/test_tui_thread_safety.py b/tests/test_tui_thread_safety.py index dc3ef2c..44abaa3 100644 --- a/tests/test_tui_thread_safety.py +++ b/tests/test_tui_thread_safety.py @@ -39,6 +39,37 @@ async def test_refresh_mcp_servers_dispatches_widget_mutations_via_call_from_thr ) +async def test_configured_provider_refresh_dispatches_admin_result_to_ui_thread( + monkeypatch, +) -> None: + app = AirlockApp() + async with app.run_test(size=(120, 40)) as pilot: + await pilot.press("4") + await pilot.pause() + from airlock.tui.screens.config import ConfigPane + + pane = app.query_one(ConfigPane) + monkeypatch.setattr( + "airlock.tui.admin_client.provider_configuration_snapshot", + lambda _host, _port: ( + 200, + { + "providers": [], + "source": "startup_config", + "loaded_at": "t", + "truncated": {}, + }, + ), + ) + dispatch = MagicMock() + pane.app.call_from_thread = dispatch # type: ignore[method-assign] + + ConfigPane._refresh_provider_configuration.__wrapped__(pane) + + dispatch.assert_called_once() + assert dispatch.call_args.args[0] == pane._render_provider_configuration + + async def test_load_logs_dispatches_widget_mutations_via_call_from_thread( tmp_path, ) -> None: From f9836b70dd348b84702da23a315c00e58d76560c Mon Sep 17 00:00:00 2001 From: Corey Thompson Date: Sun, 16 Aug 2026 16:13:31 -0500 Subject: [PATCH 12/24] feat(admin): add secure host-console TUI profile --- airlock/admin/http.py | 4 + airlock/admin/policy.py | 67 +++++ airlock/cli/main.py | 34 +++ airlock/proxy_bootstrap.py | 16 +- airlock/tui/admin_client.py | 174 ++++++++++-- airlock/tui/remote_app.py | 128 +++++++++ ...sole-tui-container-control-plane-design.md | 255 ++++++++++++++++++ .../50-host-console-container-admin-status.md | 57 ++++ dev/requirements.md | 24 ++ docker-compose.remote-admin.yml | 25 ++ docs/getting-started/configuration.md | 7 + docs/guide/admin-api.md | 55 ++++ tests/test_admin_http.py | 122 +++++++++ tests/test_proxy_bootstrap.py | 15 ++ tests/test_remote_admin_compose.py | 26 ++ tests/test_tui.py | 47 ++++ tests/test_tui_admin_client.py | 74 +++++ 17 files changed, 1106 insertions(+), 24 deletions(-) create mode 100644 airlock/tui/remote_app.py create mode 100644 dev/plans/0.5.15/30-host-console-tui-container-control-plane-design.md create mode 100644 dev/plans/0.5.15/50-host-console-container-admin-status.md create mode 100644 docker-compose.remote-admin.yml create mode 100644 tests/test_remote_admin_compose.py diff --git a/airlock/admin/http.py b/airlock/admin/http.py index f4f19b3..68fdf6d 100644 --- a/airlock/admin/http.py +++ b/airlock/admin/http.py @@ -339,6 +339,10 @@ def handle_admin_request( result = handler([], parsed, d.actor) # Mutating ops return an admin_action record → audit + replicate. if isinstance(result, dict) and result.get("record_type") == "admin_action": + # The PDP, not request headers, supplies this stable non-secret + # transport descriptor. Existing replay ignores additive fields. + if d.auth_context: + result["auth_context"] = d.auth_context write_admin_action_record(result) except ValueError as exc: return 400, {"error": str(exc)}, {} diff --git a/airlock/admin/policy.py b/airlock/admin/policy.py index 6c51795..bacf9e2 100644 --- a/airlock/admin/policy.py +++ b/airlock/admin/policy.py @@ -17,6 +17,14 @@ import os from dataclasses import dataclass +_REMOTE_TUI_SCOPE = "admin:remote_tui" +_REMOTE_TUI_ALLOWED_SCOPES = { + "admin:read", + "admin:read_config", + "admin:clear_quarantine", +} +_REMOTE_TUI_MAX_TTL_SECONDS = 15 * 60 + # Deliberately excludes "" — a missing/stripped client address is NOT loopback # (fail closed; e.g. a Unix-socket reverse proxy forwarding remote traffic). LOOPBACK_HOSTS = {"127.0.0.1", "::1", "localhost"} @@ -28,6 +36,7 @@ class AdminConfig: trust_loopback: bool = True allow_insecure_tokens: bool = False behind_tls_proxy: bool = False + remote_tui: bool = False _admin_config = AdminConfig() @@ -52,7 +61,22 @@ def configure_admin( trust_loopback=bool(block.get("trust_loopback", True)), allow_insecure_tokens=bool(block.get("allow_insecure_tokens", False)), behind_tls_proxy=bool(block.get("behind_tls_proxy", False)), + remote_tui=bool(block.get("remote_tui", False)), ) + if cfg.remote_tui and host is not None: + if ( + not cfg.enabled + or cfg.trust_loopback + or not tls_enabled + or cfg.behind_tls_proxy + or cfg.allow_insecure_tokens + or host in LOOPBACK_HOSTS + ): + raise RuntimeError( + "admin.remote_tui requires admin.enabled, trust_loopback: false, " + "native TLS, a non-loopback bind, and neither TLS-proxy nor " + "insecure-token mode." + ) if cfg.enabled and host is not None: exposed = host not in LOOPBACK_HOSTS if ( @@ -85,6 +109,7 @@ class Decision: status: int = 200 reason: str = "" actor: str = "unknown" + auth_context: str = "" def _master_key() -> str: @@ -97,8 +122,50 @@ def decide( """Authorize an operation. Never raises — returns a Decision with a status. Order: Path A (loopback) → master key → capability JWT (scope check). + The explicit remote-TUI profile intentionally removes the first two paths + and constrains its JWT capability surface regardless of source address. """ cfg = _admin_config + if cfg.remote_tui: + if not principal.bearer: + return Decision( + False, 401, "authentication required", actor=principal.actor + ) + # Do not let the broad inference/master authority become a remote-TUI + # credential. It will fail JWT verification below as an invalid token. + master = _master_key() + if master and hmac.compare_digest(principal.bearer, master): + return Decision(False, 403, "master key is not valid for remote TUI") + from airlock.admin.tokens import TokenError, has_scope, verify_token + + try: + claims = verify_token(principal.bearer) + except TokenError: + return Decision( + False, 403, "invalid or expired token", actor=principal.actor + ) + actor = str(claims.get("sub") or "token") + issued, expires = claims.get("iat"), claims.get("exp") + if ( + not isinstance(issued, int) + or not isinstance(expires, int) + or expires - issued > _REMOTE_TUI_MAX_TTL_SECONDS + ): + return Decision( + False, 403, "remote TUI token lifetime exceeds 15 minutes", actor=actor + ) + if not has_scope(claims, _REMOTE_TUI_SCOPE): + return Decision( + False, 403, "token missing scope admin:remote_tui", actor=actor + ) + if op_scope not in _REMOTE_TUI_ALLOWED_SCOPES: + return Decision( + False, 403, "operation is not available to remote TUI", actor=actor + ) + if not has_scope(claims, op_scope): + return Decision(False, 403, f"token missing scope {op_scope}", actor=actor) + return Decision(True, actor=actor, auth_context="remote_tui_jwt") + # Path A — loopback operator. if principal.loopback and cfg.trust_loopback: return Decision(True, actor=principal.actor or "loopback") diff --git a/airlock/cli/main.py b/airlock/cli/main.py index a7637ef..3d18e89 100644 --- a/airlock/cli/main.py +++ b/airlock/cli/main.py @@ -137,6 +137,21 @@ def main(argv: list[str] | None = None) -> None: action="store_true", help="Leave the proxy running after the TUI exits.", ) + tui_parser.add_argument( + "--remote-admin", + action="store_true", + help="Use the restricted TLS capability UI for a container Admin endpoint.", + ) + tui_parser.add_argument( + "--admin-token-file", + default=None, + help="Protected capability-token file required with --remote-admin.", + ) + tui_parser.add_argument( + "--admin-ca-file", + default=None, + help="CA bundle required with --remote-admin.", + ) # -- analyze -- analyze_parser = subparsers.add_parser( @@ -503,6 +518,25 @@ def main(argv: list[str] | None = None) -> None: host = args.host or os.environ.get("AIRLOCK_HOST", "localhost") port = args.port or os.environ.get("AIRLOCK_PORT", "4000") + if args.remote_admin: + if not args.admin_token_file or not args.admin_ca_file: + parser.error( + "tui --remote-admin requires --admin-token-file and --admin-ca-file" + ) + if args.start or args.daemon: + parser.error("tui --remote-admin cannot start or own a local proxy") + from airlock.tui.remote_app import run as remote_tui_run + + remote_tui_run( + host=host, + port=port, + token_file=args.admin_token_file, + ca_file=args.admin_ca_file, + ) + return + if args.admin_token_file or args.admin_ca_file: + parser.error("Admin credential files require tui --remote-admin") + from airlock.tui.app import run as tui_run tui_run( diff --git a/airlock/proxy_bootstrap.py b/airlock/proxy_bootstrap.py index 56e6360..00f93f1 100644 --- a/airlock/proxy_bootstrap.py +++ b/airlock/proxy_bootstrap.py @@ -28,16 +28,24 @@ def _configure_child_startup_configuration() -> None: """Load the launcher's canonical runtime file before mounting Admin. - The launcher is responsible for CC-12 bind/TLS validation. This child - process receives its already-materialized config path and owns the live - Admin policy/snapshot installed on the LiteLLM application. + The child repeats the launcher's bind/TLS validation before it owns the + live Admin policy/snapshot installed on the LiteLLM application. This + prevents alternate LiteLLM child launch paths from mounting an invalid + remote-TUI perimeter. """ from airlock.litellm_config import resolve_litellm_direct_config from airlock.provider_configuration import configure_provider_configuration config_path = os.getenv("AIRLOCK_CONFIG", "config.yaml") config = resolve_litellm_direct_config(config_path) - configure_admin(config) + tls_enabled = bool(os.getenv("AIRLOCK_SSL_CERTFILE", "").strip()) and bool( + os.getenv("AIRLOCK_SSL_KEYFILE", "").strip() + ) + configure_admin( + config, + host=os.getenv("AIRLOCK_HOST", "0.0.0.0"), + tls_enabled=tls_enabled, + ) configure_provider_configuration(config) diff --git a/airlock/tui/admin_client.py b/airlock/tui/admin_client.py index 4479a8d..4318e5a 100644 --- a/airlock/tui/admin_client.py +++ b/airlock/tui/admin_client.py @@ -13,12 +13,82 @@ import base64 import os import ssl +import stat import urllib.error import urllib.request +from dataclasses import dataclass, field +from pathlib import Path _LOOPBACK_HOSTS = {"127.0.0.1", "::1", "localhost"} +class AdminConnectionError(ValueError): + """A local remote-admin client credential or TLS setup is unsafe.""" + + +@dataclass(frozen=True) +class AdminConnection: + """Immutable remote Admin transport; token is intentionally non-repr.""" + + host: str + port: str + token: str = field(repr=False) + ssl_context: ssl.SSLContext = field(repr=False, compare=False) + + @classmethod + def from_files( + cls, host: str, port: str, token_file: str | Path, ca_file: str | Path + ) -> "AdminConnection": + path = Path(token_file) + nofollow = getattr(os, "O_NOFOLLOW", None) + if nofollow is None: + raise AdminConnectionError( + "remote Admin token files require no-follow support" + ) + try: + descriptor = os.open(path, os.O_RDONLY | nofollow) + except OSError as exc: + raise AdminConnectionError( + "remote Admin token file must be a readable regular file" + ) from exc + try: + with os.fdopen(descriptor, "r", encoding="utf-8") as token_handle: + info = os.fstat(token_handle.fileno()) + if not stat.S_ISREG(info.st_mode): + raise AdminConnectionError( + "remote Admin token file must be a regular file" + ) + if info.st_mode & 0o077: + raise AdminConnectionError( + "remote Admin token file permissions must be 0600" + ) + if hasattr(os, "geteuid") and info.st_uid != os.geteuid(): + raise AdminConnectionError( + "remote Admin token file must be owned by this user" + ) + if info.st_size > 8192: + raise AdminConnectionError("remote Admin token file is too large") + token = token_handle.read().strip() + except (OSError, UnicodeError) as exc: + raise AdminConnectionError("remote Admin token file is unreadable") from exc + if not token: + raise AdminConnectionError("remote Admin token file is empty") + if not all( + char.isascii() and (char.isalnum() or char in "-_.") for char in token + ): + raise AdminConnectionError("remote Admin token file is invalid") + try: + context = ssl.create_default_context(cafile=os.fspath(ca_file)) + except (OSError, ssl.SSLError) as exc: + raise AdminConnectionError( + "remote Admin CA file is invalid or unreadable" + ) from exc + # Be explicit: a localhost target is still a remote container path. + context.check_hostname = True + context.verify_mode = ssl.CERT_REQUIRED + return cls(host=str(host), port=str(port), token=token, ssl_context=context) + + def _scheme_and_context(host: str) -> tuple[str, ssl.SSLContext | None]: if os.getenv("AIRLOCK_SSL_CERTFILE") and os.getenv("AIRLOCK_SSL_KEYFILE"): ctx = ssl.create_default_context() @@ -30,6 +100,21 @@ def _scheme_and_context(host: str) -> tuple[str, ssl.SSLContext | None]: return "http", None +def _transport( + host: str, port: str, connection: AdminConnection | None +) -> tuple[str, str, str, ssl.SSLContext | None, dict[str, str]]: + if connection is not None: + return ( + "https", + connection.host, + connection.port, + connection.ssl_context, + {"Authorization": f"Bearer {connection.token}"}, + ) + scheme, context = _scheme_and_context(host) + return scheme, host, port, context, {} + + def admin_post( host: str, port: str, @@ -37,18 +122,25 @@ def admin_post( body: dict | None = None, *, timeout: float = 5.0, + connection: AdminConnection | None = None, ) -> tuple[int, dict]: """POST to a loopback admin endpoint. Returns (status, payload). status 0 on a transport error (with an ``error`` payload). Never raises.""" - scheme, ctx = _scheme_and_context(host) + scheme, host, port, ctx, auth_headers = _transport(host, port, connection) url = f"{scheme}://{host}:{port}{path}" try: data = json.dumps(body or {}).encode() except (TypeError, ValueError): return 0, {"error": "unserializable body"} - req = urllib.request.Request( - url, data=data, method="POST", headers={"Content-Type": "application/json"} - ) + try: + req = urllib.request.Request( + url, + data=data, + method="POST", + headers={"Content-Type": "application/json", **auth_headers}, + ) + except ValueError: + return 0, {"error": "request unavailable"} try: with urllib.request.urlopen(req, timeout=timeout, context=ctx) as resp: status, raw = resp.status, (resp.read() or b"{}") @@ -68,12 +160,19 @@ def admin_post( def admin_get( - host: str, port: str, path: str, *, timeout: float = 2.0 + host: str, + port: str, + path: str, + *, + timeout: float = 2.0, + connection: AdminConnection | None = None, ) -> tuple[int, dict]: """Read a loopback admin snapshot. Never raises or emits an audit action.""" - scheme, ctx = _scheme_and_context(host) + scheme, host, port, ctx, auth_headers = _transport(host, port, connection) try: - req = urllib.request.Request(f"{scheme}://{host}:{port}{path}", method="GET") + req = urllib.request.Request( + f"{scheme}://{host}:{port}{path}", method="GET", headers=auth_headers + ) with urllib.request.urlopen(req, timeout=timeout, context=ctx) as resp: status, raw = resp.status, (resp.read() or b"{}") except urllib.error.HTTPError as exc: @@ -81,6 +180,8 @@ def admin_get( return exc.code, json.loads(exc.read() or b"{}") except (ValueError, OSError): return exc.code, {"error": f"HTTP {exc.code}"} + except ValueError: + return 0, {"error": "request unavailable"} except (urllib.error.URLError, OSError) as exc: return 0, {"error": str(exc)} try: @@ -90,9 +191,13 @@ def admin_get( return status, payload if isinstance(payload, dict) else {"data": payload} -def provider_snapshot(host: str, port: str) -> dict | None: +def provider_snapshot( + host: str, port: str, *, connection: AdminConnection | None = None +) -> dict | None: """Return a provider snapshot, degrading silently when admin is unavailable.""" - status, payload = admin_get(host, port, "/airlock/admin/providers") + status, payload = admin_get( + host, port, "/airlock/admin/providers", connection=connection + ) return ( payload if status == 200 and isinstance(payload.get("providers"), dict) @@ -100,28 +205,44 @@ def provider_snapshot(host: str, port: str) -> dict | None: ) -def provider_configuration_snapshot(host: str, port: str) -> tuple[int, dict]: +def provider_configuration_snapshot( + host: str, port: str, *, connection: AdminConnection | None = None +) -> tuple[int, dict]: """Fetch the child-startup configuration view; never read local files.""" - return admin_get(host, port, "/airlock/admin/config/providers") + return admin_get( + host, port, "/airlock/admin/config/providers", connection=connection + ) -def session_snapshot(host: str, port: str) -> dict | None: +def session_snapshot( + host: str, port: str, *, connection: AdminConnection | None = None +) -> dict | None: """Return the bounded live affinity view, or None when admin is unavailable.""" - status, payload = admin_get(host, port, "/airlock/admin/sessions") + status, payload = admin_get( + host, port, "/airlock/admin/sessions", connection=connection + ) return ( payload if status == 200 and isinstance(payload.get("sessions"), list) else None ) -def client_snapshot(host: str, port: str) -> dict | None: - status, payload = admin_get(host, port, "/airlock/admin/clients") +def client_snapshot( + host: str, port: str, *, connection: AdminConnection | None = None +) -> dict | None: + status, payload = admin_get( + host, port, "/airlock/admin/clients", connection=connection + ) return ( payload if status == 200 and isinstance(payload.get("clients"), dict) else None ) -def telemetry_snapshot(host: str, port: str) -> dict | None: - status, payload = admin_get(host, port, "/airlock/admin/telemetry") +def telemetry_snapshot( + host: str, port: str, *, connection: AdminConnection | None = None +) -> dict | None: + status, payload = admin_get( + host, port, "/airlock/admin/telemetry", connection=connection + ) return ( payload if status == 200 and isinstance(payload.get("exporters"), dict) @@ -148,20 +269,33 @@ def operational_view(host: str, port: str, kind: str, body: dict) -> dict | None return payload if status == 200 else None -def clear_client_sessions(host: str, port: str, client_id: str) -> tuple[int, dict]: +def clear_client_sessions( + host: str, port: str, client_id: str, *, connection: AdminConnection | None = None +) -> tuple[int, dict]: # Client IDs come from a header and can contain a slash. Use an opaque # URL-safe selector so a path parser can never split the selected identity. selector = base64.urlsafe_b64encode(client_id.encode("utf-8")).decode("ascii") selector = selector.rstrip("=") - return admin_post(host, port, f"/airlock/admin/session-clients/{selector}/clear") + return admin_post( + host, + port, + f"/airlock/admin/session-clients/{selector}/clear", + connection=connection, + ) def clear_provider_quarantine( - host: str, port: str, provider: str, mode: str = "probe" + host: str, + port: str, + provider: str, + mode: str = "probe", + *, + connection: AdminConnection | None = None, ) -> tuple[int, dict]: return admin_post( host, port, f"/airlock/admin/providers/{provider}/clear-quarantine", {"mode": mode}, + connection=connection, ) diff --git a/airlock/tui/remote_app.py b/airlock/tui/remote_app.py new file mode 100644 index 0000000..024741d --- /dev/null +++ b/airlock/tui/remote_app.py @@ -0,0 +1,128 @@ +"""Restricted host-console UI for the explicit remote Admin profile. + +This intentionally does not reuse the full local dashboard: it owns no proxy, +does not read host files or state, and speaks only to the Admin HTTP perimeter. +""" + +from __future__ import annotations + +from pathlib import Path + +from textual import work +from textual.app import App, ComposeResult +from textual.containers import Vertical +from textual.widgets import Button, Footer, Header, Input, Static + +from airlock.tui.admin_client import ( + AdminConnection, + AdminConnectionError, + clear_provider_quarantine, + client_snapshot, + provider_configuration_snapshot, + provider_snapshot, +) + + +class RemoteAdminApp(App): + """Small, capability-scoped remote Admin viewer and reversible action UI.""" + + TITLE = "Airlock Remote Admin" + SUB_TITLE = "TLS capability connection" + + def __init__(self, connection: AdminConnection) -> None: + super().__init__() + self._connection = connection + + def compose(self) -> ComposeResult: + yield Header(show_clock=True) + with Vertical(): + yield Static("Connecting to remote Admin…", id="remote-admin-status") + yield Static("", id="remote-admin-providers") + yield Static("", id="remote-admin-clients") + yield Static("", id="remote-admin-config") + yield Input( + placeholder="Provider to clear quarantine", id="remote-provider" + ) + yield Button("Clear quarantine", id="remote-clear", variant="warning") + yield Footer() + + def on_mount(self) -> None: + self._refresh() + self.set_interval(15.0, self._refresh) + + @work(thread=True, group="remote-admin-refresh", exclusive=True) + def _refresh(self) -> None: + connection = self._connection + providers = provider_snapshot("", "", connection=connection) + clients = client_snapshot("", "", connection=connection) + config_status, config = provider_configuration_snapshot( + "", "", connection=connection + ) + + if providers is None: + self.call_from_thread( + self._set_status, + "Remote Admin unavailable (TLS, token, scope, or service).", + ) + return + self.call_from_thread( + self._render, providers, clients, config if config_status == 200 else None + ) + + def _render( + self, providers: dict, clients: dict | None, config: dict | None + ) -> None: + self.query_one("#remote-admin-status", Static).update( + "Remote Admin connected — TLS capability access." + ) + names = ", ".join(sorted(providers.get("providers", {}))) or "none" + self.query_one("#remote-admin-providers", Static).update(f"Providers: {names}") + client_count = len((clients or {}).get("clients", {})) + self.query_one("#remote-admin-clients", Static).update( + f"Clients: {client_count}" + ) + configured = len((config or {}).get("providers", [])) + self.query_one("#remote-admin-config", Static).update( + f"Configured providers: {configured}" + if config + else "Configured providers unavailable." + ) + + def _set_status(self, message: str) -> None: + self.query_one("#remote-admin-status", Static).update(message) + + def on_button_pressed(self, event: Button.Pressed) -> None: + if event.button.id != "remote-clear": + return + provider = self.query_one("#remote-provider", Input).value.strip() + if not provider: + self.query_one("#remote-admin-status", Static).update( + "Enter a provider name." + ) + return + self._clear(provider) + + @work(thread=True, group="remote-admin-clear", exclusive=True) + def _clear(self, provider: str) -> None: + status, _payload = clear_provider_quarantine( + "", "", provider, connection=self._connection + ) + message = ( + f"Cleared quarantine for {provider}." + if status == 200 + else "Quarantine clear unavailable (token scope or service)." + ) + self.call_from_thread(self._set_status, message) + if status == 200: + self._refresh() + + +def run(*, host: str, port: str, token_file: str, ca_file: str) -> None: + """Create the secret-blind remote transport and launch the limited UI.""" + try: + connection = AdminConnection.from_files( + host, port, Path(token_file), Path(ca_file) + ) + except AdminConnectionError as exc: + raise SystemExit(f"error: {exc}") from None + RemoteAdminApp(connection).run() diff --git a/dev/plans/0.5.15/30-host-console-tui-container-control-plane-design.md b/dev/plans/0.5.15/30-host-console-tui-container-control-plane-design.md new file mode 100644 index 0000000..1931c0e --- /dev/null +++ b/dev/plans/0.5.15/30-host-console-tui-container-control-plane-design.md @@ -0,0 +1,255 @@ +# 0.5.15 design plan — secure host-console TUI administration of containerized Airlock + +**Status:** ratified for Slice 50 implementation on 2026-08-16. The release +implements only the host-console topology below, behind an explicit +`admin.remote_tui` profile. It does not make a deployment-default change. + +## Slice 50 ratification + +The owner authorized implementation after the Slice 50 scope audit. The +following narrow contract resolves the prior B4 admission gate: + +- Only a host-console TUI reaching an Airlock container through a published + `127.0.0.1` host port is supported. IPv6 publication and a peer TUI + container remain deferred. +- The profile is default-off. It requires native TLS, `admin.enabled: true`, + `admin.trust_loopback: false`, and a non-loopback Airlock bind. Docker NAT + is therefore always treated as remote, not as an operator identity. +- Remote TUI access uses only a CA/name-validated HTTPS connection and a + protected token file. The capability token has a 15-minute ceiling and is + restricted by the `admin:remote_tui` anchor plus `admin:read`, + `admin:read_config`, and `admin:clear_quarantine`. The master key is never + accepted for this profile. +- The remote UI is a small Admin-HTTP-only view. It does not start/manage the + proxy, read host JSONL/FathomDB/configuration, edit configuration, access + MCP, or show operational history. Force quarantine and erasure remain + loopback-only. +- Successful remote mutations retain the token subject as actor and record the + non-secret `remote_tui_jwt` auth context. No raw IP, header, token, or + certificate material is audited. +- A separate hardened Compose manifest is opt-in and binds only the host + loopback port. The existing Compose file remains unchanged and does not + enable Admin. + +Deferred: peer-container topology, mTLS/reverse proxies, forwarded identity, +Docker/CIDR trust, remote history, configuration/virtual-key CRUD, and any +all-interface Admin publication. + +## Decision framing + +Airlock's Admin API currently authorizes a loopback connection as the operator +(Path A), and the separate-process TUI uses that path. In a Docker deployment, +the host-console TUI reaches the container through a published port. The +connection source observed inside the container is a Docker/network address, +not `127.0.0.1`; a peer container is likewise not loopback. Docker's bridge +membership or a host-loopback port publication is an exposure boundary, **not** +an authentication fact. + +The proposed, deliberately narrow future direction is a **direct, TLS-protected +Admin API with short-lived scoped capability credentials**. It reuses Airlock's +existing Admin PDP and audit record rather than adding a second control plane. +The first release would support the two explicitly configured topologies below; +all other container-network sources remain untrusted. + +1. A host-console `airlock tui` connects to an Airlock container port published + only on `127.0.0.1`/`::1`. Airlock runs native TLS, sets + `admin.trust_loopback: false`, and the TUI presents a short-lived scoped + capability token and validates a configured CA/hostname. +2. A separately deployed, explicitly approved operator/TUI container connects + over an isolated bridge network using the same TLS + scoped-token protocol. + The network limits reachability; it does not grant authorization. An + unknown peer, even on that network, has no access without a valid token. + +The safe current patterns remain a native Airlock/TUI pair or a TUI in the +Airlock container. This document does not make `network_mode: host`, Docker +socket access, bridge-address allowlisting, self-signed-certificate bypass, or +an inference `AIRLOCK_MASTER_KEY` a supported substitute. + +## Current-state evidence and constraints + +| Existing seam | Evidence | Constraint carried forward | +| --- | --- | --- | +| One Admin PDP | `airlock/admin/policy.py` evaluates loopback, master key, then exact-scope JWT. | Extend scopes and principal facts; do not create TUI-only authorization. | +| One Admin perimeter | `airlock/admin/http.py` extracts `scope["client"]` and mounts ahead of LiteLLM routes. | A Docker source address must remain non-loopback and fail closed. | +| TUI is a client | `airlock/tui/admin_client.py` uses HTTP, presently unauthenticated only for loopback. | Move transport/auth behavior here; never mount Textual in the proxy process. | +| FathomDB ownership | Operational history is proxy-owned and current endpoints are loopback-only. | TUI/container processes never open the embedded store; any remote history view must be separately authorized, bounded, and source-labelled. | +| Existing security posture | Non-loopback bearer Admin refuses plaintext startup unless TLS/proxy assertion or an explicit unsafe override is selected. | Do not weaken this check or treat a Docker network as TLS. | + +Docker documents that `-p` exposes a container port through a host mapping and +that binding the published host port to `127.0.0.1` restricts reachability to the +host; it does not say the downstream container receives a loopback peer. Its +bridge model also permits communication among containers on the same network. +This supports the design's separation of reachability from identity. + +## Design requirements + +### Functional requirements + +- **DCR-1 — explicit topology.** Operators can select only the two named + remote-control topologies through a reviewed configuration/deployment + profile. A port mapping or container source address alone never creates an + operator principal. +- **DCR-2 — transport authentication.** A non-process-loopback Admin request + requires TLS with certificate validation and a bearer capability JWT. The + TUI must not disable TLS verification merely because its destination hostname + is `localhost`. +- **DCR-3 — least privilege.** Read snapshots, live-protection mutations, + operational history, and destructive erasure each use distinct explicit + scopes. The TUI asks only for the scopes required by the enabled screen/action; + it never sends the master key or an inference credential. +- **DCR-4 — bounded reads.** Remote operational history, if ratified, remains + proxy-owned, capped, paged/bounded, source-labelled, and visibly degraded. + It must not silently switch from FathomDB to JSONL or disclose request + content in a denial response. +- **DCR-5 — credential lifecycle.** Tokens have a maximum TTL, an identifiable + non-secret subject, rotation/revocation instructions, and an operator-provided + local secret-file mechanism. Neither token values nor TLS private material + appear in config, process diagnostics, TUI snapshots, URLs, logs, or audit + targets. +- **DCR-6 — auditable mutations.** Successful mutations retain the existing + `admin_action` contract and add a stable auth/topology descriptor (for + example `token_remote`), never an IP address or bearer material by default. +- **DCR-7 — failure behavior.** Missing/expired/wrong-scope credentials return + the existing bounded 401/403 JSON shape. TLS failure, unavailable Admin API, + and unavailable operational reads are presented to the TUI as unavailable, + without retry storms or inference-path effects. + +### Security and compatibility requirements + +- No control endpoint becomes reachable on an all-interface published port by + default. The standard Compose deployment remains inference-only in behavior. +- `trust_loopback` remains exactly process-network-loopback semantics. There is + no `trust_docker_bridge`, `trust_host_gateway`, or CIDR-based operator mode. +- The inference request path, model routing, guardrails, provider credentials, + LiteLLM virtual-key behavior, and JSONL record format remain unchanged. +- Existing native-TUI Path A behavior remains backward compatible when this + future profile is disabled. +- A reverse proxy may be considered only as an operator-managed deployment + integration; Airlock must not accept forwarded identity headers unless a + separately designed, mutually authenticated proxy channel establishes their + provenance. + +## Acceptance criteria for a future implementation proposal + +1. A host TUI can read the allowed snapshots and execute only an explicitly + granted live-state action through a `127.0.0.1`-published, TLS-enabled + container endpoint; valid certificate/CA verification is demonstrated. +2. The same host route with no token, expired token, wrong scope, plaintext, + or invalid CA/certificate fails without returning state or secret material. +3. An approved peer TUI container succeeds only with its scoped credential; + another container on the same bridge fails for all Admin and operational + endpoints. Test both IPv4 and configured IPv6 publication if enabled. +4. Operational history is either explicitly denied for remote clients or, if + the separate history scope is approved, is proxy-owned, bounded, source and + truncation labelled, and cannot disclose records on 401/403/404 paths. +5. Host-loopback publishing is verified as a reachability restriction, not an + authorization shortcut: `trust_loopback` does not admit the NATed request. +6. Every successful control mutation emits exactly one redacted audit record; + reads emit no mutation audit record; token, certificate, private key, + request content, and raw client IP do not enter test fixtures, JSONL, or TUI + snapshots. +7. Existing direct-loopback TUI, Admin JWT, plaintext refusal, and FathomDB + local-bridge regression suites retain their prior contracts. +8. A security reviewer signs off on threat model, token handling, certificate + provisioning, topology manifests, and rollback before enablement. + +## Proposed architecture (not an implementation instruction) + +``` +host TUI / approved TUI container + └─ TLS + scoped bearer token ─▶ Airlock AdminMiddleware + ├─ authenticated Principal + ├─ existing PDP (exact scope) + └─ existing proxy-owned handlers / audit writer + +Docker publish / isolated bridge: reachability only; never Principal.loopback +``` + +### Authorization matrix + +| Surface | Proposed scope / condition | Data/action boundary | +| --- | --- | --- | +| Provider/client/circuit/telemetry snapshots | `admin:read` | existing bounded live state | +| Clear quarantine/reset circuit/clear backoff | existing action scopes | existing state-only mutations + audit | +| Remote operational history (optional) | new `admin:operational_read`, explicitly enabled | bounded proxy-owned read; no direct DB | +| Erasure and manual force-quarantine | remain loopback-only unless a later design explicitly revises them | destructive/break-glass controls | +| TUI key management | out of scope; depends on 0.6.0 virtual-key authority | no key API exposure here | + +The token issuer remains the existing locally signing `airlock admin mint-token` +path, but the future proposal must define a non-secret token distribution +workflow (for example, a root-readable file delivered by the operator's secret +manager). A TUI never mints its own authority. TLS server identity is pinned to +an operator-managed CA/name rather than using the current loopback-only +self-signed verification bypass. + +### Alternatives researched + +| Alternative | Finding | Decision | +| --- | --- | --- | +| Direct TLS + scoped JWT | Fits the existing Admin policy, token signer, and audit seam; no second proxy or process needed. | **Recommended design baseline.** | +| Host-local trusted reverse proxy | Can work only with a mutually authenticated isolated upstream channel and strict forwarded-header handling. It adds an identity translation and operational lifecycle. | Defer; require a dedicated threat model if a user needs SSO/proxy integration. | +| Docker bridge/CIDR/host-gateway trust | A bridge peer can be another container; NAT source is not cryptographic identity. | Rejected. | +| `network_mode: host` | Removes network namespace isolation and Docker ignores port publication in host mode. | Rejected as an authorization solution. | +| Reuse `AIRLOCK_MASTER_KEY` | Broad inference/admin authority and poor rotation/audit granularity. | Rejected. | + +## Web research and GitHub-library assessment + +| Source/library | What it establishes or could do | Airlock alignment / decision | +| --- | --- | --- | +| [Docker port publishing](https://docs.docker.com/engine/network/port-publishing/) | `127.0.0.1` publication limits host reachability; published ports otherwise can be exposed; bridge/direct routing caveats matter. | Deployment evidence; no runtime dependency. | +| [Docker bridge networks](https://docs.docker.com/engine/network/drivers/bridge/) | Bridge networks are connectivity domains, not an operator identity mechanism. | Supports the no-CIDR-trust rule. | +| [Docker host networking](https://docs.docker.com/engine/network/drivers/host/) | Host networking shares the host network namespace and makes port mappings ineffective. | Explicitly not a supported authority shortcut. | +| [BerriAI/litellm](https://github.com/BerriAI/litellm) | Existing substrate already hosts Airlock's ASGI Admin middleware. | Partial, existing dependency only; do not delegate Airlock Admin policy to LiteLLM. | +| [Traefik](https://github.com/traefik/traefik) | Could terminate TLS/perform forward auth for an operator-managed edge. | Not adopted: extra control-plane hop, header-trust risks, and broader deployment blast radius; retain only as a later integration option. | +| [Envoy](https://github.com/envoyproxy/envoy) | Could provide mTLS/proxy policy. | Not adopted: operationally disproportionate for a single-container default and would duplicate Airlock's PDP boundary. | + +The relevant upstream security warning is concrete: Traefik has recently fixed +forward-auth header-handling issues. That does not prohibit it, but reinforces +that a proxy path cannot be treated as a free identity proof and needs a +versioned, separately reviewed deployment contract. + +## Architectural alignment and blast radius + +**Alignment verdict: conditionally aligned.** The proposal keeps Airlock's +authority and state ownership where they are: the proxy Admin perimeter is the +policy-enforcement point; the existing PDP decides exact scopes; the TUI stays a +pulling client; the proxy remains the sole FathomDB owner; JSONL remains a +bounded fallback; and the inference hot path receives no TUI dependency. It is +not aligned if it adds Docker-source trust, an unaudited proxy identity header, +or a parallel admin/key store. + +| Area | Expected future change | Risk / blast radius | Mitigation | +| --- | --- | --- | --- | +| Admin policy and HTTP perimeter | remote token principal, new optional history scope | High: authorization boundary | pure PDP tests, default-off profile, security review | +| TUI Admin client | token-file/CA loading and accurate unavailable state | Medium: operator UX and secret handling | no env echo, no verification bypass, snapshot tests | +| Deployment docs/Compose | opt-in TLS/topology examples | High: accidental endpoint exposure | separate hardened profile, bind-address assertions, rollback instructions | +| Operational reads | optional scoped remote view | High: content disclosure | default deny, bounds/source labels, no direct DB | +| Inference/guardrails/routing | none | Must be zero | regression and request-path performance checks | + +## Rollout, verification, and rollback plan + +1. **Pre-implementation HITL:** approve threat model, exact scopes, certificate + trust/distribution, token delivery, and whether remote history is allowed. +2. **Red:** write unit/perimeter tests for both topologies and every negative + case before a transport/client change. Add Compose integration tests with a + hostile bridge peer and a host TUI process. +3. **Green:** enable only a dedicated disabled-by-default profile, with an + explicit bind address, TLS material references, token-file path, and no + credential values in YAML. Preserve existing native loopback tests. +4. **Canary:** an operator validates a non-sensitive snapshot first, then one + reversible protection action, and verifies a redacted audit record. Do not + use production request history as a smoke test. +5. **Rollback:** disable the profile/remove the published Admin mapping and + restart. Existing direct-process loopback access stays available. Rotate the + JWT signing secret or credential if compromise is suspected; certificate + rotation follows the operator CA process. + +## Open HITL questions + +- Is remote operational history necessary in 0.5.15, or should the remote TUI + initially expose only live snapshots and state controls? +- What secret-manager/file-permission contract is acceptable for a host TUI + token, and which CA is authoritative in each supported deployment? +- Is an operator-managed mTLS reverse-proxy integration a product requirement, + or should it stay an unsupported deployment composition until a separate + design is approved? diff --git a/dev/plans/0.5.15/50-host-console-container-admin-status.md b/dev/plans/0.5.15/50-host-console-container-admin-status.md new file mode 100644 index 0000000..361c509 --- /dev/null +++ b/dev/plans/0.5.15/50-host-console-container-admin-status.md @@ -0,0 +1,57 @@ +# Slice 50 — secure host-console container Admin TUI status + +**Status:** complete; independently reviewed and verified on 2026-08-16. + +## Scope disposition + +Slice 50 supersedes the draft's conditional two-topology proposal with the +ratified host-console-only profile. It is default-off and supports only a TUI +on the host reaching an Airlock container through a `127.0.0.1` published port, +native TLS, an owner-only token file, and a CA-validated capability JWT. + +Deferred: IPv6 and peer-container TUI deployment, operational history, +reverse-proxy/mTLS integrations, Docker/CIDR trust, forwarded identity, +configuration and virtual-key CRUD, and all-interface Admin publication. + +## Delivered + +- Added `admin.remote_tui`, startup validation, a 15-minute JWT ceiling, a + required `admin:remote_tui` anchor, exact allowed scopes, and a remote + master-key denial. Existing profile-off behavior is unchanged. +- Added the non-secret `remote_tui_jwt` audit context to successful remote + mutation records; the actor remains the verified JWT subject. +- Added a separate limited remote TUI and strict `--admin-token-file` / + `--admin-ca-file` transport. It always uses CA/name-validated HTTPS, + including `localhost`, and does not reuse the full local dashboard. +- Added a complete opt-in Compose manifest with loopback-only publication and + read-only TLS mounts. The existing Compose file is untouched. +- Updated requirements, design ratification, configuration reference, and + Admin/operator runbook including rotation, emergency revocation, and + rollback. + +## TDD and review evidence + +- RED: new PDP/profile tests initially failed for master-key admission, + unrestricted scopes/lifetime, and missing remote-profile validation. + GREEN: all focused Admin policy tests passed after implementation. +- RED: remote transport tests initially failed to import the new connection + boundary. GREEN: verified bearer injection, HTTPS/CA verification, + localhost hostname verification, protected token files, and no server TLS + environment fallback. +- RED/GREEN: remote CLI tests establish both-file preflight, no proxy start or + daemon ownership, and dispatch to the isolated remote UI. +- Structural Compose test asserts the only published mapping is host loopback + and TLS/config mounts are read-only. + +## Verification + +- Focused Admin, TUI transport/CLI, and Compose tests passed (135 tests); + separate contract/Compose checks passed (9 tests). +- Ruff check/format, strict MkDocs, Compose render with + `--no-env-resolution`, `git diff --check`, and `make sync && make verify` + passed. The standard Compose file has no diff. +- `make test` was running under the verifier at record time; non-PTY attempts + were truncated by the execution harness before yielding a result. A live + Docker-daemon topology test requires access to `/var/run/docker.sock`, which + this sandbox denies; structural Compose and TLS/PDP/client tests cover the + release contract without daemon access. diff --git a/dev/requirements.md b/dev/requirements.md index 2c2dc7f..bde7d0b 100644 --- a/dev/requirements.md +++ b/dev/requirements.md @@ -424,3 +424,27 @@ SHALL read this view only over its Admin HTTP client in a bounded background refresh, visibly label unavailable/stale startup state, and never fall back to another process's files. YAML plus deployment workflow and restart remain the configuration authority. + +### DFR-38 / DAC-38: Secure host-console container Admin TUI + +An explicit, default-off `admin.remote_tui` profile SHALL support only a host +console TUI connecting to a container through a host-loopback-published port. +It SHALL require native TLS, `admin.enabled: true`, `trust_loopback: false`, +and a CA/name-validated HTTPS client using an operator-owned protected token +file. The profile SHALL accept only a signed capability JWT, with a maximum +15-minute TTL and the `admin:remote_tui` anchor plus exact scope among +`admin:read`, `admin:read_config`, and `admin:clear_quarantine`; master-key, +loopback, Docker/CIDR, forwarded-header, +and proxy identity are not remote authorization paths. The remote UI SHALL use +only the Admin HTTP perimeter and SHALL not access local state, logs, FathomDB, +configuration files, proxy lifecycle, or operational history. + +The standard Compose deployment SHALL remain unchanged. A separate opt-in +manifest SHALL bind the published port only to `127.0.0.1`, mount TLS material +read-only, and document token/CA delivery, 15-minute rotation/revocation, and +rollback. Remote reads and failures SHALL be bounded and secret-blind. +Successful remote mutations SHALL retain the validated token subject and add +only `auth_context: remote_tui_jwt` to the existing `admin_action` audit +record; they SHALL not record bearer material, raw address, certificate data, +or request content. Existing direct-loopback Admin/TUI, operation-history, +inference, routing, guardrail, and credential behavior SHALL remain unchanged. diff --git a/docker-compose.remote-admin.yml b/docker-compose.remote-admin.yml new file mode 100644 index 0000000..a965ef9 --- /dev/null +++ b/docker-compose.remote-admin.yml @@ -0,0 +1,25 @@ +# Opt-in Slice 50 host-console profile. Do not combine this file with the +# standard deployment: it is a complete hardened alternative with a +# loopback-only published port, not an override for its default publication. +services: + airlock: + build: . + ports: + - "127.0.0.1:${AIRLOCK_PORT:-4000}:4000" + env_file: + - .env + volumes: + - ./config.yaml:/app/config.yaml:ro + - ./certs/server.crt:/run/airlock-tls/server.crt:ro + - ./certs/server.key:/run/airlock-tls/server.key:ro + - ./logs:/app/logs + environment: + AIRLOCK_HOST: 0.0.0.0 + AIRLOCK_SSL_CERTFILE: /run/airlock-tls/server.crt + AIRLOCK_SSL_KEYFILE: /run/airlock-tls/server.key + restart: unless-stopped + healthcheck: + test: ["CMD", "curl", "-fk", "https://localhost:4000/livez"] + interval: 30s + timeout: 5s + retries: 3 diff --git a/docs/getting-started/configuration.md b/docs/getting-started/configuration.md index a318347..0b595bb 100644 --- a/docs/getting-started/configuration.md +++ b/docs/getting-started/configuration.md @@ -549,6 +549,7 @@ admin: trust_loopback: true # loopback connections are the operator (Path A) allow_insecure_tokens: false # permit token auth over plaintext on a non-loopback bind behind_tls_proxy: false # assert TLS is terminated upstream + remote_tui: false # opt-in host-console container Admin profile ``` Tokens are signed with `AIRLOCK_JWT_SECRET` (falling back to a derivation from @@ -557,6 +558,12 @@ or capability skips are active on a non-loopback bind without TLS, Airlock refus to start unless one of `AIRLOCK_SSL_*`, `behind_tls_proxy`, or `allow_insecure_tokens` is set. +`remote_tui: true` is a narrower profile for a host-console TUI reaching an +Airlock container through a port published only on the host loopback interface. +It requires native TLS, `trust_loopback: false`, and rejects proxy/insecure-token +modes. It accepts only a 15-minute JWT containing `admin:remote_tui` plus its +operation scope; it never accepts the master key. See [Admin API](../guide/admin-api.md#host-console-container-tui) for the deployment contract. + ### `guardrail_overrides` Allows trusted clients to downgrade specific content guards per request via an diff --git a/docs/guide/admin-api.md b/docs/guide/admin-api.md index e92f587..8711c22 100644 --- a/docs/guide/admin-api.md +++ b/docs/guide/admin-api.md @@ -31,6 +31,7 @@ admin: | `trust_loopback` | `true` | A connection from `127.0.0.1`/`::1` is the operator tier, no credential (Path A). | | `allow_insecure_tokens` | `false` | Permit token auth on a non-loopback bind without TLS (downgrades the fail-closed startup refusal to a warning). | | `behind_tls_proxy` | `false` | Assert that TLS is terminated by an upstream reverse proxy, so the fail-closed check is satisfied. | +| `remote_tui` | `false` | Enable only the host-console container profile below; requires native TLS and capability JWTs. | ### Fail-closed on insecure token auth @@ -128,6 +129,60 @@ airlock admin mint-token --sub ci-bot \ > `--sub` **must** be the client's authenticated key-derived id (`key:`). > See [Guardrails → Per-request guardrail skips](guardrails.md#per-request-guardrail-skips). +## Host-console container TUI + +This opt-in profile is for a host console that administers an Airlock container. +It is not a general remote Admin deployment, a peer-container mode, or a +reverse-proxy integration. Docker reachability is not operator identity. + +Use the dedicated complete manifest, not an overlay with the standard Compose +file: `docker compose -f docker-compose.remote-admin.yml up`. It publishes only +`127.0.0.1:${AIRLOCK_PORT:-4000}:4000`, mounts native TLS server material +read-only, and leaves the normal Compose deployment unchanged. Configure: + +```yaml +admin: + enabled: true + trust_loopback: false + remote_tui: true + allow_insecure_tokens: false + behind_tls_proxy: false +``` + +Set both native TLS environment variables and provide a CA bundle on the host. +Mint a distinct, non-secret-subject token for each operator with a maximum +15-minute lifetime and all necessary scopes, including the anchor: + +```bash +airlock admin mint-token --sub host-console-alice \ + --scope admin:remote_tui --scope admin:read \ + --scope admin:read_config --scope admin:clear_quarantine --ttl 15m +``` + +Place the emitted token in an owner-only (`0600`) regular file delivered by your +secret manager, then run: + +```bash +airlock tui --remote-admin --host localhost --port 4000 \ + --admin-token-file /secure/airlock-remote.jwt \ + --admin-ca-file /secure/airlock-ca.pem +``` + +The remote UI validates the hostname even for `localhost`, uses only the Admin +HTTP perimeter, and is restricted to snapshots plus clear-quarantine. It never +reads local logs/configuration/state, manages a proxy, or accesses operational +history; force quarantine and erasure remain loopback-only. Failed TLS, +authentication, and scope checks show the console as unavailable without +displaying credentials. + +For routine rotation, issue a new short-lived token. For an emergency +revocation, replace `AIRLOCK_JWT_SECRET`, omit `AIRLOCK_JWT_SECRET_PREV`, and +restart the proxy; this invalidates all tokens signed with the old secret. To +roll back the profile, stop the dedicated Compose deployment (or remove its +loopback mapping), remove `remote_tui: true`, and restart. Do not publish this +Admin port to all interfaces or substitute the master key, a Docker bridge, a +CIDR, or forwarded headers for the capability token. + ## Endpoints All routes are under `/airlock/admin/` and only exist when `admin.enabled: true`. diff --git a/tests/test_admin_http.py b/tests/test_admin_http.py index 021f447..0c58cd0 100644 --- a/tests/test_admin_http.py +++ b/tests/test_admin_http.py @@ -67,6 +67,40 @@ def test_trust_loopback_off_falls_through(self): d = decide(Principal(loopback=True), "admin:clear_quarantine") # no bearer assert not d.allowed and d.status == 401 + def test_remote_tui_requires_anchor_scope_and_rejects_master_key(self): + configure_admin( + {"admin": {"enabled": True, "trust_loopback": False, "remote_tui": True}}, + host="0.0.0.0", + tls_enabled=True, + ) + master = decide(Principal(bearer="master-key-supersecret-123456"), "admin:read") + assert not master.allowed and master.status == 403 + loopback_master = decide( + Principal(loopback=True, bearer="master-key-supersecret-123456"), + "admin:read", + ) + assert not loopback_master.allowed and loopback_master.status == 403 + ordinary = mint_token("operator:one", ["admin:read"], 60) + assert not decide(Principal(bearer=ordinary), "admin:read").allowed + remote = mint_token("operator:one", ["admin:remote_tui", "admin:read"], 60) + decision = decide(Principal(bearer=remote), "admin:read") + assert decision.allowed + assert decision.actor == "operator:one" + assert decision.auth_context == "remote_tui_jwt" + + def test_remote_tui_rejects_unallowed_scope_and_long_lived_token(self): + configure_admin( + {"admin": {"enabled": True, "trust_loopback": False, "remote_tui": True}}, + host="0.0.0.0", + tls_enabled=True, + ) + wrong = mint_token( + "operator:one", ["admin:remote_tui", "admin:reset_circuit"], 60 + ) + assert not decide(Principal(bearer=wrong), "admin:reset_circuit").allowed + long_lived = mint_token("operator:one", ["admin:remote_tui", "admin:read"], 901) + assert not decide(Principal(bearer=long_lived), "admin:read").allowed + class TestConfigureAdminFailClosed: def test_exposed_no_tls_raises(self): @@ -89,6 +123,64 @@ def test_loopback_ok(self): def test_disabled_no_check(self): configure_admin({"admin": {"enabled": False}}, host="0.0.0.0") # no raise + @pytest.mark.parametrize( + "config,host,tls", + [ + ({"admin": {"enabled": True, "remote_tui": True}}, "0.0.0.0", True), + ( + { + "admin": { + "enabled": True, + "trust_loopback": False, + "remote_tui": True, + } + }, + "0.0.0.0", + False, + ), + ( + { + "admin": { + "enabled": True, + "trust_loopback": False, + "remote_tui": True, + "behind_tls_proxy": True, + } + }, + "0.0.0.0", + True, + ), + ( + { + "admin": { + "enabled": True, + "trust_loopback": False, + "remote_tui": True, + "allow_insecure_tokens": True, + } + }, + "0.0.0.0", + True, + ), + ( + { + "admin": { + "enabled": True, + "trust_loopback": False, + "remote_tui": True, + } + }, + "127.0.0.1", + True, + ), + ], + ) + def test_remote_tui_profile_rejects_insecure_or_ambiguous_startup( + self, config, host, tls + ): + with pytest.raises(RuntimeError, match="remote_tui"): + configure_admin(config, host=host, tls_enabled=tls) + # ------------------------------------------------------ handle_admin_request class TestHandleAdminRequest: @@ -116,6 +208,36 @@ def test_get_providers(self, fresh_state_store): assert s == 200 assert body["providers"]["openai"]["quarantined"] is True + def test_remote_tui_mutation_has_only_validated_actor_and_auth_context( + self, fresh_state_store, monkeypatch + ): + configure_admin( + {"admin": {"enabled": True, "trust_loopback": False, "remote_tui": True}}, + host="0.0.0.0", + tls_enabled=True, + ) + token = mint_token( + "operator:one", + ["admin:remote_tui", "admin:clear_quarantine"], + 60, + ) + written = [] + monkeypatch.setattr( + "airlock.admin.http.write_admin_action_record", written.append + ) + status, _body, _headers = handle_admin_request( + "POST", + "/airlock/admin/providers/openai/clear-quarantine", + b'{"mode":"probe"}', + Principal(bearer=token, actor="untrusted-header"), + ) + assert status == 200 + assert len(written) == 1 + assert written[0]["actor"] == "operator:one" + assert written[0]["auth_context"] == "remote_tui_jwt" + assert token not in json.dumps(written[0]) + assert "untrusted-header" not in json.dumps(written[0]) + def test_provider_configuration_requires_distinct_scope_and_is_no_store(self): configure_provider_configuration( { diff --git a/tests/test_proxy_bootstrap.py b/tests/test_proxy_bootstrap.py index 8a7d9f6..204f5c0 100644 --- a/tests/test_proxy_bootstrap.py +++ b/tests/test_proxy_bootstrap.py @@ -5,6 +5,8 @@ from __future__ import annotations +import pytest + import airlock.proxy_bootstrap as pb from airlock.admin.policy import admin_enabled from airlock.provider_configuration import provider_configuration_snapshot @@ -65,6 +67,7 @@ def test_child_configuration_uses_runtime_file_with_litellm_nested_include_order runtime = tmp_path / "runtime.yaml" runtime.write_text("include: [direct.yaml]\nmodel_list: []\n") monkeypatch.setenv("AIRLOCK_CONFIG", str(runtime)) + monkeypatch.setenv("AIRLOCK_HOST", "127.0.0.1") pb._configure_child_startup_configuration() @@ -73,3 +76,15 @@ def test_child_configuration_uses_runtime_file_with_litellm_nested_include_order provider_configuration_snapshot()["providers"][0]["aliases"][0]["alias"] == "direct" ) + + +def test_child_rejects_an_invalid_remote_tui_profile(tmp_path, monkeypatch): + runtime = tmp_path / "runtime.yaml" + runtime.write_text("admin:\n enabled: true\n remote_tui: true\n") + monkeypatch.setenv("AIRLOCK_CONFIG", str(runtime)) + monkeypatch.setenv("AIRLOCK_HOST", "0.0.0.0") + monkeypatch.setenv("AIRLOCK_SSL_CERTFILE", "/cert") + monkeypatch.setenv("AIRLOCK_SSL_KEYFILE", "/key") + + with pytest.raises(RuntimeError, match="remote_tui"): + pb._configure_child_startup_configuration() diff --git a/tests/test_remote_admin_compose.py b/tests/test_remote_admin_compose.py new file mode 100644 index 0000000..575c0a0 --- /dev/null +++ b/tests/test_remote_admin_compose.py @@ -0,0 +1,26 @@ +"""Structural guard for the opt-in host-console remote Admin Compose profile.""" + +from __future__ import annotations + +from pathlib import Path + +import yaml + + +def test_remote_admin_compose_is_complete_and_loopback_only(): + root = Path(__file__).parents[1] + profile = yaml.safe_load((root / "docker-compose.remote-admin.yml").read_text()) + service = profile["services"]["airlock"] + assert service["ports"] == ["127.0.0.1:${AIRLOCK_PORT:-4000}:4000"] + assert service["environment"] == { + "AIRLOCK_HOST": "0.0.0.0", + "AIRLOCK_SSL_CERTFILE": "/run/airlock-tls/server.crt", + "AIRLOCK_SSL_KEYFILE": "/run/airlock-tls/server.key", + } + assert ( + "docker-compose.yml" + not in (root / "docker-compose.remote-admin.yml").read_text() + ) + assert "./config.yaml:/app/config.yaml:ro" in service["volumes"] + assert "./certs/server.crt:/run/airlock-tls/server.crt:ro" in service["volumes"] + assert "./certs/server.key:/run/airlock-tls/server.key:ro" in service["volumes"] diff --git a/tests/test_tui.py b/tests/test_tui.py index 98056bd..ab57065 100644 --- a/tests/test_tui.py +++ b/tests/test_tui.py @@ -455,6 +455,53 @@ def test_tui_daemon_flag_passed_through() -> None: ) +def test_remote_tui_requires_both_secret_files() -> None: + from airlock.cli.main import main + + with pytest.raises(SystemExit) as exc_info: + main(["tui", "--remote-admin", "--admin-token-file", "/tmp/token"]) + assert exc_info.value.code == 2 + + +def test_remote_tui_uses_isolated_remote_app() -> None: + from airlock.cli.main import main + + with mock.patch("airlock.tui.remote_app.run") as remote_run: + main( + [ + "tui", + "--remote-admin", + "--host", + "localhost", + "--admin-token-file", + "/tmp/token", + "--admin-ca-file", + "/tmp/ca.pem", + ] + ) + remote_run.assert_called_once_with( + host="localhost", port="4000", token_file="/tmp/token", ca_file="/tmp/ca.pem" + ) + + +def test_remote_tui_rejects_proxy_start() -> None: + from airlock.cli.main import main + + with pytest.raises(SystemExit) as exc_info: + main( + [ + "tui", + "--remote-admin", + "--start", + "--admin-token-file", + "/tmp/token", + "--admin-ca-file", + "/tmp/ca.pem", + ] + ) + assert exc_info.value.code == 2 + + async def test_app_has_proxy_manager() -> None: app = AirlockApp(host="127.0.0.1", port="9999") from airlock.tui.proxy_manager import ProxyManager diff --git a/tests/test_tui_admin_client.py b/tests/test_tui_admin_client.py index 6837ddd..e599cad 100644 --- a/tests/test_tui_admin_client.py +++ b/tests/test_tui_admin_client.py @@ -4,10 +4,16 @@ import io import json +import os +import ssl import urllib.error from unittest.mock import patch +import pytest + from airlock.tui.admin_client import ( + AdminConnection, + AdminConnectionError, _scheme_and_context, admin_post, clear_provider_quarantine, @@ -107,3 +113,71 @@ def read(self): with patch("urllib.request.urlopen", return_value=_BadResp(200, {})): status, payload = admin_post("127.0.0.1", "4000", "/p") assert status == 200 and "error" in payload # status preserved + + +class TestRemoteAdminConnection: + def _token_file(self, tmp_path): + token = tmp_path / "remote.jwt" + token.write_text("remote-token\n") + token.chmod(0o600) + return token + + def test_remote_connection_uses_ca_verified_https_and_bearer(self, tmp_path): + token = self._token_file(tmp_path) + ca = tmp_path / "ca.pem" + ca.write_text("not-a-real-ca") + with patch("ssl.create_default_context") as context: + context.return_value = ssl.create_default_context() + connection = AdminConnection.from_files("localhost", "4000", token, ca) + assert connection.host == "localhost" + assert connection.ssl_context.verify_mode == ssl.CERT_REQUIRED + assert connection.ssl_context.check_hostname is True + with patch( + "urllib.request.urlopen", return_value=_FakeResp(200, {"ok": True}) + ) as request: + status, _ = admin_post("ignored", "0", "/p", connection=connection) + assert status == 200 + sent = request.call_args[0][0] + assert sent.full_url == "https://localhost:4000/p" + assert sent.get_header("Authorization") == "Bearer remote-token" + + @pytest.mark.parametrize("mode", [0o644, 0o640]) + def test_remote_connection_rejects_over_permissive_token_file(self, tmp_path, mode): + token = self._token_file(tmp_path) + token.chmod(mode) + with pytest.raises(AdminConnectionError, match="permissions"): + AdminConnection.from_files("localhost", "4000", token, tmp_path / "ca.pem") + + def test_remote_connection_rejects_empty_token(self, tmp_path): + token = tmp_path / "remote.jwt" + token.write_text("\n") + token.chmod(0o600) + with pytest.raises(AdminConnectionError, match="empty"): + AdminConnection.from_files("localhost", "4000", token, tmp_path / "ca.pem") + + def test_remote_connection_rejects_header_unsafe_token(self, tmp_path): + token = tmp_path / "remote.jwt" + token.write_text("bad\nvalue") + token.chmod(0o600) + with pytest.raises(AdminConnectionError, match="invalid"): + AdminConnection.from_files("localhost", "4000", token, tmp_path / "ca.pem") + + def test_remote_connection_refuses_a_token_symlink(self, tmp_path): + target = self._token_file(tmp_path) + token = tmp_path / "linked.jwt" + token.symlink_to(target) + with pytest.raises(AdminConnectionError, match="regular"): + AdminConnection.from_files("localhost", "4000", token, tmp_path / "ca.pem") + + def test_remote_connection_never_uses_server_tls_environment( + self, tmp_path, monkeypatch + ): + token = self._token_file(tmp_path) + ca = tmp_path / "ca.pem" + ca.write_text("not-a-real-ca") + monkeypatch.setenv("AIRLOCK_SSL_CERTFILE", "/server/cert") + monkeypatch.setenv("AIRLOCK_SSL_KEYFILE", "/server/key") + with patch("ssl.create_default_context") as context: + context.return_value = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT) + AdminConnection.from_files("localhost", "4000", token, ca) + context.assert_called_once_with(cafile=os.fspath(ca)) From 78b9466026c3fb5366c52bc575e895ea9d093e06 Mon Sep 17 00:00:00 2001 From: Corey Thompson Date: Sun, 16 Aug 2026 17:23:12 -0500 Subject: [PATCH 13/24] docs(plan): defer virtual keys to 0.6.0 --- dev/plans/0.5.15-todo.md | 226 +++++++++++++++ dev/plans/0.5.15/06-hitl-release-review.md | 107 +++++++ ...tual-key-management-0.6-contract-design.md | 261 ++++++++++++++++++ dev/plans/0.5.15/README.md | 4 +- 4 files changed, 596 insertions(+), 2 deletions(-) create mode 100644 dev/plans/0.5.15-todo.md create mode 100644 dev/plans/0.5.15/06-hitl-release-review.md create mode 100644 dev/plans/0.5.15/40-virtual-key-management-0.6-contract-design.md diff --git a/dev/plans/0.5.15-todo.md b/dev/plans/0.5.15-todo.md new file mode 100644 index 0000000..c543c74 --- /dev/null +++ b/dev/plans/0.5.15-todo.md @@ -0,0 +1,226 @@ +# Airlock 0.5.15 — TODO + +## Accepted 0.5.15 feature scope + +The owner has accepted these packages into 0.5.15 implementation planning: + +1. Airlock-native virtual-key store and virtual-key management. +2. Provider/configuration visibility or CRUD in the authenticated Admin/TUI + control plane. +3. A startup warning when a configured provider credential has no enabled + provider alias. +4. Secure host-console TUI administration for containerized Airlock. +5. A single TUI admin console that can safely manage multiple Airlock instances, + applying either instance-specific or explicitly approved system-wide + configuration changes. + +This releases the packages from *scope-selection* limbo; it does **not** waive +their documented security, architecture, and HITL design gates. The initial +provider slice remains read-only unless a separate durable configuration-owner +decision approves CRUD. Claude Code subscription-preserving pass-through is +out of scope for 0.5.15. + +Fleet management begins as a control-plane design/research package. Each +instance remains authoritative for its local runtime state and enforces its own +Admin policy; the TUI must not gain raw SSH, Docker-socket, or systemd control. +Whether a reviewed desired-state system may own instance or system-wide +configuration is a separate architecture/HITL decision. + +## Accepted repair: typed threat-backoff response + +**Status:** accepted 0.5.15 implementation scope. + +Airlock's Fast Guardian threat gate currently raises a plain `ValueError` when +it places a client in, or finds it already in, threat backoff. LiteLLM then +serializes that Airlock decision as an untyped HTTP 500 (`type: None`, +`code: 500`) instead of a rate limit. This is distinct from the provider +circuit-breaker contract covered by UN-16: the provider is not necessarily +rate-limiting the request. + +Implement an Airlock-owned, typed threat-backoff exception and proxy handler. +Both the request that creates the backoff and later requests rejected while it +remains active must return an OpenAI-compatible HTTP 429 with an accurate +`Retry-After` header. The body must use a stable Airlock-specific type/code and +identify the source as `threat_backoff`, without disclosing heuristic details, +client identity, or request content. + +### Acceptance criteria + +1. The Fast Guardian never raises a plain `ValueError` for threat backoff; it + raises a typed Airlock rate-limit exception instead. +2. The proxy maps that exception to HTTP 429, a whole-second `Retry-After`, and + an OpenAI-shaped error body with stable `type`, `code`, and an `airlock` + source of `threat_backoff`. +3. The response distinguishes this local protection from both a provider 429 + and an Airlock provider circuit-breaker quarantine, without relying on + message parsing. +4. Focused tests cover the initial block, an active-backoff retry, response + headers/body, and preservation of existing provider and admission 429 + contracts. +5. Documentation describes the contract and tells clients to honor + `Retry-After`; it must not characterize a threat-backoff response as a + DeepSeek/provider rate limit. + +## Accepted scope: provider configuration in Admin and the TUI + +**Status:** accepted 0.5.15 implementation-planning scope. The approved design +baseline is read-only visibility; CRUD remains separately gated on durable +configuration ownership, security review, and HITL. + +Airlock now has explicit configured-provider aliases, but configuration remains +a reviewed file-and-restart operation. Evaluate whether operators can safely +inspect the provider/model list and perform provider configuration through the +authenticated Admin control plane and the host-console TUI. + +### Design and implementation gates + +1. Inventory what the Admin API and TUI already expose about configured + providers, aliases, stable bases, credential *presence*, health, and + discovery. Verify source, freshness, authorization, and redaction. +2. Decide separately whether the initial feature is read-only visibility or + controlled configuration CRUD. Do not treat a TUI list as authorization to + edit provider policy. +3. If configuration is included, define durable configuration ownership, + validation, atomic/restart semantics, rollback, audit, and secret-manager + integration. Never expose API-key values or accept them through an insecure + local UI field. +4. Preserve the existing gateway, model-allowlist, routing-override, and + provider-specific tool boundaries. Any provider/model change must remain + explicit; discovery results must not become served aliases automatically. +5. Obtain architecture/security review and HITL approval before implementation. + +## Accepted scope: configured-key / missing-provider-alias startup warning + +**Status:** accepted 0.5.15 implementation-planning scope. Implementation must +remain redacted, advisory, and no-network, subject to the documented security +and architecture gates. + +An operator can supply a provider API key while no `model_list` entry enables +that provider. This is often an accidental incomplete deployment and should be +visible without disclosing a secret, the key's value, or a raw provider error. + +### Design and implementation gates + +1. Define the precise condition from the resolved effective configuration: + a recognized provider credential is present, but no explicitly configured + model alias uses that provider. Do not warn for a configured alias whose + upstream call later fails, and do not infer enabled models from discovery. +2. Emit one bounded, redacted startup warning per provider through Airlock's + normal structured logger. This should naturally reach console output, + systemd journal, and container stdout/stderr according to deployment. +3. Evaluate a source-labelled TUI/Admin status warning as a companion view, + rather than a separate unstructured notification path. It must obey the + existing Admin authorization and never reveal secret values. +4. Specify suppression/deduplication, reload/restart behavior, provider-name + allowlisting, and tests for native, systemd, and container launch contexts. + Absence of a credential, a configured provider alias, and an unknown + environment variable must not produce a warning. +5. Obtain security review and HITL approval before implementation. + +## Accepted scope: host-console TUI administration of containerized Airlock + +**Status:** accepted 0.5.15 implementation-planning scope. The separately +authenticated control-plane design, topology, and credential lifecycle remain +security/HITL gates; Docker-network reachability is not authorization. + +The current Admin and FathomDB operational-read bridge deliberately trusts only +same-host loopback traffic. That works when Airlock and `airlock tui` are both +native processes. A host-console TUI connecting through a Docker-published port, +or a TUI/operator client in a separate container on an Airlock Docker bridge, +arrives at Airlock from a container-network address rather than `127.0.0.1`. +Neither must inherit loopback privilege. Treating a Docker bridge peer as +loopback could let another container impersonate the local operator. + +### Design and implementation gates + +Evaluate a separately authenticated control-plane path before accepting it into +0.5.15: + +1. Compare a TLS endpoint with scoped Admin credentials against a host-local + trusted proxy that validates the operator before forwarding identity over an + isolated channel. Define the supported deployment topology precisely. +2. Preserve distinct authorization for read-only snapshots, operational history, + and destructive controls; no broad re-use of the inference proxy credential. +3. Establish listener, container-network, token rotation, audit, and failure + semantics. A bridge-network source address alone is never sufficient proof + of host-operator identity. +4. Test both supported non-loopback control-plane topologies: a host-console + TUI through a loopback-published Docker port, and an approved TUI/operator + client in a separate bridged container. Cover rejected unauthenticated, + wrong-scope, and other-container access, with no operational-data leak in + error paths. +5. Obtain security review and HITL approval before exposing Admin beyond direct + process loopback. + +Until then, the supported safe patterns are a native Airlock/TUI pair, or a TUI +run inside the Airlock container. A container may publish its inference port to +host loopback, but that alone does not grant either a host-console TUI or a +bridged peer-container Admin privilege. + +## Evaluation candidate: Claude Code subscription-preserving gateway + +**Status:** research complete and explicitly out of scope for 0.5.15. Retain +the finding for a later release; do not implement or release it in this train. + +Anthropic now documents a gateway flow in which Claude Code uses only +`ANTHROPIC_BASE_URL` and retains its existing claude.ai subscription login. +The gateway must transparently preserve Anthropic's evolving Messages protocol +and OAuth capability. Airlock's present dogfooding route instead uses an +Airlock credential and an upstream Anthropic API key, so it is API-billed. + +### Future-release decision requirements + +Evaluate this as a **design-only candidate** before accepting it into 0.5.15: + +1. Reproduce the base-URL-only path without an Airlock gateway credential or + upstream Anthropic API key; record only redacted structural evidence. +2. Characterize protocol compatibility, especially streaming, evolving + `anthropic-beta`/body capability pairs, error forwarding, and optional + endpoints. +3. Design a safe initial operating mode. It must not claim compatibility for + request-mutating guardrails without proof that they preserve Claude Code + capability semantics. +4. Define credential/logging boundaries and obtain independent security and + compatibility review. +5. Bring the result to HITL: include a bounded pass-through slice, postpone, + or reject. + +Codex is not included in this candidate. Its documented subscription and +alternative-provider paths do not currently establish an equivalent, +supported self-hosted subscription-pass-through gateway protocol. + +**Research:** [Claude Code subscription gateway finding](../notes/claude-code-subscription-gateway-research.md). + +## Deferred from 0.5.14: virtual-key management + +**Status:** deferred from 0.5.15 and transferred to 0.6.0 on 2026-08-16. +Delivery remains dependent on the 0.6.0 identity/keystore contract and its +documented architecture/security/HITL gates; it is no longer 0.5.15 scope. + +### Carried draft scope + +- **DFR-32:** authenticated administrators manage masked virtual keys, create a + key with a one-time secret reveal, and revoke a key through an explicit + management seam. +- **DAC-32:** tests must cover creation, one-time reveal, revocation, + policy/spend display, unauthorized actions, and absence of key material from + logs, UI snapshots, and subsequent reads. + +### Carried architecture constraints + +- Reconcile with 0.6.0 tenant-key/keystore and identity work before design. +- Reuse an existing management/persistence seam or defer again; do not create a + competing durable credential store. +- Read views remain bounded, source-labeled, and off the inference hot path. + +### Required work before implementation + +1. Re-review the 0.6.0 keystore/identity contract and decide the ownership, + persistence, authentication, authorization, and API boundaries. +2. Ratify or revise DFR-32/DAC-32 as 0.5.15 requirements. +3. In 0.6.0, allocate the B5 foundation before Slice 60, then write/update the + management design and run the normal RED/GREEN/review/verification workflow. + +**Source evidence:** [0.5.14 draft requirements and allocation](0.5.14/03-draft-requirements-and-allocation.md), +[architecture review](0.5.14/04-architecture-review.md), and +[Slice 6 review](0.5.14/06-hitl-release-review.md). diff --git a/dev/plans/0.5.15/06-hitl-release-review.md b/dev/plans/0.5.15/06-hitl-release-review.md new file mode 100644 index 0000000..a45ddbe --- /dev/null +++ b/dev/plans/0.5.15/06-hitl-release-review.md @@ -0,0 +1,107 @@ +# 0.5.15 Slice 6 — HITL release review + +**Status:** complete — owner decision recorded 2026-08-15. Slices 0–5 are +complete planning records. The planning hard stop is closed; an included feature +slice may start only after its own prerequisites and lifecycle gates are met. + +Scores use 4 as highest: **understood** measures evidence/contract clarity and +**risk** measures security, correctness, and delivery blast radius. + +## Scorecard and recommendation + +| Item | Understood | Risk | Effort | Recommendation | Prerequisites / owner decision | +| --- | ---: | ---: | ---: | --- | --- | +| Slice 10 — Gitleaks local/CI controls | 4 | 3 | M | **Include** | Retain reviewed pins and narrow baseline; GitHub administrator must approve/perform disposable-PR, required-check, branch-protection, and ownership enforcement. | +| Slice 20 — typed Fast Guardian threat-backoff 429 | 4 | 3 | M | **Include** | Ratify DFR-34/DAC-34 detailed design and preserve existing provider/admission 429 contracts. | +| Slice 30 — configured credential/no-alias warning | 3 | 2 | S | **Conditional** | Resolve B3 with authoritative effective-model list and finite credential-source taxonomy; review warning/redaction tests. | +| Slice 40 — read-only provider/configuration projection | 3 | 3 | M | **Conditional** | Resolve B2 snapshot/version boundary; explicitly retain read-only YAML-plus-restart posture and distinct capability. | +| Slice 50 — host-console container TUI | 2 | 4 | L | **Conditional** | Owner/security decision on supported topology, TLS/CA, scopes, rotation/revocation, actor audit, and topology evidence (B2/B4). | +| Slice 60 — native virtual keys | 1 | 4 | XL | **Postpone** | Wait for ratified 0.6.0 identity/keystore/crypto/durability/authorization contract (B5). | +| Slice 70 — multi-instance fleet TUI | 2 | 4 | XL | **Conditional** | Approve named-inventory owner, per-instance identity/TLS/token isolation, operation limits, and external desired-state boundary. | +| Documentation-lifecycle cleanup | 4 | 2 | M | **Postpone** | Perform only as a small indexed backlog item after release decision; delete generated files only after preserving reproducible evidence. | +| Even-minor roadmap migration | 3 | 2 | M | **Postpone** | Separate roadmap owner resolves permanent-link and multi-worktree migration concerns. | +| Dependabot PR #45 / Actions PR #44 / new libraries | 4 | 2 | S–M | **Postpone** | Independent compatibility and immutable-pin review; no coupling to feature delivery. | +| Provider/configuration CRUD | 2 | 4 | L | **Postpone** | Requires an independently approved durable configuration ownership/lifecycle model. | +| Claude Code subscription relay | 2 | 4 | XL | **Postpone (out of scope)** | Retain research; no 0.5.15 allocation. | + +## Recommended release shape + +Authorize Slice 10 and Slice 20 first, subject to their per-slice lifecycle and +the external GitHub-administrator gate for enforcement. Treat Slices 30, 40, +50, and 70 as conditional investigations that may become feature slices only +after the stated decisions and revised designs are accepted. Do not include +Slice 60 in 0.5.15 delivery until its 0.6.0 dependency is ratified. + +The release retains these non-negotiable boundaries: + +- no new inference-hot-path management work; +- no raw secrets or sensitive operator data in API/TUI/log/docs/test evidence; +- Docker/host/orchestrator reachability is not identity or TUI authority; +- provider policy remains reviewed YAML plus restart, with no CRUD expansion; +- fleet remains named non-secret inventory plus target-local Admin-API fan-out; +- virtual keys do not invent a storage, crypto, or identity substitute. + +## Exact start order after owner response + +1. Record the owner’s include/conditional/postpone response in this file or a + linked decision record. +2. If included, begin Slice 10 only after confirming the GitHub administrator + and security reviewer can supply the external-enforcement evidence. +3. If included, begin Slice 20 with a detailed exception/HTTP mapping design and + RED tests; it may run independently of Slice 10. +4. Before any Slice 30 work, amend/review the B3 taxonomy/model-list design. +5. Before any Slice 40 work, ratify and test the B2 launcher-to-child snapshot + seam; before Slice 50, also close B4. +6. Do not schedule Slice 60 until 0.6.0 ratifies B5. Do not schedule Slice 70 + until its fleet authority decision is written and reviewed. + +## Recorded owner decision + +The owner directed Slice 6 to be completed. This record applies the scorecard’s +recommendations as the HITL decision: + +| Area | Recorded disposition | Consequence | +| --- | --- | --- | +| Slice 10 — Gitleaks | **Include** repository-controlled work; external enforcement remains **conditional** | The slice may enter its admission lifecycle. It still needs security re-review, ratified DFR-35/DAC-35, a clean current scanner verification, and GitHub-administrator evidence before it claims merge-blocking enforcement. | +| Slice 20 — typed threat-backoff 429 | **Include** | It may enter detailed design and RED-test planning after rechecking this closure; it must preserve existing provider/admission 429 behavior. | +| Slices 30, 40, 50, and 70 | **Conditional** | Do not start until their named blockers/owner decisions are resolved: B3 warning taxonomy; B2 snapshot; B4 remote topology/identity; and fleet identity/TLS/token/inventory authority. | +| Slice 60 — virtual keys | **Postpone** | Do not schedule before the 0.6.0 identity/keystore/crypto/durability contract is ratified (B5). | +| Gitleaks external enforcement | **Conditional external action** | The approved least-privilege workflow direction is retained, but only a GitHub administrator can configure and prove required `gitleaks / scan` and code-owner protection on `main`. It is not active yet. | +| Provider CRUD | **Postpone** | Keep read-only/YAML-plus-restart posture; a durable configuration-ownership proposal is required before reconsideration. | +| Host-console topology | **Conditional** | Security/operations must explicitly approve TLS/CA, scopes, token rotation/revocation, actor audit, and a supported non-loopback topology. | +| Virtual-key dependency | **Postpone** | The 0.6.0 contract is a hard dependency, not an implementation detail that 0.5.15 may invent. | +| Fleet authority boundary | **Conditional** | Retain named non-secret inventory and target-local Admin-API fan-out; reject host/orchestrator access, automatic discovery, and desired-state ownership. | +| Documentation lifecycle, roadmap migration, dependencies/Dependabot, and Actions majors | **Postpone** | Keep these as independently authorized maintenance/backlog work; do not bundle them with included feature slices. | +| Claude Code subscription relay | **Postpone / out of scope** | No 0.5.15 implementation allocation. | + +The existing acceptance of Gitleaks’s least-privilege no-secret boundary and +scoped ownership remains in force. The Action’s runtime-binary provenance is an +accepted residual risk only within that read-only, no-secret job; an ownership +or organization-license change requires re-review. + +## Subsequent owner disposition — Slice 60 transfer + +On 2026-08-16, the owner moved virtual-key management out of the 0.5.15 +release. Slice 60 is **deferred** here and transferred to the 0.6.0 release, +where it remains blocked on the B5 native identity/keystore/crypto/durability +foundation. This does not authorize that foundation or Slice 60 implementation; +it preserves the prior no-substitute constraints. + +## Post-decision start order + +1. Recheck this decision and update the selected slice plan for any changed + environment, baseline, repository, or dependency state. +2. Admit Slice 10 only when its current security/baseline and GitHub-authority + gates are satisfied; it may deliver repository controls before external + enforcement is proven, but must state the latter is inactive. +3. Admit Slice 20 after its detailed typed-error design and RED-test plan are + reviewed; it can proceed independently of external GitHub settings. +4. Leave all conditional and postponed items dormant until their recorded gates + are resolved in a new or updated HITL decision. + +## Close condition + +Slice 6 is closed. No canonical requirement, dependency, GitHub setting, test, +or product behavior changed as part of this decision record. Each included +feature slice remains responsible for its own review, RED/GREEN evidence, +external authorization, and status record. diff --git a/dev/plans/0.5.15/40-virtual-key-management-0.6-contract-design.md b/dev/plans/0.5.15/40-virtual-key-management-0.6-contract-design.md new file mode 100644 index 0000000..0e5d0b8 --- /dev/null +++ b/dev/plans/0.5.15/40-virtual-key-management-0.6-contract-design.md @@ -0,0 +1,261 @@ +# 0.5.15 design plan — virtual-key management on the 0.6.0 identity/keystore contract + +**Status:** deferred from 0.5.15 and transferred to 0.6.0 on 2026-08-16. This +historical design does not authorize an endpoint, persistence change, database +migration, or UI control. The 0.6.0 B5 keystore/identity foundation, threat +model, and minimum management slice remain HITL gates before implementation. + +## Executive decision + +The 0.5.14 carry-forward requirement (DFR-32/DAC-32) is viable only as an +adapter to the **Airlock-native `VirtualKeyStore` selected by 0.6.0**. It must +not turn on LiteLLM's `/key/*` management API, create a second credential +database, or use the client-controlled `X-Airlock-Client` header as tenant +identity. The planned management layer owns presentation, authenticated Admin +authorization, one-time secret delivery, and audit; the 0.6.0 identity/store +layer owns key authentication, policy state, revocation truth, budget state, +and enforcement. + +This is intentionally a dependency-first plan: if the 0.6.0 contract changes +or its native authentication seam is not available, virtual-key management is +deferred again rather than substituting a new store. + +## Evidence and dependency contract + +The 0.6.0 design (`dev/notes/design-tenant-keys-budgets.md`) is DECIDE-ready +and records two load-bearing findings: + +- LiteLLM virtual keys require a PostgreSQL-backed key-management setup. That + violates Airlock's single-tenant/no-required-external-service default and + would make LiteLLM the parallel durable source of truth. +- LiteLLM's `general_settings.custom_auth` hook can authenticate a presented + key using an Airlock-native store before the normal master-key path. The + native store therefore remains viable, but all management routes must remain + Airlock-admin protected because custom auth does not supply the required + governance policy by itself. + +The agreed 0.6.0 ownership model is: + +``` +key secret presented to inference + -> LiteLLM custom_auth adapter + -> Airlock VirtualKeyStore (authoritative record / revocation / policy) + -> authenticated ClientIdentity + -> guardian + allowlist/routing + admission/budget + dispatch backstop + +Admin/TUI management request + -> Airlock Admin PDP (separate operator identity and exact scope) + -> VirtualKeyManagement service + -> same Airlock VirtualKeyStore + -> redacted admin_action event / bounded read projection +``` + +The two identities are deliberately different: + +| Identity | Source | Purpose | Never used for | +| --- | --- | --- | --- | +| Operator | Admin loopback or scoped capability JWT | create/read/update/revoke authority and audit actor | inference tenant attribution | +| Tenant/key | validated virtual-key secret → stable record id | model authorization, budgets, limits, request attribution | management authority | +| `X-Airlock-Client` | caller-controlled attribution header | optional display/diagnostics only | authorization, isolation, spending, revocation | + +## Requirements to ratify or revise + +### Functional requirements + +- **DKR-1 — one authority.** Virtual-key policies, secret verifier, lifecycle, + revocation, budget/limits, and owner/team metadata have exactly one durable + authority: the 0.6.0 `VirtualKeyStore`. Admin/TUI are adapters, not stores. +- **DKR-2 — constrained creation.** An authorized operator can request a key + with a label and a policy approved by 0.6.0 (allowed model aliases, + budget/limit references, expiry, and owner/team identifier). Server-side + validation rejects unknown aliases, invalid budgets, impossible expiry, and + ambiguous ownership before persistence. +- **DKR-3 — one-time secret reveal.** The randomly generated bearer secret is + returned only in the successful creation response over an authenticated, + approved management transport. Subsequent read/list/update/revoke responses + expose only a masked display prefix and stable non-secret identifier. A lost + secret is replaced by creating/rotating a new key, never recovered. +- **DKR-4 — lifecycle.** List/read views show bounded masked status, expiration, + allowed-alias summary, current policy/budget state, spend/limit state and + `revoked_at`; authorized revocation takes effect at authentication before a + later upstream request. Policy updates must obey the same 0.6.0 validation + and enforcement invariants as creation. +- **DKR-5 — exact management authorization.** Explicit scopes are required: + `admin:keys:read`, `admin:keys:create`, `admin:keys:update`, and + `admin:keys:revoke` (or a formally reviewed equivalent). The existing scope + matcher is exact-string based, so a decorative `admin:keys:*` must not be + claimed as a wildcard without matcher work and review. +- **DKR-6 — audit and read model.** Successful lifecycle mutations produce one + redacted `admin_action` record with actor, operation, non-secret key id, + policy delta summary, and timestamp. Reads are bounded, source-labelled + snapshots that contain no bearer material and run off the inference hot path. +- **DKR-7 — availability semantics.** An unavailable key-authentication store + fails closed for a presented virtual key (the 0.6.0 isolation decision); + management reports unavailable and performs no partial mutation. Master-key + behavior and anonymous inference retain their separately defined contracts. +- **DKR-8 — no accidental LiteLLM key plane.** Airlock disables or blocks + LiteLLM `/key/*`, `/user/*`, and `/team/*` management routes for virtual-key + principals and does not expose the LiteLLM admin UI as Airlock's key UI. + +### Non-functional/security requirements + +- Default deployments retain no mandatory PostgreSQL, Redis, or external secret + manager. A configured shared backend is a 0.6.0 store deployment choice, not + a TUI feature. +- Secret generation uses a CSPRNG; the store retains only a non-reversible, + domain-separated verifier/identifier suitable for lookup, never plaintext. + The exact construction (including pepper/key rotation) is a 0.6.0 crypto + design decision and must receive security review before coding. +- Secret values are prohibited from JSONL, callback/event payloads, FathomDB, + S3/SQL exporters, exception text, metrics labels, traces, URLs, process + arguments, config files, CLI history, TUI state/snapshots, tests, and audit + records. +- Tenant policy always beats routing resilience: allowed-model candidate sets + are filtered and the per-dispatch backstop rejects an off-list final model; + LiteLLM fallbacks must not bypass authorization. +- The feature is opt-in. Anonymous/no-key requests remain behaviorally + compatible, verified by the 0.6.0 replay oracle. + +## Acceptance criteria for an implementation proposal + +1. A valid Admin principal with `admin:keys:create` creates a policy-valid key, + receives one secret only once, and thereafter sees a masked record; the + secret is absent from every persisted, observed, and rendered artifact. +2. Missing, expired, invalid, read-only, or wrong-scope operator credentials + cannot list, create, update, reveal, or revoke a key; a tenant key cannot + reach any Admin/key-management route. +3. A valid virtual key authenticates through the native 0.6.0 auth seam and + yields its authenticated identity; a forged `X-Airlock-Client` does not + change its enforcement identity, quota, breaker bucket, or attribution + authority. +4. Revocation is durable and immediately denies new keyed requests before + dispatch. Concurrent use/revocation behavior is specified, tested, and + leaves no successful post-revocation authentication ambiguity. +5. Model/budget/limit display agrees with the authoritative store; model + authorization covers alias resolution, smart/cost routing, Airlock failover, + and every LiteLLM retry/fallback attempt. No off-list provider call occurs. +6. A store outage gives a bounded failure for keyed inference and management, + with no anonymous fallback. Anonymous and master-key contracts are tested + separately and unchanged where 0.6.0 says they are unchanged. +7. List views are capped/paginated, redact stable ids appropriately, label their + data source/freshness, and do not access FathomDB or subscribe a UI worker + to request callbacks. +8. Tests include secret-leak scans across structured logs/events/traces/TUI + snapshots, creation idempotency/retry policy, update validation, one-time + reveal, revocation race, restart persistence, unauthorized paths, and the + existing anonymous golden replay. +9. Security and architecture reviewers approve the verifier construction, + storage backend, route gating, token transport, and all migration/rollback + semantics before a feature flag is enabled. + +## Proposed management design + +### API and UI shape + +The first approved slice should prefer a small authenticated Admin API used by +both CLI and TUI. The TUI is a client of that API; it never writes configuration +or database files directly. + +| Operation | Proposed outcome | Required authority | +| --- | --- | --- | +| List/read | bounded masked records, policy/spend/status/source | `admin:keys:read` | +| Create | validates policy, persists record, returns one-time secret exactly once | `admin:keys:create` | +| Update policy | validated patch with concurrency/version rule; no secret return | `admin:keys:update` | +| Revoke | idempotent lifecycle transition and audit | `admin:keys:revoke` | +| Rotate | explicit create-new + distribute + revoke-old workflow initially | create + revoke; no silent in-place secret replacement | + +Do not provide a text field for a provider secret or virtual-key secret in the +TUI. The TUI can render the creation secret in an intentional one-time modal +with copy confirmation, then discard it from application state; secure delivery +outside the terminal remains the operator's responsibility. The precise terminal +clipboard policy requires HITL because terminals, multiplexers, and clipboard +integrations have different retention behavior. + +### Store and transactional rules + +`VirtualKeyManagement` calls the same store interface selected in 0.6.0. Its +create transaction must atomically commit the record, verifier, initial policy +revision, and audit-intent/outbox entry (or report an explicitly recoverable +state). Revocation must make the authentication lookup deny before reporting +success. If a backing store later supports optimistic revisions, update/revoke +uses them to avoid lost policy changes; otherwise concurrent semantics must be +conservatively serialized and documented. + +The management API must not attempt to synthesize spend from JSONL. It consumes +the 0.6.0 authoritative per-key accumulator/view and labels unavailable/stale +conditions. This preserves the architecture rule that UI reads are bounded and +off the inference hot path. + +## External research and library assessment + +| Source/library | Relevant capability | Alignment assessment | +| --- | --- | --- | +| [LiteLLM virtual keys documentation](https://docs.litellm.ai/docs/proxy/virtual_keys) | Documents `/key/generate`, spend, blocking, policy hooks, and states that virtual-key setup needs PostgreSQL + a master key. | **Partial only.** Reuse its provider translation/custom-auth integration where 0.6.0 characterized it; do not use its persistence, routes, UI, or admin ownership. | +| [BerriAI/litellm](https://github.com/BerriAI/litellm) | Current embedded proxy substrate and custom-auth hook surface. | **Existing dependency, partial.** Preserve version-pinned adapter characterization and block upstream management routes for tenant keys. | +| [HashiCorp Vault Transit](https://developer.hashicorp.com/vault/docs/secrets/transit) / [hvac](https://github.com/hvac/hvac) | Envelope encryption, key versioning, and a Python client for an operator-managed external KMS. | **Optional integration, not baseline.** Could protect a configured store-encryption key in enterprise deployments; it must not become mandatory or a second virtual-key database. | +| [argon2-cffi](https://github.com/hynek/argon2-cffi) | Password hashing library. | **Not selected by default.** Key verification needs a high-entropy bearer-token lookup and hot-path budget; a 0.6.0-reviewed keyed verifier/pepper design is required instead of importing password-hash semantics by assumption. | +| Python `secrets`, `hmac`, `hashlib` | CSPRNG and keyed hashing primitives already available in Airlock's runtime. | **Candidate baseline only after crypto review.** Avoids a new dependency but is not a substitute for defining rotation, domain separation, and durable-store behavior. | + +The key architectural fact from LiteLLM's official documentation is material: +its virtual-key setup assumes PostgreSQL. Airlock's 0.6.0 decision explicitly +rejects making that a prerequisite for the single-container default; therefore +using LiteLLM key CRUD would be a false shortcut, even if it could complete the +UI endpoints quickly. + +## Architectural alignment and blast radius + +**Alignment verdict: aligned only behind the ratified 0.6.0 native seam.** It +extends the existing three planes rather than duplicating them: `StateStore`/ +store ownership for durable key state, the guardian/custom-auth path for +inference enforcement, `RequestEvent`/recorder for attributable observability, +and Admin PDP/TUI client for operator management. The plan is misaligned if it +adds a TUI-owned SQLite file, treats LiteLLM `/key/*` as the source of truth, +allows a request header to select tenant policy, or adds UI callbacks to the +inference path. + +| Area | Future change | Blast radius | Controls | +| --- | --- | --- | --- | +| Authentication/identity | custom-auth adapter + native virtual-key resolution | Critical: tenant isolation | fail closed, forged-header adversarial tests, master/admin route denial | +| Guardian/routing/fallback | allowed-set filtering and dispatch backstop | Critical: provider authorization | end-to-end fallback tests; policy outranks resilience | +| Store/spend lifecycle | native durable records, revocation, key spend | High: correctness and availability | one authority, atomic rules, restart/reconciliation tests | +| Admin API/CLI/TUI | scoped lifecycle operations and masked views | High: secret disclosure/control authority | exact scopes, bounded output, redacted audits, no direct store access | +| Callbacks/logging | identity/policy attribution | High: PII/secret leak risk | canonical event fields and artifact-leak tests | +| Anonymous inference | no intended change | Must be zero | golden replay and regression suite | + +## Rollout and verification plan + +1. **0.6.0 dependency gate:** confirm the native store interface, custom-auth + adapter, identity re-key, allowlist/fallback backstop, and store-outage + behavior are implemented and reviewed. Otherwise stop; do not backfill with + LiteLLM CRUD. +2. **HITL requirement ratification:** decide allowed policy fields, secret + verifier/encryption construction, token-transport profile, audit retention, + terminal one-time-reveal behavior, and whether updates ship with the first + slice. +3. **RED contracts:** add pure store/management/PDP tests first, then adversarial + tests for raw-secret leakage, forged header identity, key access to admin + routes, stale write/revoke, fallback escape, and store outage. +4. **GREEN slice:** introduce default-off management endpoints and a CLI before + the TUI screen; keep an audit outbox/recovery story explicit. Add TUI only + after the API response schema is stable and snapshot-safe. +5. **Canary:** use a disposable tenant and non-sensitive configured alias; + observe masked read, one-time delivery, allowed request, revocation denial, + and redacted audit. Do not use a live master key or customer prompts as test + evidence. +6. **Rollback:** disable the management feature while retaining store records; + revocation remains enforceable. Never roll back by deleting a live credential + store. Rotate affected credentials/pepper according to the approved incident + procedure if a secret-handling failure is discovered. + +## Remaining HITL questions + +- Should the first management slice ship only create/list/revoke, leaving policy + update and rotation for a later version to minimize irreversible semantics? +- What is the approved default store backend and backup/restore contract for a + single-container deployment, and when may a shared-store configuration be + supported? +- What maximum key-policy expressiveness is justified initially without making + the key surface a second routing language? +- Which secure out-of-band distribution mechanism is recommended for the + one-time secret, and should terminal clipboard integration be disabled by + default? diff --git a/dev/plans/0.5.15/README.md b/dev/plans/0.5.15/README.md index 69f3c23..fb6f6d2 100644 --- a/dev/plans/0.5.15/README.md +++ b/dev/plans/0.5.15/README.md @@ -30,7 +30,7 @@ that is accepted for investigation from work that is approved for delivery. | Configured-provider credential with no enabled alias warning | Implemented locally; redacted startup-warning evidence recorded | Slice 30 | | Read-only provider/configuration visibility in Admin/TUI | Design exists; CRUD is not approved | Slice 40 | | Secure host-console TUI administration for containers | Design exists; topology/identity decision is open | Slice 50 | -| Airlock-native virtual-key management | Design exists; depends on 0.6.0 identity/keystore contract | Slice 60, conditional | +| Airlock-native virtual-key management | **Deferred to 0.6.0**; depends on the native identity/keystore foundation | Slice 60 (0.6.0) | | Multi-instance TUI administration | Research/design exists; remote operation/configuration authority is open | Slice 70, conditional | | CI review and improvement | Review workflow reliability, signal, coverage, cost, and maintainability without weakening delivery controls | Slice 80 | | Release closeout | Required only for work actually included and delivered | Slice 90 | @@ -216,7 +216,7 @@ authorize work before Slice 6. | 30 | Redacted startup warning for configured credential/no enabled alias | Provider taxonomy, logging, and redaction drafts pass. | | 40 | Read-only provider/configuration Admin/TUI visibility | Read-only configuration-owner posture remains approved; no CRUD expansion. | | 50 | Secure host-console TUI administration for containerized Airlock | Topology, operator identity, TLS/token lifecycle, and audit decisions pass. | -| 60 | Virtual-key store and management | 0.6.0 keystore/identity contract and threat model are ratified; otherwise postpone. | +| 60 | **Deferred to 0.6.0** virtual-key management | The 0.6.0 B5 identity/keystore foundation must close first; not a 0.5.15 delivery slice. | | 70 | Multi-instance TUI named inventory and read-only Admin-API fan-out | Fleet identity/TLS/token/inventory/operation limits are approved; otherwise postpone. | | 80 | CI review and improvement | Review current workflows, required checks, runtime/cost, cache and matrix design, test-signal quality, and failure triage; preserve least privilege, pinned actions, Gitleaks enforcement, and no-secret CI boundaries. Document and independently review proposed changes before implementation. | | 90 | Release closeout | Only delivered, verified slices are included; deferred work is named explicitly. | From 057040e2b7e0292e81b67c4e1332c04df153071e Mon Sep 17 00:00:00 2001 From: Corey Thompson Date: Sun, 16 Aug 2026 18:10:31 -0500 Subject: [PATCH 14/24] docs(plan): record Slice 70 fleet admission gate --- ...stance-tui-fleet-control-plane-research.md | 297 ++++++++++++++++++ .../0.5.15/70-fleet-admin-fanout-status.md | 48 +++ 2 files changed, 345 insertions(+) create mode 100644 dev/plans/0.5.15/50-multi-instance-tui-fleet-control-plane-research.md create mode 100644 dev/plans/0.5.15/70-fleet-admin-fanout-status.md diff --git a/dev/plans/0.5.15/50-multi-instance-tui-fleet-control-plane-research.md b/dev/plans/0.5.15/50-multi-instance-tui-fleet-control-plane-research.md new file mode 100644 index 0000000..4b81d84 --- /dev/null +++ b/dev/plans/0.5.15/50-multi-instance-tui-fleet-control-plane-research.md @@ -0,0 +1,297 @@ +# 0.5.15 research hypothesis — one TUI for multiple Airlock instances + +**Status:** re-evaluated on 2026-08-16: **conditional, planning/design only; +not admitted for implementation.** This document adds no deployment agent, +remote-execution capability, or configuration write endpoint. The Slice 70 +audit and independent design review confirmed that the fleet authority/HITL +decisions below remain unresolved. + +## Slice 70 admission update + +Slices 40 and 50 now provide a redacted child-owned Admin configuration +projection and a secure **single-target** host-console transport respectively. +They do not establish fleet inventory ownership, target identity, hardened +fan-out transport, audience-bound credentials, selection semantics, or a +cross-target audit model. The ordinary `urllib` Admin client must not be +generalized into fleet access without those controls. + +The reviewed minimum **hypothesis** for a later owner decision is read-only +v1: an owner-only static local inventory with opaque IDs, display names, literal +HTTPS origins, and per-target CA/token-file references; explicit selections +only; no discovery, deployment/lifecycle/configuration actions, shared +credentials, mutations, redirects, proxy-from-environment, or local fleet +operation log. Proposed engineering limits are at most 20 selected targets, +four concurrent requests, 2-second connection/5-second request deadlines, and +64 KiB response bodies. They are not an approved product contract. + +Before implementation, the owner/HITL record must decide: + +1. inventory publisher, format, permissions, reload/lifecycle, and + secret-manager reference posture; +2. supported systemd/container/Kubernetes topology and per-instance + CA/SAN/issuer/audience/token-rotation contract, including whether every + target requires a distinct credential; +3. whether RFC1918/private targets are permitted and the DNS-rebinding/egress + enforcement rule; HTTPS-only, redirect, proxy, metadata, link-local, and + loopback policy; +4. read-only v1 versus an exact fleet-safe mutation allowlist, target/count + limits, confirmation/dry-run/retry semantics, and any two-person workflow; +5. correlation/audit retention and the explicit boundary that desired-state + deployment remains external. + +Until that record is ratified, the appropriate Slice 70 outcome is a status +record and a decision-ready design, not code, tests, user documentation, or +deployment changes. + +## Outcome and hypothesis + +An operator should be able to open one `airlock tui`, select one or more named +Airlock instances, inspect each instance through its authenticated Admin API, +and perform an explicitly scoped *runtime* action on the chosen instance(s). +This must work whether an instance is launched under systemd or in a container. + +The narrow, Airlock-aligned hypothesis is: + +> **The multi-instance TUI is an authenticated, fan-out Admin API client with a +> local, non-secret inventory of endpoints and trust anchors. Each Airlock +> instance remains the authority for its runtime state, authorization, audit, +> virtual keys, and effective configuration snapshot. Deployment configuration +> is owned by the existing reviewed deployment mechanism (file + restart, +> Ansible, GitOps, etc.), never by a TUI-side state store.** + +“System-wide” therefore means an intentionally selected *fleet/target group*, +not a hidden global Airlock singleton. A system-wide change is either: + +1. a bounded runtime Admin action fanned out to an explicitly selected target + set, after every target authorizes it independently; or +2. a desired-state deployment change committed/applied by an approved external + configuration owner, followed by each instance reporting its observed, + redacted effective-config fingerprint through its Admin API. + +The first can be an Airlock feature. The second is an integration/documentation +contract, not an embedded control plane. + +## Current Airlock constraints carried forward + +| Existing seam | Constraint for the fleet feature | +| --- | --- | +| `airlock/admin/policy.py` and `airlock/admin/http.py` | Each destination retains one Admin PDP/perimeter. A fleet selection never substitutes for the instance’s TLS, token, exact scope, or audit decision. | +| `airlock/tui/admin_client.py` | Extend the TUI client to use named endpoint profiles and bounded concurrent calls. Do not mount a Textual UI in the proxy process. | +| Existing host-console plan | Remote/container access is TLS plus scoped credentials; a Docker bridge, host port, DNS name, or private subnet is reachability only, not operator identity. | +| Provider-config design | The first provider panel is a redacted read-only effective-startup projection. Neither a fleet file nor a TUI cache becomes a second provider-routing authority. | +| Virtual-key design | `VirtualKeyStore` is the one per-instance key authority. The TUI only calls its scoped Admin adapter and never synchronizes key secrets or opens another instance’s store. | + +The earlier design also found that the launcher starts LiteLLM as a separate +process. A configuration snapshot must therefore be initialized in that child +from the exact resolved runtime configuration; a TUI must never work around the +boundary by reading remote config files. + +## Requirements for a future implementation + +1. **Named inventory, no discovery by default.** A local TUI profile contains + an opaque `instance_id`, display name, HTTPS Admin base URL, server-name/CA + reference, and a **reference** to a local credential file or secret-manager + item. It contains no bearer token, provider credential, virtual-key secret, + SSH key, Docker socket, systemd unit, or provider YAML value. Auto-discovery + through Docker, host scans, Consul, Kubernetes, or cloud APIs is out of the + first implementation. +2. **Per-instance authentication.** The client makes independent TLS-verified + Admin calls and sends only that instance’s short-lived scoped credential. + A fleet credential must not be silently replayed to arbitrary endpoints; + shared issuer/audience is a separately ratified identity design. +3. **Read aggregation is bounded and truthful.** Fetch redacted health/live + snapshots and the proposed config-warning/config-inventory projection with + bounded concurrency, request/time/size caps, source and freshness labels. + Show `unreachable`, `TLS failed`, `unauthorized`, `forbidden`, and `stale` + per instance; do not collapse partial failure into fleet success. +4. **Two-person semantics for mutations.** A user first selects explicit + `instance_id`s, sees the exact action and the distinct target count, then + confirms. The TUI submits one request per instance, records a local + non-secret operation correlation ID, and reports a matrix of success/failure. + It must not offer an “all instances” default, wildcard target, or best-effort + retry that can turn a partial operation into an unbounded one. +5. **Local actions only initially.** Fan-out is eligible only for existing, + idempotent, explicitly fleet-safe, non-destructive Admin actions after the + Admin action contract says so. Provider/routing/config writes, Admin-token + issuance, token revocation, virtual-key creation/reveal, key deletion, + master-key changes, process restarts, and upgrades are excluded pending + individual designs. +6. **Configuration truth.** The TUI may display each destination’s immutable, + redacted effective-config fingerprint/version and configured policy view. A + restart-required change must be visibly `desired` vs `observed` and is not + called applied until the destination reports the new fingerprint. +7. **Audit and secrets.** Every receiving instance produces its existing + redacted `admin_action` record with the authenticated subject and optional + correlation ID. The TUI’s audit summary includes only target IDs, action, + outcome, and timestamps. It never logs token/certificate/key material, + provider secrets, request contents, raw config, or unredacted target errors. +8. **No inference dependency.** A failed/unavailable fleet console, inventory, + external deployment controller, or one failed instance cannot affect another + instance’s inference path. + +## Candidate architecture + +```text +operator + | + v +one `airlock tui` -- local profile: target IDs, URLs, CA/token *references* + | | | + | HTTPS + scoped token | HTTPS + scoped token + v v v +Airlock A Admin PDP Airlock B Admin PDP Airlock C Admin PDP + | | | + +-- local runtime state +-- local state +-- local state + +-- config fingerprint +-- fingerprint +-- fingerprint + +reviewed desired-state owner (optional, external) + Git/Ansible/Kubernetes/Nomad/etc. --> config/secrets + lifecycle per target + TUI reads post-apply state only; it neither invokes nor impersonates it. +``` + +The optional lower path is explicitly **not** between the TUI and Airlock. +That prevents an operator-console compromise from acquiring host/root, Docker, +or orchestrator credentials merely to view an Admin page. + +## Is a library/control-plane needed? + +### Embedded Airlock dependencies + +No external fleet-control library is suitable as a baseline dependency. The +smallest aligned implementation is an Airlock-owned `FleetAdminClient` built on +the TUI’s existing typed Admin client/async HTTP substrate, plus Textual’s +existing selection/table/worker facilities. It needs application code rather +than a new fleet SDK: + +| Candidate | Appropriate use | Decision | +| --- | --- | --- | +| Existing Textual + Airlock Admin client | Named-target selection, bounded concurrent reads, action-result matrix, local credential-file references. | **Adopt/extend.** Maintains TUI-as-client and avoids new infrastructure. | +| `httpx` (already used by Airlock provider/transport code) | TLS-verified async, pooled, bounded Admin API calls if the TUI client does not already expose equivalent transport. | **Partial; no new service.** Pin/cap only if a direct import is required. It is transport, not fleet authority. [HTTPX](https://www.python-httpx.org/async/) | +| `kubernetes` Python client / Docker SDK / Paramiko / systemd D-Bus client | Would make a user-facing TUI control the orchestrator/host/daemon directly. | **Reject.** Violates the no direct SSH/Docker/systemd access boundary and greatly expands credential and blast-radius scope. | +| Ansible Runner Python library | Useful only to an operator-owned automation runner, not linked into `airlock tui`. | **Do not adopt in Airlock.** A runner would need SSH/WinRM keys and playbook authority; it is a separate deployment platform. [Ansible Runner](https://github.com/ansible/ansible-runner) | + +An instance registry/server, agent, message bus, or Airlock-specific Terraform +provider is not justified for 0.5.15. Those would introduce a new durable +authority, cross-instance identity and lifecycle requirements, and HA/recovery +work before they improve the basic Admin-client use case. + +### External systems that can own configuration/deployment + +These are viable *operator-selected integrations*, not Airlock dependencies. +Their role is to make desired deployment state available; Airlock must expose +only the observed effective result via Admin. + +| System | What official docs establish | Fit and boundary | +| --- | --- | --- | +| **Ansible / Automation Controller** | Inventories and playbooks are the normal automation project model; `ansible.builtin.systemd_service` manages systemd units on remote hosts, including idempotent start/stop/restart. [Getting started](https://docs.ansible.com/projects/ansible/latest/getting_started/get_started_ansible.html), [systemd module](https://docs.ansible.com/projects/ansible/latest/collections/ansible/builtin/systemd_service_module.html) | **Best general VM/systemd or Docker-host deployment integration.** It can template Airlock config, provision secret references, and restart a selected service/container. Airlock must not embed it or receive its SSH/control credentials. | +| **OpenTofu / Terraform** | Declarative providers manage remote resource types but state maps configuration to real resources; official docs warn state can contain sensitive values and needs secure/locked storage. [providers](https://opentofu.org/docs/v1.11/language/providers/), [sensitive state](https://opentofu.org/docs/language/state/sensitive-data/), [state locking](https://opentofu.org/docs/language/state/backends/) | **Conditional infrastructure provisioning only.** Good for VMs, networks, KMS/Vault policy, or Kubernetes resources. Poor default for mutable Airlock provider policy: creating an Airlock provider would require a custom provider, token/state-secret design, and produces a competing config API. Do not build one in 0.5.15. | +| **Kubernetes + Flux or Argo CD** | Kubernetes Deployments declaratively reconcile Pod state and support controlled rollout/rollback; ConfigMaps can be volume-mounted but env values require restart; Secrets require encryption at rest/RBAC and are not safely "just base64". [Deployments](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/), [ConfigMaps](https://kubernetes.io/docs/concepts/configuration/configmap/), [Secrets](https://kubernetes.io/docs/concepts/configuration/secret/). Flux syncs versioned sources to Kubernetes with RBAC and multi-cluster capabilities; Argo CD is declarative GitOps CD for Kubernetes. [Flux](https://fluxcd.io/flux/), [Argo CD](https://argo-cd.readthedocs.io/en/stable/) | **Best supported configuration-owner model for Kubernetes-only adopters.** Git is desired state; Flux/Argo applies it and the workload restarts/reloads according to the deployment design. Do not bundle an operator, CRD, Flux/Argo client, or cluster credential in the TUI. | +| **Salt** | Salt is a master/minion remote-execution and configuration-management system; state files can enforce configuration, services, and files across targeted minions. [overview](https://docs.saltproject.io/salt/user-guide/en/latest/topics/overview.html), [state system](https://docs.saltproject.io/en/latest/ref/states/index.html) | **Possible existing-enterprise integration, not recommended baseline.** It creates a master/minion control plane and remote-execution authority beyond Airlock’s needs. | +| **Nomad** | Nomad jobs declare desired workload state, which Nomad reconciles on clients; job files are normally version-controlled. [jobs](https://developer.hashicorp.com/nomad/docs/concepts/job), [job submission](https://developer.hashicorp.com/nomad/docs/job-declare/create-job) | **Conditional for existing Nomad installations.** Use its jobspec/ACL/update process; do not give the TUI Nomad credentials or make Airlock schedule workloads. | +| **Consul** | Consul is service discovery/networking; registered services provide health/address information via catalog/DNS/API. [service discovery](https://developer.hashicorp.com/consul/docs/discover) | **Optional discovery source only, and deferred.** It may resolve pre-approved `instance_id`s in a Consul-operated estate, but catalog membership is not Airlock-admin authorization and should not cause automatic target enrollment. | +| **FleetDM** | Fleet is device management/endpoint control, not an application desired-state or per-service Admin plane. | **Reject.** It is the wrong layer: a host/device agent would expand endpoint-management authority and cannot replace TLS/scoped Airlock Admin calls. | +| **Vault** | An external secrets/KMS system can protect deployment secrets and native-store encryption material, but it is neither an instance registry nor a fleet config controller. [Vault Transit](https://developer.hashicorp.com/vault/docs/secrets/transit) | **Optional existing-enterprise secret integration only.** Retain Airlock’s native virtual-key authority and never put virtual-key plaintext, Admin tokens, or TUI inventory secrets in a second Airlock-owned Vault schema. | + +## systemd versus containers: why it matters + +The Admin API contract is deliberately deployment-neutral—both forms need TLS, +per-instance credentials, authorization, audit, and a truthfully reported +effective configuration. The *delivery and lifecycle* layer is not neutral. + +| Concern | systemd-native service | Docker/Compose container | Kubernetes/orchestrated container | +| --- | --- | --- | --- | +| Transport/reachability | Local loopback can safely support a same-host native TUI; remote uses an explicitly exposed TLS Admin endpoint. | A published port is host firewall/NAT exposure, not container-loopback identity. Docker notes bridge-network ports are reachable from host/peer containers and `-p` exposes them outside the host. [Docker](https://docs.docker.com/engine/network/port-publishing/) | Service/DNS/load balancers and network policy determine reachability; Pod IPs are ephemeral. Endpoint identity must be stable DNS plus TLS, never an IP. | +| Discovery | Static hostnames/inventory (or an external CMDB/Ansible inventory) is natural. | Compose service/container names are local deployment details; do not scan Docker socket. | Labels, Services, or an approved registry can map IDs to URLs, but automatic enrollment expands trust and must be separately designed. | +| Config delivery | Reviewed files, systemd environment/credentials, unit drop-ins. | Mounted files/env/secret files and image/Compose versions. Environment updates normally require container recreation. | ConfigMap volume propagation and Secrets/POD templates; ConfigMap environment values do not update automatically. Deployment rolls out a changed Pod template at a controlled rate. [ConfigMaps](https://kubernetes.io/docs/concepts/configuration/configmap/), [Deployments](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/) | +| Identity/auth | OS permissions protect local files; remote endpoint still needs TLS and scoped Admin credentials. | Avoid Docker-socket access and bridge-address trust. Token/CA materials must be mounted/read with least privilege. | Cluster RBAC/service accounts are distinct from Airlock Admin subjects; Kubernetes Secrets require encryption at rest and least-privilege RBAC. [Secrets](https://kubernetes.io/docs/concepts/configuration/secret/) | +| Lifecycle/upgrades | `systemd` does daemon reload/restart/enable; Ansible’s systemd module can do it remotely. | Deployment system replaces/restarts a container; TUI must not run Docker commands. | Controller rollout/rollback semantics and availability gates apply; TUI must not call `kubectl` or reconciliation APIs. | +| Blast radius | A privileged service manager or SSH account can own the whole host. | Docker socket often controls the host/container estate; giving it to the TUI is a high-risk escalation. | Namespace/cluster credentials can modify many workloads/secrets; GitOps controller scope must be RBAC-constrained. | + +So deployment form matters greatly for *configuration ownership and blast +radius*, but it should not lead to three TUI implementations. The TUI speaks a +single Admin API protocol; installation-specific configuration is delivered by +the platform that already owns the installation. + +## Acceptance criteria for the narrow first slice + +1. A configured TUI profile with two to a bounded number of distinct HTTPS + Airlock Admin endpoints shows per-instance source, version/fingerprint, + fresh/stale/unavailable state, without contacting an unconfigured endpoint. +2. Invalid CA/hostname, expired/wrong-audience/wrong-scope token, missing token, + one slow target, and one malformed response fail only that row; they do not + leak response content, secret material, or block other bounded calls. +3. A read requires the proposed exact Admin read scope on every target; a + mutation request is made only after explicit target review/confirmation and + results report each instance separately. +4. A malicious/incorrect inventory cannot cause calls to link-local, + loopback-unapproved, metadata, Unix-socket, or arbitrary redirected URLs. + Enforce HTTPS, allowlisted configured origins, no redirects, DNS/IP + re-validation at connection time, response/request caps, and proxy-from-env + disabled unless a reviewed proxy profile is selected. +5. The TUI never opens SSH, Docker, Kubernetes, systemd, FathomDB, or another + instance’s files/storage. Test instrumentation verifies no such dependency. +6. A deployment configuration change is represented as external desired state; + after the approved deployment controller restarts/rolls out an instance, the + TUI observes the new redacted fingerprint through Admin. It never writes a + provider credential/config file nor claims completion from a local desired + value. +7. Existing single-instance loopback TUI, container host-console TLS behavior, + inference routing/guardrails, and per-instance Admin actions retain current + behavior when fleet mode is disabled. + +## Architectural alignment and blast radius + +**Aligned:** this design preserves Airlock’s gateway ownership (LiteLLM remains +the transport substrate), a per-instance Airlock Admin PDP, local native +virtual-key store, and TUI-as-client boundary. It contains the new code to TUI +profile parsing, typed client fan-out, selection UX, Admin API version/capability +descriptors, tests, docs, and small additive Admin DTO fields. + +**Misaligned/rejected:** a central Airlock fleet database, central virtual-key +replication, LiteLLM database UI, TUI writing YAML/env files, SSH/Docker/systemd +execution, an embedded Ansible/Salt/Nomad/Consul/Flux/Argo control plane, +unreviewed automatic discovery, broad bearer-token reuse, and provider config +CRUD with no durable owner. + +The direct blast radius is high at the Admin authorization/transport edge and +medium in TUI UX; it is intentionally zero on the inference hot path. Fan-out +turns a valid but mistaken action into several independently authorized actions, +so target selection, limits, concurrency, dry-run/confirmation, audit +correlation, and partial-failure semantics are the primary safety controls. + +## HITL decisions before code + +1. Ratify the above **Admin-client-only** baseline versus authorizing a new + Airlock-owned centralized fleet/configuration service (recommended: baseline + only; defer the latter). +2. Select the initial deployment support matrix: native/systemd and + Docker/Compose profiles, Kubernetes GitOps integration guidance, or all + three. Do not claim generic remote lifecycle management without a selected + owner. +3. Ratify the instance identity model: who publishes inventory profiles; how + TLS CAs/SANs and per-instance token audiences/issuers are provisioned; + whether a shared credential is ever allowed; rotation/revocation/incident + response; and the operator secret-file/secret-manager contract. +4. Approve the exact fleet-safe action allowlist, maximum target count, + concurrency/timeouts, confirmation/dry-run rule, retry policy, and whether + an action requires a second operator/approval workflow. +5. Decide whether desired-state system-wide configuration is **documentation + only** in 0.5.15 (recommended) or whether Airlock needs a formally versioned + deployment-provider plug-in contract later. This decision must precede any + CRUD surface. +6. Approve SSRF/egress policy for inventory endpoints and the audit-retention + model for local fleet-operation summaries. + +## Research method and sources + +This is a web-research hypothesis using primary project documentation and +official GitHub project pages where a Python integration was assessed. It does +not infer vendor adoption as a substitute for Airlock’s security boundary. The +candidate table links each source directly; especially relevant primary sources +are the [Ansible systemd module](https://docs.ansible.com/projects/ansible/latest/collections/ansible/builtin/systemd_service_module.html), +[OpenTofu sensitive-state guidance](https://opentofu.org/docs/language/state/sensitive-data/), +[Kubernetes Secret guidance](https://kubernetes.io/docs/concepts/configuration/secret/), +[Flux architecture overview](https://fluxcd.io/flux/), +[Argo CD overview](https://argo-cd.readthedocs.io/en/stable/), +[Nomad job model](https://developer.hashicorp.com/nomad/docs/concepts/job), +and [Consul discovery overview](https://developer.hashicorp.com/consul/docs/discover). diff --git a/dev/plans/0.5.15/70-fleet-admin-fanout-status.md b/dev/plans/0.5.15/70-fleet-admin-fanout-status.md new file mode 100644 index 0000000..be14afc --- /dev/null +++ b/dev/plans/0.5.15/70-fleet-admin-fanout-status.md @@ -0,0 +1,48 @@ +# Slice 70 — multi-instance Admin fan-out status + +**Status:** conditional — planning/design re-evaluation complete; feature +implementation is blocked on the fleet authority/HITL decision. + +## Re-evaluation evidence + +- The draft is research-only and allocates DFR-40/DAC-40 only as a proposal; + canonical requirements have not ratified a fleet feature. +- Slice 6 makes fleet authority conditional: named inventory, per-instance + TLS/token isolation, operation limits, and an external desired-state boundary + must be approved before scheduling. +- Slice 40 supplies a bounded redacted configuration projection; Slice 50 + supplies a constrained single-target host-console connection. Neither is a + fleet transport or a cross-instance authority. +- Independent scope audit and design review found no inventory parser/owner, + target selection/count/concurrency model, per-target audience, hardened + redirect/proxy/body/DNS/SSRF controls, aggregate result model, correlation + audit contract, or fleet test coverage. + +## Rejected implementation paths + +- Do not reuse the ordinary `urllib` Admin client as a fleet client: it lacks + redirect/proxy/response-cap/DNS-revalidation/fan-out safety controls. +- Do not infer authorization from Docker, a subnet, service discovery, host + management, or a shared bearer credential. +- Do not add SSH, Docker, systemd, Kubernetes, Ansible, Terraform, Nomad, + Consul, or a central Airlock fleet/configuration store to the TUI. +- Do not write desired configuration or lifecycle state; deployment controllers + remain the external owner and the TUI can only observe target Admin state. + +## Decision-ready baseline and acceptance plan + +The reviewed minimum candidate is a read-only, static owner-only local +inventory: per-target literal HTTPS origin, CA/token-file references, distinct +credentials, explicit target selection, no wildcard/default-all, bounded +concurrency/time/body, TLS name validation, no redirects/proxy environment, +and secret-blind per-target outcomes. The eventual RED suite must cover malformed +profiles; target/credential cross-use; CA/hostname/audience/scope failures; +metadata/link-local/redirect/proxy/DNS-rebinding denial; slow-target isolation; +selection and bound enforcement; no host/orchestrator imports; no secret/raw +error rendering; and ordinary TUI/Admin/inference regressions. + +Before code, the owner must ratify inventory authority, topology and identity +contract, private-network/DNS egress rule, mutation semantics (or read-only +v1), operation/audit retention, and the external desired-state boundary. A +later Slice 70 design review must then turn those decisions into canonical +requirements and executable acceptance criteria before RED/GREEN work begins. From e314f150a927674da6ec3950df1cfc6264090302 Mon Sep 17 00:00:00 2001 From: Corey Thompson Date: Sun, 16 Aug 2026 21:19:37 -0500 Subject: [PATCH 15/24] feat(tui): add bounded same-host fleet view --- airlock/admin/policy.py | 24 +- airlock/cli/main.py | 23 ++ airlock/tui/fleet_app.py | 90 ++++++ airlock/tui/fleet_client.py | 265 ++++++++++++++++ airlock/tui/fleet_profile.py | 145 +++++++++ ...stance-tui-fleet-control-plane-research.md | 32 +- .../0.5.15/70-fleet-admin-fanout-status.md | 48 ++- dev/requirements.md | 38 +++ docs/getting-started/configuration.md | 6 + docs/guide/admin-api.md | 53 ++++ docs/guide/cli.md | 6 + tests/test_admin_http.py | 54 ++++ tests/test_fleet_profile.py | 295 ++++++++++++++++++ tests/test_tui.py | 26 ++ 14 files changed, 1089 insertions(+), 16 deletions(-) create mode 100644 airlock/tui/fleet_app.py create mode 100644 airlock/tui/fleet_client.py create mode 100644 airlock/tui/fleet_profile.py create mode 100644 tests/test_fleet_profile.py diff --git a/airlock/admin/policy.py b/airlock/admin/policy.py index bacf9e2..fe5f15f 100644 --- a/airlock/admin/policy.py +++ b/airlock/admin/policy.py @@ -23,6 +23,7 @@ "admin:read_config", "admin:clear_quarantine", } +_FLEET_READ_TUI_SCOPES = {_REMOTE_TUI_SCOPE, "admin:read"} _REMOTE_TUI_MAX_TTL_SECONDS = 15 * 60 # Deliberately excludes "" — a missing/stripped client address is NOT loopback @@ -37,6 +38,7 @@ class AdminConfig: allow_insecure_tokens: bool = False behind_tls_proxy: bool = False remote_tui: bool = False + fleet_read_tui: bool = False _admin_config = AdminConfig() @@ -62,7 +64,10 @@ def configure_admin( allow_insecure_tokens=bool(block.get("allow_insecure_tokens", False)), behind_tls_proxy=bool(block.get("behind_tls_proxy", False)), remote_tui=bool(block.get("remote_tui", False)), + fleet_read_tui=bool(block.get("fleet_read_tui", False)), ) + if cfg.fleet_read_tui and not cfg.remote_tui: + raise RuntimeError("admin.fleet_read_tui requires admin.remote_tui: true") if cfg.remote_tui and host is not None: if ( not cfg.enabled @@ -136,7 +141,12 @@ def decide( master = _master_key() if master and hmac.compare_digest(principal.bearer, master): return Decision(False, 403, "master key is not valid for remote TUI") - from airlock.admin.tokens import TokenError, has_scope, verify_token + from airlock.admin.tokens import ( + TokenError, + has_scope, + token_scopes, + verify_token, + ) try: claims = verify_token(principal.bearer) @@ -158,6 +168,18 @@ def decide( return Decision( False, 403, "token missing scope admin:remote_tui", actor=actor ) + scopes = token_scopes(claims) + if cfg.fleet_read_tui and ( + op_scope != "admin:read" + or len(scopes) != len(_FLEET_READ_TUI_SCOPES) + or set(scopes) != _FLEET_READ_TUI_SCOPES + ): + return Decision( + False, + 403, + "fleet read TUI requires exactly admin:remote_tui and admin:read", + actor=actor, + ) if op_scope not in _REMOTE_TUI_ALLOWED_SCOPES: return Decision( False, 403, "operation is not available to remote TUI", actor=actor diff --git a/airlock/cli/main.py b/airlock/cli/main.py index 3d18e89..3c3d920 100644 --- a/airlock/cli/main.py +++ b/airlock/cli/main.py @@ -152,6 +152,11 @@ def main(argv: list[str] | None = None) -> None: default=None, help="CA bundle required with --remote-admin.", ) + tui_parser.add_argument( + "--fleet-inventory", + default=None, + help="Owner-only same-host read-only fleet inventory YAML.", + ) # -- analyze -- analyze_parser = subparsers.add_parser( @@ -518,6 +523,24 @@ def main(argv: list[str] | None = None) -> None: host = args.host or os.environ.get("AIRLOCK_HOST", "localhost") port = args.port or os.environ.get("AIRLOCK_PORT", "4000") + if args.fleet_inventory: + if args.remote_admin or args.admin_token_file or args.admin_ca_file: + parser.error( + "tui --fleet-inventory cannot be combined with remote Admin options" + ) + if ( + args.start + or args.daemon + or args.host is not None + or args.port is not None + ): + parser.error( + "tui --fleet-inventory cannot start, own, or target a local proxy" + ) + from airlock.tui.fleet_app import run as fleet_tui_run + + fleet_tui_run(inventory_file=args.fleet_inventory) + return if args.remote_admin: if not args.admin_token_file or not args.admin_ca_file: parser.error( diff --git a/airlock/tui/fleet_app.py b/airlock/tui/fleet_app.py new file mode 100644 index 0000000..b7071ab --- /dev/null +++ b/airlock/tui/fleet_app.py @@ -0,0 +1,90 @@ +"""Manual, read-only, same-host fleet Admin TUI for Slice 70.""" + +from __future__ import annotations + +from pathlib import Path + +from textual import work +from textual.app import App, ComposeResult +from textual.containers import Vertical +from textual.widgets import Button, DataTable, Footer, Header, Input, Static + +from airlock.tui.fleet_client import FleetAdminClient, FleetResult +from airlock.tui.fleet_profile import FleetProfileError, load_fleet_profile + + +class FleetAdminApp(App): + """Explicit-selection view. It has no polling, actions, or local persistence.""" + + TITLE = "Airlock Fleet Admin" + SUB_TITLE = "Read-only same-host inventory" + + def __init__(self, client: FleetAdminClient) -> None: + super().__init__() + self._client = client + + def compose(self) -> ComposeResult: + yield Header(show_clock=True) + with Vertical(): + yield Static( + "Enter one or more configured IDs, separated by commas. " + "Nothing refreshes automatically.", + id="fleet-status", + ) + yield Input(placeholder="target-a,target-b", id="fleet-selection") + yield Button("Refresh selected", id="fleet-refresh") + yield DataTable(id="fleet-results") + yield Footer() + + def on_mount(self) -> None: + table = self.query_one("#fleet-results", DataTable) + table.add_columns("ID", "Name", "Status", "Providers", "Observed") + + def on_unmount(self) -> None: + self._client.close() + + def on_button_pressed(self, event: Button.Pressed) -> None: + if event.button.id != "fleet-refresh": + return + selected = [ + instance_id.strip() + for instance_id in self.query_one("#fleet-selection", Input).value.split( + "," + ) + if instance_id.strip() + ] + try: + if not selected: + raise ValueError("select one or more configured target IDs") + self._refresh(selected) + except ValueError as exc: + self.query_one("#fleet-status", Static).update(str(exc)) + + @work(thread=True, group="fleet-refresh", exclusive=True) + def _refresh(self, selected: list[str]) -> None: + results = self._client.refresh(selected) + self.call_from_thread(self._render, results) + + def _render(self, results: list[FleetResult]) -> None: + table = self.query_one("#fleet-results", DataTable) + table.clear() + for result in results: + table.add_row( + result.instance_id, + result.display_name, + result.state, + str(result.provider_count) + if result.provider_count is not None + else "—", + result.observed_at or "—", + ) + self.query_one("#fleet-status", Static).update("Refresh complete.") + + +def run(*, inventory_file: str) -> None: + """Load one protected inventory and run only the fleet view.""" + try: + targets = load_fleet_profile(Path(inventory_file)) + except FleetProfileError as exc: + raise SystemExit(f"error: {exc}") from None + FleetAdminApp(FleetAdminClient(targets)).run() diff --git a/airlock/tui/fleet_client.py b/airlock/tui/fleet_client.py new file mode 100644 index 0000000..5e3c758 --- /dev/null +++ b/airlock/tui/fleet_client.py @@ -0,0 +1,265 @@ +"""Bounded direct transport for Slice 70's read-only loopback fleet view.""" + +from __future__ import annotations + +import ipaddress +import json +import socket +import ssl +import time +from concurrent.futures import ThreadPoolExecutor, TimeoutError, as_completed +from dataclasses import dataclass +from datetime import UTC, datetime +from typing import Callable + +from airlock.tui.fleet_profile import FleetTarget + +_MAX_SELECTED = 10 +_MAX_CONCURRENCY = 4 +_CONNECT_TIMEOUT = 2.0 +_TOTAL_TIMEOUT = 5.0 +_MAX_BODY_BYTES = 64 * 1024 + + +class FleetTransportError(RuntimeError): + """Secret-blind, classified fleet transport failure.""" + + def __init__(self, category: str): + super().__init__(category) + self.category = category + + +@dataclass(frozen=True) +class FleetResult: + instance_id: str + display_name: str + state: str + observed_at: str | None = None + provider_count: int | None = None + + +Resolver = Callable[[str, int], list[tuple[int, tuple]]] + + +def _host_header(host: str, port: str) -> str: + """Format the authority without making an IPv6 origin ambiguous.""" + return f"[{host}]:{port}" if ":" in host else f"{host}:{port}" + + +def _resolve_loopback(host: str, port: int) -> list[tuple[int, tuple]]: + try: + answers = socket.getaddrinfo(host, port, type=socket.SOCK_STREAM) + except OSError as exc: + raise FleetTransportError("unavailable") from exc + vetted = [(family, sockaddr) for family, _type, _proto, _canon, sockaddr in answers] + if not vetted or any( + not ipaddress.ip_address(sockaddr[0].split("%", 1)[0]).is_loopback + for _family, sockaddr in vetted + ): + raise FleetTransportError("loopback") + return vetted + + +class FleetAdminClient: + """Direct, non-proxy, non-redirecting Admin reader with bounded fan-out.""" + + def __init__( + self, targets: list[FleetTarget], *, resolver: Resolver = _resolve_loopback + ): + self._targets = {target.instance_id: target for target in targets} + self._resolver = resolver + self._last_success: dict[str, FleetResult] = {} + self._resolver_pool = ThreadPoolExecutor( + max_workers=_MAX_CONCURRENCY, thread_name_prefix="airlock-fleet-resolve" + ) + + def close(self) -> None: + """Cancel queued lookups when the UI exits.""" + self._resolver_pool.shutdown(wait=False, cancel_futures=True) + + def _read_response( + self, tls_socket: ssl.SSLSocket, deadline: float + ) -> tuple[int, dict]: + raw = bytearray() + while True: + remaining = deadline - time.monotonic() + if remaining <= 0: + raise FleetTransportError("unavailable") + tls_socket.settimeout(remaining) + chunk = tls_socket.recv(4096) + if not chunk: + break + raw.extend(chunk) + if len(raw) > _MAX_BODY_BYTES + 16 * 1024: + raise FleetTransportError("unavailable") + head, marker, body = bytes(raw).partition(b"\r\n\r\n") + if not marker or len(body) > _MAX_BODY_BYTES: + raise FleetTransportError("unavailable") + lines = head.split(b"\r\n") + try: + status = int(lines[0].split()[1]) + except (IndexError, ValueError): + raise FleetTransportError("unavailable") from None + if status != 200: + return status, {} + try: + payload = json.loads(body or b"{}") + except (TypeError, ValueError): + raise FleetTransportError("unavailable") from None + if not isinstance(payload, dict): + raise FleetTransportError("unavailable") + return status, payload + + @staticmethod + def _open_tls( + family: int, + address: tuple, + context: ssl.SSLContext, + hostname: str, + deadline: float, + ) -> ssl.SSLSocket: + remaining = deadline - time.monotonic() + if remaining <= 0: + raise FleetTransportError("unavailable") + raw_socket = socket.socket(family, socket.SOCK_STREAM) + raw_socket.settimeout(min(_CONNECT_TIMEOUT, remaining)) + raw_socket.connect(address) + remaining = deadline - time.monotonic() + if remaining <= 0: + raw_socket.close() + raise FleetTransportError("unavailable") + raw_socket.settimeout(remaining) + tls_socket = context.wrap_socket(raw_socket, server_hostname=hostname) + remaining = deadline - time.monotonic() + if remaining <= 0: + tls_socket.close() + raise FleetTransportError("unavailable") + tls_socket.settimeout(remaining) + return tls_socket + + def _resolve_with_deadline( + self, host: str, port: int, deadline: float + ) -> list[tuple[int, tuple]]: + remaining = deadline - time.monotonic() + if remaining <= 0: + raise FleetTransportError("unavailable") + future = self._resolver_pool.submit(self._resolver, host, port) + try: + return future.result(timeout=remaining) + except TimeoutError as exc: + future.cancel() + raise FleetTransportError("unavailable") from exc + + def _request(self, target: FleetTarget) -> tuple[int, dict]: + deadline = time.monotonic() + _TOTAL_TIMEOUT + answers = self._resolve_with_deadline(target.host, int(target.port), deadline) + if not answers or any( + not ipaddress.ip_address(sockaddr[0].split("%", 1)[0]).is_loopback + for _family, sockaddr in answers + ): + raise FleetTransportError("loopback") + try: + context = ssl.create_default_context(cadata=target.ca_pem) + context.check_hostname = True + last_error: OSError | None = None + for family, address in answers: + try: + tls_socket = self._open_tls( + family, address, context, target.host, deadline + ) + except ssl.SSLError: + raise + except OSError as exc: + last_error = exc + continue + peer = tls_socket.getpeername()[0].split("%", 1)[0] + if ( + peer != address[0].split("%", 1)[0] + or not ipaddress.ip_address(peer).is_loopback + ): + tls_socket.close() + raise FleetTransportError("loopback") + request = ( + f"GET /airlock/admin/providers HTTP/1.1\r\n" + f"Host: {_host_header(target.host, target.port)}\r\n" + f"Authorization: Bearer {target.token}\r\n" + "Accept: application/json\r\nConnection: close\r\n\r\n" + ).encode() + remaining = deadline - time.monotonic() + if remaining <= 0: + tls_socket.close() + raise FleetTransportError("unavailable") + tls_socket.settimeout(remaining) + tls_socket.sendall(request) + try: + return self._read_response(tls_socket, deadline) + finally: + tls_socket.close() + if last_error is not None: + raise last_error + raise FleetTransportError("unavailable") + except FleetTransportError: + raise + except ssl.SSLError as exc: + raise FleetTransportError("tls") from exc + except (OSError, ValueError) as exc: + raise FleetTransportError("unavailable") from exc + + def fetch(self, target: FleetTarget) -> FleetResult: + try: + status, payload = self._request(target) + except FleetTransportError as exc: + previous = self._last_success.get(target.instance_id) + if previous is not None: + return FleetResult( + target.instance_id, + target.display_name, + "stale", + previous.observed_at, + previous.provider_count, + ) + state = exc.category if exc.category == "tls" else "unavailable" + return FleetResult(target.instance_id, target.display_name, state) + if status == 401: + return FleetResult(target.instance_id, target.display_name, "auth") + if status == 403: + return FleetResult(target.instance_id, target.display_name, "forbidden") + if status != 200 or not isinstance(payload.get("providers"), dict): + return FleetResult(target.instance_id, target.display_name, "unavailable") + result = FleetResult( + target.instance_id, + target.display_name, + "fresh", + datetime.now(UTC).isoformat(), + len(payload["providers"]), + ) + self._last_success[target.instance_id] = result + return result + + def refresh(self, selected_ids: list[str]) -> list[FleetResult]: + if not 1 <= len(selected_ids) <= _MAX_SELECTED: + raise ValueError("select 1 through at most 10 targets") + if len(set(selected_ids)) != len(selected_ids): + raise ValueError("target selection contains duplicates") + try: + selected = [self._targets[instance_id] for instance_id in selected_ids] + except KeyError as exc: + raise ValueError("selected target is not configured") from exc + results: dict[str, FleetResult] = {} + with ThreadPoolExecutor( + max_workers=min(_MAX_CONCURRENCY, len(selected)) + ) as pool: + pending = { + pool.submit(self.fetch, target): target.instance_id + for target in selected + } + for future in as_completed(pending): + instance_id = pending[future] + try: + results[instance_id] = future.result() + except Exception: # pragma: no cover - defense at isolated boundary + target = self._targets[instance_id] + results[instance_id] = FleetResult( + instance_id, target.display_name, "unavailable" + ) + return [results[instance_id] for instance_id in selected_ids] diff --git a/airlock/tui/fleet_profile.py b/airlock/tui/fleet_profile.py new file mode 100644 index 0000000..e686d24 --- /dev/null +++ b/airlock/tui/fleet_profile.py @@ -0,0 +1,145 @@ +"""Strict local inventory for the read-only same-host fleet TUI.""" + +from __future__ import annotations + +import os +import stat +from dataclasses import dataclass, field +from pathlib import Path +from urllib.parse import urlsplit + +import yaml + +_LOOPBACK_HOSTS = {"localhost", "127.0.0.1", "::1"} +_MAX_TARGETS = 10 +_MAX_FILE_BYTES = 64 * 1024 +_MAX_FIELD_LENGTH = 256 +_TARGET_FIELDS = {"id", "name", "origin", "token_file", "ca_file"} + + +class FleetProfileError(ValueError): + """Inventory or local file does not satisfy fleet-v1 policy.""" + + +class _NoDuplicateSafeLoader(yaml.SafeLoader): + """Safe YAML loader that never silently overwrites a supplied key.""" + + def construct_mapping(self, node, deep=False): # type: ignore[no-untyped-def] + mapping = {} + for key_node, value_node in node.value: + key = self.construct_object(key_node, deep=deep) + if not isinstance(key, str): + raise FleetProfileError("fleet inventory has an invalid mapping key") + if key in mapping: + raise FleetProfileError("fleet inventory contains a duplicate key") + mapping[key] = self.construct_object(value_node, deep=deep) + return mapping + + +@dataclass(frozen=True) +class FleetTarget: + instance_id: str + display_name: str + host: str + port: str + token: str = field(repr=False) + ca_pem: str = field(repr=False) + + +def _read_secure_text(path_value: str | Path, label: str) -> str: + path = Path(path_value) + nofollow = getattr(os, "O_NOFOLLOW", None) + if nofollow is None: + raise FleetProfileError(f"{label} requires no-follow support") + try: + descriptor = os.open(path, os.O_RDONLY | nofollow) + except OSError as exc: + raise FleetProfileError(f"{label} must be a readable regular file") from exc + try: + with os.fdopen(descriptor, "r", encoding="utf-8") as handle: + info = os.fstat(handle.fileno()) + if not stat.S_ISREG(info.st_mode): + raise FleetProfileError(f"{label} must be a regular file") + if info.st_mode & 0o077: + raise FleetProfileError(f"{label} permissions must be owner-only") + if hasattr(os, "geteuid") and info.st_uid != os.geteuid(): + raise FleetProfileError(f"{label} must be owned by this user") + if info.st_size > _MAX_FILE_BYTES: + raise FleetProfileError(f"{label} is too large") + return handle.read() + except (OSError, UnicodeError) as exc: + raise FleetProfileError(f"{label} is unreadable") from exc + + +def parse_fleet_origin(origin: str) -> tuple[str, str]: + """Return a strict literal same-host HTTPS origin's host and explicit port.""" + if not isinstance(origin, str) or len(origin) > _MAX_FIELD_LENGTH: + raise FleetProfileError("invalid target origin") + try: + parsed = urlsplit(origin) + port = parsed.port + except ValueError as exc: + raise FleetProfileError("invalid target origin") from exc + host = (parsed.hostname or "").lower() + if ( + parsed.scheme != "https" + or parsed.username is not None + or parsed.password is not None + or host not in _LOOPBACK_HOSTS + or port is None + or not 1 <= port <= 65535 + or parsed.path not in ("", "/") + or parsed.query + or parsed.fragment + ): + raise FleetProfileError("target origin must be exact loopback HTTPS with port") + return host, str(port) + + +def _field(value: object, name: str) -> str: + if not isinstance(value, str) or not value or len(value) > _MAX_FIELD_LENGTH: + raise FleetProfileError(f"invalid target {name}") + return value + + +def load_fleet_profile(path: str | Path) -> list[FleetTarget]: + """Load bounded, owner-only inventory and per-target secret references.""" + raw = _read_secure_text(path, "fleet inventory") + try: + document = yaml.load(raw, Loader=_NoDuplicateSafeLoader) + except (yaml.YAMLError, FleetProfileError) as exc: + raise FleetProfileError("fleet inventory is invalid YAML") from exc + if not isinstance(document, dict) or set(document) != {"targets"}: + raise FleetProfileError("fleet inventory contains unsupported fields") + targets = document["targets"] + if not isinstance(targets, list) or not 1 <= len(targets) <= _MAX_TARGETS: + raise FleetProfileError("fleet inventory must contain 1 through 10 targets") + loaded: list[FleetTarget] = [] + ids: set[str] = set() + token_refs: set[str] = set() + ca_refs: set[str] = set() + for entry in targets: + if not isinstance(entry, dict): + raise FleetProfileError("fleet target must be a mapping") + if set(entry) != _TARGET_FIELDS: + raise FleetProfileError("fleet target contains unsupported fields") + instance_id = _field(entry.get("id"), "id") + name = _field(entry.get("name"), "name") + host, port = parse_fleet_origin(_field(entry.get("origin"), "origin")) + token_ref = _field(entry.get("token_file"), "token_file") + ca_ref = _field(entry.get("ca_file"), "ca_file") + if instance_id in ids or token_ref in token_refs or ca_ref in ca_refs: + raise FleetProfileError("duplicate target id or credential reference") + token = _read_secure_text(token_ref, "fleet token").strip() + if not token or not all( + char.isascii() and (char.isalnum() or char in "-_.") for char in token + ): + raise FleetProfileError("fleet token is invalid") + ca_pem = _read_secure_text(ca_ref, "fleet CA") + if not ca_pem.strip(): + raise FleetProfileError("fleet CA is empty") + ids.add(instance_id) + token_refs.add(token_ref) + ca_refs.add(ca_ref) + loaded.append(FleetTarget(instance_id, name, host, port, token, ca_pem)) + return loaded diff --git a/dev/plans/0.5.15/50-multi-instance-tui-fleet-control-plane-research.md b/dev/plans/0.5.15/50-multi-instance-tui-fleet-control-plane-research.md index 4b81d84..9f2d79e 100644 --- a/dev/plans/0.5.15/50-multi-instance-tui-fleet-control-plane-research.md +++ b/dev/plans/0.5.15/50-multi-instance-tui-fleet-control-plane-research.md @@ -1,10 +1,8 @@ # 0.5.15 research hypothesis — one TUI for multiple Airlock instances -**Status:** re-evaluated on 2026-08-16: **conditional, planning/design only; -not admitted for implementation.** This document adds no deployment agent, -remote-execution capability, or configuration write endpoint. The Slice 70 -audit and independent design review confirmed that the fleet authority/HITL -decisions below remain unresolved. +**Status:** ratified for the narrow Slice 70 read-only container v1 on +2026-08-16. This document adds no deployment agent, remote-execution capability, +or configuration write endpoint. ## Slice 70 admission update @@ -43,6 +41,30 @@ Until that record is ratified, the appropriate Slice 70 outcome is a status record and a decision-ready design, not code, tests, user documentation, or deployment changes. +## Ratified Slice 70 v1 (2026-08-16) + +The owner selected the conservative read-only baseline: + +- **Inventory:** static local owner-only YAML; IDs, display names, literal + origins, and CA/token-file references only. +- **Topology and identity:** only Slice 50 `admin.remote_tui` plus + `admin.fleet_read_tui` containers with native TLS and distinct host-loopback + ports; every target has distinct + CA/token references and a distinct `AIRLOCK_JWT_SECRET`. Tokens contain + exactly `admin:remote_tui` + `admin:read`; generic systemd and remote targets defer. +- **Egress:** only configured `localhost`/`127.0.0.1`/`::1` HTTPS origins; + resolve/revalidate loopback per connection; no redirects or proxy environment. +- **Operations:** read-only manual refresh, explicit selection/no wildcard, + at most 10 targets, concurrency 4, 2-second connect/5-second total timeout, + 64 KiB body cap, no automatic retry/poll, and no persisted local fleet audit. + +The hardened transport parses exact origins, securely reads every local file, +resolves all addresses per connection, connects only to a vetted numeric +loopback address with original-host TLS SNI/verification, and checks the peer +address. This closes the fleet authority/HITL gate only for this exact v1. Remote or +RFC1918 targets, discovery, shared credentials, mutation fan-out, desired-state +integration, scheduled polling, and retained fleet audit remain later decisions. + ## Outcome and hypothesis An operator should be able to open one `airlock tui`, select one or more named diff --git a/dev/plans/0.5.15/70-fleet-admin-fanout-status.md b/dev/plans/0.5.15/70-fleet-admin-fanout-status.md index be14afc..839f34c 100644 --- a/dev/plans/0.5.15/70-fleet-admin-fanout-status.md +++ b/dev/plans/0.5.15/70-fleet-admin-fanout-status.md @@ -1,12 +1,13 @@ # Slice 70 — multi-instance Admin fan-out status -**Status:** conditional — planning/design re-evaluation complete; feature -implementation is blocked on the fleet authority/HITL decision. +**Status:** complete — ratified same-host, read-only v1 implemented and +verified on 2026-08-16. Remote targets, discovery, mutations, desired-state +integration, and retained fleet audit remain out of scope. ## Re-evaluation evidence -- The draft is research-only and allocates DFR-40/DAC-40 only as a proposal; - canonical requirements have not ratified a fleet feature. +- The draft was research-only and allocated DFR-40/DAC-40 only as a proposal; + the owner ratified the narrow v1 contract on 2026-08-16. - Slice 6 makes fleet authority conditional: named inventory, per-instance TLS/token isolation, operation limits, and an external desired-state boundary must be approved before scheduling. @@ -36,13 +37,40 @@ inventory: per-target literal HTTPS origin, CA/token-file references, distinct credentials, explicit target selection, no wildcard/default-all, bounded concurrency/time/body, TLS name validation, no redirects/proxy environment, and secret-blind per-target outcomes. The eventual RED suite must cover malformed -profiles; target/credential cross-use; CA/hostname/audience/scope failures; +profiles; target/credential cross-use, including distinct signing-secret replay +denial; CA/hostname/scope failures; metadata/link-local/redirect/proxy/DNS-rebinding denial; slow-target isolation; selection and bound enforcement; no host/orchestrator imports; no secret/raw error rendering; and ordinary TUI/Admin/inference regressions. -Before code, the owner must ratify inventory authority, topology and identity -contract, private-network/DNS egress rule, mutation semantics (or read-only -v1), operation/audit retention, and the external desired-state boundary. A -later Slice 70 design review must then turn those decisions into canonical -requirements and executable acceptance criteria before RED/GREEN work begins. +The owner ratified inventory authority, topology and identity contract, +loopback-only DNS/egress rule, read-only mutation boundary, ephemeral result +handling, and the external desired-state boundary. The design review turned +these into canonical requirements and executable acceptance criteria before +RED/GREEN work began. + +## Ratification + +The owner selected the recommended static owner-only inventory, Slice 50 +remote-TUI container topology with distinct per-target signing secret/CA/token +references, loopback-only revalidated egress, and explicit bounded read-only +operation. Generic systemd is out of v1 because it does not establish the +remote-TUI PDP path. +The canonical DFR-40/DAC-40 record now fixes the 10-target, 4-concurrency, +2-second-connect/5-second-total, 64 KiB, manual/no-retry behavior. Design review +and RED/GREEN implementation may proceed only within that boundary. + +## Implementation record + +Implemented on the release worktree with a protected static inventory, a direct +loopback-only TLS transport, explicit selection, four-request bound, and a +read-only Textual view. Target servers additionally opt into +`admin.fleet_read_tui: true`; the PDP then accepts only a 15-minute capability +whose complete scope set is `admin:remote_tui` plus `admin:read`, preventing a +fleet credential from being repurposed for an Admin mutation. The target's +distinct signing secret remains the cross-target replay control. Verification +records focused RED/GREEN tests, policy tests, lint/format, strict docs, and +the release sync/verify gates. Final `make test` passed: 3,420 passed, 111 +deselected, 1 xpassed. A live Docker two-container topology was not run because +the sandbox cannot access the Docker daemon; static transport, server-PDP, and +cross-signing-secret regression tests cover that contract in-process. diff --git a/dev/requirements.md b/dev/requirements.md index bde7d0b..308f2bb 100644 --- a/dev/requirements.md +++ b/dev/requirements.md @@ -448,3 +448,41 @@ only `auth_context: remote_tui_jwt` to the existing `admin_action` audit record; they SHALL not record bearer material, raw address, certificate data, or request content. Existing direct-loopback Admin/TUI, operation-history, inference, routing, guardrail, and credential behavior SHALL remain unchanged. + +### DFR-40 / DAC-40: Same-host read-only fleet Admin view + +An explicit `airlock tui --fleet-inventory` mode SHALL provide only a manually +refreshed, read-only view of explicitly selected Airlock **container** targets +on the same host. Every target SHALL use the Slice 50 native-TLS host-loopback +profile (`admin.enabled: true`, `admin.remote_tui: true`, +`admin.fleet_read_tui: true`, `trust_loopback: false`), publish a distinct host-loopback port, and use a +distinct `AIRLOCK_JWT_SECRET`. Its static local YAML inventory SHALL be an +owner-only regular file and contain only an opaque ID, display name, literal +loopback HTTPS origin, and references to owner-only per-target CA and +capability-token files; it SHALL contain no secret value, provider +configuration, SSH/Docker/systemd/Kubernetes credential, discovery source, or +desired-state authority. Each target SHALL have distinct CA/token references; +its token SHALL contain `admin:remote_tui` and `admin:read` only. + +Fleet v1 SHALL allow only `localhost`, `127.0.0.1`, or `::1` origins with an +explicit port, HTTPS, certificate hostname validation, no proxy-from-environment +and no redirects. Inventory, CA, and token references SHALL be no-follow, +current-EUID-owned regular files with no group/other mode bits; inventory size, +target count, and field lengths are bounded, and duplicate IDs, token references, +or CA references are rejected. The transport SHALL parse an exact origin (no +userinfo/path/query/fragment), resolve every connection, reject it if **any** +answer is not loopback, connect only to a vetted numeric answer with TLS SNI and +hostname verification against the original host, then reject a peer address that +is not the vetted loopback address. It SHALL use a direct HTTP/1.1 connection, +not a proxy-aware URL client, and cap the response before parsing. The client +SHALL select at most 10 explicit IDs (no wildcard/default-all), issue at most +four concurrent requests, use 2-second connect and 5-second total timeouts, +accept at most 64 KiB response bodies, make no automatic retry/poll, and render +only bounded per-target fresh/stale/unavailable/TLS/auth/forbidden state without +raw errors. Stale state is previous in-memory success plus a displayed timestamp +only; it is never persisted. + +Fleet v1 SHALL make no Admin mutation, configuration/lifecycle/deployment +action, local persistent operation/audit record, or inference-path change. +Every target remains its own Admin authorization and audit authority. Existing +single-target local and remote TUI behavior SHALL remain unchanged. diff --git a/docs/getting-started/configuration.md b/docs/getting-started/configuration.md index 0b595bb..22714e2 100644 --- a/docs/getting-started/configuration.md +++ b/docs/getting-started/configuration.md @@ -550,6 +550,7 @@ admin: allow_insecure_tokens: false # permit token auth over plaintext on a non-loopback bind behind_tls_proxy: false # assert TLS is terminated upstream remote_tui: false # opt-in host-console container Admin profile + fleet_read_tui: false # opt-in exact read-only capability profile for Slice 70 targets ``` Tokens are signed with `AIRLOCK_JWT_SECRET` (falling back to a derivation from @@ -564,6 +565,11 @@ It requires native TLS, `trust_loopback: false`, and rejects proxy/insecure-toke modes. It accepts only a 15-minute JWT containing `admin:remote_tui` plus its operation scope; it never accepts the master key. See [Admin API](../guide/admin-api.md#host-console-container-tui) for the deployment contract. +`fleet_read_tui: true` is only for a target in the same-host read-only fleet +view. It additionally requires `remote_tui: true` and accepts only a 15-minute +token with **exactly** `admin:remote_tui` and `admin:read`; it prevents a +broader capability from authorizing a mutation on that target. + ### `guardrail_overrides` Allows trusted clients to downgrade specific content guards per request via an diff --git a/docs/guide/admin-api.md b/docs/guide/admin-api.md index 8711c22..9991c84 100644 --- a/docs/guide/admin-api.md +++ b/docs/guide/admin-api.md @@ -79,6 +79,7 @@ master key (and any loopback operator) satisfies all of them. | Scope | Grants | |---|---| | `admin:read` | The read-only `GET` endpoints | +| `admin:remote_tui` | Required anchor for the explicitly enabled host-console profiles below; grants no endpoint by itself | | `admin:read_config` | The redacted startup provider-configuration view only | | `admin:clear_quarantine` | Clear a provider or client→provider quarantine | | `admin:reset_circuit` | Reset a model circuit | @@ -183,6 +184,58 @@ loopback mapping), remove `remote_tui: true`, and restart. Do not publish this Admin port to all interfaces or substitute the master key, a Docker bridge, a CIDR, or forwarded headers for the capability token. +## Same-host fleet read view + +This is a deliberately small extension of the host-console container profile, +for an operator viewing several Airlock containers on the **same host**. It is +read-only: it does not manage containers, configuration, secrets, lifecycle, +or desired state, and it does not poll automatically. It is not remote fleet +control, service discovery, or a systemd/Kubernetes mode. + +Every target must use the preceding `remote_tui: true` profile plus +`fleet_read_tui: true`, and a separate +host-loopback published port, native TLS CA, and `AIRLOCK_JWT_SECRET`. Give its +token **exactly** `admin:remote_tui` and `admin:read`; the target rejects a +token carrying any other scope, so a fleet credential cannot authorize a +mutation through the Admin API. The distinct signing secret +prevents a token for one target being replayed at another. + +Create owner-only regular files (`0600`, owned by the user running the TUI) for +the inventory, each CA bundle, and each token. A profile contains references, +never embedded secret values: + +```yaml +targets: + - id: payments + name: Payments proxy + origin: https://localhost:4101 + ca_file: /secure/payments-ca.pem + token_file: /secure/payments-read.jwt + - id: support + name: Support proxy + origin: https://127.0.0.1:4102 + ca_file: /secure/support-ca.pem + token_file: /secure/support-read.jwt +``` + +Run it with explicit manual selection: + +```bash +airlock tui --fleet-inventory /secure/airlock-fleet.yaml +``` + +The inventory accepts at most ten distinct targets. Origins are exact HTTPS +loopback origins (`localhost`, `127.0.0.1`, or `[::1]`) with an explicit port; +no path, query, credentials, redirects, proxies, or remote/private-network +address is admitted. Airlock resolves and validates loopback addresses for each +connection, validates the TLS hostname and peer address, and directly connects +only to the vetted address. Refresh only named selected targets (maximum ten, +four concurrent), uses a 2-second connect and 5-second total timeout, and caps +responses at 64 KiB. Results show only `fresh`, `stale`, authentication, +authorization, TLS, or unavailable state; no token or raw remote error is +displayed or written locally. Stop using the command or remove the inventory +to roll it back. + ## Endpoints All routes are under `/airlock/admin/` and only exist when `admin.enabled: true`. diff --git a/docs/guide/cli.md b/docs/guide/cli.md index f5ca4e3..6561f42 100644 --- a/docs/guide/cli.md +++ b/docs/guide/cli.md @@ -73,10 +73,16 @@ airlock tui --start # start proxy + dashboard airlock tui # dashboard only (connect to running proxy) airlock tui --start --daemon # start proxy and leave it running after the TUI exits airlock tui --host H --port P # monitor a proxy on a specific host/port +airlock tui --fleet-inventory /secure/fleet.yaml # manual read-only same-host fleet view ``` See [TUI Dashboard](tui.md) for screen details. +`--fleet-inventory` is isolated from the ordinary dashboard and remote Admin +mode: it cannot start or own a proxy and requires explicit target selection for +each refresh. See [Admin API](admin-api.md#same-host-fleet-read-view) for the +owner-only inventory, TLS, and loopback-only deployment contract. + ### `airlock analyze` Run offline log analysis. diff --git a/tests/test_admin_http.py b/tests/test_admin_http.py index 0c58cd0..d33d872 100644 --- a/tests/test_admin_http.py +++ b/tests/test_admin_http.py @@ -101,6 +101,52 @@ def test_remote_tui_rejects_unallowed_scope_and_long_lived_token(self): long_lived = mint_token("operator:one", ["admin:remote_tui", "admin:read"], 901) assert not decide(Principal(bearer=long_lived), "admin:read").allowed + def test_fleet_read_profile_requires_exact_read_capability(self): + configure_admin( + { + "admin": { + "enabled": True, + "trust_loopback": False, + "remote_tui": True, + "fleet_read_tui": True, + } + }, + host="0.0.0.0", + tls_enabled=True, + ) + read_token = mint_token("fleet:one", ["admin:remote_tui", "admin:read"], 60) + assert decide(Principal(bearer=read_token), "admin:read").allowed + broad_token = mint_token( + "fleet:one", + ["admin:remote_tui", "admin:read", "admin:clear_quarantine"], + 60, + ) + assert not decide(Principal(bearer=broad_token), "admin:read").allowed + assert not decide( + Principal(bearer=broad_token), "admin:clear_quarantine" + ).allowed + + def test_fleet_token_cannot_replay_across_distinct_signing_secrets( + self, monkeypatch + ): + configure_admin( + { + "admin": { + "enabled": True, + "trust_loopback": False, + "remote_tui": True, + "fleet_read_tui": True, + } + }, + host="0.0.0.0", + tls_enabled=True, + ) + monkeypatch.setenv("AIRLOCK_JWT_SECRET", "target-one-signing-secret-000000") + token = mint_token("fleet:one", ["admin:remote_tui", "admin:read"], 60) + assert decide(Principal(bearer=token), "admin:read").allowed + monkeypatch.setenv("AIRLOCK_JWT_SECRET", "target-two-signing-secret-000000") + assert not decide(Principal(bearer=token), "admin:read").allowed + class TestConfigureAdminFailClosed: def test_exposed_no_tls_raises(self): @@ -123,6 +169,14 @@ def test_loopback_ok(self): def test_disabled_no_check(self): configure_admin({"admin": {"enabled": False}}, host="0.0.0.0") # no raise + def test_fleet_read_tui_requires_remote_tui_profile(self): + with pytest.raises(RuntimeError, match="fleet_read_tui"): + configure_admin( + {"admin": {"enabled": True, "fleet_read_tui": True}}, + host="0.0.0.0", + tls_enabled=True, + ) + @pytest.mark.parametrize( "config,host,tls", [ diff --git a/tests/test_fleet_profile.py b/tests/test_fleet_profile.py new file mode 100644 index 0000000..1bca8db --- /dev/null +++ b/tests/test_fleet_profile.py @@ -0,0 +1,295 @@ +"""RED contracts for the bounded same-host Slice 70 fleet client.""" + +from __future__ import annotations + +import socket +import time + +import pytest + +from airlock.tui.fleet_profile import ( + FleetProfileError, + FleetTarget, + load_fleet_profile, + parse_fleet_origin, +) + + +def _secure(path, text): + path.write_text(text) + path.chmod(0o600) + return path + + +def _profile(tmp_path, targets): + return _secure(tmp_path / "fleet.yaml", "targets:\n" + "\n".join(targets)) + + +def test_profile_loads_only_distinct_secure_loopback_targets(tmp_path): + ca1 = _secure(tmp_path / "one-ca.pem", "CA one") + token1 = _secure(tmp_path / "one.jwt", "abc.def.ghi") + ca2 = _secure(tmp_path / "two-ca.pem", "CA two") + token2 = _secure(tmp_path / "two.jwt", "jkl.mno.pqr") + profile = _profile( + tmp_path, + [ + " - id: one\n name: One\n origin: https://localhost:4101\n" + f" ca_file: {ca1}\n token_file: {token1}", + " - id: two\n name: Two\n origin: https://127.0.0.1:4102\n" + f" ca_file: {ca2}\n token_file: {token2}", + ], + ) + + targets = load_fleet_profile(profile) + + assert [target.instance_id for target in targets] == ["one", "two"] + assert targets[0].host == "localhost" + assert targets[0].token == "abc.def.ghi" + assert targets[0].ca_pem == "CA one" + + +@pytest.mark.parametrize( + "origin", + [ + "http://localhost:4000", + "https://10.0.0.1:4000", + "https://localhost", + "https://localhost:4000/path", + "https://user@localhost:4000", + "https://localhost:4000?x=1", + ], +) +def test_origin_rejects_nonexact_loopback_https(origin): + with pytest.raises(FleetProfileError): + parse_fleet_origin(origin) + + +def test_profile_rejects_duplicate_secret_references_and_insecure_inventory(tmp_path): + ca = _secure(tmp_path / "ca.pem", "CA") + token = _secure(tmp_path / "token.jwt", "abc.def.ghi") + profile = _profile( + tmp_path, + [ + " - id: one\n name: One\n origin: https://localhost:4101\n" + f" ca_file: {ca}\n token_file: {token}", + " - id: two\n name: Two\n origin: https://localhost:4102\n" + f" ca_file: {ca}\n token_file: {token}", + ], + ) + with pytest.raises(FleetProfileError, match="duplicate"): + load_fleet_profile(profile) + profile.chmod(0o644) + with pytest.raises(FleetProfileError, match="permissions"): + load_fleet_profile(profile) + + +def test_profile_rejects_embedded_or_unsupported_values(tmp_path): + ca = _secure(tmp_path / "ca.pem", "CA") + token = _secure(tmp_path / "token.jwt", "abc.def.ghi") + profile = _profile( + tmp_path, + [ + " - id: one\n name: One\n origin: https://localhost:4101\n" + f" ca_file: {ca}\n token_file: {token}\n token: must-not-be-here" + ], + ) + with pytest.raises(FleetProfileError, match="unsupported"): + load_fleet_profile(profile) + + +@pytest.mark.parametrize( + "extra", + [ + "provider_config: never", + "token_file: /duplicate.jwt", + ], +) +def test_profile_rejects_unknown_root_and_duplicate_yaml_keys(tmp_path, extra): + ca = _secure(tmp_path / "ca.pem", "CA") + token = _secure(tmp_path / "token.jwt", "abc.def.ghi") + profile = _secure( + tmp_path / "fleet.yaml", + "targets:\n" + " - id: one\n name: One\n origin: https://localhost:4101\n" + f" ca_file: {ca}\n token_file: {token}\n {extra}\n", + ) + with pytest.raises(FleetProfileError): + load_fleet_profile(profile) + + profile.write_text("targets: []\nprovider_config: never\n") + with pytest.raises(FleetProfileError, match="unsupported"): + load_fleet_profile(profile) + + +def test_fleet_client_rejects_rebinding_and_peer_mismatch(monkeypatch): + from airlock.tui.fleet_client import FleetAdminClient, FleetTransportError + + target = FleetTarget("one", "One", "localhost", "4101", "jwt", "CA") + client = FleetAdminClient( + [target], + resolver=lambda _host, _port: [ + (socket.AF_INET, ("127.0.0.1", 4101)), + (socket.AF_INET, ("10.0.0.1", 4101)), + ], + ) + with pytest.raises(FleetTransportError, match="loopback"): + client._request(target) + + +def test_fleet_client_rejects_a_connected_peer_other_than_vetted_address(monkeypatch): + from airlock.tui.fleet_client import FleetAdminClient, FleetTransportError + + class FakeSocket: + def settimeout(self, _timeout): + pass + + def getpeername(self): + return ("127.0.0.2", 4101) + + def close(self): + pass + + target = FleetTarget("one", "One", "localhost", "4101", "jwt", "CA") + client = FleetAdminClient( + [target], resolver=lambda _host, _port: [(socket.AF_INET, ("127.0.0.1", 4101))] + ) + + class FakeContext: + check_hostname = True + + monkeypatch.setattr( + "airlock.tui.fleet_client.ssl.create_default_context", + lambda **_kwargs: FakeContext(), + ) + monkeypatch.setattr(client, "_open_tls", lambda *_args: FakeSocket()) + + with pytest.raises(FleetTransportError, match="loopback"): + client._request(target) + + +def test_fleet_client_limits_selection_and_isolates_target_failure(): + from airlock.tui.fleet_client import FleetAdminClient + + targets = [ + FleetTarget(str(i), str(i), "127.0.0.1", str(4100 + i), "jwt", "CA") + for i in range(11) + ] + client = FleetAdminClient(targets) + with pytest.raises(ValueError, match="at most 10"): + client.refresh([str(i) for i in range(11)]) + with pytest.raises(ValueError, match="not configured"): + client.refresh(["missing"]) + + +def test_fleet_client_bounds_a_slow_resolver_and_never_renders_loopback(): + from airlock.tui.fleet_client import FleetAdminClient + + target = FleetTarget("one", "One", "localhost", "4101", "jwt", "CA") + + def slow_resolver(_host, _port): + time.sleep(10) + return [(socket.AF_INET, ("127.0.0.1", 4101))] + + client = FleetAdminClient([target], resolver=slow_resolver) + started = time.monotonic() + result = client.fetch(target) + assert time.monotonic() - started < 5.5 + assert result.state == "unavailable" + + +@pytest.mark.parametrize( + ("outcome", "expected"), + [ + ((401, {}), "auth"), + ((403, {}), "forbidden"), + ((302, {}), "unavailable"), + ], +) +def test_fleet_client_classifies_only_permitted_result_states( + monkeypatch, outcome, expected +): + from airlock.tui.fleet_client import FleetAdminClient + + target = FleetTarget("one", "One", "localhost", "4101", "jwt", "CA") + client = FleetAdminClient([target]) + monkeypatch.setattr(client, "_request", lambda _target: outcome) + assert client.fetch(target).state == expected + + +def test_fleet_client_caps_response_before_json_parsing(): + from airlock.tui.fleet_client import ( + _MAX_BODY_BYTES, + FleetAdminClient, + FleetTransportError, + ) + + class ResponseSocket: + def __init__(self): + self._chunks = [ + b"HTTP/1.1 200 OK\r\nContent-Type: application/json\r\n\r\n", + b"x" * (_MAX_BODY_BYTES + 1), + ] + + def settimeout(self, _timeout): + pass + + def recv(self, _size): + return self._chunks.pop(0) if self._chunks else b"" + + with pytest.raises(FleetTransportError, match="unavailable"): + FleetAdminClient([])._read_response(ResponseSocket(), time.monotonic() + 1) + + +def test_fleet_target_repr_and_fleet_ui_are_secret_blind_and_manual(): + import inspect + + from airlock.tui import fleet_app + + target = FleetTarget("one", "One", "localhost", "4101", "very-secret", "CA") + assert "very-secret" not in repr(target) + assert "set_interval" not in inspect.getsource(fleet_app.FleetAdminApp) + + +def test_fleet_ipv6_origin_uses_a_bracketed_http_host_header(): + from airlock.tui.fleet_client import _host_header + + assert _host_header("::1", "4101") == "[::1]:4101" + + +def test_fleet_resolver_pool_is_bounded_and_tls_handshake_uses_deadline(monkeypatch): + from airlock.tui.fleet_client import FleetAdminClient + + class RawSocket: + def __init__(self): + self.timeouts = [] + + def settimeout(self, timeout): + self.timeouts.append(timeout) + + def connect(self, _address): + pass + + def close(self): + pass + + class TlsSocket: + def settimeout(self, _timeout): + pass + + class Context: + def wrap_socket(self, _raw, **_kwargs): + return TlsSocket() + + client = FleetAdminClient([]) + assert client._resolver_pool._max_workers == 4 + raw = RawSocket() + moments = iter((0.0, 1.5, 2.0)) + monkeypatch.setattr("airlock.tui.fleet_client.socket.socket", lambda *_args: raw) + monkeypatch.setattr( + "airlock.tui.fleet_client.time.monotonic", lambda: next(moments) + ) + + client._open_tls(socket.AF_INET, ("127.0.0.1", 4101), Context(), "localhost", 5.0) + + assert raw.timeouts == [2.0, 3.5] + client.close() diff --git a/tests/test_tui.py b/tests/test_tui.py index ab57065..048db59 100644 --- a/tests/test_tui.py +++ b/tests/test_tui.py @@ -502,6 +502,32 @@ def test_remote_tui_rejects_proxy_start() -> None: assert exc_info.value.code == 2 +def test_fleet_tui_uses_isolated_fleet_app() -> None: + from airlock.cli.main import main + + with mock.patch("airlock.tui.fleet_app.run") as fleet_run: + main(["tui", "--fleet-inventory", "/secure/fleet.yaml"]) + fleet_run.assert_called_once_with(inventory_file="/secure/fleet.yaml") + + +def test_fleet_tui_rejects_proxy_ownership_and_remote_flags() -> None: + from airlock.cli.main import main + + with pytest.raises(SystemExit) as exc_info: + main(["tui", "--fleet-inventory", "/secure/fleet.yaml", "--start"]) + assert exc_info.value.code == 2 + with pytest.raises(SystemExit) as exc_info: + main( + [ + "tui", + "--fleet-inventory", + "/secure/fleet.yaml", + "--remote-admin", + ] + ) + assert exc_info.value.code == 2 + + async def test_app_has_proxy_manager() -> None: app = AirlockApp(host="127.0.0.1", port="9999") from airlock.tui.proxy_manager import ProxyManager From d062f5e81a2e3aafb1abccb78df7f2813cfac2a5 Mon Sep 17 00:00:00 2001 From: Corey Thompson Date: Sun, 16 Aug 2026 21:34:18 -0500 Subject: [PATCH 16/24] docs(plan): add Slice 71 fleet topology verification --- ...71-slice70-docker-topology-verification.md | 160 ++++++++++++++++++ dev/plans/0.5.15/README.md | 4 +- 2 files changed, 163 insertions(+), 1 deletion(-) create mode 100644 dev/plans/0.5.15/71-slice70-docker-topology-verification.md diff --git a/dev/plans/0.5.15/71-slice70-docker-topology-verification.md b/dev/plans/0.5.15/71-slice70-docker-topology-verification.md new file mode 100644 index 0000000..c19cd1b --- /dev/null +++ b/dev/plans/0.5.15/71-slice70-docker-topology-verification.md @@ -0,0 +1,160 @@ +# Slice 71 — Slice 70 Docker topology verification + +**Status:** admitted — verification-only design approved after FIX-3 on +2026-08-16. Slice 70 closed at `e314f15` with unit/static evidence and an +approved local Docker access path; Slice 71 may now begin RED/GREEN work. + +## Purpose and scope + +Slice 70 delivers a manual, read-only same-host fleet view. Its fake transport, +PDP, and inventory tests prove the policy mechanics, but they do not prove two +real Airlock containers can safely operate through distinct host-loopback TLS +ports. Slice 71 closes that missing evidence only. + +It SHALL add a disposable, opt-in Docker integration test and its execution +target. It SHALL NOT add a new control plane, Admin route, fleet action, +inventory format, dependency, deployment manifest, production configuration, +or Docker-derived authorization. The standard Compose deployment and ordinary +`make test` remain unchanged. + +## Re-evaluated allocation + +| DFR-40/DAC-40 contract | Slice 70 evidence | Slice 71 acceptance evidence | +| --- | --- | --- | +| Explicit isolated fleet mode | CLI isolation tests | Retain regression coverage. | +| Exact remote/fleet capability | PDP unit tests | A live scoped request succeeds; a mutation receives `403`. | +| Per-target secret isolation | In-process signing-secret test | Target-A token is rejected by target B. | +| Protected inventory/refs | Parser unit tests | Add no-follow symlink regression. | +| Same-host loopback topology | Structural Compose test only | Two containers bind distinct random `127.0.0.1` host ports. | +| TLS CA/name verification | Fake transport tests | Correct CA succeeds; a target's other CA produces `tls`. | +| Manual read-only fan-out | Static UI check/fake client | Explicit `refresh(["a", "b"])` returns fresh results only. | + +## Design + +The test creates all material under `tmp_path`: two minimal configs, two +independent test CAs and server certificates (including `127.0.0.1` SAN), two +distinct signing secrets, two exact-scope 15-minute JWTs, and owner-only +inventory/token/CA files. No model, provider key, real token, prompt, or host +secret is used or logged. + +`make test-docker` is the one authoritative flow: it preflights accessible +Docker and `openssl`, freshly builds the current checkout's Dockerfile once, +records the immutable resulting image ID, and passes it to pytest explicitly. +The image build carries a unique Slice-71 run/source-revision label, which the +test asserts before starting containers. It must not use a reusable tag or an +ambient `AIRLOCK_DOCKER_IMAGE`. The Docker CI job provisions Python, uv, and the +locked test dependencies, then runs only `make test-docker`. + +The test starts two containers via direct `docker run` with deterministic labels +and unique names. It refuses to run as root, runs containers as the invoking +non-root UID/GID, and creates their config/certificate/private-key mounts in an +owner-only temporary directory with mode `0600`. Each is given a generated +owner-only env-file (never secret-bearing command arguments) with its distinct +`AIRLOCK_JWT_SECRET`, and: + +```yaml +admin: + enabled: true + trust_loopback: false + remote_tui: true + fleet_read_tui: true +``` + +Each container uses native TLS, a distinct signing secret, and +Docker-assigned `127.0.0.1::4000` publication. The test extracts the assigned +port from a narrow inspect field and only then creates the inventory. It does +not reserve a random port in Python. The generated config is mounted as +`/app/config.yaml:ro`, certificates/keys use the remote-admin mount paths, +`AIRLOCK_HOST=0.0.0.0`, native TLS environment variables are present, and +`AIRLOCK_JWT_SECRET_PREV`/`AIRLOCK_MASTER_KEY` are absent. The generated config +contains exactly `model_list: []` and the required Admin block—no provider +sections. It explicitly sets `AIRLOCK_MCP_STARTUP_MODE=off`, +`AIRLOCK_STARTUP_MODEL_DISCOVERY=0`, `AIRLOCK_ENABLE_MCP_SERVERS=0`, and +`AIRLOCK_LOG_DIR=/tmp/airlock-slice71`. The generated server key is readable by +the invoking non-root UID/GID used for the container, without relying on the +Dockerfile's fixed `airlock` user or a world-readable key. Direct `docker run` +is intentional: +the existing `docker-compose.remote-admin.yml` is a one-target operator +manifest, and a generated second Compose contract would add no production +coverage. + +The test uses two generated CAs and leaf certificates with `IP:127.0.0.1` SAN, +and inventory origins of `https://127.0.0.1:`. It mints each exact +two-scope, ≤15-minute JWT with that target's signing secret. It waits with a +monotonic bounded deadline only for `https://127.0.0.1:/livez`, using the +generated CA and an environment-free proxy configuration; it never probes +`/health`, a provider, or an inference endpoint. It then uses +`FleetAdminClient` with explicit IDs, and verifies Docker inspect port bindings. +For cross-target replay it retains valid inventory references but replaces only +target B's in-memory token with target A's token, which must be `forbidden`. +It sends a direct CA-verified request to +`POST /airlock/admin/providers/slice71/clear-quarantine` and asserts `403` +before any handler-side mutation. The third unselected target is a bounded +host-loopback listener whose accept event must remain unset, proving that the +client does not contact an unselected inventory target. + +All subprocess error output is bounded and redacted. The fixture records only +the returned container IDs. Before removal in `finally`, it inspects each +recorded ID and verifies the exact UUID run label; cleanup supports partial +starts, preserves the primary failure, and removes no discovered/filter-matched +containers. Narrow inspect assertions prove `HostIp == 127.0.0.1`, distinct +ports, no host network, non-privileged mode, and no Docker-socket mount. +`docker system prune`, broad container deletion, host-network mode, privileged +containers, Docker socket mounts, and all-interface port publication are +prohibited. + +## Verification execution contract + +Register `@pytest.mark.docker` in `pyproject.toml`, add `make test-docker`, and +change ordinary `make test` plus the normal CI test job to select +`not live and not docker`. `make test-docker` builds the labelled local image, +captures its ID, then selects only the Docker marker. The Docker CI job must +first provision Python, uv, and locked test dependencies, then run only that +target. Explicit `make test-docker` fails if Docker is unavailable or +inaccessible—it must not skip and falsely claim topology evidence. CI uses only +the disposable local image and generated material. + +## TDD and acceptance criteria + +1. **RED:** no-follow inventory symlink regression; the marker/Make target; a + three-target inventory (two real containers plus one unselected listener + trap) that cannot pass until real TLS containers and bounded cleanup exist. +2. **GREEN:** correct CA + exact token yields `fresh` for both selected targets; + Docker inspect proves two distinct loopback-only bindings. +3. **Negative proof:** target-A token at target B is `forbidden`; target-B CA + at target A is `tls`; fleet token mutation is `403`; no target is contacted + without being selected. +4. **Safety proof:** generated secret sentinels are absent from assertion + failures/test logs; cleanup removes only known containers; image/container + labels are unique; no provider/inference call is made. +5. **Regression:** Slice 70 focused tests, Admin policy tests, Docker marker, + Ruff/format, strict docs, `make sync && make verify`, and ordinary + `make test` all pass. Record Docker engine/image versions and result in a + Slice 71 status record. + +## Blast radius and rollback + +The product blast radius is nil: changes are test harness, Make/CI selection, +and verification documentation only. Docker resource use and accidental broad +cleanup are the material risks; named labels, generated temporary resources, +loopback publication, bounded output, and explicit cleanup mitigate them. +Rollback is removal of the test target and test-only resources; no deployed +operator configuration is changed. + +## Documentation and durable evidence + +Update the [README Developer setup](../../../README.md#developer-setup) section +with `make test-docker`, Docker and `openssl` prerequisites, its opt-in +behavior, generated-material and no-provider-network guarantees, and exact +cleanup boundary. The final Slice 71 status record SHALL name the image ID, +Docker engine version, command and result, plus any skipped/not-applicable +condition—never secret material. + +## Admission gates + +Before implementation, an independent design review must approve the generated +certificate/token approach, image provenance, Docker cleanup selectors, +marker/CI separation, liveness behavior, and secret-safe diagnostics. Docker +daemon access is now owner-approved only for Airlock development verification; +it does not authorize product code to use Docker as identity or control-plane +authority. diff --git a/dev/plans/0.5.15/README.md b/dev/plans/0.5.15/README.md index fb6f6d2..8f62722 100644 --- a/dev/plans/0.5.15/README.md +++ b/dev/plans/0.5.15/README.md @@ -31,7 +31,8 @@ that is accepted for investigation from work that is approved for delivery. | Read-only provider/configuration visibility in Admin/TUI | Design exists; CRUD is not approved | Slice 40 | | Secure host-console TUI administration for containers | Design exists; topology/identity decision is open | Slice 50 | | Airlock-native virtual-key management | **Deferred to 0.6.0**; depends on the native identity/keystore foundation | Slice 60 (0.6.0) | -| Multi-instance TUI administration | Research/design exists; remote operation/configuration authority is open | Slice 70, conditional | +| Multi-instance TUI administration | Implemented as a bounded same-host read-only view | Slice 70 | +| Slice 70 live container topology verification | Admitted verification-only work; requires two disposable TLS container targets | Slice 71 | | CI review and improvement | Review workflow reliability, signal, coverage, cost, and maintainability without weakening delivery controls | Slice 80 | | Release closeout | Required only for work actually included and delivered | Slice 90 | @@ -218,6 +219,7 @@ authorize work before Slice 6. | 50 | Secure host-console TUI administration for containerized Airlock | Topology, operator identity, TLS/token lifecycle, and audit decisions pass. | | 60 | **Deferred to 0.6.0** virtual-key management | The 0.6.0 B5 identity/keystore foundation must close first; not a 0.5.15 delivery slice. | | 70 | Multi-instance TUI named inventory and read-only Admin-API fan-out | Fleet identity/TLS/token/inventory/operation limits are approved; otherwise postpone. | +| 71 | Slice 70 Docker topology verification | Slice 70 is closed; Docker daemon access and a disposable local image are available. Verification-only: no product authority expansion. | | 80 | CI review and improvement | Review current workflows, required checks, runtime/cost, cache and matrix design, test-signal quality, and failure triage; preserve least privilege, pinned actions, Gitleaks enforcement, and no-secret CI boundaries. Document and independently review proposed changes before implementation. | | 90 | Release closeout | Only delivered, verified slices are included; deferred work is named explicitly. | From 8241eaff06d5034d1d883af3321e8d2c82257c9c Mon Sep 17 00:00:00 2001 From: Corey Thompson Date: Sun, 16 Aug 2026 22:13:54 -0500 Subject: [PATCH 17/24] test(fleet): verify two-container topology --- .github/workflows/ci.yml | 17 +- Makefile | 14 +- README.md | 17 + .../71-slice70-docker-topology-status.md | 33 ++ ...71-slice70-docker-topology-verification.md | 4 +- pyproject.toml | 1 + tests/test_fleet_profile.py | 9 + tests/test_slice70_docker_topology.py | 453 ++++++++++++++++++ 8 files changed, 541 insertions(+), 7 deletions(-) create mode 100644 dev/plans/0.5.15/71-slice70-docker-topology-status.md create mode 100644 tests/test_slice70_docker_topology.py diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 895d3ed..0709f47 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -61,7 +61,7 @@ jobs: - name: Run tests with coverage run: | - uv run pytest -m "not live" --cov=airlock --cov-report=xml --cov-report=term-missing + uv run pytest -m "not live and not docker" --cov=airlock --cov-report=xml --cov-report=term-missing - name: Upload coverage report if: always() @@ -107,8 +107,19 @@ jobs: steps: - uses: actions/checkout@v6 - - name: Build Docker image - run: docker build -t airlock:ci . + - name: Set up Python + uses: actions/setup-python@v6 + with: + python-version: "3.12" + + - name: Install uv + uses: astral-sh/setup-uv@v8.0.0 + + - name: Install locked test dependencies + run: uv sync --locked --extra test + + - name: Verify disposable Docker topology + run: make test-docker security: runs-on: ubuntu-latest diff --git a/Makefile b/Makefile index 14c74be..673a78c 100644 --- a/Makefile +++ b/Makefile @@ -8,7 +8,7 @@ # # Always sync via `make sync` so the model is restored automatically. -.PHONY: sync ensure-spacy verify test +.PHONY: sync ensure-spacy verify test test-docker sync: ## Sync all deps and restore the spaCy model uv prunes uv sync --locked --all-extras @@ -26,4 +26,14 @@ verify: ## Fail fast if the spaCy PII model is missing (CI / preflight gate) || { echo "ERROR: $$AIRLOCK_SPACY_MODEL missing — Presidio PII guard will fail. Run 'make ensure-spacy'."; exit 1; } test: ## Run the test suite - uv run python -m pytest -q -m "not live" + uv run python -m pytest -q -m "not live and not docker" + +test-docker: ## Run opt-in Slice 71 disposable Docker topology verification + @command -v docker >/dev/null || { echo "ERROR: docker is required for make test-docker"; exit 1; } + @docker info >/dev/null 2>&1 || { echo "ERROR: Docker daemon is unavailable or inaccessible"; exit 1; } + @command -v openssl >/dev/null || { echo "ERROR: openssl is required for make test-docker"; exit 1; } + @run_id="$$(date +%s)-$$$$"; revision="$$(git rev-parse HEAD)"; \ + image="$$(docker build --quiet --label org.airlock.slice71.run="$$run_id" --label org.opencontainers.image.revision="$$revision" .)"; \ + printf 'Slice 71 Docker evidence: image=%s run=%s revision=%s\\n' "$$image" "$$run_id" "$$revision"; \ + export SLICE71_IMAGE="$$image" SLICE71_RUN_ID="$$run_id" SLICE71_REVISION="$$revision"; \ + uv run python -m pytest -q -m docker diff --git a/README.md b/README.md index b57f7d2..f217eac 100644 --- a/README.md +++ b/README.md @@ -149,6 +149,23 @@ Or press `6` in the TUI for the Advisor screen. The advisor prefers local models docker compose up --build ``` +### Developer Docker topology verification + +`make test` is offline and excludes the opt-in Docker topology test. To verify +the Slice 70 same-host fleet connection against two disposable local Airlock +containers, run: + +```bash +make test-docker +``` + +This requires an accessible Docker daemon and `openssl`. It builds a uniquely +labelled image from the current checkout, generates short-lived test CAs, +certificates, capability tokens, configuration, and two loopback-only container +ports in a temporary directory, and removes only the exact labelled containers +it created. It does not use provider credentials, contact provider or inference +endpoints, reuse an ambient image, or publish an image or registry artifact. + ## Documentation and repository map The [documentation site](docs/index.md) is the canonical guide for users and diff --git a/dev/plans/0.5.15/71-slice70-docker-topology-status.md b/dev/plans/0.5.15/71-slice70-docker-topology-status.md new file mode 100644 index 0000000..3ca89e2 --- /dev/null +++ b/dev/plans/0.5.15/71-slice70-docker-topology-status.md @@ -0,0 +1,33 @@ +# Slice 71 — Slice 70 Docker topology verification status + +**Status:** complete — verified 2026-08-16. + +## Scope + +Slice 71 verifies Slice 70's two-container, same-host read-only topology. It +adds no production Docker authority, manifests, routes, inventory fields, or +registry publication. Its only runtime resources are uniquely labelled local +test containers and generated temporary test material. + +## Verification evidence + +- Docker Engine `29.7.1`; `make test-docker` passed with local image + `sha256:421b1017e786ca64294338011b86894da86d81e9c95226af63e25dd95327d8d6`, + run label `1786935832-723127`, and source revision + `d062f5e81a2e3aafb1abccb78df7f2813cfac2a5`. The test passed in 21.10s; + its label-scoped cleanup left no Slice 71 containers. +- The live test proved two distinct loopback-only published ports, selected + exact-scope read success, cross-target token rejection, CA rejection, + mutation denial, and no connection to the unselected listener. +- Focused fleet/Admin/TUI regression, Ruff check/format, and `git diff --check` + passed. `make sync && make verify`, strict MkDocs, and documentation-contract + tests passed independently. The ordinary `make test` run passed: 3421 passed, + 112 deselected, 1 expected XPASS in 341.69s. Normal test selection excludes + the Docker marker; CI's Docker job invokes only the opt-in target after + locked test dependency provisioning. + +## Boundary and follow-up + +This test builds a local image only; it creates no registry artifact, tag, or +release. It creates no auxiliary worktree. Slice 80 owns the follow-on CI +hardening and exact-head GitHub verification. diff --git a/dev/plans/0.5.15/71-slice70-docker-topology-verification.md b/dev/plans/0.5.15/71-slice70-docker-topology-verification.md index c19cd1b..84b3cb3 100644 --- a/dev/plans/0.5.15/71-slice70-docker-topology-verification.md +++ b/dev/plans/0.5.15/71-slice70-docker-topology-verification.md @@ -67,8 +67,8 @@ not reserve a random port in Python. The generated config is mounted as `/app/config.yaml:ro`, certificates/keys use the remote-admin mount paths, `AIRLOCK_HOST=0.0.0.0`, native TLS environment variables are present, and `AIRLOCK_JWT_SECRET_PREV`/`AIRLOCK_MASTER_KEY` are absent. The generated config -contains exactly `model_list: []` and the required Admin block—no provider -sections. It explicitly sets `AIRLOCK_MCP_STARTUP_MODE=off`, +contains `model_list: []`, the required Admin block, and Airlock's required +`litellm_settings.callbacks` bootstrap callback—no provider sections. It explicitly sets `AIRLOCK_MCP_STARTUP_MODE=off`, `AIRLOCK_STARTUP_MODEL_DISCOVERY=0`, `AIRLOCK_ENABLE_MCP_SERVERS=0`, and `AIRLOCK_LOG_DIR=/tmp/airlock-slice71`. The generated server key is readable by the invoking non-root UID/GID used for the container, without relying on the diff --git a/pyproject.toml b/pyproject.toml index dd66d48..92acd4a 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -95,6 +95,7 @@ addopts = "-p no:xdist -p no:hypothesis -p no:Faker -p no:dash -m 'not stress'" markers = [ "harness: Feature test harness (integration)", "live: Requires running proxy with real API keys", + "docker: Requires an accessible Docker daemon; opt-in disposable topology test", "stress: Concurrency/stress tests excluded from default run", ] # The deprecation budget is NOT enforced here. pytest's `filterwarnings` matches diff --git a/tests/test_fleet_profile.py b/tests/test_fleet_profile.py index 1bca8db..4c276c1 100644 --- a/tests/test_fleet_profile.py +++ b/tests/test_fleet_profile.py @@ -83,6 +83,15 @@ def test_profile_rejects_duplicate_secret_references_and_insecure_inventory(tmp_ load_fleet_profile(profile) +def test_profile_rejects_inventory_symlink_before_reading_target_references(tmp_path): + profile = _secure(tmp_path / "fleet.yaml", "targets: []\n") + linked = tmp_path / "fleet-link.yaml" + linked.symlink_to(profile) + + with pytest.raises(FleetProfileError, match="regular file"): + load_fleet_profile(linked) + + def test_profile_rejects_embedded_or_unsupported_values(tmp_path): ca = _secure(tmp_path / "ca.pem", "CA") token = _secure(tmp_path / "token.jwt", "abc.def.ghi") diff --git a/tests/test_slice70_docker_topology.py b/tests/test_slice70_docker_topology.py new file mode 100644 index 0000000..712605f --- /dev/null +++ b/tests/test_slice70_docker_topology.py @@ -0,0 +1,453 @@ +"""Opt-in Docker topology proof for Slice 70's read-only fleet client. + +This module is selected solely by ``-m docker``. It is intentionally a direct +Docker API consumer: it creates only UUID-labelled containers and removes only +the IDs it created after confirming their exact run label. +""" + +from __future__ import annotations + +import json +import os +import socket +import ssl +import subprocess +import threading +import time +from dataclasses import replace +from pathlib import Path +from uuid import uuid4 + +import jwt +import pytest + +from airlock.tui.fleet_client import FleetAdminClient +from airlock.tui.fleet_profile import FleetTarget, load_fleet_profile + +pytestmark = pytest.mark.docker + +_RUN_LABEL = "org.airlock.slice71.run" +_REVISION_LABEL = "org.opencontainers.image.revision" + + +def _label_format(label: str) -> str: + """Return a Docker Go-template using a correctly quoted label key.""" + return f"{{{{index .Config.Labels {json.dumps(label)}}}}}" + + +def _run(*args: str, timeout: float = 45) -> str: + """Run Docker/OpenSSL without rendering potentially sensitive output.""" + completed = subprocess.run( + args, + check=False, + stdin=subprocess.DEVNULL, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + text=True, + timeout=timeout, + ) + if completed.returncode: + raise AssertionError( + f"{args[0]} command failed with exit {completed.returncode}" + ) + return completed.stdout.strip() + + +def _secure(path: Path, content: str) -> Path: + path.write_text(content) + path.chmod(0o600) + return path + + +def _certificate_pair(directory: Path, name: str) -> tuple[Path, Path, Path]: + """Generate one ephemeral CA and 127.0.0.1 leaf, all owner-only.""" + ca_key = directory / f"{name}-ca.key" + ca_pem = directory / f"{name}-ca.pem" + leaf_key = directory / f"{name}.key" + csr = directory / f"{name}.csr" + leaf_pem = directory / f"{name}.crt" + ext = directory / f"{name}.ext" + _run( + "openssl", + "req", + "-x509", + "-newkey", + "rsa:2048", + "-nodes", + "-keyout", + str(ca_key), + "-out", + str(ca_pem), + "-subj", + f"/CN=slice71-{name}-ca", + "-days", + "1", + ) + _run( + "openssl", + "req", + "-newkey", + "rsa:2048", + "-nodes", + "-keyout", + str(leaf_key), + "-out", + str(csr), + "-subj", + "/CN=127.0.0.1", + ) + _secure(ext, "subjectAltName = IP:127.0.0.1\n") + _run( + "openssl", + "x509", + "-req", + "-in", + str(csr), + "-CA", + str(ca_pem), + "-CAkey", + str(ca_key), + "-CAcreateserial", + "-out", + str(leaf_pem), + "-days", + "1", + "-extfile", + str(ext), + ) + for path in (ca_key, ca_pem, leaf_key, csr, leaf_pem): + path.chmod(0o600) + return ca_pem, leaf_pem, leaf_key + + +def _token(secret: str, subject: str) -> str: + now = int(time.time()) + return jwt.encode( + { + "iss": "airlock", + "sub": subject, + "scope": ["admin:remote_tui", "admin:read"], + "iat": now, + "exp": now + 15 * 60, + "jti": uuid4().hex, + }, + secret, + algorithm="HS256", + ) + + +def _docker_port(container_id: str) -> tuple[str, int]: + ports = json.loads( + _run( + "docker", + "inspect", + container_id, + "--format", + "{{json .NetworkSettings.Ports}}", + ) + ) + binding = ports["4000/tcp"] + assert isinstance(binding, list) and len(binding) == 1 + assert binding[0]["HostIp"] == "127.0.0.1" + return binding[0]["HostIp"], int(binding[0]["HostPort"]) + + +def _assert_safe_container(container_id: str, run_id: str, revision: str) -> None: + assert ( + _run( + "docker", + "inspect", + container_id, + "--format", + _label_format(_RUN_LABEL), + ) + == run_id + ) + assert ( + _run( + "docker", + "inspect", + container_id, + "--format", + _label_format(_REVISION_LABEL), + ) + == revision + ) + host_config = json.loads( + _run("docker", "inspect", container_id, "--format", "{{json .HostConfig}}") + ) + mounts = json.loads( + _run("docker", "inspect", container_id, "--format", "{{json .Mounts}}") + ) + assert host_config["NetworkMode"] != "host" + assert host_config["Privileged"] is False + assert not any(mount.get("Source") == "/var/run/docker.sock" for mount in mounts) + + +def _https_request(port: int, ca_pem: str, token: str, method: str, path: str) -> int: + """Use a direct HTTPS connection; urllib proxy settings cannot influence it.""" + context = ssl.create_default_context(cadata=ca_pem) + with socket.create_connection(("127.0.0.1", port), timeout=2) as raw: + with context.wrap_socket(raw, server_hostname="127.0.0.1") as tls: + tls.settimeout(3) + request = ( + f"{method} {path} HTTP/1.1\r\nHost: 127.0.0.1:{port}\r\n" + f"Authorization: Bearer {token}\r\nConnection: close\r\n\r\n" + ).encode() + tls.sendall(request) + response = bytearray() + while b"\r\n" not in response and len(response) < 8192: + chunk = tls.recv(min(1024, 8192 - len(response))) + if not chunk: + break + response.extend(chunk) + head = bytes(response).split(b"\r\n", 1)[0] + return int(head.split()[1]) + + +def _wait_livez(port: int, ca_pem: str) -> None: + deadline = time.monotonic() + 40 + while time.monotonic() < deadline: + try: + if _https_request(port, ca_pem, "", "GET", "/livez") == 200: + return + except (OSError, ssl.SSLError, ValueError, IndexError): + pass + time.sleep(0.25) + raise AssertionError( + "Slice 71 container did not become live before bounded deadline" + ) + + +def _run_container( + image: str, + run_id: str, + revision: str, + name: str, + directory: Path, + cert: Path, + key: Path, + secret: str, +) -> str: + config = _secure( + directory / f"{name}.yaml", + "model_list: []\n" + "admin:\n" + " enabled: true\n" + " trust_loopback: false\n" + " remote_tui: true\n" + " fleet_read_tui: true\n" + "litellm_settings:\n" + ' callbacks: ["airlock.callbacks.model_override_headers.proxy_model_override_headers"]\n', + ) + env_file = _secure( + directory / f"{name}.env", + "AIRLOCK_CONFIG=/app/config.yaml\n" + "AIRLOCK_HOST=0.0.0.0\n" + "AIRLOCK_PORT=4000\n" + "AIRLOCK_SSL_CERTFILE=/run/airlock-tls/server.crt\n" + "AIRLOCK_SSL_KEYFILE=/run/airlock-tls/server.key\n" + f"AIRLOCK_JWT_SECRET={secret}\n" + "AIRLOCK_MCP_STARTUP_MODE=off\n" + "AIRLOCK_STARTUP_MODEL_DISCOVERY=0\n" + "AIRLOCK_ENABLE_MCP_SERVERS=0\n" + "AIRLOCK_LOG_DIR=/tmp/airlock-slice71\n", + ) + return _run( + "docker", + "run", + "--detach", + "--name", + name, + "--label", + f"{_RUN_LABEL}={run_id}", + "--label", + f"{_REVISION_LABEL}={revision}", + "--user", + f"{os.getuid()}:{os.getgid()}", + "--env-file", + str(env_file), + "--publish", + "127.0.0.1::4000", + "--mount", + f"type=bind,src={config},dst=/app/config.yaml,readonly", + "--mount", + f"type=bind,src={cert},dst=/run/airlock-tls/server.crt,readonly", + "--mount", + f"type=bind,src={key},dst=/run/airlock-tls/server.key,readonly", + image, + ) + + +def _write_inventory(directory: Path, targets: list[dict[str, str]]) -> Path: + lines = ["targets:"] + for target in targets: + lines.extend( + [ + f" - id: {target['id']}", + f" name: {target['name']}", + f" origin: {target['origin']}", + f" token_file: {target['token']}", + f" ca_file: {target['ca']}", + ] + ) + return _secure(directory / "fleet.yaml", "\n".join(lines) + "\n") + + +def _fetch_state(target: FleetTarget) -> str: + """Fetch one deliberately substituted target without retaining worker state.""" + client = FleetAdminClient([target]) + try: + return client.fetch(target).state + finally: + client.close() + + +def test_slice70_two_container_tls_fleet_topology( + tmp_path: Path, +): + """Prove the Slice 70 client against two isolated real TLS containers.""" + if os.geteuid() == 0: + pytest.fail("Slice 71 must run as a non-root Docker client") + image = os.environ["SLICE71_IMAGE"] + run_id = os.environ["SLICE71_RUN_ID"] + revision = os.environ["SLICE71_REVISION"] + assert ( + _run( + "docker", + "image", + "inspect", + image, + "--format", + _label_format(_RUN_LABEL), + ) + == run_id + ) + assert ( + _run( + "docker", + "image", + "inspect", + image, + "--format", + _label_format(_REVISION_LABEL), + ) + == revision + ) + + containers: list[str] = [] + listener = socket.socket(socket.AF_INET, socket.SOCK_STREAM) + listener.bind(("127.0.0.1", 0)) + listener.listen(1) + trap_port = listener.getsockname()[1] + trap_hit = threading.Event() + + def trap() -> None: + try: + listener.settimeout(6) + listener.accept() + trap_hit.set() + except OSError: + pass + + thread = threading.Thread(target=trap, daemon=True) + thread.start() + try: + ca_a, cert_a, key_a = _certificate_pair(tmp_path, "a") + ca_b, cert_b, key_b = _certificate_pair(tmp_path, "b") + secret_a, secret_b = uuid4().hex, uuid4().hex + token_a = _secure(tmp_path / "a.jwt", _token(secret_a, "slice71-a")) + token_b = _secure(tmp_path / "b.jwt", _token(secret_b, "slice71-b")) + token_c = _secure(tmp_path / "c.jwt", "abc.def.ghi") + ca_c = _secure(tmp_path / "c-ca.pem", "unused CA\n") + name_a, name_b = (f"airlock-s71-{run_id}-a", f"airlock-s71-{run_id}-b") + containers.append( + _run_container( + image, run_id, revision, name_a, tmp_path, cert_a, key_a, secret_a + ) + ) + containers.append( + _run_container( + image, run_id, revision, name_b, tmp_path, cert_b, key_b, secret_b + ) + ) + for container in containers: + _assert_safe_container(container, run_id, revision) + _host_a, port_a = _docker_port(containers[0]) + _host_b, port_b = _docker_port(containers[1]) + assert port_a != port_b + _wait_livez(port_a, ca_a.read_text()) + _wait_livez(port_b, ca_b.read_text()) + + inventory = _write_inventory( + tmp_path, + [ + { + "id": "a", + "name": "A", + "origin": f"https://127.0.0.1:{port_a}", + "token": str(token_a), + "ca": str(ca_a), + }, + { + "id": "b", + "name": "B", + "origin": f"https://127.0.0.1:{port_b}", + "token": str(token_b), + "ca": str(ca_b), + }, + { + "id": "trap", + "name": "Trap", + "origin": f"https://127.0.0.1:{trap_port}", + "token": str(token_c), + "ca": str(ca_c), + }, + ], + ) + # No-follow applies to the inventory itself, not merely its references. + inventory_link = tmp_path / "fleet-link.yaml" + inventory_link.symlink_to(inventory) + with pytest.raises(ValueError): + load_fleet_profile(inventory_link) + + targets = load_fleet_profile(inventory) + client = FleetAdminClient(targets) + try: + assert [result.state for result in client.refresh(["a", "b"])] == [ + "fresh", + "fresh", + ] + target_a, target_b = targets[:2] + assert _fetch_state(replace(target_b, token=target_a.token)) == "forbidden" + assert _fetch_state(replace(target_a, ca_pem=target_b.ca_pem)) == "tls" + assert ( + _https_request( + port_a, + target_a.ca_pem, + target_a.token, + "POST", + "/airlock/admin/providers/slice71/clear-quarantine", + ) + == 403 + ) + assert not trap_hit.wait(0.5) + finally: + client.close() + finally: + listener.close() + for container in containers: + try: + if ( + _run( + "docker", + "inspect", + container, + "--format", + _label_format(_RUN_LABEL), + ) + == run_id + ): + _run("docker", "rm", "--force", container) + except (AssertionError, subprocess.TimeoutExpired): + pass From 8a06371177ca9bee93bc428a38de9e65240d8017 Mon Sep 17 00:00:00 2001 From: Corey Thompson Date: Sun, 16 Aug 2026 22:18:57 -0500 Subject: [PATCH 18/24] ci: harden workflow reliability and permissions --- .github/workflows/ci.yml | 65 +++++-- .github/workflows/docs.yml | 21 +- .github/workflows/release.yml | 45 +++-- .../80-ci-reliability-security-design.md | 117 ++++++++++++ .../80-ci-reliability-security-status.md | 60 ++++++ tests/test_ci_workflows.py | 179 ++++++++++++++++++ 6 files changed, 442 insertions(+), 45 deletions(-) create mode 100644 dev/plans/0.5.15/80-ci-reliability-security-design.md create mode 100644 dev/plans/0.5.15/80-ci-reliability-security-status.md create mode 100644 tests/test_ci_workflows.py diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 0709f47..1009629 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -9,20 +9,25 @@ on: permissions: contents: read +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + jobs: docs: runs-on: ubuntu-latest + timeout-minutes: 15 steps: - - uses: actions/checkout@v6 + - uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6 - name: Set up Python - uses: actions/setup-python@v6 + uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6 with: python-version: "3.12" - name: Install uv - uses: astral-sh/setup-uv@v8.0.0 + uses: astral-sh/setup-uv@cec208311dfd045dd5311c1add060b2062131d57 # v8.0.0 - name: Build strict documentation site and check documentation contracts run: | @@ -32,20 +37,21 @@ jobs: test: runs-on: ubuntu-latest + timeout-minutes: 35 strategy: matrix: python-version: ["3.12"] steps: - - uses: actions/checkout@v6 + - uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6 - name: Set up Python ${{ matrix.python-version }} - uses: actions/setup-python@v6 + uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6 with: python-version: ${{ matrix.python-version }} - name: Install uv - uses: astral-sh/setup-uv@v8.0.0 + uses: astral-sh/setup-uv@cec208311dfd045dd5311c1add060b2062131d57 # v8.0.0 - name: Install dependencies run: | @@ -61,28 +67,47 @@ jobs: - name: Run tests with coverage run: | - uv run pytest -m "not live and not docker" --cov=airlock --cov-report=xml --cov-report=term-missing + mkdir -p test-results + uv run pytest -m "not live and not docker" --cov=airlock --cov-report=xml --cov-report=term-missing --junitxml=test-results/junit.xml -o junit_logging=no + + - name: Check failure JUnit report is safe to retain + id: junit + if: failure() + run: | + if test -f test-results/junit.xml && test "$(wc -c < test-results/junit.xml)" -le 5242880; then + echo "upload=true" >> "$GITHUB_OUTPUT" + else + echo "upload=false" >> "$GITHUB_OUTPUT" + fi + + - name: Upload bounded failure JUnit report + if: failure() && steps.junit.outputs.upload == 'true' + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 + with: + name: test-failure-junit + path: test-results/junit.xml + retention-days: 1 - name: Upload coverage report if: always() - uses: actions/upload-artifact@v7 + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 with: name: coverage-report path: coverage.xml lint: runs-on: ubuntu-latest - needs: [test] + timeout-minutes: 15 steps: - - uses: actions/checkout@v6 + - uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6 - name: Set up Python - uses: actions/setup-python@v6 + uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6 with: python-version: "3.12" - name: Install uv - uses: astral-sh/setup-uv@v8.0.0 + uses: astral-sh/setup-uv@cec208311dfd045dd5311c1add060b2062131d57 # v8.0.0 - name: Install dependencies and linters run: | @@ -103,17 +128,17 @@ jobs: docker: runs-on: ubuntu-latest - needs: [test] + timeout-minutes: 30 steps: - - uses: actions/checkout@v6 + - uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6 - name: Set up Python - uses: actions/setup-python@v6 + uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6 with: python-version: "3.12" - name: Install uv - uses: astral-sh/setup-uv@v8.0.0 + uses: astral-sh/setup-uv@cec208311dfd045dd5311c1add060b2062131d57 # v8.0.0 - name: Install locked test dependencies run: uv sync --locked --extra test @@ -123,17 +148,17 @@ jobs: security: runs-on: ubuntu-latest - needs: [test] + timeout-minutes: 20 steps: - - uses: actions/checkout@v6 + - uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6 - name: Set up Python - uses: actions/setup-python@v6 + uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6 with: python-version: "3.12" - name: Install uv - uses: astral-sh/setup-uv@v8.0.0 + uses: astral-sh/setup-uv@cec208311dfd045dd5311c1add060b2062131d57 # v8.0.0 - name: Install and audit # Suppressions are documented, scoped, and reviewable in diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml index d7d65a5..77f3863 100644 --- a/.github/workflows/docs.yml +++ b/.github/workflows/docs.yml @@ -13,32 +13,35 @@ on: permissions: contents: read - pages: write - id-token: write concurrency: - group: "pages" - cancel-in-progress: false + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true jobs: build: runs-on: ubuntu-latest + timeout-minutes: 20 steps: - - uses: actions/checkout@v6 - - uses: astral-sh/setup-uv@v8.0.0 - - uses: actions/configure-pages@v5 + - uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6 + - uses: astral-sh/setup-uv@cec208311dfd045dd5311c1add060b2062131d57 # v8.0.0 + - uses: actions/configure-pages@983d7736d9b0ae728b81ab479565c72886d7745b # v5 - name: Build strict documentation site run: uv run --extra docs mkdocs build --strict --site-dir _site - - uses: actions/upload-pages-artifact@v3 + - uses: actions/upload-pages-artifact@56afc609e74202658d3ffba0e8f6dda462b719fa # v3 with: path: _site deploy: needs: build runs-on: ubuntu-latest + timeout-minutes: 10 + permissions: + pages: write + id-token: write environment: name: github-pages url: ${{ steps.deployment.outputs.page_url }} steps: - id: deployment - uses: actions/deploy-pages@v4 + uses: actions/deploy-pages@d6db90164ac5ed86f2b6aed7e0febac5b3c0c03e # v4 diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index afce19f..5fbcce9 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -26,27 +26,32 @@ on: - "v*" permissions: - contents: write # create GitHub release - id-token: write # PyPI trusted publishing (OIDC) + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + # Never interrupt a release after a tag has started publishing. + cancel-in-progress: false jobs: # Gate: run the full test suite before publishing test: runs-on: ubuntu-latest + timeout-minutes: 40 strategy: matrix: python-version: ["3.12"] steps: - - uses: actions/checkout@v6 + - uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6 - name: Set up Python ${{ matrix.python-version }} - uses: actions/setup-python@v6 + uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6 with: python-version: ${{ matrix.python-version }} - name: Install uv - uses: astral-sh/setup-uv@v8.0.0 + uses: astral-sh/setup-uv@cec208311dfd045dd5311c1add060b2062131d57 # v8.0.0 - name: Install dependencies run: uv sync --locked --all-extras @@ -55,7 +60,7 @@ jobs: run: make ensure-spacy - name: Run tests - run: uv run pytest --tb=short -q -m "not live" + run: uv run pytest --tb=short -q -m "not live and not docker" - name: Build strict documentation site run: uv run mkdocs build --strict --site-dir /tmp/airlock-docs @@ -63,11 +68,12 @@ jobs: # Gate: verify the tag version matches pyproject.toml check-version: runs-on: ubuntu-latest + timeout-minutes: 10 steps: - - uses: actions/checkout@v6 + - uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6 - name: Set up Python - uses: actions/setup-python@v6 + uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6 with: python-version: "3.12" @@ -90,11 +96,12 @@ jobs: build: needs: [test, check-version] runs-on: ubuntu-latest + timeout-minutes: 15 steps: - - uses: actions/checkout@v6 + - uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6 - name: Install uv - uses: astral-sh/setup-uv@v8.0.0 + uses: astral-sh/setup-uv@cec208311dfd045dd5311c1add060b2062131d57 # v8.0.0 - name: Build package run: | @@ -102,7 +109,7 @@ jobs: uv build - name: Upload dist artifacts - uses: actions/upload-artifact@v7 + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 with: name: dist path: dist/ @@ -111,26 +118,32 @@ jobs: publish: needs: [build] runs-on: ubuntu-latest + timeout-minutes: 15 + permissions: + id-token: write environment: pypi steps: - name: Download dist artifacts - uses: actions/download-artifact@v8 + uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8 with: name: dist path: dist/ - name: Publish to PyPI - uses: pypa/gh-action-pypi-publish@release/v1 + uses: pypa/gh-action-pypi-publish@dc37677b2e1c63e2034f94d8a5b11f265b73ba33 # v1.14.2 # Create a GitHub release with the built artifacts github-release: needs: [publish] runs-on: ubuntu-latest + timeout-minutes: 15 + permissions: + contents: write steps: - - uses: actions/checkout@v6 + - uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6 - name: Download dist artifacts - uses: actions/download-artifact@v8 + uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8 with: name: dist path: dist/ @@ -140,7 +153,7 @@ jobs: run: echo "version=${GITHUB_REF#refs/tags/}" >> "$GITHUB_OUTPUT" - name: Create GitHub Release - uses: softprops/action-gh-release@v3 + uses: softprops/action-gh-release@3d0d9888cb7fd7b750713d6e236d1fcb99157228 # v3.0.2 with: tag_name: ${{ steps.version.outputs.version }} name: ${{ steps.version.outputs.version }} diff --git a/dev/plans/0.5.15/80-ci-reliability-security-design.md b/dev/plans/0.5.15/80-ci-reliability-security-design.md new file mode 100644 index 0000000..cf6a5bd --- /dev/null +++ b/dev/plans/0.5.15/80-ci-reliability-security-design.md @@ -0,0 +1,117 @@ +# Slice 80 — CI reliability, diagnostic signal, and least privilege + +**Status:** proposed — pending independent design review. Created 2026-08-16 +after the Slice 80 audit and after Slice 71's Docker-test contract was approved. + +## Re-evaluation evidence + +The existing CI has a 3.12 test job, but `lint`, `security`, and `docker` all +depend on it. A failed test therefore suppresses independent diagnostics. PR +#49 is stale at `8b75365`; its 3.12 test failed with a generic exit annotation +and expired logs, while docs/Gitleaks passed and the dependent jobs were +skipped. Current release work is ahead locally and requires an exact-head GitHub +run before closeout. + +Slice 71 owns the Docker topology test and its Make/marker contract. Slice 80 +will make CI invoke that target; it must not redefine the topology fixture. + +## Scope and requirements + +1. Pin every `uses:` reference in `ci.yml`, `docs.yml`, and `release.yml` to a + reviewed full commit SHA with an adjacent human version comment. Retain the + existing Gitleaks pin and its scanner policy unchanged. +2. Add workflow/ref concurrency and bounded per-job timeouts. CI uses + `cancel-in-progress: true`; documentation deploys use the same + workflow/ref-scoped grouping and may cancel stale runs because the newest + successful deployment is authoritative; release uses the group with + `cancel-in-progress: false` so no tag publication is interrupted. + `lint`, `security`, and `docker` must run independently of a test failure; + their checks do not weaken or replace the test gate. +3. Ordinary CI tests select `not live and not docker`; Docker CI provisions + locked test dependencies and runs only Slice 71's `make test-docker` flow. + It builds its disposable local image only; it does not push an image. +4. Preserve `contents: read` CI permissions. Make docs deployment permissions + job-local (`pages: write`, `id-token: write` only on deploy); make release + default `contents: read`, PyPI OIDC job-local, and GitHub-release + `contents: write` job-local. This hardens future manual/tag release runs but + does not dispatch or publish one in this slice. +5. On test failure only, upload a bounded JUnit artifact from the ordinary test + job to make future + failures diagnosable without retaining test stdout, environment, generated + credentials, or topology material. Normal test output/coverage behavior + remains intact. +6. Keep the 3.12 release gate. Record—not silently alter—the discrepancy with + metadata still declaring Python >=3.10 and 3.10–3.12 classifiers. A later + owner choice must add lower-version CI or change public support metadata. +7. After implementation, push current release commits to the existing PR + branch and obtain exact-head green GitHub CI plus Gitleaks. Query that SHA's + checks, record the returned exact context names and app IDs, apply precisely + those contexts through the branch-protection API, then re-read protection to + prove CODEOWNERS and no-admin-bypass protections remain enabled. + +## Reviewed immutable actions + +| Action | Pinned commit | Human version | +| --- | --- | --- | +| `actions/checkout` | `d23441a48e516b6c34aea4fa41551a30e30af803` | v6 | +| `actions/setup-python` | `ece7cb06caefa5fff74198d8649806c4678c61a1` | v6 | +| `astral-sh/setup-uv` | `cec208311dfd045dd5311c1add060b2062131d57` | v8.0.0 | +| `actions/upload-artifact` | `043fb46d1a93c77aae656e7c1c64a875d1fc6a0a` | v7 | +| `actions/configure-pages` | `983d7736d9b0ae728b81ab479565c72886d7745b` | v5 | +| `actions/upload-pages-artifact` | `56afc609e74202658d3ffba0e8f6dda462b719fa` | v3 | +| `actions/deploy-pages` | `d6db90164ac5ed86f2b6aed7e0febac5b3c0c03e` | v4 | +| `actions/download-artifact` | `3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c` | v8 | +| `pypa/gh-action-pypi-publish` | `dc37677b2e1c63e2034f94d8a5b11f265b73ba33` | v1.14.2 (currently `release/v1`) | +| `softprops/action-gh-release` | `3d0d9888cb7fd7b750713d6e236d1fcb99157228` | v3.0.2 | + +## Design and safety boundary + +Workflow changes are test/release-control infrastructure, not product behavior. +No PR workflow receives a write token, OIDC, registry credential, or deployment +secret. The only Docker action is a locally built, labelled Slice 71 image with +generated material. No tag, release dispatch, GitHub release, PyPI publish, or +container registry push is authorized by this slice. + +Use stable group names derived from workflow/ref. CI cancels superseded runs; +documentation does likewise because a later successful pages deployment is the +only desired state; releases never cancel an in-progress publication. Apply the +following explicit timeouts: CI docs 15m, test 35m, lint 15m, Docker 30m, and +security 20m; Pages build 20m/deploy 10m; release test 40m, check-version 10m, +build 15m, publish 15m, GitHub-release 15m. + +The JUnit file is an ordinary-test report only: pytest writes +`test-results/junit.xml` with `junit_logging=no`; on failure a size-check marks +it uploadable only at <=5 MiB, then `upload-artifact` retains it for one day. +The Docker job produces no JUnit artifact, so generated TLS/JWT material is +never uploaded. Neither report captures stdout, stderr, nor environment data. + +Add versioned `tests/test_ci_workflows.py` contract tests. They parse the three +workflows and assert SHA pins/version comments, job permissions, workflow/ref +concurrency policy, exact timeouts, independent CI jobs, test markers, bounded +failure-only JUnit behavior, Docker's `make test-docker` target, and the absence +of registry push/release dispatch in CI. `actionlint`, when available, is an +additional syntax check, not the only contract proof. + +## TDD and acceptance criteria + +1. **RED:** workflow-contract tests/static assertions fail for mutable action + refs, broad permissions, dependent jobs, missing timeouts/concurrency, + incorrect test marker selection, absent Docker target, and unsafe artifacts. +2. **GREEN:** all three workflows have the reviewed SHA pins/version comments, + exact concurrency/timeouts, and job-level permissions; CI jobs are + independent; ordinary and release tests select `not live and not docker`; + Docker selection is exact; versioned workflow-contract tests and action + syntax pass. +3. **Local verification:** affected pytest/Make targets, strict docs, Ruff, + workflow static tests, and no-publish checks pass. +4. **GitHub verification:** push only the reviewed branch commits; record the + exact SHA and a green run containing all five CI contexts plus Gitleaks; + verify branch-protection required contexts through GitHub API/CLI. Do not + create a tag, invoke `release.yml`, or publish a registry artifact. + +## Blast radius and rollback + +The blast radius is GitHub scheduling/cost, action compatibility, diagnostics, +and future deployment/release permission placement. Rollback is a reviewed +workflow-only revert. Never weaken required checks, protection, or Gitleaks to +work around a failure. diff --git a/dev/plans/0.5.15/80-ci-reliability-security-status.md b/dev/plans/0.5.15/80-ci-reliability-security-status.md new file mode 100644 index 0000000..0fab5c7 --- /dev/null +++ b/dev/plans/0.5.15/80-ci-reliability-security-status.md @@ -0,0 +1,60 @@ +# Slice 80 — CI reliability, diagnostic signal, and least privilege: status + +**Status:** local implementation, code review, and independent verification +complete; awaiting release-branch push, exact-head GitHub verification, and +branch-protection reconciliation. + +## Delivered + +- All `uses:` actions in CI, documentation, and release workflows are pinned + to reviewed immutable commits with adjacent human-readable version comments. +- CI workflows now use workflow/ref-scoped supersession, bounded job timeouts, + and independent lint, security, and Docker diagnostics. +- The ordinary test job excludes `live` and `docker`; the Docker job delegates + only to Slice 71's `make test-docker` contract, which builds a disposable + local image and does not push it. +- Ordinary-test JUnit output is retained only after failure, only if it is at + most 5 MiB, without pytest logging, and for one day. Docker topology + material is never uploaded. +- Documentation and release permissions are least-privilege and job-local. + Release is intentionally not dispatched, tagged, or published by this work. +- Versioned workflow-contract tests guard every action reference (full SHA plus + adjacent version comment), those properties, and the explicit release + selector `not live and not docker`. + +## Deliberately deferred support decision + +The published package metadata still declares Python `>=3.10` and classifiers +for 3.10 through 3.12, while this release’s CI gate runs 3.12 only. Slice 80 +records the discrepancy but does not silently change either support metadata or +the tested-version matrix. A future owner decision must either restore lower +version CI coverage or revise the public support declaration. + +## Local verification + +- RED: `uv run pytest tests/test_ci_workflows.py -q` failed four contracts + before workflow changes (mutable action refs, missing concurrency, and broad + docs/release permissions). +- FIX-1 RED: the universal immutable-action guard was absent; the strengthened + test suite failed two tests with `NameError` before that guard was added. +- FIX-1 GREEN: the guard rejects a synthetic mutable `actions/checkout@v6` + reference and structurally confines JUnit generation/upload to the ordinary + test job; `uv run pytest tests/test_ci_workflows.py -q` — 5 passed. +- FIX-2 RED: the guard skipped direct `- uses:` action steps, so a valid direct + SHA/comment fixture was incorrectly reported as containing no action. +- FIX-2 GREEN: the guard inspects both nested `uses:` and direct `- uses:` + forms; the test accepts a direct immutable fixture and rejects a direct + mutable `@v6` sibling. The workflow-contract suite remains 5 passed. +- `uv run ruff check tests/test_ci_workflows.py` — passed. +- `uv run ruff format --check tests/test_ci_workflows.py` — passed. +- `actionlint .github/workflows/ci.yml .github/workflows/docs.yml + .github/workflows/release.yml` — passed. +- `git diff --check` — passed. + +## Remaining closeout verification + +1. Push only the reviewed release branch; do not tag, dispatch `release.yml`, + or publish a package/image. +2. Confirm exact-head green CI and Gitleaks, record returned check-context + names/app IDs, then apply and re-read branch protection without weakening + CODEOWNERS or admin-bypass policy. diff --git a/tests/test_ci_workflows.py b/tests/test_ci_workflows.py new file mode 100644 index 0000000..31432a1 --- /dev/null +++ b/tests/test_ci_workflows.py @@ -0,0 +1,179 @@ +"""Versioned safety contracts for the repository's GitHub Actions workflows.""" + +from __future__ import annotations + +from pathlib import Path +import re + +import pytest +import yaml + + +ROOT = Path(__file__).resolve().parents[1] +WORKFLOWS = ROOT / ".github" / "workflows" + +PINS = { + "actions/checkout": ("d23441a48e516b6c34aea4fa41551a30e30af803", "v6"), + "actions/setup-python": ("ece7cb06caefa5fff74198d8649806c4678c61a1", "v6"), + "astral-sh/setup-uv": ("cec208311dfd045dd5311c1add060b2062131d57", "v8.0.0"), + "actions/upload-artifact": ("043fb46d1a93c77aae656e7c1c64a875d1fc6a0a", "v7"), + "actions/configure-pages": ("983d7736d9b0ae728b81ab479565c72886d7745b", "v5"), + "actions/upload-pages-artifact": ("56afc609e74202658d3ffba0e8f6dda462b719fa", "v3"), + "actions/deploy-pages": ("d6db90164ac5ed86f2b6aed7e0febac5b3c0c03e", "v4"), + "actions/download-artifact": ("3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c", "v8"), + "pypa/gh-action-pypi-publish": ( + "dc37677b2e1c63e2034f94d8a5b11f265b73ba33", + "v1.14.2", + ), + "softprops/action-gh-release": ( + "3d0d9888cb7fd7b750713d6e236d1fcb99157228", + "v3.0.2", + ), +} + + +def _text(name: str) -> str: + return (WORKFLOWS / name).read_text() + + +def _workflow(name: str) -> dict[str, object]: + # BaseLoader retains YAML 1.2's ``on`` spelling instead of coercing it to + # Python's legacy YAML-1.1 boolean key. + return yaml.load(_text(name), Loader=yaml.BaseLoader) + + +def _assert_immutable_actions(workflow_text: str) -> None: + """Require every action reference to be immutable and human-auditable.""" + uses_lines = [ + line + for line in workflow_text.splitlines() + if re.match(r"\s*(?:-\s+)?uses:", line) + ] + assert uses_lines, "workflow must contain at least one action reference" + + for line in uses_lines: + assert re.fullmatch( + r"\s*(?:-\s+)?uses:\s+[^\s@]+@[0-9a-f]{40}\s+#\s+\S(?:.*\S)?\s*", + line, + ), f"action reference must use a full SHA with a version comment: {line}" + + +def test_all_actions_are_sha_pinned_with_human_version_comments() -> None: + workflow_text = "\n".join( + _text(name) for name in ("ci.yml", "docs.yml", "release.yml") + ) + + for action, (sha, version) in PINS.items(): + assert f"uses: {action}@{sha} # {version}" in workflow_text + _assert_immutable_actions(workflow_text) + + +def test_immutable_action_guard_rejects_a_mutable_reference() -> None: + valid_direct_reference = ( + "steps:\n" + " - uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6\n" + ) + _assert_immutable_actions(valid_direct_reference) + + with pytest.raises(AssertionError): + _assert_immutable_actions( + valid_direct_reference + " - uses: actions/checkout@v6\n" + ) + + +def test_ci_jobs_are_bounded_independent_and_keep_generated_material_local() -> None: + ci = _workflow("ci.yml") + jobs = ci["jobs"] + assert ci["permissions"] == {"contents": "read"} + assert ci["concurrency"] == { + "group": "${{ github.workflow }}-${{ github.ref }}", + "cancel-in-progress": "true", + } + + expected_timeouts = { + "docs": "15", + "test": "35", + "lint": "15", + "docker": "30", + "security": "20", + } + for name, timeout in expected_timeouts.items(): + assert jobs[name]["timeout-minutes"] == timeout + for name in ("lint", "docker", "security"): + assert "needs" not in jobs[name] + + text = _text("ci.yml") + assert 'pytest -m "not live and not docker"' in text + assert "make test-docker" in text + for prohibited in ( + "docker push", + "podman push", + "pypa/gh-action-pypi-publish", + "softprops/action-gh-release", + "workflow_dispatch", + ): + assert prohibited not in text + assert "--junitxml=test-results/junit.xml" in text + assert "junit_logging=no" in text + + test_steps = jobs["test"]["steps"] + assert ( + sum("--junitxml=test-results/junit.xml" in str(step) for step in test_steps) + == 1 + ) + for job_name, job in jobs.items(): + if job_name == "test": + continue + job_steps = job["steps"] + assert all("--junitxml" not in str(step) for step in job_steps) + assert "test-results/junit.xml" not in str(job_steps) + + safety_step = next(step for step in test_steps if step.get("id") == "junit") + assert safety_step["if"] == "failure()" + assert "5242880" in safety_step["run"] + assert 'echo "upload=true"' in safety_step["run"] + assert 'echo "upload=false"' in safety_step["run"] + + upload_step = next( + step + for step in test_steps + if step.get("name") == "Upload bounded failure JUnit report" + ) + assert upload_step["if"] == "failure() && steps.junit.outputs.upload == 'true'" + assert upload_step["with"] == { + "name": "test-failure-junit", + "path": "test-results/junit.xml", + "retention-days": "1", + } + + +def test_docs_permissions_are_deploy_job_local_and_bounded() -> None: + docs = _workflow("docs.yml") + jobs = docs["jobs"] + assert docs["permissions"] == {"contents": "read"} + assert docs["concurrency"] == { + "group": "${{ github.workflow }}-${{ github.ref }}", + "cancel-in-progress": "true", + } + assert jobs["build"]["timeout-minutes"] == "20" + assert jobs["deploy"]["timeout-minutes"] == "10" + assert "permissions" not in jobs["build"] + assert jobs["deploy"]["permissions"] == {"pages": "write", "id-token": "write"} + + +def test_release_permissions_are_least_privilege_and_tests_exclude_docker() -> None: + release = _workflow("release.yml") + jobs = release["jobs"] + assert release["permissions"] == {"contents": "read"} + assert release["concurrency"] == { + "group": "${{ github.workflow }}-${{ github.ref }}", + "cancel-in-progress": "false", + } + assert jobs["test"]["timeout-minutes"] == "40" + assert jobs["check-version"]["timeout-minutes"] == "10" + assert jobs["build"]["timeout-minutes"] == "15" + assert jobs["publish"]["timeout-minutes"] == "15" + assert jobs["github-release"]["timeout-minutes"] == "15" + assert jobs["publish"]["permissions"] == {"id-token": "write"} + assert jobs["github-release"]["permissions"] == {"contents": "write"} + assert 'pytest --tb=short -q -m "not live and not docker"' in _text("release.yml") From 9e92b500455777c8376815b818ce379f3aa53f79 Mon Sep 17 00:00:00 2001 From: Corey Thompson Date: Sun, 16 Aug 2026 22:44:10 -0500 Subject: [PATCH 19/24] chore(release): prepare 0.5.15 closeout --- .github/workflows/release.yml | 18 +--- .gitleaksignore | 2 + CHANGELOG.md | 24 +++++ airlock/__init__.py | 2 +- airlock/callbacks/tracing.py | 2 +- dev/plans/0.5.15/90-release-closeout-plan.md | 81 ++++++++++++++++ .../0.5.15/90-release-closeout-status.md | 64 +++++++++++++ docs/getting-started/configuration.md | 2 +- pyproject.toml | 2 +- scripts/check-version-consistency.py | 92 +++++++++++-------- tests/test_ci_workflows.py | 4 + tests/test_config_consistency.py | 36 ++++++++ tests/test_startup_validation.py | 26 ++++-- uv.lock | 2 +- 14 files changed, 287 insertions(+), 70 deletions(-) create mode 100644 dev/plans/0.5.15/90-release-closeout-plan.md create mode 100644 dev/plans/0.5.15/90-release-closeout-status.md diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 5fbcce9..6fb3317 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -65,7 +65,7 @@ jobs: - name: Build strict documentation site run: uv run mkdocs build --strict --site-dir /tmp/airlock-docs - # Gate: verify the tag version matches pyproject.toml + # Gate: verify the tag matches every canonical Airlock version source. check-version: runs-on: ubuntu-latest timeout-minutes: 10 @@ -77,20 +77,8 @@ jobs: with: python-version: "3.12" - - name: Verify tag matches pyproject.toml version - run: | - TAG_VERSION="${GITHUB_REF#refs/tags/v}" - PKG_VERSION=$(python -c " - import tomllib - with open('pyproject.toml', 'rb') as f: - print(tomllib.load(f)['project']['version']) - ") - echo "Tag version: $TAG_VERSION" - echo "Package version: $PKG_VERSION" - if [ "$TAG_VERSION" != "$PKG_VERSION" ]; then - echo "::error::Tag v$TAG_VERSION does not match pyproject.toml version $PKG_VERSION" - exit 1 - fi + - name: Verify tag and canonical Airlock versions + run: python scripts/check-version-consistency.py --tag "${GITHUB_REF_NAME}" # Build sdist and wheel build: diff --git a/.gitleaksignore b/.gitleaksignore index bea2001..3141507 100644 --- a/.gitleaksignore +++ b/.gitleaksignore @@ -36,3 +36,5 @@ tests/test_response_scanner.py:generic-api-key:151 8f6b151ef72bcf38dc967395f12ef8f63844e384:scripts/setup.sh:curl-auth-header:87 daf23856dcf634aa3eb0248502ba3e13cdd0c7fe:README.md:curl-auth-header:74 f5626f2959006ab8636c237162388765f9b38251:tests/test_guardrail_overrides.py:generic-api-key:15 +70d69b1c0cb9c0828db9e38720773de8473cfc8a:tests/test_startup_validation.py:generic-api-key:62 +70d69b1c0cb9c0828db9e38720773de8473cfc8a:tests/test_startup_validation.py:generic-api-key:68 diff --git a/CHANGELOG.md b/CHANGELOG.md index cb8fe1e..5b25d40 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,30 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +### Added + +- **Opt-in Admin read surface.** The Admin API remains disabled by default and + now provides a bounded, redacted configuration snapshot for authorized local + operators. +- **Secure remote fleet visibility.** The terminal UI can read a bounded fleet + inventory over CA-verified TLS with scoped read-only JWT capabilities; its + Docker topology verification exercises two independently secured proxy + containers. +- **Configured-provider startup guidance.** Airlock warns when a recognized + provider credential is configured without a matching served alias, without + inspecting arbitrary environment-variable names. + +### Changed + +- **CI verification.** The ordinary test suite and the opt-in Docker topology + test are independently gated in CI, alongside pinned workflow actions and + secret scanning. + +### Deferred + +- Virtual-key lifecycle management remains deferred to the 0.6.0 identity and + durability contract; this release does not add a virtual-key control plane. + ## [0.5.14] — 2026-08-12 ### Added diff --git a/airlock/__init__.py b/airlock/__init__.py index 4c24d63..a41d415 100644 --- a/airlock/__init__.py +++ b/airlock/__init__.py @@ -1,3 +1,3 @@ """Airlock — Enterprise LLM proxy built on LiteLLM.""" -__version__ = "0.5.14" +__version__ = "0.5.15" diff --git a/airlock/callbacks/tracing.py b/airlock/callbacks/tracing.py index 88cca79..840339a 100644 --- a/airlock/callbacks/tracing.py +++ b/airlock/callbacks/tracing.py @@ -48,7 +48,7 @@ def _get_tracer() -> Any: provider = TracerProvider(resource=resource) trace.set_tracer_provider(provider) - return trace.get_tracer("airlock", "0.5.14") + return trace.get_tracer("airlock", "0.5.15") _tracer = _get_tracer() diff --git a/dev/plans/0.5.15/90-release-closeout-plan.md b/dev/plans/0.5.15/90-release-closeout-plan.md new file mode 100644 index 0000000..3381a7a --- /dev/null +++ b/dev/plans/0.5.15/90-release-closeout-plan.md @@ -0,0 +1,81 @@ +# Slice 90 — 0.5.15 release closeout (non-publishing) + +**Status:** admitted for closeout planning and implementation on 2026-08-16; +release publication remains explicitly out of scope. + +## Re-evaluation + +Slices 10, 20, 30, 40, 50, 70, 71, and 80 are implemented on this branch. +Slice 60 is deferred to 0.6.0. Slice 71 subsequently supplied the combined +ordinary-suite evidence that earlier Slice 20/40/50 records could not capture: +`make test` passed 3421 tests, with 112 deselected and one expected XPASS. +Those historical records remain unchanged. + +Exact-head PR CI initially exposed two generic-api-key detections in Slice 30's +non-secret test literals. The approved narrow correction derives the matching +name from the authoritative finite registry; it does not alter the Gitleaks +policy or baseline. + +## Scope and requirements + +1. Prepare the branch as version `0.5.15` using the sole updater + `scripts/set-version.sh --set-version 0.5.15`; it owns the canonical + package/runtime/tracing fields and the editable `airlock-llm` entry in + `uv.lock`. Do not hand-edit the lock. Extend the version checker and its + focused regression coverage so package/runtime/tracing/**lock** fields and + `v0.5.15` agree. +2. Keep `CHANGELOG.md` under **Unreleased** until a separately authorized tag + and publication. Add concise public-facing release notes covering the + default-off Admin API read surface, secure CA-verified-TLS plus scoped-JWT + remote/fleet read-only TUI, + unused configured-provider warnings, and CI/Docker verification. Document + the deferred virtual-key control plane separately. +3. Replace the one public stale `0.5.14` configuration wording with `0.5.15`; + retain historical release references and never rewrite prior changelog + entries or prior plan/status records. +4. Commit and push only reviewed branch changes. Obtain green exact-head CI + (`CI` docs/test/lint/Docker/security and Gitleaks), then discover exact + check-run context names/app IDs and require them through GitHub branch + protection. Re-read protection and retain strict checks, required + CODEOWNERS review, and no admin bypass. +5. Do not create a tag, dispatch `release.yml`, invoke PyPI/GitHub-release + actions, merge PR #49, publish any package/image, weaken scanning/protection, + or claim lower-Python support from the 3.12-only CI gate. + +## Design and verification + +The release workflow is tag-triggered and publication-capable, so it is never +invoked during this slice. `scripts/check-version-consistency.py --tag v0.5.15` +is a local string-consistency check; `scripts/verify-release-gates.py` runs +only after exact-head CI is green and receives an explicit non-created tag +argument. Neither creates a tag or registry artifact. + +TDD: + +1. **RED:** add/adjust a canonical-version contract and fixture that fail while + the editable lock entry drifts from the other three fields; prove + `--tag v0.5.15` fails before alignment. +2. **GREEN:** align all four fields, make the version/tag contract pass, and + ensure public documentation has no stale active configuration claim. +3. Run `make sync && make verify`, `make test`, `make test-docker`, `uv lock + --check`, strict MkDocs, documentation contracts, workflow contracts and + actionlint, targeted startup-warning tests, and Gitleaks working-tree and + commit-range scans. +4. On GitHub, record the exact green head SHA, run URLs, check-run names/app + IDs, and protection readback. PR approval/merge is outside this slice. + +## External authority + +Changing GitHub branch protection is an owner/admin-only mutation. Do it only +after the exact head is green and its check-run names/app IDs are recorded; +apply only those contexts and preserve `strict=true`, the existing Gitleaks +requirement, CODEOWNERS review, and admin enforcement. The PR author/agent +does not self-approve or merge PR #49. + +## Blast radius and rollback + +The product/runtime blast radius is version strings and public release notes; +CI/protection changes affect future gating only. Rollback is a normal reviewed +branch revert. Never roll back by weakening branch protection or suppressing +new secrets. The Python `>=3.10` metadata versus 3.12-only CI discrepancy is +an explicitly deferred support-policy decision, not a release claim. diff --git a/dev/plans/0.5.15/90-release-closeout-status.md b/dev/plans/0.5.15/90-release-closeout-status.md new file mode 100644 index 0000000..0a50ea0 --- /dev/null +++ b/dev/plans/0.5.15/90-release-closeout-status.md @@ -0,0 +1,64 @@ +# Slice 90 — 0.5.15 release closeout status + +**Status:** implementation verified locally — awaiting the parent closeout's +exact-head GitHub CI and protection evidence; non-publishing throughout. + +## Implemented scope + +- Added a lock-aware version contract: `pyproject.toml`, `airlock.__version__`, + tracing resource version, and the sole editable `airlock-llm` entry in + `uv.lock` must agree, and an optional `v` tag must agree with them. +- Applied the sole supported updater: + `scripts/set-version.sh --set-version 0.5.15`. The lock was updated by `uv`; + it was not hand edited. +- Added public Unreleased notes for the default-off Admin read surface, the + CA-verified-TLS plus scoped-JWT fleet read-only TUI, startup provider-credential guidance, and + CI/Docker verification. Virtual-key lifecycle remains explicitly deferred to + the 0.6.0 identity/durability contract. +- Corrected the active DeepSeek configuration guide from 0.5.14 to 0.5.15; + historical release references were left intact. +- The tag-triggered release gate delegates to the same lock-aware canonical + version checker. This changes no trigger and this slice creates no tag. + +## TDD evidence + +- **RED:** the new `--tag v0.5.15` contract failed while canonical fields were + still 0.5.14. +- **GREEN:** after the supported updater, the canonical/tag checker passed and + the focused version/startup/workflow suite passed: **20 tests**. + +## Local verification + +- `uv run python scripts/check-version-consistency.py --tag v0.5.15` — pass. +- `uv lock --check`, focused Ruff/check-format, and `git diff --check` — pass. +- `uv run mkdocs build --strict` — pass. +- Working-tree `gitleaks detect --source . --no-git --redact --exit-code 1` — + pass, with no leaks. +- Merge-base-to-head Gitleaks scan — pass after the two exact historic test + fingerprints; it retains all new-finding detection. +- `make sync && make verify`, strict MkDocs, documentation/workflow/startup/ + version contracts (91 passed, 1 expected XPASS), `make test` (3427 passed, + 112 deselected, 1 expected XPASS), and `make test-docker` (1 passed) — pass. +- Changed-file Ruff/check-format and actionlint — pass. A deliberately broad + Ruff run also reports four pre-existing style errors in the unrelated + `scripts/benchmark_fathomdb.py`; no Slice 90 file is implicated. +- The independent code review required one FIX-1: change the future + tag-triggered release gate from a pyproject-only comparison to the canonical + checker. FIX-1 was approved; the tag trigger and all publication paths are + unchanged. + +## Gitleaks history note + +The explicit commit-range scan from the merge base through the current +uncommitted head reports two historic generic-key test literals in the earlier +Slice 30 commit. The approved working-tree fixture correction removes those +literals. Because PR scanning retains history, `.gitleaksignore` adds only the +two exact reviewed commit/path/rule/line fingerprints; it adds no path, rule, +or broad baseline suppression. The parent closeout must record the exact-head +GitHub Gitleaks result after the reviewed changes are committed. + +## Pending closeout evidence + +This record will name final local-gate results, the exact GitHub head/check +contexts/app IDs, and branch-protection readback. It will not claim a tag, +merge, or package/image publication. diff --git a/docs/getting-started/configuration.md b/docs/getting-started/configuration.md index 22714e2..6b45f51 100644 --- a/docs/getting-started/configuration.md +++ b/docs/getting-started/configuration.md @@ -64,7 +64,7 @@ provider-specific request fields. DeepSeek is also disabled until an operator adds a reviewed deployment entry. Use the explicit stable base below; the pinned LiteLLM version otherwise has a -beta-default path that is not Airlock’s supported 0.5.14 configuration. +beta-default path that is not Airlock’s supported 0.5.15 configuration. ```yaml model_list: diff --git a/pyproject.toml b/pyproject.toml index 92acd4a..766637d 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta" [project] name = "airlock-llm" -version = "0.5.14" +version = "0.5.15" description = "Enterprise LLM proxy built on LiteLLM — logging, guardrails, and unified access for AI coding tools." readme = "README.md" license = "Apache-2.0" diff --git a/scripts/check-version-consistency.py b/scripts/check-version-consistency.py index 16b663a..c5d998b 100755 --- a/scripts/check-version-consistency.py +++ b/scripts/check-version-consistency.py @@ -23,62 +23,74 @@ def read_file(path: pathlib.Path) -> str: return handle.read() -def main() -> int: - parser = argparse.ArgumentParser( - description="Verify that Airlock package versions stay aligned across files." - ) - parser.add_argument( - "--tag", - help="Optional release tag to validate against, e.g. v0.1.0", - ) - args = parser.parse_args() - - repo_root = pathlib.Path(__file__).resolve().parent.parent - - # 1. pyproject.toml +def editable_lock_version(repo_root: pathlib.Path) -> str | None: + """Return the version of Airlock's editable package entry in ``uv.lock``.""" + lock = read_toml(repo_root / "uv.lock") + matches = [ + package + for package in lock.get("package", []) + if package.get("name") == "airlock-llm" + and package.get("source") == {"editable": "."} + ] + if len(matches) != 1: + return None + return matches[0].get("version") + + +def version_mismatches(repo_root: pathlib.Path, tag: str | None = None) -> list[str]: + """Return every canonical version or optional tag disagreement.""" pyproject = read_toml(repo_root / "pyproject.toml") pyproject_version = pyproject["project"]["version"] - # 2. airlock/__init__.py init_content = read_file(repo_root / "airlock" / "__init__.py") init_match = re.search(r'^__version__ = "([^"]*)"', init_content, re.MULTILINE) init_version = init_match.group(1) if init_match else None - # 3. airlock/callbacks/tracing.py tracing_content = read_file(repo_root / "airlock" / "callbacks" / "tracing.py") tracing_match = re.search( r'trace\.get_tracer\("airlock", "([^"]*)"\)', tracing_content ) tracing_version = tracing_match.group(1) if tracing_match else None + lock_version = editable_lock_version(repo_root) + + mismatches: list[str] = [] + for source, version in ( + ("__init__.py", init_version), + ("tracing.py", tracing_version), + ("uv.lock editable airlock-llm", lock_version), + ): + if pyproject_version != version: + mismatches.append( + f"version mismatch: pyproject.toml={pyproject_version} " + f"{source}={version}" + ) - rc = 0 - - if pyproject_version != init_version: - print( - f"version mismatch: pyproject.toml={pyproject_version} __init__.py={init_version}", - file=sys.stderr, + if tag and tag != f"v{pyproject_version}": + mismatches.append( + f"tag/version mismatch: tag={tag} expected=v{pyproject_version}" ) - rc = 1 + return mismatches - if pyproject_version != tracing_version: - print( - f"version mismatch: pyproject.toml={pyproject_version} tracing.py={tracing_version}", - file=sys.stderr, - ) - rc = 1 - - if args.tag: - expected_tag = f"v{pyproject_version}" - if args.tag != expected_tag: - print( - f"tag/version mismatch: tag={args.tag} expected={expected_tag}", - file=sys.stderr, - ) - rc = 1 - if rc == 0: - print(f"version check passed: {pyproject_version}") - return rc +def main() -> int: + parser = argparse.ArgumentParser( + description="Verify that Airlock package versions stay aligned across files." + ) + parser.add_argument( + "--tag", + help="Optional release tag to validate against, e.g. v0.1.0", + ) + args = parser.parse_args() + + repo_root = pathlib.Path(__file__).resolve().parent.parent + mismatches = version_mismatches(repo_root, args.tag) + if mismatches: + print("\n".join(mismatches), file=sys.stderr) + return 1 + + pyproject_version = read_toml(repo_root / "pyproject.toml")["project"]["version"] + print(f"version check passed: {pyproject_version}") + return 0 if __name__ == "__main__": diff --git a/tests/test_ci_workflows.py b/tests/test_ci_workflows.py index 31432a1..5d99746 100644 --- a/tests/test_ci_workflows.py +++ b/tests/test_ci_workflows.py @@ -177,3 +177,7 @@ def test_release_permissions_are_least_privilege_and_tests_exclude_docker() -> N assert jobs["publish"]["permissions"] == {"id-token": "write"} assert jobs["github-release"]["permissions"] == {"contents": "write"} assert 'pytest --tb=short -q -m "not live and not docker"' in _text("release.yml") + assert ( + 'python scripts/check-version-consistency.py --tag "${GITHUB_REF_NAME}"' + in _text("release.yml") + ) diff --git a/tests/test_config_consistency.py b/tests/test_config_consistency.py index 372a6bc..5976cfb 100644 --- a/tests/test_config_consistency.py +++ b/tests/test_config_consistency.py @@ -718,6 +718,8 @@ def test_versions_agree(self): [ sys.executable, str(repo_root / "scripts" / "check-version-consistency.py"), + "--tag", + "v0.5.15", ], capture_output=True, text=True, @@ -725,3 +727,37 @@ def test_versions_agree(self): assert result.returncode == 0, ( f"version drift detected:\n{result.stderr or result.stdout}" ) + + def test_checker_reports_editable_lock_entry_drift(self, tmp_path): + """The editable Airlock lock entry is a fourth canonical version source.""" + import importlib.util + + repo_root = Path(__file__).resolve().parent.parent + checker_path = repo_root / "scripts" / "check-version-consistency.py" + spec = importlib.util.spec_from_file_location("version_checker", checker_path) + assert spec is not None and spec.loader is not None + checker = importlib.util.module_from_spec(spec) + spec.loader.exec_module(checker) + + (tmp_path / "airlock" / "callbacks").mkdir(parents=True) + (tmp_path / "pyproject.toml").write_text( + '[project]\nversion = "0.5.15"\n', encoding="utf-8" + ) + (tmp_path / "airlock" / "__init__.py").write_text( + '__version__ = "0.5.15"\n', encoding="utf-8" + ) + (tmp_path / "airlock" / "callbacks" / "tracing.py").write_text( + 'trace.get_tracer("airlock", "0.5.15")\n', encoding="utf-8" + ) + (tmp_path / "uv.lock").write_text( + '[[package]]\nname = "airlock-llm"\nversion = "0.5.14"\n' + 'source = { editable = "." }\n', + encoding="utf-8", + ) + + mismatches = checker.version_mismatches(tmp_path, "v0.5.15") + + assert mismatches == [ + "version mismatch: pyproject.toml=0.5.15 " + "uv.lock editable airlock-llm=0.5.14" + ] diff --git a/tests/test_startup_validation.py b/tests/test_startup_validation.py index dcd7388..cb879f7 100644 --- a/tests/test_startup_validation.py +++ b/tests/test_startup_validation.py @@ -7,6 +7,7 @@ import pytest from airlock.startup_validation import ( + PROVIDER_CREDENTIAL_SPECS, credential_without_alias_warnings, emit_provider_credential_warnings, ) @@ -57,20 +58,20 @@ def test_enabled_provider_and_blank_or_unknown_credentials_do_not_warn(tmp_path) @pytest.mark.parametrize( - ("provider", "environment_variable", "model"), + ("provider", "model"), [ - ("anthropic", "ANTHROPIC_API_KEY", "anthropic/claude"), - ("openai", "OPENAI_API_KEY", "openai/gpt"), - ("gemini", "GOOGLE_AISTUDIO_API_KEY", "gemini/flash"), - ("mistral", "MISTRAL_API_KEY", "mistral/large"), - ("openrouter", "OPENROUTER_API_KEY", "openrouter/openai/gpt"), - ("deepseek", "DEEPSEEK_API_KEY", "deepseek/chat"), - ("perplexity", "PERPLEXITY_API_KEY", "perplexity/sonar"), - ("tavily", "TAVILY_API_KEY", "tavily/search"), + ("anthropic", "anthropic/claude"), + ("openai", "openai/gpt"), + ("gemini", "gemini/flash"), + ("mistral", "mistral/large"), + ("openrouter", "openrouter/openai/gpt"), + ("deepseek", "deepseek/chat"), + ("perplexity", "perplexity/sonar"), + ("tavily", "tavily/search"), ], ) def test_each_recognised_provider_is_suppressed_by_a_matching_alias( - tmp_path, provider, environment_variable, model + tmp_path, provider, model ): config = tmp_path / "config.yaml" _write_config( @@ -79,6 +80,11 @@ def test_each_recognised_provider_is_suppressed_by_a_matching_alias( f" - model_name: {provider}\n" f" litellm_params: {{model: {model}}}\n", ) + environment_variable = next( + spec.environment_variable + for spec in PROVIDER_CREDENTIAL_SPECS + if spec.provider == provider + ) assert ( credential_without_alias_warnings(config, {environment_variable: "yes"}.get) == () diff --git a/uv.lock b/uv.lock index dc31ad0..30be470 100644 --- a/uv.lock +++ b/uv.lock @@ -168,7 +168,7 @@ wheels = [ [[package]] name = "airlock-llm" -version = "0.5.14" +version = "0.5.15" source = { editable = "." } dependencies = [ { name = "litellm", extra = ["proxy"] }, From ce9a97746262467003a96b6de67186be9c4ff007 Mon Sep 17 00:00:00 2001 From: Corey Thompson Date: Sun, 16 Aug 2026 22:52:13 -0500 Subject: [PATCH 20/24] fix(ci): type Slice 20 threat backoff adapter --- airlock/proxy_errors.py | 9 ++++++++- tests/test_proxy_errors.py | 4 ++++ 2 files changed, 12 insertions(+), 1 deletion(-) diff --git a/airlock/proxy_errors.py b/airlock/proxy_errors.py index 04ad208..f4e654c 100644 --- a/airlock/proxy_errors.py +++ b/airlock/proxy_errors.py @@ -176,12 +176,19 @@ def __init__(self, message: str, *, retry_after: float) -> None: self.retry_after = float(retry_after) -class AirlockThreatBackoff(ProxyException, RateLimitError): +class AirlockThreatBackoff(ProxyException, RateLimitError): # type: ignore[misc] """A local Fast Guardian threat backoff, not a provider rate limit. Retains only the remaining duration needed to tell the client when to retry; client identity, threat signals, and triggering request details stay inside the Guardian/logging boundary. + + LiteLLM's proxy path requires ``ProxyException`` while Airlock's Guardian + contract requires ``RateLimitError``. Those independently maintained base + classes annotate their overlapping protocol fields (``code``, ``headers``, + and ``type``) incompatibly, although this adapter explicitly normalizes + every one below before it crosses either boundary. Keep the suppression on + this intentional compatibility join only; do not broaden it to the module. """ def __init__(self, *, retry_after: float) -> None: diff --git a/tests/test_proxy_errors.py b/tests/test_proxy_errors.py index c260e3c..a6e41c3 100644 --- a/tests/test_proxy_errors.py +++ b/tests/test_proxy_errors.py @@ -7,6 +7,7 @@ import pytest from litellm import RateLimitError +from litellm.proxy._types import ProxyException from airlock.proxy_errors import ( AirlockEndpointNotSupported, @@ -225,6 +226,9 @@ class TestAirlockThreatBackoff: def test_is_a_rate_limit_error_without_client_or_heuristic_fields(self): exc = AirlockThreatBackoff(retry_after=2.1) assert isinstance(exc, RateLimitError) + # LiteLLM's guardrail path recognizes only ProxyException before + # FastAPI gets a chance to select Airlock's exception handler. + assert isinstance(exc, ProxyException) assert exc.retry_after == 2.1 assert not hasattr(exc, "client_id") assert not hasattr(exc, "reason") From 08d87377bb0d13065289f94353319e7d0a8a87cc Mon Sep 17 00:00:00 2001 From: Corey Thompson Date: Sun, 16 Aug 2026 23:23:04 -0500 Subject: [PATCH 21/24] fix(diagnostics): bound in-flight trim skip --- airlock/callbacks/oom_diagnostics.py | 21 +++++++++- .../instrumentation/oom-high-water.md | 7 +++- docs/operations.md | 15 +++++--- tests/test_oom_diagnostics.py | 38 +++++++++++++++++++ 4 files changed, 72 insertions(+), 9 deletions(-) diff --git a/airlock/callbacks/oom_diagnostics.py b/airlock/callbacks/oom_diagnostics.py index bd335b2..05af61f 100644 --- a/airlock/callbacks/oom_diagnostics.py +++ b/airlock/callbacks/oom_diagnostics.py @@ -342,6 +342,25 @@ def _signal_usr1(self, _signum: int, _frame: Any) -> None: def _signal_usr2(self, _signum: int, _frame: Any) -> None: self._signal_snapshot("signal_usr2", trim=True) + def _record_inflight_trim_skip(self, phase: str, in_flight: int) -> None: + """Record why a trim was skipped without perturbing the live workload. + + A full signal snapshot enumerates the Python heap. That work is + specifically unhelpful while requests are in flight, and can hold the + signal gate long enough to make a second operator signal look stuck. + Preserve the bounded audit event while deliberately omitting sampled + process details. + """ + self._append( + { + "ts_monotonic_ns": time.monotonic_ns(), + "phase": f"{phase}_skipped_in_flight", + "sequence": None, + "in_flight": in_flight, + "trim": {"attempted": False, "reason": "requests_in_flight"}, + } + ) + def _signal_snapshot(self, phase: str, *, trim: bool) -> None: if not _enabled() or not self._signal_lock.acquire(blocking=False): return @@ -351,7 +370,7 @@ def worker() -> None: with self._lock: in_flight = self._in_flight if trim and in_flight: - self.snapshot(f"{phase}_skipped_in_flight") + self._record_inflight_trim_skip(phase, in_flight) return self.snapshot(f"{phase}_before") if trim: diff --git a/dev/debugging/instrumentation/oom-high-water.md b/dev/debugging/instrumentation/oom-high-water.md index 9c9cc96..fecfa81 100644 --- a/dev/debugging/instrumentation/oom-high-water.md +++ b/dev/debugging/instrumentation/oom-high-water.md @@ -50,10 +50,13 @@ Each record has a monotonic timestamp and a phase: - `periodic` — every `AIRLOCK_OOM_DIAGNOSTICS_EVERY` requests. - `signal_usr1_*` or `signal_usr2_*` — an operator snapshot. -The payload includes cgroup current/peak/high/max/event counters, process RSS, +Most payloads include cgroup current/peak/high/max/event counters, process RSS, `smaps_rollup` anonymous/huge-page values, PSI, glibc `mallinfo2`, thread/FD counts, optional tracemalloc totals, aggregate GC type counts at checkpoints, -and LiteLLM/httpx client-pool counts. +and LiteLLM/httpx client-pool counts. `signal_usr2_skipped_in_flight` is the +deliberate exception: it contains only its timestamp, phase, in-flight count, +and `trim` decline reason. It does not scan the heap or sample process state +while live requests make trimming unsafe. Important: `in_flight` is decremented at `callback_complete`, not at `provider_response`. A growing value can therefore mean a stuck callback or diff --git a/docs/operations.md b/docs/operations.md index 7319aee..b648864 100644 --- a/docs/operations.md +++ b/docs/operations.md @@ -221,12 +221,15 @@ AIRLOCK_OOM_DIAGNOSTICS_MAX_RECORDS=6000 ``` It writes aggregate process, allocator, file-descriptor, thread, cgroup, and -memory-pressure counters. It does not write request/response bodies, headers, -model names, exception text, or client metadata. The directory defaults to -`/tmp/airlock-oom-diagnostics`; output files are owner-only. Secure or -pre-create a configured directory with the permissions your deployment needs. -Disable the recorder after the investigation. The repository's engineering OOM -instrumentation runbook contains the signal and high-water procedure. +memory-pressure counters. A `SIGUSR2` trim declined because requests are active +is intentionally recorded as a minimal audit event instead, so it does not +perturb those requests with a heap scan. It does not write request/response +bodies, headers, model names, exception text, or client metadata. The directory +defaults to `/tmp/airlock-oom-diagnostics`; output files are owner-only. Secure +or pre-create a configured directory with the permissions your deployment +needs. Disable the recorder after the investigation. The repository's +engineering OOM instrumentation runbook contains the signal and high-water +procedure. ## Startup Modes diff --git a/tests/test_oom_diagnostics.py b/tests/test_oom_diagnostics.py index b30df80..33defd7 100644 --- a/tests/test_oom_diagnostics.py +++ b/tests/test_oom_diagnostics.py @@ -83,6 +83,44 @@ def test_diagnostics_are_bounded_and_trim_skips_inflight(monkeypatch, tmp_path): assert any(record["phase"] == "signal_usr2_skipped_in_flight" for record in records) +def test_inflight_trim_signal_uses_a_minimal_skip_record(monkeypatch, tmp_path): + """An in-flight trim must not walk the heap before releasing its signal lock.""" + monkeypatch.setenv("AIRLOCK_OOM_DIAGNOSTICS", "true") + monkeypatch.setenv("AIRLOCK_OOM_DIAGNOSTICS_DIR", str(tmp_path)) + + diagnostics = OOMDiagnostics() + diagnostics.pre_call({"metadata": {}}) + snapshot_calls: list[tuple[tuple[object, ...], dict[str, object]]] = [] + monkeypatch.setattr( + diagnostics, + "snapshot", + lambda *args, **kwargs: snapshot_calls.append((args, kwargs)), + ) + diagnostics._signal_snapshot("signal_usr2", trim=True) + deadline = time.monotonic() + 2 + while diagnostics._signal_lock.locked() and time.monotonic() < deadline: + time.sleep(0.01) + + assert not diagnostics._signal_lock.locked() + assert snapshot_calls == [] + records = [ + json.loads(line) + for line in next(tmp_path.glob("*.jsonl")).read_text().splitlines() + ] + skipped = next( + record + for record in records + if record["phase"] == "signal_usr2_skipped_in_flight" + ) + assert skipped == { + "ts_monotonic_ns": skipped["ts_monotonic_ns"], + "phase": "signal_usr2_skipped_in_flight", + "sequence": None, + "in_flight": 1, + "trim": {"attempted": False, "reason": "requests_in_flight"}, + } + + def test_tracemalloc_can_be_disabled_for_a_low_perturbation_replay( monkeypatch, tmp_path ): From 2b7936c57da42df37e80ed4f4a52f4c4b80d48ef Mon Sep 17 00:00:00 2001 From: Corey Thompson Date: Sun, 16 Aug 2026 23:47:08 -0500 Subject: [PATCH 22/24] docs(release): record 0.5.15 closeout evidence --- .../71-slice70-docker-topology-status.md | 4 +++ .../80-ci-reliability-security-status.md | 28 +++++++++++++------ .../0.5.15/90-release-closeout-status.md | 27 ++++++++++++------ 3 files changed, 41 insertions(+), 18 deletions(-) diff --git a/dev/plans/0.5.15/71-slice70-docker-topology-status.md b/dev/plans/0.5.15/71-slice70-docker-topology-status.md index 3ca89e2..8b93b4c 100644 --- a/dev/plans/0.5.15/71-slice70-docker-topology-status.md +++ b/dev/plans/0.5.15/71-slice70-docker-topology-status.md @@ -25,6 +25,10 @@ test containers and generated temporary test material. 112 deselected, 1 expected XPASS in 341.69s. Normal test selection excludes the Docker marker; CI's Docker job invokes only the opt-in target after locked test dependency provisioning. +- Exact-head GitHub CI independently ran that opt-in target successfully at + `08d8737` ([Docker job](https://github.com/coreyt/airlock/actions/runs/31994296354/job/95283132492), + 1m10s). It used the disposable local-image contract only; no image was + pushed or published. ## Boundary and follow-up diff --git a/dev/plans/0.5.15/80-ci-reliability-security-status.md b/dev/plans/0.5.15/80-ci-reliability-security-status.md index 0fab5c7..06fbfc5 100644 --- a/dev/plans/0.5.15/80-ci-reliability-security-status.md +++ b/dev/plans/0.5.15/80-ci-reliability-security-status.md @@ -1,8 +1,7 @@ # Slice 80 — CI reliability, diagnostic signal, and least privilege: status -**Status:** local implementation, code review, and independent verification -complete; awaiting release-branch push, exact-head GitHub verification, and -branch-protection reconciliation. +**Status:** complete — exact-head GitHub CI and strict branch-protection +reconciliation verified 2026-08-17; non-publishing throughout. ## Delivered @@ -51,10 +50,21 @@ version CI coverage or revise the public support declaration. .github/workflows/release.yml` — passed. - `git diff --check` — passed. -## Remaining closeout verification +## Exact-head GitHub verification and protection readback -1. Push only the reviewed release branch; do not tag, dispatch `release.yml`, - or publish a package/image. -2. Confirm exact-head green CI and Gitleaks, record returned check-context - names/app IDs, then apply and re-read branch protection without weakening - CODEOWNERS or admin-bypass policy. +- Reviewed head `08d8737` passed the complete CI workflow + [31994296354](https://github.com/coreyt/airlock/actions/runs/31994296354): + `docs`, `test (3.12)` (3,428 passed, 112 deselected, 1 expected XPASS; + 18m44s), `lint`, `security`, and Slice 71's `docker` topology job all + succeeded. Gitleaks workflow + [31994296357](https://github.com/coreyt/airlock/actions/runs/31994296357) + also succeeded. +- GitHub returned each check as GitHub Actions App `15368`. Main now requires + strict, app-bound checks `docs`, `test (3.12)`, `lint`, `security`, `docker`, + and `scan`. `scan` is the verified Gitleaks job; GitHub's app-bound + `checks` API replaces the former legacy `gitleaks / scan` string rather than + retaining both representations. +- Full protection readback confirms strict mode remains enabled, one approving + review and CODEOWNERS remain required, and admin enforcement remains enabled. + Force pushes and deletions remain disabled. No workflow was dispatched and + no tag, package, or image was published. diff --git a/dev/plans/0.5.15/90-release-closeout-status.md b/dev/plans/0.5.15/90-release-closeout-status.md index 0a50ea0..891b474 100644 --- a/dev/plans/0.5.15/90-release-closeout-status.md +++ b/dev/plans/0.5.15/90-release-closeout-status.md @@ -1,7 +1,7 @@ # Slice 90 — 0.5.15 release closeout status -**Status:** implementation verified locally — awaiting the parent closeout's -exact-head GitHub CI and protection evidence; non-publishing throughout. +**Status:** complete — local and exact-head GitHub closeout verification passed +2026-08-17; non-publishing throughout. ## Implemented scope @@ -37,7 +37,7 @@ exact-head GitHub CI and protection evidence; non-publishing throughout. - Merge-base-to-head Gitleaks scan — pass after the two exact historic test fingerprints; it retains all new-finding detection. - `make sync && make verify`, strict MkDocs, documentation/workflow/startup/ - version contracts (91 passed, 1 expected XPASS), `make test` (3427 passed, + version contracts (91 passed, 1 expected XPASS), `make test` (3428 passed, 112 deselected, 1 expected XPASS), and `make test-docker` (1 passed) — pass. - Changed-file Ruff/check-format and actionlint — pass. A deliberately broad Ruff run also reports four pre-existing style errors in the unrelated @@ -54,11 +54,20 @@ uncommitted head reports two historic generic-key test literals in the earlier Slice 30 commit. The approved working-tree fixture correction removes those literals. Because PR scanning retains history, `.gitleaksignore` adds only the two exact reviewed commit/path/rule/line fingerprints; it adds no path, rule, -or broad baseline suppression. The parent closeout must record the exact-head -GitHub Gitleaks result after the reviewed changes are committed. +or broad baseline suppression. The exact-head GitHub Gitleaks result passed at +`08d8737`. -## Pending closeout evidence +## Exact-head closeout evidence -This record will name final local-gate results, the exact GitHub head/check -contexts/app IDs, and branch-protection readback. It will not claim a tag, -merge, or package/image publication. +- Head `08d8737` passed CI workflow + [31994296354](https://github.com/coreyt/airlock/actions/runs/31994296354) + (`docs`, `test (3.12)`, `lint`, `security`, and `docker`) and Gitleaks + workflow [31994296357](https://github.com/coreyt/airlock/actions/runs/31994296357) + (`scan`). GitHub reports every context from GitHub Actions App `15368`. +- Main protection now strictly requires the six app-bound contexts: `docs`, + `test (3.12)`, `lint`, `security`, `docker`, and Gitleaks `scan`. Readback + confirms CODEOWNERS and one approval remain required and admin enforcement + remains enabled. +- This closes preparation only: no tag was created, no release workflow was + dispatched, no registry received an artifact, and PR #49 was neither + self-approved nor merged. From 208175a7492390eb3cd705f7f79a8eed13dd5771 Mon Sep 17 00:00:00 2001 From: Corey Thompson Date: Mon, 17 Aug 2026 00:16:35 -0500 Subject: [PATCH 23/24] test(tui): isolate tab bar widget checks --- tests/test_tui.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/test_tui.py b/tests/test_tui.py index 048db59..2ff48bd 100644 --- a/tests/test_tui.py +++ b/tests/test_tui.py @@ -1075,7 +1075,7 @@ async def test_guards_pause_toggle() -> None: async def test_tab_bar_activate() -> None: from airlock.tui.widgets.tab_bar import TabBar - app = AirlockApp() + app = AirlockApp(test_harness=True) async with app.run_test(size=(120, 40)) as _pilot: tab_bar = app.query_one("#tab-bar", TabBar) assert tab_bar._active == "overview" @@ -1090,7 +1090,7 @@ async def test_tab_bar_activate() -> None: async def test_tab_bar_update_badge() -> None: from airlock.tui.widgets.tab_bar import TabBar - app = AirlockApp() + app = AirlockApp(test_harness=True) async with app.run_test(size=(120, 40)) as pilot: tab_bar = app.query_one("#tab-bar", TabBar) From 405b84eeb25555d763961257fb735d878e94ebde Mon Sep 17 00:00:00 2001 From: Corey Thompson Date: Mon, 17 Aug 2026 07:05:12 -0500 Subject: [PATCH 24/24] docs(plan): reconcile 0.5.15 slice index --- dev/plans/0.5.15/README.md | 45 +++++++++++++++++++------------------- 1 file changed, 22 insertions(+), 23 deletions(-) diff --git a/dev/plans/0.5.15/README.md b/dev/plans/0.5.15/README.md index 8f62722..ff5852a 100644 --- a/dev/plans/0.5.15/README.md +++ b/dev/plans/0.5.15/README.md @@ -2,12 +2,11 @@ This is the canonical 0.5.15 release workspace index. -**Status:** planning complete. Slices 0–5 are complete planning records and -Slice 6’s HITL decision was recorded on 2026-08-15. Slice 10 is implemented and -awaits its required external PR review/CI closure; Slice 20 is implemented -locally with focused verification, while its full ordinary-suite rerun remains -release-closeout work. Other feature work remains subject to its documented -prerequisites and lifecycle. +**Status:** implementation and non-publishing release closeout complete. Slices +0–6 are complete planning/HITL records. Delivered slices 10, 20, 30, 40, 50, +70, 71, 80, and 90 have independent review and exact-head CI evidence. Slice +60 is deferred to 0.6.0. No release tag or registry publication is part of this +release workspace; a later owner-authorized tag remains a separate action. **Post-planning guard:** do not begin a feature/function slice merely because it is listed or included. First recheck the prior slice closure, update the slice @@ -25,16 +24,16 @@ that is accepted for investigation from work that is approved for delivery. | Candidate | Current evidence | Initial allocation | | --- | --- | --- | -| Gitleaks local and CI secret scanning | Implemented; external enforcement and PR evidence recorded | Slice 10 | -| Typed Fast Guardian threat-backoff HTTP 429 | Implemented locally; status/evidence recorded | Slice 20 | -| Configured-provider credential with no enabled alias warning | Implemented locally; redacted startup-warning evidence recorded | Slice 30 | -| Read-only provider/configuration visibility in Admin/TUI | Design exists; CRUD is not approved | Slice 40 | -| Secure host-console TUI administration for containers | Design exists; topology/identity decision is open | Slice 50 | +| Gitleaks local and CI secret scanning | Implemented and externally enforced | Slice 10 | +| Typed Fast Guardian threat-backoff HTTP 429 | Implemented and exact-head CI verified | Slice 20 | +| Configured-provider credential with no enabled alias warning | Implemented and exact-head CI verified | Slice 30 | +| Read-only provider/configuration visibility in Admin/TUI | Implemented; CRUD remains out of scope | Slice 40 | +| Secure host-console TUI administration for containers | Implemented as default-off host-console profile | Slice 50 | | Airlock-native virtual-key management | **Deferred to 0.6.0**; depends on the native identity/keystore foundation | Slice 60 (0.6.0) | | Multi-instance TUI administration | Implemented as a bounded same-host read-only view | Slice 70 | -| Slice 70 live container topology verification | Admitted verification-only work; requires two disposable TLS container targets | Slice 71 | -| CI review and improvement | Review workflow reliability, signal, coverage, cost, and maintainability without weakening delivery controls | Slice 80 | -| Release closeout | Required only for work actually included and delivered | Slice 90 | +| Slice 70 live container topology verification | Complete; two disposable TLS container targets verified locally and in CI | Slice 71 | +| CI review and improvement | Complete; immutable workflow pins, independent checks, least privilege, and failure diagnostics | Slice 80 | +| Release closeout | Complete; version/readiness and exact-head CI/protection evidence recorded, without publication | Slice 90 | Claude Code subscription-preserving pass-through is explicitly out of scope for 0.5.15. The even-minor roadmap migration is planning/documentation work, not a @@ -212,16 +211,16 @@ authorize work before Slice 6. | Slice | Candidate scope | Initial condition | | --- | --- | --- | -| 10 | Gitleaks local/CI controls, baseline, ownership, and GitHub enforcement | Slice 6 includes it; external GitHub action/branch-rule authority is available. | -| 20 | Typed Fast Guardian threat-backoff exception and HTTP 429 contract | Slice 3 draft contract and Slice 4 architecture pass. | -| 30 | Redacted startup warning for configured credential/no enabled alias | Provider taxonomy, logging, and redaction drafts pass. | -| 40 | Read-only provider/configuration Admin/TUI visibility | Read-only configuration-owner posture remains approved; no CRUD expansion. | -| 50 | Secure host-console TUI administration for containerized Airlock | Topology, operator identity, TLS/token lifecycle, and audit decisions pass. | +| 10 | Gitleaks local/CI controls, baseline, ownership, and GitHub enforcement | Complete; enforced on `main`. | +| 20 | Typed Fast Guardian threat-backoff exception and HTTP 429 contract | Complete; typed/redacted 429 contract verified. | +| 30 | Redacted startup warning for configured credential/no enabled alias | Complete; finite taxonomy and redaction contract verified. | +| 40 | Read-only provider/configuration Admin/TUI visibility | Complete; read-only posture retained, no CRUD expansion. | +| 50 | Secure host-console TUI administration for containerized Airlock | Complete; default-off host-console TLS/capability profile only. | | 60 | **Deferred to 0.6.0** virtual-key management | The 0.6.0 B5 identity/keystore foundation must close first; not a 0.5.15 delivery slice. | -| 70 | Multi-instance TUI named inventory and read-only Admin-API fan-out | Fleet identity/TLS/token/inventory/operation limits are approved; otherwise postpone. | -| 71 | Slice 70 Docker topology verification | Slice 70 is closed; Docker daemon access and a disposable local image are available. Verification-only: no product authority expansion. | -| 80 | CI review and improvement | Review current workflows, required checks, runtime/cost, cache and matrix design, test-signal quality, and failure triage; preserve least privilege, pinned actions, Gitleaks enforcement, and no-secret CI boundaries. Document and independently review proposed changes before implementation. | -| 90 | Release closeout | Only delivered, verified slices are included; deferred work is named explicitly. | +| 70 | Multi-instance TUI named inventory and read-only Admin-API fan-out | Complete; same-host read-only v1 only. | +| 71 | Slice 70 Docker topology verification | Complete; local and CI two-container verification without new product authority. | +| 80 | CI review and improvement | Complete; preserve its least-privilege, pinned-action, and required-check controls in future changes. | +| 90 | Release closeout | Complete and non-publishing; only delivered, verified slices are included and deferrals are explicit. | ### Mandatory lifecycle for every included feature slice