From 346fadafc9717c744145c6faef5979de0aca088b Mon Sep 17 00:00:00 2001 From: Lorenzo S <46894435+Suffro@users.noreply.github.com> Date: Fri, 25 Sep 2026 02:59:10 +0200 Subject: [PATCH] Build the documentation as a VitePress site Turn docs/ into a VitePress site with a home page, getting started, core concepts, testing, production, backends, reference, CI and architecture sections. Existing pages keep their paths and were audited against the implementation; new pages add a quickstart verified against the PyPI package, a CLI reference, a complete contract reference, and separate System One, HTTP and custom-backend pages. The README becomes a short landing page linking into the docs, with absolute GitHub links since it is also the PyPI project page. No hosting or deployment is configured. Co-Authored-By: Claude Opus 5.5 --- .../backend-plugins-via-entry-points.md | 2 +- .../core-dependencies-and-internal-metrics.md | 2 +- .../decisions/systemone-builtin-backend.md | 2 +- .context/state/current.md | 22 +- .context/truth/architecture.md | 7 +- .context/truth/conventions.md | 18 +- CHANGELOG.md | 9 + CONTRIBUTING.md | 3 +- README.md | 195 +- docs/.vitepress/config.mts | 145 + docs/.vitepress/theme/HeroTerminal.vue | 20 + docs/.vitepress/theme/custom.css | 225 ++ docs/.vitepress/theme/index.ts | 14 + docs/architecture.md | 123 +- docs/backends.md | 368 +-- docs/ci.md | 206 +- docs/cli.md | 258 ++ docs/contracts.md | 312 +- docs/custom-backends.md | 169 ++ docs/datasets.md | 94 + docs/decisions.md | 174 ++ docs/fuzzing.md | 155 + docs/gates.md | 98 + docs/http.md | 166 ++ docs/index.md | 89 + docs/installation.md | 92 + docs/introduction.md | 91 + docs/metrics.md | 133 + docs/policy.md | 138 +- docs/production.md | 197 +- docs/properties.md | 339 ++- docs/quickstart.md | 293 ++ docs/regression.md | 139 +- docs/reports.md | 221 +- docs/systemone.md | 257 ++ package-lock.json | 2562 +++++++++++++++++ package.json | 15 + src/decguard/backends/http.py | 2 +- src/decguard/backends/python.py | 2 +- src/decguard/metrics/core.py | 2 +- tests/integration/real/choice.yaml | 2 +- tests/integration/real/noul.yaml | 2 +- tests/integration/real/score.yaml | 2 +- tests/integration/test_real_backends.py | 2 +- 44 files changed, 6532 insertions(+), 835 deletions(-) create mode 100644 docs/.vitepress/config.mts create mode 100644 docs/.vitepress/theme/HeroTerminal.vue create mode 100644 docs/.vitepress/theme/custom.css create mode 100644 docs/.vitepress/theme/index.ts create mode 100644 docs/cli.md create mode 100644 docs/custom-backends.md create mode 100644 docs/datasets.md create mode 100644 docs/decisions.md create mode 100644 docs/fuzzing.md create mode 100644 docs/gates.md create mode 100644 docs/http.md create mode 100644 docs/index.md create mode 100644 docs/installation.md create mode 100644 docs/introduction.md create mode 100644 docs/metrics.md create mode 100644 docs/quickstart.md create mode 100644 docs/systemone.md create mode 100644 package-lock.json create mode 100644 package.json diff --git a/.context/decisions/backend-plugins-via-entry-points.md b/.context/decisions/backend-plugins-via-entry-points.md index 6604106..2cb7db6 100644 --- a/.context/decisions/backend-plugins-via-entry-points.md +++ b/.context/decisions/backend-plugins-via-entry-points.md @@ -17,7 +17,7 @@ are untrusted input, and the plan forbids executing arbitrary code from contract `DecisionResult` semantics. - Python code can also be used directly through `CallableBackend` in the Python API. - The generic HTTP backend speaks a small documented JSON protocol, `decguard.http/0.1` - (docs/backends.md), with `label_map` for differing label names. Credentials come only + (docs/http.md), with `label_map` for differing label names. Credentials come only from environment variables. - No real Open-Jev/Kev adapter ships in Step 1: their APIs were not available to design against, and CI must not need model downloads or GPUs. The opt-in external test diff --git a/.context/decisions/core-dependencies-and-internal-metrics.md b/.context/decisions/core-dependencies-and-internal-metrics.md index 01413b9..124290c 100644 --- a/.context/decisions/core-dependencies-and-internal-metrics.md +++ b/.context/decisions/core-dependencies-and-internal-metrics.md @@ -37,5 +37,5 @@ Build backend: `hatchling`. Tooling: `uv` (lockfile `uv.lock` committed), `ruff` ## Consequences - `pip install decguard` stays light and pure-Python. -- Metric definitions are ours to keep correct; `docs/reports.md` documents every formula +- Metric definitions are ours to keep correct; `docs/metrics.md` documents every formula and the tests pin them. Any change to a metric definition is a report-format change. diff --git a/.context/decisions/systemone-builtin-backend.md b/.context/decisions/systemone-builtin-backend.md index c3f04e4..718c052 100644 --- a/.context/decisions/systemone-builtin-backend.md +++ b/.context/decisions/systemone-builtin-backend.md @@ -64,4 +64,4 @@ github.com/jaredpalmer/kev. and downloads about 1.8 GB of weights, which are cached by pinned revision. That is why the workflow is manual. - Changing the pinned Kev commit, checkpoint or Jev model means updating the workflow env, - `docs/backends.md` and the real contracts together. + `docs/systemone.md` and the real contracts together. diff --git a/.context/state/current.md b/.context/state/current.md index d638e38..0f34a28 100644 --- a/.context/state/current.md +++ b/.context/state/current.md @@ -2,10 +2,14 @@ ## Current focus -`0.1.0` is released: tag `v0.1.0` on `ca2373a`, published to PyPI by `release.yml` -(Trusted Publishing, attestations) with the same wheel and sdist on the GitHub Release. -The repository is public. `0.1.1` (on `main`, not yet tagged) only fixes the README, which -is the PyPI project page: install with `pip install decguard`, absolute GitHub links. +Documentation site (branch `docs/vitepress`): `docs/` is now a VitePress site with +getting started, concepts, testing, production, backends, reference, CI and architecture +sections; the README is a short landing page (absolute GitHub links, since it is the PyPI +project page). Hosting/deployment of the site is deliberately not configured yet. + +Released: `0.1.0` (tag `v0.1.0` on `ca2373a`) and `0.1.1` (tag `v0.1.1` on `20b6b02`, +README fix only), both published to PyPI by `release.yml` (Trusted Publishing, +attestations) with the same files on the GitHub Releases. The repository is public. ## Recent relevant changes @@ -38,6 +42,11 @@ is the PyPI project page: install with `pip install decguard`, absolute GitHub l metadata contains only compatible permissive licenses plus Certifi's MPL-2.0; `uv.lock` is current and every registry artifact is hash-pinned. No runtime dependency was added for the real backends. +- Documentation: VitePress site in `docs/` (`npm run docs:build` fails on dead links); + existing page paths kept, backends split into `docs/systemone.md`, `docs/http.md` and + `docs/custom-backends.md`, metrics moved to `docs/metrics.md`, the fuzzing workflow to + `docs/fuzzing.md`. + Quickstart output was produced by the PyPI package. - Release workflow: pushing a `vX.Y.Z` tag validates the tag against `pyproject.toml` and CHANGELOG, builds and checks the distributions once, publishes them to PyPI with Trusted Publishing (environment `pypi`, no stored token), then creates the GitHub Release with @@ -45,8 +54,9 @@ is the PyPI project page: install with `pip install decguard`, absolute GitHub l ## Next -- Release `0.1.1` (push `v0.1.1` on the `main` commit with the README fix) so PyPI shows the - corrected page; a released version's PyPI description cannot be edited. +- Merge the documentation PR, then choose and configure hosting for the VitePress site + (separate task; the site has no `base` or deployment workflow). Once a public docs URL + exists, point `project.urls.Documentation` in `pyproject.toml` and the README at it. ## Blockers diff --git a/.context/truth/architecture.md b/.context/truth/architecture.md index 7ace1dd..66c5337 100644 --- a/.context/truth/architecture.md +++ b/.context/truth/architecture.md @@ -50,6 +50,9 @@ explicit policies and SDK) are implemented. records, summaries, property checks, `rejudge`, load-time `verify_run`), `render.py` (terminal). - `engine.py`: `run_test(mode="test"|"fuzz"|"all")` orchestration shared by CLI and API. +- `docs/`: the user documentation as a VitePress site (`docs/.vitepress/config.mts`, a + small theme extension in `docs/.vitepress/theme/`); Node tooling is dev-only + (`package.json`, `package-lock.json` at the root). Not part of the Python package. - `cli/`: typer app with `validate`, `test` (`--all`), `fuzz`, `report`, `diff`, `replay`, offline `check`, and policy `run`. @@ -69,8 +72,8 @@ policy → action directive (fallback is not invoked automatically). ## External systems -- Decision backends over HTTP (`decguard.http/0.1` or System One, docs/backends.md) or - installed plugins. None is needed for the default test suite; a local stdlib HTTP server +- Decision backends over HTTP (`decguard.http/0.1`, docs/http.md; System One, + docs/systemone.md) or installed plugins. None is needed for the default test suite; a local stdlib HTTP server backs the integration tests. - Real backends, opt-in only: a self-hosted Kev server (upstream `jaredpalmer/kev`, Kev-0.8B at a pinned revision) and Jev via OpenRouter (`typesafe/jev-1.13`, `OPENROUTER_API_KEY`), diff --git a/.context/truth/conventions.md b/.context/truth/conventions.md index 1f683ec..4e3b1db 100644 --- a/.context/truth/conventions.md +++ b/.context/truth/conventions.md @@ -15,7 +15,13 @@ test` and `decguard test --all` (`test_examples_pass`, `test_examples_pass_all_checks`). - Deterministic outputs that users may store (mock hash distribution, fuzz RNG stream) are pinned by tests; changing them is a CHANGELOG-worthy format change. -- Docs for users in `docs/`; user-visible changes go in `CHANGELOG.md` under Unreleased. +- Docs for users in `docs/`, a VitePress site: every page is listed in the sidebar in + `docs/.vitepress/config.mts`; links between pages are relative `.md` links so they work + on GitHub too. The implementation is authoritative: command output shown in the docs is + copied from real runs. User-visible changes go in `CHANGELOG.md` under Unreleased. +- Hosting/deployment of the docs is not configured (no `base`, no workflow). +- `README.md` is also the PyPI project page: link with absolute GitHub URLs, not + repository-relative paths. ## Development workflow @@ -26,10 +32,18 @@ uv run mypy uv run pytest ``` +Documentation (Node.js 18+, dev-only): + +```bash +npm ci +npm run docs:dev # local preview +npm run docs:build # production build; fails on dead links between pages +``` + CI (`.github/workflows/ci.yml`) runs these on Ubuntu and macOS for Python 3.11–3.13 and installs the built wheel in a clean environment. -Real-backend checks are opt-in and never part of the default run (docs/backends.md): +Real-backend checks are opt-in and never part of the default run (docs/systemone.md): `uv run pytest -m real_kev` (a Kev server on 127.0.0.1:8009) and `uv run pytest -m real_jev` (`OPENROUTER_API_KEY`). Once selected, a missing prerequisite is a failure, not a skip. In CI they run only in `.github/workflows/real-backends.yml`. diff --git a/CHANGELOG.md b/CHANGELOG.md index bc1ded2..9e40c35 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,15 @@ versioned separately. ## [Unreleased] +### Added + +- Documentation site built with VitePress in `docs/` (`npm run docs:dev`, + `npm run docs:build`): getting started, core concepts, testing, production, backends, + CLI/contract/report reference, CI and architecture. New pages include a verified + quickstart, a CLI reference and a complete contract reference; the backends guide is + split into System One, HTTP and custom-backend pages. The README is now a short landing + page that links into these docs. + ## [0.1.1] - 2026-09-25 ### Fixed diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 24e8103..990984c 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -87,6 +87,7 @@ does not run the paid real-backend tests. ## Adding a backend -See [docs/backends.md](docs/backends.md). Built-in adapters live in +See [docs/custom-backends.md](docs/custom-backends.md) and +[docs/backends.md](docs/backends.md). Built-in adapters live in `src/decguard/backends/`; third-party ones should be separate packages registered under the `decguard.backends` entry-point group. diff --git a/README.md b/README.md index e5c3638..9e357cf 100644 --- a/README.md +++ b/README.md @@ -1,33 +1,33 @@ # DecGuard -Test, verify and gate **probabilistic AI decision models** — models that answer a typed -question (pick an option, say yes/no, assign a level) with a probability for each answer. +Reliability testing for **probabilistic AI decisions**: models that answer a typed +question — pick an option, say yes or no, assign a level — with a probability for each +answer. -DecGuard sits above your decision backends. You describe the decision once in a small -**contract**, run a dataset through any backend, and get one reproducible **reliability -report** with CI-friendly PASS / WARN / FAIL gates. +You describe the decision once in a small **Decision Contract**, run a dataset through +any backend, and get one reproducible **reliability report** with CI-friendly +PASS / WARN / FAIL gates. DecGuard measures accuracy and calibration, fuzzes behavioral +invariants, compares model versions, and checks production records offline. -> Status: alpha (v0.1). Contracts, backends, golden-dataset testing, metrics, reports, metamorphic fuzzing, -> regression/replay, offline post-deployment checks, explicit cascade policies and the -> Python SDK are implemented. The `systemone` backend is validated end to end against a -> real Kev-0.8B server and real Jev through OpenRouter, on Linux and macOS. +> Status: alpha (v0.1). The contract format and reports are versioned; the `systemone` +> backend is validated end to end against a real Kev-0.8B server and real Jev through +> OpenRouter, on Linux and macOS. ## Install ```bash -pip install decguard # or: uv tool install decguard +pip install decguard # or: uv tool install decguard ``` -Python 3.11+. +Python 3.11+. Four runtime dependencies (pydantic, typer, httpx, PyYAML); no GPU, +server or account. -## 5-minute quickstart - -The quickstart uses the example contract and datasets from the repository: +## Quickstart ```bash git clone --depth 1 https://github.com/suffro/decguard && cd decguard decguard validate examples/refund/decguard.yaml -decguard test examples/refund/decguard.yaml --all --output stable.json +decguard test examples/refund/decguard.yaml --all ``` ```text @@ -41,30 +41,33 @@ DecGuard 0.1.1 · refund_request (choice) · PASS Checks PASS min_accuracy accuracy 1 >= 0.9 PASS max_ece ece 0.139 <= 0.15 + PASS option_order.max_violation_rate option_order.violation_rate 0 <= 0 (per case: max_tv_distance 0.03) ... PASS: all gates hold ``` -A contract looks like this ([full reference](https://github.com/suffro/decguard/blob/main/docs/contracts.md)): +A contract ([reference](https://github.com/suffro/decguard/blob/main/docs/contracts.md)): ```yaml schema_version: "0.1" decision: name: refund_request + description: How should this customer refund request be handled? type: choice # choice | noul | score options: [refund, reject, review] backend: - provider: http # or: systemone (Kev, Jev), mock, an installed plugin - url: https://decisions.internal/v1/decide - model: open-jev-2b - bearer_token_env: DECISIONS_TOKEN + provider: systemone # or: mock, http, an installed plugin + model: typesafe/jev-1.13 + url: https://openrouter.ai/api/alpha/decisions + bearer_token_env: OPENROUTER_API_KEY dataset: cases.jsonl # {"input": ..., "expected": ...?, "metadata": ...?} per line evaluation: confidence_threshold: 0.8 # below this, a decision counts as an abstention + probability_tolerance: 0.02 # Jev rounds probabilities to two decimals requirements: # hard gates -> exit code 1 min_accuracy: 0.95 @@ -72,125 +75,67 @@ requirements: # hard gates -> exit code 1 warnings: # soft gates -> WARN max_latency_p95_ms: 300 -``` - -### Find what golden tests miss - -The refund example also defines `order_sensitive`, a backend that quietly favours whichever -option is listed first. Its ordinary accuracy still looks perfect, but calibration and the -metamorphic properties catch it: -```bash -decguard test examples/refund/decguard.yaml -b order_sensitive # calibration FAIL -decguard fuzz examples/refund/decguard.yaml -b order_sensitive -o fuzz.json -``` - -```text - Properties seed 42 - option_order 33 compared · 27 violating (81.8%) · flips 0 (0.0%) · max TV 0.100 - label_format 22 compared · 0 violating (0.0%) · flips 0 (0.0%) · max TV 0.000 - ... - FAIL option_order.max_violation_rate option_order.violation_rate 0.8182 violates <= 0 (per case: ...) - Property failures (10 of 27; replay with `decguard replay`) - option_order/r1/0 - tv_distance 0.1 > 0.03 - sent: options [reject, refund, review] · The blender arrived damaged and won't turn on. -FAIL: at least one requirement does not hold +properties: # metamorphic checks (decguard fuzz / test --all) + option_order: + max_tv_distance: 0.03 ``` -The broken run exits `1`, records the distribution drift and stores minimal failing -examples. Replay it, then show that the same backend also breaks calibration and the -regression limit: - -```bash -decguard replay fuzz.json # reproduces; exit 1 -decguard test examples/refund/decguard.yaml -b order_sensitive -o candidate.json -decguard diff stable.json candidate.json -c examples/refund/decguard.yaml -``` +The [quickstart guide](https://github.com/suffro/decguard/blob/main/docs/quickstart.md) walks through writing your first contract +and dataset. -### Check production records and route runtime decisions +## Capabilities -Analyze records your application collected—fully offline—and gate aggregate, drift and -metadata-segment reliability: +| Command | What it does | +| --- | --- | +| `decguard validate` | Check the contract, backend settings and dataset offline. | +| `decguard test` | Golden-dataset metrics (accuracy, calibration, coverage, latency, errors) and gates; `--all` adds properties. | +| `decguard fuzz` | Metamorphic properties: option order, label format, irrelevant/repeated context, whitespace, paraphrase, noul inversion, score monotonicity. Seeded and minimized. | +| `decguard replay` | Re-send stored property failures; exit 1 if they still fail. | +| `decguard diff` | Compare a baseline and a candidate run: flips, distribution and confidence shifts, calibration, errors, segments. | +| `decguard report` | Show a stored report; `--contract` re-applies edited gates without re-running the model. | +| `decguard check` | Analyze collected production records offline: calibration, drift, metadata segments. | +| `decguard run` | Make one decision and apply the contract's policy: `accept`, `abstain`, `fallback` or `human_review`. | + +Exit codes: **0** pass (or warn), **1** a reliability gate failed, **2** configuration or +runtime error. + +Backends: an offline `mock`, a generic `http` protocol, `systemone` for **Kev** +(self-hosted), **Jev through OpenRouter** and TypeSafe endpoints — validated with real Kev +and Jev inference on Linux and macOS — plus Python plugins. -```bash -decguard check examples/refund/decguard.yaml \ - --dataset examples/refund/production.jsonl \ - --output production-report.json -``` +## Documentation -Compare an intentionally overconfident, incorrect batch to see calibration drift and -localized segment failures (exit `1`): +The full documentation is a [VitePress](https://vitepress.dev) site in [`docs/`](https://github.com/suffro/decguard/tree/main/docs): -```bash -decguard check examples/refund/decguard.yaml \ - --dataset examples/refund/production-drift.jsonl \ - --baseline examples/refund/production.jsonl -``` +- Getting started: [What is DecGuard?](https://github.com/suffro/decguard/blob/main/docs/introduction.md) · + [Installation](https://github.com/suffro/decguard/blob/main/docs/installation.md) · [Quickstart](https://github.com/suffro/decguard/blob/main/docs/quickstart.md) +- Concepts: [Decisions and results](https://github.com/suffro/decguard/blob/main/docs/decisions.md) · + [Golden datasets](https://github.com/suffro/decguard/blob/main/docs/datasets.md) · [Requirements and warnings](https://github.com/suffro/decguard/blob/main/docs/gates.md) +- Testing: [Metrics](https://github.com/suffro/decguard/blob/main/docs/metrics.md) · [Metamorphic properties](https://github.com/suffro/decguard/blob/main/docs/properties.md) · + [Fuzzing and replay](https://github.com/suffro/decguard/blob/main/docs/fuzzing.md) · [Regression diffs](https://github.com/suffro/decguard/blob/main/docs/regression.md) +- Production: [Production checks](https://github.com/suffro/decguard/blob/main/docs/production.md) · + [Policies and Python SDK](https://github.com/suffro/decguard/blob/main/docs/policy.md) +- Backends: [Overview](https://github.com/suffro/decguard/blob/main/docs/backends.md) · [System One: Kev and Jev](https://github.com/suffro/decguard/blob/main/docs/systemone.md) · + [HTTP](https://github.com/suffro/decguard/blob/main/docs/http.md) · [Custom backends](https://github.com/suffro/decguard/blob/main/docs/custom-backends.md) +- Reference: [CLI](https://github.com/suffro/decguard/blob/main/docs/cli.md) · [Decision Contract](https://github.com/suffro/decguard/blob/main/docs/contracts.md) · + [Reports and formats](https://github.com/suffro/decguard/blob/main/docs/reports.md) +- [CI with GitHub Actions](https://github.com/suffro/decguard/blob/main/docs/ci.md) · [Architecture](https://github.com/suffro/decguard/blob/main/docs/architecture.md) -The same contract can deterministically route one normalized decision: +To browse it locally (Node.js 18+): ```bash -decguard run examples/refund/decguard.yaml "The blender arrived damaged" -decguard run examples/refund/decguard.yaml "The parcel never arrived" -decguard run examples/refund/decguard.yaml "Can someone call me?" -``` - -These three calls deterministically return `accept`, a `fallback -> strong_model` -directive, and `human_review`. The fallback is not invoked automatically. - -See [post-deployment checks](https://github.com/suffro/decguard/blob/main/docs/production.md) and the -[policy/SDK guide](https://github.com/suffro/decguard/blob/main/docs/policy.md). - -## Commands - -| Command | What it does | -| --- | --- | -| `decguard validate ` | Check the contract, backend settings and dataset offline. | -| `decguard test ` | Run the dataset through a backend (`--backend NAME` picks a named one), print the report, `--output report.json` to keep it. | -| `decguard fuzz ` | Check the contract's metamorphic properties (option order, label format, irrelevant context, whitespace, paraphrase, noul inversion, score monotonicity); deterministic per `--seed`. | -| `decguard test --all` | Golden gates and properties in one report. | -| `decguard replay ` | Re-send stored property failures (`--id` for one); exit 1 if they still fail. | -| `decguard diff ` | Answer flips, confidence/distribution shifts, calibration, latency, error and per-segment changes; `--contract` applies `regression` gates. | -| `decguard report ` | Show a stored report; `--contract` re-applies (possibly edited) gates without re-running the model. | -| `decguard check --dataset ` | Analyze production JSONL/JSON offline, optionally against `--baseline`, with aggregate and segment gates. | -| `decguard run ` | Make one decision and apply the contract's ordered policy. | - -Exit codes: **0** pass (or warn; `--fail-on-warn` turns warn into 1), **1** a reliability -gate failed, **2** configuration or runtime error. Use `--format json` for machine-readable -output on stdout. - -### In CI - -```yaml -- run: uv tool install decguard -- run: decguard test decguard.yaml --all --output decguard-report.json -- uses: actions/upload-artifact@v4 - if: always() - with: { name: decguard-report, path: decguard-report.json } +npm ci +npm run docs:dev ``` -A full workflow with a regression diff is in [docs/ci.md](https://github.com/suffro/decguard/blob/main/docs/ci.md). - -## Documentation - -- [Decision Contract reference](https://github.com/suffro/decguard/blob/main/docs/contracts.md) -- [Backends: System One (Kev, Jev), HTTP, plugins](https://github.com/suffro/decguard/blob/main/docs/backends.md) -- [Reports, metrics and gates](https://github.com/suffro/decguard/blob/main/docs/reports.md) -- [Metamorphic properties, fuzzing and replay](https://github.com/suffro/decguard/blob/main/docs/properties.md) -- [Regression diffs](https://github.com/suffro/decguard/blob/main/docs/regression.md) -- [Post-deployment checks and record format](https://github.com/suffro/decguard/blob/main/docs/production.md) -- [Runtime cascade policy and Python SDK](https://github.com/suffro/decguard/blob/main/docs/policy.md) -- [Architecture and integration philosophy](https://github.com/suffro/decguard/blob/main/docs/architecture.md) -- [CI with GitHub Actions](https://github.com/suffro/decguard/blob/main/docs/ci.md) -- [Contributing](https://github.com/suffro/decguard/blob/main/CONTRIBUTING.md) · [Changelog](https://github.com/suffro/decguard/blob/main/CHANGELOG.md) - ## What DecGuard is not -Not a model, not a generic LLM eval framework, not a hosted service or dashboard. It owns -the decision-specific reliability layer (contract, normalized results, metrics, gates, -report) and stays backend-neutral; other tools plug in as optional integrations. +Not a model, a model server, a generic LLM evaluation framework or a hosted service. It +owns the decision-specific reliability layer — contract, normalized results, metrics, +gates, reports — and stays backend-neutral. -## License +## Contributing and license -Apache-2.0 +See [CONTRIBUTING.md](https://github.com/suffro/decguard/blob/main/CONTRIBUTING.md) and the [changelog](https://github.com/suffro/decguard/blob/main/CHANGELOG.md). +Licensed under Apache-2.0. diff --git a/docs/.vitepress/config.mts b/docs/.vitepress/config.mts new file mode 100644 index 0000000..d8fad10 --- /dev/null +++ b/docs/.vitepress/config.mts @@ -0,0 +1,145 @@ +import { readFileSync } from 'node:fs' +import { defineConfig } from 'vitepress' + +// The nav shows the package version from pyproject.toml, so it follows each release. +const pyproject = readFileSync(new URL('../../pyproject.toml', import.meta.url), 'utf8') +const version = pyproject.match(/^version = "([^"]+)"/m)?.[1] ?? 'unknown' + +const repository = 'https://github.com/suffro/decguard' +const description = + 'Reliability testing for probabilistic AI decisions: decision contracts, calibration, ' + + 'metamorphic fuzzing, regression gates and offline production checks.' + +export default defineConfig({ + lang: 'en-US', + title: 'DecGuard', + titleTemplate: ':title · DecGuard', + description, + + // Hosting-neutral: no `base`, no sitemap hostname and no clean-URL rewrites. + lastUpdated: true, + metaChunk: true, + + head: [ + ['meta', { name: 'theme-color', content: '#0e7c66' }], + ['meta', { property: 'og:type', content: 'website' }], + ['meta', { property: 'og:site_name', content: 'DecGuard' }], + ['meta', { property: 'og:title', content: 'DecGuard' }], + ['meta', { property: 'og:description', content: description }], + ['meta', { name: 'twitter:card', content: 'summary' }], + ['meta', { name: 'twitter:title', content: 'DecGuard' }], + ['meta', { name: 'twitter:description', content: description }], + ], + + markdown: { + theme: { light: 'github-light', dark: 'github-dark' }, + }, + + themeConfig: { + siteTitle: 'DecGuard', + + nav: [ + { text: 'Guide', link: '/quickstart', activeMatch: '^/(introduction|installation|quickstart|decisions|datasets|gates|metrics|properties|fuzzing|regression|production|policy|ci|architecture)' }, + { text: 'Backends', link: '/backends', activeMatch: '^/(backends|systemone|http|custom-backends)' }, + { + text: 'Reference', + activeMatch: '^/(cli|contracts|reports)', + items: [ + { text: 'CLI', link: '/cli' }, + { text: 'Decision Contract', link: '/contracts' }, + { text: 'Reports and formats', link: '/reports' }, + ], + }, + { + text: `v${version}`, + items: [ + { text: 'Changelog', link: `${repository}/blob/main/CHANGELOG.md` }, + { text: 'PyPI', link: 'https://pypi.org/project/decguard/' }, + { text: 'Contributing', link: `${repository}/blob/main/CONTRIBUTING.md` }, + ], + }, + ], + + sidebar: [ + { + text: 'Getting Started', + items: [ + { text: 'What is DecGuard?', link: '/introduction' }, + { text: 'Installation', link: '/installation' }, + { text: 'Quickstart', link: '/quickstart' }, + ], + }, + { + text: 'Core Concepts', + items: [ + { text: 'Decisions and results', link: '/decisions' }, + { text: 'Golden datasets', link: '/datasets' }, + { text: 'Requirements and warnings', link: '/gates' }, + ], + }, + { + text: 'Testing & Reliability', + items: [ + { text: 'Golden tests and metrics', link: '/metrics' }, + { text: 'Metamorphic properties', link: '/properties' }, + { text: 'Fuzzing and replay', link: '/fuzzing' }, + { text: 'Regression diffs', link: '/regression' }, + ], + }, + { + text: 'Production', + items: [ + { text: 'Production checks', link: '/production' }, + { text: 'Policies and Python SDK', link: '/policy' }, + ], + }, + { + text: 'Backends', + items: [ + { text: 'Overview and mock', link: '/backends' }, + { text: 'System One: Kev and Jev', link: '/systemone' }, + { text: 'Generic HTTP', link: '/http' }, + { text: 'Custom backends', link: '/custom-backends' }, + ], + }, + { + text: 'Reference', + items: [ + { text: 'CLI', link: '/cli' }, + { text: 'Decision Contract', link: '/contracts' }, + { text: 'Reports and formats', link: '/reports' }, + ], + }, + { + text: 'CI and Architecture', + items: [ + { text: 'CI with GitHub Actions', link: '/ci' }, + { text: 'Architecture', link: '/architecture' }, + ], + }, + ], + + outline: { level: [2, 3], label: 'On this page' }, + + search: { provider: 'local' }, + + socialLinks: [{ icon: 'github', link: repository }], + + editLink: { + pattern: `${repository}/edit/main/docs/:path`, + text: 'Edit this page on GitHub', + }, + + lastUpdated: { + text: 'Last updated', + formatOptions: { dateStyle: 'medium' }, + }, + + docFooter: { prev: 'Previous', next: 'Next' }, + + footer: { + message: 'Released under the Apache-2.0 License.', + copyright: 'DecGuard contributors', + }, + }, +}) diff --git a/docs/.vitepress/theme/HeroTerminal.vue b/docs/.vitepress/theme/HeroTerminal.vue new file mode 100644 index 0000000..a3ccaac --- /dev/null +++ b/docs/.vitepress/theme/HeroTerminal.vue @@ -0,0 +1,20 @@ + + diff --git a/docs/.vitepress/theme/custom.css b/docs/.vitepress/theme/custom.css new file mode 100644 index 0000000..e04ce9b --- /dev/null +++ b/docs/.vitepress/theme/custom.css @@ -0,0 +1,225 @@ +/* DecGuard: a restrained layer over the VitePress default theme. */ + +:root { + --vp-c-brand-1: #0e7c66; + --vp-c-brand-2: #0b6a57; + --vp-c-brand-3: #09594a; + --vp-c-brand-soft: rgba(14, 124, 102, 0.12); + + --dg-pass: #1a7f37; + --dg-warn: #9a6700; + --dg-fail: #cf222e; + + --vp-home-hero-name-color: var(--vp-c-brand-1); + --vp-button-brand-bg: var(--vp-c-brand-1); + --vp-button-brand-hover-bg: var(--vp-c-brand-2); + --vp-button-brand-active-bg: var(--vp-c-brand-3); + + --dg-terminal-bg: #0f1419; + --dg-terminal-fg: #d6dde4; + --dg-terminal-dim: #8b98a5; + --dg-terminal-pass: #56d364; +} + +.dark { + --vp-c-brand-1: #3cc9a7; + --vp-c-brand-2: #2fb193; + --vp-c-brand-3: #24977d; + --vp-c-brand-soft: rgba(60, 201, 167, 0.14); + + --dg-pass: #56d364; + --dg-warn: #e3b341; + --dg-fail: #f85149; + + --vp-button-brand-text: #0b1512; + --vp-button-brand-hover-text: #0b1512; + --vp-button-brand-active-text: #0b1512; + + --dg-terminal-bg: #0b0f14; +} + +/* Typography */ + +.vp-doc h1 { + letter-spacing: -0.02em; +} + +.vp-doc h2 { + letter-spacing: -0.01em; +} + +.vp-doc table { + font-size: 14px; +} + +.vp-doc th, +.vp-doc td { + padding: 7px 12px; +} + +.vp-doc td code, +.vp-doc th code { + white-space: nowrap; +} + +/* Home: hero */ + +.VPHero .name { + font-weight: 750; + letter-spacing: -0.03em; +} + +.VPHero .text { + font-weight: 650; + letter-spacing: -0.02em; +} + +/* The terminal replaces the hero image: size it to its content, not a square logo box. */ +.VPHero.has-image .image { + order: 3; + min-width: 0; + margin: 40px 0 0; +} + +.VPHero.has-image .image-container { + width: auto; + height: auto; + transform: none; +} + +.VPHero.has-image .image-bg { + display: none; +} + +@media (min-width: 960px) { + .VPHero.has-image .image { + order: 2; + margin: 0; + display: flex; + align-items: center; + justify-content: flex-end; + } +} + +.dg-terminal { + margin: 0 auto; + width: 100%; + max-width: min(400px, 100%); + min-width: 0; + border: 1px solid var(--vp-c-divider); + border-radius: 10px; + background: var(--dg-terminal-bg); + box-shadow: 0 8px 28px rgba(0, 0, 0, 0.12); + overflow: hidden; + text-align: left; +} + +.dg-terminal-bar { + display: flex; + gap: 6px; + padding: 10px 14px; + border-bottom: 1px solid rgba(255, 255, 255, 0.06); +} + +.dg-terminal-bar span { + width: 10px; + height: 10px; + border-radius: 50%; + background: rgba(255, 255, 255, 0.16); +} + +.dg-terminal pre { + margin: 0; + padding: 14px 18px 16px; + overflow-x: auto; + font-family: var(--vp-font-family-mono); + font-size: 13px; + line-height: 1.6; + color: var(--dg-terminal-fg); +} + +.dg-terminal code { + font-family: inherit; +} + +@media (max-width: 480px) { + .dg-terminal pre { + padding: 12px 14px 14px; + font-size: 12px; + } +} + +@media (min-width: 960px) { + .VPHero .text { + font-size: 48px; + line-height: 56px; + } +} + +.dg-terminal .dg-prompt, +.dg-terminal .dg-dim { + color: var(--dg-terminal-dim); +} + +.dg-terminal .dg-pass { + color: var(--dg-terminal-pass); + font-weight: 600; +} + +.dg-terminal figcaption { + padding: 0 18px 12px; + font-size: 12px; + color: var(--dg-terminal-dim); +} + +/* Home: feature grid */ + +.VPFeatures .item .VPFeature { + border: 1px solid var(--vp-c-divider); + background-color: var(--vp-c-bg); + transition: border-color 0.2s; +} + +.VPFeatures .item .VPFeature.link:hover { + border-color: var(--vp-c-brand-1); +} + +.VPFeatures .VPFeature .box { + padding: 20px 22px; +} + +.VPFeatures .VPFeature .title { + font-size: 15px; +} + +.VPFeatures .VPFeature .details { + font-size: 14px; + line-height: 22px; +} + +/* Inline status markers, e.g. PASS */ + +.dg-status { + display: inline-block; + padding: 0 7px; + border-radius: 4px; + font-family: var(--vp-font-family-mono); + font-size: 0.85em; + font-weight: 600; + line-height: 1.6; +} + +.dg-status.pass { + color: var(--dg-pass); + background: color-mix(in srgb, var(--dg-pass) 12%, transparent); +} + +.dg-status.warn { + color: var(--dg-warn); + background: color-mix(in srgb, var(--dg-warn) 14%, transparent); +} + +.dg-status.fail { + color: var(--dg-fail); + background: color-mix(in srgb, var(--dg-fail) 12%, transparent); +} diff --git a/docs/.vitepress/theme/index.ts b/docs/.vitepress/theme/index.ts new file mode 100644 index 0000000..e53ee60 --- /dev/null +++ b/docs/.vitepress/theme/index.ts @@ -0,0 +1,14 @@ +import { h } from 'vue' +import type { Theme } from 'vitepress' +import DefaultTheme from 'vitepress/theme' +import HeroTerminal from './HeroTerminal.vue' +import './custom.css' + +export default { + extends: DefaultTheme, + Layout() { + return h(DefaultTheme.Layout, null, { + 'home-hero-image': () => h(HeroTerminal), + }) + }, +} satisfies Theme diff --git a/docs/architecture.md b/docs/architecture.md index ec292b0..8652c88 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -1,28 +1,107 @@ -# Architecture and integration philosophy +--- +description: How DecGuard is structured, how data flows through it, the design principles it keeps, and what it deliberately does not do. +--- -DecGuard is a local Python library and CLI around a small provider-neutral core: +# Architecture + +DecGuard is a local Python library and CLI around a small, provider-neutral core. There +is no server, database, telemetry collector or hosted account: every command reads files, +optionally calls a backend, and writes a JSON document. ```text Decision Contract - | - +--> backend adapter --> normalized DecisionResult --> golden/property report - | -production JSONL -----------------------------------------> production report - | -DecisionResult + explicit policy ------------------------> route directive + │ + ├──> backend adapter ──> normalized DecisionResult ──> golden / property report + │ │ + │ two reports ──> regression diff + │ + ├──> production JSONL ─────────────────────────────> production report + │ + └──> DecisionResult + explicit policy ─────────────> route directive ``` -- Contracts own typed decision semantics, reliability gates, metamorphic properties, - post-deployment checks and deterministic policies. -- Thin adapters normalize local functions, HTTP/System-One-compatible endpoints and - third-party `decguard.backends` entry points into `DecisionResult`. -- Golden testing, fuzzing, diff/replay and production checks emit versioned JSON plus a - concise terminal view with consistent PASS/WARN/FAIL exit semantics. -- The CLI and Python SDK call the same engines. No server, database, telemetry collector, - GPU, paid model or hosted account is required. - -DecGuard deliberately does not recreate model servers, generic red-team frameworks, -training systems or observability platforms. External projects can plug in at the backend -or paraphrase-provider boundaries; DecGuard retains the decision-specific contract, -normalization, comparison, gating and reproducibility layer. The default core remains -usable when no optional integration is installed. +- **Contracts** own the typed decision semantics, reliability gates, metamorphic + properties, post-deployment checks and deterministic policies. +- **Adapters** normalize local functions, HTTP and System One endpoints, and third-party + `decguard.backends` entry points into one `DecisionResult`. +- **Engines** — golden testing, fuzzing, diff, replay, production checks and policy + evaluation — are shared by the CLI and the Python API, and emit versioned JSON plus a + concise terminal view with consistent PASS / WARN / FAIL exit semantics. + +## Data flow + +**Golden test.** contract file → load and validate → decision spec + backend; dataset +file → validated cases → run with bounded concurrency, dataset order preserved → results +or per-case errors → metrics → gates → report → JSON file, terminal text, exit code. + +**Fuzz.** After the golden run, each enabled property generates transformations per case +from the seed → options presented as shown, sent to the backend, mapped back → compared +with the original → failing transformations minimized → recorded in the same report. + +**Diff.** Two stored reports → cases matched by id → shifts, metric deltas, segments → +regression gates → diff report. + +**Production.** Collected records → strict normalization → aggregate and segment metrics +→ optional baseline drift → gates → production report. No backend is called. + +**Runtime.** Backend result → ordered policy → action directive. A fallback is returned, +not invoked. + +## Components + +| Package | Responsibility | +| --- | --- | +| `decguard.contracts` | schema `0.1` models, safe loading with duplicate-key rejection, contract hash | +| `decguard.decisions` | decision spec, request, raw `Prediction`, normalized `DecisionResult`, probability validation | +| `decguard.datasets` | golden cases from JSONL/JSON, validated against the contract labels | +| `decguard.backends` | `DecisionBackend` interface; `mock`, `http`, `systemone`; `CallableBackend`; entry-point registry | +| `decguard.runner` | bounded thread pool; per-case errors | +| `decguard.metrics` | pure-stdlib metrics and distribution comparisons | +| `decguard.fuzz` | seeded RNG, transformations, paraphrase providers, comparison, minimization, replay | +| `decguard.regression` | report diffs and regression gates | +| `decguard.production` | production records, metrics, drift, segments, production reports | +| `decguard.policy` | pure first-match route evaluation | +| `decguard.reports` | gates, the report model with load-time verification, terminal rendering | +| `decguard.engine`, `decguard.sdk`, `decguard.cli` | orchestration, the `DecGuard` SDK, the command line | + +## Design principles + +- **Never repair, never drop.** Malformed backend output, invalid dataset rows and invalid + production records are errors. Probabilities are validated, not rescaled. +- **Deterministic and reproducible.** Metrics use exact summation in dataset order; fuzz + transformations derive from a SHA-256 stream keyed by seed, property and case id; the + mock backend's output is pinned by tests. +- **Reports are verifiable.** Every stored report is re-checked on load: results, counts, + metrics, property comparisons, checks and status must all agree. +- **Exit codes are API.** `0` pass/warn, `1` gate failed, `2` could not evaluate. + Unexpected exceptions map to `2`, never `1`. +- **Contracts are data.** Safe YAML, strict schemas, no import paths, no secrets; plugins + are installed packages, never code referenced by a contract. +- **Credentials only from the environment.** They never appear in reports, errors or + logs; authenticated healthchecks stay on the backend's origin. +- **Small dependency footprint.** Four runtime dependencies: pydantic, typer, httpx and + PyYAML. Metrics are implemented in the standard library. + +## Integration philosophy + +DecGuard deliberately does not recreate model servers, generic LLM evaluation or +red-teaming frameworks, training systems or observability platforms. External projects +plug in at two boundaries: + +- **backends** — anything that returns label probabilities, through `http`, `systemone` + or a `decguard.backends` plugin; +- **paraphrase providers** — `file`, OpenAI-compatible APIs, or a `decguard.paraphrasers` + plugin. + +DecGuard keeps the decision-specific layer: the contract, normalization, comparisons, +gating and reproducibility. The core stays fully usable when no optional integration is +installed. + +## Non-goals for v0.1 + +- Automatically invoking fallback backends or optimizing routing policies. A `fallback` + route is a directive; cascade execution stays in the application. +- Ingesting telemetry or running as a service. Production checks read exported files. +- A general analytics query language. Segments are simple groups by scalar metadata + values. +- Hosting models or managing credentials. diff --git a/docs/backends.md b/docs/backends.md index ae5a226..df82a0f 100644 --- a/docs/backends.md +++ b/docs/backends.md @@ -1,312 +1,138 @@ -# Backends +--- +description: How DecGuard reaches decision models - built-in providers, named backends, credentials, healthchecks, error handling and the offline mock backend. +--- -Every backend implements the same interface and returns the same normalized -`DecisionResult`, so one dataset can run against any of them. +# Backends -```python -class DecisionBackend: - def predict(self, request: DecisionRequest) -> Prediction: ... # adapters implement - def decide(self, request) -> DecisionResult: ... # timing + validation, shared - def metadata(self) -> BackendMetadata: ... # provenance for reports - def healthcheck(self) -> HealthStatus: ... # ok / unhealthy / unknown - def close(self) -> None: ... -``` +A **backend** is how DecGuard reaches a decision model. Every backend implements the same +small interface and returns the same validated +[`DecisionResult`](./decisions.md#the-normalized-decisionresult), so one contract and one +dataset run unchanged against any of them. -Adapters only return raw label probabilities. `decide` measures latency and validates -them: exactly the contract's labels, finite, non-negative, summing to 1 within -`evaluation.probability_tolerance`. Anything else is an `invalid_response` error for that -case: it is counted and reported, never repaired. +## Choosing a backend -Per-case failures are typed: `timeout`, `unavailable`, `invalid_response`, -`backend_error`, or `exception` (an unexpected error from a plugin). If every case fails, -or the healthcheck reports `unhealthy`, the run stops with exit code 2. +| Provider | Use it for | Page | +| --- | --- | --- | +| `mock` | offline, deterministic tests, demos and CI; simulating defects | [below](#mock) | +| `systemone` | System One decision APIs: self-hosted **Kev**, **Jev** through OpenRouter, compatible TypeSafe endpoints | [System One](./systemone.md) | +| `http` | any endpoint that speaks DecGuard's small JSON protocol, or a thin proxy in front of your model | [Generic HTTP](./http.md) | +| plugin | a Python package registering a provider under `decguard.backends` | [Custom backends](./custom-backends.md#plugin-backends) | +| `CallableBackend` | a Python function, from the Python API only | [Custom backends](./custom-backends.md#python-callable) | -## `mock` +## Default and named backends -Deterministic and offline, for tests, examples and CI. +The contract's `backend` is the default. `backends` declares named alternatives: the +model you are replacing, a candidate, a staging endpoint, a fallback target for a +[policy](./policy.md). ```yaml -backend: +backend: # "default" provider: mock model: refund-mock-v1 - rules: # first match wins, ignoring case and whitespace - - contains: "damaged" - probabilities: {refund: 0.96, reject: 0.01, review: 0.03} - default: {refund: 0.2, reject: 0.15, review: 0.65} # when no rule matches - seed: 0 # without `default`: seeded hash of the input - model_version: "1" - position_bias: 0.0 # deliberate defect: mass moved to the option shown first -``` - -Rule and default distributions are validated against the contract labels up front. The -mock reads options as a well-behaved model would: reordered options and reformatted -labels (`REFUND`, `B) refund`, `"refund"`, `[refund]`) get the same answer. Set -`position_bias` to simulate an order-sensitive model; `decguard fuzz` should catch it. - -## `systemone` - -For System One decision APIs: [Kev](https://github.com/jaredpalmer/kev) servers you run -yourself, TypeSafe's Jev, and Jev through OpenRouter. They share one wire format (`state`, -typed `questions`, typed `answers`), so one adapter covers all of them. - -```yaml -decision: - name: support_team - type: choice - description: Which team should handle this customer support ticket? # the question asked - options: [billing, shipping, technical] - -backend: # Jev through OpenRouter's Decisions API - provider: systemone - model: typesafe/jev-1.13 # required; pin a version, not an alias - url: https://openrouter.ai/api/alpha/decisions - bearer_token_env: OPENROUTER_API_KEY - timeout_s: 60 - max_retries: 2 # connection failures, HTTP 429, 502-504 and 529 only + default: {refund: 0.2, reject: 0.15, review: 0.65} backends: - kev: # a local Kev server + kev: provider: systemone model: kev-latest url: http://127.0.0.1:8009/v1/systemone - health_url: http://127.0.0.1:8009/v1/models - typesafe: # TypeSafe's own API + jev: provider: systemone - model: jev-1.13.0 - url: https://api.typesafe.ai/v1/systemone - bearer_token_env: TYPESAFE_API_KEY - -evaluation: - probability_tolerance: 0.02 # Jev rounds probabilities to two decimals (see below) + model: typesafe/jev-1.13 + url: https://openrouter.ai/api/alpha/decisions + bearer_token_env: OPENROUTER_API_KEY ``` -Settings are those of `http` (URL checks, `health_url`, timeouts, retries, environment -credentials, `label_map` for choice and score) plus `instructions`, the question asked about -every input. It defaults to the decision's `description`; one of the two is required. - -Each case is sent as one question about the case input, keyed by the decision name: - -```json -{ - "model": "typesafe/jev-1.13", - "state": "I was charged twice for order #4411.", - "questions": { - "support_team": { - "type": "choice", - "instructions": "Which team should handle this customer support ticket?", - "criteria": {"billing": null, "shipping": null, "technical": null} - } - } -} +```bash +decguard test decguard.yaml # default +decguard test decguard.yaml --backend jev # named ``` -| decision | sent | answer used | -| --- | --- | --- | -| `choice` | options as `criteria` keys, descriptions `null`, in the order shown | `probabilities` by option; `choice` must be the most probable option | -| `noul` | no labels (a yes/no question) | `noul` is the probability of the first (positive) contract label; the second gets `1 − noul` | -| `score` | levels as the ordered `criteria` list, lowest first | `probabilities` by level index (`"0"` = lowest); `legend`, when present, must echo the levels sent | +Every backend has `provider` (required) and `model` (optional for most providers, +required for `systemone`). All other keys are provider settings, validated by the +provider: unknown keys are errors. `decguard validate` builds every declared backend +without making network calls, so configuration mistakes surface before a run. The name +`default` is reserved for the top-level backend. -Put guidance for the options into `description` or `instructions`. During fuzzing, options -and levels are sent in the order and form shown and mapped back to the contract labels, -as with `http`. A noul question has no labels, so `option_order` and `label_format` send the -same request for it. +## Credentials -Answers are checked, never repaired: the response must answer exactly this question with the -same type, and a missing or malformed answer is an `invalid_response` for that case. The -provider's `confidence` field measures something else, so DecGuard ignores it: its -confidence is the selected label's probability, as for every backend. Per result, DecGuard -records the model that answered as `model_version` (for Jev the dated snapshot, e.g. -`typesafe/jev-1.13-20260917`) and adds `request_id`, `upstream_provider` (OpenRouter's -`provider`) and `usage` (`input_tokens`, `output_tokens`, `cost`) to `metadata`. The -report's backend lists the `served_models` and `upstream_providers` seen during the run. - -Jev returns probabilities rounded to two decimals, so K options can sum to 1 ± K × 0.005. -Set `evaluation.probability_tolerance` to cover that (0.02 for up to four options); the sum -is still checked and never rescaled. Kev rounds to four decimals. - -## `http` - -For endpoints that speak DecGuard's own small JSON protocol. +Credentials come **only from environment variables**. A contract names the variable, +never the value: ```yaml backend: - provider: http - url: https://decisions.internal/v1/decide - health_url: https://decisions.internal/healthz # optional GET, 2xx = healthy - model: open-jev-2b - timeout_s: 30 - max_retries: 0 # retries only connection failures and HTTP 502/503/504 - retry_backoff_s: 0.5 # doubled after each retry - bearer_token_env: DECISIONS_TOKEN # Authorization: Bearer $DECISIONS_TOKEN - headers_from_env: {X-Tenant: DECISIONS_TENANT} - headers: {X-Client: decguard} # static, non-secret only - label_map: {"true": "yes", "false": "no"} # backend label -> contract label -``` - -Credentials are read from environment variables when the first request is made. Literal -`Authorization`/API-key headers, URL userinfo and credential-like query parameters are -rejected. When environment-backed headers are configured, `health_url` must have the same -origin as `url`, so a healthcheck cannot forward credentials elsewhere. Reports store the -URL without its query string. Redirects are not followed, untrusted HTTP error bodies are -not recorded, and values under common credential keys in backend metadata are redacted. - -### Protocol `decguard.http/0.1` - -Request (`POST url`): - -```json -{ - "protocol": "decguard.http/0.1", - "case_id": "r1", - "model": "open-jev-2b", - "decision": {"name": "refund_request", "type": "choice", "labels": ["refund", "reject", "review"]}, - "input": "The blender arrived damaged." -} -``` - -`labels` are the backend's names (after `label_map`), in the order the options should be -presented. During fuzzing (`option_order`, `label_format`) they may be reordered or -reformatted; answer with probabilities keyed by the labels exactly as sent, and DecGuard -maps them back to the contract labels. Response (`200`): - -```json -{ - "probabilities": {"refund": 0.91, "reject": 0.02, "review": 0.07}, - "model": "open-jev-2b", - "model_version": "2026-09-01", - "metadata": {"tokens": 41} -} -``` - -Only `probabilities` is required. Endpoints with a different shape need a thin proxy or a -plugin backend (below). - -## Python code - -From Python, wrap any function: - -```python -from decguard.backends import CallableBackend -from decguard.engine import run_test - - -def decide(request): - return {"refund": 0.7, "reject": 0.1, "review": 0.2} - - -report = run_test("decguard.yaml", backend=CallableBackend(decide, model="rules-v2")) -print(report.status, report.metrics.classification.accuracy) -``` - -Contracts never reference import paths (a contract must not be able to run arbitrary -code). To make a Python backend usable from contracts and the CLI, package it and register -it under the `decguard.backends` entry-point group: - -```toml -# your package's pyproject.toml -[project.entry-points."decguard.backends"] -openjev = "decguard_openjev:OpenJevBackend" + provider: systemone + model: typesafe/jev-1.13 + url: https://openrouter.ai/api/alpha/decisions + bearer_token_env: OPENROUTER_API_KEY # sends Authorization: Bearer $OPENROUTER_API_KEY ``` -```python -from typing import Self -from pydantic import BaseModel, ConfigDict -from decguard.backends import DecisionBackend, validate_settings -from decguard.decisions import Prediction - - -class OpenJevSettings(BaseModel): - model_config = ConfigDict(extra="forbid") # reject typos in contracts - device: str = "cpu" +The HTTP-based backends (`http`, `systemone`) enforce this: +- `bearer_token_env` sends `Authorization: Bearer `; `headers_from_env` maps any + other header to a variable. +- Literal `Authorization` or API-key headers, credentials embedded in the URL, and + credential-like query parameters (`api_key`, `token`, `key`, ...) are rejected. +- Variables are read when the first request is made. A missing variable stops the run with + exit code `2`. +- With environment-backed headers, `health_url` must share the backend URL's origin, so a + healthcheck cannot forward a credential elsewhere. +- Reports store URLs without their query string, redirects are not followed, untrusted + HTTP error bodies are not recorded, and values under common credential keys in backend + metadata are redacted. -class OpenJevBackend(DecisionBackend): - provider = "openjev" +In CI, map the variable from a secret; see [CI](./ci.md#credentials). - def __init__(self, settings, *, name, model): - super().__init__(name=name, model=model) - self.settings = settings +## Healthchecks - @classmethod - def from_config(cls, config, *, name, decision) -> Self: - # validate settings only; no network or model loading here - return cls( - validate_settings(OpenJevSettings, config, name=name), name=name, model=config.model - ) +Before a run, `test`, `fuzz` and `run` call the backend's healthcheck (disable with +`--no-healthcheck`). The result is `ok`, `unhealthy` or `unknown`; only `unhealthy` stops +the run (exit code `2`). HTTP backends are `unknown` unless `health_url` is set, in which +case any 2xx answer to a `GET` is healthy. The result is stored in the report's `health`. - def predict(self, request): - ... # compute {label: probability} for request.decision.labels - return Prediction(probabilities=..., model_version="...") -``` - -Then `provider: openjev` works in any contract. Built-in provider names cannot be -shadowed by plugins. Backends are called from several threads at once (up to -`evaluation.max_concurrency`), so `predict` must be thread-safe. - -## Real-backend validation - -Two opt-in test groups run genuine inference through the `systemone` backend on the -contracts in `tests/integration/real/` (one choice, one noul and one score case each). Nothing -is mocked. A default `pytest` run skips them. Selected with `-m`, they fail rather than skip -when the server, checkpoint or credential is missing. - -### Kev (`-m real_kev`) - -Upstream Kev, pinned to commit `09ff745d52a0f23954e3b0f5a608bf4c7c6aebb4`, serving Kev-0.8B, -the smallest current checkpoint (a LoRA adapter and pointer head on -`Qwen/Qwen3.5-0.8B-Base`), at Hub revision `9a45d25eb2ab761841196625383fa1dff0e56c1e`. Kev -needs Python 3.12 or 3.13 and `uv`. It runs on CPU (fp32) without a GPU, and on MLX (bf16) -on Apple Silicon. The first start downloads about 1.8 GB of weights. +## Validation and errors -```bash -git clone https://github.com/jaredpalmer/kev.git && cd kev -git checkout 09ff745d52a0f23954e3b0f5a608bf4c7c6aebb4 -uv sync --locked --extra serve -uv run --locked --extra serve python -m kev.serve \ - --run jaredpalmer/kev-0.8b@9a45d25eb2ab761841196625383fa1dff0e56c1e --port 8009 -``` +Adapters only return raw label probabilities. DecGuard measures latency and validates the +answer: exactly the contract's labels, finite, non-negative, summing to 1 within +`evaluation.probability_tolerance`. Anything else is an `invalid_response` error for that +case — counted and reported, never repaired. Per-case failures are typed (`timeout`, +`unavailable`, `invalid_response`, `backend_error`, `exception`); see +[Errors are results too](./decisions.md#errors-are-results-too). -Then, in the DecGuard checkout, once `curl -s 127.0.0.1:8009/v1/models` answers: +If every case fails, or the healthcheck reports `unhealthy`, the run stops with exit code +`2`. -```bash -DECGUARD_KEV_RUN=jaredpalmer/kev-0.8b@9a45d25eb2ab761841196625383fa1dff0e56c1e \ - uv run pytest -m real_kev -``` +Backends are called from up to `evaluation.max_concurrency` threads at once (default 4). -`DECGUARD_KEV_RUN` is optional: when set, the test fails unless the server's -`GET /v1/models` reports that checkpoint. The Kev server needs no credential (set -`KEV_API_KEY` on the server and `bearer_token_env` in the contract to require one). - -### Jev through OpenRouter (`-m real_jev`) +## `mock` -`typesafe/jev-1.13` through OpenRouter's Decisions API -(`POST https://openrouter.ai/api/alpha/decisions`). The only credential is an OpenRouter API -key in the environment: +A deterministic, offline backend for tests, examples and CI. It needs no network and +always gives the same answer for the same input. -```bash -export OPENROUTER_API_KEY=... # never commit it or put it in a contract -uv run pytest -m real_jev +```yaml +backend: + provider: mock + model: refund-mock-v1 + rules: # first match wins, ignoring case and whitespace + - contains: "damaged" + probabilities: {refund: 0.96, reject: 0.01, review: 0.03} + default: {refund: 0.2, reject: 0.15, review: 0.65} # when no rule matches + seed: 0 # without `default`: a seeded hash of the input decides + model_version: "1" + position_bias: 0.0 # deliberate defect: mass moved to the option shown first ``` -This makes three billed Jev calls (about $0.00004 in total at the current price) and one -call with a deliberately invalid key, which OpenRouter rejects with HTTP 401 and does not -bill. The tests check that neither key appears in any output, error or report. - -### GitHub Actions - -`.github/workflows/real-backends.yml` runs both groups on `ubuntu-latest` and `macos-latest` -with Python 3.13, as four jobs: `REAL KEV` and `REAL JEV` on each OS. It never runs on -ordinary commits. Start it from the Actions tab (`workflow_dispatch`), or add the -`real-backends` label to a pull request from this repository. It needs the repository secret -`OPENROUTER_API_KEY`, which only the Jev test step receives. The Kev weights are cached by -pinned revision. The JSON reports, Kev's model card and server log are kept as artifacts. - -### Any other backend (`-m external`) - -```bash -DECGUARD_EXTERNAL_CONTRACT=/absolute/path/to/real-decguard.yaml \ - uv run pytest -m external -``` +| Setting | Default | Meaning | +| --- | --- | --- | +| `rules` | `[]` | checked in order; a rule matches when `contains` appears in the input (case- and whitespace-insensitive; object inputs are matched on their JSON form) | +| `default` | none | distribution when no rule matches | +| `seed` | `0` | without `default`, unmatched inputs get a pseudo-random distribution from a seeded hash of the input, stable across processes | +| `model_version` | none | reported as the result's `model_version` | +| `position_bias` | `0.0` | share of probability mass (0–1) moved to the option presented first | -This runs `decguard test` on your own contract and requires a non-`mock` backend with at -least one valid decision. +Rule and default distributions are validated against the contract labels up front. The +mock reads options the way a well-behaved model would: reordered options and reformatted +labels (`REFUND`, `B) refund`, `"refund"`, `[refund]`) get the same answer, so it passes +the [metamorphic properties](./properties.md). Set `position_bias` to simulate an +order-sensitive model; the [fuzzing guide](./fuzzing.md#catching-an-order-sensitive-model) +shows `decguard fuzz` catching it. diff --git a/docs/ci.md b/docs/ci.md index 2097778..31e67a9 100644 --- a/docs/ci.md +++ b/docs/ci.md @@ -1,42 +1,105 @@ -# DecGuard in CI (GitHub Actions) +--- +description: Use DecGuard as a reliability gate in GitHub Actions - golden tests and properties on pull requests, regression diffs for model upgrades, and scheduled production checks. +--- -Exit codes are the interface: **0** pass (or warn), **1** a reliability gate or property -failed, **2** the run could not be evaluated. A job therefore fails whenever the model -breaks the contract, and fails differently (2) when the configuration is broken. +# CI with GitHub Actions + +Exit codes are the interface between DecGuard and CI: + +| Code | CI outcome | +| --- | --- | +| `0` | pass (or warn) — the step succeeds | +| `1` | a reliability gate or property failed — the model broke the contract | +| `2` | the run could not be evaluated — the configuration, data or backend is broken | + +A job therefore fails whenever the model breaks the contract, and fails *differently* when +the check itself is broken. No wrapper script is needed. + +## Gate every pull request + +The smallest useful workflow validates the contract, runs the golden gates and properties +together, and keeps the report even when the gate fails: ```yaml name: decision-model -on: [pull_request] + +on: + pull_request: jobs: decguard: runs-on: ubuntu-latest - env: - DECISIONS_TOKEN: ${{ secrets.DECISIONS_TOKEN }} # referenced by bearer_token_env steps: - uses: actions/checkout@v4 - - uses: astral-sh/setup-uv@v6 - - run: uv tool install decguard + - uses: actions/setup-python@v5 + with: + python-version: "3.12" + - run: pip install decguard==0.1.1 - name: Validate the contract (offline) run: decguard validate decguard.yaml - name: Golden gates and metamorphic properties + run: decguard test decguard.yaml --all --output decguard-report.json + + - uses: actions/upload-artifact@v4 + if: always() + with: + name: decguard-report + path: decguard-report.json +``` + +With the offline [`mock`](./backends.md#mock) backend this needs no secrets; with a real +backend, add its credential as below. + +## Credentials + +Contracts name environment variables; the workflow maps them from repository secrets. +Scope a secret to the step that needs it: + +```yaml + - name: Golden gates against Jev + env: + OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }} # bearer_token_env in the contract + run: decguard test decguard.yaml --all --backend jev --output jev.json +``` + +DecGuard never writes credentials into reports, errors or logs. A missing variable is a +configuration error (exit `2`), not a gate failure. + +## Regression gate for a model upgrade + +When a pull request changes the model — a new checkpoint, version or backend — test both +the current and the candidate model on the same golden dataset and diff them: + +```yaml +name: model-upgrade + +on: + pull_request: + +jobs: + regression: + runs-on: ubuntu-latest + env: + DECISIONS_TOKEN: ${{ secrets.DECISIONS_TOKEN }} # referenced by bearer_token_env + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-python@v5 + with: + python-version: "3.12" + - run: pip install decguard==0.1.1 + + - name: Candidate - golden gates and properties run: decguard test decguard.yaml --all --backend candidate --output candidate.json - - name: Regression against the production model + - name: Baseline - the model in production if: always() - run: | - decguard test decguard.yaml --backend production --output baseline.json - decguard diff baseline.json candidate.json --contract decguard.yaml --output diff.json + run: decguard test decguard.yaml --backend production --output baseline.json - - name: Post-deployment batch gate + - name: Regression limits if: always() - run: | - decguard check decguard.yaml \ - --dataset collected-production.jsonl \ - --baseline previous-production-report.json \ - --output production-report.json + run: decguard diff baseline.json candidate.json --contract decguard.yaml --output diff.json - uses: actions/upload-artifact@v4 if: always() @@ -46,21 +109,102 @@ jobs: candidate.json baseline.json diff.json - production-report.json ``` -- Pin a `fuzz.seed` in the contract (or pass `--seed`) so every CI run checks the same +`if: always()` keeps the later steps running after a failed gate, so one run shows every +problem. The contract declares both backends: + +```yaml +backend: + provider: http + url: https://decisions.internal/v1/decide + model: refund-v3 + bearer_token_env: DECISIONS_TOKEN + +backends: + production: + provider: http + url: https://decisions.internal/v1/decide + model: refund-v3 + bearer_token_env: DECISIONS_TOKEN + candidate: + provider: http + url: https://decisions.internal/v1/decide + model: refund-v4 + bearer_token_env: DECISIONS_TOKEN + +regression: + max_answer_flip_rate: 0.02 + max_accuracy_drop: 0.01 + max_ece_increase: 0.02 +``` + +To avoid re-running the baseline on every pull request, store the production model's +report once and diff against it. + +## Scheduled production check + +[`decguard check`](./production.md) is offline, so it runs after whatever job exports your +decision records. A nightly workflow: + +```yaml +name: production-reliability + +on: + schedule: + - cron: "0 5 * * *" + workflow_dispatch: + +jobs: + check: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-python@v5 + with: + python-version: "3.12" + - run: pip install decguard==0.1.1 + + - name: Export yesterday's decisions + run: ./scripts/export-decisions.sh > production.jsonl # your own export + + - name: Post-deployment gates + run: | + decguard check decguard.yaml \ + --dataset production.jsonl \ + --baseline reference/production.jsonl \ + --output production-report.json + + - uses: actions/upload-artifact@v4 + if: always() + with: + name: production-report + path: production-report.json +``` + +The baseline can be a reference batch kept in the repository, as here, or a previous +production report. DecGuard does not ingest telemetry or need access to the model. + +## Tips + +- **Pin versions.** Pin the DecGuard version, and pin model versions in the contract (for + example `typesafe/jev-1.13` rather than an alias), so a change is always a reviewed diff. +- **Pin the seed.** Set `fuzz.seed` in the contract so every run checks the same transformations; change it deliberately to explore new ones. -- Keep the JSON reports as artifacts. A failure can then be reproduced locally with - `decguard replay candidate.json`. -- `--fail-on-warn` turns warning gates into failures, for a stricter main branch. -- `decguard check` is offline: schedule the same command in cron or Actions after your own - collection/export job. DecGuard does not ingest telemetry or need service credentials. +- **Keep reports as artifacts.** A failure can then be inspected with + `decguard report`, replayed with `decguard replay`, or re-gated with + `decguard report --contract`. +- **Stricter main branch.** Add `--fail-on-warn` where warnings must block. +- **Cost.** With a billed backend, every golden case and every transformed case is a + request. Keep property `samples` small in pull-request workflows and run broader fuzzing + on a schedule. +- **Concurrency.** Lower `evaluation.max_concurrency` (or `--max-concurrency`) for + rate-limited backends; `systemone` backends also retry HTTP 429 when `max_retries` is set. ## Real-backend compatibility (this repository) -DecGuard's own `Real backends` workflow (`.github/workflows/real-backends.yml`) makes genuine -decisions through a Kev server started on the runner and through Jev on OpenRouter, on Ubuntu -and macOS. It is manual (`workflow_dispatch`, or the `real-backends` label on a pull request) -and passes `OPENROUTER_API_KEY` from repository secrets to the Jev test step only. See -[backends](backends.md#real-backend-validation) for running the same checks locally. +DecGuard's own `Real backends` workflow makes genuine decisions through a Kev server +started on the runner and through Jev on OpenRouter, on Ubuntu and macOS. It is manual +(`workflow_dispatch`, or the `real-backends` label on a pull request) and passes +`OPENROUTER_API_KEY` to the Jev test step only. See +[Real-backend validation](./systemone.md#real-backend-validation). diff --git a/docs/cli.md b/docs/cli.md new file mode 100644 index 0000000..f4bb292 --- /dev/null +++ b/docs/cli.md @@ -0,0 +1,258 @@ +--- +description: Every decguard command with its arguments, options, examples and exit codes. +--- + +# CLI reference + +```text +decguard [--version] [--help] COMMAND [ARGS]... +``` + +| Command | Purpose | +| --- | --- | +| [`validate`](#decguard-validate) | check a contract, its backends and dataset, offline | +| [`test`](#decguard-test) | run the golden dataset and apply the gates (`--all`: plus properties) | +| [`fuzz`](#decguard-fuzz) | check the metamorphic properties | +| [`replay`](#decguard-replay) | re-send stored property failures | +| [`diff`](#decguard-diff) | compare a baseline and a candidate report | +| [`report`](#decguard-report) | show a stored report, optionally re-applying edited gates | +| [`check`](#decguard-check) | analyze collected production records offline | +| [`run`](#decguard-run) | make one decision and apply the contract's policy | + +`decguard --help` prints the same options. `python -m decguard` is equivalent to +`decguard`. + +## Common options + +| Option | Commands | Meaning | +| --- | --- | --- | +| `--format`, `-f` `text\|json` | all except `validate` | what is printed to stdout; default `text` | +| `--output`, `-o FILE` | `test`, `fuzz`, `report`, `diff`, `check` | also write the JSON report to `FILE` | +| `--fail-on-warn` | `test`, `fuzz`, `report`, `diff`, `check` | exit `1` when only warnings are violated | +| `--backend`, `-b NAME` | `test`, `fuzz`, `replay`, `run` | named backend from the contract's `backends` (default: `backend`) | +| `--healthcheck` / `--no-healthcheck` | `test`, `fuzz`, `run` | check backend health first; default on | + +Paths inside a contract (`dataset`, a paraphrase `file`) are relative to the contract +file, not to the current directory. + +## `decguard validate` + +Check a contract, every backend's settings and the dataset, **without calling any +backend**. Also builds the paraphrase provider when `paraphrase` is enabled. + +```text +decguard validate CONTRACT [--dataset FILE] +``` + +| Option | Meaning | +| --- | --- | +| `--dataset`, `-d FILE` | validate this dataset instead of the contract's | + +```bash +decguard validate decguard.yaml +``` + +```text +OK decguard.yaml + decision refund_request (choice): refund, reject, review + backends default=mock + dataset cases.jsonl: 6 cases, 6 labeled + gates requirements: 2, warnings: 2 + properties none (add 'properties' to fuzz) + production segments: none; requirements: 0; segment requirements: 0 + policy none + hash sha256:ab7ab0ad031d6130242da713be280ba3ee50dd6ad14e1665cc9a8e785ff03ecd +``` + +Exits `0` when valid, `2` otherwise. Credentials are not read (they are read at the first +request), so validation works without secrets. + +## `decguard test` + +Run the golden dataset through a backend, compute metrics and apply `requirements` and +`warnings`. With `--all`, also check the metamorphic properties into the same report. + +```text +decguard test CONTRACT [OPTIONS] +``` + +| Option | Meaning | +| --- | --- | +| `--dataset`, `-d FILE` | golden dataset; overrides the contract's `dataset` | +| `--backend`, `-b NAME` | named backend | +| `--output`, `-o FILE` | write the JSON report | +| `--format`, `-f text\|json` | stdout format | +| `--max-concurrency N` | parallel backend calls, 1–256 (default: `evaluation.max_concurrency`) | +| `--fail-on-warn` | exit `1` on WARN | +| `--healthcheck` / `--no-healthcheck` | check backend health before running (default on) | +| `--all` | also check the contract's properties (`mode: all`) | +| `--seed N` | with `--all`: transformation seed (default: `fuzz.seed`, else `0`) | +| `--property`, `-p NAME` | with `--all`: run only this property (repeatable) | +| `--minimize` / `--no-minimize` | with `--all`: shrink failing transformations (default: `fuzz.minimize`) | + +```bash +decguard test decguard.yaml --output report.json +decguard test decguard.yaml --all --backend candidate --fail-on-warn +``` + +See [Golden tests and metrics](./metrics.md). + +## `decguard fuzz` + +Check the contract's metamorphic properties only (`mode: fuzz`). Deterministic for a +given seed. + +```text +decguard fuzz CONTRACT [OPTIONS] +``` + +Options are those of `test` without `--all`: `--dataset`, `--backend`, `--seed`, +`--property`, `--minimize/--no-minimize`, `--output`, `--format`, `--max-concurrency`, +`--fail-on-warn`, `--healthcheck/--no-healthcheck`. + +```bash +decguard fuzz decguard.yaml -p option_order --seed 7 -o fuzz.json +``` + +The contract must enable at least one property, and `--property` must name an enabled +one; otherwise the command exits `2`. The same applies to `test --all`. See +[Metamorphic properties](./properties.md) and [Fuzzing and replay](./fuzzing.md). + +## `decguard replay` + +Re-send the property failures stored in a `fuzz` or `test --all` report and check whether +they still fail. Also regenerates each transformation from the seed and confirms it +matches the stored one. + +```text +decguard replay REPORT [--id ID]... [--contract FILE] [--backend NAME] [--format text|json] +``` + +| Option | Meaning | +| --- | --- | +| `--id ID` | replay only this transformed case, `//` (repeatable); default: every failure | +| `--contract`, `-c FILE` | contract to use (default: the one recorded in the report) | +| `--backend`, `-b NAME` | backend to replay against (default: the one recorded in the report) | +| `--format`, `-f text\|json` | stdout format; `json` prints a replay report | + +```bash +decguard replay fuzz.json --id option_order/r1/2 --backend candidate +``` + +Exits `1` if any failure reproduces, `0` if none does. + +## `decguard diff` + +Compare two stored reports of the same decision, case by case, and apply the contract's +`regression` gates. + +```text +decguard diff BASELINE CANDIDATE [OPTIONS] +``` + +| Option | Meaning | +| --- | --- | +| `--contract`, `-c FILE` | apply this contract's `regression` gates and `evaluation` settings | +| `--segment-by`, `-s KEY` | case metadata key to segment by (repeatable); overrides `regression.segment_by` | +| `--output`, `-o FILE` | write the JSON diff | +| `--format`, `-f text\|json` | stdout format | +| `--fail-on-warn` | exit `1` on WARN | + +```bash +decguard diff baseline.json candidate.json --contract decguard.yaml --output diff.json +``` + +Without `--contract`, only the implicit `max_error_rate_increase: 0` applies. See +[Regression diffs](./regression.md). + +## `decguard report` + +Show a stored report and exit with its status. With `--contract`, recompute metrics and +re-apply the (possibly edited) gates and property limits to the stored results, without +calling the model. + +```text +decguard report REPORT [--contract FILE] [--output FILE] [--format text|json] [--fail-on-warn] +``` + +| Option | Meaning | +| --- | --- | +| `--contract`, `-c FILE` | re-evaluate against this contract's gates | +| `--output`, `-o FILE` | write the (re-evaluated) JSON report | +| `--format`, `-f text\|json` | stdout format | +| `--fail-on-warn` | exit `1` on WARN | + +```bash +decguard report report.json --contract decguard.yaml +``` + +Loading re-verifies the whole report; a tampered or inconsistent report is rejected with +exit code `2`. See [Reports](./reports.md). + +## `decguard check` + +Analyze production records your application collected, offline, and apply the contract's +`production` gates. + +```text +decguard check CONTRACT --dataset FILE [--baseline FILE] [OPTIONS] +``` + +| Option | Meaning | +| --- | --- | +| `--dataset`, `-d FILE` | collected production records, `.jsonl` or `.json` (**required**) | +| `--baseline FILE` | baseline production dataset, or a previous `decguard check` JSON report | +| `--output`, `-o FILE` | write the production report | +| `--format`, `-f text\|json` | stdout format | +| `--fail-on-warn` | exit `1` on WARN | + +```bash +decguard check decguard.yaml -d production.jsonl --baseline last-week.json -o production-report.json +``` + +No backend is called. See [Production checks](./production.md). + +## `decguard run` + +Make one decision with the contract's backend and apply its `policy`. + +```text +decguard run CONTRACT INPUT [--backend NAME] [--input-json] [--id ID] [--format text|json] +``` + +| Option | Meaning | +| --- | --- | +| `--backend`, `-b NAME` | backend that makes the decision | +| `--input-json` | parse `INPUT` as a JSON object | +| `--id ID` | case/correlation id recorded in the result (default `runtime`) | +| `--format`, `-f text\|json` | `text`: one line; `json`: the full decision and route | +| `--healthcheck` / `--no-healthcheck` | check backend health first (default on) | + +```bash +decguard run decguard.yaml "The parcel never arrived" +``` + +```text +fallback -> strong_model · selected refund · confidence 0.900 · route 1 +``` + +Exits `0` for every action, `2` if the contract has no `policy` or the decision fails. The +fallback backend is not invoked. See [Policies and Python SDK](./policy.md). + +## Exit codes + +| Code | Meaning | +| --- | --- | +| `0` | PASS; WARN unless `--fail-on-warn`; `replay`: no failure reproduced; `run`: a decision was made; `validate`: the contract is valid | +| `1` | a reliability gate failed (`test`, `fuzz`, `report`, `diff`, `check`); WARN with `--fail-on-warn`; `replay`: a failure still reproduces | +| `2` | configuration or runtime error: invalid contract, dataset, records or report; unknown backend or provider; missing credential; unhealthy backend; every case failed; invalid command-line usage; internal error | + +Unexpected exceptions map to `2`, never to `1`. Error messages go to stderr as +`error: ...`; set `DECGUARD_DEBUG=1` to print a traceback for internal errors. + +## Environment variables + +| Variable | Meaning | +| --- | --- | +| `DECGUARD_DEBUG` | print a traceback for internal errors | +| names in `bearer_token_env`, `headers_from_env`, `api_key_env` | backend and paraphrase-provider credentials; see [Credentials](./backends.md#credentials) | diff --git a/docs/contracts.md b/docs/contracts.md index 9e5aaa6..ff96e98 100644 --- a/docs/contracts.md +++ b/docs/contracts.md @@ -1,125 +1,265 @@ -# Decision Contract reference (schema 0.1) +--- +description: Complete reference for the Decision Contract, schema 0.1 - every section and field, defaults, validation rules and invalid combinations. +--- -A contract is a YAML (`.yaml`/`.yml`) or JSON (`.json`) file. Validation is strict: -unknown keys, duplicate keys and wrong types are errors that name the offending field. -Run `decguard validate ` to check one without calling any backend. +# Decision Contract reference + +This page describes **schema `0.1`**, the contract format read by DecGuard `0.1.x`. For an +introduction, see [Decisions and results](./decisions.md) and the +[quickstart](./quickstart.md). + +## File format + +A contract is a YAML (`.yaml`, `.yml`) or JSON (`.json`) file, UTF-8, at most 1 MB. + +Validation is strict. **Unknown keys, duplicate keys and wrong types are errors** that +name the offending field, so typos cannot silently disable a gate: + +```text +error: decguard.yaml: invalid contract: + requirements.min_acuracy: unknown field (check the spelling) +``` + +YAML is parsed with a safe loader; contracts never execute code or reference import paths. +Run [`decguard validate`](./cli.md#decguard-validate) to check a contract without calling +any backend. ```yaml -schema_version: "0.1" # required, quoted string +schema_version: "0.1" # required decision: {...} # required -backend: {...} # required, the default backend -backends: {name: {...}} # optional alternative backends -dataset: cases.jsonl # optional, relative to the contract file -evaluation: {...} # optional -requirements: {...} # optional hard gates -warnings: {...} # optional soft gates -properties: {...} # optional metamorphic properties (decguard fuzz / test --all) -fuzz: {...} # optional fuzz settings: seed, text_field, minimization -regression: {...} # optional regression gates (decguard diff) -production: {...} # optional post-deployment gates and metadata segments -policy: {...} # optional ordered runtime routes +backend: {...} # required: the default backend +backends: {name: {...}} # named alternative backends +dataset: cases.jsonl # golden dataset, relative to the contract +evaluation: {...} # metric settings +requirements: {...} # hard gates (golden tests) +warnings: {...} # soft gates (golden tests) +properties: {...} # metamorphic properties +fuzz: {...} # fuzz settings +regression: {...} # decguard diff gates +production: {...} # decguard check gates and segments +policy: {...} # runtime routes ``` +**Names** — decision names, backend names and provider names — start with a letter or +digit and contain only letters, digits, `_`, `.` and `-`. + +## `schema_version` + +Required. The string `"0.1"`. Quote it in YAML: an unquoted `0.1` is a number and is +rejected. + ## `decision` -Common fields: `name` (letters, digits, `_`, `.`, `-`), `description` (optional), `type`. +Required. The question and its labels. -| type | fields | labels | -| --- | --- | --- | -| `choice` | `options: [a, b, c]` (at least 2, unique) | unordered options | -| `noul` | `labels: [yes, no]` (optional, exactly 2) | boolean; positive label first | -| `score` | `levels: [1, 2, 3, 4]` (at least 2, unique) | ordered, lowest first; numbers become strings | +| Field | Types | Required | Meaning | +| --- | --- | --- | --- | +| `name` | all | yes | decision name; appears in reports, backend requests and production records | +| `type` | all | yes | `choice`, `noul` or `score` | +| `description` | all | no | the question in words; required by [`systemone`](./systemone.md) backends unless they set `instructions` | +| `options` | `choice` | yes | at least 2 unique labels | +| `labels` | `noul` | no | exactly 2 labels, positive first; default `[yes, no]` | +| `levels` | `score` | yes | at least 2 unique levels, lowest first; numbers become strings | -The label order in the contract is the canonical order: probabilities in results and -reports follow it, and when two labels tie for the highest probability the one listed -first is selected. +Labels must be non-empty, unique and have no surrounding spaces. Their order is the +canonical order: results list probabilities in this order, and ties for the highest +probability go to the label listed first. ## `backend` and `backends` -```yaml -backend: - provider: mock # built-in: mock, http; or an installed plugin's name - model: my-model # optional, passed to the backend - ...provider settings... - -backends: - candidate: - provider: http - url: https://... -``` +`backend` (required) is the default backend, named `default`. `backends` maps names to +alternative backends, selected with `--backend NAME` or named by +[policy](#policy) fallback routes. The name `default` is reserved. -`decguard test --backend candidate` runs the named backend; the top-level one is called -`default`. Provider settings are described in [backends.md](backends.md). +| Field | Required | Meaning | +| --- | --- | --- | +| `provider` | yes | `mock`, `http`, `systemone`, or an installed plugin's name | +| `model` | no (yes for `systemone`) | model identifier passed to the backend | +| *other keys* | — | provider settings, validated by the provider; unknown keys are errors | -## `dataset` +Provider settings: + +- [`mock`](./backends.md#mock): `rules`, `default`, `seed`, `model_version`, `position_bias`; +- [`http`](./http.md#settings): `url`, `health_url`, `timeout_s`, `max_retries`, + `retry_backoff_s`, `bearer_token_env`, `headers_from_env`, `headers`, `label_map`; +- [`systemone`](./systemone.md#settings): the `http` settings plus `instructions`; +- plugins: whatever their settings model defines ([Custom backends](./custom-backends.md)). -Path to a golden dataset, relative to the contract. `decguard test --dataset` overrides it. +Contracts hold environment-variable **names**, never secret values. See +[Credentials](./backends.md#credentials). -- `.jsonl`: one JSON object per line (blank lines ignored). -- `.json`: a list of objects. +## `dataset` -Each case: `input` (string or JSON object, required), `expected` (a contract label, -optional), `id` (optional, defaults to `case-`), `metadata` (optional object). Cases -without `expected` still count for coverage, latency and error rate. Invalid rows, -unknown labels and duplicate ids stop loading; nothing is skipped silently. +Optional. Path to the [golden dataset](./datasets.md) (`.jsonl` or `.json`), relative to +the contract file. `--dataset` overrides it on `validate`, `test` and `fuzz`. Commands +that need a dataset fail with exit code `2` when neither is given. ## `evaluation` -| key | default | meaning | -| --- | --- | --- | -| `confidence_threshold` | none | decisions with confidence below it are abstentions; enables coverage metrics | -| `calibration_bins` | 10 | equal-width bins for ECE | -| `probability_tolerance` | 0.001 | how far a backend's probabilities may sum away from 1 | -| `max_concurrency` | 4 | parallel backend calls (`--max-concurrency` overrides) | +Optional. Settings shared by all metrics. + +| Field | Default | Range | Meaning | +| --- | --- | --- | --- | +| `confidence_threshold` | none | 0–1 | decisions below it are abstentions; enables coverage metrics and gates | +| `calibration_bins` | `10` | 1–1000 | equal-width bins for ECE | +| `probability_tolerance` | `0.001` | > 0, ≤ 0.1 | how far a backend's probabilities may sum away from 1 | +| `max_concurrency` | `4` | 1–256 | parallel backend calls (`--max-concurrency` overrides) | ## `requirements` and `warnings` -Both accept the same gates. A violated requirement fails the run (exit 1); a violated -warning turns PASS into WARN (exit 0 unless `--fail-on-warn`). +Optional. Golden-test gates, applied by `test`, `test --all` and `report`. Both sections +accept the same keys. A violated requirement fails the run (exit `1`); a violated warning +turns PASS into WARN. See [Requirements and warnings](./gates.md). -| gate | metric | holds when | +| Gate | Range | Holds when | | --- | --- | --- | -| `min_accuracy` | accuracy | `>=` | -| `min_macro_f1` | macro-F1 | `>=` | -| `max_nll` | negative log-likelihood | `<=` | -| `max_brier` | multi-class Brier score (0–2) | `<=` | -| `max_ece` | expected calibration error | `<=` | -| `min_coverage` | share of decisions at/above the threshold | `>=` | -| `max_abstention_rate` | share below the threshold | `<=` | -| `min_selective_accuracy` | accuracy at/above the threshold | `>=` | -| `max_error_rate` | share of cases the backend failed | `<=` | -| `max_latency_p95_ms` | 95th percentile latency | `<=` | -| `max_ordinal_mae` | mean level distance (score only) | `<=` | - -Coverage gates require `evaluation.confidence_threshold`. `max_error_rate` defaults to `0` -as a requirement: any case the backend could not decide fails the run unless you allow -some. A gate whose metric cannot be computed (for example accuracy with no labeled cases) -fails. +| `min_accuracy` | 0–1 | accuracy ≥ value | +| `min_macro_f1` | 0–1 | macro-F1 ≥ value | +| `max_nll` | ≥ 0 | negative log-likelihood ≤ value | +| `max_brier` | 0–2 | Brier score ≤ value | +| `max_ece` | 0–1 | expected calibration error ≤ value | +| `min_coverage` | 0–1 | share at/above `confidence_threshold` ≥ value | +| `max_abstention_rate` | 0–1 | share below `confidence_threshold` ≤ value | +| `min_selective_accuracy` | 0–1 | accuracy at/above `confidence_threshold` ≥ value | +| `max_error_rate` | 0–1 | errored cases / all cases ≤ value; **implicit requirement `0`** | +| `max_latency_p95_ms` | ≥ 0 | p95 backend latency ≤ value | +| `max_ordinal_mae` | ≥ 0 | mean level distance ≤ value (`score` only) | -## Contract hash +Metric definitions: [Golden tests and metrics](./metrics.md). + +## `properties` -Reports record `sha256:` of the normalized contract (canonical JSON of the parsed model), -so reformatting or reordering keys does not change it but any semantic change does. -Contracts contain environment-variable names, never secret values. +Optional. Metamorphic properties, checked by `fuzz` and `test --all`. Each key enables +one property; see [Metamorphic properties](./properties.md) for behavior and examples. -## `properties` and `fuzz` +Shared fields: -Metamorphic properties (option order, label format, irrelevant context, whitespace, -paraphrase, noul inversion, score monotonicity) and their tolerances. See -[properties.md](properties.md). +| Field | Default | Meaning | +| --- | --- | --- | +| `enabled` | `true` | `false` disables the property but keeps its settings | +| `level` | `requirement` | `requirement` or `warning` | +| `max_tv_distance` | none | per-case limit, 0–1 (not `monotonic`) | +| `max_js_divergence` | none | per-case limit, 0–1 (not `monotonic`) | +| `max_abs_delta` | none | per-case limit, 0–1 (not `monotonic`) | +| `max_confidence_delta` | none | per-case limit, 0–1 (not `monotonic`) | +| `max_violation_rate` | `0` | share of transformed cases allowed to break a per-case limit | +| `max_flip_rate` | none | share of transformed cases whose selected label may change | + +Every property except `monotonic` needs at least one of the four per-case limits or +`max_flip_rate`. + +| Property | Decisions | Specific fields (defaults) | +| --- | --- | --- | +| `option_order` | choice, noul | `samples` (3) | +| `label_format` | all | `styles` (all of `upper`, `lower`, `capitalized`, `lettered`, `numbered`, `quoted`, `bracketed`), `aliases` ({}), `samples` (2) | +| `irrelevant_context` | all | `fragments` (8 neutral sentences), `insertions` (1), `samples` (2) | +| `repeated_context` | all | `fragments` (same defaults), `repeats` (3), `samples` (1) | +| `whitespace` | all | `edits` (3), `samples` (2) | +| `paraphrase` | all | `source` (`{provider: identity}`), `samples` (1) | +| `inversion` | noul | `rewrites` (required: list of `{find, replace}`), `samples` (1) | +| `monotonic` | score | `field` (required), `direction` (`increasing`), `deltas` ([1.0]), `max_level_decrease` (0.0) | + +## `fuzz` + +Optional. Settings for every property run. + +| Field | Default | Meaning | +| --- | --- | --- | +| `seed` | `0` | every transformation derives from (seed, property, case id); `--seed` overrides | +| `text_field` | none | for object inputs: the top-level key whose string value text properties edit | +| `minimize` | `true` | shrink failing transformations; `--minimize/--no-minimize` overrides | +| `max_minimize_calls` | `32` | backend calls allowed per failing case while minimizing (0–1000) | ## `regression` -Limits on how much a candidate run may degrade relative to a baseline, applied by -`decguard diff`. See [regression.md](regression.md). +Optional. Gates applied by [`decguard diff --contract`](./regression.md). Gate keys at the +top level are requirements; the same keys under `regression.warnings` are warnings. Every +gate holds when the value is ≤ the limit. + +| Field | Meaning | +| --- | --- | +| `max_answer_flip_rate` | share of cases whose selected label changed | +| `max_accuracy_drop`, `max_macro_f1_drop` | baseline − candidate | +| `max_ece_increase`, `max_brier_increase`, `max_nll_increase` | candidate − baseline | +| `max_mean_tv_distance` | mean total variation distance between matched distributions | +| `max_mean_confidence_shift` | mean absolute confidence shift | +| `max_error_rate_increase` | candidate − baseline error rate; **implicit requirement `0`** | +| `max_latency_p95_increase_ms` | candidate − baseline p95 latency | +| `max_segment_accuracy_drop` | worst accuracy drop over segments of at least `min_segment_size` cases | +| `warnings` | the same gates at warning level | +| `segment_by` | case metadata keys that define segments (default `[]`) | +| `min_segment_size` | smallest segment considered by `max_segment_accuracy_drop` (default `5`) | ## `production` -Offline aggregate, baseline-drift and per-segment gates applied by `decguard check`. -Production data is separate from the golden `dataset`. See [production.md](production.md) -for its JSONL schema, metrics and gate reference. +Optional. Gates and segments applied by [`decguard check`](./production.md). + +| Field | Meaning | +| --- | --- | +| `segments` | metadata keys; one group per scalar value, plus `` | +| `requirements`, `warnings` | aggregate gates | +| `segment_requirements`, `segment_warnings` | gates applied to every segment group | + +Aggregate gates: `min_accuracy`, `max_error_rate`, `max_ece`, `max_brier`, `max_nll`, +`min_threshold_coverage`, `max_abstention_rate`, `max_fallback_rate`, and the baseline +gates `max_confidence_tv_distance`, `max_accuracy_drop`, `max_ece_increase` (need +`--baseline`). Segment gates: the same without the three baseline gates. See the +[gate table](./production.md#metrics-drift-and-gates). ## `policy` -An ordered, deterministic route table over decision confidence with `accept`, `abstain`, -`fallback` and `human_review` actions. See [policy.md](policy.md). +Optional. An ordered, first-match route table used by `decguard run` and the +[Python SDK](./policy.md#python-sdk). + +| Field | Meaning | +| --- | --- | +| `routes` | at least one route | +| `routes[].when.confidence_gte` | condition: confidence ≥ this value (0–1); omit on the final route | +| `routes[].action` | `accept`, `abstain`, `fallback` or `human_review` | +| `routes[].backend` | for `fallback` only (required there): a backend declared in this contract | + +## Invalid combinations + +Besides type and range errors, the contract is rejected when: + +| Rule | Error example | +| --- | --- | +| `min_coverage`, `max_abstention_rate` or `min_selective_accuracy` without `evaluation.confidence_threshold` | `requirements.min_coverage needs evaluation.confidence_threshold to be set` | +| `production` `min_threshold_coverage` without `evaluation.confidence_threshold` | `production min_threshold_coverage needs evaluation.confidence_threshold to be set` | +| `max_ordinal_mae` on a non-`score` decision | `requirements.max_ordinal_mae only applies to score decisions` | +| a property that does not apply to the decision type | `properties.option_order applies to choice and noul decisions, not score` | +| a property (except `monotonic`) with no limit | `set at least one of max_tv_distance, ... or max_flip_rate` | +| `label_format.aliases` for labels the decision does not have | `properties.label_format.aliases: unknown labels [...]` | +| `regression.max_segment_accuracy_drop` without `segment_by` | `max_segment_accuracy_drop needs segment_by (case metadata keys)` | +| a backend named `default` under `backends` | `'default' is reserved for the top-level 'backend'` | +| policy: the last route has a `when`, or an earlier route has none | `the final route must be unconditional` | +| policy: `confidence_gte` thresholds not strictly descending | `confidence_gte thresholds must be strictly descending` | +| policy: `fallback` without `backend`, or `backend` on another action | `fallback routes require 'backend'` | +| policy: `backend` not declared in the contract | `unknown backend 'x'; the contract defines: ...` | +| `systemone` without `model`, or without `decision.description` / `instructions` | `systemone backend 'default': 'model' is required` | +| `label_map` targets that are not contract labels, or two labels mapped to one | `label_map targets [...] are not contract labels` | +| `label_map` on a `systemone` `noul` decision | `label_map does not apply to noul decisions` | +| HTTP: literal credential headers, credentials in the URL, credential-like query parameters | `'Authorization' looks like a credential; ...` | +| HTTP: `bearer_token_env` and an `Authorization` header from the environment together | `set either bearer_token_env or an Authorization header, not both` | +| HTTP: `health_url` on another origin while credentials come from the environment | `health_url must use the backend URL's origin ...` | + +Backend errors (the last six rows) are reported by `decguard validate`, which builds every +declared backend, and by any command that uses the backend. + +## Contract hash + +Reports record `sha256:` of the normalized contract: canonical JSON of the parsed model, +with defaults filled in. Reformatting, comments and key order do not change it; any +semantic change does. `decguard validate` prints it. + +## Complete example + +The repository's +[refund example](https://github.com/suffro/decguard/blob/main/examples/refund/decguard.yaml) +uses every section: a mock default backend, an order-sensitive candidate, a fallback +target and an HTTP staging backend; golden gates; six properties; regression limits; +production gates with segments; and a three-route policy. The +[severity](https://github.com/suffro/decguard/blob/main/examples/severity/decguard.yaml) +(`score`, object inputs, `monotonic`) and +[spam](https://github.com/suffro/decguard/blob/main/examples/spam/decguard.yaml) (`noul`, +`label_format`) examples cover the other decision types. diff --git a/docs/custom-backends.md b/docs/custom-backends.md new file mode 100644 index 0000000..b364a4c --- /dev/null +++ b/docs/custom-backends.md @@ -0,0 +1,169 @@ +--- +description: Connect a model through Python - wrap a function with CallableBackend, or package a DecisionBackend plugin usable from contracts and the CLI. +--- + +# Custom backends + +When a model is reachable from Python, you can connect it without an HTTP endpoint: + +- **[`CallableBackend`](#python-callable)** wraps a plain function. It is for the Python + API (tests, notebooks, the SDK), not for contracts. +- **[Plugin backends](#plugin-backends)** are `DecisionBackend` subclasses in an installed + package, registered under the `decguard.backends` entry-point group. Contracts and the + CLI use them by `provider` name. + +Contracts never reference Python import paths. A contract is data and must not be able to +run arbitrary code, so the only way to make Python code available to contracts is an +installed package that registers it. + +## The interface + +```python +class DecisionBackend: + provider: ClassVar[str] + + def predict(self, request: DecisionRequest) -> Prediction: ... # implemented by adapters + def decide(self, request, *, tolerance) -> DecisionResult: ... # shared: timing + validation + def metadata(self) -> BackendMetadata: ... # provenance for reports + def healthcheck(self) -> HealthStatus: ... # ok / unhealthy / unknown + def close(self) -> None: ... # release resources +``` + +Adapters implement **`predict`** and return raw probabilities. The shared `decide` +measures latency, validates the probabilities and builds the +[`DecisionResult`](./decisions.md#the-normalized-decisionresult), so results cannot +diverge between backends. + +- **`DecisionRequest`** has `case_id`, `decision` (a spec with `name`, `type` and + `labels`) and `input` (a string or a dict). +- **`request.decision.labels`** are the labels **in the order and form to present**. While + fuzzing, [`option_order`](./properties.md#option-order) and + [`label_format`](./properties.md#label-format) reorder or reformat them. Key your + probabilities by these labels exactly; DecGuard maps them back to the contract labels. +- **`Prediction`** has `probabilities` (a mapping label → probability) and optional + `model`, `model_version` and `metadata`. +- Raise `decguard.errors.BackendError` (or `BackendTimeout`, `BackendUnavailable`, + `InvalidResponse`) for a failure that concerns one request; it becomes a per-case error. + Any other exception is recorded as an `exception` error for that case. +- `predict` is called from several threads at once, up to `evaluation.max_concurrency`, so + it must be thread-safe. + +## Python callable + +Wrap any function that takes a `DecisionRequest` and returns a mapping of label to +probability, or a `Prediction`: + +```python +from decguard.backends import CallableBackend +from decguard.engine import run_test + + +def decide(request): + # request.decision.labels: the labels to answer for, as presented + return {"refund": 0.7, "reject": 0.1, "review": 0.2} + + +report = run_test("decguard.yaml", backend=CallableBackend(decide, model="rules-v2")) +print(report.status, report.metrics.classification.accuracy) +``` + +`run_test` is the engine behind the CLI. It takes the same inputs as `decguard test` — +`dataset`, `backend` (a name or an instance), `max_concurrency`, `check_health`, +`mode` (`"test"`, `"fuzz"` or `"all"`), `seed`, `properties`, `minimize` — and returns the +`Report`. The same `CallableBackend` works with the runtime +[SDK](./policy.md#python-sdk): `DecGuard.from_contract("decguard.yaml", backend=CallableBackend(decide))`. + +## Plugin backends + +A plugin is a normal Python package. This one loads a local model lazily and exposes it +as `provider: localmodel`: + +```python +# decguard_localmodel/__init__.py +import threading +from typing import Self + +from pydantic import BaseModel, ConfigDict + +from decguard.backends import DecisionBackend, HealthStatus, validate_settings +from decguard.decisions import DecisionRequest, Prediction + + +class LocalModelSettings(BaseModel): + model_config = ConfigDict(extra="forbid") # typos in the contract are errors + checkpoint: str + device: str = "cpu" + + +class LocalModelBackend(DecisionBackend): + provider = "localmodel" + + def __init__(self, settings: LocalModelSettings, *, name: str, model: str | None) -> None: + super().__init__(name=name, model=model) + self.settings = settings + self._model = None + self._lock = threading.Lock() + + @classmethod + def from_config(cls, config, *, name, decision) -> Self: + # Validate settings only. No network or model loading here: `decguard validate` + # builds every backend and must stay offline. + settings = validate_settings(LocalModelSettings, config, name=name) + return cls(settings, name=name, model=config.model) + + def _loaded(self): + with self._lock: # predict() is called from several threads + if self._model is None: + self._model = load_my_model(self.settings.checkpoint, self.settings.device) + return self._model + + def predict(self, request: DecisionRequest) -> Prediction: + options = list(request.decision.labels) # in the order and form to present + probabilities = self._loaded().predict(request.input, options) + return Prediction(probabilities=probabilities, model_version=self.settings.checkpoint) + + def healthcheck(self) -> HealthStatus: + self._loaded() + return HealthStatus(status="ok", detail=f"loaded {self.settings.checkpoint}") +``` + +Register it in the package's `pyproject.toml`: + +```toml +[project.entry-points."decguard.backends"] +localmodel = "decguard_localmodel:LocalModelBackend" +``` + +Once the package is installed in the same environment as DecGuard, any contract can use +it. Keys besides `provider` and `model` are passed to your settings model: + +```yaml +backend: + provider: localmodel + model: refund-local + checkpoint: ./checkpoints/refund-v3 + device: cpu +``` + +`validate_settings` reports errors against the contract path, like the built-in backends: + +```text +error: invalid settings for provider 'localmodel': + backend.devcie: unknown field (check the spelling) +``` + +Rules for plugins: + +- `from_config` must not do network I/O or load models, so `decguard validate` works + offline. Do that lazily in `predict` or `healthcheck`. +- Built-in provider names (`mock`, `http`, `systemone`) cannot be shadowed. +- An unknown provider name is a configuration error (exit `2`) that lists the available + providers. +- Override `metadata()` to add provenance (for example a model hash) to reports, and + `close()` to release resources. + +## Paraphrase plugins + +The [`paraphrase`](./properties.md#paraphrase) property has its own extension point: a +`decguard.fuzz.paraphrase.Paraphraser` subclass registered under the +`decguard.paraphrasers` entry-point group. diff --git a/docs/datasets.md b/docs/datasets.md new file mode 100644 index 0000000..e884f40 --- /dev/null +++ b/docs/datasets.md @@ -0,0 +1,94 @@ +--- +description: The golden dataset format, labeled and unlabeled cases, metadata for segments, and how to build a dataset that catches real regressions. +--- + +# Golden datasets + +A **golden dataset** is a fixed set of inputs, most with a known correct answer, that a +model version is tested against. It is the input of `decguard test`, `decguard fuzz` and +`decguard test --all`. The same file, run against two model versions, is what +[`decguard diff`](./regression.md) compares. + +Golden datasets are for testing **before** deployment. Decisions collected **after** +deployment use a different, richer record format: see +[production checks](./production.md#record-schema). + +## Format + +A dataset is either: + +- **`.jsonl`** — one JSON object per line; blank lines are ignored (recommended); +- **`.json`** — a JSON list of objects. + +```json +{"id": "r1", "input": "The blender arrived damaged and won't turn on.", "expected": "refund", "metadata": {"locale": "en"}} +{"id": "r5", "input": "The invoice shows a different price than the website did.", "expected": "review", "metadata": {"locale": "en"}} +{"id": "r11", "input": "Can someone call me about my order?", "metadata": {"locale": "en"}} +``` + +| Field | Required | Meaning | +| --- | --- | --- | +| `input` | yes | what the model sees: a string or a JSON object | +| `expected` | no | the correct label; must be one of the contract's labels (score levels may be numbers) | +| `id` | no | unique case id; defaults to `case-` by position (1-based) | +| `metadata` | no | a JSON object of attributes such as `locale` or `customer_tier` | + +The contract points to the dataset with `dataset:`, relative to the contract file. +`decguard validate --dataset` and `decguard test --dataset` override it for one run. + +## Strict loading + +Nothing is skipped silently. Loading stops with exit code `2`, naming the line or +position, when a case: + +- is not valid JSON, or has an unknown field; +- has no `input`; +- has an `expected` label that is not in the contract; +- repeats an `id` already used. + +An empty dataset is also an error. The report stores the file's `sha256:` hash, so you can +tell exactly which dataset produced a result. + +## Labeled and unlabeled cases + +Cases without `expected` still run. They count toward coverage, abstention, latency and +error rate, and they are transformed by [metamorphic properties](./properties.md), which +compare the model with itself and need no gold answer. Accuracy, macro-F1, calibration +(ECE, Brier, NLL) and selective accuracy use **labeled** cases only. + +A gated metric that cannot be computed fails its gate: `min_accuracy` on a dataset with no +labeled cases is a failure, not a pass. + +## Object inputs + +When the model needs structured input, pass a JSON object: + +```json +{"id": "s2", "input": {"text": "Checkout latency degraded by 40% in eu-west.", "affected_users": 8000}, "expected": 3} +``` + +Text properties then need `fuzz.text_field` to know which key to edit, and the +[`monotonic`](./properties.md#monotonic-score) property changes a numeric field. The +[severity example](https://github.com/suffro/decguard/tree/main/examples/severity) uses +both. + +## Metadata and segments + +`metadata` is carried through to every report. Its keys become **segments** in +regression diffs (`regression.segment_by`, `decguard diff --segment-by`), which report +accuracy and flip rate per value, and can gate on the worst segment. Use scalar values +(strings, numbers, booleans) for keys you want to segment by. + +## Building a useful dataset + +- **Stable ids.** Diffs match cases by `id` across runs. Set ids explicitly and never + reuse one for a different input. +- **Real distribution first, then edge cases.** Sample inputs the model will actually see, + then add the ambiguous and adversarial cases you care about. Label edge cases carefully: + one wrong label moves both accuracy and calibration. +- **Cover every label.** Macro-F1 and per-label precision/recall are only meaningful when + each label has support. +- **Enough cases for calibration.** ECE bins cases by confidence; with a handful of cases, + single decisions dominate it. Set gates with the dataset size in mind. +- **Version it with the contract.** Commit the dataset next to `decguard.yaml` so a change + to either is reviewed together. diff --git a/docs/decisions.md b/docs/decisions.md new file mode 100644 index 0000000..b32b2f7 --- /dev/null +++ b/docs/decisions.md @@ -0,0 +1,174 @@ +--- +description: Probabilistic decisions, the three decision types, the Decision Contract and the normalized DecisionResult every backend produces. +--- + +# Decisions and results + +## Probabilistic decisions + +A **decision** is a typed question asked about an input, with a fixed set of possible +answers called **labels**. A probabilistic decision model does not just return an answer: +it returns a probability for every label. + +```text +input "The blender arrived damaged and won't turn on." +question refund_request: refund, reject or review? +answer {refund: 0.96, reject: 0.01, review: 0.03} → selected refund, confidence 0.96 +``` + +DecGuard treats that distribution as the model's output. The **selected** label is the +most probable one and the **confidence** is its probability. Everything DecGuard +measures — accuracy, calibration, coverage, distribution shifts — derives from the +distributions, which is why the contract fixes the set of labels up front. + +Inputs are either a string or a JSON object (for example +`{"text": "...", "affected_users": 8000}`). + +## Decision types + +Three types cover the questions decision models answer. The type decides how labels are +declared and which metrics and properties apply. + +### Choice + +Pick one of two or more unordered options. + +```yaml +decision: + name: refund_request + type: choice + options: [refund, reject, review] +``` + +### Noul + +A yes/no question. `labels` is optional and defaults to `[yes, no]`; the **positive label +comes first**. + +```yaml +decision: + name: is_spam + type: noul + labels: [spam, ham] +``` + +### Score + +A level on an ordered scale, listed **from lowest to highest**. Numbers are allowed and +become strings (`1` → `"1"`). Score decisions also get the ordinal error metric +(`ordinal_mae`) and the [`monotonic`](./properties.md#monotonic-score) property. + +```yaml +decision: + name: incident_severity + type: score + levels: [1, 2, 3, 4] +``` + +### Label order + +The order in the contract is the **canonical order**. Probabilities in results and +reports always follow it, and when two labels tie for the highest probability, the one +listed first is selected. Labels must be unique, non-empty and have no surrounding spaces. + +## The Decision Contract + +The contract is the single YAML (or JSON) file that every command reads. Only three keys +are required; each other section adds one capability. + +| Section | Purpose | Used by | +| --- | --- | --- | +| `schema_version` | contract format, currently `"0.1"` (required) | all | +| `decision` | name, type and labels (required) | all | +| `backend`, `backends` | the default backend (required) and named alternatives | `test`, `fuzz`, `replay`, `run` | +| `dataset` | golden dataset path | `validate`, `test`, `fuzz` | +| `evaluation` | confidence threshold, calibration bins, probability tolerance, concurrency | all metrics | +| `requirements`, `warnings` | golden-test gates | `test`, `report` | +| `properties`, `fuzz` | metamorphic properties and fuzz settings | `fuzz`, `test --all` | +| `regression` | limits for baseline vs candidate | `diff` | +| `production` | post-deployment gates and metadata segments | `check` | +| `policy` | ordered runtime routes | `run`, Python SDK | + +Contracts are data only. They are parsed with a safe YAML loader, reject unknown keys and +duplicate keys, never reference Python import paths, and hold environment-variable +**names**, never secret values. Each report records a `sha256:` hash of the normalized +contract, so formatting changes do not change it but any semantic change does. + +The [Decision Contract reference](./contracts.md) lists every field. + +## The normalized `DecisionResult` + +Every backend's raw answer is validated and normalized into the same structure. This is +one decision from the quickstart, as stored in a report: + +```json +{ + "case_id": "r1", + "decision": "refund_request", + "decision_type": "choice", + "labels": ["refund", "reject", "review"], + "probabilities": {"refund": 0.96, "reject": 0.01, "review": 0.03}, + "selected": "refund", + "confidence": 0.96, + "backend": "default", + "provider": "mock", + "model": "refund-mock-v1", + "model_version": null, + "latency_ms": 0.0073, + "metadata": {} +} +``` + +| Field | Meaning | +| --- | --- | +| `case_id` | dataset case id (or the id passed to `run` / the SDK) | +| `decision`, `decision_type`, `labels` | the contract's decision and its canonical labels | +| `probabilities` | one probability per label, in canonical order | +| `selected`, `confidence` | the most probable label (ties to the first) and its probability | +| `backend`, `provider` | the contract's backend name (`default` or a key of `backends`) and its provider | +| `model`, `model_version` | what the backend reports it served, when known | +| `latency_ms` | time spent in the backend call | +| `metadata` | backend-specific extras (for example token usage); secret-like keys are redacted | + +### Validation, never repair + +Before a result exists, its probabilities must: + +- cover **exactly** the contract's labels, no more and no fewer; +- be finite, non-negative numbers; +- sum to 1 within `evaluation.probability_tolerance` (default `0.001`). + +Anything else is an `invalid_response` error for that case. DecGuard never rescales, +clips or fills in probabilities, and never drops a case. Confidence is always derived from +the probabilities: if a provider returns its own "confidence" field, DecGuard ignores it. + +The same invariants are re-checked whenever a stored report is loaded, so a tampered or +corrupted report is rejected. + +### Errors are results too + +A case the backend could not answer is recorded with a typed error instead of a result: + +| Kind | Cause | +| --- | --- | +| `timeout` | the backend did not answer within its timeout | +| `unavailable` | connection failure, or a retryable HTTP status after retries | +| `invalid_response` | malformed output, wrong labels, probabilities that do not validate | +| `backend_error` | the backend returned a non-retryable error status (for example HTTP 400 or 500) | +| `exception` | an unexpected exception from a plugin backend | +| `paraphrase_error` | fuzzing only: the paraphrase provider failed for this case | + +Errors count toward `error_rate`, which is an [implicit requirement](./gates.md#implicit-gates) +of `0`. If **every** case fails, or the backend's healthcheck reports it unhealthy, the run +stops with exit code `2`: there is nothing to evaluate. + +## Backend neutrality + +Because every backend produces the same `DecisionResult`, everything downstream — metrics, +gates, fuzz comparisons, regression diffs, policies — is written once and works with any +model. Switching from the mock to a local Kev server or to Jev through OpenRouter changes +the `backend` section and nothing else. Two reports from different backends can be +[diffed](./regression.md) directly. + +Adapters stay thin: they translate the request to the model's wire format and return raw +label probabilities. See [Backends](./backends.md). diff --git a/docs/fuzzing.md b/docs/fuzzing.md new file mode 100644 index 0000000..753472c --- /dev/null +++ b/docs/fuzzing.md @@ -0,0 +1,155 @@ +--- +description: Run metamorphic fuzzing, read property failures, reproduce them from the seed, minimize them, and replay stored failures against any backend. +--- + +# Fuzzing and replay + +`decguard fuzz` checks the contract's [metamorphic properties](./properties.md). It is +deterministic: the same seed always generates the same transformed inputs, failures are +shrunk to minimal examples, and every failure can be replayed from the stored report. + +```bash +decguard fuzz decguard.yaml # every enabled property +decguard test decguard.yaml --all # golden gates + properties +decguard fuzz decguard.yaml -p option_order --seed 7 -o fuzz.json +``` + +| Option | Meaning | +| --- | --- | +| `--property`, `-p NAME` | run only this property (repeatable); default: all enabled | +| `--seed N` | transformation seed; default: `fuzz.seed` in the contract, else `0` | +| `--minimize` / `--no-minimize` | shrink failing transformations; default: `fuzz.minimize` (`true`) | +| `--backend`, `-b NAME` | run against a named backend | +| `--output`, `-o FILE` | write the JSON report, needed for `replay` | + +`fuzz` and `test --all` accept the same options as `test`; see the [CLI reference](./cli.md#decguard-fuzz). + +## Catching an order-sensitive model + +The repository's +[refund example](https://github.com/suffro/decguard/blob/main/examples/refund/decguard.yaml) +defines a second backend, `order_sensitive`, that quietly moves 10% of the probability +mass to whichever option is shown first. Its answers never change, so golden accuracy +still looks perfect. The `option_order` property catches it: + +```bash +decguard fuzz examples/refund/decguard.yaml -b order_sensitive -p option_order -o fuzz.json +``` + +```text +DecGuard 0.1.1 · refund_request (choice) · FAIL + ... + Properties seed 42 + option_order 33 compared · 27 violating (81.8%) · flips 0 (0.0%) · max TV 0.100 + + Checks + PASS option_order.max_error_rate option_order.error_rate 0 <= 0 + FAIL option_order.max_violation_rate option_order.violation_rate 0.8182 violates <= 0 (per case: max_tv_distance 0.03) + PASS option_order.max_flip_rate option_order.flip_rate 0 <= 0 + + Property failures (10 of 27; replay with `decguard replay`) + option_order/r1/1 + tv_distance 0.1 > 0.03 + sent: options [review, refund, reject] · The blender arrived damaged and won't turn on. + option_order/r1/2 + tv_distance 0.1 > 0.03 + sent: options [review, reject, refund] · The blender arrived damaged and won't turn on. + minimal: options [reject, refund, review] · The blender arrived damaged and won't turn on. (tv_distance 0.1 > 0.03) + ... + +FAIL: at least one requirement does not hold +``` + +How to read it: + +- **33 compared** — 11 cases × 3 permutations. **27 violating** broke the per-case limit + `max_tv_distance: 0.03`; the violation rate 0.8182 exceeds the allowed `0`. +- **flips 0** — the selected answer never changed. Only the distribution check reveals + the bias, which is why limits are distribution-aware. +- **`option_order/r1/2`** — the id of one transformed case: property, case id, sample + index. Use it with `decguard replay --id`. +- **`sent`** — exactly what the backend received. **`minimal`** — the smallest version of + the transformation that still fails (see [minimization](#minimization)): a single swap of + the first two options is enough. + +The run exits `1`. + +## Seeds and reproducibility + +Transformations draw from a SHA-256-based random stream keyed by +`(seed, property, case id)`. As a result: + +- the same seed gives the same transformations on every platform and Python version; +- adding cases or properties does not change the transformations of the others; +- a failure seen in CI is reproduced locally by running the same command with the same seed. + +Pin `fuzz.seed` in the contract so every CI run checks the same inputs, and change it +deliberately (or pass `--seed`) to explore new ones. Reports record the seed, the fuzz and +property settings, the contract and dataset hashes, the backend metadata, and every +transformed case with its steps and any paraphrase provenance. + +## Minimization + +For `option_order`, `label_format`, `irrelevant_context`, `repeated_context` and +`whitespace`, a failing transformation is shrunk greedily until no smaller version still +fails, or until `fuzz.max_minimize_calls` backend calls (default 32 per failing case) are +spent: + +- inserted fragments and whitespace edits are removed one at a time; +- reformatted labels are reverted one at a time; +- permutations are moved toward the original order one adjacent swap at a time. + +The smallest failing version is stored as `reduced` next to the full one and printed as +`minimal`. Turn minimization off with `fuzz.minimize: false` or `--no-minimize`. + +## Replay + +`decguard replay` re-sends stored property failures to a backend and checks whether they +still fail: + +```bash +decguard replay fuzz.json # every stored failure +decguard replay fuzz.json --id option_order/r1/2 # one of them +decguard replay fuzz.json --backend candidate # against another backend +``` + +For each failure, replay: + +1. re-sends the original case and the stored transformed case, including the reduced one, + **exactly as stored**; +2. judges them again with the recorded limits; +3. regenerates the transformation from the seed and confirms it matches the stored one. + +```text +DecGuard 0.1.1 · replay fuzz.json · 1 case(s) + backend order_sensitive: mock + contract examples/refund/decguard.yaml + + option_order/r1/2: REPRODUCED · regenerated from seed: yes + transformed: options [review, reject, refund] · The blender arrived damaged and won't turn on. + tv_distance 0.1 > 0.03 + reduced: options [reject, refund, review] · The blender arrived damaged and won't turn on. + tv_distance 0.1 > 0.03 + +FAIL: at least one failure reproduced +``` + +By default replay uses the backend recorded in the report. Replay exits `1` if any failure still reproduces and `0` if none does, so it doubles as a +check that a fix works: replay the failures of the old model against the new one with +`--backend`. `--format json` prints a replay report (`replay_version` 0.1). Use +`--contract` to replay with a different contract than the one recorded in the report. + +## Re-judging without the model + +`decguard report fuzz.json --contract edited.yaml` re-applies edited property limits and +levels to the stored transformed cases without calling the backend. Transformations are +not regenerated. This is the fastest way to tune tolerances against real results. + +## In CI + +- Pin the seed; keep the JSON report as a build artifact. +- A failing CI run is then reproducible locally with `decguard replay report.json`. +- Use `level: warning` for properties you are still calibrating, and `requirement` once + they are stable. + +See [CI with GitHub Actions](./ci.md). diff --git a/docs/gates.md b/docs/gates.md new file mode 100644 index 0000000..8649bf5 --- /dev/null +++ b/docs/gates.md @@ -0,0 +1,98 @@ +--- +description: How requirements and warnings turn metrics into PASS, WARN and FAIL, the implicit gates, and the exit codes CI relies on. +--- + +# Requirements and warnings + +A **gate** is a threshold on one metric: `min_accuracy: 0.9` holds when accuracy is at +least 0.9. The contract sorts gates into two levels: + +- **`requirements`** — hard gates. A violated requirement **fails** the run. +- **`warnings`** — soft gates. A violated warning turns PASS into **WARN**, but the run + still succeeds. + +```yaml +requirements: + min_accuracy: 0.95 + max_ece: 0.05 + +warnings: + max_latency_p95_ms: 300 +``` + +Both sections accept the same gates. Gates that are not set are not checked. The same +pattern is used everywhere DecGuard gates something: + +| Where | Requirements | Warnings | Reference | +| --- | --- | --- | --- | +| Golden tests (`test`, `report`) | `requirements` | `warnings` | [gates](./contracts.md#requirements-and-warnings) | +| Properties (`fuzz`, `test --all`) | `level: requirement` (default) | `level: warning` | [properties](./properties.md#limits) | +| Regression (`diff`) | top-level `regression` gates | `regression.warnings` | [regression](./regression.md#gates) | +| Production (`check`) | `production.requirements`, `segment_requirements` | `production.warnings`, `segment_warnings` | [production](./production.md#metrics-drift-and-gates) | + +## Status + +Each gate produces one **check**. The run's status is the worst check: + +| Status | When | Exit code | +| --- | --- | --- | +| PASS | every check holds | `0` | +| WARN | every requirement holds; at least one warning does not | `0`, or `1` with `--fail-on-warn` | +| FAIL | at least one requirement does not hold | `1` | + +Every check line shows the gate, the measured value and the threshold, and says whether +it is a warning: + +```text + PASS min_accuracy accuracy 1 >= 0.9 + FAIL max_ece ece 0.2329 violates <= 0.2 + WARN min_coverage coverage 0.8333 violates >= 0.9 (warning) +``` + +## Implicit gates + +Some requirements apply even when the contract does not set them, because a run that +silently loses decisions has not demonstrated anything: + +- **`max_error_rate: 0`** for golden tests. Any case the backend could not decide fails the + run, unless you set `requirements.max_error_rate` to tolerate some. +- **`.max_error_rate: 0`** for every enabled property. A transformed case the + backend fails on (or whose original case failed) means the property was not checked. + This one cannot be relaxed. +- **`max_error_rate_increase: 0`** for regression diffs. A candidate that errors on cases + the baseline decided fails, even without `--contract`. + +## Missing values + +A gate whose metric **cannot be computed** does not hold: `min_accuracy` with no labeled +cases, `max_segment_accuracy_drop` with no segment large enough, drift gates without a +baseline. The check message says why (`accuracy could not be computed (no eligible +cases)`). At requirement level the run fails; at warning level it warns. + +## Exit codes + +Exit codes are part of DecGuard's public interface and are the same for every command: + +| Code | Meaning | +| --- | --- | +| `0` | PASS, or WARN without `--fail-on-warn` | +| `1` | a reliability gate failed (or WARN with `--fail-on-warn`); `replay`: a stored failure still reproduces | +| `2` | the run could not be evaluated: invalid contract, dataset or report; unknown backend; missing credential; unhealthy backend; every case failed; invalid command-line usage; or an internal error | + +Code `1` always means "the model broke the contract" and code `2` always means "the check +itself could not run". Unexpected exceptions map to `2`, never to `1`, so a crash can never +be mistaken for a model regression. Set `DECGUARD_DEBUG=1` to print a traceback for +internal errors. + +## Choosing thresholds + +- **Start from a measured baseline.** Run `decguard test` on the current model, then set + requirements slightly below what it achieves, so the gate catches regressions without + failing on noise. +- **Use warnings for trends.** Latency and coverage often belong in `warnings` first, then + move to `requirements` once they are stable. +- **Tighten on the main branch.** `--fail-on-warn` turns every warning into a failure where + you want the strictest gate. +- **Re-evaluate without re-running.** [`decguard report --contract`](./cli.md#decguard-report) + applies edited thresholds to a stored report instantly, so you can tune gates against + real results without calling the model again. diff --git a/docs/http.md b/docs/http.md new file mode 100644 index 0000000..dc43c29 --- /dev/null +++ b/docs/http.md @@ -0,0 +1,166 @@ +--- +description: The generic http backend, its settings, and the decguard.http/0.1 JSON protocol for connecting any decision endpoint. +--- + +# Generic HTTP + +The `http` backend calls any endpoint that speaks DecGuard's small JSON protocol, +`decguard.http/0.1`: one `POST` per decision, returning a probability per label. Use it +for your own model servers, or put a thin proxy in front of an endpoint with a different +shape. (System One APIs such as Kev and Jev have their own +[built-in backend](./systemone.md).) + +```yaml +backend: + provider: http + url: https://decisions.internal/v1/decide + health_url: https://decisions.internal/healthz + model: refund-v3 + timeout_s: 30 + max_retries: 2 + bearer_token_env: DECISIONS_TOKEN +``` + +## Settings + +| Setting | Default | Meaning | +| --- | --- | --- | +| `url` | required | absolute `http://` or `https://` decision endpoint | +| `model` | none | sent as `model` in every request | +| `health_url` | none | optional `GET` endpoint; any 2xx answer means healthy | +| `timeout_s` | `30` | per-request timeout in seconds (up to 600) | +| `max_retries` | `0` | retries (0–5) for connection failures and HTTP 502/503/504 only | +| `retry_backoff_s` | `0.5` | wait before the first retry, doubled after each one | +| `bearer_token_env` | none | environment variable sent as `Authorization: Bearer ` | +| `headers_from_env` | `{}` | header name → environment variable holding its value | +| `headers` | `{}` | static, **non-secret** headers | +| `label_map` | `{}` | backend label → contract label, for endpoints that name labels differently | + +```yaml +backend: + provider: http + url: https://decisions.internal/v1/decide + headers_from_env: {X-Tenant: DECISIONS_TENANT} + headers: {X-Client: decguard} + label_map: {"true": "yes", "false": "no"} +``` + +Security rules: + +- Credentials come from environment variables only. Literal `Authorization`, `X-Api-Key` + or `Api-Key` headers, credentials in the URL, and credential-like query parameters are + rejected when the contract is loaded. Set either `bearer_token_env` or an + `Authorization` entry in `headers_from_env`, not both. +- When environment-backed headers are configured, `health_url` must have the same origin + as `url`, so a healthcheck cannot forward credentials elsewhere. +- Redirects are not followed. Retries happen only when the request never reached the + server (connection failures) or the server answered 502/503/504. +- HTTP error bodies are not recorded (they are untrusted and can echo private input), and + reports store the URL without its query string. + +## Protocol `decguard.http/0.1` + +### Request + +```http +POST /v1/decide +Content-Type: application/json +``` + +```json +{ + "protocol": "decguard.http/0.1", + "case_id": "r1", + "model": "refund-v3", + "decision": {"name": "refund_request", "type": "choice", "labels": ["refund", "reject", "review"]}, + "input": "The blender arrived damaged." +} +``` + +| Field | Meaning | +| --- | --- | +| `protocol` | always `decguard.http/0.1` | +| `case_id` | dataset case id, or the id given to `run` / the SDK | +| `model` | the backend's `model` setting, or `null` | +| `decision.name`, `decision.type` | from the contract | +| `decision.labels` | the labels to present, in order, in the backend's names (after `label_map`) | +| `input` | the case input: a string or a JSON object | + +`decision.labels` is not always the contract order. During fuzzing, `option_order` and +`label_format` reorder or reformat them (`["review", "refund", "reject"]`, +`["REFUND", "REJECT", "REVIEW"]`, ...). Present them to the model as given, and key the +answer by the labels **exactly as sent**; DecGuard maps them back to the contract labels. + +### Response + +```json +{ + "probabilities": {"refund": 0.91, "reject": 0.02, "review": 0.07}, + "model": "refund-v3", + "model_version": "2026-09-01", + "metadata": {"tokens": 41} +} +``` + +| Field | Required | Meaning | +| --- | --- | --- | +| `probabilities` | yes | one probability per label sent: finite, non-negative, summing to 1 within `evaluation.probability_tolerance` | +| `model` | no | the model that answered (string) | +| `model_version` | no | its version (string) | +| `metadata` | no | an object stored with the result; common credential keys are redacted | + +Status handling: + +| Answer | Result for the case | +| --- | --- | +| `200` with a valid body | a `DecisionResult` | +| `200` with invalid JSON, missing or malformed `probabilities` | `invalid_response` | +| `502`, `503`, `504` (after retries) or connection failure | `unavailable` | +| any other non-2xx status | `backend_error` | +| no answer within `timeout_s` | `timeout` | + +## A minimal endpoint + +A complete endpoint using only the Python standard library. Replace `my_model` with a +call to your model: + +```python +import json +from http.server import BaseHTTPRequestHandler, HTTPServer + + +def my_model(text, options): + """Replace with your model: one probability per option, keyed exactly as given.""" + return {option: 1 / len(options) for option in options} + + +class Handler(BaseHTTPRequestHandler): + def do_POST(self): + request = json.loads(self.rfile.read(int(self.headers["Content-Length"]))) + labels = request["decision"]["labels"] # in the order and form to present + body = json.dumps( + { + "probabilities": my_model(request["input"], labels), + "model": "refund-rules", + "model_version": "2026-09-01", + } + ).encode() + self.send_response(200) + self.send_header("Content-Type", "application/json") + self.send_header("Content-Length", str(len(body))) + self.end_headers() + self.wfile.write(body) + + +HTTPServer(("127.0.0.1", 8080), Handler).serve_forever() +``` + +```yaml +backend: + provider: http + url: http://127.0.0.1:8080/decide + model: refund-rules +``` + +If your model runs in the same Python process as DecGuard, you can skip HTTP entirely with +a [custom backend](./custom-backends.md). diff --git a/docs/index.md b/docs/index.md new file mode 100644 index 0000000..09e3ffe --- /dev/null +++ b/docs/index.md @@ -0,0 +1,89 @@ +--- +layout: home +title: DecGuard +titleTemplate: Reliability testing for probabilistic AI decisions + +hero: + name: DecGuard + text: Reliability testing for probabilistic AI decisions. + tagline: Define decision contracts, test behavioral invariants, detect regressions, and gate unreliable model versions before production. + actions: + - theme: brand + text: Get Started + link: /quickstart + - theme: alt + text: View on GitHub + link: https://github.com/suffro/decguard + +features: + - title: Decision Contracts + details: Define typed Choice, Noul and Score decisions once, with their gates, properties and policies, in one versioned YAML file. + link: /decisions + linkText: Decisions and results + - title: Reliability Testing + details: Measure accuracy, calibration, confidence, coverage, latency and error rate on a golden dataset, with PASS / WARN / FAIL gates. + link: /metrics + linkText: Metrics + - title: Metamorphic Fuzzing + details: Stress option order, label format, irrelevant context, whitespace, paraphrases, noul inversion and score monotonicity. Seeded, minimized, replayable. + link: /properties + linkText: Properties + - title: Regression Gates + details: Compare two model or backend versions case by case and fail CI on answer flips, calibration loss, new errors or segment regressions. + link: /regression + linkText: Regression diffs + - title: Production Checks + details: Analyze decision records your application already collected, offline, with baseline drift and per-segment gates. + link: /production + linkText: Production checks + - title: Backend Neutral + details: Run the same contract against mock, HTTP, System One (Kev, Jev through OpenRouter) or your own plugin backends. + link: /backends + linkText: Backends +--- + +## One contract, every check + +A Decision Contract describes the decision and what "reliable" means for it. Every +command reads the same file. + +```yaml +schema_version: "0.1" + +decision: + name: refund_request + description: How should this customer refund request be handled? + type: choice + options: [refund, reject, review] + +backend: # Jev through OpenRouter; or mock, http, a plugin + provider: systemone + model: typesafe/jev-1.13 + url: https://openrouter.ai/api/alpha/decisions + bearer_token_env: OPENROUTER_API_KEY + +dataset: cases.jsonl + +evaluation: + probability_tolerance: 0.02 # Jev rounds probabilities to two decimals + +requirements: # violated -> FAIL, exit 1 + min_accuracy: 0.9 + max_ece: 0.1 + +properties: # decguard fuzz / decguard test --all + option_order: + max_tv_distance: 0.03 +``` + +| Command | Question it answers | +| --- | --- | +| [`decguard test`](./cli.md#decguard-test) | Does the model meet its accuracy, calibration and coverage gates on the golden dataset? | +| [`decguard fuzz`](./cli.md#decguard-fuzz) | Does the answer stay stable when irrelevant details of the input change? | +| [`decguard diff`](./cli.md#decguard-diff) | Is the candidate version worse than the one it replaces? | +| [`decguard check`](./cli.md#decguard-check) | Is the deployed model still calibrated on real traffic, in every segment? | +| [`decguard run`](./cli.md#decguard-run) | Given this confidence, should the application accept, abstain, fall back or ask a human? | + +Exit codes are the interface: `0` pass, `1` a reliability gate failed, `2` the run could not +be evaluated. Start with the [quickstart](./quickstart.md), or read +[what DecGuard is](./introduction.md) first. diff --git a/docs/installation.md b/docs/installation.md new file mode 100644 index 0000000..2ff9779 --- /dev/null +++ b/docs/installation.md @@ -0,0 +1,92 @@ +--- +description: Install the DecGuard CLI and Python library from PyPI or from source. +--- + +# Installation + +DecGuard is a Python package with a command-line interface. It needs **Python 3.11 or +newer** and is tested on Linux and macOS with Python 3.11, 3.12 and 3.13. + +## From PyPI + +```bash +pip install decguard +``` + +Or install the CLI as an isolated tool with [uv](https://docs.astral.sh/uv/): + +```bash +uv tool install decguard +``` + +Check the installation: + +```bash +decguard --version +``` + +```text +decguard 0.1.1 +``` + +`decguard` with no arguments, or `decguard --help`, lists the commands and their +options. Running `python -m decguard` is equivalent. + +## Dependencies + +DecGuard has four runtime dependencies: `pydantic`, `typer`, `httpx` and `PyYAML`. There +are no optional extras: the mock, HTTP and System One backends, metrics, fuzzing, +regression diffs, production checks and the SDK all ship in the base package. No GPU, +model weights, database or service account is required. + +Backends that serve real models run wherever you run them. DecGuard only needs network +access to their URL and, when they require one, a credential in an +[environment variable](./backends.md#credentials). + +## As a library + +The same package provides the Python API used by the CLI: + +```python +from decguard import DecGuard # runtime policy SDK +from decguard.engine import run_test # golden tests and fuzzing from Python +from decguard.backends import CallableBackend # wrap a Python function as a backend +``` + +See [Policies and Python SDK](./policy.md) and [Custom backends](./custom-backends.md). + +## From source + +To try unreleased changes or contribute: + +```bash +git clone https://github.com/suffro/decguard +cd decguard +uv tool install . # or, inside a virtual environment: pip install . +``` + +For development (tests, linting, type checks), use the locked environment: + +```bash +uv sync +uv run pytest +``` + +The repository's +[CONTRIBUTING.md](https://github.com/suffro/decguard/blob/main/CONTRIBUTING.md) describes +the full workflow. + +## In CI + +Install from PyPI in the job, pinning the version you tested with: + +```yaml +- uses: actions/setup-python@v5 + with: + python-version: "3.12" +- run: pip install decguard==0.1.1 +``` + +See [CI with GitHub Actions](./ci.md) for complete workflows. + +Next: [write your first contract](./quickstart.md). diff --git a/docs/introduction.md b/docs/introduction.md new file mode 100644 index 0000000..7ca500b --- /dev/null +++ b/docs/introduction.md @@ -0,0 +1,91 @@ +--- +description: What DecGuard is, the problem it solves, and where it fits next to your decision models. +--- + +# What is DecGuard? + +DecGuard tests, verifies and gates **probabilistic AI decision models**: models that +answer a typed question about an input — pick one of several options, say yes or no, +assign a level on a scale — and return a probability for every possible answer. + +You describe the decision once in a small YAML file, the **Decision Contract**. DecGuard +runs a dataset through any backend that serves the model, normalizes every answer into the +same shape, and produces one reproducible **reliability report** with PASS / WARN / FAIL +gates and CI-friendly exit codes. + +```text +Decision Contract ─┬─> backend (mock, HTTP, System One, plugin) ─> DecisionResult ─> report ─> exit 0 / 1 / 2 +golden dataset ────┘ +``` + +## Why test decisions differently? + +A decision model is usually wired to an action: refund the customer, escalate the +ticket, block the message. Its probabilities matter as much as its top answer, because +they decide when the application trusts the model and when it asks a human. Accuracy on a +test set does not tell you: + +- whether a confidence of 0.9 really means "right nine times out of ten" + ([calibration](./metrics.md#calibration)); +- how many decisions clear your confidence threshold, and how accurate those are + ([coverage and selective accuracy](./metrics.md#coverage-and-selective-accuracy)); +- whether the answer changes when the options are listed in a different order, the labels + are formatted differently, or an irrelevant sentence is added + ([metamorphic properties](./properties.md)); +- whether a new model version flips answers or loses calibration compared with the one in + production ([regression diffs](./regression.md)); +- whether the deployed model is still calibrated on real traffic, in every locale or + customer tier ([production checks](./production.md)). + +DecGuard answers each of these with a deterministic metric and an explicit gate, so a +reliability regression fails a build the same way a broken unit test does. + +## What you get + +| Capability | Command | Page | +| --- | --- | --- | +| Validate a contract, its backends and dataset offline | `decguard validate` | [CLI](./cli.md#decguard-validate) | +| Golden-dataset metrics and gates | `decguard test` | [Golden tests and metrics](./metrics.md) | +| Metamorphic fuzzing with seeded, minimized, replayable failures | `decguard fuzz`, `test --all`, `replay` | [Properties](./properties.md), [Fuzzing](./fuzzing.md) | +| Baseline vs candidate comparison with regression gates | `decguard diff` | [Regression diffs](./regression.md) | +| Re-apply edited gates to stored results without re-running the model | `decguard report --contract` | [Reports](./reports.md) | +| Offline checks of collected production records, drift and segments | `decguard check` | [Production checks](./production.md) | +| Deterministic runtime routing: accept, abstain, fallback, human review | `decguard run`, Python SDK | [Policies and SDK](./policy.md) | + +## Three decision types + +| Type | Question | Example | +| --- | --- | --- | +| `choice` | Which of these options? | `refund`, `reject` or `review` for a refund request | +| `noul` | Yes or no? | Is this message spam? | +| `score` | Which level on an ordered scale? | Incident severity from 1 to 4 | + +See [Decisions and results](./decisions.md) for how each type is declared and normalized. + +## Backend neutral + +DecGuard does not serve models. It calls them through thin adapters that all produce the +same validated `DecisionResult`, so one contract and one dataset can run against: + +- the offline, deterministic [`mock`](./backends.md#mock) backend (tests, demos, CI); +- any endpoint speaking DecGuard's small [HTTP protocol](./http.md); +- [System One](./systemone.md) decision APIs: a self-hosted **Kev** server, TypeSafe's + **Jev** through OpenRouter, or a compatible TypeSafe endpoint. This backend has been + validated with real Kev and real Jev inference, on Linux and macOS; +- your own Python code, in-process or packaged as a [plugin](./custom-backends.md). + +## What DecGuard is not + +DecGuard is not a model, a model server, a generic LLM evaluation framework, a red-teaming +suite, an observability platform or a hosted service. It needs no server, database, GPU, +account or telemetry collector. It owns the decision-specific reliability layer — contract, +normalized results, metrics, comparisons, gates and reproducible reports — and leaves +everything else to the tools you already use. See [Architecture](./architecture.md). + +## Status + +DecGuard `0.1.1` is on PyPI. The contract format (`schema_version: "0.1"`) and the report +formats are versioned independently of the package; changes to their meaning come with a +version bump and a changelog entry. The project is Apache-2.0 licensed. + +Next: [install DecGuard](./installation.md), then follow the [quickstart](./quickstart.md). diff --git a/docs/metrics.md b/docs/metrics.md new file mode 100644 index 0000000..297b8d2 --- /dev/null +++ b/docs/metrics.md @@ -0,0 +1,133 @@ +--- +description: What decguard test does, and the exact definition of every golden-test metric - accuracy, calibration, confidence coverage, latency and errors. +--- + +# Golden tests and metrics + +`decguard test` runs the [golden dataset](./datasets.md) through one backend, computes +reliability metrics, applies the contract's [requirements and warnings](./gates.md), and +produces one [report](./reports.md). + +```bash +decguard test decguard.yaml # default backend +decguard test decguard.yaml --backend candidate # a named backend +decguard test decguard.yaml --output report.json # keep the full report +decguard test decguard.yaml --all # plus metamorphic properties +``` + +## What a run does + +1. Loads and validates the contract and the dataset. Any problem stops the run with exit + code `2`. +2. Calls the backend's healthcheck (skip it with `--no-healthcheck`). An `unhealthy` + backend stops the run with exit code `2`; `unknown` (no health endpoint) continues. +3. Sends every case to the backend, up to `evaluation.max_concurrency` at a time (default + 4, `--max-concurrency` overrides). Results keep dataset order whatever the concurrency. +4. Validates each answer into a [`DecisionResult`](./decisions.md#the-normalized-decisionresult), + or records a typed error for the case. +5. Computes the metrics below, evaluates the gates, and prints the report. + +Metrics are deterministic: they are computed in dataset order with exact floating-point +summation, so the same results always give bit-identical numbers. That is what lets +[`decguard report --contract`](./cli.md#decguard-report) re-apply edited gates to a stored +report without calling the model. + +## Which cases each metric uses + +- **Classification and calibration** metrics use **labeled** cases the backend decided. +- **Coverage** and **latency** use all decided cases. +- **Errors** are counted over all cases and never dropped. + +## Accuracy + +- **`accuracy`** — share of labeled cases whose selected label is the expected one. +- **`macro_f1`** — unweighted mean F1 over the labels that appear as expected or + predicted (scikit-learn's `average="macro"`; undefined precision or recall count as 0). + The report's `per_label` has precision, recall, F1 and support for each label. +- **`ordinal_mae`** (score decisions only) — mean |predicted level − expected level| in + scale steps. Predicting 3 when the answer is 4 costs 1; predicting 1 costs 3. + +## Calibration + +A model is **calibrated** when its confidence matches its accuracy: of all decisions made +at 80% confidence, about 80% are right. Calibration is reported separately from accuracy +on purpose. A model can be accurate and badly calibrated, and that matters as soon as its +confidence drives an action, such as a [policy](./policy.md) that automates only +high-confidence decisions. + +- **`ece`** — expected calibration error over top-label confidence, with + `evaluation.calibration_bins` equal-width bins (default 10). Bin *i* covers + (i/n, (i+1)/n]; the first bin also includes 0. ECE is the case-weighted mean + |accuracy − mean confidence| over bins. The report's `reliability` lists count, mean + confidence and accuracy per bin, which is the data behind a reliability diagram. +- **`brier`** — multi-class Brier score: the mean of Σₖ (pₖ − yₖ)² over labels, from 0 + (perfect) to 2. For two labels this is twice the binary `(p − y)²` form. +- **`nll`** — mean negative log-likelihood, −ln p(expected). p is floored at 1e-15, so a + confident miss adds about 34.5 instead of infinity. + +ECE is easy to read; Brier and NLL also reward sharp, correct probabilities and punish +confident mistakes harder. Gate on ECE first, then add `max_brier` or `max_nll` when you +need them. + +## Coverage and selective accuracy + +Set `evaluation.confidence_threshold` to the confidence at which your application would +act on a decision automatically. Decisions below it are **abstentions**. + +- **`coverage`** — share of decided cases with confidence ≥ the threshold. +- **`abstention_rate`** — share below it (1 − coverage). +- **`selective_accuracy`** — accuracy of labeled cases at or above the threshold: how good + the decisions you would automate are. + +The gates `min_coverage`, `max_abstention_rate` and `min_selective_accuracy` require the +threshold to be set; contract validation rejects them otherwise. For routing that acts on +these thresholds at runtime, see [Policies](./policy.md). + +## Latency + +Time spent in the backend call, in milliseconds: mean, p50, p90, p95, p99 (linear +interpolation, NumPy's default) and max. `max_latency_p95_ms` gates the 95th +percentile. Latency includes network time for remote backends and depends on +`max_concurrency`, so compare latency between runs made with the same settings. + +## Errors + +- **`error_rate`** — errored cases / all cases, with `errors_by_kind` (`timeout`, + `unavailable`, `invalid_response`, `backend_error`, `exception`). + +`max_error_rate` is an [implicit requirement of 0](./gates.md#implicit-gates). A backend +that fails on every case stops the run with exit code `2`. + +## Gate reference + +| Gate | Metric | Holds when | +| --- | --- | --- | +| `min_accuracy` | accuracy | value ≥ threshold | +| `min_macro_f1` | macro-F1 | ≥ | +| `max_ordinal_mae` | ordinal MAE (score decisions only) | ≤ | +| `max_ece` | expected calibration error | ≤ | +| `max_brier` | multi-class Brier score (0–2) | ≤ | +| `max_nll` | negative log-likelihood | ≤ | +| `min_coverage` | share of decisions at/above the threshold | ≥ | +| `max_abstention_rate` | share below the threshold | ≤ | +| `min_selective_accuracy` | accuracy at/above the threshold | ≥ | +| `max_error_rate` | share of cases the backend failed (defaults to 0) | ≤ | +| `max_latency_p95_ms` | 95th percentile latency | ≤ | + +All gates are accepted under both `requirements` and `warnings`. See the +[contract reference](./contracts.md#requirements-and-warnings) for validation rules. + +## Reading failures + +When gates fail, the terminal report lists failing examples — wrong answers and errors — +with their inputs: + +```text + Failures (1 of 1) + r7 expected review, got refund (0.96) + input: The lid was damaged, but I just want a replacement lid. +``` + +The JSON report keeps every case in `results`, including the full distribution, so you can +inspect confident mistakes, sort by confidence, or feed the report to +[`decguard diff`](./regression.md). diff --git a/docs/policy.md b/docs/policy.md index 77e0950..1ade8ea 100644 --- a/docs/policy.md +++ b/docs/policy.md @@ -1,7 +1,16 @@ -# Runtime cascade policy and Python SDK +--- +description: Deterministic runtime routing on decision confidence - accept, abstain, fallback and human_review - from the CLI and the Python SDK. +--- -A policy is an explicit ordered route table. The first matching route wins and the final -route must be unconditional, which makes evaluation total and deterministic. +# Policies and Python SDK + +Testing tells you how reliable a model is at each confidence level. A **policy** turns +that into a runtime rule: which decisions the application may act on automatically, and +which it should abstain from, escalate to a stronger model, or send to a person. + +A policy is an explicit, ordered route table in the contract. The first matching route +wins and the final route must be unconditional, so every decision gets exactly one action, +deterministically. ```yaml backends: @@ -19,17 +28,86 @@ policy: - action: human_review ``` -Supported actions are `accept`, `abstain`, `fallback` and `human_review`. A fallback route -must name a backend declared by the same contract. Conditional `confidence_gte` thresholds -must be strictly descending; ambiguous/unreachable routes fail contract validation. +## Actions + +| Action | Directive to the application | +| --- | --- | +| `accept` | act on the model's `selected` label | +| `abstain` | do not decide; record the abstention | +| `fallback` | ask the backend named in `backend` (a backend declared in the same contract) | +| `human_review` | send the case to a person | + +::: warning Fallback is a directive, not an automatic call +DecGuard v0.1 **does not** invoke the fallback backend. It returns +`action: fallback` and the backend's name; your application decides whether and how to +call it. This keeps retries, cost limits and side effects under the application's control +and keeps the policy engine a pure function rather than an orchestration service. +::: + +## Validation rules -Test one route from the CLI: +The contract is rejected when the policy is ambiguous or incomplete: + +- at least one route; the **last** route has no `when`, and only the last; +- `confidence_gte` thresholds (0–1) are **strictly descending**, so no route is + unreachable; +- `fallback` routes must name a `backend` that the contract declares (`default` or a key of + `backends`); other actions must not set `backend`. + +`confidence_gte` is the only condition in schema 0.1. It compares against the +[normalized confidence](./decisions.md#the-normalized-decisionresult): the selected label's +probability. + +## From the CLI + +`decguard run` makes one decision with the contract's backend and applies the policy: ```bash -decguard run decguard.yaml "The blender arrived damaged" --format json +decguard run examples/refund/decguard.yaml "The blender arrived damaged" +decguard run examples/refund/decguard.yaml "The parcel never arrived" +decguard run examples/refund/decguard.yaml "Can someone call me?" ``` -Embed the identical engine in Python: +```text +accept · selected refund · confidence 0.960 · route 0 +fallback -> strong_model · selected refund · confidence 0.900 · route 1 +human_review · selected review · confidence 0.650 · route 2 +``` + +`--format json` prints the full result: + +```json +{ + "result": { + "case_id": "runtime", + "decision": "refund_request", + "decision_type": "choice", + "labels": ["refund", "reject", "review"], + "probabilities": {"refund": 0.9, "reject": 0.02, "review": 0.08}, + "selected": "refund", + "confidence": 0.9, + "backend": "default", + "provider": "mock", + "model": "refund-mock-v1", + "model_version": null, + "latency_ms": 0.0069, + "metadata": {} + }, + "action": "fallback", + "fallback_backend": "strong_model", + "route_index": 1 +} +``` + +Options: `--backend NAME` to decide with another configured backend, `--input-json` to +pass an object input (`decguard run decguard.yaml '{"text": "..."}' --input-json`), +`--id` to set the case/correlation id (default `runtime`), and `--no-healthcheck`. `run` +exits `0` whatever the action; it exits `2` if the contract has no `policy` or the backend +fails. + +## Python SDK + +The SDK embeds the same engine in your application: ```python from decguard import DecGuard @@ -38,19 +116,39 @@ with DecGuard.from_contract("decguard.yaml") as guard: decision = guard.decide("The blender arrived damaged", case_id="request-42") if decision.action == "accept": - use(decision.result) + apply(decision.result.selected) elif decision.action == "fallback": - enqueue_for(decision.fallback_backend) + enqueue_for(decision.fallback_backend, decision.result) elif decision.action == "abstain": - record_abstention() -else: - send_to_human_review() + record_abstention(decision.result) +else: # "human_review" + send_to_human_review(decision.result) ``` -`fallback` is an explicit directive; DecGuard v0.1 does not automatically invoke the next -backend. This keeps retry/cost/side-effect control in the embedding application and avoids -turning the policy engine into an implicit optimizer or orchestration service. +- **`DecGuard.from_contract(contract, *, backend=None, check_health=False)`** loads and + validates the contract. `contract` is a path; `backend` is the name of a configured + backend, or a `DecisionBackend` instance such as a + [`CallableBackend`](./custom-backends.md#python-callable) for an in-process model. It + raises if the contract has no `policy`. +- **`guard.decide(input, *, case_id="sdk")`** calls the backend, validates the answer with + the same rules as `decguard test` (including `evaluation.probability_tolerance`), and + returns a `PolicyDecision`. +- **`PolicyDecision`** has `result` (the [`DecisionResult`](./decisions.md#the-normalized-decisionresult)), + `action`, `fallback_backend` (set for `fallback` only) and `route_index`. +- Use `DecGuard` as a context manager, or call `close()`, to release the backend's + connections. A backend instance you pass in is not closed for you. + +Backend failures raise `decguard.errors.BackendError` subclasses (`BackendTimeout`, +`BackendUnavailable`, `InvalidResponse`); handle them like any other failed dependency call. + +## Choosing thresholds + +Pick route thresholds from measured reliability, not intuition: -`DecGuard.from_contract(..., backend="name")` selects a configured backend. Library users -can instead pass a `DecisionBackend` instance such as `CallableBackend` for an in-process -provider. Backend results still go through the same probability validation used by tests. +1. Run [`decguard test`](./metrics.md) with `evaluation.confidence_threshold` set to a + candidate threshold, and read coverage and selective accuracy. +2. Check the report's calibration bins: a threshold is only meaningful if confidence is + calibrated around it. +3. After deployment, write the policy's `action` into your [production + records](./production.md#record-schema) and gate `max_fallback_rate`, + `max_abstention_rate` and per-segment accuracy with `decguard check`. diff --git a/docs/production.md b/docs/production.md index d0a3ba3..d580e94 100644 --- a/docs/production.md +++ b/docs/production.md @@ -1,8 +1,16 @@ -# Post-deployment checks +--- +description: Analyze decisions your application already collected, offline - calibration and reliability after deployment, baseline drift, metadata segments and production gates. +--- -`decguard check` analyzes decisions already collected by your application. It is an -offline batch command: it does not call a model, upload data, start a daemon or require a -DecGuard service. +# Production checks + +A model that passed its golden tests can still drift after deployment: traffic changes, +a new locale arrives, an upstream model version changes silently. `decguard check` +analyzes decisions your application **already collected** and applies post-deployment +gates to them. + +It is an offline batch command. It does not call a model, upload data, start a daemon, +ingest telemetry or require a DecGuard service. You export records; DecGuard reads a file. ```bash decguard check decguard.yaml \ @@ -11,15 +19,82 @@ decguard check decguard.yaml \ --output production-report.json ``` -The baseline may be another production JSONL/JSON dataset or a JSON report previously -written by `decguard check`. Exit codes are the usual `0` pass/warn, `1` failed gate and -`2` invalid configuration, dataset or report. +| Option | Meaning | +| --- | --- | +| `--dataset`, `-d FILE` | collected production records, `.jsonl` or `.json` (required) | +| `--baseline FILE` | a previous production dataset, or a JSON report from an earlier `decguard check` | +| `--output`, `-o FILE` | write the production report (`check_version` 0.1) | +| `--format json` | print the report as JSON | +| `--fail-on-warn` | exit `1` on warnings | + +Exit codes are the usual `0` pass/warn, `1` failed gate, `2` invalid configuration, +records or baseline. + +## Example + +The refund example ships two small batches. The first is healthy: + +```bash +decguard check examples/refund/decguard.yaml --dataset examples/refund/production.jsonl +``` + +```text +DecGuard 0.1.1 · refund_request (choice) · PASS + + Records 4 total · 4 with correctness · 4 with label outcome + Outcomes accuracy 1.000 · error 0.000 + Calibration ECE 0.160 · Brier 0.060 · NLL 0.185 + Confidence mean 0.840 · p50 0.875 · p95 0.951 + Threshold confidence >= 0.8: coverage 0.750 + Routing abstain 0.000 · fallback 0.500 · review 0.250 + + Segments + PASS locale=en · 2 records · accuracy 1.000 · ECE 0.070 + PASS locale=it · 2 records · accuracy 1.000 · ECE 0.250 + + Checks + PASS min_accuracy accuracy 1 >= 0.9 + PASS max_ece ece 0.16 <= 0.25 + PASS min_threshold_coverage threshold_coverage 0.75 >= 0.5 + PASS max_fallback_rate fallback_rate 0.5 <= 0.6 + PASS segment[locale=en].min_accuracy segment[locale=en].accuracy 1 >= 0.8 + PASS segment[locale=it].min_accuracy segment[locale=it].accuracy 1 >= 0.8 + +PASS: all post-deployment gates hold +``` + +The second is intentionally overconfident and wrong. Checked against the first as +baseline: -## Production record schema +```bash +decguard check examples/refund/decguard.yaml \ + --dataset examples/refund/production-drift.jsonl \ + --baseline examples/refund/production.jsonl +``` + +```text + Outcomes accuracy 0.000 · error 1.000 + Calibration ECE 0.975 · Brier 1.924 · NLL 4.432 + Confidence mean 0.975 · p50 0.975 · p95 0.989 + ... + Baseline dataset · 4 records + Drift confidence TV 0.750 · mean delta 0.135 · accuracy drop 1.000 · ECE increase 0.815 + + Segments + FAIL locale=en · 2 records · accuracy 0.000 · ECE 0.985 + FAIL locale=it · 2 records · accuracy 0.000 · ECE 0.965 + ... +FAIL: at least one post-deployment requirement does not hold +``` + +Confidence went **up** while accuracy collapsed: the pattern that makes an uncalibrated +model dangerous, because a confidence-based policy would automate every one of these +decisions. The run exits `1`. -JSONL is recommended (one object per line); a JSON list is also accepted. Required fields -are `decision`, `probabilities` and `selected`. `id` defaults to `record-` and -`confidence` defaults to the selected label's probability. All other fields are optional. +## Record schema + +Production records are backend-neutral: any application can write them, whatever model +it calls. JSONL (one object per line) is recommended; a JSON list is also accepted. ```json { @@ -43,35 +118,53 @@ are `decision`, `probabilities` and `selected`. `id` defaults to `record-` an } ``` -- Probabilities must contain exactly the contract labels, be finite/non-negative and sum - to 1 within `evaluation.probability_tolerance`. `selected` and optional `confidence` - must agree with their argmax; malformed records are rejected, never repaired or skipped. -- `outcome` is the observed label. When present, DecGuard derives `outcome_correct`; if - both are supplied they must agree. `outcome_correct` alone supports accuracy and ECE - when the actual label cannot be retained. Brier and NLL require `outcome`. -- `action` is `accept`, `abstain`, `fallback` or `human_review`. Missing routing and outcome - fields remain unknown; DecGuard does not count them as false. -- `latency_ms`, `cost` and `fallback` are retained so future tooling can study cascades - without changing this public format. v0.1 does not optimize policies. - -The complete runnable sample is -[`examples/refund/production.jsonl`](../examples/refund/production.jsonl). -An intentionally overconfident and incorrect comparison batch is -[`examples/refund/production-drift.jsonl`](../examples/refund/production-drift.jsonl). +| Field | Required | Meaning | +| --- | --- | --- | +| `decision` | yes | must equal the contract's `decision.name` | +| `probabilities` | yes | exactly the contract labels; finite, non-negative, summing to 1 within `evaluation.probability_tolerance` | +| `selected` | yes | must be the most probable label (ties to the first in contract order) | +| `id` | no | defaults to `record-` | +| `confidence` | no | defaults to the selected label's probability; must agree with it when given | +| `timestamp` | no | ISO 8601 with a timezone | +| `input` | no | the decision input, string or object | +| `backend`, `model`, `model_version` | no | provenance | +| `outcome` | no | the label actually observed later; DecGuard derives `outcome_correct` from it | +| `outcome_correct` | no | whether the decision turned out right, when the actual label cannot be kept; must agree with `outcome` when both are given | +| `latency_ms`, `cost` | no | non-negative numbers, summarized in the report | +| `action` | no | `accept`, `abstain`, `fallback` or `human_review` — the [policy](./policy.md) route taken | +| `fallback` | no | whether a fallback was used; must agree with `action` when both are given | +| `metadata` | no | an object; scalar values can be used as [segments](#metadata-segments) | + +Records are validated strictly, like backend answers: a malformed record stops the check +with exit code `2`, naming the line. Nothing is repaired or skipped. Missing optional +fields stay **unknown**: a record without `outcome` is not counted as wrong, and one +without `action` is not counted as accepted. + +`latency_ms`, `cost` and `fallback` are kept in the format so cascade behavior can be +studied without changing it; v0.1 reports them but does not optimize policies. + +The runnable samples are +[`production.jsonl`](https://github.com/suffro/decguard/blob/main/examples/refund/production.jsonl) +and +[`production-drift.jsonl`](https://github.com/suffro/decguard/blob/main/examples/refund/production-drift.jsonl). ## Metrics, drift and gates The report includes: -- accuracy and error rate over records with known correctness; -- ECE over known correctness, plus Brier/NLL where label outcomes exist; -- coverage at `evaluation.confidence_threshold`; -- abstention, fallback and human-review rates over records with routing information; -- confidence histogram/mean/percentiles, latency and cost summaries; -- against a baseline: confidence-histogram total variation, confidence mean delta, - accuracy drop and ECE increase. +- **outcomes** — accuracy and error rate (1 − accuracy) over records with known + correctness; +- **calibration** — ECE over records with known correctness; Brier and NLL where the + label `outcome` is known; +- **confidence** — histogram, mean and percentiles; +- **threshold coverage** — share at or above `evaluation.confidence_threshold`; +- **routing** — abstention, fallback and human-review rates over records with routing + information; +- **latency and cost** summaries; +- against a **baseline**: confidence-histogram total variation distance, mean confidence + delta, accuracy drop and ECE increase. -Configure gates separately from pre-deployment golden gates: +Production gates are configured separately from pre-deployment golden gates: ```yaml evaluation: @@ -97,7 +190,35 @@ production: max_fallback_rate: 0.3 ``` -Each configured metadata key creates one simple group per scalar value (including a -`` group). Segment gates are evaluated independently for every group, so a -localized failure cannot be hidden by the aggregate. Objects and arrays are rejected as -segment values; this is deliberately not a general analytics query language. +| Gate | Aggregate | Per segment | Value | +| --- | --- | --- | --- | +| `min_accuracy` | ✓ | ✓ | accuracy over records with known correctness | +| `max_error_rate` | ✓ | ✓ | 1 − accuracy | +| `max_ece` | ✓ | ✓ | expected calibration error | +| `max_brier` | ✓ | ✓ | Brier score (needs `outcome`) | +| `max_nll` | ✓ | ✓ | negative log-likelihood (needs `outcome`) | +| `min_threshold_coverage` | ✓ | ✓ | share at/above `evaluation.confidence_threshold` (required) | +| `max_abstention_rate` | ✓ | ✓ | share of routed records with `action: abstain` | +| `max_fallback_rate` | ✓ | ✓ | share of records that used a fallback | +| `max_confidence_tv_distance` | ✓ | | confidence-histogram TV distance to the baseline | +| `max_accuracy_drop` | ✓ | | baseline − current accuracy | +| `max_ece_increase` | ✓ | | current − baseline ECE | + +Aggregate gates go under `requirements` / `warnings`; per-segment gates under +`segment_requirements` / `segment_warnings`. A drift gate without `--baseline` cannot be +computed and therefore does not hold. + +## Metadata segments + +Each key in `production.segments` creates one group per scalar value found in record +`metadata`, plus a `` group for records without the key. Segment gates are +evaluated **independently for every group**, so a failure in one locale cannot be hidden +by a healthy aggregate. Objects and arrays are rejected as segment values; this is +deliberately not a general analytics query language. + +## Running it on a schedule + +`decguard check` fits wherever you already export data: a nightly job, a cron entry, or a +scheduled GitHub Actions workflow that downloads yesterday's records and keeps the report +as an artifact. Pass the previous run's report as `--baseline` to gate on drift. See +[CI](./ci.md#scheduled-production-check). diff --git a/docs/properties.md b/docs/properties.md index 71ff1fe..8e98803 100644 --- a/docs/properties.md +++ b/docs/properties.md @@ -1,179 +1,272 @@ -# Metamorphic properties and fuzzing +--- +description: Metamorphic properties - option order, label format, irrelevant and repeated context, whitespace, paraphrase, noul inversion and score monotonicity - with their limits and defaults. +--- -A property says how a decision must behave when the input changes in a controlled way. Most -properties are **invariances**: an irrelevant sentence, a different option order or extra -whitespace must not change the answer. Two are not: `inversion` expects the answer to -**swap**, and `monotonic` expects a score to **move in one direction**. +# Metamorphic properties -```bash -decguard fuzz decguard.yaml # properties only -decguard test decguard.yaml --all # golden gates + properties -decguard fuzz decguard.yaml -p option_order --seed 7 --output fuzz.json -``` +Golden tests check answers to fixed inputs. A **metamorphic property** checks how the +answer behaves when the input changes in a controlled way, so it needs no extra labels and +catches failures a golden set cannot see. + +Most properties are **invariances**: listing the options in a different order, formatting +the labels differently, adding an irrelevant sentence or extra whitespace must not change +the decision. Two are not: [`inversion`](#inversion-noul) expects a yes/no answer to +**swap**, and [`monotonic`](#monotonic-score) expects a score to **move in one direction**. + +Properties run with `decguard fuzz` (properties only) or `decguard test --all` (golden +gates and properties in one report). How runs are seeded, minimized and replayed is +described in [Fuzzing and replay](./fuzzing.md). + +## How a property is checked -Each dataset case is first decided as is (the *original*). Every property then derives -transformed cases from it, sends them to the same backend and compares the two -probability distributions. Everything is recorded in the report: the transformation steps, -what was sent, the result (mapped back to contract labels), the comparison and the -violations. +1. Each dataset case is decided as is: the **original**. +2. The property derives one or more **transformed** cases from it, deterministically from + the [seed](./fuzzing.md#seeds-and-reproducibility). +3. Each transformed case is sent to the same backend. +4. The two probability distributions are compared, and the comparison is checked against + the property's limits. -## Contract sections +Everything is recorded in the report: the transformation steps, what was sent, the result +(mapped back to contract labels), the comparison and any violation. + +**Option presentation is separate from the input.** `option_order` and `label_format` do +not edit the input text; they change how the labels are presented to the model. Backends +receive the labels in the order and form shown (`request.decision.labels` in Python, +`decision.labels` over [HTTP](./http.md), the question's `criteria` for +[System One](./systemone.md)) and answer with probabilities keyed by those labels. DecGuard +maps them back to the contract labels before comparing. + +## Configuring properties ```yaml fuzz: - seed: 0 # every transformation derives from (seed, property, case id) - text_field: text # object inputs: the key text transformations edit - minimize: true # shrink failing transformations - max_minimize_calls: 32 # backend calls per failing case while shrinking + seed: 42 properties: option_order: samples: 3 - max_tv_distance: 0.03 # per transformed case - max_flip_rate: 0 # share of transformed cases whose answer changes + max_tv_distance: 0.03 # per transformed case + max_flip_rate: 0 # share of transformed cases whose answer changes irrelevant_context: insertions: 2 max_tv_distance: 0.05 - level: warning # requirement (default) fails the run; warning only warns + level: warning # requirement (default) fails the run; warning only warns ``` -A property is enabled by being present (`enabled: false` turns it off). It needs at least -one limit. +A property is enabled by being present; `enabled: false` turns it off while keeping its +settings. Every property except `monotonic` needs at least one limit. Contract validation +rejects a property that does not apply to the decision type, for example `monotonic` on a +`choice` decision. + +### Settings shared by every property + +| Key | Default | Meaning | +| --- | --- | --- | +| `enabled` | `true` | turn the property off without deleting it | +| `level` | `requirement` | `requirement` fails the run on violation; `warning` turns PASS into WARN | +| `samples` | per property | transformed cases generated per dataset case | -### Limits (all properties except `monotonic`) +### Limits -| key | per transformed case | meaning | +| Key | Scope | Meaning | | --- | --- | --- | -| `max_tv_distance` | yes | total variation distance: ½ Σ \|p − q\|, 0–1 | -| `max_js_divergence` | yes | Jensen-Shannon divergence in bits, 0–1 (symmetric, finite) | -| `max_abs_delta` | yes | largest change of any one label's probability | -| `max_confidence_delta` | yes | change of the selected label's confidence | +| `max_tv_distance` | per transformed case | total variation distance, ½ Σ \|p − q\|, 0–1 | +| `max_js_divergence` | per transformed case | Jensen-Shannon divergence in bits, 0–1 (symmetric, finite) | +| `max_abs_delta` | per transformed case | largest change of any single label's probability | +| `max_confidence_delta` | per transformed case | change of the selected label's confidence | | `max_violation_rate` | aggregate, default `0` | share of transformed cases allowed to break a per-case limit | | `max_flip_rate` | aggregate | share of transformed cases whose selected label may change | Every comparison also records the confidence delta, whether the answer flipped and -whether the ranking of the labels changed. A failing check therefore shows how far the -distribution moved as well as whether the top answer flipped. +whether the ranking of the labels changed. A failing check shows how far the distribution +moved as well as whether the top answer flipped: a model can keep the same answer and +still shift 20% of its probability mass because of the option order. + +### Checks produced -Checks per property: `.max_violation_rate` (when a per-case limit is set), -`.max_flip_rate` (when set) and `.max_error_rate`. The error check is always a -requirement fixed at 0: if the backend fails on a transformed case, or the original case -failed, the property was not demonstrated. If no case could be transformed, the check -fails too. +Each enabled property adds up to three checks to the report: -## Properties +- `.max_violation_rate` — when a per-case limit is set; +- `.max_flip_rate` — when set; +- `.max_error_rate` — always, as a requirement fixed at `0`. If the backend fails on + a transformed case, or the original case failed, the property was not demonstrated. + If no case could be transformed at all, the check fails too. -| property | decisions | transformation | expected | +## Property catalog + +| Property | Decisions | Transformation | Expected | | --- | --- | --- | --- | -| `option_order` | choice, noul | present the options in another order (`samples` distinct permutations) | invariant | -| `label_format` | all | reformat the labels: `styles` from `upper`, `lower`, `capitalized`, `lettered` (`A) refund`), `numbered` (`1. refund`), `quoted`, `bracketed`; plus `aliases` | invariant | -| `irrelevant_context` | all | insert `insertions` sentences from `fragments` at sentence boundaries | invariant | -| `repeated_context` | all | insert one fragment `repeats` times | invariant | -| `whitespace` | all | `edits` whitespace changes (double spaces, line breaks, tabs, leading/trailing) | invariant | -| `paraphrase` | all | replace the text with paraphrases from `source` | invariant | -| `inversion` | noul | apply the first matching `rewrites` (`find` → `replace`, first occurrence) | answer swaps | -| `monotonic` | score | add each of `deltas` to the numeric input `field` | score does not move against `direction` | - -Option presentation is separate from the input. Backends receive the options in the -order and form shown (`request.decision.labels`, or `decision.labels` over HTTP), answer -with probabilities keyed by those labels, and DecGuard maps them back to the contract -labels before comparing. For `noul`, `label_format` covers affirmative/negative surface -variation (`YES`/`NO`, `"yes"`, or aliases such as `true`/`false`). - -`fragments` defaults to neutral English sentences. Supply your own that are truly -irrelevant for your decision. Text transformations need string inputs, or -`fuzz.text_field` for object inputs. A case that a property cannot transform (no text, -no matching rewrite, missing numeric field) is listed under `properties.skipped` with the -reason; it is never dropped silently. +| [`option_order`](#option-order) | choice, noul | present the options in another order | invariant | +| [`label_format`](#label-format) | all | reformat the labels, or use aliases | invariant | +| [`irrelevant_context`](#irrelevant-context) | all | insert irrelevant sentences | invariant | +| [`repeated_context`](#repeated-context) | all | insert one irrelevant sentence several times | invariant | +| [`whitespace`](#whitespace) | all | change whitespace | invariant | +| [`paraphrase`](#paraphrase) | all | replace the text with a paraphrase | invariant | +| [`inversion`](#inversion-noul) | noul | negate the input with a literal rewrite | answer swaps | +| [`monotonic`](#monotonic-score) | score | increase or decrease a numeric input field | score does not move against the direction | -### `inversion` (noul) +### `option_order` -```yaml -properties: - inversion: - rewrites: - - {find: " is eligible", replace: " is not eligible"} - max_tv_distance: 0.1 # distance between before and after with the labels swapped - max_flip_rate: 0 # the answer must invert -``` +Presents the options as distinct permutations of the contract order. Catches position +bias: models that favour whichever option is listed first. -Only use rewrites whose meaning you know inverts the decision. This is a separate -property from the invariances, so a model that ignores negation fails here while it may -pass everything else. +| Key | Default | +| --- | --- | +| `samples` | `3` distinct permutations (1–100) | -### `monotonic` (score) +### `label_format` + +Changes the surface form of the labels. + +| Key | Default | Meaning | +| --- | --- | --- | +| `styles` | all seven | any of `upper` (`REFUND`), `lower`, `capitalized` (`Refund`), `lettered` (`A) refund`), `numbered` (`1. refund`), `quoted` (`"refund"`), `bracketed` (`[refund]`) | +| `aliases` | `{}` | alternative wordings per label; variant *i* uses every label's *i*-th alias | +| `samples` | `2` | variants per case | + +For `noul` decisions this covers affirmative/negative surface variation, such as `YES` / +`NO`, `"yes"`, or aliases like `true` / `false`: ```yaml properties: - monotonic: - field: affected_users - direction: increasing # more users must not lower the severity - deltas: [100, 10000] - max_level_decrease: 0.05 # allowed move of the expected level, in scale steps - max_flip_rate: 0 # selected level must not move against the direction + label_format: + styles: [upper, capitalized, quoted] + aliases: {spam: [junk], ham: [legitimate]} + max_tv_distance: 0.02 ``` -The expected level is Σ index × probability over the ordered levels. +### `irrelevant_context` + +Inserts sentences that should not affect the decision at sentence boundaries of the text. + +| Key | Default | Meaning | +| --- | --- | --- | +| `fragments` | eight neutral English sentences | the sentences to insert | +| `insertions` | `1` (1–10) | sentences inserted per transformed case | +| `samples` | `2` | transformed cases per case | + +The default fragments are generic ("The weather was mild that day."). Supply your own +that are truly irrelevant for **your** decision; a sentence about shipping is not +irrelevant to a refund decision. -### `paraphrase` providers +### `repeated_context` + +Inserts one fragment several times, to catch models distracted by repetition. + +| Key | Default | +| --- | --- | +| `fragments` | the same defaults as `irrelevant_context` | +| `repeats` | `3` (2–20) | +| `samples` | `1` | + +### `whitespace` + +Applies whitespace edits: doubled spaces, line breaks, tabs, leading and trailing blanks. + +| Key | Default | +| --- | --- | +| `edits` | `3` (1–20) edits per transformed case | +| `samples` | `2` | + +### `paraphrase` + +Replaces the text with paraphrases from a provider. ```yaml properties: paraphrase: - source: {provider: identity} # default: the same text again (checks determinism) + source: {provider: identity} # the default max_tv_distance: 0.0 ``` -- `identity`: no model needed. It re-asks the same input, so it catches backends that - answer differently each time. -- `file`: curated paraphrases, deterministic and reviewable: `source: {provider: file, path: - paraphrases.jsonl}` with rows `{"id": "", "paraphrases": [...]}` or +| Key | Default | +| --- | --- | +| `source` | `{provider: identity}` | +| `samples` | `1` (1–20) paraphrases per case | + +Providers: + +- **`identity`** — re-asks the same input. No model needed; it catches backends that + answer differently each time (non-determinism). +- **`file`** — curated paraphrases, deterministic and reviewable: + `source: {provider: file, path: paraphrases.jsonl}`, relative to the contract, with rows + `{"id": "", "paraphrases": ["...", "..."]}` or `{"input": "", "paraphrases": [...]}`. -- `openai`: any OpenAI-compatible chat-completions API: - - `source: {provider: openai, model: gpt-4o-mini, url: https://api.openai.com/v1, api_key_env: OPENAI_API_KEY}`; - - temperature 0 and the run seed are sent; - - the model and system fingerprint are recorded; - - `api_key_env: null` for local servers. - - LLM output is not guaranteed to be reproducible, so replay uses the stored text. -- Plugins: a `decguard.fuzz.paraphrase.Paraphraser` subclass under the +- **`openai`** — any OpenAI-compatible chat-completions API: + + ```yaml + source: + provider: openai + model: gpt-4o-mini + url: https://api.openai.com/v1 # default + api_key_env: OPENAI_API_KEY # default; null for local servers without a key + timeout_s: 60 # default + temperature: 0.0 # default + ``` + + The run seed is sent with the request, and the model and system fingerprint are recorded. + LLM output is not guaranteed to be reproducible, so [replay](./fuzzing.md#replay) uses + the stored paraphrase text. +- **Plugins** — a `decguard.fuzz.paraphrase.Paraphraser` subclass registered under the `decguard.paraphrasers` entry-point group. -Paraphrase failures are recorded as `paraphrase_error` for the case. +A provider failure is recorded as a `paraphrase_error` for the case. -## Minimization +### `inversion` (noul) -For `option_order`, `label_format`, `irrelevant_context`, `repeated_context` and -`whitespace`, a failing transformation is shrunk greedily until no smaller version still -fails, or until `max_minimize_calls` is spent: -- inserted fragments and whitespace edits are removed one at a time; -- reformatted labels are reverted one at a time; -- permutations are moved closer to the original order one adjacent swap at a time. +Applies the first matching literal rewrite (`find` → `replace`, first occurrence) that +negates the input. The answer must **swap**: the comparison is made with the labels +swapped, so `max_tv_distance` measures the distance from the perfectly inverted answer. -The smallest failing version is stored as `reduced` next to the full one. +```yaml +properties: + inversion: + rewrites: + - {find: " is eligible", replace: " is not eligible"} + max_tv_distance: 0.1 + max_flip_rate: 0 # the answer must invert +``` + +| Key | Default | Meaning | +| --- | --- | --- | +| `rewrites` | required, at least one | tried in order | +| `samples` | `1` (1–20) | applicable rewrites used per case | + +Only use rewrites whose meaning you know inverts the decision. This is a separate property +from the invariances, so a model that ignores negation fails here while it may pass +everything else. -## Reproducibility and replay +### `monotonic` (score) + +Adds each of `deltas` to a numeric field of object inputs. The score must not move against +`direction`. -Transformations draw from a SHA-256-based stream keyed by `(seed, property, case id)`. -The same seed gives the same transformations on every platform and Python version, and -adding cases or properties does not change the others. Reports record: -- the seed and the fuzz settings; -- the property settings; -- the contract hash, dataset hash and backend metadata; -- every transformed case with its steps and any paraphrase provenance. +```yaml +fuzz: + text_field: text # other properties edit input.text -```bash -decguard replay fuzz.json # every stored failure -decguard replay fuzz.json --id option_order/r1/0 # one of them -decguard replay fuzz.json --backend candidate # against another backend +properties: + monotonic: + field: affected_users + direction: increasing # more affected users must not lower the severity + deltas: [100, 10000] + max_level_decrease: 0.05 + max_flip_rate: 0 # the selected level must not move against the direction ``` -Replay does three things: -1. re-sends the original case and each stored failing transformation, including the - reduced one, exactly as stored; -2. judges them again; -3. regenerates the transformation from the seed to confirm it matches the stored one. +| Key | Default | Meaning | +| --- | --- | --- | +| `field` | required | top-level numeric key of object inputs | +| `direction` | `increasing` | `increasing`: a larger value must not lower the score; `decreasing`: the opposite | +| `deltas` | `[1.0]` | positive amounts added to the field, one transformed case each | +| `max_level_decrease` | `0.0` | allowed move of the expected level against the direction, in scale steps | + +The **expected level** is Σ index × probability over the ordered levels, so this compares +the whole distribution, not only the top answer. + +## Inputs a property cannot transform -It exits 1 if a failure still reproduces and 0 if none does. `decguard report fuzz.json ---contract edited.yaml` re-applies edited limits and levels to the stored cases without -calling the backend. +Text transformations need string inputs, or [`fuzz.text_field`](./contracts.md#fuzz) to +name the key to edit in object inputs. A case a property cannot transform — no text, no +matching rewrite, a missing numeric field — is listed under `properties.skipped` in the +report with the reason. It is never dropped silently. diff --git a/docs/quickstart.md b/docs/quickstart.md new file mode 100644 index 0000000..c3fb9bb --- /dev/null +++ b/docs/quickstart.md @@ -0,0 +1,293 @@ +--- +description: Write a Decision Contract and a golden dataset, validate them, and run your first reliability test in a few minutes. +--- + +# Quickstart + +In this guide you describe a refund-handling decision, give DecGuard a handful of +labeled examples, and run a reliability test against an offline mock backend. Every +command and output below was produced by `decguard 0.1.1` installed from PyPI. + +You need DecGuard [installed](./installation.md) and an empty directory: + +```bash +pip install decguard +mkdir refund-demo && cd refund-demo +``` + +## 1. Describe the decision + +A support tool receives a customer message and must decide whether to **refund**, +**reject** or send the request to manual **review**. Create `decguard.yaml`: + +```yaml +schema_version: "0.1" + +decision: + name: refund_request + description: Decide how to handle a customer refund request. + type: choice + options: [refund, reject, review] + +backend: + provider: mock + model: refund-mock-v1 + rules: + - contains: "damaged" + probabilities: {refund: 0.96, reject: 0.01, review: 0.03} + - contains: "never arrived" + probabilities: {refund: 0.90, reject: 0.02, review: 0.08} + - contains: "changed my mind" + probabilities: {refund: 0.05, reject: 0.85, review: 0.10} + default: {refund: 0.20, reject: 0.15, review: 0.65} + +dataset: cases.jsonl + +evaluation: + confidence_threshold: 0.8 + +requirements: + min_accuracy: 0.9 + max_ece: 0.2 + +warnings: + min_coverage: 0.6 + max_latency_p95_ms: 500 +``` + +What each section says: + +- **`decision`** — the question. A `choice` decision picks one of its `options`; the model + returns a probability for each. `description` documents the question (and is what + [System One](./systemone.md) backends are asked). +- **`backend`** — what answers it. The [`mock`](./backends.md#mock) backend is offline and + deterministic: the first rule whose `contains` text appears in the input answers, and + `default` answers everything else. You will swap it for a real model in step 7. +- **`dataset`** — the golden examples, relative to the contract. +- **`evaluation.confidence_threshold`** — decisions below 0.8 confidence count as + abstentions: cases your application would not act on automatically. +- **`requirements`** — hard gates. If one does not hold, the run **fails** (exit code 1). +- **`warnings`** — soft gates. If one does not hold, the run **warns** (exit code 0). + +## 2. Add a golden dataset + +Create `cases.jsonl`, one JSON object per line. `input` is what the model sees; +`expected` is the correct label. + +```json +{"id": "r1", "input": "The blender arrived damaged and won't turn on.", "expected": "refund"} +{"id": "r2", "input": "My parcel never arrived, tracking stopped two weeks ago.", "expected": "refund"} +{"id": "r3", "input": "I changed my mind about the colour, please take it back.", "expected": "reject"} +{"id": "r4", "input": "The invoice shows a different price than the website did.", "expected": "review"} +{"id": "r5", "input": "Screen was damaged in transit, photos attached.", "expected": "refund"} +{"id": "r6", "input": "I changed my mind, found it cheaper elsewhere.", "expected": "reject"} +``` + +Real datasets are larger, and may include unlabeled cases and `metadata` for +segmentation. See [Golden datasets](./datasets.md). + +## 3. Validate + +`decguard validate` checks the contract, every backend's settings and the dataset, +**without calling any backend**: + +```bash +decguard validate decguard.yaml +``` + +```text +OK decguard.yaml + decision refund_request (choice): refund, reject, review + backends default=mock + dataset cases.jsonl: 6 cases, 6 labeled + gates requirements: 2, warnings: 2 + properties none (add 'properties' to fuzz) + production segments: none; requirements: 0; segment requirements: 0 + policy none + hash sha256:ab7ab0ad031d6130242da713be280ba3ee50dd6ad14e1665cc9a8e785ff03ecd +``` + +Validation is strict. A typo is an error that names the field, and the command exits +with code `2`: + +```text +error: decguard.yaml: invalid contract: + requirements.min_acuracy: unknown field (check the spelling) +``` + +## 4. Test + +```bash +decguard test decguard.yaml --output report.json +``` + +```text +DecGuard 0.1.1 · refund_request (choice) · PASS + + contract decguard.yaml sha256:ab7ab0ad031d + dataset cases.jsonl sha256:5cd7ea766986 + backend default: mock · refund-mock-v1 + + Cases 6 total · 6 decided · 0 errored · 6 labeled + Accuracy 1.000 macro-F1 1.000 + Calibration ECE 0.138 Brier 0.046 NLL 0.157 (10 bins) + Selective confidence >= 0.8: coverage 0.833 abstention 0.167 accuracy 1.000 + Latency ms mean 0.0 p50 0.0 p95 0.0 max 0.0 + + Checks + PASS min_accuracy accuracy 1 >= 0.9 + PASS max_ece ece 0.1383 <= 0.2 + PASS max_error_rate error_rate 0 <= 0 + PASS min_coverage coverage 0.8333 >= 0.6 (warning) + PASS max_latency_p95_ms latency_p95_ms 0.007822 <= 500 (warning) + +PASS: all gates hold +``` + +What DecGuard measured: + +- **Cases** — every case got a valid answer. A backend failure would count as an error, + never be dropped. +- **Accuracy / macro-F1** — how often the selected label is the expected one. +- **Calibration** — whether confidence matches reality. The model is always right here but on + average only 86% confident, so the expected calibration error (ECE) is 0.138: it is + *under*-confident. ECE, Brier and NLL are explained in + [Golden tests and metrics](./metrics.md#calibration). +- **Selective** — 5 of 6 decisions (0.833) clear the 0.8 threshold. The sixth, the invoice + question, got `review` at 0.65: correct, but not confident enough to automate. +- **Latency** — time spent in the backend call. The mock answers in microseconds. +- **Checks** — one line per gate. `max_error_rate` was not configured: it is an + [implicit requirement](./gates.md#implicit-gates) that no case may error. + +`--output` also wrote the complete report, including every decision, to `report.json`. +[`decguard report report.json`](./cli.md#decguard-report) prints it again at any time. + +## 5. Check behavior, not only answers + +Golden tests check answers to fixed inputs. **Metamorphic properties** check that the +answer does not change when it should not: here, when the options are presented in a +different order, or an irrelevant sentence is added to the message. Append to +`decguard.yaml`: + +```yaml +properties: + option_order: + max_tv_distance: 0.03 + irrelevant_context: + max_tv_distance: 0.05 +``` + +`max_tv_distance` is how far the probability distribution may move (total variation +distance, 0–1) on any transformed case. Run the golden gates and the properties together: + +```bash +decguard test decguard.yaml --all +``` + +```text + Properties seed 0 + option_order 18 compared · 0 violating (0.0%) · flips 0 (0.0%) · max TV 0.000 + irrelevant_context 11 compared · 0 violating (0.0%) · flips 0 (0.0%) · max TV 0.000 + + Checks + ... + PASS option_order.max_error_rate option_order.error_rate 0 <= 0 + PASS option_order.max_violation_rate option_order.violation_rate 0 <= 0 (per case: max_tv_distance 0.03) + PASS irrelevant_context.max_error_rate irrelevant_context.error_rate 0 <= 0 + PASS irrelevant_context.max_violation_rate irrelevant_context.violation_rate 0 <= 0 (per case: max_tv_distance 0.05) + +PASS: all gates hold +``` + +Transformations are derived from a seed, so every run checks exactly the same inputs. +[Fuzzing and replay](./fuzzing.md) shows a model that fails this check. + +## 6. PASS, WARN and FAIL + +Every run ends in one status, and the exit code follows from it: + +| Status | Meaning | Exit code | +| --- | --- | --- | +| PASS | every requirement and every warning holds | `0` | +| WARN | every requirement holds, at least one warning does not | `0` (`1` with `--fail-on-warn`) | +| FAIL | at least one requirement does not hold | `1` | +| error | the run could not be evaluated (invalid contract, unreachable backend, ...) | `2` | + +**A warning.** Raise the coverage warning to `min_coverage: 0.9` and run +`decguard test decguard.yaml` again. Coverage is 0.833, so: + +```text + WARN min_coverage coverage 0.8333 violates >= 0.9 (warning) + +WARN: requirements hold, some warning gates do not +``` + +The exit code is still `0`, so a pipeline continues; add `--fail-on-warn` where +warnings must block. + +**A failure.** Put `min_coverage` back to `0.6` and add a case the mock gets wrong — the +customer mentions damage but wants a replacement, which should go to review: + +```json +{"id": "r7", "input": "The lid was damaged, but I just want a replacement lid.", "expected": "review"} +``` + +```text + Checks + FAIL min_accuracy accuracy 0.8571 violates >= 0.9 + FAIL max_ece ece 0.2329 violates <= 0.2 + PASS max_error_rate error_rate 0 <= 0 + PASS min_coverage coverage 0.8571 >= 0.6 (warning) + PASS max_latency_p95_ms latency_p95_ms 0.007859 <= 500 (warning) + + Failures (1 of 1) + r7 expected review, got refund (0.96) + input: The lid was damaged, but I just want a replacement lid. + +FAIL: at least one requirement does not hold +``` + +The model is wrong **and** 96% confident, so accuracy drops and calibration fails with +it. The failing example is printed with its input, and the command exits `1`. This is the +signal a CI job gates on. + +## 7. Connect a real model + +Only the `backend` section changes. For example, to ask TypeSafe's Jev through +OpenRouter's Decisions API: + +```yaml +backend: + provider: systemone + model: typesafe/jev-1.13 + url: https://openrouter.ai/api/alpha/decisions + bearer_token_env: OPENROUTER_API_KEY + +evaluation: + confidence_threshold: 0.8 + probability_tolerance: 0.02 # Jev rounds probabilities to two decimals +``` + +```bash +export OPENROUTER_API_KEY=... # the key stays in the environment, never in the contract +decguard test decguard.yaml +``` + +You can also keep the mock as the default and add real models under +[`backends`](./contracts.md#backend-and-backends), then choose one with +`decguard test decguard.yaml --backend NAME`. See [Backends](./backends.md) for Kev, +HTTP endpoints and your own Python code. + +## Next steps + +- Understand the building blocks: [Decisions and results](./decisions.md), + [Golden datasets](./datasets.md), [Requirements and warnings](./gates.md). +- Explore the complete refund example in the repository, + [`examples/refund`](https://github.com/suffro/decguard/tree/main/examples/refund): an + order-sensitive model, regression limits, production records and a routing policy. +- Find what golden tests miss with [metamorphic properties](./properties.md) and + [fuzzing](./fuzzing.md). +- Compare model versions with [regression diffs](./regression.md). +- Check deployed models with [production checks](./production.md), and route live + decisions with [policies](./policy.md). +- Gate pull requests with [GitHub Actions](./ci.md). diff --git a/docs/regression.md b/docs/regression.md index 5c6eb36..526fed9 100644 --- a/docs/regression.md +++ b/docs/regression.md @@ -1,8 +1,12 @@ +--- +description: Compare a baseline and a candidate model run case by case, and fail CI on answer flips, calibration loss, distribution shifts, new errors or segment regressions. +--- + # Regression diffs -`decguard diff` compares two stored reports of the same decision, typically the current -model (baseline) and a new model or version (candidate), and fails CI when the candidate -regresses beyond the contract's limits. +`decguard diff` compares two stored reports of the same decision — typically the model in +production (**baseline**) and a new model, version or backend (**candidate**) — and fails +when the candidate regresses beyond the contract's `regression` limits. ```bash decguard test decguard.yaml -o baseline.json @@ -10,23 +14,85 @@ decguard test decguard.yaml --backend candidate -o candidate.json decguard diff baseline.json candidate.json --contract decguard.yaml ``` -Cases are matched by id. Only matched cases are compared, and both runs' metrics are -recomputed on them with the same `evaluation` settings, so the numbers stay comparable -even if the datasets differ slightly. Unmatched cases are counted. +Aggregate gates can hide a regression: two models with the same accuracy can disagree on +many cases, or one can be much less calibrated. A diff compares the runs **case by case**. + +## How runs are compared + +- Cases are **matched by id**. Only matched cases are compared; cases present in only one + report are counted. +- Both runs' metrics are **recomputed on the matched cases** with the same `evaluation` + settings (the contract's with `--contract`, otherwise the candidate report's), so the numbers stay comparable even if the + datasets differ slightly. +- Both reports must be for the same decision (name, type and labels). +- The diff records both runs' provenance: backend, model and version, contract and dataset + hashes, and whether the dataset was the same. + +## Example + +The refund example's `order_sensitive` backend keeps every answer but shifts probability +mass toward the first option. Compared with the stable model: + +```bash +decguard test examples/refund/decguard.yaml -o stable.json +decguard test examples/refund/decguard.yaml -b order_sensitive -o candidate.json +decguard diff stable.json candidate.json -c examples/refund/decguard.yaml +``` + +```text +DecGuard 0.1.1 · diff refund_request (choice) · FAIL + + baseline stable.json default: mock · refund-mock-v1 + candidate candidate.json order_sensitive: mock · refund-mock-v2 + + Cases 11 matched · 11 compared · 0 only in baseline · 0 only in candidate + Answers 0 flipped (0.0%) · 0 newly wrong · 0 newly correct + Errors 0 new · 0 recovered · 0 still + Shift mean TV 0.051 max TV 0.097 mean JS 0.008 + Confidence mean shift -0.039 mean |shift| 0.045 max |shift| 0.093 + + Metric baseline candidate delta + accuracy 1.000 1.000 +0.000 + macro_f1 1.000 1.000 +0.000 + ece 0.139 0.175 +0.036 + brier 0.049 0.078 +0.030 + nll 0.159 0.208 +0.049 + coverage 0.727 0.545 -0.182 + error_rate 0.000 0.000 +0.000 + latency_p50_ms 0.005 0.005 +0.000 + latency_p95_ms 0.012 0.009 -0.003 + + Segments + locale=en n=11 accuracy 1.000 -> 1.000 flips 0.0% + + Checks + PASS max_answer_flip_rate answer_flip_rate 0 <= 0.05 + PASS max_accuracy_drop accuracy_drop -0 <= 0.02 + PASS max_ece_increase ece_increase 0.0361 <= 0.05 + FAIL max_mean_tv_distance mean_tv_distance 0.05082 violates <= 0.03 + PASS max_error_rate_increase error_rate_increase 0 <= 0 + +FAIL: the candidate regresses beyond the contract's limits +``` + +Accuracy and answers are unchanged, but the candidate is less confident (coverage at the +0.8 threshold drops from 0.727 to 0.545), less calibrated, and its distributions moved by +a mean total variation distance of 0.051, beyond the allowed 0.03. The diff exits `1`. ## What is compared -- **Answers**: flips (selected label changed), newly wrong and newly correct labeled cases. -- **Distributions**: mean and max total variation distance, mean Jensen-Shannon +- **Answers** — flips (selected label changed), newly wrong and newly correct labeled cases. +- **Distributions** — mean and max total variation distance, mean Jensen-Shannon divergence. -- **Confidence**: mean signed shift, mean and max absolute shift. -- **Metrics**: accuracy, macro-F1, ordinal MAE, ECE, Brier, NLL, coverage, error rate, - latency p50/p95 (baseline, candidate, delta). -- **Failures**: cases that newly error, recover, or still error. -- **Segments**: per value of each `segment_by` metadata key (or `--segment-by KEY`), the - accuracy before and after and the flip rate. +- **Confidence** — mean signed shift, mean and max absolute shift. +- **Metrics** — accuracy, macro-F1, ordinal MAE, ECE, Brier, NLL, coverage, error rate, + latency p50/p95: baseline, candidate and delta. +- **Errors** — cases that newly error, recover, or still error. +- **Segments** — per value of each `segment_by` metadata key (or `--segment-by KEY`): + accuracy before and after, and flip rate. -Every changed case is listed with its input, so a regression can be inspected directly. +Every flipped, newly errored or recovered case is listed under `changes` with its input, +both answers and confidences, so a regression can be inspected directly. ## Gates @@ -42,22 +108,43 @@ regression: max_mean_confidence_shift: 0.03 ``` -The top-level gates are requirements (exit 1); `warnings` takes the same gates (WARN). +Gates at the top level of `regression` are requirements (exit `1`); `regression.warnings` +takes the same gates at warning level. -| gate | value | +| Gate | Value checked (must be ≤ the limit) | | --- | --- | | `max_answer_flip_rate` | flips / cases both runs decided | -| `max_accuracy_drop`, `max_macro_f1_drop` | baseline − candidate | -| `max_ece_increase`, `max_brier_increase`, `max_nll_increase` | candidate − baseline | -| `max_mean_tv_distance` | mean total variation distance | +| `max_accuracy_drop` | baseline − candidate accuracy | +| `max_macro_f1_drop` | baseline − candidate macro-F1 | +| `max_ece_increase` | candidate − baseline ECE | +| `max_brier_increase` | candidate − baseline Brier score | +| `max_nll_increase` | candidate − baseline NLL | +| `max_mean_tv_distance` | mean total variation distance between matched distributions | | `max_mean_confidence_shift` | mean absolute confidence shift | | `max_error_rate_increase` | candidate − baseline error rate; **defaults to 0** | | `max_latency_p95_increase_ms` | candidate − baseline p95 latency | -| `max_segment_accuracy_drop` | worst accuracy drop over segments with ≥ `min_segment_size` cases (needs `segment_by`) | +| `max_segment_accuracy_drop` | worst accuracy drop over segments with ≥ `min_segment_size` matched cases (needs `segment_by`) | + +| Setting | Default | Meaning | +| --- | --- | --- | +| `segment_by` | `[]` | case `metadata` keys whose values define segments | +| `min_segment_size` | `5` | smallest segment that `max_segment_accuracy_drop` considers | + +Without `--contract`, only the implicit `max_error_rate_increase: 0` applies. A gate whose +value cannot be computed (for example accuracy without labeled cases) fails. + +## Output + +A terminal summary by default; `--format json` prints the diff report to stdout and +`--output diff.json` writes it (`diff_version` 0.1, see [Reports](./reports.md#diff-report)). +`--fail-on-warn` makes warning-level regressions exit `1`. -Without `--contract`, only the implicit `max_error_rate_increase: 0` applies. A gate -whose value cannot be computed (e.g. accuracy without labels) fails. +## Typical uses -Output: a terminal summary, `--format json`, and `--output diff.json`. The JSON diff -(`diff_version` 0.1) records both runs' provenance (backend, model/version, contract and -dataset hashes). +- **Model upgrade in CI.** Test the production backend and the candidate on the same + golden dataset in one job, diff them, and block the merge on regressions. See + [CI](./ci.md#regression-gate-for-a-model-upgrade). +- **Backend migration.** Moving from a self-hosted Kev server to Jev through OpenRouter, or + the other way round, is a diff between two backends of the same contract. +- **Stored baseline.** Keep the production model's report as a build artifact and diff each + candidate against it, without re-running the baseline. diff --git a/docs/reports.md b/docs/reports.md index 4b45b2d..4cd958d 100644 --- a/docs/reports.md +++ b/docs/reports.md @@ -1,79 +1,156 @@ -# Reports, metrics and gates +--- +description: The JSON reports DecGuard writes - test/fuzz reports, diff, replay and production reports - their integrity checks, and every input format at a glance. +--- -`decguard test` produces one report. `--output report.json` writes it as JSON, -`--format json` prints it to stdout; the default terminal view is a summary of the same -data. +# Reports and formats -## JSON report (`report_version` 0.1) +Every command that evaluates something produces a versioned JSON document. The terminal +output is a summary of the same data: `--format json` prints the full document to stdout, +and `--output FILE` writes it. -| field | content | +| Document | Written by | Version field | +| --- | --- | --- | +| [Report](#report) | `test`, `fuzz`, `test --all`, `report` | `report_version: "0.1"` | +| [Diff report](#diff-report) | `diff` | `diff_version: "0.1"` | +| [Replay report](#replay-report) | `replay --format json` | `replay_version: "0.1"` | +| [Production report](#production-report) | `check` | `check_version: "0.1"` | + +All of them carry `decguard_version`, `created_at`, `status` (`pass`, `warn` or `fail`) +and `exit_code`. Their formats are versioned independently of the package: a change in +meaning comes with a version bump and a changelog entry. + +## Report + +The report of `decguard test` and `decguard fuzz`. + +| Field | Content | | --- | --- | | `report_version`, `decguard_version`, `created_at` | format and tool provenance | | `mode` | `test` (golden gates), `fuzz` (properties) or `all` (both) | -| `status`, `exit_code` | `pass` / `warn` / `fail`, and 0 / 0 / 1 | -| `contract` | name, type, labels, schema version, `sha256:` hash, path | -| `dataset` | path, `sha256:` hash of the file bytes, case and labeled counts | -| `backend`, `health` | backend name, provider, model, version, redacted details; healthcheck result | -| `evaluation`, `requirements`, `warnings` | the evaluation and gate settings used | -| `metrics` | `counts`, `classification`, `calibration`, `selective`, `latency` | -| `checks` | one entry per gate: level, metric, comparison, threshold, value, status | +| `status`, `exit_code` | `pass` / `warn` / `fail`, and `0` / `0` / `1` | +| `contract` | `name`, `type`, `labels`, `schema_version`, `hash` (`sha256:`), `path` | +| `dataset` | `path`, `hash` (`sha256:` of the file bytes), `n_cases`, `n_labeled` | +| `backend` | `name`, `provider`, `model`, `model_version`, `details` (redacted provider details) | +| `health` | healthcheck `status` (`ok` / `unhealthy` / `unknown`) and `detail` | +| `evaluation`, `requirements`, `warnings` | the settings and gates used | +| `metrics` | `counts`, `classification`, `calibration` (with `reliability` bins), `selective`, `latency` | +| `checks` | one entry per gate: `gate`, `level`, `metric`, `comparison`, `threshold`, `value`, `status`, `message` | | `failures` | failing examples (errors and wrong answers) with their inputs | -| `results` | every case: input, expected, metadata, and a `result` or an `error` | -| `properties` | `fuzz`/`all` only: seed, fuzz and property settings, per-property `summaries`, every transformed case in `pairs` (steps, input, option presentation, result, comparison, violations, `reduced` example), `skipped` cases with reasons | - -Loading a report re-checks every stored result: probabilities over exactly the contract -labels, in canonical order, summing to 1 within the recorded `probability_tolerance`; -`selected` equal to the argmax (ties to the first label) and `confidence` equal to its -probability; and exactly one of `result` or `error` per case. Transformed cases are -checked the same way. Their comparison and violations must also match what their stored -results and the recorded property settings imply. Dataset counts, aggregate metrics, -failure examples, property summaries, checks, status and exit code are recomputed and -must agree. A tampered or corrupted report is rejected with exit code 2. - -Because `results` holds every normalized decision, `decguard report --contract -` can recompute metrics and re-apply gates offline; recomputation from the same -results is bit-identical. For `fuzz`/`all` reports it also re-applies the edited property -limits and levels to the stored transformed cases (transformations are not regenerated). - -`decguard diff` writes a separate diff report (`diff_version` 0.1, see -[regression.md](regression.md)); `decguard replay --format json` a replay report -(`replay_version` 0.1, see [properties.md](properties.md)); and `decguard check` a -post-deployment report (`check_version` 0.1, see [production.md](production.md)). - -## Metrics - -Classification and calibration metrics use **labeled** cases the backend decided. Coverage -and latency use all decided cases. Errors are counted separately and never dropped. - -- **accuracy**: share of labeled cases where the selected label is the expected one. -- **macro_f1**: unweighted mean F1 over labels that appear as expected or predicted - (scikit-learn's `average="macro"`; undefined precision/recall count as 0). - `per_label` has precision, recall, F1 and support for each label. -- **ordinal_mae** (score only): mean |predicted level − expected level| in scale steps. -- **nll**: mean −ln p(expected). p is floored at 1e-15, so a confident miss adds about 34.5 - instead of infinity. -- **brier**: multi-class Brier score, mean of Σₖ (pₖ − yₖ)², range 0–2. (For two labels this - is twice the binary `(p − y)²` form.) -- **ece**: expected calibration error over top-label confidence with `calibration_bins` - equal-width bins; bin *i* covers (i/n, (i+1)/n], the first also includes 0. `reliability` - lists count, mean confidence and accuracy per bin. -- **coverage / abstention_rate**: share of decisions with confidence ≥ / < - `evaluation.confidence_threshold`. **selective_accuracy**: accuracy on labeled cases at or - above it. -- **error_rate**: errored cases / all cases, with `errors_by_kind`. -- **latency**: mean, p50/p90/p95/p99 (linear interpolation, NumPy's default) and max of the - time spent in the backend call, in milliseconds. - -Calibration is reported separately from accuracy on purpose: a model can be accurate and -badly calibrated, which matters as soon as its confidence drives an action. - -## Status and exit codes - -- **FAIL** (exit 1): at least one requirement does not hold, including the implicit - `max_error_rate: 0` (and `.max_error_rate: 0`), a gated metric could not be - computed, or a requirement-level property is violated. -- **WARN** (exit 0, or 1 with `--fail-on-warn`): requirements hold, a warning gate does not. -- **PASS** (exit 0): everything holds. -- Exit **2**: the run could not be evaluated: invalid contract or dataset, unknown backend, - missing credentials, unhealthy backend, every case failing, invalid CLI usage, or an - internal error. +| `results` | every case: `case_id`, `input`, `expected`, `metadata`, and either a `result` or an `error` | +| `properties` | `fuzz`/`all` only: see below | + +A check entry: + +```json +{ + "gate": "min_accuracy", + "level": "requirement", + "metric": "accuracy", + "comparison": ">=", + "threshold": 0.9, + "value": 1.0, + "status": "pass", + "message": "accuracy 1 >= 0.9" +} +``` + +A result entry holds the normalized +[`DecisionResult`](./decisions.md#the-normalized-decisionresult); an error entry holds its +`kind` and message. + +### `properties` + +Present in `fuzz` and `all` reports: + +| Field | Content | +| --- | --- | +| `seed` | the transformation seed used | +| `fuzz`, `config` | the fuzz settings and every property's settings | +| `summaries` | per property: counts of transformed, skipped, errored and evaluated cases; violations and flips with their rates; max TV, JS and deltas | +| `pairs` | every transformed case: `id` (`//`), `steps`, `input`, `presentation` (option shown → contract label), `result` or `error`, `comparison`, `violations`, and a `reduced` minimal example when minimized | +| `skipped` | cases a property could not transform, with the reason | + +### Integrity on load + +Loading a report — with `report`, `diff` or `replay` — re-verifies it completely: + +- every stored result has probabilities over exactly the contract labels, in canonical + order, summing to 1 within the recorded `probability_tolerance`; `selected` is the + argmax (ties to the first label) and `confidence` its probability; +- each case has exactly one of `result` or `error`; +- transformed cases are checked the same way, and their comparisons and violations must + match what their results and the recorded property settings imply; +- dataset counts, aggregate metrics, failure examples, property summaries, checks, status + and exit code are recomputed and must agree with the stored values. + +A tampered or corrupted report is rejected with exit code `2`. + +### Re-evaluation + +Because `results` holds every normalized decision, +`decguard report report.json --contract edited.yaml` recomputes metrics and re-applies the +edited gates offline. Recomputation from the same results is bit-identical. For `fuzz` and +`all` reports it also re-applies edited property limits and levels to the stored +transformed cases; transformations are not regenerated. + +## Diff report + +Written by [`decguard diff`](./regression.md). + +| Field | Content | +| --- | --- | +| `contract` | the decision compared | +| `baseline`, `candidate` | provenance of each run: `path`, `created_at`, `mode`, `contract_hash`, `dataset`, `backend` | +| `same_dataset` | whether both runs used the same dataset file (by hash) | +| `evaluation` | settings used to recompute metrics on matched cases | +| `counts` | cases per run, matched, only in one run, compared; answer flips, newly wrong/correct; newly errored, recovered, still errored | +| `shift` | answer flip rate; mean and max TV distance; mean JS divergence; mean, mean absolute and max absolute confidence delta | +| `metrics` | per metric: `baseline`, `candidate`, `delta` | +| `segments` | per segment: key, value, size, accuracy before and after, flip rate | +| `checks` | regression gates, same shape as report checks | +| `changes` | each flipped, newly errored or recovered case with its input, answers and confidences | + +## Replay report + +Printed by `decguard replay --format json`. + +| Field | Content | +| --- | --- | +| `source` | the report replayed | +| `contract_path`, `contract_hash`, `contract_changed` | the contract used, and whether it differs from the one recorded | +| `backend` | the backend replayed against | +| `outcomes` | per replayed failure: `id`, `property`, `case_id`; `regenerated` (the seed regenerates the stored transformation; `null` for non-deterministic paraphrase providers); the new `original` result or `original_error`; the re-judged `examples` (transformed and reduced); and `reproduced` | + +## Production report + +Written by [`decguard check`](./production.md). + +| Field | Content | +| --- | --- | +| `contract`, `evaluation` | the decision and settings used | +| `config` | the `production` section: segments and gates | +| `dataset` | `path`, `hash`, `n_records`, `n_outcomes`, `n_correctness` | +| `baseline` | `source` (`dataset` or `report`), `path`, `hash`, `n_records` and `metrics`, or `null` | +| `metrics` | `counts`, `outcomes`, `calibration`, `confidence` (with histogram), `threshold`, `routing`, `latency_ms`, `cost` | +| `drift` | `confidence_tv_distance`, `confidence_mean_delta`, `accuracy_drop`, `ece_increase` (`null` without a baseline) | +| `segments` | per metadata group: key, value, metrics, checks | +| `checks` | aggregate and segment checks | +| `records` | every normalized record | + +A production report can be passed back as `--baseline` to the next `decguard check`; its +records are validated again against the contract when it is loaded. + +## Input formats + +| Format | Used by | Reference | +| --- | --- | --- | +| Decision Contract (YAML/JSON) | every command | [Decision Contract](./contracts.md) | +| Golden dataset (JSONL/JSON) | `validate`, `test`, `fuzz` | [Golden datasets](./datasets.md#format) | +| Production records (JSONL/JSON) | `check` | [Record schema](./production.md#record-schema) | +| Paraphrase file (JSONL) | `paraphrase` with `provider: file` | [Paraphrase providers](./properties.md#paraphrase) | +| `decguard.http/0.1` request/response | `http` backend | [HTTP protocol](./http.md#protocol-decguard-http-0-1) | +| System One request/answer | `systemone` backend | [Wire format](./systemone.md#wire-format) | + +Status semantics and exit codes are described in +[Requirements and warnings](./gates.md#exit-codes) and the +[CLI reference](./cli.md#exit-codes). diff --git a/docs/systemone.md b/docs/systemone.md new file mode 100644 index 0000000..7bb77b2 --- /dev/null +++ b/docs/systemone.md @@ -0,0 +1,257 @@ +--- +description: Use the built-in systemone backend with a self-hosted Kev server, Jev through OpenRouter's Decisions API, or a compatible TypeSafe System One endpoint. +--- + +# System One: Kev and Jev + +The built-in `systemone` backend speaks the **System One** decision wire format: an input +(`state`), typed `questions`, and typed `answers` with probabilities. One adapter covers +every service that uses it: + +- **Kev** — a [Kev](https://github.com/jaredpalmer/kev) server you run yourself; +- **Jev through OpenRouter** — TypeSafe's Jev via OpenRouter's Decisions API; +- **TypeSafe** — compatible TypeSafe System One endpoints, directly. + +The adapter has been validated with real inference against a self-hosted Kev-0.8B server +and against Jev (`typesafe/jev-1.13`) through OpenRouter, on Linux and macOS, for +`choice`, `noul` and `score` decisions. See [Real-backend validation](#real-backend-validation). + +## The question + +System One models answer a question about the input, so the question must be stated. +DecGuard uses the decision's `description`, or the backend's `instructions` setting when +it is set. One of the two is required. + +```yaml +decision: + name: support_team + type: choice + description: Which team should handle this customer support ticket? + options: [billing, shipping, technical] +``` + +Put guidance about the options into the description (or `instructions`): for `choice` +decisions, option names are sent without descriptions. + +## Jev through OpenRouter + +```yaml +backend: + provider: systemone + model: typesafe/jev-1.13 + url: https://openrouter.ai/api/alpha/decisions + bearer_token_env: OPENROUTER_API_KEY + timeout_s: 60 + max_retries: 2 + +evaluation: + probability_tolerance: 0.02 # Jev rounds probabilities to two decimals +``` + +The only credential is an OpenRouter API key, and it belongs in the +**`OPENROUTER_API_KEY`** environment variable — never in the contract: + +```bash +export OPENROUTER_API_KEY=... +decguard test decguard.yaml +``` + +- **Pin a model version** (`typesafe/jev-1.13`), not a moving alias. Each result records + the dated snapshot that actually answered as `model_version` (for example + `typesafe/jev-1.13-20260917`), so a silent upstream change shows up in + [regression diffs](./regression.md). +- **Probability tolerance.** Jev returns probabilities rounded to two decimals, so K + labels can sum to 1 ± K × 0.005. Set `evaluation.probability_tolerance` to cover it + (`0.02` for up to four labels). The sum is still checked and never rescaled. +- **Cost.** Every case is one billed request, as is every transformed case during + fuzzing. Token usage and cost are recorded per result (see [below](#what-gets-recorded)). + +## Self-hosted Kev + +Run a Kev server, then point the backend at its System One endpoint. Kev serves +`/v1/systemone` for decisions and `/v1/models` for the loaded model, which works as a +healthcheck: + +```yaml +backend: + provider: systemone + model: kev-latest + url: http://127.0.0.1:8009/v1/systemone + health_url: http://127.0.0.1:8009/v1/models + timeout_s: 300 # CPU inference; the first request also warms the model up +``` + +This is the flow DecGuard validates in CI, with upstream Kev at a pinned commit serving +Kev-0.8B, the smallest current checkpoint (a LoRA adapter and pointer head on +`Qwen/Qwen3.5-0.8B-Base`). Kev needs Python 3.12 or 3.13 and `uv`. It runs on CPU (fp32) +without a GPU, and on MLX (bf16) on Apple Silicon. The first start downloads about 1.8 GB +of weights. + +```bash +git clone https://github.com/jaredpalmer/kev.git && cd kev +git checkout 09ff745d52a0f23954e3b0f5a608bf4c7c6aebb4 +uv sync --locked --extra serve +uv run --locked --extra serve python -m kev.serve \ + --run jaredpalmer/kev-0.8b@9a45d25eb2ab761841196625383fa1dff0e56c1e --port 8009 +``` + +Once `curl -s 127.0.0.1:8009/v1/models` answers, `decguard test` can use it. Kev rounds +probabilities to four decimals, so the default tolerance is usually enough. + +A local Kev server needs no credential. To require one, set `KEV_API_KEY` on the server +and `bearer_token_env` (naming a variable that holds the same key) in the contract. + +## TypeSafe endpoints + +A TypeSafe System One endpoint uses the same adapter with its own URL, model name and key +variable: + +```yaml +backends: + typesafe: + provider: systemone + model: jev-1.13.0 + url: https://api.typesafe.ai/v1/systemone + bearer_token_env: TYPESAFE_API_KEY +``` + +The real-backend validation covers Kev and Jev through OpenRouter; direct TypeSafe +endpoints share the wire format but are not part of that validation. + +## Comparing backends + +Declare several System One backends in one contract and run the same dataset through +each, then [diff](./regression.md) them: + +```yaml +backend: # a local Kev server + provider: systemone + model: kev-latest + url: http://127.0.0.1:8009/v1/systemone + health_url: http://127.0.0.1:8009/v1/models + timeout_s: 300 + +backends: + jev: # Jev through OpenRouter + provider: systemone + model: typesafe/jev-1.13 + url: https://openrouter.ai/api/alpha/decisions + bearer_token_env: OPENROUTER_API_KEY + timeout_s: 60 + max_retries: 2 + +evaluation: + probability_tolerance: 0.02 +``` + +```bash +decguard test decguard.yaml -o kev.json +decguard test decguard.yaml --backend jev -o jev.json +decguard diff kev.json jev.json --contract decguard.yaml +``` + +## Settings + +`systemone` accepts every [`http` setting](./http.md#settings) — URL checks, `health_url`, +timeouts, retries, environment credentials, `label_map` — plus: + +| Setting | Default | Meaning | +| --- | --- | --- | +| `model` | required | the model to ask; sent with every request | +| `instructions` | `decision.description` | the question asked about every input | + +Differences from `http`: + +- retries (`max_retries`) also cover HTTP **429** and **529**, which TypeSafe and + OpenRouter ask clients to retry, besides connection failures and 502–504; +- `label_map` applies to `choice` and `score` decisions only, since no labels are sent + for `noul`. + +## Wire format + +Each case is sent as one question about the case input, keyed by the decision name: + +```json +{ + "model": "typesafe/jev-1.13", + "state": "I was charged twice for order #4411.", + "questions": { + "support_team": { + "type": "choice", + "instructions": "Which team should handle this customer support ticket?", + "criteria": {"billing": null, "shipping": null, "technical": null} + } + } +} +``` + +| Decision | Sent | Answer used | +| --- | --- | --- | +| `choice` | options as `criteria` keys, descriptions `null`, in the order shown | `probabilities` by option; `choice` must be the most probable option | +| `noul` | no labels (a yes/no question) | `noul` is the probability of the first (positive) contract label; the second gets `1 − noul` | +| `score` | levels as the ordered `criteria` list, lowest first | `probabilities` by level index (`"0"` = lowest); `legend`, when present, must echo the levels sent | + +During fuzzing, options and levels are sent in the order and form shown and mapped back +to the contract labels. A `noul` question has no labels, so `option_order` and +`label_format` send the same request for it. + +Answers are checked, never repaired. The response must answer exactly this question with +the same type; a missing or malformed answer is an `invalid_response` for that case. The +provider's own `confidence` field measures something else, so DecGuard ignores it: its +confidence is the selected label's probability, as for every backend. + +## What gets recorded + +Per result, DecGuard records the model that answered as `model_version` and adds to +`metadata`: + +- `request_id`; +- `upstream_provider` — OpenRouter's `provider` field; +- `usage` — `input_tokens`, `output_tokens` and `cost`. + +The report's backend section lists the `served_models` and `upstream_providers` seen during +the run. API keys never appear in reports, errors or terminal output. + +## Real-backend validation + +The DecGuard repository runs genuine inference through the `systemone` backend on three +small contracts in +[`tests/integration/real/`](https://github.com/suffro/decguard/tree/main/tests/integration/real) +— one `choice`, one `noul` and one `score` decision. Nothing is mocked. These tests are +opt-in: a default `pytest` run skips them, and once selected they fail rather than skip +when the server, checkpoint or credential is missing. + +**Kev** (`-m real_kev`), with a Kev server started as [above](#self-hosted-kev): + +```bash +DECGUARD_KEV_RUN=jaredpalmer/kev-0.8b@9a45d25eb2ab761841196625383fa1dff0e56c1e \ + uv run pytest -m real_kev +``` + +`DECGUARD_KEV_RUN` is optional: when set, the test fails unless the server's +`GET /v1/models` reports that checkpoint. + +**Jev through OpenRouter** (`-m real_jev`): + +```bash +export OPENROUTER_API_KEY=... +uv run pytest -m real_jev +``` + +This makes three billed Jev calls (about $0.00004 in total at the time of writing) and one +call with a deliberately invalid key, which OpenRouter rejects with HTTP 401 and does not +bill. The tests check that neither key appears in any output, error or report. + +**GitHub Actions.** The repository's manual `Real backends` workflow +([`.github/workflows/real-backends.yml`](https://github.com/suffro/decguard/blob/main/.github/workflows/real-backends.yml)) +runs both groups on `ubuntu-latest` and `macos-latest` with Python 3.13. It is started +from the Actions tab or by adding the `real-backends` label to a pull request, never on +ordinary commits. Only the Jev test step receives the `OPENROUTER_API_KEY` secret. + +**Any other backend** (`-m external`) runs `decguard test` on your own contract, which +must use a non-`mock` backend and produce at least one valid decision: + +```bash +DECGUARD_EXTERNAL_CONTRACT=/absolute/path/to/real-decguard.yaml \ + uv run pytest -m external +``` diff --git a/package-lock.json b/package-lock.json new file mode 100644 index 0000000..8e18f81 --- /dev/null +++ b/package-lock.json @@ -0,0 +1,2562 @@ +{ + "name": "decguard-docs", + "lockfileVersion": 3, + "requires": true, + "packages": { + "": { + "name": "decguard-docs", + "license": "Apache-2.0", + "devDependencies": { + "vitepress": "1.6.4" + } + }, + "node_modules/@algolia/abtesting": { + "version": "1.25.0", + "resolved": "https://registry.npmjs.org/@algolia/abtesting/-/abtesting-1.25.0.tgz", + "integrity": "sha512-rSTin9Uta23uaewYVQEp8XI9T3iA/zrg0/1G2vhf8oFFDxFL5vybnZ5IQwsVAg4JpKxPX4/WYNKdcfWrZymk7w==", + "dev": true, + "license": "MIT", + "dependencies": { + "@algolia/client-common": "5.59.0", + "@algolia/requester-browser-xhr": "5.59.0", + "@algolia/requester-fetch": "5.59.0", + "@algolia/requester-node-http": "5.59.0" + }, + "engines": { + "node": ">= 14.0.0" + } + }, + "node_modules/@algolia/autocomplete-core": { + "version": "1.17.7", + "resolved": "https://registry.npmjs.org/@algolia/autocomplete-core/-/autocomplete-core-1.17.7.tgz", + "integrity": "sha512-BjiPOW6ks90UKl7TwMv7oNQMnzU+t/wk9mgIDi6b1tXpUek7MW0lbNOUHpvam9pe3lVCf4xPFT+lK7s+e+fs7Q==", + "dev": true, + "license": "MIT", + "dependencies": { + "@algolia/autocomplete-plugin-algolia-insights": "1.17.7", + "@algolia/autocomplete-shared": "1.17.7" + } + }, + "node_modules/@algolia/autocomplete-plugin-algolia-insights": { + "version": "1.17.7", + "resolved": "https://registry.npmjs.org/@algolia/autocomplete-plugin-algolia-insights/-/autocomplete-plugin-algolia-insights-1.17.7.tgz", + "integrity": "sha512-Jca5Ude6yUOuyzjnz57og7Et3aXjbwCSDf/8onLHSQgw1qW3ALl9mrMWaXb5FmPVkV3EtkD2F/+NkT6VHyPu9A==", + "dev": true, + "license": "MIT", + "dependencies": { + "@algolia/autocomplete-shared": "1.17.7" + }, + "peerDependencies": { + "search-insights": ">= 1 < 3" + } + }, + "node_modules/@algolia/autocomplete-preset-algolia": { + "version": "1.17.7", + "resolved": "https://registry.npmjs.org/@algolia/autocomplete-preset-algolia/-/autocomplete-preset-algolia-1.17.7.tgz", + "integrity": "sha512-ggOQ950+nwbWROq2MOCIL71RE0DdQZsceqrg32UqnhDz8FlO9rL8ONHNsI2R1MH0tkgVIDKI/D0sMiUchsFdWA==", + "dev": true, + "license": "MIT", + "dependencies": { + "@algolia/autocomplete-shared": "1.17.7" + }, + "peerDependencies": { + "@algolia/client-search": ">= 4.9.1 < 6", + "algoliasearch": ">= 4.9.1 < 6" + } + }, + "node_modules/@algolia/autocomplete-shared": { + "version": "1.17.7", + "resolved": "https://registry.npmjs.org/@algolia/autocomplete-shared/-/autocomplete-shared-1.17.7.tgz", + "integrity": "sha512-o/1Vurr42U/qskRSuhBH+VKxMvkkUVTLU6WZQr+L5lGZZLYWyhdzWjW0iGXY7EkwRTjBqvN2EsR81yCTGV/kmg==", + "dev": true, + "license": "MIT", + "peerDependencies": { + "@algolia/client-search": ">= 4.9.1 < 6", + "algoliasearch": ">= 4.9.1 < 6" + } + }, + "node_modules/@algolia/client-abtesting": { + "version": "5.59.0", + "resolved": "https://registry.npmjs.org/@algolia/client-abtesting/-/client-abtesting-5.59.0.tgz", + "integrity": "sha512-bm2XN0hCSMYwStSsCBT0/PUB2BDxoyR1Lnub3c392HMEy9bi8PUSW8vR6zltVEKWG2t4PQFWMD5C07bmJxGgPg==", + "dev": true, + "license": "MIT", + "dependencies": { + "@algolia/client-common": "5.59.0", + "@algolia/requester-browser-xhr": "5.59.0", + "@algolia/requester-fetch": "5.59.0", + "@algolia/requester-node-http": "5.59.0" + }, + "engines": { + "node": ">= 14.0.0" + } + }, + "node_modules/@algolia/client-analytics": { + "version": "5.59.0", + "resolved": "https://registry.npmjs.org/@algolia/client-analytics/-/client-analytics-5.59.0.tgz", + "integrity": "sha512-XOFPOTa69WuqHR6c5tMgnUUwwqQgNSzMpxmhrgA9KmxRf8WIqEa0cokHJvohk5CYb7CZx0xeSL6Bk2IUJ7Lv7Q==", + "dev": true, + "license": "MIT", + "dependencies": { + "@algolia/client-common": "5.59.0", + "@algolia/requester-browser-xhr": "5.59.0", + "@algolia/requester-fetch": "5.59.0", + "@algolia/requester-node-http": "5.59.0" + }, + "engines": { + "node": ">= 14.0.0" + } + }, + "node_modules/@algolia/client-common": { + "version": "5.59.0", + "resolved": "https://registry.npmjs.org/@algolia/client-common/-/client-common-5.59.0.tgz", + "integrity": "sha512-PC8ipLOYFKRTfIUY1J3FJxS6ryzWziaXmIX9/sNMoUR8L+XhF7hX2QAeUI87YVl5zujvlYTSOb+HAIqKXDjyHQ==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">= 14.0.0" + } + }, + "node_modules/@algolia/client-insights": { + "version": "5.59.0", + "resolved": "https://registry.npmjs.org/@algolia/client-insights/-/client-insights-5.59.0.tgz", + "integrity": "sha512-yFNcCMM5fHiyoR0HuxMrzy+VjDcmhFUZTm2IJ2DHwGsVH3B5SEob4zTmeEZ3j/AZqNnmrJOCKl/tJnBg7+84bA==", + "dev": true, + "license": "MIT", + "dependencies": { + "@algolia/client-common": "5.59.0", + "@algolia/requester-browser-xhr": "5.59.0", + "@algolia/requester-fetch": "5.59.0", + "@algolia/requester-node-http": "5.59.0" + }, + "engines": { + "node": ">= 14.0.0" + } + }, + "node_modules/@algolia/client-personalization": { + "version": "5.59.0", + "resolved": "https://registry.npmjs.org/@algolia/client-personalization/-/client-personalization-5.59.0.tgz", + "integrity": "sha512-GYja6HkDt2VrQhWmB2cLx3Z5fDwI9no7q+xwCWcFrTPm5CLH9QZvK0XLO9co1FcNIDxxkMaP3AXM39/XcecEOg==", + "dev": true, + "license": "MIT", + "dependencies": { + "@algolia/client-common": "5.59.0", + "@algolia/requester-browser-xhr": "5.59.0", + "@algolia/requester-fetch": "5.59.0", + "@algolia/requester-node-http": "5.59.0" + }, + "engines": { + "node": ">= 14.0.0" + } + }, + "node_modules/@algolia/client-query-suggestions": { + "version": "5.59.0", + "resolved": "https://registry.npmjs.org/@algolia/client-query-suggestions/-/client-query-suggestions-5.59.0.tgz", + "integrity": "sha512-Wofg7bMpWh8N5qDDZs0wy6whc+KMmdNsxgrIGp9Ug2s1Bka0uq7AjyckdxReKPHLA0Q1qH8X/SNh0wn2t2X2Lw==", + "dev": true, + "license": "MIT", + "dependencies": { + "@algolia/client-common": "5.59.0", + "@algolia/requester-browser-xhr": "5.59.0", + "@algolia/requester-fetch": "5.59.0", + "@algolia/requester-node-http": "5.59.0" + }, + "engines": { + "node": ">= 14.0.0" + } + }, + "node_modules/@algolia/client-search": { + "version": "5.59.0", + "resolved": "https://registry.npmjs.org/@algolia/client-search/-/client-search-5.59.0.tgz", + "integrity": "sha512-fHnALZfbEnODczGk14Y/1YBRApp6UEpZUTexGcMUPzY7RDc7q4HN2Y6jh0KUG8Jo9B+q+wOoQEc3ziR0ErbuFg==", + "dev": true, + "license": "MIT", + "dependencies": { + "@algolia/client-common": "5.59.0", + "@algolia/requester-browser-xhr": "5.59.0", + "@algolia/requester-fetch": "5.59.0", + "@algolia/requester-node-http": "5.59.0" + }, + "engines": { + "node": ">= 14.0.0" + } + }, + "node_modules/@algolia/ingestion": { + "version": "1.59.0", + "resolved": "https://registry.npmjs.org/@algolia/ingestion/-/ingestion-1.59.0.tgz", + "integrity": "sha512-Fa38s1mHgoaLCT117sfJ6P78rtxUt93CYxBYpq1VOIcslaF+cH+1h5uikAPsKfxLnsuEDZHiBiaI6eNPTsQMRA==", + "dev": true, + "license": "MIT", + "dependencies": { + "@algolia/client-common": "5.59.0", + "@algolia/requester-browser-xhr": "5.59.0", + "@algolia/requester-fetch": "5.59.0", + "@algolia/requester-node-http": "5.59.0" + }, + "engines": { + "node": ">= 14.0.0" + } + }, + "node_modules/@algolia/monitoring": { + "version": "1.59.0", + "resolved": "https://registry.npmjs.org/@algolia/monitoring/-/monitoring-1.59.0.tgz", + "integrity": "sha512-NyNsRSqM2tF1MX7ZGw/j4rduoEJiQ5wfvbSo/CFVdEzYCjyxbGFRPOZyS/GETJG9rk1TdHOq8NZqKqk/u1fm4g==", + "dev": true, + "license": "MIT", + "dependencies": { + "@algolia/client-common": "5.59.0", + "@algolia/requester-browser-xhr": "5.59.0", + "@algolia/requester-fetch": "5.59.0", + "@algolia/requester-node-http": "5.59.0" + }, + "engines": { + "node": ">= 14.0.0" + } + }, + "node_modules/@algolia/recommend": { + "version": "5.59.0", + "resolved": "https://registry.npmjs.org/@algolia/recommend/-/recommend-5.59.0.tgz", + "integrity": "sha512-nXBK2uygWtvbCOffMWqqfjjcEtp9enDKY5/2Pw/Hhw8VVaOyK2ThbGK04bNX/Le+qoK1VHYkodWGVTKfw0Otkw==", + "dev": true, + "license": "MIT", + "dependencies": { + "@algolia/client-common": "5.59.0", + "@algolia/requester-browser-xhr": "5.59.0", + "@algolia/requester-fetch": "5.59.0", + "@algolia/requester-node-http": "5.59.0" + }, + "engines": { + "node": ">= 14.0.0" + } + }, + "node_modules/@algolia/requester-browser-xhr": { + "version": "5.59.0", + "resolved": "https://registry.npmjs.org/@algolia/requester-browser-xhr/-/requester-browser-xhr-5.59.0.tgz", + "integrity": "sha512-yb+4afX/zja8QwX0KmV4/ae2kyYkRknsE79CRusYS2U5D8qwn2wq0cn1x12f3tm3F3+5FgmDdbuGdQTuCLSKMQ==", + "dev": true, + "license": "MIT", + "dependencies": { + "@algolia/client-common": "5.59.0" + }, + "engines": { + "node": ">= 14.0.0" + } + }, + "node_modules/@algolia/requester-fetch": { + "version": "5.59.0", + "resolved": "https://registry.npmjs.org/@algolia/requester-fetch/-/requester-fetch-5.59.0.tgz", + "integrity": "sha512-Lp52TmpA1QtNmdHzs505Xwf4tgII7RAapkDSF9AAtWIBETcGaTIaXci4Rd9nS6SdaWK1BWTSqAPp5wzh6UN3Ag==", + "dev": true, + "license": "MIT", + "dependencies": { + "@algolia/client-common": "5.59.0" + }, + "engines": { + "node": ">= 14.0.0" + } + }, + "node_modules/@algolia/requester-node-http": { + "version": "5.59.0", + "resolved": "https://registry.npmjs.org/@algolia/requester-node-http/-/requester-node-http-5.59.0.tgz", + "integrity": "sha512-YYHLEs5rC6oRTFwT7bJaKBR9NSFzikeVHGAT1ffATmUQzR8XqyhHXjoieAUoR7fBU6I5cfy1FVeh5GoRgriuzA==", + "dev": true, + "license": "MIT", + "dependencies": { + "@algolia/client-common": "5.59.0" + }, + "engines": { + "node": ">= 14.0.0" + } + }, + "node_modules/@babel/helper-string-parser": { + "version": "7.29.7", + "resolved": "https://registry.npmjs.org/@babel/helper-string-parser/-/helper-string-parser-7.29.7.tgz", + "integrity": "sha512-Pb5ijPrZ89GDH8223L4UP8i6QApWxs04RbPQJTeWDV0/keR2E36MeKnyr6LYmUUvqRRI+Iv87SuF1W6ErINzYw==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=6.9.0" + } + }, + "node_modules/@babel/helper-validator-identifier": { + "version": "7.29.7", + "resolved": "https://registry.npmjs.org/@babel/helper-validator-identifier/-/helper-validator-identifier-7.29.7.tgz", + "integrity": "sha512-qehxGkRj55h/ff8EMaJ+cYhyaKlHIxqYDn682wQD7RNp9UujOQsHog2uS0r2vzr4pW+sXf90NeeayjcNaX3fFg==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=6.9.0" + } + }, + "node_modules/@babel/parser": { + "version": "7.29.9", + "resolved": "https://registry.npmjs.org/@babel/parser/-/parser-7.29.9.tgz", + "integrity": "sha512-CjXrNHTnvqBVqHgdBysY3vk2T8tpJHb5/RMeHJBTyVa9xgugCB0CJTx/3oO8RV2QRQP391RWpB7D6hLjm8V9uA==", + "dev": true, + "license": "MIT", + "dependencies": { + "@babel/types": "^7.29.8" + }, + "bin": { + "parser": "bin/babel-parser.js" + }, + "engines": { + "node": ">=6.0.0" + } + }, + "node_modules/@babel/types": { + "version": "7.29.8", + "resolved": "https://registry.npmjs.org/@babel/types/-/types-7.29.8.tgz", + "integrity": "sha512-Vj1jF3cPfxg7OAfoI7QnVKLoILlm2JF9pnVHrX8qx7AHMiYWT+NDAA7jChlNgRS4WTLc/fD1lXLmPixluj+3Gg==", + "dev": true, + "license": "MIT", + "dependencies": { + "@babel/helper-string-parser": "^7.29.7", + "@babel/helper-validator-identifier": "^7.29.7" + }, + "engines": { + "node": ">=6.9.0" + } + }, + "node_modules/@docsearch/css": { + "version": "3.8.2", + "resolved": "https://registry.npmjs.org/@docsearch/css/-/css-3.8.2.tgz", + "integrity": "sha512-y05ayQFyUmCXze79+56v/4HpycYF3uFqB78pLPrSV5ZKAlDuIAAJNhaRi8tTdRNXh05yxX/TyNnzD6LwSM89vQ==", + "dev": true, + "license": "MIT" + }, + "node_modules/@docsearch/js": { + "version": "3.8.2", + "resolved": "https://registry.npmjs.org/@docsearch/js/-/js-3.8.2.tgz", + "integrity": "sha512-Q5wY66qHn0SwA7Taa0aDbHiJvaFJLOJyHmooQ7y8hlwwQLQ/5WwCcoX0g7ii04Qi2DJlHsd0XXzJ8Ypw9+9YmQ==", + "dev": true, + "license": "MIT", + "dependencies": { + "@docsearch/react": "3.8.2", + "preact": "^10.0.0" + } + }, + "node_modules/@docsearch/react": { + "version": "3.8.2", + "resolved": "https://registry.npmjs.org/@docsearch/react/-/react-3.8.2.tgz", + "integrity": "sha512-xCRrJQlTt8N9GU0DG4ptwHRkfnSnD/YpdeaXe02iKfqs97TkZJv60yE+1eq/tjPcVnTW8dP5qLP7itifFVV5eg==", + "dev": true, + "license": "MIT", + "dependencies": { + "@algolia/autocomplete-core": "1.17.7", + "@algolia/autocomplete-preset-algolia": "1.17.7", + "@docsearch/css": "3.8.2", + "algoliasearch": "^5.14.2" + }, + "peerDependencies": { + "@types/react": ">= 16.8.0 < 19.0.0", + "react": ">= 16.8.0 < 19.0.0", + "react-dom": ">= 16.8.0 < 19.0.0", + "search-insights": ">= 1 < 3" + }, + "peerDependenciesMeta": { + "@types/react": { + "optional": true + }, + "react": { + "optional": true + }, + "react-dom": { + "optional": true + }, + "search-insights": { + "optional": true + } + } + }, + "node_modules/@esbuild/aix-ppc64": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/aix-ppc64/-/aix-ppc64-0.21.5.tgz", + "integrity": "sha512-1SDgH6ZSPTlggy1yI6+Dbkiz8xzpHJEVAlF/AM1tHPLsf5STom9rwtjE4hKAF20FfXXNTFqEYXyJNWh1GiZedQ==", + "cpu": [ + "ppc64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "aix" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/android-arm": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/android-arm/-/android-arm-0.21.5.tgz", + "integrity": "sha512-vCPvzSjpPHEi1siZdlvAlsPxXl7WbOVUBBAowWug4rJHb68Ox8KualB+1ocNvT5fjv6wpkX6o/iEpbDrf68zcg==", + "cpu": [ + "arm" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "android" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/android-arm64": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/android-arm64/-/android-arm64-0.21.5.tgz", + "integrity": "sha512-c0uX9VAUBQ7dTDCjq+wdyGLowMdtR/GoC2U5IYk/7D1H1JYC0qseD7+11iMP2mRLN9RcCMRcjC4YMclCzGwS/A==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "android" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/android-x64": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/android-x64/-/android-x64-0.21.5.tgz", + "integrity": "sha512-D7aPRUUNHRBwHxzxRvp856rjUHRFW1SdQATKXH2hqA0kAZb1hKmi02OpYRacl0TxIGz/ZmXWlbZgjwWYaCakTA==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "android" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/darwin-arm64": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/darwin-arm64/-/darwin-arm64-0.21.5.tgz", + "integrity": "sha512-DwqXqZyuk5AiWWf3UfLiRDJ5EDd49zg6O9wclZ7kUMv2WRFr4HKjXp/5t8JZ11QbQfUS6/cRCKGwYhtNAY88kQ==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "darwin" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/darwin-x64": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/darwin-x64/-/darwin-x64-0.21.5.tgz", + "integrity": "sha512-se/JjF8NlmKVG4kNIuyWMV/22ZaerB+qaSi5MdrXtd6R08kvs2qCN4C09miupktDitvh8jRFflwGFBQcxZRjbw==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "darwin" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/freebsd-arm64": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/freebsd-arm64/-/freebsd-arm64-0.21.5.tgz", + "integrity": "sha512-5JcRxxRDUJLX8JXp/wcBCy3pENnCgBR9bN6JsY4OmhfUtIHe3ZW0mawA7+RDAcMLrMIZaf03NlQiX9DGyB8h4g==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "freebsd" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/freebsd-x64": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/freebsd-x64/-/freebsd-x64-0.21.5.tgz", + "integrity": "sha512-J95kNBj1zkbMXtHVH29bBriQygMXqoVQOQYA+ISs0/2l3T9/kj42ow2mpqerRBxDJnmkUDCaQT/dfNXWX/ZZCQ==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "freebsd" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/linux-arm": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/linux-arm/-/linux-arm-0.21.5.tgz", + "integrity": "sha512-bPb5AHZtbeNGjCKVZ9UGqGwo8EUu4cLq68E95A53KlxAPRmUyYv2D6F0uUI65XisGOL1hBP5mTronbgo+0bFcA==", + "cpu": [ + "arm" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/linux-arm64": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/linux-arm64/-/linux-arm64-0.21.5.tgz", + "integrity": "sha512-ibKvmyYzKsBeX8d8I7MH/TMfWDXBF3db4qM6sy+7re0YXya+K1cem3on9XgdT2EQGMu4hQyZhan7TeQ8XkGp4Q==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/linux-ia32": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/linux-ia32/-/linux-ia32-0.21.5.tgz", + "integrity": "sha512-YvjXDqLRqPDl2dvRODYmmhz4rPeVKYvppfGYKSNGdyZkA01046pLWyRKKI3ax8fbJoK5QbxblURkwK/MWY18Tg==", + "cpu": [ + "ia32" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/linux-loong64": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/linux-loong64/-/linux-loong64-0.21.5.tgz", + "integrity": "sha512-uHf1BmMG8qEvzdrzAqg2SIG/02+4/DHB6a9Kbya0XDvwDEKCoC8ZRWI5JJvNdUjtciBGFQ5PuBlpEOXQj+JQSg==", + "cpu": [ + "loong64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/linux-mips64el": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/linux-mips64el/-/linux-mips64el-0.21.5.tgz", + "integrity": "sha512-IajOmO+KJK23bj52dFSNCMsz1QP1DqM6cwLUv3W1QwyxkyIWecfafnI555fvSGqEKwjMXVLokcV5ygHW5b3Jbg==", + "cpu": [ + "mips64el" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/linux-ppc64": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/linux-ppc64/-/linux-ppc64-0.21.5.tgz", + "integrity": "sha512-1hHV/Z4OEfMwpLO8rp7CvlhBDnjsC3CttJXIhBi+5Aj5r+MBvy4egg7wCbe//hSsT+RvDAG7s81tAvpL2XAE4w==", + "cpu": [ + "ppc64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/linux-riscv64": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/linux-riscv64/-/linux-riscv64-0.21.5.tgz", + "integrity": "sha512-2HdXDMd9GMgTGrPWnJzP2ALSokE/0O5HhTUvWIbD3YdjME8JwvSCnNGBnTThKGEB91OZhzrJ4qIIxk/SBmyDDA==", + "cpu": [ + "riscv64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/linux-s390x": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/linux-s390x/-/linux-s390x-0.21.5.tgz", + "integrity": "sha512-zus5sxzqBJD3eXxwvjN1yQkRepANgxE9lgOW2qLnmr8ikMTphkjgXu1HR01K4FJg8h1kEEDAqDcZQtbrRnB41A==", + "cpu": [ + "s390x" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/linux-x64": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/linux-x64/-/linux-x64-0.21.5.tgz", + "integrity": "sha512-1rYdTpyv03iycF1+BhzrzQJCdOuAOtaqHTWJZCWvijKD2N5Xu0TtVC8/+1faWqcP9iBCWOmjmhoH94dH82BxPQ==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/netbsd-x64": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/netbsd-x64/-/netbsd-x64-0.21.5.tgz", + "integrity": "sha512-Woi2MXzXjMULccIwMnLciyZH4nCIMpWQAs049KEeMvOcNADVxo0UBIQPfSmxB3CWKedngg7sWZdLvLczpe0tLg==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "netbsd" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/openbsd-x64": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/openbsd-x64/-/openbsd-x64-0.21.5.tgz", + "integrity": "sha512-HLNNw99xsvx12lFBUwoT8EVCsSvRNDVxNpjZ7bPn947b8gJPzeHWyNVhFsaerc0n3TsbOINvRP2byTZ5LKezow==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "openbsd" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/sunos-x64": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/sunos-x64/-/sunos-x64-0.21.5.tgz", + "integrity": "sha512-6+gjmFpfy0BHU5Tpptkuh8+uw3mnrvgs+dSPQXQOv3ekbordwnzTVEb4qnIvQcYXq6gzkyTnoZ9dZG+D4garKg==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "sunos" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/win32-arm64": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/win32-arm64/-/win32-arm64-0.21.5.tgz", + "integrity": "sha512-Z0gOTd75VvXqyq7nsl93zwahcTROgqvuAcYDUr+vOv8uHhNSKROyU961kgtCD1e95IqPKSQKH7tBTslnS3tA8A==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "win32" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/win32-ia32": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/win32-ia32/-/win32-ia32-0.21.5.tgz", + "integrity": "sha512-SWXFF1CL2RVNMaVs+BBClwtfZSvDgtL//G/smwAc5oVK/UPu2Gu9tIaRgFmYFFKrmg3SyAjSrElf0TiJ1v8fYA==", + "cpu": [ + "ia32" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "win32" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/win32-x64": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/win32-x64/-/win32-x64-0.21.5.tgz", + "integrity": "sha512-tQd/1efJuzPC6rCFwEvLtci/xNFcTZknmXs98FYDfGE4wP9ClFV98nyKrzJKVPMhdDnjzLhdUyMX4PsQAPjwIw==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "win32" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@iconify-json/simple-icons": { + "version": "1.2.97", + "resolved": "https://registry.npmjs.org/@iconify-json/simple-icons/-/simple-icons-1.2.97.tgz", + "integrity": "sha512-Z6PmM6jvWVU7rmLzqbxCLJA7SeqCnK6lMHXtjyzHPLZebwuoojBEcHHycppahDzEeN9xujRkrV5J8QpOhMyNrg==", + "dev": true, + "license": "CC0-1.0", + "dependencies": { + "@iconify/types": "*" + } + }, + "node_modules/@iconify/types": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/@iconify/types/-/types-2.0.0.tgz", + "integrity": "sha512-+wluvCrRhXrhyOmRDJ3q8mux9JkKy5SJ/v8ol2tu4FVjyYvtEzkc/3pK15ET6RKg4b4w4BmTk1+gsCUhf21Ykg==", + "dev": true, + "license": "MIT" + }, + "node_modules/@jridgewell/sourcemap-codec": { + "version": "1.6.0", + "resolved": "https://registry.npmjs.org/@jridgewell/sourcemap-codec/-/sourcemap-codec-1.6.0.tgz", + "integrity": "sha512-T7jf+5zgsZHwNJ4lvQ7/aezbyk0nNX+zJVWpmHA7VYsEx7a7qr5Rg5IbtJFqkgze5Y2sruq1RUY8Q837Od7iFw==", + "dev": true, + "license": "MIT" + }, + "node_modules/@napi-rs/lzma-linux-x64-gnu": { + "version": "1.5.1", + "resolved": "https://registry.npmjs.org/@napi-rs/lzma-linux-x64-gnu/-/lzma-linux-x64-gnu-1.5.1.tgz", + "integrity": "sha512-oTXEIha4SsuXdTA4Iyskj0kpdx2yVXdhd75c2v3xGrHFfVMsbhTPZU/nMPL4sWKo4pBHm3aucLaqGlF696dTyQ==", + "cpu": [ + "x64" + ], + "dev": true, + "libc": [ + "glibc" + ], + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": "^22.20 || ^24.12 || >=25" + } + }, + "node_modules/@rollup/rollup-android-arm-eabi": { + "version": "4.63.5", + "resolved": "https://registry.npmjs.org/@rollup/rollup-android-arm-eabi/-/rollup-android-arm-eabi-4.63.5.tgz", + "integrity": "sha512-J25QJU+B78T4FhhBsNpLJyVWOi31mwtpcMwywHmOKH65Q9IWGA81gPj+dnwlhU8wktVriYE+tFAaQgrnJRzAZg==", + "cpu": [ + "arm" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "android" + ] + }, + "node_modules/@rollup/rollup-android-arm64": { + "version": "4.63.5", + "resolved": "https://registry.npmjs.org/@rollup/rollup-android-arm64/-/rollup-android-arm64-4.63.5.tgz", + "integrity": "sha512-LDopB3zuZM5Ux9TT2luNEBJW/tYbGU2g1d+VpKk6I+gSKDb+/7sYE6M225gRQt4RbMX6MSwMsVR/phdjVUgRLg==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "android" + ] + }, + "node_modules/@rollup/rollup-darwin-arm64": { + "version": "4.63.5", + "resolved": "https://registry.npmjs.org/@rollup/rollup-darwin-arm64/-/rollup-darwin-arm64-4.63.5.tgz", + "integrity": "sha512-wlJEERGfeuHeBavCL2qVnNacOK43NDoZM4sjkeRPymd04OAE9T1zBqDJgmZ+CIsPTYKwdzpUC8vmOw84dwY4Tg==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "darwin" + ] + }, + "node_modules/@rollup/rollup-darwin-x64": { + "version": "4.63.5", + "resolved": "https://registry.npmjs.org/@rollup/rollup-darwin-x64/-/rollup-darwin-x64-4.63.5.tgz", + "integrity": "sha512-4nJJGg5jbo2wwPP4JP+LfEBA3bvP8rU9CLuhp7jWvq9sxEyhjQFTFdrqi+/dHEin/pd8jpT0vcehIpnZtmEdcQ==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "darwin" + ] + }, + "node_modules/@rollup/rollup-freebsd-arm64": { + "version": "4.63.5", + "resolved": "https://registry.npmjs.org/@rollup/rollup-freebsd-arm64/-/rollup-freebsd-arm64-4.63.5.tgz", + "integrity": "sha512-DrZbyCDF1hneuO6jRbvZ2D7+PIBM6yIwYnJpg2vIk58T+wuFpiaGZrfUr59lDWw45bg+IrpTGLPiNi/Fk4w3Cg==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "freebsd" + ] + }, + "node_modules/@rollup/rollup-freebsd-x64": { + "version": "4.63.5", + "resolved": "https://registry.npmjs.org/@rollup/rollup-freebsd-x64/-/rollup-freebsd-x64-4.63.5.tgz", + "integrity": "sha512-gqfUVMJMB3mehqywxp6hTBFfgtMQykZY19+cfiaYP0toIJLb/1DZRJHVkQQGP13W4TAwfZDWeg1qBcheTRioXQ==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "freebsd" + ] + }, + "node_modules/@rollup/rollup-linux-arm-gnueabihf": { + "version": "4.63.5", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-arm-gnueabihf/-/rollup-linux-arm-gnueabihf-4.63.5.tgz", + "integrity": "sha512-CFmhpvAwzSaWMlN3VN7UtmoTihlZNzoP0juQib5TQRnYUyDV8dXeWOp29sobWAT6gXl/hQgAClLlEiYozQG3OQ==", + "cpu": [ + "arm" + ], + "dev": true, + "libc": [ + "glibc" + ], + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-arm-musleabihf": { + "version": "4.63.5", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-arm-musleabihf/-/rollup-linux-arm-musleabihf-4.63.5.tgz", + "integrity": "sha512-Uc9H8eXCOayV6JLTH5bXKMId6qbhNHa818/BgYjm4jrlq3vZquC9cqyvHBw17xy5Mnj5f+I3gFK5JcEf3hSqrw==", + "cpu": [ + "arm" + ], + "dev": true, + "libc": [ + "musl" + ], + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-arm64-gnu": { + "version": "4.63.5", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-arm64-gnu/-/rollup-linux-arm64-gnu-4.63.5.tgz", + "integrity": "sha512-VcPr/szv/1BFw112Kt//fxulXt/JPqzzidU84iW68L2DdjnOO8QFUv2zTSYBEPHD6movBD4z+bbr5y60GYM7Jw==", + "cpu": [ + "arm64" + ], + "dev": true, + "libc": [ + "glibc" + ], + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-arm64-musl": { + "version": "4.63.5", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-arm64-musl/-/rollup-linux-arm64-musl-4.63.5.tgz", + "integrity": "sha512-BnxtJ5/91BrIHYIkGrmjz/lbMhqEHt1dPFqIxIFR+jPn0xVc/oUSCtIT089zfp5ufwGDlYz2UC+Fe1SRBpYFbQ==", + "cpu": [ + "arm64" + ], + "dev": true, + "libc": [ + "musl" + ], + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-loong64-gnu": { + "version": "4.63.5", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-loong64-gnu/-/rollup-linux-loong64-gnu-4.63.5.tgz", + "integrity": "sha512-LrYcHZwF+fAMNKHYTOQ5osWM4AZF7YF6D+XtsjDyEvljtt11twc+zHVXBLNEjxVSUnKYsOhvVz4Z213eW02COQ==", + "cpu": [ + "loong64" + ], + "dev": true, + "libc": [ + "glibc" + ], + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-loong64-musl": { + "version": "4.63.5", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-loong64-musl/-/rollup-linux-loong64-musl-4.63.5.tgz", + "integrity": "sha512-nj7QKQePAAUpCpJHtg0pR0W/b92A9NO17JS3BAQmHDn/yhmkir2p8llrKY9TOhleKIaSzy1JhxS3T9FVld6coA==", + "cpu": [ + "loong64" + ], + "dev": true, + "libc": [ + "musl" + ], + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-ppc64-gnu": { + "version": "4.63.5", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-ppc64-gnu/-/rollup-linux-ppc64-gnu-4.63.5.tgz", + "integrity": "sha512-5ylkX6dWMeBKge9nTU+Rxfb+ZfaCIJ9lRqIFaK0eAMcWp7OJbYnLveLgXmm0VrvuLKb8qIK+mHyH0qu88RM+iA==", + "cpu": [ + "ppc64" + ], + "dev": true, + "libc": [ + "glibc" + ], + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-ppc64-musl": { + "version": "4.63.5", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-ppc64-musl/-/rollup-linux-ppc64-musl-4.63.5.tgz", + "integrity": "sha512-oHK4ZHYFDKjZviK34I+NwgfbGxgI7ztrNxj2hPTSSNFgeq1a/lEd7dHV2fdGAuTH4Iym3RHJg+vAbWaWG4B7Zg==", + "cpu": [ + "ppc64" + ], + "dev": true, + "libc": [ + "musl" + ], + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-riscv64-gnu": { + "version": "4.63.5", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-riscv64-gnu/-/rollup-linux-riscv64-gnu-4.63.5.tgz", + "integrity": "sha512-UcetmHZ6XOXuUByiKZyQmb55ZPr0LABr3Ec/HB9wKZn6CEAFWZkE+hsJErJ9hbPBC7nI0dKuELx7CoV6IM7TMg==", + "cpu": [ + "riscv64" + ], + "dev": true, + "libc": [ + "glibc" + ], + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-riscv64-musl": { + "version": "4.63.5", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-riscv64-musl/-/rollup-linux-riscv64-musl-4.63.5.tgz", + "integrity": "sha512-C5CmDPQBtvjVo8cgQsBs+w6WB0JLkiixhgi6hVLV11hERWdn/p0XcPU2OUcZzac9BPOFq7SbaHFa8r3SWEysCQ==", + "cpu": [ + "riscv64" + ], + "dev": true, + "libc": [ + "musl" + ], + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-s390x-gnu": { + "version": "4.63.5", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-s390x-gnu/-/rollup-linux-s390x-gnu-4.63.5.tgz", + "integrity": "sha512-lHVQHJFKsuuxLMi3MQO9XVL8Tje3JR82CzB+QDKC5NWBcsIWuwsn9uIM5e3lBhI+fF1/s63qnyYqsg65+8rV/w==", + "cpu": [ + "s390x" + ], + "dev": true, + "libc": [ + "glibc" + ], + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-x64-gnu": { + "version": "4.63.5", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-x64-gnu/-/rollup-linux-x64-gnu-4.63.5.tgz", + "integrity": "sha512-3W9bTFcQNJn71cSJVM9RKIiZOy8DO/XLDii8Uv/Pm6WKqDRj7JV3ZfuXIEfyuy5LXpIzAbB/1M4Ukp9GKNa7nA==", + "cpu": [ + "x64" + ], + "dev": true, + "libc": [ + "glibc" + ], + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-x64-musl": { + "version": "4.63.5", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-x64-musl/-/rollup-linux-x64-musl-4.63.5.tgz", + "integrity": "sha512-VDC7rRJlee/scpki96GZ27Omf6yU87s1YXwVTpjE5841faVlDYYT565rgfmoR1U0sqL7z5ivQSDjcsF6VRXyBA==", + "cpu": [ + "x64" + ], + "dev": true, + "libc": [ + "musl" + ], + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-openbsd-x64": { + "version": "4.63.5", + "resolved": "https://registry.npmjs.org/@rollup/rollup-openbsd-x64/-/rollup-openbsd-x64-4.63.5.tgz", + "integrity": "sha512-z86Ok2p4pTdv5xqCKZsTooO7yBEiaJR/HzU3Wx8RmWsPoLppnMKROhJusQob8B3IE1ghC343kUW9rC2r+Wf3ig==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "openbsd" + ] + }, + "node_modules/@rollup/rollup-openharmony-arm64": { + "version": "4.63.5", + "resolved": "https://registry.npmjs.org/@rollup/rollup-openharmony-arm64/-/rollup-openharmony-arm64-4.63.5.tgz", + "integrity": "sha512-IzQmj+xXwQFGhMAMKMQVXkMwMZN3TqkJgAE0nSsqvVwWWciP4AIPMmWRqOQ2GfX7TUDZr+xqGFcBS36CRPGw0g==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "openharmony" + ] + }, + "node_modules/@rollup/rollup-win32-arm64-msvc": { + "version": "4.63.5", + "resolved": "https://registry.npmjs.org/@rollup/rollup-win32-arm64-msvc/-/rollup-win32-arm64-msvc-4.63.5.tgz", + "integrity": "sha512-F6qpTaPc9bwBH85kjy0/BLmLSW1uv7AoOXCoRIkg2arlgCYlWYcAbiMkvZuAcaWk9TpCRG//okznLAqLGshkMw==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "win32" + ] + }, + "node_modules/@rollup/rollup-win32-ia32-msvc": { + "version": "4.63.5", + "resolved": "https://registry.npmjs.org/@rollup/rollup-win32-ia32-msvc/-/rollup-win32-ia32-msvc-4.63.5.tgz", + "integrity": "sha512-igoDsTFhhwECBeGbUuLeIk7t8Y1apa+cs6mDWpx2EZ0ch7oEQgzHbFUXN9euoHekCAQzXdXApAGkV6jznS7tWw==", + "cpu": [ + "ia32" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "win32" + ] + }, + "node_modules/@rollup/rollup-win32-x64-gnu": { + "version": "4.63.5", + "resolved": "https://registry.npmjs.org/@rollup/rollup-win32-x64-gnu/-/rollup-win32-x64-gnu-4.63.5.tgz", + "integrity": "sha512-U3teMeMbXFmaM5D+OTJpsOXd+wV/qftIeYF9kBKL4v73641qyJmoXFtA28DQLsnmlyayEsTe72xpLHrArq6vHw==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "win32" + ] + }, + "node_modules/@rollup/rollup-win32-x64-msvc": { + "version": "4.63.5", + "resolved": "https://registry.npmjs.org/@rollup/rollup-win32-x64-msvc/-/rollup-win32-x64-msvc-4.63.5.tgz", + "integrity": "sha512-ypfC34F3RKXvCXBglGqGMsUSMKlgwd1HX9AOAlx9RoZZ6GaI42YHVeKpzg3JG+wpBUJYTG+NNZhqbDWL8tBZkw==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "win32" + ] + }, + "node_modules/@shikijs/core": { + "version": "2.5.0", + "resolved": "https://registry.npmjs.org/@shikijs/core/-/core-2.5.0.tgz", + "integrity": "sha512-uu/8RExTKtavlpH7XqnVYBrfBkUc20ngXiX9NSrBhOVZYv/7XQRKUyhtkeflY5QsxC0GbJThCerruZfsUaSldg==", + "dev": true, + "license": "MIT", + "dependencies": { + "@shikijs/engine-javascript": "2.5.0", + "@shikijs/engine-oniguruma": "2.5.0", + "@shikijs/types": "2.5.0", + "@shikijs/vscode-textmate": "^10.0.2", + "@types/hast": "^3.0.4", + "hast-util-to-html": "^9.0.4" + } + }, + "node_modules/@shikijs/engine-javascript": { + "version": "2.5.0", + "resolved": "https://registry.npmjs.org/@shikijs/engine-javascript/-/engine-javascript-2.5.0.tgz", + "integrity": "sha512-VjnOpnQf8WuCEZtNUdjjwGUbtAVKuZkVQ/5cHy/tojVVRIRtlWMYVjyWhxOmIq05AlSOv72z7hRNRGVBgQOl0w==", + "dev": true, + "license": "MIT", + "dependencies": { + "@shikijs/types": "2.5.0", + "@shikijs/vscode-textmate": "^10.0.2", + "oniguruma-to-es": "^3.1.0" + } + }, + "node_modules/@shikijs/engine-oniguruma": { + "version": "2.5.0", + "resolved": "https://registry.npmjs.org/@shikijs/engine-oniguruma/-/engine-oniguruma-2.5.0.tgz", + "integrity": "sha512-pGd1wRATzbo/uatrCIILlAdFVKdxImWJGQ5rFiB5VZi2ve5xj3Ax9jny8QvkaV93btQEwR/rSz5ERFpC5mKNIw==", + "dev": true, + "license": "MIT", + "dependencies": { + "@shikijs/types": "2.5.0", + "@shikijs/vscode-textmate": "^10.0.2" + } + }, + "node_modules/@shikijs/langs": { + "version": "2.5.0", + "resolved": "https://registry.npmjs.org/@shikijs/langs/-/langs-2.5.0.tgz", + "integrity": "sha512-Qfrrt5OsNH5R+5tJ/3uYBBZv3SuGmnRPejV9IlIbFH3HTGLDlkqgHymAlzklVmKBjAaVmkPkyikAV/sQ1wSL+w==", + "dev": true, + "license": "MIT", + "dependencies": { + "@shikijs/types": "2.5.0" + } + }, + "node_modules/@shikijs/themes": { + "version": "2.5.0", + "resolved": "https://registry.npmjs.org/@shikijs/themes/-/themes-2.5.0.tgz", + "integrity": "sha512-wGrk+R8tJnO0VMzmUExHR+QdSaPUl/NKs+a4cQQRWyoc3YFbUzuLEi/KWK1hj+8BfHRKm2jNhhJck1dfstJpiw==", + "dev": true, + "license": "MIT", + "dependencies": { + "@shikijs/types": "2.5.0" + } + }, + "node_modules/@shikijs/transformers": { + "version": "2.5.0", + "resolved": "https://registry.npmjs.org/@shikijs/transformers/-/transformers-2.5.0.tgz", + "integrity": "sha512-SI494W5X60CaUwgi8u4q4m4s3YAFSxln3tzNjOSYqq54wlVgz0/NbbXEb3mdLbqMBztcmS7bVTaEd2w0qMmfeg==", + "dev": true, + "license": "MIT", + "dependencies": { + "@shikijs/core": "2.5.0", + "@shikijs/types": "2.5.0" + } + }, + "node_modules/@shikijs/types": { + "version": "2.5.0", + "resolved": "https://registry.npmjs.org/@shikijs/types/-/types-2.5.0.tgz", + "integrity": "sha512-ygl5yhxki9ZLNuNpPitBWvcy9fsSKKaRuO4BAlMyagszQidxcpLAr0qiW/q43DtSIDxO6hEbtYLiFZNXO/hdGw==", + "dev": true, + "license": "MIT", + "dependencies": { + "@shikijs/vscode-textmate": "^10.0.2", + "@types/hast": "^3.0.4" + } + }, + "node_modules/@shikijs/vscode-textmate": { + "version": "10.0.2", + "resolved": "https://registry.npmjs.org/@shikijs/vscode-textmate/-/vscode-textmate-10.0.2.tgz", + "integrity": "sha512-83yeghZ2xxin3Nj8z1NMd/NCuca+gsYXswywDy5bHvwlWL8tpTQmzGeUuHd9FC3E/SBEMvzJRwWEOz5gGes9Qg==", + "dev": true, + "license": "MIT" + }, + "node_modules/@types/estree": { + "version": "1.0.9", + "resolved": "https://registry.npmjs.org/@types/estree/-/estree-1.0.9.tgz", + "integrity": "sha512-GhdPgy1el4/ImP05X05Uw4cw2/M93BCUmnEvWZNStlCzEKME4Fkk+YpoA5OiHNQmoS7Cafb8Xa3Pya8m1Qrzeg==", + "dev": true, + "license": "MIT" + }, + "node_modules/@types/hast": { + "version": "3.0.5", + "resolved": "https://registry.npmjs.org/@types/hast/-/hast-3.0.5.tgz", + "integrity": "sha512-rp/ezSWaD1m44dPKICGhiskI13nVr7qTloFwDa/IYkhhf5nzwP+zIQcIJh3WIFSBOy/H1PzB40jPjMDksN4F+g==", + "dev": true, + "license": "MIT", + "dependencies": { + "@types/unist": "*" + } + }, + "node_modules/@types/linkify-it": { + "version": "5.0.0", + "resolved": "https://registry.npmjs.org/@types/linkify-it/-/linkify-it-5.0.0.tgz", + "integrity": "sha512-sVDA58zAw4eWAffKOaQH5/5j3XeayukzDk+ewSsnv3p4yJEZHCCzMDiZM8e0OUrRvmpGZ85jf4yDHkHsgBNr9Q==", + "dev": true, + "license": "MIT" + }, + "node_modules/@types/markdown-it": { + "version": "14.2.0", + "resolved": "https://registry.npmjs.org/@types/markdown-it/-/markdown-it-14.2.0.tgz", + "integrity": "sha512-NoQ2yGlLWj4wpxMs+TYmRKk3thDrQ97agr7sFqfLsAlvoS8SNQuTrlObhFqG9iugdTtgOE9jpJ6FNM4ZGsa5xQ==", + "dev": true, + "license": "MIT", + "dependencies": { + "@types/linkify-it": "^5", + "@types/mdurl": "^2" + } + }, + "node_modules/@types/mdast": { + "version": "4.0.4", + "resolved": "https://registry.npmjs.org/@types/mdast/-/mdast-4.0.4.tgz", + "integrity": "sha512-kGaNbPh1k7AFzgpud/gMdvIm5xuECykRR+JnWKQno9TAXVa6WIVCGTPvYGekIDL4uwCZQSYbUxNBSb1aUo79oA==", + "dev": true, + "license": "MIT", + "dependencies": { + "@types/unist": "*" + } + }, + "node_modules/@types/mdurl": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/@types/mdurl/-/mdurl-2.0.0.tgz", + "integrity": "sha512-RGdgjQUZba5p6QEFAVx2OGb8rQDL/cPRG7GiedRzMcJ1tYnUANBncjbSB1NRGwbvjcPeikRABz2nshyPk1bhWg==", + "dev": true, + "license": "MIT" + }, + "node_modules/@types/unist": { + "version": "3.0.3", + "resolved": "https://registry.npmjs.org/@types/unist/-/unist-3.0.3.tgz", + "integrity": "sha512-ko/gIFJRv177XgZsZcBwnqJN5x/Gien8qNOn0D5bQU/zAzVf9Zt3BlcUiLqhV9y4ARk0GbT3tnUiPNgnTXzc/Q==", + "dev": true, + "license": "MIT" + }, + "node_modules/@types/web-bluetooth": { + "version": "0.0.21", + "resolved": "https://registry.npmjs.org/@types/web-bluetooth/-/web-bluetooth-0.0.21.tgz", + "integrity": "sha512-oIQLCGWtcFZy2JW77j9k8nHzAOpqMHLQejDA48XXMWH6tjCQHz5RCFz1bzsmROyL6PUm+LLnUiI4BCn221inxA==", + "dev": true, + "license": "MIT" + }, + "node_modules/@ungap/structured-clone": { + "version": "1.4.0", + "resolved": "https://registry.npmjs.org/@ungap/structured-clone/-/structured-clone-1.4.0.tgz", + "integrity": "sha512-1mEZtMKPM09vDmQt5y7YvmN2+DFTP7Tg0EWXdic8/C6VRnpb33e4ghisCIE3WZjsE2N8mf+QV1Zqh7ZFYLWInQ==", + "dev": true, + "license": "ISC" + }, + "node_modules/@vitejs/plugin-vue": { + "version": "5.2.4", + "resolved": "https://registry.npmjs.org/@vitejs/plugin-vue/-/plugin-vue-5.2.4.tgz", + "integrity": "sha512-7Yx/SXSOcQq5HiiV3orevHUFn+pmMB4cgbEkDYgnkUWb0WfeQ/wa2yFv6D5ICiCQOVpjA7vYDXrC7AGO8yjDHA==", + "dev": true, + "license": "MIT", + "engines": { + "node": "^18.0.0 || >=20.0.0" + }, + "peerDependencies": { + "vite": "^5.0.0 || ^6.0.0", + "vue": "^3.2.25" + } + }, + "node_modules/@vue/compiler-core": { + "version": "3.5.43", + "resolved": "https://registry.npmjs.org/@vue/compiler-core/-/compiler-core-3.5.43.tgz", + "integrity": "sha512-zdiLhnbe1QQqgDT8xZMpNmyqZ3qlI+/Q/FHQco57Kwl/b05HhCzN6eVGN9QU9rbga4CrS0H5SYY8VGHZCt/1Hg==", + "dev": true, + "license": "MIT", + "dependencies": { + "@babel/parser": "^7.29.8", + "@vue/shared": "3.5.43", + "entities": "^7.0.1", + "estree-walker": "^2.0.2", + "source-map-js": "^1.2.1" + } + }, + "node_modules/@vue/compiler-dom": { + "version": "3.5.43", + "resolved": "https://registry.npmjs.org/@vue/compiler-dom/-/compiler-dom-3.5.43.tgz", + "integrity": "sha512-PEZoAk3NQmsn/ejMzSOCyTYqwGqczrWm70PuhBKjjv1+TCoQAaO/zOqNwjV+honlNstT5ILxtc+8r8UUfj+iEQ==", + "dev": true, + "license": "MIT", + "dependencies": { + "@vue/compiler-core": "3.5.43", + "@vue/shared": "3.5.43" + } + }, + "node_modules/@vue/compiler-sfc": { + "version": "3.5.43", + "resolved": "https://registry.npmjs.org/@vue/compiler-sfc/-/compiler-sfc-3.5.43.tgz", + "integrity": "sha512-FCbrG3XNCRl+js3huuKx4IVHBLTvMkJhVepjbxSPu1gn4yWLaYtGNQdjJGZaMytXB6qb76qQDDmDSLy/vkmleQ==", + "dev": true, + "license": "MIT", + "dependencies": { + "@babel/parser": "^7.29.8", + "@vue/compiler-core": "3.5.43", + "@vue/compiler-dom": "3.5.43", + "@vue/compiler-ssr": "3.5.43", + "@vue/shared": "3.5.43", + "estree-walker": "^2.0.2", + "magic-string": "^0.30.21", + "postcss": "^8.5.28", + "source-map-js": "^1.2.1" + } + }, + "node_modules/@vue/compiler-ssr": { + "version": "3.5.43", + "resolved": "https://registry.npmjs.org/@vue/compiler-ssr/-/compiler-ssr-3.5.43.tgz", + "integrity": "sha512-GF62orf7KiJX9RqrHNGrYBudsQGD0OhJ5nDs90O8UiDuS40+YMYomiXu6w6EuvtXuRDc3MSNis3EaaSKAVSWpg==", + "dev": true, + "license": "MIT", + "dependencies": { + "@vue/compiler-dom": "3.5.43", + "@vue/shared": "3.5.43" + } + }, + "node_modules/@vue/devtools-api": { + "version": "7.7.10", + "resolved": "https://registry.npmjs.org/@vue/devtools-api/-/devtools-api-7.7.10.tgz", + "integrity": "sha512-KxtEpUOOpFz/qOGRrAwA36QF7DqIA+FXgCYit9mk9wjbaZt0sXOFz81ElOZtKA4HbWHUdwNjZHBFsFFyp5BZiA==", + "dev": true, + "license": "MIT", + "dependencies": { + "@vue/devtools-kit": "^7.7.10" + } + }, + "node_modules/@vue/devtools-kit": { + "version": "7.7.10", + "resolved": "https://registry.npmjs.org/@vue/devtools-kit/-/devtools-kit-7.7.10.tgz", + "integrity": "sha512-3WNi2Kq4tbpVbmhml7RiphmAt0279oh3fKNeWMQIrltfX8Q91b4i5PL8DtyNKdwmcsGrV4fg+erwWOmD05CLIw==", + "dev": true, + "license": "MIT", + "dependencies": { + "@vue/devtools-shared": "^7.7.10", + "birpc": "^2.3.0", + "hookable": "^5.5.3", + "mitt": "^3.0.1", + "perfect-debounce": "^1.0.0", + "speakingurl": "^14.0.1", + "superjson": "^2.2.2" + } + }, + "node_modules/@vue/devtools-shared": { + "version": "7.7.10", + "resolved": "https://registry.npmjs.org/@vue/devtools-shared/-/devtools-shared-7.7.10.tgz", + "integrity": "sha512-wOPslzB8vTvpxwdaOcR2qAbwmuSP0L+rhpoC6Cf56V3Jip+HWb7PQQXOUPgBNQARpXsbQX/+mvi8kKucmBGRwQ==", + "dev": true, + "license": "MIT", + "dependencies": { + "rfdc": "^1.4.1" + } + }, + "node_modules/@vue/reactivity": { + "version": "3.5.43", + "resolved": "https://registry.npmjs.org/@vue/reactivity/-/reactivity-3.5.43.tgz", + "integrity": "sha512-G/c9GyOZNI2jVaaS6OX1EF1SSFSv7H0ERqNTl4+DTFMlZmB5eVAB53aLQNam/7NL2NPtaDD7RdVrzf8uJzMuOA==", + "dev": true, + "license": "MIT", + "dependencies": { + "@vue/shared": "3.5.43" + } + }, + "node_modules/@vue/runtime-core": { + "version": "3.5.43", + "resolved": "https://registry.npmjs.org/@vue/runtime-core/-/runtime-core-3.5.43.tgz", + "integrity": "sha512-hU6U6VnVhBGQDpvlnnDlIB8ZGJBiOcgk2lh/0InltHiz3D8oSkluvuvY+do1G2H3+udeKFsmaBlgVYP7gXQzEw==", + "dev": true, + "license": "MIT", + "dependencies": { + "@vue/reactivity": "3.5.43", + "@vue/shared": "3.5.43" + } + }, + "node_modules/@vue/runtime-dom": { + "version": "3.5.43", + "resolved": "https://registry.npmjs.org/@vue/runtime-dom/-/runtime-dom-3.5.43.tgz", + "integrity": "sha512-Bb2Jc0YjjJdMt1SJmb9b2L/IWd3I8lIT9x9eS/xvvP9CiVgna0ffua74xKRmt4/uSJ+0r4iN8ex1jrqkhQGWQw==", + "dev": true, + "license": "MIT", + "dependencies": { + "@vue/reactivity": "3.5.43", + "@vue/runtime-core": "3.5.43", + "@vue/shared": "3.5.43", + "csstype": "^3.2.3" + } + }, + "node_modules/@vue/server-renderer": { + "version": "3.5.43", + "resolved": "https://registry.npmjs.org/@vue/server-renderer/-/server-renderer-3.5.43.tgz", + "integrity": "sha512-l2Ygjv9NehV94PSBxNWsAHC0j/eIIKbn92mBuWXAPGLnn6HfJ6MH5ubsd+Nk0YoZ5FRuxWI1P2VSoh+dbPQhCQ==", + "dev": true, + "license": "MIT", + "dependencies": { + "@vue/compiler-ssr": "3.5.43", + "@vue/runtime-dom": "3.5.43", + "@vue/shared": "3.5.43" + } + }, + "node_modules/@vue/shared": { + "version": "3.5.43", + "resolved": "https://registry.npmjs.org/@vue/shared/-/shared-3.5.43.tgz", + "integrity": "sha512-uksS7YGMR5NZyr4JNq0Rp+QyLns0ueaz20KwzIPW9R0LH1Vnt4E+XUM29PNseEbf1www2gOuhuDi5AKOIXag9Q==", + "dev": true, + "license": "MIT" + }, + "node_modules/@vueuse/core": { + "version": "12.8.2", + "resolved": "https://registry.npmjs.org/@vueuse/core/-/core-12.8.2.tgz", + "integrity": "sha512-HbvCmZdzAu3VGi/pWYm5Ut+Kd9mn1ZHnn4L5G8kOQTPs/IwIAmJoBrmYk2ckLArgMXZj0AW3n5CAejLUO+PhdQ==", + "dev": true, + "license": "MIT", + "dependencies": { + "@types/web-bluetooth": "^0.0.21", + "@vueuse/metadata": "12.8.2", + "@vueuse/shared": "12.8.2", + "vue": "^3.5.13" + }, + "funding": { + "url": "https://github.com/sponsors/antfu" + } + }, + "node_modules/@vueuse/integrations": { + "version": "12.8.2", + "resolved": "https://registry.npmjs.org/@vueuse/integrations/-/integrations-12.8.2.tgz", + "integrity": "sha512-fbGYivgK5uBTRt7p5F3zy6VrETlV9RtZjBqd1/HxGdjdckBgBM4ugP8LHpjolqTj14TXTxSK1ZfgPbHYyGuH7g==", + "dev": true, + "license": "MIT", + "dependencies": { + "@vueuse/core": "12.8.2", + "@vueuse/shared": "12.8.2", + "vue": "^3.5.13" + }, + "funding": { + "url": "https://github.com/sponsors/antfu" + }, + "peerDependencies": { + "async-validator": "^4", + "axios": "^1", + "change-case": "^5", + "drauu": "^0.4", + "focus-trap": "^7", + "fuse.js": "^7", + "idb-keyval": "^6", + "jwt-decode": "^4", + "nprogress": "^0.2", + "qrcode": "^1.5", + "sortablejs": "^1", + "universal-cookie": "^7" + }, + "peerDependenciesMeta": { + "async-validator": { + "optional": true + }, + "axios": { + "optional": true + }, + "change-case": { + "optional": true + }, + "drauu": { + "optional": true + }, + "focus-trap": { + "optional": true + }, + "fuse.js": { + "optional": true + }, + "idb-keyval": { + "optional": true + }, + "jwt-decode": { + "optional": true + }, + "nprogress": { + "optional": true + }, + "qrcode": { + "optional": true + }, + "sortablejs": { + "optional": true + }, + "universal-cookie": { + "optional": true + } + } + }, + "node_modules/@vueuse/metadata": { + "version": "12.8.2", + "resolved": "https://registry.npmjs.org/@vueuse/metadata/-/metadata-12.8.2.tgz", + "integrity": "sha512-rAyLGEuoBJ/Il5AmFHiziCPdQzRt88VxR+Y/A/QhJ1EWtWqPBBAxTAFaSkviwEuOEZNtW8pvkPgoCZQ+HxqW1A==", + "dev": true, + "license": "MIT", + "funding": { + "url": "https://github.com/sponsors/antfu" + } + }, + "node_modules/@vueuse/shared": { + "version": "12.8.2", + "resolved": "https://registry.npmjs.org/@vueuse/shared/-/shared-12.8.2.tgz", + "integrity": "sha512-dznP38YzxZoNloI0qpEfpkms8knDtaoQ6Y/sfS0L7Yki4zh40LFHEhur0odJC6xTHG5dxWVPiUWBXn+wCG2s5w==", + "dev": true, + "license": "MIT", + "dependencies": { + "vue": "^3.5.13" + }, + "funding": { + "url": "https://github.com/sponsors/antfu" + } + }, + "node_modules/algoliasearch": { + "version": "5.59.0", + "resolved": "https://registry.npmjs.org/algoliasearch/-/algoliasearch-5.59.0.tgz", + "integrity": "sha512-wUXzaeI7B526W4y1gFg3lcxgDZ67XSgRJIiellYWOas/pLpO7rOtxm9Gr2E/C8aiSDXIgx1q5TdSsvK67Uakqw==", + "dev": true, + "license": "MIT", + "dependencies": { + "@algolia/abtesting": "1.25.0", + "@algolia/client-abtesting": "5.59.0", + "@algolia/client-analytics": "5.59.0", + "@algolia/client-common": "5.59.0", + "@algolia/client-insights": "5.59.0", + "@algolia/client-personalization": "5.59.0", + "@algolia/client-query-suggestions": "5.59.0", + "@algolia/client-search": "5.59.0", + "@algolia/ingestion": "1.59.0", + "@algolia/monitoring": "1.59.0", + "@algolia/recommend": "5.59.0", + "@algolia/requester-browser-xhr": "5.59.0", + "@algolia/requester-fetch": "5.59.0", + "@algolia/requester-node-http": "5.59.0" + }, + "engines": { + "node": ">= 14.0.0" + } + }, + "node_modules/birpc": { + "version": "2.9.0", + "resolved": "https://registry.npmjs.org/birpc/-/birpc-2.9.0.tgz", + "integrity": "sha512-KrayHS5pBi69Xi9JmvoqrIgYGDkD6mcSe/i6YKi3w5kekCLzrX4+nawcXqrj2tIp50Kw/mT/s3p+GVK0A0sKxw==", + "dev": true, + "license": "MIT", + "funding": { + "url": "https://github.com/sponsors/antfu" + } + }, + "node_modules/ccount": { + "version": "2.0.1", + "resolved": "https://registry.npmjs.org/ccount/-/ccount-2.0.1.tgz", + "integrity": "sha512-eyrF0jiFpY+3drT6383f1qhkbGsLSifNAjA61IUjZjmLCWjItY6LB9ft9YhoDgwfmclB2zhu51Lc7+95b8NRAg==", + "dev": true, + "license": "MIT", + "funding": { + "type": "github", + "url": "https://github.com/sponsors/wooorm" + } + }, + "node_modules/character-entities-html4": { + "version": "2.1.0", + "resolved": "https://registry.npmjs.org/character-entities-html4/-/character-entities-html4-2.1.0.tgz", + "integrity": "sha512-1v7fgQRj6hnSwFpq1Eu0ynr/CDEw0rXo2B61qXrLNdHZmPKgb7fqS1a2JwF0rISo9q77jDI8VMEHoApn8qDoZA==", + "dev": true, + "license": "MIT", + "funding": { + "type": "github", + "url": "https://github.com/sponsors/wooorm" + } + }, + "node_modules/character-entities-legacy": { + "version": "3.0.0", + "resolved": "https://registry.npmjs.org/character-entities-legacy/-/character-entities-legacy-3.0.0.tgz", + "integrity": "sha512-RpPp0asT/6ufRm//AJVwpViZbGM/MkjQFxJccQRHmISF/22NBtsHqAWmL+/pmkPWoIUJdWyeVleTl1wydHATVQ==", + "dev": true, + "license": "MIT", + "funding": { + "type": "github", + "url": "https://github.com/sponsors/wooorm" + } + }, + "node_modules/comma-separated-tokens": { + "version": "2.0.3", + "resolved": "https://registry.npmjs.org/comma-separated-tokens/-/comma-separated-tokens-2.0.3.tgz", + "integrity": "sha512-Fu4hJdvzeylCfQPp9SGWidpzrMs7tTrlu6Vb8XGaRGck8QSNZJJp538Wrb60Lax4fPwR64ViY468OIUTbRlGZg==", + "dev": true, + "license": "MIT", + "funding": { + "type": "github", + "url": "https://github.com/sponsors/wooorm" + } + }, + "node_modules/copy-anything": { + "version": "4.1.1", + "resolved": "https://registry.npmjs.org/copy-anything/-/copy-anything-4.1.1.tgz", + "integrity": "sha512-AoT6Imdr98feSpFfmFwTFN73ccdr7uFPf27cBCgYvyyRyn1BzLRxMvrHNmwXO5LJMddRy4Rdhw2b1h7vSMKsEw==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=18" + }, + "funding": { + "url": "https://github.com/sponsors/mesqueeb" + } + }, + "node_modules/csstype": { + "version": "3.2.3", + "resolved": "https://registry.npmjs.org/csstype/-/csstype-3.2.3.tgz", + "integrity": "sha512-z1HGKcYy2xA8AGQfwrn0PAy+PB7X/GSj3UVJW9qKyn43xWa+gl5nXmU4qqLMRzWVLFC8KusUX8T/0kCiOYpAIQ==", + "dev": true, + "license": "MIT" + }, + "node_modules/dequal": { + "version": "2.0.3", + "resolved": "https://registry.npmjs.org/dequal/-/dequal-2.0.3.tgz", + "integrity": "sha512-0je+qPKHEMohvfRTCEo3CrPG6cAzAYgmzKyxRiYSSDkS6eGJdyVJm7WaYA5ECaAD9wLB2T4EEeymA5aFVcYXCA==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=6" + } + }, + "node_modules/devlop": { + "version": "1.1.0", + "resolved": "https://registry.npmjs.org/devlop/-/devlop-1.1.0.tgz", + "integrity": "sha512-RWmIqhcFf1lRYBvNmr7qTNuyCt/7/ns2jbpp1+PalgE/rDQcBT0fioSMUpJ93irlUhC5hrg4cYqe6U+0ImW0rA==", + "dev": true, + "license": "MIT", + "dependencies": { + "dequal": "^2.0.0" + }, + "funding": { + "type": "github", + "url": "https://github.com/sponsors/wooorm" + } + }, + "node_modules/emoji-regex-xs": { + "version": "1.0.0", + "resolved": "https://registry.npmjs.org/emoji-regex-xs/-/emoji-regex-xs-1.0.0.tgz", + "integrity": "sha512-LRlerrMYoIDrT6jgpeZ2YYl/L8EulRTt5hQcYjy5AInh7HWXKimpqx68aknBFpGL2+/IcogTcaydJEgaTmOpDg==", + "dev": true, + "license": "MIT" + }, + "node_modules/entities": { + "version": "7.0.1", + "resolved": "https://registry.npmjs.org/entities/-/entities-7.0.1.tgz", + "integrity": "sha512-TWrgLOFUQTH994YUyl1yT4uyavY5nNB5muff+RtWaqNVCAK408b5ZnnbNAUEWLTCpum9w6arT70i1XdQ4UeOPA==", + "dev": true, + "license": "BSD-2-Clause", + "engines": { + "node": ">=0.12" + }, + "funding": { + "url": "https://github.com/fb55/entities?sponsor=1" + } + }, + "node_modules/esbuild": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/esbuild/-/esbuild-0.21.5.tgz", + "integrity": "sha512-mg3OPMV4hXywwpoDxu3Qda5xCKQi+vCTZq8S9J/EpkhB2HzKXq4SNFZE3+NK93JYxc8VMSep+lOUSC/RVKaBqw==", + "dev": true, + "hasInstallScript": true, + "license": "MIT", + "bin": { + "esbuild": "bin/esbuild" + }, + "engines": { + "node": ">=12" + }, + "optionalDependencies": { + "@esbuild/aix-ppc64": "0.21.5", + "@esbuild/android-arm": "0.21.5", + "@esbuild/android-arm64": "0.21.5", + "@esbuild/android-x64": "0.21.5", + "@esbuild/darwin-arm64": "0.21.5", + "@esbuild/darwin-x64": "0.21.5", + "@esbuild/freebsd-arm64": "0.21.5", + "@esbuild/freebsd-x64": "0.21.5", + "@esbuild/linux-arm": "0.21.5", + "@esbuild/linux-arm64": "0.21.5", + "@esbuild/linux-ia32": "0.21.5", + "@esbuild/linux-loong64": "0.21.5", + "@esbuild/linux-mips64el": "0.21.5", + "@esbuild/linux-ppc64": "0.21.5", + "@esbuild/linux-riscv64": "0.21.5", + "@esbuild/linux-s390x": "0.21.5", + "@esbuild/linux-x64": "0.21.5", + "@esbuild/netbsd-x64": "0.21.5", + "@esbuild/openbsd-x64": "0.21.5", + "@esbuild/sunos-x64": "0.21.5", + "@esbuild/win32-arm64": "0.21.5", + "@esbuild/win32-ia32": "0.21.5", + "@esbuild/win32-x64": "0.21.5" + } + }, + "node_modules/estree-walker": { + "version": "2.0.2", + "resolved": "https://registry.npmjs.org/estree-walker/-/estree-walker-2.0.2.tgz", + "integrity": "sha512-Rfkk/Mp/DL7JVje3u18FxFujQlTNR2q6QfMSMB7AvCBx91NGj/ba3kCfza0f6dVDbw7YlRf/nDrn7pQrCCyQ/w==", + "dev": true, + "license": "MIT" + }, + "node_modules/focus-trap": { + "version": "7.8.0", + "resolved": "https://registry.npmjs.org/focus-trap/-/focus-trap-7.8.0.tgz", + "integrity": "sha512-/yNdlIkpWbM0ptxno3ONTuf+2g318kh2ez3KSeZN5dZ8YC6AAmgeWz+GasYYiBJPFaYcSAPeu4GfhUaChzIJXA==", + "dev": true, + "license": "MIT", + "dependencies": { + "tabbable": "^6.4.0" + } + }, + "node_modules/fsevents": { + "version": "2.3.3", + "resolved": "https://registry.npmjs.org/fsevents/-/fsevents-2.3.3.tgz", + "integrity": "sha512-5xoDfX+fL7faATnagmWPpbFtwh/R77WmMMqqHGS65C3vvB0YHrgF+B1YmZ3441tMj5n63k0212XNoJwzlhffQw==", + "dev": true, + "hasInstallScript": true, + "license": "MIT", + "optional": true, + "os": [ + "darwin" + ], + "engines": { + "node": "^8.16.0 || ^10.6.0 || >=11.0.0" + } + }, + "node_modules/hast-util-to-html": { + "version": "9.0.5", + "resolved": "https://registry.npmjs.org/hast-util-to-html/-/hast-util-to-html-9.0.5.tgz", + "integrity": "sha512-OguPdidb+fbHQSU4Q4ZiLKnzWo8Wwsf5bZfbvu7//a9oTYoqD/fWpe96NuHkoS9h0ccGOTe0C4NGXdtS0iObOw==", + "dev": true, + "license": "MIT", + "dependencies": { + "@types/hast": "^3.0.0", + "@types/unist": "^3.0.0", + "ccount": "^2.0.0", + "comma-separated-tokens": "^2.0.0", + "hast-util-whitespace": "^3.0.0", + "html-void-elements": "^3.0.0", + "mdast-util-to-hast": "^13.0.0", + "property-information": "^7.0.0", + "space-separated-tokens": "^2.0.0", + "stringify-entities": "^4.0.0", + "zwitch": "^2.0.4" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/unified" + } + }, + "node_modules/hast-util-whitespace": { + "version": "3.0.0", + "resolved": "https://registry.npmjs.org/hast-util-whitespace/-/hast-util-whitespace-3.0.0.tgz", + "integrity": "sha512-88JUN06ipLwsnv+dVn+OIYOvAuvBMy/Qoi6O7mQHxdPXpjy+Cd6xRkWwux7DKO+4sYILtLBRIKgsdpS2gQc7qw==", + "dev": true, + "license": "MIT", + "dependencies": { + "@types/hast": "^3.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/unified" + } + }, + "node_modules/hookable": { + "version": "5.5.3", + "resolved": "https://registry.npmjs.org/hookable/-/hookable-5.5.3.tgz", + "integrity": "sha512-Yc+BQe8SvoXH1643Qez1zqLRmbA5rCL+sSmk6TVos0LWVfNIB7PGncdlId77WzLGSIB5KaWgTaNTs2lNVEI6VQ==", + "dev": true, + "license": "MIT" + }, + "node_modules/html-void-elements": { + "version": "3.0.0", + "resolved": "https://registry.npmjs.org/html-void-elements/-/html-void-elements-3.0.0.tgz", + "integrity": "sha512-bEqo66MRXsUGxWHV5IP0PUiAWwoEjba4VCzg0LjFJBpchPaTfyfCKTG6bc5F8ucKec3q5y6qOdGyYTSBEvhCrg==", + "dev": true, + "license": "MIT", + "funding": { + "type": "github", + "url": "https://github.com/sponsors/wooorm" + } + }, + "node_modules/magic-string": { + "version": "0.30.21", + "resolved": "https://registry.npmjs.org/magic-string/-/magic-string-0.30.21.tgz", + "integrity": "sha512-vd2F4YUyEXKGcLHoq+TEyCjxueSeHnFxyyjNp80yg0XV4vUhnDer/lvvlqM/arB5bXQN5K2/3oinyCRyx8T2CQ==", + "dev": true, + "license": "MIT", + "dependencies": { + "@jridgewell/sourcemap-codec": "^1.5.5" + } + }, + "node_modules/mark.js": { + "version": "8.11.1", + "resolved": "https://registry.npmjs.org/mark.js/-/mark.js-8.11.1.tgz", + "integrity": "sha512-1I+1qpDt4idfgLQG+BNWmrqku+7/2bi5nLf4YwF8y8zXvmfiTBY3PV3ZibfrjBueCByROpuBjLLFCajqkgYoLQ==", + "dev": true, + "license": "MIT" + }, + "node_modules/mdast-util-to-hast": { + "version": "13.2.1", + "resolved": "https://registry.npmjs.org/mdast-util-to-hast/-/mdast-util-to-hast-13.2.1.tgz", + "integrity": "sha512-cctsq2wp5vTsLIcaymblUriiTcZd0CwWtCbLvrOzYCDZoWyMNV8sZ7krj09FSnsiJi3WVsHLM4k6Dq/yaPyCXA==", + "dev": true, + "license": "MIT", + "dependencies": { + "@types/hast": "^3.0.0", + "@types/mdast": "^4.0.0", + "@ungap/structured-clone": "^1.0.0", + "devlop": "^1.0.0", + "micromark-util-sanitize-uri": "^2.0.0", + "trim-lines": "^3.0.0", + "unist-util-position": "^5.0.0", + "unist-util-visit": "^5.0.0", + "vfile": "^6.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/unified" + } + }, + "node_modules/micromark-util-character": { + "version": "2.1.1", + "resolved": "https://registry.npmjs.org/micromark-util-character/-/micromark-util-character-2.1.1.tgz", + "integrity": "sha512-wv8tdUTJ3thSFFFJKtpYKOYiGP2+v96Hvk4Tu8KpCAsTMs6yi+nVmGh1syvSCsaxz45J6Jbw+9DD6g97+NV67Q==", + "dev": true, + "funding": [ + { + "type": "GitHub Sponsors", + "url": "https://github.com/sponsors/unifiedjs" + }, + { + "type": "OpenCollective", + "url": "https://opencollective.com/unified" + } + ], + "license": "MIT", + "dependencies": { + "micromark-util-symbol": "^2.0.0", + "micromark-util-types": "^2.0.0" + } + }, + "node_modules/micromark-util-encode": { + "version": "2.0.1", + "resolved": "https://registry.npmjs.org/micromark-util-encode/-/micromark-util-encode-2.0.1.tgz", + "integrity": "sha512-c3cVx2y4KqUnwopcO9b/SCdo2O67LwJJ/UyqGfbigahfegL9myoEFoDYZgkT7f36T0bLrM9hZTAaAyH+PCAXjw==", + "dev": true, + "funding": [ + { + "type": "GitHub Sponsors", + "url": "https://github.com/sponsors/unifiedjs" + }, + { + "type": "OpenCollective", + "url": "https://opencollective.com/unified" + } + ], + "license": "MIT" + }, + "node_modules/micromark-util-sanitize-uri": { + "version": "2.0.1", + "resolved": "https://registry.npmjs.org/micromark-util-sanitize-uri/-/micromark-util-sanitize-uri-2.0.1.tgz", + "integrity": "sha512-9N9IomZ/YuGGZZmQec1MbgxtlgougxTodVwDzzEouPKo3qFWvymFHWcnDi2vzV1ff6kas9ucW+o3yzJK9YB1AQ==", + "dev": true, + "funding": [ + { + "type": "GitHub Sponsors", + "url": "https://github.com/sponsors/unifiedjs" + }, + { + "type": "OpenCollective", + "url": "https://opencollective.com/unified" + } + ], + "license": "MIT", + "dependencies": { + "micromark-util-character": "^2.0.0", + "micromark-util-encode": "^2.0.0", + "micromark-util-symbol": "^2.0.0" + } + }, + "node_modules/micromark-util-symbol": { + "version": "2.0.1", + "resolved": "https://registry.npmjs.org/micromark-util-symbol/-/micromark-util-symbol-2.0.1.tgz", + "integrity": "sha512-vs5t8Apaud9N28kgCrRUdEed4UJ+wWNvicHLPxCa9ENlYuAY31M0ETy5y1vA33YoNPDFTghEbnh6efaE8h4x0Q==", + "dev": true, + "funding": [ + { + "type": "GitHub Sponsors", + "url": "https://github.com/sponsors/unifiedjs" + }, + { + "type": "OpenCollective", + "url": "https://opencollective.com/unified" + } + ], + "license": "MIT" + }, + "node_modules/micromark-util-types": { + "version": "2.0.2", + "resolved": "https://registry.npmjs.org/micromark-util-types/-/micromark-util-types-2.0.2.tgz", + "integrity": "sha512-Yw0ECSpJoViF1qTU4DC6NwtC4aWGt1EkzaQB8KPPyCRR8z9TWeV0HbEFGTO+ZY1wB22zmxnJqhPyTpOVCpeHTA==", + "dev": true, + "funding": [ + { + "type": "GitHub Sponsors", + "url": "https://github.com/sponsors/unifiedjs" + }, + { + "type": "OpenCollective", + "url": "https://opencollective.com/unified" + } + ], + "license": "MIT" + }, + "node_modules/minisearch": { + "version": "7.2.0", + "resolved": "https://registry.npmjs.org/minisearch/-/minisearch-7.2.0.tgz", + "integrity": "sha512-dqT2XBYUOZOiC5t2HRnwADjhNS2cecp9u+TJRiJ1Qp/f5qjkeT5APcGPjHw+bz89Ms8Jp+cG4AlE+QZ/QnDglg==", + "dev": true, + "license": "MIT" + }, + "node_modules/mitt": { + "version": "3.0.1", + "resolved": "https://registry.npmjs.org/mitt/-/mitt-3.0.1.tgz", + "integrity": "sha512-vKivATfr97l2/QBCYAkXYDbrIWPM2IIKEl7YPhjCvKlG3kE2gm+uBo6nEXK3M5/Ffh/FLpKExzOQ3JJoJGFKBw==", + "dev": true, + "license": "MIT" + }, + "node_modules/nanoid": { + "version": "3.3.19", + "resolved": "https://registry.npmjs.org/nanoid/-/nanoid-3.3.19.tgz", + "integrity": "sha512-Y2tUNy4ouw6tq5oDSKeQYGOyhkUBhNOcGV/02KC+6kd9eDGqdZd++mjMiIDilrBYvjEnCYvVtsuHCuP+okSfug==", + "dev": true, + "funding": [ + { + "type": "github", + "url": "https://github.com/sponsors/ai" + } + ], + "license": "MIT", + "bin": { + "nanoid": "bin/nanoid.cjs" + }, + "engines": { + "node": "^10 || ^12 || ^13.7 || ^14 || >=15.0.1" + } + }, + "node_modules/oniguruma-to-es": { + "version": "3.1.1", + "resolved": "https://registry.npmjs.org/oniguruma-to-es/-/oniguruma-to-es-3.1.1.tgz", + "integrity": "sha512-bUH8SDvPkH3ho3dvwJwfonjlQ4R80vjyvrU8YpxuROddv55vAEJrTuCuCVUhhsHbtlD9tGGbaNApGQckXhS8iQ==", + "dev": true, + "license": "MIT", + "dependencies": { + "emoji-regex-xs": "^1.0.0", + "regex": "^6.0.1", + "regex-recursion": "^6.0.2" + } + }, + "node_modules/perfect-debounce": { + "version": "1.0.0", + "resolved": "https://registry.npmjs.org/perfect-debounce/-/perfect-debounce-1.0.0.tgz", + "integrity": "sha512-xCy9V055GLEqoFaHoC1SoLIaLmWctgCUaBaWxDZ7/Zx4CTyX7cJQLJOok/orfjZAh9kEYpjJa4d0KcJmCbctZA==", + "dev": true, + "license": "MIT" + }, + "node_modules/picocolors": { + "version": "1.1.1", + "resolved": "https://registry.npmjs.org/picocolors/-/picocolors-1.1.1.tgz", + "integrity": "sha512-xceH2snhtb5M9liqDsmEw56le376mTZkEX/jEb/RxNFyegNul7eNslCXP9FDj/Lcu0X8KEyMceP2ntpaHrDEVA==", + "dev": true, + "license": "ISC" + }, + "node_modules/postcss": { + "version": "8.5.28", + "resolved": "https://registry.npmjs.org/postcss/-/postcss-8.5.28.tgz", + "integrity": "sha512-RRuzqDtt5Y9h3quz5hWhK+TPnsmVs6WwSU6LkJMeY4HstUEDuYTG8UJSdawMRzmzAtV+KEoG8N3Qg2qLy5vM/A==", + "dev": true, + "funding": [ + { + "type": "opencollective", + "url": "https://opencollective.com/postcss/" + }, + { + "type": "tidelift", + "url": "https://tidelift.com/funding/github/npm/postcss" + }, + { + "type": "github", + "url": "https://github.com/sponsors/ai" + } + ], + "license": "MIT", + "dependencies": { + "nanoid": "^3.3.18", + "picocolors": "^1.1.1", + "source-map-js": "^1.2.1" + }, + "engines": { + "node": "^10 || ^12 || >=14" + } + }, + "node_modules/preact": { + "version": "10.29.8", + "resolved": "https://registry.npmjs.org/preact/-/preact-10.29.8.tgz", + "integrity": "sha512-ej2aVZ+vZ8WO7tvlQWRM9N63A0KzF9q4mWJfDUHgYaIofWY9hu74QdnQrjoPMmZi2/nZ5gN0bJCQF49xQqx09Q==", + "dev": true, + "license": "MIT", + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/preact" + }, + "peerDependencies": { + "preact-render-to-string": ">=5" + }, + "peerDependenciesMeta": { + "preact-render-to-string": { + "optional": true + } + } + }, + "node_modules/property-information": { + "version": "7.2.0", + "resolved": "https://registry.npmjs.org/property-information/-/property-information-7.2.0.tgz", + "integrity": "sha512-IAtzIB6sUiWaJYrX9smp3V46pBGbBeLFRGdh25kg1334VcBlD8HzhPeNIWQH9zhGmo2itIe25EHt9dQP7G5hmg==", + "dev": true, + "license": "MIT", + "funding": { + "type": "github", + "url": "https://github.com/sponsors/wooorm" + } + }, + "node_modules/regex": { + "version": "6.1.0", + "resolved": "https://registry.npmjs.org/regex/-/regex-6.1.0.tgz", + "integrity": "sha512-6VwtthbV4o/7+OaAF9I5L5V3llLEsoPyq9P1JVXkedTP33c7MfCG0/5NOPcSJn0TzXcG9YUrR0gQSWioew3LDg==", + "dev": true, + "license": "MIT", + "dependencies": { + "regex-utilities": "^2.3.0" + } + }, + "node_modules/regex-recursion": { + "version": "6.0.2", + "resolved": "https://registry.npmjs.org/regex-recursion/-/regex-recursion-6.0.2.tgz", + "integrity": "sha512-0YCaSCq2VRIebiaUviZNs0cBz1kg5kVS2UKUfNIx8YVs1cN3AV7NTctO5FOKBA+UT2BPJIWZauYHPqJODG50cg==", + "dev": true, + "license": "MIT", + "dependencies": { + "regex-utilities": "^2.3.0" + } + }, + "node_modules/regex-utilities": { + "version": "2.3.0", + "resolved": "https://registry.npmjs.org/regex-utilities/-/regex-utilities-2.3.0.tgz", + "integrity": "sha512-8VhliFJAWRaUiVvREIiW2NXXTmHs4vMNnSzuJVhscgmGav3g9VDxLrQndI3dZZVVdp0ZO/5v0xmX516/7M9cng==", + "dev": true, + "license": "MIT" + }, + "node_modules/rfdc": { + "version": "1.4.1", + "resolved": "https://registry.npmjs.org/rfdc/-/rfdc-1.4.1.tgz", + "integrity": "sha512-q1b3N5QkRUWUl7iyylaaj3kOpIT0N2i9MqIEQXP73GVsN9cw3fdx8X63cEmWhJGi2PPCF23Ijp7ktmd39rawIA==", + "dev": true, + "license": "MIT" + }, + "node_modules/rollup": { + "version": "4.63.5", + "resolved": "https://registry.npmjs.org/rollup/-/rollup-4.63.5.tgz", + "integrity": "sha512-KRWwmNLlPw5M7HcdYfm15oBv9n9LPtjzpzCIxS/phwqvPyxHSoKX6Y2YU3pxSPfy0CLquVgsx/j/hBi6OvH1Nw==", + "dev": true, + "license": "MIT", + "dependencies": { + "@types/estree": "1.0.9" + }, + "bin": { + "rollup": "dist/bin/rollup" + }, + "engines": { + "node": ">=18.0.0", + "npm": ">=8.0.0" + }, + "optionalDependencies": { + "@napi-rs/lzma-linux-x64-gnu": "1.5.1", + "@rollup/rollup-android-arm-eabi": "4.63.5", + "@rollup/rollup-android-arm64": "4.63.5", + "@rollup/rollup-darwin-arm64": "4.63.5", + "@rollup/rollup-darwin-x64": "4.63.5", + "@rollup/rollup-freebsd-arm64": "4.63.5", + "@rollup/rollup-freebsd-x64": "4.63.5", + "@rollup/rollup-linux-arm-gnueabihf": "4.63.5", + "@rollup/rollup-linux-arm-musleabihf": "4.63.5", + "@rollup/rollup-linux-arm64-gnu": "4.63.5", + "@rollup/rollup-linux-arm64-musl": "4.63.5", + "@rollup/rollup-linux-loong64-gnu": "4.63.5", + "@rollup/rollup-linux-loong64-musl": "4.63.5", + "@rollup/rollup-linux-ppc64-gnu": "4.63.5", + "@rollup/rollup-linux-ppc64-musl": "4.63.5", + "@rollup/rollup-linux-riscv64-gnu": "4.63.5", + "@rollup/rollup-linux-riscv64-musl": "4.63.5", + "@rollup/rollup-linux-s390x-gnu": "4.63.5", + "@rollup/rollup-linux-x64-gnu": "4.63.5", + "@rollup/rollup-linux-x64-musl": "4.63.5", + "@rollup/rollup-openbsd-x64": "4.63.5", + "@rollup/rollup-openharmony-arm64": "4.63.5", + "@rollup/rollup-win32-arm64-msvc": "4.63.5", + "@rollup/rollup-win32-ia32-msvc": "4.63.5", + "@rollup/rollup-win32-x64-gnu": "4.63.5", + "@rollup/rollup-win32-x64-msvc": "4.63.5", + "fsevents": "~2.3.2" + } + }, + "node_modules/search-insights": { + "version": "2.17.3", + "resolved": "https://registry.npmjs.org/search-insights/-/search-insights-2.17.3.tgz", + "integrity": "sha512-RQPdCYTa8A68uM2jwxoY842xDhvx3E5LFL1LxvxCNMev4o5mLuokczhzjAgGwUZBAmOKZknArSxLKmXtIi2AxQ==", + "dev": true, + "license": "MIT", + "peer": true + }, + "node_modules/shiki": { + "version": "2.5.0", + "resolved": "https://registry.npmjs.org/shiki/-/shiki-2.5.0.tgz", + "integrity": "sha512-mI//trrsaiCIPsja5CNfsyNOqgAZUb6VpJA+340toL42UpzQlXpwRV9nch69X6gaUxrr9kaOOa6e3y3uAkGFxQ==", + "dev": true, + "license": "MIT", + "dependencies": { + "@shikijs/core": "2.5.0", + "@shikijs/engine-javascript": "2.5.0", + "@shikijs/engine-oniguruma": "2.5.0", + "@shikijs/langs": "2.5.0", + "@shikijs/themes": "2.5.0", + "@shikijs/types": "2.5.0", + "@shikijs/vscode-textmate": "^10.0.2", + "@types/hast": "^3.0.4" + } + }, + "node_modules/source-map-js": { + "version": "1.2.1", + "resolved": "https://registry.npmjs.org/source-map-js/-/source-map-js-1.2.1.tgz", + "integrity": "sha512-UXWMKhLOwVKb728IUtQPXxfYU+usdybtUrK/8uGE8CQMvrhOpwvzDBwj0QhSL7MQc7vIsISBG8VQ8+IDQxpfQA==", + "dev": true, + "license": "BSD-3-Clause", + "engines": { + "node": ">=0.10.0" + } + }, + "node_modules/space-separated-tokens": { + "version": "2.0.2", + "resolved": "https://registry.npmjs.org/space-separated-tokens/-/space-separated-tokens-2.0.2.tgz", + "integrity": "sha512-PEGlAwrG8yXGXRjW32fGbg66JAlOAwbObuqVoJpv/mRgoWDQfgH1wDPvtzWyUSNAXBGSk8h755YDbbcEy3SH2Q==", + "dev": true, + "license": "MIT", + "funding": { + "type": "github", + "url": "https://github.com/sponsors/wooorm" + } + }, + "node_modules/speakingurl": { + "version": "14.0.1", + "resolved": "https://registry.npmjs.org/speakingurl/-/speakingurl-14.0.1.tgz", + "integrity": "sha512-1POYv7uv2gXoyGFpBCmpDVSNV74IfsWlDW216UPjbWufNf+bSU6GdbDsxdcxtfwb4xlI3yxzOTKClUosxARYrQ==", + "dev": true, + "license": "BSD-3-Clause", + "engines": { + "node": ">=0.10.0" + } + }, + "node_modules/stringify-entities": { + "version": "4.0.4", + "resolved": "https://registry.npmjs.org/stringify-entities/-/stringify-entities-4.0.4.tgz", + "integrity": "sha512-IwfBptatlO+QCJUo19AqvrPNqlVMpW9YEL2LIVY+Rpv2qsjCGxaDLNRgeGsQWJhfItebuJhsGSLjaBbNSQ+ieg==", + "dev": true, + "license": "MIT", + "dependencies": { + "character-entities-html4": "^2.0.0", + "character-entities-legacy": "^3.0.0" + }, + "funding": { + "type": "github", + "url": "https://github.com/sponsors/wooorm" + } + }, + "node_modules/superjson": { + "version": "2.2.6", + "resolved": "https://registry.npmjs.org/superjson/-/superjson-2.2.6.tgz", + "integrity": "sha512-H+ue8Zo4vJmV2nRjpx86P35lzwDT3nItnIsocgumgr0hHMQ+ZGq5vrERg9kJBo5AWGmxZDhzDo+WVIJqkB0cGA==", + "dev": true, + "license": "MIT", + "dependencies": { + "copy-anything": "^4" + }, + "engines": { + "node": ">=16" + } + }, + "node_modules/tabbable": { + "version": "6.5.0", + "resolved": "https://registry.npmjs.org/tabbable/-/tabbable-6.5.0.tgz", + "integrity": "sha512-wieBHXygIm7OyQOu5hQlkk62/WyCFYGlWg7L6/ZCUZwx0o398Zkn4pVmMyfYhfMG8kGrj/Krt8eIk6UKC6VzwA==", + "dev": true, + "license": "MIT" + }, + "node_modules/trim-lines": { + "version": "3.0.1", + "resolved": "https://registry.npmjs.org/trim-lines/-/trim-lines-3.0.1.tgz", + "integrity": "sha512-kRj8B+YHZCc9kQYdWfJB2/oUl9rA99qbowYYBtr4ui4mZyAQ2JpvVBd/6U2YloATfqBhBTSMhTpgBHtU0Mf3Rg==", + "dev": true, + "license": "MIT", + "funding": { + "type": "github", + "url": "https://github.com/sponsors/wooorm" + } + }, + "node_modules/unist-util-is": { + "version": "6.0.1", + "resolved": "https://registry.npmjs.org/unist-util-is/-/unist-util-is-6.0.1.tgz", + "integrity": "sha512-LsiILbtBETkDz8I9p1dQ0uyRUWuaQzd/cuEeS1hoRSyW5E5XGmTzlwY1OrNzzakGowI9Dr/I8HVaw4hTtnxy8g==", + "dev": true, + "license": "MIT", + "dependencies": { + "@types/unist": "^3.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/unified" + } + }, + "node_modules/unist-util-position": { + "version": "5.0.0", + "resolved": "https://registry.npmjs.org/unist-util-position/-/unist-util-position-5.0.0.tgz", + "integrity": "sha512-fucsC7HjXvkB5R3kTCO7kUjRdrS0BJt3M/FPxmHMBOm8JQi2BsHAHFsy27E0EolP8rp0NzXsJ+jNPyDWvOJZPA==", + "dev": true, + "license": "MIT", + "dependencies": { + "@types/unist": "^3.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/unified" + } + }, + "node_modules/unist-util-stringify-position": { + "version": "4.0.0", + "resolved": "https://registry.npmjs.org/unist-util-stringify-position/-/unist-util-stringify-position-4.0.0.tgz", + "integrity": "sha512-0ASV06AAoKCDkS2+xw5RXJywruurpbC4JZSm7nr7MOt1ojAzvyyaO+UxZf18j8FCF6kmzCZKcAgN/yu2gm2XgQ==", + "dev": true, + "license": "MIT", + "dependencies": { + "@types/unist": "^3.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/unified" + } + }, + "node_modules/unist-util-visit": { + "version": "5.1.0", + "resolved": "https://registry.npmjs.org/unist-util-visit/-/unist-util-visit-5.1.0.tgz", + "integrity": "sha512-m+vIdyeCOpdr/QeQCu2EzxX/ohgS8KbnPDgFni4dQsfSCtpz8UqDyY5GjRru8PDKuYn7Fq19j1CQ+nJSsGKOzg==", + "dev": true, + "license": "MIT", + "dependencies": { + "@types/unist": "^3.0.0", + "unist-util-is": "^6.0.0", + "unist-util-visit-parents": "^6.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/unified" + } + }, + "node_modules/unist-util-visit-parents": { + "version": "6.0.2", + "resolved": "https://registry.npmjs.org/unist-util-visit-parents/-/unist-util-visit-parents-6.0.2.tgz", + "integrity": "sha512-goh1s1TBrqSqukSc8wrjwWhL0hiJxgA8m4kFxGlQ+8FYQ3C/m11FcTs4YYem7V664AhHVvgoQLk890Ssdsr2IQ==", + "dev": true, + "license": "MIT", + "dependencies": { + "@types/unist": "^3.0.0", + "unist-util-is": "^6.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/unified" + } + }, + "node_modules/vfile": { + "version": "6.0.3", + "resolved": "https://registry.npmjs.org/vfile/-/vfile-6.0.3.tgz", + "integrity": "sha512-KzIbH/9tXat2u30jf+smMwFCsno4wHVdNmzFyL+T/L3UGqqk6JKfVqOFOZEpZSHADH1k40ab6NUIXZq422ov3Q==", + "dev": true, + "license": "MIT", + "dependencies": { + "@types/unist": "^3.0.0", + "vfile-message": "^4.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/unified" + } + }, + "node_modules/vfile-message": { + "version": "4.0.3", + "resolved": "https://registry.npmjs.org/vfile-message/-/vfile-message-4.0.3.tgz", + "integrity": "sha512-QTHzsGd1EhbZs4AsQ20JX1rC3cOlt/IWJruk893DfLRr57lcnOeMaWG4K0JrRta4mIJZKth2Au3mM3u03/JWKw==", + "dev": true, + "license": "MIT", + "dependencies": { + "@types/unist": "^3.0.0", + "unist-util-stringify-position": "^4.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/unified" + } + }, + "node_modules/vite": { + "version": "5.4.21", + "resolved": "https://registry.npmjs.org/vite/-/vite-5.4.21.tgz", + "integrity": "sha512-o5a9xKjbtuhY6Bi5S3+HvbRERmouabWbyUcpXXUA1u+GNUKoROi9byOJ8M0nHbHYHkYICiMlqxkg1KkYmm25Sw==", + "dev": true, + "license": "MIT", + "dependencies": { + "esbuild": "^0.21.3", + "postcss": "^8.4.43", + "rollup": "^4.20.0" + }, + "bin": { + "vite": "bin/vite.js" + }, + "engines": { + "node": "^18.0.0 || >=20.0.0" + }, + "funding": { + "url": "https://github.com/vitejs/vite?sponsor=1" + }, + "optionalDependencies": { + "fsevents": "~2.3.3" + }, + "peerDependencies": { + "@types/node": "^18.0.0 || >=20.0.0", + "less": "*", + "lightningcss": "^1.21.0", + "sass": "*", + "sass-embedded": "*", + "stylus": "*", + "sugarss": "*", + "terser": "^5.4.0" + }, + "peerDependenciesMeta": { + "@types/node": { + "optional": true + }, + "less": { + "optional": true + }, + "lightningcss": { + "optional": true + }, + "sass": { + "optional": true + }, + "sass-embedded": { + "optional": true + }, + "stylus": { + "optional": true + }, + "sugarss": { + "optional": true + }, + "terser": { + "optional": true + } + } + }, + "node_modules/vitepress": { + "version": "1.6.4", + "resolved": "https://registry.npmjs.org/vitepress/-/vitepress-1.6.4.tgz", + "integrity": "sha512-+2ym1/+0VVrbhNyRoFFesVvBvHAVMZMK0rw60E3X/5349M1GuVdKeazuksqopEdvkKwKGs21Q729jX81/bkBJg==", + "dev": true, + "license": "MIT", + "dependencies": { + "@docsearch/css": "3.8.2", + "@docsearch/js": "3.8.2", + "@iconify-json/simple-icons": "^1.2.21", + "@shikijs/core": "^2.1.0", + "@shikijs/transformers": "^2.1.0", + "@shikijs/types": "^2.1.0", + "@types/markdown-it": "^14.1.2", + "@vitejs/plugin-vue": "^5.2.1", + "@vue/devtools-api": "^7.7.0", + "@vue/shared": "^3.5.13", + "@vueuse/core": "^12.4.0", + "@vueuse/integrations": "^12.4.0", + "focus-trap": "^7.6.4", + "mark.js": "8.11.1", + "minisearch": "^7.1.1", + "shiki": "^2.1.0", + "vite": "^5.4.14", + "vue": "^3.5.13" + }, + "bin": { + "vitepress": "bin/vitepress.js" + }, + "peerDependencies": { + "markdown-it-mathjax3": "^4", + "postcss": "^8" + }, + "peerDependenciesMeta": { + "markdown-it-mathjax3": { + "optional": true + }, + "postcss": { + "optional": true + } + } + }, + "node_modules/vue": { + "version": "3.5.43", + "resolved": "https://registry.npmjs.org/vue/-/vue-3.5.43.tgz", + "integrity": "sha512-o5qZoksdnjIKvW1srZ3ab7pcDNYAerBjRe54D0LBLfRdCYFrSgBHVXokMas35czQc0//lmx4/tuY4ZNQ+Rf2Ng==", + "dev": true, + "license": "MIT", + "dependencies": { + "@vue/compiler-dom": "3.5.43", + "@vue/compiler-sfc": "3.5.43", + "@vue/runtime-dom": "3.5.43", + "@vue/server-renderer": "3.5.43", + "@vue/shared": "3.5.43" + }, + "peerDependencies": { + "typescript": "*" + }, + "peerDependenciesMeta": { + "typescript": { + "optional": true + } + } + }, + "node_modules/zwitch": { + "version": "2.0.4", + "resolved": "https://registry.npmjs.org/zwitch/-/zwitch-2.0.4.tgz", + "integrity": "sha512-bXE4cR/kVZhKZX/RjPEflHaKVhUVl85noU3v6b8apfQEc1x4A+zBxjZ4lN8LqGd6WZ3dl98pY4o717VFmoPp+A==", + "dev": true, + "license": "MIT", + "funding": { + "type": "github", + "url": "https://github.com/sponsors/wooorm" + } + } + } +} diff --git a/package.json b/package.json new file mode 100644 index 0000000..fcf55ee --- /dev/null +++ b/package.json @@ -0,0 +1,15 @@ +{ + "name": "decguard-docs", + "private": true, + "description": "Documentation site for DecGuard (VitePress).", + "license": "Apache-2.0", + "type": "module", + "scripts": { + "docs:dev": "vitepress docs", + "docs:build": "vitepress build docs", + "docs:preview": "vitepress preview docs" + }, + "devDependencies": { + "vitepress": "1.6.4" + } +} diff --git a/src/decguard/backends/http.py b/src/decguard/backends/http.py index cfecfaf..7397bf7 100644 --- a/src/decguard/backends/http.py +++ b/src/decguard/backends/http.py @@ -1,6 +1,6 @@ """Generic HTTP backend for decision endpoints (System One APIs: see ``systemone.py``). -Protocol ``decguard.http/0.1`` (see docs/backends.md):: +Protocol ``decguard.http/0.1`` (see docs/http.md):: POST {"protocol": "decguard.http/0.1", "case_id": "...", "model": "..." | null, diff --git a/src/decguard/backends/python.py b/src/decguard/backends/python.py index 0299e4e..9d87f29 100644 --- a/src/decguard/backends/python.py +++ b/src/decguard/backends/python.py @@ -14,7 +14,7 @@ class CallableBackend(DecisionBackend): """Backend backed by ``fn(request) -> {label: probability}`` (or a ``Prediction``). Contracts cannot reference arbitrary Python code; to use a Python provider from the CLI, - package it as a ``decguard.backends`` entry point (see docs/backends.md). + package it as a ``decguard.backends`` entry point (see docs/custom-backends.md). """ provider = "python" diff --git a/src/decguard/metrics/core.py b/src/decguard/metrics/core.py index 0ebf58c..7a6c96c 100644 --- a/src/decguard/metrics/core.py +++ b/src/decguard/metrics/core.py @@ -3,7 +3,7 @@ Kept dependency-free and deterministic: sums use :func:`math.fsum` over inputs in their given order, so the same results always produce bit-identical metrics. Definitions follow the usual conventions (scikit-learn for F1/NLL, Guo et al. 2017 for ECE); see -docs/reports.md. +docs/metrics.md. """ from __future__ import annotations diff --git a/tests/integration/real/choice.yaml b/tests/integration/real/choice.yaml index d600bc6..975a0f6 100644 --- a/tests/integration/real/choice.yaml +++ b/tests/integration/real/choice.yaml @@ -1,4 +1,4 @@ -# Real-backend compatibility check (pytest -m real_kev / -m real_jev; docs/backends.md). +# Real-backend compatibility check (pytest -m real_kev / -m real_jev; docs/systemone.md). # The default backend is a local Kev server; `jev` is TypeSafe's Jev on OpenRouter. schema_version: "0.1" diff --git a/tests/integration/real/noul.yaml b/tests/integration/real/noul.yaml index 897731b..19fbf14 100644 --- a/tests/integration/real/noul.yaml +++ b/tests/integration/real/noul.yaml @@ -1,4 +1,4 @@ -# Real-backend compatibility check (pytest -m real_kev / -m real_jev; docs/backends.md). +# Real-backend compatibility check (pytest -m real_kev / -m real_jev; docs/systemone.md). # The default backend is a local Kev server; `jev` is TypeSafe's Jev on OpenRouter. schema_version: "0.1" diff --git a/tests/integration/real/score.yaml b/tests/integration/real/score.yaml index 0c48591..35c6495 100644 --- a/tests/integration/real/score.yaml +++ b/tests/integration/real/score.yaml @@ -1,4 +1,4 @@ -# Real-backend compatibility check (pytest -m real_kev / -m real_jev; docs/backends.md). +# Real-backend compatibility check (pytest -m real_kev / -m real_jev; docs/systemone.md). # The default backend is a local Kev server; `jev` is TypeSafe's Jev on OpenRouter. schema_version: "0.1" diff --git a/tests/integration/test_real_backends.py b/tests/integration/test_real_backends.py index 0dbb4a3..b3c4010 100644 --- a/tests/integration/test_real_backends.py +++ b/tests/integration/test_real_backends.py @@ -1,6 +1,6 @@ """Opt-in end-to-end checks against real System One backends. Nothing here is mocked. - uv run pytest -m real_kev # needs a Kev server on 127.0.0.1:8009 (docs/backends.md) + uv run pytest -m real_kev # needs a Kev server on 127.0.0.1:8009 (docs/systemone.md) uv run pytest -m real_jev # needs OPENROUTER_API_KEY; three billed Jev calls Without ``-m`` these tests are skipped. Once selected with ``-m``, a missing server,