Distributed AI evaluation infrastructure and deployment gating experiments for generative AI systems.
Agent Vigilo explores what LLM and agent evaluation infrastructure can look like beyond ad hoc scripts: versioned WASM evaluators, durable evaluation runs, worker/coordinator execution, normalized results, and pass/fail gates that can sit in CI or release workflows.
It focuses on the parts of AI evaluation that become hard as systems grow: idempotent distributed work, durable event delivery, evaluator isolation, retry-safe persistence, and auditable results.
- Run evaluations like infrastructure: PostgreSQL-backed state, RabbitMQ work distribution, Rust workers, and deterministic state guards.
- Ship versioned evaluators: publish WASI Preview 2 WebAssembly evaluators with strict WIT contracts.
- Protect the runtime: Wasmtime fuel, memory, timeout, log, and concurrency limits isolate evaluator execution.
- Avoid lost events: durable outbox ledger plus hot delivery queue, RabbitMQ publisher confirms, and idempotency keys.
- Gate deployments: turn evaluator measurements and host-owned profile policy into dimension scores, total aggregate scores, and reproducible release decisions.
Each evaluator invocation returns one measurement or abstention plus optional diagnostics. The profile binding owns normalization, threshold, dimension, weight, requiredness, and blocking policy. Missing, errored, abstained, duplicated, or invalid required output produces an errored execution aggregate with no score. An execution passes when completeness is satisfied, aggregate_score >= min_execution_score, and no host-derived blocking result fails.
Run scorecard gates enforce dimension or evaluator thresholds across the full run, including configured case-group and tag slices. Shards publish bounded rollups; the coordinator merges those rollups and persists the authoritative scorecard without scanning execution rows centrally.
See Measurement Normalization for copyable binary, numeric, curve, threshold, and ordinal policy examples.
A run can fail operationally because work did not complete, or complete with a failed gate because evaluation policy failed.
- Getting started: run your first evaluation.
- CLI reference: canonical commands for evaluators, runs, databases, and rebalancing.
- Architecture overview: containers, components, flows, and state diagrams.
- Scale-out and shard migration: 128 logical run shards and expansion guidance.
- Worker runtime: chunk claiming, evaluator execution, and result persistence.
- Runtime limits: Wasm evaluator sandbox and worker concurrency controls.
- Outbox lifecycle: durable event publication and retry behavior.
- Publishing evaluators: build and publish versioned WASM evaluators.
- Evaluator compatibility: versioned WIT ABIs, host adapters, and immutable run pinning.
Rust, Tokio, PostgreSQL, SQLx, RabbitMQ, Wasmtime, WASI Preview 2, WIT, Docusaurus.
GitHub Actions is the source of truth for build verification. To install the optional local Git hooks:
chmod +x scripts/hooks/pre-commit scripts/hooks/pre-push
git config core.hooksPath scripts/hooks| Tier | Scope | Required services | GitHub Actions job |
|---|---|---|---|
| Unit and contract | Service-free Rust behavior across the workspace | None | Unit and Contract Tests |
| Database integration | SQL, migrations, transactions, advisory locks, leases, and concurrency | One PostgreSQL server; DATABASE_URL must use a role that can create test databases |
Database Integration Tests |
| Migration | Greenfield schema application through the CLI setup path | One fresh PostgreSQL database | Migration Tests |
| End-to-end | Routing and the real distributed runtime across process and protocol boundaries | Two PostgreSQL servers, RabbitMQ, evaluator Wasm, and the test HTTP agent | End-to-End Tests |
| Performance services | Perf topology ownership, collector reset, and cleanup integration | Docker Engine with Compose; images are declared in performance/compose.yml |
Opt-in local tier |
Run the service-free tier with:
cargo test --workspace --locked --lib --binsThe repository-local performance harness is a separate opt-in workflow. Its static check remains service-free and validates that performance tooling stays out of the shipped dependency graph:
cargo perf checkThe explicit live-service integration tier is:
cargo test -p xtask --locked --features performance-services --test performance_services -- --nocaptureIt is feature-gated and therefore does not expand normal workspace, database,
migration, or end-to-end test commands. Build, workload, artifact, and
isolation details are in performance/README.md.
After setting DATABASE_URL, run every PostgreSQL-backed SQLx test with:
cargo test -p vigilo --locked --bin vigilo -- --ignored --nocapture --test-threads=4Within the vigilo binary target, #[ignore] is reserved for this database
integration tier so the command remains complete.
The end-to-end tier runs multi_database_routing and multi_database_e2e
separately because they require the complete distributed dependency set. The
workflow builds the evaluator Wasm and supplies the required database,
messaging, and opt-in environment variables.
The pre-commit hook runs nightly rustfmt only. The pre-push hook runs clippy, the service-free Rust tier, and the web typecheck. Database integration, migration, end-to-end, evaluator Wasm, and web production build checks run as separate required CI jobs.
The service-free, PostgreSQL, and end-to-end jobs collect coverage in parallel
while running their existing test tiers. A final job merges the three reports,
requires at least 80% aggregate line coverage, and uploads that report under the
rust-all Codecov flag. Project coverage and the repository badge therefore
reflect code reached through unit, integration, and distributed runtime tests.
Only PostgreSQL test fixture sources are excluded; production query and table
modules remain in the coverage denominator.
SQLx creates each isolated PostgreSQL test database with CREATE DATABASE,
which clones PostgreSQL's built-in template1 database. The PostgreSQL coverage
job migrates template1 once, so each test validates the recorded migrations
instead of rebuilding every partition and index. The separate migration job
still applies the complete migration set to an empty database.
Accumulate the same reports locally after starting the services and setting the environment variables required by each tier:
cargo llvm-cov clean --workspace
cargo llvm-cov --no-report --workspace --locked --lib --bins
cargo llvm-cov --no-report -p vigilo --locked --bin vigilo -- --ignored --nocapture --test-threads=4
cargo llvm-cov --no-report -p vigilo --locked --test multi_database_routing -- --nocapture
VIGILO_E2E_MULTI_DATABASE=1 cargo llvm-cov --no-report -p vigilo --locked --test multi_database_e2e -- --nocapture
cargo llvm-cov report --fail-under-lines 80 --ignore-filename-regex '[/\\]postgres_tests(\.rs|[/\\])' --lcov --output-path lcov.infoThe integration commands require the database, messaging, evaluator Wasm, and
test agent dependencies described above. Configure the repository secret
CODECOV_TOKEN for the aggregate CI upload.
Agent Vigilo is an active systems project focused on reliable AI evaluation, LLM evaluation workflows, agent testing, and deployment gates. The implementation favors explicit contracts, durable state transitions, and operational diagrams over black-box orchestration.